Exactly. If you don't want those people to know you use Signal you don't have to. Look up the recent update to phone number privacy. In fact I just took this screenshot: 
Discussion
Nah, not that either. You're still missing my point. If a contact (friend, family, etc) outside of Signal has your phone number saved with your real name, and you use that same phone number for your Signal account, and your friend/family use a service that uploads their contact data with your real name/number in it, then they have effectively doxxed you as the owner of the Signal account to three-letter agencies.
What you're describing is the simple concept of a social graph. Those three letter agencies could gain a list of contacts for who knows who from services like Google, but if you have PNP in Signal set in such a way where discoverability is set to no one, how would even the three-letter agencies know that you're using Signal?