No, you don't understand what I'm saying. I'm not talking about your Signal contacts. I'm talking about other people who have your phone number saved in their contacts (the one people use to make phone calls from, not from Signal) and have used other services that have uploaded those contacts.
Discussion
Exactly. If you don't want those people to know you use Signal you don't have to. Look up the recent update to phone number privacy. In fact I just took this screenshot: 
Nah, not that either. You're still missing my point. If a contact (friend, family, etc) outside of Signal has your phone number saved with your real name, and you use that same phone number for your Signal account, and your friend/family use a service that uploads their contact data with your real name/number in it, then they have effectively doxxed you as the owner of the Signal account to three-letter agencies.
What you're describing is the simple concept of a social graph. Those three letter agencies could gain a list of contacts for who knows who from services like Google, but if you have PNP in Signal set in such a way where discoverability is set to no one, how would even the three-letter agencies know that you're using Signal?