The same way it's none of anyones business what me and someone else are talking about, it is none of their business who I'm talking to. That's why I would prefer to hide the social graph.

About assigning ID vs ID to a connection...you can use one-time connection IDs similar to stealth addresses on Monero iirc

I get your point about the risk of talking to people you don't know IRL and compromised channels. But they still wouldn't be able to know who else you are talking to, or you want to use it to talk to people you know IRL, it still matters.