Yes this has to be designed better to be intuitive with the privacy aspects.

If you start a chat with someone, nobody knows about it.

But adding someone to your contact list is the same as your follow list in other Nostr clients, so those are inherently public.