They basically have “view only” privileges with your npub.
Your nsec (which you should always keep secret) is what “signs” or gives your npub permission to do things like follow other users or post notes.
At least, I’m 95% sure that “following” another account requires a signed event.
Yes, your contact list (list of npubs you follow) is an event type itself, so would need to be signed. Each individual follow is just an update to that list/event, that then gets relayed out so other clients replicate the list, I believe. NIP-02 defined this: https://github.com/nostr-protocol/nips/blob/master/02.md
