They basically have “view only” privileges with your npub.

Your nsec (which you should always keep secret) is what “signs” or gives your npub permission to do things like follow other users or post notes.

At least, I’m 95% sure that “following” another account requires a signed event.