Its Nostr, if you unfollow they'll never know, and if they somehow do find out, just blame it on the client. You can't lose.