Some nostr client should include the ability to leave a note as to why you unfollowed an npub. Or just tag the note that made you say "nah"