I'm not talking to you through a "server". What you're seeing is a Nostr bridge that acts like a Fediverse server. I'm not a "member" of that server (momostr.pink), nor any server.
Nostr is a different protocol / architecture. Rather than storing your identity and social graph on some particular server, like Mastodon, identities are simply public/private key pairs, independent of any one server or operator.
Posts and other content are shared by "relays", which only exist to pass notes around. This makes censorship effectively impossible, because you can generally always find a relay that will carry your note.
Thus, there's nothing like the situation you describe, where one disgruntled user can get you "banned" for an entire section of the user base.