I know exactly what you mean but perhaps just go on a muting and blocking mayhem rather than leaving? I do believe that decentralised social media platforms like nostr have a future but yes, people will always form groups, seek leaders and eventually try to force their will on the "outsiders".
I've read a study on Tribalism a good year ago and it was an eye opener. Look, we divide ourselves by country of origin, colour of skin, religion and more in the legacy world. In cryptography we now divide ourselves by Bitcoiner, Shitcoiner, Altcoiner, Nocoiner, Newcoiner etc.
Eventually always resulting in one outcome. Discrimination of the border groups resulting in weakening our unity as people. It's absurd but works well for those who hold the power and control the crowd.
If we could step away from our tribal desires we could unite and change the world. It's absurd isn't it?