I’m not disagreeing with your point necessarily, but in reality every society has its own “rules”.
These certainly shouldn’t form some arbitrary social contract that you or anyone must adhere to, but the principle of “when in Rome” tends to be quite useful to keep things smooth.
For example, we drive on the left in Aus. Our brains are wired to look right then left because of this. We’re wired to stand on the left of an escalator walk on the left side of a path past someone else.
These are “rules” that we all accept, from local communities through to the nation.
Now throw in a bunch of Asians who drive on the right, whose traffic was built for motorbikes not cars, and whose idea of personal space isn’t the same as an Australian’s - now you’ve got disharmony because people came to Rome and wanted to live like they’re still in Delhi or Hanoi.
Everything should stem from natural rights, and communities should govern themselves. The modern challenge is that ideological and moral lenses are applied at too high a level such that there is constant disharmony and rather than seperate so people can maintain their identities, we just get smushed together more which makes it worse.