This sounds great in theory and I have the same instinct, but I can't pretend there isn't plenty of historical evidence that technological advances that makes weapons more accessible has often destroyed large civilizations, and it is not as rosey as decentralisation of power, it often results in a devastating retardation of societies and obviously their standards of living.
Out of the top of my head, Iron seems to have contributed to the Bronze age collapsing where pottery and writing (things we can observe and use as proxy for quality of life) basically disappeared for a generation or two.
Roman empire was brutal, especially against the periphery and the "barbarians", but more often than not these barbarians were just tribes attracted to the standard of living of the empire and want some of it.
Once the empire fell, many farmers had a great time as they didn't have to pay taxes for a while, but what followed was called a dark age... And rightly so, plenty of Roman society's sophistication and quality of life disappeared... Certainly things got more violent.
So I wonder if we shouldn't wish for wilderness, instead just try to build many smaller sovereign states and cities .. but we can't just pretend we don't need functional security systems at scale.
It sucks to admit that, but I am geographically and culturally too close to too many societies absolutely devastated not by a totalitarian government but by a dysfunctional one + rampant violence, war lords + external investors in such violence.
So I don't know how to express this position between anarchist radical individualism and mindless collectivism... But there is a minimum viable organisational size for life to be worth living... I am afraid