My biggest worry is not even about AI or fiat or war or whatever. It's really an abstract worry about the human predilection for simple, concise, models and rules to explain and govern the world. When what we've done is built a society of such complexity, that there's no stable set of parameters within the limits of human cognition that is stable.

Some libertarians and anarchists think that formulation is the non-aggression principle, self-ownership and homesteading. Some progressives think it's about taxing the rich and deconstructing all aspects of culture and society. Religious conservatives think it's a set of divine rules.

I happen to think they're all wrong, and for similar epistemic reasons, setting aside the very fun metaphysical assumptions creating the various axiomatic groundings for this normative systems of mediating social conflict.