Did you read Rothbard on the subject of private law and justice? For a New Liberty is great.

Or The Enterprise of Law by Bruce Benson.

I think Chaos Theory by Bob Murphy and maybe the Machinery of Freedom also hit on it.

There are some pretty developed theories around it. It seems to me, without the coercive state, many HOA type opt-in communities would have somewhat differing methods, but all handle criminals more justly than our system.