What do you mean by defamation? From my quick research, it requires proving that someone did the following four things:

1. Made a false statement purporting to be fact;

2. Published or communicated that statement to at least one third party;

3. Attributed fault (usually negligence, or “actual malice” for public figures); and

4. Caused harm (e.g. injury to reputation, emotional distress, pecuniary loss)

We can certainly argue that the existing implementations (i.e. state court systems) do not handle such cases well

I think in a free society where disputes were resolved in the absence of state court systems, some form of defamation laws would likely form. If I were a restaurant owner, I would want some form of “defamation insurance” to cover any potential loss of income from situations such as a competitor creating false bad reviews about my business to drive my customers away, to his restaurant. Most businesses would probably want such insurance (and if such events were unlikely, the insurance would be very cheap). And in such an insurance contract, insurance companies would want to include a clause stating that the insured business is subject to penalties if they are caught defaming other businesses. Any business that did not purchase such insurance would eventually start to look very suspicious.