Could just move to support resolution of https://handshake.org/ domains, you can buy one on https://www.namebase.io/ or even on namecheap if you are ok with an extension they own (namebase by contrast lets you get your own extension).
There are already systems to prevent squatting (automatic Vickrey auction for new registrations, etc.) and these work well enough. The NIP-05 system would remain the same, still pop a json same place, just allow for resolution of handshake. So it wouldn't break the design of the current NIP-05 system.
As for bitcoin there are some that at first glance seem to be like what you mean, like https://www.namecoin.org/dot-bit/ , but these actually are forks of bitcoin and don't require any bitcoin ownership, so it's the same concept as handshake more or less (another chain), but handshake is the one with traction and ease.