The hash and signatures are based on the JSON structures and string encodings (hex, base64, bech32, etc) instead of the raw binary. Which means, even if you do run everything in binary, you end up going back to the text protocol to do everything.

But one could create a parallel spec with binary payloads in mind. Clients could offer both as long as they use the same crypto primitives.