this is a painful rabbithole you are diving into there

we have to define our threat model, and you must not think outside of it for reasons you are experiencing

we have to trust our server's physical security, for example, or otherwise we have to have physical hardening on our servers, which is a great increase in cost

it can be mitigated by making encryption schemes that defeat physical breaches, but there is limits to how strong you can make this security, especially with scale, cryptography gets astronomically expensive at scale, the math is absurdly expensive compared to simple ordinary computations, the overflows and so forth involved tend to be in the dozens if not hundreds of cycles per operation