Well you aren't really giving Alby to your clients. You allow Alby to sign the events "offline" and then it sends the signed event to the client so your key is never directly exposed to the client. You can have Alby "forget" which clients you want it to sign for.

nos2x might be a better solution for you, if you really want to be extra super careful. I think you can even use coldcard and seedsigner hardware wallets if you want to go full tinfoil hat.

(note: I think I might incorrect in my previous post. I don't think Alby has any access to your nsec private key and it is only stored in your personal browser. I will need to double check, or maybe someone can correct me)