Don’t put your cold storage in it; that’s not what it’s for.
And you can symmetrically encrypt with a pass phrase so that a client needs your explicit authorization beyond access to your nsec
Ultimately, as in anything that frees individuals, is up to each person to choose how to use the tools.