You should be fine, that's exactly the point of using signer extensions: You don't give your private key to 3rd party websites, but websites instead request signatures from an extension.