That clears my mind indeed. I tend to overthink too much.

So Sparrow has access to the public keys, and the hardware wallet to the private keys.