So right now you're running on general-purpose hardware, and trusting it to manage your private key.
If you were to use a hardware wallet, you would transfer that trust to the hardware wallet. You wouldn't be adding a trusted party, you would replacing one with the other.