I agree with you that you have to opt-in to use that feature. But with Ledger being closed source, I have to trust their word without verifying.

The point of cold storage in my opinion was that the keys should never leave the device. I just don't want another possible attack vector to worry about.