But the private keys are held in the same piece of hardware not unlike a Yubikey right?

So much like a physical key, if you can grab the device, you have the universal password.

PINs provide some protection in this scenario but only if they're secure. Someone using an insecure password is likely to put their birthday, or something else equally easy to guess, as their PIN.

Biometrics fixes the above but creates a whole new rabbit hole of privacy violation for obvious reasons.