The hardware is only as secure as the software controlling it. What ledger is doing is plain stupid. An interesting counter approach is what Block/Bitkey is doing to never let the key seed leave the hardware and rely on multisig for backup/recovery.