Are Ledger hardware wallets really that bad for storing your bitcoin?

Well, yes and no: the good news is that the wallet is still more secure than the average phone/computer wallet setup. Also, the Ledger Recover feature is completely optional and allows users to create their own setup to potentially avoid single points of failure.

But Ledger’s main drawback is that you can’t verify it and compromising the device might be possible via firmware update.