I like that take and I certainly prefer the simplicity of single sig + passphrase over multisig.
However, while you can probably avoid a $5 wrench attack by using multiple passphrases, there are still a couple of things I worry about with single sig.
1) Bad entropy in the random number generator of your chosen hardware wallet
2) Some type of manufacturer attack. They 'preprogrammed' the hardware wallets to generate known seeds and will one day 'retirement' attack all of their wallets by sweeping everything.
3) Malicious firmware update
4) Supply chain attack on your hardware wallet while enroute to you
I think most of these attacks can be mitigated by using a 2/3 multi-vendor multisig. But I'm not sure any of them can be mitigated with confidence using a singlesig + passphrase.
Perhaps these are extremely low probability that they aren't worth worrying about or perhaps there is a way to mitigate using singlesig I'm not aware of...