That's correct. If it's a high entropy passphrase. My pass phrase is around 300 bits of entropy. Excluding the hard coded hashing function. Which makes the entropy even greater.
Trezor does not store the passphrase in the HW. So it can't be hacked. But If you lose the passphrase at 256-300 bits of entropy you're screwed..๐๐