Yeah I forgot about the hardware itself.

For some HW devices it's possible to extract the seed from it, so without a pass phrase you have multiple ways of being owned (HW compromised, or paper / metal backup stolen).