To be clear, knowing that the chain code is in the xpub and it is one you supplied yourself. This is how you would know it isn't malicious.
Discussion
Yes, of you provide a chain code yourself and it has enough entropy, then it works somewhat. But it's not better than a 2-of-2 between the device and the software wallet in terms of security model, and I think the latter is easier to analyze.