Yes it can happen to company for sure. My point is there is always trust. You choose open source because you trust a bigger scale of people to verify for you which is totally reasonable.
There are differences between what hardware wallet does and what security chip does to a hardware wallet. You of course can have fully open sourced hardware wallet that’s nothing wrong with it but be aware you do not have security protection against physical hack. Security can not be open sourced, I don’t know how to properly explain it, may be it is like the lock to your home that you will never open source your key pattern to people on the street. But you know even if the lock manufacturer had a copy of the key pattern they never know where you home is (if you bought with cash at store for example).