Yeah the term "wallet" is a misnomer that causes confusion. I've helped a couple friends get started in Bitcoin, and they think the device holds their stack. Explaining that their "bitcoins" exist on the blockchain, and the reason they wanted a device, required some unlearning on their part before they understood how it actually works. Less a wallet, and more a key is a better metaphor.