Verification is important, and most people probably don't do it, but making things more difficult to verify doesn't make the situation better. The fact that something can be verified if likely a deterrent or at least makes attacks more challenging. That's my view on FOSS software. For example, very few people have audited the Linux kernel. Even the people working on it are working on parts. But I trust it more than something like Windows because it CAN be audited.
Discussion
But doing something and having the ability to do something are different issues. Most people probably don't because they know fuck all about electronics, kernels, etc.