I don't think it's even a privacy issue. A lot of middle-class Americans aren't comfortable lying on their tax forms. And there's that one question that is phrased in such a generic way that if you bought even a single cup of coffee you're technically lying if you don't report it.
It sucks, but that's the situation. To me it's very similar to drug legalization; a lot of libertarian "cool kids" puff up their chest and go "well I've got my dealer so who cares"... But widespread adoption doesn't happen until something can go at least mostly above-board.
This is the *one thing* I wanted from the supposed "bitcoin president": a de minimum exemption. Still waiting!