Holding any culpability for voters is unfair because people are essentially voting under duress.

If you don't participate, other people will vote to take your stuff/rights. So you're forced to vote out of self defense.

Alt: the theory of authority is that the gov is acting on "the will of the people". If no one voted, they can't claim to have the consent of the governed