Holding any culpability for voters is unfair because people are essentially voting under duress.
If you don't participate, other people will vote to take your stuff/rights. So you're forced to vote out of self defense.
Alt: the theory of authority is that the gov is acting on "the will of the people". If no one voted, they can't claim to have the consent of the governed