npubs are cheap. It would cost nothing to a spammer to vote.

Meaningful votes to you perhaps are only from users in your network: your follows and perhaps their follows

Another rule is for a vote to count it must meet a minimum PoW.