The Fediverse does it like fiatjaf says, similar to Likes. However, the most accurate view of the poll will be on the poll author's home server.

On Nostr, we should a COUNT for votes on the poll author's relay.

When clients cast votes, they should deliver them to the poll author's relay.

This is a principle that clients should do in general for "p" tags.