The issue isn't really about you directly harming certain people. The issue is that whenever you vote, you are effectively upholding a system i.e the government, by giving them powers that you do not have yourself.

For instance, you don't have the right to murder, steal from, or enslave people. But using the power of the ballot box, you authorize the government to perform actions on your behalf, like murder through war and the death penalty, theft through civil asset forfeiture and taxation, and slavery through the Selective Service and prison labor.

Saying you support a candidate is one thing, but granting them an authority that is used is hurt people is a whole new level.