He makes no difference to me.

The person who firmly stands with me against him, and doesn't exist, makes a difference to me. I wish they existed.