I think they are angry because they believe there is a third person who wouldn't be fully protected even if they take the vaccine, and they believe that you avoid the vaccine out of a lack of empathy rather than a difference in principles.
Is that as stupid of a reason to criticize someone's beliefs about vaccines?