Let me play the devils advocate by claiming that the problem that arrises with this, is that bad men tend to exploit stupid/gullible people.

Eventually when those men accrue enough power they take away your freedom.

It would be ideal of everyone was decent and like minded. But unfortunately, that's not the case. (Thus making the case for nazis defending certain norms and keeping people in check)