They control block templates, so they are a censorship resistance risk.
For the 51% attack risk, I do see it as less of a danger than the raw mathematics suggest (mainly for the reason you're implying). But it's still not good.
Discussion
I can't see anyone pulling off a 51% attack at this point. the cost is too high.