More competition is good, right?

BUT:

I was wondering the other day:

What would happen if someone built a sufficiently large quantum computer optimized for bitcoin mining?

Mining difficulty would adjust so hard that it would be almost impossible to "win" blocks with normal ASICs - so the quantum miner would have the ability to perform a 51% attack.

What do you think?