If those (not many) middlemen are the only ones using Bitcoin, then they can easily be targeted and corrupted. It also means that in combination with centralized mining, it is easier to take away the decision making about which chain to ignore or not.