My pastor recently talked about this and gave some interesting insight. Apparently the people were required to use a special currency in the temple and so the money changers would set up and provide exchange services as people arrived from wherever they came. But the money changers would charge exorbitant exchange rates and take advantage people. Others would inspect the sacrifices people brought to the temple and declare them unfit and then sell them an “acceptable” sacrifice while overcharging them for it. So it wasn’t just that business was being conducted, but they were being dishonest and extorting the people.