Plus if it ever got to be very large, governments and large corporate entities would farm mine with CPUs anyway. I think RandomX secures a marginal increase in decentralization only in the case that the money is not the top crypto asset. Decoupling general purpose CPUs from hashing crypto is, I think, quite necessary for the market for general purpose computers to avoid becoming prohibitively expensive and to remain focused on what a PC is meant for.

If Monero got really big like Bitcoin, it would either distort the PC market like Ethereum and other coins did to GPUs or it would result in the creation of specialized CPUs that are not really a great substitute for normal ass CPUs, a complex ASIC. You could mitigate that by continual revision of RandomX assuming there is always a software change that can consistently prevent specialized CPUs, but that still leaves the dual problems of PC market distortion and the same perceived problem in Bitcoin, a few miners having most of the hash power.

This last problem is only an issue when your everyday normie miner uses a centralized failure point or coordination mechanism (like a single mining pool sending out block templates like F2Pool or whoever). A Pareto distribution of miners is still expected and not problematic because of the rewards that go to miners who do not try to censor nor double spend, the moment any large actor in the space does that, as the mining market is free and the transaction fees for not censoring are generally higher. Monero mining would follow that same paradigm. The algorithm just determines which devices are being used like ASICs and which chains its security budget is directly competing against (DOGEcoin is winning against Litecoin as miners mine LTC and sell for DOGE: XMR competes with Wownero in the same way if I understand correctly).