it is slightly harder to audit the total supply of XMR because you have to trust some additional cryptographic primitives and assume that they were implemented correctly. but again, nobody has successfully proven invisible inflation and people have tried really really hard to attack the monero network for a variety of different reasons for over a decade now...

the belief partly comes from what happened to a previous project. monero is a codebase fork of bytecoin, which was the first implementation of the cryptonote whitepaper. the old bytecoin was unfairly launched and it was later discovered to have also had an inflation bug. people either know about this and are still spooked, or they learned to be apprehensive through cultural osmosis.