What if people just do things behind your back and you can't catch them on it, and you aren't present, and you cannot figure out what they are doing, how they're lying/deceiving? And nobody actually informs you of the things being done.

It isn't giving power away. It's people just doing things behind your back and everyone going along.