I don't think it's the money itself, but the responsibility that comes with it. To make that much money, you're typically very high up in the hierarchy, which comes with many people leaning on you.

I run a small business, and the responsibility that comes with it does take time away from my family, at times. I couldn't imagine what comes with leading 100-1,000 people.