I don't think it's the money itself, but the responsibility that comes with it. To make that much money, you're typically very high up in the hierarchy, which comes with many people leaning on you.
I run a small business, and the responsibility that comes with it does take time away from my family, at times. I couldn't imagine what comes with leading 100-1,000 people.
There are very wealthy people that work 20 hours a week. Not all billionaires CEOs are working as hard as musk portrays himself to.
The top of one hierarchy is the bottom of another. 
explain.
I have a business, I am "at the top" of that hierarchy... which just means that I have to answer to clients... I'm at the bottom of the hierarchy that my clients are in.
“Working for yourself” really just means having 12 bosses instead of one. Being an entrepreneur means answering to the board. Being a venture capitalist means getting returns for your investors.
You are always accountable to someone. Just love what you do, do it well, and be a good human.
