It's called division of labour. The one hauling the stones isn't the same one planning and organizing or the one doing fine ornaments.

Same division of labour led to a high food supply, feeding a lot of people to shit the streets.

If you can't feed them anymore, you send them to war, so they either die or gain new territory.

Also 500 years is a lot of time to build stuff, if you haven't electricity or other stuff. You have no industry besides food, transport, clothes, weapons (blacksmiths) and housing. Once this needs are fulfilled, any surplus wealth goes into that structures. There was no consumer industry.

Everybody is your workforce, there is no business to run, no industry, no schools, no nothing. They just work 60+ hours a week basically for food and shelter.