Diet makes a huge difference to population. British WWI soldiers were smaller on average than Australian soldiers, for example.

Jared Diamond's 'Guns, Germs and Steel' does a good job at explaining such realities of geography and available food.