Also, common sense.

People need to live somewhere, so unless they move to a town with a lot of extra houses (true in some places, but not all places in America), there will be upward pressure on house prices proportionate to the influx of people.