This is definitely part (but not all) of it. Even in the 60's and 70's when housing was cheapest here fertility was dropping. Much of Europe were seeing significant declines in fertility before WW1 (which caused a mini baby boom afterwards) and this was before the welfare state.

As the Roman empire matured the same thing happened. It really seems like this is an unsolvable consequence of society... Working the way people want it to.