Typically there's a built-in control mechanism where you're most biologically attracted to someone who's quite genetically similar to you that you'll be spreading more of your genes but not too close so that you can add variation and pick up improvements.

Doesn't work quite so well these days when most of us are spiteful mutants, fed and happy, and regularly mix in very expanded genetic pools.