Plenty of ethnic groups suffered genocide, never heard about devolving because of that. ;D

Realistically you can't kill aptitude within bloodline in a snap, recessive genes would kick in in a generation or two. Principal of "return to the mean" would arise as well.