actually, i think that was at the center of why the non-fallen angels were so against it. firstly, they just dominated and threatened the human women they "took as wives" which is disgusting by itself, but the progeny were dangerous and probably cultivated to be violent pets by the fallen angels. i'm guessing that probably in most cases they gained physique but didn't especially gain so much in intelligence. or at least, they dominated them while they were still relatively small. probably also, the women often died in childbirth as well because of them being substantially larger, i'd be guessing that probably they were typically in the range of the biggest infants you see today, which can cause a lot of complications in birth.
so there is a lot of reasons why it would be objectionable, most of them to do with the mindset, but probably some degree of physical issues as well.