It’s possible to curate your family. My kids’ grandma is the mother of my wife’s ex-husband, a man we have no contact with. She’s cool, he isn’t. The kids are raised and it’s all good. I suppose they never knew any different and that’s just fine.