GM
I think one of the best things about being an adult is the power to choose who you associate with. I have not been shy about cutting people out of my life and have done so on multiple occasions.
It's good to stop and reflect though, you never know when you might be a contributing factor. My only rule of thumb is that once I cut ties there's no looking back, so I tend to wait until I'm overwhelmingly sure.