I see what you mean. Yes, there is a danger of that. I have personally tried to stay as open minded as I can, but I’m certainly not perfect on it. But if it’s something that causes pain or irritation constantly, i may have to make a choice.
The key is to follow up with thought about why it got under my skin - does it mean I’m wrong? Or weak? Or intolerant of other valid viewpoints?
Without introspection and being willing to challenge one’s self, I agree that you can certainly stunt your own growth by surrounding yourself with yes-men.