I totally understand your point and can relate.. I found myself in that position and it happened a couple of times I did unfollow.

It is indeed a strange feeling when some that you considered an amazing person, can just shift your opinion with sometime a single note.. but that's fine, I do believe happens because people here feel safer to express opinions they know they couldn't anywhere, they can drop some masks.