You could have more or less followers, but at the end of the day, does it truly define who you are as a person? Personally I try not to think about things like that and go on with my day.
I think the real question here is if the people following you are genuinely interested in what you have to say, regardless of who you are.
I greatly enjoy interesting conversations like the question you posted here.