It would make more sense to refer to them as subscribers rather than followers and people say follower counts don't matter because a lot of them are bots, dummy accounts, etc, and also because you can't get a true count of them since your only getting those numbers from the relays you ask.

If you post to a blastr relay like I do then you probably have subscribers on other relays that you don't read from that you don't even know about.

What does matter is reputation. It takes time and effort to build a reputation and you don't want to just throw that away.