I've never liked the idea of followers. For me, changing it to contacts makes sense if I "follow" and "they follow me." On Twitter, it also served the purpose of indicating how DMs could be exchanged.
Regardless of the name, would it be possible to make it optionally displayed?