It doesn't seem to affect followers, only your follows. I tried to use nostrgram that some other people had used to get theirs back but it didn't work for me.
Discussion
My understanding is that follows and followers are both posted to relays just like notes are, so if you remove all the relays one of them is posted on you won't see it anymore.
You may be right and maybe I got lucky but I know a few people who seemed to only lose follows. I would also think that the people following you would re-find you through other relays eventually since their follow is still in their config.
They might still follow you, but you might not see it if the follow is only posted to a relay you don't use but they do.
Nostrgram was a life saver for me.