Could try removing them one by one and refreshing your followers list each time. Moght not be fool proof though as some ppl might have followed you on a relay they don't use anymore.
Discussion
It doesn't seem to affect followers, only your follows. I tried to use nostrgram that some other people had used to get theirs back but it didn't work for me.
My understanding is that follows and followers are both posted to relays just like notes are, so if you remove all the relays one of them is posted on you won't see it anymore.
You may be right and maybe I got lucky but I know a few people who seemed to only lose follows. I would also think that the people following you would re-find you through other relays eventually since their follow is still in their config.
They might still follow you, but you might not see it if the follow is only posted to a relay you don't use but they do.