My understanding is that follows and followers are both posted to relays just like notes are, so if you remove all the relays one of them is posted on you won't see it anymore.
Discussion
You may be right and maybe I got lucky but I know a few people who seemed to only lose follows. I would also think that the people following you would re-find you through other relays eventually since their follow is still in their config.
They might still follow you, but you might not see it if the follow is only posted to a relay you don't use but they do.