You may be right and maybe I got lucky but I know a few people who seemed to only lose follows. I would also think that the people following you would re-find you through other relays eventually since their follow is still in their config.
Discussion
They might still follow you, but you might not see it if the follow is only posted to a relay you don't use but they do.