You may be right and maybe I got lucky but I know a few people who seemed to only lose follows. I would also think that the people following you would re-find you through other relays eventually since their follow is still in their config.