It's the app deleting the contact lists from memory. it's VERY heavy to keep all your followers in the device.

Also, folllwer count is absolutely irrelevant in Nostr because you can just create a million keys that follow you in about a second.