I remember Vitor said there's a bug, if your followers haven't loaded yet, but you anything in your relay list, you will loose followers.

Maybe it's fixed in the recent version.