I felt the same way but I bit the bullet and did it.

You'll get the followers back quickly, don't sweat it.