Yeah, it happens. I often notice (in my notifications) people starting following me, who were following before.