Not possible, when somebody follows you they publish their entire contact list again, so you don't know when they followed only you