Because the follower graph is used to identify spammers. If non spammers follow spammers, this gets harder to do, hence the penalty.