Showing just a thumbnail profile image and sometimes also a name in notifications, I believe, contributes towards identity confusion.
There are also those that you know, but don't follow, and don't have the "follow" icon — this is mostly relevant for Amethyst.