To follow = to post a signed msg saying "I follow you"

If your follower posted this to relays A, B, C and you're connecting to relays X, Y, Z then you'll never know they follow you.

There are other ways to "follow" silently, without publishing anything.