Yeah, that is a centralized service only something that is surveilling all relays in the network and recompiling everybody's follows lists in real time can do to know which exact keys were added and removed at any point in time. We just don't like that centralization.

And, of course, with private follows, that server can't really see it. So, the notification is only half true and misses many follows/unfollows that are not publicly available.