I notice the same thing. You’re not crazy. I’ve always suspected that it has to do with the funky-ness of how the relays work. But I honestly don’t know. I’d be interested if you get a good answer.

It seems to depend on the client too. This was a while ago, so it may have been fixed, but I noticed that two different clients, despite having the same relay list, was different in what they showed - one of them only showed some of the notes of the people I followed, while the other showed, as far as I know, all of them. I think it was Coracle that didn't show all, and Iris that did, but I'm not 100% certain. Currently I'm using Satellite, as Iris became worse over time and eventually stopped working in my browser. I haven't tried it for a few months.