I've noticed that too & I keep a fairly small follow list. In my case, it seems to be dependent in what connection I am currently using. There are accts who's notes I cannot see unless I am at a particular location, despite shared relays. I suppose waiting long enough might provide time to find & fetch but at some point that just becomes unrealistic.