Could be an issue with relays, especially if you switch between various clents frequently and they don't all use your same relay list. Snort.social has a nice feature that allows you to follow everyone that someone else is following. It's a good way to bootstrap a follow list, and then whittle it down from there as you see people in your feed that you don't want to see.