Yes. When you request notes from your followers, you have to send that request to each of the relays you read from with the pubkeys for everyone you follow. This can be pushing several dozen kilobytes of data on each request to each relay. Then each relays takes that request and has to look at its database for notes recently authored by those pubkeys before sending back. More follows will take more time or until it hits the match limit and returns early. Then your client deduplicates and sorts for display.

In an ideal world, clients would be able to define lists, and have a way to pass just the list identifier instead of its contents when requesting notes to at least reduce some network overhead.