I would first recommend that if a client is connecting to "thousands of relays" because you follow thousands of people and they are all on different relays, then that client should only download the events of one person per relay. If John is on John's relay, only download John's notes from John's relay, not everybody elses, because otherwise you will get tons of duplicates. And if your client remembers what time period it asked for last time, it can avoid asking for those same notes again next time.

However, It still is a lot of SSL connections to manage. And I think that servers helping clients with the load are a good idea in general, without specifically recommending any architecture (because I'm entirely unsure).