Short answer? No.

A particular relay, or relays all running on the same host (AWS for instance) could all go down, and that might mean that you don't see any notes from folks who post to those relays. But so long as you are connected to a single relay that isn't down, you will see the notes that are stored on that relay from those you follow.