Users are not tied to relays. Their client may be fluid. Take a look at https://mikedilger.com/gossip-relay-model.mp4.

But they could also rebroadcast notes they reply to that they got from other relays to the relays they normally write to, so their followers that know about these relays may easily fetch the notes.

Or other approaches can be done.

Currently most clients use the dumbest approach you can imagine, as far as I know, except, so far, https://github.com/mikedilger/gossip, which uses the model described in the video above.