This is equivalent to saying people can always switch clients if the client starts censoring. That has major costs in terms of user experience.

Imagine I'm following Alex Jones, and the hub relay I use starts censoring him. Not only do I not get any notice that that's happened (shadow bans are notoriously hard to detect), I have to manually find his relay out of band and add it to my client to see his material again. The gossip model (in its ideal form) is just the client doing this automatically.