Duplicates would be easy to remove as the notes would be digitally signed and can be uniquely identified. Idk how it happens behind the scenes but I'm sure amethyst and damas and all other existing client apps do this
Discussion
They do, and it's inefficient and wasteful. If you want to see a note, you ask for it from 20 servers. Those 20 servers will send you all the same note. The bandwidth on 19 of them was wasted. Bandwidth adds up to real money. That's just a side effect of how nostr works. Maybe the payoff is worth it, but I think it should at least be understood as one of the tradeoffs, and perhaps there's a solution to it worth exploring.
I wonder if the relays can be tiered? So perhaps... client connects to just 3 main relays. Each of those 3 relays basically sort and filter from 10 each. Yes there is still duplication but less in total from a per-relay point of view.
I've been playing around with building a nostr client so I can understand nostr better, and when I'm searching for a particular note, I do a serial search in which I search the first relay, and if it returns the note, I stop the search, otherwise search the next relay, so there are ways around it to one degree or another. That's the 'fun' part of being an engineer of sorts is seeing problems, thinking of solutions, then seeing the problems those solutions created and seeing which of those you can 'fix'.
I've also thought about that, a smart relay that connects to other relays and dedupes the data before sending it to the client. At that point though, it starts to move away from dumb relays, so may be moving away from one of the core ideas of nostr.