Isn't everything ID'd? If clients hit relays in succession, if could get the Note ID's and only actually grab "new"?
This would reduce overhead on Clients and Relays (esp if relays are hit in random order to spread the load).
Or even if grabbing all is unavoidable, can locally dump the dupes.