Ah, that explains it. I was imagining standard filter behavior, but this is totally reasonable. It is, however, an instance of negative filtering, which some people are rabidly against. It's just baked into the spec rather than included in a filter object.
Discussion
Who are these people and why are they against it?
I think I get it, but I think that comes from a misunderstanding. The default "dumb relay" mode of Nostr was made for public broadcast microblogging data organized in a simple chronological feed of people you follow. If we want other use cases we have to adapt.