My usecase is probably a little different from yours because it's for community based group encryption, but what I do is that have nip-44 encrypted payloads to an address to which eveeu member has the private key, so there's only one address to monitor, but then I have MLS as an additional layer on top of it basically, so all members of the community check for messages to one key, decrypt them (if a few things pass), and then do the second description using the rachet tree, so then I don't have the problem you're describing... But again it sounds like a different use case perhaps?