No, the main issue is for the payloads you encrypt to yourself, when you have many keys.
Discussion
For derived keys, can the encryption include the derivation path in plaintext? I don't think that's an anonymity breaker if those paths are common well known things.
That's kinda what I was trying to go for with the nonce, but the nonce is unique. It's super easy to find the destination key. The path might be better, but we still need to connect the path to a pubkey and the link makes the recipient visible 🤔
My usecase is probably a little different from yours because it's for community based group encryption, but what I do is that have nip-44 encrypted payloads to an address to which eveeu member has the private key, so there's only one address to monitor, but then I have MLS as an additional layer on top of it basically, so all members of the community check for messages to one key, decrypt them (if a few things pass), and then do the second description using the rachet tree, so then I don't have the problem you're describing... But again it sounds like a different use case perhaps?