isn't that why you would have several options in your NIP-65 in/outbox advertisment?

also, you know, if the clients would support NIP-42 (does gossip btw?) reading from someone's outbox when one is the addressed recipient should be permissible and could only be gated as such by using NIP-42

implement NIP-42, that's to all client devs, not just you (you know i'm not using yours because i can't test with it)