The main issue is that if the user has inbox relays that AUTH, the server cannot auth as the user and thus notifications never show up. :(
Discussion
Couldn't you have an npub just for the notifications, you then giftwrap to that npub and further giftwrap to the user's npub.
That way you don't know who the notification is actually for (its gift wrapped to a unique npub that's only used for that), and then it's further wrapped to the user, so you don't need auth nor trust from relays, it can be pushed to any public relay
And maybe you rotate that npub. Increase the derivation index every day
That requires all the other NIP-17 players to support separate key aliaes. We have a PR open in the NIP repo, but I don't think it is going to happen.