I tend to agree. Unfortunately it'll be up to clients to implement this well.
I would love to see NIP-04 go away completely and NIP-17 only be used in very low risk environments.
Once we get broad traction with MLS-based clients it'll be trivial to show whether a user can be contacted that way or if you need to fall back to NIP-17 (or double ratchet). There really isn't ANY reason to use NIP-04 at this point.