It's deterministic. You are not wrong.

It's not the right way to do it, there needs to be entropy between the key and the usage.

It brings up a good question about trust because you really can't trust either the client or a network connected (especially not in-browser) keychain.

Is there some specific reason why you aren't going with an embedded return encryption key? This at least gives you a buffer of protecting future messages via entropy and doesn't let you gain more than the past out of a key breach.

Going further than this requires the use of rendezvous points and such similar things. Preparing entropy ahead of time inherently requires you conceal the location where it is stored, or it's a key to future messages.