NIP46 involves too many participants, and if it's done over the internet, there are many potential issues and it can be slow. Jumble directly implements NIP46 using nostr-tools, so I'll need to check the source code to find the problem.
Discussion
Have you thought about implementing NIP-55 for Amber?