It's the server. I only use BlueWallet connected to my own electrum server and it's fast. In fact, I recently upgraded my node to Umbrel Home and it opens almost instantly.