I think tor is just half of the story. I had exact same problem with blue wallet on pi4, so I gave it up. But now I have Umbrel on Ubuntu on old laptop with intel i5 4th gen or so. Still tor and the same network setup, but Samourai wallet runs through it like a charm. And basically everything is normally fast. I did not try blue wallet on it, but if you want, I can try and let you know.