I am not familiar with Portainer or Tailscale, apologies. What I would suggest is to test that your relay is able to accept websocket connections at all prior to using with a client.
I have a section here where I am setting up a relay with auth, and test the connection with wscat.

https://rodbishop.npub.pro/post/52120fe7/
If all else fails open a GitHub issue with the devs. Utxo and Anthony are quite responsive