Bridging E2EE services will always rely on "handing the keys" as you say to an intermediate you have to trust.

You could possibly fix the issue by running an encapsulated E2EE scheme where the clients use their own network as a connection layer and then handshake with *actual* E2EE between themselves.