Metadata (who,when) is stored on the server, and accessible to the admin. This is necessary for delivery under federation. If you do the protocol fully decentralized (an option with Dendrite), then it is fully e2ee and only the nodes involved have the metadata.