Interesting idea and approach, you're right about the tradeoffs with MLS. I implemented something similar to this a few years ago, which worked by creating a single group key and sharing it through encrypted messages to members. This hides everything from the server, and allows for sending keys in-band (you could send via invite link as well if you preferred). You could also control how often group keys were rotated, giving the group owner control over whether new members get message history (unfortunately with no guarantees about post compromise security or forward secrecy, although if you trust the relay to delete old messages you could get it that way).

I abandoned this because message delivery was flaky for orthogonal reasons (relay rate limits and a poor relay selection implementation), as well as because MLS was coming. It could be revived though, and might be a decent trade off balance for certain applications.