Agreed about the invite flow.
My simple/moron solve for that initially is: human trust.
Group members have a private QR code that will allow anyone in who scans it. So people should be extremely discriminating with handing it out.
When you have a _valuable_ and super high trust unit like that, people are naturally incentivized to protect it.