this seems equivalent to Matrix rooms. Sounds complicated, but I'm sure it's doable.
What I don't think would work is making posts authored before you followed somebody visible to them after you follow them.
Discussion
Matrix encryption is not optimal, but yes its similar.
Indeed, this has forward secrecy and new follows don't see old posts.