I think private non-rebroadcasting relays are the only way that access restriction is going to work. And even then white listing npubs isn't enough. You would need access to an area or content individually signed by an extension of nsec bunker.
People may not like it, but it is necessary for the growth of nostr. People will continue to use telegram, Facebook, OF, etc as long as that where they can have voluntary walled gardens. And we do need those, as long as people choose them.