Not sure if this is what your looking for but you can use wireguard and iptabels to restric access per user.