Are you using a commercial VPN or your own?
It's possible you may need port forwarding, however a lot of VPNs such as Mullvad and IVPN have disabled this feature.
Split tunnelling is another possible solution if the VPN app you use provides it.
That's all I can suggest for now.