Are you using a commercial VPN or your own?

It's possible you may need port forwarding, however a lot of VPNs such as Mullvad and IVPN have disabled this feature.

Split tunnelling is another possible solution if the VPN app you use provides it.

That's all I can suggest for now.