This is extremely advanced but if you're game, there are plenty of ways you can configure routing rules so that it uses the vpn for specific IPs except some specific ones.

I have some rules like that precisely because of this sort of issues.