That's tough dude. I'll keep it short:

Happy wife, happy life is a lie as big as fiat.

You (the man's) happiness is paramount. Do what you think is best for you. Don't go along with stuff you don't want. It's never too late to walk away, and by the sound of it, you do but you don't wanna admit it to yourself.

Source: I've had a similar situation and much experience with getting fucked over in relationships.