Not only does this make sense, but it is actually what the roots of the word would imply:

Com: with or alongside

Passio: (passion) to suffer, endure, or strive

When we say we have "passion" for something, what we ought to mean is that we are willing to suffer for it. I have passion for my wife and family, for the right ordering of our society, and for freedom.