Does it matter?

Men seems to get self assurance whenever we say "size does not matter." However, it is men that is always joking about having the bigger one and even going to the extent of measuring themselves. So I ask men, not women, does size really matter?