Yeah, so the diamonds he (supposedly) gave her. That's not how I understood the sentence but that is the only one I can make sense.
I think you're right.
Discussion
How did you understood ?
It says "I never hated a man enough to give him diamonds back." It says "[..] give him [..]", suggesting new/other diamonds, as opposed to "[..] give *his* [..]" suggests returning the ones he gave.
So made me think: diamonds as sign of hate/hating.