How did you understood ?
Discussion
It says "I never hated a man enough to give him diamonds back." It says "[..] give him [..]", suggesting new/other diamonds, as opposed to "[..] give *his* [..]" suggests returning the ones he gave.
So made me think: diamonds as sign of hate/hating.