It says "I never hated a man enough to give him diamonds back." It says "[..] give him [..]", suggesting new/other diamonds, as opposed to "[..] give *his* [..]" suggests returning the ones he gave.

So made me think: diamonds as sign of hate/hating.