I'm just thinking stress may play a part, I get a bit jittery too if I have coffee when I'm really stressed, but who knows really.

Straight coffee has been shown to be quite beneficial to you, so I personally don't see any reason to try and ween yourself off it if you enjoy it. But it makes sense you would if it doesn't make you feel good.