You shouldn't eliminate caffeine from your diet.

Unlike sugary drinks and added sugars, which are not necessary because there are plenty of common sources of carbs that you will consume anyway, coffee is basically the only consistent source of antioxidants that modern Western people will have readily available to them.

You can get caffeine from tea, guarana, yerba mate, cola nuts... You will see that most of these are made into drinks. That's a better way to avoid sugars and fats. If you don't mind them, caffeine is also present in chocolate...