A reasonable definition for poison: A substance that causes injury, illness, or death, especially by chemical means.

Caffeine doesn't seem to fit.