All the parts of the plant except the fruit don't want to be eaten. And they are at war with us, creating toxins of various kinds, growing thorns, growing too tough to chew, doing whatever they can to protect themselves from being eaten. But we are at war with them, developing enzymes to denature these toxins. And because we evolved fighting these toxins, turning away from them changes our environment for the worse... we are better off eating the non-fruit vegetables now even though they have these toxins in them.

Protein and fat didn't develop toxins, it developed teeth and claws. If you can get past those, the meat and fat are safe foods.