So I'm one of these people.

It's not the fact that the oils are from seeds. It's the fact that they're processed and extracted in high heat. Heat damages fats and fatty acids, causes them to oxidize and react in strange ways. They're complex molecules and many can cause metabolic disruption.

Cold pressed seed oils are fine, provided of course there's no other toxins in there like glyphosate or something natural in the plant seed that's bad or taken up from the soil. Also, different plant oils have different fats and fatty acids in them, too much of some is not good and all that, but that applies to any source of fat and isn't specific to seeds.