An LLM is basically a bunch of layers that transform data into other data

Some of those probably at some step encode some “knowledge” of sorts

The problem is that it will work whatever you throw at it, even if it is a nonexistent thing, and return the most sensible “knowledge” according to the training data