I know, it’s happened to me before. Like it literally made up nostr nips when I asked specific questions about them. But I guarantee it actually does report certain things, like 100% without a doubt.
I saw your conversation with poppin willow, one thing to be aware of with gtp bots is they do something called "hallucinating" where when asked a question they don't have an answer to they will confidently give you a completely made up answer with absolutely no evidence or reasoning to back it up instead of just admitting they don't know.
Chat gtp can't be trusted as a reliable source on its own programming for that reason.
https://en.m.wikipedia.org/wiki/Hallucination_(artificial_intelligence)
Discussion
Best to assume everything you send to it is reported and data mined in several different ways.