Nostr Web Client

Sometimes when I find myself not finishing a book but am still curious to at least know the ending, I ask AI to summarize it for me.

But today I wanted to remember something from a book I fully read many years ago, and asked AI to summarize it for me, and noticed that it was totally wrong at multiple points. Dramatically so. I pointed out that it was wrong and it’s like “my bad, here’s the right answer.”

So I asked how it could confidently make mistakes like that. Like what specifically happened in this case?

And it talked about how it extrapolates from stories and thus if something avoids the usual tropes, it could get it wrong. It said if I ask for citations it could help it prove its accuracy (which I didn’t think to do for fiction, I mean the entire objective answer is in one book, there’s not like multiple conflicting answers here).

So I was like “okay what happened to (this character) at the end? With citations.”

It gave me a wrong answer with misleading citations. So I pointed that out.

And it was basically like “oh wow, you’re right, I know that is disappointing to happen twice.” And some more boilerplate.

So I asked “I mean, I should just disregard all your prior summaries, right?”

And it more or less was like “Yeah. But we can revisit them if you want.”

By the end I felt like JK Simmons in Burn After Reading.

Jonathan 7mo ago

The model’s performance on summarization is going to depend most of all on how popular the book is. Remember, in this context you’re relying on the model’s memorization of the book, which only really works if there’s a lot of stuff written about it. This also means the book can’t be new.

TLDR: Just use a long context model like Gemini and put the whole book into your prompt when asking it to summarize instead of trusting the LLM to remember.

Reply to this note

Please Login to reply.

Discussion

Lyn Alden 7mo ago

I did it with very popular books.

I did find that Gemini was not making the summarization errors that ChatGPT was making.

Jonathan 7mo ago

If the books are super popular it might be ok, but again not super trustworthy. It’s like asking a friend who read the book years ago to try and remember it for you. Gemini might be a bit better if it has web search enabled by default.

Just pasting the whole book into the prompt is going to be much more like asking a friend to give you a summary who is currently holding the book and can instantaneously read the whole thing.