"Lost in the Middle: How Language Models Use Long Contexts" investigates the performance of language models in utilizing long input contexts. The study focuses on multi-document question answering and key-value retrieval tasks that require language models to identify and use relevant information within their input contexts. The analysis reveals that language models tend to perform better when the relevant information is located at the beginning or end of the input context but struggle when accessing relevant information in the middle of long contexts. The research also finds that as the input context becomes longer, model performance decreases, even for models designed for long-context processing. The document provides insights into how language models use their input contexts and suggests new evaluation protocols for future models. It includes experiments with open and closed language models and investigates the impact of model architecture, query-aware contextualization, and instruction fine-tuning. A case study on open-domain question answering with retriever-reader models highlights the trade-off between adding more information to the input context and the model's ability to reason effectively over it. Overall, the research sheds light on the limitations and challenges of language models in effectively utilizing long input contexts for various tasks。

#chatgpt #AI #LLM