Researchers have discovered that transformer models, a type of neural network widely used in natural language processing, struggle with long input sequences. As the input length increases, token embeddings become more similar, causing the model to lose its ability to capture important semantic information. This phenomenon is known as "length-induced embedding collapse." A theoretical analysis and empirical observations validate this effect, highlighting a fundamental challenge in scaling transformer models to handle longer inputs.

Source: https://dev.to/mikeyoung44/transformer-models-struggle-with-long-inputs-due-to-embedding-collapse-3h1g