“If the total number of parameters in this one-layer transformer is less than the size of a domain, then transformers provably cannot solve the compositional task,”

https://www.quantamagazine.org/chatbot-software-begins-to-face-fundamental-limitations-20250131/