There might still be some juice left to squeeze here. A week ago Google dropped a successor to the transformer architecture that scales memory so context windows can grow significantly larger and with better performance:

https://arxiv.org/abs/2501.00663