WOW. Meta just open-sourced a Github repo for LLM Training.

Meta Lingua is a minimal and fast LLM training and inference library designed for research

📊 Key features

- Minimal and fast LLM training/inference library for research

- Uses modifiable PyTorch components for experimenting with architectures, losses, data

- Enables end-to-end training, inference, evaluation

- Provides tools for understanding speed and stability

- Structured with core 'lingua' library and 'apps' to showcase usage

🚀 Lingua's performance comparison to other models

- 1B models trained on 60B tokens match DCLM (DataComp-LM) baseline performance on many tasks

- 7B models (Mamba, Llama) show strong results on benchmarks like ARC, MMLU, BBH

- Llama 7B squared ReLU 1T tokens model achieves high scores across tasks