Projects
Research and engineering projects, from ideas and model training to working tools.
Tiny LLM Ablation
Small language models trained from scratch: the gains and tradeoffs of MoE, recurrence, MoD, UL2, and diffusion at equal data budgets.
T5Gemma 2 Speculators
Training diffusion speculators to accelerate T5Gemma 2, with a plugin for serving them in vLLM.