Key Features
Resume buffer
Reallocates redundant meta-training compute toward longer failure regimes, scaling learned-optimizer meta-training to long inner problems.
Progressive expert supervision
Enforces decoupled progressive expert supervision, providing stable meta-learning signals that additionally lead to generalizable LOs.
Consistent downstream improvements
ELO-LOs substantially outperform their base LOs.
Cheap to meta-train
Every ELO-LO is trained in under 7 GPU-hours.
Experiments
ELO-LOs generalize beyond their meta-training distribution
Meta-trained only on four tiny 8×8 image classification tasks with a one-layer MLP, ELO-LOs generalize to vision and language pretraining tasks with up to 1.3B parameters. ELO-Celo2 outperforms well-tuned AdamW across all tasks and even surpasses Muon on GPT-2 (30M and 124M), while Muon remains best on GPT-3 (1.3B).
Evaluation on ImageNet Benchmarks
| Method | ResNet-50 | ViT-B/16 | Avg. | ||||
|---|---|---|---|---|---|---|---|
| IN-1K | IN-Real | IN-V2 | IN-1K | IN-Real | IN-V2 | ||
| AdamW | 75.81±0.07 | 82.17±0.02 | 63.39±0.15 | 74.03±0.11 | 80.72±0.18 | 61.30±0.27 | 72.90 |
| small_fc | 65.31±0.10 | 73.66±0.03 | 53.51±0.11 | 32.10±0.94 | 37.09±1.10 | 25.69±0.88 | 47.89 |
| CL-small_fc | 71.91±0.30 | 79.23±0.26 | 59.58±0.30 | 67.65±0.22 | 74.88±0.16 | 54.73±0.27 | 68.00 |
| Celo2 | 75.76±0.11 | 82.34±0.07 | 63.39±0.27 | 75.86±0.05 | 81.74±0.05 | 62.76±0.09 | 73.64 |
| ELO-small_fc (Ours) | 74.08±0.16 | 81.13±0.12 | 62.10±0.30 | 75.15±0.07 | 81.69±0.05 | 62.08±0.10 | 72.70 |
| ELO-Celo2 (Ours) | 76.11±0.09 | 82.70±0.02 | 64.03±0.39 | 76.42±0.28 | 82.26±0.31 | 63.89±0.32 | 74.24 |
Best top1-validation accuracy on ImageNet-1K validation, ImageNet-ReaL, and ImageNet-V2 for ResNet-50 and ViT-B/16 trained from scratch, mean ± std over 3 seeds. Each method is trained on ImageNet-1K at 224x224 resolution for 50K steps with batch size 2048. The best value in each column is in bold.
How to use ELO-trained LOs in PyTorch?
ELO optimizers ship in PyLO with CUDA-accelerated kernels. Install from source:
git clone https://github.com/Belilovsky-Lab/pylo
cd pylo
pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cu118
PYLO_CUDA=1 pip install --no-build-isolation .
python -m pylo.util.patch_mup
Drop ELO_CELO2_CUDA into a standard training loop like any other PyTorch optimizer:
import torch
from pylo.optim import ELO_CELO2_CUDA
model = torch.nn.Linear(10, 2)
num_steps = 1000
optimizer = ELO_CELO2_CUDA(model.parameters(), lr=3.16e-4, weight_decay=0.1, adam_lr_mult=20)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_steps, eta_min=3.16e-5)
for step in range(num_steps):
optimizer.zero_grad()
loss = loss_fn(model(input), target)
loss.backward()
optimizer.step()
scheduler.step()
Citation
If you use ELO in your research, please cite our paper:
@article{huang2026efficient,
title={Efficient Long-Horizon Learning for Learned Optimization},
author={Huang, Xiaolong and Th{\'e}rien, Benjamin and Harrison, James and Belilovsky, Eugene},
journal={arXiv preprint arXiv:2607.06772},
year={2026}
}