Efficient Long-Horizon Learning for Learned Optimization

Xiaolong Huang1,3, Benjamin Thérien1,4, James Harrison2, Eugene Belilovsky1,3

1Mila – Quebec AI Institute   2Google DeepMind   3Concordia University   4Université de Montréal

Paper Code (JAX) Code (PyTorch)

TL;DR: We introduce ELO, an efficient long-horizon meta-training algorithm for learned optimization.

Abstract

Learned optimization aims to improve upon hand-designed optimizers (e.g., Adam and Muon) by meta-learning small neural network optimizers over a distribution of tasks. While recent work has greatly advanced the architectural design and inductive biases of learned optimizers (LOs), their meta-training remains biased toward short-unroll learning on particular tasks, resulting in redundant computation and leaving LOs often unable to compete with hand-designed optimizers. We introduce Efficient Long-hOrizon (ELO) learning, an efficient meta-training algorithm that (1) reallocates wasted meta-training compute to longer failure regimes, achieving efficient long-horizon learning, and (2) enforces decoupled progressive expert supervision, providing stable meta-learning signals that additionally improve the generalization of LOs.

Our empirical study evaluates ELO for meta-training both element-wise and matrix-based LOs. Across downstream language modeling (GPT-2 and GPT-3 on FineWeb) and image classification (ViT-B/16, ResNet-50 on ImageNet-1K) tasks, ELO substantially improves the long-unroll performance and out-of-distribution generalization of the base LOs. In particular, ELO-Celo2 outperforms well-tuned AdamW and Muon on GPT-2 (30M and 124M) language pretraining. Notably, all ELO baselines require less than 7 H100 GPU-hours for meta-training.

Key Features

Resume buffer

Reallocates redundant meta-training compute toward longer failure regimes, scaling learned-optimizer meta-training to long inner problems.

Progressive expert supervision

Enforces decoupled progressive expert supervision, providing stable meta-learning signals that additionally lead to generalizable LOs.

Consistent downstream improvements

ELO-LOs substantially outperform their base LOs.

Cheap to meta-train

Every ELO-LO is trained in under 7 GPU-hours.

Overview of ELO: resume buffer and progressive expert supervision
Overview of ELO. Left: under the same compute budget, the resume buffer restarts unrolls from stored states instead of random initialization, shifting compute from easy early stages to failure-prone later regions. Right: progressive expert supervision moves from imitating an expert optimizer to pure task-loss exploitation, stabilizing meta-training throughout and encouraging the learning of superior update rules.

Experiments

ELO-LOs generalize beyond their meta-training distribution

Meta-trained only on four tiny 8×8 image classification tasks with a one-layer MLP, ELO-LOs generalize to vision and language pretraining tasks with up to 1.3B parameters. ELO-Celo2 outperforms well-tuned AdamW across all tasks and even surpasses Muon on GPT-2 (30M and 124M), while Muon remains best on GPT-3 (1.3B).

Validation loss of GPT-2 (30M, 124M) and GPT-3 (1.3B) pretraining on FineWeb
Validation loss of GPT pretraining on FineWeb. GPT-2 (30M/124M) is trained for 0.6B/2.5B tokens, and GPT-3 (1.3B) for ~10B tokens.

Evaluation on ImageNet Benchmarks

Method ResNet-50 ViT-B/16 Avg.
IN-1KIN-RealIN-V2 IN-1KIN-RealIN-V2
AdamW75.81±0.0782.17±0.0263.39±0.1574.03±0.1180.72±0.1861.30±0.2772.90
small_fc65.31±0.1073.66±0.0353.51±0.1132.10±0.9437.09±1.1025.69±0.8847.89
CL-small_fc71.91±0.3079.23±0.2659.58±0.3067.65±0.2274.88±0.1654.73±0.2768.00
Celo275.76±0.1182.34±0.0763.39±0.2775.86±0.0581.74±0.0562.76±0.0973.64
ELO-small_fc (Ours)74.08±0.1681.13±0.1262.10±0.3075.15±0.0781.69±0.0562.08±0.1072.70
ELO-Celo2 (Ours)76.11±0.0982.70±0.0264.03±0.3976.42±0.2882.26±0.3163.89±0.3274.24

Best top1-validation accuracy on ImageNet-1K validation, ImageNet-ReaL, and ImageNet-V2 for ResNet-50 and ViT-B/16 trained from scratch, mean ± std over 3 seeds. Each method is trained on ImageNet-1K at 224x224 resolution for 50K steps with batch size 2048. The best value in each column is in bold.

How to use ELO-trained LOs in PyTorch?

ELO optimizers ship in PyLO with CUDA-accelerated kernels. Install from source:

git clone https://github.com/Belilovsky-Lab/pylo
cd pylo
pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cu118
PYLO_CUDA=1 pip install --no-build-isolation .
python -m pylo.util.patch_mup

Drop ELO_CELO2_CUDA into a standard training loop like any other PyTorch optimizer:

import torch
from pylo.optim import ELO_CELO2_CUDA

model = torch.nn.Linear(10, 2)

num_steps = 1000
optimizer = ELO_CELO2_CUDA(model.parameters(), lr=3.16e-4, weight_decay=0.1, adam_lr_mult=20)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_steps, eta_min=3.16e-5)

for step in range(num_steps):
    optimizer.zero_grad()
    loss = loss_fn(model(input), target)
    loss.backward()
    optimizer.step()
    scheduler.step()

Citation

If you use ELO in your research, please cite our paper:

@article{huang2026efficient,
    title={Efficient Long-Horizon Learning for Learned Optimization},
    author={Huang, Xiaolong and Th{\'e}rien, Benjamin and Harrison, James and Belilovsky, Eugene},
    journal={arXiv preprint arXiv:2607.06772},
    year={2026}
}