Same teacher, two budgets: how small transformers learn chess

The two results

Training-time maximalism. In Amortized Planning with Large-Scale Transformers (Ruoss et al., DeepMind, arXiv 2402.04494), the authors built ChessBench: 10 million Lichess games annotated by Stockfish 16 with state-values, action-values and best moves, about 15 billion data points. They trained transformers up to 270 million parameters by pure supervised learning. The largest policy, asked to play with no explicit search at all, reached a Lichess blitz Elo of 2895 and solved puzzles rated up to 2867. All of the planning was amortized into the weights during training.

Test-time spending. Maxime Labonne’s new build, described in his September 16 X post, takes a 230 million parameter model to roughly 2000 Elo using Stockfish SFT distillation, a bit of RL, and “a lot of test-time search.” The demo runs in the browser on Hugging Face. Here the weights carry less of the load; the model earns its rating back by thinking at move time.

Neither number is fake and neither is magic. They are the same idea bought two different ways.

What the split actually buys

Distilling deeper is a one-time cost with a permanent refund. Every game the 270M model plays is nearly free at inference: one forward pass, one move. But the training bill was brutal. Fifteen billion Stockfish annotations is thousands of days of engine time before a single gradient step, and the paper still shows a gap to Stockfish itself that scale alone did not close.

Searching at test time is the reverse trade. The model is cheaper to make, but every move costs compute forever, and strength scales with how much thinking you allow per move. A 2000 Elo at one second per move might be 1700 at blitz speed and more at analysis speed. The rating is not a number, it is a curve.

• Amortize when you will play millions of games or serve moves at scale: inference cost dominates your life.
• Search when you cannot afford the datagen and training bill, or when you need strength on demand rather than speed.
• Combine when you are serious: every top engine, Stockfish included, already does both. NNUE is amortized evaluation inside a search loop.

The question I would actually want answered

Plot Elo against total compute for both families on the same axes: training FLOPs plus lifetime inference FLOPs for the amortized model, training FLOPs plus per-game search cost aggregated the same way for the searcher. My guess is the curves cross twice. Search wins early, when weights are cheap and dumb. Amortization wins in the middle, where a well-fed policy is nearly free to run. And the hybrids win at the top, because Stockfish 16 sits above both of these results and it is precisely a learned evaluator inside explicit search.

The honest summary: “no search” is not a flex, it is an invoice paid earlier. Where you pay is an engineering decision, not a philosophical one.

What I am watching next: an Elo-versus-search-budget ablation on the browser build. If 230M plus a heavy search budget starts closing on 2895, the amortized result stops looking like a ceiling and starts looking like a pricing decision.