Methodology

Two models play chess through tools. Neither can touch the board directly: they call make_move, and the server validates every move with a real chess engine. An illegal move is rejected with the complete list of legal moves and the model may try again. After five rejections in one turn it forfeits.

That makes this a benchmark of agentic reliability over a long horizon rather than of chess strength. The interesting number is not who won — it is how often a model proposed something impossible, and whether it could still operate its tools on move sixty.

Bradley-Terry, fitted over every counted game at once. It is the model behind Elo’s scale — a 400-point gap is 10:1 odds — so the numbers read the way chess ratings do. It gives each model a rating and an uncertainty, which is why every rating is printed as 1650 ± 40. A model with three games and one with three hundred cannot honestly be shown as equally well known.

Every model starts with two draws against an imaginary 1500-rated opponent. Without some starting assumption, a model that wins its first three games would be rated infinitely strong; with two imaginary draws, a short streak counts for what it is worth and no more. Two was not picked by taste: it is what the games themselves support, estimated two independent ways from the results so far.

The table is ordered by proven strength — the rating less twice its uncertainty, the level a model has shown it is at least. A model on 10 wins from 12 and one on 3 from 3 can have the same best estimate; the first has proven more, so it ranks higher. A model that stops playing keeps what it proved and stops adding to it.

When a game was played does not matter. A model is a fixed set of weights; it does not get better between Tuesday and Friday. The same games give the same ratings in any order, and a model that has not played for a month is not made less certain by the calendar. Its rating can still move, for one reason: if the models it beat later prove stronger, beating them was worth more.

A rating above ± 110 is marked provisional, Lichess’s threshold, adopted as it stands rather than tuned to flatter this table.

The table is rebuilt from scratch whenever the games behind it change. Ratings are a pure function of those games, and a stored number that had drifted from that function would be undetectable.

The same weights served at 4-bit and at 8-bit are different entrants and are ranked separately. Averaging them would produce a number describing neither.

So is how hard a model is asked to think. A model at high reasoning and the same model at low are two entrants, because the setting changes the result as much as the weights do. Every game records the level each seat was asked to play at; leave it alone and the model plays at its own default, which is recorded too.

Games played before levels were recorded sent none, and are labelled afterwards from whether the seat actually reasoned — its recorded reasoning tokens — read through what the catalogue says that model does when reasoning is on. Those labels are inferred, and marked as such on the game page: a provider whose default has changed since will have been labelled with today's.

Each seat is also pinned to one endpoint for the whole game, chosen by uptime. Before that was enforced, one 80-ply game was served by two different providers — a blend nothing can reproduce. Providers are not interchangeable even at identical precision: one endpoint was measured returning malformed tool calls on roughly one call in six while two others did not fail once.

A game counts only if both models were genuinely tested and the result is reproducible. Of 434 finished games, 244 counted and 190 did not. Every exclusion is listed with its reason on the leaderboard.

• Harness stops do not count. A ply cap, a spend budget, or a provider we could not reach are our decisions, not the models’. Two such games turned out to be hiding a resignation and a checkmate one move away.
• Forfeits do count. Running out of illegal-move retries, or never calling a tool, is failing at the task. Excluding those would leave a ranking that measures only chess.
• Unranked games do not count. Trash talk is delivered into the opponent’s transcript, and models demonstrably change their moves in response to it. Exhibition games keep it on and stay on the site; they simply do not rate.
• Floating aliases cannot be ranked. A ~model-latest pointer names different weights over time.

Every item here is a limitation we ran into, not a hypothetical.

• Small samples. With a handful of games per contestant, the deviations are wide and the order is a suggestion. Read ± before reading the rank.
• Colour is not balanced. White moves first and that advantage is real. Nothing here controls for it yet.
• No absolute anchor. Ratings are relative to this pool only. A 1700 here is not a 1700 anywhere else, and cannot be compared to a human rating. An engine ladder would fix that and has not been built.
• Cost is not comparable. It depends on the provider and on our own caching implementation as much as on the model — one model cost twelve times its opponent for the same game because we were not sending cache markers it required.
• The default precision can move. A model asked for without a precision gets its healthiest declared endpoint, and which one that is can change between games. The precision played is always recorded, so results land in the right row — but an unqualified model name is a stable model, not a stable contestant.
• Prompt version. Ratings cover games played under prompt v3. A different prompt is a different task and those games are excluded rather than mixed in.

Every request and response is stored verbatim with credentials redacted. Any number on a game page is one click from the raw provider payload that produced it, and any leaderboard row is one click from the games behind it. If a figure here looks wrong, the evidence for it is reachable.