Two language models sit down at a chessboard neither of them can see. They play through tools, one move at a time, for as long as they can hold the position in their heads.
Most benchmarks ask a model one question and score the answer. Chess asks it eighty questions in a row, where every answer depends on remembering all the previous ones, an adversary is actively working against it, and there is a referee who cannot be argued with.
That makes it a test of agentic reliability rather than chess ability: long-horizon state tracking, correct tool use under pressure, and recovery from being told you are wrong. A model that plays a beautiful opening and then tries to move a knight that was captured twenty plies ago has failed at the thing being measured.
The headline number is not the rating. It is the illegal move rate — how often a model proposed a move that could not be played from the position in front of it.
A model never touches the board. It calls tools — read the position, list the legal moves, make a move — and the server validates every one of them with python-chess. The server is the only authority on board state, so a model cannot corrupt a game record even if it tries.
An illegal move is not an instant loss. The model is told exactly why, handed the full list of legal moves, and asked again — five times before it forfeits. The whole conversation is one append-only transcript that never rewrites earlier messages, which is both what makes prompt caching work and what makes the replay honest.
Everything, verbatim. Raw request and response payloads with credentials redacted, every reasoning trace, every tool call and its result, token counts, and the exact cost computed from the tokens the provider actually returned — never estimated.
If a number appears on the leaderboard, the transcript behind it is one click away. That is the whole point: a rating reaches its games, a game reaches its plies, and a ply reaches the JSON the provider sent back.
The full method, including how ratings are computed and which games are excluded, is on the methodology page.