About Chessmark

Most benchmarks ask a model one question and score the answer. Chess asks it eighty questions in a row, where every answer depends on remembering all the previous ones, an adversary is actively working against it, and there is a referee who cannot be argued with.

That makes it a test of agentic reliability rather than chess ability: long-horizon state tracking, correct tool use under pressure, and recovery from being told you are wrong. A model that plays a beautiful opening and then tries to move a knight that was captured twenty plies ago has failed at the thing being measured.

The headline number is not the rating. It is the illegal move rate — how often a model proposed a move that could not be played from the position in front of it.

A model never touches the board. It calls tools — read the position, list the legal moves, make a move — and the server validates every one of them with python-chess. The server is the only authority on board state, so a model cannot corrupt a game record even if it tries.

An illegal move is not an instant loss. The model is told exactly why, handed the full list of legal moves, and asked again — five times before it forfeits. The whole conversation is one append-only transcript that never rewrites earlier messages, which is both what makes prompt caching work and what makes the replay honest.

Everything, verbatim. Raw request and response payloads with credentials redacted, every reasoning trace, every tool call and its result, token counts, and the exact cost computed from the tokens the provider actually returned — never estimated.

If a number appears on the leaderboard, the transcript behind it is one click away. That is the whole point: a rating reaches its games, a game reaches its plies, and a ply reaches the JSON the provider sent back.

• The sample is small. 244 ranked games of 433 finished. Every rating carries its deviation for exactly this reason — read the ± before the number.
• A contestant is a model at a precision and a reasoning level. The same weights served at 4-bit and 8-bit, or asked to think hard and barely at all, are different entrants and are ranked apart, because they do not play the same.
• The endpoint matters too. One provider returned malformed tool calls on roughly one call in six while others serving identical weights returned none. Each seat pins one endpoint for a whole game, and the provider is recorded per call.
• This is not a chess engine ranking. Every contestant here would lose to free software from 2005. The interesting question is not how well they play, but how long they can stay coherent.

The board sounds are from Kenney, who releases game assets into the public domain.