A strong general-purpose VLM still fails on a chessboard

We were building Chess Lens, an educational chess assistant that looks at a board and helps a player think through the position. The plan was to use VideoDB to ingest the board visually, follow the game as it changes, and turn the current position into structured data for teaching, analysis and next-move guidance. None of that works unless the first step is exact. The system has to read the board correctly, every square, every time.

This post is about how we tested that step. We ran GPT-5.4 and Claude Opus 4.7 through the same board-reading task under several thinking configurations. Claude Opus 4.7 trailed the strongest GPT-5.4 runs throughout, and the aggregate scores did not explain why. The explanation was in the reasoning traces and in the individual failures, and it turned out to be a narrow, specific limitation rather than a general one.

What we measured, and why FEN

Chess Lens needs the current position in FEN, the standard notation for one exact chess position. FEN writes the board as eight rows, from rank 8 down to rank 1. Each row is a string of piece letters and digits: an uppercase letter is a white piece, a lowercase letter is a black piece, and a digit counts consecutive empty squares. A row like PP3P2 means two white pawns, three empty squares, one white pawn and two empty squares.

We scored only the piece-placement field. The model received a board image and had to return which piece was on which square, in that format. We compared its output with the expected FEN for the same point in the game, and a position counted as correct only if the whole field matched.

That is strict, and it is strict in the way Chess Lens needs. If a pawn is shifted by one file, the position may still read as plausible in prose, but the FEN is wrong, and everything downstream of it is wrong too.

How the test was set up

We took one game and evaluated 72 board positions from it. For each position the model had to work out the board orientation, identify every visible piece, assign each piece to its square, and emit the piece-placement field.

The score is exact-match accuracy: the number of positions where the returned field matched the expected FEN, out of 72. We ran both models under several thinking configurations, from the default with no extended thinking up to the highest settings each model offers. The configuration names below follow the API settings we used: the thinking level, and whether the run returned a reasoning summary.

The token budget changed the result

Our first run capped the output at 1024 tokens. That turned out to be too low for the high-reasoning configurations. Some runs spent the whole budget thinking and never produced the final structured answer, which the scorer counts as a parse error.

The very low scores for GPT-5.4 xhigh and Claude Opus 4.7 max are truncation failures, not vision failures. Those runs used too many tokens and usually did not reach the answer.

We raised the limit to 4096 tokens and ran everything again.

With the larger budget, Claude Opus 4.7 improved substantially, most of all at max thinking, where it went from 2.78% to 84.72%. It still did not catch up with GPT-5.4, whose best runs were at or near 100%. The gap survived the fix, so the token budget was not the explanation.

Why the aggregate scores hid the problem

At that point we stopped looking at the totals and read the intermediate reasoning summaries and the failed outputs instead.

GPT-5.4’s summaries were procedural and row-specific. They walked the board rank by rank:

For row 8, I see: a8 has a black rook, f8 has a black rook, h8 has a black king…

Moving to rank 2, I see the white pawns at a2, b2, c2, with gaps where pieces have moved…

Claude Opus 4.7’s summaries were more often global descriptions of the position:

I’m looking at a chess board layout with pieces positioned across the rows, showing what appears to be a mid-game or puzzle position…

Black has pawns scattered across the board with the king on f6, white has a rook on d1, a king on e3, and a few pawns positioned strategically.

That difference explains a lot of the remaining errors. Claude Opus 4.7 usually understood the board as a chess position, with the right pieces and the right general shape, but it was less reliable at preserving the exact square-by-square layout when writing it down.

What the failures looked like

We went through the failed positions by hand. Most errors were local. A piece shifted by one file. One extra or one missing empty square in a row. A gap left by a moved pawn that was not recorded. The right piece family on the wrong square. The right piece with the wrong color, which in FEN is the wrong case. Three examples:

This pattern matters because of how FEN compresses a row. The piece count can be right, the row can add up to eight squares, and the string can still describe a different position. Claude Opus 4.7 was not failing at chess understanding. It was failing at precise spatial localization, and FEN punishes exactly that.

What the model card already says

The result lines up with Anthropic’s own vision documentation:

Spatial reasoning: Claude’s spatial reasoning abilities are limited. It may struggle with tasks requiring precise localization or layouts, like reading an analog clock face or describing exact positions of chess pieces.

FEN generation is that task in its purest form. It requires exact localization across 64 squares, and a model that recognizes the board but places one piece one file over is wrong.

What this means for downstream vision tasks

Chess Lens exposed a narrow but important failure mode. Understanding the board is not enough when the output format requires exact coordinates. Claude Opus 4.7’s outputs were often reasonable at the level of the position, but FEN is scored at the level of the square. One shifted piece changes the board state, one wrong empty-square count changes the row, and a model can be directionally right and still fail the task.

The practical lesson for us was that downstream tasks need their own evaluations. The model that performs best in general is not automatically the best model for the specific step you are trying to automate, and the only way to know is to test that step directly, with a scorer as strict as the task.

@article{nagaonkar2026strong,
  author = {S Nagaonkar},
  title = {A strong general-purpose VLM still fails on a chessboard},
  journal = {VideoDB Labs},
  year = {2026},
  month = {may},
  note = {https://videodb.io/blog/claude-chessboard-spatial-reasoning},
}