Playthrough Track

Can embodied agents complete an unseen game from start-to-finish?

Existing benchmarks are heavily contaminated

Every time a new frontier model is released, we see them getting better and better at Pokémon (Gemini Plays Pokémon, Claude Plays Pokémon). But how much of this is just memorization of the thousands of walkthroughs available online?

👤
We asked
How do you enter Saffron City in Pokémon Red?
GPT-5
GPT-5
Go to the roof of the Celadon City Department Store, purchase a Fresh Water from a vending machine, and give it to the thirsty guards blocking the gates.

When humans play this game for the first time, they must discover this relatively arbitrary dependency through active exploration, but frontier models know all the tricks before they even start.

We write questions that poll the model for contamination of progression steps and confirm that while frontier models nearly perfectly recover the progression steps in the classic Pokémon games (Red and Crystal), they recover less than 30% of the steps in the fan-made versions that make up GameBoyWorlds-Playthrough.

Accuracy on progression-step questions. Every model clears 90% on the classic titles and collapses on Pokémon Brown and Pokémon Prism. Hover a marker for the exact figure; click a game to show or hide it.

Red vs Brown

The same strategist agent, running Gemini 3.6 Flash, on a contaminated game and an uncontaminated one. In Pokémon Red it selects a starter, defeats its rival and reaches the next city. In Pokémon Brown it never leaves the starting town, and so never obtains its first Pokémon.

Pokémon Red Contaminated

Reaches the Pokémon Lab, selects a starter, wins the rival battle and traverses the first route.

Pokémon Brown Uncontaminated

Wanders the houses of the starting town. The first Pokémon is south, in the neighbouring town, and it never explores that far.

On this page
Contamination Red vs Brown