A testbed for short-horizon task execution where the action space consists of only GameBoy buttons, with no special harnesses, and the observation space consists of screen frames, with no observation engineering. Agents see the game as humans do, and play the game as humans do. Each of the five game series marks a dedicated training game and a distinct, test-only game. Agents are allowed indefinite access to dedicated training games but are provided no demonstrations, documentation, or rewards, and must ground themselves in the environment through self-directed exploration and by inferring actionable knowledge from their own experience. With 50 hand-crafted tasks per game, GameBoyWorlds-Execution tests agents on a wide range of 500 tasks, including navigation, interaction, combat and game-specific control across 10 distinct games.

We adapt curiosity-based RL to serve as a search procedure to identify `interesting' snippets of gameplay. Our core intuition is that subtrajectories which end in a highly novel observation likely implicitly contain information regarding key, game-specific mechanics. We sweep these subtrajectories with the VLM and distill insights from them into game-specific guidance documents to aid task execution. In truth, this strategy achieves only partial success, but it consistently outperforms both world modelling and autonomous skill discovery-based approaches.

Tests end-to-end game completion in two fan-made Pokémon games. We show that while frontier models have been pre-exposed to official releases such as Pokémon Red, they lack essential information on the games in our testbed, recovering less than 30% of their progression steps. Instead of relying on their parametric knowledge to succeed, agents must learn from their own experience and autonomously improve over the course of the playthrough.