Summary: OpenAI’s GPT-5.6 Sol had already solved open math problems such as the cycle double cover conjecture and beaten games like Pokémon FireRed. So when it managed only 7.8% on ARC-AGI-3, a benchmark built around 2D puzzle games, the team had to ask: were 2D puzzle games unusually hard, or was something else at work? ARC-AGI-3 is built to measure how well AI agents learn and reason inside unfamiliar 2D games, with no explicit instructions to guide them.
A Surprising 7.8% on ARC-AGI-3
OpenAI’s GPT-5.6 Sol had already solved open math problems such as the cycle double cover conjecture and beaten games like Pokémon FireRed. So when it managed only 7.8% on ARC-AGI-3, a benchmark built around 2D puzzle games, the team had to ask: were 2D puzzle games unusually hard, or was something else at work?
ARC-AGI-3 is built to measure how well AI agents learn and reason inside unfamiliar 2D games, with no explicit instructions to guide them. Its harness is deliberately generic, offering no tools or special features, so that model weaknesses stand out more clearly and comparisons stay fairer. Digging deeper, OpenAI found that a large part of the model’s confusion did not come from the model at all—it came from two settings in that harness.
Two Harness Flaws: Reasoning Thrown Away and Rolling Truncation
The official ARC-AGI-3 harness carried two design choices that badly limited the model:
- Discarded reasoning: Once a game action was taken, all private reasoning was discarded. GPT-5.6 Sol had to work the game out anew on every action, with no memory of its earlier thinking. It could see a record of past moves and brief notes, but none of the plans, insights, or thoughts that produced them.
- Rolling truncation: When the conversation context went past 175,000 characters, the oldest messages were deleted. The model therefore lost its memory of earlier actions little by little, which compounded the loss of reasoning.
Taken together, these two features explain why GPT-5.6 Sol could not build on what it learned as a run went on. Each turn effectively restarted the model, and even its action history kept getting erased.
The Fix: Retaining Reasoning and Compaction in the Responses API
OpenAI’s models are trained to think in private reasoning messages before they produce a reply or a tool call. Those messages are kept as part of the conversation history, and when a conversation grows too long it is summarized and continued. That is how the models are deployed in ChatGPT and Codex.
To mirror this production setup, OpenAI built the ARC-AGI-3 harness on the Responses API. Managing context is simple with it: for GPT-5.6, supplying the previous response ID retains reasoning automatically across tool calls and turns.
With reasoning retained, two big changes showed up:
- Less thinking time: GPT-5.6 Sol spent less time before each action, because it no longer had to read the game from scratch on every turn.
- Better learning and strategy: Once it could remember its earlier thoughts, the model learned far better over time and followed coherent strategies.
The next gain came from replacing rolling truncation with compaction, another Responses API setting. Compaction summarizes older context rather than throwing it away, so earlier observations and actions survive. This let GPT-5.6 Sol hold on to what it had learned across longer runs and reach a higher score with fewer output tokens.
The Results: Triple the Score, 6x Fewer Tokens
Under the official harness, GPT-5.6 Sol scored 13.3% on the ARC-AGI-3 public set. With retained reasoning and compaction enabled, it scored 38.3%—roughly a 3x improvement, reached while using 6x fewer output tokens. The scores are measured in Relative Human Action Efficiency (RHAE), a metric that compares model performance against a human baseline. Drawing on official gameplay logs, OpenAI estimates that the average human tester scored 48%.
An animation in the blog post shows the difference: with better memory, GPT-5.6 Sol thinks less per action and proceeds much faster. The two central columns illustrate how differently each harness uses the model’s 175K context window. A note adds that OpenAI’s implementation enforces a limit of 175,000 tokens instead of characters, though the two turn out to be similar because the vast majority of the text is action grids tokenized at a 1:1 ratio.
What This Means for Developers and Evaluators
OpenAI’s conclusion is a reminder that evals seldom measure models in isolation—they also measure a bundle of less visible choices about API settings, harness design, and prompting. Nor is this the first time the company has been surprised by a low score on a public benchmark and then found that the eval runner was built on a generic harness which dropped reasoning messages.
For API developers aiming at maximum performance, OpenAI recommends the same settings it deploys in its own products:
- Use the Responses API, not the legacy Chat Completions API
- Retain reasoning
- Use compaction
For anyone comparing models, the recommendation is to lean on evals that use these settings, since they come closest to real-world use in ChatGPT and Codex.
Practical Lessons for Product Builders
The lesson here is valuable for anyone building AI products: when a model underperforms on a benchmark, check whether the test environment matches your actual deployment. Changing settings can be more effective than changing the model itself.
If you are on the Responses API, retaining reasoning and enabling compaction are simple ways to lift performance on long-horizon tasks. These are not benchmark-only switches—they reflect how the models are trained and deployed in production, so real-world applications are likely to benefit as well.
Before you trust a benchmark score, ask: does the harness retain reasoning? Does it use compaction or rolling truncation? The answers could change your conclusions dramatically.