The Surprising Score: 7.8% on ARC-AGI-3
When OpenAI’s GPT-5.6 Sol—a model that has solved open math problems like the cycle double cover conjecture and beaten games like Pokémon FireRed—scored just 7.8% on ARC-AGI-3, a benchmark of 2D puzzle games, the team had to ask: were 2D puzzle games unusually difficult, or was something else going on?
ARC-AGI-3 is designed to measure how well AI agents learn and reason in unfamiliar 2D games without explicit instructions. The benchmark uses an intentionally generic harness, without tools or special features, to make model shortcomings more visible and comparisons fairer. But as OpenAI dug deeper, they found that much of the model’s confusion wasn’t inherent to the model itself—it was due to two settings in the harness.
The Two Harness Flaws: Discarded Reasoning and Rolling Truncation
The official ARC-AGI-3 harness had two design choices that severely handicapped the model:
- Discarded reasoning: After each game action, all private reasoning was thrown away. GPT-5.6 Sol was forced to figure out the game anew with every action, unable to remember its past thinking. It could see a record of past moves and brief notes, but not the plans, insights, or thoughts that led to them.
- Rolling truncation: When the conversation context exceeded 175,000 characters, the oldest messages were deleted. This meant the model gradually lost memory of its earlier actions, compounding the problem of lost reasoning.
Together, these features explained why GPT-5.6 Sol struggled to learn over time. The model was effectively starting from scratch on every turn, and even its action history was being erased.
The Fix: Retained Reasoning and Compaction in the Responses API
OpenAI’s models are trained to think with private reasoning messages before outputting replies or tool calls. These messages are retained as part of the conversation history, and if the conversation grows too long, it’s summarized and continued. This is how the models are deployed in ChatGPT and Codex.
To match this production setup, OpenAI implemented the ARC-AGI-3 harness using the Responses API. The API makes it easy to manage context: for GPT-5.6, passing the previous response ID automatically retains reasoning across tool calls and turns.
With reasoning retained, two big changes appeared:
- Less thinking time: GPT-5.6 Sol spent less time before each action because it no longer had to interpret the game from scratch every turn.
- Better learning and strategy: When it could remember its past thoughts, the model was much better at learning over time and employing coherent strategies.
The next improvement came from replacing rolling truncation with compaction, another Responses API setting. Compaction summarizes older context instead of discarding it, preserving earlier observations and actions. This allowed GPT-5.6 Sol to maintain what it had learned across longer runs, achieving a higher score with fewer output tokens.
The Results: 3x Score, 6x Fewer Tokens
With the official harness, GPT-5.6 Sol scored 13.3% on the ARC-AGI-3 public set. With retained reasoning and compaction enabled, it scored 38.3%. That’s roughly a 3x improvement, while using 6x fewer output tokens. The scores measure Relative Human Action Efficiency (RHAE), a metric comparing model performance to a human baseline. Based on official gameplay logs, OpenAI estimates the average human tester scored 48%.
An animation in the blog post illustrates the difference: with better memory, GPT-5.6 Sol thinks less per action and proceeds much faster. The two central columns show how the model’s 175K context window is used differently by each harness. Note: OpenAI’s implementation uses a limit of 175,000 tokens instead of characters, but this ends up being similar because the vast majority of text is action grids tokenized at a 1:1 ratio.
Why This Matters for Developers and Evaluators
OpenAI’s conclusion is a reminder that evals rarely measure models in isolation—they also measure a bundle of less visible choices about API settings, harness design, and prompting. This isn’t the first time they’ve been surprised by low scores on a public benchmark and then discovered the eval runner was using a generic harness that dropped reasoning messages.
For API developers trying to maximize performance, OpenAI recommends using the same settings they deploy in their own products:
- Use the Responses API, not the legacy Chat Completions API
- Retain reasoning
- Use compaction
For those comparing models, they recommend relying on evals that use these settings, as they best match real-world use in ChatGPT and Codex.
Practical Takeaways for Product Builders
This case offers a valuable lesson for anyone building AI products: when a model underperforms on a benchmark, check whether the test environment matches your actual deployment. Adjusting settings can be more effective than tweaking the model itself.
If you’re using the Responses API, retaining reasoning and enabling compaction are straightforward ways to improve performance on long-horizon tasks. These settings are not just for benchmarks—they reflect how models are trained and deployed in production, so they’re likely to yield benefits in real-world applications too.
Before you trust a benchmark score, ask: does the harness retain reasoning? Does it use compaction or rolling truncation? The answers could change your conclusions dramatically.
Sources
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.