On April 22, 2026, developer Charles Azam published a deep-dive experiment: he read four open-source coding agent codebases cover to cover, forked three of them to run the same model — GLM-4.7 — and raced them on the 89 real-world tasks of Terminal-Bench 2.0. Claude Code, being closed-source, could not be forked; it ran natively through ZAI’s Anthropic-compatible endpoint.
Why it matters: model capability is usually treated as the only variable in agent performance. This study pins the model and isolates the harness — same model, same tasks, and a gap of more than 2x between the best and worst scaffold.
Setup: One Model, GLM-4.7, on Terminal-Bench 2.0
The four contestants come from very different engineering cultures: Codex (OpenAI; a 52-crate Rust Cargo workspace), Gemini CLI (Google; a TypeScript/React monorepo), Mistral Vibe (Python 3.12+), and OpenCode (TypeScript on Bun). The model is ZAI’s open-weight GLM-4.7, which the author notes ranks on par with Claude Sonnet 4.5 on Artificial Analysis.
The benchmark is Terminal-Bench 2.0 (run via the Harbor harness): 89 Docker-isolated, time-constrained tasks covering debugging, multi-file refactoring, test generation, and build fixing. Each score is a 5-run average with roughly 5% variance. The whole experiment burned about 5 billion tokens, funded entirely by ZAI’s $60/month subscription — which meant around 100 tokens per second and frequent timeouts.
The 2x Gap: Vibe at 0.35 vs Codex at 0.15
Final GLM-4.7 standings: Mistral Vibe 0.35, Claude Code 0.29, Gemini CLI 0.23, OpenCode 0.21, Codex 0.15. For reference, Terminal-Bench’s own leaderboard shows Claude Code and Terminus 2 scoring around 0.33 with GLM-4.7 via direct API, so the magnitudes line up. The source of the spread, in the author’s words, is “the code around the model.”
The fork difficulty tells the same story in reverse. Mistral Vibe was easy: 13 files, 1 commit, a single ZAIAdapter extending the OpenAI adapter. Gemini CLI took 49 files and 6+ commits, including an 812-line protocol translator. Codex was the worst — deep Rust surgery to translate OpenAI’s Responses API, losing prompt caching and native shell tool calls along the way, with 5-minute rebuild cycles. The author’s ranking: “Adapter pattern >> translation layer >> type system surgery.”
Five Architectural Dimensions Behind the Gap
The study isolates five design decisions that actually move the score:
- File editing: forgiving edit tools win — OpenCode ships a 9-strategy fallback chain; Codex’s custom patch DSL is a dialect GPT was fine-tuned on, and it punishes non-frontier models
- Sandboxing: the spectrum runs from Codex’s five OS-level layers (bubblewrap, seccomp) to OpenCode’s “nothing at all, approval prompts only”
- Context management: Mistral Vibe’s proactive compaction beats Codex’s reactive truncation on multi-step tasks
- Error handling: Mistral Vibe’s two-layer split (silent infrastructure errors, visible task errors) is called the best compromise
- Memory: only Codex genuinely learns across sessions, via a two-phase memory extraction pipeline
The Ranking Survives an Upgrade to GLM-5
The author then re-ran everything on GLM-5: Mistral Vibe went from 0.35 to 0.48 (+38%), Claude Code from 0.29 to 0.40 (+40%), and Gemini CLI from 0.23 to 0.35 (preliminary, 34 of 89 tasks, +54%). The ordering did not budge, and the Vibe-to-Claude-Code ratio barely moved from 1.21x to 1.20x — architectural advantages persist across model generations. The cost: GLM-5 timed out far more often, on 24 of 88 tasks for Claude Code (27%) and 21 of 89 for Mistral Vibe (24%).
Lessons for Harness Builders
The author’s conclusion is one line: “Scaffolding still dominates.” For teams choosing or building an agent harness, three takeaways. First, pick the harness before the model — migration cost and vendor coupling are decided by the adapter design, and custom protocols are long-term liabilities. Second, open-weight models are now genuinely competitive in the right harness; from GLM here to purpose-built agentic coders like Qwen3.6-35B-A3B, they belong on any serious shortlist. Third, stop treating model upgrades as the only lever — engineering choices like edit-tool forgiveness and compaction timing contribute on the same order of magnitude as a model generation. As the author closes: the most interesting thing about coding agents isn’t the loop — it’s everything around it.
Sources
- I Forked 4 CLI coding agents to Run the Same Model. I found a 2x gap — Charles Azam
- Show HN discussion — Hacker News
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.