Summary: Developer Charles Azam forked three open-source coding agents, standardized them on GLM-4.7, and ran them alongside Claude Code on 89 Terminal-Bench 2.0 tasks. With the model fixed, Mistral Vibe scored 0.35 and Codex 0.15, a gap above two times attributable to harness design. Re-running selected agents on GLM-5 improved scores without changing the ranking.
Developer Charles Azam published a deep-dive experiment on April 22, 2026. He worked through the source of four open-source coding agents end to end, then forked three to run the identical model — GLM-4.7 — and ran them against the 89 real-world tasks of Terminal-Bench 2.0. Claude Code is closed-source and couldn’t be forked; it competed natively over ZAI’s Anthropic-compatible endpoint.
The point: model quality usually takes all the credit or blame for how an agent performs. By freezing the model, this study measures the harness alone — identical model, identical tasks — and still finds the strongest and weakest scaffolds separated by better than 2x.
How the Test Was Built: GLM-4.7 Everywhere, Terminal-Bench 2.0 as the Course
The four entrants grew out of distinct engineering cultures: Codex comes from OpenAI as a 52-crate Rust Cargo workspace, Gemini CLI is Google’s TypeScript/React monorepo, Mistral Vibe is Python 3.12+, and OpenCode is TypeScript on Bun. The shared engine is ZAI’s open-weight GLM-4.7, which the author places alongside Claude Sonnet 4.5 on Artificial Analysis.
The course is Terminal-Bench 2.0, executed through the Harbor harness: 89 tasks, each Docker-isolated and time-limited, spanning debugging, multi-file refactoring, test generation, and build fixing. Each score averages 5 runs, wobbling by about 5%. The whole thing consumed roughly 5 billion tokens on ZAI’s $60/month subscription — about 100 tokens per second, with timeouts a routine occurrence.
Where They Landed: 0.35 for Vibe, 0.15 for Codex, and a 2x Spread
Final standings on GLM-4.7: Mistral Vibe 0.35, Claude Code 0.29, Gemini CLI 0.23, OpenCode 0.21, Codex 0.15. For scale, Terminal-Bench’s official leaderboard has Claude Code and Terminus 2 around 0.33 with GLM-4.7 over the direct API — same ballpark. Asked where the spread comes from, the author’s answer is blunt: “the code around the model.”
Fork difficulty tells the mirror-image story. Mistral Vibe was the light lift: 13 files, 1 commit, and a single ZAIAdapter extending the OpenAI adapter. Gemini CLI demanded 49 files and 6+ commits, among them an 812-line protocol translator. Codex was the hardest by far — major surgery deep in the Rust code to re-express OpenAI’s Responses API, prompt caching and native shell tool calls dropped along the way, plus 5-minute rebuild cycles. The author’s verdict on these approaches: “Adapter pattern >> translation layer >> type system surgery.”
Five Design Decisions That Move the Score
The analysis pulls out five design choices that visibly change the outcome:
- File editing: tools that tolerate imperfect edits come out ahead — OpenCode carries a 9-strategy fallback chain, while Codex’s custom patch DSL is a dialect GPT was fine-tuned on and penalizes anything non-frontier
- Sandboxing: at one extreme sits Codex with five OS-level layers (bubblewrap, seccomp); at the other, OpenCode with “nothing at all, approval prompts only”
- Context management: across multi-step tasks, Mistral Vibe compacts proactively and comes out ahead of Codex, which truncates reactively
- Error handling: the two-layer split credited to Mistral Vibe — infrastructure errors handled silently, task errors kept visible — gets named the best compromise
- Memory: Codex alone truly learns across sessions, drawing on a two-phase memory extraction pipeline
Swap in GLM-5 and the Order Holds
The author then ran it all again on GLM-5. Mistral Vibe climbed from 0.35 to 0.48 (+38%), Claude Code from 0.29 to 0.40 (+40%), and Gemini CLI from 0.23 to 0.35 (+54%, preliminary, over 34 of 89 tasks). Nobody changed places, and the ratio of Vibe’s score to Claude Code’s barely shifted, from 1.21x to 1.20x — whatever an architecture gives you survives a model generation. The price: GLM-5 ran over the time limit far more often — on 24 of 88 tasks for Claude Code (27%) and 21 of 89 for Mistral Vibe (24%).
What Harness Builders Should Take Away
The whole study lands on one line: “Scaffolding still dominates.” For anyone choosing or building an agent harness, three takeaways. One: settle on the harness before the model — migration cost and vendor coupling get fixed at the adapter layer, and a custom protocol is a long-term liability. Two: open-weight models now compete for real in a suitable harness; from the GLM tested here to purpose-built agentic coders like Qwen3.6-35B-A3B, they belong on any serious shortlist. Three: don’t treat a model upgrade as the only lever — engineering calls like how forgiving the edit tools are and when compaction kicks in move results on the same order of magnitude as a model generation. The author’s parting thought: what makes coding agents interesting isn’t the loop, it’s everything built around it.