Summary: Hugging Face published its agent-tooling benchmark on June 18 to measure work behind answers, not only match rates. It scores match rate, median time and tokens, silent-failure error rate, and marker adoption across bare, clone, and skill tiers. In the study, large models gained speed, while small models saw no gain or lost correctness and spent far more tokens.
Hand the same sentiment-classification task to two agents and both can land on the right label — by utterly different routes. One writes dozens of lines of Python, pulls in Transformers, trips over a shape error, and runs everything twice more; the other fires off a single line of CLI. Conventional benchmarks log these as two interchangeable wins, yet what actually reaches your product’s cost line is the journey: turns spent, tokens burned, failures along the way, and whether the official tool was ever touched. “Is it agentic enough?”, which Hugging Face published on June 18, aims straight at that blind spot — it worries less about whether the answer is right than about how much work the agent put in to arrive at it.
The question has history. After Hugging Face reshaped the hf CLI into an agent-optimized form, agents burned roughly 1.3–1.8× fewer tokens, with 6× at the top end. The gains are real, but proving the worth of every later tool change demands a measurement you can rerun — so the benchmark got built first, and the changes followed.
Grading the Answer Alone Hides What It Really Cost
Answer-only grading carries two blind spots. Large open models have pushed completion on routine tasks close to saturation, so match rates no longer tell them apart; what still separates them is effort — turns, tokens, time elapsed, and whether the route stayed clean. Then there is the agent that skips your official API and rebuilds the logic itself: its answer is still correct, but at a completely different cost. The benchmark responds with four scoring axes: match rate (substring, regex, or exact), median time and tokens (new, cached, and generated counted separately), error rate (with a guard for silent failures — zero output and no tool calls), and marker adoption, which scores behavior rather than outcome.
Three Execution Tiers Turn “Was the Documentation Read?” into a Measurable Variable
Every task runs once under each of three tiers, and no tier contains another. Bare is pip install transformers, nothing else. Clone is a complete checkout of the source. Skill stuffs the CLI documentation plus task examples into the agent’s context. The three setups mirror the three ways an agent can actually know your tool: from training memory alone, by reading the source, or by having the docs handed over explicitly.
For the moment the harness accepts deterministic tasks only — ones whose answers can be exact-matched without ambiguity; model-as-a-judge remains future work. The conservatism is on purpose: lay the measurement foundation first, then stretch toward open-ended tasks.
Inside the Harness
The whole thing ships as one CLI: agent-eval. Once you define tasks and expected answers, it fans every models × revisions × tasks combination out to Hugging Face Jobs — each run becomes its own independent job tied to one model, one revision, and one task, and the entire sweep runs in parallel on identical hardware, squeezing environmental variance down to a minimum. Open models do the work, driven through the pi coding agent, and the finished report is published as an HF Space with interactive visualizations and a shared-tasks-only switch, so different models can be compared like-for-like.
Markers are what turn “what I want the agent to do” into patterns you can measure. A profile — a compact per-tool plugin that shows the harness how to build and drive a given library — supplies the definitions, and a marker can match shell commands, code, files the agent read, or the final answer. The cli marker, for one, records that the agent reached for the transformers CLI instead of writing Python by hand.
Large Models Bank Time and Pay in Tokens
The Transformers study pins one strong model in place and walks the tool through its git revisions — v5.8.0, then v5.9.0, out to the commit where the CLI and Skill land. Kimi-K2.6, GLM-5.1, and MiniMax-M2.7, the three large open models in the test, all agree: the Skill commit cuts the time a task takes. This CLI arrived in a single commit and sits in no model’s training data, so bare and clone runs essentially never pick it up; on the skill tier, 55.3% of runs genuinely do.
The savings are only half the ledger. On the clone tier, close to one run in three reads the new interface first, and median input tokens climb from about 4k to 6.4k — less time, more tokens, a genuine trade-off.
Small Models, Opposite Outcome: the Same Change Damages Correctness
The small-model experiment flips the design over: the revision carrying CLI and Skill is held fixed while the sweep moves across models, with size, quantization, and provider all treated as variables. The story turns. On the clone tier, Qwen3-4B digs heavily through the new CLI’s source — median new tokens leap from about 2.4k to 23k — while time and output swell along with it and the match rate gains nothing. Qwen3-14B has it worse: once the Skill is in place, match slides from 67% on bare to 43%, classify-sentiment plunges from 100% on clone to 0%, and 39 of the 56 skill runs either mistake the CLI for a directly callable tool or give up outright.
One change, two destinies: speed for strong models, damage for weak ones. Without trajectory-level measurement, a maintainer could ship CLI + Skill convinced it helps everyone, while it actually plants friction in front of part of the audience.
Practical Guidance for Tool Maintainers
Nothing about the method is exclusive to Hugging Face; any tool you can drive from a CLI can copy it. Define a few deterministic tasks, mark the behavior you expect in a profile, sweep the model sizes you plan to support, and then read the distributions of match, tokens, errors, and markers. A few judgments worth taking away: validate API changes across model sizes, because the benefit is lopsided — in this case the winners and the casualties were the largest and the smallest models; documentation inside context is not the same as a tool, and since a Skill is context rather than callable, weak models will mix the two up, so the docs should explicitly direct “read, then invoke via shell”; and do not stop at match rate — silent failures and marker adoption tend to expose interface problems sooner than wrong answers do.
Hugging Face turned its conclusion into a product as well: the companion project Upskill emits a Skill only when measurement proves it genuinely helps small models. One safety boundary to close on: the harness runs the coding agent with bypassed permissions, so keep it to local execution on trusted code.