Summary: LangChain published an FAQ on agent evals on October 15, 2026, written by Divya Santhanam from the LangSmith team. It answers the questions teams actually ask when their agents stop being unit-testable: where to start, offline versus online evals, which evaluator types to use, and whether an LLM judge can be trusted. The useful part is how small the first step turns out to be.
Why agents broke unit testing
Traditional software gives you the same output for the same input, so an exact-match assertion works. Agents fail that assumption three ways, according to the LangChain post: outputs vary between runs, more than one answer can be correct, and the agent picks its own path — so a right answer can arrive via a tool call it never needed, or an action it took without permission.
Agent evals score runs with checks that tolerate that variance. The catch, which the post is honest about, is that there is no standard suite. Your evals depend on your product, your team, and what you’re optimizing for.
The starting recipe is smaller than you think
The recommended first move: read a sample of traces, pick one behavior you care about (for a travel assistant, whether it books the date and price range the user asked for), and build a dataset of ten to twenty examples from real traces. Then run a baseline experiment so future agent versions have something to beat. Every production failure you find afterward becomes a new dataset example, and that loop — find, add, improve, rerun — is the whole improvement cycle.
If you haven’t shipped yet, LangChain suggests dogfooding with your team or design partners and mining those traces instead.
Offline and online evals feed each other
Offline evals run before you ship: fixed inputs, optional reference outputs, comparable experiments across versions. Online evals score production traces as they arrive, usually with no reference output, using code checks and LLM judges. You need both — failures the online evals catch become new offline examples. That trace-to-dataset habit is the same discipline behind tracing agent calls in production; if you’re already piping Claude API calls through OpenTelemetry, you have the raw material (we’ve covered that setup before).
Dataset hygiene matters too: keep examples together when they test the same behavior with the same evaluators, split into a new dataset when inputs or purposes differ — say, a small PR-gated set versus a larger pre-release one.
Four evaluators, aimed at four targets
LangChain names four evaluator types: code evaluators (deterministic, cheap, limited), LLM-as-a-judge (flexible, needs calibration), human review (slow but the ground truth for subjective quality), and pairwise comparison for choosing between two prompts or models. Any of them can score a single step, the final response, a full conversation, or the end state of an environment.
On trusting LLM judges, the advice is concrete: compare judge scores against human scores on a sample, ask narrow questions (“does the reply mention every layover?” beats “is this good?”), and prefer binary pass/fail or labeled categories over 1-to-10 scales, which judges struggle to apply consistently.
Benchmarks are just offline evals with containers
When agents execute code and edit files, a judge reading the reply isn’t enough. Benchmarks like Terminal-Bench give each task its own containerized environment and score what the agent did — a test script inspects the state left behind, like whether booking records match the request. LangChain’s Harbor, the open source harness behind Terminal-Bench, logs each run as a LangSmith experiment. The team says they used this setup to improve Deep Agents on Terminal-Bench 2.0 and to build IssueBench for LangSmith Engine.
The takeaway for your own agent: the same loop works at any scale. Pull failures from traces, turn the recurring ones into containerized tasks, define what good looks like, and hill climb on the score. Start with ten traces, not a benchmark.