On June 2, 2026, Microsoft’s Responsible AI team open-sourced ASSERT — Adaptive Spec-driven Scoring for Evaluation and Regression Testing — a framework that turns plain-language descriptions of how an AI system should behave into generated, scored test suites. It is available now on GitHub under the MIT license, with a CLI package named assert-ai.
The gap it targets is familiar to anyone shipping AI features. Frontier labs publish general evaluations for safety, compliance, sycophancy, and alignment, but a company still has to verify that an agent behaves as intended inside its own product — something public benchmarks simply do not cover. ASSERT is Microsoft’s attempt to turn that check from artisanal work into routine infrastructure.
What ASSERT Is
ASSERT was built by the product, science, engineering, and design teams inside Microsoft’s Responsible AI organization, and it is positioned as a requirement-driven evaluation harness. The input does not have to be a carefully drafted spec: you can point it at a PRD, a threat model, or even an incident report from a past failure. It is designed to run while you build the system, after deployment, and as continuous monitoring.
From Spec to Test Suite
The pipeline has four steps. A developer writes high-level, natural-language descriptions of the system’s goals, policies, and behavioral boundaries. ASSERT converts them into a structured set of acceptable and unacceptable behaviors. It then generates problem scenarios and test cases — single-turn and multi-turn — and runs them against the target system. Finally, an LLM judge scores the conversations against your policies. Along the way it records execution paths, including intermediate actions and tool calls, so you can see where a run went wrong, not merely that it did. Developers can optionally supply system context, tools, and constraints to customize the evaluation.
The example TechCrunch gives is concrete: a document research agent that must never email people outside the company, must restrict confidential information to C-level executives, and must produce concise summaries that keep prior context. ASSERT generates tests that verify ongoing compliance with each of those rules.
Metrics, Sandbox, and CI Gating
ASSERT reports two headline metrics: impermissible behavior violations — the harm side, where an agent does something it never may — and permissible behavior violations — the trade-off side, where it should do something and does not. Splitting the two keeps a single aggregate score from hiding the nature of the failure.
The integration surface is broad. It reaches hosted models through LiteLLM, covering more than 100 endpoints, and agent frameworks such as LangGraph, CrewAI, and AutoGen through OpenInference/OTel tracing. A Docker sandbox lets you exercise risky tool calls safely, and a curated preset library covers safety, fairness, and common agentic failure modes. Artifacts are local-first JSON/JSONL files with a bundled web viewer, and a GitHub Action gates regressions inside CI. There are two entry points: a guided skill for coding assistants like Claude Code, Copilot, and Cursor, or a manual CLI. ASSERT can even generate a governance policy from observed failures — its ACS step — and re-run the evaluation to measure how much the policy improved things.
Why Now
Sarah Bird, chief product officer of Responsible AI at Microsoft, put it bluntly: “One of the things we’ve learned is that evaluations are absolutely critical to making good decisions.” Without understanding a system’s behavior, “it’s really hard to know if it’s meeting your organization’s bar,” and trustworthy systems require evaluating “many more dimensions” that are application-specific.
The backdrop is an industry-wide shift toward repeatable testing and regression checks as models grow more capable. Stanford’s HELM, MLCommons’ AILuminate benchmark, and METR’s evaluations anchor the public-benchmark side of that shift; ASSERT aims at the application side. The repository notes that the method follows Agarwal et al. (2026) on AI-assisted systematization, and that the pipeline design draws inspiration from Anthropic’s open-source Bloom and Petri behavioral-evaluation frameworks.
What It Means for Developers
Three practical effects. First, behavior verification moves left: generate tests from your spec during development, instead of discovering via customer complaints that your agent emailed the wrong person. Second, evaluations become files: local JSON/JSONL artifacts plus a CI gate turn “does the agent still follow policy” into a diffable, reviewable check. Third, this complements rather than replaces public benchmarks — general evals answer whether a model is good; ASSERT answers whether it behaves in your product. Teams need both layers, and the second one has always been the one missing decent tooling.
Sources
- New Microsoft tool lets devs spin up AI behavior tests using text descriptions — TechCrunch
- responsibleai/ASSERT — GitHub
- AILuminate — MLCommons
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.