Summary: Microsoft’s Responsible AI team released ASSERT on GitHub on June 2, 2026 as an MIT-licensed framework that turns plain-language behavior specs into AI test scenarios. It executes single-turn and multi-turn cases, scores impermissible and permissible-behavior violations separately, records tool-call paths, and integrates with LiteLLM, tracing frameworks, Docker, and CI. The article positions it as application-specific regression testing rather than a replacement for public benchmarks.
ASSERT — short for Adaptive Spec-driven Scoring for Evaluation and Regression Testing — arrived on GitHub as an open-source release from Microsoft’s Responsible AI team on June 2, 2026. Describe in plain language how your AI system is supposed to behave, and the framework builds test scenarios from that description, runs them, and assigns scores. The project is published under the MIT license, and its CLI package goes by the name assert-ai.
Any team shipping AI features will recognize the gap this tool aims at. General evaluations covering safety, compliance, sycophancy, and alignment exist on the lab side, but what a company actually needs to confirm is whether its agent plays by the rules inside its own product context — territory public benchmarks never touch. ASSERT is Microsoft’s bid to move that verification out of artisanal, hand-finished effort and into everyday infrastructure.
Getting to Know ASSERT
Four groups within Microsoft’s Responsible AI organization — product, science, engineering, and design — built ASSERT, which positions itself as a requirement-driven evaluation harness. What you feed it doesn’t have to be a meticulously drafted specification: a PRD, a threat model, or even the incident report from a past failure can serve as the starting point. The harness is meant to be used at three moments — while the system is being built, after it ships, and as continuous monitoring over time.
From a Written Spec to a Full Test Suite
Four steps make up the pipeline. The developer starts by describing goals, policies, and behavioral boundaries in high-level natural language. ASSERT reshapes those descriptions into a structured list of acceptable and unacceptable behaviors. Next come problem scenarios and test cases — single-turn and multi-turn conversations included — which run against the target system; an LLM judge then grades each conversation against your policies. Execution paths are recorded along the way, intermediate actions and tool calls among them, so you can trace exactly which step went off the rails instead of merely learning that something failed. To fit the evaluation to their own system, developers can additionally provide system context, tools, and constraints.
The scenario TechCrunch describes is concrete: a document research agent that may not email anyone outside the company, may hand confidential information only to C-level executives, and must write concise summaries that preserve prior context. For each of those rules, ASSERT produces tests checking that the system keeps complying over time.
Two Metrics, a Docker Sandbox, and CI Gates
Two headline metrics come out of a run. Impermissible behavior violations sit on the harm side — the agent did something it may never do. Permissible behavior violations sit on the trade-off side — something it should have done went undone. Keeping the two categories scored separately stops one blended total from obscuring what kind of failure actually happened.
The integrations reach far. LiteLLM opens up more than 100 model endpoints; OpenInference/OTel tracing brings in agent frameworks such as LangGraph, CrewAI, and AutoGen. High-risk tool calls can be exercised safely inside the built-in Docker sandbox, and a curated preset library of behaviors comes stocked with safety, fairness, and agentic failure modes. Evaluation output takes the form of local-first JSON/JSONL files with a web viewer for browsing, and a GitHub Action stands watch against regressions in CI. There are two ways in: a guided skill aimed at coding assistants like Claude Code, Copilot, and Cursor, or a hands-on CLI. The framework can even derive a governance policy from real, observed failures — the ACS step — and then run the evaluation again to quantify how much things improved.
Why This Moment
“One of the things we’ve learned is that evaluations are absolutely critical to making good decisions.” Sarah Bird, chief product officer of Responsible AI at Microsoft, put it that plainly. Without a grasp of how a system behaves, “it’s really hard to know if it’s meeting your organization’s bar” — and trustworthy systems call for evaluation across “many more dimensions” at the application level.
The larger backdrop is an industry-wide turn toward repeatable testing and regression checks as models grow more capable. Stanford’s HELM, MLCommons’ AILuminate, and METR’s evaluations hold up the public-benchmark side of that turn; the application side is where ASSERT slots in. According to the repo, the method follows Agarwal et al. (2026) on AI-assisted systematization, and the pipeline design borrows ideas from Anthropic’s open-source Bloom and Petri behavioral-evaluation frameworks.
Three Takeaways for Developers
The practical effects number three. Behavior verification shifts left: tests are generated from the spec during development, so a misdirected email surfaces in a test run instead of through customer complaints. Evaluations become files: local JSON/JSONL artifacts together with a CI gate make “does the agent still follow policy” a check that can be diffed and reviewed like any other. And the tool complements public benchmarks rather than displacing them — general evals speak to whether a model is good, while ASSERT speaks to whether it behaves inside your product. Both layers matter, and the second has always been the one short on decent tooling.