Summary: On November 6, 2026, Pydantic AI became a supported agent inside AgentEnv Framework, Scale AI’s open source system for testing agents against realistic apps and data instead of toy prompts. If you ship Pydantic AI agents, you can now run them through reproducible multi-step tasks with verifiers — and swap in other agents on the exact same setup for head-to-head comparison.
Why a fixed dataset is not enough
The pitch from the Pydantic team’s announcement starts with a builder pain point: an eval on a static dataset catches regressions, but it misses the agent that sends the right email to the wrong person because the inbox looked different yesterday. AgentEnv Framework attacks that by making the test itself a live, seeded world rather than a list of prompts.
The mechanics: an environment is a composable stack of realistic apps — email, Slack, a help desk — preloaded with seed data. A task is a graph of steps that deploys the environment, deploys an agent, prompts it, and verifies the outcome. Tasks pin their versions, so rerunning a task reproduces the original conditions. Environments expose their tools over MCP, and agents connect over A2A, which is why one environment can host many tasks and many agents on equal footing.
What actually runs when you pick pydantic-ai
The integration is not a wrapper around a CLI. The pydantic-ai agent runs Pydantic AI in process, with each task being one run of an Agent using the Coder capability from Pydantic AI Harness — the same coding toolset CLAI is built on: shell, read_file, write_file, edit_file, list_files, grep, and delegate_task. There is a sensible security boundary here: shell commands can see PATH, HOME, and the locale, but never the model key.
Model routing is flexible. It talks to any OpenAI-compatible endpoint, and a vllm: prefix on the model name switches to Pydantic AI’s vLLM provider, which shapes requests for the model’s family — Qwen, DeepSeek, GLM, Llama, and others — the way a vLLM server expects.
State handling is worth knowing before your first run. Each context maps to one conversation kept on the agent’s own disk, so follow-up tasks in that context continue the same conversation while the agent is up. Moving to a new instance goes through the snapshot flow, which saves and restores the conversation and, if requested, the workspace. MCP servers, skills, peer agents, and live trajectories all arrive through standard A2A extensions.
The smoke test, end to end
Getting started is two commands:
uv tool install agentenv-framework
agent-env run helloNothing is configured by default, so you must supply a model endpoint and export PYDANTIC_AI_GATEWAY_API_KEY (the env: prefix in the config reads it at runtime). The announcement’s example points at the Logfire Gateway — an EU organization gets gateway-eu.pydantic.dev instead of gateway-us.pydantic.dev, and you keep the base URL as-is because the agent appends /chat/completions itself:
# .agentenv/config.toml
[model]
base_url = "https://gateway-us.pydantic.dev/proxy/openai"
api_key = "env:PYDANTIC_AI_GATEWAY_API_KEY"
[model.roles]
agent = "gpt-5.5"
[agents]
default_a2a_agent_id = "pydantic-ai"The framework ships an example bundle called mail-mcp: an MCP server with a fixed inbox of three emails, one of which is an invoice, plus a verifier that checks the invoice can be found. Its stock task only deploys the environment and runs the verifier. To put Pydantic AI in the loop, you add two steps to tasks/smoke.json — a deploy_agent step and a prompt_agent step asking the agent to find the invoice and say who sent it, followed by the outcome verifier:
[
{ "id": "env", "type": "deploy_env", "env_id": "mail" },
{ "id": "assistant", "type": "deploy_agent", "env_ids": ["mail"], "a2a_agent_id": "pydantic-ai", "agent_name": "assistant", "depends_on": [ { "task_step_id": "env" } ] },
{ "id": "find-invoice", "type": "prompt_agent", "agent_name": "assistant", "prompt_id": "find-invoice", "prompt": "Find the invoice in the inbox and tell me who sent it.", "depends_on": [ { "task_step_id": "assistant" } ] },
{ "id": "finds-the-invoice", "type": "env_outcome_verifier", "env_id": "mail", "file_artifact_id": "check_mail", "verifier_id": "finds-the-invoice", "depends_on": [ { "task_step_id": "find-invoice" } ] }
]Run the copy with agent-env run ./my-mail. The agent receives the inbox’s search_emails tool over MCP, its reply and full trajectory land in the run output, the verifier scores the result, and — if you turned on telemetry under Gateway, Settings, Telemetry — every model call shows up as a span in Logfire with zero changes to the agent code.
The comparison trick is the part I’d actually use: change a2a_agent_id to claude-code and run again. Same environment, same prompt, same verifier, different agent. That is a controlled A/B on your eval infrastructure, not a vibes-based bake-off.
Costs, limits, and what the source does not say
Two practical caveats from the announcement. First, the initial run builds the gateway the environment deploys on, and that takes a few minutes — budget for it before assuming your smoke test is broken. Second, eval realism ultimately depends on the environment’s seed data; the framework makes runs reproducible, but it cannot make thin seed data informative.
The source also establishes no benchmarks, pass rates, or performance comparisons between Pydantic AI and other supported agents, and it says nothing about production adoption or long-term stability of the integration. Treat this as a new testing capability, not a proven leaderboard result.
What to do next
Install the framework, run the hello task, then extend the mail-mcp bundle until it resembles a workflow you actually own — a triage queue, a refund pipeline, whatever your agents touch. Since the pydantic-ai agent here is the same agent you run in production, a passing task is meaningful signal about your real deployment. If you’re already tuning prompts for cache efficiency in Pydantic AI, as we covered on prompt caching earlier, this gives you a place to verify those prompts still behave under realistic multi-step load.
Sources
View the original post on X