Summary: When the answer an agent hands back can decide where a patrol vessel goes, reliability stops being something a benchmark score can express. That framing moves agent reliability out of the model problem and into the engineering problem, and the post’s value is splitting it into four layers a builder can act on. Shippy is the maritime agent built by Ai2’s Skylight team, working in maritime domain awareness — a setting the team puts bluntly: a wrong answer has real impacts.
When the answer an agent hands back can decide where a patrol vessel goes, reliability stops being something a benchmark score can express. Shippy is the maritime agent built by Ai2’s Skylight team, working in maritime domain awareness — a setting the team puts bluntly: a wrong answer has real impacts. In their technical post they reach a direct conclusion: in high-stakes operational domains, the difficulty is not in choosing a stronger model. It is in constructing a system that can be trusted to be correct, to hold inside its own limits, and to remain stable from one task to the next — validated against live data that keeps moving.
That framing moves agent reliability out of the model problem and into the engineering problem, and the post’s value is splitting it into four layers a builder can act on.
Breaking the agent into three parts: soul, skills, config
Shippy splits the definition of its agent into three pieces, each one free to evolve on its own:
| Layer | Contents | Responsibility |
|---|---|---|
| Soul | system prompt | persona and the boundaries on behavior |
| Skills | markdown with structured frontmatter | workflows that handle particular request types |
| Config | everything else in the runtime settings | harness, model, runtime |
Skills are written to the same agent-skills spec that Claude Code and Codex use, and they reach across Skylight API queries (events, vessel data), EEZ and MPA boundaries, vessel-track interpretation, and deep links into the interactive map. The track-reading skill sits on top of the activity classifications Skylight’s existing models — Atlantes among them — already produce, instead of sending the agent back to re-read raw signals.
Soul and skills are packaged into a versioned Docker image, with secrets injected only at runtime. Pointing the system at a different LLM or agent harness is a config change rather than a rebuild — the harness in place today is OpenClaw, an open-source agent framework, running Claude Opus 4.6 as the model. Where that separation truly pays off is governance: Shippy issues no legal judgment on whether a vessel is breaking the law, and it does not speculate past what the data will support. Those no-go zones are stated in the system prompt rather than buried in fine-tuning, which leaves them auditable, version-controlled, and eligible for review and eval.
Deterministic tools: wrapping a complex API in a purpose-built CLI
Early prototypes had Shippy assembling API calls from scratch. The Skylight API carries dozens of input types, nested filters, pagination cursors, and involved geometry, and the outcome was a steady trickle of subtle bugs: malformed pagination quietly dropping results, geometry encoded incorrectly, and queries that looked right while returning wrong data because a filter type had been misread. What makes this class of bug unpleasant is that it looks correct — the agent carries on reasoning over bad data, and only someone downstream eventually notices.
The remedy is to converge the interface. Shippy issues no raw API calls; everything travels through a purpose-built CLI, where a single skylight events search with typed filter flags does the work and the CLI handles authentication, pagination, and structured output. The CLI ships with full --help and error messages clear enough for an agent to recover from its own mistake rather than guess. Output never goes through a shell pipe; it lands in a local JSON file, because large result sets once hit pipe buffer limits and broke downstream tools such as jq, and having it on disk lets later steps reuse the same results. Place-name resolution follows the same rule: asked about “fishing activity in Panama’s EEZ,” the skill calls Skylight’s regions API to turn the name into a boundary polygon rather than guessing or hard-coding coordinates, and it credits results with deep links and sources like Global Fishing Watch and TMT.
Beneath the CLI sits a standardized API: events, vessels, regions, satellite imagery, and vessel tracks all expose two operations — search and aggregate — with inputs and outputs defined as typed schemas carrying field-level descriptions. The point of the layering is independent testability: the API keeps its own suite, a human or an agent can drive the CLI, and skills point at CLI commands instead of reinventing them. Every layer trims the room the layer below has to get something wrong.
Mothership: a sandbox for every session
Skylight is a free platform that, by the team’s own numbers, serves 300+ partners in 70+ countries, mostly government agencies and NGOs, which makes multi-tenant isolation one of the project’s biggest engineering challenges. So the team built Mothership, an agent hosting platform that gives every user session a dedicated Kubernetes deployment. Pods carry the agent runtime, skills, and CLI; the user’s Skylight JWT is injected at provisioning, so API calls are scoped automatically to that user’s data; nothing created in a session is shared across users; the sandbox can write code and run multi-step analysis, but its network reaches only the services it needs.
The detail worth pausing on is where isolation lives: not in filtering at the application layer, but in runtime injection and the deployment boundary. Filtering at the application layer rests on “remembering to check every time.” Mothership rests on something else: “there is no path across the boundary at all.”
Grading the whole agent: live data plus an LLM judge
A static benchmark cannot grade an agent that is wired into a system — how it picks tools, how it queries live data, when it decides to stop. Shippy’s eval therefore scores the whole agent (model plus skills plus sandbox) on real data. Subject-matter experts write the scenarios and rubrics, set the weights, and annotate ground truth; an LLM judge scores every criterion from 0 to 1 and writes out its reasoning for passing or failing it; the weighted total is measured against a fixed pass threshold.
On the execution side, a plugin written for Harbor, an open evaluation framework, launches a real Shippy session on the version under test, runs it in parallel against the same real data users encounter, and produces timestamped results plus a report comparing scores with the prior run. Any change to skills, to the model, or to the underlying data triggers another run. The positioning that matters: eval is the release gate — a version that regresses does not reach end users.
What works today, and the failure modes that persist
The most recent eval round showed guardrail tasks holding steady: military intelligence requests refused correctly, user data isolation maintained, sources attributed accurately. The failure modes were equally specific: patrol-planning tasks where Shippy crossed over into tactical recommendations, boundary simplification that caused events to be missed, and one instance where the agent invented a CLI command that does not exist. A model benchmark shows none of this, and every item went straight back into revising the skills.
Limits, and guidance for builders
The limits deserve the same candor. There is no cross-thread memory yet, which means an analyst restates their jurisdiction each session. The map returns deep links only, with agent-driven UI control still ahead. Easy and hard queries currently share one model, and the team intends to add model routing that sends simple lookups to smaller ones. Shippy is opening to early adopters on a rolling basis so it can be stress-tested. Mothership itself was designed from the start as general agent hosting, and these lessons are slated to move into Ai2’s other projects, beginning with EarthRanger.
Three takeaways a builder can carry off. First, keep behavioral boundaries in a version-controlled system prompt rather than in fine-tuning or in team folklore — only boundaries you can audit are boundaries you can test. Second, instead of putting a complex API in front of the agent as-is, spend the effort on a typed, self-documenting CLI, which leaves the nondeterminism in the model and the determinism in the tools. Third, make eval a release gate rather than an after-action report: whole-agent, live data, weighted rubrics, with regressions blocked before they reach end users. Reliability in production agents is not something a stronger model buys you; it accumulates as you shrink the error surface one layer at a time.