On January 3, 2026, NBC News published a field report from the Alaska Court System: to help residents navigate probate without a lawyer, the courts built a generative AI chatbot called AVA, the Alaska Virtual Assistant. The project was scoped as a three-month effort. It ran for fifteen months, shed most of its ambition along the way, and is now scheduled to launch in late January 2026 — “if all goes as planned.”
The value of this story is not “AI project struggles.” It is the detail. A Q&A bot grounded in official court documents looks like the safest possible shape for generative AI. Under a 100% accuracy requirement, though, the engineering bears no resemblance to the demo. For any team building AI products for law, medicine, or government services, AVA is close to a textbook case.
From Three Months to Fifteen
AVA was built by Tom Martin, a lawyer and founder of legal tech firm LawDroid, with seed funding from the National Center for State Courts (NCSC), which also stationed consultant Aubrie Souza on the project. The problem it targets is concrete: transferring a deceased person’s estate is paperwork-heavy, most families have no attorney, and the court’s self-help resources and helpline are permanently understaffed.
The technical approach is not exotic: an enhanced retrieval-augmented generation pipeline that anchors every answer to the court’s existing self-help web content, with citations to verifiable sources and suggested follow-up questions. The team repeatedly compared OpenAI and Anthropic models for “accuracy and tone,” and built a knowledge base to block the model from leaning on outside sources. The Thomson Reuters Institute was already describing AVA as “in the final testing phase before launching” in February 2025 — nearly a year before NBC’s report.
What Hallucination Looks Like in Probate Law
The failures are worth reading closely if you build vertical-domain bots. AVA invented resources — at one point directing users to seek help from an Alaska law school alumni network that does not exist. Stacey Marz, administrative director of the Alaska Court System, put the requirement bluntly: “With a project like this, we need to be 100% accurate, and that’s really difficult with this technology.” Souza’s version: they “had trouble with hallucinations, regardless of the model.”
Tone turned out to be its own minefield. Early versions were relentlessly empathetic, offering condolences to every grieving user — until one fed-back user said, “I’m tired of everybody in my life telling me that they’re sorry for my loss.” The team removed the sympathy entirely. Developer Martin’s observation about model behavior is equally useful: some models are poor rule-followers and “kind of want to prove that they’re the smartest guy in the room” — precisely the trait a legal tool must never have.
The Reality of Testing and Cost
To evaluate accuracy, the team designed a 91-question probate test suite. Running it in full was too labor-intensive, so it was cut to 16 questions. Marz described the whole effort as “just so very labor-intensive.” The cost structure is therefore easy to read: inference is trivially cheap — about 11 cents for 20 AVA queries under one setup — while human review, content preparation, and iteration are where the budget actually goes.
The launch design shows the retreat in every corner. The official FAQ states plainly that AVA “uses artificial intelligence (AI) to answer your questions and may make mistakes.” Every answer carries links back to the court’s self-help probate pages so users can verify against the source of truth. And the bot no longer attempts to replicate the full role of human facilitators — as Marz put it, “We’re not confident that the bots can work in that fashion.”
Three Lessons for High-Stakes Deployment
AVA’s fifteen months compress into three lessons for product teams.
First, treat scope reduction as a legitimate engineering move. AVA earned its launch ticket by retreating from “virtual facilitator” to “document navigator” — the smaller scope is not failure, it is an honest response to the accuracy bar. Second, evaluation is a recurring labor cost. Cutting 91 questions to 16 was not corner-cutting but resource reality; it also means post-launch regression testing needs a standing owner, not a one-time acceptance gate. Third, do not underestimate a “simple” grounded Q&A project. A Deloitte survey found fewer than 6% of local government practitioners prioritize AI for service delivery — and AVA explains why: it is not reluctance, it is the accuracy threshold plus review cost that keeps most projects out.
Sources
- Alaska’s court system built an AI chatbot. It didn’t go smoothly. — NBC News
- Chatbots for pro se litigants — Thomson Reuters Institute
- AVA FAQ — Alaska Court System
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.