On June 17, 2026, startup Pramaana Labs announced a $27 million seed round led by Khosla Ventures, with Accel, BoldCap, Nexus Venture Partners, Premji Invest, and Unbound participating. What the company builds is unusual: it brings formal verification — the mathematical-proof discipline — into LLM products, adding a deterministic checker on top of model output for domains where a mistake costs someone their health, money, or freedom.
The timing matters. Just one day earlier, on June 16, another reliability-focused startup, Probably, announced a $9 million seed. Two rounds in two days is a signal: verifiable LLM output has graduated from research talking point to investable category.
What Formal Verification Is, and Why Use It on LLMs
Formal verification uses mathematics to prove that a system satisfies its specification. It has long lived in chip design and theorem proving, and Pramaana’s core tool is the open-source proof language LEAN — the system mathematicians use to verify proofs. Pramaana borrows the same machinery to check an LLM’s reasoning.
Co-founder and CEO Ranjan Rajagopalan frames it plainly: “The world’s hardest problems are not unsolvable. They are unformalized.” The underlying thesis is that every domain where being wrong can cost someone their health, money, or freedom has rules — and rules can be written in a verifiable form.
An LLM With a Deterministic Verification Layer
The architecture is two-tier. A conventional LLM handles natural-language flexibility and understanding; a deterministic verification layer checks every output the model produces against the rules. The LLM stops being the answer machine and becomes the proposer — the final answer carries the verifier’s stamp.
For each use case, Pramaana builds its own LEAN-style verification system under the supervision of domain experts. That is laborious, and it is also the moat. The precedent is France’s CATALA project, which translated much of the country’s tax and benefit system into executable code — proof that formalizing a rule-bound world is engineering-feasible.
Law, Tax, and Drug Discovery: High-Stakes Domains First
Pramaana’s first three verticals are law, drug discovery, and tax preparation, and the team maps onto that bet. Danny Werfel, former IRS commissioner, works on tax law; professors from IIT Delhi, IIT Madras, and UC Berkeley oversee the cybersecurity and drug-discovery systems.
Tax is the cleanest illustration. Rajagopalan argues the tax code is “like math in the sense that you have a lot of rules that you need to abide by,” and that “once you have a codified version of it, the reasoning on top of it starts becoming deterministic.” That is precisely where LLMs are weakest — rule-dense reasoning with zero error tolerance — and exactly what the verification layer patches.
Probably, Same Week: LLM Reliability Becomes a Trend
Probably, announced the day before, is a variation on the same theme. Founded by Peter Elias and backed by Andreessen Horowitz, its first product is a data-science tool: an LLM answers questions over complex datasets, every result ships with a citation and an audit trail, and underneath sits a harness Elias calls a “data science mech suit” — a deterministic validator that rejects any output inconsistent with the dataset, with the LLM itself trained against that validator.
Two details stand out. Probably currently runs a model “four classes weaker than the frontier models,” on local hardware like a desktop instead of a data center, slashing token costs — evidence for Elias’s principle that “the better your harness engineering is, the weaker the model can be.” He is blunt about why big labs have not tried this: “they make money the more times you have to correct the model.”
What It Means for Builders
Three conclusions. First, reliability is moving up from prompt tricks to architecture: the two-tier design — LLM proposes, deterministic verifier disposes — is becoming the standard pattern for high-stakes applications. Second, a verification layer lets you reach near-deterministic accuracy with smaller, cheaper models, which rewrites the cost structure. Third, formalized rule bases (CATALA for French tax law being the model) are the new moat — whoever encodes a domain’s rules first holds the ticket to AI in that domain.
Sources
- Pramaana Labs raises $27 million seed round from Khosla Ventures to bring formal verification to AI — TechCrunch
- Probably raises $9M to build a more reliable kind of AI — TechCrunch
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.