Summary: NIST’s CAISI released an official DeepSeek V4 Pro evaluation on May 1, 2026 using preregistered benchmarks, held-out items, fixed budgets, and IRT weighting. It estimated Elo 800 (±28) and placed the model about eight months behind the US frontier, near GPT-5. Results varied sharply by domain: math was near parity, while abstract reasoning, cybersecurity, software engineering, and natural sciences trailed.
On May 1, 2026, the Center for AI Standards and Innovation (CAISI), part of the National Institute of Standards and Technology (NIST), released its official evaluation of DeepSeek V4 Pro, an open-weight model out of China. The short version: of all the Chinese models CAISI has assessed so far, this one is the most capable — and it still trails the US frontier by roughly eight months, landing near where OpenAI’s GPT-5 left off.
The ranking alone is not what makes this report worth a developer’s attention; the machinery behind the numbers matters just as much. CAISI commits to its benchmark suite in advance and weights results with Item Response Theory (IRT), closing off much of the room that data contamination and cherry-picking normally exploit. Self-reported scores have grown nearly impossible to compare; against that backdrop, an auditable, government-grade evaluation process is coming together — and pre-release review is exactly where it is headed.
Inside the Methodology
Three design choices define the exercise. The benchmark suite is locked in before any model runs, so nothing gets chosen based on results. The tests lean on held-out items the public never sees — PortBench, built by CAISI itself, plus the semi-private split of ARC-AGI-2 — which keeps training-data contamination in check. And every model answers under matched conditions: token budgets and scaffolding are held constant, after which an IRT-inspired method produces an aggregate capability estimate. Coverage spans five domains: cybersecurity, software engineering, natural sciences, abstract reasoning, and mathematics.
The Numbers: Elo 800 Across Five Domains
Begin with the headline figure: DeepSeek V4 Pro earns an IRT-estimated Elo of 800 (±28), while the comparison models post 1260 for GPT-5.5, 999 for Opus 4.6, and 749 for GPT-5.4 mini. Across domains the profile is strikingly uneven. Math sits nearly at parity with the strongest US models — 97 percent on OTIS-AIME-2025, and 96 percent on each of PUMaC 2024 and SMT 2025. Abstract reasoning slumps to 46 percent on the ARC-AGI-2 semi-private set, and cybersecurity drops further still, to 32 percent on CTF-Archive-Diamond (a figure imputed via IRT). Software engineering and the natural sciences trail the frontier as well. Awkwardly for agent builders, the three weakest domains are three that agentic workloads lean on hardest.
Where Vendor Claims and Independent Testing Diverge
The second storyline is the distance between what the vendor says and what independent testing measures. DeepSeek’s own technical report placed V4 Pro alongside Opus 4.6 and GPT-5.4, two models that had shipped roughly two months earlier. CAISI’s measurements instead return the model to GPT-5 territory, eight months back. Matthias Bastian of The Decoder makes the key distinction: this is not automatically evidence of fabrication, but the systematic distance between a marketing document free to pick its benchmarks and comparators, and an evaluation registered before the fact. Buying decisions should rest on the latter.
The Price Advantage and the Politics Around It
Price is where DeepSeek genuinely pulls ahead. Set against GPT-5.4 mini — the lone US reference model that cleared CAISI’s filtering criteria — DeepSeek V4 came in cheaper on five of the seven benchmarks, with the margin spanning 53 percent cheaper to 41 percent more expensive. Agent workloads run through millions of tokens while premium US models keep becoming costlier, and a spread like that rewrites the shortlist: Cursor has already fine-tuned a lower-cost coding model on the Chinese open-weight Kimi K2.5. Sam Altman conceded the tension himself on X, writing that “just being smarter is still the most important thing” — a caution that capability-per-dollar cuts in both directions once the capability gap is real. The political angle warrants its own skepticism: The Decoder notes that Artificial Analysis’s Intelligence Index shows the US-China gap holding roughly steady, and that CAISI, as a government agency, “likely has its own political agenda.” The split over open-source routes that our opening-of-year outlook flagged at the start of the year now arrives with an official-evaluation lens stacked on top.
Practical Takeaways for Developers
Three items worth acting on. Choose by domain, not by aggregate score: pipelines dominated by math can give open-weight models a serious look, while security-sensitive and long-chain reasoning tasks warrant continued caution. Borrow CAISI’s playbook — benchmarks committed in advance, held-out items, fixed token budgets — because that procedure stands up to contamination far better than yet another pass over public leaderboards. And watch the timing: in that same week, CAISI announced a widening of pre-release evaluation for US frontier models. Government benchmarking is turning from a research instrument into regulatory infrastructure, and learning how it scores models amounts to knowing where the compliance bar of tomorrow sits.