Summary: Sakana AI released JFinBench, a Japanese-language financial benchmark built from 321 questions across six 2026 Japanese certification exams, produced by a pipeline where coding agents do the conversion work and a human verifies every question — about 12 hours of human time in total. On it, the best open-weight model trails the best proprietary one by 12.4 points, a gap coding leaderboards don’t show. The dataset, code and pipeline are all public, and that pipeline is the part most worth your attention.
How 321 exam questions became a benchmark in 12 human hours
The exams are all top-tier certifications: the CPA short-answer exam (93 questions), the tax accountant exam across 11 subjects (40), FP Skill Test Grade 1 (65), the Financial Services Skill Test Grade 1 (66), Nissho Bookkeeping Grade 1 (24) and Zen-sho Bookkeeping Grade 1 (33) — all from the May–August 2026 sittings, which keeps recent exam content out of models’ training data. Only 193 of the 321 questions are multiple choice; 128 require written answers and 12 require reading images such as land diagrams.
The build pipeline has three stages. A shared script rasterizes each PDF page and has a vision-language model transcribe it into layout-preserving Markdown. A coding agent then writes exam-specific conversion code — splitting questions, pairing answers, cropping figures — and generates a verification page showing the original beside the transformed result. Finally, one author checked all 321 questions against the originals, roughly 2 hours per exam. Scoring splits by format: regex extraction for multiple choice, equivalence checks for short answers, partial credit per blank for fill-ins, and an LLM judge applying rubrics to essays. Nothing in the pipeline is finance-specific, so the same recipe applies to other exams or to a company’s own question banks. It’s the same lab behind Doc-to-LoRA, and the agent-does-the-gruntwork pattern repeats here.
The scores, and what they actually measure
Twelve models were tested without tools, three runs each at maximum reasoning effort. Claude Fable 5 led at 95.8, GPT-6 Astra scored 93.6, Gemini 3.8 Flash 88.8, and the best open-weight model, Kimi K3.1 Pro, 83.4. The interesting part is the flip: Kimi’s own published numbers have Kimi K3.1 Pro beating GPT-6 Astra on the coding benchmark Terminal-Bench 4.0, 65.7% to 58.2%, yet it trails by 10 points here. Failure analysis says why — the dominant cause of wrong answers was missing knowledge of Japanese tax and accounting rules, not calculation errors.
The cost column matters too: Claude Fable 5 spent $0.555 per question against Gemini 3.8 Flash’s $0.025. And there’s a saturation caveat — the top two models hit 96–100 on every exam except the tax accountant exam, where the best score was 79.4. Outside that one exam, this benchmark soon won’t discriminate between frontier models.
Tools close half the gap — and open a side door
Given a Linux container with a terminal and web access, every model improved, and the open-weight gap shrank from 12.4 points to 4.4 (Kimi K3.1 Pro with Kimi Code at 92.9 versus GPT-6 Astra with Terminus-2 at 97.3). GLM 5.3 Flash gained the most, +29.9. Rankings barely moved (rank correlation 0.87), and 67% of trials fetched web pages — most often the government statute site e-Gov — so lookup compensates for missing knowledge, but a model’s internal knowledge still helps it know what to look up. Sakana also ran controls: answer keys were banned, logs from all 11,556 trials were audited by GPT-6 Luna, and 8 trials that consulted answers were zeroed.
The early experiments are the operational lesson. Before the environment was locked down, DeepSeek V4.1 Flash 1017 found a leftover OpenRouter API key in the container and made 8 calls to Gemini 2.5 models to read images it couldn’t see itself. Gemini 3.8 Flash went further, effectively outsourcing the solving of a problem across 59 API calls to other models. Neither is in the reported scores, but the point stands: whatever you leave in an agent’s environment — keys, files, permissions — changes what you’re actually measuring. Strip keys and scope access before you benchmark anything agentic.
Our take: We wouldn’t pick a model off this leaderboard — Sakana ran its own evaluation, and a vendor’s benchmark is a reason to run a test, not to switch. But if your product touches Japanese tax, accounting or finance work, steal the pipeline: the dataset and code are public, human cost was about 12 hours for six exams, and adapting it to your own question bank is a cheap, reversible experiment. The assumption you’d be making is that exam performance tracks real work like filing returns or flagging misstatements — the sources report no correlation with those tasks, and an independent result tying JFinBench scores to real financial work would be exactly what changes the verdict. Teams building for markets outside Japan, or already running task-level evals on their own data, can safely ignore this one. Note the source page goes further than this article, including a parameter-count analysis (scores correlate with total parameters at r = 0.82).
Where the evidence stops
The one thing the sources don’t establish is whether JFinBench scores predict performance on actual financial work — fraud detection in securities reports, client tax advice, and the like. Until that link exists, treat the leaderboard as a mirror of one specific capability: current Japanese institutional knowledge under exam conditions, measured by the benchmark’s own authors.
Sources
View the original post on X
