Summary: Winfunc Research’s N-Day-Bench tests whether frontier models can find real vulnerabilities disclosed after their knowledge cutoffs, using monthly rotating post-cutoff cases. Its opening run placed GPT-5.4 at 83.93, GLM-5.1 at 80.13, Claude Opus 4.6 at 79.95, and Gemini 3.1 Pro at 68.50. The authors and article still caution that the judge model overlaps the contest and that guided sink tracing does not equal open-ended bug discovery.
Security firm Winfunc’s research team closed out another run of its new benchmark, N-Day-Bench, on the evening of April 13, 2026. It asks exactly one question: handed a real codebase, can a frontier language model surface vulnerabilities that were already known but disclosed only after the model’s knowledge cutoff (N-Days)? The opening leaderboard put OpenAI’s GPT-5.4 on top at 83.93, Z.ai’s GLM-5.1 at 80.13, and Anthropic’s Claude Opus 4.6 at 79.95 — three models packed inside four points — while Google’s Gemini 3.1 Pro finished more than 15 points behind.
The design matters more here than the order of names. Contamination remains the oldest objection to security evals: a model memorizes vulnerabilities from its training data and then “replays” them in the exam. N-Day-Bench’s response is blunt — questions rotate monthly, and only advisories published after the tested models’ cutoffs are admitted, so recall buys nothing. Every trace is public, and each score traces back to specific shell commands and code.
The Case for Bugs Born After the Cutoff
A fixed question set decays into a memorization test; an unbounded rolling one makes runs impossible to compare. N-Day-Bench steers a middle course by anchoring monthly editions to cutoff timestamps: each pool holds only advisories published after the previous cutoff and on or before the current one, and the model roster is upgraded monthly to each lab’s latest version and checkpoint. Per the site, every model runs the same harness on the same context, with “no leeway for reward hacking.”
Filtering the Cases: 47 of 1,000 Make the Cut
Cases come from GitHub Security Advisories, scanned newest-first through the GraphQL API. The bar is strict: the advisory must explicitly name a repository (not inferred from package metadata), the repo must exceed 10,000 stars, and the advisory must point to exactly one unambiguous fix commit. Selection moves in 500-advisory windows until 50 slots fill; this month’s run scanned 1,000 advisories, accepted 47, and skipped 953. The harness checks out the fix commit’s sole parent — the codebase in its pre-patch, vulnerable state — while the fix diff goes only to the question writer and is withheld from the model under test. Ambiguous cases (multi-repo advisories, merge commits, unresolvable refs) are discarded instead of approximated, and a round-robin pick per repository keeps one popular project from dominating the pool. The stated philosophy: a smaller, clean dataset beats a larger one assembled from guesswork.
How Curator, Finder, and Judge Split the Work
Three agents with deliberately unequal roles. The Curator, pinned to GPT-5.4 at medium reasoning, studies the advisory context and patch excerpts and returns a structured case with an answer key: expected vulnerability class, components, sink paths, and required evidence. The Finder — the model under test — receives the case and a read-only bash shell, gets at most 24 tool-use steps, and always traces backward from a known sink rather than hunting in the open. The Judge, also GPT-5.4, scores against nothing but a digest-based blind label. The sandbox enforces an overlay filesystem mounted read-only at /workspace, git shimmed into read-only stubs, a 12-second timeout on each command, and caps on call depth, total commands, and awk/sed loop iterations. Grading follows a fixed five-dimension rubric — target alignment 30%, source-to-sink reasoning 30%, impact and exploitability 20%, evidence quality 10%, and overclaim control 10%.
The Opening Leaderboard: Three Models in Four Points
GPT-5.4 leads at 83.93 (44 submissions, 1.07 average findings); GLM-5.1 follows at 80.13 with the best findings-per-submission figure, 1.23; Claude Opus 4.6 posts 79.95 (43 submissions, 1.16); Moonshot’s Kimi K2.5 records 77.18 across 37 submissions; Gemini 3.1 Pro sits at 68.50 with the weakest findings rate, 0.91. The run began at 5:03 PM on April 13 and wrapped at 8:53 PM the same day — about 3 hours 50 minutes, zero failures. A cluster this tight says frontier models have largely mastered the constrained task of tracing backward from a given sink; the gaps show up in report quality, not in whether the bug is found. Gemini’s 15-point gap is the number its team should be probing.
Three Reservations Before Trusting the Numbers
Hold three caveats in mind. The Curator and Judge both run on GPT-5.4 — which also enters the contest and finishes first; blind labels curb bias, but the site concedes that stylistic fingerprinting and length-correlated bias stay possible. The Judge emits the whole score object in a single pass with no server-side weighted recomputation — flexibility bought at the cost of mechanical consistency. And tracing backward from a known sink is far easier than open-ended vulnerability discovery, so scores should not be extrapolated to real bug-bounty work. Even so, monthly question refreshes, complete trace transparency, and a read-only harness make this one of the few credible continuous trackers of LLM security capability. Note the incentive, too: Winfunc sells autonomous code-auditing AI agents, so the leaderboard doubles as a public demo of its own product’s machinery.