Every model we evaluated sat the same fixed slice of our item bank, a little over a tenth of it. The rest had never been served to anyone, and under the design we were running, they never would be.
This is the story of a bug that produces no error message, corrupts no data, and quietly converts a held-out benchmark into a fixed exam that every future model can, in principle, prepare for. We suspect it exists in other evaluation pipelines right now, which is why we are writing it up in detail instead of quietly fixing it: what caused it, why the obvious fix doesn't work, how to audit your own pipeline for it this afternoon, and what the correct fix costs.
The reasonable decision that caused it
Early on we wanted comparability above all: every model should sit the same exam. The obvious implementation is a deterministic draw, a fixed selection seed that samples the same items from the bank for every evaluation. Same items, same distribution, perfectly fair head-to-head comparisons, and the score differences between models are exactly model differences. Every evaluation engineer has written this line of code, usually with a comment that says "for reproducibility."
The failure mode is what the fixed seed does to the word "held-out." Our private items are served to vendor-controlled endpoints, which see every prompt in plaintext, as any black-box evaluation of a commercial API must. Serve the same items to every endpoint forever, and those items are held out from the public but not from the models being measured; every submission is another disclosure of the same exam. Meanwhile the items nobody ever sees provide exactly zero measurement value while costing full authoring, sourcing, and veterinary review effort. The bank was 88% decoration guarding a 12% exam.
A flow diagram covering: Fixed seed (before); (Item bank); Vendor endpoints; Zero measurement value; Memorizable exam; Per-run sampling (now); Exposure spread and measured; Paraphrase perturbation on rotating items.
Why the obvious fix doesn't work
The first idea everyone has is a bigger fixed paper: serve two or three times as many items and the exposure problem shrinks, surely. It doesn't; it just gets more expensive. Any fixed selection, whatever its size, is the same exam disclosed to every endpoint on every evaluation, and disclosure accumulates. The size of the paper changes how long memorization takes to matter, not whether it does. Worse, a bigger fixed paper burns more of the bank into permanent exposure, shrinking the reserve that makes future measurement trustworthy.
The second idea is rotation on a schedule: swap the paper quarterly. Better, but now the benchmark has seasons, scores are only comparable within a season, and each season is still a fixed exam for its duration, with the added excitement of a discontinuity at every rotation that readers will misread as models improving or regressing.
The actual fix has to remove the concept of "the" exam entirely.
The fix, and what it costs
Every evaluation now draws its own paper: a fresh, stratified sample from the held-out pool, with nothing carried between evaluations. An item asked in one evaluation is exactly as likely as any other to be asked in the next, so exposure spreads across the bank and is measured rather than concentrated and ignored. Rotating items are additionally served with semantically preserving paraphrase, and any paraphrased dosing or toxicity stem is re-verified by a licensed veterinarian before it is served, because a paraphrase that changes the correct dose is not a perturbation, it is a new and wrong question. Draw seeds are recorded permanently, so every published score traces to the exact paper that produced it.
The honest part: this costs precision, and we pay for it in public. Two models scored on different draws from the same bank differ partly because the models differ and partly because the papers do. That is exactly why our published intervals bootstrap over the draw, why ranks publish as bands rather than an order, and why two models whose paired difference is not significant are declared tied instead of decorated with fake decimals. The uncertainty was always there; the fixed paper hid it by measuring something easier to be precise about. A benchmark that reports fixed-exam precision while serving a memorizable exam has chosen the wrong thing to be precise about.
How to audit your own pipeline this afternoon
If your benchmark serves a private set to endpoints you don't control, three queries against your serving logs tell you whether you have our bug.
First, bank coverage: across all evaluations to date, what fraction of the private bank has ever been served? If the answer is a small, stable fraction, your bank is decoration. Second, per-item exposure by endpoint: for the items that have been served, how many distinct vendor endpoints have seen each one? A histogram with a spike at "all of them" is the fixed-exam signature. Third, the overlap matrix: for each pair of consecutive evaluations, what share of items is identical? Near 100% overlap means every submission was a disclosure of the next submission's exam.
None of these queries requires new infrastructure; they are joins on logs you already keep. The uncomfortable part is only what they might say.
General lessons for any private-set benchmark
Three, in descending order of comfort. Comparability and contamination resistance trade off directly, and the trade is not optional; you choose which one to sacrifice, and only one of those choices can be repaired with statistics afterward. Reproducibility of the pipeline is not the same as validity of the measurement; our fixed seed was perfectly reproducible, which is precisely what made it perfectly disclosable. And a held-out set is a claim about your subjects, not your readers: "the public has never seen these items" is a different sentence from "the models being measured have never seen these items," and only the second one protects the measurement.
We kept every item in the bank. Now all of them work for a living.
Common questions
Doesn't per-run sampling make model comparisons unfair, since models sit different papers? It makes comparisons honest about their uncertainty. Papers are stratified to the same category and species proportions, intervals are computed over the draw, and ranks publish as bands with statistical ties. A fixed paper's tighter-looking comparisons were partly measuring shared exposure, which is a worse unfairness wearing better clothes.
Can two models still be compared head-to-head? Yes: differences are tested as paired statistical questions with multiplicity control, and models that cannot be separated honestly share a rank band. What disappears is decimal theater, not comparison.
What stops memorization of the whole bank over many evaluations? Spread exposure is measured, rotating items are paraphrase-perturbed, per-model contamination signals are computed from public-versus-private performance gaps and membership-inference screening, and the bank grows. Contamination control is a posture, not a single trick; the full protocol is in the methodology.
Why publish this at all? Because the failure mode is quiet, common, and checkable in an afternoon, and because a benchmark that hides its own design errors is asking for exactly the trust it hasn't earned. The instances are private. The method is public. The asymmetry is the point.
Data and reproducibility
Per-run sampling, stratification proportions, paraphrase rules, and the exposure accounting are specified in the methodology and the paper. Draw seeds are recorded per evaluation, so any published score can be traced to the exact paper that produced it.