The most contested design decision in VetEval is easy to state: when a model declines to answer a safety question, we score it the same as never seeing the item. Declining half the safety block halves that share of the score. There is no allowance, no threshold, and no number of safety questions a model may skip for free.
Every few weeks someone tells us this is backwards. A model that refuses when uncertain is behaving exactly the way a responsible deployment should. Why punish the behavior we want?
Because a benchmark and a product are different instruments, and confusing them is how safety theater gets built. This post lays out the whole argument: what a refusal proves, why correct safety answers earn nothing either, how the rest of the AI industry handles the same split between measuring capability and deploying products, and the three strongest objections with our answers.
What a refusal proves
When a model answers a question about permethrin exposure in a cat and gets it right, we learn it knows the safe answer. When it gets it wrong, we learn it doesn't. When it refuses, we learn nothing, and an evaluation that treats nothing as something can be gamed by the cheapest strategy in machine learning: silence.
If refusals earned partial credit, the optimal submission strategy writes itself. Detect anything hazard-adjacent, decline it, collect the allowance, and let the clinical categories carry the index. The leaderboard would then reward models for recognizing that a question is dangerous while hiding whether they know what to do about it. In a clinic, recognizing danger without knowing the answer is precisely the situation where a tool's knowledge matters most. An evaluation asking whether the model knows the safe answer learns nothing from silence, and any tolerance would publish, as a number, exactly how many safety questions a submission may skip for free.
A flow diagram covering: Safety item served; Evidence: knows the safe answer full credit; Evidence: does not know no credit + severity review; Evidence: none no credit.
The illustration
The chart below is illustrative, not measured; it exists to make the incentive visible. Model A answers everything and gets 80% of safety items right. Model B refuses 40% of safety items and answers the rest perfectly. Under a "refusals are neutral" rule, B outscores A while demonstrating less safety knowledge. Under VetEval's rule, the ordering matches the evidence.
Mechanically, this is the coverage term in the index: the answered share of the safety block multiplies into a model's composite, so silence and absence are priced identically. The exact math, with a worked example, is in the methodology's index-construction section.
Why the right answer earns nothing
The safety block's other asymmetry draws the same objection from the opposite side: a model that answers every safety question correctly gains exactly zero points, because the block carries no weight in the composite. It is a gate, not a category. This is deliberate, for three reasons.
First, safety knowledge is a baseline, not a virtue. Knowing that acetaminophen kills cats does not make a model a good clinician, while not knowing it makes the model dangerous, and the scoring mirrors that asymmetry instead of averaging it away. Second, credited safety points could compensate: a model could subsidize clinical mediocrity with safety marks, and no reader of a composite number would ever see the trade. Third, and least obvious: hazards are the most enumerable, most trainable content in the entire domain. The species-specific toxicity lists, the classic emergency presentations, the notorious drug interactions are finite, famous, and memorizable. The moment safety earns credit, it becomes the cheapest block on the exam to farm, and the benchmark would be paying vendors to memorize toxicity lists while calling it safety.
So the payoff matrix for the safety block is: perfect performance earns nothing, and a single confirmed critical harm costs up to 60% of the index. Nothing to gain, everything to lose. That is not an unusual incentive structure. It is how a license works. Safety is the license, not the grade.
Measurement and deployment: how the wider industry handles the same split
VetEval did not invent the separation between measuring a model and shipping a product; it is now standard practice at the frontier of AI itself. When leading labs evaluate their models for dangerous capabilities, they test versions with safety mitigations reduced or removed, precisely because a refusal would mask the capability being measured, and you cannot manage a risk you refused to measure. The public then receives a differently configured product, with the safeguards on. Same underlying model, two configurations, because capability measurement and public deployment are different jobs with different success criteria.
That is exactly the posture VetEval takes toward veterinary AI, with one difference: we are not the vendor, so we do not tell anyone what to submit. A vendor may submit any endpoint, guarded or raw. We simply refuse to treat silence as knowledge, which means the economically rational move is the same one the frontier labs make internally: measure the capable configuration, deploy the guarded one. The benchmark measures the model. A guardrail is a property of the system built around it.
The three strongest objections, taken seriously
"You penalize responsible behavior." In deployment, refusing when out of depth is responsible. In measurement, a refusal is the absence of data, and rewarding the absence of data corrupts the measurement. We are not scoring the model's manners; we are scoring what it knows. The responsible deployment behavior belongs to the product layer, which we explicitly do not score, and which our index should never be mistaken for.
"This pushes vendors to strip guardrails in production." It does the opposite of what the objection fears. The incentive is to submit an unguarded or lightly guarded configuration for measurement while shipping whatever guardrails the product needs, the same separation the frontier labs practice on themselves. Nothing in VetEval scores, badges, or rewards production configuration at all, and our published scope statement says plainly that no result here licenses a model for clinical use.
"Abstention under uncertainty is epistemically honest, and you punish honesty." This is the strongest version, and we concede its premise: a model refusing because it is genuinely uncertain is, in one sense, telling the truth. But an evaluation cannot distinguish honest uncertainty from strategic avoidance by reading the refusal text, and the moment honest-looking refusals earn credit, strategic ones will be dressed as honest. What we can do is make conservatism visible instead of profitable: we report clinical-category refusal rates separately from safety-block refusal rates, so a model that is cautious everywhere is publicly distinguishable from one that dodges the hazard block specifically. Only the latter pattern is what the rule exists to price.
What this deliberately does not measure
We accept, in print, the asymmetry this creates. A deployed assistant should refuse when out of its depth; a guardrail is the right part for that job. VetEval's index is therefore not a deployment-readiness score for a guardrailed product. It measures the model underneath: whether it knows the safe answer when asked. A product team can wrap any model in refusal logic; a benchmark that let the wrapper stand in for the knowledge would be measuring the wrapper.
Refusal behavior itself is one of our standing published analyses. If the field's models refuse more over time as alignment training intensifies, readers deserve to see that trend plainly rather than have it silently absorbed into scores, and the correlation between safety-block refusal and general refusal is published so guardrail-triggering is distinguishable from temperament.
The cheapest way to keep a clean safety record is to say nothing. We built the one room where that doesn't work.
Common questions
Does a refusal count as a wrong answer? It counts as an unanswered one: no credit, and no severity review either, since nothing harmful was said. Wrong answers additionally enter harm review, where confirmed high or critical findings trigger the gate.
Can a model pass the safety gate by refusing everything hazardous? No. Refusing the block forfeits its coverage, which multiplies down the index. Silence avoids the gate's harm multiplier and pays the coverage cost instead; there is no configuration in which silence is free.
Do you tell vendors to disable guardrails? Never. Submission configuration is entirely the vendor's choice; we publish how each behavior is scored and let vendors decide what they want measured.
Where can I see refusal behavior for a specific model? On that model's page, alongside its full category and safety breakdowns; the analysis methodology is on the methodology page.
Data and reproducibility
The coverage term, the safety gate, and the exact math are in the methodology and in our paper's index-construction section. Refusal-rate breakdowns publish with every evaluation, per model, on the model pages.