Imagine two AI assistants. Each answers 95 out of 100 clinical questions correctly. On paper they're identical, and most AI benchmarks would rank them side by side.
Now look at what each got wrong. The first gave five incomplete answers: a differential missing one cause, a plan that skipped a follow-up test. The second gave four of those, plus one answer telling an owner that acetaminophen would help their cat.
Acetaminophen is lethal to cats. One of these assistants is imperfect. The other can kill a patient. The percentage can't tell them apart.
That gap is why I built VetEval, a benchmark for how language models handle veterinary clinical work. I'm a veterinarian, and none of the existing AI benchmarks measured the thing that matters most in our work.
Veterinary medicine breaks the usual scoring
Most benchmarks assume a wrong answer is a wrong answer. Getting question 40 wrong costs roughly what getting question 41 wrong costs. In our field that assumption fails, and it fails in ways any vet will recognise.
Species changes everything. Something harmless in one animal is fatal in another. Permethrin kills cats. Xylitol, grapes and onions are dangerous for dogs. Lilies cause kidney failure in cats. A model that carries advice from one species to another isn't slightly off; it produces the textbook poisoning case.
Doses are calculated, not looked up. Our patients run from a 2 kg kitten to a 700 kg horse. A plausible-looking mg/kg figure with a unit slipped, on a drug with a narrow safety margin, is dangerous in a way a vague differential usually isn't.
Some cases can't wait. A bloated, retching large-breed dog. A male cat straining with no urine. An animal that just ate something toxic. A model that answers calmly without saying "go to a vet now" has caused the harm itself.
Safety can only take points away
So VetEval still produces a single score, because people need one number to compare tools. What changes is what can happen to that number.
The clinical questions earn the score across six areas of practice, weighted by how often each comes up in real caseload. Then we look at the worst harm the model produced anywhere in the run, and that caps the result.
A model with no harmful answers keeps its score. A single confirmed critical harm, such as a lethal drug for the species, a lethal dose, or a missed emergency, cuts the result to 40% of what it would have been. A model scoring 85 on clinical questions would publish at 34.
This is deliberate. A model never earns a point for not poisoning a cat. Not poisoning a cat is the minimum, not an achievement.
There's a catch we had to close. If safety questions can only cost points, a model could simply refuse to answer them. So the score is also multiplied by the share of safety questions the model actually engaged with. Staying silent doesn't protect a score.
We're careful about the difference between refusing and referring. "This sounds like an emergency, take your dog to a vet now" is a real clinical answer, and it's scored on its merits. Declining to engage at all is something else.
A fresh exam every time
The second problem with AI benchmarks is contamination. When the questions are public, models end up trained on them, and the test starts measuring memory instead of reasoning.
Our questions are written by licensed veterinarians, reviewed, and kept private. And no single fixed exam exists. Every evaluation draws its own random sample from the private bank.
Random selection sounds like it would make results impossible to check. It doesn't, because each run's draw is recorded before the first question is sent. Every published score can be traced back to the exact questions that produced it.
We also refuse to test any model whose provider might keep what we send it. A provider that retains our questions could train on them, and a model trained on the exam scores higher without being any better. No zero data retention, no test.
Checking the grader, too
Multiple-choice answers grade themselves. Open clinical answers, the kind a vet would actually write, need judging, and at VetEval that's done by AI graders working against a rubric.
That makes the grader a risk in its own right. So before any grader's marks count, it has to grade a set of cases that veterinarians have already scored, and it has to agree with them closely enough. If it doesn't, the results don't get published. No model grades its own answers.
When two models can't be told apart, we say so
The last thing we changed is how rankings are shown. Every score carries a confidence interval, a range the true score very likely falls within. When two models' ranges overlap, calling one "number 1" and the other "number 2" claims a precision the data doesn't have.
So on the VetEval leaderboard, statistically inseparable models share a tier. It's less exciting than a clean top ten. It's also true.
What this means if you're choosing an AI tool
You don't need a benchmark to ask better questions. When a vendor tells you their assistant is "95% accurate", ask:
- What did it get wrong, and how badly? An average hides the one answer that matters.
- Were the questions veterinary, and species-specific? Human medical benchmarks don't test feline toxicology.
- Could the model have seen the test before? Public questions measure memory.
- Does it tell owners when to come in? Escalation is a clinical skill, not a courtesy.
- Does the provider keep what you type? That matters for your clients' data as much as for any benchmark.
Read more
Current results for every evaluated model, with their intervals and safety records, are on the VetEval leaderboard. Standings change as new models are tested, which is exactly why they live there and not in an article.
The full methodology, including the scoring arithmetic, the statistics behind the intervals, and the limitations we can't engineer away, is in the whitepaper: