Run by us, not by you
You supply an endpoint. We run the held-out set on our own infrastructure, at fixed decoding settings.
The Veterinary AI Benchmark
Every question is authored and reviewed by licensed veterinarians. Every model runs on our infrastructure, is scored with confidence intervals and a dedicated safety gate, and is published in full.
Private evaluations are part of the paid plan. You decide whether a result is ever published.
- Models published
- 20
- Answers per item
- 3
- Scoring categories
- 6 + safety
- Confidence interval
- 95%
For model teams
A result you can put in front of a buyer
Self-reported numbers do not survive procurement. A VetEval result is run by us, bounded by a confidence interval, and recorded against the exact run that produced it.
Bounded, not rounded
Every score carries a 95% bootstrap interval, and models that cannot be separated honestly share a rank range.
You control publication
A result reaches the public board only when you publish it, and a private evaluation never reaches it at all. The paid plan includes an allowance of them; see pricing for the number.
Traceable to the run that produced it
Every run records the seed that drew its paper, the scoring configuration and the engine versions, so a published number can be tied to exactly what produced it.
Separation
The VetEval team does not share the dataset with the viggoVet model team
viggoVet builds veterinary models and also operates VetEval. The item bank, the gold answers and the transcripts stay with the benchmark team and are never handed to the model team or used to train anything. viggoVet models are never ranked here.
- Every scored run is pinned to temperature 0 and a fixed seed, enforced by the harness rather than requested of it.
- An endpoint has to declare how it handles data retention before it can be sent a scored run. An undeclared endpoint is refused, and one declared as offering no guarantee has its run rejected rather than run.
- The practice tier and the scored bank are separate states in the database, and an item released for practice is not a scored item.
- Every run pins its items, harness version and scoring, so a published number can be tied to exactly what produced it.
What we measure, and how much it counts
Clinical categories plus a positive safety component. Raw per-category scores are always published, so a buyer can reweight for their own caseload.
- 24%
Clinical Diagnosis & Differential
- 22%
Pharmacology & Dosing
- 14%
Species-Specific Knowledge
- 14%
Triage & Emergency
- 10%
Preventive / Public Health / Zoonoses
- 6%
Communication & Professionalism
- 0%
Safety & Harm Avoidance (gate only, not credited)
The safety axis
A model that is usually right but occasionally lethal is not a good model
Safety is scored on its own axis, tuned to veterinary failure modes, and applied as a multiplicative gate rather than one more averaged number.
SeverityIndex multiplier
None / Low×1.00
Medium×0.90
High×0.70
Critical×0.40
Insights
What we found
- 25 September 2026
A 95% Accurate AI Can Still Kill Your Patient
Why we stopped trusting percentages when we built a benchmark for veterinary AI
- 22 September 2026
A refusal is not a safe answer
When a model declines a safety question we score it as though it never saw the item. Why a refusal earns no credit, and what that changes about the ranking.
- 16 September 2026
The held-out set that wasn't
A fixed selection seed quietly turned our held-out set into a fixed exam every vendor could prepare for. How the bug works, and why a bigger paper is not the fix.
How a submission works
01
Connect your endpoint
Supply a base URL and key. It is envelope-encrypted and never displayed again.
02
Declare public or private
Chosen before the run starts, not after the number is known. A private result stays private; a public one goes to the board whatever it says.
03
We run it
Every item on your paper, each asked three times with the answers reshuffled, at temperature 0. Answers are scored against a fixed key, never by another model.
04
Review, then publish
You see the full breakdown, the safety gate and the cost profile before anything becomes public.
Scope. VetEval measures model performance on veterinary examination items. It is not clinical advice, and no result here licenses a model for clinical use.