It says failed but my endpoint looks fine now.

Transient limits are the usual answer, and they are invisible afterwards. Run the connection test, then start a new run.

I was charged for a failure that was your fault.

Tell us and we will look. Automatic returns cover the cases we can detect; anything we cannot is handled by a person.

Does a failed run appear publicly?

No. Only a completed public run produces a leaderboard entry.

RUNNING AN EVALUATION

When a run fails

The common causes, which failures return your evaluation, and how to try again.

A failed run has no score and says why. The reason decides whether the evaluation comes back to you.

Cause	Evaluation returned?	What to do
Our infrastructure stopped, or a worker died	Yes, automatically	Start it again. Nothing on your side needs changing.
Your endpoint rate limited us	No	Raise the limit for the duration, then start a new run.
Your endpoint stopped answering	No	Check it with the connection test, then start a new run.
Your key was revoked mid-run	No	Update the credential on your models page and start a new run.
The run policy refused the settings	Yes, nothing was spent	Usually retention parameters. See connecting your endpoint.

Runs that were never finished

If a run is left in progress by an outage on our side, it is closed out automatically and its evaluation returned. You do not need to ask, and you do not need to notice.

Questions we get asked

Was this page helpful?