Skip to main content
Seven events ship in v1. Each fires exactly once per state transition; none fan out per-rater. Subscribe to specific types in enabled_events on the endpoint. Use ["*"] to receive everything — recommended only for warehouse-sync integrations.

​
Envelope

All events share the same outer shape:
request.id traces back to the API request that triggered the event, useful for correlating customer-side and server-side logs.

​
Per-event payloads

Fires after POST /v1/evaluations returns 201. Payload mirrors the HTTP response.
Fires on every Evaluation.status transition. Reading the current status alone is insufficient — listen for this event if you need to react on Draft → Running distinct from Running → Ended.
Fires once when all queued AutouserRuns reach a terminal state (completed or failed) AND any pending human rating windows close. The single most useful event for “wake up when the eval is done” integrations.
Fires per AutouserRun when it reaches status: completed. If you queued 6 runs (agentCount: 3 × 2 comparisons), expect six events.
status and stop_reason answer different questions. status says whether the PIPELINE finished — did the worker walk every comparison and write every rating. stop_reason says whether the browser SESSION finished. A run is routinely status: "completed" with stop_reason: "api-error", meaning the rows were written around a transcript an infrastructure failure cut off at an arbitrary turn.Values: completed (the agent decided it had seen enough), max-turns (it spent its whole turn budget — a parameter of the study, applied to every run alike), api-error (the session was cut short by an infrastructure failure), blocked (a bot/Cloudflare interstitial ended at least one navigation), or null for a run written before the column existed.Autousers’ own agreement statistics treat a completed + api-error run as not a peer of one that ran through, because two sessions truncated at two arbitrary points did not do comparable work. Its ratings are still real judgements of what it saw and still count toward each design’s averages. If you reconstruct rater panels downstream, apply the same rule or your numbers will not match ours.
Fires when an AutouserRun errors out. A failed run does NOT consume autouser-rating quota, but you may want to alert on it (the design might be unreachable, or the worker might have crashed).
Fires per Rating row insert — humans AND autousers. The single firehose event for warehouse-sync use cases. High volume on busy evals; budget your receiver accordingly.
Filter on excluded before you aggregate. Some rating rows are recorded but are not judgements about the design — a human rater’s pre-qualification skip, or an autouser whose navigation was killed by a bot/Cloudflare interstitial. The latter is scored at the floor of every dimension by design, so an unfiltered average silently reports a scraper failure as a bad design.excluded is the predicate. skip_reason is the raw token and is groupable: anything starting automation_failure: is a pipeline failure that needs an infrastructure fix (usually a residential proxy), not a research finding. exclusion_reason is the same thing as a sentence. dimension_ratings still carries the floored scores — deliberately, so that a consumer who forgets to filter sees a conspicuous all-minimum row rather than a plausible-looking midpoint.excluded is present on every payload, including false. If a row you stored has no excluded key at all it predates this field: re-fetch it from GET /v1/evaluations/{id}/ratings before trusting it.
Fires when an autouser’s rubric is locked via POST /v1/autousers/{id}/calibration/freeze. Downstream pipelines should listen for this before promoting an autouser to production.

​
Adding new events

New event types are additive — new entries on Autousers-Event-Type, new payload shapes — and ship without a version bump. Build your receiver to ignore unknown event types: