# Approve an access request Source: https://docs.autousers.ai/api-reference/access-requests/approve-an-access-request https://app.autousers.ai/api/v1/openapi.json post /api/v1/evaluations/{id}/access-requests/{requestId}/approve # Decline an access request Source: https://docs.autousers.ai/api-reference/access-requests/decline-an-access-request https://app.autousers.ai/api/v1/openapi.json post /api/v1/evaluations/{id}/access-requests/{requestId}/decline # Get the caller's outstanding access request Source: https://docs.autousers.ai/api-reference/access-requests/get-the-callers-outstanding-access-request https://app.autousers.ai/api/v1/openapi.json get /api/v1/evaluations/{id}/access-requests/mine # Inbox of pending access requests Source: https://docs.autousers.ai/api-reference/access-requests/inbox-of-pending-access-requests https://app.autousers.ai/api/v1/openapi.json get /api/v1/access-requests/inbox Returns access requests across all evaluations the caller administers. Cursor-paginated. # List access requests for an evaluation Source: https://docs.autousers.ai/api-reference/access-requests/list-access-requests-for-an-evaluation https://app.autousers.ai/api/v1/openapi.json get /api/v1/evaluations/{id}/access-requests # Request access to an evaluation Source: https://docs.autousers.ai/api-reference/access-requests/request-access-to-an-evaluation https://app.autousers.ai/api/v1/openapi.json post /api/v1/evaluations/{id}/access-requests # Withdraw / delete an access request Source: https://docs.autousers.ai/api-reference/access-requests/withdraw-delete-an-access-request https://app.autousers.ai/api/v1/openapi.json delete /api/v1/evaluations/{id}/access-requests/{requestId} # List API keys Source: https://docs.autousers.ai/api-reference/api-keys/list-api-keys https://app.autousers.ai/api/v1/openapi.json get /api/v1/api-keys Returns metadata only — the plain-text token is shown exactly once at creation time. **Session auth only**: bearer-auth on this route is rejected. # Mint an API key Source: https://docs.autousers.ai/api-reference/api-keys/mint-an-api-key https://app.autousers.ai/api/v1/openapi.json post /api/v1/api-keys Returns the plain-text bearer token in the `token` field — the only time the value is exposed. **Session auth only**: bearer-auth on this route is rejected. # Rename or rescope an API key Source: https://docs.autousers.ai/api-reference/api-keys/rename-or-rescope-an-api-key https://app.autousers.ai/api/v1/openapi.json patch /api/v1/api-keys/{id} # Revoke an API key Source: https://docs.autousers.ai/api-reference/api-keys/revoke-an-api-key https://app.autousers.ai/api/v1/openapi.json delete /api/v1/api-keys/{id} # Identity fan-in Source: https://docs.autousers.ai/api-reference/auth/identity-fan-in https://app.autousers.ai/api/v1/openapi.json get /api/v1/auth/whoami Returns the principal behind the current request — useful for CLI / TUI start-screens that need to confirm who they're acting as. # Create a rubric Source: https://docs.autousers.ai/api-reference/autousers/create-a-rubric https://app.autousers.ai/api/v1/openapi.json post /api/v1/autousers/{id}/rubrics # Create an autouser persona Source: https://docs.autousers.ai/api-reference/autousers/create-an-autouser-persona https://app.autousers.ai/api/v1/openapi.json post /api/v1/autousers # Delete a rubric Source: https://docs.autousers.ai/api-reference/autousers/delete-a-rubric https://app.autousers.ai/api/v1/openapi.json delete /api/v1/autousers/{id}/rubrics/{rubricId} # Delete an autouser persona Source: https://docs.autousers.ai/api-reference/autousers/delete-an-autouser-persona https://app.autousers.ai/api/v1/openapi.json delete /api/v1/autousers/{id} # Draft an autouser from a natural-language prompt Source: https://docs.autousers.ai/api-reference/autousers/draft-an-autouser-from-a-natural-language-prompt https://app.autousers.ai/api/v1/openapi.json post /api/v1/autousers/draft-from-prompt Returns a `SuggestedRubric` + `AutouserDraft` shape that the caller can refine and POST to `/api/v1/autousers`. # Duplicate an autouser persona Source: https://docs.autousers.ai/api-reference/autousers/duplicate-an-autouser-persona https://app.autousers.ai/api/v1/openapi.json post /api/v1/autousers/{id}/duplicate # Extract an autouser persona from unstructured input Source: https://docs.autousers.ai/api-reference/autousers/extract-an-autouser-persona-from-unstructured-input https://app.autousers.ai/api/v1/openapi.json post /api/v1/autousers/extract # Freeze the calibrated rubric Source: https://docs.autousers.ai/api-reference/autousers/freeze-the-calibrated-rubric https://app.autousers.ai/api/v1/openapi.json post /api/v1/autousers/{id}/calibration/freeze Freezing locks the calibrated rubric for production use. Subsequent rating runs reference the frozen version regardless of further calibration cycles. # Get a rubric Source: https://docs.autousers.ai/api-reference/autousers/get-a-rubric https://app.autousers.ai/api/v1/openapi.json get /api/v1/autousers/{id}/rubrics/{rubricId} # Get an autouser persona Source: https://docs.autousers.ai/api-reference/autousers/get-an-autouser-persona https://app.autousers.ai/api/v1/openapi.json get /api/v1/autousers/{id} # Get calibration status Source: https://docs.autousers.ai/api-reference/autousers/get-calibration-status https://app.autousers.ai/api/v1/openapi.json get /api/v1/autousers/{id}/calibration # List autouser personas Source: https://docs.autousers.ai/api-reference/autousers/list-autouser-personas https://app.autousers.ai/api/v1/openapi.json get /api/v1/autousers Includes both team-scoped and system autousers by default. Cursor-paginated. # List rubrics Source: https://docs.autousers.ai/api-reference/autousers/list-rubrics https://app.autousers.ai/api/v1/openapi.json get /api/v1/autousers/{id}/rubrics # Optimize calibration parameters Source: https://docs.autousers.ai/api-reference/autousers/optimize-calibration-parameters https://app.autousers.ai/api/v1/openapi.json post /api/v1/autousers/{id}/calibration/optimize # Start calibration Source: https://docs.autousers.ai/api-reference/autousers/start-calibration https://app.autousers.ai/api/v1/openapi.json post /api/v1/autousers/{id}/calibration/start # Trigger a calibration cycle Source: https://docs.autousers.ai/api-reference/autousers/trigger-a-calibration-cycle https://app.autousers.ai/api/v1/openapi.json post /api/v1/autousers/{id}/calibration # Update a rubric Source: https://docs.autousers.ai/api-reference/autousers/update-a-rubric https://app.autousers.ai/api/v1/openapi.json patch /api/v1/autousers/{id}/rubrics/{rubricId} # Update an autouser persona Source: https://docs.autousers.ai/api-reference/autousers/update-an-autouser-persona https://app.autousers.ai/api/v1/openapi.json patch /api/v1/autousers/{id} # Create a builder-chat conversation Source: https://docs.autousers.ai/api-reference/conversations/create-a-builder-chat-conversation https://app.autousers.ai/api/v1/openapi.json post /api/v1/conversations # Delete a conversation Source: https://docs.autousers.ai/api-reference/conversations/delete-a-conversation https://app.autousers.ai/api/v1/openapi.json delete /api/v1/conversations/{id} # Extract structured fields from a conversation Source: https://docs.autousers.ai/api-reference/conversations/extract-structured-fields-from-a-conversation https://app.autousers.ai/api/v1/openapi.json post /api/v1/conversations/{id}/extract # Get a conversation Source: https://docs.autousers.ai/api-reference/conversations/get-a-conversation https://app.autousers.ai/api/v1/openapi.json get /api/v1/conversations/{id} # List builder-chat conversations Source: https://docs.autousers.ai/api-reference/conversations/list-builder-chat-conversations https://app.autousers.ai/api/v1/openapi.json get /api/v1/conversations # Materialize a conversation into an evaluation Source: https://docs.autousers.ai/api-reference/conversations/materialize-a-conversation-into-an-evaluation https://app.autousers.ai/api/v1/openapi.json post /api/v1/conversations/{id}/evaluations Promotes a chat-built draft into a real Evaluation row, transferring autousers, dimensions, and stimuli atomically. # Post a message (returns SSE stream) Source: https://docs.autousers.ai/api-reference/conversations/post-a-message-returns-sse-stream https://app.autousers.ai/api/v1/openapi.json post /api/v1/conversations/{id}/messages Posts a user message and streams the assistant response back as SSE. Long-lived; treat as a streaming RPC rather than a request/response. # Update a conversation Source: https://docs.autousers.ai/api-reference/conversations/update-a-conversation https://app.autousers.ai/api/v1/openapi.json patch /api/v1/conversations/{id} # Create a dimension (deprecated — use /templates) Source: https://docs.autousers.ai/api-reference/dimensions-deprecated/create-a-dimension-deprecated-—-use-templates https://app.autousers.ai/api/v1/openapi.json post /api/v1/dimensions # Delete a dimension (deprecated — use /templates/{id}) Source: https://docs.autousers.ai/api-reference/dimensions-deprecated/delete-a-dimension-deprecated-—-use-templates- https://app.autousers.ai/api/v1/openapi.json delete /api/v1/dimensions/{id} # Duplicate a dimension (deprecated) Source: https://docs.autousers.ai/api-reference/dimensions-deprecated/duplicate-a-dimension-deprecated https://app.autousers.ai/api/v1/openapi.json post /api/v1/dimensions/{id}/duplicate # Get a dimension (deprecated — use /templates/{id}) Source: https://docs.autousers.ai/api-reference/dimensions-deprecated/get-a-dimension-deprecated-—-use-templates- https://app.autousers.ai/api/v1/openapi.json get /api/v1/dimensions/{id} # Get a specific dimension version (deprecated) Source: https://docs.autousers.ai/api-reference/dimensions-deprecated/get-a-specific-dimension-version-deprecated https://app.autousers.ai/api/v1/openapi.json get /api/v1/dimensions/{id}/versions/{versionId} # List dimension versions (deprecated) Source: https://docs.autousers.ai/api-reference/dimensions-deprecated/list-dimension-versions-deprecated https://app.autousers.ai/api/v1/openapi.json get /api/v1/dimensions/{id}/versions # List dimensions (deprecated — use /templates) Source: https://docs.autousers.ai/api-reference/dimensions-deprecated/list-dimensions-deprecated-—-use-templates https://app.autousers.ai/api/v1/openapi.json get /api/v1/dimensions # Revert a dimension (deprecated) Source: https://docs.autousers.ai/api-reference/dimensions-deprecated/revert-a-dimension-deprecated https://app.autousers.ai/api/v1/openapi.json post /api/v1/dimensions/{id}/revert # Snapshot a new dimension version (deprecated) Source: https://docs.autousers.ai/api-reference/dimensions-deprecated/snapshot-a-new-dimension-version-deprecated https://app.autousers.ai/api/v1/openapi.json post /api/v1/dimensions/{id}/versions # Update a dimension (deprecated — use /templates/{id}) Source: https://docs.autousers.ai/api-reference/dimensions-deprecated/update-a-dimension-deprecated-—-use-templates- https://app.autousers.ai/api/v1/openapi.json patch /api/v1/dimensions/{id} # Cancel running autouser runs Source: https://docs.autousers.ai/api-reference/evaluations/cancel-running-autouser-runs https://app.autousers.ai/api/v1/openapi.json post /api/v1/evaluations/{id}/stop-autousers # Create a public rater Source: https://docs.autousers.ai/api-reference/evaluations/create-a-public-rater https://app.autousers.ai/api/v1/openapi.json post /api/v1/evaluations/{id}/public-raters # Create an evaluation Source: https://docs.autousers.ai/api-reference/evaluations/create-an-evaluation https://app.autousers.ai/api/v1/openapi.json post /api/v1/evaluations # Delete a comparison Source: https://docs.autousers.ai/api-reference/evaluations/delete-a-comparison https://app.autousers.ai/api/v1/openapi.json delete /api/v1/evaluations/{id}/comparisons/{comparisonId} # Delete an autouser run Source: https://docs.autousers.ai/api-reference/evaluations/delete-an-autouser-run https://app.autousers.ai/api/v1/openapi.json delete /api/v1/evaluations/{id}/autouser-runs/{runId} # Delete an evaluation Source: https://docs.autousers.ai/api-reference/evaluations/delete-an-evaluation https://app.autousers.ai/api/v1/openapi.json delete /api/v1/evaluations/{id} # Dual-lookup by id or shareToken Source: https://docs.autousers.ai/api-reference/evaluations/dual-lookup-by-id-or-sharetoken https://app.autousers.ai/api/v1/openapi.json get /api/v1/evaluations/{id}/resolve Accepts either an evaluation cuid or a public `shareToken` in the path; useful for clients that have an opaque link and want to fan out to the canonical resource. # Enqueue autouser runs Source: https://docs.autousers.ai/api-reference/evaluations/enqueue-autouser-runs https://app.autousers.ai/api/v1/openapi.json post /api/v1/evaluations/{id}/run-autousers Spawns one autouser run per `(autouserId, comparisonId)` pairing. Returns the list of newly-enqueued run ids; runs progress asynchronously and emit `autouser_run.completed` / `autouser_run.failed` webhooks (Phase 4). # Export ratings as CSV / JSON Source: https://docs.autousers.ai/api-reference/evaluations/export-ratings-as-csv-json https://app.autousers.ai/api/v1/openapi.json get /api/v1/evaluations/{id}/export # Get a comparison Source: https://docs.autousers.ai/api-reference/evaluations/get-a-comparison https://app.autousers.ai/api/v1/openapi.json get /api/v1/evaluations/{id}/comparisons/{comparisonId} # Get aggregated results Source: https://docs.autousers.ai/api-reference/evaluations/get-aggregated-results https://app.autousers.ai/api/v1/openapi.json get /api/v1/evaluations/{id}/results Returns rating aggregates, agreement summary, and per-comparison breakdowns for a finalised evaluation. # Get AI agreement narrative Source: https://docs.autousers.ai/api-reference/evaluations/get-ai-agreement-narrative https://app.autousers.ai/api/v1/openapi.json get /api/v1/evaluations/{id}/agreement-insights # Get AI insights summary Source: https://docs.autousers.ai/api-reference/evaluations/get-ai-insights-summary https://app.autousers.ai/api/v1/openapi.json get /api/v1/evaluations/{id}/ai-insights # Get an evaluation by id Source: https://docs.autousers.ai/api-reference/evaluations/get-an-evaluation-by-id https://app.autousers.ai/api/v1/openapi.json get /api/v1/evaluations/{id} # Get autouser run detail Source: https://docs.autousers.ai/api-reference/evaluations/get-autouser-run-detail https://app.autousers.ai/api/v1/openapi.json get /api/v1/evaluations/{id}/autouser-runs/{runId} # Get per-turn telemetry Source: https://docs.autousers.ai/api-reference/evaluations/get-per-turn-telemetry https://app.autousers.ai/api/v1/openapi.json get /api/v1/evaluations/{id}/turns # Get per-turn telemetry for a run Source: https://docs.autousers.ai/api-reference/evaluations/get-per-turn-telemetry-for-a-run https://app.autousers.ai/api/v1/openapi.json get /api/v1/evaluations/{id}/autouser-runs/{runId}/turns # Inter-rater agreement Source: https://docs.autousers.ai/api-reference/evaluations/inter-rater-agreement https://app.autousers.ai/api/v1/openapi.json get /api/v1/evaluations/{id}/agreement Krippendorff α and Cohen κ summaries comparing autousers vs. humans, broken down by dimension. # List evaluations Source: https://docs.autousers.ai/api-reference/evaluations/list-evaluations https://app.autousers.ai/api/v1/openapi.json get /api/v1/evaluations Returns evaluations the caller has access to via team membership or direct share. Cursor-paginated using `starting_after` / `ending_before`. # List public raters Source: https://docs.autousers.ai/api-reference/evaluations/list-public-raters https://app.autousers.ai/api/v1/openapi.json get /api/v1/evaluations/{id}/public-raters # List run artifacts Source: https://docs.autousers.ai/api-reference/evaluations/list-run-artifacts https://app.autousers.ai/api/v1/openapi.json get /api/v1/evaluations/{id}/autouser-runs/{runId}/artifacts Returns signed URLs for screenshots, traces, and other artifacts captured during the run. URLs expire; do not cache. # Regenerate AI agreement narrative Source: https://docs.autousers.ai/api-reference/evaluations/regenerate-ai-agreement-narrative https://app.autousers.ai/api/v1/openapi.json post /api/v1/evaluations/{id}/agreement-insights # Regenerate AI insights Source: https://docs.autousers.ai/api-reference/evaluations/regenerate-ai-insights https://app.autousers.ai/api/v1/openapi.json post /api/v1/evaluations/{id}/ai-insights # Server-Sent Events progress stream Source: https://docs.autousers.ai/api-reference/evaluations/server-sent-events-progress-stream https://app.autousers.ai/api/v1/openapi.json get /api/v1/evaluations/{id}/autouser-stream Long-lived SSE stream of autouser-run progress events. Heartbeat every 15s. Prefer webhooks (Phase 4) for production integrations. # Snapshot of autouser-run status Source: https://docs.autousers.ai/api-reference/evaluations/snapshot-of-autouser-run-status https://app.autousers.ai/api/v1/openapi.json get /api/v1/evaluations/{id}/autouser-status # Transfer ownership Source: https://docs.autousers.ai/api-reference/evaluations/transfer-ownership https://app.autousers.ai/api/v1/openapi.json post /api/v1/evaluations/{id}/transfer Move an evaluation to another team the caller administers. # Update a comparison Source: https://docs.autousers.ai/api-reference/evaluations/update-a-comparison https://app.autousers.ai/api/v1/openapi.json patch /api/v1/evaluations/{id}/comparisons/{comparisonId} # Update an evaluation Source: https://docs.autousers.ai/api-reference/evaluations/update-an-evaluation https://app.autousers.ai/api/v1/openapi.json patch /api/v1/evaluations/{id} # Wizard autosave Source: https://docs.autousers.ai/api-reference/evaluations/wizard-autosave https://app.autousers.ai/api/v1/openapi.json patch /api/v1/evaluations/{id}/draft Patches the evaluation draft state. Used by the in-product wizard for autosave; safe for clients to call repeatedly. # Get a single event Source: https://docs.autousers.ai/api-reference/events/get-a-single-event https://app.autousers.ai/api/v1/openapi.json get /api/v1/events/{id} Full event detail rendered as the same Stripe-style envelope the dispatcher POSTs. Cross-team access returns 404. # List webhook events for the team Source: https://docs.autousers.ai/api-reference/events/list-webhook-events-for-the-team https://app.autousers.ai/api/v1/openapi.json get /api/v1/events Cursor-paginated event log for the caller's team, newest first. Returns the same Stripe-style envelope the dispatcher POSTs to receivers — polling consumers and push consumers share parsing code. Filter by `type` (exact match) or `object_id` (the primary object the event is about, e.g. an evaluation cuid). # Confirm a direct upload completed Source: https://docs.autousers.ai/api-reference/files/confirm-a-direct-upload-completed https://app.autousers.ai/api/v1/openapi.json post /api/v1/files/confirm Marks an `UploadedFile` row as `uploaded` after the client's PUT to GCS succeeded. Verifies object metadata and persists the final size. # Delete an uploaded file Source: https://docs.autousers.ai/api-reference/files/delete-an-uploaded-file https://app.autousers.ai/api/v1/openapi.json delete /api/v1/files/{fileId} # GCS configuration self-test Source: https://docs.autousers.ai/api-reference/files/gcs-configuration-self-test https://app.autousers.ai/api/v1/openapi.json get /api/v1/files/doctor Diagnoses the GCS bucket + service-account configuration. Used by `autousers files doctor` and the support team. # Multipart upload (small files) Source: https://docs.autousers.ai/api-reference/files/multipart-upload-small-files https://app.autousers.ai/api/v1/openapi.json post /api/v1/files Multipart `multipart/form-data` upload for small stimuli (<25 MB images). For larger files prefer the 3-step signed-URL flow (`/files/signed-url` → direct PUT → `/files/confirm`). # Read an uploaded file (signed redirect) Source: https://docs.autousers.ai/api-reference/files/read-an-uploaded-file-signed-redirect https://app.autousers.ai/api/v1/openapi.json get /api/v1/files/{fileId} Performs a permission check then 302-redirects to a short-lived signed GCS URL. Do not cache. # Request a signed PUT URL for direct GCS upload Source: https://docs.autousers.ai/api-reference/files/request-a-signed-put-url-for-direct-gcs-upload https://app.autousers.ai/api/v1/openapi.json post /api/v1/files/signed-url # Accept an invite (cookie session required) Source: https://docs.autousers.ai/api-reference/invites/accept-an-invite-cookie-session-required https://app.autousers.ai/api/v1/openapi.json post /api/v1/invites/{token} # Delete an invite Source: https://docs.autousers.ai/api-reference/invites/delete-an-invite https://app.autousers.ai/api/v1/openapi.json delete /api/v1/evaluations/{id}/invites/{inviteId} # Preview an invite (no auth) Source: https://docs.autousers.ai/api-reference/invites/preview-an-invite-no-auth https://app.autousers.ai/api/v1/openapi.json get /api/v1/invites/{token} The opaque `token` itself authenticates this read. Returns enough metadata to render an accept-invite page (evaluation name, owner, expiry). # Resend invite email Source: https://docs.autousers.ai/api-reference/invites/resend-invite-email https://app.autousers.ai/api/v1/openapi.json post /api/v1/evaluations/{id}/invites/{inviteId}/resend # Update an invite Source: https://docs.autousers.ai/api-reference/invites/update-an-invite https://app.autousers.ai/api/v1/openapi.json patch /api/v1/evaluations/{id}/invites/{inviteId} # URL → thumbnail (Figma-plugin helper) Source: https://docs.autousers.ai/api-reference/link-preview/url-→-thumbnail-figma-plugin-helper https://app.autousers.ai/api/v1/openapi.json get /api/v1/link-preview SSRF-guarded fetch + 24h cache. Used by the Figma plugin to render thumbnails for design-link references. # List notifications for the caller Source: https://docs.autousers.ai/api-reference/notifications/list-notifications-for-the-caller https://app.autousers.ai/api/v1/openapi.json get /api/v1/notifications # Mark a notification as read Source: https://docs.autousers.ai/api-reference/notifications/mark-a-notification-as-read https://app.autousers.ai/api/v1/openapi.json post /api/v1/notifications/{id}/read # Mark all notifications as read Source: https://docs.autousers.ai/api-reference/notifications/mark-all-notifications-as-read https://app.autousers.ai/api/v1/openapi.json post /api/v1/notifications/mark-all-read # List OAuth-connected apps Source: https://docs.autousers.ai/api-reference/oauth/list-oauth-connected-apps https://app.autousers.ai/api/v1/openapi.json get /api/v1/oauth/connected-apps Returns the OAuth 2.1 client authorizations the caller has granted. Session auth only. # Revoke an OAuth client authorization Source: https://docs.autousers.ai/api-reference/oauth/revoke-an-oauth-client-authorization https://app.autousers.ai/api/v1/openapi.json delete /api/v1/oauth/connected-apps # Bulk delete ratings Source: https://docs.autousers.ai/api-reference/ratings/bulk-delete-ratings https://app.autousers.ai/api/v1/openapi.json delete /api/v1/evaluations/{id}/ratings # List ratings Source: https://docs.autousers.ai/api-reference/ratings/list-ratings https://app.autousers.ai/api/v1/openapi.json get /api/v1/evaluations/{id}/ratings # Submit a rating Source: https://docs.autousers.ai/api-reference/ratings/submit-a-rating https://app.autousers.ai/api/v1/openapi.json post /api/v1/evaluations/{id}/ratings Upserts on `(evaluationId, comparisonId, userId)` so re-submission updates the existing record. # Delete the caller's BYOK key Source: https://docs.autousers.ai/api-reference/settings/delete-the-callers-byok-key https://app.autousers.ai/api/v1/openapi.json delete /api/v1/settings/byok # Get the caller's BYOK settings Source: https://docs.autousers.ai/api-reference/settings/get-the-callers-byok-settings https://app.autousers.ai/api/v1/openapi.json get /api/v1/settings/byok # Probe a BYOK key without persisting Source: https://docs.autousers.ai/api-reference/settings/probe-a-byok-key-without-persisting https://app.autousers.ai/api/v1/openapi.json post /api/v1/settings/byok/probe # Save a BYOK key Source: https://docs.autousers.ai/api-reference/settings/save-a-byok-key https://app.autousers.ai/api/v1/openapi.json post /api/v1/settings/byok Rate-limited to 5 requests / minute / user. # Test the saved BYOK key end-to-end Source: https://docs.autousers.ai/api-reference/settings/test-the-saved-byok-key-end-to-end https://app.autousers.ai/api/v1/openapi.json post /api/v1/settings/byok/test # Toggle BYOK on/off Source: https://docs.autousers.ai/api-reference/settings/toggle-byok-onoff https://app.autousers.ai/api/v1/openapi.json patch /api/v1/settings/byok # Create an evaluation share Source: https://docs.autousers.ai/api-reference/shares/create-an-evaluation-share https://app.autousers.ai/api/v1/openapi.json post /api/v1/evaluations/{id}/shares # Delete an evaluation share Source: https://docs.autousers.ai/api-reference/shares/delete-an-evaluation-share https://app.autousers.ai/api/v1/openapi.json delete /api/v1/evaluations/{id}/shares # List evaluation shares Source: https://docs.autousers.ai/api-reference/shares/list-evaluation-shares https://app.autousers.ai/api/v1/openapi.json get /api/v1/evaluations/{id}/shares # Update an evaluation share Source: https://docs.autousers.ai/api-reference/shares/update-an-evaluation-share https://app.autousers.ai/api/v1/openapi.json patch /api/v1/evaluations/{id}/shares # Create a team Source: https://docs.autousers.ai/api-reference/teams/create-a-team https://app.autousers.ai/api/v1/openapi.json post /api/v1/teams # Delete a team Source: https://docs.autousers.ai/api-reference/teams/delete-a-team https://app.autousers.ai/api/v1/openapi.json delete /api/v1/teams/{id} # Get a team Source: https://docs.autousers.ai/api-reference/teams/get-a-team https://app.autousers.ai/api/v1/openapi.json get /api/v1/teams/{id} # Invite a member by email Source: https://docs.autousers.ai/api-reference/teams/invite-a-member-by-email https://app.autousers.ai/api/v1/openapi.json post /api/v1/teams/{id}/members # Leave a team Source: https://docs.autousers.ai/api-reference/teams/leave-a-team https://app.autousers.ai/api/v1/openapi.json post /api/v1/teams/{id}/leave # List team members Source: https://docs.autousers.ai/api-reference/teams/list-team-members https://app.autousers.ai/api/v1/openapi.json get /api/v1/teams/{id}/members # List teams the caller belongs to Source: https://docs.autousers.ai/api-reference/teams/list-teams-the-caller-belongs-to https://app.autousers.ai/api/v1/openapi.json get /api/v1/teams # Remove a member Source: https://docs.autousers.ai/api-reference/teams/remove-a-member https://app.autousers.ai/api/v1/openapi.json delete /api/v1/teams/{id}/members/{memberId} # Transfer the Owner role Source: https://docs.autousers.ai/api-reference/teams/transfer-the-owner-role https://app.autousers.ai/api/v1/openapi.json post /api/v1/teams/{id}/transfer-admin # Update a member's role Source: https://docs.autousers.ai/api-reference/teams/update-a-members-role https://app.autousers.ai/api/v1/openapi.json patch /api/v1/teams/{id}/members/{memberId} # Update a team Source: https://docs.autousers.ai/api-reference/teams/update-a-team https://app.autousers.ai/api/v1/openapi.json patch /api/v1/teams/{id} # Create a question template Source: https://docs.autousers.ai/api-reference/templates/create-a-question-template https://app.autousers.ai/api/v1/openapi.json post /api/v1/templates # Delete a question template Source: https://docs.autousers.ai/api-reference/templates/delete-a-question-template https://app.autousers.ai/api/v1/openapi.json delete /api/v1/templates/{id} # Draft a template from a natural-language prompt Source: https://docs.autousers.ai/api-reference/templates/draft-a-template-from-a-natural-language-prompt https://app.autousers.ai/api/v1/openapi.json post /api/v1/templates/draft-from-prompt # Duplicate a template Source: https://docs.autousers.ai/api-reference/templates/duplicate-a-template https://app.autousers.ai/api/v1/openapi.json post /api/v1/templates/{id}/duplicate # Get a question template Source: https://docs.autousers.ai/api-reference/templates/get-a-question-template https://app.autousers.ai/api/v1/openapi.json get /api/v1/templates/{id} # Get a specific template version Source: https://docs.autousers.ai/api-reference/templates/get-a-specific-template-version https://app.autousers.ai/api/v1/openapi.json get /api/v1/templates/{id}/versions/{versionId} # List question templates Source: https://docs.autousers.ai/api-reference/templates/list-question-templates https://app.autousers.ai/api/v1/openapi.json get /api/v1/templates Templates are the canonical product noun for question sets shown to evaluators. The legacy `/api/v1/dimensions/*` family is kept as an alias and is marked `Deprecation: true` with `Sunset: 2027-04-04`. # List template versions Source: https://docs.autousers.ai/api-reference/templates/list-template-versions https://app.autousers.ai/api/v1/openapi.json get /api/v1/templates/{id}/versions # Revert a template to a prior version Source: https://docs.autousers.ai/api-reference/templates/revert-a-template-to-a-prior-version https://app.autousers.ai/api/v1/openapi.json post /api/v1/templates/{id}/revert # Snapshot the current template state into a new version Source: https://docs.autousers.ai/api-reference/templates/snapshot-the-current-template-state-into-a-new-version https://app.autousers.ai/api/v1/openapi.json post /api/v1/templates/{id}/versions # Update a question template Source: https://docs.autousers.ai/api-reference/templates/update-a-question-template https://app.autousers.ai/api/v1/openapi.json patch /api/v1/templates/{id} # Aggregated usage + plan + remaining quota Source: https://docs.autousers.ai/api-reference/usage/aggregated-usage-+-plan-+-remaining-quota https://app.autousers.ai/api/v1/openapi.json get /api/v1/usage Returns the caller's plan tier, current period window, used/limit/remaining counters per quota dimension. Phase 2 of the public-API plan extends this with monthly autouser-rating + human-rating + evaluation counters. # Create a webhook endpoint Source: https://docs.autousers.ai/api-reference/webhooks/create-a-webhook-endpoint https://app.autousers.ai/api/v1/openapi.json post /api/v1/webhooks Register a new webhook endpoint. The plaintext signing `secret` is returned ONCE in the response body (`whsec_*`, ~38 chars) and is not recoverable later — persist it immediately or call `POST /v1/webhooks/{id}/rotate-secret` to mint a new one. Admin role or higher; `webhooks:write` scope. # Delete a webhook endpoint Source: https://docs.autousers.ai/api-reference/webhooks/delete-a-webhook-endpoint https://app.autousers.ai/api/v1/openapi.json delete /api/v1/webhooks/{id} Hard-delete the endpoint. Pending and historical deliveries cascade-delete with the endpoint; events on the team survive (they're independent of any single endpoint). Admin role or higher. # Get a single webhook delivery Source: https://docs.autousers.ai/api-reference/webhooks/get-a-single-webhook-delivery https://app.autousers.ai/api/v1/openapi.json get /api/v1/webhooks/deliveries/{id} Returns the full delivery row including the captured `response_body` (truncated to 4KB at write time) and the linked `event_id`. Cross-team access returns 404. # Get a webhook endpoint Source: https://docs.autousers.ai/api-reference/webhooks/get-a-webhook-endpoint https://app.autousers.ai/api/v1/openapi.json get /api/v1/webhooks/{id} Returns the endpoint detail. Plaintext secret is NOT included — only `secret_prefix`. Cross-team access returns 404 to avoid leaking endpoint existence. # List deliveries for an endpoint Source: https://docs.autousers.ai/api-reference/webhooks/list-deliveries-for-an-endpoint https://app.autousers.ai/api/v1/openapi.json get /api/v1/webhooks/{id}/deliveries Cursor-paginated delivery log for this endpoint, newest first. Filter by `status` to inspect only failed / pending / delivered / abandoned attempts. Defaults to 25 rows per page; `limit` caps at 100. # List webhook endpoints Source: https://docs.autousers.ai/api-reference/webhooks/list-webhook-endpoints https://app.autousers.ai/api/v1/openapi.json get /api/v1/webhooks List the webhook endpoints registered on the caller's team. Newest first. The plaintext signing `secret` is NEVER returned here — only `secret_prefix`. Returns `503 feature_not_available` until `AUTOUSERS_WEBHOOKS_ENABLED` is set. # Replay a failed delivery Source: https://docs.autousers.ai/api-reference/webhooks/replay-a-failed-delivery https://app.autousers.ai/api/v1/openapi.json post /api/v1/webhooks/deliveries/{id}/retry Enqueue a NEW delivery row pointing at the same event + endpoint, `attempt=1`, `scheduled_for=now`. The original row is preserved for audit; the worker drains the new row on its next tick. Pending rows are rejected (the worker hasn't tried yet). Disabled endpoints are rejected. # Rotate the signing secret Source: https://docs.autousers.ai/api-reference/webhooks/rotate-the-signing-secret https://app.autousers.ai/api/v1/openapi.json post /api/v1/webhooks/{id}/rotate-secret Atomically replace the endpoint's signing secret. The new plaintext `secret` is returned ONCE — same UX as create. Receivers that need a graceful rotation window should temporarily verify against both the old and the new secret; the verifier in `lib/webhooks/signing.ts` already supports multiple `v1=` values per signature header. Admin role or higher. # Send a synthetic test event Source: https://docs.autousers.ai/api-reference/webhooks/send-a-synthetic-test-event https://app.autousers.ai/api/v1/openapi.json post /api/v1/webhooks/{id}/test Enqueue a `webhook.test` event and a single matching delivery on this endpoint, regardless of the endpoint's `enabled_events` filter. Useful for smoke-testing connectivity and signature verification without waiting for a real evaluation to finish. Disabled endpoints are rejected — re-enable first. # Update a webhook endpoint Source: https://docs.autousers.ai/api-reference/webhooks/update-a-webhook-endpoint https://app.autousers.ai/api/v1/openapi.json patch /api/v1/webhooks/{id} Partial update of `url`, `description`, `enabled_events`, `api_version`, or `status`. Disabling does not delete deliveries — set `status: "disabled"` to pause dispatch and `"enabled"` to resume. Admin role or higher. # Authentication Source: https://docs.autousers.ai/authentication Four auth paths, thirteen scopes, one rule: bearer wins. Every `/v1` request authenticates as one of four principals. The server resolves them in a fixed order; the first match wins. | Order | Path | Carrier | Identity | Scopes | | ----- | ---------------- | ----------------------------------------- | --------------------- | ------------------------ | | 1 | Session | Supabase cookie | User | implicit (full) | | 2 | API key | `Authorization: Bearer ak_live_*` | API key → User + Team | explicit allowlist | | 3 | OAuth 2.1 | `Authorization: Bearer ` (aud-bound) | User + Team | explicit (`scope` claim) | | 4 | Figma plugin JWT | `Authorization: Bearer ` | User | implicit (full) | `session` and `figma-token` principals have **no** `scopes` field — they authenticate as a real user and inherit that user's team memberships. `apikey` and `oauth` principals must present an explicit scope per route or get `403 authorization_error: Missing required scope: ...`. Scopes are layered on top of, not in place of, the route's permission check. An `ak_live_*` key with `evaluations:write` scoped to team A still cannot write evaluations on team B — `assertTeamAccess` rejects the team mismatch. ## Path 1 — Session cookie (web app) Used by the dashboard at `app.autousers.ai`. Set automatically by Supabase Auth on sign-in. There is nothing for an integrator to do here. ```bash theme={null} curl https://app.autousers.ai/api/v1/auth/whoami \ --cookie "sb-...=..." ``` ## Path 2 — API key (`ak_live_*`) The right choice for **server-to-server**, **CI/CD**, **the CLI**, and **the MCP server**. Mint at **Settings → API keys**; you'll see the plaintext value once. ```bash theme={null} curl https://app.autousers.ai/api/v1/auth/whoami \ -H "Authorization: Bearer ak_live_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" ``` Properties: * Prefixed `ak_live_`; sha256-hashed at rest (never stored plaintext). * Bound to the team it was minted on. All requests act on that team. * Scopes are explicit and immutable for the life of the key. To change scopes, mint a new key and revoke the old one. * Optional expiry; default is no expiry. * Last-used IP and timestamp are recorded for audit. ## Path 3 — OAuth 2.1 access token Audience-bound HS256 JWT minted by `/api/oauth/token` after a user authorizes a third-party MCP app or SDK. Tokens are short-lived (≤15 minutes) and **must** carry an `aud` claim matching the resource server they are presented to (RFC 8707). A token minted for `/mcp` will **not** verify at `/api/v1/*`, and vice versa. ```bash theme={null} curl https://app.autousers.ai/api/v1/auth/whoami \ -H "Authorization: Bearer eyJ..." ``` Token payload: ```json theme={null} { "sub": "usr_clxq3...", "team": "team_clxq3...", "scope": "evaluations:read autousers:read", "aud": "https://app.autousers.ai/api/v1", "exp": 1714867200 } ``` Access tokens are **stateless**. Revoking the underlying refresh token stops future refreshes but does not invalidate access tokens already in flight; they expire naturally at their TTL. This is the standard OAuth 2.1 trade-off. ## Path 4 — Figma plugin JWT Issued by `/api/auth/figma/*` for the Autousers Figma plugin only. Stored in `figma.clientStorage`. 7-day expiry. HS256, signed with a secret distinct from the OAuth signing key — Figma JWTs cannot be replayed as OAuth tokens or vice versa. You will not need this unless you are forking the Figma plugin. ## Scopes Scopes are **resource:action** strings. Use the wildcard sparingly. | Scope | Grants | | ------------------- | ------------------------------------------------------ | | `*` | Admin — every action on every resource family. | | `templates:read` | List, read templates / dimensions. | | `templates:write` | Create, update, delete templates / dimensions. | | `templates:*` | All template actions. | | `evaluations:read` | List, read evaluations and their nested resources. | | `evaluations:write` | Create, update, delete evaluations. | | `evaluations:*` | All evaluation actions. | | `autousers:read` | List, read autousers, calibration, runs. | | `autousers:write` | Create, update, delete autousers; trigger calibration. | | `autousers:*` | All autouser actions. | | `ratings:read` | List ratings on evaluations the caller can read. | | `ratings:write` | Submit, bulk-delete ratings. | | `ratings:*` | All rating actions. | When **webhooks ship** (see [Changelog](/changelog)) three more scopes are added: `webhooks:read`, `webhooks:write`, `events:read`. The literal scope is checked first, then the family wildcard (`evaluations:*`), then the admin wildcard (`*`). ## Key rotation Mint a new key with the same scopes, deploy it, then revoke the old key. ```bash theme={null} # 1. Mint replacement curl -X POST https://app.autousers.ai/api/v1/api-keys \ --cookie "sb-...=..." \ -H "Content-Type: application/json" \ -d '{ "name": "ci-2026-q3", "scopes": ["evaluations:read", "evaluations:write"] }' # 2. Update your secret store with the new key. # 3. Revoke the old key. curl -X DELETE https://app.autousers.ai/api/v1/api-keys/ \ --cookie "sb-...=..." ``` `/api/v1/api-keys/*` is **session-only**. You cannot mint or revoke keys with another `ak_live_*` key — that prevents a leaked key from bootstrapping itself into permanence. Use the dashboard or a session cookie. ## Best practices * Mint a **separate key per environment** (CI, staging, prod). Revoke blast radius is the key, not the account. * Use the **narrowest scope** that works. Reach for `evaluations:read` before `evaluations:*`, and `*` only for break-glass admin tooling. * **Rotate keys** annually or when an employee with key access leaves. * **Set an expiry** on keys you only need for a known window (a launch, a contractor engagement). * Never log the `Authorization` header. Never paste a key into a chat transcript or a screenshot. # Changelog Source: https://docs.autousers.ai/changelog Every API-affecting change, dated, in plain English. Subscribe to the [RSS feed](https://docs.autousers.ai/changelog/rss.xml) or watch this page for every change to the public surface — new routes, new webhooks, new error codes, deprecations, breaking-version splits. We log every change. Even the additive, non-breaking ones. *** ## 2026-05-04 — Public API beta launch * **OpenAPI 3.1 spec** at `https://app.autousers.ai/api/v1/openapi.json` — full coverage of `/api/v1/*` (143+ operations across 80+ paths). * **Mintlify docs** at `https://docs.autousers.ai` (DNS pending). * **Webhooks (beta, flag-gated)** — 7 v1 events, Stripe-style HMAC-SHA256 signing, exponential backoff over \~3 days, `POST /v1/webhooks/{id}/test` for synthetic events. * **Stripe-style cursor params** — `starting_after`/`ending_before` now canonical on every list endpoint; legacy `cursor` param emits `Deprecation: true` headers (sunset 2027-04-04). * **CORS opens for `docs.autousers.ai`** — playground "Try it" works against production. * **`Autousers-Signature`, `Idempotency-Key`, `Autousers-Version`, `X-RateLimit-*`, `X-Request-Id`, `Deprecation`/`Sunset`/`Link`** documented and exposed via `Access-Control-Expose-Headers`. Coming next: tier-aware rate limiting + monthly quotas (flag-gated, observation-only today), `Idempotency-Key` replay (flag-gated, passthrough today), and typed SDKs for TypeScript and Python (planned, generated from the same OpenAPI spec the live API reference is built from). *** ## 2026-05-04 — Public API beta launch (extended notes) The big one. Three things shipped together: ### `/v1/*` is now the documented public surface * 100% of publishable routes are in the OpenAPI spec at [`/api/v1/openapi.json`](https://app.autousers.ai/api/v1/openapi.json). * The in-page playground works against production with a real `ak_live_*` key. * Concepts, errors, scopes, and pagination are documented at [`docs.autousers.ai`](https://docs.autousers.ai). ### Webhooks (beta, flag-gated) Seven event types ship behind the `AUTOUSERS_WEBHOOKS_ENABLED` flag: * `evaluation.created` * `evaluation.status_changed` * `evaluation.completed` * `autouser_run.completed` * `autouser_run.failed` * `rating.created` * `calibration.frozen` While the flag is off, every `/v1/webhooks/*` route returns `503 Service Unavailable`. The flip-the-switch announcement will land in the next entry. New scopes: `webhooks:read`, `webhooks:write`, `events:read`. Tier-gated to **Pro and Enterprise**. See [Webhooks → Overview](/webhooks/overview). ### Mintlify docs at `docs.autousers.ai` Hand-curated content for authentication, error envelope, scope vocabulary, signature verification (TS/Python/Go/Ruby), retry schedule, and four production recipes. Auto-generated route reference from the live OpenAPI spec. The product help center at [`autousers.ai/help`](https://autousers.ai/help) stays where it is — this is the developer reference. *** ## What's still rolling out A handful of items are documented but enforcement is still rolling out during beta. Build to the contract today; nothing changes when the switch flips. * **Tier-aware rate limits and monthly quotas.** The `X-RateLimit-Limit/Remaining/Reset` headers are populated; over-limit requests log but do not yet block. See [Rate limits](/rate-limits). * **`Idempotency-Key` enforcement.** Keys are accepted and recorded; full replay-on-conflict semantics ship next. * **TypeScript and Python SDKs.** Planned. The OpenAPI spec is the source of truth; SDK generation will be wired off it once the Wave 1 hardening (rate-limit + idempotency enforcement) lands. * **`Autousers-Version` request header.** Date-pinned versioning is designed but not yet live; default to current behaviour. * **Audit log API and status page.** Both queued for Wave 3 of the public-API rollout. *** ## Format conventions Every entry uses one of four labels in its heading: | Label | Meaning | | -------------- | ------------------------------------------------------------------------------ | | **Added** | New endpoint, new field, new event, new scope. Non-breaking. | | **Changed** | Behaviour adjusted within the existing contract. | | **Deprecated** | Marked with `Deprecation: true` + `Sunset` header. ≥12 month migration window. | | **Removed** | Sunset window has elapsed. Route now 410s. | Breaking changes never ship inside `/v1`. See [Versioning](/versioning) for the full policy. # AI insights Source: https://docs.autousers.ai/concepts/ai-insights Synthesised narratives over your ratings. Generated on demand, cached, regeneratable. Once you have ratings flowing, two AI surfaces summarise them: 1. **Standard insights** (`/ai-insights`) — narrative + structured action items per evaluation or comparison. 2. **Agreement insights** (`/agreement-insights`) — narrative explaining why raters agreed or disagreed, dimension by dimension. Both are generated by Gemini, cached on the row, and stale-detected by rating count. ## Standard insights ```bash theme={null} curl https://app.autousers.ai/api/v1/evaluations/$EVAL_ID/ai-insights \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` ```json theme={null} { "id": "ains_clxq3...", "evaluationId": "eval_clxq3...", "comparisonId": null, "insightType": null, "recommendation": "Strengthen trust signals on checkout v2 — every persona flagged it.", "insights": "Six raters across three personas converged on three themes: ...", "structured": { "actionItems": [ { "priority": "P1", "title": "Add a security badge near the CTA", "rationale": "..." }, { "priority": "P2", "title": "Surface review count on the order summary", "rationale": "..." } ] }, "ratingCountAtGeneration": 24, "generatedAt": "2026-05-04T11:05:00.000Z" } ``` If the row is **stale** (i.e. the evaluation has new ratings since generation), the response includes `stale: true`. Regenerate: ```bash theme={null} curl -X POST https://app.autousers.ai/api/v1/evaluations/$EVAL_ID/ai-insights \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` Per-comparison insights: ```bash theme={null} curl "https://app.autousers.ai/api/v1/evaluations/$EVAL_ID/ai-insights?comparisonId=$COMP_ID" \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` ## Agreement insights A separate row keyed on `insightType: "agreement"`. Where standard insights ask *"what did raters say?"*, agreement insights ask *"where did raters disagree, and why?"*. ```bash theme={null} curl https://app.autousers.ai/api/v1/evaluations/$EVAL_ID/agreement-insights \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` The narrative is calibrated to the Krippendorff α score: | α range | Tone | | -------------- | --------------------------------------------------------------------------------------- | | α ≥ 0.8 | "Strong agreement on X, Y, Z. Trust the result." | | 0.6 ≤ α \< 0.8 | "Moderate agreement; spotlight on dimension D." | | α \< 0.6 | "Raters disagreed substantially on D and E. Consider re-running with a clearer rubric." | ## Costs Each insight regeneration is a single Gemini completion (\~$0.001–$0.005 depending on rating count). Negligible. Doesn't count against autouser quota; counts against the per-user platform Gemini quota only when on the platform key (`User.geminiApiKey` BYOK callers don't count). ## Caching We **don't** auto-regenerate. The cache is invalidated only when you POST to the `/ai-insights` route. This keeps the cost predictable and prevents accidental cost cascades when a flurry of ratings comes in. A typical pattern in CI: ```ts theme={null} // 1. Wait for autouser_run.completed webhooks until all expected runs finish. // 2. Fetch insights once. // 3. Pin to the report. const insights = await fetch( `https://app.autousers.ai/api/v1/evaluations/${evalId}/ai-insights`, { method: "POST", headers: { Authorization: `Bearer ${KEY}` } } ).then((r) => r.json()); writeReport(insights.recommendation, insights.structured.actionItems); ``` ## Limitations * The model has access only to the rating data, not the actual designs. If you want it to reason about visual fidelity, post screenshots as files and reference them from `customDimensions[].context`. * Insights do not include dimensional scores you didn't actually collect. If you want it to reason about Accessibility, include an Accessibility dimension on the template. * Output is in English regardless of `Autousers-Locale` (currently no i18n on this surface — on the roadmap). # Autousers Source: https://docs.autousers.ai/concepts/autousers AI personas that drive a real browser, follow a calibrated rubric, and emit structured ratings. An **autouser** is a UX-research persona implemented as a Computer Use agent. It receives a stimulus (URL, screenshot, or video), navigates or inspects it the way a real first-time user would, and emits a Rating in the same shape a human rater produces. ## Anatomy | Field | Type | Notes | | ------------------- | ------- | -------------------------------------------------------- | | `id` | string | `auto_` for system; `auto_` for team. | | `name` | string | "First-time buyer", "Power user", "Skeptical evaluator". | | `role` | string | One-line persona summary surfaced to the model. | | `systemPrompt` | string | Full instructions. The model's context. | | `isSystem` | boolean | `true` for built-ins; `false` for team-created. | | `visibility` | enum | `private` (team) or `public` (team-publishable). | | `calibrationStatus` | enum | `uncalibrated`, `calibrating`, `calibrated`, `frozen`. | | `activeRubricId` | string? | The frozen rubric in use, if any. | System autousers are visible to every team. Team autousers are scoped to the team that created them. ## Listing ```bash theme={null} curl https://app.autousers.ai/api/v1/autousers \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` The default response includes both system and your team's autousers. Filter with `?source=system|team`. ## Creating a custom autouser ```bash theme={null} curl -X POST https://app.autousers.ai/api/v1/autousers \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Healthcare-portal patient", "role": "A patient managing chronic conditions through a hospital portal.", "systemPrompt": "You are a 58-year-old patient with hypertension and type-2 diabetes...", "visibility": "private" }' ``` A new autouser starts `uncalibrated` — it can rate, but its scores have no inter-rater reliability data yet. ## Calibration Calibration is how an autouser learns to **agree with itself** across runs. We feed it a small panel of comparisons, run it N times, measure the consistency of its ratings, and either freeze the rubric (locking in stable behaviour) or iterate the system prompt. Lifecycle: ``` uncalibrated ──► calibrating ──► calibrated ──► frozen ``` | State | Meaning | | -------------- | ------------------------------------------------------------------------------ | | `uncalibrated` | Never run a calibration pass. | | `calibrating` | A `CalibrationRun` is in flight. | | `calibrated` | Stable Krippendorff α ≥ 0.6 across recent self-runs. | | `frozen` | The rubric is locked. New evaluations use this rubric forever or until thawed. | Trigger: ```bash theme={null} curl -X POST https://app.autousers.ai/api/v1/autousers/$AUTOUSER_ID/calibration/start \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` Watch: ```bash theme={null} curl https://app.autousers.ai/api/v1/autousers/$AUTOUSER_ID/calibration/status \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` Freeze when stable: ```bash theme={null} curl -X POST https://app.autousers.ai/api/v1/autousers/$AUTOUSER_ID/calibration/freeze \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` Freezing emits a `calibration.frozen` webhook. Downstream pipelines should listen for it before promoting an autouser to production. ## Runs When you call `/v1/evaluations/{id}/run-autousers`, every entry in `selectedAutousers` is expanded by `agentCount` into individual `AutouserRun` rows. Each row tracks: * `status`: `pending` → `running` → `completed` or `failed`. * `currentStep`, `currentAction`, `currentNarration`: live worker progress. * `inputTokens`, `outputTokens`, `estimatedCostUsd`: cost telemetry. * `artifactsPath`: GCS prefix for video, screenshots, transcripts. A failed run does **not** consume autouser-rating quota. A completed run produces one Rating per Comparison; total ratings = `agentCount × comparisonCount`. ```bash theme={null} curl https://app.autousers.ai/api/v1/evaluations/$EVAL_ID/autouser-runs/$RUN_ID \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` ## Cost Autouser runs price against `gemini-3-flash-preview` (the only Gemini SKU with native Computer Use). Typical run: $0.04–$0.12 depending on page complexity, navigation depth, and dimension count. Use `dryRun` on the parent evaluation to forecast before queueing. ## Built-in personas We ship a roster of system autousers covering common roles — first-time buyer, power user, skeptical evaluator, accessibility-first user, support-call-prone novice. They are calibrated against an internal benchmark panel and updated quarterly. Custom personas always override built-ins for your team. # Calibration Source: https://docs.autousers.ai/concepts/calibration Measure and improve how closely an autouser's ratings agree with your human raters using Cohen's Kappa, then freeze the rubric when agreement is strong. Calibration measures how well an autouser agrees with your human raters. It works by comparing the autouser's dimension-level scores against human scores on the same evaluation and computing Cohen's Kappa — a statistical measure of inter-rater agreement that accounts for chance. When agreement is low, the calibration workflow shows you exactly where the autouser diverged from humans, and an AI-assisted optimizer suggests rubric changes to close the gap. Once you are satisfied with agreement, you freeze the rubric version so the autouser always rates against a stable, locked definition. Calibration is available on the **Team plan and above** (Team, Pro, BYOK, and Enterprise). It is not available on Free or Indie plans. ## When to calibrate Calibrate a custom autouser when: * You have just created a new custom autouser and want to verify it agrees with your team's judgment before using it in production evaluations. * Agreement scores on your evaluation results are lower than expected. * You have updated the autouser's system prompt and want to confirm the change improved (not degraded) agreement. You do not need to calibrate built-in autousers — they are maintained by Autousers and validated before release. ## Understanding Cohen's Kappa Cohen's Kappa (κ) measures agreement between two raters while correcting for the level of agreement you would expect by chance alone. It ranges from −1 to 1. | Kappa range | Interpretation | | ----------- | --------------------------------------------------------------------------- | | \< 0.2 | Poor agreement — the autouser and humans are rating very differently | | 0.2 – 0.4 | Fair agreement — some alignment, but substantial divergence remains | | 0.4 – 0.6 | Moderate agreement — acceptable for exploratory use, worth optimizing | | 0.6 – 0.8 | Substantial agreement — the autouser is a reliable proxy for human judgment | | > 0.8 | Near-perfect agreement — the autouser closely tracks human ratings | Aim for κ ≥ 0.6 before relying on an autouser as your primary rater. Freeze the rubric once you reach the agreement level your team considers acceptable. ## The calibration workflow Calibration requires an evaluation that has both autouser ratings and human ratings on the same comparisons. If you do not yet have human ratings, invite raters via a shareable link or add team members as raters before starting calibration. In the dashboard, open your autouser and go to the **Calibration** tab, then click **Start calibration**. Select the evaluation whose human ratings you want to calibrate against. Via the MCP or API, call `autousers_calibration_start` with the autouser ID and evaluation ID: ```json theme={null} { "id": "au_your_autouser_id", "evaluationId": "ev_your_evaluation_id" } ``` The system pairs the autouser's scores with human scores on the same dimensions and computes Cohen's Kappa overall and per dimension. Check the computed Kappa score and the list of disagreements — individual ratings where the autouser and humans scored the same dimension differently. Via MCP or API, call `autousers_calibration_status_get` to retrieve the current calibration state, overall Kappa, per-dimension Kappa, and the disagreement list: ```json theme={null} { "id": "au_your_autouser_id" } ``` If agreement is lower than you want, use **Optimize** to send the disagreement data to the AI for rubric suggestions. The optimizer analyzes where the autouser diverged from humans and proposes specific changes to the rubric criteria text to align the autouser's scoring behavior. In the dashboard, click **Optimize** on the calibration screen. Via MCP or API, call `autousers_calibration_optimize` with the autouser ID and the disagreement payload: ```json theme={null} { "id": "au_your_autouser_id", "disagreements": [ { "ratingId": "rat_123", "humanReasoning": "The navigation was confusing because..." } ] } ``` Review the suggested changes, edit them if needed, then re-run calibration to see the updated Kappa score. When you are satisfied with the agreement level, freeze the current rubric version. Freezing locks the rubric so it cannot be modified and sets it as the autouser's active rubric for all future evaluations. In the dashboard, click **Freeze rubric**. Via MCP or API, call `autousers_calibration_freeze`: ```json theme={null} { "id": "au_your_autouser_id", "commitMessage": "v2 — optimized for checkout flow evaluations" } ``` After freezing, the autouser's calibration status changes to `"frozen"` and a `calibration.frozen` webhook fires (see [events](/webhooks/events)) so your downstream tooling can promote the new rubric automatically. ## MCP tools for calibration If you work with Autousers through the MCP server, these four tools cover the full calibration workflow: | Tool | What it does | | ---------------------------------- | ---------------------------------------------------------------------------------- | | `autousers_calibration_start` | Compute Cohen's Kappa between the autouser and human ratings on a given evaluation | | `autousers_calibration_status_get` | Return the current calibration status, Kappa scores, and disagreement list | | `autousers_calibration_optimize` | Send disagreements to AI for rubric improvement suggestions | | `autousers_calibration_freeze` | Lock the current rubric version and set it as active | The MCP prompt `calibrate-autouser` runs the full calibration loop for you — start, review, optimize, and freeze — in a single guided workflow. You can also use `triage-low-agreement` to surface disagreements and get suggested fixes without committing to a new rubric version. ## See also * [Autousers](/concepts/autousers) — what an autouser is and how to create one. * [Ratings & agreement](/concepts/ratings-and-agreement) — the rating shape and the agreement formula. * [`calibration.frozen` webhook](/webhooks/events#calibration-frozen) — fires when a rubric is locked, so downstream pipelines can promote it. # Evaluations Source: https://docs.autousers.ai/concepts/evaluations The study container. SSE for a single design, SxS for a comparison. Lifecycle: Draft → Running → Ended. An **evaluation** is the unit of UX research. It owns one or more **comparisons** (the things being rated), a **template** (the dimensions to rate against), the **autousers** queued to rate them, and the **ratings** those autousers and your humans produce. ## Types | `type` | Stimuli | When to use it | | ------ | -------------------------------------------------------------------- | ------------------------ | | `SSE` | Single Stimulus Evaluation — N independent designs (`designUrls[]`). | "Rate the new checkout." | | `SxS` | Side-by-Side Comparison — N pairs (`comparisonPairs[]`). | "Is v2 better than v1?" | The shape determines what the rater sees: SSE renders a single panel per comparison, SxS renders A and B together. ## Lifecycle ``` Draft ──────► Running ──────► Ended │ │ │ └─ autousers queued, ratings flowing │ └─ wizard state, no work yet ``` Status transitions emit a `evaluation.status_changed` webhook (see [events](/webhooks/events)). | Status | Meaning | | --------- | ---------------------------------------------------------------- | | `Draft` | Wizard state. Editable. No autousers queued, no ratings allowed. | | `Running` | Autousers may be queued. Public share link is live. | | `Ended` | Closed for new ratings. Results are read-only. | ## Creating an evaluation The minimum viable SSE evaluation: ```bash theme={null} curl -X POST https://app.autousers.ai/api/v1/evaluations \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" \ -H "Content-Type: application/json" \ -H "Idempotency-Key: $(uuidgen)" \ -d '{ "name": "Checkout v2", "type": "SSE", "status": "Draft", "designUrls": [ { "url": "https://staging.example.com/checkout", "label": "v2", "stimulusType": "URL" } ], "selectedAutousers": [ { "autouserId": "auto_first_time_buyer", "agentCount": 3 } ], "selectedDimensionIds": ["overall", "trust", "clarity"], "evaluationMethod": "ai" }' ``` Always pass `dryRun: true` first to validate and price the run without committing. See [Quickstart](/quickstart#4-dry-run-a-new-evaluation). The response includes a `links` object with absolute URLs to the preview, review, edit, results, and public share pages — surface these in your UI rather than constructing URLs yourself. ## Running the autousers Creating an evaluation does **not** queue autousers — that's a separate call so you can stage a draft without spend. ```bash theme={null} curl -X POST https://app.autousers.ai/api/v1/evaluations/$EVAL_ID/run-autousers \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` The endpoint flips the evaluation to `Running` (if it was `Draft`), expands `selectedAutousers` by `agentCount` into individual `AutouserRun` rows, and enqueues them on the GKE worker. Each run takes 1–6 minutes depending on the design complexity. Subscribe to the `autouser_run.completed` webhook for completion. Or poll `GET /v1/evaluations/{id}/autouser-status` for an aggregate snapshot. Or open `GET /v1/evaluations/{id}/autouser-stream` for an SSE event stream. ## Reading results ```bash theme={null} # Aggregated dimension scores + rater counts curl https://app.autousers.ai/api/v1/evaluations/$EVAL_ID/results \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" # Per-rating raw data curl https://app.autousers.ai/api/v1/evaluations/$EVAL_ID/ratings \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` Use `/results` for dashboards (it's already aggregated). Use `/ratings` for warehouse sync (full row-level data, paginate). ## Sharing Every evaluation has a public share token. Sharing modes: | `shareAccess` | Behaviour | | -------------------- | ---------------------------------------------- | | `TEAM_ONLY` | Default. Only team members can view. | | `LINK_ONLY` | Anyone with the share URL can rate. | | `PASSWORD_PROTECTED` | Requires `sharePassword` (≥4 chars). | | `EMAIL_GATED` | Public raters supply name/email before rating. | See [Teams & permissions](/concepts/teams-and-permissions) for the share ACL. ## Deletion ```bash theme={null} curl -X DELETE https://app.autousers.ai/api/v1/evaluations/$EVAL_ID \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` Cascades to comparisons, ratings, autouser runs, ai insights, shares, invites, access requests. **Irreversible.** Quota counts already burned on autouser runs are **not** refunded. ## See also * [Autousers](/concepts/autousers) — how the personas queue, run, calibrate. * [Templates](/concepts/templates) — what gets rated. * [Ratings & agreement](/concepts/ratings-and-agreement) — the scoring shape. # Files & stimuli Source: https://docs.autousers.ai/concepts/files Three-step signed-URL upload. Direct-to-GCS, never through our origin. When your stimulus is a URL, just pass it on the evaluation create body. When it's a screenshot, video, or PDF, use the file upload flow. The upload is a **three-step signed-URL handshake** — you talk to our API to get a signed URL, you `PUT` the bytes directly to Google Cloud Storage, then you tell us you're done. No bytes ever traverse our origin. ## Why three steps * **Cost:** signed URLs are free; piping 200MB videos through Vercel functions is not. * **Speed:** GCS regional edge upload beats round-tripping through us. * **Security:** we never see the file bytes; you never see our service account. ## Flow ``` 1. POST /v1/files/init → { fileId, signedUrl, headers, expiresAt } 2. PUT → 204 (direct to GCS) 3. POST /v1/files/{id}/finalize → { fileId, sha256, sizeBytes, ready: true } ``` Then reference the `fileId` on `designUrls[].fileId` or `comparisonPairs[].sideAFileId` when creating an evaluation. ## Step 1 — initialise ```bash theme={null} curl -X POST https://app.autousers.ai/api/v1/files/init \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "filename": "checkout-v2.png", "mimeType": "image/png", "sizeBytes": 482103 }' ``` Response: ```json theme={null} { "fileId": "file_clxq3...", "signedUrl": "https://storage.googleapis.com/autousers-assets/...?X-Goog-Signature=...", "headers": { "Content-Type": "image/png", "x-goog-content-length-range": "0,524288000" }, "expiresAt": "2026-05-04T10:21:08.000Z" } ``` Signed URLs expire in **15 minutes**. ## Step 2 — PUT the bytes Use the headers from step 1 verbatim. Adding extra headers will void the signature. ```bash theme={null} curl -X PUT "$SIGNED_URL" \ -H "Content-Type: image/png" \ -H "x-goog-content-length-range: 0,524288000" \ --data-binary "@checkout-v2.png" ``` A successful PUT returns `204 No Content`. ## Step 3 — finalise ```bash theme={null} curl -X POST https://app.autousers.ai/api/v1/files/$FILE_ID/finalize \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` ```json theme={null} { "fileId": "file_clxq3...", "sha256": "9a3f8e...", "sizeBytes": 482103, "mimeType": "image/png", "ready": true } ``` Finalise validates that the bytes actually landed and computes a sha256 checksum. **Do not skip it** — an unfinalised file is invisible to evaluation create. ## Reading a file ```bash theme={null} curl https://app.autousers.ai/api/v1/files/$FILE_ID \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` The route gates on team membership and returns a short-lived signed GET URL. Render images directly from that URL; do not proxy them through your application. ## Limits | Type | Max size | Notes | | ----------------- | -------- | --------------------------------------- | | `image/png` | 25 MB | Recommended for static design captures. | | `image/jpeg` | 25 MB | | | `application/pdf` | 100 MB | One PDF, multiple pages OK. | | `video/mp4` | 500 MB | H.264 / H.265. | Files are retained for the life of the evaluation. Deleting an evaluation deletes its files. ## Doctor A self-test endpoint to diagnose upload issues: ```bash theme={null} curl https://app.autousers.ai/api/v1/files/doctor \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` Returns a list of recent failed uploads on your team and the likely cause (signed URL expired, Content-Type mismatch, finalise never called). # Concepts Source: https://docs.autousers.ai/concepts/overview How Autousers thinks. Read this before writing your first integration. Autousers is an AI-first UX evaluation platform. The data model is shaped by that — five nouns explain almost everything you'll do through the API. ## The five nouns A study. Either an SSE (single design review) or an SxS (side-by-side comparison). Has a lifecycle Draft → Running → Ended. An AI persona that drives a real browser, navigates the design, and emits structured ratings. A reusable question set, composed of dimensions like Usability, Visual Design, Accessibility. A single rater's verdict on a single comparison. Humans and autousers produce the same shape. The ownership boundary. Every resource lives on exactly one team. ## How they fit together ``` Team └── Evaluation (status: Draft | Running | Ended; type: SSE | SxS) ├── Comparison (one per stimulus, or one per A/B pair) │ └── Rating (per rater × per comparison × per dimension) ├── Autouser (selected persona, expanded by agentCount) │ └── AutouserRun │ └── Rating ├── Template (one per evaluation; composed of dimensions) └── AiInsight (auto-generated narrative + structured action items) ``` A typical SxS workflow looks like: 1. `POST /v1/evaluations` with `type: "SxS"`, two `comparisonPairs`, three `selectedAutousers`, a chosen template. 2. `POST /v1/evaluations/{id}/run-autousers` queues the AI runs. 3. (Optional) raters land on the public share link and produce human ratings. 4. `GET /v1/evaluations/{id}/agreement` computes Krippendorff α across all raters. 5. `GET /v1/evaluations/{id}/ai-insights` returns a synthesised narrative. ## Where the work happens | Component | Runs in | Bills against | | -------------- | -------------------------- | ------------------------------------------- | | Autouser run | GKE Autopilot, browser pod | `autouser_ratings_mo` quota + token cost | | Human rating | The rater's browser | `human_ratings_mo` quota | | Agreement calc | Postgres + Node, on demand | Free, cached on `Evaluation.agreementCache` | | AI insights | Gemini, on demand | Negligible — single completion | ## What's deliberately not in `/v1` * **Bulk evaluation creation** — every eval has nuance; we'd rather you loop than express that complexity in a single payload. * **Synchronous autouser runs** — too slow. The pattern is "POST → webhook" (see [Webhooks](/webhooks/overview)). * **Public listing of all autousers** — system autousers (`isSystem: true`) are visible to all; team autousers are scoped to the team. ## Concept reference If you're new to UX evaluation as a discipline (not just to this API), the expanded definitions below cover the seven nouns the platform turns on. Skip this section if you've already integrated with similar tools — the five-noun model above is enough. An evaluation is the study container. It holds everything needed to run a UX study: the URLs or design files being tested, the autousers and human raters assigned to it, the dimensions being rated, and all the resulting ratings and scores. Autousers supports two evaluation types: * **SSE (single experience)** — evaluates one URL or design file. Each rater assesses the experience on its own merits across the selected dimensions. Use SSE for baseline quality checks, regression testing, or first-impression studies. * **SxS (side-by-side)** — compares two URLs or design files head-to-head. Each rater sees both sides and rates them against each other. Use SxS when you want to know which of two designs performs better. Evaluations belong to a team and can be shared with collaborators or external stakeholders via a shareable link. An autouser is an AI persona that acts as a rater in an evaluation. When you run an autouser against an evaluation, it launches a browser session, navigates your live URL using Computer Use, and then produces a structured rating for each selected dimension — including a numeric score and written rationale. Autousers come in two kinds: * **Built-in autousers** are maintained by Autousers and available to every team. They cover a range of representative user archetypes — different goals, device preferences, accessibility needs, and interaction styles. * **Custom autousers** are team-scoped personas you define yourself. You write the role, background, and evaluation rubric, and the autouser applies that lens consistently across every run. Because autousers browse live URLs, they evaluate your product the way a real user would — not a static screenshot or a mock. Their ratings reflect actual navigation behaviour. A template is a reusable evaluation configuration. It stores a set of dimensions, instructions, and other settings so you don't have to reconfigure the same study from scratch each time. Templates are scoped to your team. When you create an evaluation using a template, the template's dimensions and settings are copied into the evaluation — changes to the template afterwards don't affect existing evaluations. Custom dimensions you define in the evaluation wizard are automatically saved as team templates so you can reuse them in future evaluations. A dimension is an axis on which a design is rated. Every rating is structured as a score per dimension, which means you get granular feedback rather than a single overall number. Autousers includes four built-in dimensions: * **Overall** — a holistic quality score for the experience. * **Usability** — how easy the design is to navigate and use. * **Visual** — the quality of the visual design, layout, and aesthetics. * **Accessibility** — how well the design accommodates users with different needs. You can also define custom dimensions for your specific evaluation goals — for example, "Onboarding clarity", "Trust signals", or "Checkout friction". Custom dimensions are saved to your team's template library automatically. A rating is the structured output produced by a rater — either an autouser or a human — for one design in an evaluation. Each rating contains a score per selected dimension and written commentary explaining the score. Ratings from multiple raters on the same evaluation are aggregated into: * **Aggregate scores** — the average per dimension across all raters. * **Per-rater breakdowns** — individual scores and rationale so you can see where raters agree or diverge. * **AI insights** — a synthesised summary of key findings and recommendations generated from the full set of ratings. Human raters can be invited to an evaluation via a shareable link. They complete ratings in the Autousers rater interface without needing an account, depending on your sharing settings. Calibration is the process of measuring how closely an autouser agrees with human raters on the same designs, and then improving the autouser's rubric when agreement is low. The agreement score is expressed as **Cohen's Kappa** — a statistical measure that accounts for chance agreement. A Kappa of 1.0 means perfect agreement; 0.0 means no better than chance. The calibration workflow: 1. Run the same evaluation with both autousers and human raters. 2. Start calibration — Autousers computes pairwise Kappa scores between each autouser and each human rater. 3. If agreement is low, use the **Optimise** tool to send disagreements to the AI for rubric suggestions. 4. Adjust the autouser's rubric based on the suggestions and re-run calibration. 5. When agreement is stable and acceptable, **freeze** the rubric version. The frozen version becomes the active rubric for future runs. Calibration is available on Team, Pro, BYOK, and Enterprise plans. See the [calibration page](/concepts/calibration) for the full workflow with code samples. A team is the organisational unit in Autousers. Evaluations, autousers, templates, and custom dimensions all belong to a team. When you sign up, Autousers creates a personal team for you automatically. Team members have one of three roles: * **Owner** — full access, including billing and team settings. * **Editor** — can create and run evaluations, manage autousers and templates. * **Viewer** — read-only access to evaluations and results. You can belong to multiple teams — for example, one for your organisation and one for a client project. API keys are issued per user but operate within the scope of the teams you belong to. ## See also * [Authentication](/authentication) — who can read which evaluations. * [Webhooks](/webhooks/events) — what events these objects emit. * [API reference](/api-reference) — every route, every field. # Ratings & agreement Source: https://docs.autousers.ai/concepts/ratings-and-agreement Per-rater verdicts and the reliability math that aggregates them. A **Rating** is a single rater's verdict on a single comparison. Humans and autousers produce the same shape so downstream analytics doesn't have to branch on `raterType`. ## Shape ```json theme={null} { "id": "rat_clxq3...", "evaluationId": "eval_clxq3...", "comparisonId": "cmp_clxq3...", "raterType": "human", "userId": "usr_clxq3...", "publicRaterId": null, "autouserId": null, "autouserRunId": null, "rubricVersion": "v3", "dimensionRatings": { "overall": 4, "trust": 3, "clarity": 5 }, "openTextResponses": { "overall": "Felt fast but the trust signals were thin.", "trust": "No security badge, no review count." }, "factors": null, "justification": "Liked the simplicity, missed the social proof.", "skipReason": null, "timeSpentSeconds": 87, "timingData": { /* ... */ }, "createdAt": "2026-05-04T10:21:08.123Z" } ``` Discriminator: | Field | Set when | | --------------- | ----------------------------------------- | | `userId` | Authenticated user submitted this rating. | | `publicRaterId` | Anonymous public rater (no account). | | `autouserId` | An autouser run produced it. | | `autouserRunId` | The specific run row. | Exactly one of `(userId, publicRaterId)` is set on human ratings; exactly one of `(autouserId, autouserRunId)` is set on autouser ratings. ## Listing ```bash theme={null} curl "https://app.autousers.ai/api/v1/evaluations/$EVAL_ID/ratings?limit=100" \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` Cursor-paginate with `starting_after`. See [Pagination](/pagination). ## Submitting a human rating via the API Most ratings come from the dashboard or the public share link. If you need to submit one programmatically (e.g. wiring up a custom rater UI): ```bash theme={null} curl -X POST https://app.autousers.ai/api/v1/evaluations/$EVAL_ID/ratings \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" \ -H "Content-Type: application/json" \ -H "Idempotency-Key: $(uuidgen)" \ -d '{ "comparisonId": "cmp_clxq3...", "dimensionRatings": { "overall": 4, "trust": 3, "clarity": 5 }, "justification": "Smooth flow, weak trust signals.", "timeSpentSeconds": 87 }' ``` ## Agreement Once you have ratings from ≥3 raters per comparison, agreement metrics become useful. The `/agreement` endpoint computes Krippendorff α and, when there are exactly two raters, Cohen κ. ```bash theme={null} curl https://app.autousers.ai/api/v1/evaluations/$EVAL_ID/agreement \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` ```json theme={null} { "krippendorff": { "alpha": 0.74, "n_raters": 6, "n_items": 4 }, "byDimension": { "overall": { "alpha": 0.81 }, "trust": { "alpha": 0.62 }, "clarity": { "alpha": 0.79 } }, "ratingCount": 24, "cachedAt": "2026-05-04T11:02:13.000Z" } ``` ### What the numbers mean | α range | Reading | | -------------- | ---------------------------------------------------- | | α \< 0.4 | No agreement. Treat results as anecdote, not signal. | | 0.4 ≤ α \< 0.6 | Weak. Useful directional, not for promotion gating. | | 0.6 ≤ α \< 0.8 | Acceptable. Most teams ship gates at α ≥ 0.6. | | α ≥ 0.8 | Strong. Suitable for automated CI gates. | Cohen κ uses the same scale. When >2 raters are present we report only Krippendorff (κ is undefined for >2 raters). ## Caching Agreement is cached on `Evaluation.agreementCache` and only recomputed when the rating count changes. The first call after a new rating is slightly slower (\~100ms) as it warms the cache; subsequent calls are instant. ## Streaming ratings into a warehouse The shape is stable — `dimensionRatings` is a JSON map, `factors` and `openTextResponses` are JSON. Subscribe to the `rating.created` webhook (see [Events](/webhooks/events)) and append rows to BigQuery / Snowflake as they arrive. Use `Autousers-Event-Id` as the dedup key on insert. See the [Looker / BigQuery recipe](/integrations/recipes#looker-bigquery-sync). # Teams & permissions Source: https://docs.autousers.ai/concepts/teams-and-permissions The ownership boundary. One resource, one team, four roles. Every resource in Autousers — Evaluation, Autouser, Template, ApiKey, WebhookEndpoint — lives on exactly one **Team**. Permissions cascade from team membership; sharing breaks open select per-evaluation exceptions. ## Roles | Role | Read | Edit own | Edit team | Manage members | Billing | | ------ | :--: | :------: | :-------: | :------------: | :-----: | | Viewer | ✓ | | | | | | Editor | ✓ | ✓ | ✓ | | | | Admin | ✓ | ✓ | ✓ | ✓ | | | Owner | ✓ | ✓ | ✓ | ✓ | ✓ | Every team has exactly **one Owner**. Use `POST /v1/teams/{id}/transfer-admin` to hand off. ## Auth and team binding | Principal | Team binding | | ------------ | --------------------------------------------------------------- | | Session | All teams the user belongs to. | | `ak_live_*` | Pinned to the team it was minted on. Cannot act on other teams. | | OAuth 2.1 | The team the user consented under at `/authorize` time. | | Figma plugin | All teams the user belongs to. | API key callers cannot pivot to a different team mid-request. To act on a different team, mint a new key on that team. ## Reading your teams ```bash theme={null} curl https://app.autousers.ai/api/v1/teams \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` ## Members ```bash theme={null} curl https://app.autousers.ai/api/v1/teams/$TEAM_ID/members \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` ```bash theme={null} # Invite a new member curl -X POST https://app.autousers.ai/api/v1/teams/$TEAM_ID/members \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "email": "alice@example.com", "role": "Editor" }' ``` Role changes emit no webhook today; subscribe to the audit log API (coming Wave 3) for that signal. ## Per-evaluation sharing Beyond team membership, you can share a single evaluation with people who are **not** on the team. Three primitives: | Primitive | When to use it | | ----------------------------- | ------------------------------------------------------------------- | | **Public share token** | Mass distribution. One URL, anyone with it can rate. | | **`EvaluationShare`** | Named individuals — a specific email gets a specific permission. | | **`EvaluationInvite`** | Pre-account invitation — sends an email, becomes a Share on accept. | | **`EvaluationAccessRequest`** | Someone with the link who lacks access asks for it. | Edit-on-share is gated by `editorsCanShare` on the evaluation row — default `true`. When the Owner flips it off, only Admin+ can edit sharing. ```bash theme={null} # List shares curl https://app.autousers.ai/api/v1/evaluations/$EVAL_ID/shares \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" # Add a share curl -X POST https://app.autousers.ai/api/v1/evaluations/$EVAL_ID/shares \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "email": "external@example.com", "permission": "Viewer" }' ``` ## "Not found" vs "no access" We deliberately conflate the two. A 404 from `/v1/evaluations/{id}` means either "no such row" or "you lack permission". This avoids leaking the existence of resources to outsiders. The single exception: cross-team requests where the caller has a membership on **a** team but not the resource's team return a 403 with `code: evaluation_no_access` and a `details` object containing the resource's team name and the caller's email. The dashboard uses this to render a "switch account" empty state. ## Deletion Deleting a team cascades to every resource on it — evaluations, autousers, templates, API keys, webhook endpoints. **Irreversible.** Owner-only. ```bash theme={null} curl -X DELETE https://app.autousers.ai/api/v1/teams/$TEAM_ID \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` The Owner gets a final-warning email 24 hours before any team-deletion endpoint is wired up to the dashboard. (As of this writing, team deletion is dashboard-only — no API route.) # Templates Source: https://docs.autousers.ai/concepts/templates Reusable question sets composed of dimensions. The canonical noun. A **template** is a reusable rating question set. Each template is composed of one or more **dimensions** like Usability, Visual Design, Accessibility, Trust, Clarity. Every Rating attaches a value to each dimension on the active template. **Templates is the canonical noun.** The legacy alias `/v1/dimensions/*` continues to work but carries `Deprecation: true` and `Sunset: Sun, 04 Apr 2027 00:00:00 GMT`. New code should use `/v1/templates/*` exclusively. ## Built-in vs custom | Kind | Owner | Visible to | Editable by | | -------- | ------ | -------------------------- | ----------- | | Built-in | system | All teams | Read-only | | Custom | team | Members of the owning team | Editor+ | Built-ins cover the common rubrics — UMUX-Lite, NPS-style trust scales, visual-design heuristics, accessibility shorthand. Forking a built-in into a custom template is a one-call duplicate. ## Listing ```bash theme={null} curl "https://app.autousers.ai/api/v1/templates?limit=50" \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` ## Creating a custom template ```bash theme={null} curl -X POST https://app.autousers.ai/api/v1/templates \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" \ -H "Content-Type: application/json" \ -H "Idempotency-Key: $(uuidgen)" \ -d '{ "name": "Mobile commerce — short rubric", "description": "Three dimensions, 1-5 scale.", "dimensions": [ { "id": "trust", "label": "Trust", "scale": "1-5" }, { "id": "clarity", "label": "Clarity", "scale": "1-5" }, { "id": "speed", "label": "Perceived speed", "scale": "1-5" } ] }' ``` Templates are versioned (`DimensionVersion` rows) — editing a template in active use does not retroactively change the dimensions of past ratings. ## The `selectedDimensionIds` field When creating an evaluation, you pass `selectedDimensionIds: string[]` on the create body. Each id is either: * A built-in dimension key (`overall`, `trust`, `clarity`, `accessibility`, …). * A custom dimension id from your team's templates. * A new id you supply on `customDimensions[]` — autosaved into your team's templates atomically with the evaluation create. The autosave behaviour means you can compose a one-off rubric inline on a dryRun, see the cost, then commit — and the new dimensions land in `/v1/templates` for future reuse without an extra round-trip. ## Why "templates" and not "dimensions" The original schema called these "dimensions" — single-axis scales like Usability or Accessibility. But the user's mental model is the **set**, not the axis: "the accessibility template", not "the accessibility dimension". The route alias rename in 2026-04 reflects that. Internally the rows still live in the `dimensions` table; externally the noun is template. ## Sunsetting an alias Until 04 Apr 2027: ```http theme={null} GET /v1/dimensions HTTP/1.1 200 OK Deprecation: true Sunset: Sun, 04 Apr 2027 00:00:00 GMT Link: ; rel="successor-version" ``` After 04 Apr 2027 the route 410s. See [Versioning](/versioning). # Errors Source: https://docs.autousers.ai/errors One envelope, six error types, every response carries X-Request-Id. Every error response shares a single envelope. The HTTP status is derived from the `error.type` field — clients should branch on `type`, not on status code. ```json theme={null} { "error": { "message": "Evaluation not found", "type": "not_found_error", "code": "evaluation_not_found", "request_id": "0192c7f8-dead-beef-9abc-def012345678" } } ``` ## Envelope fields | Field | Type | Always set | Notes | | ------------ | ------ | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `message` | string | yes | Human-readable. Self-sufficient for surfacing to end users. | | `type` | enum | yes | One of six values, see below. | | `code` | string | no | Machine-readable sub-classifier (e.g. `evaluation_not_found`). | | `param` | string | no | The offending request field, when applicable (`type: invalid_request_error`). | | `doc_url` | string | no | Link to the relevant doc page. | | `request_id` | string | yes | uuid v7. Same value as the `X-Request-Id` response header. | | `details` | object | no | Structured context. Today only set for cross-team `authorization_error` (`code: evaluation_no_access`). Treat as additive — `message` always tells the full story. | ## Error types and status codes | `error.type` | HTTP status | When | | ----------------------- | ----------- | ------------------------------------------------------------------- | | `authentication_error` | 401 | Missing, malformed, expired, or revoked credentials. | | `authorization_error` | 403 | Authenticated, but lacks the required scope or team permission. | | `invalid_request_error` | 400 | Malformed JSON, validation failure, missing required field. | | `not_found_error` | 404 | Resource does not exist, or the caller has no permission to see it. | | `rate_limit_error` | 429 | Per-tier RPM or monthly quota exceeded. Carries `Retry-After`. | | `api_error` | 500 | Unexpected server error. Quote the `request_id` to support. | We **conflate "not found" and "no permission"** by design. A 404 from `/v1/evaluations/{id}` means either "no such row" or "you lack membership on its team" — we don't disclose which, to avoid leaking the existence of resources to outsiders. ## `X-Request-Id` Every response — success or error — carries an `X-Request-Id` header. Error responses also embed the same value in `error.request_id` so it survives a copy-paste from a screenshot. ```http theme={null} HTTP/1.1 404 Not Found Content-Type: application/json X-Request-Id: 0192c7f8-dead-beef-9abc-def012345678 ``` The id is a **uuid v7**, sortable by time, which makes log scans cheap. It is the single piece of context you should include in every support ticket. Do not include the request body; we can pull that ourselves from the request id. ```bash theme={null} curl https://app.autousers.ai/api/v1/evaluations/eval_does_not_exist \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" \ -i ``` ## Common error codes A non-exhaustive list of `error.code` values you may encounter. Codes are stable; new ones are added without a version bump. The evaluation id does not exist, or the caller has no permission. 404. Cross-team request — the resource exists but lives on a team the caller is not a member of. The `details` field carries the resource team name and the caller's email so the UI can render a "switch account" empty state. 403. An `ak_live_*` or OAuth principal lacks the scope this route demands. The `message` names the scope. 403. The supplied `Idempotency-Key` matches a prior request, but the request body differs. 409. See [Idempotency](/idempotency). The route is gated to a higher pricing tier. Today this fires on webhook endpoints for Free / Team callers. 403. Per-tier RPM cap hit. Read `Retry-After`. 429. See [Rate limits](/rate-limits). Monthly quota for autouser ratings, human ratings, or evaluations exhausted. 429\. Carries no `Retry-After` — the quota resets at the start of the next billing period. ## Handling errors Branch on `error.type` first, `error.code` second. Never branch on the HTTP status — the mapping is stable but the field is the contract. ```ts theme={null} async function autousersFetch(path: string, init?: RequestInit) { const res = await fetch(`https://app.autousers.ai${path}`, { ...init, headers: { ...init?.headers, Authorization: `Bearer ${process.env.AUTOUSERS_API_KEY}`, }, }); if (!res.ok) { const body = await res.json(); const requestId = res.headers.get("x-request-id"); if (body.error?.type === "rate_limit_error") { const retryAfter = Number(res.headers.get("retry-after") ?? 60); await new Promise((r) => setTimeout(r, retryAfter * 1000)); return autousersFetch(path, init); } throw new Error(`${body.error?.message} (request_id=${requestId})`); } return res.json(); } ``` ## Reporting a bug Email `contact@autousers.ai` with the `request_id`, the route, and the local time of the request. Do not paste the request body — we can pull it. Do not paste the API key — we cannot help if you do, and will revoke the key on receipt. # Idempotency Source: https://docs.autousers.ai/idempotency Safe retries on billable POSTs. One key per logical request, 24-hour TTL. Set the `Idempotency-Key` header on any retried POST so a network hiccup never double-charges the team. ```bash theme={null} curl -X POST https://app.autousers.ai/api/v1/evaluations \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" \ -H "Content-Type: application/json" \ -H "Idempotency-Key: $(uuidgen)" \ -d '{ "name": "...", "type": "SSE", "...": "..." }' ``` If the request reaches us, we record the response. If you retry with the **same key** within 24 hours, we replay the original response — same status, same body, same `X-Request-Id` — without re-running the work. **Beta status (2026-05-04):** the `Idempotency-Key` header is accepted but enforcement is rolling out. During beta the server records keys but does not yet replay; behaviour is "best-effort safe retries". Watch the [Changelog](/changelog). ## When to use it Any time **the request mutates state and a retry could double-bill**. In practice that is: | Route | Why idempotency matters | | ----------------------------------------- | ------------------------------------------------ | | `POST /v1/evaluations` | Creates an evaluation. Re-runs spawn duplicates. | | `POST /v1/evaluations/{id}/ratings` | Submits a rating. Re-submits skew agreement. | | `POST /v1/evaluations/{id}/run-autousers` | Queues paid autouser runs. The expensive one. | | `POST /v1/api-keys` | Mints a key. Re-mints leak credentials. | | `POST /v1/autousers` | Creates an autouser persona. | | `POST /v1/templates` | Creates a template / dimension. | `GET`, `PATCH`, and `DELETE` are already idempotent at the protocol level — `Idempotency-Key` is ignored. ## Key format * Up to 255 characters. * We recommend a UUID v4 generated by your client per **logical request**, not per HTTP attempt. A retry uses the same key. * Scope: per team. Two teams using the same key string see independent records. ```ts theme={null} import { randomUUID } from "node:crypto"; async function createEvaluationOnce(payload: unknown) { const idempotencyKey = randomUUID(); for (let attempt = 0; attempt < 3; attempt++) { try { return await fetch("https://app.autousers.ai/api/v1/evaluations", { method: "POST", headers: { Authorization: `Bearer ${process.env.AUTOUSERS_API_KEY}`, "Content-Type": "application/json", "Idempotency-Key": idempotencyKey, }, body: JSON.stringify(payload), }); } catch (err) { if (attempt === 2) throw err; await new Promise((r) => setTimeout(r, 1000 * 2 ** attempt)); } } } ``` ## Mismatched bodies If you replay a key with a **different body**, we 409: ```json theme={null} { "error": { "message": "Idempotency-Key was previously used with a different request body.", "type": "invalid_request_error", "code": "idempotency_key_reused", "request_id": "0192c7f8-..." } } ``` The check is a sha256 of the canonical request body. The cure is a new key — pick one fresh UUID per logical request. ## TTL Records are kept **24 hours** from first use, then garbage-collected. After that the key is reusable. In practice, generate fresh keys; reuse is for retries within a single workflow. ## Combining with retries `Idempotency-Key` and exponential backoff are complementary. The first makes the server safe to retry; the second decides when. ```ts theme={null} import { randomUUID } from "node:crypto"; async function postIdempotent(path: string, body: unknown) { const key = randomUUID(); for (let attempt = 0; attempt < 5; attempt++) { const res = await fetch(`https://app.autousers.ai${path}`, { method: "POST", headers: { Authorization: `Bearer ${process.env.AUTOUSERS_API_KEY}`, "Content-Type": "application/json", "Idempotency-Key": key, }, body: JSON.stringify(body), }); if (res.ok) return res.json(); if (res.status >= 500 || res.status === 429) { await new Promise((r) => setTimeout(r, 1000 * 2 ** attempt)); continue; } throw new Error(`HTTP ${res.status}`); } } ``` ## Webhooks: idempotency on the receiver Your webhook receiver should be idempotent too. Use the `Autousers-Event-Id` header as the dedup key. See [Retry & replay](/webhooks/retry-and-replay). # Chrome extension Source: https://docs.autousers.ai/integrations/chrome-extension Live-rate the URL in your tab. Smoke tests in seconds. The Autousers Chrome extension lets you launch an evaluation against the URL in the active tab. Useful for product managers checking a staging URL, designers smoke-testing a deploy preview, or sales running a live demo. ## Install [Chrome Web Store → Autousers](https://chrome.google.com/webstore/detail/autousers/) Available for Chromium-based browsers (Chrome, Edge, Brave, Arc, Opera). ## First-run 1. Click the Autousers icon in the toolbar. 2. Click **Sign in**. A browser tab opens at `app.autousers.ai/extension/pair`. 3. Sign in if necessary. Pick a team. 4. The extension captures a session cookie + a short-lived API key bound to the team. The credential is stored in the extension's [`chrome.storage.session`](https://developer.chrome.com/docs/extensions/reference/api/storage) — wiped when the browser closes. Re-pair on next launch. ## Workflow Any `https://` URL works. The extension does not require any markup on the target page. The popup opens. The current tab URL is pre-filled. Same picker as the Figma plugin. Built-ins first, custom personas below. The extension calls `POST /v1/evaluations` with `dryRun: true`, shows the cost, then on confirm re-issues without `dryRun`. The popup shows a live progress bar fed by the `autouser-stream` SSE endpoint. Click **Open in dashboard** for the full experience. ## Live-rate mode A second flow lets the extension open a sidebar on the current page and surface autouser ratings as they arrive: 1. Click **Live-rate** in the popup. 2. The extension injects a sidebar iframe onto the page. 3. Autousers run; ratings appear in the sidebar in real time, anchored to the dimensions on the active template. 4. The sidebar persists across navigation within the same origin until you close it. Useful for screen-share demos and live product reviews. ## Permissions The extension requests: | Permission | Why | | ----------- | -------------------------------------------------------------- | | `activeTab` | Read the URL of the current tab on user click. | | `storage` | Persist the session token between popup opens. | | `scripting` | Inject the live-rate sidebar iframe on user request. | | `cookies` | Read the `app.autousers.ai` session cookie for first-run pair. | It does **not** request broad host permissions. The extension cannot read or modify pages other than the one you click **Live-rate** on. ## Authentication paths The extension uses two paths, depending on flow: * **Popup → API**: short-lived `ak_live_*` key minted at pair time and stored in `chrome.storage.session`. Scopes: `evaluations:read`, `evaluations:write`, `autousers:read`. * **Live-rate sidebar**: same key. Both expire on browser close. There is no long-lived credential. ## Privacy * The extension does not screenshot or scrape the active tab. The autouser run pulls its own copy of the URL via the GKE worker, which navigates the page in a clean browser. * Live-rate sidebar data is loaded over CORS from `app.autousers.ai`; no page DOM is read. * No analytics; no third-party requests. # CLI Source: https://docs.autousers.ai/integrations/cli @autousers/cli — every /v1 route from your terminal, in CI, in a Makefile. The Autousers CLI wraps `/v1` with a typed, scriptable interface. Use it in CI to gate deploys on UX scores, in cron to bootstrap nightly evals, or interactively to explore an evaluation while debugging. ## Install ```bash theme={null} npm i -g @autousers/cli ``` Global install. Self-updates with `autousers self-update`. Source on GitHub at [`autousers-ai/cli`](https://github.com/autousers-ai/cli). ## Auth ```bash theme={null} # Interactive — opens a browser to mint a key. autousers auth login # Or use an existing key. export AUTOUSERS_API_KEY=ak_live_... ``` `auth login` mints the key with sensible defaults (`evaluations:read`, `evaluations:write`, `autousers:read`, `autousers:write`, `ratings:read`). Pass `--scopes ""` to override. The key is stored in the system keychain (macOS Keychain, libsecret on Linux, Windows Credential Locker). Never written to disk in plaintext. ## TUI Run without arguments to drop into a terminal UI: ```bash theme={null} autousers ``` Lists evaluations, drills into runs, shows live progress, follows deliveries. Designed for the local-debugging case where you want to see what the API sees without writing a script. ## Commands A representative subset: ```bash theme={null} # Evaluations autousers eval list autousers eval get $EVAL_ID autousers eval create --template accessibility \ --url https://staging.example.com/checkout \ --autousers first-time-buyer:3,power-user:2 autousers eval delete $EVAL_ID autousers eval results $EVAL_ID autousers eval agreement $EVAL_ID # Autousers autousers autouser list autousers autouser create --from-prompt "A patient managing chronic conditions..." autousers autouser calibrate $AUTOUSER_ID autousers autouser freeze $AUTOUSER_ID # Templates autousers template list autousers template create --from-file ./trust-template.json # Webhooks (when the feature flag flips) autousers webhooks list autousers webhooks create --url https://your.app/webhooks --events evaluation.completed autousers webhooks trigger evaluation.completed --endpoint $ENDPOINT_ID autousers webhooks deliveries --endpoint $ENDPOINT_ID --limit 50 # Misc autousers whoami autousers usage autousers self-update ``` `--help` on any command for the full surface. ## Output formats ```bash theme={null} # Human-readable (default) autousers eval list # JSON for scripting autousers eval list --json # Selective fields with jq autousers eval list --json | jq '.[] | {id, name, status}' ``` ## In CI The `--json` mode + a non-zero exit code on failure makes the CLI a drop-in step in any pipeline: ```yaml theme={null} # .github/workflows/eval-checkout.yml name: Evaluate checkout on: pull_request: paths: ["app/checkout/**"] jobs: eval: runs-on: ubuntu-latest steps: - uses: actions/checkout@v5 - uses: actions/setup-node@v5 with: { node-version: 22 } - run: npm i -g @autousers/cli - name: Smoke eval env: AUTOUSERS_API_KEY: ${{ secrets.AUTOUSERS_API_KEY }} run: | # Create + run + wait + assert α >= 0.6 EVAL_ID=$(autousers eval create \ --template accessibility \ --url ${{ steps.preview.outputs.preview_url }} \ --autousers first-time-buyer:3 \ --json | jq -r .id) autousers eval run $EVAL_ID autousers eval wait $EVAL_ID --timeout 600 ALPHA=$(autousers eval agreement $EVAL_ID --json | jq -r .krippendorff.alpha) [ "$(echo "$ALPHA >= 0.6" | bc)" -eq 1 ] || exit 1 ``` See the [GitHub Actions recipe](/integrations/recipes#github-actions-gate) for a more complete example. ## Profiles Switch between teams without re-logging-in: ```bash theme={null} autousers profile list autousers profile add personal --key ak_live_xxx autousers profile add work --key ak_live_yyy autousers profile use work ``` Every command honours the active profile; pass `--profile ` to override per-command. ## Updates The CLI checks npm for updates on each invocation (cached for 24h) and prints a one-line nudge when a new version is out. ```bash theme={null} autousers self-update # or npm i -g @autousers/cli@latest ``` # Figma plugin Source: https://docs.autousers.ai/integrations/figma-plugin Queue an autouser pass on a frame without leaving Figma. The Autousers Figma plugin lets designers create and run an evaluation directly on a frame, view, or prototype URL — no context switch to the dashboard. ## Install Open the Figma plugin manager and search **Autousers**, or visit [`autousers.ai/figma`](https://autousers.ai/figma). The plugin runs in the Figma sandbox; nothing local to install. ## First-run pairing The first time you open the plugin in a Figma file, it walks you through a one-time pairing: 1. Click **Pair with Autousers** in the plugin UI. 2. Figma opens `app.autousers.ai/figma/pair?code=...`. 3. Sign in if not already signed in; pick the team. 4. Return to Figma; the plugin completes pairing automatically. After pairing, the plugin holds a 7-day Figma JWT in `figma.clientStorage`. It refreshes silently before expiry. The JWT is signed with `AUTOUSERS_JWT_SECRET`, distinct from the OAuth signing key — Figma tokens cannot be replayed against the OAuth surface or vice versa. ## Workflow Pick the design you want evaluated. The plugin captures a 2x PNG of the selected frame. Choose 1–N autousers from the team's roster. Built-ins are listed first; custom personas below. Select a template (rubric). Inline-create custom dimensions if needed. The plugin calls `POST /v1/evaluations` with `dryRun: true` and shows the per-run cost and the total. Click **Run** to commit. The plugin subscribes to the `autouser-stream` SSE endpoint and shows live action labels as the autousers navigate. When the evaluation completes, the plugin offers a **View results** button that opens `app.autousers.ai/evaluations/$ID/results` in your browser. ## Architecture The plugin is a dual-bundle: * **Sandbox** (`code.ts`): runs in the Figma plugin runtime. Reads the current selection, captures the frame as PNG, calls the API. * **iframe UI** (Preact): the visible plugin window. Talks to the sandbox via the standard Figma plugin postMessage bridge. Both bundles call **the same `/v1` API** with the Figma plugin JWT. There is nothing Figma-specific about the API surface. ## Authentication paths The plugin uses **only** the Figma JWT path documented in [Authentication](/authentication#path-4-figma-plugin-jwt). It does not hold an `ak_live_*` key — the JWT is bound to the user, expires after 7 days, and is scoped to all teams the user belongs to. To use the plugin under a different team, switch your active team in the dashboard and reload the plugin. ## What it does NOT do * It does not upload the actual Figma file. Only the captured PNG of the selected frame plus a metadata blob (frame name, file id, page id) is sent. * It does not export designs to anywhere else; ratings and results live in `app.autousers.ai`. * It does not access other plugins' state. ## Forking the plugin Source lives at [`autousers/figma`](https://github.com/autousers/figma). The MIT-licensed bundle is what we ship to the Figma community store. Fork freely; if you publish a fork, drop the "Autousers" branding. See `docs/FIGMA_PLUGIN.md` in the main repo for build instructions. # MCP server Source: https://docs.autousers.ai/integrations/mcp-server The Autousers MCP server exposes 39 tools so you can create evaluations, run autousers, and read results directly from any MCP-aware AI client. The Autousers MCP server lets you run UX evaluations without leaving your AI coding tool. Connect once and you can create evaluations, queue autousers against live URLs, read results, and calibrate personas — all from Claude, Cursor, VS Code, ChatGPT, or any MCP-aware client. The server is hosted at `https://mcp.autousers.ai/mcp`; no package installation is required for most clients. ## Connect Paste `https://mcp.autousers.ai/mcp` into your client's MCP connector UI. OAuth 2.1 launches in the browser on first use — no API key needed for interactive sessions. | Client | How to connect | | --------------------- | ------------------------------------------------------------------ | | **Claude.ai** | Settings → Connectors → Add custom connector → paste URL | | **Claude Desktop** | Settings → Connectors → Add custom connector → paste URL | | **Cursor** | See config below | | **VS Code + Copilot** | See config below | | **ChatGPT** | Settings → Connectors → Developer Mode → Add connector → paste URL | | **Codex CLI** | See config below | **Cursor** — `~/.cursor/mcp.json`: ```json theme={null} { "mcpServers": { "autousers": { "url": "https://mcp.autousers.ai/mcp" } } } ``` **VS Code + GitHub Copilot** — `.vscode/mcp.json` (workspace) or `~/.config/Code/User/mcp.json` (global): ```json theme={null} { "servers": { "autousers": { "type": "http", "url": "https://mcp.autousers.ai/mcp" } } } ``` **Claude Code CLI**: ```bash theme={null} claude mcp add --transport http autousers https://mcp.autousers.ai/mcp ``` **Codex CLI**: ```bash theme={null} codex mcp add autousers --url https://mcp.autousers.ai/mcp ``` For environments without a browser — CI pipelines, scripted workflows — run the server as a local stdio process using your API key. ```bash theme={null} npx -y @autousers/mcp ``` Pass your key as an environment variable in your client config: ```json theme={null} { "mcpServers": { "autousers": { "command": "npx", "args": ["-y", "@autousers/mcp"], "env": { "AUTOUSERS_API_KEY": "ak_live_..." } } } } ``` Or pass it as a Bearer header when using the direct URL with Claude Code: ```bash theme={null} claude mcp add --transport http autousers https://mcp.autousers.ai/mcp \ --header "Authorization: Bearer $AUTOUSERS_API_KEY" ``` Clients that only support stdio — Cline, Zed, Continue, Goose — can connect via the [mcp-remote](https://github.com/geelen/mcp-remote) shim. **Generic stdio config** (Cline and similar): ```json theme={null} { "mcpServers": { "autousers": { "command": "npx", "args": ["-y", "mcp-remote@>=0.1.16", "https://mcp.autousers.ai/mcp"] } } } ``` **Zed** — `~/.config/zed/settings.json`: ```json theme={null} { "context_servers": { "autousers": { "command": { "path": "npx", "args": ["-y", "mcp-remote@>=0.1.16", "https://mcp.autousers.ai/mcp"] } } } } ``` **Continue** — `~/.continue/config.yaml`: ```yaml theme={null} mcpServers: - name: autousers command: npx args: - -y - "mcp-remote@>=0.1.16" - https://mcp.autousers.ai/mcp ``` ## Authentication OAuth is recommended for interactive sessions (Claude.ai, Claude Desktop, Cursor, VS Code, ChatGPT). API keys are the right choice for CI pipelines and headless scripts. ### OAuth 2.1 When you add the server URL to a supported client, an OAuth 2.1 PKCE flow launches in the browser. After you approve, the client receives a short-lived access token (\~15 minutes) that rotates automatically via a refresh token. No static credentials are stored on the client. ### API keys API keys are long-lived bearer tokens for headless environments. Pass them as `Authorization: Bearer ak_live_...` — either as an HTTP header in direct-URL clients or as the `AUTOUSERS_API_KEY` environment variable for stdio. Mint keys at [app.autousers.ai/settings/api-keys](https://app.autousers.ai/settings/api-keys). Keys are shown **once** at creation — store them in a secrets manager immediately. ### Scopes | Scope | Grants | | ------------------- | --------------------------------------------------- | | `templates:read` | List and fetch templates | | `templates:write` | Create, update, delete, duplicate templates | | `evaluations:read` | List, fetch, export, view results | | `evaluations:write` | Create, update, delete, share evaluations | | `autousers:read` | List autousers, fetch runs and telemetry | | `autousers:write` | Create, update, delete, run, calibrate autousers | | `ratings:read` | List ratings for an evaluation | | `webhooks:read` | List endpoints, deliveries, and inspect signatures | | `webhooks:write` | Create, rotate, retry, and delete webhook endpoints | | `events:read` | List the event log feeding webhook deliveries | ## Tools The server exposes 39 tools across four categories. Read-only tools carry `readOnlyHint=true` and are safe to call without side effects. ### Templates (6) | Tool | Description | | --------------------- | ---------------------------------------------- | | `templates_list` | List question templates available to your team | | `templates_get` | Fetch a single template by ID | | `templates_create` | Create a new team-scoped template | | `templates_update` | Patch a template (only supplied fields change) | | `templates_delete` | Hard-delete a template | | `templates_duplicate` | Deep-clone a template into a destination team | ### Evaluations (14) | Tool | Description | | ----------------------------- | --------------------------------------------------------------- | | `evaluations_list` | List evaluations the caller can see | | `evaluations_get` | Fetch one evaluation including config and comparisons | | `evaluations_create` | Create an SSE or SxS evaluation; optionally queue autouser runs | | `evaluations_update` | Patch fields on an evaluation | | `evaluations_delete` | Delete an evaluation and its dependent rows | | `evaluations_save_draft` | Merge wizard fields into a draft evaluation | | `evaluations_ratings_list` | List human and autouser ratings | | `evaluations_results_get` | Aggregate stats and per-rater summaries | | `evaluations_agreement_get` | Pairwise Cohen's Kappa inter-rater agreement | | `evaluations_ai_insights_get` | AI-authored summary, key findings, and recommendations | | `evaluations_export_get` | Download results as JSON or CSV | | `evaluations_share_create` | Grant a per-user VIEWER / EDITOR / OWNER share | | `evaluations_shares_list` | List explicit per-user shares | | `evaluations_transfer` | Transfer evaluation ownership to another user | ### Autousers (15) | Tool | Description | | ---------------------------------- | ------------------------------------------------- | | `autousers_list` | List autousers (built-in + custom) | | `autousers_get` | Fetch a single autouser by ID | | `autousers_create` | Create a team-scoped custom autouser | | `autousers_update` | Patch a custom autouser | | `autousers_delete` | Soft-delete a custom autouser | | `autousers_duplicate` | Deep-clone an autouser into a team | | `autousers_run` | Queue autouser runs against an evaluation | | `autousers_run_stop` | Cancel pending or running autouser runs | | `autouser_status_get` | Run statuses and summary counts for an evaluation | | `autouser_run_get` | Fetch one autouser run with full context | | `autouser_run_turns_list` | Per-turn token and cost telemetry for a run | | `autousers_calibration_start` | Compute Cohen's Kappa vs human ratings | | `autousers_calibration_status_get` | Get calibration status | | `autousers_calibration_freeze` | Freeze a rubric version and set it as active | | `autousers_calibration_optimize` | Send disagreements to AI for rubric suggestions | ### Settings (4) | Tool | Description | | ------------------- | ----------------------------------------------------- | | `teams_list` | List every team the caller belongs to | | `teams_get` | Fetch one team's detail | | `team_members_list` | List team members with roles and profiles | | `usage_get` | Usage rollup: free-run pool remaining and token spend | ## Prompts Five canned workflows are registered as MCP prompts. Invoke them from your client's slash menu — the server pre-fetches relevant data and guides the AI through each workflow end-to-end. | Prompt | Description | | ---------------------- | ---------------------------------------------------------- | | `evaluate-url` | Create an SSE evaluation against a URL and surface results | | `compare-designs` | Create an SxS evaluation between two URLs | | `analyze-results` | Summarise evaluation results with key findings | | `calibrate-autouser` | Run calibration and freeze the rubric when stable | | `triage-low-agreement` | Surface autouser/human disagreements and suggest fixes | ## Resources Three readable resource URIs let your AI client fetch structured data directly: | URI | Description | | ----------------------------- | ---------------------------------------- | | `autousers://evaluation/{id}` | Full evaluation object including results | | `autousers://template/{id}` | Template definition with all dimensions | | `autousers://autouser/{id}` | Autouser persona with rubric | ## Configuration | Variable | Default | Description | | -------------------- | -------------------------- | ---------------------------------------------------- | | `AUTOUSERS_API_KEY` | — | Bearer token for headless/stdio auth (`ak_live_...`) | | `AUTOUSERS_BASE_URL` | `https://app.autousers.ai` | Override the API host (for self-hosted or local dev) | ## Diagnostics Run the bundled doctor command to verify your configuration: ```bash theme={null} npx -y -p @autousers/mcp autousers-mcp-doctor ``` It checks that `AUTOUSERS_API_KEY` is set (for stdio auth), that the API host is reachable, and that at least one tool call succeeds. The command exits non-zero on any misconfiguration, so it integrates cleanly into CI pre-flight checks. # Integrations Source: https://docs.autousers.ai/integrations/overview Six first-party clients, all built on the same /v1 surface. The same REST API powers every Autousers surface. Pick the integration closest to where the work happens. Mount Autousers tools into Claude Desktop, Claude.ai, Cursor, or any MCP-aware agent. `autousers eval create --template=accessibility` in CI, in a Makefile, in a Slack bot. Queue an autouser pass on a frame without leaving Figma. Live-rate any URL in your tab. Useful for quick smoke tests. Direct HTTP. Available today, no client needed. **Typed SDKs are on the roadmap.** A TypeScript SDK (`@autousers/sdk`) and a Python equivalent are planned — generated from the same [OpenAPI spec](https://app.autousers.ai/api/v1/openapi.json) the REST API reference is built from. Until they ship, the cleanest paths are the CLI (for CI / scripts) or hand-rolled `fetch` against the documented routes (for application code). ## What lives where | Surface | Auth | Best for | Distribution | | ---------------- | ---------------- | ----------------------------------------------- | ------------------------- | | MCP server | API key or OAuth | Agentic workflows, conversational eval creation | npm + Smithery + uvx | | CLI | API key | CI/CD, batch scripts, quick local commands | npm `@autousers/cli` | | Figma plugin | Plugin JWT | Designer-led evaluations, quick frame review | Figma Community | | Chrome extension | API key + cookie | Live-rate any URL, smoke tests, demos | Chrome Web Store | | REST API | API key or OAuth | Anything else — direct HTTP, any language | `app.autousers.ai/api/v1` | ## Why so many surfaces UX research lives in a lot of places. The product team uses Figma. The research team uses notebooks. The platform team wants `npx`. The agentic crowd wants MCP. We meet each of them where they work; one `/v1` underneath. ## See also Slack notifications, GitHub Actions gates, Linear tickets, Looker sync. Every route, every field, with try-it-now. # Recipes Source: https://docs.autousers.ai/integrations/recipes Slack notifications, GitHub Actions gates, Linear tickets, Looker sync. Four worked examples that customers actually run in production. ## 1. Slack notification on `evaluation.completed` A Vercel function receiver that posts to Slack when an eval finishes. ```ts theme={null} // app/api/webhooks/autousers/route.ts import crypto from "node:crypto"; export const runtime = "nodejs"; const SECRET = process.env.AUTOUSERS_WEBHOOK_SECRET!; const SLACK = process.env.SLACK_WEBHOOK_URL!; function verify(rawBody: string, header: string | null) { if (!header) return false; const parts = Object.fromEntries( header.split(",").map((p) => p.split("=") as [string, string]) ); const t = Number(parts.t); if (!t || Math.abs(Date.now() / 1000 - t) > 300) return false; const expected = crypto .createHmac("sha256", SECRET) .update(`${t}.${rawBody}`) .digest("hex"); const a = Buffer.from(parts.v1, "hex"); const b = Buffer.from(expected, "hex"); return a.length === b.length && crypto.timingSafeEqual(a, b); } export async function POST(req: Request) { const raw = await req.text(); if (!verify(raw, req.headers.get("autousers-signature"))) { return new Response("invalid signature", { status: 400 }); } const event = JSON.parse(raw); if (event.type !== "evaluation.completed") { return new Response(null, { status: 204 }); } const e = event.data.object; const alpha = e.summary?.krippendorffAlpha?.toFixed(2) ?? "n/a"; const text = [ `:white_check_mark: *${e.name}* finished`, `Krippendorff α: *${alpha}* (${e.summary.ratingCount} ratings)`, ``, ].join("\n"); await fetch(SLACK, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ text }), }); return new Response(null, { status: 200 }); } ``` Register the endpoint once: ```bash theme={null} curl -X POST https://app.autousers.ai/api/v1/webhooks \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://your.app/api/webhooks/autousers", "enabled_events": ["evaluation.completed"] }' ``` ## 2. GitHub Actions gate on Krippendorff α Block the merge when the agreement on the staging deploy drops below a threshold. ```yaml theme={null} # .github/workflows/eval-gate.yml name: UX gate on: pull_request: paths: ["app/**"] jobs: ux-gate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v5 - uses: actions/setup-node@v5 with: { node-version: 22 } - run: npm i -g @autousers/cli - name: Smoke eval against preview deploy env: AUTOUSERS_API_KEY: ${{ secrets.AUTOUSERS_API_KEY }} PREVIEW_URL: ${{ steps.preview.outputs.preview_url }} run: | set -euo pipefail EVAL_ID=$(autousers eval create \ --template accessibility \ --url "$PREVIEW_URL" \ --autousers first-time-buyer:3,accessibility-first:2 \ --json | jq -r .id) autousers eval run "$EVAL_ID" autousers eval wait "$EVAL_ID" --timeout 900 ALPHA=$(autousers eval agreement "$EVAL_ID" --json | jq -r .krippendorff.alpha) echo "alpha=$ALPHA" THRESHOLD=0.6 if [ "$(echo "$ALPHA < $THRESHOLD" | bc)" -eq 1 ]; then echo "::error ::UX α=$ALPHA below threshold $THRESHOLD" echo "Open https://app.autousers.ai/evaluations/$EVAL_ID/results" exit 1 fi ``` The CLI's `eval wait` blocks on the underlying `evaluation.completed` webhook (or polls if no webhook receiver is configured for the team). You don't need to register a webhook to use this recipe — but it finishes faster if you do. ## 3. Linear ticket on `autouser_run.failed` Auto-create a Linear ticket when a run errors. Useful for catching flaky stimuli, expired auth on staging, or outright worker bugs. ```ts theme={null} // app/api/webhooks/autousers/route.ts (continued from recipe 1) if (event.type === "autouser_run.failed") { const r = event.data.object; await fetch("https://api.linear.app/graphql", { method: "POST", headers: { "Content-Type": "application/json", Authorization: process.env.LINEAR_API_KEY!, }, body: JSON.stringify({ query: ` mutation IssueCreate($input: IssueCreateInput!) { issueCreate(input: $input) { success issue { id identifier url } } } `, variables: { input: { teamId: process.env.LINEAR_TEAM_ID!, title: `Autouser run failed: ${r.id}`, description: [ `**Evaluation:** ${r.evaluationId}`, `**Persona:** ${r.autouserId}`, `**Error:** ${r.error}`, `**Cost burned:** $${r.estimatedCostUsd}`, ``, `https://app.autousers.ai/evaluations/${r.evaluationId}/autouser-runs/${r.id}`, ].join("\n"), priority: 2, labelIds: [process.env.LINEAR_FAILURE_LABEL_ID!], }, }, }), }); } ``` Subscribe with `enabled_events: ["autouser_run.failed"]` (or `["*"]` if you have a single fan-out receiver). ## 4. Looker / BigQuery sync on `rating.created` Stream every rating into a warehouse table for trend analysis. The schema mirrors the [Rating shape](/concepts/ratings-and-agreement). ```ts theme={null} // app/api/webhooks/autousers/route.ts (continued) import { BigQuery } from "@google-cloud/bigquery"; const bq = new BigQuery(); if (event.type === "rating.created") { const r = event.data.object; await bq .dataset(process.env.BQ_DATASET!) .table("autousers_ratings") .insert( [ { insert_id: event.id, // BigQuery dedup on this rating_id: r.id, evaluation_id: r.evaluationId, comparison_id: r.comparisonId, rater_type: r.raterType, autouser_id: r.autouserId, autouser_run_id: r.autouserRunId, user_id: r.userId, public_rater_id: r.publicRaterId, rubric_version: r.rubricVersion, dimension_ratings: JSON.stringify(r.dimensionRatings), factors: r.factors ? JSON.stringify(r.factors) : null, justification: r.justification, time_spent_seconds: r.timeSpentSeconds, created_at: r.createdAt, }, ], { ignoreUnknownValues: false } ); } ``` Schema: ```sql theme={null} CREATE TABLE `your_project.autousers.autousers_ratings` ( rating_id STRING, evaluation_id STRING, comparison_id STRING, rater_type STRING, autouser_id STRING, autouser_run_id STRING, user_id STRING, public_rater_id STRING, rubric_version STRING, dimension_ratings JSON, factors JSON, justification STRING, time_spent_seconds INT64, created_at TIMESTAMP ) PARTITION BY DATE(created_at) CLUSTER BY evaluation_id; ``` Use BigQuery's `insertId` deduplication or a manual MERGE keyed on `rating_id` to handle webhook retries. For Looker, point a view at the partitioned table; α and dimension distributions update in near-real time as deliveries land. # Introduction Source: https://docs.autousers.ai/introduction REST API, webhooks, and SDKs for AI-driven UX evaluation. The Autousers API lets you create evaluations, queue AI personas ("autousers") to rate designs, collect human ratings, compute inter-rater agreement, and stream events into your own systems. This is the **developer reference**. For end-user product help (videos, walkthroughs, conceptual onboarding) see the [help center](https://autousers.ai/help). ## What you can build Block a Vercel deploy or a PR merge if a UX evaluation drops below a Krippendorff α threshold. Queue an autouser pass from inside Figma or a PR comment. Get structured ratings back in minutes, not days. Stream every rating into Looker, BigQuery, or Snowflake via webhooks and watch trends across releases. Mount the Autousers MCP server into Claude or Cursor; let the agent create and analyse evaluations. ## API surfaces The same `/v1` surface powers four first-party clients. Each is a thin wrapper over this REST API. | Surface | Auth | Use it for | | ---------------- | ------------------------ | ------------------------------------------------- | | Web app | Supabase session cookie | The dashboard at `app.autousers.ai`. | | `ak_live_*` keys | Bearer API key | Server-to-server, CI/CD, the CLI, the MCP server. | | OAuth 2.1 | Audience-bound HS256 JWT | Third-party MCP apps (Claude Desktop, Cursor). | | Figma plugin JWT | HS256 JWT (7-day expiry) | The `Autousers` Figma plugin. | See [Authentication](/authentication) for full details on each path. ## Conventions The API is **Stripe-style**: * JSON request and response bodies, `application/json`. * Cursor pagination via `starting_after` / `ending_before` (see [Pagination](/pagination)). * Errors share a single envelope with a stable `error.type` taxonomy (see [Errors](/errors)). * Every response carries an `X-Request-Id` header — quote it in support tickets. ```bash theme={null} curl https://app.autousers.ai/api/v1/auth/whoami \ -H "Authorization: Bearer ak_live_..." ``` ## Where to start Mint a key, list evaluations, dry-run a new one. Five minutes. The four auth paths, the 13-scope vocabulary, key rotation. Stop polling. Verify signatures. Retry semantics. MCP server, CLI, Figma plugin, Chrome extension, SDKs. ## Stability The `/v1` surface is stable. We will not ship a breaking change inside `/v1`. Once a route, request shape, or response shape is published, it stays. See [Versioning](/versioning) for the deprecation lifecycle. ## Talk to us * Bugs and feature requests: [github.com/autousers-ai](https://github.com/autousers-ai) * Security: `security@autousers.ai` * Sales: `sales@autousers.ai` # Pagination Source: https://docs.autousers.ai/pagination Stripe-style cursor pagination — stable across writes, no offsets, no skipped rows. List endpoints return at most 100 items per page, ordered most-recent first. The response envelope is uniform: ```json theme={null} { "data": [ /* up to `limit` items */ ], "has_more": true, "next_cursor": "eval_clxq3..." } ``` `total_count` is **not** included by default — it is only set when it is cheap to compute (notably absent on `/v1/evaluations`). If you need a count, page through to exhaustion. ## Query parameters | Param | Type | Default | Notes | | ---------------- | ------- | ------- | --------------------------------------------- | | `limit` | integer | 20 | 1–100. Larger pages reduce RPM consumption. | | `starting_after` | string | — | Cursor — fetch items strictly after this id. | | `ending_before` | string | — | Cursor — fetch items strictly before this id. | `starting_after` and `ending_before` are mutually exclusive. The cursor value is always an item id from a prior page; do not synthesise it. ## Forward pagination ```bash theme={null} # Page 1 curl "https://app.autousers.ai/api/v1/evaluations?limit=50" \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` ```json theme={null} { "data": [ { "id": "eval_05", "...": "..." }, "...", { "id": "eval_56", "...": "..." } ], "has_more": true, "next_cursor": "eval_56" } ``` ```bash theme={null} # Page 2 — pass the previous next_cursor curl "https://app.autousers.ai/api/v1/evaluations?limit=50&starting_after=eval_56" \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` Loop until `has_more === false`. ## Walking a list to exhaustion ```ts TypeScript theme={null} async function listAllEvaluations() { const out: Evaluation[] = []; let cursor: string | undefined; while (true) { const url = new URL("https://app.autousers.ai/api/v1/evaluations"); url.searchParams.set("limit", "100"); if (cursor) url.searchParams.set("starting_after", cursor); const res = await fetch(url, { headers: { Authorization: `Bearer ${process.env.AUTOUSERS_API_KEY}` }, }); const page = await res.json(); out.push(...page.data); if (!page.has_more) break; cursor = page.next_cursor; } return out; } ``` ```python Python theme={null} import os, requests def list_all_evaluations(): out, cursor = [], None while True: params = {"limit": 100} if cursor: params["starting_after"] = cursor r = requests.get( "https://app.autousers.ai/api/v1/evaluations", headers={"Authorization": f"Bearer {os.environ['AUTOUSERS_API_KEY']}"}, params=params, timeout=30, ) r.raise_for_status() page = r.json() out.extend(page["data"]) if not page["has_more"]: return out cursor = page["next_cursor"] ``` ## Why cursors, not offsets Cursor pagination is **stable across writes**. With offset pagination, inserting a new evaluation while you page mid-list shifts every subsequent item by one and silently skips a row. Cursors anchor on a specific id and are immune. ## Ordering Default order is `updatedAt DESC` for the first page, then `id DESC` once a cursor is supplied. Order is **not configurable**. If you need a different ordering (oldest first, by name), fetch the page and sort client-side. ## Edge cases An empty page (`data: []`, `has_more: false`) is a valid terminal state — do not retry. It just means there is nothing on this team matching your filter. Cursors are **not portable across endpoints**. A cursor from `/v1/evaluations` cannot be used on `/v1/autousers`. A cursor referring to a deleted item still works — it's a sort anchor, not a row read. The page after the deleted item returns as if it were still there. # Quickstart Source: https://docs.autousers.ai/quickstart Mint a key, make your first call, dry-run an evaluation. Five minutes. By the end of this page you will have: 1. An `ak_live_*` API key. 2. Confirmed authentication with `/auth/whoami`. 3. Listed your team's evaluations. 4. Dry-run a new evaluation (no spend, no DB write). 5. Created a real evaluation. ## 1. Mint an API key Go to **Settings → API keys** in the dashboard and click **Create key**. Create API key dialog Pick the scopes you need. For this quickstart, select `evaluations:read`, `evaluations:write`, and `autousers:read`. See [Authentication](/authentication#scopes) for the full vocabulary. The plaintext key is shown **once**. Store it as `AUTOUSERS_API_KEY` in your secret manager. We store only its sha256 hash; if you lose it, mint a new one. Keys begin with `ak_live_`. Treat them like passwords — never check them into source, never paste them into chat. They inherit the team membership of whoever minted them. ```bash theme={null} export AUTOUSERS_API_KEY=ak_live_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ``` ## 2. Confirm authentication ```bash theme={null} curl https://app.autousers.ai/api/v1/auth/whoami \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` Response: ```json theme={null} { "userId": "usr_clxq3...", "teamId": "team_clxq3...", "source": "apikey", "scopes": ["evaluations:read", "evaluations:write", "autousers:read"] } ``` If you see `{"error":{"type":"authentication_error",...}}`, your key is wrong or revoked. Mint a fresh one. ## 3. List evaluations ```bash theme={null} curl "https://app.autousers.ai/api/v1/evaluations?limit=5" \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` The response is a Stripe-style paginated list: ```json theme={null} { "data": [ { "id": "eval_clxq3...", "name": "Checkout v2 vs v1", "type": "SxS", "status": "Ended", "...": "..." } ], "has_more": false } ``` Pass `starting_after=` to fetch the next page. See [Pagination](/pagination). ## 4. Dry-run a new evaluation Every billable POST that runs autousers supports `dryRun: true`. The server validates the payload, returns a cost estimate, and **writes nothing** to the database. Use this in CI to fail fast on a malformed config before spending tokens. ```bash theme={null} curl https://app.autousers.ai/api/v1/evaluations \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Smoke — checkout v2", "type": "SSE", "status": "Draft", "designUrls": [ { "url": "https://staging.example.com/checkout", "label": "v2", "stimulusType": "URL" } ], "selectedAutousers": [ { "autouserId": "auto_first_time_buyer", "agentCount": 3 } ], "selectedDimensionIds": ["overall", "trust", "clarity"], "evaluationMethod": "ai", "dryRun": true }' ``` Response: ```json theme={null} { "dryRun": true, "persisted": false, "autousersQueued": false, "wouldRun": { "autouserCount": 3, "comparisonCount": 1, "totalRuns": 3 }, "costEstimate": { "usd": 0.273, "tokens": { "input": 18000, "output": 4200 } }, "warnings": [], "note": "PREVIEW ONLY — this evaluation has NOT been created. Re-issue without dryRun:true to persist." } ``` The dry-run echoes the validated payload as `wouldCreate` so the caller can confirm exactly what the live POST would persist. ## 5. Create the real evaluation Drop `dryRun: true` and re-issue the same request: ```bash theme={null} curl https://app.autousers.ai/api/v1/evaluations \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "...": "...same payload, no dryRun..." }' ``` Response (status `201`): ```json theme={null} { "id": "eval_clxq3...", "name": "Smoke — checkout v2", "status": "Draft", "links": { "preview": "https://app.autousers.ai/evaluations/eval_clxq3.../preview", "review": "https://app.autousers.ai/evaluations/eval_clxq3.../review", "results": "https://app.autousers.ai/evaluations/eval_clxq3.../results" } } ``` To actually run the autousers, `POST` to `/v1/evaluations/{id}/run-autousers`. See the [Evaluations concept guide](/concepts/evaluations). ## Next steps Get notified when an autouser run completes. `autousers eval create --template=accessibility` in GitHub Actions. # Rate limits & quotas Source: https://docs.autousers.ai/rate-limits Per-tier RPM, monthly quotas, and the headers you need to respect. The Autousers API enforces two limits independently: 1. **RPM** — a sliding-window cap on requests per minute, per team. 2. **Monthly quotas** — counts of autouser ratings, human ratings, and evaluations created per team per billing period. Both are tied to your subscription plan. **Beta status (2026-05-04):** tier-aware RPM and monthly quota enforcement is **rolling out**. During beta, the server logs every over-limit request with its `X-Request-Id` but does not yet block. The `X-RateLimit-*` headers are populated on a best-effort basis. Watch the [Changelog](/changelog) for the flip-the-switch announcement before you rely on these limits in production. Build to the contract below now; nothing changes when enforcement lands. ## Per-plan limits These mirror the [pricing page](https://autousers.ai/pricing) exactly. | Plan | RPM | Autouser ratings / mo | Human ratings / mo | Evaluations / mo | | ---------- | ------ | --------------------- | ------------------ | ---------------- | | Free | 60 | 50 | 250 | 5 | | Team | 120 | 500 | 5,000 | 100 | | Pro | 600 | 2,000 | 20,000 | unlimited | | Enterprise | custom | custom | custom | unlimited | Limits apply to the **team** the API key is bound to, not the user. Multiple keys on one team share the same budget. ## Response headers Every `/v1` response carries: | Header | Meaning | | ----------------------- | ------------------------------------------------- | | `X-RateLimit-Limit` | The team's per-minute limit (e.g. `120`). | | `X-RateLimit-Remaining` | Requests remaining in the current sliding window. | | `X-RateLimit-Reset` | Unix epoch seconds when the window resets. | | `Retry-After` | Set on 429 only. Seconds to wait before retrying. | Read these on **every** response — not just 429s. Throttling proactively is cheaper than retrying. ## The 429 response ```http theme={null} HTTP/1.1 429 Too Many Requests Content-Type: application/json Retry-After: 12 X-RateLimit-Limit: 120 X-RateLimit-Remaining: 0 X-RateLimit-Reset: 1714867260 X-Request-Id: 0192c7f8-... ``` ```json theme={null} { "error": { "message": "Rate limit exceeded — 120 requests per minute on plan team. Retry in 12s.", "type": "rate_limit_error", "code": "rate_limit_exceeded", "request_id": "0192c7f8-..." } } ``` Monthly-quota exhaustion uses `code: quota_exceeded` and **does not** carry `Retry-After` — the quota resets at the start of the next billing period (visible at `GET /v1/usage`). ## Self-throttling Call `GET /v1/usage` to read the team's plan and remaining budget without firing a real workload request: ```bash theme={null} curl https://app.autousers.ai/api/v1/usage \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` ```json theme={null} { "plan": "team", "period": { "start": "2026-05-01T00:00:00Z", "end": "2026-06-01T00:00:00Z" }, "limits": { "rpm": 120, "autouser_ratings_mo": 500, "human_ratings_mo": 5000, "evals_mo": 100 }, "used": { "autouser_ratings_mo": 312, "human_ratings_mo": 1840, "evals_mo": 19 }, "remaining": { "autouser_ratings_mo": 188, "human_ratings_mo": 3160, "evals_mo": 81 } } ``` ## Exponential backoff with jitter Treat 429 and 5xx as retryable. Cap at 5 attempts; add jitter so retries don't synchronise across replicas. ```ts theme={null} async function fetchWithBackoff(input: RequestInfo, init?: RequestInit) { const max = 5; for (let attempt = 0; attempt < max; attempt++) { const res = await fetch(input, init); if (res.ok) return res; const retryable = res.status === 429 || res.status >= 500; if (!retryable || attempt === max - 1) { throw new Error( `Autousers API ${res.status} — request_id=${res.headers.get("x-request-id")}` ); } const serverHint = Number(res.headers.get("retry-after") ?? 0); const exp = Math.min(60_000, 1000 * 2 ** attempt); const jitter = Math.random() * 250; const wait = serverHint > 0 ? serverHint * 1000 : exp + jitter; await new Promise((r) => setTimeout(r, wait)); } throw new Error("unreachable"); } ``` ## Limits on specific routes A handful of routes have **stricter** per-user limits independent of plan tier: | Route | Limit | Why | | ------------------------------ | ----------------- | --------------------------------- | | `POST /v1/settings/byok/test` | 5 / minute / user | Probing a third-party Vertex key. | | `POST /v1/settings/byok/probe` | 5 / minute / user | Same. | | `POST /v1/auth/figma/*` | 10 / minute / IP | Unauthenticated pairing surface. | These are documented per-route in the [API reference](/api-reference). ## What does not count * `GET /v1/auth/whoami` — free, unmetered, ping anytime. * `GET /v1/usage` — free, unmetered. * Webhook **deliveries** (us → you). Your endpoint receiving a webhook is not a request you made; it doesn't consume RPM. ## Enterprise Custom RPM, unmetered monthly quotas, and dedicated capacity are available on Enterprise. Contact `sales@autousers.ai`. # Versioning Source: https://docs.autousers.ai/versioning What's stable, what's safe to add, and how we sunset what's not. The `/v1` surface is **stable**. We will not ship a breaking change inside `/v1`. Once a route, request shape, or response shape is published, it stays — that's the whole point of a `v1`. ## What counts as breaking These changes require a new version namespace. They will never ship inside `/v1`: * Removing or renaming a field in a response. * Removing or renaming an endpoint. * Changing the type of a field (e.g. string → number). * Adding a required request field. * Tightening validation (e.g. a previously-accepted value now rejected). * Changing the meaning of a value (e.g. `"Draft"` → `"draft"`). ## What's safe to ship without a version bump * Adding a new endpoint. * Adding a new optional field to a request. * Adding a new field to a response. * Adding new enum values (where consumers were already prepared to handle unknown values — we document this per-route). * Loosening validation. * Performance improvements. * New `error.code` values inside an existing `error.type`. Build your client tolerantly — ignore unknown response fields, treat unknown enum values as a degraded-but-safe default. ## When breaking changes happen A new versioned namespace ships (e.g. `/v2/foo`). The old `/v1/foo` stays alive **at least 12 months** with the deprecation headers below. Existing consumers get the full 12 months to migrate. ```http theme={null} Deprecation: true Sunset: Sun, 04 Apr 2027 00:00:00 GMT Link: ; rel="successor-version" ``` Watch for these headers in production responses; surface them in your own logs so a quietly-introduced sunset doesn't become a Friday emergency. ## Aliases When we rename an internal noun to a product noun, the old route stays working as an alias for the full sunset window. Today the canonical example is **templates** vs **dimensions**: | Canonical | Alias (deprecated) | Status | | ----------------- | ------------------ | ------------------------------------------------------------ | | `/v1/templates/*` | `/v1/dimensions/*` | `Deprecation: true`, `Sunset: Sun, 04 Apr 2027 00:00:00 GMT` | Both keep working through 04 Apr 2027. After that the alias 410s. The [reference](/api-reference) only documents `/templates`. ## Header-pinned versions (future) If we ever need version-specific behaviour on the same URL — rare — clients will pin via: ```http theme={null} Autousers-Version: 2026-05-04 ``` The default is the current version. Old dated versions stay reachable for at least 12 months from the day they are superseded. **Not yet implemented.** When it ships, we will add a per-route version selector and the request/response examples in the [reference](/api-reference) will gain a version dropdown. ## How to track changes | Channel | What lands there | | ------------------------------------ | ---------------------------------------------------- | | [Changelog](/changelog) | Every API-affecting change, dated, in plain English. | | `Deprecation` / `Sunset` headers | Per-route, machine-readable. | | Email (Pro + Enterprise) | 60 / 14 days before sunset, sent to team Owners. | | Status banner on `docs.autousers.ai` | At 90 / 30 / 7 days before sunset. | ## What we will not do * Rename fields silently. Renames ship as **additions** for one version and **removals** in the next; the rename always crosses a version bump. * Repurpose `error.type` values. New conditions get new types. * Change the response shape of a route inside a version, even by adding required fields. Additive optional fields only. ## OpenAPI as source of truth The full machine-readable contract lives at [`/api/v1/openapi.json`](https://app.autousers.ai/api/v1/openapi.json). SDKs, the in-page playground, the CLI, and the MCP server all derive from it. If the spec and this prose disagree, the spec wins; file an issue. # Event reference Source: https://docs.autousers.ai/webhooks/events Every v1 event, when it fires, what it carries. Seven events ship in v1. Each fires exactly once per state transition; none fan out per-rater. | Event | Fires when | Object | | --------------------------- | -------------------------------------------------------------- | ---------------------- | | `evaluation.created` | `POST /v1/evaluations` succeeds (status `Draft` or `Running`). | `Evaluation` | | `evaluation.status_changed` | `Evaluation.status` transitions. | `Evaluation` | | `evaluation.completed` | All `AutouserRun`s + ratings finalize. | `Evaluation` + summary | | `autouser_run.completed` | `AutouserRun.status` flips to `completed`. | `AutouserRun` | | `autouser_run.failed` | `AutouserRun.status` flips to `failed`. | `AutouserRun` + error | | `rating.created` | A new `Rating` row inserts (human OR autouser). | `Rating` | | `calibration.frozen` | An autouser's rubric is frozen. | `Autouser` + rubric | Subscribe to specific types in `enabled_events` on the endpoint. Use `["*"]` to receive everything — recommended only for warehouse-sync integrations. ## Envelope All events share the same outer shape: ```json theme={null} { "id": "event_clxq3...", "type": "evaluation.completed", "api_version": "2026-05-04", "created": 1714867200, "team_id": "team_clxq3...", "data": { "object": { /* the resource */ } }, "request": { "id": "req_01HXY...", "idempotency_key": null } } ``` `request.id` traces back to the API request that triggered the event, useful for correlating customer-side and server-side logs. ## Per-event payloads Fires after `POST /v1/evaluations` returns 201. Payload mirrors the HTTP response. ```json theme={null} { "id": "event_clxq3created", "type": "evaluation.created", "api_version": "2026-05-04", "created": 1714867200, "team_id": "team_clxq3...", "data": { "object": { "id": "eval_clxq3...", "teamId": "team_clxq3...", "name": "Checkout v2 vs v1", "type": "SxS", "status": "Draft", "shareToken": "shr_clxq3...", "shareAccess": "TEAM_ONLY", "allowMultipleRatings": false, "createdAt": "2026-05-04T10:00:00.000Z", "updatedAt": "2026-05-04T10:00:00.000Z", "links": { "preview": "https://app.autousers.ai/evaluations/eval_clxq3.../preview", "review": "https://app.autousers.ai/evaluations/eval_clxq3.../review", "results": "https://app.autousers.ai/evaluations/eval_clxq3.../results" } } }, "request": { "id": "req_01HXY...", "idempotency_key": null } } ``` Fires on every `Evaluation.status` transition. Reading the current status alone is insufficient — listen for this event if you need to react on `Draft → Running` distinct from `Running → Ended`. ```json theme={null} { "id": "event_clxq3statc", "type": "evaluation.status_changed", "api_version": "2026-05-04", "created": 1714867260, "team_id": "team_clxq3...", "data": { "object": { "id": "eval_clxq3...", "status": "Running", "previous_status": "Draft", "...": "...full Evaluation..." } }, "request": { "id": "req_01HXY...", "idempotency_key": null } } ``` Fires once when **all** queued `AutouserRun`s reach a terminal state (`completed` or `failed`) AND any pending human rating windows close. The single most useful event for "wake up when the eval is done" integrations. ```json theme={null} { "id": "event_clxq3compl", "type": "evaluation.completed", "api_version": "2026-05-04", "created": 1714867800, "team_id": "team_clxq3...", "data": { "object": { "id": "eval_clxq3...", "name": "Checkout v2 vs v1", "status": "Ended", "summary": { "ratingCount": 24, "autouserRatingCount": 18, "humanRatingCount": 6, "krippendorffAlpha": 0.74, "completedRunCount": 6, "failedRunCount": 0 }, "...": "...full Evaluation..." } }, "request": { "id": "req_01HXY...", "idempotency_key": null } } ``` Fires per `AutouserRun` when it reaches `status: completed`. If you queued 6 runs (`agentCount: 3 × 2 comparisons`), expect six events. ```json theme={null} { "id": "event_clxq3runOK", "type": "autouser_run.completed", "api_version": "2026-05-04", "created": 1714867440, "team_id": "team_clxq3...", "data": { "object": { "id": "arun_clxq3...", "evaluationId": "eval_clxq3...", "autouserId": "auto_first_time_buyer", "autouserType": "builtin", "status": "completed", "ratingsCreated": 4, "inputTokens": 18204, "outputTokens": 4123, "estimatedCostUsd": 0.0623, "credentialSource": "platform", "artifactsPath": "gs://autousers-assets/runs/arun_clxq3.../", "startedAt": "2026-05-04T10:01:12.000Z", "completedAt": "2026-05-04T10:04:00.000Z", "createdAt": "2026-05-04T10:01:00.000Z" } }, "request": { "id": "req_01HXY...", "idempotency_key": null } } ``` Fires when an `AutouserRun` errors out. A failed run does NOT consume autouser-rating quota, but you may want to alert on it (the design might be unreachable, or the worker might have crashed). ```json theme={null} { "id": "event_clxq3runFAIL", "type": "autouser_run.failed", "api_version": "2026-05-04", "created": 1714867500, "team_id": "team_clxq3...", "data": { "object": { "id": "arun_clxq3...", "evaluationId": "eval_clxq3...", "autouserId": "auto_first_time_buyer", "status": "failed", "error": "Stimulus URL returned 503 after 3 retries.", "ratingsCreated": 0, "inputTokens": 4102, "outputTokens": 0, "estimatedCostUsd": 0.002, "startedAt": "2026-05-04T10:01:12.000Z", "completedAt": "2026-05-04T10:02:30.000Z" } }, "request": { "id": "req_01HXY...", "idempotency_key": null } } ``` Fires per Rating row insert — humans AND autousers. The single firehose event for warehouse-sync use cases. High volume on busy evals; budget your receiver accordingly. ```json theme={null} { "id": "event_clxq3rating", "type": "rating.created", "api_version": "2026-05-04", "created": 1714867268, "team_id": "team_clxq3...", "data": { "object": { "id": "rat_clxq3...", "evaluationId": "eval_clxq3...", "comparisonId": "cmp_clxq3...", "raterType": "autouser", "userId": null, "publicRaterId": null, "autouserId": "auto_first_time_buyer", "autouserRunId": "arun_clxq3...", "rubricVersion": "v3", "dimensionRatings": { "overall": 4, "trust": 3, "clarity": 5 }, "openTextResponses": { "overall": "Felt fast but trust signals thin." }, "factors": null, "justification": "Liked simplicity, missed social proof.", "skipReason": null, "timeSpentSeconds": 42, "createdAt": "2026-05-04T10:21:08.000Z" } }, "request": { "id": "req_01HXY...", "idempotency_key": null } } ``` Fires when an autouser's rubric is locked via `POST /v1/autousers/{id}/calibration/freeze`. Downstream pipelines should listen for this before promoting an autouser to production. ```json theme={null} { "id": "event_clxq3frozen", "type": "calibration.frozen", "api_version": "2026-05-04", "created": 1714868000, "team_id": "team_clxq3...", "data": { "object": { "autouser": { "id": "auto_clxq3...", "name": "Healthcare-portal patient", "calibrationStatus": "frozen", "activeRubricId": "rub_clxq3..." }, "rubric": { "id": "rub_clxq3...", "version": 3, "krippendorffAlpha": 0.81, "frozenAt": "2026-05-04T10:30:00.000Z" } } }, "request": { "id": "req_01HXY...", "idempotency_key": null } } ``` ## Adding new events New event types are additive — new entries on `Autousers-Event-Type`, new payload shapes — and ship without a version bump. Build your receiver to **ignore unknown event types**: ```ts theme={null} const KNOWN_TYPES = new Set([ "evaluation.created", "evaluation.status_changed", "evaluation.completed", "autouser_run.completed", "autouser_run.failed", "rating.created", "calibration.frozen", ]); if (!KNOWN_TYPES.has(req.headers["autousers-event-type"])) { return res.status(204).end(); // ack and drop } ``` # Webhooks overview Source: https://docs.autousers.ai/webhooks/overview Stop polling. Subscribe to events. Verify signatures. Retry deterministically. Webhooks let your system react to Autousers events in near-real time instead of polling. We POST a signed JSON payload to a URL you control; your endpoint returns 2xx; we move on. **Beta status (2026-05-04):** webhooks are flag-gated behind `AUTOUSERS_WEBHOOKS_ENABLED=1`. While the flag is off, every `/v1/webhooks/*` route returns `503 Service Unavailable`. Watch the [Changelog](/changelog) for the public flip. ## When to use webhooks | Use case | Polling | Webhooks | | ----------------------------------------------- | :------: | :------: | | Notify Slack when an evaluation completes | bad | ✓ | | Append every rating to a BigQuery table | terrible | ✓ | | Block a CI deploy on Krippendorff α | OK | ✓ | | Auto-create a Linear ticket on autouser failure | bad | ✓ | | Show live progress in a UI | OK¹ | OK² | ¹ Use `/v1/evaluations/{id}/autouser-stream` (server-sent events) for in-page live progress instead. ² Webhooks fan out to your backend; surface that to the UI via your own WebSocket / SSE. Polling autouser status burns RPM budget — a 50-comparison eval × 4 personas = 200 jobs, each taking minutes. Webhooks fire **once** per state change. ## How it works ``` ┌──────────┐ ┌──────────────────┐ │ Autouser │ state transition │ WebhookEvent │ │ Run ├───────────────────►│ inserted │ └──────────┘ └────────┬─────────┘ │ ▼ ┌──────────────────┐ │ WebhookDelivery │ │ enqueued (per │ ──► your endpoint │ matching │ POST + sig │ endpoint) │ └──────────────────┘ ``` 1. An event happens (e.g. `evaluation.completed`). 2. We insert a `WebhookEvent` row — the immutable record. 3. For every endpoint subscribed to that type, we enqueue a `WebhookDelivery`. 4. A worker POSTs the payload, signed with your endpoint's secret. 5. Your endpoint returns 2xx. We mark `delivered`. Done. 6. On non-2xx or timeout, we retry — see [Retry & replay](/webhooks/retry-and-replay). ## What we POST Every request includes: ```http theme={null} POST /your/path HTTP/1.1 Host: your.app Content-Type: application/json User-Agent: Autousers-Webhooks/1.0 Autousers-Signature: t=1714867200,v1=8e3a... Autousers-Event-Id: event_clxq3... Autousers-Event-Type: evaluation.completed Autousers-Delivery-Id: whd_clxq3... ``` ```json theme={null} { "id": "event_clxq3...", "type": "evaluation.completed", "api_version": "2026-05-04", "created": 1714867200, "team_id": "team_clxq3...", "data": { "object": { "id": "eval_clxq3...", "...": "..." } }, "request": { "id": "req_01HXY...", "idempotency_key": null } } ``` The body is the **fat payload** — full snapshot of the object at emit time. No need for a follow-up GET to fetch detail. ## Setting up an endpoint ```bash theme={null} curl -X POST https://app.autousers.ai/api/v1/webhooks \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://your.app/webhooks/autousers", "description": "prod — slack notifier", "enabled_events": ["evaluation.completed", "autouser_run.failed"] }' ``` The response includes the **plaintext webhook secret once**. Store it in your secret manager; we keep only its sha256 hash. ```json theme={null} { "id": "whe_clxq3...", "url": "https://your.app/webhooks/autousers", "secret": "whsec_3xampLeOnLySh0wnOnceK33pSafe", "enabled_events": ["evaluation.completed", "autouser_run.failed"], "status": "enabled" } ``` ## Tier gating Webhooks are restricted to **Pro and Enterprise** plans. Free and Team callers get `403 plan_upgrade_required` on `/v1/webhooks/*` routes. This matches the [pricing card](https://autousers.ai/pricing) — the "webhooks + REST API" line item. ## Required scopes | Action | Scope | | ----------------------------------------------------- | ---------------- | | List, get endpoints; list deliveries | `webhooks:read` | | Create, update, delete endpoints; rotate secret; test | `webhooks:write` | | Read the underlying event log (`/v1/events`) | `events:read` | ## Next All seven v1 events with example payloads. Production-ready snippets in TypeScript, Python, Go, Ruby. The backoff schedule. The receiver contract. The auto-disable rules. Send synthetic events. Use ngrok locally. # Signature verification Source: https://docs.autousers.ai/webhooks/signature-verification HMAC-SHA256 over `${timestamp}.${raw_body}`. Stripe-style. Five-minute replay window. Every webhook delivery carries a signature header. **Verify it.** Refusing to verify is the single most common way customers ship exploitable webhook receivers; we do not consider an unverified endpoint a working integration. ## The header ```http theme={null} Autousers-Signature: t=1714867200,v1=8e3a4d9c1f... ``` | Field | Meaning | | ----- | --------------------------------------------- | | `t` | Unix epoch seconds at signing time. | | `v1` | Hex-encoded HMAC-SHA256 of `${t}.${rawBody}`. | The signing secret is the `whsec_*` plaintext value you received once when creating the endpoint. Rotate via `POST /v1/webhooks/{id}/rotate-secret` — the new plaintext is shown once; the old secret keeps verifying for 24 hours so deploys can roll. ## The verification rules A payload is valid iff **all four** hold: 1. The signature header parses (one `t`, one `v1`). 2. `Math.abs(Date.now()/1000 - t) <= toleranceSec` (default **300**, i.e. 5 minutes). 3. `HMAC_SHA256(secret, "${t}.${rawBody}").hex() === v1`. 4. Comparison uses a **constant-time** equality check (`crypto.timingSafeEqual` or equivalent). Use the **raw bytes** of the request body. Re-serialising via `JSON.stringify(JSON.parse(body))` will corrupt the signature on any payload with non-canonical key order. ## Reference implementations ```ts TypeScript / Node theme={null} import crypto from "node:crypto"; export function verifyAutousersSignature( rawBody: string | Buffer, header: string | undefined, secret: string, toleranceSec = 300 ): boolean { if (!header) return false; const parts = Object.fromEntries( header.split(",").map((p) => { const [k, ...rest] = p.split("="); return [k.trim(), rest.join("=")]; }) ); const t = Number(parts.t); const sig = parts.v1; if (!t || !sig) return false; if (Math.abs(Date.now() / 1000 - t) > toleranceSec) return false; const payload = typeof rawBody === "string" ? Buffer.from(`${t}.${rawBody}`) : Buffer.concat([Buffer.from(`${t}.`), rawBody]); const expected = crypto .createHmac("sha256", secret) .update(payload) .digest("hex"); // Both buffers must be the same length for timingSafeEqual. const a = Buffer.from(sig, "hex"); const b = Buffer.from(expected, "hex"); if (a.length !== b.length) return false; return crypto.timingSafeEqual(a, b); } // Express handler — note `express.raw()` to keep the bytes intact. import express from "express"; const app = express(); app.post( "/webhooks/autousers", express.raw({ type: "application/json" }), (req, res) => { const ok = verifyAutousersSignature( req.body, // Buffer because of express.raw() req.header("autousers-signature"), process.env.AUTOUSERS_WEBHOOK_SECRET! ); if (!ok) return res.status(400).send("invalid signature"); const event = JSON.parse(req.body.toString("utf8")); // ... handle event ... res.status(200).end(); } ); ``` ```python Python theme={null} import hmac, hashlib, time def verify_autousers_signature( raw_body: bytes, header: str | None, secret: str, tolerance_sec: int = 300, ) -> bool: if not header: return False try: parts = dict(p.strip().split("=", 1) for p in header.split(",")) t = int(parts["t"]) sig = parts["v1"] except (KeyError, ValueError): return False if abs(time.time() - t) > tolerance_sec: return False expected = hmac.new( secret.encode("utf-8"), f"{t}.".encode("utf-8") + raw_body, hashlib.sha256, ).hexdigest() return hmac.compare_digest(sig, expected) # FastAPI handler — request.body() returns the raw bytes. from fastapi import FastAPI, Request, HTTPException import os, json app = FastAPI() @app.post("/webhooks/autousers") async def autousers_webhook(request: Request): body = await request.body() header = request.headers.get("autousers-signature") if not verify_autousers_signature( body, header, os.environ["AUTOUSERS_WEBHOOK_SECRET"] ): raise HTTPException(status_code=400, detail="invalid signature") event = json.loads(body) # ... handle event ... return {"ok": True} ``` ```go Go theme={null} package autousers import ( "crypto/hmac" "crypto/sha256" "encoding/hex" "errors" "strconv" "strings" "time" ) // VerifySignature returns nil iff the header is present, parses, the // timestamp is within tolerance, and the HMAC matches. func VerifySignature( rawBody []byte, header, secret string, tolerance time.Duration, ) error { if header == "" { return errors.New("missing signature header") } parts := map[string]string{} for _, p := range strings.Split(header, ",") { kv := strings.SplitN(strings.TrimSpace(p), "=", 2) if len(kv) == 2 { parts[kv[0]] = kv[1] } } tStr, sig := parts["t"], parts["v1"] if tStr == "" || sig == "" { return errors.New("malformed signature header") } tInt, err := strconv.ParseInt(tStr, 10, 64) if err != nil { return err } skew := time.Since(time.Unix(tInt, 0)) if skew < 0 { skew = -skew } if skew > tolerance { return errors.New("timestamp outside tolerance") } mac := hmac.New(sha256.New, []byte(secret)) mac.Write([]byte(strconv.FormatInt(tInt, 10) + ".")) mac.Write(rawBody) expected := hex.EncodeToString(mac.Sum(nil)) if !hmac.Equal([]byte(sig), []byte(expected)) { return errors.New("signature mismatch") } return nil } // http.Handler example // // import "io" // func handler(w http.ResponseWriter, r *http.Request) { // body, _ := io.ReadAll(r.Body) // if err := VerifySignature( // body, // r.Header.Get("Autousers-Signature"), // os.Getenv("AUTOUSERS_WEBHOOK_SECRET"), // 5*time.Minute, // ); err != nil { // http.Error(w, err.Error(), http.StatusBadRequest) // return // } // // handle event // w.WriteHeader(http.StatusOK) // } ``` ```ruby Ruby theme={null} require "openssl" require "json" module Autousers def self.verify_signature(raw_body, header, secret, tolerance_sec: 300) return false if header.nil? || header.empty? parts = header.split(",").map { |p| p.strip.split("=", 2) }.to_h t = parts["t"]&.to_i sig = parts["v1"] return false unless t && sig return false if (Time.now.to_i - t).abs > tolerance_sec expected = OpenSSL::HMAC.hexdigest( OpenSSL::Digest.new("sha256"), secret, "#{t}.#{raw_body}", ) # constant-time compare Rack::Utils.secure_compare(sig, expected) end end # Sinatra handler — read the raw body before any middleware munges it. post "/webhooks/autousers" do request.body.rewind raw = request.body.read unless Autousers.verify_signature( raw, request.env["HTTP_AUTOUSERS_SIGNATURE"], ENV.fetch("AUTOUSERS_WEBHOOK_SECRET"), ) halt 400, "invalid signature" end event = JSON.parse(raw) # ... handle event ... status 200 end ``` ## Common pitfalls **Don't re-serialise the body.** Frameworks that parse JSON before your handler runs will hand you a Ruby Hash / Python dict / JS object whose `JSON.stringify` representation does **not** match the bytes we signed. Use the raw-body middleware (`express.raw`, `Request.body()`, `request.body.read`). **Don't compare with `===` / `==` / `.equal?`.** A naive equality check leaks signature bytes through timing. Use `timingSafeEqual`, `hmac.compare_digest`, `hmac.Equal`, or `Rack::Utils.secure_compare`. **Don't trust `req.body.timestamp` over the header `t`.** Some CDNs rewrite or strip request bodies. The `t` in the header is what we signed; verify against that, not against the JSON body. ## Secret rotation Rotate when an employee with secret access leaves, after any suspected leak, and at least annually: ```bash theme={null} curl -X POST https://app.autousers.ai/api/v1/webhooks/$ENDPOINT_ID/rotate-secret \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" ``` ```json theme={null} { "id": "whe_clxq3...", "secret": "whsec_n3wRotat3dS3cretShownOnceK33pSafe", "rotatedAt": "2026-05-04T11:00:00.000Z", "previousSecretValidUntil": "2026-05-05T11:00:00.000Z" } ``` The previous secret keeps verifying for **24 hours**, so a rolling deploy that updates secrets across replicas does not drop deliveries. After 24 hours, only the new secret is valid. # Testing webhooks Source: https://docs.autousers.ai/webhooks/testing Send synthetic events. Tunnel localhost. Replay deliveries. Don't wait for a real evaluation to complete before testing your endpoint. Three options, in order of how often you'll use them. ## 1. Send a synthetic event The fastest loop. Fires a real signed POST with a representative payload at the endpoint URL on file. No DB side effects. ```bash theme={null} curl -X POST https://app.autousers.ai/api/v1/webhooks/$ENDPOINT_ID/test \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "type": "evaluation.completed" }' ``` Response: ```json theme={null} { "deliveryId": "whd_test_clxq3...", "status": "pending", "synthetic": true, "scheduledFor": "2026-05-04T10:00:00.000Z" } ``` The synthetic payload uses `event_id` prefix `event_test_*` so your receiver can branch on test vs prod if needed (or treat them identically — recommended). You can request any of the seven event types in the `type` field. The payload mirrors a real event of that type, with `team_id` set to your team and `data.object.id` carrying a `_test_` infix. ## 2. Tunnel localhost For active development, expose your local server with a tunnel so we can reach it. Three good options: ```bash theme={null} # Static-domain free tier: install once, login, tunnel. brew install ngrok ngrok config add-authtoken $YOUR_NGROK_TOKEN ngrok http 3000 ``` ``` Forwarding https://hippo-rare-koi.ngrok-free.app -> http://localhost:3000 ``` Use `https://hippo-rare-koi.ngrok-free.app/webhooks/autousers` as the endpoint URL when creating the webhook. ngrok logs every inbound request at `http://localhost:4040`. ```bash theme={null} brew install cloudflared cloudflared tunnel --url http://localhost:3000 ``` Cloudflare prints an ephemeral `https://*.trycloudflare.com` URL. If you're already on Vercel, push a branch and use the preview URL — it's a real cert, real DNS, free, and survives reboots. ``` https://your-app-git-feature-webhook.vercel.app/api/webhooks/autousers ``` ## 3. Use webhook.site for blind smoke tests When you want to confirm we send what we say we send — no receiver code needed: 1. Open [webhook.site](https://webhook.site) and copy the unique URL. 2. Create an endpoint pointing at it. 3. Fire `POST /v1/webhooks/$ID/test`. 4. Inspect the request on webhook.site — headers, body, signature. Useful for verifying signature math by pasting the body and the secret into a local script. ## CLI workflow The Autousers CLI wraps the synthetic-event flow: ```bash theme={null} # Install npm i -g @autousers/cli # Login and select team autousers auth login autousers team select # Fire a synthetic event at every enabled endpoint on the team autousers webhooks trigger evaluation.completed # Tail recent deliveries autousers webhooks deliveries --endpoint whe_clxq3... --limit 20 # Replay a specific delivery autousers webhooks replay whd_clxq3... ``` See [the CLI reference](/integrations/cli). ## A first-call sanity check A 30-second loop to confirm everything works end-to-end. ```bash theme={null} # 1. Mint an API key with webhooks:write scope (in the dashboard). export AUTOUSERS_API_KEY=ak_live_... # 2. Create an endpoint pointing at webhook.site. ENDPOINT=$(curl -s -X POST https://app.autousers.ai/api/v1/webhooks \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://webhook.site/your-uuid", "enabled_events": ["evaluation.completed"] }' | jq -r .id) # 3. Fire a synthetic event. curl -X POST https://app.autousers.ai/api/v1/webhooks/$ENDPOINT/test \ -H "Authorization: Bearer $AUTOUSERS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "type": "evaluation.completed" }' # 4. Refresh webhook.site — see the request, copy the body and the # Autousers-Signature header. # 5. Verify the signature with the snippet from # /webhooks/signature-verification using your endpoint's secret. ``` ## Common questions No. Synthetic events are free and do not consume RPM, monthly quota, or evaluation quota. They do produce a real `WebhookDelivery` row in the log so you can verify retry behaviour end-to-end. Only if it returns non-2xx — the same retry schedule applies to synthetic deliveries. Use this to test your receiver's idempotency. Yes. The CLI is unauthenticated except for the API key — drop `AUTOUSERS_API_KEY` in CI env and run `autousers webhooks trigger ...` as a step. See the [GitHub Actions recipe](/integrations/recipes#github-actions-gate).