Skip to main content
A Rating is a single rater’s verdict on a single comparison. Humans and autousers produce the same shape so downstream analytics doesn’t have to branch on raterType.

​
Shape

Discriminator: Exactly one of (userId, publicRaterId) is set on human ratings; exactly one of (autouserId, autouserRunId) is set on autouser ratings.

​
Listing

Cursor-paginate with starting_after. See Pagination.

​
Submitting a human rating via the API

Most ratings come from the dashboard or the public share link. If you need to submit one programmatically (e.g. wiring up a custom rater UI):

​
Agreement

The /agreement endpoint reports how much your raters agreed with each other.

​
The unit is the (design, dimension) pair

One item is one design scored on one dimension. Three raters scoring one design on four dimensions is twelve paired observations, not three. Items drawn from the same rating are not independent, so a large item count from a single design is weaker evidence than the same count spread across several designs.

​
A rater is an agent, not a persona

Running one persona with agentCount: 3 starts three independent runs — three sessions, three transcripts, three judge calls — and they count as three raters in raterCount. independentRaterCount is the coarser count — distinct personas and people, so three agents of one persona are 1 — and it is the one to gate on when the question is “how many points of view”, not “how many trials”. Anonymous raters who arrive through a share link are told apart by their PublicRater row, so three strangers rating the same design are three raters, not one.

​
intraRater is replicate agreement, not test-retest

The same three runs are also three passes of ONE instrument over the same item, and intraRater reports how often that instrument gave itself the same score. It counts every unordered pair of replicates, so it does not depend on the order the runs happened to finish in — those agents run concurrently. It is a different quantity from overallAgreementPercent on the same 0-100 axis. Do not read it against the agreement bands below: they say when two raters agreeing is trustworthy, and the bar for an instrument reproducing its own score is higher.

​
What the numbers mean

overallAgreementPercent is direct-count agreement: the share of scored items where two raters gave the same score (for SxS, picked the same side). It is not a rescaled kappa, so read it against the chance floor of your scale — 20% on a 5-point scale, 14% on 7-point, ~33% for SxS direction. The kappa equivalents come from converting Landis & Koch’s bands through the chance floor of the scale — 20% for a 5-point SSE dimension, ~33% for an SxS direction, which is why the SxS column sits higher for the same kappa. They shift again on other scale lengths; use scaleGroups when your dimensions mix them. kappa.value is chance-corrected, and kappa.isMean tells you whether it is a single Cohen’s κ or an item-count-weighted mean of several — one per rater pair, and one per scale group within a pair. Kappas are never pooled across scale lengths, because chance agreement depends on how many points a rater could have picked. Krippendorff’s α appears as alpha when every dimension shares one scale.

​
When there is no number

kappa.value is null — not 0 — whenever kappa is undefined. The common case is a unanimous panel: with no variance the arithmetic is 0/0, and both Cohen’s κ and Krippendorff’s α return 1.0 by convention. kappa.blocked says which case it is and what would change it. overallAgreementPercent is null only when nothing can be measured at all, and notComputable then carries the reason (no_ratings, single_rater, no_shared_items) plus a remedy.
Two fields changed meaning at cache_version 8. Version 6 is the last one any client saw — there was never a released version 7. Branch on cache_version >= 8, or read the replacement fields, which need no version check.
  • insufficient used to mean “no overall kappa” and now means “no figure of any kind”. A unanimous panel is no longer insufficient, because it has a real agreement percentage. Read kappa_available for the old signal.
  • sample_size used to count rater-pair × comparison observations and now counts rater-pair × (design, dimension) observations, so the same evaluation reports a larger number. agreement_unit names the unit on the wire.
The same version changed rater_count: a persona run with agentCount: 3 now reports three raters rather than one. independent_rater_count carries the persona-level count. overall_kappa_is_mean is new, and is true whenever overall_kappa is a weighted mean of several kappas rather than one Cohen’s κ.

​
Caching

Agreement is cached on Evaluation.agreementCache and recomputed whenever any input it was built from has changed — a rating added, removed or edited, or the evaluation’s dimension config touched. The count alone used to be the test, which could not see an edited rating and so let a cached figure outlive the ratings behind it. The first call after a change is slightly slower (~100ms) as it warms the cache; subsequent calls are instant.

​
Streaming ratings into a warehouse

The shape is stable — dimensionRatings is a JSON map, factors and openTextResponses are JSON. Subscribe to the rating.created webhook (see Events) and append rows to BigQuery / Snowflake as they arrive. Use Autousers-Event-Id as the dedup key on insert. See the Looker / BigQuery recipe.