Quality Control
Consensus thresholds, Inter-Rater Agreement scores, and response validation in crowd stages.
Crowd quality control ensures that pipeline verdicts produced by crowd stages are reliable, consistent, and defensible. A crowd stage whose workers disagree or whose responses are low-quality would produce a verdict with little probative value. To prevent this, Crowdee applies three interlocking quality mechanisms to every crowd stage: minimum response counts, consensus thresholds, and Inter-Rater Agreement (IRA) scoring. Together these mechanisms mean that a crowd stage can only produce a verdict when a sufficient number of workers have responded and their responses demonstrate meaningful agreement.
Minimum Responses
The minResponses parameter specifies the minimum number of unique workers who must submit a valid response before a crowd stage is eligible to proceed. This parameter is defined in the pipeline definition and ranges from 2 to 5 depending on the pipeline and crowd tier.
A higher minResponses value provides greater statistical reliability because the verdict reflects more independent observations. It also increases cost and potentially extends latency, since the stage cannot complete until enough workers have responded. The value is therefore calibrated per pipeline to reflect the trade-off appropriate for that task's risk level and typical content volume.
Worker responses that fail schema validation — for example, because a required field was left blank — do not count toward minResponses. Only clean, validated submissions increment the response counter.
Consensus Threshold
Once minResponses has been reached, Crowdee evaluates whether the consensusThreshold has been met. The consensus threshold is the percentage of collected responses that must agree on the majority answer for the stage to conclude with a definitive verdict.
The threshold ranges from 60% to 100% depending on the pipeline and crowd tier. If the threshold is not met after collecting minResponses responses, additional workers are recruited and the evaluation repeats. If consensus cannot be achieved after the maximum configured retry limit, the stage verdict is recorded as inconclusive and the pipeline continues with that uncertainty carried forward into the synthesis stage.
A stage that concludes as inconclusive does not mean the pipeline has failed — the synthesis stage factors in the inconclusive crowd result alongside all other stage findings when producing the final verdict and confidence score. A run can still conclude as authentic or manipulated even if one crowd stage was inconclusive, provided other stages provide sufficient supporting evidence.
IRA Score
Inter-Rater Agreement measures how consistently workers agree across a crowd stage, independent of which answer was chosen. Crowdee computes the IRA score using Cohen's kappa, which accounts for the possibility of agreement by chance.
The IRA score is included in the stage record as a numeric value between −1 and 1. It feeds directly into the confidence calculation for the final verdict: a crowd stage with high IRA is treated as providing stronger evidence than one where workers were internally divided.
A high IRA score (above 0.7) indicates strong worker agreement, which boosts the crowd stage's weight in the final verdict synthesis. A low IRA score (below 0.4) signals mixed or unreliable responses and reduces that stage's contribution to the overall confidence score. An IRA score near zero suggests responses were no more consistent than random chance, and the stage's contribution to the final verdict will be minimal.
IRA scores are visible in the stage-level detail of the verification run API response and in the Platform's pipeline run detail view. They are also included in compliance exports, where they can support arguments about the quality of human review applied to a content decision.
Response Validation
Every worker response is validated against the SurveyJS template schema before it is accepted. Validation checks include required field completion, data type conformance, minimum length requirements for free-text fields, and any conditional validation rules defined in the template (such as requiring a follow-up explanation when the primary answer is "Cannot determine").
Responses that fail validation are rejected without being counted. The worker is notified immediately and given the opportunity to correct and resubmit. Repeated submission of invalid responses within a task is flagged as an anomalous quality signal.
Beyond per-response validation, Crowdee tracks response patterns across a worker's history. Workers who consistently produce responses at the extreme end of response time distributions — either implausibly fast (indicating random clicking) or suspiciously slow (indicating possible distraction or task abandonment) — are flagged for review. Workers who show a pattern of contradicting the consensus on crowd stages where consensus was achieved at high confidence are also flagged. These signals feed into the automated quality monitoring system described in Workers.
Expert vs General Crowd
The two crowd tiers — general crowd and expert crowd — use different parameter configurations that reflect the different reliability characteristics of each worker pool.
| Parameter | General crowd | Expert crowd |
|---|---|---|
minResponses | 3–5 | 2 |
consensusThreshold | 60–70% | 100% |
| Worker tags | general, image_forensics, video_forensics, native_speaker | fact_checker, identity_verification |
| Cost per stage | Lower | Higher |
| Typical latency | 15–40 min | 30–60 min |
| Use case | High-volume, broad review | High-stakes, defensible verdicts |
The expert tier's 100% consensus requirement means that if the two expert workers disagree, the outcome is always inconclusive — there is no majority to establish. This is intentional: an expert verdict carries significant weight in the final synthesis, and it would be misleading to produce a definitive expert verdict on the basis of a 50/50 split.
Gold Questions
Crowdee periodically injects known-answer "gold" questions into worker task queues to continuously assess accuracy in real conditions. Gold questions are genuine-looking tasks for which the correct answer is known in advance. Workers are not told which specific questions are gold, though they are informed that gold questions exist as part of the onboarding process.
When a worker answers a gold question incorrectly, that response is flagged. Isolated failures do not trigger consequences — workers occasionally make honest mistakes on edge-case content. However, a pattern of failures across multiple gold questions triggers an automatic quality review. Workers who fail quality reviews are suspended from the pool pending investigation and may be permanently removed.
Gold question results are not included in crowd stage response counts. A worker's gold question answer is evaluated separately and does not contribute to the stage verdict or IRA calculation. This ensures that quality monitoring does not inadvertently distort verification outcomes.
How is this guide?
Workers
The Crowdee crowd panel, worker specializations, and tag-to-pipeline mapping.
Compliance Overview
Regulatory frameworks that Crowdee is designed to support.