Crowdsourcing Overview

How Crowdee's crowdsourcing system works — both as automated crowd stages in verification pipelines and as directly published crowd jobs.

Crowdee's crowdsourcing system distributes structured review tasks to a vetted global pool of crowd workers and domain experts. Human judgment is incorporated at defined points in the verification process, and also directly via platform-managed crowd job campaigns.

Crowdsourcing Modes

Crowdee supports two distinct modes of crowdsourcing, which can be used independently or in combination.

Pipeline crowd stages — automated crowd review embedded in Tier 2 and Tier 3 verification pipelines. When a pipeline reaches a crowd stage, Crowdee automatically distributes tasks to eligible workers, aggregates responses, applies consensus thresholds and IRA scoring, and resumes the pipeline with the crowd verdict. You do not create or manage these tasks directly — they are driven entirely by the pipeline configuration.

Direct crowd jobs — platform-managed campaigns where you define a survey template, optionally attach an input data set to generate task variants, set per-task rewards and validity windows, and publish the job to the worker pool. Workers claim task slots independently; you review and accept or reject each answer, then export results or convert them into datasets and input data for further pipeline runs. Direct crowd jobs are independent of verification pipelines and are the right tool for data collection, annotation, qualification testing, and iterative labelling workflows.

Role of Crowd in Verification

Crowd stages serve as quality gates within a pipeline. The AI runs first, producing structured findings and a preliminary verdict. Crowd workers then review the AI findings and apply their own independent judgment to the same content. This hybrid model consistently reduces both false positives and false negatives compared to either AI-only or human-only approaches, because each method compensates for the other's weaknesses.

AI stages are fast and consistent but can miss cultural nuance, misread context, or fail on unusual compression artefacts. Human workers catch what the model misses. Conversely, crowd stages are subject to individual worker variation, which is why consensus thresholds and Inter-Rater Agreement scoring are applied before a crowd stage can conclude. The result is a verdict that carries the complementary strengths of both.

Pipeline tiers that include crowd stages:

Tier 2 (AI + Crowd): verify-image-deep, verify-video-full, verify-audio-deepfake, verify-news-article, verify-social-post
Tier 3 (Context-driven or multi-file, with optional crowd): verify-url-credibility, verify-identity-claim, verify-multimedia-cross-modal

Tier 1 pipelines are AI-only and do not involve crowd stages.

Crowdsourcing Architecture

Crowd Jobs

Create, publish, and manage crowdsourcing jobs directly — define tasks, set rewards, and control the full job lifecycle.

Answers

Review, accept, reject, and export collected worker answers, or convert them into datasets and input data for further pipeline runs.

Task Templates

How crowd tasks are structured using SurveyJS — the questionnaire each worker sees during a crowd stage.

Input Data

Datasets, lists, and groups that drive task variant generation for bulk or parameterised crowd review.

Workers

The Crowdee crowd panel, worker specialization tags, and how task routing matches tasks to eligible workers.

Quality Control

Consensus thresholds, Inter-Rater Agreement scoring, and response validation — how Crowdee ensures crowd verdicts are reliable.

Crowd Stage Lifecycle

A crowd stage begins when a running pipeline reaches a stage of type crowd or expert. The overall run status transitions from running to awaiting_crowd and the pipeline pauses until the crowd stage is resolved.

Task preparation — When the pipeline reaches a crowd stage, a task is assembled using the crowd stage's configured SurveyJS template, the file's enrichment metadata, and any applicable verification context. The task is prepared as a structured questionnaire ready for distribution.

Worker matching and distribution — The task is distributed to eligible workers whose tags match the crowd stage's workerTags requirement. Workers are selected automatically by Crowdee's matching system based on tags, language, and current availability. You cannot select individual workers.

Worker responses — Workers complete the structured questionnaire independently. Each response is validated against the SurveyJS template schema. Incomplete or invalid responses are rejected and the worker is prompted to resubmit.

Aggregation and consensus check — Once the minimum number of responses (minResponses) has been collected, Crowdee checks whether the consensusThreshold is met. If the required percentage of workers agree on the majority answer, the crowd stage records a verdict and the pipeline resumes.

Additional recruitment (if needed) — If minResponses have been collected but the consensusThreshold has not been reached, additional workers are recruited. If consensus cannot be achieved after the maximum number of retries, the stage verdict is recorded as inconclusive.

Pipeline resumes — The crowd stage result — verdict, confidence, and Inter-Rater Agreement score — is written to the stage record and made available to the next stage. For pipelines that follow a crowd stage with an AI synthesis stage, the AI receives the crowd findings as part of its context.

Enabling Crowdsourcing

The crowdsourcing module must be enabled on your organisation before crowd stages can execute. When crowdsourcing is not enabled, pipelines that include crowd stages will fail at the crowd stage with a configuration error rather than routing tasks to workers.

To enable crowdsourcing, contact hello@crowdee.ai. Enterprise and Managed plan customers can enable crowdsourcing as part of their onboarding. Usage is billed per crowd stage execution, with pricing differentiated between general crowd and expert tiers.

Crowdsourcing is included by default in Managed and Enterprise plans. If you are on a self-service plan and want to run Tier 2 or Tier 3 pipelines, reach out to discuss access.