AI Output Evaluation

Verification and Language Technology pipelines produce a verdict and an explanation, but a confidence score alone doesn't tell you whether that explanation actually makes sense to a human. AI output evaluation asks a crowd panel to rate exactly that — how clear, well-evidenced, and actionable a given AI result is — and logs every rating as a durable feedback signal for that AI component.

Requesting an Evaluation

Evaluations are requested against a completed run, identified by a source type and id:

curl -X POST https://api.crowdee.ai/v2/projects/{projectId}/ai-output-evaluations \
  -H "X-API-Key: crw_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "sourceType": "verification_pipeline_run",
    "sourceId": "run_01j9x...",
    "minResponses": 3
  }'

The response is an evaluation batch:

{
  "id": "batch_01j9x...",
  "sourceType": "verification_pipeline_run",
  "sourceId": "run_01j9x...",
  "status": "dispatched",
  "minResponses": 3,
  "aggregateTransparencyScore": null
}

Behind the scenes, requesting an evaluation automatically generates a crowd task showing the run's pipeline, verdict, and explanation, and dispatches it to minResponses crowd workers — you don't need to build a task template yourself.

Evaluating Your Own AI's Output

sourceType: "external_submission" lets you request an evaluation for an AI result produced outside Crowdee — your own model, a third-party API, anything with a verdict and an explanation. There's no pre-existing internal run to point at, so use the dedicated endpoint instead, which takes the output directly:

curl -X POST https://api.crowdee.ai/v2/projects/{projectId}/ai-output-evaluations/external \
  -H "X-API-Key: crw_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "modelName": "Acme Fraud Detector v3",
    "verdict": "likely fraudulent",
    "explanation": "Transaction pattern matches known card-testing behaviour: 14 micro-charges across 6 minutes to unrelated merchants.",
    "confidence": 87,
    "evidenceUrls": ["https://internal.acme.example/case/8841"],
    "minResponses": 3
  }'

Under the hood, the submitted fields are stored as a one-variant input data set (one list per field) rather than a dedicated table — sourceId in the resulting batch points at that input data set's id, the same identifier you'd get back from the regular input-data API. Everything downstream — GET /v2/ai-output-evaluations, .../results, and the feedback log — works identically regardless of source type.

Cost

Each crowd response costs 15 credits, regardless of the source pipeline. Total cost is minResponses × 15 credits, blocked upfront when the batch is dispatched:

What Reviewers Rate

Each reviewer rates the AI result on four dimensions, 1–5 each:

Reviewers can also flag the result for retraining or prompt review, and leave free-text notes.

Batch Status and Scoring

Once enough responses arrive, Crowdee computes a transparency score (0–100) per response — clarity, evidence sufficiency, and actionability contribute positively, bias risk contributes negatively — and averages across all responses into the batch's aggregateTransparencyScore.

GET https://api.crowdee.ai/v2/ai-output-evaluations/{batchId}
X-API-Key: crw_YOUR_API_KEY

{
  "id": "batch_01j9x...",
  "status": "completed",
  "minResponses": 3,
  "responseCount": 3,
  "aggregateTransparencyScore": "78.00"
}

Fetch each individual rating and any free-text feedback:

GET https://api.crowdee.ai/v2/ai-output-evaluations/{batchId}/results
X-API-Key: crw_YOUR_API_KEY

The Feedback Loop

Every submitted rating is also written to an append-only feedback log, keyed by the AI pipeline it evaluated. This is the record behind improving an AI component over time — Crowdee does not automatically retrain or adjust prompts based on it; it's a durable signal for your team (or ours) to act on manually.

curl -X PATCH https://api.crowdee.ai/v2/ai-output-evaluations/feedback/{feedbackLogId} \
  -H "X-API-Key: crw_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"appliedNotes": "Reworded verdict-synthesis prompt to cite specific evidence."}'

Marking a feedback entry as applied records who applied it and when, and any notes — a lightweight audit trail for prompt or pipeline changes made in response to crowd feedback.