# AI Output Evaluation (/docs/concepts/ai-output-evaluation)



Verification and Language Technology pipelines produce a verdict and an explanation, but a confidence score alone doesn't tell you whether that explanation actually makes sense to a human. AI output evaluation asks a crowd panel to rate exactly that — how clear, well-evidenced, and actionable a given AI result is — and logs every rating as a durable feedback signal for that AI component.

<Callout type="info">
  This is a request-scoped feature, not something you configure once: you request an evaluation for a specific completed run, a crowd panel rates it, and you get back an aggregate score plus per-response feedback.
</Callout>

## Requesting an Evaluation [#requesting-an-evaluation]

Evaluations are requested against a completed run, identified by a source type and id:

```bash
curl -X POST https://api.crowdee.ai/v2/projects/{projectId}/ai-output-evaluations \
  -H "X-API-Key: crw_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "sourceType": "verification_pipeline_run",
    "sourceId": "run_01j9x...",
    "minResponses": 3
  }'
```

| Field                | Required               | Description                                                                                            |
| -------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------ |
| `sourceType`         | **Required**           | `"verification_pipeline_run"`, `"lt_pipeline_run"`, or `"external_submission"` (see below).            |
| `sourceId`           | **Required**           | The id of the completed run to evaluate.                                                               |
| `evaluationCriteria` | Optional               | Freeform JSON describing what reviewers should focus on, surfaced alongside the run in the crowd task. |
| `minResponses`       | Optional (default `3`) | How many crowd responses to collect before the batch is scored.                                        |

The response is an evaluation batch:

```json
{
  "id": "batch_01j9x...",
  "sourceType": "verification_pipeline_run",
  "sourceId": "run_01j9x...",
  "status": "dispatched",
  "minResponses": 3,
  "aggregateTransparencyScore": null
}
```

Behind the scenes, requesting an evaluation automatically generates a crowd task showing the run's pipeline, verdict, and explanation, and dispatches it to `minResponses` crowd workers — you don't need to build a task template yourself.

<Callout type="warn">
  Each requested evaluation blocks credits from your organisation balance, the same way a crowd job does. If your balance is insufficient, the request fails with `403 Forbidden`.
</Callout>

## Evaluating Your Own AI's Output [#evaluating-your-own-ais-output]

`sourceType: "external_submission"` lets you request an evaluation for an AI result produced outside Crowdee — your own model, a third-party API, anything with a verdict and an explanation. There's no pre-existing internal run to point at, so use the dedicated endpoint instead, which takes the output directly:

```bash
curl -X POST https://api.crowdee.ai/v2/projects/{projectId}/ai-output-evaluations/external \
  -H "X-API-Key: crw_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "modelName": "Acme Fraud Detector v3",
    "verdict": "likely fraudulent",
    "explanation": "Transaction pattern matches known card-testing behaviour: 14 micro-charges across 6 minutes to unrelated merchants.",
    "confidence": 87,
    "evidenceUrls": ["https://internal.acme.example/case/8841"],
    "minResponses": 3
  }'
```

| Field                                 | Required     | Description                                                         |
| ------------------------------------- | ------------ | ------------------------------------------------------------------- |
| `modelName`                           | **Required** | Name of the AI system/model that produced the output.               |
| `verdict`                             | **Required** | The verdict or label your AI produced.                              |
| `explanation`                         | **Required** | Your AI's reasoning for the verdict — this is what the crowd rates. |
| `confidence`                          | Optional     | Your AI's self-reported confidence, 0–100.                          |
| `evidenceUrls`                        | Optional     | Supporting links, if any.                                           |
| `evaluationCriteria` / `minResponses` | Optional     | Same meaning as the standard endpoint.                              |

Under the hood, the submitted fields are stored as a one-variant [input data set](/docs/crowdsourcing/input-data) (one list per field) rather than a dedicated table — `sourceId` in the resulting batch points at that input data set's id, the same identifier you'd get back from the regular input-data API. Everything downstream — `GET /v2/ai-output-evaluations`, `.../results`, and the feedback log — works identically regardless of source type.

## Cost [#cost]

Each crowd response costs **15 credits**, regardless of the source pipeline. Total cost is `minResponses × 15` credits, blocked upfront when the batch is dispatched:

| `minResponses` | Cost        |
| -------------- | ----------- |
| 3 (default)    | 45 credits  |
| 5              | 75 credits  |
| 10             | 150 credits |

## What Reviewers Rate [#what-reviewers-rate]

Each reviewer rates the AI result on four dimensions, 1–5 each:

| Dimension            | Question                                                                            |
| -------------------- | ----------------------------------------------------------------------------------- |
| Clarity              | Is the explanation easy to understand?                                              |
| Evidence sufficiency | Is the reasoning backed by specific evidence?                                       |
| Actionability        | Is it clear what to do with this result?                                            |
| Bias risk            | How likely is this result to be skewed or unbalanced? (1 = low risk, 5 = high risk) |

Reviewers can also flag the result for retraining or prompt review, and leave free-text notes.

## Batch Status and Scoring [#batch-status-and-scoring]

| Status       | Meaning                                                       |
| ------------ | ------------------------------------------------------------- |
| `pending`    | Batch created but not yet dispatched (transient).             |
| `dispatched` | Crowd task is live, awaiting responses.                       |
| `completed`  | `minResponses` reached; an aggregate score has been computed. |

Once enough responses arrive, Crowdee computes a **transparency score** (0–100) per response — clarity, evidence sufficiency, and actionability contribute positively, bias risk contributes negatively — and averages across all responses into the batch's `aggregateTransparencyScore`.

```bash
GET https://api.crowdee.ai/v2/ai-output-evaluations/{batchId}
X-API-Key: crw_YOUR_API_KEY
```

```json
{
  "id": "batch_01j9x...",
  "status": "completed",
  "minResponses": 3,
  "responseCount": 3,
  "aggregateTransparencyScore": "78.00"
}
```

Fetch each individual rating and any free-text feedback:

```bash
GET https://api.crowdee.ai/v2/ai-output-evaluations/{batchId}/results
X-API-Key: crw_YOUR_API_KEY
```

## The Feedback Loop [#the-feedback-loop]

Every submitted rating is also written to an append-only feedback log, keyed by the AI pipeline it evaluated. This is the record behind improving an AI component over time — Crowdee does not automatically retrain or adjust prompts based on it; it's a durable signal for your team (or ours) to act on manually.

```bash
curl -X PATCH https://api.crowdee.ai/v2/ai-output-evaluations/feedback/{feedbackLogId} \
  -H "X-API-Key: crw_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"appliedNotes": "Reworded verdict-synthesis prompt to cite specific evidence."}'
```

Marking a feedback entry as applied records who applied it and when, and any notes — a lightweight audit trail for prompt or pipeline changes made in response to crowd feedback.

<Callout type="info">
  The evaluation data model identifies the AI output being rated by `sourceType` and `sourceId` rather than assuming it always comes from a Crowdee pipeline — Crowdee's own verification and Language Technology pipeline runs, and externally submitted AI output, are all evaluated through the same batch, evaluation, and feedback-log tables.
</Callout>