Continuous AI Monitoring

AI output evaluation rates a single run on demand. Continuous monitoring is the always-on version of the same idea: you define a schedule once, and Crowdee periodically samples a recent run, dispatches a crowd evaluation for it, and tracks how the resulting transparency score moves over time — flagging a run when it deviates meaningfully from your schedule's own trailing average.

Creating a Schedule

curl -X POST https://api.crowdee.ai/v2/monitoring-schedules \
  -H "X-API-Key: crw_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "sourceType": "verification_pipeline_run",
    "cadence": "daily",
    "sampleCount": 1
  }'

What Happens on Each Tick

On its configured cadence, a schedule:

1. Looks at completed runs in scope (organisation- or project-wide, matching sourceType) from the preceding period that haven't already been sampled by this schedule.
2. If none are found, records a monitoring_runs row with sampledCount: 0 and status: "completed" — an empty period is not an error.
3. Otherwise, dispatches an AI output evaluation batch for the sampled run exactly as POST /v2/projects/{projectId}/ai-output-evaluations would, blocking credits from your organisation balance the same way.
4. Once the crowd panel reaches quorum, the batch's aggregate transparency score is written back to the monitoring run, along with a driftFlag comparing it against the schedule's trailing average of prior completed runs.

GET https://api.crowdee.ai/v2/monitoring-schedules/{scheduleId}/runs
X-API-Key: crw_YOUR_API_KEY

[
  {
    "id": "mrun_01j9x...",
    "periodStart": "2026-07-02T00:00:00.000Z",
    "periodEnd": "2026-07-03T00:00:00.000Z",
    "sampledCount": 1,
    "batchId": "batch_01j9x...",
    "aggregateScore": "62.00",
    "driftFlag": true,
    "status": "completed"
  }
]

Cost

A tick that finds nothing to sample is free. A tick that dispatches an evaluation batch costs the same as a manual AI Output Evaluation — 45 credits by default (15 credits per crowd response × 3 responses) — blocked from your organisation balance at dispatch time. There is no separate monitoring-specific fee.

Drift Detection

driftFlag is set when a run's aggregate transparency score deviates from the schedule's own trailing average of prior completed runs by more than a fixed threshold. There's no baseline to compare against on a schedule's first sampled run, so driftFlag is always false until at least one prior run has completed — drift detection is inherently a relative, self-baselining signal per schedule, not an absolute quality bar.

Use PATCH to pause a schedule (isActive: false) without losing its history, adjust its cadence, or change what it samples:

curl -X PATCH https://api.crowdee.ai/v2/monitoring-schedules/{scheduleId} \
  -H "X-API-Key: crw_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"cadence": "hourly"}'

Deleting a schedule (DELETE /v2/monitoring-schedules/{scheduleId}) stops future ticks; past monitoring runs and their evaluation batches are unaffected.