Pipelines

The Data Platform pipeline catalog covers three kinds of pipeline, all callable through one unified API surface (GET /v2/data-platform/pipelines/catalog, POST /v2/data-platform/pipelines/:slug/run):

• Language Technology (7 pipelines) — extract or transform content, returning transcripts, translations, detected languages, named entities, or extracted text as structured JSON. Unlike Verification Pipelines, they don't produce an authenticity verdict.
• Cleaning & transformation (3 pipelines) — clean-audio-silence, convert-format, pii-redact-text. Each processes every file in a dataset version and produces a new derived version.
• Dataset management (2 pipelines) — dedup-content-hash (flags duplicates in place, no derived version) and split-dataset-version (partitions a version into train/val/test versions, synchronous, no queue). crowd-label-classify is covered separately below since it creates a crowd job rather than processing files directly.

Language Technology Pipelines

Each LT pipeline runs on one file at a time. Results are stored as structured JSON on the run record and can be retrieved via the API or viewed inline in the platform.

Cleaning, Transformation & Dataset Management Pipelines

These pipelines only ever run against a dataset version (targetType: "dataset_version") — never an ad-hoc set of project files.

Deduplication review: dedup-content-hash never deletes anything automatically. Review flagged duplicate groups in the run's result, then soft-exclude a file from the version with PATCH /v2/datasets/:datasetId/versions/:versionId/files/:fileId { "excluded": true } (reversible — pass false to re-include it).

Crowd-assisted labeling: crowd-label-classify creates an input_data_sets/input_data_lists pair (one task variant per file), a survey template, and a crowd job — one label per file, majority-vote consensus resolved independently per file once minResponsesPerFile (default 3) workers have answered that file. Because crowd jobs are project-scoped, the request body must include a projectId. See Crowdsourcing Jobs for how the underlying job/task/answer runtime works.

How to Trigger a Pipeline

You can run a pipeline from the platform UI or via the API.

From the Platform

Open a dataset version (or, for Language Technology pipelines, a project's Files tab), select a pipeline, pick the files to process if applicable, fill in any required context fields, and click Run.

Via the API

Unified endpoint (recommended) — works for every pipeline in the catalog:

POST /v2/data-platform/pipelines/{slug}/run

{
  "targetType": "dataset_version",
  "datasetId": "dataset_abc123",
  "versionId": "version_def456",
  "contextData": { "target_language": "English" }
}

Or against an ad-hoc set of project files (Language Technology pipelines only):

{
  "targetType": "project_file",
  "projectId": "project_abc123",
  "fileIds": ["file_abc123"],
  "contextData": {}
}

The response is { "runId": "...", "fileCount": N } — plus trainVersionId/valVersionId/testVersionId for split-dataset-version, or crowdJobId for crowd-label-classify.

Legacy endpoints (Language Technology pipelines only, still functional as deprecated aliases):

POST /v2/lt-pipelines/project/{projectId}/run
POST /v2/lt-pipelines/dataset/{datasetId}/versions/{versionId}/run

Both return a runIds array — one ID per file — rather than the unified endpoint's single runId per invocation.

Checking Results

Poll or fetch a single run via the unified endpoint:

GET /v2/data-platform/pipelines/runs/{runId}

List runs, filtered by project, dataset version, or status:

GET /v2/data-platform/pipelines/runs?datasetVersionId={versionId}&status=completed

A run includes its per-file, per-stage breakdown (stageRuns). The legacy GET /v2/lt-pipelines/runs/{runId} and GET /v2/lt-pipelines/project/{projectId}/runs endpoints remain available for Language Technology pipelines triggered through the legacy routes.

Language Technology Pipeline Pages