Datasets

Versioned collections of media files for training and testing verification pipelines.

Datasets are managed collections of media files that exist independently of projects. Unlike project files, which are attached to a single research workflow, datasets are reusable: you can upload a corpus of audio clips once and apply different cleaning and enrichment pipelines without duplicating the underlying files.

Every dataset is versioned. The system tracks how files evolve through cleaning and enrichment so you always have a clear audit trail from raw input to processed output.

Modalities

Each dataset has a single modality that describes the type of content it contains.

Modality	Content types
`image`	JPEG, PNG, WebP, TIFF, GIF
`audio`	MP3, WAV, FLAC, OGG, M4A
`video`	MP4, MOV, WebM, AVI
`text`	TXT, MD, HTML
`document`	PDF
`multimodal`	Mixed types within a single dataset

The modality determines which cleaning pipelines are available and which enrichment workers apply when you enrich uploaded files.

Versioning

Every dataset starts with one version: the raw version. This version is immutable — once files are uploaded they cannot be replaced, only added or removed. All subsequent processing creates new derived versions.

raw  ──→  cleaned  ──→  cleaned_and_enriched
 └──────────────────────→ (enrichment from raw)

Version states

State	Meaning
`raw`	Initial version; files uploaded directly by the user
`cleaning`	A cleaning pipeline job is in progress
`cleaned`	Cleaning completed successfully
`enriching`	Per-file enrichment jobs are running
`cleaned_and_enriched`	Enrichment completed on a cleaned version
`failed`	Processing failed; see the version's `error` field for details

Enrichment can be triggered from either a raw or cleaned version. Starting enrichment always creates a new derived version, so the source version is never mutated.

Cleaning Pipelines

A cleaning pipeline is a background job that processes every file in a version and produces a new derived version with the cleaned output. The pipeline slug identifies the transformation to apply.

Available cleaning pipelines:

Slug	Modality	Description
`clean-audio-silence`	`audio`	Trims leading and trailing silence (threshold −50 dBFS, minimum 0.3 s) using ffmpeg

The full list is served at GET /v2/datasets/catalog/cleaning-pipelines. Each pipeline declares the modalities it supports so the API and UI can filter compatible options automatically.

Files that the cleaning pipeline does not need to modify are still linked to the new version unchanged. Only files where silence was actually detected are replaced.

File Provenance

When a cleaning pipeline produces a modified file it records the sourceFileId — the ID of the original raw file. This link is stored in dataset_version_files and is returned on version detail responses.

{
  "id": "dvf_abc123",
  "fileId": "file_cleaned_xyz",
  "sourceFileId": "file_raw_abc"
}

Use sourceFileId to trace any processed file back to its original. The field is null for files that were not modified.

Using Datasets with Verification Pipelines

Once a dataset version has been enriched, its files carry the same technical metadata as regular project files and can be submitted to verification runs. Export presigned download URLs for a version via GET .../versions/{versionId}/export, then upload the files to a project if you need to reference them by project file ID.

See Files & Enrichment for the enrichment fields extracted per media type.