Datasets API

Create and manage versioned media file collections, apply cleaning pipelines, and export processed files.

Datasets are organization-scoped versioned file collections. Files are uploaded to an immutable raw version; cleaning and enrichment jobs create new derived versions without modifying the source.

Endpoint Overview

Method	Path	Description
`GET`	`/v2/datasets`	List all datasets in the active organization
`POST`	`/v2/datasets`	Create a dataset
`GET`	`/v2/datasets/{datasetId}`	Get dataset with all versions
`PATCH`	`/v2/datasets/{datasetId}`	Update name or description
`DELETE`	`/v2/datasets/{datasetId}`	Delete dataset and all versions
`POST`	`/v2/datasets/{datasetId}/files`	Upload files to the raw version
`DELETE`	`/v2/datasets/{datasetId}/files/{fileId}`	Remove a file from the raw version
`GET`	`/v2/datasets/{datasetId}/versions`	List all versions
`GET`	`/v2/datasets/{datasetId}/versions/{versionId}`	Get version with paginated file listing
`POST`	`/v2/datasets/{datasetId}/versions/{versionId}/clean`	Trigger a cleaning pipeline
`POST`	`/v2/datasets/{datasetId}/versions/{versionId}/enrich`	Trigger enrichment on all files
`GET`	`/v2/datasets/{datasetId}/versions/{versionId}/export`	Get presigned download URLs
`GET`	`/v2/datasets/catalog/cleaning-pipelines`	List available cleaning pipelines

Create a Dataset

POST /v2/datasets

Request body

Field	Type	Required	Description
`name`	string	Yes	Human-readable display name
`internalId`	string	Yes	Slug used in S3 paths; letters, numbers, hyphens, and underscores only; unique within the organization
`modality`	string	Yes	One of `image`, `audio`, `video`, `text`, `document`, `multimodal`
`description`	string	No	Optional description

curl -X POST https://api.crowdee.ai/v2/datasets \
  -H "X-API-Key: crw_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Interview Audio 2024-Q4",
    "internalId": "interview-audio-q4-2024",
    "modality": "audio",
    "description": "Raw interview recordings for speaker verification"
  }'

Response (201):

{
  "id": "ds_abc123",
  "name": "Interview Audio 2024-Q4",
  "internalId": "interview-audio-q4-2024",
  "modality": "audio",
  "description": "Raw interview recordings for speaker verification",
  "organizationId": "org_xyz789",
  "createdAt": "2024-11-15T10:00:00Z"
}

A raw version is created automatically alongside the dataset. Retrieve it via GET /v2/datasets/{datasetId} (returns the dataset with a versions array).

Upload Files

POST /v2/datasets/{datasetId}/files

Adds files to the dataset's raw version. Submit a multipart/form-data request with one or more files under the key file[].

curl -X POST https://api.crowdee.ai/v2/datasets/ds_abc123/files \
  -H "X-API-Key: crw_YOUR_API_KEY" \
  -F "file[]=@recording1.mp3" \
  -F "file[]=@recording2.wav"

Returns an array of created file objects. Enrichment is not started automatically — call the enrich endpoint separately when ready.

Files are stored in S3 at datasets/{organizationId}/{internalId}/raw/. The internalId you set at creation time is permanent and becomes part of every file path in this dataset.

List Versions

GET /v2/datasets/{datasetId}/versions

Returns all versions in creation order. Use the state field to determine which versions are ready for further processing.

curl https://api.crowdee.ai/v2/datasets/ds_abc123/versions \
  -H "X-API-Key: crw_YOUR_API_KEY"

Response:

[
  {
    "id": "ver_raw001",
    "state": "raw",
    "label": null,
    "parentVersionId": null,
    "cleaningPipelineSlug": null,
    "fileCount": 12,
    "error": null,
    "createdAt": "2024-11-15T10:00:00Z"
  },
  {
    "id": "ver_clean001",
    "state": "cleaned",
    "label": "Silence trimmed",
    "parentVersionId": "ver_raw001",
    "cleaningPipelineSlug": "clean-audio-silence",
    "fileCount": 12,
    "error": null,
    "createdAt": "2024-11-15T10:05:00Z"
  }
]

Get Version with Files

GET /v2/datasets/{datasetId}/versions/{versionId}

Query parameters

Parameter	Type	Default	Max	Description
`page`	integer	`1`	—	Page number (1-based)
`limit`	integer	`50`	`200`	Files per page

curl "https://api.crowdee.ai/v2/datasets/ds_abc123/versions/ver_clean001?page=1&limit=50" \
  -H "X-API-Key: crw_YOUR_API_KEY"

Response:

{
  "id": "ver_clean001",
  "state": "cleaned",
  "label": "Silence trimmed",
  "fileCount": 12,
  "files": [
    {
      "id": "dvf_001",
      "fileId": "file_cleaned_001",
      "sourceFileId": "file_raw_001",
      "file": {
        "id": "file_cleaned_001",
        "name": "recording1_cleaned.mp3",
        "size": 1843200,
        "type": "audio/mpeg",
        "createdAt": "2024-11-15T10:05:30Z"
      }
    },
    {
      "id": "dvf_002",
      "fileId": "file_raw_002",
      "sourceFileId": null,
      "file": {
        "id": "file_raw_002",
        "name": "recording2.wav",
        "size": 4096000,
        "type": "audio/wav",
        "createdAt": "2024-11-15T10:01:00Z"
      }
    }
  ],
  "total": 12,
  "page": 1,
  "limit": 50
}

sourceFileId is non-null only for files that were modified by a cleaning pipeline — it points back to the original raw file. Files that passed through the cleaning pipeline unchanged have sourceFileId: null.

Clean a Version

POST /v2/datasets/{datasetId}/versions/{versionId}/clean

Starts a cleaning pipeline job on the given version. The source version must be in raw or cleaned state. A new version is created with state cleaning; it transitions to cleaned on success or failed on error.

Request body

Field	Type	Required	Description
`pipelineSlug`	string	Yes	Cleaning pipeline to apply (e.g. `clean-audio-silence`)
`label`	string	No	Human-readable label for the new version

curl -X POST https://api.crowdee.ai/v2/datasets/ds_abc123/versions/ver_raw001/clean \
  -H "X-API-Key: crw_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "pipelineSlug": "clean-audio-silence",
    "label": "Silence trimmed Nov 2024"
  }'

Response (200) — the new derived version (state cleaning):

{
  "id": "ver_clean001",
  "state": "cleaning",
  "label": "Silence trimmed Nov 2024",
  "parentVersionId": "ver_raw001",
  "cleaningPipelineSlug": "clean-audio-silence",
  "fileCount": 0,
  "error": null,
  "createdAt": "2024-11-15T10:05:00Z"
}

Poll GET /v2/datasets/{datasetId}/versions/{versionId} until state becomes cleaned or failed.

Only one cleaning job can run at a time per dataset. If another version is already in cleaning state, the request returns 400.

Enrich a Version

POST /v2/datasets/{datasetId}/versions/{versionId}/enrich

Queues a per-file enrichment job for every file in the given version. The version must be in raw or cleaned state. A new derived version with state enriching is created and transitions to cleaned_and_enriched once all files have been processed.

curl -X POST https://api.crowdee.ai/v2/datasets/ds_abc123/versions/ver_clean001/enrich \
  -H "X-API-Key: crw_YOUR_API_KEY"

Response (200): the newly created version object in enriching state.

Enrichment uses the same per-MIME-type workers as project files (enrich-audio, enrich-video, enrich-image, enrich-text). Poll the new version until its state is cleaned_and_enriched or failed.

Export a Version

GET /v2/datasets/{datasetId}/versions/{versionId}/export

Returns presigned S3 download URLs (24-hour expiry) for every file in the version.

curl https://api.crowdee.ai/v2/datasets/ds_abc123/versions/ver_clean001/export \
  -H "X-API-Key: crw_YOUR_API_KEY"

Response:

{
  "files": [
    {
      "fileId": "file_cleaned_001",
      "name": "recording1_cleaned.mp3",
      "url": "https://s3.example.com/datasets/.../recording1_cleaned.mp3?X-Amz-Expires=86400&..."
    },
    {
      "fileId": "file_raw_002",
      "name": "recording2.wav",
      "url": "https://s3.example.com/datasets/.../recording2.wav?X-Amz-Expires=86400&..."
    }
  ],
  "expiresInSeconds": 86400
}

Cleaning Pipeline Catalog

GET /v2/datasets/catalog/cleaning-pipelines

Lists all available cleaning pipelines with their supported modalities.

curl https://api.crowdee.ai/v2/datasets/catalog/cleaning-pipelines \
  -H "X-API-Key: crw_YOUR_API_KEY"

Response:

[
  {
    "slug": "clean-audio-silence",
    "name": "Silence Trimmer",
    "description": "Removes leading and trailing silence from audio files",
    "supportedModalities": ["audio"]
  }
]

Use the supportedModalities array to filter cleaning options based on your dataset's modality before presenting them to users.

On this page