Cross-Modal Multimedia Consistency

Multi-file AI analysis and crowd cross-reference to verify consistency across related images, videos, and audio files.

The Cross-Modal Multimedia Consistency pipeline accepts 2 to 10 media files of mixed types (images, videos, and audio) and checks whether they are mutually consistent — i.e. whether they could plausibly originate from the same real-world event. It is designed for investigative teams that have gathered multiple pieces of purported evidence and need to determine whether they are authentic and coherent as a set.

Unlike per-file pipelines that assess individual pieces of media in isolation, this pipeline treats the entire file set as the unit of analysis. It first processes each file independently, then cross-references the results to detect inconsistencies across time, location, lighting, audio environment, and metadata. A crowd stage involving five workers provides a human layer of cross-referencing before the final verdict is synthesised.

Pipeline Details

Property	Value
Pipeline ID	`verify-multimedia-cross-modal`
Tier	3
Estimated Duration	~90 min
Credit Cost	4,500 credits per run
Crowd Stages	Yes (5 workers, 60% consensus)
Files Required	Yes — 2–10 media files (mixed types: image, video, audio)

Required Inputs

Files

Between 2 and 10 media files. Mixed file types are accepted — images, videos, and audio files may all be included in the same run. All files must belong to the same project and have enrichmentStatus: "completed" before the run is created. See Files & Enrichment for details.

Context Keys

Key	Required	Description
`claimed_event`	Optional	Event or news story the files purport to document.
`claimed_date`	Optional	Claimed date of the event.
`claimed_location`	Optional	Claimed location of the event.

Stages

Individual Analysis (AI)

Each file in the set is individually analysed for integrity and contextual signals — extracting timestamps, GPS coordinates, device metadata, ambient audio characteristics, lighting conditions, and any other signals available for that file type. This stage produces a per-file profile that feeds into the consistency check.

Consistency Check (AI)

Cross-references all per-file profiles to assess whether the files are mutually consistent. The AI checks whether timestamps align, whether locations are geographically plausible relative to each other, whether lighting and shadow directions are coherent across images and video frames, and whether audio ambient signals match the visual environments shown. Any cross-modal discrepancies are flagged with an explanation.

Crowd Cross-Reference (Crowd)

Five workers with the general tag independently review the complete set of files alongside the AI consistency findings. Workers assess whether the files feel coherent as a set and flag anything the AI may have missed. A minimum of 60% consensus (3 out of 5 workers) is required to proceed to a definitive verdict.

Unified Verdict (AI Synthesis)

Aggregates the individual file analyses, the consistency check findings, and the crowd cross-reference results into a single cross-modal verdict for the entire file set. The verdict reflects whether the files are consistent with each other and with the claimed event context — not whether any individual file is authentic in isolation.

Starting a Run

curl -X POST https://api.crowdee.ai/v2/projects/{projectId}/verification-runs \
  -H "X-API-Key: crw_YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "pipeline_id": "verify-multimedia-cross-modal",
    "file_ids": ["{imageFileId}", "{videoFileId}", "{audioFileId}"],
    "context": {
      "claimed_event": "Protest at Brandenburg Gate",
      "claimed_date": "2024-09-22",
      "claimed_location": "Berlin, Germany"
    }
  }'

Example Response

{
  "id": "run_01j9x...",
  "status": "completed",
  "verdict": "inconclusive",
  "confidence": 61,
  "pipeline_id": "verify-multimedia-cross-modal",
  "scorecard": {
    "image_integrity": "pass",
    "video_integrity": "pass",
    "audio_timestamp_consistency": "fail",
    "location_coherence": "pass",
    "crowd_consensus": "pass"
  },
  "explanation": "Image and video files are mutually consistent with the claimed event, date, and location. However, the audio file's embedded timestamp is inconsistent with the video — the audio was recorded approximately 4 hours earlier than the video metadata indicates. This discrepancy prevents a confident authentic verdict."
}

Pass all file IDs in the file_ids array — between 2 and 10 files. All files must belong to the same project and have enrichmentStatus: "completed".

The verdict reflects consistency across the file set, not the authenticity of individual files. For per-file authentication, run individual pipelines first and use this pipeline to check cross-modal coherence.