Web Research
How Crowdee performs live web search and crawling as part of source-tracing pipelines.
Most Crowdee verification pipelines reason from an AI model's training knowledge — fast, but limited to what the model already "knows." Web research stages are different: they perform a live search or crawl of the current web before handing the result to an LLM for analysis. This is what powers Source Research & Credibility Assessment and any future pipeline that needs to trace a claim back to its actual current source, rather than reason about it in the abstract.
How a Web Research Stage Runs
Resolve the query
The stage inspects the run's context data and determines either a specific URL to fetch directly, or a search query to run when no URL is already known.
Search or fetch
If a URL is known, the stage fetches that page directly. Otherwise it searches the web and retrieves the full content of the top matching results.
Analyze
The retrieved page content — title, URL, and cleaned text — is passed to an LLM alongside the stage's task-specific instructions. The model's output is validated against the stage's structured output schema, same as any AI stage.
If a specific result fails to load (dead link, blocked, timeout), that result falls back to its search snippet rather than failing the whole stage — the stage only fails outright if no sources could be gathered at all.
Failure Modes
| Condition | Behavior |
|---|---|
| A fetched page returns no usable content | That source is dropped; the stage continues with remaining sources |
| No sources could be gathered at all | Stage fails; the pipeline run transitions to failed |
Because a web research stage reflects the live state of the web rather than a fixed corpus, re-running the same pipeline on the same subject at a later date can produce a different result if the underlying web content has changed. This is expected — the stage is tracing current sources, not returning a cached judgment.
How is this guide?
Synthetic Data Detection
How Crowdee scores the likelihood that content was AI-generated, and where that signal appears in a verification run.
AI Output Evaluation
Using the crowd to rate the transparency and explainability of AI-generated verification results, and feed that back into improving them.