Data Methodology
How HeadlineCrunch clusters coverage, identifies framing differences, and sources its data.
Data sources
HeadlineCrunch indexes coverage from configurable RSS feeds, licensed news APIs, and public records. The platform respects
robots.txt and publisher access restrictions, never bypasses paywalls or technical controls, and stores only the metadata needed for search, comparison, attribution, and lawful short excerpts. Wire-service duplicates are detected and not counted as independent sources. Affiliate, sponsored, partner, and press-release content is clearly labeled.The clustering process
- Ingestion: Each article is parsed for its headline, byline, publish time, outlet, and a short lawful excerpt.
- Entity extraction: AI identifies the people, organizations, locations, and topics each article references.
- Matching: Articles are grouped into a cluster when they concern the same underlying event — not merely sharing a name or topic. Materially different developments are kept separate, even when adjacent in time.
- Deduplication: Wire-service republications are detected and folded, so five copies of one AP report count as one source, not five.
- Updates and corrections: When an article is updated or a correction issued, the cluster is refreshed and earlier versions are preserved in an internal audit history.
- Editorial review: Administrators can merge, divide, rename, or archive clusters from the Manage Clusters tool.
How AI is used
AI generates the neutral overview and structured coverage comparison on each story page from the linked source articles only. It prefers original reporting, official records, court documents, transcripts, datasets, and direct statements, and weights them more heavily. Every substantive assertion links to its source. Summaries regenerate when major new information arrives. AI-generated text may contain errors — always follow the links to the original reporting.
Identifying framing differences
For each article the system records the article type (reporting, analysis, opinion, fact-check, press release, live coverage), the primary facts emphasized, the tone, the framing, the topics emphasized, and notable topics absent from the coverage. These fields are surfaced in the “Broadly Reported,” “Coverage Differences,” and “Unresolved or Disputed” blocks and in the side-by-side comparison table. The platform describes framing; it never infers a motive without evidence, and it attributes opinions and allegations to those making them.
Coverage Prism and divergence
The Coverage Prism Bar visualizes how many unique outlets are covering a story and how far apart their headlines diverge. Outlet count uses unique outlets (after wire-service deduplication); headline divergence is a normalized measure of lexical and emphasis difference across the cluster’s headlines. A wider, more saturated bar signals broader and more varied coverage.
Update cadence and history
Clusters are re-evaluated as new articles arrive. The trending monitor flags clusters that reach five or more articles within a 60-minute window. Each cluster keeps a persistent URL and a history of updates so readers can see how coverage evolved.
What we do not do
We do not republish full articles, fabricate sources or quotations, present predictions as facts, infer motive, or count wire duplicates as independent confirmation. We do not allow sponsors to influence editorial analysis. “Not mentioned” means a fact appears in other credible reporting and is absent here — not that it was concealed.
Limitations and demo data
This prototype is populated with illustrative data labeled “Demo Content.” Where a live news API, paid data license, email service, or subscription processor is not yet connected, a clearly labeled integration point and demo data are used temporarily.