Deduping 11k Stories a Day Without Killing the Feed

The aggregator I help run pulls roughly 11,000 story candidates a day from about 40 feeds, and for months the front page kept showing the same wire story three or four times — once per outlet, slightly different headline, different URL, same body text. This was on Postgres 15, one 4-core box, candidates table around 9M rows and growing ~11k/day. The dedupe job ran every 10 minutes and had drifted to 6+ minutes per run, mostly because of a LIKE '%...%' match on titles that no index could help with.

What I did was stop trying to be clever about headline similarity and match on what actually repeats: content. I added a content_hash column computed at ingest from a normalized body (strip HTML, collapse whitespace, lowercase, drop a trailing source attribution line), plus a title_shingle column for near-duplicates.

ALTER TABLE candidates ADD COLUMN content_hash bytea;
ALTER TABLE candidates ADD COLUMN title_shingle bigint[];
CREATE INDEX ON candidates (content_hash);
CREATE INDEX ON candidates USING gin (title_shingle);

The hash in Python, before insert:

import hashlib, re

def normalize(body: str) -> str:
    text = re.sub(r'<[^>]+>', ' ', body)
    text = re.sub(r'\s+', ' ', text).strip().lower()
    text = re.sub(r'\s*[-—]\s*[a-z0-9 .]{3,30}$', '', text)
    return text

def content_hash(body: str) -> bytes:
    return hashlib.blake2b(normalize(body).encode(), digest_size=16).digest()

Exact dupes collapse on the hash. For near-dupes I build 3-word shingles over the title, keep the 32 smallest hashes, and use Postgres array overlap (&&) against the gin index. Anything overlapping on 8+ of 32 shingles gets flagged for a human, not auto-merged.

The reason for two mechanisms is that they fail differently. The content hash is cheap and exact but useless when one outlet rewrites the lede — same event, different bytes. Shingles catch that, at the cost of false positives on recurring phrases (“ceasefire talks resume” hits every few weeks). That’s why shingles only flag; auto-merge only ever happens on an exact hash match within a 48-hour window.

UPDATE candidates c SET duplicate_of = d.id
FROM candidates d
WHERE c.content_hash = d.content_hash
  AND c.id > d.id
  AND c.fetched_at BETWEEN d.fetched_at - interval '48 hours'
                       AND d.fetched_at + interval '48 hours'
  AND c.duplicate_of IS NULL;

Where it breaks: paywalled feeds ship a stub body, so ten different stories hash to the same near-empty string. I now skip hashing when normalized length < 400 chars and let those through. Also, live blogs mutate in place, so the same URL re-hashes differently every hour — I key those on URL instead. And the 48-hour window is a guess; a two-week-old story resurfacing after an update slipped past it twice before I widened it for a handful of slow feeds.

Runtime dropped from ~6 minutes to ~40 seconds, and front-page duplicates went from visible daily to rare enough that readers stopped complaining. The shingle flag queue is the part I still don’t trust — false positives run maybe 1 in 12, and I haven’t found a threshold that beats reading them.

If you’ve run near-dup detection on short text at this volume, what did you use for the similarity step, and did you ever get comfortable auto-merging on anything short of an exact hash?