A Cron Job That Fired Twice: Notes From an Ingestion Pipeline

Last spring I inherited a small news-ingestion service: 14 worker processes on four nodes, pulling roughly 900 feeds, writing about 120k rows a day into Postgres 14, with a Redis 6 queue in front. The symptom was banal and infuriating: every few days, a handful of stories appeared twice in the aggregation feed, always within a 20-second window, never reproducible on a laptop.

The queue was the usual sorted-set-plus-hash setup. A worker pulled a feed URL out of the ready set and set a lease key:

# worker loop, simplified
item = r.zpopmin('ready', count=1)
if not item:
    continue
feed_url, score = item[0]
lease = f'lease:{feed_url}'
if not r.set(lease, worker_id, nx=True, ex=45):
    continue                       # someone else owns it
if not r.set(f'hash:{feed_url}', 'processing', xx=True):
    r.delete(lease)
    continue
try:
    ingest(feed_url)
finally:
    r.delete(lease)

The bug was in zpopmin(..., count=1): it returned what the server thought was the lowest item, but with replica reads enabled for the ready set, a worker could be handed a URL that a primary-side worker had already popped milliseconds earlier. The lease normally caught this — except when the second writer’s set landed after the first worker’s delete in the finally. Then both ran ingest and both wrote rows with different fetched_at values, so the dedupe index on (story_id, fetched_at::date) did not help.

What I did, in order, because I did not want to rewrite the queue:

1. Stopped reading ready from replicas (READONLY was being set per-connection by an old wrapper). That alone cut duplicates by maybe 80%.
2. Replaced the lease delete with a compare-and-delete Lua script so a worker cannot delete another worker’s lease:

-- kev: lease key, argv[1]: worker id
if redis.call('GET', KEYS[1]) == ARGV[1] then
  return redis.call('DEL', KEYS[1])
end
return 0

3. Made the insert idempotent with INSERT ... ON CONFLICT (story_id, source_id) DO NOTHING, and moved fetched_at out of the uniqueness constraint; it is metadata, not identity.

Why this shape: the lease is a correctness mechanism, so it has to live on the same node that hands out work. Reading the ready set from a replica bought us latency headroom we did not need — the p99 of zpopmin was 0.4 ms against the primary — while quietly breaking mutual exclusion. The compare-and-delete costs one extra round trip per job, about 3% throughput on our numbers; the ON CONFLICT clause costs us the ability to record every re-fetch, which I decided was fine because we already keep raw payloads in object storage.

Where this fails: if a worker is partitioned from Redis but not from the database, the lease expires and a second worker starts while the first is still writing. The compare-and-delete does not save you there; only a fencing token checked at write time does. I have not implemented that yet, and I know it. Also, ON CONFLICT DO NOTHING hides genuine upstream changes — a corrected headline looks like a duplicate. We now compare a content hash and update instead of skipping, which reintroduced a smaller version of the same race for about a week.

One thing I still cannot explain cleanly: the duplicate window clustered around 45 seconds, which is exactly the lease TTL, suggesting some jobs were taking longer than the lease without the worker noticing. If you have run a lease-based queue where the renewal path was an afterthought, how did you detect the slow jobs before users did?