The aggregator I help run pulls roughly 11,000 story candidates a day from about 40 feeds, and for months the front page kept showing the same wire story three or four times — once per outlet, slightl
Two winters ago I ran the read side of a news aggregator not unlike this one: a PHP front page, a Python ranker writing into MySQL 5.7, and nginx 1.18 in front doing fastcgi_cache. The symptom was sub
Last spring I inherited a small news-ingestion service: 14 worker processes on four nodes, pulling roughly 900 feeds, writing about 120k rows a day into Postgres 14, with a Redis 6 queue in front. The