When a USGS Feed Update Quietly Renamed My Stations

I run a small ShakeMap-adjacent pipeline on a Raspberry Pi 4 (8 GB, Debian 12, Python 3.11) that pulls the USGS GeoJSON summary feeds every five minutes and writes a rolling CSV I use for a local dashboard. Around 40k events accumulated over about three years. Two weeks ago I noticed my station-to-event join had silently dropped roughly 9% of rows for events older than ~2019, and a handful of recent events had null mag where the feed clearly showed one.

The cause was not the feed. It was my own normalization layer. I had been keying events on properties.ids, splitting the comma-separated string and taking the first token as canonical. That works most of the time, but the ordering of that list is not stable across feed updates, and some events carry network-prefixed ids (us, ci, nc) that I was stripping inconsistently. My join then matched on a truncated id and quietly produced nothing rather than failing loudly.

What fixed it was making the canonical key explicit and never derived from list order:

import re

def canonical_id(props):
    raw = props.get("ids", "").strip(",")
    if not raw:
        return None
    tokens = [t.strip() for t in raw.split(",") if t.strip()]
    # prefer the authoritative network id, fall back to first
    for t in tokens:
        if t.startswith(("us", "ci", "nc", "ak", "hv")):
            return t
    return tokens[0]

def parse_mag(props):
    m = props.get("mag")
    try:
        return float(m)
    except (TypeError, ValueError):
        return None  # do not coerce to 0.0

I also added an assertion at ingest that every event has a non-null canonical_id and a mag that parses to float or None, and I started writing rejected rows to a separate quarantine.jsonl instead of dropping them. That file is now the first thing I check when counts look off.

The mechanism matters here: GeoJSON summary feeds are a convenience view, not a stable schema contract. Field presence varies by event and by network, and the ids list is a set of aliases, not an ordered primary key. Treating it as ordered is the bug. The cost of fixing it this way is that I now carry a quarantine path and a slightly slower ingest loop, which at 40k rows is irrelevant but would matter if I were backfilling millions.

Where this breaks: events with no network-prefixed id at all (some regional feeds) fall through to tokens[0], which is exactly the unstable case I was trying to avoid. I have seen that on a few hundred older events in my set, and I have not found a clean rule for them yet. Also, my prefix list is hand-maintained, so a new network will silently miss the preferred branch. And coercing missing magnitude to 0.0 was a mistake I made early on; it made a mag filter look healthy while hiding nulls, and I only caught it because a plot had a suspicious floor at zero.

If you are keying on ids from any summary feed, what does your canonical key actually point at, and have you checked how often the alias order changes for the events you care about?