Seismic station clock drift: catch it before the waveforms lie

A station I help keep alive is a three-channel broadband feed (HHZ/HHN/HHE, 100 sps) that writes 10-minute miniSEED chunks and ships them nightly. Last spring I noticed P arrivals from a local M2.7 about 40 km away landing ~0.9 s late on HHZ relative to HN/HHE, then the lag growing to about 1.4 s over three weeks. The data looked fine in a quick plot. That is the trap: a quiet station with a drifting clock still plots fine.

My first move was not to touch the instrument. I pulled a day of chunks and measured cross-channel coherence and, more usefully, compared the station’s own event triggers against a nearby reference station using a simple arrival-difference check.

# pick one local event, extract 60 s around the theoretical S minus 5 s
obspy-dataselect -F miniSEED -S "STA1.*" -s "2024-04-11T02:17:00" -e "2024-04-11T02:18:00" > seg.mseed
# cross-correlate the vertical channels of STA1 and REF1
python3 - <<'PY'
from obspy import read
from obspy.signal.cross_correlation import correlate, xcorr_max
a = read('seg.mseed').select(id='STA1..HHZ')[0]
b = read('ref.mseed').select(id='REF1..HHZ')[0]
shift, value = xcorr_max(correlate(a.data, b.data, 2000))
print(shift/100.0, value)   # seconds, normalised coefficient
PY

Three events gave lags of 0.88, 0.91 and 1.02 s, all one sign, all growing. That is clock, not path.

The fix was boring: the box had no PPS input and its ntpd was pointed at a pool server reachable only over a saturated LTE link, with maxpoll left at default. I moved it to a GPS-disciplined source, set minpoll 4 maxpoll 6, and logged ntpq -p offsets to a file every minute. Offset fell from ~900 ms to under 2 ms in a day. The cost: more polling traffic, a GPS antenna that needs a sky view, and a handful of chunks around the switchover that I flagged rather than silently edited. I did not time-shift old data; I wrote a sidecar note per day so anyone merging the archive knows which chunks are suspect.

Where this fails: cross-correlation needs a shared, impulsive arrival. If the reference station is more than ~150 km away or the event is emergent, the lag estimate is noise. On a station with two dead channels you cannot cross-check at all; you are back to a known-location blast or a calibration pulse. I have also been burned by correlating on a clipped channel, which produced a confident 0.0 s shift and hid a real 300 ms drift for a week.

The thing I keep relearning: check timing before you interpret any travel-time residual. What is your cheapest routine check that catches a slow clock drift before it contaminates a catalog?