We run a small fleet of application servers in a single French datacenter, mostly Debian 12, with logs shipped to a central collector that does correlation on timestamps. Everything had been fine for about fourteen months. Then one Tuesday a support ticket came in where a user’s session appeared to end before it started, and the audit trail showed a password reset two minutes in the future. The drift was not the same on every host: two boxes were about 90 seconds ahead, one was 40 seconds behind, the rest were within a few hundred milliseconds. Log volume is modest, maybe 30 GB/month, so we noticed it late because nothing had actually broken — the timestamps were just quietly lying.
The cause was chrony pointing at a single stratum-2 server that our hosting provider publishes for France, plus makestep limited to the first three updates. That server had been having a bad month. On the two fast hosts, chrony had been refusing to step because the offset was under the panic threshold but above what it would correct gradually during business hours, and our maxslewrate was conservative. What I actually changed was not exotic:
# /etc/chrony/chrony.conf
pool fr.pool.ntp.org iburst maxsources 4
makestep 1.0 -1
maxslewrate 83333
logdir /var/log/chrony
log measurements statistics tracking
makestep 1.0 -1 means step whenever the offset exceeds one second, regardless of how many updates have happened. That is aggressive, and it is the tradeoff: on a machine with a monotonic clock assumption in application code, a backwards step can confuse timers. We accepted it because these hosts do not run anything that depends on wall-clock monotonicity across a step, and because a 90-second lie is worse than a one-off jump. Setting maxsources 4 was the real fix — one server is a single point of truth, and a pool gives chrony something to vote with. I also enabled measurement logging, which is how I confirmed afterwards that the offsets were actually converging rather than just looking plausible.
Where this falls apart: if your provider blocks outbound UDP 123, the pool is useless and you will not see it in the config, only in chronyc sources -v showing nothing reachable. I have also seen containers inherit the host clock but run their own chrony, which is a fight you should not start — leave time to the host. And if you do step backwards on a host running a database, expect WAL and heartbeat logic to have opinions about it. We excluded the two Postgres boxes from the aggressive makestep and gave them a wider tolerance instead.
What I still do not know is whether the provider’s server was actually misconfigured or whether our upstream path to it was the problem, because by the time I looked the offset had corrected itself. If you run hosts in France, what does your chronyc tracking actually say right now, and how many sources are behind it?
0 条评论
还没有评论，写下第一条。