A Paris datacenter move broke our cron jobs, not our servers

We moved a small internal service from one Paris-region provider to another in March. Same city, same nominal latency, same Debian 12 image, Node 20.11, a single Postgres 15.6 instance with about 40 GB of data and roughly 120 requests per minute. The cutover looked clean: DNS TTL was low, health checks were green, p99 stayed under 60 ms. What broke was everything scheduled.

The old host ran systemd timers in Europe/Paris. The new host’s base image had /etc/localtime pointing at UTC. Nobody noticed because the application logs were already in UTC, so the dashboards looked identical. The first sign was a daily aggregation job running at 02:00 UTC instead of 02:00 Paris time, which during the March DST window meant it overlapped with a backup that nobody had touched in a year.

What we did, in order:

# confirm what the host actually believes
timedatectl
# check the timer itself, not just the service
systemctl list-timers --all | grep -E 'aggregate|backup'
systemctl cat aggregate.timer

The timer had OnCalendar=*-*-* 02:00:00 with no timezone suffix, so it inherited the host’s zone. We fixed the unit rather than the host, because pinning the host was what hid the problem before:

[Timer]
OnCalendar=*-*-* 02:00:00 Europe/Paris
Persistent=true

Then systemctl daemon-reload and a systemctl restart aggregate.timer. We did the same for the backup timer and for a cleanup job that had silently been running an hour late since the move. We also added a check to our deploy script that fails if timedatectl show -p Timezone --value is not Europe/Paris, because the next image will probably reset it again.

Why not just set the host timezone and move on? Because the unit file is the thing under version control. The host image is disposable; the timer definition is not. Putting the zone in OnCalendar makes the schedule survive a rebuild in any region, and it makes the intent readable to whoever opens the file next. The cost is that you now have two sources of truth if someone also sets the host zone, and you have to remember that Persistent=true will fire a missed run immediately after boot, which is not always what you want for a job that talks to an external API with rate limits.

Where this falls apart: OnCalendar with an explicit zone does not help if the job itself formats dates using the process locale. We had a CSV export that used toLocaleDateString('fr-FR') and wrote 01/03/2024, which our downstream parser read as January 3rd. That one we fixed by writing ISO 8601 and leaving locale out of storage entirely. Also, if you run in containers, the timer zone and the container’s TZ can disagree, and Persistent=true on a short-lived container is close to meaningless.

We caught this because a human noticed a report was an hour off, not because monitoring did. I still do not have a good alert for “job ran at the wrong wall-clock time but succeeded.” If you have one that is not just comparing timestamps in a dashboard, I would like to hear how you set the expected window.