French date formats broke our logs, so we pinned the locale

A service I help maintain runs on a small VPS in Roubaix, Debian 12, Python 3.11, logging to journald and shipping rotated files to a Grafana/Loki box. Nothing exotic. Around the time we added a second app instance, log lines coming from that box started carrying dates like 12/03/2024 14:07 while the old instance emitted 2024-03-12T14:07:31. Same code, same requirements.txt. The mismatch was enough to break our Loki parsers, which had been written against ISO timestamps. We only noticed because a dashboard panel went blank for one instance and someone asked whether the node was down.

The cause was locale, not code. The new box was provisioned from an image that set LANG=fr_FR.UTF-8, and something in the stack (a third-party lib formatting dates via strftime('%x')) honored it. We could have patched the parser, but I’d rather not have two timestamp dialects in one log store. So we pinned it explicitly at the process level and in the unit file:

# /etc/systemd/system/app.service
[Service]
Environment=LANG=C.UTF-8 LC_ALL=C.UTF-8 TZ=Europe/Paris

And for good measure, in the app’s entrypoint:

import locale, os, time
os.environ.setdefault("TZ", "Europe/Paris")
time.tzset()
locale.setlocale(locale.LC_TIME, "C.UTF-8")

We kept TZ=Europe/Paris rather than UTC because the humans reading the raw logs are in France and DST switches twice a year — better to be explicit than to rely on the host’s /etc/localtime being right, which it wasn’t on that image.

The trade-off: C.UTF-8 loses French month names in any user-facing strings, so we format those in the application layer with babel instead of relying on the process locale. That’s more code but predictable. It also means anyone who shells in and runs date sees the system locale, not ours; we’ve had one confused debugging session from that already.

Where this fails: if a dependency calls setlocale(LC_ALL, '') at import time, it will undo our pin. We hit that with an old PDF library and ended up wrapping its call site. Also, LC_ALL=C.UTF-8 will bite you if you ever sort accented strings with sort — café and cafe behave differently than under fr_FR. And if you run containers, the base image’s /etc/localtime can still leak in via mounted volumes, so check that too.

The question I keep coming back to: do you pin the locale per service, or do you standardize the host image and trust it? I’ve been doing per-service because the images drift, but it’s more duplication than I’d like.