The Night Our PHP Cron Overlapped and Double-Charged 3,000 Subscriptions

PHP 8.1, Ubuntu 22.04, php-fpm plus a systemd timer that ran bin/console billing:charge-due every five minutes. Table subscriptions had about 400k rows, maybe 3k due on any given night. The job took 40-90 seconds normally, so the timer’s AccuracySec=1s seemed harmless. Until a night when the payments API got slow, the run took 11 minutes, and the next two timer firings started before the first finished. We charged a chunk of subscriptions twice. The API’s idempotency key was the invoice ID, and we were generating a fresh invoice row per run, so every duplicate run looked like a brand-new charge.

What fixed it was not a bigger interval. I put a lock around the whole worker, using MySQL’s GET_LOCK because we already had a connection and I did not want to add Redis just for this:

$pdo->exec("SET SESSION lock_wait_timeout = 1");
$stmt = $pdo->prepare("SELECT GET_LOCK(:name, 0)");
$stmt->execute(['name' => 'billing:charge-due']);
if ((int) $stmt->fetchColumn() !== 1) {
    fwrite(STDERR, "already running, exiting\n");
    exit(0);
}

register_shutdown_function(function () use ($pdo) {
    $pdo->prepare("SELECT RELEASE_LOCK(:name)")
        ->execute(['name' => 'billing:charge-due']);
});

GET_LOCK with zero timeout returns immediately, so an overlapping run exits instead of queueing up. The shutdown function matters because the lock is tied to the session; if the process dies without releasing, the connection closes and MySQL drops it anyway. I also changed the idempotency key to a hash of subscription_id + billing_period_start, so even a duplicate run cannot create a second charge.

The trade-off: GET_LOCK is per MySQL server, so this only works while all workers point at the same database. If you shard billing across databases, you need one lock per shard or something external. It also serialises the job completely, which is fine at 90 seconds and painful if a run ever takes an hour.

Where it bit me: the first version used SET SESSION lock_wait_timeout but a pooled connection had been reused with a previous value, and I spent an evening confused about why a second run waited 30 seconds instead of exiting. Also, exit(0) inside a Symfony command skips finally blocks, so any cleanup you rely on will not run.

Does anyone here use a filesystem lock (flock) instead of a database lock for this, and how do you handle the case where the lock file lives on NFS?