The PHP 8.2 Fiber That Blocked on flock and Stalled 60 Workers

We run a PHP 8.2 worker pool on Kubernetes, 60 pods, each processing image derivatives from an S3-style bucket. Each job: download, resize with GD, write back. Throughput was fine—about 40 jobs/sec across the pool—until we added a per-file lock so two workers could not resize the same source twice after a retry. That change looked harmless. Two days later, pod CPU dropped to near zero and queue depth went from ~200 to 30k while workers reported healthy.

The lock was a flock on a local temp file keyed by object hash:

$fp = fopen($lockPath, 'c');
if (!flock($fp, LOCK_EX | LOCK_NB)) {
    throw new RuntimeException('locked');
}
try {
    return $this->resize($job);
} finally {
    flock($fp, LOCK_UN);
    fclose($fp);
}

The retry handler caught RuntimeException, waited 250ms, and retried up to 20 times. That was the first problem: LOCK_NB failing is not an error, it is backpressure, and we turned it into a retry storm. The second problem was subtler. We had just started using Fibers for the download step, so the resize ran inside Fiber::suspend()/resume(). When a fiber suspended while holding the file handle, the finally block ran only on normal completion. On cancellation, the flock was released when the process exited, not when the fiber was abandoned. Workers held locks for jobs they had already dropped.

What fixed it was not a bigger timeout. We moved the lock out of the fiber entirely, into a small try/finally around the whole job in the worker loop, and made the lock acquisition blocking with a deadline instead of non-blocking with retries:

$deadline = microtime(true) + 5.0;
while (!flock($fp, LOCK_EX | LOCK_NB)) {
    if (microtime(true) > $deadline) {
        throw new RuntimeException('lock timeout');
    }
    usleep(50_000);
}

That gave us an explicit ceiling and removed the retry amplifier. The mechanism matters: flock is tied to the file description, not to the PHP call stack, so fiber suspension does not release it. The cost is that we now serialize per object hash across the whole pod, which is fine at 60 pods but would not scale to one pod doing thousands of distinct hashes. It also does not protect across pods—our temp dir is pod-local.

Where this fails: if the lock file lives on NFS or a shared volume, flock semantics depend on the mount and can block the whole process. We also saw a case where a worker was OOM-killed mid-hold, and the lock persisted until the temp file was cleaned. And if you use pcntl_fork, the child inherits the open description—closing it in the parent does not release the lock.

What does your worker pool do when a lock is already held: fail fast, queue, or spin? I am curious whether anyone has a cleaner cancellation story for fibers holding OS resources.