When opcache.validate_timestamps Bit Us In Staging

Staging ran PHP 8.2 on FPM with a shared NFS mount for releases, ~180 files per deploy, and opcache.validate_timestamps=0. Deploys were atomic symlink swaps, so we assumed nothing needed revalidation. Then we spent two days chasing “the fix isn’t live.” A deploy would go out, curl would show old templates, and php -r "var_dump(opcache_get_status()['scripts']);" on one worker showed the old inode cached while another worker had the new one. Same code, different behavior depending on which FPM child handled the request.

The cause is that opcache keys cached scripts by the resolved realpath. With symlink swaps, current/index.php resolves to releases/2024-06-11/index.php, which is a different path than yesterday’s release. That part should invalidate correctly. What bit us was the opposite: a hotfix that edited a file in place on the shared mount. With validate_timestamps off, no worker ever noticed the mtime change. The symlink swap was fine; the in-place edit was the landmine.

My fix was not to turn validation back on (that adds a stat per request on every included file and hurt our p99). Instead I made the deploy do the invalidation explicitly:

# after symlink swap, per pool
for pid in $(pgrep -f 'php-fpm: pool app'); do
  kill -USR2 "$pid"
done

SIGUSR2 tells FPM to reload its opcache in place, no restart, no dropped connections. We also added a guard in CI that fails the build if a release directory contains a file modified after the build started, which catches the in-place hotfix habit before it reaches staging.

The tradeoff is real: every deploy now touches every worker, and on a 12-worker box that is ~300ms of added latency while opcache repopulates. We accepted that for correctness. If your workers are on separate hosts, you need a fan-out, and that fan-out is a new thing to get wrong.

Where this falls apart: opcache with file_cache enabled still reads that secondary cache, and USR2 invalidates it too, so the first request after reload can be a cold start that reads from disk. If your NFS mount is slow, that first request after deploy can climb. I saw one staging p99 hit 1.4s right after reload before it settled. Also, if anyone edits a file inside a release directory directly, this whole scheme is silent again. The CI guard is the only thing standing between you and that.

Curious how others handle invalidation when releases live on a shared mount: do you fan out USR2 per host, or do you just run validate_timestamps everywhere and eat the stat cost?