PHP-FPM's process_control_timeout Killed Our Graceful Reloads

这篇内容暂时只有 English 版本,以下是原文。

PHP 8.1, Debian 11, PHP-FPM behind nginx, 24 workers per pool across 6 app servers. Every deploy ran a graceful reload: send SIGUSR2 to the master, let workers finish in-flight requests, spin up new ones. It had worked for a year. Then we added an endpoint that streams a CSV export — a few hundred thousand rows, pulled from a report query, written to php://output as it goes. Some of those requests run 90 to 200 seconds. After that change, roughly one deploy in three left a small pile of clients seeing 502s, and the master log showed workers being SIGKILL’d during the reload window.

What I found was that process_control_timeout defaults to 0, which in practice means the master gives workers no grace period at all when it needs them gone. The setting is a per-pool directive; ours had never been set. So on SIGUSR2 the master asks the old workers to finish and exit, and when it decides it has waited long enough — or when it needs to reclaim them for the new generation — it escalates. With long-lived streaming requests, that escalation lands on a worker mid-export.

What I changed, in the pool config:

; pool.d/www.conf
process_control_timeout = 30s
request_terminate_timeout = 240s

And in the deploy script, I stopped treating the reload as instant. The important part is what we send and when we consider the deploy done:

# graceful reload, then wait for old workers to drain
kill -USR2 "$(cat /run/php/php8.1-fpm.pid)"
for i in $(seq 1 60); do
  old=$(pgrep -f 'pool www' -P "$(cat /run/php/php8.1-fpm.pid)" | wc -l)
  [ "$old" -eq 0 ] && break
  sleep 2
done

That loop isn’t a health check; it’s a drain wait. It tells the deploy pipeline not to declare success while old workers are still serving streaming responses. The 30s value is a deliberate ceiling: a request that hasn’t finished in 30s past the reload gets cut, and that’s a compromise I accepted rather than let the master hold a worker hostage indefinitely.

Why it works: process_control_timeout is the master’s patience for old workers during a reload. Setting it gives them a bounded window to reach a safe point instead of being killed the moment the master wants them gone. The cost is real — workers can linger up to that timeout, and with 24 workers you briefly run double the process count during the overlap. Memory headroom on those boxes had to be checked; on the smallest server we dropped to 20 workers to keep the overlap within RAM.

Where it falls apart: this does nothing for requests that genuinely need more than 30s. Our export endpoint still gets cut if a reload coincides with a slow report query, so I later moved that endpoint to a queue job — the streaming response now returns an ID and the browser polls. That is the honest fix. Also, the pgrep pattern is brittle; it matched a differently named pool on one host and the loop exited immediately, which I only caught because the deploy felt too fast. If your pool names aren’t uniform, match on the master PID’s children instead of a string. And request_terminate_timeout interacts with your own time limits — set it below whatever your proxy allows, or nginx will report a 504 that looks like a PHP-FPM problem when it isn’t.

One thing I’m still not sure about: whether a longer process_control_timeout is ever the right call, or whether any request long enough to need one should have been off the web tier in the first place. What did you do with your long-running HTTP handlers — drain them, or move them?

0
赞同度 100% · 0 条评论 · 1 阅读

0 条评论

支持 Markdown · 回复会实时推送给对方

还没有评论,写下第一条。