Persistent broken pipe errors (~1-2% of requests)

[sid-stripe]

Description

We're seeing persistent WriteFailedException: Failed to write request to socket [broken pipe] errors at a ~1-2% error rate across all Lambda instances. Bref recovers by restarting FPM, but the current invocation returns a 500.

The error pattern on a single Lambda instance:

1. Request N completes normally (e.g., 220ms)
2. Request N+1 arrives ~6ms later
3. WriteFailedException immediately — the FPM socket is already dead
4. Bref restarts FPM, next request works

Errors also spontaneously drop to near-zero (~0.01%) for 7-14 hours without any deployment, then climb back to ~1-2%. This suggests state corruption that self-heals through Lambda instance recycling.

How to reproduce

We haven't found a reliable reproduction — it's intermittent and only manifests in production under real traffic. It has persisted since our initial Bref migration and across PHP 8.1 and 8.4.

• Bref: 2.4.18 (Docker images, not layers)
• Docker image: bref/php-84-fpm:2
• PHP: 8.4.18
• AWS service: Lambda (via API Gateway HTTP API)
• Lambda config: 1024 MB memory, 28s timeout
• FPM config: Bref defaults (pm=static, max_children=1, log_limit=8192)
• Extensions: redis (Predis pure PHP client), intl, opcache, pcntl, posix, pdo_mysql
• Framework: Laravel 10

What we've ruled out

• SIGPIPE (PHP-FPM children die with SIGPIPE since 8.3.10 #1854) — zero matches in CloudWatch
• JIT segfaults (FastCgiCommunicationFailed on every second request #842) — opcache.jit = disable, verified from image
• OOM — memory well within Lambda limits
• Excimer profiler — removed from Dockerfile entirely, errors persist
• Sentry tracing/profiling — fully disabled, errors persist
• Redis/Valkey — Predis runs in child only, master never touches it
• Large responses — no correlation between response size and errors
• stderr log_limit — max log message is ~2KB, well under the 8192 limit
• Cold starts — orders of magnitude fewer than error count
• PHP version — reproduced on both 8.1 and 8.4

Questions

• We noticed the catch block in FpmHandler.php (line 162) calls $this->stop() before capturing
  proc_get_status($this->fpm), so the exit code and termination signal are lost. Would you accept a PR adding
  this logging on the failure path?

[mnapoli]

@sid-stripe thank you for this very detailed issue, it is really great, especially everything that you ruled out or tried.

Yes, feel free to send PRs to add more logging for these scenarios, they are the worst to debug, so any extra log that doesn't impact the happy path is welcome.

Do you have any theories on what could cause the FPM master to die silently? We're happy to run any
diagnostic steps or share additional logs/data.

You already explored a lot of options already, I'm not sure what more to suggest to be honest. One thing that comes to mind is to check anything pcntl/posix-related (even though I would expect the FPM worker to cause the issue, not the parent, but 🤷). For example are you using signals in the application? E.g. I tried in the past to use signals, this didn't work well with FPM: #895 (comment) And make sure you don't "fork", or since you use Laravel make sure you don't use anything that could fork or set up signals (stuff that runs "after the response is sent" on a server, e.g. the defer() function, things like that).

I also released Bref v3 last week, I know it's probably not something you want to try here but considering all the low-level changes in the image (e.g. Amazon Linux 2023 and system library changes) it could be worth a try if that's an option.

Is there any known interaction between Lambda freeze/thaw and FPM? Could the cgroup freezer leave the master in a broken state after thaw?

Not that I know of. I've seen problematic scenarios where Lambda would timeout before the FPM child would timeout, which would lead to the child worker still "working" on the next request, causing FPM to be busy and things to be borked. I'm not saying this is what is happening here, but just mentioning in case it gives you some ideas.

[sid-stripe]

Thanks for the pointers. I will do some more debugging and get back

[sid-stripe]

Hey @mnapoli!

Update on our investigation — I added proc_get_status() logging to the catch block and found that the FPM master is actually still alive when the broken pipe occurs:

• proc_get_status() shows running=true, signaled=false, exitcode=-1. The master is not crashing.
• The FPM worker or socket is dead. The write fails because the reader end (worker) has closed its end of the Unix socket, likely due to the worker dying between invocations.
• The error is transient. After Bref restarts FPM, the very next request to the same Lambda instance succeeds. The current code already does the right thing (restart FPM) but then discards the recovery by throwing FastCgiCommunicationFailed.

Since WriteFailedException means the request never reached PHP-FPM (the write to the socket failed), it's safe to retry once against the freshly restarted process. We've been running this patch in production and broken pipes now recover transparently instead of returning 500s.

I've opened a draft PR with this fix: #2081 — happy to get your thoughts on it.