Skip to content

fix(swoole): bound the per-connection output buffer (default 512KB) - #31

Open
loks0n wants to merge 3 commits into
mainfrom
fix/bound-slow-consumer-buffer
Open

fix(swoole): bound the per-connection output buffer (default 512KB)#31
loks0n wants to merge 3 commits into
mainfrom
fix/bound-slow-consumer-buffer

Conversation

@loks0n

@loks0n loks0n commented Aug 28, 2026

Copy link
Copy Markdown

Root cause of repeated Appwrite Cloud realtime OOMKills (fra1: OnCall #1098, #1175, and
further kills on 2026-08-28 and 2026-08-31).

The bug

A websocket client that stops draining makes the reactor buffer its undelivered frames in
process memory while push() keeps returning true. Swoole's cap on that is
SW_SOCKET_BUFFER_SIZE8MB per connection (include/swoole_config.h:43), checked as
out_buffer->length() >= _socket->buffer_size in src/server/master.cc:1561.

8MB per connection is harmless for a handful of connections. For the ~1200 an Appwrite
realtime container holds it is a 9.6GB ceiling, against a 1GiB container. Measured with
300 non-draining connections in Appwrite's production image (PHP 8.5.9, Swoole 6.2.2):

worker  (pid 11)   36,152 →     36,644 kB   flat
reactor (pid  8)   73,848 →  2,344,232 kB   ← +2.27 GB

All of it in the reactor, none in the PHP worker — PHP's peak heap was 4MB against a 128M
memory_limit. That is why this never looked like a leak, and why no application-level change
moved it.

The fix

Adapter\Swoole takes a $socketBufferSize at construction, defaulting to
512KB (DEFAULT_SOCKET_BUFFER_SIZE). Exposure becomes size × connections, and push()
returns false for a connection that is over, which send() turns into a close.

Verified end to end through Server + Adapter\Swoole — 300 connections that never read, 200
events of 50KB each:

cap RSS growth stalled connections still open
0 (Swoole's 8MB) +2269 MB 301 — none shed
512KB (new default) +176 MB 1 — all 300 shed

send() acting on the false is the other half. Dropping the frame silently would leave the
client out of sync: in a capped-but-not-closed run, 37500 events vanished from clients that
stayed connected and never learned. Closing lets them reconnect and resubscribe from a known
state, which for a realtime protocol is the recoverable failure. It also stops the failure
repeating — failures fall from 37800 to 300, one per connection, because without the close
every later event re-fails against the same stuck connection forever.

Two things worth knowing for review

It must be set on the listen port, not the server. ListenPort captures
Socket::default_buffer_size when it is constructed
(include/swoole_server.h:200), while the server-level socket_buffer_size option only
mutates that static later, from start() (ext-src/php_swoole.cc:478) — by which point the
port has its own copy. An earlier revision of this PR set it server-side, and a 512KB value
measured identically to no cap at all. That is also why it is a constructor argument rather
than a setter: it has to reach the port before start().

send_yield must be off (done in the same constructor). Left on, an over-budget push
suspends and the worker accumulates the backlog against PHP's memory_limit, fatalling
the worker instead of shedding the one connection that is behind. Verified: Allowed memory size of 134217728 bytes exhausted ... Swoole\WebSocket\Server->push().

buffer_output_size is not the knob — setting it changed nothing.

Behaviour change

This changes the default, so it is a behaviour change: a client far enough behind is now
disconnected rather than having up to 8MB held on its behalf. That is deliberate — the old
default fails silently and takes the whole process with it — but it warrants a major, and
2.0.0 was cut shortly before this PR existed.

Pass socketBufferSize: 0 to restore the previous behaviour exactly.

Testing

composer lint (PSR-12) and composer check (PHPStan level max) pass. The measurements drive
a real Swoole\WebSocket\Server in Appwrite's production image rather than a stub: the failure
lives entirely in the reactor's handling of a socket that will not accept writes, which a mock
server cannot exhibit. Existing e2e coverage in tests/e2e/AdapterTest.php continues to
exercise send/broadcast/disconnect.

Consumed by appwrite/appwrite#13415.

🤖 Generated with Claude Code

A websocket client that stops draining makes the Swoole reactor buffer
every undelivered frame in process memory, without limit, while push()
keeps returning true. Reproduced in Appwrite's production image (PHP
8.5.9, Swoole 6.2.2): 300 non-draining connections fanned 2930MB and grew
RSS by 2.27GB, all of it in the reactor rather than the PHP worker, with
60000/60000 pushes reported successful. This is the cause of repeated
realtime OOMKills.

setSocketBufferSize() caps it. Memory then bounds at size x connections,
and push() returns false for a connection that is over budget:

  uncapped        RSS +2271MB   60000 ok / 0 failed
  cap 1MB         RSS  +333MB   22200 ok / 37800 failed
  cap 1MB + close RSS  +333MB   22200 ok / 300 failed, 300 shed

send() now acts on that false. Dropping the frame would leave the client
silently out of sync -- in the capped-but-not-closed run above, 37500
events vanished from clients that stayed connected. Closing lets them
reconnect and resubscribe from a known state, which for a realtime
protocol is the recoverable failure.

send_yield is disabled alongside the cap. Left on, an over-budget push
suspends and the worker accumulates the backlog against PHP's
memory_limit, which fatals the worker instead of shedding one connection.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Aug 28, 2026

Copy link
Copy Markdown

Greptile Summary

The PR bounds Swoole’s per-connection output buffering and disconnects clients whose buffers cannot accept another frame.

  • Adds a 512KB default socket-buffer limit configurable through the adapter constructor.
  • Applies the limit to the listen port before startup and disables send_yield.
  • Closes connections when push() reports failure.

Confidence Score: 5/5

The PR appears safe to merge based on the available follow-up evidence.

No blocking failure remains.

Important Files Changed

Filename Overview
src/WebSocket/Adapter/Swoole.php Adds pre-start output-buffer configuration and closes lagging connections after failed pushes; no eligible follow-up finding was established.

Reviews (3): Last reviewed commit: "fix(swoole): default the cap to 512KB an..." | Re-trigger Greptile

Swoole reads the config once, when start() hands it over, so a setter
called after that point silently does nothing. Taking it as a constructor
argument makes the value immutable and the timing unambiguous.

Default 0 leaves the buffer uncapped, so behaviour is unchanged for
existing callers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
loks0n added a commit to appwrite/appwrite that referenced this pull request Aug 31, 2026
Follows utopia-php/websocket#31 moving it from a setter to a constructor
argument: Swoole reads the config once at start(), so a later setter would
silently do nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two corrections to the previous commits.

The cap has to be set on the listen port, not the server. ListenPort
captures Socket::default_buffer_size when it is constructed
(include/swoole_server.h:200), and the server-level socket_buffer_size
option only mutates that static later, from start()
(ext-src/php_swoole.cc:478) -- by which point the port has its own copy.
Set there it silently does nothing, which is why a 512KB server-level
value measured identically to no cap at all.

The behaviour it guards against is also not "unbounded", as the earlier
message said. Swoole's default is SW_SOCKET_BUFFER_SIZE, 8MB per
connection (include/swoole_config.h:43), and the overflow check compares
out_buffer->length() against it (src/server/master.cc:1561). So exposure
is 8MB x connections: fine for a handful, 9.6GB for the ~1200 connections
an Appwrite realtime container holds.

Verified through Server + Adapter\Swoole in Appwrite's production image,
300 connections that never read, 200 events of 50KB:

  cap 0 (Swoole 8MB)   RSS +2269MB   301 connections still open
  cap 512KB (default)  RSS  +176MB     1 connection still open

Default is now 512KB rather than opt-in, so the failure mode is bounded
without every caller having to know about it. Pass 0 for the old
behaviour.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@loks0n loks0n changed the title fix(swoole): bound the per-connection output buffer fix(swoole): bound the per-connection output buffer (default 512KB) Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant