fix(swoole): bound the per-connection output buffer (default 512KB) - #31
Open
loks0n wants to merge 3 commits into
Open
fix(swoole): bound the per-connection output buffer (default 512KB)#31loks0n wants to merge 3 commits into
loks0n wants to merge 3 commits into
Conversation
A websocket client that stops draining makes the Swoole reactor buffer every undelivered frame in process memory, without limit, while push() keeps returning true. Reproduced in Appwrite's production image (PHP 8.5.9, Swoole 6.2.2): 300 non-draining connections fanned 2930MB and grew RSS by 2.27GB, all of it in the reactor rather than the PHP worker, with 60000/60000 pushes reported successful. This is the cause of repeated realtime OOMKills. setSocketBufferSize() caps it. Memory then bounds at size x connections, and push() returns false for a connection that is over budget: uncapped RSS +2271MB 60000 ok / 0 failed cap 1MB RSS +333MB 22200 ok / 37800 failed cap 1MB + close RSS +333MB 22200 ok / 300 failed, 300 shed send() now acts on that false. Dropping the frame would leave the client silently out of sync -- in the capped-but-not-closed run above, 37500 events vanished from clients that stayed connected. Closing lets them reconnect and resubscribe from a known state, which for a realtime protocol is the recoverable failure. send_yield is disabled alongside the cap. Left on, an over-budget push suspends and the worker accumulates the backlog against PHP's memory_limit, which fatals the worker instead of shedding one connection. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Greptile SummaryThe PR bounds Swoole’s per-connection output buffering and disconnects clients whose buffers cannot accept another frame.
Confidence Score: 5/5The PR appears safe to merge based on the available follow-up evidence. No blocking failure remains. Important Files Changed
Reviews (3): Last reviewed commit: "fix(swoole): default the cap to 512KB an..." | Re-trigger Greptile |
Swoole reads the config once, when start() hands it over, so a setter called after that point silently does nothing. Taking it as a constructor argument makes the value immutable and the timing unambiguous. Default 0 leaves the buffer uncapped, so behaviour is unchanged for existing callers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
loks0n
added a commit
to appwrite/appwrite
that referenced
this pull request
Aug 31, 2026
Follows utopia-php/websocket#31 moving it from a setter to a constructor argument: Swoole reads the config once at start(), so a later setter would silently do nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two corrections to the previous commits. The cap has to be set on the listen port, not the server. ListenPort captures Socket::default_buffer_size when it is constructed (include/swoole_server.h:200), and the server-level socket_buffer_size option only mutates that static later, from start() (ext-src/php_swoole.cc:478) -- by which point the port has its own copy. Set there it silently does nothing, which is why a 512KB server-level value measured identically to no cap at all. The behaviour it guards against is also not "unbounded", as the earlier message said. Swoole's default is SW_SOCKET_BUFFER_SIZE, 8MB per connection (include/swoole_config.h:43), and the overflow check compares out_buffer->length() against it (src/server/master.cc:1561). So exposure is 8MB x connections: fine for a handful, 9.6GB for the ~1200 connections an Appwrite realtime container holds. Verified through Server + Adapter\Swoole in Appwrite's production image, 300 connections that never read, 200 events of 50KB: cap 0 (Swoole 8MB) RSS +2269MB 301 connections still open cap 512KB (default) RSS +176MB 1 connection still open Default is now 512KB rather than opt-in, so the failure mode is bounded without every caller having to know about it. Pass 0 for the old behaviour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause of repeated Appwrite Cloud realtime OOMKills (fra1: OnCall #1098, #1175, and
further kills on 2026-08-28 and 2026-08-31).
The bug
A websocket client that stops draining makes the reactor buffer its undelivered frames in
process memory while
push()keeps returningtrue. Swoole's cap on that isSW_SOCKET_BUFFER_SIZE— 8MB per connection (include/swoole_config.h:43), checked asout_buffer->length() >= _socket->buffer_sizeinsrc/server/master.cc:1561.8MB per connection is harmless for a handful of connections. For the ~1200 an Appwrite
realtime container holds it is a 9.6GB ceiling, against a 1GiB container. Measured with
300 non-draining connections in Appwrite's production image (PHP 8.5.9, Swoole 6.2.2):
All of it in the reactor, none in the PHP worker — PHP's peak heap was 4MB against a 128M
memory_limit. That is why this never looked like a leak, and why no application-level changemoved it.
The fix
Adapter\Swooletakes a$socketBufferSizeat construction, defaulting to512KB (
DEFAULT_SOCKET_BUFFER_SIZE). Exposure becomessize × connections, andpush()returns
falsefor a connection that is over, whichsend()turns into a close.Verified end to end through
Server+Adapter\Swoole— 300 connections that never read, 200events of 50KB each:
0(Swoole's 8MB)send()acting on thefalseis the other half. Dropping the frame silently would leave theclient out of sync: in a capped-but-not-closed run, 37500 events vanished from clients that
stayed connected and never learned. Closing lets them reconnect and resubscribe from a known
state, which for a realtime protocol is the recoverable failure. It also stops the failure
repeating — failures fall from 37800 to 300, one per connection, because without the close
every later event re-fails against the same stuck connection forever.
Two things worth knowing for review
It must be set on the listen port, not the server.
ListenPortcapturesSocket::default_buffer_sizewhen it is constructed(
include/swoole_server.h:200), while the server-levelsocket_buffer_sizeoption onlymutates that static later, from
start()(ext-src/php_swoole.cc:478) — by which point theport has its own copy. An earlier revision of this PR set it server-side, and a 512KB value
measured identically to no cap at all. That is also why it is a constructor argument rather
than a setter: it has to reach the port before
start().send_yieldmust be off (done in the same constructor). Left on, an over-budget pushsuspends and the worker accumulates the backlog against PHP's
memory_limit, fatallingthe worker instead of shedding the one connection that is behind. Verified:
Allowed memory size of 134217728 bytes exhausted ... Swoole\WebSocket\Server->push().buffer_output_sizeis not the knob — setting it changed nothing.Behaviour change
This changes the default, so it is a behaviour change: a client far enough behind is now
disconnected rather than having up to 8MB held on its behalf. That is deliberate — the old
default fails silently and takes the whole process with it — but it warrants a major, and
2.0.0 was cut shortly before this PR existed.
Pass
socketBufferSize: 0to restore the previous behaviour exactly.Testing
composer lint(PSR-12) andcomposer check(PHPStan level max) pass. The measurements drivea real
Swoole\WebSocket\Serverin Appwrite's production image rather than a stub: the failurelives entirely in the reactor's handling of a socket that will not accept writes, which a mock
server cannot exhibit. Existing e2e coverage in
tests/e2e/AdapterTest.phpcontinues toexercise send/broadcast/disconnect.
Consumed by appwrite/appwrite#13415.
🤖 Generated with Claude Code