-
Notifications
You must be signed in to change notification settings - Fork 686
fix(socket-mode): tear down leaked sockets on disconnect (#2709) #2710
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
c90c987
test(socket-mode): add failing tests reproducing socket leak (#2709)
WilliamBergamin dbb2015
fix(socket-mode): tear down leaked sockets on disconnect (#2709)
WilliamBergamin bff860a
Merge branch 'main' into fix-issue-#2709
WilliamBergamin 3341c4e
chore(socket-mode): extract buildDefaultDispatcher() and drop redunda…
WilliamBergamin 86cd21c
docs(socket-mode): tighten dispatcher option JSDoc
WilliamBergamin e4936e4
chore(socket-mode): reframe changeset as the #2709 socket-leak fix
WilliamBergamin 4962460
refactor(socket-mode): rename connect callback param to `callback`
WilliamBergamin 654a27b
test(socket-mode): cover buildDefaultDispatcher() connect hook
WilliamBergamin 360516d
Merge branch 'main' into fix-issue-#2709
WilliamBergamin File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| --- | ||
| "@slack/socket-mode": patch | ||
| --- | ||
|
|
||
| Fix leaked TCP sockets on disconnect ([#2709](https://github.com/slackapi/node-slack-sdk/issues/2709)). Moving from `ws` to undici's `WebSocket` in 3.0.0 dropped every force-close mechanism, so `ESTABLISHED` sockets were never torn down and accumulated toward Slack's per-app connection cap. | ||
|
|
||
| - `disconnect()` now arms a 30s close-handshake timeout, so a peer that never replies to the close frame no longer leaves the socket stuck in `CLOSING` indefinitely. | ||
| - On the default dispatcher path, the underlying TCP socket is now captured and force-destroyed during cleanup. A user-supplied `dispatcher` owns its own socket and instead relies on the close-handshake timeout — this limitation is documented on the `dispatcher` option. | ||
| - The `close` handler no longer schedules a reconnect on stale or overlapping close events, so at most one reconnect is scheduled per disconnect. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,14 +1,18 @@ | ||
| import { channel } from 'node:diagnostics_channel'; | ||
| import type { Socket } from 'node:net'; | ||
| import type { TLSSocket } from 'node:tls'; | ||
|
|
||
| import type { EventEmitter } from 'eventemitter3'; | ||
| import { CloseEvent, type Dispatcher, ErrorEvent, MessageEvent, ping, WebSocket } from 'undici'; | ||
| import { Agent, buildConnector, CloseEvent, type Dispatcher, ErrorEvent, MessageEvent, ping, WebSocket } from 'undici'; | ||
|
|
||
| import { SMWebsocketError } from './errors'; | ||
| import log, { type Logger, LogLevel } from './logger'; | ||
| import type { SocketModeDispatcher } from './SocketModeOptions'; | ||
|
|
||
| export const WS_READY_STATES = ['CONNECTING', 'OPEN', 'CLOSING', 'CLOSED']; | ||
|
|
||
| const CLOSE_HANDSHAKE_TIMEOUT_MS = 30_000; | ||
|
|
||
| interface PingPongMessage { | ||
| websocket: WebSocket; | ||
| payload: Buffer; | ||
|
|
@@ -38,7 +42,11 @@ export interface SlackWebSocketOptions { | |
| logger?: Logger; | ||
| /** @description Delay between this client sending a `ping` message, in milliseconds. */ | ||
| pingInterval?: number; | ||
| /** @description An undici Dispatcher used to establish the WebSocket connection (e.g. ProxyAgent). */ | ||
| /** | ||
| * @description An undici Dispatcher used to establish the WebSocket connection (e.g. ProxyAgent). | ||
| * Overrides the default dispatcher, which force-destroys the underlying socket on cleanup; a custom | ||
| * dispatcher cannot, so a stalled close handshake falls back to a timeout instead. | ||
| */ | ||
| dispatcher?: SocketModeDispatcher; | ||
| /** @description Whether this WebSocket should DEBUG log ping and pong events. `false` by default. */ | ||
| pingPongLoggingEnabled?: boolean; | ||
|
|
@@ -68,6 +76,8 @@ export class SlackWebSocket { | |
|
|
||
| private websocket: WebSocket | null; | ||
|
|
||
| private defaultSocket: Socket | TLSSocket | null = null; | ||
|
|
||
| /** | ||
| * The last timetamp that this WebSocket received pong from the server | ||
| */ | ||
|
|
@@ -89,6 +99,11 @@ export class SlackWebSocket { | |
| */ | ||
| private clientPingTimeout: NodeJS.Timeout | undefined; | ||
|
|
||
| /** | ||
| * Reference to the timer that force-closes the connection if the peer never completes the close handshake | ||
| */ | ||
| private closeHandshakeTimeout: ReturnType<typeof setTimeout> | undefined; | ||
|
|
||
| private openHandler: (() => void) | null = null; | ||
| private errorHandler: ((event: Event) => void) | null = null; | ||
| private messageHandler: ((event: Event) => void) | null = null; | ||
|
|
@@ -136,7 +151,10 @@ export class SlackWebSocket { | |
| public connect(): void { | ||
| this.logger.debug('Initiating new WebSocket connection.'); | ||
|
|
||
| this.websocket = new WebSocket(this.options.url, { dispatcher: this.options.dispatcher as Dispatcher }); | ||
| const dispatcher: Dispatcher = this.options.dispatcher | ||
| ? (this.options.dispatcher as Dispatcher) | ||
| : this.buildDefaultDispatcher(); | ||
| this.websocket = new WebSocket(this.options.url, { dispatcher }); | ||
|
|
||
| this.openHandler = () => { | ||
| this.logger.debug('WebSocket open event received (connection established)!'); | ||
|
|
@@ -205,6 +223,27 @@ export class SlackWebSocket { | |
| SlackWebSocket.pongChannel.subscribe(this.pongHandler); | ||
| } | ||
|
|
||
| /** | ||
| * The `connect` hook captures the underlying socket into `this.defaultSocket` so `cleanup()` can | ||
| * force-destroy it: undici's `WebSocket` hides its socket and detaches it from the pool at upgrade, | ||
| * leaving no other way to close a stalled peer. | ||
| */ | ||
| private buildDefaultDispatcher(): Dispatcher { | ||
| const baseConnect = buildConnector({}); | ||
| return new Agent({ | ||
| connect: (opts, callback) => { | ||
| baseConnect(opts, (err, socket) => { | ||
| if (!socket) { | ||
| callback(err ?? new Error('Socket Mode connector returned no socket'), null); | ||
| return; | ||
| } | ||
| this.defaultSocket = socket; | ||
| callback(null, socket); | ||
| }); | ||
| }, | ||
| }); | ||
| } | ||
|
|
||
| /** | ||
| * Disconnects the WebSocket connection with Slack, if connected. | ||
| */ | ||
|
|
@@ -216,15 +255,17 @@ export class SlackWebSocket { | |
| this.logger.debug('Terminating WebSocket (close frame received).'); | ||
| this.cleanup(); | ||
| } else if (this.websocket.readyState === WebSocket.CLOSING) { | ||
| // A close frame was already sent but the peer hasn't responded. Force-terminate rather than | ||
| // waiting for the ws library's closeTimeout (~30s) while the ping monitor logs repeated warnings. | ||
| this.logger.debug('Terminating WebSocket (close frame sent but no response, force-terminating).'); | ||
| this.cleanup(); | ||
| } else { | ||
| // If we haven't received a close frame yet, then we send one to the peer, expecting to receive a close frame | ||
| // in response. | ||
| this.logger.debug('Sending close frame (status=1000).'); | ||
| this.websocket.close(1000); // 1000 = Normal Closure | ||
| this.closeHandshakeTimeout = setTimeout(() => { | ||
| this.logger.warn('Peer did not complete the close handshake in time; forcing cleanup.'); | ||
| this.cleanup(); | ||
| }, CLOSE_HANDSHAKE_TIMEOUT_MS); | ||
| } | ||
| } else { | ||
| this.logger.debug('WebSocket already disconnected, flushing remainder.'); | ||
|
|
@@ -250,9 +291,14 @@ export class SlackWebSocket { | |
| if (this.pongHandler) SlackWebSocket.pongChannel.unsubscribe(this.pongHandler); | ||
| this.pingHandler = null; | ||
| this.pongHandler = null; | ||
| if (this.defaultSocket && !this.defaultSocket.destroyed) { | ||
| this.defaultSocket.destroy(); | ||
| } | ||
| this.defaultSocket = null; | ||
| this.websocket = null; | ||
| clearTimeout(this.serverPingTimeout); | ||
| clearInterval(this.clientPingTimeout); | ||
| clearTimeout(this.closeHandshakeTimeout); | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧠 praise: Clever pattern to avoid confused logs! |
||
| // Emit event back to client letting it know connection has closed (in case it needs to reconnect if | ||
| // reconnecting is enabled) | ||
| this.options.client.emit('close'); | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👁️🗨️ thought: This might be nice to surface as an option although I understand it's for default
dispatcherso might be confusing in some cases...There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I thought about this also 🤔 But I think we can add this in later if we need to