Skip to content

fix(websocket): drain httpx-ws reader thread before closing sync sessions - #164

Open
cshape wants to merge 2 commits into
fishaudio:mainfrom
cshape:fix/drain-sync-ws-reader
Open

cshape wants to merge 2 commits into
fishaudio:mainfrom
cshape:fix/drain-sync-ws-reader

Conversation

@cshape

@cshape cshape commented Sep 11, 2026

Copy link
Copy Markdown

Symptom

Intermittent [SSL: WRONG_VERSION_NUMBER], record layer failure, or [X509] PEM lib errors on the WebSocket session opened right after a completed sync WebSocket TTS session (client.tts.stream_websocket(...) and legacy WebSocketSession.tts(...)). Linux only; worse on loaded hosts. The async client is unaffected.

Mechanism

httpx-ws's sync WebSocketSession reads on a background thread and closes the socket from the calling thread (close() writes the close frame, then stream.close()).

A read that has not started yet is harmless: CPython's SSLSocket._real_close sets _sslobj = None before closing the fd, so a later read falls through to socket.recv on fd -1 and gets EBADF.

The culprit is the read that is already in flight. On Linux, close() does not interrupt a thread blocked in recv(). OpenSSL (read_ahead off) fetches a TLS record header and body with separate read(fd) calls on the fd number cached in its BIO. When the server's close-frame reply arrives, the second read() lands on the next connection, which has reused the same fd number, and that new session fails during its TLS handshake / 101 read. macOS wakes the blocked reader with EBADF on close(), which is why it is unaffected there.

Fix

fishaudio.core._ws_utils.drain_reader(ws), called in a finally before leaving the with connect_ws(...) block in both sync paths (fishaudio/resources/tts.py and fish_audio_sdk/websocket.py). It:

  1. sends CloseConnection(1000) via ws.send(...), so the server still sees a clean 1000 close (not 1006) and wsproto moves to LOCAL_CLOSING, which makes httpx-ws's later close() skip its own frame;
  2. shutdown(SHUT_RDWR) on the socket from ws.stream (fallback: response.extensions["network_stream"]), which wakes the in-flight read with EOF;
  3. joins _background_receive_task (5 s timeout; only reached when the reader is stuck, where httpx-ws's own untimed join would hang afterwards anyway).

Everything is best-effort; failures are logged at debug. The sender future is still awaited/raised as before. Async paths untouched.

Validation

GCP us-central1 Linux VM (Python 3.12, httpx-ws 0.9.0), 8 parallel processes x 10 sessions each, per batch:

  • unpatched: 47/160 sessions with SSL failures
  • patched (previous revision of this PR): 0/320
  • patched (this revision, close frame sent first): 0/320 SSL failures across 4 batches (new 0/80, legacy 0/80, new 0/80, legacy 0/80)

The remaining non-SSL failures in those batches (19/320, WebSocket disconnected unexpectedly / WebSocketErr) are a separate server-side issue that also appears without this patch.

Unit tests: helper ordering (close frame, shutdown, join), stream fallback, send failure, shutdown failure, missing socket/thread, debug logging on unexpected errors; both sync paths call the helper before __exit__ (asserted on mock_calls order); and a real httpx_ws.WebSocketSession over a socket.socketpair() that checks the reader thread exits, no threading.excepthook fires, state is LOCAL_CLOSING, and the peer receives a 1000 close frame before FIN. Passes on Python 3.9 (httpx-ws 0.7.2), 3.12 (0.8.1) and 3.13 (0.9.0).

Residual risk

If httpx-ws has already closed the socket on one of its own error paths (keepalive ping timeout, or a sender WriteError triggering its close()), the fd is already freed, shutdown() fails with EBADF, and the in-flight read can still race. That can only be fixed inside httpx-ws's close() (shut down / join before closing the fd). Upstream fix: frankie567/httpx-ws#150. Once that lands and we bump the dependency, this helper becomes redundant but harmless.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Summary by CodeRabbit

  • Bug Fixes

    • Improved reliability when ending text-to-speech WebSocket streams.
    • Ensures remaining connection messages are processed during normal completion and error handling.
    • Helps prevent stalled or lingering connections during audio streaming cleanup.
  • Tests

    • Added coverage for successful and failed streaming sessions, connection shutdown behavior, and cleanup resilience.

@coderabbitai

coderabbitai Bot commented Sep 11, 2026

Copy link
Copy Markdown

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 6cc99727-6024-44b7-ba53-c7a9a5c5977b

📥 Commits

Reviewing files that changed from the base of the PR and between 4481509 and 090c91b.

📒 Files selected for processing (4)
  • src/fish_audio_sdk/websocket.py
  • src/fishaudio/core/_ws_utils.py
  • src/fishaudio/resources/tts.py
  • tests/unit/test_ws_utils.py

Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.


📝 Walkthrough

Walkthrough

The PR adds a WebSocket reader-draining utility and calls it from synchronous TTS cleanup paths. Tests cover close-frame handling, socket shutdown, reader-thread termination, failure handling, and legacy and current TTS sessions.

Changes

WebSocket cleanup

Layer / File(s) Summary
Reader draining utility
src/fishaudio/core/_ws_utils.py
Adds drain_reader, which sends a close frame, shuts down the socket, joins the background reader thread, and logs handled failures.
TTS cleanup integration
src/fish_audio_sdk/websocket.py, src/fishaudio/resources/tts.py
Calls drain_reader in finally blocks after synchronous TTS receive processing and before WebSocket cleanup.
Cleanup validation
tests/unit/test_ws_utils.py
Tests close-frame behavior, socket shutdown, reader termination, failure handling, and both current and legacy TTS WebSocket paths.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 090c9

The synchronous TTS cleanup now drains WebSocket readers before session closure, with coverage for normal, failing, and real-socket behavior. No actionable merge risk remains.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 14.29% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 14 functions across 4 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: draining the httpx-ws reader thread before closing synchronous WebSocket sessions.
  • Fix all pre-merge checks with AI
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Sep 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines Coverage Δ
src/fishaudio/core/_ws_utils.py 100.00% <100.00%> (ø)
src/fishaudio/resources/tts.py 95.12% <100.00%> (+0.12%) ⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant