Skip to content

ssh, internal: flush the worker's queued output on every call - #1217

Merged
ejohnstown merged 6 commits into
wolfSSL:masterfrom
yosuke-wolfssl:fix/worker-deadlock
Sep 14, 2026
Merged

ejohnstown merged 6 commits into
wolfSSL:masterfrom
yosuke-wolfssl:fix/worker-deadlock

Conversation

@yosuke-wolfssl

@yosuke-wolfssl yosuke-wolfssl commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Problem

A read-only application on a non-blocking socket stalls permanently. A channel read credits the window, ChannelCreditWindow() queues a CHANNEL_WINDOW_ADJUST, and the socket write blocks; the peer has spent its window and goes silent waiting for it. The application calls wolfSSH_worker(), as wolfssh/ssh.h directs — but the worker gated its flush on DoReceive()'s return, and an idle socket makes DoReceive() return WS_FATAL_ERROR, which the gate did not list. No write is attempted, on that call or any later one.

The fix (src/ssh.c)

wolfSSH_worker() flushes whenever output is queued and the session is live:

if (ssh != NULL && !ssh->disconnected && ssh->outputBuffer.length != 0) {

ssh->error keeps the receive's code when the receive failed, and the close's when a WS_CHANNEL_CLOSED pass hard-failed its flush; every other status keeps the send's, and WS_REKEYING is withheld on a failed flush. The second DoReceive(), its WS_WINDOW_FULL arm, the WOLFSSH_TEST_BLOCK fork and the separate WS_CHANNEL_CLOSED flush all go — they existed to work around the gate. Ignoring ret is deliberate: an idle receive and a hard one both surface as WS_FATAL_ERROR, so a narrower gate would reintroduce the same bug. SendPacketFlush() now records its code on every transport failure, so two rules hold on wolfSSH_SendPacket(): a later write to ssh->error on the same pass must be conditional on the flush succeeding, and success is never written there.

What the wider flush exposed

  • BundlePacket() left an unpadded, un-MAC'd partial packet queued on a framing failure. The old gate never sent it; the new one would. Its failure arm drops it.
  • wolfSSH_shutdown() mapped its close-read's WS_CHAN_RXD and WS_EOF to WS_SUCCESS, reporting a clean teardown with bytes owed. It now reports WS_WANT_WRITE, or the send's error when that send failed.
  • ReceiveScpMessage() found the event through wolfSSH_get_error(), which the flush now overwrites, so a WS_EXTDATA pass fell to default: and stranded the peer's stderr. It takes the event from the return.
  • ScpStreamRead() handed WS_EXTDATA to all three of its callers as an error: a confirmation read reported success without reading one, and a file-body read aborted mid-transfer. It drains the stderr and reads again. This predates the PR.

Consumers

A queued write now reports WS_WANT_WRITE where it reported WS_WANT_READ, and the return carries the event rather than the flush status.

  • Tolerate it — three shell loops treated anything but WS_WANT_READ as fatal.
  • Wait for writability — the Windows wolfsshd drain and sftp_worker()'s retry retried with no wait at all; both now use a new tcp_select_write() in wolfssh/test.h and continue only while it reports WS_SELECT_SEND_READY.
  • Route on the return — four shell loops and ReceiveScpMessage() dispatched off wolfSSH_get_error(), so a WS_CHAN_RXD pass with a blocked flush skipped the drain. They now take the event from the return, the owed write from wolfSSH_OutputPending(), and read wolfSSH_get_error() only to tell a transient failure from a terminal one.
  • Keep draining, under the lockFlushQueuedSend() exited on the first pass once the return carried the event, reporting success with bytes queued. It samples wolfSSH_get_error() inside the mutex and reports a failed send over the event it rode in on.
  • Don't skip the teardown drainscpclient gated its close-message drain on the shutdown result, so the new WS_WANT_WRITE closed the socket with the reply queued. Only that return is fixed: a WS_CHANNEL_CLOSED teardown still skips the drain and closes with our close reply unsent, which is a follow-up.

portfwd, client and scpclient also accept WS_WANT_WRITE from wolfSSH_shutdown(). ssh.h now tells callers to read the return and wolfSSH_get_error() as independent channels on every pass, rather than enumerating which value lands where; WS_WINDOW_FULL comes off the return list. Master's new application-driven channel mode hands the session to wolfSSH_worker() after user auth, so out-of-tree callers land on this contract directly.

One new public API

wolfSSH_OutputPending() becomes WOLFSSH_API. An application had no way to ask whether a write was still owed except by reading ssh->error, a field written twice a pass — by the receive, then by the flush. OpenSSH answers this with ssh_packet_have_data_to_write() and OpenSSL with SSL_want_write(); wolfSSH had the same predicate and kept it private. The library now gates its own flush on it.

A send that fails outright also takes wolfSSH_worker()'s return in place of the event it arrived with; only WS_CHANNEL_CLOSED keeps the return there.

Tests

Twenty new unit tests cover the flush on an idle receive, the owed flush across calls, and what ret and ssh->error hold after a receive, send, buffer, callback or framing failure — including a clean WS_SUCCESS receive whose flush short-writes, driven by a peer window adjust, which was the last uncovered arm.

The WS_CHANNEL_CLOSED-with-a-short-flush arm is covered, by test_DoChannelCloseWantWrite(), which five reviews have now called untested. Deleting && sendRet != WS_WANT_WRITE drops the suite to 164/165 with that test the only failure.

Verification

  • unit.test 165 passed / 0 failed; regress.test passed. Clean under gcc-13 -Werror across 6 configurations, plus lint.
  • Network contention (-DWOLFSSH_TEST_BLOCK, scripts/sftp.test) passes at WOLFSSH_BLOCK_PROB 70/50/30 in 47s/21s/12s, re-run on the current base.
  • No negative control for the SCP changes or FlushQueuedSend(): scripts/scp.test skips under WOLFSSH_TEST_BLOCK, and FlushQueuedSend() is static in a file no test binary links. Recorded as follow-ups.

Known limitations, not addressed here

Six caller sites mishandle an owed flush, all inherited, and deferred because none has a test that can verify a fix. For the three shell loops the consequence does change: where a short flush used to break the loop and drop the session, tolerating WS_WANT_WRITE leaves it waiting on a read-only select(). On the idle-receive path this PR fixes, master parked there too.

File Fault
examples/echoserver/echoserver.c:1023 select() watches read fds with a NULL timeout
ide/Espressif/.../echoserver.c:984 same
apps/wolfsshd/wolfsshd.c:2251 select() has an 800us timeout, but the guard below gates the worker on rc != 0, so a timeout iteration never retries
examples/client/client.c:1213 and :1230 the worker's return is matched against a whitelist omitting WS_FATAL_ERROR, so an idle receive reaches err_sys(). Master reached the same whitelist, so this PR does not change its reachability
examples/echoserver/echoserver.c:1566 the SFTP loop's continue skips the tcp_select() below it, and its || re-enters on any pending send, so it spins rather than waits
ide/Espressif/.../echoserver.c:1464 the same loop guarded with &&, so it skips the wait without the spin

ssh_worker() has a separate open finding, 10544; both patch the same function, so that lands first.

@yosuke-wolfssl yosuke-wolfssl self-assigned this Aug 31, 2026
Copilot AI lite review requested due to automatic review settings August 31, 2026 05:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@wolfSSL-Fenrir-bot wolfSSL-Fenrir-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fenrir Automated Review — PR #1217

Scan targets checked: wolfssh-bugs, wolfssh-src

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

Comment thread src/ssh.c Outdated
Comment thread src/ssh.c Outdated

@wolfSSL-Fenrir-bot wolfSSL-Fenrir-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fenrir Automated Review — PR #1217

Scan targets checked: wolfssh-bugs, wolfssh-src

Findings: 2
2 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

Comment thread src/ssh.c
Comment thread apps/wolfsshd/wolfsshd.c
Comment thread src/ssh.c
Comment thread apps/wolfsshd/wolfsshd.c

@wolfSSL-Fenrir-bot wolfSSL-Fenrir-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fenrir Automated Review — PR #1217

Scan targets checked: wolfssh-bugs, wolfssh-src

Fenrir result: Approved ✅

No new issues found in the changed files.

Advisory only — this automated result does not count as a GitHub approval.

@wolfSSL-Fenrir-bot
wolfSSL-Fenrir-bot dismissed stale reviews from themself September 1, 2026 01:12

Fenrir's latest completed scan found no issues; clearing the prior automated change request.

Comment thread src/ssh.c Outdated

@wolfSSL-Fenrir-bot wolfSSL-Fenrir-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fenrir Automated Review — PR #1217

Scan targets checked: wolfssh-bugs, wolfssh-src

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

Comment thread src/ssh.c
Comment thread src/ssh.c

@wolfSSL-Fenrir-bot wolfSSL-Fenrir-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fenrir Automated Review — PR #1217

Scan targets checked: wolfssh-bugs, wolfssh-src

Findings: 2
2 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

Comment thread apps/wolfsshd/wolfsshd.c Outdated
Comment thread examples/echoserver/echoserver.c Outdated

@wolfSSL-Fenrir-bot wolfSSL-Fenrir-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fenrir Automated Review — PR #1217

Scan targets checked: wolfssh-src, wolfssh-bugs

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

Comment thread examples/client/client.c

@wolfSSL-Fenrir-bot wolfSSL-Fenrir-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fenrir Automated Review — PR #1217

Scan targets checked: wolfssh-src, wolfssh-bugs

Findings: 3
3 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

Comment thread apps/wolfssh/wolfssh.c Outdated
Comment thread examples/scpclient/scpclient.c
Comment thread tests/unit.c
- The echoserver and Espressif shell loops and the Windows
  wolfsshd shell loop treat a WS_WANT_WRITE from wolfSSH_worker()
  as non-fatal.

@wolfSSL-Fenrir-bot wolfSSL-Fenrir-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fenrir Automated Review — PR #1217

Scan targets checked: wolfssh-src, wolfssh-bugs

Findings: 4
4 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

Comment thread src/ssh.c Outdated
Comment thread src/ssh.c
Comment thread apps/wolfsshd/wolfsshd.c
Comment thread apps/wolfsshd/wolfsshd.c Outdated

@wolfSSL-Fenrir-bot wolfSSL-Fenrir-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fenrir Automated Review — PR #1217

Scan targets checked: wolfssh-src, wolfssh-bugs

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

Comment thread apps/wolfsshd/wolfsshd.c Outdated

@wolfSSL-Fenrir-bot wolfSSL-Fenrir-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fenrir Automated Review — PR #1217

Scan targets checked: wolfssh-src, wolfssh-bugs

Findings: 2
2 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

Comment thread apps/wolfsshd/wolfsshd.c Outdated
Comment thread src/wolfscp.c
yosuke-wolfssl and others added 5 commits September 14, 2026 16:35
- wolfSSH_worker() calls wolfSSH_SendPacket() whenever
  ssh->outputBuffer holds bytes and the session is not
  disconnected. ssh->error keeps the receive's code when the
  receive failed, and the close's when a WS_CHANNEL_CLOSED pass
  hard-failed its flush; WS_REKEYING is withheld on a failed
  flush. Drops the second DoReceive(), the WOLFSSH_TEST_BLOCK
  fork and the separate WS_CHANNEL_CLOSED flush.
- BundlePacket() resets ssh->outputBuffer.length to
  ssh->packetStartIdx when the framing fails. wolfSSH_shutdown()
  reports WS_WANT_WRITE when its close-read leaves output queued,
  and the send's own error in place of it when that send failed.
  SendPacketFlush() records its code in ssh->error on every
  transport failure path, and wolfSSH_TriggerKeyExchange() writes
  it only when SendKexInit() fails.
- portfwd, client and scpclient accept WS_WANT_WRITE from
  wolfSSH_shutdown(); in scpclient the close-message drain runs
  on it.
- wolfssh/ssh.h drops WS_WINDOW_FULL from wolfSSH_worker() and says
  to read the return and wolfSSH_get_error() as independent channels
  on every pass.
- Twenty unit tests and the extended TestWorkerReportsDisconnect
  cover what ret and ssh->error hold after a receive, send, buffer,
  callback or framing failure.
- tcp_select_write() joins tcp_select(), with WS_SELECT_SEND_READY
  at the end of the enum. tcp_select() passes NULL for writefds and
  cannot wait on the write side.
- The Windows wolfsshd window-change drain and sftp_worker()'s
  handshake flush retry wait on it. The drain records a give-up in
  ret, and sftp_worker() breaks only on WS_SELECT_ERROR_READY.
- The echoserver, Espressif and both wolfsshd shell loops, and
  ReceiveScpMessage(), take the event from wolfSSH_worker()'s
  return when it carries one, in place of reading ssh->error alone.
  The two echoservers cover WS_CHAN_RXD, WS_REKEYING,
  WS_CHANNEL_CLOSED and WS_EOF; the wolfsshd loops and
  ReceiveScpMessage() cover the ones they have arms for.
- The POSIX wolfsshd loop takes an owed write into wantWrite before
  the event overwrites rc, and its WS_WANT_WRITE arm runs the
  channel drain below in place of skipping it.
- FlushQueuedSend() folds the receive's own statuses into
  WS_SUCCESS inside its loop and keeps flushing while
  wolfSSH_get_error() reports WS_WANT_WRITE within the deadline, in
  place of looping on the worker's return. It reports
  WS_WANT_WRITE when the deadline leaves the packet queued.
ReceiveScpConfirmation() takes a WS_EXTDATA off ScpStreamRead()'s
return, the rule ssh.h now states for reading an event.
FlushQueuedSend() samples the owed write with the session still
locked, since the peer reader thread writes ssh->error too, and
reports a send that failed there over the event it rode in on.

- A pass that delivers stderr while its flush short-writes returns
  WS_EXTDATA with WS_WANT_WRITE in ssh->error, so the confirmation
  read aborted the transfer with the stderr undrained.
- wolfSSH_stream_read() on the reader thread clears ssh->error with
  this thread's packet still queued, which ended the flush early and
  reported it a success.
- An event return with a hard send error latched collapsed to
  WS_SUCCESS, telling the caller a queued packet reached a socket
  that was gone.
- ScpStreamRead() calls _DumpExtendedData() and reads again when
  wolfSSH_stream_read() returns WS_EXTDATA, reporting the drain's
  own failure when it has one.
- ReceiveScpConfirmation() drops its WS_EXTDATA arm and reports a
  negative read directly.
- ReceiveScpMessage() returns _DumpExtendedData()'s failure in place
  of discarding it.
- wolfSSH_OutputPending() moves from wolfssh/internal.h to
  wolfssh/ssh.h as a WOLFSSH_API taking a const WOLFSSH*, defined
  in src/ssh.c beside the other public calls.
- wolfSSH_worker() puts the send's code in the return in place of
  an event when the flush fails outright, and gates its flush on
  wolfSSH_OutputPending(). wolfssh/ssh.h states both.
- Both wolfsshd shell loops, both echoservers and
  ReceiveScpMessage() dispatch on wolfSSH_worker()'s return, and
  call wolfSSH_get_error() only to tell a transient failure from a
  terminal one. The POSIX wolfsshd loop sets wantWrite from
  wolfSSH_OutputPending().
- Five worker tests expect the send's code where they expected the
  event, and tests/testsuite.c calls wolfSSH_OutputPending().

@wolfSSL-Fenrir-bot wolfSSL-Fenrir-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fenrir Automated Review — PR #1217

Scan targets checked: wolfssh-src, wolfssh-bugs

Findings: 3
3 finding(s) posted as inline comments (see file-level comments below)

💬 1 finding(s) from an earlier review are still open and were not re-posted:

  • ESP-IDF echoserver stalls on queued output — ide/Espressif/ESP-IDF/examples/wolfssh_echoserver/main/echoserver.c:1172

This review was generated automatically by Fenrir. Reported findings require changes before merge.

Comment thread apps/wolfsshd/wolfsshd.c
Comment thread examples/echoserver/echoserver.c
Comment thread ide/Espressif/ESP-IDF/examples/wolfssh_echoserver/main/echoserver.c
@ejohnstown
ejohnstown dismissed wolfSSL-Fenrir-bot’s stale review September 14, 2026 17:40

Last review turned up already deferred issues.

@ejohnstown
ejohnstown merged commit a14e6d0 into wolfSSL:master Sep 14, 2026
187 of 189 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants