fix(readable): keep body bytes that arrive after setEncoding()

[mcollina]

Fixes #5611. Alternative to #5612, which fixes the same bug by retaining a second copy of the body.

The bug

setEncoding() installs a StringDecoder (since #5003), so from that point state.buffer holds decoded strings, and the trailing bytes of a multi-byte sequence split across a chunk boundary sit inside the decoder, in neither place. #5003 worked around this by snapshotting the raw chunks that happened to be buffered at the time of the call into kPreservedBuffer. Chunks arriving after the call were never added to it, and consumeStart() treated the two sources as mutually exclusive:

const preserved = consume.stream[kPreservedBuffer]
if (preserved && preserved.length > 0) {
  // ... only the snapshot
} else if (state.bufferIndex) {

so everything outside the snapshot was dropped, silently:

body.setEncoding('utf8')
await new Promise(r => setTimeout(r, 150))   // let the body arrive
await body.text()                            // "abc�" instead of "abc傳def"

The fix

Remove kPreservedBuffer and read the one source of truth the stream already keeps:

• consumePush() re-encodes a string chunk back to bytes, so consume.length stays a byte count. Without that .arrayBuffer()/.bytes() return a zero-filled buffer of the wrong length, because a string's length is in UTF-16 code units and Uint8Array.prototype.set(string) writes nothing.
• consumeStart() then appends the bytes the decoder is holding for an incomplete sequence, which are absent from every string in state.buffer.

That covers chunks buffered before setEncoding(), after it, and after the consume was requested, because all of them end up in exactly one of those two places. The "which entry of state.buffer is the duplicate?" problem that makes reading both sources unworkable only exists while kPreservedBuffer does; deleting it removes the ambiguity rather than arbitrating it.

Net effect on lib/: -15 lines, push() untouched, and nothing retained beyond the stream's own buffer. #5003 is not undone — setEncoding() + for await and + on('data') still decode a split multi-byte character correctly.

Why not keep the raw chunks

#5612 makes push() append every raw chunk while an encoding is set, which is a second full copy of everything buffered, allocated on the hot path. Backpressure bounds it (api-request.js pauses when push() returns false), but not at one highWaterMark: with a decoder installed state.length counts decoded characters, so 64 KiB of buffered CJK is ~192 KiB of retained bytes on top. The release also lags a push() behind the first emitted chunk.

Trade-off

Re-encoding a decoded string is byte-exact for latin1, hex, base64, utf16le and valid UTF-8, and only applies to what was buffered before the consume started — chunks arriving later reach consumePush() as raw Buffers. It is lossy in two corners: invalid UTF-8 bytes under setEncoding('utf8') come back as U+FFFD, and high-bit bytes under setEncoding('ascii') come back masked to 7 bits, if a .bytes()/.arrayBuffer()/.blob() consume reads them out of the pre-consume buffer. Both mean "decode this as text, then hand me the raw bytes back", and a plain node:stream Readable loses the same information. Retaining raw chunks is the only way to be exact there, at the cost above.

Tests

test/readable.js gains the 7 cases from #5612 (@marko1olo) — chunks pushed after setEncoding(), a partial character buffered, utf8/hex/base64/latin1 before any chunk arrives, and .json() — and test/client-request.js gains the end-to-end repro from #5611 over a real socket. All were confirmed failing on main first, with the truncation, ERR_INVALID_ARG_TYPE from Buffer.concat, and a zero-filled Uint8Array.

test/readable.js 18/18, test/client-request.js 49/49, npm run test:unit 1459 pass / 0 fail, npm run test:fetch 47/47, npm run test:typescript and npm run lint clean. Node 24, Linux.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.49%. Comparing base (c76fe46) to head (390377d).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5620      +/-   ##
==========================================
- Coverage   93.49%   93.49%   -0.01%     
==========================================
  Files         110      110              
  Lines       38429    38414      -15     
==========================================
- Hits        35931    35916      -15     
  Misses       2498     2498              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.