Revise backpressure examples in streams documentation#90
Conversation
Discussed here: https://stackoverflow.com/questions/79821132/node-js-back-pressure-documentation-whats-the-relevance-of-their-example Original backpressure example contains following oddities: 1) Incorrect zip example 2) Incorrect assumption that zip console command loads full file into memory. Updated examples to demonstrate file compression using Node.js streams with backpressure handling. Signed-off-by: Dmitry Baskakov <dmitry@bask.ws>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
PR SummaryLow Risk Overview The section now walks through three Node-only steps on the same ~9 GB file: The removed narrative that compared a “corrupt” zip output to a successful stream decompress is gone; the existing Reviewed by Cursor Bugbot for commit 7ce0776. Bugbot is set up for automated code reviews on this repo. Configure here. |
👋 Codeowner Review RequestThe following codeowners have been identified for the changed files: Team reviewers: @nodejs/streams Please review the changes when you have a chance. Thank you! 🙏 |
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Updates the backpressure module to better illustrate memory/throughput implications of buffering vs streaming in Node.js compression examples.
Changes:
- Replaces the
zip(1)example withfs.readFileSync+zlib.gzipSyncexamples (CJS + ESM). - Adds an explicit “streams without backpressure” example plus an explanation of why it can OOM.
- Removes the prior note about comparing resulting archives.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| In one scenario, we will take a large file (approximately ~9 GB) and compress it | ||
| using the familiar [`zip(1)`][] tool. | ||
| In one scenario, we will read a large file (approximately ~9 GB) using `fs.readFileSync`: and compress it |
| writeFileSync('The.Matrix.1080p.mkv.gz', compressed); | ||
| ``` | ||
|
|
||
| This fails on two separate limits, whichever you hit first: the buffer size cap or heap exhaustion. Lets rewite it another way, using Node.js' [`Stream`][] but without backpressure: |
| While that will take a few minutes to complete, in another shell we may run | ||
| a script that takes Node.js' module [`zlib`][], that wraps around another | ||
| compression tool, [`gzip(1)`][]. | ||
| Neither of two writes respects backpressure. Stage 1 keeps pushing into gzip even after gzip.write() returns false, and stage 2 keeps pushing into out even after out.write() returns false. Both internal buffers can grow without bound, so this is very prone to running out of memory. On a large compressible file both numbers climb fast and it heads for `JavaScript heap out of memory`. |
| inp.on('data', (chunk) => gzip.write(chunk)); | ||
| inp.on('end', () => gzip.end()); | ||
| gzip.on('data', (chunk) => out.write(chunk)); | ||
| gzip.on('end', () => out.end()); |
|
|
||
| ```cjs | ||
| const { createReadStream, createWriteStream } = require('node:fs'); | ||
| const gzip = require('node:zlib').createGzip(); |
There was a problem hiding this comment.
| const gzip = require('node:zlib').createGzip(); | |
| const { createGzip } = require('node:zlib'); | |
| const gzip = createGzip(); |
| While that will take a few minutes to complete, in another shell we may run | ||
| a script that takes Node.js' module [`zlib`][], that wraps around another | ||
| compression tool, [`gzip(1)`][]. | ||
| Neither of two writes respects backpressure. Stage 1 keeps pushing into gzip even after gzip.write() returns false, and stage 2 keeps pushing into out even after out.write() returns false. Both internal buffers can grow without bound, so this is very prone to running out of memory. On a large compressible file both numbers climb fast and it heads for `JavaScript heap out of memory`. |
There was a problem hiding this comment.
| Neither of two writes respects backpressure. Stage 1 keeps pushing into gzip even after gzip.write() returns false, and stage 2 keeps pushing into out even after out.write() returns false. Both internal buffers can grow without bound, so this is very prone to running out of memory. On a large compressible file both numbers climb fast and it heads for `JavaScript heap out of memory`. | |
| Neither of two writes respects backpressure. Stage 1 keeps pushing into gzip even after `gzip.write()` returns false, and stage 2 keeps pushing into out even after `out.write()` returns false. Both internal buffers can grow without bound, so this is very prone to running out of memory. On a large compressible file both numbers climb fast and it heads for `JavaScript heap out of memory`. |
| compression tool, [`gzip(1)`][]. | ||
| Neither of two writes respects backpressure. Stage 1 keeps pushing into gzip even after gzip.write() returns false, and stage 2 keeps pushing into out even after out.write() returns false. Both internal buffers can grow without bound, so this is very prone to running out of memory. On a large compressible file both numbers climb fast and it heads for `JavaScript heap out of memory`. | ||
|
|
||
| To resolve this, we may use pipe, which pauses the read when write() returns false. When `gzip.write()` returns `false`, `pipe` calls `pause()` on the read stream, halting disk reads. Once gzip works through its backlog and the buffer empties, it emits a `'drain'` event, and `pipe` calls `resume()` to start reading again. |
There was a problem hiding this comment.
| To resolve this, we may use pipe, which pauses the read when write() returns false. When `gzip.write()` returns `false`, `pipe` calls `pause()` on the read stream, halting disk reads. Once gzip works through its backlog and the buffer empties, it emits a `'drain'` event, and `pipe` calls `resume()` to start reading again. | |
| To resolve this, we may use pipe, which pauses the read when `write()` returns false. When `gzip.write()` returns `false`, `pipe` calls `pause()` on the read stream, halting disk reads. Once gzip works through its backlog and the buffer empties, it emits a `'drain'` event, and `pipe` calls `resume()` to start reading again. |
Discussed here: https://stackoverflow.com/questions/79821132/node-js-back-pressure-documentation-whats-the-relevance-of-their-example Original backpressure example contains following oddities: 1) Incorrect zip example 2) Incorrect assumption that zip console command loads full file into memory. Updated examples to demonstrate file compression using Node.js streams with backpressure handling.