Stream package extraction to bound peak memory usage - #410
Conversation
|
All commit authors signed the Contributor License Agreement. |
| with open(dest, "wb") as f: | ||
| f.write(zf.read(member)) | ||
| with zf.open(member) as source, open(dest, "wb") as f: | ||
| shutil.copyfileobj(source, f, length=1024 * 1024) |
There was a problem hiding this comment.
How badly does it affect the maximum memory usage if we make this use 10MB instead of 1MB? I'd like our standard Python distributions to still only use one call per file, and the biggest file we include is (currently) around 7MB.
There was a problem hiding this comment.
I measured this on Windows 11 with CPython 3.14.3, using the PR's actual extract_package function and changing only the copyfileobj buffer size in an in-memory benchmark variant.
For synthetic ZIP_DEFLATED archives containing one member each, the median tracemalloc peak across five extractions was:
| Uncompressed member size | 1 MiB buffer | 10 MiB buffer | Increase |
|---|---|---|---|
| 7 MiB | 3.34 MiB | 17.32 MiB | 13.98 MiB |
| 64 MiB | 3.78 MiB | 30.67 MiB | 26.90 MiB |
Archive creation was outside the measured region. Each extraction used a fresh destination, and I verified the extracted contents afterward. The payload was a repeating bytes(range(256)) pattern, so these are highly compressible synthetic cases, not measurements of standard runtime packages. These figures measure peak traced Python allocations during extraction, not total process RSS; they include zipfile/decompression allocations as well as the copy buffer.
The 10 MiB buffer therefore costs more than just the extra 9 MiB of buffer capacity, but still bounds memory for oversized members. A 7 MiB file fits in one data-bearing read/write with that buffer (copyfileobj also performs a final EOF read).
Punisheroot has also posted complementary measurements using real 3.12/3.13/3.14 packages here: #409 (comment) . Their reported results likewise support 10 MiB as a compromise: single data-bearing reads for normal distribution files, with bounded memory for oversized files.
Based on these results, 10 MiB looks reasonable to me for the goal you described.
There was a problem hiding this comment.
Following up with measurements using real official runtime packages rather than synthetic files.
I tested the AMD64 ZIP distributions for Python 3.12.10, 3.13.15, and 3.14.7 from the official Windows install-manager index, verifying each download against its published SHA-256. Together they contain 10,241 files and 376,099,043 uncompressed bytes; the largest member is 6,945,272 bytes.
Setup: Windows 11 (build 26200), CPython 3.14.3 AMD64. Each run used a fresh process and destination and extracted all three packages sequentially using this PR's extract_package function. The variants changed only the copy operation: original zf.read(), 1 MiB copyfileobj, or 10 MiB copyfileobj. There was one warm-up per variant, then five measured runs per variant in rotating order.
| Variant | Median extraction time | Time range | Median tracemalloc peak | Median peak process working set |
|---|---|---|---|---|
| Original whole-member read | 9.50 s | 9.23–14.09 s | 25.05 MiB | 50.66 MiB |
| 1 MiB buffer | 11.91 s | 9.67–15.57 s | 7.33 MiB | 40.86 MiB |
| 10 MiB buffer | 10.23 s | 9.78–20.11 s | 21.70 MiB | 50.75 MiB |
Moving from 1 MiB to 10 MiB increased the median traced peak by 14.38 MiB and peak working set by 9.88 MiB. Compared with the original, 10 MiB reduced traced allocations somewhat, but the process working-set peak was essentially unchanged on these normal-sized packages.
A separate instrumented extraction confirmed that all 10,164 non-empty files completed in exactly one data-bearing read with 10 MiB. There were 20,405 reads including EOF reads across all 10,241 files. This diagnostic run was excluded from the measurements above.
SHA-256 output manifests, including relative paths and per-file digests, matched across all 18 benchmark executions and the separate read-count run.
Limitations: downloads, imports, integrity hashing, and cleanup were excluded from extraction timing. tracemalloc was enabled during all timed runs, so timings include its overhead. Windows working-set peaks were read with GetProcessMemoryInfo immediately after extraction, before hashing, and include the interpreter/tracing machinery. OS caches were not flushed and background activity was not controlled. The timing ranges overlap substantially, so I would not claim a reliable speed improvement from this run.
This confirms that 10 MiB meets the single-data-read goal for these distributions, at the cost of giving up much of the normal-package memory saving of 1 MiB. Oversized members remain a separate case, covered by the earlier synthetic measurements. No production code has been changed for this benchmark.
There was a problem hiding this comment.
Let's make it 10MB then. In practice, I think this will help most with actual "bad" cases such as a custom package that contains a 2GB file in it, which will no longer attempt to allocate all of that in one go.
|
Thanks! |
2 participants
Fixes #409.
Extract ZIP members in 1 MiB chunks instead of materializing each uncompressed member in memory. Destination validation, repair behavior, overwrite warnings, and progress reporting are unchanged.
Adds regression coverage for ZIP and NuGet archives, multi-chunk files, empty files, existing files, and repair. The four new cases fail against the original extraction function and pass with the change.
Validation on Windows, CPython 3.14.3: