Size streaming GeoTIFF write budget from source chunks, not output tiles#3008
Open
brendancol wants to merge 2 commits into
Open
Size streaming GeoTIFF write budget from source chunks, not output tiles#3008brendancol wants to merge 2 commits into
brendancol wants to merge 2 commits into
Conversation
…3007) The streaming writer's horizontal-segment budget was computed from the output tile geometry (tile_h * tile_w). For a map_overlap source such as open_geotiff(chunks=...).xrs.slope() that undercounts: slicing one tile-row pulls every source chunk-row the band touches plus a one-chunk halo. A source chunked taller than the tile on a wide raster therefore pulled several full-width source chunk-rows per compute and OOMed, despite streaming_buffer_bytes. Size the budget from the source chunk-row span (touched chunk-rows + a one-chunk halo each side) so peak per-compute memory tracks the soft cap. Falls back to the tile height for numpy-from-dask / unknown-chunk arrays. On the repro (2048x8192 f32, chunks=512, slope, 8 MB cap) peak source bytes per compute drop from 32 MB to 12.6 MB.
brendancol
commented
Jun 7, 2026
Contributor
Author
brendancol
left a comment
There was a problem hiding this comment.
PR Review: source-aware streaming write budget
Reviewed the full diff and the surrounding streaming-writer context. The change is narrow and the logic holds up.
Blockers
None.
Suggestions
- The one-chunk halo is added unconditionally, so the budget treats every source as if it had a
map_overlaphalo. A genuinely non-overlap wide read (plain windowed read, no slope/aspect) will therefore segment about one step earlier than it strictly needs to. The writer can't cheaply tell overlap from non-overlap, and over-segmenting only costs an extra compute call near the buffer ceiling -- the safe direction. Worth a one-line note in the code comment that the non-overlap case is intentionally conservative, but not worth detection machinery.
Nits
_writer.py:import bisectsits inside the helper rather than at module top. That matches the file's existing style (os/tempfileare imported inside_write_streaming), so it's consistent -- leaving as-is is fine.- Strip layout (
tiled=False) still has no horizontal segmentation, so an overlap source in strip mode keeps its old unbounded peak. The issue scopes that out and the docstring now says so; flagging only so it isn't forgotten.
What looks good
- The budget is sized from the real driver.
_max_streaming_row_spanaccounts for the source chunk-rows a tile-band touches plus the one-chunk halo, and the one-chunk bound is correct becausemap_overlaprequiresdepth <= chunk size. - The fix only resizes column segments; the slices stay tile-aligned and cover the full raster, so output bytes are unchanged. The 146-test streaming + dask-integration suite staying green backs that up.
test_overlap_source_respects_buffer_3007instruments actual source-chunk materialization rather than asserting on an internal counter, and it fails on the pre-fix writer (32 MB vs 8 MB cap). That's a real regression guard, not a smoke test.- Helper has direct unit coverage for the three interesting shapes (band inside a tall chunk, middle band with halo on both sides, single full-height chunk).
- Unknown-chunk / numpy-from-dask inputs fall back to the tile height via the
getattr+try/except, so the new path can't raise on a chunkless array.
Checklist
- Output unchanged (slices still tile-aligned and complete)
- All streaming backends reach the same path; fix is backend-agnostic
- NaN handling untouched (no change to pixel transforms)
- Edge cases covered (tall chunk, ragged halo, single chunk, fallback)
- Dask chunk boundaries handled (halo bound is provably tight)
- No premature materialization added (helper is pure arithmetic)
- Benchmark not needed (memory-budget fix, no hot loop touched)
- README matrix N/A (no new function)
- Docstrings updated in both
_write_streamingand publicto_geotiff
brendancol
commented
Jun 7, 2026
Contributor
Author
brendancol
left a comment
There was a problem hiding this comment.
Follow-up review (after aeec97f)
Disposition of the first-pass findings:
- Fixed -- the suggestion to flag the non-overlap conservatism in-code. Added a comment at the budget block (
aeec97fc) explaining that a non-overlap read carries no halo, so the one-chunk halo is intentionally conservative there and may segment one step early. - Dismissed (style match) --
import bisectinside the helper. The file already importsos/tempfileinside_write_streaming, so the local import is consistent with the surrounding code. - Dismissed (out of scope, documented) -- strip-layout (
tiled=False) overlap peak. The issue scopes strip mode out and the docstring now states it does no horizontal segmentation.
No new findings. The one-line follow-up is comment-only; the streaming budget + helper tests stay green (7 passed).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3007
The streaming
to_geotiffwriter bounds peak memory withstreaming_buffer_bytes, but sized its horizontal-segment budget from the output tile geometry (tile_h * tile_w). That undercounts amap_overlapsource: slicing one tile-row out ofopen_geotiff(chunks=...).xrs.slope()materializes every source chunk-row the band touches plus a one-chunk halo. A source chunked taller than the tile (e.g.chunks=512,tile_size=256) on a wide raster pulled several full-width source chunk-rows per compute and OOMed, despite the buffer._max_streaming_row_span: the worst-case source rows a tile-band materializes (touched source chunk-rows plus a one-chunk halo each side;map_overlapdepth never exceeds one chunk, so that is an upper bound).bytes_per_tile_colfrom that span instead ofth. Falls back to the tile height for numpy-from-dask or unknown-chunk arrays. The column halo of a 2D overlap adds a bounded couple of source chunk-columns, sostreaming_buffer_bytesstays a soft cap.On the repro (2048x8192 float32,
chunks=512, slope, 8 MB cap) peak source bytes per compute drop from 32 MB to 12.6 MB.Backend coverage: writer path is backend-agnostic (numpy / cupy / dask+numpy / dask+cupy all reach the same streaming writer); the fix affects the dask streaming branch.
Test plan:
test_overlap_source_respects_buffer_3007-- instruments source-chunk materialization and asserts peak per compute stays within 2x the buffer (fails pre-fix at 4x)test_overlap_source_roundtrip_small_buffer_3007-- lazy slope under a tight cap matches eager slopetest_max_streaming_row_span_3007-- helper unit testTestStreamingBufferBudget+ dask integration suite still green (146 passed, 1 skipped)