Replace the D8 code-to-offset if/elif chain with a lookup table (#3738) - #3743
Open
brendancol wants to merge 2 commits into
Open
Replace the D8 code-to-offset if/elif chain with a lookup table (#3738)#3743brendancol wants to merge 2 commits into
brendancol wants to merge 2 commits into
Conversation
_code_to_offset mapped a D8 direction code to its (dy, dx) neighbour offset with an eight-way if/elif chain. _flow_accum_cpu calls it twice per cell and stream_link_d8, flow_length_d8, flow_path_d8, stream_order_d8 and hand_d8 import it for their own per-cell loops. Index two module-level int64 arrays by the code instead. numba freezes module-level arrays as compile-time constants, so the call compiles to a range check and two loads. The range check is written as an inside-the-box test (0 <= c <= 128) on purpose: int(nan) is INT64_MIN in numba and there is no bounds checking, so a rejection-form guard would let a NaN code index out of bounds (same class as the erode fix in #3703). Non-power-of-two codes in range and anything outside 0..128 keep returning (0, 0), and the returns stay integer. On a 1000x2000 float64 flow-direction raster _flow_accum_cpu dropped from 56.3 ms to 39.7 ms (median of 7 after warmup) with identical results, on both the clean raster and one built from a DEM with 30% NaN. Add a parametrised test pinning the mapping for all eight codes, 0, in-range non-codes, out-of-range ints, floats and NaN, plus a check that only the eight D8 entries of the tables are populated.
brendancol
commented
Sep 4, 2026
brendancol
left a comment
Contributor
Author
There was a problem hiding this comment.
PR Review: Replace the D8 code-to-offset if/elif chain with a lookup table (#3738)
Blockers (must fix before merge)
None.
Suggestions (should fix, not blocking)
-
xrspatial/hydro/flow_accumulation_d8.py:214-216: the comment says ac < 0 or c > 128guard would not stop a NaN code. That is not what happens here.cis already an integer at that point, and on x86int(nan)lands onINT64_MIN, whichc < 0does reject. The hazard that actually bites is comparing the float before the conversion (code < 0 or code > 128is False for NaN, so it falls through toint(nan)), or relying on whatfptosidoes with NaN at all, since LLVM treats that as poison and other targets return a different value (aarch64fcvtzsgives 0). Reword the comment to describe that, so the next reader does not learn the wrong rule from it. -
xrspatial/hydro/flow_accumulation_d8.py:213: given the above, an explicitif code != code: return 0, 0beforeint(code)would make NaN handling independent of the conversion semantics instead of depending on the integer guard catching whateverfptosiproduced. The bounds-checked test proves the current x86 behaviour but not the portable one. Worth a quick timing check that the extra compare is free on the 1000x2000 case; if it is, add it.
Nits (optional improvements)
-
xrspatial/hydro/tests/test_flow_accumulation_d8.py:664-668and:677: the two new tests import_code_to_offset,_code_to_offset_py,_D8_DXand_D8_DYinside the function bodies. The rest of the file imports at module level; move these up to match. - The PR touches the kernel behind
FlowAccumulationinbenchmarks/benchmarks/flow_accumulation.py, but the path-based labeler may not tag hydro asperformance. Add the label by hand so the asv job runs on this PR.
What looks good
- The guard is written as
0 <= c <= 128rather than a rejection test, and the PR includes a negative control showingNUMBA_BOUNDSCHECK=1raises on the unguarded form with a NaN code. - Returns stay
int64from the tables, sor + dyarithmetic in the seven importing modules is unchanged. The old literal tuples also typed asint64, so no recompilation surprises for callers. - The A/B against
origin/maincovers both a clean raster and one with 30% NaN, usingarray_equal(equal_nan=True)rather thanallclose. _code_to_offset_pyis left alone, so the boundary-stitching code in watershed_d8, flow_length_d8 and hand_d8 that uses it is unaffected.- The parametrised test pins the mapping explicitly (not by comparing to the Python version), so a future edit to either table is caught.
Checklist
- Algorithm matches reference (same mapping as the chain, verified for every probe)
- All implemented backends produce consistent results (numpy and dask+numpy share the kernel; GPU untouched)
- NaN handling is correct (see suggestion on the comment wording)
- Edge cases are covered by tests (0, 3, 5, 129, 255, -1, 1e9, floats, NaN)
- Dask chunk boundaries handled correctly (no change to tiling)
- No premature materialization or unnecessary copies
- Benchmark exists (
FlowAccumulation) - README feature matrix: not applicable, private helper
- Docstrings present and accurate
…est imports (#3738) Return (0, 0) for a NaN code before calling int() rather than relying on the integer guard catching whatever the float-to-int conversion produced (INT64_MIN on x86, 0 on aarch64, poison to LLVM). Reword the guard comment to describe the real hazard, a rejection test on the float that is False for NaN, instead of claiming an integer rejection test would miss it. The extra compare is free: 36.5 ms and 37.1 ms on two runs of the 1000x2000 case versus 39.7 ms before it, all within noise, output still identical to origin/main. Move the private imports in the new tests up to module level to match the rest of the file.
brendancol
commented
Sep 4, 2026
brendancol
left a comment
Contributor
Author
There was a problem hiding this comment.
PR Review: Replace the D8 code-to-offset if/elif chain with a lookup table (#3738), follow-up
Second pass after 2405f0c.
Blockers (must fix before merge)
None.
Suggestions (should fix, not blocking)
None.
Nits (optional improvements)
None.
Disposition of the first-pass findings
- Guard comment wording: fixed.
flow_accumulation_d8.py:213-221now names the real hazard (a rejection test on the float is False for NaN) instead of claiming an integer rejection test would miss it. - Explicit NaN check before
int(): fixed.if code != code: return 0, 0sits ahead of the conversion, so the table index no longer depends on whatfptosidoes with NaN on a given target. Timed at 36.5 ms and 37.1 ms on the 1000x2000 case versus 39.7 ms without it, so the compare is in the noise, and the A/B againstorigin/mainis still bitwise identical on both rasters. - Module-level test imports: fixed.
test_flow_accumulation_d8.py:6-11. performancelabel: added to the PR.
What looks good
- The NaN path is now tested three ways: the parametrised case, the same test under
NUMBA_BOUNDSCHECK=1, and the A/B on a 30% NaN raster. - The bench script's
xrspatial.__file__assertion guards against the worktree-vs-main editable install trap, so the numbers are for the branch under review.
Checklist
- Algorithm matches reference
- All implemented backends produce consistent results
- NaN handling is correct
- Edge cases are covered by tests
- Dask chunk boundaries handled correctly
- No premature materialization or unnecessary copies
- Benchmark exists
- README feature matrix: not applicable
- Docstrings present and accurate
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3738
_code_to_offsetinflow_accumulation_d8.pymapped a D8 code to its(dy, dx)offset with an eight-wayif/elifchain._flow_accum_cpucalls it twice per cell, and stream_link_d8, flow_length_d8, flow_path_d8, stream_order_d8 and hand_d8 import it for their own per-cell loops.int64lookup arrays indexed by the code. numba freezes module-level arrays as compile-time constants, so the call is now a range check and two loads.0 <= c <= 128.int(nan)isINT64_MINinside numba and there is no bounds checking, so a rejection-form guard would let a NaN code index out of bounds (same class as theerode()fix in erode(): NaN input causes an out-of-bounds read and floods the output with NaN #3703). Codes like 3 or 5, anything above 128, negative codes and NaN keep returning(0, 0), and the returns stay integer sor + dyin the callers is unchanged._code_to_offset_pyis untouched.On a 1000x2000 float64 flow-direction raster (Gaussian bump plus
default_rng(71942).normal(0, 2)noise),_flow_accum_cpuwent from 56.3 ms to 39.7 ms, median of 7 after warmup, with identical results. The output is also identical on a raster built from the same DEM with 30% random NaN, checked withnp.array_equal(..., equal_nan=True)against the function pulled fromorigin/main.Backends: numpy and dask+numpy share this CPU kernel. cupy and dask+cupy use the device-side dispatch and are not touched.
Test plan
_code_to_offsetcovering the eight codes, 0, 3, 5, 129, 255, -1, 1e9, float codes and NaN, asserting the same tuples the old chain returned.NUMBA_BOUNDSCHECK=1, plus a negative control confirming the flag raisesIndexErroron an unguarded lookup with a NaN code.pytest xrspatial/hydro/tests -k "d8 or flow_accum or flow_length or stream_link or stream_order or watershed or flow_path or hand or basin": 872 passed.origin/mainon the clean and 30% NaN rasters: bitwise identical.The
FlowAccumulationasv class inbenchmarks/benchmarks/flow_accumulation.pyalready covers this path, so no benchmark change.