[carloea2]

What changes were proposed in this PR?

Hash LARGE_BINARY URI keys using Java-compatible UTF-16 code units. This supports tuple hashing and shuffle routing for ordinary and supplementary Unicode URIs while preserving null hashing.

Refreshed onto upstream main at 86fbe56. The full frontend CI command passes locally without frontend changes.

Any related issues, documentation, discussions?

Closes #8185
Closes #8243

Replaces #8186 and consolidates #8244.

How was this PR tested?

Fresh detached worktree, Python 3.12 and Node 24.14.1 on Windows. The five tuple and partitioner regressions fail with KeyError before the source fix.

After generating Python protobuf files with bash bin/python-proto-gen.sh, from amber:

python -m pytest src/test/python/core/models src/test/python/core/architecture/sendsemantics/test_partitioners.py --deselect src/test/python/core/models/test_tuple.py::TestTuple::test_hash -q --tb=short -p no:cacheprovider
python -m ruff check src/main/python src/test/python
python -m ruff format --check src/main/python src/test/python
git diff --check

368 passed, one expected failure. Two parameterized cases of the existing Windows epoch test were deselected; that separate change is #8957. All LARGE_BINARY regressions ran. Ruff passed for all 217 Python files.

From frontend:

corepack yarn install --immutable
corepack yarn test:ci
corepack yarn format:ci

248 test files passed: 7,020 tests passed and one skipped. Formatting and lint checks passed. Remote frontend coverage needs the frontend label, which requires a maintainer to add.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Codex

[carloea2]

This replaces stale-closed #8186. The problem still reproduces on current main. The patch passes its regression, 349 broader tests and Ruff checks.

/request-review @eugenegujing

[carloea2]

/request-review @eugenegujing

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• Contributors with relevant context: @eugenegujing, @Ma77Ball, @aglinxinyuan
  You can notify them by mentioning @eugenegujing, @Ma77Ball, @aglinxinyuan in a comment.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.17%. Comparing base (86fbe56) to head (0285b24).

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8955      +/-   ##
============================================
- Coverage     92.17%   92.17%   -0.01%     
  Complexity     4987     4987              
============================================
  Files          1284     1284              
  Lines         55605    55608       +3     
  Branches       7120     7120              
============================================
  Hits          51256    51256              
- Misses         2591     2593       +2     
- Partials       1758     1759       +1     

Flag	Coverage Δ		*Carryforward flag
access-control-service	77.44% <ø> (ø)		Carriedforward from 86fbe56
agent-service	99.54% <ø> (ø)		Carriedforward from 86fbe56
amber	86.84% <ø> (ø)		Carriedforward from 86fbe56
computing-unit-managing-service	64.02% <ø> (ø)		Carriedforward from 86fbe56
config-service	87.61% <ø> (ø)		Carriedforward from 86fbe56
file-service	82.29% <ø> (ø)		Carriedforward from 86fbe56
frontend	96.48% <ø> (ø)		Carriedforward from 86fbe56
notebook-migration-service	83.77% <ø> (ø)		Carriedforward from 86fbe56
pyamber	98.48% <100.00%> (-0.07%)	⬇️
workflow-compiling-service	77.95% <ø> (ø)		Carriedforward from 86fbe56

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 3 worse · ⚪ 10 noise (<±5%) · 0 without baseline

Compared against main 86fbe56 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	544	0.332	17,478/24,340/24,340 us	🟢 -10.7% / 🔴 +72.1%
⚪	bs=100 sw=10 sl=64	1,241	0.758	79,897/96,660/96,660 us	⚪ within ±5% / 🟢 -11.0%
🔴	bs=1000 sw=10 sl=64	1,459	0.891	684,090/796,395/796,395 us	🔴 +9.0% / 🟢 +24.3%

Baseline details

Latest main 86fbe56 from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	544 tuples/sec	571 tuples/sec	881.37 tuples/sec	-4.7%	-38.3%
bs=10 sw=10 sl=64	MB/s	0.332 MB/s	0.348 MB/s	0.538 MB/s	-4.6%	-38.3%
bs=10 sw=10 sl=64	p50	17,478 us	16,278 us	11,399 us	+7.4%	+53.3%
bs=10 sw=10 sl=64	p95	24,340 us	27,262 us	14,143 us	-10.7%	+72.1%
bs=10 sw=10 sl=64	p99	24,340 us	27,262 us	17,513 us	-10.7%	+39.0%
bs=100 sw=10 sl=64	throughput	1,241 tuples/sec	1,281 tuples/sec	1,148 tuples/sec	-3.1%	+8.1%
bs=100 sw=10 sl=64	MB/s	0.758 MB/s	0.782 MB/s	0.701 MB/s	-3.1%	+8.2%
bs=100 sw=10 sl=64	p50	79,897 us	77,049 us	89,815 us	+3.7%	-11.0%
bs=100 sw=10 sl=64	p95	96,660 us	100,966 us	95,717 us	-4.3%	+1.0%
bs=100 sw=10 sl=64	p99	96,660 us	100,966 us	107,566 us	-4.3%	-10.1%
bs=1000 sw=10 sl=64	throughput	1,459 tuples/sec	1,487 tuples/sec	1,175 tuples/sec	-1.9%	+24.2%
bs=1000 sw=10 sl=64	MB/s	0.891 MB/s	0.908 MB/s	0.717 MB/s	-1.9%	+24.3%
bs=1000 sw=10 sl=64	p50	684,090 us	673,797 us	885,446 us	+1.5%	-22.7%
bs=1000 sw=10 sl=64	p95	796,395 us	730,678 us	933,809 us	+9.0%	-14.7%
bs=1000 sw=10 sl=64	p99	796,395 us	730,678 us	979,021 us	+9.0%	-18.7%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,367.50,200,128000,544,0.332,17477.58,24339.52,24339.52
1,100,10,64,20,1611.34,2000,1280000,1241,0.758,79897.36,96659.98,96659.98
2,1000,10,64,20,13703.62,20000,12800000,1459,0.891,684090.12,796394.82,796394.82

[carloea2]

/request-review @mengw15

Could you review the large binary hash routing? The regression and broader tests pass locally, and CI is green.

[carloea2]

/request-review @sshiv012 @parshimers

[eugenegujing]

LGTM. One small thing: the test only checks Tuple.hash, but the KeyError actually came out of HashBasedShufflePartitioner._route. There's already test_same_key_routes_to_same_receiver_deterministically in test_partitioners.py using an INTEGER key, might be worth copying it with a LARGE_BINARY key so the partitioner path is covered too.

[carloea2]

@eugenegujing Added the partitioner-level LARGE_BINARY test you suggested, including a non-BMP URI. It reproduces the original KeyError. The focused suites give 144 passes; the existing Windows epoch-hashing test remains separate in #8957. Ruff passes, and CI has restarted.

[carloea2]

@mengw15, could you add the frontend label so those CI checks run too? The full frontend suite passes locally, but I don't have permission to add the label.

[carloea2]

/request-review @mengw15

Refreshed onto main with partitioner coverage. All required checks are green. Could you review this head?