Repository navigation

feat: Joom 1.1-1 — Delta scan, correctness fixes, wide-row performance, cost-based engine choice - #2

Merged

msafonov merged 72 commits into

branch-1.1from

joom/1.1-dev

Oct 3, 2026

Commits on Sep 26, 2026
- fix: retain Comet execution for warehouse workaround regressions
  
  notimesea
  authored and
  msafonov
  committed
  65be8b7
- fix: preserve zero partitions when native scan files are pruned
  
  notimesea
  authored and
  msafonov
  committed
  45fad9e
Commits on Sep 27, 2026
- revert: drop the sort merge reservation cap
  
  msafonov
  and
  claude
  committed
  d153b3a
- feat: spill all non-streaming native window shapes
  
  msafonov
  and
  claude
  committed
  e90fcec
- test: compare spilled window shapes with Spark
  
  msafonov
  and
  claude
  committed
  e435466
- test: reproduce native sort failing to spill after the Spark share shrinks
  
  msafonov
  and
  claude
  committed
  943936f
- build: vendor datafusion-physical-plan 55.1.0 unchanged
  
  msafonov
  and
  claude
  committed
  6eaf12e
- fix: keep native sort spill memory when the Spark share shrinks
  
  msafonov
  and
  claude
  committed
  880575b
- fix: keep the spill merge's headroom across all merge passes
  
  msafonov
  and
  claude
  committed
  b3cd74a
- fix: reserve the encoded rows a merge keeps for reuse
  
  msafonov
  and
  claude
  committed
  d7d47f3
- fix: count a single-column merge's sort key once
  
  msafonov
  and
  claude
  committed
  d82e3a3
- fix: leave as much memory again for the final spill merge's consumers
  
  msafonov
  and
  claude
  committed
  69e6389
- fix: let a sort's spill merge split a run batch wider than its budget
  
  msafonov
  and
  claude
  committed
  a4dc994
- fix: keep a sort spill's batches reserved until they are written
  
  msafonov
  and
  claude
  committed
  426ca8e
- fix: count buffers a sort's chunks share once
  
  msafonov
  and
  claude
  committed
  1b8d480
- fix: reserve a buffer a sort's buffered batches share once
  
  msafonov
  and
  claude
  committed
  6ae0323
- fix: keep the merge headroom for a sort's final in-memory merge
  
  msafonov
  and
  claude
  committed
  5b75a38
Commits on Sep 28, 2026
- fix: reserve what a sort's spill merge holds for each run
  
  msafonov
  and
  claude
  committed
  399e6c9
- fix: restore sort merge reservation cap and native test contract
  
  msafonov
  and
  codex
  committed
  4ff3708
- fix: allow small Comet batches and cap JVM shuffle batches at use time
  
  msafonov
  and
  codex
  committed
  ea38aad
- debug: measure wide batches at shuffle decode and FFI output
  
  msafonov
  and
  codex
  committed
  ca6b963
- test: bound unreserved sort output at the JVM handoff
  
  msafonov
  and
  codex
  committed
  5f6cbb8
- test: measure wide binary payload copies in sort kernels
  
  msafonov
  and
  codex
  committed
  39f49a0
- fix: avoid repeated copies of wide binary payloads in native sort
  
  msafonov
  and
  codex
  committed
  be99cc1
- test: measure dictionary expansion before wide binary sort
  
  msafonov
  and
  codex
  committed
  5da7791
- fix: select direct shuffle reads during initial planning
  
  msafonov
  and
  codex
  committed
  4956d83
- test: benchmark binary columnar-to-row allocation overhead
  
  msafonov
  and
  codex
  committed
  c0b0448
- perf: avoid intermediate binary copies in row materialization
  
  msafonov
  and
  codex
  committed
  3d99265
- test: cover default Kryo broadcast payloads
  
  msafonov
  and
  codex
  committed
  a52c168
- perf: reuse unique dictionary values instead of casting in JVM shuffle row conversion
  
  msafonov
  and
  claude
  committed
  d9ea392
- feat: revert isolated native operators feeding Spark operators
  
  msafonov
  and
  claude
  committed
  2eda535
Commits on Sep 29, 2026
- feat: run each query stage in one engine and pick shuffle formats per boundary
  
  msafonov
  and
  claude
  committed
  edaf6ca
- perf: evaluate small window partitions together and coalesce window output
  
  msafonov
  and
  claude
  committed
  95dc9ae
- fix: do not fail an aggregate whose first reservation was refused when it spills
  
  msafonov
  and
  claude
  committed
  85bef53
- fix: let the task's other memory consumers make the JVM shuffle writer spill
  
  msafonov
  and
  claude
  committed
  b65fdbe
- revert: remove UnifyStageEngines
  
  msafonov
  and
  claude
  committed
  0a92f71
- revert: remove RevertIsolatedNativeOperators
  
  msafonov
  and
  claude
  committed
  4f92985
- feat: pick shuffle and broadcast formats from the engines on both sides
  
  msafonov
  and
  claude
  committed
  01df400
- feat: choose each operator's engine by a cost over the whole plan
  
  msafonov
  and
  claude
  committed
  fd190a6
- feat: native Delta scan (port of apache/datafusion-comet#5365)
  
  msafonov
  and
  claude
  committed
  c69bd9c
- fix(delta): claim GCS scans whose Hadoop auth keys only select ADC
  
  msafonov
  and
  claude
  committed
  a435236
- fix: do not hash decimals above precision 18 in native shuffle (port of apache/datafusion-comet#6005)
  
  msafonov
  and
  claude
  committed
  88cfb29
- feat: run sorts of wide rows with narrow keys in Spark when Spark reads them
  
  msafonov
  and
  claude
  committed
  2d3cc89
- feat: move sorts with variable-width payloads to Spark on the initial plan
  
  msafonov
  and
  claude
  committed
  909335e
- test: time native sorts of wide binary payloads end to end
  
  msafonov
  and
  claude
  committed
  3b37b4a
- perf: sort the keys of wide rows and gather each payload once in native sort
  
  msafonov
  and
  claude
  committed
  d796f09
Commits on Sep 30, 2026
- fix: gather a native sort's output from the batches it references
  
  msafonov
  and
  claude
  committed
  f66011d
- refactor: keep wide-row sorts in Spark across AQE re-plans without per-thread state
  
  msafonov
  and
  claude
  committed
  9ff3736
- fix: count the encoded rows a merge keeps instead of summing them per batch
  
  msafonov
  and
  claude
  committed
  b2771dd
- perf: coalesce small input batches of a native sort before buffering them
  
  msafonov
  and
  claude
  committed
  f7a13eb
Commits on Oct 1, 2026
- fix(delta): do not resolve subqueries when planning reads the scan's partitioning
  
  msafonov
  and
  claude
  committed
  0e17ce3
- fix: rebase the offsets of sliced build batches when coalescing a broadcast
  
  msafonov
  and
  claude
  committed
  d913126
- fix: keep the streamed order of a sort-merge join with a join filter
  
  msafonov
  and
  claude
  committed
  f761b31
- perf: coalesce small blocks when reading a Comet shuffle
  
  msafonov
  and
  claude
  committed
  2d30872
- feat: keep shuffles of wide rows in Spark
  
  msafonov
  and
  claude
  committed
  c2562fb
- perf: reuse generated columnar-to-row projections across tasks
  
  msafonov
  and
  claude
  committed
  9560d4a
- feat: decide wide-row sorts and shuffles by leaf columns
  
  msafonov
  and
  claude
  committed
  b83e917
Commits on Oct 2, 2026
- fix: spill a large in-memory sort before producing its output
  
  msafonov
  and
  claude
  committed
  568955d
- perf: gather wide shuffle rows from the batches each partition references
  
  msafonov
  and
  claude
  committed
  94abcc0
- feat: price the cost-based engine choice by rows and leaf columns
  
  msafonov
  and
  claude
  committed
  d1148b2
- feat: price the cost-based engine choice per row and decide it on every plan
  
  msafonov
  and
  claude
  committed
  cea0ae5
- feat: enable the cost-based engine choice by default
  
  msafonov
  and
  claude
  committed
  f9793a7
Commits on Oct 3, 2026
- feat: price the cost-based engine choice with the calib29 measurements
  
  msafonov
  and
  claude
  committed
  1f7b89b
- fix: clone the shared sort source with Arc::clone in the handoff test
  
  msafonov
  and
  claude
  committed
  6b11f2e
- feat: keep the cost-based engine choice disabled by default
  
  msafonov
  and
  claude
  committed
  71a7c07
- test: give the child sort spill metrics test less memory so the sort spills
  
  msafonov
  and
  claude
  committed
  2b28fed
- feat: keep PartitionAggregateWindowExec disabled by default
  
  msafonov
  and
  claude
  committed
  bc64dfc
- fix: pool C2R projections only after every reader of their rows is done
  
  msafonov
  and
  claude
  committed
  8e06527
- ci: list the missing suites so check-suites.py passes
  
  msafonov
  and
  claude
  committed
  bb53574
- style: drop a needless string interpolator flagged by scalafix
  
  msafonov
  and
  claude
  committed
  c8b47b4
- build: compile with Scala 2.13 and Rust 1.99
  
  msafonov
  and
  claude
  committed
  1a52c8c
- style: drop an unused pattern binding flagged by scalafix on Scala 2.13
  
  msafonov
  and
  claude
  committed
  4f150e1

Commits

Commits on Sep 26, 2026

Commits on Sep 27, 2026

Commits on Sep 28, 2026

Commits on Sep 29, 2026

Commits on Sep 30, 2026

Commits on Oct 1, 2026

Commits on Oct 2, 2026

Commits on Oct 3, 2026