fix(precompute): force-close open windows on worker shutdown - #376
Merged
Merged
Conversation
flush_all only advances the watermark by +1ms (plus the wall-clock fallback, whose grace may not have elapsed for a one-shot batch), so a batch whose records all fall in a single window — event-time never advancing past the window end — leaves the trailing window open and its data never reaches the store. Add Worker::force_close_all(), invoked from the Shutdown handler after the final flush_all(): for every bucket with open panes, advance to a finite bound (max_pane + window_size_ms) and emit every remaining window, covering both active_panes (sample aggregation) and sketch_panes (OTLP sketches). A finite bound is used deliberately — WindowManager::closed_windows enumerates window starts one slide at a time, so passing i64::MAX would loop ~i64::MAX/slide times and overflow. Idempotent: drained panes are removed and their wall-clock bookkeeping pruned. Tests: shutdown_force_close_emits_trailing_sample_window and shutdown_force_close_emits_trailing_sketch_window. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
flush_allonly advances the per-group watermark by+1ms(plus the wall-clock fallback, which is gated on grace having elapsed on a flush tick). For a one-shot batch ingest whose records all fall in a single window — so event-time never advances past the window end — the trailing window never closes,emit_batchis never called, and that window's data never reaches the store.The
Shutdownhandler previously just calledflush_all(), so the trailing window was lost.Fix
Add
Worker::force_close_all(), invoked from theShutdownhandler after the finalflush_all(). For every bucket with open panes it closes and emits every remaining window unconditionally (no more samples will arrive after shutdown), covering both pane maps:active_panes(sample aggregation) →merge_panes_for_windowsketch_panes(OTLP-delivered sketches) →merge_sketch_panes_for_windowTo advance past the open windows it uses a finite bound derived from the largest open pane (
max_pane + window_size_ms), noti64::MAX:WindowManager::closed_windowsenumerates window starts one slide at a time, soi64::MAXwould loop ~i64::MAX/slidetimes and overflow.max_pane + window_size_msis the smallest watermark that closes the latest open window.Idempotent: drained panes are removed from both maps and their wall-clock bookkeeping is pruned, so a second call emits nothing.
Tests
shutdown_force_close_emits_trailing_sample_window(active_panes path)shutdown_force_close_emits_trailing_sketch_window(sketch_panes path, 10 DDSketches at frozen event-time → one merged window emitted withtotal_count() == 10)Verification
cargo check --lib --tests— clean (pre-existing warnings only)precompute_engine::worker— 30 passed (incl. 2 new)precompute_engine(module) — 241 passedContext
Companion to ProjectASAP/ASAPQuery#400, which ports the wall-clock grace fallback into the frontend and adds the same shutdown force-close there. This PR adds the shutdown force-close to the backend, where the wall-clock fallback already exists.
🤖 Generated with Claude Code