Skip to content

Window implementation candidates: sources, rewrites, and simplification questions #705

Description

@milindsrivastava1997

Context

A query's WindowImplementationCandidate list (PlanningQuery.window_implementations) is built from several sources and then rewritten in several places. This issue records how that works today and lists open questions, with the goal of simplifying the interface.

Where candidates come from

# Source Code
S1 Snapshot implementation.window_candidates: HashMap<PromQL, Vec<WindowImplementationCandidate>> (provider-priced, never rewritten) control_plane/src/physical/compiler.rs:226, :547, :621
S2 HTTP PhysicalPlanQueryRequest.window_implementations (compile-and-publish / cost-manifests), passed through as-is control_plane/src/main.rs:578, :959
S3 derived_window_candidates (one production caller: planning_request), used when S1 has no entry for the query control_plane/src/physical/compiler.rs:2435

S3 today, for each range-selector window W (or the lookback if the query has none), with evaluation interval E:

  • E < W && W % E == 0: Sliding, slide = E, two candidates: Pane{E} and FullWindow
  • otherwise: Tumbling, slide = W, one candidate: Pane{W}
  • Cost: derived_window_cost scales the snapshot's single implementation_cost template.

Rewrites and filters

# What Code
R1 After logical selection, for S3 queries only: candidates at derived-maintenance windows are collapsed to one Tumbling Pane{W} (…-derived-cohort) and repriced compiler.rs:684-727
R2 Hybrid mode drops states with no candidate at their window; each state keeps only candidates at its own window compiler.rs:1082, :1188
R3 Pane sharing makes shared-pane-<fp>, priced from window_implementations.first(), S3 queries only physical/pane_reuse.rs:80

Consumption: validate_window_implementationsControlPlaneCostModel::with_window_implementation_costsselect_lifecycle → Planner plan_summary_maintenance_lifecyclesselected_window_implementation_id.

Questions

  1. Can S1 be deleted? No production caller sets implementation.window_candidates, and no checked-in snapshot does either. Its only users are tests: compiler.rs (supplied_window_candidates_are_not_replaced_by_the_derivation, ~:5482), and data_plane/tests/asapquery_compatibility_process_e2e.rs:789 (run_shared_dashboard(multi_pane=true)), which injects a 5s pane with a hand-set cost to force the choice. Deleting it removes window_candidates, the key check at :547, the contains_key branches at :621/:673/:741, and makes synthesized_window_queries always cover every query. What should replace provider-priced pane quotes in those tests? Should the backend keep a way to accept measured per-pane costs, or is the derived formula the intended source of truth?
  2. Should S2 derive candidates too? The HTTP path is the only entry for DistributedCollectors (the snapshot path requires BackendLocalRemoteWrite). Its only in-repo caller is data_plane/tests/backend_process_e2e.rs. Should it keep caller-supplied candidates, or share S3?
  3. Intermediate pane sizes. Proposal: also offer pane p with E < p < W, p % E == 0, W % p == 0. But WindowMaterializationLayout::validate (crates/asap_types/src/aggregation_config.rs:50) requires the pane to divide both window and slide. With slide = E, a flat Pane{p} with p > E is rejected, and readouts at every E tick could not align with pane ends anyway. Options:
    • a) HierarchicalRollup { base_pane_secs: E, levels_secs: [p] }. The compiler accepts it only as Extension("backend.exact-hierarchical-rollup.v1") (compiler.rs:2561), and the data plane has no implementation today.
    • b) Finer panes p | gcd(E, W), p < E. These never beat Pane{E} for one query, but could enable sharing across queries with different E.
    • c) Change the slide to p. That changes evaluation alignment, so off-tick evaluations fall back to exact.
      Which is intended?
  4. Tumbling fallback when W % E != 0 or E > W. Example: W = 300s, E = 45s produces Tumbling Pane{300}, slide 300. Evaluations every 45s rarely line up with pane ends, so most reads should fail the serving pane-alignment check and go to exact. Should S3 use Pane{gcd(E, W)} with slide gcd(E, W) instead?
  5. R1 / R3 only apply to S3 queries. If S1 goes away, is the "synthesized" distinction still needed at all?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions