diff --git a/docs/design_docs/empirical-o11y-execution-plan.md b/docs/design_docs/empirical-o11y-execution-plan.md new file mode 100644 index 00000000..7ffbdcac --- /dev/null +++ b/docs/design_docs/empirical-o11y-execution-plan.md @@ -0,0 +1,67 @@ +# Offline evidence and o11y planning evaluation + +This is the historical offline-evidence milestone plan. The current prototype +adds actual backend-only data-plane execution using the supplied OpenMetrics +dataset, as described in the [current evaluation guide](../user-guide/o11y-replay.md). +Benchmark production and execution tooling now belong to ASAPQuery-backend; +planner-only coverage and synthetic cached-result timing are not its acceptance criteria. + +Audience: developers reproducing issue #322 and the planner/control-plane evaluation. + +## Scope + +Use offline sketch-bench CPU, elapsed time, state/memory, disk (when measured), +and errors against offline ground truth. No runtime ground truth, posterior +feedback, or self-estimated accuracy is required. Offline accuracy observations +do not establish formal guarantees on unseen distributions. + +## Execution and ownership + +1. Pin isolated planner and control-plane worktrees from fetched main branches. + Preserve existing worktrees, including unresolved cost-model changes. +2. Evidence agent: implement the versioned artifact, validation, compatibility + matching, public cost-model integration, and fallback/decision tests. +3. Benchmark agent: run actual sketch-bench algorithms on uniform and Zipf + inputs, preserve raw output, export artifacts with source and environment + provenance, and document reproducible commands. +4. Replay agent: run the existing o11y PromQL corpus through the backend parser, + its ASAPPlanner call, and the backend typed binder (not a planner-only entry), + export binding/fallback coverage and planning latency, and separately identify + supplemental sketch workloads. +5. Integration owner: connect compatible evidence to the downstream control + plane, validate dependency compatibility, review other agents' changes, run + integration checks, and report evidence-supported comparisons. + +Steps 2–4 run concurrently after agreeing the artifact contract. Integration +uses their completed interfaces and measurements. Query-pattern changes are +limited to demonstrated blockers with regression coverage; unsupported query +semantics remain explicit in the report. + +## Acceptance + +- Versioned schema and example; explicit units, algorithm/configuration, + dataset/distribution, environment, collection/validity times, sample count, + dispersion, and benchmark/model provenance. +- At least two actual sketch algorithms measured offline on uniform and skewed + inputs. CPU is distinct from elapsed time; serialization size is distinct + from heap memory and disk I/O. Unmeasured fields stay unavailable. +- Matching evidence affects a public planning cost/ranking/lifecycle boundary. + Tests cover changed decisions, missing/stale/mismatched evidence, invalid + values, and unchanged accuracy guarantees. +- The planner and control-plane evaluation preserve exact fallbacks and expose + unsupported shapes, measured provenance, and limits on benefit estimates. +- Reproducible measurement and replay commands, raw machine-readable results, + and a concise report distinguish actual measurements from modeled totals. + +## Comparison rules + +Compare equivalent tasks, data, parameters, windows, horizons, and evaluation +cadences. Whole-plan estimates include build/update/readout, retained windows, +sharing, and raw residual work where evidence exists. Missing raw baseline or +physical evidence means an unavailable whole-plan speedup, not zero cost. +Microbenchmark algorithm comparisons are labeled separately. Disk usage and +network savings require their own measurements or explicit models. + +Repeated-query break-even can be computed only when compatible exact baseline, +sketch build/maintenance, and readout measurements are present. It is an offline +estimate and does not establish deployed end-to-end latency improvement. diff --git a/docs/offline-o11y-final-2026-09-08.md b/docs/offline-o11y-final-2026-09-08.md new file mode 100644 index 00000000..f361557c --- /dev/null +++ b/docs/offline-o11y-final-2026-09-08.md @@ -0,0 +1,102 @@ +# Historical Offline Sketch Evidence and o11y Report + +This report records the earlier offline experiment, not the backend-only +prototype's execution results. Its synthetic exact-result cache experiment is +out of the current prototype scope; its source tool was removed from this PR. +Current work uses user-provided OpenMetrics data and queries entering the backend, +followed by ASAPPlanner and actual data-plane execution. No execution speedup or +resource reduction is established by the historical results below. + +Audience: developers. This report covers the offline evidence path for #322, +without real-time error feedback. The original dirty working directory was left unchanged. + +## Completed Integration + +- Versioned artifacts, JSON Schema, and a public CostModel provider match parameters, query semantics, distributions, environments, and validity intervals. Unknown measurements remain unavailable. +- Actual CMS/CountSketch parameter sweeps measure offline error, separate construction/update/read/merge CPU, live heap allocations, serialized size, and allocated file blocks. +- Fixed-snapshot point-frequency comparisons use explicit observed-error thresholds, formal parameter minima, CPU/retained byte-seconds weights, and an exact baseline on the same data. +- The backend filters unsupported layouts before recommending and binding frequency configurations. When exact execution wins or evidence is insufficient, it preserves the original query without silently rounding selected parameters. +- o11y planner/control-plane replay and exact fixed-snapshot reference measurements cover seven explicitly supported queries. + +For interfaces and reproduction instructions, see the [evidence contract](developer_docs/offline-sketch-evidence.md), +[backend benchmark commands](https://github.com/ProjectASAP/ASAPQuery-backend/blob/codex/empirical-o11y-322/tools/empirical-bench/README.md), and [backend-entry replay guide](user-guide/o11y-replay.md). + +## Measured Results + +sketch-bench is pinned to revision `87f619e843fd2e4da784160d4e205a0d0d55f032`. +The uniform and Zipf datasets each contain 20,000 i64 keys, with a key space of +1,000, seed 42, and Zipf exponent 1.1. There are 18 sketch configurations and +two exact baselines; CPU measurements use five trials after two warmups. +Accuracy comes from one offline experiment per fixed dataset, not a guarantee +across distributions or real-time ground truth. The comparison assumes one +build, 1,000 queries, 300 seconds of retention, and no merges. + +| Scenario | Uniform / Zipf results | +| --- | --- | +| CPU-only, with a 1% or 5% observed mean relative-error limit | Both select exact execution; no sketch CPU benefit | +| Generic comparator, with memory weighting | CMS 2720×5; retained heap decreases by 81.68%, while CPU increases by approximately 0.037 / 0.182 ms, respectively | +| Backend-compatible configuration, with memory weighting | CMS 4096×5; retained heap decreases by approximately 72.4%, while CPU also increases | + +The memory-weighted objective is `CPU ns + 0.01 × retained byte-seconds`. These +weights express an illustrative preference, not a conversion measured from +hardware. Exact retained heap is 296,976 B; CMS 2720×5 uses 54,400 B, and +4096×5 uses 81,920 B. Memory figures measure requested allocation bytes using +a separate System allocator probe; CPU measurements use jemalloc. These memory +figures are not process RSS. Serialized size and allocated file blocks do not +establish disk-throughput benefits. + +See `results-sweep/MEASUREMENTS.md` in the [verified experiment archive](https://github.com/ProjectASAP/ASAPQuery-backend/blob/codex/empirical-o11y-322/tools/empirical-bench/ARTIFACTS.md). +Earlier frequency measurements and control-plane replay in `results/` remain +as the initial baseline. Final frequency comparisons and control-plane results +use `results-sweep/`; costs from the two runs must not be mixed. + +## o11y Coverage and Limitations + +The replay reuses the repository's existing 27 PromQL fixtures. It does not run +upstream LLM-agent scoring or use real scenario data. + +The supported evaluation entry point is now the backend's `offline_planner_replay`: +queries enter its parser, call ASAPPlanner, and return through its typed binder. +The standalone planner-only replay was removed. Its candidate/lifecycle counts +below remain historical diagnostics, not backend support coverage. + +| Check | Result | +| --- | --- | +| Planner candidate coverage | 26/27 have exact summary candidates; 1/27 uses raw fallback; no sketch candidates | +| Complete lifecycle costs in the original replay | Insufficient evidence; all 27/27 conservatively fall back to raw execution | +| Control-plane exact/default/empirical binding | All three modes bind 27/27 successfully; each retains the original query in 5 root plans | +| Seven exact reference queries | Raw execution and cached-result reads are measured on fixed synthetic gauge snapshots, using the public CostModel and actual selection flow | +| Remaining 20 queries | No complete reference implementation; benefits remain unavailable | + +New fallbacks preserve the original IR for selector, sort, and comparison/filter +roots, increasing successful binding from 24/27 to 27/27. Binding does not mean +every plan is executable by the summary executor. Column updates in the new IR +are supported; keyed/non-column updates and warm-tier BinaryOp remain explicitly rejected. + +The seven reference queries cover instant sums, window maxima, and sums of +window averages: 300 series, three jobs, one finite gauge sample per minute, +and 60 repeated reads of the same snapshot. The retained state is the complete +exact query result, not a sliding window with advancing time. Measurements +exclude network, disk, protocol serialization, and output-label materialization; +memory figures count logical value bytes. Benefits from repeated reads cannot +be extrapolated to production end-to-end speedups. Machine-readable results are +available as `results/o11y-exact-snapshot.json` in that archive. + +Quantile measurements, real o11y traces, time drift, post-merge error, online +update and retirement costs, and complete production benefits remain outside +this coverage. Point-frequency error cannot be applied to quantiles or count_over_time. + +## Validation and Delivery + +- Planner mapping: 349 unit tests pass. +- Devtools: 3 exact benchmark tests and 4 replay tests pass; the recommendation CLI is verified with six actual requests. +- Benchmark export: 5 Python tests pass. +- Backend: 633 control-plane and 973 data-plane library tests pass; 7 new integration tests pass. +- Agents cross-reviewed the provider, measurement, and binding code; artifact text, JSON, source hashes, and provenance were checked successfully. + +The backend pins the published planner core commit +`dfdf6b5c1f7d667394a4ea1f56fb3786af04228d`. Normal builds and final replay do not +require local Cargo source patches. Core planner code, benchmark tools, and backend +integration are reviewed in separate PRs. Full generated results are distributed +as an experiment artifact, not embedded in the source diff. No PR is automatically +merged and the issue is not automatically closed. diff --git a/docs/user-guide/o11y-replay.md b/docs/user-guide/o11y-replay.md new file mode 100644 index 00000000..b6e4ab71 --- /dev/null +++ b/docs/user-guide/o11y-replay.md @@ -0,0 +1,85 @@ +# Offline o11y evaluation through the backend + +For developers evaluating the actual control-plane planning integration. +The canonical path is: + +`PromQL corpus -> ASAPQuery-backend parser -> ASAPPlanner -> backend typed binder/fallback` + +There is no standalone planner-only `o11y_replay` command. Run the backend's +`offline_planner_replay` example through its reproducibility wrapper instead. + +## Run the control-plane replay + +From an ASAPQuery-backend checkout, with its normal sibling dependencies available: + +```sh +python3 tools/run-offline-planner-replay.py \ + --queries /path/to/ASAPPlanner/crates/frontend-promql/tests/observability/data/o11y_bench_promql.txt \ + --evidence /path/to/planner-evidence.json \ + --context /path/to/context.json \ + --output /tmp/control-plane-o11y.json +``` + +The backend uses its pinned planner dependency; a planner source checkout is +only needed here to locate the vendored query fixture. Omit evidence and context +together to run exact/default modes only. Supplying both adds empirical mode. +An optional `--planner /path/to/ASAPPlanner` enables a local source override for +development; it is not the default evaluation path. + +The [backend guide](https://github.com/ProjectASAP/ASAPQuery-backend/blob/codex/empirical-o11y-322/control_plane/docs/offline-sketch-evidence.md) +documents the actual parser/binder and evidence assumptions. +The [benchmark driver](https://github.com/ProjectASAP/ASAPQuery-backend/blob/codex/empirical-o11y-322/tools/empirical-bench/README.md) +and [artifact instructions](https://github.com/ProjectASAP/ASAPQuery-backend/blob/codex/empirical-o11y-322/tools/empirical-bench/ARTIFACTS.md) +also live in the backend repository. + +## Interpret coverage conservatively + +- A parse failure is not a planner or backend execution result. +- A bound plan means the backend accepted a plan returned through its planner integration. +- `root_raw_fallback` means the complete original query remains selected; this is not summary acceleration. +- `raw_subtrees` includes source scans underneath summaries and cannot be counted as root fallback. +- `planning_elapsed_ns` measures the parser/binder path, not query execution. +- Binding does not establish deployable placement, successful data-plane execution, or resource savings. + +The recorded final evaluation bound all 27 fixtures in exact/default/empirical +modes; each mode retained five raw root fallbacks. Earlier standalone planner +candidate counts are historical diagnostics, not backend support coverage. +No collectors, storage services, query servers or upstream agent scoring run. +Point-frequency error measurements do not establish error bounds for these +PromQL queries. + +## Backend-only execution prototype + +The target is real data-plane execution, not just binding coverage: +OpenMetrics data and o11y queries enter ASAPQuery-backend; the backend calls +ASAPPlanner and executes either a valid ASAP plan or a local exact baseline. +No ASAPCollector or Prometheus process is required by the intended standalone +prototype. The implementation and its current limitations belong to +[backend PR #524](https://github.com/ProjectASAP/ASAPQuery-backend/pull/524). + +The supplied dataset is `/mydata/metrics.txt`, an OpenMetrics file with 305,026 +samples across 106 series. Preserve it read-only. The backend adapter owns +timestamp conversion and Remote Write encoding; OpenMetrics text cannot be +posted directly to `/api/v1/write`. The currently pinned upstream task corpus +has 28 query occurrences (24 unique queries); do not equate that corpus with the +older 27-query fixture used for the historical binding report. + +Acceptance requires matching exact/ASAP query results before reporting latency +or resource reductions, with identical data, labels and evaluation times. +Report construction/ingestion costs separately from steady-state queries. +Unsupported queries and exact fallbacks are not accelerated successes. +Do not substitute the earlier synthetic result-cache experiment for this test. + +## Optional backend diagnostic + +The small `offline_recommend` diagnostic also lives in ASAPQuery-backend. +It compares measured fixed-snapshot frequency scenarios, not PromQL execution: + +```sh +cargo run -p control_plane --example offline_recommend -- \ + /path/to/comparison-evidence.json /path/to/request.json +``` + +All benchmark producers and execution tools live in the backend repository. +This Planner PR contains documentation only; the public evidence and resource +contracts are reviewed separately in #357.