docs: define backend-driven o11y evaluation scope

docs/design_docs/empirical-o11y-execution-plan.md

@@ -0,0 +1,67 @@
+# Offline evidence and o11y planning evaluation
+
+This is the historical offline-evidence milestone plan. The current prototype
+adds actual backend-only data-plane execution using the supplied OpenMetrics
+dataset, as described in the [current evaluation guide](../user-guide/o11y-replay.md).
+Benchmark production and execution tooling now belong to ASAPQuery-backend;
+planner-only coverage and synthetic cached-result timing are not its acceptance criteria.
+
+Audience: developers reproducing issue #322 and the planner/control-plane evaluation.
+
+## Scope
+
+Use offline sketch-bench CPU, elapsed time, state/memory, disk (when measured),
+and errors against offline ground truth. No runtime ground truth, posterior
+feedback, or self-estimated accuracy is required. Offline accuracy observations
+do not establish formal guarantees on unseen distributions.
+
+## Execution and ownership
+
+1. Pin isolated planner and control-plane worktrees from fetched main branches.
+   Preserve existing worktrees, including unresolved cost-model changes.
+2. Evidence agent: implement the versioned artifact, validation, compatibility
+   matching, public cost-model integration, and fallback/decision tests.
+3. Benchmark agent: run actual sketch-bench algorithms on uniform and Zipf
+   inputs, preserve raw output, export artifacts with source and environment
+   provenance, and document reproducible commands.
+4. Replay agent: run the existing o11y PromQL corpus through the backend parser,
+   its ASAPPlanner call, and the backend typed binder (not a planner-only entry),
+   export binding/fallback coverage and planning latency, and separately identify
+   supplemental sketch workloads.
+5. Integration owner: connect compatible evidence to the downstream control
+   plane, validate dependency compatibility, review other agents' changes, run
+   integration checks, and report evidence-supported comparisons.
+
+Steps 2–4 run concurrently after agreeing the artifact contract. Integration
+uses their completed interfaces and measurements. Query-pattern changes are
+limited to demonstrated blockers with regression coverage; unsupported query
+semantics remain explicit in the report.
+
+## Acceptance
+
+- Versioned schema and example; explicit units, algorithm/configuration,
+  dataset/distribution, environment, collection/validity times, sample count,
+  dispersion, and benchmark/model provenance.
+- At least two actual sketch algorithms measured offline on uniform and skewed
+  inputs. CPU is distinct from elapsed time; serialization size is distinct
+  from heap memory and disk I/O. Unmeasured fields stay unavailable.
+- Matching evidence affects a public planning cost/ranking/lifecycle boundary.
+  Tests cover changed decisions, missing/stale/mismatched evidence, invalid
+  values, and unchanged accuracy guarantees.
+- The planner and control-plane evaluation preserve exact fallbacks and expose
+  unsupported shapes, measured provenance, and limits on benefit estimates.
+- Reproducible measurement and replay commands, raw machine-readable results,
+  and a concise report distinguish actual measurements from modeled totals.
+
+## Comparison rules
+
+Compare equivalent tasks, data, parameters, windows, horizons, and evaluation
+cadences. Whole-plan estimates include build/update/readout, retained windows,
+sharing, and raw residual work where evidence exists. Missing raw baseline or
+physical evidence means an unavailable whole-plan speedup, not zero cost.
+Microbenchmark algorithm comparisons are labeled separately. Disk usage and
+network savings require their own measurements or explicit models.
+
+Repeated-query break-even can be computed only when compatible exact baseline,
+sketch build/maintenance, and readout measurements are present. It is an offline
+estimate and does not establish deployed end-to-end latency improvement.

docs/offline-o11y-final-2026-09-08.md

@@ -0,0 +1,102 @@
+# Historical Offline Sketch Evidence and o11y Report
+
+This report records the earlier offline experiment, not the backend-only
+prototype's execution results. Its synthetic exact-result cache experiment is
+out of the current prototype scope; its source tool was removed from this PR.
+Current work uses user-provided OpenMetrics data and queries entering the backend,
+followed by ASAPPlanner and actual data-plane execution. No execution speedup or
+resource reduction is established by the historical results below.
+
+Audience: developers. This report covers the offline evidence path for #322,
+without real-time error feedback. The original dirty working directory was left unchanged.
+
+## Completed Integration
+
+- Versioned artifacts, JSON Schema, and a public CostModel provider match parameters, query semantics, distributions, environments, and validity intervals. Unknown measurements remain unavailable.
+- Actual CMS/CountSketch parameter sweeps measure offline error, separate construction/update/read/merge CPU, live heap allocations, serialized size, and allocated file blocks.
+- Fixed-snapshot point-frequency comparisons use explicit observed-error thresholds, formal parameter minima, CPU/retained byte-seconds weights, and an exact baseline on the same data.
+- The backend filters unsupported layouts before recommending and binding frequency configurations. When exact execution wins or evidence is insufficient, it preserves the original query without silently rounding selected parameters.
+- o11y planner/control-plane replay and exact fixed-snapshot reference measurements cover seven explicitly supported queries.
+
+For interfaces and reproduction instructions, see the [evidence contract](developer_docs/offline-sketch-evidence.md),
+[backend benchmark commands](https://github.com/ProjectASAP/ASAPQuery-backend/blob/codex/empirical-o11y-322/tools/empirical-bench/README.md), and [backend-entry replay guide](user-guide/o11y-replay.md).
+
+## Measured Results
+
+sketch-bench is pinned to revision `87f619e843fd2e4da784160d4e205a0d0d55f032`.
+The uniform and Zipf datasets each contain 20,000 i64 keys, with a key space of
+1,000, seed 42, and Zipf exponent 1.1. There are 18 sketch configurations and
+two exact baselines; CPU measurements use five trials after two warmups.
+Accuracy comes from one offline experiment per fixed dataset, not a guarantee
+across distributions or real-time ground truth. The comparison assumes one
+build, 1,000 queries, 300 seconds of retention, and no merges.
+
+| Scenario | Uniform / Zipf results |
+| --- | --- |
+| CPU-only, with a 1% or 5% observed mean relative-error limit | Both select exact execution; no sketch CPU benefit |
+| Generic comparator, with memory weighting | CMS 2720×5; retained heap decreases by 81.68%, while CPU increases by approximately 0.037 / 0.182 ms, respectively |
+| Backend-compatible configuration, with memory weighting | CMS 4096×5; retained heap decreases by approximately 72.4%, while CPU also increases |
+
+The memory-weighted objective is `CPU ns + 0.01 × retained byte-seconds`. These
+weights express an illustrative preference, not a conversion measured from
+hardware. Exact retained heap is 296,976 B; CMS 2720×5 uses 54,400 B, and
+4096×5 uses 81,920 B. Memory figures measure requested allocation bytes using
+a separate System allocator probe; CPU measurements use jemalloc. These memory
+figures are not process RSS. Serialized size and allocated file blocks do not
+establish disk-throughput benefits.
+
+See `results-sweep/MEASUREMENTS.md` in the [verified experiment archive](https://github.com/ProjectASAP/ASAPQuery-backend/blob/codex/empirical-o11y-322/tools/empirical-bench/ARTIFACTS.md).
+Earlier frequency measurements and control-plane replay in `results/` remain
+as the initial baseline. Final frequency comparisons and control-plane results
+use `results-sweep/`; costs from the two runs must not be mixed.
+
+## o11y Coverage and Limitations
+
+The replay reuses the repository's existing 27 PromQL fixtures. It does not run
+upstream LLM-agent scoring or use real scenario data.
+
+The supported evaluation entry point is now the backend's `offline_planner_replay`:
+queries enter its parser, call ASAPPlanner, and return through its typed binder.
+The standalone planner-only replay was removed. Its candidate/lifecycle counts
+below remain historical diagnostics, not backend support coverage.
+
+| Check | Result |
+| --- | --- |
+| Planner candidate coverage | 26/27 have exact summary candidates; 1/27 uses raw fallback; no sketch candidates |
+| Complete lifecycle costs in the original replay | Insufficient evidence; all 27/27 conservatively fall back to raw execution |
+| Control-plane exact/default/empirical binding | All three modes bind 27/27 successfully; each retains the original query in 5 root plans |
+| Seven exact reference queries | Raw execution and cached-result reads are measured on fixed synthetic gauge snapshots, using the public CostModel and actual selection flow |
+| Remaining 20 queries | No complete reference implementation; benefits remain unavailable |
+
+New fallbacks preserve the original IR for selector, sort, and comparison/filter
+roots, increasing successful binding from 24/27 to 27/27. Binding does not mean
+every plan is executable by the summary executor. Column updates in the new IR
+are supported; keyed/non-column updates and warm-tier BinaryOp remain explicitly rejected.
+
+The seven reference queries cover instant sums, window maxima, and sums of
+window averages: 300 series, three jobs, one finite gauge sample per minute,
+and 60 repeated reads of the same snapshot. The retained state is the complete
+exact query result, not a sliding window with advancing time. Measurements
+exclude network, disk, protocol serialization, and output-label materialization;
+memory figures count logical value bytes. Benefits from repeated reads cannot
+be extrapolated to production end-to-end speedups. Machine-readable results are
+available as `results/o11y-exact-snapshot.json` in that archive.
+
+Quantile measurements, real o11y traces, time drift, post-merge error, online
+update and retirement costs, and complete production benefits remain outside
+this coverage. Point-frequency error cannot be applied to quantiles or count_over_time.
+
+## Validation and Delivery
+
+- Planner mapping: 349 unit tests pass.
+- Devtools: 3 exact benchmark tests and 4 replay tests pass; the recommendation CLI is verified with six actual requests.
+- Benchmark export: 5 Python tests pass.
+- Backend: 633 control-plane and 973 data-plane library tests pass; 7 new integration tests pass.
+- Agents cross-reviewed the provider, measurement, and binding code; artifact text, JSON, source hashes, and provenance were checked successfully.
+
+The backend pins the published planner core commit
+`dfdf6b5c1f7d667394a4ea1f56fb3786af04228d`. Normal builds and final replay do not
+require local Cargo source patches. Core planner code, benchmark tools, and backend
+integration are reviewed in separate PRs. Full generated results are distributed
+as an experiment artifact, not embedded in the source diff. No PR is automatically
+merged and the issue is not automatically closed.

docs/user-guide/o11y-replay.md

@@ -0,0 +1,85 @@
+# Offline o11y evaluation through the backend
+
+For developers evaluating the actual control-plane planning integration.
+The canonical path is:
+
+`PromQL corpus -> ASAPQuery-backend parser -> ASAPPlanner -> backend typed binder/fallback`
+
+There is no standalone planner-only `o11y_replay` command. Run the backend's
+`offline_planner_replay` example through its reproducibility wrapper instead.
+
+## Run the control-plane replay
+
+From an ASAPQuery-backend checkout, with its normal sibling dependencies available:
+
+```sh
+python3 tools/run-offline-planner-replay.py \
+  --queries /path/to/ASAPPlanner/crates/frontend-promql/tests/observability/data/o11y_bench_promql.txt \
+  --evidence /path/to/planner-evidence.json \
+  --context /path/to/context.json \
+  --output /tmp/control-plane-o11y.json
+```
+
+The backend uses its pinned planner dependency; a planner source checkout is
+only needed here to locate the vendored query fixture. Omit evidence and context
+together to run exact/default modes only. Supplying both adds empirical mode.
+An optional `--planner /path/to/ASAPPlanner` enables a local source override for
+development; it is not the default evaluation path.
+
+The [backend guide](https://github.com/ProjectASAP/ASAPQuery-backend/blob/codex/empirical-o11y-322/control_plane/docs/offline-sketch-evidence.md)
+documents the actual parser/binder and evidence assumptions.
+The [benchmark driver](https://github.com/ProjectASAP/ASAPQuery-backend/blob/codex/empirical-o11y-322/tools/empirical-bench/README.md)
+and [artifact instructions](https://github.com/ProjectASAP/ASAPQuery-backend/blob/codex/empirical-o11y-322/tools/empirical-bench/ARTIFACTS.md)
+also live in the backend repository.
+
+## Interpret coverage conservatively
+
+- A parse failure is not a planner or backend execution result.
+- A bound plan means the backend accepted a plan returned through its planner integration.
+- `root_raw_fallback` means the complete original query remains selected; this is not summary acceleration.
+- `raw_subtrees` includes source scans underneath summaries and cannot be counted as root fallback.
+- `planning_elapsed_ns` measures the parser/binder path, not query execution.
+- Binding does not establish deployable placement, successful data-plane execution, or resource savings.
+
+The recorded final evaluation bound all 27 fixtures in exact/default/empirical
+modes; each mode retained five raw root fallbacks. Earlier standalone planner
+candidate counts are historical diagnostics, not backend support coverage.
+No collectors, storage services, query servers or upstream agent scoring run.
+Point-frequency error measurements do not establish error bounds for these
+PromQL queries.
+
+## Backend-only execution prototype
+
+The target is real data-plane execution, not just binding coverage:
+OpenMetrics data and o11y queries enter ASAPQuery-backend; the backend calls
+ASAPPlanner and executes either a valid ASAP plan or a local exact baseline.
+No ASAPCollector or Prometheus process is required by the intended standalone
+prototype. The implementation and its current limitations belong to
+[backend PR #524](https://github.com/ProjectASAP/ASAPQuery-backend/pull/524).
+
+The supplied dataset is `/mydata/metrics.txt`, an OpenMetrics file with 305,026
+samples across 106 series. Preserve it read-only. The backend adapter owns
+timestamp conversion and Remote Write encoding; OpenMetrics text cannot be
+posted directly to `/api/v1/write`. The currently pinned upstream task corpus
+has 28 query occurrences (24 unique queries); do not equate that corpus with the
+older 27-query fixture used for the historical binding report.
+
+Acceptance requires matching exact/ASAP query results before reporting latency
+or resource reductions, with identical data, labels and evaluation times.
+Report construction/ingestion costs separately from steady-state queries.
+Unsupported queries and exact fallbacks are not accelerated successes.
+Do not substitute the earlier synthetic result-cache experiment for this test.
+
+## Optional backend diagnostic
+
+The small `offline_recommend` diagnostic also lives in ASAPQuery-backend.
+It compares measured fixed-snapshot frequency scenarios, not PromQL execution:
+
+```sh
+cargo run -p control_plane --example offline_recommend -- \
+  /path/to/comparison-evidence.json /path/to/request.json
+```
+
+All benchmark producers and execution tools live in the backend repository.
+This Planner PR contains documentation only; the public evidence and resource
+contracts are reviewed separately in #357.