Skip to content

docs: define backend-driven o11y evaluation scope - #358

Closed
zzylol wants to merge 4 commits into
codex/empirical-o11y-322from
codex/offline-benchmark-tools-322
Closed

zzylol wants to merge 4 commits into
codex/empirical-o11y-322from
codex/offline-benchmark-tools-322

Conversation

@zzylol

@zzylol zzylol commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Why

Refs #322 and depends on #357. Document the corrected repository boundary and backend-driven o11y evaluation scope.

Before this PR

The proposed evaluation mixed planner-only replay, benchmark production and synthetic fixed-snapshot timing. Those tools did not establish the requested backend execution benefits.

After this PR

This PR now contains only three English documentation files:

  • The backend-entry evaluation guide and current prototype acceptance criteria.
  • The earlier offline execution plan, explicitly marked historical.
  • The earlier measurement report, explicitly marked historical and not proof of backend execution benefits.

All sketch-bench producers/probes and the small offline recommendation diagnostic belong in ProjectASAP/ASAPQuery-backend#524. The standalone planner-only o11y_replay is removed. The synthetic o11y_exact_bench is removed from this PR and is not part of the prototype. No Rust binaries, Cargo dependency changes, benchmark scripts, or generated result files remain in this diff.

Current prototype target — backend work, not completed by this PR

Use the supplied /mydata/metrics.txt OpenMetrics data and upstream o11y queries. Data and queries enter ASAPQuery-backend, which calls ASAPPlanner and performs real data-plane execution without ASAPCollector or Prometheus runtime dependencies.

Compare exact and ASAP query results on the same data, labels and evaluation times before reporting latency/CPU/memory/storage benefits. Report build/ingestion costs separately. Unsupported queries and exact fallbacks must not count as accelerated successes. The earlier 27-query binding fixture is not the current upstream corpus (28 query occurrences, 24 unique queries).

The runtime implementation is still in progress in the backend. This documentation PR does not claim end-to-end speedups, lower resource use, full query support, or completion of that prototype.

Review and validation

Stacked on the #357 branch; merge #357 first, then retarget to main. Documentation-only diff; local whitespace checks pass. Historical artifacts remain available at https://github.com/ProjectASAP/ASAPPlanner/releases/tag/offline-evidence-2026-09-08.

@zzylol zzylol changed the title feat(bench): add offline evaluation tools and artifact-backed reports docs: define backend-driven o11y evaluation scope Sep 8, 2026
@zzylol

zzylol commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

Closing at the author’s request: evaluation documentation belongs in ASAPQuery-backend. All three documents have been moved, link-adjusted, committed and pushed in ProjectASAP/ASAPQuery-backend#524 (commit c2a632e). The backend also now owns the sketch-bench tools. No documentation from this PR needs to merge into ASAPPlanner. Core shared resource/evidence work remains in #357. This closure does not claim completion or performance benefits for the backend-only runtime prototype.

@zzylol zzylol closed this Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant