eval: report partial Alibaba dashboard measurements - #651
Merged
zzylol merged 4 commits intoSep 12, 2026
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Preserve the real Alibaba dashboard measurements collected before the user requested stopping the experiment.
What
Add the Rust comparison harness, data preparation and evaluation design, plus 15 completed service runs out of 54 planned runs. Include raw plans/measurements, dataset provenance, a partial report and performance plots. Execution is stopped; this is not the completed experiment.
How
Prepare all 240 official three-minute archives (12 hours, 2,646,269,187 retained observations). Calibrate on a 10M-event reservoir from hours 0–6; replay hours 6–12 without sampling after one hour of warmup. Compare six baselines on count and Top-3 dashboards with 1m/10m/60m windows refreshed every minute.
Before this PR
The dashboard comparison branch had no executable Alibaba observation-contract experiment or these real-data measurements.
After this PR
Reviewers can inspect the partial report, plots and raw per-query samples, including planning, update, merge, readout, payload memory and accuracy violations. This PR is stacked on #602.
Trial 0 total CPU seconds:
Verification
Limitations
Service repetitions are incomplete/unbalanced. Edge and latency catalogs exist, but their held-out runs were not started. The displayed table is consistently trial 0, not a three-trial aggregate. ERP uses a custom calibration catalog, not synthetic nearest-shape matching; the shared ERP result is exact fallback, not a successful approximate-sketch result. AutoSketch is a CPU adaptation with extensions. Analytical sizing is not full cost-model optimization. Timers measure operator regions, not deployed backend/network latency; memory is logical payload, not isolated RSS. Original datasets and oracle caches are not vendored into Git.