feat(o11y): add matched exact comparison and scoped resource reports - #526
Merged
Merged
Conversation
zzylol
marked this pull request as ready for review
September 8, 2026 20:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Compare every o11ybench query occurrence with Prometheus using identical query strings, evaluation timestamps and input batches. Report label/sample completeness, numerical differences, failures, latency and process resource observations without claiming benchmark acceptance or speedup.
Before this PR, the execution harness recorded responses without a matched exact comparison. After this PR, missing groups or different values remain visible and suppress the aggregate latency ratio; response ordering alone does not count as an error. Optional separate fallback and baseline Prometheus processes avoid intentional sharing of their caches, and fallback CPU is charged to the backend. Per-process CPU affinity and virtual-address-space limits are explicit optional controls.
Validation: 13 Python comparison/replay tests passed, including odd-sized corpus order alternation and fallback CPU accounting; git diff --check passed. These tests use fixtures and do not establish real-workload performance.
Limitations: exact floating-point equality is deliberately conservative; approximate differences are reported without an invented tolerance. CPU has process/tick scope, HWM is process lifetime, RLIMIT_AS is not a physical-memory or aggregate process budget, external service startup and guaranteed cold caches remain unmeasured. The runner now uses #525’s explicit finite-input drain before measuring queries; input remains sealed for the process lifetime. This is not a live-ingestion watermark or proof of every readout’s coverage. Provider costs remain in their declared units, estimated/measured cost ratios remain null, and acceptance_complete remains false. This patch adds no logical/hybrid execution engine and depends only on the parent #525 interfaces.