feat(experimentation): results aggregation query and payload builder#7781
Draft
gagantrivedi wants to merge 3 commits into
Draft
feat(experimentation): results aggregation query and payload builder#7781gagantrivedi wants to merge 3 commits into
gagantrivedi wants to merge 3 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. 3 Skipped Deployments
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## feat/experiment-stats-kernel #7781 +/- ##
================================================================
- Coverage 98.57% 98.50% -0.07%
================================================================
Files 1462 1461 -1
Lines 56550 56409 -141
================================================================
- Hits 55743 55567 -176
- Misses 807 842 +35 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
ClickHouse rejects an ON clause mixing left and right columns in an inequality; verified against ClickHouse 24.8 with seeded data.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
docs/if required so people know about the feature. (deferred — internal; docs land with the results UI.)Changes
Contributes to the experimentation results scorecard (v0.2). The ClickHouse aggregation + pure payload builder that feeds the stats kernel (#7769). No model, endpoints or task — those, plus the ORM-coupled orchestrator (metric specs + per-environment expected shares), land in the next PR.
(n, sum, sum_squares), plus per-variant identity counts for SRM. Per-aggregation unit value:occurrence0/1,count,sum,mean(zero when an identity has no matching events). No conversion window in v0.2 (window = exposure → as_of).build_results_summary— pure: runscompare_to_controlper treatment andsrm_p_valueon the counts. Data floor (n ≥ 50/arm, ≥ 5 conversions/arm for occurrence) → inferencenull; SRM only above 100 total identities; chance-to-win flipped for lower-is-better metrics. Raw stats kept for every variant (durable record for future engines); means/status/names derived client-side.MetricSpec/MetricResult/ResultsSummary(reusing the kernel'sVariantStats/Inference);asdictyields the wire shape directly.How did you test this code?
VariantStatsmapping, per-aggregation unit-value expressions, post-exposure join + quarantine exclusion + half-open window, params, no-metrics count-only path.asdict.pytest tests/unit/experimentation/— 284 passed;ruff+mypystrict clean.Note
The query's ClickHouse-specific semantics (LEFT JOIN attribution,
avgIfover zero matching rows,toFloat64OrZero) are asserted by SQL substring only — there is no ClickHouse in CI (parked). Themeanaggregation guardsavgIfwith a count check; this is the least-covered path until ClickHouse-in-CI lands.