Skip to content

feat(experimentation): results aggregation query and payload builder#7781

Draft
gagantrivedi wants to merge 3 commits into
feat/experiment-stats-kernelfrom
feat/experiment-results-query
Draft

feat(experimentation): results aggregation query and payload builder#7781
gagantrivedi wants to merge 3 commits into
feat/experiment-stats-kernelfrom
feat/experiment-results-query

Conversation

@gagantrivedi

Copy link
Copy Markdown
Member
  • I have read the Contributing Guide.
  • I have added information to docs/ if required so people know about the feature. (deferred — internal; docs land with the results UI.)
  • I have filled in the "Changes" section below.
  • I have filled in the "How did you test this code" section below.

Changes

Contributes to the experimentation results scorecard (v0.2). The ClickHouse aggregation + pure payload builder that feeds the stats kernel (#7769). No model, endpoints or task — those, plus the ORM-coupled orchestrator (metric specs + per-environment expected shares), land in the next PR.

  • Results query — one single-pass query: the v0.1 exposures CTE (first-exposure dedup, quarantine, half-open window) joined to post-exposure metric events, with per-metric conditional aggregation → per metric × variant sufficient statistics (n, sum, sum_squares), plus per-variant identity counts for SRM. Per-aggregation unit value: occurrence 0/1, count, sum, mean (zero when an identity has no matching events). No conversion window in v0.2 (window = exposure → as_of).
  • build_results_summary — pure: runs compare_to_control per treatment and srm_p_value on the counts. Data floor (n ≥ 50/arm, ≥ 5 conversions/arm for occurrence) → inference null; SRM only above 100 total identities; chance-to-win flipped for lower-is-better metrics. Raw stats kept for every variant (durable record for future engines); means/status/names derived client-side.
  • Dataclasses MetricSpec / MetricResult / ResultsSummary (reusing the kernel's VariantStats / Inference); asdict yields the wire shape directly.

How did you test this code?

  • Query (faked client): row→VariantStats mapping, per-aggregation unit-value expressions, post-exposure join + quarantine exclusion + half-open window, params, no-metrics count-only path.
  • Builder: per-treatment inference, identity/conversion floors, zero-control-mean and missing-control nulling, lower-is-better flip, SRM balanced/imbalanced/not-computable, exact wire shape via asdict.
  • pytest tests/unit/experimentation/ — 284 passed; ruff + mypy strict clean.

Note

The query's ClickHouse-specific semantics (LEFT JOIN attribution, avgIf over zero matching rows, toFloat64OrZero) are asserted by SQL substring only — there is no ClickHouse in CI (parked). The mean aggregation guards avgIf with a count check; this is the least-covered path until ClickHouse-in-CI lands.

@vercel

vercel Bot commented Jun 15, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments
Project Deployment Actions Updated (UTC)
docs Ignored Ignored Preview Jun 15, 2026 11:39am
flagsmith-frontend-preview Ignored Ignored Preview Jun 15, 2026 11:39am
flagsmith-frontend-staging Ignored Ignored Preview Jun 15, 2026 11:39am

Request Review

@github-actions github-actions Bot added api Issue related to the REST API feature New feature or request labels Jun 15, 2026
@codecov

codecov Bot commented Jun 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.50%. Comparing base (32c8582) to head (bdfef40).

Additional details and impacted files
@@                       Coverage Diff                        @@
##           feat/experiment-stats-kernel    #7781      +/-   ##
================================================================
- Coverage                         98.57%   98.50%   -0.07%     
================================================================
  Files                              1462     1461       -1     
  Lines                             56550    56409     -141     
================================================================
- Hits                              55743    55567     -176     
- Misses                              807      842      +35     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions github-actions Bot added feature New feature or request and removed feature New feature or request labels Jun 15, 2026
ClickHouse rejects an ON clause mixing left and right columns in an
inequality; verified against ClickHouse 24.8 with seeded data.
@github-actions github-actions Bot added feature New feature or request and removed feature New feature or request labels Jun 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api Issue related to the REST API feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant