Live, right now
byProject accuracy : [100, 100, 100] # on 2377 / 602 / 508 reviewed, 0 reversals
totals.accuracyPct : 100
Both #9718 (withhold accuracy where a reversal is unobservable) and #9768 (bound the denominator to the retention window) are deployed. Neither fixed this, because both addressed the window the denominator covers and not which PRs belong in it.
Root cause
A reversal is, by definition, a human overturning an engine auto-action — recordReversalSignals only ever records one against a PR the engine merged or closed. So the population the numerator can be drawn from is auto-actions.
The denominator is github_app.pr_public_surface_published: every PR that got a review surface, including the many the engine only commented on, held, or advised. Those PRs could never have produced a reversal, so every one of them pushes 1 - reversed/decided toward 100% for a reason unrelated to gate quality.
#9768 narrowed that set to a 90-day window, which was a real and necessary fix — an immortal denominator against a pruned numerator drifts on its own. But a windowed count of reviewed PRs is still the wrong population.
The weekly trend already gets this right on the numerator side: loadReversalDayRows (src/services/public-accuracy-trend.ts) anchors reversals on agent.action.* rows with dry-runs excluded. Its own denominator is still published-PR-based, so the same defect is present there, just masked because the own-ledger series currently reports null for recent weeks.
Fix
Make the accuracy denominator, on both surfaces, the distinct PRs this deployment auto-actioned in the window — same event types, same outcome = 'completed' filter, same dry-run exclusion the numerator's anchor already uses. Volume columns (reviewed, merged, closed) keep publishing lifetime/fleet-folded counts, which are measured and correct; only the ratio's denominator narrows.
This also makes #9718's separate observability probe redundant and it should be removed rather than left beside the thing that supersedes it: no auto-actions means decided is 0 and the answer is null by construction. A structural guarantee is better than a heuristic that can disagree with the denominator sitting next to it.
Expected effect in production
The hosted Worker does not execute reviews (#9676), so recent auto-actions are absent or few. byProject should go null or drop to a small honest number instead of asserting 100% across 3,487 PRs.
Live, right now
Both #9718 (withhold accuracy where a reversal is unobservable) and #9768 (bound the denominator to the retention window) are deployed. Neither fixed this, because both addressed the window the denominator covers and not which PRs belong in it.
Root cause
A reversal is, by definition, a human overturning an engine auto-action —
recordReversalSignalsonly ever records one against a PR the engine merged or closed. So the population the numerator can be drawn from is auto-actions.The denominator is
github_app.pr_public_surface_published: every PR that got a review surface, including the many the engine only commented on, held, or advised. Those PRs could never have produced a reversal, so every one of them pushes1 - reversed/decidedtoward 100% for a reason unrelated to gate quality.#9768 narrowed that set to a 90-day window, which was a real and necessary fix — an immortal denominator against a pruned numerator drifts on its own. But a windowed count of reviewed PRs is still the wrong population.
The weekly trend already gets this right on the numerator side:
loadReversalDayRows(src/services/public-accuracy-trend.ts) anchors reversals onagent.action.*rows with dry-runs excluded. Its own denominator is still published-PR-based, so the same defect is present there, just masked because the own-ledger series currently reports null for recent weeks.Fix
Make the accuracy denominator, on both surfaces, the distinct PRs this deployment auto-actioned in the window — same event types, same
outcome = 'completed'filter, same dry-run exclusion the numerator's anchor already uses. Volume columns (reviewed,merged,closed) keep publishing lifetime/fleet-folded counts, which are measured and correct; only the ratio's denominator narrows.This also makes #9718's separate observability probe redundant and it should be removed rather than left beside the thing that supersedes it: no auto-actions means
decidedis 0 and the answer is null by construction. A structural guarantee is better than a heuristic that can disagree with the denominator sitting next to it.Expected effect in production
The hosted Worker does not execute reviews (#9676), so recent auto-actions are absent or few.
byProjectshould go null or drop to a small honest number instead of asserting 100% across 3,487 PRs.