docs(canary): record the first fleet read — check 3 is dirty - #3030
Open
vanceingalls wants to merge 4 commits into
Open
docs(canary): record the first fleet read — check 3 is dirty#3030vanceingalls wants to merge 4 commits into
vanceingalls wants to merge 4 commits into
Conversation
Calibration check 3 says cohort flips MUST be zero. Pre-release data showed one install of 15 reporting inverse assignments for both calibration canaries in two cli_command events at the same millisecond, then stable forever after. Cause: two CLI processes started concurrently before install-state.json exists each mint their own bucketSeed, the last writer wins the file, and the loser keeps its in-memory seed for its process lifetime. Bounded to a machine's first runs and self-healing, so accepted rather than fixed — serializing it needs cross-process locking in the telemetry config path. Documents the distinctive signature (disagreement confined to an install's earliest events) so the fleet read is not misattributed to a bucketer defect, and corrects the passing-criteria line that claimed check 3 passes outright. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
Supersedes the earlier framing of the first-mint seed race as a sufficient explanation for cohort flips. The first fleet read (0.7.92-0.7.94, n=9,702) shows it explains only ~23% of them. Check 1 passes, but only once CI installs are excluded from the denominator: they are excluded from enrolment yet still emit "false", so the naive read came back 9.22%/48.64% and looked like a failure. Both land on target (9.49%/49.76%) with CI removed. That denominator rule is now in the check. Check 3 is dirty. 107 calibration-50 flips; only 25 match the first-mint signature, while 17 sustain the minority for a mean 7.7 hours and ~125 events. A registry edit mid-window is ruled out — enrolment is consistent across all three shipped versions. Documents the split so the accepted exception is not read as covering the whole population, and calls for the decision-reason property this section already promised to add if the check came back dirty. Also records the independence result (5.18% vs 4.71% expected, z=2.16). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
vanceingalls
changed the title
The first-read section labelled n = 9,702 as non-CI. It is not: 9,702 is all installs, 9,455 are non-CI, and the flip counts were computed without the CI filter while the accuracy figures were computed with it. Conflating those two populations is the exact error this section warns about one paragraph earlier. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MDX reads a bare `<` as the start of a JSX tag, so `| <= 5s ...` failed Validate docs with 'Unexpected character = (U+003D) before name'. Uses the typographic operator and an entity instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first fleet read is in (
0.7.92–0.7.94, 9,702 installs of which 9,455 are non-CI), and it changes what this doc needed to say. The original version of this PR framed the first-mint seed race as a sufficient explanation for cohort flips. It is not — it explains about 23% of them.Check 1 — accuracy: passes, but the definition had a trap
Naive column is all 9,702 installs; CI-excluded column is the 9,455 non-CI.
CI installs are excluded from percentage enrolment but still emit the assignment as
"false", so they sit in the denominator and bias every canary low. The naive read is significantly under target on both — anyone following the check literally would have reported a bucketer failure that does not exist. The denominator rule is now part of the check.Check 3 — stability: dirty
107 installs flipped on
calibration-50(1.1%), 40 oncalibration-10— over all 9,702 installs, CI included. Split by how long the minority assignment persisted:Seventeen installs sustained the minority for a mean 7.7 hours and ~125 events (two Docker, ~430 each). By this check's own definition that is a real defect, not an accepted limit.
Ruled out: a registry edit mid-window. Enrolment is consistent across all three shipped versions (9.53/9.83/9.36% and 49.51/53.88/49.88%), so the percentages did stay fixed. Leading unproven hypothesis: a shared
distinct_idacross containers that each hold their own seed.Why the revision matters more than the original note
Left as written, the note would have licensed dismissing all 107 flips as known-innocent when 82 are not. That is worse than having no note — it converts an open defect into a documented non-issue.
This section already contained its own escape hatch: "Emitting a reason property is deliberately skipped — add it only if this check comes back dirty." It has. Without the decision reason, a deliberate
HF_CANARY_*override is indistinguishable from a genuine flip, which is precisely why the remaining 82 cannot be attributed.Consequence
#2840 stays held. Nothing should ramp on a bucketer whose cohorts are not provably stable. Next step is emitting
forced_on/forced_off/in_cohort/out_of_cohortand re-reading the table.Also records the independence result: 5.18% overlap vs 4.71% expected (n = 9,523 non-CI, z = 2.16) — slight positive correlation, small in absolute terms, worth re-checking rather than acting on.
🤖 Generated with Claude Code