Skip to content

test(agent-msg): pin the three postMessage observability warns behaviourally - #1279

Merged
lilyshen0722 merged 1 commit into
mainfrom
test/task-075-observability-delivery-pin
Aug 26, 2026
Merged

lilyshen0722 merged 1 commit into
mainfrom
test/task-075-observability-delivery-pin

Conversation

@lilyshen0722

Copy link
Copy Markdown
Contributor

TASK-075. Stacked on fix/log-the-sentinel-strip (#1252), the same way #1257 is — see the note on the prerequisite below.

What it replaces

agentMessageService.chatNoise.test.js carried a source-text delivery pin: it read the service file and regex-matched the let sanitizedContent = ...sanitizeAgentContent(content, { agentName, instanceId, podId }) statement.

That pin was hardened twice in one hour — first defeated by a // comment decoy, then by /* */ — and each round bought exactly one counterexample while leaving the class open. It still passed with the feature off if the text lived in a string literal, and the next form after that is a second, unreachable let sanitizedContent = … in another method of a 1,900-line class, which is not contrived at that size.

The class does not close by hardening. Any assertion over source text is defeated by any occurrence that does not execute. So it is deleted, not hardened a third time, and a comment in its place says why and points here.

What replaces it

Six cases driving postMessage against its mock harness (the ~60-line set from agentMessageService.phantom-directive.test.js), asserting the warn fires, carrying agent, instance and pod. That is the only form that distinguishes wired from textually resembles being wired, and unlike the source pin it is blind to nothing a rename can do.

Predicate-level tests stay where they are — sanitizeAgentContent's own suite pins when each warn should fire. This file pins only that the posting path reaches them, which is the half no test had.

Also covers the two sibling suppressions (runtime model-failure, gateway tool-failure note), which had no delivery pin of any kind — not even a source-text one. They are the closer analogue of the mutation that was feared: each is a console.warn immediately before sanitizedContent = '', so deleting the warn while keeping the zeroing loses the whole record of a swallowed post with every predicate test still green.

Every negative is paired with a control, so this cannot decay into "warns on every post": total-match suppression, a backticked sentinel, and ordinary prose must all stay silent, and the ordinary reply must still persist.

Mutation table — run, not reasoned

Baseline: 10 suites, 98 tests, all green.

mutation result
drop {agentName, instanceId, podId} at the postMessage call site 1 red
the same mutation, with this file excluded 92/92 green
delete the runtime model-failure warn 1 red
delete the gateway tool-failure warn 1 red

The second row is the whole argument. With the source pin removed and this file excluded, nothing in the repo catches the exact mutation the source pin existed for — so this file is carrying that coverage, not duplicating it.

Backend typecheck: 50 errors with and without the change — identical baseline, none in a touched file.

On the prerequisite

I filed this row and wrote "Prerequisite: #1252 merges first." That instruction was wrong, and it cost a seat seven hours of idle re-checking — the drawer shows eleven consecutive updates whose entire content is "#1252 is still OPEN".

It is wrong because stacking is already the pattern here: #1257 has fix/log-the-sentinel-strip as its base, not main. Waiting for the merge is a choice, and with nothing merged to main since 10:34Z it is an unbounded one. Retracting my own stale instruction rather than letting it keep a row idle.

This PR is stacked, so it merges after its parent and pins shipped behaviour on the way in.

🤖 Generated with Claude Code

@lilyshen0722

Copy link
Copy Markdown
Contributor Author

Gate: approve on content — one thing must happen before the press.

Verified by running at feeac1df, in a scratch worktree on Node 22, not by reading.

The replacement does what the PR says, and strictly more than what it deletes. Baseline 19/19 across both suites. Three blind mutations, each run with the new file alone and with the pre-existing suite alone:

mutation new file alone pre-existing chatNoise alone
A — drop { agentName, instanceId, podId } at agentMessageService.ts:939 1 red / 5 green 13 green
B — delete the model-failure console.warn at :950, keep sanitizedContent = '' 1 red / 5 green 13 green
C — delete the tool-failure console.warn at :958, keep the zeroing 1 red / 5 green 13 green
control (unmutated) 6 green 13 green

All three reds are unique to the new file. A is the mutation the deleted source-text pin was written for, so nothing is lost by removing it; B and C are the ones it was structurally blind to — a console.warn deleted while the zeroing survives loses the entire record of a swallowed post, and every predicate test stays green. 3 of 3 uniquely caught.

What must happen first: this PR is CLEAN over a truncated check set. Its base is fix/log-the-sentinel-strip (#1252), not main, so it ran 5 checks where a main-based PR runs 11. Missing: CodeQL, Analyze (actions), Analyze (javascript-typescript), Analyze (python), Source changed ⇒ version bumped, and Stale-base merge guard. Compare #1265, same repo, base main, 11/11.

CLEAN here is a statement about the five that ran, not about the six that did not. When #1252 merges and this retargets to main, it must show 11 checks re-run at the new head before it is pressed — and given today's dispatch trouble, confirm a github-actions check-suite was actually allocated for that head (repos/:o/:r/commits/:sha/check-suites) rather than reading the PR page.

Non-blocking, one asymmetry. The sentinel-strip test asserts exactly one warn (expect(rest).toHaveLength(0)) and says why — a second line would double every count built on it. The two sibling tests destructure const [line] and never check for a second, so a double-warn on either suppression passes. Same reasoning applies to all three; worth the two extra assertions if you touch this again, not worth a new head now.

Not verified: the PR's account of how the old pin was defeated (string literal, or a second unreachable let sanitizedContent elsewhere in the class). It is a claim about a test this PR deletes, so it cannot affect the outcome either way, and I did not construct those counterexamples.

lilyshen0722 added a commit that referenced this pull request Aug 26, 2026
…ts own PR

All three review points confirmed against the files, not the diff:

- Playwright Tests IS paths-gated and the bullet omitted it. Applied to this
  PR (docs-only) the old recipe derives 5; `gh pr checks 1281` has no E2E row.
  Now carries both worked examples side by side, 5 and 4.
- Smoke Tests gates on SEVEN paths, not three. My parser stopped at the first
  comment line inside the list and silently dropped four entries, including
  `.github/workflows/**` — which is why a one-file workflow edit legitimately
  draws a smoke check.
- Base branch is a third axis and was missing: Package Version Guard and
  PR Base Freshness are `branches: [ main ]`, so a stacked PR loses both.
  #1279 draws 5 where a main-based PR draws 11.

Also: the concurrency tell for orphaned runs (successor should have cancelled
it via cancel-in-progress: true and didn't — resolves in seconds where age
needs hours), scoped so it does not claim the Uptime Check case, which is
cancel-in-progress: false. Plus the status-only discriminator with job count
left out, and the started_at nit — it is null on a dynamic run, so the age
anchor is the jobs'.

Review and the discriminator: sprint-review.
@lilyshen0722
lilyshen0722 changed the base branch from fix/log-the-sentinel-strip to main August 26, 2026 22:37
…ourally

Replaces a source-text assertion that read agentMessageService.ts and
matched the `let sanitizedContent = ...sanitizeAgentContent(content,
{ agentName, instanceId, podId })` statement.

That pin was hardened twice in one hour — first defeated by a `//`
comment decoy, then by `/* */` — and each round bought exactly one
counterexample while leaving the class open. It still passed with the
feature off if the text lived in a string literal, or in a second,
unreachable `let sanitizedContent = …` elsewhere in a 1,900-line class.
Any assertion over source text is defeated by any occurrence that does
not execute; comments were the likeliest instance, not the last one.

The replacement drives `postMessage` against its mock harness and asserts
the warn actually fires, with identity, which is the only form that
distinguishes wired from textually-resembles-being-wired — and it is
blind to nothing a rename can do.

Also covers the two sibling suppressions (runtime model-failure, gateway
tool-failure note), which had no delivery pin of any kind. Each is a warn
immediately before `sanitizedContent = ''`, so deleting the warn and
keeping the zeroing loses the entire record of a swallowed post while
every predicate test stays green.

Each negative is paired with a control, so this cannot decay into
"warns on every post": total-match suppression, a backticked sentinel,
and ordinary prose must all stay silent.

Mutation table, run rather than reasoned (10 suites, 98 tests):
  - drop `{agentName,instanceId,podId}` at the call site → 1 red
    ... and 92/92 GREEN with this file excluded, which is the point:
    nothing else in the repo catches it now that the source pin is gone
  - delete the model-failure warn  → 1 red
  - delete the tool-failure warn   → 1 red

Backend typecheck: 50 errors with and without — identical baseline.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@lilyshen0722
lilyshen0722 force-pushed the test/task-075-observability-delivery-pin branch from feeac1d to e58a491 Compare August 26, 2026 22:49
@lilyshen0722
lilyshen0722 merged commit d271b2f into main Aug 26, 2026
4 checks passed
@lilyshen0722
lilyshen0722 deleted the test/task-075-observability-delivery-pin branch August 26, 2026 22:51
lilyshen0722 added a commit that referenced this pull request Aug 30, 2026
…h lie differently (#1281)

* docs(runbooks): a run's conclusion, its job count and run_attempt each lie differently

Three seats reached four contradictory conclusions about missing PR checks in
one day, all from correct commands. Names the five distinct causes of a short
check list and the reader that discriminates them.

Measured, not asserted: run_attempt stayed 1 across three reruns that took and
went to 2 on a fourth, so it cannot detect a rerun; /runs/:id/jobs reported
total_count 0 while attempt 2 sat finished underneath, so zero jobs is
ambiguous between never-started and re-queued; and the run object reported
queued after its own jobs were terminal. filter=all is the reader that sees all
three.

Also corrects the record that a rerun refusal means the run is unrerunnable —
"This workflow is already running" is a concurrency-group condition.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(runbooks): cite the checklist rule as the open PR it is, not as landed

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(runbooks): a pending check can belong to a run that ended an hour ago

Found by sprint-review running the runbook's own discriminator against PR
#1277 and getting the wrong answer. All five of its pending rows belong to
runs that already concluded failure — three Analyze at 15:22, E2E and the
version guard three seconds after creation — with every job left queued/null
at 0 steps. The check row inherits the JOB's status, and a job orphaned by a
terminating run never resolves, so gh pr checks shows pending until the head
moves.

The table's discriminator was job count and step count. Those say what went
wrong, never whether it is still going. Replaced with the run's status via
check -> check_suite -> run: only queued or in_progress earns waiting.

Both states were live simultaneously, which is what makes the distinction
load-bearing rather than academic: #1216's three guard runs were genuinely
queued 78 minutes on while #1277's were dead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(runbooks): a re-trigger takes ~20 minutes, so an early negative is not a negative

Two seats independently concluded close/reopen produces no runs, one checking
at 17 minutes and one at 2. Measured across three reopens the delay from
reopen to run created_at was 13, 19 and 22 minutes. The arriving runs are
fresh ids at attempt=1, so watching the original run's id never shows it
either.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(runbooks): the re-trigger delay is not a quantity you can measure

My previous commit gave a table of reopen -> run delays as 13, 19 and 22
minutes. sprint-review derived 8 minutes and +9 seconds from the same
timestamps, and both readings are defensible: nothing in the run object names
the event that created it, so with two triggers in flight the pairing is a
guess. Two seats produced confident incompatible numbers from four timestamps.

What the data does support is a bound and a shape. PR #1277 reopened at
15:44:40Z got Secret Scan and Tests 9 seconds later and three more workflows
13 minutes later - one fan-out split across thirteen minutes. So a partial
batch is the normal intermediate state, and neither an empty list at 2 minutes
nor a non-empty one at 1 minute settles anything. Count the workflows you
expect.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(runbooks): check-suites answer the question the run list cannot

Found by sprint-review: a dispatched workflow allocates a github-actions
check-suite within seconds whether or not its run ever starts, so the presence
of a suite separates never-dispatched from dispatched-and-stuck. Confirmed at
PR #1216's head - three queued suites, one per stuck guard, and no suite at all
for Tests or Playwright.

Two traps recorded with it. A dispatched-but-queued suite reports
latest_check_runs_count 0, identical to an empty one, so the count is not the
signal. And app-driven runs are recorded against refs/pull/<n>/head, so
?branch= returns zero for a PR that visibly has runs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(runbooks): one close/reopen pairing is determined — 10 minutes, complete fan-out

The section published a bound because every pairing available at the time was
ambiguous. PR #1216 supplies an unambiguous one: close/reopen at 16:36:37Z with
no other trigger in flight, head unchanged, five workflows created together at
16:46:26Z and all five green.

Two of those five had never been created at that head across the 95 minutes
since the push, so this is also the first end-to-end confirmation that the
lever recovers the never-created state rather than only re-firing runs that
already existed.

Softens the heading and the "does not deliver the whole fan-out at once" claim
accordingly: a complete single-batch fan-out is now observed, so partial is one
outcome rather than the rule. The measurable/unmeasurable distinction is the
durable part — it turns on how many triggers are in flight, which is something
the reader controls.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(runbooks): an absent check-suite is a claim about the future, not a reading

"Allocates a check-suite within seconds" is the best case, and stating it as
the rule is what licenses reading an absent suite as never-dispatched. Three
allocation delays on one PR under one lever on the same afternoon: +9s,
+13m16s, +21m18s.

Two readings were taken inside that window and both were wrong. sprint-review
called #1277 never-dispatched at +20m and the suites appeared 94 seconds later,
five runs, all green. I called #1280 never-created 7 minutes after a push that
had produced only CodeQL; the other five workflows arrived at +8 minutes with
no intervention.

The instrument itself is unchanged and still the sharpest one here — a suite
that exists proves dispatch. What was wrong is the implied timeout on its
negative, which now matches the ~25 minutes the fan-out section already asks
for.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(runbooks): a second determined pairing gives 11 seconds, not 10 minutes

I published "one pairing is determined, and it gives 10 minutes" an hour ago.
#1271 reopened at 17:37:06Z produced all five runs at 17:37:17Z. Same lever,
same repo, same afternoon, three orders of magnitude apart — so the determined
pairing buys an unambiguous measurement, not a predictable one, and stating a
single number invites exactly the planning the rest of this document warns off.

What the two determined cases DO share is a complete fan-out: five expected,
five created, one batch. Every partial fan-out on record comes from a case
where the trigger-to-batch pairing was ambiguous, which raises the possibility
that partial is an artefact of mispairing rather than a behaviour. Flagged as
n=2 rather than asserted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(runbook): partial fan-out is a behaviour, not a mispairing artefact (n=3)

- #1271 reopened 17:37:05Z, not :06 — twelve seconds, not eleven.
- Retract the n=2 "partial may be an artefact" flag. #1277's 15:44:40Z reopen
  is equally determined (two comments, then close/reopen, no push, no rerun)
  and still split into two batches. Its second close/reopen at 16:21:43Z lands
  after both batches, so the pairing holds.
- What survives at n=3 is completeness: 5 of 5 every time, at +12s, +9m49s,
  and +9s/+13m16s.
- Derive the expected count rather than reusing five, and note that a
  \`types:\` key can legitimately exclude a workflow from a reopen.
- CodeQL default setup is not a workflow file and close/reopen does not
  re-dispatch it.
- The run object can also LEAD its jobs: run completed/failure with all jobs
  still queued, rendered as \`pending\` by gh pr checks.

* docs(runbook): a queued run is not evidence of a queue

Measured repo-wide: 11 queued runs, all 11 outlived by a completed successor
of the same workflow on the same branch. Live queue depth zero. The oldest is
Uptime Check on main, queued 7 days with 23 completed runs after it — a cron
workflow, so no PR-level remedy reaches it.

- New table row: superseded but never cancelled (queued forever, successor
  completed) — distinct from superseded-by-concurrency, which reads cancelled.
- "Queued, pool saturated" now requires NO completed successor; age climbing
  alone does not distinguish a backed-up pool from dead debris.
- Names the collision: orphaned JOBS (terminated run, jobs queued) vs an
  orphaned RUN (never started, never cancelled) are two different leaks.
- Gives the one-call successor check.

Credit: sprint-review ran the repo-wide sweep.

* docs(runbook): three exclusion axes, not one — the recipe failed on its own PR

All three review points confirmed against the files, not the diff:

- Playwright Tests IS paths-gated and the bullet omitted it. Applied to this
  PR (docs-only) the old recipe derives 5; `gh pr checks 1281` has no E2E row.
  Now carries both worked examples side by side, 5 and 4.
- Smoke Tests gates on SEVEN paths, not three. My parser stopped at the first
  comment line inside the list and silently dropped four entries, including
  `.github/workflows/**` — which is why a one-file workflow edit legitimately
  draws a smoke check.
- Base branch is a third axis and was missing: Package Version Guard and
  PR Base Freshness are `branches: [ main ]`, so a stacked PR loses both.
  #1279 draws 5 where a main-based PR draws 11.

Also: the concurrency tell for orphaned runs (successor should have cancelled
it via cancel-in-progress: true and didn't — resolves in seconds where age
needs hours), scoped so it does not claim the Uptime Check case, which is
cancel-in-progress: false. Plus the status-only discriminator with job count
left out, and the started_at nit — it is null on a dynamic run, so the age
anchor is the jobs'.

Review and the discriminator: sprint-review.

* docs(runbook): gh pr checks collapses by name and hides orphaned rows

Two corrections earned after this PR's head, both on PR #1277:

- `gh pr checks` dedupes to the newest row per check name; the
  `statusCheckRollup` that computes UNSTABLE does not. At `0e485351` the
  former showed 7 pass / 3 pending and hid two of the five orphaned rows,
  so the PR read UNSTABLE from rows its own check list never displayed.
- The table's remedy for orphaned jobs was `gh run rerun`. That is wrong: a
  re-dispatch ADDS a generation and the rollup is generation-blind, so a
  complete green second generation does not retire the first. Only a new
  SHA clears it. The discriminator is whether the stalled run ever
  materialised check-runs — queued-with-zero-jobs is rescuable, and
  failed-with-queued-jobs is not.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(runbook): the check denominator has a fourth axis — the trigger event

`Release Safety` also declares `pull_request_review`, and `branches:` is not
applied to that event. Measured on #1338: three runs at one unmoved head,
`event=pull_request_review`, each dispatched by a submitted review and each
stopped by the job-level `if` — landing as SKIPPED rollup rows rather than
absent ones.

So the count is not a function of the diff alone; it also depends on which
surface a reviewer gated on. #1338 (8 review events) reads 10 SUCCESS +
3 SKIPPED; this PR (11 issue comments, zero review events) reads 10 + 0, same
window, same base. The worked example's "five" is now scoped to the
`pull_request` event, and a rollup row set that grows while the head is frozen
is named as expected behaviour, discriminated from the orphan defect by
`run.status` — these are completed/skipped, not queued.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant