Skip to content

fix(security-audit): end each orchestrator wait call before the Bash cap - #626

Draft
dormouse-bot wants to merge 4 commits into
mainfrom
fix/audit-wait-loop-returns
Draft

fix(security-audit): end each orchestrator wait call before the Bash cap#626
dormouse-bot wants to merge 4 commits into
mainfrom
fix/audit-wait-loop-returns

Conversation

@dormouse-bot

@dormouse-bot dormouse-bot commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

The nightly audit has now gone two nights without a verdict, and last night it published no report at all — issue #598. The cause is in the orchestrator's wait loop, not in the deadline it enforces.

A wait call that reaches the harness's ten-minute Bash cap is moved to the background rather than returned. It hands back no fragment listing, no DEADLINE, and nothing to decide the next step on, so whether the orchestrator re-issued a third time was a judgement call rather than a step in the recipe. Run 34457954349 (2026-09-10) happened to re-issue, its 25-minute deadline fell inside that third call, and it merged and published both completed domains as INCONCLUSIVE. Run 34581574869 (2026-09-11) spent one extra call listing the fragments between re-issues, which shifted the phase enough that a fourth call would have been needed, ended its turn at 21 minutes instead — four minutes short of its own deadline — and the session terminated with §3 never reached. supply-chain and ci-and-secrets had both written VERDICT: PASS fragments by 09:02; neither reached the issue, which says only _No audit-report.md was produced._.

So each call now ends itself at 540 seconds and prints STILL WAITING or DEADLINE, and §2 tells the orchestrator to act on that line: re-issue on the first, merge on the second. §4 gains the backstop the lost run needed — the turn must not end while audit-report.md does not exist. Nothing else about the technique changes: the deadline is still persisted in $RUNNER_TEMP, still accumulates across re-issues, and the loop is still the same until over the three fragment files.

Because the orchestrator now acts on what each call printed, the call has to end cleanly: the ls after the loop is 2>/dev/null || echo "no fragments yet", since on a slow night's first call it otherwise wrote to stderr and exited 2. And the per-call timeout: 600000 is load-bearing in a way it was not — at the harness default of two minutes the call is backgrounded before the 540-second break can print, so no answer comes back at all. BASH_DEFAULT_TIMEOUT_MS: "600000" in the audit step's env:, beside AUDIT_PAT, makes forgetting it harmless rather than fatal.

The second commit is separable and you may not want it: application-security has missed the 25-minute deadline two nights running, while roughly 13 of the job's timeout-minutes: 40 went unused on both, so it moves the deadline into that headroom (32 minutes). It is the knob, not a diagnosis — the domain spent last night's run fanning out to 15 nested subagents, re-running vitest, deploy-lint-selftest and the other repo lints, and then busy-waiting ~10 minutes on its own children, so 32 minutes may still not be enough for it. What the first commit guarantees regardless is that the two domains that do finish get published.

How the loop was verified

The sh block was extracted verbatim from the committed .github/audit/orchestrator.md, syntax-checked with bash -n, and run against a fake $RUNNER_TEMP with the two constants shrunk for the test:

  • no fragments, per-call cap reached → prints STILL WAITING, then no fragments yet, exit 0
  • re-issued → reads back the same persisted deadline rather than recomputing it
  • deadline already past → prints DEADLINE
  • all three fragments present → loop body never runs, ls lists them

pnpm lint:specs passes; docs/specs/security-audit.md came in 3 words over its 1,750-word budget and was tightened to fit rather than ratcheted. The FAIL IF covering the wait loop is scoped to the capped call, since the all-fragments call prints neither answer and §2 already names its ls listing as its own.

A Bash call that reaches the harness's ten-minute cap is moved to the
background rather than returned, so the wait loop handed the orchestrator
no fragment list and no DEADLINE — nothing to decide the next step on.
Whether it re-issued a third time was then a judgement call: run
34457954349 did, its deadline landed inside that call, and it merged and
published both completed domains; run 34581574869 spent one extra call
checking the fragments, ended its turn four minutes short of the
deadline, and published no report at all.

Each call now breaks on its own 540-second sub-cap and prints STILL
WAITING or DEADLINE, so every re-issue is driven by the loop's own
output, and §4 states that the turn must not end before audit-report.md
exists.
application-security has now failed to report inside the 25-minute
deadline two nights running — it expired on 2026-09-10, and on
2026-09-11 the domain was still sweeping when the run ended at 21
minutes. Both nights left roughly 13 of the job's 40 minutes unused, so
the deadline moves into that headroom and timeout-minutes: 40 stays
above it.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 11, 2026

Copy link
Copy Markdown

Deploying mouseterm with  Cloudflare Pages  Cloudflare Pages

Latest commit: e0a9399
Status: ✅  Deploy successful!
Preview URL: https://17dab339.mouseterm.pages.dev
Branch Preview URL: https://fix-audit-wait-loop-returns.mouseterm.pages.dev

View logs

@dormouse-bot dormouse-bot left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feedback on work in progress — not a merge verdict.

On the separable second commit: the headroom is fine. On run 34457954349 the steps before the agent step took 21 seconds and the steps after it took 4, so a 32-minute wait puts the job at roughly 33 of its 40 minutes — timeout-minutes: 40 does not need to move with the deadline.

Three inline findings. One point beyond the diff: timeout: 600000 is now load-bearing in a way it was not. Before this change a short timeout only cost extra re-issues; now it costs the printed answer entirely, and the only thing holding it is the orchestrator reading one sentence. BASH_DEFAULT_TIMEOUT_MS: 600000 in the audit step's env: block — beside AUDIT_PAT, which already shows the action forwards step env into the agent's Bash — would make a forgotten timeout harmless rather than fatal, if the harness honors it.

Comment thread .github/audit/orchestrator.md Outdated
Comment thread docs/specs/security-audit.md Outdated
Comment thread .github/audit/orchestrator.md Outdated
Three corrections from review, plus the env backstop behind them.

The `ls` that follows the loop wrote to stderr and exited 2 whenever no
fragment existed yet, which is the most common `STILL WAITING` call on a
slow night. That was invisible while capped calls were backgrounded and
their tail never read; now §2 tells the orchestrator to act on what the
call printed, so the call has to end cleanly. `2>/dev/null || echo "no
fragments yet"` gives it a tail and exit 0.

The `timeout: 600000` sentence still described the pre-sub-cap world,
where a short timeout only bought more re-issues. At the harness default
of two minutes the call is now backgrounded before the loop's own
540-second break can print, so no answer comes back at all — the run
34581574869 failure the block exists to end.

`BASH_DEFAULT_TIMEOUT_MS: 600000` in the audit step's `env:` makes that
harmless rather than fatal: the prompt still asks for the per-call
`timeout`, but forgetting it no longer costs the printed answer. The
AUDIT_PAT entry beside it already shows the action forwards step env to
the agent.

The spec's `FAIL IF` asserted that every call prints `STILL WAITING` or
`DEADLINE`; a call that finds all three fragments never enters the loop body
and prints neither, answering with the `ls` listing alone, as §2
already says. Narrowed to the capped call, word-neutral so the 1,750-word
budget needs no ratchet.

@dormouse-bot dormouse-bot left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feedback on work in progress — not a merge verdict. Mark it ready when you want the full review.

Two inline findings, both on material this commit added. The loop itself behaves as the four documented cases claim — I re-ran the committed block against a fake $RUNNER_TEMP with the sub-cap shrunk to 2s and each case printed what §2 says it does.

Comment thread .github/audit/orchestrator.md Outdated
Comment thread .github/workflows/security-audit.yaml
The orchestrator was told to act on "the loop's own last line", but the
`ls` after the loop always runs, so a re-issue call's last line is
`no fragments yet` or a partial listing — neither of the two answers. Act
on what the call printed instead, and call the two things decisions: the
`ls` listing all three fragments is one of the inputs, not a third answer
the loop was supposed to print.

Also state the true scope of `BASH_DEFAULT_TIMEOUT_MS`: it is a
process-wide default, so it raises the cap on the domain subagents' Bash
calls too, not just the orchestrator's wait.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant