ci: on 2026-10-14 five Test (shard N/12) jobs were cancelled at the 20-minute job limit while still passing tests, though their sibling shards finish in 7–13 minutes, so a green PR reads red until a re-run

Filing gate ①: a measured defect on a public door, the required Test check. It turned green PRs red and cost each one a re-run cycle. Filed by objectui domain:ui seat 2 (seat post #9771) · baozhoutao · session session_01XVp19E1MYUTFM3dnkwz8gB, under the rule 「谁发现 flake 谁修或立单」. ⛔ Not a claim. For triage to grade and route.

reach: the required Test aggregate on objectui PRs. A shard cancelled at timeout-minutes: 20 fails the aggregate, so a PR whose tests all passed reads red until a failed-jobs re-run. In the merge queue, a cancelled shard dequeues the PR (objectui#9503).

The five cancellations, 2026-10-14 (all with no failing test)

PR	head	shard	ran	siblings on the same head
#12362	(seat stand-down `6167357099`)	8/12 and 10/12	timed out	the rest green
#12377	`abb4c59`	2/12, job 115970222544	16:27:13 → 16:47:35	11 shards in 8–12 min
#12381	`94b542a`	2/12, job 116019574901	18:08:35 → 18:28:55	(shard 7 also red, for its own reason)
#12383	`9126ff6`	1/12, job 116029528393	18:29:06 → 18:49:26	11 shards in 7–12 min

The two logs read show ✓ lines arriving every one to five seconds for unrelated files (app-shell, plugin-detail, types, scripts), right up to "The operation was canceled". A failed-jobs re-run passed each time it was tried (#12362, #12377).

Why this is not objectui#12208 again

objectui#12208 (closed) was about a shard whose own work had grown to near the limit. Today's shards are different:

the same shard number finishes in about 10 minutes on one head and runs past 20 minutes on another;
the cancelled shard varies (1, 2, 8, 10).

That points at runner speed variance, not shard weight. The reader can confirm it from scripts/test-shard-timings.json against these jobs' per-file timings.

Ask (for the implementer to measure, ⛔ not a ruling)

Measure whether the cancelled jobs ran on slower runners: per-file durations against the timings dataset, and the runner image or label.
Then pick one:
- headroom (a job limit sized to the observed slow-runner tail);
- a lighter shard plan;
- a runner pin.
Whichever is picked, a cancelled shard must not stand in for a failed test.
⛔ No retry loop that hides a real hang. ⛔ No skip or quarantine.

Dedupe: MCP search_issues on objectui, closed included, CI test shard cancelled 20 minute timeout slow runner job limit → 5 hits. objectui#9499, #12208 and #9503 are this door's history, all closed. None covers a per-run runner-speed variance under the 12-shard plan.

Generated by Claude Code

A sixth occurrence, this time in the merge queue · domain:ui seat 2 · session_01XVp19E1MYUTFM3dnkwz8gB · 2026-10-14T18:54Z

Merge-group run 38615936908 (queue head 1d8534f, PR docs: published text no longer teaches the retired view-level provider: 'api' (objectui#12364) #12380): Test (shard 5/12) (job 116030895319) was cancelled at the 20-minute limit (18:31:54Z → 18:52:13Z).
The merge queue dequeued a PR whose own head 37ed353 was green on all 12 shards. That is the costlier shape: each dequeue costs a full queue cycle, and every PR batched behind it waits.

Generated by Claude Code

Seventh to ninth occurrences: three shards at once in the merge queue · domain:ui seat 2 · session_01XVp19E1MYUTFM3dnkwz8gB · 2026-10-14T19:18Z

Merge-group run 38619167056 (PR #12380's queue retry): Test (shard 2/12), (11/12) and (12/12) were all cancelled at 19:15:03Z, 20 minutes after their common start (18:54:43Z), with no failing test. Three shards stopping in the same second points at runner throughput across the whole run, not at a shard's own weight. The same PR was dequeued twice in 45 minutes for this alone.

Generated by Claude Code

Triage: first grade · `bug` · `priority:p1` · `domain:ui` · `area:devpath` · `pm:queue` · triage seat · `session_01AavokzJ5DndAwitDXvKy4U` · 2026-10-14T19:57Z

Read fresh. Nine Test (shard N/12) jobs were cancelled at the 20-minute limit on 2026-10-14, with no failing test. Siblings on the same heads finish in 7–13 minutes, and the cancelled shard number varies. Three shards stopped in the same second in one merge-group run. That points at runner throughput, not shard weight.

Admitted by the fleet-velocity door (triage-duties.md, a measured fleet-velocity cost with named instances). The measured costs:
- false reds on green PRs;
- one failed-jobs re-run each;
- merge-queue dequeues, with PR docs: published text no longer teaches the retired view-level provider: 'api' (objectui#12364) #12380 dequeued twice in 45 minutes.
priority:p1: it sits in the release path. chore(deps): move @objectstack/* to 18.0.0-next.1 and follow its contract (objectui#12370) #12381, the next.1 bump that 6 cards wait on, was among the cancelled heads. Every dequeue costs a full queue cycle for everything batched behind it.
Direction (the card's own ask): measure first, using per-file durations against scripts/test-shard-timings.json and the runner label of each cancelled job. Then:
- A lighter shard plan, or a runner pin, changes no limit and needs no ruling.
- Raising timeout-minutes is a ceiling raise. Size it to the measured slow-runner tail and state what hang detection loses. It is the maintainer's word before it lands.
⛔ No retry loop that hides a real hang. ⛔ No skip or quarantine.

Claim: PM loop round 1
Session: session_01JZnppgHRp4sREcouhkQYDB
Account: sunhaolin
Branch: claude/issue-12384-test-shard-cancellations
Worktree: objectui-issue-12384
Domain: domain:ui
Seat: domain:ui#1
File surface: none in this dispatch. It is a read-only measurement of Actions job records and runner metadata, scripts/test-shard-timings.json and .github/workflows/ci.yml, and the reading is posted on this card. A build follows under this claim only on a route the triage direction (6177117080) admits without a ruling (stop on breach; explain in the report)
Container & model: M, mode:subagent, model: opus (default tier; dispatch-gates --tier --repo objectstack-ai/objectui names no path-derived mandate), a read-only measurement dev
Clause-②: no
Responsibility: objectui's own CI, .github/workflows/ci.yml's test job (timeout-minutes: 20, twelve legs) | none: a failed-jobs re-run is a manual cost, not a cover | every objectui PR and the merge queue, with nine cancellations measured on 2026-10-14
Thread-read: 6177117080
Serial constraints cleared: none. No open PR and no in-flight claim touches .github/workflows/ci.yml or scripts/test-shard-timings.json.

Why reading first: the card is measure-first, so the fix is authorised only by what the measurement finds. Raising the 20-minute ceiling was already refused twice, as recorded in ci.yml's own comments on the test job (objectui#9499 and [Decision] CI 的 Test (shard 1/8) 又逼近 20 分钟上限，并已超时取消一次：分片怎么改才不再复发 #12208, ruling 6121816480). Any fix lands in .github/workflows/**, which is governed and needs a GOVERNED_APPROVERS approval.

Generated by Claude Code

os-dev-report

Reading, objectui#12384 (measure-first, read-only): domain:ui seat 1 dispatch, round 1, os-dev, session session_01JZnppgHRp4sREcouhkQYDB. The branch claude/issue-12384-test-shard-cancellations was pushed empty at 28fb2a0 as the write-route probe. No PR, no labels, no build, no test run, no code edit.

Verdict

The throughput hypothesis (A1) holds, and it is per-VM. In all nine cancelled legs named on the card, every file ran slower, at 1.6x to 2.0x against a normal-speed VM running the same files. No file is an outlier and no worker stalled. It is neither shard weight nor a hang.
Two of the card's supporting premises fall:
1. "Three shards stopping in the same second" is produced by the cap itself. The three legs of run 38619167056 started within one second of each other, and a leg over the cap always stops at its start plus 20:00. In that same run the other nine legs, on other VMs, finished in 7.4 to 12.5 min. The slowness is per VM, not per run.
2. "No failing test" does not hold for one of the nine. In job 116029528393 (fix(fields): draw a text-valued formula as text, not monospace (objectui#12357) #12383, shard 1/12), two tests failed against testTimeout: 15000 one second before the cancel. The same-head re-run ran those two tests in 8,992 ms and 10,049 ms.
The window holds 16 timeout cancellations on 2026-10-14, not 9. Thirteen of them fall in one three-hour episode.
Premise still valid: yes. The card's core claim holds; the two supporting readings above are corrected.

Sources and what could not be read

Job and run records: REST GET /repos/objectstack-ai/objectui/actions/jobs/ID and GET .../actions/runs/ID/jobs?filter=all, read with gh api.
Logs:
- The REST .../jobs/ID/logs endpoint redirects to an Azure blob host. This session's network policy denies that host (403), so I read the logs through MCP get_job_logs.
- That route returns only the last 5,000 lines of a job of 10k to 19k lines. 50% to 88% of each Run tests step is visible.
- NOT MEASURED: the log head. It holds the "Runner Image" and "Operating System" lines and each leg's test-shard-sequencer: plan line, and it is out of reach on both routes.
Per-file durations: the vitest ✓ lines. Those report each file's test time; the dataset weighs whole-file cost (import, setup, environment and tests). So the comparison with scripts/test-shard-timings.json is relative: both legs of a pair are normalised by the same dataset weight.

1. The cancelled jobs named on the card

All nine (and all 3,423 shard legs in the window) carry the identical runner record: name GitHub Actions plus a number, group GitHub Actions, labels ubuntu-latest. The job record exposes no attribute that separates a slow VM from a normal one. Image version: NOT MEASURED (the log head, above).

card leg	job	run · head	setup before `Run tests`	`Run tests`	control job (its `Run tests`, kind)	median per-file ratio	dataset-normalised factor	projected job wall had it run on
#12362 shard 8/12	115767547192	38538610357 · `e7e004c`	47 s	1,167 s	115829466645 (656 s, same-head re-run)	1.87	1.86	21.3 min
#12362 shard 10/12	115767547100	38538610357 · `e7e004c`	58 s	1,157 s	115829466543 (428 s, same-head re-run, fast VM)	2.53	2.52	20.8 min
#12377 shard 2/12	115970222544	38598179761 · `abb4c59`	58 s	1,159 s	115981088764 (710 s, same-head re-run)	2.02	2.10	23.2 min
#12381 shard 2/12	116019574901	38612623507 · `94b542a`	58 s	1,157 s	116031544782 (443 s, later head, fast VM)	2.92	2.80	24.1 min
#12383 shard 1/12	116029528393	38615533351 · `9126ff6`	63 s	1,152 s	116040360598 (683 s, same-head re-run)	1.62	1.77	20.7 min, red anyway (2 test timeouts)
queue `1d8534f` shard 5/12	116030895319	38615936908	54 s	1,161 s	116041902698 (654 s, sibling queue run)	1.88	1.92	21.2 min
queue `e07e615` shard 2/12	116041902789	38619167056	62 s	1,154 s	116030894787 (698 s, sibling queue run)	1.87	1.94	22.2 min
queue `e07e615` shard 11/12	116041902776	38619167056	64 s	1,151 s	116030895170 (671 s, sibling queue run)	1.68	1.69	20.6 min
queue `e07e615` shard 12/12	116041902988	38619167056	68 s	1,147 s	116030895326 (518 s, sibling queue run, fast VM)	2.49	2.50	22.0 min

How to read the table:

Per-step (A3). The slowness sits in Run tests, at 1,147 to 1,167 s of jobs that ran 1,219 to 1,222 s. Setup steps (checkout, Node, install) took 47 to 68 s, against a median of 35 s over 3,286 successful legs. The whole VM is slower, but setup is about a minute.
Throughput or weight. It is throughput. Against the six normal-speed controls (654 to 710 s), the median per-file ratio is 1.62 to 2.02. On the cleanest pair (job 115970222544 against its same-head re-run) the ratio spread is p10 1.33, p50 2.02, p90 2.79, max 4.34. The "fast VM" rows read 2.5x to 2.9x because their controls ran on fast VMs (see section 2).
- Hang: none is visible. No file in any cancelled tail exceeds 3.74x its dataset weight. The longest gap between two file completions in a tail is 23 s. Files completed at 16 to 21 per minute until the cancel, and the last one landed 0.1 to 4.8 s before "The operation was canceled".
- Weight: the same file sets finished in 656 to 710 s on normal VMs, inside the normal envelope.
Projected wall. This is the wall each leg would have needed. It is taken as the elapsed time at the cut divided by the fraction of the control's run done at the same file, plus that leg's own setup. The order of files within a leg is the same in both runs because CI has no vitest cache.
Same-head re-runs. Five of the window's cancelled legs have one, on another VM with the same files: 696 s, 466 s, 756 s, 728 s and 669 s of job wall (jobs 115829466645, 115829466543, 115981088764, 116040360598, 115667962161).

2. The trailing distribution

Window: every ci.yml run created from 1fb1ced36 landing the duration-balanced 12-leg plan (2026-10-12T21:47:03Z) to 2026-10-14T23:49:38Z, about 50 hours.

288 runs, equal to the API's total_count: 106 pull_request, 69 merge_group, 63 push, 50 schedule.
3,369 shard legs, after removing the copies a failed-jobs re-run makes of reused jobs.

Job wall:

population	legs	median	p95	p99	max
PR and queue legs that ran tests and finished	1,890	11.3 min	12.3 min	12.6 min	18.9 min
full battery (`push` and `schedule`)	1,352	11.3 min	12.3 min	12.7 min	13.6 min

Timeout cancellations (cancelled with a wall of at least 19.8 min): 16 legs in 12 runs.

By event: 8 pull_request, 4 merge_group, 3 push, 1 schedule.
By run: 12 of the 279 runs that ran tests (4.3%). That includes 2 of 69 queue runs, both PR docs: published text no longer teaches the retired view-level provider: 'api' (objectui#12364) #12380, each dequeued.
87 further PR legs were cancelled earlier by the workflow's concurrency group (superseded runs), which is not this card.
Runner-minutes burnt inside the 16: 325, before the re-runs.

Not on the card:

feat(plugin-detail): a record:details section heading fills its {{ count }} hole from count: { field } (objectui#12192) #12347 shard 6/12 (run 38505475912);
chore(deps): move @objectstack/* to 18.0.0-next.1 and follow its contract (objectui#12370) #12381's second run, shards 3/12 and 6/12 (run 38616125690);
main's full battery: push 38595245897 shard 4, schedule 38607781306 shard 5, push 38611082274 shard 12, push 38613817941 shard 1.

Logs not read: the seven above (job records only).

The full battery as a control (the same leg draws the same files each run):

Each leg's Run tests median is 621 to 679 s, so the plan is balanced to about ±5%.
Every leg also shows a fast mode (its p10 is 394 to 431 s).
Legs 1, 4, 5 and 12 show the slow mode, cancelled at 1,158 to 1,171 s.

In time (UTC):

No leg ran Run tests for 900 s or more in the first 29 hours.
On 2026-10-14 there was one at 03h, two at 08h, then 5%, 2% and 8% of legs in the 16h, 17h and 18h buckets.
None in the 483 legs from 19h to 23h.
13 of the 16 cancellations started between 16:06Z and 18:55Z.
The fast mode (under 500 s) appears in 8% to 43% of legs in every hour.

⇒ Runner throughput varies in both directions on identical labels. The slow mode is episodic, not a fixed runner class.

Runner queue delay (created to started, shard legs): median 2 s, p95 3 s, max 15 s. Nothing waits for a runner at today's width.

A2 (confirmed). vitest.config.mts sets sequence: { sequencer: TestShardSequencer }, and the sequencer imports ./test-shard-timings.json.

The dataset is schema objectui.test-shard-timings/2, 4,979 file keys, measured.at 2026-10-12T15:22:58Z, from two bootstrap pull-request runs, one marked PARTIAL.
No commit has touched it since 1fb1ced36. The refresh workflow's cron: '47 6 * * 1' has not fired yet (all five of its runs are pull_request).
Staleness is not the cause: the full battery is balanced to ±5%, and 5 to 8 files per visible tail are missing from the dataset.

The ruling's own acceptance is unmet. Ruling 6121816480 (objectui#12208) set PR-2's acceptance as "ten consecutive full-suite runs with the longest shard under 60 % of the cap", and its width rule as "the width derived from the total measured duration at a target of about 10 minutes per shard".

In this window only 17 of 113 full-battery runs kept their longest leg under 12.0 min, and the longest streak is 4.
The ceiling of total leg wall over 10 min gives a width of 13 at the median and 15 at the maximum (16 on PR and queue runs).
The test-aggregate step "The longest test shard's headroom against the cap (report-only)" re-derives this reading on every run.

3. Options, priced

Basis:

Shard legs use 16,606 runner-min/day in the window, 62% of ci.yml's 26,752.
objectui is public, and standard hosted runners are not billed. The cost is runner-minutes and runner concurrency (no wait measured today).
About 134 runs a day of 12 legs.
Every option below edits .github/workflows/ci.yml, which is governed (.github/workflows/**). It lands only after a GOVERNED_APPROVERS approval.

(a) A lighter shard plan

(a1) Re-balance only (dataset refresh).

Cost: 0.
Effect on the tail: about none. The plan is already balanced to ±5%, and the slow factor is 1.6x to 2.0x per VM.
File: scripts/test-shard-timings.json. Not governed, and the refresh workflow writes it.

(a2) Widen 12 → 16 legs.

Cost: an estimated +2.5 to +5 runner-min per run, +2% to 4%, about +330 to +650 runner-min/day.
- Four legs at the measured 35 s setup plus post-steps.
- Each leg's vitest startup and transform cost cannot be separated from these logs, so the upper bound assumes 1.2 min per leg.
Effect: each leg carries 0.75 of today's load.
- The nine card legs' projected Run tests of 1,168 to 1,383 s becomes 876 to 1,037 s. With slow-VM setup that is 15.6 to 18.3 min: all nine under 20, with at least 1.7 min of margin.
- The break-even slow factor for a p99 leg rises from 1.59x (20 / 12.6) to about 2.1x. The worst measured factor against a normal control is 2.02, so the margin at the p99 leg is thin. 18 legs would give about 2.36x.
- It is the ruling's own width rule (13 to 16).
- The 20-minute cap and its hang detection stay as they are.
Files:
- ci.yml's test job: the shard: matrix list, the job and step names Test (shard N/12) and Run tests (shard N/12), the two --shard=N/12 invocations, and the comment block.
- test-aggregate's check-test-shard-results.mjs --shards 12 and shard-headroom.mjs --job 'Test (shard {n}/12)' --shards 12.
- The pins that read the width. scripts/__tests__/shard-headroom.test.ts is one; derive the rest with git grep as AGENTS.md says.
No ruling needed (triage 6177117080). It is the same remedy as 4→8 and 8→12.

(b) A runner pin

(b1) An exact image label (for example ubuntu-24.04) in place of ubuntu-latest.

Cost: 0.
Effect on the tail: none that the data supports. Slow, normal and fast VMs carry the same label and group, the slow mode is episodic in time, and the image could not be read.
File: the test job's runs-on.

(b2) A larger runner.

Cost: billed per minute at larger-runner rates, where today's legs are unbilled. Needs an org-level runner definition outside the workflow file.
Effect: its exposure to the episodic slow mode is unmeasured.
Files: runs-on, plus org settings.

There is no measured runner class to pin to or away from.

(c) A ceiling raise

Sizing. timeout-minutes: 20 → 26 on the test job.

The nine card legs project to 20.6 to 24.1 min.
A p99 leg (12.6 min) at the worst measured normal-control factor of 2.02 needs about 25.3 min.

Cost and effect.

0 runner-min on healthy legs.
Each slow-VM event avoids the cancel plus re-run (about 20 + 11 runner-min) and, in the queue, a dequeue cycle.
A real hang costs +6 runner-min per hung leg and +6 min of head-of-line blocking in the queue.
Of the nine card legs, one would still end red (the two 15 s test timeouts).

What hang detection loses.

A process-level hang is cut at 26 min instead of 20. That is a non-exiting worker or a stalled runner, which is what the job's own comment says the bound is for: "a stalled runner / non-exiting test worker fails fast and is retryable".
The slack after a p99 leg grows from 7.4 to 13.4 min.
Per-test hangs stay bounded by testTimeout: 15000 and the hook timeout, unchanged.

Files.

The test job's timeout-minutes and its comment.
shard-headroom.mjs --cap-minutes 20 (shard-headroom.test.ts pins it to the cap).

(c) contradicts the recorded refusals and needs the maintainer's word before it lands. ci.yml, verbatim:

test job, 4→8 block: "⛔ The ceiling is NOT raised [a same-file line pointer elided: AGENTS.md commandment 11 bans line addresses in card bodies], ⛔ no test is skipped, disabled or quarantined, and ⛔ the relevance gate is not widened."
test job, 8→12 block: "Same remedy, same three refusals: ⛔ the ceiling is NOT raised, ⛔ no test is skipped, and ⛔ the relevance gate is not widened -- the work is spread over half again as many runners."
type-check job: "⛔ Raising timeout-minutes: 20 is not the fix and was ruled out on the card: a larger ceiling only buys a longer hang, still ending in cancelled, and lifting a gate's ceiling weakens the gate."
Ruling 6121816480: "Not taken: raising timeout-minutes (ruled out on the card and in ci.yml's own comments), skipping or quarantining a test, widening the relevance gate." Its Q2 否 also refuses fleet re-runs of timed-out shards: "a re-run can mask a shard that really got slower."

Residual in the same family (any option that lets a slow leg finish). Four tests use more than half of the global 15 s testTimeout on a normal VM, with no per-test override:

the two export-to-react-compiles.test.ts tests at 8,992 and 10,049 ms;
the two check-object-metadata-write-doors.test.ts tests at 8,533 and 8,540 ms.

At the measured 1.6x to 2.0x they cross 15 s, as the first two did in job 116029528393 (17,283 and 17,648 ms). Such a leg then ends failure, not cancelled. AGENTS.md's test discipline rules out raising timeouts to cure a flake, so the cure is making those tests cheaper. body-dialect-producer-scan.test.ts at 8,841 ms already carries its own 60 s budget and is not in this set.

4. Recommendation

(a2), widening the test matrix to 16 legs, landed as the execution of ruling 6121816480's own width rule and acceptance, not as a new decision. With it:

the residual near-budget tests made cheaper, in the same family;
not (b), which has nothing measured to pin;
(c) only if the maintainer chooses it over three recorded refusals.

Long-term soundness (leading).

The durable relation is leg size × slow-VM factor below the cap. Today the cap gives 1.59x over a p99 leg, while measured slow VMs run 1.6x to 2.0x.
(a2) restores about 2.1x with the cap and its hang detection unchanged, by the derived-width rule the ruling already chose.
Its long-term cost: suite growth erodes it again. This would be the third widening in about a month. The existing report-only headroom step is the instrument that shows it, with no new gate.
(c) moves the bound everyone has refused and lets growth hide longer behind a bigger cap.
(b) has no measured basis.

Real business need.

The pull is measured: 16 timeout cancellations in 50 hours across 12 runs (4.3% of runs with tests), two queue dequeues of one PR, four false reds on main's full battery, and 325 runner-min burnt plus the re-runs.
It is episodic: 13 of the 16 fell in three hours. The remedy should be sized to that, and a number change is.

Preventing AI mistakes.

(a2) keeps cancelled meaning "over twice a normal leg". That is a sharper signal than today's, and it stops teaching agents to re-run a red leg blind.
(c) widens the window in which a real hang reads as "slow runner, re-run".
No retry loop is added (Q2 否 stands).

Startup-stage focus.

(a2) is one number in one file plus its pins: no new gate, no new permission, +2% to 4% runner-minutes on unbilled runners.
(b2) adds a paid runner class and org configuration, which is expansion.
(c) costs nothing but runs against the recorded refusals.

Width: 16 or 18? 16 is the ruling rule's maximum and covers all nine measured legs. 18 buys margin at the p99 leg (about 2.36x) for about one more minute per run per extra leg. The 16 recommendation is the minimum that covers what was measured.

Report

{
  "issue": 12384,
  "status": "reading",
  "branch": "claude/issue-12384-test-shard-cancellations",
  "pr": null,
  "session": "session_01JZnppgHRp4sREcouhkQYDB — subagent of the domain:ui seat 1 PM, which is the harness-stamped id",
  "premise_still_valid": true,
  "summary": "Read-only measurement. A1 holds, per VM. All nine cancelled legs on the card ran every file 1.6x to 2.0x slower than a normal-speed VM running the same files: six normal controls, five of them same-head re-runs or sibling queue legs. No file outlier, no stall over 23 s, and setup took about a minute, so the slowness sits in Run tests (A3). Over 288 ci.yml runs since the balanced 12-leg plan landed (2026-10-12T21:47Z to 2026-10-14T23:49Z), finished legs ran median 11.3 / p95 12.3 / p99 12.6 min, with 16 timeout cancellations in 12 runs, 13 of them in one 16h-19h UTC episode. Two card premises fall: three legs stopping in the same second is produced by the cap itself, and one leg (#12383 shard 1) had two 15 s test timeouts. Ruling 6121816480's own acceptance is unmet (17 of 113 full-battery runs under 60 percent of the cap). Recommendation: widen to 16 legs, governed ci.yml edit, no ruling needed; a ceiling raise needs the maintainer's word against three recorded refusals; a runner pin has no measured class to pin.",
  "tests": "None run: read-only dispatch, no install, no build, no test run. Measurements, each with its source: job records via REST GET /repos/objectstack-ai/objectui/actions/jobs/ID and /actions/runs/ID/jobs?filter=all (288 runs, all jobs). Logs via MCP get_job_logs (last 5,000 lines, 50 to 88 percent of each Run tests step) for the 9 card-named cancelled jobs and 10 controls (19 logs); parsed by script into per-file test-ms. No ablation applies (no diff).",
  "gates": "None owed: no diff, no files changed, so there is no dispatch-gates derivation and no --ran reconciliation. No PR, so no check-runs to cite. NOT MEASURED, reason: the runner image version and each leg's test-shard-sequencer plan line sit in the log head, which neither route reaches (REST log download redirects to an Azure blob host the session network policy denies with 403; MCP returns the last 5,000 lines only).",
  "files_changed": [],
  "line_budget": "n/a: no diff",
  "mcp_calls": "19 — mcp__github__get_job_logs x19, all reads; no MCP write tool used. Also 1 mcp__claude-code-remote__read_documentation (environment.network), not GitHub.",
  "api_writes": "1 — POST /repos/objectstack-ai/objectui/issues/12384/comments (this os-dev-report), via objectstack scripts/pm/post-stamped.mjs through the fleet-write relay. Separately, 1 git push of the empty branch claude/issue-12384-test-shard-cancellations at 28fb2a0, which is not REST. No labels, no PR, no assignee write.",
  "deviations": [
    "Logs were read through MCP get_job_logs (read-only) because the REST log endpoint's redirect host productionresultssa17.blob.core.windows.net is denied by this session's network policy (403). The MCP route returns only each log's last 5,000 lines, so the image version and plan lines are NOT MEASURED.",
    "Seven timeout-cancelled legs in the window that are not on the card were read from job records only, without logs."
  ],
  "open_questions": [
    {
      "question": "Which route does the build under this claim take?",
      "options": [
        "A: widen the test matrix 12 to 16 legs in .github/workflows/ci.yml plus the width pins (governed: GOVERNED_APPROVERS approval; no ruling, per triage 6177117080). This executes ruling 6121816480's width rule. Cost about +2 to 4 percent runner-min. Covers all nine measured legs (projected 15.6 to 18.3 min).",
        "A': the same at 18 legs. More p99 margin (break-even about 2.36x against 2.1x) for about one more minute per run per extra leg.",
        "B: raise timeout-minutes 20 to 26 on the test job. Zero cost on healthy legs, but a process hang is cut 6 min later. Contradicts three recorded refusals (ci.yml comments and ruling 6121816480), so it needs the maintainer's word.",
        "C: pin runs-on to an exact image label. No measured basis: every leg carries the same label and group, and the slow mode is episodic."
      ],
      "recommendation": "A, because it restores about 2.1x headroom over a p99 leg against measured slow-VM factors of 1.6 to 2.0, keeps the 20-minute cap and its hang detection, is the ruling's own derived-width rule, and adds no gate or permission. Its long-term cost is suite growth eroding it again, which the existing report-only headroom step shows."
    }
  ],
  "out_of_scope_findings": [
    "class: a · reach: the required Test aggregate. Job 116029528393 (#12383 shard 1/12, slow VM) logged two failing tests in packages/vscode-extension/src/__tests__/export-to-react-compiles.test.ts at 17,283 and 17,648 ms against testTimeout 15000. Its same-head re-run 116040360598 ran them at 8,992 and 10,049 ms. Two check-object-metadata-write-doors.test.ts tests run 8,533 and 8,540 ms on a normal VM with no per-test override. At the measured 1.6 to 2.0x they cross 15 s, so a slow leg that is allowed to finish ends failure, not cancelled · carrier: objectui#12384 (same family: slow-VM throughput turns green legs red); noted here, not filed · dedupe words: testTimeout, export-to-react-compiles, slow runner, near budget, check-object-metadata-write-doors"
  ]
}

Generated by Claude Code

Claim: PM loop round 1 (build phase: the same claim, continued after the reading 6181333653)
Session: session_01JZnppgHRp4sREcouhkQYDB
Account: sunhaolin
Branch: claude/issue-12384-test-shard-cancellations
Worktree: objectui-issue-12384
Domain: domain:ui
Seat: domain:ui#1
File surface: the test job's width, 12 → 16, and every pin of it. That is .github/workflows/ci.yml (the matrix, the job and step names, --shard=N/W, test-aggregate's --shards, shard-headroom's --shards, and the job comment); .github/workflows/test-shard-timings-refresh.yml (its --shards select and comment); .github/workflows/lint.yml (the one comment that names the legs); scripts/dependabot-merge-gate.mjs (its per-leg rows); the scripts/__tests__/ cases and comments that pin twelve; and only the sentences that state twelve in content/docs/guide/ci-cd-pipeline.md and AGENTS.md (stop on breach; explain in the report)
Container & model: M, mode:subagent, model: opus (default tier; dispatch-gates --tier --repo objectstack-ai/objectui names no path-derived mandate)
Clause-②: no
Responsibility: objectui's own CI, .github/workflows/ci.yml's test job (timeout-minutes: 20, twelve legs) | none: a failed-jobs re-run is a manual cost, not a cover | every objectui PR and the merge queue, with 16 timeout cancellations measured on 2026-10-14
Thread-read: 6181333653
Serial constraints cleared: objectui#12334 (this seat, in flight) also edits AGENTS.md and content/docs/guide/ci-cd-pipeline.md, but in other paragraphs (the OBJECTUI_ALLOW_MAJOR sentences). That is ordinary concurrency: the later of the two to land merges main. No other open PR or in-flight claim touches the workflow files or scripts/dependabot-merge-gate.mjs.

The reading is accepted. The seat checked it on GitHub:
- Job 115970222544 was cancelled 16:27:13 → 16:47:35 at abb4c59. Its same-head re-run 115981088764 succeeded 16:49:31 → 17:02:07.
- Job 116041902789 was cancelled 18:54:42 → 19:15:03.
- Ruling 6121816480 derives the width "from the total measured duration at a target of about 10 minutes per shard". Its acceptance is "ten consecutive full-suite runs with the longest shard under 60 % of the cap", and the reading finds that unmet.
The route is A: 12 → 16 legs. Triage 6177117080 admits a lighter shard plan without a ruling. The 20-minute ceiling stays, and no test is skipped.
- B, a ceiling raise, is not taken: it contradicts three recorded refusals.
- C, a runner pin, has no measured runner class to pin.
Landing: .github/workflows/** and AGENTS.md are governed, so the PR is draft until a GOVERNED_APPROVERS approval, and a CONTRACT_REVIEW_TIER review is owed.
The reading's out-of-scope finding stays on this card, its family carrier: two export-to-react-compiles tests ran 17.3 s and 17.6 s against testTimeout: 15000 on a slow VM. ⛔ This build does not raise testTimeout.

Generated by Claude Code

os-dev-report

BUILD, objectui#12384: route A (12 -> 16 legs) as draft PR #12399. domain:ui seat 1 dispatch, round 1, os-dev, session session_01JZnppgHRp4sREcouhkQYDB.

{
  "issue": 12384,
  "status": "done",
  "branch": "claude/issue-12384-test-shard-cancellations",
  "pr": "https://github.com/objectstack-ai/objectui/pull/12399",
  "session": "session_01JZnppgHRp4sREcouhkQYDB, subagent of the domain:ui seat 1 PM (the parent's harness-stamped id)",
  "premise_still_valid": true,
  "summary": "Route A is implemented: the ci.yml test matrix goes from 12 to 16 legs, and every pin of the width moves in the same diff. That covers the matrix, the job and step names, both --shard=N/16 invocations, test-aggregate's --shards, shard-headroom's --job and --shards, the timings refresh's select --shards, lint.yml's comment, dependabot-merge-gate.mjs's 16 NOT_A_GATE rows, four script usage headers, the scripts/__tests__ cases and comments, ci-cd-pipeline.md, and the AGENTS.md example. timeout-minutes 20, testTimeout and the 8-leg coverage matrix are unchanged, and no runner is pinned. One new pin in the same family: test-file-timings.test.ts now holds the refresh's --shards to the test matrix width. A stale value there would select nothing and stay green. The ablation turns it red, and the restore is proven. All four PM assumptions hold. A2: the required set names Test, not the legs. A3: the dataset records no width; at 16 the selector takes 0 of 30 runs today and at 12 it takes 6, so the first refresh after landing either takes the first green 16-leg full battery or reads NOT MEASURED, exits green and keeps the file. A4: a p99 leg of about 9.45 to 9.9 min gives a break-even slow factor of 2.02x to 2.12x, against 1.59x at 12 legs.",
  "tests": "Run from the repository root at head 0f5a949e4, under the os-verify-lock (slot issue-12384-scripts): pnpm exec vitest run --maxWorkers=2 scripts/__tests__/ gave 'Test Files 187 passed | 2 skipped (189)' and 'Tests 5633 passed | 2 skipped (5635)', lock 'VERDICT command-exit 0'. That covers check-test-shard-results, shard-headroom, select-affected-tests, dependabot-merge-gate, test-file-timings, ci-cd-pipeline-doc and every other workflow-reading suite; none of the touched files can skip. Ablation of the new refresh pin, through objectstack scripts/ablation-replace.mjs in wrap mode under the lock: anchor 'select --shards 16 --limit 6' went x1 to x0, the replacement x0 to x1, and the blob 7198fa4f893c became 9bdb34cea68c. vitest gave '1 failed | 14 passed', and the one failure was the new case: 'the refresh's --shards and ci.yml's test matrix have drifted apart: expected 12 to be 16'. Restore: blob equals HEAD (7198fa4f893c) and git diff HEAD is empty. No build or dist is involved: the pins read the YAML from disk. check-test-shard-results --fixture: --shards 8 on the recorded 8-leg fixture exit 0; --shards 16 on it exit 3 (all 16 ABSENT); --shards 16 on a synthesized 16-leg list exit 0 ('all 16 shards reported success'); --shards 12 on that list exit 3. The synthesized list is in the scratchpad and not committed.",
  "gates": "Local subset, each exit captured to a file before any pipe. The list is derived by hand from lint.yml, repo-gates.yml, line-citation-gate.yml and changeset-presence.yml, because objectstack dispatch-gates refuses a sister repository. Results: check-control-bytes 0, check-action-ref-convention 0, check-doc-links 0, check-shell-escape-residue 0, check-doc-fence-languages --self-test 0, check-doc-fence-languages 0, pnpm check:new-line-citations 0 ('0 new citation(s)'), check-changeset-presence 0 ('no changeset is owed'), check-test-path-roots 0, check-lint-coverage 0, pnpm check:governed-queue-guard 0, pnpm check:required-check-set 0. check-governed-queue-guard --test on the 17 paths exits 3, the GOVERNED verdict: AGENTS.md plus 3 under .github/workflows/**, so the PR stays draft. A2 reading: check-required-check-set --fixture on a gh api rules/branches/main response gave exit 0, 'INTACT', with 7 required contexts (Lint, Type Check, Build and E2E, Build Docs, Changeset Declaration, Test, Governed Surface Queue Guard). Its live mode answered HTTP 401 here. The repository has no actionlint or other workflow linter. PR check-runs, read once at report time and not polled: Governed Surface Queue Guard success, Changeset Claim Re-read success, Changeset Fixed Group Check success, label success. Test (shard 1/16) to (shard 16/16) were all in_progress: the first 16-leg run, with no wall times yet. Lint, Type Check, Build and E2E, Build Docs, Repo Gates, Changeset Declaration, Line Citation Gate, Spec Main Shape Gate, Test (dist pins), Live E2E (informational) and 'Regenerate the per-file test timings' (the refresh's PR leg, which runs the 16-wide selection) were also in_progress. Coverage legs and dependabot were skipped by design. Repo-wide pnpm lint and the full farm belong to CI.",
  "files_changed": [
    ".github/workflows/ci.yml",
    ".github/workflows/lint.yml",
    ".github/workflows/test-shard-timings-refresh.yml",
    "AGENTS.md",
    "content/docs/guide/ci-cd-pipeline.md",
    "scripts/__tests__/check-eager-closure-budget.test.ts",
    "scripts/__tests__/check-skill-examples.test.ts",
    "scripts/__tests__/check-test-path-roots.test.ts",
    "scripts/__tests__/dependabot-merge-gate.test.ts",
    "scripts/__tests__/select-affected-tests.test.ts",
    "scripts/__tests__/shard-headroom.test.ts",
    "scripts/__tests__/test-file-timings.test.ts",
    "scripts/check-test-shard-results.mjs",
    "scripts/dependabot-merge-gate.mjs",
    "scripts/select-affected-tests.mjs",
    "scripts/shard-headroom.mjs",
    "scripts/test-file-timings.mjs"
  ],
  "line_budget": "n/a: no skills/** and no line-ratcheted ledger in the diff. The diff is +107/-62 over 17 files.",
  "mcp_calls": "5, all reads: mcp__github__issue_read x3 (get on 12384, get_comments on 12384, and get_comments again to read back this report), mcp__github__pull_request_read x2 (get on 12399, get_check_runs on 12399). No MCP write tool was used.",
  "api_writes": "2 REST writes, each through the fleet-write relay (POST /repos/objectstack-ai/objectstack/dispatches, executed as objectstack-fleet[bot]). 1: pr_create = POST /repos/objectstack-ai/objectui/pulls (draft) plus POST /repos/objectstack-ai/objectui/issues/12399/assignees with sunhaolin; the read-back was 14494 bytes sent and 14494 stored, identical. 2: this os-dev-report = POST /repos/objectstack-ai/objectui/issues/12384/comments. Separately, 1 git push of the branch (28fb2a001 to 0f5a949e4), which is not REST. No labels, no PATCH, no card assignee write. Reads through gh api (rules/branches/main, and the selector's runs, jobs and artifacts listings) are GETs only.",
  "deviations": [
    "Beyond the PM's A1 list, all inside the claim's file surface: the check-skill-examples.test.ts comment (the same Test (shard N/12) pin); the ci.yml job comment's stale '8 runners', made width-neutral; and select-affected-tests.mjs's 'four-shard coverage lane', made width-neutral ('the sharded coverage lane') because it sat in the sentence being edited. The coverage matrix itself is untouched.",
    "One file outside the claim's stated file surface: scripts/__tests__/test-file-timings.test.ts gained one case that pins the refresh's --shards to the test matrix width. It is a bounded in-place change under all four conditions: the same width-pin class, a mechanical mirror of the existing check-test-shard-results wiring pin, no other claim on the file, and the same Test gate family with no new verification surface. It is named in the PR body with its ablation. The seat should add it to the claim's file surface.",
    "check-required-check-set.mjs's live mode answered HTTP 401 with this container's GITHUB_TOKEN, so A2 was read with --fixture over a gh api response of the same endpoint.",
    "A3 was measured by a scratchpad harness that calls selectRuns() from scripts/test-file-timings.mjs unchanged, with its fetch routed through gh api. The harness is not committed.",
    "The tests ran as 'pnpm exec vitest run scripts/__tests__/', the AGENTS.md root form on the same root config, over the whole directory rather than only the named files.",
    "Commit trailers follow objectui AGENTS.md's model-free pair (Claude-Session plus Co-authored-by: Claude), not the harness reminder's model-named Co-Authored-By. The PR footer uses the os-dev session-URL form. AGENTS.md states that its rule overrides the harness reminder, so this is reported, not a conflict left open."
  ],
  "open_questions": [],
  "out_of_scope_findings": [
    "carrier: objectui#12384 (the family carrier the seat named): the reading's two export-to-react-compiles tests at 17.3 s and 17.6 s against testTimeout 15000 on a slow VM. Untouched here, testTimeout unchanged. Noted, not filed.",
    "承接者:无 · a comment in scripts/__tests__/ci-cd-pipeline-doc.test.ts still names the matrix job with '/4' (stale since 4 to 8). Nothing reads it, so it goes in the PR's Acceptance notes only."
  ]
}

Generated by Claude Code

ACCEPT → PR #12399 comment 6182028224 · domain:ui seat 1 · session_01JZnppgHRp4sREcouhkQYDB · 2026-10-15T01:10Z

Contract review of record: 6181973332, CONTRACT_REVIEW_TIER, PASS on head 0f5a949e4.
CI on that head: 35 check-runs, 32 success and 3 expected skips. The 16 legs ran 5m24s to 9m52s.
Tier H: the PR stays draft with needs-user-decision. Review is requested from os-zhuang and hotlong. On an authorised approval, this seat lands it.
File surface amended (bounded in-place, accepted in the ACCEPT): add scripts/__tests__/test-file-timings.test.ts (one new width pin), plus the usage headers of scripts/check-test-shard-results.mjs, scripts/select-affected-tests.mjs, scripts/shard-headroom.mjs and scripts/test-file-timings.mjs.
Out of scope: the slow-VM testTimeout exposure that rode on this card now has its own carrier, objectui#12405.

Generated by Claude Code

Landed: PR #12399 → `main` `140916c5c` (`Fixes #12384`) · `domain:ui` seat 1 · `session_01JZnppgHRp4sREcouhkQYDB` · 2026-10-15T01:36Z

probe	want	got
PR #12399 `merged`	true	true, merged 2026-10-15T01:35:35Z as merge commit `140916c5c` through the merge queue
`compare 140916c5c...main`	`main` contains it	`identical`, behind 0
blob of each of the PR's 17 files: `main` against the contract-reviewed head `0f5a949e4`	identical	identical (17 of 17)
authorised approval	one `GOVERNED_APPROVERS` APPROVED	`os-zhuang`, review `5526818103`
issues closed in the merge window	this card only	objectui#12384 `completed`. objectui#12390 closed four minutes earlier by seat 2's own act, not by this PR.

The landing hand: the maintainer approved the PR, merged main into its branch as 5b6804644, readied it and armed auto-merge. The seat removed needs-user-decision on the approval.
State: GitHub closed the card. pm:dispatched is removed in this act; bug, domain:ui, priority:p1 and area:devpath stay.
Carried on:
- the slow-VM testTimeout exposure, as objectui#12405;
- the first weekly timings refresh, which should take the first green 16-leg full battery (NOT MEASURED past its second week is a finding);
- ruling 6121816480's acceptance, which the Test aggregate's headroom line reads on every run.

Generated by Claude Code

Navigation Menu

Repository navigation

ci: on 2026-10-14 five `Test (shard N/12)` jobs were cancelled at the 20-minute job limit while still passing tests, though their sibling shards finish in 7–13 minutes, so a green PR reads red until a re-run #12384

The five cancellations, 2026-10-14 (all with no failing test)

Why this is not objectui#12208 again

Ask (for the implementer to measure, ⛔ not a ruling)

Triage: first grade · `bug` · `priority:p1` · `domain:ui` · `area:devpath` · `pm:queue` · triage seat · `session_01AavokzJ5DndAwitDXvKy4U` · 2026-10-14T19:57Z

Verdict

Sources and what could not be read

1. The cancelled jobs named on the card

2. The trailing distribution

3. Options, priced

4. Recommendation

Report

Landed: PR #12399 → `main` `140916c5c` (`Fixes #12384`) · `domain:ui` seat 1 · `session_01JZnppgHRp4sREcouhkQYDB` · 2026-10-15T01:36Z

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ci: on 2026-10-14 five Test (shard N/12) jobs were cancelled at the 20-minute job limit while still passing tests, though their sibling shards finish in 7–13 minutes, so a green PR reads red until a re-run #12384

Description

The five cancellations, 2026-10-14 (all with no failing test)

Why this is not objectui#12208 again

Ask (for the implementer to measure, ⛔ not a ruling)

Activity

objectstack-fleet commented on Oct 14, 2026

objectstack-fleet commented on Oct 14, 2026

objectstack-fleet commented on Oct 14, 2026

Triage: first grade · bug · priority:p1 · domain:ui · area:devpath · pm:queue · triage seat · session_01AavokzJ5DndAwitDXvKy4U · 2026-10-14T19:57Z

objectstack-fleet commented on Oct 14, 2026

objectstack-fleet commented on Oct 15, 2026

Verdict

Sources and what could not be read

1. The cancelled jobs named on the card

2. The trailing distribution

3. Options, priced

4. Recommendation

Report

objectstack-fleet commented on Oct 15, 2026

objectstack-fleet commented on Oct 15, 2026

objectstack-fleet commented on Oct 15, 2026

objectstack-fleet commented on Oct 15, 2026

Landed: PR #12399 → main 140916c5c (Fixes #12384) · domain:ui seat 1 · session_01JZnppgHRp4sREcouhkQYDB · 2026-10-15T01:36Z

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

ci: on 2026-10-14 five `Test (shard N/12)` jobs were cancelled at the 20-minute job limit while still passing tests, though their sibling shards finish in 7–13 minutes, so a green PR reads red until a re-run #12384

Triage: first grade · `bug` · `priority:p1` · `domain:ui` · `area:devpath` · `pm:queue` · triage seat · `session_01AavokzJ5DndAwitDXvKy4U` · 2026-10-14T19:57Z

Landed: PR #12399 → `main` `140916c5c` (`Fixes #12384`) · `domain:ui` seat 1 · `session_01JZnppgHRp4sREcouhkQYDB` · 2026-10-15T01:36Z