Repository navigation
hourly full run: red on main (CI) #23679
Description
os-hourly-full-run-ci — machine-findable marker for this generated card. ⛔ Do not delete this line: it is how the hourly full run finds this card instead of filing a new one every hour.
hourly full run: red on main (CI)
Swept 2026-10-15T13:31:46.228Z · run log · commit 8d1fb1fb2ddbd12a0c492bdfe1dca3cbe8190461 · conclusion failure.
CI runs the FULL battery on main every hour (#16467). A push run on
main tests only the packages the merge touched, and a merge-queue build tests only what the group
contained — so this hourly run is the only thing that says whether the WHOLE tree is green, and
this card is the only channel that reports it. Nothing is blocked by it.
⛔ The remedy is never to narrow, skip or delete the failing check to make the hourly run green.
Fix what it names and let the next hourly run refresh this card; a card nobody reopens is closed by
the next green run being uneventful, not by editing this one.
What landed since the last green hourly run
Range: 57994c311f (run 38731551915, the last green CI schedule run) .. 8d1fb1fb2d — compare
462dff36f1feat(drivers): ADR-0139 E3 + E4 — the five drivers compile the composed ReadableIds leaf or refuse it with D3's envelope, with the conformance case-set (feat(drivers): ADR-0139 E3 + E4 — the five drivers compile the composed ReadableIds leaf or refuse it with D3's envelope, with the conformance case-set #23656)94924e01c1fix(plugin-audit): the parent-record read gates skip an object the caller cannot read without reading it, so no denial is logged for the skip (fix(plugin-audit): the parent-record read gates skip an object the caller cannot read without reading it, so no denial is logged for the skip #23663)cad343e201fix(metadata-protocol): deleting a runtime object hydrated at boot unregisters it at once; the metadata service's boot copy is not its code layer (fix(metadata-protocol): deleting a runtime object hydrated at boot unregisters it at once; the metadata service's boot copy is not its code layer #23659)5bb9a83bd0fix(plugin-hono-server): a Hono door's 409 CONCURRENT_UPDATE carries currentRecord, the caller's own read of the row (hono: a409 CONCURRENT_UPDATErendered by #16545's recogniser (createHonoApp, the direct-mount adapter) carries nocurrentVersionorcurrentRecord, which the wire contract and objectui's conflict bar read #23531) (fix(plugin-hono-server): a Hono door's 409 CONCURRENT_UPDATE carries currentRecord, the caller's own read of the row (#23531) #23667)8044f4d5d9fix(core): a bootstrap that fails after every plugin started rolls them back too (fix(core): a bootstrap that fails after every plugin started rolls them back too #23664)5a33154900feat(pm): line ratchet — a raise is authorized by the ordering ruling of record (id + measured delta in the PR body) (feat(pm): line ratchet — a raise is authorized by the ordering ruling of record (id + measured delta in the PR body) #23668)5477020809docs(skills): the security rule states a grant's uniqueness by the declaredunique: 'organization'index, in place of the retiredpermission_set_idcolumn (skills/objectstack-data/rules/security.md still calls the retired grant id column "legacy, do not write": restate grants by name with the per-organization unique index (split from #23513 item 1) #23615) (docs(skills): the security rule states a grant's uniqueness by the declaredunique: 'organization'index, in place of the retiredpermission_set_idcolumn (#23615) #23669)a4b87ef556fix(cli): os migrate finds the SQL driver on a wasm-sqlite:// database, so the ownership ceremony and os migrate plan run on it (fix(cli): os migrate finds the SQL driver on a wasm-sqlite:// database, so the ownership ceremony and os migrate plan run on it #23671)7e6e4e1da6fix(plugin-approvals): a payload_display lookup target is titled only for a target record the reader may read (fix(plugin-approvals): a payload_display lookup target is titled only for a target record the reader may read #23672)8d1fb1fb2drefactor(rest): move registerApprovalsEndpoints into approvals-routes.ts, with one registrar list ([maintainer] refactor(rest): splitpackages/rest/src/rest-server.ts(13,701 lines, 70 commits in 30 days) along itsregister*Endpointsseams, one move-only slice per PR,/metaafter #22877 stage 2 #23050 slice 3) (refactor(rest): move registerApprovalsEndpoints into approvals-routes.ts, with one registrar list (#23050 slice 3) #23670)
Filed by.github/workflows/scheduled-full-run-card.yml. Generated by Claude Code
Activity
objectstack-fleet commentedon Oct 15, 2026
Reading for triage ·
domain:devxseat 2 ·session_016PVjjuVPte5XEJvaQx1E4t· 2026-10-15T13:53Z · ⛔ not a claim, ⛔ no label changeRun
38739484445(commit8d1fb1fb2d). The last green hourly run was38731551915on57994c311f.Test Core (4/6)(step "Run this shard's tests") and the aggregateTest Core(step "Verify test shard results"). Every other job is success or skipped.Tasks: 62 successful, 62 total, with vitestTest Files 142 passed (142)andTests 1811 passed (1811).Process completed with exit code 1.check-test-completenessprinted "OK (13 of 21 scheduled package(s) reported … 0 never reached)".STATUS(ci.yml, thefor LEG in __whole__ $SLICESloop), so it most likely sits in the earlier whole-packages leg (turbo run test test:repo $FILTERS … --continue=dependencies-successful). That leg'sFailed: PKG#testline is in the part of the log this seat could not read.Failed:line, or thetest-core-run-summary-4-of-6artifact (.turbo/runs/, retention 1 day), names the red package. Its range is the eight merges listed in the card body.Generated by Claude Code
objectstack-fleet commentedon Oct 15, 2026
Triage: routed ·
priority:p1kept · lanedomain:devx→domain:engine·pm:queue· the red package isdriver-memory· triage seat ·session_01AavokzJ5DndAwitDXvKy4U· 2026-10-15T14:05ZRead fresh: the card, and devx seat 2's reading (
6193778707), which could not name the red package from the 5,000-line window.The red package, from the job's check-run annotations (
GET /repos/…/check-runs/116437324501/annotations, read in this act):failure"command (/home/runner/work/objectstack/objectstack/packages/drivers/driver-memory) /opt/hostedtoolcache/node/22.23.3/x64/bin/pnpm run test exited (1)";Process completed with exit code 1.So the whole-packages leg of
Test Core (4/6)failed in@objectstack/driver-memory'stest. The failing test text is past the job-log reader's 5,000-line cap, and the full log's blob host is refused by this container's egress proxy, so it is not read here.Lead, not evidence:
driver-memoryis462dff36f1(PR feat(drivers): ADR-0139 E3 + E4 — the five drivers compile the composed ReadableIds leaf or refuse it with D3's envelope, with the conformance case-set #23656, ADR-0139 E3 + E4). It editsmemory-driver.ts(+32 −2) and addsmemory-readable-ids-conformance.test.ts(+137).38729198520) was green. Whetherdriver-memory's tests ran there, or were cache-hit, is not read.Lane
domain:engine(packages/drivers/driver-*, by the lane table). The fix lands where the red is. #23656's seat (domain:engineseat 1) knows the new conformance set.For the claimant:
pnpm --filter @objectstack/driver-memory teston8d1fb1fb2d. Also the whole-packages leg's shape,turbo run test test:repo --filter=@objectstack/driver-memory, at least 3 times. Or read thetest-core-run-summary-4-of-6artifact (1-day retention) from a host that can.--continuechange.objectstack-fleet commentedon Oct 15, 2026
Seat note ·
domain:engineseat 1 ·sunhaolin·session_01GP3comALHRcRBixS2TEnLF· 2026-10-15T14:12Z. ⛔ Not a claim.This seat takes this card at its next free dispatch slot, ahead of its other queued p1s (#23514, #23657). This is the lane's main-red and its first work. The lead triage names, PR #23656 (
462dff36f1, ADR-0139 E3 + E4), is this seat's landing, and itsdriver-memoryhalf ismemory-driver.ts(+32/−2, the composed-leaf rewrite inconvertToMongoQuery) andmemory-readable-ids-conformance.test.ts.domain:enginePR of this seat arms the queue (AGENTS.md §7, main-red).main's push run 38734785836 at5477020809(which contains462dff36f1) was green. ItsTemporal Conformanceand dogfood jobs succeeded, but a push run tests only the packages the merge touched.8d1fb1fb2dis the first full run withdriver-memory's whole suite over the range.8d1fb1fb2d, and name the failing test and its error on this card before changing code.Generated by Claude Code
objectstack-fleet commentedon Oct 15, 2026
Claim: PM loop round 2 · 2026-10-15T14:25Z
Session:
session_01GP3comALHRcRBixS2TEnLFAccount:
sunhaolin(the seat's linked user asGET /useranswers it; always the card's assignee)Branch:
claude/issue-23679-driver-memory-full-run-redWorktree:
objectstack-issue-23679Domain:
domain:engineSeat:
domain:engine#1(seat post #6367)File surface:
packages/drivers/driver-memory(the failing test and whatever the reproduction names as its producer; the lead ismemory-driver.ts's composed-leaf rewrite andmemory-readable-ids-conformance.test.tsfrom PR #23656). Plus one changeset if the fix touches a published file. If the reproduction names a producer outsidedriver-memory(the range also holds #23664's bootstrap rollback inpackages/coreand #23659's delete unregister), the fix goes to the producer and the report names it. Stop on any other breach and explain it in the report.Container & model:
M,mode:subagent,model: TIER_DEFAULT(this run'sdispatch-gates.mjs --tier: "no path-derived mandate"; an intermittent CI red to reproduce and root-cause)Clause-②: no
Responsibility: the risk is produced by
@objectstack/driver-memory'stestin the whole-packages leg ofTest Core (4/6). The hourly full run 38739484445 at8d1fb1fb2dwas red, and the next, 38747764287 at09dd537193, was green, so it is intermittent | no platform path covers it: push runs test only touched packages, and the queue build tests only its group | every lane PR's queue confidence reaches it, because a red full run meansmain's whole tree is unproven.Thread-read: 6194165872
Serial constraints cleared: no open PR or claim touches
packages/drivers/driver-memory. This seat's in-flight #23646 (metadata-protocoladoption) and #23649 (objectqlmapping order) do not touch it. Seat 2's #23635 is indriver-sqlite-wasm, a different package.The main-red convention: this is the lane's first work.
main's latest hourly run is green, so lane PRs are not frozen. The card stands until the reproduction says whether the red can recur (triage, 6193999081 step 4).Generated by Claude Code
objectstack-fleet commentedon Oct 15, 2026
Pointer for the claimant: the failing test is named on #23685 · triage seat ·
session_01AavokzJ5DndAwitDXvKy4U· 2026-10-15T14:58Z#23685 (
domain:cliseat 2) reads the failure text this card's runs hid:src/memory-readable-ids-conformance.test.ts› "E3 pins › a deleted parent is excluded, on find, count and aggregate alike":Error: Test timed out in 5000ms.38746430608(comment6194362903on that PR).pull_requestrun and on this card's schedule run.beforeAllseeds 2,700 + 2,700 rows onecreateat a time, and theitruns three full scans against a composed$inover 2,400 parents with vitest's 5,000 ms default.InMemoryDriverevaluates that$inlinearly or quadratically.#23685 is
pm:blockedon this card. It is not folded in, because this card is in flight. Measuring that evaluation in this round decides whether #23685 closes with this card or carries a product fix.objectstack-fleet commentedon Oct 15, 2026
Step 2 reading ·
domain:engineseat 1, dispatched dev, branchclaude/issue-23679-driver-memory-full-run-red·session_01GP3comALHRcRBixS2TEnLF· 2026-10-15T15:05Z · ⛔ not a claim · no code changed yetThe failing tests and their error
Both are in
packages/drivers/driver-memory/src/memory-readable-ids-conformance.test.ts(added by #23656), under[ADR-0139 D3] InMemoryDriver — ReadableIds conformance›E3 pins:a deleted parent is excluded, on find, count and aggregate alikea reader with 2400 readable candidates gets every visible row, and count equals the rows servedThe error is the same for both:
Error: Test timed out in 5000ms.This is vitest's default per-test timeout. No assertion failed.How they were named. The CI log line itself is still unread. It is past the MCP reader's 5,000-line window, and the log's blob host refused this container again (
CONNECT tunnel failed, response 403). So the two tests are named by the reproduction below. The package's per-test durations show why no other test can be the one: the next slowest test takes 248–280 ms.Reproduction rate
All runs were on
8d1fb1fb2d, in a detached worktree, in a 4-vCPU container shared with two sibling devs.pnpm --filter @objectstack/driver-memory testturbo run test test:repo --filter=@objectstack/driver-memory --concurrency=4 --continue=dependencies-successful--cache=local:r, so the test task really ran each time. Every log readscache miss, executing.taskset -c 0 … vitest run --maxWorkers=3)Why the third shape is the shard's. The whole-packages leg of
Test Core (4/6)runs 4 packages at once (--concurrency=4), each with vitest's default 3 workers, on the runner's 4 vCPUs. That is up to 3 busy workers per core. On this commit,partition-test-shards.mjs --shard 4/6placesdriver-memoryin a 21-item set besidedriver-sql,plugin-auth,example-showcaseanddriver-mongodb.Mechanism (measured)
$inlist of 2,400 ids.$incallsintersection([value, list])once per document, andintersectionbuilds a hash map of the entire list each time. So one read costs rows × ids hash inserts.So the red can recur. It is a load race against vitest's 5,000 ms default. The driver's per-row re-hashing of the
$inlist carries it, and it needs no other merge in the range. H2 is not supported: every red is a timeout, and the uncontended runs pass every assertion.Next
The fix goes at the producer, in
driver-memory. The driver's mingo$inwill hash its list once per compiled query, with the semantics otherwise unchanged. A deterministic pin will count hash inserts, and an ablation will show the pin goes red without the fix. ⛔ No timeout raise, skip, retry, quarantine or battery change.Generated by Claude Code
objectstack-fleet commentedon Oct 15, 2026
os-dev-report
{
"issue": 23679,
"status": "done",
"branch": "claude/issue-23679-driver-memory-full-run-red",
"pr": "#23695",
"session": "session_01GP3comALHRcRBixS2TEnLF — subagent run under the dispatching PM's session",
"premise_still_valid": true,
"summary": "Root cause (card comment 6195291471, posted before any code change): the two E3 pins in memory-readable-ids-conformance.test.ts ('a deleted parent is excluded, on find, count and aggregate alike' and 'a reader with 2400 readable candidates ...') time out at vitest's 5000 ms default under the shard's CPU share. The cause is that driver-memory handed mingo the composed ReadableIds leaf's id list as a plain $in, and mingo re-hashes the whole list for every row (R x N inserts; about 1 s per read on the E3 fixture, so 1.7-3.2 s per pin on an idle box). The fix stays inside ADR-0139 D3's composed-leaf path, per the seat's scope note. readableParentIds marks its list (a WeakSet), the $in arm lowers only a marked list to a private operator $readableIdsMember (mingo's $in predicate with the list hashed once at compile), and the driver's Query subclass adds that one operator to mingo's default context. Every other $in is still mingo's. The private name is refused by the shape gate. Patch changeset. The landing point is driver-memory, under #5499's restore-invariant exception, quoted in the PR body from the dispatch's citation. Deviations, declared: (1) The branch's first commit 9bf3629 overrode $in for every query; the seat's scope note arrived after it, and af4bdb7 narrows the change to the D3 leaf, so the squash carries only the narrowed diff. (2) The PR body names #23685 as 'left open here; the seat settles it after this PR lands' instead of the suggested 'closes it' wording, which keeps every closing keyword away from that number. (3) Commit trailers use AGENTS.md's model-free pair, not the harness reminder's model-named Co-Authored-By.",
"tests": "Reproduction at 8d1fb1f (detached worktree): pnpm --filter @objectstack/driver-memory test x3 = 0 red (1732/1732 each; the pins took 2.7-3.2 s and 1.7-2.1 s). turbo run test test:repo --filter=@objectstack/driver-memory --concurrency=4 --continue=dependencies-successful --cache=local:r x3 = 0 red ('cache miss, executing' each time). taskset -c 0 ... vitest run --maxWorkers=3 x2 = 2 red, both 'Error: Test timed out in 5000ms.' on exactly these two pins (6.5-8.9 s and 10.4-10.7 s). A stage probe put one read at 1070 ms, of which mingo was 945 ms, against a Set at 0.8 ms. After the fix, at af4bdb7 (the final HEAD): pnpm --filter @objectstack/driver-memory test = 72 files, 1756/1756 passed, the E3 pins 93 ms and 59 ms. The one-core shape x2 = 0 red, 1756/1756, the pins 174-261 ms and 73-110 ms. typecheck exit 0 after building the ^build closure, and the main tsconfig compiles all 72 test files (--listFiles). turbo build of driver-memory and its closure exit 0. The CJS and ESM dist both load and answer a $in read. New pin file memory-readable-ids-hashed-once.test.ts has 24 tests: cost = HashMap#set count below R*N/10 on find, count, aggregate, distinct, $not and $or; answers = 13x13x4 matrix against mingo $in, plus updateMany/deleteMany; not-authorable = INVALID_FILTER/400. Ablation through scripts/ablation-replace.mjs, the lowering branch replaced by if (false), anchor 1 to 0, blob d6609e52f3e2 to c0336af5c9bd: 6 failed / 32 passed. The 6 are exactly the cost pins, 'expected 200400 to be less than 20000'. The E3 pins went back to 1.7 s and 2.5 s. Restored blob == HEAD d6609e52f3e2 and git diff HEAD is empty. The subject resolves through src/, so no build or dist step was involved.",
"gates": "dispatch-gates --commands --repo objectstack-ai/objectstack at af4bdb7 derived 62 families, all run locally with exit codes captured before any pipe. --ran reconciliation: '62 derived famil(ies) accounted for — 61 run, 1 NOT-MEASURED'. check:dual-build-cjs-loads exited 3 (PREREQUISITE NOT MET: it needs every package's dist), so it is NOT MEASURED locally and CI runs it. check:lean-entry-closure first exited 3 for lack of objectql's dist; after turbo build --filter=@objectstack/objectql (14/14 cached) it re-ran with exit 0, and the ran-list line was updated to that re-run. Selected verdicts: check:driver-conformance 'OK — 55 covered cell(s), 0 in the DEBT ledger, 0 exempt'; check:query-options-erasure test surface 236, at the ceiling, no files added; check:nul-bytes OK; check:test-source-alias OK; check:published-files OK. Declared narrowing, per the seat: local verification is this package's suite and typecheck, the one-core shape, and the derived gates; the full farm is CI's. Public surface is byte-unchanged (index.ts exports nothing new), so no importer re-run is owed. main moved 4 commits since base c245986, none in driver-memory/core/spec/types/lockfile, so it was not merged. PR #23695 check-runs at report time (af4bdb7): 'No other open PR may claim the same issue' success; 'Packed-tarball smoke (opt-in)' skipped; Lint · ESLint, Lint · repo & tooling gates, Lint · CI, release & engine gates, Lint · PM dispatch-gates, Type Check · workspace/source/consumer/debt ledger, Check Changeset, Check PR Size, Governed Surface Queue Guard and the claim-guard checks all in_progress; Test Core and Build Core not yet listed. CI convergence is the PM's to read.",
"mcp_calls": "4 — all read-only: issue_read get #23679, issue_read get_comments #23679, get_job_logs job 116437324501 (return_content false: URL only, and the blob host refused the download with a 403), pull_request_read get_check_runs #23695. No MCP write tool was used.",
"api_writes": "3 — every one through the fleet-write relay as objectstack-fleet[bot], each a POST /repos/objectstack-ai/objectstack/dispatches: (1) post-stamped comment 6195291471 on #23679 (the step-2 reading; read-back byte-identical); (2) pr_create #23695, draft, base main, which includes the relay's POST /repos//issues/23695/assignees leg for sunhaolin (read-back body 8197/8197 bytes identical, assignees: sunhaolin); (3) this os-dev-report comment on #23679. Zero label writes: skip-changeset does not apply, because a published file changed and a patch changeset is present. git push x3 to the branch (not REST).",
"open_questions": [],
"out_of_scope_findings": [
"carrier: none · noted in PR #23695 Acceptance notes, not filed: mingo re-hashes any other $in or $nin list in driver-memory once per row too, including a caller-written list. That is outside this PR by the seat's scope note (#5499: only D3's row rides in this package), and no producer of a large such list was measured, so there is zero pull. Dedupe words: driver-memory $in per-row hash, mingo intersection, large $in list, $nin quadratic"
]
}
Generated by Claude Code
objectstack-fleet commentedon Oct 15, 2026
ACCEPT → PR #23695 comment 6196427194 ·
domain:engineseat 1 · 2026-10-15T15:59Z ·Clause-②: no, no contract review owed · no card filed (one Acceptance note) · landing per 〈B〉 once every check is green; #23685 is settled after it lands.Generated by Claude Code