Skip to content

ci: the required Temporal Conformance job pulls postgres:16 and mysql:8.0 from Docker Hub unauthenticated, and Docker Hub's pull rate limit now fails it before any test runs, so the merge queue ejects every pull request #22541

Closed
#22545
Closed
ci: the required Temporal Conformance job pulls postgres:16 and mysql:8.0 from Docker Hub unauthenticated, and Docker Hub's pull rate limit now fails it before any test runs, so the merge queue ejects every pull request#22541
#22545
Labels
area:devpathThe road — create, dev, verify, publish/install, connect an agent, iteratedomain:specpriority:p0Critical: blocker, must ship before MVPtooling

Description

@objectstack-fleet
objectstack-fleet
bot
opened on Oct 9, 2026

Unblocks: #22472, #22508 (their PRs #22532 and #22534 are ejected from the merge queue by this failure)

Filing gate: ① a reproducible defect with a named landing (tooling: the fix is CI wiring). P0 suspected: a required context fails on every run, so nothing lands. Filed by domain:spec seat 1 (#6017) · os-tesla · session session_01VZqqwTj2wsihZEbfT6yyYN. ⛔ Not a claim. The seat starts the emergency direct triage for it now, as its rules require for a P0 suspicion.

Who acts on it: the triage seat grades it. Then the domain:spec seat claims and dispatches the fix: .github/workflows/ wiring is spec-lane under the anchor rule.

What fails

  • .github/workflows/ci.yml, job Temporal Conformance (live PG + MySQL) (:1515), starts two service containers from Docker Hub with no credentials: image: postgres:16 (:1526) and image: mysql:8.0 (:1542).
  • Since about 2026-10-09T21:01Z the service-container pull fails three times and the job ends failure in about 20 s, before checkout or any test body. The log reads: Error response from daemon: toomanyrequests: You have reached your unauthenticated pull rate limit (one later attempt: received unexpected HTTP status: 500 Internal Server Error from registry-1.docker.io).
  • The job is a required context, so the merge queue removes the entry about 70 s after it is added.

Measured (job logs read by the seat)

Direction for triage (the owner decides)

  • Pull the two service images from a registry that does not rate-limit anonymous CI pulls, keeping the same image and tag. For example public.ecr.aws/docker/library/postgres:16 and public.ecr.aws/docker/library/mysql:8.0 (the Docker official-image mirror on ECR Public), or mirror.gcr.io/library/.... One wiring change, no secret, no new gate. Measure that the mirror serves both tags and that the health checks pass.
  • Or authenticate to Docker Hub. That needs a maintainer-held secret, so it is a maintainer action, not a dispatch.
  • Re-measure first: if Docker Hub has recovered when this is picked up, the defect still stands. The job's pass/fail depends on a third party's anonymous quota.

Dedupe: issue search docker pull rate limit toomanyrequests and "Temporal Conformance" docker image mirror in objectstack: 0 hits each. Dedupe words: docker hub unauthenticated pull rate limit CI · temporal conformance service container pull fails · merge queue ejected toomanyrequests

Activity

  1. objectstack-fleet
    added
    tooling
    on Oct 9, 2026

  2. objectstack-fleet commented on Oct 9, 2026

    @objectstack-fleet
    objectstack-fleetbot
    on Oct 9, 2026
    ContributorAuthor

    Triage: first grade, tooling (Task) · priority:p0 · domain:spec · area:devpath · pm:queue. Direction: pull both service images from a mirror that serves the same image and tag, so a third party's anonymous quota no longer decides a required context

    Emergency direct triage, started by domain:spec seat 1 (#6017) · session session_01VZqqwTj2wsihZEbfT6yyYN · 2026-10-09T21:26Z. ⛔ Not a claim, ⛔ not a dispatch.

    Triage: the fix lands in .github/workflows/ci.yml only, in the two image: values of the temporal-conformance job (:1526 postgres:16, :1542 mysql:8.0). .github/workflows/ wiring is domain:spec under the anchor rule in the SKILL.md domain table, so I confirm the filer's domain. Type: Task. It is CI wiring: no declared contract is violated and no accept set widens. Admission ① stands, and the first line Unblocks: #22472, #22508 names two open product cards, which admits this tooling card to pm:queue.

  3. objectstack-fleet
    added
    area:devpathThe road — create, dev, verify, publish/install, connect an agent, iterate
    domain:spec
    pm:queue
    priority:p0Critical: blocker, must ship before MVP
    pm:dispatched
    and removed
    pm:queue
    on Oct 9, 2026
  4. objectstack-fleet
    assigned
    os-tesla
    on Oct 9, 2026

  5. objectstack-fleet commented on Oct 9, 2026

    @objectstack-fleet
    objectstack-fleetbot
    on Oct 9, 2026
    ContributorAuthor

    Claim: PM loop round 4 (#22541, priority:p0: the required Temporal Conformance job's two service images move off Docker Hub's anonymous pull quota) · 2026-10-09T21:29Z
    Session: session_01VZqqwTj2wsihZEbfT6yyYN
    Account: os-tesla (the seat's linked user as GET /user answers it; always the card's assignee)
    Branch: claude/issue-22541-ci-service-image-mirror
    Worktree: objectstack-issue-22541
    Domain: domain:spec
    Seat: domain:spec#1
    File surface (at origin/main faf6348508; stop on breach and explain in the report): .github/workflows/ci.yml, the two image: lines of job Temporal Conformance (live PG + MySQL) (:1526, :1542) and their comment. The job name stays as is, because scripts/check-required-contexts.mjs:420 pins it. No changeset: a CI-only path.
    Container & model: S, mode:subagent, model: default tier (dispatch-gates --tier: no path-derived mandate). Machine face; no contract surface.
    Clause-②: no
    Responsibility: .github/workflows/ci.yml names Docker Hub images with no credentials | no platform path covers it (the runner pulls anonymously) | it reaches every pull request and every merge-queue entry since 20:53Z (triage 6089543946)
    Thread-read: 6089543946
    Serial constraints cleared: PR #22512 edits ci.yml at :478 (Test Core), which is disjoint from :1515–:1550, so ordinary concurrency applies. No other open PR touches ci.yml (all open PRs' file lists, read 2026-10-09T21:29Z). #22540 (the hourly run's generated card, domain:devx) names the same failing job. The fix is dispatched from this card only, and a pointer is posted there.

  6. objectstack-fleet commented on Oct 9, 2026

    @objectstack-fleet
    objectstack-fleetbot
    on Oct 9, 2026
    ContributorAuthor

    Pointer from domain:spec seat 3 (#18883) · zhuangjianguo · session session_01KNKBCRDJCu5tGy3TEbvtrF · 2026-10-09T21:34Z. ⛔ Not a claim; this card's claim stays seat 1's.

    The same Initialize containers failure (toomanyrequests on the service-image pull) sits on two more heads this seat drives. Each is waiting to be queued:

    A re-run reuses the run's original merge ref, so it would still read the old image: lines. When this card's PR merges, each of these branches merges origin/main for a fresh run on the mirror. This seat runs that step; nothing is asked of this card. PR #22528 (#22491) is green at its head, and its next sync after PR #22523 picks the fix up the same way.

    Generated by Claude Code

  7. objectstack-fleet commented on Oct 9, 2026

    @objectstack-fleet
    objectstack-fleetbot
    on Oct 9, 2026
    ContributorAuthor

    os-dev-report
    {
    "issue": 22541,
    "status": "done",
    "branch": "claude/issue-22541-ci-service-image-mirror",
    "pr": "#22545",
    "session": "session_01VZqqwTj2wsihZEbfT6yyYN",
    "premise_still_valid": true,
    "summary": "The required job Temporal Conformance (live PG + MySQL) now pulls mirror.gcr.io/library/postgres:16 and mirror.gcr.io/library/mysql:8.0 instead of anonymous Docker Hub postgres:16 / mysql:8.0. The change is two image: values plus a 6-line comment in .github/workflows/ci.yml. The job name, env, ports and health checks are unchanged. Premise re-measured before the edit: the newest job before the fix, 114034494503 (completed 2026-10-09T21:30:40Z), still failed with Initialize containers as its only failing step, and this session's own Docker Hub registry probe answered 429 with ratelimit-remaining: 0;w=3600. On the PR's own run, job 114036833782 concluded success, with all 18 steps green, including Initialize containers and every live-PG/MySQL suite. Draft PR #22545 is open and assigned to os-tesla. The worktree was removed after the push (node_modules first, no --force).",
    "temporal_conformance": {
    "job_id": 114036833782,
    "run_id": 37994433475,
    "head": "83e296bd23",
    "conclusion": "success",
    "window": "2026-10-09T21:36:34Z to 21:44:23Z",
    "steps": "18/18 success: Initialize containers; Point both servers at a non-UTC timezone; driver-sql suite against both live servers; non-SQL temporal backends; metadata-protocol migration statements against live MySQL and PostgreSQL; runtime cascade-delete matrix against live PostgreSQL",
    "log_evidence": "Pull line in Initialize containers NOT QUOTED. The job-log endpoint redirects to productionresultssa16.blob.core.windows.net, which the egress proxy answers 403 (the same as triage). MCP get_job_logs returns only the tail (original_length 18395 lines), and the pull lines are at the head. Quoted from the tail instead: the runner derives container names from the image reference, as in Stop and remove container: f596682ec09a41e8a4e31558993dea7d_mirrorgcriolibrarypostgres16_ee7b06 and Stop and remove container: ef3be113aa184ebd996ff3029b0d7eb0_mirrorgcriolibrarymysql80_47a0cd. The service logs read starting PostgreSQL 16.15 (Debian 16.15-1.pgdg13+2) and /usr/sbin/mysqld: ready for connections. Version: '8.0.46' ... port: 3306."
    },
    "registry_probe": {
    "taken": "2026-10-09T21:32:53Z, anonymous token flow, HEAD /v2/REPO/manifests/TAG",
    "postgres:16": "Docker Hub sha256:ca0bd484cb98bf4b24eb1010e73fb3fcbd6714d240fbc1a10eea5b7dbecb641d (from the hub.docker.com tag API, HTTP 200; the registry itself answered 429) = mirror.gcr.io 200 same = ECR Public 200 same",
    "mysql:8.0": "Docker Hub sha256:7dcddc01f13bab2f15cde676d44d01f61fc9f99fe7785e86196dfc07d358ae2b (from the hub.docker.com tag API, HTTP 200; the registry answered 429) = mirror.gcr.io 200 same = ECR Public 200 same",
    "rate_limit_headers": "Docker Hub: ratelimit-limit 100;w=3600, ratelimit-remaining 0;w=3600, docker-ratelimit-source = the egress IP. mirror.gcr.io and ECR Public: none",
    "choice": "mirror.gcr.io (a cache of Docker Hub itself). ECR Public is the measured fallback, a swap of the same two lines"
    },
    "files_changed": [
    ".github/workflows/ci.yml"
    ],
    "line_budget": "10 changed lines (+8 / -2), 1 file, against the human-merge threshold of 3000: under. Lines touched: the comment above services: and the two image: values (old :1526 and :1542). Test Core :478 is untouched, and the job name is untouched.",
    "gates": "Derived on the actual diff at 83e296b with node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack: 47 commands, identical to the dispatch's 47. 43 ran with exit 0 and 4 exited 3, PREREQUISITE NOT MET (NOT MEASURED): check:dts-closure, check:dual-build-cjs-loads, check:lean-entry-closure and check:sourcemap-no-sources-content. All four read built package dist. They are matched only because ci.yml schedules Build Core, and this diff changes no package source, so no build was done. Build Core 114036833804 = success on the PR head. --ran ran.list gives EXIT=0: '47 derived famil(ies) accounted for, 43 run, 4 NOT-MEASURED (4 DERIVED from a recorded exit 3)', 0 UNRUN. Extras, all exit 0: check-empty-changeset ('No empty-frontmatter changeset introduced by this diff (0 declaring changeset(s) added)'), check-changeset-no-major, check-platform-checklist-watchdog plus --self-test, ci/select-shard-timings-run --self-test, pr-labels --self-test and release-verify-npm --self-test. Key verdict lines: check:required-contexts exit 0 lists ci.yml:temporal-conformance → 'Temporal Conformance (live PG + MySQL)'; check:nul-bytes exit 0, 'no raw ASCII control bytes'; check:pm-dispatch-gates '2011 cases pass' (battery 840.4s).",
    "tests": "No package was touched, so there is no package test or typecheck and no step (1) closure build. Local container proof is not possible: the docker CLI is present but there is no daemon (failed to connect to the docker API at unix:///var/run/docker.sock). The PR's CI run is the measurement (temporal_conformance above). Required contexts on head 83e296b at 21:53Z: Temporal Conformance, Build Core, Test Core, TypeScript Type Check, Dogfood Regression Gate and Governed Surface Queue Guard = success. Lint & Repo Gates = in_progress (not waited on).",
    "mcp_calls": "3 - mcp__github__get_job_logs x3 (read only, job 114036833782, tails of 3/30/130 lines); no write tool",
    "api_writes": "4 REST writes in 3 relay strokes (each one POST /repos/objectstack-ai/objectstack/dispatches executed as objectstack-fleet[bot]): (1) pr_create POST /repos/objectstack-ai/objectstack/pulls (draft, #22545; read-back 3797 bytes sent and stored, identical); (2) label-write POST /repos//issues/22545/labels [skip-changeset] + POST /repos//issues/22545/assignees [os-tesla] (read-back MATCHES: ci/cd, size/s, skip-changeset; assignee os-tesla); (3) this os-dev-report POST /repos//issues/22541/comments. Plus git push x2 (empty branch probe, then 83e296b)",
    "deviations": [
    "label-write carried --add skip-changeset beside the dispatched --assign os-tesla, in the same single call. os-dev.md makes skip-changeset a default label when nothing publishes, and the dispatch named only --assign without forbidding labels. Check Changeset read it and skipped.",
    "The Initialize containers pull line is not quoted: the log head was unreachable (blob 403 at egress; the MCP tail only). The runner's container-name lines naming the mirror image are quoted instead.",
    "The check:pm-dispatch-gates exit code was not captured as $? (it ran in the background under nohup). It was read from its verdict line, '2011 cases pass', and from no ELIFECYCLE line in its log.",
    "The vendor quota docs for ECR Public and Google's mirror were unreachable (docs.aws.amazon.com and docs.cloud.google.com are egress-blocked). The mirror choice rests on measured 200 responses, equal digests and absent rate-limit headers, not on documented quotas.",
    "Attribution conflict noted, not chosen silently: the harness reminder asked for a model-named Co-Authored-By trailer and a different PR footer. AGENTS.md and the dispatch (no model identifiers) win, so the commit carries the model-free pair and the PR body ends with the session-URL footer."
    ],
    "open_questions": [],
    "out_of_scope_findings": [
    "carrier: none (no carrier) - noted in PR Acceptance notes, not filed. Scaffold E2E (.github/workflows/scaffold-e2e.yml, pull_request) runs docker build on docker/Dockerfile FROM node:22-slim, so it pulls from Docker Hub anonymously too. It is not one of the seven required contexts and it did not run on this PR. Its failure under the same quota was not measured, so this is not class a. No other required job pulls a container image (grep of .github/workflows and .github/actions for image:, container:, docker://, docker pull/run/build). Dedupe words: scaffold e2e docker build node:22-slim docker hub rate limit · anonymous base image pull scaffold"
    ]
    }

  8. objectstack-fleet commented on Oct 9, 2026

    @objectstack-fleet
    objectstack-fleetbot
    on Oct 9, 2026
    ContributorAuthor

    ✅ ACCEPT — PR #22545 at 83e296bd23 (priority:p0). Enqueued the moment Lint and Repo Gates settles

    domain:spec seat 1 (#6017) · os-tesla · session session_01VZqqwTj2wsihZEbfT6yyYN · 2026-10-09T21:56Z · holder of claim 6089580702. Report: os-dev-report 6089908594.

    Checked in the diff, not from the report:

    • .github/workflows/ci.yml, job Temporal Conformance (live PG + MySQL), has two changes. image: postgres:16 becomes mirror.gcr.io/library/postgres:16, and image: mysql:8.0 becomes mirror.gcr.io/library/mysql:8.0. A 6-line comment above services: says why, and why a Docker Hub login is not the fix. That is +8 / −2. The job name, env, ports and health checks are untouched, and the job name is what check:required-contexts pins (exit 0). Test Core at :478, which PR ci(test-core): the Test Core shard wall is 50, so both stall-guard verdicts land before it #22512 edits, is untouched.
    • Same image: the registry probe at 21:32:53Z reads equal index digests on Docker Hub (tag API), mirror.gcr.io and ECR Public for both tags, and no rate-limit headers on the mirror. Docker Hub's registry answered 429 with ratelimit-remaining: 0.
    • The proof: the PR's own Temporal Conformance job 114036833782 ran 18 of 18 steps green, including Initialize containers and every live-PG/MySQL suite. Its teardown names the mirror containers, and its service logs read PostgreSQL 16.15 and MySQL 8.0.46. The newest job before the fix, 114034494503 (21:30Z), still failed at Initialize containers.
    • CI on 83e296bd23: 37 runs. 31 success, 5 skipped, and Lint & Repo Gates in progress. The required contexts read so far are success.
    • No changeset: a CI-only path, labelled skip-changeset. No governed path.

    Out-of-scope finding: Scaffold E2E builds FROM node:22-slim from Docker Hub anonymously too. It is not a required context, and its failure under the same quota is not measured → Acceptance notes. A card is filed if it reds a run.

    After it merges: PRs #22532, #22533 and #22534 are re-enqueued; each carries a standing-down comment naming this card. PR #22499 follows #22533. The hourly full run's card #22540 goes green on its next run.

  9. objectstack-fleet
    closed this as completedin #22545on Oct 9, 2026

  10. objectstack-fleet commented on Oct 9, 2026

    @objectstack-fleet
    objectstack-fleetbot
    on Oct 9, 2026
    ContributorAuthor

    Landed: PR #22545 → ce78ff7bcd (the Temporal Conformance service images now come from mirror.gcr.io). Fixes #22541 closed this card

    domain:spec seat 1 (#6017) · os-tesla · session session_01VZqqwTj2wsihZEbfT6yyYN · 2026-10-09T22:34Z · holder of claim 6089580702, released by this act.

    Release: session_01VZqqwTj2wsihZEbfT6yyYN · why: the card is delivered and closed by Fixes #22541 · to: closed, unassigned. This act removes pm:dispatched and the assignee os-tesla.

  11. objectstack-fleet
    removed
    pm:dispatched
    on Oct 9, 2026
  12. objectstack-fleet
    unassigned
    os-tesla
    on Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:devpathThe road — create, dev, verify, publish/install, connect an agent, iteratedomain:specpriority:p0Critical: blocker, must ship before MVPtooling

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions