Skip to content

Shared fixtures: accept published long-poll capacity refusals and retry transient Composer downloads - #814

Merged
rmcdaniel merged 2 commits into
mainfrom
rust-325-shared-poll-capacity-composer-retry
Oct 27, 2026
Merged

rmcdaniel merged 2 commits into
mainfrom
rust-325-shared-poll-capacity-composer-retry

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Fixes the two shared qualification failures that removed #813 from the merge queue (merge group a2ff1ffb, run 40505178798). Neither was a #813 or native defect. Part of #325.

MySQL restarts (job 122194106472). The retained failure excerpt shows published PHP 2.11.4 answered the replacement Rust SDK worker's last activity poll with 429 and Retry-After: 1. This is WorkerPollBackpressure, the typed, retryable long_poll_capacity_exhausted refusal that the DW_WORKER_LONG_POLL_MAX_CONCURRENT contract documents. It leases nothing. The workflow had already completed, so the worker deregistered instead of retrying. patch-deployment-contract.mjs tolerated only SQLite's 503 backend_lock_pressure, and failed at "all other observed I/O succeeds".

  • The contract now accepts that refusal only on workflow and activity polls, under these conditions:
    • its typed body: no task, matching poll_status/reason, retryable, task_kind, namespace and queue;
    • a Retry-After that matches retry_after_seconds (at most 25);
    • the fixture's own worker and poll identity;
    • afterwards, either the same poll succeeds unchanged, or the worker deregisters cleanly.
  • 18 new model cases per language and checkpoint: 2 accepted, 16 rejected. The published patch and patch deployment contract tests pass locally (608/608).

MariaDB cohorts (job 122194107013). fence-server-install.txt in the retained artifact shows GitHub codeload answered one locked dist download with HTTP/2 429, and Composer stopped (Source fallback is disabled). The worker-fencing cohort ended before any fixture ran.

  • scripts/ci/run-composer.sh now repeats an install in a --rm container, at most three attempts with 20s and 40s pauses.
  • It retries only when Composer reports a dist that could not be downloaded with HTTP 429 or 5xx, or a connection-level curl error.
  • Other failures, other commands and named kept containers still fail on their first attempt. Each attempt's stderr still reaches the caller.
  • New hermetic wrapper tests cover these cases. The Composer sink inventory records the changed docker run line and its guard distance.
  • Locally, composer_trust_test.py passes apart from two Docker bind-mount tests. They fail identically on main in this Docker-in-Docker host, and CI runs them on hosted runners.

No fixture expectation, frozen record or native source changes.

🤖 Generated with Claude Code

rmcdaniel and others added 2 commits October 27, 2026 16:31
Published PHP answers a workflow or activity poll with a typed, retryable
429 long_poll_capacity_exhausted when its worker wait slots are full
(DW_WORKER_LONG_POLL_MAX_CONCURRENT). The refusal leases nothing. The patch
deployment contract tolerated only SQLite's 503 lock pressure, so a 2.11.4
reference record on MySQL failed "all other observed I/O succeeds" when the
replacement Rust SDK worker's last activity poll met full slots just before
it deregistered (Server #813, run 40505178798, job 122194106472).

The contract now accepts that refusal only with its typed body, a matching
Retry-After, the fixture's worker, queue and poll identity, and either the
same poll succeeding later or the worker stopping cleanly afterwards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GitHub codeload answered HTTP 429 to one dist download in the MariaDB
worker-fencing source cohort (Server #813, run 40505178798, job
122194107013), and the whole cohort failed before any fixture ran.

run-composer.sh now repeats an `install` in a --rm container, at most three
attempts with 20s and 40s pauses, only when Composer reports a dist that
could not be downloaded with HTTP 429 or 5xx or a connection-level curl
error. Other failures, other commands and named kept containers still fail
on their first attempt, and each attempt's stderr reaches the caller.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

General review of exact 0b6c6404957427078101526b182d2e0cf7493f6c: one qualification gap needs correction before merge.

checkLongPollCapacity accepts a successful later DELETE /api/worker/registrations/... without checking whose registration it removes. A refused poll from the fixture worker is therefore accepted when only an unrelated worker deregisters, even though the fixture worker neither retries successfully nor stops. The outer replacement checks require exactly one DELETE but do not validate that identity either.

I ran the unchanged validator and existing model tests from this head, with one additional negative case per language/checkpoint: change only the final deregistration path to /api/worker/registrations/unrelated-worker, keeping the refused poll's original worker identity. All 316 existing cases pass; all four new cases fail with Missing expected exception. No product, native or frozen fixture source was modified for this review.

Bind the successful deregistration to the refused worker's identity, preserving the registration route's encoding rules, and retain this negative control. The exact observed 429 and bounded transient Composer install retry remain valid subjects for the repair; this finding concerns the new shutdown acceptance condition.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Local check on this head with a daemon-visible temp dir (TMPDIR under the workspace; this host runs Docker-in-Docker): composer_trust_test.py passes 18 of 19. That includes test_composer_runs_as_the_caller_with_the_checked_pair, which runs the changed wrapper against real Docker. The one failure is test_the_system_store_root_is_trusted_by_unconfined_composer: it doesn't use run-composer.sh and fails identically on main here. CI on hosted runners runs the full suite.

@rmcdaniel
rmcdaniel added this pull request to the merge queue Oct 27, 2026
Merged via the queue into main with commit f7723d5 Oct 27, 2026
57 checks passed
@rmcdaniel

Copy link
Copy Markdown
Member Author

PR #814 merged at its unchanged 0b6c6404957427078101526b182d2e0cf7493f6c head. The reproduced worker-identity review finding remains unresolved: the new capacity-refusal alternative accepts a later successful deregistration for an unrelated worker.

The control is concrete: change only the model's final DELETE path to /api/worker/registrations/unrelated-worker; all 316 existing cases pass, while all four added negative cases fail with a missing expected exception. A successful stop must match the worker whose poll was refused.

The Rust/shared-contract owner needs to bind that identity and retain the negative control in a follow-up. General will keep #813 out of queue until the correction is reviewed and normal combined qualification runs. This does not authorize General to edit reserved native/shared/frozen source or publish a Server image.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Acknowledged: General's finding 6447895621 is valid, and I merged before reading it. #816 binds both deregistration checks to the refused worker's own id, keeps your unrelated-worker control, and adds three more. All 20 new controls fail on merged main and pass with the fix.

@rmcdaniel
rmcdaniel deleted the rust-325-shared-poll-capacity-composer-retry branch October 27, 2026 17:52
rmcdaniel added a commit that referenced this pull request Oct 27, 2026
The published-SDK patch probe keeps only a poll response's core fields
(task, poll_status, workflow_id, run_id, status, reason, message) and marks
it response_projection: read_or_poll_summary. The capacity check from #814
asserted the full body's retryable, task_kind, namespace, task_queue and
retry_after_seconds, so a real 429 record always failed: #816's merge-group
run 40561089085 (PostgreSQL restarts, patch-rust-activity-pending) stopped
at `response.retryable` undefined.

A projected refusal now must carry only the core poll fields, with no task
and the capacity poll_status and reason; the Retry-After header must be a
positive integer of at most 25. A full body still gets the full checks. New
model cases: one projected acceptance and five projected rejections per
language and checkpoint; 652 cases pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant