Skip to content

Retry the PHP reference's lock-pressure refusals in parity probes - #515

Merged
rmcdaniel merged 10 commits into
mainfrom
rust-325-probe-lock-pressure
Oct 20, 2026
Merged

rmcdaniel merged 10 commits into
mainfrom
rust-325-probe-lock-pressure

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Part of #325. Replaces the closed #511, whose wrapper transport hid SDK capabilities.

Shared runs keep failing on the published PHP SQLite reference's documented HTTP 503 backend_lock_pressure refusal. It shares its database file with its maintenance loop, and SQLite refuses a writer while another connection writes. Seen so far:

The refusal means nothing was committed, and worker mutations stay fenced by task and attempt, so repeating the request cannot record a second outcome. The probes treated it as fatal.

Change

  • Probe SDK clients. probe.php, the probe files it loads and cooperative-restart.php build their clients through parityClient() (lock-pressure-middleware.php). A caller's transport is kept. Otherwise the default is the SDK's own Psr18Transport over Guzzle with an innermost retry middleware, so the transport keeps every capability the SDK checks (bounded requests, payload transfer). The observed-transport stacks in probe.php, cooperative-restart.php and the namespace and patch-deployment probes add the same middleware innermost, so recording middleware sees only the final reply.
  • Retry rule. lockPressureOperationRetryDelay retries a non-poll request only on status 503, reason backend_lock_pressure and retryable: true. Polls keep lockPressurePollRetryDelay's stricter rule that task is null. Both stop at 15 retries, 30 s of waiting and the request's own timeout. A request body that cannot be replayed is never retried. Retries are reported once on stderr at exit, so observations stay comparable across PHP, native and embedded targets.
  • Rust SDK actor. The corrected-marker actor retries its deregistration on the same refusal under the same bounds (corrected-markers/rust/src/lock_pressure.rs).
  • probe-failure.php keeps its plain client and 1 s timeouts.
  • activity-deadline-probe.php keeps its plain client too. It retries its own lock-pressure polls and records each retry, and its contract checks those against a recording proxy outside the client.

Two fixes since the first push, both found by this PR's own runs:

  • Guzzle wraps a handler stack in reverse, so HandlerStack::unshift had put the retry outermost, and recorders logged each refused attempt. The patch deployment contract then rejected a refused activity poll. The retry is now pushed after each recorder, which makes it innermost.
  • Retrying below the SDK hid the activity deadline probe's poll retries from its own accounting, so that probe was reverted to its plain client.

Diagnosable Rust test failures

The parity jobs write cargo test output to evidence files, so a failed Rust test left only its exit code in the step log (the execution-and-codec failure on this PR's previous head showed nothing more). An ERR trap in the three test steps now runs rust-test-failures.sh, which prints the panics and failed test names from those files.

Checks

  • ServerParityLockPressureRetryTest adds operation cases: deferred worker refusals and control-plane refusals are retried, while missing or false retryable, another reason, another status and the budgets are not, and a poll through the operation entry point still needs task: null.
  • The actor crate's new unit tests cover the same rule.
  • A local mock check with the parity SDK installed as CI installs it refused two polls, then succeeded through parityClient(), with the SDK still recognizing the transport's bounded-request capability.
  • A local run of the execution-and-codec step's fixtures and SDK directions (the step that failed on Preserve explicit worker command declarations in native registration #508) passed with these changes. No refusal happened in that run, so it shows the transport change is transparent rather than exercising a retry.
  • rust-test-failures.sh run against a local failing suite output printed both panics and the closing failure list, and printed nothing for passing output.

🤖 Generated with Claude Code

durable-workflow-ops and others added 2 commits October 20, 2026 07:56
The published PHP SQLite reference shares its database file with its
maintenance loop and sometimes refuses a request with its documented HTTP 503
backend_lock_pressure reply: retryable, nothing committed, worker mutations
fenced by task and attempt. Probe runs failed on these refusals for polls,
worker operations and a Rust SDK actor's deregistration.

- Parity probes now build their SDK clients through parityClient(), whose
  default transport is the SDK's own Psr18Transport over Guzzle with an
  innermost retry middleware. The transport keeps every capability the SDK
  checks (bounded requests and payload transfer), unlike the earlier
  wrapper. Observed-transport stacks add the same middleware innermost, so
  recording middleware sees only the final reply.
- lockPressureOperationRetryDelay retries any non-poll request only on
  status 503, reason backend_lock_pressure and retryable true, within 15
  retries, 30 s of waiting and the request's own timeout. Polls keep the
  stricter rule that no task was handed out. Bodies that cannot be replayed
  are never retried. Retries are reported once on stderr.
- The Rust SDK actor retries its deregistration on the same refusal, under
  the same bounds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
durable-workflow-ops and others added 4 commits October 20, 2026 10:39
The parity jobs write Rust test output to evidence files, so a failed
cargo test left only its exit code in the step log. An ERR trap now prints
the panics and the failed test names from those files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Docker workflow isolation policy only accepts repository helpers in
statically known positions, as the container log tails already use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Guzzle wraps a handler stack in reverse, so HandlerStack::unshift put the
retry outermost and each recorder logged the refused attempt as well as
the retried reply. The patch deployment contract then rejected a
published PHP activity-poll refusal it did not expect. The retry is now
pushed after each recorder, which makes it innermost, so recorders see
only the final reply, as the middleware intended.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
durable-workflow-ops and others added 2 commits October 20, 2026 13:33
This probe retries lock-pressure poll refusals itself and records each
retry, while a recording proxy outside the client logs every refused poll.
Retrying below the SDK hid those retries from the probe, so the contract
found a refused poll without its recorded retry. The probe keeps its plain
client.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Parity run 39431491496 on 0e5843b49: execution-and-codec failed in step 18 (query deadlines and installed Worker authority). This PR doesn't change that step.

What failed: against native, the published Python SDK 2.5.15 client in late-handler-sync mode raised worker shutdown timed out while stopping the query poller thread. Its synchronous 6-second handler returned and was fenced with query_execution_budget_expired, after one claim, so this isn't the known boundary replay. The query thread then didn't exit within the client's 5-second shutdown budget.

Local reproduction: a loop of only this case (native, observer, Python client, --cpus=1 each) passed 40 of 40. Recorded as undiagnosed in the port log's known gaps. I'll rerun the failed job once the run completes.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Merged main (83f4d6b43, the docs-only #517) into this branch as a44c739a4 to meet the up-to-date requirement. The rerun of 39431491496 passed all 13 parity jobs on 0e5843b49, and main added nothing under the parity paths since, so I cancelled the duplicate parity run 39449468233. The SQLite library suite passes on the merge. Merging when PHPUnit and Sonar pass.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Main moved again with #525 (PHP workflow definition evolution: app code, PHP feature tests and docs, nothing under the parity paths). Merged as f94e33e7c and cancelled the duplicate parity run 39451098197 for the same reason as above. Merging when PHPUnit and Sonar pass on this head.

@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel merged commit 2d5fbcc into main Oct 20, 2026
9 of 14 checks passed
@rmcdaniel
rmcdaniel deleted the rust-325-probe-lock-pressure branch October 20, 2026 15:52
rmcdaniel pushed a commit that referenced this pull request Oct 20, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants