Investigate first-call PHP service rate rejection during SQLite maintenance

General's PHP repair is published in Server 2.11.5 / Core 2.15.1, source 721b2831975cc898cc39d7528c5ba7e8441fe744, through merged #809 and successful normal release 40478372197. Both public registries use index sha256:f4f2ec64a6522ee18a31542a30554873dbaa2f5d3af3ccb52deb914591699c20.

Source qualification passed 42 actual two-process cases / 644 assertions across PostgreSQL 17, MySQL 8, MariaDB 10.11 and file-backed SQLite, plus 112 consumer tests / 1023 assertions. Canonical admission precedes quota; only an uncommitted writer is refunded; raced ownership/current policy are checked; uncertain committed calls carry durable identity and explicit retry safety.

Published acceptance: 10 fresh SQLite namespaces with the unchanged service-calls requests and concurrent one-second repair/schedule loop all returned first rated 200 / second 429, first held 200 / second 429, and repeated synchronous held 200. Same-owner replay retained identity and foreign replay was refused 403. The strict full fixture remains non-passing because its older expected responses omit the new metadata field and still expect a 86400-second wait maximum and foreign replay 200. Exact scope and contract deltas.

Next: Rust owner qualifies its updated reference/native contract without weakening first-success/second-throttle; General completes committed-outcome/recovery installed acceptance. Keep #796 open until those checks are complete. #794 / Workflow #947 and #931 retain distinct installed Nexus and authority-recovery requirements.

Original report and acceptance context:

The Rust owner handed this PHP reference failure to General so native implementation can continue independently.

In Server #795's first queue qualification run 40418397301, the retained php.failure-excerpt.txt reportedly marks service-calls (product-fail): rated[0]. The published PHP Server 2.11.1 / Core 2.10.2 reference returned HTTP 429 rate_limit_exceeded to the fixture's first logical execution of an operation limited to one request per minute, around 2026-10-26 23:26:30 UTC. The same fixture passed in #795's PR run and #793's group. The Rust owner requeued #795 at exact head c8b87bd16; General does not own that queue action or native source.

The report also notes SQLite database is locked messages in the maintenance loop after 23:28. The retained tail does not contain the original 23:26 admission. A cache counter being consumed before a failed/rolled-back database write is a hypothesis, not an established cause. The source repair and reproductions are recorded above and in the issue comments.

General's next action is an isolated published-2.11.1 SQLite reproduction with the one-second maintenance loop and the existing frozen tests/Fixtures/ServerParity/service-calls.json. Capture actual wire attempts, the first rate decision, audit rows and the maintenance/write outcome, using bounded fresh namespaces. Distinguish a duplicate/retried HTTP attempt from a first admission and inspect shared rate-counter ownership. Preserve the fixture's first-success/second-throttled contract; do not skip or relax its assertion.

This is a fixture/reference qualification failure with no confirmed customer impact. PHP fixes and qualification belong here; native frozen-fixture changes remain with the Rust owner. Server #794 / Workflow #947 retain the separate workflow-binding defects.

Security/reliability source triage separates two failure modes and broadens the current-release scope:

The first-call 429 has a concrete rate-reservation leak path. SharedServiceBoundaryPolicy::evaluate() increments the shared rate counter before ServiceBoundaryAuditRecorder::record() writes the durable row. If that write throws under SQLite lock pressure, ServiceCallBoundary calls release(), but the policy tracks/releases only concurrency keys; rate keys remain consumed until their TTL. A retry can therefore be the first recorded call while correctly observing an already-consumed rate window and returning 429. This is fail-closed denial/accounting drift, not an authorization bypass.
A retryable lock-pressure response can follow a committed call mutation. Admission/audit is an autocommitted write before DefaultServiceControlPlane::execute(). Handler transitions and the controller's final $admission->call->refresh() are later database operations. Any SQLite lock exception there is globally rendered as backend_lock_pressure, HTTP 503, retryable: true. The parity client's bounded middleware then replays the POST because that public response explicitly says to retry. Without an idempotency key, the controller creates another audit/call row. The Rust owner's second group observation—five handler_failed rows where the reviewed fixture has four—is consistent with a first handler-failure row committing before a later lock error caused the whole POST to be replayed.

The second observation proves duplicate durable call creation for the failing-handler case; it does not yet prove that an external handler effect ran twice. The same boundary is nevertheless replay-unsafe for valid unkeyed operations: activity/carrier bindings receive fresh handler identifiers on a fresh row, and workflow-backed handlers can mutate their target before a later service-call refresh fails. Every replay is authenticated and re-authorized, so this remains a duplicate-operation/integrity and availability risk rather than an authorization escape.

Server 2.11.3 remains affected: its ServiceCallBoundary and shared-rate logic are unchanged, its execute route still commits admission before execute/refresh, and SQLite still maps any later lock exception to a retryable control-plane 503. Workflow 2.14.2's added authority check does not make the HTTP mutation idempotent.

Fix/acceptance boundary:

Compensate an allowed rate reservation when the durable audit write fails; preserve normal rate consumption after the audit succeeds.
Do not advertise an unkeyed service-call mutation as safely retryable once any call/handler state may have committed. Either make the full mutation atomic through the response boundary, or return a reconcilable existing service_call_id / require a stable idempotency key and prove replay reuses it.
Add deterministic injected-lock tests after (a) counter reservation, (b) audit save, (c) handler transition, and (d) final refresh. A bounded retry must leave one rate debit, one call row, one handler identity, and at most one target effect.
Keep the fixture's first-success/second-throttled and four-handler-failure expectations unchanged. Do not hide this by disabling the maintenance loop or weakening comparison.

No incident/operator page or Rust source gate is warranted from current evidence. This belongs to General's released PHP correction in #796; Rust can continue independently.

The first isolated published-2.11.1 SQLite batch did not reproduce the rate failure: all 10 fresh namespaces returned first rated 200, second rated 429, and other-caller 200 using the unchanged service-call request fixture from c8b87bd16. The finite 1-second repair/schedule loop was active; one real 503 lock-pressure retry occurred on broken/execute, but none on the rated requests. This agrees with Native's isolated 12/12 result and does not explain or waive the whole-corpus failures.

Next is deterministic injection at reservation/audit/handler/final-refresh boundaries and then unchanged whole-corpus published acceptance. Preserve the distinction between failed-admission rollback (refund its rate/concurrency reservation) and release after a committed admission (keep its rate charge). A 503 after durable unkeyed mutation must not encourage a blind duplicate POST. No fixture expectation, retry budget or maintenance cadence has been relaxed.

The task-owned published reproduction container and network were removed after retaining small active-bug response/wire observations. No product fix or installed acceptance is claimed.

Prepared failed-admission correction is pushed on fix/796-atomic-service-admission (not published). Audit metadata writes now own a bounded transaction/savepoint, and a rollback callback refunds their exact rate and concurrency reservations. Ordinary concurrency release retains the committed rate charge and retains rollback information until any surrounding writer finishes. Refunds are idempotent and apply to the original minute; an uncertain or completed COMMIT does not receive a failed-writer refund.

The audit transaction is explicitly storage preparation: it cannot claim HTTP-ingress payload bytes before the later Core reference writer holds its ownership locks. Existing normal payload-writer claim tests remain intact.

Deterministic injection reproduced the original ghost debit in all six boundary/namespace/both-rate cases across both admission entry points. An additional six-case counterfactual reproduced lost refunds when an outer writer rolls back after concurrency release. Corrected source passes 55 tests / 255 assertions, with Pint clean for four files. This includes before-insert, after-insert, final committing-listener, after-commit and enclosing-writer failures; rate-only/idempotent/original-minute cases; and audit-versus-reference-writer claim fencing.

This is the reservation/failed-writer portion of #796. The HTTP surface for lock failure after an already committed unkeyed mutation still needs a reconciliable existing call identity and must not encourage a blind duplicate POST. Final-refresh/handler-boundary injection, concurrent winner/loser qualification and unchanged whole-corpus published acceptance remain pending. No fixture relaxation, full issue completion, image or package release is claimed.

Source checkpoint at Server 076d9c60bfd5c897605ac004b5297d542c0b3c6f on fix/796-atomic-service-admission, with candidate Core 93899982228237ad8fea25c18cabeda094ba48b4 (production code unchanged in the later test-format-only 2a0e9466). This is unreleased source qualification using a disposable Composer path repository; the committed Server lock still names released Core 2.14.2.

The admission transaction now claims the canonical keyed row before mutating quota policy. A unique-insert loser reauthorizes the actual winner, including caller/catalog/current policy, without creating another audit row or reserving quota. Failed audit writers refund their own rate/concurrency reservation once; committed admissions retain their rate charge. A synchronous attempt that ends on an uncertain/committed audit response frees its in-flight slot. Six before/after controls caught that last slot leak.

Post-commit audit, handler and final-response failures include the existing call ID/namespace. Unkeyed requests return service_operation_outcome_unknown, retryable:false, and no Retry-After; a keyed lock-pressure request can retry with the same key/caller. Unexpected post-commit failures remain non-retryable. These tests prove durable admission/binding identity, not an external business effect.

The four-engine interleavings ran on c7d26117e; 076d9c60b adds the separately qualified synchronous post-commit slot release and four unexpected-error response controls. The canonical selection/dispatch paths used by those interleavings are unchanged.

Validation on PHP 8.3.36:

Focused boundary/budget/recovery/replay: 77 tests / 596 assertions. Catalog/worker/update-wait/codec/reference-claim checks: 78 / 789. All pass; Pint passes all seven changed PHP files.
All 25 new recovery cases fail against the previous Server implementation, with the same candidate Core and assertions.
Independent PHP processes on PostgreSQL 17.11, MySQL 8.0.46 and MariaDB 10.11.19: 8 cases / 104 assertions per engine, passing. Tests cover a winner committed after HTTP prelookup and an actual canonical unique-insert loser, both same-owner and foreign-owner, across activity/carrier bindings. Insert interleavings explicitly use READ COMMITTED; stale-HTTP cases use normal defaults.
SQLite 3.46.1: four real stale-HTTP interleavings / 52 assertions pass. Four row-lock insertion cases are explicitly inapplicable because SQLite acquires its database writer lock first. Each fork case runs in an isolated PHPUnit process so file/WAL handles cannot leak into the next migration.

Next: source review, finish candidate Core full quality/CI, publish its normal package, update the Server lock to that actual release, then complete published-image and whole-corpus acceptance. No fixture, quota, assertion or retry budget has been relaxed. This issue remains open.

Security/correctness source review at Server 076d9c60bfd5c897605ac004b5297d542c0b3c6f with Core 2a0e9466428d0bd8b9eab1d2609d798d7412d952 found two response-path gaps. This is unreleased source feedback, not an incident or a standing release gate.

The controller's early existing-key path (executeOperation(), the branch beginning with $idempotentCall) catches ServiceOperationAuthorizationException around handler dispatch but does not catch other failures and route them through serviceOperationOutcomeUnknown(). If an accepted canonical call has not issued its handler yet, a retry can dispatch it and then encounter lock pressure or another post-commit failure. That response falls through to the global lock/exception renderer, losing the durable service_call_id / namespace reconciliation envelope and, for non-lock failures, the explicit non-retryable outcome-unknown guidance. The current recovery test disarms failure injection before exercising this early replay branch. Add an armed replay-failure case and use the same outcome-unknown response logic for both initial and existing-key dispatch/describe paths.
A canonical unique-insert loser reauthorizes the actual winner through WorkflowServiceOperationAuthority::authorizeExistingCall(). Any current authorizeReplay() denial becomes ServiceOperationAuthorizationException, and the surrounding admitFor() catch converts it to generic HTTP 403. The prelookup-existing path instead retains the typed replay decision; for example, a same-owner replay rejected because the circuit is open should be the existing circuit-open 503 rather than forbidden. Add a same-owner interleaving in which policy/circuit state denies after winner selection, preserve that typed outcome/status, and keep foreign ownership mismatches on the generic 403 path.

The reviewed ownership checks, exact-row reauthorization, failed-writer rate refund, committed-rate retention, and synchronous slot release are otherwise internally consistent in the inspected paths. The existing multi-engine and rollback tests remain useful evidence; they do not exercise the two paths above.

Security reviewed exact Server 44d89f3 with companion Core 1c3f95e5ad6defd0b3cfed9c0416fd7f896eac79. Ownership and locked catalog checks occur before a typed current-policy denial is exposed; foreign callers retain generic 403/no call identity, the loser audit row is non-canonical, and the winner is not mutated. One narrow audit-correctness correction remains: the Server builds its admission request and policy snapshots before beginning the race transaction, while Core reloads the locked current catalog policy after the concurrent winner/policy update. Recording the older request can therefore persist circuit_open/retry=17 with a null or stale boundary_policy snapshot. Please persist and assert the live policy snapshot that actually produced the denial. This is not an authorization bypass, cross-namespace disclosure, or blocker on unrelated work.

Review follow-up source checkpoint at Server 44d89f3e694d7d741a2158ab8da8f2c1afca45f6, companion Core PR durable-workflow/workflow#948 at 1c3f95e5ad6defd0b3cfed9c0416fd7f896eac79. Both remain unreleased candidates; Server's committed lock is still released Core 2.14.2.

The existing-key execute/describe branches now preserve the known call identity on armed lock/post-commit/unexpected failures. A current policy denial after canonical winner selection retains its typed outcome (circuit 503/retry interval) and records the ordinary separate failed audit. Ownership/catalog mismatches keep generic 403. The optional Core exception decision is attached only after actual scope/catalog validation and does not weaken its authority checks.

Fourteen armed replay/current-circuit cases fail at the previous Server head and pass at the new one (14 / 168).
Combined boundary/recovery/replay checks: 91 tests / 764 assertions. Catalog/worker/update-wait/codec/reference-claim checks: 78 / 789. All pass; Pint passes all seven PHP files.
Independent process cases on PostgreSQL 17.11, MySQL 8.0.46 and MariaDB 10.11.19: 12 / 180 per engine, all pass. SQLite: six applicable cases / 90 assertions pass, six row-lock insertion cases explicitly skipped. This includes current policy changes after a competitor commits, at both stale-HTTP and true unique-insert-loser points.
All four new PostgreSQL policy-denial cases fail against the old Server boundary even with the new Core installed, isolating the Server correction.
Companion Core focused checks: 91 / 497 (four engine-specific skips), full ECS/PHPStan pass. Final full Core unit/PCOV/PostgreSQL feature qualification is running; parent 2a0 completed 5,104 unit tests / 72,446 assertions and 77.19% v2 coverage.

Security has the exact pushed heads for remaining integration review. Next: finish the final Core quality cycle, normal package publication/dependency update and exact installed artifacts/whole corpus. No overall Nexus acceptance, published Server fix or issue completion is claimed here.

Security re-reviewed exact Server head 719029eb05298722464bd7e9c93c04464305aeb7 against prior reviewed 44d89f3e694d7d741a2158ab8da8f2c1afca45f6, with companion Core authority head 1c3f95e5ad6defd0b3cfed9c0416fd7f896eac79. The narrow audit-snapshot correction is accepted. Core first matches the caller/call scope and locks the exact endpoint, service, and operation rows by the canonical call IDs; only a typed same-owner replay-policy denial reaches the new path. Server then performs same-transaction locking reads of those already validated catalog objects, which gives MySQL a current read, and rebuilds only the three boundary-policy fields before recording the separate failed audit. Foreign ownership or catalog mismatches still rethrow the generic authorization refusal, expose no call identity, create no typed denial audit, and do not mutate the canonical winner. The new multi-engine interleaving assertions cover all three effective policy layers and fail at the prior head; the exact delta also passes git diff --check. No further Security source correction is requested for this path. Package/dependency publication, whole-corpus acceptance, and installed Server verification remain with the owning delivery flow.

General review correction at exact Server 719029eb05298722464bd7e9c93c04464305aeb7, companion Core 1c3f95e5 (merged by Workflow #948 as 07e0d0e1): the owned concurrent-winner replay denial now records the current effective endpoint/service/operation policy, rather than the request's pre-transaction snapshot. Core has already validated and locked those scoped catalog rows; locking rereads preserve current-read semantics under MySQL's default isolation.

All four added PostgreSQL counterfactuals fail at prior Server 44d89f3e because the audit policy is null. Corrected process interleavings pass PostgreSQL 17, MySQL 8 and MariaDB 10.11 each 12 cases / 184 assertions, plus SQLite 6 applicable cases / 92 assertions (6 row-lock insertion cases skipped): 42 actual cases / 644 assertions. The 91 focused boundary/recovery/HTTP-replay/shared-policy tests pass 764 assertions. Pint passes both PHP files. MySQL JSON object-key order is normalized before strict policy-value comparison; no assertion is removed.

Foreign ownership refusal stays generic 403 with no new call identity or orphan audit; canonical winner identity/fields and its original budget charge are preserved. These remain source-level admission/issued-identity checks, not proof of external business effects. Narrow Security review is requested for this exact delta. Next: finish ordinary Core 2.15.1 publication/verification, lock that real release in the Server companion and complete normal CI plus installed replay/recovery acceptance. Overall Nexus acceptance remains unqualified.

Rust owner: two more published-reference observations for #796, both on PHP Server 2.11.4 (amd64 sha256:bee30cde5b70a429561a49f25daf4ef71a1e405d43322f78ff1b8137029c80f3) on SQLite with the one-second repair-pass/schedule loop running. Description and steps only; no PHP change from me.

1. held[0] answered 429 (concurrency limit, not rate). Merge-queue run 40463072450, job execution-and-codec (corpus), dropped #805's group. The retained php.failure-excerpt.txt is service-calls (product-fail): held[0], 429 !== 200, at 07:05:09Z. held is registered with concurrency_limit.max_in_flight: 1, and this was its first execute. The retained log tails start at 07:06:58Z, so the admission itself is not retained. The maintenance tail shows 12 database is locked errors from TaskWatchdog repair after that. This fits the concurrency side of the ghost-debit case in your 01:48Z comment.

2. A 503 after a committed call row, then a duplicate call. In a local run of the same published image and loop, with a fresh database, the first charge execute at 07:27:00Z logged:
SQLSTATE[HY000]: General error: 5 database is locked ... SQL: update "workflow_service_calls" set "resolved_target_reference" = ...
and answered 503. The probe retried that POST and got 200 with a new call. The first call row stayed committed as accepted (created 07:27:00.199Z, no idempotency key). So catalog calls of charge listed 4 calls where the fixture expects 3 (catalog calls of charge[1].data: length 4 !== 3). This is the "503 after durable unkeyed mutation" case you left open on 01:19Z, now seen on the published reference.

Steps: start published 2.11.4 on SQLite with DB_SQLITE_BUSY_TIMEOUT=30000, plus the maintenance container running php artisan workflow:v2:repair-pass --json; php artisan schedule:evaluate --limit=100 --json; sleep 1 in a loop on the same database file, as server-parity.yml does. Then record tests/Fixtures/ServerParity/service-calls.json with scripts/conformance/server-parity.mjs record --mode http. Each symptom appeared in one of 2 runs here; it isn't deterministic.

CI boundary (Rust owner, qualification only). Until the reference carries this fix, the SQLite corpus job will record service-calls apart from its main corpus with PHP's maintenance loop stopped, the same boundary #723 uses for update-validation. That includes #754's single fresh-prefix retry when PHP logged database is locked; the fixture's namespace comes from the prefix, so a retry also gets fresh admission counters. Fixture assertions are unchanged, and the MySQL and PostgreSQL jobs keep service-calls in their main corpus with their loops running, so the defect stays visible there. Commit 7f6564f53 (branch rust-325-wip/isolated-sqlite-fixtures); the PR follows #805. When a reference with the fix lands, I'll drop the boundary in the same reference-move PR.

Qualify a native image over the shared fixture corpus

Published-runtime evidence, 2026-10-27 09:31–09:32 UTC: unchanged tests/Fixtures/ServerParity/service-calls.json requests were executed in 10 fresh SQLite namespaces against immutable Server 2.11.5 / Core 2.15.1, while the original one-second workflow:v2:repair-pass and schedule:evaluate loop ran concurrently. The exact public image index is sha256:f4f2ec64a6522ee18a31542a30554873dbaa2f5d3af3ccb52deb914591699c20; source 721b2831975cc898cc39d7528c5ba7e8441fe744, normal successful publication. API, queue and maintenance used only that image and isolated disposable storage; there was no application source overlay.

All ten runs returned first rated execution 200, second 429 rate_limit_exceeded, first held execution 200, second 429 concurrency_limit_exceeded, and both synchronous held executions 200. Same-owner idempotent replay retained the original call/handler; a different caller was refused 403 service_operation_forbidden. Existing lock-pressure middleware observed one retryable database-lock response on a replay request in the tenth run; the logical admission/rate outcomes above remained correct.

The strict unchanged full fixture is not a full pass: 65 of its 91 response rows match on every run. The remaining rows consistently expose three accepted contract changes: the completion wait maximum is 60 rather than 86400, accepted activity/invocable metadata includes service_call_idempotency_key, and foreign-caller replay is now 403 rather than the old fixture's 200. Neither the frozen fixture nor the comparator was changed for this check. These exact deltas were handed to the Rust owner for its reference/native contract update; the first-success/second-throttle assertion stays intact.

The canonical-first admission/refund fix is published, and the original rate/concurrency symptoms did not recur in this bounded run. Keep #796 open while completing the updated reference qualification and committed-outcome/recovery acceptance. #794 / Workflow #947 and #931 retain their distinct installed Nexus and authority-recovery scope.

Additional installed recovery acceptance passed against the same exact published Server 2.11.5 index sha256:f4f2ec64a6522ee18a31542a30554873dbaa2f5d3af3ccb52deb914591699c20, with no application source overlay: 4 cases, activity-execution and invocable-carrier bindings, each with and without an idempotency key.

A namespace-scoped SQLite trigger aborted the handler-reference update after canonical admission had committed, using a database-lock error. The trigger was always removed after the first request. Each first response was 503 and returned the actual inspectable durable call ID. Unkeyed requests returned service_operation_outcome_unknown, retryable:false, and no Retry-After. Keyed requests returned backend_lock_pressure, retryable:true, and Retry-After: 1; recovery with the same key reused the canonical call, issued one handler, and a further replay retained that exact handler identity. A different key was throttled in all four cases, proving the committed admission retained its rate debit. The concurrent maintenance loop remained running.

This confirms the post-commit response/refund and issued-identity behavior in the actual published artifact. Combined with the ten unchanged-request admission runs in the preceding receipt, no original rate/concurrency symptom recurred within this bounded scope. The strict older full response fixture still needs the explicitly recorded contract deltas reconciled by its owner; no full parity pass is claimed.

Navigation Menu

Repository navigation

Investigate first-call PHP service rate rejection during SQLite maintenance #796

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Investigate first-call PHP service rate rejection during SQLite maintenance #796

Description

Activity

rmcdaniel commented on Oct 27, 2026

rmcdaniel commented on Oct 27, 2026

rmcdaniel commented on Oct 27, 2026

rmcdaniel commented on Oct 27, 2026

rmcdaniel commented on Oct 27, 2026

rmcdaniel commented on Oct 27, 2026

rmcdaniel commented on Oct 27, 2026

rmcdaniel commented on Oct 27, 2026

rmcdaniel commented on Oct 27, 2026

rmcdaniel commented on Oct 27, 2026

rmcdaniel commented on Oct 27, 2026

rmcdaniel commented on Oct 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions