Repository navigation
Investigate intermittent approved-update HTTP 500 in published PHP MariaDB corpus #825
Description
General accepts the PHP investigation handed off by the native Server owner. This is an intermittent published-PHP qualification failure; its root cause is not yet established.
The owner handoff identifies update-validation / approved_update returning HTTP500 instead of the expected202 on the PHP side of the MariaDB10.11 corpus in #822 run40578777402, job122454294740, using published Server2.11.5. The earlier #816 run40561089085, MariaDB job122392116543 reported the same response shape on published2.11.4. Neighboring full corpus runs pass; that does not explain either500.
The retained PHP log is a tail and does not retain the correlated request exception. A later external-payload exception belongs to a different fixture and must not be used as this failure's diagnosis. This is distinct from a demonstrated native-source regression. The normal #822 failed-job rerun is owned by the native maintainer; General will retain the original failures regardless of that rerun's result.
Current evidence: the owner correlated a later state-updates HTTP 500 with SQLSTATE 40001 / InnoDB 1213 while inserting workflow_tasks, reported in the request-specific receipt. The original approved-update 500s above remain uncorrelated. Both original published MariaDB probe contracts passed once in a controlled baseline, which does not explain the intermittent failures.
General pushed 517e2647512e7f94722050ace5d33fecceb48ef6 with owned-root MySQL/MariaDB admission retries after rollback. Twenty real-engine controls and 82 related tests pass; source and qualification receipt. Follow-up head adaad79ac327fda10d40dd4df96f5ce5e7b7dc1f also fences the selected run inside retried admission. A real deadlock concurrent with continue-as-new previously let the direct path record an unknown-update rejection after the requested run changed. All 24 real-engine controls, 78 related tests and 54 controller tests pass; the strengthened final four race cases pass with 140 assertions. See follow-up qualification. The approved path already refused stale validation. Task-owned database fixtures/network have been removed.
Merged PR: #828, main 759abaa3b4eb7d931cf256b1aab2950768c2c0a9. The combined PR head passed 16 checks with 7 conditional skips; the normal merge group passed all 43 Shared Server jobs and its full PHP feature qualification. Source review and the specific Sonar maintainability decision are recorded. No published-image acceptance is claimed.
Next action: qualify main, publish through the protected Server release path with the appropriate qualified Core dependency, then repeat the affected full MariaDB corpus against the exact published image. Keep the original approved-update failures open until their disposition is supported.
Acceptance:
- Preserve the original response, exact published PHP/Core tuple, database version, fixture runner and workload; qualify whether the input and expected202 remain valid.
- Reproduce or retain correlated evidence that distinguishes product storage/validation failure from fixture/environment failure. Do not assign a cause from a neighboring log entry.
- Correct any confirmed product defect with focused regression coverage, normal source qualification and published-consumer follow-through. Keep required checks and original failing evidence.
- Verify the affected full MariaDB corpus after repair; a passing rerun alone does not close this investigation.
Activity
rmcdaniel commentedon Oct 27, 2026
General accepted the published-PHP investigation from the Rust owner. Original artifact metadata independently confirms artifact
12773370526,mysql-execution-mariadb-10.11-corpus, run40578777402, at source0e7e955244d7a30fb19694bb04c263b90c44e582. The archive is 15,382 bytes and is not expired. Its normal GitHub download timed out, so I have not independently inspected its contents or correlated the HTTP 500 with an exception.The reports for published PHP 2.11.4 and 2.11.5 remain separate evidence. The unrelated payload exception in the retained log tail is not a root-cause finding. A passing failed-job rerun would not establish the cause. Next: recover the original request/case correlation and exercise that published-PHP MariaDB path independently, retaining any first failure before changing code.
rmcdaniel commentedon Oct 27, 2026
The Rust owner supplied a later, request-correlated report from run 40578777402, attempt 2, job 122482789008: the published PHP 2.11.5 record returned HTTP 500 for the
state-updatesfixture's selected-runset_payloadupdate at 21:40:40 UTC. The reported PHP exception was SQLSTATE40001, MariaDB error1213, on insertion intoworkflow_tasks, with maintenance running. The published-PHP recording failed before the native recording ran. This is an owner-reported correlation; General has not independently recovered that log. It does not establish the exception behind the earlierapproved_updatefailures.Source inspection traced remote workflow updates through Server's
ExternalWorkflowUpdateAdmission, including the outer approval transaction inValidatedExternalWorkflowUpdateAdmission. Their defaultControlPlaneMutationRetrierpath only retries SQLite, so the reported MariaDB path is a candidate retry gap. Core's embedded update admission also uses a single-attempt transaction, but it is a separate path and is not yet implicated in this remote fixture failure. Existing Core command-deadlock coverage exercises MySQL signal/cancellation cases and explicitly skips MariaDB. No reproduced root cause or delivered fix is claimed.Next: exercise the exact original probe inputs against published PHP 2.11.5/Core 2.15.1 on isolated MariaDB, retain correlated request and exception evidence, then qualify a scoped retry correction with rollback, idempotency, caller-owned transaction, and unrelated-error controls. Passing reruns do not close this issue.
rmcdaniel commentedon Oct 27, 2026
The isolated published-PHP reproduction harness is working. Original
state-updatesandupdate-validationprobes from runner source0e7e955244d7a30fb19694bb04c263b90c44e582each completed once and passed the canonicalcheckObservationcontract. The tuple is published Server 2.11.5, Core 2.15.1, normal locked PHP SDK 2.2.6, MariaDB 10.11.19. The database was bootstrapped with the frozen published 2.5.14 layout and migrated with the published current image. The repair/schedule maintenance loop stayed active; the isolated server's deadlock counter remained zero.This first pair did not reproduce either intermittent HTTP 500. It is not a root-cause finding or closure evidence. The exact selected images/input hashes and small original observations are retained only for this active investigation; disposable data, raw logs and task resources will be removed after the correlated diagnostic receipt.
The next useful check is a deterministic rollback/deadlock control at the actual Server external-admission boundary. A correction must retry an owned root transaction, preserve a caller-owned transaction, and avoid repeating validator execution or committed effects. The validated path has an outer approval transaction around parent admission, so blindly enabling nested retries would be insufficient. No product source has changed for this issue yet.
rmcdaniel commentedon Oct 27, 2026
Implemented and pushed the scoped correction at
517e2647512e7f94722050ace5d33fecceb48ef6, including actual merged #824. The application change isb8c685e73onfix/825-external-update-deadlocks; only the two external-update admission classes and a regression test differ from current main.Server now asks Laravel to retry owned root MySQL/MariaDB admission transactions using the existing configured attempt budget. Nested approval admission remains owned by its outer approval transaction. Caller-owned transactions, other storage connections, unrelated failures and after-commit failures retain their original behavior. Validator approval remains outside the retried admission, and the retained approval is rechecked on each attempt.
Before/after evidence:
1213while accessingworkflow_tasks.This reproduces and fixes the root-transaction retry gap at the actual Server external-admission boundary. It does not independently identify the exception behind the earlier uncorrelated
approved_update500s, or establish acceptance of a newly published Server image. Those failures remain retained. The original published 2.11.5 baseline still needs replacement only after a qualified publication and coordination with the native fixture owner.No PR or release has been created yet while CI throughput repair #826 is under qualification. Next: defensive exact-source review, protected checks/merge, qualified publication and published-artifact verification of the reported corpus cases.
rmcdaniel commentedon Oct 28, 2026
General pushed follow-up head
adaad79ac327fda10d40dd4df96f5ce5e7b7dc1fonfix/825-external-update-deadlocks.A controlled real InnoDB deadlock now also exercises a worker's actual continue-as-new operation between rollback and retry. Before the run guard, MariaDB's direct case retried twice, answered
unknown_updateHTTP 404 and left one durable update row after the requested run changed. The approved case already refused stale validation. The first loose probe allowed admission to commit before continuation and was discarded as an invalid interleaving; the retained negative control holds the history range so rollback actually precedes continuation.The controller now passes the selected run ID into external admission. Fresh admission rechecks it while holding the instance/run locks on each retry, and refuses a changed run with the existing
historical_run_command_rejectedHTTP 409. The guard runs after retained-key recovery, preserving committed receipts, and before validation can start for a different selected run. Ordinary successful response metadata is unchanged. No native or frozen-fixture source changed.Validation on the source logic: all 24 real-engine controls passed on MySQL 8.0.46 and MariaDB 10.11.19 (604 assertions), all 78 related request/validation/routing/fence/retrier tests passed (934 assertions), and all 54 control-plane/controller tests passed (819 assertions). After strengthening the race assertions, its four final cases passed with 140 assertions: one rolled-back insert attempt, correct original run/reason, and zero admitted command/update/history rows. Pint passed all four changed PHP files. This supplements the original rollback-only retry proof; it does not correlate the original approved-update HTTP 500 reports.
Next: exact-source review, normal CI/PR progression after the current CI bottleneck repair, then published-image acceptance and the full affected MariaDB corpus. No PR, tag or release is claimed yet.
rmcdaniel commentedon Oct 28, 2026
Merged #828 through the normal queue as
759abaa3b4eb7d931cf256b1aab2950768c2c0a9. All 43 merge-group Shared Server jobs and the merge-group PHP feature qualification pass. The PR head had 16 successful checks and 7 conditional skips, with its source/Sonar review recorded in the PR.This puts the demonstrated owned-transaction retry and selected-run corrections on main. It is not a published-image acceptance result. Next: qualify main, publish through the protected release path with the appropriate qualified Core dependency, and repeat the full affected MariaDB corpus against the exact published image. The originally reported approved-update HTTP 500s remain uncorrelated and this issue stays open.
rmcdaniel commentedon Oct 29, 2026
Handoff from #854's merge-queue run 40841285248 (MariaDB 10.11 corpus lane, job 123339409484). General owns the PHP follow-through.
tests/Fixtures/ServerParityBaselines/php-2.11.5.json; linux/amd64durableworkflow/server@sha256:bcc672859a399d1df234fc304531ab06b7bfa8acad7cb2c9da95f8e7ab5c98b4).many-updatesprobe sent its 16thPOST /api/workflows/{id}/update/set(wait_for: accepted, with arequest_id) while the installed maintenance loop (workflow:v2:repair-passthenschedule:evaluate) was running. The answer was an uncaught500with body{"message":"Server Error"}.SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction, on aselect * from workflow_tasks where workflow_run_id = …in the update admission.request_id) again, at most three times in all. Each retry writes a value-free note (many_updates_submit_retry: index, attempt, status) that the recorder keeps even when the probe passes. So a recovered 500 stays visible as a defect signal, not error-free qualification.