Skip to content

Allow replay-compatible workflow definition upgrades and explain routing - #525

Merged
rmcdaniel merged 8 commits into
mainfrom
fix/worker-definition-evolution-518
Oct 20, 2026
Merged

rmcdaniel merged 8 commits into
mainfrom
fix/worker-definition-evolution-518

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Product correction in progress

Part of #518. PHP-only source on main 8511872f0ecf8d015eb55ec9e78ca745b94b70cb, which includes qualified PHP 150f3fbc and the separately owned CI-only #501 integration.

The default service claim gate permanently rejects a changed workflow source hash before the SDK can replay. It also ignores the published pinning opt-out. Existing embedded Core 2.7.2 already prefers a matching known class and falls back to the current class when the old definition is absent.

Source correction 57de9ec39b4107f32d2a7bcb6b070a4158cf3dc1 (whole tree 7f734fb91d81e5ba39579f21145eb9cbaab0fce5) aligns service routing with that policy: prefer a fresh eligible worker advertising the recorded definition, permit a changed definition to reach strict replay when no eligible original remains, and honor explicit opt-out. Original history/fingerprints, build/namespace/queue/type/capability and lease fences remain authoritative. A source hash is not proof that arbitrary application edits are replay-compatible; incompatible command replay must still fail explicitly.

Run detail/debug, queue detail and repair responses now expose the same read-only definition-routing decision. Allowed current-code replay is advisory, not a fabricated block. Only due Ready work with no eligible definition route reports a block. Queue examples and worker identities are bounded and truncation is explicit; namespace guards remain. Repair enrichment preserves its durable command outcome/reason. The authenticated worker wire version is recorded in an existing server-owned metrics key for replay eligibility, not accepted from client metrics. No new schema or client field.

Test-first evidence

  • The unchanged normal source passes all 19 existing version-pinning tests (98 assertions).
  • Three actual HTTP feature cases: default replacement and explicit opt-out fail on the original; matching-live-peer preference passes. The default failure is the original ready task not being delivered, not a setup refusal.
  • Parent e03bd5fc3 routing, registration, build/type/capability/protocol isolation, run/debug/queue/repair and surrounding suites: 189 tests / 2,155 assertions PASS. This includes eight ineligible-peer cases, busy-peer preference, immutable stored deadlines, source preference opt-out, original build-pin refusal, typed replay-protocol eligibility and client-metric spoof refusal. Final canonical selected/current-run correction: 95 tests / 1,279 assertions PASS.
  • Initial corrected assertion compared Carbon object identities; it now compares exact stored deadline bytes. A protocol fixture initially reused an existing history sequence; its separate corrected fixture uses the next actual sequence. Diagnostics initially omitted due microsecond timestamps; the implementation now uses UTC6 and includes null availability. A repair assertion assumed a particular outcome; it now verifies the actual response against the stored immutable command decision. Those unsuccessful observations are not counted as passes.
  • Four actual final-head normal Python SDK2.6.1 originals pass: default compatible replacement; live matching-peer preference; explicit opt-out; deliberately incompatible replay refused with no completion. Three separate cold Clients preserve exact int64/null/Unicode results. Eight application Workers exit0; eight first-attempt tasks/zero repairs/30 retained events, original prefixes/fingerprints and stored UTC6 60/120-second deadlines unchanged. Full independent receipt and limits.
  • Initial fixture/audit observations stay unsuccessful and separate. Development PHP API launcher requires137/noOOM, so this is not graceful API shutdown qualification. Fresh normal CI/Sonar and eventual published-image acceptance are pending. No issue closure or release claim.

No Rust/native/shared runner/frozen schema/catalog/OpenAPI/Composer/version change. This draft is not a release or a qualified fix. General retains PHP implementation and exact source handoff before any native adoption.

Current exact-source follow-through and release safety

Source follow-through is now e692df2196a3e973903e10ea2fec22487ba1707b / whole b4280b968732e0a84f0599033525845c2c040ea3. It integrates current main 1638681911a473b86484a1782b4654bab80ed209 without changing its native benchmark additions. Fresh normal gates are qualifying this combined tree; no prior whole-tree pass transfers.

The four actionable maintainability findings on prior 57de9ec3 were addressed in source, without threshold changes or suppression. A new actual HTTP regression also preserves the existing distinction between omitted client metrics and explicit process_metrics: null: omission retains them, explicit null clears them, and neither can overwrite the authenticated server-owned worker protocol. The old implementation failed that null regression; final routing/pinning/debug/queue/control-plane selection passes 114 tests / 1,746 assertions. Fresh analyzer resolution remains pending.

There is now a separate release-safety dependency: shared221 tracks source-verified service SDK entry-binding errors that can become terminal workflow failures before application versioning code runs. Core866 independently reproduces the embedded counterpart, including silently substituted DTO input. Current518 evidence proves command-history incompatibility refusal, not safe handling of every entry-signature/input-schema change. I will qualify that SDK boundary before publishing518's routing fallback; keeping old workers forever is not the product resolution. The independent PgBouncer524 fix can proceed through its own exact source/image gates.

No source-hash policy can prove arbitrary application code compatible. Original run/input/history/claim/deadline authority and the real SDK refusal decision remain required. This is an owned implementation/qualification dependency, with no operator action needed.

Current coordinated entry-safety follow-through

Exact ebfcbff19 incorporates the independently qualified Server2.9.8 main ancestry without altering native/shared/frozen source. Earlier source/installed observations above remain attributed to their recorded heads.

A genuine SDK source original on immutable published Server2.9.8 exposed a missing runtime boundary: the structured workflow_entry_arguments_mismatch task failure creates a ready retry instead of replay-blocking. The original remains unsuccessful and no retry loop or compatible recovery is inferred from it. Three new actual HTTP regressions reproduce that unwanted retry for PHP/Python/Rust failure types. The correction recognizes the canonical reason explicitly; supported repair reclaims the same original task with exact original input/history/deadlines.

Default changed-definition fallback also now requires the worker's additive workflow_entry_arguments_validation declaration. Older SDK implementations cannot silently gain that route. Matching recorded definitions and the explicit preference opt-out retain their existing contracts. Run/queue diagnostics report entry_validation_unavailable and the SDK-upgrade action when that declaration is missing. No protocol floor, schema or SDK-version heuristic is added.

Final local evolution/routing selection: 25 tests /297 assertions PASS. Surrounding genuine retry/owner/failure protocol selection: 8 tests /89 assertions PASS, preserving ordinary worker-process retries. SDK PHP#196, Python#176 and Rust#106 supply the corrected high-level Worker declarations; all normal exact-head gates, fresh combined source-backed business originals, published artifacts and API/Waterline acceptance remain pending. This draft is not release-qualified and does not close #518/#221.

Latest qualification

Three fresh genuine source-backed Rust Worker originals pass pre-entry refusal, cold replay refusal followed by compatible repair of the same original, and ordinary terminal body failure. Original input/history/deadlines remain exact. Full source evidence and retained unsuccessful observations. Published fixed-package/image acceptance remains required.

Current source is 85a81e67e2db0469a86831efe0171e8ce69dbdf4. It removes the valid extra-return finding through an equivalent final-return expression. Exact Sonar workflow/app and independent API review pass: gate OK, zero OPEN/CONFIRMED/REOPENED findings, 0.1% new duplication and 100% hotspot review. No rules, thresholds or exclusions changed. The MySQL topology, contracts and boundary checks pass; the PHPUnit feature suite and bounded-growth smoke remain live. No earlier whole-tree pass transfers.

@rmcdaniel

Copy link
Copy Markdown
Member Author

PHP correction is in draft #525 at 57de9ec39b4107f32d2a7bcb6b070a4158cf3dc1, whole tree 7f734fb91d81e5ba39579f21145eb9cbaab0fce5.

The default now prefers a fresh eligible recorded-definition peer, then lets a replacement undergo strict replay when no eligible matching peer remains. The published opt-out removes that preference. Build pins, namespace/queue/type, capability and actual wire-version eligibility, registration identity and claim/lease fences remain. Original history, fingerprint and deadlines are not rewritten. Run detail/debug, queue detail and repair expose the same routing policy; repair preserves its stored command decision.

Qualification:

  • Tests first: two new actual HTTP regressions fail on unchanged source; matching-live-peer control and all 19 old pinning tests pass.
  • Parent implementation e03bd5fc3: 189 focused and surrounding PHP tests / 2,155 assertions pass. Final canonical selected/current-run enrichment correction: 95 focused tests / 1,279 assertions pass.
  • Four fresh originals on final 57de9ec39, normally installed Python SDK 2.6.1 (33 Python distribution files independently match RECORD): default replacement completes the original after an actual get_version edit; a live gated matching peer keeps preference and completes when released; effective pinning=false permits replacement despite that peer; an incompatible condition key is leased but reports NonDeterministicReplayError/condition_key_changed, with zero business completion.
  • Eight actual application Worker processes exit 0. Three independent cold Client processes return exact signed-int64 maximum/null/Unicode results. Eight first-attempt workflow tasks, zero repairs, 30 retained events. All original history prefixes/fingerprints remain equal. Stopped query-only SQLite audits verify the unchanged original 60-second run and 120-second execution deadlines at stored microsecond precision, typed durable replay refusal and unchanged database/WAL hashes.

Earlier probe/setup and audit mistakes remain unsuccessful, including the first observer's incorrect history URL. That preliminary original is separate and retains one stale setup registration; final-cohort registrations are zero. No rows were deleted to manufacture an audit pass. The development PHP API launcher needed forced shutdown (137/no OOM), so graceful API shutdown is not qualified here.

Normal CI and Sonar for the exact new head are queued. No merge, release, installed Server image or native/shared/frozen adoption claim. The original published-image failure remains failed. The next gate is normal exact-head CI/Sonar, followed by publication and genuine installed-image acceptance. Native/shared/runner/OpenAPI/Composer/version paths remain unchanged.

@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel
rmcdaniel marked this pull request as ready for review October 20, 2026 15:37
@rmcdaniel
rmcdaniel merged commit 64c6968 into main Oct 20, 2026
13 checks passed
@rmcdaniel
rmcdaniel deleted the fix/worker-definition-evolution-518 branch October 20, 2026 15:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Changed Python workflow definitions silently strand in-flight runs

2 participants