Skip to content

Rust #325: record local activities like Server 2.11.1 - #766

Merged
rmcdaniel merged 20 commits into
mainfrom
rust-325-record-local-activity
Oct 26, 2026
Merged

rmcdaniel merged 20 commits into
mainfrom
rust-325-record-local-activity

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

Part of #325. Native now serves the record_local_activity workflow task command (worker protocol 1.18+) the way published Server 2.11.1 does with Workflow 2.10.2.

What changes

  • Capabilities. Native advertises the local activity contract in server_capabilities and lists the command among supported workflow task commands. Prepared local activities stay unadvertised, with no groups and no cancellation policies.
  • Admission. The portable worker affinity guard runs first. Then the command normalizer applies its field, outcome, attempt and payload rules, with PHP's messages.
  • Storage. The bridge records the activity's execution, attempts, history and failure under one authored sequence. Every row ID is generated by the server. The worker's attempt_id is stored only as worker_attempt_id metadata and is never resolved as a reference.
  • Activity view. The run's activity view and attempt status follow Workflow 2.10+. Each attempt carries failure_reason, and an expired attempt keeps a recorded lease loss as its stop reason.

Completion budget (Workflow #930, phase A)

Security held this PR until the package bounded completions. Workflow #934 (merged as 4eb49a20) set that rule, and native now applies it.

  • Timing caps. duration_ms and heartbeat elapsed_ms are capped at 7 days and backoff_seconds at 1 day, each with the package's field error. The walk back from completion is overflow-checked, and refused with local_activity_timestamp_out_of_range before it reaches the earliest storable timestamp.
  • Early check. After a run's deadline close, which is never metered, the local activities' exact events and rows (other commands count one event) are checked against DW_V2_LIMIT_HISTORY_TRANSACTION_SIZE and _ROWS. A completion past either is refused with completion_budget_exceeded.
  • Meter.
    • Each history event charges one event, its bytes as PHP measures them (against _BYTES) and two rows. Each local activity row is charged before it is written.
    • A terminal event's worker attribution is charged too.
    • A refusal answers history_transaction_limit_exceeded, writes nothing and keeps the lease. A tripped meter also refuses the commit.
  • Visibility. Operator metrics, health and backend capabilities publish all three limits. config_audit honors them, with their legacy aliases.
  • A MySQL fix the matrix found. The run summary projection's history page could filesort whole rows ("Out of sort memory"), so any completion with one event over 256 KiB failed on MySQL. The query now forces the run-and-sequence index.

Every Security finding along the way is answered in this PR's comments; the final source acceptance is #766 (comment).

Fixture

tests/Fixtures/ServerParity/record-local-activity.json was recorded three times from published 2.11.1, with identical results. Raw workers cover:

  • the advertised contract and registration agreement;
  • the protocol floor;
  • capability and ownership gates;
  • transport, scope and normalizer refusals;
  • a success after a retried attempt;
  • a failure after retries, and a timeout;
  • the protocol 1.18 shape without attempts.

One published defect is bounded rather than reproduced. PHP's validated() rebuilds a nested report in rule order, so a report whose first attempt lacks an optional field is refused as unordered.

Qualification at 7f57016e

This head is main through #768. Its tree (b85f11f7) is the Security-accepted tree of backup commit 3f6a388c.

  • SQLite: cargo test --lib --test execution passes, lib 306 and execution 519.
  • PostgreSQL 17 and MySQL 8.0:
    • completion_budget passes 10/10, including the event, row and byte exact/one-over matrix and both terminal attribution controls.
    • local_activities, structural_limits and waterline_projection_guards pass too.

The 2.5.22 activity metadata corpus predates failure_reason. Its test now requires that field to be null, since no corpus case records a lease loss, and compares the rest as recorded.

🤖 Generated with Claude Code

rmcdaniel and others added 7 commits October 24, 2026 15:21
… 2.11.1

Raw workers complete workflow tasks with record_local_activity commands:
the advertised local_activities contract and registration agreement, the
protocol floor, capability and ownership gates, transport, scope and
normalizer refusals, a success after a retried attempt, a failure after
retries and a timeout, and the protocol 1.18 shape without attempts. History
payloads, the run's activity view and status are recorded with identities as
case-keeping labels and attempt times as offsets from each close. Recorded
three times, identical. One published defect is bounded: transport
validation reorders a nested report whose first entry lacks an optional
field, so the normalizer refuses it as unordered.
Native now serves the record_local_activity workflow task command (worker
protocol 1.18 and newer) as published Server 2.11.1 does with Workflow
2.10.2:

- It advertises the local activity contract in server_capabilities and
  lists the command among the supported workflow task commands. Prepared
  local activities stay unadvertised, with no groups and no cancellation
  policies.
- The portable worker affinity guard runs first, then the command
  normalizer's field, outcome, attempt and payload rules with PHP's
  messages.
- The bridge records the activity's execution, attempts, history and
  failure under one authored sequence.
- The run's activity view and attempt status follow Workflow 2.10+: each
  attempt carries failure_reason, and an expired attempt keeps a recorded
  lease loss as its stop reason.

One published defect is not reproduced: PHP's validated() rebuilds a
nested report in rule order, so a report whose first attempt lacks an
optional field is refused as unordered. The record-local-activity fixture
bounds that difference.

The 2.5.22 activity metadata corpus predates the attempt failure_reason.
Its test now requires that field to be null (no corpus case records a
lease loss) and compares the rest as recorded. A payload unit test now
expects PHP's final message for a completed record, whose missing-result
check replaces the resolver's message under the same key.

Native passes the record-local-activity fixture recorded from published
2.11.1, along with the capability-sensitive fixtures (cluster-info,
admission, worker listing, namespaces, worker replacement).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-activity

# Conflicts:
#	.github/workflows/server-parity.yml
…-activity

Conflicts with #745 resolved as unions. Capabilities advertise both the
local-activity manifest and message streams.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nto rust-325-record-local-activity

The worker capability manifest keeps this branch's local activity entries
and main's worker session and update validation entries. The comparator,
probe and node test lists keep both sides' fixtures and tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ust-325-record-local-activity

The comparator node test list keeps both sides' tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Security exact-head review of 38ee6ca5598da59270f0ce34121125100c27a5cb: hold merge for two availability boundaries.

  1. Valid local-activity timing values can panic the request path instead of returning a protocol refusal. The normalizer accepts every non-negative signed 64-bit value for elapsed_ms, duration_ms, and backoff_seconds. Those values then reach unchecked Chrono construction/arithmetic at lines 925-937 and 1025-1027. With the locked Chrono behavior, backoff_seconds = i64::MAX panics while constructing the duration; duration_ms = i64::MAX or elapsed_ms = i64::MAX overflows the supported DateTime range. An authenticated, capability-advertising worker holding the workflow-task lease can therefore make the handler unwind with a syntactically valid completion instead of receiving a deterministic refusal. The package-owned PHP contract accepts the same unbounded integers, so fix the owning contract and port the bound/checked behavior here rather than creating a native-only rule. Required regressions: each extreme field, just-inside/just-outside the accepted boundary, no durable mutation on refusal, the task remains usable, and the server remains responsive on SQLite/PostgreSQL/MySQL.

  2. The per-command heartbeat cap composes into an extreme completion transaction and severe payload amplification. A completion admits 100 commands, while each local-activity command admits 1,000 heartbeats independently. One single-attempt command writes one scheduled event, one started event, 1,000 heartbeat events, and one terminal event; a valid 100-command request can therefore request at least 100,300 history events in one transaction, plus execution/attempt rows and projections. Minimal heartbeat JSON for that batch remains below the default 2 MiB request ceiling. Worse, every heartbeat event copies the running activity snapshot, including arguments (construction, heartbeat append), so one large valid payload can be persisted hundreds or thousands of times. append performs the writes and projections without enforcing the history budget; the 10,000-event threshold is advisory only.

    Add a deterministic whole-completion budget before mutation for projected durable events/rows and serialized history bytes; a per-command cap is insufficient. Either avoid repeating large immutable payloads in heartbeat snapshots or include the repeated bytes in that budget. Add exact-boundary and one-over regressions across all three databases, including a large valid argument plus heartbeats; refusal must leave the lease and all durable rows unchanged. The same aggregate behavior exists in the package-owned PHP implementation, so this also needs an owning Workflow issue/fix and downstream Server qualification.

I found no local-activity external-payload prefetch/SSRF path at this head: external preparation selects only complete_workflow.result and schedule_activity.arguments, while local-activity external storage is refused before persistence. The server-generated durable IDs and non-resolving treatment of worker attempt_id / cancellation_scope_id also hold in this pass.

The record-local-activity fixture was recorded on SQLite, so each local
activity event's workflow task snapshot names the sqlite connection. On
the PostgreSQL and MySQL corpus lanes that snapshot names pgsql or mysql,
the backend that recorded it, and the contract compared the name exactly.
It now requires one known storage connection per observation and compares
it as {connection}. Every other step still compares exactly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

New head: d6b96bd0b5953017b0cf840af4b9172a89814807.

On 38ee6ca55, five corpus lanes differed in one value: the record-local-activity fixture's task connection. The fixture was recorded on SQLite, so its three local-activity events name sqlite. Native on PostgreSQL names pgsql, and native on MySQL and MariaDB names mysql. Those are the names PHP's task snapshots use on the same backends.

The contract now requires a single known storage connection per observation and compares it as {connection}. The reviewed fixture is unchanged, and every other step still compares exactly. A new unit test covers both directions: pgsql and mysql pass, while a mixed observation or an unknown name is refused. All 2,264 comparator node tests pass.

🤖 Generated with Claude Code

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security review of the exact 38ee6ca55..d6b96bd0b delta: no additional blocker in the connection-normalization change.

The delta changes only the record-local-activity comparator and its unit test. It replaces a backend-local sqlite / pgsql / mysql value with one placeholder after requiring that the observation contain at most one spelling and that every observed spelling be in that allowlist. The reviewed fixture and every other comparison remain exact. This proves consistency plus an allowlisted name; it does not prove that a MySQL lane reported mysql rather than another allowed spelling, which is acceptable only because the published fixture contract intentionally ignores the connection value.

The two availability blockers in comment 6401983037 remain merge blockers. Workflow #930 must first ship the package-owned timing and whole-completion admission contract, then this PR must port and qualify it on SQLite, PostgreSQL, and MySQL. Current hosted matrix jobs were still running when I checked, so this is not full-head merge acceptance.

@rmcdaniel

Copy link
Copy Markdown
Member Author

All 56 checks pass at exact head d6b96bd0b5953017b0cf840af4b9172a89814807, including the five corpus lanes that failed on the connection name. Security has no new finding on this delta (6402480001). The PR stays held for the two package-owned availability blockers, which I'm fixing first in Workflow #930 (revised design: durable-workflow/workflow#930 (comment)). Once the package carries the rule, I'll port it here and ask for re-review.

🤖 Generated with Claude Code

rmcdaniel and others added 2 commits October 25, 2026 22:34
Native applies phase A of Workflow #930 (Workflow PR #934) to
record_local_activity and to every workflow task completion:

- record_local_activity caps duration_ms and heartbeat elapsed_ms at
  7 days and backoff_seconds at 1 day, with the package's field errors.
  The walk back from completion is summed with checked arithmetic and
  refused with local_activity_timestamp_out_of_range before it reaches
  the earliest timestamp every supported column stores.
- After a run's deadline close, which is never metered, the local
  activities' exact events and rows (other commands at one event) are
  checked against DW_V2_LIMIT_HISTORY_TRANSACTION_SIZE and _ROWS, and a
  completion past them is refused with completion_budget_exceeded.
- The rest of the completion runs under a meter. Each history event
  charges one event, its HistoryBudget bytes and two rows, and each local
  activity row is charged before it is written. A refusal answers
  history_transaction_limit_exceeded and rolls back, leaving the lease. A
  tripped meter also refuses the commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-record-local-activity

# Conflicts:
#	.github/workflows/server-parity.yml
@rmcdaniel

Copy link
Copy Markdown
Member Author

Native port of Workflow #930 Phase A, ready for review. It mirrors the rule Security accepted and that was merged as Workflow #934 (4eb49a20).

The exact commit is bbc6025149892fd11812f896fbaf88e97bec0de7. It is on the backup ref rust-325-wip/record-local-activity, not yet pushed to this PR's head, so it does not start a second full CI run while #768 is active. It has main 2dd58d25 (#767, #764) merged in. Port commit: 0ed8de6c3.

What it does, in local_activities.rs, history_meter.rs, store.rs, transaction.rs and memo_limits.rs:

  • Caps. duration_ms and heartbeat elapsed_ms are capped at 604,800,000, and backoff_seconds at 86,400. Each uses the package's field error.
  • Walk and floor. backdated_ms sums the walk with checked arithmetic (None on overflow). storable refuses an earliest instant before 1970-01-01T00:00:01Z with local_activity_timestamp_out_of_range. Both run before any write.
  • Early check. It runs after the run's deadline close, which happens before the meter exists and so is never metered. The local activities' exact events and rows, plus one event for each other command, are checked against DW_V2_LIMIT_HISTORY_TRANSACTION_SIZE and DW_V2_LIMIT_HISTORY_TRANSACTION_ROWS. Past either, the completion is refused with 409 completion_budget_exceeded.
  • Meter.
    • Store::record charges each history event: one event, its PHP json_encode length plus the event type against DW_V2_LIMIT_HISTORY_TRANSACTION_BYTES, and two rows. Each local activity execution, attempt and failure row is charged before its insert.
    • A refusal answers 409 history_transaction_limit_exceeded with run_id null. The error drops the transaction, which rolls back.
    • A tripped meter also refuses Transaction::commit, so a swallowed refusal cannot commit.
  • No counterpart for staging or event deferral. Native never stores an external payload while applying a completion (workers upload first, and a completion only claims references), and it dispatches no lifecycle events.

Tests

  • Unit tests: each timing field at its maximum, one over, and with i64::MAX, refused by that field alone; backoff required only when absent; the walk's span rule, the longest report, overflow, and the floor boundary; and the meter's measure, limits, a tripped refusal and its commit refusal.
  • Execution tests (completion_budget.rs):
    • An over-budget completion writes nothing and the same lease completes afterwards.
    • The byte meter refuses, leaves no history, execution or attempt rows, and keeps the lease.
    • The row budget counts a local activity's rows exactly (9 refused, 10 admitted).
    • A run past its deadline times out before the budget is checked.

Local results at bbc602514

  • SQLite: cargo test --lib --test execution passes, 295 unit and 508 execution tests.
  • PostgreSQL 17 and MySQL 8.0: completion_budget and local_activities pass, 8/8 each.

Known gaps: the doc's entry records that native is ahead of published 2.11.1 here. Operator metrics' structural-limit snapshot stays as 2.11.1 reports it until the PHP Server takes the Workflow release.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security review of exact backup commit bbc6025149892fd11812f896fbaf88e97bec0de7: hold promotion to the PR head for an operational safety boundary. Review of the remaining delta continues.

This candidate now reads and enforces configurable history_events, history_bytes, and history_rows limits in StructuralLimits and the completion meter. However, operator_metrics::structural_limits() still publishes a hard-coded history_transaction_size: 5000 and omits history_transaction_bytes and history_transaction_rows. The system-health response copies that incomplete snapshot, while the nested backend-capability snapshot is built from StructuralLimits::default() instead of the active configuration.

Consequently, a server configured with another event ceiling—or with the check disabled—reports 5,000 anyway, and neither of the other two enforced hard ceilings is inspectable. Operators cannot explain a completion_budget_exceeded / history_transaction_limit_exceeded refusal from the advertised runtime state.

This is the native counterpart of finding 4 on Workflow #934. Its accepted Phase A source added all three values to StructuralLimits::snapshot() and proved them through HealthCheck, OperatorMetrics, and BackendCapabilities. Deferring the native snapshot until a later PHP Server release is not safe: this exact native candidate already changes enforcement and therefore must make those controls truthful at the same boundary.

Before promotion, publish the active configured values (0 when disabled) in the configured and effective structural-limit views, including the health and nested backend-capability surfaces. Add controls with three distinctive non-default values plus disabled values, so a hard-coded/default snapshot cannot pass. Preserve backend-specific adjustments without replacing the active ceilings.

No Rust tests were run on the lab host. This finding is from exact-source comparison with the merged Workflow Phase A contract.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Second exact-commit blocker on bbc6025149892fd11812f896fbaf88e97bec0de7: the startup configuration contract was not updated with the enforcement code.

StructuralLimits::from_env() now honors:

  • DW_V2_LIMIT_HISTORY_TRANSACTION_SIZE
  • DW_V2_LIMIT_HISTORY_TRANSACTION_BYTES
  • DW_V2_LIMIT_HISTORY_TRANSACTION_ROWS

and their WORKFLOW_V2_... legacy aliases. But config_audit::SETTINGS still classifies ..._SIZE as Ignored, and has no entries for ..._BYTES or ..._ROWS.

The missing primary names are therefore reported as unknown DW_* variables. With DW_ENV_AUDIT_STRICT=1, configuring either newly enforced byte or row ceiling stops native startup with the generic unknown-variable refusal. The size entry also makes the durable config contract claim that an actively enforced setting changes nothing native observes. The legacy byte/row aliases are silently read without the expected deprecation mapping because their primary entries do not exist.

Before promotion, register all three as Honored with their exact legacy aliases. Add audit controls proving that strict mode accepts each primary name, legacy names produce only the established deprecation warning and still select the same value, and no enforced setting is listed as ignored or unknown. Reconcile the port documentation's honored/ignored counts at the same time.

This is separate from the incomplete structural-limit runtime snapshot in the prior comment: fixing telemetry alone would still leave documented non-default configuration unable to start under the supported strict audit mode.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Third exact-commit promotion blocker on bbc6025: the original aggregate-budget acceptance matrix is not yet represented by the native controls.

The completion_budget execution module proves an exact event boundary (4 admitted, 7 refused) and an exact row boundary (9 refused, 10 admitted). Its byte control uses one arbitrary 4,000-byte refusal followed by a smaller unrelated completion; it does not prove the exact byte boundary and one-over behavior. The local helper uses a small fixed argument and 200-character heartbeat details, so it does not exercise the reported large-valid-argument plus heartbeat amplification. On refusal, the byte test compares history and counts only activity_executions and activity_attempts; it does not prove that every durable sibling/projection table remains unchanged. The PostgreSQL/MySQL handoff reports the same focused eight tests, so repeating them on three backends does not fill those semantic gaps.

The original Security requirement in comment 6401983037 was exact-boundary and one-over coverage for events, rows, and serialized history bytes on SQLite, PostgreSQL, and MySQL; a large valid argument plus heartbeats; and proof that refusal leaves the lease and all durable rows unchanged. Please add that matrix before promotion. Derive the byte boundary from the same PHP-compatible event measurement used by the meter so the control cannot pass by relying on an incidental 4,000-byte estimate. Snapshot all completion-owned durable tables before and after the refused request, then prove the same lease succeeds with an admitted request.

This evidence gap is independent of the runtime-snapshot and config-audit blockers already recorded. No Rust tests were run on the lab host.

rmcdaniel and others added 2 commits October 25, 2026 23:09
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d-local-activity

# Conflicts:
#	.github/workflows/server-parity.yml
#	docs/rust-server-port.md
rmcdaniel added a commit that referenced this pull request Oct 25, 2026
Security's review of 6676720:

- events() now orders the projected events from their boundary, time and
  source order alone, then builds, encodes and counts one event at a time.
  The first event past DW_V2_LIMIT_HISTORY_TRANSACTION_BYTES stops the
  build before it is kept or any later event is built. TooLarge reports
  how many events were built, and a test-only counter proves an early
  over-limit event builds none of the thousands after it.
- history_rows is removed from this change: #766 adds it with its
  enforcement, audit and disclosure.
- The no-write snapshots count every table of the test database,
  workflow_links included.
- A lock barrier control holds each backend's transition lock while the
  projection's install waits on it: an append then finds no instance and
  writes nothing, and once the install commits an append is refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel and others added 5 commits October 25, 2026 23:34
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…snapshot helpers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The byte matrix's large completion (a ~270 KB argument and 20 heartbeats)
failed on MySQL 8.0 with "Out of sort memory": the run summary projection
pages history with SELECT ... CAST(payload AS CHAR) ... ORDER BY sequence
LIMIT 16, and MySQL may answer it with a filesort that carries a whole row;
one large payload exceeds the default 256 KiB sort buffer and fails the
completion with 503 storage_unavailable. The query now forces the
workflow_run_id/sequence index, as the history page query already does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Answer to Security's three holds on bbc60251:

New exact commit: e363afac03ca9c8da45b9ebdc9a44e94693d0491, on the backup ref rust-325-wip/record-local-activity. It is not pushed to this PR's head yet; it activates after #768 merges.

It contains #768 (merged locally at 3e62112d), because #768 also honors the event and byte limits and carries their shared snapshot and audit code. The #766-only delta is the port plus the row limit, the matrix and one MySQL fix:

git diff 3e62112dd1758beeb224fba1c43d541404d071dd e363afac03ca9c8da45b9ebdc9a44e94693d0491

1. The snapshot publishes the active limits

operator_metrics::structural_limits() publishes the active history_transaction_size, history_transaction_bytes and history_transaction_rows (0 when disabled). The backend capabilities snapshot is built from the active configuration and memo limit instead of StructuralLimits::default(). Backend adjustments and issues are preserved.

Control: every_surface_reports_the_active_history_transaction_limits, in six views: the operator metrics, the nested backend capabilities and the health snapshot, each configured and effective. It checks:

  • distinctive values: 4321, 987654 and 13579;
  • disabled values: Some(0) and None, which both read 0;
  • the defaults: 5000, 16777216 and 20000.

2. The config audit honors all three

  • DW_V2_LIMIT_HISTORY_TRANSACTION_SIZE is Honored in SETTINGS.
  • DW_V2_LIMIT_HISTORY_TRANSACTION_BYTES and _ROWS are in a new NATIVE_HONORED list, with their WORKFLOW_V2_... legacy aliases. They are not in PHP's contract yet.
    • That list is excluded from the unknown-name warning.
    • Its legacy names give the established deprecation warning.
    • A test asserts they stay outside PHP's contract and are not listed twice.

Control: strict mode starts with all three primary names set. The legacy names produce exactly the two deprecation warnings and nothing else. No enforced setting is ignored or unknown.

The port doc's counts are reconciled: PHP's 164 contract settings, of which native honors 49, ignores 51 and refuses 64 non-default values. Native also honors _BYTES and _ROWS.

3. The acceptance matrix

completion_budget.rs runs each limit at its exact value (admitted) and one less (refused) on a fresh run, then the same lease completes a wait.

  • Events: [local(3), wait()] is 7 events. Admitted at 7; refused with completion_budget_exceeded at 6.
  • Rows: [local(0), wait()] is 10 rows. Admitted at 10; refused at 9.
  • Bytes, with a large valid argument: [large(20), wait()] is a ~270 KB argument and 20 heartbeats with details.
    • The boundary is measured from the stored history the same way the meter charges it: event type plus compact JSON payload, summed over the completion's events and read straight from the database on each backend. That comes to several MB.
    • Admitted at that value; refused with history_transaction_limit_exceeded at one less.
  • On refusal:
    • every table's row count is unchanged, from sqlite_master or information_schema;
    • every task's status, lease owner, lease expiry and attempt count, and every run's status, history position and update time, are unchanged;
    • the same lease then completes.

A MySQL defect the matrix found

On MySQL 8.0, the large completion failed with 503 storage_unavailable before any limit applied. The cause is error 1038, "Out of sort memory", found with the general query log.

The run summary projection pages history with SELECT … CAST(payload AS CHAR) … ORDER BY sequence LIMIT 16. MySQL may plan that as a filesort that carries the whole row, and a single ~360 KB payload exceeds the default 256 KiB sort buffer. The MySQL query now forces the workflow_run_id, sequence index, as the history page query already does (commit d4fe36e40). The test's own history read does the same.

This defect exists independently of the budget: before this fix, any completion on MySQL with one history event larger than the sort buffer failed the same way.

Local qualification at e363afac03ca9c8da45b9ebdc9a44e94693d0491

  • SQLite: cargo test --lib --test execution passes, 305 unit and 517 execution tests.
  • PostgreSQL 17 and MySQL 8.0, for completion_budget, local_activities, structural_limits and waterline_projection_guards: 20/20 each.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security exact-successor review of backup commit e363afac03ca9c8da45b9ebdc9a44e94693d0491 (the 3e62112d..e363afac #766-only delta): the three requested corrections are materially present in source. The configured/effective metric, health, and backend surfaces publish all three active limits; strict audit accepts the primary names and maps the legacy names; and the three-backend matrix now exercises exact/one-over event, row, and measured multi-megabyte byte boundaries with refusal rollback and lease reuse. The MySQL summary query forces the existing run/sequence index and is appropriately scoped.

One byte-budget bypass remains before promotion. Store::record() charges a history event only after its task/command snapshots are assembled and before the insert. Later in the same completion, attribute_worker_terminal() reloads WorkflowCompleted or WorkflowFailed, adds the authenticated worker principal fields through command_context::worker_terminal(), and updates the stored payload. That post-record payload growth is never charged to HistoryTransactionMeter. A terminal completion admitted at the exact byte limit can therefore persist a final history transaction over the configured hard ceiling; the current large-local-activity matrix never records a terminal event and cannot catch this.

Build the terminal attribution into the payload before Store::record() measures/inserts it, or charge the exact serialized growth before the update without double-counting existing bytes. Add authenticated-worker exact/one-over controls for both WorkflowCompleted and WorkflowFailed; the refused request must leave the terminal event/run/task absent or unchanged and keep the same lease usable.

This is source-only review; no Rust build or runtime action was performed on the lab or development VPS. Promotion/full-head acceptance remains held on this correction, exact hosted evidence, and the separately recorded #768 SQLite interleaving blocker.

rmcdaniel and others added 2 commits October 26, 2026 00:33
A completed or failed run's terminal event is recorded, charged, and then
rewritten to name the worker that completed the task. That growth was
never charged, so a completion admitted at the exact byte limit could
store more history than the limit allows. The attribution's growth is now
charged before the update, measured on the same stored payload before and
after, so nothing is charged twice. Controls for WorkflowCompleted and
WorkflowFailed admit the stored terminal history at its exact size and
refuse it one byte short; both fail without the charge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Answer to #766 (comment). New exact commit: 3f6a388ce9d649619637da268aab8124ead65b21, on the backup ref rust-325-wip/record-local-activity. It includes #768 at 6838f47f.

Fix. attribute_worker_terminal measures the stored terminal payload (php_json_len), applies worker_terminal, measures it again, and charges the growth to the byte meter (Transaction::charge_bytes) before the UPDATE.

  • The event and its recorded bytes were charged when it was recorded, so only the growth is added, measured on the same payload before and after. Nothing is charged twice, and a shrink is not refunded.
  • A refusal there trips the meter, so the transaction rolls back and the commit is refused too.

Controls, for both terminal events (a_completed_run_s_attributed_terminal_event_is_charged, a_failed_run_s_attributed_terminal_event_is_charged):

  1. A probe run completes [large(5), complete_workflow | fail_workflow]. Its stored terminal event names the worker (command.source = worker_protocol, principal_id present).
  2. The byte boundary is measured from the stored history after attribution, the way the meter charges it.
  3. A fresh run is admitted at exactly that size.
  4. Another fresh run is refused one byte short with history_transaction_limit_exceeded. All table counts, task leases and run state are unchanged, so the terminal event, run close and task were all rolled back.
  5. The same lease then completes a wait.

With the charge removed, both controls fail.

Local qualification at this commit

  • SQLite: cargo test --lib --test execution passes, 306 unit and 519 execution tests.
  • PostgreSQL 17 and MySQL 8.0: completion_budget passes 10/10 each (the 8 earlier tests plus these 2), run at 5f4fd3fca.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security exact-successor review — source boundary only.

Reviewed backup-ref merge 3f6a388ce9d649619637da268aab8124ead65b21 (tree b85f11f7341ea9af9f34c882383394ede421d24e). Its #766-only correction is 5f4fd3fca4ca26b7b4ed632ceae17242ec4db8a1 (tree 7e486dbc165d10fb75985f2eba2123c44ac38b2f, parent e363afac03ca9c8da45b9ebdc9a44e94693d0491). The other merge parent is the already-reviewed #768 head 6838f47feb76fc563a45b0a9f53cf4f048e447b1; stable patch IDs prove that merge carries exactly the reviewed 3e62112d..6838f47f delta, without extra conflict-resolution changes. git diff --check passed.

The terminal-attribution metering blocker from comment 6405437415 is materially closed:

  • attribute_worker_terminal() reads the already-stored terminal payload, measures it with the same PHP-compatible JSON meter used for event admission, adds the server-derived worker attribution, and charges only the positive serialized growth before issuing the UPDATE;
  • the completion and failure payload constructors are server-controlled and contain no pre-existing command, so this path adds rather than replaces/shrinks untrusted command material;
  • a refused growth charge propagates before the update, trips the transaction meter, and prevents commit, rolling back the earlier task/run/history/completion mutations;
  • separate WorkflowCompleted and WorkflowFailed controls derive the exact stored post-attribution size, admit it exactly, refuse one byte short, compare every table plus task and run state, and prove the same lease can then complete a smaller wait.

This closes Security's source-level change request. It is not merge acceptance yet: PR #766 still points at d6b96bd0b5953017b0cf840af4b9172a89814807, so the accepted tree must first become the immutable PR head and complete its hosted checks. Security did not run a local Rust build or runtime action.

@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

Hosted qualification at exact head 7f57016e768ff0539beba35f2a03dfcd649f0b67 (tree b85f11f7341ea9af9f34c882383394ede421d24e, identical to the Security-accepted tree of 3f6a388c):

  • The checks are terminal: 56 passed and 1 path-inapplicable job was skipped, with none failing or pending.
  • The merge state is clean against main ac39e3a4.

SonarCloud decision. The quality gate is OK, with no bugs, vulnerabilities or hotspots. Sonar reports three rust:S3776 cognitive-complexity notes in rust/src/runtime/local_activities.rs:

  • line 273, the heartbeat normalizer (33);
  • line 418, the attempt-report normalizer (74);
  • line 721, normalize (46).

They are the native ports of WorkflowCommandNormalizer's local activity rules, which run in PHP's order and keep PHP's error-message precedence (Laravel's validated() ordering, including the bounded defect described in the PR body). The #930 caps added one branch to each.

Splitting these functions risks changing the order in which field errors are produced, and that order is what the parity fixture compares. I'm accepting the notes for this change. A later refactor can split them by rule group behind the existing fixture and unit tests.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Exact-head receipt for #766, as Security asked in chat 1870.

Tree identity

  • Accepted commit 3f6a388ce9d649619637da268aab8124ead65b21 has tree b85f11f7341ea9af9f34c882383394ede421d24e.
  • Current head 7f57016e768ff0539beba35f2a03dfcd649f0b67 has tree b85f11f7341ea9af9f34c882383394ede421d24e.
  • The trees are identical. The head is 3f6a388c merged with main ac39e3a49b066b4935f4894272cb2894406b4632, which only adds Rust #325: Waterline v1 workflow import #768's tree, already in 3f6a388c.

Hosted runs at 7f57016e, all completed with success:

Check-run rollup for the commit: 57 success and 1 skipped (path-inapplicable), with none failing or pending. The merge state is clean.

The MySQL large-probe path

  • Job mysql-execution (mysql-8.0, native-tests), https://github.com/durable-workflow/server/actions/runs/40259289924/job/121369422310. Its MySQL server is 8.0.46. Artifact 12623868215 (mysql-execution-tests.txt) records:
    • test completion_budget::a_run_past_its_deadline_closes_before_the_budget_is_checked ... ok
    • test completion_budget::a_completed_run_s_attributed_terminal_event_is_charged ... ok
    • test completion_budget::the_meter_counts_a_local_activity_s_rows ... ok
    • test completion_budget::the_byte_limit_admits_its_exact_value_and_refuses_one_more ... ok
    • test completion_budget::the_row_limit_admits_its_exact_value_and_refuses_one_more ... ok
    • The shard's result: test result: ok. 260 passed; 0 failed; 0 ignored; 0 measured; 260 filtered out; finished in 294.10s.
  • Job mysql-execution (mariadb-10.11, native-tests), https://github.com/durable-workflow/server/actions/runs/40259289924/job/121369422398. Its server is 10.11.19-MariaDB. Artifact 12624388576 records all nine completion_budget tests passing, and test result: ok. 519 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 580.35s.

the_byte_limit_admits_its_exact_value_and_refuses_one_more is the large-argument probe: about 270 KB with 20 heartbeats, exact and one under. It passing on MySQL 8.0 also exercises the forced-index summary query fix.

@rmcdaniel
rmcdaniel merged commit 086ba7f into main Oct 26, 2026
58 checks passed
@rmcdaniel
rmcdaniel deleted the rust-325-record-local-activity branch October 26, 2026 03:07
@rmcdaniel

Copy link
Copy Markdown
Member Author

Merged as 086ba7fafa9bf76eb10a28d537ef3b5b448dea02, pinned to exact head 7f57016e768ff0539beba35f2a03dfcd649f0b67.

Native now serves record_local_activity with Workflow #930 Phase A's completion budget. Workflow #930 stays open for Phase B.

Next in #325: cancellation scopes.

rmcdaniel added a commit that referenced this pull request Oct 26, 2026
…llation-scopes

# Conflicts:
#	.github/workflows/server-parity.yml
#	rust/src/runtime/store.rs
rmcdaniel added a commit that referenced this pull request Oct 26, 2026
…-operations

Conflicts kept both sides: the parity contract, probe and workflow test list
now carry the Nexus operations and record_local_activity entries together.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel added a commit that referenced this pull request Oct 26, 2026
…nto rust-325-workflow-stream-writes

The port guide now counts 142 served route methods: stream writes and the
two workflow imports each add two to the base 138.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel added a commit that referenced this pull request Oct 26, 2026
…ust-325-bridge-adapters

Conflicts kept both sides: the router, delegates, modules, parity contract,
probe and workflow test list carry bridge adapters with the workflow
imports and record_local_activity. The port guide counts 141 served route
methods: bridge webhooks add one and the two imports two to the base 138.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant