Repository navigation
Release message streams at retention, and keep PHP streams at takeover - #801
Merged
Merged
Conversation
Native's retention pass held every run whose instance had inbound message streams, because PHP's MessageStreamRetentionCleanup was never ported. A namespace that used message streams therefore kept those runs forever. The prune now plans and applies the stream cleanup in the run's own transaction. The instance's last retained run, once terminal, takes its streams and items with it. An earlier run's consumed items lose their payloads once a retained later run's cursor checkpoint covers them. Tested on SQLite, PostgreSQL, MySQL and MariaDB. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Native serves inbound message streams and durable workflow streams from PHP's own tables, and now releases them at retention as PHP does, so takeover carries their rows over instead of refusing the source. Each closed or errored durable stream gets the reclaim deadline native's retention pass reads: PHP 2.11.3's reclaim_at, or closed_at plus its retention for earlier releases, as 2.11.3's migration backfills it. Inspection now counts stream rows that hold an external payload reference, which takeover still refuses. Only workflow_messages, which native does not deliver yet, stays refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…325-stream-retention
The opt-in for native's maintenance loop (#797) was missing from the config audit's native-only names, so setting it logged "Unknown DW_ variables set in environment". The native image smoke test showed it. The shipped-environment test now sets it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#797 merged; #799, this PR and the native image are next. External payload takeover stays refused: PHP keeps references in about 30 locations in their URI form, so it is its own project, and the release-candidate scope question now includes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An independent review of the stream retention and stream takeover commits found no serious defects. This fixes what it did find: - A resumed MySQL takeover now picks step 10's deadline source by layout, as SQLite and PostgreSQL do, not by whether reclaim_at still exists. A 2.11.3 takeover killed after that column's drop had copied every deadline already, so it adopts nothing more instead of adding closed_at + retention rows for streams PHP had emptied. - The current run counts as retained only when its run row exists, as in PHP. - A run whose instance has an inbound stream item holding an external reference stays held (external_payload_cleanup_unavailable). Nothing writes one today; this keeps a later change from stranding objects. - The payload scan's comment no longer says stream item rows are refused. The retention test now puts the cursor exactly on the pruned run's last consumed message, and points the instance at a missing current run. Each new assertion fails against the code it guards (checked by mutation). SQLite: lib 338, execution 560. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SonarCloud failed this PR's quality gate on duplication (4.5% of new lines against a 3% limit): STREAM_REFERENCES repeated the same 6-line query for PostgreSQL, SQLite and MySQL, differing only in the ::text casts. One macro now builds it from the payload and metadata column expressions; SQLite and MySQL share one constant. The SQL each backend runs is unchanged. SQLite lib: 338 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This PR's first full run lost execution-and-codec (native-tests): rustc was killed with signal 9 compiling the library tests inside the 6 GB container. A cold cargo test --all-targets --no-run with two jobs peaks at 6.8 GB (cgroup memory.peak, measured locally on this tree); main passed only narrowly. The three containers that compile test targets (the all-targets run and the MySQL and PostgreSQL native test runs) now get 8 GB. The binary-only build keeps 6 GB. A hosted runner has 16 GB, which leaves room for the database container beside them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Member
Author
|
Merged through the merge queue at 2026-10-27T04:16:45Z, 21 minutes after enqueue (03:55:29Z, pinned to head
|
rmcdaniel
added a commit
that referenced
this pull request
Oct 27, 2026
The checkpoint names the verified 2.11.4 artifacts, the Core 2.14 expectation changes native now follows, the poll_transport position and the Workflow #931 boundary, with what was checked. The Rust port's current state lists #799 and #801 as merged, #803 and this move as next, and 2.11.4 as the compared reference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
12 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Part of #325. Pre-RC native Rust (tier C).
1. Release message streams at retention as PHP does (
d7abee65b)Native's retention pass held any run that had
workflow_messagesrows or whose instance had inbound message streams, withmessage_stream_cleanup_unavailable. Published PHP'sMessageStreamRetentionCleanupwas never ported. So a namespace that used message streams kept those runs' details forever.prune_runnow plans and applies the stream cleanup in the run's own prune transaction, under the transition lock. The plan follows PHP'splanForRun, read before the run's summary goes:message_stream_items_deletedandmessage_streams_deleted.cursor_checkpoint_run_idis a retained later run, meaning another run summary of the instance or its current run.payload_blobbecomes''andpayload_released_atis set.message_stream_items_compacted.cleanup_blocked_*markers naming this run are cleared.workflow_messagesrows go with the run's other details, as before.Not ported: PHP's
markBlocked, which marks a stream when external-payload cleanup blocks a prune. Native never deletes external objects. A run holding an external reference stays held asexternal_payload_cleanup_unavailable, and native refuses external references as message input.Tests, on SQLite, PostgreSQL, MySQL and MariaDB:
retention_releases_consumed_messages_and_removes_the_last_runs_streams: pruning a continued run compacts the two messages it consumed and keeps the third. Pruning the last run deletes the stream and its three items.retention_keeps_consumed_messages_no_retained_checkpoint_covers: without a retained checkpoint, nothing is compacted and the stream stays.2. Takeover keeps PHP message and workflow streams (
4e9012c58)Takeover refused any source with rows in
workflow_messages, either kind of stream, or their items. Stream rows now carry over unchanged.dw_stream_reclaimsdeadline. That is 2.11.3'sreclaim_at, or for earlier layoutsclosed_atplusretention_seconds(3600 unless set). Reported asstream_reclaims_recorded. MySQL and MariaDB do it in journal step 10.stream_reference_inspection. A stream row holding an external payload reference refuses withupgrade_source_external_payloads_unsupported.workflow_messagesstays refused, now asupgrade_source_workflow_messages_unsupported. Only embedded PHP workflows write it, and native does not run them.Tests:
takeover_keeps_php_streams_and_records_their_reclaim_deadlines(SQLite).3. Register
DW_RUST_MAINTENANCE_LOOPas a native setting (01ec49bf3)#797 added the opt-in, but the config audit did not know it. So setting it logged "Unknown DW_ variables set in environment". The local native image smoke test showed this. The audit test now includes it in the shipped environment.
4. Docs
docs/rust-server-port.md"Current state" is refreshed for October 27 and after #797 (8b2a19df1,3da8f54fd), including the recommendation that the release candidate keeps refusing external payload takeover. Checkpoint:docs/rust-server-port-checkpoints/2026-10-27-message-stream-retention.md.5. Fixes from an independent review (
ec23a5ad7)The review of commits 1 and 2 found no serious defects. Fixed:
reclaim_atwas dropped had already copied every deadline, so it no longer addsclosed_at + retentionrows for streams PHP had emptied.The retention test now puts the cursor exactly on the pruned run's last consumed message (
<=matters), and points the instance at a missing current run. Each new assertion was checked by mutation: it fails against the code it guards.Local results:
6. One macro for the stream reference count (
63de18803)SonarCloud failed the first run's quality gate on duplicated new lines (4.5% against 3%).
STREAM_REFERENCESrepeated one query for each backend. A macro now builds it from the column expressions, and the SQL each backend runs is unchanged.7. 8 GB for the Rust test builds (
f9610d429)The first full run lost
execution-and-codec (native-tests): rustc was killed with signal 9 compiling the library tests in the 6 GB container. A coldcargo test --all-targets --no-runwith two jobs peaks at 6.8 GB (cgroupmemory.peak, measured locally on this tree), and main had passed only narrowly. The three containers that compile test targets now get 8 GB; the binary-only build keeps 6 GB. Both organization policies pass.Still refused at takeover
workflow_messages(embedded PHP workflows only) and external payload references.🤖 Generated with Claude Code