Repository navigation
Do not count sequence number 0 against the reliable send window - #451
Open
victorpaleologue wants to merge 1 commit into
Open
victorpaleologue wants to merge 1 commit into
victorpaleologue wants to merge 1 commit into
Conversation
A reader proxy starts with all_acked_before = 0 and keeps it until the reader's first ACKNACK. Writer::refresh_acked_frontier runs as soon as a reader matches and passes the minimum over the reliable readers to WriterSendBuffer::set_acked_frontier, which stores it as the acknowledgement frontier, replacing the buffer's own initial value of 1. has_room counts the unacknowledged samples as last_seq - acked_before + 1. Sequence numbers start at 1, so with a frontier of 0 that count includes sample 0, which does not exist: a fresh writer with a matched reader that has not acknowledged anything counts one sample in flight before it has written any. The window then holds window_limit - 1 samples until every reliable reader has sent an ACKNACK, and a writer whose window is 1 (ResourceLimits max_samples = 1) refuses its first write, blocking for max_blocking_time and returning WouldBlock. set_acked_frontier now reads a frontier below 1 as 1, the value an ACKNACK cannot go below either (handle_ack_nack clamps readerSNState.base the same way). Only the flow-control frontier changes: the reader proxies keep their initial value, so heartbeat and repair decisions, which compare against all_acked_before directly, are unchanged. The new unit test admits the first sample into a window of one with a reader frontier of 0, which failed before this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Until a matched reliable reader sends its first ACKNACK, the reliable send window counts one sample that does not exist. The window holds one sample fewer than its limit, and a writer whose window is 1 refuses its first write. This makes the acknowledgement frontier never go below 1, which is what the
TODOinWriter::refresh_acked_frontierasks for.The bug
RtpsReaderProxystarts withall_acked_before: SequenceNumber::zero()in all three constructors (new,from_reader,from_discovered_reader_data), and keeps it until the reader's first ACKNACK.handle_ack_nacknever stores anything below 1 — it clampsreaderSNState.basewithmax(base, SequenceNumber::from(1)), and its comment explains that base 1 is how a reader says "nothing received".Writer::update_reader_proxycallsrefresh_acked_frontieras soon as a reader matches. That passes the minimumacked_up_to_before()over the reliable readers — 0 for a reader that has not acknowledged yet — toWriterSendBuffer::set_acked_frontier, which stores it asacked_before, replacing the buffer's own initial value of 1.has_roomthen computesOn a fresh writer that is
0 - 0 + 1 = 1: one sample in flight before any has been written. So, until every reliable reader has sent an ACKNACK:window_limit - 1samples, andResourceLimits { max_samples: 1, .. }) refuses its first write: it blocks formax_blocking_time, then returnsWouldBlock("reliable send window full").Since 0.14.0 the window is
ResourceLimits::max_samples(8192 when unset), so with default QoS this costs one slot out of 8192. In 0.13, where the window was theKeepLastdepth, it caused visible failures in ROS 2.ros2-clientpublishes with reliableKeepLast(1)QoS in several places. Its action status writer (transient-local,KeepLast(1)) timed out when a goal was accepted, and theros_discovery_infowriter returnedWouldBlockregularly.The fix
set_acked_frontierreads a frontier below 1 as 1 (max(sn, SequenceNumber::new(1))). That is the same floorhandle_ack_nackapplies to an ACKNACK, so it is the value the frontier takes after the reader's first ACKNACK anyway.The fix is in the send buffer rather than in the reader proxy on purpose. The heartbeat tick and the repair path compare
last_change < rp.all_acked_beforedirectly, and starting proxies at 1 would change their behaviour on a writer that has not written yet — for example, the periodic heartbeat tick would stop sending to a fresh reader. Clamping where the window arithmetic lives changes flow control only. TheTODOinrefresh_acked_frontierkeeps its other point, slow first-match repair.Tests
The new unit test
writer_send_buffer::tests::unacknowledged_reader_frontier_leaves_full_windowcreates a reliable buffer with a window of 1 and sets the frontier toSome(SequenceNumber::zero()), as a matched reader that has not ACKNACKed yet does. It then checks that the first write is admitted, the second is refused, and an ACKNACK frontier of 2 makes room again. Onmaster(c076919) the first assertion fails. With this change the test passes.cargo test -- --test-threads=1: 694 + 2 + 2 + 58 passcargo test --features=security -- --test-threads=1: 704 + 2 + 2 + 58 passcargo +nightly fmt -- --check,cargo +nightly clippy --all --lib --bins --examples,cargo +nightly clippy --tests --examples(also with--all-features),cargo doc, andcargo +1.88.0 checkI did not add a two-participant test. The window is wrong only between a match and the reader's first ACKNACK, which comes back within a round trip of the HEARTBEAT the writer sends on match. A test that writes inside that interval depends on timing, so it would not be deterministic.
Questions for reviewers
These are related, but this PR does not change them:
matched_reader_updatemarks every existing sequence number as pending GAP for the new reader, because the reader is not owed those samples. They still count against the window until that reader's first ACKNACK. Take a writer that has writtenwindow_limitsamples or more, even with all of them acknowledged by the current readers. When a new reliable reader matches, the writer refuses writes until that ACKNACK arrives. Should a new volatile reader's flow-control frontier start atlast_seq + 1, or the send buffer'sfirst_seq, instead?KEEP_LAST. Should aKEEP_LASTwriter block on unacknowledged samples at all? DDS 1.4 §2.2.3.14 lets a reliablewriteblock if the modification would cause data to be lost or aRESOURCE_LIMITSlimit to be exceeded. UnderKEEP_LAST, replacing the oldest sample of an instance is what the history policy asks for, so arguably it is not loss and does not need to block. Since 0.14.0 the window comes frommax_samples, so this is rarely visible, but aKEEP_LASTwriter withmax_samplesset still blocks.🤖 Generated with Claude Code