lightningd: don't force-close when fulfilled HTLC removal is in progress - #9431
Open
vincenzopalazzo wants to merge 2 commits into
Open
Conversation
3 tasks
Collaborator
Author
|
Hi @daywalker90 👋 — this replaces #9408 (identical commits) and fixes #8899: nodes force-closing healthy channels on the fulfilled-HTLC removal deadline race (real-world logs in the issue from a BTCPay node). This is the most release-critical of my three v26.09 PRs — could you take a look before the RC? #9427 and #9428 are the other two. |
vincenzopalazzo
added a commit
to vincenzopalazzo/lightning
that referenced
this pull request
Aug 17, 2026
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
vincenzopalazzo
added a commit
to vincenzopalazzo/lightning
that referenced
this pull request
Aug 17, 2026
Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
vincenzopalazzo
force-pushed
the
test/reproduce-issue-8899-fulfilled-htlc-deadline
branch
3 times, most recently
from
August 18, 2026 08:37
d5360be to
0d1bd4a
Compare
CLN force-closes with "Fulfilled HTLC SENT_REMOVE_HTLC cltv hit deadline" even though it holds the preimage and just needs to reconnect to send update_fulfill_htlc upstream. test_fulfilled_htlc_deadline_no_force_close sets up l1->l2->l3, pays, then disconnects l2 from l1 right before update_fulfill_htlc is sent (-WIRE_UPDATE_FULFILL_HTLC): the incoming HTLC is left stuck in SENT_REMOVE_HTLC, and mining to the deadline triggers the force-close. test_fulfilled_htlc_deadline_reconnect covers the edge case where the deadline fires while removal is pending and l2 reconnects to finish the job. These fail until the next commit. Reproduces: ElementsProject#8899 Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
Once an incoming HTLC is fulfilled and has reached SENT_REMOVE_HTLC or later, removal is already in progress: channeld has been told to send update_fulfill_htlc upstream, or will be on reconnect. Force-closing here is counterproductive: 1. the preimage is in the DB, so onchaind can claim on-chain anyway, 2. the cooperative path (reconnect + fulfill) is cheaper and faster, 3. if the peer goes on-chain, onchaind handles it. So log an UNUSUAL and let the normal state machine finish the removal. This changes test_htlc_no_force_close and test_htlc_in_timeout, which depended on the old force-close: the fulfilling peer no longer force-closes, the offering peer does, and onchaind claims via the preimage. With this, the tests from the previous commit pass. Fixes: ElementsProject#8899 Changelog-Fixed: lightningd: don't force-close a channel when removal of a fulfilled HTLC is already in progress Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
vincenzopalazzo
force-pushed
the
test/reproduce-issue-8899-fulfilled-htlc-deadline
branch
from
August 18, 2026 08:58
0d1bd4a to
3ad7f7e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #8899.
Two commits, red then green: the first adds reproducer tests (they fail
on master with the force-close below), the second stops the force-close
and adapts the two existing tests which relied on it.
The bug
CLN force-closes the channel when a fulfilled incoming HTLC hits its
deadline, even though removal is already in progress:
But if the HTLC has reached SENT_REMOVE_HTLC (or later), channeld has
been told to send update_fulfill_htlc upstream, or will be on reconnect:
the preimage is in the DB, so onchaind can always claim on-chain, the
cooperative path is cheaper and faster, and if the peer goes on-chain,
onchaind handles it. Force-closing just burns fees.
After the fix we log an UNUSUAL and let the normal state machine finish
the removal:
The tests
test_fulfilled_htlc_deadline_no_force_close(reproducer): l1->l2->l3line graph, pay, disconnect l2 from l1 right before update_fulfill_htlc
is sent (
-WIRE_UPDATE_FULFILL_HTLC), so the incoming HTLC is stuck inSENT_REMOVE_HTLC; mining to the deadline used to force-close.
test_fulfilled_htlc_deadline_reconnect: same, but l2 reconnects afterthe deadline fires and completes the removal cooperatively.
test_htlc_no_force_close/test_htlc_in_timeoutare adapted: thefulfilling peer no longer force-closes, the offering peer does, and
onchaind claims via the preimage.
(Also dropped an unrelated event-notifications.md doc blob that had
accidentally landed on this branch.)