Skip to content

[ISSUE #11170] Preserve transaction subscriptions before first heartbeat - #11171

Open
qianye1001 wants to merge 3 commits into
apache:developfrom
qianye1001:codex/fix-transaction-heartbeat-grace
Open

qianye1001 wants to merge 3 commits into
apache:developfrom
qianye1001:codex/fix-transaction-heartbeat-grace

Conversation

@qianye1001

@qianye1001 qianye1001 commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Which Issue(s) This PR Fixes

Brief Description

A transaction send can add a group-to-cluster subscription before the producer's first heartbeat registers its channel with the proxy. If the transaction heartbeat scan runs in between, it removes the subscription. The later producer heartbeat does not rebuild it, so a producer that sends no further transaction messages can miss transaction checks.

Retain recently active subscriptions while the group is not yet online:

  • Add lastActiveTimestamp in milliseconds, using System.currentTimeMillis(), to the existing ClusterData entries and refresh it on subscription updates and online scans.
  • Skip broker heartbeats immediately when the group is offline, preserving the behavior required by Stop send heartbeat to broker when ProducerGroup is offline in proxy #5862.
  • Reclaim offline subscriptions after the existing channelExpiredTimeout (120 seconds by default). Explicit unsubscription still removes them immediately.

ClusterData currently has no timestamp. A channel's last-update time is unavailable before its first registration, and per-message TransactionData expiry has a different lifecycle. This change adds no new configuration, map, or background task.

An earlier proposal by @redlsz, #8320, addressed the same race by introducing a ClusterDataSet.lastAddTimestamp and a separate transaction-group timeout. It was closed as stale without being merged. This patch keeps the existing collection structure and timeout configuration. Unlike #8320, it sends no broker heartbeats while the producer group is offline, including during the grace period; it also refreshes subscription activity when the group is observed online.

How Did You Test This Change?

  • The delayed-first-heartbeat regression fails against unmodified develop at 80e1ae55773c2330d2005b86f020ed028f649e94 because the subscription is removed.

  • Added six deterministic regression tests covering delayed registration, never-registered expiry, repeated subscription updates to the same cluster, online activity, replacement, and explicit unsubscription. All six pass with the fix; they use no timing sleeps.

  • Ran with Amazon Corretto 11.0.23 and Maven 3.9.8:

    mvn -B -pl proxy -am \
      -Dtest=ClusterTransactionServiceTest,AbstractTransactionServiceTest,TransactionDataManagerTest \
      -Dsurefire.failIfNoSpecifiedTests=false test

    Result: 16 passed, 0 failures/errors, 1 skipped. The skip is the existing macOS guard in TransactionDataManagerTest.testWaitTransactionDataClear. Checkstyle and SpotBugs passed.

Validation is at the service/unit-test level; a full broker/client E2E run has not been performed for this patch.

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 49.42%. Comparing base (80e1ae5) to head (34a10ee).

Additional details and impacted files
@@              Coverage Diff              @@
##             develop   #11171      +/-   ##
=============================================
- Coverage      49.51%   49.42%   -0.10%     
+ Complexity     14290    14266      -24     
=============================================
  Files           1390     1390              
  Lines         103129   103138       +9     
  Branches       13485    13487       +2     
=============================================
- Hits           51068    50978      -90     
- Misses         45909    45983      +74     
- Partials        6152     6177      +25     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@qianye1001
qianye1001 marked this pull request as ready for review September 16, 2026 05:43

@RockteMQ-AI RockteMQ-AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This PR fixes a race condition where a transaction send can add a subscription before the producer's first heartbeat, causing the transaction heartbeat scan to remove it prematurely. The fix adds a grace period using lastActiveTimestamp on ClusterData entries, allowing subscriptions to survive until channelExpiredTimeout elapses.

Verdict: APPROVED

The implementation is clean, minimal, and well-tested:

  • Correctness: Properly addresses the race condition without breaking existing behavior. Explicit unsubscription still works immediately.
  • Performance: Minimal overhead (one long per ClusterData, cheap System.currentTimeMillis() calls).
  • Tests: 6 comprehensive regression tests covering all scenarios (delayed registration, expiry, refresh, replacement, explicit unsubscribe). Tests are deterministic with no timing sleeps.
  • Compatibility: No API or wire protocol changes. Fully backward compatible.

The approach of reusing channelExpiredTimeout is elegant — no new configuration needed. The code changes are surgical and preserve the existing structure.

Minor observation: The test helper expireSubscriptions() uses reflection to access the private lastActiveTimestamp field. This is acceptable for testing but makes the test sensitive to field renames. Consider adding a package-private setter for testing if this becomes a maintenance burden.


Automated review by "github-manager-bot"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Proxy removes transaction subscriptions before the first producer heartbeat

4 participants