Enable configurable multi-lease acquisition per cycle in change feed processor#47606
Enable configurable multi-lease acquisition per cycle in change feed processor#47606jeet1995 merged 4 commits intoAzure:mainfrom
Conversation
|
Thank you for your contribution @stas-openai! We will review the pull request and get back to you soon. |
There was a problem hiding this comment.
Pull request overview
This PR introduces an opt-in configuration (setMaxLeasesToAcquirePerCycle) to the Change Feed Processor (CFP) that allows acquiring multiple leases per balancing cycle, improving rebalance/convergence time during scale-out and rolling deployments. The default value of 0 preserves the legacy conservative behavior of acquiring at most one lease per cycle when multiple workers exist.
Key Changes:
- Added
maxLeasesToAcquirePerCyclefield and accessors toChangeFeedProcessorOptionswith validation - Updated
EqualPartitionsBalancingStrategyto honor the new option for expired/unowned lease acquisition while keeping the legacy 1-lease-per-cycle behavior for stealing - Wired the new option through both CFP implementations (EPK and PK/Incremental versions)
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
ChangeFeedProcessorOptions.java |
Adds the new maxLeasesToAcquirePerCycle field with getter/setter and validation |
IncrementalChangeFeedProcessorImpl.java |
Passes the new option to the EqualPartitionsBalancingStrategy constructor |
ChangeFeedProcessorImplBase.java |
Passes the new option to the EqualPartitionsBalancingStrategy constructor |
EqualPartitionsBalancingStrategy.java |
Implements the multi-lease acquisition logic while maintaining legacy behavior for stealing |
CHANGELOG.md |
Documents the new feature in the changelog |
PartitionLoadBalancerImplTests.java (pkversion) |
Adds test verifying all leases returned by strategy are attempted |
PartitionLoadBalancerImplTests.java (epkversion) |
Adds test verifying all leases returned by strategy are attempted |
EqualPartitionsBalancingStrategyTests.java |
Adds comprehensive unit tests for the new multi-lease acquisition behavior |
API Change CheckAPIView identified API level changes in this PR and created the following API reviews |
|
@microsoft-github-policy-service agree company="OpenAI" |
|
Thank you @stas-openai for your contributions to improve the Change feed processor logic, we will review this PR in the upcoming week (once everyone is back from vacation) and will get back to you with any comments, thanks again, your contributions are deeply appreciated! |
FabianMeiswinkel
left a comment
There was a problem hiding this comment.
Looks good to me - Thanks for the contribution!
|
/azp run java - cosmos - tests |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/check-enforcer override |
|
QueryValidationTests.queryPlanCacheSinglePartitionParameterizedQueriesCorrectness is failing which is unrelated to this change (follow up in a separate PR). |
|
Woah, appreciate this! Thank you. |
Description
This PR adds an opt-in configuration to Change Feed Processor (CFP) to allow acquiring more than one lease per lease-
acquire cycle, which can significantly improve rebalance/convergence time during scale-out and rolling deployments
(especially when hosts use ephemeral identities, e.g., Kubernetes rollouts).
Today, when multiple CFP workers exist, the default EqualPartitionsBalancingStrategy is intentionally conservative and
effectively attempts to take ownership of at most one lease per cycle. In large lease sets / high physical partition
counts, this can make rebalancing very gradual and can temporarily reduce processing throughput after deployments.
Changes included:
strategy.
No Swagger regeneration involved.
All SDK Contribution checklist:
General Guidelines and Best Practices
merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR,
see this page.
Testing Guidelines