FLINK-38334: Fix MySQL CDC source stuck in INITIAL_ASSIGNING#4278
Open
morozov wants to merge 1 commit intoapache:masterfrom
Open
FLINK-38334: Fix MySQL CDC source stuck in INITIAL_ASSIGNING#4278morozov wants to merge 1 commit intoapache:masterfrom
morozov wants to merge 1 commit intoapache:masterfrom
Conversation
When a table is excluded from configuration after a restart from savepoint, the MySQL CDC source could get stuck in the INITIAL_ASSIGNING state. This happened because table exclusion cleanup was only performed when isAssigningFinished() was true, but the assigner couldn't finish because excluded table splits were never reported as finished. The fix separates two concerns in captureNewlyAddedTables(): - Adding new tables: should only happen when isAssigningFinished() - Removing excluded tables: must happen regardless of assigner status Added integration test TableExclusionDuringSnapshotIT that reproduces the issue by using a blocking hook to take a savepoint during INITIAL_ASSIGNING phase, then restarting with a table excluded from configuration.
Contributor
Author
|
@leonardBang could you approve running the workflow? |
Contributor
Author
|
The tests are failing due a Docker API incompatibility issue. It looks like you guys are working on it in #4275. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a table is excluded from configuration after a restart from savepoint, the MySQL CDC source could get stuck in the
INITIAL_ASSIGNINGstate. This happened because table exclusion cleanup was only performed whenisAssigningFinished()wastrue, but the assigner couldn't finish because excluded table splits were never reported as finished.The fix separates two concerns in
captureNewlyAddedTables():isAssigningFinished()Added integration test
TableExclusionDuringSnapshotITthat reproduces the issue by using a blocking hook to take a savepoint duringINITIAL_ASSIGNINGphase, then restarting with a table excluded from configuration.