[FLINK-40051][pipeline-connector/hudi] Fix schema operator UID with operator UID prefix#4460
Open
qiuyanjun888 wants to merge 1 commit into
Open
[FLINK-40051][pipeline-connector/hudi] Fix schema operator UID with operator UID prefix#4460qiuyanjun888 wants to merge 1 commit into
qiuyanjun888 wants to merge 1 commit into
Conversation
…perator UID prefix
Contributor
|
Could we avoid hard-coding the
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pipeline.operator.uid.prefixis configured.Root Cause
FlinkPipelineComposerderives the schema operator UID frompipeline.operator.uid.prefix(for example,<prefix>-schema-operator) when the prefix option is set. However,HudiDataSinkFactorystill passedpipeline.schema.operator.uiddirectly toHudiDataSink, so the Hudi sink used the deprecated default schema operator UID instead of the actual prefix-derived schema operator UID. This mismatch can make Hudi coordinator requests fail withNo coordinator registered for operator.Fix
HudiDataSinkFactoryderive the schema operator UID frompipeline.operator.uid.prefixwhen that option is explicitly configured.pipeline.schema.operator.uidbehavior when the prefix option is not set.HudiDataSinkFactoryTest#testSchemaOperatorUidFollowsOperatorUidPrefix.Validation
mvn -B -ntp -pl flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-hudi -am -Dtest=HudiDataSinkFactoryTest#testSchemaOperatorUidFollowsOperatorUidPrefix -DfailIfNoTests=false testmvn -B -ntp -pl flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-hudi -am -DfailIfNoTests=false testgit diff --checkJira
Reviewers
Could @cshuo or @voonhous please review when convenient?
AI assistance
This PR was prepared with AI assistance. The focused regression test and Hudi module tests listed above were run locally.