[FLINK-34807][mysql] Add MariaDB GTID support#4468
Open
leosanqing wants to merge 2 commits into
Open
Conversation
Author
|
@yuxiqian hi, could you help me review this PR? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
• Fixes FLINK-34807
Motivation
We use TDSQL(based on MariaDB) in our production environment.
When the MariaDB cluster changes, for example during scaling, migration, or failover, or when a Flink job restarts, the connector may be routed to a different MariaDB server. Although the new server belongs to the same GTID domain, its server ID may differ and its replicated GTID position may temporarily lag behind the position stored in the Flink checkpoint.
The current MySQL CDC connector cannot recover automatically in this situation because its GTID handling is based on MySQL's UUID:interval format and does not understand MariaDB's
domain-server-sequenceformat.An earlier PR, #2494, attempted to add MariaDB support. This PR adopts one important finding from that work: the MariaDB replication capability must be set to
4so thatMARIADB_GTIDevents are emitted and the consumed GTID position can advance.However, the earlier implementation does not fully address the production recovery scenarios described above. This PR introduces a more complete and extensible solution.
Description
This PR adds dialect-aware MariaDB GTID support to the MySQL CDC connector.
MariaDB and MySQL use incompatible GTID models:
uuid:intervaldomain-server-sequenceA new
GtidStrategyabstraction separates the parsing, comparison, containment, and recovery behavior of the two dialects.The implementation:
scan.dialectwithmysql,mariadb, andautomodes.@@gtid_binlog_pos.4so that GTID events advance the checkpointed offset.Comparison with the Earlier MariaDB PR
We use the earlier implementation in #2494 at first, but In pre-prod env we find several limitations could cause recovery problems.
1. Server ID is treated as part of the recovery identity(Main)
The earlier implementation uses
MariadbGtidSet.isContainedWithin()directly. That comparison includes the MariaDB server ID.In a MariaDB cluster, the server ID may change after failover, migration, scaling, or routing to another replica, while the GTID domain continues to represent the same logical stream. In production environments, especially with cloud databases that have a proxy layer, this scenario is very common.
This PR compares MariaDB GTIDs by domain and sequence. The server ID is preserved in the stored GTID text, but it is not used to determine whether one position contains another.
2. The restored GTID may be ahead of the selected replica(sometimes)
The earlier implementation passes the checkpointed GTID directly to the binlog client:
After a routing change, the selected replica may temporarily lag behind the checkpointed position.
3. MariaDB handling is embedded in the MySQL code path
4. Test coverage does not cover production recovery scenarios
Tests
Added or updated tests for:
Verified with JDK 11, Docker, and MariaDB 11.4: