Skip to content

[FLINK-34807][mysql] Add MariaDB GTID support#4468

Open
leosanqing wants to merge 2 commits into
apache:masterfrom
leosanqing:feature/mariadb
Open

[FLINK-34807][mysql] Add MariaDB GTID support#4468
leosanqing wants to merge 2 commits into
apache:masterfrom
leosanqing:feature/mariadb

Conversation

@leosanqing

@leosanqing leosanqing commented Jul 11, 2026

Copy link
Copy Markdown

• Fixes FLINK-34807

Motivation

We use TDSQL(based on MariaDB) in our production environment.

When the MariaDB cluster changes, for example during scaling, migration, or failover, or when a Flink job restarts, the connector may be routed to a different MariaDB server. Although the new server belongs to the same GTID domain, its server ID may differ and its replicated GTID position may temporarily lag behind the position stored in the Flink checkpoint.

The current MySQL CDC connector cannot recover automatically in this situation because its GTID handling is based on MySQL's UUID:interval format and does not understand MariaDB's domain-server-sequence format.

An earlier PR, #2494, attempted to add MariaDB support. This PR adopts one important finding from that work: the MariaDB replication capability must be set to 4 so that MARIADB_GTID events are emitted and the consumed GTID position can advance.

However, the earlier implementation does not fully address the production recovery scenarios described above. This PR introduces a more complete and extensible solution.

Description

This PR adds dialect-aware MariaDB GTID support to the MySQL CDC connector.

MariaDB and MySQL use incompatible GTID models:

  • MySQL: uuid:interval
  • MariaDB: domain-server-sequence

A new GtidStrategy abstraction separates the parsing, comparison, containment, and recovery behavior of the two dialects.

The implementation:

  • Adds config option scan.dialect with mysql, mariadb, and auto modes.
  • Add MariaDbGtidStrategy (MOST IMPORTTANT)
  • Reads the local MariaDB binlog position from @@gtid_binlog_pos.
  • Compares MariaDB GTIDs by domain and sequence instead of server ID.
  • Corrects a restored GTID against the GTID position available on the target server.
  • Enables MariaDB replication capability 4 so that GTID events advance the checkpointed offset.

Comparison with the Earlier MariaDB PR

We use the earlier implementation in #2494 at first, but In pre-prod env we find several limitations could cause recovery problems.

1. Server ID is treated as part of the recovery identity(Main)

The earlier implementation uses MariadbGtidSet.isContainedWithin() directly. That comparison includes the MariaDB server ID.

In a MariaDB cluster, the server ID may change after failover, migration, scaling, or routing to another replica, while the GTID domain continues to represent the same logical stream. In production environments, especially with cloud databases that have a proxy layer, this scenario is very common.

This PR compares MariaDB GTIDs by domain and sequence. The server ID is preserved in the stored GTID text, but it is not used to determine whether one position contains another.

2. The restored GTID may be ahead of the selected replica(sometimes)

The earlier implementation passes the checkpointed GTID directly to the binlog client:

After a routing change, the selected replica may temporarily lag behind the checkpointed position.

3. MariaDB handling is embedded in the MySQL code path

4. Test coverage does not cover production recovery scenarios

Tests

Added or updated tests for:

  • MariaDB domain-based comparison and containment.
  • Restored GTID correction against the target server.
  • Binlog offset serialization and recovery.

Verified with JDK 11, Docker, and MariaDB 11.4:

mvn test \
  -pl flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc \
  -Dtest=MariaDbDialectITCase,MariaDbDialectSavepointITCase \
  -Dspotless.check.skip=true \
  -Drat.skip=true

@leosanqing

Copy link
Copy Markdown
Author

@yuxiqian hi, could you help me review this PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant