diff --git a/ticdc/ticdc-architecture.md b/ticdc/ticdc-architecture.md index a4d6a237a99d8..2ef96f474d9fe 100644 --- a/ticdc/ticdc-architecture.md +++ b/ticdc/ticdc-architecture.md @@ -129,26 +129,16 @@ In the TiCDC classic architecture, DDL replication operations are strictly seria ### Changes in DDL replication behavior -- The classic TiCDC architecture does not support DDLs that swap table names (for example, `RENAME TABLE a TO c, b TO a, c TO b;`). The new architecture supports such DDLs. +TiCDC's new and classic architectures differ in the replication behavior of `RENAME` DDLs as follows: -- The new architecture unifies and simplifies the filtering rules for `RENAME` DDLs. +- **Swapping table names**: For DDLs that use an intermediate table name to swap two table names (for example, `RENAME TABLE a TO c, b TO a, c TO b;`), the classic architecture does not support replication, while the new architecture does. During replication, all interdependent renaming steps must be preserved. +- **Table filtering**: In both architectures, whether to replicate a `RENAME` statement is determined by whether the old table name matches the filter rule. If the old table name does not match but the new table name does, the changefeed reports an error and stops replicating. For multi-table `RENAME`, the classic architecture also requires the target database to be within the replication scope, while the new architecture evaluates each renaming step separately. For example, when `filter.rules = ['test.t*']`, the new architecture can replicate `RENAME TABLE test.t1 TO ignore.t1, test.t2 TO test.t22;` and then stop replicating the renamed `ignore.t1`. For more information, see [DDL replication considerations for renaming tables](/ticdc/ticdc-ddl.md#ddl-replication-considerations-for-renaming-tables). - - In the classic architecture, the filtering logic is as follows: +### Differences in DDL SQL text - - Single-table renaming: a DDL statement is replicated only if the old table name matches the filter rule. - - Multi-table renaming: a DDL statement is replicated only if both old and new table names match the filter rules. +The DDL SQL text that TiCDC replicates downstream might differ from the original upstream statement or the output of the classic architecture. The new architecture rewrites single-table rename statements as `RENAME TABLE` statements with fully qualified database and table names. When splitting multi-table rename or batch table creation events, it also adds a trailing semicolon to each resulting statement. SQL rewriting might also occur for views, cross-database `CREATE TABLE ... LIKE ...` statements, unnamed indexes, and table routing. - - In the new architecture, for both single-table and multi-table renaming, a DDL statement is replicated as long as old table names in the statement match the filter rules. - - Take the following filter rule as an example: - - ```toml - [filter] - rules = ['test.t*'] - ``` - - - In the classic architecture: for single-table renaming, such as `RENAME TABLE test.t1 TO ignore.t1`, the old table name `test.t1` matches the rule, so it will be replicated. For a multi-table renaming, such as `RENAME TABLE test.t1 TO ignore.t1, test.t2 TO test.t22;`, because the new table name `ignore.t1` does not match the rule, it will not be replicated. - - In the new TiCDC architecture: because the old table names in both `RENAME TABLE test.t1 TO ignore.t1` and `RENAME TABLE test.t1 TO ignore.t1, test.t2 TO test.t22;` match the rules, both DDL statements will be replicated. +If downstream consumers need to parse DDL SQL, or if you need to compare the outputs of the new and classic architectures during an upgrade, see [Handling DDL SQL text](/ticdc/ticdc-ddl.md#handling-of-ddl-sql-text). ## Limitations diff --git a/ticdc/ticdc-ddl.md b/ticdc/ticdc-ddl.md index 985d4dad2db3c..4a6177b16c98a 100644 --- a/ticdc/ticdc-ddl.md +++ b/ticdc/ticdc-ddl.md @@ -72,7 +72,7 @@ During the execution of the `ADD INDEX` or `CREATE INDEX` DDL operation in the d ### DDL replication considerations for renaming tables -Due to the lack of some context during the replication process, TiCDC has some constraints on the replication of `RENAME TABLE` DDLs. +TiCDC determines whether to replicate `RENAME TABLE` DDLs based on the database and table names before and after renaming and the filter rules. The following examples assume that the tables meet the replication requirements and that no additional DDL event filter rules are configured. #### Rename a single table in a DDL statement @@ -92,14 +92,20 @@ TiCDC processes this type of DDL as follows: | `RENAME TABLE test.t1 TO test.t2` | Replicate | `test.t1` matches the filter rule | | `RENAME TABLE test.t1 TO ignore.t1` | Replicate | `test.t1` matches the filter rule | | `RENAME TABLE ignore.t1 TO ignore.t2` | Ignore | `ignore.t1` does not match the filter rule | -| `RENAME TABLE test.n1 TO test.t1` | Report an error and exit the replication | The old table name `test.n1` does not match the filter rule, but the new table name `test.t1` matches the filter rule. This operation is invalid. In this case, refer to the error message for handling. | -| `RENAME TABLE ignore.t1 TO test.t1` | Report an error and exit the replication | Same reason as above. | +| `RENAME TABLE test.n1 TO test.t1` | Report an error and stop replication | The old table name `test.n1` does not match the filter rule, but the new table name `test.t1` matches the filter rule. This operation is invalid. In this case, refer to the error message for handling. | +| `RENAME TABLE ignore.t1 TO test.t1` | Report an error and stop replication | Same reason as above. | + +These rules apply to both the classic and new architectures. If the old table name matches the filter rule but the new table name does not, TiCDC replicates the `RENAME TABLE` DDL and then stops replicating subsequent data changes for that table. If the old table name does not match but the new table name matches, TiCDC reports the `CDC:ErrSyncRenameTableFailed` error because replicating only the rename cannot backfill data that was not previously replicated for that table. #### Rename multiple tables in a DDL statement -If a DDL statement renames multiple tables, TiCDC replicates the DDL statement only when the **old database name**, **old table names**, and **new database name** all match the filter rule. +When a DDL statement renames multiple tables, the new architecture evaluates each rename step according to the single-table rename rules: + +- If the old table name matches the filter rule, TiCDC replicates the corresponding rename step. +- If neither the old nor the new table name matches, TiCDC ignores that step. +- If the old table name does not match but the new table name does, the changefeed reports an error and replication stops. -In addition, TiCDC does not support the `RENAME TABLE` DDL that swaps the table names. The following is an example. +In addition, the classic architecture requires the target database of each replicated rename step to be within the replication scope. It also does not support `RENAME TABLE` statements that swap table names. Assume that the configuration file of your changefeed is as follows: @@ -110,17 +116,50 @@ rules = ['test.t*'] TiCDC processes this type of DDL as follows: -| DDL | Whether to replicate | Reason for the handling | -| --- | --- | --- | -| `RENAME TABLE test.t1 TO test.t2, test.t3 TO test.t4` | Replicate | All database names and table names match the filter rule. | -| `RENAME TABLE test.t1 TO test.ignore1, test.t3 TO test.ignore2` | Replicate | The old database name, the old table names, and the new database name match the filter rule. | -| `RENAME TABLE test.t1 TO ignore.t1, test.t2 TO test.t22;` | Report an error | The new database name `ignore` does not match the filter rule. | -| `RENAME TABLE test.t1 TO test.t4, test.t3 TO test.t1, test.t4 TO test.t3;` | Report an error | The `RENAME TABLE` DDL swaps the names of `test.t1` and `test.t3` in one DDL statement, which TiCDC cannot handle correctly. In this case, refer to the error message for handling. | +| DDL | Classic architecture | New architecture | Reason | +| --- | --- | --- | --- | +| `RENAME TABLE test.t1 TO test.t2, test.t3 TO test.t4` | Replicate | Replicate | Both the old and new database and table names match the filter rule | +| `RENAME TABLE test.t1 TO test.ignore1, test.t3 TO test.ignore2` | Replicate | Replicate | In both architectures, the old table names match the filter rule. The classic architecture additionally requires the target database to be within the replication scope. After replicating the rename, TiCDC stops replicating subsequent data changes for the renamed tables | +| `RENAME TABLE test.t1 TO ignore.t1, test.t2 TO test.t22` | Report an error | Replicate | The target database `ignore` is not within the replication scope. After replicating the rename in the new architecture, TiCDC stops replicating `ignore.t1` and continues replicating `test.t22` | +| `RENAME TABLE test.t1 TO test.t2, ignore.t1 TO ignore.t2` | Replicate only the rename step of `test.t1` | Replicate only the rename step of `test.t1` | Neither `ignore.t1` nor `ignore.t2` matches the filter rule | +| `RENAME TABLE test.t1 TO test.t2, ignore.t1 TO test.t3` | Report an error | Report an error | `ignore.t1` does not match but `test.t3` matches the filter rule | +| `RENAME TABLE test.t1 TO test.t4, test.t3 TO test.t1, test.t4 TO test.t3` | Report an error | Replicate | The two table names are swapped through the temporary table name `test.t4`, and the new architecture preserves the order of each rename step | + +In the new architecture, TiCDC coordinates the replication progress of all tables involved in a multi-table rename, but reconstructs the downstream SQL as multiple single-table `RENAME TABLE` statements. The MySQL sink executes multiple sub-statements separated by semicolons, while the Kafka and Pulsar sinks split them into single-table DDL messages. Therefore, downstream execution does not guarantee the atomicity of a single upstream multi-table rename statement. When using a temporary table name to swap table names, keep all mutually dependent steps and avoid removing any of them through filter rules. ### DDL statement considerations When executing cross-database DDL statements (such as `CREATE TABLE db1.t1 LIKE t2`) in the upstream, it is recommended that you explicitly specify all relevant database names in DDL statements (such as `CREATE TABLE db1.t1 LIKE db2.t2`). Otherwise, cross-database DDL statements might not be executed correctly in the downstream due to the lack of database name information. +### Handling of DDL SQL text + +TiCDC parses, formats, or rebuilds DDL SQL. Therefore, the SQL text replicated downstream is not guaranteed to exactly match the original upstream statement. Both the classic and new architectures normalize keyword case, identifier quotes, spaces, and some synonymous syntax, and both also handle TiDB-specific syntax for compatibility. DDLs such as `DROP TABLE`, `DROP VIEW`, and `EXCHANGE PARTITION` all have SQL rebuilding logic in both architectures. + +The new architecture might also modify DDL SQL in the following ways. Except for MySQL sink rewrites specifically noted, these changes also affect the SQL field in message protocols that support DDL. The exact output format might vary by TiCDC version. + +| DDL or scenario | SQL text handling in the new architecture | Applicable conditions | +| --- | --- | --- | +| Single-table rename | Uniformly rebuilt as `RENAME TABLE db.old_table TO db.new_table` with full database and table names | Includes `ALTER TABLE ... RENAME AS ...`, `ALTER TABLE ... RENAME TO ...`, and regular `RENAME TABLE ... TO ...`. It is also rewritten when table routing is not enabled | +| Multi-table rename | Rebuilt in the original order into multiple single-table `RENAME TABLE` statements, with each sub-statement ending with `;` | The MySQL sink receives the combined SQL; paths that output one table at a time, such as Kafka and Pulsar, split it into multiple messages | +| Batch table creation | Each split `CREATE TABLE` statement ends with `;` | The upstream batch table creation produces one DDL event containing multiple tables, and the downstream outputs it as single-table events | +| `CREATE TABLE ... LIKE ...` | Adds the missing database qualifier to the table referenced by `LIKE` | The original SQL does not qualify the database name of the referenced table, and the DDL metadata indicates that the referenced table and the target table are in different databases | +| `CREATE VIEW` and `CREATE OR REPLACE VIEW` | Rebuilds SQL using the view query stored by TiDB and the database and table metadata at the DDL time, and might add database qualifiers to table and column references, normalize identifier case, or add column aliases | When names in the view definition need to be parsed; table routing does not need to be enabled | +| Anonymous secondary indexes | Adds the upstream-generated index names for unnamed indexes in `ALTER TABLE ... ADD INDEX/KEY/...`, including multiple change operations in one `ALTER TABLE` | Processed only before the MySQL sink executes DDL on MySQL or TiDB, and does not apply to SQL in MQ messages | +| Databases, tables, or partitions whose names contain backticks | When rebuilding SQL, escapes backticks in names as two backticks | Includes scenarios such as rename, `DROP TABLE`, `DROP VIEW`, and `EXCHANGE PARTITION` that require SQL rebuilding | +| Table routing | Rewrites database names, table names, and related references according to routing rules, and regenerates SQL | `target-schema` or `target-table` is configured. For more information, see [DDL behavior of table routing](/ticdc/ticdc-table-routing.md#ddl-behavior) | + +For example, assume that the current database is `db`, table routing is not enabled, and the following rename events are not filtered. The outputs for single-table rename are as follows. Statement-ending semicolons are omitted in the table. + +| Upstream SQL | Classic architecture output | New architecture output | +| --- | --- | --- | +| `ALTER TABLE a RENAME AS b` | ``ALTER TABLE `a` RENAME AS `b` `` | ``RENAME TABLE `db`.`a` TO `db`.`b` `` | +| `ALTER TABLE a RENAME TO b` | ``ALTER TABLE `a` RENAME AS `b` `` | ``RENAME TABLE `db`.`a` TO `db`.`b` `` | +| `RENAME TABLE a TO b` | ``RENAME TABLE `a` TO `b` `` | ``RENAME TABLE `db`.`a` TO `db`.`b` `` | + +For `RENAME TABLE a TO b, c TO d`, when output as single-table events, the first SQL in the classic architecture is ``RENAME TABLE `db`.`a` TO `db`.`b` ``, and in the new architecture it is ``RENAME TABLE `db`.`a` TO `db`.`b`;``. The trailing semicolon comes from the multi-table DDL splitting in the new architecture. + +When developing downstream consumers or comparing the outputs of the classic and new architectures, parse the SQL and use the DDL type and database and table metadata to assess its meaning instead of comparing SQL text directly. Database names, table names, index names, and their letter case still need to be validated according to the specific scenario to avoid overlooking name resolution or table routing errors. + ### Notes on using event filter rules to filter DDL events If a filtered DDL statement involves table creation or deletion, TiCDC only filters out the DDL statement without affecting the replication behavior of DML statements. The following is an example. @@ -146,6 +185,8 @@ ignore-event = ["create table", "drop table", "truncate table", "rename table"] | `RENAME TABLE test.t1 TO test.t2;` | Ignore | Replicate | `test.t1` matches the event filter rule, so the `RENAME TABLE` event is ignored. The replication of DML events remains unaffected. | | `RENAME TABLE test.t1 TO test.ignore;` | Ignore | Ignore | `test.t1` matches the event filter rule, so the `RENAME TABLE` event is ignored. `test.ignore` matches the event filter rule, so both DDL and DML events are ignored. | +When using `ignore-sql` to filter multi-table rename operations, note that the matching input differs between the classic and new architectures. In the classic architecture, `ignore-sql` is matched against each SQL statement after splitting the operation into single-table DDL events. In the new architecture, `ignore-sql` is matched against the reconstructed SQL for the entire multi-table rename operation when event filter rules are applied to individual tables. If the `matcher` of the same event filter rule matches multiple tables, `ignore-sql` matching one rename step might also filter other rename steps covered by that rule. Before switching architectures, check the actual matching results of such filter rules. + > **Note:** > > - When replicating data to a database, use the event filter to filter DDL events with caution. Ensure that the upstream and downstream database schemas remain consistent during replication. Otherwise, TiCDC might report errors or cause undefined replication behavior. diff --git a/ticdc/ticdc-sink-to-kafka.md b/ticdc/ticdc-sink-to-kafka.md index a59163127df22..47c67e52f24ca 100644 --- a/ticdc/ticdc-sink-to-kafka.md +++ b/ticdc/ticdc-sink-to-kafka.md @@ -273,9 +273,13 @@ DDLs that are related to a specific table are called table-level DDLs, such as ` For example, for a dispatcher like `matcher = ['test.*'], topic = "{schema}_{table}"`, DDL events are dispatched as follows: -- If a single table is involved in the DDL event, the DDL event is sent to the corresponding topic as is. For example, for the DDL event `drop table test.table1`, the event is sent to the topic named `test_table1`. +- If a single table is involved in the DDL event, the DDL event is sent to the corresponding topic. For example, for the DDL event `drop table test.table1`, the event is sent to the topic named `test_table1`. - If multiple tables are involved in the DDL event (`rename table` / `drop table` / `drop view` may involve multiple tables), the DDL event is split into multiple events and sent to the corresponding topics. For example, for the DDL event `rename table test.table1 to test.table10, test.table2 to test.table20`, the event `rename table test.table1 to test.table10` is sent to the topic named `test_table1` and the event `rename table test.table2 to test.table20` is sent to the topic named `test_table2`. +The SQL text in DDL messages might be reformatted or reconstructed. In the new architecture, single-table `ALTER TABLE ... RENAME` statements are rewritten as `RENAME TABLE` statements with fully qualified database and table names. When this architecture splits a multi-table `RENAME TABLE` statement, each resulting SQL statement ends with a semicolon. Consumers must support these SQL forms when parsing. + +Splitting a multi-table rename does not guarantee the atomicity of the original upstream DDL statement. When swapping table names, consumers must process all related rename steps in the correct order. For more information, see [DDL replication considerations for renaming tables](/ticdc/ticdc-ddl.md#ddl-replication-considerations-for-renaming-tables) and [Handling of DDL SQL text](/ticdc/ticdc-ddl.md#handling-of-ddl-sql-text). + ### Partition dispatchers You can use `partition = "xxx"` to specify a partition dispatcher. It supports five dispatchers: `default`, `index-value`, `columns`, `table`, and `ts`. The dispatcher rules are as follows: diff --git a/ticdc/ticdc-sink-to-pulsar.md b/ticdc/ticdc-sink-to-pulsar.md index 4772afaa5528e..329b3c2d0a94d 100644 --- a/ticdc/ticdc-sink-to-pulsar.md +++ b/ticdc/ticdc-sink-to-pulsar.md @@ -310,13 +310,17 @@ DDL statements such as `ALTER TABLE` and `CREATE TABLE` that are related to a sp For example, for a `dispatchers` configuration like `matcher = ['test.*'], topic = "{schema}_{table}"`, the DDL events are despatched as follows: -- If a DDL event only involves a single table, the DDL event is dispatched to the appropriate topic as it is. For example, for the DDL event `DROP TABLE test.table1`, the event is dispatched to the topic named `test_table1`. +- If a DDL event only involves a single table, the DDL event is dispatched to the appropriate topic. For example, for the DDL event `DROP TABLE test.table1`, the event is dispatched to the topic named `test_table1`. - If a DDL event involves more than one table (`RENAME TABLE`, `DROP TABLE`, and `DROP VIEW` might all involve more than one table), the single DDL event is split into multiple ones and dispatched to appropriate topics. For example, for the DDL event `RENAME TABLE test.table1 TO test.table10, test.table2 TO test.table20`, the processing is as follows: - Dispatch the DDL event for `RENAME TABLE test.table1 TO test.table10` to a topic named `test_table1`. - Dispatch the DDL event for `RENAME TABLE test.table2 TO test.table20` to a topic named `test_table2`. +The SQL text in DDL messages might be reformatted or reconstructed. In the new architecture, single-table `ALTER TABLE ... RENAME` statements are rewritten as `RENAME TABLE` statements with fully qualified database and table names. When this architecture splits a multi-table `RENAME TABLE` statement, each resulting SQL statement ends with a semicolon. Consumers must support these SQL forms when parsing. + +Splitting a multi-table rename does not guarantee the atomicity of the original upstream DDL statement. When swapping table names, consumers must process all related rename steps in the correct order. For more information, see [DDL replication considerations for renaming tables](/ticdc/ticdc-ddl.md#ddl-replication-considerations-for-renaming-tables) and [Handling of DDL SQL text](/ticdc/ticdc-ddl.md#handling-of-ddl-sql-text). + ### Partition dispatcher Currently, TiCDC only supports consumers to consume messages using the exclusive subscription model, that is, each consumer can consume messages from all partitions in a topic.