Add debezium avro protocol intro.#23270
Open
ginkgoch wants to merge 1 commit into
Open
Conversation
ti-chi-bot
Bot
added
contribution
|
Hi @ginkgoch. Thanks for your PR. I'm waiting for a pingcap member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
ti-chi-bot
Bot
added
missing-translation-status
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a new documentation file ticdc/ticdc-debezium-avro-protocol.md detailing the TiCDC Debezium Avro Protocol, including its usage, data format, configuration, and compatibility. The reviewer's feedback focuses on improving the documentation's clarity, readability, and consistency by suggesting the use of active voice, proper articles, consistent terminology (such as TopicNameStrategy), and formatting adjustments in accordance with the technical writing style guide.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Comment on lines
+3
to
+7
| TiCDC Debezium Avro protocol combines Debezium-style change event semantics with Confluent Avro wire format. | ||
|
|
||
| In this protocol, TiCDC still produces Debezium envelope fields such as `before`, `after`, `source`, `op`, and `ts_ms`, but message bytes are encoded in Avro binary and schema ids are managed by Schema Registry. | ||
|
|
||
| Compared with `protocol=debezium`, `protocol=debezium-avro` is better suited for downstream systems that already use Avro deserializers and Schema Registry. |
Contributor
There was a problem hiding this comment.
According to the technical writing style guide, we should prefer active voice and ensure consistent terminology. Let's use active voice, correct 'schema ids' to 'schema IDs', and add appropriate articles ('the') for better clarity.
Suggested change
| TiCDC Debezium Avro protocol combines Debezium-style change event semantics with Confluent Avro wire format. | |
| In this protocol, TiCDC still produces Debezium envelope fields such as `before`, `after`, `source`, `op`, and `ts_ms`, but message bytes are encoded in Avro binary and schema ids are managed by Schema Registry. | |
| Compared with `protocol=debezium`, `protocol=debezium-avro` is better suited for downstream systems that already use Avro deserializers and Schema Registry. | |
| The TiCDC Debezium Avro protocol combines Debezium-style change event semantics with the Confluent Avro wire format.\n\nIn this protocol, TiCDC still produces Debezium envelope fields such as `before`, `after`, `source`, `op`, and `ts_ms`, but TiCDC encodes message bytes in Avro binary and manages schema IDs using the Schema Registry.\n\nCompared with `protocol=debezium`, `protocol=debezium-avro` is better suited for downstream systems that already use Avro deserializers and the Schema Registry. |
References
- Prefer present tense and active voice. Avoid passive voice overuse. (link)
Comment on lines
+26
to
+28
| > **Note:** | ||
| > | ||
| > Debezium Avro uses Schema Registry topic-name subject behavior. One Kafka topic should contain data for only one table. You need to configure topic dispatchers to route each table to an independent topic. |
Contributor
There was a problem hiding this comment.
To maintain consistency with other documentation files (such as ticdc-avro-protocol.md), let's refer to the specific subject naming strategy TopicNameStrategy in backticks.
Suggested change
| > **Note:** | |
| > | |
| > Debezium Avro uses Schema Registry topic-name subject behavior. One Kafka topic should contain data for only one table. You need to configure topic dispatchers to route each table to an independent topic. | |
| > **Note:**\n>\n> Debezium Avro uses the Schema Registry `TopicNameStrategy` subject naming strategy. One Kafka topic should contain data for only one table. You need to configure topic dispatchers to route each table to an independent topic. |
References
- Use consistent terminology. Code snippets, command names, options, and paths should be in backticks. (link)
Comment on lines
+39
to
+43
| TiCDC converts a DML event into a Kafka event. The key and value are encoded as Debezium-model payloads over Avro wire format. | ||
|
|
||
| ### Wire format | ||
|
|
||
| Each Kafka message is encoded as: |
Contributor
There was a problem hiding this comment.
Let's rewrite this section to use active voice and improve readability.
Suggested change
| TiCDC converts a DML event into a Kafka event. The key and value are encoded as Debezium-model payloads over Avro wire format. | |
| ### Wire format | |
| Each Kafka message is encoded as: | |
| TiCDC converts a DML event into a Kafka event. TiCDC encodes the key and value as Debezium-model payloads over the Avro wire format.\n\n### Wire format\n\nTiCDC encodes each Kafka message as follows: |
References
- Prefer present tense and active voice. Avoid passive voice overuse. (link)
Comment on lines
+77
to
+83
| When TiCDC converts this key schema to Avro and registers it in Schema Registry: | ||
|
|
||
| - `connect.name` keeps the full logical name: `{{ClusterID}}.{{SchemaName}}.{{TableName}}Key` | ||
| - Avro `name` becomes: `{{TableName}}Key` | ||
| - Avro `namespace` becomes: `{{ClusterID}}.{{SchemaName}}` | ||
|
|
||
| All parts are sanitized to Avro-compatible identifiers. |
Contributor
There was a problem hiding this comment.
Let's use active voice and add appropriate articles to improve the clarity of the record naming rules.
Suggested change
| When TiCDC converts this key schema to Avro and registers it in Schema Registry: | |
| - `connect.name` keeps the full logical name: `{{ClusterID}}.{{SchemaName}}.{{TableName}}Key` | |
| - Avro `name` becomes: `{{TableName}}Key` | |
| - Avro `namespace` becomes: `{{ClusterID}}.{{SchemaName}}` | |
| All parts are sanitized to Avro-compatible identifiers. | |
| When TiCDC converts this key schema to Avro and registers it in the Schema Registry:\n\n- `connect.name` keeps the full logical name: `{{ClusterID}}.{{SchemaName}}.{{TableName}}Key`\n- The Avro `name` becomes: `{{TableName}}Key`\n- The Avro `namespace` becomes: `{{ClusterID}}.{{SchemaName}}`\n\nTiCDC sanitizes all parts to Avro-compatible identifiers. |
References
- Prefer present tense and active voice. Avoid passive voice overuse. (link)
Comment on lines
+172
to
+184
| For value-related records, TiCDC applies the same conversion rule: | ||
|
|
||
| - Envelope `connect.name`: `{{ClusterID}}.{{SchemaName}}.{{TableName}}Envelope` | ||
| - Envelope Avro `name`: `{{TableName}}Envelope` | ||
| - Envelope Avro `namespace`: `{{ClusterID}}.{{SchemaName}}` | ||
|
|
||
| Nested row records follow the same pattern (for example, `{{TableName}}` under the same namespace). | ||
|
|
||
| Debezium Avro encodes DML event types as follows: | ||
|
|
||
| - For insert events, `op = "c"`, `before = null`, and `after` contains new row data. | ||
| - For update events, `op = "u"`, and `after` contains updated row data. If old value output is enabled, `before` is included. | ||
| - For delete events, `op = "d"`, `after = null`, and `before` contains deleted row data. |
Contributor
There was a problem hiding this comment.
Let's use active voice, add appropriate articles, and capitalize the event types ('Insert', 'Update', 'Delete') to maintain consistency with the rest of the TiCDC documentation.
Suggested change
| For value-related records, TiCDC applies the same conversion rule: | |
| - Envelope `connect.name`: `{{ClusterID}}.{{SchemaName}}.{{TableName}}Envelope` | |
| - Envelope Avro `name`: `{{TableName}}Envelope` | |
| - Envelope Avro `namespace`: `{{ClusterID}}.{{SchemaName}}` | |
| Nested row records follow the same pattern (for example, `{{TableName}}` under the same namespace). | |
| Debezium Avro encodes DML event types as follows: | |
| - For insert events, `op = "c"`, `before = null`, and `after` contains new row data. | |
| - For update events, `op = "u"`, and `after` contains updated row data. If old value output is enabled, `before` is included. | |
| - For delete events, `op = "d"`, `after = null`, and `before` contains deleted row data. | |
| For value-related records, TiCDC applies the same conversion rules:\n\n- The envelope `connect.name` becomes: `{{ClusterID}}.{{SchemaName}}.{{TableName}}Envelope`\n- The envelope Avro `name` becomes: `{{TableName}}Envelope`\n- The envelope Avro `namespace` becomes: `{{ClusterID}}.{{SchemaName}}`\n\nNested row records follow the same pattern (for example, `{{TableName}}` under the same namespace).\n\nDebezium Avro encodes DML event types as follows:\n\n- For Insert events, `op = "c"`, `before = null`, and `after` contains new row data.\n- For Update events, `op = "u"`, and `after` contains updated row data. If `debezium-output-old-value` is enabled, `before` is included.\n- For Delete events, `op = "d"`, `after = null`, and `before` contains deleted row data. |
References
- Use consistent terminology. (link)
Comment on lines
+190
to
+197
| | Parameter | Description | Default | | ||
| | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------- | --------- | | ||
| | `protocol=debezium-avro` | Enables the Debezium Avro protocol | N/A | | ||
| | `schema-registry` | Confluent Schema Registry URL | Required | | ||
| | `enable-tidb-extension` | Adds TiDB-specific metadata such as `tidb_type`, and enables internal watermark / DDL encoding paths when used with `avro-enable-watermark` | `false` | | ||
| | `avro-decimal-handling-mode` | Decimal handling mode, `precise` or `string` | `precise` | | ||
| | `avro-bigint-unsigned-handling-mode` | Unsigned BIGINT handling mode, `long` or `string` | `long` | | ||
| | `avro-enable-watermark` | Enables watermark / DDL messages for internal testing | `false` | |
Contributor
There was a problem hiding this comment.
Let's improve the parameter descriptions in the table by using active voice, adding articles, and ending descriptions with periods for consistency.
Suggested change
| | Parameter | Description | Default | | |
| | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------- | --------- | | |
| | `protocol=debezium-avro` | Enables the Debezium Avro protocol | N/A | | |
| | `schema-registry` | Confluent Schema Registry URL | Required | | |
| | `enable-tidb-extension` | Adds TiDB-specific metadata such as `tidb_type`, and enables internal watermark / DDL encoding paths when used with `avro-enable-watermark` | `false` | | |
| | `avro-decimal-handling-mode` | Decimal handling mode, `precise` or `string` | `precise` | | |
| | `avro-bigint-unsigned-handling-mode` | Unsigned BIGINT handling mode, `long` or `string` | `long` | | |
| | `avro-enable-watermark` | Enables watermark / DDL messages for internal testing | `false` | | |
| | Parameter | Description | Default |\n| ------------------------------------ | --------------------------------------------------------------------------------------------------------------------- | --------- |\n| `protocol=debezium-avro` | Enables the Debezium Avro protocol. | N/A |\n| `schema-registry` | The Confluent Schema Registry URL. | Required |\n| `enable-tidb-extension` | Adds TiDB-specific metadata such as `tidb_type`, and enables internal watermark and DDL encoding paths when used with `avro-enable-watermark`. | `false` |\n| `avro-decimal-handling-mode` | The decimal handling mode, which can be `precise` or `string`. | `precise` |\n| `avro-bigint-unsigned-handling-mode` | The unsigned `BIGINT` handling mode, which can be `long` or `string`. | `long` |\n| `avro-enable-watermark` | Enables watermark and DDL messages for internal testing. | `false` | |
References
- Clarity, simplicity, completeness, and readability. (link)
Comment on lines
+199
to
+207
| The `debezium-output-old-value` setting is controlled by sink configuration and decides whether update events include old row values in `before`. | ||
|
|
||
| `enable-tidb-extension` is optional in normal Debezium Avro scenarios. Base Debezium envelope fields are available without it. | ||
|
|
||
| ### Constraints | ||
|
|
||
| - Debezium Avro requires a Schema Registry endpoint. | ||
| - A Kafka topic should carry one table schema. | ||
| - `force-replicate` must be disabled when using Avro or Debezium Avro. |
Contributor
There was a problem hiding this comment.
Let's use active voice and second person ('you') to make the constraints and configuration descriptions clearer and more direct.
Suggested change
| The `debezium-output-old-value` setting is controlled by sink configuration and decides whether update events include old row values in `before`. | |
| `enable-tidb-extension` is optional in normal Debezium Avro scenarios. Base Debezium envelope fields are available without it. | |
| ### Constraints | |
| - Debezium Avro requires a Schema Registry endpoint. | |
| - A Kafka topic should carry one table schema. | |
| - `force-replicate` must be disabled when using Avro or Debezium Avro. | |
| The sink configuration controls the `debezium-output-old-value` setting, which determines whether update events include old row values in `before`.\n\n`enable-tidb-extension` is optional in normal Debezium Avro scenarios. You can access the base Debezium envelope fields without enabling it.\n\n### Constraints\n\n- Debezium Avro requires a Schema Registry endpoint.\n- A Kafka topic should carry only one table schema.\n- You must disable `force-replicate` when using the Avro or Debezium Avro protocol. |
References
- Write in second person ('you') when addressing users. Avoid passive voice overuse. (link)
Comment on lines
+209
to
+217
| ## DDL events and schema changes | ||
|
|
||
| Debezium Avro does not rely on downstream DDL event consumption. When table schema changes, TiCDC generates and registers new schemas through Schema Registry during DML processing. | ||
|
|
||
| If schema compatibility checks fail in Schema Registry, changefeed enters an error state. | ||
|
|
||
| For operational details, read: | ||
|
|
||
| - [TiCDC Avro Protocol - DDL events and schema changes](/ticdc/ticdc-avro-protocol.md#ddl-events-and-schema-changes) |
Contributor
There was a problem hiding this comment.
Let's add appropriate articles and use 'see' instead of 'read' for referring to other sections, which is more standard in technical documentation.
Suggested change
| ## DDL events and schema changes | |
| Debezium Avro does not rely on downstream DDL event consumption. When table schema changes, TiCDC generates and registers new schemas through Schema Registry during DML processing. | |
| If schema compatibility checks fail in Schema Registry, changefeed enters an error state. | |
| For operational details, read: | |
| - [TiCDC Avro Protocol - DDL events and schema changes](/ticdc/ticdc-avro-protocol.md#ddl-events-and-schema-changes) | |
| ## DDL events and schema changes\n\nDebezium Avro does not rely on downstream DDL event consumption. When a table schema changes, TiCDC generates and registers new schemas through the Schema Registry during DML processing.\n\nIf schema compatibility checks fail in the Schema Registry, the changefeed enters an error state.\n\nFor operational details, see:\n\n- [TiCDC Avro Protocol - DDL events and schema changes](/ticdc/ticdc-avro-protocol.md#ddl-events-and-schema-changes) |
References
- Clarity, simplicity, completeness, and readability. (link)
Comment on lines
+235
to
+241
| ## Compatibility | ||
|
|
||
| Schema evolution compatibility is determined by Schema Registry policy. Incompatible schema changes can stop the changefeed. | ||
|
|
||
| Reference: | ||
|
|
||
| - [TiCDC Avro Protocol - Compatibility](/ticdc/ticdc-avro-protocol.md#compatibility) No newline at end of file |
Contributor
There was a problem hiding this comment.
Let's use active voice and add a newline at the end of the file to follow standard POSIX file formatting.
Suggested change
| ## Compatibility | |
| Schema evolution compatibility is determined by Schema Registry policy. Incompatible schema changes can stop the changefeed. | |
| Reference: | |
| - [TiCDC Avro Protocol - Compatibility](/ticdc/ticdc-avro-protocol.md#compatibility) | |
| ## Compatibility\n\nThe Schema Registry policy determines the schema evolution compatibility. Incompatible schema changes can cause the changefeed to stop.\n\nReference:\n\n- [TiCDC Avro Protocol - Compatibility](/ticdc/ticdc-avro-protocol.md#compatibility)\n |
References
- Clarity, simplicity, completeness, and readability. Avoid passive voice overuse. (link)
Comment on lines
+11
to
+13
| When you use Kafka as the downstream sink, specify `protocol=debezium-avro` in `sink-uri`, and provide the Schema Registry endpoint. | ||
|
|
||
| The configuration example is as follows: |
Contributor
There was a problem hiding this comment.
Let's improve the clarity and flow by adding appropriate articles and using active phrasing.
Suggested change
| When you use Kafka as the downstream sink, specify `protocol=debezium-avro` in `sink-uri`, and provide the Schema Registry endpoint. | |
| The configuration example is as follows: | |
| When you use Kafka as the downstream sink, specify `protocol=debezium-avro` in the `sink-uri`, and provide the Schema Registry endpoint.\n\nThe following is a configuration example: |
References
- Clarity, simplicity, completeness, and readability. (link)
qiancai
added
needs-cherry-pick-master
ti-chi-bot
Bot
removed
the
missing-translation-status
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First-time contributors' checklist
What is changed, added or deleted? (Required)
We recently added ticdc debezium-avro protocol support, need to add the document for it.
Which TiDB version(s) do your changes apply to? (Required)
Tips for choosing the affected version(s):
By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.
For details, see tips for choosing the affected versions.
What is the related PR or file link(s)?
Do your changes match any of the following descriptions?