Skip to content

[flink][cdc] Preserve binary values in Debezium JSON and MySQL records - #10879

Open
taoran92 wants to merge 1 commit into
apache:masterfrom
taoran92:fix/cdc-json-binary-field-corruption
Open

taoran92 wants to merge 1 commit into
apache:masterfrom
taoran92:fix/cdc-json-binary-field-corruption

Conversation

@taoran92

@taoran92 taoran92 commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

Purpose

Closes #10878.

Synchronizing a MySQL BLOB can silently corrupt binary data because plain Debezium bytes values are decoded into a Java string before the sink converts that string back to bytes. For example, the 70-byte PNG in the issue becomes 92 bytes even with UTF-8 as the default charset.

Keep plain binary values as Base64 in intermediate CDC records and carry explicit per-field encoding metadata from both the Debezium JSON and MySQL parsers. Decode marked fields when converting sink rows and projecting keys, checking the decoded byte length for binary target types. Preserve existing text conversions and computed-column behavior, while keeping BIT, DECIMAL and geometry logical types on their existing conversion paths.

Tests

  • Added JSON/MySQL parser-to-row, key projection and local append-table write/read coverage for null, empty, text and arbitrary binary values, including all 256 byte values.
  • Added coverage for marked versus ordinary string fields, binary length validation, existing text target columns, case conversion, and Java/Flink Kryo serialization round trips.
  • Ran 66 targeted CDC tests: no failures, errors or skipped tests.
  • Replayed the captured snapshot and binlog events through both parsers, then wrote and read a local primary-key table: the master commit returns 92-byte invalid images; this change preserves the original 70 bytes and the PNGs decode successfully.

@taoran92
taoran92 marked this pull request as ready for review October 29, 2026 15:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] MySQL CDC corrupts BLOB data during synchronization

1 participant