[26.2] Iceberg support for decoded message keys and headers#1804
[26.2] Iceberg support for decoded message keys and headers#1804kbatuigas wants to merge 8 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
✅ Deploy Preview for redpanda-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
d77121c to
0127264
Compare
0127264 to
f2690c7
Compare
|
|
||
| == Configure key, value, and header translation | ||
|
|
||
| For Redpanda clusters version 26.2 and later, in addition to the <<supported-iceberg-modes,supported modes>>, `redpanda.iceberg.mode` also accepts a section-based syntax that lets you independently configure how Redpanda translates the record key, value, and headers into the Iceberg table. The `key_value`, `value_schema_id_prefix`, and `value_schema_latest` modes are shorthand for common combinations of these sections. |
There was a problem hiding this comment.
@wdberkeley could you help confirm that this will also be live for the next Cloud release and that we don't need to conditionalize it for self-managed only? (Not sure which Cloud SME to tag)
|
|
||
| |=== | ||
|
|
||
| By default (`layout=flat`), Redpanda places each decoded value field as a top-level column in the generated table, alongside the `redpanda` system struct. If a decoded value field is named `redpanda`, Redpanda moves it into the `redpanda` system struct as a `data` field, to avoid colliding with the record metadata column of the same name. |
There was a problem hiding this comment.
@wdberkeley I think there is a discrepancy here with the TOI, could you please confirm the behavior for the system struct?
1 participant
Description
This pull request updates the documentation to introduce and clarify new capabilities for the
redpanda.iceberg.modetopic property, allowing users to independently configure how record keys, values, and headers are translated into Iceberg tables. It also improves guidance and references for configuring these options and updates related descriptions and links.Iceberg integration enhancements:
redpanda.iceberg.mode, enabling independent configuration of key, value, and header translation, including schema-decoding and UTF-8 string options.property-overrides.jsonto explain the new configuration options and reference the improved documentation.Documentation improvements:
Resolves https://redpandadata.atlassian.net/browse/2194
Review deadline: 23 July
Page previews
Checks