[Iceberg CDC] Add Changelog readers and update resolver

[ahmedabu98]

Adds read transforms that consume planned Iceberg changelog tasks and turn them into Beam CDC rows.

ReadFromChangelogs reads task batches produced by ChangelogScanner:

• Unidirectional task batches are read and emitted directly as INSERT or DELETE rows
• Large bidirectional task batches are read with full rows, filtered by primary-key overlap range, and routed into keyed insert/delete streams for downstream update resolution.
  • Records outside the overlap range are safely emitted directly because they cannot match an opposing change.

LocalResolveDoFn reads and resolves small bidirectional task batches in memory.

CdcResolver centralizes the logic for reconciling deletes and inserts for the same primary key. It emits changed pairs as UPDATE_BEFORE / UPDATE_AFTER, and leaves unmatched rows as DELETE or INSERT. Duplicate no-op pairs with identical non-PK fields are dropped.

Part of #38831

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces the core infrastructure for reading and resolving Iceberg CDC changelogs within Apache Beam. It implements a routing mechanism that distinguishes between unidirectional and bidirectional changelog tasks, ensuring that records are either emitted directly or routed for update resolution based on primary key overlap. By centralizing the reconciliation logic, this change enables consistent handling of CDC data, including the identification of update pairs and the filtering of redundant operations.

Highlights

• Read Transform Implementation: Added ReadFromChangelogs transform to process and route Iceberg changelog tasks based on their type and primary key overlap.
• CDC Resolution Logic: Implemented CdcResolver to reconcile insert and delete operations into update events, including support for Copy-on-Write deduplication.
• In-Memory Processing: Added LocalResolveDoFn to efficiently resolve small bidirectional changelog batches entirely in memory.
• Configuration Enhancements: Updated IcebergScanConfig to support watermark columns and added robust validation for CDC-specific configurations.

New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces support for Iceberg Change Data Capture (CDC) processing in Apache Beam, adding classes like CdcResolver, CdcRowDescriptor, LocalResolveDoFn, OverlapRange, and ReadFromChangelogs to handle bi-directional changelog resolution, Copy-on-Write deduplication, and primary key overlap detection. The review feedback highlights several critical issues: a potential ClassCastException in IcebergUtils when casting nested StructLike objects directly to Record, and multiple schema mismatches in OverlapRange, ReadFromChangelogs, and LocalResolveDoFn where projectors are constructed without accounting for metadata columns, which would cause position mismatches during record processing.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[ahmedabu98]

R: @talatuyarer

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

[talatuyarer]

On a table with a non-PK fixed column, a copy-on-write rewrite's delete/insert pair never matches in the hash index, so unchanged rows are emitted as spurious UPDATE_BEFORE/UPDATE_AFTER pairs.

Can we derive both hash and equals from one mechanism

[ahmedabu98]

Can we derive both hash and equals from one mechanism

We could build a single content-based key per overlapping record and do a equals/hashCode for each lookup, but that would be less performant

Correct that the old code was problematic for fixed types cuz Objects.hashCode() would be identity based for byte[]. I covered this by making the hash logic array-aware. This way, we only check deep-equality on a hash collision. I also added tests that a CoW pair with a non-PK fixed column is recognized and dropped.

I'd rather keep it this way cuz it's more performant, but let me know what you think.

[talatuyarer]

Consider noting that while records are buffered based on the 128MB compressed SPLIT_SIZE, the current implementation lacks an inflation factor for decoded data. With no safety check for the actual memory footprint in LocalResolveDoFn, a single process() call can exceed 500MB of heap usage. Furthermore, since the overlap bounds can be null when metrics are missing, the scanner defaults to buffering everything, which risks OOM errors.

[ahmedabu98]

I added a scaling factor in ChangelogScanner to make a rough estimate of the decoded bytes. This reduces the byte size threshold of batches getting routed to LocalResolveDoFn, so we won't see big batches making it there. Should help us avoid OOMs, and we can adjust the factor when we see how it's handled with real usage