Iceberg: correctly retry commit on conflict

[DAlperin]

Fixes #11229. Paired with #35397 for tests.

[cursor]

PR Summary

Medium Risk
Changes Iceberg sink commit/retry behavior and fencing checks, which can affect exactly-once progress and data-loss prevention if incorrect. Scope is localized but touches core commit path and conflict handling.

Overview
Fixes Iceberg sink commit retries to correctly rebuild and commit transactions against the latest table metadata when conflicts occur.

The commit path now retries with carried-forward table state (retry_async_with_state), reloads the table on both commit conflicts and failed apply, and adds an additional fencing check that aborts if a newer mz-sink-version is detected while retrying.

^{Written by Cursor Bugbot for commit 940e235. This will update automatically on new commits. Configure here.}

[github-actions]

Thanks for opening this PR! Here are a few tips to help make the review process smooth for everyone.

PR title guidelines

• Use imperative mood: "Fix X" not "Fixed X" or "Fixes X"
• Be specific: "Fix panic in catalog sync when controller restarts" not "Fix bug" or "Update catalog code"
• Prefix with area if helpful: compute: , storage: , adapter: , sql:

Pre-merge checklist

• The PR title is descriptive and will make sense in the git log.
• This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
• If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.
• This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
• If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
• If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).

[def-]

Thanks, rebased the test on your PR: https://buildkite.com/materialize/nightly/builds/15550
Edit: Green!

[DAlperin]

bugbot run

[cursor]

Duplicated error message in apply failure path

Low Severity

The action.apply(tx) result is first wrapped with .context("Failed to apply data file addition to iceberg table transaction"), then in the Err branch, the resulting error e is formatted into a new anyhow! with the exact same prefix message. Since Display on the anyhow::Error shows the outermost context, the final error message will read "Failed to apply data file addition to iceberg table transaction: Failed to apply data file addition to iceberg table transaction: <original error>". Either the .context() call or the custom anyhow! message is redundant.

 

[antiguru]

We have a wrapper type in ore!

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{Bugbot Autofix is ON, but it could not run because the branch was deleted or merged before autofix could start.}