Native Iceberg Write: close cleanup ownership gap during JVM handoff #6581

What is the problem the feature request solves?

As part of #5649, Comet's native Iceberg write path needs failure cleanup semantics that are safe across the entire native → JVM lifecycle.

#5618 added task-attempt cleanup for native Iceberg writes, but there is still a small ownership gap during the handoff.

Today, native AbortOnDrop is disarmed once the output batch has been constructed and yielded. The JVM only takes cleanup ownership after it reads that batch, decodes the written-file locations, and registers them with WrittenFileCleanup.

A failure between those two points can therefore leave an uncommitted data file without an active cleanup owner.

This is also relevant to #6240. Native merge-on-read writes will extend the executor-side writer to produce both data files and position-delete files, so the write lifecycle needs a reliable cleanup ownership contract before the MOR path builds on top of it.

Describe the potential solution

Keep native cleanup armed until the JVM explicitly completes the handoff:

1. Native writes and tracks every generated file location.
2. Native yields the output batch while keeping AbortOnDrop armed.
3. The JVM reads the locations and registers them with WrittenFileCleanup.
4. The JVM performs the final EOF poll on the native stream.
5. That EOF poll acts as the acknowledgement that ownership has transferred, allowing native cleanup to disarm.

There is intentionally a short period where native and JVM cleanup overlap. Both paths are best-effort, so attempting to delete the same uncommitted file twice is safe and preferable to an ownership gap.

Additional context

Related

• [EPIC] Production-quality native Iceberg writes #5649 — [EPIC] Production-quality native Iceberg writes
• Support Native Iceberg MOR write #6240 — Support Native Iceberg MOR write
• Failure-injection tests for native Iceberg writes #5646 — Failure-injection tests for native Iceberg writes
• Native Iceberg write: clean up task-attempt data files on task failure #5618 — Native Iceberg write: clean up task-attempt data files on task failure — this is a follow-up for the remaining handoff window
• Remove orphaned files in comet iceberg split writers #5277 — Remove orphaned files in Comet Iceberg split writers
• Accelerate row-level MERGE / UPDATE / DELETE plans (MergeRowsExec, ReplaceDataExec, WriteDeltaExec) #5122 — Accelerate row-level MERGE / UPDATE / DELETE plans
• feat: add native support for MergeRowsExec (row-level MERGE INTO) #5318 — Native MergeRowsExec support

For the MOR work specifically, #5122 / #5318 cover the row-level execution side while #6240 covers the native WriteDelta path. This issue hardens the underlying native writer lifecycle that those paths will rely on.