Spark 4.1: Support parallel orphan file deletion

[aarushigupta132]

Summary

This draft adds a parallel_deletes option to the Spark 4.1 remove_orphan_files procedure. When enabled, orphan files are deleted from Spark executors rather than sequentially on the driver.

The option is disabled by default, preserving the existing driver-side deletion behavior. FileIO implementations that support bulk operations delete one Spark partition at a time; other implementations delete files individually within each partition. Orphan paths continue to flow back to the driver so the procedure can produce its existing result count and path sample.

Motivation

remove_orphan_files currently identifies orphan files with Spark but performs deletion on the driver. Even when listing and the metadata anti-join are distributed, deletion throughput is therefore limited by one node. Executor-side deletion allows the work to scale with the number of Spark partitions.

This is complementary to #17790, which distributes prefix-based file listing. The cached orphan-file dataset remains the correct input to deletion because files must not be removed until the metadata anti-join and prefix-mismatch validation have completed.

Open questions and follow-ups

This is a draft because the API and failure semantics need review. In particular:

• dry_run => true currently supplies a no-op delete function, which the parallel path bypasses. The two options must be rejected together or made safe before this is ready to merge.
• The current implementation materializes a complete Spark partition before invoking bulk deletion. It should retain bounded batches comparable to the existing DELETE_GROUP_SIZE.
• The interaction with max_concurrent_deletes and custom delete functions should be validated explicitly.
• Executor-side retries and speculative execution can cause a partition to be deleted more than once. The expected idempotency and failure-reporting behavior should be documented and tested.
• Driver-side logs become executor-side logs for the actual delete operations when this option is enabled.

Testing

Tests are still to be added for executor-side bulk and non-bulk deletion, dry-run behavior, option interactions, partition batching, and task retry behavior.

[aarushigupta132]

Closing