AWS, Azure, Core, GCP, Spark: Add support for delimiter-based prefix listing

[tom-s-powell]

Summary

This PR extends SupportsPrefixOperations with delimited prefix listing. The new listPrefix(prefix, delimiter) method returns files directly below a prefix and common sub-prefixes grouped by the delimiter. Results are returned lazily as pages so that implementations can retain the pagination behaviour of the underlying storage service.

The supportsPrefixListingWithDelimiter(prefix, delimiter) method lets a FileIO report whether it supports a specific delimiter. The new operation has default implementations, so existing SupportsPrefixOperations implementations remain compatible.

Delimited prefix listing is implemented for S3, GCS, Azure Data Lake Storage, and Hadoop FileIO. Wrapper implementations, including ResolvingFileIO and EncryptingFileIO, delegate the capability to their underlying FileIO.

Motivation

When prefix_listing=true, remove_orphan_files currently lists all files from the Spark driver on one thread. For tables with many files, this can create a driver bottleneck and increase driver memory use.

The Hadoop listing path already limits driver-side recursive listing. It discovers directories to a configured depth and then distributes the remaining directory listings across Spark executors. This PR applies the same approach to prefix-based listing.

I've be happy to split the PRs if preferred but figured it would be useful to understand the full scope of change.

Spark behaviour

The new prefix_listing_max_seed_depth argument controls how many levels of common sub-prefixes are discovered on the driver before listing is distributed to Spark executors.

When prefix_listing_max_seed_depth=0, which is the default, the table location is used as a single seed prefix. Listing therefore uses one Spark partition, but it runs on an executor instead of the driver.

When the value is greater than zero, the driver uses delimited prefix listing to discover seed prefixes up to that depth. Spark then distributes those prefixes, subject to the configured listing parallelism, so that executors can list them concurrently. Files found during seed discovery are retained and combined with the executor-side results.

This reduces driver-side work and memory use. For tables with a suitable directory or object-key layout, it also allows listing to run in parallel.

Testing

Unit tests added for all FileIO implementations and Spark RemoveOrphanFilesProcedure (all versions).

Alternatives considered

I considered introducing a SupportsDirectoryListingOperations interface or some such that extends SupportsPrefixOperations and deals specifically with listing directories that remove_orphan_files cares about. This avoided the supportsPrefixListingWithDelimiter but felt a bit more of an invasive change.

[tom-s-powell]

Raised #17594 separately to address this specific change