[FLINK-6417][connector/filesystem] Add opt-in glob enumeration for FileSource

[Cribbee]

What is the purpose of the change

Propose explicit glob selection for DataStream FileSource, addressing the compatibility concerns in FLINK-6417 and #15436. Existing enumerators remain unchanged and continue treating input paths literally. This does not change legacy FileInputFormat or SQL filesystem paths.

The new @PublicEvolving API is a proposal; the API/FLIP prerequisite in AGENTS.md still needs community guidance and approval.

Brief change log

• Enable glob expansion only through setFileEnumerator(() -> new GlobFileEnumerator(delegate)).
• Traverse from the fixed prefix and prune nonmatching directories. Support segment wildcards, character classes and standalone **; deduplicate concrete files before delegating filtering and split generation.
• Keep directory traversal policy separate from delegate file filtering. Delegate directory-specific subclass hooks are not invoked; existing default enumerators and their hooks are untouched.
• Propagate traversal and split-generation errors; only a missing glob prefix or no matches yields no files.
• In a separate prerequisite commit, fix the connector architecture rule to inspect array component types. Public arrays pass; internal arrays remain prohibited. Only resolved historical false positives were removed.

Verifying this change

Added regression coverage for literal paths, grammar, pruning, filters, overlapping inputs, error propagation, split preservation, bounded/continuous reading, TM/JM failover, and checkpoint-restored file sets.

Local Java 17 validation at a7ccd2010b6c1746e681a5ddf66756824a6772dd:

• File connector verify: 284 unit tests (6 existing migration skips), 28 integration tests; no failures.
• Real HDFS: 2 tests; native S3 with SeaweedFS 4.47: 3 tests; no failures or skips.
• Production architecture checks and rule regressions: 24 tests, also passing with violation-store updates disabled.

Official Azure build 79246 passed at this exact revision: all 13 PR CI jobs, including the six test groups, packaging/licensing, and four E2E groups. The logs confirm actual execution of the glob, HDFS/native S3, and 24 production architecture/rule tests without failures or skips.

Root Spotless, Checkstyle and license-header checks also pass locally. Local full clean verify encountered runtime temporary-storage/test-isolation failures; all 96 tests in the four affected/control classes pass both with corrected local isolation and in the normal official CI, without runtime source or assertion changes. The local full command itself is not claimed as passing.

A supplemental disposable MinIO run before the sync also verified actual bounded/continuous records. Other S3 plugins are not claimed as tested.

The architecture regression has 14 positive/negative cases, including GlobFilePathFilter[] rejection. Its public-array cases fail before the rule fix.

Does this pull request potentially affect one of the following parts:

• Dependencies: yes, test-scoped Flink dependencies for filesystem integration tests; no production dependencies added.
• Public API: yes, a new @PublicEvolving GlobFileEnumerator; approval remains pending.
• Serializers: no.
• Runtime per-record code paths: no; file enumeration only.
• Deployment or recovery: no protocol/runtime changes; discovery and restore regression tests added.
• S3 filesystem connector: integration coverage added; no S3 implementation changes.

Documentation

• New feature: yes, explicitly opt-in.
• Documented in the DataStream filesystem guide and JavaDocs, including literal syntax, filtering boundaries, error behavior, permissions and listing costs.

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Generated-by: Codex (GPT-5); Codex CLI 0.153.0

[flinkbot]

CI report:

• a7ccd20 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[Cribbee]

Hi @snuyanzin, apologies for the unsolicited ping. I noticed that you recently reviewed filesystem connector path-handling work such as #27553. Would you mind taking a quick look at this approach and, if it seems appropriate, helping assign FLINK-6417?

Unlike the earlier #15436, this PR implements glob handling in the default FileSource enumerators, preserves non-glob behavior, and uses only the generic FileSystem API. Thank you!

[snuyanzin]

Thank you for the contribution
however have read the comments in jira issue?
It tells about breaking changes and I don't see how it is mitigated here

[Cribbee]

Thanks for reviewing. I revisited FLINK-6417 and #15436, and I agree that moving the logic into FileSource does not by itself address the compatibility concerns. The current implementation still reinterprets literal paths containing glob characters and does not fully preserve subclass/filter behavior.

I propose keeping the existing enumerators unchanged and making glob expansion explicitly opt-in via setFileEnumerator(...). The implementation would also need to avoid traversing unrelated subtrees and handle filtering and traversal errors correctly.

Do you think this is the right direction for FileSource? I’ll also address the architecture-test failure

[Cribbee]

Updated in a7ccd20: the existing enumerators are unchanged, and glob expansion is explicitly opt-in through setFileEnumerator(...). Traversal is prefix-pruned; the delegate receives unique concrete files for filtering and splitting. Directory-filter and subclass-hook boundaries are documented and tested.

Local connector verify, HDFS/native S3 integration tests, and production architecture checks pass. The original internal-API dependency is gone; a separate commit fixes the architecture rule's public-array false positives while retaining internal-array checks. Full CI is still pending.

This adds a @PublicEvolving API, so the FLIP prerequisite in AGENTS.md remains open. Could you advise on the FLIP scope and whether the architecture-rule fix should be tracked separately? Thank you.

[Cribbee]

@flinkbot run azure