Skip to content

Improve Controller ingestFromURI filesystem validation - #19238

Draft
xiangfu0 wants to merge 1 commit into
apache:masterfrom
xiangfu0:xiangfu0/ingest-uri-local-fs-hardening-master
Draft

Improve Controller ingestFromURI filesystem validation#19238
xiangfu0 wants to merge 1 commit into
apache:masterfrom
xiangfu0:xiangfu0/ingest-uri-local-fs-hardening-master

Conversation

@xiangfu0

Copy link
Copy Markdown
Contributor

Summary

  • build on the existing default-off ingestFromURI local-filesystem policy by resolving and validating the exact filesystem before ingestion work starts
  • reject local path, subclass, alias, and delegate forms while keeping configured and request-provided remote filesystems working
  • keep request filesystem lifecycle ownership explicit, return generic API errors, and document the compatibility opt-in
  • ensure independently created Hadoop filesystem adapters own the clients they close

Existing behavior on master

Master already contains dd6520c7267 / #18660, which introduced the default-off setting and the initial direct URI and class checks. This change builds on that implementation to cover remaining delegate, lifecycle, ordering, and error-handling cases.

Behavior before this change

Filesystem selection could happen during copying, and endpoint-provided implementations were registered in shared factory state. Rejected requests could also create Controller staging directories before source validation.

Testing

  • PinotFSFactoryTest
  • ControllerConfTest
  • FileIngestionHelperTest
  • PinotIngestionRestletResourceStatelessTest
  • HadoopPinotFSTest
  • Spotless, Checkstyle, and license checks for pinot-spi, pinot-controller, and pinot-hdfs
  • affected-module test compilation with compiler warnings enabled

@codecov-commenter

codecov-commenter commented Aug 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.73%. Comparing base (cb89c0e) to head (f02926f).

❗ There is a different number of reports uploaded between BASE (cb89c0e) and HEAD (f02926f). Click for more details.

HEAD has 14 uploads less than BASE
Flag BASE (cb89c0e) HEAD (f02926f)
unittests 2 1
java-25 4 1
temurin 4 1
lane-a 1 0
integration 2 0
integration2 2 0
lane-b 1 0
unittests2 1 0
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #19238      +/-   ##
============================================
- Coverage     66.97%   57.73%   -9.25%     
+ Complexity     1417        1    -1416     
============================================
  Files          3453     2656     -797     
  Lines        218858   159139   -59719     
  Branches      34787    26098    -8689     
============================================
- Hits         146591    91873   -54718     
+ Misses        60556    59473    -1083     
+ Partials      11711     7793    -3918     
Flag Coverage Δ
integration ?
integration2 ?
java-25 57.73% <100.00%> (-9.25%) ⬇️
lane-a ?
lane-b ?
temurin 57.73% <100.00%> (-9.25%) ⬇️
unittests 57.73% <100.00%> (-9.25%) ⬇️
unittests1 57.73% <100.00%> (+0.01%) ⬆️
unittests2 ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@xiangfu0
xiangfu0 force-pushed the xiangfu0/ingest-uri-local-fs-hardening-master branch from 974e777 to f02926f Compare August 13, 2026 09:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants