Skip to content

Add DataSource/FileSource proto hooks and FileScanConfig serde#23683

Open
kumarUjjawal wants to merge 2 commits into
apache:mainfrom
kumarUjjawal:issue-23497-datasource-proto-hooks
Open

Add DataSource/FileSource proto hooks and FileScanConfig serde#23683
kumarUjjawal wants to merge 2 commits into
apache:mainfrom
kumarUjjawal:issue-23497-datasource-proto-hooks

Conversation

@kumarUjjawal

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

File scan serialization currently depends on central type downcasts in datafusion-proto.

This PR adds the foundation needed to move each data source to its own protobuf hooks while preserving the existing wire format and fallback behavior.

It unblocks the concrete source migrations tracked by #23516, #23517, and #23518.

What changes are included in this PR?

  • Add feature-gated try_to_proto hooks to DataSource and FileSource.
  • Delegate serialization from DataSourceExec through FileScanConfig to its concrete FileSource.
  • Add shared FileScanConfig protobuf encoding and decoding in the datasource crate.
  • Keep the existing central serializer as the fallback for sources that still return Ok(None).
  • Preserve the legacy protobuf wire format.
  • Add compatibility test

Are these changes tested?

Yes

Are there any user-facing changes?

This adds non-breaking, feature-gated public serialization hooks for data source and file source implementations.

@github-actions github-actions Bot added proto Related to proto crate datasource Changes to the datasource crate labels Jul 18, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.23773% with 122 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.69%. Comparing base (1734d4b) to head (bc9a8e5).

Files with missing lines Patch % Lines
...atafusion/datasource/src/file_scan_config/proto.rs 86.46% 10 Missing and 39 partials ⚠️
datafusion/proto/src/physical_plan/mod.rs 88.54% 20 Missing and 17 partials ⚠️
datafusion/datasource/src/file_scan_config/mod.rs 48.57% 35 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #23683      +/-   ##
==========================================
+ Coverage   80.67%   80.69%   +0.01%     
==========================================
  Files        1087     1088       +1     
  Lines      367541   368315     +774     
  Branches   367541   368315     +774     
==========================================
+ Hits       296529   297200     +671     
- Misses      53355    53400      +45     
- Partials    17657    17715      +58     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

datasource Changes to the datasource crate proto Related to proto crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Proto: add DataSource/FileSource try_to_proto hook + port FileScanConfig serde

2 participants