Add datafusion.execution.soft_max_bytes_per_output_file config parameter

[Tpt]

Rationale for this change

Allows to roughly limit the size of parquet files when the number of rows is a hard-to-use estimator.
This is especially useful when a column contains potentially large blobs.

What changes are included in this PR?

• Adds the config parameter
• Adds an implementation aligned with datafusion.execution.soft_max_rows_per_output_file config parameter
• Adds a test

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v55.0.0 (current)
       Built [  60.402s] (current)
     Parsing datafusion v55.0.0 (current)
      Parsed [   0.032s] (current)
    Building datafusion v55.0.0 (baseline)
       Built [  54.283s] (baseline)
     Parsing datafusion v55.0.0 (baseline)
      Parsed [   0.033s] (baseline)
    Checking datafusion v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.938s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 118.034s] datafusion
    Building datafusion-common v55.0.0 (current)
       Built [  31.144s] (current)
     Parsing datafusion-common v55.0.0 (current)
      Parsed [   0.057s] (current)
    Building datafusion-common v55.0.0 (baseline)
       Built [  30.824s] (baseline)
     Parsing datafusion-common v55.0.0 (baseline)
      Parsed [   0.059s] (baseline)
    Checking datafusion-common v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   1.013s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---

Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field ExecutionOptions.soft_max_bytes_per_output_file in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:894

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  64.690s] datafusion-common
    Building datafusion-datasource v55.0.0 (current)
       Built [  40.329s] (current)
     Parsing datafusion-datasource v55.0.0 (current)
      Parsed [   0.029s] (current)
    Building datafusion-datasource v55.0.0 (baseline)
       Built [  39.492s] (baseline)
     Parsing datafusion-datasource v55.0.0 (baseline)
      Parsed [   0.029s] (baseline)
    Checking datafusion-datasource v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.365s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  81.746s] datafusion-datasource
    Building datafusion-sqllogictest v55.0.0 (current)
       Built [  89.732s] (current)
     Parsing datafusion-sqllogictest v55.0.0 (current)
      Parsed [   0.022s] (current)
    Building datafusion-sqllogictest v55.0.0 (baseline)
       Built [  91.359s] (baseline)
     Parsing datafusion-sqllogictest v55.0.0 (baseline)
      Parsed [   0.022s] (baseline)
    Checking datafusion-sqllogictest v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.126s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 184.720s] datafusion-sqllogictest

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.64%. Comparing base (20d1c56) to head (94e622b).

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #24924   +/-   ##
=======================================
  Coverage   81.64%   81.64%           
=======================================
  Files        1123     1123           
  Lines      410248   410273   +25     
  Branches   410248   410273   +25     
=======================================
+ Hits       334940   334967   +27     
+ Misses      55617    55614    -3     
- Partials    19691    19692    +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.