[SPARK-58063][SQL] Add column pruning for BIN BY

[vranes]

What changes were proposed in this pull request?

This PR adds column pruning to the logical BinBy operator, so a Project over BIN BY trims fully-unused pass-through columns from the operator's child.

A ColumnPruning arm for Project(_, b: BinBy) rewrites the child via prunedChild, keeping the range and DISTRIBUTE inputs the kernel reads (requiredAttrs = p.references -- b.producedAttributes ++ b.references):

case p @ Project(_, b: BinBy)
    if !b.child.outputSet.subsetOf(p.references -- b.producedAttributes ++ b.references) =>
  val requiredAttrs = p.references -- b.producedAttributes ++ b.references
  p.copy(child = b.copy(child = prunedChild(b.child, requiredAttrs)))

This is the plain prunedChild form used by Expand / MergeRows, and it adds no field to BinBy. It is enabled by the produced-attributes shape from SPARK-57858: the scaled DISTRIBUTE columns and appended columns are produced attributes with fresh ExprIds, so BinBy.references lists exactly the child inputs the kernel reads, and prunedChild can never prune those away.

An earlier revision followed the Generate shape: a unrequiredChildIndex: Seq[Int] field plus a requiredChildOutput lazy val and an output override, so the range columns could stay in the child (read for bin math) while dropping from output. Per review, that was dropped: the set the field can ever exclude is provably {rangeStart, rangeEnd} -- p.references (at most two fixed-width timestamps, and only when the range columns are not selected, which the outer Project trims anyway), so the field bought at most ~16 bytes/row and added positional operator surface. The Generate analogy does not transfer because Generate excludes generator.references, which are arbitrary-width. BinBy.output stays child.output.map(replace) ++ appendedAttributes.

Why are the changes needed?

BIN BY is row-multiplying, so forwarding unused pass-through columns through the operator wastes work proportional to the fan-out. Pruning them at the child, as Generate / Expand already do, avoids carrying columns the query does not use.

Does this PR introduce any user-facing change?

No. BIN BY is gated off by default (spark.sql.binByRelationOperator.enabled, SPARK-57440). This is an optimizer-only change that does not alter query results.

How was this patch tested?

• ColumnPruningSuite: a Project over BinBy prunes the unused pass-through column while keeping the range and DISTRIBUTE inputs in the child. Uses a new binBy test DSL helper (dsl/package.scala, alongside generate).
• ResolveBinBySuite: existing analyzer coverage, green (the pruning arm is optimizer-only; BinBy is unchanged from master).

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Anthropic)

[cloud-fan]

0 blocking, 1 non-blocking, 0 nits. Correct and result-safe; one non-blocking design suggestion on the pruning mechanism.

Design / architecture (1)

• Optimizer.scala:1139: unrequiredChildIndex machinery buys only <=16 bytes/row and the Generate analogy doesn't transfer to BinBy — see inline

Verification

Traced the Project(_, BinBy) rewrite: the pruning Project (prunedChild) keeps every kernel-read input (requiredAttrs ⊇ b.references), and the unrequired set reduces to {rangeStart, rangeEnd} -- p.references — DISTRIBUTE columns are gated out, so no kernel input is ever pruned. Result-equivalent for empty / single / many-row input, and the synthesized Project forwards child attributes with unchanged nullability/dataType/exprId (no metadata mismatch). The rewrite is correct; the finding below is a simplification, not a correctness concern.