Support Spark-version-specific expectations in datafusion-spark sqllogictest files

[andygrove]

Is your feature request related to a problem or challenge?

The sqllogictest files under datafusion/sqllogictest/test_files/spark/ hard-code
their expected results, so each file encodes the behavior of exactly one Spark
version. There is currently no way to express that a function's behavior changed
between Spark versions.

This matters because Spark's built-in expressions do change across releases. A
few of the axes that move: ANSI mode became the default in Spark 4.0, collation
support was added to many string functions in 4.0, and individual expressions
have picked up new accepted input types or altered edge-case handling between
3.5 and 4.2.

Today the only options when we discover such a divergence are:

1. Assert one version's behavior and silently be wrong about the others.
2. Leave the case untested entirely.

Neither leaves a durable record, and neither lets CI catch a regression on the
version we did not pick.

Concretely, a test author who finds that f(x) returns NULL on Spark 3.5 and
throws on Spark 4.2 has nowhere to put that knowledge except a comment.

Describe the solution you'd like

A way to scope a .slt file, or an individual query within one, to a Spark
version or version range.

There is prior art worth borrowing from. Apache DataFusion Comet solves the
analogous problem for its SQL file tests with a file-level directive:

-- MinSparkVersion: 3.5

which skips the whole file when running against an older Spark. See
docs/source/contributor-guide/sql-file-tests.md in apache/datafusion-comet.

Two shapes seem plausible here, and I do not have a strong preference yet:

Option A: file-level directive. Mirror Comet directly, e.g. a
# MinSparkVersion: 4.0 comment directive honored by the sqllogictest runner.
Simple, but coarse: a single divergent query forces the whole file to be scoped,
or the divergent cases to be split into a separate file.

Option B: per-query labels. sqllogictest already supports skipif <label> /
onlyif <label>. Exposing the target Spark version as a label would allow:

onlyif spark-4.2
query I
SELECT ...
----
42

onlyif spark-3.5
query I
SELECT ...
----
NULL

This is finer-grained and keeps both behaviors visible side by side at the test
site, which is where a reader wants them. It needs a decision on what the label
namespace looks like and how a range (rather than an exact version) is expressed.

Either option needs a notion of "which Spark version is this test suite currently
targeting", since there is no Spark process in the loop on the DataFusion side.
That target is a declared baseline rather than something detected at runtime.

Describe alternatives you've considered

Keeping the status quo and recording divergences only in comments. This is what
we do implicitly today. It loses the ability to test the non-baseline behavior at
all, and the comments drift out of date with no signal.

Additional context

This came up while designing an agent skill for auditing datafusion-spark
expression implementations against the Spark source. The audit compares a
function's behavior across several Spark releases, and cross-version divergences
are a routine finding. With no mechanism to express them, the audit can only
leave a comment and link here.

The related epic is #15914.