IN LIST: retain short lists with specialized filters

[geoffreyclaude]

Which issue does this PR close?

• Part of Further improve performance of IN list evaluation #19241.
• Stacked on #25186.

Rationale for this change

On main, the expression simplifier rewrites singleton IN lists, and two- or three-item lists on a column, into left-deep OR / AND comparison chains. DataFusion now has specialized static InList filters for several physical representations. For supported constant lists, expanding two or three items prevents those specialized physical filters from being used.

The rewrite should therefore depend on the available physical representation, not only on list length. This PR retains two- and three-item InList expressions when a specialized static filter can evaluate them, while preserving the existing comparison rewrite for dynamic or generic lists.

Singleton lists are still simplified to equality. Although singleton InList evaluation is faster for several representations in isolation, equality remains the canonical logical form and can benefit optimizer and pushdown paths that this microbenchmark does not measure.

What changes are included in this PR?

• Gives the short-list simplifier access to type information.
• Retains two- and three-item constant lists for supported primitive, temporal, decimal, fixed-size-binary, and inline byte-view representations, including dictionary wrappers.
• Keeps the existing OR / AND expansion for generic types, out-of-line byte views, and non-constant lists.
• Decouples the preimage rewrite limit from the short-list physical-filter decision; its existing three-value limit is unchanged.
• Adds a SQL-planned benchmark for list sizes one through four and updates affected plan snapshots.

What is the testing strategy for this PR?

Coverage includes the retained and expanded boundaries; IN and NOT IN; typed nulls; dynamic lists; nested dictionaries; supported and unsupported fixed-size-binary widths; inline and out-of-line Utf8View / BinaryView values; and the independent preimage limit.

Focused validation on the stacked branch:

• cargo fmt --all
• cargo clippy --all-targets --all-features -- -D warnings
• cargo test -p datafusion-optimizer --lib (808 tests)
• Directly modified SQL logic test files, plus negative_zero
• cargo test -p datafusion-substrait roundtrip_inlist (5 tests)
• cargo check -p datafusion --bench in_list_rewrite
• cargo bench -p datafusion --bench in_list_rewrite -- --noplot

The full extended workspace suite was not run locally; broader coverage is left to CI.

Are there any user-facing changes?

This PR changes optimized plan shape, but does not change query results or public APIs. For supported constant-list representations, optimized plans retain two- and three-item InList expressions instead of expanding them into comparison chains.

The lower stacked PR, #25186, independently fixes signed-zero equality for retained floating-point lists and must land before this plan-shape change.

Local benchmark snapshot

The new benchmark parses and analyzes both forms from SQL, asserts their physical shapes and equal outputs, then measures only physical-expression evaluation. It covers five representative types; list sizes 1-4; batches of 1, 64, and 8,192 rows; input and list nulls; miss, balanced, and skewed match rates; and both IN and NOT IN: 300 paired scenarios and 600 Criterion functions in total.

These indicative measurements were collected locally on an Apple M1 Max (10 cores, 64 GB) with #25187 stacked on #25186. The branch hashes have since changed during rebasing; the benchmark and optimizer logic measured here are unchanged. Absolute timings are machine-specific.

The table focuses on the two sizes whose optimizer policy changes. Each ratio is the median comparison chain / retained InList time across 16 scenarios; values above 1 favor retaining InList.

Representation	2 items	Wins	3 items	Wins	Decision
FixedSizeBinary(8)	17.26x	16/16	17.99x	16/16	Retain
Float64	3.29x	16/16	3.96x	16/16	Retain
Int32	1.85x	16/16	2.27x	16/16	Retain
inline Utf8View	1.34x	14/16	1.35x	14/16	Retain
generic Utf8	1.25x	11/16	1.77x	12/16	Keep OR / AND

Generic Utf8 remains mixed, including regressions for skewed and null-containing lists, so this PR deliberately leaves it on the existing comparison rewrite. List sizes one and four serve as boundary controls; their optimizer policy is unchanged.