[branch-54] Fix evaluation of lambda with multiple parameters when some are unused (backport of #22853, adapted)

[gstvg]

Which issue does this PR close?

Adapted backport of #22853 (non-breaking). Note that the fix is totally different from the mainline PR

Rationale for this change

Despite all built-in higher-order functions being single parameter in the 54 release, the public higher-order trait supports lambdas with multiple parameters, which would trigger the bug. I found a case in Sail, for example
See mainline PR #22853 description as well

What changes are included in this PR?

Currently the lambda body and the captured record batch are projected to avoid copying uncaptured columns.

This PR instead replace uncaptured columns with cheap NullArray's, so no projection is necessary, and therefore no wrong projection happens.

In contrast, the mainline PR collects the lambda parameters that are actually used, and skips the evaluation and insertion of unused ones into the evalution batch, so no errors occurs (with the bonus of avoiding unused parameter evaluation at the cost of a breaking change)

Are these changes tested?

sqllogictests in new file multi_arg_lambda.slt

Are there any user-facing changes?

No

[alamb]

I don't understand this PR

It claims to be a backport of

• fix(lambda): only push referenced params into the merged batch #22853

However, that PR and this PR says:

This is a latent bug today — no in-tree higher-order function exercises it — but it blocks #22689 (transform_values, which uses (k, v) -> body lambdas) and any future HOF that takes more than one parameter.

Which I read to be that this isn't a user visible bug

However, the PR contains a slt test that implies this is a user visible issue

I think we need to explain more clearly what we are doing more clearly before backporting this to a release

[alamb]

can we please add comments about the rationale for this function? It isn't at all clear to me

[gstvg]

b1f83e8

[gstvg]

@alamb I did a quick read of the mainline PR description without giving it much attention before replicating it here, my bad. The public higher-order trait supports lambdas with multiple parameters but in the 54 release all built-in ones are single parameter, so this is only visible for user defined higher-order functions. I found a case in Sail, for example

Any query like

SELECT array_filter(a, (v, i) -> i > 1) -- first parameter not used

would fail without the workaround Sail is using. I updated this PR description

For context, before being merged, the array_transform in the lambda PR used to support two parameters (value, index) but I removed it to reduce the PR size, leaving this code path untested (at that time projection wasn't used so this bug didn't surfaced). In retrospect, this was not the best decision