fix(optimizer): preserve unreferenced UDTF and anonymous-function projections

[fivetran-kwoodbeck]

pushdown_projections was pruning unreferenced select-list projections by checking for a hardcoded tuple of known set-returning expression classes (exp.Explode, exp.Inline, exp.Unnest). This silently dropped projections containing unregistered set-returning functions.

Updated pushdown_projections.py to remove SET_RETURNING_FUNCTIONS and replaced it with (exp.Anonymous, exp.UDTF). exp.UDTF catches all registered set-returning functions via the class hierarchy; exp.Anonymous is the conservative catch-all for unregistered ones. Also, added UDTF to Inline's base classes since it was always semantically a UDTF.

Bug Sample:

-- Input
SELECT t.a FROM (SELECT a, STACK(2, b, c) AS s FROM x) t

-- Wrong output (STACK pruned — rows halved)
SELECT x.a AS a FROM x AS x

-- Expected output (STACK preserved)
SELECT t.a AS a FROM (SELECT x.a AS a, STACK(2, x.b, x.c) AS s FROM x AS x) AS t

STACK parses as exp.Anonymous and was not in the hardcoded list, so it was pruned. The same latent bug affects any dialect with unregistered set-returning functions.

[github-actions]

SQLGlot Integration Test Results

✅ All tests passed

Comparing:

• this branch (sqlglot:fix/spark-stack-pushdown-projections @ sqlglot 9b1e506)
• baseline (main @ sqlglot 1d753d7)

Overall

main: 182937 total, 163862 passed (pass rate: 89.6%)

sqlglot:fix/spark-stack-pushdown-projections: 182937 total, 163862 passed (pass rate: 89.6%)

Transitions:
No change

✅ All tests passed

[ebarkhordar]

I ran the SQL from your description against 9bdb961 in a clean python:3.12-slim container. The pushdown_projections change does what you describe, but the example in the description still loses STACK through the default optimize() pipeline, because merge_subqueries carries its own narrower guard at sqlglot/optimizer/merge_subqueries.py:306 that matches exp.Explode only.

qualify + pushdown_projections alone:
  SELECT `t`.`a` AS `a` FROM (SELECT `x`.`a` AS `a`, STACK(2, `x`.`b`, `x`.`c`) AS `s` FROM `x` AS `x`) AS `t`
full optimize():
  SELECT `x`.`a` AS `a` FROM `x` AS `x`

Adding exp.UDTF and exp.Anonymous to that one isinstance at merge_subqueries.py:306, and changing nothing else, produces the output your description expects:

WITH `t` AS (SELECT `x`.`a` AS `a`, STACK(2, `x`.`b`, `x`.`c`) AS `s` FROM `x` AS `x`) SELECT `t`.`a` AS `a` FROM `t` AS `t`

EXPLODE survives the full pipeline either way, since exp.Explode is what both guards already match.

Second and smaller, on the line this PR changes. exp.UDTF does not reach every registered set-returning function. exp.ExplodingGenerateSeries, which is what postgres parses GENERATE_SERIES into, subclasses GenerateSeries(Expression, Func), so it is neither a UDTF nor Anonymous:

in : SELECT t.a FROM (SELECT a, GENERATE_SERIES(1, 3) AS s FROM x) t
out: SELECT "x"."a" AS "a" FROM "x" AS "x"

pushdown_projections prunes that one itself, so it is not a regression here, the old hardcoded tuple missed it too. It does mean the class hierarchy on its own is not the completeness the description claims. Adding exp.ExplodingGenerateSeries at line 206 covers it.

I ran pure Python only, not the mypyc build.

[georgesittas]

Both points are valuable, thanks for sharing.

[fivetran-kwoodbeck]

Thanks @ebarkhordar, good catch! We'll follow up with the merge_subqueries fix in another PR.