Skip to content

Join dynamic filter pushdown through AggregateExec can confuse same-named grouping columns #25264

Description

@haohuaijin

Describe the bug

Enabling join dynamic filter pushdown can silently discard a matching row when the filter passes through an AggregateExec that groups on two columns with the same name (here a.id and b.id from a nested join).

With the same data and SQL, disabling datafusion.optimizer.enable_join_dynamic_filter_pushdown returns one row; enabling it returns zero rows, without an error.

To Reproduce

Run the following SQL in a fresh datafusion-cli session. The two /tmp/ Parquet paths must not already exist.

SET datafusion.optimizer.join_reordering = false;

COPY (SELECT 'a1' AS id, 'x1' AS ty)
TO '/tmp/df_dynamic_filter_aggregate_a.parquet'
STORED AS PARQUET;

COPY (SELECT 'x1' AS id)
TO '/tmp/df_dynamic_filter_aggregate_b.parquet'
STORED AS PARQUET;

CREATE EXTERNAL TABLE ta
STORED AS PARQUET
LOCATION '/tmp/df_dynamic_filter_aggregate_a.parquet';

CREATE EXTERNAL TABLE tb
STORED AS PARQUET
LOCATION '/tmp/df_dynamic_filter_aggregate_b.parquet';

-- Returns one row: (x1, a1, x1).
SET datafusion.optimizer.enable_join_dynamic_filter_pushdown = false;

SELECT s.id, a.id, b.id
FROM tb s
JOIN (
    SELECT a.id, b.id
    FROM ta a LEFT JOIN tb b ON a.ty = b.id
    GROUP BY a.id, b.id
)
ON s.id = b.id;

-- Incorrectly returns zero rows.
SET datafusion.optimizer.enable_join_dynamic_filter_pushdown = true;

SELECT s.id, a.id, b.id
FROM tb s
JOIN (
    SELECT a.id, b.id
    FROM ta a LEFT JOIN tb b ON a.ty = b.id
    GROUP BY a.id, b.id
)
ON s.id = b.id;

Expected behavior

No response

Additional context

AggregateExec resolves pushed-down filter columns by name instead of by grouping position, so the filter on the grouping column b.id is applied to a.id. Found while working on #25244; the fix is included in #25259.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions