fix: cap the field list in "No field named" errors - #24931
Open
akashjainn wants to merge 1 commit into
Open
Conversation
Selecting an unknown column from a wide table produces an error message that
lists every field in the schema. On a 200 column table that is ~2.8k characters
of output, and schemas with hundreds or thousands of columns are ordinary in
analytics workloads, so in practice the useful part of the message -- the name
that was not found, and the "Did you mean" suggestion -- is pushed off screen by
the list that follows it.
Cap the listed fields at 20 and summarise the rest, so the same query now
reports:
Error: Schema error: No field named not_a_column.
Valid fields are wide.col_0, ..., wide.col_19 and 180 others.
2.8k characters down to 356. Schemas at or below the cap are unchanged, which
keeps the list useful where it is short enough to read and leaves the existing
message assertions in dfschema, column, expr_rewriter and the dataframe tests
untouched.
Postgres omits the valid-field list entirely, but DataFusion's list is helpful
on narrow schemas and the closest-match suggestion already handles the common
typo case, so this keeps the list and bounds it rather than removing it.
Tests cover a narrow schema, exactly the cap, one field past the cap (singular
"1 other"), and a 200 field schema.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #5332.
Which issue does this PR close?
Closes #5332.
Rationale for this change
Selecting a column that doesn't exist lists every field in the schema. On a 200 column table that's about 2.8k characters, and the useful part of the message - the name that wasn't found and the "Did you mean" suggestion - ends up scrolled off the top.
Filed in 2023, still reproduces.
What changes are included in this PR?
Cap the list at 20 fields and summarise the rest:
2.8k characters down to 356. Schemas at or under 20 fields print the same as before, so the existing message assertions in dfschema.rs, column.rs, expr_rewriter and the dataframe tests are untouched - they're all well under that.
The issue suggested either dropping the list like Postgres does or truncating it. I kept it and bounded it, since the list is useful on narrow schemas and closest_valid_field already handles the common typo. Happy to switch to the Postgres behaviour instead if you'd rather.
Are these changes tested?
Four unit tests in datafusion/common/src/error.rs: a narrow schema, exactly 20 fields, 21 fields (checks it says "1 other" not "1 others"), and 200 fields.
cargo test -p datafusion-common: 608 passed. datafusion-expr --lib: 258 passed. cargo fmt clean.
Two things in my checkout that aren't from this change and reproduce on unmodified main: the arrow_test_data doctest fails because the testing/ submodule isn't initialised, and clippy reports an unused import of crate::config::TableParquetOptions.
Are there any user-facing changes?
The wording of SchemaError::FieldNotFound when a schema has more than 20 fields. No API change.