Skip to content

Preserve column COLLATE through transforms - #955

Open
Mo0nl19ht wants to merge 1 commit into
simonw:mainfrom
Mo0nl19ht:fix-transform-collate
Open

Mo0nl19ht wants to merge 1 commit into
simonw:mainfrom
Mo0nl19ht:fix-transform-collate

Conversation

@Mo0nl19ht

@Mo0nl19ht Mo0nl19ht commented Nov 17, 2026 •

Copy link
Copy Markdown

Fixes:

table.transform() rebuilds columns from PRAGMA table_info, which does not report collation, so a column declared name TEXT COLLATE NOCASE became name TEXT and comparisons on it turned case-sensitive. (A column-level UNIQUE already kept its collation, because parse_uniques() reads it, but the column itself lost it.)

Change

A new parse_column_collations() in create_table_parser.py reads each column's COLLATE from the schema. It reuses the collation scan that _column_uniques() already did, now moved into a _column_collation() helper. transform_sql() passes the result to create_table_sql() through a private _column_collations argument, the same way _column_comments works. Collations follow renamed columns and are dropped with dropped columns. Everything else about the generated schema is unchanged.

The transform section of docs/python-api.rst now mentions that column collations are preserved.

Tests

  • test_parse_column_collations in tests/test_create_table_parser.py: quoted names, VARCHAR(10), and ignoring COLLATE inside CHECK, comments and table-level UNIQUE
  • test_transform_preserves_column_collation and test_transform_collation_follows_renamed_column_and_drop in tests/test_transform.py
  • test_transform_preserves_column_collation in tests/test_cli.py, for sqlite-utils transform

The new tests fail on main and pass with this change. Locally the full suite passes (1505 passed, 16 skipped), and so do black, flake8, mypy, pyright, cog and codespell. ty check sqlite_utils reports one redundant-cast in insert_all() (db.py:4792 on main). It fails the same way on main with ty 0.0.104, and the latest main CI run fails at the same run ty step, so it is unrelated to this change.


Written with help from Claude Code.


📚 Documentation preview 📚: https://sqlite-utils--955.org.readthedocs.build/en/955/

table.transform() rebuilt each column from PRAGMA table_info, which does
not report collation, so a column declared "name TEXT COLLATE NOCASE"
became plain "name TEXT" and comparisons on it turned case-sensitive.

Parse column collations from the existing schema with a new
parse_column_collations() helper and pass them to create_table_sql()
via a private _column_collations argument. Collations follow renamed
columns and are dropped with dropped columns.

Refs simonw#954
@Mo0nl19ht
Mo0nl19ht force-pushed the fix-transform-collate branch from 0b71df5 to 6dec072 Compare November 17, 2026 08:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant