Skip to content

Copy the original schema in table.duplicate() - #957

Open
Mo0nl19ht wants to merge 1 commit into
simonw:mainfrom
Mo0nl19ht:fix-duplicate-schema
Open

Mo0nl19ht wants to merge 1 commit into
simonw:mainfrom
Mo0nl19ht:fix-duplicate-schema

Conversation

@Mo0nl19ht

@Mo0nl19ht Mo0nl19ht commented Nov 17, 2026 •

Copy link
Copy Markdown

Fixes:

table.duplicate() and sqlite-utils duplicate used CREATE TABLE ... AS SELECT, so the copy lost the primary key, NOT NULL, UNIQUE, CHECK and DEFAULT constraints, foreign keys, STRICT, WITHOUT ROWID, indexes and rowids, and INTEGER columns became INT. The documentation says the schema is copied.

Behavior change

The copy now enforces the original table's constraints. Code that duplicated a table and then wrote rows into the copy that the original would reject (duplicate primary keys, NULL in a NOT NULL column, values failing a CHECK, dangling foreign keys with PRAGMA foreign_keys on) will now get an IntegrityError. Column types in the copy are the declared types (INTEGER, not INT).

Change

The copy is created from the original table's CREATE TABLE statement with only the table name changed. Rows are then copied with their rowid values (except for WITHOUT ROWID tables). Generated columns are left out of the INSERT so the copy computes them itself.

  • Indexes are recreated from their CREATE INDEX statements under new names, since index names must be unique in a database. The original table name in the index name is replaced (idx_pets_name becomes idx_pets_copy_name). Other index names are prefixed with the new table name. A numeric suffix is added if the name is taken.
  • Self-referencing foreign keys and table.column references (in CHECK constraints, generated columns and partial or expression indexes) are rewritten to the new name. Without this the copy's foreign key would point at the original table, and the qualified references would make CREATE TABLE fail with no such column.
  • The AUTOINCREMENT counter is carried over, as transform() does since sqlite-utils transform removes the AUTOINCREMENT keyword #602, so the copy does not reuse IDs deleted from the original.
  • Triggers are not copied.
  • Virtual tables such as FTS tables are still copied with CREATE TABLE ... AS SELECT, as before.

The renaming uses the existing lexer in create_table_parser.py (new rename_create_statement(), rewrite_table_references() and parse_without_rowid()), so quoting, comments and string literals are left alone.

The duplicate sections of docs/python-api.rst and docs/cli.rst now say what is and is not copied.

Tests

The existing test_duplicate expected INT for its INTEGER columns. That was the bug: CREATE TABLE ... AS SELECT rewrites declared types to their affinity. It now expects INTEGER.

New tests in tests/test_duplicate.py cover:

  • the copied CREATE TABLE statement and enforced constraints
  • rowids, WITHOUT ROWID, STRICT and generated columns
  • recreated and renamed indexes, including name collisions
  • self-referencing foreign keys and the AUTOINCREMENT counter
  • table.column references and names that need quoting
  • triggers not being copied
  • FTS tables keeping their current behavior

tests/test_cli.py adds a test for sqlite-utils duplicate.

The new tests fail on main and pass with this change. Locally the full suite passes (1511 passed, 16 skipped), and so do black, flake8, mypy, pyright, cog --check, codespell and a Sphinx build with -W. I also ran the suite on Python 3.10 against SQLite 3.23.1 and 3.46 built the way test-sqlite-support.yml does, with no failures. The STRICT and generated column tests are skipped on 3.23.1.

ty check sqlite_utils reports one redundant-cast in insert_all() (db.py:4792 on main). It fails the same way on main, so it is unrelated to this change.


Written with help from Claude Code.


📚 Documentation preview 📚: https://sqlite-utils--957.org.readthedocs.build/en/957/

table.duplicate() and "sqlite-utils duplicate" used CREATE TABLE ... AS
SELECT, so the copy lost the primary key, NOT NULL, UNIQUE, CHECK and
DEFAULT constraints, foreign keys, STRICT, WITHOUT ROWID, indexes and
rowid values, and INTEGER columns became INT.

The copy is now created from the original CREATE TABLE statement with
only the table name changed, and rows are copied along with their
rowids. Indexes are recreated under new names, since index names must
be unique. Self-referencing foreign keys and table.column references
point at the copy, and the AUTOINCREMENT counter is carried over.
Triggers are not copied. Virtual tables are still copied with
CREATE TABLE ... AS SELECT.

This is a behavior change: the copy now enforces the original table's
constraints.

Refs simonw#956
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant