fix: attach the window a live writer is filling, under one lock

[AlexeyShalaev]

The problem

On a table that is being written to, the window the application is inserting into — the current month, every time — could not be attached at all.

_attach_with_reconcile moved the window's rows out of the DEFAULT partition in reconcile_default_rows' own transaction and retried the ATTACH in another. The move holds SHARE ROW EXCLUSIVE on DEFAULT, so writers queue behind it and are let in the moment it commits — exactly the gap before ATTACH takes its ACCESS EXCLUSIVE and scans. One row is enough to fail the scan. With DEFAULT_CONFLICT_MAX_RETRIES = 2 the window bought two full scans of a two-million-row DEFAULT under ACCESS EXCLUSIVE, a compensating move-back, and no progress; the tick reported error set and issues empty, and partition_data let the IntegrityError out instead of reporting the window.

Reproduced with the reporter's lab on postgres:17-alpine, 2M rows, a writer inserting every 5 ms.

The design

The move and the attach now share a transaction. A new repository operation, reconcile_and_attach, takes its locks up front — EXCLUSIVE on the parent, ACCESS EXCLUSIVE on the partition and on the DEFAULT sibling — moves what is left of the window, and attaches. Nothing can put a row into DEFAULT between the move and the scan, because nothing has been able to reach DEFAULT since before the move, and a failed attach rolls the move back with it.

_attach_with_reconcile keeps its shape: attempt 1 is still the plain, cheap attach_partition, so a parent with no DEFAULT partition — or one holding nothing for the window — pays for nothing else. A DEFAULT conflict runs the bulk reconcile_default_rows exactly as before, under the lighter SHARE ROW EXCLUSIVE that leaves readers of DEFAULT alone, and attempt 2 is the atomic form, which takes only what the writer added while that ran, plus the scan ATTACH does anyway. The exclusive window is a tail and a scan, not a month.

The parent's lock is EXCLUSIVE, one level above ATTACH's own, and it is there for the writer. An INSERT picks its partition from the set it saw when it took ROW EXCLUSIVE on the parent: one that got that far and then queued on DEFAULT's lock comes out of the wait still aimed at DEFAULT, and is rejected by the constraint the attach just narrowed — a lost write, and what a plain ATTACH does to a live writer today. Blocking at the parent means no insert is ever mid-routing while the partition set changes: the writer waits, re-plans, and lands in the new partition with its statement unchanged. I found this the honest way — the first cut used SHARE UPDATE EXCLUSIVE and the new integration test failed on the writer's insert, not on the attach. EXCLUSIVE does not conflict with ACCESS SHARE, so readers of the other partitions carry on.

The second half of the report: a window that still cannot be attached is now a default_holds_rows topology finding rather than the driver's exception. apply() already records those in result.issues and goes on — what the failures table in concepts/execution.md has always promised — and partition_data catches it and returns a move issue with complete=False. That also covers the one case the lock cannot reach, a foreign DEFAULT partition, which LOCK TABLE refuses outright.

What I rejected

• More retries. No retry count converges against a steady writer, and each one is a full scan of DEFAULT under ACCESS EXCLUSIVE.
• Moving the whole window inside the locked transaction, with no bulk pre-drain. Simpler, but it holds ACCESS EXCLUSIVE on DEFAULT for the length of a month's rows, blocking readers of DEFAULT too.
• Going straight to the locked form instead of trying the plain attach first. It saves one failed scan, but puts an extra statement and a heavier lock on every attach against a table that has a DEFAULT partition — including the common case where DEFAULT holds nothing for the window and the plain attach just works.
• Accepting the writer's failed insert as PostgreSQL's own behaviour and documenting a retry. A migration must not cost the application writes.

Public surface

reconcile_and_attach is an addition to the PartitionRepository protocol in both mirrors: an implementation written from scratch gains one method, subclasses of PostgresPartitionRepository are untouched. It is the one method with a transaction boundary in its contract, and guide/extending.md says so.

Also on this page

Following the guide's last step verbatim fails on a BIGSERIAL table: the sequence is owned by the column it was declared on, so after the swap it belongs to events_legacy while every partition's id default draws from it, and DROP TABLE events_legacy is refused. The lab reaches step 6 and dies there. ALTER SEQUENCE events_id_seq OWNED BY events.id first, and the drop goes through with the ids intact; CASCADE takes the sequence and every default with it. Measured on 17 and written into the guide as its own note.

Verified

The reporter's lab, unmodified, against this branch:

--- 3. the first tick: this month and two ahead, this month's rows move out of DEFAULT
    created 3, issues 0, error None, 806 ms
    live rows: 85 in this month's partition, 0 left in DEFAULT
    writer: 43 inserts, p50 0.5 ms, longest 424 ms, 2 over 100 ms
--- 4. the drain: partition_data in batches of 50,000, oldest month first
    5 calls, 47 batches, 1,964,338 rows moved into 12 partitions in 10.6 s
    writer: 1480 inserts, p50 0.7 ms, longest 43 ms, 0 over 100 ms
    reader: 349 counts of the oldest month, lowest count 29501 of 129501, below full in 6 of 349 samples

No writer errors in either step, no issues, and nothing raised out of the drain. On master the same run prints created 0, issues 0, error 'IntegrityError: … updated partition constraint for default partition "events_legacy" would be violated by some row', leaves 87 left in DEFAULT, and ends on that exception in step 4.

With the ALTER SEQUENCE above added to the lab's step 6, the run finishes and prints its tree — twelve full months, the current one live and filling, and the writer's own rows all present:

--- 6. detach and drop the empty DEFAULT
    rows left in events_legacy: 0
    DETACH PARTITION events_legacy; DROP TABLE events_legacy: 134 ms
--- the tree afterwards
    events__2025_09           128944 rows  2 indexes  FOR VALUES FROM ('2025-09-01 00:00:00+00') TO ('2025-10-01 00:00:00+00')
    ...
    events__2026_08           169246 rows  2 indexes  FOR VALUES FROM ('2026-08-01 00:00:00+00') TO ('2026-09-01 00:00:00+00')
    events__2026_09            37242 rows  2 indexes  FOR VALUES FROM ('2026-09-01 00:00:00+00') TO ('2026-10-01 00:00:00+00')
    events__2026_10                0 rows  2 indexes  FOR VALUES FROM ('2026-10-01 00:00:00+00') TO ('2026-11-01 00:00:00+00')
    events__2026_11                0 rows  2 indexes  FOR VALUES FROM ('2026-11-01 00:00:00+00') TO ('2026-12-01 00:00:00+00')
    2,001,931 rows through the parent; the writer inserted 1931, 1931 are there
    the writer's statement never changed: INSERT INTO events (created_at, kind, payload) VALUES (now(), 'live', $1)

New integration tests drive a writer inserting into the window while the attach runs, in both mirrors — the aio pair with a task, the sync pair with a thread in test_concurrency.py. Against master's library sources all four fail with the reported CheckViolation, three runs out of three; against this branch they pass, five runs out of five.

Gate: uv run --extra pydantic-settings pytest --cov=pg_partsmith --cov-report=term --cov-fail-under=90 --cov-report=xml:coverage.xml — 3374 passed, 20 skipped, 97.99% coverage, plus ruff check, ruff format --check and mypy pg_partsmith clean.

Closes #64

[codecov]

Codecov Report

❌ Patch coverage is 97.97980% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pg_partsmith/aio/repositories/creator.py	96.55%	0 Missing and 1 partial ⚠️
pg_partsmith/sync/repositories/creator.py	96.66%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!