From 42d9c06d3644b4fc451ffc9484415e83c2215d33 Mon Sep 17 00:00:00 2001 From: Alexey Shalaev <75322386+AlexeyShalaev@users.noreply.github.com> Date: Sun, 6 Sep 2026 18:48:46 +0300 Subject: [PATCH] docs: what the guides said that the code no longer does The grant comment in the container guide claimed USAGE and CREATE on the schema are enough to create and attach partitions; PostgreSQL lets only the parent's owner do either, so the comment now says the grant is needed and not sufficient. The rest is drift since 1.2.0 through 1.5.0: PostgreSQL 15 through 18 under test, backfill among the commands, the tree read over pg_inherits, three differences between the mirrors, after_* hook exceptions re-raised, a Python block hook no longer holding a stop, eight hook moments, the hand-maintained mirror files as the script lists them, and 1.5.x as the release the security policy supports. --- CONTRIBUTING.md | 4 ++-- README.md | 18 ++++++++++++------ SECURITY.md | 2 +- docs/agents.md | 2 +- docs/getting-started/installation.md | 2 +- docs/guide/archiving.md | 4 ++-- docs/guide/cli.md | 10 +++++----- docs/guide/container.md | 8 ++++---- docs/guide/running.md | 5 +++-- pg_partsmith/cli/__init__.py | 7 ++++--- 10 files changed, 35 insertions(+), 27 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b9b74f3..0f10121 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -133,8 +133,8 @@ uv run python scripts/sync_mirror.py uv run ruff check --fix pg_partsmith/sync && uv run ruff format pg_partsmith/sync ``` -The lock managers, `maintainer.py`, `repositories/{resolver,fk_manager,timeouts}.py` are -maintained by hand. +The lock managers, `maintainer.py`, `command_hooks.py`, `python_hooks.py` and +`repositories/{resolver,fk_manager,timeouts,pin}.py` are maintained by hand. The tests follow the same rule. `tests/integration/sync/` is generated from `tests/integration/aio/`: diff --git a/README.md b/README.md index b058846..361a242 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,7 @@ Three ways in, one version number: at their own modulus; partitions the scheme did not produce are reported, never touched; foreign tables are inspected, never dropped - **Async and sync** — `pg_partsmith.aio` on `AsyncEngine`, `pg_partsmith.sync` on `Engine` -- **A command line** — `pg-partsmith inspect / plan / validate / apply` over a YAML or +- **A command line** — `pg-partsmith inspect / plan / validate / apply / backfill` over a YAML or JSON document, with a saved plan as the artifact between plan and apply, and exit codes a CronJob and a CI step can read - **A container image** — `ghcr.io/bedrock-python/pg-partsmith`, for stacks with no Python @@ -187,7 +187,8 @@ LifecyclePolicy(retention=ExpireIf(AllOf((KeepNewest(count=2), ## What maintenance does -1. **Inspect** — one catalog round-trip reads the whole tree (`pg_partition_tree`), the +1. **Inspect** — one catalog round-trip reads the whole tree (a recursive walk over + `pg_inherits`, which still sees a half-detached partition), the marker-tagged detached orphans, and — only when a policy asks — sizes and row estimates. 2. **Plan** — the planner walks the scheme and the tree together. At a `RANGE` level it decides which windows must exist ahead of the cursor (the clock, or `max(key)` for an @@ -212,7 +213,8 @@ test, not a promise. - A hash set at a modulus the config no longer uses is preserved if complete and repaired *at its own modulus* if not; mixed moduli leaving a gap are reported, never guessed at. - A plan made at 10:00 and applied at 10:05 refuses to drop a table that was recreated in - between (`PlanStaleError`, reported as an issue). + between (`PlanStaleError`: the run stops there, or records the issue under + `continue_on_error`). ## Sync usage @@ -236,8 +238,10 @@ service = PartitionLifecycleService( result = PartitionMaintainer(service).run_maintenance_safe(config) ``` -Two behavioural differences: `ddl_timeout_seconds` is enforced server-side via -`statement_timeout` per statement, and the Redis lock renews from a background thread. +Three behavioural differences: `ddl_timeout_seconds` is enforced server-side via +`statement_timeout` per statement, the Redis lock renews from a background thread and can +only warn when the lease is lost, and a Python block hook runs inline rather than on a +thread of its own. ## Hooks @@ -267,7 +271,9 @@ planned and the size the policy measured, when it asked for one. | `before_drop` / `after_drop` | around a drop — `before_drop` is the last chance to read the data | | `on_event` | every one of the above, for an audit trail or metrics in one method | -`before_*` exceptions abort that operation; `after_*` exceptions are logged. +`before_*` exceptions abort that operation; `after_*` exceptions are logged and re-raised +too — the operation already happened, but the run stops there (or records the issue under +`continue_on_error`). ## Documentation diff --git a/SECURITY.md b/SECURITY.md index 25cdc44..af6541e 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -4,7 +4,7 @@ | Version | Supported | |---------|-----------| -| 1.2.x | ✅ the current release; fixes ship as the next patch or minor | +| 1.5.x | ✅ the current release; fixes ship as the next patch or minor | | older | ❌ upgrade to the current release first | The library, the command line and the container image share one version number, so a fix diff --git a/docs/agents.md b/docs/agents.md index 9f1b8c9..8a0bad2 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -7,7 +7,7 @@ | | | |---|---| | Package | `pg-partsmith` on PyPI, import root `pg_partsmith` | -| Requires | Python 3.11+, PostgreSQL 15+ (tested on 15, 16 and 17), SQLAlchemy 2 | +| Requires | Python 3.11+, PostgreSQL 15+ (tested on 15 through 18), SQLAlchemy 2 | | Install | `pip install pg-partsmith` · extras: `cli`, `redis-locks`, `pydantic-settings` | | Async | `pg_partsmith.aio`, on an `AsyncEngine` | | Sync | `pg_partsmith.sync` — same class names, same arguments, no `await` | diff --git a/docs/getting-started/installation.md b/docs/getting-started/installation.md index 5f8be6c..c2e175e 100644 --- a/docs/getting-started/installation.md +++ b/docs/getting-started/installation.md @@ -5,7 +5,7 @@ | | | |---|---| | Python | 3.11 or newer | -| PostgreSQL | 15 or newer — the integration suite runs on 15, 16 and 17 | +| PostgreSQL | 15 or newer — the integration suite runs on 15 through 18 | | SQLAlchemy | 2.x, with `asyncpg` for the async API or `psycopg2` for the sync one | | Privileges | ownership of the partitioned table (to create, attach, detach and drop partitions) — no superuser | diff --git a/docs/guide/archiving.md b/docs/guide/archiving.md index 568e6f6..15ca6f7 100644 --- a/docs/guide/archiving.md +++ b/docs/guide/archiving.md @@ -27,7 +27,7 @@ Every one of them takes the same `PartitionEvent`: | Field | What it is | |---|---| -| `phase` | which of the six moments this is | +| `phase` | which of the eight moments this is | | `config` | the table's configuration — the calendar, the codec, the policy in force | | `partition` | the partition itself: `name`, `bounds`, `oid`, `subpartition_type` | | `window` | the period it covers; `None` for a member of a root `HASH` or `LIST` | @@ -64,7 +64,7 @@ logs. ### One method for every phase -An audit trail or a metrics counter wants all six moments and treats them alike. That is +An audit trail or a metrics counter wants all eight moments and treats them alike. That is `on_event`, which fires for every phase in addition to the method named after it — so a hook implementing both is called twice, on purpose: diff --git a/docs/guide/cli.md b/docs/guide/cli.md index 0ac826b..090148a 100644 --- a/docs/guide/cli.md +++ b/docs/guide/cli.md @@ -46,8 +46,9 @@ What a stopped run leaves behind is what the library promises for any cancellati most a detached, unattached table (attach is the last step of a creation) or a marked, half-detached partition, and the next run converges both. -A block of Python in a hook cannot be interrupted mid-way — the signal is honoured the -moment the block returns. A command can, and is. +A block of Python in a hook runs on a thread of its own, so a stop does not wait for it: +the run is cancelled and exits, and a block still running at that point is abandoned with +the process. A command's child is terminated on purpose. ## Shell completion @@ -180,9 +181,8 @@ therefore reviewable for its locks without the CLI. **Why there is no `--sql`.** Printing the statements would be the obvious next thing, and it is deliberately not offered. The DDL is built at execution time from decisions that are -only made then — whether a `DETACH CONCURRENTLY` has to fall back to the blocking form -because a DEFAULT sibling exists, how rows a DEFAULT partition holds for a new window get -moved, which orphan marker gets cleared — and a rendering that left those out would be +only made then — how rows a DEFAULT partition holds for a new window get moved, which +orphan marker gets cleared — and a rendering that left those out would be read as a transcript in exactly the cases a review exists to catch. The plan is the contract: what, why, how big, what it locks. diff --git a/docs/guide/container.md b/docs/guide/container.md index d2a8d3c..a90da3d 100644 --- a/docs/guide/container.md +++ b/docs/guide/container.md @@ -66,12 +66,12 @@ its own rather than the application's: ```sql CREATE ROLE partsmith LOGIN PASSWORD '…'; --- Enough to create partitions in the schema and attach them +-- The schema: needed, and not enough on its own GRANT USAGE, CREATE ON SCHEMA public TO partsmith; --- Enough to detach and drop them: PostgreSQL requires ownership of both the --- partition and the parent for ATTACH/DETACH, so the role owns the tables it --- maintains, or is a member of the role that does. +-- The tables: PostgreSQL lets only the parent's owner run CREATE TABLE … PARTITION OF, +-- ATTACH and DETACH, and only a table's owner drop it, so the role owns the tables +-- it maintains, or is a member of the role that does. ALTER TABLE public.events OWNER TO partsmith; -- or: GRANT app_owner TO partsmith; ``` diff --git a/docs/guide/running.md b/docs/guide/running.md index 442b446..8f7b3bc 100644 --- a/docs/guide/running.md +++ b/docs/guide/running.md @@ -3,7 +3,7 @@ `pg-partsmith` is a process that runs one command and exits with a code. It has no scheduler of its own, on purpose: cron, a CronJob, a CI schedule or a systemd timer already exist wherever it will run, and a long-lived daemon holding DDL credentials would -need its own liveness story for no gain. Every scenario below is the same four commands +need its own liveness story for no gain. Every scenario below is the same five commands in a different harness. The commands, and which ones change anything: @@ -14,6 +14,7 @@ The commands, and which ones change anything: | `inspect` | no | no | no | a terminal, a dashboard | | `plan` | no | no | no | review, monitoring (`--check`, `--output metrics`) | | `apply` | yes | yes | with `--allow-hooks` | a schedule, an init step | +| `backfill` | yes | yes | with `--allow-hooks` | a Job, once, when an existing table gets partitions | `apply` creates and re-attaches by default and retires nothing; `--allow-destructive` adds detaches and drops. That is the whole difference between "safe at startup" and @@ -491,7 +492,7 @@ production is the same alert as the CronJob version, from the other side. Airflow's `KubernetesPodOperator` or `DockerOperator`, Argo Workflows, Nomad's periodic jobs, Dagster, Rundeck: anything that runs a container and reads an exit code runs this. -The contract is the four commands, the three inputs, and the codes — nothing about the +The contract is the five commands, the three inputs, and the codes — nothing about the harness is assumed. Treat `6` as success and `2` (under `--check`) as a signal, not a failure, and the rest maps onto whatever the scheduler calls "failed". diff --git a/pg_partsmith/cli/__init__.py b/pg_partsmith/cli/__init__.py index ca2ed7d..c45e3a0 100644 --- a/pg_partsmith/cli/__init__.py +++ b/pg_partsmith/cli/__init__.py @@ -4,9 +4,10 @@ pip install "pg-partsmith[cli]" -Four commands over a configuration document and a DSN: ``inspect``, ``plan`` -and ``validate`` issue no DDL, and ``apply`` withholds every destructive -operation unless it is asked for them. ``plan`` and ``apply`` stay separable, +Five commands over a configuration document and a DSN: ``inspect``, ``plan`` +and ``validate`` issue no DDL, ``apply`` withholds every destructive +operation unless it is asked for them, and ``backfill`` moves a DEFAULT +partition's rows into the partitions they belong in. ``plan`` and ``apply`` stay separable, with a saved plan as the artifact between them: what this reads, prints and writes is the library's own :class:`~pg_partsmith.MaintenancePlan`, not a summary of one.