diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b9b74f3..0f10121 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -133,8 +133,8 @@ uv run python scripts/sync_mirror.py uv run ruff check --fix pg_partsmith/sync && uv run ruff format pg_partsmith/sync ``` -The lock managers, `maintainer.py`, `repositories/{resolver,fk_manager,timeouts}.py` are -maintained by hand. +The lock managers, `maintainer.py`, `command_hooks.py`, `python_hooks.py` and +`repositories/{resolver,fk_manager,timeouts,pin}.py` are maintained by hand. The tests follow the same rule. `tests/integration/sync/` is generated from `tests/integration/aio/`: diff --git a/README.md b/README.md index b058846..361a242 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,7 @@ Three ways in, one version number: at their own modulus; partitions the scheme did not produce are reported, never touched; foreign tables are inspected, never dropped - **Async and sync** — `pg_partsmith.aio` on `AsyncEngine`, `pg_partsmith.sync` on `Engine` -- **A command line** — `pg-partsmith inspect / plan / validate / apply` over a YAML or +- **A command line** — `pg-partsmith inspect / plan / validate / apply / backfill` over a YAML or JSON document, with a saved plan as the artifact between plan and apply, and exit codes a CronJob and a CI step can read - **A container image** — `ghcr.io/bedrock-python/pg-partsmith`, for stacks with no Python @@ -187,7 +187,8 @@ LifecyclePolicy(retention=ExpireIf(AllOf((KeepNewest(count=2), ## What maintenance does -1. **Inspect** — one catalog round-trip reads the whole tree (`pg_partition_tree`), the +1. **Inspect** — one catalog round-trip reads the whole tree (a recursive walk over + `pg_inherits`, which still sees a half-detached partition), the marker-tagged detached orphans, and — only when a policy asks — sizes and row estimates. 2. **Plan** — the planner walks the scheme and the tree together. At a `RANGE` level it decides which windows must exist ahead of the cursor (the clock, or `max(key)` for an @@ -212,7 +213,8 @@ test, not a promise. - A hash set at a modulus the config no longer uses is preserved if complete and repaired *at its own modulus* if not; mixed moduli leaving a gap are reported, never guessed at. - A plan made at 10:00 and applied at 10:05 refuses to drop a table that was recreated in - between (`PlanStaleError`, reported as an issue). + between (`PlanStaleError`: the run stops there, or records the issue under + `continue_on_error`). ## Sync usage @@ -236,8 +238,10 @@ service = PartitionLifecycleService( result = PartitionMaintainer(service).run_maintenance_safe(config) ``` -Two behavioural differences: `ddl_timeout_seconds` is enforced server-side via -`statement_timeout` per statement, and the Redis lock renews from a background thread. +Three behavioural differences: `ddl_timeout_seconds` is enforced server-side via +`statement_timeout` per statement, the Redis lock renews from a background thread and can +only warn when the lease is lost, and a Python block hook runs inline rather than on a +thread of its own. ## Hooks @@ -267,7 +271,9 @@ planned and the size the policy measured, when it asked for one. | `before_drop` / `after_drop` | around a drop — `before_drop` is the last chance to read the data | | `on_event` | every one of the above, for an audit trail or metrics in one method | -`before_*` exceptions abort that operation; `after_*` exceptions are logged. +`before_*` exceptions abort that operation; `after_*` exceptions are logged and re-raised +too — the operation already happened, but the run stops there (or records the issue under +`continue_on_error`). ## Documentation diff --git a/SECURITY.md b/SECURITY.md index 25cdc44..af6541e 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -4,7 +4,7 @@ | Version | Supported | |---------|-----------| -| 1.2.x | ✅ the current release; fixes ship as the next patch or minor | +| 1.5.x | ✅ the current release; fixes ship as the next patch or minor | | older | ❌ upgrade to the current release first | The library, the command line and the container image share one version number, so a fix diff --git a/docs/agents.md b/docs/agents.md index 9f1b8c9..8a0bad2 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -7,7 +7,7 @@ | | | |---|---| | Package | `pg-partsmith` on PyPI, import root `pg_partsmith` | -| Requires | Python 3.11+, PostgreSQL 15+ (tested on 15, 16 and 17), SQLAlchemy 2 | +| Requires | Python 3.11+, PostgreSQL 15+ (tested on 15 through 18), SQLAlchemy 2 | | Install | `pip install pg-partsmith` · extras: `cli`, `redis-locks`, `pydantic-settings` | | Async | `pg_partsmith.aio`, on an `AsyncEngine` | | Sync | `pg_partsmith.sync` — same class names, same arguments, no `await` | diff --git a/docs/getting-started/installation.md b/docs/getting-started/installation.md index 5f8be6c..c2e175e 100644 --- a/docs/getting-started/installation.md +++ b/docs/getting-started/installation.md @@ -5,7 +5,7 @@ | | | |---|---| | Python | 3.11 or newer | -| PostgreSQL | 15 or newer — the integration suite runs on 15, 16 and 17 | +| PostgreSQL | 15 or newer — the integration suite runs on 15 through 18 | | SQLAlchemy | 2.x, with `asyncpg` for the async API or `psycopg2` for the sync one | | Privileges | ownership of the partitioned table (to create, attach, detach and drop partitions) — no superuser | diff --git a/docs/guide/archiving.md b/docs/guide/archiving.md index 568e6f6..15ca6f7 100644 --- a/docs/guide/archiving.md +++ b/docs/guide/archiving.md @@ -27,7 +27,7 @@ Every one of them takes the same `PartitionEvent`: | Field | What it is | |---|---| -| `phase` | which of the six moments this is | +| `phase` | which of the eight moments this is | | `config` | the table's configuration — the calendar, the codec, the policy in force | | `partition` | the partition itself: `name`, `bounds`, `oid`, `subpartition_type` | | `window` | the period it covers; `None` for a member of a root `HASH` or `LIST` | @@ -64,7 +64,7 @@ logs. ### One method for every phase -An audit trail or a metrics counter wants all six moments and treats them alike. That is +An audit trail or a metrics counter wants all eight moments and treats them alike. That is `on_event`, which fires for every phase in addition to the method named after it — so a hook implementing both is called twice, on purpose: diff --git a/docs/guide/cli.md b/docs/guide/cli.md index 0ac826b..090148a 100644 --- a/docs/guide/cli.md +++ b/docs/guide/cli.md @@ -46,8 +46,9 @@ What a stopped run leaves behind is what the library promises for any cancellati most a detached, unattached table (attach is the last step of a creation) or a marked, half-detached partition, and the next run converges both. -A block of Python in a hook cannot be interrupted mid-way — the signal is honoured the -moment the block returns. A command can, and is. +A block of Python in a hook runs on a thread of its own, so a stop does not wait for it: +the run is cancelled and exits, and a block still running at that point is abandoned with +the process. A command's child is terminated on purpose. ## Shell completion @@ -180,9 +181,8 @@ therefore reviewable for its locks without the CLI. **Why there is no `--sql`.** Printing the statements would be the obvious next thing, and it is deliberately not offered. The DDL is built at execution time from decisions that are -only made then — whether a `DETACH CONCURRENTLY` has to fall back to the blocking form -because a DEFAULT sibling exists, how rows a DEFAULT partition holds for a new window get -moved, which orphan marker gets cleared — and a rendering that left those out would be +only made then — how rows a DEFAULT partition holds for a new window get moved, which +orphan marker gets cleared — and a rendering that left those out would be read as a transcript in exactly the cases a review exists to catch. The plan is the contract: what, why, how big, what it locks. diff --git a/docs/guide/container.md b/docs/guide/container.md index d2a8d3c..a90da3d 100644 --- a/docs/guide/container.md +++ b/docs/guide/container.md @@ -66,12 +66,12 @@ its own rather than the application's: ```sql CREATE ROLE partsmith LOGIN PASSWORD '…'; --- Enough to create partitions in the schema and attach them +-- The schema: needed, and not enough on its own GRANT USAGE, CREATE ON SCHEMA public TO partsmith; --- Enough to detach and drop them: PostgreSQL requires ownership of both the --- partition and the parent for ATTACH/DETACH, so the role owns the tables it --- maintains, or is a member of the role that does. +-- The tables: PostgreSQL lets only the parent's owner run CREATE TABLE … PARTITION OF, +-- ATTACH and DETACH, and only a table's owner drop it, so the role owns the tables +-- it maintains, or is a member of the role that does. ALTER TABLE public.events OWNER TO partsmith; -- or: GRANT app_owner TO partsmith; ``` diff --git a/docs/guide/running.md b/docs/guide/running.md index 442b446..8f7b3bc 100644 --- a/docs/guide/running.md +++ b/docs/guide/running.md @@ -3,7 +3,7 @@ `pg-partsmith` is a process that runs one command and exits with a code. It has no scheduler of its own, on purpose: cron, a CronJob, a CI schedule or a systemd timer already exist wherever it will run, and a long-lived daemon holding DDL credentials would -need its own liveness story for no gain. Every scenario below is the same four commands +need its own liveness story for no gain. Every scenario below is the same five commands in a different harness. The commands, and which ones change anything: @@ -14,6 +14,7 @@ The commands, and which ones change anything: | `inspect` | no | no | no | a terminal, a dashboard | | `plan` | no | no | no | review, monitoring (`--check`, `--output metrics`) | | `apply` | yes | yes | with `--allow-hooks` | a schedule, an init step | +| `backfill` | yes | yes | with `--allow-hooks` | a Job, once, when an existing table gets partitions | `apply` creates and re-attaches by default and retires nothing; `--allow-destructive` adds detaches and drops. That is the whole difference between "safe at startup" and @@ -491,7 +492,7 @@ production is the same alert as the CronJob version, from the other side. Airflow's `KubernetesPodOperator` or `DockerOperator`, Argo Workflows, Nomad's periodic jobs, Dagster, Rundeck: anything that runs a container and reads an exit code runs this. -The contract is the four commands, the three inputs, and the codes — nothing about the +The contract is the five commands, the three inputs, and the codes — nothing about the harness is assumed. Treat `6` as success and `2` (under `--check`) as a signal, not a failure, and the rest maps onto whatever the scheduler calls "failed". diff --git a/pg_partsmith/cli/__init__.py b/pg_partsmith/cli/__init__.py index ca2ed7d..c45e3a0 100644 --- a/pg_partsmith/cli/__init__.py +++ b/pg_partsmith/cli/__init__.py @@ -4,9 +4,10 @@ pip install "pg-partsmith[cli]" -Four commands over a configuration document and a DSN: ``inspect``, ``plan`` -and ``validate`` issue no DDL, and ``apply`` withholds every destructive -operation unless it is asked for them. ``plan`` and ``apply`` stay separable, +Five commands over a configuration document and a DSN: ``inspect``, ``plan`` +and ``validate`` issue no DDL, ``apply`` withholds every destructive +operation unless it is asked for them, and ``backfill`` moves a DEFAULT +partition's rows into the partitions they belong in. ``plan`` and ``apply`` stay separable, with a saved plan as the artifact between them: what this reads, prints and writes is the library's own :class:`~pg_partsmith.MaintenancePlan`, not a summary of one.