Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,8 +133,8 @@ uv run python scripts/sync_mirror.py
uv run ruff check --fix pg_partsmith/sync && uv run ruff format pg_partsmith/sync
```

The lock managers, `maintainer.py`, `repositories/{resolver,fk_manager,timeouts}.py` are
maintained by hand.
The lock managers, `maintainer.py`, `command_hooks.py`, `python_hooks.py` and
`repositories/{resolver,fk_manager,timeouts,pin}.py` are maintained by hand.

The tests follow the same rule. `tests/integration/sync/` is generated from
`tests/integration/aio/`:
Expand Down
18 changes: 12 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ Three ways in, one version number:
at their own modulus; partitions the scheme did not produce are reported, never touched;
foreign tables are inspected, never dropped
- **Async and sync** — `pg_partsmith.aio` on `AsyncEngine`, `pg_partsmith.sync` on `Engine`
- **A command line** — `pg-partsmith inspect / plan / validate / apply` over a YAML or
- **A command line** — `pg-partsmith inspect / plan / validate / apply / backfill` over a YAML or
JSON document, with a saved plan as the artifact between plan and apply, and exit codes
a CronJob and a CI step can read
- **A container image** — `ghcr.io/bedrock-python/pg-partsmith`, for stacks with no Python
Expand Down Expand Up @@ -187,7 +187,8 @@ LifecyclePolicy(retention=ExpireIf(AllOf((KeepNewest(count=2),

## What maintenance does

1. **Inspect** — one catalog round-trip reads the whole tree (`pg_partition_tree`), the
1. **Inspect** — one catalog round-trip reads the whole tree (a recursive walk over
`pg_inherits`, which still sees a half-detached partition), the
marker-tagged detached orphans, and — only when a policy asks — sizes and row estimates.
2. **Plan** — the planner walks the scheme and the tree together. At a `RANGE` level it
decides which windows must exist ahead of the cursor (the clock, or `max(key)` for an
Expand All @@ -212,7 +213,8 @@ test, not a promise.
- A hash set at a modulus the config no longer uses is preserved if complete and repaired
*at its own modulus* if not; mixed moduli leaving a gap are reported, never guessed at.
- A plan made at 10:00 and applied at 10:05 refuses to drop a table that was recreated in
between (`PlanStaleError`, reported as an issue).
between (`PlanStaleError`: the run stops there, or records the issue under
`continue_on_error`).

## Sync usage

Expand All @@ -236,8 +238,10 @@ service = PartitionLifecycleService(
result = PartitionMaintainer(service).run_maintenance_safe(config)
```

Two behavioural differences: `ddl_timeout_seconds` is enforced server-side via
`statement_timeout` per statement, and the Redis lock renews from a background thread.
Three behavioural differences: `ddl_timeout_seconds` is enforced server-side via
`statement_timeout` per statement, the Redis lock renews from a background thread and can
only warn when the lease is lost, and a Python block hook runs inline rather than on a
thread of its own.

## Hooks

Expand Down Expand Up @@ -267,7 +271,9 @@ planned and the size the policy measured, when it asked for one.
| `before_drop` / `after_drop` | around a drop — `before_drop` is the last chance to read the data |
| `on_event` | every one of the above, for an audit trail or metrics in one method |

`before_*` exceptions abort that operation; `after_*` exceptions are logged.
`before_*` exceptions abort that operation; `after_*` exceptions are logged and re-raised
too — the operation already happened, but the run stops there (or records the issue under
`continue_on_error`).

## Documentation

Expand Down
2 changes: 1 addition & 1 deletion SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

| Version | Supported |
|---------|-----------|
| 1.2.x | ✅ the current release; fixes ship as the next patch or minor |
| 1.5.x | ✅ the current release; fixes ship as the next patch or minor |
| older | ❌ upgrade to the current release first |

The library, the command line and the container image share one version number, so a fix
Expand Down
2 changes: 1 addition & 1 deletion docs/agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
| | |
|---|---|
| Package | `pg-partsmith` on PyPI, import root `pg_partsmith` |
| Requires | Python 3.11+, PostgreSQL 15+ (tested on 15, 16 and 17), SQLAlchemy 2 |
| Requires | Python 3.11+, PostgreSQL 15+ (tested on 15 through 18), SQLAlchemy 2 |
| Install | `pip install pg-partsmith` · extras: `cli`, `redis-locks`, `pydantic-settings` |
| Async | `pg_partsmith.aio`, on an `AsyncEngine` |
| Sync | `pg_partsmith.sync` — same class names, same arguments, no `await` |
Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
| | |
|---|---|
| Python | 3.11 or newer |
| PostgreSQL | 15 or newer — the integration suite runs on 15, 16 and 17 |
| PostgreSQL | 15 or newer — the integration suite runs on 15 through 18 |
| SQLAlchemy | 2.x, with `asyncpg` for the async API or `psycopg2` for the sync one |
| Privileges | ownership of the partitioned table (to create, attach, detach and drop partitions) — no superuser |

Expand Down
4 changes: 2 additions & 2 deletions docs/guide/archiving.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Every one of them takes the same `PartitionEvent`:

| Field | What it is |
|---|---|
| `phase` | which of the six moments this is |
| `phase` | which of the eight moments this is |
| `config` | the table's configuration — the calendar, the codec, the policy in force |
| `partition` | the partition itself: `name`, `bounds`, `oid`, `subpartition_type` |
| `window` | the period it covers; `None` for a member of a root `HASH` or `LIST` |
Expand Down Expand Up @@ -64,7 +64,7 @@ logs.

### One method for every phase

An audit trail or a metrics counter wants all six moments and treats them alike. That is
An audit trail or a metrics counter wants all eight moments and treats them alike. That is
`on_event`, which fires for every phase in addition to the method named after it — so a
hook implementing both is called twice, on purpose:

Expand Down
10 changes: 5 additions & 5 deletions docs/guide/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,9 @@ What a stopped run leaves behind is what the library promises for any cancellati
most a detached, unattached table (attach is the last step of a creation) or a marked,
half-detached partition, and the next run converges both.

A block of Python in a hook cannot be interrupted mid-way — the signal is honoured the
moment the block returns. A command can, and is.
A block of Python in a hook runs on a thread of its own, so a stop does not wait for it:
the run is cancelled and exits, and a block still running at that point is abandoned with
the process. A command's child is terminated on purpose.

## Shell completion

Expand Down Expand Up @@ -180,9 +181,8 @@ therefore reviewable for its locks without the CLI.

**Why there is no `--sql`.** Printing the statements would be the obvious next thing, and
it is deliberately not offered. The DDL is built at execution time from decisions that are
only made then — whether a `DETACH CONCURRENTLY` has to fall back to the blocking form
because a DEFAULT sibling exists, how rows a DEFAULT partition holds for a new window get
moved, which orphan marker gets cleared — and a rendering that left those out would be
only made then — how rows a DEFAULT partition holds for a new window get moved, which
orphan marker gets cleared — and a rendering that left those out would be
read as a transcript in exactly the cases a review exists to catch. The plan is the
contract: what, why, how big, what it locks.

Expand Down
8 changes: 4 additions & 4 deletions docs/guide/container.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,12 +66,12 @@ its own rather than the application's:
```sql
CREATE ROLE partsmith LOGIN PASSWORD '…';

-- Enough to create partitions in the schema and attach them
-- The schema: needed, and not enough on its own
GRANT USAGE, CREATE ON SCHEMA public TO partsmith;

-- Enough to detach and drop them: PostgreSQL requires ownership of both the
-- partition and the parent for ATTACH/DETACH, so the role owns the tables it
-- maintains, or is a member of the role that does.
-- The tables: PostgreSQL lets only the parent's owner run CREATE TABLE … PARTITION OF,
-- ATTACH and DETACH, and only a table's owner drop it, so the role owns the tables
-- it maintains, or is a member of the role that does.
ALTER TABLE public.events OWNER TO partsmith;
-- or: GRANT app_owner TO partsmith;
```
Expand Down
5 changes: 3 additions & 2 deletions docs/guide/running.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
`pg-partsmith` is a process that runs one command and exits with a code. It has no
scheduler of its own, on purpose: cron, a CronJob, a CI schedule or a systemd timer
already exist wherever it will run, and a long-lived daemon holding DDL credentials would
need its own liveness story for no gain. Every scenario below is the same four commands
need its own liveness story for no gain. Every scenario below is the same five commands
in a different harness.

The commands, and which ones change anything:
Expand All @@ -14,6 +14,7 @@ The commands, and which ones change anything:
| `inspect` | no | no | no | a terminal, a dashboard |
| `plan` | no | no | no | review, monitoring (`--check`, `--output metrics`) |
| `apply` | yes | yes | with `--allow-hooks` | a schedule, an init step |
| `backfill` | yes | yes | with `--allow-hooks` | a Job, once, when an existing table gets partitions |

`apply` creates and re-attaches by default and retires nothing; `--allow-destructive`
adds detaches and drops. That is the whole difference between "safe at startup" and
Expand Down Expand Up @@ -491,7 +492,7 @@ production is the same alert as the CronJob version, from the other side.

Airflow's `KubernetesPodOperator` or `DockerOperator`, Argo Workflows, Nomad's periodic
jobs, Dagster, Rundeck: anything that runs a container and reads an exit code runs this.
The contract is the four commands, the three inputs, and the codes — nothing about the
The contract is the five commands, the three inputs, and the codes — nothing about the
harness is assumed. Treat `6` as success and `2` (under `--check`) as a signal, not a
failure, and the rest maps onto whatever the scheduler calls "failed".

Expand Down
7 changes: 4 additions & 3 deletions pg_partsmith/cli/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,10 @@

pip install "pg-partsmith[cli]"

Four commands over a configuration document and a DSN: ``inspect``, ``plan``
and ``validate`` issue no DDL, and ``apply`` withholds every destructive
operation unless it is asked for them. ``plan`` and ``apply`` stay separable,
Five commands over a configuration document and a DSN: ``inspect``, ``plan``
and ``validate`` issue no DDL, ``apply`` withholds every destructive
operation unless it is asked for them, and ``backfill`` moves a DEFAULT
partition's rows into the partitions they belong in. ``plan`` and ``apply`` stay separable,
with a saved plan as the artifact between them: what this reads, prints and
writes is the library's own :class:`~pg_partsmith.MaintenancePlan`, not a
summary of one.
Expand Down
Loading