Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -66,4 +66,6 @@ tmp/
temp/

examples/
# Documentation examples are reviewed source, not local sample projects.
!docs/examples/
lambda-exporter/
13 changes: 13 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,19 @@ Changes are recorded in [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)

## [Unreleased]

## [0.8.0]

0.8 gives CI a machine-readable view of a migration project. `status`, `history`, `lint` and `diff` take `--json` and print one versioned document to stdout, and the README has a GitHub Actions job that fails a build when the live schema drifts from a committed snapshot. Without `--json` nothing changes: output and exit codes are the same as 0.7, and there is no `upgrade-env`.

Every document carries `"schema_version": 1`. The format can still change before 1.0.

To stay on 0.7, pin `ch-migrate-cli==0.7.0`.

### Added

- Versioned `--json` output and published JSON Schemas for status, history, lint, and diff, with machine-readable errors, graph-resolved applied revisions, explicit waiver reasons, and CI-friendly exit codes.
- A SHA-pinned GitHub Actions drift gate that preserves the JSON report on failure; its install, credential, comparison, and exit steps run against the integration fixture for both matching and out-of-band schemas.

## [0.7.0]

0.7 waits for the work a migration starts. `up` doesn't record a migration as applied until the mutations and `ON CLUSTER` DDL its statements started have finished, and it shows their progress while it waits. A normal `pip install -U` reaches it from 0.6, so read this before upgrading:
Expand Down
105 changes: 101 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@ This README describes the source checkout, which may be ahead of PyPI. To try an

## Upgrade from 0.x

**From 0.7:** 0.8 needs no `upgrade-env`, and nothing changes unless you pass
`--json` (see [Drift detection in CI](#drift-detection-in-ci)). A loose requirement
such as `ch-migrate-cli` upgrades to 0.8; pin `ch-migrate-cli==0.7.0` to stay on 0.7.

**From 0.6:** 0.7 needs no `upgrade-env`. `up` now waits for the mutations and
`ON CLUSTER` DDL its statements start (see [Migrations that wait](#migrations-that-wait)),
so give CI jobs time to finish or pass `--timeout`. Where `bootstrap` couldn't grant
Expand Down Expand Up @@ -473,12 +477,68 @@ def upgrade():
create_dictionary("history/dictionaries/dict_users/001_create.sql")
```

## Drift detection in CI

After applying and verifying migrations in an environment, capture its expected
schema:

```bash
ch-migrate snapshot dev
mv migrations/sql/snapshots/20261002_120000 migrations/sql/snapshots/dev
ch-migrate diff dev --snapshot-dir migrations/sql/snapshots/dev --json
```

`snapshot` writes a new timestamped directory every time; replace the example
timestamp with the one it reports. Give each environment's snapshot a fixed name,
such as `snapshots/dev`, so `CH_MIGRATE_SNAPSHOT_DIR` never changes and the next
snapshot shows up in review as a diff against this one. Review and commit that
directory alongside the migration head. Do not regenerate the snapshot
automatically in the drift job: doing so would accept an out-of-band change as the
new expected schema. Re-snapshot after an intentional migration or an explicitly
reviewed reconciliation, not merely to turn a failed check green. To re-snapshot,
run `snapshot` again, delete the old `snapshots/dev`, and move the new directory
into its place, so the commit also shows objects that were dropped.

Copy [the GitHub Actions example](docs/examples/github-actions-drift.yml) into the
consumer repository's `.github/workflows/`. It runs on a daily schedule and
same-repository pull requests; fork PRs are excluded because they have no database
secret. It does not use `pull_request_target`. All actions are pinned by commit.
The checkout must contain `config.yaml` and the committed snapshot, and the runner
must be able to reach the selected server.

Configure these repository variables:

| Variable | Value |
|---|---|
| `CH_MIGRATE_PACKAGE` | A pinned `ch-migrate-cli` requirement, such as `ch-migrate-cli==0.8.0`. 0.8.0 is the first release with `diff --json`; there is no fallback to an older one. |
| `CH_MIGRATE_ENV` | An environment name in `config.yaml`, such as `dev`. |
| `CH_MIGRATE_SNAPSHOT_DIR` | That environment's committed snapshot directory. |

Set the repository secret `DRIFT_MIGRATION_PASSWORD` to the password for that
environment's configured user. Prefer a dedicated schema-inspection account,
not an admin account. The job writes a private `.env.local` without printing the
value and removes the file even when comparison fails.

`diff --json` exits **0** for matching schemas, **1** for drift, and **2** for a
configuration/connection error. The workflow saves `drift.json`, uploads it even
on comparison failure, then fails the job for either nonzero result. The report
names changed tables and fields. ch-migrate's own test suite runs the example's
shell steps against a real ClickHouse server, installing from the source checkout,
for both a matching and a drifted schema.

Without `--snapshot-dir`, `diff` sorts entries in `migrations/sql/snapshots/` by
name and chooses the last one. It does not select a snapshot by environment or
recorded migration revision. Use an explicit path in CI, especially when one
repository holds snapshots for several environments.

## Command reference

Every command accepts `--help`. Top-level `ch-migrate --version` reports the installed package version. `ENV` below names an entry in `config.yaml`.

Output lines start with `→` for a step, `✓` for a result, `!` for a warning and `✗` for an error; warnings and errors go to stderr. Colour is dropped when output is not a terminal or `NO_COLOR` is set, and lines are never wrapped, so paths and SQL can be copied or grepped.

`status`, `history`, `lint`, and `diff` accept `--json`. JSON mode writes one document to stdout; diagnostics go to stderr. Every document includes `"schema_version": 1` and `"command"`. The format can still change before 1.0; `schema_version` says which one a document follows. Errors include an `error` string rather than inventing a successful empty result. The schemas linked below use JSON Schema draft 2020-12.

### `init`

`ch-migrate init [PATH] [-n NAME]` initializes the current directory by default. `-n/--name` sets the project name; otherwise it uses the directory name.
Expand Down Expand Up @@ -511,22 +571,53 @@ Example: `ch-migrate down dev --revision base`

### `status`

`ch-migrate status ENV` shows connection information, applied/pending counts, and head status, and names the `up` command when migrations are pending. No command-specific options. Status is a report: it exits 0 when the database is unreachable (with a warning) or migrations are pending, and exits 1 only when the configuration or `migrations/versions/` is missing, so CI can run it as a non-blocking check.
`ch-migrate status ENV [--json]` shows connection information, applied/pending counts, and head status, and names the `up` command when migrations are pending. Status is a report: human output exits 0 when the database is unreachable (with a warning) or migrations are pending, and exits 1 only when the configuration or `migrations/versions/` is missing, so CI can run it as a non-blocking check. JSON mode uses nonzero exits for unreachable or pending state.

Example: `ch-migrate status dev`

With `--json`, [the status schema](docs/schemas/status.schema.json) includes
`database`, `current_heads`, `script_heads`, `pending`, `applied`, and `at_head`.
`applied` includes ancestors resolved through the revision graph, not just stored
heads. JSON exits 0 when the head sets match exactly, 1 when pending or diverged,
and 2 when configuration or database state cannot be read. Human-mode exits are
unchanged.

```bash
ch-migrate status dev --json | jq -e .current_heads
# Check the actual head set, not the last element of applied:
ch-migrate status dev --json | jq -e '.current_heads == ["PINNED_REVISION"]'
```

Replace `PINNED_REVISION` with the required revision. In a shell pipeline, enable
`set -o pipefail` if the caller must also preserve ch-migrate's exit code.

### `history`

`ch-migrate history ENV` displays the revision graph and applied state. No command-specific options.
`ch-migrate history ENV [--json]` displays the revision graph and applied state.

Example: `ch-migrate history dev`

`ch-migrate history dev --json` follows [the history schema](docs/schemas/history.schema.json).
Each entry in `revisions` includes its revision, all `down_revisions` (including
merge parents), description, create date, path, applied status, and irreversible
reason. A reversible revision has `irreversible: null`. If database state is
unavailable, `applied` is null, the document includes `error`, and JSON exits 2.
An unknown database head also makes applied status unknown instead of claiming
that every local revision is unapplied.

### `lint`

`ch-migrate lint [ENV]` analyzes upgrade statements, not downgrade SQL. Without `ENV`, it checks revisions after the gate baseline statically without credentials or a connection. With an environment, it checks only pending revisions in that scope and adds live size and dependency checks. If it cannot determine the pending set, it fails rather than silently checking a different scope. No command-specific options. Errors exit nonzero; warnings and waiver INFO lines alone do not.
`ch-migrate lint [ENV] [--json]` analyzes upgrade statements, not downgrade SQL. Without `ENV`, it checks revisions after the gate baseline statically without credentials or a connection. With an environment, it checks only pending revisions in that scope and adds live size and dependency checks. If it cannot determine the pending set, it fails rather than silently checking a different scope. Errors exit nonzero; warnings and waiver INFO lines alone do not.

Example: `ch-migrate lint`

`ch-migrate lint --json` follows [the lint schema](docs/schemas/lint.schema.json).
`findings` include rule, severity, source file/line, message, `blocking`, and
`waived`. `blocking` means the finding currently blocks `up`; a waived finding
has its written reason and does not block. `counts` reports total, error, warning,
info, blocking, and waived findings. JSON and human output use the same exit
codes: 1 for lint/configuration errors, otherwise 0.

Findings name the project-relative SQL file and statement line. Inline Python SQL
points to its `op.execute` call. Extraction reads `run_sql`/`read_sql` file
references and literal or f-string `op.execute` arguments without importing
Expand All @@ -543,10 +634,16 @@ Example: `ch-migrate deps dev --validate migrations/sql/history/tables/logs/chan

### `diff`

`ch-migrate diff ENV [-s PATH]` compares the live schema with the latest snapshot. `-s/--snapshot-dir PATH` chooses another snapshot. Exit code 0 means no drift; 1 means drift or an execution error.
`ch-migrate diff ENV [-s PATH] [--json]` compares the live schema with the latest snapshot. `-s/--snapshot-dir PATH` chooses another snapshot. Exit code 0 means no drift; 1 means drift. JSON errors exit 2; human-mode execution errors retain exit 1.

Example: `ch-migrate diff dev --snapshot-dir migrations/sql/snapshots/20261002_120000`

`ch-migrate diff dev --json` follows [the diff schema](docs/schemas/diff.schema.json).
`objects` identifies each object's type, name, status, and structural `details`
(field, local value, remote value, and message). `in_sync` is true only when every
object matches. Use the exit code as the drift gate; an error document does not
contain a fabricated `in_sync: true`.

### `snapshot`

`ch-migrate snapshot ENV [-e GLOB] [-f GLOB]` writes CREATE statements to a timestamped snapshot directory. `-e/--exclude` and `-f/--filter` accept repeated glob patterns for excluded and included objects.
Expand Down
2 changes: 1 addition & 1 deletion ch_migrate/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
from ch_migrate import read_sql, get_db, get_env_config, create_dictionary
"""

__version__ = "0.7.0"
__version__ = "0.8.0"


from typing import Any
Expand Down
Loading
Loading