From 0a65c438904ea1caede6a2a56a2710cbb78f67da Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Mon, 27 Jul 2026 18:07:23 -0600 Subject: [PATCH] Document what a decoding slot actually does with a columnar table The published limitation said changes are not emitted through logical decoding and to use physical replication. True, and it leaves out the part a user finds out the hard way. A slot is not silent about a columnar table. The metadata catalog is ordinary heap, so a slot subscribed to everything delivers inserts and updates to pgcolumnar.storage, row_group, column_chunk, zone_map and delete_vector as the table is written, including encoded chunk descriptors as bytea, and not one of the table's own rows. Verified against a real slot on PG18 with test_decoding: zero changes named the columnar table, and the stream carried the internal bookkeeping instead. That is worse than silence, because it looks like the consumer is working. So the entry now says to filter the pgcolumnar schema out of a publication, and points at a recipe for capturing the changes themselves. The recipe is a row trigger writing to a heap table, which works because #182 made after-row triggers fire on columnar tables. Every statement in it was run before it was written down, and the sample output is copied from that run rather than composed. Four properties are documented because each is a thing a reader would otherwise have to discover: an UPDATE arrives as one UPDATE even though a columnar update is internally a delete and an insert; the capture is transactional, so a rolled back statement leaves nothing; it costs a heap row per changed row, which is worth thinking about before a bulk load; and nothing prunes the capture table. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_012uKWWwBDt5TWWS5DR2tzDb --- docs/limitations.md | 20 ++++++++++-- docs/user-guide.md | 80 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 97 insertions(+), 3 deletions(-) diff --git a/docs/limitations.md b/docs/limitations.md index 6f10216..b1ff85d 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -165,9 +165,23 @@ on whether the filter was pushed down. columnar tables, which are WAL-logged relations. - `pg_dump` and `pg_restore` handle columnar tables. The target server must have the extension installed and preloaded. -- Logical decoding reads heap-tuple WAL records. Changes to columnar tables are - not emitted through logical decoding, so logical replication does not carry - them. Use physical replication for columnar tables. +- Logical decoding reads heap-tuple WAL records. Columnar data reaches WAL as + full-page images, which carry no tuple structure, so changes to columnar + tables are not emitted through logical decoding and logical replication does + not carry them. Use physical replication for columnar tables. + + A decoding slot is not silent about a columnar table, which is the part worth + knowing before relying on one. The metadata catalog is made of ordinary heap + tables, so a slot delivers inserts and updates to `pgcolumnar.storage`, + `pgcolumnar.row_group`, `pgcolumnar.column_chunk`, `pgcolumnar.zone_map` and + `pgcolumnar.delete_vector` as the table is written. A consumer subscribed to + all tables therefore receives a stream of internal bookkeeping, including + encoded chunk descriptors as bytea, and none of the table's own rows. Filter + the `pgcolumnar` schema out of any publication. + + To capture changes from a columnar table today, use a row trigger that writes + to a heap table and decode that. The recipe is in + [user-guide.md](user-guide.md#capture-changes-for-replication-or-cdc). ## Import and export type coverage diff --git a/docs/user-guide.md b/docs/user-guide.md index f9cbc0a..30460ad 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -166,6 +166,86 @@ EXPLAIN (ANALYZE, COSTS OFF) SELECT id FROM events WHERE ts >= '2026-01-01'; -- Files: 4 ``` +## Capture changes for replication or CDC + +Logical decoding does not carry changes to a columnar table. Columnar data +reaches WAL as full-page images, which carry no tuple structure for a decoder to +read, so a replication slot never sees the rows. See +[limitations](limitations.md#replication-and-backup). + +A slot is not quiet about the table, though. The metadata catalog is made of +ordinary heap tables, so a slot subscribed to everything delivers the internal +bookkeeping and none of the data: + +``` +table pgcolumnar.row_group: INSERT: storage_id[bigint]:10000000000 group_number[bigint]:1 ... +table pgcolumnar.column_chunk: INSERT: ... encoding_descriptor[bytea]:'\x020001...' +table pgcolumnar.zone_map: INSERT: ... minimum[bytea]:'\x0100000000000000' ... +table pgcolumnar.delete_vector: UPDATE: ... bitmap[bytea]:'\x03' deleted_count[integer]:2 +``` + +Exclude the `pgcolumnar` schema from any publication. + +To capture the changes themselves, write them to a heap table with a row trigger +and decode that table. Row triggers fire on columnar tables and see the same +rows a heap table would. + +```sql +CREATE TABLE events (id bigint primary key, kind text, amount int) + USING pgcolumnar; + +CREATE TABLE events_cdc ( + seq bigserial primary key, + op text, + id bigint, + kind text, + amount int +); + +CREATE FUNCTION events_capture() RETURNS trigger LANGUAGE plpgsql AS $$ +BEGIN + IF TG_OP = 'DELETE' THEN + INSERT INTO events_cdc (op, id, kind, amount) + VALUES ('DELETE', OLD.id, OLD.kind, OLD.amount); + ELSE + INSERT INTO events_cdc (op, id, kind, amount) + VALUES (TG_OP, NEW.id, NEW.kind, NEW.amount); + END IF; + RETURN NULL; +END +$$; + +CREATE TRIGGER events_cdc_t AFTER INSERT OR UPDATE OR DELETE ON events + FOR EACH ROW EXECUTE FUNCTION events_capture(); +``` + +The capture table is a heap table, so a slot carries it: + +``` +table public.events_cdc: INSERT: seq[bigint]:1 op[text]:'INSERT' id[bigint]:1 kind[text]:'sale' amount[integer]:100 +table public.events_cdc: INSERT: seq[bigint]:3 op[text]:'UPDATE' id[bigint]:1 kind[text]:'sale' amount[integer]:150 +table public.events_cdc: INSERT: seq[bigint]:4 op[text]:'DELETE' id[bigint]:2 kind[text]:'refund' amount[integer]:50 +``` + +Four properties of this arrangement are worth knowing. + +An `UPDATE` arrives as one `UPDATE`. A columnar update is a delete of the old row +and an insert of a new one internally, but the trigger fires once per row event, +so the capture table records what the statement did rather than how the storage +did it. + +The capture is transactional. The trigger runs in the same transaction as the +statement, so a rolled back transaction leaves no captured rows, and the captured +rows commit with the data. + +It costs a heap row per changed row, in write time and in space. Bulk loads are +the case to think about before enabling it: a load of ten million rows writes ten +million heap rows as well. + +The capture table needs pruning. Nothing deletes from it. Delete rows once the +consumer has confirmed them, and remember that the deletes are themselves +decoded unless the consumer filters them. + ## Next steps - Operate tables in production: [Administration](administration.md).