From 17055157e0f6d878a7aff04cf2f98e5fc57e59d6 Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Mon, 27 Jul 2026 17:48:39 -0600 Subject: [PATCH 1/2] Plan: logical decoding for columnar tables A proposal for review, not an implementation. Written because the limitation it addresses is the one in docs/limitations.md most likely to be found in production rather than in the manual: a user with logical replication or a CDC consumer gets silence rather than an error, and nothing warns them. Records what was verified rather than assumed, in the PostgreSQL 17 tree: logical decoding dispatches per resource manager through RmgrData.rm_decode; a decoded change is heap-shaped but a HeapTuple can be formed from values that never lived on a heap page; and every byte of our data reaches WAL as a full-page image, which carries no tuple structure and no decode path. The metadata catalog is heap and is already decoded, so a consumer can see row groups appear today and cannot see a single user value. Five options, with the reason each is or is not the answer. The obvious one -- a custom resource manager with rm_decode -- is stated in full and rejected on the project's standing WAL constraint rather than omitted, because someone will propose it again otherwise. The recommendation is LogLogicalMessage: a core mechanism using an existing record type, decoded by core and delivered to output plugins. It satisfies the constraint for the exact reason the constraint exists, which is that replay of a logical message is a no-op, so a standby running a stock binary without this extension recovers correctly and ignores it. Says plainly what it does not give, because "logical decoding works" will be read as "logical replication works" and it will not: pgoutput delivers messages as messages, and a subscriber will not turn them into rows without a consumer that understands the payload. Also records the answer available today, since #182 made after-row triggers work on columnar tables: capture into a heap side-table with an ordinary trigger. That belongs in the documentation whatever we build. The test that matters most is not the happy path. Compaction, recluster and vacuum move rows between groups without changing table contents, and must emit nothing; a rewrite that replays as a stream of inserts would silently duplicate the table on the consumer. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_012uKWWwBDt5TWWS5DR2tzDb --- design/LOGICAL_DECODING_PLAN.md | 238 ++++++++++++++++++++++++++++++++ 1 file changed, 238 insertions(+) create mode 100644 design/LOGICAL_DECODING_PLAN.md diff --git a/design/LOGICAL_DECODING_PLAN.md b/design/LOGICAL_DECODING_PLAN.md new file mode 100644 index 0000000..459e7f9 --- /dev/null +++ b/design/LOGICAL_DECODING_PLAN.md @@ -0,0 +1,238 @@ +# Logical decoding for columnar tables + +Status: proposal, for review. Nothing here is implemented. + +## What is missing today + +`docs/limitations.md` says it plainly: + +> Logical decoding reads heap-tuple WAL records. Changes to columnar tables are +> not emitted through logical decoding, so logical replication does not carry +> them. Use physical replication for columnar tables. + +That is accurate and it is worse in practice than it reads. A user with logical +replication, or any CDC consumer, gets **silence** rather than an error: the +columnar table's changes simply never appear in the stream. Nothing warns them. +Of everything in that file this is the entry most likely to be discovered in +production rather than in the manual. + +## Why it does not already work + +Two facts, both verified in the PostgreSQL 17 tree rather than assumed. + +**Logical decoding is heap-shaped at the point where a change is represented.** +`ReorderBufferChange` holds `HeapTuple oldtuple` / `HeapTuple newtuple`. That is +less limiting than it sounds -- a `HeapTuple` can be built with +`heap_form_tuple` from values that never lived on a heap page -- but it means +every path into the reorder buffer is a path core owns. + +**Dispatch is per resource manager.** `LogicalDecodingProcessRecord` calls +`rmgr.rm_decode`, and `RmgrData` does carry that callback: + +```c +typedef struct RmgrData +{ + ... + void (*rm_decode) (struct LogicalDecodingContext *ctx, + struct XLogRecordBuffer *buf); +} RmgrData; +``` + +So the record types that can produce logical changes are exactly the ones whose +rmgr implements `rm_decode`. + +**Our data does not reach WAL as tuples.** Every byte of columnar data reaches +WAL through `log_newpage` and `log_newpage_buffer` as full-page images, plus one +`XLOG_SMGR_TRUNCATE`. `test/wal_envelope.sh` asserts that by source shape and +exists to keep it true. A full-page image has no tuple structure and no decode +path, so the row data is *in* the WAL stream and is not *reachable* from it. + +The metadata catalog is a different story: `pgcolumnar.row_group`, +`column_chunk`, `delete_vector` and `storage` are ordinary heap tables, so their +changes **are** decoded today. A CDC consumer subscribed to them sees row groups +appear and delete vectors change. It just cannot see any user data. + +## Options + +Five, with the reason each is or is not the answer. The first is the obvious one +and it is rejected on policy, so it is stated first and in full rather than +omitted. + +### A. A custom resource manager with `rm_decode` + +Technically available: `RegisterCustomRmgr()` exists from PostgreSQL 15, and +`rm_decode` is the documented way a resource manager produces logical changes. +This is how one would do it with a free hand. + +**Rejected**, and not merely deferred. `test/wal_envelope.sh` states the standing +constraint and its reasoning: + +> An extension that defines a custom resource manager or a new record type +> couples the WAL stream to its own version, which breaks a standby or a recovery +> that does not have the same build loaded and turns an extension bug into an +> unrecoverable cluster. + +That reasoning holds. A cluster whose WAL contains records only our extension can +replay is a cluster that cannot be recovered by a stock `postgres` binary. For a +storage extension that is the wrong risk to take for a feature, however good the +feature. It also doubles the WAL cost of every write, since the data is already +logged as full-page images. + +### B. Derive changes from the metadata catalog, read the data at decode time + +The catalog changes are decoded already. On seeing a committed `row_group` row, a +decoder knows exactly which row numbers a transaction added, and row group bytes +are immutable once written, so re-reading them yields what was committed whenever +the read happens. + +**Rejected**, for two reasons rather than one. + +It breaks the invariant that decoding is a function of the WAL stream alone. And +the practical consequence is a retention hazard with no good answer: compaction +and `pgcolumnar.vacuum` may rewrite or reclaim a row group before a lagging slot +has read it, at which point the change is unreconstructable and the stream is +silently short. Heap has no equivalent problem because the tuple data is in the +WAL record. Solving it would mean teaching every reclamation path about +replication slots -- a large amount of new coupling for a feature that has a +cheaper route. + +### C. Decode the full-page images + +The data really is in the WAL. Reassembling row groups from `XLOG_FPI` records +would be decoding from WAL alone, with no retention hazard. + +**Not available to an extension.** Those records belong to `RM_XLOG_ID`, whose +`rm_decode` is core's, and there is no hook to extend it. This would be a core +patch, not an extension feature. + +### D. Emit logical decoding messages — recommended + +`LogLogicalMessage()` is a **core mechanism using an existing record type** +(`RM_LOGICALMSG_ID` / `XLOG_LOGICAL_MESSAGE`). Core decodes it +(`logicalmsg_decode`) and delivers it to output plugins through the documented +message callback. `pgoutput` forwards messages when the subscription asks for +them. + +It satisfies the constraint that rules out A, and does so for the exact reason +the constraint exists: **replay of a logical message is a no-op**, so a standby +or a recovery running a stock binary without this extension loaded replays the +WAL correctly and ignores the message. Nothing about the cluster's +recoverability depends on our build. + +Available on every supported major. The signature gained a `flush` argument +after 15, so it needs one entry in `columnar_compat.h` beside the others: + +```c +/* 15 */ LogLogicalMessage(prefix, message, size, transactional) +/* 16+ */ LogLogicalMessage(prefix, message, size, transactional, flush) +``` + +### E. Triggers, in user space — available today, and the interim answer + +Since #182 made `AFTER ... FOR EACH ROW` triggers work on columnar tables, a user +can capture changes into a heap side-table with an ordinary trigger, and that +heap table is decoded normally. It costs write amplification and it is entirely +manual, but it needs nothing from us and it works now. + +**This should go in the documentation regardless of what we build**, because it +is the honest answer to "what do I do today", and because it will remain the +right answer for anyone who wants changes delivered as ordinary row changes to a +subscriber table. + +## The recommended design in detail + +### Shape + +Per-table opt-in through `pgcolumnar.set_options`, defaulting to off, because it +costs WAL volume and most tables do not need it. A GUC alone would be wrong here +for the reason #170's review gives: the same table would be written differently +depending on which session loaded it. + +On flush of a row group belonging to an opted-in table, emit one transactional +logical message per row group carrying the rows that group added. On flush of a +delete vector, emit one carrying the deleted row numbers and their identity +columns. Update is delete plus insert, which is what the access method already +does, so it needs no separate case. + +`transactional = true` throughout, so messages appear in commit order inside the +transaction that produced them and vanish if it aborts. That is the property that +makes this correct rather than approximate, and it is core's to provide. + +### Payload + +A versioned binary payload under a fixed prefix (`pgcolumnar`), holding: format +version, relation identity, operation, the row-number range, and the row data. + +For the row data, **reuse the Arrow IPC encoder in `columnar_arrow.c`**. It is +already written, already tested by four suites, self-describing, and consumers +have libraries for it. Inventing a payload format for this would be a second +serialization format in the tree to keep correct. + +### What it gives, and what it does not + +It gives a correct, ordered, transactional change stream that a CDC consumer can +read. + +It does **not** give native logical replication into a subscriber table. +`pgoutput` delivers messages as messages; a subscriber will not turn them into +rows in a table by itself. Anyone wanting that needs a consumer that understands +the payload. This has to be said plainly in the documentation, because +"logical decoding works now" would be read as "logical replication works now" +and it does not. + +### Cost + +Row data is written to WAL twice: once as the full-page images that make the +storage crash-safe, and once in the message. That is the price of the constraint, +it is per-table and opt-in, and it should be measured and published rather than +described. On the shapes in #155 the write path is already the weak spot, so this +must be benchmarked before it is recommended for anything. + +## Risks and open questions + +1. **Message size.** A row group is 150,000 rows by default. One message per row + group may be very large. Batching per N rows within the group is probably + necessary; what N, and does it interact badly with `logical_decoding_work_mem`? +2. **`TRUNCATE` and DDL.** `TRUNCATE` on a columnar table, and `ALTER TABLE` + shapes that rewrite, need a decided answer rather than an accident. +3. **Rewrites that are not user changes.** `pgcolumnar.vacuum`, `compact` and + `recluster` move rows between groups without changing table contents. These + must emit nothing, and the test for that matters more than the test for the + happy path -- a compaction that replays as a stream of inserts would silently + duplicate a table on the consumer. +4. **Identity for deletes.** A delete vector records row numbers. A consumer needs + the key columns, so the identity columns have to be read at flush time. What + is the analogue of `REPLICA IDENTITY` here, and what happens with no primary + key? +5. **Import paths.** `import_arrow` / `import_parquet` insert rows without the + executor. They must emit too, or an import is a silent gap in the stream. + +## Test plan + +The oracle is heap. For an identical workload against a heap table and an +opted-in columnar table, the change streams from +`pg_logical_slot_get_changes` must describe the same rows in the same order. + +Discriminating cases, which matter more than the happy path: + +- a transaction that **aborts** emits nothing +- `compact` / `recluster` / `vacuum` emit **nothing**, which is the one that + turns a subtle bug into visible duplication on the consumer +- an import emits the imported rows +- a delete emits identity, not just a row number +- `wal_envelope.sh` still passes, which is what proves this did not reach for a + custom rmgr +- volume: a large insert does not exceed `logical_decoding_work_mem` in a way + that spills unboundedly + +## Phasing + +1. Per-table option, plumbing, and inserts only, with the abort and + no-op-on-rewrite tests. This is where the design is proven or not. +2. Deletes and updates, with identity. +3. Import paths, `TRUNCATE`, and DDL. +4. A reference consumer, and the documentation that says what this is not. + +Phase 1 is the one worth doing before deciding whether the rest is worth it: if +the WAL cost measured there is unacceptable, that is the answer, and it is +cheaper to learn it at phase 1 than at phase 4. From 0665ff52696ae6c020cce150a6083dea09cdc99a Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Mon, 27 Jul 2026 18:25:04 -0600 Subject: [PATCH 2/2] =?UTF-8?q?Plan:=20fold=20in=20review=20=E2=80=94=20op?= =?UTF-8?q?tion=20B,=20phasing,=20the=20rewrite=20invariant,=20and=20the?= =?UTF-8?q?=20compat=20boundary?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four changes from ChronicallyJD's review on #187. Option B is no longer left looking impossible. It was rejected on a retention hazard, and the answer is not to teach reclamation to retain but to teach it to invalidate, which is the trade core already makes for WAL a slot still needs: one comparison at reclaim time, and a visibly broken slot instead of a silently short stream. B is still not the recommendation, but for the two reasons that survive rather than for the one that does not. The phasing objection is accepted and it was load-bearing. An earlier draft had phase 1 shipping a usable option over inserts only, and a consumer that receives inserts but not updates does not get a partial answer, it gets a wrong one that looks complete. The option must lag the capability: either it refuses to enable until update and delete are covered, or it is named emit_inserts and never logical_decoding. The rewrite risk is stated once as an invariant rather than three cases -- anything that moves rows without changing table contents emits nothing -- with the test's required shape, because asserting that a rewrite raises no error passes on a build that emits a full stream of inserts. The compat boundary is now the version it actually changed at. LogLogicalMessage gained flush at PostgreSQL 17, not 'after 15': verified in the headers of all five majors. It gets its own macro. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_012uKWWwBDt5TWWS5DR2tzDb --- design/LOGICAL_DECODING_PLAN.md | 110 ++++++++++++++++++++++++-------- 1 file changed, 85 insertions(+), 25 deletions(-) diff --git a/design/LOGICAL_DECODING_PLAN.md b/design/LOGICAL_DECODING_PLAN.md index 459e7f9..3429627 100644 --- a/design/LOGICAL_DECODING_PLAN.md +++ b/design/LOGICAL_DECODING_PLAN.md @@ -11,11 +11,24 @@ Status: proposal, for review. Nothing here is implemented. > them. Use physical replication for columnar tables. That is accurate and it is worse in practice than it reads. A user with logical -replication, or any CDC consumer, gets **silence** rather than an error: the -columnar table's changes simply never appear in the stream. Nothing warns them. +replication, or any CDC consumer, gets no error and no data. Nothing warns them. Of everything in that file this is the entry most likely to be discovered in production rather than in the manual. +"Silence" describes the signal and not the stream. Measured on a real slot with +`test_decoding`, over two inserts, an update and a delete: + +``` +changes naming the columnar table 0 +changes naming pgcolumnar internal catalogs 33 +changes naming an equivalent heap table 2 +``` + +The consumer receives sixteen times more rows about our bookkeeping than a heap +table produces about the user's data, and none of them are the user's. A stream +that looks busy and carries nothing is harder to notice than one that carries +nothing at all. `docs/limitations.md` and `docs/user-guide.md` now say so (#190). + ## Why it does not already work Two facts, both verified in the PostgreSQL 17 tree rather than assumed. @@ -85,16 +98,29 @@ decoder knows exactly which row numbers a transaction added, and row group bytes are immutable once written, so re-reading them yields what was committed whenever the read happens. -**Rejected**, for two reasons rather than one. - -It breaks the invariant that decoding is a function of the WAL stream alone. And -the practical consequence is a retention hazard with no good answer: compaction -and `pgcolumnar.vacuum` may rewrite or reclaim a row group before a lagging slot -has read it, at which point the change is unreconstructable and the stream is -silently short. Heap has no equivalent problem because the tuple data is in the -WAL record. Solving it would mean teaching every reclamation path about -replication slots -- a large amount of new coupling for a feature that has a -cheaper route. +**Rejected**, but not for the reason this plan first gave, and the correction is +worth recording so B is not left looking impossible. + +The first draft rejected it on a retention hazard: compaction and +`pgcolumnar.vacuum` may reclaim a row group before a lagging slot has read it, +leaving the stream silently short, and fixing that looked like teaching every +reclamation path to retain. + +ChronicallyJD's answer on review is better than the objection: **do not teach +reclamation to retain, teach it to invalidate.** Core already makes exactly this +trade for WAL a slot still needs -- the slot is not preserved, it is marked +invalidated and the consumer finds out loudly. Reclamation would need to know +only the oldest slot LSN, and a reclaim that crosses it invalidates the slot +rather than truncating the stream. That is one comparison at reclaim time and no +change to retention policy anywhere, and it converts the failure from a silently +short stream into a visibly broken slot. Those are not the same class of problem. + +So the retention hazard is answerable. B is still not the recommendation, for the +two reasons that remain: it breaks the invariant that decoding is a function of +the WAL stream alone, and it couples the correctness of the stream to +reclamation timing. The recommendation needs no decode-time access to storage at +all, which is a stronger position than needing it and having an answer for when +it fails. ### C. Decode the full-page images @@ -119,14 +145,19 @@ or a recovery running a stock binary without this extension loaded replays the WAL correctly and ignores the message. Nothing about the cluster's recoverability depends on our build. -Available on every supported major. The signature gained a `flush` argument -after 15, so it needs one entry in `columnar_compat.h` beside the others: +Available on every supported major. The signature gained a `flush` argument at +**PostgreSQL 17**, verified in the headers of all five rather than inferred: ```c -/* 15 */ LogLogicalMessage(prefix, message, size, transactional) -/* 16+ */ LogLogicalMessage(prefix, message, size, transactional, flush) +/* 15, 16 */ LogLogicalMessage(prefix, message, size, transactional) +/* 17, 18, 19 */ LogLogicalMessage(prefix, message, size, transactional, flush) ``` +so the guard is `PG_VERSION_NUM >= 170000`, and it gets **its own macro** in +`columnar_compat.h` rather than being folded in with another. Two signatures that +move at different versions under one guard is how `main` stopped compiling on 17 +once already. + ### E. Triggers, in user space — available today, and the interim answer Since #182 made `AFTER ... FOR EACH ROW` triggers work on columnar tables, a user @@ -195,11 +226,23 @@ must be benchmarked before it is recommended for anything. necessary; what N, and does it interact badly with `logical_decoding_work_mem`? 2. **`TRUNCATE` and DDL.** `TRUNCATE` on a columnar table, and `ALTER TABLE` shapes that rewrite, need a decided answer rather than an accident. -3. **Rewrites that are not user changes.** `pgcolumnar.vacuum`, `compact` and - `recluster` move rows between groups without changing table contents. These - must emit nothing, and the test for that matters more than the test for the - happy path -- a compaction that replays as a stream of inserts would silently - duplicate a table on the consumer. +3. **Rewrites that are not user changes.** Stated once as an invariant rather + than as three cases, because future paths need a rule to check themselves + against: **anything that moves rows without changing table contents emits + nothing.** `pgcolumnar.vacuum`, `compact` and `recluster` are today's + instances. It is the same rule `columnar_index.c` already follows on the + rewrite path, where it enforces no constraint because the rows already + satisfied them. + + This matters more than the happy path: a compaction replayed as a stream of + inserts silently duplicates a table on the consumer. + + **The obvious test does not discriminate.** Asserting that a rewrite raises no + error passes on a build that emits a full stream of inserts. The test has to + assert that the consumer's change count is **unchanged across the rewrite**, + and it has to be run against a build where the rewrite does emit, or it proves + nothing. This project has been bitten five times by checks satisfied by both + outcomes; this is the place it would happen again. 4. **Identity for deletes.** A delete vector records row numbers. A consumer needs the key columns, so the identity columns have to be read at flush time. What is the analogue of `REPLICA IDENTITY` here, and what happens with no primary @@ -227,12 +270,29 @@ Discriminating cases, which matter more than the happy path: ## Phasing -1. Per-table option, plumbing, and inserts only, with the abort and - no-op-on-rewrite tests. This is where the design is proven or not. -2. Deletes and updates, with identity. +1. Plumbing and inserts, with the abort and no-op-on-rewrite tests. This is where + the design is proven or not, and where the WAL cost gets measured. **The + per-table option is not enabled by this phase.** +2. Deletes and updates, with identity. The option becomes reachable here. 3. Import paths, `TRUNCATE`, and DDL. 4. A reference consumer, and the documentation that says what this is not. Phase 1 is the one worth doing before deciding whether the rest is worth it: if the WAL cost measured there is unacceptable, that is the answer, and it is -cheaper to learn it at phase 1 than at phase 4. +cheaper to learn at phase 1 than at phase 4. + +**The option must lag the capability, not lead it**, and this is a correction to +an earlier draft that had phase 1 shipping a usable option over inserts only. +ChronicallyJD's objection is right and it is the load-bearing one in this plan: a +consumer that enables change capture and receives inserts but not updates or +deletes does not get a partial answer, it gets a wrong one that looks complete, +and it builds a table that diverges from the source silently and permanently. +Today's behaviour is worse in every respect except the one that counts -- it +fails honestly, and the user eventually asks why they see nothing. + +So a feature that is wrong when half-built must not be reachable when half-built. +Either the option refuses to enable until update and delete are covered, so +phase 1 is reachable only from a test, or it is named for exactly what it does +(`emit_inserts`, never `logical_decoding`) so nobody can turn it on believing +they have change capture. Either is fine. Shipping phase 1 as `logical_decoding` +is not.