Feature: Apache Iceberg lake tables as an extension (datalake_fdw skeleton)

[MisterRaindrop]

Important

This PR has been repurposed, in place, from kernel syntax to an extension.

Review here converged on doing lake tables without kernel changes, and
@tuhaihe asked
for the new commits to land in this PR so the review context stays in one
place. So rather than opening a third PR, the branch now carries:

1. the original 26 kernel commits, untouched, so the 64 inline review
   comments stay anchored to the code they were written about;
2. one commit that reverts all of them (the tree afterwards is identical to
   the merge base, verified by comparing tree object ids);
3. the extension itself, moved over from Feature: Apache Iceberg lake tables as an extension (datalake_fdw skeleton) #1881.

Nothing was force-pushed. The kernel work is therefore still recoverable two
ways: revert the single revert commit, or check out
archive/iceberg-ddl-kernel-pr1842 on the author's fork.

The diff is now 100% different from what was reviewed and approved earlier
in this thread, so the existing approval no longer applies to it. Please
re-review from scratch. The original description is kept, struck through, at
the bottom.

What does this PR do?

Adds contrib/datalake_fdw: Apache Iceberg lake tables as a table access
method, with no kernel changes. A lake table is an ordinary
CREATE TABLE ... USING iceberg WITH (catalog = '...', volume = '...'), where
catalog and volume name foreign servers created through two foreign-data
wrappers this extension registers.

This is deliberately a skeleton, published to get the shape reviewed before
the implementation follows. CREATE TABLE and DROP TABLE work end to end
against a stub metadata engine -- no catalog service, object store, Arrow or JVM
involved -- and everything that would touch data reports a clean
iceberg: <operation> is not supported yet. The interfaces the later work plugs
into ship whole so they can be argued about now: the metadata-engine vtable with
a capability bitmap the registry validates and dispatches through, the
reader/writer format interfaces, and the storage facade.

The question this change is really asking is whether the seam is in the
right place -- see Additional Context.

CREATE SERVER cat FOREIGN DATA WRAPPER iceberg_catalog_fdw
  OPTIONS (type 'rest', uri 'https://catalog.example:443', warehouse 's3://bucket/wh');
CREATE USER MAPPING FOR CURRENT_USER SERVER cat
  OPTIONS (client_id '...', client_secret '...');   -- credentials only ever live here

CREATE SERVER vol FOREIGN DATA WRAPPER iceberg_volume_fdw
  OPTIONS (base_path 's3://bucket/prefix', region 'us-east-1');
CREATE USER MAPPING FOR CURRENT_USER SERVER vol
  OPTIONS (access_key_id '...', secret_access_key '...');

CREATE TABLE t (a int, b text) USING iceberg
  WITH (catalog = 'cat', volume = 'vol', purge_on_drop = false);

Type of Change

• Bug fix (non-breaking change)
• New feature (non-breaking change)
• Breaking change (fix or feature with breaking changes)
• Documentation update

Test Plan

• Unit tests added/updated
• Integration tests added/updated
• Passed make installcheck
• Passed make -C src/test installcheck-cbdb-parallel

Impact

Performance: none. Nothing outside the module changes behaviour, and no
statement gains work: the utility and object-access hooks match on the access
method and return immediately for every other relation.

User-facing changes: a new extension, plus two lines of registration
(contrib/Makefile, and a CI matrix entry so the suite actually runs). The
extension installs process-wide hooks, so it must be in
shared_preload_libraries; _PG_init refuses to load any other way rather than
half-initialising. One consequence worth documenting when this ships: because
gpconfig -c shared_preload_libraries replaces the value rather than
appending to it, overwriting that GUC on a cluster where the extension is in use
will break the databases that have it installed.

Dependencies: none. The module builds with no new external dependency -- the
C++ translation units are stubs today, and the object-store SDK and Arrow arrive
with the changes that need them.

Checklist

• Followed contribution guide
• Added/updated documentation
• Reviewed code for security implications
• This PR contains AI-assisted code generation
• Requested review from cloudberry committers

Additional Context

This is the upstream half of an implementation that already exists. A
datalake_fdw with the same shape -- the same two hooks, the same table access
method, largely the same file names -- runs in production elsewhere, and the
intent is that this becomes the single source for it rather than a second
lookalike. Two consequences a reviewer should see explicitly:

The mapping accessor is deliberately shaped to that implementation. A lake
table's mapping to its catalog and volume servers is read only through
pg_iceberg_get_table_info(), whose signature and result types
(IcebergTableInfo, IcebergTableOptions) match it field for field. That
implementation keeps the same mapping in a system catalog of its own, which an
extension cannot add, so here the function reads reloptions instead -- but the
difference stops inside the function body, and the layers above it can be the
same code on both sides.

The SQL vocabulary follows the Iceberg specification, not the released
implementation. Catalog servers take uri and warehouse, and the REST catalog
type is rest. An earlier revision of this change used the other tree's url /
warehouse_location_prefix / polaris on the reasoning that one vocabulary
across both trees was worth more than either name; review pointed out that the
REST catalog specification exists precisely so that these keys are the same
across implementations, and that argument wins. polaris is kept as an accepted
alias for rest so existing SQL keeps working. Convergence with the other tree
is unaffected in substance: the macro names, struct names, field names and
parse-function split are unchanged, so only the string a user types differs.
hadoop and s3 are in the vocabulary but refused until something implements
them.

Semantics that are decided (and why they are in the SQL surface)

• DROP TABLE does not delete lake data unless the table was created with
  purge_on_drop = true. It is a table option rather than a GUC or a separate
  function so that it travels with the table, cannot turn one DROP into a
  destructive one through session state, and is carried out by a dump.
• Table identity is the (catalog, namespace, table) name triple, matching
  the existing implementation.
• Concurrency is Iceberg's: compare-and-swap on the metadata pointer, with
  the external catalog owning the metadata. Nothing here takes distributed locks.
• A schema that has diverged from the lake is an error, not something to
  adapt to silently. The external table is the truth.

What was and was not verified

Stated plainly, because a skeleton is worth less if you have to guess:

Verified on a three-segment cluster: make installcheck 3/3, and
order-independent (each file alone and in reverse order, on a fresh database);
zero build warnings; exactly 8 exported symbols; a schema-level pg_dump/restore
round trip; Apache RAT clean. Separately, an 84-statement smoke pass over
ordinary objects -- heap/AO/partitioned/temp/unlogged tables, CTAS, matviews,
inherited and typed tables, views, sequences, indexes, functions, another
wrapper's servers and user mappings, every ALTER form, COPY, TRUNCATE, VACUUM
FULL, CLUSTER, REINDEX -- with the library preloaded, both with and without the
extension installed, plus two concurrency probes. Zero errors: the hooks are
inert for everything that is not a lake table.

Verified by CI (this was listed as unverified when the change was first
published, and has since run): the ic-datalake-fdw entry added here passes on
Rocky 8, 9 and 10, and ic-cbdb-parallel passes on every platform.

Still not verified: the error-detail channel has no regression coverage, for
the reason given below.

Why this branch also touches contrib/interconnect

The second commit changes two lines in another module's test, which would
otherwise look like unrelated scope. It is here because that test made this PR's
CI red and the failure is worth fixing rather than retrying:
contrib/interconnect/sql/interconnect.sql restarts the cluster with
gpstop -raiq. An immediate shutdown skips the shutdown checkpoint, so the next
startup runs crash recovery; while the postmaster is in PM_RECOVERY it rejects
the connection gpstart makes to read the segment configuration ("the database
system is not accepting connections"), gpstop -r exits CRITICAL, psql gives up
at the following \c, and every statement in the file is skipped -- leaving the
cluster with only the coordinator running, so the suites that run after it in the
same job fail too. A fast shutdown checkpoints on the way out, so no recovery
runs and the state that produces that error is never entered. Measured before
and after with recovery deliberately slowed: -raiq failed 2/2, -rafq passed
3/3. Fast is also what the rest of the tree already uses; this file was the only
-raiq. Happy to split it into its own PR if that is preferred.

Known limits, stated rather than left to be found

• The mapping stores server names, so ALTER SERVER ... RENAME on a referenced
  server is refused. Storing OIDs instead would allow it, at the cost of owning
  dump/restore translation; the accessor above is the only thing that would
  change.
• A full pg_dump of a database containing a lake table fails, because dumping
  table contents means scanning. A schema-level dump round-trips. The data
  belongs to the lake, so it arguably should not be in the dump at all -- but
  pg_dump decides from relkind, which an extension cannot influence, so that
  needs a separate conversation.
• Every ALTER TABLE form except OWNER TO is refused while the access method
  is unfinished. purge_on_drop is therefore fixed at CREATE time.
• The error-detail channel has no regression coverage, because no statement can
  make the stub engine fail.

Naming

datalake_fdw is inherited from the existing implementation and is a poor fit:
what this installs is a table access method plus two wrappers, not one FDW. A
better name is welcome while it is still cheap to change.

Discussion: #1683   ·   Superseded PR: #1881

---

Original description — kernel-side DDL scaffolding (superseded; kept for the review history)

Summary

Kernel-side DDL scaffolding for Iceberg lake tables, per the design proposal in #1683 (this PR is the "core syntax" milestone of the roadmap posted there). It adds the system catalogs, parse nodes, grammar, commands, and client-tool awareness that a datalake provider extension builds on. No table AM implementation or provider is included — those are later milestones (agent / FDW phases in the roadmap).

CREATE FOREIGN CATALOG hive_cat SERVER hive_cat_srv OPTIONS (type 'hive', uri 'thrift://...');
CREATE FOREIGN VOLUME s3_vol SERVER s3_srv OPTIONS (path 's3://bucket/prefix');
CREATE ICEBERG TABLE t (a int, b text) CATALOG hive_cat VOLUME s3_vol OPTIONS (fileformat 'parquet');
DROP CATALOG hive_cat;  DROP VOLUME s3_vol;

What's included (structured for commit-by-commit review)

1. catalog: pg_foreign_catalog, pg_foreign_volume (metadata service / storage location handles, hanging off pg_foreign_server), pg_lake_table (per-relation lake metadata keyed by relid). All three have TOAST tables; name indexes are single-column unique (see "Design decisions").
2. nodes: CreateLakeTableStmt (embeds CreateStmt), CreateForeignCatalogStmt, CreateForeignVolumeStmt + hand-maintained node-support functions.
3. parser: CREATE ICEBERG TABLE ... CATALOG ... VOLUME ... OPTIONS (...), CREATE FOREIGN CATALOG|VOLUME ... SERVER ..., DROP CATALOG|VOLUME via drop_type_name.
4. commands (foreign catalog/volume): create/drop with IF [NOT] EXISTS, dependencies, ownership, object addressing / COMMENT / event-trigger integration, syscaches, QD→QE dispatch with synchronized OIDs.
5. commands (CREATE ICEBERG TABLE): resolves catalog/volume (explicit clause or iceberg_default_catalog/iceberg_default_volume GUCs), validates the statement uses the iceberg AM, creates the relation + TOAST + pg_lake_table row + dependencies; guards ALTER TABLE SET ACCESS METHOD / SET DISTRIBUTED BY both directions; lake tables are forced DISTRIBUTED RANDOMLY.
6. bin: pg_dump skips iceberg tables (by AM name, with a warning — same pattern as PAX); psql tab completion.
7. tests: lake_table regression in greenplum_schedule (DDL lifecycle, duplicate/USING rejection, TOAST wide rows, ownership, dependency/CASCADE, QD/QE dispatch checks) + catalog-expected refresh across regress, singlenode_regress, and pax copies.

Design decisions to review

• Provider decoupling: the kernel resolves the AM strictly by name (get_table_am_oid("iceberg")); no hardcoded AM OID and no extension-name checks. Everything degrades gracefully (with a hint) when no provider is installed.
• Provider integration contract: CreateLakeTable() runs after DefineRelation() and ends with CommandCounterIncrement() + InvokeObjectPostCreateHook(LakeTableRelationId, ...) — a provider performs remote Iceberg creation from that object-access hook (on QD and QEs), where catalog/volume/options metadata is already visible. relation_set_new_filelocator remains local-storage-only.
• Catalog/volume names are database-global (single-column unique index), like pg_foreign_server — every reference syntax (DROP CATALOG x, the CATALOG x clause, the GUCs) identifies them by bare name, so the uniqueness scope matches what the syntax can express. (The alternative — per-server names like user mappings — would require server-qualified reference syntax everywhere.)
• CREATE ICEBERG TABLE rejects a USING clause naming any other AM; without it the statement implies USING iceberg.

Known limitations / open questions (deliberately out of scope here)

• pg_dump / pg_upgrade: FOREIGN CATALOG/VOLUME objects and pg_lake_table metadata are not dumped yet; iceberg tables are skipped with a warning (PAX precedent). Restore/upgrade support is planned as a follow-up.
• Permission model: creating a catalog/volume requires USAGE on its server. Referencing one from CREATE ICEBERG TABLE currently checks existence only — whether that should require USAGE on the underlying server, or dedicated ACLs (GRANT USAGE ON CATALOG), is an open question we'd like reviewer input on.
• The GUC defaults are synced to QEs and behave like default_tablespace w.r.t. objects dropped after SET.

Discussion: #1683

[MisterRaindrop]

1. Does this AM refer to the minimal Iceberg AM implementation that does not support scans or inserts? Is that correct?
2. One more question: CREATE ICEBERG TABLE USING
   Is this syntax designed to support extensions via other frameworks? From my understanding, there should be no need to specify USING am if we're already using CREATE ICEBERG TABLE, right?

[MisterRaindrop]

At this stage, I believe the regression testing coverage is sufficient, test stub is enough. Introducing the AM layer at this point would only add unnecessary complexity with no practical benefit, as we still cannot run any real data workloads with it.

[MisterRaindrop]

@andr-sokolov It is ready for review

[yjhjstz]

We can align syntax with crunchydata.com

Building a Postgres Data Warehouse using Iceberg.pdf

> Iceberg as a Postgres table format Capture queries, writes, & schema changes to provide a transactional table experience for Iceberg in S3.

create table chats (
message_id bigserial not null,
thread_id bigint not null,
…
) using iceberg;

@leborchuk @andr-sokolov

[andr-sokolov]

@yjhjstz How does crunchydata specify catalog? I didn't find this syntax in their presentation

[leborchuk]

Building a Postgres Data Warehouse using Iceberg.pdf

Yes, looks good. But how we setup catalog and lakehouse format? There should exist some definition ... It's good for pre-configured database instances, but your own database needs a more advanced syntax.

[yjhjstz]

Em, currently how about change CREATE LAKE TABLE lake_test_t0 -> CREATE TABLE lake_test_t0, no need LAKE ?

[MisterRaindrop]

@andr-sokolov @leborchuk @yjhjstz I looked into how Crunchy Data Warehouse (now open-sourced by Snowflake as pg_lake) handles the catalog — hope this helps answer the question above.

TL;DR: Crunchy doesn't specify a catalog at CREATE time, because Postgres itself acts as the catalog.

• Managed Iceberg tables are created with plain CREATE TABLE ... USING iceberg. The Iceberg metadata lives inside Postgres and is exposed through an iceberg_tables view compatible with the Iceberg JDBC/SQL catalog protocol, so Spark / pyiceberg / iceberg-rust connect to Postgres as the catalog (JDBC with Postgres credentials; catalog_name = the database name). (Crunchy docs)
• I couldn't find any syntax for attaching an external catalog (Glue / HMS / REST) — the product doesn't seem to have that concept, which would explain why it doesn't appear in the presentation.
• Storage location is a GUC default plus an optional per-table reloption: SET crunchy_iceberg.default_location_prefix TO 's3://...', or CREATE TABLE ... USING iceberg WITH (location = 's3://...'). With neither set, the cluster's bundled managed storage is used — so in the demo nothing needs to be specified at all. @leborchuk's reading ("good for pre-configured database instances") matches what I found.

One more data point that may be relevant: pg_lake is currently adding external Iceberg REST catalog support as a client (pg_lake#94), and that roadmap mentions handling multiple endpoints/credentials "likely using a Postgres SERVER-like concept" — quite close to what CREATE FOREIGN CATALOG ... SERVER does in this PR. So the two designs may be converging from different starting points.

For reference, the two syntax shapes side by side:

• Plain CREATE TABLE ... USING iceberg: the catalog/volume bindings move into reloptions, e.g. WITH (catalog 'c', volume 'v'), with GUC defaults; no new grammar, and plain DROP TABLE applies (the Spark / pg_lake convention).
• CREATE LAKE TABLE ... USING <format> CATALOG c VOLUME v: catalog/volume are first-class clauses with dedicated CREATE/DROP LAKE TABLE commands (the current PR).

[yjhjstz]

if we use CREATE TABLE ... USING iceberg WITH (catalog 'c', volume 'v'), we need not change kernel code, iceberg will be totally an extension .

[MisterRaindrop]

@yjhjstz Agreed — this looks technically feasible, so I sketched what the fully-extension shape could look like.

Since Cloudberry's table AM API already supports per-AM reloptions validation (TableAmRoutine->amoptions, unlike vanilla PG16), WITH (catalog ..., volume ...) can be validated by the iceberg AM itself with no kernel changes. The catalog/volume objects could reuse the existing FDW infrastructure (SERVER + USER MAPPING), which gives ACLs, pg_dump support, QD/QE dispatch and dependency tracking for free:

CREATE EXTENSION pg_iceberg;   -- provides the table AM, two FDWs, and hooks

CREATE SERVER hive_cat FOREIGN DATA WRAPPER iceberg_catalog
    OPTIONS (type 'hive', uri 'thrift://metastore:9083');
CREATE USER MAPPING FOR alice SERVER hive_cat
    OPTIONS (user '...', password '...');

CREATE SERVER s3_vol FOREIGN DATA WRAPPER iceberg_volume
    OPTIONS (base_path 's3://bucket/prefix');

CREATE TABLE t (a int, b text) USING iceberg
    WITH (catalog 'hive_cat', volume 's3_vol');

DROP TABLE t;   -- plain DROP TABLE applies

A few things the extension would need to own, instead of kernel guards:

• registering table metadata and pg_depend entries (table → server) via object_access_hook, so DROP SERVER stays protected and CASCADE works;
• intercepting paths that don't make sense for iceberg (CTAS, partition children, ALTER TABLE ... SET ACCESS METHOD) via ProcessUtility_hook, and forcing random distribution;
• the library would need to be in shared_preload_libraries so the hooks are always installed.

This is essentially the pg_lake architecture, and USER MAPPING would also give a cleaner credentials story than embedding keys in options. If we go this way, the kernel scaffolding in this PR (the grammar, the three catalogs, the guards) would no longer be needed — the work would move into the datalake extension instead. WDYT?
@leborchuk @andr-sokolov

[andr-sokolov]

@MisterRaindrop If possible, let's not change the kernel to reduce conflicts when updating to new versions of PostgreSQL. I think it's better to use an extension.

[MisterRaindrop]

@andr-sokolov @yjhjstz
I open this draft PR about lake table ext (create table xxx using iceberg xxx), please give some suggest and review it.
#1881

[tuhaihe]

@andr-sokolov @yjhjstz I open this draft PR about lake table ext (create table xxx using iceberg xxx), please give some suggest and review it. #1881

Hi, is there any way to introduce your new commits to this place, where we have a context for review? FYI. @MisterRaindrop

[MisterRaindrop]

@tuhaihe as you asked, the extension work now lives in this PR instead of a separate one, so the review context stays in one place. @andr-sokolov @yjhjstz please note what changed below before looking at the diff again.

How it was moved, and why this way. The branch was not force-pushed. It now has, in order:

1. the original 26 kernel commits, untouched -- so your 64 inline comments stay anchored to the code they were written about;
2. Revert the kernel-side lake-table DDL scaffolding -- one commit that undoes all of them. The tree after it is identical to the merge base, verified by comparing tree object ids rather than by reading a diff;
3. the two extension commits, moved over from Feature: Apache Iceberg lake tables as an extension (datalake_fdw skeleton) #1881. The resulting diff is identical to what Feature: Apache Iceberg lake tables as an extension (datalake_fdw skeleton) #1881 showed -- I compared the two patches line by line; the only differences are blob hashes and one hunk offset, because the two branches had different merge bases.

Rewriting the branch would have been shorter, but it would have orphaned the review history, and it would have made going back expensive. This way the kernel scaffolding is one git revert away if the extension approach turns out to be wrong. The pre-revert tip is also kept as archive/iceberg-ddl-kernel-pr1842 on my fork.

Please re-review from scratch. The diff is now 100% different from what was reviewed and approved earlier in this thread, so the earlier approval does not apply to what is here now. I would rather say that explicitly than have it merged on the strength of it.

One thing that will look out of place: the second commit changes two lines in contrib/interconnect/sql/interconnect.sql. That test restarts the cluster with gpstop -raiq; an immediate shutdown skips the shutdown checkpoint, so the next startup runs crash recovery, and while the postmaster is in PM_RECOVERY it rejects the connection gpstart makes to read the segment configuration. gpstop -r then exits CRITICAL, psql gives up at the \c that follows, the whole file is skipped, and the cluster is left with only the coordinator running -- which is why pg_trgm and indexscan failed after it in the same job. A fast shutdown checkpoints on the way out, so that state is never entered. Measured with recovery deliberately slowed: -raiq failed 2/2, -rafq passed 3/3. Fast is what the rest of the tree already uses; this file was the only -raiq. Happy to split it into its own PR if you would rather keep this one to the extension.

The original description is kept at the bottom of the PR body, struck through. #1881 is superseded by this PR.

[tuhaihe]

Is datalake_fdw enabled by default? We can disable it by default and enable it using the --enable-datalake_fdw configure option, the way like PAX.

[MisterRaindrop]

Good call -- done in 6f40dd0, following the PAX mechanism: PGAC_ARG_BOOL defaulting to no, substituted into Makefile.global, and a conditional in contrib/Makefile that moves the directory to ALWAYS_SUBDIRS when disabled so the clean targets still reach it.

Three things I decided while doing it, all cheap to reverse if you disagree:

• The option is spelled --enable-datalake-fdw, with dashes. Every option in this tree uses dashes (--enable-tap-tests, --enable-ic-udp2, --enable-catalog-ext), and autoconf maps the dashes to enable_datalake_fdw for the Makefile variable, so the directory name still matches. Happy to use the underscore form if you would rather it read exactly like the directory.
• No AC_DEFINE. PAX and ic-udp2 define a macro because C code tests it; nothing here does, and adding an unread macro would mean touching src/include/pg_config.h.in for no reader. --enable-pxf and --enable-orafce already use this shorter four-argument form.
• --enable-datalake-fdw is also added to devops/build/automation/cloudberry/scripts/configure-cloudberry.sh, next to --enable-pax. Without it the ic-datalake-fdw CI job would install nothing and silently test nothing. I deliberately did not add it to coverity.yml and sonarqube.yml, which also pass --enable-pax -- say the word if you want the module in the weekly scans too.

One thing you should know about configure. I regenerated it by hand rather than running autoconf over the whole file, because the committed configure and configure.ac are currently out of sync in both directions:

• configure.ac has a restructured PAX liburing check (the case $host_os in linux*) block with the macOS fallback comment) that was never regenerated into configure;
• configure has a Darwin python shared-library lookup (trying .dylib in addition to DLSUFFIX) that does not exist in configure.ac.

A full autoconf run therefore produced twelve hunks, only four of which were mine; taking all of them would have swept those two unrelated changes into this PR and, in the python case, reverted working behaviour. So I applied only the four hunks for this option and left the drift alone. It is worth someone reconciling those two separately -- I did not want to do it inside this PR.

Verified by configuring a real tree both ways, not by inspection:

	configure says	Makefile.global	SUBDIRS	ALWAYS_SUBDIRS
--enable-datalake-fdw	... yes	enable_datalake_fdw = yes	has datalake_fdw	--
default	... no	enable_datalake_fdw = no	--	has datalake_fdw