Antalya 26.6: Implement TRUNCATE TABLE for Iceberg Engine (REST …

[zvonand]

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Frontport for Antalya 26.3

• 26.1 Antalya port - Implement TRUNCATE TABLE for Iceberg Engine (REST catalog support) Feature: Support TRUNCATE TABLE for Iceberg engine #1529, It's a frontport from 26.1, contains:
  • feat(iceberg): Implement TRUNCATE TABLE for Iceberg Engine (REST catalog support) — Core implementation: metadata-only truncation generating a new overwrite snapshot with empty manifest list, committed atomically via REST catalog
  • fix(iceberg): pass new_snapshot to updateMetadata in IcebergStorageSink — Fixed silent breakage of all INSERTs on REST catalog tables (wrong JSON object passed to catalog->updateMetadata)
  • fix(iceberg): restore return false in RestCatalog::updateMetadata — Preserve retry contract; add LOG_WARNING for diagnostics
  • fix(iceberg): revert Mutations.cpp updateMetadata to pass new_snapshot — Same fix as IcebergStorageSink, applied to ALTER TABLE DELETE/UPDATE path
  • refactor(iceberg): add comment explaining Avro zigzag encoding — Reviewer-requested documentation for manual Avro OCF serialization
  • refactor(iceberg): address code review feedback on TRUNCATE implementation — Named zero arguments, helper functions, restart integration test (feature (iceberg): Implement TRUNCATE TABLE for Iceberg Engine (REST … #1655 by @il9ue).

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All with Aarch64
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

Cherry-picked from #1655.

---

Adaptation notes for antalya-26.3

The following API changes between antalya-26.1 and antalya-26.3 required adaptation:

• RestCatalog constructor signature updated to match antalya-26.3 calling convention in DataLakeConfiguration.h
• DatabaseDataLakeSetting member names updated
• Cargo.lock restored to antalya-26.3 version (our changes do not touch any Rust code)

All feature logic is identical to the merged antalya-26.1 implementation.

[github-actions]

Workflow [PR], commit [dfeeb88]

[zvonand]

@blau-ai

[blau-ai]

CI triage

Verdict: 2 failing checks — both PR-caused, same root cause. Nothing flaky/infra/pre-existing.

Both red checks (Build (arm_tidy) and Fast test) fail at compile time on the same code. Every other job shows skipping because it is gated behind the build — those are not independent failures. This is a backport of #1655 (written against antalya-26.3), and the Iceberg-writes API on antalya-26.6 has diverged, so the ported code no longer compiles here.

There are two distinct compile problems:

1. Duplicate Iceberg field aliases in Constant.h — blocks the whole build (hit first, e.g. by Fast test)

Constant.h:145:1: error: redefinition of 'f_deleted_records'
Constant.h:126:1: note: previous definition is here
Constant.h:144:1: error: redefinition of 'f_deleted_data_files'
...
2 errors generated. → ninja: build stopped: subcommand failed.

antalya-26.6 already defines both aliases (origin/antalya-26.6:Constant.h:142-143, now at :144-145). The PR re-adds them at :126/:128, so they collide. Fix is to drop the two lines the PR added:

--- a/src/Storages/ObjectStorage/DataLakes/Iceberg/Constant.h
+++ b/src/Storages/ObjectStorage/DataLakes/Iceberg/Constant.h
@@ -123,9 +123,7 @@ DEFINE_ICEBERG_FIELD_ALIAS(partition_specs, partition-specs);
 DEFINE_ICEBERG_FIELD_ALIAS(spec_id, spec-id);
 DEFINE_ICEBERG_FIELD_ALIAS(added_records, added-records);
-DEFINE_ICEBERG_FIELD_ALIAS(deleted_records, deleted-records);
 DEFINE_ICEBERG_FIELD_ALIAS(added_data_files, added-data-files);
-DEFINE_ICEBERG_FIELD_ALIAS(deleted_data_files, deleted-data-files);
 DEFINE_ICEBERG_FIELD_ALIAS(added_delete_files, added-delete-files);

2. IcebergMetadata::truncate() uses the 26.3 FileNamesGenerator / MetadataGenerator API (arm_tidy, 5 errors)

IcebergMetadata.cpp:653:30: error: no matching constructor for initialization of 'FileNamesGenerator'
    note: candidate constructor not viable: requires 4 arguments, but 5 were provided
IcebergMetadata.cpp:659:30: error: no matching constructor for initialization of 'FileNamesGenerator'
IcebergMetadata.cpp:667:70: error: no member named 'generateMetadataName' in 'DB::FileNamesGenerator'
235321 warnings and 5 errors generated.

On antalya-26.6 the writer API changed:

• Constructor is 4-arg, FileNamesGenerator(table_location, bool use_uuid_in_metadata, compression, format) (FileNamesGenerator.h:32). There is no config_path / no separate transactional-path argument — truncate() passes 5 args. The data-path override is now set separately via setDataLocation().
• generateMetadataName() no longer exists — use generateMetadataPathWithInfo(), which returns a struct with a .path (IcebergPathFromMetadata) (FileNamesGenerator.h:47).
• generateNextMetadata(...) returns NextMetadataResult{ snapshot, manifest_list_path } — only 2 fields (MetadataGenerator.h:22-29), but truncate() unpacks 3 ([new_snapshot, manifest_list_name, storage_manifest_list_name]). Storage paths are now obtained by running the metadata path through IcebergPathResolver::resolve(...).

Good news: the is_truncate=true overload of generateNextMetadata already exists on this branch (MetadataGenerator.h:43), so no MetadataGenerator changes are needed — only the caller in truncate().

Concrete fix: rewrite the FileNamesGenerator setup and path handling in IcebergMetadata::truncate() to follow the pattern this branch's writer already uses:

• construction + setDataLocation() — see IcebergWrites.cpp:738-748;
• generateMetadataPathWithInfo() → generateNextMetadata() → resolver.resolve(manifest_list_path) — see IcebergWrites.cpp:952-970.

Note: clang-tidy stops reporting after 5 errors, so once the constructor/generateMetadataName calls are fixed, expect to reconcile a couple of follow-on mismatches in the same function (the 3-vs-2 structured binding above, and the generateManifestList / storage-path arguments). The rest of the diff (Context::createCopy, the new IDataLakeMetadata::truncate virtual, IcebergWrites.cpp zigzag helper) compiled cleanly.

---

I can prepare these fixes — the trivial Constant.h dedup is safe, and I can adapt truncate() to the 26.6 API as above (I can't build locally, so CI on the resulting PR would be the validation). Do you want me to open a blau/* PR with the fix, or commit directly to feature/antalya-26.6/pr-1655? I'll hold until you confirm the delivery.

_{Evidence: run 30468779903 — jobs Build (arm_tidy) and Fast test, head 7a636452.}

[zvonand]

@blau-ai commit the fix directly to this branch

[blau-ai]

Done — committed the fix directly to feature/antalya-26.6/pr-1655 as requested.

Commit: 984609c — Fix antalya-26.6 build: adapt Iceberg TRUNCATE to 26.6 writer API

What changed (both compile errors from the triage above)

1. Constant.h — macro redefinition (blocked the whole build)
Dropped the two field aliases the backport re-added (deleted_records, deleted_data_files); antalya-26.6 already defines both further down the file.

2. IcebergMetadata::truncate() — 26.3 writer API no longer exists (arm_tidy)
Rewrote the path/metadata handling to follow this branch's writer (IcebergStorageSink, IcebergWrites.cpp) so REST (transactional) and non-transactional catalogs go through the same IcebergPathResolver:

• 4-arg FileNamesGenerator(resolver.getTableLocation(), is_transactional, compression, format) — the old 5-arg config_path form is gone; the data-path override is now a separate setDataLocation() concern (a no-op for a metadata-only truncate, which writes no data files);
• generateMetadataPathWithInfo() replaces the removed generateMetadataName();
• generateNextMetadata(...) now returns {snapshot, manifest_list_path} (2 fields, was 3) — storage paths come from resolver.resolve(...), and the catalog path from resolver.resolveForCatalog(...);
• generateManifestList(resolver, …, {}, new_snapshot, {}, …) — first arg is the resolver (was the generator), and the empty manifest-entry/size args are now typed vectors.

The metadata-only truncate semantics (empty overwrite snapshot superseding all previous snapshots) are unchanged; the is_truncate=true overload of generateNextMetadata already exists on this branch, so no MetadataGenerator change was needed.

Validation

I can't build ClickHouse in this environment, so this is validated by CI on this PR, not locally. The two red checks (Build (arm_tidy), Fast test) both failed at compile time on exactly these two spots; a fresh run should now get past compilation. I'll be watching — ping me if anything downstream (e.g. the follow-on mismatches clang-tidy would have hidden behind its 5-error cap) still trips.

[mkmkme]

The issue is reproducible locally. Will investigate and try to fix

[zvonand]

@blau-ai

[blau-ai]

CI triage for #2125

Verdict: of all the red checks, only one is actually caused by this PR — the DCO sign-off. Everything else is pre-existing on antalya-26.6, environmental/infra, or regression-suite-vs-build version skew. None of the test failures touch the Iceberg TRUNCATE code paths this PR adds.

Summary: 1 PR-caused (DCO), rest not (0 test regressions attributable to this PR).

---

🔴 PR-caused (fix required)

DCO — one commit is missing its Signed-off-by line:

• ✅ merge commits & member commits are skipped by policy
• ❌ 984609c "Fix antalya-26.6 build: adapt Iceberg TRUNCATE to 26.6 writer API (Antalya 26.6: Implement TRUNCATE TABLE for Iceberg Engine (REST … #2125)" → Sign-off not found

Fix (must be done by a human — I can't force-push). Since this is your own branch and only that commit needs it:

git rebase HEAD~6 --signoff
git push --force-with-lease origin feature/antalya-26.6/pr-1655

Or, to avoid rewriting history, add a DCO remediation commit. Going forward, git commit -s adds the trailer automatically.

---

🟢 Not caused by this PR

Stateless tests (amd_debug parallel, amd_debug distributed/s3, arm_binary parallel) — the single real [ FAIL ] in each is the same test:

04337_iceberg_v3_row_lineage_reserved_field_id — result differs, actual output is missing the trailing ICEBERG_SPECIFICATION_VIOLATION line (the reader didn't reject an unmapped, non-reserved field id 2147483447).

This is a reader-side reserved-field-id boundary check. This PR does not touch it — it only modifies the Iceberg write / TRUNCATE path (IcebergWrites.cpp, MetadataGenerator.cpp) plus include/extern additions. Proof it's pre-existing: the test was merged into antalya-26.6 on 2026-08-11 (backport #2198 / upstream ClickHouse#107377) and the identical [ FAIL ] reproduces on unrelated PR #2197 (cluster-discovery, no Iceberg changes) and on the current base-branch tip run. Likely the reader-side fix from ClickHouse#107377 didn't fully land with the test — worth a separate issue against the backport, not this PR.
The other non-green lines in those jobs are infra: 03441_deltalake_* and 04070_url_base_setting are [ BROKEN ] = "INVESTIGATE - S3 sometimes unreachable" / "fragile timeout on debug" (auto-excluded, don't gate).

Regression — Iceberg (1)/(2), release & aarch64 — the failing features are all feature support validation gates for features not present in this build, i.e. the regression suite is newer than the 26.6 build:

• Swarm object_storage_cluster_join_mode is not supported by this build
• Iceberg partition timezone is not supported by this build
• Export merge tree partition is not supported by this build

Plus, on aarch64 only, [ Error ] in icebergS3 table function/glue catalog/* and iceberg cache/glue catalog/* — AWS Glue catalog access (external infra), not TRUNCATE. None of these are the TRUNCATE feature.

Regression — S3Export (part)/(partition), release & aarch64 — Export merge tree part/partition is not supported by this build; the feature is not present in this version. Same suite-vs-build version skew. Unrelated to this PR.

Regression — Swarms, release & aarch64 — Swarm object_storage_cluster is not supported by this build. Same version skew (note the 24 retries all passed).

Stress test (arm_release) — no crash / sanitizer / logical-error signature anywhere in the log; the failure is Code: 210. DB::NetException: Connection refused (localhost:9000) — the server didn't come up in that iteration (Failures: 2/3). Environmental/flaky. Safe to re-run.

Grype Scan (server, non-alpine) — the scan reports "Completed with 0 high/critical vulnerabilities" but the job ran 6h 0m → it timed out. The alpine/keeper variants passed. Infra timeout; this PR changes no dependencies or Dockerfiles. Safe to re-run.

PR — aggregate gate; red only because the children above are red. No independent failure.

---

⚠️ Coverage gap worth noting

The feature's own integration test, test_storage_iceberg_no_spark/test_iceberg_truncate.py (added/edited by this PR), was skipped — the Integration tests (amd_asan_ubsan, targeted) job resolved to skipped, and CI did detect it as a changed_integration_test. So TRUNCATE itself was not actually exercised by this run. Before merge I'd recommend making sure that targeted integration job runs (re-trigger integration tests / confirm it isn't being gated out) so the feature gets a real green signal — none of the failing checks above validate the TRUNCATE path.

---

Analysis is CI-evidence-based (I can't build/run ClickHouse here). Bottom line: fix the DCO sign-off; the remaining reds are pre-existing/infra/version-skew and don't block on account of this PR's code.