Skip to content

Antalya 26.6: Add commit info to partition exports table - #2210

Open
zvonand wants to merge 2 commits into
antalya-26.6from
feature/antalya-26.6/pr-1832
Open

Antalya 26.6: Add commit info to partition exports table#2210
zvonand wants to merge 2 commits into
antalya-26.6from
feature/antalya-26.6/pr-1832

Conversation

@zvonand

@zvonand zvonand commented Aug 14, 2026

Copy link
Copy Markdown
Member

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add Iceberg metadata file paths and data file paths to the system.replicated_partition_exports to improve observability and debugging. Partially tackles #1824

Also replaces clickhouse snowflakeid implementation with uuidv4 (#1832 by @arthurpassos).

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Performance tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All with Aarch64
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

Combined port of 1 PR(s) (group pr-1832). Cherry-picked from #1832.

arthurpassos and others added 2 commits August 14, 2026 10:43
…next commit)

---
Original cherry-pick message follows:

Merge pull request #1832 from Altinity/expand-replicated-partition-exports-columns

Add commit info to partition exports table
# Conflicts:
#	docs/en/antalya/partition_export.md
#	src/Storages/ObjectStorage/DataLakes/IDataLakeMetadata.h
#	src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.cpp
#	src/Storages/ObjectStorage/StorageObjectStorage.cpp
Kept antalya-26.6's metadata_snapshot / configuration arguments while adopting the PR's ExportPartitionCommitInfo return values.
@zvonand zvonand added releasy Created/managed by RelEasy antalya-26.6 forwardport This is a frontport of code that existed in previous Antalya versions ai-resolved Port conflict auto-resolved by Claude labels Aug 14, 2026
@zvonand zvonand mentioned this pull request Aug 14, 2026
55 tasks
@github-actions

Copy link
Copy Markdown

Workflow [PR], commit [c1efccc]

@zvonand

zvonand commented Aug 14, 2026

Copy link
Copy Markdown
Member Author

@blau-ai

@blau-ai

blau-ai commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

CI triage for #2210

Verdict — 5 distinct failures: 2 PR-related, 3 not.

Check Classification PR-caused?
Regression iceberg_2 (release + aarch64) Feature regression (this cherry-pick) Yes
Regression s3_export_part (release + aarch64) Feature regression (this cherry-pick) Yes
Regression swarms (release + aarch64) Flaky / infra (node-failure tests) No
Stateless amd_debug, distributed plan…03599_lightweight_delete_vertical_merge Unrelated subsystem, likely flaky No
Stateless amd_debug, sequential00024_random_counters 600 s timeout under debug load No

The top-level PR check is just the aggregate rollup of the above.


PR-related (need attention)

iceberg_2 and s3_export_part regression suites — every failure lands inside the exact feature this PR reworks (/iceberg/export partition/… and /export part/…); this PR is a conflict-resolved cherry-pick of #1832 (c1efccca "Resolve conflicts in cherry-pick of #1832", preceded by a commit with raw conflict markers).

Decisive evidence it's this port, not a pre-existing 26.6 issue:

  • The identical change (original Add commit info to partition exports table #1832) passed the full s3_export_part suite on antalya-26.3 (1 module (1 ok), full 2h+ run). After the cherry-pick onto 26.6 the same suite fails.
  • On base antalya-26.6 these scenarios never ran — the scheduled base run (31713806632) reports "Export merge tree part/partition is not supported by this build; the feature is not present in this version" and executes only ~6 feature-support scenarios. So there is no green baseline on 26.6, but the failures are clearly introduced once this PR builds the reworked path.
  • Failure signatures are behavioral, not infra:
    • s3_export_part: early, server-healthy assertion — /export part/error handling/pending mutations expects exitcode == 237 + PENDING_MUTATIONS_NOT_ALLOWED in output, which is no longer returned; plus /concurrent alter/during minio interruption/* hanging to a 7 m01 s timeout, which then cascades instant failures into concurrent other/*, stress/*, rbac, table functions.
    • iceberg_2: 242 scenario failures, almost all bare AssertionError (output / system.replicated_partition_exports content mismatches), spanning sanity, transactions, manifest integrity, system monitoring, casting, schema compatibility — i.e. the tests that assert on this table's columns and transaction_id.

Suggested next steps for the author:

  1. Re-audit the conflict resolution in the commit path — the resolution mixed 26.6's metadata_snapshot/configuration args with Add commit info to partition exports table #1832's new ExportPartitionCommitInfo return values in IcebergMetadata::commitExportPartitionTransaction / commitImportPartitionTransactionImpl and StorageObjectStorage::commitExportPartitionTransaction. Confirm the reworked retry loop (if (commit_info) return *commit_info; else retry, throw UNFINISHED after MAX_TRANSACTION_RETRIES) preserves Add commit info to partition exports table #1832's original commit/conflict semantics, and that the pending-mutations/patch-parts guard still fires before the commit proceeds so exitcode 237 / PENDING_MUTATIONS_NOT_ALLOWED is returned.
  2. Pull the testflows expected-vs-actual diff to pin the iceberg_2 mismatches: artifact iceberg_2-artifacts-release_zookeeper from run 31787415505 (the --log-failed output doesn't inline the assertion bodies).
  3. Check the Altinity/clickhouse-regression ref used by this branch matches Add commit info to partition exports table #1832's new schema (UUIDv4 transaction_id + destination_file_paths / committed_* columns). If the suite pinned on 26.6 still expects the old snowflake-int transaction_id/old column set, the system monitoring assertions will fail regardless of the C++ being correct.
  4. The minio-interruption 7-minute hang is the highest-value single fix — it's what triggers the downstream cascade. Look for a missing timeout/abort/retry bound in the reworked manifest-commit path when the destination object store is interrupted.

Note: the Integration tests jobs (which include test_export_replicated_mt_partition_to_iceberg / …_to_object_storage, directly modified by this PR) show skipping and did not report — worth running them explicitly, as they'd localize the regression faster than the full regression suites.

Not PR-related

  • Regression swarms — 3 failing scenarios (node failure/check restart swarm node, object storage cluster profile events with overloaded node, task rescheduling with bucket granularity), all QUERY_WAS_CANCELLED (Code 394) during deliberate node kills, with heavy retry churn (515 retries (430 ok, 84 failed)). No mention of partition exports / transaction_id. Classic swarm node-failure flakiness. Safe to re-run.
  • 03599_lightweight_delete_vertical_merge (distributed-plan stateless) — lightweight-delete/vertical-merge subsystem, untouched by this PR (the diff only touches export-partition / Iceberg / object-storage / the exports system table). Not caused here; likely flaky. Safe to re-run.
  • 00024_random_counters (sequential stateless) — [ FAIL ] 600.03 sec = the 600 s test timeout under debug-build sequential load. Infra/timeout, unrelated subsystem. Safe to re-run.

I did not push anything — this was a triage request. If you'd like, I can (a) dig into the iceberg_2 expected-vs-actual artifact to pinpoint the exact column/assertion mismatch, or (b) prepare a fix for the conflict-resolution regression on a blau/* branch as a separate PR. Just say which.

(Analysis from run 31787415505 @ c1efccca; I can't build/run ClickHouse in this container, so correctness of any fix would be validated by re-running CI.)

@zvonand

zvonand commented Aug 14, 2026

Copy link
Copy Markdown
Member Author

RelEasy analyze-fails — UNRELATED

run completed at 2026-08-14T19:04:33Z

  • Head SHA: c1efccca5e (feature/antalya-26.6/pr-1832)
  • Tests considered: 2 across 2 CI shard(s)
  • Outcomes: 0 done · 0 partial · 2 unrelated · 0 unresolved
  • Commits added by AI: 0 (—)
  • Anthropic cost: $1.4126

Per-shard outcomes

⏭️ UNRELATED — Stateless tests (amd_debug, distributed plan, s3 storage, parallel)

1 failed test(s) considered — cost $0.8387
full report

AI narration

I'll start by triaging the failure against this PR's diff.

Triage

PR diff scope (git log antalya-26.6..HEAD → the two PR commits 61fb4babb52 + c1efccca5eb, a cherry-pick of #1832 "Add commit info to partition exports table"):

  • src/Storages/MergeTree/ExportReplicatedMergeTreePartition*, ExportPartitionManifestUpdatingTask.*, ExportPartitionUtils.cpp
  • src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.*, StorageObjectStorage*.{h,cpp}, IDataLakeMetadata.h
  • src/Storages/System/StorageSystemReplicatedPartitionExports.*, src/Common/FailPoint.cpp (+1 failpoint), IStorage.h, docs, integration tests for partition export.

Nothing in the diff touches lightweight deletes, vertical merges, MergeTask, or system-log flushing.

03599_lightweight_delete_vertical_merge — the CI excerpt shows the failure was Code: 159 … Timeout exceeded (180 s) while flushing system log 'SystemLogQueue<PartLogElement>' on SYSTEM FLUSH LOGS part_log;, i.e. a loaded-runner timeout, and praktika's own randomized-settings diagnosis reported 38 reruns, 0 failed ("failure is not reproducible (likely a transient issue)"). Classified NOT-THIS-PR.

Verification

Ran the test locally with this shard's runner flags (--s3-storage --no-stateful --no-sequential, per ci/jobs/functional_tests.py--distributed-plan is a server-install option, not a runner arg):

  • First attempt failed spuriously because the prescribed rm -rf ci/tmp clipped parts out of the running server's own system.part_log store dir (filesystem error: in file_size … skp_idx_event_time_index.cmrk2) — my own environment damage, not the test. After TRUNCATE TABLE system.part_log SYNC:
[1 / 1] 03599_lightweight_delete_vertical_merge:   [ OK ] 0.18 sec.
1 tests passed. 0 tests skipped.

No code was changed; git status --porcelain is empty and no commits were added.

Summary

  • 03599_lightweight_delete_vertical_merge[unrelated] — CI failure was a 180 s SYSTEM FLUSH LOGS part_log timeout under runner load; praktika's own 38 reruns all passed, it passes locally on this branch, and the PR diff only touches partition-export/Iceberg code with no relation to lightweight deletes, vertical merges, or system-log flushing.

UNRELATED

Triage

PR diff scope (git log antalya-26.6..HEAD → the two PR commits 61fb4babb52 + c1efccca5eb, a cherry-pick of #1832 "Add commit info to partition exports table"):

  • src/Storages/MergeTree/ExportReplicatedMergeTreePartition*, ExportPartitionManifestUpdatingTask.*, ExportPartitionUtils.cpp
  • src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.*, StorageObjectStorage*.{h,cpp}, IDataLakeMetadata.h
  • src/Storages/System/StorageSystemReplicatedPartitionExports.*, src/Common/FailPoint.cpp (+1 failpoint), IStorage.h, docs, integration tests for partition export.

Nothing in the diff touches lightweight deletes, vertical merges, MergeTask, or system-log flushing.

03599_lightweight_delete_vertical_merge — the CI excerpt shows the failure was Code: 159 … Timeout exceeded (180 s) while flushing system log 'SystemLogQueue<PartLogElement>' on SYSTEM FLUSH LOGS part_log;, i.e. a loaded-runner timeout, and praktika's own randomized-settings diagnosis reported 38 reruns, 0 failed ("failure is not reproducible (likely a transient issue)"). Classified NOT-THIS-PR.

Verification

Ran the test locally with this shard's runner flags (--s3-storage --no-stateful --no-sequential, per ci/jobs/functional_tests.py--distributed-plan is a server-install option, not a runner arg):

  • First attempt failed spuriously because the prescribed rm -rf ci/tmp clipped parts out of the running server's own system.part_log store dir (filesystem error: in file_size … skp_idx_event_time_index.cmrk2) — my own environment damage, not the test. After TRUNCATE TABLE system.part_log SYNC:
[1 / 1] 03599_lightweight_delete_vertical_merge:   [ OK ] 0.18 sec.
1 tests passed. 0 tests skipped.

No code was changed; git status --porcelain is empty and no commits were added.

Summary

  • 03599_lightweight_delete_vertical_merge[unrelated] — CI failure was a 180 s SYSTEM FLUSH LOGS part_log timeout under runner load; praktika's own 38 reruns all passed, it passes locally on this branch, and the PR diff only touches partition-export/Iceberg code with no relation to lightweight deletes, vertical merges, or system-log flushing.

UNRELATED

⏭️ UNRELATED — Stateless tests (amd_debug, sequential)

1 failed test(s) considered — cost $0.5739
full report

AI narration

I'll start by triaging the failure against the PR diff.

Triage

PR diff (git diff antalya-26.6..HEAD --stat): a large cherry-pick of #1832 — 1684 files, dominated by Iceberg/Puffin, object-storage cluster table functions, partition-export (system.exports, MergeTreePartExportManifest), hybrid watermarks, text-index and system-table changes. Neither tests/queries/0_stateless/00024_random_counters.sql nor its .reference is touched by the diff (git diff … -- <those paths> → empty).

00024_random_counters: tagged stateful, no-parallel, no-msan — ~1500 SELECT uniq(UserID), sum(Sign) FROM test.visits WHERE CounterID = … queries over the standard visits dataset, then a rerun block with min_bytes_to_use_direct_io=1 / use_uncompressed_cache=1. The CI failure is not a result mismatch but Reason: Timeout! Killing process group … on a Debug (sequential) shard with randomized settings. The praktika report's own randomized-settings diagnosis states the rerun with identical randomized settings passed (Runs: 1, Failed: 0, Passed: 1 — "All reruns passed. The failure is not reproducible"), i.e. CI itself already classified it as transient.

There is no path from this diff to a slowdown of plain MergeTree uniq/sum scans over test.visits: the changed code paths (object storage / Iceberg / export manifests / cluster table functions) are not exercised by this test at all. I cannot write the required one-sentence "this PR broke this test because ", so per the scoping rule I made no code change (also, reproducing locally would require the multi-GB stateful hits/visits bootstrap, and a wall-clock timeout on different hardware is not a meaningful reproduction signal anyway).

No build was run, no commits were made, working tree is clean (git status --porcelain → no output).

Results

  • 00024_random_counters[unrelated] — timed out (not a diff mismatch) on the Debug sequential shard; CI's own same-settings rerun passed, the test file is untouched by this PR, and the PR's diff (Iceberg/object-storage/export code) is not exercised by these test.visits MergeTree aggregate queries.

UNRELATED

Triage

PR diff (git diff antalya-26.6..HEAD --stat): a large cherry-pick of #1832 — 1684 files, dominated by Iceberg/Puffin, object-storage cluster table functions, partition-export (system.exports, MergeTreePartExportManifest), hybrid watermarks, text-index and system-table changes. Neither tests/queries/0_stateless/00024_random_counters.sql nor its .reference is touched by the diff (git diff … -- <those paths> → empty).

00024_random_counters: tagged stateful, no-parallel, no-msan — ~1500 SELECT uniq(UserID), sum(Sign) FROM test.visits WHERE CounterID = … queries over the standard visits dataset, then a rerun block with min_bytes_to_use_direct_io=1 / use_uncompressed_cache=1. The CI failure is not a result mismatch but Reason: Timeout! Killing process group … on a Debug (sequential) shard with randomized settings. The praktika report's own randomized-settings diagnosis states the rerun with identical randomized settings passed (Runs: 1, Failed: 0, Passed: 1 — "All reruns passed. The failure is not reproducible"), i.e. CI itself already classified it as transient.

There is no path from this diff to a slowdown of plain MergeTree uniq/sum scans over test.visits: the changed code paths (object storage / Iceberg / export manifests / cluster table functions) are not exercised by this test at all. I cannot write the required one-sentence "this PR broke this test because ", so per the scoping rule I made no code change (also, reproducing locally would require the multi-GB stateful hits/visits bootstrap, and a wall-clock timeout on different hardware is not a meaningful reproduction signal anyway).

No build was run, no commits were made, working tree is clean (git status --porcelain → no output).

Results

  • 00024_random_counters[unrelated] — timed out (not a diff mismatch) on the Debug sequential shard; CI's own same-settings rerun passed, the test file is untouched by this PR, and the PR's diff (Iceberg/object-storage/export code) is not exercised by these test.visits MergeTree aggregate queries.

UNRELATED


🤖 Posted automatically by releasy analyze-fails. Re-run the command to refresh.

@blau-ai

blau-ai commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

CI triage

Verdict: 4 distinct failure groups. 2 are not caused by this PR (Swarms + Stateless — unrelated tests, flaky/timeout, also red on antalya-26.6). 2 are in the export‑partition feature this PR extends (s3_export_part, Iceberg export partition) and are blocking — they need to be looked at before merge. I can't fully attribute those to this PR's delta vs. the underlying feature yet, for the reason in §3.

Reduced "releasy" workflow, so most jobs show skipping; only the checks below actually ran.


1. Stateless tests — NOT PR‑caused (flaky / timeout)

Job Failing test Signal
amd_debug, sequential 00024_random_counters [ FAIL ] 600.03 sec — hit the exact 600 s test timeout → hung, not a functional fail
amd_debug, parallel 03599_lightweight_delete_vertical_merge [ FAIL ] 181.01 sec

Neither test touches export code (the diff only changes src/Storages/*Export*, Iceberg/ObjectStorage commit, system.replicated_partition_exports, FailPoint.cpp, and docs). 00024_random_counters timing out at exactly 600 s is a classic hang/infra timeout. → Safe to re‑run; not this PR.

2. Swarms (aarch64 + release) — NOT PR‑caused (pre‑existing / flaky infra)

Only 2–3 scenarios fail, all unrelated to exports: node failure / check restart swarm node, object storage cluster profile events with overloaded node, task rescheduling / rescheduling with bucket granularity. The same suite is red on the base branch antalya-26.6 with ~2 failing scenarios (1583 scenarios (…2 failed…)). → Pre‑existing/flaky; not this PR.

3. s3_export_part + Iceberg export partition (iceberg_2) — PR‑area, BLOCKING

These are the exact feature this PR builds on, and they fail broadly:

  • s3_export_part: 9/17 features failed, ~19 scenarios (error handling, concurrent alter, concurrent other, stress).
  • iceberg_2: 89/207 features failed, 242 scenarios — including the most basic export partition/.../sanity/export single partition and manifest integrity / snapshot summary total-records matches exported row count.

Why I can't yet pin these on this PR specifically: on the base branch antalya-26.6 these trees never execute — feature‑gated off:

✘ '/s3/minio/export tests/export part/feature support validation' ᐅ Export merge tree part is not supported by this build; the feature is not present in this version.

So there is no green baseline — this branch is the first place these scenarios actually run to completion. The failures are therefore either (a) a real regression introduced somewhere in the export feature chain (possibly this PR's conflict resolution — the branch history shows an AI‑resolved cherry‑pick of #1832), or (b) the regression suite's expected system.replicated_partition_exports snapshots not matching the 6 new columns this PR adds.

Narrowing signal: the sibling suite s3_export_partition is GREEN on this PR (1 module (1 ok), both arches). So partition‑level export to plain S3 works end‑to‑end; only part‑level export and iceberg export fail. That points at the part‑export / commit / commit‑info path rather than a total breakage.

What I could not do: the exact assertion diffs live only in the testflows report.html artifacts on S3 (iceberg2, s3part). The console logs only emit a bare AssertionError under output_format: new-fails, and I have no network access to fetch the reports (WebFetch/curl disabled) and cannot build/run ClickHouse in this container. So I can't hand you a verified one‑line fix for these yet.

Concrete things to check (hypotheses, in priority order):

  1. New commit multi‑op in ExportPartitionUtils::commit — the COMPLETED transition changed from a single trySet(status) to tryMulti([set status, create commit_info]). Verify the commit_info znode's parent path exists at commit time and that a non‑ZNODEEXISTS multi failure (e.g. the set status op failing) isn't silently turning a normal commit into a thrown NETWORK_ERROR. If the commit_info create can abort the multi, status never flips to COMPLETED and every test that polls system.replicated_partition_exports for COMPLETED will assert/time out — which matches the breadth.
  2. std::optional back‑compat fields (parquet_compression_method, output_format_compression_level, parquet_row_group_size[_bytes]): fromJsonString now leaves these nullopt when absent, and commit() only applies the setting when present. For a fresh export on this branch the initiator should always serialize them, but confirm the writer path still sets them (a missing row‑group size could change produced Parquet and break manifest‑integrity / row‑count assertions).
  3. Confirm whether the regression suite already expects the 6 new columns (query_id, destination_file_paths, committed_metadata_file, committed_manifest_list, committed_manifest_file, committed_marker_file). If a scenario does SELECT * / snapshot‑compares the table, the added columns alone would fail it and the fix is on the regression‑repo side, not here.

4. Aggregate PR check — just the rollup of the above; no independent error.


Bottom line: Swarms + Stateless are noise (re‑run). The export‑partition suites are the real signal and must be triaged before this merges. I can help further if you either (a) paste one failing scenario's assertion from the report.html, or (b) want me to review the #1832 conflict resolution / the commit multi‑op diff line‑by‑line for the bug. I can't fetch the S3 reports or build/run here myself.

@blau-ai · evidence: run 31787415505 (PR) vs base run 31713806632 on antalya-26.6

@blau-ai

blau-ai commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

CI triage for #2210

Verdict: 9 failing checks. None is a clean, provably PR-introduced hard failure I can pin from the logs, but two regression suites land squarely on this PR's feature area and deserve the author's eyes. Breakdown: 2 stateless = flaky/unrelated, swarms = flaky (branch baseline), iceberg_2 + s3_export_part = pre-existing "known-red" on antalya-26.6, but exercised on this PR's changed code — needs a look at the report.html I can't fetch.

Caveat: I can't build/run ClickHouse or open the S3 report.html artifacts (no network tooling in my container). The TestFlows --log-failed output truncates most tracebacks to a bare AssertionError, so for the regression suites I'm reasoning from failure location + one leaked exception + cross-PR comparison, not the full assertion diffs.


Not caused by this PR

1. Stateless 00024_random_counters (amd_debug, sequential) — flaky/timeout
[909/909] 00024_random_counters: [ FAIL ] 600.03 sec — 600s is the exact test timeout. It's a heavy stateful, no-parallel query over test.visits, has nothing to do with partition export/Iceberg, and this PR touches no stateless tests. Debug-build slowness. Safe to re-run.

2. Stateless 03599_lightweight_delete_vertical_merge (amd_debug, distributed plan / s3 / parallel) — flaky/unrelated
[ FAIL ] 181.01 sec. Lightweight-delete + vertical-merge test, outside this PR's code area. Not PR-caused. Safe to re-run.

3. Regression Swarms (aarch64 + release) — flaky
1521 scenarios (1519 ok, 2 failed) / (1518 ok, 3 failed) — 2–3 scenario failures out of 1521. This suite is also red on recently merged PRs into antalya-26.6 (#2201, #2184), so it's branch baseline noise, not this PR.


Pre-existing "known-red" on the base branch, but touching this PR's code — please verify

4. Regression Iceberg (2) iceberg_2 (aarch64 + release)
207 features (87 ok, 89 failed, 31 skipped). All 89 failures are under /iceberg/export partition/... — the exact feature this PR changes; the non-export Iceberg features pass. One real server exception leaked through the truncation:

Code: 57. DB::Exception: Iceberg table with path data/iceberg_<uuid>/ already exists. (TABLE_ALREADY_EXISTS)
  at /iceberg/export partition/.../transactions/commit durable across post-publish exception

The other ~88 are bare result-content AssertionErrors (the query succeeds, the assertion on its output fails) — consistent with this PR's schema/behavior change: new query_id / destination_file_paths / committed_metadata_file|manifest_list|manifest_file|marker_file columns and transaction_id switching from a snowflake integer to a UUID string.

5. Regression S3Export (part) s3_export_part (aarch64 + release)
17 features (8 ok, 9 failed). All failures under /s3/minio/export tests/export part/... (error handling, concurrent alter). The leaked assertion is a process-exit-code check: assert results[0].exitcode == 237 — a behavioral expectation in the error-handling scenarios, not a schema diff.

Why "pre-existing" but still flagged: these exact suites (iceberg_2, s3_export_part, plus iceberg_1, s3_export_partition) are failing on already-merged PRs into antalya-26.6 — e.g. #2201 (merged 2026-08-14) and #2184 (merged 2026-08-12) both show them red, and #2201's iceberg_2 ran the full ~2h26m just like here. So red regression is the branch's baseline and did not block those merges.

But don't dismiss them outright: #2210's iceberg_2 ran a much larger export-partition suite (207 features vs 34 on #2201 — a different clickhouse-regression version), and every failure is in the code this PR modifies. I therefore can't rule out that this PR (or the conflict resolution in c1efccca5eb) shifted the pass/fail set. Settling it needs the S3 report.html diffs, which I can't reach.


Suggested next steps

  • Re-run the two stateless jobs and Swarms — expected flaky/timeout.
  • For iceberg_2 / s3_export_part: open the report.html (linked on the checks) and diff the failing-scenario list against a base antalya-26.6 run. In particular, check whether the regression suite version used here already expects the new committed_* / destination_file_paths / query_id columns and the UUID transaction_id, or still expects the old snowflake-int transaction_id — a suite/code schema mismatch would explain the ~88 result-mismatch assertions wholesale.
  • Sanity-check the forward-port resolution: the commit durable across post-publish exception recovery scenario threw TABLE_ALREADY_EXISTS, so please confirm c1efccca5eb didn't alter the commit/recovery path (the once-only commit_info znode write done atomically with the status→COMPLETED transition).

If you'd like, I can go deeper on any single suite, or draft a fix once we know whether the break is code vs. regression-suite expectations — just point me at it.

@zvonand

zvonand commented Aug 15, 2026

Copy link
Copy Markdown
Member Author

CI triage for #2210 — correction

Verdict: none of the CI failures are caused by this PR.

This supersedes my earlier triage comment, which classified iceberg_2 and s3_export_part as regressions from the conflict resolution. That was wrong — it reasoned from the diff instead of comparing against a baseline. Below is the result of diffing the actual testflows result trees.

Method

Byte-level diff of the per-scenario result trees for every regression suite, against two independent baselines on the same branch:

All at x86_64/with_analyzer/zookeeper/without_thread_fuzzer.

The 17 failing checks = 5 distinct failures

Check Classification PR-caused?
Regression iceberg_2 (release + aarch64) Pre-existing on antalya-26.6; suite is ahead of the branch No
Regression s3_export_part (release + aarch64) Pre-existing; strict subset of the baselines No
Regression swarms (release + aarch64) Flaky, plus one pre-existing base-branch failure No
Stateless amd_debug, distributed plan03599_lightweight_delete_vertical_merge SYSTEM FLUSH LOGS timeout, unrelated subsystem No
Stateless amd_debug, sequential00024_random_counters 600 s harness timeout; CI's own rerun passed No

The top-level PR check is the aggregate rollup of the above.


iceberg_2 — identical to baseline, zero delta

332 failures on this PR. Exactly the same 332 on #2209 and on #2208 — no scenario fails here that passes there, and none the other way round.

Root cause: the pinned Altinity/clickhouse-regression suite (974b380ca9213a0f88f37a1c7f10793fab861ac3) tests features that do not exist on antalya-26.6:

  • 251 × EXPORT PARTITION is not implemented for engine MergeTree. (NOT_IMPLEMENTED) — the whole plain merge tree subtree (548 of the 664 failing entries) exercises plain-MergeTree export, which is not present in this branch.
  • 42 × Unrecognized option '--export_merge_tree_part_schema_mismatch_mode' — the setting does not exist on 26.6.
  • The remainder are test-isolation leftovers in the same suites (TABLE_ALREADY_EXISTS, already exported or it is being exported).

This will stay red on every 26.6 pull request until the suite feature-gates those two things; it is a clickhouse-regression fix, not a ClickHouse one.

s3_export_part — strictly better than baseline

28 failures here vs 29 on both #2209 and #2208. The set here is a strict subset: /s3/minio/export tests/export part/concurrent alter/after export fails on both baselines and passes on this PR. Nothing fails here that passes there.

That includes /export part/error handling/pending mutations (the exitcode == 237 / PENDING_MUTATIONS_NOT_ALLOWED case) and the /concurrent alter/during minio interruption hang, which my earlier comment called out as decisive evidence — both fail identically on the baselines.

swarms — flaky, plus one base-branch failure

The failures rotate across pull requests and architectures, so they do not track this PR:

Scenario #2208 #2209 #2210 x86 #2210 aarch64
node failure/check restart swarm node Fail OK Fail Fail
feature/object storage cluster profile events with overloaded node OK OK Fail OK
task rescheduling/rescheduling with bucket granularity OK Fail Fail Fail

object storage cluster profile events with overloaded node fails on x86 and passes on aarch64 at the same SHA. check restart swarm node fails with Code: 394 ... QUERY_WAS_CANCELLED during a deliberate node kill.

One of these is not flaky and deserves its own issue: task rescheduling/rescheduling with bucket granularity fails with the identical count — Expected 200000 total rows, but got 168783 — on #2209 and on both architectures of #2210. Deterministic, and reproducible on a settings-only pull request, so it is a base-branch issue. This PR touches no swarm scheduling code; its only StorageObjectStorageCluster change is a voidExportPartitionCommitInfo return-type plumb.

Stateless tests

  • 03599_lightweight_delete_vertical_mergeCode: 159. DB::Exception: Timeout exceeded (180 s) while flushing system log 'DB::SystemLogQueue<DB::PartLogElement>' on SYSTEM FLUSH LOGS part_log, under debug + s3 + distributed plan. Lightweight-delete / vertical-merge subsystem, untouched by this PR.
  • 00024_random_counters[ FAIL ] 600.03 sec, the harness test timeout under amd_debug sequential load. CI's own randomized-settings diagnosis re-ran it with the same settings and it passed: "All reruns passed. The failure is not reproducible (likely a transient issue)."

Positive evidence that the port is correct

My earlier comment claimed the integration tests were skipped. They were not — they ran and passed:

Integration tests (arm_binary, distributed plan), all 4 shards, 0 failures out of 6002 results. Those shards include the files this PR modifies:

  • test_export_replicated_mt_partition_to_object_storage/test.py (shard 2)
  • test_export_replicated_mt_partition_to_iceberg/test.py (shard 3)

which are exactly the tests asserting the new system.replicated_partition_exports commit-info columns and the UUIDv4 transaction_id. Regression release/aarch64 s3_export_partition, iceberg_1 and all parquet* suites passed on both architectures as well.

Conflict-resolution audit

Re-checked c1efccca against 0329de17 (the original #1832 merge):

  • No leftover conflict markers in src/, docs/ or tests/.
  • IcebergMetadata.cppstorage_manifest_entry_pathstorage_manifest_entry_name is a required rename: 26.6 renamed the local, and both hold resolver.resolve(manifest_entry_path), so the value is unchanged.
  • IcebergMetadata::commitExportPartitionTransaction — the retry loop preserves Add commit info to partition exports table #1832's semantics: if (commit_info) return *commit_info; otherwise retry, throwing UNFINISHED after MAX_TRANSACTION_RETRIES.
  • StorageObjectStorage::commitExportPartitionTransaction — keeps 26.6's metadata_snapshot / configuration arguments while returning Add commit info to partition exports table #1832's ExportPartitionCommitInfo.
  • docs/en/antalya/partition_export.md — retains the full new content (destination_file_paths and the committed_* sections).

Recommendation

Nothing to fix in this PR. Re-running swarms is the only action that could turn a check green; iceberg_2 and s3_export_part are branch-level noise that no change here can clear.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-resolved Port conflict auto-resolved by Claude antalya-26.6 forwardport This is a frontport of code that existed in previous Antalya versions releasy Created/managed by RelEasy

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants