Antalya 26:6 Fix alter operations for iceberg - #2157
Conversation
|
ADD COLUMN DROP COLUMN RENAME COLUMN |
| { | ||
| switch (type->getTypeId()) | ||
| { | ||
| case TypeIndex::UInt8: |
There was a problem hiding this comment.
Added support for bool
| return {"string", true}; | ||
| case TypeIndex::UUID: | ||
| return {"uuid", true}; | ||
| case TypeIndex::Decimal32: |
There was a problem hiding this comment.
Added support for iceberg decimal types.
….com/Altinity/ClickHouse into antalya_26_6_fix_alter_table_iceberg
|
Decimal support |
|
I asked Claude to analyze this PR and it came with some review. Please have a look: https://gist.github.com/mkmkme/b28a41f136a62d5a6d41a73dee0132bd |
….com/Altinity/ClickHouse into antalya_26_6_fix_alter_table_iceberg
….com/Altinity/ClickHouse into antalya_26_6_fix_alter_table_iceberg
… fix medium defects
|
@mkmkme can u check again please |
@subkanthi the new one is quite cleaner, but there're still some issues worth fixing: https://gist.github.com/mkmkme/98ffbceed6d86e14339537481a5f162c |
|
|
||
| } | ||
|
|
||
| MetadataGenerator::MetadataGenerator(Poco::JSON::Object::Ptr metadata_object_) |
There was a problem hiding this comment.
Just noting that dropping a column used in partitioning or ordering has had its behaviour changed, especially that of an ordering column which is now completely rejected.
There was a problem hiding this comment.
This is an iceberg spec:
(Section "Schema Evolution") explicitly forbids this: a column that is part of the current sort order or partition spec cannot be removed without first changing the sort order/partition spec to no longer reference it.
Thanks @mkmkme , i will take a shot at fixing 1, 2, for #5 Glue catalog we have separate issues which I thought is better to go in a separate PR. ALTER TABLE commit kept losing to concurrent modifications after 100 retries There was no concurrent modification. And the orphan v<N+1>.metadata.json makes every subsequent ALTER on that table repeat this until someone deletes it by hand. The loop structure is pre-existing; the misleading message is new, and the unused cleanup failpoint suggests the real fix was intended. |
|
|
Could you fix the conflicts please? |
|
Agent found a potential issue: Nested ADD COLUMN + commit-unknown recoveryiceberg_alter_catalog_commit_reported_as_failed makes RestCatalog::updateSchema apply the catalog commit and then return false. The ALTER retry path is supposed to notice the column is already there via MetadataGenerator::isAddColumnApplied and treat the op as succeeded. That works for a primitive ADD COLUMN z Nullable(String) (test_writes_alter_when_commit_is_reported_as_failed). It does not work for nested types. SET allow_insert_into_iceberg = 1, enable_nullable_tuple_type = 1;
ALTER TABLE … ADD COLUMN tuple_col Nullable(Tuple(Int32, String));
Code: 36. DB::Exception: Column tuple_col already exists. (BAD_ARGUMENTS)The catalog already committed the column. Recovery missed it, so the retry called generateAddColumnMetadata again. CauseisAddColumnApplied (MetadataGenerator.cpp ~181–199) rebuilds the expected Iceberg type with getIcebergType(type, last_column_id) using the current last-column-id (already advanced by the successful commit). For primitives that is a string ("string", "long", …) and the compare succeeds. For Array / Map / Tuple, getIcebergType allocates new nested field ids from that high watermark. icebergTypesEqual stringifies the JSON, so element-id / key-id / value-id / struct field ids do not match the committed type → returns false → retry → Column already exists. |
….com/Altinity/ClickHouse into antalya_26_6_fix_alter_table_iceberg
PR CI Triage SummaryIceberg ALTER coverage passed (iceberg_1). None of these are caused by this PR. Same jobs are already red on MasterCI antalya-26.6 iceberg_2 — EXPORT PARTITION tests against a 26.6 binary that does not have that feature. Unrelated to Iceberg ALTER. s3_export_part — same export-feature mismatch, then cascade after the server died. Sanity cases passed. settings — missing snapshots for new 26.6 settings. Already failing on MasterCI. Stateless amd_debug + distributed plan + s3 — three 600s timeouts. Passed on the other stateless jobs in this run. Known flakes. swarms — historically flaky node-restart test. Already failing on MasterCI. tiered_storage_minio — disk-name flake. Passed on the other tiered-storage jobs in this run and on MasterCI. |
continuation of work from #1841
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
...
Documentation entry for user-facing changes
...
CI/CD Options
Exclude tests:
Regression jobs to run: