Skip to content

feat(clickhouse): support DataLake catalog source mappings - #6153

Open
mday-io wants to merge 3 commits into
SQLMesh:mainfrom
mday-io:feat/clickhouse-datalake-catalog-mapping
Open

mday-io wants to merge 3 commits into
SQLMesh:mainfrom
mday-io:feat/clickhouse-datalake-catalog-mapping

Conversation

@mday-io

@mday-io mday-io commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Why

SQLMesh sources can use catalog.schema.table names, but ClickHouse DataLakeCatalog exposes them as database."schema.table". Without a mapping, SQLMesh cannot render these source references correctly for ClickHouse reads and model materialization.

How

Add datalake_catalog_mapping to the ClickHouse connection configuration. For example, {lake: iceberg} renders lake.analytics.events as iceberg."analytics.events". The mapping defaults to empty and applies only to parsed query sources. Raw SQL strings and INSERT targets remain unchanged.

Direct reads preserve unmapped catalogs. Materialization rejects unknown source catalogs when a mapping or injected virtual catalog is active. The existing restriction on cross-catalog view creation remains in place.

Includes configuration documentation and a dedicated ClickHouse, Iceberg REST, and MinIO integration fixture in the PR Docker CI matrix.

Test Plan

  • make style passed all four hooks on the current revision.
  • pytest -n 0 tests/core/engine_adapter/test_clickhouse.py passed 107 tests, including exact raw-SQL pass-through and unchanged INSERT-target checks.
  • Earlier full validation passed make fast-test with 2,933 passed and 13 skipped, and make doc-test with 29 passed.
  • The original live fixture passed 19 cases twice, including a clean restart and reseed. The audit removed one error-inference case and replaced it with stronger unit assertions. All 18 remaining live cases collect successfully.
  • Workflow YAML and Docker Compose configuration checks passed.
  • Complete the current GitHub CI run, including the 18-case DataLake suite. The latest local live rerun failed before startup because Colima's containerd metadata database returned an input/output error. Fixture teardown succeeded.

Checklist

  • I have run make style and fixed any issues
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test)
  • My commits are signed off (git commit -s) per the DCO

Signed-off-by: Michael Day <mdaytn@gmail.com>
Signed-off-by: Michael Day <mdaytn@gmail.com>
Signed-off-by: Michael Day <mdaytn@gmail.com>
@mday-io
mday-io marked this pull request as ready for review October 19, 2026 12:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant