Skip to content

0.49 gcs deltalake - #95

Open
zhoucai-pingcap wants to merge 9 commits into
0.49from
0.49-gcs-deltalake
Open

0.49 gcs deltalake#95
zhoucai-pingcap wants to merge 9 commits into
0.49from
0.49-gcs-deltalake

Conversation

@zhoucai-pingcap

Copy link
Copy Markdown
Collaborator

Summary

Add Google Cloud Storage (GCS) support for Delta Lake sinks, harden write recovery when Delta log views go stale (e.g. after external compaction), and shrink Docker image size for release builds.

Changes

Delta Lake / GCS

  • Enable the gcs feature on the deltalake crate so tables can be written to GCS in addition to S3 and Azure.
  • Centralize stale Delta log detection in is_stale_delta_log_error (shared by the writer and sinks).
  • On stale log / missing _delta_log segment errors:
    • Retry inside DeltaOpsManager after reloading the table.
    • Evict the cached writer and retry once in deltalake, topsql_data_deltalake, and topsql_meta_deltalake sinks (with a short backoff).
  • Stop panicking on these corruption-like errors; log and recover so the pipeline can continue.

Docker / release size

  • Release profile: strip = true, panic = "abort", opt-level = "z".
  • Slim the Debian Dockerfile (multi-stage, drop unused runtime packages).
  • Support alternate bases via DOCKER_BASE (alpine / minimal) and add Makefile targets:
    • release-docker-alpine
    • release-docker-minimal
    • release-docker-nextgen-alpine

Tests

  • Add tests/deltalake_stale_log_test.rs to simulate external compaction removing a _delta_log JSON file and verify write recovery after reopening the writer.

Commits (vs 0.49)

  • 393f2b5 deltalake gcs
  • 6636544 / 10bc14c optimize docker image size
  • 6865bfc fix delta lake write issue
  • (plus merge commits from 0.49)

Test plan

  • Build release binary and confirm GCS Delta Lake writes succeed with valid GCS credentials / storage options
  • Run unit tests for is_stale_delta_log_error
  • Run tests/deltalake_stale_log_test.rs and confirm recovery after simulated log compaction
  • Smoke-test deltalake / topsql_data_deltalake / topsql_meta_deltalake sinks against S3 and GCS
  • Verify Debian Docker image still builds; optionally verify Alpine/minimal targets if those Dockerfiles are shipped with this PR
  • Confirm image size reduction vs previous 0.49 Debian image

@ti-chi-bot ti-chi-bot Bot added the size/XXL label Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant