Build: Pull MinIO images from Quay by smaheshwar-pltr · Pull Request #18071 · apache/iceberg

smaheshwar-pltr · 2026-09-11T22:24:55Z

Docker Hub is no longer serving minio/minio (https://hub.docker.com/r/minio/minio) or minio/mc (https://hub.docker.com/r/minio/mc), stopping the AWS integration tests and Kafka Connect integration tests from starting on main (+ open PRs).

This PR switches to the corresponding Quay images, as the related PRs above do - those other projects are encountering the same issue as us.

Note: we have discussed moving away from MinIO before, but I don't think that has progressed - see:

Docker Hub no longer serves the MinIO server and client repositories. Use the existing Quay images so AWS and Kafka integration tests can start, and update the Flink quickstart references as well. Generated-by: Codex

Keep the runnable Compose example working after the MinIO Docker Hub repositories became unavailable. Generated-by: Codex

nssalian

LGTM. Confirmed the quay.io/minio/* images (including the RELEASE.2024-12-18T13-15-44Z tag, quay.io/minio/mc:latest and quay.io/minio/minio:latest) resolve locally.
Pending CI to complete.
It is worth bringing this up in the mailing list of long term use of MinIO versus its alternatives

szehon-ho

Yea it seems to be breaking CI (I hit it too). I am ok with this short term fix to unblock folks, we can discuss the long term later (move off MinIO)

smaheshwar-pltr · 2026-09-11T23:16:06Z

Thanks both for the reviews, agreed that this a strong data point to move off MinIO - we had devlist discussions here previously, I'll send out an email now to revive it.

Include the MinIO image fix from apache#18071. Generated-by: Codex

smaheshwar-pltr · 2026-09-12T01:22:37Z

Mailing list thread: https://lists.apache.org/thread/k5hjjtmj1z3gdwtrlf4v5ovz1wp1p14q

(cherry picked from commit cff60bc) Co-authored-by: Sreesh Maheshwar <maheshwarsreesh@gmail.com>

### What changes were proposed in this pull request? Point `MinIOContainer.DEFAULT_IMAGE` at `quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z`. It's the same MinIO release, pulled from MinIO's quay.io organization instead of Docker Hub. ### Why are the changes needed? Docker Hub removed the `minio/minio` repository, so every integration test that starts `MinIOContainer` fails with `pull access denied`. That includes `FilesetS3TokenConnectionIT` in Backend Integration Test and `IcebergRESTMinIOTokenAuthorizationIT`. This is a temporary fix, the same one apache/iceberg#18071 and apache/doris#67897 merged. Replacing MinIO is tracked in the discussion on #13111. Fix: #13111 ### Does this PR introduce _any_ user-facing change? No. Test infrastructure only. ### How was this patch tested? - `docker pull minio/minio:RELEASE.2025-09-07T16-13-09Z` fails with `pull access denied`; `docker pull quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z` succeeds. - `./gradlew :integration-test-common:spotlessCheck :integration-test-common:compileTestJava` passes. - CI on this PR runs `FilesetS3TokenConnectionIT`.

CI 两个 Docker 相关 job 长期恒红,本次定位到**同一根因**: - `Compose 集成测试` —— 失败步骤 `全拓扑构建与启动`,耗时**约 2 秒**; - `E2E 三浏览器矩阵` —— 失败步骤 `矩阵 spec(E2E_MATRIX=1)`,耗时**约 6 秒** (= Playwright 启动 + `global-setup` 的 `topologyRunning` 2 秒轮询 + 同一 compose 拒绝);该 job 的 `globalSetup` 经 `helpers/compose.ts#composeAppUp()` 与前者**共用同一套 compose 拓扑**。 ## 根因(本机非破坏性复现,逐字原文) `compose/deps.yaml` 的 `minio` 与 `compose/app.yaml` 的 `verifier-minio-init` 使用 Docker Hub 的 `minio/minio:latest` / `minio/mc:latest`。**Docker Hub 上的 `minio/minio` 与 `minio/mc` 已不再接受匿名拉取**: ``` $ docker pull minio/minio:latest Error response from daemon: pull access denied for minio/minio, repository does not exist or may require 'docker login' (exit 1) $ docker manifest inspect minio/minio:latest denied: requested access to the resource is denied unauthorized: authentication required ``` `minio/mc:latest` 同形。CI runner **无 Docker Hub 凭证、无本地缓存** ⇒ compose 在拉取阶段被 registry 立即拒绝(**无任何层下载**,故秒级失败),两条 job 同时打红。 **为什么本地一直绿**:本机**已缓存**这两个镜像,compose 不触发拉取 ⇒ 该错在开发机上结构性不可见。**又一例「本地绿、CI 恒红」**。 **时间量级对照**:无层下载的 registry 拒绝路径本机实测 6.7 秒(Windows + CLI 启动开销),CI 上 2 秒量级自洽。 **上游迁移佐证**(多项目已跟进切 quay.io): - gotempsh/temps — 「switch MinIO images from Docker Hub to quay.io」 gotempsh/temps#963 - apache/iceberg — 「Build: Pull MinIO images from Quay」 apache/iceberg#18071 - camunda/camunda — 「test: use quay hosted minio image」 camunda/camunda#63189 ## 修法两处镜像改为 **quay.io** 并**固定 tag**(原为 `:latest`,是本次脆弱性的来源): | 位置 | 改前 | 改后 | |---|---|---| | `compose/deps.yaml`(minio) | `minio/minio:latest` | `quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z` | | `compose/app.yaml`(verifier-minio-init) | `minio/mc:latest` | `quay.io/minio/mc:RELEASE.2025-08-13T08-35-41Z` | 两个 tag **正是本机已缓存、既有拓扑一直在用的版本**(`minio --version` / `mc --version` 实测读取)⇒ 行为等价、零语义变化,只是把来源换成可匿名拉取者并把版本钉死。各加注释说明「**不得改回 Docker Hub**」及原因。 ## 验证 - `quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z` 与 `quay.io/minio/mc:RELEASE.2025-08-13T08-35-41Z`:`docker manifest inspect` 均 OK (对比 Docker Hub 侧 denied); - **实拉实测**:`docker compose -f compose/deps.yaml up -d --wait` ⇒ **exit 0**, postgres / redis / **minio** / db-roles-init 全部 **Healthy**(真正走了一次 quay 拉取,非缓存命中); - `docker pull quay.io/minio/mc:...` ⇒ `Status: Downloaded newer image`, `mc --version` = `RELEASE.2025-08-13T08-35-41Z`(与 init 脚本既有实跑版本一致); - `docker compose config --quiet` 两份文件组合 exit 0; - **清理**:`down -v` exit 0,容器与卷零残留(已核对 `docker ps -a` / `docker volume ls`)。

smaheshwar-pltr added 2 commits September 11, 2026 23:19

Build: Pull MinIO images from Quay

75b6e7f

Docker Hub no longer serves the MinIO server and client repositories. Use the existing Quay images so AWS and Kafka integration tests can start, and update the Flink quickstart references as well. Generated-by: Codex

Docs: Use Quay images in the Spark quickstart

bf835e7

Keep the runnable Compose example working after the MinIO Docker Hub repositories became unavailable. Generated-by: Codex

github-actions Bot added docs AWS KAFKACONNECT labels Sep 11, 2026

This was referenced Sep 11, 2026

ci: pull MinIO images from Quay apache/iceberg-rust#3202

Merged

ci: use Quay and GitHub releases for MinIO apache/iceberg-cpp#933

Merged

nssalian approved these changes Sep 11, 2026

View reviewed changes

smaheshwar-pltr mentioned this pull request Sep 11, 2026

ci: pull MinIO images from Quay apache/iceberg-go#2008

Merged

nssalian requested review from RussellSpitzer and amogh-jahagirdar September 11, 2026 22:49

szehon-ho approved these changes Sep 11, 2026

View reviewed changes

szehon-ho merged commit cff60bc into apache:main Sep 11, 2026
49 checks passed

smaheshwar-pltr added a commit to smaheshwar-pltr/iceberg that referenced this pull request Sep 11, 2026

Merge main with the upstream MinIO registry fix

c0c3806

Include the MinIO image fix from apache#18071. Generated-by: Codex

smaheshwar-pltr deleted the build/minio-quay-images branch September 12, 2026 01:22

huaxingao added a commit that referenced this pull request Sep 12, 2026

Build: Pull MinIO images from Quay (#18071) (#18075)

884277c

(cherry picked from commit cff60bc) Co-authored-by: Sreesh Maheshwar <maheshwarsreesh@gmail.com>

This was referenced Sep 13, 2026

[Bug report] MinIO integration tests fail: minio/minio is no longer available on Docker Hub apache/gravitino#13111

Closed

[#13111] fix(test): Pull the MinIO test image from quay.io apache/gravitino#13112

Merged

nevzheng mentioned this pull request Sep 14, 2026

[Improvement] Replace MinIO with a maintained S3 integration-test backend apache/gravitino#13154

Closed

6 tasks

Navigation Menu

Build: Pull MinIO images from Quay - #18071

Build: Pull MinIO images from Quay#18071
szehon-ho merged 2 commits into
apache:mainfrom
smaheshwar-pltr:build/minio-quay-images

smaheshwar-pltr commented Sep 11, 2026 •

edited

Loading

Uh oh!

nssalian left a comment •

edited

Loading

Uh oh!

szehon-ho left a comment •

edited

Loading

Uh oh!

Uh oh!

smaheshwar-pltr commented Sep 11, 2026

Uh oh!

smaheshwar-pltr commented Sep 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

smaheshwar-pltr commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nssalian left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

szehon-ho left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

smaheshwar-pltr commented Sep 11, 2026

Uh oh!

smaheshwar-pltr commented Sep 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

smaheshwar-pltr commented Sep 11, 2026 •

edited

Loading

nssalian left a comment •

edited

Loading

szehon-ho left a comment •

edited

Loading