fix(upgrade): move ingestion images to Python 3.12 and strip build-only tooling by harshsoni2024 · Pull Request #30878 · open-metadata/OpenMetadata

harshsoni2024 · 2026-08-03T14:34:19Z

Describe your changes:

Follow-up to #30751. That PR patched PySpark's vulnerable jars in place; this one
addresses the rest of the CVE surface in the ingestion images — the frozen base
layer and build tooling that ships into the runtime image.

python:3.10-slim-bookworm is EOL and no longer rebuilt, so its Debian layer and
bundled Python tooling are frozen. All four ingestion Dockerfiles move to
python:3.12-slim-bookworm / apache/airflow:3.2.2-python3.12.

Key changes

Base bump to Python 3.12 across ingestion/Dockerfile{,.ci} and
ingestion/operators/docker/Dockerfile{,.ci}. 3.12's ensurepip ships pip only —
no setuptools, no wheel — which removes the vendored wheel-0.45.1
behind CVE-2026-24049.
Bug fix: ingestion/Dockerfile pinned the Airflow Python 3.10 constraints
file while building on a python3.12 base → now constraints-3.12.txt.
Regenerated airflow-constraints-3.2.2.txt from upstream's 3.12 variant. Both
files come from the same 2026-05-26 generation run, so this is the matching sibling:
it drops the py<3.11 backports (backports.strenum, exceptiongroup, tomli) and
refreshes numpy/scipy/scikit-learn.
Interpreter-level pip pinned to 26.2, before the USER switch. The existing
pip install --upgrade pip runs after that switch, so it only reached ~/.local
and left the base image's copy on disk for scanners (CVE-2025-8869, CVE-2026-3219,
CVE-2026-6357, CVE-2026-8643 — fixed in 26.1.2). Python 3.12 alone is not enough:
its bundled pip is 25.0.1.
datamodel-code-generator + black uninstalled after codegen in the two CI
images. Neither is imported at runtime; they exist only to run
datamodel_generation.py, and account for 8 HIGH findings.

Type of change:

Bug fix
Improvement

High-level design:

Why the airflow images need /usr/local/bin/python -m pip rather than plain pip:
three independent reasons, all verified in the base image —

as root, pip resolves to airflow's /root/bin/pig wrapper, which exit 1s (so a
plain pip line fails the build, it doesn't silently no-op);
that wrapper execs ${HOME}/.local/bin/pip even for non-root, so it can never
reach the interpreter-level install;
bare python as root resolves to /home/airflow/.local/bin/python, so
python -m pip also targets ~/.local.

Only /usr/local/bin/python (→ /usr/python/bin/python3) reaches the copy scanners
report. The slim images are ordinary python:3.12-slim-bookworm with no wrapper, so
plain pip is correct there.

Alternative rejected — bumping datamodel-code-generator instead of removing it.
0.64.0+ is CVE-free, but scripts/datamodel_generation.py monkeypatches
datamodel_code_generator.model.pydantic.types.IMPORT_SECRET_STR (the
CustomSecretStr swap applied to every generated password field), and that module
path no longer exists in 0.26+ — probed on 0.25.6 / 0.64.0 / 0.71.0. The script also
post-processes generated output with exact string replacements that would fail
silently on a formatting change. A pin bump therefore needs a regenerate-and-diff of
the full src/metadata/generated/ tree; tracked separately. The uninstall runs after
the final pip install so no extra can pull the packages back in.

Alternative rejected — lifting setuptools<81. setuptools ≥83 fixes
CVE-2026-59890 but no longer ships pkg_resources, and
sqlalchemy_redshift/__init__.py:1 imports it unguarded at module top level, with
sqlalchemy-redshift==0.8.14 (2023) being the latest release. Left as a documented
residual.

Rollout / compatibility. cx_Oracle 8.3.0 and mysqlclient==2.1.1 have no cp312
wheel but both build cleanly from sdist on 3.12 (the toolchain and pkg-config are
already installed) — verified in a container. Bumping mysqlclient to 2.2.x for a
prebuilt wheel is a sensible follow-up but is not required here.

Tests:

Use cases covered

All four ingestion images build on Python 3.12.
The airflow image installs openmetadata-managed-apis under the matching 3.12
constraints file.
The shipped images no longer carry datamodel-code-generator / black, and their
interpreter-level pip is ≥ 26.1.2.

Unit tests

Not applicable — Dockerfile / build-constraint changes only, no Python logic.

Backend integration tests

Not applicable (no backend API changes).

Ingestion integration tests

Not applicable (no connector changes).

Playwright (UI) tests

Not applicable (no UI changes).

Manual testing performed

docker buildx build --check clean on all four Dockerfiles.
Verified pip install --upgrade pip as root in apache/airflow:3.2.2-python3.12
exits 1, and that /usr/local/bin/python -m pip install "pip==26.2" exits 0 and
lands in /usr/python/lib/python3.12/site-packages.
Installed openmetadata-airflow-apis under the regenerated constraints file in the
3.12 airflow base — succeeds, setuptools stays at the pinned 80.10.2, and
pkg_resources still imports.
Confirmed cx_Oracle 8.3.0 and mysqlclient==2.1.1 build from sdist on
python:3.12-slim-bookworm.
Confirmed pip uninstall -y datamodel-code-generator black exits 0 when the
packages are absent, so the layer is a safe no-op.

…park jar surgery The ingestion-slim image (ingestion/operators/docker/Dockerfile*) shipped vulnerable jars bundled by PySpark 3.5.6 that the earlier CVE fix (#30516) never patched — that fix only touched the airflow image (ingestion/Dockerfile*), so the PySpark jar surgery never reached the slim image. AWS Inspector kept flagging zookeeper, jackson-asl, netty, and a phantom black finding on it. Extract the jar patch into a single SHA256-pinned script and run it in all four ingestion Dockerfiles so the two images can no longer drift: - zookeeper 3.6.3 -> 3.7.2 CVE-2023-44981 (SASL quorum auth bypass) - jackson-mapper/core-asl 1.9.13 CVE-2019-10202 (removed; no upstream fix) - netty-codec-http 4.1.96 -> 4.1.135 CVE-2026-42581 / CVE-2026-42584 (both fixed in 4.1.133.Final; staying on the 4.1.x line keeps binary compatibility with PySpark's sibling netty 4.1.96 jars — verified pyspark and SparkSession still import) Also strip spacy's bundled tests/package/requirements.txt from the slim images: it pins an old black that scanners misreport as an installed package (CVE-2026-31900). The file is test-only and never imported at runtime. Left as documented residuals (no drop-in fix for this runtime): - derby 10.14.2.0 (CVE-2022-46337): only fix needs Java 21, image is Java 17; vulnerable LDAP path unused - jetty 9.4.x (CVE-2026-2332): shaded inside hadoop-client-runtime / spark-core uber-jars, not a swappable standalone jar; clears on a future pyspark bump Verified against a local build of the slim CI image: patched jars present, jackson-asl and the black fixture gone, pyspark imports cleanly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…cleanup into jar patch Address review feedback on the PySpark CVE-jar remediation: - Detection no longer fails open. Previously `python -c 'import pyspark' 2>/dev/null || true` treated a broken-but-installed pyspark the same as an absent one, letting an image with unpatched jars ship silently. The script now distinguishes three cases via exit code: genuinely absent (skip), installed-but-unimportable (fail the build), importable (patch). - Fold the spaCy test-fixture cleanup (spacy/tests/**/requirements.txt, the phantom black CVE-2026-31900) into patch_pyspark_jars.sh so it runs wherever the patch runs, and drop the separate per-Dockerfile `find ... -delete` RUN lines that could drift. Self-guarding and idempotent: no-op when pyspark is absent (INGESTION_DEPENDENCY=slim builds), active when present (default `all`). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…lerant delete Address review feedback on patch_pyspark_jars.sh: - The spaCy fixture cleanup was placed after the early `exit 0` that fires when PySpark is absent, so a build with spaCy but no PySpark (e.g. INGESTION_DEPENDENCY=pii-processor — spaCy comes from pii-processor/sample-data, PySpark from deltalake, independent extras) skipped the strip and reintroduced the CVE-2026-31900 phantom finding. Extract it into strip_spacy_scanner_fixture() and call it before the PySpark exit branches so it runs on every path. - Restore the best-effort delete: `find ... -delete` under `set -euo pipefail` aborted the build on any undeletable file (read-only layer). Back to `-delete 2>/dev/null || true` so the fixture strip stays non-fatal. ]

…ils closed instead of skipping

python:3.10-slim-bookworm is EOL and no longer rebuilt, so its Debian layer and bundled Python tooling are frozen. Move all four ingestion images to python:3.12-slim-bookworm / apache/airflow:3.2.2-python3.12. - ingestion/Dockerfile pinned the Airflow *Python 3.10* constraints file while building on a python3.12 base; point it at constraints-3.12.txt. - Regenerate the vendored airflow-constraints-3.2.2.txt from upstream's 3.12 variant. Both come from the same 2026-05-26 generation run, so this is the matching sibling: it drops the py<3.11 backports (backports.strenum, exceptiongroup, tomli) and refreshes numpy/scipy/scikit-learn. Verified openmetadata-airflow-apis still installs under it and that the new setuptools==82.0.1 pin stays inert against the deliberate setuptools<81. - Upgrade the interpreter-level pip to a pinned 26.2 before dropping privileges. The existing `pip install --upgrade pip` runs after the USER switch, so it only reached ~/.local and left the base image's copy on disk for scanners (CVE-2025-8869, CVE-2026-3219, CVE-2026-6357, CVE-2026-8643, fixed in 26.1.2). Python 3.12 alone is not enough - its bundled pip is 25.0.1. The airflow images need the explicit /usr/local/bin/python: as root, `pip` hits airflow's wrapper and exits 1, and bare `python` resolves to /home/airflow/.local/bin/python, so both would miss the very copy this line exists to fix. - Uninstall datamodel-code-generator and the black it depends on after the codegen step in the two CI images. Neither is imported at runtime; they exist only to run datamodel_generation.py, and account for eight HIGH findings. Bumping the pin is not an option - both codegen scripts monkeypatch datamodel_code_generator.model.pydantic.types.IMPORT_SECRET_STR (the CustomSecretStr swap), which no longer exists in 0.26+. The uninstall runs after the final pip install so no extra can pull them back in. Python 3.12 also ships ensurepip without setuptools/wheel, removing the vendored wheel-0.45.1 behind CVE-2026-24049. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

github-actions · 2026-08-03T14:34:30Z

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

No GitHub issue is linked. Link an issue in the Development section of the PR (or add Fixes #12345 to the description). For a same-org cross-repo issue, add Fixes open-metadata/<repo>#123 to the description.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

gitar-bot · 2026-08-03T14:36:59Z

+rm -f zookeeper-*.jar zookeeper-jute-*.jar \
+      jackson-mapper-asl-*.jar jackson-core-asl-*.jar \
+      netty-codec-http-*.jar
+
+fetch_jar zookeeper-3.7.2.jar \
+  org/apache/zookeeper/zookeeper/3.7.2/zookeeper-3.7.2.jar \
+  b12d6fb4afd7b3849d3a9a5a38b9260c23a12e1ea58ca8c8d775880249cb8eac
+
+fetch_jar zookeeper-jute-3.7.2.jar \
+  org/apache/zookeeper/zookeeper-jute/3.7.2/zookeeper-jute-3.7.2.jar \
+  ad15d812b1f01f373638443adcda0e23fda549d65ced2be8c4f64bef33b5d774
+
+fetch_jar netty-codec-http-4.1.135.Final.jar \
+  io/netty/netty-codec-http/4.1.135.Final/netty-codec-http-4.1.135.Final.jar \
+  4018529d3d6aecf4044b98c75d9a90c91839ddf49c7aa484c5ac81c90a15da02


⚠️ Edge Case: Netty module version skew after codec-http bump

The script deletes only netty-codec-http-*.jar and re-fetches netty-codec-http-4.1.135.Final, while PySpark 3.5.6's sibling Netty jars (netty-common, netty-buffer, netty-handler, netty-transport, netty-codec, netty-resolver) remain at 4.1.96.Final. Netty does not guarantee forward binary compatibility across patch releases within 4.1.x, so a 4.1.135 codec-http calling a method added after 4.1.96 in a sibling module can raise NoSuchMethodError/LinkageError at runtime on the Hive/Deltalake path. The build's SHA-pinned checks won't catch this since it's a runtime linkage issue. Consider bumping all bundled netty-*-4.1.96 jars to 4.1.135.Final together (or the lowest common release ≥4.1.133 that fixes the CVEs) so the module set stays version-consistent.

_{Was this helpful? React with 👍 / 👎}

gitar-bot · 2026-08-03T14:37:00Z

Code Review ⚠️ Changes requested 0 resolved / 1 findings

Migrates ingestion images to Python 3.12 and strips build tooling to address CVE vulnerabilities, but leaves a netty-codec-http module version skew open.

⚠️

Edge Case: Netty module version skew after codec-http bump

📄 ingestion/scripts/patch_pyspark_jars.sh:127-141

The script deletes only netty-codec-http-*.jar and re-fetches netty-codec-http-4.1.135.Final, while PySpark 3.5.6's sibling Netty jars (netty-common, netty-buffer, netty-handler, netty-transport, netty-codec, netty-resolver) remain at 4.1.96.Final. Netty does not guarantee forward binary compatibility across patch releases within 4.1.x, so a 4.1.135 codec-http calling a method added after 4.1.96 in a sibling module can raise NoSuchMethodError/LinkageError at runtime on the Hive/Deltalake path. The build's SHA-pinned checks won't catch this since it's a runtime linkage issue. Consider bumping all bundled netty-*-4.1.96 jars to 4.1.135.Final together (or the lowest common release ≥4.1.133 that fixes the CVEs) so the module set stays version-consistent.

🤖 Prompt for agents

Code Review: Migrates ingestion images to Python 3.12 and strips build tooling to address CVE vulnerabilities, but leaves a netty-codec-http module version skew open.

1. ⚠️ Edge Case: Netty module version skew after codec-http bump
   Files: ingestion/scripts/patch_pyspark_jars.sh:127-141

   The script deletes only `netty-codec-http-*.jar` and re-fetches `netty-codec-http-4.1.135.Final`, while PySpark 3.5.6's sibling Netty jars (netty-common, netty-buffer, netty-handler, netty-transport, netty-codec, netty-resolver) remain at 4.1.96.Final. Netty does not guarantee forward binary compatibility across patch releases within 4.1.x, so a 4.1.135 codec-http calling a method added after 4.1.96 in a sibling module can raise NoSuchMethodError/LinkageError at runtime on the Hive/Deltalake path. The build's SHA-pinned checks won't catch this since it's a runtime linkage issue. Consider bumping all bundled netty-*-4.1.96 jars to 4.1.135.Final together (or the lowest common release ≥4.1.133 that fixes the CVEs) so the module set stays version-consistent.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

`Compact`
`gitar display:verbose`

_{Was this helpful? React with 👍 / 👎 | Gitar | Powered by Gitar — free for open source}

Copilot

Pull request overview

This PR upgrades the ingestion Docker images to Python 3.12 bases (including Airflow 3.2.2 python3.12), refreshes interpreter-level pip early to reduce scanner findings, consolidates PySpark jar/CVE remediation into a shared script, and updates the Airflow constraints to match the Python 3.12 variant.

Changes:

Bump ingestion image bases to python:3.12-slim-bookworm and apache/airflow:3.2.2-python3.12, and align the Airflow constraints URL to constraints-3.12.txt.
Pin/refresh interpreter-level pip to 26.2 before dropping privileges, and run PySpark/spaCy CVE cleanup via ingestion/scripts/patch_pyspark_jars.sh.
Regenerate ingestion/airflow-constraints-3.2.2.txt and uninstall build-only codegen tooling (datamodel-code-generator, black) in CI images.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
ingestion/scripts/patch_pyspark_jars.sh	New shared script to patch PySpark-bundled vulnerable jars and remove spaCy scanner fixture deterministically.
ingestion/operators/docker/Dockerfile.ci	Slim CI image moved to Python 3.12, pins interpreter pip, runs jar/fixture patch, and uninstalls codegen tooling.
ingestion/operators/docker/Dockerfile	Slim runtime image moved to Python 3.12, pins interpreter pip, and runs jar/fixture patch.
ingestion/Dockerfile.ci	Airflow CI image moved to Python 3.12, pins interpreter pip via explicit interpreter path, runs jar/fixture patch, and uninstalls codegen tooling.
ingestion/Dockerfile	Airflow runtime image moved to Python 3.12, fixes constraints URL to `constraints-3.12.txt`, pins interpreter pip, and runs jar/fixture patch.
ingestion/airflow-constraints-3.2.2.txt	Updated to the regenerated Python 3.12-compatible constraints set for Airflow 3.2.2.

+# datamodel-code-generator (and the black it depends on) exist only to run the codegen
+# step above -- nothing imports them at runtime. Drop them so the shipped image stops
+# reporting their CVEs. Runs after the final pip install so an extra cannot pull them
+# back in. Bumping the pin instead is not an option: the codegen script monkeypatches
+# datamodel_code_generator.model.pydantic.types.IMPORT_SECRET_STR, gone in 0.26+.
+RUN pip uninstall -y datamodel-code-generator black


github-actions · 2026-08-03T15:25:29Z

✅ Playwright Results — workflow succeeded

Validated commit b387340bffd40c2b442d81d542e75f377ae625b5 in Playwright run 30823339366, attempt 1.

✅ 105 passed · ❌ 0 failed · 🟡 0 flaky · ⏭️ 3 skipped · 🧰 0 lifecycle flaky

Performance

Blocking targets: ✅ met · Optimization targets: 🟡 in progress

Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting.

🕒 Full workflow signal wall (to summary) 49m 20s

⏱️ Max setup 3m 13s · max shard execution 11m 22s · max shard-job elapsed before upload 19m 31s · reporting 4s

🌐 208.55 requests/attempt · 1.78 app boots/UI scenario · 0.00% common-shard skew

Optimization targets still in progress:

Browser traffic was 208.55 requests per attempt (convergence target: fewer than 200).
Application boot ratio was 1.78 per UI scenario (206 boots / 116 scenarios; convergence target: at most 1).

Shard	Passed	Skipped
✅ Shard chromium-01	46	3
✅ Shard ingestion-01	30	0
✅ Shard ingestion-02	29	0

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

harshsoni2024 and others added 8 commits July 31, 2026 14:41

Merge branch 'main' into fix/patch_pyspark_jars_ingestion_slim

933a30e

fix(ingestion): use find_spec so a broken pyspark submodule import fa…

2758bd0

…ils closed instead of skipping

fix: spacy cleanup

0668f63

Merge branch 'main' into fix/patch_pyspark_jars_ingestion_slim

2a0331e

harshsoni2024 requested a review from a team as a code owner August 3, 2026 14:34

Copilot AI review requested due to automatic review settings August 3, 2026 14:34

github-actions Bot added Ingestion safe to test Add this label to run secure Github workflows on PRs labels Aug 3, 2026

Copilot started reviewing on behalf of harshsoni2024 August 3, 2026 14:34 View session

gitar-bot Bot reviewed Aug 3, 2026

View reviewed changes

Copilot AI reviewed Aug 3, 2026

View reviewed changes

harshsoni2024 temporarily deployed to test August 3, 2026 15:01 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(upgrade): move ingestion images to Python 3.12 and strip build-only tooling - #30878

fix(upgrade): move ingestion images to Python 3.12 and strip build-only tooling#30878
harshsoni2024 wants to merge 8 commits into
mainfrom
fix/bump_ingestion_img_python_3_12

harshsoni2024 commented Aug 3, 2026

Uh oh!

github-actions Bot commented Aug 3, 2026

Uh oh!

gitar-bot Bot Aug 3, 2026

Uh oh!

gitar-bot Bot commented Aug 3, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

harshsoni2024 commented Aug 3, 2026

Describe your changes:

Type of change:

High-level design:

Tests:

Use cases covered

Unit tests

Backend integration tests

Ingestion integration tests

Playwright (UI) tests

Manual testing performed

Uh oh!

github-actions Bot commented Aug 3, 2026

❌ PR checklist incomplete

Uh oh!

gitar-bot Bot Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

gitar-bot Bot commented Aug 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

github-actions Bot commented Aug 3, 2026

✅ Playwright Results — workflow succeeded

Performance

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants