Skip to content

GH-48028: [Python][Packaging] Update quay.io base manylinux and musllinux images for Python wheels and remove cp313t #50082

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
raulcd wants to merge 9 commits into
base: main
Choose a base branch
from
+31 −149
Draft

GH-48028: [Python][Packaging] Update quay.io base manylinux and musllinux images for Python wheels and remove cp313t #50082

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
d15a603
GH-48028: [Python][Packaging] Update quay.io base manylinux and musll…
raulcd Jun 2, 2026
473ca7a
Try using mold linker instead of ld shipped with musllinux
raulcd Jun 3, 2026
bd3d2e5
Make mold unconditional on musllinux
raulcd Jun 3, 2026
5da462a
Drop Python 3.13 free threaded builds
raulcd Jun 3, 2026
df5d426
Remove unnecessary requirements specific for 3.13t
raulcd Jun 3, 2026
e1a4895
Push missing file save
raulcd Jun 3, 2026
a02798b
Remove now unused is_free_threaded function
raulcd Jun 3, 2026
fb256c0
Remove now misleading comment and update windows-wheel build image re…
raulcd Jun 3, 2026
386cf7d
Udate build_date for Python 3.14.5
raulcd Jun 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,5 +98,5 @@ VCPKG="9b965a116838c6cdcd36bca60d1b81b030c8ab8d" # 2026.05.27 (not release, u
# ci/docker/python-*-windows-*.dockerfile or the vcpkg config.
# This is a workaround for our CI problem that "archery docker build" doesn't
# use pulled built images in dev/tasks/python-wheels/github.windows.yml.
PYTHON_WHEEL_WINDOWS_IMAGE_REVISION=2026-05-27
PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION=2026-05-27
PYTHON_WHEEL_WINDOWS_IMAGE_REVISION=2026-06-03
PYTHON_WHEEL_WINDOWS_TEST_IMAGE_REVISION=2026-06-03
59 changes: 0 additions & 59 deletions ci/docker/linux-apt-python-313-freethreading.dockerfile
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion ci/docker/python-free-threaded-wheel-manylinux-test-imports.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
ARG base
FROM ${base}

ARG python_version=3.13
ARG python_version=3.14

ENV DEBIAN_FRONTEND=noninteractive

Expand Down
2 changes: 1 addition & 1 deletion ci/docker/python-free-threaded-wheel-manylinux-test-unittests.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
ARG base
FROM ${base}

ARG python_version=3.13
ARG python_version=3.14

ENV DEBIAN_FRONTEND=noninteractive

Expand Down
5 changes: 2 additions & 3 deletions ci/docker/python-free-threaded-wheel-musllinux-test-imports.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
ARG base
FROM ${base}

ARG python_version=3.13
ARG python_version=3.14
ARG arch=aarch64
ARG build_date

Expand All @@ -35,8 +35,7 @@ RUN apk update && \
# See available releases at: https://github.com/astral-sh/python-build-standalone/releases
RUN set -e; \
case "${python_version}" in \
3.13) python_patch_version="3.13.9";; \
3.14) python_patch_version="3.14.0";; \
3.14) python_patch_version="3.14.5";; \
esac && \
Comment thread
raulcd marked this conversation as resolved.
Comment thread
raulcd marked this conversation as resolved.
Comment thread
raulcd marked this conversation as resolved.
curl -L -o python.tar.zst \
https://github.com/astral-sh/python-build-standalone/releases/download/${build_date}/cpython-${python_patch_version}+${build_date}-${arch}-unknown-linux-musl-freethreaded+lto-full.tar.zst && \
Expand Down
5 changes: 2 additions & 3 deletions ci/docker/python-free-threaded-wheel-musllinux-test-unittests.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
ARG base
FROM ${base}

ARG python_version=3.13
ARG python_version=3.14
ARG arch=aarch64
ARG build_date

Expand All @@ -37,8 +37,7 @@ RUN apk update && \
# See available releases at: https://github.com/astral-sh/python-build-standalone/releases
RUN set -e; \
case "${python_version}" in \
3.13) python_patch_version="3.13.9";; \
3.14) python_patch_version="3.14.0";; \
3.14) python_patch_version="3.14.5";; \
esac && \
Comment thread
raulcd marked this conversation as resolved.
Comment thread
raulcd marked this conversation as resolved.
Comment thread
raulcd marked this conversation as resolved.
curl -L -o python.tar.zst \
https://github.com/astral-sh/python-build-standalone/releases/download/${build_date}/cpython-${python_patch_version}+${build_date}-${arch}-unknown-linux-musl-freethreaded+lto-full.tar.zst && \
Expand Down
18 changes: 6 additions & 12 deletions ci/docker/python-free-threaded-wheel-windows-test-vs2022.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,10 @@ ARG base
# hadolint ignore=DL3006
FROM ${base}

ARG python=3.13
ARG python=3.14

# hadolint ignore=SC1072
RUN (if "%python%"=="3.13" setx PYTHON_VERSION "3.13.1") & \
(if "%python%"=="3.14" setx PYTHON_VERSION "3.14.0")
RUN (if "%python%"=="3.14" setx PYTHON_VERSION "3.14.5")
Comment thread
raulcd marked this conversation as resolved.

Comment thread
raulcd marked this conversation as resolved.
Comment thread
raulcd marked this conversation as resolved.
SHELL ["powershell", "-NoProfile", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
RUN $version = $env:PYTHON_VERSION; \
Expand All @@ -40,20 +39,15 @@ RUN $version = $env:PYTHON_VERSION; \
ENV PYTHON_CMD="py -${python}t"

SHELL ["cmd", "/S", "/C"]
RUN %PYTHON_CMD% -m pip install -U pip setuptools & \
if "%python%"=="3.13" ( \
setx REQUIREMENTS_FILE "requirements-wheel-test-3.13t.txt" \
) else ( \
setx REQUIREMENTS_FILE "requirements-wheel-test.txt" \
)

COPY python/requirements-wheel-test-3.13t.txt python/requirements-wheel-test.txt C:/arrow/python/
RUN %PYTHON_CMD% -m pip install -U pip setuptools

COPY python/requirements-wheel-test.txt C:/arrow/python/
# Cython and Pandas wheels for free-threaded are not released yet
RUN %PYTHON_CMD% -m pip install \
--extra-index-url https://pypi.anaconda.org/scientific-python-nightly-wheels/simple \
--pre \
--prefer-binary \
-r C:/arrow/python/%REQUIREMENTS_FILE%
-r C:/arrow/python/requirements-wheel-test.txt

ENV PYTHON="${python}t"
ENV PYTHON_GIL=0
8 changes: 8 additions & 0 deletions ci/docker/python-wheel-musllinux.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,14 @@ RUN apk add --no-cache \
# We will be able to use the main repo once we move to alpine 3.22 or later.
RUN apk add --no-cache --repository=https://dl-cdn.alpinelinux.org/alpine/edge/community mono

# The linker shipped in Alpine (ld 2.44) causes issue with the generated wheel
# on x86_64 which makes "import pyarrow" abort at startup with:
# Invalid file descriptor data passed to EncodedDescriptorDatabase::Add()
# See: GH-48028
# Link with mold to work around it.
ENV ARROW_USE_MOLD=ON
RUN apk add --no-cache mold

# A system Python is required for ninja and vcpkg in this Dockerfile.
# On musllinux_1_2 a system python is installed (3.12) but pip is not
# We therefore override the PATH with Python 3.10 in /opt/python
Expand Down
5 changes: 2 additions & 3 deletions ci/scripts/install_python.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@ versions=([3.10]=3.10.11
[3.11]=3.11.9
[3.12]=3.12.10
[3.13]=3.13.9
[3.13t]=3.13.9
[3.14]=3.14.0
[3.14t]=3.14.0)

Expand All @@ -52,7 +51,7 @@ if [ "$platform" = "macOS" ]; then
wget "https://www.python.org/ftp/python/${full_version}/${fname}"

echo "Installing Python..."
if [[ $2 == "3.13t" ]] || [[ $2 == "3.14t" ]]; then
if [[ $2 == "3.14t" ]]; then
# Extract the base version without 't' suffix
base_version="${version%t}"
# See https://github.com/python/cpython/issues/120098#issuecomment-2151122033 for more info on this.
Expand Down Expand Up @@ -80,7 +79,7 @@ EOF
rm "$fname"

python="/Library/Frameworks/Python.framework/Versions/${version}/bin/python${version}"
if [[ $2 == "3.13t" ]] || [[ $2 == "3.14t" ]]; then
if [[ $2 == "3.14t" ]]; then
base_version="${version%t}"
python="/Library/Frameworks/PythonT.framework/Versions/${base_version}/bin/python${base_version}t"
fi
Expand Down
12 changes: 1 addition & 11 deletions ci/scripts/python_wheel_unix_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,19 +100,9 @@ if [ "${CHECK_WHEEL_CONTENT}" == "ON" ]; then
--path "${source_dir}/python/repaired_wheels"
fi

is_free_threaded() {
python -c "import sysconfig; print('ON' if sysconfig.get_config_var('Py_GIL_DISABLED') else 'OFF')"
}

if [ "${CHECK_UNITTESTS}" == "ON" ]; then
# Install testing dependencies
if [ "$(is_free_threaded)" = "ON" ] && [[ "${PYTHON:-}" == *"3.13"* ]]; then
echo "Free-threaded Python 3.13 build detected"
python -m pip install -U -r "${source_dir}/python/requirements-wheel-test-3.13t.txt"
else
echo "Regular Python build detected"
python -m pip install -U -r "${source_dir}/python/requirements-wheel-test.txt"
fi
python -m pip install -U -r "${source_dir}/python/requirements-wheel-test.txt"
Comment thread
raulcd marked this conversation as resolved.

# Execute unittest, test dependencies must be installed
python -c 'import pyarrow; pyarrow.create_library_symlinks()'
Expand Down
2 changes: 2 additions & 0 deletions ci/scripts/python_wheel_xlinux_build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ echo "=== (${PYTHON_VERSION}) Building Arrow C++ libraries ==="
: ${ARROW_SUBSTRAIT:=ON}
: ${ARROW_S3:=ON}
: ${ARROW_TENSORFLOW:=ON}
: ${ARROW_USE_MOLD:=OFF}
: ${ARROW_WITH_BROTLI:=ON}
: ${ARROW_WITH_BZ2:=ON}
: ${ARROW_WITH_LZ4:=ON}
Expand Down Expand Up @@ -128,6 +129,7 @@ cmake \
-DARROW_SUBSTRAIT=${ARROW_SUBSTRAIT} \
-DARROW_TENSORFLOW=${ARROW_TENSORFLOW} \
-DARROW_USE_CCACHE=ON \
-DARROW_USE_MOLD=${ARROW_USE_MOLD} \
-DARROW_WITH_BROTLI=${ARROW_WITH_BROTLI} \
-DARROW_WITH_BZ2=${ARROW_WITH_BZ2} \
-DARROW_WITH_LZ4=${ARROW_WITH_LZ4} \
Expand Down
35 changes: 4 additions & 31 deletions compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,6 @@ x-hierarchy:
- ubuntu-ruby
- ubuntu-python
- ubuntu-python-sdist-test
- ubuntu-python-313-freethreading
- ubuntu-r
- ubuntu-r-only-r
- ubuntu-cpp-bundled
Expand Down Expand Up @@ -1132,32 +1131,6 @@ services:
/arrow/ci/scripts/cpp_build.sh /arrow /build &&
/arrow/ci/scripts/python_sdist_test.sh /arrow"

############################ Python free-threading ##########################

ubuntu-python-313-freethreading:
# Usage:
# docker compose build ubuntu-cpp
# docker compose build ubuntu-python-313-freethreading
# docker compose run --rm ubuntu-python-313-freethreading
# Parameters:
# ARCH: amd64, arm64v8, ...
# UBUNTU: 22.04, 24.04
image: ${REPO}:${ARCH}-ubuntu-${UBUNTU}-python-313-freethreading
build:
context: .
dockerfile: ci/docker/linux-apt-python-313-freethreading.dockerfile
cache_from:
- ${REPO}:${ARCH}-ubuntu-${UBUNTU}-python-313-freethreading
args:
base: ${REPO}:${ARCH}-ubuntu-${UBUNTU}-cpp
shm_size: *shm-size
environment:
<<: [*common, *ccache]
# Bundled build of OpenTelemetry needs a git client
ARROW_WITH_OPENTELEMETRY: "OFF"
volumes: *ubuntu-volumes
command: *python-command

############################ Python wheels ##################################

# See available versions at:
Expand All @@ -1168,7 +1141,7 @@ services:
args:
arch: ${ARCH}
arch_short: ${ARCH_SHORT}
base: quay.io/pypa/manylinux_2_28_${ARCH_ALIAS}:2025-06-04-496f7e1
base: quay.io/pypa/manylinux_2_28_${ARCH_ALIAS}:2026.05.31-3
Comment thread
raulcd marked this conversation as resolved.
manylinux: 2_28
python: ${PYTHON}
python_abi_tag: ${PYTHON_ABI_TAG}
Expand All @@ -1193,7 +1166,7 @@ services:
args:
arch: ${ARCH}
arch_short: ${ARCH_SHORT}
base: quay.io/pypa/musllinux_1_2_${ARCH_ALIAS}:2025-06-04-496f7e1
base: quay.io/pypa/musllinux_1_2_${ARCH_ALIAS}:2026.05.31-3
musllinux: 1_2
python: ${PYTHON}
python_abi_tag: ${PYTHON_ABI_TAG}
Expand Down Expand Up @@ -1234,7 +1207,7 @@ services:
args:
base: "${ARCH}/alpine:${ALPINE_LINUX}"
python_version: ${PYTHON}
build_date: "20251014" # python-build-standalone release date
build_date: "20260510" # python-build-standalone release date
arch: ${ARCH_ALIAS}
context: .
dockerfile: ci/docker/python-free-threaded-wheel-musllinux-test-imports.dockerfile
Expand Down Expand Up @@ -1278,7 +1251,7 @@ services:
args:
base: "${ARCH}/alpine:${ALPINE_LINUX}"
python_version: ${PYTHON}
build_date: "20251014" # python-build-standalone release date
build_date: "20260510" # python-build-standalone release date
arch: ${ARCH_ALIAS}
context: .
dockerfile: ci/docker/python-free-threaded-wheel-musllinux-test-unittests.dockerfile
Expand Down
9 changes: 0 additions & 9 deletions dev/tasks/tasks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,6 @@ tasks:
("3.11", "cp311", "cp311"),
("3.12", "cp312", "cp312"),
("3.13", "cp313", "cp313"),
("3.13", "cp313", "cp313t"),
("3.14", "cp314", "cp314"),
("3.14", "cp314", "cp314t")] %}

Expand Down Expand Up @@ -538,14 +537,6 @@ tasks:
PYTEST_ARGS: "-m hypothesis -W error"
image: conda-python-pandas

test-ubuntu-22.04-python-313-freethreading:
ci: github
template: docker-tests/github.linux.yml
params:
env:
UBUNTU: 22.04
image: ubuntu-python-313-freethreading

{% for debian_version in ["13"] %}
test-debian-{{ debian_version }}-python-3-amd64:
ci: github
Expand Down
5 changes: 0 additions & 5 deletions python/requirements-test-3.13t.txt
View file
Open in desktop

This file was deleted.

9 changes: 0 additions & 9 deletions python/requirements-wheel-test-3.13t.txt
View file
Open in desktop

This file was deleted.

Loading