fix(pii): install CUDA torch on amd64 so GLiNER can run on GPU

[TheodoreSpeaks]

Problem

The published sim/pii image installs a CPU-only torch build. On the ECS GPU fleet (g4dn.xlarge / NVIDIA T4), running with PII_ENGINE=gliner + PII_DEVICE=cuda crashes at model load:

RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False.

The GPU is genuinely attached — the nvidia container runtime injects it. The image's torch simply has no CUDA compiled in.

docker/pii.Dockerfile already had an ARG TORCH_INDEX_URL defaulting to .../whl/cpu, with a comment saying the GPU variant just needs --build-arg. But no CI job ever passes a build arg, so every published image silently took the CPU default.

Fix

Select the wheel index from Docker's built-in TARGETARCH:

• amd64 → cu128 (GPU-capable; also runs on CPU)
• arm64 → cpu

A CUDA torch falls back to CPU when no GPU is present, so one image keeps serving both the Fargate CPU tasks and the EC2 GPU tasks from the same sim/pii:staging tag. No ci.yml change, no CDK change — rebuild and push, then re-enable the infra flag.

Also adds a build-time assert so an amd64 image can never silently regress to a +cpu wheel again — that failure currently only surfaces at GLiNER load time on a GPU host.

Why cu128, and why arm64 is different

• cu121 was never viable. That index stops at torch 2.5.1; using it would downgrade the 2.11.0 pin by six minor versions.
• cu128 keeps Turing. Per pytorch v2.11.0 .ci/manywheel/build_cuda.sh, the 12.8) case yields 7.5;8.0;8.6;9.0;10.0;12.0 — 7.5 is the T4.
• Driver floor is met. CUDA 12.x minor-version compatibility needs driver >=525. amzn2-ami-ecs-gpu-hvm-2.0.20260707-x86_64-ebs installs nvidia-driver-latest-dkms (>=535).
• arm64 must stay CPU. cu128 publishes no aarch64 wheel at 2.11.0 (it stops at 2.9.1), and the build-ghcr-arm64 job builds this same Dockerfile for linux/arm64. A blanket CUDA default would break that job.
• No CUDA base image needed. The pip CUDA wheels bundle their own runtime (cudnn, cublas, cuda_runtime, …); the host AMI supplies the driver. Base stays python:3.12-slim-bookworm.

Verification

Built the changed block for linux/amd64 and ran it:

VER   2.11.0+cu128
CUDA  12.8
AVAIL False          <- expected: no GPU on the build box
ARCH  sm_75 sm_80 sm_86 sm_90 sm_100 sm_120

torch.version.cuda is set (a CUDA build, not +cpu) and sm_75 is present, which is the root cause addressed. CUDA runtime libs ship in the image (nvidia/{cublas,cudnn,cuda_runtime,nccl,...}, libtorch_cuda.so ~912 MB).

linux/arm64 build resolves to 2.11.0+cpu / cuda=None, and the assert passes on both arches.

Note: torch.cuda.get_arch_list() early-returns [] with no GPU present — torch._C._cuda_getArchFlags() is the correct accessor on a CPU box.

Cost

torch wheel 181 MB → 782 MB, plus the nvidia-* runtime packages: roughly +3-4 GB installed. The Fargate CPU tasks pull that too. Accepted in exchange for one image, one tag, and zero infra changes.

Not verified here

The gated CPU-path suite (RUN_PII_INTEGRATION=1 PII_DEVICE=cpu python -m pytest tests) needs a full ~13 GB image build, which is emulated on an arm64 dev box. Device plumbing is untouched (server.py:34 maps unset PII_DEVICE to None → auto-detect → cpu), and torch.cuda.is_available() is False without a GPU, so the CPU path behaves exactly as before.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QHNEWVrh7k89m8Wtqzhs18

[TheodoreSpeaks]

@greptile review

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
docs	Ready	Preview, Comment	Jul 10, 2026 12:51am

[cursor]

PR Summary

Medium Risk
Changes only the container build (torch variant and size); runtime app code is untouched, but a wrong wheel or failed assert would break amd64/arm64 PII image builds and GPU GLiNER load paths.

Overview
Fixes CPU-only torch in published sim/pii images, which caused PII_ENGINE=gliner + PII_DEVICE=cuda to fail on ECS GPU hosts even when a GPU was attached.

docker/pii.Dockerfile selects the PyTorch wheel index from Docker TARGETARCH: amd64 → cu128, arm64 → cpu. CUDA torch still runs on CPU-only Fargate tasks, so one tag serves both CPU and GPU fleets without CI/CD changes. Comments document cu128 vs cu121, T4 (sm_75), driver floor, and why arm64 stays CPU.

After all pip install steps, a build-time assert checks that amd64 images have torch.version.cuda set (guarding against later deps downgrading to +cpu).

Tradeoff: larger amd64 image (~+3–4 GB installed) in exchange for a single image workflow.

^{Reviewed by Cursor Bugbot for commit d088912. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Greptile Summary

This PR updates the PII Docker image so GLiNER can use CUDA torch on amd64 builds. The main changes are:

• Selects the torch wheel index from the Docker target architecture.
• Installs CUDA torch for amd64 and CPU torch for arm64.
• Adds a build-time check that the final installed torch wheel matches the target architecture.

Confidence Score: 5/5

This looks safe to merge.

• No blocking issues found in the changed code.
• The torch validation now runs after the dependency installs that could affect the selected wheel.

Important Files Changed

Filename	Overview
docker/pii.Dockerfile	Updates torch wheel selection by architecture and validates the final installed torch build after dependency installation.

_{Reviews (4): Last reviewed commit: "fix(pii): assert torch CUDA state after ..." | Re-trigger Greptile}

[greptile-apps]

Empty Target Architecture Fails Build

When this image is built with the documented plain docker build -f docker/pii.Dockerfile -t sim-pii . path and Docker does not inject BuildKit platform args, TARGETARCH is empty. The new case then takes the unsupported branch and the PII image build fails before installing torch, whereas the previous Dockerfile had a working CPU-wheel default.

[greptile-apps]

Torch Index Override Is Ignored

The old TORCH_INDEX_URL build arg was the documented control point for selecting the torch wheel index. After replacing it with TORCH_CUDA_INDEX_URL and TORCH_CPU_INDEX_URL, any existing build wrapper that still passes --build-arg TORCH_INDEX_URL=... silently downloads from the hard-coded upstream PyTorch index instead of the requested mirror or CPU-only index.

[greptile-apps]

Greptile Summary

This PR updates the PII image build and spreadsheet dependency handling. The main changes are:

• Select CUDA PyTorch for amd64 and CPU PyTorch for arm64.
• Add a build-time check for the expected torch CUDA state.
• Replace the SheetJS CDN tarball with the @e965/xlsx npm mirror.
• Update the lockfile for the new xlsx package source.

Confidence Score: 4/5

The torch build check should run after all Python dependencies are installed.

• The amd64 image can pass the CUDA check before a later dependency install has a chance to change torch.
• Local Docker builds can fail when the builder does not provide TARGETARCH.
• The xlsx package alias keeps the existing bare import shape.

docker/pii.Dockerfile

Important Files Changed

Filename	Overview
docker/pii.Dockerfile	Adds architecture-based torch wheel selection and a CUDA check, but the check runs before later dependencies can affect the final torch install.
apps/sim/package.json	Switches xlsx from the SheetJS CDN tarball to the @e965/xlsx npm alias at the same version.
bun.lock	Updates the locked xlsx resolution, bin metadata, and integrity entry for the npm alias.

Comments Outside Diff (1)

1. docker/pii.Dockerfile, line 72-79 (link)

   Final Torch Wheel Unchecked

   The CUDA assertion runs before requirements-gliner.txt is installed, but that later install brings in gliner, which declares a torch dependency and resolves from the default index. If pip ever replaces or downgrades the preinstalled CUDA wheel during that step, the build still passes this assertion and the published amd64 image can regress to the same GPU runtime crash this change is meant to prevent.

_{Reviews (2): Last reviewed commit: "fix(pii): install CUDA torch on amd64 so..." | Re-trigger Greptile}

[greptile-apps]

Empty Target Architecture Fails

The selector now requires Docker to inject TARGETARCH, but the documented local build path for this image is plain docker build -f docker/pii.Dockerfile -t sim-pii .. On builders that do not provide automatic platform args, TARGETARCH is empty, so supported local amd64 builds hit the fallback arm and fail with unsupported TARGETARCH:.

Suggested change

	case "${TARGETARCH}" in \
	amd64) torch_index="${TORCH_CUDA_INDEX_URL}" ;; \
	arm64) torch_index="${TORCH_CPU_INDEX_URL}" ;; \
	*) echo "unsupported TARGETARCH: ${TARGETARCH}" >&2; exit 1 ;; \
	esac && \
	pip install torch==${TORCH_VERSION} --index-url "${torch_index}"
	arch="${TARGETARCH:-$(dpkg --print-architecture)}" && \
	case "${arch}" in \
	amd64) torch_index="${TORCH_CUDA_INDEX_URL}" ;; \
	arm64) torch_index="${TORCH_CPU_INDEX_URL}" ;; \
	*) echo "unsupported TARGETARCH: ${arch}" >&2; exit 1 ;; \
	esac && \
	pip install torch==${TORCH_VERSION} --index-url "${torch_index}"

[TheodoreSpeaks]

Thanks — went and checked both against the actual builder behavior. Neither reproduces, and here's the evidence.

P1 — "Empty Target Architecture Fails Build" — not reachable

The premise is that docker build -f docker/pii.Dockerfile -t sim-pii . (documented in apps/sim/lib/guardrails/README.md:35) may run on a builder that doesn't inject TARGETARCH. In practice the only such builder is the legacy one, and this Dockerfile has never been buildable with it — it already contains 8 RUN --mount=type=cache directives on staging, before this PR:

$ git show origin/staging:docker/pii.Dockerfile | grep -c 'mount=type=cache'
8

Forcing the legacy builder fails on the first such RUN, long before the torch step:

$ DOCKER_BUILDKIT=0 docker build -f docker/pii.Dockerfile .
the --mount option requires BuildKit. Refer to https://docs.docker.com/go/buildkit/ ...

Any builder that can build this file is BuildKit, and BuildKit always populates TARGETARCH when the ARG is declared. Confirmed:

# BuildKit (default)
#5 [2/2] RUN echo "TARGETARCH=[arm64]"

# legacy builder
TARGETARCH=[]        <- but it can never get to the torch step anyway

So the ${TARGETARCH:-$(dpkg --print-architecture)} fallback would be dead code. This repo's CLAUDE.md is explicit about failing fast rather than adding fallback paths, so the *) arm stays as the loud failure.

One thing the suggestion would have missed even if the premise held: it only rewrites the pip install RUN. The assert on the next line also interpolates ${TARGETARCH}, so on a legacy amd64 build it would install cu128 while computing want=False and then fail the assert. A correct fix would have to thread the resolved arch through both.

P2 — "Torch Index Override Is Ignored" — no such wrapper exists

TORCH_INDEX_URL has zero references anywhere outside this Dockerfile:

$ grep -rn 'TORCH_INDEX_URL' --exclude-dir=.git . | grep -v '^./docker/pii.Dockerfile'
(no matches)

No CI job passed it (all three pii build jobs in ci.yml pass no build-args: at all — that's precisely the bug this PR fixes: the arg existed but nothing ever set it). And BuildKit does not silently ignore an unknown build arg; it warns:

WARNING: one or more build args were not consumed: [TORCH_INDEX_URL]

Reintroducing it as a third override would add a second way to express the same thing, so I'd rather leave the two explicit TORCH_CUDA_INDEX_URL / TORCH_CPU_INDEX_URL args.

---

Also force-pushed to drop an unrelated xlsx/bun.lock commit that this branch had inherited from its base — that's why the earlier review saw apps/sim/package.json. The PR is now one commit touching only docker/pii.Dockerfile.

[TheodoreSpeaks]

@greptile review

[TheodoreSpeaks]

The torch build check should run after all Python dependencies are installed. The amd64 image can pass the CUDA check before a later dependency install has a chance to change torch.

Good catch — acted on this in d088912.

First I checked whether it bites today. requirements-gliner.txt is installed without --index-url, so it resolves against PyPI, but it pins no torch, and torch 2.11.0 already satisfies gliner's constraint — pip leaves it alone:

BEFORE_GLINER 2.11.0+cu128 12.8
AFTER_GLINER  2.11.0+cu128 12.8

The only Installing collected packages: ... torch line in the build log is the cu128 install itself. So no live regression.

But the structural point stands: nothing pins torch in those later requirements files, so a future gliner/transformers bump could pull a different wheel from PyPI and the assert — sitting above them — would never see it. Moved it below the last pip install (after requirements-dev.txt) so it certifies the torch that actually ships rather than the torch that was just installed.

Confirmed ARG TARGETARCH still expands that far down the stage (ARG scope runs to end of stage):

amd64: late_expansion_want= True
arm64: late_expansion_want= False

Trade-off accepted: a wrong wheel index now fails later in the build instead of immediately. That only costs CI time, and correctness of the invariant is worth more than a fast failure.

[TheodoreSpeaks]

@greptile review