fix(nvvm): keep address-space-qualified pointers out of the generic space

[nh13]

Refs #402.

The defect

Every access to an #[address_space(shared)] (or constant) static is emitted as a generic access behind an addrspacecast. load, volatile_load, gep, inbounds_gep and check_store all cast their pointer operand through type_ptr_to, which always yields an addrspace 0 pointer, and bitcast/pointercast insert an addrspacecast (3 -> 0) to make that type-check.

Correctness then rests entirely on libNVVM's InferAddressSpaces pass folding the cast back into the access. It usually does -- which is why static shared memory mostly works today, and why a small reproducer does not reveal anything. When the fold misses, the access is emitted as a generic ld/st against the raw shared window offset. That is an out-of-bounds access at runtime, with no error or warning at compile time.

That is exactly the failure in #402: 106 of 108 accesses in the reporting kernel became ld.shared/st.shared, and 2 -- both to the same symbol, both inside a data-dependent branch -- stayed generic, so compute-sanitizer reported an invalid 4-byte write to 0x26c, the raw shared offset of that symbol's slot.

Here is the emitted NVVM IR for the kernel added in this PR, before this change:

%22 = getelementptr inbounds i32,
        i32* addrspacecast (i32 addrspace(3)* getelementptr inbounds (..., @SH, ...) to i32*), i64 %18
store i32 %1, i32* %22, align 4
%23 = getelementptr inbounds i32,
        i32* addrspacecast (i32 addrspace(3)* getelementptr inbounds (..., @SH, ...) to i32*), i64 %18
%24 = load i32, i32* %23, align 4

Both the GEP and the access are in the generic space, and the only thing standing between that and a bad access is a libNVVM optimization pass.

The fix

Cast the pointer operand in the address space it already points into, via a new pointercast_preserving_addrspace helper. Same kernel, after:

%22 = getelementptr inbounds i32, i32 addrspace(3)* getelementptr inbounds (..., @SH, ...), i64 %18
store i32 %1, i32 addrspace(3)* %22, align 4
%23 = getelementptr inbounds i32, i32 addrspace(3)* getelementptr inbounds (..., @SH, ...), i64 %18
%24 = load i32, i32 addrspace(3)* %23, align 4

The access is now shared by construction rather than by inference.

raw_eq's integer-compare fast path (intrinsic.rs) had the same defect -- it bitcast both operands to a generic pointer before loading -- and is fixed with it.

Casts that are semantically required are deliberately left alone: a pointer passed to a callee whose parameter type is a generic pointer, and the memcmp branch of raw_eq, still go through bitcast, which addrspacecasts to generic exactly as before. Those are the cases where a generic pointer is genuinely what the callee expects.

On testing, and what I could not show

I was not able to construct a small kernel whose PTX differs before and after. I tried a lane-0 store plus post-__syncwarp broadcast read, a select between two shared statics (a shape InferAddressSpaces handles poorly), and forcing -Copt-level=0; in every case libNVVM recovered the shared space both before and after, so the PTX is identical. This matches the reporter's own experience in #402 -- their synthetic reproducer did not trigger it either, and the trigger appears to depend on surrounding context such as function size, inlining, or register pressure.

So this PR should be judged on the IR shape, which is verifiable and shown above, rather than on a PTX-level regression test. The included dis test pins the shared-memory codegen shape -- st.shared/ld.shared on the raw symbol offset, no cvta.shared -- which guards against regression but would not have caught the original bug.

Verification

In ghcr.io/rust-gpu/rust-cuda-ubuntu24-cuda12:

• Full compiletest suite green across compute_61,compute_75,compute_90 (64 passed, 3 ignored).
• cargo fmt --all -- --check and rustfmt --check tests/compiletests/ui/**/*.rs clean.
• cargo clippy -p rustc_codegen_nvvm clean with RUSTFLAGS=-Dwarnings.
• Atomics on a shared static were a specific concern, since atomic_op carried a FIXME stating that it assumes all pointers are generic. Emitted PTX for fetch_add/load on a shared AtomicU32 is byte-identical before and after: the llvm.nvvm.isspacep.* predicates declare generic parameters, so check_call casts for those calls while the atomic itself stays on the shared pointer, giving atom.shared.add.u32. The FIXME is updated to say what is actually left to do -- skip the runtime predicate dance when the address space is statically known.

🤖 Generated with Claude Code