[GatewayJ]

Related issues

Depends on rustfs/rustfs-ucx#4. This PR targets main at 78f6c910e68e23fa6c6e430e1c89faeb2a130ff9; the validated head is 3807cc5bb5e6e47091ee6772b0844fbd807f0a48 and pins UCX aba9d32fd2d1da2f7686b9421a3883c512356e38. Local validation compiled this UCX checkout before the final dependency-pin commit; that final commit changes only vendor/rustfs-ucx.

Problem and resulting behavior

Single-part native file uploads read the complete file into memory before calling the transport provider. The updated path passes the opened file through a shared known-length AsyncRead helper, so the provider can consume it through bounded transfer windows.

• Validate exact byte count and EOF before accepting success.
• When a calculated SHA-256 is requested, scan with a 64 KiB buffer, validate source length, and rewind the same file before upload admission.
• Allow Auto-mode HTTP fallback only after NotAccepted without any source polling. Rejection after source polling returns OutcomeUnknown.
• Route the existing bounded multipart part buffer through the same helper. Preserve conditional writes, persisted-checksum validation, and HTTP request-body behavior.
• Pin the bounded native completion driver, cancellation-safe access-end cleanup, and reclamation of Reliable Data/Grant windows whose publication never committed to the journal.

Validation

• cargo +1.99.0 fmt --all --check and workspace Clippy with --locked --offline -- -D warnings passed.
• Workspace tests: 1,993 passed, 0 failed, 2 existing ignored. Native consumer tests: 30 passed, 0 failed. Regression coverage includes conditional uploads, unconsumed-source fallback, partial-consumption rejection, bounded checksum calculation/rewind, and changed source length.
• Locked, offline release builds of the native CLI and comparison harness passed. Dependency lock files remain identical to the baseline setup. Native rc SHA-256: 12982afd2b5610d54a0d956c20b252ded719d5b7c1eb1af0698e761c1ec075ef; harness SHA-256: 0d0c0fb3b41938f481bda97cbcec70372ae2cd7b5db00331d09dd6ed194960c6.
• The comparison harness uses a Vec workload. Actual CLI file-upload memory was measured separately with the native rc binary above.

Actual rc cp uploaded and downloaded the same 64 MiB ordinary file using strict S3 over RDMA, verified HTTPS, and one attempt. Both variants passed exact length and full SHA-256 verification. Upload RSS sampled every 200 ms decreased from 167.38 MiB to 105.00 MiB, a 62.38 MiB reduction (37.27%); each upload had five samples. Observed VmHWM decreased from 168.08 to 105.77 MiB. These are sampled observations with unobserved intervals before exit, not exact full-lifetime memory peaks. Download RSS was 104.11 → 105.15 MiB. This supports the bounded file-streaming allocation change independently of the Vec stress harness.

Both smoke runs completed without CLI shutdown diagnostics; server resource samples reached zero after shutdown. The CLI has no structured successful shutdown report, so physical CLI closure remains unconfirmed by that interface. Command duration includes initialization, hashing, and shutdown and is not used as sustained throughput.

Local S3-over-RDMA comparison

The end-to-end comparison used enterprise 636c55d, CLI 3807cc5, and UCX runtime aba9d32, with the original baseline binaries and unchanged native libraries. Four servers and four strict-RDMA clients shared one i9-13900H host; UCX used tcp,self over loopback and verified HTTPS control requests. The workload and configuration matched between variants, with 64 × 4 MiB and 32 × 64 MiB objects. All compilation completed before the load. These results measure the combined changes across the three PRs and do not establish physical-RDMA NIC performance or a new HTTP comparison.

Eight 60-second GET pairs improved successful throughput by 4.44%–22.07%, with zero measured request errors, complete before/after content validation, and all client scopes closed. Selected pairs:

Window / internode operations	Object / concurrency	Baseline → candidate MiB/s	Baseline → candidate p95 ms
1 MiB / 32	4 MiB / 16	343.46 → 383.38	233.23 → 204.99
4 MiB / 128	64 MiB / 32	664.92 → 771.66	3294.21 → 3057.21
4 MiB / 128	64 MiB / 64	647.13 → 789.92	6817.64 → 5898.85

Sampled CPU time per successful GiB decreased by 5.67%–24.23% on clients and 5.63%–16.85% on servers, with 97.63%–99.75% CPU sample coverage. At 4 MiB windows / 64 MiB objects / concurrency 64, average S3 native payload increased from 0.984 to 2.707 MiB. Window waiting remains significant; concurrency 32 → 64 improved throughput by only 2.37%. The default 4 MiB baseline also had higher swap-in, so the entire gain cannot be assigned to one code change.

Two 20-second PUT rounds used opposite variant order. All 16 formal cases and content checks passed, and client scopes closed. Throughput varied with storage state and order; the measurements do not support a stable general PUT gain. Internode native payload approximately doubled from 262,160 to 524,320 bytes per request, while server CPU seconds/GiB estimates decreased by 5.15%–25.14%. The 64 MiB/concurrency-8 cases had 62.23%–77.41% host I/O wait. Each server had one local root on the shared NVMe filesystem, so this setup does not measure the behavioral benefit of sharing sync permits between several roots in one process.

With 1 MiB windows and internode limit 128, concurrency 84 completed 786 requests without external errors or observed internal capacity rejections: 797.34 MiB/s, p95 8.61 seconds. Concurrency 88 had zero external errors but internal admission rejections/503s and lower throughput. Concurrency 96 returned 34 external failures. These are bounded 60-second observations on 32 keys, not an absolute concurrency limit.

Fault cases separately preserve request outcomes, recovery, and closure evidence. Three-second pause, 40-second pause, GET/PUT cancellation, client termination, and saturated-cache node restart all passed subsequent full-content reads. Timeout and cancellation cases retain their request errors; the killed client has no complete report. Cancelled PUTs stopped further writes without replay, and identical-content validation cannot determine whether those cancelled writes committed. Some faulted server scopes remained retained, including after the short pause. Saturated-cache restart restored reads but one client retained an old 64 MiB arena with two operation records and one sink-slot owner; the replacement generation closed. The diagnostic does not establish the remaining window's publication state. No forced release of unconfirmed remote access is introduced. Source inspection confirms a pre-existing recovery boundary: an expired reliable journal entry releases control permits, while later received frames still require contiguous lane sequence numbers. A permanently missing urgent sequence can therefore prevent Stop/Stopped delivery. Current fault captures lack per-record sequence and cleanup-state details and do not establish which retained records encountered that sequence gap.

Normal candidate GET and PUT phases had zero observed active ownership during the final idle interval and fresh zero samples after server shutdown for agents, registered budgets, generations, operation records, and native requests; shutdown logs contained no retained/failure events. The two tuned baseline GET phases already retained internode records before shutdown and reported shutdown failures afterward. Fault-resource retention above remains a separate result.

CI: 11 checks passed. Dependency Advisories fails on the existing unmaintained pin-utils 0.1.1 entry, RUSTSEC-2026-0441; the complete root Cargo.lock blob is identical in the PR base and validated head. Full Integration was skipped and has no execution result. Advisory job.

Review

• Correctness and replay safety: no findings after checking exact length/EOF, provider acceptance, and persisted-checksum errors.
• Resource lifetime and cancellation: no findings after checking reader lifetime and provider ownership.
• Bounded memory: checksum preparation uses a fixed-size buffer; multipart preserves the existing part-size bound.

Risks and rollback

Calculated checksums require a full bounded-buffer read before upload admission. HTTP fallback retains its existing immutable request-body allocation. Revert the upload changes and UCX dependency update together if rollback is required.