Bank striping across N devices: WASTE_BANK_SHARDS, record-granularity round-robin - #53
Bank striping across N devices: WASTE_BANK_SHARDS, record-granularity round-robin#53mfethe1 wants to merge 1 commit into
Conversation
55948c0 to
09ada61
Compare
WASTE_BANK_SHARDS=dirA,dirB,... opens each layer's bank as N shard files, expert e on shard e%N at offset (e/N)*rec_bytes. Round-robin because the per-token demand is k experts of ONE layer — layer-granularity placement would leave every read of a token on one drive. Unstriped is the N=1 case of the same code path; env unset = exact prior behavior. - waste_bank: fd[16] + n_shards; all 6 deref sites migrated - load: manifest-driven shard open, fail-closed on short shard sets - bank_fetch: shard+offset resolution, byte-exact by construction - tools/split_banks.py: split + byte-for-byte verify modes - evidence: tiny.waste striped across 2 dirs, logits byte-identical (sha de62689a...), make check 48/0, ASan+UBSan striped run clean 0 findings
09ada61 to
bc710d5
Compare
|
Repointed the PR branch to a clean single-commit base on current Verified on the rebased head before pushing: |
1 participant
What
Optional multi-device expert-bank striping. Set
WASTE_BANK_SHARDS=/mnt/a,/mnt/band each layer's bank opens as N shard files (bank basename in each dir); experteis read from sharde % Nat offset(e / N) * rec_bytes.tools/split_banks.pyproduces and byte-verifies shard sets. Unset env = exact current behavior — unstriped is the N=1 case of the same code path.Why
Per-token demand is top-k experts of a single layer — one file today, so one device serves every expert read of a token. Round-robin record placement spreads a token's reads across devices, so effective bandwidth scales toward min(N, readers) with the existing async reader pool (
WASTE_IO_THREADS). On a 2-drive rig this targets ~1.8-2x; on 4 NVMe with more readers, more.No container format change: shards are plain files named after each bank. Fail-closed: any missing or short shard refuses the load.
Evidence
de62689a…, argmax + max-logit equal) — correctness gate is identity, not tolerancetools/split_banks.py --mode verify: every record byte-identical post-splitmake check48 passed / 0 failed (13 skips: no real container locally)Not claimed
No device A/B numbers yet — needs a 2-drive rig with a real container. This PR is the mechanism; the bandwidth measurement follows on hardware.