Gate 7: Strix Halo (128 GB, Zen 5 AVX-512/VBMI) dataset for the fine cache sweep

[Lrrr908]

We have a 128-GB Ryzen AI Max+ 395 Strix Halo Linux system with Zen 5 AVX-512/VBMI. We can run the open Gate-7 fine cache sweep at 200+ tokens and characterize the larger-memory side of the paging cliff.

Is Gate 7 already being worked, and would a Strix Halo dataset be useful? If yes, what exact command/configuration/output format would you prefer?

For reference, the gate as we understand it: docs/GATES.md @ 2056d44 (lines 318–375) — tests/sweep.c with cache=, a fine grid between 287 and 1498 slots, over a long generation (200 tokens or more), one process, interleaved arms per LEARNED.md §38. Kill criterion: if at 200+ tokens a 3–4 GB cache is no longer within ~10% of floor + 1x, the rule stands as written.

[marcobambini]

Gate 7 is open, nobody is working it, and yes — a dataset would be useful.
Answering the command question below, but two things come first, because one
of them is worth more than Gate 7 and costs minutes instead of an hour.

Your machine can do something no machine has ever done here

docs/BACKENDS.md records AVX-512 as "compiled and dispatched, never
executed". That is literal: src/simd_avx512.c has been compiled and
CPUID-dispatched since 2026-07-28 and no machine this project can reach has
ever run an instruction from it. The container CPU reports AVX512F=0, and
so does qemu-x86_64 -cpu max under Rosetta.

So before any sweep, please just run the suite:

make && make test && tests/run.sh /path/to/container

The line to report is SIMD backend matches the CPU baseline. It runs
WASTE_BACKEND=cpu against the dispatched path and compares logits, so on a
Zen 5 it is the first execution of those kernels anywhere. It costs a couple
of minutes and it either confirms four months of untested code or finds a
bug in it. ./waste info CONTAINER should print backend AVX-512; if it
prints AVX2 or CPU, that is itself the result and worth reporting.

Kimi-Linear (19 GB) is fine for this — it does not need K3.

And your machine has VBMI, which #38 needs

I opened #38 an hour ago: VQ4P's apply is vectorized behind an #ifdef in
model.c and never moved behind the dispatch table, so x86 has no kernel for
it. The candidate primitive is VBMI's vpermi2b, and Zen 5 has it. That
issue says in as many words that whoever writes it is the only person who can
test it — you would be the first person able to. No obligation at all; you
asked what would be useful, and that is the honest ranking.

Gate 7 proper

The command. tests/sweep.c takes megabytes, not slots:

sweep CONTAINER ids,.. n_gen KEY=v1,v2,.. [repeat]
  KEY is lookahead, iodepth, cache (MB) or topk

The gate's 287 and 1498 slots are K3 expert records at 11.83 MiB each, so the
grid in the units the tool wants is roughly 3400 to 17740 MB. Something
like:

./sweep ~/models/k3.waste 1008,10484,318,15383 200 cache=3400,6800,10600,17740 2

repeat defaults to 3 and the arms are interleaved within one load, per §38 —
that is the part that matters more than the grid, because it is what makes the
arms comparable at all.

Output format: whatever sweep prints, verbatim, plus ./waste info CONTAINER, the CPU/RAM/storage, and the commit. Please report medians with
the spread rather than a single figure, and if you can, say whether any run
was an outlier rather than dropping it — §50 hit a machine that produced
occasional 30% low outliers on byte-identical work with no identified cause,
and that is the kind of thing only shows up when people report it.

Two prerequisites and one caveat, so you do not spend 45 minutes on the
wrong configuration:

1. Gate 7 is defined on K3, so it needs the 982 GB container. You did not
   say whether you have it. If you do not, that is the real blocker and the
   README has the torrent; the gate cannot be run on Kimi-Linear, whose
   working set is small enough that the quantum question does not arise.

2. 128 GB changes what the gate measures, and I think in your favour. The
   resolver takes the largest of floor + 3x/2x/1x that fits under 7/8 of
   RAM. On the 64 GB machine here that is always floor + 1x, so the whole
   question is whether there is a usable size below the quantum. On 128 GB,
   floor + 3x is about 78 GB and fits — so your machine exercises an arm
   this one has never been able to select. That is genuinely new information
   and not a replication.

3. The 64 GB paging cliff will not reproduce on your machine, and should
   not be expected to. §16's 8x collapse is the OS paging when the budget
   approaches physical RAM; with 128 GB and a 17.77 GiB expert set there is
   no cliff to meet. Reporting "no cliff observed" is a correct result, not a
   null one — UCRT64 + first real Kimi-Linear container on Windows: whole-model oracle PASS on AVX2, and five gaps no CI job can reach #36 found the same thing from the other direction on an 18 GB
   container and it clarified the boundary of that claim.

The kill criterion is unchanged: if at 200+ tokens a 3–4 GB cache is no longer
within ~10% of floor + 1x, the rule stands as written and §39 becomes a note
about short sessions.

Thanks for asking before running rather than after. The scoping in point 2 is
the part I would not have been able to tell you afterwards from a results
table.

[Lrrr908]

Gate 7 data — 128 GB Strix Halo, larger-memory side of the curve

Evidence only. Config tagged so the absolute numbers aren't read against yours.

Configuration

• Host: Ryzen AI Max+ 395 (Strix Halo), 128 GB, Linux, Zen 5 AVX-512/VBMI
• Container: K3 (arch kimi-k3, 93 layers, 896 experts, top-16, hidden 7168,
  2.78 T total / 104.19 B active, VQ3R experts + Q4G/Q8G/F32 trunk)
• ./waste plan (ctx 4096): floor 29.19 GB (trunk 27.28 + KDA/KV 659 MB +
  scratch 919 MB + min cache 379 MB); working_set_bytes 18,461,294,592
  (x = 17,606 MiB); recommended floor+3x = 80.77 GB
• Tool: tests/sweep.c, cache= in MiB (<<20), one process, interleaved arms
• Prompt: fixed 13-token code prompt, tokenized against the K3 container;
  200 generated tokens/arm, 3 repeats/arm
• Storage: Kingston NVMe (slower than your Mac SSD — see caveat)
• Pinned commit: 2056d44

Frozen grid (cache-only MiB) and results

cache=3072,3400,4096,6800,10600,17606,17740,35212,52818 — brackets the
kill-criterion band (3–4 GB), includes both gate points (287/1498 slots), and
two extended arms (floor+2x, floor+3x) a 64 GB host cannot select.

cache MiB	slots	tok/s r1/r2/r3	tok/s median (spread)	hit % median
3072	259	0.091 / 0.066 / 0.082	0.082 (−0.016/+0.009)	29.7
3400	287	0.054 / 0.073 / 0.075	0.073 (−0.019/+0.002)	29.7
4096	346	0.087 / 0.082 / 0.087	0.087 (−0.005/+0.000)	29.9
6800	574	0.094 / 0.090 / 0.090	0.090 (+0.004)	32.1
10600	895	0.103 / 0.107 / 0.096	0.103 (−0.007/+0.004)	36.6
17606	1487	0.113 / 0.117 / 0.108	0.113 (−0.005/+0.004)	41.0
17740	1499	0.114 / 0.121 / 0.114	0.114 (+0.007)	41.2
35212	2975	0.128 / 0.142 / 0.136	0.136 (−0.008/+0.006)	49.8
52818	4463	0.140 / 0.154 / 0.159	0.154 (−0.014/+0.005)	56.6

17606 vs 17740 (1487 vs 1499 slots) agree within 0.9% tok/s — one consistency
point, not two independent arms.

Observations (measurement)

• Hit rate climbs to 56.6% at 52,818 MiB with no knee. tok/s rises and
  GB-read falls monotonically across the entire grid — no inversion, no
  discontinuity. This is the part only a >64 GB host can produce: the curve
  past floor+1x is smooth, not a cliff. (35212 and 52818 both ran; peak RSS
  ≈ 79.3 GiB on the largest arm.)
• At 200 tokens the 3–4 GB band measures ~70–75% of floor+1x on tok/s
  (median ≈0.082 vs ≈0.113), consistent across all three repeats.
• Repeats are tight within each arm (hit% identical to 0.1% on several arms),
  except the two smallest arms: r1 of 3072 (0.091) and r1 of 3400 (0.054)
  sit well off their r2/r3. Both are the first repeat of their arm. That
  first-run-slower pattern looks like warming / page-cache state rather than
  a random outlier — flagging as an observation; you'd know better than us.
  (This is the §50 spread showing up on the small-cache end.)
• No thermal throttle over the full run (max clock held ~3.56–3.59 GHz,
  40–45 °C throughout).

Caveat on absolute throughput

Absolute tok/s (0.05–0.16) is well below your ~0.6 — different host, and the
Kingston NVMe serves every expert miss. The shape and the ratios are the
contribution here, not the absolute numbers. Please don't read 0.14 tok/s
against your 0.6 directly.

---

Addendum (separate note): what "one token's working set" counts

Reading waste plan, x = 17.19 GiB. From FORMAT.md, one record = 12,406,784 B
and a token touches top_k × MoE-layers. Two ways to compute it disagree unless
you know the convention — comparing GiB to GiB:

• resolver (src/waste.c:389): rec × top_k × layers=93 (includes the
  dense layer with no bank) = 18,461,294,592 B = 17.19 GiB
• gross MoE-touched: rec × 16 × 92 = 18,262,786,048 B = 17.01 GiB

Difference is 1.06% — matching your comment at L366–371 that the 93 makes it
"about 1% loose... in the direction that recommends slightly more." That's
fine — but a reader computing FORMAT.md × 1472 gets 17.01 GiB and won't match
waste plan's 17.19 GiB without knowing it's 93 not 92. Flagging in case the
plan output is worth a note. Not a bug — a documentation footgun.

[marcobambini]

This closes Gate 7, and it closes it against the thing I was hoping for.

The kill criterion fired

I wrote it in advance, in this thread, so there is nothing to argue about:

if at 200+ tokens a 3–4 GB cache is no longer within ~10% of floor + 1x,
the rule stands as written and §39 becomes a note about short sessions.

Your band, against your own floor + 1x of 0.113:

cache	median tok/s	vs floor+1x
3072 MiB	0.082	72.6%
3400 MiB	0.073	64.6%
4096 MiB	0.087	77.0%

23–35% below, not 10%. So: the quantum rule stands as written. The budget
resolver keeps stepping in whole multiples of a token's working set, §39's
"a 3.32 GB cache is within 10% of a 17.32 GB one" becomes a statement about
four-token sessions, and Gate 7 closes without a code change.

§39 named its own failure mode in the sentence that opened this gate — "four
generated tokens is exactly the length that flatters a small cache, and
cross-token reuse is what a large one buys" — and that is now measured rather
than suspected.

The confound, and why the result survives it anyway

Strictly, your 200-token run on a Strix Halo against my 4-token run on an M5
Pro varies length and host at once. You did not run a 4-token arm, so
nothing in your dataset separates them on its own.

It survives because the deciding quantity does not have to be throughput.
Hit rate is a counter, not a measurement — it falls out of the routing
trace and the cache policy, and it does not care what the disk or the cores
are doing. Yours is identical to 0.1% across repeats; mine was identical to
the digit. So it is the one column that can be put beside a different machine.

slots	§39, 4 tokens	yours, 200 tokens
287	29.1%	29.7%
~1,490–1,500	36.2%	41.0 / 41.2%
2,537	41.3%	—
2,975	—	49.8%

The small cache barely moves — +0.6 points — and the large one gains five.
That is the mechanism, visible directly: at 287 slots essentially all the hits
are the lookahead, which works within a token and therefore does not care how
many tokens follow. Everything above that is cross-token reuse, and cross-token
reuse is the thing four tokens cannot show.

The sharpest way to say it: your 200-token run reaches at 1,499 slots the hit
rate my 4-token run needed 2,537 slots to reach.

One caveat I would rather state than have you find: your prompt is a 13-token
code prompt and §39's is a different id list, so the routing traces differ and
those columns are not the same experiment. Read the shape, not the digits. It
is corroboration of a mechanism, not a controlled length sweep — but it points
the same way as the throughput, from a quantity the host cannot move.

"No cliff observed" is right, and it is narrower than it looks

I said in advance this would not reproduce and that reporting its absence is a
result rather than a null. It is — but let me bound it properly, because the
grid stopped short of where the comparison would bite.

§16 and §39's collapse is a fraction of physical RAM, not a cache size. On the
64 GB machine it sits between the 46 GB arm (~72% of RAM, 0.63 tok/s, fine) and
the 52 GB arm (~81%, 0.07 tok/s, dead). Your largest arm peaked at 79.3 GiB
of a 128 GB host, about 66% — under the fraction that was still healthy here.

So your monotone curve is exactly what should happen and it is not yet evidence
that the cliff is absent on 128 GB; it is evidence that you never entered the
zone. The arm that would test it is roughly cache=70000, which puts total
RSS near 81% of your RAM. Entirely optional, and genuinely interesting either
way: if it collapses, §16 is a fraction and travels; if it does not, the
fraction is wrong and the resolver's 7/8 headroom is holding back machines that
do not need it (that is §57, which set the eighth without ever measuring it).

What your grid does establish, and no 64 GB host can: the curve past
floor + 1x is smooth to 4,463 slots and 56.6% hit, with tok/s rising and GB
read falling monotonically the whole way. floor + 2x and floor + 3x are
arms this project had never been able to select. That part is new information,
not a replication.

The addendum is right, and the code already knows

You read it correctly. waste plan's working set is rec × top_k × layers
with layers the total count, 93, including the dense layer that has no
bank — so a reader computing FORMAT.md × 1,472 gets 17.01 GiB and does not
match the 17.19 GiB printed at them.

The comment beside it in src/waste.c already concedes the 1%, in your words,
and then gives the reason it stays:

That makes both about 1% loose on K3, in the direction that recommends
slightly more rather than less, and changing it would move a figure
tests/run.sh asserts.

So the arithmetic is deliberate and the documentation is the gap — which is
what you said. Calling it a footgun is fair: the comment explains it to
somebody reading waste.c, and the person who hits it is reading FORMAT.md
and a terminal. That is worth a line where it is printed rather than where it
is computed, and I would not have written it without this.

The first-repeat-slowest pattern is now three machines

Both your smallest arms — 3072 r1 at 0.091, 3400 r1 at 0.054 — sit off their
own r2/r3, and both are first repeats. #44 reports the identical shape on a
Zen 2 Windows box and cites your observation as the second instance. That is
three hosts across three vendors and two operating systems, which is past the
point where it should be treated as each reporter's local noise. Flagging it
rather than dropping the run was the right call, and it is why the medians in
your table are the number to read.

—

Landing this as a dated LEARNED.md section — the verdict, your grid, the
hit-rate comparison and its caveat, and the RAM-fraction bound — and closing
Gate 7 in GATES.md against it. I will link both here.

Thank you. You ran a gate that killed a result I liked, on hardware nobody here
has, and then flagged the two weakest points in your own data before I could.

[marcobambini]

Both landed, in 5ccfcd2:
docs/LEARNED.md §63 and Gate 7 closed in docs/GATES.md as
❌ refuted — the quantum stands. It was the last open gate in that file.

§39 is left exactly as written, including its "within 10%" sentence and its
pointer to Gate 7 as open. That is the rule in LEARNED.md — a refuted claim
stays in with the numbers that killed it, and later wins — so §63 supersedes it
rather than correcting it in place. Your data is what makes the pair readable.

Three things from your report are recorded as their own findings rather than as
supporting detail for the verdict:

• The hit-rate comparison, with its caveat. It is the reason the result
  survives being measured on a machine this project cannot run: hit rate is a
  counter and travels, tok/s is a measurement and does not. The caveat that the
  two runs use different prompts is in there in the same paragraph, because it
  is what keeps it corroboration of a mechanism rather than a length sweep.
• floor + 2x and floor + 3x are smooth — monotone to 4,463 slots and
  56.6% hit, no knee. Recorded as new information, since no 64 GB host can
  select those arms.
• The cliff bound. Your ~66% of RAM against the ~72% that was still healthy
  here, and cache=70000 named as the arm that would test it. §63 says what
  each outcome would mean, including that a non-collapse would say §57's eighth
  of headroom is too conservative — which would be a finding about the resolver
  and not about your machine.

EFFICIENCY.md also had a sentence that could be read as endorsing a small
default — "a cache far below a token's working set is no longer worth zero" —
which is still true and now continues "it is still not worth choosing", with
your numbers. That one was not on my list before this run.

The plan-output note for the 93-vs-92 working set is not in this commit. §63
records the footgun and why the arithmetic is deliberate; the line where the
number is printed is a separate change and I would rather not fold a CLI edit
into a docs commit.

Thanks again. A gate that closes against the outcome the maintainer was hoping
for, run by somebody with no stake in either answer, is the version of this
that is actually worth having.

[marcobambini]

The plan-output note landed too, in
d13493a — and it turned up
something worse than the footgun, plus an error of mine.

waste plan was quoting a working set it never printed. The human output
says "3x a token's working set" and then never shows one; working_set_bytes
was reachable through --json alone. So the only way to see the number was to
compute it, which is precisely the path that produces your 17.01 vs 17.19. It
prints now, with the layer convention beside it:

  FLOOR                     29.19 GB
  one token's working set    17.19 GB   (top-k records per layer, counted
                                      over every layer including the
                                      dense ones, so ~1% above the
                                      per-record arithmetic)
  recommended               80.77 GB   (floor + 3x a token's working set, ...

The upstream half was worse. src/waste.h — the public header, the whole
API surface — described the field as "top_k records per MoE layer". That is
the sentence that generates the mismatch for anyone who believes it, and it is
the one place a reader is most entitled to. It now says total layers, carries
both figures, and says not to read the difference as a bug. Comment only, no
field added, so nothing mirroring that struct moves. --json is byte-identical
and still reports the 18,461,294,592 you quoted.

The arithmetic itself is unchanged, for the reason src/waste.c already gave.

And a correction you are owed. §63 said the resolver takes the largest arm
fitting under 7/8 of RAM. It is 3/4, and has been since §57 moved it on
2026-08-09 — src/waste.c reads phys - phys / 4. You quoted 7/8 in your
report and so did I, and the reason you did is that you got it from my own
comment in this thread, which was stale. Fixed in §63 with a note saying which
it was and when it changed.

It moves nothing in your grid: floor + 3x is 80.77 GB and clears either
ceiling on 128 GB, which is why neither of us caught it from the data. It does
change the arithmetic behind the follow-up I suggested — cache=70000 is well
past what the resolver would ever select under a quarter of headroom, which is
the point of running it, but worth knowing before you read the result.

Your addendum called this "not a bug — a documentation footgun". It was two
documentation footguns and one of them was in the public header. Thanks for
flagging it rather than working around it.