Skip to content

guides: add Tested Models, the measured picture instead of a claim list - #4

Open
webdevtodayjason wants to merge 1 commit into
mainfrom
fable/tested-models
Open

guides: add Tested Models, the measured picture instead of a claim list#4
webdevtodayjason wants to merge 1 commit into
mainfrom
fable/tested-models

Conversation

@webdevtodayjason

Copy link
Copy Markdown
Contributor

What was missing

guides/models.mdx ended with a heading that said "Tested models include:" over a
list of model families that had never been benchmarked through AINode: Llama 3 405B,
Mixtral, Command R+, DeepSeek V3. Meanwhile the four runs we have actually measured
were sitting in bench/results/*.json in the product repo with no page pointing at
them and no numbers on the docs site at all.

So the docs asserted testing we had not done, and hid the testing we had.

What I decided

A new page, guides/tested-models.mdx, that defines its own bar. A model is on
it only if it was launched through AINode rather than a hand-written docker run,
somebody measured it on our hardware with engine-reported numbers, and the run is
committed as a JSON record a reader can open. Stated up front, so the page is
falsifiable.

Same table as the product README, same generator. The rows come from
scripts/render-bench-table.py in the product repo, spliced between
{/* BEGIN bench-table */} markers (JSX comments, since MDX rejects HTML ones). Each
row links to its JSON on GitHub. Regenerate with:

python3 scripts/render-bench-table.py \
    --readme ../ainode-docs/guides/tested-models.mdx \
    --begin '{/* BEGIN bench-table */}' --end '{/* END bench-table */}'

The gaps get a section, with a Warning callout saying they are real gaps rather
than omissions.
Vision on Ornith 1.5 (launched text-only with
--limit-mm-per-prompt {"image":0,"video":0} after the multimodal warmup OOM-killed
the engine core). TP=4 through the current 0.5.x launch path. The Dell C4130 that
joined as a worker on 2026-09-11 with no AINode-launched model. The GLM-5.3-Flash
TP=2 pair that runs raw outside AINode. DeepSeek V4 and V4.1 Flash.

scripts/ainode-bench.py is described, not promised. It is not on main yet, so
the page says so in an Info callout, describes the intended shape from
bench/SCHEMA.md (one JSON per run in bench/results/, named
<stamp>-<model-slug>-<label>.json), and then shows the flags of
scripts/bench-serve.py, which is on main today and takes the same measurements.

A submit-a-result section with the one rule that matters: leave out a section you
did not measure. A missing key renders "not measured", which is the answer we want;
filling it with a plausible number is what gets a result rejected.

Fixed the claim that prompted the page. models.mdx now says "Model families
known to load and serve here include" and links to the new page in one sentence.
That list was never a list of tested models.

Nav: guides/tested-models goes right after guides/models in the Guides group.

Proof

mintlify broken-links  ->  success no broken links found
@mdx-js/mdx compile    ->  ok guides/tested-models.mdx, guides/models.mdx, guides/distributed.mdx
docs.json              ->  parses

The CLI would not run through npx on this machine (npm error ECOMPROMISED: Lock compromised during build:link), so mintlify was installed into a scratch directory
and run from there against this working tree.

🤖 Generated with Claude Code

guides/models.mdx ended with "Tested models include:" and then a list of model
families nobody had benchmarked through the product. Meanwhile four real runs with
real numbers were sitting in bench/results/*.json in the product repo with no page
pointing at them.

Added guides/tested-models.mdx:

- What "tested" means here, stated as three conditions: launched through AINode
  (not raw docker), measured on our own hardware with engine-reported numbers, and
  committed as a JSON record anyone can read.
- The table of four runs, generated by scripts/render-bench-table.py in the product
  repo between {/* BEGIN bench-table */} markers, each row linking to its JSON on
  GitHub. Regenerate with:
    python3 scripts/render-bench-table.py \
        --readme ../ainode-docs/guides/tested-models.mdx \
        --begin '{/* BEGIN bench-table */}' --end '{/* END bench-table */}'
- What is tested and what is not. The gaps are named: vision on Ornith 1.5, TP=4
  through the current 0.5.x launch path, the Dell C4130 worker with no
  AINode-launched model, the GLM-5.3-Flash pair that runs raw, DeepSeek V4 and
  V4.1 Flash.
- How to run the bench. scripts/ainode-bench.py is described from bench/SCHEMA.md
  and flagged as on the way, since it is not on main yet; scripts/bench-serve.py
  is on main today and its real flags are shown.
- How to submit a result, including the rule that a section you did not measure is
  left out rather than estimated.

Changed models.mdx's "Tested models include" to "Model families known to load and
serve here include" and cross-linked the new page, because that list was never a
list of tested models.

Nav: guides/tested-models sits right after guides/models in the Guides group.

Proof: mintlify broken-links reports no broken links. @mdx-js/mdx compiles all
three touched pages. docs.json parses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant