guides: add Tested Models, the measured picture instead of a claim list - #4
Open
webdevtodayjason wants to merge 1 commit into
Open
guides: add Tested Models, the measured picture instead of a claim list#4webdevtodayjason wants to merge 1 commit into
webdevtodayjason wants to merge 1 commit into
Conversation
guides/models.mdx ended with "Tested models include:" and then a list of model
families nobody had benchmarked through the product. Meanwhile four real runs with
real numbers were sitting in bench/results/*.json in the product repo with no page
pointing at them.
Added guides/tested-models.mdx:
- What "tested" means here, stated as three conditions: launched through AINode
(not raw docker), measured on our own hardware with engine-reported numbers, and
committed as a JSON record anyone can read.
- The table of four runs, generated by scripts/render-bench-table.py in the product
repo between {/* BEGIN bench-table */} markers, each row linking to its JSON on
GitHub. Regenerate with:
python3 scripts/render-bench-table.py \
--readme ../ainode-docs/guides/tested-models.mdx \
--begin '{/* BEGIN bench-table */}' --end '{/* END bench-table */}'
- What is tested and what is not. The gaps are named: vision on Ornith 1.5, TP=4
through the current 0.5.x launch path, the Dell C4130 worker with no
AINode-launched model, the GLM-5.3-Flash pair that runs raw, DeepSeek V4 and
V4.1 Flash.
- How to run the bench. scripts/ainode-bench.py is described from bench/SCHEMA.md
and flagged as on the way, since it is not on main yet; scripts/bench-serve.py
is on main today and its real flags are shown.
- How to submit a result, including the rule that a section you did not measure is
left out rather than estimated.
Changed models.mdx's "Tested models include" to "Model families known to load and
serve here include" and cross-linked the new page, because that list was never a
list of tested models.
Nav: guides/tested-models sits right after guides/models in the Guides group.
Proof: mintlify broken-links reports no broken links. @mdx-js/mdx compiles all
three touched pages. docs.json parses.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was missing
guides/models.mdxended with a heading that said "Tested models include:" over alist of model families that had never been benchmarked through AINode: Llama 3 405B,
Mixtral, Command R+, DeepSeek V3. Meanwhile the four runs we have actually measured
were sitting in
bench/results/*.jsonin the product repo with no page pointing atthem and no numbers on the docs site at all.
So the docs asserted testing we had not done, and hid the testing we had.
What I decided
A new page,
guides/tested-models.mdx, that defines its own bar. A model is onit only if it was launched through AINode rather than a hand-written
docker run,somebody measured it on our hardware with engine-reported numbers, and the run is
committed as a JSON record a reader can open. Stated up front, so the page is
falsifiable.
Same table as the product README, same generator. The rows come from
scripts/render-bench-table.pyin the product repo, spliced between{/* BEGIN bench-table */}markers (JSX comments, since MDX rejects HTML ones). Eachrow links to its JSON on GitHub. Regenerate with:
python3 scripts/render-bench-table.py \ --readme ../ainode-docs/guides/tested-models.mdx \ --begin '{/* BEGIN bench-table */}' --end '{/* END bench-table */}'The gaps get a section, with a Warning callout saying they are real gaps rather
than omissions. Vision on Ornith 1.5 (launched text-only with
--limit-mm-per-prompt {"image":0,"video":0}after the multimodal warmup OOM-killedthe engine core). TP=4 through the current 0.5.x launch path. The Dell C4130 that
joined as a worker on 2026-09-11 with no AINode-launched model. The GLM-5.3-Flash
TP=2 pair that runs raw outside AINode. DeepSeek V4 and V4.1 Flash.
scripts/ainode-bench.pyis described, not promised. It is not onmainyet, sothe page says so in an Info callout, describes the intended shape from
bench/SCHEMA.md(one JSON per run inbench/results/, named<stamp>-<model-slug>-<label>.json), and then shows the flags ofscripts/bench-serve.py, which is onmaintoday and takes the same measurements.A submit-a-result section with the one rule that matters: leave out a section you
did not measure. A missing key renders "not measured", which is the answer we want;
filling it with a plausible number is what gets a result rejected.
Fixed the claim that prompted the page.
models.mdxnow says "Model familiesknown to load and serve here include" and links to the new page in one sentence.
That list was never a list of tested models.
Nav:
guides/tested-modelsgoes right afterguides/modelsin the Guides group.Proof
The CLI would not run through
npxon this machine (npm error ECOMPROMISED: Lock compromisedduringbuild:link), so mintlify was installed into a scratch directoryand run from there against this working tree.
🤖 Generated with Claude Code