Add August 2026 model benchmarks - #41
Conversation
- Add Gemini 3.7 Flash, Muse Spark 1.2, Grok 4.6, Kimi K3, and Claude Opus 5 - Add --models/--list-models flags to run the benchmark for a subset of models - Add Meta provider logo - Benchmark results for both with-skills and without-skills modes
Greptile SummaryThe PR adds five August 2026 benchmark models and their result datasets, introduces model filtering and listing options for the benchmark runner, and adds Meta provider branding.
Confidence Score: 4/5The PR should not merge until filtered runs stop reporting already-complete selected models as newly evaluated. The runner assigns a fresh timestamp before checking whether a selected model has any unfinished questions, causing unchanged results to receive a false public “Last updated” date. Files Needing Attention: benchmark/src/index.ts Important Files Changed
Prompt To Fix All With AI### Issue 1
benchmark/src/index.ts:459
**Skipped models receive fresh timestamps**
When a filtered run selects a model whose existing results are already complete, this assignment refreshes its `runDate` before `runBenchmark` skips every question, causing unchanged results to be displayed as newly evaluated on the model detail page.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Reviews (2): Last reviewed commit: "Address review: preserve runDate for uns..." | Re-trigger Greptile |
| } | ||
|
|
||
| // Only models actually run get a fresh timestamp; others keep theirs | ||
| models[model.id].runDate = new Date().toISOString(); |
There was a problem hiding this comment.
Skipped models receive fresh timestamps
When a filtered run selects a model whose existing results are already complete, this assignment refreshes its runDate before runBenchmark skips every question, causing unchanged results to be displayed as newly evaluated on the model detail page.
Knowledge Base Used:
Prompt To Fix With AI
This is a comment left during a code review.
Path: benchmark/src/index.ts
Line: 459
Comment:
**Skipped models receive fresh timestamps**
When a filtered run selects a model whose existing results are already complete, this assignment refreshes its `runDate` before `runBenchmark` skips every question, causing unchanged results to be displayed as newly evaluated on the model detail page.
**Knowledge Base Used:**
- [Benchmark orchestration](https://app.greptile.com/appwrite/-/custom-context/knowledge-base/appwrite/arena/-/docs/benchmark-orchestration.md)
- [Leaderboard results and ranking](https://app.greptile.com/appwrite/-/custom-context/knowledge-base/appwrite/arena/-/docs/leaderboard-results-and-ranking.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Summary
--models <ids>/--list-modelsflags to the benchmark runner so a subset of models can be run instead of the whole roster (useful when a model gets stuck).ProviderLogo.Results (overall)
Note:
stealth/ox-alphawas also benchmarked but excluded — it does not support tool calling on OpenRouter, so it failed every MCQ (answered via tool call) and produced no usable with-skills results.