Skip to content

Add August 2026 model benchmarks - #41

Merged
atharvadeosthale merged 2 commits into
mainfrom
new-models-august-2026
Aug 24, 2026
Merged

Add August 2026 model benchmarks#41
atharvadeosthale merged 2 commits into
mainfrom
new-models-august-2026

Conversation

@atharvadeosthale

Copy link
Copy Markdown
Member

Summary

  • Adds five new models to the Arena benchmark: Gemini 3.7 Flash, Muse Spark 1.2, Grok 4.6, Kimi K3, and Claude Opus 5, with full results for both with-skills and without-skills modes.
  • Adds --models <ids> / --list-models flags to the benchmark runner so a subset of models can be run instead of the whole roster (useful when a model gets stuck).
  • Adds the Meta provider logo to ProviderLogo.

Results (overall)

Model Without skills With skills
Claude Opus 5 97.4% 97.4%
Kimi K3 95.0% 96.9%
Muse Spark 1.2 94.8% 97.8%
Grok 4.6 91.6% 97.8%
Gemini 3.7 Flash 93.6% 96.0%

Note: stealth/ox-alpha was also benchmarked but excluded — it does not support tool calling on OpenRouter, so it failed every MCQ (answered via tool call) and produced no usable with-skills results.

- Add Gemini 3.7 Flash, Muse Spark 1.2, Grok 4.6, Kimi K3, and Claude Opus 5
- Add --models/--list-models flags to run the benchmark for a subset of models
- Add Meta provider logo
- Benchmark results for both with-skills and without-skills modes
@greptile-apps

greptile-apps Bot commented Aug 24, 2026

Copy link
Copy Markdown

Greptile Summary

The PR adds five August 2026 benchmark models and their result datasets, introduces model filtering and listing options for the benchmark runner, and adds Meta provider branding.

  • Adds Gemini 3.7 Flash, Muse Spark 1.2, Grok 4.6, Kimi K3, and Claude Opus 5 configurations.
  • Publishes with-skills and without-skills benchmark results for the new models.
  • Adds --models and --list-models runner options, including validation and per-model timestamp preservation.
  • Adds Meta monochrome and color provider icons.

Confidence Score: 4/5

The PR should not merge until filtered runs stop reporting already-complete selected models as newly evaluated.

The runner assigns a fresh timestamp before checking whether a selected model has any unfinished questions, causing unchanged results to receive a false public “Last updated” date.

Files Needing Attention: benchmark/src/index.ts

Important Files Changed

Filename Overview
benchmark/src/index.ts Adds model-selection CLI support and preserves existing timestamps, but still refreshes selected models that perform no benchmark work.
benchmark/src/config.ts Adds configuration and provider metadata for five new benchmark models.
src/components/ProviderLogo.tsx Adds Meta monochrome and color logo mappings.
src/data/results-with-skills.json Adds with-skills results for the five new models and updates aggregate benchmark metadata.
src/data/results-without-skills.json Adds without-skills results for the five new models and updates aggregate benchmark metadata.

Fix all with Greploop Fix All in Claude Code Fix All in Codex

Prompt To Fix All With AI
### Issue 1
benchmark/src/index.ts:459
**Skipped models receive fresh timestamps**

When a filtered run selects a model whose existing results are already complete, this assignment refreshes its `runDate` before `runBenchmark` skips every question, causing unchanged results to be displayed as newly evaluated on the model detail page.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Reviews (2): Last reviewed commit: "Address review: preserve runDate for uns..." | Re-trigger Greptile

Comment thread benchmark/src/index.ts
Comment thread benchmark/src/index.ts
Comment thread benchmark/src/index.ts
}

// Only models actually run get a fresh timestamp; others keep theirs
models[model.id].runDate = new Date().toISOString();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Skipped models receive fresh timestamps

When a filtered run selects a model whose existing results are already complete, this assignment refreshes its runDate before runBenchmark skips every question, causing unchanged results to be displayed as newly evaluated on the model detail page.

Knowledge Base Used:

Prompt To Fix With AI
This is a comment left during a code review.
Path: benchmark/src/index.ts
Line: 459

Comment:
**Skipped models receive fresh timestamps**

When a filtered run selects a model whose existing results are already complete, this assignment refreshes its `runDate` before `runBenchmark` skips every question, causing unchanged results to be displayed as newly evaluated on the model detail page.

**Knowledge Base Used:**
- [Benchmark orchestration](https://app.greptile.com/appwrite/-/custom-context/knowledge-base/appwrite/arena/-/docs/benchmark-orchestration.md)
- [Leaderboard results and ranking](https://app.greptile.com/appwrite/-/custom-context/knowledge-base/appwrite/arena/-/docs/leaderboard-results-and-ranking.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Claude Code Fix in Codex

@atharvadeosthale
atharvadeosthale merged commit 0285fa2 into main Aug 24, 2026
1 check passed
@atharvadeosthale
atharvadeosthale deleted the new-models-august-2026 branch August 24, 2026 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant