Commit 74f3957
committed
Add provider-neutral Claude/OpenAI benchmarks and fix temperature for reasoning models
- benchmarks/benchmark_claude.py: end-to-end recovery + sampled audit +
classification, defaults anthropic/claude-sonnet-5 with --provider/--model/
--base-url overrides, smoke + mock modes, per-dataset checkpoints
- benchmarks/benchmark_claude_classification.py: focused correctness harness
with the same provider options and offline fake client
- OpenAIClient: omit temperature for gpt-5/gpt-6/o-series reasoning models
(API rejects it with 400); reasoning effort stays at API default (medium)
- benchmark_buffdata PRICING: add gpt-5.6-sol / gpt-5.6 rates1 parent f08e12c commit 74f3957
4 files changed
Lines changed: 786 additions & 11 deletions
File tree
- benchmarks
- buffdata/engine
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
162 | 162 | | |
163 | 163 | | |
164 | 164 | | |
| 165 | + | |
| 166 | + | |
165 | 167 | | |
166 | 168 | | |
167 | 169 | | |
| |||
0 commit comments