Skip to content

Commit 74f3957

Browse files
committed
Add provider-neutral Claude/OpenAI benchmarks and fix temperature for reasoning models
- benchmarks/benchmark_claude.py: end-to-end recovery + sampled audit + classification, defaults anthropic/claude-sonnet-5 with --provider/--model/ --base-url overrides, smoke + mock modes, per-dataset checkpoints - benchmarks/benchmark_claude_classification.py: focused correctness harness with the same provider options and offline fake client - OpenAIClient: omit temperature for gpt-5/gpt-6/o-series reasoning models (API rejects it with 400); reasoning effort stays at API default (medium) - benchmark_buffdata PRICING: add gpt-5.6-sol / gpt-5.6 rates
1 parent f08e12c commit 74f3957

4 files changed

Lines changed: 786 additions & 11 deletions

File tree

benchmarks/benchmark_buffdata.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -162,6 +162,8 @@
162162
PRICING = {
163163
"gemini-3.7-flash": {"input": 0.075, "output": 0.30},
164164
"claude-sonnet-5": {"input": 3.00, "output": 15.00},
165+
"gpt-5.6-sol": {"input": 4.00, "output": 20.00},
166+
"gpt-5.6": {"input": 4.00, "output": 20.00},
165167
"default": {"input": 1.00, "output": 5.00},
166168
}
167169

0 commit comments

Comments
 (0)