Summary
Braintrust::Eval::Runner#run_eval_case calls eval_context.tracer_provider.force_flush synchronously after every single case, before running scorers/classifiers:
# Flush spans so they're queryable via BTQL, then build trace
eval_context.tracer_provider.force_flush if eval_context.tracer_provider.respond_to?(:force_flush)
kase.trace = build_trace(eval_span)
This exists so a scorer/classifier that declares trace: can BTQL-query that case's spans immediately with a freshness guarantee. That's a reasonable feature, but the flush runs unconditionally, even when nothing in the eval consumes trace: at all — which is the common case for eval suites that only use input/expected/output/metadata.
Impact
Measured directly against a Ruby eval with ~139 cases, parallelism: 1 (the default), and no trace:-consuming scorers: force_flush took 10-25+ seconds per call, dwarfing everything else (a real per-case Bedrock/LLM task call was 5-20s, an LLM-judge scorer call was ~2s). This SDK-level overhead alone accounts for the large majority of the eval's total wall-clock time (~24 minutes for 139 cases).
I also tried increasing parallelism to see if the (I/O-bound) wait would overlap across cases — it didn't help; one force_flush call actually got slower (48s) under concurrency, suggesting the calls contend with each other rather than overlapping cleanly.
Worth noting: the BatchSpanProcessor Braintrust sets up already exports spans asynchronously every ~5s regardless of this call, so results still show up in the Braintrust UI live without it, and the SDK's own at_exit hook still guarantees a final flush before the process exits. The per-case synchronous flush is purely a freshness guarantee for immediate in-process BTQL trace lookups.
Suggested fix
Only pay this cost when something will actually use it. Scorer/Classifier instances already expose #call_parameters for KeywordFilter's own introspection — the same mechanism can detect whether any registered scorer/classifier declares trace: (or accepts arbitrary kwargs via **kwargs, which might use it), and skip the per-case flush/trace-build entirely when none do.
I have a draft PR with this approach: introduce a Runner#callable_wants_trace? check computed once in #initialize, and guard the existing flush/build_trace call behind it. Verified against the existing test suite (all passing, including the existing trace: behavior tests) plus two new tests asserting force_flush is/isn't called depending on whether a scorer declares trace:. Happy to open it if useful — wanted to raise the issue with numbers first in case there's a different preferred direction (e.g. an explicit opt-out flag instead of auto-detection).
Summary
Braintrust::Eval::Runner#run_eval_casecallseval_context.tracer_provider.force_flushsynchronously after every single case, before running scorers/classifiers:This exists so a scorer/classifier that declares
trace:can BTQL-query that case's spans immediately with a freshness guarantee. That's a reasonable feature, but the flush runs unconditionally, even when nothing in the eval consumestrace:at all — which is the common case for eval suites that only useinput/expected/output/metadata.Impact
Measured directly against a Ruby eval with ~139 cases,
parallelism: 1(the default), and notrace:-consuming scorers:force_flushtook 10-25+ seconds per call, dwarfing everything else (a real per-case Bedrock/LLM task call was 5-20s, an LLM-judge scorer call was ~2s). This SDK-level overhead alone accounts for the large majority of the eval's total wall-clock time (~24 minutes for 139 cases).I also tried increasing
parallelismto see if the (I/O-bound) wait would overlap across cases — it didn't help; oneforce_flushcall actually got slower (48s) under concurrency, suggesting the calls contend with each other rather than overlapping cleanly.Worth noting: the
BatchSpanProcessorBraintrust sets up already exports spans asynchronously every ~5s regardless of this call, so results still show up in the Braintrust UI live without it, and the SDK's ownat_exithook still guarantees a final flush before the process exits. The per-case synchronous flush is purely a freshness guarantee for immediate in-process BTQL trace lookups.Suggested fix
Only pay this cost when something will actually use it.
Scorer/Classifierinstances already expose#call_parametersforKeywordFilter's own introspection — the same mechanism can detect whether any registered scorer/classifier declarestrace:(or accepts arbitrary kwargs via**kwargs, which might use it), and skip the per-case flush/trace-build entirely when none do.I have a draft PR with this approach: introduce a
Runner#callable_wants_trace?check computed once in#initialize, and guard the existing flush/build_tracecall behind it. Verified against the existing test suite (all passing, including the existingtrace:behavior tests) plus two new tests assertingforce_flushis/isn't called depending on whether a scorer declarestrace:. Happy to open it if useful — wanted to raise the issue with numbers first in case there's a different preferred direction (e.g. an explicit opt-out flag instead of auto-detection).