Skip per-case force_flush when no scorer/classifier uses trace: - #211
Draft
Wil Alvarez (satanas) wants to merge 1 commit into
Draft
Skip per-case force_flush when no scorer/classifier uses trace:#211Wil Alvarez (satanas) wants to merge 1 commit into
Wil Alvarez (satanas) wants to merge 1 commit into
Conversation
Runner#run_eval_case force-flushes the tracer_provider synchronously after every case so a trace:-consuming scorer/classifier can BTQL-query it immediately. That flush runs unconditionally today, even when nothing in the eval declares trace: at all, and measures at 10-25s+ per case in practice - dwarfing typical task/scorer time for evals that don't need it. Scorer/Classifier already expose #call_parameters for KeywordFilter's own introspection; reuse it here to detect whether anything registered actually wants trace: (or accepts **kwargs, which might), and skip the flush and the (otherwise-unused) Trace object entirely when nothing does. The async BatchSpanProcessor still exports normally, and the SDK's own at_exit flush still guarantees final delivery, so no data is lost. Fixes braintrustdata#210
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #210
What
Runner#run_eval_casecallseval_context.tracer_provider.force_flushsynchronously after every case, unconditionally, before running scorers/classifiers. This exists so atrace:-consuming scorer/classifier can BTQL-query that case's spans immediately with a freshness guarantee - but it runs even when nothing in the eval declarestrace:at all.Impact (see #210 for full numbers)
Measured 10-25s+ per
force_flushcall against a ~139-case eval with notrace:-consuming scorers, dwarfing typical per-case task/scorer time. Increasingparallelismdidn't help (one flush got slower, 48s, under concurrency).Fix
Scorer/Classifieralready expose#call_parametersforKeywordFilter's own kwarg-filtering. This reuses that same introspection:Runner#initializecomputes once whether any registered scorer or classifier declarestrace:(or accepts arbitrary**kwargs, which might use it), and the existing flush/build_tracecall is skipped entirely when nothing does.Nothing about the async
BatchSpanProcessorexport path changes - spans still export every ~5s regardless, and the SDK's ownat_exitflush still guarantees final delivery. This only removes the synchronous per-case wait for evals that never look attrace:.Testing
trace:-behavior tests inrunner_test.rb(test_scorer_declaring_trace_receives_eval_trace,test_trace_works_with_parallelism, etc.) - none of them changed behavior, since they all declaretrace:and are correctly detected as needing it.force_flushis not called when no scorer declarestrace:, and still is called (once per case) when one does.1302 runs, 1978 assertions, 0 failures, 0 errors(plus the 2 new tests).standardrbclean on both changed files.Open to a different shape here (e.g. an explicit opt-out flag instead of auto-detection) if that's preferred - wanted to lead with a working reference implementation since I already had the repro in hand.