feat: online mode selects production traces for scoring - #2
Open
RobertCrupa wants to merge 6 commits into
Open
RobertCrupa wants to merge 6 commits into
RobertCrupa wants to merge 6 commits into
Conversation
Add mode: datasetRun (default, unchanged) | online. Online mode selects finished production apify-ai traces from [checkpoint ?? now-24h, now - 30 min request timeout - 3 min export lag), reading them as observations from GET /api/public/v2/observations (events_only instance) with traceTags / name / metadata.completed string filters, samples ceil(sampleRate * completed) capped at maxItems after a Fisher-Yates shuffle, checkpoints the window's upper bound in the default KV store (ONLINE_CHECKPOINT) after selection succeeds, and writes the tracesInWindow / completedTraces / sampled / judged / failedToJudge counters to OUTPUT. judged and failedToJudge stay 0 behind a typed judgeOnline seam for #269/#270. windowStart/windowEnd overrides never touch the checkpoint. Pure window, filter, sampling and pagination logic is unit-tested with fakes (vitest); an eslint config mirroring the runner's is added so the judge can be linted.
…eam#267) Review round 1. Drop the stringObject metadata.completed condition from the completed-trace filter: the agent sets it as trace metadata on every span of that name, so it added no selectivity and an unproven match could have zeroed completedTraces forever while the checkpoint advanced. Warn when a window has traces but no completion spans (broken gate signature) and when the checkpoint is at or past the safe upper bound. Restore the datasetRunId guard before the Langfuse env check so error precedence in datasetRun mode is unchanged. Make the #270 seam say the checkpoint write must move behind the score write. Add a test for the langfuseObservationFetcher adapter. Drop the schema's empty required array and a redundant Math.max guard.
The eslint config only knew the build tsconfig, so scripts/create-score-configs.ts was a parse error and test/ was ignored. tsconfig.check.json already covers all three directories. The jest globals were never used; the tests import from vitest.
The traceTags filter on GET /api/public/v2/observations matches per observation and the tag is only on the root AGENT span, so the completion filter's tag AND name conditions could never both hold: online evals would sample nothing on every run while the checkpoint kept advancing. Verified live on 2026-09-07 traffic: tag alone 3 AGENT rows, name alone 3 TOOL rows, both together 0. Drop the tag from completedFilter and stop intersecting the two id sets, since they are not nested: the completion span's own start time is the selection key, and the same live day showed only 1 of 3 trace ids in common. tracesInWindow stays as a documented coverage counter over root spans. A window with traffic but no completion span at all now also returns checkpoint: null, so the window is retried instead of being burned silently while the contract is undeployed or the span name has drifted.
Neither selection query constrained the environment, so dev, staging and prod turns of the same service, emitting the same span names into one project, were selected as one population. Over the 14 days to 2026-09-09 the project held dev, staging and sdk-experiment traffic and no prod traffic at all, and on 2026-09-07 the 3 rows matching a single span name split 2 dev + 1 staging. Add an online `environment` input (default prod, the value apify-ai-agent emits from its own environment setting) and put a stringOptions condition into the shared bounds so both queries inherit it. A staging backfill is `environment: "staging"`. A broken completion gate also left the run SUCCEEDED, which no alert covers. Expose it as `isGateBroken` and have the online branch call Actor.fail() after OUTPUT is written. Drop "the trace contract is not deployed" as a stated cause: with it undeployed the coverage query is 0 too, so the gate never fires. README: document the environment input, the ~33 min effective ingestion-lag tolerance the upper bound implies and the overlap mitigation left to #270, and correct the last-page cursor wording.
An unknown environment returns zero rows without an error, which is the idle-window path: green run, checkpoint advancing, nothing judged. The agent only ever emits prod, staging or dev.
RobertCrupa
added this pull request to stack #6
September 9, 2026 15:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes apify/ai-team#267. Part of apify/ai-team#249, stacked on the #268 PR.
Known: whether the span-name filter matches on the live instance is unverified; a run with traces in the window but zero completed logs a warning naming that as a broken gate. #270 must move the checkpoint write behind the score write.