feat: write online scores idempotently and keep them past retention - #4
Open
RobertCrupa wants to merge 4 commits into
Open
RobertCrupa wants to merge 4 commits into
RobertCrupa wants to merge 4 commits into
Conversation
…ai-team#270) Every verdict from the online judge is written to Langfuse twice: once on the trace, once under an invented datasetRunId apify-ai-online-YYYY-MM-DD so the score survives the 30-day trace sweep. Score ids are readable, <traceId>-<scoreName>-p<promptVersion>-i<JUDGE_IMPL_VERSION>-<judgeModel>, with a -run suffix on the archival copy; a bumped version writes beside the old score. Sampled traces already judged under the same version are dropped before judging, so a re-run spends nothing on the LLM. A daily rollup (pass rate and n per score, coverage counters) is upserted as dataset item rollup-YYYY-MM-DD in apify-ai-online-rollups, merging runs of the same day. The checkpoint write moves out of selectTraces (writeCheckpoint option) and happens only after the scores and the rollup are written; a rollup failure fails the run and leaves the window to be retried. OUTPUT gains scoresSkipped, scoresWritten, failedToWrite and rollupItemId.
…the window day (ai-team#270 review) A batch whose writes all failed was reported as success: the rollup came from every judged verdict, the checkpoint advanced and the run exited 0, so a Langfuse outage silently lost the window and its LLM spend. writeOnlineScores now returns the verdicts that were written, only those feed the rollup, the rollup coverage gains scoresWritten and failedToWrite, and an all-failed batch skips the rollup, keeps the checkpoint and exits non-zero. Run ids and rollup items are keyed on the UTC date of the window start, not the write, so a backfill lands on its own day; the daily run therefore labels D-1 when it writes at ~06:0x on D (README says so for the #271 monitor). The rollup is skipped when nothing was sampled or judged. README and comments now describe the ReplacingMergeTree semantics (same-day rewrite replaces the row, cross-day retry of a half-written trace leaves two rows) and the retry double count of the window counters. A round-trip test pins writer and reader to each other.
… a judge failure (ai-team#270 round 3) A live verification against langfuse.apify.dev (2026-09-09) confirmed every API assumption. datasets.create is idempotent by name, so ensureRollupDataset calls it unconditionally; the events_only hedges become "verified live"; the README records the experimentId read filter and the experiment subject kind, that ANNOTATION needs a configId (API is the only usable source), that BOOLEAN scores read back as JSON booleans in environment default, and that no public API deletes a dataset (ROLLUP_DATASET_NAME is permanent; pinned by a test). isNotFound stays on statusCode, now pinned by a test that constructs the client's real NotFoundError (@langfuse/core added as a devDependency for it). finishOnlineRun gains the judge-side twin guard: when every trace given to the judge failed to judge, no rollup, no checkpoint, AllJudgementsFailedError and a non-zero exit. Window override docs say the ISO 8601 value needs an offset.
…cs claim (ai-team#270 follow-ups) The all-judgements-failed guard is judged === 0 && failedToJudge > 0, which is what sampled - scoresSkipped > 0 && judged === 0 meant, so the extra scoresSkipped option on finishOnlineRun goes. The README no longer claims the metrics aggregation of the run copy was verified live: read-back via experimentId and the subject kind were, aggregation under an invented run is the one open inference, and the working endpoint in events_only mode is GET /api/public/v2/metrics. The two wordings for the held checkpoint are one.
RobertCrupa
added this pull request to stack #6
September 9, 2026 15:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes apify/ai-team#270. Part of apify/ai-team#249, stacked on the #269 PR.
Every Langfuse API assumption was verified live on 2026-09-09 with throwaway objects (score create and same-day replace, the run subject without a run, exactly-one-subject, EVAL rejected, datasets and items in events_only mode, v3 read with a traceId list). One inference remains and is marked as such: metrics-API aggregation of run-attached scores. No public API deletes a dataset, so the rollup dataset name is permanent once created.