chore: schedule the daily online eval run and document its alerts - #5
Open
RobertCrupa wants to merge 3 commits into
Open
RobertCrupa wants to merge 3 commits into
RobertCrupa wants to merge 3 commits into
Conversation
…-team#271) Add actors/judge/schedule/online-daily.json, a complete POST /v2/schedules body for the online judge: 06:00 UTC daily, isExclusive, 1 h timeout, 1 GB, online-mode defaults pinned in runInput (mode online, sampleRate 0.2, maxItems 100, judgeModel deepseek/deepseek-v4-flash, concurrency 4), restartOnError false so a crash surfaces as a FAILED run. schedule/README.md justifies each field, lists the Actor env secrets (LANGFUSE_BASE_URL, LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY), gives the curl to create it and the Apify run-status alert for a dead run. Add actors/judge/monitors/agent-judge-pass-rate.md: the Langfuse alert to enter by hand (no public API): Scores (boolean), avg, score name agent_judge, 1-day window, operator <, warning 0.8, alert 0.6 in one alert, Notify after sustained NO_DATA (6 h) for the silently dead Actor, Slack via an automation. Thresholds are loose until a baseline exists; the planned second alert on agent_judge_argumentCorrectness is described. Nothing is created in Apify or Langfuse. Mention both modes in the Actor description and point the README at the new files.
…team#271) Raise the schedule timeout to 7200 s: the judge LLM call is 2 attempts x 120 s plus a 2 s sleep, so 100 traces at concurrency 4 is 25 x 242 s = 100.8 min before the observation and tools/list fetches. Note that a run dying or timing out after selection has already moved the checkpoint until #270 reorders the write, so recovery is a manual windowStart/windowEnd run. Mark the scores-boolean webhook view string and the notifications.email POST field as unverified, replace the unverified error code with plain words, phrase the sustained NO_DATA delay as "6 h or the nearest offered value", and explain the daily NO_DATA flicker so nobody lowers the delay or switches to treat-as-0.
…m#271) The #271 docs were written before #270 landed under them. Correct the claims that are no longer true and one that never was. - schedule/README.md: the checkpoint now moves only after the window's scores and rollup succeed, so a timed-out or crashed run leaves the window to be retried on the next tick. The window overrides are for backfills only. Also fix the next-tick gap after a 2 h timeout (22 h, not 23 h). - monitors/agent-judge-pass-rate.md: CreateScoreRequest has no timestamp field, so both score copies carry the write time, not the trace's. Drop the pinned-timestamp claim and the timing rationale for preferring the run copy (retention still prefers it), restate the daily NO_DATA gap accordingly and record the D-1 day label the Actor README points here for. - schedule/online-daily.json: pin environment: prod in the run input, so the schedule records that production runs score prod traffic only. - README.md: move the schedule and alerting section above the v1 scope notes.
RobertCrupa
added this pull request to stack #6
September 9, 2026 15:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes apify/ai-team#271. Part of apify/ai-team#249, stacked on the #270 PR. Docs and config only; nothing is created in Apify or Langfuse by this PR.
Facts checked against the Langfuse Alerts docs and the Apify docs are listed in each file; the remaining assumptions (the exact NO_DATA delay values the UI offers, whether `notifications.email` is accepted on the POST body) are called out as such.