feat: score sampled production turns against the online rubric - #3
Open
RobertCrupa wants to merge 4 commits into
Open
RobertCrupa wants to merge 4 commits into
RobertCrupa wants to merge 4 commits into
Conversation
…am#269) Online mode now judges each sampled trace instead of returning zeros. The turn is reconstructed from the GENERATION observations' mapped input/output (never the attributes bag), rendered for the judge with tool results included and every payload capped head-plus-tail at 4096 chars, and scored: argumentCorrectness deterministically with ajv against the live Apify MCP tool schemas (hash recomputed with a port of the agent's toolSchemaHash, schemaMatch reported), the other five criteria plus the separate holistic agent_judge verdict in one LLM call driven by the Langfuse-managed apify-ai-online-judge prompt. errorRecovery is omitted on turns with no tool error regardless of the model's answer. Verdicts are returned as a typed OnlineVerdicts value; nothing is written to Langfuse yet (TODO #270 in judgeOnline). The resolve-or-seed prompt logic moves to prompt.ts and serves both modes unchanged. judgeModel defaults per mode: deepseek/deepseek-v4-flash online, anthropic/claude-sonnet-4.6 for dataset runs.
…'s real shape (ai-team#269)
Review round 1. The agent's exporter writes one chat GENERATION per turn
(input = pre-loop messages, output = final text, no tool_call parts) and
one TOOL observation per tool call, so steps now come from the TOOL
observations ordered by startTime (name from the span, arguments from
input, result from output, isError from level/statusMessage/success/
isError), with message-part parsing kept only as a fallback. Fixtures
are rebuilt from that shape and the three facts a first live trace must
confirm are documented.
Trace metadata is read from three candidate locations (top-level key,
nested langfuse object, the attributes.mastra.metadata.langfuse JSON
string); a trace with none is judged with outcome unknown and counted in
metadataMissing, logged and put in OUTPUT. judgeOnlineTrace now returns
{verdicts, traceMetadataFound}; the OnlineVerdicts types are unchanged.
The toolSchemaHash port still hashes the raw JSON schema; a test built
the way Mastra wraps tools pins that the agent's current hash ignores
schema content, and the comment and README say schemaMatch stays false
until the agent-side fix lands, with validation on the live schemas
regardless. Also: the turn is fenced in unique delimiters and declared
data, not instructions; prior messages are capped at 10 with a dropped
count; the fallback treats error-typed tool outputs as errors; the log
says plainly when no live toolset was loaded.
…i-team#269) Verified against staging Langfuse on 2026-09-09. Every live trace carries two GENERATIONs: the turn's sonnet chat and a later haiku chat that Memory uses for the thread title and compaction. Taking every GENERATION let the memory one become the judged turn, with the summarisation prompt as the question, the generated title as the answer, and its span id in every comment. The turn's generations are now the ones whose top-level sessionId is non-empty, with the TOOL-span parent and then the earliest as fallbacks; the rest are counted as excludedGenerations. A call whose span recorded no arguments is no longer validated against the schema, where it became a bogus "/ must be object" failure. The criterion is omitted with a reason instead, and arguments and results fall back to attributes.gen_ai.tool.call.arguments / .result when the mapped input/output is absent. Also from the live data: the model arrives as the `model` field, not providedModelName, so it was always undefined; a failed tool call is level ERROR with an empty statusMessage and a serialised MastraError on output, so isErrorOutput matches that shape and the dead attributes.success read is gone; expandMetadata is accepted by the endpoint and forwarded by the SDK. Test fixtures are real observation rows with only the long payloads trimmed.
…i-team#269)
The fixture claimed verbatim fidelity it did not have: the TOOL result was a
172-character rewrite with a top-level `total: null` that does not exist live,
and every row silently dropped the endpoint's eight non-`attributes.*` metadata
keys. Re-fetched both trace rows and the ERROR observation from staging:
- the TOOL result now carries the real five top-level keys and the first of its
five actor objects in full, with the other four as one visible trim marker,
and a test pins that envelope;
- `callerOrigin`, `scope.*` and `resourceAttributes.*` are kept as they arrive;
- the docstrings name every departure from live, including the error row's
dropped `resourceAttributes.process.*` / `.host.*` families.
Also: `isErrorOutput` now requires `domain` and `category` alongside `name` and
an `id`/`cause`, which all eight live MastraErrors carry, so a user-controlled
record shaped `{name:'Error', id}` is not read as a failure. Documented that one
TOOL observation is one step, so parallel calls count as N, and reworded
`planEfficiency` and the README to say tool calls rather than steps.
RobertCrupa
added this pull request to stack #6
September 9, 2026 15:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes apify/ai-team#269. Part of apify/ai-team#249, stacked on the #267 PR.
Verified live against langfuse.apify.dev on 2026-09-09 (staging traces): the export shape, the two-generation pattern, the TOOL input/output mapping, the error shape and the `sessionId` discriminator. `test/fixtures/live-trace.ts` holds the real rows. Not yet verifiable: the trace-contract metadata path (`feat/trace-contract` is not deployed).
Known, agent-side: the agent's `toolSchemaHash` currently ignores schema content (Mastra wraps `inputSchema` in functions the hash drops), so `schemaMatch` will read false until apify-ai-agent hashes the raw JSON schema. Validation still runs against live schemas.