feat: eval trace contract package - #7
Open
RobertCrupa wants to merge 3 commits into
Open
RobertCrupa wants to merge 3 commits into
RobertCrupa wants to merge 3 commits into
Conversation
Introduces the shared trace contract (JSON Schemas + ajv) and deterministic check engine that both the workflow-runner and judge Actors depend on, plus the workspace plumbing (tsconfig, deploy script, CI) needed to build it.
apify-service-account
added
the
tested
workspace.file compiled globs with two chained replaces, so the second (`*` -> `[^/]*`) reprocessed the `.*` the first (`**` -> `.*`) had just inserted: `**/*.json` became a regex with exactly one literal `/` and matched neither `a/b/c.json` nor `c.json`. Compile the glob in a single pass instead, with `**/` also matching zero directories. answer.grounded pooled every number from the agent's own tool-call inputs, so a fabricated figure the agent passed to an Actor or echoed in a Bash command grounded itself, and the `applicable: false` path was practically unreachable. Ground only against dataset and reference items, as the check's contract says. reference's itemsOverlap and countWithinPct flattened every dataset the session touched, ignoring the check's `actor` scoping that apify.items honours, so a wrong Actor the agent tried first failed the comparison. Regression tests cover all three.
The agent-evals job and the root scenarios scripts name `actors/judge`, `actors/workflow-runner` and `tools`, which arrive in later PRs. npm errors on `npm -w tools run scenarios:check`, and worse, silently no-ops unresolvable `-w` flags in a multi-workspace `npm run`, so the build and test steps passed while exercising `contract` alone. Select the workspaces that are actually checked out before building and testing, and route the scenarios scripts through `--workspaces --if-present`. Both grow to cover the judge, the runner and tools as those workspaces land, with no further CI change.
RobertCrupa
added this pull request to stack #12
September 21, 2026 13:26
artogahr
approved these changes
Sep 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the trace contract package (`contract/`): JSON Schemas + ajv validation and the deterministic check engine, shared by both the workflow-runner and judge Actors. Also includes the workspace plumbing needed to build it (tsconfig base, deploy script, CI workflow) and doc updates in CLAUDE.md.
No runtime behavior yet — this is the foundation the next PRs in the stack build on.
🤖 Generated with Claude Code