From 0993785ba0f6e4afaeff127f7b7079e9f1076bcc Mon Sep 17 00:00:00 2001 From: "warp-agent-staging[bot]" <240773466+warp-agent-staging[bot]@users.noreply.github.com> Date: Wed, 26 Aug 2026 22:04:46 +0000 Subject: [PATCH 1/4] docs(factories): editorial pass on scorer documentation Align the scorer mental model and terminology across the Factories section: classify (not grade) completed runs (not conversations or completed work), consolidate the duplicate scorer/benchmark explanation on the dashboard page into a pointer to the canonical measure-and-improve page, and add a Scorer terminology.md entry. Co-Authored-By: Warp --- .agents/references/terminology.md | 3 +++ .../docs/factories/factory-dashboard.mdx | 4 +--- .../docs/factories/how-factories-work.mdx | 2 +- src/content/docs/factories/index.mdx | 2 +- .../docs/factories/measure-and-improve.mdx | 18 +++++++++--------- 5 files changed, 15 insertions(+), 14 deletions(-) diff --git a/.agents/references/terminology.md b/.agents/references/terminology.md index 1520dce2c..88a5f4f5b 100644 --- a/.agents/references/terminology.md +++ b/.agents/references/terminology.md @@ -334,6 +334,9 @@ Docs match the screen; the fix belongs in the app. - **factory dashboard** — The web app surface for operating a single factory: its work items, runs, agents, automations, and settings. *Usage note:* Lowercase common noun. Distinct from **Dashboard**, the metrics page inside it, which is also the factory's landing page — bold **Dashboard** when you mean that page, and leave "factory dashboard" unbolded when you mean the surface. Replaced "control room," a docs-only coinage that appeared nowhere in the product. +- **Scorer** — A configured LLM judge that classifies completed runs against criteria you write, such as "did the agent run the tests before opening a PR?" A Scorer assigns a label (a classification with a score), not a freeform numeric grade. + *Usage note:* Capitalize "Scorer"/"Scorers" when referring to the feature or a configured instance ("create a Scorer," "Scorer cards"); lowercase only for a generic instance count or file listing ("two scorers," alongside "skills" in an example tree). Say "classify," never "grade" — the docs draw this distinction deliberately. The unit a Scorer evaluates is a **run** (a single agent execution), not a "conversation" or "completed work." `measure-and-improve.mdx` is the canonical page for what a Scorer is and how to configure one; other pages link there rather than repeating the definition. + - **AI sovereignty** — Warp Factories' positioning around customer ownership and control of inference, hosting, and data exhaust (agent conversations, evals, memories) for their factory. ## Technical terms diff --git a/src/content/docs/factories/factory-dashboard.mdx b/src/content/docs/factories/factory-dashboard.mdx index 6a94a23ea..2c49e8b82 100644 --- a/src/content/docs/factories/factory-dashboard.mdx +++ b/src/content/docs/factories/factory-dashboard.mdx @@ -88,9 +88,7 @@ When an agent proposes a change to a Warp-managed definition, its work item on * ## Score and benchmark -A Scorer is an LLM judge that classifies completed runs against a rubric you define, scoped to the agents you choose and sampled at a rate you set. Each Scorer has a **Self-improvement** toggle: when it's on, the factory periodically triages the runs that Scorer marks as failing and files fixes for recurring issues. The **Self-improvement** page tracks the pull requests those fixes open, linking each one to the run that produced it. - -**Benchmarks** compares harness, model, and runner configurations against a fixed set of tasks with success criteria. Benchmark results are only as reliable as the Scorers behind them, so read them alongside [Measure and improve a factory](/factories/measure-and-improve/). +Scorer cards on the **Dashboard** page and the **Self-improvement** page surface your Scorer results; **Benchmarks** compares harness, model, and runner configurations against a fixed set of tasks. See [Configure Scorers](/factories/measure-and-improve/#configure-scorers) and [Compare configurations with benchmarks](/factories/measure-and-improve/#compare-configurations-with-benchmarks) for what each one is and how to set it up. ## Change factory settings diff --git a/src/content/docs/factories/how-factories-work.mdx b/src/content/docs/factories/how-factories-work.mdx index ba5f59894..d60fd5875 100644 --- a/src/content/docs/factories/how-factories-work.mdx +++ b/src/content/docs/factories/how-factories-work.mdx @@ -63,7 +63,7 @@ The first two are workflow policy, written into the foreman's instructions; edit ## How the factory improves itself -Your factory is self-improving, and you define what "better" means. [Scorers](/factories/measure-and-improve/) grade completed runs against criteria you write, and [Self-improvement](/factories/measure-and-improve/#configure-and-review-self-improvement) groups the failures they flag into follow-up runs that propose fixes — to the application code or to the factory's own definition. Every proposal arrives as a change for your review; nothing is adopted on its own. +Your factory is self-improving, and you define what "better" means. [Scorers](/factories/measure-and-improve/) classify completed runs against criteria you write, and [Self-improvement](/factories/measure-and-improve/#configure-and-review-self-improvement) groups the failures they flag into follow-up runs that propose fixes — to the application code or to the factory's own definition. Every proposal arrives as a change for your review; nothing is adopted on its own. The factory's definition is open to the same loop. Anyone on the team, or an agent, can propose changes to its instructions, skills, models, or other [definition files](/factories/factory-as-code/), and definitions stored in GitHub go through pull request review and [configuration checks](/factories/factory-as-code/#pull-request-checks-for-github-backed-factories) before a change reaches the production branch. diff --git a/src/content/docs/factories/index.mdx b/src/content/docs/factories/index.mdx index 2dcbb6cfa..55153c7c2 100644 --- a/src/content/docs/factories/index.mdx +++ b/src/content/docs/factories/index.mdx @@ -39,7 +39,7 @@ Warp Factories is designed for engineering teams with repeatable work that exten * **Definitions as code** - [Version-controlled definition files](/factories/factory-as-code/) describe your repositories, agents, automations, runners, skills, and MCP servers, so factory changes get the same review, history, and rollback as code changes. * **Integrations and the Factory MCP** - Work flows in from [Slack](/factories/integrations/slack/), [GitHub](/factories/integrations/github/), [GitLab](/factories/integrations/gitlab/), [Linear](/factories/integrations/linear/), and [Jira](/factories/integrations/jira/), plus direct runs and schedules. The [Factory MCP](/factories/factory-mcp/) connects coding agents and other MCP clients. * **Model and harness choice** - Each agent can use a different model and [supported harness](/platform/harnesses/), including the Warp Agent, Claude Code, and Codex. -* **Measurement and self-improvement** - The [factory dashboard](/factories/factory-dashboard/) shows work-item status, runs, automations, costs, and benchmarks. [Scorers](/factories/measure-and-improve/) grade completed work, and [Self-improvement](/factories/measure-and-improve/#configure-and-review-self-improvement) turns repeated failures into follow-up work the factory proposes for review. +* **Measurement and self-improvement** - The [factory dashboard](/factories/factory-dashboard/) shows work-item status, runs, automations, costs, and benchmarks. [Scorers](/factories/measure-and-improve/) classify completed runs, and [Self-improvement](/factories/measure-and-improve/#configure-and-review-self-improvement) turns repeated failures into follow-up work the factory proposes for review. * **Infrastructure control** - Run on Warp-hosted infrastructure, or self-host execution on an eligible Enterprise plan. Teams can also connect supported inference providers, scope secrets, and (if eligible) store transcripts, artifacts, and run attachments in their own S3 or GCS buckets. See [infrastructure and security](/factories/infrastructure-and-security/) for the available controls. ## How Warp Factories relates to other Warp products diff --git a/src/content/docs/factories/measure-and-improve.mdx b/src/content/docs/factories/measure-and-improve.mdx index d07c50666..f968da777 100644 --- a/src/content/docs/factories/measure-and-improve.mdx +++ b/src/content/docs/factories/measure-and-improve.mdx @@ -1,7 +1,7 @@ --- title: Measure and improve a factory description: >- - Measure factory activity and costs, evaluate completed conversations, compare + Measure factory activity and costs, evaluate completed runs, compare agent configurations, and turn failures into follow-up work. sidebar: label: "Measure and improve" @@ -12,7 +12,7 @@ Warp Factories tracks what your factory produces and how well it performs, so yo | Feature | What it tells you | | --- | --- | | Dashboard metrics | How much work the factory produced, and what it cost. | -| Scorers | Whether completed conversations meet criteria you define. | +| Scorers | Whether completed runs meet criteria you define. | | Benchmarks | How different configurations perform on the same tasks. | | Self-improvement | Which repeated failures get investigated and turned into follow-up work. | @@ -42,7 +42,7 @@ Use the **Dashboard** page to pick which runs to investigate, not to conclude wh ## Configure Scorers -A **Scorer** uses an LLM judge to classify completed conversations against criteria you write, such as "did the agent run the tests before opening a PR?" Scorers classify conversations rather than grading them on a numeric scale. Keep each Scorer focused on one question so its failures point to a specific fix. For Scorers defined as files in a factory definition, see the two scorers in [`02-sdlc-issue-to-pr`](https://github.com/warpdotdev/warp-factory-examples/tree/main/examples/02-sdlc-issue-to-pr) in the [warp-factory-examples](https://github.com/warpdotdev/warp-factory-examples) repository. +A **Scorer** uses an LLM judge to classify completed runs against criteria you write — for example, "did the agent run the tests before opening a PR?" It assigns a label, not a numeric grade, so keep each Scorer focused on one question its failures can point back to. For Scorers defined as files in a factory definition, see the two scorers in [`02-sdlc-issue-to-pr`](https://github.com/warpdotdev/warp-factory-examples/tree/main/examples/02-sdlc-issue-to-pr) in the [warp-factory-examples](https://github.com/warpdotdev/warp-factory-examples) repository. Configure these fields: @@ -50,14 +50,14 @@ Configure these fields: * **Judge instructions** - The criteria the judge checks for. * **Judge model** - The model that acts as the judge. * **Classifications** - The labels the judge can assign, each with a score. -* **Pass threshold** - The score a conversation needs to pass. -* **Sample rate** - The share of the selected agents' completed conversations to evaluate. +* **Pass threshold** - The score a run needs to pass. +* **Sample rate** - The share of the selected agents' completed runs to evaluate. {/* VISUAL: The Scorer create/edit form (judge instructions, judge model, classifications, pass threshold, sample rate). */} -While the sample rate is above 0, scoring runs automatically: shortly after a sampled conversation completes, the judge evaluates it and records a classification, a score, and its reasoning. To stop automatic scoring, set the sample rate to 0. +While the sample rate is above 0, scoring happens automatically: shortly after a sampled run completes, the judge evaluates it and records a classification, a score, and its reasoning. To stop automatic scoring, set the sample rate to 0. -You can also score any single conversation on demand, which is useful for testing new judge instructions before raising the sample rate. Scoring a conversation again replaces its previous result from that Scorer. +You can also score any single run on demand, which is useful for testing new judge instructions before raising the sample rate. Scoring a run again replaces its previous result from that Scorer. Changing **Pass threshold** updates how past scores display as pass or fail; the recorded results don't change. @@ -99,9 +99,9 @@ flowchart LR Improve -.-> Adopt ``` -1. **Define a Scorer.** Pick one agent and one failure mode you can observe. Write the judge instructions and classifications, then score a few conversations manually and compare the judge's results against your own review. +1. **Define a Scorer.** Pick one agent and one failure mode you can observe. Write the judge instructions and classifications, then score a few runs manually and compare the judge's results against your own review. 2. **Collect a baseline.** Let automatic scoring run until results reflect normal work. Record the Scorer settings, date range, and relevant costs. -3. **Inspect failures.** Read the judge's reasoning and the underlying conversations. Look for causes like missing context, unclear instructions, or missing tools. Turn on Self-improvement when the same failure keeps repeating. +3. **Inspect failures.** Read the judge's reasoning and the underlying runs. Look for causes like missing context, unclear instructions, or missing tools. Turn on Self-improvement when the same failure keeps repeating. 4. **Benchmark a candidate.** Compare configurations of that agent on the same tasks, with enough repetitions to trust the difference. 5. **Review and adopt.** If the evidence supports the change, make it. Review Self-improvement pull requests with the same standards as human-authored ones. 6. **Keep monitoring.** Leave the Scorer active and compare new results against your baseline. Revise the Scorer, or set its sample rate to 0, when its criteria no longer match what your team needs. From f3b974d669db035437d86cd9434c4371c0b7fdea Mon Sep 17 00:00:00 2001 From: "warp-agent-staging[bot]" <240773466+warp-agent-staging[bot]@users.noreply.github.com> Date: Wed, 26 Aug 2026 22:20:14 +0000 Subject: [PATCH 2/4] docs(factories): absorb PR #584's scorers//scorer.md syntax Fold in PR #584's factory-as-code.mdx addition (definitions-as-code syntax for scorers//scorer.md) so QUALITY-1907 ships one consolidated PR per the requester's follow-up, rather than two overlapping ones. Align its wording with the terminology already standardized in this PR (it already said classify/run, so no verb or unit changes were needed), add the missing description field reference, fix a missing blank line before the passingScore heading, and cross-link it from measure-and-improve.mdx and vice versa. Co-Authored-By: Warp --- .../docs/factories/factory-as-code.mdx | 65 ++++++++++++++++++- .../docs/factories/measure-and-improve.mdx | 2 +- 2 files changed, 65 insertions(+), 2 deletions(-) diff --git a/src/content/docs/factories/factory-as-code.mdx b/src/content/docs/factories/factory-as-code.mdx index ca996e9ed..8264e3fb0 100644 --- a/src/content/docs/factories/factory-as-code.mdx +++ b/src/content/docs/factories/factory-as-code.mdx @@ -2,7 +2,7 @@ title: Factory definition syntax description: >- Look up every file and key in a factory definition: factory.yaml, agents, - automations, runners, and skills. + automations, runners, scorers, and skills. sidebar: label: "Definitions as code" --- @@ -46,6 +46,9 @@ automations/ automation.md runners/ linux-build.yaml +scorers/ + tests-run/ + scorer.md skills/ repository-conventions/ SKILL.md @@ -319,6 +322,66 @@ Optional. The compute size, as `vcpus` and `memoryGb`. Omit it to take the works The operating system and architecture. `os` is `linux` (the default) or `macos`, and `arch` is `x86_64` (the default on Linux) or `aarch64` (the only option on macOS). Linux runners require `linux.dockerImage`, the container image the sandbox boots, so every Linux runner declares a `platform` section. macOS runners accept an optional `mac.version` (`"14"`, `"15"`, `"26"`, or `"27"`; quote it, and it defaults to `"26"`). +## `scorers//scorer.md` + +Optional. Each file defines a scorer: an LLM judge that classifies a sample of an agent's finished runs against a rubric. The directory segment is only a stable filesystem slug — the required `name` field is the scorer's identity. The YAML frontmatter declares the classification contract, and the Markdown body after the closing `---` fence is the rubric. See [Configure Scorers](/factories/measure-and-improve/#configure-scorers) for how scores are used. + +```markdown title="scorers/tests-run/scorer.md" +--- +name: tests-run +description: Checks whether implementation runs include test evidence. +agents: + - reviewer +labels: + - value: tests_run + description: The transcript contains a test command and its result. + score: 1 + - value: tests_skipped + score: 0 +passingScore: 1 +samplingRate: 25 +model: claude-4-5-haiku +--- +Evaluate whether the agent ran the relevant tests before finishing. Return +exactly one declared label. +``` + +### `name` + +Required. The scorer's identity. Renaming it is a content edit, not a directory move. + +### `description` + +Optional. A short summary of what the scorer checks. + +### `agents` + +Required. The agents (by name) whose runs this scorer evaluates. + +### `output` + +Optional. The scorer output form. `classification` is the current supported value. + +### `labels` + +Required. The classifications the judge may return, each with a `value`, a numeric `score` from 0 through 1, and an optional `description`. At least one label must score at or above `passingScore` and at least one below it. + +### `passingScore` + +Required. The threshold, from 0 through 1, at or above which a run counts as passing. + +### `samplingRate` + +Optional. The percentage of eligible runs to score. Defaults to 25. + +### `model` + +Required. The model that judges the runs. + +### `selfImprovement` + +Optional. When `true`, failing scores can feed the factory's self-improvement flow, which proposes definition changes as pull requests. Defaults to `false`. + ## Skills A skill is a directory containing a `SKILL.md`, not a YAML key. Skills under `skills/` are available to every agent in the factory; skills under `agents//skills/` are available only to that agent. See [Skills for agents](/agents/capabilities/skills/). diff --git a/src/content/docs/factories/measure-and-improve.mdx b/src/content/docs/factories/measure-and-improve.mdx index f968da777..27c629308 100644 --- a/src/content/docs/factories/measure-and-improve.mdx +++ b/src/content/docs/factories/measure-and-improve.mdx @@ -42,7 +42,7 @@ Use the **Dashboard** page to pick which runs to investigate, not to conclude wh ## Configure Scorers -A **Scorer** uses an LLM judge to classify completed runs against criteria you write — for example, "did the agent run the tests before opening a PR?" It assigns a label, not a numeric grade, so keep each Scorer focused on one question its failures can point back to. For Scorers defined as files in a factory definition, see the two scorers in [`02-sdlc-issue-to-pr`](https://github.com/warpdotdev/warp-factory-examples/tree/main/examples/02-sdlc-issue-to-pr) in the [warp-factory-examples](https://github.com/warpdotdev/warp-factory-examples) repository. +A **Scorer** uses an LLM judge to classify completed runs against criteria you write — for example, "did the agent run the tests before opening a PR?" It assigns a label, not a numeric grade, so keep each Scorer focused on one question its failures can point back to. For Scorers defined as files in a factory definition, see the [`scorers//scorer.md` syntax](/factories/factory-as-code/#scorersnamescorermd) and the two scorers in [`02-sdlc-issue-to-pr`](https://github.com/warpdotdev/warp-factory-examples/tree/main/examples/02-sdlc-issue-to-pr) in the [warp-factory-examples](https://github.com/warpdotdev/warp-factory-examples) repository. Configure these fields: From dd34b74181992953bb9941e9a539e16bad049e26 Mon Sep 17 00:00:00 2001 From: "warp-agent-staging[bot]" <240773466+warp-agent-staging[bot]@users.noreply.github.com> Date: Wed, 26 Aug 2026 23:03:59 +0000 Subject: [PATCH 3/4] fix(check_for_broken_links): don't strip placeholder text from backtick headings slugify_heading() approximates the anchor ids Starlight's github-slugger emits, but it stripped -style placeholders inside backtick code spans as if they were real HTML/JSX tags. A code span's content renders as literal text, so `scorers//scorer.md` keeps "name" in its visible heading text and its real anchor id -- the checker's old approximation diverged from the actual Astro build, both producing a false positive on a correct link (this PR's new #scorersnamescorermd cross-link) and silently missing a real one (factory-as-code.mdx's pre-existing #runnersyaml link, fixed alongside this). Co-Authored-By: Warp --- .../check_for_broken_links/check_links.py | 15 ++++++++- .../test_check_links.py | 31 ++++++++++++++++++- .../docs/factories/factory-as-code.mdx | 2 +- 3 files changed, 45 insertions(+), 3 deletions(-) diff --git a/.agents/skills/check_for_broken_links/check_links.py b/.agents/skills/check_for_broken_links/check_links.py index 4a5af7fd6..224d55572 100644 --- a/.agents/skills/check_for_broken_links/check_links.py +++ b/.agents/skills/check_for_broken_links/check_links.py @@ -81,10 +81,23 @@ def slugify_heading(text): """Approximate github-slugger, which is what Starlight uses for anchor ids.""" text = re.sub(r'\{[^}]*\}', '', text) # MDX expressions, e.g. {VARS.X} + + # A code span's content renders as literal text -- markdown never parses + # `` inside backticks as an HTML/JSX tag, so a heading like + # `` ## `scorers//scorer.md` `` keeps "name" in its visible text and + # thus its anchor id. Stash code-span content before the inline-HTML strip + # below so that strip only touches real markup, then restore it verbatim. + code_spans = [] + + def _stash_code(match): + code_spans.append(match.group(1)) + return f'\x00{len(code_spans) - 1}\x00' + + text = re.sub(r'`([^`]*)`', _stash_code, text) # code spans (stashed) text = re.sub(r'<[^>]+>', '', text) # inline HTML/JSX - text = re.sub(r'`([^`]*)`', r'\1', text) # code spans text = re.sub(r'\[([^\]]*)\]\([^)]*\)', r'\1', text) # links keep their text text = re.sub(r'[*_]{1,3}', '', text) # emphasis + text = re.sub(r'\x00(\d+)\x00', lambda m: code_spans[int(m.group(1))], text) text = text.strip().lower() text = re.sub(r'[^\w\- ]+', '', text, flags=re.UNICODE) return text.replace(' ', '-') diff --git a/.agents/skills/check_for_broken_links/test_check_links.py b/.agents/skills/check_for_broken_links/test_check_links.py index 88f181a61..da82f709a 100644 --- a/.agents/skills/check_for_broken_links/test_check_links.py +++ b/.agents/skills/check_for_broken_links/test_check_links.py @@ -37,6 +37,29 @@ ] +# (description, heading text as it appears after '## ', expected slug) +# A code span's content renders as literal text -- markdown never parses +# `` inside backticks as an HTML/JSX tag, so these placeholders must +# stay in the anchor id instead of being stripped like real markup. +SLUG_CASES = [ + ( + "backtick heading with an angle-bracket placeholder keeps the placeholder text", + "`scorers//scorer.md`", + "scorersnamescorermd", + ), + ( + "a different backtick+placeholder heading keeps its own placeholder text too", + "`agents//agent.md`", + "agentsnameagentmd", + ), + ( + "plain-text heading is unaffected by the code-span handling", + "Configure Scorers", + "configure-scorers", + ), +] + + def main() -> int: failures = 0 @@ -59,7 +82,13 @@ def main() -> int: failures += 0 if ok else 1 print(f" [{'PASS' if ok else 'FAIL'}] {description:<58} broken={broken_urls}") - total = len(CASES) + 1 + for description, heading, expected_slug in SLUG_CASES: + slug = check_links.slugify_heading(heading) + ok = slug == expected_slug + failures += 0 if ok else 1 + print(f" [{'PASS' if ok else 'FAIL'}] {description:<58} slug={slug!r}") + + total = len(CASES) + len(SLUG_CASES) + 1 print() if failures: print(f"{failures} of {total} cases regressed.") diff --git a/src/content/docs/factories/factory-as-code.mdx b/src/content/docs/factories/factory-as-code.mdx index 8264e3fb0..88e9a83ef 100644 --- a/src/content/docs/factories/factory-as-code.mdx +++ b/src/content/docs/factories/factory-as-code.mdx @@ -185,7 +185,7 @@ For a definition that runs a different harness per agent, with managed-secret au ### `agentDefaults.runner` -The name of a runner defined under [`runners/`](#runnersyaml) that provides the compute for runs. +The name of a runner defined under [`runners/`](#runnersnameyaml) that provides the compute for runs. ### `agentDefaults.environmentId` From aaa41d095c639a24bbfdc968fcf43b1fecefa76f Mon Sep 17 00:00:00 2001 From: hongyi-chen Date: Thu, 27 Aug 2026 00:07:25 +0000 Subject: [PATCH 4/4] Address review: scorer agents format, Scorers vs Self-improvement surfaces - factory-as-code: document agents as a list of one or more agent names, cross-link the agents/ section, and add scorers/skills to the intro file list - factory-dashboard: Scorer results live on the Scorers page; Self-improvement lists the pull requests the flow opens after analyzing failing runs - measure-and-improve: name the Scorers and Self-improvement pages so both sections say where to configure and review each feature Co-Authored-By: Warp --- src/content/docs/factories/factory-as-code.mdx | 4 ++-- src/content/docs/factories/factory-dashboard.mdx | 2 +- src/content/docs/factories/measure-and-improve.mdx | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/src/content/docs/factories/factory-as-code.mdx b/src/content/docs/factories/factory-as-code.mdx index 88e9a83ef..cb64a42a6 100644 --- a/src/content/docs/factories/factory-as-code.mdx +++ b/src/content/docs/factories/factory-as-code.mdx @@ -8,7 +8,7 @@ sidebar: --- import { VARS } from '@data/vars'; -Every factory is defined by files: a `factory.yaml` plus directories of agents, automations, and runners, versioned in a Git repository. The files are the source of truth — when they change, Warp updates the factory to match. This page describes every file and key in a definition. +Every factory is defined by files: a `factory.yaml` plus directories of agents, automations, runners, scorers, and skills, versioned in a Git repository. The files are the source of truth — when they change, Warp updates the factory to match. This page describes every file and key in a definition. Definition files are YAML and Markdown. Keys are case-sensitive. @@ -356,7 +356,7 @@ Optional. A short summary of what the scorer checks. ### `agents` -Required. The agents (by name) whose runs this scorer evaluates. +Required. The agents whose runs this scorer evaluates, as a list of one or more agent names. Each name matches an agent defined under [`agents/`](#agentsnameagentmd). ### `output` diff --git a/src/content/docs/factories/factory-dashboard.mdx b/src/content/docs/factories/factory-dashboard.mdx index 2c49e8b82..92be33d9a 100644 --- a/src/content/docs/factories/factory-dashboard.mdx +++ b/src/content/docs/factories/factory-dashboard.mdx @@ -88,7 +88,7 @@ When an agent proposes a change to a Warp-managed definition, its work item on * ## Score and benchmark -Scorer cards on the **Dashboard** page and the **Self-improvement** page surface your Scorer results; **Benchmarks** compares harness, model, and runner configurations against a fixed set of tasks. See [Configure Scorers](/factories/measure-and-improve/#configure-scorers) and [Compare configurations with benchmarks](/factories/measure-and-improve/#compare-configurations-with-benchmarks) for what each one is and how to set it up. +**Scorers** is where you create Scorers and read their results. **Self-improvement** lists the pull requests the self-improvement flow opens after analyzing runs your Scorers mark as failing, and **Benchmarks** compares harness, model, and runner configurations against a fixed set of tasks. See [Configure Scorers](/factories/measure-and-improve/#configure-scorers), [Configure and review Self-improvement](/factories/measure-and-improve/#configure-and-review-self-improvement), and [Compare configurations with benchmarks](/factories/measure-and-improve/#compare-configurations-with-benchmarks) for what each one is and how to set it up. ## Change factory settings diff --git a/src/content/docs/factories/measure-and-improve.mdx b/src/content/docs/factories/measure-and-improve.mdx index 27c629308..f9f21c893 100644 --- a/src/content/docs/factories/measure-and-improve.mdx +++ b/src/content/docs/factories/measure-and-improve.mdx @@ -42,7 +42,7 @@ Use the **Dashboard** page to pick which runs to investigate, not to conclude wh ## Configure Scorers -A **Scorer** uses an LLM judge to classify completed runs against criteria you write — for example, "did the agent run the tests before opening a PR?" It assigns a label, not a numeric grade, so keep each Scorer focused on one question its failures can point back to. For Scorers defined as files in a factory definition, see the [`scorers//scorer.md` syntax](/factories/factory-as-code/#scorersnamescorermd) and the two scorers in [`02-sdlc-issue-to-pr`](https://github.com/warpdotdev/warp-factory-examples/tree/main/examples/02-sdlc-issue-to-pr) in the [warp-factory-examples](https://github.com/warpdotdev/warp-factory-examples) repository. +A **Scorer** uses an LLM judge to classify completed runs against criteria you write — for example, "did the agent run the tests before opening a PR?" It assigns a label, not a numeric grade, so keep each Scorer focused on one question its failures can point back to. Create Scorers on the factory dashboard's **Scorers** page, which also holds each Scorer's results. For Scorers defined as files in a factory definition, see the [`scorers//scorer.md` syntax](/factories/factory-as-code/#scorersnamescorermd) and the two scorers in [`02-sdlc-issue-to-pr`](https://github.com/warpdotdev/warp-factory-examples/tree/main/examples/02-sdlc-issue-to-pr) in the [warp-factory-examples](https://github.com/warpdotdev/warp-factory-examples) repository. Configure these fields: @@ -81,7 +81,7 @@ Every benchmark also runs **Correctness**, a built-in Scorer that marks each tri Turn on **Self-improvement** for each Scorer whose failures you want investigated automatically. Self-improvement groups related failures and files follow-up tasks as ordinary agent runs. A follow-up run can propose changes to application code. It can also improve the factory itself: when you manage your factory as [definitions as code](/factories/factory-as-code/), its prompts, skills, and configuration are version-controlled files, so a follow-up run can open a pull request against the factory definition the same way it would against application code. Nothing is adopted without your review. -Each Self-improvement pull request includes a **Regressions addressed** section that links the failing runs and Scorer results behind it, so you can trace the change back to its evidence. +The factory dashboard's **Self-improvement** page lists the pull requests these follow-up runs open. Each pull request includes a **Regressions addressed** section that links the failing runs and Scorer results behind it, so you can trace the change back to its evidence. ## Run a practical improvement loop