Add agent-factory example: the output gate to SageRoute's run gate - #1
Add agent-factory example: the output gate to SageRoute's run gate#1codejunkie99 wants to merge 1 commit into
Conversation
SageRoute decides whether a run is still worth paying for. This example is the other half: a line that turns a job description into a certified agent, and gates every output that agent produces. Both halves call Sage. The router asks "does this run need intervention" mid-task; the factory asks "is this output right" after the answer. Same primitive, two moments. Six files, 846 lines, standard library only. Runs with no keys at all (deterministic backend, every run flags to a human and says why). The parts worth reading: - broker.py enforces tool grants outside the model, so a poisoned ticket reaching for billing:refund dies at the boundary, not in the prompt - prove runs with dry=True; an agent under test that can write to the suite it is judged against is not being tested - certify refuses without a sealed run, under the pass bar, or if the ABOM changed after the run it was proved against - restamp propagates a master fix and revokes every certificate stamped from the old one - harvest runs a certified builder agent that writes the next eval cases, into proposed.jsonl only; promoting one stays a human edit Includes four SVG diagrams and a README documenting the seven guards, each a reproducible exit-1.
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
| "use `factory.py harvest <agent>` instead of `run evalsmith`." | ||
| ) | ||
| raw, backend = ctx.llm(PROMPT.replace("{run}", json.dumps(record)), offline=_offline) | ||
| try: |
There was a problem hiding this comment.
🟡 Medium agents/evalsmith.py:48
When the model returns syntactically valid JSON that isn't an object — such as [] — json.loads succeeds and case.setdefault("expect", {}) throws AttributeError because setdefault is called on a list. The except only catches JSON parsing errors, so this exception aborts the run instead of falling back to _offline. Similarly, {} parses fine and becomes {"expect": {}}, which is sent to evals:propose and returned with an empty input and empty expectations — an unusable proposal silently enters the catalog. Consider validating that case is a dict with the required fields inside the try block, so schema failures fall back to _offline just like parse failures do.
Also found in 1 other location(s)
examples/agent-factory/agents/triager.py:55
The fallback only covers JSON extraction/parsing, not schema validation. If the model returns syntactically valid JSON such as
{}or omitslabel,draft, orescalate, parsing succeeds and the later direct indexing raisesKeyError; the triage run aborts instead of using_offlineas intended.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @examples/agent-factory/agents/evalsmith.py around line 48:
When the model returns syntactically valid JSON that isn't an object — such as `[]` — `json.loads` succeeds and `case.setdefault("expect", {})` throws `AttributeError` because `setdefault` is called on a list. The `except` only catches JSON parsing errors, so this exception aborts the run instead of falling back to `_offline`. Similarly, `{}` parses fine and becomes `{"expect": {}}`, which is sent to `evals:propose` and returned with an empty `input` and empty expectations — an unusable proposal silently enters the catalog. Consider validating that `case` is a dict with the required fields inside the `try` block, so schema failures fall back to `_offline` just like parse failures do.
Also found in 1 other location(s):
- examples/agent-factory/agents/triager.py:55 -- The fallback only covers JSON extraction/parsing, not schema validation. If the model returns syntactically valid JSON such as `{}` or omits `label`, `draft`, or `escalate`, parsing succeeds and the later direct indexing raises `KeyError`; the triage run aborts instead of using `_offline` as intended.
| "issues:comment", | ||
| "billing:refund" | ||
| ], | ||
| "knowledge": "docs/support-playbook.md", |
There was a problem hiding this comment.
🟡 Medium masters/triager.json:18
The knowledge field is inert: agents.triager:run constructs its prompt from a hard-coded PROMPT, and no code loads or injects docs/support-playbook.md (or any knowledge value) into the agent context at invocation time. The shipped triager therefore ignores its configured support playbook, and variants created via --set knowledge=... silently inherit the same behavior—producing triage decisions that can contradict the intended playbook. Either wire the knowledge file into the agent context at runtime or document that the field is currently unimplemented.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @examples/agent-factory/masters/triager.json around line 18:
The `knowledge` field is inert: `agents.triager:run` constructs its prompt from a hard-coded `PROMPT`, and no code loads or injects `docs/support-playbook.md` (or any `knowledge` value) into the agent context at invocation time. The shipped triager therefore ignores its configured support playbook, and variants created via `--set knowledge=...` silently inherit the same behavior—producing triage decisions that can contradict the intended playbook. Either wire the `knowledge` file into the agent context at runtime or document that the field is currently unimplemented.
| {run}""" | ||
|
|
||
|
|
||
| def _offline(prompt): |
There was a problem hiding this comment.
🟡 Medium agents/evalsmith.py:22
_offline always sets expect.label to the rejected output's label and flips escalate, but a gate can flag a run because the label was wrong while escalation was already correct. For such a trace, the generated eval case canonizes the incorrect label and reverses a correct escalation expectation — promoting it would teach the suite the wrong behavior. The fallback assumes every flag means both fields are wrong. Consider only inverting the field the gate actually flagged, or document why blanket inversion is intended.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @examples/agent-factory/agents/evalsmith.py around line 22:
`_offline` always sets `expect.label` to the rejected output's label and flips `escalate`, but a gate can flag a run because the label was wrong while escalation was already correct. For such a trace, the generated eval case canonizes the incorrect label and reverses a correct escalation expectation — promoting it would teach the suite the wrong behavior. The fallback assumes every flag means both fields are wrong. Consider only inverting the field the gate actually flagged, or document why blanket inversion is intended.
| return out | ||
|
|
||
|
|
||
| def cmd_restamp(args): |
There was a problem hiding this comment.
🟡 Medium agent-factory/factory.py:132
cmd_restamp fails to propagate master fixes to existing variants: values that a variant inherited unchanged from the old master become treated as overrides as soon as the master is edited, so dict(master, **overrides, ...) writes those old values back over the fix. The restamped variant ends up identical to what it was before the master was changed.
The overrides dict on line 142 is computed by comparing each variant field against the already-updated master (old[k] != master.get(k)). Any field the variant inherited from the old master that the master edit changed will match this condition and be carried forward as an override, overwriting the new master value. To fix this, overrides must be identified explicitly (e.g., recorded at stamp time) or compared against the previous master, not the updated one.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @examples/agent-factory/factory.py around line 132:
`cmd_restamp` fails to propagate master fixes to existing variants: values that a variant inherited unchanged from the old master become treated as overrides as soon as the master is edited, so `dict(master, **overrides, ...)` writes those old values back over the fix. The restamped variant ends up identical to what it was before the master was changed.
The `overrides` dict on line 142 is computed by comparing each variant field against the *already-updated* master (`old[k] != master.get(k)`). Any field the variant inherited from the old master that the master edit changed will match this condition and be carried forward as an override, overwriting the new master value. To fix this, overrides must be identified explicitly (e.g., recorded at stamp time) or compared against the *previous* master, not the updated one.
| for i, (case, got) in enumerate(results): | ||
| missed = {k: {"want": v, "got": got.get(k)} | ||
| for k, v in case["expect"].items() if got.get(k) != v} | ||
| if i in scores: | ||
| drafts.append(scores[i]) | ||
| if scores[i] < rubric["min"]: | ||
| missed["draft"] = {"want": f">={rubric['min']}", "got": round(scores[i], 2)} | ||
| if missed: | ||
| failures.append({"id": case["id"], "input": case["input"], "missed": missed}) |
There was a problem hiding this comment.
🟠 High agent-factory/factory.py:212
cmd_prove does not treat missing draft rubric scores as failures. When sage.batch returns [] (no key or outage) or a partial response, no scores[i] entry is created for that case, so the case is judged only on exact-match expect fields and can pass. cmd_certify then accepts the resulting sealed score and certifies an agent whose required draft safety rubric was never evaluated. Printing UNSCORED to the console does not prevent certification because the record itself contains no signal that scoring was incomplete.
The issue is that a case with a draft but no score is neither added to failures nor counted as a draft failure. Consider treating an unscored draft as a missed draft expectation (e.g., recording missed["draft"] when rubric is set, the output contains a draft, and no score was returned), so the score drops below the pass bar and cmd_certify refuses to sign.
for i, (case, got) in enumerate(results):
missed = {k: {"want": v, "got": got.get(k)}
for k, v in case["expect"].items() if got.get(k) != v}
- if i in scores:
+ if rubric and isinstance(got.get("draft"), str) and i not in scores:
+ missed["draft"] = {"want": f">={rubric['min']}", "got": None}
+ elif i in scores:
drafts.append(scores[i])
if scores[i] < rubric["min"]:
missed["draft"] = {"want": f">={rubric['min']}", "got": round(scores[i], 2)}
if missed:🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @examples/agent-factory/factory.py around lines 212-220:
`cmd_prove` does not treat missing draft rubric scores as failures. When `sage.batch` returns `[]` (no key or outage) or a partial response, no `scores[i]` entry is created for that case, so the case is judged only on exact-match `expect` fields and can pass. `cmd_certify` then accepts the resulting sealed score and certifies an agent whose required draft safety rubric was never evaluated. Printing `UNSCORED` to the console does not prevent certification because the `record` itself contains no signal that scoring was incomplete.
The issue is that a case with a draft but no score is neither added to `failures` nor counted as a draft failure. Consider treating an unscored draft as a missed `draft` expectation (e.g., recording `missed["draft"]` when `rubric` is set, the output contains a draft, and no score was returned), so the score drops below the pass bar and `cmd_certify` refuses to sign.
| f.write(json.dumps(case) + "\n") | ||
|
|
||
|
|
||
| def invoke(abom, task, agent, target=None, dry=False): |
There was a problem hiding this comment.
🟠 High agent-factory/factory.py:95
invoke returns llm.LAST["cost_usd"] as the total cost, but llm.LAST is reset by llm.complete on every call — so when an agent's entrypoint calls ctx.llm more than once, only the final call's cost is reported and earlier calls' costs are silently discarded. cmd_run then compares this understated cost against cost_envelope_usd, so a multi-call run that exceeds the envelope can be allowed to act instead of being forced to a human. Consider accumulating cost per invocation (e.g., a callback that sums each complete result) rather than reading the global after the fact.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @examples/agent-factory/factory.py around line 95:
`invoke` returns `llm.LAST["cost_usd"]` as the total cost, but `llm.LAST` is reset by `llm.complete` on every call — so when an agent's entrypoint calls `ctx.llm` more than once, only the final call's cost is reported and earlier calls' costs are silently discarded. `cmd_run` then compares this understated cost against `cost_envelope_usd`, so a multi-call run that exceeds the envelope can be allowed to act instead of being forced to a human. Consider accumulating cost per invocation (e.g., a callback that sums each `complete` result) rather than reading the global after the fact.
| if not (REGISTRY / "evalsmith.card.json").exists(): | ||
| sys.exit("evalsmith is not certified. the line only hires from the registry.") |
There was a problem hiding this comment.
🟡 Medium agent-factory/factory.py:364
cmd_harvest checks only that evalsmith.card.json exists, but never rejects a recalled card or verifies that the card's abom_digest matches digest("evalsmith"). A recalled evalsmith, or one whose ABOM changed after certification, still runs and writes proposals — bypassing the revocation and tamper checks that cmd_run enforces. Consider applying the same recalled and abom_digest guards used in cmd_run before invoking the builder.
| if not (REGISTRY / "evalsmith.card.json").exists(): | |
| sys.exit("evalsmith is not certified. the line only hires from the registry.") | |
| if not (REGISTRY / "evalsmith.card.json").exists(): | |
| sys.exit("evalsmith is not certified. the line only hires from the registry.") | |
| card = json.loads((REGISTRY / "evalsmith.card.json").read_text()) | |
| if card.get("recalled"): | |
| sys.exit(f"evalsmith recalled: {card['recalled']}. re-certify before harvesting.") | |
| if card["abom_digest"] != digest("evalsmith"): | |
| sys.exit("evalsmith's ABOM changed since certification. re-prove and re-certify.") |
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @examples/agent-factory/factory.py around lines 364-365:
`cmd_harvest` checks only that `evalsmith.card.json` exists, but never rejects a recalled card or verifies that the card's `abom_digest` matches `digest("evalsmith")`. A recalled `evalsmith`, or one whose ABOM changed after certification, still runs and writes proposals — bypassing the revocation and tamper checks that `cmd_run` enforces. Consider applying the same `recalled` and `abom_digest` guards used in `cmd_run` before invoking the builder.
| if card["abom_digest"] != digest(args.agent): | ||
| sys.exit("the ABOM changed since certification. re-prove and re-certify.") | ||
|
|
||
| try: |
There was a problem hiding this comment.
🟠 High agent-factory/factory.py:297
cmd_run invokes the agent with live tools before checking the output gate or certificate tier, so an agent that fails the gate (or a C0/C1 agent) still executes every granted tool call with real side effects. The acted flag is set to false afterward, but the side effects have already happened — the gate and tier cannot prevent them, only label the run after the fact.
The invoke call on line 298 runs the agent with live tools (no dry=True), so tool calls like billing:refund or drafts:write take effect before the gate is evaluated. To prevent side effects from agents that should only observe or draft, either run the agent dry first and only re-invoke with live tools when the gate and tier permit action, or route tool calls through the broker based on the certificate tier.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @examples/agent-factory/factory.py around line 297:
`cmd_run` invokes the agent with live tools before checking the output gate or certificate tier, so an agent that fails the gate (or a `C0`/`C1` agent) still executes every granted tool call with real side effects. The `acted` flag is set to `false` afterward, but the side effects have already happened — the gate and tier cannot prevent them, only label the run after the fact.
The `invoke` call on line 298 runs the agent with live tools (no `dry=True`), so tool calls like `billing:refund` or `drafts:write` take effect before the gate is evaluated. To prevent side effects from agents that should only observe or draft, either run the agent dry first and only re-invoke with live tools when the gate and tier permit action, or route tool calls through the broker based on the certificate tier.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 765a5c4c8f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| overrides = {k: v for k, v in old.items() | ||
| if k in master and old[k] != master.get(k) and k not in keep} | ||
| new = dict(master, **keep, **overrides, master_digest=fresh) | ||
| (MASTERS / f"{name}.json").write_text(json.dumps(new, indent=2) + "\n") |
There was a problem hiding this comment.
When a master field is changed, every existing variant differs from the new master, so this comprehension misclassifies the variant's stale inherited value as an override and reapplies it over the updated master. For example, changing the master's gate_question and running restamp leaves every variant on the old question, defeating propagation of the fix; explicit overrides need to be recorded separately or compared with the version of the master from which the variant was stamped.
Useful? React with 👍 / 👎.
| try: | ||
| output, trace, cost, route = invoke(abom, args.input, args.agent) |
There was a problem hiding this comment.
When a broker implementation has real side effects, invoke executes those tools before the Sage result, cost envelope, or certificate tier is evaluated below. Consequently a failed gate, an over-budget run, or even a C0 "observe" certificate can already have performed its writes; evals:propose is an included persistent example. Tool calls need to be staged or otherwise constrained by the tier and committed only after the gate passes.
Useful? React with 👍 / 👎.
| failures.append({"id": case["id"], "input": case["input"], "missed": missed}) | ||
|
|
||
| scored = len(scores) | ||
| score = (len(cases) - len(failures)) / len(cases) |
There was a problem hiding this comment.
When SAGE_API_KEY is absent or batch scoring fails, scores is empty and no draft is added to missed, so exact-match outputs produce a score of 1.0 despite every draft being explicitly reported as UNSCORED. Because certification checks only this score, an agent can be certified without the prose-safety evaluation that the rubric is intended to enforce; missing scores for rubric-bearing cases must make the proof incomplete or failing.
Useful? React with 👍 / 👎.
| if not (REGISTRY / "evalsmith.card.json").exists(): | ||
| sys.exit("evalsmith is not certified. the line only hires from the registry.") |
There was a problem hiding this comment.
If evalsmith's card has been recalled or its ABOM has changed since certification, this existence-only check still accepts the card and harvest invokes the builder directly, bypassing the digest and recall checks in cmd_run. Thus a recalled or stale builder can continue writing proposals; harvest should apply the same certificate validation before invocation.
Useful? React with 👍 / 👎.
1 participant
SageRoute decides whether a run is still worth paying for. This adds the other half as a worked example: a line that turns a job description into a certified agent, and gates every output that agent produces.
Both halves call Sage. The router asks does this run need intervention mid-task; the factory asks is this output right after the answer. Same primitive, two moments.
Six files, 846 lines, standard library only. Runs with no keys at all — deterministic backend, and every run flags to a human and says why.
Worth reviewing
broker.pyenforces tool grants outside the model. A poisoned ticket reaching forbilling:refunddies at the boundary, not in the promptproveruns withdry=True— an agent under test that can write to the suite it is judged against is not being testedcertifyrefuses without a sealed run, under the pass bar, or if the ABOM changed after the run it was proved againstrestamppropagates a master fix and revokes every certificate stamped from the old oneharvestruns a certified builder agent that writes the next eval cases intoproposed.jsonlonly; promoting one stays a human editVerification
python3 factory.py selfcheckpasses with no keys. The seven guards in the README are each a reproducible exit-1.Includes four SVG diagrams.
🤖 Generated with Claude Code
Note
Add agent-factory example integrating output gating via SageRoute
examples/agent-factory/demonstrating a stamp → prove → certify → run → harvest workflow for managed LLM agents.factory.pyimplements the CLI pipeline: ABOM-based agent definitions, certification with human sign-off, production runs with cost envelope enforcement, Sage output gating, and recall/selfcheck commands.sage.pyprovides a fail-open Sage client for yes/no, scale, choice, and batch decisions; output gate incmd_runusessage.yesnoto approve or suppress agent output.llm.pyroutes completions through SageRoute when configured, with offline fallback and token-based cost estimation.agents/triager.py(support ticket triage with broker-gated billing actions) andagents/evalsmith.py(proposes new eval cases from flagged production traces).📊 Macroscope summarized 765a5c4. 10 files reviewed, 0 issues evaluated, 0 issues filtered, 0 comments posted
🗂️ Filtered Issues
No issues evaluated.