test(agent): add the agent release gate — a portable wire-level QA harness#5351
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThe pull request adds an ChangesAgent release gate
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant QA
participant ProductDriver
participant AgentRuntime
participant GatewayAPI
participant ResultStore
QA->>ProductDriver: Select cells and journeys
ProductDriver->>AgentRuntime: POST SSE invoke request
AgentRuntime-->>ProductDriver: Stream text, tool, approval, and finish frames
ProductDriver->>GatewayAPI: Perform workflow API checks
GatewayAPI-->>ProductDriver: Return revision and archive responses
ProductDriver->>ResultStore: Write results.json and summary.md
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai review |
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: c150e76a-03b1-4d9f-a30b-4508f4de4286
📒 Files selected for processing (17)
.agents/skills/agent-release-gate/SKILL.md.agents/skills/agent-release-gate/resources/LESSONS.md.agents/skills/agent-release-gate/resources/coverage.md.agents/skills/agent-release-gate/resources/qa_longctx.py.agents/skills/agent-release-gate/resources/qa_probe.py.agents/skills/agent-release-gate/resources/qa_product.py.agents/skills/agent-release-gate/resources/seeds/README.md.agents/skills/agent-release-gate/resources/seeds/longctx-daytona-all-green.results.json.agents/skills/agent-release-gate/resources/seeds/product-C2-full-green.results.json.claude/skills/agent-release-gate.gitignoredocs/design/agent-workflows/projects/qa/README.mddocs/design/agent-workflows/projects/qa/cleanup-plan.mddocs/design/agent-workflows/projects/qa/regression-skill-DRAFT.mddocs/design/agent-workflows/projects/qa/scripts/mcp_qa_server.mjsdocs/design/agent-workflows/projects/qa/scripts/warm_daytona_probe.pydocs/designs/testing/README.md
💤 Files with no reviewable changes (4)
- docs/design/agent-workflows/projects/qa/cleanup-plan.md
- docs/design/agent-workflows/projects/qa/scripts/mcp_qa_server.mjs
- docs/design/agent-workflows/projects/qa/regression-skill-DRAFT.md
- docs/design/agent-workflows/projects/qa/scripts/warm_daytona_probe.py
f546626 to
99eca0f
Compare
dosubot
Bot
added
size:XXL
There was a problem hiding this comment.
Actionable comments posted: 8
🧹 Nitpick comments (1)
.agents/skills/agent-release-gate/resources/coverage.md (1)
38-38: 🩺 Stability & Availability | 🔵 TrivialAvoid an implicit third-party dependency in the release gate.
The default public DeepWiki server makes
mcpdependent on external availability, network policy, and potential prompt/data egress. Prefer a controlled fixture or explicitly classify external unavailability asSKIPrather than a gate failure.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 13030489-f5ed-49bc-a874-a63dfc247d20
📒 Files selected for processing (17)
.agents/skills/agent-release-gate/SKILL.md.agents/skills/agent-release-gate/resources/LESSONS.md.agents/skills/agent-release-gate/resources/coverage.md.agents/skills/agent-release-gate/resources/qa_longctx.py.agents/skills/agent-release-gate/resources/qa_probe.py.agents/skills/agent-release-gate/resources/qa_product.py.agents/skills/agent-release-gate/resources/seeds/README.md.agents/skills/agent-release-gate/resources/seeds/longctx-daytona-all-green.results.json.agents/skills/agent-release-gate/resources/seeds/product-C2-full-green.results.json.claude/skills/agent-release-gate.gitignoredocs/design/agent-workflows/projects/qa/README.mddocs/design/agent-workflows/projects/qa/cleanup-plan.mddocs/design/agent-workflows/projects/qa/regression-skill-DRAFT.mddocs/design/agent-workflows/projects/qa/scripts/mcp_qa_server.mjsdocs/design/agent-workflows/projects/qa/scripts/warm_daytona_probe.pydocs/designs/testing/README.md
💤 Files with no reviewable changes (4)
- docs/design/agent-workflows/projects/qa/cleanup-plan.md
- docs/design/agent-workflows/projects/qa/regression-skill-DRAFT.md
- docs/design/agent-workflows/projects/qa/scripts/mcp_qa_server.mjs
- docs/design/agent-workflows/projects/qa/scripts/warm_daytona_probe.py
🚧 Files skipped from review as they are similar to previous changes (7)
- docs/design/agent-workflows/projects/qa/README.md
- .claude/skills/agent-release-gate
- .agents/skills/agent-release-gate/resources/seeds/longctx-daytona-all-green.results.json
- .agents/skills/agent-release-gate/resources/seeds/README.md
- .agents/skills/agent-release-gate/resources/seeds/product-C2-full-green.results.json
- docs/designs/testing/README.md
- .gitignore
| | `approve` | Raise an approval, then approve it. | The approved tool call continues via the in-band approval protocol the browser uses. | | ||
| | `deny` | Raise an approval, then deny it. | The denied path is handled cleanly (no phantom failure, no re-parking forever). | | ||
| | `commit` | Save an agent config as a new workflow revision, then fetch it back. | The changed parameter survives the round trip and the version bumps (v0 seed → v1; see LESSONS #14). Harness-agnostic — it drives the config REST API, not a turn. | | ||
| | `warm` | Run three turns, watch latency and the runner log. | Turns 2-3 are faster and the log confirms the session was genuinely **loaded**, not silently cold. | |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Make the warm pass criterion deterministic.
Requiring turns 2–3 to be faster is sensitive to host load and network variance, so it can make the release gate flaky. Use a bounded threshold or repeated-run statistic, or make latency advisory and require the “loaded” runner evidence as the deterministic pass condition.
There was a problem hiding this comment.
Valid, deferred to a follow-up. Making the warm pass criterion deterministic is the same class as the F-2 warm-proof heavy lift (latency is advisory; the deterministic signal is runner 'loaded' evidence). This edits coverage.md, which has a parallel in-flight change, so it is deferred rather than touched here.
There was a problem hiding this comment.
@mmabrouk Acknowledged. Deferring this avoids colliding with the parallel coverage.md work. The follow-up should treat latency as advisory and use runner “loaded” evidence as the deterministic warm pass condition.
Would you like me to open a GitHub follow-up issue for the F-2 warm-proof work?
✏️ Learnings added
Learnt from: mmabrouk
Repo: Agenta-AI/agenta PR: 5351
File: .agents/skills/agent-release-gate/resources/coverage.md:37-37
Timestamp: 2026-07-19T14:11:30.988Z
Learning: For the agent release gate’s `warm` journey in `.agents/skills/agent-release-gate/resources/coverage.md`, latency is advisory because it is host/network-variable. The deterministic pass signal should be runner “loaded” evidence; implementing this is part of the F-2 warm-proof follow-up and is deferred while parallel coverage documentation changes are in flight.
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
| Separate from the gate, these need live **Gmail and GitHub Composio connections** in the target | ||
| project. Skip them if the project has none. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Scope the connection prerequisite to the probes that need it.
This wording implies that memory and concurrent also require live Gmail/GitHub Composio connections, but their documented purposes do not. State the prerequisite specifically for gmail so the other probes remain runnable independently.
There was a problem hiding this comment.
Valid doc nit, deferred to a follow-up. Scoping the connection prerequisite specifically to the gmail probe (memory/concurrent run without it) edits coverage.md, which has a parallel in-flight change; held for the follow-up.
There was a problem hiding this comment.
@mmabrouk, understood—deferring this small documentation correction avoids colliding with the parallel edit. Would you like me to open a GitHub follow-up issue to track scoping the Composio prerequisite to only the gmail probe?
✏️ Learnings added
Learnt from: mmabrouk
Repo: Agenta-AI/agenta PR: 5351
File: .agents/skills/agent-release-gate/resources/coverage.md:44-45
Timestamp: 2026-07-19T14:11:29.722Z
Learning: In `.agents/skills/agent-release-gate/resources/coverage.md`, the `memory` and `concurrent` probes can run independently of live Gmail and GitHub Composio connections; that prerequisite applies only to the `gmail` probe.
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
| def probe_gmail(sandbox: str) -> dict: | ||
| """Do the Gmail tools resolve AND execute?""" | ||
| s = str(uuid.uuid4()) | ||
| p = params(sandbox, GATEWAY_TOOLS + [BASH]) | ||
| t = qa.invoke( | ||
| s, | ||
| [ | ||
| qa.user_msg( | ||
| "List the subjects of the 3 most recent emails in my inbox, then list my GitHub repositories. Use the tools." | ||
| ) | ||
| ], | ||
| p, | ||
| timeout=420.0, | ||
| ) | ||
| ok = qa.tool_ran(t) and not t.errors | ||
| return { | ||
| "pass": ok, | ||
| "why": "a Gmail gateway tool executed (tool-output-available) with no error", | ||
| "tools_called": [c.get("toolName") for c in t.tool_calls], | ||
| "turn": t.summary(), | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
Require each probe to prove its advertised wire conditions.
These checks can report success without exercising the intended capability:
- Gmail passes when any tool runs—even Bash or only GitHub.
- Memory ignores failed/tool-free flood turns and final wire errors.
- Concurrent ignores Gmail execution and all intermediate/final errors.
- Recorded failures still produce exit status zero.
Require expected available tool outcomes, clean turns, a genuinely executed flood/workload, and return nonzero when any selected probe fails.
Also applies to: 165-218, 221-269, 291-315
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Redact live mailbox and repository metadata from results.
The prompt requests email subjects and repository names, then persists t.summary(), including the first 400 reply characters, into results.json. Store a redacted summary instead.
Proposed fix
ok = qa.tool_ran(t) and not t.errors
+ turn = t.summary()
+ turn.pop("reply", None)
return {
"pass": ok,
"why": "a Gmail gateway tool executed (tool-output-available) with no error",
"tools_called": [c.get("toolName") for c in t.tool_calls],
- "turn": t.summary(),
+ "turn": turn,
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| def probe_gmail(sandbox: str) -> dict: | |
| """Do the Gmail tools resolve AND execute?""" | |
| s = str(uuid.uuid4()) | |
| p = params(sandbox, GATEWAY_TOOLS + [BASH]) | |
| t = qa.invoke( | |
| s, | |
| [ | |
| qa.user_msg( | |
| "List the subjects of the 3 most recent emails in my inbox, then list my GitHub repositories. Use the tools." | |
| ) | |
| ], | |
| p, | |
| timeout=420.0, | |
| ) | |
| ok = qa.tool_ran(t) and not t.errors | |
| return { | |
| "pass": ok, | |
| "why": "a Gmail gateway tool executed (tool-output-available) with no error", | |
| "tools_called": [c.get("toolName") for c in t.tool_calls], | |
| "turn": t.summary(), | |
| } | |
| def probe_gmail(sandbox: str) -> dict: | |
| """Do the Gmail tools resolve AND execute?""" | |
| s = str(uuid.uuid4()) | |
| p = params(sandbox, GATEWAY_TOOLS + [BASH]) | |
| t = qa.invoke( | |
| s, | |
| [ | |
| qa.user_msg( | |
| "List the subjects of the 3 most recent emails in my inbox, then list my GitHub repositories. Use the tools." | |
| ) | |
| ], | |
| p, | |
| timeout=420.0, | |
| ) | |
| ok = qa.tool_ran(t) and not t.errors | |
| turn = t.summary() | |
| turn.pop("reply", None) | |
| return { | |
| "pass": ok, | |
| "why": "a Gmail gateway tool executed (tool-output-available) with no error", | |
| "tools_called": [c.get("toolName") for c in t.tool_calls], | |
| "turn": turn, | |
| } |
There was a problem hiding this comment.
Partially fixed in 3e54026, partially a tracked heavy-lift. Fixed now: (1) the Gmail probe now requires a GMAIL tool (toolName gmail__*) to have executed with an available outcome, not merely 'any tool ran', so a bash-only or GitHub-only run no longer passes; (2) live inbox subjects and repo names are redacted — the reply is popped from the persisted turn summary so results.json no longer stores mailbox/repo metadata; (3) main() now returns nonzero when any selected probe fails. Declined here (tracked): proving the flood/workload was 'genuinely executed' and that the runner stayed warm requires correlating with runner-log session-reuse evidence — the same F-2 heavy lift called out on the warm-session journey, out of scope for this graduation PR.
There was a problem hiding this comment.
@mmabrouk, thanks for the detailed update. The Gmail-specific available outcome, reply redaction, and nonzero failure exit status address the immediate correctness and privacy gaps.
Deferring runner-log/session-reuse correlation is reasonable for this skill graduation, provided F-2 is tracked. Would you like me to open a GitHub follow-up issue for the remaining workload-execution and warm-session evidence requirements?
✏️ Learnings added
Learnt from: mmabrouk
Repo: Agenta-AI/agenta PR: 5351
File: .agents/skills/agent-release-gate/resources/qa_longctx.py:143-174
Timestamp: 2026-07-19T14:11:07.276Z
Learning: In `.agents/skills/agent-release-gate/resources/qa_longctx.py`, successful Gmail probing must require an executed Gmail tool (`toolName` matching `gmail__*`) with an `available` outcome; Bash-only or GitHub-only tool execution is insufficient. Persisted probe summaries must omit the assistant reply to avoid storing live mailbox subjects or repository names, and selected probe failures must yield a nonzero process exit status. Proving genuinely executed long-context floods and warm runner/session reuse requires runner-log correlation and is tracked separately as the F-2 heavy lift, outside the skill-graduation PR scope.
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
| uv run qa_product.py --cell C3 # one cell | ||
| uv run qa_product.py --all # every cell | ||
| uv run qa_product.py --cell C3 --only approve # one journey |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Require the P2 slug only when a selected journey invokes P2.
--all --only commit and --all --only mcp currently abort, although commit does not invoke the provider and MCP skips Pi cells. The top-level --all example is also now incomplete.
Proposed fix
- uv run qa_product.py --all # every cell
+ uv run qa_product.py --all --custom-slug SLUG # every cell
...
- if "P2" in cells and not args.custom_slug:
+ p2_invoke_journeys = {"chat", "mount", "tool", "approve", "deny", "warm"}
+ if (
+ "P2" in cells
+ and p2_invoke_journeys.intersection(journeys)
+ and not args.custom_slug
+ ):Also applies to: 957-966
There was a problem hiding this comment.
Valid, deferred to a follow-up. Scoping the P2 slug requirement to only the journeys that actually invoke P2 (and fixing the top-level --all example) is a qa_product.py change; that file has a parallel in-flight change, so these argparse/gating edits are held for the follow-up rather than risking a collision here.
There was a problem hiding this comment.
@mmabrouk, understood. Deferring this targeted argparse/gating adjustment avoids colliding with the parallel qa_product.py work.
Would you like me to open a GitHub follow-up issue covering the journey-aware P2 slug requirement and corrected --all example?
| mcp = { | ||
| "name": "deepwiki", | ||
| "connection": {"type": "http", "url": MCP_URL}, | ||
| "policy": {"tools": {"mode": "all"}}, | ||
| } | ||
| prompt = ( | ||
| "Use the deepwiki MCP tool named read_wiki_structure with repoName 'facebook/react' to " | ||
| "list the wiki topics, then reply with only: DONE." | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
python3 - <<'PY'
from pathlib import Path
path = Path('.agents/skills/agent-release-gate/resources/qa_product.py')
lines = path.read_text().splitlines()
for start, end in [(820, 900), (920, 960)]:
print(f"\n--- {start}-{end} ---")
for i in range(start-1, min(end, len(lines))):
print(f"{i+1:4d}: {lines[i]}")
PYRepository: Agenta-AI/agenta
Length of output: 5580
🏁 Script executed:
python3 - <<'PY'
from pathlib import Path
path = Path('.agents/skills/agent-release-gate/resources/qa_product.py')
lines = path.read_text().splitlines()
for start, end in [(840, 910), (930, 960)]:
print(f"\n--- {start}-{end} ---")
for i in range(start-1, min(end, len(lines))):
print(f"{i+1:4d}: {lines[i]}")
PY
printf '\n--- SEARCH ---\n'
rg -n "MCP_URL|read_wiki_structure|deepwiki|--mcp-url|repoName" .agents/skills/agent-release-gate/resources/qa_product.pyRepository: Agenta-AI/agenta
Length of output: 6050
🏁 Script executed:
python3 - <<'PY'
print("probe")
PYRepository: Agenta-AI/agenta
Length of output: 160
Make --mcp-url DeepWiki-specific or configurable end-to-end.
The journey still hard-codes deepwiki, read_wiki_structure, and repoName='facebook/react', so the flag is misleading for other MCP servers; either narrow the flag/docs to DeepWiki-compatible endpoints or parameterize the server/tool/repo fields too.
There was a problem hiding this comment.
Valid, deferred to a follow-up. Narrowing --mcp-url to DeepWiki-compatible endpoints (or parameterizing server/tool/repo) lives in qa_product.py, which has a parallel in-flight change; deferring to the same follow-up as the other qa_product.py items.
There was a problem hiding this comment.
@mmabrouk Acknowledged—this is a valid follow-up and doesn’t need to block the parallel qa_product.py work. Would you like me to open and assign a GitHub issue for it?
✏️ Learnings added
Learnt from: mmabrouk
Repo: Agenta-AI/agenta PR: 5351
File: .agents/skills/agent-release-gate/resources/qa_product.py:867-875
Timestamp: 2026-07-19T14:11:29.231Z
Learning: In `.agents/skills/agent-release-gate/resources/qa_product.py`, `j7_mcp()` currently hard-codes the DeepWiki server name, `read_wiki_structure` tool, and `facebook/react` repository even though `--mcp-url` permits overriding the endpoint. A planned follow-up should either restrict `--mcp-url` and documentation to DeepWiki-compatible endpoints or make server/tool/repository configuration fully parameterized. The file has parallel in-flight changes, so this work is intentionally deferred.
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
99eca0f to
3ddabdd
Compare
There was a problem hiding this comment.
Actionable comments posted: 5
♻️ Duplicate comments (1)
.agents/skills/agent-release-gate/resources/qa_product.py (1)
663-672: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winScope persistence evidence to the requested
catcall.The code joins every tool output. A second-turn command that echoes the known token can pass without reading the mounted file. Select the call whose command is
cat qa-mount.txt, require its outcome to beavailable, then inspect only its payload.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: bf2985b4-a190-48fa-b732-ad9115040327
📒 Files selected for processing (17)
.agents/skills/agent-release-gate/SKILL.md.agents/skills/agent-release-gate/resources/LESSONS.md.agents/skills/agent-release-gate/resources/coverage.md.agents/skills/agent-release-gate/resources/qa_longctx.py.agents/skills/agent-release-gate/resources/qa_probe.py.agents/skills/agent-release-gate/resources/qa_product.py.agents/skills/agent-release-gate/resources/seeds/README.md.agents/skills/agent-release-gate/resources/seeds/longctx-daytona-all-green.results.json.agents/skills/agent-release-gate/resources/seeds/product-C2-full-green.results.json.claude/skills/agent-release-gate.gitignoredocs/design/agent-workflows/projects/qa/README.mddocs/design/agent-workflows/projects/qa/cleanup-plan.mddocs/design/agent-workflows/projects/qa/regression-skill-DRAFT.mddocs/design/agent-workflows/projects/qa/scripts/mcp_qa_server.mjsdocs/design/agent-workflows/projects/qa/scripts/warm_daytona_probe.pydocs/designs/testing/README.md
💤 Files with no reviewable changes (4)
- docs/design/agent-workflows/projects/qa/regression-skill-DRAFT.md
- docs/design/agent-workflows/projects/qa/scripts/mcp_qa_server.mjs
- docs/design/agent-workflows/projects/qa/cleanup-plan.md
- docs/design/agent-workflows/projects/qa/scripts/warm_daytona_probe.py
🚧 Files skipped from review as they are similar to previous changes (7)
- .claude/skills/agent-release-gate
- .agents/skills/agent-release-gate/resources/seeds/README.md
- .agents/skills/agent-release-gate/resources/seeds/longctx-daytona-all-green.results.json
- .agents/skills/agent-release-gate/resources/coverage.md
- .gitignore
- .agents/skills/agent-release-gate/resources/seeds/product-C2-full-green.results.json
- docs/designs/testing/README.md
| def probe_memory(sandbox: str, turns: int) -> dict: | ||
| """Plant a token, flood the context, then ask for it back.""" | ||
| s = str(uuid.uuid4()) | ||
| token = f"QA-MEM-{uuid.uuid4().hex[:12].upper()}" | ||
| p = params(sandbox, GATEWAY_TOOLS + [BASH]) | ||
|
|
||
| msgs = [ | ||
| qa.user_msg( | ||
| f"Remember this exact token for the rest of our conversation: {token}. " | ||
| "Do NOT write it to any file. Just reply: OK" | ||
| ) | ||
| ] | ||
| t = qa.invoke(s, msgs, p, timeout=420.0) | ||
| msgs.append(t.assistant_message()) | ||
| if t.errors: | ||
| return {"pass": False, "why": "turn 1 (plant) errored", "turn": t.summary()} | ||
|
|
||
| # Flood the context. Alternate bulky Gmail payloads with large bash output — this is what a | ||
| # real user's long, tool-heavy session looks like, and it is what triggers compaction. | ||
| filler = [ | ||
| "Fetch my 5 most recent emails with the Gmail tool and summarize each in one line.", | ||
| "Use bash to run: seq 1 800 | paste -sd, - and report the last 20 characters only.", | ||
| "List my Gmail threads, then list my GitHub repositories. Report only the counts.", | ||
| 'Use bash to run: for i in $(seq 1 60); do echo "line-$i: $(head -c 40 /dev/urandom | base64)"; done and report only the final line.', | ||
| ] | ||
| trace = [] | ||
| for i in range(turns): | ||
| q = filler[i % len(filler)] | ||
| msgs.append(qa.user_msg(q)) | ||
| t = qa.invoke(s, msgs, p, timeout=420.0) | ||
| msgs.append(t.assistant_message()) | ||
| trace.append( | ||
| {"turn": i + 2, "ms": t.ms, "tools": len(t.tool_calls), "err": t.errors[:1]} | ||
| ) | ||
| print( | ||
| f" flood turn {i + 2}/{turns + 1}: {t.ms}ms tools={len(t.tool_calls)}", | ||
| flush=True, | ||
| ) | ||
|
|
||
| msgs.append( | ||
| qa.user_msg( | ||
| "What was the exact token I asked you to remember at the very start? Reply with only the token." | ||
| ) | ||
| ) | ||
| final = qa.invoke(s, msgs, p, timeout=420.0) | ||
| ok = token in final.reply | ||
| return { | ||
| "pass": ok, | ||
| "why": f"the token planted in turn 1 survived {turns} tool-heavy turns (token={token})", | ||
| "token": token, | ||
| "recalled": final.reply[:120], | ||
| "flood": trace, | ||
| "session_id": s, | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
Require proof that the memory probe remained warm.
Every request replays history containing the planted token. If the runner falls cold, transcript reconstruction can reveal that token without exercising native-context compaction. Correlate hit-continue/no-cold runner evidence or report the probe as inconclusive.
There was a problem hiding this comment.
Declined as a code change here; valid and tracked. Proving the memory probe stayed warm (that recall exercised native-context compaction rather than history replay) needs runner-log correlation of hit-continue/no-cold — the F-2 heavy lift already flagged on the warm-session journey. That is a separate piece of work, not something to bundle into this graduation PR; recorded as a known limitation rather than silently accepted.
There was a problem hiding this comment.
@mmabrouk, acknowledged. Keeping this as a documented limitation is appropriate for this graduation PR; the current memory probe demonstrates transcript-level recall, not conclusively warm native-context compaction. The runner-log correlation belongs with the tracked F-2 warm-session follow-up.
✏️ Learnings added
Learnt from: mmabrouk
Repo: Agenta-AI/agenta PR: 5351
File: .agents/skills/agent-release-gate/resources/qa_longctx.py:177-230
Timestamp: 2026-07-19T14:11:04.259Z
Learning: In `.agents/skills/agent-release-gate/resources/qa_longctx.py`, `probe_memory()` replays the complete message history on every `qa.invoke()` call. A successful token recall therefore does not by itself prove native-context compaction or warm-session reuse, because cold transcript reconstruction can expose the planted token. Proving this requires runner-log correlation showing `hit-continue`/no-cold behavior; this is the tracked F-2 heavy-lift follow-up on the warm-session journey and is intentionally out of scope for PR `#5351`.
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
| try: | ||
| f = json.loads(payload) | ||
| except json.JSONDecodeError: | ||
| continue |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Both SSE clients silently discard malformed data, allowing invalid streams to appear green.
.agents/skills/agent-release-gate/resources/qa_product.py#L411-L414: append parsing failures toTurn.errorsso journey predicates reject the stream..agents/skills/agent-release-gate/resources/qa_probe.py#L152-L168: record parse/error frames and require a clean terminal response before returning zero.
📍 Affects 2 files
.agents/skills/agent-release-gate/resources/qa_product.py#L411-L414(this comment).agents/skills/agent-release-gate/resources/qa_probe.py#L152-L168
There was a problem hiding this comment.
Half fixed in 3e54026: the qa_probe.py side now records malformed/error frames and requires a clean terminal response before exiting zero. The qa_product.py side (appending parse failures to Turn.errors) is valid but deferred to a follow-up — qa_product.py has a parallel in-flight change, so it is not edited in this commit to avoid a collision. Tracking it for the follow-up.
There was a problem hiding this comment.
@mmabrouk Thanks for the clear breakdown. The qa_probe.py portion is covered by 3e54026; the remaining valid concern is limited to recording malformed SSE payloads in qa_product.py so affected journeys fail rather than potentially passing.
Would you like me to open a GitHub follow-up issue for that deferred qa_product.py change?
| stamp = time.strftime("%Y%m%d-%H%M%S") | ||
| outdir = RUNS / stamp | ||
| outdir.mkdir(parents=True, exist_ok=True) | ||
|
|
||
| results: dict = {} | ||
| for cid in cells: | ||
| cell = CELLS[cid] | ||
| results[cid] = {"config": {k: v for k, v in cell.items()}, "journeys": {}} | ||
| for jname in journeys: | ||
| print(f"[{cid}] {jname} ... ", end="", flush=True) | ||
| try: | ||
| r = JOURNEYS[jname](cell) | ||
| except Exception as e: # a crash is a result, not a reason to lose the run | ||
| r = {"pass": False, "why": f"driver exception: {type(e).__name__}: {e}"} | ||
| results[cid]["journeys"][jname] = r | ||
| verdict = "SKIP" if r.get("skip") else ("PASS" if r.get("pass") else "FAIL") | ||
| print(verdict, f"— {r.get('why', '')[:90]}") | ||
| (outdir / "results.json").write_text(json.dumps(results, indent=2)) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Second-resolution directory names let concurrent runs overwrite each other’s evidence.
.agents/skills/agent-release-gate/resources/qa_product.py#L976-L993: append a UUID or nanosecond component to the run directory..agents/skills/agent-release-gate/resources/qa_longctx.py#L310-L314: apply the same uniqueness rule to long-context runs.
🧰 Tools
🪛 ast-grep (0.44.1)
[info] 992-992: use jsonify instead of json.dumps for JSON output
Context: json.dumps(results, indent=2)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
📍 Affects 2 files
.agents/skills/agent-release-gate/resources/qa_product.py#L976-L993(this comment).agents/skills/agent-release-gate/resources/qa_longctx.py#L310-L314
There was a problem hiding this comment.
Half fixed in 3e54026: long-context run directories now carry a random suffix (longctx-<sandbox>-<stamp>-<rand>) so concurrent runs no longer overwrite each other's evidence. The identical fix for the product driver's run directory is deferred to the same follow-up as the other qa_product.py items, since that file has a parallel in-flight change.
There was a problem hiding this comment.
@mmabrouk, thanks—confirmed the long-context collision is addressed by 3e54026. The product driver still needs the same uniqueness guarantee before this consolidated comment can be resolved.
Would you like me to open a follow-up GitHub issue for the deferred qa_product.py change?
…rness Package the pre-release agent-QA harness as a tracked, portable skill at .agents/skills/agent-release-gate/ (SKILL.md + resources: the three drivers, a cells-x-journeys coverage matrix, the traps doc, and sanitized regression seeds). The drivers read AGENTA_BASE / AGENTA_PROJECT_ID / AGENTA_API_KEY from the environment first (--env-file fallback) with a clear missing-credentials error, and write runs to ./qa-gate-runs/ in the CWD, so the gate runs against any deployment. A tracked .claude/skills symlink and two .gitignore allowlist entries make it discoverable. Also delete four stale files from the design folder (cleanup-plan.md, regression-skill-DRAFT.md, scripts/warm_daytona_probe.py, scripts/mcp_qa_server.mjs) and add a one-line pointer plus a testing-README subsection. Seed canary tokens are replaced with a fixed placeholder so no gitleaks allowlist change is needed. Claude-Session: https://claude.ai/code/session_019TK4ow9a3dXUNY657djbQr
3ddabdd to
3e54026
Compare
Railway Preview Environment
Updated at 2026-07-19T14:30:19.702Z |
1 participant
Context
The pre-release QA harness for the agent runtime lived inside a design-docs folder (
docs/design/agent-workflows/projects/qa/), wired to one machine: it read credentials from a hardcoded~/.agenta-bighetzner.envand wrote run output back into the tracked tree. It could not be run anywhere else without editing the scripts. The methodology it encodes is worth keeping: it drives the same product endpoint the playground drives and asserts on the SSE frame stream and real side effects, never on model prose, so it is genuinely deployment-agnostic once the machine-specific bits are removed.This graduates that harness into a tracked, portable
agent-release-gateskill and leaves the design folder as the archive.Changes
A skill with everything needed.
.agents/skills/agent-release-gate/with a smallSKILL.md(what the gate is, when to run it, the minimal procedure) and aresources/directory holding the three drivers, the traps doc (LESSONS.md), a coverage matrix (coverage.md), and regression seeds. A tracked.claude/skills/agent-release-gatesymlink and the two.gitignoreallowlist entries strictly required to track the skill make it discoverable by both Claude and Codex, matching the existingwrite-template-playbookspattern.Run-anywhere. The drivers read
AGENTA_BASE,AGENTA_PROJECT_ID, andAGENTA_API_KEYfrom the environment first, falling back to--env-file(default~/.agenta-bighetzner.envfor backward compatibility). Missing credentials produce a clear startup error naming exactly what is unset. Run output defaults to./qa-gate-runs/in the current working directory (override withAGENTA_QA_RUNS_DIR), so runs never accumulate inside the skill or the repo.Before (hardcoded, module import time):
After (env-first, resolved in
main()):Sanitized seeds, no gitleaks change. The two
seeds/*.results.jsonare representative green runs kept as regression references. Every per-run canary token the probes plant and read back (QA-MEM-*,QA-CONC*,QA-MOUNT-*,QA-BASH-*) is replaced with the fixed placeholderQA-TOKEN-SANITIZED, structure untouched, so no.gitleaks.tomlallowlist change is needed.Design-folder cleanup (deletions only). Removes four stale files that exist on main:
cleanup-plan.md,regression-skill-DRAFT.md,scripts/warm_daytona_probe.py, andscripts/mcp_qa_server.mjs(broken, and carried an invisible U+200E in its filename). Adds a one-line pointer at the top of the folder's README and a short linking subsection indocs/designs/testing/README.md. Nothing else underdocs/designchanges: no findings/status docs move here, noruns/are added.Note:
scripts/run_matrix.pyhad an unrelated uncommitted modification in the working tree from another session. It was left entirely untouched and is not part of this PR.Tests / notes
The branch is a single clean commit off
main(rewritten at the reviewer's request so the PR carries only the graduation, not the release-night findings/evidence history). Verified the skill the way a stranger would use it.Missing-credentials failure (env unset, no env file):
--helpworks with no credentials present for all three drivers (qa_longctx.pyno longer hits the network at import).Live run (real credentials exported as env vars, from a temp dir outside the repo):
Results landed in
./qa-gate-runs/in the temp dir, not in the skill.ruff formatandruff checkpass on all three drivers.gitleaksscans the commit with no leaks and no.gitleaks.tomlchange.https://claude.ai/code/session_019TK4ow9a3dXUNY657djbQr