Prevent daily AIC guardrail blocks after pre-agent failures - #60355
Conversation
Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
|
Introduced in #60136 . Not in a stable release yet . |
There was a problem hiding this comment.
🟡 Changes recommended
Execution evidence is omitted from the aggregate usage upload, and some started markers precede engine setup rather than model invocation.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Adds execution-state evidence so the daily AIC guardrail can distinguish pre-model failures from missing accounting after execution starts.
Changes:
- Emits and propagates agent/detection execution evidence.
- Validates evidence identity before accepting zero usage.
- Adds regression tests and recompiles workflow lock files.
File summaries
| File | Description |
|---|---|
pkg/workflow/threat_detection_steps.go |
Adds detection evidence lifecycle and upload. |
pkg/workflow/threat_detection_steps_test.go |
Tests detection evidence ordering. |
pkg/workflow/threat_detection_external.go |
Uploads external-detector evidence. |
pkg/workflow/daily_aic_workflow_guardrail_test.go |
Tests pre-agent failure compilation. |
pkg/workflow/compiler_yaml_post_agent.go |
Adds agent evidence to artifacts. |
pkg/workflow/compiler_yaml_main_job.go |
Initializes agent evidence. |
pkg/workflow/compiler_yaml_ai_execution.go |
Generates execution evidence steps. |
actions/setup/sh/collect_usage_artifact_files.sh |
Collects evidence for aggregation. |
actions/setup/js/daily_aic_component_coverage.test.cjs |
Tests fail-closed evidence handling. |
actions/setup/js/daily_aic_component_coverage.cjs |
Validates zero-usage evidence. |
actions/setup/js/check_daily_aic_workflow_guardrail.cjs |
Passes run identity to validation. |
.github/workflows/update-astro.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/typist.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/tidy.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-workflow-call.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-workflow-call-with-inputs.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-update-cross-repo-pr.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-test-tools.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-temporary-id.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-service-ports.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-pydantic.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-project.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-pi.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-otel-backends.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-opencode.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-multi-pr.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-kiro.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-goose.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-drive.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-deepseek-harness.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-cursor.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-crush.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-create-cross-repo-pr.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-copilot.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-copilot-sub-agents.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-copilot-small.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-copilot-sdk.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-copilot-mai.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-copilot-auto.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-copilot-arm.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-copilot-aoai-entra.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-copilot-aoai-apikey.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-codex.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-claude.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-ci.lock.yml |
Regenerates compiled agent evidence. |
.github/workflows/smoke-checkout-pr-dispatch.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-call-workflow.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-aider.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-agent-scoped-approved.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-agent-public-none.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-agent-public-approved.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-agent-all-none.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/smoke-agent-all-merged.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/security-review.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/safe-output-health.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/research.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/release.lock.yml |
Regenerates compiled agent evidence. |
.github/workflows/pdf-summary.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/notion-issue-summary.lock.yml |
Regenerates compiled agent evidence. |
.github/workflows/jsweep.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/hippo-embed.lock.yml |
Regenerates compiled agent evidence. |
.github/workflows/go-fan.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/firewall.lock.yml |
Regenerates compiled agent evidence. |
.github/workflows/example-permissions-warning.lock.yml |
Regenerates compiled agent evidence. |
.github/workflows/dev.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/daily-malicious-code-scan.lock.yml |
Regenerates compiled agent evidence. |
.github/workflows/daily-harness-experiment-proposer.lock.yml |
Regenerates compiled agent evidence. |
.github/workflows/daily-github-docs-seo-optimizer.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/codex-github-remote-mcp-test.lock.yml |
Regenerates compiled agent evidence. |
.github/workflows/bot-detection.lock.yml |
Regenerates compiled agent evidence. |
.github/workflows/artifacts-summary.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/archie.lock.yml |
Regenerates compiled evidence flow. |
.github/workflows/ai-moderator.lock.yml |
Regenerates compiled agent evidence. |
.github/workflows/agentic-token-optimizer.lock.yml |
Regenerates compiled agent evidence. |
.github/workflows/ace-editor.lock.yml |
Regenerates compiled agent evidence. |
Review details
- Files reviewed: 254/310 changed files
- Comments generated: 3
- Review effort level: Balanced
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
| if [ -f /tmp/gh-aw/agent_execution.json ]; then cp /tmp/gh-aw/agent_execution.json /tmp/gh-aw/usage/agent/execution.json || true; fi | ||
| if [ -f /tmp/gh-aw/threat-detection/execution.json ]; then cp /tmp/gh-aw/threat-detection/execution.json /tmp/gh-aw/usage/detection/execution.json || true; fi |
There was a problem hiding this comment.
Fixed in 7a738e9. The aggregate usage upload now includes /tmp/gh-aw/usage/agent/execution.json and /tmp/gh-aw/usage/detection/execution.json, with regression assertions for both paths.
| if !data.UseSamples { | ||
| for _, line := range generateComponentExecutionEvidenceStep("agent", "started", agentExecutionEvidencePath, "") { | ||
| yaml.WriteString(line) | ||
| } | ||
| } |
There was a problem hiding this comment.
Fixed in 7a738e9 and tightened in 273a8c5. Agent started evidence is injected into the actual agentic_execution command, and AWF-backed commands write it immediately before the AWF invocation.
| steps = append(steps, generateComponentExecutionEvidenceStep("detection", "started", detectionExecutionEvidencePath, detectionStepCondition)...) | ||
| steps = append(steps, c.buildDetectionEngineExecutionStep(data)...) |
There was a problem hiding this comment.
Fixed in 7a738e9 and tightened in 273a8c5. Detection started evidence is injected into the actual detector execution command, after setup/log preparation and immediately before AWF invocation.
|
✅ Ponytail Reviewer completed successfully! Cannot run ponytail review: /tmp/gh-aw/agent/pr-diff.patch is empty (0 bytes), so there are no changed lines to review. Warning Firewall blocked 1 domainThe following domain was blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "ab.chatgpt.com"See Network Configuration for more information.
|
|
✅ Test Quality Sentinel completed test quality analysis. Test Quality Sentinel skipped because pre-fetch PR data was unavailable: unable to fetch PR file list
|
|
Warning Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding. What happenedThe threat detection engine failed to produce results. Review the workflow run logs for details.
|
|
No ADR enforcement needed: PR does not have the 'implementation' label and has <=100 new lines of code in business logic directories (prefetch summary: 93 additions, no custom config).
|
|
🧠 Matt Pocock Skills Reviewer failed during the skills-based review. Warning Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding. What happenedThe threat detection engine failed to produce results. Review the workflow run logs for details. Warning Firewall blocked 3 domainsThe following domains were blocked by the firewall during workflow execution:
[!TIP] tools:
github:
mode: gh-proxySee GitHub Tools for more information on To allow these domains, add them to the network:
allowed:
- defaults
- "api.github.com"
- "github.com"
- "raw.githubusercontent.com"See Network Configuration for more information.
|
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Fixed in |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Fixed in |
The daily AIC guardrail treated every failed component job as potentially billable, so setup failures with zero model usage poisoned subsequent activations. It must distinguish execution that never started from missing accounting after execution began.
Execution evidence
not_startedevidence for agent and detection jobs.startedimmediately before model execution.Fail-closed accounting
not_startedevidence.startedevidence when accounting is missing.Artifact propagation
{ "version": 1, "component": "agent", "run_id": 34650415396, "run_attempt": 1, "state": "not_started" }