Skip to content

Prevent daily AIC guardrail blocks after pre-agent failures - #60355

Merged
pelikhan merged 6 commits into
mainfrom
copilot/daily-aic-guardrail-fix
Sep 12, 2026
Merged

Prevent daily AIC guardrail blocks after pre-agent failures#60355
pelikhan merged 6 commits into
mainfrom
copilot/daily-aic-guardrail-fix

Conversation

Copilot AI commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

The daily AIC guardrail treated every failed component job as potentially billable, so setup failures with zero model usage poisoned subsequent activations. It must distinguish execution that never started from missing accounting after execution began.

  • Execution evidence

    • Initialize authoritative not_started evidence for agent and detection jobs.
    • Transition evidence to started immediately before model execution.
    • Bind evidence to the component, run ID, and run attempt.
  • Fail-closed accounting

    • Accept zero usage only for valid, matching not_started evidence.
    • Continue rejecting absent, malformed, stale, or started evidence when accounting is missing.
    • Preserve producer freshness and rerun-attempt validation.
  • Artifact propagation

    • Include execution evidence in component and aggregate usage artifacts.
    • Upload detection evidence even when pre-execution setup fails.
{
  "version": 1,
  "component": "agent",
  "run_id": 34650415396,
  "run_attempt": 1,
  "state": "not_started"
}
  • Regression coverage
    • Covers skipped components, provable pre-execution failures, valid usage, missing or malformed accounting, detection parity, and stale rerun evidence.

Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix daily AI-credit guardrail block after pre-agent job failure Prevent daily AIC guardrail blocks after pre-agent failures Sep 12, 2026
Copilot AI requested a review from lpcox September 12, 2026 00:22
@pelikhan

Copy link
Copy Markdown
Collaborator

Introduced in #60136 . Not in a stable release yet .

@pelikhan
pelikhan marked this pull request as ready for review September 12, 2026 00:39
Copilot AI balanced review requested due to automatic review settings September 12, 2026 00:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Execution evidence is omitted from the aggregate usage upload, and some started markers precede engine setup rather than model invocation.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds execution-state evidence so the daily AIC guardrail can distinguish pre-model failures from missing accounting after execution starts.

Changes:

  • Emits and propagates agent/detection execution evidence.
  • Validates evidence identity before accepting zero usage.
  • Adds regression tests and recompiles workflow lock files.
File summaries
File Description
pkg/workflow/threat_detection_steps.go Adds detection evidence lifecycle and upload.
pkg/workflow/threat_detection_steps_test.go Tests detection evidence ordering.
pkg/workflow/threat_detection_external.go Uploads external-detector evidence.
pkg/workflow/daily_aic_workflow_guardrail_test.go Tests pre-agent failure compilation.
pkg/workflow/compiler_yaml_post_agent.go Adds agent evidence to artifacts.
pkg/workflow/compiler_yaml_main_job.go Initializes agent evidence.
pkg/workflow/compiler_yaml_ai_execution.go Generates execution evidence steps.
actions/setup/sh/collect_usage_artifact_files.sh Collects evidence for aggregation.
actions/setup/js/daily_aic_component_coverage.test.cjs Tests fail-closed evidence handling.
actions/setup/js/daily_aic_component_coverage.cjs Validates zero-usage evidence.
actions/setup/js/check_daily_aic_workflow_guardrail.cjs Passes run identity to validation.
.github/workflows/update-astro.lock.yml Regenerates compiled evidence flow.
.github/workflows/typist.lock.yml Regenerates compiled evidence flow.
.github/workflows/tidy.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-workflow-call.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-workflow-call-with-inputs.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-update-cross-repo-pr.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-test-tools.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-temporary-id.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-service-ports.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-pydantic.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-project.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-pi.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-otel-backends.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-opencode.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-multi-pr.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-kiro.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-goose.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-drive.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-deepseek-harness.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-cursor.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-crush.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-create-cross-repo-pr.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-copilot.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-copilot-sub-agents.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-copilot-small.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-copilot-sdk.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-copilot-mai.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-copilot-auto.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-copilot-arm.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-copilot-aoai-entra.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-copilot-aoai-apikey.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-codex.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-claude.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-ci.lock.yml Regenerates compiled agent evidence.
.github/workflows/smoke-checkout-pr-dispatch.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-call-workflow.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-aider.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-agent-scoped-approved.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-agent-public-none.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-agent-public-approved.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-agent-all-none.lock.yml Regenerates compiled evidence flow.
.github/workflows/smoke-agent-all-merged.lock.yml Regenerates compiled evidence flow.
.github/workflows/security-review.lock.yml Regenerates compiled evidence flow.
.github/workflows/safe-output-health.lock.yml Regenerates compiled evidence flow.
.github/workflows/research.lock.yml Regenerates compiled evidence flow.
.github/workflows/release.lock.yml Regenerates compiled agent evidence.
.github/workflows/pdf-summary.lock.yml Regenerates compiled evidence flow.
.github/workflows/notion-issue-summary.lock.yml Regenerates compiled agent evidence.
.github/workflows/jsweep.lock.yml Regenerates compiled evidence flow.
.github/workflows/hippo-embed.lock.yml Regenerates compiled agent evidence.
.github/workflows/go-fan.lock.yml Regenerates compiled evidence flow.
.github/workflows/firewall.lock.yml Regenerates compiled agent evidence.
.github/workflows/example-permissions-warning.lock.yml Regenerates compiled agent evidence.
.github/workflows/dev.lock.yml Regenerates compiled evidence flow.
.github/workflows/daily-malicious-code-scan.lock.yml Regenerates compiled agent evidence.
.github/workflows/daily-harness-experiment-proposer.lock.yml Regenerates compiled agent evidence.
.github/workflows/daily-github-docs-seo-optimizer.lock.yml Regenerates compiled evidence flow.
.github/workflows/codex-github-remote-mcp-test.lock.yml Regenerates compiled agent evidence.
.github/workflows/bot-detection.lock.yml Regenerates compiled agent evidence.
.github/workflows/artifacts-summary.lock.yml Regenerates compiled evidence flow.
.github/workflows/archie.lock.yml Regenerates compiled evidence flow.
.github/workflows/ai-moderator.lock.yml Regenerates compiled agent evidence.
.github/workflows/agentic-token-optimizer.lock.yml Regenerates compiled agent evidence.
.github/workflows/ace-editor.lock.yml Regenerates compiled agent evidence.
Review details
  • Files reviewed: 254/310 changed files
  • Comments generated: 3
  • Review effort level: Balanced

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +43 to +44
if [ -f /tmp/gh-aw/agent_execution.json ]; then cp /tmp/gh-aw/agent_execution.json /tmp/gh-aw/usage/agent/execution.json || true; fi
if [ -f /tmp/gh-aw/threat-detection/execution.json ]; then cp /tmp/gh-aw/threat-detection/execution.json /tmp/gh-aw/usage/detection/execution.json || true; fi

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7a738e9. The aggregate usage upload now includes /tmp/gh-aw/usage/agent/execution.json and /tmp/gh-aw/usage/detection/execution.json, with regression assertions for both paths.

Comment on lines +531 to +535
if !data.UseSamples {
for _, line := range generateComponentExecutionEvidenceStep("agent", "started", agentExecutionEvidencePath, "") {
yaml.WriteString(line)
}
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7a738e9 and tightened in 273a8c5. Agent started evidence is injected into the actual agentic_execution command, and AWF-backed commands write it immediately before the AWF invocation.

Comment thread pkg/workflow/threat_detection_steps.go Outdated
Comment on lines 114 to 115
steps = append(steps, generateComponentExecutionEvidenceStep("detection", "started", detectionExecutionEvidencePath, detectionStepCondition)...)
steps = append(steps, c.buildDetectionEngineExecutionStep(data)...)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7a738e9 and tightened in 273a8c5. Detection started evidence is injected into the actual detector execution command, after setup/log preparation and immediately before AWF invocation.

@github-actions

github-actions Bot commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Ponytail Reviewer completed successfully!

Cannot run ponytail review: /tmp/gh-aw/agent/pr-diff.patch is empty (0 bytes), so there are no changed lines to review.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • ab.chatgpt.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "ab.chatgpt.com"

See Network Configuration for more information.

Generated by Ponytail Reviewer for #60355

@github-actions

github-actions Bot commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel skipped because pre-fetch PR data was unavailable: unable to fetch PR file list

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

⚠️ PR Code Quality Reviewer failed during code quality review.

Warning

Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding.

What happened

The threat detection engine failed to produce results.

Review the workflow run logs for details.

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

⚠️ Security scanning failed for Design Decision Gate 🏗️. Review the logs for details.

No ADR enforcement needed: PR does not have the 'implementation' label and has <=100 new lines of code in business logic directories (prefetch summary: 93 additions, no custom config).

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer failed during the skills-based review.

Warning

Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding.

What happened

The threat detection engine failed to produce results.

Review the workflow run logs for details.

Warning

Firewall blocked 3 domains

The following domains were blocked by the firewall during workflow execution:

  • api.github.com
  • github.com
  • raw.githubusercontent.com

[!TIP]
api.github.com is blocked because GitHub API access uses the built-in GitHub tools by default. Instead of adding api.github.com to network.allowed, use tools.github.mode: gh-proxy for direct pre-authenticated GitHub CLI access without requiring network access to api.github.com:

tools:
  github:
    mode: gh-proxy

See GitHub Tools for more information on gh-proxy mode.

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.github.com"
    - "github.com"
    - "raw.githubusercontent.com"

See Network Configuration for more information.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@pelikhan

Copy link
Copy Markdown
Collaborator

Copilot AI and others added 2 commits September 12, 2026 02:36
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Sep 12, 2026

Copy link
Copy Markdown
Contributor Author

@copilot fix https://github.com/github/gh-aw/actions/runs/34664537831/job/103473733385?pr=60355

Fixed in 7a738e9 and refined in 273a8c5. The referenced CI failure was the wasm golden test; the goldens and generated locks are now regenerated, and started evidence is written from the actual execution command path.

@pelikhan

Copy link
Copy Markdown
Collaborator

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Sep 12, 2026

Copy link
Copy Markdown
Contributor Author

@copilot fix https://github.com/github/gh-aw/actions/runs/34668568262/job/103488221053#step:4:1

Fixed in 8454ccb. The failing CGO test expected checkout to remain the first runtime step; agent not_started evidence now initializes after the initial checkout phase while still preceding runtime/custom/pre-agent setup and model execution.

@pelikhan
pelikhan merged commit 08c167a into main Sep 12, 2026
37 of 39 checks passed
@pelikhan
pelikhan deleted the copilot/daily-aic-guardrail-fix branch September 12, 2026 03:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Daily AIC guardrail blocks after a pre-agent job failure with zero usage

4 participants