[Copilot]

Copilot CLI can exit 0 after a server-side tool_search_call without reading the PR or producing a review. gh-aw currently accepts that exit as success before the collector reports missing_terminal_safe_output.

• Detect premature completion
  Inspect the invocation’s exact native session, with structured stdout as fallback. Recognize a final empty main-assistant response containing only completed server-side tool searches and no local tool requests.

• Fail without replaying work
  Return a non-retryable copilot_tool_search_only_completion failure when no terminal safe output exists. Leave safe-output files and collector validation unchanged.

• Avoid false classifications
  Preserve completed outputs, subsequent responses, and refusals; ignore sub-agent messages and reject malformed native logs as evidence.

This addresses gh-aw’s success classification—not continuation itself. CLI-versus-service ownership remains unresolved, and release notes through 1.0.105 identify no verified fix.

• Fixes Copilot reviewer exits successfully after tool discovery without a terminal safe output #72702

[pelikhan]

use filter in readJsonFile to apply filter inline and shortcut quickly. see jsonl helpers

[Copilot]

Applied the inline event filter to the native session reader and stdout fallback. Commit: 46359b83.

[pelikhan]

@copilot use filter in readJsonFile to apply filter inline and shortcut quickly. see jsonl helpers

[Copilot]

@copilot use filter in readJsonFile to apply filter inline and shortcut quickly. see jsonl helpers

Applied the inline event filter to the native session reader and stdout fallback. Commit: 46359b83.

[github-actions]

✅ Ponytail Reviewer completed successfully!

Ponytail Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Ponytail Reviewer · gpt56 · ~9 AIC · ⌖ ~6 AIC · ⊞ 14.8K · ◷

Generated by Ponytail Reviewer for #72939

[github-actions]

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

[github-actions]

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · haiku45 · ~5 AIC · ⌖ ~7 AIC · ◇ ~3 AIC · ⊞ 8.9K · ◷

🧪 Test quality analysis by Test Quality Sentinel

[github-actions]

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

[github-actions]

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · opus55 · ~11 AIC · ⌖ ~33 AIC · ◇ ~1 AIC · ⊞ 1.9K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

[github-actions]

Comment Memory

Peek at saved memory (pr-code-quality-reviewer)

reviewed_at: 2026-10-29T14:59:00Z
review_event: COMMENT
top_themes: [session-directory traversal]
files_reviewed: [actions/setup/js/copilot_harness.cjs, actions/setup/js/copilot_harness.test.cjs]
comment_count: 1

Note

This comment is managed by comment memory.

Expand the saved memory block to view or edit the persistent context for this thread.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~1 AIC · ⊞ 1.6K · ◷
_{Comment /review to run again}

[github-actions]

Review result

One correctness issue found: the detector stops after the first readable session directory, so later configured locations may never be checked. I left an inline comment with the impact and a regression-test suggestion.

Details

The specialist pass completed; its export concern was disproven by the current module, and the remaining session-context concern is covered by the existing field mapping and tests. No tests were run because this was a static review.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~1 AIC · ⊞ 1.6K
_{Comment /review to run again}

[github-actions]

This returns after the first readable session directory even when it does not contain a tool-search-only completion, so a valid record in a later configured directory is never examined; this can leave a discovery-only invocation classified as successful and defer the failure to the collector.

💡 Check every candidate session directory

The session directory list can contain multiple sources (for example the configured SDK base plus the default Copilot locations). Only return immediately when the predicate is true; otherwise continue through the remaining files, and preserve the malformed/unreadable distinction before falling back to stdout. Add a test with a non-matching first file and matching second file.

[github-actions]

Skills-Based Review 🧠

Applied /diagnosing-bugs to this fix for premature Copilot success classification (Fixes #72702). Triage via pr-triage: bug_fix, high-impact files copilot_harness.cjs (core detection) and copilot_harness.test.cjs (coverage).

One correctness bug found that undermines the fix's own goal — see inline comment.

📋 Key Themes & Highlights

Key Themes

• Early-return bug in directory scan (copilot_harness.cjs:1262): hasCopilotToolSearchOnlyCompletion returns on the first readable session directory even when it doesn't contain a match, so later directories in copilotSessionStateDirectories() are never examined. This can silently defer a real discovery-only completion to the collector instead of catching it — the exact regression class this PR is meant to close. This specific defect was already called out by an automated review pass on the current head commit (46359b83), and it remains unresolved in the diff.
• Test coverage gap: the existing "reads only the exact native session" test only ever populates a single COPILOT_HOME directory, so it can't catch the early-return regression. A case with a non-matching/missing first directory and a matching second directory would pin the fix down.

Positive Highlights

• ✅ Careful scoping to the exact session UUID (regex-validated) avoids picking up an unrelated session by recency in a shared HOME — good defensive design per /diagnosing-bugs (minimizing false-positive blast radius).
• ✅ isCopilotToolSearchOnlyCompletion correctly excludes sub-agent and nested tool-call messages via copilotSessionContext, and the refusal/last-message handling is well tested with clear edge-case names (empty content, non-empty content, refusals, in-progress/other server tool types).
• ✅ The end-to-end harness test (it.each([false, true])) nicely proves the non-retryable failure class only fires when there's no terminal safe output, directly guarding against double side effects.

@copilot please address the review comments above.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · ~111 AIC · ⌖ ~16 AIC · ⊞ 10.2K
_{Comment /matt to run again}

[github-actions]

[/diagnosing-bugs] This for loop returns on the first directory that does not throw ENOENT, regardless of whether it actually found a tool-search-only completion. copilotSessionStateDirectories() returns up to three candidate directories; if the first readable one yields false (wrong session content, or a stale/empty file), the remaining directories are never checked — even though a real match could live in one of them. This can mask a discovery-only completion as success, exactly the failure mode this PR sets out to fix.

💡 Check every candidate directory before giving up

Only short-circuit when the predicate is actually true; otherwise fall through to the next directory, and only drop to the stdout fallback once every directory has been tried (or thrown non-ENOENT). Something like:

for (const directory of copilotSessionStateDirectories()) {
  try {
    const events = readJsonlFile(path.join(directory, sessionId, "events.jsonl"), { strict: true, lineFilter: copilotAssistantResponseLineFilter });
    if (isCopilotToolSearchOnlyCompletion(events)) return true;
    return false; // file read succeeded and is authoritative for this session
  } catch (error) {
    const missing = error !== null && typeof error === "object" && "code" in error && error.code === "ENOENT";
    if (!missing) log("Could not inspect Copilot completion events; leaving missing-output validation to the collector");
  }
}

Add a regression test with a non-matching/missing first directory and a matching second directory (per the review thread already on this PR) to lock in the fix — the current test suite only exercises a single COPILOT_HOME directory.

@copilot please address this.

[Copilot]

🟡 Changes recommended

Malformed native-session records can bypass strict parsing and cause incorrect non-retryable failures, while an unregistered failure class drops retry events and tracing attributes.

2 open issues

• Strictly parse all native-session records before filtering · New
• Register copilot search-only completion failure class · New

What changed in this PR

Updates gh-aw’s Copilot execution harness to reject discovery-only responses as successful completion, addressing #72702 without attempting to continue or replay the task.

Changes:

• Detects final empty main-assistant responses containing only completed server-side tool searches, using the exact native session with structured-output fallback.
• Returns a non-retryable failure when no terminal safe output exists, leaving safe-output files and collector validation unchanged.
• Adds tests for session selection, later responses, refusals, subagents, and terminal-output protection.

File	Description
actions/​setup/​js/​copilot_harness.test.cjs	Adds completion-detection and harness exit-behavior tests.
actions/​setup/​js/​copilot_harness.cjs	Adds discovery-only detection and a non-retryable completion guard.

_{Review effort: Max (auto)}