Skip to content

docs: fix inaccurate SDK/runtime claims found in docs audit - #2064

Merged
SteveSandersonMS merged 4 commits into
mainfrom
docs/fix-audit-findings
Jul 28, 2026
Merged

docs: fix inaccurate SDK/runtime claims found in docs audit#2064
SteveSandersonMS merged 4 commits into
mainfrom
docs/fix-audit-findings

Conversation

@patniko

@patniko patniko commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Ran a documentation audit that cross-checked concrete claims in docs/ against both the SDK repo and the copilot-agent-runtime repo (source of truth for protocol/CLI behavior). This PR fixes the five high-confidence, verified inaccuracies. Each was re-verified against source before editing.

Fixes

Doc Was Now (verified)
setup/local-cli.md, setup/bundled-cli.md "Go SDK does not bundle a CLI, must always provide Connection" Go embeds a CLI via go tool bundler and reads COPILOT_CLI_PATH (go/README.md:99-134, go/types.go:43)
setup/bundled-cli.md (Java) configure via Connection or COPILOT_CLI_PATH Java uses setCliPath / setCliUrl, or copilot on PATH (CopilotClientOptions.java:172,202); no Connection/env var
hooks/user-prompt-submitted.md rate-limit example returns reject / rejectReason output type only has modifiedPrompt/additionalContext/suppressOutput (go/types.go:672-676) — example rewritten
auth/byok.md Azure apiVersion "default 2024-10-21" omitted uses GA versionless v1 route (runtime/generated/api.schema.json:22541-22543)
features/custom-agents.md "parent session effort is not inherited" runtime inherits parent effort for same-model subagents (runtime #13388)

Validation

  • Re-ran just validate-docs-extract (only changed TS sample is a partial block; .validation/ is git-ignored).
  • just validate-docs-ts -> 178 files pass.

Not included (follow-ups for separate PRs)

The audit also surfaced larger gaps that need net-new docs rather than corrections:

  • Undocumented SDK features: tool-search config, typed ExP assignments, session canvasProvider, in-process FFI transport, enableManagedSettings, tool metadata passthrough.
  • Undocumented runtime surface: ~64 session event types not in docs, managed MCP policy (allowedMcpServers/deniedMcpServers), sidekick events, sandbox auth injection, plugin Rust port / Open Plugin Spec v1.
  • Minor additions: OTel COPILOT_OTEL_* env vars, protocol version / bundled CLI version in troubleshooting/compatibility.md, compaction events in session docs.

Two audit findings were investigated and intentionally not changed (false positives): the error-handling.md errorType claim (doc is already correct) and the mcp-debugging.md MCP_DEBUG/NODE_DEBUG env vars (those configure the user's own MCP server, not a runtime toggle).

Cross-checked doc claims against the SDK and copilot-agent-runtime
repos and corrected five verified inaccuracies:

- setup/local-cli.md, setup/bundled-cli.md: the Go SDK is not
  bundle-less — it embeds a CLI via `go tool bundler` and reads
  COPILOT_CLI_PATH; corrected the "must always provide Connection"
  guidance.
- setup/bundled-cli.md: Java configures the CLI via setCliPath /
  setCliUrl (or `copilot` on PATH), not `Connection` /
  COPILOT_CLI_PATH.
- hooks/user-prompt-submitted.md: the hook output has no
  reject/rejectReason; replaced the rate-limit example and
  best-practice bullet with the real additionalContext behavior.
- auth/byok.md: omitted Azure apiVersion uses the GA versionless v1
  route, not a 2024-10-21 default.
- features/custom-agents.md: the runtime now inherits parent
  reasoning effort for same-model subagents (#13388); corrected the
  "parent effort is not inherited" statement.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: af5056fb-e73c-4bc2-bc11-753fd633cb98
Copilot AI review requested due to automatic review settings July 23, 2026 18:32
@patniko
patniko requested a review from a team as a code owner July 23, 2026 18:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Corrects verified documentation inaccuracies across CLI setup, hooks, BYOK configuration, and custom-agent behavior.

Changes:

  • Clarifies Go and Java CLI resolution.
  • Corrects hook and Azure API behavior.
  • Documents reasoning-effort inheritance.
Show a summary per file
File Description
docs/setup/local-cli.md Updates Go CLI setup guidance.
docs/setup/bundled-cli.md Corrects Go and Java CLI behavior.
docs/hooks/user-prompt-submitted.md Replaces unsupported prompt rejection fields.
docs/features/custom-agents.md Clarifies reasoning-effort resolution.
docs/auth/byok.md Corrects Azure API-version routing.

Review details

Comments suppressed due to low confidence (1)

docs/hooks/user-prompt-submitted.md:495

  • additionalContext cannot gate a request; it only influences the model after the request has already proceeded. Use advisory wording such as “redirect” rather than retaining another implication of enforcement. The replacement also uses the colon required for labeled list items in docs/.
1. **Prefer `additionalContext` for soft controls** - This hook cannot hard-reject a prompt; to discourage or gate a request, add context explaining the constraint rather than expecting a block.
  • Files reviewed: 5/5 changed files
  • Comments generated: 5
  • Review effort level: Medium

Comment thread docs/setup/local-cli.md Outdated
Comment thread docs/setup/bundled-cli.md Outdated
Comment thread docs/hooks/user-prompt-submitted.md Outdated
Comment thread docs/auth/byok.md Outdated
Comment thread docs/features/custom-agents.md Outdated
Document the Go PATH fallback, reframe the hook example as advisory, and correct Azure versionless routing references across SDK docs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 28, 2026 10:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

Comments suppressed due to low confidence (3)

docs/hooks/user-prompt-submitted.md:439

  • Record the current submission before testing the threshold. As written, the tenth prompt produces no notice, and prompts that receive a notice are never added to promptTimestamps, so the counter stops representing actual usage. Push now first and report the actual count.
      if (promptTimestamps.length >= NOTICE_THRESHOLD) {
        // This is advisory context for the model, not an enforced rate limit.
        // Enforce hard limits before calling session.send().
        return {
          additionalContext: `The user has sent ${NOTICE_THRESHOLD} prompts in the last minute. Suggest waiting before sending more.`,

docs/features/custom-agents.md:262

  • This correction leaves all public SDK API documentation stating the opposite: nodejs/src/types.ts:1646-1647, python/copilot/session.py:1083-1084, go/types.go:952-954, dotnet/src/Types.cs:2661-2664, rust/src/types.rs:645-648, and java/src/main/java/com/github/copilot/rpc/CustomAgentConfig.java:298-302 still say the parent effort is not inherited. Update those comments too so users relying on IDE/API docs receive the corrected same-model behavior.
Set `model` and `reasoningEffort` to override the parent session's model settings while a custom agent runs. When `reasoningEffort` is omitted, the SDK sends no per-agent override and the runtime resolves the effort from its own precedence: a per-call client option, the resolved model's default, or the agent definition all take priority; otherwise the runtime inherits the parent session's effort only when the subagent runs the same model as the parent. When the subagent resolves to a different model, it falls back to that model's default instead of inheriting the parent's effort. Python uses `reasoning_effort`, .NET uses `ReasoningEffort`, Go uses `ReasoningEffort`, Java uses `setReasoningEffort`, and Rust uses `with_reasoning_effort`.

docs/hooks/user-prompt-submitted.md:494

  • Use a colon between the bold label and its description, as required by .github/instructions/docs-style.instructions.md:79-82.
1. **Use `additionalContext` for advisory guidance** - This hook cannot reject a prompt or enforce policy. Enforce hard limits before calling `session.send()`.
  • Files reviewed: 12/12 changed files
  • Comments generated: 0 new
  • Review effort level: Medium

@github-actions

This comment has been minimized.

Correct the advisory threshold example and align Azure routing and reasoning-effort API documentation across all six SDKs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 28, 2026 10:54

@SteveSandersonMS SteveSandersonMS left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the documentation corrections against the SDK and runtime implementations. All review feedback is addressed consistently across the six SDKs.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

Comments suppressed due to low confidence (4)

docs/setup/local-cli.md:5

  • This still implies that using the Go SDK is itself a reason to supply a local CLI explicitly. NewClient(nil) can use a bundled CLI or resolve copilot from PATH, so Go users do not need this setup unless they want to override that resolution. Please narrow this guidance to the override/pinning case.
**Use when:** You need to pin a specific CLI version, or work with the Go SDK (which does not include a CLI automatically).

docs/setup/bundled-cli.md:148

  • This note is prose inside a <details> language tab. The docs pipeline requires consecutive language tabs to contain only a code fence (plus an optional validation marker); prose here prevents the group from normalizing correctly. Move this note outside the tab group and label it **(Java)**.
> [!NOTE]
> The Java SDK does not bundle or embed the Copilot CLI. Install the CLI separately and either make `copilot` available on your `PATH` or set its location with `setCliPath(...)` (or connect to a running CLI server with `setCliUrl(...)`).

docs/setup/local-cli.md:81

  • This note is prose inside a <details> language tab. The docs pipeline requires consecutive language tabs to contain only a code fence (plus an optional validation marker); prose here prevents the group from normalizing correctly. Move this note outside the tab group and label it **(Go)**.
> [!NOTE]
> The Go SDK does not ship a CLI automatically. Install `copilot` on `PATH`, set the `COPILOT_CLI_PATH` environment variable, embed a CLI with the [bundler tool](../../go/README.md#distributing-your-application-with-an-embedded-github-copilot-cli), or point `StdioConnection.Path` at an installed binary.

docs/setup/bundled-cli.md:82

  • This note is prose inside a <details> language tab. The docs pipeline requires consecutive language tabs to contain only a code fence (plus an optional validation marker); prose here prevents the group from normalizing correctly. Move this note outside the tab group and label it **(Go)**.

This issue also appears on line 147 of the same file.

> [!NOTE]
> Unlike Node.js, Python, and .NET, the Go SDK does not include a CLI as an automatic dependency. With no explicit path, `NewClient(nil)` uses an embedded CLI when available, then falls back to `copilot` on `PATH`. To embed a CLI, run the [bundler tool](../../go/README.md#distributing-your-application-with-an-embedded-github-copilot-cli) at build time. You can also set `COPILOT_CLI_PATH` or point a `Connection` at an existing binary. See [Local CLI Setup](./local-cli.md) for details.
  • Files reviewed: 15/15 changed files
  • Comments generated: 0 new
  • Review effort level: Medium

@github-actions

This comment has been minimized.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 28, 2026 11:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review details

  • Files reviewed: 15/15 changed files
  • Comments generated: 0 new
  • Review effort level: Medium

@github-actions

Copy link
Copy Markdown
Contributor

Cross-SDK Consistency Review ✅

This PR updates documentation and comments across all six SDK implementations (Node.js/TypeScript, Python, Go, .NET, Java, Rust) for two related changes:

  1. Azure API version behavior (AzureProviderOptions.apiVersion): All SDKs now correctly document that when the field is omitted, the runtime uses the GA versionless v1 route (removing the previously inaccurate default value of "2024-10-21").

  2. Custom agent reasoning effort resolution: All six SDKs consistently update the CustomAgentConfig.reasoningEffort / reasoning_effort / ReasoningEffort comments to describe the runtime precedence: resolves from model configuration, inherits parent effort only when subagent runs the same model.

All language implementations are updated in parallel. API surface and semantics are consistent across languages, accounting for appropriate naming conventions (camelCase for TS/Java, snake_case for Python/Rust, PascalCase for Go/.NET). No cross-SDK inconsistencies found.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by SDK Consistency Review Agent for #2064 · 18.8 AIC · ⌖ 5.19 AIC · ⊞ 5.2K ·

@SteveSandersonMS
SteveSandersonMS added this pull request to the merge queue Jul 28, 2026
Merged via the queue into main with commit 5a887d7 Jul 28, 2026
57 of 58 checks passed
@SteveSandersonMS
SteveSandersonMS deleted the docs/fix-audit-findings branch July 28, 2026 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants