docs-review: OpenRouter + six-criteria rubric + APM skills - #352
theletterf wants to merge 8 commits into
Conversation
Remove every sizing concern from issue-scope: the sizer sub-agent, the Cost & benefit section, effort buckets, ownership, audience, bill of materials, and good-for-ai. Scope now judges issue quality, searches the Elastic docs corpus for affected pages, verifies the request against any linked code, says how to tackle the work, and lists questions to go deeper. Structural changes from the 2026-09-15 analysis: - One agent. The three sub-agents delegated under codex and not under Claude, and the best production comment came from the run that did the work itself. - Docs MCP search is mandatory and first. It had never been called in any run under either engine. - Drop the APM import and its job. The installed skills were never invoked. - Linked code is evidence, not a prerequisite. No PR caps confidence at Medium instead of forcing a partial outcome. - Labels: human-needed on the red quality gate only, create-if-missing false. Drop "Not assessable" — with one agent that is a run failure. Compiled with gh-aw v0.88.7, zero warnings. README and example updated; example drops the dead project-instructions-path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Move the workflow from the Copilot engine to Claude via OpenRouter (anthropic/claude-sonnet-5). Add ANTHROPIC_API_KEY to the secrets manifest; consumers must now pass this secret in their caller workflow. Import review-criteria.md from elastic/elastic-docs-skills at compile time so both the local docs-review-pr skill and this workflow share one authoritative rubric. Structured the prompt around the rubric's six criteria (User focus, Technical accuracy, Applicability, Maintainability, Language, Style) and removed the embedded prose checklists that duplicated what the rubric and APM skills already cover. Updated the review body format to match the skill's six-criteria output. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…skill invocations Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
GPT 5.6 Luna costs ~2× more per run due to higher cache-read pricing on long-prompt workloads, and the Skill tool is not available in the codex engine — both reasons to stay on claude-sonnet-5. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…rmat endpoint OpenRouter's Anthropic-format /v1/messages endpoint requires model IDs without the provider prefix (claude-sonnet-5, not anthropic/claude-sonnet-5). The anthropic/ prefix is for the OpenAI-compatible endpoint only. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Use the gh-aw 'sonnet' alias (OpenRouter resolves it) instead of a hardcoded model ID. Use https://openrouter.ai/api as the base URL — Claude Code SDK appends /v1/messages, giving the correct path. Adding /v1 to the base URL caused a double-path 404. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Example review output from the latest sandbox run (run 35345724174, triggered on docs-actions-sandbox#141 — an APM logs page refresh with real docs-content material):
See the review comments directly on the sandbox PR: https://github.com/elastic/docs-actions-sandbox/pull/141 |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Requesting changes: the docs-review caller secret name in the example and setup docs does not match the reusable workflow input OPENROUTER_API_KEY, which causes authentication failure for consumers following the published setup.
What is this? | From workflow: PR Review
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
| with: | ||
| review-scope: docs-subtree | ||
| secrets: | ||
| ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} |
There was a problem hiding this comment.
[HIGH] Example passes a non-consumed secret name
The reusable workflow consumes secrets.OPENROUTER_API_KEY (mapped to ANTHROPIC_API_KEY internally). Passing ANTHROPIC_API_KEY here means the callee never receives OPENROUTER_API_KEY, so runs fail when callers follow this example and README guidance. Use the same external secret name the workflow expects to keep invocation and docs consistent.
| ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} | |
| OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }} |
Claude Code makes fallback calls to api.anthropic.com during startup; blocking them surfaces a warning in the run logs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Requesting changes due to a blocking workflow setup mismatch in the updated installation guidance.
What is this? | From workflow: PR Review
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
| Add the following to the caller workflow before running this workflow: | ||
|
|
||
| - `permissions.copilot-requests: write` | ||
| - A repo or organization secret named `ANTHROPIC_API_KEY` containing your OpenRouter API key. Pass it with `secrets: inherit` or as an explicit secret in the `uses:` call. |
There was a problem hiding this comment.
[HIGH] Install docs use a non-consumed secret name
This workflow reads ${{ secrets.OPENROUTER_API_KEY }} (mapped to ANTHROPIC_API_KEY internally), so telling callers to define ANTHROPIC_API_KEY causes authentication failures when they follow the install steps. Document OPENROUTER_API_KEY as the required caller secret name to match the reusable workflow contract.
1 participant
Summary
claude-sonnet-5via the Anthropic-format endpoint)elastic/elastic-docs-skills/skills/review/review-pr/references/review-criteria.mdas an inlined fragment, so the workflow and the local review skill share one authoritative sourceSkilltool invocations for all six APM docs skills (docs-check-style,docs-flag-jargon-skill,docs-frontmatter-audit,docs-content-type-checker,docs-applies-to-tagging,docs-check-contradictions) — the agent actively runs them rather than treating them as passive contextTest plan
elastic/docs-actions-sandboxPR Add scoped targeting modes to docs quality sweeps #141 (APM logs refresh content)pre_activation,activation,apm,agent,detection,safe_outputs,conclusion)primary_model: anthropic/claude-sonnet-5confirmed in usage artifact252 AIC per run ($2.50); ~$1.23/run on lighter PRs without heavy cache_writeNotes
The correct OpenRouter setup for the claude engine is:
ANTHROPIC_BASE_URL: https://openrouter.ai/api(no/v1— the SDK appends it)model: sonnet(gh-aw alias; OpenRouter resolves it toanthropic/claude-sonnet-5)Using a hardcoded model ID or appending
/v1to the base URL both produce 404s on OpenRouter's Anthropic-format endpoint.🤖 Generated with Claude Code