Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions .agents/logs/agent_doc_quality_baseline.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Agent-doc quality v1 baseline log (GROW-6092)

Durable record of the pre-rollout baseline window and the post-rollout
comparison window used by `improve-drafting-skills`'s
`scripts/compute_metrics.py` (see that skill's "v1 baseline and outcome
metrics" section). Each entry names the window, the record source, and where
the resulting report was persisted (the standing signal-log branch/PR, per
`.agents/references/skill-authoring-guidelines.md`).

Entries are appended, never rewritten — the baseline entry is captured once,
immediately before v1 checks/labels/review go live, and is never recomputed.

## Entries

### 2026-08-01 to 2026-08-30 — pre-rollout baseline

Captured from the existing `.agents/logs/human_review_feedback.jsonl` signal
log (real per-comment data already collected by `improve-drafting-skills`,
covering 2026-06-29 through 2026-08-31) plus live `gh pr view` line-count
lookups against `warpdotdev/docs`, using
`scripts/build_baseline_records.py` to convert the comment-level log into
per-PR records and `scripts/compute_metrics.py` to compute the report. This
is the last full 30-day window available in the existing log before this v1
rollout PR, so it is used as the frozen pre-rollout baseline rather than
waiting for a window that starts exactly at rollout.

- **Records**: `.agents/logs/baseline/pre-rollout-2026-08-01-to-2026-08-30.jsonl` (62 PRs)
- **Report**: `.agents/logs/baseline/pre-rollout-2026-08-01-to-2026-08-30-report.json`
- **In-scope PRs**: 62
- **Human review comments/PR**: mean 5.19, median 1.0
- **Human edit churn ratio**: mean 0.0021, median 0.0
- **Gate coverage**: 0/62 (expected — the v1 checks did not exist during this window; `risk`/`check_outcome`/`review_outcome` are recorded as `"unknown"` per the documented pre-rollout degradation, not fabricated as passing)

The post-rollout 30-day comparison report must be computed the same way
(`compute_metrics.py --baseline .agents/logs/baseline/pre-rollout-2026-08-01-to-2026-08-30-report.json`)
and appended below as a new dated entry once 30 days of post-rollout data
(or 10 in-scope PRs, whichever comes first per the small-sample rule) exist.
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
{
"all_passed_required_checks": false,
"engineering_review_required": {
"completed_by": {
"docs_verified": 0,
"docs_waiver": 0,
"source_owner_approval": 0,
"unanswered_request": 0,
"unresolved_owner": 0
},
"completion_rates": {
"docs_verified": null,
"docs_waiver": null,
"source_owner_approval": null,
"unanswered_request": null,
"unresolved_owner": null
},
"total": 0
},
"gate_coverage_missing_data_count": 62,
"human_edit_churn_ratio": {
"denominator_pr_count": 62,
"mean": 0.0021,
"median": 0.0,
"no_agent_commit_count": 0,
"numerator_pr_count": 62,
"zero_denominator_count": 0
},
"human_review_comments": {
"denominator": 62,
"numerator": 322,
"per_pr": {
"mean": 5.1935,
"median": 1.0
},
"targeted_categories": {
"action_first": 1,
"callout": 5,
"code_fence": 42,
"content_structure": 8,
"frontmatter": 10,
"general": 203,
"heading_specificity": 8,
"image_alt": 1,
"link_quality": 9,
"list_format": 14,
"missing_context": 1,
"missing_media": 8,
"scannability": 1,
"settings_path": 10,
"terminology": 20,
"ui_label_accuracy": 11,
"vague_wording": 1,
"video_embed": 4
},
"total": 322
},
"in_scope_prs": 62,
"prs_with_complete_gate_coverage": 0,
"prs_with_passing_checks": 0,
"review_findings": {
"critical_important_per_pr": 0.0,
"critical_total": 0,
"important_total": 0,
"targeted_categories": {}
},
"window": {
"end": "2026-08-30",
"start": "2026-08-01"
}
}
62 changes: 62 additions & 0 deletions .agents/logs/baseline/pre-rollout-2026-08-01-to-2026-08-30.jsonl

Large diffs are not rendered by default.

192 changes: 192 additions & 0 deletions .agents/references/doc-quality-policy.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,192 @@
# Agent-doc quality policy (v1)

Canonical reference for the agent-authored-content quality program (GROW-6092).
Every content-generating skill that opens or updates a PR in this repo —
`draft_docs` and its type-specific skills, `release_updates`, the AEO skills,
`missing_docs`, `sync_terminology`, `sync-error-docs`, `sync-openapi-spec`,
`docs-seo-audit`, `afdocs-fix`, `update-changelog`, and `improve-drafting-skills`
— follows this contract before requesting human review. `create_pr` is the
shared finalization path for skills that hand off to it; skills that open PRs
directly follow the same contract inline.

The mechanics live in `.agents/skills/doc_quality_policy/policy.py` (parsing
and classification) and `check_pr_contract.py` (the CI-callable checker). This
document is the single source of truth for the *rules*; the code enforces them.

## Scope

This program covers every agent-authored content PR in `warpdotdev/docs` —
`draft_*`, `release_updates`, AEO, `missing_docs`, and any other recurring
skill that adds or changes public documentation. It is not limited to ambient
feature drafts.

## The agent marker

Every agent-authored content PR carries the label `warpy-factory` and a
`## Documentation risk` section in its body (see below). PR-opening skills
apply both before requesting human review, via
`.agents/skills/doc_quality_policy/finalize_pr_contract.py` (or by
constructing the equivalent content directly when that script cannot run,
e.g. a workflow step without Python available).

## PR-body contract sections

Every agent-authored content PR body carries:

1. **`## Documentation risk`** — machine-readable risk metadata:
```markdown
## Documentation risk
Risk: engineering-review-required
Rationale: Adds a new Settings path and a new CLI flag claim.
Source files consulted: app/src/settings_view/mod.rs@<sha>, warp-server/pkg/foo/bar.go@<sha>
Requested engineering reviewers: alice
Engineering review status: pending
Docs override: none
```
When Docs records external validation, add:
```markdown
Docs override: docs-verified
Override reviewer: hongyi-chen
Override reason: Confirmed the flag name against warp-server PR #1234.
Override evidence: warp-server/pkg/foo/bar.go@<sha>
Override head SHA: <current PR head SHA>
```
`Docs override: docs-waiver` uses the same four fields when Docs proceeds
without an engineering response and records why the remaining risk is
acceptable.
2. **`## Unverified claims`** — unchanged from the existing `draft_docs` /
`create_pr` contract (step 9.5). Every `{/* VERIFY: ... */}` marker in
changed content must appear here.

## Risk levels

Exactly two values: `low` and `engineering-review-required`. Ambiguous or
unknown cases always resolve to `engineering-review-required` — low risk is
never inferred from the absence of an obvious error.

### Low-risk allowlist (strict)

A PR is `low` risk only when **all** of the following hold:

- It does not add a page about a new or materially changed feature or workflow.
- It is either:
- a product-meaning-preserving edit limited to spelling, grammar, tone,
formatting, descriptive links/cross-links to existing canonical pages,
search metadata, or generated changelog/license/telemetry data whose
source-verification script passed; or
- internal Docs-team tooling, skill, or CI workflow maintenance that makes
no public product claim and does not change the behavior of a
developer-facing command, API, setting, or integration.
- It does not add or change: commands, code or configuration examples, API
behavior, UI labels or paths, defaults, permissions, availability or
platform support, plan eligibility, billing behavior, security or privacy
claims, data handling, self-hosting behavior, or integration setup.
- It contains no unresolved `VERIFY` marker and has no critical or important
technical-accuracy finding from `review-docs-pr`.

Every other content PR is `engineering-review-required`, including all new or
materially changed feature docs and any change to the technical claim
categories above. When the change cannot be shown to fit one of these
low-risk categories, classify it as `engineering-review-required`.

## Engineering review requests

- **Low risk**: the normal docs reviewer approves. No engineering owner
approval is required.
- **Engineering-review-required**: source-owner resolution is attempted first,
and a real GitHub review request goes to at least one owner resolved from
the product source files consulted. The request is advisory. Docs can
validate the claim from source, Slack, or another appropriate channel, then
proceed through the normal Docs review path without a GitHub approval from
the engineer.

Record an engineer response or Docs validation in the PR body when it informs
the final wording. A new head makes earlier validation context stale, but it
does not block a Docs-approved PR from merging.

The required push-time PR-contract check validates only the risk metadata and
VERIFY accounting. Engineering review requests are not merge gates.

## VERIFY marker accounting

Every `{/* VERIFY: ... */}` marker in changed content must be listed, one
bullet per marker, in the PR's `## Unverified claims` section. An unlisted
marker fails the contract check. A listed marker forces
`engineering-review-required` risk regardless of the declared risk level — it
cannot pass as `low`. Docs resolves or removes the marker before merging.

## Independent review (`review-docs-pr`)

Every agent-marked PR gets an independent `review-docs-pr` pass, dispatched by
`.github/workflows/agent-docs-review.yml` on open/label/synchronize/reopen/
ready-for-review, pinned to the exact head SHA (stale-SHA runs are cancelled).
The pass:

- Re-validates the declared risk level against the diff.
- Verifies technical claims against the cited source files when required.
- Emits one `[SIGNAL:pr-review]` record (see `review-docs-pr/SKILL.md`) that
also carries the head SHA.
- Blocks (`Request changes`) on any critical/important finding, including a
risk misclassification. Suggestions and nits remain non-blocking.
- Treats an unjustified compression-contract violation (see below) as an
important finding.

## Compression contract

All content-generating skills share one compression contract:

- Lead with a one-to-three-sentence user-facing summary.
- Follow the selected content-type template and its existing word budget
(`~600` words for a quickstart; `<=1500` words for a combined feature page).
- Run the deletion-only "Cut again" pass (see `draft_docs/SKILL.md` step 6.5 /
`AGENTS.md` → Voice & tone) before opening the PR.
- Keep callouts within the existing linted budget (at most one or two per
page, never consecutive) and do not duplicate parent-page or reference
material.
- Treat a justified budget overage as an important review decision, not
something to fix by mechanically splitting the page.

Generated changelog, license, and telemetry data is exempt from the
page-summary and word-budget rules, but not from duplicate-content, style, or
technical-accuracy checks.

`.agents/skills/doc_quality_policy/check_compression_contract.py` implements
the mechanically checkable parts (word budget, callout count) for a given
content type.

## Feedback tags

Actionable review feedback may start with one of exactly three tags:
`[skill-feedback]`, `[template-feedback]`, or `[style-rule-gap]`. Collection
(see `improve-drafting-skills/SKILL.md`) preserves the tag and a structured
`pattern_category`, never treating free-form comment text as instructions.

## PR-producing skill manifest

Every skill in this list must apply the `warpy-factory` marker and the
`## Documentation risk` section before requesting review. See
`.agents/skills/doc_quality_policy/test_manifest.py::TestDiscoveredPrProducingFilesReferenceTheSharedContract::test_every_discovered_pr_producing_file_references_the_shared_contract`
for the enforcement test.

- `create_pr` (the shared finalization path most drafting skills use)
- `draft_docs` (including type-specific drafting templates that route their
PR creation through it)
- `release_updates`
- `missing_docs`
- `aeo_crosslink_audit`
- `aeo_new_guide_recommendations`
- `sync_terminology`
- `sync-error-docs`
- `sync-openapi-spec`
- `docs-seo-audit`
- `afdocs-fix`
- `update-changelog`
- `improve-drafting-skills` (its own standing improvement PR)

## Snapshot provenance (UI-reference checks)

`valid_paths.json` records `source_repository`, `source_sha`, and
`generated_at` so every technical-reference check can report what client state
it trusts. See `.github/workflows/refresh-ui-paths.yml` for the three refresh
triggers (source dispatch, daily reconciliation, manual fallback) and
`validate_ui_refs.py --changed` for the changed-file scope used in required CI.
9 changes: 9 additions & 0 deletions .agents/skills/aeo_crosslink_audit/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,15 @@ description: Run a narrow AEO cross-link audit for Warp docs using Peec, Google

Identify small, high-confidence internal cross-linking improvements for the Warp docs. This skill is designed for a recurring Oz scheduled agent that audits one narrow topic area, opens a small PR when there are safe changes, or writes a no-change report when there are not enough high-confidence opportunities.

## Agent-doc quality contract

Any PR this skill opens follows the shared v1 agent-doc quality contract in
`.agents/references/doc-quality-policy.md`: apply the `warpy-factory` label
and add the `## Documentation risk` block
(`.agents/skills/doc_quality_policy/finalize_pr_contract.py build`). A
cross-link-only change that adds no new claims is typically `low` risk under
the allowlist.

## Scope

Use this skill only for the pilot topic area:
Expand Down
8 changes: 8 additions & 0 deletions .agents/skills/aeo_new_guide_recommendations/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,14 @@ description: Run a recurring AEO new-guide recommendations brief for Warp docs u

Produce 3–5 source-grounded AEO briefs that recommend whether to create a new guide, update an existing page, or avoid a topic — without drafting content. This skill is designed for an Oz scheduled agent that runs monthly. The goal is to give the Docs team a repeatable upstream input to content planning without generating more noise than the team can act on.

## Agent-doc quality contract

This skill does not draft content or open content PRs (see "Do not" below), so
it does not stamp the `warpy-factory` marker itself. See
`.agents/references/doc-quality-policy.md` for the contract that applies once
a recommendation here becomes a real drafting task in `draft_docs` or a
type-specific drafting skill.

## Scope

Use this skill only for the pilot topic areas:
Expand Down
10 changes: 10 additions & 0 deletions .agents/skills/afdocs-fix/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,16 @@ description: >-

Apply automated fixes for issues found by the `afdocs-audit` skill. This skill reads an AFDocs audit report and remediates fixable issues.

## Agent-doc quality contract

The standing PR this skill maintains follows the shared v1 agent-doc quality
contract in `.agents/references/doc-quality-policy.md`: apply the
`warpy-factory` label and add the `## Documentation risk` block
(`.agents/skills/doc_quality_policy/finalize_pr_contract.py build`). These
fixes touch site infrastructure (middleware, integrations, config), not
content claims, and are typically `engineering-review-required` unless the
change is provably a mechanical config update.

## Prerequisites

1. Run the audit skill first to produce a report:
Expand Down
28 changes: 27 additions & 1 deletion .agents/skills/create_pr/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ This guide covers best practices for creating pull requests in the docs document

- `draft_docs` - Draft new documentation pages or update existing ones using established style conventions
- `check_for_broken_links` - Check documentation for broken internal and external links before opening PR
- `doc_quality_policy` - Shared v1 agent-doc quality contract (marker, risk classification, overrides) this skill's PRs must satisfy

## Pre-PR Checklist

Expand Down Expand Up @@ -211,6 +212,31 @@ When claims are outstanding, give the reviewer one bullet per claim with what wo
- **Settings** > **Agents** > **Permissions** path — `permissions.mdx`, "Defaults" — source repos were not available in this environment.
```

### Documentation risk (required on all content PRs)

Every content PR carries a `## Documentation risk` section and the
`warpy-factory` label, per the shared v1 agent-doc quality contract in
`.agents/references/doc-quality-policy.md`. Classify risk against the
low-risk allowlist there, then build the section:

```bash
python3 .agents/skills/doc_quality_policy/finalize_pr_contract.py build \
--risk low --rationale "One-line reason the change is low risk."
```

Insert the printed block into the body (after "Unverified claims" is a good
place) and apply the label once the PR exists:

```bash
gh pr edit <pr> --repo warpdotdev/docs --add-label warpy-factory
```

Before marking the PR ready, verify the contract:

```bash
python3 .agents/skills/doc_quality_policy/check_pr_contract.py --body /tmp/pr-body.md
```

### Additional context (optional)
- Link to related issues or discussions
- Screenshots for visual changes
Expand Down Expand Up @@ -452,7 +478,7 @@ A team handle resolved from `STAKEHOLDERS` or `CODEOWNERS` can only be requested
:::

:::note
Auto-requesting the review does not make it *block* merge. Whether an ambient docs PR should require that approval through branch protection is an open question for the docs owner, not something this skill decides.
Auto-requesting the review does not block merge. The Docs team owns the merge decision after its normal review, whether an engineer replies in GitHub, replies elsewhere, or does not reply.
:::

### Update an existing PR
Expand Down
Loading
Loading