docs: extend AGENTS.md with conventions mined from PR reviews

[julian-risch]

Related Issues

• none

Proposed Changes:

Our AGENTS.md documents the mechanics — hatch, tests, mypy, release notes — but none of the judgement reviewers actually apply. That knowledge lives only in review comments, so contributor agents rediscover it one review round at a time.

This adds 117 rules mined with pydantic/braindump (see Scaling open source with AI) from 2,746 PR review comments written by 14 deepset reviewers between 2025-07-01 and 2026-08-14. The hand-written sections of AGENTS.md are untouched; the mined rules follow underneath.

Layout. Rules are split by scope rather than piled into one file, so agents load them lazily — only when work touches that directory:

File	Rules	Loaded
AGENTS.md	70	always (~2.8k tok, down from 5.0k as one file)
test/AGENTS.md	17	when touching tests
docs-website/docs/AGENTS.md	11	when editing component docs
releasenotes/notes/AGENTS.md	8	when writing a release note
docs-website/AGENTS.md	6	when editing the docs site
haystack/hooks/compaction/AGENTS.md	4	when touching compaction

CLAUDE.md becomes a one-line @AGENTS.md import, and each directory with an AGENTS.md gets the same. Claude Code reads CLAUDE.md, not AGENTS.md (docs), so the import is what puts the guidance in context. Codex and Cursor read AGENTS.md directly, so it stays the single source of truth.

Verified with canary fixtures (claude -p, file tools disabled so the model can only answer from loaded context):

Setup	Canary in context?
CLAUDE.md = @AGENTS.md	yes
AGENTS.md alone, no CLAUDE.md	no — Claude Code does not read AGENTS.md on its own
CLAUDE.md = the old prose, tools disabled	no — nothing is preloaded
nested sub/CLAUDE.md = @AGENTS.md, cwd at root, after reading sub/thing.py	yes — loads lazily, which is what the split relies on
nested, cwd inside sub/	yes

For fairness: with Read enabled the old prose form did work — the model went and read AGENTS.md in 3/3 runs. The import's advantage is that it is deterministic and costs no tool-call round-trip, not that the old form was broken.

Pipeline: download → extract (1 LLM call per comment) → embed & cluster → dedupe → place → group → generate. 2,643 candidate generalisations collapsed into 442 deduplicated rules, of which 117 survived scoring and review. Rules seen in only one PR are penalised, so what remains is what reviewers say repeatedly.

Handling the 3.0 boundary. The corpus has to span 13 months to be large enough — only ~220 team review comments exist since 3.0 shipped on 2026-07-20 — so most of it is 2.x-era, and a naive run would tell agents to use OpenAIGenerator. Rules were therefore filtered against the current tree before inclusion, deterministically rather than by asking a model:

• A git diff of v2.31.0..HEAD over git ls-files (not the filesystem — the deleted component directories still exist on disk holding __pycache__) yields 75 removed symbols: 45 that moved to haystack-core-integrations, 30 gone entirely.
• Each rule's identifiers are checked against that set. Verdicts: 395 current, 45 rewritten, 2 stale. Removed symbols appearing only in a rule's example never condemn the rule — the reviewer just happened to be looking at that component.
• Rewrites keep the insight and drop the dead name, e.g. "For AsyncPipeline.stream()-style APIs, define, document and test the exact error semantics…" → "For Pipeline streaming APIs, …".
• A recency weighting (v3-era share plus a 300-day half-life on the newest evidence) demotes rules whose only support is old.
• Six hand-written rules state 3.0 conventions explicitly, so ToolInvoker and AsyncPipeline appear exactly twice in the file — in the two rules that warn against them.

Five rules removed and nine generalised on review. Clustering preserves whichever example the source review comments happened to discuss, so several rules read as though they only applied to one class or path — e.g. "preserve all runtime config, including Watsonx max_retries", or an assertion stated only for DocumentSplitter when core ships 11 splitter classes. Those now state the convention, keeping the specific case only where it illustrates it.

Four of the removals were about rule quality rather than scope. One was a genuine defect: "Use typing.Union[...] instead of | unions until Python 3.10+ is required" contradicted the rule three lines above it and is factually wrong — requires-python is already >=3.10, and 106 modules use PEP 604 unions against 12 using Optional[. The staleness gate could not catch it: it diffs symbols, and this is a convention change with no symbol footprint. The other three ("Revert review-rejected changes exactly", "Compute values once per function", "Delete lines explicitly marked to remove") carry no project-specific information.

How did you test it?

Markdown-only change, so no test suite applies. Verified instead:

• No stale rule leaked. Every rule joined back to its verdict: 114 current, 8 rewritten, 0 stale, 0 unknown ids.
• Symbol sanity. All 185 backticked identifiers and 5 file paths across the six files resolve in the current tree, except ToolInvoker and AsyncPipeline in the two deliberate warnings.
• Stale bucket read by hand. Small enough (2 rules) to inspect individually; both genuinely name removed classes in their core text.
• Extraction quality gated by a 50-comment smoke test before committing to the full run.

Notes for the reviewer

• What to actually review: the rules themselves. Skim for anything you disagree with — a rule you would not enforce in review should not be here. The files carry no provenance metadata: judge each rule on whether it reads as a convention this project actually holds, not on where it came from.
• The rules are weighted toward the highest-volume reviewers (sjrl ~1,000 actionable comments, anakin87 ~480), so they reflect their emphases more than the team average.
• Verification limitation worth knowing: every recent PR is inside the training corpus, so I could not test against held-out reviews. Rules do generalise across PRs by construction, but "would this have pre-empted comment X" was not independently measurable.
• The gate is deliberately biased toward keeping rules: a false stale silently deletes real knowledge, a false current is one dated bullet you will catch here.
• Tooling (braindump patches, staleness gate, merge script) lives in a local braindump fork, not in this repo. Happy to push it somewhere shared if useful for re-running.

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have updated the related issue with new insights and changes.
• I have added unit tests and updated the docstrings. — documentation-only change.
• I've used one of the conventional commit types for my PR title: docs:.
• I have documented my code.
• I have added a release note file — not user-facing; AGENTS.md is contributor documentation.
• I have run pre-commit hooks and fixed any issue.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
haystack-docs	Ready	Preview	Aug 15, 2026 9:51am