Skip to content

chore(harness): inject PR template + review-handling guidance#2214

Open
richardsolomou wants to merge 4 commits into
mainfrom
posthog-code/pr-template-and-review-hooks
Open

chore(harness): inject PR template + review-handling guidance#2214
richardsolomou wants to merge 4 commits into
mainfrom
posthog-code/pr-template-and-review-hooks

Conversation

@richardsolomou
Copy link
Copy Markdown
Member

@richardsolomou richardsolomou commented May 19, 2026

Problem

The PostHog Code cloud harness — i.e. the system prompt produced by AgentServer#buildCloudSystemPrompt in packages/agent/src/server/agent-server.ts — instructs the agent to open PRs with a generic ## Summary / ## Test plan body. It doesn't:

  • check the repo for a .github/pull_request_template.md (or the org's .github repo template) and use it as the body,
  • spell out how to handle PR review comments after pushing fixes (so threads get silently re-pushed with no reply or resolution), or
  • search the repo for matching open issues to link via Closes #N / Refs #N so GitHub auto-links/auto-closes them.

The first two commits on this branch tried to solve the first two by adding a Claude-Code SessionStart hook under .claude/. That was the wrong target: the PostHog Code harness is agent-server.ts, not Claude Code's local config.

Refs #1642 — that issue tracks the same area but is about the SDK firing SessionStart; out of scope here.

Changes

packages/agent/src/server/agent-server.ts#buildCloudSystemPrompt — augment each branch of the cloud system prompt:

  • Auto-PR, no existing PR. Insert a "Before opening the PR, prepare the body" step that tells the agent to (a) detect a repo-level PR template at .github/pull_request_template.md and the usual variants, (b) fall back to the org's .github repo via gh api, and (c) run gh issue list --state open --search for matching open issues and include Closes #<n> (or Refs #<n>) lines in the body.
  • Auto-PR, existing PR. After "push to the existing PR branch", add a step requiring the agent to reply on each PR review thread it addressed (gh api -X POST /repos/{owner}/{repo}/pulls/{n}/comments/{id}/replies) and resolve the conversation via the resolveReviewThread GraphQL mutation, with a hint to first list unresolved threads via the reviewThreads(first:100) GraphQL query so the agent targets the right ones.
  • No-repository mode. Extend the "if the user asks for a PR" line to apply the same template + issue-link guidance to the freshly cloned repo.

packages/agent/src/server/agent-server.test.ts — extend the existing buildCloudSystemPrompt cases to assert the new content for each branch (PR template path, related-issue search, review-thread reply + resolve, no-repo guidance). Earlier commits' .claude/ hook attempt is reverted.

How did you test this?

  • pnpm --filter agent exec vitest run src/server/agent-server.test.ts — 40 / 40 pass once POSTHOG_CODE_INTERACTION_ORIGIN is unset (the cloud-task shell preset it to slack, which leaks into vitest and breaks three pre-existing tests that depend on the env being clean — unrelated to this change).
  • pnpm --filter agent typecheck — passes.
  • lint-staged ran biome + repo-wide typecheck on commit and passed.
  • Dogfooded the issue-linking step against this PR: gh issue list --search across pr template, review thread, SessionStart, link issue, pull request template, gh pr create, etc. — ensure claude SDK runs SessionStart hook #1642 is the only related open issue and it's a dependency, so it's linked as Refs rather than Closes.

Publish to changelog?

no


Created with PostHog Code

Injects the repo's PR template (or the org's `.github` repo template as a
fallback) into agent context at session start, and adds guidance to always
reply and resolve PR review threads when addressing review comments.

Generated-By: PostHog Code
Task-Id: 041ddd4c-ea89-4cd1-9697-cdae40e30a3f
Extend the SessionStart hook context with guidance to search for open
issues that match the branch's work via `gh issue list --search` and
include `Closes #N` / `Refs #N` links in the PR description.

Generated-By: PostHog Code
Task-Id: 041ddd4c-ea89-4cd1-9697-cdae40e30a3f
@richardsolomou richardsolomou requested a review from a team May 19, 2026 04:33
@richardsolomou richardsolomou marked this pull request as ready for review May 19, 2026 04:33
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented May 19, 2026

Prompt To Fix All With AI
Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
.claude/hooks/check-pr-template.sh:46
The `base64 -d` flag is Linux/GNU-specific. On macOS the native `base64` utility uses `-D` (uppercase) for decode; passing `-d` prints an error and exits non-zero. Because `pipefail` is set, the pipe fails and the `|| true` catch suppresses it silently, so the org-level `.github` template fallback never activates on macOS developer machines. Using `--decode` (GNU) plus a `-D` fallback makes the line portable.

```suggestion
      content=$(gh api "/repos/${owner}/.github/contents/${path}" --jq '.content' 2>/dev/null | (base64 --decode 2>/dev/null || base64 -D 2>/dev/null) || true)
```

Reviews (1): Last reviewed commit: "chore(harness): instruct agent to link m..." | Re-trigger Greptile

Comment thread .claude/hooks/check-pr-template.sh Outdated
@richardsolomou richardsolomou marked this pull request as draft May 19, 2026 04:36
@richardsolomou richardsolomou added the Create Release This will trigger a new release label May 19, 2026
…the cloud system prompt

Wrong target on the previous two commits: the agent harness is
`packages/agent/src/server/agent-server.ts#buildCloudSystemPrompt`, not
Claude Code's `.claude/settings.json`. Revert the Claude-Code hook attempt
and put the guidance in the actual cloud task system prompt instead.

- Auto-PR (no existing PR) path: instruct the agent to check
  `.github/pull_request_template.md` (and variants) for a body template,
  fall back to the org's `.github` repo via `gh api`, and search for
  matching open issues with `gh issue list --search` to include
  `Closes #N` / `Refs #N` links in the body.
- Auto-PR (existing PR) path: instruct the agent to reply on each PR
  review thread it addressed and resolve the conversation via the
  `resolveReviewThread` GraphQL mutation, and to list unresolved threads
  first so it can target the ones it fixed.
- No-repo path: add a brief reminder to apply both behaviors when the
  user explicitly asks for a PR from a freshly cloned repo.
- Extend agent-server tests to lock in the new prompt content for each
  branch (`gh pr create` path, existing-PR path, no-repo path).

Generated-By: PostHog Code
Task-Id: 041ddd4c-ea89-4cd1-9697-cdae40e30a3f
@richardsolomou richardsolomou marked this pull request as ready for review May 19, 2026 04:57
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented May 19, 2026

Prompt To Fix All With AI
Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
packages/agent/src/server/agent-server.ts:1638
The `\\$id` escape in a TypeScript template literal produces the literal string `\$id` in the prompt. Inside a bash single-quoted string, backslash is **not** an escape character, so `gh` receives the query with `\$id` as the variable name — but GraphQL requires `$id` (without the backslash). The command would fail with a parse error at runtime. Since TypeScript template literals only interpolate `${...}` (with curly braces), a bare `$id` is already treated as literal text and needs no escaping.

```suggestion
   - Resolve the thread via the \`resolveReviewThread\` GraphQL mutation: \`gh api graphql -f query='mutation($id:ID!){resolveReviewThread(input:{threadId:$id}){thread{isResolved}}}' -f id="<thread-node-id>"\`.
```

Reviews (2): Last reviewed commit: "chore(harness): move PR template + revie..." | Re-trigger Greptile

Comment thread packages/agent/src/server/agent-server.ts Outdated
…ippet

A TypeScript template literal only interpolates ${...}; a bare $id is
already literal text. The `\\$id` escape produced a stray backslash in
the prompt — bash single quotes preserve it verbatim, so gh would send
`mutation(\$id:ID!)` to GraphQL and fail with a parse error.

Generated-By: PostHog Code
Task-Id: 041ddd4c-ea89-4cd1-9697-cdae40e30a3f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Create Release This will trigger a new release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant