create-ol-pull-request: audit factual claims before opening the PR - #258
Open
blarghmatey wants to merge 2 commits into
Open
create-ol-pull-request: audit factual claims before opening the PR#258blarghmatey wants to merge 2 commits into
blarghmatey wants to merge 2 commits into
Conversation
Several past PRs shipped confident-but-wrong claims (a metrics window too narrow to be meaningful, a wrong framework-default assertion, a mischaracterized IAM denial) that reviewers had to catch and force a retraction on. Add a pre-submit step that lists every factual/behavioral claim in the drafted body, checks each against its strongest evidence source (Prometheus/Grafana, library source, deployed infra state, or a failing-then-passing test), and requires dropping or correcting anything that can't be verified — before the PR opens, not after a reviewer flags it. Skips cleanly for mechanical changes with nothing to audit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XqAaVVSbbp412kVEYbKY1K
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a factual-claim audit before creating mitodl pull requests.
Changes:
- Verifies behavioral claims against authoritative evidence.
- Renumbers PR creation to Step 6.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
blarghmatey
added a commit
to blarghmatey/dotfiles
that referenced
this pull request
Aug 19, 2026
Pairs with the create-ol-pull-request skill's new claim-audit step (mitodl/agent-kit#258): the first attempt at any `gh pr create` command is blocked with a reminder to verify factual claims in the body first. Re-running the identical command lets it through — a one-time speed bump, not a permanent block, since the hook has no way to know the audit actually happened. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XqAaVVSbbp412kVEYbKY1K
Copilot's review of the sibling PR (#251) made the point against the same wording this table carries: seven days proves absence during seven days, so a flat window still lets through the confident-but-wrong assertions the claim audit exists to catch. Keep seven days as the floor for trend claims, where the window is there to stop a blip reading as a trend, and require an absence claim to either cover the period it names or be narrowed to the window actually queried. Matches the corrected wording landing in #251. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CqvGU3ssvASjwcfFGExNPL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What are the relevant tickets?
N/A
Description (What does it do?)
create-ol-pull-request, run beforegh pr create: extract every factual/behavioral claim from the drafted body (production behavior, metrics, library defaults, "this fixes X"), verify each against its strongest evidence source, and mark it VERIFIED / UNVERIFIABLE / CONTRADICTED.toolhive-swe-{ci,qa,prod}tier, ≥7-day window), the actual library source/docs, deployed infra state (pointing at thedeploy-verificationskill for rollout claims), or a failing-then-passing test for "this fixes X" claims.descriptionto mention the audit step.How can this be tested?
npx skills-ref validate ./skills/process/create-ol-pull-request— passes.prek run --files skills/process/create-ol-pull-request/SKILL.md— all checks pass (markdownlint, trailing whitespace, secrets, etc.).