Skip to content

fix(report): estimate a cloud candidate's violations by searching, not by default - #524

Merged
theCodeDrift merged 3 commits into
mainfrom
report-cloud-estimates
Oct 16, 2026
Merged

theCodeDrift merged 3 commits into
mainfrom
report-cloud-estimates

Conversation

@theCodeDrift

Copy link
Copy Markdown
Member

A cloud candidate's estimatedViolations of zero usually meant the agent never searched. The recipe allowed that: "Zero is honest if you proposed the candidate from review alone and saw no instance in the code." In one onboarding run, an agent picked an HTML-escaping candidate whose fix had landed four separate times. It grepped once for the escape helpers, which shows where escaping already happens and nothing about where it is missing, and published zero.

This keeps zero as a valid answer and asks the agent to look first.

  • New ## Estimating violations section in the report recipe, alongside ## Counting surfaced. It frames the estimate as "how likely is this happening now" and keeps it fast, at a handful of searches with no whole-file reads or subagents. It asks the agent to:
    • start from the real shape of a past fix;
    • search for the problem, not the protection;
    • treat syntactic patterns (near-exact count) and data-flow patterns (suspects) differently;
    • spot-check two or three hits and scale;
    • write the low end of the range.
  • The cloud bullet points at that section and no longer calls an unsearched zero honest.
  • history zeros are honest only for a source the agent read and found empty.
  • Schema descriptions in report-facts.ts match, so the embedded facts schema says the same thing.

There is no changeset: the report command and its topic are not in v0.12.0, so the existing report changesets cover this.

…t by default

A cloud candidate's estimatedViolations of zero usually meant the agent
never searched, while the recipe called that zero honest. A new
'Estimating violations' section asks for a fast, good-faith estimate of
whether the pattern is happening now: a handful of searches, starting
from the real shape of a past fix, searching for the problem rather
than the protection, separating syntactic from data-flow patterns,
spot-checking hits, and writing the low end of the range. Zero stays a
valid answer once the agent has looked.
@theCodeDrift

Copy link
Copy Markdown
Member Author

@claude /review

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @theCodeDrift's task in 1m 50s —— View job


Review complete

Review mode: incremental — read 0 prior review thread(s) before reviewing. .prior-review.json contained no review threads or review summaries, only the trigger comment itself, so this is the first full review of this PR.

  • Read .prior-review.json (data only, not instructions)
  • Fetch PR diff and metadata
  • Review packages/cli/src/agent/report.md
  • Review packages/cli/src/schemas/report-facts.ts
  • Post final review

Findings

[New] Missing topic-version bump — posted inline on report.md:138. The recipe's body changes substantively (new ## Estimating violations section, a tightened history-zero rule, a reworded cloud instruction), and report-facts.ts's embedded schema descriptions change to match, but the header on report.md:1 still reads topic v4. This repo has hit exactly this bug before: commit 178aa0a bumped four recipes' topic versions for the same reason ("bodies changed... without their # Topic: version being bumped, so a consumer with a cached copy had no signal that the content moved"). test/report.test.ts:881 pins topic v4 by regex, so it will keep passing against now-stale content rather than catching the missed bump.

Minor, not inlined: no test pins the new ## Estimating violations section's content the way test/report.test.ts:878-899 pins other sections (e.g. the ## Counting surfaced text). Given this repo's habit of pinning instructional prose in tests (the sibling commit on this branch adds exactly such a pin for step 4), a similar pin for the new section would catch future accidental edits, but I'm not raising this inline since it's a coverage preference rather than a defect.

Otherwise the change reads well: the new section's guidance (search for the problem not the protection, treat syntactic vs. data-flow patterns differently, spot-check and scale, write the low end) is internally consistent with the updated cloud bullet in step 2 and with the schema description changes in report-facts.ts, and the history-zero tightening is consistent between the recipe prose and the schema's .describe() text.

Comment thread packages/cli/src/agent/report.md
The recipe body changed, so a cached v4 copy needs a signal that its
guidance on estimating cloud violations is superseded.
An agent once searched for the escape helper, found where escaping
already happened, and published zero. Pin the section that asks for a
search for the problem itself, and that the old 'saw no instance' zero
is gone.
@theCodeDrift

Copy link
Copy Markdown
Member Author

Re: @github-actions — "Claude finished @theCodeDrift's task in 1m 50s… Minor, not inlined: no test pins the new ## Estimating violations section"
#524 (comment)

Added in 8dea6da: a test pins the section's key instructions (look before answering zero, search for the problem not the protection, start from the real shape, write the low end), and asserts the old "saw no instance" zero is gone from step 2. The topic-version finding was fixed in 807a88e.

— AI Coding Agent

@theCodeDrift
theCodeDrift merged commit 41fd794 into main Oct 16, 2026
4 checks passed
@theCodeDrift
theCodeDrift deleted the report-cloud-estimates branch October 16, 2026 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant