Regenerate the reviewer-feedback eval baseline against the shipped prompt

[trask]

The recorded baseline in eval/reviewer_feedback_cases.json named TOP_LEVEL_REVIEWER_FEEDBACK_BATCH_PROMPT_TEMPLATE, a prompt that #196 replaced and #210 deleted, so its labels described a classifier that no longer runs and a scored regression was as likely to be the old prompt showing through as a real one. This re-measures every case against the shipped REVIEWER_FEEDBACK_PROMPT_TEMPLATE at gpt-5.4-mini, five runs, batched one pull request at a time as the dashboard batches. Fifteen stable cases change label, eleven of them approvals carrying optional nits moving to no_author_action, which is the change #196 set out to make.

eval/regenerate_baseline.py is what produced the file, so the next prompt change can refresh it instead of retiring it. It drives from the cases already recorded rather than re-collecting from GitHub, because the original collection kept only pull requests that were open on the day it ran. It writes the field names #187 settled on, where action_labels maps the shipped binary's verdicts onto themselves because that binary already answers in the recorded labels.

The cases also record the root_timestamp the dashboard orders a pull request's items by and are stored in that order, and the one case holding an untruncated body now holds the truncated one, so the batches measured are the batches production sends.

Also adds the five review summaries from open-telemetry/opentelemetry-java-instrumentation#19152, the thread that prompted this, and adjudicates the shortest of them, AI-generated review, as no_author_action.

[opentelemetry-pr-dashboard]

Pull request dashboard status

Merged · refreshed 2026-08-04 13:35 UTC

Status above doesn't look right?

• Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

[Copilot]

Pull request overview

Regenerates reviewer-feedback evaluation data against the shipped classifier while removing the retired classifier infrastructure.

Changes:

• Adds a cached baseline-regeneration utility.
• Refreshes 616 evaluation cases and adds five review summaries.
• Removes obsolete classifier code, scoring vocabulary, and tests.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
classification.py	Removes the retired classifier path.
test_top_level_actions.py	Updates tests for the shipped classifier.
eval/score_reviewer_feedback.py	Removes obsolete scoring vocabulary.
eval/regenerate_baseline.py	Adds reproducible baseline generation.
eval/reviewer_feedback_cases.json	Records the refreshed baseline and new cases.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 2 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

.github/scripts/pull-request-dashboard/eval/regenerate_baseline.py:12

• The docstring says the cache is under .cache/baseline/, but the code writes under .github/scripts/pull-request-dashboard/.cache/baseline (relative to repo root). Update the docstring to reflect the actual location (or clarify it’s relative to the dashboard script directory) so users can find/clean the cache confidently.

"""Re-measure the recorded baseline in reviewer_feedback_cases.json.

Drives from the cases already in the file rather than re-collecting them from
GitHub. Re-collecting would silently change the set, because the original
collection kept only pull requests that were open on the day it ran.

Raw responses are cached under .cache/baseline/, so a failed or interrupted run
resumes without paying for the calls it already made, and a change to how
answers are read costs nothing to apply.

Run manually; it makes several hundred model calls.
"""

.github/scripts/pull-request-dashboard/eval/regenerate_baseline.py:32

• The docstring says the cache is under .cache/baseline/, but the code writes under .github/scripts/pull-request-dashboard/.cache/baseline (relative to repo root). Update the docstring to reflect the actual location (or clarify it’s relative to the dashboard script directory) so users can find/clean the cache confidently.

CACHE_DIR = Path(__file__).resolve().parent.parent / ".cache" / "baseline"

.github/scripts/pull-request-dashboard/eval/regenerate_baseline.py:141

• The script fans out model calls concurrently (default --workers 12) without retry/backoff. If the underlying service throttles or has transient failures, batches will become unobserved, skewing stability counts and potentially forcing reruns. Consider (mandatory if throttling is realistic in your environment) adding a small retry policy for known transient failures and/or reducing the default concurrency (or making it opt-in) to improve baseline reproducibility.

    with ThreadPoolExecutor(max_workers=workers) as pool:
        results = list(pool.map(work, tasks))

[Copilot]

Pull request overview

Copilot reviewed 1 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (1)

.github/scripts/pull-request-dashboard/eval/regenerate_baseline.py:149

• The new baseline derivation is not covered by tests, even though this repository tests the adjacent scorer and validates the generated fixture. A regression here could silently relabel all 616 cases while the fixture-shape tests still pass. Please add focused unit tests for rebuild (stable/flaky/partially unanswered trials and preservation of adjudication) and the duplicate/invalid-response handling in answers.

def rebuild(payload: dict, trials: list[dict[str, str]], model: str) -> dict:

[Copilot]

Pull request overview

Copilot reviewed 2 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (1)

.github/scripts/pull-request-dashboard/eval/regenerate_baseline.py:49

• This does not always reproduce the prompt sent by production. derive_top_level_items truncates every top-level body to 1,200 characters before classification (dashboard.py:729, utils.py:7), but this path sends the recorded body verbatim. The newly added pr-review-4660197614 body in the fixture is several thousand characters and is not marked truncated, so its five recorded results were measured against input the dashboard never sends. Apply the same production truncation (or store the exact derived body) and regenerate the baseline.

            "discussion_id": c["id"],
            "requester": c["requester"],
            "pr_author": c["pr_author"],
            "body": c["body"],
        }

[Copilot]

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (1)

.github/scripts/pull-request-dashboard/eval/regenerate_baseline.py:173

• batch_cases preserves list order, while production orders top-level items by root_timestamp (dashboard.py:772). Sorting the regenerated fixture by the opaque string ID changes that order—especially by putting all pr-issue-comment-* entries before pr-review-* entries—and can change prompt ordering and 10-item batch boundaries on the next regeneration. Preserve the recorded per-PR order (or record and sort by the production timestamp) instead of sorting by ID.

    cases.sort(key=lambda c: (c["repo"], c["pull_request"], c["id"]))

[Copilot]

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated no new comments.