Fix the Review Lens plugin's wire contract, and 20 defects behind it - #419
Merged
Conversation
An audit of the Review Lens (code review) plugin across its engine, fetch, dispatch, domain, persistence and React layers. Problem The headline defect made the plugin unusable end to end. scene_payload forwarded the code_review node's nested rows with `dict(row)`, so they reached the wire carrying the review engine's snake_case keys - `group_title`, `patch_truncated`, `previous_path` - where CodeReviewWalkthroughGroupRow and CodeReviewFileRow declare `groupTitle`, `patchTruncated` and `previousPath`. The generated client validator treats a missing required field as fatal and bindTopic.ts DROPS a snapshot that fails validation, so from the first successful PR fetch onward every scene snapshot for the session was rejected client-side: the whole canvas froze, not just this node. Nothing caught it because the one wire test passed empty nested lists and the frontend test built its fixtures by hand in camelCase. The rest were found in the same pass: - Release risk ignored high and critical REVIEW FINDINGS, so a critical defect the model filed as a finding rather than an error rendered as "Needs Revision, 95/100, low risk" above its own red card. - looks_like_a_review accepted a list whose entries normalization then discarded, resurrecting the fabricated 72/100 grade it exists to stop. - An infinite category score or line number raised OverflowError - not a ValueError - out of the one path that must always degrade to a fallback, surfacing as "Review Lens run failed". - The model diff cap (45,000) was smaller than the fetch cap (60,000), so a 50KB diff was cut a second time while every user-visible surface still said the review saw all of it. - The unified diff was appended to the prompt under a plain heading with no fence, and attacker-controlled file paths - newlines intact - were interpolated outside it. - toDiffFence used a fixed 3-backtick fence, which any diff line containing ``` closes, rendering the rest of a third-party PR as live markdown in the node. - Cancel was a dead control during a fetch or an Ask, and cancelling a run freed the registry slot but left the node busy for up to 600s. - A timed-out run left no error banner at all; deleting the node mid-run produced a spurious "unknown node" toast. - parse_pr_url returned `..` as an owner or repo, retargeting the api.github.com URL built by concatenation downstream. - The diff download followed redirects with the token attached and buffered the whole body before applying any cap. - The fallback pre-screen shipped an eight-category 82/100 scorecard it had invented, and the node rendered "No review yet - run a review first." directly above that pre-screen's own findings. - Restore was the only write path into these fields with no caps. - review_engine.py sat inside the mypy gate with all 30 of its function bodies unchecked, because none of them are annotated. Change Nested rows are now built explicitly at the wire builder, in camelCase, following the toolCalls precedent - which also drops the per-file patch bodies (up to ~600KB per node on every republish, read by nothing). Everything else is fixed at the layer that owns it; every fix carries a comment naming the behavior it replaces. The engine's fallback scorecard now reports only categories a heuristic actually lowered, and the node distinguishes "not reviewed" from "reviewed but not graded". Six false-positive and six false-negative fallback heuristics were corrected (JS `regex.exec`, commented-out code, `pprint`; multi-line `except Exception: pass`, `check_output(shell=True)`, `os.popen`). check_untyped_defs is enabled for the plugin's modules, which found a real narrowing bug immediately. Test plan - 3,189 backend tests pass (54 anyio DeprecationWarning collection errors are pre-existing on clean main and unrelated). - 2,204 frontend tests across 93 files pass. - mypy, ruff, eslint and the codegen drift check are all clean; the bundle-size ceiling still has headroom. - New: a wire-boundary test validating a fully populated review row against SceneNodeRow itself, which fails on the old code; adversarial prompt-fence and path-injection tests; a parametrized table pinning each corrected heuristic; verdict/risk gate pins; restore-cap pins. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Follow-up to the audit fixes in the previous commit - the smaller items that were found in the same pass. Problem - _decode_text_bytes tried four encodings with an errors="replace" tail, three parts of which were unreachable: utf-8-sig can only succeed where utf-8 already did (a BOM is valid UTF-8), and latin-1 maps all 256 byte values, so it never raises and nothing after it can run. Reaching latin-1 meant silently mojibaking a diff - every non-ASCII byte became a plausible-looking wrong character, with that text then reviewed, persisted, and shown as the file's real content. - test_walkthrough_caps_groups_and_paths put exactly one file in each of twenty directories, so every group held a single path and its per-group assertion could not fail for any implementation. MAX_WALKTHROUGH_PATHS_PER_GROUP was uncovered everywhere. - canonical_pr_slug's docstring called it "the short human label shown on the node". Nothing calls it; the view composes that string itself. - codeReviewDiffChars was fetched, put on the wire and compared by the memo comparator while no JSX referenced it - so the comparator's "only the fields this view actually reads" claim was false, and the one number saying how much diff the review actually saw was invisible. Change The decode chain is utf-8 with an explicit BOM strip, falling back to errors="replace" - U+FFFD is visible, which is the point. The cap test now puts more files in each directory than the cap allows and asserts both caps bind. canonical_pr_slug's docstring says what it is rather than claiming a caller it does not have. Diff size renders in the Setup tab's identity block, which makes the comparator's claim true. Test plan - 3,192 backend tests pass (54 anyio DeprecationWarning collection errors are pre-existing on clean main). - 2,204 frontend tests pass; the one ChartNodeView failure seen in a full-suite run is a canvas/lazy-load flake - that file is untouched by this branch and passes 28/28 in isolation. - mypy, ruff, eslint, codegen drift and the bundle ceiling are all clean. - New: three tests pinning the decode behavior (mis-encoded input marked rather than mojibaked, BOM stripped, ordinary UTF-8 untouched). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An audit of the Review Lens (code review) plugin across its engine, fetch, dispatch, domain, persistence and React layers.
Problem
The headline defect made the plugin unusable end to end.
scene_payloadforwarded thecode_reviewnode's nested rows withdict(row), so they reached the wire carrying the review engine's snake_case keys —group_title,patch_truncated,previous_path— whereCodeReviewWalkthroughGroupRowandCodeReviewFileRowdeclaregroupTitle,patchTruncatedandpreviousPath. The generated client validator treats a missing required field as fatal andbindTopic.tsdrops a snapshot that fails validation, so from the first successful PR fetch onward every scene snapshot for the session was rejected client-side: the whole canvas froze, not just this node.Nothing caught it because the one wire test passed empty nested lists, and the frontend test built its fixtures by hand in camelCase — so the drift was invisible from both sides.
The rest were found in the same pass:
looks_like_a_reviewaccepted a list whose entries normalization then discarded, resurrecting the fabricated 72/100 grade it exists to stopOverflowError— not aValueError— out of the one path that must always degrade to a fallbacktoDiffFenceused a fixed 3-backtick fence, which any diff line containing ``` closes, rendering the rest of a third-party PR as live markdownparse_pr_urlreturned..as an owner or repo, retargeting theapi.github.comURL built by concatenation downstreamqa-entryhad no CSS rule; the Ask draft was destroyed on every rejection path; tabs had no panel association or keyboard navigationreview_engine.pysat inside the mypy gate with all 30 of its function bodies unchecked, because none of them are annotatedChange
Nested rows are built explicitly at the wire builder, in camelCase, following the
toolCallsprecedent — which also drops the per-file patch bodies (up to ~600KB per node on every republish, read by nothing). Everything else is fixed at the layer that owns it, and every fix carries a comment naming the behavior it replaces.The engine's fallback scorecard now reports only categories a heuristic actually lowered, and the node distinguishes "not reviewed" from "reviewed but not graded". Six false-positive and six false-negative fallback heuristics were corrected (JS
regex.exec, commented-out code,pprint; multi-lineexcept Exception: pass,check_output(shell=True),os.popen).check_untyped_defsis enabled for the plugin's modules, which found a real narrowing bug immediately.Test plan
DeprecationWarningcollection errors are pre-existing on cleanmainand unrelated — verified by stashing.mypy,ruff,eslintand the codegen drift check are clean; the bundle-size ceiling still has headroom.SceneNodeRowitself (fails on the old code); adversarial prompt-fence and path-injection tests; a parametrized table pinning each corrected heuristic; verdict/risk gate pins; restore-cap pins.🤖 Generated with Claude Code