Skip to content

Explain inter-rater reliability before the first reviewer pair completes - #852

Merged
InfinityBowman merged 1 commit into
mainfrom
feat/reliability-empty-state
Sep 19, 2026
Merged

InfinityBowman merged 1 commit into
mainfrom
feat/reliability-empty-state

Conversation

@InfinityBowman

Copy link
Copy Markdown
Owner

Follow-up to #850. The inter-rater reliability section only rendered once at least one reviewer pair had produced comparable answers, so on a dual-review project where nobody has finished yet the feature left no trace on the overview at all - no heading, no hint that the numbers are coming.

What changed

  • The section keeps its heading whenever any study has two different reviewers assigned, and replaces the cards with one line: "Agreement appears once both reviewers of a study have completed their appraisals. N studies have two reviewers."
  • The subtitle ("Before reconciliation, pooled across reviewer pairs") is suppressed while empty, since it describes numbers that are not on screen.
  • Single-reviewer projects still render nothing. An empty reliability card there is not a hint, it implies a workflow the user did not choose.

This matches the existing Results section pattern ("Figures and tables appear once the first study is finalized.").

Not covered here

calculateProjectReliability also drops a cell when both reviewer checklists are complete but the pair is rejected - same assignee, one unassigned, or missing checklist data. To the user that is indistinguishable from "no pairs yet" even though the pair is finished. Surfacing it needs the calculator to return a skipped-cell count alongside the tools, which changes its signature, so it is left for a separate change.

Verification

  • pnpm --filter web test - 324 passed, including a new case covering the empty state
  • pnpm typecheck - clean

https://claude.ai/code/session_01H77BMkZ9GhK8D98TG4cpTg

The section only rendered once a reviewer pair had produced comparable
answers, so a dual-review project with nothing finished yet showed no
trace of the feature at all. It now keeps its heading and says what
unlocks the numbers whenever any study has two different reviewers,
and stays hidden entirely on single-reviewer projects.

Claude-Session: https://claude.ai/code/session_01H77BMkZ9GhK8D98TG4cpTg
@coderabbitai

coderabbitai Bot commented Sep 19, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 43 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: a09642d8-60f4-4e0c-b398-6cd7ab27457b

📥 Commits

Reviewing files that changed from the base of the PR and between 3d6b9a2 and f3e883c.

📒 Files selected for processing (3)
  • packages/web/src/components/project/overview-tab/OverviewTab.tsx
  • packages/web/src/components/project/overview-tab/ReliabilitySection.tsx
  • packages/web/src/components/project/overview-tab/__tests__/ReliabilitySection.test.tsx

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@InfinityBowman
InfinityBowman merged commit dc1117c into main Sep 19, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant