Skip to content

Add panel-level agreement for expert panels (#18) - #22

Merged
JUhalt merged 1 commit into
masterfrom
feat/v0.3-panel-agreement
Sep 15, 2026
Merged

JUhalt merged 1 commit into
masterfrom
feat/v0.3-panel-agreement

Conversation

@JUhalt

@JUhalt JUhalt commented Sep 14, 2026

Copy link
Copy Markdown
Owner

Summary

Closes #18. Adds a panel-level agreement coefficient for expert panels. Item-level I-CVI and modified kappa are unchanged; this adds one number for how consistently the panel rated the whole item set.

  • panel_agreement() (new export). Krippendorff's alpha is the default, computed from the coincidence matrix (Krippendorff, 2011), with the measurement level chosen by the researcher: ordinal (default), nominal, or interval.
  • Gwet's AC1 is available with method = "ac1" but is never the default. Its output always prints the critique in Vach & Gerke (2023).
  • Intervals are percentile bootstraps that resample items with all of their ratings intact (Zapf et al., 2016). seed makes them reproducible; B = 0 skips them. When some resamples cannot be computed, the output says how many and warns that few items give an imprecise interval.
  • expert_validity() relevance mode gains agreement, agreement_level, agreement_B, and seed. scale_summary gains agreement, agreement_low, and agreement_high; details$agreement holds the full result. AC1 there uses the relevant/not-relevant decision at relevance_cut. Panels with fewer than two experts or two items report no coefficient and say why.
  • Readable output. The share of identical rating pairs prints beside the coefficient, with a plain explanation of why alpha can be low on a close-agreeing panel.
  • Docs. Glossary term, a new section in the expert-panel vignette, a "Reading panel agreement" section in the reading-output vignette, README, NEWS, ROADMAP, and the pkgdown reference index.

Behavior change to note

expert_validity() relevance mode now runs a 1,000-resample bootstrap by default (about 0.2 to 0.6 seconds on typical panels) and draws from the random number generator unless seed is set or agreement_B = 0.

Verification

  • Krippendorff (2011) worked example reproduced: coincidence matrix entries, ordinal difference values, and alpha = .743 (nominal), .815 (ordinal), .849 (interval).
  • Agrees with irr::kripp.alpha() to 1e-10 at all three levels.
  • AC1 reproduces the Wongpakaran et al. (2013) table: observed agreement 16/19 and AC1 within .001 of the printed .752 (the exact value is .7527, so the paper appears to truncate). Kappa from the same table reproduces the printed .565, confirming the cell orientation. AC1 with missing ratings matches Gwet's irrCAC computation.
  • devtools::test(): 296 tests, 1,683 assertions, 0 failures, 0 skips.
  • Spell check clean.
  • R CMD check --as-cran (R 4.6.1, Windows): 0 errors, 0 warnings, 0 notes. An earlier local run reported 2 errors that traced to Windows' 260-character path limit in the check directory, not to package code; the rerun in a short directory is clean.

🤖 Generated with Claude Code

Adds panel_agreement(): Krippendorff's alpha by default, with the
measurement level chosen by the researcher (ordinal, nominal, interval),
and Gwet's AC1 as a never-default option that prints the Vach & Gerke
(2023) critique. Intervals are percentile bootstraps that resample items
with all ratings intact, following Zapf et al. (2016).

expert_validity() relevance mode now reports agreement, agreement_low and
agreement_high in scale_summary, with the full result in
details$agreement. Item-level I-CVI and modified kappa are unchanged.
Output shows the share of identical rating pairs beside the coefficient
and explains why alpha can be low on a close-agreeing panel.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@JUhalt JUhalt added this to the v0.3.0 milestone Sep 14, 2026
@JUhalt
JUhalt merged commit 7bc14ef into master Sep 15, 2026
8 checks passed
@JUhalt
JUhalt deleted the feat/v0.3-panel-agreement branch September 15, 2026 10:52
@JUhalt JUhalt mentioned this pull request Sep 15, 2026
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[v0.3] Panel-level agreement coefficients for expert panels

1 participant