fix: run the unmocked commit-check test on the CLI's defaults, not the checkout's by shenxianpeng · Pull Request #286 · commit-check/commit-check-action

shenxianpeng · 2026-09-15T06:46:54Z

Why #284 went red

main_test.py::TestRealCommitCheckBinary pipes a message to the real, unmocked commit-check and asserts the verdict. It ran from this repo's checkout, so the CLI read commit-check.toml (which inherits the org config over the network), and — on a runner with no git identity — weighed ignore_authors against the author of HEAD. On a Dependabot PR that author is dependabot[bot], which the org config ignores, so every message check skipped and the test saw skip where it asserted pass/fail. That is what failed all six test jobs and coverage on #284.

Reproduced identically on commit-check 2.17.0 and 2.18.0: the version bump is not the cause. The test was added in #279 and #284 is simply the first Dependabot PR since.

The HEAD-author fallback in commit-check is intentional (commit-check#461, covered by its engine_test.py), and commit-check's own suite guards against exactly this with a pinned_author fixture. So the fix belongs here, in the test.

Fix (test only, `main.py` untouched)

Pin everything the verdict was picking up from the surroundings:

an explicit, empty --config, so no repo or org config applies and ignore_authors names nobody (also: no network fetch in the test suite);
a known identity via GIT_CONFIG_GLOBAL/GIT_CONFIG_SYSTEM, so the author lookup never falls through to HEAD;
no inherited CCHK_* variable, since the environment overrides the file.

Verified

Exact CI scenario (repo with the org-inheriting config, HEAD authored dependabot[bot], no git identity anywhere): unfixed test fails as on chore(deps): bump commit-check from 2.17.0 to 2.18.0 #284; fixed test passes.
Adversarial: inherited CCHK_IGNORE_AUTHORS naming the pinned identity, outer git config nulled: fixed test passes.
Full suite: 199 passed, 8 subtests. pre-commit: clean.

Supersedes #285 (closed) — same diagnosis, but this is the complete isolation CodeRabbit asked for there, as a single test-only change.

#284

Nothing wrong with the bump itself. Once this merges, @dependabot rebase (or re-running its checks) should turn it green.

Summary by CodeRabbit

Tests
- Improved integration test reliability by isolating tests from local configuration, Git identity settings, and environment variables.
- Ensured message validation tests consistently use explicit configuration and produce predictable results.

coderabbitai · 2026-09-15T06:47:02Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 14e4653c-a8f8-416a-be77-ad35ab179263

📥 Commits

Reviewing files that changed from the base of the PR and between a17482d and 2822b24.

📒 Files selected for processing (1)

main_test.py

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

Walkthrough

The integration test now pins configuration, git identity, and environment variables. CLI checks use the explicit temporary configuration instead of ambient repository or organization settings.

Changes

Deterministic integration tests

Layer / File(s)	Summary
Pin test configuration and identity `main_test.py`	The test setup creates an empty configuration file, sets a known git identity, removes `CCHK_*` variables, and passes the configuration path to `run_check_json`. The class documentation records the previous ambient-configuration behavior.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other · Severity of issue fixed: Low

Merge Risk: ⚪ Minimal · up to 2822b

The test-only change isolates temporary state correctly and explicitly controls CLI configuration, with no established production or merge-blocking risk.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 1 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: the unmocked commit-check test now uses isolated CLI defaults instead of checkout configuration.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/real-binary-test-runs-on-defaults

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-09-15T06:47:14Z

Commit Check

✅ All 5 checks passed

Show all 5 checks

Commit message
  ✔ PR title (fix: run the unmocked commit-check test on the CLI's defa...)
  ✔ Commit 1/1 (2822b24) (fix: run the unmocked commit-check test on the CLI's defa...)
Branch
  ✔ Branch (fix/real-binary-test-runs-on-defaults)
Author
  ✔ Author name (Xianpeng Shen)
  ✔ Author email (xianpeng.shen@gmail.com)

commit-check 2.18.0 · Rules reference

codecov · 2026-09-15T06:47:43Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.11%. Comparing base (707724d) to head (2822b24).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #286   +/-   ##
=======================================
  Coverage   95.11%   95.11%           
=======================================
  Files           1        1           
  Lines         614      614           
=======================================
  Hits          584      584           
  Misses         30       30

Flag	Coverage Δ
unittests	`95.11% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…e checkout's TestRealCommitCheckBinary pipes a message to the real commit-check and asserts the verdict. It ran from this repo's checkout, so the CLI read commit-check.toml -- which inherits the org config over the network -- and, on a runner with no git identity, weighed ignore_authors against the author of HEAD. On a Dependabot PR that author is dependabot[bot], which the org config ignores, so every message check skipped and the test saw "skip" where it asserted "pass" or "fail" (#284, on every OS and Python in the matrix, plus coverage). The same on 2.17.0 and 2.18.0: the bump is not the cause, the test just had never met a bot commit since #279 added it. A piped message describes no commit. Pin everything the verdict was picking up from the surroundings: an explicit, empty --config so no repo or org config applies and ignore_authors names nobody; a known identity at git's global and system levels so the author lookup never falls through to HEAD; and no inherited CCHK_* variable, which would override the file. commit-check's own suite pins the same two lookups for the same reason (its pinned_author fixture); this is that pin from outside the process. No change to main.py.

shenxianpeng requested a review from a team as a code owner September 15, 2026 06:46

shenxianpeng force-pushed the fix/real-binary-test-runs-on-defaults branch from 1250436 to 2822b24 Compare September 15, 2026 06:56

shenxianpeng added the bug Something isn't working label Sep 15, 2026

shenxianpeng merged commit 6866772 into main Sep 15, 2026
21 checks passed

shenxianpeng deleted the fix/real-binary-test-runs-on-defaults branch September 15, 2026 07:00

Navigation Menu

Uh oh!

Uh oh!

fix: run the unmocked commit-check test on the CLI's defaults, not the checkout's - #286

fix: run the unmocked commit-check test on the CLI's defaults, not the checkout's#286
shenxianpeng merged 1 commit into
mainfrom
fix/real-binary-test-runs-on-defaults

shenxianpeng commented Sep 15, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Sep 15, 2026 •

edited

Loading

Walkthrough

Changes

❌ Failed checks (1 warning)

Uh oh!

github-actions Bot commented Sep 15, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Sep 15, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

shenxianpeng commented Sep 15, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why #284 went red

Fix (test only, main.py untouched)

Verified

#284

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

❌ Failed checks (1 warning)

Uh oh!

github-actions Bot commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Commit Check

Uh oh!

codecov Bot commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

shenxianpeng commented Sep 15, 2026 •

edited by coderabbitai Bot

Loading

Fix (test only, `main.py` untouched)

coderabbitai Bot commented Sep 15, 2026 •

edited

Loading

github-actions Bot commented Sep 15, 2026 •

edited

Loading

codecov Bot commented Sep 15, 2026 •

edited

Loading