Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion .bazelignore
Original file line number Diff line number Diff line change
@@ -1 +1 @@
cli/src/test/resources/workspaces
tests/resources/workspaces
15 changes: 4 additions & 11 deletions .bazelrc
Original file line number Diff line number Diff line change
@@ -1,10 +1,5 @@
run -c opt --show_loading_progress=false --show_progress=false --ui_event_filters=error
run:verbose -c dbg --show_loading_progress=true --show_progress=true --ui_event_filters=info,error,debug
# https://github.com/mockito/mockito/issues/1879
test --sandbox_tmpfs_path=/tmp

# Use a hermetic JDK for tests.
common --java_runtime_version=remotejdk_21

# Enable platform-specific config
common --enable_platform_specific_config
Expand All @@ -19,7 +14,7 @@ build:macos --host_cxxopt=-std=c++17
build:windows --cxxopt=/std:c++17
build:windows --host_cxxopt=/std:c++17

# Release artifacts: `bazel build //release:bazel-diff-rust --config=release`.
# Release artifacts: `bazel build //release:bazel-diff --config=release`.
# CI uses nothing but this config, so the flags a published binary is built with
# live here rather than in the workflow files. The C++ standard flags come from
# the platform-specific configs above.
Expand All @@ -39,7 +34,7 @@ build:release --@rules_rust//rust/settings:lto=thin
# property of the toolchain, which would change every build, not just releases.
build:release --@rules_rust//rust/settings:extra_rustc_flags=-Cstrip=symbols

# Linux release artifacts: `bazel build //release:bazel-diff-rust --config=release-musl`
# Linux release artifacts: `bazel build //release:bazel-diff --config=release-musl`
# (amd64) or `--config=release-musl-arm64` (arm64). Everything --config=release
# does, but built for a musl platform, which makes the published Linux binary
# statically linked: it runs on any distribution instead of requiring the glibc
Expand Down Expand Up @@ -78,10 +73,8 @@ build --output_groups=+rustfmt_checks
# coverage mode anyway, so plain `bazel test` runs are never affected.
coverage --coverage_output_generator=//tools/coverage:lcov_merger

# Releases run the suite on cold runners (see release.yaml, which splits it
# into a Java job and a Rust job), where the long suites -- //tests:e2e_test
# (size = "enormous") and //cli:E2ETest (timeout = "eternal") -- have no
# headroom left: v44.0.0 died on
# Releases run the suite on cold runners (see release.yaml), where the long
# e2e suite under //tests has no headroom left: v44.0.0 died on
# "//tests:e2e_test TIMEOUT in 3600.0s" and published nothing. Per-size timeouts
# are there to catch hangs on PRs, where each suite gets its own runner and an
# overrun means something is wrong. A release has no such signal to give, so the
Expand Down
3 changes: 1 addition & 2 deletions .bcr/presubmit.yml
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,6 @@ bcr_test_module:
- '--cxxopt=-std=c++17'
- '--host_cxxopt=-std=c++17'
build_targets:
- '//:bazel-diff-rust'
- '//:bazel-diff'
test_targets:
- '//cli/...'
- '//:rust_tests'
23 changes: 12 additions & 11 deletions .claude/skills/coverage-status/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,15 @@ description: Use to check the current main-source line coverage of the bazel-dif

# Checking coverage status

bazel-diff has a 90% main-source line-coverage gate enforced in CI ([.github/workflows/ci.yaml](../../../.github/workflows/ci.yaml)). The enforcement code lives in [tools/coverage_check.py](../../../tools/coverage_check.py); the gate runs on every `test-jre21` matrix entry as `bazelisk run //tools:coverage-check -- bazel-out/_coverage/_coverage_report.dat`.
bazel-diff has a 90% main-source line-coverage gate enforced in CI ([.github/workflows/ci.yaml](../../../.github/workflows/ci.yaml)). The repo-wide enforcement code lives in [tools/coverage_check.py](../../../tools/coverage_check.py); the gate runs on every `coverage` matrix entry as `bazelisk run //tools:coverage-check -- bazel-out/_coverage/_coverage_report.dat`. On top of that, individual test targets declare per-target minimums that the Rust LCOV merger under [tools/coverage/](../../../tools/coverage/) enforces during the coverage run itself.

## Fastest path: just check the current state

```bash
make coverage
```

This runs `bazel coverage --combined_report=lcov //cli/... //tools:coverage_check_test`, then `bazel run //tools:coverage-check` against the combined report. It prints a per-file table sorted worst-first, the overall percentage, and PASS/FAIL against the threshold. Exits non-zero when below.
This runs `bazel coverage --combined_report=lcov //src:cli_tests //src:rust_tests //tools:coverage_check_test //tools/coverage/... //tools/go/...`, then `bazel run //tools:coverage-check` against the combined report (twice: once for `src/`, once scoped to `tools/go/`). It prints a per-file table sorted worst-first, the overall percentage, and PASS/FAIL against the threshold. Exits non-zero when below.

If a coverage report already exists in `bazel-out/_coverage/_coverage_report.dat` (e.g. from a prior `bazel coverage` invocation) and you only want to re-check the threshold without re-running the test suite:

Expand All @@ -36,27 +36,28 @@ The per-file table looks like:
```
COV% LINES (hit/total) FILE
-------- ----------------- ----
0.00% 0 / 4 cli/src/main/kotlin/com/bazel_diff/Main.kt
62.67% 47 / 75 cli/src/main/kotlin/com/bazel_diff/bazel/BazelModService.kt
85.10% 401 / 471 src/main.rs
93.42% 1263 / 1352 src/server.rs
...
100.00% 23 / 23 cli/src/main/kotlin/com/bazel_diff/hash/TargetHash.kt
100.00% 23 / 23 src/fingerprint.rs

Overall main-source line coverage: 90.37% (1455 / 1610)
Overall main-source line coverage: 94.37% (4816 / 5103)
Threshold: 90.00%
PASS
```

- **Sort order is ascending** — worst-covered files appear first, so the gap is at the top.
- **Test sources (`/test/`) are excluded** — Bazel's Kotlin instrumentation reports them too, but they don't count toward the threshold (otherwise well-covered tests would hide thin production-code coverage).
- **Files with `LF:0` are dropped** — that's an "no instrumentation data" signal (e.g. Bazel's Python coverage when no toolchain coverage tool is configured), and treating it as 0% would tank the threshold.
- **Only `src/` counts** (plus `tools/coverage_check.py`) — the e2e crate under `tests/` and the tooling under `tools/` are excluded, so well-covered test code cannot hide thin production coverage. The Rust unit tests live in `#[cfg(test)]` modules inside the `src/` files, so their lines are counted along with the code they cover; that is the same scope the per-target minimum on `//src:rust_tests` uses.
- **Files with `LF:0` are dropped** — that's a "no instrumentation data" signal (e.g. Bazel's Python coverage when no toolchain coverage tool is configured), and treating it as 0% would tank the threshold.
- **Threshold is inclusive at the boundary** — exactly 90.00% passes; 89.99% fails.

## Looking at CI's number, not local

Local coverage may be lower than CI's because the slow `//cli:E2ETest` is often skipped or fails on dev machines (JDK-env sandbox issues). The number CI sees comes from the `test-jre21` matrix; download the `coverage-report-jre21-*` artifact from a recent run for an exact snapshot, or just read the "Overall main-source line coverage:" line in the workflow log.
The number CI sees comes from the `coverage` matrix; download the `coverage-report-*` artifact from a recent run for an exact snapshot, or just read the "Overall main-source line coverage:" line in the workflow log.

## Configuration

- **Threshold**: `COVERAGE_THRESHOLD` env var (default 90). Set in [.github/workflows/ci.yaml](../../../.github/workflows/ci.yaml) for CI, or on the command line to experiment: `COVERAGE_THRESHOLD=85 make coverage`.
- **Include prefixes**: `--include` flag or `$COVERAGE_INCLUDE` env var (default `cli/src/main/,tools/coverage_check.py`). Only files starting with one of these prefixes count.
- **Self-coverage of the checker**: `tools/coverage_check.py` is itself in the default include list, but Bazel's Python coverage emits `LF:0` for it today because the Python toolchain in `MODULE.bazel` has no `coverage_tool` configured. The 24-case py_test (`bazel test //tools:coverage_check_test`) is the actual proof the checker is exercised.
- **Include prefixes**: `--include` flag or `$COVERAGE_INCLUDE` env var (default `src/,tools/coverage_check.py`). Only files starting with one of these prefixes count. CI runs a second pass with `--include tools/go/` so Go is gated independently.
- **Self-coverage of the checker**: `tools/coverage_check.py` is itself in the default include list, but Bazel's Python coverage emits `LF:0` for it today because the Python toolchain in `MODULE.bazel` has no `coverage_tool` configured. The py_test (`bazel test //tools:coverage_check_test`) is the actual proof the checker is exercised.
- **Per-target minimums**: declared through `coverage_enforced_test` / `coverage_minimum_env` in [tools/coverage/defs.bzl](../../../tools/coverage/defs.bzl); a failing target exits with code 33 and its test log carries the per-file breakdown.
54 changes: 19 additions & 35 deletions .claude/skills/improve-coverage/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,62 +1,46 @@
---
name: improve-coverage
description: Use when you need to raise main-source line coverage in the bazel-diff repo, write tests for an under-covered Kotlin file, fix a CI failure on the 90% coverage gate, or pick the highest-leverage files to test next. Triggers on requests like "the coverage gate is failing, fix it", "write tests for X", "we need more coverage", or "what should I test to get to 90%".
description: Use when you need to raise main-source line coverage in the bazel-diff repo, write tests for an under-covered Rust module, fix a CI failure on the 90% coverage gate, or pick the highest-leverage files to test next. Triggers on requests like "the coverage gate is failing, fix it", "write tests for X", "we need more coverage", or "what should I test to get to 90%".
---

# Improving coverage to clear the 90% gate

bazel-diff enforces a 90% main-source line-coverage gate on every PR (see [coverage-status](../coverage-status/SKILL.md) for the inspection side). When the gate fails or you want to raise the bar, the workflow is: pick the worst-covered files, write small focused unit tests, re-run the gate locally before pushing.
bazel-diff enforces a 90% main-source line-coverage gate on every PR (see [coverage-status](../coverage-status/SKILL.md) for the inspection side), plus per-target minimums on `//src:rust_tests` and `//src:cli_tests` that fire during `bazel coverage`. When the gate fails or you want to raise the bar, the workflow is: pick the worst-covered files, write small focused unit tests, re-run the gate locally before pushing.

## 1. Pick the right files to target

Run `make coverage` (or check the latest CI artifact) and look at the top of the sorted table. Prioritise files by **uncovered-lines-per-test-effort**, not by lowest percentage:

- **Highest-leverage**: small files at 0% (e.g. enum classes, value objects, single-method utilities) — one short unit test usually moves the needle without much code.
- **Highest absolute gain**: large files with moderate coverage (e.g. `BazelQueryService.kt` at 303 lines / 93%) — closing a small percentage gap covers many lines.
- **Highest-leverage**: small helpers at 0% (a parser, a converter, a value type's `Display`) — one short unit test usually moves the needle without much code.
- **Highest absolute gain**: large files with moderate coverage (`src/server.rs`, `src/bazel.rs`, `src/main.rs`) — closing a small percentage gap covers many lines.
- **Lowest leverage**: tiny files at 50–80% where the remaining branches are error paths needing fault injection or refactors.

A worked example: the PR that added the gate ([#356](https://github.com/Tinder/bazel-diff/pull/356)) raised coverage from 88.76% → 90.37% by adding five small files of tests — `BazelTargetTypeTest`, `VersionProviderTest`, `BazelDiffTest`, `StderrLoggerTest`, `BazelTargetTest` — for a total of 26 newly-covered lines.

## 2. Write the test

Existing tests follow a consistent shape:

- Live under `cli/src/test/kotlin/...` mirroring the main source path.
- Use **JUnit 4** (`@Test`, `@Before`, `@After`, `org.junit.Assert.assertThrows`).
- Use **assertk** for assertions (`assertk.assertThat`, with explicit imports for each assertion like `assertk.assertions.isEqualTo`). Forgetting `import assertk.assertions.contains` on a `String.contains` assertion produces a confusing receiver-mismatch error — import every assertion you use.
- Use **mockito-kotlin** when mocking is required (existing examples: `cli/src/test/kotlin/com/bazel_diff/bazel/BazelClientTest.kt`).
- Use **koin** for DI-test setup (existing pattern: `cli/src/test/kotlin/com/bazel_diff/interactor/CalculateImpactedTargetsInteractorIssue335Test.kt`).

Tiny files (enums, value objects, small command classes) usually only need a few targeted tests. Looking at the bytes via `assertThat(BazelTargetType.entries).hasSize(N).containsExactlyInAnyOrder(...)` is enough to cover an enum's declaration lines.

## 3. Register the test target in cli/BUILD

Every test needs its own `kt_jvm_test` entry:
- Unit tests live in a `#[cfg(test)] mod tests` at the bottom of the `src/` file they cover, and are run by `//src:rust_tests` (library) and `//src:cli_tests` (`main.rs`).
- They test pure transformations with injected data: query planning, repository lowering, hash computation and module-impact decisions take their inputs as values, not as a live Bazel. Nothing under `src/` spawns a fake `bazel` executable — subprocess integration is the e2e suite's job.
- Filesystem cases use `tempfile::TempDir`; server cases in `src/server.rs` bind a loopback port and drive the real HTTP handler.
- Anything that needs a real Bazel workspace is an e2e case under `tests/e2e/` (see [tools/e2e/README.md](../../../tools/e2e/README.md) for the per-case target split and `make regen-e2e`). E2E cases are slow and carry no per-target coverage minimum, so prefer a unit test whenever the logic can be reached without Bazel.

```python
kt_jvm_test(
name = "BazelTargetTypeTest",
test_class = "com.bazel_diff.bazel.BazelTargetTypeTest",
runtime_deps = [":cli-test-lib"],
)
```

The `:cli-test-lib` glob picks up the new test source automatically; the explicit `kt_jvm_test` rule is what makes it executable via `bazel test //cli:<name>`.

## 4. Verify locally before pushing
## 3. Verify locally before pushing

```bash
bazel test //cli:<YourNewTest> # one-off run of the new test
make coverage # full gate
cargo test # fastest inner loop (unit + e2e crate)
bazel test //:rust_tests # what CI runs, with the pinned toolchain
make coverage # the full gate
```

The local number may be lower than CI's because `//cli:E2ETest` often fails or is excluded on dev machines (JDK-env sandbox issues). If you've added tests for a file that's also exercised by E2E (e.g. `BazelQueryService.kt`), the CI delta will be smaller than the local delta — count only the lines that weren't already covered by E2E.
`bazel coverage` is what makes the per-target minimums fire: plain `bazel test` never invokes the LCOV merger. If a target falls below its minimum, its test log ends with the merger's per-file breakdown and the action exits 33.

Lint gates run on every build through the clippy and rustfmt aspects in `.bazelrc`; `make format` (`bazel run //tools/format:rustfmt`) fixes formatting with the exact rustfmt CI uses.

## 5. Things that don't work / aren't worth attempting
## 4. Things that don't work / aren't worth attempting

- **`Main.kt`** — calls `exitProcess(...)` which kills the JVM; can't be tested in-process without a SecurityManager hack or refactor. Stays at 0%; the threshold tolerates it.
- **`throw IllegalArgumentException(...)` branches in resource-loading code** like `VersionProvider.kt` — the production code resolves the classloader from `this::class.java`, with no injection seam to swap it for one missing the resource. Refactor or skip.
- **`else -> BazelTargetType.UNKNOWN` branches** — only reachable when Bazel's `Build.Target.Discriminator` adds a new enum value the production code doesn't recognise. Triggering today would require a hand-forged proto with a reserved discriminator number, which the protobuf builder rejects.
- **`main()` in `src/main.rs`** — exits the process; the testable surface is the command functions it dispatches to, which the `cli_tests` target already covers. Test those, not `main`.
- **`unreachable!` / exhaustive-match fallbacks** on generated protobuf enums — only reachable when Bazel's `Target.Discriminator` grows a new value. Not worth a hand-forged proto.
- **Network error paths in the S3 cache tier** — covered by the loopback mock in `src/server.rs`'s tests; do not add real-bucket tests.

## When the gate fails on a flake, not on a coverage drop

Expand Down
7 changes: 4 additions & 3 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Test fixture workspaces are deliberately constructed inputs to the E2E tests
# (e.g. module_bazel_comment exercises comment preservation, cquery_failing_target
# encodes a specific failure scenario). Exclude them from aspect_rules_lint
# formatters (buildifier/ktfmt) so the cron jobs never rewrite the very content
# under test. `rules-lint-ignored` is honored by //cli/format and //cli/format:buildifier.
cli/src/test/resources/** rules-lint-ignored
# formatters (buildifier/rustfmt) so the cron jobs never rewrite the very content
# under test. `rules-lint-ignored` is honored by //tools/format:buildifier and
# //tools/format:rustfmt.
tests/resources/** rules-lint-ignored
2 changes: 1 addition & 1 deletion .github/workflows/assert_static_binary.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

# Asserts that a published Linux binary is statically linked, i.e. that it
# carries no dependency on the glibc of the runner that built it. Shared by the
# `release-artifacts` job in ci.yaml (per-PR check) and the `rust-binaries` job
# `release-artifacts` job in ci.yaml (per-PR check) and the `binaries` job
# in release.yaml (before the asset is uploaded), so a regression cannot reach a
# release without failing a PR first.
#
Expand Down
13 changes: 4 additions & 9 deletions .github/workflows/buildifier.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name: Buildifier

# Runs buildifier over all Starlark/BUILD files on a daily schedule and opens a
# PR only if the formatter produced changes. Mirrors the ktfmt format cron: it
# PR only if the formatter produced changes. Mirrors the coverage-badge cron: it
# force-pushes a dedicated, bot-owned branch so re-runs reuse one PR instead of
# accumulating stale branches, and it never touches master directly.
#
Expand All @@ -10,7 +10,7 @@ name: Buildifier
# generated PR limited to formatting-only changes.
on:
schedule:
# 12:30 UTC daily (between the ktfmt format and coverage-badge crons).
# 12:30 UTC daily (half an hour before the coverage-badge cron).
- cron: '30 12 * * *'
workflow_dispatch:

Expand All @@ -21,11 +21,6 @@ jobs:
contents: write
pull-requests: write
steps:
- name: Setup Java JDK
uses: actions/setup-java@v4
with:
distribution: 'temurin'
java-version: '21'
- name: Setup Go environment
uses: actions/setup-go@v5
with:
Expand All @@ -35,7 +30,7 @@ jobs:
run: go install github.com/bazelbuild/bazelisk@latest && export PATH=$PATH:$(go env GOPATH)/bin
- uses: actions/checkout@v4
- name: Run buildifier
run: ~/go/bin/bazelisk run //cli/format:buildifier --enable_bzlmod=true --enable_workspace=false --lockfile_mode=off
run: ~/go/bin/bazelisk run //tools/format:buildifier --enable_bzlmod=true --enable_workspace=false --lockfile_mode=off
- name: Open PR if formatting changed
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Expand All @@ -60,5 +55,5 @@ jobs:
--base master \
--head "$BRANCH" \
--title "ci: apply buildifier formatting" \
--body "Automated buildifier run via \`bazel run //cli/format:buildifier\`. This PR contains formatting-only changes to BUILD/Starlark files."
--body "Automated buildifier run via \`bazel run //tools/format:buildifier\`. This PR contains formatting-only changes to BUILD/Starlark files."
fi
Loading