Skip to content

ci: add daily checkpoint-availability audit job - #1881

Open
gtrrz-victor wants to merge 5 commits into
mainfrom
feat/checkpoint-audit-job
Open

ci: add daily checkpoint-availability audit job#1881
gtrrz-victor wants to merge 5 commits into
mainfrom
feat/checkpoint-audit-job

Conversation

@gtrrz-victor

@gtrrz-victor gtrrz-victor commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

https://entire.io/gh/entireio/cli/trails/959

Why

entire trail resume 955 failed with "checkpoint not found": checkpoint 01KYSGR4W1ZATZRW47J510EZGJ was referenced by a commit trailer on feat/repo-mirror-use (reached origin), but its git-refs ref refs/entire/checkpoints/GJ/01KYSGR4W1… was never pushed to the checkpoint remote entireio/cli-checkpoints. The git-refs backend enqueues checkpoint refs for a best-effort pre-push drain; when that drain never runs or fails silently, the branch commit and the checkpoint ref diverge — and the gap is invisible until someone tries to resume.

This adds an active daily audit so orphaned checkpoints surface within 24h instead of at resume time.

What

  • scripts/checkpoint-audit.sh — walks every branch commit in a window (--branches --remotes --since), extracts Entire-Checkpoint trailers using git's own %(trailers:…) formatting (handles squash-merge multi-trailers), and diffs them against one git ls-remote refs/entire/checkpoints/*. The checkpoint ID is the ref leaf, so membership is a plain string-set lookup (works for ULID + legacy hex, no shard math).
    • Exit 0 all present · 1 missing (prints a context table) · 2 remote/setup error, so a broken remote never reads as "0 missing".
    • The checkpoint remote is public, so the audit needs no credentials. The script still honors ENTIRE_CHECKPOINT_TOKEN (RFC 7617 basic-header auth, matching the CLI) if the remote ever goes private.
  • .github/workflows/checkpoint-audit.yml — daily 07:00 UTC cron + workflow_dispatch. Fails the job on missing checkpoints; reports via GITHUB_STEP_SUMMARY, an uploaded artifact (.md + .json), and Slack. No App token / secrets needed (default GITHUB_TOKEN checkout + anonymous public fetch).
  • mise.tomlcheckpoint:audit task for local runs.

Report shape

Checkpoint Commit Author Branch(es) Date Subject
01KYSGR4W1ZATZRW47J510EZGJ 4e229c05 Matthias Wenz feat/repo-mirror-use 2026-07-30T14:42:18+02:00 feat(repo): add entire repo mirror use

Verification

  • Positive: 7-day window → exit 1, flagged the known-missing 01KYSGR4W1… (and caught a second real orphan). Valid table + JSON. ~3s.
  • Remote-error: bogus token → exit 2 (not 0).
  • Anonymous git ls-remote against cli-checkpoints confirmed working with no credentials (repo is public).
  • shellcheck / mise run lint:shellcheck / gofmt / gomod / YAML valid / all uses: SHA-pinned. No Go files touched.

Note

Slack reuses the existing E2E_SLACK_WEBHOOK_URL secret. Nothing else to provision.

🤖 Generated with Claude Code

Adds a scheduled audit that catches checkpoints referenced by commit
trailers but missing from the checkpoint remote — the failure behind
`entire trail resume` reporting "checkpoint not found" when a git-refs
checkpoint ref reached origin as a commit trailer but was never pushed
to github.com/entireio/cli-checkpoints.

- scripts/checkpoint-audit.sh: walks all-branch commits in a window,
  extracts Entire-Checkpoint trailers via git's own trailer formatting,
  and diffs them against `git ls-remote refs/entire/checkpoints/*`.
  Exit 0 = all present, 1 = missing (with a context table: checkpoint,
  commit, author, branch, date, subject), 2 = remote/setup error so a
  broken remote never reads as "0 missing". RFC 7617 token auth matches
  the CLI (ENTIRE_CHECKPOINT_TOKEN), never creds-in-URL.
- .github/workflows/checkpoint-audit.yml: daily 07:00 UTC cron (plus
  workflow_dispatch). Fails the job on missing checkpoints and reports
  via GITHUB_STEP_SUMMARY, an uploaded artifact, and Slack.
- mise.toml: `checkpoint:audit` task for local runs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 01KYVVBETH3GTRBRPKWVQBN0JK
Copilot AI review requested due to automatic review settings July 31, 2026 10:29

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 3 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit a00238f. Configure here.

Comment thread scripts/checkpoint-audit.sh Outdated
Comment thread scripts/checkpoint-audit.sh Outdated
Comment thread .github/workflows/checkpoint-audit.yml
entireio/cli-checkpoints is a public repo, so the audit's read-only
`git ls-remote` needs no credentials. Remove the create-github-app-token
step and ENTIRE_CHECKPOINT_TOKEN wiring; anonymous HTTPS fetch is enough.
The script still honors ENTIRE_CHECKPOINT_TOKEN if the remote ever goes
private.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 01KYVVVJ8N5XPJVRTPD400B67N

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not ready to approve

The audit script and workflow have a few correctness/operational issues (portability, documented exit-code behavior, and failure messaging accuracy) that should be fixed before relying on it for automated monitoring.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

Adds a daily CI audit to proactively detect “orphaned” checkpoints: commits that reference an Entire-Checkpoint trailer whose corresponding refs/entire/checkpoints/* ref was never pushed to the checkpoint remote, preventing failures from being discovered only at entire trail resume time.

Changes:

  • Adds scripts/checkpoint-audit.sh to scan recent branch/remotes commits for Entire-Checkpoint trailers and compare them against remote checkpoint refs, emitting markdown + JSON reports and distinct exit codes.
  • Adds a scheduled + manually-dispatchable GitHub Actions workflow to run the audit daily, upload artifacts, and notify Slack on failures.
  • Adds a mise task (checkpoint:audit) to run the audit locally.
File summaries
File Description
scripts/checkpoint-audit.sh New audit script that enumerates checkpoint refs on the remote, scans recent commits for checkpoint trailers, and reports missing refs.
.github/workflows/checkpoint-audit.yml New daily workflow that runs the audit, uploads reports, and posts a Slack notification on failures.
mise.toml Adds a local checkpoint:audit task to run the audit script.
Review details

Suppressed comments (1)

.github/workflows/checkpoint-audit.yml:85

  • The final failure step always reports “missing checkpoints”, but the audit step can also fail with exit code 2 for remote/auth/setup errors. Use a more accurate, generic error message so operators don’t chase the wrong root cause.
      - name: Fail if checkpoints are missing
        if: ${{ steps.audit.outcome == 'failure' }}
        run: |
          echo "::error::checkpoint audit reported missing checkpoints"
          exit 1
  • Files reviewed: 3/3 changed files
  • Comments generated: 4
  • Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

Comment thread scripts/checkpoint-audit.sh Outdated
Comment thread scripts/checkpoint-audit.sh
Comment thread .github/workflows/checkpoint-audit.yml
Comment thread scripts/checkpoint-audit.sh
Bot review (Cursor Bugbot + Copilot) on the audit script and workflow:

- Branch lookup no longer aborts the run: the `git branch --contains`
  pipeline gets `|| true` so grep exiting 1 (all branches filtered, or
  none) under `set -e` can't kill the script before the report renders.
- Multi-branch cells join with ", " — `paste -sd', '` treated the
  delimiter as a circular char list and alternated "," and " ".
- git-log failure (e.g. run outside a repo) now exits 2 with a clear
  message instead of leaking git's raw 128.
- json_str escapes tab and the other C0 controls a commit subject can
  legally carry, so the JSON report is always valid.
- mktemp uses a portable ${TMPDIR:-/tmp}/…XXXXXX template (BSD/macOS).
- Workflow captures the script's exit code and distinguishes missing (1)
  from remote/setup error (2): the Slack alert no longer claims "missing
  checkpoints" when the audit simply couldn't run. upload-artifact gets
  if-no-files-found: ignore for the no-report error path.
- Stop tracking the runtime report artifacts and gitignore them (they
  were committed by an earlier `git add -A`).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 01KYVWHKDPJH5WYMANGZ9SPF4Z
@gtrrz-victor
gtrrz-victor marked this pull request as ready for review July 31, 2026 10:50
@gtrrz-victor
gtrrz-victor requested a review from a team as a code owner July 31, 2026 10:50
@gtrrz-victor
gtrrz-victor requested a review from Copilot August 3, 2026 08:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not ready to approve

The audit script’s token-auth header casing differs from the CLI’s established Authorization: Basic ... behavior and the workflow permissions should align with other artifact-uploading workflows to avoid CI runtime failures.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (3)

scripts/checkpoint-audit.sh:69

  • When ENTIRE_CHECKPOINT_TOKEN is set, the script injects an HTTP auth header using http.extraheader=AUTHORIZATION: basic .... The CLI’s existing implementation and tests use the standard-cased Authorization: Basic ...; using a non-standard header name/scheme casing risks GitHub not recognizing the credentials in some environments.
  auth_b64=$(printf 'x-access-token:%s' "$TOKEN" | base64 | tr -d '\n')
  remote_refs=$(git -c "http.extraheader=AUTHORIZATION: basic ${auth_b64}" \
    ls-remote "$CHECKPOINT_URL" 'refs/entire/checkpoints/*') || {

scripts/checkpoint-audit.sh:148

  • The markdown table only escapes | in the commit subject; author and branches can also contain | (author names are free-form, and git refnames can include |), which would break the table formatting in the report and step summary.
    while IFS=$'\x1e' read -r cp short author branches cdate subject; do
      subject=${subject//|/\\|}
      echo "| \`${cp}\` | \`${short}\` | ${author} | ${branches} | ${cdate} | ${subject} |"

.github/workflows/checkpoint-audit.yml:14

  • This workflow uploads artifacts, but its top-level permissions: block only grants contents: read. In this repo, workflows that both restrict permissions and upload artifacts also grant actions: read (e.g., .github/workflows/e2e.yml). Without it, artifact upload can fail in orgs that default to read-only tokens.
permissions:
  contents: read

  • Files reviewed: 3/4 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

- Token auth header uses standard casing `Authorization: Basic`, matching
  the CLI's implementation (checkpoint/remote/git.go) instead of the
  non-standard `AUTHORIZATION: basic`.
- Markdown table escapes `|` in the author and branches cells too, not
  just the subject — author names are free-form and a pipe is a legal
  git refname character, either of which would break the table.
- Workflow grants `actions: read` alongside `contents: read` so
  actions/upload-artifact works under a restricted token, matching
  e2e.yml / e2e-checkpoint-store.yml.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 01KZ3BQF65CD8199EEWH1FFFH9

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not ready to approve

The audit script currently buffers the full git ls-remote output in memory, which can become a scaling issue as the checkpoint remote grows.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (2)

scripts/checkpoint-audit.sh:85

  • The git ls-remote output is captured into the remote_refs shell variable, which needlessly holds the entire remote ref listing in memory before processing. As the checkpoint remote grows, this can become a significant memory/CPU overhead for the daily workflow. Stream ls-remote directly into the awk/sort pipeline and only write the extracted IDs to the temp file.
echo "Enumerating checkpoint refs on ${CHECKPOINT_REPO} ..." >&2
if [ -n "$TOKEN" ]; then
  auth_b64=$(printf 'x-access-token:%s' "$TOKEN" | base64 | tr -d '\n')
  # Standard header name/scheme casing, matching the CLI (Authorization: Basic).
  remote_refs=$(git -c "http.extraheader=Authorization: Basic ${auth_b64}" \
    ls-remote "$CHECKPOINT_URL" 'refs/entire/checkpoints/*') || {

mise.toml:35

  • The task description implies ENTIRE_CHECKPOINT_TOKEN must be set, but the script and workflow are explicitly designed to work anonymously while the checkpoint repo is public. This is misleading for local runs.
[tasks."checkpoint:audit"]
description = "Audit that recent commits' checkpoints exist on the checkpoint remote (set ENTIRE_CHECKPOINT_TOKEN)"
run = "scripts/checkpoint-audit.sh"
  • Files reviewed: 3/4 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants