Skip to content

chore(ci): claims-drift job and weekly GitHub signals - #730

Open
ajitpratap0 wants to merge 1 commit into
mainfrom
chore/claims-drift-and-signals
Open

ajitpratap0 wants to merge 1 commit into
mainfrom
chore/claims-drift-and-signals

Conversation

@ajitpratap0

Copy link
Copy Markdown
Owner

Summary

  • Adds --claims [--about "text"] to scripts/gen-readme-metrics: checks that llms.txt (performance line, dialect count, SQL-99 estimate) and the GitHub About text agree with scripts/gen-readme-metrics/metrics.json, and fails when metrics.json or benchmarks/compare/results/latest.json is older than the latest release tag.
  • metrics.json gains measured_at (2026-10-13) so freshness is checkable.
  • New weekly workflow .github/workflows/claims-drift.yml: runs the README check, a seeded-mismatch self-test, and the claims check against the live About text. A second job appends a weekly comment (stars, forks, and with the optional SIGNALS_TOKEN secret: 14d views, clones, top referrers) to a "Weekly GitHub signals" tracking issue, created on first run. pkg.go.dev "Imported by" stays a manual note in each comment.
  • Q10 decision: GITHUB_TOKEN cannot edit About, so the job only detects drift and fails; a maintainer updates About by hand. No admin PAT is stored.

Heads up: the live About text currently says "1.5M+ ops/sec", so the first scheduled run will fail until About is updated by hand (e.g. gh repo edit --description ... with a measured, stage-labelled figure or no number).

Test Plan

  • task check passes
  • go test ./scripts/gen-readme-metrics/ covers seeded llms.txt and About mismatches and stale dates
  • go run ./scripts/gen-readme-metrics --claims passes; with --about "fast 1.5M+ ops/sec" it exits 1
  • actionlint v1.7.7 clean on the new workflow
  • After merge: run the workflow via workflow_dispatch; optionally add the SIGNALS_TOKEN secret for traffic data

Breaking Changes

None.

Closes #684
Part of #634

🤖 Generated with Claude Code

https://claude.ai/code/session_01RkzVu3JY9Y6BRWfjesWts9

Add a --claims mode to gen-readme-metrics that verifies llms.txt, the GitHub
About text and benchmark dates against metrics.json, and a weekly workflow
that runs it plus logs stars, traffic and referrers to a tracking issue.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RkzVu3JY9Y6BRWfjesWts9
@vercel

vercel Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
gosqlx-website Ready Ready Preview Oct 16, 2026 1:00am UTC

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 28s —— View job


PR review: claims-drift job and weekly signals

I read the diff only. I did not run the tests or the workflow.

The design is sound. Making the check detect drift instead of storing an admin PAT is a good call. The tests cover the main mismatch cases. Using flag.Visit to tell an explicit --about "" from an omitted flag is correct. I found no blocking issues, but there are a few things worth fixing.

Issues and suggestions

  1. The self-test can pass for the wrong reason (claims-drift.yml, "Self-test" step).

    • The step only checks that the command exits non-zero, and it sends stderr to /dev/null.
    • Anything else that fails makes it pass: no v<version> tag, a missing llms.txt, a stale measured_at, or a compile error.
    • Capture the output instead and assert it contains the About-specific message:
      out="$(go run ./scripts/gen-readme-metrics --claims --about '1.5M+ ops/sec' 2>&1)" && { echo "::error::seeded mismatch not detected"; exit 1; }
      grep -q 'About text claims' <<<"$out" || { echo "::error::failed for the wrong reason"; echo "$out"; exit 1; }
  2. Failures in the traffic section behave inconsistently (signals job).

    • The script uses set -euo pipefail. A 403 from SIGNALS_TOKEN inside the $(...) in an echo is swallowed, because echo still succeeds. That leaves a blank Views or Clones value in the comment.
    • A failure in the final referrers call aborts the whole step, so no comment is posted at all.
    • Consider wrapping the traffic block (e.g. if ! { ...; } >> "$body") and writing a "traffic fetch failed" note instead. Stars and forks would still get logged.
  3. Fragile llms.txt matching (claims.go, checkLLMS).

    • The check is an exact substring match. It will also fail on reflowed text or a different hardware string.
    • That is probably intended as strictness, but the failure message prints only the expected string. Adding a hint to run the generator, or pointing at the llms.txt line, would make fixes quicker.
    • "- SQL dialects: %d\n" also breaks if the file has CRLF line endings.
  4. Regex edge cases (claims.go, opsClaim and nsClaim).

    • opsClaim accepts only ops/sec or parses/sec. Text such as "1.5M ops per second" or "1.5M operations/sec" would slip through.
    • A rounded figure like "~3M ops/sec" is flagged as a mismatch, because only 1 or 2 decimals are accepted. That is fine if intended, but worth documenting.
    • nsClaim could match unrelated text such as "10 ns", but the \b makes that unlikely.
  5. Test gaps (claims_test.go).

    • runClaims and latestTagDate are untested. That means the latest.json generatedAt branch, including the "generatedAt not found" path, has no coverage.
    • Only checkFresh with a given date string is tested.
    • A small test with a temp dir and a fixture would cover it.
  6. Freshness semantics (claims.go, latestTagDate).

    • git log -1 --format=%cs v<version> returns the tagged commit's date, not the tag creation date. This is usually close enough, but note it.
    • Each new release tag makes the job fail until metrics.json and latest.json are re-measured. That matches the stated intent.
  7. Workflow operations

    • The signals job's issue lookup uses search, which can lag a little. If the first run races with a manual dispatch, two tracking issues could be created. This is unlikely and low impact.
    • GitHub disables scheduled workflows after 60 days without repo activity. This is unlikely to matter for an active repo.
    • The claims-drift job will fail on the first scheduled run until About is edited, as the PR description says. A failed scheduled run only emails people who watch it. You could have the job open or comment on an issue, but that is optional.
  8. Security

    • Nothing concerning. Permissions are minimal and per-job, and actions are pinned by SHA.
    • SIGNALS_TOKEN is only exported inside the step. The About text is passed through a quoted shell variable, not interpolated into the script, so there is no injection risk.

Summary

Item 1 (the self-test) and item 2 (traffic failure handling) are the ones I'd fix before merge. The rest are optional polish.

I can't push edits to .github/workflows/, so I haven't changed anything.

This branch was successfully deployed

1 active deployment
Preview — ecb4dbdf Deployed Oct 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

chore(ci): claims-drift job and weekly GitHub signals [5.3]

1 participant