Skip to content

feat: branch and tag discovery, on-demand snapshots without pinning, FIFO capacity eviction - #14

Merged
qiansc merged 1 commit into
mainfrom
feat/refs-on-demand
Nov 8, 2026
Merged

qiansc merged 1 commit into
mainfrom
feat/refs-on-demand

Conversation

@qiansc

@qiansc qiansc commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Background

Users want to work with code on branches that are not monitored and on tags: resolve a branch name to a commit, list and search branches, and search and resolve tags. Until now a repository had at most 8 monitored branches, resolve rejected other branches with BRANCH_NOT_MONITORED and could not resolve tags, preparing a full SHA pinned it automatically, and a full snapshot budget refused new work.

Changes

  • refs (new): MCP tool refs and GET /v1/repo/refs list and search a registered repository's branches and tags from the remote, newest first, with name, author, title and time filters and paging. Catalogs live in memory; each refresh lists the remote first (beyond 200,000 per kind only names are kept), then downloads commit and tag objects without file trees into a scratch directory that is parsed and removed.
  • resolve (changed): accepts any branch or tag name and refs/heads|tags/..., returning the commit, whether it is indexed and its commit details; monitored branches keep the synchronized head. Remote lookups are coalesced, cached (including "not found"), bounded (64 running or waiting) and back off on failure; resolution uses the refs actually downloaded, and unverifiable objects return a retryable error.
  • prepare / snapshots: preparing a full SHA keeps an ordinary snapshot. Retention runs after each preparation and index task. At 4096 snapshots or the storage budget, the oldest unprotected snapshots are released oldest first in batches of at most 64 at safe points (before Git fetches, before builds, after tasks).
  • Collection recovery: the affected path group's retention marker is removed before versions are revoked (the update is abandoned if that fails); a separate background pass with each repository's Git timeout, and every job before it starts, finishes pending Git collection; inspection failures are logged with a sanitized reason.
  • Docs: docs/interfaces.md, docs/snapshots.md, README; website copy and the Connect page.

Risks

  • Refreshes and lookups contact the code host; they are bounded per repository and instance, back off, and are limited by download size (256 MiB / 16 MiB), a 4 GiB total for scratch downloads and the free-space reserve. Failures do not affect existing search and reads.
  • Capacity eviction removes old ordinary snapshots; branch heads, pins and snapshots of active jobs are protected, and released versions can be prepared again.

Backward compatibility

  • resolve adds fields; commit/indexed/committer_time and the behavior for monitored branches and full SHAs are unchanged.
  • Existing pins are kept as they are.
  • SNAPSHOT_LIMIT / REPOSITORY_STORAGE_LIMIT are returned only when protected snapshots fill the limit; new codes REF_*, REFS_*, REMOTE_*, RECLAIM_FAILED.
  • Deploy the service alone; no data migration.

Testing

  • make check (gofmt, go vet), check-web, go test ./..., go test -race ./internal/service/ ./internal/gitstore/ and the production build pass.
  • New tests: internal/gitstore/refs_test.go (real Git fixture: listing, exact lookup, filtered fetch, tag of a tree, download abort, marker invalidation and permission errors); internal/service/refs_test.go (resolve forms, ambiguity, not found, non-commit, moved refs, refs filters, sorting, paging, names-only, truncation, first-listing preparation, failed refresh keeps the catalog, private repositories 404, lookup queue bound); internal/service/eviction_test.go (FIFO order, protection, batches, error causes, release before fetch, end-to-end sync after eviction, aborted invalidation, per-group invalidation, inaccessible repository).
  • Against GitHub microsoft/vscode (6,356 refs): first catalog 10.2 s, cached query 26 µs, remote exact resolve 6.2 s, no scratch directories left.

🤖 Generated with Claude Code

…FIFO capacity eviction

- refs (MCP tool and GET /v1/repo/refs) lists and searches a registered
  repository's branches and tags from the remote, newest first, with author,
  title and time filters; catalogs are parsed into memory from tree-less
  scratch downloads that are always removed.
- resolve accepts any branch or tag name of a registered repository and
  reports whether the commit is indexed; monitored branches keep the
  synchronized head. Remote lookups are coalesced, cached, bounded and back
  off on failure.
- Preparing a full SHA no longer pins it. Retention runs after each
  preparation and index task; at 4096 snapshots or the storage budget the
  oldest unprotected snapshots are released in batches at safe points.
- Git collection left by interrupted releases is finished by a separate
  background pass and before jobs start; retention markers are invalidated
  per path group before versions are revoked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@qiansc
qiansc merged commit 78191c5 into main Nov 8, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant