feat: branch and tag discovery, on-demand snapshots without pinning, FIFO capacity eviction

README.md

@@ -19,7 +19,7 @@ It is built for the workload AI coding agents generate: many small, precise `sea
 
 - **Milliseconds, not seconds.** Indexed search typically completes in **under 5 ms** on the server. Deployed in the same region as its callers, a full request — network included — stays **well under 100 ms**, which is faster than running `grep` or `rg` over a local checkout.
 - **Reproducible answers.** Branches and tags resolve to a commit before anything is read, so a search result, the file it points to and a later diff all refer to the same snapshot.
-- **One contract, four surfaces.** Browser, HTTP, MCP and CLI share the same operations: `repositories`, `availability`, `resolve`, `search`, `read`, `list`, `diff` and `prepare`. MCP adds `read_many` for several reads in one call. See [interfaces](docs/interfaces.md).
+- **One contract, four surfaces.** Browser, HTTP, MCP and CLI share the same operations: `repositories`, `availability`, `resolve`, `search`, `read`, `list`, `diff` and `prepare`. MCP adds `read_many` for several reads in one call and `refs` to find branches and tags. See [interfaces](docs/interfaces.md).
 - **Agent-ready.** A stateless Streamable HTTP MCP endpoint works with any MCP client; anonymous access is read-only and limited to public repositories.
 - **Snapshot retention.** Shallow branch-head synchronization, on-demand pinned SHAs and per-repository storage budgets; see [snapshot configuration](docs/snapshots.md).
 - **Small footprint.** One process manages Git mirrors, the Zoekt child process and background indexing. SQLite by default; PostgreSQL and MongoDB are supported.

docs/interfaces.md

@@ -5,8 +5,8 @@ All routes live under `/sourcegraph/`. Repository names use `owner/repository`,
 | Surface | Entry | Operations |
 | --- | --- | --- |
 | Browser | `/sourcegraph/` | Search, code reader, repositories, Connect page |
-| MCP | `/sourcegraph/mcp` (Streamable HTTP, stateless JSON responses) | `repositories`, `availability`, `resolve`, `search`, `read`, `read_many`, `list`, `diff`, `prepare` |
-| HTTP | Routes below | Same operations except `read_many`; read several ranges with repeated `read` calls |
+| MCP | `/sourcegraph/mcp` (Streamable HTTP, stateless JSON responses) | `repositories`, `availability`, `resolve`, `refs`, `search`, `read`, `read_many`, `list`, `diff`, `prepare` |
+| HTTP | Routes below | Same operations as MCP except `read_many`; read several ranges with repeated `read` calls |
 | CLI | `output/bin/sourcegraph-cli` | `repositories`, `availability`, `resolve`, `search`, `read`, `list`, `diff`, `prepare` |
 
 Use MCP discovery to obtain the current tool input schemas. `GET /sourcegraph/v1/capabilities` lists the HTTP operations.
@@ -29,6 +29,38 @@ For an MCP client supporting Streamable HTTP:
 
 `read_many` reads several files or line ranges in one call. Repository read plugins enrich `read` and `read_many`; see [repository plugins](ref/repository-plugins.md).
 
+Other tools accept a monitored branch or a full SHA. To work with any other branch or a tag, find it with `refs` if needed, `resolve` it to a full SHA, and use that SHA. When `indexed` is false, a repository manager prepares that SHA first.
+
+### Branches, tags and resolve
+
+`resolve` takes `repo` and an optional `revision` and returns `commit/indexed/kind/source/monitored`, plus `ref/committer_time/created_at/author/title` when known and a `hint` when the commit has no index (a repository manager, or this caller if it may prepare, can prepare the SHA). It never downloads code or builds an index.
+
+| `revision` | Resolution | `source` |
+| --- | --- | --- |
+| Omitted, or a monitored branch (also as `refs/heads/<name>`) | Last synchronized head; the remote is not asked, so it may lag the remote's latest commit. A bare name equal to a monitored branch is not checked for a same-named tag; pass `refs/tags/<name>` for the tag | `observed` |
+| Full 40-character SHA | An observed or retained version; otherwise `REVISION_NOT_ELIGIBLE` | `observed` |
+| Any other branch or tag name, `refs/heads/<name>`, `refs/tags/<name>` | The repository's ref catalog when it is fresh and not truncated; otherwise the remote is asked for exactly these names and the matching ref is downloaded to a scratch directory, without file trees, to confirm its object type and commit details, then deleted | `catalog` or `remote` |
+
+A bare name that is both a branch and a tag returns `REF_AMBIGUOUS` listing both; pass the full form. A missing name returns `REF_NOT_FOUND`, and a tag confirmed to end at a tree or blob returns `REF_NOT_COMMIT`. Resolution uses the ref as actually downloaded, so a branch that moves between listing and download resolves to the downloaded commit; an object that cannot be verified returns a retryable `REMOTE_UNAVAILABLE`, never `REF_NOT_COMMIT`. Annotated tags resolve to their commit; `created_at/author/title` describe the tag and `committer_time` the commit. Short SHAs, `HEAD` and revision expressions are still rejected. Identical concurrent remote lookups share one request, and results, including "not found", are cached for 30 seconds (at most 10,000 entries). Remote lookups run at most 2 per repository and 8 per instance, at most 64 may be running or waiting (beyond that, `REMOTE_UNAVAILABLE` with `Retry-After`), and they back off per repository after failures, from 10 seconds up to 5 minutes. Visibility is checked before every cached answer.
+
+`refs` lists a registered repository's branches or tags. Optional arguments: `kind` (`branch` default, `tag`, `all`); `query`, `author` and `title`, case-insensitive substrings of the name, the author or tagger, and the commit title or tag message first line; `since/until` (RFC 3339 or `YYYY-MM-DD`, compared with `created_at`); `sort` (`created`, newest first, the default, or `name`); `first` (1–100, default 30); `after` (the previous `next_cursor`, with the other arguments unchanged). It returns `repo/catalog/sort/refs/next_cursor/has_more` and sometimes a `hint`. Each entry has `name/ref/kind/monitored/indexed`; entries verified to end at a commit add `commit/created_at/committer_time/author/title`, annotated tags add `annotated`, and tags of trees, blobs or unverifiable objects carry `object/target_type` instead of `commit`. `created_at` is the commit time for branches and lightweight tags and the tagging time for annotated tags.
+
+`catalog.state` is `ready`; `stale` (expired, or its refresh failed: the last successful catalog is returned with its `refreshed_at` while a refresh runs in the background); or `names_only`, with `catalog.reason` `filter_unsupported` (the server cannot send commit objects without trees), `ref_limit` (more than 200,000 branches or tags) or `size_limit` (the download exceeded 256 MiB). A `names_only` catalog sorts by name when `sort` is omitted; an explicit `author/title/since/until` or `sort=created` returns `REFS_FILTER_UNAVAILABLE`. `catalog.truncated: true` means only the first 200,000 names are listed and `query` searches only those; `resolve` with a complete name does not depend on the catalog. The cursor is an offset, so a refresh between pages can repeat or skip a few entries.
+
+Catalogs live in service memory. A refresh lists the remote first, so the ref limit applies before anything is downloaded, then downloads branch and tag commit and tag objects, without file trees, into a scratch directory that is parsed and deleted. A catalog stays fresh for 5 minutes or three times its last refresh duration, whichever is longer; an expired catalog is served while one background refresh per repository runs (at most 4 per instance). Every download is checked each second against its size limit (256 MiB for a catalog, 16 MiB for a resolve lookup), a 4 GiB total for all scratch downloads and the instance free-space reserve, and is stopped and deleted when any is exceeded; scratch directories left by a stopped process are removed at startup. When catalogs together exceed 1,000,000 refs, the least recently used ones are dropped.
+
+Without any catalog, a request waits for the first refresh within its remaining deadline less a 2-second margin and otherwise returns `REFS_CATALOG_PREPARING` with `Retry-After` while the refresh continues. A failed refresh without an earlier catalog returns the remote error; with one, the stale catalog is returned. Listing never indexes.
+
+| Error | Meaning |
+| --- | --- |
+| `REF_NOT_FOUND`, `REF_AMBIGUOUS`, `REF_NOT_COMMIT` | No such branch or tag (names are case-sensitive; search with `refs`); a branch and a tag share the name; the tag does not end at a commit |
+| `REFS_CATALOG_PREPARING` | The first listing is still being prepared; retry after `Retry-After`, or resolve a complete name meanwhile |
+| `REFS_FILTER_UNAVAILABLE` | The catalog has names only; filter and sort by name |
+| `REMOTE_ACCESS_DENIED` | The code-source token this repository uses cannot read its remote; a repository manager should check it. The caller's own key is not the cause |
+| `REMOTE_UNAVAILABLE` | The remote is unavailable, timed out or backing off; retry after `Retry-After` |
+| `INSUFFICIENT_STORAGE` | The instance's free disk space is below its reserve; not the repository budget |
+| `BRANCH_NOT_MONITORED` | Returned by tools other than `resolve`, and by prepare, for an unmonitored branch; resolve it to a SHA first |
+
 ## HTTP
 
 Repository routes take `?repo=owner/repository`.
@@ -41,7 +73,8 @@ Repository routes take `?repo=owner/repository`.
 | `GET /sourcegraph/v1/code-sources` | Configured code sources in display order: `id/name/host/username/anonymous/editable/locked` |
 | `GET /sourcegraph/v1/repo/availability` | Indexed commits and readiness; `/v1/repo/commits` is an alias |
 | `GET /sourcegraph/v1/repo/plugins` | Read plugins available for a revision |
-| `POST /sourcegraph/v1/repo/resolve` | Resolve a branch or SHA to a fixed commit |
+| `POST /sourcegraph/v1/repo/resolve` | Resolve a branch, tag or SHA to a fixed commit; see [Branches, tags and resolve](#branches-tags-and-resolve) |
+| `GET /sourcegraph/v1/repo/refs` | List and search branches and tags (`kind/query/author/title/since/until/sort/first/after`) |
 | `POST /sourcegraph/v1/repo/read` | Read a line range |
 | `POST /sourcegraph/v1/repo/list` | List a directory |
 | `POST /sourcegraph/v1/repo/diff` | Changed paths between two commits |

docs/snapshots.md

@@ -4,13 +4,21 @@ Synchronization fetches depth-one snapshots of the watched branch heads. The fir
 
 To prepare a specific baseline, use **Create index** with its full lowercase 40-character commit SHA, or the prepare API with that revision. Missing snapshots are fetched directly from the registered repository, without deepening its full history. The Git server must permit fetching that SHA. A rejected request fails without substituting another commit. The branch selector identifies the watched scope; it does not prove ancestry for an explicitly supplied SHA.
 
-Explicit SHA preparation pins that snapshot. In repository settings, **Pinned snapshots** lists the versions to keep independently of ordinary retention. Removing a SHA releases its pin; it becomes eligible for expiry at the next successful sync. Adding a pin alone does not fetch or index an absent snapshot: prepare the SHA first. A missing or failed index remains visible in task status.
+A snapshot prepared by SHA is an ordinary snapshot: preparation does not pin it. To work with an unmonitored branch or a tag, resolve it to a full SHA first (see [interfaces](interfaces.md#branches-tags-and-resolve)). In repository settings, **Pinned snapshots** lists the versions to keep independently of retention and capacity eviction; pins are only added there. Removing a SHA releases its pin and returns it to ordinary retention. Adding a pin alone does not fetch or index an absent snapshot: prepare the SHA first. A missing or failed index remains visible in task status.
 
-Ordinary snapshots are retained by time since first observation, or by the configured count of most recently observed ordinary snapshots. Pins, current branch heads and active tasks are additionally protected. There are at most 4096 retained snapshots per path group. Commit choices show up to 20 observed snapshots, not the complete Git history.
+Ordinary snapshots are retained by time since first observation, or by the configured count of most recently observed ordinary snapshots. Pins, current branch heads and commits of queued or running tasks are protected and never released. Retention runs after each successful sync and again as soon as a preparation publishes its index. When a path group reaches 4096 snapshots, the oldest unprotected one is released instead of refusing the new snapshot. Commit choices show up to 20 observed snapshots, not the complete Git history.
 
 After a successful sync, expired snapshots lose query eligibility. Git collection then removes objects no longer reachable from any retained snapshot, preserving the complete trees and available file objects of retained versions. Index generations no retained version references are collected once their manifest is older than five minutes: right after the sync that applied retention, before a job would fail on the repository storage budget, and in a background round every minute that collects repository by repository (a repository with a running job is deferred to the next round). The status page shows pending expired indexes, deferred repositories, the last reclaim and the last issue. Adding snapshots alone does not rewrite all packs; collection runs when versions expire and periodically during synchronization. No synchronization means no automatic snapshot expiry. **Clean old indexes** keeps branch-head and pinned indexes; when a latest index is not ready it removes only expired indexes and keeps every retained version's index (`expired_only: true` in the preview).
 
-Each repository has a **Storage budget** (default 100 GiB, configurable from 1 GiB to 10 TiB) covering Git objects, index shards and active preparation files. Preparation stops when the sampled budget is reached. This is a protective budget, not an exact filesystem quota: checks occur between operations and during long-running work, so short-lived overshoot is possible. Increase the budget or release unused snapshots and clean indexes when capacity blocks preparation. Git compaction needs temporary working space and may exceed the repository budget while reclaiming objects; the global free-space reserve remains enforced.
+Each repository has a **Storage budget** (default 100 GiB, configurable from 1 GiB to 10 TiB) covering Git objects, index shards and active preparation files. When it is reached, the oldest unprotected snapshots and their indexes are released, by first observation, instead of refusing work. Eviction runs only at safe points: before the Git fetch of a sync or a SHA preparation, before an index build starts (reserving 1.2 times the size of the group's previous index), and after a preparation or index task finishes. It works in batches of at most 64:
+
+1. The batch leaves the control data in one update while holding the query lock, so reads of those versions finish first; the Git retention marker of each path group that loses versions is removed before the update commits, so collection cannot be skipped as already done; if the marker cannot be removed, nothing is committed and `RECLAIM_FAILED` is returned. They then answer `REVISION_NOT_ELIGIBLE` and can be prepared again.
+2. Indexes no longer referenced are removed, then Git keeps only retained snapshots and repacks reachable objects, after checking instance free space.
+3. Usage is measured again, and further batches follow until the reservation fits or nothing unprotected remains.
+
+Failures name their cause. `REPOSITORY_STORAGE_LIMIT` means protected snapshots alone fill the budget and `SNAPSHOT_LIMIT` that they fill 4096 versions: remove pins you no longer need or raise the budget. `INSUFFICIENT_STORAGE` means the instance's free space is below its reserve, which is not the repository budget. `RECLAIM_FAILED` means collection did not finish and continues in the background. The reservation is an estimate: a running build releases nothing (its files are in use) and is refused only above 120% of the budget, always within the instance free-space reserve, and the release after it brings usage back under budget. This is a protective budget, not an exact filesystem quota, so short-lived overshoot is possible. Git compaction needs temporary working space and may exceed the repository budget while reclaiming objects.
+
+The control data is the source of truth, and physical collection follows it idempotently. Every change that releases versions (retention after a sync or a preparation, budget eviction, the 4096 limit) removes the affected path groups' retention markers before it commits and is abandoned if that fails, and a sync reads the repository again after eviction before recording new versions; a missing marker, or one naming a released commit, means Git collection is unfinished, and only a successful collection writes it again. Besides the index round, a separate pass every minute (one at a time) checks each repository's marker and, once it holds that repository's preparation slot (busy repositories wait for the next pass), finishes Git collection using the repository's Git timeout rather than the index round's 30-second budget. Preparation and index tasks do the same before they start. An interrupted process or a failing later sync therefore does not keep released space.
 
 `SOURCEGRAPH_MIN_FREE_BYTES` protects the instance's remaining disk space across repositories. Keep enough additional free capacity for replacement packs and index builds. Back up the control database, including pin and retention settings; Git snapshots and indexes reside on the instance volume.