feat: private repositories visible only to creators, managers and administrators

docs/interfaces.md

@@ -50,11 +50,15 @@ Repository routes take `?repo=owner/repository`.
 
 Repository registration, maintenance, statistics and site settings live under `/sourcegraph/api/admin/` and are used by the website's administration pages. Use the Connect page for current request examples.
 
+### Repository visibility
+
+Each repository is public or private (`public_read`, default `true`; repositories registered before the field existed stay public). Anonymous callers read public repositories only, and only while the site allows public reading. A private repository is visible only to its creator, its managers, site administrators signed in on the website and the deployment credential. Personal API keys read as their owner: public repositories plus the private ones the owner created or manages, even when the owner is a site administrator. Every other caller gets the same `404 REPOSITORY_NOT_FOUND` as for an unregistered name from details, search, read, list, diff, plugins, prepare and MCP tools, and catalogs list only visible repositories. Statistics omit all repository names for anonymous callers; other callers without site-wide read keep only registered repositories they can see plus the `_overflow` aggregate, so unregistered, deleted and hidden names look the same. Registering an existing name returns `409 REPOSITORY_EXISTS` with its registrant in `creator` (`id/name/email/department`); the website suggests asking them to be added as a manager, or a site administrator when no registrant is recorded. Background synchronization, indexing and cleanup are not narrowed by visibility.
+
 ### Repository managers
 
 Site administrators manage every repository. The site setting **Let signed-in users register repositories** (`access.allow_user_repositories`) defaults to off; older settings without the field read as off, so upgrading does not change who can register. When it is on, any signed-in user can register a repository and becomes its only manager. Repositories registered before managers existed treat their creator as manager until the list is first edited; after that the stored list is used even when empty, leaving management to administrators. A repository has at most 50 managers.
 
-Repository managers and administrators can use every `?repo=` management route on that repository: edit settings, preview, prepare and add indexes, list commit options, clean up, cancel jobs, delete and purge, and add or remove managers, including themselves. Others receive `403 REPOSITORY_MANAGEMENT_DENIED`; registration while the setting is off returns `403 REGISTRATION_DISABLED`. Users, site settings, global code-source tokens and other site-wide routes stay administrator-only. Personal API keys read what their owner reads and may call only the prepare route besides that.
+Repository managers and administrators can use every `?repo=` management route on that repository: edit settings, preview, prepare and add indexes, list commit options, clean up, cancel jobs, delete and purge, and add or remove managers, including themselves. Others receive `403 REPOSITORY_MANAGEMENT_DENIED`; registration while the setting is off returns `403 REGISTRATION_DISABLED`. Users, site settings, global code-source tokens and other site-wide routes stay administrator-only. Personal API keys read what their owner reads (see [Repository visibility](#repository-visibility)) and may call only the prepare route besides that; a site administrator's key prepares only repositories it can see.
 
 | Route | Purpose |
 | --- | --- |
@@ -92,7 +96,7 @@ sourcegraph-cli read -repo context4ai/context -path README.md -start-line 1 -end
 | `SOURCEGRAPH_URL` | Service URL, default `https://context4ai.org/sourcegraph`; must be a credential-free HTTPS URL |
 | `SOURCEGRAPH_API_TOKEN` | Optional bearer credential, at least 32 characters |
 
-Personal API keys are created from a signed-in user's settings. A key can prepare (MCP `prepare` or `POST /api/admin/v1/repo/prepare`) a repository its owner manages, or any repository when its owner is currently a site administrator; this is checked on every call. MCP lists `prepare` only for keys whose owner is an administrator or manages at least one repository (refreshed within a minute). Keys are limited to 10 prepare calls a minute.
+Personal API keys are created from a signed-in user's settings. A key can prepare (MCP `prepare` or `POST /api/admin/v1/repo/prepare`) a repository its owner manages, or any repository it can see when its owner is currently a site administrator; this is checked on every call. MCP lists `prepare` only for keys whose owner is an administrator or manages at least one repository (refreshed within a minute). Keys are limited to 10 prepare calls a minute.
 
 Manual preparation is paced per repository across the website, HTTP and MCP. A request identical to a queued or running job (same group, kind and revision) returns that job and adds no queue load. Any other new request within 30 seconds of the repository's last accepted manual request returns `429 RATE_LIMITED` with `Retry-After`; one request queuing every monorepo group is a single request. Scheduled work is not paced.
 

internal/httpserver/private_repository_test.go

@@ -0,0 +1,199 @@
+package httpserver
+
+import (
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+	"testing"
+	"time"
+
+	"github.com/context4ai/sourcegraph/internal/control"
+	"github.com/context4ai/sourcegraph/internal/sitesettings"
+	"github.com/context4ai/sourcegraph/internal/sso"
+	"github.com/modelcontextprotocol/go-sdk/mcp"
+)
+
+type bearerTransport string
+
+func (b bearerTransport) RoundTrip(r *http.Request) (*http.Response, error) {
+	r = r.Clone(r.Context())
+	r.Header.Set("Authorization", "Bearer "+string(b))
+	return http.DefaultTransport.RoundTrip(r)
+}
+
+// personalToken stores an API key for user and returns the raw token.
+func (s *site) personalToken(user string) string {
+	raw := "sgk_" + hexID("key-" + user)[:43]
+	h := sha256.Sum256([]byte(raw))
+	s.people.mu.Lock()
+	defer s.people.mu.Unlock()
+	if s.people.keys == nil {
+		s.people.keys = map[string]sso.APIKey{}
+	}
+	s.people.keys[hex.EncodeToString(h[:])] = sso.APIKey{ID: "k-" + user[:8], UserID: user, Hash: hex.EncodeToString(h[:]), Created: time.Now()}
+	return raw
+}
+
+// mcpSees reports whether the repositories and availability tools expose repo.
+func mcpSees(t *testing.T, endpoint, token, repo string) (listed, available bool) {
+	t.Helper()
+	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+	defer cancel()
+	transport := &mcp.StreamableClientTransport{Endpoint: endpoint}
+	if token != "" {
+		transport.HTTPClient = &http.Client{Transport: bearerTransport(token)}
+	}
+	session, err := mcp.NewClient(&mcp.Implementation{Name: "visibility-test", Version: "1"}, nil).Connect(ctx, transport, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer session.Close()
+	res, err := session.CallTool(ctx, &mcp.CallToolParams{Name: "repositories", Arguments: map[string]any{}})
+	if err != nil || res.IsError {
+		t.Fatalf("repositories: %v %v", err, res)
+	}
+	body, _ := json.Marshal(res)
+	listed = strings.Contains(string(body), repo)
+	res, err = session.CallTool(ctx, &mcp.CallToolParams{Name: "availability", Arguments: map[string]any{"repo": repo}})
+	if err != nil {
+		t.Fatal(err)
+	}
+	return listed, !res.IsError
+}
+
+func TestPrivateRepositoryVisibility(t *testing.T) {
+	admin, alice, bob, carol := hexID("admin"), hexID("alice"), hexID("bob"), hexID("carol")
+	s := newSite(t, map[string]string{"admin": admin, "alice": alice, "bob": bob, "carol": carol}, admin)
+	rev := s.settings.Snapshot().Access.Revision
+	if _, err := s.settings.Update(context.Background(), sitesettings.AccessName, rev, []byte(`{"public_read":true,"allow_registration":true,"allow_user_repositories":true}`), "sso:admin"); err != nil {
+		t.Fatal(err)
+	}
+	private := `{"name":"context4ai/private","policy":{"branches":["main"],"count":10},"code_provider":"github","public_read":false}`
+	if code, body := s.call(alice, "POST", "/sourcegraph/api/admin/v1/repos", private, "Idempotency-Key", "register-private"); code != 202 {
+		t.Fatalf("register: %d %v", code, body)
+	}
+	// A taken name answers 409 with its registrant, so others can ask to be added as managers.
+	code, conflict := s.call(bob, "POST", "/sourcegraph/api/admin/v1/repos", private, "Idempotency-Key", "register-bob")
+	creator, _ := conflict["creator"].(map[string]any)
+	if code != 409 || errorCode(conflict) != "REPOSITORY_EXISTS" || creator["name"] != "alice" || creator["email"] != "alice@example.com" {
+		t.Fatalf("taken name: %d %v", code, conflict)
+	}
+	s.db.mu.Lock()
+	r := s.db.repos["context4ai/private"]
+	r.ManagersSet, r.Managers = true, []string{alice, carol}
+	s.db.repos[r.Name] = r
+	s.db.mu.Unlock()
+	// Registered before per-repository visibility: no public_read, still public.
+	legacy := "context4ai/legacy"
+	if err := s.db.Create(context.Background(), control.Repository{ID: control.ID(legacy), Name: legacy, Enabled: true, Revision: 1, Heads: map[string]string{}, Versions: []control.Version{}}); err != nil {
+		t.Fatal(err)
+	}
+
+	server := httptest.NewServer(s.handler)
+	defer server.Close()
+	endpoint := server.URL + "/sourcegraph/mcp"
+	catalog := func(cookie, bearer string) string {
+		r := httptest.NewRequest("GET", "/sourcegraph/v1/repos", nil)
+		if cookie != "" {
+			r.AddCookie(&http.Cookie{Name: "sourcegraph_session", Value: "cookie-" + cookie})
+		}
+		if bearer != "" {
+			r.Header.Set("Authorization", "Bearer "+bearer)
+		}
+		w := httptest.NewRecorder()
+		s.handler.ServeHTTP(w, r)
+		if w.Code != 200 {
+			t.Fatalf("catalog: %d %s", w.Code, w.Body.String())
+		}
+		return w.Body.String()
+	}
+	availability := "/sourcegraph/v1/repo/availability?repo=context4ai%2Fprivate"
+	service := strings.Repeat("t", 32)
+
+	// Website: the creator and site administrators see it, other users and anonymous visitors do not.
+	for _, c := range []struct {
+		name, user string
+		want       bool
+	}{{"creator", alice, true}, {"manager", carol, true}, {"administrator", admin, true}, {"other user", bob, false}, {"anonymous", "", false}} {
+		code, _ := s.call(c.user, "GET", availability, "")
+		listed := strings.Contains(catalog(c.user, ""), "context4ai/private")
+		if (code == 200) != c.want || listed != c.want {
+			t.Errorf("website %s: availability %d, listed %v, want %v", c.name, code, listed, c.want)
+		}
+		if !c.want && code != 404 {
+			t.Errorf("website %s: hidden repository must answer 404, got %d", c.name, code)
+		}
+		if !strings.Contains(catalog(c.user, ""), legacy) {
+			t.Errorf("website %s: legacy repository hidden", c.name)
+		}
+	}
+
+	// MCP and HTTP: a personal token reads as its owner, an administrator's included.
+	for _, c := range []struct {
+		name, token string
+		want        bool
+	}{{"anonymous", "", false}, {"creator token", s.personalToken(alice), true}, {"other user token", s.personalToken(bob), false}, {"administrator token", s.personalToken(admin), false}, {"service credential", service, true}} {
+		listed, available := mcpSees(t, endpoint, c.token, "context4ai/private")
+		if listed != c.want || available != c.want {
+			t.Errorf("MCP %s: listed %v, available %v, want %v", c.name, listed, available, c.want)
+		}
+		if got := strings.Contains(catalog("", c.token), "context4ai/private"); got != c.want {
+			t.Errorf("HTTP %s: listed %v, want %v", c.name, got, c.want)
+		}
+		if listed, available := mcpSees(t, endpoint, c.token, legacy); !listed || !available {
+			t.Errorf("MCP %s: legacy repository hidden", c.name)
+		}
+	}
+
+	// Every repository route answers a hidden repository exactly like an unregistered name.
+	send := func(cookie, bearer, method, path, body string) string {
+		r := httptest.NewRequest(method, path, strings.NewReader(body))
+		if cookie != "" {
+			r.AddCookie(&http.Cookie{Name: "sourcegraph_session", Value: "cookie-" + cookie})
+			r.Header.Set("Origin", testOrigin)
+			r.Header.Set("X-CSRF-Token", "csrf-"+cookie)
+		}
+		if bearer != "" {
+			r.Header.Set("Authorization", "Bearer "+bearer)
+		}
+		r.Header.Set("Idempotency-Key", "visibility-probe")
+		w := httptest.NewRecorder()
+		s.handler.ServeHTTP(w, r)
+		var out map[string]any
+		_ = json.Unmarshal(w.Body.Bytes(), &out)
+		return http.StatusText(w.Code) + " " + errorCode(out)
+	}
+	routes := []struct{ method, path, body string }{
+		{"POST", "/sourcegraph/api/search", `{"Pattern":"x"}`},
+		{"POST", "/sourcegraph/v1/repo/read", `{"Path":"README.md"}`},
+		{"POST", "/sourcegraph/v1/repo/list", `{}`},
+		{"POST", "/sourcegraph/v1/repo/diff", `{"Base":"main","Head":"main"}`},
+		{"POST", "/sourcegraph/v1/repo/resolve", `{}`},
+		{"GET", "/sourcegraph/v1/repo/plugins", ""},
+		{"GET", "/sourcegraph/v1/repo/commits", ""},
+		{"POST", "/sourcegraph/api/admin/v1/repo/prepare", `{"kind":"sync"}`},
+	}
+	for _, c := range []struct {
+		name, cookie, bearer string
+		want                 bool
+	}{
+		{"creator session", alice, "", true}, {"manager session", carol, "", true}, {"administrator session", admin, "", true},
+		{"creator token", "", s.personalToken(alice), true}, {"manager token", "", s.personalToken(carol), true}, {"service credential", "", service, true},
+		{"other user session", bob, "", false}, {"other user token", "", s.personalToken(bob), false}, {"administrator token", "", s.personalToken(admin), false}, {"anonymous", "", "", false},
+	} {
+		for _, route := range routes {
+			private := send(c.cookie, c.bearer, route.method, route.path+"?repo=context4ai%2Fprivate", route.body)
+			missing := send(c.cookie, c.bearer, route.method, route.path+"?repo=context4ai%2Fmissing", route.body)
+			if c.want && strings.HasSuffix(private, "REPOSITORY_NOT_FOUND") {
+				t.Errorf("%s %s: visible repository answered %s", c.name, route.path, private)
+			}
+			if !c.want && private != missing {
+				t.Errorf("%s %s: hidden repository answered %q, unregistered name %q", c.name, route.path, private, missing)
+			}
+		}
+	}
+}

internal/httpserver/repository_access.go

@@ -63,6 +63,12 @@ func authorizeDelegated(ctx context.Context, r *http.Request, s *service.Service
 	return nil
 }
 
+// registrationConflict is the 409 for a taken name, with its registrant.
+type registrationConflict struct {
+	contract.ErrorResponse
+	Creator *sso.Person `json:"creator"`
+}
+
 // repositoryView adds who created and manages a repository, and whether the
 // caller may manage it. People are shown to signed-in callers only.
 type repositoryView struct {

internal/httpserver/repository_access_test.go

@@ -32,6 +32,7 @@ type people struct {
 	mu       sync.Mutex
 	users    map[string]sso.User
 	sessions map[string]sso.Session
+	keys     map[string]sso.APIKey
 	access   sso.AccessControl
 }
 
@@ -77,6 +78,14 @@ func (p *people) GetSession(_ context.Context, id string) (sso.Session, error) {
 	return sso.Session{}, sso.ErrNotFound
 }
 func (p *people) DeleteSession(context.Context, string) error { return nil }
+func (p *people) FindAPIKey(_ context.Context, hash string) (sso.APIKey, error) {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	if k, ok := p.keys[hash]; ok {
+		return k, nil
+	}
+	return sso.APIKey{}, sso.ErrNotFound
+}
 
 // repositories is an in-memory service.Store.
 type repositories struct {
@@ -126,6 +135,7 @@ type site struct {
 	handler  http.Handler
 	settings *sitesettings.Service
 	db       *repositories
+	people   *people
 }
 
 // fakeIssuer serves just enough OIDC discovery for sso.New; no login runs.
@@ -167,7 +177,7 @@ func newSite(t *testing.T, users map[string]string, admins ...string) *site {
 		t.Fatal(err)
 	}
 	t.Cleanup(s.Close)
-	return &site{t: t, handler: RuntimeWithSettings(config.Defaults(), slog.New(slog.NewTextHandler(io.Discard, nil)), s, strings.Repeat("t", 32), settings, auth), settings: settings, db: db}
+	return &site{t: t, handler: RuntimeWithSettings(config.Defaults(), slog.New(slog.NewTextHandler(io.Discard, nil)), s, strings.Repeat("t", 32), settings, auth), settings: settings, db: db, people: store}
 }
 
 // call sends a browser request as user (empty for anonymous) and decodes the JSON body.

internal/httpserver/runtime.go

@@ -271,6 +271,7 @@ func runtime(c config.Config, log *slog.Logger, s *service.Service, token string
 		}
 		var result any
 		var err error
+		var existsCreator *sso.Person
 		decode := func(v any) bool {
 			b, e := io.ReadAll(http.MaxBytesReader(w, r.Body, c.MaxBodyBytes))
 			if e == nil {
@@ -333,7 +334,15 @@ func runtime(c config.Config, log *slog.Logger, s *service.Service, token string
 						if q.SyncIntervalMinutes != nil {
 							interval = *q.SyncIntervalMinutes
 						}
-						result, err = s.RegisterWithGitTimeout(r.Context(), q.Name, policy, interval, q.IndexMode, q.PathGroups, q.PublicRead, q.CodeProvider, q.GitTimeoutMinutes, r.Header.Get("Idempotency-Key"))
+						var row control.Repository
+						row, err = s.RegisterWithGitTimeout(r.Context(), q.Name, policy, interval, q.IndexMode, q.PathGroups, q.PublicRead, q.CodeProvider, q.GitTimeoutMinutes, r.Header.Get("Idempotency-Key"))
+						result = row
+						// A taken name names its registrant, so the caller can ask to become a manager.
+						var ce *contract.Error
+						if row.CreatorID != "" && browserAuth != nil && errors.As(err, &ce) && ce.Code == "REPOSITORY_EXISTS" {
+							creator := browserAuth.People(r.Context(), []string{row.CreatorID})[0]
+							existsCreator = &creator
+						}
 					}
 					status = 202
 				}
@@ -495,7 +504,11 @@ func runtime(c config.Config, log *slog.Logger, s *service.Service, token string
 				if ce.RetryAfterSeconds > 0 {
 					w.Header().Set("Retry-After", strconv.Itoa(ce.RetryAfterSeconds))
 				}
-				writeError(w, id, status, ce.Code, ce.Message)
+				if existsCreator != nil {
+					writeJSON(w, status, registrationConflict{ErrorResponse: contract.ErrorResponse{Error: ce.Message, Meta: contract.Meta{Protocol: "zoekt-single-repo-v1", RequestID: id, Status: "error", Code: ce.Code}}, Creator: existsCreator})
+				} else {
+					writeError(w, id, status, ce.Code, ce.Message)
+				}
 			} else {
 				status = 503
 				writeError(w, id, status, "DEPENDENCY_UNAVAILABLE", "Operation could not complete within its budget.")

internal/identity/identity.go

@@ -52,10 +52,24 @@ func IsAdmin(ctx context.Context) bool {
 	return value
 }
 
+// ReadsAllRepositories reports whether repository visibility does not narrow
+// the caller: an administrator session, the deployment credential, or
+// background work that carries no request identity. Every HTTP request is
+// anonymous, a session or a personal token by the time it reaches the service,
+// so an empty actor never comes from a caller.
+func ReadsAllRepositories(ctx context.Context) bool {
+	if IsAdmin(ctx) {
+		return true
+	}
+	actor, _ := ctx.Value(actorKey{}).(string)
+	return actor == "" && !IsAnonymous(ctx)
+}
+
 type ownerAdminKey struct{}
 
 // WithTokenOwnerAdmin records that a personal token's owner is currently an
-// administrator. It only widens index preparation, never site management.
+// administrator. It only widens index preparation of repositories the token can
+// see, never repository visibility or site management.
 func WithTokenOwnerAdmin(ctx context.Context) context.Context {
 	return context.WithValue(ctx, ownerAdminKey{}, true)
 }

internal/service/prepare_access.go

@@ -9,11 +9,16 @@ import (
 
 // AuthorizePrepare admits site administrators, the deployment credential, a
 // personal token whose owner is currently an administrator, and the
-// repository's managers through their session or personal token.
+// repository's managers through their session or personal token. An
+// administrator's token prepares only repositories it can see.
 func (s *Service) AuthorizePrepare(ctx context.Context, name string) error {
-	if identity.IsAdmin(ctx) || identity.TokenOwnerAdmin(ctx) {
+	if identity.IsAdmin(ctx) {
 		return nil
 	}
+	if identity.TokenOwnerAdmin(ctx) {
+		_, err := s.Repo(ctx, name)
+		return err
+	}
 	user := identity.UserID(ctx)
 	if user == "" || identity.IsAnonymous(ctx) {
 		return contract.Fail("ACCESS_DENIED", "Index preparation requires a repository manager or an administrator.", 403)