Source-code protection: gate raw source (thinking trace + citation fragments) behind source:view (#66)

docs/superpowers/specs/2026-08-20-source-code-protection-design.md

@@ -0,0 +1,185 @@
+# Source-code protection design (#66)
+
+## Problem
+
+The knowledge agent ingests private source repos and reads raw source via the
+`read_source` tool to answer questions. Reading is fine and must stay
+unrestricted — the agent needs the code to answer well. The problem is at the
+**user-facing output**: today the raw source can reach the user verbatim,
+turning the Q&A surface into a source-download channel.
+
+Requirement: the agent may freely read code, but must not hand the user
+complete source.
+
+## Source channels to the user
+
+There are exactly three ways source content reaches a user today:
+
+1. **The answer** — `token`/`done` SSE events (`server.py:345`, `:431`). The
+   model can paste source into its free-form answer text.
+2. **The thinking trace** — `thinking` SSE event (`server.py:341`). Reasoning
+   text (Anthropic extended thinking; OpenAI-compatible reasoning models) can
+   quote source.
+3. **Citation fragments** — the `source` SSE event (`server.py:410`) carries
+   only `repo/file_path/line_start/line_end` **references, not the body**. The
+   body reaches the user only when they click a citation, which calls
+   `GET /api/graph/source` (`graph_api.py:85`) — the only endpoint that returns
+   raw `lines`. Gated today by `CAP_EXPLORE` alone.
+
+The `read_source` tool's raw return value is **not** streamed to the client
+(only `search_entities` emits a `tool_result` with a count; `read_source` emits
+just the reference `source` event). So the tool output body never reaches the
+user except through channel 1 or 3.
+
+## Decisions (locked)
+
+- **Answer channel: prompt only, always on.** No code enforcement, no
+  post-hoc redaction, no streaming line-cap. A system-prompt rule instructs the
+  model to explain and quote minimally and to decline full-file dumps. Applies
+  to every role, including admin (it is guidance, not a gate).
+- **Thinking + citations: one role capability**, default-deny.
+- **Dropped from the original issue (YAGNI):** aggregate per-conversation
+  output budget, streaming line-cap, post-hoc verbatim-diff redaction, per-repo
+  strictness knobs. Prompt-only + a single role gate meets the requirement.
+
+## Design
+
+### New capability: `source:view`
+
+Grants visibility of raw source detail — the thinking trace **and**
+citation-fragment bodies.
+
+- Added to `auth.py`: new constant `CAP_SOURCE_VIEW = "source:view"`, appended
+  to `ALL_CAPABILITIES` (canonical UI/display order). No `:manage` sibling (it
+  is a bare grant like `chat`/`explore`), so `_MANAGE_IMPLIES_VIEW` is
+  untouched.
+- **Not** added to `DEFAULT_CAPABILITIES` → default-deny. `admin` bypasses all
+  capability checks as usual.
+- Added to `web/src/capabilities.ts` (`CAP`, `CAPABILITY_OPTIONS`) so it renders
+  in the role editor.
+
+> **Amended after review of the running build.** The original design kept the
+> `file:line` reference rows visible to all chat users and gated only the body.
+> In practice a restricted user still saw the reasoning trace and clickable
+> citations, and — critically — the reference metadata was only hidden in the
+> UI, still crossing the wire in the SSE stream. Decision revised to
+> **answer-only** for users without `source:view`: the `/chat` stream carries
+> only the answer (`token`/`done`/`error`), and the reasoning trace + all
+> source/citation references are withheld **server-side** (not just hidden in
+> the client). The server still records the full trace/sources to the audit
+> log. See "Enforcement point 2" below.
+
+### Enforcement point 1 — answer (prompt)
+
+Extend the agent system prompt in `agent.py` (the numbered guidance block) with
+a source-protection rule: explain behavior and structure, quote only short
+illustrative snippets, never reproduce complete or near-complete files; if the
+user asks to print/dump a whole file, decline and offer to explain instead. The
+existing citation guidance (use `read_source` so files appear in the Sources
+panel) is unchanged — reading and citing still happen; only bulk reproduction in
+the answer is discouraged.
+
+### Enforcement point 2 — thinking trace (`server.py`)
+
+The chat stream handler already resolves the caller's `role` (see
+`server.py:236`, `:274`). Compute a per-request flag:
+
+```
+can_view_source = (user.role == ROLE_ADMIN) or (CAP_SOURCE_VIEW in caps)
+```
+
+where `caps` is the expanded capability set for the role. When
+`can_view_source` is false, the stream is **answer-only**: withhold every
+non-answer event so the reasoning trace and all source references never cross
+the wire. Concretely, gate these `yield _sse(...)` calls on `can_view_source`:
+
+- `thinking` — the extended-thinking deltas.
+- `tool` (`on_tool_start`) and `tool_result` — tool names/inputs (a
+  `read_source` input *is* a file/line reference).
+- `source` — the clickable citation reference.
+- `done` — send `"sources": []` instead of the accumulated list.
+
+Only `token`, `done` (answer text), and `error` reach a restricted user. The
+audit accumulators (`tool_calls`, `sources`) are still built and written to the
+audit log — only the client stream is gated, not the server-side record.
+
+Why server-side and not just client hiding: the client re-labels the model's
+between-tool narration (`token` text before a `tool` event) as a "thinking"
+step, so hiding only `kind==="thinking"` rows still showed a trace; and any
+client-only gate leaves the references visible to anyone reading the raw SSE
+stream. Gating at emission is the actual boundary.
+
+### Enforcement point 3 — citation fragment (`graph_api.py`)
+
+`GET /api/graph/source` is the only endpoint returning raw `lines`. Its
+dependency becomes `require_cap(CAP_SOURCE_VIEW)` — `source:view` **replaces**
+`CAP_EXPLORE` here rather than being added to it. This is deliberate: `/source`
+backs the **chat citation preview** (`SourceCard`), not just the Explorer, so a
+chat user granted `source:view` but not `explore` must still reach it;
+requiring `explore` as well would break citation previews for exactly that role.
+`source:view` is the source-specific grant, so gating `/source` on it alone is
+the correct condition. Without the cap the endpoint returns 403; the
+`search`/`entity`/`neighbors` endpoints keep `CAP_EXPLORE` and are unaffected.
+
+Admin keeps working automatically: `require_cap` checks `effective_caps`
+(`auth.py:390`), and `effective_caps` returns `set(ALL_CAPABILITIES)` for admin
+(`auth.py:377`) — so adding `source:view` to `ALL_CAPABILITIES` grants it to
+admin with no special-casing.
+
+### Frontend (`web/src/`)
+
+The app already fetches `capabilities` from `/auth/me` and gates UI with
+`hasCap`. Thread the flag `hasCap(caps, CAP.sourceView)` into `Chat.tsx` as
+`canViewSource`. These are defense-in-depth — the server now withholds the
+events, so they matter only if a future stream change regresses:
+
+- **Reasoning trace:** `renderTrace` returns `null` when `canViewSource` is
+  false — no thinking steps, no tool-call rows.
+- **Sources panel:** the citation side-panel (`activeSource`) is not rendered
+  when `canViewSource` is false. `SourceCard` also keeps its own guard, skipping
+  the `readSource` fetch and showing a restricted note.
+- **Role editor (`Users.tsx`):** `CAPABILITY_OPTIONS` gains `source:view` so
+  admins can grant it.
+
+## Migration / behavior change
+
+Default-deny means existing non-admin roles lose the thinking panel and fragment
+previews on upgrade until an admin grants `source:view`. This is the intended
+protection posture (confirmed). No data migration is required — roles that
+predate this cap simply don't hold it. Admin is unaffected.
+
+## Testing (TDD)
+
+Backend, mirroring existing auth/graph/server tests:
+
+- `expand_capabilities` includes `source:view` when granted; unknown/absent
+  behaves as before.
+- `GET /api/graph/source` → 403 for a role without `source:view`; 200 for a role
+  with it; 200 for admin.
+- Chat stream: `thinking` events are suppressed for a role without the cap and
+  present for a role with it (and for admin). Assert other event types
+  (`token`, `source`, `done`) are unaffected in both cases.
+- Sanity: a role with `source:view` but the endpoint otherwise scoped still
+  respects role `entry_keys` scope (no scope bypass introduced).
+
+Frontend: follow existing `web/src` test conventions if present; otherwise a
+manual check that thinking rows and fragment previews are hidden without the cap
+and shown with it.
+
+## Out of scope
+
+- Any change to `read_source` tool access or what the model reads into context.
+- Aggregate output budgets, streaming caps, redaction, per-repo strictness.
+- Reworking how citations are represented for privileged users (unchanged);
+  restricted users simply receive no citation events.
+
+## Touched files
+
+- `src/tpk/auth.py` — new capability constant + `ALL_CAPABILITIES`.
+- `src/tpk/agent.py` — system-prompt source-protection rule.
+- `src/tpk/server.py` — compute `can_view_source`, gate `thinking` events.
+- `src/tpk/graph_api.py` — require `source:view` on `/api/graph/source`.
+- `web/src/capabilities.ts` — new cap + `CAPABILITY_OPTIONS`.
+- `web/src/Chat.tsx` — gate thinking rows + `SourceCard` preview/click.
+- `web/src/Users.tsx` — cap appears in role editor (via `CAPABILITY_OPTIONS`).
+- Tests alongside the backend modules above.

src/tpk/agent.py

@@ -145,7 +145,14 @@ def system_prompt(repos) -> str:
 6. Your last message MUST be a normal assistant reply containing the
    answer text itself — never end the conversation on a tool call or with
    an empty message, and never leave the answer only in your private
-   reasoning."""
+   reasoning.
+7. PROTECT SOURCE CODE. Read and search the code freely to ground your
+   answer, and quote only the SHORT snippets needed to explain a point —
+   but never reproduce complete or near-complete files, and never
+   reconstruct a whole file across several quotes. If the user asks you to
+   print, dump, export, or output the full contents of a file, decline and
+   offer to explain what it does or show the specific lines relevant to
+   their question instead."""
 
 
 def build_agent(

src/tpk/auth.py

@@ -29,12 +29,14 @@
 CAP_CORPUS_MANAGE = "corpus:manage"
 CAP_USERS_VIEW = "users:view"
 CAP_USERS_MANAGE = "users:manage"
+CAP_SOURCE_VIEW = "source:view"
 
 # Order is the canonical UI/display order.
 ALL_CAPABILITIES = [
     CAP_CHAT, CAP_EXPLORE,
     CAP_CORPUS_VIEW, CAP_CORPUS_MANAGE,
     CAP_USERS_VIEW, CAP_USERS_MANAGE,
+    CAP_SOURCE_VIEW,
 ]
 _CAP_SET = frozenset(ALL_CAPABILITIES)
 # `:manage` grants its `:view` sibling for free.

src/tpk/graph_api.py

@@ -85,7 +85,7 @@ async def neighbors(id: str, direction: str = "both", depth: int = 1,
     @router.get("/source")
     async def source(repo: str, file_path: str,
                      line_start: int = Query(..., ge=1), line_end: int = Query(..., ge=1),
-                     user=Depends(auth.require_cap(auth_mod.CAP_EXPLORE))):
+                     user=Depends(auth.require_cap(auth_mod.CAP_SOURCE_VIEW))):
         try:
             lines = await _scoped(user, lambda: kg.read_source(repo, file_path, line_start, line_end))
         except ValueError as exc:

src/tpk/server.py

@@ -264,6 +264,7 @@ async def chat(req: ChatRequest, user: User = Depends(auth.require_cap(auth_mod.
             audit_provider, audit_model = "", ""
 
         scope = None
+        role = None
         turn_limit = 0  # effective daily token budget for this user (0 = unlimited)
         if user.role != auth_mod.ROLE_ADMIN:
             try:
@@ -305,6 +306,12 @@ async def chat(req: ChatRequest, user: User = Depends(auth.require_cap(auth_mod.
                         },
                     )
 
+        # Whether this user may see the reasoning trace + source references.
+        # Single source of truth: `effective_capabilities` already encodes
+        # admin -> all, missing/unreadable role -> none (fail closed), else the
+        # view-expanded role caps — the same helper the API/UI gates use.
+        can_view_source = auth_mod.CAP_SOURCE_VIEW in auth_mod.effective_capabilities(user, role)
+
         async def stream():
             # Set inside stream(), not the handler body: the generator runs
             # after `chat` returns (StreamingResponse drives it lazily), so
@@ -337,12 +344,21 @@ async def stream():
                             # with tool calls in event order — the UI groups
                             # consecutive deltas into a thinking step.
                             thinking = _chunk_thinking(chunk)
-                            if thinking:
+                            if thinking and can_view_source:
                                 yield _sse({"type": "thinking", "text": thinking})
                             text = _chunk_text(chunk)
                             if text:
                                 full.append(text)
-                                yield _sse({"type": "token", "text": text})
+                                # Token deltas interleave the model's between-tool
+                                # narration (reasoning) with answer text and can't
+                                # be told apart mid-stream. For users without
+                                # source:view the stream is answer-only: withhold
+                                # token deltas entirely and deliver the answer in
+                                # the `done` event (built from `full`/final_text),
+                                # so no narration crosses the wire or lingers in
+                                # the client buffer to surface on an error.
+                                if can_view_source:
+                                    yield _sse({"type": "token", "text": text})
                         elif kind == "on_chat_model_end":
                             # The last model turn's message is the authoritative
                             # answer — token deltas can miss it entirely for
@@ -355,13 +371,19 @@ async def stream():
                             # (the tool loop makes several) for the daily budget.
                             turn_tokens += _usage_tokens(output)
                         elif kind == "on_tool_start":
-                            yield _sse(
-                                {
-                                    "type": "tool",
-                                    "name": event.get("name", ""),
-                                    "input": event.get("data", {}).get("input", {}),
-                                }
-                            )
+                            # Trace/source events are withheld from users without
+                            # source:view — the API stream carries only the answer
+                            # (token/done/error). Reference metadata (tool names,
+                            # file paths, line ranges) must not cross the wire, not
+                            # just be hidden client-side.
+                            if can_view_source:
+                                yield _sse(
+                                    {
+                                        "type": "tool",
+                                        "name": event.get("name", ""),
+                                        "input": event.get("data", {}).get("input", {}),
+                                    }
+                                )
                         elif kind == "on_tool_end":
                             name = event.get("name", "")
                             args = event.get("data", {}).get("input") or {}
@@ -389,9 +411,12 @@ async def stream():
                                     if isinstance(items, list):
                                         audit_entry["count"] = len(items)
                                         audit_entry["unit"] = "matches"
-                                        yield _sse({"type": "tool_result", "name": name,
-                                                    "input": args, "count": len(items),
-                                                    "unit": "matches"})
+                                        # audit count is recorded above regardless;
+                                        # only the client event is cap-gated.
+                                        if can_view_source:
+                                            yield _sse({"type": "tool_result", "name": name,
+                                                        "input": args, "count": len(items),
+                                                        "unit": "matches"})
                                 except Exception:
                                     logger.exception("failed to emit search tool_result; skipping")
                             # Surface cited sources for the read_source rows.
@@ -416,8 +441,12 @@ async def stream():
                                             "line_end": line_end,
                                             "kind": "file",
                                         }
+                                        # `sources` is accumulated for the audit
+                                        # record below regardless; the client event
+                                        # (the clickable citation) is cap-gated.
                                         sources.append(source)
-                                        yield _sse(source)
+                                        if can_view_source:
+                                            yield _sse(source)
                             except Exception:
                                 logger.exception(
                                     "failed to process on_tool_end source event; skipping"
@@ -428,7 +457,10 @@ async def stream():
                         "or rephrase."
                     )
                     answer_text = done_text
-                    yield _sse({"type": "done", "text": done_text, "sources": sources})
+                    # Withhold the citation list from users without source:view —
+                    # the done event carries the answer only.
+                    yield _sse({"type": "done", "text": done_text,
+                                "sources": sources if can_view_source else []})
                 except Exception as exc:  # stream errors must reach the client
                     # Log the full exception server-side; the client only gets
                     # the exception's class name, never the raw message, which

tests/test_agent.py

@@ -47,6 +47,13 @@ def test_system_prompt_contains_corpus_and_citation_rules():
     assert "neighbors" in p and "path_between" in p
 
 
+def test_system_prompt_forbids_dumping_complete_source():
+    p = system_prompt(REPOS)
+    # The model may explain and quote minimally, but must not reproduce whole files.
+    assert "never reproduce complete" in p
+    assert "decline" in p
+
+
 def test_recursion_limit_constant():
     assert RECURSION_LIMIT == 40
 

tests/test_capabilities.py

@@ -10,8 +10,8 @@
 from tpk import auth
 from tpk.auth import (
     ALL_CAPABILITIES, CAP_CHAT, CAP_CORPUS_MANAGE, CAP_CORPUS_VIEW,
-    CAP_EXPLORE, CAP_USERS_MANAGE, CAP_USERS_VIEW, Role, User,
-    effective_capabilities, expand_capabilities,
+    CAP_EXPLORE, CAP_SOURCE_VIEW, CAP_USERS_MANAGE, CAP_USERS_VIEW,
+    DEFAULT_CAPABILITIES, Role, User, effective_capabilities, expand_capabilities,
 )
 from tpk.server import create_app
 
@@ -47,6 +47,18 @@ def test_effective_capabilities_admin_and_scoped():
     assert effective_capabilities(scoped, None) == set()
 
 
+def test_source_view_capability_registered_and_default_deny():
+    assert CAP_SOURCE_VIEW == "source:view"
+    assert CAP_SOURCE_VIEW in ALL_CAPABILITIES
+    # Default-deny: brand-new roles must not silently hold it.
+    assert CAP_SOURCE_VIEW not in DEFAULT_CAPABILITIES
+    # It is a bare grant (no :manage sibling), so expand is identity.
+    assert expand_capabilities([CAP_SOURCE_VIEW]) == {CAP_SOURCE_VIEW}
+    # Admin holds every capability, including this one.
+    admin = User("a", "", auth.ROLE_ADMIN)
+    assert CAP_SOURCE_VIEW in effective_capabilities(admin, None)
+
+
 # -- integration fixtures --------------------------------------------------
 
 pytestmark = requires_timeplus