From e3bc340a597503949d8e21ace90470dea591f74e Mon Sep 17 00:00:00 2001 From: Alexandre Daubois Date: Wed, 28 Oct 2026 09:00:20 +0100 Subject: [PATCH] Rewrite the documentation for readability --- README.md | 40 +++++++++++++----------- docs/configuration.md | 73 +++++++++++++++++++++++++++++-------------- docs/development.md | 48 ++++++++++++++++++---------- docs/how-it-works.md | 60 +++++++++++++++++++++++------------ docs/security.md | 73 +++++++++++++++++++++++++++++-------------- docs/setup.md | 39 +++++++++++++---------- 6 files changed, 215 insertions(+), 118 deletions(-) diff --git a/README.md b/README.md index 64da83b..11b15f5 100644 --- a/README.md +++ b/README.md @@ -2,24 +2,28 @@ [![CI](https://github.com/php/tamis/actions/workflows/ci.yml/badge.svg)](https://github.com/php/tamis/actions/workflows/ci.yml) -First-pass triage of private vulnerability reports on GitHub. When a new security advisory comes in, tamis has an LLM agent investigate it against the repository and posts its findings as a comment on the advisory. +tamis does a first-pass triage of private vulnerability reports on GitHub. When a new security advisory arrives, an LLM agent investigates it against the repository and tamis posts the findings as a comment on the advisory. For each report, tamis: -- lists public duplicates in the comment, and embargoed advisories only when they are exact duplicates, by identifier, -- finds public issues and pull requests already fixing or hardening the same code, +- lists public duplicates, and names an embargoed advisory only when it is an exact duplicate, +- finds public issues and pull requests that already fix or harden the same code, - checks that the files and functions the report cites exist, - reads the security policy and flags what may exclude the report, - suggests a CWE and a CVSS 4.0 base vector. -It informs the maintainers' decision and never makes it. +The maintainers keep the decision. tamis only prepares it. -The agent runs in an isolated container driven by [harness](https://github.com/alpha-omega-security/harness), the Alpha-Omega library behind [scrutineer](https://github.com/alpha-omega-security/scrutineer). [Security](docs/security.md) details what that sandbox allows. +The agent runs in an isolated container driven by [harness](https://github.com/alpha-omega-security/harness), the Alpha-Omega library behind [scrutineer](https://github.com/alpha-omega-security/scrutineer). [Security](docs/security.md) describes what the sandbox allows. + +## Before installing + +GitHub cannot post confidential advisory comments yet, so the reporter reads the comment. tamis also sends the report to the model provider you choose. Check that your disclosure policy allows both. ## Quick start -1. Create a GitHub App for your organisation with the "Repository security advisories" permission in read and write, and install it on the project repository only. -2. In a private repository, store the App's client ID and private key, and an Anthropic or OpenAI API key, or a [Codex access token](docs/configuration.md#codex-access-token). +1. Create a GitHub App for your organisation with the "Repository security advisories" permission in read and write. Install it on the project repository only. +2. In a private repository, store the App's client ID and private key, and an Anthropic or OpenAI API key or a [Codex access token](docs/configuration.md#codex-access-token). 3. Add this workflow to the private repository: ```yaml @@ -57,7 +61,7 @@ jobs: repository: your-org/your-project fetch-depth: 0 persist-credentials: false - # Replace this placeholder with the full SHA of a reviewed Tamis commit. + # Replace this placeholder with the full SHA of a reviewed tamis commit. - uses: php/tamis@REPLACE_WITH_REVIEWED_FULL_COMMIT_SHA with: repository: your-org/your-project @@ -67,23 +71,21 @@ jobs: api-key: ${{ secrets.ANTHROPIC_API_KEY }} github-token: ${{ steps.app.outputs.token }} bot-login: your-app-slug[bot] - # List every branch your project supports; these refs must exist in the checkout. + # List every branch your project supports. Each ref must exist in the checkout. references: master origin/release-branch ``` -Set `since` to the installation date so existing advisories are left alone. Keep the workflow repository private: the action uploads reports there as artifacts, retained for seven days. The [setup guide](docs/setup.md) covers source branches and provider spending limits. +Set `since` to the installation date so tamis leaves existing advisories alone. Keep the workflow repository private, since the action uploads the reports there as artifacts kept for seven days. The [setup guide](docs/setup.md) covers source branches and provider spending limits. ## Documentation -- [Setup](docs/setup.md): GitHub App, private repository, secrets and first run. -- [Configuration](docs/configuration.md): inputs, project guidance and engines. -- [How it works](docs/how-it-works.md): the triage pipeline and the comment it posts. -- [Security](docs/security.md): threat model, isolation and known limits. -- [Development](docs/development.md): building, testing and running locally. - -## Before installing - -The reporter can read the comment, since the GitHub API cannot post confidential comments yet, and the report is sent to the model provider you choose. Make sure your disclosure policy allows both. +| Page | Contents | +|---|---| +| [Setup](docs/setup.md) | GitHub App, private repository, secrets and first run | +| [Configuration](docs/configuration.md) | Inputs, budgets, project guidance and engines | +| [How it works](docs/how-it-works.md) | The triage pipeline and the comment it posts | +| [Security](docs/security.md) | Threat model, isolation and known limits | +| [Development](docs/development.md) | Building, testing and running locally | ## License diff --git a/docs/configuration.md b/docs/configuration.md index e4ce79d..a0587ec 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -4,49 +4,74 @@ | Input | Default | Description | |---|---|---| -| `api-key` | | API key of the model provider, or a Codex access token with `openai`. Required. | -| `github-token` | | Token able to read and comment the repository advisories. Required. | -| `bot-login` | owner of the token | Login the comments are posted as, e.g. `my-app[bot]`. Required with a GitHub App token. | -| `repository` | current repository | Repository owning the advisories. | -| `path` | `.` | Path of the repository checkout, relative to the workspace. | -| `since` | | Only advisories created after this date (`2026-10-22` or RFC 3339) are triaged. Required unless `advisory` is set. | +| `api-key` | | Required. API key of the model provider, or a Codex access token with `engine: openai`. | +| `github-token` | | Required. Token able to read and comment the repository advisories. | +| `references` | | Required, at most ten. Whitespace-separated local git references, one per supported branch. tamis analyses the first and locates the cited code on all of them. A missing reference fails the analysis. | +| `since` | | Triage only advisories created after this date, `2026-10-22` or RFC 3339. Required unless `advisory` is set. | | `advisory` | | Triage this GHSA only, whatever its state and creation date. | -| `engine` | `claude` | `claude` (Claude Code and the Anthropic API) or `openai` (Codex and the OpenAI API). | -| `model` | `claude-opus-5-5` / `gpt-5.5` | Model used by the reader, duplicate search and exact confirmation. `gpt-6.1-sol` with a Codex access token. | -| `daybreak` | `false` | Turn on OpenAI's Daybreak cyber access program in Codex. `openai` engine only, and the API key's organisation must be approved for it. | -| `runner-image` | tamis agent image, pinned by digest | Image the agent runs in. It must contain the agent CLI, git, curl and `harness-proxy`. A custom image skips the attestation check. | -| `references` | | Whitespace-separated local git references, one for each supported branch. The first is analysed, the code the report cites is located on all of them. Required, at most ten. A missing reference fails the analysis. | +| `bot-login` | Owner of the token | Login tamis posts as, such as `my-app[bot]`. Required with a GitHub App token. | +| `repository` | Current repository | Repository owning the advisories. | +| `path` | `.` | Path of the repository checkout, relative to the workspace. | +| `engine` | `claude` | `claude` for Claude Code and the Anthropic API, `openai` for Codex and the OpenAI API. | +| `model` | `claude-opus-5-5` or `gpt-5.5` | Model of the reader, the duplicate search and the exact confirmation. `gpt-6.1-sol` with a Codex access token. | +| `daybreak` | `false` | Turns on OpenAI's Daybreak cyber access program in Codex. Needs `engine: openai` and an API key whose organisation OpenAI approved for it. | +| `runner-image` | tamis agent image, pinned by digest | Image the agent runs in. It must contain the agent CLI, git, curl and `harness-proxy`. The action skips the attestation check for a custom image. | -Outside GitHub Actions, the same settings are read from environment variables: `LLM_API_KEY`, `GH_TOKEN`, `BOT_LOGIN`, `REPOSITORY`, `REPO_PATH`, `SINCE`, `ADVISORY`, `ENGINE`, `MODEL`, `DAYBREAK`, `RUNNER_IMAGE` and `REFERENCES`. `OUTPUT_DIR` must name a private directory for the reports. The action creates that directory in the runner's temporary storage and uploads it to the private workflow repository with seven-day retention. +The action refuses to run from a public workflow repository or from a `uses:` that is not a full commit SHA. Update that SHA through a reviewed pull request. -The action requires a private workflow repository and a full commit SHA in `uses:`. Update that SHA through a reviewed pull request. +Outside GitHub Actions, tamis reads the same settings from environment variables: `LLM_API_KEY`, `GH_TOKEN`, `BOT_LOGIN`, `REPOSITORY`, `REPO_PATH`, `SINCE`, `ADVISORY`, `ENGINE`, `MODEL`, `DAYBREAK`, `RUNNER_IMAGE` and `REFERENCES`. `OUTPUT_DIR` must name a private directory for the reports. The action creates it in the runner's temporary storage and uploads it to the workflow repository with a seven-day retention. ## Source branches -Fetch the configured references with `fetch-depth: 0`. For php-src, include the default branch and every PHP branch covered by your disclosure policy, for example `master origin/PHP-8.4` when those are the branches you support. Keep the list current as releases enter or leave support. Tamis analyses each reference separately and includes its exact commit in both copies of the result; it does not infer the maintained branches from a reporter's prose. No source code is executed. +Fetch the references with `fetch-depth: 0`. For php-src, list the default branch and each PHP branch your disclosure policy covers, such as `master origin/PHP-8.4`, and update the list as releases enter or leave support. + +tamis analyses each reference on its own and records its exact commit in both copies of the result. It does not guess the maintained branches from the reporter's prose, and it never executes the source code. ## Private reports -The private artifact contains a full `.md` report with every duplicate proposed by the global search and a `.comment.md` copy of the reporter-visible text, limited to published duplicates and separately confirmed exact unpublished duplicates. The full report also notes a duplicate search or confirmation that could not complete. Each copy names the source commit and input revision it covers. When an agent run fails, an `.agent-error.txt` file keeps the agent's error output and the error events of its JSON output, which may quote the report. +The private artifact holds, per advisory: + +| File | Contents | +|---|---| +| `.md` | The full report, with every duplicate the global search proposed and any duplicate search or confirmation that could not complete | +| `.comment.md` | The text the reporter sees, limited to published duplicates and to the unpublished exact duplicates that passed the [one-candidate confirmation](how-it-works.md#3-duplicate-search) | +| `.agent-error.txt` | Written when an agent run fails. It keeps the agent's error output and the error events of its JSON output, which may quote the report. | + +Each report names the source commit and the input revision it covers. ## Resource budgets -An execution permits at most 800 provider requests and 256 MiB of cumulative request bodies. Each advisory shares 200 requests and 64 MiB across its reader run and comparisons; each agent token additionally permits 150 requests, not counting the refused routes Codex calls on every run. A request body cannot exceed 1 MiB. API responses request at most 16,000 output tokens; the ChatGPT backend has its own output limits. The thirty-minute advisory deadline also cancels host-side git commands. At most ten unpublished exact candidates are separately confirmed. At most one report per known reporter is attempted per execution, and a report is only started while the execution still has a whole advisory budget and thirty minutes left of its 3 hours 50 minutes, otherwise it waits for the next execution. SDK retries are disabled, and an agent run that fails without a valid report is retried once after a minute. +| Scope | Limit | +|---|---| +| Execution | 800 provider requests and 256 MiB of request bodies in total, over at most 3 hours 50 minutes | +| Advisory | 200 requests and 64 MiB, shared by its reader run and its comparisons, and a 30-minute deadline that also cancels host-side git commands | +| Agent token | 150 requests, not counting the refused routes Codex calls on each run | +| Request | 1 MiB of body. API responses ask for at most 16,000 output tokens, and the ChatGPT backend applies its own output limits. | +| Exact confirmations | 10 unpublished candidates per advisory | +| Reports | One per known reporter and per execution | + +tamis starts a report only while the execution still holds a whole advisory budget and thirty minutes of its time. Otherwise the report waits for the next execution. SDK retries are off, but tamis retries an agent run that fails without a valid report once, after a minute. When a budget runs out, the report stays retriable and tamis does not mark it complete. + +These bounds cap one execution, not the spending across scheduled runs. Provider pricing, tokenisation and ChatGPT credit policies differ, so request and byte limits give no monetary guarantee. On the provider side: + +- use a dedicated account or project with an enforced spending or credit ceiling, +- restrict the models, and disable hosted tools and network allowlists, +- configure billing alerts. -These bounds limit each execution, not spending across scheduled runs. Use a dedicated provider account/project with an enforced spending or credit ceiling, restrict models, disable hosted tools and network allowlists there, and configure billing alerts. Provider pricing, tokenisation and ChatGPT credit policies differ, so byte/request limits are not a monetary guarantee. Keep a workflow `timeout-minutes` of at least 240 and a single concurrency group. Budget exhaustion leaves a report retriable rather than marking it permanently complete. +In the workflow, keep a `timeout-minutes` of at least 240 and a single concurrency group. ## Project guidance -A `.github/tamis.md` file in the project is appended to the agent's instructions. Use it for what the model cannot guess: what is in scope, what the project's public API is, how the project scores CVSS, which configurations are realistic. +tamis appends a `.github/tamis.md` file from the project to the agent's instructions. Use it for what the model cannot guess, such as the scope, the public API, how the project scores CVSS and which configurations are realistic. ## Security policy -The agent checks the report against the files that describe what the project treats as a security issue. tamis looks for them among the tracked files of the repository root, `.github/` and `docs/`: +The agent checks the report against the files that define what the project treats as a security issue. tamis looks for them among the tracked files of the repository root, `.github/` and `docs/`: -- `SECURITY` with a `.md`, `.markdown`, `.rst`, `.txt` or `.adoc` extension, in any case, -- then `AGENTS.md` and `CLAUDE.md`. +1. `SECURITY` with a `.md`, `.markdown`, `.rst`, `.txt` or `.adoc` extension, in any case, +2. then `AGENTS.md` and `CLAUDE.md`. -The comment lists the files it was checked against. Without any of them, the section is left out. +The comment lists the files the agent checked. Without any of them, tamis leaves the section out. ## Engines @@ -56,8 +81,8 @@ The comment lists the files it was checked against. Without any of them, the sec | `openai` | Codex | OpenAI Chat Completions | `api.openai.com` | | `openai` with a Codex access token | Codex | Codex, in a container without GitHub access | `chatgpt.com` | -The agent CLI versions are pinned in [`docker/agent/Dockerfile`](../docker/agent/Dockerfile). +[`docker/agent/Dockerfile`](../docker/agent/Dockerfile) pins the agent CLI versions. ## Codex access token -A ChatGPT workspace without OpenAI API access can create a [Codex access token](https://learn.chatgpt.com/docs/enterprise/access-tokens) in its admin console and pass it as `api-key` with `engine: openai`. It is recognised by its `at-` prefix. The ChatGPT backend does not serve the API models, so the default model becomes `gpt-6.1-sol`. Prefer a token of a service account, with the shortest expiration you can rotate. +A ChatGPT workspace without OpenAI API access can create a [Codex access token](https://learn.chatgpt.com/docs/enterprise/access-tokens) in its admin console and pass it as `api-key` with `engine: openai`. tamis recognises it by its `at-` prefix. The ChatGPT backend does not serve the API models, so the default model becomes `gpt-6.1-sol`. Prefer the token of a service account, with the shortest expiration you can rotate. diff --git a/docs/development.md b/docs/development.md index ebde244..511c2c3 100644 --- a/docs/development.md +++ b/docs/development.md @@ -1,53 +1,67 @@ # Development -Requirements: Go 1.27.3 and Docker. +You need Go 1.27.3 and Docker. ## Build and test ```bash go build ./cmd/tamis -``` - -```bash go test -race ./... ``` -Four fuzz targets cover the comment escaper, duplicate-search clues, file reading and the provider protocol filter: +Four fuzz targets cover the comment escaper, the duplicate-search clues, file reading and the provider protocol filter: + +| Target | Package | +|---|---| +| `FuzzText` | `./internal/report` | +| `FuzzClues` | `./internal/report` | +| `FuzzReadFileStaysInTrackedFiles` | `./internal/repo` | +| `FuzzProviderRequest` | `./internal/modelproxy` | ```bash go test ./internal/report -run '^$' -fuzz '^FuzzText$' -fuzztime 60s ``` -The others are `FuzzClues` in `./internal/report`, `FuzzReadFileStaysInTrackedFiles` in `./internal/repo`, and `FuzzProviderRequest` in `./internal/modelproxy`. CI runs formatting, `go mod tidy`, `go vet`, the tests, the four fuzz targets and Docker isolation tests for both agent images on every push. Make both isolation jobs required release checks. +On each push, CI checks formatting and `go mod tidy`, then runs `go vet`, the tests, the four fuzz targets and the Docker isolation tests of both agent images. Make both isolation jobs required release checks. -Build each agent image and run its isolation tests locally without a real API key: +## Isolation tests + +Build an agent image and run its isolation tests locally, without a real API key: ```bash docker build --target codex -t tamis-test:codex docker/agent TAMIS_TEST_IMAGE=tamis-test:codex go test -race -count=1 -tags integration ./internal/sandbox ``` -Repeat for `claude`. The tests use fake model servers, check direct host/Internet/DNS access and validate that the pinned CLI can reach the filtered proxy. Missing Docker or an unset test image fails the suite. +Repeat with `claude`. The tests use fake model servers, check direct host, Internet and DNS access, and confirm the pinned CLI reaches the filtered proxy. The suite fails without Docker or with `TAMIS_TEST_IMAGE` unset. ## Agent images -The agent images are built from [`docker/agent/Dockerfile`](../docker/agent/Dockerfile), one target per agent: +[`docker/agent/Dockerfile`](../docker/agent/Dockerfile) builds the agent images, one target per agent: ```bash docker build --target claude -t tamis-agent:claude docker/agent ``` -The `Agent image` workflow publishes both to `ghcr.io/php/tamis-agent` with a build attestation when that directory changes on `main`, then opens a pull request updating the digests in [`internal/sandbox/images.json`](../internal/sandbox/images.json). tamis only runs the images pinned there, so new images reach users once that pull request is merged and released. +When that directory changes on `main`, the `Agent image` workflow publishes both images to `ghcr.io/php/tamis-agent` with a build attestation. It then opens a pull request that updates the digests in [`internal/sandbox/images.json`](../internal/sandbox/images.json). tamis only runs the images pinned there, so new images reach users once that pull request lands in a release. To bump an agent CLI, update its tag and the sha256 of each architecture's asset together in the Dockerfile. The workflow needs "Allow GitHub Actions to create and approve pull requests" in the repository settings. ## Releases -Pushing a `vYYYY.MM.DD.N` tag, such as `v2026.10.25.1`, runs the `Release` workflow, which builds tamis for Linux on amd64 and arm64, attests both binaries and publishes them in a GitHub release. When the action's pinned commit has such a release tag, the action downloads the binary for the runner and checks its attestation against that commit before running it. Any other pinned commit is built from source. +A `vYYYY.MM.DD.N` tag such as `v2026.10.25.1` runs the `Release` workflow. It builds tamis for Linux on amd64 and arm64, attests both binaries and publishes them in a GitHub release. + +When the action's pinned commit carries such a tag, the action downloads the binary for the runner and checks its attestation against that commit before running it. For any other commit, the action builds tamis from source. ## Dependency updates -[Renovate](https://docs.renovatebot.com/) keeps Go modules, pinned actions, base images and the agent CLIs up to date, following [`renovate.json`](../renovate.json). It updates each CLI's tag and checksums in one pull request, keeps `harness-proxy` in the agent image on the same version as the `harness` module, and waits seven days after a release before proposing it. It needs the [Renovate GitHub App](https://github.com/apps/renovate) installed on the repository. +[Renovate](https://docs.renovatebot.com/) keeps the Go modules, pinned actions, base images and agent CLIs up to date, following [`renovate.json`](../renovate.json). It: + +- updates each CLI's tag and checksums in one pull request, +- keeps `harness-proxy` in the agent image on the same version as the `harness` module, +- waits seven days after a release before proposing it. + +It needs the [Renovate GitHub App](https://github.com/apps/renovate) installed on the repository. ## Running locally @@ -57,7 +71,7 @@ tamis reads its configuration from environment variables, so it runs on any mach REPOSITORY=your-org/your-project REPO_PATH=. ADVISORY=GHSA-xxxx-xxxx-xxxx ENGINE=claude GH_TOKEN="$(gh auth token)" LLM_API_KEY="$ANTHROPIC_API_KEY" RUNNER_IMAGE=tamis-agent:claude REFERENCES=HEAD OUTPUT_DIR=/private/path/to/reports ./tamis ``` -The comment is posted for real. An advisory with a completed comment from the configured bot is skipped, including after edits to the report or followup comments. +tamis posts the comment for real. It skips an advisory that already has a completed comment from the configured bot, even after edits to the report or new followup comments. ## Layout @@ -66,9 +80,11 @@ The comment is posted for real. An advisory with a completed comment from the co | `cmd/tamis` | Configuration and entry point | | `internal/triage` | Advisory selection and the triage pipeline | | `internal/sandbox` | The agent container, through harness | -| `internal/modelproxy` | The proxy keeping the model key on the host | -| `internal/engine` | The in-process model loop of exact duplicate confirmation | -| `internal/tools` | The exact duplicate confirmation's read-only tools | +| `internal/modelproxy` | The proxy that keeps the model key on the host | +| `internal/budget` | Request and byte budgets shared by every agent and comparator of a run | +| `internal/usage` | Requests and tokens spent per advisory, for the log | +| `internal/engine` | The in-process model loop of the exact duplicate confirmation | +| `internal/tools` | The read-only tools of the exact duplicate confirmation | | `internal/report` | Report schema, verification and rendering | | `internal/repo` | Read-only access to tracked files | | `internal/github` | GitHub REST client | diff --git a/docs/how-it-works.md b/docs/how-it-works.md index 1a7ae6d..5102052 100644 --- a/docs/how-it-works.md +++ b/docs/how-it-works.md @@ -1,48 +1,68 @@ # How it works -Each run goes through four stages. Two models are involved, and no model ever sees both the report and the repository's other advisories. +Each run goes through four stages. Two models take part, and no model sees both the report and the repository's other advisories. ## 1. Selection -tamis lists the repository's advisories and keeps those in the `triage` state created after `since`, or only the one given in `advisory`. It skips an advisory when a comment from `bot-login` contains the Tamis marker. A marker posted by another account is ignored. +tamis lists the repository's advisories and keeps those in the `triage` state created after `since`, or only the one given in `advisory`. It skips an advisory when a comment from `bot-login` holds the tamis marker, and ignores a marker posted by any other account. -A failed triage posts a comment saying so, with its own marker, and is retried on the next runs. After three failures, the comment asks a maintainer to review the report and the advisory is left alone. An advisory is triaged once: editing the report afterwards does not start a new triage. +A failed triage posts a comment that says so, with its own marker, and tamis retries it on the next runs. After three failures, the comment asks a maintainer to review the report and tamis stops retrying. tamis triages an advisory once, so an edit to the report afterwards starts no new triage. -The log names the advisory being triaged and each step as it starts: copying the source, reading the report, comparing with other advisories, posting the comment. Each step and each triaged or failed advisory also logs its duration, the model requests and the input and output tokens it spent, with cached input tokens counted as input. +The log names the advisory and each step as it starts: copying the source, reading the report, comparing with other advisories and posting the comment. Each step, and each triaged or failed advisory, also logs its duration, its model requests and its input and output tokens. Cached input tokens count as input. ## 2. Reading the report An agent, Claude Code or Codex, investigates the report in a throwaway container: -- the report and its non-bot followup comments are in `/work/report.md`, and the selected source branch is mounted read-only in `/work/src` with its history, -- it can search public issues and pull requests through the GitHub API, +- `/work/report.md` holds the report and its non-bot followup comments, +- `/work/src` holds the selected source branch, read-only and with its history, +- the agent can search public issues and pull requests through the GitHub API, - it reads the security policy files when the repository has some, -- it writes its findings to `/work/report.json`, following a JSON schema, as soon as it has a first answer and then as it learns more. When the run stops on its budget or an error, a valid report already written is kept in the private artifact and the advisory is retried, and the last attempt posts it with a note that it may be incomplete. +- it writes its findings to `/work/report.json` against a JSON schema, first as soon as it has an answer, then as it learns more. -The JSON is decoded strictly: an unknown or mistyped field sends the error back to the agent for one correction round. The agent reports the faulty statement (file, function, line), the public entry points it goes through, the code the report cites, related issues, policy exclusions, CWE and CVSS. The agent analyses the first configured source reference. The code the report cites is then located on each other reference, and the final report names every exact commit. A missing branch fails closed. +The agent reports the faulty statement (file, function, line), the public entry points that reach it, the code the report cites, related issues, policy exclusions, the CWE and the CVSS. + +tamis rejects an unknown or mistyped field in the JSON and sends the error back to the agent for one correction round. + +When the run stops on its budget or on an error, tamis keeps a valid report already written in the private artifact and retries the advisory. The last attempt posts that report with a note that it may be incomplete. + +The agent analyses the first configured reference. tamis then locates the cited code on each other reference, and the final report names every exact commit. A missing branch fails the triage. ## 3. Duplicate search -One offline agent then searches every other advisory of the repository for the same bug, whatever their state: published, in triage, draft or closed, since a closed duplicate tells the maintainers the issue was already handled. It gets the report, the clues whose location exists in the checkout, and one file per advisory, which it greps and reads the way a maintainer would, and the code read-only for the cases the texts leave open. The report, the clues and the advisories remain hostile data. It answers with the identifiers of the duplicates it found, each with a confidence and an exact flag, and an identifier outside the advisories it was given is refused. +One offline agent searches every other advisory of the repository for the same bug, whatever its state: published, in triage, draft or closed. A closed duplicate tells the maintainers someone already handled the issue. The agent gets: + +- the report, +- the clues whose location exists in the checkout, +- one file per advisory, which it greps and reads the way a maintainer would, +- the code, read-only, for the cases the texts leave open. -The full search result is preserved in the private report. Each unpublished candidate called exact with high confidence is checked again in its own model context, which receives the report, that candidate without its identifier, the checked clues and the code. It returns one `exact` boolean. Only the host knows which candidate that answer belongs to and copies a confirmed identifier into the public comment. All models share the advisory and run budgets. The log records the confirmation duration, usage and number of confirmed candidates. +tamis treats the report, the clues and the advisories as hostile data. The agent answers with the identifiers of the duplicates it found, each with a confidence and an exact flag. tamis refuses an identifier outside the advisories it gave. -Both stages compare concrete defects, including secondary findings in a report. A different PoC, version, configuration or severity estimate does not turn the same defective operation into a different bug. An exact match requires positive evidence of a shared defect; a similar symptom or a broad patch fixing several independent defects is insufficient. An exact match does not settle any additional findings in either report, which maintainers must still review before closing it. +The private report keeps the full search result. tamis checks each unpublished candidate marked exact with high confidence again, in its own model context. That context receives the report, the candidate without its identifier, the checked clues and the code, and returns one `exact` boolean. Only the host knows which candidate the answer belongs to, and it copies a confirmed identifier into the public comment. All models share the advisory and run budgets. The log records the duration and usage of the confirmations and the number of confirmed candidates. -Matching is separate from judging validity: an issue already fixed or rejected under the security policy can still be a duplicate. A finding explicitly split into a separate advisory, however, no longer belongs to the current report's scope. +Both stages compare concrete defects, secondary findings of a report included: + +- a different PoC, version, configuration or severity estimate leaves the same defective operation the same bug, +- an exact match needs positive evidence of a shared defect, and a similar symptom or a broad patch fixing several independent defects does not qualify, +- an exact match does not settle the other findings of either report, which maintainers still review before closing it. + +Matching stays separate from validity. An issue already fixed, or rejected under the security policy, can still be a duplicate. A finding the report explicitly removes, excludes or moves to another advisory no longer counts as one of its findings. ## 4. Verification and comment -Before posting, tamis checks every claim against GitHub and the checkout: +Before posting, tamis checks each claim against GitHub and the checkout: | Claim | Check | |---|---| -| Duplicates | Must be one of the advisories the search was given. Its complete result is preserved privately. An unpublished advisory appears publicly, by identifier only, after its exact, high-confidence result passes a separate one-candidate confirmation. | -| Issues and pull requests | Must exist. Titles and links come from GitHub, not from the model. | -| Cited code | Only references named in the report are kept, each marked as found in the repository or not. A cited line and the code the report quotes there are kept only when the report contains them. The quote is looked up in the file at the scanned commit, nearest the cited line, and when it is gone, `git log -S` names the commit that last changed it, since reports often target another version. | -| CWE and CVSS | Must be well-formed. CWE names come from the MITRE catalogue. | +| Duplicates | Each one must belong to the advisories the search received. The private report keeps the complete result. An unpublished advisory appears in the comment, by identifier only, once its exact, high-confidence result passes the one-candidate confirmation. | +| Issues and pull requests | Each one must exist. Titles and links come from GitHub, not from the model. | +| Cited code | tamis keeps only references named in the report, each marked as found in the repository or not, and keeps a cited line and its quoted code only when the report contains them. It looks the quote up in the file at the scanned commit, nearest the cited line. When the quote is gone, `git log -S` names the commit that last changed it, since reports often target another version. | +| CWE and CVSS | Each must be well-formed. CWE names come from the MITRE catalogue. | + +tamis drops the claims that fail a check, and the comment counts them. -Claims that fail a check are dropped, and the comment counts them. Just before publication, the current report and non-bot comments are read again; if they changed, no result is posted. The public comment is saved before publication. A failed duplicate search leaves the duplicates out of the comment; a failed exact confirmation leaves that unpublished candidate out. Both are marked in the private report and do not invalidate the published triage. A failure before publication leaves the advisory retriable. +Just before publication, tamis reads the report and its non-bot comments again and posts nothing if they changed. It saves the public comment before posting it. A failed duplicate search leaves the duplicates out of the comment, and a failed exact confirmation leaves that unpublished candidate out. The private report records both failures, and the triage still gets published. A failure before publication leaves the advisory retriable. ## The comment @@ -81,6 +101,6 @@ Plausible signed `char` passed to `isxdigit()` in `php_url_decode_ex()`, the giv ``` -The duplicates and related issues stay visible. The cited code, the security policy and the weakness estimate are folded so the comment scans quickly. +The duplicates and related issues stay visible. The cited code, the security policy and the weakness estimate sit in a folded block so the comment reads fast. -Every piece of model text is escaped, so the comment cannot contain links, images, HTML or mentions coming from the model. The links to cited lines are built from the repository name, the commit hash and the tracked path, and the quoted code itself is never shown. +tamis escapes all model text, so the comment cannot carry links, images, HTML or mentions coming from the model. It builds the links to cited lines from the repository name, the commit hash and the tracked path, and never shows the quoted code. diff --git a/docs/security.md b/docs/security.md index 383d930..13aa6d4 100644 --- a/docs/security.md +++ b/docs/security.md @@ -1,49 +1,76 @@ # Security -tamis reads text written by an outside reporter, with access to embargoed advisories and an API key, and posts a comment the reporter can read. The design assumes the report is hostile. +tamis reads text from an outside reporter, holds access to embargoed advisories and an API key, and posts a comment the reporter can read. The design assumes a hostile report. ## Threats | Threat | Mitigation | |---|---| -| A prompt injection makes the agent leak other advisories | The duplicate search is one offline agent that reads the report next to every other advisory, embargoed and closed ones included, so it can search them the way a maintainer would. Its complete result is preserved privately; published matches remain reporter-visible. Each unpublished exact, high-confidence candidate is confirmed in a separate model context holding that candidate only; the host owns its identifier and publishes it only when this second model returns `exact: true`. A hostile report can still push confirmations to disclose candidate identifiers, but an instruction hidden in one candidate only reaches that candidate's confirmation. A failed search or confirmation leaves unconfirmed unpublished identifiers out of the comment. | -| The agent leaks secrets | The container holds no secret. The model key stays on the runner, the container receives a clean single-branch clone with an empty `.git/config`, and the GitHub token never enters it. | -| The agent exfiltrates data over the network | The container sits on an isolated internal network and only knows a private egress sidecar. The model proxy pins the model and rejects remote URLs, remote MCP servers, hosted tools, stored conversations and background calls. | -| The comment loads a remote image carrying data, or pings people | All model text is escaped: no links, images, HTML, entities or mentions. | -| A reporter posts the marker to block triage | Only comments from `bot-login` containing the marker are considered. | -| A report changes while being triaged | The title, description, metadata and non-bot comments are hashed. They are re-read before publication; a changed revision cancels the post. Completed advisories are not automatically triaged again after later edits. | -| A reporter consumes the model budget | Advisory and run budgets are shared across branches, agents and comparators. Requests, bytes, output tokens and runtime are bounded; one known reporter cannot monopolise a run. A dedicated provider spending ceiling bounds costs across runs. | -| A release tag is replaced | The action requires a full reviewed commit SHA. Agent images are additionally pinned by digest and verified against their build attestation. | -| Logs of a public repository expose the report | tamis logs the identifier of the advisory it works on, its steps, counters, status codes and the routes, JSON keys and `type`, `role` and `execution` values of requests the model proxy refuses. It never logs report text, the identifiers of other advisories or provider error bodies itself, but a hostile agent chooses the routes, keys and values of its requests, so it can write the report, or with a Codex access token the advisories it searches, into the log. GitHub also prints step inputs, so the action refuses to run from a public repository. | +| A prompt injection makes the agent leak other advisories | See [Duplicate search isolation](#duplicate-search-isolation). | +| The agent leaks secrets | The container holds no secret. The model key stays on the runner, the GitHub token never enters the container, and the container gets a clean single-branch clone with an empty `.git/config`. | +| The agent exfiltrates data over the network | The container sits on an isolated internal network and only reaches a private egress sidecar. The model proxy pins the model and rejects remote URLs, remote MCP servers, hosted tools, stored conversations and background calls. | +| The comment loads a remote image carrying data, or pings people | tamis escapes all model text: no links, images, HTML, entities or mentions. | +| A reporter posts the marker to block triage | tamis only considers markers in comments from `bot-login`. | +| A report changes during triage | tamis hashes the title, description, metadata and non-bot comments, reads them again before publication and cancels the post when the revision changed. It does not triage a completed advisory again after a later edit. | +| A reporter drains the model budget | Branches, agents and comparators share the advisory and run budgets. tamis bounds requests, bytes, output tokens and runtime, and one known reporter cannot take a whole run. A provider spending ceiling bounds costs across runs. | +| Someone replaces a release tag | The action requires a full reviewed commit SHA. tamis also pins agent images by digest and verifies them against their build attestation. | +| Logs of a public repository expose the report | The action refuses to run from a public repository. See [Logs](#logs). | + +## Duplicate search isolation + +The duplicate search is one offline agent that reads the report next to every other advisory, embargoed and closed ones included, so it can search them the way a maintainer would. The private report keeps its complete result, and the comment shows published matches to the reporter. + +A separate model context confirms each unpublished exact, high-confidence candidate, and holds that candidate only. The host owns the candidate's identifier and publishes it only when this second model returns `exact: true`. A failed search or confirmation keeps unconfirmed unpublished identifiers out of the comment. + +A hostile report can still push the confirmations to disclose candidate identifiers. An instruction hidden in one candidate only reaches that candidate's confirmation. + +## Logs + +tamis logs the identifier of the advisory it works on, its steps, counters and status codes. For requests the model proxy refuses, it logs the routes, the JSON keys and the `type`, `role` and `execution` values. + +tamis itself never logs report text, the identifiers of other advisories or provider error bodies. A hostile agent still chooses the routes, keys and values of its requests, so it can write the report into the log, or with a Codex access token the advisories it searches. GitHub also prints step inputs, which is why the action refuses a public repository. ## The agent container The agent runs through [harness](https://github.com/alpha-omega-security/harness), the isolation layer of [scrutineer](https://github.com/alpha-omega-security/scrutineer): - a throwaway container per advisory, with a read-only root filesystem, all capabilities dropped and no new privileges, -- one single-branch clone of the analysed branch per execution, mounted read-only into every agent run, containing no untracked files and an empty `.git/config`, while the other branches are worktrees that only tamis reads, -- an internal bridge network with both gateway modes set to `isolated` and IPv6 explicitly disabled, whose only peer is an authenticated egress proxy allowing the model proxy and `api.github.com`, -- the workspace mounted read-only except for `report.json`, with memory, CPU, process, file-size, output-capture and Docker-log limits, -- an image containing only the agent CLI, git, curl, ripgrep and `harness-proxy`, with every download checked against a pinned checksum, +- one single-branch clone of the analysed branch per execution, mounted read-only into each agent run, with no untracked files and an empty `.git/config`, while only tamis reads the other branches, checked out as worktrees, +- an internal bridge network with both gateway modes set to `isolated` and IPv6 disabled, whose only peer is an authenticated egress proxy that allows the model proxy and `api.github.com`, +- a read-only workspace except for `report.json`, with memory, CPU, process, file-size, output-capture and Docker log limits, +- an image with only the agent CLI, git, curl, ripgrep and `harness-proxy`, each download checked against a pinned checksum, - that image pinned by digest in each release, and checked by the action against its build attestation before use, like the tamis binary of a release. ## The model proxy -The model key never enters the container. A proxy on the runner gives each agent a random token revoked when the run ends. It only accepts `POST /v1/messages`, `POST /v1/messages/count_tokens` or `POST /v1/responses`, or with a Codex access token `POST /backend-api/codex/responses` and a `GET` of the token check, the workspace configuration and the model catalogue, whose only permitted query is `client_version`. It pins the configured model and charges a shared request/byte budget. Protocol fields, input item types and tool definitions are checked against allowlists, including tools under `additional_tools`, tool search results and namespaces. Only local tools and client-side tool search pass; hosted execution, remote retrieval, persisted conversation references and unknown protocol extensions fail closed. Provider capability headers from the agent are discarded. Stateless encrypted reasoning blocks are permitted, since the CLI resends them as conversation data. +The model key never enters the container. A proxy on the runner gives each agent a random token and revokes it when the run ends. + +The proxy accepts only these routes: + +- `POST /v1/messages`, `POST /v1/messages/count_tokens` and `POST /v1/responses`, +- with a Codex access token, `POST /backend-api/codex/responses` and a `GET` of the token check, the workspace configuration and the model catalogue, with `client_version` as the only permitted query. + +It pins the configured model and charges a shared request and byte budget. It checks protocol fields, input item types and tool definitions against allowlists, tools under `additional_tools`, tool search results and namespaces included. Only local tools and client-side tool search pass. Hosted execution, remote retrieval, persisted conversation references and unknown protocol extensions fail closed. The proxy discards provider capability headers from the agent. It lets stateless encrypted reasoning blocks through, since the CLI resends them as conversation data. + +A Codex access token stays on the runner the same way. Codex only sends it over HTTPS, so the proxy serves a certificate from a CA generated for the process and drops the CA key once it signs the certificate. The agent gets a fake `at-` token. The proxy: + +- accepts only the token check, the workspace configuration, which it answers as empty, and `POST /backend-api/codex/responses`, +- strips the email, user and workspace identifiers from the token check's answer, +- adds the real workspace identifier to upstream requests itself. -A Codex access token stays on the runner the same way. Codex only sends it over HTTPS, so the proxy serves a certificate from a CA generated for the process, whose key is dropped once the certificate is signed. The agent gets a fake `at-` token and the proxy only accepts the token check, the workspace configuration, which it answers as empty, and `POST /backend-api/codex/responses`. It strips the email, user and workspace identifiers from the token check's answer and adds the real workspace identifier to upstream requests itself. Codex's ChatGPT features that call other routes (plugins, apps, image generation, analytics) are turned off. +tamis turns off the Codex ChatGPT features that call other routes: plugins, apps, image generation and analytics. ## Known limits -- The reporter reads the comment. A crafted report can still steer the agent's own text, for example its summary or its policy check. -- Filtered paths and identifiers are still hostile text, not a prompt-injection defence. Their code location is checked before comparison; the output's privacy boundary does not depend on the model following instructions. -- Re-reading before publication narrows the update race but GitHub offers no conditional comment creation tied to an advisory revision. Completed advisories are not automatically revisited after a later edit. Results display the digest they cover. -- Request/byte limits bound one run, not the cost of an unlimited series of runs. Enforce a provider spending ceiling and a workflow timeout as described in the setup guide. -- The report is sent to the model provider. Check its data retention terms against your disclosure policy. -- Docker Engine 28 or newer is required for an internal network without a host gateway. +- The reporter reads the comment. A crafted report can still steer the agent's own text, such as its summary or its policy check. +- Filtered paths and identifiers remain hostile text and offer no prompt-injection defence. tamis checks their code location before comparison, and the privacy boundary of the output does not rely on the model following instructions. +- Reading the report again before publication narrows the update race, but GitHub offers no conditional comment creation tied to an advisory revision. tamis does not revisit a completed advisory after a later edit. Each result shows the digest it covers. +- Request and byte limits bound one run, not the cost of an endless series of runs. Enforce a provider spending ceiling and a workflow timeout as the [setup guide](setup.md#2-create-a-private-repository) describes. +- tamis sends the report to the model provider. Check its data retention terms against your disclosure policy. +- An internal network without a host gateway needs Docker Engine 28 or newer. - The advisory comments API is in public preview and may change. -CI builds both agent images and runs the Docker isolation tests on `ubuntu-latest`. Treat both `isolation` jobs as required checks before releasing or updating the deployment SHA. They test the model path, direct and proxied host access, DNS and IPv4/IPv6 egress without real provider credentials. +CI builds both agent images and runs the Docker isolation tests on `ubuntu-latest`. They test the model path, direct and proxied host access, DNS, and IPv4 and IPv6 egress, without real provider credentials. Treat both `isolation` jobs as required checks before releasing or updating the deployment SHA. ## Reporting a vulnerability diff --git a/docs/setup.md b/docs/setup.md index 82e77db..86490d1 100644 --- a/docs/setup.md +++ b/docs/setup.md @@ -1,49 +1,56 @@ # Setup -tamis runs as a scheduled GitHub Actions workflow. Installing it takes a GitHub App, a private repository and two API keys. +tamis runs as a scheduled GitHub Actions workflow. You need a GitHub App, a private repository and a model provider key. ## 1. Create the GitHub App -Open this address with `YOUR-ORG` replaced by your organisation name. The form is pre-filled with the only permission tamis needs, "Repository security advisories" in read and write, and no webhook. +Open this address with `YOUR-ORG` replaced by your organisation name. The form comes pre-filled with the only permission tamis needs, "Repository security advisories" in read and write, and no webhook. ```text https://github.com/organizations/YOUR-ORG/settings/apps/new?name=YOUR-ORG-tamis&description=First-pass%20triage%20of%20new%20security%20advisories&url=https://github.com/php/tamis&public=false&webhook_active=false&repository_advisories=write ``` -After creating it: +Once the App exists: -1. Note its client ID and its slug, shown in its public page address. +1. Note its client ID, and its slug from its public page address. 2. Generate a private key. 3. Install it on the project repository only, with "Only select repositories". -Comments are posted as `[bot]`, and tamis recognises its own comments by that login. +tamis posts as `[bot]` and recognises its own comments by that login. -A personal access token of a repository security manager works too, with the `repository_advisories:write` scope, in which case the comments are posted under that account and `bot-login` can be omitted. +A personal access token of a repository security manager with the `repository_advisories:write` scope also works. tamis then posts under that account, and you can omit `bot-login`. ## 2. Create a private repository -Run the workflow from a private repository, for example `your-org/tamis-runner`, rather than from the public project. Workflow logs, step inputs and run times of a public repository are public, and tamis handles embargoed reports. +Run the workflow from a private repository such as `your-org/tamis-runner`, not from the public project. A public repository exposes its workflow logs, step inputs and run times, and tamis handles embargoed reports. -In that repository, create an `tamis` environment and add: +In that repository, create a `tamis` environment and add: | Kind | Name | Value | |---|---|---| -| Variable | `TAMIS_CLIENT_ID` | the App client ID | -| Secret | `TAMIS_PRIVATE_KEY` | the App private key | -| Secret | `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` | the model provider key, or a [Codex access token](configuration.md#codex-access-token) | +| Variable | `TAMIS_CLIENT_ID` | The App client ID | +| Secret | `TAMIS_PRIVATE_KEY` | The App private key | +| Secret | `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` | The model provider key, or a [Codex access token](configuration.md#codex-access-token) | Restrict the environment to the default branch so other workflows cannot read its secrets. -Use a dedicated provider project/account with a hard spending or credit limit. Alerts alone do not stop charges. Approve the configured models and their retention/residency terms. See [resource budgets](configuration.md#resource-budgets) for the application limits. +Use a dedicated provider project or account with a hard spending or credit limit, because alerts alone do not stop charges. Approve the configured models and their retention and residency terms. [Resource budgets](configuration.md#resource-budgets) lists the limits tamis enforces itself. ## 3. Add the workflow -Copy the workflow from the [README](../README.md#quick-start) and replace `your-org`, `your-project`, `your-app-slug` and the Tamis commit placeholder. Pin the full SHA of a reviewed commit; the action refuses a branch or tag reference. Review updates in the private deployment repository. Set `since` to today's date: only advisories in the `triage` state created after it are processed. Set `references` to every branch covered by your security policy, the default branch first. A report is analysed on the first one, and the code it cites is located on each. +Copy the workflow from the [README](../README.md#quick-start), then: -Keep `persist-credentials: false` on the checkout, so no token is written to `.git/config`, and `fetch-depth: 0`, so the agent can read the history. +- replace `your-org`, `your-project`, `your-app-slug` and the commit placeholder, +- pin the full SHA of a reviewed commit, since the action refuses a branch or tag, and review later updates in the private repository, +- set `since` to today's date, so tamis only processes advisories in the `triage` state created after it, +- set `references` to every branch your security policy covers, default branch first. tamis analyses the report on the first one and locates the cited code on each. + +Keep `persist-credentials: false` on the checkout so no token lands in `.git/config`, and `fetch-depth: 0` so the agent can read the history. ## 4. First run -Trigger the workflow by hand with the `advisory` input set to a test GHSA. That advisory is triaged whatever its state and date, and the comment is posted on it. Download the private `tamis-reports` artifact and check that the comment names embargoed advisories only as exact duplicates, then let the schedule run. Artifact access follows the private runner repository's permissions; grant access only to the security team. +Trigger the workflow by hand with the `advisory` input set to a test GHSA. tamis triages that advisory whatever its state and date, and posts the comment on it. + +Download the private `tamis-reports` artifact and check that the comment names embargoed advisories only as exact duplicates. Then let the schedule take over. Artifact access follows the permissions of the private repository, so grant it to the security team only. -GitHub Actions has no trigger for security advisories, so the schedule polls every 15 minutes. Runs without a new advisory do not start the agent nor call the model. +GitHub Actions has no trigger for security advisories, so the schedule polls every 15 minutes. A run with no new advisory neither starts the agent nor calls the model.