From b70210a7740dce152f9d8af4d98e8906104910d5 Mon Sep 17 00:00:00 2001 From: "randomizedcoder dave.seddon.ca@gmail.com" Date: Sun, 8 Nov 2026 15:35:24 -0800 Subject: [PATCH 1/7] =?UTF-8?q?docs:=20security-hardening=20program=20?= =?UTF-8?q?=E2=80=94=20ConfigService.Set=20advisory=20and=20phased=20harde?= =?UTF-8?q?ning=20plan?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds docs/security-hardening/ with the full security assessment of the gRPC ConfigService.Set filesystem-write exposure and the program to close it: - security-vulnerability-configservice-set-fs-write.md — the advisory: the Set -> filesystem flow, the five Set-reachable write primitives, the fleet container deployment and permission model (NET_ADMIN/ SYS_ADMIN, read-only rootfs, private pidns), what the Nov 5 incident IOCs prove, and the defense layers L0-L6 - plan-configservice-set-hardening.md — workstreams W0-W10, issue register I1-I16, open decisions OD-1-OD-9, fleet rollout runbook - implementation-plan.md — phases 0-4 mapped to PRs with gates and a rollback matrix - phase-1-code-hardening.md — build-time elimination (dest_file/capture tags), protovalidate path patterns, runtime path policy + frozen fields, universal validating interceptor, Set audit, CI gates - phase-2-kernel-confinement.md — Landlock self-confinement: the ruleset, ABI degradation policy, re-exec persistence, and the full table-driven test matrices - phase-3-writer-split.md — privilege separation: writer process with a path-free proto envelope over a UDS (protovalidate both ends), SO_PEERCRED auth, monotonic seq, crash-safe commit dance, collector :ro flip, the complete write inventory, and the target admission checklist - phase-4-egress-fleet-lockdown.md — egress pinning in three layers (build tags closing OD-9, Landlock v4 network rules, security groups), escape-hatch removal, v1.x retirement, standing CI gates docs/README.md links the set from the security section. --- docs/README.md | 4 + .../security-hardening/implementation-plan.md | 152 ++++++ .../phase-1-code-hardening.md | 269 +++++++++++ .../phase-2-kernel-confinement.md | 260 +++++++++++ .../phase-3-writer-split.md | 365 +++++++++++++++ .../phase-4-egress-fleet-lockdown.md | 181 ++++++++ .../plan-configservice-set-hardening.md | 267 +++++++++++ ...ulnerability-configservice-set-fs-write.md | 438 ++++++++++++++++++ 8 files changed, 1936 insertions(+) create mode 100644 docs/security-hardening/implementation-plan.md create mode 100644 docs/security-hardening/phase-1-code-hardening.md create mode 100644 docs/security-hardening/phase-2-kernel-confinement.md create mode 100644 docs/security-hardening/phase-3-writer-split.md create mode 100644 docs/security-hardening/phase-4-egress-fleet-lockdown.md create mode 100644 docs/security-hardening/plan-configservice-set-hardening.md create mode 100644 docs/security-hardening/security-vulnerability-configservice-set-fs-write.md diff --git a/docs/README.md b/docs/README.md index 45678288..d6f43f65 100644 --- a/docs/README.md +++ b/docs/README.md @@ -65,6 +65,10 @@ gRPC defaults to TCP port `8889`, with optional Unix sockets and bearer/HMAC aut Current TCP/UDS, TTL/hop-limit, bearer/HMAC, and read/admin protections, with defaults, tradeoffs, recommended deployment profiles, secret handling, and verification examples. Authentication is off by default; native TLS and the connection/accept-rate limits proposed in the earlier design are not implemented. +### [Security advisory: `ConfigService.Set` filesystem writes](security-hardening/security-vulnerability-configservice-set-fs-write.md) + +`Set` re-executes the daemon with an attacker-chosen config and, absent path validation, becomes a root file-write primitive (`file:` destinations, `capture_path`, `baseline_file`, `ipmeta_cache_path`, unix listeners). The advisory maps every primitive, documents the container deployment and its permission model (runtime settings → Linux primitives, what `NET_ADMIN`+`SYS_ADMIN` really grant, what is restricted/filtered), checks the classic container-escape path idioms target-by-target, and lays out the layered fix: build-time elimination, protovalidate path rules, a runtime path policy with frozen fields, a universal validating interceptor, Landlock self-confinement, and a chrooted state-writer process split. The [hardening plan](security-hardening/plan-configservice-set-hardening.md) tracks the workstreams; the [implementation plan](security-hardening/implementation-plan.md) sequences the phases and PRs; the phase drill-downs go deep: [phase 1](security-hardening/phase-1-code-hardening.md) (build tags, proto validators, path policy, interceptor, audit, CI gates), [phase 2](security-hardening/phase-2-kernel-confinement.md) (Landlock self-confinement, full test matrices), [phase 3](security-hardening/phase-3-writer-split.md) (the writer-process split with its path-free proto protocol and the complete write inventory), and [phase 4](security-hardening/phase-4-egress-fleet-lockdown.md) (egress pinning, escape-hatch removal, v1.x retirement, standing CI gates). + ### [Observability](observability.md) Prometheus metrics, Go `pprof` endpoints, optional Pyroscope continuous profiling, and the startup capability check that explains exactly which Linux capabilities are required. diff --git a/docs/security-hardening/implementation-plan.md b/docs/security-hardening/implementation-plan.md new file mode 100644 index 00000000..43d3e006 --- /dev/null +++ b/docs/security-hardening/implementation-plan.md @@ -0,0 +1,152 @@ +# Implementation plan: locking down xtcp2 + +| | | +|---|---| +| **Status** | Active — checkboxes track execution; supersedes the sequencing section of the [hardening plan](plan-configservice-set-hardening.md) | +| **Repos** | `randomizedcoder/xtcp2` (code), the image repo (image), the deployment repo (deploy), the infra repo (SSM/IAM/network) | +| **Docs** | [Advisory](security-vulnerability-configservice-set-fs-write.md) — the problem (primitives, permission model, escape idioms). [Hardening plan](plan-configservice-set-hardening.md) — workstreams W0–W9 with design detail, estimates, and open decisions. **This doc** — execution order, PR-level steps, phase gates, and rollback. | + +Reading order for an implementer: skim the advisory's §2 (primitives) and §4 (permission model), then execute the phases below, dipping into the workstream sections of the hardening plan as each PR needs detail. + +## Contents + +- [Phase 0 — Immediate ops response (start today, no release)](#phase-0--immediate-ops-response-start-today-no-release) +- [Phase 1 — Code hardening release (vNext)](#phase-1--code-hardening-release-vnext) +- [Phase 2 — Kernel self-confinement](#phase-2--kernel-self-confinement) +- [Phase 3 — Privilege separation (writer split)](#phase-3--privilege-separation-writer-split) +- [Phase 4 — Egress and fleet lockdown](#phase-4--egress-and-fleet-lockdown) +- [Issue → phase tracking](#issue--phase-tracking) +- [Timeline and effort](#timeline-and-effort) +- [Gates and rollback](#gates-and-rollback) + +--- + +## Phase 0 — Immediate ops response (start today, no release) + +> **Status: descoped (2026-11 decision).** The auth converge (P0.1) remains recommended before or alongside Phase 1, but Phase 1 does not depend on it. Retained below for reference. + +**Goal:** close the unauthenticated `:8889` exposure. Pure ops — no code, no image. + +- [ ] **P0.1 — Merge and converge the v2.0.x ansible revision** (I10): listener auth (`raw`, per-host read + admin tokens) on every host. Canary via `ansible/xtcp2.yml`, then fleet converge; verify with `docker inspect` env on a sample. +- [ ] **P0.2 — Interim containment** for any host that cannot converge the v2.0.x revision yet: firewall `:8889` to operator sources, or `-e xtcp2_enabled=false`. +- [ ] **P0.3 — Detection**: alert on any `Set` that changes `dest`, and on repeated `FailedPrecondition` from `Set` (stolen-token probing). + +**Exit criteria:** every fleet host requires bearer tokens on `:8889`. + +**Effort:** 1–2 days, on-call infra. + +--- + +## Phase 1 — Code hardening release (vNext) + +> Drill-down with PR-level implementation detail: [phase-1-code-hardening.md](phase-1-code-hardening.md). + +**Goal:** no `Set` input — however crafted — can express a filesystem write, a path change, or an egress repoint; the debug writers are not in the production binary at all. + +PRs (repo `xtcp2` unless noted; each lands with its tests and doc updates from W9): + +- [ ] **PR-1 — build-time elimination** ([W0](plan-configservice-set-hardening.md#w0--build-time-elimination-l0)): `dest_file` build tag on the `file` destination; `capture` build tag on the netlink writer; both fatal at startup when requested but uncompiled (`destinationLookupError` path); `nix/versions.nix` flavor matrix + [build-flavors.md](../build-flavors.md) updated; decide [OD-8](plan-configservice-set-hardening.md#open-decisions) (`xtcp2-min`) and [OD-9](plan-configservice-set-hardening.md#open-decisions) (network stdlib destinations). +- [ ] **PR-2 — proto path validators** ([W1](plan-configservice-set-hardening.md#w1--protovalidate-path-rules-l1)): the segment pattern + `dest` scheme allowlist + `unix_socket_mode lte 384` + `ListenerEndpoint` cross-rule on all nine path-bearing fields; regenerate `gen/`; hostile-path table test. +- [ ] **PR-3 — runtime path policy + frozen fields** ([W2](plan-configservice-set-hardening.md#w2--runtime-path-policy--frozen-fields-l2)): `pkg/xtcp/set_path_policy.go` (`-writableDir`, canonicalize, component `Lstat`, root containment, `-allowFileDest`); wired into the `Set` handler **and** startup **and** `loadReconfigureConfig`; `set_frozen_fields.go` diff with `-setAllowFrozenFields` escape hatch; unit tests for reject/accept tables. +- [ ] **PR-4 — universal validating interceptor** ([W3](plan-configservice-set-hardening.md#w3--universal-validating-interceptor-l3)): unary interceptor + stream `RecvMsg` wrapper; descriptor-walking guard test so no future RPC skips validation. +- [ ] **PR-5 — `Set` audit logging** ([W6](plan-configservice-set-hardening.md#w6--set-audit-logging)): field-diff log lines (never secret values), token id, `xtcp2_set_applied_total{applied,frozen_rejected}` metric. +- [ ] **PR-6 — regression gates** ([W8](plan-configservice-set-hardening.md#w8--tests-ci-and-fleet-config-gates)): end-to-end advisory regression (all five primitives blocked at their layers); fleet golden-config CI gate (ansible-rendered configs from all DCs validated on every proto change). +- [ ] **PR-7 — image + role** (the image repo, the deployment repo): build and publish the new production flavor; pin bump; fleet-gate test asserts the flavor rejects `dest file:` and `write_files`. + +**Release gate:** full unit/race/microVM matrix green; golden-config gate green; 24 h canary; dashboard shows no unexpected `xtcp2_set_applied_total` rejections from legitimate automation. + +**Exit criteria:** fleet image where `Set` cannot reach any write primitive (L0 kills #1–2 in the binary; L1–L3 reject everything else; frozen fields stop `dest`/`s3_endpoint` repointing). + +**Effort:** ~1.5–2 weeks engineering + release. + +--- + +## Phase 2 — Kernel self-confinement + +> Drill-down with the ruleset design and the full table-driven test matrices: [phase-2-kernel-confinement.md](phase-2-kernel-confinement.md). + +**Goal:** even a validation bypass or an RCE inside the daemon cannot write outside the operator-approved roots. + +- [ ] **PR-8 — Landlock** ([W4](plan-configservice-set-hardening.md#w4--landlock-self-confinement-l4)): `pkg/sandbox` ruleset at startup (read-only `/proc`, `/sys`, `/run/netns`, `/run/docker/netns`, docker.sock, lldpd, tokens, ASN paths; write only `-writableDir` roots); ABI feature-detect with `xtcp2_sandbox_active` gauge; re-exec persistence test (restriction survives the soft restart); microVM write-denial scenario. +- [ ] **PR-9 — optional pivot_root belt** ([W4](plan-configservice-set-hardening.md#w4--landlock-self-confinement-l4)): self-`pivot_root` with the existing `CAP_SYS_ADMIN` (`CAP_SYS_CHROOT` is dropped); prioritize for bare-metal deployments where no container pidns defeats `/proc/1/root`. +- [ ] **PR-10 — state-dir guard + audit ship** ([W7](plan-configservice-set-hardening.md#w7--deployment-guardrails-l6)): size alert (quota per OD-7 later) on `/var/lib/xtcp2`; `Set` audit lines shipped and alerted on. + +**Gate:** a deliberately crippled build (path policy compiled out) still cannot write outside the state dir on a canary host — kernel-enforced proof, not validation. + +**Effort:** ~1 week. + +--- + +## Phase 3 — Privilege separation (writer split) + +> Drill-down with the writer protocol design and the full table-driven test matrices: [phase-3-writer-split.md](phase-3-writer-split.md). + +**Goal:** the collector holds zero write rules; the two legitimate writes live behind a chrooted, capability-less writer with a path-free API. + +- [ ] **PR-11 — writer** ([W5](plan-configservice-set-hardening.md#w5--state-writer-process-split-l5)): `proto/state_writer/v1` (`WriteBaseline`, `PublishIpmetaCache` — no path fields on the wire); `xtcp2 -writer` mode (UDS + `SO_PEERCRED`, fixed targets, size caps, rate limits, version numbers, temp+fsync+rename, `flock`); collector seams via the `internal/baseline` injection point and `ipasn.PublishLookupCache`; degrade-not-exit metrics; tests: no-path-parameter structural test, `kill -9` mid-write atomicity, version replay rejection. +- [ ] **PR-12 — config-surface shrink**: `baseline_file` / `ipmeta_cache_path` become writer-owned constants — removed from the `Set`-mutable surface entirely (already in the frozen list; now gone from the runtime path). +- [ ] **PR-13 — deployment** (the deployment repo): writer container (state dir `:rw`, shared socket dir, no network, uid 65534); collector state dir flips `:ro`; systemd units (`RootDirectory=`, `cap_drop ALL`, `AF_UNIX`) for bare metal; canary via the fleet gate; revert via `xtcp2_revert.yml` + restoring the `:rw` mount. + +**Exit criteria:** collector's Landlock ruleset contains **zero** write rules; `kill -9` the writer mid-baseline leaves the old baseline intact and the monitor degrades-not-exits; `Set` cannot name any path anywhere in the config. + +**Effort:** ~2–3 weeks (proto + writer + collector seams + deployment + microVM). + +--- + +## Phase 4 — Egress and fleet lockdown + +> Drill-down with the layer design and the full table-driven test matrices: [phase-4-egress-fleet-lockdown.md](phase-4-egress-fleet-lockdown.md). + +**Goal:** a stolen admin token can only tune poll knobs; everything else requires a redeploy, and the network layer enforces the egress set. + +- [ ] **PR-14 — close the escape hatch**: remove `-setAllowFrozenFields` once the OD-6 migration window closes; frozen fields become permanent. +- [ ] **PR-15 — egress pinning** (the deployment repo, the infra repo): container egress restricted to the per-DC S3 endpoints (network policy / security groups); `:8889` loopback/operators only, or UDS listeners; UFW rules reviewed with the monitoring lockdown role. +- [ ] **PR-16 — retire the v1.x revision** (I10 permanent): role defaults make disabling listener auth impossible; fleet-gate test asserts the auth env; retire the old revision branches. +- [ ] **PR-17 — standing CI gate** ([W8](plan-configservice-set-hardening.md#w8--tests-ci-and-fleet-config-gates)): advisory regression + golden-config run on every proto, daemon, image, and role change — the regression tests are the permanent enforcement of this whole program. + +**Final verification:** run the [end-state verification checklist](plan-configservice-set-hardening.md#end-state-verification-checklist) against a canary host **holding the admin token** — every primitive must fail safely at its layer. + +**Effort:** ~1 week. + +--- + +## Issue → phase tracking + +| Issue | Fixed in | Layer(s) | +|---|---|---| +| I1–I5 write primitives | Phase 1 (reject) + Phase 2 (kernel) + Phase 3 (API has no paths) | L0–L5 | +| I6 no path/scheme validation | Phase 1 (PR-2, PR-3) | L1, L2 | +| I7 per-handler validation drift | Phase 1 (PR-4) | L3 | +| I8 runtime egress repoint | Phase 1 (frozen fields) + Phase 4 (network) | L2, L6 | +| I9 auth off by default | Phase 0 (converge v2.0.x) | L6 | +| I10 unauthenticated v1.x fleet revision | Phase 0 + Phase 4 (retire) | L6 | +| I11 no state-dir quota | Phase 2 (alert) → Phase 4 (quota per OD-7) | L6 | +| I12 no `Set` audit trail | Phase 1 (PR-5) | L6 | +| I13 collector holds needless write access | Phase 3 | L4, L5 | +| I15 debug writers in production image | Phase 1 (PR-1) | L0 | + +## Timeline and effort + +| Phase | Elapsed (from start) | Content | +|---|---|---| +| 0 | days 0–2 | auth converge | +| 1 | weeks 1–3 | PR-1…PR-7, release vNext + fleet converge | +| 2 | weeks 3–4 | Landlock (+ optional pivot_root), quota/alert, audit ship | +| 3 | weeks 4–7 | writer split, `:ro` flip, systemd units | +| 4 | weeks 7–8 | egress pinning, v1.x retirement, standing gates | + +Phases 1 and 2 partially overlap (Landlock PR-8 can start as soon as PR-3's `-writableDir` lands); Phase 3's proto work can start during Phase 2. Nothing later is required for anything earlier to ship. + +## Gates and rollback + +- **Per release:** golden-config gate (never break a legit fleet config) + advisory regression (every primitive blocked) + hostile-path table + interceptor guard test. +- **Rollback matrix:** + +| Phase | Rollback | +|---|---| +| 1 | previous image tag + role pin (validations simply stop rejecting) | +| 2 | previous image tag (Landlock ruleset gone); quota/alert are additive | +| 3 | `xtcp2_revert.yml` + restore the collector's `:rw` state mount (writer container can stay, unused) | +| 4 | network policy rules (per-DC, reversible); `-setAllowFrozenFields` restoration if a genuine migration is blocked | + +- **Success definition (end state):** with the admin token in hand on a canary host, nothing in the [end-state verification checklist](plan-configservice-set-hardening.md#end-state-verification-checklist) succeeds in writing outside `/var/lib/xtcp2`, changing egress, or naming any path — and `/var/lib/xtcp2` itself is quota-guarded, audited, and (post-Phase 3) writable only by the capability-less writer. diff --git a/docs/security-hardening/phase-1-code-hardening.md b/docs/security-hardening/phase-1-code-hardening.md new file mode 100644 index 00000000..4ee79715 --- /dev/null +++ b/docs/security-hardening/phase-1-code-hardening.md @@ -0,0 +1,269 @@ +# Phase 1 — Code hardening release (drill-down) + +| | | +|---|---| +| **Status** | Not started — checkboxes track execution | +| **Tracks** | [Implementation plan](implementation-plan.md) Phase 1 → [hardening plan](plan-configservice-set-hardening.md) workstreams W0–W3, W6, W8 → [advisory](security-vulnerability-configservice-set-fs-write.md) layers L0–L3 | +| **Goal** | After this phase, no `Set` input — however crafted — can express a filesystem write, a path change, or an egress repoint, and the debug writers are absent from the production binary | +| **Repos** | `randomizedcoder/xtcp2` (PR-1…PR-6), the image repo + the deployment repo (PR-7) | + +## Contents + +- [Scope](#scope) +- [PR map, branches, and parallelization](#pr-map-branches-and-parallelization) +- [PR-1 — build-time elimination (`dest_file`, `capture`)](#pr-1--build-time-elimination-dest_file-capture) +- [PR-2 — protovalidate path rules](#pr-2--protovalidate-path-rules) +- [PR-3 — runtime path policy + frozen fields](#pr-3--runtime-path-policy--frozen-fields) +- [PR-4 — universal validating interceptor](#pr-4--universal-validating-interceptor) +- [PR-5 — `Set` audit logging](#pr-5--set-audit-logging) +- [PR-6 — regression and CI gates](#pr-6--regression-and-ci-gates) +- [PR-7 — image and role](#pr-7--image-and-role) +- [Decisions to close during Phase 1](#decisions-to-close-during-phase-1) +- [Release checklist](#release-checklist) +- [Definition of done](#definition-of-done) + +## Scope + +Phase 1 is pure code and build changes — no new processes, no kernel confinement (Phase 2), no deployment-topology changes (Phase 3). It closes issues I1–I8 and I15 at the validation and binary level: + +| Layer | Effect after Phase 1 | +|---|---| +| L0 (PR-1) | the production image contains no `file:` destination and no netlink capture writer — a hostile `Set` naming either returns `FailedPrecondition` (never a restart loop) | +| L1 (PR-2) | every path-bearing config field has a shape pattern; hostile paths are rejected by protovalidate before any Go code runs | +| L2 (PR-3) | a Go path policy confines every path to operator-approved roots and freezes egress/listener/path fields against runtime changes | +| L3 (PR-4) | every RPC input is validated by an interceptor — per-handler discipline becomes structural | + +## PR map, branches, and parallelization + +``` +PR-1 sec/w0-build-tags ─────┐ +PR-2 sec/w1-proto-paths ────┤ +PR-3 sec/w2-path-policy ───┼──▶ PR-6 sec/w8-regression ──▶ PR-7 (image, role) +PR-4 sec/w3-interceptor ────┘ ▲ +PR-5 sec/w6-set-audit (after PR-3: reuses its diff helper) +``` + +- PR-1, PR-2, PR-3, PR-4 are independent — four parallel tracks. PR-1 and PR-3 both touch the `Set` handler's validation block (`pkg/xtcp/grpc_configService.go` ~lines 186–262); land PR-1 first there to minimize rebase churn. +- PR-5 after PR-3 (it logs PR-3's frozen-field diff). +- PR-6 integrates; PR-7 is the release (repos: the image repo, the deployment repo). + +--- + +## PR-1 — build-time elimination (`dest_file`, `capture`) + +**Objective:** the production binary cannot write a file on `Set`'s behalf even if every other layer is bypassed — the code is not in it. + +**Files** + +- `pkg/xtcp/destinations_file.go` — add `//go:build dest_file` to the file header. The `init()` registration disappears from untagged builds; `knownSchemes` (`pkg/xtcp/destinations_core.go:70`) keeps `file` unconditionally so the lookup reports *"known but not compiled in"*. +- `pkg/xtcp/init_destinations.go` — no change needed for startup: `validateInput` (`pkg/xtcp/input_validation.go`) already calls `lookupDestinationFactory` and returns `destinationLookupError` (telling the operator to rebuild with `-tags dest_file`) — startup stays fatal, as today. +- `pkg/xtcp/grpc_configService.go` — **add a `Set`-side guard** so an untagged build returns `FailedPrecondition` instead of soft-restarting into a crash loop. Mirror the enricher pattern exactly — `checkConfigEnrichersCompiledIn` (`pkg/xtcp/enrich_core.go:206-228`) documents the two-severity rationale: startup = fatal, `Set` = `FailedPrecondition` ("Set re-execs the same binary… refusing it is strictly better than restarting into it"). New helper `checkDestCompiledIn(cfg) error` + `checkWriteFilesCompiledIn(cfg) error` beside it. +- New `pkg/xtcp/capture_gate_on.go` (`//go:build capture`) / `capture_gate_off.go` (`//go:build !capture`) — `const captureCompiledIn = true/false`, so the `write_files > 0` check compiles in both worlds. +- `nix/versions.nix` — `destinationFlavors`: make `full` an explicit list including `"file"` (`full = [ "file" "kafka" "nats" "nsq" "valkey" "s3parquet" ]`, replacing `null`); `min`/per-library flavors unchanged (no `file`); new `debugTools = [ "capture" ]` tag axis applied to the `-debug` attrs in `nix/binaries.nix` (or fold `capture` into the debug build's tag list — final wiring is this PR's to settle). +- Docs: [build-flavors.md](../build-flavors.md) (file moves from "always compiled" to tagged; new `capture` axis), [output-and-destinations.md](../output-and-destinations.md) (`file:` is debug-flavor only), [grpc-api.md](../grpc-api.md) (`Set` behavior on uncompiled schemes). + +**Tests** + +- `Set` with `dest: "file:/tmp/x"` on the default (untagged) test build → `codes.FailedPrecondition`, error names `-tags dest_file` (follow the `TestConfigService_Set_*` patterns in `pkg/xtcp/grpc_configService_test.go`). +- `Set` with `write_files: 1` on the default build → `FailedPrecondition` naming the `capture` tag; startup path (`validateInput`-level) with the same config → fatal error. +- Tagged build (`go test -tags 'dest_file capture' ./pkg/xtcp/`): `newFileDest` constructs and writes under a `t.TempDir()`; capture writer writes one file and decrements the budget. +- `lookupDestinationFactory("file")` without the tag → `destLookupNotCompiledIn` (gate-pair test file, both worlds). + +**Acceptance:** `nix build .#xtcp2-s3parquet` binary: startup with `dest file:…` or `write_files>0` exits non-zero with the rebuild hint; `Set` with either returns `FailedPrecondition` and the daemon keeps running. `.#xtcp2` (full) accepts both. Fleet golden configs (PR-6 gate) unaffected. + +**Risks:** a fleet config legitimately using `file:`/`write_files` (none known — fleet uses `s3parquet`; the golden-config gate proves it). Decision [OD-8](plan-configservice-set-hardening.md#open-decisions) (`xtcp2-min` keeps `file:`?) — recommend dropping it there too. + +--- + +## PR-2 — protovalidate path rules + +**Objective:** shape validation on every path-bearing field, enforced by the `protovalidate.Validate` calls that already run in every RPC. + +**Files:** `proto/xtcp_config/v1/xtcp_config.proto` (then `nix run .#regen-protos` → `gen/go`, `gen/dart`, …; `make regen-protos` in the [repo Makefile](../../Makefile) wraps it). + +The path pattern (document it once in the proto as a comment, reference per field): + +``` +segment = [A-Za-z0-9._-]*[A-Za-z0-9_-]+[A-Za-z0-9._-]* // ≥1 non-dot char ⇒ blocks "." and ".." +path = "^(/segment)+/?$" // absolute, no empty segments +``` + +**Field rules to add** (current rules verified at these field numbers): + +| Field | Current | Add | +|---|---|---| +| `capture_path` (51) | `min_len 1, max_len 80` | `pattern` above | +| `link_monitor.baseline_file` (LM field 2) | `max_len 4096` | `pattern` above | +| `asn_db_path` (241), `ipmeta_bootstrap_path` (243), `ipmeta_cache_path` (244) | `max_len 255` | `pattern` above | +| `docker_socket_path` (202), `lldpd_socket_path` (211) | `max_len 255` | `pattern` above | +| `xtcp_proto_file` (63) | `min_len 1` | `pattern` above | +| `dest` (60) | `min_len 4, max_len 512` | scheme pattern (below) | +| `ListenerEndpoint.address` (2) | `max_len 512` | charset pattern; message-level CEL cross-rule: `network == unix` ⇒ path pattern | +| `unix_socket_mode` (3) | `lte 511` | tighten to `lte 384` (0600) | + +**`dest` scheme pattern** — the real syntax (verified from `validateInput`, `pkg/xtcp/input_validation.go`): bare `null`/`stdout`/`stderr`, or `:payload` where path-style schemes take one colon and network schemes take two: + +``` +^(null|stdout|stderr|(tcp|udp|unix|unixgram|file|http|https|kafka|nats|nsq|valkey|s3parquet):[^[:cntrl:]]+)$ +``` + +Per-scheme payload shape (host:port vs path) stays in Go — `validateInput`'s switch already does the two-colon check; PR-3 adds root containment for `file:`. + +**Tests** + +- Hostile-path table (also used by PR-3): `../etc/passwd`, `/run/x/../../`, embedded NUL, `…/` (unicode ellipsis), `/proc/self/mem`, `/etc//hostname`, `/etc/hostname/`, `/etc/hostname`, symlink-into-root, `/var/lib/xtcp2/ok` (accept), `/tmp/cap` (accept only with a root configured). +- `dest` table: `stdout`, `null`, `file:/ok`, `kafka:h:9092`, `http://h/x` (accept); `FILE:/x`, `evil://x`, `stdout:extra`, control chars (reject). +- Cross-rule: `ListenerEndpoint{network: unix, address: "not a path"}` → `InvalidArgument`. + +**Acceptance:** every hostile case rejected at the proto layer; the fleet golden configs (all DCs, both role revisions) still validate (PR-6 gate runs them in CI). + +**Risks:** a legitimate path the pattern rejects — mitigated by the golden-config gate and [OD-8/OD-9](plan-configservice-set-hardening.md#open-decisions) review during this PR. + +--- + +## PR-3 — runtime path policy + frozen fields + +**Objective:** semantics CEL cannot express — deployment roots, symlink discipline, and a runtime diff against the running config. + +**Files** + +- New `pkg/xtcp/set_path_policy.go`: + +```go +type PathPolicy struct { + Roots []string // from -writableDir; empty ⇒ no filesystem-writing config allowed + AllowFileDest bool // from -allowFileDest +} + +// ValidateConfigPaths canonicalizes each path-bearing field (filepath.Clean + +// EvalSymlinks, error on failure), verifies no path component is a symlink +// (component-wise os.Lstat), and requires containment in a root. +func (p PathPolicy) ValidateConfigPaths(c *xtcp_config.XtcpConfig) error +``` + + Covered fields: `capture_path`, `write_files`, `link_monitor.baseline_file`, `asn_db_path`, `ipmeta_bootstrap_path`, `ipmeta_cache_path`, `docker_socket_path`, `lldpd_socket_path`, `xtcp_proto_file`, and unix `ListenerEndpoint.address`. `dest` scheme `file:` requires `AllowFileDest`. + +- New `pkg/xtcp/set_frozen_fields.go`: `func DiffFrozenConfigFields(running, incoming *xtcp_config.XtcpConfig) (changed []string)` over the frozen set: `dest`, `s3_endpoint`/`s3_bucket`/S3 credential fields, kafka/nats/nsq/valkey options, both listener endpoints + auth settings, `docker_socket_path`, `lldpd_socket_path`, `asn_db_path`, `ipmeta_*`, `baseline_file`, `capture_path`, `write_files`, `xtcp_proto_file`. (`marshal_to` stays runtime-mutable — format changes are legitimate ops; PR-5's audit line covers it.) +- `pkg/xtcp/grpc_configService.go` — wire both checks into `Set` after the existing enricher check, before `reconfigureFunc`: path violation → `codes.InvalidArgument` (field named); frozen-field change → `codes.FailedPrecondition` (frozen fields named), unless `-setAllowFrozenFields`. The running config is already in reach: `xtcpConfigService.config` (`grpc_configService.go:25`). +- `cmd/xtcp2/xtcp2.go` — register flags alongside the existing set (~line 454): `-writableDir` (repeatable), `-allowFileDest`, `-setAllowFrozenFlags` (default false). Run the same policy at **startup** and in `loadReconfigureConfig` (~line 1199) so an env-carried `XTCP_CONFIG_JSON` cannot bypass it after the soft restart. + +**Tests** + +- Reject/accept table for the policy (uses PR-2's hostile-path list): no roots configured ⇒ any path-bearing field ⇒ `InvalidArgument`; `/var/lib/xtcp2` root ⇒ in-root accept, out-of-root reject; symlink component reject (create one in `t.TempDir()`). +- Frozen-diff: change `dest` → rejected; change only `poll_frequency` → applied; `-setAllowFrozenFields` honored. +- Env-carried: set `XTCP_CONFIG_JSON` to a config with `capture_path`, assert the reload path rejects it. + +**Acceptance:** a `Set` that changes any path field, `dest` scheme, or egress endpoint is rejected before `reconfigureFunc`; a tuning-only `Set` (poll frequency, envelope caps) still applies and soft-restarts. + +**Risks:** operators using `Set` for dest migration (documented escape hatch + rollout note); legitimate symlinked state dir (document: roots must be canonical). + +--- + +## PR-4 — universal validating interceptor + +**Objective:** "every input is validated" becomes structural, not per-handler discipline. + +**Files** + +- New `pkg/xtcp/grpc_interceptors.go`: + +```go +func validatingUnaryInterceptor(ctx context.Context, req any, + info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (any, error) { + if m, ok := req.(proto.Message); ok { + if err := protovalidate.Validate(m); err != nil { + return nil, status.Error(codes.InvalidArgument, err.Error()) + } + } + return handler(ctx, req) +} +``` + + plus a stream interceptor wrapping `RecvMsg` (validates every received message — `FlatRecords`/`PollFlatRecords` requests are empty today, so cost is nil, but future fields are covered). +- `pkg/xtcp/grpc_server.go:85` — add both to the `grpc.NewServer(opts...)` options via `grpc.ChainUnaryInterceptor` / `grpc.ChainStreamInterceptor`. + +**Tests** + +- Descriptor-walking guard test: for every method of every registered service, either the request message has no fields, or a crafted-invalid message is rejected with `InvalidArgument` — so a new RPC with request fields cannot merge without a validation case. +- Backstop test: delete a per-handler `validate()` call in a test build and assert the interceptor still rejects (proves the guarantee does not depend on handler code). + +**Acceptance:** guard test green on all services; measured overhead on `PollFlatRecords` streams negligible (requests are empty). + +**Risks:** double validation on `Set` (cold path — fine); benchmark the stream path anyway. + +--- + +## PR-5 — `Set` audit logging + +**Objective:** every applied `Set` — and every rejection worth alerting on — is logged with a caller identity and shipped. + +**Files** + +- `pkg/listenerauth` — small extension: attach a **token fingerprint** to the auth context (sha256 of the token + listener name, first 12 hex chars — never the raw token). Nothing exists today; this is the only new surface. +- `pkg/xtcp/grpc_configService.go` — in `Set`, log the outcome at INFO: `event=set_applied token= outcome=applied|invalid_path|frozen_rejected|not_compiled_in fields=` (never secret values). +- Metric: `xtcp2_set_applied_total{outcome}` (follow the existing counter conventions, `c.pC`). +- Docs: [metrics.md](../metrics.md), [operations.md](../operations.md) (alert on `dest` changes and on repeated `frozen_rejected`). + +**Tests:** log-content assertions per outcome; counter increments per outcome; fingerprint stable across calls, raw token never logged. + +**Acceptance:** a canary `Set` round-trip produces one audit line + metric delta; the alert rule fires on a `dest` change. + +--- + +## PR-6 — regression and CI gates + +**Objective:** the phase's guarantees become permanent CI enforcement, and the fleet's real configs can never be broken by a tightening. + +**Files** + +- New `pkg/xtcp/set_hardening_regression_test.go` — the advisory's five primitives end-to-end through `Set` (against a service with a test `reconfigureFunc`): `dest file:`, `write_files`+`capture_path`, `baseline_file`, `ipmeta_cache_path`, unix listener — each rejected with the expected code at the expected layer; plus a tuning-only `Set` that applies. +- Golden-config gate — new `-validateConfig ` mode in `cmd/xtcp2/xtcp2.go`, analogous to `-healthcheck` (`:545`): load + protovalidate + full startup validation, exit 0/1 without starting the daemon. CI then renders the ansible role's configs (all DCs, both role revisions — a small script checked into the deployment repo or `ci/`) and validates each with the release binary. This makes CI prove "no legitimate fleet config is rejected" on every proto/daemon change. +- CI wiring: run the regression test + golden-config validation on every PR touching `proto/`, `pkg/xtcp/`, or `cmd/`. + +**Acceptance:** CI fails if (a) any primitive becomes reachable again, or (b) any golden fleet config is rejected. + +--- + +## PR-7 — image and role + +**Objective:** ship it. + +- Image repo: bump the `xtcp2` flake input to the release commit; build `oci-xtcp2-s3parquet-enrich` (now without `dest_file`/`capture`); smoke test in CI: the image binary with `dest file:…` / `write_files=1` exits non-zero naming the build tag; publish the image tag. +- Deployment repo: pin `xtcp2_image` to the new tag; verify the fleet-gate test still passes (it asserts image↔role compatibility); the binary-level assertions live in `xtcp2` CI (PR-6), not in the role. +- Rollout per the [fleet rollout runbook](plan-configservice-set-hardening.md#fleet-rollout-runbook): canary first, watch `xtcp2_set_applied_total` for legitimate-automation rejections (expect none — operators redeploy), then fleet converge. + +--- + +## Decisions to close during Phase 1 + +| # | Decision | Recommendation | +|---|---|---| +| [OD-8](plan-configservice-set-hardening.md#open-decisions) | does `xtcp2-min` keep `file:`? | drop — only `full`/`-debug` carry the tag | +| [OD-9](plan-configservice-set-hardening.md#open-decisions) | drop network stdlib destinations from the production flavor? | yes — production needs `s3parquet` only; blocks the DEST-repoint exfil class structurally | +| new | flag name for the config-validation mode (`-validateConfig`) | keep it — mirrors `-healthcheck`, usable by CI and ansible | +| new | freeze `marshal_to`? | no — runtime format changes are legitimate ops; PR-5's audit line covers it | + +## Release checklist + +- [ ] All six code PRs merged; `make regen-protos` output committed (`gen/` in sync) +- [ ] Full matrix green: unit + race + per-flavor (`dest_file`/`capture` worlds both tested) + microVM suite +- [ ] Golden-config gate green for both ansible role revisions +- [ ] Advisory regression test green (five primitives blocked) +- [ ] New image built, smoke-tested (rejects `file:`/`write_files`), published +- [ ] Role pin bumped; canary 24 h; `xtcp2_set_applied_total` reviewed +- [ ] Fleet converge; dashboards for `xtcp2_set_applied_total{outcome}` + the `dest`-change alert live + +## Definition of done + +From [implementation-plan.md](implementation-plan.md#phase-1--code-hardening-release-vnext): + +> **Exit criteria:** fleet image where `Set` cannot reach any write primitive — L0 kills #1–2 in the binary; L1–L3 reject everything else; frozen fields stop `dest`/`s3_endpoint` repointing. + +Concretely, with the admin token on a canary host running the new image: + +- [ ] `Set` with `dest = "file:/var/lib/xtcp2/pwn"` → `FailedPrecondition` (not compiled in) — no file, no restart loop +- [ ] `Set` with `dest = "file:…"` on a `full`-flavor binary without `-allowFileDest` → `InvalidArgument` +- [ ] `Set` with any path field outside `-writableDir` roots → `InvalidArgument`, field named +- [ ] `Set` changing `s3_endpoint`/`dest` scheme/listeners → `FailedPrecondition` (frozen) +- [ ] `Set` changing only poll knobs applies; soft restart works; one audit line logged +- [ ] Every RPC rejects a crafted-invalid message with `InvalidArgument` (interceptor) diff --git a/docs/security-hardening/phase-2-kernel-confinement.md b/docs/security-hardening/phase-2-kernel-confinement.md new file mode 100644 index 00000000..f892e3c5 --- /dev/null +++ b/docs/security-hardening/phase-2-kernel-confinement.md @@ -0,0 +1,260 @@ +# Phase 2 — Kernel self-confinement (drill-down) + +| | | +|---|---| +| **Status** | Not started — checkboxes track execution | +| **Tracks** | [Implementation plan](implementation-plan.md) Phase 2 → [hardening plan](plan-configservice-set-hardening.md) workstreams W4 (Landlock, optional pivot_root) + W7 (state-dir guard, audit ship) → [advisory](security-vulnerability-configservice-set-fs-write.md) layer L4 | +| **Goal** | After this phase, even a **validation bypass or an RCE inside the daemon** cannot write outside operator-approved roots — the guarantee moves from code to the kernel | +| **Prereq** | Phase 1 (PR-3's `-writableDir` flags feed the ruleset; PR-5's metric feeds the alert) | + +## Contents + +- [Scope](#scope) +- [PR map and branches](#pr-map-and-branches) +- [PR-8 — Landlock self-confinement](#pr-8--landlock-self-confinement) +- [PR-8 tests](#pr-8-tests) +- [PR-9 — optional pivot_root belt](#pr-9--optional-pivot_root-belt) +- [PR-10 — state-dir guard and audit shipping](#pr-10--state-dir-guard-and-audit-shipping) +- [Decisions to close during Phase 2](#decisions-to-close-during-phase-2) +- [Release checklist](#release-checklist) +- [Definition of done](#definition-of-done) + +## Scope + +Phase 1 made hostile `Set` inputs **invalid**. Phase 2 assumes Phase 1 is bypassed — a compromised daemon, a future validation bug, a hostile env-carried config — and confines the process at the syscall layer anyway: + +| Property | Mechanism | Survives | +|---|---|---| +| No write/create/unlink outside roots | Landlock filesystem ruleset applied at startup | `fork` **and** `execve` — including `Set`'s soft restart; not bypassable even by a privileged process | +| `/proc/sys` writes blocked (kernel backstop for the `core_pattern` idiom) | `/proc` granted read-only | same | +| State-dir fill detected + alerted | `xtcp2_state_dir_bytes` gauge + alert rules | — | +| Stolen-token probing visible | `FailedPrecondition` rate alert on PR-5's `xtcp2_set_applied_total` | — | + +Not in scope: network confinement (Landlock v4 network rules are a possible future layer — TCP bind/connect scoping — tracked as an open question, not planned here); capability reduction (the daemon keeps `NET_ADMIN`+`SYS_ADMIN` by design, see advisory §4.3). + +## PR map and branches + +``` +PR-8 sec/w4-landlock ──▶ PR-10 sec/w7-guardrails ──▶ release + role bump +PR-9 sec/w4-pivot-root (optional, independent, after PR-8) +``` + +--- + +## PR-8 — Landlock self-confinement + +**Objective:** a startup-applied Landlock ruleset that survives the soft restart and cannot be escaped by the process itself; degraded gracefully (warn + gauge, never a failed daemon) where the kernel lacks support. + +### Files + +- New `pkg/sandbox/sandbox.go` — the ruleset builder + `Constrain`. Mirror the `requiredCap` pattern from `pkg/xtcp/init_capabilities.go` (a declarative list + an injectable probe so the ruleset logic is table-testable without a kernel): + - `requiredPath{ paths []string, access AccessFSS, reason string }` — the table below is its content; + - `Constrain(cfg Config) Result` where `Result{ ABI int, Active bool, Rules int, Err error }`; + - the apply step sets `PR_SET_NO_NEW_PRIVS` (the go-landlock library does this itself) and applies the ruleset with `Restrict`. +- New `pkg/sandbox/sandbox_child_test.go` + `sandbox_test.go` — the [test harness](#pr-8-tests). +- `cmd/xtcp2/xtcp2.go` — one call site, ordered **after** flag parsing, config load, `InputValidation`, and the capability check, and **before** any destination/store construction: `sandbox.Constrain(...)`; skipped in `-healthcheck` and `-validateConfig` modes. +- `pkg/xtcp/prometheus.go` — new gauges beside the existing factory registrations (`prometheus.go:52-84`): `xtcp2_sandbox_active` (1/0), `xtcp2_sandbox_abi` (0 = unsupported). +- `nix/versions.nix` — new dependency → update `go.mod`/`go.sum` and `goVendorHash` (no vendor dir in-repo). + +### Dependency choice (decide in this PR) + +| Option | Pros | Cons | +|---|---|---| +| `github.com/landlock-lsm/go-landlock` v4 | idiomatically wraps ABI detection, `BestEffort()` degradation, rule construction | one new (pure-Go, tiny) dependency + nix hash bump | +| raw `golang.org/x/sys/unix` (already a dep, `v0.49.0`) | zero new dependencies; the syscall constants (`LandlockCreateRuleset`/`AddRule`/`RestrictSelf`, `LANDLOCK_ACCESS_FS_*`) are all present | we re-implement ABI detection and rule encoding (~150 lines, testable) | + +Recommendation: **go-landlock** — ABI handling is where the subtle bugs live, and the library is upstream-blessed. The API sketch below is illustrative; confirm exact method names against the go-landlock v4 README during implementation. + +### The ruleset + +Write roots come from Phase 1's `-writableDir` flags (empty by default = **no write rules at all** — post-Phase-3 that is the permanent state). Read grants are the daemon's verified needs (advisory §4.2): + +| Path(s) | Access (`LANDLOCK_ACCESS_FS_*`) | Why | +|---|---|---| +| dir of `os.Executable()` (`/bin` in the image) | `EXECUTE \| READ_FILE` | the `Set` soft restart `syscall.Exec`s the same binary — without this, the first `Set` kills the daemon (corner case C3) | +| `/proc` | `READ_FILE \| READ_DIR` | namespace discovery (`/proc/self/ns/net`, `/proc/self/status`); read-only grant is also the kernel backstop against `/proc/sys` and `/proc/sysrq-trigger` writes | +| `/sys` | `READ_FILE \| READ_DIR` | interface/ethtool sysfs reads | +| `/run/netns`, `/run/docker/netns` | `READ_FILE \| READ_DIR` | named-netns discovery | +| `/run/docker.sock` | `READ_FILE \| WRITE_FILE` | `connect(2)` to a UDS requires rw access on the socket file under Landlock — a **scoped per-path grant** (a socket cannot be used as a file-write target; `open(2)` on it fails regardless) | +| `/run/lldpd.socket` (where present) | `READ_FILE \| WRITE_FILE` | same — LLDP enrichment datagram socket | +| `/sys/fs/cgroup` | `READ_FILE \| READ_DIR` | container-id resolution | +| `/etc` | `READ_FILE \| READ_DIR` | tokens + S3 secret mounts, `/etc/resolv.conf`, `/etc/hosts` (pure-Go resolver), TLS roots | +| `/dev/null` | `READ_FILE \| WRITE_FILE` | runtime; the only `/dev` write grant | +| `/dev/urandom`, `/dev/random` | `READ_FILE` | `crypto/rand` seeding | +| `/share/xtcp2/ipmeta` (where present) | `READ_FILE \| READ_DIR` | ASN bootstrap artifact | +| each `-writableDir` root | `READ_FILE \| READ_DIR \| WRITE_FILE \| MAKE_DIR \| MAKE_REG \| MAKE_SOCK \| MAKE_SYM \| REMOVE_FILE \| REMOVE_DIR \| REFER \| TRUNCATE` | baseline + ipmeta atomic writes (`CreateTemp`→`fsync`→`rename` needs `MAKE_REG`+`REMOVE_FILE`+`REFER`); UDS listener bind needs `MAKE_SOCK`+`REMOVE_FILE` (`unlink_stale_unix_socket`); `REFER` (ABI 2) and `TRUNCATE` (ABI 3) degrade on older kernels | + +Notes baked into the design: + +- **Same-dir atomic rename works from ABI 1** — `REFER` only gates *cross-directory* moves, and the stores never truncate (they replace via rename). This is why the ABI floor for write roots is v1, not v3 (see [D-table](#test-d--abi-policy-testsandboxabipolicy)). +- **UDS listener addresses must live under a write root** (bind creates a socket file). This matches and re-states Phase 1's L2 rule; the Landlock denial is the kernel backstop. +- **Existing fds opened before `Constrain` stay usable** — Landlock evaluates at `open(2)` time. The wiring order (constrain before destination/store init) is what makes this moot; test B20 documents the semantic. +- **`no-new-privileges`** is set by the apply step; the container already sets it — idempotent. + +--- + +## PR-8 tests + +Three layers: **A/E/D** are pure table-driven unit tests (no kernel, all platforms), **B/C** run in a self-re-exec'd child process on Linux (Landlock needs no privileges; skip when the kernel is < 5.13), and the microVM suite covers the container-shaped integration. + +**Harness (B/C):** the test binary re-execs itself with `XTCP2_SANDBOX_CHILD=1`; the child reads its ruleset parameters from env, applies `Constrain`, runs **one** named probe, and reports `OK`, `EACCES`, `EPERM`, `ENOENT`, … on stderr. The parent drives the table, sets up `t.TempDir()` paths via env, and asserts each outcome. Gate: `//go:build linux`, `t.Skip` when `sandbox.ProbeABI() < 1`. + +### Test A — ruleset construction (`TestSandboxRuleset`, pure) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| A1 | P | one write root + default read paths | `Roots=["/var/lib/xtcp2"]` | rules = the [table above](#the-ruleset) exactly: read-only grants + one write grant on the root; `Result{ABI probed, Active true, Rules == expected count}` | +| A2 | N | no roots configured | `Roots=[]` | **zero** `WRITE_FILE/MAKE_*` bits in any rule; read grants unchanged | +| A3 | B | root with redundant slashes and trailing `/` | `Roots=["/var/lib/xtcp2//sub/"]` | rule path is `filepath.Clean`ed; exactly one rule | +| A4 | N | root does not exist | `Roots=["/nope/deep"]` | startup error naming the root (path must exist at rule time) | +| A5 | C | root is `/` (operator footgun) | `Roots=["/"]` | allowed (explicit operator choice) but startup log carries a loud warning; rule mask is the write set | +| A6 | C | overlapping roots | `Roots=["/var/lib", "/var/lib/xtcp2"]` | both rules emitted (Landlock unions); no dedup error, no failure | +| A7 | P | UDS listener enabled under a root | `Roots=["/run/xtcp2"]`, listener `network: unix` | root mask includes `MAKE_SOCK \| REMOVE_FILE`; address inside root passes policy | +| A8 | N | lldpd socket absent (host without lldpd) | no `/run/lldpd.socket` | rule omitted; no error | +| A9 | B | root equals the binary's dir | root == dir of `os.Executable()` | two grants on one path (exec + write) coexist; no conflict | +| A10 | C | duplicate identical roots | `Roots=["/a","/a"]` | single rule (dedup) | + +### Test B — confinement behavior (child-process harness) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| B1 | P | create a file inside the root | `OpenFile(root+"/f", O_CREATE\|O_WRONLY)` | `OK` | +| B2 | P | `MkdirAll` inside the root | `os.MkdirAll(root+"/ipmeta/nested")` | `OK` | +| B3 | P | baseline-style atomic write inside the root | `CreateTemp(root)` → write → `fsync` → `rename` over target | `OK` (exercises `MAKE_REG`+`REMOVE_FILE`+`REFER`/same-dir rename) | +| B4 | N | create a file outside any root | write to `$HOME`-adjacent scratch | `EACCES` | +| B5 | N | create a file in `/tmp` (not a root) | `OpenFile("/tmp/pwn")` | `EACCES` | +| B6 | N | write `/etc/hostname` (a classic marker target) | `OpenFile("/etc/hostname", O_WRONLY\|O_APPEND)` | `EACCES` | +| B7 | N | append under the read-only `/run` grants | write a file next to the docker.sock mount | `EACCES` | +| B8 | N | `ipmeta` primitive outside roots | `os.MkdirAll("/opt/x/y")` + `os.Create` | `EACCES` on the mkdir | +| B9 | N | unlink a file outside roots | `os.Remove("/tmp/x")` | `EACCES` | +| B10 | N | `chmod` a file outside roots | `os.Chmod("/tmp/x", 0777)` | `EACCES` (open-for-write fails before the mode change) | +| B11 | N | rename from root to outside | `os.Rename(root+"/f", "/tmp/f")` | `EACCES` (cross-dir `REFER` + target-dir write both denied) | +| B12 | P | rename within the root (replace existing target) | `os.Rename(root+"/tmp", root+"/current")` | `OK` — proves baseline/ipmeta atomicity survives | +| B13 | P | connect to a granted UDS | socket created by parent under a granted path; child dials | `OK` | +| B14 | N | connect to an ungranted UDS | socket outside granted paths | `EACCES` | +| B15 | P | read `/proc/self/status` | plain read | `OK` | +| B16 | P | read `/etc/resolv.conf`, `/etc/hosts` | plain reads (resolver prerequisites) | `OK` | +| B17 | N | write under `/proc/sys` | open `/proc/sys/kernel/hostname` for write | `EACCES` — the `core_pattern`-idiom kernel backstop | +| B18 | B | open a **nonexistent** path under a granted dir | `OpenFile("/etc/no-such-file")` | `ENOENT` — Landlock must not mask normal errno semantics | +| B19 | C | symlink inside a root pointing outside | symlink → `/etc`; write through it | `EACCES` (Landlock resolves the symlink; the *target's* grants govern) | +| B20 | C | fd opened **before** `Constrain` stays writable | child opens file pre-constrain, writes post-constrain | `OK` — documents kernel semantics; the wiring order (constrain before store init) is what neutralizes it | +| B21 | P | open `/dev/urandom`, `/dev/null` | read + write respectively | `OK` | +| B22 | N | create a **symlink** outside roots (`MAKE_SYM`) | `os.Symlink` in `/tmp` | `EACCES` | +| B23 | C | bind a UDS listener outside any root | `net.Listen("unix", "/tmp/x.sock")` | `EACCES` — kernel backstop for the planted-socket primitive | + +### Test C — persistence across the soft restart (`TestSandboxReexecPersistence`) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| C1 | P | re-exec succeeds under the ruleset | child applies ruleset → `syscall.Exec(self, args, env)` | exec permitted (binary dir has `EXECUTE\|READ_FILE`) — without that grant this test fails, which is exactly the regression it guards | +| C2 | P | write inside root still works post-exec | after C1's exec, child writes in root | `OK` | +| C3 | N | write outside root still denied post-exec | after C1's exec, child writes outside | `EACCES` — the guarantee survives `syscall.Exec` | +| C4 | N | exec a **different** binary not granted execute | `syscall.Exec("/bin/sh")` (test helper equivalent) | `EACCES` at exec | +| C5 | B | `Set`-shaped flow end-to-end | child 1 applies → execs child 2 with a hostile "config" in env that names an out-of-root path | child 2's write attempt returns `EACCES` even though child 2 re-derived its ruleset from the same args | + +### Test D — ABI policy (`TestSandboxABIPolicy`, pure — the version→policy decision table) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| D1 | P | ABI 4 (Ubuntu 24.04 / 6.8) | probed version 4 | full ruleset incl. `REFER`+`TRUNCATE`; `xtcp2_sandbox_active 1`, `xtcp2_sandbox_abi 4` | +| D2 | B | ABI 2 (5.19) | probed version 2 | `REFER` kept, `TRUNCATE` dropped; write roots allowed (same-dir atomic rename needs neither); one info-level log line | +| D3 | B | ABI 1 (5.13) | probed version 1 | `REFER` and `TRUNCATE` dropped; same-dir rename still permitted (v1 rights cover it); active 1; info log | +| D4 | N | ABI 0 / `ENOSYS` (unsupported kernel) | probe fails with not-supported | **warn** + `xtcp2_sandbox_active 0` + daemon continues (OD-2: degrade, never a failed daemon) | +| D5 | C | `EPERM` from a seccomp-blocked `landlock_*` syscall (possible on old Docker default profiles) | probe returns EPERM | same as D4: warn + gauge 0 + continue, message names the likely cause | +| D6 | C | ABI ≥ 2 but UDS listener outside all roots | listener path not under a write root | `Constrain` returns a configuration error (loud at startup, not a runtime bind surprise) | + +### Test E — root validation (`TestSandboxRootValidation`, pure) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| E1 | N | root does not exist | `Roots=["/nope"]` | startup error naming the root | +| E2 | C | root is a symlink | root → real dir via `EvalSymlinks` | rule emitted for the **resolved** target; startup log notes the canonicalization | +| E3 | B | root == binary dir | as A9 | accepted; both grants present | +| E4 | N | empty-string root | `Roots=[""]` | startup error (reject before Clean produces `.`) | +| E5 | C | relative root | `Roots=["state"]` | rejected — roots must be absolute (mirror of PR-2's proto pattern) | + +--- + +## PR-9 — optional pivot_root belt + +**Objective** (advisory §5.4 variant): hide the host filesystem from path resolution entirely, in-process, using the `CAP_SYS_ADMIN` the daemon already holds (`CAP_SYS_CHROOT` is dropped, so `chroot()` itself is unavailable). + +- New `cmd/xtcp2/pivot.go` (or `pkg/sandbox/pivot.go`): `unshare(CLONE_NEWNS)` → read-only bind-mount farm → `pivot_root(".")` → drop the old root. **Gate behind a flag** (`-sandboxPivotRoot`), default off — Landlock (PR-8) is the primary. +- Bind-mount farm (everything from the Landlock table, minus `/dev/null`/`urandom` device handling which needs a fresh `devtmpfs`/tmpfs + mknod — note `CAP_MKNOD` is dropped, so /dev must be carried as a bind, not rebuilt). +- Tests (gated `//go:build linux`, requires root — microVM tier): farm table where each row is one required path verified reachable post-pivot (DNS resolution, TLS roots, netns discovery, lldpd connect, baseline write, re-exec); plus a negative row — a path *not* in the farm resolves to `ENOENT` rather than the host tree. +- Priority: bare-metal deployments only (in the container, Landlock + the private pidns already cover everything pivot_root adds). + +## PR-10 — state-dir guard and audit shipping + +**Objective:** the residual writable surface (the state dir, pre-Phase-3) is watched, and Phase 1's audit lines reach the monitoring pipeline with actionable alerts. + +### In `xtcp2` (this repo) + +- `pkg/xtcp/state_dir.go` — `xtcp2_state_dir_bytes` gauge: walk `-writableDir` roots on a low cadence (the ASN refresh timer is fine; the dir is small), never on the poll path. Also `xtcp2_state_dir_walk_errors_total{reason}` for skipped entries. +- Register beside the existing gauges (`pkg/xtcp/prometheus.go`). + +### In the deployment repo + +- Set `WRITABLE_DIR=/var/lib/xtcp2` (via the daemon's flag/env bridge) so the fleet actually confines — without it, Landlock applies with **zero** write rules and the baseline writer fails; the rollout note in the runbook covers this ordering (image first, env second). +- vmalert rules (+ `promtool test rules` case files, checked into the repo): + - `Xtcp2StateDirFilling`: `xtcp2_state_dir_bytes > ` for 15m. + - `Xtcp2SandboxInactive`: `xtcp2_sandbox_active == 0` on any host (fleet kernels are ≥ 6.8 — a 0 means an unexpected kernel or seccomp block). + - `Xtcp2SetAbuse`: `rate(xtcp2_set_applied_total{outcome="frozen_rejected"}[5m]) > 0.1` (stolen-token probing) and any `dest` change (Phase 1 PR-5). +- Ship `event=set_applied` lines through the existing log pipeline to the ops log index. + +### Test G — state-dir walk (`TestStateDirBytes`, pure) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| G1 | P | nested dirs + files | tree with 3 files (10, 100, 1000 B) + subdir | gauge = 1110 (file sizes; dirs not counted — document the convention) | +| G2 | N | unreadable subdir | chmod 000 dir (as another uid, or simulate via injected walk error) | subdir skipped; `xtcp2_state_dir_walk_errors_total{reason}` +1; walk succeeds | +| G3 | B | empty root | empty dir | gauge 0, no error | +| G4 | C | symlink inside the tree | symlink → file elsewhere | not followed; link itself not counted (`filepath.WalkDir` semantics, documented) | +| G5 | P | large file | one 256 MiB file | gauge exact; walk time bounded (assert < 1 s) | +| G6 | C | root disappears mid-walk | injected remove between walk ticks | walk error, gauge holds last value, counter +1, next tick recovers | + +### Test H — alert rules (`promtool test rules`, in the deployment repo) + +| ID | Cat | Description | Input series | Expected outcome | +|---|---|---|---|---| +| H1 | P | state dir over threshold | `xtcp2_state_dir_bytes` = threshold + 1 B for 15m | `Xtcp2StateDirFilling` fires at 15m | +| H2 | B | exactly at threshold | = threshold for 15m | fires (comparison is `>`, stays pending at exactly threshold — assert the documented operator choice) | +| H3 | N | under threshold | = threshold − 1 B | no alert | +| H4 | C | gauge absent (daemon down) | no series for 10m | `absent()`-based `Xtcp2NoStateDirMetric` fires (separate from the daemon-down alert) | +| H5 | P | frozen-rejected burst | 5 rejections in 5m from one host | `Xtcp2SetAbuse` fires with that host label | + +--- + +## Decisions to close during Phase 2 + +| # | Decision | Recommendation | +|---|---|---| +| OD-2 (already defaulted) | Landlock-absent kernel behavior | warn + `xtcp2_sandbox_active 0` + continue (D4/D5) | +| new | dependency: go-landlock vs raw x/sys | go-landlock | +| new | ABI floor for write roots | v1 (same-dir atomic rename needs no `REFER`/`TRUNCATE`); v0 disables writes with a warning | +| new | Landlock network rules (TCP bind/connect scoping) | out of scope this phase; revisit after Phase 4 egress pinning | +| OD-7 | quota mechanism | alert-only here (Test H); XFS project quota deferred to Phase 4 | + +## Release checklist + +- [ ] PR-8 merged: ruleset construction (A), child-harness confinement (B), re-exec persistence (C), ABI policy (D), root validation (E) all green on dev kernels and CI +- [ ] microVM suite: container-shaped scenario — daemon with a state-dir root, `Set` applies, writes inside root succeed, `/etc/hostname`-style writes fail at the kernel +- [ ] Gauge pair (`xtcp2_sandbox_active`/`_abi`) exported and documented in [metrics.md](../metrics.md) +- [ ] go-landlock (or x/sys) dependency landed via nix (`goVendorHash` updated) +- [ ] PR-10: `WRITABLE_DIR` env set by the role **after** the image bump; vmalert rules + `promtool` cases merged; audit lines visible in the log index +- [ ] Canary 24 h: `xtcp2_sandbox_active == 1` on all canaries; zero `xtcp2_state_dir_walk_errors_total` surprises; no S3 upload failures (a broken read grant would show up exactly there — see the ruleset table's `/etc` row) + +## Definition of done + +From [implementation-plan.md](implementation-plan.md#phase-2--kernel-self-confinement): + +> **Gate:** a deliberately crippled build (path policy compiled out) still cannot write outside the state dir on a canary host — kernel-enforced proof, not validation. + +Concretely, on a canary host running a **test build with Phase 1's path policy stubbed out**: + +- [ ] `Set` with `dest = "file:/etc/cron.d/pwn"` applies (no validation left) → soft restart → the file-write attempt fails with `EACCES` in the daemon logs; **no file exists** on the host +- [ ] same for `capture_path`, `baseline_file`, `ipmeta_cache_path` outside roots, and a planted UDS outside roots +- [ ] baseline + ipmeta writes inside `/var/lib/xtcp2` still succeed (the fleet keeps working) +- [ ] `Set` (tuning-only) applies and the daemon re-execs — confinement survives +- [ ] `xtcp2_sandbox_active` is 1 on every canary; an alert fires if any host reports 0 +- [ ] filling `/var/lib/xtcp2` past the threshold raises `Xtcp2StateDirFilling` within one cycle diff --git a/docs/security-hardening/phase-3-writer-split.md b/docs/security-hardening/phase-3-writer-split.md new file mode 100644 index 00000000..22e6a2b7 --- /dev/null +++ b/docs/security-hardening/phase-3-writer-split.md @@ -0,0 +1,365 @@ +# Phase 3 — Privilege separation: the writer split (drill-down) + +| | | +|---|---| +| **Status** | Not started — checkboxes track execution | +| **Tracks** | [Implementation plan](implementation-plan.md) Phase 3 → [hardening plan](plan-configservice-set-hardening.md) workstream W5 → [advisory](security-vulnerability-configservice-set-fs-write.md) layer L5 | +| **Goal** | The collector process — the part that talks to the network, parses attacker-influenced data, and serves gRPC — holds **no filesystem write capability at all**. The only two legitimate writes move to a separate, minimal, unprivileged writer process | +| **Prereqs** | Phase 1 (frozen path fields so `Set` can't move targets; build tags) and Phase 2 (`pkg/sandbox`, reused by both processes) | + +## Contents + +- [Scope](#scope) +- [PR map and branches](#pr-map-and-branches) +- [PR-11 — the writer process](#pr-11--the-writer-process) +- [PR-11 tests](#pr-11-tests) +- [PR-12 — collector split and config shrink](#pr-12--collector-split-and-config-shrink) +- [PR-12 tests](#pr-12-tests) +- [PR-13 — deployment and the `:ro` flip](#pr-13--deployment-and-the-ro-flip) +- [PR-13 tests (microVM)](#pr-13-tests-microvm) +- [Appendix A — the write inventory and the target wish-list](#appendix-a--the-write-inventory-and-the-target-wish-list) +- [Decisions to close during Phase 3](#decisions-to-close-during-phase-3) +- [Release checklist](#release-checklist) +- [Definition of done](#definition-of-done) + +## Scope + +The daemon has exactly two legitimate writes (advisory §2): the link-monitor baseline JSON (≤ 64 KiB, `baseline.MaxFileBytes`) and the ipmeta parquet lookup cache. Everything else `Set` can reach is a debug primitive that Phase 1 eliminated at build time. + +| Threat | Before Phase 3 | After Phase 3 | +|---|---|---| +| Collector RCE writes anywhere validation allows | one trusted process, one ruleset | collector has zero file-write rules — the kernel answer is "no", always | +| `Set` flips a path field | rejected (Phase 1) / kernel-denied (Phase 2) | same, and the path fields no longer exist in the collector's write path at all | +| Collector RCE **as root** wants to poison state | — | must speak the writer's fixed 3-op API: two named targets, one size cap each, monotonic seq — no paths, no arbitrary bytes past schema checks | +| Writer itself compromised | — | unprivileged uid, no capabilities, no network (container `network_mode: none`), Landlock write rules only on its two target dirs | + +What this phase does **not** fix: the collector can still reach network egress (Phase 4) and the writer accepts a well-formed request from uid 0 (or a stolen shared secret — see [decisions](#decisions-to-close-during-phase-3)) and will overwrite its two fixed targets with schema-valid data; that bounded surface is the accepted residual. + +## PR map and branches + +``` +PR-11 sec/w5-writer ──▶ PR-12 sec/w5-collector ──▶ PR-13 sec/w5-deploy ──▶ release + role bump + (needs PR-1/PR-3; (deployment repo; + sandbox from PR-8) PR-13 last) +``` + +--- + +## PR-11 — The writer process + +**Objective:** a minimal writer daemon — same binary, `-writer` mode — exposing four operations over one unix socket. The security model lives in the parameter list: **no request anywhere carries a path.** + +### Files + +- New `pkg/writerd/writerd.go` — server, request validation, target registry, seq state, crash-safe commit dance +- New `pkg/writerd/proto.go` — the `[len u32]` frame + proto envelope codec (below) +- New `proto/xtcp_writer/v1/writer.proto` — message-only envelope (no `service` definition), annotated with `buf.validate` rules, generated via the usual `make regen-protos` +- New `pkg/writerd/client.go` — the collector-side client +- `cmd/xtcp2/xtcp2.go` — `-writer` mode: writer never parses `XtcpConfig`, never registers gRPC services, never initializes enrichers/netlink; it reads its own flags only +- `pkg/xtcp/prometheus.go` — collector-side `xtcp2_writer_requests_total{op,result}` (client view) + proxy registration for the writer's own counters +- `nix/versions.nix` — no new dependencies: protobuf runtime (`v1.36.12`) and protovalidate-go are already direct dependencies; `golang.org/x/sys/unix` suffices for `SO_PEERCRED` + +### The API + +| Op | Name | Request | Writer-side effect | Writer-side validation | +|---|---|---|---|---| +| 0 | `PING` | — | reply | none | +| 1 | `WRITE_BASELINE` | `payload []byte`, `seq uint64` | temp+fsync+rename onto the configured baseline target (`0600`) | oneof arm selects the `baseline` target; protovalidate `max_len 65536` (== `baseline.MaxFileBytes`); payload parses as a `wireRecord` with `Version == 1` (`baseline.Parse` reused); `seq` monotonic (`gte 1`) | +| 2 | `WRITE_IPMETA` | `payload []byte`, `seq uint64` | temp + preserve-previous (`current → previous`, both writer-configured) + rename + `syncDir` — the exact dance of `ipasn.PublishLookupCache` (`ipasn.go:373`) | oneof arm selects the `ipmeta` target; protovalidate `max_len` == `-ipmetaMaxBytes` (default 64 MiB); `seq` monotonic (`gte 1`) | +| 3 | `GET_STATS` | — | counters + last `seq` per target | none | + +The two write ops port existing code, not new logic: baseline's atomic dance comes from `pkg/linkmonitor/internal/baseline/files.go`; ipmeta's from `pkg/ipasn/ipasn.go` (`MkdirAll 0750`, `preservePrevious`, `syncDir`). Validation of the *ipmeta payload* stays collector-side (`validateCacheCandidate` needs the refresh loop's `Stats`); the writer validates size, schema only where cheap (baseline), and seq. + +### Wire format and the protocol decision + +Frame: little-endian `[len u32]`, then one protobuf `Request`; reply frame `[len u32]`, one protobuf `Response` (oneof `ok` / `err{code,detail}` / `stats`). The envelope is proto — deliberately **not** a gRPC service: + +```proto +// proto/xtcp_writer/v1/writer.proto — message-only, no service definition. +message WriteBaselineRequest { + uint64 seq = 1 [(buf.validate.field).uint64.gte = 1]; + bytes payload = 2 [(buf.validate.field).bytes.max_len = 65536]; // == baseline.MaxFileBytes +} +message WriteIpmetaRequest { + uint64 seq = 1 [(buf.validate.field).uint64.gte = 1]; + bytes payload = 2 [(buf.validate.field).bytes.max_len = 67108864]; // == -ipmetaMaxBytes default +} +message Request { + oneof op { + PingRequest ping = 1; + WriteBaselineRequest write_baseline = 2; + WriteIpmetaRequest write_ipmeta = 3; + StatsRequest stats = 4; + } +} +// Response mirrors: oneof { Ok ok = 1; Err err = 2; StatsResponse stats = 3; } +``` + +Per-op caps live in per-message rules — the oneof expresses them natively where a flat message could not. `.Validate()` runs on **both ends**: collector-side before send (catches collector bugs at the source), writer-side after unmarshal (catches everything else). + +**Why not gRPC over the UDS?** Both processes run the same binary and gRPC is already in it, so this is about *exposure*, not dependency weight: + +| Dimension | gRPC over UDS | proto envelope + tiny frame (chosen) | +|---|---|---| +| Parser facing a pwned collector | the full HTTP/2 stack — SETTINGS, HPACK, flow control, trailers, keepalive; a CVE history (rapid reset et al.) | `len u32` + proto unmarshal | +| Validation | protovalidate via PR-4's universal interceptor (genuine reuse) | protovalidate via `.Validate()` — same rules | +| `SO_PEERCRED` | custom listener-wrapper code (no stock support) | direct `GetsockoptUcred` on the accepted conn | +| Deadlines/timeouts | free (ctx) | ~20 lines in the client (covered by D3) | +| Knobs to get right | `MaxRecvMsgSize`, keepalive, streams, compression codecs, reflection, health service — each a misconfiguration risk | none | +| Evolvability | new RPCs at will | new oneof arms — the friction is deliberate (a frozen API is the security property) | + +A team standardizing on one IPC mechanism could defensibly choose gRPC-over-UDS anyway; the hardening checklist is then: `MaxRecvMsgSize` at the payload cap + margin, no reflection service, no health service, no compression codecs imported, keepalive disabled, one stream per connection — and the same filter chain below. + +**Writer receive pipeline — a series of fail-closed filters, each stage metric-tagged (`xtcp2_writerd_requests_total{op,filter}`):** + +1. frame `len` checked against the global cap (64 MiB + margin) **before allocation** — `make(4 GiB)` is unreachable (protobuf-go `v1.36.12` has no `UnmarshalOptions.MaxSize`; this check is the allocation guard) +2. `proto.Unmarshal` → `malformed` on parse failure; unknown fields are **rejected explicitly** — the default stores them (`decode.go` `m.SetUnknown`), so the writer asserts `GetUnknown() == nil` after unmarshal +3. `.Validate()` (protovalidate: `gte 1`, per-op `max_len`) → `invalid` +4. op/target allow-list — the op's target must have been passed via `-writerTarget` → `unconfigured_target` +5. `seq` monotonic per target (sidecar state) → `stale_seq` +6. payload schema check (baseline `wireRecord` parse) → `invalid_payload` +7. commit dance (temp → fsync → rename → sidecar) + +### Targets, auth, seq, locking + +- **Targets** come from repeated `-writerTarget name=path` flags — the writer's own config, reachable by nothing `Set` controls. The request's oneof arm selects the target — the envelope carries no strings at all; an op whose target was not configured is rejected (`unconfigured_target`). +- **Auth:** `SO_PEERCRED` on each accepted connection — peer uid must equal the configured collector uid (fleet: 0; check injectable for tests like `capgetFunc` in `init_capabilities.go`). Socket dir `0750`, socket `0660`. Both are *layers*: uid 0 already passes the cred check, so the real auth is the API surface itself (see decisions). +- **Seq (monotonic per target):** writer is the source of truth. Persisted as a sidecar `.ver` (decimal, atomic, written *after* the rename). Commit order: payload → fsync → rename → sidecar. A crash between rename and sidecar leaves the sidecar one step behind; a replay at the old seq is then accepted once — harmless because both targets are full-file writes (idempotent). Rollback protection has a documented ≤ 1-step window across a writer crash; anything older is rejected (`stale_seq`). +- **Locking:** the baseline's lifetime `flock` (`files.go` `ErrLocked`) becomes per-op, held by the writer; the collector's post-split load is a lock-free read (it is no longer the baseline "owner"). The writer-init bootstrap (below) re-imports an existing target without a sidecar. +- **Metrics:** `xtcp2_writerd_requests_total{op,result}`, `xtcp2_writerd_last_seq{target}` — exported through `GET_STATS` and re-registered by the collector (proxy), so vmagent scraping is unchanged and the writer keeps **no TCP listener**. +- **Writer self-confinement:** `pkg/sandbox` with write rules only on the two target dirs; `-healthcheck` mode probes the socket. In the fleet, the writer container adds the docker boundary on top (`network_mode: none`, `cap_drop ALL`, uid 65534) — see PR-13. + +--- + +## PR-11 tests + +Harness: `pkg/writerd` mirrors the injectable-`operations` seam from `baseline/files.go` (`operations` struct with `mkdirAll/temporary/rename/remove/...` funcs) plus an injectable peer-cred lookup. Pure tests (A–D, G) run on any platform; the fault-injection rows run the commit dance stepwise. + +### Test A — request validation (`TestWriterdValidate`) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| A1 | P | valid baseline write | 1 KiB valid wireRecord, seq 1 | accepted; file at configured target == payload, mode `0600`; reply OK | +| A2 | N | op whose target is not configured | op 2 against a writer started baseline-only (`-writerTarget baseline=…`) | rejected `unconfigured_target`; nothing written | +| A3 | N | attacker stuffs path-like bytes into the payload | path bytes inside the payload | no string in the envelope is ever resolved as a path: schema-invalid → `invalid_payload`; schema-valid → written verbatim to the one fixed target (the no-path-parameters property — there is no field to abuse) | +| A4 | B | payload exactly at the baseline cap | `len == baseline.MaxFileBytes` (65 536) | accepted | +| A5 | N | payload one byte over | 65 537 B | rejected `invalid` at `.Validate()` (`max_len 65536`) — the framing cap (B4) is the layer above it; nothing written | +| A6 | P | baseline payload with `Version: 1`, all fields | well-formed wireRecord | accepted | +| A7 | N | baseline payload not baseline JSON | `Version: 2` / missing fields / garbage | rejected `invalid_payload` — writer-side schema check | +| A8 | N | seq 0 | any write op | rejected `invalid` at `.Validate()` (`gte 1`) — before the seq filter ever sees it | +| A9 | N | replay: seq == last seen | two requests, same seq | first accepted, second rejected `stale_seq` | +| A10 | N | rollback: seq < last seen | write at last-5 | rejected `stale_seq` | +| A11 | B | first request after writer-init import | target pre-exists, sidecar imported, seq 1 | accepted | +| A12 | C | two writes, same seq, delivered back-to-back | serialized requests | exactly one accepted, one `stale_seq`; exactly one rename observed | +| A13 | B | ipmeta payload exactly at the cap | `len == ipmetaMaxBytes` | accepted | +| A14 | N | ipmeta payload cap+1 | `len == ipmetaMaxBytes+1` | rejected `invalid` at `.Validate()` (`max_len`) | +| A15 | C | bootstrap: target exists, no sidecar | writer-init run first | init imports (sidecar `0`), writer starts, write at seq 1 accepted | +| A16 | N | zero-length payload | op 1, `payloadLen 0` | rejected `malformed` | + +### Test B — framing (`TestWriterdFraming`) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| B1 | P | encode/decode round trip | valid frame with payload | parsed fields identical | +| B2 | N | truncated header | 16 of 17 header bytes | `malformed`, connection closed, no write attempted | +| B3 | N | unknown op | op `0xFF` | error reply `unknown_op` | +| B4 | N | payload length lies huge | `payloadLen 0xFFFFFFFF` | rejected against the global cap **before allocation** (no OOM), connection closed | +| B5 | C | payload shorter than declared, then EOF | header says 100 B, sender sends 50 and closes | `malformed`; no partial file written | +| B6 | C | trailing bytes after payload | valid frame + 4 junk bytes | `malformed` (strict framing) | +| B7 | P | empty-payload op | `GET_STATS` frame | valid, stats reply | +| B8 | B | error reply at max size | err text 1024 B | encoded fine; collector decodes it | +| B9 | N | envelope violates protovalidate: seq 0 | `WriteBaselineRequest{seq: 0}` | rejected `invalid` at `.Validate()` (writer side; the collector side rejected it before send — same rules, both ends) | +| B10 | N | protovalidate per-op cap: baseline payload 65 537 B | hand-built envelope over `max_len` | rejected `invalid` at `.Validate()` — layered under the framing cap (B4) | +| B11 | N | unknown/forward-incompatible field in the envelope | hand-crafted frame with field 99 set | rejected `malformed` — explicit `GetUnknown() == nil` check, fail closed (protobuf-go stores unknown fields by default) | +| B12 | P | both-ends validation round trip | valid request through collector-side `.Validate()` → marshal → writer unmarshal → `.Validate()` | both sides pass; mutating any bound field flips exactly the writer side (the layering is observable) | + +### Test C — peer auth (`TestWriterdPeerAuth`) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| C1 | P | peer uid equals the collector uid | injected cred uid 0 | request served | +| C2 | N | peer uid not allowed | injected cred uid 1234 | rejected at handshake with `auth` error; counter `{op,result="auth"}` +1; nothing written | +| C3 | C | wrong-uid peer blocked by file perms first | socket `0660 root:root`, dir `0750`, peer uid 1234 | connect fails `EACCES` before any protocol bytes | +| C4 | N | writer never listens on TCP | server constructed, inspect listener calls | only `unix` network listeners; any config attempt to add TCP is a startup error | +| C5 | C | peer-cred lookup fails | injected lookup returns EBADF | connection closed, counter `auth`, writer stays up | + +### Test D — client resilience (`TestWriterdClient`, fake server) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| D1 | P | happy-path save | fake server replies OK | `SaveResult` with no error; request bytes match A1's shape | +| D2 | N | writer down at collector startup | no socket | collector still starts; first save fails with a typed error; `xtcp2_writer_requests_total{result="error"}` +1; no crash | +| D3 | B | slow writer beyond client timeout | fake server delays | deadline honored, error surfaces, no goroutine leak (leakcheck) | +| D4 | C | connection drops mid-request, then writer recovers | fake server closes once | client errors, reconnects on next attempt, second save succeeds | +| D5 | P | stats proxy | fake server serves counters | collector re-exports them under the `xtcp2_writerd_*` names | +| D6 | P | seq bootstrapping | `GET_STATS` returns last seq 7 for `baseline` | collector's next save uses seq 8 (writer stays the source of truth across collector restarts) | + +### Test G — crash safety (`TestWriterdCrashSafety`, injected faults) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| G1 | P | fault after temp write, before rename | injected `rename` error | target unchanged (old bytes); sidecar unchanged; temp file remains until sweep | +| G2 | P | fault mid-rename is atomic | injected `rename` failure — rename is atomic in the kernel | target is old or new, never partial | +| G3 | C | fault after rename, before sidecar | injected `remove`/sidecar error | target holds NEW bytes, sidecar one behind; next request at last+1 accepted (documented replay window) | +| G4 | P | temp sweep | stale temp file (mtime > TTL) + a fresh one | stale removed on next request; fresh kept | +| G5 | N | write denied by Landlock (misconfigured rule) | injected EACCES on temp create | error reply, writer stays up, counter `denied`; nothing written | +| G6 | C | disk full | injected ENOSPC at temp write | error reply, target intact, writer stays up | + +--- + +## PR-12 — Collector split and config shrink + +**Objective:** the collector writes through the client; the write-side path fields leave its settable config; its Landlock ruleset drops the write roots. + +Seams (verified in code — both are single call sites): + +- **Baseline:** implement `model.BaselineStore` with a client-backed `Save(ctx, record) (SaveResult, error)` — `storageExecutor` (`lifecycle_store.go`) consumes it unchanged; `Load` stays fs-backed read-only. +- **ipmeta:** the refresh loop's call at `enrich_asn.go:206` (`PublishLookupCache(cachePath, previousLookupPath(cachePath), artifact, before)`) becomes: build the projection bytes collector-side (same `writeRowsZstd` output + `validateCacheCandidate`), then `client.WriteIpmeta(payload, seq)`; the writer does the file dance. + +Config shrink (on top of Phase 1's frozen-field rejection): + +| Field | Before Phase 3 | After Phase 3 | +|---|---|---| +| `link_monitor.baseline_file` | frozen; read + write target | read path only — writes route to the writer's `baseline` target; value must equal the writer's target (startup check, mismatch is fatal) | +| `ipmeta_cache_path` | frozen; read + write target | read path only, same treatment | +| `WRITABLE_DIR` env (fleet) | state dir write root | removed from the collector — zero file-write Landlock rules | + +Collector post-split ruleset delta (from the [Phase 2 table](phase-2-kernel-confinement.md#the-ruleset)): drop every write-root row; add `READ_FILE|WRITE_FILE` on `/run/xtcp2/writer.sock` (UDS connect grant — the only write-classified access left; `open(2)` on a socket fails regardless) and, only where a collector UDS listener is configured, `MAKE_SOCK|REMOVE_FILE` on its listen dir. The fleet's collectors are TCP-only, so the shipped ruleset has no make/remove bits at all. + +## PR-12 tests + +### Test E — collector integration (`TestCollectorSplit`, fake writer) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| E1 | P | baseline save round trip | trigger a learn/save via client | writer receives bytes that decode to a wireRecord equal to the record; `SaveResult` OK | +| E2 | P | baseline load unchanged | file on the read path | load succeeds without contacting the writer; lock-free read does not conflict with a concurrent writer op | +| E3 | N | `Set` changes `baseline_file` | runtime `Set` | frozen-rejected (Phase 1 regression re-run, now against the shrunken config) | +| E4 | N | `Set` changes `ipmeta_cache_path` | runtime `Set` | frozen-rejected | +| E5 | P | `Set` of ordinary tuning fields with the split on | e.g. poll cadence | applies normally, soft restart survives (writer socket reconnects) | +| E6 | P | ASN publish round trip | fake writer | collector sends projection bytes; writer's dance called once with `preserve-previous`; next load reads the new cache; previous path holds the old cache | +| E7 | N | writer down during ASN refresh | socket absent | refresh errors, counter, collector serves from in-memory cache; next refresh recovers | +| E8 | C | first-ever startup, no baseline file | empty state dir | collector proceeds without baseline; writer-init creates dir + sidecar (F6 covers the ops step) | +| E9 | N | hostile `file:` `Set` replayed | `dest:"file:/proc/1/root/root/.ssh/authorized_keys"` | rejected by validation (Phase 1) — and with the ruleset flip, nothing writable exists anyway (cross-layer regression row) | +| E10 | C | collector restarts mid-session | restart between two saves | seq continues from `GET_STATS` (D6), no stale-seq rejection of the first save | + +--- + +## PR-13 — Deployment and the `:ro` flip + +**Objective:** the fleet collector container ends with no writable host path; the writer is the only thing in the deployment that can write, and only to the state dir. + +Container pair (deployment role, mirroring the existing `ipfeed` second-container pattern): + +| | `xtcp2` (collector) | `xtcp2_writer` (writer) | +|---|---|---| +| uid | 0 (unchanged — keeps `NET_ADMIN`/`SYS_ADMIN`) | 65534 | +| capabilities | `cap_drop ALL` + NET_ADMIN, SYS_ADMIN (unchanged) | `cap_drop ALL` — **none added** | +| network | host (unchanged, ports 9088/8889/9089) | `network_mode: none` — no interfaces, no egress at all | +| mounts | as today, **minus** the rw state dir (kept `ro` for loads) + socket volume | `/var/lib/xtcp2:rw` (the only writable mount), socket volume, nothing else — no docker.sock, no netns, no S3 secret | +| Landlock | zero file-write rules | write rules on the two target dirs | +| command | unchanged | `xtcp2 -writer -sock /run/xtcp2/writer.sock -writerTarget baseline=... -writerTarget ipmeta=...` | + +Socket sharing: a docker named volume mounted rw in both containers, containing only the socket dir (`0750`) and socket (`0660`). Whether the collector's mount can be `ro` (connect(2) on a UDS under an ro mount) is decided by F-table row F8 — if it works, the collector ends with **zero rw mounts, full stop**. + +Alert rules (vmalert + `promtool test rules` cases): `Xtcp2WriterDown` (client-side `rate(xtcp2_writer_requests_total{result="error"}[5m]) > 0`, sustained), `Xtcp2WriterSeqStuck` (last seq unchanged while link churn is high — silent save failure), plus `Xtcp2SandboxInactive` extended to both containers. + +Rollout order — each step is independently shippable and reversible: + +1. **Image bump** (contains `-writer` mode; behavior unchanged — no container uses it yet). +2. **Writer container added** (idle; collector still writes directly; `WRITABLE_DIR` still set). +3. **Collector switch**: `XTCP_WRITER_SOCK` env set → client mode on (fallback: unset → direct write, old behavior). +4. **`:ro` flip**: remove `WRITABLE_DIR` from the collector, state dir `ro` in the collector container — after a canary soak proves writer mode end-to-end. + +## PR-13 tests (microVM) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| F1 | P | writer up, pair healthy | fresh deploy | socket `0660` in dir `0750`; handshake OK; both processes report `xtcp2_sandbox_active 1`; writer `/proc/self/status` `CapEff: 0` | +| F2 | P | collector has no writable host path | `docker inspect` + crippled build (Phase 1 policy compiled out) attempting a write anywhere | kernel denies every file write outside the socket volume; `Set` with a hostile path applies but writes fail `EACCES` | +| F3 | N | writer container killed mid-operation | `docker kill` writer | collector keeps collecting (netlink path unaffected), error counter rises, `Xtcp2WriterDown` fires within one cycle; recovery on restart | +| F4 | P | baseline lands end-to-end | induce a baseline learn on the canary | `/var/lib/xtcp2` file exists, owner `65534:65534`, mode `0600`, contents == wire JSON | +| F5 | P | ASN publish rotates end-to-end | trigger ASN refresh | cache written by writer; previous path holds the prior cache; collector's next load serves the new one | +| F6 | C | fresh volume, sidecar missing | first boot on a new host | writer **refuses to start** (fail-closed) with a one-line runbook message; after the documented init invocation (imports existing targets, writes sidecars) it starts and serves | +| F7 | N | hostile `Set` payloads replayed | `dest` `file:` targets from advisory §4.7 | validation rejects; nothing is written anywhere — kernel gate proof (E9's microVM twin) | +| F8 | C | collector socket mount `ro` | mount the named volume `ro` in the collector container | document the verified answer: if connect(2) succeeds → ship `ro` (zero rw mounts); if `EROFS` → keep `rw`, note the socket volume's contents are one socket file (DoS-only loss) | +| F9 | C | soft restart with the split on | ordinary tuning `Set` | exec + reconnect clean; one save after restart succeeds (D6/E10 twin at deploy level) | + +## Appendix A — the write inventory and the target wish-list + +**A. Everything the daemon process can write today** — re-audited against current main by grepping `os.Create / OpenFile / WriteFile / CreateTemp / Rename / MkdirAll / Symlink / Remove / Chmod` across `pkg/` and `cmd/`, then classifying every hit: + +| # | Site | Reachable via | Writes what, where | Class | Disposition | +|---|---|---|---|---|---| +| 1 | `pkg/xtcp/destinations_file.go:28` | **`Set`** (`dest:"file:"`) | records, `O_APPEND`, arbitrary path, `0600` | debug export | PR-1 `dest_file` build tag; out of prod flavor | +| 2 | `pkg/xtcp/netlinker.go:84` | **`Set`** (`write_files` + `capture_path`) | netlink capture dump, arbitrary path | debug | PR-1 `capture` build tag | +| 3 | `pkg/linkmonitor/internal/baseline/files.go` | internal (linkmonitor learn) | baseline JSON, ≤ 64 KiB, temp+rename | **state** | **writer op `WRITE_BASELINE`** | +| 4 | `pkg/ipasn/ipasn.go:373-439` | internal (ASN refresh) | ipmeta cache + previous, temp+rename+syncDir | **state** | **writer op `WRITE_IPMETA`** | +| 5 | `pkg/listener/listener.go:149,165,214` | **`Set`** (listener config) | UDS bind / chmod ≤0777 / unlink stale — sockets, not files | sockets | narrow Landlock grants (`MAKE_SOCK\|REMOVE_FILE` on the listen dir); fleet listeners are TCP-only | +| 6 | `pkg/linkmonitor/internal/linuxio/record_linux.go:149` | env `LINKMONITOR_NLRECORD=` at **deploy** time — not `Set`-reachable (env survives the soft restart unchanged but no `Set` field touches it) | `--.pcap` netlink recordings, `O_EXCL`, into the env dir | debug recording | stays env-gated; Landlock requires the dir under a writable root; never set in the fleet | +| 7 | `pkg/linkmonitor/internal/rdmacapture/manifest.go` | standalone `rdma-capture-manifest` tool only | manifest + payloads | analysis artifact | not linked into the daemon; no change | + +Non-writes verified: `pkg/misc/misc.go:88,116` are `O_RDONLY` config loads; the other `cmd/*` binaries (`zstd-probe`, `clickhouse_*`, `kafka_to_clickhouse`) are standalone tools writing their own dumps — separate mains, not the daemon. Rows 1–5 are the advisory's five `Set`-reachable primitives; rows 6–7 complete the picture for **all** daemon-process writes. Landlock treats them identically regardless of reachability: no grant, no write. + +**B. The wish-list — every plausible "where would someone want to write" and the answer:** + +| Need | How it is served | Verdict | +|---|---|---| +| baseline state | writer `WRITE_BASELINE` | exists | +| ipmeta cache | writer `WRITE_IPMETA` | exists | +| local record export (the `file:` need) | `stdout`/`stderr` dests + redirect; kafka/nats/nsq/valkey/s3parquet dests | dev builds only; if a fleet need ever materializes: a **named journal op** with fixed path + cap + rotation — never a generic write | +| netlink pcap capture (`capture_path`, `LINKMONITOR_NLRECORD`) | debug image + writable root, time-boxed, Landlock-scoped | never a writer op — chatty and unbounded by nature | +| node_exporter textfile snapshot | vmagent scrapes the daemon's metrics port directly | refuse; if ever genuinely needed: small fixed-file writer op (cap ~64 KiB) | +| config persistence / snapshot | fetch via the gRPC config API | refuse — the API is the interface, files are not | +| log files | stdout/stderr → docker/journald | refuse file logging | +| core dumps | host `core_pattern` + ulimits (container: ulimit 0) | **never daemon-settable** — this is the classic `/proc/sysrq-trigger`/`core_pattern` escape idiom | +| state export / backup | host-side ops (ansible/tar); the dir is writer-owned (uid 65534) | runbook item, not a daemon write | +| credentials / certs | read-only mounts | the daemon never writes credentials | +| Prometheus / vmagent `file_sd` | vmagent's own directory | refuse — separation of duties | + +**C. Admission checklist for a new write target** — the rule that keeps the property: + +1. a named oneof arm in `writer.proto` — the envelope gains no strings +2. fixed path from the writer's own `-writerTarget` flags, never from a request or `XtcpConfig` +3. size cap via protovalidate `max_len` + a semantic schema check where one exists (baseline's `wireRecord` parse is the model) +4. `seq` monotonic + sidecar + the atomic commit dance +5. Landlock grant scoped to that target's directory only +6. an alert watching the target +7. a row in table A above, code review, and both containers redeployed + +Adding a write destination is an engineering decision with review and a redeploy — **never a runtime toggle**. Contrast with today: `Set` flips `dest` to any path on the host in a single unauthenticated call. + +## Decisions to close during Phase 3 + +| # | Decision | Recommendation | +|---|---|---| +| OD-4 (socket auth strength) | `SO_PEERCRED` uid alone vs + shared secret | ship uid + file perms this phase (API surface is the real auth); add a 0400 shared-secret file mounted ro in both containers if review wants the uid ambiguity closed | +| new | writer IPC format: gRPC over UDS vs proto envelope + tiny frame | **proto envelope + tiny frame** ([rationale](#wire-format-and-the-protocol-decision)) — protovalidate both ends, no HTTP/2 stack on the writer; gRPC variant only behind its hardening checklist if the team standardizes on one IPC mechanism | +| new | ipmeta payload cap default | 64 MiB (measured bootstrap × headroom) | +| new | sidecar ordering (rename→sidecar) | accept the ≤ 1-step replay window across a writer crash (full-file writes make replays idempotent); document in the runbook | +| new | baseline `flock` ownership | writer holds per-op; collector reads lock-free; `ErrLocked` now means "two writers" (misconfig, fatal in writer) | +| new | socket volume `ro` in collector | decided by F8 | +| new | writer seq source of truth | writer (via `GET_STATS`); collector never persists seq | + +## Release checklist + +- [ ] PR-11: tests A–D, G green (pure, any platform); protocol code reviewed line-by-line (it is the entire attack surface) +- [ ] PR-12: tests E green; `enrich_asn.go` seam and `BaselineStore` implementation reviewed for one-call-site fidelity +- [ ] Metrics: `xtcp2_writer_*` (collector) + `xtcp2_writerd_*` (proxy) in [metrics.md](../metrics.md) +- [ ] PR-13: container pair deployed to canary; F1–F9 executed and recorded (F8's answer folded into the role) +- [ ] `promtool test rules` cases for `Xtcp2WriterDown`/`Xtcp2WriterSeqStuck` merged with the role +- [ ] Rollout steps 1–4 run in order on the fleet with a soak between 3 and 4; rollback = unset `XTCP_WRITER_SOCK` + restore `WRITABLE_DIR` (documented in the runbook) + +## Definition of done + +From [implementation-plan.md](implementation-plan.md#phase-3--privilege-separation-writer-split): *the collector container holds no write capability to any host path — the `:ro` flip lands.* + +On a canary pair running a **deliberately crippled build** (Phase 1 path policy stubbed out): + +- [ ] `docker inspect xtcp2` shows the collector's only rw mounts are container-internal (`/tmp` tmpfs) and the socket volume (or, per F8, `ro` there too); the state dir and rootfs are `ro` +- [ ] the writer is the only process in the deployment able to write, and only to its two configured targets; `CapEff 0`, no network interface, uid 65534 +- [ ] `Set` with `dest:"file:/proc/1/root/root/.ssh/authorized_keys"` applies (validation stubbed) → the write fails at the kernel — no authorized_keys file appears on the host; same for every other path primitive +- [ ] a baseline learn lands in `/var/lib/xtcp2` via the writer (owner `65534`, `0600`); an ASN refresh rotates the cache + previous files via the writer +- [ ] killing the writer degrades, never breaks: collector keeps collecting, `Xtcp2WriterDown` fires, everything recovers on restart +- [ ] the writer protocol surface is exactly the 4 ops: a fuzzer against the socket (1 h, dumb mode) produces no write outside the two targets, no crash, no OOM, no leaked fds +- [ ] alert runbook covers: writer down, sidecar init on fresh hosts, socket volume loss (recreate + restart pair) diff --git a/docs/security-hardening/phase-4-egress-fleet-lockdown.md b/docs/security-hardening/phase-4-egress-fleet-lockdown.md new file mode 100644 index 00000000..74939678 --- /dev/null +++ b/docs/security-hardening/phase-4-egress-fleet-lockdown.md @@ -0,0 +1,181 @@ +# Phase 4 — Egress and fleet lockdown (drill-down) + +| | | +|---|---| +| **Status** | Not started — checkboxes track execution | +| **Tracks** | [Implementation plan](implementation-plan.md) Phase 4 → [hardening plan](plan-configservice-set-hardening.md) W7 (egress pinning, v1.x retirement) + W8 (standing gates) → [advisory](security-vulnerability-configservice-set-fs-write.md) layer L6 | +| **Goal** | A stolen admin token can only tune poll knobs; everything else requires a redeploy, and the network layer enforces the egress set | +| **Prereqs** | Phases 1–3 shipped (frozen fields to remove, writer split to leave alone, sandbox to extend with network rules) | + +## Contents + +- [Scope](#scope) +- [PR map and branches](#pr-map-and-branches) +- [PR-14 — close the escape hatch](#pr-14--close-the-escape-hatch) +- [PR-15 — egress pinning](#pr-15--egress-pinning) +- [PR-16 — retire the v1.x revision](#pr-16--retire-the-v1x-revision) +- [PR-17 — standing CI gates](#pr-17--standing-ci-gates) +- [Tests](#tests) +- [Decisions to close during Phase 4](#decisions-to-close-during-phase-4) +- [Release checklist](#release-checklist) +- [Definition of done](#definition-of-done) + +## Scope + +After Phases 1–3 the `Set` surface is: path fields rejected, writers not compiled, kernel denies writes, collector has no write capability at all. What remains exposed: + +| Residual exposure | Owner | Fix here | +|---|---|---| +| `dest` can still be repointed to **any network scheme the binary compiled in** by anything that can write config — today that includes the always-compiled stdlib destinations `tcp`, `udp`, `http`, `https`, `unix`, `unixgram` (the exfil class) | PR-15 | +| The frozen-field escape hatch (`-setAllowFrozenFields`) is still a flag away from undoing Phase 1's central control | PR-14 | +| The v1.x fleet revision — unauthenticated `:8889` — may still be running on some hosts (I10) | PR-16 | +| A compromised collector can dial arbitrary hosts (kernel does not yet constrain egress) | PR-15 | +| Nothing structurally stops a future PR from reintroducing a write site, a flavor regression, or a protovalidate gap | PR-17 | + +The verification target is the [end-state verification checklist](plan-configservice-set-hardening.md#end-state-verification-checklist) from the hardening plan — executed **holding the admin token**. + +## PR map and branches + +``` +PR-14 sec/w6-hatch-close (xtcp2) ─┐ +PR-15 sec/w7-egress (xtcp2 + image + deployment-repo + SG) ─┼─▶ PR-17 sec/w8-standing-gates (CI) +PR-16 sec/w7-retire-v1x (deployment-repo) ─┘ +``` + +--- + +## PR-14 — Close the escape hatch + +**Objective:** frozen fields ([phase 1 PR-3](phase-1-code-hardening.md)) become permanent; OD-6's migration window closes. + +- Remove `-setAllowFrozenFields` from `cmd/xtcp2/xtcp2.go` and the `Set` handler branch that honors it; frozen-field rejection becomes unconditional. +- Before removal, one release cycle of `xtcp2_set_applied_total{outcome="frozen_allowed"}` == 0 across the fleet (the PR-5 metric proves nobody used the hatch) — the fleet-gate test asserts it. +- Docs: [operations.md](../operations.md) loses the hatch paragraph; the hardening plan's OD-6 row closes. + +## PR-15 — Egress pinning + +**Objective:** the DEST-repoint exfil class becomes structurally impossible and the kernel plus cloud layer enforce the egress set — three independent layers: + +**Layer 1 — build (closes OD-9, decision [recorded](plan-configservice-set-hardening.md#open-decisions): yes).** Gate the network stdlib destinations behind the established `dest_` build-tag pattern: `dest_tcp`, `dest_udp`, `dest_http` (covers `https`), `dest_unix`, `dest_unixgram` on `pkg/xtcp/destinations_{tcp,udp,http,unix,unixgram}.go` — today all five are **unconditional** (`knownSchemes`, `destinations_core.go:70`). `nix/versions.nix` flavor lists drop them from the production flavor (the fleet image is `oci-xtcp2-s3parquet-enrich`, which already excludes kafka/nats/nsq/valkey); `full` and the debug flavor keep them. Production needs `s3parquet` + `null`/`stdout` only. The existing `destinationLookupError` behavior (fatal at startup when a requested scheme is not compiled in; `FailedPrecondition` from `Set`) means a repointed config **cannot start**, fleet-wide. + +**Layer 2 — kernel (Landlock network rules, ABI 4; the Phase 2 deferred decision).** The collector's `pkg/sandbox` ruleset gains TCP network rules: `CONNECT_TCP` scoped to port 443, `BIND_TCP` scoped to the listener ports (9088, 8889, 9089). Honest limits, documented in-code: Landlock net rules are **port-scoped, not host-scoped** (a 443 listener anywhere passes the kernel check) and ABI 4 does not cover UDP — DNS keeps working and UDP egress stays governed by Layer 3. Degradation mirrors Phase 2: ABI < 4 → warn + `xtcp2_sandbox_net 0` + continue. The writer needs nothing — its container is `network_mode: none` (Phase 3). + +**Layer 3 — cloud/host (deployment + infra repos).** Security groups: host egress restricted to the per-DC S3 endpoints and the operator VPN; UFW OUTPUT rules reviewed with the monitoring lockdown role. This is the only layer that constrains by **host**, closing Landlock's port-only gap. Detection: the host Falco pipeline gains an alert for outbound connections to non-S3 hosts from the collector's uid. + +**Also here (closes OD-7):** state-dir quota — XFS project quota on `/var/lib/xtcp2` for the writer container, plus a small writerd addition from Phase 3: per-target rate cap (default 12 writes/hour/target — a legitimate baseline learns a handful of times an hour) so the seq monotonicity can't be ground into disk fill by an attacker with a valid request shape. + +## PR-16 — Retire the v1.x revision + +**Objective:** the unauthenticated revision stops existing (I10 permanent). + +- Role defaults make disabling listener auth impossible: `LISTENER_AUTH_MODE` and the token mounts become non-overridable role invariants (fail the play if unset/empty); the fleet-gate Molecule test asserts the auth env + token file presence on every host. +- Migrate any stragglers, then retire the v1.x revision branches and their image tags. +- `:8889` exposure: keep raw-token auth (v2.0.x behavior) and add SG/UFW scoping to operator subnets only — the admin port becomes unreachable from anywhere else. + +## PR-17 — Standing CI gates + +**Objective:** the properties of this whole program cannot silently regress. Gates (all wired as required checks): + +1. **Write-site inventory drift** — CI greps the same write-syscall classes as [Appendix A](phase-3-writer-split.md#appendix-a--the-write-inventory-and-the-target-wish-list) across `pkg/`, diffs against a committed allowlist; a new write site fails the build with a message pointing at the Appendix A admission checklist. +2. **Flavor/destination matrix** — builds each nix flavor and asserts compiled/not per scheme against a committed matrix (the K-table below is its spec); `nix flake check` drift guards in the image repo extend to it. +3. **Golden-config validation** — [phase 1 PR-6](phase-1-code-hardening.md)'s `-validateConfig` runs against ansible-rendered golden configs for **both** repos' current revisions on every role change, not just daemon changes. +4. **Advisory regression suites** — the table-driven suites from phases 1–3 (`TestSandbox*`, `TestWriterd*`, `TestCollectorSplit`, `TestConfigService_Set_*` hostile-path rows) are required checks by name. +5. **Scheduled end-to-end proof** — weekly workflow runs the crippled-build scenario (validation stubbed): all five primitives fail at the kernel, egress repoint fails at build, in the [definition of done](#definition-of-done) shape. +6. **Sandbox-active assertion** — the smoke test asserts `xtcp2_sandbox_active == 1` (skip-aware for runner kernels < 5.13; current hosted runners are ≥ 6.5). + +## Tests + +Table-driven, same conventions (Cat: P positive, N negative, B boundary, C corner). J/K/N are pure or CI-level (any platform); L is the Phase 2 child-process harness extended with network rules (Linux, ABI ≥ 4, skip-aware); M is microVM + cloud (executed and recorded, not automated). + +### Test J — hatch removal (`TestSetFrozenHatchClosed`, pure) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| J1 | N | `Set` changes a frozen field with no flag in play | any frozen path/dest field via admin token | rejected `FailedPrecondition`, frozen forever — no flag exists to allow it | +| J2 | N | operator passes the removed flag | `-setAllowFrozenFields=true` on the new binary | `flag.Parse` fatal, unknown flag (loud, not silent ignore) | +| J3 | C | old binary + new config that used the hatch | config rendered for a hatch migration | startup validation failure naming the frozen field — redeploy is the only path (documented in the runbook) | +| J4 | P | fleet-gate pre-check | `xtcp2_set_applied_total{outcome="frozen_allowed"}` over the last release cycle | zero across the fleet — the PR-14 precondition assertion | +| J5 | B | ordinary tuning `Set` after removal | poll cadence change | applies + soft restart normally (removal touched nothing else) | + +### Test K — flavor/destination matrix (`TestDestinationFlavorMatrix`, per-flavor builds) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| K1 | P | `full` flavor resolves everything | lookup all 15 `knownSchemes` | all resolve to factories | +| K2 | P | production flavor (s3parquet-enrich) keeps what it needs | lookup `s3parquet`, `null`, `stdout`, `stderr` | all resolve; `libraryDefaultDest` auto-default still selects `s3parquet` | +| K3 | N | production flavor drops the exfil class | lookup `tcp`, `udp`, `http`, `https`, `unix`, `unixgram` in the prod flavor | none compiled: config requesting one → startup fatal; runtime `Set` → `FailedPrecondition` (the `destinationLookupError` two-severity path) | +| K4 | N | production flavor still drops library dests | lookup `kafka`/`nats`/`nsq`/`valkey` | not compiled (pre-existing flavor gates unchanged — regression row) | +| K5 | B | `min` flavor per OD-8 | lookup `file` | not compiled (OD-8: dropped from min; only `full`/debug carry `dest_file`) | +| K6 | C | unknown scheme entirely | `dest: "foo://x:1"` in any flavor | startup fatal (existing behavior preserved, not converted into a `Set`-time-only error) | +| K7 | C | a host upgraded with a legacy config naming a now-uncompiled scheme | old config with `dest: "tcp://..."` on new image | startup fails loudly with the not-compiled message — the fleet upgrade runbook covers the config fix (no silent destination change) | + +### Test L — Landlock network rules (`TestSandboxNetwork`, child-process harness) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| L1 | P | TCP connect to an allowed port | dial `127.0.0.1:443` (local listener) | connect succeeds | +| L2 | N | TCP connect to a non-allowed port | dial `127.0.0.1:8080` | `EACCES` at connect — an `http://attacker:8080` repoint, kernel-denied | +| L3 | P | TCP bind on a listener port | bind `:8889` in the child | bind succeeds (ruleset grants `BIND_TCP` on the listener ports only) | +| L4 | N | TCP bind on a foreign port | bind `:44444` | `EACCES` — a smuggled listener cannot open a C2 port even with root | +| L5 | C | ABI < 4 kernel | probed network ABI 0–3 | warn + `xtcp2_sandbox_net 0` + continue (degrade, never a failed daemon — mirrors D4/D5) | +| L6 | C | UDP is untouched by ABI 4 rules | DNS lookup through `/etc/resolv.conf` resolver | succeeds — documented gap, governed by Layer 3 (SG) instead | +| L7 | P | UDS connect unaffected by TCP rules | dial the writer socket | succeeds (network rules are TCP-only; FS grants govern sockets) | +| L8 | C | seccomp-blocked `landlock_add_rule` for net rules | injected EPERM at net-rule add | FS ruleset still applied; warn specifically about the missing net rules (`xtcp2_sandbox_net 0`, `xtcp2_sandbox_active 1`) | + +### Test M — fleet egress and exposure (microVM + security groups, recorded) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| M1 | P | S3 upload path intact | trigger ASN refresh on a canary | upload succeeds to the per-DC endpoint on 443 | +| M2 | N | arbitrary egress blocked at the host layer | `curl http://:443` and `:8080` from the collector container | denied by SG (443 non-S3 included — closes Landlock's port-only gap) | +| M3 | P | DNS intact | resolver query for the S3 endpoint | resolves (UDP 53 untouched) | +| M4 | P | metrics scrape intact | vmagent scrape of 9088 from the docker-bridge subnets | scrape succeeds | +| M5 | N | `:8889` from a non-operator subnet | connect from a random host | denied at SG/UFW | +| M6 | P | `:8889` from operator subnet with the admin token | authorized `Set` (tuning field) | applies, audited (PR-5 line shipped) | +| M7 | N | unauthenticated `Set` replay on the converged fleet | unauthenticated `Set` to `:8889`, v1.x shape | rejected (raw-token auth) — unauthenticated access is dead fleet-wide | +| M8 | B | state-dir quota enforced | writer spammed with schema-valid 64 KiB writes past the project quota | writes fail with quota errors once the limit is hit; `Xtcp2StateDirFilling` had already fired (alert precedes quota); collector unaffected | +| M9 | C | per-target writer rate cap | 20 valid baseline writes in an hour | first ~12 accepted, rest rejected `rate_limited` with counter — no disk-fill grind through the seq mechanism | + +### Test N — CI gates (each gate's failure mode) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| N1 | N | PR adds a write site without an inventory row | `os.WriteFile` added in `pkg/xtcp/foo.go`, allowlist untouched | write-site drift gate fails; failure message points at the Appendix A admission checklist | +| N2 | P | same PR with allowlist + inventory row | allowlist + Appendix A row updated | gate passes — the checklist is the process, the gate is the reminder | +| N3 | N | PR drops a protovalidate path pattern | `max_len`/pattern removed from a path field in `xtcp_config.proto` | golden-config gate fails on re-validation | +| N4 | N | PR changes nix flavors without updating the matrix | flavor list edited, matrix spec untouched | flavor/destination matrix gate fails | +| N5 | P | scheduled end-to-end proof | weekly crippled-build workflow | all five primitives fail at the kernel + egress repoint fails at build — green run archived as the program's standing receipt | +| N6 | N | regression suite deleted or renamed | a named required check's test file removed | CI fails (required-check-by-name), not silently skipped | + +## Decisions to close during Phase 4 + +| # | Decision | Recommendation | +|---|---|---| +| OD-9 | drop network stdlib destinations from the production flavor | **yes** (closes here): `dest_tcp/udp/http/unix/unixgram` tags; prod keeps `s3parquet` + `null`/`stdout` | +| OD-6 | hatch lifetime | closes with PR-14 after one clean release cycle (J4 proves zero use) | +| OD-7 | state-dir quota | XFS project quota + writerd rate cap (M8/M9) | +| new | Landlock net-rule scope | ship port-scoped 443-connect + listener-port bind; host-level pinning stays with SG (documented division: kernel covers ports, cloud covers hosts) | +| new | per-DC S3 endpoint list source of truth | the infra repo variables, consumed by SG + frozen S3 fields (single list, no drift) | +| new | egress alert owner | the host Falco pipeline | + +## Release checklist + +- [ ] PR-14 merged after J4 verified zero hatch use across a full release cycle +- [ ] PR-15 build side: five destinations tagged, flavor lists updated, image rebuilt; K1–K7 green +- [ ] PR-15 network side: SG changes canaried **one DC first** (the riskiest change in the program — a wrong rule breaks S3 uploads fleet-wide; rollback = revert SG), L1–L8 green where kernels allow, M1–M4 recorded +- [ ] PR-16: stragglers migrated, v1.x branches + image tags retired, M5–M7 recorded +- [ ] PR-17: all six gates wired as required checks + weekly scheduled proof; N1–N6 exercised with a deliberately-bad PR before merge +- [ ] [metrics.md](../metrics.md): `xtcp2_sandbox_net`, writerd rate/quota counters documented +- [ ] Rollback matrix from [implementation-plan.md](implementation-plan.md) updated: each Phase 4 layer rolls back independently (SG revert; flavor bump back; hatch re-add is a code revert, not a flag) + +## Definition of done + +The [end-state verification checklist](plan-configservice-set-hardening.md#end-state-verification-checklist), executed on a canary host **holding the admin token** — every primitive must fail safely at its layer: + +- [ ] **Filesystem:** `Set` with `dest:"file:…"` (any of the advisory §4.7 path idioms incl. `/proc/1/root/root/.ssh/authorized_keys`) → validation rejects; crippled build (validation stubbed) → kernel denies; nothing written anywhere on the host +- [ ] **State:** baseline + ipmeta land via the writer only, owned `65534:65534` `0600`; quota + rate cap enforce (M8/M9) +- [ ] **Egress:** `Set` with `dest:"http://attacker:8080"` → `FailedPrecondition`, not compiled in the prod flavor; crippled build + RCE-simulated dial → Landlock denies the port; SG denies the host; the outbound-connection alert fires — four layers, each independently sufficient +- [ ] **Frozen fields:** `dest`, S3 endpoints, listeners, all path fields → rejected unconditionally, no flag exists (J1/J2); a tuning-only `Set` still applies and audits +- [ ] **Auth:** unauthenticated `Set` on `:8889` rejected everywhere (M7); `:8889` unreachable off-operator subnets (M5); no v1.x host remains in inventory +- [ ] **Standing:** weekly crippled-build proof green (N5); a write-site PR fails CI until the inventory row exists (N1/N2); deleting a regression suite fails CI (N6) +- [ ] **Program:** every layer of the advisory's L0–L6 table has a ✅, each backed by a named required check — the hardening plan's end-state checklist signs off diff --git a/docs/security-hardening/plan-configservice-set-hardening.md b/docs/security-hardening/plan-configservice-set-hardening.md new file mode 100644 index 00000000..57918546 --- /dev/null +++ b/docs/security-hardening/plan-configservice-set-hardening.md @@ -0,0 +1,267 @@ +# Plan: hardening `ConfigService.Set` and the filesystem write surface + +| | | +|---|---| +| **Status** | Plan — not yet implemented; checkboxes track progress. Execution order, PR-level steps, and phase gates: [implementation-plan.md](implementation-plan.md) | +| **Tracks** | [security-vulnerability-configservice-set-fs-write.md](security-vulnerability-configservice-set-fs-write.md) (advisory, layers L1–L6) | +| **Repos** | `randomizedcoder/xtcp2` (this repo), the image repo (image), the deployment repo (deployment), the infra repo (SSM/IAM) | +| **Owner / review** | TBD | + +## Contents + +- [Goals and non-goals](#goals-and-non-goals) +- [Issue register](#issue-register) +- [Workstreams](#workstreams) + - [W0 — build-time elimination (L0)](#w0--build-time-elimination-l0) + - [W1 — protovalidate path rules (L1)](#w1--protovalidate-path-rules-l1) + - [W2 — runtime path policy + frozen fields (L2)](#w2--runtime-path-policy--frozen-fields-l2) + - [W3 — universal validating interceptor (L3)](#w3--universal-validating-interceptor-l3) + - [W4 — Landlock self-confinement (L4)](#w4--landlock-self-confinement-l4) + - [W5 — state-writer process split (L5)](#w5--state-writer-process-split-l5) + - [W6 — `Set` audit logging](#w6--set-audit-logging) + - [W7 — deployment guardrails (L6)](#w7--deployment-guardrails-l6) + - [W8 — tests, CI, and fleet-config gates](#w8--tests-ci-and-fleet-config-gates) + - [W9 — documentation](#w9--documentation) +- [Sequencing and dependencies](#sequencing-and-dependencies) +- [Fleet rollout runbook](#fleet-rollout-runbook) +- [End-state verification checklist](#end-state-verification-checklist) +- [Open decisions](#open-decisions) +- [References](#references) + +## Goals and non-goals + +**Goals** + +1. No gRPC input, however crafted, can make the daemon write outside operator-approved roots. +2. Every RPC input is validated — structurally, not by handler discipline. +3. The guarantees above survive a fully attacker-controlled `Set` call, a bypass of the validation code, and the soft-restart `syscall.Exec` — enforced by the kernel where possible. +4. Runtime config changes cannot repoint egress (`dest`, `s3_endpoint`) or listeners/auth without a redeploy. +5. Every applied `Set` is auditable. +6. The collector process ends up with **zero** filesystem write capability. + +**Non-goals** (tracked in the advisory's [honest limits](security-vulnerability-configservice-set-fs-write.md#7-honest-limits--what-this-does-not-fix)) + +- Containing the collector's `NET_ADMIN`/`SYS_ADMIN`/`setns` powers or the `docker.sock` mount — inherent to the product's function; a Docker-API proxy is a separate future design. +- Network egress confinement at the L7 layer (only the *config* surface is frozen here). +- The ipfeed sidecar's write path (it already runs as uid 65534 into one subdirectory; adopting the writer is optional follow-up). + +## Issue register + +| # | Issue | Where | Severity | Fixed by | +|---|---|---|---|---| +| I1 | `dest = "file:"` → create/append arbitrary 0600 root files | `pkg/xtcp/destinations_file.go:24` | High | W1, W2, W4, W5 | +| I2 | `write_files` + `capture_path` → raw netlink dumps to arbitrary paths | `pkg/xtcp/netlinker.go:70` | High | W1, W2, W4 | +| I3 | `baseline_file` → atomic write to arbitrary path | `pkg/linkmonitor/config.go:95`, `internal/baseline` | High | W1, W2, W4, W5 | +| I4 | `ipmeta_cache_path` → file write + `MkdirAll` arbitrary directories | `pkg/ipasn/ipasn.go:371` | High | W1, W2, W4, W5 | +| I5 | unix listeners bind arbitrary paths, chmod ≤0777 | `pkg/listener/listener.go` | Medium | W1, W2, W4 | +| I6 | No path/scheme checks anywhere in the `Set` validation chain | `pkg/xtcp/grpc_configService.go:186-262` | High | W1, W2 | +| I7 | Validation is per-handler (drift risk; empty requests unvalidated by convention) | all services | Medium | W3 | +| I8 | Runtime repoint of `dest`/`s3_endpoint` (exfil with preserved SigV4 keys) | `Set` | High | W2 | +| I9 | Auth off by default; wildcard `:8889` | defaults | High (deployment) | W7 | +| I10 | v1.x fleet role revision: gRPC admin surface unauthenticated | deployment role (v1.x revision) | Critical | W7 | +| I11 | No quota on `/var/lib/xtcp2` → host-rootfs disk-fill | ansible role | Medium | W7 | +| I12 | No audit trail of applied `Set` calls | `Set` handler | Medium | W6 | +| I13 | Collector holds write access it does not need (state-integrity/persistence risk under RCE) | architecture | Medium | W4, W5 | +| I15 | `file:` destination (stdlib, no build tag) and the netlink capture writer (no build gate) are compiled into **every** production image | `pkg/xtcp/destinations_file.go`, `pkg/xtcp/netlinker.go:70`, [build-flavors.md](../build-flavors.md) | High | W0 | + +## Workstreams + +### W0 — build-time elimination (L0) + +Advisory §5.7. Primitives #1/#2 live in code production never uses; remove it from the production binary. Fits the existing flavor axes ([build-flavors.md](../build-flavors.md)) — the `dest_` build-tag convention and the "not compiled in" error path already exist (`pkg/xtcp/destinations_core.go`). + +- [ ] Move the `file` destination behind a `dest_file` build tag: `destinations_file.go`'s `init()` registration becomes tag-gated; `knownSchemes` keeps `file` so the error path can say "not compiled into this binary; rebuild with `-tags dest_file`" (a config naming an uncompiled scheme is a **fatal startup error**, `destinationLookupError` — a restart loop, never a write) +- [ ] Gate the netlink capture writer behind a `capture` build tag (`netlinker.go` `captureToFileIfEnabled` + the `write_files`/`capture_path` wiring); `write_files > 0` without the tag = fatal startup error, mirroring the enricher convention +- [ ] Update `nix/versions.nix` flavor matrix + [build-flavors.md](../build-flavors.md): the production `s3parquet` flavor excludes both tags; `full`/`-debug` keep them; `xtcp2-min` per [OD-8](#open-decisions) +- [ ] Decide (OD-9) whether the production flavor also drops the network stdlib destinations (`tcp`, `udp`, `http`, `https`, `unix`, `unixgram`) — production needs `s3parquet` + `null`/`stdout`; makes the DEST-repoint exfil class structurally impossible +- [ ] Image repo: build + publish the new production flavor; deployment repo pin bump; fleet-gate test asserts the flavor rejects `dest file:` and `write_files` +- [ ] Docs: [output-and-destinations.md](../output-and-destinations.md) (`file:` now debug-flavor only), [build-flavors.md](../build-flavors.md), [grpc-api.md](../grpc-api.md) (`Set` behavior on uncompiled schemes) + +**Acceptance**: on the production image, `Set` with `dest = "file:/tmp/pwn"` is a fatal startup error — no file is ever created; `write_files = 1` likewise; debug flavors unchanged. +**Estimate**: 2–3 days (tags + nix matrix + image + role bump). **Risk**: a legitimate `file:`/`write_files` production use (none known — fleet uses `s3parquet`); golden-config gate (W8) catches any fleet config that breaks. + +### W1 — protovalidate path rules (L1) + +Belt layer 1: shape validation in `proto/xtcp_config/v1/xtcp_config.proto`, enforced by the existing `protovalidate.Validate` calls. No Go changes required for the rules to take effect (they run today), only regeneration + tests. + +- [ ] Add the segment pattern (blocks relative paths, `.`/`..` segments, empty `//`, NULs, spaces, control/unicode tricks — see advisory §5.1) as a shared comment + per-field `pattern` on: + - [ ] `capture_path` (51) + - [ ] `link_monitor.baseline_file` (LinkMonitorConfig field 2) + - [ ] `ipmeta_cache_path` (244), `ipmeta_bootstrap_path` (243), `asn_db_path` (241) + - [ ] `docker_socket_path` (202), `lldpd_socket_path` (211) +- [ ] `dest` (60): scheme allowlist pattern mirroring `knownSchemes` (`pkg/xtcp/destinations_core.go:70`): `^(null|stdout|stderr|file|tcp|http|https|udp|unix|unixgram|kafka|nats|nsq|valkey|s3parquet)://[^[:cntrl:]]+$` +- [ ] `ListenerEndpoint`: charset pattern on `address`; message-level CEL cross-rule — when `network == unix`, address must match the path pattern +- [ ] Tighten `unix_socket_mode` from `lte 511` → `lte 384` (0600) — pending [OD-1](#open-decisions) +- [ ] Regenerate all targets (`gen/go`, `gen/dart`, …) via the existing make/nix protobuf targets +- [ ] Golden-config gate: validate the **fleet's actual configs** (ansible role defaults + per-DC env, incl. the v1.x revision's) against the new rules in CI — catches any legitimate path we just broke, before release (part of W8) + +**Acceptance**: hostile-path table (W8) rejected at the proto layer; fleet golden configs pass. +**Estimate**: 2–3 days. **Risk**: over-tight pattern rejects a legitimate deployment → mitigated by the golden-config gate. + +### W2 — runtime path policy + frozen fields (L2) + +Belt layer 2: semantics that CEL cannot express (deployment roots, symlink race, running-config diff). This is the workstream that closes I1–I6, I8 outright. + +- [ ] New file `pkg/xtcp/set_path_policy.go`: + - [ ] `PathPolicy` struct: roots from a new repeatable daemon flag `-writableDir` (register alongside existing flags, `cmd/xtcp2/xtcp2.go` ~line 454); empty default = no filesystem-writing config allowed + - [ ] `ValidatePaths(cfg *xtcp_config.XtcpConfig) error`: for every path-bearing field — canonicalize (`filepath.Clean` + `filepath.EvalSymlinks`, reject on error), verify each component via `os.Lstat` is not a symlink, then require containment in a root + - [ ] Validate `dest` scheme against compiled-in `knownSchemes`; reject `file:` unless `-allowFileDest` is set +- [ ] Wire the policy: + - [ ] `Set` handler (`pkg/xtcp/grpc_configService.go`, after existing validation, before `reconfigureFunc`) — returns `codes.InvalidArgument` with the offending field named + - [ ] startup config load **and** `loadReconfigureConfig` (`cmd/xtcp2/xtcp2.go:1199`) — so a hostile env-carried config cannot bypass it after the soft-restart exec +- [ ] Frozen-field diff (new file `pkg/xtcp/set_frozen_fields.go`): compare incoming vs running config for `dest`, `s3_endpoint`/`s3_bucket`/S3 credential fields, kafka/nats/nsq/valkey options, both listener endpoints + auth settings, `docker_socket_path`, `lldpd_socket_path`, `asn_db_path`, `ipmeta_*`, `baseline_file`, `capture_path`, `write_files`, `xtcp_proto_file`; reject changes with `codes.FailedPrecondition` naming the frozen fields. Escape hatch: `-setAllowFrozenFields` (default false, documented as migration-only). Source of the running snapshot: the config captured at startup (same one `reconfigureFunc` records). + +**Acceptance**: unit tests below; a `Set` that changes any path field, `dest` scheme, or egress endpoint is rejected before restart; a `Set` that only changes poll knobs still applies. +**Estimate**: 3–5 days. **Risk**: operators legitimately using `Set` for dest migration → escape-hatch flag + rollout note in the runbook. + +### W3 — universal validating interceptor (L3) + +Belt layer 3: make "every input is validated" structural instead of per-handler discipline. + +- [ ] New file `pkg/xtcp/grpc_interceptors.go`: unary `validatingInterceptor` calling `protovalidate.Validate` on every request implementing `proto.Message`; return `codes.InvalidArgument` +- [ ] Streaming: wrap `RecvMsg` (validate each received message) for `FlatRecords`/`PollFlatRecords` — currently empty messages, cost nil, but future fields are covered automatically +- [ ] Install via `grpc.ChainUnaryInterceptor` where the server is constructed (`pkg/xtcp/grpc_server.go`) +- [ ] Descriptor-walking guard test: every RPC in every service either (a) rejects a crafted invalid request with `InvalidArgument`, or (b) has an empty request message — asserted from the descriptor, so a new RPC with fields cannot merge without a validation case + +**Acceptance**: guard test passes; removing any per-handler `validate()` call still results in rejected bad input (interceptor backstop). +**Estimate**: 2 days. **Risk**: double-validation cost on `Set` (cold path — negligible); measure `PollFlatRecords` stream overhead anyway (should be ~0; empty requests). + +### W4 — Landlock self-confinement (L4) + +Belt layer 4: kernel enforcement that survives `syscall.Exec` (soft restart) and cannot be escaped even by the privileged process itself. + +- [ ] Dependency: `github.com/landlock-lsm/go-landlock` (check nixpkgs availability; vendor otherwise) +- [ ] New package `pkg/sandbox`: + - [ ] `Constrain(opts)` called once at daemon startup (`cmd/xtcp2/xtcp2.go`), before any config-driven writer is constructed: read-only rules for `/proc`, `/sys`, `/run/netns`, `/run/docker/netns`, `/run/docker.sock`, the lldpd socket dir, ASN bootstrap/cache paths, listener token files; write rules only for `-writableDir` roots (until W5 lands, after which: none) + - [ ] Carve-outs: UDS listener dir (only when a unix listener is configured); capture dir (only when `write_files` > 0 with a root configured) + - [ ] ABI feature detection: kernel < 5.13 → log a warning, set `xtcp2_sandbox_active 0`, continue (degrade-not-exit); otherwise `1` — new Prometheus gauge (add to [metrics.md](../metrics.md) in W9) +- [ ] Soft-restart interplay test: restriction persists across `syscall.Exec` (unit test with a re-exec of a helper binary attempting an out-of-root write) +- [ ] microVM integration scenario: daemon running, `os.WriteFile` outside roots fails with `EACCES`; state-dir writes still succeed (pre-W5) / fail (post-W5) +- [ ] Optional extra belt (advisory §5.4): self-`pivot_root` confinement using the existing `CAP_SYS_ADMIN` (`chroot()` itself is impossible — `CAP_SYS_CHROOT` is dropped); carries the full bind-mount farm (`/proc` ro, `/run/netns`, `/run/docker/netns`, `/sys/fs/cgroup`, docker + lldpd sockets, state dir, tokens, ASN files, `/dev/null` + `urandom`, `/etc/resolv.conf` + `hosts` + `ssl/certs` for S3 TLS/DNS). Persists across the soft-restart exec. Landlock stays primary; prioritize for bare-metal runs where no container pidns defeats `/proc/1/root` (advisory §4.7) + +**Acceptance**: microVM green; gauge reports 1 on fleet kernels (Ubuntu 24.04, 6.8 — ABI ≥4). +**Estimate**: 3–5 days. **Risk**: unexpected write path in the daemon (e.g. runtime temp files) breaks under Landlock → run the full test matrix + a staging soak before fleet rollout; the gauge + log line makes misses visible in prod. + +### W5 — state-writer process split (L5) + +Belt layer 5: the collector ends up with zero write rules; the two legitimate writes move behind a path-free API. Full design in advisory §5.5. + +- [ ] New proto `proto/state_writer/v1/state_writer.proto`: `service StateWriter { rpc WriteBaseline(bytes) …; rpc PublishIpmetaCache(bytes) … }` — **no path parameters on the wire, ever** +- [ ] New daemon mode `xtcp2 -writer` (`cmd/xtcp2/writer.go`): + - [ ] gRPC server on UDS `/run/xtcp2/writer.sock`; `SO_PEERCRED` on accept, reject unexpected uids + - [ ] Fixed targets inside its root: `link-monitor-baseline.json`, `ipmeta/current.lookup.parquet.zst` (+ previous) + - [ ] Size caps (baseline ≤ 64 KiB = `baseline.MaxFileBytes`; cache ≤ 192 MiB), per-RPC rate limits, monotonic version numbers (reject rollback/replay), mode 0600 + - [ ] Reuse the existing atomic-write discipline (`pkg/linkmonitor/internal/baseline/files.go`): `CreateTemp` → write → `fsync` → `rename` → dir fsync; baseline `flock` moves inside the writer +- [ ] Collector side: + - [ ] `baseline` store: new RPC-backed implementation behind the existing injectable `temporary`/`rename` seam (`pkg/linkmonitor/internal/baseline/files.go:46`); collector reads the file directly through the ro mount + - [ ] `ipasn.PublishLookupCache` (`pkg/ipasn/ipasn.go:373`): route content through the writer RPC; keep the reload/rollback semantics + - [ ] Failure mode: writer unreachable → log + `xtcp2_state_writer_errors_total` counter + degrade-not-exit (monitor keeps last-known state) — matches today's behavior when the baseline is unwritable +- [ ] Deployment (W7): second container `xtcp2_writer` (same image, state dir `:rw`, socket dir shared, no network); collector container flips state dir to `:ro` and mounts the socket dir; systemd units `xtcp2-writer.service` (`RootDirectory=`, uid 65534, `cap_drop ALL`, `RestrictAddressFamilies=AF_UNIX`) + ordering for bare metal +- [ ] Remove `baseline_file`/`ipmeta_cache_path` from the `Set`-mutable surface entirely (they become writer-owned constants; the fields stay in the proto for compat but are frozen — W2 list already includes them) + +**Acceptance**: collector's Landlock ruleset has zero write rules; writer tests prove no request field can influence the target path (structurally: no such field exists); kill -9 mid-write leaves old baseline intact; version replay rejected. +**Estimate**: 10–15 days (proto + writer + collector seams + deployment + microVM). **Risk**: largest item — stage after W4 so the fleet is already protected; the `:ro` flip for the collector is the irreversible step (roll back by restoring the `:rw` mount). + +### W6 — `Set` audit logging + +- [ ] Log every applied `Set` at INFO with: caller token id (from listenerauth context), frozen-vs-changed field diff summary (field names + old→new for non-secret fields; never log values of `s3_*` credentials) +- [ ] Structured line format the log shipper can alert on (`event=set_applied …`) +- [ ] Metric `xtcp2_set_applied_total{frozen_rejected,applied}` so a stolen-token pattern (repeated `FailedPrecondition`) is visible on dashboards +- [ ] Advisory: `xtcp2_rotate_listener_tokens` runbook step on suspected exposure + +**Estimate**: 1–2 days. **Depends on**: W2 (field diff helper reusable here). + +### W7 — deployment guardrails (L6) + +Changes in the deployment repo (`ansible/roles/monitoring/xtcp2/`) and the image repo; no Go code. + +- [ ] **I10 (critical)**: land/merge the v2.0.x role revision (listener auth raw, per-host read/admin tokens, ro token mounts); until merged, firewall 8889 to operator hosts + loopback on those hosts +- [ ] Retire the v1.x revision: make `LISTENER_AUTH_MODE` impossible to disable via role defaults; add a `test_xtcp2_fleet_gate.py` assertion that auth env is present +- [ ] Bind guidance: prefer UDS for the control plane (`xtcp2ctl` workflow documented), else loopback + `tasks/firewall.yml` rules for 8889 with TTL 3 +- [ ] I11: state-dir guard — minimum: node-level alert on `/var/lib/xtcp2` usage (vmagent); preferred: XFS project quota (or systemd quota for bare metal) sized to ~2× the ASN cache (~200 MiB headroom) +- [ ] Egress pinning: restrict container egress to the per-DC S3 endpoints (network policy / security group); pair with W2's frozen `s3_endpoint` +- [ ] W5 deployment: second container + `:ro` collector state dir + shared socket dir; canary via the existing fleet-gate mechanism; revert via `xtcp2_revert.yml` +- [ ] Pin the new image tag in the image repo; bump the deployment repo role defaults in lockstep (role gate test asserts image↔config compatibility) + +**Estimate**: 3–5 days spread across phases. **Owner**: infra. + +### W8 — tests, CI, and fleet-config gates + +- [ ] Hostile-path table test (unit, `pkg/xtcp`): `../etc/passwd`, `/run/x/../../`, NUL, `…/` (unicode ellipsis), `/proc/self/mem`, trailing `/`, `//`, symlink-into-root, `/var/lib/xtcp2/ok` (accept), `/tmp/cap` (accept only when a root is configured) — assert proto layer (W1) and Go policy (W2) verdicts +- [ ] `Set` policy tests: path fields rejected; frozen-field change rejected; valid tuning `Set` applies; escape-hatch flag honored +- [ ] Interceptor guard test (W3) + per-service invalid-message cases +- [ ] Landlock: unit re-exec persistence test + microVM scenario (W4) +- [ ] Writer tests (W5): size cap, rate limit, version replay, kill -9 atomicity, peer-credential rejection +- [ ] **Fleet golden-config gate (CI)**: ansible role renders real configs (all DCs, both role revisions) → `protovalidate` them in CI on every proto change; blocks a rule that would reject production +- [ ] Advisory regression: end-to-end gRPC test running the five primitives of advisory §2 and asserting each is blocked at its layer + +### W9 — documentation + +- [ ] [security.md](../security.md): new "Runtime reconfiguration" section — path policy, frozen fields, Landlock gauge, writer process +- [ ] [grpc-api.md](../grpc-api.md): `Set` frozen-field list + error codes (`InvalidArgument` path, `FailedPrecondition` frozen) + `-setAllowFrozenFields` migration note +- [ ] [operations.md](../operations.md): writer deployment (container + systemd), `-writableDir`, quota guidance, token rotation +- [ ] [metrics.md](../metrics.md): `xtcp2_sandbox_active`, `xtcp2_state_writer_errors_total`, `xtcp2_set_applied_total` +- [ ] [README.md](../README.md): link this plan from the advisory entry +- [ ] Update the advisory's layer table with ✅ per workstream as they land + +## Sequencing and dependencies + +``` +Phase 0 (today): the v2.0.x auth merge (I10) — ops action, no release needed +Phase 1 (release vNext): W0 (image rebuild) + W1 ─▶ W2 ─▶ W3 (pure code/build; closes I1-I6, I8, I15) + └──▶ W8 (hostile-path + golden-config gates) ──▶ W9 +Phase 2: W4 (Landlock, optional pivot_root) ──▶ W6 (Set audit) ──▶ W7 quota + audit ship +Phase 3: W5 (writer split) ──▶ W7 (second container, :ro flip) ──▶ W9 +Phase 4: W7 egress pinning + v1.x retirement; W8 end-to-end advisory regression as a standing gate +``` + +- W1, W2, W3 are independent of each other — parallelizable; W2 needs W1's pattern only for shared test fixtures. +- W4 should land **before** W5 (fleet already kernel-protected while the split is built). +- W5 is the only workstream that changes deployment topology; it is also fully reversible until the collector's `:ro` flip. +- Every phase ships independently; no phase requires the next. + +## Fleet rollout runbook + +1. Merge Phase 1 → tag release → image repo bump → ansible role pin bump; fleet-gate test green. +2. Canary via the existing canary mechanism (`ansible/xtcp2.yml`); watch for `InvalidArgument`/`FailedPrecondition` spikes from legit automation using `Set` (expect none — operators use redeploy, but verify `xtcp2_set_applied_total`). +3. If a legit `Set` migration surfaces: document `-setAllowFrozenFields`, time-box it, remove after migration. +4. Phase 2 canary: watch `xtcp2_sandbox_active` (expect 1; 0 on any pre-5.13 kernel → raise a ticket, host is unprotected at L4). +5. Phase 3 canary: writer-up check, baseline alert still fires on link-down (synthetic test), ASN refresh metric, then the `:ro` flip; revert path = `xtcp2_revert.yml` + restore `:rw` mount. +6. Post-rollout: run the [end-state verification checklist](#end-state-verification-checklist) against one canary host with the admin token. + +## End-state verification checklist + +With admin token in hand on a canary host, every item must fail safely: + +- [ ] `Set` with `dest = "file:/etc/cron.d/pwn"` → `InvalidArgument` (W1/W2) and, with rules bypassed, `EACCES` from Landlock (W4) +- [ ] `Set` with `capture_path = /var/spool/cron/` + `write_files = 1` → rejected; no file created +- [ ] `Set` with `link_monitor.baseline_file = /run/pwn.json` → rejected +- [ ] `Set` with `ipmeta_cache_path = /etc/xtcp2/pwn/current.lookup.parquet.zst` → rejected; no directories created anywhere new +- [ ] `Set` with unix listener `address = /etc/ld.so.preload`, `unix_socket_mode = 0777` → rejected (pattern + `lte 384`) +- [ ] `Set` changing `s3_endpoint` to an attacker host → `FailedPrecondition` (frozen), even with `-setAllowFrozenFields` unset network policy blocks the egress (W7) +- [ ] Legit tuning `Set` (poll frequency) applies and the daemon soft-restarts with the Landlock ruleset intact (`xtcp2_sandbox_active` stays 1) +- [ ] Direct process check: as the collector uid, `os.WriteFile("/tmp/x")` fails (zero write rules, post-W5); the writer still publishes a fresh ASN cache +- [ ] `kill -9` the writer mid-baseline-write: old baseline intact, monitor degrades-not-exits, counter increments +- [ ] `/var/lib/xtcp2` fill attempt (legal writes within quota bounds) trips the size alert +- [ ] Production image (W0): `Set` with `dest = "file:/tmp/pwn"` → fatal startup error, no file ever created (the code is absent from the binary); `write_files = 1` → fatal startup error + +## Open decisions + +| # | Decision | Default proposal | +|---|---|---| +| OD-1 | `unix_socket_mode` cap: 0600 or allow 0660 (shared-group socket)? | 0600 (`lte 384`); relax only if a consumer needs group access | +| OD-2 | Landlock-absent kernel behavior: warn-and-continue vs refuse to start | Warn + `xtcp2_sandbox_active 0` + alert; fleet is ≥6.8 so this is a bare-metal edge case | +| OD-3 | `file:` destination: keep behind `-allowFileDest` or remove entirely in the s3parquet build flavor | Keep behind flag (build flavors already gate schemes) | +| OD-4 | Writer transport: UDS gRPC (chosen) vs loopback TCP | UDS — peer credentials, no port surface; loopback TCP rejected because every host process shares it | +| OD-5 | Does the ipfeed sidecar adopt the writer? | No — out of scope; it is already uid 65534, single subdirectory, no caps | +| OD-6 | Frozen-field escape hatch lifetime | Time-boxed to one migration window, then remove the flag | +| OD-7 | State-dir quota mechanism (alert-only vs XFS project quota) | Alert-only in Phase 2 (cheap), quota when a host survives to Phase 3 | +| OD-8 | Does `xtcp2-min` keep `file:` (it is the stdlib-only flavor)? | Drop it there too; only `full`/`-debug` carry the tag | +| OD-9 | Drop the network stdlib destinations (`tcp`/`udp`/`http`/`https`/`unix`/`unixgram`) from the production flavor? | Yes — production needs `s3parquet` only; kills the DEST-repoint exfil class structurally | + +## References + +- Advisory: [security-vulnerability-configservice-set-fs-write.md](security-vulnerability-configservice-set-fs-write.md) — primitives, reachability, container analysis, layer rationale +- Code touchpoints: `pkg/xtcp/grpc_configService.go`, `pkg/xtcp/destinations_core.go` + `destinations_file.go`, `pkg/xtcp/netlinker.go`, `pkg/xtcp/input_validation.go`, `pkg/linkmonitor/config.go` + `internal/baseline/`, `pkg/ipasn/ipasn.go`, `pkg/listener/listener.go`, `pkg/listenerauth/policy.go`, `cmd/xtcp2/xtcp2.go` (flags, `loadReconfigureConfig:1199`, `execSelfWithConfig:1211`) +- Deployment: the deployment repo → `ansible/roles/monitoring/xtcp2/` (defaults, tasks, firewall, fleet-gate test); the image repo (flake, settings docs) +- Upstream: [Landlock](https://docs.kernel.org/userspace-api/landlock.html), [go-landlock](https://github.com/landlock-lsm/go-landlock), [buf protovalidate](https://github.com/bufbuild/protovalidate) diff --git a/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md b/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md new file mode 100644 index 00000000..ba34ce4d --- /dev/null +++ b/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md @@ -0,0 +1,438 @@ +# Security advisory: `ConfigService.Set` is a filesystem-write primitive + +| | | +|---|---| +| **Status** | Draft advisory — no CVE issued (internal assessment) | +| **Severity** | High (authenticated callers, default deployments: unauthenticated) | +| **Affected** | Every xtcp2 version with `ConfigService.Set` + `file:` destination / `capture_path` / `baseline_file` / `ipmeta_cache_path` / unix listeners — i.e. all current releases | +| **Not affected** | Read-only clients (`Get`, `FlatRecords`, `PollFlatRecords`); hot-path setters (`SetPollFrequency`, `SetS3Upload`, `SetEnvelopeFlush`, `Trigger*` — none touch the filesystem) | +| **Scope** | `ConfigService.Set` runtime reconfiguration; see [security.md](../security.md) for listener-auth posture and [grpc-api.md](../grpc-api.md) for the Set soft-restart design | + +## Contents + +- [Executive summary](#executive-summary) +- [1. How `Set` reaches the filesystem](#1-how-set-reaches-the-filesystem) +- [2. The write primitives](#2-the-write-primitives) +- [3. Who can call `Set`](#3-who-can-call-set) +- [4. The container deployment and its permission model](#4-the-container-deployment-and-its-permission-model) + - [4.1 How the container is deployed](#41-how-the-container-is-deployed) + - [4.2 Runtime settings → Linux primitives](#42-runtime-settings--linux-primitives) + - [4.3 What `NET_ADMIN` + `SYS_ADMIN` actually grant](#43-what-net_admin--sys_admin-actually-grant) + - [4.4 What is restricted and filtered](#44-what-is-restricted-and-filtered) + - [4.5 Can `Set` write to the host filesystem?](#45-can-set-write-to-the-host-filesystem) + - [4.6 Adjacent (non-`Set`) escalation paths](#46-adjacent-non-set-escalation-paths) + - [4.7 Container-escape path idioms](#47-container-escape-path-idioms) +- [5. Remediation plan — belt and braces](#5-remediation-plan--belt-and-braces) + - [5.1 Layer 1 — protovalidate rules on every path-bearing field](#51-layer-1--protovalidate-rules-on-every-path-bearing-field) + - [5.2 Layer 2 — runtime path policy in the `Set` handler (incl. frozen fields)](#52-layer-2--runtime-path-policy-in-the-set-handler-incl-frozen-fields) + - [5.3 Layer 3 — validate every input, everywhere](#53-layer-3--validate-every-input-everywhere) + - [5.4 Layer 4 — kernel self-confinement: Landlock (and chroot alternatives)](#54-layer-4--kernel-self-confinement-landlock-and-chroot-alternatives) + - [5.5 Layer 5 — privilege separation: a dedicated state-writer process](#55-layer-5--privilege-separation-a-dedicated-state-writer-process) + - [5.6 Layer 6 — deployment and runtime guardrails](#56-layer-6--deployment-and-runtime-guardrails) + - [5.7 Layer 0 — build-time elimination of write-capable debug features](#57-layer-0--build-time-elimination-of-write-capable-debug-features) +- [6. What each layer stops](#6-what-each-layer-stops) +- [7. Honest limits — what this does not fix](#7-honest-limits--what-this-does-not-fix) +- [8. Testing plan](#8-testing-plan) +- [9. Rollout sequencing](#9-rollout-sequencing) +- [10. References](#10-references) + +## Executive summary + +`ConfigService.Set` replaces the **entire** daemon configuration at runtime and soft-restarts the process (`syscall.Exec`, config carried via the `XTCP_CONFIG_JSON` env var — `cmd/xtcp2/xtcp2.go:42`, `:1199`, `:1211`). The validation chain (`pkg/xtcp/grpc_configService.go` lines 186–262: protovalidate length/enum rules, enricher-compiled-in check, secret preservation) contains **no path or scheme validation**. A caller who can reach the gRPC listener and pass admin access therefore controls five distinct filesystem-write primitives, four of which accept **arbitrary absolute paths**: + +1. `dest = "file:"` — create/append mode-0600 root-owned files with attacker-influenced content (`pkg/xtcp/destinations_file.go:24`). +2. `write_files > 0` + `capture_path = ` — raw netlink dumps to attacker-chosen paths (`pkg/xtcp/netlinker.go:70`). +3. `link_monitor.baseline_file = ` — baseline store (atomic temp+rename) writes there; only check today is "nonempty, no NUL" (`pkg/linkmonitor/config.go:95`). +4. `ipmeta_cache_path = ` — parquet cache writer with `os.MkdirAll(dir, 0o750)` — an arbitrary **directory-creation** primitive (`pkg/ipasn/ipasn.go:371`). +5. `grpc_listener`/`prometheus_listener` with `network: unix` — bind a socket at any path and chmod it up to 0777 (proto rule `lte: 511`) (`pkg/listener/listener.go`). + +Authentication is **off by default** and the default gRPC listener is wildcard TCP `:8889` ([security.md](../security.md)). In the reference container deployment, the daemon runs as uid 0 with `CAP_NET_ADMIN`+`CAP_SYS_ADMIN` and — in the current role revision — the **only** writable host path is the `/var/lib/xtcp2` bind mount. So the answer to "can `Set` write to the host filesystem from inside the container?" is: **yes, but only under `/var/lib/xtcp2`** (disk-fill DoS, baseline suppression, ASN-cache poisoning, planted files) — which is still serious, because those writes corrupt the very state the monitor's alerts and enrichment depend on. Everything else on the host is read-only to the daemon; the adjacent escalation paths out of the container are `docker.sock:ro` (full Docker API) and the netlink/`setns` capabilities, not `Set`. + +The remediation plan is layered so that every layer independently blocks the primitives above: build-time removal of the `file:` destination and the netlink capture writer from production images (L0), protovalidate rules on all path fields (L1), a Go runtime path policy + frozen-field diff in the `Set` handler (L2), a server-side validating interceptor so every RPC input is always validated (L3), kernel self-confinement via Landlock (L4), privilege separation — moving the two remaining legitimate writes into a chrooted, capability-less helper process with a path-free API (L5) — and deployment/runtime guardrails (L6). + +--- + +## 1. How `Set` reaches the filesystem + +``` +gRPC Set(XtcpConfig) + └─ protovalidate (length/enum/secret rules only — no path or scheme checks) + └─ enricher-compiled-in check, secret preservation pkg/xtcp/grpc_configService.go:186-262 + └─ reconfigureFunc + └─ marshal config → env XTCP_CONFIG_JSON cmd/xtcp2/xtcp2.go:42 + └─ execSelfWithConfig → syscall.Exec(self) cmd/xtcp2/xtcp2.go:1211 (soft restart) + └─ new process builds destinations/stores from the config + ├─ file: destination pkg/xtcp/destinations_file.go:24 os.OpenFile(O_CREATE|O_APPEND, 0600) + ├─ capture writer pkg/xtcp/netlinker.go:70 os.WriteFile(capturePath+"netlink."+RFC3339Nano, 0600) + ├─ baseline store pkg/linkmonitor/internal/baseline os.CreateTemp + rename + fsync + ├─ ipmeta cache pkg/ipasn/ipasn.go:371 os.Create + rename + MkdirAll(0o750) + └─ unix listeners pkg/listener/listener.go bind + chmod (up to 0777) +``` + +The hot-path setters never reach the filesystem: `SetPollFrequency`, `TriggerPoll`, `TriggerPollBurst`, `SetS3Upload`, `SetEnvelopeFlush` only touch in-memory state. Only full `Set` (which re-execs the daemon) can influence paths. + +## 2. The write primitives + +| # | Config field(s) | Primitive | Where | Content control | +|---|---|---|---|---| +| 1 | `dest = "file:"` | create/append 0600 files, arbitrary path | `pkg/xtcp/destinations_file.go:24` | structured records — hostname/location ≤253, label/tag ≤40, daemon_version ≤253 (`pkg/xtcp/deserialize.go:104`); marshaller framing prevents raw newline injection | +| 2 | `write_files` (50) + `capture_path` (51) | raw netlink dumps, `netlink.`, 0600 | `pkg/xtcp/netlinker.go:70` | raw kernel netlink bytes; filename suffix fixed | +| 3 | `link_monitor.baseline_file` | atomic baseline write, arbitrary path | `pkg/linkmonitor/config.go:95`, `internal/baseline/files.go:46` | JSON baseline content (link states) | +| 4 | `ipmeta_cache_path` (244) | parquet cache write + `MkdirAll(dir, 0o750)` — arbitrary **directory creation** | `pkg/ipasn/ipasn.go:371` | zstd parquet rows (ASN ranges); fixed filenames `current/previous.lookup.parquet.zst` | +| 5 | `*_listener` `network: unix` + `address` + `unix_socket_mode` | bind socket at any path, chmod ≤0777 | `pkg/listener/listener.go` | socket file only; existing non-socket paths are refused; stale sockets are unlinked (`:142-152`) — not an arbitrary-delete primitive | + +Ruled out during the assessment: `unlink_stale_unix_socket` only unlinks paths that are already unix sockets; S3/Kafka/HTTP destination setters and link-monitor RPCs (`Rebaseline`/`Resync`) perform no filesystem writes beyond the baseline store in #3. + +## 3. Who can call `Set` + +- `Set` requires `ACCESS_ADMIN` (`pkg/xtcp/grpc_access.go`, `pkg/listenerauth/policy.go`). +- **Authentication is off by default**; with no credential configured, `Policy.Enabled()` is false and every method — including admin ones — passes (`pkg/listenerauth/policy.go:230`). +- Default gRPC listener is wildcard TCP `:8889`; default UDS deployments inherit filesystem permissions instead. +- The daemon requires uid 0 with `CAP_NET_ADMIN`/`CAP_SYS_ADMIN` for netlink `inet_diag` and `setns()` — files created by `Set`-driven writers are root-owned. + +**Bottom line:** any network-reachable client of an unauthenticated deployment can turn `Set` into a root file-write primitive. In authenticated deployments, the admin token holder can. + +## 4. The container deployment and its permission model + +xtcp2 ships as a `FROM scratch` image containing one static binary, deployed by the `monitoring/xtcp2` role in the deployment repo (image built/re-exported by the image repo). This section records how the fleet deploys it, what the container runtime grants it, and how those settings map onto Linux primitives — because the honest answer to "what can this process do to the host?" is the union of the namespace layout, the capability set, and the mount table, not the docker command alone. + +### 4.1 How the container is deployed + +The role is **opt-in per host** (`xtcp2_enabled: false` by default; enabled per DC/batch with `-e xtcp2_enabled=true`), canary-first, and `xtcp2_reinstall: true` re-creates the container each run so image/env changes always apply. Two containers run per host: + +| Container | Image | Runtime | +|---|---|---| +| `xtcp2` | `xtcp2:v2.0.0` — `FROM scratch`, one static binary, no shell | the daemon, uid 0, permissions per §4.2 | +| `ipfeed` | `xtcp2:ipfeed-collector-v2.0.0` | ASN-feed sidecar, uid 65534, fetches over HTTPS and writes one parquet into `/ipfeed/` | + +Per-host runtime materials (never baked into the image): + +- State dir `/var/lib/xtcp2` (root 0750) — link-monitor baseline, `ipmeta/` ASN cache, sidecar's `ipfeed/`. +- Listener read + admin tokens at `/etc/xtcp2/listener_{read,admin}_token` — random per host, mode 0400, rotated via `-e xtcp2_rotate_listener_tokens=true`. +- Per-DC S3 secret pulled from SSM at runtime, mode 0400 — read via file path, not env, so it never appears in `docker inspect` or `/proc//environ`. + +Because the daemon runs in the host netns (§4.2), all listeners live on the **host** stack: Prometheus `:9088` (read token), gRPC `:8889` (read/admin bearer tokens, raw mode), link-monitor detail `:9089` on `127.0.0.1` only. Listener replies carry IPv4 TTL / IPv6 hop limit 3 (`xtcp2_ipv4_ttl: 3`), clamping them to the local segment. The role's firewall task opens `:9088` inbound only from the local docker-bridge subnets (where vmagent scrapes), and only where UFW is actually active. Operators drive gRPC reads and `Set` with the vendored `xtcp2ctl` image run `--network host` with the admin token file mounted. Lifecycle: the built-in `-healthcheck` binary mode probes `/readyz` (the only workable probe on a shell-less image), `restart: unless-stopped`, log rotation (10m × 3), and resource caps — memory 256M (reservation 128M), cpus 2, `pids_limit` 2048. Fleet-constant settings (enrichers, ASN paths, link monitor, modulus) are baked into the image env; the role adds only host-specific env (`LOCATION`, `XTCP_HOSTNAME`, `LABEL`, cadence, token-file paths). + +### 4.2 Runtime settings → Linux primitives + +The container spec (`tasks/main.yml`) translates to kernel-enforced properties as follows: + +| Runtime setting | Linux primitive | Effect on the process | +|---|---|---| +| `network_mode: host` | shares the **host network namespace** | binds, connects, and netlink all operate on the host stack; `:9088`/`:8889`/`:9089` are host ports; no bridge/veth/netns isolation | +| `userns_mode: host` | no uid remap | container uid 0 **is** host uid 0; DAC and file ownership are evaluated against host ids; anything it creates is host-root-owned | +| `cgroupns_mode: host` + `/sys/fs/cgroup:ro` | whole host cgroup tree, read-only mount | socket→container attribution resolves every container's cgroup path; cgroupfs is not writable | +| `cap_drop: ALL` + `capabilities: [NET_ADMIN, SYS_ADMIN]` | bounding/permitted/effective set = exactly those two | every other capability is unavailable even as root — no `CAP_DAC_OVERRIDE`, `CAP_SYS_MODULE`, `CAP_SYS_PTRACE`, `CAP_SYS_BOOT`, `CAP_SYS_TIME`, `CAP_MKNOD`, … | +| `security_opts: no-new-privileges:true` | `PR_SET_NO_NEW_PRIVS` | no privilege gain through `execve` — this also covers `Set`'s soft restart, which `syscall.Exec`s the same binary | +| `read_only: true` rootfs + `tmpfs: /tmp` | root mount `MNT_READONLY`; private tmpfs | zero writes in the image filesystem; `/tmp` is ephemeral scratch | +| `:ro` bind mounts | per-mount read-only flags | `/run/netns`, `/run/docker/netns`, `/sys/fs/cgroup`, `/run/docker.sock`, the S3 secret, and both token files are readable, never writable | +| `{{ xtcp2_state_dir }}:/var/lib/xtcp2:rw` | the single rw bind mount | **the only path through which the container can write to the host filesystem** | +| `groups: []` | supplementary group membership | `connect()` to the 0770 lldpd control socket without needing `CAP_DAC_OVERRIDE` | +| private PID namespace (docker default, not overridden) | pidns | `/proc` inside the container shows only its own processes — host process enumeration and `/proc//environ` are not visible | +| Docker's default seccomp profile (not disabled) | syscall allow-list | blocks legacy syscalls; `setns`/`mount`/`bpf` are allowed or capability-gated, and the caps are present | +| memory / cpus / `pids_limit` | cgroup controllers | resource ceilings | + +Two consequences of "uid 0 without `CAP_DAC_OVERRIDE`" are worth spelling out: the daemon **can** read root-owned mode-0400 files (owner-class DAC — that is how it reads its own token and S3-secret mounts), and it **cannot** read files belonging to other uids (why the lldpd socket gets the group join instead of a capability grant). + +### 4.3 What `NET_ADMIN` + `SYS_ADMIN` actually grant + +The daemon's own startup check (`pkg/xtcp/init_capabilities.go`, see [observability.md](../observability.md)) marks exactly two capabilities **fatal**: `CAP_NET_ADMIN` (netlink `inet_diag` dumps) and `CAP_SYS_ADMIN` (`setns(CLONE_NEWNET)` into every other namespace for the per-netns readers). `CAP_SYS_PTRACE`, `CAP_NET_RAW`, and `CAP_SYS_RESOURCE` are warnings only. So the deployment grants the minimal set the product needs — the intuition "net admin to open the netlink sockets" is correct but incomplete: per-namespace polling also requires `SYS_ADMIN`, and the role grants exactly those two and nothing else. + +Because the container runs in the **host** network namespace and the **initial** user namespace (no userns remap), both capabilities are full-strength: + +**`CAP_NET_ADMIN` (host netns)** + +- netlink `inet_diag` dumps of every socket — the product's function; +- interface/address/route/neighbor configuration — bring links up/down, add or blackhole routes; +- netfilter: rewrite the **host** firewall (`iptables`/`nftables`); +- tc qdiscs, bridge/VRF/macvlan config, conntrack-table manipulation. + +**`CAP_SYS_ADMIN` (initial userns)** + +- `setns()` into every namespace on the host — every container's netns (the function), but also other containers' mount and PID namespaces; +- `mount()`/`umount2()` within its own mount namespace (tmpfs, bind mounts, remounts; previously-locked mounts stay locked); +- `bpf()` — `CAP_SYS_ADMIN` retains the legacy privileged bpf path; +- `perf_event_open`, namespace creation (`unshare`/`clone` with `CLONE_NEW*`), `quotactl`. + +Honest assessment: this is *not* "a locked-down exporter that happens to have net admin." The filesystem is tightly confined (§4.2), but `NET_ADMIN` + `SYS_ADMIN` in the initial userns make the process host-network-equivalent and namespace-equivalent — an RCE in the daemon can reconfigure host networking and enter any namespace regardless of `Set`. That is inherent to a host-wide socket monitor, not a deployment misconfiguration; §5's layers shrink the *filesystem* angle (L4/L5) but cannot remove this one. Seccomp does not help here: the default profile does not gate namespace entry or mounting away from a process holding these capabilities. + +### 4.4 What is restricted and filtered + +Everything an attacker (an admin-token holder, or an RCE inside the daemon) still faces: + +- **Filesystem**: read-only image rootfs; every bind mount `:ro` except `/var/lib/xtcp2`; `/tmp` is ephemeral tmpfs; no `CAP_DAC_OVERRIDE`, so other uids' files are unreadable. +- **Capabilities**: bounding set of two — no kernel module loading, no ptrace of other processes, no system clock, no `mknod`, no DAC override. +- **Privilege transitions**: `no-new-privileges` survives `execve`, including `Set`'s soft restart. +- **Process visibility**: private PID namespace — host PIDs and their environments are invisible. +- **Syscalls**: Docker's default seccomp profile (legacy set blocked). +- **`/proc`**: runc's defaults render `/proc/sys` read-only (sysctl / `core_pattern` writes blocked) and mask `/proc/kcore` and friends; the private pidns hides host PIDs — together these close the classic `/proc/1/root` and `core_pattern` escape idioms (see §4.7). +- **`/dev`**: runc populates a private tmpfs with an allow-list of character devices only (`null`, `zero`, `full`, `random`, `urandom`, `tty`, ptys, `shm`) — **no block devices exist in the container** (`/dev/sda` and friends are absent), `CAP_MKNOD` is dropped so none can be created, and the cgroup device filter denies `open()` on any non-allow-listed major/minor regardless of capabilities. Three independent guards stand between the daemon and raw disks; creating *regular files* in `/dev` touches only ephemeral tmpfs. +- **Network (app-level)**: bearer auth on `:9088`/`:8889` with per-host tokens and rotation; `/ip` detail loopback-only; TTL-3 listener replies; UFW scrape rule limited to local bridge subnets; S3 credential via a 0400 file mount, not env. +- **Resources**: memory / CPU / pids caps; log rotation. + +Not filtered (see §4.6): the Docker API through `/run/docker.sock` (a `:ro` mount is not an API restriction), host-firewall rewriting via `NET_ADMIN`, and namespace entry via `SYS_ADMIN`. + +### 4.5 Can `Set` write to the host filesystem? + +**Fleet role revision v2.0.x (current `origin/main`)** + +- Listener auth is **on**: Prometheus 9088 and gRPC 8889 both require bearer tokens (`LISTENER_AUTH_MODE=raw`), per-host random read + admin tokens at `/etc/xtcp2/listener_{read,admin}_token` (0400, ro-mounted, rotated by `xtcp2_rotate_listener_tokens`). +- The **only** writable host path is `{{ xtcp2_state_dir }}:/var/lib/xtcp2:rw` (host `/var/lib/xtcp2`, root 0750) — it holds the link-monitor baseline, `ipmeta/` ASN cache, and the sidecar's `ipfeed/` (uid 65534). + +**Answer: yes — a `Set` caller holding the admin token can write to the host, but only under `/var/lib/xtcp2`.** That still enables, from inside the container, against the host: + +- **Disk-fill DoS** on the host rootfs by repointing `capture_path`/`ipmeta_cache_path` into the state dir (or just enabling `write_files`) — the state dir has no quota. +- **Baseline suppression / alert manipulation**: `Set` can write a doctored `baseline.json` (all-links-up baseline written while links are actually down = alerts that never fire), or point `baseline_file` at a file whose content keeps the monitor "healthy". +- **ASN-cache poisoning**: `Set` writes attacker bytes to a file, then points `ipmeta_cache_path`/`asn_db_path` at it; the daemon reloads hourly and parses attacker-controlled parquet/zstd — both a parser attack surface and a way to poison the enrichment data shipped to the data lake. +- **Planted files** in `/var/lib/xtcp2` (mode 0600 root-owned) and planted unix sockets (chmod up to 0777 inside that dir). + +What `Set` **cannot** do from inside that container: write anywhere else on the host. The image rootfs is read-only, `/run` paths are `:ro`, and there is no `CAP_DAC_OVERRIDE` to escape the mount table — writes to `/tmp`, `/dev`, and `/dev/shm` succeed but land in ephemeral tmpfs, never the host disk. The mount namespace fixes the writable set: no `..` or symlink traversal escapes it, because a bind mount has no "above" — parent resolution lands in the container's own read-only rootfs. + +**Fleet role revision v1.x (previous)** + +No listener-auth env vars are set, so the gRPC admin surface on host-netns `:8889` is **unauthenticated to anything that can reach the host** — including other containers. However this revision has no writable state mount, so `Set`'s filesystem writes stay in container tmpfs (destroyed on restart). The live risks in that revision are the non-filesystem ones: `dest` repointing to attacker endpoints (socket-table exfil; `s3parquet:` to an attacker endpoint signs SigV4 with the preserved DC keys), pprof exposure, and poll-burst DoS. + +### 4.6 Adjacent (non-`Set`) escalation paths + +- `/run/docker.sock:ro` is a **mount** restriction, not an API restriction — the Docker API itself can create containers that mount the host root. It is mounted for container-id resolution (cgroup → container attribution). Treat a compromised daemon as host-root regardless of `Set`. +- `NET_ADMIN` + `SYS_ADMIN` on host network namespaces (`setns` into every netns) are required for the product's core function and are inherently host-equivalent privileges. +- These are pre-existing, by-design capabilities of the monitoring daemon; this advisory is scoped to `Set`'s write primitives. The layers below shrink them where possible. + +### 4.7 Container-escape path idioms + +How the classic file-write escape targets fare when `Set` points a `file:` destination at them in the role-deployed container (§4.2 — both the v1.x and v2.0.x revisions): + +| Path idiom | Verdict | Why | +|---|---|---| +| `/proc/1/root/root/.ssh/authorized_keys` | **fails** | private pidns: `/proc/1` is the container's own init, so `/proc/1/root` is the container's **read-only rootfs** → `OpenFile` gets `EROFS` | +| `/proc/1/root/home//.ssh/authorized_keys` | **fails** | same | +| `/etc/hostname` | **fails** | read-only rootfs → `EROFS` | +| `docker.sock` (as a `file:` target) | **fails** | a unix-socket path cannot be opened with `open(2)`; the bind is `:ro` regardless | + +The `/proc/1/root` idiom — dereferencing PID 1's root to escape a container's mount namespace — works only when the daemon **shares the host PID namespace or runs on the host directly**. The private pidns (a docker default, not overridden by the role) is the load-bearing restriction here, alongside the read-only rootfs; both deserve explicit regression guards so a future `pid_mode: host` "fix" for namespace discovery cannot silently reintroduce the escape. The only writable paths in the role-deployed container are ephemeral tmpfs — `/tmp`, `/dev`, `/dev/shm` (both revisions; never the host disk) — and `/var/lib/xtcp2` (v2.0.x; the only host path, and not on an SSH path). + +A failed write is not silent: `Set` applies the config → soft-restart `exec` → `InitDests` → `OpenFile` fails on the read-only filesystem → the daemon exits → `restart: unless-stopped` brings the container back with the clean image env (`XTCP_CONFIG_JSON` was process env, not container env). A crash-restart loop that follows a `Set` is therefore a detection signal in its own right. + +If xtcp2 runs outside this role — a hand-run container with different flags, a bare-metal/systemd deployment, `sudo xtcp2` — none of these guards apply and the same `Set` calls land as root. + +## 5. Remediation plan — belt and braces + +Each layer is independently effective; together they remove the entire primitive class. Layers are ordered by effort/benefit — L0 + L1 + L2 are small pure-code/build changes that close the hole; L4/L5 are the kernel-enforced backstops that keep the hole closed even if validation code is later bypassed. + +### 5.1 Layer 1 — protovalidate rules on every path-bearing field + +Proto rules catch shape (charset, traversal segments, control bytes, absolute-only). They cannot check deployment-specific roots — that is L2's job. The pattern + +``` +segment := [A-Za-z0-9._-]*[A-Za-z0-9_-]+[A-Za-z0-9._-]* // ≥1 non-dot char ⇒ blocks "." and ".." +pattern := "^(/segment)+/?$" +``` + +blocks relative paths, dot-dot traversal, empty `//` segments, NULs, spaces, and unicode tricks, while accepting every legitimate path used today. Apply to: + +| Field | Current rule | Add | +|---|---|---| +| `capture_path` (51) | `min_len 1, max_len 80` | `pattern` above | +| `link_monitor.baseline_file` | `max_len 4096` (Go additionally checks nonempty/no-NUL) | `pattern` above | +| `ipmeta_cache_path` (244), `ipmeta_bootstrap_path` (243), `asn_db_path` (241) | `max_len 255` | `pattern` above | +| `docker_socket_path` (202), `lldpd_socket_path` (211) | len rules | `pattern` above | +| `dest` (60) | `min_len 4, max_len 512`; scheme checked only in Go (`pkg/xtcp/input_validation.go`) | `pattern: "^(null|stdout|stderr|file|tcp|http|https|udp|unix|unixgram|kafka|nats|nsq|valkey|s3parquet)://[^[:cntrl:]]+$"` (mirror of `knownSchemes`, `pkg/xtcp/destinations_core.go:70`) | +| `ListenerEndpoint.address` (2) | `max_len 512` | charset pattern; when `network: unix`, path shape as above (message-level CEL cross-rule) | +| `unix_socket_mode` (3) | `lte 511` | tighten to `lte 384` (0600) — nothing legitimate needs a world-writable daemon socket | + +RE2 has no lookahead, so ".."-blocking must be done via the segment pattern, not a negative assertion — the shape above is the standard idiom. Fuzz the pattern with the hostile-path table in [§8](#8-testing-plan). + +### 5.2 Layer 2 — runtime path policy in the `Set` handler (incl. frozen fields) + +Add a post-validation gate in `pkg/xtcp/grpc_configService.go` before `reconfigureFunc` is invoked (and mirror it at startup in `cmd/xtcp2/xtcp2.go` config load, so a hostile env-carried config can't bypass it after soft restart): + +1. **Writable-root allowlist.** New daemon flag `-writableDir` (repeatable), default empty. Every path-bearing field above must `filepath.Clean` + `filepath.EvalSymlinks` (reject on error) into one of the roots. With no roots configured, all filesystem-writing config is rejected — `dest` must not use `file:`, `write_files` must be 0, etc. +2. **Symlink discipline.** After canonicalization, verify no path component is a symlink (component-wise `os.Lstat`), so a previously-planted symlink can't redirect a later write. +3. **Frozen-field diff.** Compare incoming vs running config for the security-critical set and reject changes at runtime (operators change these via redeploy): `dest`, `s3_endpoint`/`s3_bucket`/S3 credential fields, kafka/nats/nsq/valkey options, both listener endpoints + auth settings, `docker_socket_path`, `lldpd_socket_path`, `asn_db_path`, `ipmeta_*`, `baseline_file`, `capture_path`, `write_files`, `xtcp_proto_file`. This simultaneously kills the non-`Set`-FS but adjacent **exfil** class (repointing `dest`/`s3_endpoint` to attacker endpoints from a stolen admin token) and makes the writable-roots check for those fields a startup-only concern. +4. **`file:` dest goes debug-only.** `dest` scheme `file` is a debugging convenience (`docs/output-and-destinations.md`); require an explicit `-allowFileDest` flag for it to validate. + +### 5.3 Layer 3 — validate every input, everywhere + +Current coverage (audited): + +| Service | RPC | Validated? | +|---|---|---| +| `ConfigService` | `Get`, `Set`, `SetPollFrequency`, `TriggerPoll`, `TriggerPollBurst`, `SetS3Upload`, `SetEnvelopeFlush` | ✅ all 7 call `protovalidate.Validate` (`pkg/xtcp/grpc_configService.go:117,151,271,325,361,410,475`) | +| `LinkMonitorService` | `Rebaseline`, `Resync`, `ListInterfaces`, `ListCollectors` | ✅ via `validate()` helper (`pkg/linkmonitor/linkmonitorgrpc/server.go:126`) | +| `XTCPFlatRecordService` | `FlatRecords`, `PollFlatRecords` | n/a — request messages are empty (`xtcp_flat_record.proto:504-514`) | + +Coverage is complete today, but it is per-handler and will drift. Enforce it structurally with a **server-side validating interceptor** installed on every connection: + +```go +func validatingInterceptor(ctx context.Context, req any, info *grpc.UnaryServerInfo, + handler grpc.UnaryHandler) (any, error) { + if m, ok := req.(proto.Message); ok { + if err := protovalidate.Validate(m); err != nil { + return nil, status.Error(codes.InvalidArgument, err.Error()) + } + } + return handler(ctx, req) +} +``` + +Streaming RPCs validate in a `RecvMsg` wrapper or at handler entry (the flat-record requests are empty today, so cost is nil). Add a guard test that walks every registered service descriptor and asserts each unary handler rejects an invalid crafted message, so no future RPC can skip validation. + +### 5.4 Layer 4 — kernel self-confinement: Landlock (and chroot alternatives) + +Validation is code; Landlock is a kernel guarantee. A Landlock ruleset applied at daemon startup survives `fork` **and `execve`** — i.e. it persists through `Set`'s soft restart — and cannot be escaped even by a privileged process (a process is always confined by its own ruleset). Using `github.com/landlock-lsm/go-landlock`: + +- Read-only access to everything the daemon needs: `/proc` (namespace discovery), `/sys`, `/run/netns`, `/run/docker/netns`, `/run/docker.sock`, the lldpd socket dir, ASN bootstrap/cache paths, token files. +- **No write access anywhere** by default (or, until L5 lands: write access only beneath `-writableDir` roots, e.g. `/var/lib/xtcp2`). +- Exception carve-outs: a UDS listener dir (`/run/xtcp2`) if UDS listeners are enabled; `/tmp` only if `write_files`/capture is explicitly enabled with a root there. + +Fleet kernels are Ubuntu 24.04 (6.8) — Landlock ABI ≥4, comfortably sufficient for filesystem rules (v1, kernel 5.13+). After this lands, even a fully attacker-controlled `Set` (or a future validation bypass) cannot create, open-for-write, or chmod a file outside the state dir: the syscalls fail with `EACCES`. + +Why Landlock over the classical alternatives for the **daemon** process: + +- **chroot/pivot_root** for the collector is awkward: it legitimately needs `/proc`, `/sys`, `/run/netns`, `/run/docker/netns`, `/run/docker.sock`, `/run/lldpd.socket`, `/sys/fs/cgroup` — a bind-mount farm inside the chroot that must track host layout changes. It also requires `CAP_SYS_CHROOT` at exec time. Use chroot only for the L5 writer process, whose needs are one directory. +- **seccomp** filters syscalls, not paths — it complements (block `chmod` outside allowed modes, `link`/`symlink` outright) but cannot express "write only under `/var/lib/xtcp2`". +- **systemd** (bare-metal installs) gives much of this declaratively: `ProtectSystem=strict`, `ReadWritePaths=/var/lib/xtcp2`, `RestrictAddressFamilies=AF_UNIX AF_NETLINK AF_INET AF_INET6`, `CapabilityBoundingSet=CAP_NET_ADMIN CAP_SYS_ADMIN`, `PrivateTmp=yes`, `SystemCallFilter=@system-service`. Landlock still applies inside, and covers the container deployment where there is no systemd. + +**Self-chroot / pivot_root variant.** The daemon cannot `chroot()` itself today: `CAP_SYS_CHROOT` is dropped (§4.2). If a filesystem-only, in-process confinement is wanted in addition to Landlock, there are two options: grant the narrow `CAP_SYS_CHROOT` and chroot into the state dir at startup; or — needing **no new capability** — `unshare(CLONE_NEWNS)`, bind-mount the minimal set read-only, and `pivot_root` into it, using the `CAP_SYS_ADMIN` the daemon already holds. The bind-mount farm must carry everything the daemon needs — `/proc` (ro), `/run/netns`, `/run/docker/netns`, `/sys/fs/cgroup`, the docker + lldpd sockets, the state dir, token and ASN files, `/dev/null` + `/dev/urandom`, and `/etc/resolv.conf` + `/etc/hosts` + `/etc/ssl/certs` for S3 TLS and DNS — and a missed mount surfaces only as S3 upload failures, so it is fragile. Both variants persist across the soft-restart `exec` (same process, same mount namespace) and hide the host filesystem from path resolution entirely; neither confines networking or namespace entry. Verdict: Landlock stays the primary; `pivot_root` is worth adding mainly for **bare-metal deployments**, where no container pidns exists to make the `/proc/1/root` idiom fail (§4.7). + +### 5.5 Layer 5 — privilege separation: a dedicated state-writer process + +Yes — this is worth doing, and xtcp2 is unusually well-shaped for it: **the legitimate write set is tiny**. In production exactly two things write to disk: + +1. the link-monitor baseline (a small JSON, temp+fsync+rename), and +2. the ipmeta lookup cache (a parquet/zst file, temp+rename, on ASN refresh). + +Everything else in [§2](#2-the-write-primitives) is debug tooling (`capture_path`, `file:` dest) that production never needs, plus socket binds. So the split is clean: keep every privilege with the reader, move every write to a capability-less helper. + +**Architecture** + +``` +┌─────────────────────────────────┐ UDS /run/xtcp2/writer.sock ┌──────────────────────────────┐ +│ xtcp2 (collector) │ ─────────────────────────────────────▶ │ xtcp2-writer │ +│ uid 0, NET_ADMIN+SYS_ADMIN │ WriteBaseline(bytes) │ uid 65534, cap_drop ALL, │ +│ Landlock: NO write rules │ PublishIpmetaCache(bytes) │ chroot /var/lib/xtcp2, │ +│ state dir mounted :ro │ (no path parameters — ever) │ AF_UNIX only, no network │ +│ reads state directly │ ◀───────────────────────────────────── │ fixed targets, size caps, │ +│ holds gRPC + Prometheus │ ack/version │ temp+fsync+rename │ +└─────────────────────────────────┘ └──────────────────────────────┘ +``` + +- **Collector** (`xtcp2`): keeps the capabilities and network/netlink access its job requires, but loses *all* write ability — Landlock ruleset with zero write rules, and in the container the state dir flips to `:ro`. It reads the baseline and ASN cache directly through the read-only mount. +- **Writer** (`xtcp2 -writer`, same binary, new mode): separate process, chrooted (or `RootDirectory=` under systemd) into the state dir, non-root uid (65534, matching the existing `ipfeed` sidecar precedent), `cap_drop ALL`, `RestrictAddressFamilies=AF_UNIX`. It cannot reach the network, cannot `setns`, cannot open anything outside its chroot. + +**API design — the security is in the parameter list, not the transport.** A minimal gRPC service on a UDS, with `SO_PEERCRED` checked on connect: + +```proto +service StateWriter { + // Fixed target: /link-monitor-baseline.json. Content is the + // serialized baseline. No path parameter exists on the wire. + rpc WriteBaseline(WriteBaselineRequest) returns (WriteAck); + // Fixed targets: /ipmeta/current.lookup.parquet.zst (+ previous). + rpc PublishIpmetaCache(PublishIpmetaCacheRequest) returns (WriteAck); +} +``` + +Rules enforced by the writer, in code: + +- **No path parameters.** Targets are named constants inside the chroot. A compromised collector cannot ask it to write anywhere else — the request literally has nowhere to put a path. This is the property that makes the design robust even if the writer is reached by some other local process. +- Size caps (baseline ≤64 KiB — matches the existing store limit; cache ≤192 MiB), per-RPC rate limits, and monotonic version numbers to prevent rollback-replay of old baselines. +- Same discipline the baseline store already has (`pkg/linkmonitor/internal/baseline/files.go`): `CreateTemp` in the target dir → write → `fsync` → `rename` → `fsync(dir)`, mode 0600; the baseline `flock` moves inside the writer. + +**What moves where** + +| Concern | Today | After the split | +|---|---|---| +| Baseline write | collector (arbitrary `baseline_file`) | writer, fixed path | +| ASN cache publish | collector (arbitrary `ipmeta_cache_path`, incl. `MkdirAll`) | writer, fixed paths | +| `capture_path` / `file:` dest | collector, arbitrary paths | removed from the `Set`-mutable surface (L2); debug builds only, Landlock still bounds them | +| Unix socket binds | collector | collector — carve an exception dir only if UDS listeners are enabled | +| Baseline `flock` | in-process | inside writer | + +**Deployment** + +- Bare metal: two systemd units — `xtcp2-writer.service` (`RootDirectory=/var/lib/xtcp2`, `DynamicUser`/uid 65534, `RestrictAddressFamilies=AF_UNIX`, `ProtectSystem=strict`) started before `xtcp2.service`; the UDS is at `/run/xtcp2/writer.sock` with group-readable perms. +- Container: a second container from the same image (`xtcp2_writer`) with the state dir `:rw` and a shared socket dir; the collector container mounts the state dir `:ro` and the socket dir (connect-only needs write on the socket file, not the dir). Both already run `--network host`; the writer needs no network at all beyond the UDS. The alternative — two processes in one container — needs a supervisor in the `FROM scratch` image and is not recommended. + +**Interactions** + +- **Soft restart:** `Set` re-execs only the collector; the writer is untouched. Landlock restrictions persist across the exec (L4), so the confinement survives reconfiguration by construction. +- **Failure mode:** writer down ⇒ collector logs, bumps a counter, keeps monitoring with its last-known state — identical to today's degrade-not-exit behavior when the baseline is unwritable. `Set` is unaffected. +- **Rollback:** the writer keeps `previous.lookup.parquet.zst` semantics exactly as today. + +**Why do it, honestly.** Landlock (L4) already blocks every `Set` write primitive for ~10% of the effort. The split adds three things Landlock cannot: (a) **state integrity against a compromised collector** — an RCE inside the daemon cannot rewrite the baseline to hide link outages, poison the ASN cache that feeds fleet-wide enrichment, or plant persistence; (b) the collector needs **zero** write rules, so its Landlock policy is trivial to audit; (c) the config surface itself shrinks — `baseline_file`/`ipmeta_cache_path` stop being runtime-mutable paths entirely. It does **not** contain the collector's `SYS_ADMIN`/`NET_ADMIN`/docker-sock reach (see [§7](#7-honest-limits--what-this-does-not-fix)); it removes the filesystem angle only. Sequence it last, after L1–L4. + +### 5.6 Layer 6 — deployment and runtime guardrails + +- **Auth on, everywhere.** The v2.0.x role revision's `LISTENER_AUTH_MODE=raw` with per-host admin/read tokens should be treated as the minimum deployment bar; make it impossible to disable via role defaults (and backport to the v1.x revision). +- **gRPC reachability.** Prefer a UDS listener with 0600 perms for the control plane; if TCP is required (the `xtcp2ctl --network host` workflow), bind loopback and firewall 8889, or keep host-netns with auth + TTL 3 and audit. +- **State-dir quota + alerting.** No quota exists today; a `Set`-driven disk-fill of `/var/lib/xtcp2` exhausts the host rootfs. Add a systemd `MaximumDiskUsage`-style quota / XFS project quota (or at minimum size monitoring + alert) on the state dir. +- **Audit `Set`.** Log every applied `Set` (which fields changed, by which token id) to the daemon log and ship it; `Set` is rare enough that this is low-noise and catches a stolen admin token quickly. +- **Egress pinning.** Freeze `s3_endpoint`/`dest` (L2) and, ideally, network-policy the container to its known S3 egress so the frozen-field rule can't be sidestepped at deploy time either. +- **Read-only where possible.** With L5 in place, the collector container mounts the entire host `:ro` — nothing writable remains. +- **Rotate tokens** (`xtcp2_rotate_listener_tokens`) on any suspected exposure; the admin token is the keys to `Set`. + +### 5.7 Layer 0 — build-time elimination of write-capable debug features + +*(Numbered last for link stability; sequenced first — see [§9](#9-rollout-sequencing).)* + +Primitives #1 and #2 live in code that production never uses, yet both are compiled into every production image: + +- The `file:` destination is a **stdlib destination with no build tag** — always compiled regardless of flavor ([build-flavors.md](../build-flavors.md)); the deployed `s3parquet` image also carries `tcp`, `udp`, `http`, `https`, `unix`, and `unixgram`. +- The netlink capture writer (`write_files`/`capture_path`, `pkg/xtcp/netlinker.go:70`) has **no build gate at all** — only the runtime `write_files` budget. + +Remove them from production binaries entirely, using the mechanisms that already exist: + +- Move the `file` destination behind a `dest_file` build tag, following the `dest_` convention. The runtime already handles this cleanly: `lookupDestinationFactory` distinguishes "unknown scheme" from "known but not compiled in", and `destinationLookupError` tells the operator to rebuild with `-tags dest_file`; a config naming an uncompiled scheme is a **fatal startup error**, not a silent no-op — a `Set` that lands the daemon in a restart loop instead of a write. +- Gate the capture writer behind a `capture` build tag; `write_files > 0` without the tag must be a fatal startup error, mirroring the enricher convention. +- Build the production flavor (`oci-xtcp2-s3parquet-enrich`) without either tag; keep them in the `full`/`-debug` engineering flavors. +- Consider also dropping the network stdlib destinations from the production flavor — production needs `s3parquet` (plus `null`/`stdout` for logs). This makes the DEST-repoint exfil class structurally impossible, in addition to L2's frozen fields. + +After this layer, a `Set` call cannot even *express* a `file:` destination or a capture budget on a production host: the code that would act on them is not in the binary. This is the strongest form of L2's `-allowFileDest` escape hatch — the flag becomes the build tag. + +## 6. What each layer stops + +| Threat | L1 proto rules | L2 path policy + frozen fields | L3 interceptor | L4 Landlock | L5 writer split | L6 deploy | +|---|---|---|---|---|---|---| +| Write to arbitrary path (`file:` dest, `capture_path`) | ✅ shape | ✅ root allowlist + freeze | — (enforces L1) | ✅ kernel | ✅ API has no path | — | +| Write under `/var/lib/xtcp2` (baseline poison, cache poison, disk-fill) | ❌ | ❌ (legit use) | — | ❌ (dir allowed until L5) | ✅ collector can't write | ✅ quota + audit | +| MkdirAll anywhere (`ipmeta_cache_path`) | ✅ shape | ✅ | — | ✅ | ✅ | — | +| Planted 0777 unix socket | ✅ charset | ✅ root allowlist | — | ✅ | ✅ | — | +| Symlink redirect of a later write | ❌ (shape ok) | ✅ EvalSymlinks + component Lstat | — | ✅ (read-only dirs) | ✅ chroot | — | +| Unvalidated future RPC input | — | — | ✅ | — | — | — | +| DEST/S3 endpoint repoint exfil | — (scheme ok) | ✅ frozen fields | — | ❌ (network, not fs) | ❌ | ✅ egress pinning | +| RCE in daemon → fs writes/persistence | — | — | — | ✅ (except state dir) | ✅ (nothing writable) | — | + +Layer 0 (§5.7) removes primitives 1–2 at the binary level — the production image simply contains no code that opens files for writing on `Set`'s behalf, so no validation bypass can reach it. + +## 7. Honest limits — what this does not fix + +- **The collector's own capabilities.** `CAP_NET_ADMIN` + `CAP_SYS_ADMIN` on host namespaces plus `/run/docker.sock:ro` mean a fully compromised collector is host-equivalent regardless of L1–L5. L5 removes its *filesystem* angle; the netns/docker angle is inherent to the product's function (see [§4.3](#43-what-net_admin--sys_admin-actually-grant)). +- **Egress is only frozen, not confined.** L2 stops runtime repointing of `dest`/`s3_endpoint`; it cannot stop a redeploy by whoever controls ansible. L6 egress pinning narrows that to trusted operators. +- **SigV4 against attacker endpoints.** With the preserved S3 keys and a repointed endpoint, the minio client signs requests to the attacker; secrets are not disclosed, but data and the access key (inside the signature) are. Frozen fields close the runtime path. +- **A stolen admin token still holds every non-frozen `Set` field** (poll knobs, buffer sizes — DoS-adjacent). Rotation + audit (L6) is the answer there. +- **Landlock is not a sandbox for bugs in what we allow.** The ASN parquet parser still parses files in the state dir; with L5 those files can only come from the writer, but `asn_db_path`/`ipfeed` remain operator/sidecar-controlled inputs and need their own integrity story (checksums, expected publisher). + +## 8. Testing plan + +- **Hostile-path table test** (unit): `../etc/passwd`, `/run/x/../../`, `\x00`, `…/` (unicode ellipsis), `/proc/self/mem`, trailing `/`, `//`, symlink-into-root, `/var/lib/xtcp2/ok` (accept), `/tmp/cap` (accept only with root configured). Assert protovalidate rejects/accepts per L1, and the Go policy per L2. +- **Interceptor test:** craft an invalid message for every RPC in every service; assert `codes.InvalidArgument`. Walk descriptors so new RPCs fail CI until covered. +- **`Set` policy test:** apply a config that changes `dest` scheme, adds `capture_path`, or rewrites `baseline_file`; assert rejection before `reconfigureFunc`. Golden-file the frozen-field set. +- **Landlock test (Linux, gated):** start the daemon with a temp state dir; assert `os.WriteFile` outside allowed roots fails with `EACCES`; assert the same after a synthetic soft-restart exec. +- **Writer tests:** no request field can influence the target path (assert by structurally — the proto has no path field); size cap, rate limit, version-replay rejection; atomicity (kill -9 mid-write leaves old baseline intact). +- **microVM integration:** the split deployment — collector `:ro` state, writer chrooted, `Set` a valid runtime change (poll frequency) and confirm no regression; `Set` with path fields rejected end-to-end. + +## 9. Rollout sequencing + +| Phase | Contents | Risk | Notes | +|---|---|---|---| +| 1 — now | L0 (image rebuild) + L1 + L2 (+ L3 interceptor) | low; pure build/validation change, no behavior change for legit configs | Cuts a patch release; the image repo bump; ansible pin. Frozen-field list needs an operator escape hatch (`-setAllowFrozenFields`) for migrations | +| 2 | L4 Landlock; L6 quota + `Set` audit logging | low-medium; verify fleet kernels ≥5.13 | Ubuntu 24.04 fleet is fine; feature-detect ABI and degrade to warn if absent | +| 3 | L5 writer split; collector state dir `:ro`; ansible second container + systemd units | medium; new process + socket plumbing | The big win: collector has zero write rules; degrade-not-exit on writer outage | +| 4 | L6 egress pinning; v1.x role revision retirement | low | Closes the exfil class at the network layer | + +## 10. References + +- Code: `pkg/xtcp/grpc_configService.go` (Set, validation), `pkg/xtcp/destinations_file.go`, `pkg/xtcp/netlinker.go`, `pkg/linkmonitor/config.go`, `pkg/linkmonitor/internal/baseline/`, `pkg/ipasn/ipasn.go`, `pkg/listener/listener.go`, `pkg/listenerauth/policy.go`, `pkg/xtcp/input_validation.go`, `cmd/xtcp2/xtcp2.go` (soft restart). +- Docs: [security.md](../security.md) (auth off by default, wildcard listeners), [grpc-api.md](../grpc-api.md) (Set soft-restart), [output-and-destinations.md](../output-and-destinations.md) (`file:` destination), [observability.md](../observability.md) (startup capability check). +- Deployment: the deployment repo → `ansible/roles/monitoring/xtcp2/` (defaults, tasks, firewall), the image repo (image build, settings docs). +- Upstream: [Landlock](https://docs.kernel.org/userspace-api/landlock.html) (restrictions inherit across `execve`; not bypassable by privileged processes), [buf protovalidate](https://github.com/bufbuild/protovalidate) (RE2 — no lookahead), [go-landlock](https://github.com/landlock-lsm/go-landlock). From 81071d89b290b3a8e4a145270498a36d2d205d2a Mon Sep 17 00:00:00 2001 From: "randomizedcoder dave.seddon.ca@gmail.com" Date: Mon, 9 Nov 2026 08:30:00 -0800 Subject: [PATCH 2/7] docs(security-hardening): correct guarantees, Landlock ABI gaps, writer perms - Narrow L4/L5 claims: docker.sock bypasses both; the writer accepts schema-valid content at the next seq, so L5 bounds shape, not truth. - Landlock: chmod/chown unmediated at every ABI, truncate from ABI 3, pathname-UDS connect from ABI 9 (RESOLVE_UNIX); tests B10/B14/B24, D7. - Apply the ruleset once and inherit it across soft restarts (16-layer cap); tests C6 (20 restarts) and C7 (forged marker). - Writer files 0640 group xtcp2-state; collector joins the group and keeps a read-only state-dir grant; writer-init chowns legacy files. - Split -writableDir / -readableDir; EvalSymlinks on the deepest existing ancestor so not-yet-created outputs validate. - unix_socket_mode allowlist in: [0, 384]; lte 384 accepted 0577. - Scope egress pinning to the collector's cgroup (nftables), not host-wide SG/UFW; fix the Test D anchor and -setAllowFrozenFields. Co-Authored-By: Claude Opus 5.5 --- .../security-hardening/implementation-plan.md | 8 +++--- .../phase-1-code-hardening.md | 22 ++++++++------- .../phase-2-kernel-confinement.md | 27 ++++++++++++------- .../phase-3-writer-split.md | 20 +++++++------- .../phase-4-egress-fleet-lockdown.md | 27 ++++++++++--------- .../plan-configservice-set-hardening.md | 20 +++++++------- ...ulnerability-configservice-set-fs-write.md | 20 +++++++------- 7 files changed, 80 insertions(+), 64 deletions(-) diff --git a/docs/security-hardening/implementation-plan.md b/docs/security-hardening/implementation-plan.md index 43d3e006..81ff068e 100644 --- a/docs/security-hardening/implementation-plan.md +++ b/docs/security-hardening/implementation-plan.md @@ -46,8 +46,8 @@ Reading order for an implementer: skim the advisory's §2 (primitives) and §4 ( PRs (repo `xtcp2` unless noted; each lands with its tests and doc updates from W9): - [ ] **PR-1 — build-time elimination** ([W0](plan-configservice-set-hardening.md#w0--build-time-elimination-l0)): `dest_file` build tag on the `file` destination; `capture` build tag on the netlink writer; both fatal at startup when requested but uncompiled (`destinationLookupError` path); `nix/versions.nix` flavor matrix + [build-flavors.md](../build-flavors.md) updated; decide [OD-8](plan-configservice-set-hardening.md#open-decisions) (`xtcp2-min`) and [OD-9](plan-configservice-set-hardening.md#open-decisions) (network stdlib destinations). -- [ ] **PR-2 — proto path validators** ([W1](plan-configservice-set-hardening.md#w1--protovalidate-path-rules-l1)): the segment pattern + `dest` scheme allowlist + `unix_socket_mode lte 384` + `ListenerEndpoint` cross-rule on all nine path-bearing fields; regenerate `gen/`; hostile-path table test. -- [ ] **PR-3 — runtime path policy + frozen fields** ([W2](plan-configservice-set-hardening.md#w2--runtime-path-policy--frozen-fields-l2)): `pkg/xtcp/set_path_policy.go` (`-writableDir`, canonicalize, component `Lstat`, root containment, `-allowFileDest`); wired into the `Set` handler **and** startup **and** `loadReconfigureConfig`; `set_frozen_fields.go` diff with `-setAllowFrozenFields` escape hatch; unit tests for reject/accept tables. +- [ ] **PR-2 — proto path validators** ([W1](plan-configservice-set-hardening.md#w1--protovalidate-path-rules-l1)): the segment pattern + `dest` scheme allowlist + `unix_socket_mode in [0, 0600]` + `ListenerEndpoint` cross-rule on all nine path-bearing fields; regenerate `gen/`; hostile-path table test. +- [ ] **PR-3 — runtime path policy + frozen fields** ([W2](plan-configservice-set-hardening.md#w2--runtime-path-policy--frozen-fields-l2)): `pkg/xtcp/set_path_policy.go` (`-writableDir` for write fields, `-readableDir` for read/connect fields, canonicalize via the deepest existing ancestor, component `Lstat`, root containment, `-allowFileDest`); wired into the `Set` handler **and** startup **and** `loadReconfigureConfig`; `set_frozen_fields.go` diff with `-setAllowFrozenFields` escape hatch; unit tests for reject/accept tables. - [ ] **PR-4 — universal validating interceptor** ([W3](plan-configservice-set-hardening.md#w3--universal-validating-interceptor-l3)): unary interceptor + stream `RecvMsg` wrapper; descriptor-walking guard test so no future RPC skips validation. - [ ] **PR-5 — `Set` audit logging** ([W6](plan-configservice-set-hardening.md#w6--set-audit-logging)): field-diff log lines (never secret values), token id, `xtcp2_set_applied_total{applied,frozen_rejected}` metric. - [ ] **PR-6 — regression gates** ([W8](plan-configservice-set-hardening.md#w8--tests-ci-and-fleet-config-gates)): end-to-end advisory regression (all five primitives blocked at their layers); fleet golden-config CI gate (ansible-rendered configs from all DCs validated on every proto change). @@ -67,7 +67,7 @@ PRs (repo `xtcp2` unless noted; each lands with its tests and doc updates from W **Goal:** even a validation bypass or an RCE inside the daemon cannot write outside the operator-approved roots. -- [ ] **PR-8 — Landlock** ([W4](plan-configservice-set-hardening.md#w4--landlock-self-confinement-l4)): `pkg/sandbox` ruleset at startup (read-only `/proc`, `/sys`, `/run/netns`, `/run/docker/netns`, docker.sock, lldpd, tokens, ASN paths; write only `-writableDir` roots); ABI feature-detect with `xtcp2_sandbox_active` gauge; re-exec persistence test (restriction survives the soft restart); microVM write-denial scenario. +- [ ] **PR-8 — Landlock** ([W4](plan-configservice-set-hardening.md#w4--landlock-self-confinement-l4)): `pkg/sandbox` ruleset at startup (read-only `/proc`, `/sys`, `/run/netns`, `/run/docker/netns`, docker.sock, lldpd, tokens, ASN paths; write only `-writableDir` roots); ABI feature-detect with `xtcp2_sandbox_active` gauge; re-exec persistence test (restriction inherited, not re-applied, across > 16 consecutive soft restarts); microVM write-denial scenario. - [ ] **PR-9 — optional pivot_root belt** ([W4](plan-configservice-set-hardening.md#w4--landlock-self-confinement-l4)): self-`pivot_root` with the existing `CAP_SYS_ADMIN` (`CAP_SYS_CHROOT` is dropped); prioritize for bare-metal deployments where no container pidns defeats `/proc/1/root`. - [ ] **PR-10 — state-dir guard + audit ship** ([W7](plan-configservice-set-hardening.md#w7--deployment-guardrails-l6)): size alert (quota per OD-7 later) on `/var/lib/xtcp2`; `Set` audit lines shipped and alerted on. @@ -100,7 +100,7 @@ PRs (repo `xtcp2` unless noted; each lands with its tests and doc updates from W **Goal:** a stolen admin token can only tune poll knobs; everything else requires a redeploy, and the network layer enforces the egress set. - [ ] **PR-14 — close the escape hatch**: remove `-setAllowFrozenFields` once the OD-6 migration window closes; frozen fields become permanent. -- [ ] **PR-15 — egress pinning** (the deployment repo, the infra repo): container egress restricted to the per-DC S3 endpoints (network policy / security groups); `:8889` loopback/operators only, or UDS listeners; UFW rules reviewed with the monitoring lockdown role. +- [ ] **PR-15 — egress pinning** (the deployment repo, the infra repo): collector egress restricted to the per-DC S3 endpoints (nftables rule on the collector's cgroup, not host-wide SG/UFW); `:8889` loopback/operators only, or UDS listeners; UFW rules reviewed with the monitoring lockdown role. - [ ] **PR-16 — retire the v1.x revision** (I10 permanent): role defaults make disabling listener auth impossible; fleet-gate test asserts the auth env; retire the old revision branches. - [ ] **PR-17 — standing CI gate** ([W8](plan-configservice-set-hardening.md#w8--tests-ci-and-fleet-config-gates)): advisory regression + golden-config run on every proto, daemon, image, and role change — the regression tests are the permanent enforcement of this whole program. diff --git a/docs/security-hardening/phase-1-code-hardening.md b/docs/security-hardening/phase-1-code-hardening.md index 4ee79715..fc126b12 100644 --- a/docs/security-hardening/phase-1-code-hardening.md +++ b/docs/security-hardening/phase-1-code-hardening.md @@ -99,7 +99,7 @@ path = "^(/segment)+/?$" // absolute, no empty | `xtcp_proto_file` (63) | `min_len 1` | `pattern` above | | `dest` (60) | `min_len 4, max_len 512` | scheme pattern (below) | | `ListenerEndpoint.address` (2) | `max_len 512` | charset pattern; message-level CEL cross-rule: `network == unix` ⇒ path pattern | -| `unix_socket_mode` (3) | `lte 511` | tighten to `lte 384` (0600) | +| `unix_socket_mode` (3) | `lte 511` | `in: [0, 384]` (0 = the default 0600, or 0600 itself) — not a `lte` bound: 0577 (383) passes `lte 384` yet is world-writable; 0660 (432) only via [OD-1](plan-configservice-set-hardening.md#open-decisions) | **`dest` scheme pattern** — the real syntax (verified from `validateInput`, `pkg/xtcp/input_validation.go`): bare `null`/`stdout`/`stderr`, or `:payload` where path-style schemes take one colon and network schemes take two: @@ -114,6 +114,7 @@ Per-scheme payload shape (host:port vs path) stays in Go — `validateInput`'s s - Hostile-path table (also used by PR-3): `../etc/passwd`, `/run/x/../../`, embedded NUL, `…/` (unicode ellipsis), `/proc/self/mem`, `/etc//hostname`, `/etc/hostname/`, `/etc/hostname`, symlink-into-root, `/var/lib/xtcp2/ok` (accept), `/tmp/cap` (accept only with a root configured). - `dest` table: `stdout`, `null`, `file:/ok`, `kafka:h:9092`, `http://h/x` (accept); `FILE:/x`, `evil://x`, `stdout:extra`, control chars (reject). - Cross-rule: `ListenerEndpoint{network: unix, address: "not a path"}` → `InvalidArgument`. +- Socket-mode table: `0`, `0600` (accept); `0577`, `0601`, `0666`, `0777` (reject — `0577` is the row a `lte 384` bound would wrongly accept). **Acceptance:** every hostile case rejected at the proto layer; the fleet golden configs (all DCs, both role revisions) still validate (PR-6 gate runs them in CI). @@ -131,25 +132,28 @@ Per-scheme payload shape (host:port vs path) stays in Go — `validateInput`'s s ```go type PathPolicy struct { - Roots []string // from -writableDir; empty ⇒ no filesystem-writing config allowed + WriteRoots []string // from -writableDir; empty ⇒ no filesystem-writing config allowed + ReadRoots []string // from -readableDir; read/connect fields only AllowFileDest bool // from -allowFileDest } -// ValidateConfigPaths canonicalizes each path-bearing field (filepath.Clean + -// EvalSymlinks, error on failure), verifies no path component is a symlink -// (component-wise os.Lstat), and requires containment in a root. +// ValidateConfigPaths canonicalizes each path-bearing field: filepath.Clean, +// then EvalSymlinks on the deepest existing ancestor (an output may not exist +// yet, and EvalSymlinks fails on a missing path), then os.Lstat on each +// existing component (no symlinks). Write fields must land in a write root; +// read/connect fields in a read or write root. func (p PathPolicy) ValidateConfigPaths(c *xtcp_config.XtcpConfig) error ``` - Covered fields: `capture_path`, `write_files`, `link_monitor.baseline_file`, `asn_db_path`, `ipmeta_bootstrap_path`, `ipmeta_cache_path`, `docker_socket_path`, `lldpd_socket_path`, `xtcp_proto_file`, and unix `ListenerEndpoint.address`. `dest` scheme `file:` requires `AllowFileDest`. + Write fields (under `-writableDir`): `capture_path` (with `write_files`), `link_monitor.baseline_file`, `ipmeta_cache_path`, unix `ListenerEndpoint.address`, `file:` `dest`. Read/connect fields (under `-readableDir` or a write root): `docker_socket_path`, `lldpd_socket_path`, `asn_db_path`, `ipmeta_bootstrap_path`, `xtcp_proto_file` — requiring these under a writable root would reject every stock config (`/run/docker.sock` is not, and must not be, writable). `-readableDir` defaults to the stock paths' parent dirs, pinned by the golden-config gate. `dest` scheme `file:` requires `AllowFileDest`. - New `pkg/xtcp/set_frozen_fields.go`: `func DiffFrozenConfigFields(running, incoming *xtcp_config.XtcpConfig) (changed []string)` over the frozen set: `dest`, `s3_endpoint`/`s3_bucket`/S3 credential fields, kafka/nats/nsq/valkey options, both listener endpoints + auth settings, `docker_socket_path`, `lldpd_socket_path`, `asn_db_path`, `ipmeta_*`, `baseline_file`, `capture_path`, `write_files`, `xtcp_proto_file`. (`marshal_to` stays runtime-mutable — format changes are legitimate ops; PR-5's audit line covers it.) - `pkg/xtcp/grpc_configService.go` — wire both checks into `Set` after the existing enricher check, before `reconfigureFunc`: path violation → `codes.InvalidArgument` (field named); frozen-field change → `codes.FailedPrecondition` (frozen fields named), unless `-setAllowFrozenFields`. The running config is already in reach: `xtcpConfigService.config` (`grpc_configService.go:25`). -- `cmd/xtcp2/xtcp2.go` — register flags alongside the existing set (~line 454): `-writableDir` (repeatable), `-allowFileDest`, `-setAllowFrozenFlags` (default false). Run the same policy at **startup** and in `loadReconfigureConfig` (~line 1199) so an env-carried `XTCP_CONFIG_JSON` cannot bypass it after the soft restart. +- `cmd/xtcp2/xtcp2.go` — register flags alongside the existing set (~line 454): `-writableDir` and `-readableDir` (repeatable), `-allowFileDest`, `-setAllowFrozenFields` (default false). Run the same policy at **startup** and in `loadReconfigureConfig` (~line 1199) so an env-carried `XTCP_CONFIG_JSON` cannot bypass it after the soft restart. **Tests** -- Reject/accept table for the policy (uses PR-2's hostile-path list): no roots configured ⇒ any path-bearing field ⇒ `InvalidArgument`; `/var/lib/xtcp2` root ⇒ in-root accept, out-of-root reject; symlink component reject (create one in `t.TempDir()`). +- Reject/accept table for the policy (uses PR-2's hostile-path list): no roots configured ⇒ any path-bearing field ⇒ `InvalidArgument`; `/var/lib/xtcp2` root ⇒ in-root accept, out-of-root reject; symlink component reject (create one in `t.TempDir()`); not-yet-created output under a root (first baseline) ⇒ accept; missing parent dirs under a root (`ipmeta_cache_path` before `MkdirAll`) ⇒ accept; missing path whose deepest existing ancestor is a symlink out of the root ⇒ reject; stock `docker_socket_path=/run/docker.sock` with only `/var/lib/xtcp2` writable ⇒ accept (read root); `docker_socket_path` outside every root ⇒ reject. - Frozen-diff: change `dest` → rejected; change only `poll_frequency` → applied; `-setAllowFrozenFields` honored. - Env-carried: set `XTCP_CONFIG_JSON` to a config with `capture_path`, assert the reload path rejects it. @@ -263,7 +267,7 @@ Concretely, with the admin token on a canary host running the new image: - [ ] `Set` with `dest = "file:/var/lib/xtcp2/pwn"` → `FailedPrecondition` (not compiled in) — no file, no restart loop - [ ] `Set` with `dest = "file:…"` on a `full`-flavor binary without `-allowFileDest` → `InvalidArgument` -- [ ] `Set` with any path field outside `-writableDir` roots → `InvalidArgument`, field named +- [ ] `Set` with a write field outside `-writableDir` roots, or a read/connect field outside `-readableDir`/write roots → `InvalidArgument`, field named - [ ] `Set` changing `s3_endpoint`/`dest` scheme/listeners → `FailedPrecondition` (frozen) - [ ] `Set` changing only poll knobs applies; soft restart works; one audit line logged - [ ] Every RPC rejects a crafted-invalid message with `InvalidArgument` (interceptor) diff --git a/docs/security-hardening/phase-2-kernel-confinement.md b/docs/security-hardening/phase-2-kernel-confinement.md index f892e3c5..ad7baa4e 100644 --- a/docs/security-hardening/phase-2-kernel-confinement.md +++ b/docs/security-hardening/phase-2-kernel-confinement.md @@ -43,7 +43,7 @@ PR-9 sec/w4-pivot-root (optional, independent, after PR-8) ## PR-8 — Landlock self-confinement -**Objective:** a startup-applied Landlock ruleset that survives the soft restart and cannot be escaped by the process itself; degraded gracefully (warn + gauge, never a failed daemon) where the kernel lacks support. +**Objective:** a Landlock ruleset applied once at first start, inherited (not re-applied) across soft restarts, and not removable by the process itself; degraded gracefully (warn + gauge, never a failed daemon) where the kernel lacks support. ### Files @@ -75,8 +75,9 @@ Write roots come from Phase 1's `-writableDir` flags (empty by default = **no wr | `/proc` | `READ_FILE \| READ_DIR` | namespace discovery (`/proc/self/ns/net`, `/proc/self/status`); read-only grant is also the kernel backstop against `/proc/sys` and `/proc/sysrq-trigger` writes | | `/sys` | `READ_FILE \| READ_DIR` | interface/ethtool sysfs reads | | `/run/netns`, `/run/docker/netns` | `READ_FILE \| READ_DIR` | named-netns discovery | -| `/run/docker.sock` | `READ_FILE \| WRITE_FILE` | `connect(2)` to a UDS requires rw access on the socket file under Landlock — a **scoped per-path grant** (a socket cannot be used as a file-write target; `open(2)` on it fails regardless) | -| `/run/lldpd.socket` (where present) | `READ_FILE \| WRITE_FILE` | same — LLDP enrichment datagram socket | +| `/run/docker.sock` | `RESOLVE_UNIX` (ABI ≥ 9) | `connect(2)` to a pathname UDS is not a file open — `READ_FILE`/`WRITE_FILE` do not govern it, and below ABI 9 Landlock does not mediate it at all (every pathname socket stays reachable, B14). From ABI 9, handling `RESOLVE_UNIX` and granting it here limits connects to the granted sockets | +| `/run/lldpd.socket` (where present) | `RESOLVE_UNIX` (ABI ≥ 9) | same — LLDP enrichment socket | +| state dir (`/var/lib/xtcp2`) when it is not a write root | `READ_FILE \| READ_DIR` | baseline + ipmeta loads; once Phase 3 removes the write root, this read-only grant is what keeps the collector loading state | | `/sys/fs/cgroup` | `READ_FILE \| READ_DIR` | container-id resolution | | `/etc` | `READ_FILE \| READ_DIR` | tokens + S3 secret mounts, `/etc/resolv.conf`, `/etc/hosts` (pure-Go resolver), TLS roots | | `/dev/null` | `READ_FILE \| WRITE_FILE` | runtime; the only `/dev` write grant | @@ -86,7 +87,9 @@ Write roots come from Phase 1's `-writableDir` flags (empty by default = **no wr Notes baked into the design: -- **Same-dir atomic rename works from ABI 1** — `REFER` only gates *cross-directory* moves, and the stores never truncate (they replace via rename). This is why the ABI floor for write roots is v1, not v3 (see [D-table](#test-d--abi-policy-testsandboxabipolicy)). +- **Same-dir atomic rename works from ABI 1** — `REFER` only gates *cross-directory* moves, and the stores never truncate (they replace via rename). This is why the ABI floor for write roots is v1, not v3 (see [D-table](#test-d--abi-policy-testsandboxabipolicy-pure--the-versionpolicy-decision-table)). +- **What Landlock leaves open.** At every ABI: `chmod`/`chown`/xattrs/`utimes` (B10), and anything another process does on our behalf (dockerd via `docker.sock`). Below ABI 3: `truncate(2)`/`O_TRUNC` on any DAC-writable file — uid 0 owns most of the host (B24). Below ABI 9: pathname-UDS `connect(2)` (B14). Fleet kernels (6.8) are ABI 4, so chmod and UDS connect are open there. The startup log and the D-table name the lost guarantees per ABI; the tests assert them by probed ABI, they do not paper over them. +- **Apply once; inherit across the soft restart.** A Landlock domain is inherited across `execve`, and each `landlock_restrict_self` stacks a layer; the kernel caps a domain at 16 (`E2BIG`). Re-applying on every `Set`-triggered exec would fail on the 17th consecutive reconfiguration. `Constrain` therefore records the applied policy hash in an inherited (non-`CLOEXEC`) memfd and passes its fd number across `syscall.Exec`; a re-exec that finds a matching marker skips the apply and reports `Result{Reused: true}`. A missing or mismatched marker → apply (stacking only narrows). The marker is an fd, not an env var: a deploy-time env var could fake it on a cold start, an fd can only come from a process already inside the domain. If `E2BIG` is ever returned anyway, the inherited layers still confine the process: log loudly, keep `xtcp2_sandbox_active 1`. The same stacking applies to any seccomp filter added later. - **UDS listener addresses must live under a write root** (bind creates a socket file). This matches and re-states Phase 1's L2 rule; the Landlock denial is the kernel backstop. - **Existing fds opened before `Constrain` stay usable** — Landlock evaluates at `open(2)` time. The wiring order (constrain before destination/store init) is what makes this moot; test B20 documents the semantic. - **`no-new-privileges`** is set by the apply step; the container already sets it — idempotent. @@ -127,11 +130,11 @@ Three layers: **A/E/D** are pure table-driven unit tests (no kernel, all platfor | B7 | N | append under the read-only `/run` grants | write a file next to the docker.sock mount | `EACCES` | | B8 | N | `ipmeta` primitive outside roots | `os.MkdirAll("/opt/x/y")` + `os.Create` | `EACCES` on the mkdir | | B9 | N | unlink a file outside roots | `os.Remove("/tmp/x")` | `EACCES` | -| B10 | N | `chmod` a file outside roots | `os.Chmod("/tmp/x", 0777)` | `EACCES` (open-for-write fails before the mode change) | +| B10 | C | `chmod` a file outside roots | `os.Chmod("/tmp/x", 0777)` | `OK` on every ABI — Landlock does not mediate `chmod`/`chown`/xattrs/`utimes`; pins the documented gap | | B11 | N | rename from root to outside | `os.Rename(root+"/f", "/tmp/f")` | `EACCES` (cross-dir `REFER` + target-dir write both denied) | | B12 | P | rename within the root (replace existing target) | `os.Rename(root+"/tmp", root+"/current")` | `OK` — proves baseline/ipmeta atomicity survives | | B13 | P | connect to a granted UDS | socket created by parent under a granted path; child dials | `OK` | -| B14 | N | connect to an ungranted UDS | socket outside granted paths | `EACCES` | +| B14 | N | connect to an ungranted pathname UDS | socket outside granted paths, created by the parent (outside the domain) | `EACCES` on ABI ≥ 9 (`RESOLVE_UNIX` handled); `OK` on ABI ≤ 8 — documented gap, asserted by probed ABI | | B15 | P | read `/proc/self/status` | plain read | `OK` | | B16 | P | read `/etc/resolv.conf`, `/etc/hosts` | plain reads (resolver prerequisites) | `OK` | | B17 | N | write under `/proc/sys` | open `/proc/sys/kernel/hostname` for write | `EACCES` — the `core_pattern`-idiom kernel backstop | @@ -141,6 +144,7 @@ Three layers: **A/E/D** are pure table-driven unit tests (no kernel, all platfor | B21 | P | open `/dev/urandom`, `/dev/null` | read + write respectively | `OK` | | B22 | N | create a **symlink** outside roots (`MAKE_SYM`) | `os.Symlink` in `/tmp` | `EACCES` | | B23 | C | bind a UDS listener outside any root | `net.Listen("unix", "/tmp/x.sock")` | `EACCES` — kernel backstop for the planted-socket primitive | +| B24 | B | truncate a file outside roots | `os.Truncate("/tmp/x", 0)` on a file the parent created | `EACCES` on ABI ≥ 3; `OK` on ABI 1–2 — documented degraded guarantee | ### Test C — persistence across the soft restart (`TestSandboxReexecPersistence`) @@ -151,16 +155,19 @@ Three layers: **A/E/D** are pure table-driven unit tests (no kernel, all platfor | C3 | N | write outside root still denied post-exec | after C1's exec, child writes outside | `EACCES` — the guarantee survives `syscall.Exec` | | C4 | N | exec a **different** binary not granted execute | `syscall.Exec("/bin/sh")` (test helper equivalent) | `EACCES` at exec | | C5 | B | `Set`-shaped flow end-to-end | child 1 applies → execs child 2 with a hostile "config" in env that names an out-of-root path | child 2's write attempt returns `EACCES` even though child 2 re-derived its ruleset from the same args | +| C6 | B | more than 16 consecutive soft restarts | child applies → re-execs itself 20 times, probing in each generation | every generation: write outside root `EACCES`, inside root `OK`, no `E2BIG`; generations 2–20 report `Reused`, `landlock_restrict_self` called exactly once (injected syscall counter) | +| C7 | N | forged marker on a cold start | env names an fd that is absent or not the sandbox memfd | marker ignored; ruleset applied; `Reused false` | ### Test D — ABI policy (`TestSandboxABIPolicy`, pure — the version→policy decision table) | ID | Cat | Description | Input / setup | Expected outcome | |---|---|---|---|---| -| D1 | P | ABI 4 (Ubuntu 24.04 / 6.8) | probed version 4 | full ruleset incl. `REFER`+`TRUNCATE`; `xtcp2_sandbox_active 1`, `xtcp2_sandbox_abi 4` | -| D2 | B | ABI 2 (5.19) | probed version 2 | `REFER` kept, `TRUNCATE` dropped; write roots allowed (same-dir atomic rename needs neither); one info-level log line | -| D3 | B | ABI 1 (5.13) | probed version 1 | `REFER` and `TRUNCATE` dropped; same-dir rename still permitted (v1 rights cover it); active 1; info log | +| D1 | P | ABI 4 (Ubuntu 24.04 / 6.8) | probed version 4 | ruleset incl. `REFER`+`TRUNCATE`, no `RESOLVE_UNIX` (pathname-UDS connect unmediated, logged); `xtcp2_sandbox_active 1`, `xtcp2_sandbox_abi 4` | +| D2 | B | ABI 2 (5.19) | probed version 2 | `REFER` kept, `TRUNCATE` dropped; write roots allowed (same-dir atomic rename needs neither); `truncate(2)` outside roots is no longer denied — one warn line names the lost guarantee | +| D3 | B | ABI 1 (5.13) | probed version 1 | `REFER` and `TRUNCATE` dropped; same-dir rename still permitted (v1 rights cover it); truncation unmediated as D2; active 1; warn line | | D4 | N | ABI 0 / `ENOSYS` (unsupported kernel) | probe fails with not-supported | **warn** + `xtcp2_sandbox_active 0` + daemon continues (OD-2: degrade, never a failed daemon) | | D5 | C | `EPERM` from a seccomp-blocked `landlock_*` syscall (possible on old Docker default profiles) | probe returns EPERM | same as D4: warn + gauge 0 + continue, message names the likely cause | +| D7 | P | ABI ≥ 9 | probed version 9 | `RESOLVE_UNIX` handled; granted on the docker.sock and lldpd socket only; other pathname sockets denied (B14) | | D6 | C | ABI ≥ 2 but UDS listener outside all roots | listener path not under a write root | `Constrain` returns a configuration error (loud at startup, not a runtime bind surprise) | ### Test E — root validation (`TestSandboxRootValidation`, pure) @@ -231,7 +238,7 @@ Three layers: **A/E/D** are pure table-driven unit tests (no kernel, all platfor |---|---|---| | OD-2 (already defaulted) | Landlock-absent kernel behavior | warn + `xtcp2_sandbox_active 0` + continue (D4/D5) | | new | dependency: go-landlock vs raw x/sys | go-landlock | -| new | ABI floor for write roots | v1 (same-dir atomic rename needs no `REFER`/`TRUNCATE`); v0 disables writes with a warning | +| new | ABI floor for write roots | v1 (same-dir atomic rename needs no `REFER`/`TRUNCATE`); ABI 1–2 lose truncate protection and say so; v0 disables writes with a warning | | new | Landlock network rules (TCP bind/connect scoping) | out of scope this phase; revisit after Phase 4 egress pinning | | OD-7 | quota mechanism | alert-only here (Test H); XFS project quota deferred to Phase 4 | diff --git a/docs/security-hardening/phase-3-writer-split.md b/docs/security-hardening/phase-3-writer-split.md index 22e6a2b7..44d3a081 100644 --- a/docs/security-hardening/phase-3-writer-split.md +++ b/docs/security-hardening/phase-3-writer-split.md @@ -30,10 +30,10 @@ The daemon has exactly two legitimate writes (advisory §2): the link-monitor ba |---|---|---| | Collector RCE writes anywhere validation allows | one trusted process, one ruleset | collector has zero file-write rules — the kernel answer is "no", always | | `Set` flips a path field | rejected (Phase 1) / kernel-denied (Phase 2) | same, and the path fields no longer exist in the collector's write path at all | -| Collector RCE **as root** wants to poison state | — | must speak the writer's fixed 3-op API: two named targets, one size cap each, monotonic seq — no paths, no arbitrary bytes past schema checks | +| Collector RCE **as root** wants to poison state | — | must speak the writer's fixed 3-op API: two named targets, one size cap each, monotonic seq — no paths, no arbitrary bytes past schema checks; schema-valid poison at the next `seq` still lands, and `docker.sock` bypasses the writer entirely | | Writer itself compromised | — | unprivileged uid, no capabilities, no network (container `network_mode: none`), Landlock write rules only on its two target dirs | -What this phase does **not** fix: the collector can still reach network egress (Phase 4) and the writer accepts a well-formed request from uid 0 (or a stolen shared secret — see [decisions](#decisions-to-close-during-phase-3)) and will overwrite its two fixed targets with schema-valid data; that bounded surface is the accepted residual. +What this phase does **not** fix: the collector keeps `docker.sock`, so an RCE can have dockerd write any host path (the state dir included) without touching the writer; the collector can still reach network egress (Phase 4); and the writer accepts a well-formed request from uid 0 (or a stolen shared secret — see [decisions](#decisions-to-close-during-phase-3)) and will overwrite its two fixed targets with schema-valid data; that bounded surface is the accepted residual. ## PR map and branches @@ -64,7 +64,7 @@ PR-11 sec/w5-writer ──▶ PR-12 sec/w5-collector ──▶ PR-13 sec/w5-depl | Op | Name | Request | Writer-side effect | Writer-side validation | |---|---|---|---|---| | 0 | `PING` | — | reply | none | -| 1 | `WRITE_BASELINE` | `payload []byte`, `seq uint64` | temp+fsync+rename onto the configured baseline target (`0600`) | oneof arm selects the `baseline` target; protovalidate `max_len 65536` (== `baseline.MaxFileBytes`); payload parses as a `wireRecord` with `Version == 1` (`baseline.Parse` reused); `seq` monotonic (`gte 1`) | +| 1 | `WRITE_BASELINE` | `payload []byte`, `seq uint64` | temp+fsync+rename onto the configured baseline target (`0640`, group `xtcp2-state`) | oneof arm selects the `baseline` target; protovalidate `max_len 65536` (== `baseline.MaxFileBytes`); payload parses as a `wireRecord` with `Version == 1` (`baseline.Parse` reused); `seq` monotonic (`gte 1`) | | 2 | `WRITE_IPMETA` | `payload []byte`, `seq uint64` | temp + preserve-previous (`current → previous`, both writer-configured) + rename + `syncDir` — the exact dance of `ipasn.PublishLookupCache` (`ipasn.go:373`) | oneof arm selects the `ipmeta` target; protovalidate `max_len` == `-ipmetaMaxBytes` (default 64 MiB); `seq` monotonic (`gte 1`) | | 3 | `GET_STATS` | — | counters + last `seq` per target | none | @@ -123,7 +123,8 @@ A team standardizing on one IPC mechanism could defensibly choose gRPC-over-UDS ### Targets, auth, seq, locking - **Targets** come from repeated `-writerTarget name=path` flags — the writer's own config, reachable by nothing `Set` controls. The request's oneof arm selects the target — the envelope carries no strings at all; an op whose target was not configured is rejected (`unconfigured_target`). -- **Auth:** `SO_PEERCRED` on each accepted connection — peer uid must equal the configured collector uid (fleet: 0; check injectable for tests like `capgetFunc` in `init_capabilities.go`). Socket dir `0750`, socket `0660`. Both are *layers*: uid 0 already passes the cred check, so the real auth is the API surface itself (see decisions). +- **Auth:** `SO_PEERCRED` on each accepted connection — peer uid must equal the configured collector uid (fleet: 0; check injectable for tests like `capgetFunc` in `init_capabilities.go`). Socket dir `0750`, socket `0660`, both group `xtcp2-state`. Both are *layers*: uid 0 already passes the cred check, so the real auth is the API surface itself (see decisions). +- **Ownership:** the collector is uid 0 **without** `CAP_DAC_OVERRIDE`/`CAP_DAC_READ_SEARCH`, so it cannot read a `0600` file owned by uid 65534. The writer therefore runs as `65534` with primary group `xtcp2-state` (a fixed gid in the image and role) and umask `027`: targets `0640`, dirs `0750`, group `xtcp2-state`. The collector joins the group (`group_add`, the same pattern as the lldpd socket) and keeps a **read-only** Landlock grant on the state dir. writer-init (one-shot, root) chowns a pre-split state dir and its root-owned `0600` targets to `65534:xtcp2-state` with these modes before the writer starts. - **Seq (monotonic per target):** writer is the source of truth. Persisted as a sidecar `.ver` (decimal, atomic, written *after* the rename). Commit order: payload → fsync → rename → sidecar. A crash between rename and sidecar leaves the sidecar one step behind; a replay at the old seq is then accepted once — harmless because both targets are full-file writes (idempotent). Rollback protection has a documented ≤ 1-step window across a writer crash; anything older is rejected (`stale_seq`). - **Locking:** the baseline's lifetime `flock` (`files.go` `ErrLocked`) becomes per-op, held by the writer; the collector's post-split load is a lock-free read (it is no longer the baseline "owner"). The writer-init bootstrap (below) re-imports an existing target without a sidecar. - **Metrics:** `xtcp2_writerd_requests_total{op,result}`, `xtcp2_writerd_last_seq{target}` — exported through `GET_STATS` and re-registered by the collector (proxy), so vmagent scraping is unchanged and the writer keeps **no TCP listener**. @@ -139,7 +140,7 @@ Harness: `pkg/writerd` mirrors the injectable-`operations` seam from `baseline/f | ID | Cat | Description | Input / setup | Expected outcome | |---|---|---|---|---| -| A1 | P | valid baseline write | 1 KiB valid wireRecord, seq 1 | accepted; file at configured target == payload, mode `0600`; reply OK | +| A1 | P | valid baseline write | 1 KiB valid wireRecord, seq 1 | accepted; file at configured target == payload, mode `0640`, group `xtcp2-state`; reply OK | | A2 | N | op whose target is not configured | op 2 against a writer started baseline-only (`-writerTarget baseline=…`) | rejected `unconfigured_target`; nothing written | | A3 | N | attacker stuffs path-like bytes into the payload | path bytes inside the payload | no string in the envelope is ever resolved as a path: schema-invalid → `invalid_payload`; schema-valid → written verbatim to the one fixed target (the no-path-parameters property — there is no field to abuse) | | A4 | B | payload exactly at the baseline cap | `len == baseline.MaxFileBytes` (65 536) | accepted | @@ -253,11 +254,11 @@ Container pair (deployment role, mirroring the existing `ipfeed` second-containe | | `xtcp2` (collector) | `xtcp2_writer` (writer) | |---|---|---| -| uid | 0 (unchanged — keeps `NET_ADMIN`/`SYS_ADMIN`) | 65534 | +| uid | 0 (unchanged — keeps `NET_ADMIN`/`SYS_ADMIN`); `group_add: xtcp2-state` | 65534, primary group `xtcp2-state` | | capabilities | `cap_drop ALL` + NET_ADMIN, SYS_ADMIN (unchanged) | `cap_drop ALL` — **none added** | | network | host (unchanged, ports 9088/8889/9089) | `network_mode: none` — no interfaces, no egress at all | | mounts | as today, **minus** the rw state dir (kept `ro` for loads) + socket volume | `/var/lib/xtcp2:rw` (the only writable mount), socket volume, nothing else — no docker.sock, no netns, no S3 secret | -| Landlock | zero file-write rules | write rules on the two target dirs | +| Landlock | zero file-write rules; read-only on the state dir; `RESOLVE_UNIX` on the writer socket (ABI ≥ 9) | write rules on the two target dirs | | command | unchanged | `xtcp2 -writer -sock /run/xtcp2/writer.sock -writerTarget baseline=... -writerTarget ipmeta=...` | Socket sharing: a docker named volume mounted rw in both containers, containing only the socket dir (`0750`) and socket (`0660`). Whether the collector's mount can be `ro` (connect(2) on a UDS under an ro mount) is decided by F-table row F8 — if it works, the collector ends with **zero rw mounts, full stop**. @@ -278,11 +279,12 @@ Rollout order — each step is independently shippable and reversible: | F1 | P | writer up, pair healthy | fresh deploy | socket `0660` in dir `0750`; handshake OK; both processes report `xtcp2_sandbox_active 1`; writer `/proc/self/status` `CapEff: 0` | | F2 | P | collector has no writable host path | `docker inspect` + crippled build (Phase 1 policy compiled out) attempting a write anywhere | kernel denies every file write outside the socket volume; `Set` with a hostile path applies but writes fail `EACCES` | | F3 | N | writer container killed mid-operation | `docker kill` writer | collector keeps collecting (netlink path unaffected), error counter rises, `Xtcp2WriterDown` fires within one cycle; recovery on restart | -| F4 | P | baseline lands end-to-end | induce a baseline learn on the canary | `/var/lib/xtcp2` file exists, owner `65534:65534`, mode `0600`, contents == wire JSON | +| F4 | P | baseline lands end-to-end | induce a baseline learn on the canary | `/var/lib/xtcp2` file exists, owner `65534:xtcp2-state`, mode `0640`, contents == wire JSON; the collector's next load reads it (no `EACCES`) | | F5 | P | ASN publish rotates end-to-end | trigger ASN refresh | cache written by writer; previous path holds the prior cache; collector's next load serves the new one | | F6 | C | fresh volume, sidecar missing | first boot on a new host | writer **refuses to start** (fail-closed) with a one-line runbook message; after the documented init invocation (imports existing targets, writes sidecars) it starts and serves | | F7 | N | hostile `Set` payloads replayed | `dest` `file:` targets from advisory §4.7 | validation rejects; nothing is written anywhere — kernel gate proof (E9's microVM twin) | | F8 | C | collector socket mount `ro` | mount the named volume `ro` in the collector container | document the verified answer: if connect(2) succeeds → ship `ro` (zero rw mounts); if `EROFS` → keep `rw`, note the socket volume's contents are one socket file (DoS-only loss) | +| F10 | N | collector without the group | drop `group_add` on a canary | baseline/ipmeta loads fail `EACCES` and alert — negative control: group membership, not a capability, grants the read | | F9 | C | soft restart with the split on | ordinary tuning `Set` | exec + reconnect clean; one save after restart succeeds (D6/E10 twin at deploy level) | ## Appendix A — the write inventory and the target wish-list @@ -359,7 +361,7 @@ On a canary pair running a **deliberately crippled build** (Phase 1 path policy - [ ] `docker inspect xtcp2` shows the collector's only rw mounts are container-internal (`/tmp` tmpfs) and the socket volume (or, per F8, `ro` there too); the state dir and rootfs are `ro` - [ ] the writer is the only process in the deployment able to write, and only to its two configured targets; `CapEff 0`, no network interface, uid 65534 - [ ] `Set` with `dest:"file:/proc/1/root/root/.ssh/authorized_keys"` applies (validation stubbed) → the write fails at the kernel — no authorized_keys file appears on the host; same for every other path primitive -- [ ] a baseline learn lands in `/var/lib/xtcp2` via the writer (owner `65534`, `0600`); an ASN refresh rotates the cache + previous files via the writer +- [ ] a baseline learn lands in `/var/lib/xtcp2` via the writer (owner `65534:xtcp2-state`, `0640`); an ASN refresh rotates the cache + previous files via the writer - [ ] killing the writer degrades, never breaks: collector keeps collecting, `Xtcp2WriterDown` fires, everything recovers on restart - [ ] the writer protocol surface is exactly the 4 ops: a fuzzer against the socket (1 h, dumb mode) produces no write outside the two targets, no crash, no OOM, no leaked fds - [ ] alert runbook covers: writer down, sidecar init on fresh hosts, socket volume loss (recreate + restart pair) diff --git a/docs/security-hardening/phase-4-egress-fleet-lockdown.md b/docs/security-hardening/phase-4-egress-fleet-lockdown.md index 74939678..f5fb3bb8 100644 --- a/docs/security-hardening/phase-4-egress-fleet-lockdown.md +++ b/docs/security-hardening/phase-4-egress-fleet-lockdown.md @@ -38,7 +38,7 @@ The verification target is the [end-state verification checklist](plan-configser ``` PR-14 sec/w6-hatch-close (xtcp2) ─┐ -PR-15 sec/w7-egress (xtcp2 + image + deployment-repo + SG) ─┼─▶ PR-17 sec/w8-standing-gates (CI) +PR-15 sec/w7-egress (xtcp2 + image + deployment-repo + nft)─┼─▶ PR-17 sec/w8-standing-gates (CI) PR-16 sec/w7-retire-v1x (deployment-repo) ─┘ ``` @@ -58,9 +58,9 @@ PR-16 sec/w7-retire-v1x (deployment-repo) ─┘ **Layer 1 — build (closes OD-9, decision [recorded](plan-configservice-set-hardening.md#open-decisions): yes).** Gate the network stdlib destinations behind the established `dest_` build-tag pattern: `dest_tcp`, `dest_udp`, `dest_http` (covers `https`), `dest_unix`, `dest_unixgram` on `pkg/xtcp/destinations_{tcp,udp,http,unix,unixgram}.go` — today all five are **unconditional** (`knownSchemes`, `destinations_core.go:70`). `nix/versions.nix` flavor lists drop them from the production flavor (the fleet image is `oci-xtcp2-s3parquet-enrich`, which already excludes kafka/nats/nsq/valkey); `full` and the debug flavor keep them. Production needs `s3parquet` + `null`/`stdout` only. The existing `destinationLookupError` behavior (fatal at startup when a requested scheme is not compiled in; `FailedPrecondition` from `Set`) means a repointed config **cannot start**, fleet-wide. -**Layer 2 — kernel (Landlock network rules, ABI 4; the Phase 2 deferred decision).** The collector's `pkg/sandbox` ruleset gains TCP network rules: `CONNECT_TCP` scoped to port 443, `BIND_TCP` scoped to the listener ports (9088, 8889, 9089). Honest limits, documented in-code: Landlock net rules are **port-scoped, not host-scoped** (a 443 listener anywhere passes the kernel check) and ABI 4 does not cover UDP — DNS keeps working and UDP egress stays governed by Layer 3. Degradation mirrors Phase 2: ABI < 4 → warn + `xtcp2_sandbox_net 0` + continue. The writer needs nothing — its container is `network_mode: none` (Phase 3). +**Layer 2 — kernel (Landlock network rules, ABI 4; the Phase 2 deferred decision).** The collector's `pkg/sandbox` ruleset gains TCP network rules: `CONNECT_TCP` scoped to port 443, `BIND_TCP` scoped to the listener ports (9088, 8889, 9089). Honest limits, documented in-code: Landlock net rules are **port-scoped, not host-scoped** (a 443 listener anywhere passes the kernel check) and ABI 4 does not cover UDP (ABI 10 adds UDP rules; best-effort where present) — DNS keeps working and UDP egress stays governed by Layer 3. Degradation mirrors Phase 2: ABI < 4 → warn + `xtcp2_sandbox_net 0` + continue. The writer needs nothing — its container is `network_mode: none` (Phase 3). -**Layer 3 — cloud/host (deployment + infra repos).** Security groups: host egress restricted to the per-DC S3 endpoints and the operator VPN; UFW OUTPUT rules reviewed with the monitoring lockdown role. This is the only layer that constrains by **host**, closing Landlock's port-only gap. Detection: the host Falco pipeline gains an alert for outbound connections to non-S3 hosts from the collector's uid. +**Layer 3 — host, scoped to the collector (deployment + infra repos).** The collector runs in the host netns, so container network policy does not apply — and host-wide controls (security groups, UFW `OUTPUT`) would also restrict every co-located workload and host service. The isolation boundary is the collector's cgroup: run it under a dedicated cgroup parent (`--cgroup-parent xtcp2.slice`, created before the rule loads) and add an nftables `output` rule matching `socket cgroupv2 level 1 "xtcp2.slice"` that allows the per-DC S3 endpoints and DNS and drops the rest. Host-wide security groups change only on hosts dedicated to infrastructure. This is the only layer that constrains by **host**, closing Landlock's port-only gap. Detection: the host Falco pipeline gains an alert for outbound connections to non-S3 hosts from the collector's container (match the cgroup, not uid 0 — every host daemon is uid 0). **Also here (closes OD-7):** state-dir quota — XFS project quota on `/var/lib/xtcp2` for the writer container, plus a small writerd addition from Phase 3: per-target rate cap (default 12 writes/hour/target — a legitimate baseline learns a handful of times an hour) so the seq monotonicity can't be ground into disk fill by an attacker with a valid request shape. @@ -118,22 +118,23 @@ Table-driven, same conventions (Cat: P positive, N negative, B boundary, C corne | L3 | P | TCP bind on a listener port | bind `:8889` in the child | bind succeeds (ruleset grants `BIND_TCP` on the listener ports only) | | L4 | N | TCP bind on a foreign port | bind `:44444` | `EACCES` — a smuggled listener cannot open a C2 port even with root | | L5 | C | ABI < 4 kernel | probed network ABI 0–3 | warn + `xtcp2_sandbox_net 0` + continue (degrade, never a failed daemon — mirrors D4/D5) | -| L6 | C | UDP is untouched by ABI 4 rules | DNS lookup through `/etc/resolv.conf` resolver | succeeds — documented gap, governed by Layer 3 (SG) instead | -| L7 | P | UDS connect unaffected by TCP rules | dial the writer socket | succeeds (network rules are TCP-only; FS grants govern sockets) | +| L6 | C | UDP is untouched by ABI 4 rules | DNS lookup through `/etc/resolv.conf` resolver | succeeds — documented gap, governed by Layer 3 (collector-scoped nftables) instead | +| L7 | P | UDS connect unaffected by TCP rules | dial the writer socket | succeeds (net rules cover TCP/UDP ports only; pathname sockets are governed by `RESOLVE_UNIX` from ABI 9, unmediated below) | | L8 | C | seccomp-blocked `landlock_add_rule` for net rules | injected EPERM at net-rule add | FS ruleset still applied; warn specifically about the missing net rules (`xtcp2_sandbox_net 0`, `xtcp2_sandbox_active 1`) | -### Test M — fleet egress and exposure (microVM + security groups, recorded) +### Test M — fleet egress and exposure (microVM + host firewall, recorded) | ID | Cat | Description | Input / setup | Expected outcome | |---|---|---|---|---| | M1 | P | S3 upload path intact | trigger ASN refresh on a canary | upload succeeds to the per-DC endpoint on 443 | -| M2 | N | arbitrary egress blocked at the host layer | `curl http://:443` and `:8080` from the collector container | denied by SG (443 non-S3 included — closes Landlock's port-only gap) | +| M2 | N | arbitrary egress blocked at the host layer | `curl http://:443` and `:8080` from the collector container | denied by the collector-scoped nftables rule (443 non-S3 included — closes Landlock's port-only gap) | | M3 | P | DNS intact | resolver query for the S3 endpoint | resolves (UDP 53 untouched) | | M4 | P | metrics scrape intact | vmagent scrape of 9088 from the docker-bridge subnets | scrape succeeds | | M5 | N | `:8889` from a non-operator subnet | connect from a random host | denied at SG/UFW | | M6 | P | `:8889` from operator subnet with the admin token | authorized `Set` (tuning field) | applies, audited (PR-5 line shipped) | | M7 | N | unauthenticated `Set` replay on the converged fleet | unauthenticated `Set` to `:8889`, v1.x shape | rejected (raw-token auth) — unauthenticated access is dead fleet-wide | | M8 | B | state-dir quota enforced | writer spammed with schema-valid 64 KiB writes past the project quota | writes fail with quota errors once the limit is hit; `Xtcp2StateDirFilling` had already fired (alert precedes quota); collector unaffected | +| M10 | P | co-located workloads unaffected | the same `curl`s from a non-collector container and a host process | succeed — the egress rule matches only the collector's cgroup | | M9 | C | per-target writer rate cap | 20 valid baseline writes in an hour | first ~12 accepted, rest rejected `rate_limited` with counter — no disk-fill grind through the seq mechanism | ### Test N — CI gates (each gate's failure mode) @@ -154,27 +155,27 @@ Table-driven, same conventions (Cat: P positive, N negative, B boundary, C corne | OD-9 | drop network stdlib destinations from the production flavor | **yes** (closes here): `dest_tcp/udp/http/unix/unixgram` tags; prod keeps `s3parquet` + `null`/`stdout` | | OD-6 | hatch lifetime | closes with PR-14 after one clean release cycle (J4 proves zero use) | | OD-7 | state-dir quota | XFS project quota + writerd rate cap (M8/M9) | -| new | Landlock net-rule scope | ship port-scoped 443-connect + listener-port bind; host-level pinning stays with SG (documented division: kernel covers ports, cloud covers hosts) | -| new | per-DC S3 endpoint list source of truth | the infra repo variables, consumed by SG + frozen S3 fields (single list, no drift) | +| new | Landlock net-rule scope | ship port-scoped 443-connect + listener-port bind; host-level pinning stays with the collector-scoped nftables rule (documented division: Landlock covers ports, nftables covers hosts) | +| new | per-DC S3 endpoint list source of truth | the infra repo variables, consumed by the egress rule + frozen S3 fields (single list, no drift) | | new | egress alert owner | the host Falco pipeline | ## Release checklist - [ ] PR-14 merged after J4 verified zero hatch use across a full release cycle - [ ] PR-15 build side: five destinations tagged, flavor lists updated, image rebuilt; K1–K7 green -- [ ] PR-15 network side: SG changes canaried **one DC first** (the riskiest change in the program — a wrong rule breaks S3 uploads fleet-wide; rollback = revert SG), L1–L8 green where kernels allow, M1–M4 recorded +- [ ] PR-15 network side: egress-rule changes canaried **one DC first** (the riskiest change in the program — a wrong rule breaks S3 uploads fleet-wide; rollback = remove the rule); M10 proves co-located workloads are untouched, L1–L8 green where kernels allow, M1–M4 recorded - [ ] PR-16: stragglers migrated, v1.x branches + image tags retired, M5–M7 recorded - [ ] PR-17: all six gates wired as required checks + weekly scheduled proof; N1–N6 exercised with a deliberately-bad PR before merge - [ ] [metrics.md](../metrics.md): `xtcp2_sandbox_net`, writerd rate/quota counters documented -- [ ] Rollback matrix from [implementation-plan.md](implementation-plan.md) updated: each Phase 4 layer rolls back independently (SG revert; flavor bump back; hatch re-add is a code revert, not a flag) +- [ ] Rollback matrix from [implementation-plan.md](implementation-plan.md) updated: each Phase 4 layer rolls back independently (egress-rule revert; flavor bump back; hatch re-add is a code revert, not a flag) ## Definition of done The [end-state verification checklist](plan-configservice-set-hardening.md#end-state-verification-checklist), executed on a canary host **holding the admin token** — every primitive must fail safely at its layer: - [ ] **Filesystem:** `Set` with `dest:"file:…"` (any of the advisory §4.7 path idioms incl. `/proc/1/root/root/.ssh/authorized_keys`) → validation rejects; crippled build (validation stubbed) → kernel denies; nothing written anywhere on the host -- [ ] **State:** baseline + ipmeta land via the writer only, owned `65534:65534` `0600`; quota + rate cap enforce (M8/M9) -- [ ] **Egress:** `Set` with `dest:"http://attacker:8080"` → `FailedPrecondition`, not compiled in the prod flavor; crippled build + RCE-simulated dial → Landlock denies the port; SG denies the host; the outbound-connection alert fires — four layers, each independently sufficient +- [ ] **State:** baseline + ipmeta land via the writer only, owned `65534:xtcp2-state` `0640`; quota + rate cap enforce (M8/M9) +- [ ] **Egress:** `Set` with `dest:"http://attacker:8080"` → `FailedPrecondition`, not compiled in the prod flavor; crippled build + RCE-simulated dial → Landlock denies the port; the collector-scoped nftables rule denies the host; the outbound-connection alert fires — four layers, each independently sufficient - [ ] **Frozen fields:** `dest`, S3 endpoints, listeners, all path fields → rejected unconditionally, no flag exists (J1/J2); a tuning-only `Set` still applies and audits - [ ] **Auth:** unauthenticated `Set` on `:8889` rejected everywhere (M7); `:8889` unreachable off-operator subnets (M5); no v1.x host remains in inventory - [ ] **Standing:** weekly crippled-build proof green (N5); a write-site PR fails CI until the inventory row exists (N1/N2); deleting a regression suite fails CI (N6) diff --git a/docs/security-hardening/plan-configservice-set-hardening.md b/docs/security-hardening/plan-configservice-set-hardening.md index 57918546..9d07aeb1 100644 --- a/docs/security-hardening/plan-configservice-set-hardening.md +++ b/docs/security-hardening/plan-configservice-set-hardening.md @@ -91,7 +91,7 @@ Belt layer 1: shape validation in `proto/xtcp_config/v1/xtcp_config.proto`, enfo - [ ] `docker_socket_path` (202), `lldpd_socket_path` (211) - [ ] `dest` (60): scheme allowlist pattern mirroring `knownSchemes` (`pkg/xtcp/destinations_core.go:70`): `^(null|stdout|stderr|file|tcp|http|https|udp|unix|unixgram|kafka|nats|nsq|valkey|s3parquet)://[^[:cntrl:]]+$` - [ ] `ListenerEndpoint`: charset pattern on `address`; message-level CEL cross-rule — when `network == unix`, address must match the path pattern -- [ ] Tighten `unix_socket_mode` from `lte 511` → `lte 384` (0600) — pending [OD-1](#open-decisions) +- [ ] Replace `unix_socket_mode` `lte 511` with the allowlist `in: [0, 384]` (default/0600) — not `lte 384`, which accepts world-writable 0577 — pending [OD-1](#open-decisions) - [ ] Regenerate all targets (`gen/go`, `gen/dart`, …) via the existing make/nix protobuf targets - [ ] Golden-config gate: validate the **fleet's actual configs** (ansible role defaults + per-DC env, incl. the v1.x revision's) against the new rules in CI — catches any legitimate path we just broke, before release (part of W8) @@ -103,8 +103,8 @@ Belt layer 1: shape validation in `proto/xtcp_config/v1/xtcp_config.proto`, enfo Belt layer 2: semantics that CEL cannot express (deployment roots, symlink race, running-config diff). This is the workstream that closes I1–I6, I8 outright. - [ ] New file `pkg/xtcp/set_path_policy.go`: - - [ ] `PathPolicy` struct: roots from a new repeatable daemon flag `-writableDir` (register alongside existing flags, `cmd/xtcp2/xtcp2.go` ~line 454); empty default = no filesystem-writing config allowed - - [ ] `ValidatePaths(cfg *xtcp_config.XtcpConfig) error`: for every path-bearing field — canonicalize (`filepath.Clean` + `filepath.EvalSymlinks`, reject on error), verify each component via `os.Lstat` is not a symlink, then require containment in a root + - [ ] `PathPolicy` struct: write roots from a new repeatable daemon flag `-writableDir`, read roots from `-readableDir` (register alongside existing flags, `cmd/xtcp2/xtcp2.go` ~line 454); empty `-writableDir` = no filesystem-writing config allowed; read/connect fields (`docker_socket_path`, `lldpd_socket_path`, `asn_db_path`, `ipmeta_bootstrap_path`, `xtcp_proto_file`) check against read roots, never write roots + - [ ] `ValidatePaths(cfg *xtcp_config.XtcpConfig) error`: for every path-bearing field — canonicalize (`filepath.Clean`, then `filepath.EvalSymlinks` on the deepest existing ancestor — a not-yet-created output is valid), verify each existing component via `os.Lstat` is not a symlink, then require containment in a root of the field's class - [ ] Validate `dest` scheme against compiled-in `knownSchemes`; reject `file:` unless `-allowFileDest` is set - [ ] Wire the policy: - [ ] `Set` handler (`pkg/xtcp/grpc_configService.go`, after existing validation, before `reconfigureFunc`) — returns `codes.InvalidArgument` with the offending field named @@ -132,7 +132,7 @@ Belt layer 4: kernel enforcement that survives `syscall.Exec` (soft restart) and - [ ] Dependency: `github.com/landlock-lsm/go-landlock` (check nixpkgs availability; vendor otherwise) - [ ] New package `pkg/sandbox`: - - [ ] `Constrain(opts)` called once at daemon startup (`cmd/xtcp2/xtcp2.go`), before any config-driven writer is constructed: read-only rules for `/proc`, `/sys`, `/run/netns`, `/run/docker/netns`, `/run/docker.sock`, the lldpd socket dir, ASN bootstrap/cache paths, listener token files; write rules only for `-writableDir` roots (until W5 lands, after which: none) + - [ ] `Constrain(opts)` called once at first start (`cmd/xtcp2/xtcp2.go`; soft restarts inherit the domain via an fd marker — re-applying would hit the kernel's 16-layer cap), before any config-driven writer is constructed: read-only rules for `/proc`, `/sys`, `/run/netns`, `/run/docker/netns`, `/run/docker.sock`, the lldpd socket dir, ASN bootstrap/cache paths, listener token files; write rules only for `-writableDir` roots (until W5 lands, after which: none) - [ ] Carve-outs: UDS listener dir (only when a unix listener is configured); capture dir (only when `write_files` > 0 with a root configured) - [ ] ABI feature detection: kernel < 5.13 → log a warning, set `xtcp2_sandbox_active 0`, continue (degrade-not-exit); otherwise `1` — new Prometheus gauge (add to [metrics.md](../metrics.md) in W9) - [ ] Soft-restart interplay test: restriction persists across `syscall.Exec` (unit test with a re-exec of a helper binary attempting an out-of-root write) @@ -150,13 +150,13 @@ Belt layer 5: the collector ends up with zero write rules; the two legitimate wr - [ ] New daemon mode `xtcp2 -writer` (`cmd/xtcp2/writer.go`): - [ ] gRPC server on UDS `/run/xtcp2/writer.sock`; `SO_PEERCRED` on accept, reject unexpected uids - [ ] Fixed targets inside its root: `link-monitor-baseline.json`, `ipmeta/current.lookup.parquet.zst` (+ previous) - - [ ] Size caps (baseline ≤ 64 KiB = `baseline.MaxFileBytes`; cache ≤ 192 MiB), per-RPC rate limits, monotonic version numbers (reject rollback/replay), mode 0600 + - [ ] Size caps (baseline ≤ 64 KiB = `baseline.MaxFileBytes`; cache ≤ 192 MiB), per-RPC rate limits, monotonic version numbers (reject rollback/replay), mode 0640 group `xtcp2-state` (the collector, uid 0 without `CAP_DAC_OVERRIDE`, reads via the group) - [ ] Reuse the existing atomic-write discipline (`pkg/linkmonitor/internal/baseline/files.go`): `CreateTemp` → write → `fsync` → `rename` → dir fsync; baseline `flock` moves inside the writer - [ ] Collector side: - [ ] `baseline` store: new RPC-backed implementation behind the existing injectable `temporary`/`rename` seam (`pkg/linkmonitor/internal/baseline/files.go:46`); collector reads the file directly through the ro mount - [ ] `ipasn.PublishLookupCache` (`pkg/ipasn/ipasn.go:373`): route content through the writer RPC; keep the reload/rollback semantics - [ ] Failure mode: writer unreachable → log + `xtcp2_state_writer_errors_total` counter + degrade-not-exit (monitor keeps last-known state) — matches today's behavior when the baseline is unwritable -- [ ] Deployment (W7): second container `xtcp2_writer` (same image, state dir `:rw`, socket dir shared, no network); collector container flips state dir to `:ro` and mounts the socket dir; systemd units `xtcp2-writer.service` (`RootDirectory=`, uid 65534, `cap_drop ALL`, `RestrictAddressFamilies=AF_UNIX`) + ordering for bare metal +- [ ] Deployment (W7): second container `xtcp2_writer` (same image, state dir `:rw`, socket dir shared, no network); collector container flips state dir to `:ro` and mounts the socket dir; systemd units `xtcp2-writer.service` (`RootDirectory=`, uid 65534 + group `xtcp2-state` shared with the collector, `cap_drop ALL`, `RestrictAddressFamilies=AF_UNIX`) + ordering for bare metal - [ ] Remove `baseline_file`/`ipmeta_cache_path` from the `Set`-mutable surface entirely (they become writer-owned constants; the fields stay in the proto for compat but are frozen — W2 list already includes them) **Acceptance**: collector's Landlock ruleset has zero write rules; writer tests prove no request field can influence the target path (structurally: no such field exists); kill -9 mid-write leaves old baseline intact; version replay rejected. @@ -179,7 +179,7 @@ Changes in the deployment repo (`ansible/roles/monitoring/xtcp2/`) and the image - [ ] Retire the v1.x revision: make `LISTENER_AUTH_MODE` impossible to disable via role defaults; add a `test_xtcp2_fleet_gate.py` assertion that auth env is present - [ ] Bind guidance: prefer UDS for the control plane (`xtcp2ctl` workflow documented), else loopback + `tasks/firewall.yml` rules for 8889 with TTL 3 - [ ] I11: state-dir guard — minimum: node-level alert on `/var/lib/xtcp2` usage (vmagent); preferred: XFS project quota (or systemd quota for bare metal) sized to ~2× the ASN cache (~200 MiB headroom) -- [ ] Egress pinning: restrict container egress to the per-DC S3 endpoints (network policy / security group); pair with W2's frozen `s3_endpoint` +- [ ] Egress pinning: restrict the collector's egress to the per-DC S3 endpoints (nftables rule on its cgroup — host netns, so not container network policy; not host-wide SG/UFW, which would hit co-located workloads); pair with W2's frozen `s3_endpoint` - [ ] W5 deployment: second container + `:ro` collector state dir + shared socket dir; canary via the existing fleet-gate mechanism; revert via `xtcp2_revert.yml` - [ ] Pin the new image tag in the image repo; bump the deployment repo role defaults in lockstep (role gate test asserts image↔config compatibility) @@ -190,7 +190,7 @@ Changes in the deployment repo (`ansible/roles/monitoring/xtcp2/`) and the image - [ ] Hostile-path table test (unit, `pkg/xtcp`): `../etc/passwd`, `/run/x/../../`, NUL, `…/` (unicode ellipsis), `/proc/self/mem`, trailing `/`, `//`, symlink-into-root, `/var/lib/xtcp2/ok` (accept), `/tmp/cap` (accept only when a root is configured) — assert proto layer (W1) and Go policy (W2) verdicts - [ ] `Set` policy tests: path fields rejected; frozen-field change rejected; valid tuning `Set` applies; escape-hatch flag honored - [ ] Interceptor guard test (W3) + per-service invalid-message cases -- [ ] Landlock: unit re-exec persistence test + microVM scenario (W4) +- [ ] Landlock: unit re-exec persistence test (> 16 consecutive soft restarts) + microVM scenario (W4) - [ ] Writer tests (W5): size cap, rate limit, version replay, kill -9 atomicity, peer-credential rejection - [ ] **Fleet golden-config gate (CI)**: ansible role renders real configs (all DCs, both role revisions) → `protovalidate` them in CI on every proto change; blocks a rule that would reject production - [ ] Advisory regression: end-to-end gRPC test running the five primitives of advisory §2 and asserting each is blocked at its layer @@ -237,7 +237,7 @@ With admin token in hand on a canary host, every item must fail safely: - [ ] `Set` with `capture_path = /var/spool/cron/` + `write_files = 1` → rejected; no file created - [ ] `Set` with `link_monitor.baseline_file = /run/pwn.json` → rejected - [ ] `Set` with `ipmeta_cache_path = /etc/xtcp2/pwn/current.lookup.parquet.zst` → rejected; no directories created anywhere new -- [ ] `Set` with unix listener `address = /etc/ld.so.preload`, `unix_socket_mode = 0777` → rejected (pattern + `lte 384`) +- [ ] `Set` with unix listener `address = /etc/ld.so.preload`, `unix_socket_mode = 0777` → rejected (pattern + mode allowlist); `unix_socket_mode = 0577` alone → rejected - [ ] `Set` changing `s3_endpoint` to an attacker host → `FailedPrecondition` (frozen), even with `-setAllowFrozenFields` unset network policy blocks the egress (W7) - [ ] Legit tuning `Set` (poll frequency) applies and the daemon soft-restarts with the Landlock ruleset intact (`xtcp2_sandbox_active` stays 1) - [ ] Direct process check: as the collector uid, `os.WriteFile("/tmp/x")` fails (zero write rules, post-W5); the writer still publishes a fresh ASN cache @@ -249,7 +249,7 @@ With admin token in hand on a canary host, every item must fail safely: | # | Decision | Default proposal | |---|---|---| -| OD-1 | `unix_socket_mode` cap: 0600 or allow 0660 (shared-group socket)? | 0600 (`lte 384`); relax only if a consumer needs group access | +| OD-1 | `unix_socket_mode` cap: 0600 or allow 0660 (shared-group socket)? | 0600 (`in: [0, 384]`); add 432 (0660) to the list only if a consumer needs group access — never a `lte` bound | | OD-2 | Landlock-absent kernel behavior: warn-and-continue vs refuse to start | Warn + `xtcp2_sandbox_active 0` + alert; fleet is ≥6.8 so this is a bare-metal edge case | | OD-3 | `file:` destination: keep behind `-allowFileDest` or remove entirely in the s3parquet build flavor | Keep behind flag (build flavors already gate schemes) | | OD-4 | Writer transport: UDS gRPC (chosen) vs loopback TCP | UDS — peer credentials, no port surface; loopback TCP rejected because every host process shares it | diff --git a/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md b/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md index ba34ce4d..203f2f3e 100644 --- a/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md +++ b/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md @@ -238,7 +238,7 @@ blocks relative paths, dot-dot traversal, empty `//` segments, NULs, spaces, and | `docker_socket_path` (202), `lldpd_socket_path` (211) | len rules | `pattern` above | | `dest` (60) | `min_len 4, max_len 512`; scheme checked only in Go (`pkg/xtcp/input_validation.go`) | `pattern: "^(null|stdout|stderr|file|tcp|http|https|udp|unix|unixgram|kafka|nats|nsq|valkey|s3parquet)://[^[:cntrl:]]+$"` (mirror of `knownSchemes`, `pkg/xtcp/destinations_core.go:70`) | | `ListenerEndpoint.address` (2) | `max_len 512` | charset pattern; when `network: unix`, path shape as above (message-level CEL cross-rule) | -| `unix_socket_mode` (3) | `lte 511` | tighten to `lte 384` (0600) — nothing legitimate needs a world-writable daemon socket | +| `unix_socket_mode` (3) | `lte 511` | allowlist `in: [0, 384]` (default/0600) — a `lte` bound is not a permission check: 0577 (383) passes `lte 384` yet is world-writable | RE2 has no lookahead, so ".."-blocking must be done via the segment pattern, not a negative assertion — the shape above is the standard idiom. Fuzz the pattern with the hostile-path table in [§8](#8-testing-plan). @@ -246,7 +246,7 @@ RE2 has no lookahead, so ".."-blocking must be done via the segment pattern, not Add a post-validation gate in `pkg/xtcp/grpc_configService.go` before `reconfigureFunc` is invoked (and mirror it at startup in `cmd/xtcp2/xtcp2.go` config load, so a hostile env-carried config can't bypass it after soft restart): -1. **Writable-root allowlist.** New daemon flag `-writableDir` (repeatable), default empty. Every path-bearing field above must `filepath.Clean` + `filepath.EvalSymlinks` (reject on error) into one of the roots. With no roots configured, all filesystem-writing config is rejected — `dest` must not use `file:`, `write_files` must be 0, etc. +1. **Separate write and read roots.** New repeatable daemon flags `-writableDir` (default empty) and `-readableDir` (defaults: the stock paths' parent dirs). Write fields (`capture_path`, `baseline_file`, `ipmeta_cache_path`, unix listener `address`, `file:` `dest`) must resolve into a writable root; read/connect fields (`docker_socket_path`, `lldpd_socket_path`, `asn_db_path`, `ipmeta_bootstrap_path`, `xtcp_proto_file`) into a readable or writable root — requiring `/run/docker.sock` under a writable root would reject every stock config. Canonicalize with `filepath.Clean`, then `filepath.EvalSymlinks` on the **deepest existing ancestor** (an output such as the first baseline does not exist yet, and `EvalSymlinks` fails on a missing path); the missing remainder is accepted. With no roots configured, all filesystem-writing config is rejected — `dest` must not use `file:`, `write_files` must be 0, etc. 2. **Symlink discipline.** After canonicalization, verify no path component is a symlink (component-wise `os.Lstat`), so a previously-planted symlink can't redirect a later write. 3. **Frozen-field diff.** Compare incoming vs running config for the security-critical set and reject changes at runtime (operators change these via redeploy): `dest`, `s3_endpoint`/`s3_bucket`/S3 credential fields, kafka/nats/nsq/valkey options, both listener endpoints + auth settings, `docker_socket_path`, `lldpd_socket_path`, `asn_db_path`, `ipmeta_*`, `baseline_file`, `capture_path`, `write_files`, `xtcp_proto_file`. This simultaneously kills the non-`Set`-FS but adjacent **exfil** class (repointing `dest`/`s3_endpoint` to attacker endpoints from a stolen admin token) and makes the writable-roots check for those fields a startup-only concern. 4. **`file:` dest goes debug-only.** `dest` scheme `file` is a debugging convenience (`docs/output-and-destinations.md`); require an explicit `-allowFileDest` flag for it to validate. @@ -279,13 +279,13 @@ Streaming RPCs validate in a `RecvMsg` wrapper or at handler entry (the flat-rec ### 5.4 Layer 4 — kernel self-confinement: Landlock (and chroot alternatives) -Validation is code; Landlock is a kernel guarantee. A Landlock ruleset applied at daemon startup survives `fork` **and `execve`** — i.e. it persists through `Set`'s soft restart — and cannot be escaped even by a privileged process (a process is always confined by its own ruleset). Using `github.com/landlock-lsm/go-landlock`: +Validation is code; Landlock is a kernel guarantee. A Landlock ruleset applied at daemon startup survives `fork` **and `execve`** — i.e. it persists through `Set`'s soft restart — and cannot be escaped even by a privileged process (a process is always confined by its own ruleset). Because it is inherited, it is applied once: re-applying on every soft restart would stack a layer per restart, and the kernel caps a domain at 16 layers. Using `github.com/landlock-lsm/go-landlock`: - Read-only access to everything the daemon needs: `/proc` (namespace discovery), `/sys`, `/run/netns`, `/run/docker/netns`, `/run/docker.sock`, the lldpd socket dir, ASN bootstrap/cache paths, token files. - **No write access anywhere** by default (or, until L5 lands: write access only beneath `-writableDir` roots, e.g. `/var/lib/xtcp2`). - Exception carve-outs: a UDS listener dir (`/run/xtcp2`) if UDS listeners are enabled; `/tmp` only if `write_files`/capture is explicitly enabled with a root there. -Fleet kernels are Ubuntu 24.04 (6.8) — Landlock ABI ≥4, comfortably sufficient for filesystem rules (v1, kernel 5.13+). After this lands, even a fully attacker-controlled `Set` (or a future validation bypass) cannot create, open-for-write, or chmod a file outside the state dir: the syscalls fail with `EACCES`. +Fleet kernels are Ubuntu 24.04 (6.8) — Landlock ABI ≥4, comfortably sufficient for filesystem rules (v1, kernel 5.13+). After this lands, even a fully attacker-controlled `Set` (or a future validation bypass) cannot create, open-for-write, rename or unlink a file outside the state dir: the syscalls fail with `EACCES`. The gaps are documented guarantees, not bugs: `chmod`/`chown`/xattrs/`utimes` are unmediated at every ABI; `truncate(2)`/`O_TRUNC` only from ABI 3; pathname-UDS `connect(2)` only from ABI 9 (`RESOLVE_UNIX`) — on ABI 4 every pathname socket, `/run/docker.sock` included, stays reachable. Why Landlock over the classical alternatives for the **daemon** process: @@ -359,7 +359,7 @@ Rules enforced by the writer, in code: - **Failure mode:** writer down ⇒ collector logs, bumps a counter, keeps monitoring with its last-known state — identical to today's degrade-not-exit behavior when the baseline is unwritable. `Set` is unaffected. - **Rollback:** the writer keeps `previous.lookup.parquet.zst` semantics exactly as today. -**Why do it, honestly.** Landlock (L4) already blocks every `Set` write primitive for ~10% of the effort. The split adds three things Landlock cannot: (a) **state integrity against a compromised collector** — an RCE inside the daemon cannot rewrite the baseline to hide link outages, poison the ASN cache that feeds fleet-wide enrichment, or plant persistence; (b) the collector needs **zero** write rules, so its Landlock policy is trivial to audit; (c) the config surface itself shrinks — `baseline_file`/`ipmeta_cache_path` stop being runtime-mutable paths entirely. It does **not** contain the collector's `SYS_ADMIN`/`NET_ADMIN`/docker-sock reach (see [§7](#7-honest-limits--what-this-does-not-fix)); it removes the filesystem angle only. Sequence it last, after L1–L4. +**Why do it, honestly.** Landlock (L4) already blocks every `Set` write primitive for ~10% of the effort. The split adds three things Landlock cannot: (a) **a narrower state-write surface for a compromised collector** — direct file writes are gone, so planting files or arbitrary bytes in the state dir needs the writer's two fixed targets and their schema checks. This is not state integrity: the writer still accepts schema-valid baseline/ipmeta content at the next `seq` (a baseline that hides a link outage, a poisoned-but-well-formed ASN cache), and the collector's `docker.sock` can have dockerd write the state dir — or any host path — without touching the writer; (b) the collector needs **zero** write rules, so its Landlock policy is trivial to audit; (c) the config surface itself shrinks — `baseline_file`/`ipmeta_cache_path` stop being runtime-mutable paths entirely. It does **not** contain the collector's `SYS_ADMIN`/`NET_ADMIN`/docker-sock reach (see [§7](#7-honest-limits--what-this-does-not-fix)); it removes the filesystem angle only. Sequence it last, after L1–L4. ### 5.6 Layer 6 — deployment and runtime guardrails @@ -367,7 +367,7 @@ Rules enforced by the writer, in code: - **gRPC reachability.** Prefer a UDS listener with 0600 perms for the control plane; if TCP is required (the `xtcp2ctl --network host` workflow), bind loopback and firewall 8889, or keep host-netns with auth + TTL 3 and audit. - **State-dir quota + alerting.** No quota exists today; a `Set`-driven disk-fill of `/var/lib/xtcp2` exhausts the host rootfs. Add a systemd `MaximumDiskUsage`-style quota / XFS project quota (or at minimum size monitoring + alert) on the state dir. - **Audit `Set`.** Log every applied `Set` (which fields changed, by which token id) to the daemon log and ship it; `Set` is rare enough that this is low-noise and catches a stolen admin token quickly. -- **Egress pinning.** Freeze `s3_endpoint`/`dest` (L2) and, ideally, network-policy the container to its known S3 egress so the frozen-field rule can't be sidestepped at deploy time either. +- **Egress pinning.** Freeze `s3_endpoint`/`dest` (L2) and, ideally, pin the collector's egress to its known S3 endpoints with an nftables rule matched on its cgroup (it runs in the host netns; host-wide security groups or UFW would also restrict co-located workloads) so the frozen-field rule can't be sidestepped at deploy time either. - **Read-only where possible.** With L5 in place, the collector container mounts the entire host `:ro` — nothing writable remains. - **Rotate tokens** (`xtcp2_rotate_listener_tokens`) on any suspected exposure; the admin token is the keys to `Set`. @@ -394,19 +394,21 @@ After this layer, a `Set` call cannot even *express* a `file:` destination or a | Threat | L1 proto rules | L2 path policy + frozen fields | L3 interceptor | L4 Landlock | L5 writer split | L6 deploy | |---|---|---|---|---|---|---| | Write to arbitrary path (`file:` dest, `capture_path`) | ✅ shape | ✅ root allowlist + freeze | — (enforces L1) | ✅ kernel | ✅ API has no path | — | -| Write under `/var/lib/xtcp2` (baseline poison, cache poison, disk-fill) | ❌ | ❌ (legit use) | — | ❌ (dir allowed until L5) | ✅ collector can't write | ✅ quota + audit | +| Write under `/var/lib/xtcp2` (baseline poison, cache poison, disk-fill) | ❌ | ❌ (legit use) | — | ❌ (dir allowed until L5) | partial — no direct writes; schema-valid writes via the writer and `docker.sock` remain | ✅ quota + audit | | MkdirAll anywhere (`ipmeta_cache_path`) | ✅ shape | ✅ | — | ✅ | ✅ | — | | Planted 0777 unix socket | ✅ charset | ✅ root allowlist | — | ✅ | ✅ | — | | Symlink redirect of a later write | ❌ (shape ok) | ✅ EvalSymlinks + component Lstat | — | ✅ (read-only dirs) | ✅ chroot | — | | Unvalidated future RPC input | — | — | ✅ | — | — | — | | DEST/S3 endpoint repoint exfil | — (scheme ok) | ✅ frozen fields | — | ❌ (network, not fs) | ❌ | ✅ egress pinning | -| RCE in daemon → fs writes/persistence | — | — | — | ✅ (except state dir) | ✅ (nothing writable) | — | +| RCE in daemon → fs writes/persistence | — | — | — | partial — except state dir; `docker.sock` bypasses | partial — no writable mount; `docker.sock` bypasses | — | + +A compromised collector that speaks the Docker API bypasses L4 and L5: dockerd performs the write, outside the collector's Landlock domain and without the writer (§7). Layer 0 (§5.7) removes primitives 1–2 at the binary level — the production image simply contains no code that opens files for writing on `Set`'s behalf, so no validation bypass can reach it. ## 7. Honest limits — what this does not fix -- **The collector's own capabilities.** `CAP_NET_ADMIN` + `CAP_SYS_ADMIN` on host namespaces plus `/run/docker.sock:ro` mean a fully compromised collector is host-equivalent regardless of L1–L5. L5 removes its *filesystem* angle; the netns/docker angle is inherent to the product's function (see [§4.3](#43-what-net_admin--sys_admin-actually-grant)). +- **The collector's own capabilities.** `CAP_NET_ADMIN` + `CAP_SYS_ADMIN` on host namespaces plus `/run/docker.sock:ro` mean a fully compromised collector is host-equivalent regardless of L1–L5. L5 removes its *filesystem* angle; the netns/docker angle is inherent to the product's function (see [§4.3](#43-what-net_admin--sys_admin-actually-grant)). The same holds for L4/L5: they confine the collector's own syscalls, not what dockerd does on its request — and because the writer accepts schema-valid content at the next `seq`, L5 bounds the *shape* of state writes, not their truth. - **Egress is only frozen, not confined.** L2 stops runtime repointing of `dest`/`s3_endpoint`; it cannot stop a redeploy by whoever controls ansible. L6 egress pinning narrows that to trusted operators. - **SigV4 against attacker endpoints.** With the preserved S3 keys and a repointed endpoint, the minio client signs requests to the attacker; secrets are not disclosed, but data and the access key (inside the signature) are. Frozen fields close the runtime path. - **A stolen admin token still holds every non-frozen `Set` field** (poll knobs, buffer sizes — DoS-adjacent). Rotation + audit (L6) is the answer there. From bcff680185260c8f0966032769bce8cdc24ba901 Mon Sep 17 00:00:00 2001 From: "randomizedcoder dave.seddon.ca@gmail.com" Date: Mon, 9 Nov 2026 08:42:12 -0800 Subject: [PATCH 3/7] docs(security-hardening): setns scope, AppArmor load-bearing, consistency - setns: SYS_ADMIN enters net/UTS/IPC/cgroup/time namespaces only; mount needs CAP_SYS_CHROOT, PID only descendants. - :ro binds and the ro rootfs hold against SYS_ADMIN only because AppArmor docker-default denies mount; add it to the runtime table, restrictions and a fleet-gate regression guard (PidMode, ReadonlyRootfs, AppArmorProfile, remount probe). - dest pattern uses one colon (matches Phase 1); ipmeta cap 64 MiB everywhere; OD-3 aligned with W0; Set on an uncompiled scheme returns FailedPrecondition, not a restart loop; I15 renumbered to I14. - Phase 3 frame tests match the [len u32] + proto envelope (min_len 1, oneof required). Co-Authored-By: Claude Opus 5.5 --- .../security-hardening/implementation-plan.md | 2 +- .../phase-1-code-hardening.md | 2 +- .../phase-3-writer-split.md | 17 ++++++----- .../plan-configservice-set-hardening.md | 15 +++++----- ...ulnerability-configservice-set-fs-write.md | 30 ++++++++++--------- 5 files changed, 35 insertions(+), 31 deletions(-) diff --git a/docs/security-hardening/implementation-plan.md b/docs/security-hardening/implementation-plan.md index 81ff068e..9f16472b 100644 --- a/docs/security-hardening/implementation-plan.md +++ b/docs/security-hardening/implementation-plan.md @@ -123,7 +123,7 @@ PRs (repo `xtcp2` unless noted; each lands with its tests and doc updates from W | I11 no state-dir quota | Phase 2 (alert) → Phase 4 (quota per OD-7) | L6 | | I12 no `Set` audit trail | Phase 1 (PR-5) | L6 | | I13 collector holds needless write access | Phase 3 | L4, L5 | -| I15 debug writers in production image | Phase 1 (PR-1) | L0 | +| I14 debug writers in production image | Phase 1 (PR-1) | L0 | ## Timeline and effort diff --git a/docs/security-hardening/phase-1-code-hardening.md b/docs/security-hardening/phase-1-code-hardening.md index fc126b12..6107623b 100644 --- a/docs/security-hardening/phase-1-code-hardening.md +++ b/docs/security-hardening/phase-1-code-hardening.md @@ -24,7 +24,7 @@ ## Scope -Phase 1 is pure code and build changes — no new processes, no kernel confinement (Phase 2), no deployment-topology changes (Phase 3). It closes issues I1–I8 and I15 at the validation and binary level: +Phase 1 is pure code and build changes — no new processes, no kernel confinement (Phase 2), no deployment-topology changes (Phase 3). It closes issues I1–I8 and I14 at the validation and binary level: | Layer | Effect after Phase 1 | |---|---| diff --git a/docs/security-hardening/phase-3-writer-split.md b/docs/security-hardening/phase-3-writer-split.md index 44d3a081..e0b03b12 100644 --- a/docs/security-hardening/phase-3-writer-split.md +++ b/docs/security-hardening/phase-3-writer-split.md @@ -78,14 +78,15 @@ Frame: little-endian `[len u32]`, then one protobuf `Request`; reply frame `[len // proto/xtcp_writer/v1/writer.proto — message-only, no service definition. message WriteBaselineRequest { uint64 seq = 1 [(buf.validate.field).uint64.gte = 1]; - bytes payload = 2 [(buf.validate.field).bytes.max_len = 65536]; // == baseline.MaxFileBytes + bytes payload = 2 [(buf.validate.field).bytes = {min_len: 1, max_len: 65536}]; // == baseline.MaxFileBytes } message WriteIpmetaRequest { uint64 seq = 1 [(buf.validate.field).uint64.gte = 1]; - bytes payload = 2 [(buf.validate.field).bytes.max_len = 67108864]; // == -ipmetaMaxBytes default + bytes payload = 2 [(buf.validate.field).bytes = {min_len: 1, max_len: 67108864}]; // == -ipmetaMaxBytes default } message Request { oneof op { + option (buf.validate.oneof).required = true; PingRequest ping = 1; WriteBaselineRequest write_baseline = 2; WriteIpmetaRequest write_ipmeta = 3; @@ -155,18 +156,18 @@ Harness: `pkg/writerd` mirrors the injectable-`operations` seam from `baseline/f | A13 | B | ipmeta payload exactly at the cap | `len == ipmetaMaxBytes` | accepted | | A14 | N | ipmeta payload cap+1 | `len == ipmetaMaxBytes+1` | rejected `invalid` at `.Validate()` (`max_len`) | | A15 | C | bootstrap: target exists, no sidecar | writer-init run first | init imports (sidecar `0`), writer starts, write at seq 1 accepted | -| A16 | N | zero-length payload | op 1, `payloadLen 0` | rejected `malformed` | +| A16 | N | zero-length payload | `WriteBaselineRequest{seq: 1, payload: }` | rejected `invalid` at `.Validate()` (`min_len 1`) | ### Test B — framing (`TestWriterdFraming`) | ID | Cat | Description | Input / setup | Expected outcome | |---|---|---|---|---| | B1 | P | encode/decode round trip | valid frame with payload | parsed fields identical | -| B2 | N | truncated header | 16 of 17 header bytes | `malformed`, connection closed, no write attempted | -| B3 | N | unknown op | op `0xFF` | error reply `unknown_op` | -| B4 | N | payload length lies huge | `payloadLen 0xFFFFFFFF` | rejected against the global cap **before allocation** (no OOM), connection closed | -| B5 | C | payload shorter than declared, then EOF | header says 100 B, sender sends 50 and closes | `malformed`; no partial file written | -| B6 | C | trailing bytes after payload | valid frame + 4 junk bytes | `malformed` (strict framing) | +| B2 | N | truncated length prefix | 3 of the 4 `len` bytes, then EOF | `malformed`, connection closed, no write attempted | +| B3 | N | no op set | `Request` with an empty oneof (the oneof arm is the op; there is no op byte) | rejected `invalid` at `.Validate()` (`oneof required`) | +| B4 | N | frame length lies huge | `len 0xFFFFFFFF` | rejected against the global cap **before allocation** (no OOM), connection closed | +| B5 | C | payload shorter than declared, then EOF | `len` says 100 B, sender sends 50 and closes | `malformed`; no partial file written | +| B6 | C | trailing bytes after the envelope | valid frame + 4 junk bytes before the next `len` | `malformed` (strict framing) | | B7 | P | empty-payload op | `GET_STATS` frame | valid, stats reply | | B8 | B | error reply at max size | err text 1024 B | encoded fine; collector decodes it | | B9 | N | envelope violates protovalidate: seq 0 | `WriteBaselineRequest{seq: 0}` | rejected `invalid` at `.Validate()` (writer side; the collector side rejected it before send — same rules, both ends) | diff --git a/docs/security-hardening/plan-configservice-set-hardening.md b/docs/security-hardening/plan-configservice-set-hardening.md index 9d07aeb1..b628805c 100644 --- a/docs/security-hardening/plan-configservice-set-hardening.md +++ b/docs/security-hardening/plan-configservice-set-hardening.md @@ -62,7 +62,7 @@ | I11 | No quota on `/var/lib/xtcp2` → host-rootfs disk-fill | ansible role | Medium | W7 | | I12 | No audit trail of applied `Set` calls | `Set` handler | Medium | W6 | | I13 | Collector holds write access it does not need (state-integrity/persistence risk under RCE) | architecture | Medium | W4, W5 | -| I15 | `file:` destination (stdlib, no build tag) and the netlink capture writer (no build gate) are compiled into **every** production image | `pkg/xtcp/destinations_file.go`, `pkg/xtcp/netlinker.go:70`, [build-flavors.md](../build-flavors.md) | High | W0 | +| I14 | `file:` destination (stdlib, no build tag) and the netlink capture writer (no build gate) are compiled into **every** production image | `pkg/xtcp/destinations_file.go`, `pkg/xtcp/netlinker.go:70`, [build-flavors.md](../build-flavors.md) | High | W0 | ## Workstreams @@ -70,14 +70,14 @@ Advisory §5.7. Primitives #1/#2 live in code production never uses; remove it from the production binary. Fits the existing flavor axes ([build-flavors.md](../build-flavors.md)) — the `dest_` build-tag convention and the "not compiled in" error path already exist (`pkg/xtcp/destinations_core.go`). -- [ ] Move the `file` destination behind a `dest_file` build tag: `destinations_file.go`'s `init()` registration becomes tag-gated; `knownSchemes` keeps `file` so the error path can say "not compiled into this binary; rebuild with `-tags dest_file`" (a config naming an uncompiled scheme is a **fatal startup error**, `destinationLookupError` — a restart loop, never a write) +- [ ] Move the `file` destination behind a `dest_file` build tag: `destinations_file.go`'s `init()` registration becomes tag-gated; `knownSchemes` keeps `file` so the error path can say "not compiled into this binary; rebuild with `-tags dest_file`" (a config naming an uncompiled scheme is a **fatal startup error**, `destinationLookupError`; `Set` pre-checks compiled-in schemes and returns `FailedPrecondition` instead of soft-restarting — never a write, never a restart loop) - [ ] Gate the netlink capture writer behind a `capture` build tag (`netlinker.go` `captureToFileIfEnabled` + the `write_files`/`capture_path` wiring); `write_files > 0` without the tag = fatal startup error, mirroring the enricher convention - [ ] Update `nix/versions.nix` flavor matrix + [build-flavors.md](../build-flavors.md): the production `s3parquet` flavor excludes both tags; `full`/`-debug` keep them; `xtcp2-min` per [OD-8](#open-decisions) - [ ] Decide (OD-9) whether the production flavor also drops the network stdlib destinations (`tcp`, `udp`, `http`, `https`, `unix`, `unixgram`) — production needs `s3parquet` + `null`/`stdout`; makes the DEST-repoint exfil class structurally impossible - [ ] Image repo: build + publish the new production flavor; deployment repo pin bump; fleet-gate test asserts the flavor rejects `dest file:` and `write_files` - [ ] Docs: [output-and-destinations.md](../output-and-destinations.md) (`file:` now debug-flavor only), [build-flavors.md](../build-flavors.md), [grpc-api.md](../grpc-api.md) (`Set` behavior on uncompiled schemes) -**Acceptance**: on the production image, `Set` with `dest = "file:/tmp/pwn"` is a fatal startup error — no file is ever created; `write_files = 1` likewise; debug flavors unchanged. +**Acceptance**: on the production image, `Set` with `dest = "file:/tmp/pwn"` returns `FailedPrecondition` and the daemon keeps running; the same config at startup is a fatal error — no file is ever created; `write_files = 1` likewise; debug flavors unchanged. **Estimate**: 2–3 days (tags + nix matrix + image + role bump). **Risk**: a legitimate `file:`/`write_files` production use (none known — fleet uses `s3parquet`); golden-config gate (W8) catches any fleet config that breaks. ### W1 — protovalidate path rules (L1) @@ -89,7 +89,7 @@ Belt layer 1: shape validation in `proto/xtcp_config/v1/xtcp_config.proto`, enfo - [ ] `link_monitor.baseline_file` (LinkMonitorConfig field 2) - [ ] `ipmeta_cache_path` (244), `ipmeta_bootstrap_path` (243), `asn_db_path` (241) - [ ] `docker_socket_path` (202), `lldpd_socket_path` (211) -- [ ] `dest` (60): scheme allowlist pattern mirroring `knownSchemes` (`pkg/xtcp/destinations_core.go:70`): `^(null|stdout|stderr|file|tcp|http|https|udp|unix|unixgram|kafka|nats|nsq|valkey|s3parquet)://[^[:cntrl:]]+$` +- [ ] `dest` (60): scheme allowlist pattern mirroring `knownSchemes` (`pkg/xtcp/destinations_core.go:70`): `^(null|stdout|stderr|(tcp|udp|unix|unixgram|file|http|https|kafka|nats|nsq|valkey|s3parquet):[^[:cntrl:]]+)$` — one colon, not `://` (`file:/x`, `kafka:h:9092`; see [Phase 1 PR-2](phase-1-code-hardening.md)) - [ ] `ListenerEndpoint`: charset pattern on `address`; message-level CEL cross-rule — when `network == unix`, address must match the path pattern - [ ] Replace `unix_socket_mode` `lte 511` with the allowlist `in: [0, 384]` (default/0600) — not `lte 384`, which accepts world-writable 0577 — pending [OD-1](#open-decisions) - [ ] Regenerate all targets (`gen/go`, `gen/dart`, …) via the existing make/nix protobuf targets @@ -150,7 +150,7 @@ Belt layer 5: the collector ends up with zero write rules; the two legitimate wr - [ ] New daemon mode `xtcp2 -writer` (`cmd/xtcp2/writer.go`): - [ ] gRPC server on UDS `/run/xtcp2/writer.sock`; `SO_PEERCRED` on accept, reject unexpected uids - [ ] Fixed targets inside its root: `link-monitor-baseline.json`, `ipmeta/current.lookup.parquet.zst` (+ previous) - - [ ] Size caps (baseline ≤ 64 KiB = `baseline.MaxFileBytes`; cache ≤ 192 MiB), per-RPC rate limits, monotonic version numbers (reject rollback/replay), mode 0640 group `xtcp2-state` (the collector, uid 0 without `CAP_DAC_OVERRIDE`, reads via the group) + - [ ] Size caps (baseline ≤ 64 KiB = `baseline.MaxFileBytes`; cache ≤ `-ipmetaMaxBytes`, default 64 MiB), per-RPC rate limits, monotonic version numbers (reject rollback/replay), mode 0640 group `xtcp2-state` (the collector, uid 0 without `CAP_DAC_OVERRIDE`, reads via the group) - [ ] Reuse the existing atomic-write discipline (`pkg/linkmonitor/internal/baseline/files.go`): `CreateTemp` → write → `fsync` → `rename` → dir fsync; baseline `flock` moves inside the writer - [ ] Collector side: - [ ] `baseline` store: new RPC-backed implementation behind the existing injectable `temporary`/`rename` seam (`pkg/linkmonitor/internal/baseline/files.go:46`); collector reads the file directly through the ro mount @@ -181,6 +181,7 @@ Changes in the deployment repo (`ansible/roles/monitoring/xtcp2/`) and the image - [ ] I11: state-dir guard — minimum: node-level alert on `/var/lib/xtcp2` usage (vmagent); preferred: XFS project quota (or systemd quota for bare metal) sized to ~2× the ASN cache (~200 MiB headroom) - [ ] Egress pinning: restrict the collector's egress to the per-DC S3 endpoints (nftables rule on its cgroup — host netns, so not container network policy; not host-wide SG/UFW, which would hit co-located workloads); pair with W2's frozen `s3_endpoint` - [ ] W5 deployment: second container + `:ro` collector state dir + shared socket dir; canary via the existing fleet-gate mechanism; revert via `xtcp2_revert.yml` +- [ ] Regression-guard the load-bearing container defaults in the fleet-gate test (`docker inspect`): `HostConfig.PidMode` not `host`, `ReadonlyRootfs` true, `AppArmorProfile` == `docker-default` (not `unconfined`), every bind except the state dir `:ro`, and a probe that `mount -o remount,bind,rw` on a `:ro` bind fails `EACCES` inside the container - [ ] Pin the new image tag in the image repo; bump the deployment repo role defaults in lockstep (role gate test asserts image↔config compatibility) **Estimate**: 3–5 days spread across phases. **Owner**: infra. @@ -208,7 +209,7 @@ Changes in the deployment repo (`ansible/roles/monitoring/xtcp2/`) and the image ``` Phase 0 (today): the v2.0.x auth merge (I10) — ops action, no release needed -Phase 1 (release vNext): W0 (image rebuild) + W1 ─▶ W2 ─▶ W3 (pure code/build; closes I1-I6, I8, I15) +Phase 1 (release vNext): W0 (image rebuild) + W1 ─▶ W2 ─▶ W3 (pure code/build; closes I1-I6, I8, I14) └──▶ W8 (hostile-path + golden-config gates) ──▶ W9 Phase 2: W4 (Landlock, optional pivot_root) ──▶ W6 (Set audit) ──▶ W7 quota + audit ship Phase 3: W5 (writer split) ──▶ W7 (second container, :ro flip) ──▶ W9 @@ -251,7 +252,7 @@ With admin token in hand on a canary host, every item must fail safely: |---|---|---| | OD-1 | `unix_socket_mode` cap: 0600 or allow 0660 (shared-group socket)? | 0600 (`in: [0, 384]`); add 432 (0660) to the list only if a consumer needs group access — never a `lte` bound | | OD-2 | Landlock-absent kernel behavior: warn-and-continue vs refuse to start | Warn + `xtcp2_sandbox_active 0` + alert; fleet is ≥6.8 so this is a bare-metal edge case | -| OD-3 | `file:` destination: keep behind `-allowFileDest` or remove entirely in the s3parquet build flavor | Keep behind flag (build flavors already gate schemes) | +| OD-3 | `file:` destination: keep behind `-allowFileDest` or remove entirely in the s3parquet build flavor | Both: the `dest_file` build tag removes it from the production flavor (W0); flavors that compile it still require `-allowFileDest` (W2) | | OD-4 | Writer transport: UDS gRPC (chosen) vs loopback TCP | UDS — peer credentials, no port surface; loopback TCP rejected because every host process shares it | | OD-5 | Does the ipfeed sidecar adopt the writer? | No — out of scope; it is already uid 65534, single subdirectory, no caps | | OD-6 | Frozen-field escape hatch lifetime | Time-boxed to one migration window, then remove the flag | diff --git a/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md b/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md index 203f2f3e..fd5848ad 100644 --- a/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md +++ b/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md @@ -125,7 +125,8 @@ The container spec (`tasks/main.yml`) translates to kernel-enforced properties a | `cap_drop: ALL` + `capabilities: [NET_ADMIN, SYS_ADMIN]` | bounding/permitted/effective set = exactly those two | every other capability is unavailable even as root — no `CAP_DAC_OVERRIDE`, `CAP_SYS_MODULE`, `CAP_SYS_PTRACE`, `CAP_SYS_BOOT`, `CAP_SYS_TIME`, `CAP_MKNOD`, … | | `security_opts: no-new-privileges:true` | `PR_SET_NO_NEW_PRIVS` | no privilege gain through `execve` — this also covers `Set`'s soft restart, which `syscall.Exec`s the same binary | | `read_only: true` rootfs + `tmpfs: /tmp` | root mount `MNT_READONLY`; private tmpfs | zero writes in the image filesystem; `/tmp` is ephemeral scratch | -| `:ro` bind mounts | per-mount read-only flags | `/run/netns`, `/run/docker/netns`, `/sys/fs/cgroup`, `/run/docker.sock`, the S3 secret, and both token files are readable, never writable | +| `:ro` bind mounts | per-mount read-only flags | `/run/netns`, `/run/docker/netns`, `/sys/fs/cgroup`, `/run/docker.sock`, the S3 secret, and both token files are readable — but against a `CAP_SYS_ADMIN` process only while AppArmor denies the remount (next row) | +| AppArmor `docker-default` (applied implicitly; not overridden) | LSM profile | `deny mount` — what keeps the `:ro` binds and the read-only rootfs read-only against `CAP_SYS_ADMIN` (§4.3); also denies writes under `/proc/sys` and most of `/sys`. Load-bearing: `apparmor=unconfined`, or a host without AppArmor, removes it | | `{{ xtcp2_state_dir }}:/var/lib/xtcp2:rw` | the single rw bind mount | **the only path through which the container can write to the host filesystem** | | `groups: []` | supplementary group membership | `connect()` to the 0770 lldpd control socket without needing `CAP_DAC_OVERRIDE` | | private PID namespace (docker default, not overridden) | pidns | `/proc` inside the container shows only its own processes — host process enumeration and `/proc//environ` are not visible | @@ -136,7 +137,7 @@ Two consequences of "uid 0 without `CAP_DAC_OVERRIDE`" are worth spelling out: t ### 4.3 What `NET_ADMIN` + `SYS_ADMIN` actually grant -The daemon's own startup check (`pkg/xtcp/init_capabilities.go`, see [observability.md](../observability.md)) marks exactly two capabilities **fatal**: `CAP_NET_ADMIN` (netlink `inet_diag` dumps) and `CAP_SYS_ADMIN` (`setns(CLONE_NEWNET)` into every other namespace for the per-netns readers). `CAP_SYS_PTRACE`, `CAP_NET_RAW`, and `CAP_SYS_RESOURCE` are warnings only. So the deployment grants the minimal set the product needs — the intuition "net admin to open the netlink sockets" is correct but incomplete: per-namespace polling also requires `SYS_ADMIN`, and the role grants exactly those two and nothing else. +The daemon's own startup check (`pkg/xtcp/init_capabilities.go`, see [observability.md](../observability.md)) marks exactly two capabilities **fatal**: `CAP_NET_ADMIN` (netlink `inet_diag` dumps) and `CAP_SYS_ADMIN` (`setns(CLONE_NEWNET)` into every other network namespace for the per-netns readers). `CAP_SYS_PTRACE`, `CAP_NET_RAW`, and `CAP_SYS_RESOURCE` are warnings only. So the deployment grants the minimal set the product needs — the intuition "net admin to open the netlink sockets" is correct but incomplete: per-namespace polling also requires `SYS_ADMIN`, and the role grants exactly those two and nothing else. Because the container runs in the **host** network namespace and the **initial** user namespace (no userns remap), both capabilities are full-strength: @@ -149,28 +150,29 @@ Because the container runs in the **host** network namespace and the **initial** **`CAP_SYS_ADMIN` (initial userns)** -- `setns()` into every namespace on the host — every container's netns (the function), but also other containers' mount and PID namespaces; -- `mount()`/`umount2()` within its own mount namespace (tmpfs, bind mounts, remounts; previously-locked mounts stay locked); +- `setns()` into other network, UTS, IPC, cgroup and time namespaces — every container's netns (the function), and with them other containers' hostname (UTS) and SysV IPC / shared memory (IPC). **Not** mount or PID namespaces: `setns(CLONE_NEWNS)` also requires `CAP_SYS_CHROOT` (dropped), and `setns(CLONE_NEWPID)` only enters descendants of the caller's PID namespace, which neither sibling containers' nor the host's are; +- `mount()`/`umount2()` within its own mount namespace — tmpfs, bind mounts, and **remounts**: without userns remap the container's mounts are not locked, so capability-wise `mount -o remount,bind,rw` turns any `:ro` bind (or the read-only rootfs) writable. What blocks it today is AppArmor's `docker-default` profile (`deny mount`); Docker's seccomp profile allows `mount` once `CAP_SYS_ADMIN` is held. Verify that the new mount API (`fsmount`/`move_mount`/`mount_setattr`) is mediated on fleet kernels — if not, only a seccomp filter closes it; - `bpf()` — `CAP_SYS_ADMIN` retains the legacy privileged bpf path; - `perf_event_open`, namespace creation (`unshare`/`clone` with `CLONE_NEW*`), `quotactl`. -Honest assessment: this is *not* "a locked-down exporter that happens to have net admin." The filesystem is tightly confined (§4.2), but `NET_ADMIN` + `SYS_ADMIN` in the initial userns make the process host-network-equivalent and namespace-equivalent — an RCE in the daemon can reconfigure host networking and enter any namespace regardless of `Set`. That is inherent to a host-wide socket monitor, not a deployment misconfiguration; §5's layers shrink the *filesystem* angle (L4/L5) but cannot remove this one. Seccomp does not help here: the default profile does not gate namespace entry or mounting away from a process holding these capabilities. +Honest assessment: this is *not* "a locked-down exporter that happens to have net admin." The filesystem is tightly confined (§4.2), but `NET_ADMIN` + `SYS_ADMIN` in the initial userns make the process host-network-equivalent and namespace-equivalent — an RCE in the daemon can reconfigure host networking and enter any network, UTS, IPC or cgroup namespace regardless of `Set`. That is inherent to a host-wide socket monitor, not a deployment misconfiguration; §5's layers shrink the *filesystem* angle (L4/L5) but cannot remove this one. Docker's seccomp profile does not help here: it allows namespace entry and `mount` once these capabilities are held. AppArmor does gate mounting (below), which makes it load-bearing. ### 4.4 What is restricted and filtered Everything an attacker (an admin-token holder, or an RCE inside the daemon) still faces: -- **Filesystem**: read-only image rootfs; every bind mount `:ro` except `/var/lib/xtcp2`; `/tmp` is ephemeral tmpfs; no `CAP_DAC_OVERRIDE`, so other uids' files are unreadable. +- **Filesystem**: read-only image rootfs; every bind mount `:ro` except `/var/lib/xtcp2` — held against `CAP_SYS_ADMIN` by AppArmor's `deny mount`, not by the mount flags alone; `/tmp` is ephemeral tmpfs; no `CAP_DAC_OVERRIDE`, so other uids' files are unreadable. - **Capabilities**: bounding set of two — no kernel module loading, no ptrace of other processes, no system clock, no `mknod`, no DAC override. - **Privilege transitions**: `no-new-privileges` survives `execve`, including `Set`'s soft restart. - **Process visibility**: private PID namespace — host PIDs and their environments are invisible. - **Syscalls**: Docker's default seccomp profile (legacy set blocked). +- **LSM**: AppArmor `docker-default` — `deny mount` plus `/proc`/`/sys` write denials; the only thing stopping an RCE from remounting the `:ro` binds rw. - **`/proc`**: runc's defaults render `/proc/sys` read-only (sysctl / `core_pattern` writes blocked) and mask `/proc/kcore` and friends; the private pidns hides host PIDs — together these close the classic `/proc/1/root` and `core_pattern` escape idioms (see §4.7). - **`/dev`**: runc populates a private tmpfs with an allow-list of character devices only (`null`, `zero`, `full`, `random`, `urandom`, `tty`, ptys, `shm`) — **no block devices exist in the container** (`/dev/sda` and friends are absent), `CAP_MKNOD` is dropped so none can be created, and the cgroup device filter denies `open()` on any non-allow-listed major/minor regardless of capabilities. Three independent guards stand between the daemon and raw disks; creating *regular files* in `/dev` touches only ephemeral tmpfs. - **Network (app-level)**: bearer auth on `:9088`/`:8889` with per-host tokens and rotation; `/ip` detail loopback-only; TTL-3 listener replies; UFW scrape rule limited to local bridge subnets; S3 credential via a 0400 file mount, not env. - **Resources**: memory / CPU / pids caps; log rotation. -Not filtered (see §4.6): the Docker API through `/run/docker.sock` (a `:ro` mount is not an API restriction), host-firewall rewriting via `NET_ADMIN`, and namespace entry via `SYS_ADMIN`. +Not filtered (see §4.6): the Docker API through `/run/docker.sock` (a `:ro` mount is not an API restriction), host-firewall rewriting via `NET_ADMIN`, and network/UTS/IPC/cgroup namespace entry via `SYS_ADMIN`. ### 4.5 Can `Set` write to the host filesystem? @@ -186,7 +188,7 @@ Not filtered (see §4.6): the Docker API through `/run/docker.sock` (a `:ro` mou - **ASN-cache poisoning**: `Set` writes attacker bytes to a file, then points `ipmeta_cache_path`/`asn_db_path` at it; the daemon reloads hourly and parses attacker-controlled parquet/zstd — both a parser attack surface and a way to poison the enrichment data shipped to the data lake. - **Planted files** in `/var/lib/xtcp2` (mode 0600 root-owned) and planted unix sockets (chmod up to 0777 inside that dir). -What `Set` **cannot** do from inside that container: write anywhere else on the host. The image rootfs is read-only, `/run` paths are `:ro`, and there is no `CAP_DAC_OVERRIDE` to escape the mount table — writes to `/tmp`, `/dev`, and `/dev/shm` succeed but land in ephemeral tmpfs, never the host disk. The mount namespace fixes the writable set: no `..` or symlink traversal escapes it, because a bind mount has no "above" — parent resolution lands in the container's own read-only rootfs. +What `Set` **cannot** do from inside that container: write anywhere else on the host. The image rootfs is read-only, `/run` paths are `:ro`, and there is no `CAP_DAC_OVERRIDE` to escape the mount table (`Set` cannot call `mount`; an RCE could remount, were it not for AppArmor — §4.3) — writes to `/tmp`, `/dev`, and `/dev/shm` succeed but land in ephemeral tmpfs, never the host disk. The mount namespace fixes the writable set: no `..` or symlink traversal escapes it, because a bind mount has no "above" — parent resolution lands in the container's own read-only rootfs. **Fleet role revision v1.x (previous)** @@ -209,7 +211,7 @@ How the classic file-write escape targets fare when `Set` points a `file:` desti | `/etc/hostname` | **fails** | read-only rootfs → `EROFS` | | `docker.sock` (as a `file:` target) | **fails** | a unix-socket path cannot be opened with `open(2)`; the bind is `:ro` regardless | -The `/proc/1/root` idiom — dereferencing PID 1's root to escape a container's mount namespace — works only when the daemon **shares the host PID namespace or runs on the host directly**. The private pidns (a docker default, not overridden by the role) is the load-bearing restriction here, alongside the read-only rootfs; both deserve explicit regression guards so a future `pid_mode: host` "fix" for namespace discovery cannot silently reintroduce the escape. The only writable paths in the role-deployed container are ephemeral tmpfs — `/tmp`, `/dev`, `/dev/shm` (both revisions; never the host disk) — and `/var/lib/xtcp2` (v2.0.x; the only host path, and not on an SSH path). +The `/proc/1/root` idiom — dereferencing PID 1's root to escape a container's mount namespace — works only when the daemon **shares the host PID namespace or runs on the host directly**. The private pidns (a docker default, not overridden by the role) is the load-bearing restriction here, alongside the read-only rootfs; both deserve explicit regression guards so a future `pid_mode: host` "fix" for namespace discovery cannot silently reintroduce the escape — and so does the AppArmor profile, which keeps the read-only rootfs read-only against `CAP_SYS_ADMIN`. The only writable paths in the role-deployed container are ephemeral tmpfs — `/tmp`, `/dev`, `/dev/shm` (both revisions; never the host disk) — and `/var/lib/xtcp2` (v2.0.x; the only host path, and not on an SSH path). A failed write is not silent: `Set` applies the config → soft-restart `exec` → `InitDests` → `OpenFile` fails on the read-only filesystem → the daemon exits → `restart: unless-stopped` brings the container back with the clean image env (`XTCP_CONFIG_JSON` was process env, not container env). A crash-restart loop that follows a `Set` is therefore a detection signal in its own right. @@ -236,7 +238,7 @@ blocks relative paths, dot-dot traversal, empty `//` segments, NULs, spaces, and | `link_monitor.baseline_file` | `max_len 4096` (Go additionally checks nonempty/no-NUL) | `pattern` above | | `ipmeta_cache_path` (244), `ipmeta_bootstrap_path` (243), `asn_db_path` (241) | `max_len 255` | `pattern` above | | `docker_socket_path` (202), `lldpd_socket_path` (211) | len rules | `pattern` above | -| `dest` (60) | `min_len 4, max_len 512`; scheme checked only in Go (`pkg/xtcp/input_validation.go`) | `pattern: "^(null|stdout|stderr|file|tcp|http|https|udp|unix|unixgram|kafka|nats|nsq|valkey|s3parquet)://[^[:cntrl:]]+$"` (mirror of `knownSchemes`, `pkg/xtcp/destinations_core.go:70`) | +| `dest` (60) | `min_len 4, max_len 512`; scheme checked only in Go (`pkg/xtcp/input_validation.go`) | `pattern: "^(null|stdout|stderr|(tcp|udp|unix|unixgram|file|http|https|kafka|nats|nsq|valkey|s3parquet):[^[:cntrl:]]+)$"` (mirror of `knownSchemes`, `pkg/xtcp/destinations_core.go:70`; the real syntax is `:payload` — `file:/x`, `kafka:h:9092` — so the pattern requires one colon, not `://`, and the bare sinks take no payload) | | `ListenerEndpoint.address` (2) | `max_len 512` | charset pattern; when `network: unix`, path shape as above (message-level CEL cross-rule) | | `unix_socket_mode` (3) | `lte 511` | allowlist `in: [0, 384]` (default/0600) — a `lte` bound is not a permission check: 0577 (383) passes `lte 384` yet is world-writable | @@ -335,8 +337,8 @@ service StateWriter { Rules enforced by the writer, in code: - **No path parameters.** Targets are named constants inside the chroot. A compromised collector cannot ask it to write anywhere else — the request literally has nowhere to put a path. This is the property that makes the design robust even if the writer is reached by some other local process. -- Size caps (baseline ≤64 KiB — matches the existing store limit; cache ≤192 MiB), per-RPC rate limits, and monotonic version numbers to prevent rollback-replay of old baselines. -- Same discipline the baseline store already has (`pkg/linkmonitor/internal/baseline/files.go`): `CreateTemp` in the target dir → write → `fsync` → `rename` → `fsync(dir)`, mode 0600; the baseline `flock` moves inside the writer. +- Size caps (baseline ≤64 KiB — matches the existing store limit; cache ≤ `-ipmetaMaxBytes`, default 64 MiB), per-RPC rate limits, and monotonic version numbers to prevent rollback-replay of old baselines. +- Same discipline the baseline store already has (`pkg/linkmonitor/internal/baseline/files.go`): `CreateTemp` in the target dir → write → `fsync` → `rename` → `fsync(dir)`, mode 0640 group `xtcp2-state` (the collector reads via the group); the baseline `flock` moves inside the writer. **What moves where** @@ -382,12 +384,12 @@ Primitives #1 and #2 live in code that production never uses, yet both are compi Remove them from production binaries entirely, using the mechanisms that already exist: -- Move the `file` destination behind a `dest_file` build tag, following the `dest_` convention. The runtime already handles this cleanly: `lookupDestinationFactory` distinguishes "unknown scheme" from "known but not compiled in", and `destinationLookupError` tells the operator to rebuild with `-tags dest_file`; a config naming an uncompiled scheme is a **fatal startup error**, not a silent no-op — a `Set` that lands the daemon in a restart loop instead of a write. +- Move the `file` destination behind a `dest_file` build tag, following the `dest_` convention. The runtime already handles this cleanly: `lookupDestinationFactory` distinguishes "unknown scheme" from "known but not compiled in", and `destinationLookupError` tells the operator to rebuild with `-tags dest_file`; a config naming an uncompiled scheme is a **fatal startup error**, not a silent no-op. `Set` checks the same thing before restarting (mirroring `checkConfigEnrichersCompiledIn`) and returns `FailedPrecondition`, so the daemon keeps running — no write and no restart loop. - Gate the capture writer behind a `capture` build tag; `write_files > 0` without the tag must be a fatal startup error, mirroring the enricher convention. - Build the production flavor (`oci-xtcp2-s3parquet-enrich`) without either tag; keep them in the `full`/`-debug` engineering flavors. - Consider also dropping the network stdlib destinations from the production flavor — production needs `s3parquet` (plus `null`/`stdout` for logs). This makes the DEST-repoint exfil class structurally impossible, in addition to L2's frozen fields. -After this layer, a `Set` call cannot even *express* a `file:` destination or a capture budget on a production host: the code that would act on them is not in the binary. This is the strongest form of L2's `-allowFileDest` escape hatch — the flag becomes the build tag. +After this layer, a `Set` call cannot even *express* a `file:` destination or a capture budget on a production host: the code that would act on them is not in the binary. This is the strongest form of L2's `-allowFileDest` escape hatch: in production the build tag replaces the flag; the debug flavors that compile `file:` still require it. ## 6. What each layer stops From 46bf3590c29614dc7bea7dc445fdd3142dc4904b Mon Sep 17 00:00:00 2001 From: "randomizedcoder dave.seddon.ca@gmail.com" Date: Mon, 9 Nov 2026 11:25:45 -0800 Subject: [PATCH 4/7] docs(security-hardening): privilege brokers (Phase 5, L7), setns scope, severity - New phase-5-privilege-brokers.md: netlink broker (proto over SOCK_SEQPACKET, SCM_RIGHTS fds), Docker broker (typed gRPC, fixed GET set, projection), per-process seccomp/AppArmor profiles, three enforcement layers per request, test tables P-U, rollout and decisions - Advisory: L7 section and matrix column; setns limited to network namespaces in the container; bpf/perf_event_open and NET_ADMIN reach spelled out; severity scoped to deployment shape - Plan: W10-W12, I15-I18, OD-10-OD-12; implementation plan Phase 5 (PR-18-PR-23); Phase 4 egress rule holds against collector RCE only after Phase 5 Co-Authored-By: Claude Opus 5.5 --- docs/README.md | 2 +- .../security-hardening/implementation-plan.md | 36 +- .../phase-2-kernel-confinement.md | 2 +- .../phase-3-writer-split.md | 4 +- .../phase-4-egress-fleet-lockdown.md | 5 +- .../phase-5-privilege-brokers.md | 556 ++++++++++++++++++ .../plan-configservice-set-hardening.md | 94 ++- ...ulnerability-configservice-set-fs-write.md | 113 +++- 8 files changed, 772 insertions(+), 40 deletions(-) create mode 100644 docs/security-hardening/phase-5-privilege-brokers.md diff --git a/docs/README.md b/docs/README.md index d6f43f65..62dfc14d 100644 --- a/docs/README.md +++ b/docs/README.md @@ -67,7 +67,7 @@ Current TCP/UDS, TTL/hop-limit, bearer/HMAC, and read/admin protections, with de ### [Security advisory: `ConfigService.Set` filesystem writes](security-hardening/security-vulnerability-configservice-set-fs-write.md) -`Set` re-executes the daemon with an attacker-chosen config and, absent path validation, becomes a root file-write primitive (`file:` destinations, `capture_path`, `baseline_file`, `ipmeta_cache_path`, unix listeners). The advisory maps every primitive, documents the container deployment and its permission model (runtime settings → Linux primitives, what `NET_ADMIN`+`SYS_ADMIN` really grant, what is restricted/filtered), checks the classic container-escape path idioms target-by-target, and lays out the layered fix: build-time elimination, protovalidate path rules, a runtime path policy with frozen fields, a universal validating interceptor, Landlock self-confinement, and a chrooted state-writer process split. The [hardening plan](security-hardening/plan-configservice-set-hardening.md) tracks the workstreams; the [implementation plan](security-hardening/implementation-plan.md) sequences the phases and PRs; the phase drill-downs go deep: [phase 1](security-hardening/phase-1-code-hardening.md) (build tags, proto validators, path policy, interceptor, audit, CI gates), [phase 2](security-hardening/phase-2-kernel-confinement.md) (Landlock self-confinement, full test matrices), [phase 3](security-hardening/phase-3-writer-split.md) (the writer-process split with its path-free proto protocol and the complete write inventory), and [phase 4](security-hardening/phase-4-egress-fleet-lockdown.md) (egress pinning, escape-hatch removal, v1.x retirement, standing CI gates). +`Set` re-executes the daemon with an attacker-chosen config and, absent path validation, becomes a root file-write primitive (`file:` destinations, `capture_path`, `baseline_file`, `ipmeta_cache_path`, unix listeners). The advisory maps every primitive, documents the container deployment and its permission model (runtime settings → Linux primitives, what `NET_ADMIN`+`SYS_ADMIN` really grant, what is restricted/filtered), checks the classic container-escape path idioms target-by-target, and lays out the layered fix: build-time elimination, protovalidate path rules, a runtime path policy with frozen fields, a universal validating interceptor, Landlock self-confinement, a chrooted state-writer process split, and privilege brokers that take the capabilities and `docker.sock` out of the collector. The [hardening plan](security-hardening/plan-configservice-set-hardening.md) tracks the workstreams; the [implementation plan](security-hardening/implementation-plan.md) sequences the phases and PRs; the phase drill-downs go deep: [phase 1](security-hardening/phase-1-code-hardening.md) (build tags, proto validators, path policy, interceptor, audit, CI gates), [phase 2](security-hardening/phase-2-kernel-confinement.md) (Landlock self-confinement, full test matrices), [phase 3](security-hardening/phase-3-writer-split.md) (the writer-process split with its path-free proto protocol and the complete write inventory), [phase 4](security-hardening/phase-4-egress-fleet-lockdown.md) (egress pinning, escape-hatch removal, v1.x retirement, standing CI gates), and [phase 5](security-hardening/phase-5-privilege-brokers.md) (netlink and Docker brokers with typed, validated APIs; per-process seccomp and AppArmor profiles). ### [Observability](observability.md) Prometheus metrics, Go `pprof` endpoints, optional Pyroscope continuous profiling, and the startup capability check that explains exactly which Linux capabilities are required. diff --git a/docs/security-hardening/implementation-plan.md b/docs/security-hardening/implementation-plan.md index 9f16472b..e00628da 100644 --- a/docs/security-hardening/implementation-plan.md +++ b/docs/security-hardening/implementation-plan.md @@ -4,7 +4,7 @@ |---|---| | **Status** | Active — checkboxes track execution; supersedes the sequencing section of the [hardening plan](plan-configservice-set-hardening.md) | | **Repos** | `randomizedcoder/xtcp2` (code), the image repo (image), the deployment repo (deploy), the infra repo (SSM/IAM/network) | -| **Docs** | [Advisory](security-vulnerability-configservice-set-fs-write.md) — the problem (primitives, permission model, escape idioms). [Hardening plan](plan-configservice-set-hardening.md) — workstreams W0–W9 with design detail, estimates, and open decisions. **This doc** — execution order, PR-level steps, phase gates, and rollback. | +| **Docs** | [Advisory](security-vulnerability-configservice-set-fs-write.md) — the problem (primitives, permission model, escape idioms). [Hardening plan](plan-configservice-set-hardening.md) — workstreams W0–W12 with design detail, estimates, and open decisions. **This doc** — execution order, PR-level steps, phase gates, and rollback. | Reading order for an implementer: skim the advisory's §2 (primitives) and §4 (permission model), then execute the phases below, dipping into the workstream sections of the hardening plan as each PR needs detail. @@ -15,6 +15,7 @@ Reading order for an implementer: skim the advisory's §2 (primitives) and §4 ( - [Phase 2 — Kernel self-confinement](#phase-2--kernel-self-confinement) - [Phase 3 — Privilege separation (writer split)](#phase-3--privilege-separation-writer-split) - [Phase 4 — Egress and fleet lockdown](#phase-4--egress-and-fleet-lockdown) +- [Phase 5 — Privilege brokers](#phase-5--privilege-brokers) - [Issue → phase tracking](#issue--phase-tracking) - [Timeline and effort](#timeline-and-effort) - [Gates and rollback](#gates-and-rollback) @@ -110,6 +111,29 @@ PRs (repo `xtcp2` unless noted; each lands with its tests and doc updates from W --- +## Phase 5 — Privilege brokers + +> Drill-down with the broker protocols, filtering pipelines, syscall profiles and full test matrices: [phase-5-privilege-brokers.md](phase-5-privilege-brokers.md). + +**Goal:** a compromised collector holds no capability, no `docker.sock` and no namespace handle; the privileged reads go through two brokers whose APIs can express nothing else. + +- [ ] **PR-18 — spike** ([W10](plan-configservice-set-hardening.md#w10--netlink-broker-and-collector-de-privilege-l7)): kernel fd-semantics proofs and a zero-capability feature run. +- [ ] **PR-19 — syscall profiles** ([W12](plan-configservice-set-hardening.md#w12--syscall-profiles-l7)): per-process seccomp and AppArmor; the v1 collector profile ships with Phase 2. +- [ ] **PR-20 — netlink broker** (W10): `xtcp2 -nlbroker`, proto over `SOCK_SEQPACKET` with `SCM_RIGHTS`, collector seams. +- [ ] **PR-21 — Docker broker** ([W11](plan-configservice-set-hardening.md#w11--docker-broker-l7)): `xtcp2 -dockerbroker`, typed gRPC, fixed outbound request set, projection. +- [ ] **PR-22 — collector de-privilege** (W10): non-root uid, `cap_drop ALL`, broker mode as the fleet default. +- [ ] **PR-23 — deployment** (the deployment repo): broker containers, profiles, the five-step rollout, alerts. + +**Exit criteria:** +- the collector runs as `xtcp2` with `CapEff 0` and no `docker.sock`; +- RCE probes fail; +- monitoring output matches direct mode; +- Phase 4's egress rule survives a collector RCE. + +**Effort:** ~3–4 weeks (PR-19 ~1 week, alongside Phase 2). + +--- + ## Issue → phase tracking | Issue | Fixed in | Layer(s) | @@ -124,6 +148,10 @@ PRs (repo `xtcp2` unless noted; each lands with its tests and doc updates from W | I12 no `Set` audit trail | Phase 1 (PR-5) | L6 | | I13 collector holds needless write access | Phase 3 | L4, L5 | | I14 debug writers in production image | Phase 1 (PR-1) | L0 | +| I15 `docker.sock` in the collector | Phase 5 (PR-21, PR-23) | L7 | +| I16 `CAP_NET_ADMIN` in the collector | Phase 5 (PR-20, PR-22) | L7 | +| I17 `CAP_SYS_ADMIN` under the default seccomp profile | Phase 5 (PR-19, PR-22) | L7 | +| I18 full inspect documents in the collector | Phase 5 (PR-21) | L7 | ## Timeline and effort @@ -134,8 +162,9 @@ PRs (repo `xtcp2` unless noted; each lands with its tests and doc updates from W | 2 | weeks 3–4 | Landlock (+ optional pivot_root), quota/alert, audit ship | | 3 | weeks 4–7 | writer split, `:ro` flip, systemd units | | 4 | weeks 7–8 | egress pinning, v1.x retirement, standing gates | +| 5 | weeks 4–9 (profiles from week 3) | syscall profiles, netlink + Docker brokers, collector de-privilege | -Phases 1 and 2 partially overlap (Landlock PR-8 can start as soon as PR-3's `-writableDir` lands); Phase 3's proto work can start during Phase 2. Nothing later is required for anything earlier to ship. +Phases 1 and 2 partially overlap (Landlock PR-8 can start as soon as PR-3's `-writableDir` lands); Phase 3's proto work can start during Phase 2. Phase 5 starts once Phase 2 lands and runs alongside Phase 3; its PR-19 profiles ship with Phase 2. Nothing later is required for anything earlier to ship. ## Gates and rollback @@ -148,5 +177,6 @@ Phases 1 and 2 partially overlap (Landlock PR-8 can start as soon as PR-3's `-wr | 2 | previous image tag (Landlock ruleset gone); quota/alert are additive | | 3 | `xtcp2_revert.yml` + restore the collector's `:rw` state mount (writer container can stay, unused) | | 4 | network policy rules (per-DC, reversible); `-setAllowFrozenFields` restoration if a genuine migration is blocked | +| 5 | per rollout step: drop the `security_opt` (profiles); unset the broker flags (collector back to direct mode); restore the previous collector container spec (capabilities + mounts) | -- **Success definition (end state):** with the admin token in hand on a canary host, nothing in the [end-state verification checklist](plan-configservice-set-hardening.md#end-state-verification-checklist) succeeds in writing outside `/var/lib/xtcp2`, changing egress, or naming any path — and `/var/lib/xtcp2` itself is quota-guarded, audited, and (post-Phase 3) writable only by the capability-less writer. +- **Success definition (end state):** with the admin token in hand on a canary host, nothing in the [end-state verification checklist](plan-configservice-set-hardening.md#end-state-verification-checklist) succeeds in writing outside `/var/lib/xtcp2`, changing egress, or naming any path — and `/var/lib/xtcp2` itself is quota-guarded, audited, and (post-Phase 3) writable only by the capability-less writer — and the collector itself holds no capability and no `docker.sock` (post-Phase 5). diff --git a/docs/security-hardening/phase-2-kernel-confinement.md b/docs/security-hardening/phase-2-kernel-confinement.md index ad7baa4e..5c07b49d 100644 --- a/docs/security-hardening/phase-2-kernel-confinement.md +++ b/docs/security-hardening/phase-2-kernel-confinement.md @@ -30,7 +30,7 @@ Phase 1 made hostile `Set` inputs **invalid**. Phase 2 assumes Phase 1 is bypass | State-dir fill detected + alerted | `xtcp2_state_dir_bytes` gauge + alert rules | — | | Stolen-token probing visible | `FailedPrecondition` rate alert on PR-5's `xtcp2_set_applied_total` | — | -Not in scope: network confinement (Landlock v4 network rules are a possible future layer — TCP bind/connect scoping — tracked as an open question, not planned here); capability reduction (the daemon keeps `NET_ADMIN`+`SYS_ADMIN` by design, see advisory §4.3). +Not in scope: network confinement (Landlock v4 network rules are a possible future layer — TCP bind/connect scoping — tracked as an open question, not planned here); capability reduction ([Phase 5](phase-5-privilege-brokers.md): netlink and Docker brokers; its PR-19 syscall profiles can ship alongside this phase). ## PR map and branches diff --git a/docs/security-hardening/phase-3-writer-split.md b/docs/security-hardening/phase-3-writer-split.md index e0b03b12..7301b3cd 100644 --- a/docs/security-hardening/phase-3-writer-split.md +++ b/docs/security-hardening/phase-3-writer-split.md @@ -33,7 +33,7 @@ The daemon has exactly two legitimate writes (advisory §2): the link-monitor ba | Collector RCE **as root** wants to poison state | — | must speak the writer's fixed 3-op API: two named targets, one size cap each, monotonic seq — no paths, no arbitrary bytes past schema checks; schema-valid poison at the next `seq` still lands, and `docker.sock` bypasses the writer entirely | | Writer itself compromised | — | unprivileged uid, no capabilities, no network (container `network_mode: none`), Landlock write rules only on its two target dirs | -What this phase does **not** fix: the collector keeps `docker.sock`, so an RCE can have dockerd write any host path (the state dir included) without touching the writer; the collector can still reach network egress (Phase 4); and the writer accepts a well-formed request from uid 0 (or a stolen shared secret — see [decisions](#decisions-to-close-during-phase-3)) and will overwrite its two fixed targets with schema-valid data; that bounded surface is the accepted residual. +What this phase does **not** fix: the collector keeps `docker.sock` (until [Phase 5](phase-5-privilege-brokers.md)), so an RCE can have dockerd write any host path (the state dir included) without touching the writer; the collector can still reach network egress (Phase 4); and the writer accepts a well-formed request from uid 0 (or a stolen shared secret — see [decisions](#decisions-to-close-during-phase-3)) and will overwrite its two fixed targets with schema-valid data; that bounded surface is the accepted residual. ## PR map and branches @@ -255,7 +255,7 @@ Container pair (deployment role, mirroring the existing `ipfeed` second-containe | | `xtcp2` (collector) | `xtcp2_writer` (writer) | |---|---|---| -| uid | 0 (unchanged — keeps `NET_ADMIN`/`SYS_ADMIN`); `group_add: xtcp2-state` | 65534, primary group `xtcp2-state` | +| uid | 0 (unchanged — keeps `NET_ADMIN`/`SYS_ADMIN` until Phase 5 makes it a non-root uid with no capabilities); `group_add: xtcp2-state` | 65534, primary group `xtcp2-state` | | capabilities | `cap_drop ALL` + NET_ADMIN, SYS_ADMIN (unchanged) | `cap_drop ALL` — **none added** | | network | host (unchanged, ports 9088/8889/9089) | `network_mode: none` — no interfaces, no egress at all | | mounts | as today, **minus** the rw state dir (kept `ro` for loads) + socket volume | `/var/lib/xtcp2:rw` (the only writable mount), socket volume, nothing else — no docker.sock, no netns, no S3 secret | diff --git a/docs/security-hardening/phase-4-egress-fleet-lockdown.md b/docs/security-hardening/phase-4-egress-fleet-lockdown.md index f5fb3bb8..b226b5f7 100644 --- a/docs/security-hardening/phase-4-egress-fleet-lockdown.md +++ b/docs/security-hardening/phase-4-egress-fleet-lockdown.md @@ -31,6 +31,7 @@ After Phases 1–3 the `Set` surface is: path fields rejected, writers not compi | The v1.x fleet revision — unauthenticated `:8889` — may still be running on some hosts (I10) | PR-16 | | A compromised collector can dial arbitrary hosts (kernel does not yet constrain egress) | PR-15 | | Nothing structurally stops a future PR from reintroducing a write site, a flavor regression, or a protovalidate gap | PR-17 | +| A compromised collector holds `CAP_NET_ADMIN` in the host netns and can delete the Layer 3 egress rule | [Phase 5](phase-5-privilege-brokers.md) | The verification target is the [end-state verification checklist](plan-configservice-set-hardening.md#end-state-verification-checklist) from the hardening plan — executed **holding the admin token**. @@ -60,7 +61,7 @@ PR-16 sec/w7-retire-v1x (deployment-repo) ─┘ **Layer 2 — kernel (Landlock network rules, ABI 4; the Phase 2 deferred decision).** The collector's `pkg/sandbox` ruleset gains TCP network rules: `CONNECT_TCP` scoped to port 443, `BIND_TCP` scoped to the listener ports (9088, 8889, 9089). Honest limits, documented in-code: Landlock net rules are **port-scoped, not host-scoped** (a 443 listener anywhere passes the kernel check) and ABI 4 does not cover UDP (ABI 10 adds UDP rules; best-effort where present) — DNS keeps working and UDP egress stays governed by Layer 3. Degradation mirrors Phase 2: ABI < 4 → warn + `xtcp2_sandbox_net 0` + continue. The writer needs nothing — its container is `network_mode: none` (Phase 3). -**Layer 3 — host, scoped to the collector (deployment + infra repos).** The collector runs in the host netns, so container network policy does not apply — and host-wide controls (security groups, UFW `OUTPUT`) would also restrict every co-located workload and host service. The isolation boundary is the collector's cgroup: run it under a dedicated cgroup parent (`--cgroup-parent xtcp2.slice`, created before the rule loads) and add an nftables `output` rule matching `socket cgroupv2 level 1 "xtcp2.slice"` that allows the per-DC S3 endpoints and DNS and drops the rest. Host-wide security groups change only on hosts dedicated to infrastructure. This is the only layer that constrains by **host**, closing Landlock's port-only gap. Detection: the host Falco pipeline gains an alert for outbound connections to non-S3 hosts from the collector's container (match the cgroup, not uid 0 — every host daemon is uid 0). +**Layer 3 — host, scoped to the collector (deployment + infra repos).** The collector runs in the host netns, so container network policy does not apply — and host-wide controls (security groups, UFW `OUTPUT`) would also restrict every co-located workload and host service. The isolation boundary is the collector's cgroup: run it under a dedicated cgroup parent (`--cgroup-parent xtcp2.slice`, created before the rule loads) and add an nftables `output` rule matching `socket cgroupv2 level 1 "xtcp2.slice"` that allows the per-DC S3 endpoints and DNS and drops the rest. Host-wide security groups change only on hosts dedicated to infrastructure. The rule holds against a stolen admin token. Against a collector RCE it holds only after [Phase 5](phase-5-privilege-brokers.md) removes `CAP_NET_ADMIN` from the collector; until then, the collector can delete it. This is the only layer that constrains by **host**, closing Landlock's port-only gap. Detection: the host Falco pipeline gains an alert for outbound connections to non-S3 hosts from the collector's container (match the cgroup, not uid 0 — every host daemon is uid 0). **Also here (closes OD-7):** state-dir quota — XFS project quota on `/var/lib/xtcp2` for the writer container, plus a small writerd addition from Phase 3: per-target rate cap (default 12 writes/hour/target — a legitimate baseline learns a handful of times an hour) so the seq monotonicity can't be ground into disk fill by an attacker with a valid request shape. @@ -175,7 +176,7 @@ The [end-state verification checklist](plan-configservice-set-hardening.md#end-s - [ ] **Filesystem:** `Set` with `dest:"file:…"` (any of the advisory §4.7 path idioms incl. `/proc/1/root/root/.ssh/authorized_keys`) → validation rejects; crippled build (validation stubbed) → kernel denies; nothing written anywhere on the host - [ ] **State:** baseline + ipmeta land via the writer only, owned `65534:xtcp2-state` `0640`; quota + rate cap enforce (M8/M9) -- [ ] **Egress:** `Set` with `dest:"http://attacker:8080"` → `FailedPrecondition`, not compiled in the prod flavor; crippled build + RCE-simulated dial → Landlock denies the port; the collector-scoped nftables rule denies the host; the outbound-connection alert fires — four layers, each independently sufficient +- [ ] **Egress:** `Set` with `dest:"http://attacker:8080"` → `FailedPrecondition`, not compiled in the prod flavor; crippled build + RCE-simulated dial → Landlock denies the port; the collector-scoped nftables rule denies the host; the outbound-connection alert fires — four layers, each independently sufficient against a config-level attacker. Against a collector RCE, the nftables layer needs [Phase 5](phase-5-privilege-brokers.md) (test U4) - [ ] **Frozen fields:** `dest`, S3 endpoints, listeners, all path fields → rejected unconditionally, no flag exists (J1/J2); a tuning-only `Set` still applies and audits - [ ] **Auth:** unauthenticated `Set` on `:8889` rejected everywhere (M7); `:8889` unreachable off-operator subnets (M5); no v1.x host remains in inventory - [ ] **Standing:** weekly crippled-build proof green (N5); a write-site PR fails CI until the inventory row exists (N1/N2); deleting a regression suite fails CI (N6) diff --git a/docs/security-hardening/phase-5-privilege-brokers.md b/docs/security-hardening/phase-5-privilege-brokers.md new file mode 100644 index 00000000..fdf68fd9 --- /dev/null +++ b/docs/security-hardening/phase-5-privilege-brokers.md @@ -0,0 +1,556 @@ +# Phase 5 — Privilege brokers: netlink, Docker, and syscall profiles (drill-down) + +| | | +|---|---| +| **Status** | Not started — checkboxes track execution | +| **Tracks** | [Implementation plan](implementation-plan.md) Phase 5 → [hardening plan](plan-configservice-set-hardening.md) workstreams W10 (netlink broker), W11 (Docker broker), W12 (syscall profiles) → [advisory](security-vulnerability-configservice-set-fs-write.md) layer L7 | +| **Goal** | The collector — everything that parses netlink and parquet, serves gRPC and Prometheus, and uploads to S3 — runs as a dedicated non-root uid with **zero capabilities, no `docker.sock`, and no namespace handles**. The two privileged needs move into two single-purpose brokers whose APIs can express nothing else, and each request is enforced three times: protovalidate rules on both ends, broker-side filtering against broker-owned state, and a kernel syscall profile on the broker | +| **Prereqs** | None hard. PR-19 (syscall profiles) ships first, against today's single-process collector. Reuses `pkg/sandbox` (Phase 2) and the peer-auth and both-ends-validation patterns of the writer (Phase 3) | + +## Contents + +- [Why this phase exists](#why-this-phase-exists) +- [Kernel facts the design rests on](#kernel-facts-the-design-rests-on) +- [Target topology](#target-topology) +- [Design rules for every broker](#design-rules-for-every-broker) +- [The netlink broker](#the-netlink-broker) +- [The Docker broker](#the-docker-broker) +- [Syscall profiles for every process](#syscall-profiles-for-every-process) +- [PR map](#pr-map) +- [Tests](#tests) +- [Deployment and rollout](#deployment-and-rollout) +- [Decisions to close during Phase 5](#decisions-to-close-during-phase-5) +- [Release checklist](#release-checklist) +- [Definition of done](#definition-of-done) + +## Why this phase exists + +Phases 1–4 close the `Set` primitives and confine the collector's filesystem and egress. They leave three things that make a **compromised collector host-root-equivalent even inside the reference container** (advisory [§4.3](security-vulnerability-configservice-set-fs-write.md#43-what-net_admin--sys_admin-actually-grant), [§4.6](security-vulnerability-configservice-set-fs-write.md#46-adjacent-non-set-escalation-paths)): + +| Residual in the container | What an RCE in today's collector can do | After Phase 5 | +|---|---|---| +| `/run/docker.sock` mounted | create a privileged container that mounts the host root → host root; have dockerd write any host path (bypasses L4 Landlock and the L5 writer) | collector has no `docker.sock`. The Docker broker's API is two read RPCs returning a fixed projection; no request field reaches dockerd | +| `CAP_NET_ADMIN` in the host netns | rewrite the host firewall and routes (including Phase 4's egress rule); `SOCK_DESTROY` any TCP connection in any netns; receive TCP-MD5 keys and socket marks in inet_diag dumps | no process in the deployment holds `CAP_NET_ADMIN` — the netlink broker does not need it ([kernel facts](#kernel-facts-the-design-rests-on)) | +| `CAP_SYS_ADMIN` with Docker's default seccomp profile | `bpf()` and `perf_event_open` (allowed once the capability is held — tracing programs can read kernel memory and write user memory); `unshare`; `setns` of any type; `mount` (blocked today only by AppArmor) | collector holds nothing. The netlink broker keeps `CAP_SYS_ADMIN`, but its seccomp profile admits only `setns(…, CLONE_NEWNET)` and `socket(AF_NETLINK, SOCK_DGRAM, {SOCK_DIAG, ROUTE})` | + +## Kernel facts the design rests on + +Verified against kernel source (`torvalds/linux` master). PR-18's spike re-verifies each on fleet kernels; the P-table pins them as tests. + +1. **Netlink privilege checks need the opener *and* the sender.** `__netlink_ns_capable` (`net/netlink/af_netlink.c`) returns `file_ns_capable(, cap) && ns_capable(, cap)`. A netlink socket opened by a privileged broker and then used by a capability-less collector therefore behaves exactly like an unprivileged socket: dumps work, and every `CAP_NET_ADMIN`-gated operation fails. **Privilege does not travel with the fd** — this is what makes fd passing safe. +2. **inet_diag dumps are unprivileged.** `CAP_NET_ADMIN` only adds `INET_DIAG_MARK`, TCP-MD5 key material (`tcp_diag_get_aux` → `tcp_diag_put_md5sig` copies `tcpm_key`), ULP details, and mark conditions in bytecode filters (`net/ipv4/inet_diag.c`, `net/ipv4/tcp_diag.c`). `SOCK_DESTROY` checks `ns_capable(current, CAP_NET_ADMIN)` (`net/core/sock_diag.c`). `xtcp_flat_record` has no mark or MD5 field, so the collector loses nothing it records. It also gains something: it can no longer read MD5 keys or kill sockets. The startup check's claim that xtcp2 "cannot read any TCP socket data without" `CAP_NET_ADMIN` (`pkg/xtcp/init_capabilities.go:36`) contradicts this; PR-18 settles it attribute by attribute. +3. **A socket's netns is fixed at creation** (`sock_net(sk)`). An fd created inside a target netns dumps that netns, whichever netns the process using it is in. +4. **rtnetlink mutations** (`RTM_NEW*`/`RTM_DEL*`) are gated by `netlink_net_capable` (fact 1). A broker-opened `NETLINK_ROUTE` fd lets the collector dump addresses and routes, never change them. +5. **gRPC cannot carry file descriptors.** `SCM_RIGHTS` is ancillary data on a unix socket, and HTTP/2 framing has no place for it. So the netlink broker speaks proto messages over `SOCK_SEQPACKET`; the Docker broker, which returns only data, is a gRPC service. + +## Target topology + +``` + ┌────────────────────────────────────────────┐ + │ xtcp2 collector │ + │ uid xtcp2 (non-root), CapEff 0 │ + │ host netns: listeners 9088/8889/9089, S3 │ + │ no docker.sock, no netns mounts │ + └──┬───────────────┬───────────────┬──────────┘ + SOCK_SEQPACKET │ gRPC / UDS │ framed UDS │ + proto + SCM_RIGHTS │ │ (Phase 3) │ +┌───────────────────────▼──┐ ┌──────────▼──────────┐ ┌──▼─────────────────┐ +│ xtcp2 -nlbroker │ │ xtcp2 -dockerbroker │ │ xtcp2 -writer │ +│ uid 0, CAP_SYS_ADMIN only│ │ uid xtcp2-docker + │ │ uid xtcp2-writer │ +│ seccomp: setns(NEWNET), │ │ docker group, Cap 0 │ │ Cap 0, no network │ +│ socket(NETLINK, │ │ network_mode none │ │ state dir rw │ +│ SOCK_DIAG|ROUTE) only │ │ docker.sock, 4 fixed│ │ │ +│ /run/netns, docker/netns │ │ GETs, projection │ │ │ +└──────────────────────────┘ └─────────────────────┘ └────────────────────┘ +``` + +| Process | uid | Capabilities | Network | Mounts | Listens on | +|---|---|---|---|---|---| +| collector | dedicated `xtcp2` (non-root, not 65534) | **none** | host netns, so it can bind and connect but not configure | state dir `ro`, broker socket dirs, token and secret files (group `xtcp2`, `0440`), `/sys/fs/cgroup:ro` | 9088, 8889, 9089 (all ≥ 1024, no capability needed) | +| netlink broker | 0 | `CAP_SYS_ADMIN` only (+ `CAP_SYS_PTRACE` only for bare-metal `/proc` discovery) | host netns, which is its handle to the host namespace; seccomp forbids `AF_INET*` sockets | `/run/netns:ro`, `/run/docker/netns:ro` (rslave), its socket dir | `/run/xtcp2/nlbroker.sock` only | +| Docker broker | dedicated `xtcp2-docker`, `group_add` docker gid | none | `network_mode: none` | `docker.sock`, `/run/docker/netns:ro` (stat only), its socket dir | `/run/xtcp2/dockerbroker.sock` only | +| writer | dedicated `xtcp2-writer` (moved off 65534, which the internet-facing ipfeed sidecar also uses) | none | `network_mode: none` | state dir `rw` | per [Phase 3](phase-3-writer-split.md) | + +**Why two brokers, not one.** Membership in the docker group is host-root-equivalent; `CAP_SYS_ADMIN` limited by seccomp to `setns(CLONE_NEWNET)` is not. A combined broker would hand one compromise both. Each broker holds exactly one privilege and has its own API, profile and alert. + +## Design rules for every broker + +1. **Typed, verb-specific APIs; never a pass-through.** No request field carries a path, URL, HTTP method, raw netlink message, socket family, or syscall argument chosen by the client. The broker builds every privileged call from compiled-in constants plus identifiers it issued itself. +2. **Three independent enforcement layers per request:** + - *validators*: protovalidate on both ends — the client before send, the broker after decode; unknown fields rejected; + - *broker-side filtering*: semantic checks against broker-owned state (discovery tables, indexes), rate limits, connection limits; + - *kernel*: a seccomp allowlist and a Landlock ruleset on the broker, so a bug in the first two layers still cannot exceed the API's intent. +3. **Identifiers, not handles.** Clients name objects by broker-issued ids: a netns by `(dev, inode)` from the broker's own discovery, a container by netns inode from the broker's own index. An unknown id returns `NOT_FOUND` before any privileged call. +4. **Responses are projections.** Only fields with a consumer cross the boundary. Outbound messages are validated too: tenant-controlled strings such as container names get charset rules, and a failing entry is dropped, not repaired. +5. **Peer authentication is meaningful because the collector is no longer root.** + - `SO_PEERCRED` uid must equal the dedicated `xtcp2` uid. Once the collector is non-root, that uid names exactly one process image; uid 0 never did. + - Socket dir `0750` and socket `0660`, group `xtcp2-broker`. + - Optional second factor: `SO_PEERSEC` AppArmor label `xtcp2-collector` (verify unix-socket label support on fleet kernels). Separate PID namespaces rule out pid- or cgroup-based checks below `PIDFD_INFO_CGROUPID` (6.13). +6. **Fail closed in the brokers, degrade-not-exit in the collector.** + - Broker down: the affected enrichment or namespace set goes stale, a counter rises, and an alert fires. + - The collector never falls back to holding the privilege itself in the fleet. Direct mode exists only when no broker socket is configured, for bare `xtcp2` runs. +7. **One admission checklist for new ops**, mirroring the writer's ([Phase 3 Appendix A, C](phase-3-writer-split.md#appendix-a--the-write-inventory-and-the-target-wish-list)): + 1. a new oneof arm or RPC with validators; + 2. a compiled-in mapping to exactly one privileged call; + 3. a seccomp and Landlock delta; + 4. a test-table block; + 5. an alert; + 6. review. + + Adding a privileged capability to a broker is a design change, never a configuration option. + +## The netlink broker + +### What it replaces + +Today the collector does the privileged work itself. +- **Per-netns readers** (`netNamespaceInstance`, `pkg/xtcp/ns_net_namespace.go`): lock an OS thread, `open` the netns handle, `setns(fd, CLONE_NEWNET)`, `socket(AF_NETLINK, SOCK_DGRAM, NETLINK_INET_DIAG)`, then restore or terminate the thread. +- **The locality enricher** (`nsLocalitySnapshot`, `pkg/xtcp/enrich_locality.go:408`): does the same to dump addresses and routes. + +After the split, both call sites become one broker request that returns a socket fd. The collector keeps its zero-copy hot path — it reads the same netlink socket it reads today — and loses `setns`, the thread-tainting dance, and both capabilities. + +### Protocol — proto over `SOCK_SEQPACKET` + +`SOCK_SEQPACKET` preserves message boundaries, so one datagram is one request and no length prefix needs parsing. A successful `OpenSocket` reply carries **exactly one** fd in `SCM_RIGHTS`; every other reply carries none. The schema is message-only (no `service`), like the writer's: + +```proto +// proto/xtcp_nlbroker/v1/nlbroker.proto +syntax = "proto3"; +package xtcp_nlbroker.v1; +import "buf/validate/validate.proto"; + +enum NetlinkProtocol { + NETLINK_PROTOCOL_UNSPECIFIED = 0; + NETLINK_PROTOCOL_SOCK_DIAG = 1; // NETLINK_SOCK_DIAG (4): inet_diag dumps + NETLINK_PROTOCOL_ROUTE = 2; // NETLINK_ROUTE (0): address/route dumps (locality) +} + +enum NetnsSource { + NETNS_SOURCE_UNSPECIFIED = 0; + NETNS_SOURCE_HOST = 1; // the broker's own (host) netns + NETNS_SOURCE_RUN_NETNS = 2; // /run/netns/ + NETNS_SOURCE_DOCKER = 3; // /run/docker/netns/ + NETNS_SOURCE_PROC = 4; // /proc//ns/net (bare metal only) +} + +// Broker-issued identity. The client never sends a path. +message NetnsId { + uint64 dev = 1 [(buf.validate.field).uint64.gt = 0]; + uint64 inode = 2 [(buf.validate.field).uint64.gt = 0]; +} + +message Netns { + NetnsId id = 1 [(buf.validate.field).required = true]; + NetnsSource source = 2 [(buf.validate.field).enum = {defined_only: true, not_in: [0]}]; + string name = 3 [(buf.validate.field).string = {max_len: 255, pattern: "^[A-Za-z0-9._-]*$"}]; +} + +message ListNetnsRequest { + uint64 generation = 1; // 0 = start a new listing + uint32 offset = 2 [(buf.validate.field).uint32.lte = 65535]; +} +message ListNetnsResponse { + repeated Netns netns = 1 [(buf.validate.field).repeated.max_items = 256]; // one datagram ≪ sk_sndbuf + uint64 generation = 2; + bool more = 3; +} + +message OpenSocketRequest { + NetnsId netns = 1 [(buf.validate.field).required = true]; + NetlinkProtocol protocol = 2 [(buf.validate.field).enum = {defined_only: true, in: [1, 2]}]; +} +message OpenSocketResponse { // the fd travels in SCM_RIGHTS + NetnsId netns = 1; + NetlinkProtocol protocol = 2; +} + +message PingRequest {} +message PingResponse {} +message StatsRequest {} +message StatsResponse { + map counters = 1 [(buf.validate.field).map.max_pairs = 64]; +} + +message Error { + enum Code { CODE_UNSPECIFIED = 0; CODE_INVALID = 1; CODE_NOT_FOUND = 2; + CODE_RESOURCE_EXHAUSTED = 3; CODE_ABORTED = 4; CODE_INTERNAL = 5; } + Code code = 1 [(buf.validate.field).enum = {defined_only: true, not_in: [0]}]; + string detail = 2 [(buf.validate.field).string.max_len = 256]; +} + +message Request { + oneof op { + option (buf.validate.oneof).required = true; + PingRequest ping = 1; + ListNetnsRequest list_netns = 2; + OpenSocketRequest open_socket = 3; + StatsRequest stats = 4; + } +} +message Response { + oneof result { + option (buf.validate.oneof).required = true; + PingResponse ping = 1; + ListNetnsResponse list_netns = 2; + OpenSocketResponse open_socket = 3; + StatsResponse stats = 4; + Error error = 5; + } +} +``` + +What the schema leaves out is as important as what it includes: +- no socket type, family or protocol number: the broker always uses `AF_NETLINK`/`SOCK_DGRAM|SOCK_CLOEXEC`, and the enum maps to exactly two protocol numbers; +- no netns path; +- no generic-netlink family; +- no socket options. + +### Broker-side filtering (beyond the validators) + +Receive pipeline, fail-closed, each stage counted in `xtcp2_nlbroker_requests_total{op,stage}`: + +1. **Connection gate:** `SO_PEERCRED` uid == `-nlBrokerPeerUid`; at most one client connection (the collector), and a second `accept` is closed immediately; datagrams larger than 4 KiB arrive `MSG_TRUNC` and are treated as `malformed`. +2. **Decode:** `proto.Unmarshal`, then `GetUnknown() == nil` on every message (unknown fields rejected, not stored) → `malformed`. +3. **Validate:** protovalidate `Validate(req)` → `invalid`. +4. **Resolve:** `(dev, inode)` must be in the broker's **current** discovery table, which the broker builds from the host netns, `/run/netns`, `/run/docker/netns` and, on bare metal, the `/proc` scan. A miss returns `not_found` and makes no syscall. +5. **Re-verify the handle (TOCTOU):** the broker `open`s the path from **its own** table entry, `fstat`s the fd, and requires `(dev, inode)` to still match. It also requires `ioctl(NS_GET_NSTYPE) == CLONE_NEWNET`, so a non-network nsfs file planted under `/run/netns` is never entered. On mismatch it returns `not_found` and increments the `race` counter. +6. **Rate limit:** `OpenSocket` ≤ `-nlBrokerOpenRate` (default 100/s, burst 1000 — container churn) and `ListNetns` ≤ 20/s; excess returns `resource_exhausted`. +7. **Create:** on a dedicated locked OS thread: `setns(nsfd, CLONE_NEWNET)` → `socket(AF_NETLINK, SOCK_DGRAM|SOCK_CLOEXEC, proto)` → restore the original netns, or terminate the thread if the restore fails (the existing tainted-M rule from `ns_net_namespace.go` moves here unchanged). +8. **Post-check before sending:** `getsockopt(SO_DOMAIN) == AF_NETLINK` and `SO_PROTOCOL` == the requested protocol; then `sendmsg` with one `SCM_RIGHTS` fd, close the broker's copy, and count `{protocol, source, ok}`. + +Pagination (`ListNetns`): a listing is pinned to a `generation`. If the table changes mid-listing the broker replies `aborted`, and the collector restarts at offset 0. + +### Kernel enforcement on the netlink broker + +- **seccomp (allowlist; the noteworthy rules):** + - `setns`: arg1 `== CLONE_NEWNET` (`SCMP_CMP_EQ 0x40000000`). `nstype 0` means "any type" and is therefore **denied**. + - `socket`: (`AF_NETLINK`, `type & 0xf == SOCK_DGRAM`, `protocol ∈ {0, 4}`) or (`AF_UNIX`, `SOCK_SEQPACKET`, 0); everything else `EPERM`. + - `openat`: only with `(flags & (O_WRONLY|O_RDWR|O_CREAT|O_TRUNC)) == 0`. + - `clone`: denied when any `CLONE_NEW*` bit is set; `clone3` returns `ENOSYS` (the Go runtime uses `clone`). + - denied outright: `execve`/`execveat` (the broker never re-execs), the mount family (`mount`, `umount2`, `pivot_root`, `move_mount`, `open_tree`, `fsopen`/`fsconfig`/`fsmount`/`fspick`, `mount_setattr`), `bpf`, `perf_event_open`, `unshare`, `ptrace`, `process_vm_*`, `keyctl`/`add_key`/`request_key`, `io_uring_*`, `userfaultfd`, the `chmod`/`chown`/`*xattr` families. +- **Landlock:** + - read-only `/run/netns`, `/run/docker/netns` (and `/proc` on bare metal); + - `MAKE_SOCK` in its own socket dir; + - no other write; + - ABI ≥ 6: signal and abstract-UDS scoping. +- **AppArmor:** profile `xtcp2-nlbroker` — deny mount, ptrace, `network inet`, `network packet`, and file writes outside the socket dir. +- **Container:** + - `cap_drop: ALL` + `SYS_ADMIN`; + - `read_only: true`; + - `no-new-privileges`; + - `pids_limit: 256`, memory 64 MiB; + - no listeners besides its UDS. + +**Residual if the broker itself is fully compromised.** Fact 1 plus the seccomp rules bound it. The attacker can enter network namespaces and open `SOCK_DIAG`/`ROUTE` sockets — that is, read socket tables, addresses and routes, which is exactly what the collector legitimately sees. It cannot modify routes or the firewall, kill sockets, read MD5 keys (no `CAP_NET_ADMIN`), load BPF, mount, open inet sockets, or exec. That is the accepted residual. + +### Collector integration + +- **Netns readers:** `attemptOpenAndSetns` plus the `syscall.Socket(… NETLINK_INET_DIAG)` in `netNamespaceInstance` collapse into `nlbroker.OpenSocket(ctx, id, SOCK_DIAG)`. The `LockOSThread`/`setns`/restore code leaves the collector entirely, and with it the tainted-M thread-exhaustion class. +- **Locality:** `nsLocalitySnapshot` → `OpenSocket(id, ROUTE)`, then the existing address/route dump runs on that fd. This needs a new fd-accepting dump seam in `xtcpnl`, because `localnet.BuildSnapshot` already takes parsed `AddrInfo`/`RouteInfo`. +- **Discovery:** `ns_discover.go` is replaced by `ListNetns` at the existing discovery cadence; an unchanged `generation` skips work. +- **Mode:** `-nlBrokerSock ` set → broker mode. In broker mode, `CAP_NET_ADMIN`/`CAP_SYS_ADMIN` move from **fatal** to **unexpected**: their presence logs a warning and sets `xtcp2_collector_excess_caps 1`, so a role regression that re-adds them is visible. + +## The Docker broker + +### What it replaces + +`pkg/dockermeta` dials `docker.sock` and issues four requests: +- `GET /version`; +- `GET /containers/json`; +- `GET /containers/{id}/json`; +- `GET /events` filtered to container `start`/`die`/`destroy`. + +The consumer (`applyEnrichment`, `pkg/xtcp/enrich.go:242`) uses exactly four values per container: ID, name, image, and the netns inode (from `stat(NetworkSettings.SandboxKey)`). The full inspect document — tenant `Env` (often secrets), `Mounts`, `Labels`, `HostConfig`, `Cmd` — is decoded nowhere, but today it is delivered into the collector's memory. Read access to `docker.sock` is also write access to the whole Docker API. + +This is a **typed broker, not an HTTP proxy**. There is no request path, method or body to filter, because the collector cannot express a Docker request at all. + +### gRPC service + +```proto +// proto/xtcp_dockerbroker/v1/dockerbroker.proto +syntax = "proto3"; +package xtcp_dockerbroker.v1; +import "buf/validate/validate.proto"; + +service DockerMetaService { + // Snapshot of running containers that own a network namespace. + rpc ListContainers(ListContainersRequest) returns (ListContainersResponse); + // Index changes after a generation; FAILED_PRECONDITION if the generation + // has aged out of the broker's event ring (the client re-lists). + rpc WatchContainers(WatchContainersRequest) returns (stream ContainerEvent); +} + +message ListContainersRequest {} +message ListContainersResponse { + repeated Container containers = 1 [(buf.validate.field).repeated.max_items = 4096]; + uint64 generation = 2; +} + +message WatchContainersRequest { + uint64 since_generation = 1 [(buf.validate.field).uint64.gt = 0]; +} + +// The projection. Deliberately absent: env, labels, mounts, command, host +// config, pid, sandbox path — nothing without a consumer crosses the socket. +message Container { + string id = 1 [(buf.validate.field).string.pattern = "^[0-9a-f]{64}$"]; + string name = 2 [(buf.validate.field).string = {min_len: 1, max_len: 128, + pattern: "^[A-Za-z0-9][A-Za-z0-9_.-]*$"}]; // Docker's name charset + string image = 3 [(buf.validate.field).string = {min_len: 1, max_len: 255, + pattern: "^[A-Za-z0-9][A-Za-z0-9._/:@-]*$"}]; // ref or sha256: + uint64 netns_inode = 4 [(buf.validate.field).uint64.gt = 0]; +} + +message ContainerEvent { + enum Kind { KIND_UNSPECIFIED = 0; KIND_ADDED = 1; KIND_REMOVED = 2; } + Kind kind = 1 [(buf.validate.field).enum = {defined_only: true, not_in: [0]}]; + Container container = 2 [(buf.validate.field).required = true]; + uint64 generation = 3 [(buf.validate.field).uint64.gt = 0]; +} +``` + +The collector mirrors the index locally — `List` once, then `Watch` — so the per-record `Lookup(netnsInode)` stays an in-memory O(1) map read. No RPC sits on the hot path. There is deliberately no per-container lookup RPC: it would add surface and a per-record round trip. + +### Broker-side filtering (beyond the validators) + +1. **A fixed outbound request set.** The broker's Docker client can issue exactly the four GETs above, with URLs built from constants plus a container id that has itself passed `^[0-9a-f]{64}$`. No code path builds a URL from client input. Collector RPCs are answered from the broker's own index and **never trigger a Docker call**, so they cannot amplify against dockerd. +2. **Docker client hardening:** + - bodies read through `io.LimitReader`: list ≤ 8 MiB, inspect ≤ 1 MiB, one event line ≤ 64 KiB; + - strict decoding into projection structs (nothing else retained); + - `CheckRedirect` refuses redirects; + - no hijack or upgrade; + - per-request timeouts. +3. **Outbound validation.** Every `Container` passes protovalidate before it enters the index. A tenant container whose name or image breaks the charset is **dropped**, not sanitized, and counted in `xtcp2_dockerbroker_rejected_total{field}`. Container names and images are tenant-controlled strings that end up in telemetry rows. +4. **Sandbox path discipline.** Host-network containers (no `SandboxKey`) are omitted. `SandboxKey` must `filepath.Clean` to a direct child of `/run/docker/netns/`, or it is never `stat`ed. Inode 0 or a stat error → omitted. +5. **gRPC server hardening** (the [Phase 3 checklist](phase-3-writer-split.md#wire-format-and-the-protocol-decision)): + - UDS only; + - `MaxRecvMsgSize` 4 KiB (requests are tiny); `MaxSendMsgSize` sized for 4096 containers; + - no reflection service, no health service (health is `-healthcheck` mode); + - no compression codecs registered; + - keepalive enforcement policy; + - `MaxConcurrentStreams` 4, and one `Watch` stream per peer; + - interceptors, in order: peer-credential check (a custom `TransportCredentials` exposing `SO_PEERCRED` as `AuthInfo`; grpc-go has no stock support), then a per-RPC rate limit, then PR-4's validating interceptor. +6. **Watch backpressure.** Each stream has a bounded send queue; a slow consumer gets its stream reset (the collector re-lists) instead of unbounded broker memory. An event ring of the last 1024 generations backs `since_generation`. + +### Kernel enforcement on the Docker broker + +- **Container:** + - `network_mode: none`, `cap_drop: ALL` with nothing added; + - uid `xtcp2-docker` plus `group_add` of `docker.sock`'s gid; + - `read_only: true`, `no-new-privileges`; + - `pids_limit: 64`, memory 64 MiB. +- **seccomp:** + - `socket` only `AF_UNIX`; + - no `execve` after start, no `setns`/`unshare`/mount family/`bpf`/`perf_event_open`/`ptrace`/`keyctl`/`io_uring_*`, no `chmod`/`chown`/`xattr`. +- **Landlock:** + - `RESOLVE_UNIX` on `docker.sock` only (ABI ≥ 9; below that, a documented gap); + - read-only `/run/docker/netns`; + - `MAKE_SOCK` in its socket dir; + - no other write. +- **AppArmor:** profile `xtcp2-dockerbroker`. + +**Residual.** The broker is in the docker group, so a full compromise of the broker process is still Docker-API-equals-host-root. What changes is the attack surface that leads there: +- Today it is the whole collector: netlink and parquet parsers, a gRPC and HTTP server, the S3 client, and `Set`. +- After Phase 5 it is two typed RPCs with 4 KiB request caps, where no request field reaches Docker. + +Tightening further means dockerd over TCP+mTLS with an authorization plugin keyed on the broker's client cert, enforcing GET-only at dockerd ([decision](#decisions-to-close-during-phase-5)). + +## Syscall profiles for every process + +Ships first (PR-19) and does not depend on the brokers. Applied to today's collector, it immediately removes `bpf`, `perf_event_open`, `unshare`, mount and non-network `setns` from an RCE's reach, while the brokers are being built. + +| Process | Notable allows | Notable denies | AppArmor | +|---|---|---|---| +| collector, v1 (today's single process) | `setns` only `CLONE_NEWNET`; `socket` `AF_NETLINK`/`AF_INET`/`AF_INET6`/`AF_UNIX`; `io_uring_*` only in the `-ioUring` profile variant | `bpf`, `perf_event_open`, mount family, `unshare`, `clone` with `CLONE_NEW*`, `ptrace`, `process_vm_*`, `keyctl` family, `userfaultfd`, `chmod`/`chown`/`xattr` families (the fleet has no unix listeners to chmod) | `xtcp2-collector`: deny mount, ptrace, `network packet`/`raw`, writes outside the state dir | +| collector, v2 (broker mode) | as v1 **minus `setns`** | v1 set + `setns` | same | +| netlink broker | [above](#kernel-enforcement-on-the-netlink-broker) | | `xtcp2-nlbroker` | +| Docker broker | [above](#kernel-enforcement-on-the-docker-broker) | | `xtcp2-dockerbroker` | +| writer | `AF_UNIX` only; file ops in its target dirs | everything network; `setns`/mount/`bpf`/… | `xtcp2-writer` | + +- **Allowlists, generated and gated.** Each profile is generated from the full test matrix (microVM runs under a recording hook or `strace -f`), committed as Docker seccomp JSON, and enforced in CI. The suite runs under the profile with `SCMP_ACT_ERRNO` plus logging, so a new syscall fails CI rather than production. +- **Delivery:** + - containers: `security_opt: seccomp=.json` (runc applies it before exec, covering the whole process) plus `apparmor=`; + - bare metal: `SystemCallFilter=`/`RestrictNamespaces=net` in the unit, or an in-process filter installed at startup. +- **Apply once.** In-process filters stack across `execve` like Landlock, so the soft restart inherits the filter and must not re-install it ([Phase 2](phase-2-kernel-confinement.md#pr-8--landlock-self-confinement)). +- **Regression guard.** The fleet-gate test asserts each container's `SecurityOpt` names its profile; `unconfined` or `docker-default` fails the gate. + +## PR map + +``` +PR-19 sec/w12-syscall-profiles ───────────────────────────────▶ (ships with Phase 2) +PR-18 sec/w10-spike ──▶ PR-20 sec/w10-nlbroker ──┐ + PR-21 sec/w11-dockerbroker ┼──▶ PR-22 sec/w10-collector-deprivilege ──▶ PR-23 (deployment repo) +``` + +- [ ] **PR-18 — spike:** verify the [kernel facts](#kernel-facts-the-design-rests-on) and run every collector feature with zero capabilities on broker-opened fds (Test P); record which link-monitor reads (ethtool `GET`, RDMA, devlink) need a typed broker op, if any. +- [ ] **PR-19 — syscall profiles:** v1 collector seccomp + AppArmor profiles, the CI gate, and the fleet-gate assertion (Tests R, T). +- [ ] **PR-20 — netlink broker:** `proto/xtcp_nlbroker/v1`, `xtcp2 -nlbroker` (`pkg/nlbroker`), the client, and the collector seams (Tests Q, R). +- [ ] **PR-21 — Docker broker:** `proto/xtcp_dockerbroker/v1`, `xtcp2 -dockerbroker` (`pkg/dockerbroker`, reusing `dockermeta`'s decoding behind the fixed request set), and the collector mirror (Tests S, T). +- [ ] **PR-22 — collector de-privilege:** + - broker mode as the fleet default; + - excess-capability warning and gauge; + - token and secret files readable by group `xtcp2`; + - the capability check rewritten per mode. +- [ ] **PR-23 — deployment (deployment repo):** + - containers and profiles; + - the rollout steps below; + - alerts with `promtool` cases; + - fleet-gate assertions (Test U). + +## Tests + +Same conventions as Phases 2–4 (Cat: P positive, N negative, B boundary, C corner). P, R and T run in the child-process harness on Linux (skip-aware). Q and S are pure, using injected syscall seams and a fake dockerd. U is microVM, recorded. + +### Test P — kernel facts and feature parity (`TestBrokerFdSemantics`, spike → permanent) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| P1 | P | inet_diag dump on a broker-opened fd from a zero-cap process | broker (CAP_SYS_ADMIN) opens `SOCK_DIAG` in a test netns with 50 TCP sockets; child with `CapEff 0` dumps | all 50 sockets returned, `INET_DIAG_INFO` (tcp_info) present | +| P2 | N | `SOCK_DESTROY` through the same fd | zero-cap child sends `SOCK_DESTROY` for one socket | `EPERM`; the socket survives | +| P3 | N | route mutation on a broker-opened `ROUTE` fd | zero-cap child sends `RTM_NEWROUTE` | `EPERM`; routing table unchanged | +| P4 | P | address/route dump on a broker-opened `ROUTE` fd | locality snapshot | identical to the snapshot produced by today's `setns` path | +| P5 | C | TCP-MD5 socket in the target netns | socket with `TCP_MD5SIG` set | zero-cap dump carries no `INET_DIAG_MD5SIG`; the same dump with `CAP_NET_ADMIN` does — pins the confidentiality gain | +| P6 | B | socket mark | socket with `SO_MARK` | zero-cap dump has no `INET_DIAG_MARK`; golden-record diff shows no change (no mark field exists) | +| P7 | P | host-netns link-monitor reads with zero caps | rtnetlink link dumps, ethtool `GET`s, RDMA dumps | each `OK`, or listed in the spike report as needing a typed broker op | +| P8 | P | io_uring reader on a broker-opened fd | `-ioUring` path | works unchanged | +| P9 | P | fleet output parity | canary: broker mode vs direct mode, 24 h | identical record schemas; per-netns record counts within noise | +| P10 | C | fd outlives its netns | broker opens, netns deleted, collector dumps | dump returns empty or an error the reader already handles; no crash; fd closed on the netns-delete path | + +### Test Q — netlink broker validation and filtering (`TestNlBroker`, pure) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| Q1 | P | open `SOCK_DIAG` in a discovered netns | valid `(dev, inode)` | reply `open_socket` with exactly one `SCM_RIGHTS` fd; `SO_PROTOCOL == 4` | +| Q2 | N | protocol unspecified | `protocol = 0` | `invalid` (`in: [1, 2]`); no syscall | +| Q3 | N | protocol not in the enum | `protocol = 99` | `invalid` (`defined_only`) | +| Q4 | N | netns not in the discovery table | unknown `(dev, inode)` | `not_found`; injected `setns`/`open` counters stay 0 | +| Q5 | B | zero ids | `inode = 0` | `invalid` (`gt 0`) | +| Q6 | C | netns file replaced between discovery and open | same name, new inode | `fstat` mismatch → `not_found`, counter `race` +1 | +| Q7 | C | non-network nsfs file under `/run/netns` | bind of a UTS ns file | `NS_GET_NSTYPE != CLONE_NEWNET` → rejected; never `setns` | +| Q8 | N | unknown field in the envelope | field 99 set | `malformed` (`GetUnknown` check) | +| Q9 | N | empty oneof | `Request{}` | `invalid` (oneof required) | +| Q10 | B | oversized datagram | 4 KiB + 1 | `MSG_TRUNC` → `malformed`, connection closed | +| Q11 | N | wrong peer uid | injected cred uid 0 or 1234 | rejected at accept; no reply payload, no fd | +| Q12 | N | second concurrent client | two connections | second closed immediately; counter | +| Q13 | B | rate limit | burst + 1 opens | last one `resource_exhausted` | +| Q14 | P | thread hygiene under churn | 10 000 opens with netns create/delete churn | broker OS-thread count bounded (no tainted-M growth) | +| Q15 | C | listing across a generation change | table changes between page 1 and page 2 | page 2 `aborted`; restart at offset 0 yields a consistent list | +| Q16 | N | fd count on error replies | every error path | zero `SCM_RIGHTS` fds attached (cmsg count asserted) | +| Q17 | P | both-ends validation | client validates before send; mutate a bound field post-validate | broker-side `invalid` — the layering is observable | + +### Test R — netlink broker syscall profile (`TestNlBrokerProfile`, child harness) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| R1 | P | the two permitted privileged calls | `setns(fd, CLONE_NEWNET)` then `socket(AF_NETLINK, SOCK_DGRAM, 4)` | `OK` | +| R2 | N | `setns` with another type | `CLONE_NEWUTS`, `CLONE_NEWIPC` | `EPERM` | +| R3 | C | `setns` with `nstype 0` ("any") | net ns fd, nstype 0 | `EPERM` — the filter requires exactly `CLONE_NEWNET` | +| R4 | N | inet or packet socket | `AF_INET`, `AF_INET6`, `AF_PACKET` | `EPERM` | +| R5 | N | other netlink protocol | `NETLINK_GENERIC`, `NETLINK_NETFILTER` | `EPERM` | +| R6 | N | BPF and perf | `bpf(BPF_PROG_LOAD)`, `perf_event_open` | `EPERM` | +| R7 | N | mount family | `mount`, `umount2`, `move_mount`, `fsopen`, `mount_setattr` | `EPERM` | +| R8 | N | new namespaces | `unshare(CLONE_NEWUSER)`, `clone` with `CLONE_NEWNS` | `EPERM` | +| R9 | N | write-mode open | `openat(…, O_WRONLY)` | `EPERM` | +| R10 | N | exec | `execve` of self | `EPERM` | + +### Test S — Docker broker (`TestDockerBroker`, fake dockerd) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| S1 | P | list with a host-network container | fake dockerd: 3 containers, one host-net | 2 returned; fields exactly `id`, `name`, `image`, `netns_inode` | +| S2 | P | projection drops sensitive fields | inspect bodies carry `Env` secrets, `Mounts`, `Labels` | the serialized response contains none of those values (byte search) | +| S3 | N | tenant name with control characters | `"web\n01"` | container dropped from the index; `rejected_total{field="name"}` +1; others unaffected | +| S4 | B | name at 128 and 129 chars | two containers | 128 accepted; 129 dropped | +| S5 | N | image with whitespace | `"img latest"` | dropped | +| S6 | C | hostile `SandboxKey` | `"/etc/passwd"`, `"/run/docker/netns/../../etc"` | omitted before any `stat`; counter | +| S7 | N | Docker redirect | 3xx from fake dockerd | not followed; refresh error; previous snapshot retained | +| S8 | B | oversized list body | 8 MiB + 1 | `LimitReader` error; previous snapshot retained; counter | +| S9 | P | outbound request set is fixed | recording transport across the whole suite | only `GET` to `/version`, `/containers/json`, `/containers/<64 hex>/json`, `/events?filters=` | +| S10 | P | collector RPCs never reach dockerd | 1 000 `ListContainers` | fake-dockerd request count unchanged by them | +| S11 | P | watch delivers changes | start/die events | `KIND_ADDED`/`KIND_REMOVED` with strictly increasing `generation` | +| S12 | C | aged-out generation | `since_generation` older than the ring | `FAILED_PRECONDITION`; client re-lists | +| S13 | C | slow watch consumer | client stops reading | stream reset at the queue bound; broker memory flat | +| S14 | N | second watch from the same peer | two streams | second `RESOURCE_EXHAUSTED` | +| S15 | N | wrong peer uid | injected cred | `Unauthenticated` | +| S16 | B | request above `MaxRecvMsgSize` | 4 KiB + 1 | `ResourceExhausted` from gRPC | +| S17 | N | reflection and health absent | reflection and health clients | `Unimplemented` | +| S18 | C | dockerd down at broker start | no socket | empty snapshot with a stale flag; collector attribution degrades; recovers when dockerd returns | + +### Test T — Docker broker and collector profiles (`TestDockerBrokerProfile`, `TestCollectorProfile`) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| T1 | N | Docker broker inet socket | `socket(AF_INET)` | `EPERM` | +| T2 | N | Docker broker dials another unix socket | connect to a non-`docker.sock` UDS | `EACCES` on ABI ≥ 9; reachable on ABI ≤ 8 (documented gap, asserted by probed ABI) | +| T3 | N | Docker broker exec or write | `execve`; write outside its socket dir | `EPERM`; `EACCES` | +| T4 | N | collector v1: BPF, perf, mount, non-net `setns` | each call | `EPERM` | +| T5 | P | collector v1: the full feature suite | unit + microVM matrix under the profile | green; profile log shows zero denials | +| T6 | N | collector v2: any `setns` | `setns(fd, CLONE_NEWNET)` | `EPERM` — broker mode removes the syscall entirely | + +### Test U — deployment (microVM, recorded) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| U1 | P | collector is unprivileged | `docker inspect` + `/proc/self/status` | `User` = `xtcp2`; `CapAdd` empty; `CapEff 0`; no `docker.sock`, `/run/netns` or `/run/docker/netns` mount | +| U2 | P | monitoring parity | 20 containers, churn | per-netns socket records and container attribution identical to direct mode (P9's microVM twin) | +| U3 | N | collector RCE simulation | test-build hook runs a probe list in-process | `nft` flush, `ip route add`, `bpf`, `setns`, `SOCK_DESTROY`, `connect(docker.sock)` all fail (`EPERM`/`ENOENT`) | +| U4 | P | the egress rule holds against the collector | U3 plus an attempt to delete Phase 4's nftables rule | rule intact — closes the gap noted in [Phase 4](phase-4-egress-fleet-lockdown.md#definition-of-done) | +| U5 | N | netlink broker killed | `docker kill` | existing fds keep the readers running; new namespaces unmonitored; `Xtcp2NlBrokerDown` fires; recovers on restart | +| U6 | N | Docker broker killed | `docker kill` | attribution goes stale (flagged); `Xtcp2DockerBrokerDown` fires; recovers | +| U7 | P | soft restart in broker mode | tuning `Set` | re-exec reconnects to both brokers; no capability needed at any point | +| U8 | C | socket dir permissions | another uid tries both sockets | `EACCES` at connect | +| U9 | N | profile regression | redeploy a container with `seccomp=unconfined` | fleet-gate fails the play | + +## Deployment and rollout + +Five steps, each shippable and reversible on its own: + +1. **Profiles on today's collector (PR-19):** custom seccomp + AppArmor on the existing container; canary; T5 green. Rollback: drop the `security_opt`. +2. **Image with broker modes** (idle — no container uses them). +3. **Broker containers added** next to the collector (idle). +4. **Collector in broker mode, capabilities still present:** `-nlBrokerSock`, `-dockerBrokerSock` set; parity soak (P9, U2). Rollback: unset the flags. +5. **De-privilege:** + - collector uid → `xtcp2`; + - `cap_drop: ALL` with nothing added; + - `docker.sock`, `/run/netns` and `/run/docker/netns` mounts removed; + - token and secret files group `xtcp2` `0440`. + + Rollback: restore the previous container spec — the only step that touches the collector's privilege. + +Alerts (vmalert + `promtool test rules`): +- `Xtcp2NlBrokerDown` +- `Xtcp2DockerBrokerDown` +- `Xtcp2BrokerAuthRejects` (any rejected peer) +- `Xtcp2DockerBrokerRejected` (a tenant container dropped by the charset rules — benign or probing; worth seeing) +- `Xtcp2CollectorExcessCaps` (the collector regained capabilities) + +## Decisions to close during Phase 5 + +| # | Decision | Recommendation | +|---|---|---| +| new | netlink broker transport | proto over `SOCK_SEQPACKET` with `SCM_RIGHTS` — gRPC cannot pass fds, and a gRPC control call plus a side channel adds a correlation protocol for nothing | +| new | netlink broker model: pass fds vs run dumps itself | pass fds — zero-copy hot path unchanged, and kernel fact 1 strips privilege from the fd. A broker-executed op only for a `CAP_NET_ADMIN`-gated read PR-18 proves necessary, via the admission checklist | +| new | Docker access: typed gRPC broker vs filtering HTTP proxy in front of `docker.sock` | typed broker — a proxy filters an HTTP surface by method/path patterns and must also rewrite inspect bodies to keep `Env` out; the typed API makes both properties structural | +| new | collector stays in the host netns? | yes for now (listeners, link monitor, S3). Later option: with every namespace reached through broker fds, the collector could run in its own netns, turning egress pinning into ordinary container policy | +| new | dockerd over TCP+mTLS with an authz plugin enforcing GET-only for the broker | deferred — strongest bound on the Docker broker's residual; revisit once the broker is in | +| new | bare-metal `/proc` discovery | `CAP_SYS_PTRACE` lives in the netlink broker only; the collector never holds it | +| new | dedicated uids | `xtcp2`, `xtcp2-docker`, `xtcp2-writer` — none shared with each other or with the ipfeed sidecar's 65534 | + +## Release checklist + +- [ ] PR-18 spike report merged: kernel facts confirmed on fleet kernels; P1–P10 green; link-monitor reads classified +- [ ] PR-19 profiles canaried on today's collector; T4/T5 green; fleet-gate asserts the profile names +- [ ] PR-20/PR-21: Q, R, S, T green; both protocol surfaces reviewed line by line; coverage-guided fuzz targets for the nlbroker datagram decoder and the Docker broker's inbound RPCs running continuously +- [ ] PR-22: broker mode default; excess-caps gauge and alert live +- [ ] PR-23: rollout steps 1–5 on canaries with a parity soak before step 5; U1–U9 recorded +- [ ] [metrics.md](../metrics.md): `xtcp2_nlbroker_*`, `xtcp2_dockerbroker_*`, `xtcp2_collector_excess_caps` + +## Definition of done + +On a canary running Phase 5 with a deliberately crippled collector build (Phase 1 policy stubbed, RCE probe hook compiled in): + +- [ ] the collector runs as `xtcp2` with `CapEff 0`, no `docker.sock`, no namespace mounts (U1) +- [ ] every RCE probe fails: firewall and route changes, `bpf`, `perf_event_open`, `setns`, mount, `SOCK_DESTROY`, Docker API (U3); Phase 4's egress rule survives (U4) +- [ ] monitoring output is unchanged against direct mode (P9, U2) +- [ ] the netlink broker can enter only network namespaces and open only `SOCK_DIAG`/`ROUTE` sockets, enforced by seccomp (R1–R10); the Docker broker can issue only its four GETs (S9) and returns only the projection (S2) +- [ ] each broker's surface is exactly its listed ops: a descriptor test fails on any new message arm or RPC without a test-table row diff --git a/docs/security-hardening/plan-configservice-set-hardening.md b/docs/security-hardening/plan-configservice-set-hardening.md index b628805c..f4d46f27 100644 --- a/docs/security-hardening/plan-configservice-set-hardening.md +++ b/docs/security-hardening/plan-configservice-set-hardening.md @@ -3,7 +3,7 @@ | | | |---|---| | **Status** | Plan — not yet implemented; checkboxes track progress. Execution order, PR-level steps, and phase gates: [implementation-plan.md](implementation-plan.md) | -| **Tracks** | [security-vulnerability-configservice-set-fs-write.md](security-vulnerability-configservice-set-fs-write.md) (advisory, layers L1–L6) | +| **Tracks** | [security-vulnerability-configservice-set-fs-write.md](security-vulnerability-configservice-set-fs-write.md) (advisory, layers L0–L7) | | **Repos** | `randomizedcoder/xtcp2` (this repo), the image repo (image), the deployment repo (deployment), the infra repo (SSM/IAM) | | **Owner / review** | TBD | @@ -22,6 +22,9 @@ - [W7 — deployment guardrails (L6)](#w7--deployment-guardrails-l6) - [W8 — tests, CI, and fleet-config gates](#w8--tests-ci-and-fleet-config-gates) - [W9 — documentation](#w9--documentation) + - [W10 — netlink broker and collector de-privilege (L7)](#w10--netlink-broker-and-collector-de-privilege-l7) + - [W11 — Docker broker (L7)](#w11--docker-broker-l7) + - [W12 — syscall profiles (L7)](#w12--syscall-profiles-l7) - [Sequencing and dependencies](#sequencing-and-dependencies) - [Fleet rollout runbook](#fleet-rollout-runbook) - [End-state verification checklist](#end-state-verification-checklist) @@ -38,10 +41,11 @@ 4. Runtime config changes cannot repoint egress (`dest`, `s3_endpoint`) or listeners/auth without a redeploy. 5. Every applied `Set` is auditable. 6. The collector process ends up with **zero** filesystem write capability. +7. The collector holds no capabilities, no `docker.sock` and no namespace handles. The privileged reads it needs go through two brokers, each enforced by validators on both ends, broker-side filtering, and a syscall profile. **Non-goals** (tracked in the advisory's [honest limits](security-vulnerability-configservice-set-fs-write.md#7-honest-limits--what-this-does-not-fix)) -- Containing the collector's `NET_ADMIN`/`SYS_ADMIN`/`setns` powers or the `docker.sock` mount — inherent to the product's function; a Docker-API proxy is a separate future design. +- Removing the brokers' own privilege (W10/W11): the netlink broker keeps `CAP_SYS_ADMIN`, limited by seccomp to `setns(CLONE_NEWNET)`, and the Docker broker stays in the docker group. Their APIs bound the residual ([Phase 5](phase-5-privilege-brokers.md)). - Network egress confinement at the L7 layer (only the *config* surface is frozen here). - The ipfeed sidecar's write path (it already runs as uid 65534 into one subdirectory; adopting the writer is optional follow-up). @@ -63,6 +67,10 @@ | I12 | No audit trail of applied `Set` calls | `Set` handler | Medium | W6 | | I13 | Collector holds write access it does not need (state-integrity/persistence risk under RCE) | architecture | Medium | W4, W5 | | I14 | `file:` destination (stdlib, no build tag) and the netlink capture writer (no build gate) are compiled into **every** production image | `pkg/xtcp/destinations_file.go`, `pkg/xtcp/netlinker.go:70`, [build-flavors.md](../build-flavors.md) | High | W0 | +| I15 | `/run/docker.sock` mounted in the collector: Docker API access = host root after an RCE, and dockerd writes bypass W4/W5 | deployment role, `pkg/dockermeta` | Critical (post-RCE) | W11 | +| I16 | `CAP_NET_ADMIN` in the host netns: host firewall/route rewrite (defeats W7 egress pinning), `SOCK_DESTROY` of any socket, TCP-MD5 keys in inet_diag dumps | `pkg/xtcp/init_capabilities.go`, deployment role | High (post-RCE) | W10 | +| I17 | `CAP_SYS_ADMIN` under Docker's default seccomp profile: `bpf`, `perf_event_open`, `unshare`, `setns` of any type; `mount` held back only by AppArmor | deployment role | High (post-RCE) | W10, W12 | +| I18 | The collector receives full container inspect documents (tenant `Env`, mounts, labels) though it uses four fields | `pkg/dockermeta` | Medium (confidentiality) | W11 | ## Workstreams @@ -179,7 +187,7 @@ Changes in the deployment repo (`ansible/roles/monitoring/xtcp2/`) and the image - [ ] Retire the v1.x revision: make `LISTENER_AUTH_MODE` impossible to disable via role defaults; add a `test_xtcp2_fleet_gate.py` assertion that auth env is present - [ ] Bind guidance: prefer UDS for the control plane (`xtcp2ctl` workflow documented), else loopback + `tasks/firewall.yml` rules for 8889 with TTL 3 - [ ] I11: state-dir guard — minimum: node-level alert on `/var/lib/xtcp2` usage (vmagent); preferred: XFS project quota (or systemd quota for bare metal) sized to ~2× the ASN cache (~200 MiB headroom) -- [ ] Egress pinning: restrict the collector's egress to the per-DC S3 endpoints (nftables rule on its cgroup — host netns, so not container network policy; not host-wide SG/UFW, which would hit co-located workloads); pair with W2's frozen `s3_endpoint` +- [ ] Egress pinning: restrict the collector's egress to the per-DC S3 endpoints (nftables rule on its cgroup — host netns, so not container network policy; not host-wide SG/UFW, which would hit co-located workloads); pair with W2's frozen `s3_endpoint`. This holds against a stolen admin token. Against a collector RCE it holds only once W10 removes `CAP_NET_ADMIN` from the collector, which can otherwise delete the rule - [ ] W5 deployment: second container + `:ro` collector state dir + shared socket dir; canary via the existing fleet-gate mechanism; revert via `xtcp2_revert.yml` - [ ] Regression-guard the load-bearing container defaults in the fleet-gate test (`docker inspect`): `HostConfig.PidMode` not `host`, `ReadonlyRootfs` true, `AppArmorProfile` == `docker-default` (not `unconfined`), every bind except the state dir `:ro`, and a probe that `mount -o remount,bind,rw` on a `:ro` bind fails `EACCES` inside the container - [ ] Pin the new image tag in the image repo; bump the deployment repo role defaults in lockstep (role gate test asserts image↔config compatibility) @@ -205,6 +213,75 @@ Changes in the deployment repo (`ansible/roles/monitoring/xtcp2/`) and the image - [ ] [README.md](../README.md): link this plan from the advisory entry - [ ] Update the advisory's layer table with ✅ per workstream as they land +### W10 — netlink broker and collector de-privilege (L7) + +Advisory §5.8, [Phase 5](phase-5-privilege-brokers.md#the-netlink-broker). Moves `setns` and both capabilities out of the collector. + +- [ ] Spike (PR-18): + - confirm on fleet kernels that a capability-less process using a broker-opened netlink fd gets unprivileged semantics (`__netlink_ns_capable` checks opener **and** sender); + - run every collector feature with zero capabilities; + - classify the link-monitor reads. +- [ ] `proto/xtcp_nlbroker/v1/nlbroker.proto` — message-only, with protovalidate rules: + - `NetnsId{dev, inode}` with `gt 0`; + - `NetlinkProtocol` enum `in: [SOCK_DIAG, ROUTE]`; + - oneof required; + - `ListNetns` paginated at ≤ 256 entries per datagram. +- [ ] `xtcp2 -nlbroker` (`pkg/nlbroker`): + - `SOCK_SEQPACKET` UDS, replies carrying the fd in `SCM_RIGHTS`; + - a discovery table the broker owns; + - `fstat` + `NS_GET_NSTYPE` re-check before `setns`; + - rate and single-connection limits; + - `SO_PEERCRED` must equal the collector uid. +- [ ] Collector seams: + - `netNamespaceInstance`/`attemptOpenAndSetns` and `nsLocalitySnapshot` → `OpenSocket`; + - `ns_discover.go` → `ListNetns`; + - new `-nlBrokerSock` flag. +- [ ] De-privilege (PR-22): + - collector runs as uid `xtcp2` with `cap_drop ALL`; + - token and secret files group `xtcp2`, `0440`; + - capability check per mode; + - new `xtcp2_collector_excess_caps` gauge. + +**Acceptance**: P1–P10, Q1–Q17, R1–R10, U1–U5 ([Phase 5 tests](phase-5-privilege-brokers.md#tests)). +**Estimate**: 8–12 days. **Risk**: a link-monitor read that genuinely needs `CAP_NET_ADMIN` → typed broker op via the admission checklist. + +### W11 — Docker broker (L7) + +Advisory §5.8, [Phase 5](phase-5-privilege-brokers.md#the-docker-broker). Replaces the collector's `docker.sock` mount with a typed API. + +- [ ] `proto/xtcp_dockerbroker/v1/dockerbroker.proto`: + - `DockerMetaService { ListContainers; WatchContainers (server stream) }`; + - a `Container` projection with charset patterns on `id` (64 hex), `name` and `image`, and `netns_inode gt 0`. +- [ ] `xtcp2 -dockerbroker` (`pkg/dockerbroker`): + - a fixed outbound request set: `/version`, `/containers/json`, `/containers/{id}/json`, and `/events` with a constant filter; + - body caps, no redirects, strict projection decoding; + - outbound protovalidate — tenant entries that fail are dropped and counted; + - `SandboxKey` must be a direct child of `/run/docker/netns/`. +- [ ] gRPC hardening: + - UDS only, `MaxRecvMsgSize` 4 KiB; + - no reflection, health or compression; + - one watch stream per peer; + - peer-credential, rate-limit and validating interceptors. +- [ ] Collector: + - `dockermeta` replaced by a List+Watch mirror (lookups stay in-memory); + - `docker.sock` mount removed from the collector container. + +**Acceptance**: S1–S18, T1–T3, U6. **Estimate**: 5–8 days. + +### W12 — syscall profiles (L7) + +Advisory §5.8, [Phase 5](phase-5-privilege-brokers.md#syscall-profiles-for-every-process). The v1 collector profile has no dependencies and ships with Phase 2. + +- [ ] Per-process seccomp allowlists (collector v1/v2, netlink broker, Docker broker, writer): + - generated from the test matrix and committed as Docker seccomp JSON; + - `setns` only with `CLONE_NEWNET` (none at all in collector v2); + - `bpf`, `perf_event_open`, the mount family, `unshare`, `ptrace`, `keyctl` and the `chmod`/`chown`/`xattr` families denied. +- [ ] Per-process AppArmor profiles replacing `docker-default` (extends W7's regression guard to assert the profile names). +- [ ] CI runs the suite under each profile (deny + log); the fleet-gate rejects `unconfined` or a missing profile. +- [ ] Bare metal: matching `SystemCallFilter=`/`RestrictNamespaces=net` in the systemd units. + +**Acceptance**: R1–R10, T4–T6, U9. **Estimate**: 3–5 days. + ## Sequencing and dependencies ``` @@ -214,11 +291,13 @@ Phase 1 (release vNext): W0 (image rebuild) + W1 ─▶ W2 ─▶ W3 (pure co Phase 2: W4 (Landlock, optional pivot_root) ──▶ W6 (Set audit) ──▶ W7 quota + audit ship Phase 3: W5 (writer split) ──▶ W7 (second container, :ro flip) ──▶ W9 Phase 4: W7 egress pinning + v1.x retirement; W8 end-to-end advisory regression as a standing gate +Phase 5: W12 profiles (ship with Phase 2) ─▶ W10 netlink broker ∥ W11 Docker broker ─▶ collector de-privilege ``` - W1, W2, W3 are independent of each other — parallelizable; W2 needs W1's pattern only for shared test fixtures. - W4 should land **before** W5 (fleet already kernel-protected while the split is built). -- W5 is the only workstream that changes deployment topology; it is also fully reversible until the collector's `:ro` flip. +- W5 and W10/W11 change deployment topology. W5 is fully reversible until the collector's `:ro` flip; W10/W11 until the collector drops its capabilities (Phase 5 rollout step 5). +- W10/W11 can start once Phase 2 lands and run alongside Phase 3. W7's egress rule only holds against a collector RCE after W10. - Every phase ships independently; no phase requires the next. ## Fleet rollout runbook @@ -244,6 +323,9 @@ With admin token in hand on a canary host, every item must fail safely: - [ ] Direct process check: as the collector uid, `os.WriteFile("/tmp/x")` fails (zero write rules, post-W5); the writer still publishes a fresh ASN cache - [ ] `kill -9` the writer mid-baseline-write: old baseline intact, monitor degrades-not-exits, counter increments - [ ] `/var/lib/xtcp2` fill attempt (legal writes within quota bounds) trips the size alert +- [ ] Collector (W10–W12): `CapEff 0` as uid `xtcp2`, no `docker.sock`. On a crippled build, an RCE probe cannot change the firewall or routes, load BPF, `setns`, mount, kill sockets, or reach the Docker API +- [ ] Netlink broker: `setns` with any type other than `CLONE_NEWNET`, and any socket other than netlink `SOCK_DIAG`/`ROUTE` → `EPERM` (seccomp) +- [ ] Docker broker: the only Docker requests ever issued are the four fixed GETs; responses carry no `Env`/mounts/labels - [ ] Production image (W0): `Set` with `dest = "file:/tmp/pwn"` → fatal startup error, no file ever created (the code is absent from the binary); `write_files = 1` → fatal startup error ## Open decisions @@ -258,11 +340,15 @@ With admin token in hand on a canary host, every item must fail safely: | OD-6 | Frozen-field escape hatch lifetime | Time-boxed to one migration window, then remove the flag | | OD-7 | State-dir quota mechanism (alert-only vs XFS project quota) | Alert-only in Phase 2 (cheap), quota when a host survives to Phase 3 | | OD-8 | Does `xtcp2-min` keep `file:` (it is the stdlib-only flavor)? | Drop it there too; only `full`/`-debug` carry the tag | +| OD-10 | Netlink broker: pass fds vs run the dumps in the broker | Pass fds over `SOCK_SEQPACKET`. The kernel checks the sender's capabilities as well as the opener's, so the fd carries no privilege, and the hot path stays zero-copy | +| OD-11 | Docker access: typed gRPC broker vs a filtering HTTP proxy in front of `docker.sock` | Typed broker — the projection and the fixed request set are then structural, not pattern-matched | +| OD-12 | Bound the Docker broker further with dockerd TCP+mTLS and an authz plugin (GET-only for the broker's cert) | Deferred until W11 lands | | OD-9 | Drop the network stdlib destinations (`tcp`/`udp`/`http`/`https`/`unix`/`unixgram`) from the production flavor? | Yes — production needs `s3parquet` only; kills the DEST-repoint exfil class structurally | ## References - Advisory: [security-vulnerability-configservice-set-fs-write.md](security-vulnerability-configservice-set-fs-write.md) — primitives, reachability, container analysis, layer rationale +- Broker design: [phase-5-privilege-brokers.md](phase-5-privilege-brokers.md) — kernel facts, protocols, filtering pipelines, syscall profiles, test matrices - Code touchpoints: `pkg/xtcp/grpc_configService.go`, `pkg/xtcp/destinations_core.go` + `destinations_file.go`, `pkg/xtcp/netlinker.go`, `pkg/xtcp/input_validation.go`, `pkg/linkmonitor/config.go` + `internal/baseline/`, `pkg/ipasn/ipasn.go`, `pkg/listener/listener.go`, `pkg/listenerauth/policy.go`, `cmd/xtcp2/xtcp2.go` (flags, `loadReconfigureConfig:1199`, `execSelfWithConfig:1211`) - Deployment: the deployment repo → `ansible/roles/monitoring/xtcp2/` (defaults, tasks, firewall, fleet-gate test); the image repo (flake, settings docs) - Upstream: [Landlock](https://docs.kernel.org/userspace-api/landlock.html), [go-landlock](https://github.com/landlock-lsm/go-landlock), [buf protovalidate](https://github.com/bufbuild/protovalidate) diff --git a/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md b/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md index fd5848ad..9999a38c 100644 --- a/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md +++ b/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md @@ -3,7 +3,7 @@ | | | |---|---| | **Status** | Draft advisory — no CVE issued (internal assessment) | -| **Severity** | High (authenticated callers, default deployments: unauthenticated) | +| **Severity** | High in the reference container deployment (§4): the only writable host path is the state dir, so the impact is monitor-state tampering and disk-fill. Critical outside it: a bare-metal or hand-run daemon with the binary defaults (auth off, wildcard listener, root) gives any network peer a root file write at an arbitrary path | | **Affected** | Every xtcp2 version with `ConfigService.Set` + `file:` destination / `capture_path` / `baseline_file` / `ipmeta_cache_path` / unix listeners — i.e. all current releases | | **Not affected** | Read-only clients (`Get`, `FlatRecords`, `PollFlatRecords`); hot-path setters (`SetPollFrequency`, `SetS3Upload`, `SetEnvelopeFlush`, `Trigger*` — none touch the filesystem) | | **Scope** | `ConfigService.Set` runtime reconfiguration; see [security.md](../security.md) for listener-auth posture and [grpc-api.md](../grpc-api.md) for the Set soft-restart design | @@ -30,6 +30,7 @@ - [5.5 Layer 5 — privilege separation: a dedicated state-writer process](#55-layer-5--privilege-separation-a-dedicated-state-writer-process) - [5.6 Layer 6 — deployment and runtime guardrails](#56-layer-6--deployment-and-runtime-guardrails) - [5.7 Layer 0 — build-time elimination of write-capable debug features](#57-layer-0--build-time-elimination-of-write-capable-debug-features) + - [5.8 Layer 7 — privilege brokers and syscall profiles](#58-layer-7--privilege-brokers-and-syscall-profiles) - [6. What each layer stops](#6-what-each-layer-stops) - [7. Honest limits — what this does not fix](#7-honest-limits--what-this-does-not-fix) - [8. Testing plan](#8-testing-plan) @@ -46,9 +47,9 @@ 4. `ipmeta_cache_path = ` — parquet cache writer with `os.MkdirAll(dir, 0o750)` — an arbitrary **directory-creation** primitive (`pkg/ipasn/ipasn.go:371`). 5. `grpc_listener`/`prometheus_listener` with `network: unix` — bind a socket at any path and chmod it up to 0777 (proto rule `lte: 511`) (`pkg/listener/listener.go`). -Authentication is **off by default** and the default gRPC listener is wildcard TCP `:8889` ([security.md](../security.md)). In the reference container deployment, the daemon runs as uid 0 with `CAP_NET_ADMIN`+`CAP_SYS_ADMIN` and — in the current role revision — the **only** writable host path is the `/var/lib/xtcp2` bind mount. So the answer to "can `Set` write to the host filesystem from inside the container?" is: **yes, but only under `/var/lib/xtcp2`** (disk-fill DoS, baseline suppression, ASN-cache poisoning, planted files) — which is still serious, because those writes corrupt the very state the monitor's alerts and enrichment depend on. Everything else on the host is read-only to the daemon; the adjacent escalation paths out of the container are `docker.sock:ro` (full Docker API) and the netlink/`setns` capabilities, not `Set`. +Authentication is **off by default** and the default gRPC listener is wildcard TCP `:8889` ([security.md](../security.md)). In the reference container deployment, the daemon runs as uid 0 with `CAP_NET_ADMIN`+`CAP_SYS_ADMIN` and — in the current role revision — the **only** writable host path is the `/var/lib/xtcp2` bind mount. So the answer to "can `Set` write to the host filesystem from inside the container?" is: **yes, but only under `/var/lib/xtcp2`** (disk-fill DoS, baseline suppression, ASN-cache poisoning, planted files) — which is still serious, because those writes corrupt the very state the monitor's alerts and enrichment depend on. Everything else on the host is read-only to the daemon; the adjacent escalation paths out of the container are `docker.sock:ro` (full Docker API) and the netlink/`setns` capabilities, not `Set`. L7 (§5.8) moves both out of the collector. -The remediation plan is layered so that every layer independently blocks the primitives above: build-time removal of the `file:` destination and the netlink capture writer from production images (L0), protovalidate rules on all path fields (L1), a Go runtime path policy + frozen-field diff in the `Set` handler (L2), a server-side validating interceptor so every RPC input is always validated (L3), kernel self-confinement via Landlock (L4), privilege separation — moving the two remaining legitimate writes into a chrooted, capability-less helper process with a path-free API (L5) — and deployment/runtime guardrails (L6). +The remediation plan is layered so that every layer independently blocks the primitives above: build-time removal of the `file:` destination and the netlink capture writer from production images (L0), protovalidate rules on all path fields (L1), a Go runtime path policy + frozen-field diff in the `Set` handler (L2), a server-side validating interceptor so every RPC input is always validated (L3), kernel self-confinement via Landlock (L4), privilege separation — moving the two remaining legitimate writes into a chrooted, capability-less helper process with a path-free API (L5) — and deployment/runtime guardrails (L6). L7 then removes what a container does not: the `docker.sock` mount and both capabilities leave the collector for two single-purpose brokers with typed, validated APIs and tight syscall profiles. --- @@ -146,16 +147,25 @@ Because the container runs in the **host** network namespace and the **initial** - netlink `inet_diag` dumps of every socket — the product's function; - interface/address/route/neighbor configuration — bring links up/down, add or blackhole routes; - netfilter: rewrite the **host** firewall (`iptables`/`nftables`); -- tc qdiscs, bridge/VRF/macvlan config, conntrack-table manipulation. +- tc qdiscs, bridge/VRF/macvlan config, conntrack-table manipulation; +- `SOCK_DESTROY` on any socket in any netns it reaches, and TCP-MD5 keys in its inet_diag dumps (the kernel adds them for `CAP_NET_ADMIN` readers, `tcp_diag_get_aux`). **`CAP_SYS_ADMIN` (initial userns)** -- `setns()` into other network, UTS, IPC, cgroup and time namespaces — every container's netns (the function), and with them other containers' hostname (UTS) and SysV IPC / shared memory (IPC). **Not** mount or PID namespaces: `setns(CLONE_NEWNS)` also requires `CAP_SYS_CHROOT` (dropped), and `setns(CLONE_NEWPID)` only enters descendants of the caller's PID namespace, which neither sibling containers' nor the host's are; +- `setns()` into network, UTS, IPC, cgroup and time namespaces, capability-wise. In the container only **network** namespaces are reachable, because `setns` needs a handle to the namespace: + - the `/run/netns` and `/run/docker/netns` binds hold only network-namespace files; + - the private PID namespace hides other processes' `/proc//ns/*`. + + So the daemon enters every container's netns (the function) and nothing else. Mount and PID namespaces are out of reach regardless: `setns(CLONE_NEWNS)` also requires `CAP_SYS_CHROOT` (dropped), and `setns(CLONE_NEWPID)` only enters descendants of the caller's PID namespace. On bare metal, `/proc//ns/*` makes UTS, IPC (other workloads' SysV shared memory), cgroup and time namespaces reachable too; - `mount()`/`umount2()` within its own mount namespace — tmpfs, bind mounts, and **remounts**: without userns remap the container's mounts are not locked, so capability-wise `mount -o remount,bind,rw` turns any `:ro` bind (or the read-only rootfs) writable. What blocks it today is AppArmor's `docker-default` profile (`deny mount`); Docker's seccomp profile allows `mount` once `CAP_SYS_ADMIN` is held. Verify that the new mount API (`fsmount`/`move_mount`/`mount_setattr`) is mediated on fleet kernels — if not, only a seccomp filter closes it; -- `bpf()` — `CAP_SYS_ADMIN` retains the legacy privileged bpf path; -- `perf_event_open`, namespace creation (`unshare`/`clone` with `CLONE_NEW*`), `quotactl`. +- `bpf()` and `perf_event_open`: Docker's default seccomp profile allows both once `CAP_SYS_ADMIN` is held, and `docker-default` AppArmor does not restrict them. Tracing programs can read kernel memory and write other processes' memory — the most direct path from this container to the host kernel; +- namespace creation (`unshare`/`clone` with `CLONE_NEW*`, new user namespaces included), `quotactl`. + +Honest assessment: this is *not* "a locked-down exporter that happens to have net admin." The filesystem is tightly confined (§4.2), but `NET_ADMIN` + `SYS_ADMIN` in the initial userns make the process host-network-equivalent and namespace-equivalent — an RCE in the daemon can reconfigure host networking, enter any network namespace, and reach `bpf`/`perf_event_open`, regardless of `Set`. That is not inherent to a host-wide socket monitor: +- L4/L5 shrink the *filesystem* angle; +- L7 (§5.8) removes this one. Netlink privilege checks require the sender's capabilities as well as the socket opener's, so a broker can open the per-netns sockets and hand them to a collector that has no capabilities. -Honest assessment: this is *not* "a locked-down exporter that happens to have net admin." The filesystem is tightly confined (§4.2), but `NET_ADMIN` + `SYS_ADMIN` in the initial userns make the process host-network-equivalent and namespace-equivalent — an RCE in the daemon can reconfigure host networking and enter any network, UTS, IPC or cgroup namespace regardless of `Set`. That is inherent to a host-wide socket monitor, not a deployment misconfiguration; §5's layers shrink the *filesystem* angle (L4/L5) but cannot remove this one. Docker's seccomp profile does not help here: it allows namespace entry and `mount` once these capabilities are held. AppArmor does gate mounting (below), which makes it load-bearing. +Docker's seccomp profile does not help here: it allows namespace entry and `mount` once these capabilities are held. AppArmor does gate mounting (below), which makes it load-bearing. ### 4.4 What is restricted and filtered @@ -172,7 +182,7 @@ Everything an attacker (an admin-token holder, or an RCE inside the daemon) stil - **Network (app-level)**: bearer auth on `:9088`/`:8889` with per-host tokens and rotation; `/ip` detail loopback-only; TTL-3 listener replies; UFW scrape rule limited to local bridge subnets; S3 credential via a 0400 file mount, not env. - **Resources**: memory / CPU / pids caps; log rotation. -Not filtered (see §4.6): the Docker API through `/run/docker.sock` (a `:ro` mount is not an API restriction), host-firewall rewriting via `NET_ADMIN`, and network/UTS/IPC/cgroup namespace entry via `SYS_ADMIN`. +Not filtered (see §4.6): the Docker API through `/run/docker.sock` (a `:ro` mount is not an API restriction), host-firewall rewriting via `NET_ADMIN`, and network-namespace entry plus `bpf`/`perf_event_open` via `SYS_ADMIN`. L7 (§5.8) removes all three from the collector. ### 4.5 Can `Set` write to the host filesystem? @@ -196,9 +206,9 @@ No listener-auth env vars are set, so the gRPC admin surface on host-netns `:888 ### 4.6 Adjacent (non-`Set`) escalation paths -- `/run/docker.sock:ro` is a **mount** restriction, not an API restriction — the Docker API itself can create containers that mount the host root. It is mounted for container-id resolution (cgroup → container attribution). Treat a compromised daemon as host-root regardless of `Set`. -- `NET_ADMIN` + `SYS_ADMIN` on host network namespaces (`setns` into every netns) are required for the product's core function and are inherently host-equivalent privileges. -- These are pre-existing, by-design capabilities of the monitoring daemon; this advisory is scoped to `Set`'s write primitives. The layers below shrink them where possible. +- `/run/docker.sock:ro` is a **mount** restriction, not an API restriction — the Docker API itself can create containers that mount the host root. It is mounted for container-id resolution (cgroup → container attribution). Until L7 replaces the mount with a typed Docker broker (§5.8), treat a compromised daemon as host-root regardless of `Set`. +- `NET_ADMIN` + `SYS_ADMIN` on host network namespaces (`setns` into every netns) are what the product's core function needs today. L7 (§5.8) moves them into a netlink broker whose seccomp profile admits only `setns(CLONE_NEWNET)` and netlink `SOCK_DIAG`/`ROUTE` sockets. +- These are pre-existing, by-design capabilities of the monitoring daemon; this advisory is scoped to `Set`'s write primitives. L7 is the layer that removes them from the collector. ### 4.7 Container-escape path idioms @@ -219,7 +229,7 @@ If xtcp2 runs outside this role — a hand-run container with different flags, a ## 5. Remediation plan — belt and braces -Each layer is independently effective; together they remove the entire primitive class. Layers are ordered by effort/benefit — L0 + L1 + L2 are small pure-code/build changes that close the hole; L4/L5 are the kernel-enforced backstops that keep the hole closed even if validation code is later bypassed. +Each layer is independently effective; together they remove the entire primitive class. Layers are ordered by effort/benefit — L0 + L1 + L2 are small pure-code/build changes that close the hole; L4/L5 are the kernel-enforced backstops that keep the hole closed even if validation code is later bypassed; L7 takes the remaining privilege — capabilities and `docker.sock` — out of the collector. ### 5.1 Layer 1 — protovalidate rules on every path-bearing field @@ -361,7 +371,7 @@ Rules enforced by the writer, in code: - **Failure mode:** writer down ⇒ collector logs, bumps a counter, keeps monitoring with its last-known state — identical to today's degrade-not-exit behavior when the baseline is unwritable. `Set` is unaffected. - **Rollback:** the writer keeps `previous.lookup.parquet.zst` semantics exactly as today. -**Why do it, honestly.** Landlock (L4) already blocks every `Set` write primitive for ~10% of the effort. The split adds three things Landlock cannot: (a) **a narrower state-write surface for a compromised collector** — direct file writes are gone, so planting files or arbitrary bytes in the state dir needs the writer's two fixed targets and their schema checks. This is not state integrity: the writer still accepts schema-valid baseline/ipmeta content at the next `seq` (a baseline that hides a link outage, a poisoned-but-well-formed ASN cache), and the collector's `docker.sock` can have dockerd write the state dir — or any host path — without touching the writer; (b) the collector needs **zero** write rules, so its Landlock policy is trivial to audit; (c) the config surface itself shrinks — `baseline_file`/`ipmeta_cache_path` stop being runtime-mutable paths entirely. It does **not** contain the collector's `SYS_ADMIN`/`NET_ADMIN`/docker-sock reach (see [§7](#7-honest-limits--what-this-does-not-fix)); it removes the filesystem angle only. Sequence it last, after L1–L4. +**Why do it, honestly.** Landlock (L4) already blocks every `Set` write primitive for ~10% of the effort. The split adds three things Landlock cannot: (a) **a narrower state-write surface for a compromised collector** — direct file writes are gone, so planting files or arbitrary bytes in the state dir needs the writer's two fixed targets and their schema checks. This is not state integrity: the writer still accepts schema-valid baseline/ipmeta content at the next `seq` (a baseline that hides a link outage, a poisoned-but-well-formed ASN cache), and the collector's `docker.sock` can have dockerd write the state dir — or any host path — without touching the writer; (b) the collector needs **zero** write rules, so its Landlock policy is trivial to audit; (c) the config surface itself shrinks — `baseline_file`/`ipmeta_cache_path` stop being runtime-mutable paths entirely. It does **not** contain the collector's `SYS_ADMIN`/`NET_ADMIN`/docker-sock reach (see [§7](#7-honest-limits--what-this-does-not-fix)); it removes the filesystem angle only. L7 (§5.8) removes that reach. Sequence it last, after L1–L4. ### 5.6 Layer 6 — deployment and runtime guardrails @@ -391,26 +401,67 @@ Remove them from production binaries entirely, using the mechanisms that already After this layer, a `Set` call cannot even *express* a `file:` destination or a capture budget on a production host: the code that would act on them is not in the binary. This is the strongest form of L2's `-allowFileDest` escape hatch: in production the build tag replaces the flag; the debug flavors that compile `file:` still require it. -## 6. What each layer stops +### 5.8 Layer 7 — privilege brokers and syscall profiles + +*(Detailed design, protocols, filtering pipelines and test matrices: [Phase 5](phase-5-privilege-brokers.md).)* + +L4/L5 confine what the collector does with files. Even in the container, they leave it holding the three things that make a compromise host-root-equivalent: +- `docker.sock`; +- `CAP_NET_ADMIN` in the host netns; +- `CAP_SYS_ADMIN` under a seccomp profile that admits `bpf` and `perf_event_open`. -| Threat | L1 proto rules | L2 path policy + frozen fields | L3 interceptor | L4 Landlock | L5 writer split | L6 deploy | -|---|---|---|---|---|---|---| -| Write to arbitrary path (`file:` dest, `capture_path`) | ✅ shape | ✅ root allowlist + freeze | — (enforces L1) | ✅ kernel | ✅ API has no path | — | -| Write under `/var/lib/xtcp2` (baseline poison, cache poison, disk-fill) | ❌ | ❌ (legit use) | — | ❌ (dir allowed until L5) | partial — no direct writes; schema-valid writes via the writer and `docker.sock` remain | ✅ quota + audit | -| MkdirAll anywhere (`ipmeta_cache_path`) | ✅ shape | ✅ | — | ✅ | ✅ | — | -| Planted 0777 unix socket | ✅ charset | ✅ root allowlist | — | ✅ | ✅ | — | -| Symlink redirect of a later write | ❌ (shape ok) | ✅ EvalSymlinks + component Lstat | — | ✅ (read-only dirs) | ✅ chroot | — | -| Unvalidated future RPC input | — | — | ✅ | — | — | — | -| DEST/S3 endpoint repoint exfil | — (scheme ok) | ✅ frozen fields | — | ❌ (network, not fs) | ❌ | ✅ egress pinning | -| RCE in daemon → fs writes/persistence | — | — | — | partial — except state dir; `docker.sock` bypasses | partial — no writable mount; `docker.sock` bypasses | — | +L7 moves all three out of the collector. + +- **Netlink broker** (`xtcp2 -nlbroker`) — the only process that enters namespaces. It has two operations: list network namespaces, and open a netlink `SOCK_DIAG` or `ROUTE` socket inside one. The socket is returned to the collector as an fd over `SCM_RIGHTS`. + - **Why passing the fd is safe:** netlink privilege checks require the socket opener *and* the sender to hold the capability (`__netlink_ns_capable`). A collector with no capabilities that holds the fd gets unprivileged semantics: dumps work, while `SOCK_DESTROY`, route changes, socket marks and TCP-MD5 keys do not. The collector's hot path still reads the same socket, zero-copy. + - **What the broker holds:** `CAP_SYS_ADMIN` only, and no `CAP_NET_ADMIN`. Its seccomp profile admits `setns` only with `CLONE_NEWNET`, and `socket` only for those two netlink protocols. + - **Wire format:** proto over `SOCK_SEQPACKET`, because gRPC cannot carry file descriptors. +- **Docker broker** (`xtcp2 -dockerbroker`) — the only holder of `docker.sock`. It is a typed gRPC service, **not an HTTP proxy**: `ListContainers` and `WatchContainers` return a four-field projection (id, name, image, netns inode). + - It issues exactly four fixed GETs to dockerd, and no request field reaches dockerd. + - Tenant `Env`, mounts and labels never leave the broker. +- **Every broker request is enforced by three independent layers:** + - protovalidate on both ends, with unknown fields rejected; + - broker-side filtering against broker-owned state: ids the broker issued itself, a re-check that the handle hasn't changed since discovery, rate and connection limits, and outbound validation that drops tenant strings failing the charset rules; + - a per-process seccomp allowlist, AppArmor profile and Landlock ruleset. +- **Syscall profiles for every process.** These ship first, against today's collector: + - `setns` restricted to `CLONE_NEWNET`; + - `bpf`, `perf_event_open`, the mount family, `unshare`, `ptrace`, `keyctl` and the `chmod`/`chown`/`xattr` families denied; + - named AppArmor profiles replacing `docker-default`. + +**Result.** The collector runs as a dedicated non-root uid with `CapEff 0`, no `docker.sock` and no namespace handles. Peer credentials on the broker sockets now identify it, which uid 0 never did. + +**Residual.** +- The netlink broker, if compromised, can read what the collector reads, and nothing more. +- The Docker broker is still docker-group-equivalent, but the only way in is two typed RPCs with 4 KiB request caps. + +## 6. What each layer stops -A compromised collector that speaks the Docker API bypasses L4 and L5: dockerd performs the write, outside the collector's Landlock domain and without the writer (§7). +| Threat | L1 proto rules | L2 path policy + frozen fields | L3 interceptor | L4 Landlock | L5 writer split | L6 deploy | L7 brokers + profiles | +|---|---|---|---|---|---|---|---| +| Write to arbitrary path (`file:` dest, `capture_path`) | ✅ shape | ✅ root allowlist + freeze | — (enforces L1) | ✅ kernel | ✅ API has no path | — | — | +| Write under `/var/lib/xtcp2` (baseline poison, cache poison, disk-fill) | ❌ | ❌ (legit use) | — | ❌ (dir allowed until L5) | partial — no direct writes; schema-valid writes via the writer and `docker.sock` remain | ✅ quota + audit | closes the `docker.sock` path | +| MkdirAll anywhere (`ipmeta_cache_path`) | ✅ shape | ✅ | — | ✅ | ✅ | — | — | +| Planted 0777 unix socket | ✅ charset | ✅ root allowlist | — | ✅ | ✅ | — | — | +| Symlink redirect of a later write | ❌ (shape ok) | ✅ EvalSymlinks + component Lstat | — | ✅ (read-only dirs) | ✅ chroot | — | — | +| Unvalidated future RPC input | — | — | ✅ | — | — | — | — | +| DEST/S3 endpoint repoint exfil | — (scheme ok) | ✅ frozen fields | — | ❌ (network, not fs) | ❌ | ✅ egress pinning | ✅ makes the egress rule hold against RCE | +| RCE in daemon → fs writes/persistence | — | — | — | partial — except state dir; `docker.sock` bypasses | partial — no writable mount; `docker.sock` bypasses | — | ✅ no `docker.sock` — with L4/L5, nothing writable remains | +| RCE → host firewall/route rewrite, socket kill, TCP-MD5 key read (`NET_ADMIN`) | — | — | — | — | — | — | ✅ no process holds `NET_ADMIN` | +| RCE → `bpf`/`perf_event_open`/namespace creation/mount (`SYS_ADMIN`) | — | — | — | — | — | — | ✅ collector: no capabilities; broker: seccomp admits `setns(CLONE_NEWNET)` only | +| RCE → Docker API (start a host-root container) | — | — | — | — | — | — | ✅ typed broker: two read RPCs | + +A compromised collector that speaks the Docker API bypasses L4 and L5: dockerd performs the write, outside the collector's Landlock domain and without the writer (§7). L7 removes that path. Layer 0 (§5.7) removes primitives 1–2 at the binary level — the production image simply contains no code that opens files for writing on `Set`'s behalf, so no validation bypass can reach it. ## 7. Honest limits — what this does not fix -- **The collector's own capabilities.** `CAP_NET_ADMIN` + `CAP_SYS_ADMIN` on host namespaces plus `/run/docker.sock:ro` mean a fully compromised collector is host-equivalent regardless of L1–L5. L5 removes its *filesystem* angle; the netns/docker angle is inherent to the product's function (see [§4.3](#43-what-net_admin--sys_admin-actually-grant)). The same holds for L4/L5: they confine the collector's own syscalls, not what dockerd does on its request — and because the writer accepts schema-valid content at the next `seq`, L5 bounds the *shape* of state writes, not their truth. +- **The collector's capabilities, until L7; the brokers' after.** + - Until L7 lands, `CAP_NET_ADMIN` + `CAP_SYS_ADMIN` on host namespaces plus `/run/docker.sock:ro` make a fully compromised collector host-equivalent regardless of L1–L5. L4/L5 confine the collector's own syscalls, not what dockerd does on its request ([§4.3](#43-what-net_admin--sys_admin-actually-grant)). + - L7 moves that privilege into two brokers, and what remains is theirs: + - the netlink broker can read every netns's socket tables and routes, which is exactly the collector's legitimate view; + - the Docker broker is in the docker group, so compromising it is still host root — but the only way in is two typed RPCs. + - Separately, because the writer accepts schema-valid content at the next `seq`, L5 bounds the *shape* of state writes, not their truth. - **Egress is only frozen, not confined.** L2 stops runtime repointing of `dest`/`s3_endpoint`; it cannot stop a redeploy by whoever controls ansible. L6 egress pinning narrows that to trusted operators. - **SigV4 against attacker endpoints.** With the preserved S3 keys and a repointed endpoint, the minio client signs requests to the attacker; secrets are not disclosed, but data and the access key (inside the signature) are. Frozen fields close the runtime path. - **A stolen admin token still holds every non-frozen `Set` field** (poll knobs, buffer sizes — DoS-adjacent). Rotation + audit (L6) is the answer there. @@ -423,6 +474,12 @@ Layer 0 (§5.7) removes primitives 1–2 at the binary level — the production - **`Set` policy test:** apply a config that changes `dest` scheme, adds `capture_path`, or rewrites `baseline_file`; assert rejection before `reconfigureFunc`. Golden-file the frozen-field set. - **Landlock test (Linux, gated):** start the daemon with a temp state dir; assert `os.WriteFile` outside allowed roots fails with `EACCES`; assert the same after a synthetic soft-restart exec. - **Writer tests:** no request field can influence the target path (assert by structurally — the proto has no path field); size cap, rate limit, version-replay rejection; atomicity (kill -9 mid-write leaves old baseline intact). +- **Broker tests:** Phase 5's P–U tables: + - fd-semantics proofs (a capability-less holder of a broker-opened fd cannot `SOCK_DESTROY`, change routes or read MD5 keys); + - both-ends validation; + - seccomp denials; + - the fixed Docker request set; + - the collector RCE-probe scenario. - **microVM integration:** the split deployment — collector `:ro` state, writer chrooted, `Set` a valid runtime change (poll frequency) and confirm no regression; `Set` with path fields rejected end-to-end. ## 9. Rollout sequencing @@ -433,10 +490,12 @@ Layer 0 (§5.7) removes primitives 1–2 at the binary level — the production | 2 | L4 Landlock; L6 quota + `Set` audit logging | low-medium; verify fleet kernels ≥5.13 | Ubuntu 24.04 fleet is fine; feature-detect ABI and degrade to warn if absent | | 3 | L5 writer split; collector state dir `:ro`; ansible second container + systemd units | medium; new process + socket plumbing | The big win: collector has zero write rules; degrade-not-exit on writer outage | | 4 | L6 egress pinning; v1.x role revision retirement | low | Closes the exfil class at the network layer | +| 5 | L7 netlink + Docker brokers; collector de-privilege; syscall profiles (the profiles ship with phase 2) | medium; two new processes, the collector loses its capabilities | The collector ends unprivileged ([Phase 5](phase-5-privilege-brokers.md)); the phase 4 egress rule then holds against a collector RCE | ## 10. References - Code: `pkg/xtcp/grpc_configService.go` (Set, validation), `pkg/xtcp/destinations_file.go`, `pkg/xtcp/netlinker.go`, `pkg/linkmonitor/config.go`, `pkg/linkmonitor/internal/baseline/`, `pkg/ipasn/ipasn.go`, `pkg/listener/listener.go`, `pkg/listenerauth/policy.go`, `pkg/xtcp/input_validation.go`, `cmd/xtcp2/xtcp2.go` (soft restart). - Docs: [security.md](../security.md) (auth off by default, wildcard listeners), [grpc-api.md](../grpc-api.md) (Set soft-restart), [output-and-destinations.md](../output-and-destinations.md) (`file:` destination), [observability.md](../observability.md) (startup capability check). - Deployment: the deployment repo → `ansible/roles/monitoring/xtcp2/` (defaults, tasks, firewall), the image repo (image build, settings docs). +- Kernel (L7): `net/netlink/af_netlink.c` (`__netlink_ns_capable` — opener **and** sender), `net/ipv4/inet_diag.c` + `net/ipv4/tcp_diag.c` (what `CAP_NET_ADMIN` adds to dumps), `net/core/sock_diag.c` (`SOCK_DESTROY`). - Upstream: [Landlock](https://docs.kernel.org/userspace-api/landlock.html) (restrictions inherit across `execve`; not bypassable by privileged processes), [buf protovalidate](https://github.com/bufbuild/protovalidate) (RE2 — no lookahead), [go-landlock](https://github.com/landlock-lsm/go-landlock). From bc8713be7fab6644cbce11cc2149451ff434521f Mon Sep 17 00:00:00 2001 From: "randomizedcoder dave.seddon.ca@gmail.com" Date: Mon, 9 Nov 2026 12:46:25 -0800 Subject: [PATCH 5/7] docs(security-hardening): startup confinement check (PR-24, W13, I19) The daemon refuses to start (exit 78) without no-new-privs, a seccomp filter, a read-only root and a bounded capability set, and later Landlock. It checks confinement rather than container markers: those pass --privileged and fail a hardened systemd unit. A per-check -insecureAllowUnconfined / XTCP_INSECURE_ALLOW_UNCONFINED waiver is alerted on, and the NixOS module is hardened to pass without one. Phase 2's OD-2 changes from warn-and-continue to refuse unless waived. Co-Authored-By: Claude Opus 5.5 --- docs/README.md | 2 +- .../security-hardening/implementation-plan.md | 10 +- .../phase-1-code-hardening.md | 121 +++++++++++++++++- .../phase-2-kernel-confinement.md | 8 +- .../plan-configservice-set-hardening.md | 25 +++- ...ulnerability-configservice-set-fs-write.md | 7 +- 6 files changed, 152 insertions(+), 21 deletions(-) diff --git a/docs/README.md b/docs/README.md index 62dfc14d..864bbaf8 100644 --- a/docs/README.md +++ b/docs/README.md @@ -67,7 +67,7 @@ Current TCP/UDS, TTL/hop-limit, bearer/HMAC, and read/admin protections, with de ### [Security advisory: `ConfigService.Set` filesystem writes](security-hardening/security-vulnerability-configservice-set-fs-write.md) -`Set` re-executes the daemon with an attacker-chosen config and, absent path validation, becomes a root file-write primitive (`file:` destinations, `capture_path`, `baseline_file`, `ipmeta_cache_path`, unix listeners). The advisory maps every primitive, documents the container deployment and its permission model (runtime settings → Linux primitives, what `NET_ADMIN`+`SYS_ADMIN` really grant, what is restricted/filtered), checks the classic container-escape path idioms target-by-target, and lays out the layered fix: build-time elimination, protovalidate path rules, a runtime path policy with frozen fields, a universal validating interceptor, Landlock self-confinement, a chrooted state-writer process split, and privilege brokers that take the capabilities and `docker.sock` out of the collector. The [hardening plan](security-hardening/plan-configservice-set-hardening.md) tracks the workstreams; the [implementation plan](security-hardening/implementation-plan.md) sequences the phases and PRs; the phase drill-downs go deep: [phase 1](security-hardening/phase-1-code-hardening.md) (build tags, proto validators, path policy, interceptor, audit, CI gates), [phase 2](security-hardening/phase-2-kernel-confinement.md) (Landlock self-confinement, full test matrices), [phase 3](security-hardening/phase-3-writer-split.md) (the writer-process split with its path-free proto protocol and the complete write inventory), [phase 4](security-hardening/phase-4-egress-fleet-lockdown.md) (egress pinning, escape-hatch removal, v1.x retirement, standing CI gates), and [phase 5](security-hardening/phase-5-privilege-brokers.md) (netlink and Docker brokers with typed, validated APIs; per-process seccomp and AppArmor profiles). +`Set` re-executes the daemon with an attacker-chosen config and, absent path validation, becomes a root file-write primitive (`file:` destinations, `capture_path`, `baseline_file`, `ipmeta_cache_path`, unix listeners). The advisory maps every primitive, documents the container deployment and its permission model (runtime settings → Linux primitives, what `NET_ADMIN`+`SYS_ADMIN` really grant, what is restricted/filtered), checks the classic container-escape path idioms target-by-target, and lays out the layered fix: build-time elimination, protovalidate path rules, a runtime path policy with frozen fields, a universal validating interceptor, Landlock self-confinement, a chrooted state-writer process split, and privilege brokers that take the capabilities and `docker.sock` out of the collector. The [hardening plan](security-hardening/plan-configservice-set-hardening.md) tracks the workstreams; the [implementation plan](security-hardening/implementation-plan.md) sequences the phases and PRs; the phase drill-downs go deep: [phase 1](security-hardening/phase-1-code-hardening.md) (build tags, proto validators, path policy, interceptor, audit, CI gates, a startup confinement check), [phase 2](security-hardening/phase-2-kernel-confinement.md) (Landlock self-confinement, full test matrices), [phase 3](security-hardening/phase-3-writer-split.md) (the writer-process split with its path-free proto protocol and the complete write inventory), [phase 4](security-hardening/phase-4-egress-fleet-lockdown.md) (egress pinning, escape-hatch removal, v1.x retirement, standing CI gates), and [phase 5](security-hardening/phase-5-privilege-brokers.md) (netlink and Docker brokers with typed, validated APIs; per-process seccomp and AppArmor profiles). ### [Observability](observability.md) Prometheus metrics, Go `pprof` endpoints, optional Pyroscope continuous profiling, and the startup capability check that explains exactly which Linux capabilities are required. diff --git a/docs/security-hardening/implementation-plan.md b/docs/security-hardening/implementation-plan.md index e00628da..19acb76e 100644 --- a/docs/security-hardening/implementation-plan.md +++ b/docs/security-hardening/implementation-plan.md @@ -4,7 +4,7 @@ |---|---| | **Status** | Active — checkboxes track execution; supersedes the sequencing section of the [hardening plan](plan-configservice-set-hardening.md) | | **Repos** | `randomizedcoder/xtcp2` (code), the image repo (image), the deployment repo (deploy), the infra repo (SSM/IAM/network) | -| **Docs** | [Advisory](security-vulnerability-configservice-set-fs-write.md) — the problem (primitives, permission model, escape idioms). [Hardening plan](plan-configservice-set-hardening.md) — workstreams W0–W12 with design detail, estimates, and open decisions. **This doc** — execution order, PR-level steps, phase gates, and rollback. | +| **Docs** | [Advisory](security-vulnerability-configservice-set-fs-write.md) — the problem (primitives, permission model, escape idioms). [Hardening plan](plan-configservice-set-hardening.md) — workstreams W0–W13 with design detail, estimates, and open decisions. **This doc** — execution order, PR-level steps, phase gates, and rollback. | Reading order for an implementer: skim the advisory's §2 (primitives) and §4 (permission model), then execute the phases below, dipping into the workstream sections of the hardening plan as each PR needs detail. @@ -52,11 +52,12 @@ PRs (repo `xtcp2` unless noted; each lands with its tests and doc updates from W - [ ] **PR-4 — universal validating interceptor** ([W3](plan-configservice-set-hardening.md#w3--universal-validating-interceptor-l3)): unary interceptor + stream `RecvMsg` wrapper; descriptor-walking guard test so no future RPC skips validation. - [ ] **PR-5 — `Set` audit logging** ([W6](plan-configservice-set-hardening.md#w6--set-audit-logging)): field-diff log lines (never secret values), token id, `xtcp2_set_applied_total{applied,frozen_rejected}` metric. - [ ] **PR-6 — regression gates** ([W8](plan-configservice-set-hardening.md#w8--tests-ci-and-fleet-config-gates)): end-to-end advisory regression (all five primitives blocked at their layers); fleet golden-config CI gate (ansible-rendered configs from all DCs validated on every proto change). +- [ ] **PR-24 — startup confinement check** ([W13](plan-configservice-set-hardening.md#w13--startup-confinement-check-l6)): refuse to start (exit 78) without no-new-privs, a seccomp filter, a read-only root and a bounded capability set; per-check `-insecureAllowUnconfined` / `XTCP_INSECURE_ALLOW_UNCONFINED` waiver with `xtcp2_confinement_waived` gauge; hardened NixOS module passes with no waiver. Numbered after Phase 5 to keep existing PR references stable. - [ ] **PR-7 — image + role** (the image repo, the deployment repo): build and publish the new production flavor; pin bump; fleet-gate test asserts the flavor rejects `dest file:` and `write_files`. **Release gate:** full unit/race/microVM matrix green; golden-config gate green; 24 h canary; dashboard shows no unexpected `xtcp2_set_applied_total` rejections from legitimate automation. -**Exit criteria:** fleet image where `Set` cannot reach any write primitive (L0 kills #1–2 in the binary; L1–L3 reject everything else; frozen fields stop `dest`/`s3_endpoint` repointing). +**Exit criteria:** fleet image where `Set` cannot reach any write primitive (L0 kills #1–2 in the binary; L1–L3 reject everything else; frozen fields stop `dest`/`s3_endpoint` repointing), and a daemon that refuses to start unconfined unless explicitly waived. **Effort:** ~1.5–2 weeks engineering + release. @@ -152,13 +153,14 @@ PRs (repo `xtcp2` unless noted; each lands with its tests and doc updates from W | I16 `CAP_NET_ADMIN` in the collector | Phase 5 (PR-20, PR-22) | L7 | | I17 `CAP_SYS_ADMIN` under the default seccomp profile | Phase 5 (PR-19, PR-22) | L7 | | I18 full inspect documents in the collector | Phase 5 (PR-21) | L7 | +| I19 daemon starts unconfined without complaint | Phase 1 (PR-24) | L6 | ## Timeline and effort | Phase | Elapsed (from start) | Content | |---|---|---| | 0 | days 0–2 | auth converge | -| 1 | weeks 1–3 | PR-1…PR-7, release vNext + fleet converge | +| 1 | weeks 1–3 | PR-1…PR-7 and PR-24, release vNext + fleet converge | | 2 | weeks 3–4 | Landlock (+ optional pivot_root), quota/alert, audit ship | | 3 | weeks 4–7 | writer split, `:ro` flip, systemd units | | 4 | weeks 7–8 | egress pinning, v1.x retirement, standing gates | @@ -173,7 +175,7 @@ Phases 1 and 2 partially overlap (Landlock PR-8 can start as soon as PR-3's `-wr | Phase | Rollback | |---|---| -| 1 | previous image tag + role pin (validations simply stop rejecting) | +| 1 | previous image tag + role pin (validations simply stop rejecting); for PR-24 alone, waive the failing check rather than roll back | | 2 | previous image tag (Landlock ruleset gone); quota/alert are additive | | 3 | `xtcp2_revert.yml` + restore the collector's `:rw` state mount (writer container can stay, unused) | | 4 | network policy rules (per-DC, reversible); `-setAllowFrozenFields` restoration if a genuine migration is blocked | diff --git a/docs/security-hardening/phase-1-code-hardening.md b/docs/security-hardening/phase-1-code-hardening.md index 6107623b..a6b3a5e1 100644 --- a/docs/security-hardening/phase-1-code-hardening.md +++ b/docs/security-hardening/phase-1-code-hardening.md @@ -3,9 +3,9 @@ | | | |---|---| | **Status** | Not started — checkboxes track execution | -| **Tracks** | [Implementation plan](implementation-plan.md) Phase 1 → [hardening plan](plan-configservice-set-hardening.md) workstreams W0–W3, W6, W8 → [advisory](security-vulnerability-configservice-set-fs-write.md) layers L0–L3 | -| **Goal** | After this phase, no `Set` input — however crafted — can express a filesystem write, a path change, or an egress repoint, and the debug writers are absent from the production binary | -| **Repos** | `randomizedcoder/xtcp2` (PR-1…PR-6), the image repo + the deployment repo (PR-7) | +| **Tracks** | [Implementation plan](implementation-plan.md) Phase 1 → [hardening plan](plan-configservice-set-hardening.md) workstreams W0–W3, W6, W8, W13 → [advisory](security-vulnerability-configservice-set-fs-write.md) layers L0–L3, L6 | +| **Goal** | After this phase, no `Set` input — however crafted — can express a filesystem write, a path change, or an egress repoint, the debug writers are absent from the production binary, and the daemon refuses to start unconfined | +| **Repos** | `randomizedcoder/xtcp2` (PR-1…PR-6, PR-24), the image repo + the deployment repo (PR-7) | ## Contents @@ -18,13 +18,14 @@ - [PR-5 — `Set` audit logging](#pr-5--set-audit-logging) - [PR-6 — regression and CI gates](#pr-6--regression-and-ci-gates) - [PR-7 — image and role](#pr-7--image-and-role) +- [PR-24 — startup confinement check](#pr-24--startup-confinement-check) - [Decisions to close during Phase 1](#decisions-to-close-during-phase-1) - [Release checklist](#release-checklist) - [Definition of done](#definition-of-done) ## Scope -Phase 1 is pure code and build changes — no new processes, no kernel confinement (Phase 2), no deployment-topology changes (Phase 3). It closes issues I1–I8 and I14 at the validation and binary level: +Phase 1 is pure code and build changes — no new processes, no kernel confinement (Phase 2), no deployment-topology changes (Phase 3). It closes issues I1–I8, I14 and I19 at the validation and binary level: | Layer | Effect after Phase 1 | |---|---| @@ -32,6 +33,7 @@ Phase 1 is pure code and build changes — no new processes, no kernel confineme | L1 (PR-2) | every path-bearing config field has a shape pattern; hostile paths are rejected by protovalidate before any Go code runs | | L2 (PR-3) | a Go path policy confines every path to operator-approved roots and freezes egress/listener/path fields against runtime changes | | L3 (PR-4) | every RPC input is validated by an interceptor — per-handler discipline becomes structural | +| L6 (PR-24) | the daemon refuses to start without the confinement the High rating assumes (no-new-privs, a seccomp filter, a read-only root, a bounded capability set), unless the operator waives named checks | ## PR map, branches, and parallelization @@ -39,11 +41,13 @@ Phase 1 is pure code and build changes — no new processes, no kernel confineme PR-1 sec/w0-build-tags ─────┐ PR-2 sec/w1-proto-paths ────┤ PR-3 sec/w2-path-policy ───┼──▶ PR-6 sec/w8-regression ──▶ PR-7 (image, role) -PR-4 sec/w3-interceptor ────┘ ▲ +PR-4 sec/w3-interceptor ────┤ ▲ +PR-24 sec/w13-confinement ──┘ PR-5 sec/w6-set-audit (after PR-3: reuses its diff helper) ``` -- PR-1, PR-2, PR-3, PR-4 are independent — four parallel tracks. PR-1 and PR-3 both touch the `Set` handler's validation block (`pkg/xtcp/grpc_configService.go` ~lines 186–262); land PR-1 first there to minimize rebase churn. +- PR-24 is numbered after Phase 5's PRs so existing PR references stay stable; it ships in Phase 1 because it has no dependencies. +- PR-1, PR-2, PR-3, PR-4, PR-24 are independent — five parallel tracks. PR-1 and PR-3 both touch the `Set` handler's validation block (`pkg/xtcp/grpc_configService.go` ~lines 186–262); land PR-1 first there to minimize rebase churn. - PR-5 after PR-3 (it logs PR-3's frozen-field diff). - PR-6 integrates; PR-7 is the release (repos: the image repo, the deployment repo). @@ -238,6 +242,106 @@ func validatingUnaryInterceptor(ctx context.Context, req any, --- +## PR-24 — startup confinement check + +**Objective:** the advisory rates the issue High in a confined container and Critical on a bare host ([severity](security-vulnerability-configservice-set-fs-write.md)). The daemon should refuse the Critical shape by default instead of running in it silently. + +**Why check confinement, not "am I in a container":** +- Container markers (`/.dockerenv`, `/run/.containerenv`, cgroup paths, the `container=` env var) are heuristics, and they measure the wrong thing. +- A `--privileged` container passes them with no confinement at all. A hardened systemd unit fails them while being properly confined. +- The in-tree NixOS module (`nix/modules/xtcp2-service.nix`) runs on bare metal. A container check would force it onto the "not recommended" override. + +**Checks.** Each check has a stable name, used in logs, metrics and the override. + +| Check | Tier | Passes when | Read from | +|---|---|---|---| +| `no_new_privs` | required | `NoNewPrivs: 1` | `/proc/self/status` | +| `seccomp` | required | `Seccomp: 2` (a filter is loaded) | `/proc/self/status` | +| `readonly_root` | required | `/` is mounted read-only | `statfs("/")` `ST_RDONLY` | +| `capability_bound` | required | `CapBnd` ⊆ the `requiredCaps` set in `pkg/xtcp/init_capabilities.go` (`NET_ADMIN`, `SYS_ADMIN`, `SYS_PTRACE`, `NET_RAW`, `SYS_RESOURCE`) | `/proc/self/status` | +| `landlock` (from PR-8) | required | the Phase 2 ruleset applied | `pkg/sandbox` result, checked right after `Constrain` | +| `lsm` | warn | AppArmor or SELinux confinement is not `unconfined` | `/proc/self/attr/apparmor/current`, else `/proc/self/attr/current` | +| `pid_namespace` | warn | not the initial pid namespace | `/proc/self/ns/pid` inode ≠ `0xEFFFFFFC` (`PROC_PID_INIT_INO`) | + +- **Why `lsm` and `pid_namespace` only warn:** + - Many NixOS and other bare-metal hosts run no AppArmor or SELinux. + - Bare-metal namespace discovery (Method B) scans other processes in `/proc`, so it needs the host pid namespace. + - OD-13 revisits both once Phase 5's netlink broker owns discovery. +- **The fleet container passes every check today** (advisory [§4.2](security-vulnerability-configservice-set-fs-write.md#42-runtime-settings--linux-primitives)): `no-new-privileges:true`, Docker's default seccomp, a read-only rootfs, `cap_drop: ALL` + `NET_ADMIN`/`SYS_ADMIN`, `docker-default`, and a private pidns. + +**Behavior** + +- **Where it runs:** in daemon mode only, in `main()` right after flags and env are parsed, before any listener, netlink socket or Landlock call. `-healthcheck`, `-validateConfig`, `-v` and `-help` skip it. +- **When a required check fails:** + - it exits with status 78 (`EX_CONFIG`); + - the message names each failed check, what it protects against, and the fix for Docker, Kubernetes and systemd, then the override as a last resort. +- **When a warn check fails:** one WARN line per check at startup; the daemon starts. +- **Override:** `-insecureAllowUnconfined=[,…]|all`, or the `XTCP_INSECURE_ALLOW_UNCONFINED` env var, which overrides the flag like every other `XTCP_*` variable. + - Waivers are per check, so a host that can't provide one property doesn't lose the others. + - An unknown check name is fatal, so a typo never waives nothing silently. + - A waived failure logs a WARN at startup and again every hour. +- **Metrics:** + - `xtcp2_confinement_check{check}`: 1 pass, 0 fail. + - `xtcp2_confinement_waived{check}`: 1 when a failing required check is waived. The fleet alerts on any non-zero value. +- **`Set` cannot change it:** the override is argv/env only and has no config proto field, so a `Set` payload cannot carry it. The soft restart re-execs with the same argv and env, so the check re-runs with the same outcome. +- **Phase 2:** the `landlock` check replaces OD-2's warn-and-continue. A kernel without Landlock, or a seccomp-blocked `landlock_*` syscall, exits unless `landlock` is waived. Phase 4's Landlock network rules (ABI 4) stay best-effort. + +**What passes, per runtime** + +| Runtime | Settings | +|---|---| +| Docker / Compose | `--security-opt no-new-privileges`, default seccomp (not `unconfined`), `--read-only`, `--cap-drop ALL` + only the needed `--cap-add` | +| Kubernetes | `allowPrivilegeEscalation: false`, `seccompProfile: RuntimeDefault`, `readOnlyRootFilesystem: true`, `capabilities.drop: [ALL]` + only the needed `add` | +| systemd | `NoNewPrivileges=yes`, a `SystemCallFilter=` allowlist, `ProtectSystem=strict` (+ `ReadWritePaths=` for the state and runtime dirs), `CapabilityBoundingSet=`, `RestartPreventExitStatus=78` so a refusal does not restart-loop | + +**Files** + +- New `pkg/xtcp/init_confinement.go`, beside `init_capabilities.go`, with the same test seams: `readProcStatus`, `statfsRoot`, `readLSMAttr` and `statPidNS` are package variables that tests swap, like `capgetFunc`. +- `cmd/xtcp2/xtcp2.go`: the flag, env override and call site. +- `nix/modules/xtcp2-service.nix`: + - add `NoNewPrivileges`, `SystemCallFilter` (validated by the VM suite; W12's bare-metal allowlist replaces it), `ProtectSystem=strict` and `RestartPreventExitStatus=78`; + - this closes the module's existing `ProtectSystem` TODO, and the module then passes with no override. +- Any test that execs the binary outside the module sets `XTCP_INSECURE_ALLOW_UNCONFINED` explicitly, naming only the checks it fails. +- Docs: [security.md](../security.md) (checks and per-runtime settings), [operations.md](../operations.md) (exit 78 and the override), [metrics.md](../metrics.md) (both gauges). + +**Tests** (`TestConfinementCheck`, table-driven over status, statfs, LSM and pidns fixtures) + +| ID | Cat | Description | Input / setup | Expected outcome | +|---|---|---|---|---| +| X1 | P | fleet container | `NoNewPrivs 1`, `Seccomp 2`, read-only `/`, `CapBnd` = `NET_ADMIN`+`SYS_ADMIN`, `docker-default (enforce)`, private pidns | starts; every `xtcp2_confinement_check` 1; no WARN | +| X2 | P | hardened systemd unit on bare metal | required checks pass, `CapBnd` = the five allowed caps, LSM `unconfined`, initial pidns | starts; WARN for `lsm` and `pid_namespace` only; no waiver | +| X3 | N | `sudo xtcp2` on a bare host | `NoNewPrivs 0`, `Seccomp 0`, read-write `/`, full `CapBnd` | exit 78; message names all four required checks with fixes | +| X4 | N | `docker run --privileged` | `Seccomp 0`, full `CapBnd` | exit 78 naming `seccomp` and `capability_bound` | +| X5 | N | Kubernetes pod with defaults | `Seccomp 0`, `NoNewPrivs 0` | exit 78 naming `seccomp` and `no_new_privs` | +| X6 | N | one required check failing (one row per check) | X1 with that one property flipped | exit 78 naming only that check | +| X7 | P | waive all on a bare host | X3 + `XTCP_INSECURE_ALLOW_UNCONFINED=all` | starts; WARN per failed check; `xtcp2_confinement_waived` 1 for each | +| X8 | P | waive exactly the failing check | X6 (`readonly_root`) + `-insecureAllowUnconfined=readonly_root` | starts; one waiver | +| X9 | N | waiver names a different check | X6 (`seccomp`) + `-insecureAllowUnconfined=readonly_root` | exit 78 naming `seccomp` | +| X10 | N | misspelled check name | `-insecureAllowUnconfined=seccom` | fatal; names the typo and lists the valid names | +| X11 | B | `CapBnd` exactly the allowed set | the five allowed caps | `capability_bound` passes | +| X12 | B | one cap beyond the allowed set | the five + `CAP_DAC_OVERRIDE` | `capability_bound` fails, naming `CAP_DAC_OVERRIDE` | +| X13 | B | empty `CapBnd` | all caps dropped | `capability_bound` passes; the existing capability check reports the missing fatal caps separately | +| X14 | C | `NoNewPrivs` or `Seccomp` line missing | status fixture without the line | that check fails as "unknown", no panic | +| X15 | C | `/proc` unreadable | `readProcStatus` returns an error | every `/proc`-based check fails closed; the message includes the read error | +| X16 | C | SELinux, no AppArmor | AppArmor attr absent; `/proc/self/attr/current` = `system_u:system_r:container_t:s0` | `lsm` passes; `unconfined_t` → WARN | +| X17 | C | soft restart with a waiver | `Set` re-exec, env carries the waiver | check re-runs, same outcome; a `Set` payload has no field to carry a waiver | +| X18 | C | non-daemon modes on an unconfined CI runner | `-healthcheck`, `-validateConfig` | run normally; no check | +| X19 | N | Landlock not applied (after PR-8) | ABI 0, or `EPERM` from `landlock_create_ruleset` | exit 78 naming `landlock` unless it is waived | + +**Integration:** the NixOS VM tests boot the hardened module with no override and pass. One VM test turns the module's hardening off, asserts exit 78 with no restart loop, then sets the waiver and asserts the daemon starts with `xtcp2_confinement_waived` 1. + +**Acceptance:** +- the fleet image starts unchanged on a canary; +- `sudo xtcp2` on a bare host exits 78 with an actionable message; +- the NixOS module starts with no override. + +**Risks:** +- Hand-run debug sessions now need a waiver. The message prints the exact flag, so this is friction, not breakage. +- `Seccomp: 2` proves a filter is loaded, not that it is a good one. W12's fleet-gate asserts the profile itself. +- This is a guard against misdeployment, not a security boundary: code already running in the daemon is past it. + +--- + ## Decisions to close during Phase 1 | # | Decision | Recommendation | @@ -246,10 +350,12 @@ func validatingUnaryInterceptor(ctx context.Context, req any, | [OD-9](plan-configservice-set-hardening.md#open-decisions) | drop network stdlib destinations from the production flavor? | yes — production needs `s3parquet` only; blocks the DEST-repoint exfil class structurally | | new | flag name for the config-validation mode (`-validateConfig`) | keep it — mirrors `-healthcheck`, usable by CI and ansible | | new | freeze `marshal_to`? | no — runtime format changes are legitimate ops; PR-5's audit line covers it | +| [OD-13](plan-configservice-set-hardening.md#open-decisions) | make `lsm` and `pid_namespace` required? | not yet — revisit when Phase 5's netlink broker owns namespace discovery | ## Release checklist -- [ ] All six code PRs merged; `make regen-protos` output committed (`gen/` in sync) +- [ ] All seven code PRs (PR-1…PR-6, PR-24) merged; `make regen-protos` output committed (`gen/` in sync) +- [ ] Fleet image starts with no confinement waiver; NixOS VM suite green with the hardened module - [ ] Full matrix green: unit + race + per-flavor (`dest_file`/`capture` worlds both tested) + microVM suite - [ ] Golden-config gate green for both ansible role revisions - [ ] Advisory regression test green (five primitives blocked) @@ -271,3 +377,4 @@ Concretely, with the admin token on a canary host running the new image: - [ ] `Set` changing `s3_endpoint`/`dest` scheme/listeners → `FailedPrecondition` (frozen) - [ ] `Set` changing only poll knobs applies; soft restart works; one audit line logged - [ ] Every RPC rejects a crafted-invalid message with `InvalidArgument` (interceptor) +- [ ] The release binary run as root on a bare host exits 78 naming the failed checks; with `-insecureAllowUnconfined=all` it starts and `xtcp2_confinement_waived` is non-zero diff --git a/docs/security-hardening/phase-2-kernel-confinement.md b/docs/security-hardening/phase-2-kernel-confinement.md index 5c07b49d..5e5050ec 100644 --- a/docs/security-hardening/phase-2-kernel-confinement.md +++ b/docs/security-hardening/phase-2-kernel-confinement.md @@ -43,7 +43,7 @@ PR-9 sec/w4-pivot-root (optional, independent, after PR-8) ## PR-8 — Landlock self-confinement -**Objective:** a Landlock ruleset applied once at first start, inherited (not re-applied) across soft restarts, and not removable by the process itself; degraded gracefully (warn + gauge, never a failed daemon) where the kernel lacks support. +**Objective:** a Landlock ruleset applied once at first start, inherited (not re-applied) across soft restarts, and not removable by the process itself; where the kernel lacks support, the daemon refuses to start unless the `landlock` check is waived ([PR-24](phase-1-code-hardening.md#pr-24--startup-confinement-check)). ### Files @@ -165,8 +165,8 @@ Three layers: **A/E/D** are pure table-driven unit tests (no kernel, all platfor | D1 | P | ABI 4 (Ubuntu 24.04 / 6.8) | probed version 4 | ruleset incl. `REFER`+`TRUNCATE`, no `RESOLVE_UNIX` (pathname-UDS connect unmediated, logged); `xtcp2_sandbox_active 1`, `xtcp2_sandbox_abi 4` | | D2 | B | ABI 2 (5.19) | probed version 2 | `REFER` kept, `TRUNCATE` dropped; write roots allowed (same-dir atomic rename needs neither); `truncate(2)` outside roots is no longer denied — one warn line names the lost guarantee | | D3 | B | ABI 1 (5.13) | probed version 1 | `REFER` and `TRUNCATE` dropped; same-dir rename still permitted (v1 rights cover it); truncation unmediated as D2; active 1; warn line | -| D4 | N | ABI 0 / `ENOSYS` (unsupported kernel) | probe fails with not-supported | **warn** + `xtcp2_sandbox_active 0` + daemon continues (OD-2: degrade, never a failed daemon) | -| D5 | C | `EPERM` from a seccomp-blocked `landlock_*` syscall (possible on old Docker default profiles) | probe returns EPERM | same as D4: warn + gauge 0 + continue, message names the likely cause | +| D4 | N | ABI 0 / `ENOSYS` (unsupported kernel) | probe fails with not-supported | exit 78 naming `landlock` (PR-24 X19); with `landlock` waived: **warn** + `xtcp2_sandbox_active 0` + daemon continues | +| D5 | C | `EPERM` from a seccomp-blocked `landlock_*` syscall (possible on old Docker default profiles) | probe returns EPERM | same as D4 (exit 78 unless waived), message names the likely cause | | D7 | P | ABI ≥ 9 | probed version 9 | `RESOLVE_UNIX` handled; granted on the docker.sock and lldpd socket only; other pathname sockets denied (B14) | | D6 | C | ABI ≥ 2 but UDS listener outside all roots | listener path not under a write root | `Constrain` returns a configuration error (loud at startup, not a runtime bind surprise) | @@ -236,7 +236,7 @@ Three layers: **A/E/D** are pure table-driven unit tests (no kernel, all platfor | # | Decision | Recommendation | |---|---|---| -| OD-2 (already defaulted) | Landlock-absent kernel behavior | warn + `xtcp2_sandbox_active 0` + continue (D4/D5) | +| OD-2 (revised by PR-24) | Landlock-absent kernel behavior | refuse to start unless `landlock` is waived; waived: warn + `xtcp2_sandbox_active 0` + continue (D4/D5) | | new | dependency: go-landlock vs raw x/sys | go-landlock | | new | ABI floor for write roots | v1 (same-dir atomic rename needs no `REFER`/`TRUNCATE`); ABI 1–2 lose truncate protection and say so; v0 disables writes with a warning | | new | Landlock network rules (TCP bind/connect scoping) | out of scope this phase; revisit after Phase 4 egress pinning | diff --git a/docs/security-hardening/plan-configservice-set-hardening.md b/docs/security-hardening/plan-configservice-set-hardening.md index f4d46f27..1bfd75ae 100644 --- a/docs/security-hardening/plan-configservice-set-hardening.md +++ b/docs/security-hardening/plan-configservice-set-hardening.md @@ -25,6 +25,7 @@ - [W10 — netlink broker and collector de-privilege (L7)](#w10--netlink-broker-and-collector-de-privilege-l7) - [W11 — Docker broker (L7)](#w11--docker-broker-l7) - [W12 — syscall profiles (L7)](#w12--syscall-profiles-l7) + - [W13 — startup confinement check (L6)](#w13--startup-confinement-check-l6) - [Sequencing and dependencies](#sequencing-and-dependencies) - [Fleet rollout runbook](#fleet-rollout-runbook) - [End-state verification checklist](#end-state-verification-checklist) @@ -41,7 +42,8 @@ 4. Runtime config changes cannot repoint egress (`dest`, `s3_endpoint`) or listeners/auth without a redeploy. 5. Every applied `Set` is auditable. 6. The collector process ends up with **zero** filesystem write capability. -7. The collector holds no capabilities, no `docker.sock` and no namespace handles. The privileged reads it needs go through two brokers, each enforced by validators on both ends, broker-side filtering, and a syscall profile. +7. The daemon refuses to start without the confinement its severity rating assumes, unless the operator waives named checks. +8. The collector holds no capabilities, no `docker.sock` and no namespace handles. The privileged reads it needs go through two brokers, each enforced by validators on both ends, broker-side filtering, and a syscall profile. **Non-goals** (tracked in the advisory's [honest limits](security-vulnerability-configservice-set-fs-write.md#7-honest-limits--what-this-does-not-fix)) @@ -71,6 +73,7 @@ | I16 | `CAP_NET_ADMIN` in the host netns: host firewall/route rewrite (defeats W7 egress pinning), `SOCK_DESTROY` of any socket, TCP-MD5 keys in inet_diag dumps | `pkg/xtcp/init_capabilities.go`, deployment role | High (post-RCE) | W10 | | I17 | `CAP_SYS_ADMIN` under Docker's default seccomp profile: `bpf`, `perf_event_open`, `unshare`, `setns` of any type; `mount` held back only by AppArmor | deployment role | High (post-RCE) | W10, W12 | | I18 | The collector receives full container inspect documents (tenant `Env`, mounts, labels) though it uses four fields | `pkg/dockermeta` | Medium (confidentiality) | W11 | +| I19 | The daemon starts silently with no confinement (bare host as root, `--privileged`, a pod without seccomp): the advisory's Critical shape | `cmd/xtcp2/xtcp2.go`, `nix/modules/xtcp2-service.nix` | High (deployment) | W13 | ## Workstreams @@ -282,11 +285,25 @@ Advisory §5.8, [Phase 5](phase-5-privilege-brokers.md#syscall-profiles-for-ever **Acceptance**: R1–R10, T4–T6, U9. **Estimate**: 3–5 days. +### W13 — startup confinement check (L6) + +[Phase 1 PR-24](phase-1-code-hardening.md#pr-24--startup-confinement-check). No dependencies; ships in Phase 1. + +- [ ] Check confinement, not container markers. Required: `no_new_privs`, `seccomp` (filter mode), `readonly_root`, `capability_bound` (`CapBnd` within `requiredCaps`), and `landlock` once W4 lands. Warn only: `lsm`, `pid_namespace`. +- [ ] A failed required check exits 78 with per-runtime fixes (Docker, Kubernetes, systemd). +- [ ] Per-check waiver: `-insecureAllowUnconfined=|all` / `XTCP_INSECURE_ALLOW_UNCONFINED`. Unknown names are fatal; waivers warn hourly and export `xtcp2_confinement_waived{check}`. +- [ ] Argv/env only, with no config field, so `Set` cannot carry a waiver. +- [ ] Harden the NixOS module (`NoNewPrivileges`, `SystemCallFilter`, `ProtectSystem=strict`, `RestartPreventExitStatus=78`) so it passes with no waiver. +- [ ] Fleet alert on any non-zero `xtcp2_confinement_waived`. + +**Acceptance**: X1–X19. **Estimate**: 2–3 days. + ## Sequencing and dependencies ``` Phase 0 (today): the v2.0.x auth merge (I10) — ops action, no release needed Phase 1 (release vNext): W0 (image rebuild) + W1 ─▶ W2 ─▶ W3 (pure code/build; closes I1-I6, I8, I14) + W13 confinement check, independent (closes I19) └──▶ W8 (hostile-path + golden-config gates) ──▶ W9 Phase 2: W4 (Landlock, optional pivot_root) ──▶ W6 (Set audit) ──▶ W7 quota + audit ship Phase 3: W5 (writer split) ──▶ W7 (second container, :ro flip) ──▶ W9 @@ -305,7 +322,7 @@ Phase 5: W12 profiles (ship with Phase 2) ─▶ W10 netlink br 1. Merge Phase 1 → tag release → image repo bump → ansible role pin bump; fleet-gate test green. 2. Canary via the existing canary mechanism (`ansible/xtcp2.yml`); watch for `InvalidArgument`/`FailedPrecondition` spikes from legit automation using `Set` (expect none — operators use redeploy, but verify `xtcp2_set_applied_total`). 3. If a legit `Set` migration surfaces: document `-setAllowFrozenFields`, time-box it, remove after migration. -4. Phase 2 canary: watch `xtcp2_sandbox_active` (expect 1; 0 on any pre-5.13 kernel → raise a ticket, host is unprotected at L4). +4. Phase 2 canary: watch `xtcp2_sandbox_active` (expect 1). A pre-5.13 kernel now exits 78 at startup (W13 `landlock` check) unless `landlock` is waived; a waived host alerts on `xtcp2_confinement_waived`. 5. Phase 3 canary: writer-up check, baseline alert still fires on link-down (synthetic test), ASN refresh metric, then the `:ro` flip; revert path = `xtcp2_revert.yml` + restore `:rw` mount. 6. Post-rollout: run the [end-state verification checklist](#end-state-verification-checklist) against one canary host with the admin token. @@ -326,6 +343,7 @@ With admin token in hand on a canary host, every item must fail safely: - [ ] Collector (W10–W12): `CapEff 0` as uid `xtcp2`, no `docker.sock`. On a crippled build, an RCE probe cannot change the firewall or routes, load BPF, `setns`, mount, kill sockets, or reach the Docker API - [ ] Netlink broker: `setns` with any type other than `CLONE_NEWNET`, and any socket other than netlink `SOCK_DIAG`/`ROUTE` → `EPERM` (seccomp) - [ ] Docker broker: the only Docker requests ever issued are the four fixed GETs; responses carry no `Env`/mounts/labels +- [ ] Startup (W13): the release binary as root on a bare host exits 78 naming the failed checks; the fleet container and the hardened NixOS unit start with no waiver - [ ] Production image (W0): `Set` with `dest = "file:/tmp/pwn"` → fatal startup error, no file ever created (the code is absent from the binary); `write_files = 1` → fatal startup error ## Open decisions @@ -333,7 +351,7 @@ With admin token in hand on a canary host, every item must fail safely: | # | Decision | Default proposal | |---|---|---| | OD-1 | `unix_socket_mode` cap: 0600 or allow 0660 (shared-group socket)? | 0600 (`in: [0, 384]`); add 432 (0660) to the list only if a consumer needs group access — never a `lte` bound | -| OD-2 | Landlock-absent kernel behavior: warn-and-continue vs refuse to start | Warn + `xtcp2_sandbox_active 0` + alert; fleet is ≥6.8 so this is a bare-metal edge case | +| OD-2 | Landlock-absent kernel behavior: warn-and-continue vs refuse to start | Refuse to start (W13 `landlock` check) unless `landlock` is waived; a waived host runs with `xtcp2_sandbox_active 0` and alerts. Fleet is ≥6.8, so this is a bare-metal edge case | | OD-3 | `file:` destination: keep behind `-allowFileDest` or remove entirely in the s3parquet build flavor | Both: the `dest_file` build tag removes it from the production flavor (W0); flavors that compile it still require `-allowFileDest` (W2) | | OD-4 | Writer transport: UDS gRPC (chosen) vs loopback TCP | UDS — peer credentials, no port surface; loopback TCP rejected because every host process shares it | | OD-5 | Does the ipfeed sidecar adopt the writer? | No — out of scope; it is already uid 65534, single subdirectory, no caps | @@ -343,6 +361,7 @@ With admin token in hand on a canary host, every item must fail safely: | OD-10 | Netlink broker: pass fds vs run the dumps in the broker | Pass fds over `SOCK_SEQPACKET`. The kernel checks the sender's capabilities as well as the opener's, so the fd carries no privilege, and the hot path stays zero-copy | | OD-11 | Docker access: typed gRPC broker vs a filtering HTTP proxy in front of `docker.sock` | Typed broker — the projection and the fixed request set are then structural, not pattern-matched | | OD-12 | Bound the Docker broker further with dockerd TCP+mTLS and an authz plugin (GET-only for the broker's cert) | Deferred until W11 lands | +| OD-13 | Make the W13 `lsm` and `pid_namespace` checks required? | Not yet: many bare-metal hosts run no LSM, and bare-metal discovery scans host `/proc`. Revisit when W10's broker owns discovery | | OD-9 | Drop the network stdlib destinations (`tcp`/`udp`/`http`/`https`/`unix`/`unixgram`) from the production flavor? | Yes — production needs `s3parquet` only; kills the DEST-repoint exfil class structurally | ## References diff --git a/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md b/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md index 9999a38c..2251de83 100644 --- a/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md +++ b/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md @@ -3,7 +3,7 @@ | | | |---|---| | **Status** | Draft advisory — no CVE issued (internal assessment) | -| **Severity** | High in the reference container deployment (§4): the only writable host path is the state dir, so the impact is monitor-state tampering and disk-fill. Critical outside it: a bare-metal or hand-run daemon with the binary defaults (auth off, wildcard listener, root) gives any network peer a root file write at an arbitrary path | +| **Severity** | High in the reference container deployment (§4): the only writable host path is the state dir, so the impact is monitor-state tampering and disk-fill. Critical outside it: a bare-metal or hand-run daemon with the binary defaults (auth off, wildcard listener, root) gives any network peer a root file write at an arbitrary path. [PR-24](phase-1-code-hardening.md#pr-24--startup-confinement-check) makes the daemon refuse to start in that shape unless the operator waives named checks | | **Affected** | Every xtcp2 version with `ConfigService.Set` + `file:` destination / `capture_path` / `baseline_file` / `ipmeta_cache_path` / unix listeners — i.e. all current releases | | **Not affected** | Read-only clients (`Get`, `FlatRecords`, `PollFlatRecords`); hot-path setters (`SetPollFrequency`, `SetS3Upload`, `SetEnvelopeFlush`, `Trigger*` — none touch the filesystem) | | **Scope** | `ConfigService.Set` runtime reconfiguration; see [security.md](../security.md) for listener-auth posture and [grpc-api.md](../grpc-api.md) for the Set soft-restart design | @@ -382,6 +382,7 @@ Rules enforced by the writer, in code: - **Egress pinning.** Freeze `s3_endpoint`/`dest` (L2) and, ideally, pin the collector's egress to its known S3 endpoints with an nftables rule matched on its cgroup (it runs in the host netns; host-wide security groups or UFW would also restrict co-located workloads) so the frozen-field rule can't be sidestepped at deploy time either. - **Read-only where possible.** With L5 in place, the collector container mounts the entire host `:ro` — nothing writable remains. - **Rotate tokens** (`xtcp2_rotate_listener_tokens`) on any suspected exposure; the admin token is the keys to `Set`. +- **Refuse to start unconfined** ([PR-24](phase-1-code-hardening.md#pr-24--startup-confinement-check)). The severity above depends on the deployment shape, so the binary checks it. It exits 78 without no-new-privs, a seccomp filter, a read-only root, a bounded capability set and (after L4) Landlock. Container markers are not checked: `--privileged` passes them, a hardened systemd unit fails them. A per-check `-insecureAllowUnconfined` waiver exists and is alerted on. This guards against misdeployment; it is not a boundary against code already running in the daemon. ### 5.7 Layer 0 — build-time elimination of write-capable debug features @@ -449,6 +450,7 @@ L7 moves all three out of the collector. | RCE → host firewall/route rewrite, socket kill, TCP-MD5 key read (`NET_ADMIN`) | — | — | — | — | — | — | ✅ no process holds `NET_ADMIN` | | RCE → `bpf`/`perf_event_open`/namespace creation/mount (`SYS_ADMIN`) | — | — | — | — | — | — | ✅ collector: no capabilities; broker: seccomp admits `setns(CLONE_NEWNET)` only | | RCE → Docker API (start a host-root container) | — | — | — | — | — | — | ✅ typed broker: two read RPCs | +| Unconfined deployment (bare host as root, `--privileged`, pod without seccomp) | — | — | — | — | — | ✅ refuses to start unless waived | — | A compromised collector that speaks the Docker API bypasses L4 and L5: dockerd performs the write, outside the collector's Landlock domain and without the writer (§7). L7 removes that path. @@ -480,13 +482,14 @@ Layer 0 (§5.7) removes primitives 1–2 at the binary level — the production - seccomp denials; - the fixed Docker request set; - the collector RCE-probe scenario. +- **Confinement check:** Phase 1's X1–X19 table: fleet container and hardened systemd unit start; bare root, `--privileged` and a default pod exit 78; waivers are per check, typos fatal, and `Set` cannot carry one. - **microVM integration:** the split deployment — collector `:ro` state, writer chrooted, `Set` a valid runtime change (poll frequency) and confirm no regression; `Set` with path fields rejected end-to-end. ## 9. Rollout sequencing | Phase | Contents | Risk | Notes | |---|---|---|---| -| 1 — now | L0 (image rebuild) + L1 + L2 (+ L3 interceptor) | low; pure build/validation change, no behavior change for legit configs | Cuts a patch release; the image repo bump; ansible pin. Frozen-field list needs an operator escape hatch (`-setAllowFrozenFields`) for migrations | +| 1 — now | L0 (image rebuild) + L1 + L2 (+ L3 interceptor) + the L6 startup confinement check | low; pure build/validation change, no behavior change for legit configs | Cuts a patch release; the image repo bump; ansible pin. Frozen-field list needs an operator escape hatch (`-setAllowFrozenFields`) for migrations | | 2 | L4 Landlock; L6 quota + `Set` audit logging | low-medium; verify fleet kernels ≥5.13 | Ubuntu 24.04 fleet is fine; feature-detect ABI and degrade to warn if absent | | 3 | L5 writer split; collector state dir `:ro`; ansible second container + systemd units | medium; new process + socket plumbing | The big win: collector has zero write rules; degrade-not-exit on writer outage | | 4 | L6 egress pinning; v1.x role revision retirement | low | Closes the exfil class at the network layer | From 7ef6a155ddbecc7ee894bd8f2de96524fa296f4e Mon Sep 17 00:00:00 2001 From: "randomizedcoder dave.seddon.ca@gmail.com" Date: Mon, 9 Nov 2026 16:08:19 -0800 Subject: [PATCH 6/7] docs(security-hardening): PR-24 knock-ons - allowed capability set per mode (writer, both brokers, broker-mode collector) so one list cannot wave through a privileged writer - in-repo deployments that would fail the check: k8s DaemonSet (also drops the unused CAP_SYS_CHROOT), dev docker-compose, quickstart sudo examples, nix capability-check - startup-failure tests assert the message, not a non-zero exit (PR-1 acceptance, image smoke test), since exit 78 would also match - Phase 3 writer container and bare-metal units carry read_only, no-new-privileges and systemd hardening; Xtcp2ConfinementWaived alert - stale Landlock degrade-not-exit wording in W4 and the advisory - W9 docs/metrics; W7 gate noted as the check's pre-deploy twin Co-Authored-By: Claude Opus 5.5 --- .../security-hardening/implementation-plan.md | 2 +- .../phase-1-code-hardening.md | 32 ++++++++++++++++--- .../phase-3-writer-split.md | 5 +-- .../phase-5-privilege-brokers.md | 2 +- .../plan-configservice-set-hardening.md | 17 ++++++---- ...ulnerability-configservice-set-fs-write.md | 4 +-- 6 files changed, 45 insertions(+), 17 deletions(-) diff --git a/docs/security-hardening/implementation-plan.md b/docs/security-hardening/implementation-plan.md index 19acb76e..9b677a0f 100644 --- a/docs/security-hardening/implementation-plan.md +++ b/docs/security-hardening/implementation-plan.md @@ -87,7 +87,7 @@ PRs (repo `xtcp2` unless noted; each lands with its tests and doc updates from W - [ ] **PR-11 — writer** ([W5](plan-configservice-set-hardening.md#w5--state-writer-process-split-l5)): `proto/state_writer/v1` (`WriteBaseline`, `PublishIpmetaCache` — no path fields on the wire); `xtcp2 -writer` mode (UDS + `SO_PEERCRED`, fixed targets, size caps, rate limits, version numbers, temp+fsync+rename, `flock`); collector seams via the `internal/baseline` injection point and `ipasn.PublishLookupCache`; degrade-not-exit metrics; tests: no-path-parameter structural test, `kill -9` mid-write atomicity, version replay rejection. - [ ] **PR-12 — config-surface shrink**: `baseline_file` / `ipmeta_cache_path` become writer-owned constants — removed from the `Set`-mutable surface entirely (already in the frozen list; now gone from the runtime path). -- [ ] **PR-13 — deployment** (the deployment repo): writer container (state dir `:rw`, shared socket dir, no network, uid 65534); collector state dir flips `:ro`; systemd units (`RootDirectory=`, `cap_drop ALL`, `AF_UNIX`) for bare metal; canary via the fleet gate; revert via `xtcp2_revert.yml` + restoring the `:rw` mount. +- [ ] **PR-13 — deployment** (the deployment repo): writer container (state dir `:rw`, shared socket dir, no network, uid 65534); collector state dir flips `:ro`; systemd units (`RootDirectory=`, `cap_drop ALL`, `AF_UNIX`, plus `NoNewPrivileges`, `SystemCallFilter` and `ProtectSystem=strict` so PR-24 passes) for bare metal; writer container `read_only` + `no-new-privileges`; canary via the fleet gate; revert via `xtcp2_revert.yml` + restoring the `:rw` mount. **Exit criteria:** collector's Landlock ruleset contains **zero** write rules; `kill -9` the writer mid-baseline leaves the old baseline intact and the monitor degrades-not-exits; `Set` cannot name any path anywhere in the config. diff --git a/docs/security-hardening/phase-1-code-hardening.md b/docs/security-hardening/phase-1-code-hardening.md index a6b3a5e1..6e676296 100644 --- a/docs/security-hardening/phase-1-code-hardening.md +++ b/docs/security-hardening/phase-1-code-hardening.md @@ -73,7 +73,7 @@ PR-5 sec/w6-set-audit (after PR-3: reuses its diff helper) - Tagged build (`go test -tags 'dest_file capture' ./pkg/xtcp/`): `newFileDest` constructs and writes under a `t.TempDir()`; capture writer writes one file and decrements the budget. - `lookupDestinationFactory("file")` without the tag → `destLookupNotCompiledIn` (gate-pair test file, both worlds). -**Acceptance:** `nix build .#xtcp2-s3parquet` binary: startup with `dest file:…` or `write_files>0` exits non-zero with the rebuild hint; `Set` with either returns `FailedPrecondition` and the daemon keeps running. `.#xtcp2` (full) accepts both. Fleet golden configs (PR-6 gate) unaffected. +**Acceptance:** `nix build .#xtcp2-s3parquet` binary: startup with `dest file:…` or `write_files>0` exits non-zero with the rebuild hint — the test asserts the hint, not just the exit status, because PR-24's exit 78 would also satisfy "non-zero" (run it with `XTCP_INSECURE_ALLOW_UNCONFINED=all` outside a confined runtime); `Set` with either returns `FailedPrecondition` and the daemon keeps running. `.#xtcp2` (full) accepts both. Fleet golden configs (PR-6 gate) unaffected. **Risks:** a fleet config legitimately using `file:`/`write_files` (none known — fleet uses `s3parquet`; the golden-config gate proves it). Decision [OD-8](plan-configservice-set-hardening.md#open-decisions) (`xtcp2-min` keeps `file:`?) — recommend dropping it there too. @@ -236,7 +236,7 @@ func validatingUnaryInterceptor(ctx context.Context, req any, **Objective:** ship it. -- Image repo: bump the `xtcp2` flake input to the release commit; build `oci-xtcp2-s3parquet-enrich` (now without `dest_file`/`capture`); smoke test in CI: the image binary with `dest file:…` / `write_files=1` exits non-zero naming the build tag; publish the image tag. +- Image repo: bump the `xtcp2` flake input to the release commit; build `oci-xtcp2-s3parquet-enrich` (now without `dest_file`/`capture`); smoke test in CI: the image binary, run with the fleet container flags so PR-24 passes, with `dest file:…` / `write_files=1` exits non-zero and the test asserts the build-tag message (exit status alone would also match PR-24's exit 78); publish the image tag. - Deployment repo: pin `xtcp2_image` to the new tag; verify the fleet-gate test still passes (it asserts image↔role compatibility); the binary-level assertions live in `xtcp2` CI (PR-6), not in the role. - Rollout per the [fleet rollout runbook](plan-configservice-set-hardening.md#fleet-rollout-runbook): canary first, watch `xtcp2_set_applied_total` for legitimate-automation rejections (expect none — operators redeploy), then fleet converge. @@ -258,11 +258,23 @@ func validatingUnaryInterceptor(ctx context.Context, req any, | `no_new_privs` | required | `NoNewPrivs: 1` | `/proc/self/status` | | `seccomp` | required | `Seccomp: 2` (a filter is loaded) | `/proc/self/status` | | `readonly_root` | required | `/` is mounted read-only | `statfs("/")` `ST_RDONLY` | -| `capability_bound` | required | `CapBnd` ⊆ the `requiredCaps` set in `pkg/xtcp/init_capabilities.go` (`NET_ADMIN`, `SYS_ADMIN`, `SYS_PTRACE`, `NET_RAW`, `SYS_RESOURCE`) | `/proc/self/status` | +| `capability_bound` | required | `CapBnd` ⊆ the allowed set for the running mode (next table) | `/proc/self/status` | | `landlock` (from PR-8) | required | the Phase 2 ruleset applied | `pkg/sandbox` result, checked right after `Constrain` | | `lsm` | warn | AppArmor or SELinux confinement is not `unconfined` | `/proc/self/attr/apparmor/current`, else `/proc/self/attr/current` | | `pid_namespace` | warn | not the initial pid namespace | `/proc/self/ns/pid` inode ≠ `0xEFFFFFFC` (`PROC_PID_INIT_INO`) | +**Allowed capabilities per mode.** The writer and both brokers are the same binary, so one list would let a writer run with `SYS_ADMIN` unnoticed. + +| Mode | Allowed `CapBnd` | From | +|---|---|---| +| collector, direct mode | the `requiredCaps` set in `pkg/xtcp/init_capabilities.go`: `NET_ADMIN`, `SYS_ADMIN`, `SYS_PTRACE`, `NET_RAW`, `SYS_RESOURCE` | PR-24 | +| writer (`-writer`) | none | Phase 3 PR-11 | +| netlink broker (`-nlbroker`) | `SYS_ADMIN` | Phase 5 PR-20 | +| Docker broker (`-dockerbroker`) | none | Phase 5 PR-21 | +| collector, broker mode | none | Phase 5 PR-22 | + +Each later PR adds its row when it adds the mode. + - **Why `lsm` and `pid_namespace` only warn:** - Many NixOS and other bare-metal hosts run no AppArmor or SELinux. - Bare-metal namespace discovery (Method B) scans other processes in `/proc`, so it needs the host pid namespace. @@ -284,6 +296,7 @@ func validatingUnaryInterceptor(ctx context.Context, req any, - `xtcp2_confinement_check{check}`: 1 pass, 0 fail. - `xtcp2_confinement_waived{check}`: 1 when a failing required check is waived. The fleet alerts on any non-zero value. - **`Set` cannot change it:** the override is argv/env only and has no config proto field, so a `Set` payload cannot carry it. The soft restart re-execs with the same argv and env, so the check re-runs with the same outcome. +- **Relation to W7's fleet gate:** W7 asserts the container spec with `docker inspect` before deploy; this check is its runtime twin and catches anything that slips past (a hand-edited role, a different orchestrator, a bare host). - **Phase 2:** the `landlock` check replaces OD-2's warn-and-continue. A kernel without Landlock, or a seccomp-blocked `landlock_*` syscall, exits unless `landlock` is waived. Phase 4's Landlock network rules (ABI 4) stay best-effort. **What passes, per runtime** @@ -301,7 +314,13 @@ func validatingUnaryInterceptor(ctx context.Context, req any, - `nix/modules/xtcp2-service.nix`: - add `NoNewPrivileges`, `SystemCallFilter` (validated by the VM suite; W12's bare-metal allowlist replaces it), `ProtectSystem=strict` and `RestartPreventExitStatus=78`; - this closes the module's existing `ProtectSystem` TODO, and the module then passes with no override. -- Any test that execs the binary outside the module sets `XTCP_INSECURE_ALLOW_UNCONFINED` explicitly, naming only the checks it fails. +- Any test that execs the binary outside the module sets `XTCP_INSECURE_ALLOW_UNCONFINED` explicitly, naming only the checks it fails. Known case: `nix/checks/capability-check.nix` runs the bare binary without capabilities and pins the capability diagnostics. Without `all` it would hit exit 78 first and fail on the wrong message. +- `build/k8s/xtcp/xtcp-daemonset.yaml` fails every required check today: + - add `allowPrivilegeEscalation: false`, `seccompProfile: RuntimeDefault` and `readOnlyRootFilesystem: true`; + - change capabilities to `drop: [ALL]` + `add: [NET_ADMIN, SYS_ADMIN]`; + - remove `CAP_SYS_CHROOT`: it is outside the allowed set and unused (advisory §4.2). +- `build/containers/docker-compose.yml`: add `read_only: true`, `security_opt: [no-new-privileges:true]`, `cap_drop: [ALL]` + `cap_add: [NET_ADMIN, SYS_ADMIN]`. +- Quickstart examples (`sudo ./result/bin/xtcp2 …` in [output-and-destinations.md](../output-and-destinations.md) and the repo README): show the waiver explicitly, which also makes the risk visible. Offer a confined alternative: `sudo systemd-run --pty -p NoNewPrivileges=yes -p ProtectSystem=strict -p SystemCallFilter=@system-service -p CapabilityBoundingSet="CAP_NET_ADMIN CAP_SYS_ADMIN" ./result/bin/xtcp2 …`. - Docs: [security.md](../security.md) (checks and per-runtime settings), [operations.md](../operations.md) (exit 78 and the override), [metrics.md](../metrics.md) (both gauges). **Tests** (`TestConfinementCheck`, table-driven over status, statfs, LSM and pidns fixtures) @@ -327,6 +346,11 @@ func validatingUnaryInterceptor(ctx context.Context, req any, | X17 | C | soft restart with a waiver | `Set` re-exec, env carries the waiver | check re-runs, same outcome; a `Set` payload has no field to carry a waiver | | X18 | C | non-daemon modes on an unconfined CI runner | `-healthcheck`, `-validateConfig` | run normally; no check | | X19 | N | Landlock not applied (after PR-8) | ABI 0, or `EPERM` from `landlock_create_ruleset` | exit 78 naming `landlock` unless it is waived | +| X20 | N | writer with a capability (after PR-11) | `-writer`, `CapBnd` = `SYS_ADMIN` | exit 78 naming `capability_bound` and `CAP_SYS_ADMIN` | +| X21 | N | netlink broker with `NET_ADMIN` (after PR-20) | `-nlbroker`, `CapBnd` = `SYS_ADMIN`+`NET_ADMIN` | exit 78 naming `CAP_NET_ADMIN` | +| X22 | B | netlink broker with exactly `SYS_ADMIN` | `-nlbroker`, `CapBnd` = `SYS_ADMIN` | `capability_bound` passes | +| X23 | N | collector in broker mode with any capability (after PR-22) | broker flags set, `CapBnd` = `NET_ADMIN` | exit 78 naming `CAP_NET_ADMIN` | +| X24 | P | the capability nix check keeps its purpose | `capability-check.nix` with `XTCP_INSECURE_ALLOW_UNCONFINED=all` | non-zero exit with the existing capability diagnostics, not exit 78 | **Integration:** the NixOS VM tests boot the hardened module with no override and pass. One VM test turns the module's hardening off, asserts exit 78 with no restart loop, then sets the waiver and asserts the daemon starts with `xtcp2_confinement_waived` 1. diff --git a/docs/security-hardening/phase-3-writer-split.md b/docs/security-hardening/phase-3-writer-split.md index 7301b3cd..c2cbde30 100644 --- a/docs/security-hardening/phase-3-writer-split.md +++ b/docs/security-hardening/phase-3-writer-split.md @@ -129,7 +129,7 @@ A team standardizing on one IPC mechanism could defensibly choose gRPC-over-UDS - **Seq (monotonic per target):** writer is the source of truth. Persisted as a sidecar `.ver` (decimal, atomic, written *after* the rename). Commit order: payload → fsync → rename → sidecar. A crash between rename and sidecar leaves the sidecar one step behind; a replay at the old seq is then accepted once — harmless because both targets are full-file writes (idempotent). Rollback protection has a documented ≤ 1-step window across a writer crash; anything older is rejected (`stale_seq`). - **Locking:** the baseline's lifetime `flock` (`files.go` `ErrLocked`) becomes per-op, held by the writer; the collector's post-split load is a lock-free read (it is no longer the baseline "owner"). The writer-init bootstrap (below) re-imports an existing target without a sidecar. - **Metrics:** `xtcp2_writerd_requests_total{op,result}`, `xtcp2_writerd_last_seq{target}` — exported through `GET_STATS` and re-registered by the collector (proxy), so vmagent scraping is unchanged and the writer keeps **no TCP listener**. -- **Writer self-confinement:** `pkg/sandbox` with write rules only on the two target dirs; `-healthcheck` mode probes the socket. In the fleet, the writer container adds the docker boundary on top (`network_mode: none`, `cap_drop ALL`, uid 65534) — see PR-13. +- **Writer self-confinement:** `pkg/sandbox` with write rules only on the two target dirs; `-healthcheck` mode probes the socket. In the fleet, the writer container adds the docker boundary on top (`network_mode: none`, `cap_drop ALL`, `read_only`, `no-new-privileges`, uid 65534) — see PR-13. On bare metal the writer's systemd unit needs `NoNewPrivileges=yes`, a `SystemCallFilter=` allowlist, `ProtectSystem=strict` and an empty `CapabilityBoundingSet=`, or [PR-24](phase-1-code-hardening.md#pr-24--startup-confinement-check) refuses to start it. --- @@ -257,6 +257,7 @@ Container pair (deployment role, mirroring the existing `ipfeed` second-containe |---|---|---| | uid | 0 (unchanged — keeps `NET_ADMIN`/`SYS_ADMIN` until Phase 5 makes it a non-root uid with no capabilities); `group_add: xtcp2-state` | 65534, primary group `xtcp2-state` | | capabilities | `cap_drop ALL` + NET_ADMIN, SYS_ADMIN (unchanged) | `cap_drop ALL` — **none added** | +| rootfs, privileges | `read_only: true`, `no-new-privileges`, default seccomp (unchanged) | `read_only: true`, `no-new-privileges`, default seccomp — required by [PR-24](phase-1-code-hardening.md#pr-24--startup-confinement-check), whose writer-mode allowed capability set is empty | | network | host (unchanged, ports 9088/8889/9089) | `network_mode: none` — no interfaces, no egress at all | | mounts | as today, **minus** the rw state dir (kept `ro` for loads) + socket volume | `/var/lib/xtcp2:rw` (the only writable mount), socket volume, nothing else — no docker.sock, no netns, no S3 secret | | Landlock | zero file-write rules; read-only on the state dir; `RESOLVE_UNIX` on the writer socket (ABI ≥ 9) | write rules on the two target dirs | @@ -264,7 +265,7 @@ Container pair (deployment role, mirroring the existing `ipfeed` second-containe Socket sharing: a docker named volume mounted rw in both containers, containing only the socket dir (`0750`) and socket (`0660`). Whether the collector's mount can be `ro` (connect(2) on a UDS under an ro mount) is decided by F-table row F8 — if it works, the collector ends with **zero rw mounts, full stop**. -Alert rules (vmalert + `promtool test rules` cases): `Xtcp2WriterDown` (client-side `rate(xtcp2_writer_requests_total{result="error"}[5m]) > 0`, sustained), `Xtcp2WriterSeqStuck` (last seq unchanged while link churn is high — silent save failure), plus `Xtcp2SandboxInactive` extended to both containers. +Alert rules (vmalert + `promtool test rules` cases): `Xtcp2WriterDown` (client-side `rate(xtcp2_writer_requests_total{result="error"}[5m]) > 0`, sustained), `Xtcp2WriterSeqStuck` (last seq unchanged while link churn is high — silent save failure), plus `Xtcp2SandboxInactive` and `Xtcp2ConfinementWaived` (any non-zero `xtcp2_confinement_waived`, PR-24) extended to both containers. Rollout order — each step is independently shippable and reversible: diff --git a/docs/security-hardening/phase-5-privilege-brokers.md b/docs/security-hardening/phase-5-privilege-brokers.md index fdf68fd9..3ce25f5e 100644 --- a/docs/security-hardening/phase-5-privilege-brokers.md +++ b/docs/security-hardening/phase-5-privilege-brokers.md @@ -390,7 +390,7 @@ PR-18 sec/w10-spike ──▶ PR-20 sec/w10-nlbroker ──┐ - broker mode as the fleet default; - excess-capability warning and gauge; - token and secret files readable by group `xtcp2`; - - the capability check rewritten per mode. + - the capability check rewritten per mode, and [PR-24](phase-1-code-hardening.md#pr-24--startup-confinement-check)'s allowed set for the broker-mode collector set to none (the netlink broker's to `SYS_ADMIN`, the Docker broker's to none, in PR-20/PR-21). - [ ] **PR-23 — deployment (deployment repo):** - containers and profiles; - the rollout steps below; diff --git a/docs/security-hardening/plan-configservice-set-hardening.md b/docs/security-hardening/plan-configservice-set-hardening.md index 1bfd75ae..07bba598 100644 --- a/docs/security-hardening/plan-configservice-set-hardening.md +++ b/docs/security-hardening/plan-configservice-set-hardening.md @@ -145,7 +145,7 @@ Belt layer 4: kernel enforcement that survives `syscall.Exec` (soft restart) and - [ ] New package `pkg/sandbox`: - [ ] `Constrain(opts)` called once at first start (`cmd/xtcp2/xtcp2.go`; soft restarts inherit the domain via an fd marker — re-applying would hit the kernel's 16-layer cap), before any config-driven writer is constructed: read-only rules for `/proc`, `/sys`, `/run/netns`, `/run/docker/netns`, `/run/docker.sock`, the lldpd socket dir, ASN bootstrap/cache paths, listener token files; write rules only for `-writableDir` roots (until W5 lands, after which: none) - [ ] Carve-outs: UDS listener dir (only when a unix listener is configured); capture dir (only when `write_files` > 0 with a root configured) - - [ ] ABI feature detection: kernel < 5.13 → log a warning, set `xtcp2_sandbox_active 0`, continue (degrade-not-exit); otherwise `1` — new Prometheus gauge (add to [metrics.md](../metrics.md) in W9) + - [ ] ABI feature detection: kernel < 5.13 → exit 78 via W13's `landlock` check unless waived; when waived, log a warning, set `xtcp2_sandbox_active 0` and continue; otherwise `1` — new Prometheus gauge (add to [metrics.md](../metrics.md) in W9) - [ ] Soft-restart interplay test: restriction persists across `syscall.Exec` (unit test with a re-exec of a helper binary attempting an out-of-root write) - [ ] microVM integration scenario: daemon running, `os.WriteFile` outside roots fails with `EACCES`; state-dir writes still succeed (pre-W5) / fail (post-W5) - [ ] Optional extra belt (advisory §5.4): self-`pivot_root` confinement using the existing `CAP_SYS_ADMIN` (`chroot()` itself is impossible — `CAP_SYS_CHROOT` is dropped); carries the full bind-mount farm (`/proc` ro, `/run/netns`, `/run/docker/netns`, `/sys/fs/cgroup`, docker + lldpd sockets, state dir, tokens, ASN files, `/dev/null` + `urandom`, `/etc/resolv.conf` + `hosts` + `ssl/certs` for S3 TLS/DNS). Persists across the soft-restart exec. Landlock stays primary; prioritize for bare-metal runs where no container pidns defeats `/proc/1/root` (advisory §4.7) @@ -192,7 +192,7 @@ Changes in the deployment repo (`ansible/roles/monitoring/xtcp2/`) and the image - [ ] I11: state-dir guard — minimum: node-level alert on `/var/lib/xtcp2` usage (vmagent); preferred: XFS project quota (or systemd quota for bare metal) sized to ~2× the ASN cache (~200 MiB headroom) - [ ] Egress pinning: restrict the collector's egress to the per-DC S3 endpoints (nftables rule on its cgroup — host netns, so not container network policy; not host-wide SG/UFW, which would hit co-located workloads); pair with W2's frozen `s3_endpoint`. This holds against a stolen admin token. Against a collector RCE it holds only once W10 removes `CAP_NET_ADMIN` from the collector, which can otherwise delete the rule - [ ] W5 deployment: second container + `:ro` collector state dir + shared socket dir; canary via the existing fleet-gate mechanism; revert via `xtcp2_revert.yml` -- [ ] Regression-guard the load-bearing container defaults in the fleet-gate test (`docker inspect`): `HostConfig.PidMode` not `host`, `ReadonlyRootfs` true, `AppArmorProfile` == `docker-default` (not `unconfined`), every bind except the state dir `:ro`, and a probe that `mount -o remount,bind,rw` on a `:ro` bind fails `EACCES` inside the container +- [ ] Regression-guard the load-bearing container defaults in the fleet-gate test (`docker inspect`): `HostConfig.PidMode` not `host`, `ReadonlyRootfs` true, `AppArmorProfile` == `docker-default` (not `unconfined`), every bind except the state dir `:ro`, and a probe that `mount -o remount,bind,rw` on a `:ro` bind fails `EACCES` inside the container. W13 is the runtime twin of this gate - [ ] Pin the new image tag in the image repo; bump the deployment repo role defaults in lockstep (role gate test asserts image↔config compatibility) **Estimate**: 3–5 days spread across phases. **Owner**: infra. @@ -209,10 +209,10 @@ Changes in the deployment repo (`ansible/roles/monitoring/xtcp2/`) and the image ### W9 — documentation -- [ ] [security.md](../security.md): new "Runtime reconfiguration" section — path policy, frozen fields, Landlock gauge, writer process +- [ ] [security.md](../security.md): new "Runtime reconfiguration" section — path policy, frozen fields, Landlock gauge, writer process; the W13 checks and per-runtime settings - [ ] [grpc-api.md](../grpc-api.md): `Set` frozen-field list + error codes (`InvalidArgument` path, `FailedPrecondition` frozen) + `-setAllowFrozenFields` migration note -- [ ] [operations.md](../operations.md): writer deployment (container + systemd), `-writableDir`, quota guidance, token rotation -- [ ] [metrics.md](../metrics.md): `xtcp2_sandbox_active`, `xtcp2_state_writer_errors_total`, `xtcp2_set_applied_total` +- [ ] [operations.md](../operations.md): writer deployment (container + systemd), `-writableDir`, quota guidance, token rotation; exit 78 and the `-insecureAllowUnconfined` waiver +- [ ] [metrics.md](../metrics.md): `xtcp2_sandbox_active`, `xtcp2_state_writer_errors_total`, `xtcp2_set_applied_total`, `xtcp2_confinement_check`, `xtcp2_confinement_waived` - [ ] [README.md](../README.md): link this plan from the advisory entry - [ ] Update the advisory's layer table with ✅ per workstream as they land @@ -294,9 +294,12 @@ Advisory §5.8, [Phase 5](phase-5-privilege-brokers.md#syscall-profiles-for-ever - [ ] Per-check waiver: `-insecureAllowUnconfined=|all` / `XTCP_INSECURE_ALLOW_UNCONFINED`. Unknown names are fatal; waivers warn hourly and export `xtcp2_confinement_waived{check}`. - [ ] Argv/env only, with no config field, so `Set` cannot carry a waiver. - [ ] Harden the NixOS module (`NoNewPrivileges`, `SystemCallFilter`, `ProtectSystem=strict`, `RestartPreventExitStatus=78`) so it passes with no waiver. -- [ ] Fleet alert on any non-zero `xtcp2_confinement_waived`. +- [ ] Fleet alert on any non-zero `xtcp2_confinement_waived` (`Xtcp2ConfinementWaived`). +- [ ] Allowed capabilities per mode: collector direct mode = `requiredCaps`; writer, Docker broker and broker-mode collector = none; netlink broker = `SYS_ADMIN`. W5/W10/W11 add their rows. +- [ ] In-repo deployments pass: the k8s DaemonSet and the dev docker-compose gain the settings; `nix/checks/capability-check.nix` sets the waiver; quickstart docs show the waiver or a `systemd-run` recipe. +- [ ] Tests that assert a startup failure assert its message, not just a non-zero exit (W0's PR-1 acceptance and the image smoke test). -**Acceptance**: X1–X19. **Estimate**: 2–3 days. +**Acceptance**: X1–X24. **Estimate**: 3–4 days. ## Sequencing and dependencies diff --git a/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md b/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md index 2251de83..ee9a88ef 100644 --- a/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md +++ b/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md @@ -482,7 +482,7 @@ Layer 0 (§5.7) removes primitives 1–2 at the binary level — the production - seccomp denials; - the fixed Docker request set; - the collector RCE-probe scenario. -- **Confinement check:** Phase 1's X1–X19 table: fleet container and hardened systemd unit start; bare root, `--privileged` and a default pod exit 78; waivers are per check, typos fatal, and `Set` cannot carry one. +- **Confinement check:** Phase 1's X1–X24 table: fleet container and hardened systemd unit start; bare root, `--privileged` and a default pod exit 78; waivers are per check, typos fatal, and `Set` cannot carry one; the allowed capability set is per mode (writer and brokers). - **microVM integration:** the split deployment — collector `:ro` state, writer chrooted, `Set` a valid runtime change (poll frequency) and confirm no regression; `Set` with path fields rejected end-to-end. ## 9. Rollout sequencing @@ -490,7 +490,7 @@ Layer 0 (§5.7) removes primitives 1–2 at the binary level — the production | Phase | Contents | Risk | Notes | |---|---|---|---| | 1 — now | L0 (image rebuild) + L1 + L2 (+ L3 interceptor) + the L6 startup confinement check | low; pure build/validation change, no behavior change for legit configs | Cuts a patch release; the image repo bump; ansible pin. Frozen-field list needs an operator escape hatch (`-setAllowFrozenFields`) for migrations | -| 2 | L4 Landlock; L6 quota + `Set` audit logging | low-medium; verify fleet kernels ≥5.13 | Ubuntu 24.04 fleet is fine; feature-detect ABI and degrade to warn if absent | +| 2 | L4 Landlock; L6 quota + `Set` audit logging | low-medium; verify fleet kernels ≥5.13 | Ubuntu 24.04 fleet is fine; feature-detect ABI; an absent Landlock refuses to start unless the `landlock` check is waived (PR-24) | | 3 | L5 writer split; collector state dir `:ro`; ansible second container + systemd units | medium; new process + socket plumbing | The big win: collector has zero write rules; degrade-not-exit on writer outage | | 4 | L6 egress pinning; v1.x role revision retirement | low | Closes the exfil class at the network layer | | 5 | L7 netlink + Docker brokers; collector de-privilege; syscall profiles (the profiles ship with phase 2) | medium; two new processes, the collector loses its capabilities | The collector ends unprivileged ([Phase 5](phase-5-privilege-brokers.md)); the phase 4 egress rule then holds against a collector RCE | From 6504b8f76156e156017e2fe2c26c9ce8e071077b Mon Sep 17 00:00:00 2001 From: "randomizedcoder dave.seddon.ca@gmail.com" Date: Mon, 9 Nov 2026 17:01:32 -0800 Subject: [PATCH 7/7] docs(security-hardening): gRPC for all IPC; broker-side netlink sends - Netlink broker becomes a gRPC service (ListNetns, DumpSockets, DumpRoute) that builds and sends every netlink request from typed, validated fields and relays replies unfiltered as raw chunks; the collector no longer holds any netlink socket in a namespace - Kernel fact 5: a netlink socket has no receive-only mode and dump replies go only to the requester, so fd passing would leave sends filtered by the kernel alone - New pkg/ipc: gRPC over UDS for the writer and both brokers; peer uid checked in the transport handshake before HTTP/2 parsing - Writer moves to a WriterService (ipmeta as a client stream) - Netns ids become netns cookies; netns_direct build tag; collector seccomp forbids SOCK_DIAG; P12 relay benchmark gate with a shared-memory fallback; OD-10 revised, OD-14 added - Bare metal marked unsupported for the broker design Co-Authored-By: Claude Opus 5.5 --- .../security-hardening/implementation-plan.md | 6 +- .../phase-3-writer-split.md | 185 +++++----- .../phase-5-privilege-brokers.md | 330 ++++++++++-------- .../plan-configservice-set-hardening.md | 48 +-- ...ulnerability-configservice-set-fs-write.md | 15 +- 5 files changed, 325 insertions(+), 259 deletions(-) diff --git a/docs/security-hardening/implementation-plan.md b/docs/security-hardening/implementation-plan.md index 9b677a0f..2c518a83 100644 --- a/docs/security-hardening/implementation-plan.md +++ b/docs/security-hardening/implementation-plan.md @@ -116,11 +116,11 @@ PRs (repo `xtcp2` unless noted; each lands with its tests and doc updates from W > Drill-down with the broker protocols, filtering pipelines, syscall profiles and full test matrices: [phase-5-privilege-brokers.md](phase-5-privilege-brokers.md). -**Goal:** a compromised collector holds no capability, no `docker.sock` and no namespace handle; the privileged reads go through two brokers whose APIs can express nothing else. +**Goal:** a compromised collector holds no capability, no `docker.sock` and no namespace handle, and cannot send a netlink message into any namespace; the privileged reads go through two gRPC brokers whose APIs can express nothing else. -- [ ] **PR-18 — spike** ([W10](plan-configservice-set-hardening.md#w10--netlink-broker-and-collector-de-privilege-l7)): kernel fd-semantics proofs and a zero-capability feature run. +- [ ] **PR-18 — spike** ([W10](plan-configservice-set-hardening.md#w10--netlink-broker-and-collector-de-privilege-l7)): kernel-fact proofs, a dump-relay prototype with a zero-capability feature run, and the relay benchmark gate. - [ ] **PR-19 — syscall profiles** ([W12](plan-configservice-set-hardening.md#w12--syscall-profiles-l7)): per-process seccomp and AppArmor; the v1 collector profile ships with Phase 2. -- [ ] **PR-20 — netlink broker** (W10): `xtcp2 -nlbroker`, proto over `SOCK_SEQPACKET` with `SCM_RIGHTS`, collector seams. +- [ ] **PR-20 — netlink broker** (W10): `xtcp2 -nlbroker`, a gRPC service that builds and sends every netlink request and streams the replies back as raw chunks; collector seams. - [ ] **PR-21 — Docker broker** ([W11](plan-configservice-set-hardening.md#w11--docker-broker-l7)): `xtcp2 -dockerbroker`, typed gRPC, fixed outbound request set, projection. - [ ] **PR-22 — collector de-privilege** (W10): non-root uid, `cap_drop ALL`, broker mode as the fleet default. - [ ] **PR-23 — deployment** (the deployment repo): broker containers, profiles, the five-step rollout, alerts. diff --git a/docs/security-hardening/phase-3-writer-split.md b/docs/security-hardening/phase-3-writer-split.md index c2cbde30..2da9d64e 100644 --- a/docs/security-hardening/phase-3-writer-split.md +++ b/docs/security-hardening/phase-3-writer-split.md @@ -5,7 +5,8 @@ | **Status** | Not started — checkboxes track execution | | **Tracks** | [Implementation plan](implementation-plan.md) Phase 3 → [hardening plan](plan-configservice-set-hardening.md) workstream W5 → [advisory](security-vulnerability-configservice-set-fs-write.md) layer L5 | | **Goal** | The collector process — the part that talks to the network, parses attacker-influenced data, and serves gRPC — holds **no filesystem write capability at all**. The only two legitimate writes move to a separate, minimal, unprivileged writer process | -| **Prereqs** | Phase 1 (frozen path fields so `Set` can't move targets; build tags) and Phase 2 (`pkg/sandbox`, reused by both processes) | +| **Prereqs** | Phase 1 (frozen path fields so `Set` can't move targets; build tags; PR-4's validating interceptor) and Phase 2 (`pkg/sandbox`, reused by both processes) | +| **Introduces** | `pkg/ipc`, the gRPC-over-UDS stack every xtcp2 process uses to talk to another; Phase 5's two brokers reuse it | ## Contents @@ -30,7 +31,7 @@ The daemon has exactly two legitimate writes (advisory §2): the link-monitor ba |---|---|---| | Collector RCE writes anywhere validation allows | one trusted process, one ruleset | collector has zero file-write rules — the kernel answer is "no", always | | `Set` flips a path field | rejected (Phase 1) / kernel-denied (Phase 2) | same, and the path fields no longer exist in the collector's write path at all | -| Collector RCE **as root** wants to poison state | — | must speak the writer's fixed 3-op API: two named targets, one size cap each, monotonic seq — no paths, no arbitrary bytes past schema checks; schema-valid poison at the next `seq` still lands, and `docker.sock` bypasses the writer entirely | +| Collector RCE **as root** wants to poison state | — | must speak the writer's fixed write API (two write RPCs): two named targets, one size cap each, monotonic seq — no paths, no arbitrary bytes past schema checks; schema-valid poison at the next `seq` still lands, and `docker.sock` bypasses the writer entirely | | Writer itself compromised | — | unprivileged uid, no capabilities, no network (container `network_mode: none`), Landlock write rules only on its two target dirs | What this phase does **not** fix: the collector keeps `docker.sock` (until [Phase 5](phase-5-privilege-brokers.md)), so an RCE can have dockerd write any host path (the state dir included) without touching the writer; the collector can still reach network egress (Phase 4); and the writer accepts a well-formed request from uid 0 (or a stolen shared secret — see [decisions](#decisions-to-close-during-phase-3)) and will overwrite its two fixed targets with schema-valid data; that bounded surface is the accepted residual. @@ -47,88 +48,106 @@ PR-11 sec/w5-writer ──▶ PR-12 sec/w5-collector ──▶ PR-13 sec/w5-depl ## PR-11 — The writer process -**Objective:** a minimal writer daemon — same binary, `-writer` mode — exposing four operations over one unix socket. The security model lives in the parameter list: **no request anywhere carries a path.** +**Objective:** a minimal writer daemon — same binary, `-writer` mode — exposing a four-RPC gRPC service over one unix socket. The security model lives in the parameter list: **no request anywhere carries a path.** ### Files - New `pkg/writerd/writerd.go` — server, request validation, target registry, seq state, crash-safe commit dance -- New `pkg/writerd/proto.go` — the `[len u32]` frame + proto envelope codec (below) -- New `proto/xtcp_writer/v1/writer.proto` — message-only envelope (no `service` definition), annotated with `buf.validate` rules, generated via the usual `make regen-protos` +- New `pkg/ipc/` — the shared gRPC-over-UDS stack (below): peer-credential transport credentials, hardened server and client options, the interceptor chain, metrics +- New `proto/xtcp_writer/v1/writer.proto` — the `WriterService` gRPC service, annotated with `buf.validate` rules, generated via the usual `make regen-protos` - New `pkg/writerd/client.go` — the collector-side client -- `cmd/xtcp2/xtcp2.go` — `-writer` mode: writer never parses `XtcpConfig`, never registers gRPC services, never initializes enrichers/netlink; it reads its own flags only -- `pkg/xtcp/prometheus.go` — collector-side `xtcp2_writer_requests_total{op,result}` (client view) + proxy registration for the writer's own counters -- `nix/versions.nix` — no new dependencies: protobuf runtime (`v1.36.12`) and protovalidate-go are already direct dependencies; `golang.org/x/sys/unix` suffices for `SO_PEERCRED` +- `cmd/xtcp2/xtcp2.go` — `-writer` mode: writer never parses `XtcpConfig`, registers only `WriterService` (none of the collector's gRPC services, no TCP listener), never initializes enrichers/netlink; it reads its own flags only +- `pkg/xtcp/prometheus.go` — collector-side `xtcp2_writer_requests_total{rpc,result}` (client view) + proxy registration for the writer's own counters +- `nix/versions.nix` — no new dependencies: grpc-go, the protobuf runtime (`v1.36.12`) and protovalidate-go are already direct dependencies; `golang.org/x/sys/unix` suffices for `SO_PEERCRED` ### The API -| Op | Name | Request | Writer-side effect | Writer-side validation | -|---|---|---|---|---| -| 0 | `PING` | — | reply | none | -| 1 | `WRITE_BASELINE` | `payload []byte`, `seq uint64` | temp+fsync+rename onto the configured baseline target (`0640`, group `xtcp2-state`) | oneof arm selects the `baseline` target; protovalidate `max_len 65536` (== `baseline.MaxFileBytes`); payload parses as a `wireRecord` with `Version == 1` (`baseline.Parse` reused); `seq` monotonic (`gte 1`) | -| 2 | `WRITE_IPMETA` | `payload []byte`, `seq uint64` | temp + preserve-previous (`current → previous`, both writer-configured) + rename + `syncDir` — the exact dance of `ipasn.PublishLookupCache` (`ipasn.go:373`) | oneof arm selects the `ipmeta` target; protovalidate `max_len` == `-ipmetaMaxBytes` (default 64 MiB); `seq` monotonic (`gte 1`) | -| 3 | `GET_STATS` | — | counters + last `seq` per target | none | +| RPC | Request | Writer-side effect | Writer-side validation | +|---|---|---|---| +| `Ping` | — | reply | none | +| `WriteBaseline` | `payload bytes`, `seq uint64` | temp+fsync+rename onto the configured baseline target (`0640`, group `xtcp2-state`) | the RPC selects the `baseline` target; protovalidate `max_len 65536` (== `baseline.MaxFileBytes`); payload parses as a `wireRecord` with `Version == 1` (`baseline.Parse` reused); `seq` monotonic (`gte 1`) | +| `WriteIpmeta` (client stream) | a header (`seq`, `total_size`), then `data` chunks ≤ 1 MiB | chunks stream into the temp file; then preserve-previous (`current → previous`, both writer-configured) + rename + `syncDir` — the exact dance of `ipasn.PublishLookupCache` (`ipasn.go:373`) | the RPC selects the `ipmeta` target; `total_size` ≤ `-ipmetaMaxBytes` (default 64 MiB); received bytes must equal `total_size` exactly; `seq` monotonic (`gte 1`) | +| `GetStats` | — | counters + last `seq` per target | none | The two write ops port existing code, not new logic: baseline's atomic dance comes from `pkg/linkmonitor/internal/baseline/files.go`; ipmeta's from `pkg/ipasn/ipasn.go` (`MkdirAll 0750`, `preservePrevious`, `syncDir`). Validation of the *ipmeta payload* stays collector-side (`validateCacheCandidate` needs the refresh loop's `Stats`); the writer validates size, schema only where cheap (baseline), and seq. -### Wire format and the protocol decision - -Frame: little-endian `[len u32]`, then one protobuf `Request`; reply frame `[len u32]`, one protobuf `Response` (oneof `ok` / `err{code,detail}` / `stats`). The envelope is proto — deliberately **not** a gRPC service: +### Wire format: gRPC over UDS (`pkg/ipc`) + +Every xtcp2 process talks to another over gRPC on a unix socket, through one shared package. PR-11 introduces it with the writer as its first user; Phase 5's [netlink and Docker brokers](phase-5-privilege-brokers.md#design-rules-for-every-broker) reuse it unchanged. One stack means typed services, generated clients, deadlines and streaming for free, and one place to harden, fuzz and review. + +**`pkg/ipc`:** +- **Peer-credential transport credentials.** A `credentials.TransportCredentials` whose server handshake reads `SO_PEERCRED` from the accepted unix conn and rejects a wrong uid **before any HTTP/2 frame is parsed**, so only the expected peer ever reaches grpc-go's HTTP/2 server. The client handshake checks the server's uid the same way (mutual). The uid is exposed as `AuthInfo` for logs. grpc-go has no stock support, so this is ~100 lines with its own tests (Test C). +- **Server options:** + - UDS only; any TCP listener is a startup error; + - `MaxRecvMsgSize` and `MaxSendMsgSize` per service — grpc-go checks the length prefix before allocating; + - `MaxConcurrentStreams` small, per service; + - a keepalive enforcement policy; + - no reflection or health service (health is `-healthcheck` mode); + - no compression codecs registered; + - an unknown-service handler returning `Unimplemented`. +- **Interceptor chain** (unary and stream, in order): + 1. unknown fields rejected — protobuf-go stores them by default (`decode.go` `m.SetUnknown`), so the interceptor walks the message and requires `GetUnknown() == nil`; + 2. protovalidate (PR-4's validating interceptor); + 3. a per-RPC rate limit; + 4. the service's own filter stages. +- **Client side:** the same protovalidate rules before send, and a deadline on every call. +- **Metrics:** `xtcp2_ipc_requests_total{service,rpc,stage}`. ```proto -// proto/xtcp_writer/v1/writer.proto — message-only, no service definition. +// proto/xtcp_writer/v1/writer.proto +service WriterService { + rpc Ping(PingRequest) returns (PingResponse); + rpc WriteBaseline(WriteBaselineRequest) returns (WriteResponse); + // One header, then data chunks. Bytes stream into the temp file; the commit + // happens only after the stream completes at exactly total_size. + rpc WriteIpmeta(stream WriteIpmetaChunk) returns (WriteResponse); + rpc GetStats(StatsRequest) returns (StatsResponse); +} message WriteBaselineRequest { uint64 seq = 1 [(buf.validate.field).uint64.gte = 1]; - bytes payload = 2 [(buf.validate.field).bytes = {min_len: 1, max_len: 65536}]; // == baseline.MaxFileBytes + bytes payload = 2 [(buf.validate.field).bytes = {min_len: 1, max_len: 65536}]; // == baseline.MaxFileBytes } -message WriteIpmetaRequest { - uint64 seq = 1 [(buf.validate.field).uint64.gte = 1]; - bytes payload = 2 [(buf.validate.field).bytes = {min_len: 1, max_len: 67108864}]; // == -ipmetaMaxBytes default +message WriteIpmetaHeader { + uint64 seq = 1 [(buf.validate.field).uint64.gte = 1]; + uint64 total_size = 2 [(buf.validate.field).uint64 = {gte: 1, lte: 67108864}]; // == -ipmetaMaxBytes default } -message Request { - oneof op { +message WriteIpmetaChunk { + oneof part { option (buf.validate.oneof).required = true; - PingRequest ping = 1; - WriteBaselineRequest write_baseline = 2; - WriteIpmetaRequest write_ipmeta = 3; - StatsRequest stats = 4; + WriteIpmetaHeader header = 1; // first message only + bytes data = 2 [(buf.validate.field).bytes = {min_len: 1, max_len: 1048576}]; } } -// Response mirrors: oneof { Ok ok = 1; Err err = 2; StatsResponse stats = 3; } +message WriteResponse { uint64 seq = 1; } ``` -Per-op caps live in per-message rules — the oneof expresses them natively where a flat message could not. `.Validate()` runs on **both ends**: collector-side before send (catches collector bugs at the source), writer-side after unmarshal (catches everything else). - -**Why not gRPC over the UDS?** Both processes run the same binary and gRPC is already in it, so this is about *exposure*, not dependency weight: - -| Dimension | gRPC over UDS | proto envelope + tiny frame (chosen) | -|---|---|---| -| Parser facing a pwned collector | the full HTTP/2 stack — SETTINGS, HPACK, flow control, trailers, keepalive; a CVE history (rapid reset et al.) | `len u32` + proto unmarshal | -| Validation | protovalidate via PR-4's universal interceptor (genuine reuse) | protovalidate via `.Validate()` — same rules | -| `SO_PEERCRED` | custom listener-wrapper code (no stock support) | direct `GetsockoptUcred` on the accepted conn | -| Deadlines/timeouts | free (ctx) | ~20 lines in the client (covered by D3) | -| Knobs to get right | `MaxRecvMsgSize`, keepalive, streams, compression codecs, reflection, health service — each a misconfiguration risk | none | -| Evolvability | new RPCs at will | new oneof arms — the friction is deliberate (a frozen API is the security property) | +Errors are gRPC status codes: `InvalidArgument` (malformed or invalid), `FailedPrecondition` (`stale_seq`, `unconfigured_target`, `invalid_payload`), `ResourceExhausted` (size or rate), `Unauthenticated` (peer uid). `.Validate()` runs on **both ends**: collector-side before send (catches collector bugs at the source), writer-side after decode (catches everything else). Streaming ipmeta also means the writer never holds a 64 MiB payload in memory. -A team standardizing on one IPC mechanism could defensibly choose gRPC-over-UDS anyway; the hardening checklist is then: `MaxRecvMsgSize` at the payload cap + margin, no reflection service, no health service, no compression codecs imported, keepalive disabled, one stream per connection — and the same filter chain below. +**Why gRPC, given the HTTP/2 surface.** An earlier draft used a 4-byte length frame plus a proto envelope to keep HTTP/2 out of the writer. The decision now is one IPC stack for every process, and the exposure is closed inside gRPC instead: +- the peer uid is checked in the handshake, so a process other than the collector never reaches the HTTP/2 parser; +- the options above remove every optional surface (reflection, health, compression, unbounded streams); +- the residual is a compromised collector itself reaching grpc-go's HTTP/2 server, which the release checklist covers with a current grpc-go (CVE gate) and fuzz targets. -**Writer receive pipeline — a series of fail-closed filters, each stage metric-tagged (`xtcp2_writerd_requests_total{op,filter}`):** +**Writer receive pipeline — a series of fail-closed filters, each stage metric-tagged (`xtcp2_writerd_requests_total{rpc,filter}`):** -1. frame `len` checked against the global cap (64 MiB + margin) **before allocation** — `make(4 GiB)` is unreachable (protobuf-go `v1.36.12` has no `UnmarshalOptions.MaxSize`; this check is the allocation guard) -2. `proto.Unmarshal` → `malformed` on parse failure; unknown fields are **rejected explicitly** — the default stores them (`decode.go` `m.SetUnknown`), so the writer asserts `GetUnknown() == nil` after unmarshal -3. `.Validate()` (protovalidate: `gte 1`, per-op `max_len`) → `invalid` -4. op/target allow-list — the op's target must have been passed via `-writerTarget` → `unconfigured_target` -5. `seq` monotonic per target (sidecar state) → `stale_seq` -6. payload schema check (baseline `wireRecord` parse) → `invalid_payload` -7. commit dance (temp → fsync → rename → sidecar) +1. peer uid (`pkg/ipc` handshake) → `Unauthenticated` +2. message size: `MaxRecvMsgSize` 1 MiB + margin, checked by grpc-go against the length prefix **before allocation** → `ResourceExhausted` +3. unknown fields rejected → `InvalidArgument` +4. `.Validate()` (protovalidate: `gte 1`, per-RPC `max_len`, `total_size` cap) → `InvalidArgument` +5. target allow-list — the RPC's target must have been passed via `-writerTarget` → `unconfigured_target` +6. `seq` monotonic per target (sidecar state) → `stale_seq` +7. ipmeta stream discipline: header first and only first; running byte count never above `total_size`, and equal to it at end of stream; a cancelled or short stream removes the temp file → `InvalidArgument` +8. payload schema check (baseline `wireRecord` parse) → `invalid_payload` +9. commit dance (temp → fsync → rename → sidecar) ### Targets, auth, seq, locking -- **Targets** come from repeated `-writerTarget name=path` flags — the writer's own config, reachable by nothing `Set` controls. The request's oneof arm selects the target — the envelope carries no strings at all; an op whose target was not configured is rejected (`unconfigured_target`). -- **Auth:** `SO_PEERCRED` on each accepted connection — peer uid must equal the configured collector uid (fleet: 0; check injectable for tests like `capgetFunc` in `init_capabilities.go`). Socket dir `0750`, socket `0660`, both group `xtcp2-state`. Both are *layers*: uid 0 already passes the cred check, so the real auth is the API surface itself (see decisions). +- **Targets** come from repeated `-writerTarget name=path` flags — the writer's own config, reachable by nothing `Set` controls. The RPC selects the target — no request carries a string at all; an RPC whose target was not configured is rejected (`unconfigured_target`). +- **Auth:** `SO_PEERCRED` in the `pkg/ipc` handshake on each accepted connection — peer uid must equal the configured collector uid (fleet: 0; check injectable for tests like `capgetFunc` in `init_capabilities.go`). Socket dir `0750`, socket `0660`, both group `xtcp2-state`. Both are *layers*: uid 0 already passes the cred check, so the real auth is the API surface itself (see decisions). - **Ownership:** the collector is uid 0 **without** `CAP_DAC_OVERRIDE`/`CAP_DAC_READ_SEARCH`, so it cannot read a `0600` file owned by uid 65534. The writer therefore runs as `65534` with primary group `xtcp2-state` (a fixed gid in the image and role) and umask `027`: targets `0640`, dirs `0750`, group `xtcp2-state`. The collector joins the group (`group_add`, the same pattern as the lldpd socket) and keeps a **read-only** Landlock grant on the state dir. writer-init (one-shot, root) chowns a pre-split state dir and its root-owned `0600` targets to `65534:xtcp2-state` with these modes before the writer starts. - **Seq (monotonic per target):** writer is the source of truth. Persisted as a sidecar `.ver` (decimal, atomic, written *after* the rename). Commit order: payload → fsync → rename → sidecar. A crash between rename and sidecar leaves the sidecar one step behind; a replay at the old seq is then accepted once — harmless because both targets are full-file writes (idempotent). Rollback protection has a documented ≤ 1-step window across a writer crash; anything older is rejected (`stale_seq`). - **Locking:** the baseline's lifetime `flock` (`files.go` `ErrLocked`) becomes per-op, held by the writer; the collector's post-split load is a lock-free read (it is no longer the baseline "owner"). The writer-init bootstrap (below) re-imports an existing target without a sidecar. -- **Metrics:** `xtcp2_writerd_requests_total{op,result}`, `xtcp2_writerd_last_seq{target}` — exported through `GET_STATS` and re-registered by the collector (proxy), so vmagent scraping is unchanged and the writer keeps **no TCP listener**. +- **Metrics:** `xtcp2_writerd_requests_total{rpc,result}`, `xtcp2_writerd_last_seq{target}` — exported through `GetStats` and re-registered by the collector (proxy), so vmagent scraping is unchanged and the writer keeps **no TCP listener**. - **Writer self-confinement:** `pkg/sandbox` with write rules only on the two target dirs; `-healthcheck` mode probes the socket. In the fleet, the writer container adds the docker boundary on top (`network_mode: none`, `cap_drop ALL`, `read_only`, `no-new-privileges`, uid 65534) — see PR-13. On bare metal the writer's systemd unit needs `NoNewPrivileges=yes`, a `SystemCallFilter=` allowlist, `ProtectSystem=strict` and an empty `CapabilityBoundingSet=`, or [PR-24](phase-1-code-hardening.md#pr-24--startup-confinement-check) refuses to start it. --- @@ -142,13 +161,13 @@ Harness: `pkg/writerd` mirrors the injectable-`operations` seam from `baseline/f | ID | Cat | Description | Input / setup | Expected outcome | |---|---|---|---|---| | A1 | P | valid baseline write | 1 KiB valid wireRecord, seq 1 | accepted; file at configured target == payload, mode `0640`, group `xtcp2-state`; reply OK | -| A2 | N | op whose target is not configured | op 2 against a writer started baseline-only (`-writerTarget baseline=…`) | rejected `unconfigured_target`; nothing written | -| A3 | N | attacker stuffs path-like bytes into the payload | path bytes inside the payload | no string in the envelope is ever resolved as a path: schema-invalid → `invalid_payload`; schema-valid → written verbatim to the one fixed target (the no-path-parameters property — there is no field to abuse) | +| A2 | N | RPC whose target is not configured | `WriteIpmeta` against a writer started baseline-only (`-writerTarget baseline=…`) | rejected `unconfigured_target`; nothing written | +| A3 | N | attacker stuffs path-like bytes into the payload | path bytes inside the payload | no string in any request is ever resolved as a path: schema-invalid → `invalid_payload`; schema-valid → written verbatim to the one fixed target (the no-path-parameters property — there is no field to abuse) | | A4 | B | payload exactly at the baseline cap | `len == baseline.MaxFileBytes` (65 536) | accepted | | A5 | N | payload one byte over | 65 537 B | rejected `invalid` at `.Validate()` (`max_len 65536`) — the framing cap (B4) is the layer above it; nothing written | | A6 | P | baseline payload with `Version: 1`, all fields | well-formed wireRecord | accepted | | A7 | N | baseline payload not baseline JSON | `Version: 2` / missing fields / garbage | rejected `invalid_payload` — writer-side schema check | -| A8 | N | seq 0 | any write op | rejected `invalid` at `.Validate()` (`gte 1`) — before the seq filter ever sees it | +| A8 | N | seq 0 | any write RPC | rejected `invalid` at `.Validate()` (`gte 1`) — before the seq filter ever sees it | | A9 | N | replay: seq == last seen | two requests, same seq | first accepted, second rejected `stale_seq` | | A10 | N | rollback: seq < last seen | write at last-5 | rejected `stale_seq` | | A11 | B | first request after writer-init import | target pre-exists, sidecar imported, seq 1 | accepted | @@ -158,31 +177,33 @@ Harness: `pkg/writerd` mirrors the injectable-`operations` seam from `baseline/f | A15 | C | bootstrap: target exists, no sidecar | writer-init run first | init imports (sidecar `0`), writer starts, write at seq 1 accepted | | A16 | N | zero-length payload | `WriteBaselineRequest{seq: 1, payload: }` | rejected `invalid` at `.Validate()` (`min_len 1`) | -### Test B — framing (`TestWriterdFraming`) +### Test B — transport (`TestWriterdTransport`, real UDS) | ID | Cat | Description | Input / setup | Expected outcome | |---|---|---|---|---| -| B1 | P | encode/decode round trip | valid frame with payload | parsed fields identical | -| B2 | N | truncated length prefix | 3 of the 4 `len` bytes, then EOF | `malformed`, connection closed, no write attempted | -| B3 | N | no op set | `Request` with an empty oneof (the oneof arm is the op; there is no op byte) | rejected `invalid` at `.Validate()` (`oneof required`) | -| B4 | N | frame length lies huge | `len 0xFFFFFFFF` | rejected against the global cap **before allocation** (no OOM), connection closed | -| B5 | C | payload shorter than declared, then EOF | `len` says 100 B, sender sends 50 and closes | `malformed`; no partial file written | -| B6 | C | trailing bytes after the envelope | valid frame + 4 junk bytes before the next `len` | `malformed` (strict framing) | -| B7 | P | empty-payload op | `GET_STATS` frame | valid, stats reply | -| B8 | B | error reply at max size | err text 1024 B | encoded fine; collector decodes it | -| B9 | N | envelope violates protovalidate: seq 0 | `WriteBaselineRequest{seq: 0}` | rejected `invalid` at `.Validate()` (writer side; the collector side rejected it before send — same rules, both ends) | -| B10 | N | protovalidate per-op cap: baseline payload 65 537 B | hand-built envelope over `max_len` | rejected `invalid` at `.Validate()` — layered under the framing cap (B4) | -| B11 | N | unknown/forward-incompatible field in the envelope | hand-crafted frame with field 99 set | rejected `malformed` — explicit `GetUnknown() == nil` check, fail closed (protobuf-go stores unknown fields by default) | -| B12 | P | both-ends validation round trip | valid request through collector-side `.Validate()` → marshal → writer unmarshal → `.Validate()` | both sides pass; mutating any bound field flips exactly the writer side (the layering is observable) | +| B1 | P | round trip | `WriteBaseline` with a valid payload | fields identical writer-side; reply `seq` echoes | +| B2 | N | message above `MaxRecvMsgSize` | 1 MiB + margin + 1 | `ResourceExhausted` before allocation; no write attempted | +| B3 | N | unknown method | raw call to `/xtcp_writer.v1.WriterService/Delete` | `Unimplemented` | +| B4 | N | reflection and health absent | reflection and health clients | `Unimplemented` | +| B5 | N | ipmeta stream without a header | first message is `data` | `InvalidArgument`; no temp file created | +| B6 | N | second header mid-stream | header, data, header | `InvalidArgument`; temp file removed | +| B7 | B | ipmeta at exactly `total_size` | chunks summing to `total_size` | committed | +| B8 | N | ipmeta stream exceeds `total_size` | one byte over | `InvalidArgument` at the overflowing chunk; temp removed; target unchanged | +| B9 | C | client cancels mid-stream | cancel after half the bytes | temp removed; target and sidecar unchanged | +| B10 | N | short stream | stream closes below `total_size` | `InvalidArgument`; nothing renamed | +| B11 | N | unknown field in a request | field 99 set | `InvalidArgument` — the `GetUnknown() == nil` interceptor, fail closed | +| B12 | P | both-ends validation round trip | collector-side `.Validate()` → send → writer-side `.Validate()` | both pass; mutating a bound field after client validation fails only writer-side (the layering is observable) | +| B13 | B | concurrent streams | `MaxConcurrentStreams` + 1 | excess stream refused | ### Test C — peer auth (`TestWriterdPeerAuth`) | ID | Cat | Description | Input / setup | Expected outcome | |---|---|---|---|---| | C1 | P | peer uid equals the collector uid | injected cred uid 0 | request served | -| C2 | N | peer uid not allowed | injected cred uid 1234 | rejected at handshake with `auth` error; counter `{op,result="auth"}` +1; nothing written | +| C2 | N | peer uid not allowed | injected cred uid 1234 | handshake fails before any HTTP/2 frame is parsed (frame counter 0); counter `{rpc,result="auth"}` +1; nothing written | +| C6 | N | client refuses a wrong server | server runs as an unexpected uid | client handshake fails; no request sent | | C3 | C | wrong-uid peer blocked by file perms first | socket `0660 root:root`, dir `0750`, peer uid 1234 | connect fails `EACCES` before any protocol bytes | -| C4 | N | writer never listens on TCP | server constructed, inspect listener calls | only `unix` network listeners; any config attempt to add TCP is a startup error | +| C4 | N | writer never listens on TCP | server constructed, inspect listener calls | only `unix` network listeners; `pkg/ipc` makes any TCP listener a startup error | | C5 | C | peer-cred lookup fails | injected lookup returns EBADF | connection closed, counter `auth`, writer stays up | ### Test D — client resilience (`TestWriterdClient`, fake server) @@ -194,7 +215,7 @@ Harness: `pkg/writerd` mirrors the injectable-`operations` seam from `baseline/f | D3 | B | slow writer beyond client timeout | fake server delays | deadline honored, error surfaces, no goroutine leak (leakcheck) | | D4 | C | connection drops mid-request, then writer recovers | fake server closes once | client errors, reconnects on next attempt, second save succeeds | | D5 | P | stats proxy | fake server serves counters | collector re-exports them under the `xtcp2_writerd_*` names | -| D6 | P | seq bootstrapping | `GET_STATS` returns last seq 7 for `baseline` | collector's next save uses seq 8 (writer stays the source of truth across collector restarts) | +| D6 | P | seq bootstrapping | `GetStats` returns last seq 7 for `baseline` | collector's next save uses seq 8 (writer stays the source of truth across collector restarts) | ### Test G — crash safety (`TestWriterdCrashSafety`, injected faults) @@ -243,7 +264,7 @@ Collector post-split ruleset delta (from the [Phase 2 table](phase-2-kernel-conf | E7 | N | writer down during ASN refresh | socket absent | refresh errors, counter, collector serves from in-memory cache; next refresh recovers | | E8 | C | first-ever startup, no baseline file | empty state dir | collector proceeds without baseline; writer-init creates dir + sidecar (F6 covers the ops step) | | E9 | N | hostile `file:` `Set` replayed | `dest:"file:/proc/1/root/root/.ssh/authorized_keys"` | rejected by validation (Phase 1) — and with the ruleset flip, nothing writable exists anyway (cross-layer regression row) | -| E10 | C | collector restarts mid-session | restart between two saves | seq continues from `GET_STATS` (D6), no stale-seq rejection of the first save | +| E10 | C | collector restarts mid-session | restart between two saves | seq continues from `GetStats` (D6), no stale-seq rejection of the first save | --- @@ -297,8 +318,8 @@ Rollout order — each step is independently shippable and reversible: |---|---|---|---|---|---| | 1 | `pkg/xtcp/destinations_file.go:28` | **`Set`** (`dest:"file:"`) | records, `O_APPEND`, arbitrary path, `0600` | debug export | PR-1 `dest_file` build tag; out of prod flavor | | 2 | `pkg/xtcp/netlinker.go:84` | **`Set`** (`write_files` + `capture_path`) | netlink capture dump, arbitrary path | debug | PR-1 `capture` build tag | -| 3 | `pkg/linkmonitor/internal/baseline/files.go` | internal (linkmonitor learn) | baseline JSON, ≤ 64 KiB, temp+rename | **state** | **writer op `WRITE_BASELINE`** | -| 4 | `pkg/ipasn/ipasn.go:373-439` | internal (ASN refresh) | ipmeta cache + previous, temp+rename+syncDir | **state** | **writer op `WRITE_IPMETA`** | +| 3 | `pkg/linkmonitor/internal/baseline/files.go` | internal (linkmonitor learn) | baseline JSON, ≤ 64 KiB, temp+rename | **state** | **writer RPC `WriteBaseline`** | +| 4 | `pkg/ipasn/ipasn.go:373-439` | internal (ASN refresh) | ipmeta cache + previous, temp+rename+syncDir | **state** | **writer RPC `WriteIpmeta`** | | 5 | `pkg/listener/listener.go:149,165,214` | **`Set`** (listener config) | UDS bind / chmod ≤0777 / unlink stale — sockets, not files | sockets | narrow Landlock grants (`MAKE_SOCK\|REMOVE_FILE` on the listen dir); fleet listeners are TCP-only | | 6 | `pkg/linkmonitor/internal/linuxio/record_linux.go:149` | env `LINKMONITOR_NLRECORD=` at **deploy** time — not `Set`-reachable (env survives the soft restart unchanged but no `Set` field touches it) | `--.pcap` netlink recordings, `O_EXCL`, into the env dir | debug recording | stays env-gated; Landlock requires the dir under a writable root; never set in the fleet | | 7 | `pkg/linkmonitor/internal/rdmacapture/manifest.go` | standalone `rdma-capture-manifest` tool only | manifest + payloads | analysis artifact | not linked into the daemon; no change | @@ -309,8 +330,8 @@ Non-writes verified: `pkg/misc/misc.go:88,116` are `O_RDONLY` config loads; the | Need | How it is served | Verdict | |---|---|---| -| baseline state | writer `WRITE_BASELINE` | exists | -| ipmeta cache | writer `WRITE_IPMETA` | exists | +| baseline state | writer `WriteBaseline` | exists | +| ipmeta cache | writer `WriteIpmeta` | exists | | local record export (the `file:` need) | `stdout`/`stderr` dests + redirect; kafka/nats/nsq/valkey/s3parquet dests | dev builds only; if a fleet need ever materializes: a **named journal op** with fixed path + cap + rotation — never a generic write | | netlink pcap capture (`capture_path`, `LINKMONITOR_NLRECORD`) | debug image + writable root, time-boxed, Landlock-scoped | never a writer op — chatty and unbounded by nature | | node_exporter textfile snapshot | vmagent scrapes the daemon's metrics port directly | refuse; if ever genuinely needed: small fixed-file writer op (cap ~64 KiB) | @@ -323,7 +344,7 @@ Non-writes verified: `pkg/misc/misc.go:88,116` are `O_RDONLY` config loads; the **C. Admission checklist for a new write target** — the rule that keeps the property: -1. a named oneof arm in `writer.proto` — the envelope gains no strings +1. a named RPC in `writer.proto` — no request gains a string 2. fixed path from the writer's own `-writerTarget` flags, never from a request or `XtcpConfig` 3. size cap via protovalidate `max_len` + a semantic schema check where one exists (baseline's `wireRecord` parse is the model) 4. `seq` monotonic + sidecar + the atomic commit dance @@ -338,16 +359,16 @@ Adding a write destination is an engineering decision with review and a redeploy | # | Decision | Recommendation | |---|---|---| | OD-4 (socket auth strength) | `SO_PEERCRED` uid alone vs + shared secret | ship uid + file perms this phase (API surface is the real auth); add a 0400 shared-secret file mounted ro in both containers if review wants the uid ambiguity closed | -| new | writer IPC format: gRPC over UDS vs proto envelope + tiny frame | **proto envelope + tiny frame** ([rationale](#wire-format-and-the-protocol-decision)) — protovalidate both ends, no HTTP/2 stack on the writer; gRPC variant only behind its hardening checklist if the team standardizes on one IPC mechanism | +| new | writer IPC format | **gRPC over UDS on `pkg/ipc`** ([rationale](#wire-format-grpc-over-uds-pkgipc)) — one stack for the writer and both Phase 5 brokers; peer uid checked before HTTP/2 parsing; protovalidate both ends | | new | ipmeta payload cap default | 64 MiB (measured bootstrap × headroom) | | new | sidecar ordering (rename→sidecar) | accept the ≤ 1-step replay window across a writer crash (full-file writes make replays idempotent); document in the runbook | | new | baseline `flock` ownership | writer holds per-op; collector reads lock-free; `ErrLocked` now means "two writers" (misconfig, fatal in writer) | | new | socket volume `ro` in collector | decided by F8 | -| new | writer seq source of truth | writer (via `GET_STATS`); collector never persists seq | +| new | writer seq source of truth | writer (via `GetStats`); collector never persists seq | ## Release checklist -- [ ] PR-11: tests A–D, G green (pure, any platform); protocol code reviewed line-by-line (it is the entire attack surface) +- [ ] PR-11: tests A–D, G green; `pkg/ipc` and the service code reviewed line-by-line (they are the entire attack surface); grpc-go at a release with no open HTTP/2 CVE; a coverage-guided fuzz target on the writer's RPCs - [ ] PR-12: tests E green; `enrich_asn.go` seam and `BaselineStore` implementation reviewed for one-call-site fidelity - [ ] Metrics: `xtcp2_writer_*` (collector) + `xtcp2_writerd_*` (proxy) in [metrics.md](../metrics.md) - [ ] PR-13: container pair deployed to canary; F1–F9 executed and recorded (F8's answer folded into the role) @@ -365,5 +386,5 @@ On a canary pair running a **deliberately crippled build** (Phase 1 path policy - [ ] `Set` with `dest:"file:/proc/1/root/root/.ssh/authorized_keys"` applies (validation stubbed) → the write fails at the kernel — no authorized_keys file appears on the host; same for every other path primitive - [ ] a baseline learn lands in `/var/lib/xtcp2` via the writer (owner `65534:xtcp2-state`, `0640`); an ASN refresh rotates the cache + previous files via the writer - [ ] killing the writer degrades, never breaks: collector keeps collecting, `Xtcp2WriterDown` fires, everything recovers on restart -- [ ] the writer protocol surface is exactly the 4 ops: a fuzzer against the socket (1 h, dumb mode) produces no write outside the two targets, no crash, no OOM, no leaked fds +- [ ] the writer's surface is exactly the 4 RPCs (a descriptor test fails on a new one): a fuzzer against the socket (1 h) produces no write outside the two targets, no crash, no OOM, no leaked fds - [ ] alert runbook covers: writer down, sidecar init on fresh hosts, socket volume loss (recreate + restart pair) diff --git a/docs/security-hardening/phase-5-privilege-brokers.md b/docs/security-hardening/phase-5-privilege-brokers.md index 3ce25f5e..12dc971e 100644 --- a/docs/security-hardening/phase-5-privilege-brokers.md +++ b/docs/security-hardening/phase-5-privilege-brokers.md @@ -4,8 +4,9 @@ |---|---| | **Status** | Not started — checkboxes track execution | | **Tracks** | [Implementation plan](implementation-plan.md) Phase 5 → [hardening plan](plan-configservice-set-hardening.md) workstreams W10 (netlink broker), W11 (Docker broker), W12 (syscall profiles) → [advisory](security-vulnerability-configservice-set-fs-write.md) layer L7 | -| **Goal** | The collector — everything that parses netlink and parquet, serves gRPC and Prometheus, and uploads to S3 — runs as a dedicated non-root uid with **zero capabilities, no `docker.sock`, and no namespace handles**. The two privileged needs move into two single-purpose brokers whose APIs can express nothing else, and each request is enforced three times: protovalidate rules on both ends, broker-side filtering against broker-owned state, and a kernel syscall profile on the broker | -| **Prereqs** | None hard. PR-19 (syscall profiles) ships first, against today's single-process collector. Reuses `pkg/sandbox` (Phase 2) and the peer-auth and both-ends-validation patterns of the writer (Phase 3) | +| **Goal** | The collector — everything that parses netlink and parquet, serves gRPC and Prometheus, and uploads to S3 — runs as a dedicated non-root uid with **zero capabilities, no `docker.sock`, no namespace handles, and no netlink socket in any tenant netns**. The two privileged needs move into two single-purpose brokers whose APIs can express nothing else. Every netlink message xtcp2 sends into a namespace is built by the netlink broker from a typed request; the high-volume replies flow back unfiltered. Each request is enforced three times: protovalidate rules on both ends, broker-side filtering against broker-owned state, and a kernel syscall profile on the broker | +| **Prereqs** | None hard. PR-19 (syscall profiles) ships first, against today's single-process collector. Reuses `pkg/sandbox` (Phase 2) and `pkg/ipc` ([Phase 3](phase-3-writer-split.md#wire-format-grpc-over-uds-pkgipc): gRPC over UDS, peer uid checked in the handshake, validation on both ends) | +| **Scope** | The container deployment. Bare metal is unsupported for this design: [PR-24](phase-1-code-hardening.md#pr-24--startup-confinement-check) refuses to start unconfined unless the operator waives named checks, and a waived host is the operator's risk | ## Contents @@ -37,60 +38,65 @@ Phases 1–4 close the `Set` primitives and confine the collector's filesystem a Verified against kernel source (`torvalds/linux` master). PR-18's spike re-verifies each on fleet kernels; the P-table pins them as tests. -1. **Netlink privilege checks need the opener *and* the sender.** `__netlink_ns_capable` (`net/netlink/af_netlink.c`) returns `file_ns_capable(, cap) && ns_capable(, cap)`. A netlink socket opened by a privileged broker and then used by a capability-less collector therefore behaves exactly like an unprivileged socket: dumps work, and every `CAP_NET_ADMIN`-gated operation fails. **Privilege does not travel with the fd** — this is what makes fd passing safe. -2. **inet_diag dumps are unprivileged.** `CAP_NET_ADMIN` only adds `INET_DIAG_MARK`, TCP-MD5 key material (`tcp_diag_get_aux` → `tcp_diag_put_md5sig` copies `tcpm_key`), ULP details, and mark conditions in bytecode filters (`net/ipv4/inet_diag.c`, `net/ipv4/tcp_diag.c`). `SOCK_DESTROY` checks `ns_capable(current, CAP_NET_ADMIN)` (`net/core/sock_diag.c`). `xtcp_flat_record` has no mark or MD5 field, so the collector loses nothing it records. It also gains something: it can no longer read MD5 keys or kill sockets. The startup check's claim that xtcp2 "cannot read any TCP socket data without" `CAP_NET_ADMIN` (`pkg/xtcp/init_capabilities.go:36`) contradicts this; PR-18 settles it attribute by attribute. -3. **A socket's netns is fixed at creation** (`sock_net(sk)`). An fd created inside a target netns dumps that netns, whichever netns the process using it is in. -4. **rtnetlink mutations** (`RTM_NEW*`/`RTM_DEL*`) are gated by `netlink_net_capable` (fact 1). A broker-opened `NETLINK_ROUTE` fd lets the collector dump addresses and routes, never change them. -5. **gRPC cannot carry file descriptors.** `SCM_RIGHTS` is ancillary data on a unix socket, and HTTP/2 framing has no place for it. So the netlink broker speaks proto messages over `SOCK_SEQPACKET`; the Docker broker, which returns only data, is a gRPC service. +1. **Netlink privilege checks need the opener *and* the sender.** `__netlink_ns_capable` (`net/netlink/af_netlink.c`) returns `file_ns_capable(, cap) && ns_capable(, cap)`. A sender without `CAP_NET_ADMIN` is limited to unprivileged requests, whoever opened the socket. The netlink broker holds no `CAP_NET_ADMIN`, so this is the kernel backstop under the broker's own request filter. +2. **inet_diag dumps are unprivileged.** `CAP_NET_ADMIN` only adds `INET_DIAG_MARK`, TCP-MD5 key material (`tcp_diag_get_aux` → `tcp_diag_put_md5sig` copies `tcpm_key`), ULP details, and mark conditions in bytecode filters (`net/ipv4/inet_diag.c`, `net/ipv4/tcp_diag.c`). `SOCK_DESTROY` checks `ns_capable(current, CAP_NET_ADMIN)` (`net/core/sock_diag.c`). `xtcp_flat_record` has no mark or MD5 field, so nothing it records is lost, and MD5 keys and socket kills leave reach. The startup check's claim that xtcp2 "cannot read any TCP socket data without" `CAP_NET_ADMIN` (`pkg/xtcp/init_capabilities.go:36`) contradicts this; PR-18 settles it attribute by attribute. +3. **A socket's netns is fixed at creation** (`sock_net(sk)`). A socket created inside a target netns dumps that netns, whichever netns the process using it is in. The broker keeps one socket per (netns, protocol) and never stays inside a tenant netns. +4. **rtnetlink mutations** (`RTM_NEW*`/`RTM_DEL*`, every non-GET kind) are gated by `netlink_net_capable(skb, CAP_NET_ADMIN)` in `rtnetlink_rcv_msg` (fact 1). Without `CAP_NET_ADMIN`, only dumps and multicast joins succeed. +5. **A netlink socket cannot be made send-only or receive-only, and dump replies go only to the socket that sent the request.** Netlink implements no `shutdown` (`sock_no_shutdown`), seccomp cannot tell one fd from another, and the kernel unicasts a dump to the requester's port id. So whichever process reads the replies can also send anything on that socket. **Filtering what xtcp2 sends therefore means the broker both sends and reads, and relays the reply bytes.** Passing the fd to the collector would leave its sends filtered by the kernel alone. ## Target topology +All three links are gRPC over a unix socket, on the shared `pkg/ipc` stack ([Phase 3](phase-3-writer-split.md#wire-format-grpc-over-uds-pkgipc)). + ``` ┌────────────────────────────────────────────┐ │ xtcp2 collector │ │ uid xtcp2 (non-root), CapEff 0 │ │ host netns: listeners 9088/8889/9089, S3 │ - │ no docker.sock, no netns mounts │ + │ no docker.sock, no netns mounts, │ + │ no SOCK_DIAG socket, no setns │ └──┬───────────────┬───────────────┬──────────┘ - SOCK_SEQPACKET │ gRPC / UDS │ framed UDS │ - proto + SCM_RIGHTS │ │ (Phase 3) │ + gRPC / UDS │ gRPC / UDS │ gRPC / UDS │ + typed requests ↓ │ │ (Phase 3) │ + raw replies ↑ │ │ │ ┌───────────────────────▼──┐ ┌──────────▼──────────┐ ┌──▼─────────────────┐ │ xtcp2 -nlbroker │ │ xtcp2 -dockerbroker │ │ xtcp2 -writer │ │ uid 0, CAP_SYS_ADMIN only│ │ uid xtcp2-docker + │ │ uid xtcp2-writer │ -│ seccomp: setns(NEWNET), │ │ docker group, Cap 0 │ │ Cap 0, no network │ -│ socket(NETLINK, │ │ network_mode none │ │ state dir rw │ -│ SOCK_DIAG|ROUTE) only │ │ docker.sock, 4 fixed│ │ │ +│ builds + sends every │ │ docker group, Cap 0 │ │ Cap 0, no network │ +│ netlink request; relays │ │ network_mode none │ │ state dir rw │ +│ replies unfiltered │ │ docker.sock, 4 fixed│ │ │ │ /run/netns, docker/netns │ │ GETs, projection │ │ │ └──────────────────────────┘ └─────────────────────┘ └────────────────────┘ ``` | Process | uid | Capabilities | Network | Mounts | Listens on | |---|---|---|---|---|---| -| collector | dedicated `xtcp2` (non-root, not 65534) | **none** | host netns, so it can bind and connect but not configure | state dir `ro`, broker socket dirs, token and secret files (group `xtcp2`, `0440`), `/sys/fs/cgroup:ro` | 9088, 8889, 9089 (all ≥ 1024, no capability needed) | -| netlink broker | 0 | `CAP_SYS_ADMIN` only (+ `CAP_SYS_PTRACE` only for bare-metal `/proc` discovery) | host netns, which is its handle to the host namespace; seccomp forbids `AF_INET*` sockets | `/run/netns:ro`, `/run/docker/netns:ro` (rslave), its socket dir | `/run/xtcp2/nlbroker.sock` only | -| Docker broker | dedicated `xtcp2-docker`, `group_add` docker gid | none | `network_mode: none` | `docker.sock`, `/run/docker/netns:ro` (stat only), its socket dir | `/run/xtcp2/dockerbroker.sock` only | -| writer | dedicated `xtcp2-writer` (moved off 65534, which the internet-facing ipfeed sidecar also uses) | none | `network_mode: none` | state dir `rw` | per [Phase 3](phase-3-writer-split.md) | +| collector | dedicated `xtcp2` (non-root, not 65534) | **none** | host netns, so it can bind and connect but not configure. Netlink: no `SOCK_DIAG` socket and no tenant netns; host-netns `ROUTE`/`GENERIC`/`RDMA` sockets for the link monitor and `rtnlmon` until those are brokered ([decision](#decisions-to-close-during-phase-5)) | state dir `ro`, broker socket dirs, token and secret files (group `xtcp2`, `0440`), `/sys/fs/cgroup:ro` | 9088, 8889, 9089 (all ≥ 1024, no capability needed) | +| netlink broker | 0 | `CAP_SYS_ADMIN` only | host netns, which is its handle to the host namespace; seccomp forbids `AF_INET*` sockets | `/run/netns:ro`, `/run/docker/netns:ro` (rslave), its socket dir | `/run/xtcp2/nlbroker.sock` (gRPC) only | +| Docker broker | dedicated `xtcp2-docker`, `group_add` docker gid | none | `network_mode: none` | `docker.sock`, `/run/docker/netns:ro` (stat only), its socket dir | `/run/xtcp2/dockerbroker.sock` (gRPC) only | +| writer | dedicated `xtcp2-writer` (moved off 65534, which the internet-facing ipfeed sidecar also uses) | none | `network_mode: none` | state dir `rw` | `/run/xtcp2/writer.sock` (gRPC), per [Phase 3](phase-3-writer-split.md) | **Why two brokers, not one.** Membership in the docker group is host-root-equivalent; `CAP_SYS_ADMIN` limited by seccomp to `setns(CLONE_NEWNET)` is not. A combined broker would hand one compromise both. Each broker holds exactly one privilege and has its own API, profile and alert. ## Design rules for every broker -1. **Typed, verb-specific APIs; never a pass-through.** No request field carries a path, URL, HTTP method, raw netlink message, socket family, or syscall argument chosen by the client. The broker builds every privileged call from compiled-in constants plus identifiers it issued itself. +1. **Typed, verb-specific APIs; never a pass-through.** No request field carries a path, URL, HTTP method, raw netlink message, socket family, or syscall argument chosen by the client. The broker builds every privileged call, and every byte it sends to the kernel or dockerd, from compiled-in constants plus validated fields and identifiers it issued itself. 2. **Three independent enforcement layers per request:** - - *validators*: protovalidate on both ends — the client before send, the broker after decode; unknown fields rejected; + - *validators*: protovalidate on both ends — the client before send, the broker after decode; unknown fields rejected. Both run in the shared `pkg/ipc` interceptor chain ([Phase 3](phase-3-writer-split.md#wire-format-grpc-over-uds-pkgipc)), so every service gets them identically; - *broker-side filtering*: semantic checks against broker-owned state (discovery tables, indexes), rate limits, connection limits; - *kernel*: a seccomp allowlist and a Landlock ruleset on the broker, so a bug in the first two layers still cannot exceed the API's intent. -3. **Identifiers, not handles.** Clients name objects by broker-issued ids: a netns by `(dev, inode)` from the broker's own discovery, a container by netns inode from the broker's own index. An unknown id returns `NOT_FOUND` before any privileged call. +3. **Identifiers, not handles.** Clients name objects by broker-issued ids: a netns by its cookie (`SO_NETNS_COOKIE`, never reused by the kernel, unlike inode numbers) from the broker's own discovery, a container by netns inode from the broker's own index. An unknown id returns `NOT_FOUND` before any privileged call. 4. **Responses are projections.** Only fields with a consumer cross the boundary. Outbound messages are validated too: tenant-controlled strings such as container names get charset rules, and a failing entry is dropped, not repaired. 5. **Peer authentication is meaningful because the collector is no longer root.** - - `SO_PEERCRED` uid must equal the dedicated `xtcp2` uid. Once the collector is non-root, that uid names exactly one process image; uid 0 never did. + - `SO_PEERCRED` uid must equal the dedicated `xtcp2` uid, checked by `pkg/ipc`'s transport credentials during the handshake, before any HTTP/2 frame is parsed. Once the collector is non-root, that uid names exactly one process image; uid 0 never did. + - The collector checks the broker's uid the same way (mutual), so a process that wins a race for the socket path is refused. - Socket dir `0750` and socket `0660`, group `xtcp2-broker`. - Optional second factor: `SO_PEERSEC` AppArmor label `xtcp2-collector` (verify unix-socket label support on fleet kernels). Separate PID namespaces rule out pid- or cgroup-based checks below `PIDFD_INFO_CGROUPID` (6.13). 6. **Fail closed in the brokers, degrade-not-exit in the collector.** - Broker down: the affected enrichment or namespace set goes stale, a counter rises, and an alert fires. - - The collector never falls back to holding the privilege itself in the fleet. Direct mode exists only when no broker socket is configured, for bare `xtcp2` runs. + - The collector never falls back to holding the privilege itself in the fleet. Direct mode exists only when no broker socket is configured, and only in builds with the `netns_direct` tag; the fleet image is built without it, so a broker misconfiguration cannot fall back to the privileged path. 7. **One admission checklist for new ops**, mirroring the writer's ([Phase 3 Appendix A, C](phase-3-writer-split.md#appendix-a--the-write-inventory-and-the-target-wish-list)): - 1. a new oneof arm or RPC with validators; + 1. a new RPC with validators; 2. a compiled-in mapping to exactly one privileged call; 3. a seccomp and Landlock delta; 4. a test-table block; @@ -103,15 +109,19 @@ Verified against kernel source (`torvalds/linux` master). PR-18's spike re-verif ### What it replaces -Today the collector does the privileged work itself. -- **Per-netns readers** (`netNamespaceInstance`, `pkg/xtcp/ns_net_namespace.go`): lock an OS thread, `open` the netns handle, `setns(fd, CLONE_NEWNET)`, `socket(AF_NETLINK, SOCK_DGRAM, NETLINK_INET_DIAG)`, then restore or terminate the thread. -- **The locality enricher** (`nsLocalitySnapshot`, `pkg/xtcp/enrich_locality.go:408`): does the same to dump addresses and routes. +Today the collector enters every namespace, and builds and sends every netlink request itself: +- **Per-netns socket dumps** (`netNamespaceInstance`, `pkg/xtcp/ns_net_namespace.go`): lock an OS thread, `open` the netns handle, `setns(fd, CLONE_NEWNET)`, `socket(AF_NETLINK, SOCK_DGRAM, NETLINK_INET_DIAG)`, then restore or terminate the thread. The poller sends a `SOCK_DIAG_BY_FAMILY` dump (`xtcpnl.BuildNetlinkSockDiagRequest`, sent at `pkg/xtcp/poller.go:447`), and `netlinkReceiver` (`pkg/xtcp/netlinker.go`) or the io_uring path reads the replies onto `packetCh` for the decoders. +- **The locality enricher** (`nsLocalitySnapshot`, `pkg/xtcp/enrich_locality.go`): the same entry dance on a `NETLINK_ROUTE` socket, then `RTM_GETLINK`, `RTM_GETADDR` and `RTM_GETROUTE` dumps (`enrich_locality.go:486–520`). +- **Discovery** (`ns_discover.go`, `pkg/nsdiscover`). -After the split, both call sites become one broker request that returns a socket fd. The collector keeps its zero-copy hot path — it reads the same netlink socket it reads today — and loses `setns`, the thread-tainting dance, and both capabilities. +Those four dump types are everything xtcp2 sends into a namespace. The `inet_diag` request is the `ss --tcp --info` request: TCP, one address family, a state mask and an extension mask. -### Protocol — proto over `SOCK_SEQPACKET` +After the split: +- **Sends are filtered.** The collector sends nothing to the kernel over netlink in any namespace. It makes a typed gRPC call; the broker builds the request from compiled-in constants plus validated fields, sends it, and is the only process that writes to a netlink socket in any namespace. The host netns's socket dumps go through the broker too. +- **Replies are not filtered.** They are the high-volume direction and carry no risk to the kernel. The broker relays them unchanged on a server stream, and the collector feeds them to today's decoders. +- The collector loses `setns`, the thread-tainting dance, every per-netns socket, and both capabilities. -`SOCK_SEQPACKET` preserves message boundaries, so one datagram is one request and no length prefix needs parsing. A successful `OpenSocket` reply carries **exactly one** fd in `SCM_RIGHTS`; every other reply carries none. The schema is message-only (no `service`), like the writer's: +### gRPC service ```proto // proto/xtcp_nlbroker/v1/nlbroker.proto @@ -119,10 +129,13 @@ syntax = "proto3"; package xtcp_nlbroker.v1; import "buf/validate/validate.proto"; -enum NetlinkProtocol { - NETLINK_PROTOCOL_UNSPECIFIED = 0; - NETLINK_PROTOCOL_SOCK_DIAG = 1; // NETLINK_SOCK_DIAG (4): inet_diag dumps - NETLINK_PROTOCOL_ROUTE = 2; // NETLINK_ROUTE (0): address/route dumps (locality) +service NetlinkBrokerService { + // Network namespaces the broker discovered. Paginated, pinned to a generation. + rpc ListNetns(ListNetnsRequest) returns (ListNetnsResponse); + // The inet_diag dump (`ss --tcp --info`) in one netns. + rpc DumpSockets(DumpSocketsRequest) returns (stream NetlinkChunk); + // An rtnetlink link, address or route dump in one netns (locality). + rpc DumpRoute(DumpRouteRequest) returns (stream NetlinkChunk); } enum NetnsSource { @@ -130,19 +143,18 @@ enum NetnsSource { NETNS_SOURCE_HOST = 1; // the broker's own (host) netns NETNS_SOURCE_RUN_NETNS = 2; // /run/netns/ NETNS_SOURCE_DOCKER = 3; // /run/docker/netns/ - NETNS_SOURCE_PROC = 4; // /proc//ns/net (bare metal only) } -// Broker-issued identity. The client never sends a path. +// Broker-issued identity: the netns cookie. The client never sends a path. message NetnsId { - uint64 dev = 1 [(buf.validate.field).uint64.gt = 0]; - uint64 inode = 2 [(buf.validate.field).uint64.gt = 0]; + uint64 cookie = 1 [(buf.validate.field).uint64.gt = 0]; } message Netns { NetnsId id = 1 [(buf.validate.field).required = true]; - NetnsSource source = 2 [(buf.validate.field).enum = {defined_only: true, not_in: [0]}]; - string name = 3 [(buf.validate.field).string = {max_len: 255, pattern: "^[A-Za-z0-9._-]*$"}]; + uint64 inode = 2 [(buf.validate.field).uint64.gt = 0]; // joins the Docker broker's netns_inode + NetnsSource source = 3 [(buf.validate.field).enum = {defined_only: true, not_in: [0]}]; + string name = 4 [(buf.validate.field).string = {max_len: 255, pattern: "^[A-Za-z0-9._-]*$"}]; } message ListNetnsRequest { @@ -150,86 +162,104 @@ message ListNetnsRequest { uint32 offset = 2 [(buf.validate.field).uint32.lte = 65535]; } message ListNetnsResponse { - repeated Netns netns = 1 [(buf.validate.field).repeated.max_items = 256]; // one datagram ≪ sk_sndbuf + repeated Netns netns = 1 [(buf.validate.field).repeated.max_items = 1024]; uint64 generation = 2; bool more = 3; } -message OpenSocketRequest { - NetnsId netns = 1 [(buf.validate.field).required = true]; - NetlinkProtocol protocol = 2 [(buf.validate.field).enum = {defined_only: true, in: [1, 2]}]; -} -message OpenSocketResponse { // the fd travels in SCM_RIGHTS - NetnsId netns = 1; - NetlinkProtocol protocol = 2; +enum AddressFamily { + ADDRESS_FAMILY_UNSPECIFIED = 0; + ADDRESS_FAMILY_INET = 1; // AF_INET + ADDRESS_FAMILY_INET6 = 2; // AF_INET6 } -message PingRequest {} -message PingResponse {} -message StatsRequest {} -message StatsResponse { - map counters = 1 [(buf.validate.field).map.max_pairs = 64]; +// Built as struct inet_diag_req_v2 with sdiag_protocol = IPPROTO_TCP, a zeroed +// inet_diag_sockid and no attributes. There is no protocol, socket-id or +// bytecode field to set. +message DumpSocketsRequest { + NetnsId netns = 1 [(buf.validate.field).required = true]; + AddressFamily family = 2 [(buf.validate.field).enum = {defined_only: true, not_in: [0]}]; + uint32 states = 3 [(buf.validate.field).uint32 = {gt: 0, lte: 16382}]; // bits 1..13: TCP_ESTABLISHED..TCP_BOUND_INACTIVE + uint32 extensions = 4 [(buf.validate.field).uint32.lte = 255]; // idiag_ext: the 8 requestable INET_DIAG_* reads + uint32 seq = 5; // copied into nlmsg_seq; opaque to the kernel } -message Error { - enum Code { CODE_UNSPECIFIED = 0; CODE_INVALID = 1; CODE_NOT_FOUND = 2; - CODE_RESOURCE_EXHAUSTED = 3; CODE_ABORTED = 4; CODE_INTERNAL = 5; } - Code code = 1 [(buf.validate.field).enum = {defined_only: true, not_in: [0]}]; - string detail = 2 [(buf.validate.field).string.max_len = 256]; +enum RouteDump { + ROUTE_DUMP_UNSPECIFIED = 0; + ROUTE_DUMP_LINKS = 1; // RTM_GETLINK + ROUTE_DUMP_ADDRS = 2; // RTM_GETADDR + ROUTE_DUMP_ROUTES = 3; // RTM_GETROUTE } -message Request { - oneof op { - option (buf.validate.oneof).required = true; - PingRequest ping = 1; - ListNetnsRequest list_netns = 2; - OpenSocketRequest open_socket = 3; - StatsRequest stats = 4; - } +// Built as a bare dump request (ifinfomsg, ifaddrmsg or rtmsg) with +// AF_UNSPEC and no attributes. +message DumpRouteRequest { + NetnsId netns = 1 [(buf.validate.field).required = true]; + RouteDump dump = 2 [(buf.validate.field).enum = {defined_only: true, not_in: [0]}]; + uint32 seq = 3; } -message Response { - oneof result { - option (buf.validate.oneof).required = true; - PingResponse ping = 1; - ListNetnsResponse list_netns = 2; - OpenSocketResponse open_socket = 3; - StatsResponse stats = 4; - Error error = 5; - } + +// The reply format. data is one or more whole netlink messages, exactly as +// recvmsg returned them, concatenated. The broker never splits, edits or +// filters a message. The last chunk ends with NLMSG_DONE. +message NetlinkChunk { + bytes data = 1; } ``` What the schema leaves out is as important as what it includes: -- no socket type, family or protocol number: the broker always uses `AF_NETLINK`/`SOCK_DGRAM|SOCK_CLOEXEC`, and the enum maps to exactly two protocol numbers; -- no netns path; -- no generic-netlink family; -- no socket options. +- no netlink message type, flags or protocol number: each RPC maps to exactly one message type, and `DumpSockets` always asks for TCP; +- no socket id, so no single-socket lookups; +- no attributes, so no `INET_DIAG_REQ_BYTECODE` — the kernel's bytecode auditor has a CVE history, and the typed API makes it unreachable; +- no netns path, generic-netlink family or socket option. -### Broker-side filtering (beyond the validators) - -Receive pipeline, fail-closed, each stage counted in `xtcp2_nlbroker_requests_total{op,stage}`: +`seq` is the one client-chosen value that reaches the kernel: it is copied into `nlmsg_seq`, which the kernel only echoes. It keeps the collector's decoders unchanged, because the rtnetlink reader drops replies with another seq (`xtcpnl_rtnetlink.go:393`). -1. **Connection gate:** `SO_PEERCRED` uid == `-nlBrokerPeerUid`; at most one client connection (the collector), and a second `accept` is closed immediately; datagrams larger than 4 KiB arrive `MSG_TRUNC` and are treated as `malformed`. -2. **Decode:** `proto.Unmarshal`, then `GetUnknown() == nil` on every message (unknown fields rejected, not stored) → `malformed`. -3. **Validate:** protovalidate `Validate(req)` → `invalid`. -4. **Resolve:** `(dev, inode)` must be in the broker's **current** discovery table, which the broker builds from the host netns, `/run/netns`, `/run/docker/netns` and, on bare metal, the `/proc` scan. A miss returns `not_found` and makes no syscall. -5. **Re-verify the handle (TOCTOU):** the broker `open`s the path from **its own** table entry, `fstat`s the fd, and requires `(dev, inode)` to still match. It also requires `ioctl(NS_GET_NSTYPE) == CLONE_NEWNET`, so a non-network nsfs file planted under `/run/netns` is never entered. On mismatch it returns `not_found` and increments the `race` counter. -6. **Rate limit:** `OpenSocket` ≤ `-nlBrokerOpenRate` (default 100/s, burst 1000 — container churn) and `ListNetns` ≤ 20/s; excess returns `resource_exhausted`. -7. **Create:** on a dedicated locked OS thread: `setns(nsfd, CLONE_NEWNET)` → `socket(AF_NETLINK, SOCK_DGRAM|SOCK_CLOEXEC, proto)` → restore the original netns, or terminate the thread if the restore fails (the existing tainted-M rule from `ns_net_namespace.go` moves here unchanged). -8. **Post-check before sending:** `getsockopt(SO_DOMAIN) == AF_NETLINK` and `SO_PROTOCOL` == the requested protocol; then `sendmsg` with one `SCM_RIGHTS` fd, close the broker's copy, and count `{protocol, source, ok}`. +### Broker-side filtering (beyond the validators) -Pagination (`ListNetns`): a listing is pinned to a `generation`. If the table changes mid-listing the broker replies `aborted`, and the collector restarts at offset 0. +Each request runs this pipeline, fail-closed, each stage counted in `xtcp2_nlbroker_requests_total{rpc,stage}`: + +1. **Connection gate** (`pkg/ipc`): the peer uid from `SO_PEERCRED` must equal `-nlBrokerPeerUid`, checked in the handshake before any HTTP/2 frame is parsed; at most one client connection. +2. **Decode:** unknown fields rejected → `InvalidArgument`. +3. **Validate:** protovalidate → `InvalidArgument`. +4. **Resolve:** the cookie must be in the broker's **current** discovery table, which the broker builds from the host netns, `/run/netns` and `/run/docker/netns`. A miss returns `NotFound` and makes no syscall. +5. **Limit:** one dump in flight per (netns, protocol), since a netlink socket runs one dump at a time; at most `-nlBrokerMaxDumps` concurrent dumps (default 64); a per-RPC rate limit. Excess returns `ResourceExhausted`. +6. **Socket:** the broker's own cached socket for (netns, protocol). It is created on a dedicated locked OS thread: + - `open` the path from **its own** table entry, `fstat` it, and require `(dev, inode)` to match the table; + - require `ioctl(NS_GET_NSTYPE) == CLONE_NEWNET`, so a non-network nsfs file planted under `/run/netns` is never entered; + - `setns(nsfd, CLONE_NEWNET)` → `socket(AF_NETLINK, SOCK_DGRAM|SOCK_CLOEXEC, proto)` → restore the original netns, or terminate the thread if the restore fails (the tainted-M rule from `ns_net_namespace.go` moves here unchanged); + - require `SO_PROTOCOL` == the protocol and `SO_NETNS_COOKIE` == the table's cookie. + + Any mismatch returns `NotFound` and increments the `race` counter. +7. **Build:** the request bytes come from constants and validated fields only: the message type fixed by the RPC, flags `NLM_F_REQUEST|NLM_F_DUMP`, `nlmsg_pid` 0, a zeroed socket id, no attributes. Golden-byte tests pin every combination (Q2). +8. **Send and relay:** + - `sendto` the kernel, then `recvmsg` until the dump ends; + - coalesce whole datagrams into chunks of up to 256 KiB and send each as one `NetlinkChunk`; + - read only the `nlmsghdr` headers: lengths must tile each datagram (otherwise close the socket and return `Internal`), a message with another seq is dropped and counted, and the stream ends after `NLMSG_DONE`; `NLMSG_ERROR` becomes a gRPC status carrying the errno; + - a per-dump byte cap (`-nlBrokerMaxDumpBytes`, default 1 GiB) and deadline bound a runaway dump; + - a cancelled or broken stream closes that socket, because a half-read dump cannot be resumed; the next dump recreates it. + +Pagination (`ListNetns`): a listing is pinned to a `generation`. If the table changes mid-listing the broker returns `Aborted`, and the collector restarts at offset 0. + +### Reply path: simple framing, one copy + +- **Format:** a `NetlinkChunk` is one `bytes` field. On the wire that is a one-byte tag, a varint length, then the kernel's bytes, inside gRPC's 5-byte message frame. +- **Collector side:** a stream receiver replaces `netlinkReceiver` and puts each chunk on the existing `packetCh` in place of a `recvmsg` buffer. Chunks hold whole messages, so the decoders, `packetHasDone` and the dump tracker are unchanged. +- **Cost against today:** one extra copy (broker buffer → gRPC) plus HTTP/2 framing. In broker mode the collector's io_uring receive path does not apply. +- **Optional:** a custom gRPC codec for `NetlinkChunk` on the collector side that slices `data` out of the receive buffer instead of protobuf-go's copying unmarshal. Add it only if P12 shows the decode copy matters. +- **Gate (P12, in the PR-18 spike):** a benchmark against today's direct and io_uring paths at fleet-like scale. A miss selects the fallback: the broker `recvmsg`s straight into a sealed memfd ring the collector maps read-only, and gRPC carries only offsets and lengths. That is the same single copy as today, and the collector still holds no netlink socket. ### Kernel enforcement on the netlink broker - **seccomp (allowlist; the noteworthy rules):** - `setns`: arg1 `== CLONE_NEWNET` (`SCMP_CMP_EQ 0x40000000`). `nstype 0` means "any type" and is therefore **denied**. - - `socket`: (`AF_NETLINK`, `type & 0xf == SOCK_DGRAM`, `protocol ∈ {0, 4}`) or (`AF_UNIX`, `SOCK_SEQPACKET`, 0); everything else `EPERM`. + - `socket`: (`AF_NETLINK`, `type & 0xf == SOCK_DGRAM`, `protocol ∈ {0, 4}`) or (`AF_UNIX`, `SOCK_STREAM`) for its gRPC listener; everything else `EPERM`. + - `ioctl`: `NS_GET_NSTYPE` only, plus whatever the recorded profile shows the Go runtime needs. - `openat`: only with `(flags & (O_WRONLY|O_RDWR|O_CREAT|O_TRUNC)) == 0`. - `clone`: denied when any `CLONE_NEW*` bit is set; `clone3` returns `ENOSYS` (the Go runtime uses `clone`). - denied outright: `execve`/`execveat` (the broker never re-execs), the mount family (`mount`, `umount2`, `pivot_root`, `move_mount`, `open_tree`, `fsopen`/`fsconfig`/`fsmount`/`fspick`, `mount_setattr`), `bpf`, `perf_event_open`, `unshare`, `ptrace`, `process_vm_*`, `keyctl`/`add_key`/`request_key`, `io_uring_*`, `userfaultfd`, the `chmod`/`chown`/`*xattr` families. - **Landlock:** - - read-only `/run/netns`, `/run/docker/netns` (and `/proc` on bare metal); + - read-only `/run/netns` and `/run/docker/netns`; - `MAKE_SOCK` in its own socket dir; - no other write; - ABI ≥ 6: signal and abstract-UDS scoping. @@ -238,17 +268,18 @@ Pagination (`ListNetns`): a listing is pinned to a `generation`. If the table ch - `cap_drop: ALL` + `SYS_ADMIN`; - `read_only: true`; - `no-new-privileges`; - - `pids_limit: 256`, memory 64 MiB; + - `pids_limit: 256`, memory 128 MiB (it now buffers dump chunks); - no listeners besides its UDS. -**Residual if the broker itself is fully compromised.** Fact 1 plus the seccomp rules bound it. The attacker can enter network namespaces and open `SOCK_DIAG`/`ROUTE` sockets — that is, read socket tables, addresses and routes, which is exactly what the collector legitimately sees. It cannot modify routes or the firewall, kill sockets, read MD5 keys (no `CAP_NET_ADMIN`), load BPF, mount, open inet sockets, or exec. That is the accepted residual. +**Residual if the broker itself is fully compromised.** Facts 1 and 4 plus the seccomp rules bound it. The attacker can enter network namespaces and send any message on `SOCK_DIAG`/`ROUTE` sockets, but without `CAP_NET_ADMIN` the kernel admits only dumps: socket tables, links, addresses and routes, which is what the collector legitimately sees. It cannot modify routes or the firewall, kill sockets, read MD5 keys, load BPF, mount, open inet sockets, or exec. That is the accepted residual. ### Collector integration -- **Netns readers:** `attemptOpenAndSetns` plus the `syscall.Socket(… NETLINK_INET_DIAG)` in `netNamespaceInstance` collapse into `nlbroker.OpenSocket(ctx, id, SOCK_DIAG)`. The `LockOSThread`/`setns`/restore code leaves the collector entirely, and with it the tainted-M thread-exhaustion class. -- **Locality:** `nsLocalitySnapshot` → `OpenSocket(id, ROUTE)`, then the existing address/route dump runs on that fd. This needs a new fd-accepting dump seam in `xtcpnl`, because `localnet.BuildSnapshot` already takes parsed `AddrInfo`/`RouteInfo`. -- **Discovery:** `ns_discover.go` is replaced by `ListNetns` at the existing discovery cadence; an unchanged `generation` skips work. +- **Socket dumps:** in broker mode the collector holds no per-netns socket. The poller calls `DumpSockets` for each netns on each poll, and the stream receiver feeds `packetCh`. `attemptOpenAndSetns`, the `LockOSThread`/`setns`/restore code and the per-netns sockets leave the collector, and with them the tainted-M thread-exhaustion class. +- **Locality:** `nsLocalitySnapshot` → three `DumpRoute` calls whose chunks feed the existing link, address and route parsers; `localnet.BuildSnapshot` is unchanged. +- **Discovery:** `ns_discover.go` is replaced by `ListNetns` at the existing discovery cadence; an unchanged `generation` skips work. Container attribution joins on `Netns.inode`, as today. - **Mode:** `-nlBrokerSock ` set → broker mode. In broker mode, `CAP_NET_ADMIN`/`CAP_SYS_ADMIN` move from **fatal** to **unexpected**: their presence logs a warning and sets `xtcp2_collector_excess_caps 1`, so a role regression that re-adds them is visible. +- **Host-netns netlink that stays for now:** the link monitor (`pkg/linkmonitor`) and `pkg/rtnlmon` open `ROUTE`, `GENERIC` and `RDMA` sockets in the collector's own (host) netns. With `CapEff 0` the kernel admits only their dumps, GETs and multicast joins (fact 1). The broker-mode seccomp profile allows `AF_NETLINK` only for those three protocols and never `SOCK_DIAG`, and the collector has no `setns`, so it cannot reach a tenant netns at all. Moving their requests behind typed broker RPCs, so the collector's profile can deny `AF_NETLINK` entirely, is a [decision](#decisions-to-close-during-phase-5). ## The Docker broker @@ -322,14 +353,9 @@ The collector mirrors the index locally — `List` once, then `Watch` — so the - per-request timeouts. 3. **Outbound validation.** Every `Container` passes protovalidate before it enters the index. A tenant container whose name or image breaks the charset is **dropped**, not sanitized, and counted in `xtcp2_dockerbroker_rejected_total{field}`. Container names and images are tenant-controlled strings that end up in telemetry rows. 4. **Sandbox path discipline.** Host-network containers (no `SandboxKey`) are omitted. `SandboxKey` must `filepath.Clean` to a direct child of `/run/docker/netns/`, or it is never `stat`ed. Inode 0 or a stat error → omitted. -5. **gRPC server hardening** (the [Phase 3 checklist](phase-3-writer-split.md#wire-format-and-the-protocol-decision)): - - UDS only; +5. **gRPC server hardening:** the `pkg/ipc` defaults ([Phase 3](phase-3-writer-split.md#wire-format-grpc-over-uds-pkgipc)) — peer uid checked in the handshake, no reflection or health service, no compression codecs, keepalive enforcement, the shared interceptor chain — with these service settings: - `MaxRecvMsgSize` 4 KiB (requests are tiny); `MaxSendMsgSize` sized for 4096 containers; - - no reflection service, no health service (health is `-healthcheck` mode); - - no compression codecs registered; - - keepalive enforcement policy; - - `MaxConcurrentStreams` 4, and one `Watch` stream per peer; - - interceptors, in order: peer-credential check (a custom `TransportCredentials` exposing `SO_PEERCRED` as `AuthInfo`; grpc-go has no stock support), then a per-RPC rate limit, then PR-4's validating interceptor. + - `MaxConcurrentStreams` 4, and one `Watch` stream per peer. 6. **Watch backpressure.** Each stream has a bounded send queue; a slow consumer gets its stream reset (the collector re-lists) instead of unbounded broker memory. An event ring of the last 1024 generations backs `since_generation`. ### Kernel enforcement on the Docker broker @@ -362,15 +388,15 @@ Ships first (PR-19) and does not depend on the brokers. Applied to today's colle | Process | Notable allows | Notable denies | AppArmor | |---|---|---|---| | collector, v1 (today's single process) | `setns` only `CLONE_NEWNET`; `socket` `AF_NETLINK`/`AF_INET`/`AF_INET6`/`AF_UNIX`; `io_uring_*` only in the `-ioUring` profile variant | `bpf`, `perf_event_open`, mount family, `unshare`, `clone` with `CLONE_NEW*`, `ptrace`, `process_vm_*`, `keyctl` family, `userfaultfd`, `chmod`/`chown`/`xattr` families (the fleet has no unix listeners to chmod) | `xtcp2-collector`: deny mount, ptrace, `network packet`/`raw`, writes outside the state dir | -| collector, v2 (broker mode) | as v1 **minus `setns`** | v1 set + `setns` | same | +| collector, v2 (broker mode) | as v1 **minus `setns`**; `socket(AF_NETLINK)` only for `ROUTE`/`GENERIC`/`RDMA` (host-netns link monitor and `rtnlmon`) | v1 set + `setns` + `socket(AF_NETLINK, NETLINK_SOCK_DIAG)` | same | | netlink broker | [above](#kernel-enforcement-on-the-netlink-broker) | | `xtcp2-nlbroker` | | Docker broker | [above](#kernel-enforcement-on-the-docker-broker) | | `xtcp2-dockerbroker` | -| writer | `AF_UNIX` only; file ops in its target dirs | everything network; `setns`/mount/`bpf`/… | `xtcp2-writer` | +| writer | `AF_UNIX` (`SOCK_STREAM`, its gRPC listener) only; file ops in its target dirs | everything network; `setns`/mount/`bpf`/… | `xtcp2-writer` | - **Allowlists, generated and gated.** Each profile is generated from the full test matrix (microVM runs under a recording hook or `strace -f`), committed as Docker seccomp JSON, and enforced in CI. The suite runs under the profile with `SCMP_ACT_ERRNO` plus logging, so a new syscall fails CI rather than production. - **Delivery:** - containers: `security_opt: seccomp=.json` (runc applies it before exec, covering the whole process) plus `apparmor=`; - - bare metal: `SystemCallFilter=`/`RestrictNamespaces=net` in the unit, or an in-process filter installed at startup. + - bare metal (unsupported, best effort): `SystemCallFilter=`/`RestrictNamespaces=net` in the unit, or an in-process filter installed at startup. - **Apply once.** In-process filters stack across `execve` like Landlock, so the soft restart inherits the filter and must not re-install it ([Phase 2](phase-2-kernel-confinement.md#pr-8--landlock-self-confinement)). - **Regression guard.** The fleet-gate test asserts each container's `SecurityOpt` names its profile; `unconfined` or `docker-default` fails the gate. @@ -382,9 +408,9 @@ PR-18 sec/w10-spike ──▶ PR-20 sec/w10-nlbroker ──┐ PR-21 sec/w11-dockerbroker ┼──▶ PR-22 sec/w10-collector-deprivilege ──▶ PR-23 (deployment repo) ``` -- [ ] **PR-18 — spike:** verify the [kernel facts](#kernel-facts-the-design-rests-on) and run every collector feature with zero capabilities on broker-opened fds (Test P); record which link-monitor reads (ethtool `GET`, RDMA, devlink) need a typed broker op, if any. +- [ ] **PR-18 — spike:** verify the [kernel facts](#kernel-facts-the-design-rests-on) on fleet kernels; prototype the dump relay and run every collector feature against it with zero capabilities; run the P12 benchmark gate; inventory the link monitor's and `rtnlmon`'s netlink requests (ethtool `GET`, RDMA, devlink, rtnetlink) as candidate typed broker ops (Test P). - [ ] **PR-19 — syscall profiles:** v1 collector seccomp + AppArmor profiles, the CI gate, and the fleet-gate assertion (Tests R, T). -- [ ] **PR-20 — netlink broker:** `proto/xtcp_nlbroker/v1`, `xtcp2 -nlbroker` (`pkg/nlbroker`), the client, and the collector seams (Tests Q, R). +- [ ] **PR-20 — netlink broker:** `proto/xtcp_nlbroker/v1` (gRPC service), `xtcp2 -nlbroker` (`pkg/nlbroker` on `pkg/ipc`), the client, the collector's stream receiver and seams, and the `netns_direct` build tag (Tests Q, R). - [ ] **PR-21 — Docker broker:** `proto/xtcp_dockerbroker/v1`, `xtcp2 -dockerbroker` (`pkg/dockerbroker`, reusing `dockermeta`'s decoding behind the fixed request set), and the collector mirror (Tests S, T). - [ ] **PR-22 — collector de-privilege:** - broker mode as the fleet default; @@ -401,42 +427,54 @@ PR-18 sec/w10-spike ──▶ PR-20 sec/w10-nlbroker ──┐ Same conventions as Phases 2–4 (Cat: P positive, N negative, B boundary, C corner). P, R and T run in the child-process harness on Linux (skip-aware). Q and S are pure, using injected syscall seams and a fake dockerd. U is microVM, recorded. -### Test P — kernel facts and feature parity (`TestBrokerFdSemantics`, spike → permanent) +### Test P — kernel facts, relay parity and the benchmark gate (`TestBrokerRelay`, spike → permanent) | ID | Cat | Description | Input / setup | Expected outcome | |---|---|---|---|---| -| P1 | P | inet_diag dump on a broker-opened fd from a zero-cap process | broker (CAP_SYS_ADMIN) opens `SOCK_DIAG` in a test netns with 50 TCP sockets; child with `CapEff 0` dumps | all 50 sockets returned, `INET_DIAG_INFO` (tcp_info) present | -| P2 | N | `SOCK_DESTROY` through the same fd | zero-cap child sends `SOCK_DESTROY` for one socket | `EPERM`; the socket survives | -| P3 | N | route mutation on a broker-opened `ROUTE` fd | zero-cap child sends `RTM_NEWROUTE` | `EPERM`; routing table unchanged | -| P4 | P | address/route dump on a broker-opened `ROUTE` fd | locality snapshot | identical to the snapshot produced by today's `setns` path | -| P5 | C | TCP-MD5 socket in the target netns | socket with `TCP_MD5SIG` set | zero-cap dump carries no `INET_DIAG_MD5SIG`; the same dump with `CAP_NET_ADMIN` does — pins the confidentiality gain | -| P6 | B | socket mark | socket with `SO_MARK` | zero-cap dump has no `INET_DIAG_MARK`; golden-record diff shows no change (no mark field exists) | -| P7 | P | host-netns link-monitor reads with zero caps | rtnetlink link dumps, ethtool `GET`s, RDMA dumps | each `OK`, or listed in the spike report as needing a typed broker op | -| P8 | P | io_uring reader on a broker-opened fd | `-ioUring` path | works unchanged | -| P9 | P | fleet output parity | canary: broker mode vs direct mode, 24 h | identical record schemas; per-netns record counts within noise | -| P10 | C | fd outlives its netns | broker opens, netns deleted, collector dumps | dump returns empty or an error the reader already handles; no crash; fd closed on the netns-delete path | +| P1 | P | inet_diag dump without `CAP_NET_ADMIN` | broker with `CAP_SYS_ADMIN` only dumps a test netns holding 50 TCP sockets | all 50 sockets returned, `INET_DIAG_INFO` (tcp_info) present | +| P2 | N | `SOCK_DESTROY` from the broker | the broker process sends `SOCK_DESTROY` for one socket | `EPERM`; the socket survives — the kernel backstop under the broker's filter | +| P3 | N | route mutation from the broker | `RTM_NEWROUTE` on its `ROUTE` socket | `EPERM`; routing table unchanged | +| P4 | P | locality through the relay | three `DumpRoute` calls | snapshot identical to the one today's `setns` path produces | +| P5 | C | TCP-MD5 socket in the target netns | socket with `TCP_MD5SIG` set | no `INET_DIAG_MD5SIG` in the broker's dump; the same dump with `CAP_NET_ADMIN` has it — pins the confidentiality gain | +| P6 | B | socket mark | socket with `SO_MARK` | no `INET_DIAG_MARK`; golden-record diff shows no change (no mark field exists) | +| P7 | C | replies go only to the requester | two sockets in one netns; the first sends a dump | the second receives nothing — pins kernel fact 5 | +| P8 | N | no receive-only netlink socket | `shutdown(SHUT_WR)` on a netlink socket | `EOPNOTSUPP` — pins why the fd is not passed | +| P9 | P | relay is byte-exact | recording seam on the broker's `recvmsg`; a full dump | concatenated chunk bytes delivered to the collector == bytes `recvmsg` returned | +| P10 | P | fleet output parity | canary: broker mode vs direct mode, 24 h | identical record schemas; per-netns record counts within noise | +| P11 | C | netns deleted mid-dump | delete the netns while a dump streams | stream ends with an error the poller counts; broker socket closed; no crash | +| P12 | B | relay benchmark gate | 100 netns × 1 000 sockets plus a 50 000-socket host netns, at the fleet poll cadence; baseline: today's direct and io_uring paths | proposed gate: collector + broker CPU ≤ 1.3× today's collector CPU, and p99 poll completion within the poll interval. Results go in the spike report; a miss selects the shared-memory fallback | +| P13 | P | host-netns link-monitor and `rtnlmon` reads with zero caps | rtnetlink link dumps, ethtool `GET`s, RDMA dumps, multicast joins | each `OK`, or listed in the spike report as needing a typed broker op | ### Test Q — netlink broker validation and filtering (`TestNlBroker`, pure) +Injected syscall and netlink seams; the gRPC server runs on a real UDS. + | ID | Cat | Description | Input / setup | Expected outcome | |---|---|---|---|---| -| Q1 | P | open `SOCK_DIAG` in a discovered netns | valid `(dev, inode)` | reply `open_socket` with exactly one `SCM_RIGHTS` fd; `SO_PROTOCOL == 4` | -| Q2 | N | protocol unspecified | `protocol = 0` | `invalid` (`in: [1, 2]`); no syscall | -| Q3 | N | protocol not in the enum | `protocol = 99` | `invalid` (`defined_only`) | -| Q4 | N | netns not in the discovery table | unknown `(dev, inode)` | `not_found`; injected `setns`/`open` counters stay 0 | -| Q5 | B | zero ids | `inode = 0` | `invalid` (`gt 0`) | -| Q6 | C | netns file replaced between discovery and open | same name, new inode | `fstat` mismatch → `not_found`, counter `race` +1 | +| Q1 | P | `DumpSockets` in a discovered netns | valid cookie, `INET`, all states, `INFO` | a chunk stream ending in `NLMSG_DONE`; exactly one `sendto` | +| Q2 | P | golden request bytes | every RPC × family × dump kind | sent bytes equal the golden files: type fixed by the RPC, flags `REQUEST|DUMP`, pid 0, zeroed socket id, `sdiag_protocol` TCP, no attributes | +| Q3 | N | family unspecified or undefined | `family = 0`, `99` | `InvalidArgument`; zero `sendto` | +| Q4 | B | state and extension bounds | `states` 16 382 / 16 383 / 0; `extensions` 255 / 256 | 16 382 accepted, 16 383 and 0 rejected; 255 accepted, 256 rejected | +| Q5 | N | netns not in the discovery table | unknown cookie | `NotFound`; injected `open`/`setns`/`sendto` counters stay 0 | +| Q6 | C | netns file replaced between discovery and socket creation | same name, new inode | `fstat` mismatch → `NotFound`, `race` +1 | | Q7 | C | non-network nsfs file under `/run/netns` | bind of a UTS ns file | `NS_GET_NSTYPE != CLONE_NEWNET` → rejected; never `setns` | -| Q8 | N | unknown field in the envelope | field 99 set | `malformed` (`GetUnknown` check) | -| Q9 | N | empty oneof | `Request{}` | `invalid` (oneof required) | -| Q10 | B | oversized datagram | 4 KiB + 1 | `MSG_TRUNC` → `malformed`, connection closed | -| Q11 | N | wrong peer uid | injected cred uid 0 or 1234 | rejected at accept; no reply payload, no fd | -| Q12 | N | second concurrent client | two connections | second closed immediately; counter | -| Q13 | B | rate limit | burst + 1 opens | last one `resource_exhausted` | -| Q14 | P | thread hygiene under churn | 10 000 opens with netns create/delete churn | broker OS-thread count bounded (no tainted-M growth) | -| Q15 | C | listing across a generation change | table changes between page 1 and page 2 | page 2 `aborted`; restart at offset 0 yields a consistent list | -| Q16 | N | fd count on error replies | every error path | zero `SCM_RIGHTS` fds attached (cmsg count asserted) | -| Q17 | P | both-ends validation | client validates before send; mutate a bound field post-validate | broker-side `invalid` — the layering is observable | +| Q8 | C | cookie mismatch after socket creation | injected `SO_NETNS_COOKIE` differs from the table | `NotFound`, `race` +1, socket closed | +| Q9 | N | unknown field in a request | field 99 set | `InvalidArgument` | +| Q10 | B | request above `MaxRecvMsgSize` | 4 KiB + 1 | `ResourceExhausted` from gRPC | +| Q11 | N | wrong peer uid | injected cred uid 0 or 1234 | handshake fails; the HTTP/2 frame counter stays 0 | +| Q12 | N | second client connection | two connections | second closed; counter | +| Q13 | B | one dump per netns | two concurrent `DumpSockets` on one netns | second `ResourceExhausted` | +| Q14 | B | rate limit | burst + 1 calls | last one `ResourceExhausted` | +| Q15 | C | reply with a foreign seq | injected stale message mid-dump | dropped and counted; the rest of the stream intact | +| Q16 | N | malformed reply framing | `nlmsg_len` overruns the datagram | `Internal`; socket closed; the next dump recreates it | +| Q17 | B | per-dump byte cap | a dump larger than `-nlBrokerMaxDumpBytes` | `ResourceExhausted`; socket closed | +| Q18 | C | client cancels mid-dump | cancel after the first chunk | socket closed; the next dump on that netns succeeds on a new socket | +| Q19 | P | chunks never split a message | datagrams of varying sizes around the 256 KiB chunk size | every chunk is whole messages; their concatenation equals the `recvmsg` bytes | +| Q20 | C | `NLMSG_ERROR` reply | injected `-EINVAL` | stream ends with a gRPC status carrying the errno | +| Q21 | P | thread hygiene under churn | 10 000 socket creations with netns create/delete churn | broker OS-thread count bounded (no tainted-M growth) | +| Q22 | C | listing across a generation change | table changes between page 1 and page 2 | page 2 `Aborted`; restart at offset 0 yields a consistent list | +| Q23 | P | both-ends validation | client validates before send; mutate a bound field after validation | broker-side `InvalidArgument` — the layering is observable | +| Q24 | P | the surface is exactly three RPCs | service descriptor | `ListNetns`, `DumpSockets`, `DumpRoute`; a new RPC without a test-table row fails | ### Test R — netlink broker syscall profile (`TestNlBrokerProfile`, child harness) @@ -486,6 +524,7 @@ Same conventions as Phases 2–4 (Cat: P positive, N negative, B boundary, C cor | T4 | N | collector v1: BPF, perf, mount, non-net `setns` | each call | `EPERM` | | T5 | P | collector v1: the full feature suite | unit + microVM matrix under the profile | green; profile log shows zero denials | | T6 | N | collector v2: any `setns` | `setns(fd, CLONE_NEWNET)` | `EPERM` — broker mode removes the syscall entirely | +| T7 | N | collector v2: a `SOCK_DIAG` socket | `socket(AF_NETLINK, SOCK_DGRAM, NETLINK_SOCK_DIAG)` | `EPERM` — the collector cannot send a socket-diag request anywhere | ### Test U — deployment (microVM, recorded) @@ -495,7 +534,7 @@ Same conventions as Phases 2–4 (Cat: P positive, N negative, B boundary, C cor | U2 | P | monitoring parity | 20 containers, churn | per-netns socket records and container attribution identical to direct mode (P9's microVM twin) | | U3 | N | collector RCE simulation | test-build hook runs a probe list in-process | `nft` flush, `ip route add`, `bpf`, `setns`, `SOCK_DESTROY`, `connect(docker.sock)` all fail (`EPERM`/`ENOENT`) | | U4 | P | the egress rule holds against the collector | U3 plus an attempt to delete Phase 4's nftables rule | rule intact — closes the gap noted in [Phase 4](phase-4-egress-fleet-lockdown.md#definition-of-done) | -| U5 | N | netlink broker killed | `docker kill` | existing fds keep the readers running; new namespaces unmonitored; `Xtcp2NlBrokerDown` fires; recovers on restart | +| U5 | N | netlink broker killed | `docker kill` | polls fail and are counted as per-netns gaps; the collector keeps serving and uploading; `Xtcp2NlBrokerDown` fires; collection resumes on restart | | U6 | N | Docker broker killed | `docker kill` | attribution goes stale (flagged); `Xtcp2DockerBrokerDown` fires; recovers | | U7 | P | soft restart in broker mode | tuning `Set` | re-exec reconnects to both brokers; no capability needed at any point | | U8 | C | socket dir permissions | another uid tries both sockets | `EACCES` at connect | @@ -528,19 +567,21 @@ Alerts (vmalert + `promtool test rules`): | # | Decision | Recommendation | |---|---|---| -| new | netlink broker transport | proto over `SOCK_SEQPACKET` with `SCM_RIGHTS` — gRPC cannot pass fds, and a gRPC control call plus a side channel adds a correlation protocol for nothing | -| new | netlink broker model: pass fds vs run dumps itself | pass fds — zero-copy hot path unchanged, and kernel fact 1 strips privilege from the fd. A broker-executed op only for a `CAP_NET_ADMIN`-gated read PR-18 proves necessary, via the admission checklist | +| new | IPC between xtcp2 processes | gRPC over UDS for all of them, on `pkg/ipc` ([Phase 3](phase-3-writer-split.md#wire-format-grpc-over-uds-pkgipc)): typed services, protovalidate on both ends, one interceptor chain, and the peer uid checked before HTTP/2 parsing | +| new | netlink broker model: pass fds vs run the dumps itself | run the dumps. The broker builds and sends every request and relays the replies unfiltered. A passed fd lets its holder send anything on it (kernel fact 5), leaving the kernel as the only filter on what is sent | +| new | reply relay performance | a gRPC server stream of raw chunks. If P12 fails: a sealed memfd ring mapped read-only by the collector, with gRPC carrying offsets — the same single copy as today | +| new | link monitor and `rtnlmon` netlink in the collector | keep in Phase 5: host netns only, `CapEff 0` (the kernel admits only dumps, GETs and multicast joins) and no `SOCK_DIAG`. Follow-up: typed broker RPCs for the requests P13 inventories, so the collector's profile denies `AF_NETLINK` entirely | | new | Docker access: typed gRPC broker vs filtering HTTP proxy in front of `docker.sock` | typed broker — a proxy filters an HTTP surface by method/path patterns and must also rewrite inspect bodies to keep `Env` out; the typed API makes both properties structural | -| new | collector stays in the host netns? | yes for now (listeners, link monitor, S3). Later option: with every namespace reached through broker fds, the collector could run in its own netns, turning egress pinning into ordinary container policy | +| new | collector stays in the host netns? | yes for now (listeners, link monitor, `rtnlmon`, S3). Later option: once the link monitor and `rtnlmon` are brokered, the collector needs nothing from the host netns and could run in its own, turning egress pinning into ordinary container policy | | new | dockerd over TCP+mTLS with an authz plugin enforcing GET-only for the broker | deferred — strongest bound on the Docker broker's residual; revisit once the broker is in | -| new | bare-metal `/proc` discovery | `CAP_SYS_PTRACE` lives in the netlink broker only; the collector never holds it | +| new | bare metal | unsupported: PR-24 refuses to start unconfined unless waived. No process holds `CAP_SYS_PTRACE`, and discovery has no `/proc` source | | new | dedicated uids | `xtcp2`, `xtcp2-docker`, `xtcp2-writer` — none shared with each other or with the ipfeed sidecar's 65534 | ## Release checklist -- [ ] PR-18 spike report merged: kernel facts confirmed on fleet kernels; P1–P10 green; link-monitor reads classified +- [ ] PR-18 spike report merged: kernel facts confirmed on fleet kernels; P1–P13 green; the P12 gate met (or the fallback chosen); link-monitor and `rtnlmon` requests inventoried - [ ] PR-19 profiles canaried on today's collector; T4/T5 green; fleet-gate asserts the profile names -- [ ] PR-20/PR-21: Q, R, S, T green; both protocol surfaces reviewed line by line; coverage-guided fuzz targets for the nlbroker datagram decoder and the Docker broker's inbound RPCs running continuously +- [ ] PR-20/PR-21: Q, R, S, T green; both protocol surfaces reviewed line by line; coverage-guided fuzz targets for both brokers' inbound RPCs and the collector's chunk receiver running continuously - [ ] PR-22: broker mode default; excess-caps gauge and alert live - [ ] PR-23: rollout steps 1–5 on canaries with a parity soak before step 5; U1–U9 recorded - [ ] [metrics.md](../metrics.md): `xtcp2_nlbroker_*`, `xtcp2_dockerbroker_*`, `xtcp2_collector_excess_caps` @@ -553,4 +594,5 @@ On a canary running Phase 5 with a deliberately crippled collector build (Phase - [ ] every RCE probe fails: firewall and route changes, `bpf`, `perf_event_open`, `setns`, mount, `SOCK_DESTROY`, Docker API (U3); Phase 4's egress rule survives (U4) - [ ] monitoring output is unchanged against direct mode (P9, U2) - [ ] the netlink broker can enter only network namespaces and open only `SOCK_DIAG`/`ROUTE` sockets, enforced by seccomp (R1–R10); the Docker broker can issue only its four GETs (S9) and returns only the projection (S2) -- [ ] each broker's surface is exactly its listed ops: a descriptor test fails on any new message arm or RPC without a test-table row +- [ ] the collector cannot create a `SOCK_DIAG` socket (T7), and every netlink request the broker sends matches its RPC's golden bytes (Q2) +- [ ] each broker's surface is exactly its listed RPCs: a descriptor test fails on any new RPC without a test-table row diff --git a/docs/security-hardening/plan-configservice-set-hardening.md b/docs/security-hardening/plan-configservice-set-hardening.md index 07bba598..553642d6 100644 --- a/docs/security-hardening/plan-configservice-set-hardening.md +++ b/docs/security-hardening/plan-configservice-set-hardening.md @@ -43,7 +43,7 @@ 5. Every applied `Set` is auditable. 6. The collector process ends up with **zero** filesystem write capability. 7. The daemon refuses to start without the confinement its severity rating assumes, unless the operator waives named checks. -8. The collector holds no capabilities, no `docker.sock` and no namespace handles. The privileged reads it needs go through two brokers, each enforced by validators on both ends, broker-side filtering, and a syscall profile. +8. The collector holds no capabilities, no `docker.sock` and no namespace handles, and sends no netlink message into any namespace. The privileged reads it needs go through two brokers, each enforced by validators on both ends, broker-side filtering, and a syscall profile. Every xtcp2 process talks to another over gRPC on a unix socket (`pkg/ipc`). **Non-goals** (tracked in the advisory's [honest limits](security-vulnerability-configservice-set-fs-write.md#7-honest-limits--what-this-does-not-fix)) @@ -157,9 +157,10 @@ Belt layer 4: kernel enforcement that survives `syscall.Exec` (soft restart) and Belt layer 5: the collector ends up with zero write rules; the two legitimate writes move behind a path-free API. Full design in advisory §5.5. -- [ ] New proto `proto/state_writer/v1/state_writer.proto`: `service StateWriter { rpc WriteBaseline(bytes) …; rpc PublishIpmetaCache(bytes) … }` — **no path parameters on the wire, ever** +- [ ] New proto `proto/xtcp_writer/v1/writer.proto`: `service WriterService { Ping; WriteBaseline; WriteIpmeta (client stream); GetStats }` — **no path parameters on the wire, ever** +- [ ] New `pkg/ipc`: the gRPC-over-UDS stack every xtcp2 process uses (peer uid checked in the handshake before HTTP/2 parsing, hardened options, unknown-field + protovalidate + rate-limit interceptors); Phase 5's brokers reuse it - [ ] New daemon mode `xtcp2 -writer` (`cmd/xtcp2/writer.go`): - - [ ] gRPC server on UDS `/run/xtcp2/writer.sock`; `SO_PEERCRED` on accept, reject unexpected uids + - [ ] gRPC server on UDS `/run/xtcp2/writer.sock` via `pkg/ipc`; `SO_PEERCRED` in the handshake, reject unexpected uids - [ ] Fixed targets inside its root: `link-monitor-baseline.json`, `ipmeta/current.lookup.parquet.zst` (+ previous) - [ ] Size caps (baseline ≤ 64 KiB = `baseline.MaxFileBytes`; cache ≤ `-ipmetaMaxBytes`, default 64 MiB), per-RPC rate limits, monotonic version numbers (reject rollback/replay), mode 0640 group `xtcp2-state` (the collector, uid 0 without `CAP_DAC_OVERRIDE`, reads via the group) - [ ] Reuse the existing atomic-write discipline (`pkg/linkmonitor/internal/baseline/files.go`): `CreateTemp` → write → `fsync` → `rename` → dir fsync; baseline `flock` moves inside the writer @@ -221,32 +222,32 @@ Changes in the deployment repo (`ansible/roles/monitoring/xtcp2/`) and the image Advisory §5.8, [Phase 5](phase-5-privilege-brokers.md#the-netlink-broker). Moves `setns` and both capabilities out of the collector. - [ ] Spike (PR-18): - - confirm on fleet kernels that a capability-less process using a broker-opened netlink fd gets unprivileged semantics (`__netlink_ns_capable` checks opener **and** sender); - - run every collector feature with zero capabilities; - - classify the link-monitor reads. -- [ ] `proto/xtcp_nlbroker/v1/nlbroker.proto` — message-only, with protovalidate rules: - - `NetnsId{dev, inode}` with `gt 0`; - - `NetlinkProtocol` enum `in: [SOCK_DIAG, ROUTE]`; - - oneof required; - - `ListNetns` paginated at ≤ 256 entries per datagram. -- [ ] `xtcp2 -nlbroker` (`pkg/nlbroker`): - - `SOCK_SEQPACKET` UDS, replies carrying the fd in `SCM_RIGHTS`; - - a discovery table the broker owns; - - `fstat` + `NS_GET_NSTYPE` re-check before `setns`; - - rate and single-connection limits; - - `SO_PEERCRED` must equal the collector uid. + - confirm the kernel facts on fleet kernels: a sender without `CAP_NET_ADMIN` gets only unprivileged netlink semantics, and dump replies go only to the requesting socket; + - prototype the dump relay and run every collector feature against it with zero capabilities; + - run the relay benchmark gate (P12) against today's direct and io_uring paths; + - inventory the link monitor's and `rtnlmon`'s netlink requests. +- [ ] `proto/xtcp_nlbroker/v1/nlbroker.proto` — a gRPC service with protovalidate rules: + - `ListNetns`, `DumpSockets` and `DumpRoute`, the dumps as server streams of raw `NetlinkChunk { bytes data }`; + - `NetnsId{cookie}` with `gt 0`; + - `DumpSockets`: family enum, TCP state mask ≤ 16 382, extension mask ≤ 255; no protocol, socket-id or attribute field; + - `DumpRoute`: links, addresses or routes. +- [ ] `xtcp2 -nlbroker` (`pkg/nlbroker` on `pkg/ipc`): + - builds and sends every netlink request from constants plus validated fields (golden-byte tests); relays replies unfiltered, reading only `nlmsghdr` headers; + - a discovery table the broker owns; one cached socket per (netns, protocol), verified by `fstat`, `NS_GET_NSTYPE` and `SO_NETNS_COOKIE`; + - one dump per netns, concurrency, rate, byte-cap and single-connection limits; + - the peer uid must equal the collector uid. - [ ] Collector seams: - - `netNamespaceInstance`/`attemptOpenAndSetns` and `nsLocalitySnapshot` → `OpenSocket`; + - a stream receiver feeding the existing `packetCh` replaces the per-netns sockets in `netNamespaceInstance`; `nsLocalitySnapshot` → `DumpRoute`; - `ns_discover.go` → `ListNetns`; - - new `-nlBrokerSock` flag. + - new `-nlBrokerSock` flag; direct mode only in builds with the `netns_direct` tag. - [ ] De-privilege (PR-22): - collector runs as uid `xtcp2` with `cap_drop ALL`; - token and secret files group `xtcp2`, `0440`; - capability check per mode; - new `xtcp2_collector_excess_caps` gauge. -**Acceptance**: P1–P10, Q1–Q17, R1–R10, U1–U5 ([Phase 5 tests](phase-5-privilege-brokers.md#tests)). -**Estimate**: 8–12 days. **Risk**: a link-monitor read that genuinely needs `CAP_NET_ADMIN` → typed broker op via the admission checklist. +**Acceptance**: P1–P13, Q1–Q24, R1–R10, T6–T7, U1–U5 ([Phase 5 tests](phase-5-privilege-brokers.md#tests)). +**Estimate**: 10–14 days. **Risk**: the relay misses the P12 gate → the shared-memory reply ring (same single copy as today); a link-monitor read that genuinely needs `CAP_NET_ADMIN` → typed broker op via the admission checklist. ### W11 — Docker broker (L7) @@ -361,10 +362,11 @@ With admin token in hand on a canary host, every item must fail safely: | OD-6 | Frozen-field escape hatch lifetime | Time-boxed to one migration window, then remove the flag | | OD-7 | State-dir quota mechanism (alert-only vs XFS project quota) | Alert-only in Phase 2 (cheap), quota when a host survives to Phase 3 | | OD-8 | Does `xtcp2-min` keep `file:` (it is the stdlib-only flavor)? | Drop it there too; only `full`/`-debug` carry the tag | -| OD-10 | Netlink broker: pass fds vs run the dumps in the broker | Pass fds over `SOCK_SEQPACKET`. The kernel checks the sender's capabilities as well as the opener's, so the fd carries no privilege, and the hot path stays zero-copy | +| OD-10 | Netlink broker: pass fds vs run the dumps in the broker | Run the dumps in the broker, over gRPC. The broker builds and sends every request, so what reaches the kernel is filtered by xtcp2 and not only by the kernel; replies are relayed unfiltered as raw chunks. A passed fd would let the collector send anything on it. Fallback if the P12 benchmark fails: a read-only shared-memory reply ring | | OD-11 | Docker access: typed gRPC broker vs a filtering HTTP proxy in front of `docker.sock` | Typed broker — the projection and the fixed request set are then structural, not pattern-matched | | OD-12 | Bound the Docker broker further with dockerd TCP+mTLS and an authz plugin (GET-only for the broker's cert) | Deferred until W11 lands | -| OD-13 | Make the W13 `lsm` and `pid_namespace` checks required? | Not yet: many bare-metal hosts run no LSM, and bare-metal discovery scans host `/proc`. Revisit when W10's broker owns discovery | +| OD-13 | Make the W13 `lsm` and `pid_namespace` checks required? | Not yet: many bare-metal hosts run no LSM. Revisit when W10's broker owns discovery | +| OD-14 | Broker the link monitor's and `rtnlmon`'s host-netns netlink requests too? | Yes, after W10: typed broker RPCs for the requests PR-18 inventories, so the collector's seccomp profile denies `AF_NETLINK` entirely. Until then they stay capless in the host netns, where the kernel admits only GETs and multicast joins | | OD-9 | Drop the network stdlib destinations (`tcp`/`udp`/`http`/`https`/`unix`/`unixgram`) from the production flavor? | Yes — production needs `s3parquet` only; kills the DEST-repoint exfil class structurally | ## References diff --git a/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md b/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md index ee9a88ef..cdf5e901 100644 --- a/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md +++ b/docs/security-hardening/security-vulnerability-configservice-set-fs-write.md @@ -413,14 +413,14 @@ L4/L5 confine what the collector does with files. Even in the container, they le L7 moves all three out of the collector. -- **Netlink broker** (`xtcp2 -nlbroker`) — the only process that enters namespaces. It has two operations: list network namespaces, and open a netlink `SOCK_DIAG` or `ROUTE` socket inside one. The socket is returned to the collector as an fd over `SCM_RIGHTS`. - - **Why passing the fd is safe:** netlink privilege checks require the socket opener *and* the sender to hold the capability (`__netlink_ns_capable`). A collector with no capabilities that holds the fd gets unprivileged semantics: dumps work, while `SOCK_DESTROY`, route changes, socket marks and TCP-MD5 keys do not. The collector's hot path still reads the same socket, zero-copy. - - **What the broker holds:** `CAP_SYS_ADMIN` only, and no `CAP_NET_ADMIN`. Its seccomp profile admits `setns` only with `CLONE_NEWNET`, and `socket` only for those two netlink protocols. - - **Wire format:** proto over `SOCK_SEQPACKET`, because gRPC cannot carry file descriptors. +- **Netlink broker** (`xtcp2 -nlbroker`) — the only process that enters namespaces, and the only one that sends netlink messages into them. It is a gRPC service with three RPCs: list network namespaces, dump TCP sockets (the `ss --tcp --info` request), and dump links, addresses or routes. + - **Sends are filtered; replies are not.** The broker builds every request from constants plus validated fields: the message type is fixed per RPC, protocol is TCP, and there are no attributes, so the inet_diag bytecode filter is unreachable. Replies, the high-volume direction, are relayed unchanged as raw chunks on a server stream into the collector's existing decoders. The cost is one copy. + - **Why not pass the socket fd:** whoever holds a netlink socket can send anything on it, and replies go only to the socket that sent the request, so the reader must also be the sender. + - **What the broker holds:** `CAP_SYS_ADMIN` only, and no `CAP_NET_ADMIN`. The kernel requires the sender's capability for `SOCK_DESTROY`, route changes and MD5 keys (`__netlink_ns_capable`), so even a broker bug cannot issue them. Its seccomp profile admits `setns` only with `CLONE_NEWNET`, and netlink sockets only for `SOCK_DIAG` and `ROUTE`. - **Docker broker** (`xtcp2 -dockerbroker`) — the only holder of `docker.sock`. It is a typed gRPC service, **not an HTTP proxy**: `ListContainers` and `WatchContainers` return a four-field projection (id, name, image, netns inode). - It issues exactly four fixed GETs to dockerd, and no request field reaches dockerd. - Tenant `Env`, mounts and labels never leave the broker. -- **Every broker request is enforced by three independent layers:** +- **Every broker request is enforced by three independent layers**, on one gRPC-over-UDS stack (`pkg/ipc`) shared with the writer, which checks the peer uid before any HTTP/2 frame is parsed: - protovalidate on both ends, with unknown fields rejected; - broker-side filtering against broker-owned state: ids the broker issued itself, a re-check that the handle hasn't changed since discovery, rate and connection limits, and outbound validation that drops tenant strings failing the charset rules; - a per-process seccomp allowlist, AppArmor profile and Landlock ruleset. @@ -429,7 +429,7 @@ L7 moves all three out of the collector. - `bpf`, `perf_event_open`, the mount family, `unshare`, `ptrace`, `keyctl` and the `chmod`/`chown`/`xattr` families denied; - named AppArmor profiles replacing `docker-default`. -**Result.** The collector runs as a dedicated non-root uid with `CapEff 0`, no `docker.sock` and no namespace handles. Peer credentials on the broker sockets now identify it, which uid 0 never did. +**Result.** The collector runs as a dedicated non-root uid with `CapEff 0`, no `docker.sock`, no namespace handles and no socket-diag socket. Peer credentials on the broker sockets now identify it, which uid 0 never did. **Residual.** - The netlink broker, if compromised, can read what the collector reads, and nothing more. @@ -477,7 +477,8 @@ Layer 0 (§5.7) removes primitives 1–2 at the binary level — the production - **Landlock test (Linux, gated):** start the daemon with a temp state dir; assert `os.WriteFile` outside allowed roots fails with `EACCES`; assert the same after a synthetic soft-restart exec. - **Writer tests:** no request field can influence the target path (assert by structurally — the proto has no path field); size cap, rate limit, version-replay rejection; atomicity (kill -9 mid-write leaves old baseline intact). - **Broker tests:** Phase 5's P–U tables: - - fd-semantics proofs (a capability-less holder of a broker-opened fd cannot `SOCK_DESTROY`, change routes or read MD5 keys); + - kernel-fact proofs (a sender without `CAP_NET_ADMIN` cannot `SOCK_DESTROY`, change routes or read MD5 keys; dump replies reach only the requester); + - golden bytes for every netlink request the broker can send, and a byte-exact relay; - both-ends validation; - seccomp denials; - the fixed Docker request set;