Skip to content

docs: security-hardening program — ConfigService.Set advisory and phased hardening plan - #291

Open
randomizedcoder wants to merge 7 commits into
mainfrom
docs/security-hardening
Open

randomizedcoder wants to merge 7 commits into
mainfrom
docs/security-hardening

Conversation

@randomizedcoder

@randomizedcoder randomizedcoder commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner

What this is

The full security assessment of the gRPC ConfigService.Set filesystem-write exposure, and the program to close it — eight documents under docs/security-hardening/, linked from docs/README.md. Docs only: no code, proto, or build changes in this PR.

Documents

Doc Contents
Advisory The Set → filesystem flow; the five Set-reachable write primitives; who can call Set (auth off by default); the fleet container deployment and permission model (NET_ADMIN/SYS_ADMIN, read-only rootfs, private pidns — what each does and does not stop); how the classic container-escape path idioms fare; defense layers L0–L7
Hardening plan Workstreams W0–W13, issue register I1–I19, open decisions OD-1–OD-14, fleet rollout runbook, end-state verification checklist
Implementation plan Phases 0–5 mapped to PR-1…PR-24 with acceptance gates and a rollback matrix
Phase 1 Build-time elimination (dest_file/capture tags), protovalidate path patterns, runtime path policy + frozen fields, universal validating interceptor, Set audit logging, golden-config CI gate, and a startup confinement check (PR-24)
Phase 2 Landlock self-confinement: the exact ruleset, ABI degradation, re-exec persistence (the soft-restart corner case), and the table-driven test matrices
Phase 3 Privilege separation: a writer process with a path-free gRPC service over a UDS on the shared pkg/ipc stack (peer uid checked before HTTP/2 parsing, protovalidate both ends, monotonic seq, streamed ipmeta, crash-safe commit dance), collector :ro flip, the complete write inventory, and the target admission checklist
Phase 4 Egress pinning in three layers (build tags closing OD-9, Landlock v4 network rules, a collector-scoped nftables rule), escape-hatch removal, v1.x revision retirement, standing CI gates
Phase 5 Privilege brokers: a netlink broker (gRPC; builds and sends every netlink request, relays the replies unfiltered as raw chunks) and a Docker broker (typed gRPC, four-field projection) take CAP_NET_ADMIN, CAP_SYS_ADMIN and docker.sock out of the collector; per-process seccomp + AppArmor profiles

Headline findings

  • Set has no path or scheme validation, and listener auth is off by default — one unauthenticated call switches dest to file:<path> and the daemon writes root-owned files at that path on the next poll.
  • Five write primitives are Set-reachable; three are debug tooling with no build gate at all today (file:, write_files/capture_path, UDS binds).
  • The classic /proc/1/root/... escape targets fail on role-deployed containers (private pidns + read-only rootfs make /proc/1/root/... the container's own root); outside that deployment shape the same Set calls land as root.
  • The network stdlib destinations (tcp/udp/http/https/unix/unixgram) are always compiled in — the dest-repoint egress/exfil class is structural (OD-9 closes it in Phase 4).
  • Even in the container, docker.sock, CAP_NET_ADMIN in the host netns, and CAP_SYS_ADMIN (bpf/perf_event_open under Docker's default seccomp) make a compromised collector host-root-equivalent — Phase 5 removes all three.
  • The Critical shape (bare host as root, --privileged, a pod without seccomp) is now refused at startup — PR-24 checks no-new-privs, seccomp, read-only root and the capability bound; a per-check -insecureAllowUnconfined waiver is alerted on.

Design decisions worth reviewing closely

  1. L0 build-time elimination over runtime-only checks — debug writers leave the production binary entirely.
  2. Landlock as primary kernel enforcement (survives execve, including Set's soft restart; not bypassable by a privileged process) — seccomp cannot express path rules.
  3. Writer split (Phase 3) — only two legitimate writes exist (baseline JSON ≤64 KiB, ipmeta cache); the writer's protocol carries no paths anywhere, so the parameter list is the security model.
  4. gRPC over UDS for every xtcp2 process (pkg/ipc) — one typed, validated stack; the peer uid is checked in the transport handshake, so only the expected process reaches the HTTP/2 parser.
  5. Standing CI gates (Phase 4) — write-site inventory drift check, flavor matrix, golden-config validation, and a weekly crippled-build proof so none of it regresses silently.
  6. Privilege brokers (Phase 5) — the collector sends no netlink message into any namespace: the broker builds every request from a typed RPC (TCP only, no attributes, so no inet_diag bytecode) and relays replies unfiltered, at the cost of one copy (benchmark-gated). A passed fd was rejected because its holder can send anything on it. The broker lacks CAP_NET_ADMIN, so the kernel still refuses SOCK_DESTROY, route changes and MD5 keys even if the broker is buggy.
  7. Check confinement, not container markers (PR-24) — /.dockerenv-style checks pass --privileged and fail a hardened systemd unit; the in-tree NixOS module is hardened to pass with no waiver.

Verification

  • All relative links checked (docs/README.md + cross-links between all eight docs).
  • Every code-level claim in the docs was verified against main at 97bed69c (call sites, line references, build tags, nix flavors, protobuf-go semantics).

🤖 Generated with Claude Code

@randomizedcoder
randomizedcoder force-pushed the docs/security-hardening branch from 9be3b52 to f4e53a4 Compare November 9, 2026 01:27
…sed hardening plan

Adds docs/security-hardening/ with the full security assessment of the
gRPC ConfigService.Set filesystem-write exposure and the program to
close it:

- security-vulnerability-configservice-set-fs-write.md — the advisory:
  the Set -> filesystem flow, the five Set-reachable write primitives,
  the fleet container deployment and permission model (NET_ADMIN/
  SYS_ADMIN, read-only rootfs, private pidns), what the Nov 5 incident
  IOCs prove, and the defense layers L0-L6
- plan-configservice-set-hardening.md — workstreams W0-W10, issue
  register I1-I16, open decisions OD-1-OD-9, fleet rollout runbook
- implementation-plan.md — phases 0-4 mapped to PRs with gates and a
  rollback matrix
- phase-1-code-hardening.md — build-time elimination (dest_file/capture
  tags), protovalidate path patterns, runtime path policy + frozen
  fields, universal validating interceptor, Set audit, CI gates
- phase-2-kernel-confinement.md — Landlock self-confinement: the
  ruleset, ABI degradation policy, re-exec persistence, and the full
  table-driven test matrices
- phase-3-writer-split.md — privilege separation: writer process with a
  path-free proto envelope over a UDS (protovalidate both ends), SO_PEERCRED
  auth, monotonic seq, crash-safe commit dance, collector :ro flip, the
  complete write inventory, and the target admission checklist
- phase-4-egress-fleet-lockdown.md — egress pinning in three layers
  (build tags closing OD-9, Landlock v4 network rules, security groups),
  escape-hatch removal, v1.x retirement, standing CI gates

docs/README.md links the set from the security section.
@randomizedcoder
randomizedcoder force-pushed the docs/security-hardening branch from f4e53a4 to b70210a Compare November 9, 2026 01:51
randomizedcoder and others added 6 commits November 9, 2026 08:30
…er perms

- Narrow L4/L5 claims: docker.sock bypasses both; the writer accepts
  schema-valid content at the next seq, so L5 bounds shape, not truth.
- Landlock: chmod/chown unmediated at every ABI, truncate from ABI 3,
  pathname-UDS connect from ABI 9 (RESOLVE_UNIX); tests B10/B14/B24, D7.
- Apply the ruleset once and inherit it across soft restarts (16-layer
  cap); tests C6 (20 restarts) and C7 (forged marker).
- Writer files 0640 group xtcp2-state; collector joins the group and
  keeps a read-only state-dir grant; writer-init chowns legacy files.
- Split -writableDir / -readableDir; EvalSymlinks on the deepest
  existing ancestor so not-yet-created outputs validate.
- unix_socket_mode allowlist in: [0, 384]; lte 384 accepted 0577.
- Scope egress pinning to the collector's cgroup (nftables), not
  host-wide SG/UFW; fix the Test D anchor and -setAllowFrozenFields.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ency

- setns: SYS_ADMIN enters net/UTS/IPC/cgroup/time namespaces only;
  mount needs CAP_SYS_CHROOT, PID only descendants.
- :ro binds and the ro rootfs hold against SYS_ADMIN only because
  AppArmor docker-default denies mount; add it to the runtime table,
  restrictions and a fleet-gate regression guard (PidMode, ReadonlyRootfs,
  AppArmorProfile, remount probe).
- dest pattern uses one colon (matches Phase 1); ipmeta cap 64 MiB
  everywhere; OD-3 aligned with W0; Set on an uncompiled scheme returns
  FailedPrecondition, not a restart loop; I15 renumbered to I14.
- Phase 3 frame tests match the [len u32] + proto envelope (min_len 1,
  oneof required).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e, severity

- New phase-5-privilege-brokers.md: netlink broker (proto over SOCK_SEQPACKET,
  SCM_RIGHTS fds), Docker broker (typed gRPC, fixed GET set, projection),
  per-process seccomp/AppArmor profiles, three enforcement layers per request,
  test tables P-U, rollout and decisions
- Advisory: L7 section and matrix column; setns limited to network namespaces
  in the container; bpf/perf_event_open and NET_ADMIN reach spelled out;
  severity scoped to deployment shape
- Plan: W10-W12, I15-I18, OD-10-OD-12; implementation plan Phase 5 (PR-18-PR-23);
  Phase 4 egress rule holds against collector RCE only after Phase 5

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The daemon refuses to start (exit 78) without no-new-privs, a seccomp
filter, a read-only root and a bounded capability set, and later
Landlock. It checks confinement rather than container markers: those
pass --privileged and fail a hardened systemd unit. A per-check
-insecureAllowUnconfined / XTCP_INSECURE_ALLOW_UNCONFINED waiver is
alerted on, and the NixOS module is hardened to pass without one.
Phase 2's OD-2 changes from warn-and-continue to refuse unless waived.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- allowed capability set per mode (writer, both brokers, broker-mode
  collector) so one list cannot wave through a privileged writer
- in-repo deployments that would fail the check: k8s DaemonSet (also
  drops the unused CAP_SYS_CHROOT), dev docker-compose, quickstart
  sudo examples, nix capability-check
- startup-failure tests assert the message, not a non-zero exit (PR-1
  acceptance, image smoke test), since exit 78 would also match
- Phase 3 writer container and bare-metal units carry read_only,
  no-new-privileges and systemd hardening; Xtcp2ConfinementWaived alert
- stale Landlock degrade-not-exit wording in W4 and the advisory
- W9 docs/metrics; W7 gate noted as the check's pre-deploy twin

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Netlink broker becomes a gRPC service (ListNetns, DumpSockets,
  DumpRoute) that builds and sends every netlink request from typed,
  validated fields and relays replies unfiltered as raw chunks; the
  collector no longer holds any netlink socket in a namespace
- Kernel fact 5: a netlink socket has no receive-only mode and dump
  replies go only to the requester, so fd passing would leave sends
  filtered by the kernel alone
- New pkg/ipc: gRPC over UDS for the writer and both brokers; peer
  uid checked in the transport handshake before HTTP/2 parsing
- Writer moves to a WriterService (ipmeta as a client stream)
- Netns ids become netns cookies; netns_direct build tag; collector
  seccomp forbids SOCK_DIAG; P12 relay benchmark gate with a
  shared-memory fallback; OD-10 revised, OD-14 added
- Bare metal marked unsupported for the broker design

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant