From 38872af238ae1a20089d2c70757363659d3305d6 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Fri, 29 May 2026 15:31:15 +0200 Subject: [PATCH 1/2] feat(rbac): scope OIDC kubectl/Headlamp access to read-only OIDC (Dex + GitHub) was bound to cluster-admin via the oidc-admin ClusterRoleBinding, so any browser login became full admin. Make day-to-day OIDC access least-privilege (read-only) and keep admin as break-glass via the root client-certificate kubeconfig stored in the vault. - Add cluster-reader ClusterRole: cluster-scoped infra reads (nodes, PVs, storage, CRDs, API services, CSRs, RBAC objects, node/pod metrics). Excludes Secrets and all write/exec verbs. - Replace oidc-admin (cluster-admin) with two read-only bindings: oidc-view (built-in view) + oidc-cluster-reader (cluster-reader), both for User oidc:${admin_email}. - Update docs/oidc-kubectl.md: read-only model + break-glass procedure. roleRef is immutable, so this is a delete+recreate of the binding. The cert-based admin path is untouched, so there is no lockout window. Co-Authored-By: Claude Opus 4.8 --- docs/oidc-kubectl.md | 70 +++++++++++++--- .../cluster-role-bindings/kustomization.yaml | 2 +- .../cluster-role-bindings/oidc-admin.yaml | 16 ---- .../cluster-role-bindings/oidc-readonly.yaml | 38 +++++++++ .../cluster-roles/cluster-reader.yaml | 82 +++++++++++++++++++ .../cluster-roles/kustomization.yaml | 1 + 6 files changed, 182 insertions(+), 27 deletions(-) delete mode 100644 k8s/bases/infrastructure/cluster-role-bindings/oidc-admin.yaml create mode 100644 k8s/bases/infrastructure/cluster-role-bindings/oidc-readonly.yaml create mode 100644 k8s/bases/infrastructure/cluster-roles/cluster-reader.yaml diff --git a/docs/oidc-kubectl.md b/docs/oidc-kubectl.md index e355c7ac9..2bcd27292 100644 --- a/docs/oidc-kubectl.md +++ b/docs/oidc-kubectl.md @@ -2,11 +2,13 @@ This guide explains how to use [`kubelogin`](https://github.com/int128/kubelogin) to authenticate `kubectl` against the Kubernetes API via Dex and GitHub. +OIDC is the **default, day-to-day** way to reach the cluster and is intentionally **read-only** (see [RBAC](#rbac)). Write/admin access is **break-glass only** — the root client-certificate kubeconfig kept in the vault (see [Break-glass admin access](#break-glass-admin-access)). + ## Prerequisites - Access to the cluster (kubeconfig with server address) - A GitHub account that is a member of the [`devantler-tech`](https://github.com/devantler-tech) organization -- The `oidc-admin` ClusterRoleBinding must list your OIDC identity (see [RBAC](#rbac)) +- Your OIDC identity must be bound to the read-only roles (see [RBAC](#rbac)) ## 1 — Install kubelogin @@ -76,6 +78,12 @@ kubectl config set-context oidc@prod \ ```bash kubectl config use-context oidc@local # or oidc@prod kubectl get nodes + +# Confirm the identity and that access is read-only: +kubectl auth whoami # Username: oidc:ned@devantler.tech +kubectl auth can-i list pods # yes +kubectl auth can-i get secrets # no (Secrets are excluded by design) +kubectl auth can-i create deployments # no (writes are break-glass only) ``` On the first run, `kubelogin` opens a browser window. Log in with GitHub @@ -110,25 +118,66 @@ kube-apiserver validates token • groups claim → Kubernetes groups │ ▼ -RBAC: ClusterRoleBinding oidc-admin - grants cluster-admin to oidc:ned@devantler.tech +RBAC: ClusterRoleBindings oidc-view + oidc-cluster-reader + grant READ-ONLY (view + cluster-reader) to oidc:ned@devantler.tech + (admin is break-glass via the root cert — not OIDC) ``` ## RBAC -The `oidc-admin` ClusterRoleBinding -(`k8s/bases/infrastructure/cluster-role-bindings/oidc-admin.yaml`) grants -`cluster-admin` to the user whose email matches the `email` claim returned -by Dex: +OIDC access is **read-only**. Two ClusterRoleBindings in +`k8s/bases/infrastructure/cluster-role-bindings/oidc-readonly.yaml` bind the +user whose email matches the Dex `email` claim (`oidc:${admin_email}`) to: + +| Binding | Role | Grants | +|---|---|---| +| `oidc-view` | built-in `view` | read all **namespaced** resources (pods, deployments, configmaps, pod logs, events, …) — **excluding Secrets** | +| `oidc-cluster-reader` | `cluster-reader` | read **cluster-scoped** infra (nodes, PVs, storage classes, CRDs, API services, priority/runtime/ingress classes, CSRs, RBAC objects, node/pod metrics) | ```yaml subjects: - apiGroup: rbac.authorization.k8s.io kind: User - name: "oidc:ned@devantler.tech" + name: "oidc:${admin_email}" # → oidc:ned@devantler.tech ``` -To grant access to additional users, add more subjects to that file. +What is deliberately **not** granted via OIDC: Secrets (kept in the vault on +purpose) and any write/exec verb (`create`/`update`/`delete`, `pods/exec`, +`pods/portforward`). The `cluster-reader` ClusterRole is defined in +`k8s/bases/infrastructure/cluster-roles/cluster-reader.yaml`. + +To grant read-only access to additional users, add more subjects to both +bindings (or switch the subject to a Dex group such as +`oidc:devantler-tech:platform`). + +## Break-glass admin access + +There is **no admin path via OIDC** — by design. When a write or an +otherwise-forbidden operation is genuinely required, use the root +**client-certificate** kubeconfig stored in the vault: + +1. Retrieve the root kubeconfig from the vault and point `KUBECONFIG` at it + (or merge its `admin@prod` context), e.g.: + + ```bash + export KUBECONFIG=/path/to/root-kubeconfig.yaml + kubectl config use-context admin@prod + ``` + +2. Do the minimal change, then switch back to the OIDC context: + + ```bash + unset KUBECONFIG # back to ~/.kube/config + kubectl config use-context oidc@prod + ``` + +The root cert authenticates directly against the cluster CA and bypasses +OIDC/RBAC role limits (it is `cluster-admin`), so treat it accordingly: pull +it only when needed and never persist it in your day-to-day kubeconfig. + +Last-resort regeneration (if the vault copy is lost): a fresh admin +kubeconfig can be minted from the Talos control plane with +`talosctl --talosconfig kubeconfig`. ## Dex client configuration @@ -145,7 +194,8 @@ kube-apiserver's `--oidc-client-id` flag. | `error: You must be logged in to the server (Unauthorized)` | Token expired or wrong audience | Run `kubectl oidc-login setup --oidc-issuer-url=... --oidc-client-id=kubectl` to verify the flow | | Browser doesn't open | kubelogin not installed or not in `$PATH` | Verify `kubectl oidc-login --help` works | | `x509: certificate signed by unknown authority` (local) | mkcert CA not trusted | Pass `--certificate-authority-data` or install the mkcert root CA in your system trust store | -| `Forbidden` after successful login | Email doesn't match `oidc-admin` subject | Check `kubectl auth whoami` and update the ClusterRoleBinding | +| `Forbidden` on a **read** | Email not bound, or resource outside the read-only roles | Check `kubectl auth whoami`; confirm the email matches the `oidc-readonly.yaml` subjects | +| `Forbidden` on a **write** | Expected — OIDC is read-only | Use the [break-glass admin cert](#break-glass-admin-access) for writes | ## References diff --git a/k8s/bases/infrastructure/cluster-role-bindings/kustomization.yaml b/k8s/bases/infrastructure/cluster-role-bindings/kustomization.yaml index c26e8bca7..4ba1cb3a8 100644 --- a/k8s/bases/infrastructure/cluster-role-bindings/kustomization.yaml +++ b/k8s/bases/infrastructure/cluster-role-bindings/kustomization.yaml @@ -3,4 +3,4 @@ apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - flux-web-admins.yaml - - oidc-admin.yaml + - oidc-readonly.yaml diff --git a/k8s/bases/infrastructure/cluster-role-bindings/oidc-admin.yaml b/k8s/bases/infrastructure/cluster-role-bindings/oidc-admin.yaml deleted file mode 100644 index f58a634b5..000000000 --- a/k8s/bases/infrastructure/cluster-role-bindings/oidc-admin.yaml +++ /dev/null @@ -1,16 +0,0 @@ -# ClusterRoleBinding that grants cluster-admin to users authenticated -# via Dex OIDC who are members of the devantler-tech GitHub org. -# The group name matches the OIDC groups claim from Dex's GitHub connector. ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRoleBinding -metadata: - name: oidc-admin -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: cluster-admin -subjects: - - apiGroup: rbac.authorization.k8s.io - kind: User - name: "oidc:${admin_email}" diff --git a/k8s/bases/infrastructure/cluster-role-bindings/oidc-readonly.yaml b/k8s/bases/infrastructure/cluster-role-bindings/oidc-readonly.yaml new file mode 100644 index 000000000..765770d44 --- /dev/null +++ b/k8s/bases/infrastructure/cluster-role-bindings/oidc-readonly.yaml @@ -0,0 +1,38 @@ +# Read-only cluster access for the operator's OIDC identity (authenticated +# via Dex + GitHub, members of the devantler-tech org). The email matches +# the `email` claim returned by Dex; the apiserver prefixes it with `oidc:`. +# +# Daily access is intentionally read-only. ADMIN (break-glass) is NOT granted +# via OIDC — it is the root client-certificate kubeconfig/talosconfig kept in +# the vault, used only when a write is genuinely required. See +# docs/oidc-kubectl.md. +# +# Two bindings give a complete read-only picture: +# - built-in `view` → namespaced resources (minus Secrets) +# - `cluster-reader` → cluster-scoped infra (nodes, PVs, CRDs, ...) +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: oidc-view +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: view +subjects: + - apiGroup: rbac.authorization.k8s.io + kind: User + name: "oidc:${admin_email}" +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: oidc-cluster-reader +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: cluster-reader +subjects: + - apiGroup: rbac.authorization.k8s.io + kind: User + name: "oidc:${admin_email}" diff --git a/k8s/bases/infrastructure/cluster-roles/cluster-reader.yaml b/k8s/bases/infrastructure/cluster-roles/cluster-reader.yaml new file mode 100644 index 000000000..988d8f4fd --- /dev/null +++ b/k8s/bases/infrastructure/cluster-roles/cluster-reader.yaml @@ -0,0 +1,82 @@ +# Read-only ClusterRole for the cluster-scoped resources that the built-in +# `view` role deliberately omits. Bound ALONGSIDE `view` (see +# cluster-role-bindings/oidc-readonly.yaml) so an OIDC identity gets a +# complete read-only picture of the cluster: +# - `view` → namespaced resources (pods, deployments, configmaps, +# pod logs, events, ...), minus Secrets. +# - `cluster-reader` → cluster-scoped infra below. +# +# Deliberately NOT granted: Secrets (kept in the vault on purpose), and any +# write/exec verb (no pods/exec, pods/portforward, create/update/delete). +# Everything here is get/list/watch only. +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: cluster-reader +rules: + # Cluster-scoped core resources not covered by `view`. + - apiGroups: [""] + resources: + - nodes + - persistentvolumes + verbs: ["get", "list", "watch"] + # Node and pod metrics (kubectl top). + - apiGroups: ["metrics.k8s.io"] + resources: + - nodes + - pods + verbs: ["get", "list", "watch"] + # Storage topology. + - apiGroups: ["storage.k8s.io"] + resources: + - storageclasses + - volumeattachments + - csidrivers + - csinodes + - csistoragecapacities + verbs: ["get", "list", "watch"] + # API surface discovery. + - apiGroups: ["apiextensions.k8s.io"] + resources: + - customresourcedefinitions + verbs: ["get", "list", "watch"] + - apiGroups: ["apiregistration.k8s.io"] + resources: + - apiservices + verbs: ["get", "list", "watch"] + # Scheduling / runtime / ingress classes. + - apiGroups: ["scheduling.k8s.io"] + resources: + - priorityclasses + verbs: ["get", "list", "watch"] + - apiGroups: ["node.k8s.io"] + resources: + - runtimeclasses + verbs: ["get", "list", "watch"] + - apiGroups: ["networking.k8s.io"] + resources: + - ingressclasses + verbs: ["get", "list", "watch"] + # APIServer flow control (apiserver debugging). + - apiGroups: ["flowcontrol.apiserver.k8s.io"] + resources: + - flowschemas + - prioritylevelconfigurations + verbs: ["get", "list", "watch"] + # Certificate signing requests (kubelet-serving-cert-approver lives here). + # CSRs carry a public certificate request only — never a private key. + - apiGroups: ["certificates.k8s.io"] + resources: + - certificatesigningrequests + verbs: ["get", "list", "watch"] + # RBAC objects — reading these is invaluable for diagnosing "Forbidden" + # errors and exposes no secret material. This is the one place we go + # beyond the built-in `view` role's conservative defaults. + - apiGroups: ["rbac.authorization.k8s.io"] + resources: + - clusterroles + - clusterrolebindings + - roles + - rolebindings + verbs: ["get", "list", "watch"] diff --git a/k8s/bases/infrastructure/cluster-roles/kustomization.yaml b/k8s/bases/infrastructure/cluster-roles/kustomization.yaml index bf8ffb983..cba05029a 100644 --- a/k8s/bases/infrastructure/cluster-roles/kustomization.yaml +++ b/k8s/bases/infrastructure/cluster-roles/kustomization.yaml @@ -2,6 +2,7 @@ apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: + - cluster-reader.yaml - cnpg-tenant-edit.yaml - external-secrets-tenant-edit.yaml - gateway-tenant-edit.yaml From 01108c37fd2bb37d85f1a7da811818d4abaff835 Mon Sep 17 00:00:00 2001 From: Nikolai Emil Damm Date: Fri, 29 May 2026 15:45:35 +0200 Subject: [PATCH 2/2] docs(rbac): clarify cluster-reader covers some namespaced reads MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Address Copilot review on #1657: the cluster-reader ClusterRole and its binding were described as "cluster-scoped infra", but they also grant a few namespaced resources cluster-wide (RBAC roles/rolebindings and pod metrics). Reword the header comment, the binding comment, and the docs RBAC table to say "what `view` omits — mostly cluster-scoped, plus RBAC objects and pod metrics". No rule changes. Co-Authored-By: Claude Opus 4.8 --- docs/oidc-kubectl.md | 2 +- .../cluster-role-bindings/oidc-readonly.yaml | 3 ++- .../cluster-roles/cluster-reader.yaml | 14 ++++++++------ 3 files changed, 11 insertions(+), 8 deletions(-) diff --git a/docs/oidc-kubectl.md b/docs/oidc-kubectl.md index 2bcd27292..df20e1a30 100644 --- a/docs/oidc-kubectl.md +++ b/docs/oidc-kubectl.md @@ -132,7 +132,7 @@ user whose email matches the Dex `email` claim (`oidc:${admin_email}`) to: | Binding | Role | Grants | |---|---|---| | `oidc-view` | built-in `view` | read all **namespaced** resources (pods, deployments, configmaps, pod logs, events, …) — **excluding Secrets** | -| `oidc-cluster-reader` | `cluster-reader` | read **cluster-scoped** infra (nodes, PVs, storage classes, CRDs, API services, priority/runtime/ingress classes, CSRs, RBAC objects, node/pod metrics) | +| `oidc-cluster-reader` | `cluster-reader` | read what `view` omits — mostly **cluster-scoped** infra (nodes, PVs, storage classes, CRDs, API services, priority/runtime/ingress classes, CSRs) plus RBAC objects and node/pod metrics | ```yaml subjects: diff --git a/k8s/bases/infrastructure/cluster-role-bindings/oidc-readonly.yaml b/k8s/bases/infrastructure/cluster-role-bindings/oidc-readonly.yaml index 765770d44..25ed837b3 100644 --- a/k8s/bases/infrastructure/cluster-role-bindings/oidc-readonly.yaml +++ b/k8s/bases/infrastructure/cluster-role-bindings/oidc-readonly.yaml @@ -9,7 +9,8 @@ # # Two bindings give a complete read-only picture: # - built-in `view` → namespaced resources (minus Secrets) -# - `cluster-reader` → cluster-scoped infra (nodes, PVs, CRDs, ...) +# - `cluster-reader` → everything `view` omits: mostly cluster-scoped infra +# (nodes, PVs, CRDs, ...) plus RBAC objects and pod metrics --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding diff --git a/k8s/bases/infrastructure/cluster-roles/cluster-reader.yaml b/k8s/bases/infrastructure/cluster-roles/cluster-reader.yaml index 988d8f4fd..0f53f5c07 100644 --- a/k8s/bases/infrastructure/cluster-roles/cluster-reader.yaml +++ b/k8s/bases/infrastructure/cluster-roles/cluster-reader.yaml @@ -1,10 +1,12 @@ -# Read-only ClusterRole for the cluster-scoped resources that the built-in -# `view` role deliberately omits. Bound ALONGSIDE `view` (see -# cluster-role-bindings/oidc-readonly.yaml) so an OIDC identity gets a +# Read-only ClusterRole for the resources the built-in `view` role omits. +# Most are cluster-scoped (nodes, PVs, storage, CRDs, API services, classes, +# CSRs), but a few are namespaced — RBAC roles/rolebindings and pod metrics — +# granted cluster-wide here for read-only diagnostics. Bound ALONGSIDE `view` +# (see cluster-role-bindings/oidc-readonly.yaml) so an OIDC identity gets a # complete read-only picture of the cluster: -# - `view` → namespaced resources (pods, deployments, configmaps, -# pod logs, events, ...), minus Secrets. -# - `cluster-reader` → cluster-scoped infra below. +# - `view` → namespaced workloads/config (pods, deployments, +# configmaps, pod logs, events, ...), minus Secrets. +# - `cluster-reader` → everything `view` omits (see rules below). # # Deliberately NOT granted: Secrets (kept in the vault on purpose), and any # write/exec verb (no pods/exec, pods/portforward, create/update/delete).