Skip to content

[deckhouse-cli] Read RPP addresses published by the cluster - #443

Draft
Glitchy-Sheep wants to merge 9 commits into
mainfrom
fix/rpp-endpoint-candidates
Draft

[deckhouse-cli] Read RPP addresses published by the cluster#443
Glitchy-Sheep wants to merge 9 commits into
mainfrom
fix/rpp-endpoint-candidates

Conversation

@Glitchy-Sheep

@Glitchy-Sheep Glitchy-Sheep commented Aug 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

d8 plugins and d8 cli learn where registry-packages-proxy is and how to verify its TLS from the ConfigMap the cluster publishes, instead of relying on a public Ingress. This is the client half of deckhouse#21933; together they make plugins and self-update work on clusters with no public domain, no ingress controller and no cert-manager.

Context

Discovery in main prefers the public Ingress and falls back to the first ready pod. In an air-gapped cluster the Ingress serves the fake ingress.local certificate, and the pod path needs --rpp-insecure-skip-tls-verify - the sidecar's own certificate carries no IP SANs. So on exactly the clusters the proxy ADR targets, every plugin command failed.

How it works

Discovery builds a candidate list and takes the first endpoint that answers:

  1. Master addresses from the registry-packages-proxy-config ConfigMap, verified with the CA published next to them. No public domain, no DNS, no external certificate authority.
  2. The public endpoint, verified with the system roots - for a workstation with no network path to the masters.
  3. Pod IPs - kept for clusters that publish no config yet; this path still needs the insecure flag.

Around the candidate walk:

  • Reading the ConfigMap needs the same cli-download role that allows the download itself, so an operator binds one role and nothing else. A denied read only narrows the candidate list.
  • The reachability probe carries no kubeconfig credential: an endpoint gets the token only after it passed TLS verification. With the insecure flag any answering host would have received it.
  • A rejected credential (401 from the API server) stops discovery with a clear error instead of quietly degrading to fewer candidates.
  • The cluster rotates the proxy trust root together with its certificate. On a certificate verification failure the client re-reads the published CA and retries once - both during discovery and on later requests, including pulls.
  • The proxy client is built only by the network subcommands. d8 plugins list and remove work with the disk alone and no longer need a reachable cluster.
  • Explicit --rpp-endpoint, --rpp-ca-file and --rpp-insecure-skip-tls-verify still win over everything above and pin the trust.
image

Before / After

Before: in air-gap, discovery ended at the Ingress stub (x509: certificate is valid for ingress.local) or required disabling TLS verification; a dead Ingress with a live domain was a dead end.

After: d8 plugins install and d8 cli update work over masterIP:4219 with a verified certificate; the Ingress path keeps working for clients outside the cluster network.

image

Tests

  • 55 tests in internal/rpp, covering among others:
    • candidate order and the skip of an unverifiable endpoint
    • the probe sending no Authorization header while real requests keep it
    • a 401 failing discovery at both the ConfigMap and the whole-discovery level
    • trust healing for a stale CA read during discovery and for a rotation after the client was built
  • errdetect tests for both plugins and self-update diagnostics.

Notes

  • On clusters whose module publishes no ConfigMap yet, behavior stays as in main: Ingress first, then pods.
  • The 403 diagnostic names the role to bind (d8:registry-packages-proxy:cli-download); the 401 one explains when a client-certificate kubeconfig works and when only a Bearer token does.

@Glitchy-Sheep
[rpp] Read the proxy CA and addresses published by the cluster
af9fd2c 
- The module publishes `registry-packages-proxy-config` with the master
  addresses, the CA of the certificate they serve, and the public endpoint.
- Reading it gives a client everything it needs with a single `get`, instead
  of listing pods or reading the Ingress object.
- A missing ConfigMap and a denied read return an empty config, so clusters
  without the published contract keep working as before.
- Malformed content is an error: it means the contract changed.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
[rpp] Try endpoint candidates in order instead of failing on the first
ac438a5 
- Discovery returned a single endpoint, so one bad certificate stopped the
  command even when another endpoint would have worked.
- Candidates now come in order: master addresses verified with the published
  CA, then the public host, then pod IPs. Each is probed and a TLS or network
  failure moves on to the next.
- A denied read drops the candidates it would have produced and keeps the
  rest, so a narrowly permitted identity still gets a usable path.
- The Ingress is read only when the cluster publishes no public endpoint.
- The failure message names every candidate tried and why each was rejected.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
[rpp] Document why the endpoint scheme is always https
45e6d78 
Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
[rpp] Report contradictory TLS flags directly
1517643 
- `--rpp-ca-file` together with `--rpp-insecure-skip-tls-verify` came back as
  `ErrEndpointDiscovery` with the same configuration error repeated for every
  candidate, so `errors.Is` could not see `ErrUnsupportedConfig`.
- The flag pair is judged once, before any endpoint is tried.
- `collectOptions` keeps that check and `New` on one code path.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep Glitchy-Sheep self-assigned this Aug 7, 2026
@Glitchy-Sheep
Stop treating apiserver 401 as an empty discovery result
313cb5b 
- A 401 means the credentials are rejected: every later cluster call fails the same way, so discovery now surfaces it instead of degrading to fewer candidates.
- `isReadDenied` keeps only Forbidden: a narrowly permitted identity still loses just its candidates.
- Covered at both levels: `readClusterConfig` and whole-discovery `discoverCandidates`.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Probe candidate endpoints without the kubeconfig credential
8ca3ac9 
- The reachability probe runs before an endpoint passed TLS verification, and with --rpp-insecure-skip-tls-verify any answering host would receive the token.
- `Client` now keeps a credential-free probe client with the same server trust; `/healthz` needs no authentication anyway.
- The test asserts the probe arrives with no Authorization header while the following real request still carries the Bearer token.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Build the proxy client only for network plugin subcommands
e7d6c7e 
- `list` and `remove` are disk-only and now work without a reachable cluster; before, every subcommand paid for endpoint discovery up front.
- `InitPluginServices` is idempotent and called by `versions`, `contract`, `install` and `update` right before the first proxy call.
- Init errors now surface inside RunE, where `wrapProxyDiagnostics` already classifies them, so the duplicate handling in `PersistentPreRunE` is gone.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Update the 401 diagnostic for client-certificate kubeconfigs
bb6eb76 
- Recent registry-packages-proxy accepts client certificates signed by the cluster CA, so "a client-certificate kubeconfig is not enough" is no longer true everywhere.
- The cause now names both rejection reasons, and a second solution explains when a certificate kubeconfig works and when only a Bearer token does.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
@Glitchy-Sheep
Re-read the published CA and retry once on certificate failure
b85e3a6 
- The platform reissues the proxy certificate together with its root, so a client holding the pre-rotation CA failed for good; the new root lives in the same ConfigMap the old one came from.
- `healTrust` rebuilds both transports around the re-read CA; it fires only when the bundle actually changed, so a retry cannot loop.
- Wired only for discovered master endpoints; explicit --rpp-ca-file or insecure flags pin the trust.
- Covers both windows: a stale CA read during discovery and a rotation after the client was built.

Signed-off-by: Roman Berezkin <roman.berezkin@flant.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ldmonster ldmonster Awaiting requested review from ldmonster ldmonster will be requested when the pull request is marked ready for review ldmonster is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

@Glitchy-Sheep Glitchy-Sheep

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Glitchy-Sheep