Skip to content

[docs] Add GitOps configuration surfaces design proposal#16

Open
myasnikovdaniil wants to merge 6 commits into
mainfrom
daniil/gitops-surfaces-proposal
Open

[docs] Add GitOps configuration surfaces design proposal#16
myasnikovdaniil wants to merge 6 commits into
mainfrom
daniil/gitops-surfaces-proposal

Conversation

@myasnikovdaniil

@myasnikovdaniil myasnikovdaniil commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a design proposal — GitOps Configuration Surfaces — under design-proposals/gitops-surfaces/.

It defines a single GitOps primitive (GitRepository + Kustomization + a scoped ServiceAccount) and instantiates it at two trust tiers:

  • Platform GitOps (admin, cluster scope) — generalizes feat(platform): add cozystack.customizer system package cozystack#2731, with a two-class Package ownership model (hand-owned bootstrap vs chart-owned downstream) and a packageOverrides admin override layer so component-parameter changes don't race helm-controller for field ownership.
  • Tenant GitOps (tenant, namespace scope) — a catalog gitops app bound to the tenant ServiceAccount, isolated via a dedicated tenant kustomize-controller shard.

Covers admin package/parameter/golden-image configuration, tenant self-service GitOps, and arbitrary cluster resources outside the core API, plus cross-cutting secrets-as-code, DR/reproducibility, and policy-as-code.

Status

Draft — opening for community feedback per the design-proposal process. The first increment (admin surface) is already prototyped in cozystack/cozystack#2731.

Summary by CodeRabbit

  • Documentation
    • Added/updated a design proposal for “GitOps Configuration Surfaces,” defining a unified reconciled surface for admin (cluster-scoped) and tenant (namespace-scoped) workflows.
    • Describes package override behavior, security trust boundaries and tenant isolation controls, status and secrets-as-code concepts, plus upgrade/rollback considerations and an end-to-end testing plan.

Proposes a single GitOps primitive (GitRepository + Kustomization + a
scoped ServiceAccount) instantiated at two trust tiers: platform GitOps
for admins (cluster scope) and tenant GitOps for tenants (namespace
scope), plus a packageOverrides admin override layer. Generalizes
cozystack/cozystack#2731 as the first increment.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a design proposal for GitOps Configuration Surfaces in Cozystack, defining a unified primitive to enable platform GitOps for admins and tenant GitOps for users. The review feedback highlights several critical areas for improvement: first, Helm's default list-merging behavior could inadvertently overwrite default lists (like VM images) when applying packageOverrides; second, a potential security bypass exists where tenants could omit or modify the sharding label on Kustomization resources to run them on the unrestricted global controller; and third, restricting the customizer's write access to cozy-system to protect the supply chain would conflict with the need for admins to manage system-level configurations (like the cozystack-values Secret) in that namespace.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +113 to +121
packageOverrides:
linstor:
autoDiskful:
minutes: 10
vm-default-images:
images:
- name: ubuntu-24.04
url: https://example/custom.img
```

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

When implementing the packageOverrides merge logic in the platform chart, keep in mind that standard Helm merge functions (like merge or mustMerge) overwrite entire lists/arrays (such as the images list under vm-default-images) rather than merging them by key or appending to them. If an admin only wants to override or add a single image, they might inadvertently wipe out all other default images. It would be beneficial to specify in the design how list-type overrides are handled or if custom template merging logic will be used.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The packageOverrides example this comment sits on has been dropped — PackageValues is now the single override layer. The list-merge concern is addressed head-on: the Admin override layer section pins the policy as maps deep-merge, lists replace wholesale (standard Helm semantics), and calls out the vm-default-images images list as exactly this footgun — overriding one image replaces the whole list, so adding one means restating it; keyed/append-merge is an explicit per-package opt-in. (1620d3b)


A tenant must not be able to (a) reconcile into another namespace, or (b) borrow a more privileged SA. Two options, recommendation first:

1. **Dedicated tenant kustomize-controller shard** (recommended) — mirror the existing `flux-tenants` helm-controller shard. Run a kustomize-controller selecting `sharding.fluxcd.io/key: tenants`, started with `--no-cross-namespace-refs` and a default-service-account lockdown, so the restrictions apply *only* to tenant reconciliation and platform Kustomizations are unaffected. This matches a pattern Cozystack already uses.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-medium medium

Using a dedicated tenant kustomize-controller shard that filters by a sharding label (e.g., sharding.fluxcd.io/key: tenants) introduces a potential security bypass. If a tenant can create or modify Kustomization resources directly, they could omit or change this label to have their Kustomization reconciled by the default, unrestricted global kustomize-controller instead. To make this isolation robust, the design should include an admission policy (such as a ValidatingAdmissionPolicy or webhook) that enforces the sharding label on all tenant-created Kustomization resources, or configure the global controller to explicitly ignore resources in tenant namespaces.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reworked in 771ebec. The legacy flux-tenants shard this described is retired (flux-shard-operator now owns helm-controller sharding), so tenant isolation no longer "mirrors" it. Tool 2 now provisions a dedicated per-tenant kustomize-controller + SA (a separate sharding path from the helm pool), plus a fail-closed mutating webhook that stamps the shard label + serviceAccountName and a ValidatingAdmissionPolicy backstop — so a label-less or foreign-SA Kustomization is rejected at admission and never reaches the unrestricted main controller. The enforcement ships with the surface, not as deferred hardening.

Comment on lines +161 to +163
- **Tool 1 (admin):** repo write == cluster-admin, by design and by documentation. The curated ClusterRole in the prototype (patch-not-delete on Packages, no CRDs, etc.) is *defense against accident, not against intent* — anyone with repo write can escalate, so the docs must not over-claim a boundary. New trust surface introduced: an admin git repo + its credentials (protect like a kubeconfig), and a standing, self-healing reconciling credential whose blast radius equals an admin's.
- **Tool 1 footguns to narrow** (they cost nothing functionally): source access should not let the customizer rewrite the platform's *own* supply chain (the `cozystack-packages` OCIRepository in `cozy-system`), so scope write away from `cozy-system`; and field ownership on downstream Packages should eventually be enforced by a `ValidatingAdmissionPolicy` scoped to the customizer SA that rejects changes to fields currently managed by helm-controller (detected via `managedFields`), leaving the hand-owned bootstrap Package free. Until then, the `packageOverrides` layer keeps admins off chart-owned objects entirely, so the contract is mostly avoided rather than merely documented.
- **Tool 2 (tenant):** the surface runs as the tenant's own SA, so it can do nothing the tenant cannot already do via the API. The new risk is **cross-namespace / SA-borrowing**, closed by the tenant shard lockdown (`--no-cross-namespace-refs`, forced `serviceAccountName`). Tenant-supplied input = the contents of a tenant-owned repo, bounded by `cozy:tenant:*`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security-medium medium

There is a potential conflict between scoping the customizer's write access away from cozy-system (to protect the platform's supply chain like cozystack-packages) and the need for admins to manage system-level configurations. As noted in the Context section, critical admin configurations (such as cert-manager issuers or other platform values) are driven by resources residing in cozy-system (e.g., the cozystack-values Secret). If write access to cozy-system is completely blocked, admins won't be able to manage these via GitOps. Consider refining this to use fine-grained RBAC or a ValidatingAdmissionPolicy that specifically protects the supply chain resources (like OCIRepository or specific HelmReleases) while still allowing the customizer to manage other configurations in cozy-system.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch on the tension — resolved deliberately the other way. The customizer is a cluster-admin-equivalent admin tool, so it manages cozy-system (cert-manager issuers, the cozystack-values Secret, etc.) the same way Flux manages its own manifests — intentional, not a boundary to wall off. So rather than scope write away from cozy-system, any supply-chain protection (e.g. the cozystack-packages OCIRepository) is an opt-out guard, not a default exclusion. Updated the Security posture in 1620d3b.

@myasnikovdaniil

Copy link
Copy Markdown
Contributor Author

Revising Tool 1: override layer as a PackageValues resource

Updating the Tool 1 design. The two-class Package ownership model (chart-owned downstream Packages + a hand-owned admin override layer) stays — but I'm changing how the override layer is realized: a dedicated resource rather than a sub-key on the bootstrap Package. It's fully additive / non-breaking. Recording the rationale here; the proposal edit follows.

The problem. Downstream Packages are emitted by the platform chart (packages/core/platform/templates/_helpers.tpl, the cozystack.platform.package helper) with helm.sh/resource-policy: keep, so helm-controller owns their fields. Any out-of-band edit to a chart-owned Package's spec.components[].values races helm-controller for field ownership (kustomize-controller hardcodes force-ownership, so it silently wins — reproducible). Whatever the override layer is, it must not lose that race.

Solution — extend the API with a third resource:

PackageSource   (charts + variants — unchanged)
Package         (the reconciled, system/bundle-owned layer)
PackageValues   (NEW — the human/GitOps-owned override layer)

The operator merges PackageValues over the Package's values when it builds the HelmRelease. The chart never writes PackageValues; humans never write Package. There is no field to fight over — the race is removed structurally. This is not a second override Secret (the operator hardcodes a single valuesFrom: cozystack-values and resets others — internal/operator/package_reconciler.go — so a second Secret loses); PackageValues is a CR the operator itself merges, so it works with the reconciler.

No breaking changes. PackageValues is additive: with none present, the operator path is byte-identical to today (hr.Spec.Values = pkg.Spec.Components[c].Values, with cozystack-values still merged under). effective := deepMerge(packageValues, packageValuesCR) → absent CR ⇒ no change. The one discipline it needs to pay off: humans edit only PackageValues, enforced by an RBAC split (write PackageValues, read-only Package).

Why a resource rather than packageOverrides on the bootstrap Package:

PackageValues (proposed) packageOverrides (alternative)
API change new optional CRD + reconciler merge none (chart-only)
Field-ownership race eliminated (separate object) sidestepped (funnel through chart)
RBAC delegation per-package (write PackageValues, read Package) all-or-nothing (sub-key of one object)
List/map merge (e.g. vm-default-images.images is a 16-entry list) one tested merge in the operator per-package Go-template logic, untested
Override blob one small reviewable resource per package one god-object for all packages
Schema validation validatable per package later free-form JSON

packageOverrides is a reasonable lighter-weight alternative — no Go, ships purely as chart templating — and is essentially PackageValues crammed into one chart-rendered blob. The tradeoff is that it sidesteps the field-ownership race instead of removing it, can't do per-package RBAC delegation, and pushes list-merge correctness into per-package templates. Since PackageValues is additive, shipping packageOverrides first wouldn't block it — but it would make the packageOverrides key throwaway.

Feedback welcome on the precedence chain (chart defaults < cozystack-values < Package < PackageValues) and the RBAC split before I flesh out the CRD.

…layer

Recast Tool 1's override layer from a `packageOverrides` map on the bootstrap
Package (merged into downstream Packages by the platform chart) to a dedicated
`PackageValues` API kind that the operator merges into each HelmRelease.
`packageOverrides` is kept as a lighter-weight chart-only alternative/interim.

PackageValues removes the helm-controller force-ownership race structurally
(the chart never writes PackageValues; the admin never writes the chart-owned
Package), makes ownership enforceable by object-level RBAC, and is fully
additive — absent a PackageValues, the operator builds HelmReleases as today.

Updates the admin override layer, user-facing / upgrade / security / failure /
testing / rollout sections, open question 3 (now PackageValues precedence and
binding), and alternatives considered.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
@coderabbitai

coderabbitai Bot commented Jun 19, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 50ef46dc-abf4-40bc-9679-d5005a6c3510

📥 Commits

Reviewing files that changed from the base of the PR and between 4c47c08 and 6afe9a5.

📒 Files selected for processing (1)
  • design-proposals/gitops-surfaces/README.md

📝 Walkthrough

Walkthrough

Adds a design proposal for Cozystack GitOps surfaces covering admin and tenant trust tiers, override handling with PackageValues, tenant isolation enforcement, security boundaries, testing, rollout, and alternatives.

Changes

GitOps Configuration Surfaces Proposal

Layer / File(s) Summary
Metadata, context, goals, and non-goals
design-proposals/gitops-surfaces/README.md
Document header, overview, related proposals, current Flux context, stated goals, and explicit non-goals bound the proposal scope.
Shared primitive and Tool 1
design-proposals/gitops-surfaces/README.md
Defines the reconciled GitOps surface primitive and describes Tool 1 with the platform/admin trust model, Package ownership boundaries, and self-adoption flow.
PackageValues override layer
design-proposals/gitops-surfaces/README.md
Introduces the PackageValues CR, merge precedence rules, HelmRelease value composition, and status surfacing for merged values and errors.
Tenant GitOps and cross-cutting behavior
design-proposals/gitops-surfaces/README.md
Defines tenant GitOps as a catalog app with per-tenant ServiceAccount and controller isolation, then covers cross-cutting capabilities and upgrade or rollback behavior.
Security trust boundaries and edge cases
design-proposals/gitops-surfaces/README.md
Describes the trust boundaries and enumerates failure cases, tenant routing and label enforcement, uninstall semantics, re-adoption, and git-managed UI edit behavior.
Testing, rollout, open questions, and attribution
design-proposals/gitops-surfaces/README.md
Lists the testing strategy, rollout phases, resolved open questions, alternatives considered, and attribution footer.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related issues

Suggested reviewers: kvaps, lllamnyp

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: adding a GitOps configuration surfaces design proposal document.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch daniil/gitops-surfaces-proposal

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@design-proposals/gitops-surfaces/README.md`:
- Around line 130-131: The proposal currently presents RBAC enforcement as a
mechanism that prevents field-level edits, but this is actually a
human-discipline contract with no API-level enforcement. Revise lines 130-131 to
clarify that the split design structurally avoids the contract violation (by
separating concerns into PackageValues and Package) rather than enforcing it via
RBAC alone. Additionally, update the text to explicitly state that the admission
policy is a prerequisite for production use, not an optional enhancement. Make
clear that without field-ownership enforcement at the API level, admins must
manually follow the documented protocol, and any deviation recreates the
helm-controller race conditions the PackageValues design was meant to prevent.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: d7a0a413-b6cb-49ef-ba12-c1bd9336da65

📥 Commits

Reviewing files that changed from the base of the PR and between abb40d0 and 9faf7e0.

📒 Files selected for processing (1)
  • design-proposals/gitops-surfaces/README.md

Comment thread design-proposals/gitops-surfaces/README.md Outdated
@myasnikovdaniil myasnikovdaniil self-assigned this Jun 23, 2026

@IvanHunters IvanHunters left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes — solid proposal, but the tenant-tier isolation argument has a real hole that should be resolved before this leaves Draft.

  1. The tenant shard is not self-securing. Flux's main controllers run with --watch-label-selector=!sharding.fluxcd.io/key (confirmed in cozystack's own fluxcd manifests), so a Kustomization created without the shard label is reconciled by the main, unrestricted controller — no cross-namespace-ref block, no forced SA. A tenant with cozy:tenant:admin can create Kustomizations and evade the shard by omitting the label or setting a foreign spec.serviceAccountName. The shard needs an admission policy enforcing the label + SA on tenant-namespace Kustomizations as a prerequisite, not deferred hardening. (A tenant confined to its own SA is still bounded by that SA's RBAC — but nothing forces it onto its own SA once the main controller is in play.)
  2. The PackageValues field-ownership contract is discipline, not enforcement, until the ValidatingAdmissionPolicy lands: kustomize-controller always applies with client.ForceOwnership (not gated by spec.force), so an admin editing a chart-owned Package field silently steals it from helm-controller. The doc reads as "structurally removed" in one section and "advisory until policy lands" in another — please make that consistent and treat the admission policy as a production prerequisite.
  3. Open Question 1 is load-bearing: both the operator-embedded internal/fluxinstall controllers and a flux-operator-managed FluxInstance exist in the tree, and the legacy flux-tenants shard is being actively retired by a new flux-shard-operator — so "mirror the flux-tenants pattern" mirrors something on its way out. Which instance serves tenant Kustomizations, and where the lockdown flags live, needs resolving before Tool 2 is implementable.

All Flux/Helm behavior claims I checked are accurate. Minor: the cozy-system write-scoping tension (admin config lives there) deserves an explicit answer, and the list-vs-map merge policy (Open Q3) needs pinning down.

@IvanHunters IvanHunters dismissed their stale review July 4, 2026 20:43

Dismissing the review verdict: converting this to non-blocking feedback since this is a design proposal, not code. The technical points in the review comment still stand as suggestions to reconcile before implementation.

myasnikovdaniil and others added 3 commits July 6, 2026 18:08
…ator

The flux-tenants Deployment is retired; helm-controller sharding is now
driven by flux-shard-operator (an auto-scaled shard pool), and there is
no kustomize-controller sharding. Rework Tool 2 and Context accordingly:

- Context: describe the embedded flux-aio control plane and
  flux-shard-operator; note tenants hold no Flux-CRD RBAC today, so
  Tool 2 is what creates the surface (and thus the isolation problem).
- Tool 2 isolation: a dedicated kustomize-controller + SA per
  gitops-enabled tenant (a separate sharding path, not the helm-controller
  pool), a fail-closed mutating webhook that stamps the shard label +
  serviceAccountName, and a ValidatingAdmissionPolicy backstop. Enforced by
  construction and shipped with the surface, not deferred hardening.
- Resolve Open Question 1: management-plane tenant reconciliation is the
  embedded flux-aio; the FluxInstance/multitenant lever governs child
  Kubernetes clusters only (migration 21 retired it here).
- Disambiguate the deferred Tool 1 Package field-ownership VAP from the
  in-scope Tool 2 Kustomization admission enforcement.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
…verrides

Address the remaining review points as deliberate design decisions:

- Package field-ownership: the direct-edit path on chart-owned Packages
  stays open by design (it is a cluster-admin tool; locking it risks
  removing tuning knobs admins can reach today). PackageValues is the
  recommended non-racing path, not a cage; the field-ownership VAP is an
  optional opt-out guard, never a prerequisite. Reconcile the
  "structurally removed" vs "advisory" wording toward this.
- cozy-system: do not scope customizer writes away from cozy-system —
  same logic as Flux managing its own manifests. Supply-chain guards are
  opt-out, not a default namespace exclusion.
- PackageValues merge policy fixed (maps deep-merge, lists replace
  wholesale; the vm-default-images image list is the footgun).
  packageOverrides is dropped in favor of PackageValues as the single
  override mechanism.
- Resolve Open Question 4: tenant GitOps ships as a catalog app, not a
  Tenant CRD field.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
…ted apps

Tenants keep the curated Application abstraction: the GitOps surface grants
the source/kustomize authoring needed to reconcile, but not raw HelmRelease
or arbitrary-CR create rights. Since the per-tenant kustomize-controller runs
as the tenant SA, "BYO from git" means Application CRs from git, not arbitrary
HelmReleases — the tenant trust model is identical to today.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@design-proposals/gitops-surfaces/README.md`:
- Around line 206-207: The PackageValues binding contract is still ambiguous
because this section reopens the choice between 1:1-by-name and spec.packageRef
while the earlier example and failure mode already assume name binding. Update
the design text in the PackageValues section to make the binding rule explicit
and consistent, or mark the earlier example as provisional, so the README does
not imply two different contracts. Use the PackageValues heading and the binding
discussion around packageOverrides as the anchor for the edit.
- Line 184: The current guidance in the PackageValues behavior is to treat an
orphaned PackageValues object as a no-op, but this should be changed to surface
an explicit failure instead. Update the reconcile/validation handling around
PackageValues and the Package merge path so a missing target Package records a
status/validation error or fails reconciliation rather than silently succeeding.
Use the PackageValues and Package reconciliation logic to locate the
orphaned-object check and replace the no-op behavior with an explicit error
path.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: e932be38-f262-4573-b0d1-8f953447b8a4

📥 Commits

Reviewing files that changed from the base of the PR and between 9faf7e0 and 4c47c08.

📒 Files selected for processing (1)
  • design-proposals/gitops-surfaces/README.md

Comment thread design-proposals/gitops-surfaces/README.md Outdated
Comment thread design-proposals/gitops-surfaces/README.md Outdated
@myasnikovdaniil

Copy link
Copy Markdown
Contributor Author

IvanHunters thanks — this was the review that reshaped the proposal. Pushed three commits (771ebec, 1620d3b, 4c47c08):

#1 (tenant shard not self-securing) & #3 (Open Q1 / flux-tenants retiring): fully reworked. You're right that flux-tenants is on its way out and helm-controller sharding is now flux-shard-operator's auto-scaled pool — "mirror flux-tenants" pointed at something being retired. Tool 2 now delivers a dedicated kustomize-controller + SA per gitops-enabled tenant (a separate sharding path, not the helm pool), a fail-closed mutating webhook that stamps the shard label + serviceAccountName, and a ValidatingAdmissionPolicy backstop. The "omit the label → unrestricted main controller" bypass is closed at admission, by construction, and ships with the surface — not deferred hardening. Open Q1 is resolved in-doc: the management plane is the embedded flux-aio (internal/fluxinstall); the FluxInstance/multitenant lever governs child clusters only (migration 21).

#2 (PackageValues = discipline, not enforcement): you're right the wording contradicted itself, and I reconciled it. But I've landed it the other way from "VAP as prerequisite": for the admin Package path we deliberately keep direct edits on chart-owned Packages open — it's a cluster-admin-equivalent tool, and fencing those fields by default risks silently removing tuning knobs admins reach today. So PackageValues is the recommended non-racing path (not a cage), and the field-ownership VAP is an optional opt-out guard. (Distinct from the Tool 2 tenant-Kustomization VAP above, which is a prerequisite.)

Minors: cozy-system write-scoping — kept permissive by design (same logic as Flux managing its own manifests; supply-chain guard is opt-out, not a default exclusion). List-vs-map merge — pinned (maps deep-merge, lists replace wholesale, vm-default-images called out); packageOverrides dropped in favor of PackageValues as the single override layer.

Also resolved Open Q2 (tenants stay on curated Applications — no raw HelmRelease/CR rights) and Q4 (catalog app). Thanks again for the flux-shard-operator pointer — that was the load-bearing correction.

@kvaps Andrei Kvapil (kvaps) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks myasnikovdaniil — this is a well-researched proposal. I verified the factual claims against the codebase (fluxinstall label selectors, migrations 21/44, package_reconciler.go values handling, tenant RBAC) — all accurate.

PackageValues is the right call, and it solves more than the doc claims. Users running their own GitOps operators (ArgoCD, plain Flux) already hit this today: the only way to override a bundle-shipped Package is delete-and-recreate it from their repo. PackageValues fixes exactly that, and — since the merge happens in the cozystack-operator — it works with any GitOps tooling, independent of the customizer. Please add this case to the problem statement and state that independence explicitly.

1. Materialize the override into the Package itself (single source of truth, loud conflicts). Rather than merging PackageValues invisibly at HelmRelease build time, have the operator SSA-apply its content onto the matching Package's spec.components.*.values under a dedicated field manager, without force. This buys several things at once:

  • Package becomes the single materialized source of truth — the HR build path keeps reading Package only, no hidden merge layer;
  • in the common case the chart renders Packages with no values at all (_helpers.tpl emits only variant), so the operator owns the keys cleanly; where the chart does render a key, ownership is tracked per map key, so chart-owned and admin-owned keys coexist in one object;
  • a genuine collision on the same key fails the PackageValues reconcile with an SSA conflict surfaced as a status condition — chart-rendered intent wins, the admin gets a loud error instead of a silent race in either direction (note this inverts the currently fixed Package < PackageValues precedence — deliberately);
  • deleting a PackageValues (or a key from it) prunes the operator-owned fields and the Package reverts to chart state; an orphaned PackageValues (typo, missing Package) fails the apply and surfaces the same way — no silent no-op.

List semantics stay as already pinned (untyped JSON lists are atomic under SSA — replace wholesale). One caveat to document: helm-controller writes via helm's three-way merge, not SSA, so "chart wins" materializes as helm overwriting a contested key on a chart change and the operator's next apply failing loudly — an acceptable, self-announcing path.

2. Document the admin repo layout. The platform only consumes customizer.source.* + kustomization.path; everything inside the path is unspecified. Please add a reference repo structure to the proposal (and ideally an example repo before GA) — otherwise every operator invents their own, and the "environment overlays" use case stays theoretical.

3. Tenant surface: make it a parent-gated singleton, not a catalog app. Open Q4 considered "catalog app vs Tenant CRD field" — there's a middle ground that fits existing machinery: an extra module with singularResource: true (like monitoring/etcd/seaweedfs), enabled by the parent tenant (as with etcd: true today), with the repo settings then edited on the deployed instance. This gives a clean 1:1 between tenant and its dedicated kustomize-controller, avoids "5 GitOps instances = 5 controllers?" ambiguity, and puts controller-count under parent/admin control — which also answers the capacity question for large installations. Multiple repos, if ever needed, can be a list inside the single instance.

4. Bind the tenant surface to a dedicated narrow SA. Rather than a SA "carrying cozy:tenant:*", a dedicated <tenant>-gitops SA with write only on apps.cozystack.io (+ read for status) — the git path then has a strictly narrower surface than an interactive user, and the "git can only produce Application CRs" guarantee becomes explicit RBAC rather than a property of the current role set.

5. Define the dashboard ↔ git story. The surface is one-way: once an Application is git-managed, a dashboard edit gets silently reverted by drift correction on the next reconcile — a confusing UX the doc doesn't mention. Minimum: the dashboard marks git-managed Applications read-only (detectable via field manager or a label). Better: consider a two-way model where dashboard edits become commits/PRs to the tenant repo — the ReverseGitOps approach Simon Koudijs (@sunib) demoed at the last Cozystack meetup (reversegitops.dev). Simon, curious about your take here.

6. Gap: the source path is outside the isolation model (blocking for Tool 2). The dedicated-controller design covers kustomize-controller only. Tenant GitRepository objects, being label-less for source-controller, land on the shared main source-controller in cozy-fluxcd: (a) a platform component outside tenant NetworkPolicies now fetches tenant-supplied URLs (SSRF surface); (b) shared-controller DoS via large repos / aggressive intervals; (c) most importantly, source-controller serves artifacts over unauthenticated HTTP — so the per-tenant kustomize-controller's placement matters: if it runs inside the tenant namespace, the NetworkPolicy opening it to artifact storage opens every tenant pod to all artifacts, including platform manifests and other tenants' repos. The proposal should state who reconciles tenant sources, where the dedicated controller runs (platform namespace, I'd argue), and how artifact storage stays closed to tenant workloads.

7. Admission details to pin. The webhook/VAP pair must match CREATE and UPDATE (create-valid-then-strip-the-label is otherwise an escape), and the VAP must validate the shard label value equals the tenant's own shard — not mere presence — or tenant A can point its Kustomization at tenant B's controller.

8. Prune cascade on uninstall. Deleting the tenant GitOps instance deletes its Kustomization; with prune enabled, the finalizer then deletes everything it created — i.e. the tenant's Application CRs, i.e. their databases and VMs. Please add this to Failure and edge cases; I'd default the tenant surface to deletionPolicy: Orphan.

Overall: direction is right, Tool 1 + PackageValues can proceed independently (as the rollout already stages them); for Tool 2 I'd treat #6 and #7 as part of the "enforcement ships with the surface" package the doc itself commits to.

… isolation, PackageValues visibility

Incorporate @kvaps review feedback:

- Bind the tenant surface to a dedicated narrow `<tenant>-gitops` SA
  (write on apps.cozystack.io only) instead of reusing cozy:tenant:*.
- Close the source-controller gap: run the per-tenant kustomize-controller
  platform-side and deny tenant egress to the unauthenticated artifact
  endpoint; bound shared-source DoS via interval/size limits.
- Tighten admission: webhook + VAP match CREATE and UPDATE, and the VAP
  validates the shard-label value equals the tenant's own shard.
- Default the tenant Kustomization to deletionPolicy: Orphan so uninstall
  does not cascade-delete tenant apps.
- Add dashboard read-only for git-managed Applications (one-way drift).
- PackageValues: keep build-time merge + admin precedence; add Package
  status visibility and loud handling of orphaned PackageValues; record
  the SSA materialize-onto-Package variant in Alternatives.
- Keep tenant delivery as a self-service catalog app with multi-source;
  record the parent-gated extra-module singleton in Alternatives.
- Add the bring-your-own-GitOps (ArgoCD / plain Flux) case to the problem
  statement; note PackageValues is operator-side and tooling-independent.
- Document a reference admin-repo layout deliverable.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
@myasnikovdaniil

Copy link
Copy Markdown
Contributor Author

Thanks Andrei Kvapil (@kvaps) — extremely useful review. I verified each mechanism against the tree and Flux 2.8 before responding; most of this is now in the doc (pushed in the latest commit). One factual update up front that strengthens your #1: helm-controller v1.5.0 does default to server-side apply (install true, upgrade auto→inherit), and Package.spec.components.*.values is x-kubernetes-preserve-unknown-fields (per-key ownership) — so your materialize-onto-Package model is even cleaner than your "helm three-way merge" caveat assumed.

#1 — materialize onto Package / precedence. Agreed it's clean, and I've recorded it in Alternatives as a first-class option. I'm keeping the build-time merge as the default, though, for one reason: "no-force SSA onto the Package" necessarily makes the chart win any genuine overlap, which pushes the admin back onto the racy direct-edit path for exactly the keys they were trying to override; making the admin win while materializing would require force — the ForceOwnership race this proposal exists to remove. Merging one layer below (hr.Spec.Values) is the only place admin-authority and race-freedom coexist. I did adopt your two orthogonal wins: effective merged values are surfaced on Package status (single place to read), and an orphaned PackageValues is now a loud status condition, not a silent no-op. If the project decides it prefers chart-authoritative overrides with loud conflicts over admin-authoritative ones, the materialize variant is a drop-in — they differ only in who wins an overlap and where you read the result.

#2 — repo layout. Added; a reference layout + example repo is a pre-GA deliverable.

#3 — parent-gated singleton extra module. I'd like to keep it a self-service catalog app rather than a parent-gated module. The capacity concern behind the singleton is real, but it's dissolved by letting a single instance take a list of sources — one instance ⇒ one dedicated controller per gitops-enabled tenant, with a ResourceQuota/admission cap where an admin wants a hard limit. One mechanism note: singularResource in Cozystack is a dashboard/UI hint (no app sets it), so it wouldn't actually enforce a singleton — the extra module would still be plain chart gating, at the cost of self-service.

#4 — narrow SA. Adopted. The surface now binds a dedicated <tenant>-gitops SA with write on apps.cozystack.io only (the interactive tenant SA also carries sdn.cozystack.io/securitygroups write via cozy:tenant:base, which the git path doesn't need). "Git can only produce Application CRs" is now explicit RBAC.

#5 — dashboard ↔ git. Added: git-managed Applications go read-only in the dashboard (detected via field manager/label); the two-way ReverseGitOps model is noted as the intended future direction.

#6 — source path (blocking). Agreed this was the gap. Added: tenant GitRepositorys reconcile on the shared platform source-controller (no source sharding), so (a) the per-tenant kustomize-controller runs platform-side in cozy-fluxcd, (b) tenant namespaces are denied egress to the unauthenticated artifact endpoint (flux.cozy-fluxcd) via Cilium policy, and (c) GitRepository interval/size limits bound the shared-source DoS. Per-tenant source-controller stays a future option.

#7 — admission. Fixed: webhook and VAP both match CREATE and UPDATE, and the VAP validates the shard-label value equals the tenant's own shard (presence alone would let A point at B's controller).

#8 — prune cascade. Agreed — tenant Kustomization now defaults to deletionPolicy: Orphan so uninstalling GitOps doesn't garbage-collect the tenant's databases/VMs; cascading teardown is explicit opt-in. Added to Failure/edge cases.

Net: #4#8 + the BYO-GitOps problem framing are in; #1 and #3 are addressed as deliberate design choices with the alternatives recorded. Tool 1 + PackageValues can still proceed independently as staged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants