feat: add sealed secrets as platform secrets documentation

docs/for-devs/console/code-repositories.md

@@ -44,7 +44,7 @@ All registered code repositories of the Team are listed here.
 
 2. Optionally: Select `Private` If the repository in GitHub is private.
 
-3. Optionally: Select a secret that contains the authentication credentials. Only Sealed Secrets of type `basic-auth` and `ssh-auth` will be shown. If no secret is available, then first [create a Sealed Secret](sealed-secrets.md). While creating the `basic-auth` secret containing Github token, enter any string in the username input field and paste the token in the password input field.
+3. Optionally: Select a secret that contains the authentication credentials. Only Sealed Secrets of type `basic-auth` and `ssh-auth` will be shown. If no secret is available, then first [create a Sealed Secret](secrets.md). While creating the `basic-auth` secret containing Github token, enter any string in the username input field and paste the token in the password input field.
 
 4. Optionally: Test the connection to see if the authentication credentials are valid and App Platform can use the credentials to access the private repository.
 

docs/for-ops/console/secrets.md

@@ -0,0 +1,49 @@
+---
+slug: secrets
+title: Platform Secrets
+sidebar_label: Secrets
+---
+
+The **Secrets** page lists all platform secrets managed by Sealed Secrets. Platform administrators can create new secrets and update existing ones. Secret values are write-only — they are encrypted immediately on save and are never displayed in plaintext.
+
+Platform secrets are encrypted using [Bitnami Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets) and distributed to applications via the [External Secrets Operator](https://external-secrets.io) (ESO). No secrets are stored in plaintext on disk or in Git.
+
+## How Platform Secrets Work
+
+1. All sensitive configuration values (passwords, API tokens, TLS keys) are encrypted into `SealedSecret` manifests and stored in the values Git repository.
+2. The sealed-secrets controller running in the cluster decrypts them into standard Kubernetes Secrets in the `apl-secrets` namespace.
+3. ESO reads from `apl-secrets` via a `ClusterSecretStore` and distributes the secrets to each application's namespace using `ExternalSecret` resources.
+
+This means the values Git repository contains only encrypted manifests, never plaintext secrets.
+
+## Secret Lifecycle
+
+### On Installation
+
+During the bootstrap phase the platform:
+
+1. Generates an RSA-4096 key pair for the sealed-secrets controller.
+2. Encrypts all sensitive configuration values into `SealedSecret` manifests.
+3. Writes the encrypted manifests to the values repository.
+4. Deploys the sealed-secrets controller, which decrypts the manifests into Kubernetes Secrets in the `apl-secrets` namespace.
+5. Deploys ESO and the `ClusterSecretStore`, which distributes secrets to application namespaces via `ExternalSecret` resources.
+
+### On Runtime
+
+The operator polls the values Git repository for changes. When a change is detected:
+
+1. Helmfile applies updated `ExternalSecret` resources.
+2. ESO syncs the corresponding Kubernetes Secrets to each application's namespace.
+3. ESO refreshes all secrets on a hardcoded one-hour interval. Secret updates take up to one hour to propagate to application namespaces.
+
+## The Sealed Secrets Key Pair
+
+The RSA key pair used to encrypt and decrypt all platform secrets is stored as a Kubernetes TLS secret named `sealed-secrets-key` in the `sealed-secrets` namespace. The sealed-secrets controller uses this key to decrypt `SealedSecret` manifests on the cluster.
+
+:::note
+If this key pair is lost and cannot be recovered, all `SealedSecret` manifests become undecryptable. Back up the key pair immediately after installation and store it securely outside the cluster.
+:::
+
+### Back Up the Key Pair
+
+See [Sealed Secrets Key Recovery](../disaster-recovery/sealed-secrets-key.md) for the export command and instructions on how to use this backup to restore a cluster.

docs/for-ops/console/settings/key-management.md

@@ -4,6 +4,10 @@ title: Platform settings
 sidebar_label: Key Management
 ---
 
+:::caution
+SOPS-based key management was removed in v6.0.0. Upgrading from v5.x to v6.0.0 migrates your secrets automatically. See [Platform Secrets](../secrets.md) for the current approach.
+:::
+
 :::info
 The Key Management section in the Settings will NOT be visible when the installation is done by Akamai Cloud. In this case Age is used as KMS.
 :::

docs/for-ops/console/settings/platform.md

@@ -15,4 +15,8 @@ The Platform settings section offers configuration options for platform and feat
 | External DNS  | Set this to true when an external DNS zone is available to manage DNS records. (Expects required `dns:` fields to be set.) |
 | External IdP | Set this to true when bringing your own external IDP such as Azure AD. (Expects required `oidc:` fields to be set.)  |
 | Node Selector | When a label/value pair is added, all platform workloads will be scheduled on the nodes with this label/value pair |
-| Version | The installed version. Change to a new valid release to upgrade. see the [core repo](https://github.com/linode/apl-core) for all available versions |
+| Version | The installed version. Change to a new valid release to upgrade. see the [core repo](https://github.com/linode/apl-core) for all available versions |
+
+:::note
+Sensitive values configured here, such as the global pull secret, are encrypted using Sealed Secrets and stored securely in the values repository. No secrets are stored in plaintext on disk or in Git. See [Platform Secrets](../secrets.md) for more information.
+:::

docs/for-ops/disaster-recovery/overview.md

@@ -13,7 +13,7 @@ This guide has the following prerequisites and limitations that should be checke
 1. The following items should be backed up regularly by the platform administrator:
 
 - The Kubernetes secret ending in "-wildcard-cert" in namespace "istio-system" (if installed via the Linode cloud console, or using your own certificate).
-- The Kubernetes secret "otomi-sops-secrets" in namespace "otomi-pipelines".
+- The `sealed-secrets-key` Kubernetes secret in namespace `sealed-secrets`. This contains the RSA key pair required to decrypt all platform secrets. See [Sealed Secrets Key Recovery](sealed-secrets-key.md) for export instructions.
 - A download of the complete values in Platform -> Maintenance. Depending on whether these are downloaded with or without secrets, some passwords might have to be reset after recovery.
 - Optionally manual backups of databases, as covered in this guide for the CloudNative PostgreSQL Operator, should be taken.
 
@@ -33,20 +33,22 @@ This guide has the following prerequisites and limitations that should be checke
 
 When using an external Git repository (BYO Git), disaster recovery is significantly simplified. The platform configuration is already stored externally, so Gitea backup and restore is not required. To restore the platform, you only need:
 
-- The age keys (`privateKey` and `publicKey`) used for SOPS encryption.
+- The sealed-secrets key pair from the original cluster.
 - Access credentials to the external Git repository.
 
-Make sure to store the age `privateKey` securely outside the cluster.
-The `privateKey` can be retrieved from the cluster with the following command:
+Export the sealed-secrets key pair before decommissioning the cluster:
 
 ```bash
-kubectl get secret apl-sops-secrets -n apl-operator -o jsonpath='{.data.SOPS_AGE_KEY}' | base64 --decode
+kubectl get secrets -n sealed-secrets \
+  -l sealedsecrets.bitnami.com/sealed-secrets-key=active \
+  -o yaml > sealed-secrets-key.yaml
 ```
 
-See the [BYO Git installation guide](../../get-started/installation/byo-git.md) for more details.
+Store the exported file securely outside the cluster. See the [BYO Git installation guide](../../get-started/installation/byo-git.md) and [Recovery Installation](../../get-started/installation/recovery.md) for more details.
 
 ## Guides
 
 - [Gitea](gitea.md): Restoring the platform's Gitea database and repositories from the application backup
 - [Databases](platform-databases.md): Backup and restore of the CNPG databases
 - [Reinstall](platform-reinstall.md): Restoring the complete platform, including settings and data
+- [Sealed Secrets Key](sealed-secrets-key.md): Back up and restore the sealed secrets key pair used to encrypt all platform secrets

docs/for-ops/disaster-recovery/sealed-secrets-key.md

@@ -0,0 +1,78 @@
+---
+slug: sealed-secrets-key
+title: Sealed Secrets Key Recovery
+sidebar_label: Sealed Secrets Key
+---
+
+The sealed-secrets RSA key pair is the root of trust for all platform secrets. It is stored as a Kubernetes TLS secret named `sealed-secrets-key` in the `sealed-secrets` namespace. The sealed-secrets controller uses this key to decrypt all `SealedSecret` manifests in the cluster.
+
+If the key pair is lost and cannot be recovered, all `SealedSecret` manifests become undecryptable and the platform must be reinstalled with fresh secrets.
+
+## Back Up the Key Pair
+
+Back up the key pair immediately after installation and store it securely outside the cluster.
+
+Run the following command to export the key manifest:
+
+```bash
+kubectl get secrets -n sealed-secrets \
+  -l sealedsecrets.bitnami.com/sealed-secrets-key=active \
+  -o yaml > sealed-secrets-key.yaml
+```
+
+Store `sealed-secrets-key.yaml` securely — for example, in a password manager or encrypted storage in a different location from the cluster.
+
+## Restore the Key Pair on a New Cluster
+
+Use this procedure when the cluster has been destroyed but you have:
+
+- A BYO Git repository containing the existing values and `SealedSecret` manifests.
+- A backup of the sealed-secrets key pair.
+
+### Steps
+
+1. Start from the values file used for the original BYO Git installation.
+
+2. Add `installation.mode: recovery` and embed the contents of `sealed-secrets-key.yaml` under `installation.recovery.manifests`:
+
+```yaml
+installation:
+  mode: recovery
+  recovery:
+    manifests:
+      apiVersion: v1
+      items:
+        - apiVersion: v1
+          data:
+            tls.crt: <base64-encoded certificate>
+            tls.key: <base64-encoded private key>
+          kind: Secret
+          metadata:
+            labels:
+              sealedsecrets.bitnami.com/sealed-secrets-key: active
+            name: sealed-secrets-key
+            namespace: sealed-secrets
+          type: kubernetes.io/tls
+      kind: List
+      metadata:
+        resourceVersion: ""
+```
+
+The values for `tls.crt` and `tls.key` come directly from the `sealed-secrets-key.yaml` export — they are already base64-encoded.
+
+3. Install the platform:
+
+```bash
+helm install apl apl/apl -f values.yaml
+```
+
+During installation the operator provisions the sealed-secrets key before bootstrap. The sealed-secrets controller then decrypts the existing `SealedSecret` manifests from the values repository using the restored key, and ESO distributes the secrets to application namespaces.
+
+## Recovery Scenarios
+
+| Scenario | Recovery Path |
+| --- | --- |
+| Cluster destroyed, key pair backed up, values repo (BYO Git) intact | Use `installation.mode: recovery` with the backed-up key pair (this guide) |
+| Key pair lost, values repo intact | Reinstall with fresh secrets; all platform passwords will reset |
+| Values repo lost, key pair intact | Restore the values repo from a backup, then reinstall using `installation.mode: recovery` |
+| Both key pair and values repo lost | Reinstall with fresh secrets; all platform passwords will reset |

docs/for-ops/how-to/change-admin-password.md

@@ -4,6 +4,10 @@ title: Change the otomi-admin password
 sidebar_label: Change admin password
 ---
 
+:::caution
+This page describes the SOPS-based workflow for updating the admin password, which was removed in v6.0.0. Upgrading from v5.x to v6.0.0 migrates your secrets automatically. See [Update the otomi-admin password](update-admin-password.md) for the current guide.
+:::
+
 This how to provides step-by-step instructions to reset the `otomi-admin` password.
 
 ## Prerequisites:

docs/for-ops/how-to/manage-age.md

@@ -4,6 +4,10 @@ title: Manage Age for Development, Security, and Recovery
 sidebar_label: Manage Age
 ---
 
+:::caution
+SOPS-based key management was removed in v6.0.0. Upgrading from v5.x to v6.0.0 migrates your secrets automatically.
+:::
+
 :::info
 This documentation was written for SOPS that is using Age Encryption. Please check the SOPS page for other [KMS](/docs/get-started/installation/sops#use-sops-with-an-external-key-management-service-kms) providers.
 :::

docs/for-ops/how-to/update-admin-password.md

@@ -0,0 +1,72 @@
+---
+slug: update-admin-password
+title: Update the otomi-admin password
+sidebar_label: Update admin password
+---
+
+This guide provides step-by-step instructions to update the `otomi-admin` password.
+
+## Prerequisites
+
+1. Access to the Keycloak admin console.
+2. Access to the platform console (platform view).
+
+## Steps
+
+### Step 1: Generate a New Password
+
+```bash
+# Generate a random 16-character alphanumeric password
+head /dev/urandom | LC_ALL=C tr -dc A-Za-z0-9 | head -c 16
+# Example output: FPpLvZyAdAmuzc3N
+```
+
+### Step 2: Update the Password in Keycloak
+
+1. Log in to the Keycloak admin console using the platform admin credentials.
+
+2. Go to **Users** in the left-hand sidebar and find the `otomi-admin` user.
+
+3. Navigate to the **Credentials** tab.
+
+4. Change the password to the one you generated.
+
+5. Save changes (set Temporary to Disabled).
+
+:::info
+Don't make any other changes during this operation.
+:::
+
+### Step 3: Update the Password in the Platform Console
+
+1. Open the platform console and go to the **Platform** view.
+
+2. Navigate to **Secrets** in the left-hand sidebar.
+
+3. Click on `otomi-secrets` (namespace: `apl-secrets`).
+
+4. Find the `adminPassword` field in the **Secret Data** section 
+
+5. Click on the lock `🔒` icon and enter the new password in **plain text**.
+
+6. Click **Save Changes**.
+
+The platform encrypts the value using Sealed Secrets and commits the updated manifest to the values repository. No manual encryption is required.
+
+:::info
+Changes become active after the pipeline finishes applying the updated secrets.
+:::
+
+### Step 4: Verify the Changes
+
+1. Allow the pipeline to run and verify it completes successfully.
+
+2. Restart the `apl-keycloak-operator` deployment to ensure the new credentials are applied:
+
+```bash
+kubectl rollout restart deployment -n apl-keycloak-operator apl-keycloak-operator
+```
+
+3. Verify that the platform is working as expected with the new credentials.
+
+This completes the password update process for `otomi-admin`.

docs/for-ops/how-to/use-the-cli.md

@@ -57,6 +57,10 @@ otomi --help
 
 To find the options available.
 
+:::caution
+`otomi decrypt`, `otomi encrypt`, and `otomi rotate-keys` are SOPS-based commands that were removed in v6.0.0. Upgrading from v5.x to v6.0.0 migrates your secrets automatically.
+:::
+
 ## Command Reference
 
 | Command                     | Description                                                                                          |

docs/get-started/installation/byo-git.md

@@ -15,7 +15,7 @@ By default, the App Platform installs Gitea as the built-in Git repository to st
 ## Advantages of BYO Git
 
 - Use your existing Git workflows and access controls.
-- Simplified disaster recovery: the platform configuration is stored externally, so you only need to back up your age keys to restore the platform.
+- Simplified disaster recovery: the platform configuration is stored externally, so you only need to back up the sealed-secrets key pair to restore the platform.
 - No need to manage and back up the built-in Gitea instance.
 
 ## Configure the values
@@ -39,9 +39,6 @@ dns:
   provider:
     linode:
       apiToken: '<your-linode-api-token>'
-kms:
-  sops:
-    provider: age
 otomi:
   hasExternalDNS: true
   git:
@@ -52,6 +49,10 @@ otomi:
     branch: main
 ```
 
+:::note
+For v5.x clusters, also add the SOPS configuration to your `values.yaml`. See [SOPS](sops.md) for the required `kms.sops` settings.
+:::
+
 ### Git configuration options
 
 | Parameter            | Description                                    |
@@ -64,45 +65,11 @@ otomi:
 
 ## Disaster recovery with BYO Git
 
-Since the configuration parameters (the values repository) is stored outside the cluster the disaster recover process is straightforward. In the recovery mode the `values.yaml` file must contian the following parameters:
-
-- `kms.sops.age` with `privateKey` and `publicKey`
-- `otomi.git` configuration options
-- `installation.mode` set to `recovery`
-
-For example:
-
-```yaml
-cluster:
-  name: your-cluster-name
-  provider: linode
-otomi:
-  git:
-    repoUrl: https://github.com/<owner>/<repo>
-    username: <git-username>
-    password: <personal-access-token>
-    email: <git-email>
-    branch: main
-kms:
-  sops:
-    age:
-      privateKey: '<your-age-private-key>'
-      publicKey: '<your-age-public-key>'
-    provider: age
-installation:
-  mode: recovery
-```
-
-The following command can be used to re-install the App Platform:
-
-```bash
-helm install -f values.yaml apl apl/apl
-```
-
-> Make sure to store your age keys securely outside of the cluster (e.g. in a password manager or secrets vault). Without them you won't be able to decrypt the secrets stored in the git repository.
+Since the values repository is stored outside the cluster, disaster recovery is straightforward. You only need the sealed-secrets key pair from the original cluster and your Git credentials to restore the platform on a new cluster.
 
-> This procedure works out of the box for App Platform instancies that manage their own DNS records (via external-dns and cert manager).
+See [Recovery Installation](recovery.md) for step-by-step instructions and [Sealed Secrets Key Recovery](../../for-ops/disaster-recovery/sealed-secrets-key.md) for how to back up and restore the key pair.
 
-> This procedure does not cover the recovery of databases used by Gitea and/or Harbor.
+:::note
+For v5.x clusters, disaster recovery requires your age keys instead of the sealed-secrets key pair. See [Manage Age](../../for-ops/how-to/manage-age.md) for the v5.x recovery procedure.
+:::
 
-See the [disaster recovery documentation](../../for-ops/disaster-recovery/overview.md) for the full procedure.