diff --git a/CHANGELOG.md b/CHANGELOG.md index ece6af4..e7e020e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,10 @@ All notable changes to RustFS Operator are documented in this file. The format i ### Added +- Tenant `spec.oidc.extraCaCertSecretRef` for validated, reloadable custom CA trust used by + RustFS OIDC outbound connections. +- Tenant `spec.additionalVolumes` and `spec.additionalVolumeMounts` for Kubernetes-native file + injection into every RustFS Pool workload. - Tenant `spec.network` for Service IP families and IPv6 listen addresses, plus dual-stack binds for operator observability, STS, and Console sockets. - Tenant `spec.hostUsers` and OpenShift `hostUsers: false` defaults for `restricted-v3`. @@ -38,6 +42,7 @@ All notable changes to RustFS Operator are documented in this file. The format i ### Fixed - Reject multi-pool Tenants containing a single-node single-disk pool before applying pool workloads, matching RustFS startup constraints. +- Reject additional volume mounts that overlap operator-managed data, logging, TLS, or OIDC paths. - Explicit private bucket access now removes operator-managed policies, and primary Service IP family changes recreate managed Services instead of repeatedly failing immutable-field updates. @@ -46,6 +51,8 @@ All notable changes to RustFS Operator are documented in this file. The format i ### Changed +- Updated the default RustFS server image from `rustfs/rustfs:1.0.0-beta.10` to + `rustfs/rustfs:1.0.0`. - Documented that distinct-physical-disk erasure failures and a separate data-plane operator are outside this controller's scope. - Refreshed the Console dashboard and tenant navigation with semantic status colors, clearer primary diff --git a/console-web/app/(dashboard)/tenants/new/page.tsx b/console-web/app/(dashboard)/tenants/new/page.tsx index cced639..c3fbe33 100644 --- a/console-web/app/(dashboard)/tenants/new/page.tsx +++ b/console-web/app/(dashboard)/tenants/new/page.tsx @@ -20,7 +20,7 @@ import { ApiError } from "@/lib/api-client" type CreateMode = "form" | "yaml" -const DEFAULT_RUSTFS_IMAGE = "rustfs/rustfs:1.0.0-beta.10" +const DEFAULT_RUSTFS_IMAGE = "rustfs/rustfs:1.0.0" const defaultPool: CreatePoolRequest = { name: "pool-0", @@ -209,7 +209,7 @@ export default function TenantCreatePage() { required value={image} onChange={(e) => setImage(e.target.value)} - placeholder="rustfs/rustfs:1.0.0-beta.10" + placeholder="rustfs/rustfs:1.0.0" />

{t( diff --git a/deploy/rustfs-operator/README.md b/deploy/rustfs-operator/README.md index 8442e71..2bae013 100755 --- a/deploy/rustfs-operator/README.md +++ b/deploy/rustfs-operator/README.md @@ -220,6 +220,82 @@ is omitted, the operator does not set `RUSTFS_RPC_SECRET`, RustFS resolves it from its own credential configuration, and the operator does not report `RpcAuthReady` for that unmanaged value. +### Tenant OIDC Custom CA Trust + +Use `spec.oidc.extraCaCertSecretRef` when RustFS must trust a private CA for +outbound OIDC connections: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: oidc-extra-ca + namespace: storage +type: Opaque +stringData: + ca.crt: | + -----BEGIN CERTIFICATE----- + ... + -----END CERTIFICATE----- +--- +apiVersion: rustfs.com/v1alpha1 +kind: Tenant +metadata: + name: rustfs-a + namespace: storage +spec: + image: rustfs/rustfs:1.0.0 + oidc: + extraCaCertSecretRef: + name: oidc-extra-ca +``` + +The key defaults to `ca.crt`. The operator validates the PEM certificates, +mounts the selected key at `/var/run/rustfs/oidc-extra-ca/ca.pem` without a +`subPath`, and sets `RUSTFS_EXTRA_CA_CERT` to that path. Secret updates enqueue +referencing Tenants without forcing a Pod rollout. This feature targets RustFS +GA and later images. This OIDC-only trust is separate from process-wide +`spec.tls.caTrust`. + +`OidcTrustReady=True` confirms that the configured Secret key contained a valid +CA bundle during the latest reconciliation. Kubernetes projects Secret updates +to each Pod independently, so the condition does not confirm that every Pod has +observed the same version. For CA rotation, publish both the current and +replacement roots, wait for every Pod to observe the combined bundle, verify +OIDC discovery and login, switch the provider certificate, and remove the old +root only after every Pod trusts the replacement. + +### Additional RustFS Files + +Use `spec.additionalVolumes` and `spec.additionalVolumeMounts` to provide files +that do not have a dedicated Tenant field. Both fields use the Kubernetes +`Volume` and `VolumeMount` schemas and apply to the RustFS container in every +Pool. For example, the following configuration provides an unmanaged CA bundle +to `RUSTFS_EXTRA_CA_CERT`: + +```yaml +spec: + env: + - name: RUSTFS_EXTRA_CA_CERT + value: /etc/rustfs/custom-ca/ca.crt + additionalVolumes: + - name: custom-ca + secret: + secretName: custom-ca + additionalVolumeMounts: + - name: custom-ca + mountPath: /etc/rustfs/custom-ca + readOnly: true +``` + +Every additional mount must reference an additional volume. Volume names and +mount paths must not conflict with operator-managed storage, logging, TLS, or +OIDC mounts. Relative paths, `..` components, equivalent paths, and parent or +child relationships with managed mounts are rejected. Kubernetes validates the +selected volume source and projects Secret and ConfigMap updates. Avoid +`subPath` when projected updates must reach running Pods. Changing either Tenant +field updates the StatefulSet Pod template and starts a rolling update. + ### Tenant Provisioning Tenants can declare RustFS canned policies, regular users, and buckets directly in `spec.policies`, `spec.users`, and `spec.buckets`. Provisioning starts only after the Tenant workload is ready, uses `spec.credsSecret` as the RustFS admin credential source, and reports progress under `status.provisioning`. @@ -495,7 +571,7 @@ and a multi-replica Tenant temporarily runs with reduced capacity. Verify every Tenant image first. Known incompatible images are blocked before rollout, and mutable tags, digest references, or custom repositories are blocked under an effective `RuntimeDefault` profile unless the Tenant carries an image-bound -acknowledgement. Before upgrade, either pin a verified RustFS beta.9-or-later +acknowledgement. Before upgrade, either pin a verified RustFS 1.0.0 or later release tag, or verify the effective image and set `operator.rustfs.com/runtime-default-image-ack` to that exact image reference: @@ -524,7 +600,7 @@ verifying that exact digest, acknowledge the complete reference. Mutable tags ca change content without changing the annotation, so prefer an immutable digest in production. -The built-in RustFS image fallback also changes from the mutable `latest` tag to +The built-in RustFS image fallback is now `rustfs/rustfs:1.0.0`, replacing `rustfs/rustfs:1.0.0-beta.10`. Tenants without `spec.image` and without a `TENANT_RUSTFS_IMAGE` Operator environment override roll to that pinned release on reconciliation. Set `spec.image` explicitly to control future server upgrades. diff --git a/deploy/rustfs-operator/crds/tenant-crd.yaml b/deploy/rustfs-operator/crds/tenant-crd.yaml index 9e435e0..1949502 100644 --- a/deploy/rustfs-operator/crds/tenant-crd.yaml +++ b/deploy/rustfs-operator/crds/tenant-crd.yaml @@ -26,6 +26,1194 @@ spec: properties: spec: properties: + additionalVolumeMounts: + description: Additional volume mounts added to the RustFS container in every Pool. + items: + description: VolumeMount describes a mounting of a Volume within a container. + properties: + mountPath: + description: Path within the container at which the volume should be mounted. Must not contain ':'. + type: string + mountPropagation: + description: mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationNone is used. This field is beta in 1.10. When RecursiveReadOnly is set to IfPossible or to Enabled, MountPropagation must be None or unspecified (which defaults to None). + type: string + name: + description: This must match the Name of a Volume. + type: string + readOnly: + description: Mounted read-only if true, read-write otherwise (false or unspecified). Defaults to false. + type: boolean + recursiveReadOnly: + description: |- + RecursiveReadOnly specifies whether read-only mounts should be handled recursively. + + If ReadOnly is false, this field has no meaning and must be unspecified. + + If ReadOnly is true, and this field is set to Disabled, the mount is not made recursively read-only. If this field is set to IfPossible, the mount is made recursively read-only, if it is supported by the container runtime. If this field is set to Enabled, the mount is made recursively read-only if it is supported by the container runtime, otherwise the pod will not be started and an error will be generated to indicate the reason. + + If this field is set to IfPossible or Enabled, MountPropagation must be set to None (or be unspecified, which defaults to None). + + If this field is not specified, it is treated as an equivalent of Disabled. + type: string + subPath: + description: Path within the volume from which the container's volume should be mounted. Defaults to "" (volume's root). + type: string + subPathExpr: + description: Expanded path within the volume from which the container's volume should be mounted. Behaves similarly to SubPath but environment variable references $(VAR_NAME) are expanded using the container's environment. Defaults to "" (volume's root). SubPathExpr and SubPath are mutually exclusive. + type: string + required: + - mountPath + - name + type: object + type: array + x-kubernetes-list-map-keys: + - mountPath + x-kubernetes-list-type: map + additionalVolumes: + description: |- + Additional Kubernetes volumes added to every RustFS Pod. + + These volumes are mounted only when referenced by `additionalVolumeMounts`. + items: + description: Volume represents a named volume in a pod that may be accessed by any container in the pod. + properties: + awsElasticBlockStore: + description: 'awsElasticBlockStore represents an AWS Disk resource that is attached to a kubelet''s host machine and then exposed to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore' + properties: + fsType: + description: 'fsType is the filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore' + type: string + partition: + description: 'partition is the partition in the volume that you want to mount. If omitted, the default is to mount by volume name. Examples: For volume /dev/sda1, you specify the partition as "1". Similarly, the volume partition for /dev/sda is "0" (or you can leave the property empty).' + format: int32 + type: integer + readOnly: + description: 'readOnly value true will force the readOnly setting in VolumeMounts. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore' + type: boolean + volumeID: + description: 'volumeID is unique ID of the persistent disk resource in AWS (Amazon EBS volume). More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore' + type: string + required: + - volumeID + type: object + azureDisk: + description: azureDisk represents an Azure Data Disk mount on the host and bind mount to the pod. + properties: + cachingMode: + description: 'cachingMode is the Host Caching mode: None, Read Only, Read Write.' + type: string + diskName: + description: diskName is the Name of the data disk in the blob storage + type: string + diskURI: + description: diskURI is the URI of data disk in the blob storage + type: string + fsType: + description: fsType is Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. + type: string + kind: + description: 'kind expected values are Shared: multiple blob disks per storage account Dedicated: single blob disk per storage account Managed: azure managed data disk (only in managed availability set). defaults to shared' + type: string + readOnly: + description: readOnly Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. + type: boolean + required: + - diskName + - diskURI + type: object + azureFile: + description: azureFile represents an Azure File Service mount on the host and bind mount to the pod. + properties: + readOnly: + description: readOnly defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. + type: boolean + secretName: + description: secretName is the name of secret that contains Azure Storage Account Name and Key + type: string + shareName: + description: shareName is the azure share Name + type: string + required: + - secretName + - shareName + type: object + cephfs: + description: cephFS represents a Ceph FS mount on the host that shares a pod's lifetime + properties: + monitors: + description: 'monitors is Required: Monitors is a collection of Ceph monitors More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it' + items: + type: string + type: array + path: + description: 'path is Optional: Used as the mounted root, rather than the full Ceph tree, default is /' + type: string + readOnly: + description: 'readOnly is Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it' + type: boolean + secretFile: + description: 'secretFile is Optional: SecretFile is the path to key ring for User, default is /etc/ceph/user.secret More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it' + type: string + secretRef: + description: 'secretRef is Optional: SecretRef is reference to the authentication secret for User, default is empty. More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it' + properties: + name: + description: 'Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names' + type: string + required: + - name + type: object + user: + description: 'user is optional: User is the rados user name, default is admin More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it' + type: string + required: + - monitors + type: object + cinder: + description: 'cinder represents a cinder volume attached and mounted on kubelets host machine. More info: https://examples.k8s.io/mysql-cinder-pd/README.md' + properties: + fsType: + description: 'fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://examples.k8s.io/mysql-cinder-pd/README.md' + type: string + readOnly: + description: 'readOnly defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. More info: https://examples.k8s.io/mysql-cinder-pd/README.md' + type: boolean + secretRef: + description: 'secretRef is optional: points to a secret object containing parameters used to connect to OpenStack.' + properties: + name: + description: 'Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names' + type: string + required: + - name + type: object + volumeID: + description: 'volumeID used to identify the volume in cinder. More info: https://examples.k8s.io/mysql-cinder-pd/README.md' + type: string + required: + - volumeID + type: object + configMap: + description: configMap represents a configMap that should populate this volume + properties: + defaultMode: + description: 'defaultMode is optional: mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Defaults to 0644. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.' + format: int32 + type: integer + items: + description: items if unspecified, each key-value pair in the Data field of the referenced ConfigMap will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the ConfigMap, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'. + items: + description: Maps a string key to a path within a volume. + properties: + key: + description: key is the key to project. + type: string + mode: + description: 'mode is Optional: mode bits used to set permissions on this file. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.' + format: int32 + type: integer + path: + description: path is the relative path of the file to map the key to. May not be an absolute path. May not contain the path element '..'. May not start with the string '..'. + type: string + required: + - key + - path + type: object + type: array + name: + description: 'Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names' + type: string + optional: + description: optional specify whether the ConfigMap or its keys must be defined + type: boolean + required: + - name + type: object + csi: + description: csi (Container Storage Interface) represents ephemeral storage that is handled by certain external CSI drivers (Beta feature). + properties: + driver: + description: driver is the name of the CSI driver that handles this volume. Consult with your admin for the correct name as registered in the cluster. + type: string + fsType: + description: fsType to mount. Ex. "ext4", "xfs", "ntfs". If not provided, the empty value is passed to the associated CSI driver which will determine the default filesystem to apply. + type: string + nodePublishSecretRef: + description: nodePublishSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI NodePublishVolume and NodeUnpublishVolume calls. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secret references are passed. + properties: + name: + description: 'Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names' + type: string + required: + - name + type: object + readOnly: + description: readOnly specifies a read-only configuration for the volume. Defaults to false (read/write). + type: boolean + volumeAttributes: + additionalProperties: + type: string + description: volumeAttributes stores driver-specific properties that are passed to the CSI driver. Consult your driver's documentation for supported values. + type: object + required: + - driver + type: object + downwardAPI: + description: downwardAPI represents downward API about the pod that should populate this volume + properties: + defaultMode: + description: 'Optional: mode bits to use on created files by default. Must be a Optional: mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Defaults to 0644. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.' + format: int32 + type: integer + items: + description: Items is a list of downward API volume file + items: + description: DownwardAPIVolumeFile represents information to create the file containing the pod field + properties: + fieldRef: + description: 'Required: Selects a field of the pod: only annotations, labels, name, namespace and uid are supported.' + properties: + apiVersion: + description: Version of the schema the FieldPath is written in terms of, defaults to "v1". + type: string + fieldPath: + description: Path of the field to select in the specified API version. + type: string + required: + - fieldPath + type: object + mode: + description: 'Optional: mode bits used to set permissions on this file, must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.' + format: int32 + type: integer + path: + description: 'Required: Path is the relative path name of the file to be created. Must not be absolute or contain the ''..'' path. Must be utf-8 encoded. The first item of the relative path must not start with ''..''' + type: string + resourceFieldRef: + description: 'Selects a resource of the container: only resources limits and requests (limits.cpu, limits.memory, requests.cpu and requests.memory) are currently supported.' + properties: + containerName: + description: 'Container name: required for volumes, optional for env vars' + type: string + divisor: + description: Specifies the output format of the exposed resources, defaults to "1" + x-kubernetes-int-or-string: true + resource: + description: 'Required: resource to select' + type: string + required: + - resource + type: object + required: + - path + type: object + type: array + type: object + emptyDir: + description: 'emptyDir represents a temporary directory that shares a pod''s lifetime. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir' + properties: + medium: + description: 'medium represents what type of storage medium should back this directory. The default is "" which means to use the node''s default medium. Must be an empty string (default) or Memory. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir' + type: string + sizeLimit: + description: 'sizeLimit is the total amount of local storage required for this EmptyDir volume. The size limit is also applicable for memory medium. The maximum usage on memory medium EmptyDir would be the minimum value between the SizeLimit specified here and the sum of memory limits of all containers in a pod. The default is nil which means that the limit is undefined. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir' + x-kubernetes-int-or-string: true + type: object + ephemeral: + description: |- + ephemeral represents a volume that is handled by a cluster storage driver. The volume's lifecycle is tied to the pod that defines it - it will be created before the pod starts, and deleted when the pod is removed. + + Use this if: a) the volume is only needed while the pod runs, b) features of normal volumes like restoring from snapshot or capacity + tracking are needed, + c) the storage driver is specified through a storage class, and d) the storage driver supports dynamic volume provisioning through + a PersistentVolumeClaim (see EphemeralVolumeSource for more + information on the connection between this volume type + and PersistentVolumeClaim). + + Use PersistentVolumeClaim or one of the vendor-specific APIs for volumes that persist for longer than the lifecycle of an individual pod. + + Use CSI for light-weight local ephemeral volumes if the CSI driver is meant to be used that way - see the documentation of the driver for more information. + + A pod can use both types of ephemeral volumes and persistent volumes at the same time. + properties: + volumeClaimTemplate: + description: |- + Will be used to create a stand-alone PVC to provision the volume. The pod in which this EphemeralVolumeSource is embedded will be the owner of the PVC, i.e. the PVC will be deleted together with the pod. The name of the PVC will be `-` where `` is the name from the `PodSpec.Volumes` array entry. Pod validation will reject the pod if the concatenated name is not valid for a PVC (for example, too long). + + An existing PVC with that name that is not owned by the pod will *not* be used for the pod to avoid using an unrelated volume by mistake. Starting the pod is then blocked until the unrelated PVC is removed. If such a pre-created PVC is meant to be used by the pod, the PVC has to updated with an owner reference to the pod once the pod exists. Normally this should not be necessary, but it may be useful when manually reconstructing a broken cluster. + + This field is read-only and no changes will be made by Kubernetes to the PVC after it has been created. + + Required, must not be nil. + properties: + metadata: + description: May contain labels and annotations that will be copied into the PVC when creating it. No other fields are allowed and will be rejected during validation. + properties: + annotations: + additionalProperties: + type: string + description: 'Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations' + type: object + creationTimestamp: + description: |- + CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC. + + Populated by the system. Read-only. Null for lists. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata + format: date-time + type: string + deletionGracePeriodSeconds: + description: Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only. + format: int64 + type: integer + deletionTimestamp: + description: |- + DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested. + + Populated by the system when a graceful deletion is requested. Read-only. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata + format: date-time + type: string + finalizers: + description: Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list. + items: + type: string + type: array + generateName: + description: |- + GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server. + + If this field is specified and the generated name exists, the server will return a 409. + + Applied only if Name is not specified. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#idempotency + type: string + generation: + description: A sequence number representing a specific generation of the desired state. Populated by the system. Read-only. + format: int64 + type: integer + labels: + additionalProperties: + type: string + description: 'Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels' + type: object + managedFields: + description: ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object. + items: + description: ManagedFieldsEntry is a workflow-id, a FieldSet and the group version of the resource that the fieldset applies to. + properties: + apiVersion: + description: APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted. + type: string + fieldsType: + description: 'FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1"' + type: string + fieldsV1: + description: FieldsV1 holds the first JSON version format as described in the "FieldsV1" type. + type: object + manager: + description: Manager is an identifier of the workflow managing these fields. + type: string + operation: + description: Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'. + type: string + subresource: + description: Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource. + type: string + time: + description: Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over. + format: date-time + type: string + type: object + type: array + name: + description: 'Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names#names' + type: string + namespace: + description: |- + Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty. + + Must be a DNS_LABEL. Cannot be updated. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces + type: string + ownerReferences: + description: List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller. + items: + description: OwnerReference contains enough information to let you identify an owning object. An owning object must be in the same namespace as the dependent, or be cluster-scoped, so there is no namespace field. + properties: + apiVersion: + description: API version of the referent. + type: string + blockOwnerDeletion: + description: If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See https://kubernetes.io/docs/concepts/architecture/garbage-collection/#foreground-deletion for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned. + type: boolean + controller: + description: If true, this reference points to the managing controller. + type: boolean + kind: + description: 'Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' + type: string + name: + description: 'Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names#names' + type: string + uid: + description: 'UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names#uids' + type: string + required: + - apiVersion + - kind + - name + - uid + type: object + type: array + resourceVersion: + description: |- + An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources. + + Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency + type: string + selfLink: + description: 'Deprecated: selfLink is a legacy read-only field that is no longer populated by the system.' + type: string + uid: + description: |- + UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations. + + Populated by the system. Read-only. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names#uids + type: string + type: object + spec: + description: The specification for the PersistentVolumeClaim. The entire content is copied unchanged into the PVC that gets created from this template. The same fields as in a PersistentVolumeClaim are also valid here. + properties: + accessModes: + description: 'accessModes contains the desired access modes the volume should have. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1' + items: + type: string + type: array + dataSource: + description: 'dataSource field can be used to specify either: * An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot) * An existing PVC (PersistentVolumeClaim) If the provisioner or an external controller can support the specified data source, it will create a new volume based on the contents of the specified data source. When the AnyVolumeDataSource feature gate is enabled, dataSource contents will be copied to dataSourceRef, and dataSourceRef contents will be copied to dataSource when dataSourceRef.namespace is not specified. If the namespace is specified, then dataSourceRef will not be copied to dataSource.' + properties: + apiGroup: + description: APIGroup is the group for the resource being referenced. If APIGroup is not specified, the specified Kind must be in the core API group. For any other third-party types, APIGroup is required. + type: string + kind: + description: Kind is the type of resource being referenced + type: string + name: + description: Name is the name of resource being referenced + type: string + required: + - kind + - name + type: object + dataSourceRef: + description: |- + dataSourceRef specifies the object from which to populate the volume with data, if a non-empty volume is desired. This may be any object from a non-empty API group (non core object) or a PersistentVolumeClaim object. When this field is specified, volume binding will only succeed if the type of the specified object matches some installed volume populator or dynamic provisioner. This field will replace the functionality of the dataSource field and as such if both fields are non-empty, they must have the same value. For backwards compatibility, when namespace isn't specified in dataSourceRef, both fields (dataSource and dataSourceRef) will be set to the same value automatically if one of them is empty and the other is non-empty. When namespace is specified in dataSourceRef, dataSource isn't set to the same value and must be empty. There are three important differences between dataSource and dataSourceRef: * While dataSource only allows two specific types of objects, dataSourceRef + allows any non-core object, as well as PersistentVolumeClaim objects. + * While dataSource ignores disallowed values (dropping them), dataSourceRef + preserves all values, and generates an error if a disallowed value is + specified. + * While dataSource only allows local objects, dataSourceRef allows objects + in any namespaces. + (Beta) Using this field requires the AnyVolumeDataSource feature gate to be enabled. (Alpha) Using the namespace field of dataSourceRef requires the CrossNamespaceVolumeDataSource feature gate to be enabled. + properties: + apiGroup: + description: APIGroup is the group for the resource being referenced. If APIGroup is not specified, the specified Kind must be in the core API group. For any other third-party types, APIGroup is required. + type: string + kind: + description: Kind is the type of resource being referenced + type: string + name: + description: Name is the name of resource being referenced + type: string + namespace: + description: Namespace is the namespace of resource being referenced Note that when a namespace is specified, a gateway.networking.k8s.io/ReferenceGrant object is required in the referent namespace to allow that namespace's owner to accept the reference. See the ReferenceGrant documentation for details. (Alpha) This field requires the CrossNamespaceVolumeDataSource feature gate to be enabled. + type: string + required: + - kind + - name + type: object + resources: + description: 'resources represents the minimum resources the volume should have. If RecoverVolumeExpansionFailure feature is enabled users are allowed to specify resource requirements that are lower than previous value but must still be higher than capacity recorded in the status field of the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#resources' + properties: + limits: + additionalProperties: + description: "Quantity is a fixed-point representation of a number. It provides convenient marshaling/unmarshaling in JSON and YAML, in addition to String() and AsInt64() accessors.\n\nThe serialization format is:\n\n``` ::= \n\n\t(Note that may be empty, from the \"\" case in .)\n\n ::= 0 | 1 | ... | 9 ::= | ::= | . | . | . ::= \"+\" | \"-\" ::= | ::= | | ::= Ki | Mi | Gi | Ti | Pi | Ei\n\n\t(International System of units; See: http://physics.nist.gov/cuu/Units/binary.html)\n\n ::= m | \"\" | k | M | G | T | P | E\n\n\t(Note that 1024 = 1Ki but 1000 = 1k; I didn't choose the capitalization.)\n\n ::= \"e\" | \"E\" ```\n\nNo matter which of the three exponent forms is used, no quantity may represent a number greater than 2^63-1 in magnitude, nor may it have more than 3 decimal places. Numbers larger or more precise will be capped or rounded up. (E.g.: 0.1m will rounded up to 1m.) This may be extended in the future if we require larger or smaller quantities.\n\nWhen a Quantity is parsed from a string, it will remember the type of suffix it had, and will use the same type again when it is serialized.\n\nBefore serializing, Quantity will be put in \"canonical form\". This means that Exponent/suffix will be adjusted up or down (with a corresponding increase or decrease in Mantissa) such that:\n\n- No precision is lost - No fractional digits will be emitted - The exponent (or suffix) is as large as possible.\n\nThe sign will be omitted unless the number is negative.\n\nExamples:\n\n- 1.5 will be serialized as \"1500m\" - 1.5Gi will be serialized as \"1536Mi\"\n\nNote that the quantity will NEVER be internally represented by a floating point number. That is the whole point of this exercise.\n\nNon-canonical values will still parse as long as they are well formed, but will be re-emitted in their canonical form. (So always use canonical form, or don't diff.)\n\nThis format is intended to make it difficult to use these numbers without writing some sort of special handling code in the hopes that that will cause implementors to also use a fixed point implementation." + x-kubernetes-int-or-string: true + description: 'Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + requests: + additionalProperties: + description: "Quantity is a fixed-point representation of a number. It provides convenient marshaling/unmarshaling in JSON and YAML, in addition to String() and AsInt64() accessors.\n\nThe serialization format is:\n\n``` ::= \n\n\t(Note that may be empty, from the \"\" case in .)\n\n ::= 0 | 1 | ... | 9 ::= | ::= | . | . | . ::= \"+\" | \"-\" ::= | ::= | | ::= Ki | Mi | Gi | Ti | Pi | Ei\n\n\t(International System of units; See: http://physics.nist.gov/cuu/Units/binary.html)\n\n ::= m | \"\" | k | M | G | T | P | E\n\n\t(Note that 1024 = 1Ki but 1000 = 1k; I didn't choose the capitalization.)\n\n ::= \"e\" | \"E\" ```\n\nNo matter which of the three exponent forms is used, no quantity may represent a number greater than 2^63-1 in magnitude, nor may it have more than 3 decimal places. Numbers larger or more precise will be capped or rounded up. (E.g.: 0.1m will rounded up to 1m.) This may be extended in the future if we require larger or smaller quantities.\n\nWhen a Quantity is parsed from a string, it will remember the type of suffix it had, and will use the same type again when it is serialized.\n\nBefore serializing, Quantity will be put in \"canonical form\". This means that Exponent/suffix will be adjusted up or down (with a corresponding increase or decrease in Mantissa) such that:\n\n- No precision is lost - No fractional digits will be emitted - The exponent (or suffix) is as large as possible.\n\nThe sign will be omitted unless the number is negative.\n\nExamples:\n\n- 1.5 will be serialized as \"1500m\" - 1.5Gi will be serialized as \"1536Mi\"\n\nNote that the quantity will NEVER be internally represented by a floating point number. That is the whole point of this exercise.\n\nNon-canonical values will still parse as long as they are well formed, but will be re-emitted in their canonical form. (So always use canonical form, or don't diff.)\n\nThis format is intended to make it difficult to use these numbers without writing some sort of special handling code in the hopes that that will cause implementors to also use a fixed point implementation." + x-kubernetes-int-or-string: true + description: 'Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/' + type: object + type: object + selector: + description: selector is a label query over volumes to consider for binding. + properties: + matchExpressions: + description: matchExpressions is a list of label selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. + properties: + key: + description: key is the label key that the selector applies to. + type: string + operator: + description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + storageClassName: + description: 'storageClassName is the name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1' + type: string + volumeAttributesClassName: + description: 'volumeAttributesClassName may be used to set the VolumeAttributesClass used by this claim. If specified, the CSI driver will create or update the volume with the attributes defined in the corresponding VolumeAttributesClass. This has a different purpose than storageClassName, it can be changed after the claim is created. An empty string value means that no VolumeAttributesClass will be applied to the claim but it''s not allowed to reset this field to empty string once it is set. If unspecified and the PersistentVolumeClaim is unbound, the default VolumeAttributesClass will be set by the persistentvolume controller if it exists. If the resource referred to by volumeAttributesClass does not exist, this PersistentVolumeClaim will be set to a Pending state, as reflected by the modifyVolumeStatus field, until such as a resource exists. More info: https://kubernetes.io/docs/concepts/storage/volume-attributes-classes/ (Alpha) Using this field requires the VolumeAttributesClass feature gate to be enabled.' + type: string + volumeMode: + description: volumeMode defines what type of volume is required by the claim. Value of Filesystem is implied when not included in claim spec. + type: string + volumeName: + description: volumeName is the binding reference to the PersistentVolume backing this claim. + type: string + type: object + required: + - spec + type: object + type: object + fc: + description: fc represents a Fibre Channel resource that is attached to a kubelet's host machine and then exposed to the pod. + properties: + fsType: + description: fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. + type: string + lun: + description: 'lun is Optional: FC target lun number' + format: int32 + type: integer + readOnly: + description: 'readOnly is Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.' + type: boolean + targetWWNs: + description: 'targetWWNs is Optional: FC target worldwide names (WWNs)' + items: + type: string + type: array + wwids: + description: 'wwids Optional: FC volume world wide identifiers (wwids) Either wwids or combination of targetWWNs and lun must be set, but not both simultaneously.' + items: + type: string + type: array + type: object + flexVolume: + description: flexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin. + properties: + driver: + description: driver is the name of the driver to use for this volume. + type: string + fsType: + description: fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". The default filesystem depends on FlexVolume script. + type: string + options: + additionalProperties: + type: string + description: 'options is Optional: this field holds extra command options if any.' + type: object + readOnly: + description: 'readOnly is Optional: defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.' + type: boolean + secretRef: + description: 'secretRef is Optional: secretRef is reference to the secret object containing sensitive information to pass to the plugin scripts. This may be empty if no secret object is specified. If the secret object contains more than one secret, all secrets are passed to the plugin scripts.' + properties: + name: + description: 'Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names' + type: string + required: + - name + type: object + required: + - driver + type: object + flocker: + description: flocker represents a Flocker volume attached to a kubelet's host machine. This depends on the Flocker control service being running + properties: + datasetName: + description: datasetName is Name of the dataset stored as metadata -> name on the dataset for Flocker should be considered as deprecated + type: string + datasetUUID: + description: datasetUUID is the UUID of the dataset. This is unique identifier of a Flocker dataset + type: string + type: object + gcePersistentDisk: + description: 'gcePersistentDisk represents a GCE Disk resource that is attached to a kubelet''s host machine and then exposed to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk' + properties: + fsType: + description: 'fsType is filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk' + type: string + partition: + description: 'partition is the partition in the volume that you want to mount. If omitted, the default is to mount by volume name. Examples: For volume /dev/sda1, you specify the partition as "1". Similarly, the volume partition for /dev/sda is "0" (or you can leave the property empty). More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk' + format: int32 + type: integer + pdName: + description: 'pdName is unique name of the PD resource in GCE. Used to identify the disk in GCE. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk' + type: string + readOnly: + description: 'readOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk' + type: boolean + required: + - pdName + type: object + gitRepo: + description: 'gitRepo represents a git repository at a particular revision. DEPRECATED: GitRepo is deprecated. To provision a container with a git repo, mount an EmptyDir into an InitContainer that clones the repo using git, then mount the EmptyDir into the Pod''s container.' + properties: + directory: + description: directory is the target directory name. Must not contain or start with '..'. If '.' is supplied, the volume directory will be the git repository. Otherwise, if specified, the volume will contain the git repository in the subdirectory with the given name. + type: string + repository: + description: repository is the URL + type: string + revision: + description: revision is the commit hash for the specified revision. + type: string + required: + - repository + type: object + glusterfs: + description: 'glusterfs represents a Glusterfs mount on the host that shares a pod''s lifetime. More info: https://examples.k8s.io/volumes/glusterfs/README.md' + properties: + endpoints: + description: 'endpoints is the endpoint name that details Glusterfs topology. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod' + type: string + path: + description: 'path is the Glusterfs volume path. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod' + type: string + readOnly: + description: 'readOnly here will force the Glusterfs volume to be mounted with read-only permissions. Defaults to false. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod' + type: boolean + required: + - endpoints + - path + type: object + hostPath: + description: 'hostPath represents a pre-existing file or directory on the host machine that is directly exposed to the container. This is generally used for system agents or other privileged things that are allowed to see the host machine. Most containers will NOT need this. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath' + properties: + path: + description: 'path of the directory on the host. If the path is a symlink, it will follow the link to the real path. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath' + type: string + type: + description: 'type for HostPath Volume Defaults to "" More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath' + type: string + required: + - path + type: object + iscsi: + description: 'iscsi represents an ISCSI Disk resource that is attached to a kubelet''s host machine and then exposed to the pod. More info: https://examples.k8s.io/volumes/iscsi/README.md' + properties: + chapAuthDiscovery: + description: chapAuthDiscovery defines whether support iSCSI Discovery CHAP authentication + type: boolean + chapAuthSession: + description: chapAuthSession defines whether support iSCSI Session CHAP authentication + type: boolean + fsType: + description: 'fsType is the filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#iscsi' + type: string + initiatorName: + description: initiatorName is the custom iSCSI Initiator Name. If initiatorName is specified with iscsiInterface simultaneously, new iSCSI interface : will be created for the connection. + type: string + iqn: + description: iqn is the target iSCSI Qualified Name. + type: string + iscsiInterface: + description: iscsiInterface is the interface Name that uses an iSCSI transport. Defaults to 'default' (tcp). + type: string + lun: + description: lun represents iSCSI Target Lun number. + format: int32 + type: integer + portals: + description: portals is the iSCSI Target Portal List. The portal is either an IP or ip_addr:port if the port is other than default (typically TCP ports 860 and 3260). + items: + type: string + type: array + readOnly: + description: readOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false. + type: boolean + secretRef: + description: secretRef is the CHAP Secret for iSCSI target and initiator authentication + properties: + name: + description: 'Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names' + type: string + required: + - name + type: object + targetPortal: + description: targetPortal is iSCSI Target Portal. The Portal is either an IP or ip_addr:port if the port is other than default (typically TCP ports 860 and 3260). + type: string + required: + - iqn + - lun + - targetPortal + type: object + name: + description: 'name of the volume. Must be a DNS_LABEL and unique within the pod. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names' + type: string + nfs: + description: 'nfs represents an NFS mount on the host that shares a pod''s lifetime More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs' + properties: + path: + description: 'path that is exported by the NFS server. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs' + type: string + readOnly: + description: 'readOnly here will force the NFS export to be mounted with read-only permissions. Defaults to false. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs' + type: boolean + server: + description: 'server is the hostname or IP address of the NFS server. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs' + type: string + required: + - path + - server + type: object + persistentVolumeClaim: + description: 'persistentVolumeClaimVolumeSource represents a reference to a PersistentVolumeClaim in the same namespace. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims' + properties: + claimName: + description: 'claimName is the name of a PersistentVolumeClaim in the same namespace as the pod using this volume. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims' + type: string + readOnly: + description: readOnly Will force the ReadOnly setting in VolumeMounts. Default false. + type: boolean + required: + - claimName + type: object + photonPersistentDisk: + description: photonPersistentDisk represents a PhotonController persistent disk attached and mounted on kubelets host machine + properties: + fsType: + description: fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. + type: string + pdID: + description: pdID is the ID that identifies Photon Controller persistent disk + type: string + required: + - pdID + type: object + portworxVolume: + description: portworxVolume represents a portworx volume attached and mounted on kubelets host machine + properties: + fsType: + description: fSType represents the filesystem type to mount Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs". Implicitly inferred to be "ext4" if unspecified. + type: string + readOnly: + description: readOnly defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. + type: boolean + volumeID: + description: volumeID uniquely identifies a Portworx volume + type: string + required: + - volumeID + type: object + projected: + description: projected items for all in one resources secrets, configmaps, and downward API + properties: + defaultMode: + description: defaultMode are the mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set. + format: int32 + type: integer + sources: + description: sources is the list of volume projections + items: + description: Projection that may be projected along with other supported volume types + properties: + clusterTrustBundle: + description: |- + ClusterTrustBundle allows a pod to access the `.spec.trustBundle` field of ClusterTrustBundle objects in an auto-updating file. + + Alpha, gated by the ClusterTrustBundleProjection feature gate. + + ClusterTrustBundle objects can either be selected by name, or by the combination of signer name and a label selector. + + Kubelet performs aggressive normalization of the PEM contents written into the pod filesystem. Esoteric PEM features such as inter-block comments and block headers are stripped. Certificates are deduplicated. The ordering of certificates within the file is arbitrary, and Kubelet may change the order over time. + properties: + labelSelector: + description: Select all ClusterTrustBundles that match this label selector. Only has effect if signerName is set. Mutually-exclusive with name. If unset, interpreted as "match nothing". If set but empty, interpreted as "match everything". + properties: + matchExpressions: + description: matchExpressions is a list of label selector requirements. The requirements are ANDed. + items: + description: A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values. + properties: + key: + description: key is the label key that the selector applies to. + type: string + operator: + description: operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist. + type: string + values: + description: values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + matchLabels: + additionalProperties: + type: string + description: matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed. + type: object + type: object + name: + description: Select a single ClusterTrustBundle by object name. Mutually-exclusive with signerName and labelSelector. + type: string + optional: + description: If true, don't block pod startup if the referenced ClusterTrustBundle(s) aren't available. If using name, then the named ClusterTrustBundle is allowed not to exist. If using signerName, then the combination of signerName and labelSelector is allowed to match zero ClusterTrustBundles. + type: boolean + path: + description: Relative path from the volume root to write the bundle. + type: string + signerName: + description: Select all ClusterTrustBundles that match this signer name. Mutually-exclusive with name. The contents of all selected ClusterTrustBundles will be unified and deduplicated. + type: string + required: + - path + type: object + configMap: + description: configMap information about the configMap data to project + properties: + items: + description: items if unspecified, each key-value pair in the Data field of the referenced ConfigMap will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the ConfigMap, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'. + items: + description: Maps a string key to a path within a volume. + properties: + key: + description: key is the key to project. + type: string + mode: + description: 'mode is Optional: mode bits used to set permissions on this file. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.' + format: int32 + type: integer + path: + description: path is the relative path of the file to map the key to. May not be an absolute path. May not contain the path element '..'. May not start with the string '..'. + type: string + required: + - key + - path + type: object + type: array + name: + description: 'Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names' + type: string + optional: + description: optional specify whether the ConfigMap or its keys must be defined + type: boolean + required: + - name + type: object + downwardAPI: + description: downwardAPI information about the downwardAPI data to project + properties: + items: + description: Items is a list of DownwardAPIVolume file + items: + description: DownwardAPIVolumeFile represents information to create the file containing the pod field + properties: + fieldRef: + description: 'Required: Selects a field of the pod: only annotations, labels, name, namespace and uid are supported.' + properties: + apiVersion: + description: Version of the schema the FieldPath is written in terms of, defaults to "v1". + type: string + fieldPath: + description: Path of the field to select in the specified API version. + type: string + required: + - fieldPath + type: object + mode: + description: 'Optional: mode bits used to set permissions on this file, must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.' + format: int32 + type: integer + path: + description: 'Required: Path is the relative path name of the file to be created. Must not be absolute or contain the ''..'' path. Must be utf-8 encoded. The first item of the relative path must not start with ''..''' + type: string + resourceFieldRef: + description: 'Selects a resource of the container: only resources limits and requests (limits.cpu, limits.memory, requests.cpu and requests.memory) are currently supported.' + properties: + containerName: + description: 'Container name: required for volumes, optional for env vars' + type: string + divisor: + description: Specifies the output format of the exposed resources, defaults to "1" + x-kubernetes-int-or-string: true + resource: + description: 'Required: resource to select' + type: string + required: + - resource + type: object + required: + - path + type: object + type: array + type: object + secret: + description: secret information about the secret data to project + properties: + items: + description: items if unspecified, each key-value pair in the Data field of the referenced Secret will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the Secret, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'. + items: + description: Maps a string key to a path within a volume. + properties: + key: + description: key is the key to project. + type: string + mode: + description: 'mode is Optional: mode bits used to set permissions on this file. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.' + format: int32 + type: integer + path: + description: path is the relative path of the file to map the key to. May not be an absolute path. May not contain the path element '..'. May not start with the string '..'. + type: string + required: + - key + - path + type: object + type: array + name: + description: 'Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names' + type: string + optional: + description: optional field specify whether the Secret or its key must be defined + type: boolean + required: + - name + type: object + serviceAccountToken: + description: serviceAccountToken is information about the serviceAccountToken data to project + properties: + audience: + description: audience is the intended audience of the token. A recipient of a token must identify itself with an identifier specified in the audience of the token, and otherwise should reject the token. The audience defaults to the identifier of the apiserver. + type: string + expirationSeconds: + description: expirationSeconds is the requested duration of validity of the service account token. As the token approaches expiration, the kubelet volume plugin will proactively rotate the service account token. The kubelet will start trying to rotate the token if the token is older than 80 percent of its time to live or if the token is older than 24 hours.Defaults to 1 hour and must be at least 10 minutes. + format: int64 + type: integer + path: + description: path is the path relative to the mount point of the file to project the token into. + type: string + required: + - path + type: object + type: object + type: array + type: object + quobyte: + description: quobyte represents a Quobyte mount on the host that shares a pod's lifetime + properties: + group: + description: group to map volume access to Default is no group + type: string + readOnly: + description: readOnly here will force the Quobyte volume to be mounted with read-only permissions. Defaults to false. + type: boolean + registry: + description: registry represents a single or multiple Quobyte Registry services specified as a string as host:port pair (multiple entries are separated with commas) which acts as the central registry for volumes + type: string + tenant: + description: tenant owning the given Quobyte volume in the Backend Used with dynamically provisioned Quobyte volumes, value is set by the plugin + type: string + user: + description: user to map volume access to Defaults to serivceaccount user + type: string + volume: + description: volume is a string that references an already created Quobyte volume by name. + type: string + required: + - registry + - volume + type: object + rbd: + description: 'rbd represents a Rados Block Device mount on the host that shares a pod''s lifetime. More info: https://examples.k8s.io/volumes/rbd/README.md' + properties: + fsType: + description: 'fsType is the filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#rbd' + type: string + image: + description: 'image is the rados image name. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it' + type: string + keyring: + description: 'keyring is the path to key ring for RBDUser. Default is /etc/ceph/keyring. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it' + type: string + monitors: + description: 'monitors is a collection of Ceph monitors. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it' + items: + type: string + type: array + pool: + description: 'pool is the rados pool name. Default is rbd. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it' + type: string + readOnly: + description: 'readOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it' + type: boolean + secretRef: + description: 'secretRef is name of the authentication secret for RBDUser. If provided overrides keyring. Default is nil. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it' + properties: + name: + description: 'Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names' + type: string + required: + - name + type: object + user: + description: 'user is the rados user name. Default is admin. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it' + type: string + required: + - image + - monitors + type: object + scaleIO: + description: scaleIO represents a ScaleIO persistent volume attached and mounted on Kubernetes nodes. + properties: + fsType: + description: fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Default is "xfs". + type: string + gateway: + description: gateway is the host address of the ScaleIO API Gateway. + type: string + protectionDomain: + description: protectionDomain is the name of the ScaleIO Protection Domain for the configured storage. + type: string + readOnly: + description: readOnly Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. + type: boolean + secretRef: + description: secretRef references to the secret for ScaleIO user and other sensitive information. If this is not provided, Login operation will fail. + properties: + name: + description: 'Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names' + type: string + required: + - name + type: object + sslEnabled: + description: sslEnabled Flag enable/disable SSL communication with Gateway, default false + type: boolean + storageMode: + description: storageMode indicates whether the storage for a volume should be ThickProvisioned or ThinProvisioned. Default is ThinProvisioned. + type: string + storagePool: + description: storagePool is the ScaleIO Storage Pool associated with the protection domain. + type: string + system: + description: system is the name of the storage system as configured in ScaleIO. + type: string + volumeName: + description: volumeName is the name of a volume already created in the ScaleIO system that is associated with this volume source. + type: string + required: + - gateway + - secretRef + - system + type: object + secret: + description: 'secret represents a secret that should populate this volume. More info: https://kubernetes.io/docs/concepts/storage/volumes#secret' + properties: + defaultMode: + description: 'defaultMode is Optional: mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Defaults to 0644. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.' + format: int32 + type: integer + items: + description: items If unspecified, each key-value pair in the Data field of the referenced Secret will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the Secret, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'. + items: + description: Maps a string key to a path within a volume. + properties: + key: + description: key is the key to project. + type: string + mode: + description: 'mode is Optional: mode bits used to set permissions on this file. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.' + format: int32 + type: integer + path: + description: path is the relative path of the file to map the key to. May not be an absolute path. May not contain the path element '..'. May not start with the string '..'. + type: string + required: + - key + - path + type: object + type: array + optional: + description: optional field specify whether the Secret or its keys must be defined + type: boolean + secretName: + description: 'secretName is the name of the secret in the pod''s namespace to use. More info: https://kubernetes.io/docs/concepts/storage/volumes#secret' + type: string + type: object + storageos: + description: storageOS represents a StorageOS volume attached and mounted on Kubernetes nodes. + properties: + fsType: + description: fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. + type: string + readOnly: + description: readOnly defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. + type: boolean + secretRef: + description: secretRef specifies the secret to use for obtaining the StorageOS API credentials. If not specified, default values will be attempted. + properties: + name: + description: 'Name of the referent. This field is effectively required, but due to backwards compatibility is allowed to be empty. Instances of this type with an empty value here are almost certainly wrong. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names' + type: string + required: + - name + type: object + volumeName: + description: volumeName is the human-readable name of the StorageOS volume. Volume names are only unique within a namespace. + type: string + volumeNamespace: + description: volumeNamespace specifies the scope of the volume within StorageOS. If no namespace is specified then the Pod's namespace will be used. This allows the Kubernetes name scoping to be mirrored within StorageOS for tighter integration. Set VolumeName to any name to override the default behaviour. Set to "default" if you are not using namespaces within StorageOS. Namespaces that do not pre-exist within StorageOS will be created. + type: string + type: object + vsphereVolume: + description: vsphereVolume represents a vSphere volume attached and mounted on kubelets host machine + properties: + fsType: + description: fsType is filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. + type: string + storagePolicyID: + description: storagePolicyID is the storage Policy Based Management (SPBM) profile ID associated with the StoragePolicyName. + type: string + storagePolicyName: + description: storagePolicyName is the storage Policy Based Management (SPBM) profile name. + type: string + volumePath: + description: volumePath is the path that identifies vSphere volume vmdk + type: string + required: + - volumePath + type: object + required: + - name + type: object + type: array + x-kubernetes-list-map-keys: + - name + x-kubernetes-list-type: map buckets: description: Buckets that should exist in the RustFS tenant. items: @@ -761,6 +1949,25 @@ spec: nullable: true type: string type: object + oidc: + description: OpenID Connect transport settings for RustFS workloads. + nullable: true + properties: + extraCaCertSecretRef: + description: Secret key containing one or more PEM CA certificates for outbound OIDC HTTPS. + nullable: true + properties: + key: + default: ca.crt + minLength: 1 + type: string + name: + minLength: 1 + type: string + required: + - name + type: object + type: object podDeletionPolicyWhenNodeIsDown: description: |- Controls how the operator handles Pods when the node hosting them is down (NotReady/Unknown). diff --git a/deploy/rustfs-operator/templates/NOTES.txt b/deploy/rustfs-operator/templates/NOTES.txt index 91c3351..cf6f2f2 100755 --- a/deploy/rustfs-operator/templates/NOTES.txt +++ b/deploy/rustfs-operator/templates/NOTES.txt @@ -21,7 +21,7 @@ To create a Tenant resource: name: example-tenant namespace: default spec: - image: rustfs/rustfs:1.0.0-beta.10 + image: rustfs/rustfs:1.0.0 pools: - name: pool-0 servers: 2 diff --git a/docs/operator-user-guide.md b/docs/operator-user-guide.md index 96c1cc1..bb00d17 100644 --- a/docs/operator-user-guide.md +++ b/docs/operator-user-guide.md @@ -208,7 +208,7 @@ Tenant image first. Known incompatible images are blocked before rollout, and mutable tags, digest references, or custom repositories are blocked whenever the effective profile is `RuntimeDefault` unless the Tenant carries an image-bound acknowledgement. Before upgrade, either pin a verified RustFS -beta.9-or-later release tag, or verify the effective image and set +1.0.0 or later release tag, or verify the effective image and set `operator.rustfs.com/runtime-default-image-ack` to that exact image reference. The Operator also removes legacy Tenant workload Roles and RoleBindings and @@ -222,7 +222,7 @@ Granting RBAC to the generated ServiceAccount is insufficient because its Pod template disables token mounting. `createServiceAccountRbac` is retained only as an ignored compatibility field. -The built-in RustFS image fallback also changes from the mutable `latest` tag to +The built-in RustFS image fallback is now `rustfs/rustfs:1.0.0`, replacing `rustfs/rustfs:1.0.0-beta.10`. A Tenant that omits `spec.image` and has no `TENANT_RUSTFS_IMAGE` Operator environment override will therefore roll to that pinned release on reconciliation. Set `spec.image` explicitly when you want to @@ -351,7 +351,7 @@ metadata: name: dev-minimal namespace: default spec: - image: rustfs/rustfs:1.0.0-beta.10 + image: rustfs/rustfs:1.0.0 pools: - name: dev-pool servers: 1 @@ -528,17 +528,111 @@ configured Secret passes validation, the Operator reports already-running Pods; coordinated restart and hot reload are outside this feature. +#### OIDC custom CA trust + +Use `spec.oidc.extraCaCertSecretRef` when the RustFS OIDC provider is signed by +a private CA. The Secret must be in the Tenant namespace and the selected key +must contain one or more PEM-encoded CA certificates: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: oidc-extra-ca + namespace: storage +type: Opaque +stringData: + ca.crt: | + -----BEGIN CERTIFICATE----- + ... + -----END CERTIFICATE----- +--- +apiVersion: rustfs.com/v1alpha1 +kind: Tenant +metadata: + name: rustfs-a + namespace: storage +spec: + image: rustfs/rustfs:1.0.0 + oidc: + extraCaCertSecretRef: + name: oidc-extra-ca + # key defaults to ca.crt + # pools: ... +``` + +The Operator validates every certificate before applying workloads, mounts the +selected key at `/var/run/rustfs/oidc-extra-ca/ca.pem`, and sets +`RUSTFS_EXTRA_CA_CERT` to that path. The volume intentionally does not use a +`subPath`, so Kubernetes can project Secret updates into running Pods. Secret +updates enqueue all referencing Tenants and do not trigger a Pod rollout. +This feature targets RustFS GA and later images. A valid managed configuration +reports `OidcTrustReady=True`. This condition confirms that the configured +Secret key contained a valid CA bundle during the latest reconciliation. It +does not confirm that every Pod has observed the same Secret version or that +the OIDC provider is reachable. + +Kubernetes projects an updated Secret to each Pod independently. Rotate a CA by +first publishing a bundle that contains both the current and replacement roots. +Wait until every RustFS Pod has observed that bundle and verify OIDC discovery, +JWKS retrieval, and login before switching the provider certificate. Remove the +old root only after every Pod trusts the replacement root. An invalid in-place +Secret update can reach a Pod before the Operator reports +`OidcTrustReady=False`. + +This setting only extends trust for RustFS outbound OIDC connections. It is +separate from `spec.tls.caTrust`, which configures process-wide TLS and server +mTLS trust. When this field is omitted, an explicitly supplied +`RUSTFS_EXTRA_CA_CERT` in `spec.env` remains unmanaged and is passed through. + +#### Additional volumes and mounts + +Use `spec.additionalVolumes` and `spec.additionalVolumeMounts` for files that do +not have a dedicated Tenant field. The fields use the Kubernetes `Volume` and +`VolumeMount` schemas and apply to the RustFS container in every Pool: + +```yaml +spec: + env: + - name: RUSTFS_EXTRA_CA_CERT + value: /etc/rustfs/custom-ca/ca.crt + additionalVolumes: + - name: custom-ca + secret: + secretName: custom-ca + items: + - key: ca.crt + path: ca.crt + additionalVolumeMounts: + - name: custom-ca + mountPath: /etc/rustfs/custom-ca + readOnly: true + # pools: ... +``` + +Every additional mount must reference an additional volume. The Operator also +rejects duplicate names, relative paths, `..` path components, equivalent paths, +and any parent, child, or equal relationship with operator-managed data, +logging, TLS, and OIDC mounts. Kubernetes validates the volume source and +projects Secret and ConfigMap updates. Avoid `subPath` when projected updates +must reach running Pods. Changing either field changes the StatefulSet Pod +template and starts a rolling update. Install the updated CRD before applying a +Tenant that uses these fields. + ### 7.4 Workload Settings Useful Tenant-level fields: | Field | Purpose | |-------|---------| -| `image` | RustFS server image. Defaults to `TENANT_RUSTFS_IMAGE`, then the pinned fallback `rustfs/rustfs:1.0.0-beta.10`. | +| `image` | RustFS server image. Defaults to `TENANT_RUSTFS_IMAGE`, then the pinned fallback `rustfs/rustfs:1.0.0`. | | `imagePullSecret` | Image pull Secret reference. | | `imagePullPolicy` | RustFS image pull policy. | | `scheduler` | Custom scheduler name. | | `env` | Additional RustFS container environment variables. Do not override operator-managed variables. | +| `oidc` | OIDC-specific settings, including a namespaced custom CA Secret for outbound OIDC trust. | +| `additionalVolumes` | Additional Kubernetes volumes for every RustFS Pod. | +| `additionalVolumeMounts` | Additional mounts for the RustFS container in every Pool. | | `serviceAccountName` | Custom ServiceAccount for RustFS pods. | | `createServiceAccountRbac` | Deprecated compatibility field; ignored. Manage any custom ServiceAccount RBAC explicitly. | | `priorityClassName` | Tenant-level priority class. | @@ -651,6 +745,7 @@ The operator reserves these environment variables and manages them automatically - `RUSTFS_CONSOLE_ADDRESS` - `RUSTFS_CONSOLE_ENABLE` - `RUSTFS_KMS_*` variables; use `spec.encryption` instead. +- `RUSTFS_EXTRA_CA_CERT` when `spec.oidc.extraCaCertSecretRef` is configured. - TLS-related RustFS variables when Tenant TLS is enabled. For a single-pool single-node single-disk Tenant, `RUSTFS_VOLUMES` is rendered as the local data path, for example `/data/rustfs0`. Multi-pool tenants and other layouts render peer DNS URLs through the Tenant headless Service and are validated by RustFS at runtime. Set the Helm chart `clusterDomain` value when the Kubernetes cluster DNS domain is not `cluster.local`; the same domain is used for generated TLS SANs. @@ -1102,6 +1197,7 @@ Important conditions include: - `Degraded` - `SpecValid` - `CredentialsReady` +- `OidcTrustReady` - `KmsReady` - `TlsReady` - `PoolsReady` diff --git a/docs/operator-user-guide.zh-CN.md b/docs/operator-user-guide.zh-CN.md index b265f80..02f224b 100644 --- a/docs/operator-user-guide.zh-CN.md +++ b/docs/operator-user-guide.zh-CN.md @@ -190,7 +190,7 @@ StatefulSet template 如果尚未包含这些值，会在下一次 reconcile 时 较低容量运行。应先核对每个 Tenant 的镜像。已知不兼容镜像会在 rollout 前被阻断； 只要最终生效的 profile 为 `RuntimeDefault`，可变 tag、digest 引用或自定义仓库都需要 与镜像绑定的确认注解。 -升级前应固定到已验证的 RustFS beta.9 或更高 release tag；也可先验证实际生效的镜像， +升级前应固定到已验证的 RustFS 1.0.0 或更高 release tag；也可先验证实际生效的镜像， 再将 `operator.rustfs.com/runtime-default-image-ack` 设置为完全相同的镜像引用。 Operator 还会删除旧版本为 Tenant workload 创建的 Role 和 RoleBinding，并禁止其 @@ -202,8 +202,8 @@ template 变化发生一次滚动更新。若自定义镜像需要调用 Kuberne 因为其 Pod template 会禁用 token 挂载。`createServiceAccountRbac` 仅作为已忽略的 兼容字段保留。 -内置 RustFS 镜像 fallback 也会从可变的 `latest` 改为 -`rustfs/rustfs:1.0.0-beta.10`。未设置 `spec.image`，且 Operator 没有配置 +内置 RustFS 镜像 fallback 现已从 `rustfs/rustfs:1.0.0-beta.10` 更新为 +`rustfs/rustfs:1.0.0`。未设置 `spec.image`，且 Operator 没有配置 `TENANT_RUSTFS_IMAGE` 环境变量覆盖的 Tenant，会在 reconcile 时滚动到该固定版本。 若希望 RustFS 升级节奏独立于 Operator 默认值，应显式设置 `spec.image`。 @@ -326,7 +326,7 @@ metadata: name: dev-minimal namespace: default spec: - image: rustfs/rustfs:1.0.0-beta.10 + image: rustfs/rustfs:1.0.0 pools: - name: dev-pool servers: 1 @@ -500,17 +500,103 @@ spec: `RpcAuthReady=True`。更新 Secret 不会改变已运行 Pod 的进程环境；协调重启和 热加载不在此功能范围内。 +#### OIDC 自定义 CA 信任 + +当 RustFS 使用私有 CA 签发证书的 OIDC provider 时，配置 +`spec.oidc.extraCaCertSecretRef`。Secret 必须与 Tenant 位于同一 namespace，所选 key +必须包含一个或多个 PEM 编码的 CA 证书： + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: oidc-extra-ca + namespace: storage +type: Opaque +stringData: + ca.crt: | + -----BEGIN CERTIFICATE----- + ... + -----END CERTIFICATE----- +--- +apiVersion: rustfs.com/v1alpha1 +kind: Tenant +metadata: + name: rustfs-a + namespace: storage +spec: + image: rustfs/rustfs:1.0.0 + oidc: + extraCaCertSecretRef: + name: oidc-extra-ca + # key 默认是 ca.crt + # pools: ... +``` + +Operator 会在应用 workload 前校验每张证书，把所选 key 挂载到 +`/var/run/rustfs/oidc-extra-ca/ca.pem`，并将 `RUSTFS_EXTRA_CA_CERT` 设置为该路径。 +该 volume 刻意不使用 `subPath`，因此 Kubernetes 可以把 Secret 更新投射到运行中的 +Pod。Secret 更新会触发所有引用该 Secret 的 Tenant 重新 reconcile，但不会触发 Pod +滚动更新。该功能支持 RustFS GA 以及之后的镜像。有效的托管配置会报告 +`OidcTrustReady=True`。该条件表示最近一次 reconcile 检查的 Secret key 包含有效 CA +bundle，无法证明所有 Pod 已经读取相同 Secret 版本，也无法证明 OIDC provider 可以访问。 + +Kubernetes 会分别向每个 Pod 投射 Secret 更新。轮换 CA 时，应先发布同时包含当前根证书 +和新根证书的 bundle，等待每个 RustFS Pod 读取该 bundle，并检查 OIDC discovery、JWKS +读取和登录流程。完成 provider 证书切换并确认所有 Pod 信任新根证书后，才可删除旧根 +证书。无效的 Secret 原位更新可能先进入 Pod，随后 Operator 才会报告 +`OidcTrustReady=False`。 + +此配置只扩展 RustFS 出站 OIDC 连接的信任，不等同于配置进程级 TLS 和服务端 mTLS +信任的 `spec.tls.caTrust`。省略该字段时，`spec.env` 中显式提供的 +`RUSTFS_EXTRA_CA_CERT` 仍会按非托管环境变量传入。 + +#### 附加 Volume 和 VolumeMount + +对于没有 Tenant 专用字段的文件，可以使用 `spec.additionalVolumes` 和 +`spec.additionalVolumeMounts`。这两个字段采用 Kubernetes `Volume` 和 `VolumeMount` +结构，并应用到每个 Pool 的 RustFS 容器： + +```yaml +spec: + env: + - name: RUSTFS_EXTRA_CA_CERT + value: /etc/rustfs/custom-ca/ca.crt + additionalVolumes: + - name: custom-ca + secret: + secretName: custom-ca + items: + - key: ca.crt + path: ca.crt + additionalVolumeMounts: + - name: custom-ca + mountPath: /etc/rustfs/custom-ca + readOnly: true + # pools: ... +``` + +每个附加 VolumeMount 必须引用 `additionalVolumes` 中的 Volume。Operator 会拒绝重复 +名称、相对路径、`..` 路径组件、等价路径，以及与数据、日志、TLS、OIDC 管理路径形成 +相同、父目录或子目录关系的配置。Kubernetes 负责校验 VolumeSource，并把 Secret 和 +ConfigMap 更新投射到 Pod。需要让更新进入运行中的 Pod 时，请勿使用 `subPath`。修改这 +两个字段会改变 StatefulSet PodTemplate，并触发滚动更新。应用包含这些字段的 Tenant +之前，需要更新 CRD。 + ### 7.4 工作负载配置 常用 Tenant 级字段： | 字段 | 用途 | |------|------| -| `image` | RustFS server 镜像。未配置时依次使用 `TENANT_RUSTFS_IMAGE` 和固定 fallback `rustfs/rustfs:1.0.0-beta.10`。 | +| `image` | RustFS server 镜像。未配置时依次使用 `TENANT_RUSTFS_IMAGE` 和固定 fallback `rustfs/rustfs:1.0.0`。 | | `imagePullSecret` | 镜像拉取 Secret。 | | `imagePullPolicy` | RustFS 镜像拉取策略。 | | `scheduler` | 自定义 scheduler 名称。 | | `env` | 额外 RustFS 容器环境变量。不要覆盖 Operator 自动管理的变量。 | +| `oidc` | OIDC 专用设置，包括用于出站 OIDC 信任的同 namespace 自定义 CA Secret。 | +| `additionalVolumes` | 添加到每个 RustFS Pod 的 Kubernetes Volume。 | +| `additionalVolumeMounts` | 添加到每个 Pool 中 RustFS 容器的 VolumeMount。 | | `serviceAccountName` | RustFS Pod 使用的自定义 ServiceAccount。 | | `createServiceAccountRbac` | 已废弃的兼容字段，不再生效；自定义 ServiceAccount 所需 RBAC 必须显式管理。 | | `priorityClassName` | Tenant 级 PriorityClass。 | @@ -607,6 +693,7 @@ Operator 会自动管理以下环境变量： - `RUSTFS_CONSOLE_ADDRESS` - `RUSTFS_CONSOLE_ENABLE` - `RUSTFS_KMS_*` 变量；请改用 `spec.encryption` 配置 +- 配置 `spec.oidc.extraCaCertSecretRef` 时的 `RUSTFS_EXTRA_CA_CERT` - 启用 TLS 时的 RustFS TLS 相关变量 对于单 pool 的单节点单盘 Tenant，`RUSTFS_VOLUMES` 会渲染为本地数据路径，例如 `/data/rustfs0`。多 pool Tenant 和其他布局仍会通过 Tenant headless Service 渲染 peer DNS URL，并由 RustFS 在运行时校验。当 Kubernetes 集群 DNS 域不是 `cluster.local` 时，请设置 Helm chart 的 `clusterDomain`；自动生成的 TLS SAN 也会使用同一个域。 @@ -1057,6 +1144,7 @@ kubectl describe tenant -n - `Degraded` - `SpecValid` - `CredentialsReady` +- `OidcTrustReady` - `KmsReady` - `TlsReady` - `PoolsReady` diff --git a/e2e/README.md b/e2e/README.md index e157de1..6139c6c 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -86,7 +86,7 @@ operator ns: rustfs-system test namespace: rustfs-e2e-smoke tenant name: e2e-tenant console URL: http://127.0.0.1:19090 -rustfs image: rustfs/rustfs:1.0.0-beta.10 +rustfs image: rustfs/rustfs:1.0.0 storage class: local-storage PV count: 12 kind config: e2e/manifests/kind-rustfs-e2e.yaml diff --git a/e2e/src/framework/config.rs b/e2e/src/framework/config.rs index 21df04f..d140564 100644 --- a/e2e/src/framework/config.rs +++ b/e2e/src/framework/config.rs @@ -20,7 +20,7 @@ use std::time::Duration; pub const DEFAULT_CLUSTER_NAME: &str = "rustfs-e2e"; pub const DEFAULT_STORAGE_HOST_DIR_PREFIX: &str = "/tmp/rustfs-e2e-storage"; -pub const DEFAULT_RUSTFS_IMAGE: &str = "rustfs/rustfs:1.0.0-beta.10"; +pub const DEFAULT_RUSTFS_IMAGE: &str = "rustfs/rustfs:1.0.0"; pub const DEFAULT_CERT_MANAGER_VERSION: &str = "v1.16.2"; pub const KIND_WORKER_COUNT: usize = 3; diff --git a/e2e/src/framework/images.rs b/e2e/src/framework/images.rs index b7eeb1f..48672f5 100644 --- a/e2e/src/framework/images.rs +++ b/e2e/src/framework/images.rs @@ -74,7 +74,7 @@ mod tests { [ "rustfs/operator:e2e", "rustfs/console-web:e2e", - "rustfs/rustfs:1.0.0-beta.10", + "rustfs/rustfs:1.0.0", "quay.io/jetstack/cert-manager-controller:v1.16.2", "quay.io/jetstack/cert-manager-webhook:v1.16.2", "quay.io/jetstack/cert-manager-cainjector:v1.16.2", diff --git a/e2e/src/framework/kind.rs b/e2e/src/framework/kind.rs index 7c725a2..866ea7f 100644 --- a/e2e/src/framework/kind.rs +++ b/e2e/src/framework/kind.rs @@ -25,7 +25,7 @@ use crate::framework::{ const RUSTFS_FORMAT_MARKER_PATHS: [&str; 2] = [".rustfs.sys/format.json", ".minio.sys/format.json"]; const DOCKER_ROOT_UID: u32 = 0; -const CLEANUP_HELPER_FALLBACK_IMAGES: [&str; 2] = ["rustfs/rustfs:1.0.0-beta.10", "busybox:latest"]; +const CLEANUP_HELPER_FALLBACK_IMAGES: [&str; 2] = ["rustfs/rustfs:1.0.0", "busybox:latest"]; #[derive(Debug, Clone)] pub struct KindCluster { @@ -395,12 +395,11 @@ mod tests { #[test] fn ctr_import_image_command_streams_docker_archive_to_node_containerd() { let kind = KindCluster::new(E2eConfig::defaults()); - let command = - kind.ctr_import_image_command("rustfs/rustfs:1.0.0-beta.10", "rustfs-e2e-worker"); + let command = kind.ctr_import_image_command("rustfs/rustfs:1.0.0", "rustfs-e2e-worker"); assert_eq!( command.display(), - "sh -c docker save \"$1\" | docker exec --privileged -i \"$2\" ctr --namespace=k8s.io images import --digests --snapshotter=overlayfs - sh rustfs/rustfs:1.0.0-beta.10 rustfs-e2e-worker" + "sh -c docker save \"$1\" | docker exec --privileged -i \"$2\" ctr --namespace=k8s.io images import --digests --snapshotter=overlayfs - sh rustfs/rustfs:1.0.0 rustfs-e2e-worker" ); } @@ -475,7 +474,7 @@ mod tests { let command = kind .docker_clean_host_storage_command( std::path::Path::new("/tmp/rustfs-e2e-storage-1"), - "rustfs/rustfs:1.0.0-beta.10", + "rustfs/rustfs:1.0.0", 1000, 1000, ) @@ -483,12 +482,12 @@ mod tests { assert_eq!( command.display(), - "docker run --rm --pull never --user 0:0 --entrypoint /bin/sh -v /tmp/rustfs-e2e-storage-1:/e2e-storage rustfs/rustfs:1.0.0-beta.10 -c find /e2e-storage -mindepth 1 -maxdepth 1 -exec rm -rf -- {} + && chown 1000:1000 /e2e-storage" + "docker run --rm --pull never --user 0:0 --entrypoint /bin/sh -v /tmp/rustfs-e2e-storage-1:/e2e-storage rustfs/rustfs:1.0.0 -c find /e2e-storage -mindepth 1 -maxdepth 1 -exec rm -rf -- {} + && chown 1000:1000 /e2e-storage" ); assert!( kind.docker_clean_host_storage_command( std::path::Path::new("/tmp/other"), - "rustfs/rustfs:1.0.0-beta.10", + "rustfs/rustfs:1.0.0", 1000, 1000, ) @@ -528,7 +527,7 @@ mod tests { kind.cleanup_helper_images(), vec![ "rustfs/operator:e2e".to_string(), - "rustfs/rustfs:1.0.0-beta.10".to_string(), + "rustfs/rustfs:1.0.0".to_string(), "busybox:latest".to_string(), ] ); diff --git a/e2e/src/framework/resources.rs b/e2e/src/framework/resources.rs index 54f5d6a..0543b8f 100644 --- a/e2e/src/framework/resources.rs +++ b/e2e/src/framework/resources.rs @@ -366,7 +366,7 @@ mod tests { assert!(manifest.contains("kind: Tenant")); assert!(manifest.contains("namespace: rustfs-e2e-smoke")); - assert!(manifest.contains("image: rustfs/rustfs:1.0.0-beta.10")); + assert!(manifest.contains("image: rustfs/rustfs:1.0.0")); assert!(!manifest.contains("seccompProfile:")); assert!(!manifest.contains("operator.rustfs.com/runtime-default-image-ack")); assert!(manifest.contains("storageClassName: local-storage")); diff --git a/e2e/src/framework/tenant_factory.rs b/e2e/src/framework/tenant_factory.rs index 3f72321..9aee36d 100644 --- a/e2e/src/framework/tenant_factory.rs +++ b/e2e/src/framework/tenant_factory.rs @@ -266,7 +266,7 @@ mod tests { let tenant = TenantTemplate::real_cluster( "rustfs-fault-test", "fault-test-tenant", - "rustfs/rustfs:1.0.0-beta.10", + "rustfs/rustfs:1.0.0", "fast-csi", "fault-test-tenant-credentials", ) diff --git a/examples/README.md b/examples/README.md index f2bdd13..3173cb0 100755 --- a/examples/README.md +++ b/examples/README.md @@ -382,7 +382,7 @@ metadata: namespace: spec: # Container image (optional) - image: rustfs/rustfs:1.0.0-beta.10 + image: rustfs/rustfs:1.0.0 # Storage pools (required, at least one) pools: diff --git a/examples/cert-manager-ca-trust-tenant.yaml b/examples/cert-manager-ca-trust-tenant.yaml index 707a464..0dfb932 100644 --- a/examples/cert-manager-ca-trust-tenant.yaml +++ b/examples/cert-manager-ca-trust-tenant.yaml @@ -19,7 +19,7 @@ metadata: name: cert-manager-ca-trust namespace: rustfs-system spec: - image: rustfs/rustfs:1.0.0-beta.10 + image: rustfs/rustfs:1.0.0 tls: mode: certManager diff --git a/examples/cluster-expansion-tenant.yaml b/examples/cluster-expansion-tenant.yaml index 2a4f8a5..6a9c242 100755 --- a/examples/cluster-expansion-tenant.yaml +++ b/examples/cluster-expansion-tenant.yaml @@ -20,7 +20,7 @@ metadata: labels: phase: expansion spec: - image: rustfs/rustfs:1.0.0-beta.10 + image: rustfs/rustfs:1.0.0 pools: # ORIGINAL POOL (v1): Existing deployment diff --git a/examples/custom-rbac-tenant.yaml b/examples/custom-rbac-tenant.yaml index 62928ca..cba1645 100755 --- a/examples/custom-rbac-tenant.yaml +++ b/examples/custom-rbac-tenant.yaml @@ -90,7 +90,7 @@ metadata: name: cloud-identity-tenant namespace: default spec: - image: rustfs/rustfs:1.0.0-beta.10 + image: rustfs/rustfs:1.0.0 serviceAccountName: cloud-identity-sa pools: - name: pool-0 diff --git a/examples/geographic-pools-tenant.yaml b/examples/geographic-pools-tenant.yaml index 76dd1ca..e7d8772 100755 --- a/examples/geographic-pools-tenant.yaml +++ b/examples/geographic-pools-tenant.yaml @@ -37,7 +37,7 @@ metadata: architecture: multi-region compliance: gdpr-ready spec: - image: rustfs/rustfs:1.0.0-beta.10 + image: rustfs/rustfs:1.0.0 pools: # US REGION POOL diff --git a/examples/hardware-pools-tenant.yaml b/examples/hardware-pools-tenant.yaml index 3034793..c7d678a 100755 --- a/examples/hardware-pools-tenant.yaml +++ b/examples/hardware-pools-tenant.yaml @@ -37,7 +37,7 @@ metadata: architecture: heterogeneous-disks environment: production spec: - image: rustfs/rustfs:1.0.0-beta.10 + image: rustfs/rustfs:1.0.0 pools: # LARGE DISK POOL: Nodes with large SSD disks diff --git a/examples/minimal-dev-tenant.yaml b/examples/minimal-dev-tenant.yaml index d2e7837..b164e1a 100755 --- a/examples/minimal-dev-tenant.yaml +++ b/examples/minimal-dev-tenant.yaml @@ -20,7 +20,7 @@ metadata: environment: development spec: # Container image (optional - uses operator default if not specified) - image: rustfs/rustfs:1.0.0-beta.10 + image: rustfs/rustfs:1.0.0 # Minimal pool configuration. # A single-node single-disk Tenant is expressed as 1 server × 1 volume. diff --git a/examples/multi-cert-tls-tenant.yaml b/examples/multi-cert-tls-tenant.yaml index a885797..f5b26dc 100644 --- a/examples/multi-cert-tls-tenant.yaml +++ b/examples/multi-cert-tls-tenant.yaml @@ -9,7 +9,7 @@ metadata: name: multi-cert-tls namespace: rustfs-system spec: - image: rustfs/rustfs:1.0.0-beta.10 + image: rustfs/rustfs:1.0.0 tls: mode: certManager diff --git a/examples/multi-pool-tenant.yaml b/examples/multi-pool-tenant.yaml index c80166b..9cfa074 100755 --- a/examples/multi-pool-tenant.yaml +++ b/examples/multi-pool-tenant.yaml @@ -4,7 +4,7 @@ metadata: name: multi-pool-tenant namespace: rustfs-system spec: - image: rustfs/rustfs:1.0.0-beta.10 + image: rustfs/rustfs:1.0.0 # Multiple pools for distributed storage pools: diff --git a/examples/production-ha-tenant.yaml b/examples/production-ha-tenant.yaml index 89d5c20..e4293fd 100755 --- a/examples/production-ha-tenant.yaml +++ b/examples/production-ha-tenant.yaml @@ -26,7 +26,7 @@ metadata: criticality: high spec: # Use specific version tag for production (check latest at https://github.com/rustfs/rustfs/releases) - image: rustfs/rustfs:1.0.0-beta.10 + image: rustfs/rustfs:1.0.0 # Production pool configuration pools: diff --git a/examples/provisioning-tenant.yaml b/examples/provisioning-tenant.yaml index 41bf9fa..e4a611a 100644 --- a/examples/provisioning-tenant.yaml +++ b/examples/provisioning-tenant.yaml @@ -59,7 +59,7 @@ metadata: name: provisioning-demo namespace: default spec: - image: rustfs/rustfs:1.0.0-beta.10 + image: rustfs/rustfs:1.0.0 credsSecret: name: provisioning-admin-creds pools: diff --git a/examples/secret-credentials-tenant.yaml b/examples/secret-credentials-tenant.yaml index 9e4abf8..03e13aa 100755 --- a/examples/secret-credentials-tenant.yaml +++ b/examples/secret-credentials-tenant.yaml @@ -40,7 +40,7 @@ metadata: namespace: default spec: # RustFS container image - image: rustfs/rustfs:1.0.0-beta.10 + image: rustfs/rustfs:1.0.0 # ✅ RECOMMENDED: Reference Secret containing credentials (OPTIONAL) # The Secret must contain 'accesskey' and 'secretkey' keys @@ -79,7 +79,7 @@ spec: # name: secure-tenant-alt # namespace: default #spec: -# image: rustfs/rustfs:1.0.0-beta.10 +# image: rustfs/rustfs:1.0.0 # # # Reference Secret via environment variables # # Note: When using this approach, you must manually specify RUSTFS_ACCESS_KEY diff --git a/examples/simple-tenant.yaml b/examples/simple-tenant.yaml index d00e7e7..30d425d 100755 --- a/examples/simple-tenant.yaml +++ b/examples/simple-tenant.yaml @@ -5,7 +5,7 @@ metadata: namespace: rustfs-system spec: # Container image for RustFS (optional, defaults to a specific version) - image: rustfs/rustfs:1.0.0-beta.10 + image: rustfs/rustfs:1.0.0 # Pod management policy for StatefulSet (defaults to Parallel) # Parallel is recommended for multi-node RustFS deployments diff --git a/examples/spot-instance-tenant.yaml b/examples/spot-instance-tenant.yaml index 8ffacca..e7335b0 100755 --- a/examples/spot-instance-tenant.yaml +++ b/examples/spot-instance-tenant.yaml @@ -39,7 +39,7 @@ metadata: cost-profile: optimized environment: production spec: - image: rustfs/rustfs:1.0.0-beta.10 + image: rustfs/rustfs:1.0.0 pools: # CRITICAL POOL: On-demand instances diff --git a/examples/tenant-4nodes.yaml b/examples/tenant-4nodes.yaml index 342a63e..8c60558 100644 --- a/examples/tenant-4nodes.yaml +++ b/examples/tenant-4nodes.yaml @@ -9,7 +9,7 @@ metadata: labels: app: rustfs spec: - image: rustfs/rustfs:1.0.0-beta.10 + image: rustfs/rustfs:1.0.0 # Kind demos load the server image into node-local containerd. Avoid forcing # Docker Hub access when the pinned image is already present. imagePullPolicy: IfNotPresent diff --git a/src/console/handlers/tenants.rs b/src/console/handlers/tenants.rs index 3823668..1f23588 100755 --- a/src/console/handlers/tenants.rs +++ b/src/console/handlers/tenants.rs @@ -1050,7 +1050,7 @@ spec: #[test] fn json_create_validates_namespace_before_kubernetes_work() { - let mut request = minimal_create_request(Some("rustfs/rustfs:1.0.0-beta.10")); + let mut request = minimal_create_request(Some("rustfs/rustfs:1.0.0")); request.namespace = "Storage_Team".to_string(); let error = tenant_from_create_request(request) @@ -1065,7 +1065,7 @@ spec: #[test] fn raw_yaml_update_validates_image_and_acknowledgement_from_the_same_request() { let mut current = crate::tests::create_test_tenant(None, None); - current.spec.image = Some("rustfs/rustfs:1.0.0-beta.10".to_string()); + current.spec.image = Some("rustfs/rustfs:1.0.0".to_string()); let image = "registry.example.com/rustfs/rustfs@sha256:0123456789abcdef"; let mut incoming = current.clone(); diff --git a/src/context.rs b/src/context.rs index 1766131..5f3bb6a 100755 --- a/src/context.rs +++ b/src/context.rs @@ -15,7 +15,7 @@ use crate::cluster_dns::ClusterDomain; use crate::types; use crate::types::v1alpha1::encryption::LocalKmsMasterKeySecretRef; -use crate::types::v1alpha1::tenant::{RpcSecretRef, Tenant}; +use crate::types::v1alpha1::tenant::{OidcExtraCaCertSecretRef, RpcSecretRef, Tenant}; use k8s_openapi::NamespaceResourceScope; use k8s_openapi::api::core::v1::Secret; use kube::api::{DeleteParams, ListParams, ObjectList, Patch, PatchParams, PostParams}; @@ -26,6 +26,7 @@ use serde::de::DeserializeOwned; use snafu::Snafu; use snafu::futures::TryFutureExt; use std::fmt::Debug; +use std::io::Cursor; use std::path::{Component, Path, PathBuf}; use tracing::info; @@ -88,6 +89,26 @@ pub enum Error { ))] RpcSecretInvalidValue { secret_name: String, key: String }, + #[snafu(display("OIDC extra CA Secret '{}' not found", name))] + OidcExtraCaSecretNotFound { name: String }, + + #[snafu(display("spec.oidc.extraCaCertSecretRef.{} must not be blank", field))] + OidcExtraCaInvalidReference { field: String }, + + #[snafu(display( + "OIDC extra CA Secret '{}' missing required key '{}'", + secret_name, + key + ))] + OidcExtraCaSecretMissingKey { secret_name: String, key: String }, + + #[snafu(display( + "OIDC extra CA Secret '{}' key '{}' must contain at least one valid PEM certificate", + secret_name, + key + ))] + OidcExtraCaBundleInvalid { secret_name: String, key: String }, + #[snafu(display("KMS secret '{}' not found", name))] KmsSecretNotFound { name: String }, @@ -374,6 +395,48 @@ fn validate_rpc_secret_value(secret: &Secret, secret_name: &str, key: &str) -> R Ok(()) } +fn validate_oidc_extra_ca_ref(secret_ref: &OidcExtraCaCertSecretRef) -> Result<(), Error> { + for (field, value) in [("name", &secret_ref.name), ("key", &secret_ref.key)] { + if value.trim().is_empty() { + return Err(Error::OidcExtraCaInvalidReference { + field: field.to_string(), + }); + } + } + Ok(()) +} + +fn validate_oidc_extra_ca_bundle( + secret: &Secret, + secret_name: &str, + key: &str, +) -> Result<(), Error> { + let Some(value) = secret.data.as_ref().and_then(|data| data.get(key)) else { + return Err(Error::OidcExtraCaSecretMissingKey { + secret_name: secret_name.to_string(), + key: key.to_string(), + }); + }; + + let invalid = || Error::OidcExtraCaBundleInvalid { + secret_name: secret_name.to_string(), + key: key.to_string(), + }; + let mut roots = rustls::RootCertStore::empty(); + let mut reader = Cursor::new(&value.0); + let mut certificates = rustls_pemfile::certs(&mut reader).peekable(); + if certificates.peek().is_none() { + return Err(invalid()); + } + for certificate in certificates { + roots + .add(certificate.map_err(|_| invalid())?) + .map_err(|_| invalid())?; + } + + Ok(()) +} + fn status_semantically_equal( current: Option<&types::v1alpha1::status::Status>, next: &types::v1alpha1::status::Status, @@ -399,6 +462,7 @@ fn normalize_status_for_compare(status: &mut types::v1alpha1::status::Status) { pub(crate) enum SecretValidationKind { Credential, Rpc, + OidcExtraCa, Kms, } @@ -435,6 +499,7 @@ pub(crate) fn map_secret_get_error( match kind { SecretValidationKind::Credential => Error::CredentialSecretNotFound { name }, SecretValidationKind::Rpc => Error::RpcSecretNotFound { name }, + SecretValidationKind::OidcExtraCa => Error::OidcExtraCaSecretNotFound { name }, SecretValidationKind::Kms => Error::KmsSecretNotFound { name }, } } @@ -722,6 +787,33 @@ impl Context { validate_rpc_secret_value(&secret, &secret_ref.name, &secret_ref.key) } + /// Validates the optional OIDC extra root CA bundle before applying workloads. + pub async fn validate_oidc_extra_ca_secret(&self, tenant: &Tenant) -> Result<(), Error> { + let Some(secret_ref) = tenant + .spec + .oidc + .as_ref() + .and_then(|oidc| oidc.extra_ca_cert_secret_ref.as_ref()) + else { + return Ok(()); + }; + + validate_oidc_extra_ca_ref(secret_ref)?; + + let secret: Secret = match self.get(&secret_ref.name, &tenant.namespace()?).await { + Ok(secret) => secret, + Err(error) => { + return Err(map_secret_get_error( + error, + secret_ref.name.clone(), + SecretValidationKind::OidcExtraCa, + )); + } + }; + + validate_oidc_extra_ca_bundle(&secret, &secret_ref.name, &secret_ref.key) + } + /// Validates encryption configuration and the KMS Secret. /// /// Checks: @@ -1072,12 +1164,13 @@ mod validate_local_kms_tests { use super::{SecretValidationKind, map_secret_get_error}; use super::{ validate_local_kms_master_key_ref, validate_local_kms_tenant, validate_no_reserved_kms_env, - validate_rpc_secret_ref, validate_rpc_secret_value, validate_secret_utf8_non_blank, + validate_oidc_extra_ca_bundle, validate_oidc_extra_ca_ref, validate_rpc_secret_ref, + validate_rpc_secret_value, validate_secret_utf8_non_blank, }; use crate::types::v1alpha1::encryption::{LocalKmsConfig, LocalKmsMasterKeySecretRef}; use crate::types::v1alpha1::persistence::PersistenceConfig; use crate::types::v1alpha1::pool::Pool; - use crate::types::v1alpha1::tenant::RpcSecretRef; + use crate::types::v1alpha1::tenant::{OidcExtraCaCertSecretRef, RpcSecretRef}; use k8s_openapi::ByteString; use k8s_openapi::api::core::v1 as corev1; use std::collections::BTreeMap; @@ -1165,6 +1258,19 @@ mod validate_local_kms_tests { assert!(matches!(err, Error::RpcSecretNotFound { name } if name == "rpc-auth")); } + #[test] + fn oidc_extra_ca_secret_get_maps_only_404_to_not_found() { + let err = map_secret_get_error( + api_error(404, "NotFound"), + "oidc-extra-ca".to_string(), + SecretValidationKind::OidcExtraCa, + ); + + assert!( + matches!(err, Error::OidcExtraCaSecretNotFound { name } if name == "oidc-extra-ca") + ); + } + #[test] fn rpc_secret_ref_rejects_blank_name_and_key() { for secret_ref in [ @@ -1232,6 +1338,73 @@ mod validate_local_kms_tests { validate_rpc_secret_value(&secret, "rpc-auth", "rpc-secret").unwrap(); } + #[test] + fn oidc_extra_ca_ref_rejects_blank_name_and_key() { + for secret_ref in [ + OidcExtraCaCertSecretRef { + name: " ".to_string(), + key: "ca.crt".to_string(), + }, + OidcExtraCaCertSecretRef { + name: "oidc-extra-ca".to_string(), + key: "\n".to_string(), + }, + ] { + let err = validate_oidc_extra_ca_ref(&secret_ref).unwrap_err(); + assert!(matches!(err, Error::OidcExtraCaInvalidReference { .. })); + } + } + + #[test] + fn oidc_extra_ca_bundle_requires_selected_secret_key() { + let secret = corev1::Secret::default(); + + let err = validate_oidc_extra_ca_bundle(&secret, "oidc-extra-ca", "ca.crt") + .expect_err("missing CA key must fail validation"); + + assert!(matches!( + err, + Error::OidcExtraCaSecretMissingKey { secret_name, key } + if secret_name == "oidc-extra-ca" && key == "ca.crt" + )); + } + + #[test] + fn oidc_extra_ca_bundle_rejects_empty_and_malformed_certificates() { + for value in [ + b"not a certificate".to_vec(), + b"-----BEGIN CERTIFICATE-----\nbm90IGEgY2VydA==\n-----END CERTIFICATE-----\n".to_vec(), + ] { + let secret = corev1::Secret { + data: Some(BTreeMap::from([("ca.crt".to_string(), ByteString(value))])), + ..Default::default() + }; + + let err = validate_oidc_extra_ca_bundle(&secret, "oidc-extra-ca", "ca.crt") + .expect_err("invalid CA bundle must fail validation"); + assert!(matches!(err, Error::OidcExtraCaBundleInvalid { .. })); + } + } + + #[test] + fn oidc_extra_ca_bundle_accepts_multiple_pem_certificates() { + let certificate_a = rcgen::generate_simple_self_signed(vec!["a.example".to_string()]) + .expect("first CA certificate should generate"); + let certificate_b = rcgen::generate_simple_self_signed(vec!["b.example".to_string()]) + .expect("second CA certificate should generate"); + let bundle = format!("{}{}", certificate_a.cert.pem(), certificate_b.cert.pem()); + let secret = corev1::Secret { + data: Some(BTreeMap::from([( + "bundle.pem".to_string(), + ByteString(bundle.into_bytes()), + )])), + ..Default::default() + }; + + validate_oidc_extra_ca_bundle(&secret, "oidc-extra-ca", "bundle.pem") + .expect("valid CA bundle should pass validation"); + } + #[test] fn local_kms_default_key_dir_ok_single_replica() { validate_local_kms_tenant(None, &[pool(1)]).unwrap(); diff --git a/src/reconcile.rs b/src/reconcile.rs index 140277a..8220513 100755 --- a/src/reconcile.rs +++ b/src/reconcile.rs @@ -939,6 +939,10 @@ pub fn error_policy(object: Arc, error: &Error, _ctx: Arc) -> A | context::Error::RpcSecretMissingKey { .. } | context::Error::RpcSecretInvalidEncoding { .. } | context::Error::RpcSecretInvalidValue { .. } + | context::Error::OidcExtraCaSecretNotFound { .. } + | context::Error::OidcExtraCaInvalidReference { .. } + | context::Error::OidcExtraCaSecretMissingKey { .. } + | context::Error::OidcExtraCaBundleInvalid { .. } | context::Error::KmsSecretNotFound { .. } | context::Error::KmsSecretMissingKey { .. } | context::Error::KmsConfigInvalid { .. } => Duration::from_secs(60), @@ -957,6 +961,7 @@ pub fn error_policy(object: Arc, error: &Error, _ctx: Arc) -> A // Use 60-second requeue to reduce event/log spam while user fixes the issue types::error::Error::ImmutableFieldModified { .. } | types::error::Error::InvalidTenantName { .. } + | types::error::Error::InvalidAdditionalVolumeSpec { .. } | types::error::Error::KmsMigrationBlocked { .. } | types::error::Error::InvalidWorkloadSecurityProfile { .. } | types::error::Error::WorkloadSecurityIncompatible { .. } @@ -996,6 +1001,10 @@ fn reconcile_error_reason(error: &Error) -> &'static str { context::Error::RpcSecretMissingKey { .. } => "RpcSecretMissingKey", context::Error::RpcSecretInvalidEncoding { .. } => "RpcSecretInvalidEncoding", context::Error::RpcSecretInvalidValue { .. } => "RpcSecretInvalidValue", + context::Error::OidcExtraCaSecretNotFound { .. } => "OidcExtraCaSecretNotFound", + context::Error::OidcExtraCaInvalidReference { .. } => "OidcExtraCaInvalidReference", + context::Error::OidcExtraCaSecretMissingKey { .. } => "OidcExtraCaSecretMissingKey", + context::Error::OidcExtraCaBundleInvalid { .. } => "OidcExtraCaBundleInvalid", context::Error::KmsSecretNotFound { .. } => "KmsSecretNotFound", context::Error::KmsSecretMissingKey { .. } => "KmsSecretMissingKey", context::Error::KmsConfigInvalid { .. } => "KmsConfigInvalid", @@ -1007,6 +1016,9 @@ fn reconcile_error_reason(error: &Error) -> &'static str { Error::Types { source } => match source { types::error::Error::InvalidTenantName { .. } => "InvalidTenantName", types::error::Error::InvalidPoolSpec { .. } => "InvalidPoolSpec", + types::error::Error::InvalidAdditionalVolumeSpec { .. } => { + "InvalidAdditionalVolumeSpec" + } types::error::Error::ImmutableFieldModified { .. } => "ImmutableFieldModified", types::error::Error::PoolDeleteBlocked { .. } => "PoolDeleteBlocked", types::error::Error::KmsMigrationBlocked { .. } => "KmsMigrationBlocked", diff --git a/src/reconcile/phases.rs b/src/reconcile/phases.rs index b3a3405..b79fa74 100644 --- a/src/reconcile/phases.rs +++ b/src/reconcile/phases.rs @@ -98,6 +98,12 @@ pub(super) async fn validate_tenant_prerequisites( return Err(e.into()); } + if let Err(e) = tenant.validate_additional_volumes() { + let status_error = StatusError::from_types_error(&e); + patch_status_error(ctx, tenant, &status_error).await; + return Err(e.into()); + } + // Block known incompatible RustFS images before creating or rolling any StatefulSet. if let Err(e) = tenant.validate_workload_security_compatibility() { let status_error = StatusError::from_types_error(&e); @@ -125,6 +131,14 @@ pub(super) async fn validate_tenant_prerequisites( return Err(e.into()); } + // Validate OIDC trust material before rolling workloads. Kubernetes would otherwise + // reject a missing Secret key, while malformed certificate data would fail RustFS startup. + if let Err(e) = ctx.validate_oidc_extra_ca_secret(tenant).await { + let status_error = StatusError::from_context_error(&e); + patch_status_error(ctx, tenant, &status_error).await; + return Err(e.into()); + } + // Validate encryption / KMS and reject raw RUSTFS_KMS_* env overrides even when // spec.encryption is omitted or disabled. if let Err(e) = ctx.validate_kms_secret(tenant).await { diff --git a/src/status.rs b/src/status.rs index 42e6ecd..28835e8 100644 --- a/src/status.rs +++ b/src/status.rs @@ -108,6 +108,32 @@ impl StatusError { secret_name, key ), ), + context::Error::OidcExtraCaSecretNotFound { name } => Self::blocked( + Reason::OidcExtraCaSecretNotFound, + ConditionType::OidcTrustReady, + format!("OIDC extra CA Secret '{}' was not found", name), + ), + context::Error::OidcExtraCaInvalidReference { field } => Self::blocked( + Reason::OidcExtraCaInvalidReference, + ConditionType::OidcTrustReady, + format!("spec.oidc.extraCaCertSecretRef.{} must not be blank", field), + ), + context::Error::OidcExtraCaSecretMissingKey { secret_name, key } => Self::blocked( + Reason::OidcExtraCaSecretMissingKey, + ConditionType::OidcTrustReady, + format!( + "OIDC extra CA Secret '{}' is missing required key '{}'", + secret_name, key + ), + ), + context::Error::OidcExtraCaBundleInvalid { secret_name, key } => Self::blocked( + Reason::OidcExtraCaBundleInvalid, + ConditionType::OidcTrustReady, + format!( + "OIDC extra CA Secret '{}' key '{}' must contain at least one valid PEM certificate", + secret_name, key + ), + ), context::Error::KmsSecretNotFound { name } => Self::blocked( Reason::KmsSecretNotFound, ConditionType::KmsReady, @@ -170,6 +196,11 @@ impl StatusError { ConditionType::SpecValid, sanitize_message(message), ), + types::error::Error::InvalidAdditionalVolumeSpec { message, .. } => Self::blocked( + Reason::InvalidAdditionalVolumeSpec, + ConditionType::SpecValid, + sanitize_message(message), + ), types::error::Error::ImmutableFieldModified { field, .. } => Self::blocked( Reason::ImmutableFieldModified, ConditionType::SpecValid, @@ -272,6 +303,7 @@ pub struct StatusBuilder { generation: Option, now: String, rpc_secret_configured: bool, + oidc_extra_ca_configured: bool, next: Status, } @@ -281,6 +313,11 @@ impl StatusBuilder { generation: tenant.metadata.generation, now: chrono::Utc::now().to_rfc3339_opts(chrono::SecondsFormat::Secs, true), rpc_secret_configured: tenant.spec.rpc_secret.is_some(), + oidc_extra_ca_configured: tenant + .spec + .oidc + .as_ref() + .is_some_and(|oidc| oidc.extra_ca_cert_secret_ref.is_some()), next: tenant.status.clone().unwrap_or_default(), } } @@ -600,6 +637,18 @@ impl StatusBuilder { self.next .remove_condition_by_type(ConditionType::RpcAuthReady.as_str()); } + + if self.oidc_extra_ca_configured { + self.set_condition( + ConditionType::OidcTrustReady, + ConditionStatus::True, + Reason::ReconcileSucceeded, + "Configured OIDC extra CA bundle in Secret is valid".to_string(), + ); + } else { + self.next + .remove_condition_by_type(ConditionType::OidcTrustReady.as_str()); + } } fn clear_stale_blocked_conditions( @@ -737,6 +786,38 @@ mod tests { assert!(status.condition(ConditionType::WorkloadsReady).is_none()); } + #[test] + fn status_builder_maps_invalid_oidc_extra_ca_bundle() { + let mut tenant = crate::tests::create_test_tenant(None, None); + tenant.spec.oidc = Some(crate::types::v1alpha1::tenant::OidcConfig { + extra_ca_cert_secret_ref: Some( + crate::types::v1alpha1::tenant::OidcExtraCaCertSecretRef { + name: "oidc-extra-ca".to_string(), + key: "ca.crt".to_string(), + }, + ), + }); + let err = context::Error::OidcExtraCaBundleInvalid { + secret_name: "oidc-extra-ca".to_string(), + key: "ca.crt".to_string(), + }; + + let status_error = StatusError::from_context_error(&err); + let mut builder = StatusBuilder::from_tenant(&tenant); + builder.mark_error(&status_error); + let status = builder.build(); + + let condition = status.condition(ConditionType::OidcTrustReady).unwrap(); + assert_eq!(condition.status, "False"); + assert_eq!(condition.reason, "OidcExtraCaBundleInvalid"); + assert_eq!(status.current_state, "Blocked"); + assert_eq!( + crate::types::v1alpha1::status::next_actions_for_reason(&condition.reason), + vec!["replaceOidcExtraCaBundle"] + ); + assert!(status.condition(ConditionType::WorkloadsReady).is_none()); + } + #[test] fn status_builder_blocks_incompatible_workload_security() { let tenant = crate::tests::create_test_tenant(None, None); @@ -783,6 +864,29 @@ mod tests { ); } + #[test] + fn status_builder_marks_invalid_additional_volumes_as_invalid_spec() { + let tenant = crate::tests::create_test_tenant(None, None); + let err = types::error::Error::InvalidAdditionalVolumeSpec { + name: tenant.name(), + message: "spec.additionalVolumeMounts contains a managed path".to_string(), + }; + + let status_error = StatusError::from_types_error(&err); + let mut builder = StatusBuilder::from_tenant(&tenant); + builder.mark_error(&status_error); + let status = builder.build(); + + let condition = status.condition(ConditionType::SpecValid).unwrap(); + assert_eq!(condition.status, "False"); + assert_eq!(condition.reason, "InvalidAdditionalVolumeSpec"); + assert_eq!(status.current_state, "Blocked"); + assert_eq!( + crate::types::v1alpha1::status::next_actions_for_reason(&condition.reason), + vec!["fixAdditionalVolumeSpec"] + ); + } + #[test] fn successful_status_reports_rpc_auth_ready_only_for_managed_secret() { let mut unmanaged = crate::tests::create_test_tenant(None, None); @@ -811,6 +915,36 @@ mod tests { assert_eq!(condition.message, "Configured RPC Secret is valid"); } + #[test] + fn successful_status_only_reports_oidc_trust_when_configured() { + let tenant = crate::tests::create_test_tenant(None, None); + let mut builder = StatusBuilder::from_tenant(&tenant); + builder.finish_success(); + let status = builder.build(); + + assert!(status.condition(ConditionType::OidcTrustReady).is_none()); + + let mut tenant = crate::tests::create_test_tenant(None, None); + tenant.spec.oidc = Some(crate::types::v1alpha1::tenant::OidcConfig { + extra_ca_cert_secret_ref: Some( + crate::types::v1alpha1::tenant::OidcExtraCaCertSecretRef { + name: "oidc-extra-ca".to_string(), + key: "ca.crt".to_string(), + }, + ), + }); + let mut builder = StatusBuilder::from_tenant(&tenant); + builder.finish_success(); + let status = builder.build(); + + let condition = status.condition(ConditionType::OidcTrustReady).unwrap(); + assert_eq!(condition.status, "True"); + assert_eq!( + condition.message, + "Configured OIDC extra CA bundle in Secret is valid" + ); + } + #[test] fn successful_status_prunes_rpc_auth_ready_after_secret_is_unconfigured() { let mut tenant = crate::tests::create_test_tenant(None, None); diff --git a/src/tenant_reference_index.rs b/src/tenant_reference_index.rs index c68eda3..0c6dd31 100644 --- a/src/tenant_reference_index.rs +++ b/src/tenant_reference_index.rs @@ -262,7 +262,7 @@ mod tests { ConfigMapKeyReference, PolicyDocumentSource, ProvisioningPolicy, ProvisioningUser, UserCredentialsSecretRef, }; - use crate::types::v1alpha1::tenant::RpcSecretRef; + use crate::types::v1alpha1::tenant::{OidcConfig, OidcExtraCaCertSecretRef, RpcSecretRef}; use crate::types::v1alpha1::tls::{ CaTrustConfig, CertManagerTlsConfig, SecretKeyReference, TlsCertificateConfig, TlsConfig, }; @@ -305,6 +305,12 @@ mod tests { name: "rpc".to_string(), key: "secret".to_string(), }); + tenant.spec.oidc = Some(OidcConfig { + extra_ca_cert_secret_ref: Some(OidcExtraCaCertSecretRef { + name: "oidc-extra-ca".to_string(), + key: "ca.crt".to_string(), + }), + }); tenant.spec.env.push(corev1::EnvVar { name: "OIDC_CLIENT_SECRET".to_string(), value_from: Some(corev1::EnvVarSource { @@ -395,6 +401,7 @@ mod tests { "env-secret", "image-pull", "rpc", + "oidc-extra-ca", "provisioned-user-credentials", "legacy-user", "vault-token", diff --git a/src/types/error.rs b/src/types/error.rs index 31844ac..2fde6e2 100755 --- a/src/types/error.rs +++ b/src/types/error.rs @@ -39,6 +39,13 @@ pub enum Error { #[snafu(display("invalid pool specification for tenant '{}': {}", name, message))] InvalidPoolSpec { name: String, message: String }, + #[snafu(display( + "invalid additional volume specification for tenant '{}': {}", + name, + message + ))] + InvalidAdditionalVolumeSpec { name: String, message: String }, + #[snafu(display("KMS migration is blocked for tenant '{}': {}", name, message))] KmsMigrationBlocked { name: String, message: String }, diff --git a/src/types/v1alpha1.rs b/src/types/v1alpha1.rs index c46cc62..1601fae 100755 --- a/src/types/v1alpha1.rs +++ b/src/types/v1alpha1.rs @@ -172,6 +172,26 @@ mod tenant_provisioning_crd_tests { spec["properties"]["rpcSecret"]["required"], json!(["key", "name"]) ); + assert_eq!( + spec["properties"]["additionalVolumes"]["x-kubernetes-list-type"], + json!("map") + ); + assert_eq!( + spec["properties"]["additionalVolumes"]["x-kubernetes-list-map-keys"], + json!(["name"]) + ); + assert_eq!( + spec["properties"]["additionalVolumeMounts"]["x-kubernetes-list-type"], + json!("map") + ); + assert_eq!( + spec["properties"]["additionalVolumeMounts"]["x-kubernetes-list-map-keys"], + json!(["mountPath"]) + ); + assert_eq!( + spec["properties"]["additionalVolumeMounts"]["items"]["required"], + json!(["mountPath", "name"]) + ); assert_eq!(spec["properties"]["policies"]["type"], json!("array")); assert_eq!(spec["properties"]["users"]["type"], json!("array")); diff --git a/src/types/v1alpha1/status.rs b/src/types/v1alpha1/status.rs index ef9879a..0ca12f6 100755 --- a/src/types/v1alpha1/status.rs +++ b/src/types/v1alpha1/status.rs @@ -27,6 +27,7 @@ pub enum ConditionType { SpecValid, CredentialsReady, RpcAuthReady, + OidcTrustReady, KmsReady, TlsReady, PoolsReady, @@ -43,6 +44,7 @@ impl ConditionType { Self::SpecValid => "SpecValid", Self::CredentialsReady => "CredentialsReady", Self::RpcAuthReady => "RpcAuthReady", + Self::OidcTrustReady => "OidcTrustReady", Self::KmsReady => "KmsReady", Self::TlsReady => "TlsReady", Self::PoolsReady => "PoolsReady", @@ -59,6 +61,7 @@ impl ConditionType { Self::SpecValid, Self::CredentialsReady, Self::RpcAuthReady, + Self::OidcTrustReady, Self::KmsReady, Self::TlsReady, Self::PoolsReady, @@ -116,6 +119,7 @@ pub enum Reason { ReconcileSucceeded, InvalidTenantName, InvalidPoolSpec, + InvalidAdditionalVolumeSpec, ImmutableFieldModified, CredentialSecretNotFound, CredentialSecretMissingKey, @@ -126,6 +130,10 @@ pub enum Reason { RpcSecretMissingKey, RpcSecretInvalidEncoding, RpcSecretInvalidValue, + OidcExtraCaSecretNotFound, + OidcExtraCaInvalidReference, + OidcExtraCaSecretMissingKey, + OidcExtraCaBundleInvalid, KmsSecretNotFound, KmsSecretMissingKey, KmsConfigInvalid, @@ -195,6 +203,7 @@ impl Reason { Self::ReconcileSucceeded => "ReconcileSucceeded", Self::InvalidTenantName => "InvalidTenantName", Self::InvalidPoolSpec => "InvalidPoolSpec", + Self::InvalidAdditionalVolumeSpec => "InvalidAdditionalVolumeSpec", Self::ImmutableFieldModified => "ImmutableFieldModified", Self::CredentialSecretNotFound => "CredentialSecretNotFound", Self::CredentialSecretMissingKey => "CredentialSecretMissingKey", @@ -205,6 +214,10 @@ impl Reason { Self::RpcSecretMissingKey => "RpcSecretMissingKey", Self::RpcSecretInvalidEncoding => "RpcSecretInvalidEncoding", Self::RpcSecretInvalidValue => "RpcSecretInvalidValue", + Self::OidcExtraCaSecretNotFound => "OidcExtraCaSecretNotFound", + Self::OidcExtraCaInvalidReference => "OidcExtraCaInvalidReference", + Self::OidcExtraCaSecretMissingKey => "OidcExtraCaSecretMissingKey", + Self::OidcExtraCaBundleInvalid => "OidcExtraCaBundleInvalid", Self::KmsSecretNotFound => "KmsSecretNotFound", Self::KmsSecretMissingKey => "KmsSecretMissingKey", Self::KmsConfigInvalid => "KmsConfigInvalid", @@ -491,6 +504,7 @@ pub fn is_blocked_reason(reason: &str) -> bool { reason, "InvalidTenantName" | "InvalidPoolSpec" + | "InvalidAdditionalVolumeSpec" | "ImmutableFieldModified" | "CredentialSecretNotFound" | "CredentialSecretMissingKey" @@ -501,6 +515,10 @@ pub fn is_blocked_reason(reason: &str) -> bool { | "RpcSecretMissingKey" | "RpcSecretInvalidEncoding" | "RpcSecretInvalidValue" + | "OidcExtraCaSecretNotFound" + | "OidcExtraCaInvalidReference" + | "OidcExtraCaSecretMissingKey" + | "OidcExtraCaBundleInvalid" | "KmsSecretNotFound" | "KmsSecretMissingKey" | "KmsConfigInvalid" @@ -567,6 +585,7 @@ fn condition_matches_observed_generation(status: &Status, condition: &Condition) pub fn next_actions_for_reason(reason: &str) -> Vec<&'static str> { match reason { "InvalidPoolSpec" => vec!["fixPoolSpec"], + "InvalidAdditionalVolumeSpec" => vec!["fixAdditionalVolumeSpec"], "CredentialSecretNotFound" => vec!["createCredentialSecret"], "CredentialSecretMissingKey" => vec!["addRequiredSecretKey"], "CredentialSecretInvalidEncoding" => vec!["replaceSecretValueWithUtf8"], @@ -576,6 +595,10 @@ pub fn next_actions_for_reason(reason: &str) -> Vec<&'static str> { "RpcSecretMissingKey" => vec!["addRequiredRpcSecretKey"], "RpcSecretInvalidEncoding" => vec!["replaceRpcSecretValueWithUtf8"], "RpcSecretInvalidValue" => vec!["rotateRpcSecret"], + "OidcExtraCaSecretNotFound" => vec!["createOidcExtraCaSecret"], + "OidcExtraCaInvalidReference" => vec!["fixOidcExtraCaSecretRef"], + "OidcExtraCaSecretMissingKey" => vec!["addRequiredOidcExtraCaKey"], + "OidcExtraCaBundleInvalid" => vec!["replaceOidcExtraCaBundle"], "KmsSecretNotFound" => vec!["createKmsSecret"], "KmsSecretMissingKey" => vec!["addRequiredKmsSecretKey"], "KmsConfigInvalid" => vec!["fixKmsConfig"], diff --git a/src/types/v1alpha1/tenant.rs b/src/types/v1alpha1/tenant.rs index fb87d51..90cf298 100755 --- a/src/types/v1alpha1/tenant.rs +++ b/src/types/v1alpha1/tenant.rs @@ -57,6 +57,31 @@ pub struct RpcSecretRef { pub key: String, } +fn default_oidc_extra_ca_key() -> String { + "ca.crt".to_string() +} + +/// Reference to a PEM CA bundle used for outbound OIDC HTTPS connections. +#[derive(Deserialize, Serialize, Clone, Debug, KubeSchema, Default, PartialEq)] +#[serde(rename_all = "camelCase")] +pub struct OidcExtraCaCertSecretRef { + #[schemars(length(min = 1))] + pub name: String, + + #[serde(default = "default_oidc_extra_ca_key")] + #[schemars(length(min = 1))] + pub key: String, +} + +/// OpenID Connect transport configuration. +#[derive(Deserialize, Serialize, Clone, Debug, KubeSchema, Default, PartialEq)] +#[serde(rename_all = "camelCase")] +pub struct OidcConfig { + /// Secret key containing one or more PEM CA certificates for outbound OIDC HTTPS. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub extra_ca_cert_secret_ref: Option, +} + #[derive(CustomResource, Deserialize, Serialize, Clone, Debug, KubeSchema, Default)] #[kube( group = "rustfs.com", @@ -124,6 +149,22 @@ pub struct TenantSpec { #[serde(default, skip_serializing_if = "Vec::is_empty")] pub env: Vec, + /// Additional Kubernetes volumes added to every RustFS Pod. + /// + /// These volumes are mounted only when referenced by `additionalVolumeMounts`. + #[schemars( + extend("x-kubernetes-list-type" = "map", "x-kubernetes-list-map-keys" = ["name"]) + )] + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub additional_volumes: Vec, + + /// Additional volume mounts added to the RustFS container in every Pool. + #[schemars( + extend("x-kubernetes-list-type" = "map", "x-kubernetes-list-map-keys" = ["mountPath"]) + )] + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub additional_volume_mounts: Vec, + /// Tenant Service IP family policy and RustFS listen addresses. /// When omitted, generated Services inherit the cluster default and RustFS listens on IPv4. #[serde(default, skip_serializing_if = "Option::is_none")] @@ -139,6 +180,10 @@ pub struct TenantSpec { #[serde(default, skip_serializing_if = "Option::is_none")] pub tls: Option, + /// OpenID Connect transport settings for RustFS workloads. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub oidc: Option, + // #[serde(default, skip_serializing_if = "Option::is_none")] // pub request_auto_cert: Option, // @@ -261,7 +306,7 @@ pub struct TenantSpec { } impl TenantSpec { - /// Returns every Kubernetes Secret name referenced by the Tenant specification. + /// Returns Secret names whose lifecycle requires Tenant reconciliation. pub(crate) fn referenced_secret_names(&self) -> BTreeSet { let mut names = BTreeSet::new(); { @@ -289,6 +334,13 @@ impl TenantSpec { if let Some(secret) = &self.rpc_secret { insert(&secret.name); } + if let Some(secret) = self + .oidc + .as_ref() + .and_then(|oidc| oidc.extra_ca_cert_secret_ref.as_ref()) + { + insert(&secret.name); + } for user in &self.users { insert(user.credentials_secret_name()); } @@ -542,12 +594,28 @@ pub fn validate_dns1035_label(name: &str) -> Result<(), types::error::Error> { #[cfg(test)] mod tests { - use super::TenantSpec; + use super::{OidcConfig, OidcExtraCaCertSecretRef, TenantSpec}; use crate::types::v1alpha1::status::pool::PoolState; use k8s_openapi::api::apps::v1::{StatefulSet, StatefulSetSpec, StatefulSetStatus}; use k8s_openapi::apimachinery::pkg::apis::meta::v1::ObjectMeta; use std::collections::BTreeSet; + #[test] + fn oidc_extra_ca_secret_ref_defaults_to_ca_crt() { + let config: OidcConfig = serde_json::from_value(serde_json::json!({ + "extraCaCertSecretRef": { "name": "oidc-extra-ca" } + })) + .expect("OIDC configuration should deserialize"); + + assert_eq!( + config.extra_ca_cert_secret_ref, + Some(OidcExtraCaCertSecretRef { + name: "oidc-extra-ca".to_string(), + key: "ca.crt".to_string(), + }) + ); + } + #[test] fn referenced_secret_names_cover_all_tenant_secret_sources() { let spec: TenantSpec = serde_json::from_value(serde_json::json!({ @@ -561,6 +629,9 @@ mod tests { }], "credsSecret": { "name": "admin-creds" }, "rpcSecret": { "name": "rpc-auth", "key": "rpc-secret" }, + "oidc": { + "extraCaCertSecretRef": { "name": "oidc-extra-ca" } + }, "users": [ { "name": "legacy-user", "policies": ["readwrite"] }, { @@ -612,6 +683,7 @@ mod tests { "image-pull".to_string(), "legacy-user".to_string(), "local-master-key".to_string(), + "oidc-extra-ca".to_string(), "public-client-ca".to_string(), "public-tls".to_string(), "rpc-auth".to_string(), diff --git a/src/types/v1alpha1/tenant/helper.rs b/src/types/v1alpha1/tenant/helper.rs index aa6a654..95e803d 100755 --- a/src/types/v1alpha1/tenant/helper.rs +++ b/src/types/v1alpha1/tenant/helper.rs @@ -13,7 +13,7 @@ // limitations under the License. /// Default RustFS container image when neither spec.image nor TENANT_RUSTFS_IMAGE is set. -pub const DEFAULT_RUSTFS_IMAGE: &str = "rustfs/rustfs:1.0.0-beta.10"; +pub const DEFAULT_RUSTFS_IMAGE: &str = "rustfs/rustfs:1.0.0"; pub(crate) fn get_rustfs_image() -> Option { std::env::var("TENANT_RUSTFS_IMAGE").ok() diff --git a/src/types/v1alpha1/tenant/workloads.rs b/src/types/v1alpha1/tenant/workloads.rs index ba8eae0..956b2bf 100755 --- a/src/types/v1alpha1/tenant/workloads.rs +++ b/src/types/v1alpha1/tenant/workloads.rs @@ -26,18 +26,24 @@ use crate::types::v1alpha1::security_context::{ MAX_KUBERNETES_ID, PodSecurityContextOverride, effective_run_as_non_root, security_context_pair_delegates_to_platform, }; -use crate::types::v1alpha1::tls::{TlsPlan, http_probe}; +use crate::types::v1alpha1::tls::{TLS_SERVER_VOLUME_NAME, TlsPlan, http_probe}; use k8s_openapi::DeepMerge; use k8s_openapi::api::apps::v1; use k8s_openapi::api::core::v1 as corev1; use k8s_openapi::apimachinery::pkg::api::resource::Quantity; use k8s_openapi::apimachinery::pkg::apis::meta::v1 as metav1; +use std::collections::BTreeSet; +use std::path::{Component, Path, PathBuf}; const LOCAL_KMS_KEY_DIR_ENV: &str = "RUSTFS_KMS_KEY_DIR"; const LOCAL_KMS_LOCAL_KEY_DIR_ENV: &str = "RUSTFS_KMS_LOCAL_KEY_DIR"; const LOCAL_KMS_MASTER_KEY_ENV: &str = "RUSTFS_KMS_LOCAL_MASTER_KEY"; const KMS_ALLOW_INSECURE_DEV_DEFAULTS_ENV: &str = "RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS"; const RPC_SECRET_ENV: &str = "RUSTFS_RPC_SECRET"; +const OIDC_EXTRA_CA_ENV: &str = "RUSTFS_EXTRA_CA_CERT"; +const OIDC_EXTRA_CA_VOLUME: &str = "rustfs-oidc-extra-ca"; +const OIDC_EXTRA_CA_MOUNT_PATH: &str = "/var/run/rustfs/oidc-extra-ca"; +const OIDC_EXTRA_CA_FILE: &str = "ca.pem"; const VOLUME_CLAIM_TEMPLATE_PREFIX: &str = "vol"; const DEFAULT_RUN_AS_USER: i64 = 10001; const DEFAULT_RUN_AS_GROUP: i64 = 10001; @@ -720,7 +726,149 @@ fn stateful_name(tenant: &Tenant, pool: &Pool) -> String { format!("{}-{}", tenant.name(), pool.name) } +fn normalize_absolute_mount_path(path: &str) -> Option { + let path = Path::new(path); + if !path.is_absolute() { + return None; + } + + let mut normalized = PathBuf::from("/"); + for component in path.components() { + match component { + Component::RootDir | Component::CurDir => {} + Component::ParentDir => { + normalized.pop(); + } + Component::Normal(part) => normalized.push(part), + Component::Prefix(_) => return None, + } + } + Some(normalized) +} + +fn mount_paths_overlap(left: &Path, right: &Path) -> bool { + left.starts_with(right) || right.starts_with(left) +} + impl Tenant { + pub(crate) fn validate_additional_volumes(&self) -> Result<(), types::error::Error> { + let invalid = |message| types::error::Error::InvalidAdditionalVolumeSpec { + name: self.name(), + message, + }; + + let mut volume_names = BTreeSet::new(); + for volume in &self.spec.additional_volumes { + if volume.name.trim().is_empty() { + return Err(invalid( + "spec.additionalVolumes[].name must not be blank".to_string(), + )); + } + if !volume_names.insert(volume.name.as_str()) { + return Err(invalid(format!( + "spec.additionalVolumes contains duplicate volume name '{}'", + volume.name + ))); + } + } + + let mut mount_paths = BTreeSet::new(); + for mount in &self.spec.additional_volume_mounts { + let Some(normalized_mount_path) = normalize_absolute_mount_path(&mount.mount_path) + else { + return Err(invalid(format!( + "spec.additionalVolumeMounts mountPath '{}' must be absolute", + mount.mount_path + ))); + }; + if Path::new(&mount.mount_path) + .components() + .any(|component| component == Component::ParentDir) + { + return Err(invalid(format!( + "spec.additionalVolumeMounts mountPath '{}' must not contain '..' path components", + mount.mount_path + ))); + } + if !mount_paths.insert(normalized_mount_path) { + return Err(invalid(format!( + "spec.additionalVolumeMounts contains duplicate or equivalent mountPath '{}'", + mount.mount_path + ))); + } + if !volume_names.contains(mount.name.as_str()) { + return Err(invalid(format!( + "spec.additionalVolumeMounts mountPath '{}' references volume '{}', which is not declared in spec.additionalVolumes", + mount.mount_path, mount.name + ))); + } + } + + let mut managed_volume_names = BTreeSet::new(); + let mut managed_mount_paths = BTreeSet::new(); + for pool in &self.spec.pools { + for shard in 0..pool.persistence.volumes_per_server { + managed_volume_names.insert(volume_claim_template_name(shard)); + let mount_path = data_volume_mount_path(pool.persistence.path.as_deref(), shard); + if let Some(mount_path) = normalize_absolute_mount_path(&mount_path) { + managed_mount_paths.insert(mount_path); + } + } + } + + if let Some(logging) = &self.spec.logging + && logging.mode != crate::types::v1alpha1::logging::LoggingMode::Stdout + { + managed_volume_names.insert("logs".to_string()); + let mount_path = logging.mount_path.as_deref().unwrap_or("/logs"); + if let Some(mount_path) = normalize_absolute_mount_path(mount_path) { + managed_mount_paths.insert(mount_path); + } + } + + if self + .spec + .oidc + .as_ref() + .is_some_and(|oidc| oidc.extra_ca_cert_secret_ref.is_some()) + { + managed_volume_names.insert(OIDC_EXTRA_CA_VOLUME.to_string()); + managed_mount_paths.insert(PathBuf::from(OIDC_EXTRA_CA_MOUNT_PATH)); + } + + if let Some(tls) = &self.spec.tls + && tls.is_enabled() + { + managed_volume_names.insert(TLS_SERVER_VOLUME_NAME.to_string()); + if let Some(mount_path) = normalize_absolute_mount_path(&tls.mount_path) { + managed_mount_paths.insert(mount_path); + } + } + + for volume_name in volume_names { + if managed_volume_names.contains(volume_name) { + return Err(invalid(format!( + "spec.additionalVolumes volume name '{}' conflicts with an operator-managed volume", + volume_name + ))); + } + } + for mount_path in mount_paths { + if let Some(managed_mount_path) = managed_mount_paths + .iter() + .find(|managed_mount_path| mount_paths_overlap(&mount_path, managed_mount_path)) + { + return Err(invalid(format!( + "spec.additionalVolumeMounts mountPath '{}' overlaps operator-managed mount '{}'", + mount_path.display(), + managed_mount_path.display() + ))); + } + } + + Ok(()) + } + fn validate_declared_workload_security_contexts(&self) -> Result<(), types::error::Error> { let invalid_profile = |message| types::error::Error::InvalidWorkloadSecurityProfile { name: self.name(), @@ -917,7 +1065,7 @@ impl Tenant { return Err(types::error::Error::WorkloadSecurityIncompatible { name: self.name(), message: format!( - "image '{image}' uses a {} and its seccomp compatibility cannot be verified for RuntimeDefault in pool '{}'; pin a verified RustFS 1.0.0-beta.9 or later release tag (for example 1.0.0-beta.10), or verify the image and set metadata.annotations['{RUNTIME_DEFAULT_IMAGE_ACK_ANNOTATION}'] to the exact current resolved image reference; mutable references can change without changing the annotation, so a digest-qualified reference is strongly recommended", + "image '{image}' uses a {} and its seccomp compatibility cannot be verified for RuntimeDefault in pool '{}'; pin a verified RustFS 1.0.0 or later release tag, or verify the image and set metadata.annotations['{RUNTIME_DEFAULT_IMAGE_ACK_ANNOTATION}'] to the exact current resolved image reference; mutable references can change without changing the annotation, so a digest-qualified reference is strongly recommended", reason.description(), pool.name ), @@ -1267,6 +1415,51 @@ impl Tenant { (env, volumes, mounts) } + fn configure_oidc_extra_ca( + &self, + ) -> ( + Vec, + Vec, + Vec, + ) { + let Some(secret_ref) = self + .spec + .oidc + .as_ref() + .and_then(|oidc| oidc.extra_ca_cert_secret_ref.as_ref()) + else { + return (Vec::new(), Vec::new(), Vec::new()); + }; + + let env = vec![corev1::EnvVar { + name: OIDC_EXTRA_CA_ENV.to_string(), + value: Some(format!("{OIDC_EXTRA_CA_MOUNT_PATH}/{OIDC_EXTRA_CA_FILE}")), + ..Default::default() + }]; + let volumes = vec![corev1::Volume { + name: OIDC_EXTRA_CA_VOLUME.to_string(), + secret: Some(corev1::SecretVolumeSource { + secret_name: Some(secret_ref.name.clone()), + items: Some(vec![corev1::KeyToPath { + key: secret_ref.key.clone(), + path: OIDC_EXTRA_CA_FILE.to_string(), + ..Default::default() + }]), + optional: Some(false), + ..Default::default() + }), + ..Default::default() + }]; + let mounts = vec![corev1::VolumeMount { + name: OIDC_EXTRA_CA_VOLUME.to_string(), + mount_path: OIDC_EXTRA_CA_MOUNT_PATH.to_string(), + read_only: Some(true), + ..Default::default() + }]; + + (env, volumes, mounts) + } + pub fn new_statefulset(&self, pool: &Pool) -> Result { self.new_statefulset_with_tls_plan(pool, &TlsPlan::disabled()) } @@ -1290,6 +1483,7 @@ impl Tenant { cluster_domain: &str, ) -> Result { self.validate_declared_workload_security_contexts()?; + self.validate_additional_volumes()?; let labels = self.pool_labels(pool); let selector_labels = self.pool_selector_labels(pool); @@ -1321,6 +1515,8 @@ impl Tenant { ..Default::default() }); env_vars.extend(tls_plan.env.clone()); + let (oidc_env, mut oidc_volumes, mut oidc_mounts) = self.configure_oidc_extra_ca(); + env_vars.extend(oidc_env); // Add required RustFS environment variables env_vars.push(corev1::EnvVar { @@ -1409,6 +1605,15 @@ impl Tenant { if self.spec.rpc_secret.is_some() && user_env.name == RPC_SECRET_ENV { continue; } + if self + .spec + .oidc + .as_ref() + .is_some_and(|oidc| oidc.extra_ca_cert_secret_ref.is_some()) + && user_env.name == OIDC_EXTRA_CA_ENV + { + continue; + } // Remove any existing var with the same name to allow non-reserved overrides. env_vars.retain(|e| e.name != user_env.name); env_vars.push(user_env.clone()); @@ -1426,9 +1631,23 @@ impl Tenant { env_vars.extend(kms_env); pod_volumes.append(&mut kms_volumes); volume_mounts.append(&mut kms_mounts); + pod_volumes.append(&mut oidc_volumes); + volume_mounts.append(&mut oidc_mounts); pod_volumes.extend(tls_plan.volumes.clone()); volume_mounts.extend(tls_plan.volume_mounts.clone()); + let mut additional_volumes = self.spec.additional_volumes.clone(); + additional_volumes.sort_by(|left, right| left.name.cmp(&right.name)); + pod_volumes.extend(additional_volumes); + + let mut additional_volume_mounts = self.spec.additional_volume_mounts.clone(); + additional_volume_mounts.sort_by(|left, right| { + left.mount_path + .cmp(&right.mount_path) + .then_with(|| left.name.cmp(&right.name)) + }); + volume_mounts.extend(additional_volume_mounts); + let security = effective_workload_security_context( self.spec.security_context.as_ref(), pool.security_context.as_ref(), @@ -2088,7 +2307,9 @@ mod tests { }; use crate::types::v1alpha1::logging::{LoggingConfig, LoggingMode}; use crate::types::v1alpha1::security_context::{MAX_KUBERNETES_ID, PodSecurityContextOverride}; - use crate::types::v1alpha1::tenant::{RpcSecretRef, Tenant}; + use crate::types::v1alpha1::tenant::{ + OidcConfig, OidcExtraCaCertSecretRef, RpcSecretRef, Tenant, + }; use crate::types::v1alpha1::tls::{SecretKeyReference, TlsPlan}; use k8s_openapi::api::apps::v1; use k8s_openapi::api::core::v1 as corev1; @@ -2548,6 +2769,442 @@ mod tests { ); } + #[test] + fn oidc_extra_ca_mounts_selected_secret_key_without_sub_path() { + let mut tenant = crate::tests::create_test_tenant(None, None); + tenant.spec.oidc = Some(OidcConfig { + extra_ca_cert_secret_ref: Some(OidcExtraCaCertSecretRef { + name: "authentik-ca".to_string(), + key: "bundle.pem".to_string(), + }), + }); + tenant.spec.env.push(corev1::EnvVar { + name: "RUSTFS_EXTRA_CA_CERT".to_string(), + value: Some("/wrong/override.pem".to_string()), + ..Default::default() + }); + let pool = &tenant.spec.pools[0]; + + let statefulset = tenant + .new_statefulset(pool) + .expect("Should create StatefulSet with OIDC extra CA trust"); + let pod_spec = statefulset.spec.unwrap().template.spec.unwrap(); + let container = &pod_spec.containers[0]; + + assert_eq!( + env_value(container, "RUSTFS_EXTRA_CA_CERT"), + Some("/var/run/rustfs/oidc-extra-ca/ca.pem") + ); + assert_eq!( + container + .env + .as_ref() + .unwrap() + .iter() + .filter(|var| var.name == "RUSTFS_EXTRA_CA_CERT") + .count(), + 1 + ); + + let mount = container + .volume_mounts + .as_ref() + .unwrap() + .iter() + .find(|mount| mount.name == "rustfs-oidc-extra-ca") + .expect("OIDC extra CA volume mount should exist"); + assert_eq!(mount.mount_path, "/var/run/rustfs/oidc-extra-ca"); + assert_eq!(mount.read_only, Some(true)); + assert_eq!( + mount.sub_path, None, + "Secret rotation requires a directory mount" + ); + + let secret = pod_spec + .volumes + .as_ref() + .unwrap() + .iter() + .find(|volume| volume.name == "rustfs-oidc-extra-ca") + .and_then(|volume| volume.secret.as_ref()) + .expect("OIDC extra CA Secret volume should exist"); + assert_eq!(secret.secret_name.as_deref(), Some("authentik-ca")); + assert_eq!(secret.optional, Some(false)); + assert_eq!( + secret.items.as_deref(), + Some( + [corev1::KeyToPath { + key: "bundle.pem".to_string(), + path: "ca.pem".to_string(), + ..Default::default() + }] + .as_slice() + ) + ); + } + + #[test] + fn raw_oidc_extra_ca_env_remains_supported_without_managed_secret_ref() { + let mut tenant = crate::tests::create_test_tenant(None, None); + tenant.spec.env.push(corev1::EnvVar { + name: "RUSTFS_EXTRA_CA_CERT".to_string(), + value: Some("/custom/ca.pem".to_string()), + ..Default::default() + }); + let pool = &tenant.spec.pools[0]; + + let statefulset = tenant + .new_statefulset(pool) + .expect("Should preserve an unmanaged OIDC extra CA environment variable"); + let pod_spec = statefulset.spec.unwrap().template.spec.unwrap(); + let container = &pod_spec.containers[0]; + + assert_eq!( + env_value(container, "RUSTFS_EXTRA_CA_CERT"), + Some("/custom/ca.pem") + ); + assert!(pod_spec.volumes.as_ref().is_none_or(|volumes| { + volumes + .iter() + .all(|volume| volume.name != "rustfs-oidc-extra-ca") + })); + } + + #[test] + fn additional_secret_volume_supports_rustfs_extra_ca_cert() { + let mut tenant = crate::tests::create_test_tenant(None, None); + tenant.spec.env.push(corev1::EnvVar { + name: "RUSTFS_EXTRA_CA_CERT".to_string(), + value: Some("/etc/rustfs/custom-ca/ca.crt".to_string()), + ..Default::default() + }); + tenant.spec.additional_volumes.push(corev1::Volume { + name: "custom-ca".to_string(), + secret: Some(corev1::SecretVolumeSource { + secret_name: Some("custom-ca".to_string()), + optional: Some(false), + ..Default::default() + }), + ..Default::default() + }); + tenant + .spec + .additional_volume_mounts + .push(corev1::VolumeMount { + name: "custom-ca".to_string(), + mount_path: "/etc/rustfs/custom-ca".to_string(), + read_only: Some(true), + ..Default::default() + }); + + let statefulset = tenant + .new_statefulset(&tenant.spec.pools[0]) + .expect("additional CA volume should render"); + let pod_spec = statefulset.spec.unwrap().template.spec.unwrap(); + let container = &pod_spec.containers[0]; + + assert_eq!( + env_value(container, "RUSTFS_EXTRA_CA_CERT"), + Some("/etc/rustfs/custom-ca/ca.crt") + ); + assert_eq!( + pod_spec + .volumes + .as_ref() + .unwrap() + .iter() + .find(|volume| volume.name == "custom-ca") + .and_then(|volume| volume.secret.as_ref()) + .and_then(|secret| secret.secret_name.as_deref()), + Some("custom-ca") + ); + assert_eq!( + container + .volume_mounts + .as_ref() + .unwrap() + .iter() + .find(|mount| mount.name == "custom-ca") + .map(|mount| (mount.mount_path.as_str(), mount.read_only, &mount.sub_path)), + Some(("/etc/rustfs/custom-ca", Some(true), &None)) + ); + } + + #[test] + fn omitted_additional_volumes_do_not_change_tenant_json() { + let tenant = crate::tests::create_test_tenant(None, None); + let spec = serde_json::to_value(&tenant.spec).expect("Tenant spec should serialize"); + + assert!(spec.get("additionalVolumes").is_none()); + assert!(spec.get("additionalVolumeMounts").is_none()); + } + + #[test] + fn additional_volume_validation_rejects_unknown_volume_and_managed_collisions() { + let mut tenant = crate::tests::create_test_tenant(None, None); + tenant + .spec + .additional_volume_mounts + .push(corev1::VolumeMount { + name: "missing".to_string(), + mount_path: "/etc/rustfs/custom".to_string(), + ..Default::default() + }); + let error = tenant + .validate_additional_volumes() + .expect_err("unknown volume reference should be rejected"); + assert!(matches!( + error, + crate::types::error::Error::InvalidAdditionalVolumeSpec { message, .. } + if message.contains("not declared in spec.additionalVolumes") + )); + + tenant.spec.additional_volume_mounts.clear(); + tenant.spec.additional_volumes.push(corev1::Volume { + name: "vol-0".to_string(), + empty_dir: Some(corev1::EmptyDirVolumeSource::default()), + ..Default::default() + }); + let error = tenant + .validate_additional_volumes() + .expect_err("data volume name collision should be rejected"); + assert!(matches!( + error, + crate::types::error::Error::InvalidAdditionalVolumeSpec { message, .. } + if message.contains("operator-managed volume") + )); + } + + #[test] + fn additional_volume_validation_rejects_relative_and_managed_mount_paths() { + let mut tenant = crate::tests::create_test_tenant(None, None); + tenant.spec.additional_volumes.push(corev1::Volume { + name: "custom".to_string(), + empty_dir: Some(corev1::EmptyDirVolumeSource::default()), + ..Default::default() + }); + tenant + .spec + .additional_volume_mounts + .push(corev1::VolumeMount { + name: "custom".to_string(), + mount_path: "relative/path".to_string(), + ..Default::default() + }); + let error = tenant + .validate_additional_volumes() + .expect_err("relative mount path should be rejected"); + assert!(matches!( + error, + crate::types::error::Error::InvalidAdditionalVolumeSpec { message, .. } + if message.contains("must be absolute") + )); + + tenant.spec.additional_volume_mounts[0].mount_path = "/data/rustfs0".to_string(); + let error = tenant + .validate_additional_volumes() + .expect_err("data mount path collision should be rejected"); + assert!(matches!( + error, + crate::types::error::Error::InvalidAdditionalVolumeSpec { message, .. } + if message.contains("operator-managed mount") + )); + } + + #[test] + fn additional_volume_validation_rejects_parent_and_equivalent_mount_paths() { + let mut tenant = crate::tests::create_test_tenant(None, None); + tenant.spec.additional_volumes = vec![ + corev1::Volume { + name: "first".to_string(), + empty_dir: Some(corev1::EmptyDirVolumeSource::default()), + ..Default::default() + }, + corev1::Volume { + name: "second".to_string(), + empty_dir: Some(corev1::EmptyDirVolumeSource::default()), + ..Default::default() + }, + ]; + tenant.spec.additional_volume_mounts = vec![corev1::VolumeMount { + name: "first".to_string(), + mount_path: "/etc/rustfs/../custom".to_string(), + ..Default::default() + }]; + + let error = tenant + .validate_additional_volumes() + .expect_err("parent path components should be rejected"); + assert!(matches!( + error, + crate::types::error::Error::InvalidAdditionalVolumeSpec { message, .. } + if message.contains("must not contain '..'") + )); + + tenant.spec.additional_volume_mounts = vec![ + corev1::VolumeMount { + name: "first".to_string(), + mount_path: "/etc/rustfs/custom".to_string(), + ..Default::default() + }, + corev1::VolumeMount { + name: "second".to_string(), + mount_path: "/etc//rustfs/./custom/".to_string(), + ..Default::default() + }, + ]; + + let error = tenant + .validate_additional_volumes() + .expect_err("equivalent mount paths should be rejected"); + assert!(matches!( + error, + crate::types::error::Error::InvalidAdditionalVolumeSpec { message, .. } + if message.contains("duplicate or equivalent mountPath") + )); + } + + #[test] + fn additional_volume_validation_rejects_managed_mount_ancestors_and_descendants() { + let mut tenant = crate::tests::create_test_tenant(None, None); + tenant.spec.oidc = Some(OidcConfig { + extra_ca_cert_secret_ref: Some(OidcExtraCaCertSecretRef { + name: "validated-ca".to_string(), + key: "ca.crt".to_string(), + }), + }); + tenant.spec.additional_volumes.push(corev1::Volume { + name: "custom".to_string(), + empty_dir: Some(corev1::EmptyDirVolumeSource::default()), + ..Default::default() + }); + tenant + .spec + .additional_volume_mounts + .push(corev1::VolumeMount { + name: "custom".to_string(), + mount_path: "/var/run/rustfs/oidc-extra-ca/ca.pem".to_string(), + sub_path: Some("ca.pem".to_string()), + ..Default::default() + }); + + let error = tenant + .validate_additional_volumes() + .expect_err("an OIDC CA file override should be rejected"); + assert!(matches!( + error, + crate::types::error::Error::InvalidAdditionalVolumeSpec { message, .. } + if message.contains("overlaps operator-managed mount '/var/run/rustfs/oidc-extra-ca'") + )); + + tenant.spec.additional_volume_mounts[0].mount_path = "/data".to_string(); + tenant.spec.additional_volume_mounts[0].sub_path = None; + let error = tenant + .validate_additional_volumes() + .expect_err("a data mount ancestor should be rejected"); + assert!(matches!( + error, + crate::types::error::Error::InvalidAdditionalVolumeSpec { message, .. } + if message.contains("overlaps operator-managed mount '/data/rustfs0'") + )); + + tenant.spec.additional_volume_mounts[0].mount_path = + "/var/run/rustfs/oidc-extra-ca-backup".to_string(); + tenant + .validate_additional_volumes() + .expect("a similarly named sibling path should remain valid"); + } + + #[test] + fn additional_volume_order_does_not_trigger_statefulset_update() { + let mut tenant = crate::tests::create_test_tenant(None, None); + tenant.spec.additional_volumes = vec![ + corev1::Volume { + name: "second".to_string(), + empty_dir: Some(corev1::EmptyDirVolumeSource::default()), + ..Default::default() + }, + corev1::Volume { + name: "first".to_string(), + empty_dir: Some(corev1::EmptyDirVolumeSource::default()), + ..Default::default() + }, + ]; + tenant.spec.additional_volume_mounts = vec![ + corev1::VolumeMount { + name: "second".to_string(), + mount_path: "/etc/rustfs/second".to_string(), + ..Default::default() + }, + corev1::VolumeMount { + name: "first".to_string(), + mount_path: "/etc/rustfs/first".to_string(), + ..Default::default() + }, + ]; + let pool = &tenant.spec.pools[0]; + let statefulset = tenant + .new_statefulset(pool) + .expect("additional volumes should render"); + + tenant.spec.additional_volumes.reverse(); + tenant.spec.additional_volume_mounts.reverse(); + + assert!( + !tenant + .statefulset_needs_update(&statefulset, pool) + .expect("reordered map lists should compare") + ); + } + + #[test] + fn additional_volume_and_mount_changes_trigger_statefulset_update() { + let mut tenant = crate::tests::create_test_tenant(None, None); + tenant.spec.additional_volumes.push(corev1::Volume { + name: "custom-ca".to_string(), + secret: Some(corev1::SecretVolumeSource { + secret_name: Some("custom-ca".to_string()), + ..Default::default() + }), + ..Default::default() + }); + tenant + .spec + .additional_volume_mounts + .push(corev1::VolumeMount { + name: "custom-ca".to_string(), + mount_path: "/etc/rustfs/custom-ca".to_string(), + ..Default::default() + }); + let pool = &tenant.spec.pools[0]; + let statefulset = tenant + .new_statefulset(pool) + .expect("additional volume should render"); + + tenant.spec.additional_volumes[0] + .secret + .as_mut() + .unwrap() + .secret_name = Some("rotated-ca".to_string()); + assert!( + tenant + .statefulset_needs_update(&statefulset, pool) + .expect("additional volume source should compare") + ); + + tenant.spec.additional_volumes[0] + .secret + .as_mut() + .unwrap() + .secret_name = Some("custom-ca".to_string()); + tenant.spec.additional_volume_mounts[0].mount_path = "/etc/rustfs/rotated-ca".to_string(); + assert!( + tenant + .statefulset_needs_update(&statefulset, pool) + .expect("additional volume mount should compare") + ); + } + #[test] fn cert_manager_tls_statefulset_maps_secret_to_rustfs_tls_files() { let tenant = crate::tests::create_test_tenant(None, None); @@ -3199,7 +3856,7 @@ mod tests { .expect("Pod template should have spec"); assert_eq!( pod_spec.containers[0].image.as_deref(), - Some("rustfs/rustfs:1.0.0-beta.10") + Some("rustfs/rustfs:1.0.0") ); let security_context = pod_spec @@ -5018,6 +5675,40 @@ mod tests { ); } + #[test] + fn test_statefulset_oidc_extra_ca_secret_change_detected() { + let mut tenant = crate::tests::create_test_tenant(None, None); + tenant.spec.oidc = Some(OidcConfig { + extra_ca_cert_secret_ref: Some(OidcExtraCaCertSecretRef { + name: "old-oidc-ca".to_string(), + key: "ca.crt".to_string(), + }), + }); + let pool = &tenant.spec.pools[0]; + let statefulset = tenant + .new_statefulset(pool) + .expect("Should create StatefulSet"); + + tenant + .spec + .oidc + .as_mut() + .unwrap() + .extra_ca_cert_secret_ref + .as_mut() + .unwrap() + .name = "new-oidc-ca".to_string(); + + let needs_update = tenant + .statefulset_needs_update(&statefulset, pool) + .expect("Should check update need"); + + assert!( + needs_update, + "StatefulSet should need update when the OIDC extra CA Secret reference changes" + ); + } + // Test: StatefulSet diff detection - resources change #[test] fn test_statefulset_resources_change_detected() { diff --git a/src/types/v1alpha1/tls.rs b/src/types/v1alpha1/tls.rs index 6b021b3..d16f01b 100644 --- a/src/types/v1alpha1/tls.rs +++ b/src/types/v1alpha1/tls.rs @@ -20,6 +20,7 @@ use serde::{Deserialize, Serialize}; use std::collections::{BTreeMap, BTreeSet}; pub const DEFAULT_TLS_MOUNT_PATH: &str = "/var/run/rustfs/tls"; +pub(crate) const TLS_SERVER_VOLUME_NAME: &str = "rustfs-tls-server"; pub const TLS_HASH_ANNOTATION: &str = "operator.rustfs.com/tls-hash"; pub const RUSTFS_TLS_CERT_FILE: &str = "rustfs_cert.pem"; pub const RUSTFS_TLS_KEY_FILE: &str = "rustfs_key.pem"; @@ -560,8 +561,8 @@ impl TlsPlan { probe_scheme: "HTTPS", pod_template_annotations: annotations, env, - volumes: vec![projected_volume("rustfs-tls-server", sources)], - volume_mounts: vec![directory_mount("rustfs-tls-server", &mount_path)], + volumes: vec![projected_volume(TLS_SERVER_VOLUME_NAME, sources)], + volume_mounts: vec![directory_mount(TLS_SERVER_VOLUME_NAME, &mount_path)], status, } }