feat: Support configuring Iceberg REST catalog

[sbernauer]

Part of #849

Needs stackabletech/operator-rs#1293

Description

This PR allows users to connect Trino to a Iceberg REST catalog via the CRD.
As it's a breaking change (moving to a complex enum), we introduce v1alpha2 for this.

⚠️ Same as with stackabletech/operator-rs#1287, we need figure out how to correctly handle roundtrip conversions. Currently we loose data when converting a REST catalog config from v1alph2 to v1alpha1.

CRD change

See the extra/crds.yaml change. It's a bit hard to spot the diff because of the addition of v1alpha2, so I added it here for now, might be outdated.

Details

diff --git a/extra/crds.yaml b/extra/crds.yaml
index 901063b..037c2c4 100644
--- a/extra/crds.yaml
+++ b/extra/crds.yaml
@@ -3544,7 +3544,7 @@ spec:
     singular: trinocatalog
   scope: Namespaced
   versions:
-  - name: v1alpha1
+  - name: v1alpha2
     schema:
       openAPIV3Schema:
         description: The TrinoCatalog resource can be used to define catalogs in Kubernetes objects.
@@ -4053,6 +4053,100 @@ spec:
                   iceberg:
                     description: An [Apache Iceberg](https://docs.stackable.tech/home/nightly/trino/usage-guide/catalogs/iceberg) connector.
                     properties:
+                      catalog:
+                        description: |-
+                          Connection to a metadata catalog, which will be used as a storage for metadata.
+
+                          We support the following backends:
+
+                          * REST catalog
+                          * Hive metastore
+                          * User provided
+
+                          Details can be found on the corresponding documentation
+                        oneOf:
+                        - required:
+                          - rest
+                        - required:
+                          - hiveMetastore
+                        - required:
+                          - userProvided
+                        properties:
+                          hiveMetastore:
+                            description: Use a Hive metastore to store metadata.
+                            properties:
+                              configMap:
+                                description: Name of the [discovery ConfigMap](https://docs.stackable.tech/home/nightly/concepts/service_discovery) providing information about the Hive metastore.
+                                maxLength: 253
+                                minLength: 1
+                                pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
+                                type: string
+                            required:
+                            - configMap
+                            type: object
+                          rest:
+                            description: (Recommended) use a REST catalog to store metadata.
+                            properties:
+                              security:
+                                default:
+                                  none: {}
+                                description: How to authenticate against the REST catalog.
+                                oneOf:
+                                - required:
+                                  - none
+                                - required:
+                                  - oAuth2
+                                properties:
+                                  none:
+                                    description: Don't authenticate against the REST catalog (chosen by default).
+                                    type: object
+                                  oAuth2:
+                                    description: |-
+                                      Use OAuth2 to authenticate against the REST catalog.
+
+                                      Note that we only support configuring a subset of Trino's properties, you might need to use
+                                      `configOverrides` to be able to set all [available properties](https://trino.io/docs/current/object-storage/metastores.html#iceberg-specific-metastores).
+                                    properties:
+                                      credential:
+                                        description: The credential to present to the REST catalog.
+                                        oneOf:
+                                        - required:
+                                          - tokenSecretName
+                                        - required:
+                                          - credentialSecretName
+                                        properties:
+                                          credentialSecretName:
+                                            description: The Secret needs to contain the `clientId` and `clientSecret` keys.
+                                            type: string
+                                          tokenSecretName:
+                                            description: |-
+                                              Authenticate using a bearer token.
+
+                                              The Secret needs to contain the `token` key.
+                                            type: string
+                                        type: object
+                                      serverUri:
+                                        description: The endpoint to retrieve access token from OAuth2 Server.
+                                        format: uri
+                                        type: string
+                                    required:
+                                    - credential
+                                    - serverUri
+                                    type: object
+                                type: object
+                              uri:
+                                description: URL of the rest catalog server.
+                                format: uri
+                                type: string
+                            required:
+                            - uri
+                            type: object
+                          userProvided:
+                            description: |-
+                              The operator doesn't configure any catalog, the user needs to do that,
+                              e.g. using `configOverrides`.
+                            type: object
+                        type: object
                       hdfs:
                         description: |-
                           Connection to an HDFS cluster.
@@ -4068,23 +4162,6 @@ spec:
                         required:
                         - configMap
                         type: object
-                      metastore:
-                        description: |-
-                          Optional connection to a Hive Metastore, which will be used as a storage for metadata.
-
-                          The connection is optional, as Iceberg also supports other catalogs, such as a REST catalog,
-                          which (currently) can only be added using configOverrides.
-                        nullable: true
-                        properties:
-                          configMap:
-                            description: Name of the [discovery ConfigMap](https://docs.stackable.tech/home/nightly/concepts/service_discovery) providing information about the Hive metastore.
-                            maxLength: 253
-                            minLength: 1
-                            pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
-                            type: string
-                        required:
-                        - configMap
-                        type: object
                       s3:
                         description: |-
                           Connection to an S3 store.
@@ -4237,6 +4314,8 @@ spec:
                           reference:
                             type: string
                         type: object
+                    required:
+                    - catalog
                     type: object
                   postgresql:
                     description: An [PostgreSQL](https://docs.stackable.tech/home/nightly/trino/usage-guide/catalogs/postgresql) connector.

Usage examples

Up to date examples are in the PR, but some examples:

- connector:
    iceberg:
      catalog:
        hiveMetastore:
          configMap: simple-hive
- connector:
    iceberg:
      catalog:
        rest:
          uri: https://my.rest.com/iceberg
- connector:
    iceberg:
      catalog:
        rest:
          uri: https://my.secure.rest
          security:
            oAuth2:
              serverUri: https://keycloak.default.svc.cluster.local:8443/realms/test/protocol/openid-connect/token
              credential:
                credentialSecretName: my-keycloak-credentials
- connector:
    iceberg:
      catalog:
        rest:
          uri: https://my.secure.rest
          security:
            oAuth2:
              serverUri: https://keycloak.default.svc.cluster.local:8443/realms/test/protocol/openid-connect/token
              credential:
                tokenSecretName: my-keycloak-token
- connector:
    iceberg:
      catalog:
        userProvided: {}

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

Author

• Changes are OpenShift compatible
• CRD changes approved
• CRD documentation for all fields, following the style guide.
• Helm chart can be installed and deployed operator works
• Integration tests passed (for non trivial changes)
• Changes need to be "offline" compatible
• Links to generated (nightly) docs added
• Release note snippet added

Reviewer

• Code contains useful comments
• Code contains useful logging statements
• (Integration-)Test cases added
• Documentation added or updated. Follows the style guide.
• Changelog updated
• Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

• Feature Tracker has been updated
• Proper release label has been added
• Links to generated (nightly) docs added
• Release note snippet added
• Add type/deprecation label & add to the deprecation schedule
• Add type/experimental label & add to the experimental features tracker