Skip to content

OCPNODE-3880: Add criocredentialproviderconfig event handler#5487

Merged
openshift-merge-bot[bot] merged 1 commit intoopenshift:mainfrom
QiWang19:addcriocpcontroller
Mar 17, 2026
Merged

OCPNODE-3880: Add criocredentialproviderconfig event handler#5487
openshift-merge-bot[bot] merged 1 commit intoopenshift:mainfrom
QiWang19:addcriocpcontroller

Conversation

@QiWang19
Copy link
Member

@QiWang19 QiWang19 commented Dec 11, 2025

- What I did
Implement criocredentialprovierconfig that is used by crio-credential-provider plugin to fetch private mirror image pull secrets from the secret object.

The handler creates 97-pool-generated-credentialproviderconfig to rollout configurations to file /etc/kubernetes/credential-providers/[platform]-credential-provider.yaml .

workflow: https://github.com/openshift/enhancements/blob/master/enhancements/api-review/criocredentialproviderconfig-for-namespace-scoped-mirror-authentication.md#workflow-description
- How to verify it

  1. create cluster CRIOCredentialProviderConfig resource, file updated with a new section name: crio-credential-provider
apiVersion: config.openshift.io/v1alpha1
kind: CRIOCredentialProviderConfig
metadata:
  name: cluster
spec:
  matchImages:
  - docker.io
  - 123456789.dkr.ecr.us-east-1.amazonaws.com
  - "*.azurecr.io"
  - gcr.io
  - "*.*.registry.io"
  - registry.io:8080/path
$ oc get mc
97-master-generated-credentialproviderconfig       f6dc328e4c370e377ea5c878aa4c0b8feeb1b181   3.5.0             2s
97-worker-generated-credentialproviderconfig       f6dc328e4c370e377ea5c878aa4c0b8feeb1b181   3.5.0             2s

$ oc describe criocredentialproviderconfig.config.openshift.io/cluster
Name:         cluster
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  config.openshift.io/v1alpha1
Kind:         CRIOCredentialProviderConfig
Metadata:
  Creation Timestamp:  2026-02-16T21:26:17Z
  Generation:          1
  Resource Version:    42066
  UID:                 05d173ad-7b8d-4a28-afb9-b587fe565e16
Spec:
  Match Images:
    docker.io
    123456789.dkr.ecr.us-east-1.amazonaws.com
    *.azurecr.io
    gcr.io
    *.*.registry.io
    registry.io:8080/path
Status:
  Conditions:
    Last Transition Time:  2026-02-16T21:26:17Z
    Message:               Success
    Observed Generation:   1
    Reason:                MachineConfigRenderingSucceeded
    Status:                True
    Type:                  MachineConfigRendered
Events:                    <none>

sh-5.1# cat /etc/kubernetes/credential-providers/ecr-credential-provider.yaml 
apiVersion: kubelet.config.k8s.io/v1
kind: CredentialProviderConfig
providers:
- apiVersion: credentialprovider.kubelet.k8s.io/v1
  defaultCacheDuration: 12h0m0s
  matchImages:
  - '*.dkr.ecr.*.amazonaws.com'
  - '*.dkr.ecr.*.amazonaws.com.cn'
  - '*.dkr.ecr-fips.*.amazonaws.com'
  - '*.dkr.ecr.us-iso-east-1.c2s.ic.gov'
  - '*.dkr.ecr.us-isob-east-1.sc2s.sgov.gov'
  name: ecr-credential-provider
- apiVersion: credentialprovider.kubelet.k8s.io/v1
  defaultCacheDuration: 1s
  matchImages:
  - docker.io
  - 123456789.dkr.ecr.us-east-1.amazonaws.com
  - '*.azurecr.io'
  - gcr.io
  - '*.*.registry.io'
  - registry.io:8080/path
  name: crio-credential-provider
  tokenAttributes:
    cacheType: Token
    requireServiceAccount: false
    serviceAccountTokenAudience: https://kubernetes.default.svc

  1. create namespace rolebing allow serviceaccount to get secrect in the namespace e.g. namespace:mynamespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: credential-provider-secret-reader
  namespace: mynamespace 
rules:
- apiGroups: [""]
  resources: ["secrets"] 
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: credential-provider-secret-reader-binding
  namespace: mynamespace 
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: credential-provider-secret-reader
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: system:serviceaccount:mynamespace:default
  1. Create ImageDigestMirrorSet/ImageTagMirrorSet mirror configurations
  2. Create a pod containers.image is from mirror source registry
  3. Check crio-credential-provider log journalctl _COMM=crio-credential on the scheduled node

- Description for the changelog

Summary by CodeRabbit

  • New Features

    • Manage CRIO credential providers via a new CRIOCredentialProviderConfig with platform-aware credential-provider config generation and kubelet credential-provider flags on master, worker, and arbiter nodes.
  • Chores / Defaults

    • Cluster defaults updated to add credential-provider file paths and to reload/restart kubelet when those configs change.
  • Security & Permissions

    • Added cluster-scoped RBAC roles/bindings to allow nodes to access service account token-related operations.
  • Tests

    • Added comprehensive unit and controller tests for CRIO credential provider create/update/merge and ignition generation.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Dec 11, 2025

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 11, 2025
@QiWang19
Copy link
Member Author

/test all

@QiWang19 QiWang19 force-pushed the addcriocpcontroller branch 3 times, most recently from 865fa47 to 8a072e4 Compare December 13, 2025 06:29
@QiWang19
Copy link
Member Author

/test all

@QiWang19 QiWang19 changed the title Addcriocpcontroller OCPNODE-3880: Add criocredentialprovierconfig pcontroller Dec 23, 2025
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Dec 23, 2025
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Dec 23, 2025

@QiWang19: This pull request references OCPNODE-3880 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

- What I did

- How to verify it

apiVersion: config.openshift.io/v1alpha1
kind: CRIOCredentialProviderConfig
metadata:
 name: cluster
spec:
 matchImages:
 - 123456789.dkr.ecr.us-east-1.amazonaws.com
 - "*.azurecr.io"
 - "*.*.registry.io"
 - registry.io:8080/path
sh-5.1# cat /etc/kubernetes/credential-providers/gcr-credential-provider.yaml 
apiVersion: kubelet.config.k8s.io/v1
kind: CredentialProviderConfig
providers:
- apiVersion: credentialprovider.kubelet.k8s.io/v1
 args:
 - get-credentials
 - --v=3
 defaultCacheDuration: 1m0s
 matchImages:
 - gcr.io
 - '*.gcr.io'
 - '*.pkg.dev'
 - container.cloud.google.com
 name: gcr-credential-provider
- apiVersion: credentialprovider.kubelet.k8s.io/v1
 defaultCacheDuration: 1s
 matchImages:
 - registry.io:8080/path
 - 123456789.dkr.ecr.us-east-1.amazonaws.com
 - '*.azurecr.io'
 - '*.*.registry.io'
 name: crio-credential-provider
 tokenAttributes:
   cacheType: Token
   requireServiceAccount: false
   serviceAccountTokenAudience: https://kubernetes.default.svc

- Description for the changelog

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 21, 2026
@QiWang19 QiWang19 force-pushed the addcriocpcontroller branch from 8a072e4 to 75dfbfd Compare January 26, 2026 15:35
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 26, 2026
@QiWang19
Copy link
Member Author

/test all

@QiWang19 QiWang19 force-pushed the addcriocpcontroller branch from 75dfbfd to 511fde6 Compare January 27, 2026 01:46
@QiWang19
Copy link
Member Author

/test all

@QiWang19 QiWang19 force-pushed the addcriocpcontroller branch from 511fde6 to 0ff0fe8 Compare January 27, 2026 18:42
@QiWang19
Copy link
Member Author

/test all

@QiWang19 QiWang19 force-pushed the addcriocpcontroller branch from 0ff0fe8 to 5d5008b Compare January 28, 2026 15:25
@QiWang19
Copy link
Member Author

/test all

@QiWang19 QiWang19 force-pushed the addcriocpcontroller branch from 5d5008b to 7d79743 Compare January 30, 2026 14:15
@QiWang19 QiWang19 force-pushed the addcriocpcontroller branch 2 times, most recently from 147fbf2 to 3027d6a Compare February 15, 2026 16:17
@QiWang19
Copy link
Member Author

/test all

@QiWang19 QiWang19 force-pushed the addcriocpcontroller branch from 3027d6a to 9a477a8 Compare February 16, 2026 19:53
@QiWang19
Copy link
Member Author

/test all

@QiWang19 QiWang19 force-pushed the addcriocpcontroller branch from 9a477a8 to 58fa6bc Compare February 16, 2026 20:59
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Feb 16, 2026

@QiWang19: This pull request references OCPNODE-3880 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

- What I did

- How to verify it

create cluster CRIOCredentialProviderConfig resource

apiVersion: config.openshift.io/v1alpha1
kind: CRIOCredentialProviderConfig
metadata:
 name: cluster
spec:
 matchImages:
 - docker.io
 - 123456789.dkr.ecr.us-east-1.amazonaws.com
 - "*.azurecr.io"
 - gcr.io
 - "*.*.registry.io"
 - registry.io:8080/path
$ oc describe criocredentialproviderconfig.config.openshift.io/cluster
Name:         cluster
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  config.openshift.io/v1alpha1
Kind:         CRIOCredentialProviderConfig
Metadata:
 Creation Timestamp:  2026-02-16T21:26:17Z
 Generation:          1
 Resource Version:    42066
 UID:                 05d173ad-7b8d-4a28-afb9-b587fe565e16
Spec:
 Match Images:
   docker.io
   123456789.dkr.ecr.us-east-1.amazonaws.com
   *.azurecr.io
   gcr.io
   *.*.registry.io
   registry.io:8080/path
Status:
 Conditions:
   Last Transition Time:  2026-02-16T21:26:17Z
   Message:               Success
   Observed Generation:   1
   Reason:                MachineConfigRenderingSucceeded
   Status:                True
   Type:                  MachineConfigRendered
Events:                    <none>

sh-5.1# cat /etc/kubernetes/credential-providers/ecr-credential-provider.yaml 
apiVersion: kubelet.config.k8s.io/v1
kind: CredentialProviderConfig
providers:
- apiVersion: credentialprovider.kubelet.k8s.io/v1
 defaultCacheDuration: 12h0m0s
 matchImages:
 - '*.dkr.ecr.*.amazonaws.com'
 - '*.dkr.ecr.*.amazonaws.com.cn'
 - '*.dkr.ecr-fips.*.amazonaws.com'
 - '*.dkr.ecr.us-iso-east-1.c2s.ic.gov'
 - '*.dkr.ecr.us-isob-east-1.sc2s.sgov.gov'
 name: ecr-credential-provider
- apiVersion: credentialprovider.kubelet.k8s.io/v1
 defaultCacheDuration: 1s
 matchImages:
 - docker.io
 - 123456789.dkr.ecr.us-east-1.amazonaws.com
 - '*.azurecr.io'
 - gcr.io
 - '*.*.registry.io'
 - registry.io:8080/path
 name: crio-credential-provider
 tokenAttributes:
   cacheType: Token
   requireServiceAccount: false
   serviceAccountTokenAudience: https://kubernetes.default.svc

- Description for the changelog

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Feb 16, 2026

@QiWang19: This pull request references OCPNODE-3880 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

- What I did

- How to verify it

  1. create cluster CRIOCredentialProviderConfig resource
apiVersion: config.openshift.io/v1alpha1
kind: CRIOCredentialProviderConfig
metadata:
 name: cluster
spec:
 matchImages:
 - docker.io
 - 123456789.dkr.ecr.us-east-1.amazonaws.com
 - "*.azurecr.io"
 - gcr.io
 - "*.*.registry.io"
 - registry.io:8080/path
$ oc describe criocredentialproviderconfig.config.openshift.io/cluster
Name:         cluster
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  config.openshift.io/v1alpha1
Kind:         CRIOCredentialProviderConfig
Metadata:
 Creation Timestamp:  2026-02-16T21:26:17Z
 Generation:          1
 Resource Version:    42066
 UID:                 05d173ad-7b8d-4a28-afb9-b587fe565e16
Spec:
 Match Images:
   docker.io
   123456789.dkr.ecr.us-east-1.amazonaws.com
   *.azurecr.io
   gcr.io
   *.*.registry.io
   registry.io:8080/path
Status:
 Conditions:
   Last Transition Time:  2026-02-16T21:26:17Z
   Message:               Success
   Observed Generation:   1
   Reason:                MachineConfigRenderingSucceeded
   Status:                True
   Type:                  MachineConfigRendered
Events:                    <none>

sh-5.1# cat /etc/kubernetes/credential-providers/ecr-credential-provider.yaml 
apiVersion: kubelet.config.k8s.io/v1
kind: CredentialProviderConfig
providers:
- apiVersion: credentialprovider.kubelet.k8s.io/v1
 defaultCacheDuration: 12h0m0s
 matchImages:
 - '*.dkr.ecr.*.amazonaws.com'
 - '*.dkr.ecr.*.amazonaws.com.cn'
 - '*.dkr.ecr-fips.*.amazonaws.com'
 - '*.dkr.ecr.us-iso-east-1.c2s.ic.gov'
 - '*.dkr.ecr.us-isob-east-1.sc2s.sgov.gov'
 name: ecr-credential-provider
- apiVersion: credentialprovider.kubelet.k8s.io/v1
 defaultCacheDuration: 1s
 matchImages:
 - docker.io
 - 123456789.dkr.ecr.us-east-1.amazonaws.com
 - '*.azurecr.io'
 - gcr.io
 - '*.*.registry.io'
 - registry.io:8080/path
 name: crio-credential-provider
 tokenAttributes:
   cacheType: Token
   requireServiceAccount: false
   serviceAccountTokenAudience: https://kubernetes.default.svc

  1. create namespace rolebing e.g. namespace:mynamespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 name: credential-provider-secret-reader
 namespace: mynamespace 
rules:
- apiGroups: [""]
 resources: ["secrets"] 
 verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
 name: credential-provider-secret-reader-binding
 namespace: mynamespace 
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: Role
 name: credential-provider-secret-reader
subjects:
 - apiGroup: rbac.authorization.k8s.io
   kind: User
   name: system:serviceaccount:mynamespace:default
  1. Create ImageDigestMirrorSet/ImageTagMirrorSet mirror configurations
  2. Create a pod containers.image is from source
  3. Check crio-credential-provider log journalctl _COMM=crio-credential on the scheduled node

- Description for the changelog

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@QiWang19
Copy link
Member Author

/test all

@haircommander
Copy link
Member

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Mar 11, 2026
@QiWang19
Copy link
Member Author

/retest-required

ctrl.syncCRIOCredentialProviderConfigStatusOnly(err, apicfgv1alpha1.ConditionTypeMachineConfigRendered, apicfgv1alpha1.ReasonMachineConfigRenderingFailed, "could not generate CRIOCredentialProvider Ignition config: %v", err)
return err
}
if len(overlappedEntries) > 0 {
Copy link
Member

@saschagrunert saschagrunert Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When overlaps exist, this sets ConditionTypeValidated to False with ReasonConfigurationPartiallyApplied. If the user later removes the conflicting entries from the CR, no code path resets this condition back to True. The stale warning persists indefinitely.

Do we need to add an else branch (or an unconditional call after the loop) that sets ConditionTypeValidated to True when len(overlappedEntries) == 0?

builtInLabelKey = "machineconfiguration.openshift.io/mco-built-in"
configMapName = "crio-default-container-runtime"
forceSyncOnUpgrade = "force-sync-on-upgrade"
genericCredProviderConfigPath = "/etc/kubernetes/credential-providers/generic-credential-provider.yaml"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: kubeletCrioImageCredProviderConfPath duplicates constants.KubeletCrioImageCredProviderConfPath from pkg/daemon/constants/constants.go. Consider using the constants package instead of redefining the value here.

@QiWang19 QiWang19 force-pushed the addcriocpcontroller branch from c8f5763 to c953926 Compare March 12, 2026 18:38
@openshift-ci openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Mar 12, 2026
@QiWang19 QiWang19 force-pushed the addcriocpcontroller branch from c953926 to 18b41eb Compare March 12, 2026 20:38
Copy link
Member

@saschagrunert saschagrunert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@saschagrunert
Copy link
Member

/retest

@QiWang19
Copy link
Member Author

/verified by @QiWang19

@openshift-ci-robot
Copy link
Contributor

@QiWang19: This PR has been marked as verified by @QiWang19.

Details

In response to this:

/verified by @QiWang19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@QiWang19
Copy link
Member Author

/retest-required

@QiWang19
Copy link
Member Author

@isabella-janssen could you take a look?

@umohnani8
Copy link
Contributor

/approve

@QiWang19
Copy link
Member Author

/retest-required

@QiWang19
Copy link
Member Author

QiWang19 commented Mar 16, 2026

PR has a bootstrap e2e test failure that started this week. The test was passing with the previous [fix](https://github.com/QiWang19/machine-config-operator/blob/18b41eb1069358b9695019a7a9cd585ce690ea61/test/framework/envtest.go#L119-L121) but is now failing again

The failure is related to the bootstrap being run with techpreview enabled. When this feature gate is enabled, the container runtime config controller attempts to watch for TechpreviewNoUpgrade resource CRIOCredentialProviderConfig, but the test framework failed to install the CRD manifests.

@QiWang19
Copy link
Member Author

/retest-required

@QiWang19
Copy link
Member Author

rebaesd to resolve the conflict.

@QiWang19
Copy link
Member Author

/test bootstrap-unit

1 similar comment
@QiWang19
Copy link
Member Author

/test bootstrap-unit

Signed-off-by: Qi Wang <qiwan@redhat.com>
@QiWang19
Copy link
Member Author

/test bootstrap-unit

@QiWang19
Copy link
Member Author

/verified by @QiWang19

@openshift-ci-robot
Copy link
Contributor

@QiWang19: This PR has been marked as verified by @QiWang19.

Details

In response to this:

/verified by @QiWang19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@saschagrunert
Copy link
Member

/retest

@saschagrunert
Copy link
Member

/test e2e-openstack

@haircommander
Copy link
Member

/lgtm

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 17, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: haircommander, QiWang19, saschagrunert, umohnani8

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 17, 2026

@QiWang19: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-gcp-op-2of2 b4132ef link true /test e2e-gcp-op-2of2
ci/prow/e2e-openstack f4a26aa link false /test e2e-openstack

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

acknowledge-critical-fixes-only Indicates if the issuer of the label is OK with the policy. approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants