Hi everyone,
As the team responsible for our K8s infrastructure, we've noticed over the past few days that the API server is rejecting a large number of requests because the queue for that workload type is filling up:
When we tried to find the source of the requests, we came across a massive number of GET requests targeting the PersistentVolumeClaims resource:
Further investigation revealed that the 'secret-operator-csi-node-driver' DaemonSet pods were literally spamming the API server. Given the size of the cluster, this affected a correspondingly large number of pods, causing the API server queues to become overwhelmed and resulting in 429 errors.
After we restarted the DaemonSet's pods, the issue stopped immediately (see drop in the screenshots). We'd still like to know if we can figure out why it happened. We've been using this component for a while now, and as far as I can tell, this is the first time something like this has happened.
We took a look at the associated pods and you can see here that the secret operator is trying to access resources that no longer exist. We also see repeated entries with “Retrying syncing claim” where the failure counts are in the four-digit range or higher, as well as provisioning failures. I’m attaching a sample log from a random pod - with 1,000 log lines, we can cover just 2 seconds. The log volume for a single pod was 1.5 million lines per hour.
Logs-2026-07-14 15_05_17.txt
At first glance, it looks to me as though old resources somehow built up and accumulated in the loop, eventually leading to this behavior. Since a restart also fixed the problem, it can't be anything persistent. Can any of you explain this based on the logs and the behavior described?
If any additional data is needed, I'd be happy to provide it.
Thank you very much in advance, and best regards.
Hi everyone,
As the team responsible for our K8s infrastructure, we've noticed over the past few days that the API server is rejecting a large number of requests because the queue for that workload type is filling up:
When we tried to find the source of the requests, we came across a massive number of GET requests targeting the PersistentVolumeClaims resource:
Further investigation revealed that the 'secret-operator-csi-node-driver' DaemonSet pods were literally spamming the API server. Given the size of the cluster, this affected a correspondingly large number of pods, causing the API server queues to become overwhelmed and resulting in 429 errors.
After we restarted the DaemonSet's pods, the issue stopped immediately (see drop in the screenshots). We'd still like to know if we can figure out why it happened. We've been using this component for a while now, and as far as I can tell, this is the first time something like this has happened.
We took a look at the associated pods and you can see here that the secret operator is trying to access resources that no longer exist. We also see repeated entries with “Retrying syncing claim” where the failure counts are in the four-digit range or higher, as well as provisioning failures. I’m attaching a sample log from a random pod - with 1,000 log lines, we can cover just 2 seconds. The log volume for a single pod was 1.5 million lines per hour.
Logs-2026-07-14 15_05_17.txt
At first glance, it looks to me as though old resources somehow built up and accumulated in the loop, eventually leading to this behavior. Since a restart also fixed the problem, it can't be anything persistent. Can any of you explain this based on the logs and the behavior described?
If any additional data is needed, I'd be happy to provide it.
Thank you very much in advance, and best regards.