From 0fc90bf0dc05a86baefd516430c1b7a6b924c014 Mon Sep 17 00:00:00 2001 From: Komh Date: Tue, 10 Nov 2026 16:16:57 +0800 Subject: [PATCH 01/25] docs(solutions): add 9 and update 0 troubleshooting articles --- ...ations.app.k8s.io_already_exists_on_ACP.md | 97 ++++++++++++++++++ ...m_address_from_the_Katanomi_externalURL.md | 71 +++++++++++++ ...observice_exhausts_database_connections.md | 83 ++++++++++++++++ ...tion_when_ALB_runs_out_of_memory_on_ACP.md | 95 ++++++++++++++++++ ...r_does_not_match_template_labels_on_ACP.md | 87 ++++++++++++++++ ...after_deleting_the_Nexus_Services_first.md | 99 +++++++++++++++++++ ...holds_a_future-dated_certificate_on_ACP.md | 95 ++++++++++++++++++ ...rity_from_the_Connectors_CSI_kubeconfig.md | 82 +++++++++++++++ ...he_katanomi-setting-init_init_container.md | 88 +++++++++++++++++ 9 files changed, 797 insertions(+) create mode 100644 docs/en/solutions/Creating_a_custom_application_from_YAML_fails_with_applications.app.k8s.io_already_exists_on_ACP.md create mode 100644 docs/en/solutions/GitLab_Project_Hook_from_a_pipeline_trigger_uses_an_outdated_platform_address_from_the_Katanomi_externalURL.md create mode 100644 docs/en/solutions/Harbor_garbage_collection_fails_with_write_broken_pipe_to_PostgreSQL_when_jobservice_exhausts_database_connections.md create mode 100644 docs/en/solutions/Harbor_image_push_through_ALB_fails_with_use_of_closed_network_connection_when_ALB_runs_out_of_memory_on_ACP.md create mode 100644 docs/en/solutions/Native_application_update_fails_with_selector_does_not_match_template_labels_on_ACP.md create mode 100644 docs/en/solutions/Nexus_v4_migration_database_export_task_unreachable_after_deleting_the_Nexus_Services_first.md create mode 100644 docs/en/solutions/Pipeline_component_status_Unknown_because_tekton-operator-webhook-certs_holds_a_future-dated_certificate_on_ACP.md create mode 100644 docs/en/solutions/Pipeline_deploy_task_fails_with_x509_certificate_signed_by_unknown_authority_from_the_Connectors_CSI_kubeconfig.md create mode 100644 docs/en/solutions/Tekton_pipeline_Pod_rejected_by_Pod_Security_admission_because_of_the_katanomi-setting-init_init_container.md diff --git a/docs/en/solutions/Creating_a_custom_application_from_YAML_fails_with_applications.app.k8s.io_already_exists_on_ACP.md b/docs/en/solutions/Creating_a_custom_application_from_YAML_fails_with_applications.app.k8s.io_already_exists_on_ACP.md new file mode 100644 index 000000000..24093c371 --- /dev/null +++ b/docs/en/solutions/Creating_a_custom_application_from_YAML_fails_with_applications.app.k8s.io_already_exists_on_ACP.md @@ -0,0 +1,97 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Creating a custom application from YAML fails with applications.app.k8s.io already exists on ACP + +## Issue + +Creating a custom application from the YAML page of the console fails with `资源已存在` (resource already exists). Renaming the application and creating it again fails the same way, while applying the same resources with `kubectl` works. + +The error details show: + +```json +{ + "StatusCode": 409, + "reason": "AlreadyExists", + "message": "applications.app.k8s.io \"\" already exists", + "details": { "group": "app.k8s.io", "kind": "applications" } +} +``` + +An `Application` with that name does exist afterwards. It was created by the same click, and its YAML shows it never assembled: + +```yaml +spec: + assemblyPhase: Failed + componentKinds: + - group: app.k8s.io + kind: Application + - group: apps + kind: StatefulSet +status: + conditions: + - type: Error + reason: Create components failed + message: applications.app.k8s.io "" already exists + totalComponents: 0 +``` + +## Root Cause + +### Cause 1: a full `Application` object was pasted into the custom application YAML + +The YAML field of the custom application page expects the workload resources only, for example a StatefulSet. The console wraps that input in its own `app.k8s.io/v1beta1` `Application` object with the name entered on the page. + +When the pasted YAML is itself an `Application` (`apiVersion: app.k8s.io/v1beta1`, `kind: Application`), the request contains an extra Application layer. The console first creates its own `Application`, then tries to create the pasted `Application` as one of its components. That component has the same name as the Application just created, so the API server returns `409 AlreadyExists`. The outer Application ends in `assemblyPhase: Failed` with `Create components failed`, and no workload is created. + +Renaming does not help because the console Application and the pasted Application get the same name again. + +## Resolution + +### Cause 1 + +1. Remove the failed Application left behind by the earlier attempts, so the name can be used again. + + > **Warning:** Deleting an `Application` can also delete resources that belong to it. Do this only when the Application shows `assemblyPhase: Failed`, the condition `Create components failed`, and `totalComponents: 0`, which means no component was created. If the Application has components, do not delete it; check what it owns first. There is nothing to roll back for a failed Application with no components. + + ```bash + kubectl -n get application -o yaml + kubectl -n delete application + ``` + +2. Create the custom application again and paste only the workload YAML, for example the StatefulSet. Do not include an `apiVersion: app.k8s.io/v1beta1` / `kind: Application` object or its `componentKinds`, `assemblyPhase` or `status` fields. + + ```yaml + apiVersion: apps/v1 + kind: StatefulSet + metadata: + name: + spec: + ... + ``` + +3. Check that the new Application assembles and lists its workload: + + ```bash + kubectl -n get application -o jsonpath='{.spec.assemblyPhase}{"\n"}' + ``` + +## Diagnostic Steps + +1. Look at the YAML that was pasted into the page. If it starts with `apiVersion: app.k8s.io/v1beta1` and `kind: Application`, this is the cause. + +2. Find the Application that the failed attempt created. Use the full resource name: the short name `app` is not registered and returns `error: the server doesn't have a resource type "app"`. + + ```bash + kubectl get application --all-namespaces | grep + kubectl -n get application -o yaml + ``` + + Expected for this cause: `componentKinds` contains `group: app.k8s.io, kind: Application`, the condition reason is `Create components failed` with the message `applications.app.k8s.io "" already exists`, and `totalComponents: 0`. + +3. If the pasted YAML contains no `Application` object and the error still appears, rule out a duplicate submission: open the browser developer tools, select **Network**, enable **Preserve log**, and click create. Two POST requests with the same URL and body, the first returning `201` and the second `409 Conflict`, are a duplicate submission, which is a different problem from this cause. diff --git a/docs/en/solutions/GitLab_Project_Hook_from_a_pipeline_trigger_uses_an_outdated_platform_address_from_the_Katanomi_externalURL.md b/docs/en/solutions/GitLab_Project_Hook_from_a_pipeline_trigger_uses_an_outdated_platform_address_from_the_Katanomi_externalURL.md new file mode 100644 index 000000000..4858841f4 --- /dev/null +++ b/docs/en/solutions/GitLab_Project_Hook_from_a_pipeline_trigger_uses_an_outdated_platform_address_from_the_Katanomi_externalURL.md @@ -0,0 +1,71 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: [] +--- + +# GitLab Project Hook from a pipeline trigger uses an outdated platform address from the Katanomi externalURL + +## Issue + +A CI/CD pipeline is configured to build automatically when GitLab code changes. The webhook that the platform registers in the GitLab project (**Settings > Webhooks**, Project Hooks) points to an old address, for example `http:///...`, instead of the address through which the platform is actually reached, for example `https:///...`. The question is where this default address comes from and how to change it. + +## Root Cause + +### Cause 1: `spec.externalURL` of the `Katanomi` resource still holds the old address + +The address written into the GitLab Project Hook is the address that the katanomi component exposes outside the cluster. It is taken from `spec.externalURL` of the `Katanomi` resource on the global cluster (for example `katanomi-global` in the `cpaas-system` namespace). + +When the platform is reached through a new domain, a WAF or another front end, `spec.externalURL` keeps the address it was installed with. Every trigger created after that registers a hook with the old address. Hooks that already exist are not rewritten when the value changes; a trigger registers its hook when it is created. + +## Resolution + +### Cause 1 + +1. On the global cluster, read the current value: + + ```bash + kubectl get katanomis -A -o yaml | grep -n externalURL + ``` + +2. Check how many webhooks and triggers exist. Every one of them was registered with the old address, so the count shows how many pipelines must be recreated after the change: + + ```bash + kubectl get webhook -A + kubectl get trigger -A + ``` + +3. Set `spec.externalURL` to the address through which GitLab reaches the platform: + + > **Warning:** The new address is used for every hook registered afterwards, on all pipelines. GitLab must be able to reach that address, otherwise new triggers stop firing. Record the current value from step 1; roll back by setting it again. + + ```bash + kubectl -n cpaas-system edit katanomis + ``` + + Change: + + ```yaml + spec: + externalURL: https:// + ``` + +4. Delete the trigger of the affected pipeline and create it again. The recreated trigger registers the hook with the new address. + + > **Warning:** Between deleting and recreating the trigger, pushes to the GitLab project do not start the pipeline. Note the trigger settings before deleting it so it can be recreated the same way. + +5. In the GitLab project, open **Settings > Webhooks** and confirm that the hook URL now uses the new address. Remove any leftover hook that still points to the old address. + +## Diagnostic Steps + +1. Compare the hook URL in GitLab with `spec.externalURL`: + + ```bash + kubectl get katanomis -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}: {.spec.externalURL}{"\n"}{end}' + ``` + + Expected for this cause: the host of the GitLab Project Hook equals the host in `spec.externalURL`, and both differ from the address users open the platform with. + +2. After the change, create a test trigger, or recreate an existing one, and check that the new hook in GitLab uses the new address. A hook created before the change still showing the old address is expected until its trigger is recreated. diff --git a/docs/en/solutions/Harbor_garbage_collection_fails_with_write_broken_pipe_to_PostgreSQL_when_jobservice_exhausts_database_connections.md b/docs/en/solutions/Harbor_garbage_collection_fails_with_write_broken_pipe_to_PostgreSQL_when_jobservice_exhausts_database_connections.md new file mode 100644 index 000000000..fd306f7e3 --- /dev/null +++ b/docs/en/solutions/Harbor_garbage_collection_fails_with_write_broken_pipe_to_PostgreSQL_when_jobservice_exhausts_database_connections.md @@ -0,0 +1,83 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: [] +--- + +# Harbor garbage collection fails with write broken pipe to PostgreSQL when jobservice exhausts database connections + +## Issue + +Harbor garbage collection (GC) no longer frees storage. In the GC history every recent run, scheduled (`定时`) and manual (`手动`), is in the `错误` (Error) state. The GC job log stops in the mark phase with a `broken pipe` on the connection to the Harbor database on port 5432: + +```text +[INFO] [/jobservice/job/impl/gc/garbage_collection.go:144]: Garbage Collection parameters: [delete_untagged: false, dry_run: false, time_window: 2] +[INFO] [/jobservice/job/impl/gc/garbage_collection.go:154]: start to run gc in job. +[ERROR] [/jobservice/job/impl/gc/garbage_collection.go:181]: failed to get deleted Artifacts in gc job, with error: write tcp :->:5432: write: broken pipe +[ERROR] [/jobservice/job/impl/gc/garbage_collection.go:158]: failed to execute GC job at mark phase, error: write tcp :->:5432: write: broken pipe +``` + +The `harbor-jobservice` Pod log itself can look healthy, with only periodic `reap outdated job stats` lines, and the database Pod is `Running`. + +## Root Cause + +### Cause 1: jobs launched by jobservice hold all PostgreSQL connections + +Tasks launched by `harbor-jobservice` have piled up, failed and unfinished ones alike, and the connections they hold have used up the PostgreSQL connection limit. PostgreSQL then resets the connections of jobservice, so the GC job, which runs inside jobservice, loses its database connection while reading artifacts in the mark phase and fails with `write: broken pipe`. Every new GC run fails the same way as long as the connections are not released. + +Restarting `harbor-jobservice` closes the connections it holds and GC can run again. + +## Resolution + +### Cause 1 + +1. Find the jobservice Deployment in the Harbor namespace: + + ```bash + kubectl -n get deployment | grep jobservice + ``` + +2. Restart it to release its database connections. + + > **Warning:** Jobs running in jobservice at that moment (replication, scans, GC, retention) are interrupted and must be triggered again. Image push and pull are served by other Harbor components. Do this only when the GC log shows `write: broken pipe` to port 5432 and the database connections are exhausted (see Diagnostic Steps). A restart has nothing to roll back. + + ```bash + kubectl -n rollout restart deployment/ + kubectl -n rollout status deployment/ + ``` + +3. Run GC manually from **Administration > Clean Up** (**Garbage Collection**) in the Harbor portal and confirm that it finishes as `Success`. The scheduled runs then succeed again. + +## Diagnostic Steps + +1. List recent GC runs and read the log of a failed one through the Harbor API: + + ```bash + curl -sk -u 'admin:' 'https:///api/v2.0/system/gc?page=1&page_size=10' + curl -sk -u 'admin:' 'https:///api/v2.0/system/gc//log' + ``` + + Expected for this cause: `failed to execute GC job at mark phase` with `write: broken pipe` towards port 5432. A different error in the GC log is a different problem. + +2. Check the database Pod and its log: + + ```bash + kubectl get pod -A | grep -E 'harbor-database|harbor-pg' + kubectl -n logs --tail=300 + ``` + + Look for connection-limit errors such as `FATAL: sorry, too many clients already`, `remaining connection slots are reserved`, or `SQLSTATE 53300`. Messages such as `database system was not properly shut down; automatic recovery in progress` followed by `database system is ready to accept connections` show a past database restart; they explain an earlier interruption but not a GC failure that repeats on every run. + +3. Count the open connections against the limit: + + ```bash + kubectl -n exec -- psql -U postgres -c 'SHOW max_connections;' + kubectl -n exec -- psql -U postgres \ + -c "SELECT usename, client_addr, state, count(*) FROM pg_stat_activity GROUP BY 1,2,3 ORDER BY 4 DESC;" + ``` + + Expected for this cause: the total is at or near `max_connections`, and most connections come from the jobservice Pod IP. If the count is well below the limit, the broken pipe has another source, for example a database restart or a network interruption at the time of the GC run. + +4. After restarting jobservice, run step 3 again: the number of connections from jobservice drops, and a manual GC run completes. diff --git a/docs/en/solutions/Harbor_image_push_through_ALB_fails_with_use_of_closed_network_connection_when_ALB_runs_out_of_memory_on_ACP.md b/docs/en/solutions/Harbor_image_push_through_ALB_fails_with_use_of_closed_network_connection_when_ALB_runs_out_of_memory_on_ACP.md new file mode 100644 index 000000000..25599f9f7 --- /dev/null +++ b/docs/en/solutions/Harbor_image_push_through_ALB_fails_with_use_of_closed_network_connection_when_ALB_runs_out_of_memory_on_ACP.md @@ -0,0 +1,95 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Harbor image push through ALB fails with use of closed network connection when ALB runs out of memory on ACP + +## Issue + +Harbor is exposed through an ALB Layer 7 listener (for example after moving Harbor from another gateway to an ALB). Image pushes fail intermittently, mostly for large images or when several pushes run at the same time, and often succeed after a few retries. Later, image pulls can fail as well. + +The Docker client retries each layer and then gives up: + +```text +Retrying in 5 seconds +Retrying in 4 seconds +... +write tcp :->:: use of closed network connection +ERROR: Push image ://: failed! +``` + +No NodePort or hostPort conflict for the ALB port is found on the nodes. + +## Root Cause + +### Cause 1: the ALB buffers whole request bodies in memory and exceeds its memory limit + +By default, the ALB does not stream request bodies in Layer 7 forwarding. It buffers the complete request body in memory and only then forwards it to the backend. When Harbor is behind the ALB, every image layer upload is one request body, so the ALB holds whole layers in memory. + +The memory of the ALB `nginx` container is bounded by `.spec.config.resources.limits` of the `ALB2` resource, which is `256Mi` in the default ALB configuration. With large image layers and concurrent pushes, the buffered data exceeds that limit, and the process handling the upload is killed for running out of memory. The connection to the client is cut in the middle of the upload, the client reports `use of closed network connection`, and the push fails. Pulls through the same ALB can fail as well; both recover once the memory limit is raised. + +## Resolution + +### Cause 1 + +Raise the memory limit of the ALB instance that serves Harbor. Raising it to `1.5Gi` restored Harbor push and pull for large images with concurrent pushes; size it to the largest layers and the push concurrency you expect. + +1. Read the current resource settings of the ALB: + + ```bash + kubectl -n cpaas-system get alb2 -o jsonpath='{.spec.config.resources}{"\n"}' + ``` + +2. Raise the memory limit of the `nginx` container (`.spec.config.resources.limits.memory`). In the web console: **Administrator > Network Management > Load Balancer**, click **⋮ > Update** on the ALB, change the resource configuration and click **Update**. Or with `kubectl`: + + > **Warning:** Changing the ALB resources rolls the ALB Pods. Traffic through this ALB, including all other services it exposes, can be briefly interrupted while the Pods restart, longer with a single replica. Make sure the nodes running the ALB have enough free memory for the new limit. Do this only when the memory exhaustion is confirmed (see Diagnostic Steps). Roll back by setting the limit back to the value recorded in step 1. + + ```bash + kubectl -n cpaas-system patch alb2 --type=merge \ + -p '{"spec":{"config":{"resources":{"limits":{"memory":"1536Mi"}}}}}' + ``` + + Keep `.spec.config.resources.requests.memory` less than or equal to the new limit. + +3. Wait for the ALB Pods to be recreated and Ready: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep + ``` + +4. Push the large image again, also with several pushes in parallel, and confirm that the push and later pulls succeed. + +## Diagnostic Steps + +1. Confirm that the failing pushes go through the ALB: the registry address and port in the error (`:`) are the ALB address and listener port of the Harbor rule. + +2. Check the memory limit of the ALB: + + ```bash + kubectl -n cpaas-system get alb2 -o jsonpath='{.spec.config.resources.limits}{"\n"}' + ``` + + A limit of `256Mi` (the default) with images whose layers are hundreds of MB points to this cause. + +3. Watch the ALB memory while reproducing the failure with a large image or with several concurrent pushes: + + ```bash + kubectl -n cpaas-system top pod | grep + ``` + + Expected for this cause: the memory of the ALB Pod climbs to the limit during the upload, and the push fails at that moment. + +4. Look for out-of-memory kills. A killed container shows `Reason: OOMKilled` in its last state; a killed worker process inside the container appears in the kernel log of the node running the ALB Pod: + + ```bash + kubectl -n cpaas-system describe pod | grep -A5 'Last State' + journalctl -k | grep -iE 'memory cgroup out of memory|oom-kill' + ``` + + An OOM kill that matches the time of a failed push confirms this cause. If the ALB memory stays well below the limit while pushes fail, look at the network path between client, ALB and Harbor instead. + +See also `ACP_ALB_pods_OOMKilled_with_high_restart_counts_remove_memory_limits_on_global_alb2.md` for OOM kills of the `global-alb2` instance. diff --git a/docs/en/solutions/Native_application_update_fails_with_selector_does_not_match_template_labels_on_ACP.md b/docs/en/solutions/Native_application_update_fails_with_selector_does_not_match_template_labels_on_ACP.md new file mode 100644 index 000000000..c36a968b3 --- /dev/null +++ b/docs/en/solutions/Native_application_update_fails_with_selector_does_not_match_template_labels_on_ACP.md @@ -0,0 +1,87 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Native application update fails with selector does not match template labels on ACP + +## Issue + +A native application shows `失败` (Failed) in the console after an update, although its Deployments show `运行中` (Running) with all replicas ready. The update never reaches the Deployment. + +The `archon` component logs on the global cluster show the update being rejected by the Kubernetes API server: + +```text +handler.go:66] Handle request error: Deployment.apps "" is invalid: spec.template.metadata.labels: Invalid value: {"app.cpaas.io/name":".", ...}: `selector` does not match template `labels` +``` + +The same log can also contain lines such as `Warning: unknown field "spec.componentTemplates[N].apiVersion"`. The line that names the rejected Deployment is the `selector` error above. + +## Root Cause + +### Cause 1: the Deployment selector pins `app.cpaas.io/name` to a value the platform overwrites + +When the platform updates a native application, it writes the standard label `app.cpaas.io/name=.` into the Pod template of each Deployment that belongs to the application. + +A Deployment that was created earlier with a selector containing a different value for the same key, for example `app.cpaas.io/name=`, conflicts with that label. After the platform overwrites the Pod template label, `spec.selector.matchLabels` no longer matches `spec.template.metadata.labels`. Kubernetes validation rejects any Deployment whose selector does not match its own template labels, so the update fails and the native application is marked as failed. + +The existing Pods keep running because the old ReplicaSet is untouched, which is why the Deployments still report Running. + +`spec.selector` of an `apps/v1` Deployment is immutable. The selector cannot be corrected in place: the Deployment has to be recreated with a selector that agrees with the platform label. + +A configuration error in another resource of the same application can also leave the native application in an abnormal state. The `archon` log names the resource and the reason in each case, so read it before changing anything. + +## Resolution + +### Cause 1 + +1. Fix the source that generates the Deployment selector, for example the `selectorLabels` helper of the Helm chart or the template used to render the Deployment. If the selector uses the `app.cpaas.io/name` label, set its value in the `.` format, which is the value the platform injects: + + ```yaml + selector: + matchLabels: + app.cpaas.io/name: . + ``` + +2. Save the current Deployment before removing it: + + ```bash + kubectl -n get deployment -o yaml > -backup.yaml + ``` + +3. Delete the existing Deployment and redeploy the application, so the Deployment is created again with the corrected selector. + + > **Warning:** Deleting the Deployment deletes its Pods. The workload is unavailable until the new Deployment's Pods are Ready. Do this only when the `archon` log shows `` `selector` does not match template `labels` `` for this Deployment and the business allows the workload to be rebuilt. Roll back by applying the saved `-backup.yaml` (remove `metadata.resourceVersion`, `metadata.uid` and `status` first). + + ```bash + kubectl -n delete deployment + ``` + + Then redeploy the native application from the console, or re-run the release that renders the corrected template. + +4. Confirm that the native application status changes from `失败` (Failed) to Running, and that `archon` no longer logs the error for this Deployment. + +## Diagnostic Steps + +1. Find the `archon` Pods on the global cluster and search their logs for the rejected update: + + ```bash + kubectl get pods -A | grep archon + kubectl -n logs --tail=500 | grep -E 'Handle request error|does not match' + ``` + + Expected for this cause: `Deployment.apps "" is invalid: spec.template.metadata.labels ... `selector` does not match template `labels``. If the error names another resource of the application, fix that resource instead. + +2. Compare the Deployment selector with the value the platform injects: + + ```bash + kubectl -n get deployment \ + -o jsonpath='{.spec.selector.matchLabels}{"\n"}{.spec.template.metadata.labels}{"\n"}' + ``` + + Expected for this cause: the selector contains `app.cpaas.io/name` with a value other than `.`, for example the bare Deployment name. + +3. Check the other Deployments of the same application. A Deployment whose selector does not use `app.cpaas.io/name`, or already uses `.`, updates normally, which separates this cause from a general application controller problem. diff --git a/docs/en/solutions/Nexus_v4_migration_database_export_task_unreachable_after_deleting_the_Nexus_Services_first.md b/docs/en/solutions/Nexus_v4_migration_database_export_task_unreachable_after_deleting_the_Nexus_Services_first.md new file mode 100644 index 000000000..0950c4ea8 --- /dev/null +++ b/docs/en/solutions/Nexus_v4_migration_database_export_task_unreachable_after_deleting_the_Nexus_Services_first.md @@ -0,0 +1,99 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: [] +--- + +# Nexus v4 migration: database export task unreachable after deleting the Nexus Services first + +## Issue + +While migrating a platform-deployed Nexus to v4, the migration steps are run in their numbered order: + +- 1.1 Service shutdown: uninstall the old-version operator, then delete the Nexus Services with `kubectl -n ${INSTANCE_NAMESPACE} delete service -l release=${INSTANCE_NAME}`. +- 1.2 Back up Blob storage with `kubectl cp` from the Nexus Pod. +- 1.3 Back up the database by creating an **Admin - Export databases for backup** task under **Admin Dashboard > System > Tasks** in the Nexus web UI. + +After step 1.1, step 1.3 cannot be performed: the Nexus web UI, where the export task is created, is no longer reachable. + +## Root Cause + +### Cause 1: the Services were deleted before the backups were taken + +Step 1.3 is done inside the Nexus web UI, and the UI is reached through the Nexus Services. Step 1.1 deletes those Services. Running 1.1 first removes access to the UI, so the database export task cannot be created or run. + +The correct execution order is: complete steps 1.2 and 1.3 first, while the Services still exist, and only then perform step 1.1. + +## Resolution + +### Cause 1 + +1. Prepare a local backup directory with enough free disk space for the blob store and the database export, and set the variables used by the steps: + + ```bash + mkdir -p nexus-backup + export INSTANCE_NAME= + export INSTANCE_NAMESPACE= + kubectl -n ${INSTANCE_NAMESPACE} get pod -l release=${INSTANCE_NAME} + export POD_NAME= + ``` + +2. Step 1.2, back up Blob storage: + + ```bash + mkdir -p nexus-backup/blobs + kubectl -n ${INSTANCE_NAMESPACE} cp ${POD_NAME}:/nexus-data/blobs nexus-backup/blobs + ``` + +3. Step 1.3, back up the database in the Nexus web UI: + 1. Go to **Admin Dashboard > System > Tasks** and create a task. + 2. Type: **Admin - Export databases for backup**. + 3. Task name: any name. + 4. Backup location: a directory path inside the Nexus data volume directory. + 5. Frequency: manual. + 6. Run the task and confirm that it completes. + +4. Step 1.1, shut down the service. Uninstall the old-version operator first; otherwise the deleted Services are recreated automatically. Then save and delete the Services: + + > **Warning:** After the Services are deleted, Nexus is no longer reachable by users, build tools or the web UI. Do this only after steps 1.2 and 1.3 have completed and the backups are confirmed. To restore access, re-apply the saved Service manifests (remove `metadata.resourceVersion`, `metadata.uid`, `spec.clusterIP` and `spec.clusterIPs` first). + + ```bash + kubectl -n ${INSTANCE_NAMESPACE} get service -l release=${INSTANCE_NAME} -o yaml > nexus-backup/services.yaml + kubectl -n ${INSTANCE_NAMESPACE} delete service -l release=${INSTANCE_NAME} + ``` + +5. Continue with the remaining migration steps. + +If the Services were already deleted before the database export, the web UI can still be reached directly on the Pod, without a Service, by port-forwarding to the Nexus HTTP port (8081 by default in Nexus): + +```bash +kubectl -n ${INSTANCE_NAMESPACE} port-forward pod/${POD_NAME} 8081:8081 +``` + +Open `http://localhost:8081` on the machine running the command, then perform step 1.3. + +## Diagnostic Steps + +1. Check whether the Nexus Services still exist: + + ```bash + kubectl -n ${INSTANCE_NAMESPACE} get service -l release=${INSTANCE_NAME} + ``` + + `No resources found` means step 1.1 has already been run; take the database export through the port-forward described above before going on. + +2. Check that the Nexus Pod is still running, which steps 1.2 and 1.3 need: + + ```bash + kubectl -n ${INSTANCE_NAMESPACE} get pod -l release=${INSTANCE_NAME} -o wide + ``` + +3. After step 1.3, check that the export files exist in the backup location set in the task: + + ```bash + kubectl -n ${INSTANCE_NAMESPACE} exec ${POD_NAME} -- ls -la + ``` + + An empty directory means the export task did not run or did not complete. diff --git a/docs/en/solutions/Pipeline_component_status_Unknown_because_tekton-operator-webhook-certs_holds_a_future-dated_certificate_on_ACP.md b/docs/en/solutions/Pipeline_component_status_Unknown_because_tekton-operator-webhook-certs_holds_a_future-dated_certificate_on_ACP.md new file mode 100644 index 000000000..13f4a4df9 --- /dev/null +++ b/docs/en/solutions/Pipeline_component_status_Unknown_because_tekton-operator-webhook-certs_holds_a_future-dated_certificate_on_ACP.md @@ -0,0 +1,95 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: [] +--- + +# Pipeline component status Unknown because tekton-operator-webhook-certs holds a future-dated certificate on ACP + +## Issue + +On a business cluster, the cluster component **Pipeline** (the Tekton operator) shows status Unknown, for example during a routine inspection of cluster components. + +Decoding the certificates stored in the Secret `tekton-operator-webhook-certs` in the `tekton-operator` namespace shows a validity period that starts in the future: + +```text +notBefore=Oct 29 19:00:14 2027 GMT +notAfter=Nov 5 19:00:14 2027 GMT +``` + +## Root Cause + +### Cause 1: the webhook certificate was issued while the cluster clock was wrong + +The Tekton operator keeps the serving certificate of its webhook in the Secret `tekton-operator-webhook-certs`. If the cluster clock was set ahead at some point when the certificate was generated, the certificate carries a `notBefore` date in the future. + +A TLS client rejects a certificate whose `notBefore` date has not been reached yet. Even after the clock is correct again, the stored certificate stays not yet valid until that date, and the Pipeline component status is Unknown. Replacing the Secret with an empty one and restarting the operator makes the operator generate a new certificate with the current time. + +## Resolution + +### Cause 1 + +1. Check that the clock on every node of the cluster is correct before regenerating the certificate. A certificate regenerated on a wrong clock has the same problem. + + ```bash + date -u + chronyc tracking + ``` + + Run these on each node. If the time is wrong, fix time synchronization first (see `Configuring_chrony_NTP_on_Cluster_Nodes.md`). + +2. Save the current Secret: + + ```bash + kubectl -n tekton-operator get secret tekton-operator-webhook-certs -o yaml > tekton-operator-webhook-certs-backup.yaml + ``` + +3. Replace the Secret with an empty Secret of the same name. + + > **Warning:** Until the operator writes a new certificate, the Tekton operator webhook cannot serve TLS, and changes to Tekton operator resources (such as `TektonConfig`) can fail admission. Do this only when the certificate in the Secret has a `notBefore` date in the future and the node clocks are correct. Roll back by applying `tekton-operator-webhook-certs-backup.yaml` (remove `metadata.resourceVersion` and `metadata.uid` first) and restarting the operator again. + + ```bash + kubectl -n tekton-operator delete secret tekton-operator-webhook-certs + kubectl -n tekton-operator create secret generic tekton-operator-webhook-certs + ``` + +4. Restart the Tekton operator so that it generates a new certificate into the empty Secret: + + > **Warning:** The operator Pods restart. Pipelines that are already running are not stopped, but operator reconciliation pauses until the new Pods are Ready. Roll back as in step 3. + + ```bash + kubectl -n tekton-operator get deployment + kubectl -n tekton-operator rollout restart deployment + kubectl -n tekton-operator rollout status deployment + ``` + +5. Check the certificate dates again (see Diagnostic Steps) and confirm that the Pipeline component status returns to normal. + +## Diagnostic Steps + +1. List the keys of the Secret and print the validity of each certificate: + + ```bash + kubectl -n tekton-operator get secret tekton-operator-webhook-certs -o jsonpath='{.data}' | jq -r 'keys[]' + kubectl -n tekton-operator get secret tekton-operator-webhook-certs \ + -o jsonpath='{.data.}' | base64 -d | openssl x509 -noout -dates + ``` + + Expected for this cause: `notBefore` is later than the current time (`date -u`). A key that holds a private key instead of a certificate fails with `unable to load certificate ... Expecting: TRUSTED CERTIFICATE`; that is expected for that key and is not the fault. + +2. Compare with the current time on the nodes: + + ```bash + date -u + ``` + + If `notBefore` is in the past and `notAfter` is in the future, the certificate is valid and the Unknown status has another cause. If `notAfter` is in the past, the certificate has expired, which is a different problem from a future-dated certificate. + +3. Check the operator Pods and their logs for TLS errors: + + ```bash + kubectl -n tekton-operator get pods + kubectl -n tekton-operator logs deployment/ --tail=200 | grep -iE 'x509|certificate|tls' + ``` diff --git a/docs/en/solutions/Pipeline_deploy_task_fails_with_x509_certificate_signed_by_unknown_authority_from_the_Connectors_CSI_kubeconfig.md b/docs/en/solutions/Pipeline_deploy_task_fails_with_x509_certificate_signed_by_unknown_authority_from_the_Connectors_CSI_kubeconfig.md new file mode 100644 index 000000000..57831bb83 --- /dev/null +++ b/docs/en/solutions/Pipeline_deploy_task_fails_with_x509_certificate_signed_by_unknown_authority_from_the_Connectors_CSI_kubeconfig.md @@ -0,0 +1,82 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: [] +--- + +# Pipeline deploy task fails with x509 certificate signed by unknown authority from the Connectors CSI kubeconfig + +## Issue + +A Tekton pipeline task that updates a workload with `kubectl apply` (for example a `workload-apply` task that renders a Deployment and applies it) fails in its `apply` step. The task uses the kubeconfig mounted into the workspace: + +```text +==> Using kubeconfig from workspace: /workspace/kubeconfig/kubeconfig +error: error validating "/shared/rendered.yaml": error validating data: failed to download openapi: Get "https:///kubernetes//openapi/v2?timeout=32s": tls: failed to verify certificate: x509: certificate signed by unknown authority; if you choose to ignore these errors, turn validation off with --validate=false +==> Apply failed, continue to finalize flow +``` + +The following steps then fail and the PipelineRun ends as failed. The same pipeline worked before, and the platform address itself is reachable with a trusted certificate from other clients. + +## Root Cause + +### Cause 1: the Connectors CSI component keeps a certificate it loaded at startup + +When a pipeline runs, its Pod mounts a CSI volume provided by Connectors (driver `connectors.alauda.io`). At mount time the Connectors CSI component uses a CA to sign a certificate and injects it into the Pod, together with the kubeconfig the task uses. + +In Connectors versions before 1.14.0, the CSI component loads this certificate material only once, when it starts. The certificate is valid for 60 days and is rotated upstream. A CSI component that has been running longer than that keeps injecting the stale certificate, so the kubeconfig in the pipeline Pod no longer matches and the TLS check fails with `x509: certificate signed by unknown authority`. + +Connectors 1.14.0 fixes this: the certificate is rotated automatically upstream, and the CSI component detects the change and reloads it. + +## Resolution + +### Cause 1 + +**Workaround:** restart all Connectors components so that the CSI component loads the current certificate. + +1. List the Connectors workloads: + + ```bash + kubectl get deployments,daemonsets -A | grep -i connectors + ``` + +2. Restart each of them: + + > **Warning:** While the Connectors Pods restart, pipeline Pods that start at that moment cannot mount their Connectors volumes and stay in the init phase or fail; Pods that are already running keep their mounts. Do this only when the error is `x509: certificate signed by unknown authority` from the workspace kubeconfig, and preferably when no pipelines are starting. A restart has nothing to roll back; re-run any PipelineRun that failed during the restart. + + ```bash + kubectl -n rollout restart deployment/ + kubectl -n rollout restart daemonset/ + kubectl -n rollout status deployment/ + kubectl -n rollout status daemonset/ + ``` + +3. Re-run the PipelineRun and confirm that the `apply` step succeeds. + +**Permanent fix:** upgrade Connectors to 1.14.0 or later, where the CSI component refreshes the certificate when it changes. Without the upgrade, the error comes back each time the certificate is rotated while the CSI component keeps running. + +Do not use `--validate=false` as a fix. It only skips the OpenAPI schema download; the apply itself talks to the same API endpoint with the same kubeconfig and fails TLS verification in the same way. + +## Diagnostic Steps + +1. Confirm that the failing step uses the mounted kubeconfig: the log shows `Using kubeconfig from workspace: /workspace/kubeconfig/kubeconfig` before the x509 error. + +2. Check the Connectors version: + + ```bash + kubectl get csv -A | grep -i connectors + ``` + + A version earlier than 1.14.0 is affected. + +3. Check how long the Connectors Pods have been running: + + ```bash + kubectl get pods -A -o wide | grep -i connectors + ``` + + Pods older than about 60 days (the certificate validity) make this cause likely. After the restart, the next pipeline run succeeding confirms it. + +4. To rule out a platform certificate problem, check the platform address from outside the pipeline, for example in a browser or with `curl -v https:///`. A trusted certificate there, together with the failure only inside pipeline Pods, points to the injected kubeconfig and not to the platform endpoint. diff --git a/docs/en/solutions/Tekton_pipeline_Pod_rejected_by_Pod_Security_admission_because_of_the_katanomi-setting-init_init_container.md b/docs/en/solutions/Tekton_pipeline_Pod_rejected_by_Pod_Security_admission_because_of_the_katanomi-setting-init_init_container.md new file mode 100644 index 000000000..7d82ac91a --- /dev/null +++ b/docs/en/solutions/Tekton_pipeline_Pod_rejected_by_Pod_Security_admission_because_of_the_katanomi-setting-init_init_container.md @@ -0,0 +1,88 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - 4.3.3 +--- + +# Tekton pipeline Pod rejected by Pod Security admission because of the katanomi-setting-init init container + +## Issue + +A Tekton pipeline (for example one using the official `git-clone` and `buildah` tasks) cannot run in a namespace that enforces the `restricted` Pod Security Standard. The admission event names the init container `katanomi-setting-init` as not meeting the namespace security policy, in the usual Pod Security admission form: + +```text +pods "" is forbidden: violates PodSecurity "restricted:latest": ... (container "katanomi-setting-init" ...) +``` + +Setting `set-security-context: "true"` under `spec.pipeline` of the `TektonConfig` does not change the result; the same event is reported again. + +## Root Cause + +### Cause 1: the DevOps v3 components inject `katanomi-setting-init` into pipeline Pods + +`katanomi-setting-init` is not part of the Tekton task. It is an init container injected by the DevOps v3 (katanomi) components while they are installed on the cluster: + +- **v3 pipelines** do not support running in a `restricted` namespace. +- **v4 (Tekton) pipelines** also get the `katanomi-setting-init` init container injected automatically as long as the v3 components are still installed. The container does not satisfy the `restricted` policy, so the whole Pod is rejected. + +`set-security-context` in the `TektonConfig` applies to the containers Tekton creates. It does not remove or change the injected `katanomi-setting-init` container, which is why changing it has no effect. + +## Resolution + +### Cause 1 + +Choose the option that matches how the cluster still uses DevOps v3. + +**Option A: v3 is no longer needed.** Uninstall the DevOps v3 components. After that, v4 pipeline Pods no longer carry `katanomi-setting-init` and can run in a `restricted` namespace. + +> **Warning:** Uninstalling the DevOps v3 components removes the v3 pipeline functions, including Jenkins-based pipelines that still run on v3. Do this only after every pipeline that still depends on v3 has been migrated to v4 and verified. Reinstalling the v3 components is the only way back. + +**Option B: v3 and v4 must run in parallel for a while** (for example while Jenkins pipelines are still being migrated). Keep the v3 components and run the v4 pipelines in a namespace whose Pod Security level the pipeline Pods satisfy, and plan the move to a `restricted` namespace after the migration to v4 is finished. + +1. Check what a lower level would allow before changing it. The server-side dry run prints a warning for every existing Pod that would still be rejected: + + ```bash + kubectl label --dry-run=server --overwrite ns pod-security.kubernetes.io/enforce=baseline + ``` + +2. Apply the level that the pipeline Pods pass: + + > **Warning:** Lowering the enforced Pod Security level allows more privileged Pods in the namespace for every workload there, not only the pipelines. Prefer a dedicated namespace for pipelines. Record the current label first and roll back by setting it again. + + ```bash + kubectl get ns --show-labels + kubectl label --overwrite ns pod-security.kubernetes.io/enforce= + ``` + +3. Run the pipeline again and confirm that the Pod is admitted. + +## Diagnostic Steps + +1. Read the event of the rejected Pod or TaskRun and check which container is named: + + ```bash + kubectl -n get events --sort-by=.lastTimestamp | grep -i podsecurity + kubectl -n describe taskrun | sed -n '/Events/,$p' + ``` + + Expected for this cause: the violation lists `katanomi-setting-init`. If it lists only Tekton containers such as `prepare`, `place-scripts` or `step-*`, the problem is the securityContext of the Tekton containers, not the v3 injection. + +2. Check the namespace Pod Security labels: + + ```bash + kubectl get ns -o jsonpath='{.metadata.labels}{"\n"}' + ``` + + Expected: `pod-security.kubernetes.io/enforce: restricted`. + +3. Check whether the DevOps v3 (katanomi) components are still installed: + + ```bash + kubectl get katanomis.operators.katanomi.dev -A + kubectl get pods -A | grep -i katanomi + ``` + + Resources and Pods present mean the v3 components are installed and will keep injecting `katanomi-setting-init`. From faafd9d4b19c8c46b3957a34d9f7920059504efd Mon Sep 17 00:00:00 2001 From: Komh Date: Tue, 10 Nov 2026 16:17:01 +0800 Subject: [PATCH 02/25] docs(solutions): add 14 and update 0 troubleshooting articles --- ..._token_no_such_file_or_directory_on_ACP.md | 73 +++++++++++ ...g_another_platform_administrator_on_ACP.md | 67 ++++++++++ ..._v1_must_appear_in_spec_versions_on_ACP.md | 89 ++++++++++++++ ...es_data_hosts_from_cpaas-coredns_on_ACP.md | 114 ++++++++++++++++++ ...TPRoute_has_an_empty_retry_block_on_ACP.md | 101 ++++++++++++++++ ...ot_allowed_after_upgrading_to_ACP_4.3.3.md | 71 +++++++++++ ..._because_api_proxy_ca_data_is_malformed.md | 91 ++++++++++++++ ...id_password_error_on_the_ACP_login_page.md | 56 +++++++++ ...t_be_resolved_inside_the_cluster_on_ACP.md | 91 ++++++++++++++ ...r-secret_is_only_in_cpaas-system_on_ACP.md | 91 ++++++++++++++ ...h_a_PEM_certificate_and_key_on_ACP_3.12.md | 110 +++++++++++++++++ ...h_integration_to_another_cluster_on_ACP.md | 41 +++++++ ...n_with_custom-navconfig-platform_on_ACP.md | 64 ++++++++++ ...ue_to_Geneve_TX_checksum_offload_on_ACP.md | 96 +++++++++++++++ 14 files changed, 1155 insertions(+) create mode 100644 docs/en/solutions/ALB_pod_fails_to_start_after_rollback_with_serviceaccount_token_no_such_file_or_directory_on_ACP.md create mode 100644 docs/en/solutions/Built-in_admin_username_cannot_be_renamed_disable_admin_after_creating_another_platform_administrator_on_ACP.md create mode 100644 docs/en/solutions/Cluster_import_stuck_with_alb_module_Failed_grpcroutes_CRD_storedVersions_v1_must_appear_in_spec_versions_on_ACP.md create mode 100644 docs/en/solutions/CoreDNS_Host_Alias_records_do_not_take_effect_because_a_ResourcePatch_removes_data_hosts_from_cpaas-coredns_on_ACP.md create mode 100644 docs/en/solutions/Envoy_Gateway_hides_503_from_a_failing_backend_Pod_because_the_HTTPRoute_has_an_empty_retry_block_on_ACP.md create mode 100644 docs/en/solutions/Integrated_product_link_fails_with_Redirect_to_untrusted_host_is_not_allowed_after_upgrading_to_ACP_4.3.3.md create mode 100644 docs/en/solutions/Kiali_OpenID_login_fails_with_illegal_base64_data_at_input_byte_because_api_proxy_ca_data_is_malformed.md create mode 100644 docs/en/solutions/LDAP_user_who_has_never_logged_in_gets_an_invalid_password_error_on_the_ACP_login_page.md create mode 100644 docs/en/solutions/Platform_console_does_not_open_because_the_cluster_VIP_domain_name_cannot_be_resolved_inside_the_cluster_on_ACP.md create mode 100644 docs/en/solutions/Project_webhook_alert_notifications_miss_custom_headers_because_webhook-header-secret_is_only_in_cpaas-system_on_ACP.md create mode 100644 docs/en/solutions/Replace_the_platform_access_HTTPS_certificate_in_prdb_with_a_PEM_certificate_and_key_on_ACP_3.12.md create mode 100644 docs/en/solutions/Service_mesh_history_is_not_visible_after_switching_the_Elasticsearch_integration_to_another_cluster_on_ACP.md create mode 100644 docs/en/solutions/Storage_navigation_entries_added_by_plugins_or_Operators_cannot_be_hidden_with_custom-navconfig-platform_on_ACP.md create mode 100644 docs/en/solutions/kube-ovn-pinger_CrashLoopBackOff_context_deadline_exceeded_on_bonded_NICs_due_to_Geneve_TX_checksum_offload_on_ACP.md diff --git a/docs/en/solutions/ALB_pod_fails_to_start_after_rollback_with_serviceaccount_token_no_such_file_or_directory_on_ACP.md b/docs/en/solutions/ALB_pod_fails_to_start_after_rollback_with_serviceaccount_token_no_such_file_or_directory_on_ACP.md new file mode 100644 index 000000000..1070ee8a1 --- /dev/null +++ b/docs/en/solutions/ALB_pod_fails_to_start_after_rollback_with_serviceaccount_token_no_such_file_or_directory_on_ACP.md @@ -0,0 +1,73 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.16.x +--- + +# ALB pod fails to start after rollback with serviceaccount token no such file or directory on ACP + +## Issue + +After an ALB upgrade caused application errors, the ALB was rolled back urgently: the ALB operator version was set back to the version shipped with ACP 3.16, and the Deployment of the ALB instance was deleted so that the operator would recreate it. The recreated ALB pod does not start. Its log shows: + +```text +driver: use InClusterConfig +"lifecycle: get rest cfg fail" err="open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" +"run fail" err="open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" +``` + +The startup configuration printed by the pod shows `"K8s": {"Mode": "incluster", ...}`. The pod exits and restarts in a loop, and the traffic served by that ALB is not restored. + +## Root Cause + +The rolled-back ALB runs in in-cluster mode: it builds its Kubernetes client from the ServiceAccount token at the standard path `/var/run/secrets/kubernetes.io/serviceaccount/token`. + +That file only exists when Kubernetes mounts the ServiceAccount token into the pod. The ServiceAccount used by the ALB in `cpaas-system` has `automountServiceAccountToken: false`, and the pod created after the rollback does not mount the token itself, so the file is missing. Without the token the ALB cannot create its REST client and exits. + +## Resolution + +1. Find the ServiceAccount used by the ALB Deployment: + + ```bash + kubectl -n cpaas-system get deploy -o jsonpath='{.spec.template.spec.serviceAccountName}{"\n"}' + ``` + +2. Enable automatic token mounting on that ServiceAccount: + + ```bash + kubectl -n cpaas-system patch sa -p '{"automountServiceAccountToken": true}' + ``` + + To revert this change later, apply the same patch with `false`. + +3. Restart the ALB pods so that they are recreated with the token mounted. + + > **Warning:** Deleting an ALB pod interrupts the traffic it serves until the new pod is ready. Do this only after the `serviceaccount/token: no such file or directory` error is confirmed in the pod log. If the ALB has several replicas, delete one pod at a time and wait for it to become `Running` before the next. Pods are recreated by the Deployment, so no rollback is needed for this step. + + ```bash + kubectl -n cpaas-system get pods -o wide | grep + kubectl -n cpaas-system delete pod + ``` + +4. Confirm the new pod is `Running`, its log no longer shows the token error, and the applications behind the ALB respond again. + +## Diagnostic Steps + +1. Read the log of the failing ALB pod. The `get rest cfg fail` and `serviceaccount/token: no such file or directory` lines identify this issue: + + ```bash + kubectl -n cpaas-system logs --previous | grep -E 'InClusterConfig|rest cfg|serviceaccount/token' + ``` + +2. Check whether the token is mounted. Expected for this issue: the ServiceAccount prints `false`, and the pod has no volume mounted at `/var/run/secrets/kubernetes.io/serviceaccount`: + + ```bash + kubectl -n cpaas-system get sa -o jsonpath='{.automountServiceAccountToken}{"\n"}' + kubectl -n cpaas-system get pod \ + -o jsonpath='{.spec.automountServiceAccountToken}{"\n"}{range .spec.containers[*].volumeMounts[*]}{.mountPath}{"\n"}{end}' + ``` + +3. After the fix, the pod spec lists a mount at `/var/run/secrets/kubernetes.io/serviceaccount` and the log no longer reports the token error. diff --git a/docs/en/solutions/Built-in_admin_username_cannot_be_renamed_disable_admin_after_creating_another_platform_administrator_on_ACP.md b/docs/en/solutions/Built-in_admin_username_cannot_be_renamed_disable_admin_after_creating_another_platform_administrator_on_ACP.md new file mode 100644 index 000000000..41e2d9c7a --- /dev/null +++ b/docs/en/solutions/Built-in_admin_username_cannot_be_renamed_disable_admin_after_creating_another_platform_administrator_on_ACP.md @@ -0,0 +1,67 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.4 +--- + +# Built-in admin username cannot be renamed: disable admin after creating another platform administrator on ACP + +## Issue + +A security compliance assessment flags the platform's default `admin` account as a high-risk item (for example "the system has an admin account"), and asks for the default account to be renamed or removed. Administrators look for a way to change the login username of `admin`: + +- In the console, the user details and the personal center only allow changing the **display name** (and the password), not the username. +- The product documentation describes the platform administrator created at deployment as a system account that cannot be managed through user management, and has no procedure for renaming it. + +## Root Cause + +The login username of the local platform administrator created at deployment (`admin`) cannot be changed, either in the console or in the backend. The console only supports changing a user's display name. + +The `User` resource (`auth.alauda.io/v1`) of each user has a boolean `spec.is_disabled` field. Setting it to `true` on the `admin` user disables that account, which removes the default account from use without renaming it. + +## Resolution + +The supported workaround is to create a replacement platform administrator and then disable `admin`. + +1. Create a new local user with a non-default username in the console and grant it the platform administrator role. + +2. Log out, log in with the new user, and confirm it has full platform administration access. Keep this session open until step 4 is verified. + +3. On a control plane node of the `global` cluster, find the `User` resource of `admin`. Its resource name is a hash, not the username: + + ```bash + kubectl get users.auth.alauda.io | grep admin + kubectl get users.auth.alauda.io -o jsonpath='{.spec.username} is_admin={.spec.is_admin}{"\n"}' + ``` + + Expected: `admin is_admin=true`. + +4. Disable the account. + + > **Warning:** After this step nobody can log in to the console as `admin`. Run it only after step 2 confirms that the new platform administrator can log in and manage the platform, otherwise you can lock yourself out of the console. Check first that no script or integration still logs in as `admin`. To roll back, set `is_disabled` back to `false` (or remove it) on the same resource from a `global` control plane node with `kubectl`. + + ```bash + kubectl edit users.auth.alauda.io + ``` + + Add the field under `spec`: + + ```yaml + spec: + is_disabled: true + ``` + +5. Open a new private browser window and confirm that logging in as `admin` is refused while the new administrator still works. + +## Diagnostic Steps + +1. Confirm which account the assessment refers to: the local `admin` user created at deployment has `spec.username: admin` and `spec.is_admin: true`: + + ```bash + kubectl get users.auth.alauda.io -o custom-columns='NAME:.metadata.name,USERNAME:.spec.username,ADMIN:.spec.is_admin,DISABLED:.spec.is_disabled' + ``` + +2. After the change, the same command shows `DISABLED` = `true` for `admin` and the new administrator listed with `ADMIN` = `true`. diff --git a/docs/en/solutions/Cluster_import_stuck_with_alb_module_Failed_grpcroutes_CRD_storedVersions_v1_must_appear_in_spec_versions_on_ACP.md b/docs/en/solutions/Cluster_import_stuck_with_alb_module_Failed_grpcroutes_CRD_storedVersions_v1_must_appear_in_spec_versions_on_ACP.md new file mode 100644 index 000000000..0ffd04ea6 --- /dev/null +++ b/docs/en/solutions/Cluster_import_stuck_with_alb_module_Failed_grpcroutes_CRD_storedVersions_v1_must_appear_in_spec_versions_on_ACP.md @@ -0,0 +1,89 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.11 +--- + +# Cluster import stuck with the alb module Failed: grpcroutes CRD status.storedVersions v1 must appear in spec.versions on ACP + +## Issue + +Importing an existing Kubernetes cluster (for example a managed cloud cluster such as Alibaba Cloud ACK) into ACP 4.0.11 never finishes and stays at the import step. On the `global` cluster, the cluster's `alb` module is `Failed`, while other modules such as `cert-manager`, `acp-business`, `marketplace`, `auth-manager`, `base` and `sentry` are `Running`. + +On the imported cluster, the `alauda-alb2` chart release keeps failing with phase `DeployFailed` and this message: + +```text +failed to install chart acp/chart-alauda-alb2-, err: failed to install chart for release "alauda-alb2": +failed to install CRD: failed to apply CustomResourceDefinition "/grpcroutes.gateway.networking.k8s.io": +CustomResourceDefinition.apiextensions.k8s.io "grpcroutes.gateway.networking.k8s.io" is invalid: +status.storedVersions[0]: Invalid value: "v1": must appear in spec.versions +``` + +The release shows `Validate` = `True` (`ChartDownloaded`), `Sync` = `False` (`Failed`) and `Health` = `Unknown`, and a Warning `ChartSynced` event repeats thousands of times. + +## Root Cause + +The target cluster already had a `grpcroutes.gateway.networking.k8s.io` CRD before it was imported, installed by something other than the platform, for example an earlier Gateway API installation. Objects of that CRD have been stored as version `v1`, so `v1` is recorded in the CRD's `status.storedVersions`. + +The ALB chart installs its own definition of the same CRD, and that definition does not list `v1` in `spec.versions`. The Kubernetes API server rejects any CRD update that would drop a version still listed in `status.storedVersions`, so applying the chart's CRD fails. The `alauda-alb2` release cannot be installed, the `alb` module stays `Failed`, and the import never reaches the ready state. + +## Resolution + +1. On the imported cluster, confirm the conflict and list any GRPCRoute objects that would be lost: + + ```bash + kubectl get crd grpcroutes.gateway.networking.k8s.io \ + -o jsonpath='spec.versions={.spec.versions[*].name} storedVersions={.status.storedVersions}{"\n"}' + kubectl get grpcroutes.gateway.networking.k8s.io -A + ``` + + Continue only if `storedVersions` contains `v1` and the release message is the one shown in the Issue section. + +2. Back up the CRD and its objects: + + ```bash + kubectl get crd grpcroutes.gateway.networking.k8s.io -o yaml > grpcroutes-crd.yaml + kubectl get grpcroutes.gateway.networking.k8s.io -A -o yaml > grpcroutes.yaml + ``` + +3. Delete the conflicting CRD. + + > **Warning:** Deleting a CRD deletes every object of that kind in the cluster. Run this only when the `storedVersions` conflict above is confirmed, and only after step 1 shows no GRPCRoute objects you still need, or they are saved by step 2. Whatever installed the existing CRD may also depend on it. To roll back, run `kubectl create -f grpcroutes-crd.yaml` and then `kubectl create -f grpcroutes.yaml`. The objects can only be restored while their version is still served by the CRD that is installed at that time. + + ```bash + kubectl delete crd grpcroutes.gateway.networking.k8s.io + ``` + +4. The chart release is retried automatically. Watch it on the imported cluster until it is synced, and confirm the `alb` module becomes `Running` on `global`: + + ```bash + kubectl get ars -A | grep -i alb + kubectl get moduleinfo -l cpaas.io/cluster-name= # on the global cluster + ``` + + Once all modules are `Running`, the import completes. + +## Diagnostic Steps + +1. On `global`, find the module that blocks the import. A stuck import with only `alb` `Failed` points at the ALB chart: + + ```bash + kubectl get moduleinfo -l cpaas.io/cluster-name= + ``` + +2. On the imported cluster, read the ALB chart release status. The `storedVersions[0]: Invalid value: "v1": must appear in spec.versions` message on `grpcroutes.gateway.networking.k8s.io` identifies this issue. A different message, for example an API version the cluster does not serve, is a different problem: + + ```bash + kubectl get ars -A | grep -i alb + kubectl get ars -n -o yaml + ``` + +3. Confirm that the CRD existed before the import and was not created by the platform. Its creation time is earlier than the import and its versions include `v1`: + + ```bash + kubectl get crd grpcroutes.gateway.networking.k8s.io \ + -o jsonpath='{.metadata.creationTimestamp} {.spec.versions[*].name}{"\n"}' + ``` diff --git a/docs/en/solutions/CoreDNS_Host_Alias_records_do_not_take_effect_because_a_ResourcePatch_removes_data_hosts_from_cpaas-coredns_on_ACP.md b/docs/en/solutions/CoreDNS_Host_Alias_records_do_not_take_effect_because_a_ResourcePatch_removes_data_hosts_from_cpaas-coredns_on_ACP.md new file mode 100644 index 000000000..39fd934a2 --- /dev/null +++ b/docs/en/solutions/CoreDNS_Host_Alias_records_do_not_take_effect_because_a_ResourcePatch_removes_data_hosts_from_cpaas-coredns_on_ACP.md @@ -0,0 +1,114 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# CoreDNS Host Alias records do not take effect because a ResourcePatch removes data hosts from cpaas-coredns on ACP + +## Issue + +Static DNS records added in the cluster plugin **Alauda Container Platform Cluster DNS** (CoreDNS) under **Host Alias** do not take effect. Pods report `Could not resolve host` for the new name. After CoreDNS is restarted, names that used to work resolve to the wrong IP address. + +In `kube-system`, the ConfigMap `cpaas-coredns` has a `Corefile` that still contains the `hosts /etc/coredns/hosts` plugin, but the ConfigMap has no `hosts` key, and the hosts file inside the CoreDNS pods is empty or missing. + +## Root Cause + +The records entered under **Host Alias** are written to the `hosts` key of the ConfigMap `cpaas-coredns`. CoreDNS mounts that key as `/etc/coredns/hosts` and serves it with the `hosts` plugin (`hosts /etc/coredns/hosts { reload ... fallthrough }`). + +A `ResourcePatch` (`operator.alauda.io/v1alpha1`, short name `rpch`) applies JSON patches to the resources of a platform-managed release and keeps them applied. When a ResourcePatch for the `coredns` release targets `cpaas-coredns` with an operation such as: + +```yaml +spec: + jsonPatch: + - op: replace + path: /data/Corefile + value: | + ... + - op: remove + path: /data/hosts +``` + +the `hosts` key is stripped from the ConfigMap every time. This often happens when a ResourcePatch was created to customize the Corefile (for example to forward a private domain to a dedicated DNS server) and the `remove /data/hosts` operation was included with it. + +As a result: + +- new Host Alias entries never reach CoreDNS; +- after a restart, CoreDNS starts with no static records, and because of `fallthrough` the names are resolved by the upstream DNS instead, which can return a different, stale address. + +## Resolution + +1. Back up the ConfigMap and the ResourcePatch: + + ```bash + kubectl -n kube-system get cm cpaas-coredns -o yaml > cpaas-coredns.before.yaml + kubectl get rpch -A + kubectl get rpch -o yaml > coredns-resourcepatch.before.yaml + ``` + + Add `-n ` to the last command if `kubectl get rpch -A` lists the patch in a namespace. + +2. Decide whether the other changes in the patch are still needed. Read its `spec.jsonPatch`: besides `remove /data/hosts` it may replace the Corefile, for example to add a forwarding block for a private domain. + +3. Remove the patch that strips the hosts data. + + > **Warning:** Deleting the ResourcePatch also drops every other customization it carries, such as an extra forwarding block in the Corefile; names that depend on it stop resolving. Do this only when `kubectl get rpch -o yaml` shows a `remove` operation on `/data/hosts` of `cpaas-coredns`, and only after the backups in step 1. To roll back, run `kubectl apply -f coredns-resourcepatch.before.yaml`. + + - If none of its other operations are needed any more (for example, the private DNS server it forwarded to is no longer used), delete it: + + ```bash + kubectl delete rpch + ``` + + - If the Corefile customization is still needed, edit the ResourcePatch instead and remove only the `op: remove` / `path: /data/hosts` entry from `spec.jsonPatch`: + + ```bash + kubectl edit rpch + ``` + +4. Check that the hosts data is back in the ConfigMap. If it is still empty, open **Administrator** > **Marketplace** > **Cluster Plugins**, update **Alauda Container Platform Cluster DNS**, re-enter the IP addresses and domains under **Host Alias** and click **Update**. + + ```bash + kubectl -n kube-system get cm cpaas-coredns -o jsonpath='{.data.hosts}{"\n"}' + ``` + +5. Verify from a workload Pod. The `hosts` plugin reloads the file automatically once the updated ConfigMap reaches the CoreDNS pods (kubelet syncs mounted ConfigMaps periodically, so allow a minute or two): + + ```bash + kubectl -n exec -- nslookup + ``` + +Do not add a `remove /data/hosts` operation to a ResourcePatch for CoreDNS again. Keep static records in **Host Alias** and limit the ResourcePatch to the Corefile changes that cannot be made in the plugin. + +## Diagnostic Steps + +1. Check whether the ConfigMap carries the static records. A `Corefile` that references `/etc/coredns/hosts` but no `hosts` key matches this issue: + + ```bash + kubectl -n kube-system get cm cpaas-coredns -o yaml + ``` + +2. List the ResourcePatches and look for one that targets `cpaas-coredns` with `remove` on `/data/hosts`: + + ```bash + kubectl get rpch -A + kubectl get rpch -o yaml | grep -n -B2 -A2 '/data/hosts' + ``` + +3. Check the file CoreDNS actually reads: + + ```bash + kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide + kubectl -n kube-system exec -- cat /etc/coredns/hosts + ``` + + An empty result or `No such file or directory` confirms that the records are not loaded. If the CoreDNS image has no `cat`, compare the ConfigMap instead (step 1). + +4. Rule out NodeLocal DNSCache. If the Pod's `/etc/resolv.conf` points at the NodeLocal address (for example `169.254.20.10`) and a direct query to the CoreDNS Service IP succeeds while the default one fails, the records are in CoreDNS and the cache is not forwarding that name, which is a different problem: + + ```bash + kubectl -n exec -- cat /etc/resolv.conf + kubectl -n exec -- nslookup + ``` diff --git a/docs/en/solutions/Envoy_Gateway_hides_503_from_a_failing_backend_Pod_because_the_HTTPRoute_has_an_empty_retry_block_on_ACP.md b/docs/en/solutions/Envoy_Gateway_hides_503_from_a_failing_backend_Pod_because_the_HTTPRoute_has_an_empty_retry_block_on_ACP.md new file mode 100644 index 000000000..05d9666b0 --- /dev/null +++ b/docs/en/solutions/Envoy_Gateway_hides_503_from_a_failing_backend_Pod_because_the_HTTPRoute_has_an_empty_retry_block_on_ACP.md @@ -0,0 +1,101 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.4.x +--- + +# Envoy Gateway hides 503 from a failing backend Pod because the HTTPRoute has an empty retry block on ACP + +## Issue + +A Deployment behind a Service has two Pods. One of them returns HTTP `503` for every request (for example, deliberately for a test). The responses carry the Pod IP, so it is visible which Pod answered. + +- Requests sent to the Service ClusterIP reach both Pods: the healthy Pod returns `200`, the failing Pod returns `503`. +- Requests sent through Envoy Gateway only ever return `200` from the healthy Pod. No response carries the failing Pod's IP, so it looks as if the gateway sends all traffic to one Pod or never selects the failing one. + +The Service has no session affinity, and both Pods are `Ready` and listed in the EndpointSlice. + +## Root Cause + +The HTTPRoute rule contains an empty retry object: + +```yaml +rules: + - backendRefs: + - name: + port: + retry: {} +``` + +Even with no parameters, `retry: {}` makes Envoy Gateway (as shipped with ACP 4.4) apply its default retry policy to the route. That default treats `503` as retryable and uses Envoy's `previous_hosts` retry predicate, which picks a different backend host for the retry. + +A request that first reaches the failing Pod gets `503`, is retried automatically on the healthy Pod, and the client receives `200`. The failing Pod does receive traffic, but its `503` responses are never returned to the client. + +This is the expected behavior of the retry configuration. It is not a Service load-balancing problem, and the failing Pod is not excluded by the gateway. + +## Resolution + +Choose one of the following for each affected route. + +1. **The route does not need retries:** remove the `retry` field from the rule entirely. Do not leave `retry: {}` in place. + + ```bash + kubectl -n edit httproute + ``` + + Delete the `retry:` line under `spec.rules[]`, save, and send requests through the gateway again. Both `200` and `503` responses now reach the client. + +2. **The route needs retries:** configure them explicitly instead of relying on the defaults, so that you decide which status codes are retried and how often. For example: + + ```yaml + rules: + - backendRefs: + - name: + port: + retry: + codes: + - 503 + attempts: 2 + backoff: 100ms + ``` + + Enable retries on `503` with care for external interfaces, write operations and other non-idempotent requests: a retried request can be executed twice by the backend, and the retries hide backend failures from clients and from client-side monitoring. + +If routes are created from the console or from generated YAML, check whether they contain an empty `retry: {}` object that nobody added on purpose. + +## Diagnostic Steps + +1. Look for an empty or default retry configuration on the route: + + ```bash + kubectl -n get httproute -o yaml | grep -n -A4 'retry' + ``` + + `retry: {}` matches this issue. + +2. Confirm that the failing Pod still receives requests through the gateway. Its own access log shows requests coming from the Envoy Gateway proxy Pods even though no client sees its `503`: + + ```bash + kubectl -n get pod -o wide | grep + kubectl -n logs --tail=50 + ``` + +3. Rule out look-alikes: + + - Session affinity or consistent hashing would pin clients to one Pod; check the Service and any `BackendTrafficPolicy` that targets the route: + + ```bash + kubectl -n get svc -o jsonpath='{.spec.sessionAffinity}{"\n"}' + kubectl -n get backendtrafficpolicy -o yaml + ``` + + - A Pod that is not `Ready` is removed from the EndpointSlice and gets no traffic at all; check that both Pods are listed with `ready: true`: + + ```bash + kubectl -n get endpointslice -l kubernetes.io/service-name= -o yaml + ``` + +4. After removing `retry: {}`, repeat the test through the gateway. Responses from both Pods appear, which confirms the cause. diff --git a/docs/en/solutions/Integrated_product_link_fails_with_Redirect_to_untrusted_host_is_not_allowed_after_upgrading_to_ACP_4.3.3.md b/docs/en/solutions/Integrated_product_link_fails_with_Redirect_to_untrusted_host_is_not_allowed_after_upgrading_to_ACP_4.3.3.md new file mode 100644 index 000000000..24343a7bf --- /dev/null +++ b/docs/en/solutions/Integrated_product_link_fails_with_Redirect_to_untrusted_host_is_not_allowed_after_upgrading_to_ACP_4.3.3.md @@ -0,0 +1,71 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.3.3 +--- + +# Integrated product link fails with Redirect to untrusted host is not allowed after upgrading to ACP 4.3.3 + +## Issue + +After the platform is upgraded to ACP 4.3.3, an external product that was integrated into the console no longer opens from the platform. The product is listed under **Platform Settings** > **Product Configuration** as an integrated product, with its entry shown, single sign-on enabled and an external product link. + +Clicking the entry opens `/console/api/v2/token/redirect?redirect_url=`, and the page shows: + +```text +Redirect to untrusted host is not allowed +``` + +The same entry worked before the upgrade. + +## Root Cause + +In the new version, the console only performs token redirects to hosts that are explicitly allowed. The allowed hosts are listed under the `allowed-redirect-hosts` key of the ConfigMap `console-configmap` in the `cpaas-system` namespace of the `global` cluster. + +When the host of an integrated product link is not on that list, the console does not trust it and rejects the redirect. After the upgrade, products that were integrated earlier can be in this state. + +## Resolution + +1. On a control plane node of the `global` cluster, back up the ConfigMap and look at the current list: + + ```bash + kubectl -n cpaas-system get configmap console-configmap -o yaml > console-configmap.before.yaml + kubectl -n cpaas-system get configmap console-configmap -o jsonpath='{.data.allowed-redirect-hosts}{"\n"}' + ``` + +2. Edit the ConfigMap and add the host of every integrated product link (the host part of ``) to `allowed-redirect-hosts`. Follow the format of the existing entries, and leave the other keys, such as `enable-api-auth`, unchanged: + + ```bash + kubectl -n cpaas-system edit configmap console-configmap + ``` + +3. Restart the console frontend pods so that they load the new list. + + > **Warning:** While the frontend pods restart, the console can be briefly unavailable or show errors to logged-in users; workloads are not affected. Do this only after the `Redirect to untrusted host is not allowed` message is confirmed and the ConfigMap is updated. If the console misbehaves afterwards, restore the backup with `kubectl apply -f console-configmap.before.yaml` and restart the frontend pods again. + + ```bash + kubectl -n cpaas-system get deploy | grep -i frontend + kubectl -n cpaas-system rollout restart deploy + kubectl -n cpaas-system rollout status deploy + ``` + +4. Open the integrated product entry from the console again. It redirects to the product. + +## Diagnostic Steps + +1. Reproduce the failure in the browser and note the exact URL. The `/console/api/v2/token/redirect?redirect_url=...` path together with `Redirect to untrusted host is not allowed` identifies this issue. + +2. Compare the host of `redirect_url` with the allowed list. If the host is missing from `allowed-redirect-hosts`, this is the cause: + + ```bash + kubectl -n cpaas-system get configmap console-configmap -o jsonpath='{.data.allowed-redirect-hosts}{"\n"}' + ``` + +3. If the host is already listed but the error persists, check that the frontend pods were restarted after the change (their age is newer than the edit): + + ```bash + kubectl -n cpaas-system get pods | grep -i frontend + ``` diff --git a/docs/en/solutions/Kiali_OpenID_login_fails_with_illegal_base64_data_at_input_byte_because_api_proxy_ca_data_is_malformed.md b/docs/en/solutions/Kiali_OpenID_login_fails_with_illegal_base64_data_at_input_byte_because_api_proxy_ca_data_is_malformed.md new file mode 100644 index 000000000..5fbabc991 --- /dev/null +++ b/docs/en/solutions/Kiali_OpenID_login_fails_with_illegal_base64_data_at_input_byte_because_api_proxy_ca_data_is_malformed.md @@ -0,0 +1,91 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Service Mesh +ProductsVersion: [] +--- + +# Kiali OpenID login fails with illegal base64 data at input byte because api_proxy_ca_data is malformed + +## Issue + +Opening Kiali for a workload cluster through the platform (`https:///clusters//kiali`) redirects to the ACP login page. After a successful platform login, Kiali reports that the OpenID login failed. The browser console shows: + +```text +Authentication openid_error: Unable to create a Kubernetes client from the auth token: illegal base64 data at input byte +``` + +The Kiali CR uses `spec.auth.strategy: openid`, and the Kiali pod itself is running. + +## Root Cause + +With the `openid` strategy on ACP, Kiali sends the user's token to the platform's Kubernetes API proxy (`spec.auth.openid.api_proxy`) and trusts that proxy with the CA given in `spec.auth.openid.api_proxy_ca_data`. That field must hold the platform CA certificate **base64-encoded**, exactly as it is stored in the `dex.tls` Secret on the `global` cluster. + +During installation the value is normally filled in with `export PLATFORM_CA=` and `envsubst < kiali.yaml | kubectl apply -f -`, which writes `api_proxy_ca_data: ${PLATFORM_CA}` into the Kiali CR. If the CA is not copied correctly at that point (truncated, extra or missing characters, line breaks, or the decoded PEM text instead of the base64 string), the field is not valid base64. The login through Dex succeeds, but Kiali fails when it decodes the CA to build the Kubernetes client, and reports `illegal base64 data at input byte `, where `` is the position of the first invalid character. + +## Resolution + +1. On the `global` cluster, save the platform CA as a single base64 line: + + ```bash + kubectl -n cpaas-system get secret dex.tls -o jsonpath='{.data.ca\.crt}' > platform-ca.b64 + ``` + + If `platform-ca.b64` is empty, use the `tls.crt` key instead: + + ```bash + kubectl -n cpaas-system get secret dex.tls -o jsonpath='{.data.tls\.crt}' > platform-ca.b64 + ``` + + Check that the file decodes to a certificate: + + ```bash + base64 -d platform-ca.b64 | openssl x509 -noout -subject -enddate + ``` + +2. Copy `platform-ca.b64` to a host with `kubectl` access to the workload cluster where Kiali runs, without opening and re-pasting it in an editor. + +3. Find the Kiali CR and write the correct value into `api_proxy_ca_data`: + + ```bash + kubectl get kiali -A + kubectl -n patch kiali --type merge \ + -p "{\"spec\":{\"auth\":{\"openid\":{\"api_proxy_ca_data\":\"$(tr -d '\n' < platform-ca.b64)\"}}}}" + ``` + + Editing the field directly with `kubectl edit kiali` works too, as long as the value is pasted as one unbroken line. + +4. Wait for the Kiali operator to roll out the new configuration, then open Kiali through the platform again and log in. Kiali is briefly unavailable while its pod is replaced. + + ```bash + kubectl -n get pods -w | grep kiali + ``` + +## Diagnostic Steps + +1. Check that the value in the CR decodes to a certificate. A `base64: invalid input` error, or an `openssl` error, confirms this issue: + + ```bash + kubectl -n get kiali \ + -o jsonpath='{.spec.auth.openid.api_proxy_ca_data}' | base64 -d | openssl x509 -noout -subject + ``` + +2. Compare the CR value with the `global` cluster's `dex.tls` value character by character, for example by comparing their checksums: + + ```bash + kubectl -n get kiali -o jsonpath='{.spec.auth.openid.api_proxy_ca_data}' | md5sum + kubectl -n cpaas-system get secret dex.tls -o jsonpath='{.data.ca\.crt}' | md5sum # on global; use tls.crt if ca.crt is empty + ``` + +3. Check the other OpenID settings while you are there. `api_proxy` should be `/kubernetes/`, and `issuer_uri` should point to the platform Dex (`/dex`): + + ```bash + kubectl -n get kiali -o jsonpath='{.spec.auth.openid}{"\n"}' + ``` + +4. Look for the same error in the Kiali log to confirm that the failure happens when the Kubernetes client is created, not earlier at the Dex redirect: + + ```bash + kubectl -n logs deploy/kiali --tail=300 | grep -iE 'openid|base64|token' + ``` diff --git a/docs/en/solutions/LDAP_user_who_has_never_logged_in_gets_an_invalid_password_error_on_the_ACP_login_page.md b/docs/en/solutions/LDAP_user_who_has_never_logged_in_gets_an_invalid_password_error_on_the_ACP_login_page.md new file mode 100644 index 000000000..dd093adb7 --- /dev/null +++ b/docs/en/solutions/LDAP_user_who_has_never_logged_in_gets_an_invalid_password_error_on_the_ACP_login_page.md @@ -0,0 +1,56 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# LDAP user who has never logged in gets an invalid password error on the ACP login page + +## Issue + +A user tries to log in to the ACP console with a directory (LDAP / Active Directory) account for the first time. The account has no platform user yet, because directory users are only created in ACP when they log in or are synchronized. The login page reports that the password is invalid, although the password has been verified and works elsewhere. + +Other directory users log in normally, and the user is testing from a network segment that can reach the platform. Often the organization has more than one directory domain, and the failing user belongs to a domain other than the one most users are in. + +## Root Cause + +The login page does not reveal whether an account exists. When the account cannot be found and when the password is wrong, the platform returns the same invalid-password error, by design: a distinct "user not found" answer would let an attacker probe for valid account names. An invalid-password error for a user whose password is known to be correct therefore usually means that the platform **cannot find the account**. + +For an LDAP identity provider (IDP), ACP looks up a login account by combining the IDP's user **Filter** with an equality match on the configured **Login Field** attributes, below the user **Base DN**, in the directory server the IDP is connected to. A user who is not in that directory, is outside the Base DN, or is excluded by the Filter is never found, and the login fails with the invalid-password error. This happens, for example, when the user's accounts live in a separate directory domain whose accounts are not present in the directory the IDP queries. + +## Resolution + +1. Confirm which directory, Base DN and Filter the platform uses: open **Users** > **IDP** and check the LDAP IDP's server address, **Search Setting** (**Filter**, **Base DN**) and **Field Mapping** (**Login Field**). + +2. Make the user reachable through that configuration. Choose what fits the directory layout: + + - Have the directory administrator make the user's account available in the directory the IDP queries, for example by synchronizing the other domain's accounts into it, so that the user is found below the configured Base DN and matches the Filter. + - Or adjust the IDP's **Base DN** and **Filter** so that they include the user's branch, if the account is already in that directory but outside the configured search. + +3. If the IDP has **Auto Sync** enabled or you synchronize manually, run the synchronization and check that the user now appears under **Users**. Otherwise, the platform user is created at the first successful login. + +4. Ask the user to log in again with the same login attribute (for example `sAMAccountName` or `mail`) configured as **Login Field**. + +## Diagnostic Steps + +1. Read the authentication log on the `global` cluster around the failing login. Find the `apollo` pod and search its log for the username: + + ```bash + kubectl get pod -A | grep apollo + kubectl logs -n cpaas-system | grep -C 5 + ``` + +2. Query the directory with the IDP's own settings from a host whose network is equivalent to the cluster. No result means the platform cannot find the user, which explains the invalid-password error: + + ```bash + ldapsearch -H ldap://:389 -D '' -W \ + -b '' '(&(=))' dn + ``` + + One entry returned means the account is found; then the password itself, an account lock, or the account being excluded by policy is the next thing to check. + +3. Compare with a user who can log in: if that user is in a different domain or branch than the failing one, the IDP's directory or Base DN covers only the working one. + +4. Rule out network reachability by testing from a segment that is known to reach the platform. A user who cannot reach the platform at all sees a connection error, not an invalid-password error. diff --git a/docs/en/solutions/Platform_console_does_not_open_because_the_cluster_VIP_domain_name_cannot_be_resolved_inside_the_cluster_on_ACP.md b/docs/en/solutions/Platform_console_does_not_open_because_the_cluster_VIP_domain_name_cannot_be_resolved_inside_the_cluster_on_ACP.md new file mode 100644 index 000000000..1bd577643 --- /dev/null +++ b/docs/en/solutions/Platform_console_does_not_open_because_the_cluster_VIP_domain_name_cannot_be_resolved_inside_the_cluster_on_ACP.md @@ -0,0 +1,91 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Platform console does not open because the cluster VIP domain name cannot be resolved inside the cluster on ACP + +## Issue + +The ACP console cannot be opened, although every platform pod on the `global` cluster is `Running` and the API server certificates are not expired. Restarting platform components does not help. Platform components that proxy requests to the cluster may log that the cluster is temporarily not accessible. + +The cluster's VIP (the cluster access address entered at installation) is a **domain name**, not an IP address. + +## Root Cause + +When the cluster VIP is configured as a domain name, platform components inside the cluster must resolve that name to reach the cluster. Name resolution inside the cluster works as follows: + +- Pods that use cluster DNS ask CoreDNS. CoreDNS answers from its own records (such as **Host Alias** entries) and forwards every other name to the DNS servers in the node's `/etc/resolv.conf`. +- Host-network components use the node's `/etc/resolv.conf` directly. + +If the DNS servers listed in the nodes' default `/etc/resolv.conf` cannot resolve the VIP domain name, and CoreDNS has no record for it, nothing inside the cluster can resolve the cluster address. The components keep running, but their requests to the cluster fail, so the console does not open. + +A browser on a workstation may still resolve the name (for example through a local hosts file or a different DNS server), which hides the problem from the outside. + +## Resolution + +Make the VIP domain name resolvable inside the cluster, in one of two ways. + +1. **Fix the node DNS (preferred).** Make sure the DNS servers in `/etc/resolv.conf` on every cluster node can resolve the VIP domain name: add the record on those DNS servers, or point the nodes to DNS servers that have it. This covers both host-network components and Pods, because CoreDNS forwards to the same servers. + + After the change, check on each node: + + ```bash + cat /etc/resolv.conf + nslookup + ``` + +2. **Add a static record to CoreDNS.** If the node DNS servers cannot be changed, add the record to the cluster DNS plugin: + + 1. Open **Administrator** > **Marketplace** > **Cluster Plugins**, find **Alauda Container Platform Cluster DNS** and click **Update**. + 2. Under **Host Alias**, enter the VIP address as **IP** and the VIP domain name under **Domains**, then click **Update**. + + If the console cannot be opened at all, add the record from the command line on a `global` control plane node as an emergency measure, in hosts-file format, to the `hosts` key of the ConfigMap `cpaas-coredns` in `kube-system`. + + > **Warning:** `cpaas-coredns` drives DNS for every Pod in the cluster, and a syntax error in it can break all in-cluster name resolution. Do this only when step 2 of the Diagnostic Steps shows that the node DNS cannot resolve the VIP name and option 1 is not possible. Keep every existing line of `hosts` and only append the new one. To roll back, run `kubectl apply -f cpaas-coredns.before.yaml`. + + ```bash + kubectl -n kube-system get cm cpaas-coredns -o yaml > cpaas-coredns.before.yaml + kubectl -n kube-system edit cm cpaas-coredns + ``` + + ```yaml + data: + hosts: | + + + ``` + + A manual edit of this ConfigMap can be overwritten when the plugin is reconciled. Once the console opens, enter the same record under **Host Alias** so that it persists. A CoreDNS record only helps Pods that use cluster DNS. Host-network components still depend on the node DNS (option 1). + +3. Reload the console in the browser and confirm it opens. + +## Diagnostic Steps + +1. Confirm that the cluster VIP is a domain name. Find the address the cluster was installed with, for example in the `server:` field of the kubeconfig used on the control plane nodes: + + ```bash + grep 'server:' ~/.kube/config /etc/kubernetes/*.conf 2>/dev/null + ``` + +2. On a cluster node, resolve the name through the node's DNS servers. `NXDOMAIN`, `server can't find` or a timeout matches this issue: + + ```bash + nslookup + for ns in $(awk '/^nameserver/{print $2}' /etc/resolv.conf); do nslookup $ns; done + ``` + +3. Resolve the name from inside a Pod that has `nslookup` (or `getent`). `unknown host` or `can't resolve` confirms that cluster DNS cannot resolve it either: + + ```bash + kubectl -n exec -- nslookup + ``` + +4. Rule out an unreachable VIP: if the name resolves but the address does not answer, the problem is the VIP itself, not DNS: + + ```bash + curl -k -m 5 https://:6443/healthz + ``` diff --git a/docs/en/solutions/Project_webhook_alert_notifications_miss_custom_headers_because_webhook-header-secret_is_only_in_cpaas-system_on_ACP.md b/docs/en/solutions/Project_webhook_alert_notifications_miss_custom_headers_because_webhook-header-secret_is_only_in_cpaas-system_on_ACP.md new file mode 100644 index 000000000..2049829ac --- /dev/null +++ b/docs/en/solutions/Project_webhook_alert_notifications_miss_custom_headers_because_webhook-header-secret_is_only_in_cpaas-system_on_ACP.md @@ -0,0 +1,91 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Project webhook alert notifications miss custom headers because webhook-header-secret is only in cpaas-system on ACP + +## Issue + +Alert notifications sent to a third-party Webhook endpoint work for notification rules created by the platform administrator, but fail for notification rules created under **Project Management** (often the ones that cover a workload cluster). The receiving endpoint rejects the request because a required custom header is missing, for example: + +```text +Required request header ... is not present +``` + +The platform-level setup follows the product documentation: a Secret `webhook-header-secret` of type `NotificationSender` exists in `cpaas-system` on the `global` cluster, and the platform receiver group carries the annotation `cpaas.io/notification.webhook.config: webhook-header-secret`. The question is then whether workload clusters and projects use the same Secret, and where to configure it. + +## Root Cause + +The custom headers of a Webhook notification come from a `NotificationSender` Secret that the receiver group (`NotificationGroup`, `ait.alauda.io`) references by name through the `cpaas.io/notification.webhook.config` annotation. Each key under the Secret's `data` becomes an HTTP header name. + +The product documentation only describes creating that Secret in `cpaas-system`, which is where platform-level receiver groups live. Notification rules and receiver groups created under **Project Management** are tenant-isolated and are stored in the namespace whose name is the project name. A receiver group in that namespace cannot reference a Secret in `cpaas-system`, so the headers are never added and the endpoint rejects the request. + +There is no separate per-cluster configuration: what matters is the namespace of the receiver group, not the cluster that raised the alert. + +## Resolution + +Create the header Secret in the namespace named after the project, on the `global` cluster, and reference it from the project's receiver group. + +1. On a control plane node of the `global` cluster, find the receiver group used by the failing notification rule and note its namespace (the project name): + + ```bash + kubectl get notificationgroups.ait.alauda.io -A + ``` + +2. Create the Secret in that namespace. Use the same header names as the platform Secret; every value must be base64-encoded: + + ```yaml + apiVersion: v1 + kind: Secret + metadata: + name: webhook-header-secret + namespace: + type: NotificationSender + data: + : + ``` + + ```bash + kubectl apply -f webhook-header-project.yaml + ``` + + To reuse the values of the platform Secret, print them with `kubectl get secret webhook-header-secret -n cpaas-system -o yaml` and copy the `data` section. + +3. Make sure the project receiver group references the Secret. Edit it and add the annotation if it is missing: + + ```bash + kubectl edit notificationgroups.ait.alauda.io -n + ``` + + ```yaml + metadata: + annotations: + cpaas.io/notification.webhook.config: webhook-header-secret + ``` + +4. Trigger or wait for an alert covered by the project notification rule and confirm that the endpoint no longer reports the missing header. + +Repeat the steps for every project whose notification rules send to a Webhook endpoint that requires custom headers. + +## Diagnostic Steps + +1. List the receiver groups with their namespace and header annotation. A failing group in a project namespace whose annotation points to a Secret that does not exist in that namespace matches this issue: + + ```bash + kubectl get notificationgroups.ait.alauda.io -A \ + -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,HEADER_SECRET:.metadata.annotations.cpaas\.io/notification\.webhook\.config' + ``` + +2. Check that the referenced Secret exists in the same namespace as the receiver group and has the expected type and keys: + + ```bash + kubectl get secret webhook-header-secret -n -o jsonpath='{.type}{"\n"}{.data}{"\n"}' + ``` + + Expected: `NotificationSender` and the header names the endpoint requires. `NotFound` means the Secret exists only in `cpaas-system`. + +3. If the platform receiver group in `cpaas-system` fails in the same way, the Secret is not namespace-mismatched. Check instead that the annotation is present on that group and that the Secret contains the header the endpoint asks for. diff --git a/docs/en/solutions/Replace_the_platform_access_HTTPS_certificate_in_prdb_with_a_PEM_certificate_and_key_on_ACP_3.12.md b/docs/en/solutions/Replace_the_platform_access_HTTPS_certificate_in_prdb_with_a_PEM_certificate_and_key_on_ACP_3.12.md new file mode 100644 index 000000000..50c9204a3 --- /dev/null +++ b/docs/en/solutions/Replace_the_platform_access_HTTPS_certificate_in_prdb_with_a_PEM_certificate_and_key_on_ACP_3.12.md @@ -0,0 +1,110 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.12.x +--- + +# Replace the platform access HTTPS certificate in prdb with a PEM certificate and key on ACP 3.12 + +## Issue + +On ACP 3.12, the HTTPS certificate of the platform access address (the domain users open the console with) has to be replaced, for example because it is about to expire. The product documentation pages about certificates cover Ingress and load balancer certificates, not the certificate of the platform itself. + +The new certificate is often delivered as two files, such as `.key` and `.pem`, and there is no file named `.crt` or `.cert`, which makes it unclear which file goes where. + +## Root Cause + +On ACP 3.12 the platform access certificate is held in the `prdb` resource on the `global` cluster, in `spec.http.tls.cert` (certificate) and `spec.http.tls.key` (private key), both base64-encoded. Changing these fields is how the certificate is replaced. + +This procedure is for ACP 3.12. On ACP 4.x the platform access certificate is the `dex.tls` Secret in `cpaas-system`; follow "Rotate TLS Certs of Platform Access Addresses" in the product documentation instead. + +A `.pem` file is just a PEM-encoded file. When it starts with `-----BEGIN CERTIFICATE-----`, it is the certificate and is used in place of a `.cert`/`.crt` file. The `.key` file is the private key. + +## Resolution + +Run the steps on the first control plane node of the `global` cluster. + +1. Prepare the files. Copy the certificate (`.pem`) to `custom.cert` and the key to `custom.key`, then check that they belong together and cover the platform domain: + + ```bash + cp .pem custom.cert + cp .key custom.key + openssl x509 -in custom.cert -noout -subject -enddate -ext subjectAltName + openssl x509 -in custom.cert -noout -pubkey | openssl md5 + openssl pkey -in custom.key -pubout | openssl md5 + ``` + + Continue only if the two hashes are identical, `notAfter` is in the future, and the subject or subject alternative names include the platform access domain. If the issuer supplied intermediate certificates, append them to `custom.cert` after the server certificate. + +2. Back up the current resource: + + ```bash + kubectl get prdb -o yaml > /backup-prdb.yaml + ``` + +3. Generate the base64 values: + + ```bash + cat custom.cert | base64 -w 0; echo + cat custom.key | base64 -w 0; echo + ``` + +4. Write them into `prdb`. + + > **Warning:** Saving this change redeploys the platform access components with the new certificate. HTTPS access and logins can be interrupted until the rollout completes, and a wrong certificate or key makes the console unreachable over HTTPS. Do this in a maintenance window and only after step 1 succeeded. To roll back, edit `prdb` again and put back the `spec.http.tls.cert` and `spec.http.tls.key` values saved in `/backup-prdb.yaml`. + + ```bash + kubectl edit prdb + ``` + + Set `spec.http.tls.cert` to the first value and `spec.http.tls.key` to the second. Add either key if it is missing. + +5. Wait until the platform releases in `cpaas-system` are ready again (note the namespace spelling; a typo such as `cpass-system` returns `No resources found`): + + ```bash + kubectl get appreleases -n cpaas-system + ``` + +6. Check that the `dex` pod was recreated after the change. If its age shows it was not, delete it so that it restarts with the new certificate. + + > **Warning:** Users cannot log in while the `dex` pod restarts, usually for less than a minute. Delete it only if it was not recreated automatically. The Deployment recreates it, so no rollback is needed. + + ```bash + kubectl get po -A | grep dex + kubectl -n cpaas-system delete pod + ``` + +7. Log in to the platform and check the certificate the browser receives. + +## Diagnostic Steps + +1. Check the certificate currently served on the platform address: + + ```bash + echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -subject -issuer -dates + ``` + +2. Check what `prdb` holds and compare it with the served certificate: + + ```bash + kubectl get prdb -o jsonpath='{.items[0].spec.http.tls.cert}' | base64 -d | openssl x509 -noout -subject -dates + ``` + + After the replacement both commands show the new subject and dates. + +3. Identify an unlabeled file by its first line: `-----BEGIN CERTIFICATE-----` is a certificate, `-----BEGIN PRIVATE KEY-----` or `-----BEGIN RSA PRIVATE KEY-----` is a key: + + ```bash + head -1 .pem .key + ``` + +4. If the platform releases do not become ready after the edit, inspect the one that is not ready: + + ```bash + kubectl get appreleases -n cpaas-system + kubectl describe apprelease -n cpaas-system + ``` diff --git a/docs/en/solutions/Service_mesh_history_is_not_visible_after_switching_the_Elasticsearch_integration_to_another_cluster_on_ACP.md b/docs/en/solutions/Service_mesh_history_is_not_visible_after_switching_the_Elasticsearch_integration_to_another_cluster_on_ACP.md new file mode 100644 index 000000000..9af17cea5 --- /dev/null +++ b/docs/en/solutions/Service_mesh_history_is_not_visible_after_switching_the_Elasticsearch_integration_to_another_cluster_on_ACP.md @@ -0,0 +1,41 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Service Mesh +ProductsVersion: [] +--- + +# Service mesh history is not visible after switching the Elasticsearch integration to another cluster on ACP + +## Issue + +A service mesh stores its data in the platform Elasticsearch of the `global` cluster, and it has to move to the Elasticsearch of another cluster. Administrators ask whether the setting can be changed in place, and after the change they find that the data recorded before the switch no longer appears in the mesh views. + +The mesh's **Update Elasticsearch Integration Configuration** dialog shows **Source** set to **Platform** (the alternative is **External**), **Cluster** set to `global`, and an automatically filled **Access Address** that ends in `/es_proxy`. + +## Root Cause + +The Elasticsearch integration of a service mesh can be changed directly: selecting another cluster with **Source** = **Platform** points the mesh at that cluster's platform Elasticsearch, and the access address is generated automatically. + +The switch does not migrate data. Historical data stays in the Elasticsearch of the original cluster. After the switch, the mesh creates new indices in the new Elasticsearch, writes new data there, and queries only those new indices. Data recorded before the switch is therefore not visible from the mesh, although it still exists in the original Elasticsearch. + +## Resolution + +1. Open the service mesh and its **Update Elasticsearch Integration Configuration** dialog. + +2. Keep **Source** as **Platform**, select the target cluster in **Cluster**, and check that **Access Address** is regenerated for that cluster (it still ends in `/es_proxy`). + +3. Click **Update**. + + > **Warning:** After the update, the mesh writes to and reads from the new Elasticsearch only. Data recorded before the switch is not copied and is no longer shown in the mesh, although it remains in the original Elasticsearch. Plan the switch at a time when the loss of history in the mesh views is acceptable, and keep the original Elasticsearch until its data is no longer needed. To roll back, open the same dialog, select the original cluster again and click **Update**. Data written to the new cluster in the meantime stays there and is not copied back. + +4. Generate some traffic in the mesh and confirm that new data appears in the mesh views. + +## Diagnostic Steps + +1. Reopen **Update Elasticsearch Integration Configuration** and check that **Cluster** shows the new cluster and **Access Address** points to it. + +2. If the new data appears but the old data does not, this is the expected result of the switch, not a failure: the old data is still in the original cluster's Elasticsearch. + +3. If no new data appears either, check that the platform Elasticsearch of the target cluster is installed and healthy before suspecting the mesh configuration. diff --git a/docs/en/solutions/Storage_navigation_entries_added_by_plugins_or_Operators_cannot_be_hidden_with_custom-navconfig-platform_on_ACP.md b/docs/en/solutions/Storage_navigation_entries_added_by_plugins_or_Operators_cannot_be_hidden_with_custom-navconfig-platform_on_ACP.md new file mode 100644 index 000000000..f4d3a2768 --- /dev/null +++ b/docs/en/solutions/Storage_navigation_entries_added_by_plugins_or_Operators_cannot_be_hidden_with_custom-navconfig-platform_on_ACP.md @@ -0,0 +1,64 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.3.x +--- + +# Storage navigation entries added by plugins or Operators cannot be hidden with custom-navconfig-platform on ACP + +## Issue + +On ACP 4.3, an administrator wants to hide some entries under **Storage** in the left navigation of the **Administrator** view, such as **Distributed Storage**, **Local Storage** and **Object Storage**. Following "Customizing the Left Navigation" in the product documentation, `"hidden": true` is added for those entries in the `buildIn` data of the ConfigMap `custom-navconfig-platform` (namespace `cpaas-system`, `global` cluster), and the browser is refreshed. The entries are still shown. + +Hiding the whole `storage_management` group, or the documented built-in children such as `storageclasses`, works as expected. + +## Root Cause + +`custom-navconfig-platform` only customizes the **built-in** navigation. The built-in children of `storage_management` are `storageclasses` (StorageClasses), `nav_pv` (PersistentVolumes), `object_storage_class` (Object StorageClass) and `bucket` (Buckets). Keys that do not match a built-in item have no effect. + +The **Distributed Storage**, **Local Storage** and **Object Storage** entries are not built-in items: they are added to the navigation when the corresponding storage plugins or Operators are installed. There is no separate switch to hide them, so no key in `custom-navconfig-platform` removes them. + +## Resolution + +1. Decide whether the storage capability behind the entry is used. An entry contributed by a plugin or Operator disappears only when that plugin or Operator is uninstalled. + +2. If the capability is not needed, uninstall the corresponding plugin or Operator from **Administrator** > **Marketplace** (**Cluster Plugins** or **OperatorHub**, depending on how it was installed). Its navigation entry is removed with it. + + > **Warning:** Uninstalling a storage plugin or Operator removes the storage service it provides. Workloads that use volumes or buckets from it lose access to that storage, and data held only there can become unavailable. Uninstall it only after confirming that no PersistentVolumeClaim, StorageClass or bucket in any cluster still depends on it. Reinstalling the plugin or Operator restores the entry and the service, but not data that was removed with it. + +3. If the capability is in use, keep the plugin or Operator installed: its entry cannot be hidden. + +4. For the built-in items, hiding keeps working as documented. For example, to hide the PersistentVolumes entry: + + ```yaml + apiVersion: v1 + kind: ConfigMap + metadata: + name: custom-navconfig-platform + namespace: cpaas-system + data: + buildIn: | + { + "nav_pv": { + "hidden": true + } + } + ``` + + Refresh the browser after editing. + +## Diagnostic Steps + +1. Check what is currently configured. The ConfigMap must be in `cpaas-system` on the `global` cluster: + + ```bash + kubectl -n cpaas-system get configmap custom-navconfig-platform -o yaml + kubectl -n cpaas-system get configmap | grep navconfig + ``` + +2. Compare the keys under `buildIn` with the built-in storage items `storage_management`, `storageclasses`, `nav_pv`, `object_storage_class` and `bucket`. Keys such as `distributed_storage` or `local_storage` are not built-in names and are ignored. + +3. Check whether the entry belongs to an installed storage plugin or Operator: it appears after that component is installed and is absent on clusters where it is not installed. Such an entry cannot be hidden through `custom-navconfig-platform`. diff --git a/docs/en/solutions/kube-ovn-pinger_CrashLoopBackOff_context_deadline_exceeded_on_bonded_NICs_due_to_Geneve_TX_checksum_offload_on_ACP.md b/docs/en/solutions/kube-ovn-pinger_CrashLoopBackOff_context_deadline_exceeded_on_bonded_NICs_due_to_Geneve_TX_checksum_offload_on_ACP.md new file mode 100644 index 000000000..cd52fe327 --- /dev/null +++ b/docs/en/solutions/kube-ovn-pinger_CrashLoopBackOff_context_deadline_exceeded_on_bonded_NICs_due_to_Geneve_TX_checksum_offload_on_ACP.md @@ -0,0 +1,96 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# kube-ovn-pinger CrashLoopBackOff context deadline exceeded on bonded NICs due to Geneve TX checksum offload on ACP + +## Issue + +Right after a workload cluster is created on physical machines whose network uses a bond, every `kube-ovn-pinger` pod in `kube-system` is `0/1` `CrashLoopBackOff`. Other pods that talk to the API server, such as `kube-prometheus-exporter-kube-state` and `vm-operator` in `cpaas-system`, can crash-loop as well. + +The `kube-ovn-pinger` log shows that the pod cannot reach the API server through the `kubernetes` Service ClusterIP: + +```text +config.go:162] failed to get self pod kube-system/kube-ovn-pinger-: Get "https://:443/api/v1/namespaces/kube-system/pods/kube-ovn-pinger-?timeout=15s": context deadline exceeded (Client.Timeout exceeded while awaiting headers) +"failed to parse config" err=... context deadline exceeded +``` + +`` is the ClusterIP of the `default/kubernetes` Service (`443/TCP`). The nodes themselves can reach each other. + +## Root Cause + +Kube-OVN carries pod traffic between nodes in Geneve tunnels (UDP `6081`). With TX checksum offload enabled, the kernel leaves the checksum of the encapsulated packets for the NIC to fill in. On some physical NICs, the checksum computed after offload for Geneve-encapsulated packets is wrong, so the receiving side drops the packets. + +Pod-to-API-server traffic through the `kubernetes` Service crosses that path, so it times out, and `kube-ovn-pinger` exits because it cannot load its own pod. Host-to-host traffic is not encapsulated and keeps working, which is why the nodes look healthy. + +A bond is a virtual interface: the offload is actually performed by its slave (member) NICs, so the setting has to be changed on every slave NIC, not only on the bond. + +The permanent fix is a NIC firmware update from the NIC vendor. Disabling TX offload is the workaround. + +## Resolution + +1. On each affected node, list the slave NICs of the bond: + + ```bash + cat /proc/net/bonding/ | grep 'Slave Interface' + ``` + +2. Disable TX offload on every slave NIC. + + > **Warning:** Changing NIC offload settings can briefly disturb traffic on that NIC and moves checksum work to the CPU, which increases CPU usage under heavy traffic. Apply it only when the checksum-offload signature in the Diagnostic Steps is confirmed, and on production clusters apply it one node at a time. To roll back, run `ethtool -K tx on` and remove the persistent setting from step 3. + + ```bash + ethtool -K tx off + ethtool -k | grep -E 'tx-checksumming|tx-checksum-ip-generic' + ``` + + Expected: `tx-checksumming: off`. + +3. Make the setting persistent across reboots. On systems that use `network-scripts`, add the option to the configuration file of each slave NIC: + + ```bash + echo 'ETHTOOL_OPTS="-K tx off"' >> /etc/sysconfig/network-scripts/ifcfg- + ``` + + On systems managed by NetworkManager without `network-scripts`, use the equivalent persistent setting of that tool. + +4. Delete the crash-looping pods so they restart immediately instead of waiting for the back-off: + + ```bash + kubectl -n kube-system delete pod -l app=kube-ovn-pinger + ``` + + Confirm they become `1/1 Running`. + +5. Contact the NIC vendor for a firmware update that fixes Geneve checksum offload. After the firmware is updated, TX offload can be re-enabled and verified. + +## Diagnostic Steps + +1. Check whether the failure is limited to pod-to-Service traffic. From the node, the API server answers; from inside the pod network, the ClusterIP times out: + + ```bash + kubectl get svc -n default kubernetes + curl -k -m 5 https://:6443/healthz # on the node: ok + ``` + + If the node cannot reach `:6443` either, the problem is in the host network (firewall, routing), not in the overlay. + +2. Check the offload state of the bond's slave NICs. `tx-checksumming: on` on a slave NIC of the encapsulation path matches this issue; `off [fixed]` rules it out: + + ```bash + ethtool -k | grep -E 'tx-checksumming|tx-checksum-ip-generic' + ``` + +3. Capture Geneve traffic on the receiving node. Packets arriving with bad checksums, or not arriving at all while the sender transmits, point at the offload: + + ```bash + tcpdump -i -nn -vv udp port 6081 + ``` + + A capture taken on the sending node shows bad checksums whenever offload is on, because `tcpdump` sees the packet before the NIC fills in the checksum. Only a capture on the receiving side is evidence. + +4. If disabling TX offload on all slave NICs makes `kube-ovn-pinger` recover, the cause is confirmed. From 270993540e88bc159662dc2d56c910ea1499762b Mon Sep 17 00:00:00 2001 From: Komh Date: Tue, 10 Nov 2026 16:17:03 +0800 Subject: [PATCH 03/25] docs(solutions): add 9 and update 2 troubleshooting articles --- ...lickhouse-ca-certificates_Secret_on_ACP.md | 99 ++++++++++++ ...cpc-exec_feature_gate_is_enabled_on_ACP.md | 45 ++++++ ...Terminating_Due_to_Lingering_Finalizers.md | 32 ++++ ...Size_is_below_the_requested_size_on_ACP.md | 153 ++++++++++++++++++ ...AdvancedCronJob_stops_triggering_on_ACP.md | 99 ++++++++++++ ...iskby-id_paths_in_TopolvmCluster_on_ACP.md | 115 +++++++++++++ ...cannot_finish_on_a_large_cluster_on_ACP.md | 85 ++++++++++ ...ecause_of_hosts.toml_entry_order_on_ACP.md | 92 +++++++++++ .../nacos/How_to_Deploy_Nacos_2.2.md | 31 ++++ ...he_on_the_standby_global_cluster_on_ACP.md | 91 +++++++++++ ...uth_does_not_match_node-exporter_on_ACP.md | 84 ++++++++++ 11 files changed, 926 insertions(+) create mode 100644 docs/en/solutions/ClickHouse_log_storage_fails_on_HTTPS_S3_with_a_self-signed_CA_without_the_clickhouse-ca-certificates_Secret_on_ACP.md create mode 100644 docs/en/solutions/EXEC_and_Debug_entries_missing_from_Pod_and_application_pages_until_the_cpc-exec_feature_gate_is_enabled_on_ACP.md create mode 100644 docs/en/solutions/PVC_expansion_stuck_in_Resizing_because_the_Trident_backend_limitVolumeSize_is_below_the_requested_size_on_ACP.md create mode 100644 docs/en/solutions/Scheduled_etcd_backups_stop_after_an_upgrade_because_the_etcd-backup-default_AdvancedCronJob_stops_triggering_on_ACP.md create mode 100644 docs/en/solutions/TopoLVM_storage_breaks_when_VM_disk_names_change_use_devdiskby-id_paths_in_TopolvmCluster_on_ACP.md create mode 100644 docs/en/solutions/base_AppRelease_stays_failed_because_the_upgrade-sysctl-config_BroadcastJob_cannot_finish_on_a_large_cluster_on_ACP.md create mode 100644 docs/en/solutions/crictl_pull_fails_with_unexpected_commit_digest_while_ctr_pull_works_because_of_hosts.toml_entry_order_on_ACP.md create mode 100644 docs/en/solutions/etcd-sync_Pod_stuck_in_ContainerCreating_with_failed_to_sync_secret_cache_on_the_standby_global_cluster_on_ACP.md create mode 100644 docs/en/solutions/vmagent_fails_to_scrape_node-exporter_9100_metrics_because_its_basicAuth_does_not_match_node-exporter_on_ACP.md diff --git a/docs/en/solutions/ClickHouse_log_storage_fails_on_HTTPS_S3_with_a_self-signed_CA_without_the_clickhouse-ca-certificates_Secret_on_ACP.md b/docs/en/solutions/ClickHouse_log_storage_fails_on_HTTPS_S3_with_a_self-signed_CA_without_the_clickhouse-ca-certificates_Secret_on_ACP.md new file mode 100644 index 000000000..0dff06764 --- /dev/null +++ b/docs/en/solutions/ClickHouse_log_storage_fails_on_HTTPS_S3_with_a_self-signed_CA_without_the_clickhouse-ca-certificates_Secret_on_ACP.md @@ -0,0 +1,99 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.3.x +--- + +# ClickHouse log storage fails on HTTPS S3 with a self-signed CA without the clickhouse-ca-certificates Secret on ACP + +## Issue + +The ClickHouse log storage plugin is installed (from the console or from YAML) with S3 object storage, and the S3 endpoint is served over HTTPS with a certificate signed by a private or self-signed CA. The installation does not complete because ClickHouse in the ClickHouseInstallation (CHI) `cpaas-clickhouse` in `cpaas-system` cannot start. The container stdout may not show the reason. The certificate verification error for the S3 endpoint is written to `clickhouse-server.err.log` inside the ClickHouse container. + +## Root Cause + +ClickHouse verifies the TLS certificate of the S3 endpoint against the CA bundle at `/etc/ssl/certs/ca-certificates.crt` in its container. The default bundle contains only public CAs, so a certificate from a private or self-signed CA fails verification and ClickHouse cannot use the S3 disk. + +The plugin chart supports a custom CA bundle. During rendering it looks for the Secret `clickhouse-ca-certificates` (key `ca-certificates.crt`) in `cpaas-system`: + +- If the Secret exists, the chart mounts it into the ClickHouse container at `/etc/ssl/certs/ca-certificates.crt`. +- If the Secret does not exist, nothing is mounted, and the S3 certificate verification fails. + +This lookup is part of the plugin starting with ACP 4.3.2. The Secret is not created automatically and must be created by hand. The mounted file **replaces** the whole system CA bundle of the container, so it must hold the public CAs as well as the S3 CA. + +## Resolution + +1. Build a complete CA bundle: the system CA bundle followed by the CA certificate(s) of the S3 endpoint. For example, on a Linux host that has a current system bundle at the same path: + + ```bash + cat /etc/ssl/certs/ca-certificates.crt s3-ca.crt > ca-certificates.crt + ``` + + Append each additional CA certificate in PEM format, one after another. + +2. Create the Secret in `cpaas-system`: + + ```bash + kubectl -n cpaas-system create secret generic clickhouse-ca-certificates \ + --from-file=ca-certificates.crt=./ca-certificates.crt + ``` + + The same Secret as YAML: + + ```yaml + apiVersion: v1 + kind: Secret + metadata: + name: clickhouse-ca-certificates + namespace: cpaas-system + type: Opaque + stringData: + ca-certificates.crt: | + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- + ``` + +3. Redeploy the ClickHouse log storage plugin so that the chart renders again and picks up the Secret. Creating the Secret alone does not change a CHI that has already been rendered. + + > **Warning:** Redeploying the log storage plugin restarts ClickHouse, and log, event and audit writes and queries are interrupted until it is running again. Do this only after the diagnostic steps show the S3 certificate error and a CHI without the CA mount. Rollback: delete the Secret and redeploy the plugin again, which brings back the previous rendering. + +4. Check that the ClickHouse Pods become `Running` and that the plugin finishes installing: + + ```bash + kubectl -n cpaas-system get pods | grep -i clickhouse + ``` + +## Diagnostic Steps + +1. Find the ClickHouse Pods and read the server error log inside the container, not only stdout: + + ```bash + kubectl -n cpaas-system get pods | grep -i clickhouse + kubectl -n cpaas-system exec -c clickhouse -- \ + sh -c 'tail -n 200 /var/log/clickhouse-server/clickhouse-server.err.log' + ``` + + Expected for this issue: TLS or certificate verification errors for the S3 endpoint. If the log path differs, look for `clickhouse-server.err.log` under the log directory configured for the container. + +2. Check whether the CA bundle is mounted: + + ```bash + kubectl -n cpaas-system get secret clickhouse-ca-certificates + kubectl -n cpaas-system get chi cpaas-clickhouse -o yaml | grep -n -B2 -A4 clickhouse-ca-certificates + ``` + + Expected for this issue: the Secret is `NotFound` and the CHI has no volume or mount for `clickhouse-ca-certificates`. After the fix, both exist and the container has the bundle at `/etc/ssl/certs/ca-certificates.crt`. + +3. Check that the S3 endpoint certificate chains to the CA you put in the bundle: + + ```bash + openssl s_client -connect : -servername -CAfile ca-certificates.crt /dev/null | grep -E 'Verify return code' + ``` + + Expected after the bundle is correct: `Verify return code: 0 (ok)`. + +4. Rule out look-alikes. When ClickHouse logs that it cannot resolve the Keeper host, or the S3 endpoint is plain HTTP, or the endpoint uses a certificate from a public CA, the CA bundle is not the cause. diff --git a/docs/en/solutions/EXEC_and_Debug_entries_missing_from_Pod_and_application_pages_until_the_cpc-exec_feature_gate_is_enabled_on_ACP.md b/docs/en/solutions/EXEC_and_Debug_entries_missing_from_Pod_and_application_pages_until_the_cpc-exec_feature_gate_is_enabled_on_ACP.md new file mode 100644 index 000000000..423fc96e6 --- /dev/null +++ b/docs/en/solutions/EXEC_and_Debug_entries_missing_from_Pod_and_application_pages_until_the_cpc-exec_feature_gate_is_enabled_on_ACP.md @@ -0,0 +1,45 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.4.x +--- + +# EXEC and Debug entries missing from Pod and application pages until the cpc-exec feature gate is enabled on ACP + +## Issue + +The product documentation for containers describes how to open a shell in a container (**EXEC**) and how to debug a Pod with an ephemeral container (**Debug**) from the web console: **Container Platform** > **Workloads** > **Pods** > ⋮ > **Debug** or **EXEC**, or **EXEC** on a workload in the application details. In the console of an ACP 4.4.1 cluster, the ⋮ menu of the Pods and the application pages shows neither **EXEC** nor **Debug**, and the container documentation does not mention any switch for them. + +## Root Cause + +**EXEC** and **Debug** in the console are Alpha features that are controlled by the feature gate `cpc-exec`. The console shows the **EXEC** and **Debug** entries on application-related pages only when this gate is enabled. While the gate is off, the entries are hidden. + +A second Alpha gate, `cpc-exec-file`, controls file transfer inside **EXEC** and **Debug**. When it is enabled, a file transfer entry appears in the EXEC and Debug windows. + +Because both are Alpha features, they are also hidden when Alpha features are turned off for the whole platform in the **Stage Setting** section of the feature gate page. + +## Resolution + +1. Open the feature gate page at `/console-platform/feature-gate`. For example, if the platform access address is `https://demo.example.com/console-portal/`, open `https://demo.example.com/console-platform/feature-gate`. + +2. Enable `cpc-exec`: + + - To enable it for one cluster, find the gate list titled with the display name of that cluster and turn on `cpc-exec` there. + - Check that the **Alpha Feature** switch in **Stage Setting** is on, and that `cpc-exec` is not turned off in **System Settings**, which disables a gate for all clusters. + +3. Optionally, enable `cpc-exec-file` in the same place to get file transfer inside EXEC and Debug. + +4. Reload the console and open **Container Platform** > **Workloads** > **Pods** > ⋮. The **EXEC** and **Debug** entries are now shown. + +The command line does not depend on these gates. Without the console entries, `kubectl exec -it -c -n -- /bin/sh` and `kubectl debug -it --image= --target= -n ` work as usual. + +## Diagnostic Steps + +1. On the feature gate page, look up `cpc-exec` and `cpc-exec-file`. Expected: both have the stage `Alpha`. For this issue `cpc-exec` is off for the cluster, or Alpha features are off in **Stage Setting**. + +2. After enabling the gate, check the ⋮ menu of a Pod in that cluster. If the entries are still missing, make sure that you enabled the gate in the list of the cluster you are viewing and not of another one, and reload the page. + +3. If **EXEC** or **Debug** is shown but the terminal does not open or the debug container does not initialize, the gate is working and the problem is elsewhere, for example in the web terminal component or in a security policy that blocks ephemeral containers. That is not covered by this article. diff --git a/docs/en/solutions/Namespace_Stuck_in_Terminating_Due_to_Lingering_Finalizers.md b/docs/en/solutions/Namespace_Stuck_in_Terminating_Due_to_Lingering_Finalizers.md index 768ffb054..594785a35 100644 --- a/docs/en/solutions/Namespace_Stuck_in_Terminating_Due_to_Lingering_Finalizers.md +++ b/docs/en/solutions/Namespace_Stuck_in_Terminating_Due_to_Lingering_Finalizers.md @@ -36,6 +36,8 @@ A namespace wedged in `Terminating` therefore means at least one finalizer is no - The controller is running but cannot complete its cleanup (a downstream API unreachable, a missing credential, a webhook timing out). - The object is a custom resource whose CRD itself has been deleted; without the CRD the controller is gone, leaving orphaned instances. +A related case on the ACP `global` cluster does not involve a finalizer on any object. When a namespace on `global` has the same name as a workload cluster that was created and later deleted, the namespace holds `OperatorView` objects. They are served from memory by `marketplace-apiserver` in `cpaas-system`, which does not accept DELETE for them. The namespace controller cannot empty the namespace, and the namespace condition `NamespaceDeletionContentFailure` reads `Failed to delete all resource types, 1 remaining: OperatorView resource does not support DELETE requests`. Editing or patching the namespace finalizer does not help. Once `marketplace-apiserver` is stopped, its APIService `v1alpha2.app.alauda.io` has no endpoints (`False (MissingEndpoints)`), and the deletion is then blocked by `NamespaceDeletionDiscoveryFailure` with `stale GroupVersion discovery: app.alauda.io/v1alpha2` until that APIService is removed. See Step 3 below. + ## Resolution The safe fix is to find the specific resources holding finalizers, identify the owning controller, and let that controller finish its job. Forcing the finalizer off should be reserved for the case where the controller is known to be gone and its residual state is confirmed to be safe to drop. @@ -68,6 +70,28 @@ kubectl patch / -n "$NS" \ Repeat until every lingering object is gone. The namespace then transitions out of `Terminating` within a few seconds. +Step 3 — on the `global` cluster, when the namespace condition names `OperatorView resource does not support DELETE requests` and the namespace belongs to a workload cluster of the same name that no longer exists: + +> **Warning:** While `marketplace-apiserver` is scaled to zero and its APIService is removed, the APIs it serves, including `OperatorView` and `app.alauda.io/v1alpha2`, are unavailable on the `global` cluster, and console pages that read it can fail. Do this only when the namespace is confirmed to be the leftover of a deleted cluster and the condition above is present. Save both objects first. Rollback: apply the saved APIService and scale `marketplace-apiserver` back to the recorded replica count. + +```bash +NS= +kubectl get ns "$NS" -o jsonpath='{range .status.conditions[*]}{.type}: {.message}{"\n"}{end}' +kubectl -n cpaas-system get deploy marketplace-apiserver -o jsonpath='{.spec.replicas}{"\n"}' +kubectl -n cpaas-system get deploy marketplace-apiserver -o yaml > /tmp/marketplace-apiserver.yaml +kubectl get apiservice v1alpha2.app.alauda.io -o yaml > /tmp/apiservice-v1alpha2.app.alauda.io.yaml + +# 1. stop the aggregated API server that serves OperatorView +kubectl -n cpaas-system scale deploy marketplace-apiserver --replicas=0 + +# 2. the namespace now reports stale discovery for app.alauda.io/v1alpha2; remove the APIService +kubectl get apiservice | grep -v True +kubectl delete apiservice v1alpha2.app.alauda.io +kubectl get ns "$NS" +``` + +When the namespace is gone, apply `/tmp/apiservice-v1alpha2.app.alauda.io.yaml` again if the APIService has not been recreated, and scale `marketplace-apiserver` back to the recorded replica count. Then check that `kubectl get apiservice v1alpha2.app.alauda.io` shows `True`. + ## Diagnostic Steps Before forcing anything, gather evidence so you can justify the decision to remove a finalizer: @@ -78,6 +102,14 @@ Inspect the namespace object itself for its own finalizer — the `kubernetes` f kubectl get ns scheduling -o yaml | grep -A3 finalizers ``` +Read the namespace conditions. `NamespaceDeletionContentFailure` with `OperatorView resource does not support DELETE requests`, or `NamespaceDeletionDiscoveryFailure` with `stale GroupVersion discovery: app.alauda.io/v1alpha2`, points to the `global` cluster case handled in Step 3, not to a finalizer: + +```bash +kubectl get ns scheduling -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}' +kubectl get operatorview -n scheduling +kubectl get apiservice | grep -v True +``` + Check whether any CRD has been deleted while instances still exist; orphaned CRs are a very common cause: ```bash diff --git a/docs/en/solutions/PVC_expansion_stuck_in_Resizing_because_the_Trident_backend_limitVolumeSize_is_below_the_requested_size_on_ACP.md b/docs/en/solutions/PVC_expansion_stuck_in_Resizing_because_the_Trident_backend_limitVolumeSize_is_below_the_requested_size_on_ACP.md new file mode 100644 index 000000000..ad3850bb9 --- /dev/null +++ b/docs/en/solutions/PVC_expansion_stuck_in_Resizing_because_the_Trident_backend_limitVolumeSize_is_below_the_requested_size_on_ACP.md @@ -0,0 +1,153 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# PVC expansion stuck in Resizing because the Trident backend limitVolumeSize is below the requested size on ACP + +## Issue + +A PersistentVolumeClaim (PVC) on a NAS StorageClass served by NetApp Trident (driver `ontap-nas`) is expanded, for example from `5Gi` to `500Gi`, and the expansion never completes: + +- `spec.resources.requests.storage` of the PVC shows the new size (`500Gi`), but `status.capacity.storage` and the bound PV still show the old size (`5Gi`). +- The PVC carries the condition `type: Resizing` and does not leave it. +- Trying to correct the size in the console is rejected because the new capacity must be larger than the value already in `spec` (for a `500Gi` request, the console asks for at least `501Gi`). The console cannot shrink the request back. +- The Trident controller log shows the resize being refused, for example: + +```text +unsupported capacity range ... requested size: 536870912000 > the size limit: 214748364800 +``` + +The StorageClass already has `allowVolumeExpansion: true`, and there is no LimitRange or ResourceQuota in the namespace that explains the rejection. + +## Root Cause + +Trident backends can cap the size of a single volume with the backend option `limitVolumeSize`. Trident checks every create and resize request against that cap and refuses any request above it. When a backend (for example a `TridentBackendConfig` deployed through the CSI Helm chart) is configured with `limitVolumeSize: 200g`, every PVC on that backend is limited to about 200 GiB. An expansion to `500Gi` (536870912000 bytes) is above the cap (214748364800 bytes), so the storage side never grows the volume. + +Kubernetes has already accepted the larger `spec.resources.requests.storage` at this point. Kubernetes does not allow a PVC request to be lowered, so the PVC stays in `Resizing` with a request it can never reach, and the console only offers sizes above the current request. + +## Resolution + +Decide first which size the volume really needs: + +- **The needed size fits under the backend limit** (for example `100Gi` under a `200g` cap): cancel the failed expansion and expand again within the limit, using option A or option B below. +- **The needed size is above the backend limit**: the cap is a setting of the Trident backend. Changing `limitVolumeSize` is a storage administration decision that the owner of the backend and the NAS must make. After the backend limit has been raised, the pending resize can be retried. Do not edit the Trident controller Deployment. The limit is not a controller argument. + +### Option A: cancel the failed expansion and rebind the PVC to its PV + +This follows the product procedure **Recover From PVC Expansion Failure** and keeps the data on the original volume. + +> **Warning:** This procedure deletes the PVC object. If the PV reclaim policy is still `Delete` when the PVC is deleted, the volume and its data are deleted with it. Do it only after step 1 shows `Retain` on the PV. Stop the workload that mounts the PVC first. Rollback: as long as the PV is `Retain`, the data stays on the PV and can be bound again by a PVC that names it in `volumeName`. + +1. Record the PV, its reclaim policy, and the current sizes, and save both objects: + + ```bash + PV=$(kubectl -n get pvc -o jsonpath='{.spec.volumeName}') + kubectl get pv "$PV" -o jsonpath='{.spec.persistentVolumeReclaimPolicy} {.spec.capacity.storage}{"\n"}' + kubectl -n get pvc -o yaml > /tmp/.yaml + kubectl get pv "$PV" -o yaml > /tmp/"$PV".yaml + kubectl patch pv "$PV" -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' + ``` + +2. Scale the workload that uses the PVC to zero, then delete the PVC: + + ```bash + kubectl -n scale / --replicas=0 + kubectl -n delete pvc + ``` + +3. Remove the old claim reference so the PV becomes `Available`: + + ```bash + kubectl patch pv "$PV" --type json -p '[{"op":"remove","path":"/spec/claimRef"}]' + kubectl get pv "$PV" + ``` + +4. Recreate the PVC with the same name, StorageClass and access modes, `volumeName` set to the original PV, and a request that the PV can satisfy. The PV still has its old capacity (`5Gi` in the example), so the new request must not be larger than that capacity, or the PVC cannot bind: + + ```yaml + apiVersion: v1 + kind: PersistentVolumeClaim + metadata: + name: + namespace: + spec: + accessModes: + - + storageClassName: + volumeName: + resources: + requests: + storage: + ``` + +5. Restore the original reclaim policy of the PV (the value recorded in step 1): + + ```bash + kubectl patch pv "$PV" -p '{"spec":{"persistentVolumeReclaimPolicy":""}}' + ``` + +6. Expand the PVC again, this time to a size at or below the backend `limitVolumeSize`, and start the workload: + + ```bash + kubectl -n patch pvc -p '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}' + kubectl -n scale / --replicas= + ``` + +### Option B: copy the data to a new PVC + +Use this when you prefer not to touch the PV binding. + +> **Warning:** The workload must be stopped while the data is copied, and the old PVC must not be deleted until the copy is verified. Rollback: keep the old PVC and point the workload back at it. + +1. Stop the workload (scale its Deployment, StatefulSet or ReplicaSet to `0`). +2. Create a new PVC of the needed size (at or below the backend limit) on the same StorageClass. +3. Run a temporary Pod that mounts both PVCs and copy the data, for example with `cp -a` or `rsync -a`. +4. Verify the copy, point the workload at the new PVC (or recreate the old PVC name on the new volume), and start the workload. +5. Delete the old PVC only after the workload runs correctly on the new one. + +## Diagnostic Steps + +1. Compare the requested and the actual size of the PVC. A larger `spec` than `status` together with a `Resizing` condition means that the resize was accepted by Kubernetes but not done by the storage: + + ```bash + kubectl -n get pvc \ + -o jsonpath='spec={.spec.resources.requests.storage} status={.status.capacity.storage}{"\n"}{.status.conditions}{"\n"}' + kubectl get pv -o jsonpath='{.spec.capacity.storage}{"\n"}' + ``` + + Expected for this issue: `spec=500Gi status=5Gi`, a `Resizing` condition, and a PV that still shows `5Gi`. + +2. Read the PVC events. An expansion that is only slow keeps showing progress. This issue shows a failed resize: + + ```bash + kubectl -n describe pvc + ``` + +3. Find the rejection in the Trident controller log: + + ```bash + kubectl -n trident get pods + kubectl -n trident logs --all-containers --tail=500 | grep -iE 'size limit|unsupported capacity range' + ``` + + Expected: `requested size: > the size limit: `. Divide the byte values by 1073741824 to get GiB. + +4. Read the limit on the backend: + + ```bash + kubectl -n trident get tridentbackendconfig + kubectl -n trident get tridentbackendconfig -o jsonpath='{.spec.limitVolumeSize}{"\n"}' + ``` + + Expected: a value such as `200g` that is smaller than the requested size. When the backend was deployed with Helm, the same value is in the release values (`helm -n trident get values `), and that is where it has to be changed so that the next upgrade does not reset it. + +5. Rule out look-alikes. When the StorageClass has no `allowVolumeExpansion: true`, the expansion is refused right away instead of hanging in `Resizing`. A namespace LimitRange or ResourceQuota rejects the PVC edit itself, with an admission error: + + ```bash + kubectl get storageclass -o jsonpath='{.allowVolumeExpansion}{"\n"}' + kubectl -n get limitrange,resourcequota + ``` diff --git a/docs/en/solutions/Scheduled_etcd_backups_stop_after_an_upgrade_because_the_etcd-backup-default_AdvancedCronJob_stops_triggering_on_ACP.md b/docs/en/solutions/Scheduled_etcd_backups_stop_after_an_upgrade_because_the_etcd-backup-default_AdvancedCronJob_stops_triggering_on_ACP.md new file mode 100644 index 000000000..0fd297aa7 --- /dev/null +++ b/docs/en/solutions/Scheduled_etcd_backups_stop_after_an_upgrade_because_the_etcd-backup-default_AdvancedCronJob_stops_triggering_on_ACP.md @@ -0,0 +1,99 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.3.x +--- + +# Scheduled etcd backups stop after an upgrade because the etcd-backup-default AdvancedCronJob stops triggering on ACP + +## Issue + +A routine check of a cluster that was upgraded shows that the last etcd backup is several days old: + +- The newest snapshot file in the backup directory on the control plane nodes is from a date shortly after the upgrade, and nothing newer has been written. +- `kubectl get broadcastjobs -n cpaas-system` lists `etcd-backup-default-` BroadcastJobs only up to that date. No newer ones are created, even though the older ones completed (`DESIRED 3`, `SUCCEEDED 3`). The last one can also show one failed node. +- The `etcd-backup` feature is enabled, but the cluster has no etcd backup page in the console. + +## Root Cause + +Scheduled etcd backups are driven by the AdvancedCronJob (`acj`) `etcd-backup-default` in `cpaas-system`. At each scheduled time it creates a BroadcastJob (`bcj`) named `etcd-backup-default-` that runs the backup on every control plane node. The AdvancedCronJob works out its next trigger time from the BroadcastJobs it owns. + +During an upgrade, some BroadcastJobs can be created from the AdvancedCronJob without the schedule annotation. The AdvancedCronJob then fails to compute its next trigger time, and from then on it never triggers again. The object still exists and looks healthy, so the problem only shows up as backups that stop. This happens only occasionally. It is a known defect, and recreating the AdvancedCronJob clears it. + +The missing etcd backup page is a separate issue. The page is provided by the **Alauda Container Platform Cluster Enhancer** plugin, so a cluster where this plugin is not installed shows no backup page even though backups run in the background. + +## Resolution + +1. If the cluster has no etcd backup page, download the **Alauda Container Platform Cluster Enhancer** package that matches the platform version, upload it with `violet push`, and install it on the cluster from **Administrator** > **Marketplace** > **Cluster Plugins**. This restores the page. It does not by itself restart the stopped schedule. + +2. Save the AdvancedCronJob and record its schedule: + + ```bash + kubectl -n cpaas-system get acj etcd-backup-default -o yaml > /tmp/acj-etcd-backup-default.yaml + grep -n -E 'schedule|paused' /tmp/acj-etcd-backup-default.yaml + ``` + +3. Prepare a copy that can be applied again. In `/tmp/acj-etcd-backup-default.recreate.yaml`, remove `metadata.resourceVersion`, `metadata.uid`, `metadata.creationTimestamp`, `metadata.managedFields` and the whole `status` block: + + ```bash + cp /tmp/acj-etcd-backup-default.yaml /tmp/acj-etcd-backup-default.recreate.yaml + vi /tmp/acj-etcd-backup-default.recreate.yaml + ``` + +4. Delete the AdvancedCronJob and apply the saved copy. + + > **Warning:** While the AdvancedCronJob is deleted, the cluster has no scheduled etcd backup. Do this only when the diagnostic steps confirm that `etcd-backup-default` exists but has created no BroadcastJob since the last good backup, and only after `/tmp/acj-etcd-backup-default.yaml` has been saved. Rollback: `kubectl apply -f /tmp/acj-etcd-backup-default.recreate.yaml`. + + ```bash + kubectl -n cpaas-system delete acj etcd-backup-default + kubectl apply -f /tmp/acj-etcd-backup-default.recreate.yaml + kubectl -n cpaas-system get acj etcd-backup-default + ``` + +5. Verify that the schedule fires. Either wait for the next scheduled time, or set the schedule temporarily a few minutes ahead, confirm that a new BroadcastJob runs, and then set the schedule back to the value recorded in step 2: + + ```bash + kubectl -n cpaas-system edit acj etcd-backup-default # temporary schedule, then restore it + kubectl -n cpaas-system get broadcastjobs | grep etcd-backup-default + ``` + + A new `etcd-backup-default-` BroadcastJob should appear and reach `SUCCEEDED` equal to `DESIRED`, and a new snapshot file should appear on each control plane node. + +6. Repeat the procedure on every cluster that shows the same symptom. + +## Diagnostic Steps + +1. Check the dates of the snapshot files on each control plane node. Use the backup path configured for the cluster: + + ```bash + ls -lt | head + ``` + + Expected for this issue: the newest file is several days old. + +2. List the backup BroadcastJobs and compare their ages with the schedule: + + ```bash + kubectl get broadcastjobs -A | grep etcd-backup + ``` + + Expected for this issue: the newest `etcd-backup-default-` is as old as the newest snapshot, and no newer one exists. If new BroadcastJobs keep appearing but end in `FAILED`, the schedule is working and the backup run itself fails. That is a different problem: read the logs of the failed `etcd-backup` Pods instead. + +3. Confirm that the AdvancedCronJob exists and is not paused: + + ```bash + kubectl -n cpaas-system get acj etcd-backup-default -o jsonpath='{.spec.schedule} paused={.spec.paused}{"\n"}' + ``` + + Expected for this issue: the expected schedule and `paused=` empty or `false`. The object exists, but nothing has been created since the last good backup. + +4. When the backup page is missing, check in **Administrator** > **Marketplace** > **Cluster Plugins** whether **Alauda Container Platform Cluster Enhancer** is installed on that cluster. On the global cluster, the installed plugins of a cluster can also be listed with: + + ```bash + kubectl get moduleinfo -l cpaas.io/cluster-name= | grep -i enhancer + ``` + + No output means that the plugin is not installed on that cluster. diff --git a/docs/en/solutions/TopoLVM_storage_breaks_when_VM_disk_names_change_use_devdiskby-id_paths_in_TopolvmCluster_on_ACP.md b/docs/en/solutions/TopoLVM_storage_breaks_when_VM_disk_names_change_use_devdiskby-id_paths_in_TopolvmCluster_on_ACP.md new file mode 100644 index 000000000..4ad3d38fa --- /dev/null +++ b/docs/en/solutions/TopoLVM_storage_breaks_when_VM_disk_names_change_use_devdiskby-id_paths_in_TopolvmCluster_on_ACP.md @@ -0,0 +1,115 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# TopoLVM storage breaks when VM disk names change: use /dev/disk/by-id paths in TopolvmCluster on ACP + +## Issue + +The cluster nodes are virtual machines, and TopoLVM (LVM storage) uses their virtual disks. When a VM is rebuilt, its disks can come back under different kernel names (for example `/dev/sdb` becomes `/dev/sdc`, or `/dev/vdb` becomes `/dev/vdc`), and TopoLVM storage on that node becomes abnormal. The question is whether disks can be added to TopoLVM by a stable path such as `/dev/disk/by-id/...` instead. + +## Root Cause + +The disk picker in the console fills the TopolvmCluster with kernel device names such as `/dev/sdb` or `/dev/vdb`. Kernel device names are assigned at boot in discovery order and are not guaranteed to stay the same when a VM is rebuilt or its disks are reattached. After a rename, `spec.storage.deviceClasses[].classes[].devices[].name` in the TopolvmCluster no longer points to the disk that holds the volume group. + +Using kernel names is a console behavior, not a backend limitation. The TopolvmCluster accepts a `/dev/disk/by-id/...` path in `devices[].name`, both when a TopolvmCluster is created and when an existing one is edited in YAML. + +Pick the right kind of stable path: + +- Use the `by-id` link that is derived from the disk serial number or WWN (prefixes `virtio-`, `wwn-`, `scsi-`, `nvme-`). +- Do not use `/dev/disk/by-path/...`. It follows the PCI slot and changes when the slot changes, so it does not solve this problem. +- A `by-id` link is only as stable as the disk serial. If a rebuild also changes the serial, the `by-id` link changes too. Keep the disk serial unchanged on the virtualization side. + +## Resolution + +### New TopolvmCluster + +Write the TopolvmCluster in YAML and set each `devices[].name` to the serial-based `/dev/disk/by-id/...` path of the disk instead of the kernel name. + +### Existing TopolvmCluster: switch one node to by-id paths + +The switch is safe only when the path points to the same disk that is already in the volume group, and only when the change replaces paths without adding or removing disks. + +> **Warning:** Do not add or remove disks in the same edit as the path change. When the number of devices changes, the volume-group shrink logic compares device paths as strings and can treat the existing physical volume (still known by its old kernel name) as a disk to remove. Do one node at a time. Save the TopolvmCluster first. Rollback: apply the saved YAML, which puts the kernel names back. + +1. Save the current TopolvmCluster: + + ```bash + kubectl -n nativestor-system get topolvmcluster + kubectl -n nativestor-system get topolvmcluster -o yaml > /tmp/topolvmcluster-.yaml + ``` + +2. On the node, confirm that the `by-id` link resolves to the disk that `pvs` reports for the volume group: + + ```bash + ls -l /dev/disk/by-id/ + readlink -f /dev/disk/by-id/ + pvs -o pv_name,vg_name,pv_uuid + ``` + + The output of `readlink -f` must be the same device as the `pv_name` of the volume group, for example `/dev/vdb`. + +3. Edit the TopolvmCluster and change only `devices[].name` of that node under `spec.storage.deviceClasses[].classes[]` from the kernel name to the `by-id` path: + + ```bash + kubectl -n nativestor-system edit topolvmcluster + ``` + +4. Wait for the node's `topolvm-prepare-vg` run to finish and check its log. With an unchanged number of devices, the shrink step is skipped with the message `no need shrink vg for disk name change`, and the existing volume group and logical volumes are not touched: + + ```bash + kubectl -n nativestor-system get pods -o wide | grep topolvm-prepare-vg + kubectl -n nativestor-system logs | grep -i shrink + ``` + + The `lvmd.yaml` in the node's lvmd configuration needs no change. It records only the volume group name, not device paths. + +5. Align the status with the spec. The TopolvmCluster status does not switch to the `by-id` path on its own. After `topolvm-prepare-vg` has finished, edit the node's ConfigMap in `nativestor-system` (named `lvmdconfig-`). In its `status.json`, set `successClasses[].deviceStates[].name` to the same `by-id` path as in the spec, and leave `serial` unchanged. The controller syncs this status into `TopolvmCluster.status`. + + ConfigMap names are truncated for long node names, so match the ConfigMap by its `node-name` annotation: + + ```bash + kubectl -n nativestor-system get cm -l topolvm/lvmdconfig=lvmdconfig \ + -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.node-name}{"\n"}{end}' + kubectl -n nativestor-system get cm -o yaml > /tmp/.yaml + kubectl -n nativestor-system edit cm + ``` + +6. Repeat for the next node. + +### Known side effect + +On ACP 4.3.3, the storage performance monitoring page queries metrics by kernel device name. After a disk is switched to a `by-id` path, storage keeps working, but the performance page may show no data for that disk. + +## Diagnostic Steps + +1. See which device names the TopolvmCluster uses: + + ```bash + kubectl -n nativestor-system get topolvmcluster -o yaml | grep -n -A3 'devices:' + ``` + + Kernel names such as `/dev/sdb` or `/dev/vdb` mean that the node is exposed to renaming. + +2. On an affected node, compare the configured name with the disk that LVM actually uses: + + ```bash + pvs -o pv_name,vg_name,pv_uuid + ls -l /dev/disk/by-id/ | grep -v part + ``` + + If the `pv_name` of the volume group differs from the name in the TopolvmCluster, the disk was renamed. + +3. Check that the `by-id` link you plan to use is based on the serial and stays the same across a rebuild. Compare the link before and after a test rebuild, or check the serial that the virtualization platform assigns to the disk: + + ```bash + lsblk -o NAME,SERIAL,WWN + ``` + + An empty `SERIAL` and `WWN` means that the disk has no serial-based `by-id` link. Set a serial on the virtualization side first. diff --git a/docs/en/solutions/base_AppRelease_stays_failed_because_the_upgrade-sysctl-config_BroadcastJob_cannot_finish_on_a_large_cluster_on_ACP.md b/docs/en/solutions/base_AppRelease_stays_failed_because_the_upgrade-sysctl-config_BroadcastJob_cannot_finish_on_a_large_cluster_on_ACP.md new file mode 100644 index 000000000..4d8a54d75 --- /dev/null +++ b/docs/en/solutions/base_AppRelease_stays_failed_because_the_upgrade-sysctl-config_BroadcastJob_cannot_finish_on_a_large_cluster_on_ACP.md @@ -0,0 +1,85 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# base AppRelease stays failed because the upgrade-sysctl-config BroadcastJob cannot finish on a large cluster on ACP + +## Issue + +On a cluster with many nodes (more than a hundred), the `base` AppRelease (`ar`, also shown as `ars base`) in `cpaas-system` stays in a failed state. In `cpaas-system`, the BroadcastJob `upgrade-sysctl-config` is created again and again and never completes: + +```bash +kubectl -n cpaas-system get ar base +kubectl -n cpaas-system get broadcastjob upgrade-sysctl-config +``` + +## Root Cause + +The `base` AppRelease includes the BroadcastJob `upgrade-sysctl-config`, which runs a Pod on every node. The AppRelease has a timeout (`spec.timeout`) within which its resources must become ready. On a large cluster the BroadcastJob cannot finish on all nodes within that timeout. The sentry component that watches the release then judges the operation as timed out and runs it again. The new run starts over and times out the same way, so the `base` AppRelease never becomes healthy. + +## Resolution + +Use one of the two options. + +### Option 1: delete the BroadcastJob + +> **Warning:** Deleting `upgrade-sysctl-config` stops any of its Pods that are still running, so the sysctl update may not have run on every node. Do this only when the diagnostic steps show that the `base` AppRelease is failing on this BroadcastJob and that it keeps restarting. Save the object first. Rollback: `kubectl apply -f /tmp/upgrade-sysctl-config.yaml`. + +```bash +kubectl -n cpaas-system get broadcastjob upgrade-sysctl-config -o yaml > /tmp/upgrade-sysctl-config.yaml +kubectl -n cpaas-system delete broadcastjob upgrade-sysctl-config +kubectl -n cpaas-system get ar base -w +``` + +### Option 2: give the AppRelease more time + +> **Warning:** While `cluster-transformer` is scaled to zero, that platform controller is not running on this cluster. Keep the window short and scale it back as soon as the `base` AppRelease is healthy. Rollback: scale `cluster-transformer` back to the recorded replica count and patch `spec.timeout` back to the recorded value. + +1. Record the current replica count of `cluster-transformer` and the current timeout of the `base` AppRelease: + + ```bash + kubectl -n cpaas-system get deploy cluster-transformer -o jsonpath='{.spec.replicas}{"\n"}' + kubectl -n cpaas-system get ar base -o jsonpath='{.spec.timeout}{"\n"}' + ``` + +2. Stop `cluster-transformer` and raise the timeout (in seconds): + + ```bash + kubectl -n cpaas-system scale deploy/cluster-transformer --replicas=0 + kubectl -n cpaas-system patch ar base --type merge -p '{"spec":{"timeout":1800}}' + ``` + +3. Wait until the `base` AppRelease is healthy, then scale `cluster-transformer` back to the recorded replica count: + + ```bash + kubectl -n cpaas-system get ar base -w + kubectl -n cpaas-system scale deploy/cluster-transformer --replicas= + ``` + +## Diagnostic Steps + +1. Check the state of the `base` AppRelease and its conditions: + + ```bash + kubectl -n cpaas-system get ar base + kubectl -n cpaas-system get ar base -o yaml | grep -n -i -A5 'conditions\|timeout' + ``` + +2. Check the BroadcastJob and its progress. Compare `DESIRED` with the node count. A job that keeps a new, short age and never reaches `SUCCEEDED` equal to `DESIRED` is being restarted: + + ```bash + kubectl get nodes --no-headers | wc -l + kubectl -n cpaas-system get broadcastjob upgrade-sysctl-config + kubectl -n cpaas-system get pods -o wide | grep upgrade-sysctl-config + ``` + +3. Rule out a BroadcastJob that fails rather than one that is only slow. If its Pods end in `Error` or stay `Pending` on some nodes, a longer timeout does not help. Read those Pods first: + + ```bash + kubectl -n cpaas-system get pods | grep upgrade-sysctl-config | grep -v -E 'Running|Completed' + kubectl -n cpaas-system describe pod + ``` diff --git a/docs/en/solutions/crictl_pull_fails_with_unexpected_commit_digest_while_ctr_pull_works_because_of_hosts.toml_entry_order_on_ACP.md b/docs/en/solutions/crictl_pull_fails_with_unexpected_commit_digest_while_ctr_pull_works_because_of_hosts.toml_entry_order_on_ACP.md new file mode 100644 index 000000000..7d3be25bb --- /dev/null +++ b/docs/en/solutions/crictl_pull_fails_with_unexpected_commit_digest_while_ctr_pull_works_because_of_hosts.toml_entry_order_on_ACP.md @@ -0,0 +1,92 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# crictl pull fails with unexpected commit digest while ctr pull works because of hosts.toml entry order on ACP + +## Issue + +A registry (for example Harbor exposed through a NodePort, with HTTPS added in front of it by custom forwarding rules) is reachable by domain name. Pulling an image by that domain works with `docker`, `nerdctl` and `ctr`, but `crictl pull` on the same node fails, and kubelet image pulls that go through the same CRI path fail too: + +```text +"PullImage from image service failed" err="rpc error: code = FailedPrecondition desc = failed to pull and unpack image \"//:\": failed commit on ref \"unknown-sha256:\": unexpected commit digest sha256:, expected sha256:: failed precondition" +FATA[0000] pulling image: rpc error: code = FailedPrecondition desc = failed to pull and unpack image ...: failed precondition +``` + +Clearing the containerd image store (`/var/lib/containerd`) and restarting containerd and kubelet does not help, and the same error comes back. + +## Root Cause + +`crictl` talks to the containerd CRI plugin, and the CRI plugin resolves registries through the host configuration files under `/etc/containerd/certs.d/`. On the node, the wildcard configuration `/etc/containerd/certs.d/0.0.0.0_0/hosts.toml` holds two entries, one for `https://*` and one for `http://*`. containerd tries the `[host."..."]` entries of a `hosts.toml` in the order in which they appear in the file. + +When the `http://*` entry comes before the `https://*` entry, the CRI path contacts the registry domain over plain HTTP first. Behind forwarding rules that provide HTTPS for the registry, what comes back on that path is not the expected image content. Its digest does not match the digest from the manifest, and containerd aborts the commit with `unexpected commit digest ... failed precondition`. + +Other clients are not affected because they do not resolve the registry through this file in the same way. For example, `ctr` ignores `/etc/containerd/certs.d` unless `--hosts-dir` is passed, and `docker` uses its own daemon configuration. + +## Resolution + +> **Warning:** A wrong `hosts.toml` breaks every CRI image pull on the node, which includes the kubelet pulls for new Pods. Edit one node first and verify before changing the others. Rollback: copy the saved file back. + +1. Save the current file: + + ```bash + sudo cp /etc/containerd/certs.d/0.0.0.0_0/hosts.toml /tmp/hosts.toml.bak + ``` + +2. Edit `/etc/containerd/certs.d/0.0.0.0_0/hosts.toml` so that the `https://*` entry comes **before** the `http://*` entry. Keep the existing settings of each entry: + + ```toml + [host."https://*"] + capabilities = ["pull", "resolve", "push"] + skip_verify = true + + [host."http://*"] + capabilities = ["pull", "resolve", "push"] + skip_verify = true + ``` + +3. Pull the image again with `crictl`: + + ```bash + crictl pull //: + ``` + +4. Apply the same order on the other nodes that pull from this registry. + +## Diagnostic Steps + +1. Reproduce with both clients on the same node: + + ```bash + crictl pull //: + ctr -n k8s.io images pull //: + ``` + + Expected for this issue: `crictl` fails with `unexpected commit digest ... failed precondition`, while `ctr` completes. + +2. Check the order of the entries in the wildcard host configuration: + + ```bash + sudo grep -n 'host\.' /etc/containerd/certs.d/0.0.0.0_0/hosts.toml + ls /etc/containerd/certs.d/ + ``` + + Expected for this issue: `[host."http://*"]` is listed before `[host."https://*"]`. Also check whether a directory named after the registry domain exists under `/etc/containerd/certs.d/`. A host-specific `hosts.toml` there applies to that registry instead of the wildcard one, and its entry order matters in the same way. + +3. Confirm which directory the CRI plugin reads: + + ```bash + sudo grep -n -A2 'config_path' /etc/containerd/config.toml + ``` + + Expected: `config_path` points at `/etc/containerd/certs.d`. + +4. Rule out a corrupted local store. If the error remains after the entry order is corrected, check the containerd log for the pull. An identical digest error across different registries points to something else, such as a proxy that rewrites responses: + + ```bash + journalctl -u containerd --since "10 min ago" | grep -E 'commit digest|FailedPrecondition' + ``` diff --git a/docs/en/solutions/ecosystem/nacos/How_to_Deploy_Nacos_2.2.md b/docs/en/solutions/ecosystem/nacos/How_to_Deploy_Nacos_2.2.md index b85ff4a18..1a54336bb 100644 --- a/docs/en/solutions/ecosystem/nacos/How_to_Deploy_Nacos_2.2.md +++ b/docs/en/solutions/ecosystem/nacos/How_to_Deploy_Nacos_2.2.md @@ -257,3 +257,34 @@ Nacos resolves configs by composing names; the log line is expected during start ### Q8. After a Nacos 2.2.3 ephemeral instance is taken offline, the pod remains registered under the service Known community issue affecting Nacos 2.2.3. Fixed in 2.3.x. Track upstream: . To eliminate the symptom permanently, redeploy Nacos using the [Nacos 2.5 plan](./How_to_Deploy_Nacos_2.5.md). + +### Q9. How do I move Nacos members on TopoLVM to new nodes, for example before reinstalling the node operating system? + +TopoLVM volumes are local to one node, so a Nacos Pod cannot move to another node together with its PVCs. Move the members one at a time and let each one come up on a new node with new, empty volumes. Nacos configuration is kept in MySQL (see the MySQL prerequisite above), which this procedure does not touch. + +1. Prepare one new node per Nacos member (for example four new nodes for a four-member cluster), each with TopoLVM storage and enough free capacity for the member's log and data volumes. +2. Taint every node that currently runs a Nacos Pod, so that the Pods cannot be scheduled back onto them: + + ```bash + kubectl get pod -n -o wide | grep -i nacos + kubectl taint nodes nacos-migration=true:NoSchedule + ``` + +3. For one member, delete its log PVC and its data PVC first, then delete the Pod. Taking member 0 as an example: + + > **Warning:** Deleting the PVCs permanently deletes the member's local volumes on the old node. Move only one member at a time, and only while all other members are Ready, so that the cluster keeps serving. Do not reinstall an old node before every member has moved. Rollback for a member that has not been deleted yet: remove the taint from its node (`kubectl taint nodes nacos-migration=true:NoSchedule-`). + + ```bash + kubectl get pvc -n + kubectl delete pvc -n --wait=false + kubectl delete pod -n + ``` + +4. Wait until the recreated Pod is running on a new node and is Ready, with new PVCs bound: + + ```bash + kubectl get pod -n -o wide -w + kubectl get pvc -n + ``` + +5. Repeat steps 3 and 4 for the next member until every member runs on a new node. Then remove the taints from the old nodes or take the nodes out of service. diff --git a/docs/en/solutions/etcd-sync_Pod_stuck_in_ContainerCreating_with_failed_to_sync_secret_cache_on_the_standby_global_cluster_on_ACP.md b/docs/en/solutions/etcd-sync_Pod_stuck_in_ContainerCreating_with_failed_to_sync_secret_cache_on_the_standby_global_cluster_on_ACP.md new file mode 100644 index 000000000..51b6fafce --- /dev/null +++ b/docs/en/solutions/etcd-sync_Pod_stuck_in_ContainerCreating_with_failed_to_sync_secret_cache_on_the_standby_global_cluster_on_ACP.md @@ -0,0 +1,91 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# etcd-sync Pod stuck in ContainerCreating with failed to sync secret cache on the standby global cluster on ACP + +## Issue + +On the standby `global` cluster of a primary/standby (disaster recovery) setup, a new `etcd-sync` Pod in `cpaas-system` stays in `ContainerCreating`. `kubectl describe pod` shows: + +```text +Warning FailedMount ... MountVolume.SetUp failed for volume "etcd-sync-token" : failed to sync secret cache: timed out waiting for the condition +``` + +The Secrets the Pod uses (`etcd-sync-token`, `local-etcd-client`, `remote-etcd-client`) are listed by `kubectl get secret -n cpaas-system`, and other `etcd-sync` or `etcd-sync-monitor` Pods may still be `Running`. Listing Secrets across all namespaces fails with an error like: + +```text +Error from server (InternalError): Internal error occurred: unable to transform key "/registry/secrets//": invalid padding on input +``` + +## Root Cause + +The Kubernetes API server encrypts Secrets at rest with the keys in its encryption provider configuration, `/etc/kubernetes/encryption-provider.conf` on each control plane node. etcd synchronization copies the primary cluster's etcd data, including Secrets encrypted with the primary cluster's keys, into the standby cluster's etcd. The standby cluster can only decrypt that data when its control plane nodes use the same encryption provider configuration as the primary cluster. + +For that reason the standby installation procedure requires `/etc/kubernetes/encryption-provider.conf` to be copied from the first control plane node of the primary cluster to every standby control plane node **before** the standby cluster is installed. When this step is skipped, the standby cluster is installed with its own keys. The API server cannot decrypt the synchronized Secrets and returns `unable to transform key ... invalid padding on input`. The kubelet then cannot read the Secret behind the `etcd-sync-token` volume, and the Pod never leaves `ContainerCreating`. + +## Resolution + +The encryption configuration of a cluster is fixed when the cluster is installed. A standby cluster that was installed without the primary's encryption configuration has to be rebuilt. + +> **Warning:** Rebuilding the standby cluster removes it and its data. The primary cluster and its workloads are not affected, but there is no disaster recovery standby until the rebuild and the etcd synchronization have finished. Do this only when the diagnostic steps confirm the `invalid padding on input` error and a differing `encryption-provider.conf` on the standby control plane nodes. Plan a maintenance window for it. There is no rollback once the standby has been removed. Keep the primary cluster untouched throughout. + +1. Remove the standby cluster according to the product documentation. + +2. Log in to the first control plane node of the **primary** cluster and copy the encryption configuration to every control plane node of the standby cluster: + + ```bash + for i in + do + ssh "@$i" "sudo mkdir -p /etc/kubernetes/" + scp /etc/kubernetes/encryption-provider.conf "@$i:/tmp/encryption-provider.conf" + ssh "@$i" "sudo install -o root -g root -m 600 /tmp/encryption-provider.conf /etc/kubernetes/encryption-provider.conf && rm -f /tmp/encryption-provider.conf" + done + ``` + +3. Install the standby cluster again by following the standby installation steps of the global cluster disaster recovery documentation. The Platform Access Address, the certificate fields and the image repository fields must be the same as on the primary cluster. + +4. Enable etcd synchronization again and confirm that the `etcd-sync` Pods become `Running`: + + ```bash + kubectl -n cpaas-system get pods | grep etcd-sync + ``` + +## Diagnostic Steps + +1. Confirm the mount failure on the Pod: + + ```bash + kubectl -n cpaas-system get pods | grep etcd-sync + kubectl -n cpaas-system describe pod | grep -A3 FailedMount + ``` + + Expected for this issue: `failed to sync secret cache: timed out waiting for the condition` for volume `etcd-sync-token`. + +2. Confirm that the Secrets exist, so that the problem is not a missing Secret: + + ```bash + kubectl -n cpaas-system get secret etcd-sync-token local-etcd-client remote-etcd-client + ``` + + If a Secret is missing (`NotFound`), create it as described in the etcd synchronization steps instead. That is a different cause. + +3. Read Secrets through the API server: + + ```bash + kubectl get secret -A > /dev/null + ``` + + Expected for this issue: `unable to transform key "/registry/secrets/...": invalid padding on input`. This means that etcd holds Secrets the API server cannot decrypt. + +4. Compare the encryption configuration on the control plane nodes of both clusters: + + ```bash + sudo sha256sum /etc/kubernetes/encryption-provider.conf + ``` + + Run this on the first control plane node of the primary cluster and on each standby control plane node. Different checksums confirm that the file was not copied before the standby cluster was installed. diff --git a/docs/en/solutions/vmagent_fails_to_scrape_node-exporter_9100_metrics_because_its_basicAuth_does_not_match_node-exporter_on_ACP.md b/docs/en/solutions/vmagent_fails_to_scrape_node-exporter_9100_metrics_because_its_basicAuth_does_not_match_node-exporter_on_ACP.md new file mode 100644 index 000000000..119d96cc6 --- /dev/null +++ b/docs/en/solutions/vmagent_fails_to_scrape_node-exporter_9100_metrics_because_its_basicAuth_does_not_match_node-exporter_on_ACP.md @@ -0,0 +1,84 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# vmagent fails to scrape node-exporter 9100 metrics because its basicAuth does not match node-exporter on ACP + +## Issue + +After the VictoriaMetrics monitoring plugin is deployed, vmagent reports errors when it scrapes the node-exporter endpoint `https://:9100/metrics`, and the node-exporter targets are down. A manual request to the endpoint without credentials is refused with HTTP `401 Unauthorized`. The node-exporter DaemonSet `kube-prometheus-exporter-node` in `cpaas-system` itself is healthy, with all Pods scheduled, ready and available. + +## Root Cause + +The node-exporter of the VictoriaMetrics plugin protects `:9100/metrics` with TLS and HTTP basic authentication. The DaemonSet starts node-exporter with `--web.config.file=/etc/node-exporter/config.yaml`, mounted from the ConfigMap `victoriametrics-exporter-node-config` (key `config.yaml`). The TLS certificate and key come from the Secret `victoriametrics-exporter-node-ca`. The web config lists the users that are accepted (`basic_auth_users`, with a bcrypt hash of each password). + +vmagent sends the username and password configured in the `basicAuth` section of its node-exporter scrape object. When that username and password do not match a user in the node-exporter web config, every scrape is answered with `401` and all node-exporter targets go down. Nothing on the node is broken. The two sides of the credential simply disagree. + +## Resolution + +> **Warning:** Restarting vmagent and the node-exporter DaemonSet causes a short gap in node metrics. Save the objects you change first. Rollback: apply the saved YAML and restart the same components again. Do this only after the diagnostic steps show that a request with the vmagent credentials returns `401`. + +1. Save the node-exporter web config and the Secret that vmagent uses (see Diagnostic Steps for how to find it): + + ```bash + kubectl -n cpaas-system get cm victoriametrics-exporter-node-config -o yaml > /tmp/victoriametrics-exporter-node-config.yaml + kubectl -n cpaas-system get secret -o yaml > /tmp/.yaml + ``` + +2. Make the two sides consistent. Use one of these: + + - Set the username and password in the Secret that the vmagent scrape object references to a user that the node-exporter web config accepts. + - Or put the vmagent username into `basic_auth_users` of `config.yaml` in `victoriametrics-exporter-node-config`, with a bcrypt hash of the vmagent password. A bcrypt hash can be generated, for example, with: + + ```bash + htpasswd -nbBC 10 "" '' | tr -d ':\n'; echo + ``` + +3. Restart node-exporter and vmagent so that both load the new configuration: + + ```bash + kubectl -n cpaas-system rollout restart ds/kube-prometheus-exporter-node + kubectl -n cpaas-system get deploy,sts | grep -i vmagent + kubectl -n cpaas-system rollout restart / + ``` + +4. Check that the node-exporter targets are up again, and that a request with the vmagent credentials now returns metrics (see Diagnostic Steps, step 3). + +If the change is reverted after a plugin reconcile or upgrade, make the same change through the plugin configuration instead of editing the generated objects only. + +## Diagnostic Steps + +1. Read how the node-exporter is started and where its web config comes from: + + ```bash + kubectl -n cpaas-system get ds kube-prometheus-exporter-node -o yaml | grep -n -E 'web.config.file|configMap|secretName|name: ' + kubectl -n cpaas-system get cm victoriametrics-exporter-node-config -o jsonpath='{.data.config\.yaml}'; echo + ``` + + Expected: `--web.config.file=/etc/node-exporter/config.yaml`, and a `basic_auth_users` map in `config.yaml`. Note the username(s). + +2. Find the basicAuth that vmagent uses for node-exporter, and decode its username and password: + + ```bash + kubectl -n cpaas-system get servicemonitor,vmnodescrape,vmservicescrape 2>/dev/null | grep -i node + kubectl -n cpaas-system get / -o yaml | grep -n -A6 basicAuth + kubectl -n cpaas-system get secret -o jsonpath='{.data.username}' | base64 -d; echo + kubectl -n cpaas-system get secret -o jsonpath='{.data.password}' | base64 -d; echo + ``` + + A username that is not in `basic_auth_users` already confirms the mismatch. + +3. Test the endpoint from a node with and without the vmagent credentials: + + ```bash + curl -k -s -o /dev/null -w '%{http_code}\n' https://:9100/metrics + curl -k -s -o /dev/null -w '%{http_code}\n' -u ':' https://:9100/metrics + ``` + + Expected for this issue: `401` for both requests. After the fix, the request with credentials returns `200`. + +4. Rule out look-alikes. A connection error or timeout instead of `401` points to the network or to the node-exporter Pod, not to credentials. A `200` with credentials but down targets points to the scrape configuration (port, scheme, timeout), not to basicAuth. From 3f41079be326c8ee76d7445815f0092ae0c06973 Mon Sep 17 00:00:00 2001 From: Komh Date: Thu, 12 Nov 2026 08:59:40 +0800 Subject: [PATCH 04/25] docs(solutions): review fixes for the 20261110 batch - replace Chinese console labels with English ones - Nexus: clarify the v4 target in the title, copy the database export out of the Pod - Katanomi externalURL: state that it applies to DevOps v3 pipelines - ProductsVersion: use minor-version wildcards (4.0.x / 4.2.x / 4.3.x) --- ...ting_another_platform_administrator_on_ACP.md | 2 +- ...ons_v1_must_appear_in_spec_versions_on_ACP.md | 2 +- ...lications.app.k8s.io_already_exists_on_ACP.md | 2 +- ...form_address_from_the_Katanomi_externalURL.md | 2 ++ ...n_jobservice_exhausts_database_connections.md | 2 +- ...s_not_allowed_after_upgrading_to_ACP_4.3.3.md | 2 +- ...ctor_does_not_match_template_labels_on_ACP.md | 4 ++-- ...4_after_deleting_the_Nexus_Services_first.md} | 16 ++++++++++++---- ...f_the_katanomi-setting-init_init_container.md | 2 +- 9 files changed, 22 insertions(+), 12 deletions(-) rename docs/en/solutions/{Nexus_v4_migration_database_export_task_unreachable_after_deleting_the_Nexus_Services_first.md => Nexus_database_export_task_unreachable_during_migration_to_Alauda_DevOps_v4_after_deleting_the_Nexus_Services_first.md} (79%) diff --git a/docs/en/solutions/Built-in_admin_username_cannot_be_renamed_disable_admin_after_creating_another_platform_administrator_on_ACP.md b/docs/en/solutions/Built-in_admin_username_cannot_be_renamed_disable_admin_after_creating_another_platform_administrator_on_ACP.md index 41e2d9c7a..8e219ae03 100644 --- a/docs/en/solutions/Built-in_admin_username_cannot_be_renamed_disable_admin_after_creating_another_platform_administrator_on_ACP.md +++ b/docs/en/solutions/Built-in_admin_username_cannot_be_renamed_disable_admin_after_creating_another_platform_administrator_on_ACP.md @@ -4,7 +4,7 @@ kind: products: - Alauda Container Platform ProductsVersion: - - 4.2.4 + - 4.2.x --- # Built-in admin username cannot be renamed: disable admin after creating another platform administrator on ACP diff --git a/docs/en/solutions/Cluster_import_stuck_with_alb_module_Failed_grpcroutes_CRD_storedVersions_v1_must_appear_in_spec_versions_on_ACP.md b/docs/en/solutions/Cluster_import_stuck_with_alb_module_Failed_grpcroutes_CRD_storedVersions_v1_must_appear_in_spec_versions_on_ACP.md index 0ffd04ea6..108267d10 100644 --- a/docs/en/solutions/Cluster_import_stuck_with_alb_module_Failed_grpcroutes_CRD_storedVersions_v1_must_appear_in_spec_versions_on_ACP.md +++ b/docs/en/solutions/Cluster_import_stuck_with_alb_module_Failed_grpcroutes_CRD_storedVersions_v1_must_appear_in_spec_versions_on_ACP.md @@ -4,7 +4,7 @@ kind: products: - Alauda Container Platform ProductsVersion: - - 4.0.11 + - 4.0.x --- # Cluster import stuck with the alb module Failed: grpcroutes CRD status.storedVersions v1 must appear in spec.versions on ACP diff --git a/docs/en/solutions/Creating_a_custom_application_from_YAML_fails_with_applications.app.k8s.io_already_exists_on_ACP.md b/docs/en/solutions/Creating_a_custom_application_from_YAML_fails_with_applications.app.k8s.io_already_exists_on_ACP.md index 24093c371..1adb9f5e3 100644 --- a/docs/en/solutions/Creating_a_custom_application_from_YAML_fails_with_applications.app.k8s.io_already_exists_on_ACP.md +++ b/docs/en/solutions/Creating_a_custom_application_from_YAML_fails_with_applications.app.k8s.io_already_exists_on_ACP.md @@ -10,7 +10,7 @@ ProductsVersion: [] ## Issue -Creating a custom application from the YAML page of the console fails with `资源已存在` (resource already exists). Renaming the application and creating it again fails the same way, while applying the same resources with `kubectl` works. +Creating a custom application from the YAML page of the console fails with a **Resource already exists** error. Renaming the application and creating it again fails the same way, while applying the same resources with `kubectl` works. The error details show: diff --git a/docs/en/solutions/GitLab_Project_Hook_from_a_pipeline_trigger_uses_an_outdated_platform_address_from_the_Katanomi_externalURL.md b/docs/en/solutions/GitLab_Project_Hook_from_a_pipeline_trigger_uses_an_outdated_platform_address_from_the_Katanomi_externalURL.md index 4858841f4..a20c85bb2 100644 --- a/docs/en/solutions/GitLab_Project_Hook_from_a_pipeline_trigger_uses_an_outdated_platform_address_from_the_Katanomi_externalURL.md +++ b/docs/en/solutions/GitLab_Project_Hook_from_a_pipeline_trigger_uses_an_outdated_platform_address_from_the_Katanomi_externalURL.md @@ -10,6 +10,8 @@ ProductsVersion: [] ## Issue +This article applies to pipelines of the DevOps v3 (katanomi) components, whose triggers register the GitLab webhook. The `Katanomi` resource described below belongs to those v3 components. + A CI/CD pipeline is configured to build automatically when GitLab code changes. The webhook that the platform registers in the GitLab project (**Settings > Webhooks**, Project Hooks) points to an old address, for example `http:///...`, instead of the address through which the platform is actually reached, for example `https:///...`. The question is where this default address comes from and how to change it. ## Root Cause diff --git a/docs/en/solutions/Harbor_garbage_collection_fails_with_write_broken_pipe_to_PostgreSQL_when_jobservice_exhausts_database_connections.md b/docs/en/solutions/Harbor_garbage_collection_fails_with_write_broken_pipe_to_PostgreSQL_when_jobservice_exhausts_database_connections.md index fd306f7e3..361d7273b 100644 --- a/docs/en/solutions/Harbor_garbage_collection_fails_with_write_broken_pipe_to_PostgreSQL_when_jobservice_exhausts_database_connections.md +++ b/docs/en/solutions/Harbor_garbage_collection_fails_with_write_broken_pipe_to_PostgreSQL_when_jobservice_exhausts_database_connections.md @@ -10,7 +10,7 @@ ProductsVersion: [] ## Issue -Harbor garbage collection (GC) no longer frees storage. In the GC history every recent run, scheduled (`定时`) and manual (`手动`), is in the `错误` (Error) state. The GC job log stops in the mark phase with a `broken pipe` on the connection to the Harbor database on port 5432: +Harbor garbage collection (GC) no longer frees storage. In the GC history every recent run, with trigger type **Schedule** and **Manual** alike, is in the **Error** state. The GC job log stops in the mark phase with a `broken pipe` on the connection to the Harbor database on port 5432: ```text [INFO] [/jobservice/job/impl/gc/garbage_collection.go:144]: Garbage Collection parameters: [delete_untagged: false, dry_run: false, time_window: 2] diff --git a/docs/en/solutions/Integrated_product_link_fails_with_Redirect_to_untrusted_host_is_not_allowed_after_upgrading_to_ACP_4.3.3.md b/docs/en/solutions/Integrated_product_link_fails_with_Redirect_to_untrusted_host_is_not_allowed_after_upgrading_to_ACP_4.3.3.md index 24343a7bf..8eae229e4 100644 --- a/docs/en/solutions/Integrated_product_link_fails_with_Redirect_to_untrusted_host_is_not_allowed_after_upgrading_to_ACP_4.3.3.md +++ b/docs/en/solutions/Integrated_product_link_fails_with_Redirect_to_untrusted_host_is_not_allowed_after_upgrading_to_ACP_4.3.3.md @@ -4,7 +4,7 @@ kind: products: - Alauda Container Platform ProductsVersion: - - 4.3.3 + - 4.3.x --- # Integrated product link fails with Redirect to untrusted host is not allowed after upgrading to ACP 4.3.3 diff --git a/docs/en/solutions/Native_application_update_fails_with_selector_does_not_match_template_labels_on_ACP.md b/docs/en/solutions/Native_application_update_fails_with_selector_does_not_match_template_labels_on_ACP.md index c36a968b3..d63488388 100644 --- a/docs/en/solutions/Native_application_update_fails_with_selector_does_not_match_template_labels_on_ACP.md +++ b/docs/en/solutions/Native_application_update_fails_with_selector_does_not_match_template_labels_on_ACP.md @@ -10,7 +10,7 @@ ProductsVersion: [] ## Issue -A native application shows `失败` (Failed) in the console after an update, although its Deployments show `运行中` (Running) with all replicas ready. The update never reaches the Deployment. +A native application shows **Failed** in the console after an update, although its Deployments show **Running** with all replicas ready. The update never reaches the Deployment. The `archon` component logs on the global cluster show the update being rejected by the Kubernetes API server: @@ -62,7 +62,7 @@ A configuration error in another resource of the same application can also leave Then redeploy the native application from the console, or re-run the release that renders the corrected template. -4. Confirm that the native application status changes from `失败` (Failed) to Running, and that `archon` no longer logs the error for this Deployment. +4. Confirm that the native application status changes from **Failed** to **Running**, and that `archon` no longer logs the error for this Deployment. ## Diagnostic Steps diff --git a/docs/en/solutions/Nexus_v4_migration_database_export_task_unreachable_after_deleting_the_Nexus_Services_first.md b/docs/en/solutions/Nexus_database_export_task_unreachable_during_migration_to_Alauda_DevOps_v4_after_deleting_the_Nexus_Services_first.md similarity index 79% rename from docs/en/solutions/Nexus_v4_migration_database_export_task_unreachable_after_deleting_the_Nexus_Services_first.md rename to docs/en/solutions/Nexus_database_export_task_unreachable_during_migration_to_Alauda_DevOps_v4_after_deleting_the_Nexus_Services_first.md index 0950c4ea8..6c52ad88d 100644 --- a/docs/en/solutions/Nexus_v4_migration_database_export_task_unreachable_after_deleting_the_Nexus_Services_first.md +++ b/docs/en/solutions/Nexus_database_export_task_unreachable_during_migration_to_Alauda_DevOps_v4_after_deleting_the_Nexus_Services_first.md @@ -6,11 +6,11 @@ products: ProductsVersion: [] --- -# Nexus v4 migration: database export task unreachable after deleting the Nexus Services first +# Nexus database export task unreachable during migration to Alauda DevOps v4 after deleting the Nexus Services first ## Issue -While migrating a platform-deployed Nexus to v4, the migration steps are run in their numbered order: +While migrating a platform-deployed Nexus to Alauda DevOps v4 (the Nexus version itself does not change), the migration steps are run in their numbered order: - 1.1 Service shutdown: uninstall the old-version operator, then delete the Nexus Services with `kubectl -n ${INSTANCE_NAMESPACE} delete service -l release=${INSTANCE_NAME}`. - 1.2 Back up Blob storage with `kubectl cp` from the Nexus Pod. @@ -55,7 +55,15 @@ The correct execution order is: complete steps 1.2 and 1.3 first, while the Serv 5. Frequency: manual. 6. Run the task and confirm that it completes. -4. Step 1.1, shut down the service. Uninstall the old-version operator first; otherwise the deleted Services are recreated automatically. Then save and delete the Services: + The export files are written inside the Nexus data volume, and step 1.2 copies only `/nexus-data/blobs`. Copy the export out of the Pod as well, so that the database backup does not exist only on the volume being migrated: + + ```bash + mkdir -p nexus-backup/db + kubectl -n ${INSTANCE_NAMESPACE} cp ${POD_NAME}: nexus-backup/db + ls -la nexus-backup/db + ``` + +4. Step 1.1, shut down the service, only after the blob backup and the database export are both in `nexus-backup/`. Uninstall the old-version operator first; otherwise the deleted Services are recreated automatically. Then save and delete the Services: > **Warning:** After the Services are deleted, Nexus is no longer reachable by users, build tools or the web UI. Do this only after steps 1.2 and 1.3 have completed and the backups are confirmed. To restore access, re-apply the saved Service manifests (remove `metadata.resourceVersion`, `metadata.uid`, `spec.clusterIP` and `spec.clusterIPs` first). @@ -72,7 +80,7 @@ If the Services were already deleted before the database export, the web UI can kubectl -n ${INSTANCE_NAMESPACE} port-forward pod/${POD_NAME} 8081:8081 ``` -Open `http://localhost:8081` on the machine running the command, then perform step 1.3. +Open `http://localhost:8081` on the machine running the command, then perform step 1.3 and copy the export out of the Pod as described above. ## Diagnostic Steps diff --git a/docs/en/solutions/Tekton_pipeline_Pod_rejected_by_Pod_Security_admission_because_of_the_katanomi-setting-init_init_container.md b/docs/en/solutions/Tekton_pipeline_Pod_rejected_by_Pod_Security_admission_because_of_the_katanomi-setting-init_init_container.md index 7d82ac91a..6f2b6e259 100644 --- a/docs/en/solutions/Tekton_pipeline_Pod_rejected_by_Pod_Security_admission_because_of_the_katanomi-setting-init_init_container.md +++ b/docs/en/solutions/Tekton_pipeline_Pod_rejected_by_Pod_Security_admission_because_of_the_katanomi-setting-init_init_container.md @@ -4,7 +4,7 @@ kind: products: - Alauda DevOps ProductsVersion: - - 4.3.3 + - 4.3.x --- # Tekton pipeline Pod rejected by Pod Security admission because of the katanomi-setting-init init container From e067a9b52eb7431e323422ca5a6b2c1a0c75e757 Mon Sep 17 00:00:00 2001 From: Komh Date: Thu, 12 Nov 2026 21:31:34 +0800 Subject: [PATCH 05/25] docs(solutions): add 24 and update 2 troubleshooting articles --- ...er_loss_x86_to_ARM_and_planned_shutdown.md | 127 +++++++++ ...M_catalog_or_operator-managed_component.md | 261 ++++++++++++++++++ ...rade_and_the_restart_cannot_be_deferred.md | 81 ++++++ ...ugin_fails_to_install_or_upgrade_on_ACP.md | 199 +++++++++++++ ...bidden_hook_Job_or_missing_ModuleConfig.md | 175 ++++++++++++ ...r-runc_and_overwritten_runtime_binaries.md | 249 +++++++++++++++++ ..._containerd_directories_are_lost_on_ACP.md | 82 ++++++ ..._ownership_conflict_or_missing_instance.md | 146 ++++++++++ ...onfigMap_chart_and_feature_gate_changes.md | 246 +++++++++++++++++ ...luster_exec_API_returns_HTTP_500_on_ACP.md | 84 ++++++ ...ostname_IP_NIC_and_cluster_name_changes.md | 152 ++++++++++ ...asters_host_shutdown_and_disk_expansion.md | 145 ++++++++++ ...t_be_resolved_inside_the_cluster_on_ACP.md | 19 ++ ...OOMKilled_or_cluster-transformer_errors.md | 164 +++++++++++ ...is_incomplete_violet_list_and_apps_yaml.md | 184 ++++++++++++ ...fact_of_another_chart_repository_on_ACP.md | 128 +++++++++ ...ty_catalog_source_or_plugin_type_on_ACP.md | 203 ++++++++++++++ ...xtra_masters_and_control-plane_requests.md | 151 ++++++++++ ...cannot_finish_on_a_large_cluster_on_ACP.md | 70 +++++ ...r_runc_or_containerd_is_replaced_on_ACP.md | 190 +++++++++++++ ...d_bursts_or_a_database_over_2_GB_on_ACP.md | 156 +++++++++++ ...ion_frees_only_tens_of_megabytes_on_ACP.md | 146 ++++++++++ ...nager_or_scheduler_health_alerts_on_ACP.md | 199 +++++++++++++ ...iled_or_leftover_ResourcePatches_on_ACP.md | 140 ++++++++++ ...or_version_is_skipped_or_missing_on_ACP.md | 106 +++++++ ...ct_ModuleConfig_or_chart_missing_on_ACP.md | 169 ++++++++++++ 26 files changed, 3972 insertions(+) create mode 100644 docs/en/solutions/ACP_control-plane_limits_apiserver_address_change_snapshot_db_scope_total_master_loss_x86_to_ARM_and_planned_shutdown.md create mode 100644 docs/en/solutions/ACP_upgrade_stuck_on_an_operator_OLM_catalog_or_operator-managed_component.md create mode 100644 docs/en/solutions/ALB_load_balancer_Pods_restart_during_an_ACP_upgrade_and_the_restart_cannot_be_deferred.md create mode 100644 docs/en/solutions/Argo_Rollouts_Kubeflow_Base_or_a_wrong-architecture_cluster_plugin_fails_to_install_or_upgrade_on_ACP.md create mode 100644 docs/en/solutions/Cluster_plugin_install_or_upgrade_stuck_in_chart_sync_sentry_StatefulSet_Forbidden_hook_Job_or_missing_ModuleConfig.md create mode 100644 docs/en/solutions/Container_runtime_conflicts_on_ACP_nodes_Docker_podman_docker-runc_and_overwritten_runtime_binaries.md create mode 100644 docs/en/solutions/Control-plane_Pods_do_not_start_with_mkdir_var_log_pods_file_exists_after_containerd_directories_are_lost_on_ACP.md create mode 100644 docs/en/solutions/DevOps_v3_toolchain_entry_missing_after_an_upgrade_Katanomi_MonitorDashboard_ownership_conflict_or_missing_instance.md create mode 100644 docs/en/solutions/Local_overrides_reverted_or_blocking_an_ACP_upgrade_ResourcePatch_ALB_ConfigMap_chart_and_feature_gate_changes.md create mode 100644 docs/en/solutions/Node_add_check_times_out_immediately_because_the_cluster_exec_API_returns_HTTP_500_on_ACP.md create mode 100644 docs/en/solutions/Node_maintenance_on_ACP_OS_reinstall_or_replacement_hostname_IP_NIC_and_cluster_name_changes.md create mode 100644 docs/en/solutions/Node_maintenance_on_ACP_resizing_nodes_and_masters_host_shutdown_and_disk_expansion.md create mode 100644 docs/en/solutions/Plugin_install_or_uninstall_stuck_or_Marketplace_and_OperatorHub_empty_sentry_OOMKilled_or_cluster-transformer_errors.md create mode 100644 docs/en/solutions/Preparing_the_extension_package_list_for_an_ACP_upgrade_fails_or_is_incomplete_violet_list_and_apps_yaml.md create mode 100644 docs/en/solutions/Pushed_Helm_chart_version_not_shown_because_violet_reused_an_Artifact_of_another_chart_repository_on_ACP.md create mode 100644 docs/en/solutions/Pushed_package_not_listed_where_expected_global-only_plugins_cluster_affinity_catalog_source_or_plugin_type_on_ACP.md create mode 100644 docs/en/solutions/Replacing_or_rejoining_an_ACP_master_node_same-IP_replacement_etcd-join_extra_masters_and_control-plane_requests.md create mode 100644 docs/en/solutions/etcd_and_kube-apiserver_do_not_start_on_a_master_after_runc_or_containerd_is_replaced_on_ACP.md create mode 100644 docs/en/solutions/etcd_quota_stays_full_from_a_large_live_key_set_Pod_bursts_or_a_database_over_2_GB_on_ACP.md create mode 100644 docs/en/solutions/etcd_reports_database_space_exceeded_or_compaction_frees_only_tens_of_megabytes_on_ACP.md create mode 100644 docs/en/solutions/kube-apiserver_latency_alerts_noisy_apiserver_logs_and_controller-manager_or_scheduler_health_alerts_on_ACP.md create mode 100644 docs/en/solutions/sentry_at_full_CPU_or_a_redeployed_component_fails_because_of_Failed_or_leftover_ResourcePatches_on_ACP.md create mode 100644 docs/en/solutions/violet_push_fails_to_parse_a_package_or_a_new_operator_version_is_skipped_or_missing_on_ACP.md create mode 100644 docs/en/solutions/violet_push_succeeds_but_the_plugin_is_not_published_Artifact_ModuleConfig_or_chart_missing_on_ACP.md diff --git a/docs/en/solutions/ACP_control-plane_limits_apiserver_address_change_snapshot_db_scope_total_master_loss_x86_to_ARM_and_planned_shutdown.md b/docs/en/solutions/ACP_control-plane_limits_apiserver_address_change_snapshot_db_scope_total_master_loss_x86_to_ARM_and_planned_shutdown.md new file mode 100644 index 000000000..4a334a394 --- /dev/null +++ b/docs/en/solutions/ACP_control-plane_limits_apiserver_address_change_snapshot_db_scope_total_master_loss_x86_to_ARM_and_planned_shutdown.md @@ -0,0 +1,127 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# ACP control-plane limits: apiserver address change, snapshot.db scope, total master loss, x86 to ARM, and planned shutdown + +## Issue + +- A business-cluster apiserver VIP or URL has changed and the ACP console still uses the old address. +- A namespace was deleted and the only copy left is an etcd `snapshot.db` or `snapshot-etcd` file. +- All masters are lost and the only copy is the platform etcd backup; a version-specific command list for a full restore is requested. +- The `global` cluster runs on x86 and has to move to ARM. +- The whole cluster has a scheduled shutdown. + +These are procedure and product-limit questions, not a single-member etcd repair and not a database-full alarm. + +Related articles: "Replacing or rejoining an ACP master node: same-IP replacement, etcd-join, extra masters, and control-plane requests". + +## Root Cause + +### Cause 1: A business cluster apiserver address change has no standard platform procedure + +On platform versions 3.12, 3.18, and 4.2 there is no standard procedure to point the console or the `global` cluster at a new business-cluster apiserver VIP or address. + +### Cause 2: A deleted namespace cannot be restored on its own from snapshot.db + +A single namespace cannot be extracted from `snapshot.db` (running `strings` on the file does not produce the objects). The supported path is a restore of the whole cluster from an etcd backup, which returns every namespace to that backup's point in time. There is no reference procedure for starting a separate etcd just to read one namespace. + +### Cause 3: All masters lost with only an etcd backup has no supported restore + +There is no supported procedure to rebuild a cluster from the etcd backup alone after all masters are lost, on any version including 4.3, with or without bare-metal masters. An etcd snapshot stores cluster data but not the certificates and keys the cluster needs. The backup job can also save node files (`/etc/kubernetes/pki`, manifests, kubelet configuration, container runtime units), but those are version-specific; a cluster rebuilt from them is not a standard deployment and is likely to fail a later upgrade. A `snapshot-etcd` file is the platform backup, not a namespace extract and not a rebuild kit. + +### Cause 4: An x86 global cluster cannot be moved to ARM by replacing servers or importing etcd + +Replacing the servers does not change the cluster's CPU architecture, and redeploying an ARM `global` and importing the existing etcd data does not let it take over the existing business clusters. An ARM `global` is a new deployment plus a business migration. + +### Cause 5: A scheduled shutdown of the whole cluster + +A planned outage needs an etcd backup first and a normal `shutdown` rather than a power cut. Power the nodes off together when possible; otherwise shut down workers first and masters after. After power-on, platform components recover on their own. + +## Resolution + +### Cause 1: A business cluster apiserver address change has no standard platform procedure + +Do not create a procedure by rewriting kube-apiserver manifests or restoring etcd. Collect the old and new URLs and contact Alauda support for a version-specific procedure. The following prints the server URL a kubeconfig uses; it does not update the platform: + +```bash +kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}{"\n"}' +``` + +### Cause 2: A deleted namespace cannot be restored on its own from snapshot.db + +Restoring the namespace means restoring the whole cluster from the etcd backup, which is a planned-downtime decision for every namespace, not a partial extract. `file` only identifies the blob: + +```bash +file snapshot.db +``` + +### Cause 3: All masters lost with only an etcd backup has no supported restore + +Plan for this before it happens: + +1. Take full server or VM backups on a schedule, in addition to the etcd backup. +2. Protect master disks (for example with RAID) and repair failed disks promptly. +3. Run critical workloads on more than one independent cluster. + +Check how many etcd members are still running: + +```bash +kubectl -n kube-system get pods -l component=etcd -o wide +``` + +Zero running members with only `snapshot-etcd` or `snapshot.db` left is this limit. With one or more healthy members, the remaining members are repaired or replaced one at a time instead. + +### Cause 4: An x86 global cluster cannot be moved to ARM by replacing servers or importing etcd + +Plan a new ARM deployment and a migration of the business workloads. `uname -m` only confirms the current architecture: + +```bash +uname -m +kubectl get nodes -o wide +``` + +### Cause 5: A scheduled shutdown of the whole cluster + +Take the etcd backup before any power-off: run the backup on the platform's etcd backup task and confirm that a current backup record exists, or take a snapshot with `etcdctl snapshot save` from a healthy member. + +```bash +kubectl get etcdbackupconfiguration -A +``` + +> **Warning:** `shutdown -h now` powers this node off. The etcd backup must already exist. Run it on workers before masters when the nodes cannot go down together. Rollback is powering the node on again. A node that does not return after power-on is a new fault, not a step of this shutdown. + +```bash +sudo shutdown -h now +``` + +After power-on, wait for the platform components to recover and check: + +```bash +kubectl get nodes +kubectl get pods -A | grep -v -E 'Running|Completed' +``` + +## Diagnostic Steps + +```bash +kubectl -n kube-system get pods -l component=etcd -o wide +kubectl get nodes -l node-role.kubernetes.io/control-plane= -o wide +kubectl get nodes -o wide +``` + +- Address change of an existing business cluster on 3.12, 3.18, or 4.2: Cause 1. +- `snapshot.db` in hand and one missing namespace: Cause 2. +- Every master lost and the only copy is an etcd backup: Cause 3. +- x86 `global` to ARM by server swap or etcd import: Cause 4. +- Planned power-off of the whole cluster: Cause 5. diff --git a/docs/en/solutions/ACP_upgrade_stuck_on_an_operator_OLM_catalog_or_operator-managed_component.md b/docs/en/solutions/ACP_upgrade_stuck_on_an_operator_OLM_catalog_or_operator-managed_component.md new file mode 100644 index 000000000..5f85fff72 --- /dev/null +++ b/docs/en/solutions/ACP_upgrade_stuck_on_an_operator_OLM_catalog_or_operator-managed_component.md @@ -0,0 +1,261 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.16" + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# ACP upgrade stuck on an operator, OLM catalog, or operator-managed component + +## Issue + +The platform upgrade waits on an operator-managed component and does not progress. One of the following is visible: + +- `olm-registry-custom`, `olm-registry-platform`, or `olm-registry-system` is in `CrashLoopBackOff` with `unable to load in-cluster config`. +- The ARS (AppRelease) stays unready after an operator upgrade, and the `sentry` log shows a version-check Job that still checks the previous operator version. +- Argo CD `minfo` (ModuleInfo) is Failed during the upgrade to 4.0.9 and the Argo CD wait Job has no logs. +- The upgrade page stays in the Katanomi phase. +- The ASM (service mesh) upgrade fails with `bundles outdated`. +- The platform upgrade is stuck at the Tekton component. +- `redis-operator` stays in processing on the upgrade page. +- `rds-operator-updater` is stuck, or several components wait for the `rds-operator` upgrade. +- Argo CD is not ready after `bash upgrade.sh` has completed. + +## Root Cause + +### Cause 1: olm-registry Pods crash with unable to load in-cluster config + +The three `olm-registry-*` Pods crash with `unable to load in-cluster config`, and `kubectl get rpch` shows a ResourcePatch (`rpch`) that targets ConfigMap `olm-registry-config` in `cpaas-system`. The local override of that ConfigMap breaks the registry Pods. + +### Cause 2: The version-check Job still validates the previous operator version + +After an operator upgrade the ARS stays unready, and the `sentry` log shows the Job that checks the operator version and status still pinned to the previous version (for example the Job checks `v4.2.0` while `argocd-operator` is already `v4.2.1`). Editing the Job in place does not change what its Pod runs. + +### Cause 3: Argo CD minfo is Failed because the installed operator package is the wrong build + +During the upgrade to 4.0.9, the Argo CD `minfo` is Failed, the ARS error names GitOps Operator `v4.1.8`, and the Argo CD wait Job has no logs. The cluster has Argo CD Operator `4.2.1` loaded, while that upgrade requires package `4.2.0`. The version pair is specific to the 4.0.9 upgrade; for another target, compare the version named in the ARS error with the version shipped in the upgrade package. + +### Cause 4: Katanomi update is stuck on a historical Helm release + +The upgrade remains in the Katanomi phase. `helm ls -A` lists release `katanomi-global` in `cpaas-system` from the previous product generation, and the `katanomis.operators.katanomi.dev` object still points to a CatalogSource named `stable` that `kubectl get catalogsource -A` does not show. + +### Cause 5: The mesh upgrade reports bundles outdated + +The ASM upgrade fails with `bundles outdated`, although the mesh images are present on both the `global` cluster and the business cluster. Restarting `catalog-operator`, `packageserver`, `olm-operator`, and `olm-registry-system` unblocks OLM. Which of the four processes held stale data is not known. + +### Cause 6: The v3 tekton-operator and the v4 Pipeline operator both manage the same Tekton instance + +On a 4.1.2 to 4.2.1 platform upgrade the upgrade stays on the Tekton component. Both the v3 Tekton operator (`tekton-operator`) and the v4 Pipeline operator are installed and reconcile the same Tekton instance. v3 Tekton and v4 Pipeline cannot coexist; uninstalling the v3 `tekton-operator`, without changing the instance itself, clears the block. + +### Cause 7: redis-operator stays processing because its Subscription requires manual approval + +During a 4.1.3 to 4.2.1 upgrade, after the plugin packages were uploaded and `upgrade.sh` was run, `redis-operator` stays in processing. Its OperatorHub details page (Alauda Cache Service for Redis OSS) shows the old installed CSV (for example `redis-operator.v4.1.1` in `redis-system`, status `Succeeded`, channel `alpha`) and a banner that a new version is detected. The Subscription's upgrade strategy is manual, so the platform upgrade does not upgrade the operator; the new version waits for confirmation. + +### Cause 8: rds-operator or rds-operator-updater blocks the operators that follow it + +- `rds-operator-updater` is stuck (for example on a 4.1.3 to 4.1.4 upgrade). Deleting its Subscription lets the upgrade continue. +- On a workload-cluster upgrade, several components wait for the `rds-operator` upgrade. Uninstalling `rds-operator` and deleting its `minfo` and Subscription lets the upgrade continue. Why the `rds-operator` upgrade blocks the others is not known. + +If the next operators then stall because their Pods cannot be created on a node, that is a separate node-level problem. + +### Cause 9: Argo CD is not ready after upgrade.sh completes + +`bash upgrade.sh` finished, but Argo CD is not ready. This state can be ignored at this point; start the `global` cluster upgrade directly. The cause of the not-ready state has not been determined, so check Argo CD again after the global upgrade. + +## Resolution + +Match one signature before changing the cluster, and save every object before deleting it. + +### Cause 1: olm-registry Pods crash with unable to load in-cluster config + +> **Warning:** Deleting this `rpch` removes the local override of ConfigMap `olm-registry-config`; images or endpoints that the override forced fall back to the chart defaults. Precondition: only when the CrashLoop log is `unable to load in-cluster config` and the `rpch` targets `cpaas-system/olm-registry-config`. Rollback: `kubectl apply -f /tmp/rpch-olm.bak.yaml`. + +```bash +kubectl get rpch -o yaml > /tmp/rpch-olm.bak.yaml +kubectl get rpch | grep olm +kubectl delete rpch +kubectl get pods -n cpaas-system | grep olm-registry +kubectl delete pod -n cpaas-system +``` + +The replacement Pods should load the in-cluster config. + +### Cause 2: The version-check Job still validates the previous operator version + +> **Warning:** Deleting the version-check Job stops its Pod. Precondition: only when the `sentry` log quotes the old version inside that Job and an in-place edit of the Job did not change the check. Rollback: the controller recreates the Job. If the new Job still pins the old version, the installed package is wrong; use Cause 3. + +```bash +kubectl get jobs -A +kubectl delete job -n +``` + +Watch the ARS. Ready means this Job was the blocker. + +### Cause 3: Argo CD minfo is Failed because the installed operator package is the wrong build + +For the 4.0.9 upgrade, upload Argo CD Operator package `4.2.0` with `violet`, remove package `4.2.1`, and let the Argo CD `minfo` reconcile. For another target version, upload the operator package version that the upgrade bundle requires and remove the other build that the ARS error shows being installed. + +### Cause 4: Katanomi update is stuck on a historical Helm release + +> **Warning:** This deletes Helm release `katanomi-global` and its objects and removes the CatalogSource `stable` reference from the Katanomi object. Precondition: only when the upgrade is stuck in the Katanomi phase, `helm ls -n cpaas-system` shows `katanomi-global`, and no CatalogSource named `stable` exists. Save the Helm manifest first; the historical release is not reinstalled afterwards. Rollback of the Katanomi object: `kubectl apply -f /tmp/katanomi.bak.yaml`. + +```bash +helm get all -n cpaas-system katanomi-global > /tmp/katanomi-global.helm.txt +kubectl get katanomis.operators.katanomi.dev -A -o yaml > /tmp/katanomi.bak.yaml +helm delete -n cpaas-system katanomi-global +``` + +Edit the `katanomis.operators.katanomi.dev` object: remove the spec fields that name CatalogSource `stable`, and set an annotation so the object changes (for example `cpaas.io/reconcile-at` with the current UTC time). Then restart the catalog Pods: + +```bash +kubectl get pods -n cpaas-system | grep -E 'packageserver|catalog' +kubectl delete pod -n cpaas-system +``` + +Katanomi enqueues its own upgrade after those Pods are Ready. + +### Cause 5: The mesh upgrade reports bundles outdated + +Confirm that the mesh images are on the `global` cluster and the business cluster, then restart the four OLM processes. + +> **Warning:** Catalog resolution pauses until the replacement Pods are Ready, and in-flight InstallPlans retry. Precondition: only when the upgrade reports `bundles outdated` and the images are already present on both clusters. Rollback: none needed; the controllers recreate the Pods. + +```bash +kubectl get pods -n cpaas-system | grep -E 'catalog-operator|packageserver|olm-operator|olm-registry-system' +kubectl delete pod -n cpaas-system +``` + +### Cause 6: The v3 tekton-operator and the v4 Pipeline operator both manage the same Tekton instance + +List both operators and the instance they share: + +```bash +kubectl get csv -A | grep -Ei 'tekton|pipeline' +kubectl get subscriptions.operators.coreos.com -A | grep -Ei 'tekton|pipeline' +kubectl get deploy -A | grep -Ei 'tekton-operator|pipeline' +kubectl api-resources | grep -i tekton +``` + +For upstream-based Tekton operators the shared instance is the `TektonConfig` object (`operator.tekton.dev`); read it so you know what has to stay. + +> **Warning:** The v3 `tekton-operator` stops reconciling Tekton and the v4 Pipeline operator becomes the only manager of the instance. Precondition: only when both operators are installed and the upgrade is stuck at the Tekton component. Keep the Tekton instance, its CRDs, and its namespaces; remove the v3 operator only. Rollback: `kubectl apply -f` the saved objects, which installs the v3 operator again and brings back the conflict. + +When the v3 operator is installed through OLM (the commands above list a Subscription and CSV for it), remove those two objects: + +```bash +kubectl get subscriptions.operators.coreos.com -n -o yaml > /tmp/sub-tekton-operator.bak.yaml +kubectl get csv -n -o yaml > /tmp/csv-tekton-operator.bak.yaml +kubectl delete subscriptions.operators.coreos.com -n +kubectl delete csv -n +``` + +Deleting a CSV leaves the CRDs and custom resources in place. When the v3 operator was not installed through OLM, uninstall it the way it was installed, again keeping the instance. The Tekton stage moves once only the v4 Pipeline operator manages the instance. + +### Cause 7: redis-operator stays processing because its Subscription requires manual approval + +In OperatorHub, search for `redis-operator` and click **Confirm** on the new-version banner of its details page. To upgrade every operator that waits in the same way at once, use the batch upgrade under Cluster Management > Cluster > Feature Components. + +The same state from the command line: + +```bash +kubectl get subscriptions.operators.coreos.com -n redis-system -o yaml | grep -n -E 'installPlanApproval|currentCSV|installedCSV|state' +kubectl get installplan -n redis-system +``` + +`installPlanApproval: Manual` and an InstallPlan with `APPROVED` `false` are the signature. Approving that InstallPlan performs the same upgrade: + +> **Warning:** The Redis operator is upgraded to the new CSV, and the instances it manages may be reconciled by the new operator version. Precondition: only when `redis-operator` is stuck in processing, its Subscription is `Manual`, and the InstallPlan names the target version. Check the Redis instance prerequisites for the operator upgrade first. Rollback: treat this as one-way; OLM does not downgrade an approved InstallPlan. + +```bash +kubectl patch installplan -n redis-system --type=merge -p '{"spec":{"approved":true}}' +kubectl get csv -n redis-system +``` + +When the new CSV is `Succeeded`, check that the upgrade page has moved past `redis-operator`. + +### Cause 8: rds-operator or rds-operator-updater blocks the operators that follow it + +```bash +kubectl get subscriptions.operators.coreos.com -A | grep -i rds +kubectl get csv -A | grep -i rds +kubectl get minfo | grep -i rds +``` + +> **Warning:** Removing the Subscription stops OLM from managing that operator's upgrades. Uninstalling `rds-operator` stops reconciliation of the RDS-managed instances (failover, scaling, backup) until it is installed again; running database Pods are not deleted. Precondition: only when the upgrade is stuck on `rds-operator-updater` or on components that wait for `rds-operator`. Back up each object first. Rollback: `kubectl apply -f` the backups, or install `rds-operator` again from OperatorHub after the platform upgrade. + +For a stuck `rds-operator-updater`, delete its Subscription: + +```bash +kubectl get subscriptions.operators.coreos.com -n -o yaml > /tmp/sub-rds-operator-updater.bak.yaml +kubectl delete subscriptions.operators.coreos.com -n +``` + +When components wait for the `rds-operator` upgrade, uninstall `rds-operator` (from OperatorHub, or by deleting its Subscription and CSV), then delete its `minfo` on the `global` cluster: + +```bash +kubectl get subscriptions.operators.coreos.com -n -o yaml > /tmp/sub-rds-operator.bak.yaml +kubectl get csv -n -o yaml > /tmp/csv-rds-operator.bak.yaml +kubectl delete subscriptions.operators.coreos.com -n +kubectl delete csv -n +kubectl get minfo -o yaml > /tmp/minfo-rds-operator.bak.yaml +kubectl delete minfo +``` + +If RDS-managed instances run on the cluster, install the `rds-operator` version that matches the new platform version once the upgrade is complete. + +### Cause 9: Argo CD is not ready after upgrade.sh completes + +No change is needed at this point. Start the `global` cluster upgrade from the upgrade page, then check Argo CD again: + +```bash +kubectl get minfo | grep -i argo +kubectl get pods -A | grep -i argocd | grep -Ev 'Running|Completed' +``` + +If the Argo CD `minfo` is Failed after the global upgrade and the ARS error names an operator build other than the one in the upgrade package, use Cause 3. + +## Diagnostic Steps + +1. Note the component the upgrade page waits on, and whether it is installed through OLM (a CatalogSource, an operator package, or a CSV). +2. Read the OLM and operator objects on the stuck cluster: + + ```bash + kubectl get pods -n cpaas-system | grep -E 'olm-registry|olm-operator|catalog-operator|packageserver' + kubectl get catalogsource -A + kubectl get csv -A | grep -Ev 'Succeeded' + kubectl get rpch | grep -i olm + kubectl get jobs -A | grep -Ei 'check|wait' + helm ls -A | grep -i katanomi + ``` + +3. Match the first finding: + +| What you see | Cause | +| --- | --- | +| `unable to load in-cluster config` in an olm-registry Pod, and an `rpch` targets `olm-registry-config` | 1 | +| ARS unready after an operator upgrade; the version-check Job still quotes the previous version | 2 | +| Argo CD `minfo` Failed, empty wait Job, an operator build other than the one in the upgrade package is loaded | 3 | +| Katanomi phase, Helm release `katanomi-global`, CatalogSource `stable` missing | 4 | +| `bundles outdated` with the mesh images already present | 5 | +| Stuck at Tekton, both the v3 `tekton-operator` and the v4 Pipeline operator installed | 6 | +| `redis-operator` processing, new-version banner in OperatorHub, Subscription `installPlanApproval: Manual` | 7 | +| `rds-operator-updater` stuck, or several components waiting for `rds-operator` | 8 | +| Argo CD not ready right after `bash upgrade.sh`, before the global upgrade | 9 | + +4. After the change, the operator's CSV should reach `Succeeded`, the ARS should become Ready, and the upgrade page should move to the next stage. + +A message such as `redis-operator is not ready for version ` while another component (for example CoreDNS) is being upgraded, with the installed Redis operator already on a newer version, is not Cause 7. Compare the installed CSVs, the `minfo` versions, and `installedCSV` against `currentCSV` on the Subscription before acting: + +```bash +kubectl get csv -A | grep -i redis-operator +kubectl get minfo -A | grep -iE 'coredns|redis-operator' +kubectl get subscriptions.operators.coreos.com -A -o yaml | grep -n -E 'name:|installPlanApproval|currentCSV|installedCSV|state' +``` diff --git a/docs/en/solutions/ALB_load_balancer_Pods_restart_during_an_ACP_upgrade_and_the_restart_cannot_be_deferred.md b/docs/en/solutions/ALB_load_balancer_Pods_restart_during_an_ACP_upgrade_and_the_restart_cannot_be_deferred.md new file mode 100644 index 000000000..24d86e825 --- /dev/null +++ b/docs/en/solutions/ALB_load_balancer_Pods_restart_during_an_ACP_upgrade_and_the_restart_cannot_be_deferred.md @@ -0,0 +1,81 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# ALB load balancer Pods restart during an ACP upgrade and the restart cannot be deferred + +## Issue + +During an ACP upgrade the LB (ALB) Pods restart automatically, and business traffic through the load balancer is interrupted while they restart. Application owners ask: + +- Is the LB restart during the upgrade necessary? +- Can the upgrade skip the restart, so that the LB Pods are restarted by hand later, in a business downtime window? + +The question usually comes up after a test upgrade or an earlier upgrade window, once the impact on business continuity has been noticed. + +## Root Cause + +### Cause 1: The LB restart is part of the ACP upgrade and cannot be avoided + +- The LB Pod restart, and the network interruption it causes during the ACP upgrade, cannot be avoided. +- The restart cannot be skipped during the upgrade and replaced by a manual restart later. +- The upgrade has to be performed inside a maintenance window coordinated with the business owners. + +This restart comes from upgrading the cluster that runs the LB. A different case is a business-cluster ALB that rolls while only `global` is upgraded, because its Deployment no longer matches the HelmRequest on `global`. That drift-induced restart can be removed before the next global upgrade (see "Local overrides reverted or blocking an ACP upgrade: ResourcePatch, ALB ConfigMap, chart and feature gate changes", Cause 8), but removing it does not remove the restart described here. + +## Resolution + +### Cause 1: The LB restart is part of the ACP upgrade and cannot be avoided + +Treat the LB interruption as an expected part of the upgrade and plan for it: + +1. Agree on a maintenance window with the business owners before the upgrade. Tell them that traffic through the LB is interrupted while its Pods restart, for every domain and port served by those ALBs. +2. Inventory the ALB instances on the cluster being upgraded and the Pods behind each one, so the owners of every entry point can be notified. +3. Back up each `alb2` object and any overwrite ConfigMap before the upgrade, so custom nginx settings stay recoverable. +4. Agree how the interruption will be measured (for example with the probe loop in Diagnostic Steps), so the window can be closed on evidence. +5. Run the upgrade inside that window. There is no supported option to skip the LB restart and do it by hand afterwards. +6. After the upgrade, confirm that every ALB Pod is Running and Ready on the new image, and that custom settings are still rendered, before reopening business traffic. + +Read-only commands for the inventory and the post-upgrade check: + +```bash +kubectl get alb2.v2beta1.crd.alauda.io -A +kubectl -n get alb2.v2beta1.crd.alauda.io -o yaml > -alb2.yaml +kubectl -n get deploy,pods -o wide | grep +kubectl -n get pods -l service_name=alb2- \ + -o custom-columns=NAME:.metadata.name,START:.status.startTime,IMAGES:.spec.containers[*].image +``` + +If an ALB port conflict exists before the window, clear it first; otherwise the restart that the upgrade performs fails. + +## Diagnostic Steps + +1. Identify which cluster is being upgraded and on which cluster the restarted LB runs. +2. If the LB runs on the cluster being upgraded, this article applies: the restart is expected and cannot be deferred. +3. If the LB runs on a business cluster that was not being upgraded (only `global` moved), compare its Deployment with the HelmRequest on `global`; that is the drift case, not this one: + + ```bash + kubectl get helmrequests.app.alauda.io -A + ``` + +4. If the LB did not come back after the upgrade restarted it, check for an ALB port conflict. + +During the upgrade, watch the ALB Pods restart and note when each one is Ready again: + +```bash +kubectl -n get pods -l service_name=alb2- -w +``` + +Measure the interruption from a client outside the cluster with a probe against one business URL: + +```bash +while true; do + printf '%s ' "$(date +%T)" + curl -sk -o /dev/null -m 3 -w '%{http_code}\n' https://app.example.com/ || echo fail + sleep 1 +done +``` diff --git a/docs/en/solutions/Argo_Rollouts_Kubeflow_Base_or_a_wrong-architecture_cluster_plugin_fails_to_install_or_upgrade_on_ACP.md b/docs/en/solutions/Argo_Rollouts_Kubeflow_Base_or_a_wrong-architecture_cluster_plugin_fails_to_install_or_upgrade_on_ACP.md new file mode 100644 index 000000000..a174fc648 --- /dev/null +++ b/docs/en/solutions/Argo_Rollouts_Kubeflow_Base_or_a_wrong-architecture_cluster_plugin_fails_to_install_or_upgrade_on_ACP.md @@ -0,0 +1,199 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "4.1" + - "4.2" + - "4.3" +--- + +# Argo Rollouts, Kubeflow Base, or a wrong-architecture cluster plugin fails to install or upgrade on ACP + +## Issue + +A specific cluster plugin fails to install, upgrade, or uninstall: + +- An x86 plugin was installed on an ARM cluster, its init container reports an architecture mismatch, and its deletion is stuck. +- Argo Rollouts fails to deploy on ACP 4.2.2 because the init container `copy-gwapi-plugin` cannot pull `rollouts-plugin-trafficrouter-gatewayapi:v1`. +- Kubeflow Base ends in Install Failed (`安装失败`) with `namespaces "kubeflow" not found` or `illegal base64 data`, or its web UI answers `RBAC denied`. +- A Kubeflow Base upgrade fails on a cluster whose Istio uses revision-based injection, and the `kubeflow-istio-ingressgateway` Pod was never injected. + +Related articles: "Cluster plugin install or upgrade stuck in chart sync: sentry, StatefulSet Forbidden, hook Job, or missing ModuleConfig" covers failures that are not specific to one plugin. + +## Root Cause + +### Cause 1: An x86 plugin was installed on an ARM cluster, and its cleanup BroadcastJob fails too + +On an ARM cluster (for example ACP 4.3.1 ARM), the multus plugin init container reports an architecture mismatch, and after the plugin is deleted from the console it stays deleting because the cleanup BroadcastJob fails on the same mismatch. An x86 build of the plugin was deployed; its cleanup job uses the same wrong-architecture image, so the uninstall cannot finish. The known recovery is to redeploy the cluster and then deploy the plugin build for the cluster's architecture. + +### Cause 2: Argo Rollouts on 4.2.2 references the wrong init container image tag + +The Pod `argo-rollouts-` in namespace `argo-rollouts` repeats a `Warning` event for its init container `copy-gwapi-plugin`: + +```text +Failed to pull image "/acp/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi:v1": failed to pull and unpack image ... failed to resolve reference ... Head "https:///v2/acp/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/manifests/v1": tls: failed to verify certificate: x509: certificate signed by unknown authority +``` + +This is a known issue in ACP 4.2.2: the init container of the argo-rollouts chart references the incorrect tag `v1`. The pull of that tag can surface as the `x509` error above; the cause is the tag. The workaround overrides the init container through `valuesOverride` on the argo-rollouts ModuleInfo of the affected cluster, pointing it to `rollouts-plugin-trafficrouter-gatewayapi:v0.6.0-6` in the cluster's registry. + +### Cause 3: Kubeflow Base install prerequisites are not met + +The Kubeflow Base cluster plugin (for example `v1.10.10`) ends in Install Failed with one of: + +```text +failed to create NetworkPolicy "aml-api-deploy/kubeflow": namespaces "kubeflow" not found +failed to create Secret "cpaas-https-certs/istio-system": Secret in version "v1" cannot be handled as a Secret: illegal base64 data at input byte 2272 +``` + +or the plugin installs and its web UI returns `RBAC denied`, also over NodePort and with `curl` without a token. + +Kubeflow Base needs Alauda Service Mesh (ASM) deployed in the same cluster, and its install form needs the istio ingress gateway HTTPS NodePort. Installing without a service mesh, creating a gateway by hand, or pointing `oidcRedirectURL` to that gateway's load balancer IP leads to these errors; creating the `kubeflow` namespace by hand only moves the failure to the Secret error. Kubeflow Base creates its own Service `kubeflow-istio-ingressgateway`, a NodePort Service (`80:30080/TCP,443:30443/TCP` with the NodePorts chosen in the form), so no manual gateway is needed. Other form rules: + +- `oidcRedirectURL` points to an existing cluster node or the cluster VIP. +- `pgStorageClass` must be a StorageClass in the same cluster. +- The form has no setting to keep components off GPU nodes; taint the GPU nodes instead. +- The plugin template supports only NodePort; LoadBalancer is not supported. +- The ASM console lists only `gatewaydeploys.asm.alauda.io` resources, so `kubeflow-istio-ingressgateway`, a plain Deployment created by the plugin, does not appear there. + +The `RBAC denied` web UI is a product limitation, not a fault: Kubeflow Base depends on both ASM and AML (Alauda AI). Deploying Kubeflow Base standalone, without AML, is not supported. + +### Cause 4: kubeflow-istio-ingressgateway has no istio.io/rev label in its Pod template + +The gateway Pod is not injected, so its image address is not rewritten, and the Kubeflow Base upgrade fails. The `kubeflow-istio-ingressgateway` Deployment is created by the Kubeflow chart (`helm.sh/chart-name: kfbase`, release `kfbase` in `cpaas-system`) and carries `istio.io/rev: ` (for example `1-22`) on the Deployment metadata. Its Pod template has `inject.istio.io/templates: gateway`, `sidecar.istio.io/inject: "true"`, and `app` / `istio` labels, but no `istio.io/rev` label. Service Mesh v1 Istio uses revision-based injection, so the Pod template needs `istio.io/rev: ` together with `sidecar.istio.io/inject: "true"`. Injection is what rewrites the gateway image address; adding the label to the Pod template is the fix. + +## Resolution + +### Cause 1: An x86 plugin was installed on an ARM cluster, and its cleanup BroadcastJob fails too + +Confirm the mismatch: + +```bash +kubectl get nodes -o custom-columns=NAME:.metadata.name,ARCH:.status.nodeInfo.architecture +kubectl get broadcastjob -A | grep -i +kubectl -n logs --all-containers --tail=50 +``` + +> **Warning:** Redeploying the cluster destroys every workload and all data on it. This cannot be undone. Precondition: only when an x86 build of the plugin was installed on an ARM cluster, the plugin is stuck deleting because its cleanup BroadcastJob fails on architecture, and the cluster's workloads and data are backed up or disposable. Rollback: none; restore workloads from backup after the redeploy. + +Redeploy the cluster, then deploy the plugin from a package built for the cluster's architecture. Before installing any plugin on an ARM or hybrid cluster, check that the package you push is the ARM or hybrid build. + +### Cause 2: Argo Rollouts on 4.2.2 references the wrong init container image tag + +Confirm the tag in the event, then, on `global`, find and save the argo-rollouts ModuleInfo of the affected cluster: + +```bash +kubectl -n argo-rollouts get pods +kubectl -n argo-rollouts describe pod | grep -E 'copy-gwapi-plugin|rollouts-plugin-trafficrouter-gatewayapi' +kubectl get moduleinfo | grep | grep argo-rollouts +kubectl get moduleinfo -o yaml > "$HOME/.before.yaml" +``` + +> **Warning:** `valuesOverride` changes the rendered argo-rollouts chart; the controller Pod is recreated with the overridden init container, and a wrong image or mount keeps it from starting. Precondition: only on ACP 4.2.2 when the argo-rollouts Pod fails to pull `rollouts-plugin-trafficrouter-gatewayapi:v1` for `copy-gwapi-plugin`. Rollback: remove the `valuesOverride` block again. + +```bash +kubectl edit moduleinfo +``` + +Add the override under `spec`, replacing `` with the image registry address of that cluster: + +```yaml +spec: + config: {} + valuesOverride: + acp/argoproj/chart/argo-rollouts: + controller: + initContainers: + - args: + - cp /bin/rollouts-plugin-trafficrouter-gatewayapi /plugins + command: + - /bin/sh + - -c + image: /acp/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi:v0.6.0-6 + name: copy-gwapi-plugin + resources: + requests: + cpu: 100m + memory: 64Mi + limits: + cpu: 200m + memory: 128Mi + ephemeral-storage: 500Mi + volumeMounts: + - mountPath: /plugins + name: gwapi-plugin +``` + +Check that the controller Pod starts: + +```bash +kubectl get pods -A | grep argo-rollouts +``` + +### Cause 3: Kubeflow Base install prerequisites are not met + +Check the prerequisites on the target cluster before reinstalling: + +```bash +kubectl get pods -A | grep -i istio +kubectl get gatewaydeploys.asm.alauda.io -A +kubectl get sc +kubectl get svc -A | grep -E '30443|30080' +kubectl get minfo | grep | grep -iE 'aml|asm|kubeflow' +``` + +Deploy Alauda Service Mesh in the same cluster first if it is missing. Pick two free NodePorts (no output from the `grep` above means 30080 and 30443 are free), choose a `pgStorageClass` from `kubectl get sc` on that cluster, and set `oidcRedirectURL` to an existing cluster node IP or the cluster VIP, not to a load balancer IP. + +> **Warning:** Deleting a gateway created by hand for Kubeflow removes any traffic routed through it. Uninstalling a failed Kubeflow Base removes what it partly deployed. Precondition: only when Kubeflow Base failed with the errors above and the gateway was created only for Kubeflow. Export it first with `kubectl -n get gatewaydeploys.asm.alauda.io -o yaml > "$HOME/.yaml"`. Rollback: `kubectl apply -f` the export. + +```bash +kubectl -n delete gatewaydeploys.asm.alauda.io +``` + +Uninstall the failed Kubeflow Base from Cluster Plugins, install it again with the values above, and check: + +```bash +kubectl get svc -A | grep kubeflow-istio-ingressgateway +kubectl -n kubeflow get deploy,pods +kubectl -n kubeflow-oauth2-proxy get deploy,pods +curl -k -s -o /dev/null -w '%{http_code}\n' https://:30443 +``` + +All `kubeflow` and `kubeflow-oauth2-proxy` Pods should reach `Running`. The UI is reached at `https://:`; keep the Service as NodePort. If the UI answers `RBAC denied` and AML is not installed on the cluster, install Kubeflow Base together with ASM and AML; standalone use is a product requirement, not a configuration change. + +### Cause 4: kubeflow-istio-ingressgateway has no istio.io/rev label in its Pod template + +```bash +kubectl get deploy -A | grep kubeflow-istio-ingressgateway +kubectl -n get deploy kubeflow-istio-ingressgateway -o jsonpath='{.metadata.labels.istio\.io/rev}{"\n"}{.spec.template.metadata.labels}{"\n"}' +kubectl -n get deploy kubeflow-istio-ingressgateway -o yaml > "$HOME/kubeflow-istio-ingressgateway.before.yaml" +``` + +A revision on the Deployment metadata and no `istio.io/rev` in the Pod template labels is this cause. + +> **Warning:** Changing the Pod template rolls the Kubeflow ingress gateway; Kubeflow UI traffic is interrupted until the injected Pod is Ready. Precondition: only when the mesh uses revision-based injection, the gateway Pod is not injected, and the Pod template lacks `istio.io/rev`; use the revision shown on the Deployment metadata. Rollback: `kubectl apply -f "$HOME/kubeflow-istio-ingressgateway.before.yaml"`. + +```bash +kubectl -n patch deploy kubeflow-istio-ingressgateway --type=merge -p '{"spec":{"template":{"metadata":{"labels":{"istio.io/rev":"","sidecar.istio.io/inject":"true"}}}}}' +kubectl -n get pods | grep kubeflow-istio-ingressgateway +``` + +Check that the new Pod runs the rewritten image, then retry the Kubeflow Base upgrade. + +## Diagnostic Steps + +```bash +kubectl get ars -A +kubectl get minfo | grep +kubectl get broadcastjob -A +kubectl get nodes -o custom-columns=NAME:.metadata.name,ARCH:.status.nodeInfo.architecture +``` + +- An architecture mismatch in both the plugin and its cleanup BroadcastJob: Cause 1. +- A pull failure for `rollouts-plugin-trafficrouter-gatewayapi:v1` on the `copy-gwapi-plugin` init container of Argo Rollouts on 4.2.2: Cause 2. +- Kubeflow Base failing on `namespaces "kubeflow" not found` or `illegal base64 data`, or answering `RBAC denied` without AML: Cause 3. +- A failed Kubeflow Base upgrade with an uninjected `kubeflow-istio-ingressgateway` whose Pod template has no `istio.io/rev`: Cause 4. + +A cluster plugin pushed with `violet push` is registered on `global` and offered to clusters by its affinity; look for it under the target cluster's Cluster Plugins rather than pushing it to each cluster. diff --git a/docs/en/solutions/Cluster_plugin_install_or_upgrade_stuck_in_chart_sync_sentry_StatefulSet_Forbidden_hook_Job_or_missing_ModuleConfig.md b/docs/en/solutions/Cluster_plugin_install_or_upgrade_stuck_in_chart_sync_sentry_StatefulSet_Forbidden_hook_Job_or_missing_ModuleConfig.md new file mode 100644 index 000000000..7f12d6c7f --- /dev/null +++ b/docs/en/solutions/Cluster_plugin_install_or_upgrade_stuck_in_chart_sync_sentry_StatefulSet_Forbidden_hook_Job_or_missing_ModuleConfig.md @@ -0,0 +1,175 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "4.1" + - "4.2" + - "4.3" +--- + +# Cluster plugin install or upgrade stuck in chart sync: sentry, StatefulSet Forbidden, hook Job, or missing ModuleConfig + +## Issue + +A cluster plugin page shows `Installing` (Chinese UI: `安装中`) or `Deployment Failed` (`安装失败`), and the AppRelease (`ars`) or ModuleInfo carries a chart error: + +- Every plugin, including Alauda AI Essentials, is stuck in Installing, and no Pod in the cluster is abnormal. +- `metallb ChartDownloadFailed [1/1] failed to download chart acp/chart-alauda-metallb-plugin-v4.0.12, err: calculating blob path for ReaderAt: cannot calculate blob path from invalid digest: invalid checksum digest length: invalid argument`. +- The log center plugin (for example Log Storage for Elasticsearch `v4.1.3` on `global`) stays Installing, the Kafka consumer-lag alert from rule `cpaas-log-rules` keeps firing (`Kafka 消费队列滞后增长速率 > 0 且持续 1 分钟`, "Kafka consumer lag growth rate > 0 for 1 minute"), and the event is `failed to apply StatefulSet "cpaas-system/cpaas-kafka": StatefulSet.apps "cpaas-kafka" is invalid: spec: Forbidden: updates to statefulset spec for fields other than ...`. +- `Installation failed:[1/1] chart ait/chart-base-central-v4.2.44-cn Upgrading` for Alauda Container Platform Central Base during platform deployment. +- A business cluster ARS (for example `base`, before a cluster upgrade) stays `syncing` for a long time. +- A plugin upgrade on a business cluster (for example Prometheus) fails, and the error looks like a missing certificate. + +`sentry` installs and upgrades plugin charts on each cluster and records the result on the AppRelease. `cluster-transformer` on `global` turns ModulePlugins into ModuleConfigs and reconciles ModuleInfo. + +Related articles: "Argo Rollouts, Kubeflow Base, or a wrong-architecture cluster plugin fails to install or upgrade on ACP" covers plugin-specific install failures. + +## Root Cause + +### Cause 1: sentry stopped processing charts + +All plugins stay Installing with no abnormal Pod, or an AppRelease reports `ChartDownloadFailed ... invalid checksum digest length: invalid argument`. No underlying cause has been identified. Restarting `sentry` returns the plugins from Installing to normal and restores the AppRelease sync. + +### Cause 2: StatefulSet spec Forbidden after storageSize was changed in ModuleInfo + +`sentry failed to upgrade chart ait/chart-alauda-log-center-v4.1.3, err: failed to upgrade chart for release "logcenter": failed to apply StatefulSet "cpaas-system/cpaas-kafka" ... spec: Forbidden` repeats for days, and the log center plugin stays Installing. + +The Kafka `storageSize` in the log center ModuleInfo was changed. That value feeds the `volumeClaimTemplates` of the existing `cpaas-kafka` StatefulSet, which Kubernetes does not allow to change in place, so every chart upgrade fails. Kafka consumer-lag alerts on `alauda_log/ALAUDA_LOG_TOPIC/` can fire at the same time. The fix is to redeploy the log plugins with the planned PVC capacity, which deletes the existing log data. + +### Cause 3: Job remove-keep-annotations times out and the chart never leaves Upgrading + +`Installation failed:[1/1] chart ait/chart-base-central-v4.2.44-cn Upgrading`, and the `sentry` log for release `base-central` in `cpaas-system` shows: + +```text +warning: Failed to update release base-central version 4: release: not found +ServiceAccount base-central-upgrade-hook has skipped +recreate Job "remove-keep-annotations" success +Watching for changes to Job remove-keep-annotations with timeout of 2m0s +Jobs active: 0, jobs failed: 0, jobs succeeded: 0 +``` + +The `release: not found` warning alone does not identify this cause; the deciding lines are the wait on the Job and `Jobs active: 0`. The chart upgrade waits on hook Job `remove-keep-annotations`, which never becomes active, so the wait times out on every retry. Deleting the Job and running its operations by hand lets the installation complete. + +### Cause 4: Business cluster ARS stays syncing because a BroadcastJob is incomplete + +A business cluster ARS (or its `base` component) stays in `syncing` while a BroadcastJob of that cluster is incomplete: `update-containerd-config` with stuck jobs, or `upgrade-sysctl-config` with `FAILED 1` and an `Evicted` Pod. Adding a dummy value to the ARS values only triggers the same BroadcastJob again. Both forms are covered in "base AppRelease stays failed because the upgrade-sysctl-config BroadcastJob cannot finish on a large cluster on ACP". + +### Cause 5: ModuleInfo Reconcile error: ModuleConfig.cluster.alauda.io "" not found + +A plugin upgrade on a business cluster fails, and the `cluster-transformer` log shows: + +```text +ERROR controllers/moduleinfo_controller.go:61 ModuleInfo Reconcile '' error: ModuleConfig.cluster.alauda.io "prometheus-v3.16.11-def68b41a866" not found +``` + +The ModuleConfig of the old version (3.16) was deleted, and the chart version in the ModulePlugin `spec.mainChart` no longer matches any existing ModuleConfig, so ModuleInfo reconciliation fails. Setting that chart `version` to the version of the ModuleConfig that exists fixes it. + +## Resolution + +### Cause 1: sentry stopped processing charts + +```bash +kubectl get ars -A +kubectl get pods -A | grep sentry +kubectl -n cpaas-system logs --tail=300 > "$HOME/sentry-before-restart.log" +``` + +> **Warning:** Restarting `sentry` pauses chart install and sync on that cluster until the new Pod is Ready. Precondition: only when every plugin is stuck Installing with no abnormal Pod, or an AppRelease shows `ChartDownloadFailed` with `invalid checksum digest length`. Rollback: none needed; the Deployment recreates the Pod. + +```bash +kubectl -n cpaas-system delete pod +kubectl -n cpaas-system get pods -w | grep sentry +kubectl get ars -A +``` + +Plugins should leave Installing and the AppRelease should sync. Keep the saved `sentry` log for Alauda support if the problem returns. + +### Cause 2: StatefulSet spec Forbidden after storageSize was changed in ModuleInfo + +```bash +kubectl get ars -A | grep -i logcenter +kubectl -n cpaas-system describe ars logcenter | grep -A3 ChartSynced +kubectl get minfo | grep -i log +kubectl get minfo -o yaml | grep -n -i storageSize +kubectl -n cpaas-system get sts cpaas-kafka -o jsonpath='{.spec.volumeClaimTemplates[*].spec.resources.requests.storage}{"\n"}' +``` + +A `storageSize` in ModuleInfo that differs from the StatefulSet `volumeClaimTemplates` storage is this cause. The chart upgrade can only apply while the two values match or after the StatefulSet is recreated, which here means redeploying the plugin. + +> **Warning:** Redeploying the log plugin with new PVC sizes deletes the existing log storage data (the `cpaas-kafka` volumes and the stored logs). This cannot be undone. Precondition: only when the event shows the `cpaas-kafka` StatefulSet `spec: Forbidden` error, the `storageSize` change in ModuleInfo is confirmed, and the owner of the logs has confirmed the existing log data can be deleted. Rollback: none; export anything you must keep before you start. + +Uninstall the log collection plugin, confirm its PVCs are gone, and install it again with the planned PVC capacity: + +```bash +kubectl -n cpaas-system get pvc | grep -i kafka +``` + +### Cause 3: Job remove-keep-annotations times out and the chart never leaves Upgrading + +Find the Job in the `sentry` log and export it before touching it: + +```bash +kubectl -n cpaas-system logs --tail=500 | grep -E 'base-central|remove-keep-annotations|Watching for changes to Job|Jobs active' +kubectl get jobs -A | grep remove-keep-annotations +kubectl -n describe job remove-keep-annotations +kubectl -n get job remove-keep-annotations -o yaml > "$HOME/remove-keep-annotations.yaml" +``` + +> **Warning:** Deleting the Job removes the step the chart waits on; the operations that Job performs are not done until you run them yourself. Precondition: only when the chart is stuck at `chart ait/chart-base-central- Upgrading`, the `sentry` log shows the wait on Job `remove-keep-annotations` timing out with `Jobs active: 0`, and the Job is exported. Rollback: re-create the Job from the export if the chart must run it again. + +```bash +kubectl -n delete job remove-keep-annotations +``` + +Run the operations of the exported Job by hand: its container `command` and `args` are under `.spec.template.spec.containers[]` in the export, and the ServiceAccount it runs as is `.spec.template.spec.serviceAccountName`. Then check that the chart leaves `Upgrading`. Use the same procedure for another hook Job only when the `sentry` log shows that Job's wait timing out. + +### Cause 4: Business cluster ARS stays syncing because a BroadcastJob is incomplete + +```bash +kubectl get job,bcj -n cpaas-system +kubectl -n cpaas-system logs --tail=500 | grep -E 'update-containerd-config|upgrade-sysctl-config' +``` + +Follow "base AppRelease stays failed because the upgrade-sysctl-config BroadcastJob cannot finish on a large cluster on ACP": delete the `Evicted` Pod for a `FAILED` `upgrade-sysctl-config`, or raise `parallelism` for a stuck `update-containerd-config`, each with its Warning. + +### Cause 5: ModuleInfo Reconcile error: ModuleConfig.cluster.alauda.io "" not found + +```bash +kubectl -n cpaas-system logs deploy/cluster-transformer --tail=300 | grep -i 'ModuleConfig.cluster.alauda.io' +kubectl get mconfig | grep -i +kubectl get mplugin -o yaml > "$HOME/mplugin-.before.yaml" +grep -n -A8 'mainChart' "$HOME/mplugin-.before.yaml" +``` + +> **Warning:** The chart version in `spec.mainChart` decides which chart the platform installs for this plugin. Precondition: only when the `cluster-transformer` log reports `ModuleConfig.cluster.alauda.io "" not found` for the referenced version, and you set the version to one that `kubectl get mconfig` lists. Rollback: `kubectl apply -f "$HOME/mplugin-.before.yaml"`. + +```bash +kubectl edit mplugin +``` + +Set the `version` of the chart under `spec.mainChart` to the version of the existing ModuleConfig, save, and retry the upgrade. + +## Diagnostic Steps + +```bash +kubectl get ars -A +kubectl get minfo | grep +kubectl get pods -A | grep -E 'sentry|cluster-transformer' +kubectl -n cpaas-system logs deploy/cluster-transformer --tail=300 | grep -i error +kubectl get broadcastjob -A +``` + +- All plugins stuck with healthy Pods, or `invalid checksum digest length`: Cause 1. If `sentry` itself shows `OOMKilled` restarts, raise its memory instead of only restarting it. +- `StatefulSet ... spec: Forbidden` after a `storageSize` change: Cause 2. +- `chart ... Upgrading` with `Watching for changes to Job remove-keep-annotations` and `Jobs active: 0` in the `sentry` log: Cause 3. +- An ARS in `syncing` with an incomplete BroadcastJob (`update-containerd-config`, or `upgrade-sysctl-config` with an `Evicted` Pod): Cause 4. +- `ModuleInfo Reconcile ... ModuleConfig.cluster.alauda.io "" not found` from `cluster-transformer` during a plugin upgrade: Cause 5. + +For a release that reports `release: not found`, check whether Helm still has the release record before acting; a missing revision Secret matches that warning, and the hook Job lines decide Cause 3: + +```bash +kubectl get secret -n cpaas-system | grep 'sh.helm.release.v1.' +``` diff --git a/docs/en/solutions/Container_runtime_conflicts_on_ACP_nodes_Docker_podman_docker-runc_and_overwritten_runtime_binaries.md b/docs/en/solutions/Container_runtime_conflicts_on_ACP_nodes_Docker_podman_docker-runc_and_overwritten_runtime_binaries.md new file mode 100644 index 000000000..57a54d4af --- /dev/null +++ b/docs/en/solutions/Container_runtime_conflicts_on_ACP_nodes_Docker_podman_docker-runc_and_overwritten_runtime_binaries.md @@ -0,0 +1,249 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" +--- + +# Container runtime conflicts on ACP nodes: Docker, podman, docker-runc, and overwritten runtime binaries + +## Issue + +A node that is being added, or that is already in the cluster, has a container runtime that no longer matches what the platform installed. Typical reports: + +- Docker was installed on a node (by hand or by mistake) and the node became abnormal; containerd disappeared from the node. +- After a reboot the node never becomes Ready, or Pods fail with `FailedCreatePodSandBox`. +- calico does not start after a kernel and NIC driver upgrade. +- The CRI version changed by itself after a reboot. +- A deployment or node add fails because of conflicting packages or reports a `runc` error. +- On a cluster created by TKE, a new node stays at `EnsureNodeReady` with cgroup errors in the network plugin events. + +The platform installs its own containerd (or Docker with `cri-docker` on Docker clusters), `runc`, `crictl`, `kubelet`, and `kubectl` in `/usr/bin` and expects exactly those versions. Installing Docker or podman from the OS repositories, an OS upgrade, or packages preinstalled in the machine image replace or remove them. The files involved on a containerd node are `containerd`, `containerd-shim`, `containerd-shim-runc-v1`, `containerd-shim-runc-v2`, `ctr`, `ctd-decoder`, `crictl`, `critest`, `containerd-stress`, `runc`, `kubelet`, and `kubectl` in `/usr/bin`, plus `/etc/containerd/config.toml` and the containerd unit file. Same-named copies in `/usr/local/bin` are found first on `PATH`. + +## Root Cause + +### Cause 1: Docker was installed on a containerd node + +A `docker-ce` installation on a node whose cluster uses containerd has different effects; compare the node with a healthy one before choosing a step: + +- Docker is installed and the node is abnormal, but containerd still runs and its binaries are identical to a healthy node. +- `yum install docker-ce` uninstalled containerd (for example on an ACP 4.0 node). +- After a reboot the node does not return to Ready: kubelet and the runtime services fail to start because `runc` and other binaries in `/usr/bin` are now the wrong versions. +- The OVN components on the node do not run (`ovs-ovn` produces no log), and Pods fail with `FailedCreatePodSandbox`. +- On a node whose overlay and underlay traffic share one NIC, deleting containers during the cleanup invalidated the `pr-bridge` bridge: the node lost its network and SSH and went `NotReady` until it was rebooted. Such a node can be recovered by removing it from the cluster, cleaning it up, and adding it again. + +### Cause 2: An OS, kernel, or driver upgrade overwrote runc and containerd, and podman was installed + +On an ACP 4.0 ARM node, calico does not start after the kernel and NIC driver are upgraded. The `runc` and `containerd` binaries in `/usr/bin` differ from those on a healthy node, and `podman` is installed. The OS packages involved on a Kylin node include `containers-common`, `containernetworking-plugins`, `docker-runc`, `podman`, and `container-selinux`. + +### Cause 3: dockerd changed version after a reboot on a Docker-runtime cluster + +The business cluster was not upgraded, but after a node reboot the CRI is a different version and `cri-docker` is broken: `dockerd` on the node no longer matches the version healthy nodes run (for example `20.10.27-2`). Other signs: + +- Workers are `NotReady`, with node events alternating between `NodeReady` and `NodeNotReady`; no Calico Pod is failing. +- `kubectl get pod -A | grep upgra` lists `--upgrade-cri` Pods in `kube-system` at `0/1`, `ContainerCreating` or `Pending`. +- The node's machine resource in `cpaas-system` on `global` has `cri.type: docker` and the Docker version in its spec, but `cri.version: Unknown` and `containerRuntimeVersion: docker://Unknown` in its status. + +### Cause 4: Docker, docker-runc, or other conflicting packages were preinstalled before ACP installed Kubernetes + +- A 4.2.2 deployment on Kylin Linux Advanced Server V10 fails until conflicting packages are removed from the node, for example `docker-runc`, `podman`, and `containernetworking-plugins`; a separate `/usr/local/bin/runc` reports `runc version 1.0.0-rc3`. +- A node add or cluster creation (for example on 4.3.1) reports a `runc` error because a `docker-runc` package from the machine image is installed. The check line is `/usr/bin服务: 服务项检查失败: /usr/local/bin/runc 不在/usr/bin下` ("service check failed: /usr/local/bin/runc is not under /usr/bin"); `rpm -qa | grep docker` returns `docker-runc-` while `systemctl -a | grep docker` shows only `var-lib-docker.mount`. +- ACP cannot reuse a Docker already installed on the host. ACP deploys the runtime itself, and a Docker installed beforehand is not compatible with the platform deployment. + +### Cause 5: A TKE-created cluster provisions new nodes with containerd while the cluster runs Docker + +Adding a node to a TKE workload cluster stays at `EnsureNodeReady`. The node is `NotReady`, calico and kube-proxy do not start, and their events report a cgroup incompatibility. `global` and the workload cluster use Docker, but the new node gets containerd, because the cluster object on `global` carries the annotation `cpaas.io/container-runtime-version: containerd://1.6.28-4,docker://20.10.24-2`. Removing only `containerd://1.6.28-4` from that annotation does not help. + +The new node shows `Container Runtime Version: containerd://1.6.28-4` and the event `InvalidDiskCapacity` (`invalid capacity 0 on image filesystem`), while every other node runs `docker://...`. `calico-node` stays `Init:0/5`, `kube-proxy` `ContainerCreating`, with repeated `FailedCreatePodSandbox`: + +```text +Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: expected cgroupsPath to be of format "slice:prefix:name" for systemd cgroups, got "/kubepods/burstable/pod.../..." instead: unknown +``` + +## Resolution + +### Cause 1: Docker was installed on a containerd node + +Compare the node with a healthy node of the same OS, architecture, and platform version: + +```bash +kubectl get node -o wide # CONTAINER-RUNTIME must be containerd:// +rpm -qa | grep -i -E 'docker|containerd|runc|podman' +systemctl list-unit-files | grep -i -E 'docker|containerd' +systemctl is-active containerd docker +md5sum /usr/bin/runc /usr/bin/containerd /usr/bin/kubelet +rpm -qf /usr/bin/containerd /usr/bin/runc +``` + +**containerd still running and the checksums match a healthy node.** Stop and disable Docker, remove the Docker package with the OS package manager, and check that the removal did not delete the containerd unit file or `/etc/containerd/config.toml`. `runc -v` must then match a healthy node. + +> **Warning:** Removing Docker stops every container that runs under `dockerd`. Review the dependency list the package manager prints before confirming, and stop if it includes packages the platform installed. Reinstalling the package is the only rollback. + +```bash +sudo systemctl stop docker docker.socket +sudo systemctl disable docker docker.socket +sudo yum remove docker-ce docker-ce-cli +runc -v +ls -l /etc/containerd/config.toml; systemctl cat containerd | head +``` + +**Runtime binaries differ from a healthy node, or containerd is gone.** Restore them as follows. + +> **Warning:** Overwriting the runtime binaries replaces the container runtime of the node. Stop kubelet and containerd first, because a busy file cannot be overwritten. With the default `KillMode=process`, running containers keep running under `containerd-shim`, but no container can be created, started, or stopped and no image pulled until both are started again. Apply only when the checksums differ from a healthy node of the same version and architecture; an amd64 binary does not run on an ARM node. Keep the replaced files in `/root/runtime-bin-backup` for rollback. + +```bash +# on the affected node +sudo yum remove -y docker-ce docker-ce-cli +sudo systemctl stop kubelet +sudo systemctl stop containerd +sudo mkdir -p /root/runtime-bin-backup +for i in ctd-decoder containerd-shim-runc-v2 ctr containerd-shim crictl containerd-shim-runc-v1 critest containerd-stress runc containerd kubelet kubectl; do + [ -e /usr/bin/$i ] && sudo cp -a /usr/bin/$i /root/runtime-bin-backup/ +done +``` + +```bash +# on the healthy node +for i in ctd-decoder containerd-shim-runc-v2 ctr containerd-shim crictl containerd-shim-runc-v1 critest containerd-stress runc containerd kubelet kubectl +do + scp -P /usr/bin/$i root@:/usr/bin +done +``` + +If `scp` fails because containerd still holds the files, disable containerd on the affected node, reboot it, and copy again. + +> **Warning:** With containerd disabled the node runs no Pods after the reboot, and the reboot drops every session on it. Do this only on the node being repaired, only when the copy fails on busy files, and from a console that does not depend on the node's network. Rollback: `sudo systemctl enable --now containerd`. + +```bash +sudo systemctl disable containerd +sudo reboot +# after the copy from the healthy node succeeded +sudo systemctl enable --now containerd +``` + +```bash +sudo systemctl start containerd kubelet +systemctl is-active containerd kubelet +kubectl get node +``` + +Optionally, the restore sequence can also remove the node's containers before the copy, between stopping kubelet and stopping containerd (`crictl` needs containerd running): + +> **Warning:** Deleting every container on the node stops all its Pods, including the CNI agents. On a node whose overlay and underlay share one NIC, it invalidates `pr-bridge`, the node loses its network and SSH, and only a reboot brings it back. Run it only from a console that does not depend on the node's network, and only when overlay and underlay do not share a NIC on that node; otherwise skip it. It cannot be undone; the Pods are recreated after kubelet starts again. + +```bash +sudo systemctl stop kubelet +sudo crictl ps -a -q | xargs -r sudo crictl rm -f +sudo systemctl stop containerd +``` + +If the node has lost its network (the shared-NIC `pr-bridge` case), reboot it from a console first. If the node cannot be repaired in place, remove it from the cluster, run the node cleanup script, and add it again. + +### Cause 2: An OS, kernel, or driver upgrade overwrote runc and containerd, and podman was installed + +Uninstall `podman`, then restore the binaries exactly as in Cause 1: + +```bash +rpm -qa | grep -i podman +sudo yum remove -y podman +md5sum /usr/bin/runc /usr/bin/containerd +``` + +After the binaries are back, check that calico starts: `kubectl get pods -A -o wide | grep `. + +### Cause 3: dockerd changed version after a reboot on a Docker-runtime cluster + +```bash +dockerd --version; docker version --format '{{.Server.Version}}' +# on the workload cluster +kubectl get pod -A | grep upgrade-cri +# on global: the machine resource of the node +kubectl get machine -n cpaas-system | grep +kubectl get machine -n cpaas-system -o yaml | grep -n -E 'cri|containerRuntimeVersion' +``` + +`docker://Unknown` in the machine status while the spec names the Docker version matches this cause. Copy `docker` and `dockerd` from a healthy node and restart Docker. + +> **Warning:** Restarting Docker restarts every container on the node. Apply only when the version differs from the healthy nodes. Keep the replaced binaries; rollback is copying them back and restarting Docker. + +```bash +sudo systemctl stop kubelet docker +sudo mkdir -p /root/runtime-bin-backup +sudo cp -a /usr/bin/docker /usr/bin/dockerd /root/runtime-bin-backup/ +# on the healthy node +scp /usr/bin/docker /usr/bin/dockerd root@:/usr/bin/ +# on the affected node +sudo systemctl start docker kubelet +systemctl status cri-docker --no-pager +``` + +`cri-docker` recovers after Docker restarts. + +### Cause 4: Docker, docker-runc, or other conflicting packages were preinstalled before ACP installed Kubernetes + +Check the host before the add or the deployment, and remove what the platform did not install: + +```bash +rpm -qa | grep -i docker +systemctl -a | grep -i docker +for x in docker docker-client docker-common docker-engine docker-proxy docker-runc podman-docker podman containernetworking-plugins apptainer containerd kubernetes kubernetes-master kubernetes-node kubernetes-client kubernetes-kubeadm; do + rpm -qa | grep -F "$x" +done | sort -u +ls -l /usr/local/bin/runc 2>/dev/null && /usr/local/bin/runc -v +``` + +> **Warning:** Removing packages from the host cannot be undone by the platform. Do it only on a machine that is not yet in a cluster, and only for runtime packages (`docker-runc`, a preinstalled `docker-ce`, `podman`, `containernetworking-plugins`). Make sure no other service on the host depends on them. + +```bash +sudo yum remove -y docker-runc podman containernetworking-plugins +``` + +For a preinstalled Docker, uninstall it, remove its leftovers, and let the platform deploy the runtime during the Kubernetes installation. After `docker-runc` is removed the `runc` error disappears; run the node add or the deployment again. + +### Cause 5: A TKE-created cluster provisions new nodes with containerd while the cluster runs Docker + +On `global`, read the runtime annotations of the cluster and compare with the runtime of its existing nodes: + +```bash +kubectl get node -o wide # on the workload cluster: CONTAINER-RUNTIME of the existing nodes +kubectl get cls -o yaml | grep -n -E 'runtime|container-runtime' +``` + +> **Warning:** The runtime annotations decide which runtime every later node of that cluster gets. Change them only when the existing nodes run Docker and new nodes come up with containerd. Save the object first; rollback is restoring the saved annotations. + +Set both runtime annotations to the Docker runtime that the existing nodes run (the `CONTAINER-RUNTIME` value above), then remove the new node, run `cleanup.sh` on it, and add it again: + +```bash +kubectl get cls -o yaml > /tmp/cls-backup.yaml +kubectl annotate cls --overwrite \ + cpaas.io/container-runtime-version='docker://' \ + cpaas.io/container-runtime='docker://' +``` + +## Diagnostic Steps + +On the affected node and on a healthy node of the same OS and version: + +```bash +rpm -qa | grep -i -E 'docker|containerd|runc|podman' || dpkg -l | grep -i -E 'docker|containerd|runc|podman' +for b in runc containerd ctr crictl kubelet kubectl; do printf '%s ' "$b"; md5sum /usr/bin/$b | cut -d' ' -f1; done +which -a runc containerd +systemctl is-enabled containerd kubelet docker 2>/dev/null +journalctl -u containerd -u kubelet -n 50 --no-pager +``` + +- A Docker package on a containerd node: Cause 1. Whether containerd is still active and the checksums match decides which branch. +- `podman` installed and binaries changed after a kernel or driver upgrade: Cause 2. +- A Docker-runtime cluster whose node reports `docker://Unknown` after a reboot: Cause 3. +- A host that is not yet in a cluster with `docker-runc`, `podman`, or `containernetworking-plugins` installed, or a `runc` error on add: Cause 4. +- A new node on a Docker cluster that comes up with `containerd://` and the `cgroupsPath` error: Cause 5. + +A node that is `NotReady` after a reboot only because the runtime and kubelet were not enabled at boot is simpler: start and enable them. + +```bash +sudo systemctl enable --now docker kubelet +``` diff --git a/docs/en/solutions/Control-plane_Pods_do_not_start_with_mkdir_var_log_pods_file_exists_after_containerd_directories_are_lost_on_ACP.md b/docs/en/solutions/Control-plane_Pods_do_not_start_with_mkdir_var_log_pods_file_exists_after_containerd_directories_are_lost_on_ACP.md new file mode 100644 index 000000000..e958299ae --- /dev/null +++ b/docs/en/solutions/Control-plane_Pods_do_not_start_with_mkdir_var_log_pods_file_exists_after_containerd_directories_are_lost_on_ACP.md @@ -0,0 +1,82 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.1" + - "4.2" + - "4.3" +--- + +# Control-plane Pods do not start with mkdir /var/log/pods: file exists after containerd directories are lost on ACP + +## Issue + +Platform access is abnormal and a cluster shows an abnormal status. On a master, kubelet and containerd are both `active (running)`, but `crictl ps -a` prints nothing at all: no container exists on the node. The kubelet log repeats: + +- `"Error getting node" err="node \"\" not found"` and `failed to list *v1.CSIDriver` +- `"Failed to create log directory for pod" err="mkdir /var/log/pods: file exists"` for `kube-system/etcd` and `kube-system/kube-controller-manager` +- `"CreatePodSandbox for pod failed" err="mkdir /var/log/pods: file exists"` and `"Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" ..."` + +Related articles: "etcd and kube-apiserver do not start on a master after runc or containerd is replaced on ACP" covers masters whose runtime binaries were replaced. + +## Root Cause + +### Cause 1: /var/lib/containerd lost its pods and io.containerd.grpc.v1.cri directories + +Under `/var/lib/containerd` on the master, the `pods` and `io.containerd.grpc.v1.cri` directories are missing. Without them the static control-plane Pods cannot create their sandboxes, and nothing runs on the node. How the directories are lost is not known. + +`mkdir` returns `file exists` when the path already exists as something other than a directory, for example a symbolic link whose target has disappeared. On a node where `/var/log/pods` is a link into `/var/lib/containerd`, losing that target produces exactly this error. + +Stopping kubelet and containerd, creating the two directories by hand, and starting both services again lets the containers start; the cluster status returns to Normal. + +## Resolution + +### Cause 1: /var/lib/containerd lost its pods and io.containerd.grpc.v1.cri directories + +Apply only when `crictl ps -a` is empty, kubelet logs `mkdir /var/log/pods: file exists` for the static Pods, and `ls` confirms that the two directories are missing. Compare with a healthy master of the same cluster first: + +```bash +systemctl is-active kubelet containerd +sudo crictl ps -a +sudo ls -la /var/lib/containerd +ls -ld /var/log/pods +readlink -f /var/log/pods || true +``` + +On the healthy master, note the owner and mode of the same entries: + +```bash +sudo stat -c '%U:%G %a %n' /var/lib/containerd/pods /var/lib/containerd/io.containerd.grpc.v1.cri +ls -ld /var/log/pods +``` + +> **Warning:** Stopping kubelet and containerd stops container management on this master until both are started again. Do it only when this cause is confirmed: no containers in `crictl ps -a`, the two directories missing, and the `mkdir /var/log/pods: file exists` error. Creating empty directories does not restore any container data that was in them; kubelet recreates the static Pods from their manifests. Rollback: start containerd and kubelet, and `sudo rmdir` a directory you created by mistake while it is still empty. + +```bash +sudo systemctl stop kubelet containerd +sudo mkdir -p /var/lib/containerd/pods /var/lib/containerd/io.containerd.grpc.v1.cri +# set owner and mode to the values stat printed on the healthy master, for example: +sudo chown : /var/lib/containerd/pods /var/lib/containerd/io.containerd.grpc.v1.cri +sudo chmod /var/lib/containerd/pods /var/lib/containerd/io.containerd.grpc.v1.cri +sudo systemctl start containerd +sudo systemctl start kubelet +sudo crictl ps -a +``` + +The etcd, kube-apiserver, and kube-controller-manager containers should appear in `crictl ps -a` within a few minutes, and the cluster status should return to Normal. + +## Diagnostic Steps + +```bash +systemctl is-active kubelet containerd +sudo crictl ps -a +sudo journalctl -u kubelet --since "30 min ago" | grep -E 'mkdir /var/log/pods|CreatePodSandbox' | tail -20 +sudo ls -la /var/lib/containerd +ls -ld /var/log/pods; readlink -f /var/log/pods || true +``` + +- kubelet and containerd running, `crictl ps -a` empty, `mkdir /var/log/pods: file exists`, and `/var/lib/containerd/pods` missing: this cause. +- Containers exist but etcd exits with `StartError`, or kubelet logs `Failed to delete cgroup paths`: the runtime binaries were replaced; see the related article. +- `/var/log/pods` is a link whose target exists and the directories are present: this cause does not apply. diff --git a/docs/en/solutions/DevOps_v3_toolchain_entry_missing_after_an_upgrade_Katanomi_MonitorDashboard_ownership_conflict_or_missing_instance.md b/docs/en/solutions/DevOps_v3_toolchain_entry_missing_after_an_upgrade_Katanomi_MonitorDashboard_ownership_conflict_or_missing_instance.md new file mode 100644 index 000000000..1db7faf9c --- /dev/null +++ b/docs/en/solutions/DevOps_v3_toolchain_entry_missing_after_an_upgrade_Katanomi_MonitorDashboard_ownership_conflict_or_missing_instance.md @@ -0,0 +1,146 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.1" + - "4.2" +--- + +# DevOps v3 toolchain entry missing after an upgrade: Katanomi MonitorDashboard ownership conflict or missing instance + +## Issue + +- After a platform upgrade (for example 3.18.2 to 4.1.7) the DevOps pages are gone from the platform menu, `katanomi-operator` is already upgraded (for example v3.20), and the `katanomi-global` instance shows an error. +- Clicking Toolchain (Chinese UI: `工具链`) in the console redirects to a different page, and no toolchain operation can be performed. +- DevOps v3 is installed and there is no access entry for the toolchain. +- The `Alauda DevOps v3` operator is available on ACP 4.2 and the DevOps tool feature still has to be turned on. +- DevOps v3 is being retired in favor of DevOps v4, and the DevOps v3 user view has to be removed from the console, not only hidden as a product entry. + +DevOps v3 consists of the `Alauda DevOps v3` operator (`katanomi-operator`, Katanomi, the DevOps core component built on Tekton) and a `Katanomi` instance, both on the `global` cluster. + +## Root Cause + +### Cause 1: UpgradeError, MonitorDashboard "devops-api" exists and cannot be imported into the current release + +The `katanomi-global` instance of `katanomi-operator` in OperatorHub on `global` carries a condition with `reason: UpgradeError`: + +```text +failed to upgrade release: rendered manifests contain a resource that already exists. Unable to continue with update: MonitorDashboard "devops-api" in namespace "cpaas-system" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "katanomi-global"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "cpaas-system" +``` + +The DevOps v3 Helm release renders three `MonitorDashboard` (`v1alpha1`) objects in `cpaas-system`: `devops-api`, `devops-apiserver`, and `devops-controller`, each labeled `cpaas.io/dashboard.folder: devops`. Objects with those names already exist without the Helm ownership label `app.kubernetes.io/managed-by: Helm` and the annotations `meta.helm.sh/release-name` / `meta.helm.sh/release-namespace`. Helm refuses to take over an object that another owner created, so the release upgrade fails, the instance stays abnormal, and Toolchain redirects elsewhere. This is a product defect. + +The instance can show `Deployed` (`UpgradeSuccessful`) and `Running` (`RunningSuccessful`) conditions next to the `ReleaseFailed` condition with reason `UpgradeError`; the `ReleaseFailed` condition is the one that matters. Deleting the dashboard named in the message (`devops-api`) lets the operator recreate it with Helm ownership, and the DevOps pages return. + +The same conflict can block a fresh install: the `Katanomi` instance is not installed because of a duplicate `MonitorDashboard`, and the toolchain entry is missing. Deleting the pre-existing `MonitorDashboard` lets the installation recover. + +### Cause 2: The DevOps v3 operator is installed but no Katanomi instance exists + +The `Alauda DevOps v3` operator is deployed on `global` and the DevOps tool feature is still absent. The toolchain comes from a `Katanomi` instance, not from the operator alone. The Create Katanomi form proposes name `katanomi-sample` and namespace `katanomi-operator`, and has a required `External URL` field (its hint calls it the plugin webhook address) and a `Service` section with `API Port` (default `32000`), `Plugin Port` (default `32001`), and `Type` (default `NodePort`). `External URL` is the `global` cluster access address (the platform address). + +### Cause 3: The DevOps feature card stays because the Katanomi instance and the DevOps v3 operator are still installed + +Hiding the DevOps product entry leaves the DevOps v3 user view in place, because the view is provided by `katanomi-operator` and its `katanomi-sample` instance. Deleting the instance and then uninstalling DevOps v3 from OperatorHub removes the DevOps feature card. Leftover objects of the katanomi CRDs can then be cleaned up from the command line; the katanomi CRDs themselves can stay. + +## Resolution + +### Cause 1: UpgradeError, MonitorDashboard "devops-api" exists and cannot be imported into the current release + +On `global`, read the condition and inspect the dashboards named in it: + +```bash +kubectl api-resources | grep -i monitordashboard +kubectl get katanomi -A -o yaml | grep -n -B2 -A4 UpgradeError +kubectl -n cpaas-system get monitordashboard -l cpaas.io/dashboard.folder=devops +kubectl -n cpaas-system get monitordashboard devops-api devops-apiserver devops-controller -o yaml | grep -n -E '^ name:|managed-by|meta.helm.sh' +``` + +A dashboard named in the `invalid ownership metadata` message, without `app.kubernetes.io/managed-by: Helm` and the `meta.helm.sh/release-*` annotations, is this cause. Back up the dashboards: + +```bash +mkdir -p "$HOME/devops-monitordashboard-backup" +for d in devops-api devops-apiserver devops-controller; do + kubectl -n cpaas-system get monitordashboard "$d" -o yaml > "$HOME/devops-monitordashboard-backup/$d.yaml" +done +``` + +> **Warning:** Deleting a `MonitorDashboard` removes that DevOps monitoring dashboard until the DevOps v3 release creates it again. Precondition: only when the DevOps v3 instance shows `UpgradeError` (or fails to install) with `MonitorDashboard "" in namespace "cpaas-system" exists and cannot be imported into the current release: invalid ownership metadata`, only for the dashboard that the message names, and with the backups written. Rollback: `kubectl apply -f` the backup file (remove `metadata.resourceVersion` and `metadata.uid` first) if the operator does not recreate it. + +Delete the dashboard that the message names: + +```bash +kubectl -n cpaas-system delete monitordashboard +``` + +Wait for the operator to retry the release, then check the instance and that the dashboards are recreated with Helm ownership metadata: + +```bash +kubectl get katanomi -A +kubectl -n cpaas-system get monitordashboard -l cpaas.io/dashboard.folder=devops +kubectl -n cpaas-system get monitordashboard devops-api -o yaml | grep -n -E 'managed-by|meta.helm.sh' +``` + +If the retry names a different dashboard, back up and delete that one under the same Warning. Then open Toolchain again. + +### Cause 2: The DevOps v3 operator is installed but no Katanomi instance exists + +```bash +kubectl get csv -A | grep -i katanomi +kubectl get katanomi -A +``` + +No `Katanomi` object is this cause. In Platform Management > App Market Management > OperatorHub on `global`, open `Alauda DevOps v3`, go to its instances, and create a `Katanomi` instance: + +1. Keep the name and namespace the form proposes (`katanomi-sample` in `katanomi-operator`). Use a different namespace, such as `cpaas-system` set in the YAML view, only when Alauda support instructs you to for your environment. +2. Set `External URL` to the `global` cluster access address (the platform address). +3. In `Service`, set `Type` to the ingress option; keep `API Port` and `Plugin Port` unless you have a reason to change them. + +Check that the instance reconciles and the toolchain entry appears: + +```bash +kubectl get katanomi -A +kubectl -n get pods | grep -i katanomi +``` + +### Cause 3: The DevOps feature card stays because the Katanomi instance and the DevOps v3 operator are still installed + +Confirm with the owners that no DevOps v3 pipeline resource is needed; the steps below delete them. List and export what exists: + +```bash +for crd in $(kubectl get crd | grep katanomi | awk '{print $1}'); do echo "=== CRD: $crd ==="; kubectl get "$crd" -A; done +mkdir -p "$HOME/katanomi-backup" +for crd in $(kubectl get crd | grep katanomi | awk '{print $1}'); do kubectl get "$crd" -A -o yaml > "$HOME/katanomi-backup/$crd.yaml"; done +``` + +> **Warning:** Deleting `katanomi-sample` and uninstalling DevOps v3 removes the DevOps v3 user view and stops every DevOps v3 function; the forced deletion below removes all katanomi custom resources (DevOps v3 pipeline and toolchain objects) without waiting for finalizers. Apart from re-applying the exports, which does not restore runtime state, this is irreversible. Precondition: only when DevOps v3 is being retired (for example for DevOps v4) and the owners confirmed that none of its resources are needed. Rollback: install DevOps v3 again from OperatorHub, create the instance (Cause 2), and re-apply the exported objects that are still needed. + +1. In Platform Management > App Market Management > OperatorHub, open the DevOps v3 operator and delete the `katanomi-sample` instance under Instances. +2. Uninstall DevOps v3 (`katanomi-operator`) from OperatorHub. The DevOps feature card disappears. +3. Delete the remaining custom resources of every katanomi CRD: + + ```bash + for crd in $(kubectl get crd | grep katanomi | awk '{print $1}'); do kubectl delete "$crd" --all -A --force --grace-period=0; done + ``` + +Check that no katanomi objects remain (every line should print `0`): + +```bash +for crd in $(kubectl get crd | grep katanomi | awk '{print $1}'); do kubectl get "$crd" -A --no-headers 2>/dev/null | wc -l; done +``` + +## Diagnostic Steps + +```bash +kubectl get csv -A | grep -i katanomi +kubectl get katanomi -A +kubectl get katanomi -A -o yaml | grep -n -A3 'reason:' +kubectl -n cpaas-system get monitordashboard -l cpaas.io/dashboard.folder=devops +``` + +- `UpgradeError`, or a failed install, that names `MonitorDashboard "" ... invalid ownership metadata`: Cause 1. +- A `katanomi-operator` CSV with no `Katanomi` object on `global`: Cause 2. +- A DevOps v3 view that must be removed while `katanomi-sample` and the operator are still installed: Cause 3. + +If OperatorHub on `global` lists `Alauda DevOps Eventing v3` but not `Alauda DevOps v3`, the required operator package has not been uploaded. `Alauda DevOps v3` (Katanomi) is the operator required for DevOps v3 and for selecting Harbor images in ACP through the drop-down; `Alauda DevOps Eventing v3` does not need to be deployed for that. Upload the `Alauda DevOps v3` package first. diff --git a/docs/en/solutions/Local_overrides_reverted_or_blocking_an_ACP_upgrade_ResourcePatch_ALB_ConfigMap_chart_and_feature_gate_changes.md b/docs/en/solutions/Local_overrides_reverted_or_blocking_an_ACP_upgrade_ResourcePatch_ALB_ConfigMap_chart_and_feature_gate_changes.md new file mode 100644 index 000000000..86216a731 --- /dev/null +++ b/docs/en/solutions/Local_overrides_reverted_or_blocking_an_ACP_upgrade_ResourcePatch_ALB_ConfigMap_chart_and_feature_gate_changes.md @@ -0,0 +1,246 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.6" + - "3.8" + - "3.12" + - "3.16" + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Local overrides reverted or blocking an ACP upgrade: ResourcePatch, ALB ConfigMap, chart and feature gate changes + +## Issue + +An ACP upgrade is blocked by a pre-upgrade check, or it completes and a hand-edited setting returns to the catalog value. Typical signatures: + +- The pre-upgrade checklist prints `ERROR: check alb configmap patch`. +- `kubectl get rpch -A` returns a long list of `ResourcePatch` objects. +- New Pods in a Service Mesh namespace log `FailedToRetrieveImagePullSecret` naming `global-registry-auth`. +- Charts are reported missing from the local template repository after a Pod moved to another node, or hand edits disappear after a HelmRequest sync. +- ALB on a business cluster restarts while only the `global` cluster is being upgraded. +- ALB is missing after the upgrade to 4.3. +- A custom inspector role can suddenly use WebCLI to exec into Pods and log in to nodes. +- The documented fix for checklist item `check_alb_configmap_ge_3_12` fails with `unknown field "sepc.confg"`. + +ACP stores some local changes as `ResourcePatch` objects (`rpch`) and re-applies them on every reconcile. A direct edit of a generated ConfigMap or of a Helm-rendered object is not such a patch, so the next sync puts the catalog value back. + +## Root Cause + +### Cause 1: ERROR: check alb configmap patch + +The checklist names an ALB ConfigMap that was edited in place. An in-place ConfigMap edit is not an override the upgrade preserves: the upgrade rewrites the ConfigMap from the chart, and the edit is gone afterwards even though the checklist only warned about it. + +### Cause 2: ResourcePatch (rpch) entries block or silently override the upgrade + +`rpch` objects record personalized edits of platform components. Before a business-cluster upgrade (for example 4.2.2 to 4.2.5) or a `global` patch upgrade (for example 4.1.2 to 4.1.5), every entry has to be classified by what it changes. Image overrides, including old hotfix images, are not safe to carry across an upgrade. CPU and memory overrides usually are. Empty patches that only store an annotation and an update timestamp are noise. A patch whose effect nobody can explain is a stop sign. Which patches are kept is a decision for each environment; there is no global allow-list. + +### Cause 3: Subnet isolation or a temporary otel injection image is still in place + +For the upgrade from 3.18.1 to 4.2.5, subnet resource isolation and cross-subnet access that were changed away from the platform default have to be restored to the default before the upgrade and configured again from the 4.2.5 documentation afterwards. A changed Harbor storage address, the etcd-mirror synchronization of Prometheus resources, and nevermore `compression=none` can stay in place across that upgrade. + +A temporary image used so that the OpenTelemetry injection jar can start despite wrong file permissions must be checked against the target 4.x build. If the target build contains the fix, revert the temporary image before the upgrade. If it does not, the image is an image-type `rpch` and Cause 2 applies as soon as a fixed image exists. + +### Cause 4: Chart template not found, and manual edits reappear as the chart + +`/cpaas/captain-chartmuseum`, the local template repository, is a hostPath. When its Pod is rescheduled onto another node, that node does not have the files and the chart is reported missing. This happens on every move of the Pod (for example during a 3.6 to 3.8 upgrade). A related failure is a HelmRequest (`helmrequests.app.alauda.io`) resync that redeploys the chart and overwrites objects that were edited by hand. + +### Cause 5: FailedToRetrieveImagePullSecret global-registry-auth + +After a business cluster that uses Service Mesh is upgraded, new microservice Pods raise: + +`FailedToRetrieveImagePullSecret: Unable to retrieve some image pull secrets (global-registry-auth); attempting to pull the image may not succeed` + +`IstioOperator` and the injector ConfigMap `istio-sidecar-injector-1-20` list `imagePullSecrets: global-registry-auth`. Since the 3.16 injection change, that name is added to the `istio-proxy` injection. The Secret does not exist in the business namespace, so every injection logs the warning. + +### Cause 6: ALB is not enabled after the upgrade to 4.3 + +On 4.3, ALB is off unless its feature gate `alaudafeaturegates/alb2` in `cpaas-system` is enabled. Existing ALB installations can keep running. A new 4.3 deployment should not enable ALB as the long-term design: ALB is deprecated from 4.3.1, and the replacement is `ingress-nginx-operator` or Envoy Gateway. + +### Cause 7: A custom inspector role can use WebCLI on nodes + +A custom platform-inspector role that was limited before the upgrade can exec everywhere afterwards, including node login. Its manifest contains `acp-podsexec` and `acp-rootexec`. The built-in platform auditor role on 4.2 is query-only on WebCLI and has no create, delete, or node login permission; the custom role picked up the extra entries and was not trimmed. + +### Cause 8: Business-cluster ALB restarts during the global upgrade + +Only `global` is upgraded, but ALB on a business cluster restarts. The ALB Deployment on the business cluster no longer matches the HelmRequest on `global`, so the sync during the global upgrade writes the HelmRequest's desired state and the business ALB rolls. Once the pre-upgrade logs and audit data are gone, the field that differed cannot be reconstructed; only the current mismatch can be checked. + +### Cause 9: check_alb_configmap_ge_3_12 fix fails with unknown field "sepc.confg" + +The checklist reports `check_alb_configmap_ge_3_12` as failed, and applying the documented fix fails with `unknown field "sepc.confg"` (the field the fix writes is the ALB's `spec.config`). `spec.config` belongs to the `v2beta1` version of the ALB2 resource. The update was sent without selecting resource type `alb2.v2beta1.crd.alauda.io`. Repeating it with that resource type succeeds. + +## Resolution + +### Cause 1: ERROR: check alb configmap patch + +Export the ConfigMap before the upgrade and compare it afterwards. If the upgrade restored the chart copy, put the required keys back through the ALB overwrite mechanism, not through another `kubectl edit` of the ConfigMap. + +```bash +kubectl get configmap -n -o yaml > -configmap.yaml +``` + +> **Warning:** Re-applying a full ConfigMap YAML with `kubectl apply` fights the controller and can drop keys the chart added during the upgrade. Use the ALB overwrite path for that setting and keep `-configmap.yaml` as the backup. If the overwrite itself is wrong, remove the overwrite entry so the chart value returns; do not delete the ALB instance to recover one ConfigMap key. + +### Cause 2: ResourcePatch (rpch) entries block or silently override the upgrade + +Export every patch, then classify each entry: + +```bash +kubectl get rpch -A -o yaml > rpch-backup.yaml +``` + +| What the patch changes | Before the upgrade | +| --- | --- | +| Image, including an old hotfix image | Delete it. | +| CPU or memory only | May stay. | +| CoreDNS host resolution, or a dex access address | May stay through the upgrade. Delete it afterwards and move the data into plugin management or the platform access-address configuration. | +| Empty patch (annotation and timestamp only) | Delete. | +| A file under the component configuration | If nothing depends on it strongly, back up the live resource, delete the `rpch`, and restore the intended content after the upgrade. | +| Startup arguments or an environment variable of a component | May stay, but the flag's location can move. If the component then crashes or conflicts, delete that `rpch`. | +| Anything you cannot explain | Stop. Neither delete it nor keep it blindly. | + +Entries of type `Artifact` can be left out of this classification; confirm the type before you skip one. + +> **Warning:** Deleting an image `rpch` makes the component run the catalog image at the next reconcile, which can restart it and drop the hotfix. Take `rpch-backup.yaml` first. If the catalog image is worse than the hotfix and the upgrade has not started, restore that one object with `kubectl apply`. Re-applying an old image patch after the upgrade puts the old image back; do that only to recover a failed upgrade, and remove it again before the next attempt. Deleting a CPU or memory patch resets the component to the chart requests, which can cause OOM or throttling. + +```bash +kubectl delete rpch -n +``` + +### Cause 3: Subnet isolation or a temporary otel injection image is still in place + +Restore subnet resource isolation and cross-subnet access to the platform default before the 4.2.5 upgrade, then configure them again following the 4.2.5 documentation. Do not carry the pre-4.2 isolation syntax across and edit it in place. + +For the OpenTelemetry injection image, confirm whether the target 4.x build fixes the jar permission failure. If it does, point the injection back to the catalog image before the upgrade. If it does not, keep the workaround until a fixed image is uploaded, then delete it as an image `rpch` (Cause 2). + +> **Warning:** Restoring subnet isolation to the default opens or closes cross-subnet paths that the custom policy enforced. Do it in the maintenance window with the previous manifest saved. Roll back by re-applying that manifest only if the upgrade has not started; once the cluster is on 4.2.5, configure it from the new documentation instead. Reverting the otel image before the target build has the fix brings back the Pod start failure. + +### Cause 4: Chart template not found, and manual edits reappear as the chart + +Find the Pod that mounts the hostPath and the node that has the files: + +```bash +kubectl get pods -A -o json | python3 -c 'import json,sys +d=json.load(sys.stdin) +for p in d.get("items",[]): + for v in p["spec"].get("volumes",[]): + path=(v.get("hostPath") or {}).get("path") or "" + if "captain-chartmuseum" in path: + print(p["metadata"]["namespace"], p["metadata"]["name"], p["spec"].get("nodeName"), path)' +``` + +On that node, `ls /cpaas/captain-chartmuseum` must show the charts. For the duration of the upgrade, keep the Pod on that node by setting a nodeSelector `kubernetes.io/hostname=` on its owner. + +> **Warning:** A nodeSelector pins the template repository to one node; if that node is drained, the charts disappear again. Removing the nodeSelector returns the Pod to normal scheduling and loses the charts unless the files were copied to shared storage first. Do not delete the Pod to "refresh" a missing chart. + +Put every intentional difference into Helm values and change it through those values. A hand edit of a rendered Deployment is overwritten the next time the HelmRequest syncs. + +### Cause 5: FailedToRetrieveImagePullSecret global-registry-auth + +Confirm the Secret is absent, then remove `global-registry-auth` from the ServiceMesh configuration (`IstioOperator` at `spec.values.global.imagePullSecrets`) and from the injector ConfigMap that still lists it. Creating the missing Secret is not the fix. + +```bash +kubectl get secret global-registry-auth -n +kubectl get servicemesh -A +kubectl get istiooperator -A +kubectl get configmap -A | grep istio-sidecar-injector +``` + +> **Warning:** Restarting microservice Pods re-injects the sidecar and drops their current connections. Do it after neither the ServiceMesh nor the injector names `global-registry-auth`, in a window those services can restart. Roll back by restoring the previous ServiceMesh and ConfigMap YAML and restarting again. + +```bash +kubectl -n rollout restart deploy +``` + +### Cause 6: ALB is not enabled after the upgrade to 4.3 + +If the cluster already serves traffic with ALB and the gate is off, enable it. On a new 4.3 installation, do not enable ALB; plan `ingress-nginx-operator` or Envoy Gateway instead. From 4.3.1, treat enabling ALB as a temporary bridge. + +```bash +kubectl get alaudafeaturegates -n cpaas-system alb2 -o yaml +``` + +> **Warning:** Enabling the gate reconciles ALB on a cluster where it is deprecated from 4.3.1. Enable it only to keep an ALB that is already in use. To turn it off again, patch `spec.enabled` to `false` with the same annotation. Do not delete the `alb2` feature-gate object. + +```bash +kubectl patch alaudafeaturegates alb2 -n cpaas-system --type=merge -p '{"spec":{"enabled":true},"metadata":{"annotations":{"cpaas.io/ignoreme":"true"}}}' +``` + +Keep the `cpaas.io/ignoreme` annotation in the patch so the setting persists. Read the object again and require `spec.enabled: true`. + +### Cause 7: A custom inspector role can use WebCLI on nodes + +Find the custom role YAML that contains `acp-podsexec` or `acp-rootexec`, remove those entries, and keep `get`, `list`, and `watch`. The 4.2 platform auditor is the reference for query-only access. + +```bash +grep -n -e acp-podsexec -e acp-rootexec .yaml +``` + +> **Warning:** Applying the trimmed role removes node login and Pod exec from everyone bound to it. Save the previous YAML and restore it if the trim removes a permission the role still needs. Do not delete the whole role to remove two entries; that locks out the inspectors. + +```bash +kubectl apply -f .yaml +``` + +If the role is not a plain ClusterRole, apply it through the same API the platform uses for custom roles. + +### Cause 8: Business-cluster ALB restarts during the global upgrade + +Before upgrading `global`, compare the business ALB Deployment with the HelmRequest on `global` that owns it. Put the intended values on the HelmRequest so the sync is a no-op, or schedule the business ALB restart inside the global window and inform the application owners. + +```bash +kubectl get helmrequests.app.alauda.io -A +kubectl -n get deploy -o yaml > business-alb-deploy.yaml +``` + +> **Warning:** Editing the HelmRequest to match a hand-tuned Deployment rolls ALB when the request syncs. Change it at the start of the window, not in the middle of the global upgrade. To undo a bad edit, re-apply the previous HelmRequest YAML. Deleting the HelmRequest removes the release, not just the drift. + +Aligning the HelmRequest only removes this drift-induced restart. The ALB restart that comes with upgrading the cluster that runs the ALB cannot be avoided or deferred; see "ALB load balancer Pods restart during an ACP upgrade and the restart cannot be deferred". + +### Cause 9: check_alb_configmap_ge_3_12 fix fails with unknown field "sepc.confg" + +Repeat the documented update with resource type `alb2.v2beta1.crd.alauda.io` selected. In the console, pick that resource type before editing. From the command line, name the fully qualified resource: + +```bash +kubectl get alb2.v2beta1.crd.alauda.io -A +kubectl -n get alb2.v2beta1.crd.alauda.io -o yaml > -alb2.yaml +``` + +> **Warning:** Changing `spec.config` reconciles the ALB and can restart its Pods, which interrupts traffic through it briefly. Precondition: `-alb2.yaml` saved and a maintenance window. Rollback: `kubectl apply -f -alb2.yaml`. + +```bash +kubectl -n edit alb2.v2beta1.crd.alauda.io +``` + +Make the change the checklist item's documentation describes, save, and run the checklist again. + +## Diagnostic Steps + +List the overrides before judging upgrade readiness. A healthy node list does not mean the upgrade keeps local edits. + +```bash +kubectl get rpch -A +kubectl get helmrequests.app.alauda.io -A +kubectl get alaudafeaturegates -n cpaas-system alb2 -o jsonpath='{.spec.enabled}{"\n"}' +``` + +Use the first matching signature: + +1. Checklist text `check alb` with `configmap patch`: Cause 1. +2. Any `rpch` row: Cause 2. Image rows are deleted; a row is not kept just because the component is currently healthy. +3. Subnet isolation or cross-subnet access is non-default and the target is 4.2.5, or an otel injection image is a known temporary build: Cause 3. +4. A chart template cannot be found, or a hand edit vanished after a sync, and a volume uses `/cpaas/captain-chartmuseum`: Cause 4. +5. Event text naming `global-registry-auth`: Cause 5. Check the business namespace for the Secret before editing the injector. +6. ALB missing on 4.3 and `alaudafeaturegates/alb2` not enabled: Cause 6. +7. A custom inspector role can exec or log in to nodes, and its YAML contains `acp-podsexec` or `acp-rootexec`: Cause 7. +8. Business ALB restarted while only `global` changed, and its Deployment does not match the HelmRequest on `global`: Cause 8. +9. The `check_alb_configmap_ge_3_12` fix fails with `unknown field "sepc.confg"` or `unknown field "spec.config"`: Cause 9. + +A `ResourcePatch` whose owner cannot say what it changes stays a stop. Do not delete it to make `kubectl get rpch` shorter. diff --git a/docs/en/solutions/Node_add_check_times_out_immediately_because_the_cluster_exec_API_returns_HTTP_500_on_ACP.md b/docs/en/solutions/Node_add_check_times_out_immediately_because_the_cluster_exec_API_returns_HTTP_500_on_ACP.md new file mode 100644 index 000000000..84b771705 --- /dev/null +++ b/docs/en/solutions/Node_add_check_times_out_immediately_because_the_cluster_exec_API_returns_HTTP_500_on_ACP.md @@ -0,0 +1,84 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Node add check times out immediately because the cluster exec API returns HTTP 500 on ACP + +## Issue + +Adding a node to a cluster fails at the availability check with no delay. The add-node dialog returns `检查超时,请重试` ("check timed out, please retry") at once, while it still shows `节点可用性检查中...` ("node availability check in progress"). The node is reachable, and the error does not match the network state. + +In the browser developer tools, the request of the add-node dialog fails: + +```text +POST /apis/platform.tkestack.io/v1/clusters//exec?ignore_warning=true +500 Internal Server Error +``` + +The `tke-platform-api` log on the `global` cluster has no x509 error for `https://localhost:9443/...`. + +`tke-platform-api` and `tke-platform-controller` run on the `global` cluster in `cpaas-system`. On some `global` clusters (for example on ACP 4.2) there is no `tke-platform-api` Pod, and `cluster-api-provider-alauda` manages node addition instead; check which one yours runs with `kubectl get deploy -A | grep -E 'tke-platform|cluster-api-provider-alauda'`. + +Related articles: [tke-platform-api serves an expired certificate after running for a year on ACP](tke-platform-api_serves_an_expired_certificate_after_running_for_a_year_on_ACP.md), for the same immediate check timeout when the `tke-platform-api` log shows x509 errors against `localhost:9443`. + +## Root Cause + +### Cause 1: the cluster exec API of tke-platform-api returns 500 with no log cause + +The add-node dialog runs its availability check through the cluster `exec` API (`/apis/platform.tkestack.io/v1/clusters//exec`), which `tke-platform-api` serves. When that API returns HTTP 500, the dialog reports it as an immediate check timeout instead of a real one. No underlying cause has been identified for a 500 that comes without a `localhost:9443` x509 line in the log. Restarting `tke-platform-api` restores the `exec` API, and the node availability check recovers. + +## Resolution + +### Cause 1: the cluster exec API of tke-platform-api returns 500 with no log cause + +1. Confirm the 500 in the browser developer tools: the `exec?ignore_warning=true` request of the add-node dialog. + +2. Read the `tke-platform-api` log on `global`. If it shows x509 errors for `localhost:9443` (`CertificateInvalidError` or `certificate has expired`), follow the related article on the expired certificate instead. + + ```bash + kubectl -n cpaas-system get pods -o wide | grep tke-platform-api + kubectl -n cpaas-system logs deploy/tke-platform-api --tail=300 | grep -i -E 'exec|error|panic' + kubectl -n cpaas-system logs deploy/tke-platform-api --tail=500 | grep -E 'localhost:9443|CertificateInvalidError|certificate has expired' + ``` + +3. Check that the clock is correct on `global` and on the node, and that the node is reachable over SSH from `global`: + + ```bash + date -u + ``` + +4. Restart `tke-platform-api`. + + > **Warning:** Restarting `tke-platform-api` briefly interrupts cluster and node management on `global` for every cluster; running workloads are not affected. Apply only when the `exec` request returns 500 and the clock and the network to the node have been checked. Rollback: none needed; the Deployment recreates the Pods. + + ```bash + kubectl -n cpaas-system rollout restart deployment/tke-platform-api + kubectl -n cpaas-system rollout status deployment/tke-platform-api + ``` + +5. Retry the node addition. If the `exec` request still returns 500, collect the `tke-platform-api` log from the time of the request and contact support. + +## Diagnostic Steps + +```bash +# global cluster +kubectl get deploy -A | grep -E 'tke-platform|cluster-api-provider-alauda' +kubectl -n cpaas-system get pods -o wide | grep tke-platform-api +kubectl -n cpaas-system logs deploy/tke-platform-api --tail=500 | grep -E 'localhost:9443|CertificateInvalidError|certificate has expired|panic' +date -u +``` + +- The `exec` request returns 500 at once and the log has no `localhost:9443` x509 line: this article. +- The `exec` request returns 500 and the log shows `localhost:9443` with an x509 error: see [tke-platform-api serves an expired certificate after running for a year on ACP](tke-platform-api_serves_an_expired_certificate_after_running_for_a_year_on_ACP.md). +- `date -u` on `global` or on the node is wrong: fix the clock; this article does not apply. +- The `exec` request returns 200, or 504 after a long wait: a real check timeout; check SSH and network reachability to the node. +- There is no `tke-platform-api` Deployment and `cluster-api-provider-alauda` runs instead: this article does not apply; node addition is managed by `cluster-api-provider-alauda`. diff --git a/docs/en/solutions/Node_maintenance_on_ACP_OS_reinstall_or_replacement_hostname_IP_NIC_and_cluster_name_changes.md b/docs/en/solutions/Node_maintenance_on_ACP_OS_reinstall_or_replacement_hostname_IP_NIC_and_cluster_name_changes.md new file mode 100644 index 000000000..2de53ce6d --- /dev/null +++ b/docs/en/solutions/Node_maintenance_on_ACP_OS_reinstall_or_replacement_hostname_IP_NIC_and_cluster_name_changes.md @@ -0,0 +1,152 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - "3.8" + - "4.1" + - "4.2" + - "4.3" +--- + +# Node maintenance on ACP: OS reinstall or replacement, hostname, IP, NIC, and cluster name changes + +## Issue + +Planned changes to the identity of nodes that are already in a cluster raise the same questions repeatedly: + +- A machine's OS was reinstalled. How does it rejoin the cluster? +- Compute nodes must move from CentOS to Kylin, keeping their IPs, and TopoLVM is in use. +- Node hostnames are inconsistent. Can the hostnames be changed? Can the node names be changed? +- The node IPs must change. Is a redeploy needed, and how do removed nodes come back? Can the VIP and node IPs of a whole `global` and workload cluster be changed? +- VMs move to another virtualization platform and the NIC name changes (for example from `ens192` to `ens3`). +- Can the cluster be renamed? + +Related articles: "Node maintenance on ACP: resizing nodes and masters, host shutdown, and disk expansion" covers resource changes. + +## Root Cause + +### Cause 1: A reinstalled machine is a new host to the cluster + +After an OS reinstall the host has none of the files the cluster gave it, while the cluster may still hold its node object and network or storage state. Remove the node from the cluster, run the node cleanup script on the host before it rejoins, then add the node again. + +### Cause 2: Replacing the OS of compute nodes that keep their IPs + +Changing compute nodes from CentOS to Kylin Linux Advanced Server V10 (for example on ACP 4.2.1) means taking each node out of the cluster, reinstalling it, and rejoining it with the same IP. The target OS and kernel must be in the verified list of the platform version. Nodes that host log or monitoring storage, middleware, DevOps components, or TopoLVM volumes need their data moved first. + +### Cause 3: The node name is fixed at join; hostnames of IP-named nodes can change + +A joined node does not support changing its Node name; the name is fixed when the node joins. + +- Nodes named by hostname: changing the hostname afterwards can make node identity, certificates, kubelet registration, and the platform display disagree. Keep the hostnames unchanged. +- Nodes named by IP: the hostnames can be changed. + +### Cause 4: Node IPs cannot be changed in place + +- A worker's IP is changed by deleting the node in the UI and adding it again with the new IP. Changing the IPs of all machines at once is not supported (for example on TKE 3.8.1). +- Nodes already removed because their IPs changed cannot be re-added through a backend method. Run `cleanup.sh` and `init.sh` on the node and add it again under Platform Management > Cluster. The cleanup script deletes the cluster data on that node. +- A workload cluster whose IPs must all change and that runs no business workloads yet is deleted and recreated with the new IPs, rather than changed node by node. +- There is no procedure for changing both the cluster VIP and the node IPs of a running `global` cluster and its workload clusters with workloads and Harbor on them. That is an architecture and network re-planning project that needs a separate evaluation with Alauda. + +### Cause 5: VMs moved to another virtualization platform get a new NIC name + +Moving nodes to another virtualization platform requires shutting the VMs down. When IPs, disks, and paths stay the same, the only platform-side impact is the NIC name. On Calico, `IP_AUTODETECTION_METHOD` of the `calico-node` DaemonSet matches interface names (for example `interface=^eth0$`) and must also match the new name. On Kube-OVN, the tunnel interface setting of the node is the equivalent; when a migrated master's NIC name changes, `ovn-cni` can fail to start until the node annotation is updated to the new interface name. + +### Cause 6: The cluster name cannot be modified + +Renaming a cluster after it is deployed is not supported. The only option is to redeploy the cluster with the new name. Only the display name can be edited. + +## Resolution + +### Cause 1: A reinstalled machine is a new host to the cluster + +Remove the node from the cluster through the platform, run the cleanup script for your platform version on the host, prepare the host as for a new node, and add it again. The same sequence applies to nodes that must be repartitioned (for example to give `/var/lib/containerd` and `/cpaas` their own partitions) while they run no workloads: delete the node, run the cleanup script, partition, run the initialization script, add it again. + +```bash +kubectl get node +kubectl get machine -A | grep +``` + +### Cause 2: Replacing the OS of compute nodes that keep their IPs + +1. Confirm the target OS and kernel against the verified list of your version (`cat /etc/os-release` and `uname -r` on a host already running it). +2. List what runs on the nodes: log and monitoring storage, middleware, DevOps components, TopoLVM volumes. +3. If spare capacity allows, add the new nodes first. Then cordon and drain the nodes being replaced and wait until their Pods run on other nodes. +4. For TopoLVM, back up or migrate TopoLVM-backed services, then take the node out of TopoLVM following the product documentation for removing a TopoLVM storage node (see also "Preserving TopoLVM Data of KubeVirt VMs When Reinstalling Node Operating Systems on ACP"). +5. Remove the node from the cluster, reinstall the OS, prepare the host, and add it again with the same IP. + +> **Warning:** `kubectl drain` evicts every Pod on the node that is not a DaemonSet Pod; single-replica workloads are unavailable until they start elsewhere. Data on local TopoLVM volumes of that node is lost with the reinstall unless it was backed up or migrated first. Confirm the data is backed up or not needed before step 5. Rollback of a drain is `kubectl uncordon `; a reinstall cannot be rolled back. + +```bash +kubectl cordon +kubectl drain --ignore-daemonsets --delete-emptydir-data +kubectl get pods -A -o wide --field-selector spec.nodeName= +kubectl -n nativestor-system get topolvmcluster topolvm -o yaml +``` + +### Cause 3: The node name is fixed at join; hostnames of IP-named nodes can change + +Check how the nodes are named (Platform Management > Cluster Management > Nodes, or compare `NAME` with `INTERNAL-IP` below) and compare with the current OS hostname: + +```bash +kubectl get nodes -o wide +hostname; hostnamectl +``` + +- Named by hostname: keep the hostname each node had when it joined; there is no rename procedure. +- Named by IP: change the hostnames as required, then confirm every node is still `Ready`. + +### Cause 4: Node IPs cannot be changed in place + +For one worker or compute node, delete it in the UI, then on the host: + +> **Warning:** `cleanup.sh` removes the cluster data on that node (container data, kubelet and Kubernetes files). Confirm that nothing that must be kept lives in those paths before running it; it cannot be undone. + +```bash +sudo bash cleanup.sh +sudo bash init.sh +ip -br addr +``` + +Then add the node with its new IP under Platform Management > Cluster. If the add reports that a NIC cannot be found (for example `检查网卡:找不到网卡: `, "check NIC: NIC not found"), enter the interface that now carries the node IP. An expired license must be renewed before nodes can be added. For a whole cluster with no workloads, delete it and create it again with the new addresses. + +### Cause 5: VMs moved to another virtualization platform get a new NIC name + +Move one node at a time: cordon and drain it, shut it down, move it, start it, uncordon it, and only then take the next. Before the first node with the new name starts, let Calico match both names. + +> **Warning:** A wrong `IP_AUTODETECTION_METHOD` makes `calico-node` pick no address or the wrong one when the DaemonSet rolls, which breaks Pod traffic on every node. Save the current value; rollback is setting it back. + +```bash +kubectl get ds -A | grep calico-node +kubectl -n get ds calico-node -o yaml | grep -A2 IP_AUTODETECTION_METHOD +kubectl -n set env ds/calico-node IP_AUTODETECTION_METHOD='interface=^(ens192|ens3)$' +``` + +The regular expression must keep every interface name that nodes use today and add the new one. Use the namespace that `kubectl get ds -A` prints (commonly `kube-system`). + +As an alternative to moving the `global` workers, they can be replaced: add new workers; cordon the old ones; uninstall the log collection components (`nevermore`) on `global` and the business clusters and the `global` Elasticsearch log storage plugin; evict the Pods to the new workers and check `kubectl get pod -A | grep -v Running`; reinstall the Elasticsearch log storage plugin pinned to the new nodes; reinstall log collection with `global` as the storage cluster. + +### Cause 6: The cluster name cannot be modified + +Plan the name before creation. To use another name, create the cluster again with that name. For an imported cluster, re-importing it is the alternative to assess. + +```bash +kubectl get cls +``` + +## Diagnostic Steps + +```bash +kubectl get nodes -o wide +kubectl get machine -A +cat /etc/os-release; uname -r +ip -br addr +``` + +- A host whose OS was reinstalled while its node object still exists: Cause 1. +- An OS change on nodes that keep their IPs, especially with TopoLVM volumes: Cause 2. +- `NAME` equal to the hostname versus equal to `INTERNAL-IP` decides whether hostnames may change: Cause 3. +- Any node IP change: Cause 4. +- A NIC name that changed after a VM migration, with `calico-node` or `ovn-cni` failing on that node: Cause 5. +- A request to rename a cluster: Cause 6. diff --git a/docs/en/solutions/Node_maintenance_on_ACP_resizing_nodes_and_masters_host_shutdown_and_disk_expansion.md b/docs/en/solutions/Node_maintenance_on_ACP_resizing_nodes_and_masters_host_shutdown_and_disk_expansion.md new file mode 100644 index 000000000..d5dfe58d0 --- /dev/null +++ b/docs/en/solutions/Node_maintenance_on_ACP_resizing_nodes_and_masters_host_shutdown_and_disk_expansion.md @@ -0,0 +1,145 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - "3.8" + - "4.1" + - "4.2" + - "4.3" +--- + +# Node maintenance on ACP: resizing nodes and masters, host shutdown, and disk expansion + +## Issue + +Planned resource work on nodes that are already in a cluster raises the same questions repeatedly: + +- CPU or memory was hot-added to a node VM, but the node still shows the old capacity. +- The three masters of a business cluster are oversized and the CPU license is exhausted. Can they be reduced to free cores for new workers? +- Control-plane, monitoring, and log nodes must become smaller. How do I tell whether a node can be shrunk, and can the Pod requests on it be lowered? +- The hypervisor hosts need a fix. Can the platform VMs be shut down one host at a time? +- After a node disk expansion, restarted component Pods stay in the image-pulling state. + +Related articles: "Node maintenance on ACP: OS reinstall or replacement, hostname, IP, NIC, and cluster name changes" covers changes to node identity. + +## Root Cause + +### Cause 1: Node capacity does not change after a CPU or memory hot-add + +kubelet reports the CPU and memory capacity it read when it started. After a hot expansion the node keeps showing the old capacity until kubelet is restarted on that node. + +### Cause 2: Shrinking a master or other node is a per-node resize gated by Pod requests + +Reducing oversized masters (for example three 64-core, 256 GB business-cluster masters at low utilization with no business workloads or logs, reduced to 16 cores and 32 GB to free licensed cores) is feasible as a per-node resize, one master at a time. + +A node can be reduced only when both its utilization and its request rate are low, and at least 3 control-plane nodes must remain. Utilization alone is misleading: a control-plane node with 8 cores and 31.34 Gi can show 11.79% CPU utilization but an 84.00% CPU request rate (limit rate 329.10%), with requests coming largely from Pods such as `region-alb` (2050m), `vmagent-agent` (600m), `ovn-central` (300m), `kube-apiserver` (250m), and `ovs-ovn` and `kube-ovn-monitor` (200m each), well above their actual use. Pod request values like these can be lowered appropriately before the node is resized. + +### Cause 3: Host shutdowns must not take several platform VMs down at once + +A hypervisor host often carries several platform VMs. Shutting the host down stops all of them together, which can take more than one master down and lose the etcd quorum. Back up etcd first, then handle one VM at a time. + +### Cause 4: Pods stay pulling images after a disk expansion because the registry's MinIO cluster broke + +After a node disk expansion, restarted component Pods stay in the image-pulling state. The platform image registry's MinIO is abnormal: its three instances cannot form a cluster, so the registry is unavailable. A new raw disk was added directly while the node's containers were still in place and kubelet and containerd had not been stopped correctly; the existing containers then read their original data abnormally, which broke the MinIO cluster. Restarting MinIO and the registry restores it. + +## Resolution + +### Cause 1: Node capacity does not change after a CPU or memory hot-add + +> **Warning:** Restarting kubelet briefly stops status updates for that node; running Pods keep running. Do it in a maintenance window, one node at a time. There is no rollback; if kubelet does not come back, read `journalctl -u kubelet`. + +```bash +kubectl get node -o jsonpath='{.status.capacity}{"\n"}' +sudo systemctl restart kubelet +systemctl status kubelet --no-pager +kubectl get node -o jsonpath='{.status.capacity}{"\n"}' +``` + +The general cordon, drain, resize, uncordon loop is in "Safely resizing CPU and memory on Alauda Container Platform nodes". + +### Cause 2: Shrinking a master or other node is a per-node resize gated by Pod requests + +Check usage and placement first. Node names on ACP are often the node IP; use the exact name from `kubectl get nodes`. + +```bash +kubectl get nodes -o wide +kubectl top node +kubectl get pods -A -o wide --field-selector spec.nodeName= +``` + +The request rate decides as much as the usage. The node's monitoring tab in the console shows CPU and memory request rates; the same figures are in `kubectl describe node`: + +```bash +kubectl describe node | sed -n '/Allocated resources/,/Events/p' +kubectl get pods -A --field-selector spec.nodeName= \ + -o custom-columns='NAMESPACE:.metadata.namespace,POD:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory' +``` + +A node whose usage is low but whose request rate is high cannot simply be made smaller: the Pods placed there would no longer fit in its allocatable resources, and Pods that do not fit are rejected (for example `OutOfcpu`) or stay `Pending`. Lower the requests of the Pods that over-request first, then resize. Keep at least 3 control-plane nodes throughout. + +> **Warning:** A master that is powered off for the resize removes one etcd member and one API server from service. Never take a second master down until the first is back. Back up etcd before starting, and make sure the new size still covers the control-plane Pods listed above. Rollback is resizing the VM back to its old size. + +For each master in turn: shut it down, resize it, start it, restart and check kubelet, and wait until the node and its etcd Pod are back before moving on: + +```bash +sudo systemctl restart kubelet +systemctl status kubelet --no-pager +kubectl get node +kubectl -n kube-system get pods -l component=etcd -o wide +``` + +### Cause 3: Host shutdowns must not take several platform VMs down at once + +Back up etcd first. Then choose one of two plans: + +- **Business can stop:** stop the business containers, restart the compute nodes one by one and confirm each is normal before the next, then restart the control-plane nodes one by one, then start the business again. +- **Business keeps running:** for one VM at a time, mark it unschedulable and evict the business containers, confirm the business recovered elsewhere, shut down the VM for the fix, start it, make it schedulable again, then move to the next VM. + +> **Warning:** Evicting Pods interrupts single-replica workloads until they start on another node. Do not shut down several platform VMs at the same time; two masters down at once loses the etcd quorum of a three-master cluster. Data services such as MySQL or Redis in the cluster need their own plan based on how they are deployed. Rollback for a VM is `kubectl uncordon` after it is back. + +```bash +kubectl cordon +kubectl drain --ignore-daemonsets --delete-emptydir-data +# shut down, fix, and start the VM, then: +kubectl get node +kubectl uncordon +``` + +### Cause 4: Pods stay pulling images after a disk expansion because the registry's MinIO cluster broke + +Find the MinIO and registry Pods and read their logs; MinIO instances that cannot form a cluster are this cause: + +```bash +kubectl get pods -A -o wide | grep -i -E 'minio|registry' +kubectl logs -n --tail=200 +kubectl describe pod -n | sed -n '/Events/,$p' +``` + +> **Warning:** Restarting MinIO and the registry makes image pulls and pushes against the platform registry fail until the new Pods are Ready. Precondition: the stuck Pods pull from the platform registry and the MinIO logs or status show the instances are not forming a cluster. The controllers recreate the Pods, so there is nothing to roll back; the restart does not repair data that is actually damaged on disk. + +```bash +kubectl delete pod -n +kubectl delete pod -n +kubectl get pods -A -o wide | grep -i -E 'minio|registry' +``` + +The registry returns to normal and the component Pods finish pulling. For the next disk change on a node, stop kubelet, delete the containers on the node, and stop containerd before the disk is added. + +## Diagnostic Steps + +Before and after each maintenance step: + +```bash +kubectl get nodes -o wide +kubectl get node -o jsonpath='{.status.capacity}{"\n"}{.status.allocatable}{"\n"}' +kubectl get pods -A -o wide --field-selector spec.nodeName= | grep -v Running +kubectl -n kube-system get pods -l component=etcd -o wide +``` + +- Capacity still shows the old CPU or memory after a resize: kubelet has not been restarted on that node (Cause 1). +- A node with low usage but a high request rate: lower requests before shrinking it (Cause 2). +- More than one master on the same hypervisor host that is about to be shut down: Cause 3. +- Pods stuck pulling images after a disk expansion, with the platform registry's MinIO unable to form a cluster: Cause 4. +- A node that is `NotReady` after it is back is not a maintenance-plan question; see "Diagnosing a node that reports NotReady on Alauda Container Platform". diff --git a/docs/en/solutions/Platform_console_does_not_open_because_the_cluster_VIP_domain_name_cannot_be_resolved_inside_the_cluster_on_ACP.md b/docs/en/solutions/Platform_console_does_not_open_because_the_cluster_VIP_domain_name_cannot_be_resolved_inside_the_cluster_on_ACP.md index 1bd577643..4858612f4 100644 --- a/docs/en/solutions/Platform_console_does_not_open_because_the_cluster_VIP_domain_name_cannot_be_resolved_inside_the_cluster_on_ACP.md +++ b/docs/en/solutions/Platform_console_does_not_open_because_the_cluster_VIP_domain_name_cannot_be_resolved_inside_the_cluster_on_ACP.md @@ -38,6 +38,19 @@ Make the VIP domain name resolvable inside the cluster, in one of two ways. nslookup ``` + If NetworkManager or another tool generates `/etc/resolv.conf`, make the change there (for example `nmcli con mod ipv4.dns ""`), so it survives a reboot. + + CoreDNS Pods normally run with `dnsPolicy: Default` and copy the node's resolver configuration only when they are created, so they keep forwarding to the old servers until they are recreated. Check the policy, and restart CoreDNS after every node has been changed: + + > **Warning:** Restarting CoreDNS briefly interrupts DNS for Pods in the cluster while the new Pods start. Do it only after `nslookup ` works on every node. No rollback is needed for the restart itself. + + ```bash + kubectl -n kube-system get deploy | grep -i coredns + kubectl -n kube-system get deploy -o jsonpath='{.spec.template.spec.dnsPolicy}{"\n"}' + kubectl -n kube-system rollout restart deployment/ + kubectl -n kube-system rollout status deployment/ + ``` + 2. **Add a static record to CoreDNS.** If the node DNS servers cannot be changed, add the record to the cluster DNS plugin: 1. Open **Administrator** > **Marketplace** > **Cluster Plugins**, find **Alauda Container Platform Cluster DNS** and click **Update**. @@ -89,3 +102,9 @@ Make the VIP domain name resolvable inside the cluster, in one of two ways. ```bash curl -k -m 5 https://:6443/healthz ``` + +5. Rule out a look-alike: if the VIP domain resolves inside Pods and the `tke-platform-api` log on `global` contains `x509` errors for `localhost:9443`, the console failure comes from an expired certificate held by a long-running `tke-platform-api`, not from DNS: + + ```bash + kubectl -n cpaas-system logs deploy/tke-platform-api --tail=500 | grep -E 'localhost:9443|certificate has expired' + ``` diff --git a/docs/en/solutions/Plugin_install_or_uninstall_stuck_or_Marketplace_and_OperatorHub_empty_sentry_OOMKilled_or_cluster-transformer_errors.md b/docs/en/solutions/Plugin_install_or_uninstall_stuck_or_Marketplace_and_OperatorHub_empty_sentry_OOMKilled_or_cluster-transformer_errors.md new file mode 100644 index 000000000..b1f09f97c --- /dev/null +++ b/docs/en/solutions/Plugin_install_or_uninstall_stuck_or_Marketplace_and_OperatorHub_empty_sentry_OOMKilled_or_cluster-transformer_errors.md @@ -0,0 +1,164 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Plugin install or uninstall stuck, or Marketplace and OperatorHub empty: sentry OOMKilled or cluster-transformer errors + +## Issue + +- Uninstalling the monitoring or logging plugin of a cluster hangs: the plugin stays in deleting and its Pods are not removed. Removing the `finalizers` from the ModuleInfo makes the entry vanish from the console, but the resources stay on the cluster. +- All functional components of a business cluster are abnormal with `ChartDownloadFailed [1/1] failed to download chart acp/chart-acp-business-v4.0.32, err: EOF`, although the business cluster reaches `global` on ports 443 and 11443. +- `kubectl get ars -n cpaas-system` shows `prometheus` with `ChartDownloadFailed` / `failed to download chart ... err: EOF`, or `logcenter` stuck in `Syncing` with `chart ... Installing`. +- The NeuVector plugin uninstall from the console stays in uninstalling and its Pods are not deleted. +- The `global` cluster UI shows the Alauda Container Platform Marketplace component as abnormal, while `kubectl get ars` looks normal. +- On `global`, the `alauda-operators` AppRelease reports `failed to resync chart for release "install-operators": execution error at (install-operators/templates/notes.yaml:14:8)`, and OperatorHub shows no operators. + +`sentry` (Deployment `sentry` in `cpaas-system` on each cluster) installs, upgrades, and removes plugin charts and records the result on the `ars` (AppRelease) objects. `cluster-transformer` on `global` turns ModulePlugins into ModuleConfigs and reconciles ModuleInfo (`minfo`) and ClusterModule. + +Related articles: "sentry at full CPU or a redeployed component fails because of Failed or leftover ResourcePatches on ACP". + +## Root Cause + +### Cause 1: sentry is OOMKilled or short of resources, so chart sync, install, and uninstall stop + +`kubectl describe pod` on the `sentry` Pod shows `Last State: Terminated`, `Reason: OOMKilled`, `Exit Code: 137`, with a growing restart count, and the AppReleases of the cluster stay in `Syncing` / `Installing` or report `ChartDownloadFailed ... err: EOF`. The `sentry` Pods can still show `1/1 Running`. + +- System plugin uninstall fails (for example on an ACP 3.12 workload cluster): `logcenter` is `Syncing` with `[1/1] chart ait/chart-alauda-log-center-v3.12.38 Installing`, `prometheus` is `ChartDownloadFailed` with `[2/2] failed to download chart ait/chart-kube-prometheus-v3.12.9, err: EOF`, and the `sentry` container restarts with `OOMKilled` under a `4Gi` memory limit. Raising the memory limit and restarting `sentry` lets the AppReleases sync and the uninstall complete. +- A monitoring uninstall is stuck with the Prometheus ModuleInfo in deleting. Removing its `finalizers` removes the component from the console only; the backend resources stay. With `sentry` given more memory, deploying the component again from the console and uninstalling it again completes the uninstall. Removing ModuleInfo finalizers does not perform an uninstall. +- Every functional component of a business cluster is abnormal (`ChartDownloadFailed ... err: EOF`) while the network to `global` is fine: `sentry` does not have enough resources to reconcile every component at the same time. Increasing its resources returns the AppReleases to normal. + +### Cause 2: A plugin uninstall stays in uninstalling with no identified cause (NeuVector) + +The NeuVector plugin uninstall from the console stays in uninstalling, and the Pods in `neuvector` are not deleted. No underlying cause has been identified. The uninstall is finished by deleting the plugin's ModuleInfo, then the resources in the `neuvector` namespace and the namespace itself. Because deleting the ModuleInfo alone only clears the console entry (Cause 1), the namespace contents are removed by hand. + +### Cause 3: Marketplace component abnormal, get registry client is nil + +The Marketplace component shows abnormal on `global`, AppReleases look normal, and the `cluster-transformer` log repeats: + +```text +ModuleInfo Reconcile ... error: ... error calling imageExist: get registry client is nil +get moduleconfig structure error: template: :34:16: executing "" at : error calling imageExist: get registry client is nil +ClusterModule Process '' error: handle plugin marketplace failed: ClusterModule need retry +``` + +with the same `need retry` for `auth-manager`, `acp-business`, and `sentry`, and `ProcessPlugin ... get render data metallb error: ... get registry client is nil`. `cluster-transformer` cannot reach the image registry, so the `imageExist` check in the plugin templates fails and ModuleInfo / ClusterModule processing stops. Restoring network connectivity to the registry returns the component to normal. + +### Cause 4: failed to resync chart for release "install-operators" and OperatorHub on global is empty + +```text +failed to resync chart for release "install-operators": execution error at (install-operators/templates/notes.yaml:14:8) +``` + +appears on the `alauda-operators` AppRelease of `global`, and OperatorHub in the App Marketplace lists no operators. No underlying cause has been identified. Restarting `cluster-transformer` and `sentry` restores service. + +## Resolution + +### Cause 1: sentry is OOMKilled or short of resources, so chart sync, install, and uninstall stop + +On the affected cluster: + +```bash +kubectl get ars -n cpaas-system +kubectl -n cpaas-system get pods | grep sentry +kubectl -n cpaas-system describe pod | grep -E -A6 'Last State|Limits|Requests' +kubectl -n cpaas-system get deploy sentry -o yaml > "$HOME/sentry.before.yaml" +``` + +> **Warning:** Changing the `sentry` resources restarts it; chart installs, upgrades, and uninstalls on the cluster pause until the new Pod is Ready, and a higher request reserves more node memory. Precondition: only when the `sentry` container's last state is `OOMKilled` (or it cannot keep up with every AppRelease) and AppReleases are stuck in `Syncing`, `Installing`, or `ChartDownloadFailed ... EOF`. Rollback: `kubectl apply -f "$HOME/sentry.before.yaml"`. + +```bash +kubectl -n cpaas-system set resources deploy sentry --limits=memory= +kubectl -n cpaas-system rollout status deploy sentry +``` + +Choose a limit above the one the container was killed at. After a few minutes, check that `kubectl -n cpaas-system get deploy sentry -o jsonpath='{.spec.template.spec.containers[*].resources}'` still shows the new limit; if it reverted, ask Alauda support where the `sentry` resources are configured on your version. Then watch the AppReleases: + +```bash +kubectl get ars -n cpaas-system -w +``` + +For a stuck uninstall whose ModuleInfo finalizers were already removed while the resources are still on the cluster: deploy the plugin again from the console, wait until it is installed, and uninstall it again. Do not remove ModuleInfo finalizers as a way to uninstall while the plugin's resources are still on the cluster. + +One finalizer removal is valid: the Log Agent plugin stays in Uninstalling, its workloads (`nevermore` and `kubernetes-event-exporter`) are already deleted from that cluster, and only the logagent `minfo` remains, in Deleting, with `finalizers: - moduleinfo`. There, removing the finalizer on `global` finishes the uninstall. + +> **Warning:** Removing `finalizers` lets the ModuleInfo be deleted without the controller's cleanup, and anything that cleanup would have removed stays on the cluster. Precondition: only with the exact Log Agent signature above. Save the `minfo` YAML first. Rollback: none for the deleted object; deploy the plugin again from the console if it was removed by mistake. + +```bash +kubectl get minfo -o yaml > "$HOME/.yaml" +kubectl patch minfo --type=merge -p '{"metadata":{"finalizers":null}}' +``` + +### Cause 2: A plugin uninstall stays in uninstalling with no identified cause (NeuVector) + +Rule out Cause 1 first: `sentry` must be Running without `OOMKilled` restarts; otherwise fix it and retry the uninstall from the console. Then look at what is left: + +```bash +kubectl get minfo | grep -i neuvector +kubectl -n neuvector get all +kubectl get all,cm,secret,pvc -n neuvector -o yaml > "$HOME/neuvector-namespace.yaml" +``` + +> **Warning:** This removes NeuVector from the cluster without the plugin's own cleanup; container security policies and NeuVector data in that namespace are deleted. Irreversible apart from reinstalling the plugin. Precondition: only when the console uninstall of NeuVector is stuck in uninstalling, `sentry` is healthy, and the export above is written. Rollback: install the NeuVector plugin again from Cluster Plugins. + +```bash +kubectl delete minfo +kubectl -n neuvector delete all --all +kubectl delete namespace neuvector +kubectl get namespace neuvector +``` + +If the namespace stays in `Terminating`, list what is left with `kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n neuvector` and delete those objects. + +### Cause 3: Marketplace component abnormal, get registry client is nil + +```bash +kubectl -n cpaas-system logs deploy/cluster-transformer --tail=500 | grep -E 'registry client is nil|need retry' +``` + +Take the registry address from Clusters > `global` > Overview > Basic Info > Image Repository Address, and test it from the `global` control-plane nodes: + +```bash +curl -k -s -o /dev/null -w '%{http_code}\n' https:///v2/ +``` + +`000` or a timeout means the path is broken. Restore network connectivity from `global` to the registry (routing, firewall, DNS, or the registry service itself). The `need retry` errors stop and the Marketplace component returns to normal once the registry is reachable. + +### Cause 4: failed to resync chart for release "install-operators" and OperatorHub on global is empty + +```bash +kubectl get ars -A | grep -i alauda-operators +kubectl -n describe ars alauda-operators | grep -i -A3 'resync' +``` + +> **Warning:** Restarting `cluster-transformer` and `sentry` pauses plugin processing and chart sync on `global` until the new Pods are Ready; installs and upgrades in flight wait. Precondition: only when `alauda-operators` shows the `install-operators` `notes.yaml` resync error and OperatorHub on `global` is empty. Rollback: none needed; the Deployments recreate the Pods. + +```bash +kubectl -n cpaas-system rollout restart deploy/cluster-transformer deploy/sentry +kubectl -n cpaas-system rollout status deploy/cluster-transformer +kubectl -n cpaas-system rollout status deploy/sentry +kubectl get ars -A | grep -i alauda-operators +``` + +## Diagnostic Steps + +```bash +kubectl get ars -A | grep -v -E 'Synced|Ready' +kubectl -n cpaas-system get pods | grep -E 'sentry|cluster-transformer' +kubectl -n cpaas-system describe pod | grep -A5 'Last State' +kubectl -n cpaas-system logs deploy/cluster-transformer --tail=300 | grep -E 'registry client is nil|need retry' +``` + +- `sentry` `OOMKilled` (exit code `137`) with AppReleases stuck or `ChartDownloadFailed ... EOF`, or a stuck uninstall: Cause 1. +- A NeuVector uninstall stuck while `sentry` is healthy: Cause 2. +- `get registry client is nil` from `cluster-transformer`: Cause 3. +- The `install-operators` `notes.yaml` resync error with an empty OperatorHub: Cause 4. +- One `sentry` Pod near its CPU limit, or `Failed` ResourcePatches: see the related ResourcePatch article. diff --git a/docs/en/solutions/Preparing_the_extension_package_list_for_an_ACP_upgrade_fails_or_is_incomplete_violet_list_and_apps_yaml.md b/docs/en/solutions/Preparing_the_extension_package_list_for_an_ACP_upgrade_fails_or_is_incomplete_violet_list_and_apps_yaml.md new file mode 100644 index 000000000..1332aba2e --- /dev/null +++ b/docs/en/solutions/Preparing_the_extension_package_list_for_an_ACP_upgrade_fails_or_is_incomplete_violet_list_and_apps_yaml.md @@ -0,0 +1,184 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "4.0" + - "4.2" + - "4.3" +--- + +# Preparing the extension package list for an ACP upgrade fails or is incomplete: violet list and apps.yaml + +## Issue + +While collecting the extension packages for a platform upgrade: + +- The extension list exported with `violet` is pasted into the Alauda Customer Portal download page and the upload returns an error. After the YAML is fixed, Marketplace > Batch Download > Upgrade > Post-ACP v4.0 Upgrades (for example target `v4.3.0`, architecture `amd64`) shows `No Application` in the Packages list. +- After `apps.yaml` is imported, many packages are listed with several versions each. +- `apps.yaml` entries do not match on the Alauda Customer Portal (`logapi`, `logagent` reported missing), a scenario category misses `log-center`, and some entries show "no package for this version" (`暂无此版本包`). +- Before an upgrade (for example from 4.2.4 to 4.3) some extension packages are not yet released, and self-uploaded plugins have no matching version. +- `violet list` fails on 3.16.2, or after the upgrade from 3.16.x to 4.0.12 the package lookup still goes through `app.yaml`. +- While upgrading DevOps to v3.20 following the DevOps v3 upgrade documentation, the `Alauda DevOps Nexus v3` and `Alauda DevOps Harbor v3` packages cannot be found. +- On a new 4.0.7 `global` deployed with the 4.0.7 core package, the 4.0.7 extension package was pushed with `violet push`, and the VM monitoring, log collection, and ES storage plugins still show as not listed (`未上架`). +- Packages that are published (`已上架`) but not installed should be taken down (`下架`) so that production matches a test environment. +- `violet list --platform-address ... --platform-token ... --output-file ./apps.yaml` returns `null` for every plugin before an upgrade to 4.3.3, although Cluster Plugins on `global` shows installed plugins. + +## Root Cause + +### Cause 1: The extension YAML lacks the AppScenarioManifest header, or lists plugins that are not installed + +The exported YAML starts directly with the `applications:` key and lacks a top-level `AppScenarioManifest:` line above it, which causes the upload error. Separately, plain `violet list` lists every package uploaded to the platform, not only the installed ones, which produces the `No Application` ("no updates needed") result. `violet list --installed` lists only installed plugins. Adding the `AppScenarioManifest:` line and regenerating the list with `violet list --installed` fixes both. If some target-version plugins are still being released, wait for their packages before upgrading. + +### Cause 2: The upgrade prompt lists every compatible version of each package + +By default the upgrade prompt lists every extension package version that is compatible with the target version. Downloading all listed versions is the expected default. + +### Cause 3: An artifact ID changed or a scenario category missed a package + +When new artifact versions were published, some artifact IDs changed: the names gained a `-` (`log-api`, `log-agent`), so the Alauda Customer Portal treats the new version as a different application and the old `apps.yaml` entry no longer matches. Both exist under scenario category `common ACP Extension`. The 4.0 names `logagent` and `logcenter` became `log-agent` and `log-center` in 4.3. `log-center` was missing from that category and has since been added. + +For an entry with "no package for this version", the portal cannot tell whether the component was removed in the target version or merged into the core package; confirm each entry by hand. For `logagent` and `logcenter` it is the rename above. For `harbor-operator` and `nexus-operator` on 4.3 it means DevOps v3 is not supported there; migrate to DevOps v4. + +### Cause 4: Some packages for the target version are not released yet, or have no platform version + +The upgrade is held because some required extension packages are not yet released, and self-uploaded plugins show no match. Comparing every package in the environment's `apps.yaml` with the available target-version packages gives two lists: packages to download for the upgrade, and self-uploaded plugins with no matching platform version. An installed platform-provided plugin with no released target package holds the upgrade until the package is released. Self-uploaded plugins with no platform version do not affect the upgrade. For 4.3: OAM has no 4.3 packages, so nothing is downloaded for it; Ingress-nginx is no longer available as a plugin, and `ingress-nginx-operator`, included in the core package, replaces it. + +The same gap applies to the DevOps v3.20 upgrade: the `Alauda DevOps Nexus v3` and `Alauda DevOps Harbor v3` packages were published only after a request to Alauda. + +### Cause 5: After 3.16 to 4.0.12 the package list must not be taken from app.yaml + +For the upgrade from ACP 3.16.x to 4.0.12, the package list is not retrieved through `app.yaml`. The packages for that upgrade come from the dedicated `ACP Upgrade from v3.x to v4.x` batch-download scenario. `violet list` reads the application list already present in the environment and is the inventory to compare against that download set; plain `violet list` also includes packages that are uploaded but not installed. Later upgrades that start on 4.0 or later do import `apps.yaml` on the Post-ACP v4.0 Upgrades page (Cause 1). No cause is known for the `violet list` failure on 3.16.2; `violet list` works on a 3.16 `global` control-plane node with the violet client that matches the target version. + +### Cause 6: The published 4.0.7 extension package was missing plugins + +The 4.0.7 extension package published at one time did not contain several plugins. The package was rebuilt and published again; download the corrected package. + +### Cause 7: Published but not installed packages do not have to be taken down + +Packages that are published but not installed do not affect the upgrade process, and there is no takedown procedure for them. Leave them in place. + +### Cause 8: violet list returns null for every plugin before the upgrade to 4.3.3 + +With the `violet` version used before the upgrade to 4.3.3, `violet list` outputs only the application catalog and does not show installation status, so the installed state comes back as `null`. This is expected and does not mean the plugins are uninstalled. + +## Resolution + +### Cause 1: The extension YAML lacks the AppScenarioManifest header, or lists plugins that are not installed + +Regenerate the list from installed plugins only: + +```bash +violet list --installed --platform-address --platform-username --platform-password '' --output-file apps.yaml +head -n 5 apps.yaml +``` + +If the file does not begin with `AppScenarioManifest:`, add that line above `applications:` (entries carry the installed cluster and version and a `type` of `ModulePlugin`, `OperatorBundle`, or `Chart`): + +```bash +sed -i '1i AppScenarioManifest:' apps.yaml +head -n 3 apps.yaml +``` + +Import the file again on the upgrade download page and confirm that the Packages list shows updates for the plugins that are actually installed. + +### Cause 2: The upgrade prompt lists every compatible version of each package + +Download the versions the upgrade prompt lists; they are all compatible with the target version. + +### Cause 3: An artifact ID changed or a scenario category missed a package + +For each entry the portal reports as missing: + +1. Search for the name with a `-` added (`log-api`, `log-agent`) and in scenario category `common ACP Extension`. +2. Check that `log-center` is in your download set. +3. For each "no package for this version" entry, confirm in the target version's release notes or with Alauda support whether the component was removed or merged into the core package. Drop it from the list only after that confirmation. + +Compare the final download set with the installed list: + +```bash +violet list --installed --platform-address --platform-token '' +``` + +### Cause 4: Some packages for the target version are not released yet, or have no platform version + +When some target packages are not released or plugins show no match, send the environment's `apps.yaml`, the architecture, and the current platform version to Alauda support for a comparison, and download the packages on the resulting "must download" list. When the portal import shows every required package as available, no review is needed. Leave self-uploaded plugins with no platform version out of the upgrade. For OAM on 4.3 there is nothing to download; for Ingress-nginx, plan to install `ingress-nginx-operator` from the core package. If a package that a product upgrade document requires cannot be found on the download site, report the package names to Alauda support. + +```bash +kubectl get nodes -o custom-columns=NAME:.metadata.name,ARCH:.status.nodeInfo.architecture +``` + +### Cause 5: After 3.16 to 4.0.12 the package list must not be taken from app.yaml + +Download the packages for the 3.16.x to 4.0.12 upgrade from the batch-download scenario `ACP Upgrade from v3.x to v4.x` (App Marketplace > Batch Download > Upgrade). On the environment being upgraded, list what is installed, on the first control-plane node of the `global` cluster and with the violet client that matches the target version: + +```bash +violet list +``` + +Use that list as the bill of extension packages to download, leaving out rows for packages that are uploaded but not installed. If `violet list` still fails with the target-version client, contact Alauda support. + +### Cause 6: The published 4.0.7 extension package was missing plugins + +List what the downloaded package contains and compare it with the plugins that show as not listed: + +```bash +tar -tf | grep -i -E 'monitor|log|elasticsearch|es' +kubectl get moduleplugins | grep -i -E 'monitor|log|elasticsearch' +``` + +If the plugins are not in the archive, download the extension package again from the official site, push it, and check Cluster Plugins: + +```bash +violet push --platform-address --platform-username --platform-password '' +``` + +If the republished package still lacks them, report the missing plugin names to Alauda support. + +### Cause 7: Published but not installed packages do not have to be taken down + +Confirm that the packages in question are really not installed: + +```bash +violet list --platform-address --platform-username --platform-password '' +violet list --installed --platform-address --platform-username --platform-password '' +``` + +A package that appears only in the first list is published but not installed; leave it in place. An installed package is not covered by this conclusion. + +### Cause 8: violet list returns null for every plugin before the upgrade to 4.3.3 + +First rule out a command that did not reach the platform. Use the real address and token, not the template placeholders: + +```bash +violet version +violet list --platform-address "https://" --platform-token "" --output-file ./apps.yaml +ls -l ./apps.yaml +cat ./apps.yaml +``` + +If the file is written and the plugins still show `null`: + +1. Record the installed plugins and their versions from Cluster Plugins on the `global` cluster. That list is the inventory for the upgrade. +2. After the platform is upgraded to 4.3.3, run the inventory again with the `violet` client that matches 4.3.3. + +## Diagnostic Steps + +```bash +violet version +violet list --installed --platform-address --platform-token '' +head -n 3 apps.yaml +``` + +- A portal upload error on a YAML whose first line is `applications:` rather than `AppScenarioManifest:`, or `No Application` from a list that includes uninstalled plugins: Cause 1. +- Many versions per package after importing `apps.yaml`: Cause 2. +- `logapi` / `logagent` missing, `log-center` missing from a scenario, or "no package for this version": Cause 3. +- Target packages not yet released, self-uploaded plugins with no match, or packages required by a product upgrade document missing from the download site: Cause 4. +- A 3.16 to 4.0.12 lookup that still depends on `app.yaml`: Cause 5. +- Plugins still not listed after the whole extension package was pushed, because the archive does not contain them: Cause 6. +- A request to take down published but not installed packages: Cause 7. +- `null` for every plugin from `violet list` before the upgrade to 4.3.3, while Cluster Plugins shows plugins installed: Cause 8. + +When `violet list` returns nothing useful, also check what the command covers and who runs it. `violet list` lists extensions with the Aligned or Agnostic life cycle (cluster plugins of those types, installed operators, and Helm chart applications); Core plugins follow the cluster upgrade and are never listed. The platform account must have role property `System` and role name `platform-admin-system`; a `Custom` role cannot use the tool. Token-based inventory needs a recent `violet` client; older clients can fail with `failed to get user info from platform server`. diff --git a/docs/en/solutions/Pushed_Helm_chart_version_not_shown_because_violet_reused_an_Artifact_of_another_chart_repository_on_ACP.md b/docs/en/solutions/Pushed_Helm_chart_version_not_shown_because_violet_reused_an_Artifact_of_another_chart_repository_on_ACP.md new file mode 100644 index 000000000..a0edd7d6d --- /dev/null +++ b/docs/en/solutions/Pushed_Helm_chart_version_not_shown_because_violet_reused_an_Artifact_of_another_chart_repository_on_ACP.md @@ -0,0 +1,128 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Pushed Helm chart version not shown because violet reused an Artifact of another chart repository on ACP + +## Issue + +A middleware chart package (for example `nacos.ALL.v2.5.1-260420.tgz`) was pushed with `violet push` and the push finished, but the new version does not appear in the platform: Catalog > Helm Charts, the `public-charts` repository, and the version list show nothing new or only an old version. Pushing the same package again does not help; the `violet` output reports that everything already exists: + +```text +... already exists, skip push +artifact already exist, will create artifactversion cluster=global type=chart +artifactversion . already exist, skip it. cluster=global type=chart +``` + +The platform has more than one chart repository (for example the built-in chart repository and a Harbor chart repository), and the other repository already holds an older template of the same chart (for example `nacos` 2.2.3 in the Harbor chart repository, while the new package is `nacos` `v2.5.1-260420` for `public-charts`). + +The `artifact` / `artifactversion` objects exist, so this is not a push that never created them. + +## Root Cause + +### Cause 1: violet reused an Artifact owned by another chart repository + +The push succeeds: the images and the chart are stored, and an `ArtifactVersion` for the new tag is created in `cpaas-system` on the `global` cluster. The problem is the `Artifact` it is attached to. + +`Artifact` objects (`artifacts.app.alauda.io`) are named after the chart, and only one object with that name can exist in `cpaas-system`. When the chart name is already taken by an Artifact that belongs to a different chart repository, `violet` does not create a new Artifact. It logs `artifact already exist, will create artifactversion` and attaches the new `ArtifactVersion` to the existing Artifact through an ownerReference. For example: + +| Object | Labels and fields | +| --- | --- | +| Artifact `nacos` (existing) | `cpaas.io/chart-purpose: library-charts`, `cpaas.io/library: stable`, `cpaas.io/managed-by: violet`; `spec.artifactVersionSelector.matchLabels` `cpaas.io/artifact-version: nacos`; `imagePullSecrets` `ctr-library-charts`; `repository: middleware/chart-nacos` | +| ArtifactVersion `nacos.v2.5.1-260420` (new) | `cpaas.io/chart-purpose: public-charts`, `tag: v2.5.1-260420`, ownerReference to Artifact `nacos` | + +The repository label of the parent Artifact (`library-charts`) does not match the repository of the new version (`public-charts`), so the chart list for `public-charts` is not refreshed and the new version is not displayed. + +Because the `ArtifactVersion` already exists after the first push, every later push of the same package only logs `artifactversion . already exist, skip it`. Restarting platform components, re-syncing the chart repository, or pushing again does not change the parent Artifact. + +## Resolution + +### Cause 1: violet reused an Artifact owned by another chart repository + +Apply this only when all of the following are confirmed: + +- the `violet` output contains `artifact already exist, will create artifactversion`; +- `kubectl -n cpaas-system get artifact ` returns one Artifact whose `cpaas.io/chart-purpose` label names a different repository than the label on the new `ArtifactVersion`; +- the old chart in that other repository has the same name and is the version you intend to retire. + +1. Decide whether the old chart is still needed (for example, an old template that is incompatible with the current platform version and not used). Check whether any application still deploys from it: + + ```bash + kubectl get helmrequest -A | grep + kubectl get artifactversion -n cpaas-system --show-labels | grep + ``` + +2. Keep a copy of the objects before you change anything: + + ```bash + kubectl -n cpaas-system get artifact -o yaml > artifact--backup.yaml + kubectl -n cpaas-system get artifactversion . -o yaml > artifactversion--backup.yaml + ``` + +> **Warning:** Deleting the old template from the other chart repository removes that chart version from Catalog. The old version can no longer be selected for new deployments or upgrades, and applications deployed from it can no longer be upgraded or redeployed from that template. When the old `Artifact` is removed, Kubernetes garbage collection also removes the `ArtifactVersion` objects that list it as owner, including the version you just pushed; that is why the package is pushed again in step 4. Precondition: the `artifact already exist` signature and the repository-label mismatch are confirmed, the owner of the old chart agrees that it is retired, and the backups from step 2 and the old chart package (for example `helm pull oci://// --version `, or a download from Harbor) are saved. Rollback: upload the saved old package to its chart repository again; that brings the old template back, and with it the same name conflict. + +3. Delete the old `` template from the other chart repository (in the Harbor project that holds it). Then check whether the conflicting Artifact is gone: + + ```bash + kubectl -n cpaas-system get artifact --show-labels + ``` + + If it is still present with the other repository's label, collect `kubectl -n cpaas-system get artifact -o yaml` and contact Alauda support before pushing again. + +4. Push the new package again with the same command as before: + + ```bash + violet push .ALL..tgz \ + --platform-address "https://" \ + --platform-username "" \ + --platform-password "" + ``` + + This time `violet` creates the Artifact for the new chart. Images that are already present are still reported as `already exists, skip push`; that is expected. + +5. Verify the result. The `public-charts` repository lists `` under the published packages (Helm Chart), and Catalog > Helm Charts shows the application template with version ``: + + ```bash + kubectl -n cpaas-system get artifact --show-labels + kubectl -n cpaas-system get artifactversion . --show-labels + kubectl get artifactversion -A | grep + ``` + + Both objects now carry `cpaas.io/chart-purpose: public-charts`. + +## Diagnostic Steps + +1. Read the `violet push` output. `already exists, skip push` on the images only means the images were pushed before. The line that identifies this cause is `artifact already exist, will create artifactversion cluster=global type=chart`, followed on a re-push by `artifactversion . already exist, skip it`. + +2. List the chart repositories and look for a second repository that already holds the same chart name: + + ```bash + kubectl get chartrepos.app.alauda.io -A + kubectl get chartrepo -n cpaas-system -o custom-columns=NAME:.metadata.name,TYPE:.spec.type,URL:.spec.url,PHASE:.status.phase + ``` + +3. Find the Artifact and the ArtifactVersions for the chart on the `global` cluster. One Artifact that is much older than the push, plus both the old and the new version, matches this cause: + + ```bash + kubectl get artifact -A | grep + kubectl get artifactversion -A | grep + ``` + +4. Compare the repository labels of the parent Artifact and the new ArtifactVersion, and confirm the ownerReference: + + ```bash + kubectl -n cpaas-system get artifact -o yaml | grep -n -E 'chart-purpose|repository|imagePullSecrets|artifact-version' + kubectl -n cpaas-system get artifactversion . \ + -o jsonpath='purpose={.metadata.labels.cpaas\.io/chart-purpose} owner={.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name} tag={.spec.tag}{"\n"}' + ``` + +| Observation | Conclusion | +| --- | --- | +| Artifact `cpaas.io/chart-purpose` differs from the new ArtifactVersion's (for example `library-charts` vs `public-charts`), owner is that Artifact | This cause | +| Artifact and new ArtifactVersion carry the same repository label, upload succeeded, version still missing | The repository index did not refresh; check the chart repository sync state | +| ArtifactVersion of `type: chart` with empty PHASE, Artifact labels match the repository | Normal state of a chart package; chart artifacts are not reconciled and carry no phase | +| No `artifact` / `artifactversion` for the chart at all after a successful push | The push did not register the chart; this cause does not apply | diff --git a/docs/en/solutions/Pushed_package_not_listed_where_expected_global-only_plugins_cluster_affinity_catalog_source_or_plugin_type_on_ACP.md b/docs/en/solutions/Pushed_package_not_listed_where_expected_global-only_plugins_cluster_affinity_catalog_source_or_plugin_type_on_ACP.md new file mode 100644 index 000000000..093b3e47a --- /dev/null +++ b/docs/en/solutions/Pushed_package_not_listed_where_expected_global-only_plugins_cluster_affinity_catalog_source_or_plugin_type_on_ACP.md @@ -0,0 +1,203 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.2" + - "4.3" +--- + +# Pushed package not listed where expected: global-only plugins, cluster affinity, catalog source, or plugin type on ACP + +## Issue + +`violet push` succeeded and the objects exist, but the package does not show where it is expected: + +- The push targeted a DevOps or other business cluster and the Cluster Plugins page there stays empty (GitOps, `asm-global`). +- `violet push` reported success, the plugin is not under the business cluster's Cluster Plugins, and a business-side Harbor reports an image as present although it is not there. +- A cluster plugin can only be uploaded to `global`, and it is unclear which setting has to change, or how to name another cluster as the upload target. +- MetalLB was pushed with `--clusters `, the output says `moduleplugin metallb has been updated successfully cluster=global type=plugin`, and Cluster Plugins for that workload cluster (an HCS cluster) shows `No data` for MetalLB. +- The ClickHouse log plugin (`logclickhouse`) was uploaded for a dual-stack cluster and never appears in that cluster's plugin list, even after restarts and re-uploads. +- Flagger packages were pushed and are not in the expected package list. +- Kyverno was pushed, Cluster Plugins shows it as listed, and OperatorHub shows it as not listed. +- After a directory push, some `.tgz` packages show neither `present` nor `success`, and pushing again changes nothing. +- On ACP 4.0.3 no standalone ASM plugin package can be found. +- On a new ACP 4.3.0 installation, `envoy-gateway-operator` and `ingress-nginx-operator` seem absent from the marketplace and no Pods exist for them. + +Each of these is expected behavior for that package. A push whose `artifact` / `artifactversion` were never created is a different problem (see "violet push succeeds but the plugin is not published: Artifact, ModuleConfig, or chart missing on ACP"). + +## Root Cause + +### Cause 1: The plugin can only run on global + +The GitOps plugin is restricted to the `global` cluster: a successful push for a DevOps cluster does not create a plugin entry on that cluster, and raising the GitOps package to 4.3.0 does not remove the restriction. On ACP 4.0.3 the `asm-global` cluster plugin behaves the same way and is displayed only in the `global` cluster. + +### Cause 2: violet push without --clusters targets global and its built-in registry + +`violet push` without `--clusters ` uploads to the `global` cluster, and the default push target is the `global` cluster's built-in registry, not a business-side Harbor; an image check in that Harbor does not tell whether the push worked. + +For cluster plugins this is not a default that can be changed. The `--clusters` parameter is not needed when uploading a cluster plugin, because the platform distributes the plugin to clusters automatically according to its affinity configuration; if `--clusters` is given, it is ignored. A cluster plugin is always uploaded to `global`, and no plugin configuration needs to change. Older built-in help that says plugins go "only to the `global` cluster" describes the same behavior. `--clusters` matters for operator packages (`type=bundle`), not for cluster plugins (`type=plugin`). + +### Cause 3: ModulePlugin clusterAffinity does not match the target cluster + +A ModulePlugin can carry an `affinity` block, and the plugin is offered only to clusters that match it. Re-uploading and restarting `cluster-transformer` or OLM does not change that. + +- `clusterAffinity.matchLabels`: the ClickHouse log plugin `logclickhouse` (version 4.3.1) carries `cpaas.io/protocol-stack: IPv4/IPv6`, so it is not offered to a cluster without that label (for example a newly installed ACP 4.3.2 x86 dual-stack cluster). Removing that entry from `moduleplugin logclickhouse` makes the plugin appear and publishable. +- `clusterAffinity.matchExpressions`: the `metallb` ModulePlugin (`metallb.amd64.v4.2.5.tgz`) carries key `cluster-type`, operator `In`, values `Baremetal` and `DCS`. An HCS workload cluster does not match, so MetalLB is registered on `global` but never offered for that cluster, and pushing again only prints `already exists, skip push` and `moduleplugin metallb has been updated successfully cluster=global type=plugin`. Adding `HCS` to the values makes MetalLB available for that cluster. + +### Cause 4: The package is registered in the system catalog, not the platform catalog + +The Flagger packages have catalog source `system`, not `platform`, so they are not listed in the platform catalog; they appear under Installed Plugins. Both are installed automatically by the `global` cluster controller when a service mesh is created, so no manual installation is needed. + +### Cause 5: The package is a cluster plugin, not an Operator + +The `violet push` output says `moduleplugin has been created successfully cluster=global type=plugin`. Kyverno `v4.2.3` (`kyverno.ALL.v4.2.3.tgz`) is delivered as a cluster plugin (`moduleplugin`, `type=plugin`), not as an Operator, so it never appears in OperatorHub. The "Kyverno Operator" card that a search for `kyverno` finds in OperatorHub on `global` (Community, provided by Kyverno, status `未上架`, not listed) is a separate community operator; its status is unrelated, and Kyverno is not deployed from Operators. + +Cluster plugins are managed through the cluster-level CRDs `ModulePlugin`, `ModuleConfig`, and `ModuleInfo`. A plugin is published with `violet` to the `global` cluster only, and the platform then creates its `ModulePlugin` and `ModuleConfig` there; it can be installed on `global` or on any workload cluster, and installing it creates a `ModuleInfo`. Even `--clusters "global,,"` reports `cluster=global type=plugin`. + +### Cause 6: Chart-type packages show no present or success status + +After a directory push (for example with `violet` 4.2.9), `kubectl get artifactversion -A | grep -vi succe` lists ArtifactVersions in `cpaas-system` with empty PHASE and REASON columns (for example `chart-milvus-operator.v1.3.5`, `dify.v1.11.4`, `kubeflow-model-registry.v1.10.10`, `label-studio.v1.21.0-2`), and their artifact is `type: chart`. Artifacts of type `chart` are not reconciled, so their ArtifactVersion has no phase or status. `present: true` with the version in `spec` confirms the push succeeded. The empty status needs no fix. It does not prove the version is displayed; see "Pushed Helm chart version not shown because violet reused an Artifact of another chart repository on ACP" when the new chart version is also missing from the platform. + +### Cause 7: ACP 4.0.3 ships plugins inside the extension package + +In 4.0.3 the extension package is not split into individual plugin packages. It bundles every plugin, including ASM. Take the ASM plugin from the 4.0.3 extension package. + +### Cause 8: Operators shipped with the release show as ready and have no Pods until they are deployed + +`envoy-gateway-operator` and `ingress-nginx-operator` are shipped with 4.3.0. In the platform's operator list they are in the `ready` state, next to many operators in the `absent` state, which makes them easy to overlook. No Pods exist because neither operator has been deployed yet. + +## Resolution + +### Cause 1: The plugin can only run on global + +Push and install the plugin on the `global` cluster, then open Cluster Plugins on `global`. Confirm the same restriction before treating any other package as global-only; it is established for GitOps and `asm-global`. + +```bash +violet push --platform-address= --platform-username= --platform-password='' --clusters global .tgz +``` + +### Cause 2: violet push without --clusters targets global and its built-in registry + +Look for the plugin on `global` first: + +```bash +kubectl get moduleplugins | grep -i +kubectl get artifact -A | grep -i +``` + +To make an operator package (`type=bundle`) available to a business cluster, name that cluster explicitly: + +```bash +violet push --platform-address= --platform-username= --platform-password='' --clusters .tgz +``` + +For a cluster plugin (`type=plugin` in the push output), push without `--clusters`. If it is not offered to the cluster you expect, read its affinity (Cause 3) instead of pushing again with another `--clusters` value. + +### Cause 3: ModulePlugin clusterAffinity does not match the target cluster + +Read the affinity on the ModulePlugin and compare it with the labels and type of the target cluster: + +```bash +kubectl get moduleplugin -o yaml > "$HOME/moduleplugin-.before.yaml" +grep -n -A8 'affinity' "$HOME/moduleplugin-.before.yaml" +``` + +> **Warning:** Changing `clusterAffinity` makes the plugin eligible for clusters the package author excluded. Installing it on a cluster the plugin does not support can fail or misbehave. Precondition: only when the plugin is missing from the cluster plugin list, its affinity does not match the target cluster, you have confirmed the plugin is meant to run there, and the export above is kept. Rollback: `kubectl apply -f "$HOME/moduleplugin-.before.yaml"`. A later push of the same package may also restore the original ModulePlugin. + +```bash +kubectl edit moduleplugin +``` + +For a `matchLabels` entry such as `cpaas.io/protocol-stack: IPv4/IPv6` on `logclickhouse`, delete that line (and the now-empty `matchLabels` / `clusterAffinity` keys if nothing else remains). For a `matchExpressions` list such as the `metallb` `cluster-type` values, add the missing cluster type instead of removing the expression: + +```yaml +spec: + affinity: + clusterAffinity: + matchExpressions: + - key: cluster-type + operator: In + values: + - Baremetal + - DCS + - HCS +``` + +Save, then open Cluster Plugins for the target cluster. + +### Cause 4: The package is registered in the system catalog, not the platform catalog + +```bash +kubectl get packagemanifest -A -o custom-columns=NAME:.metadata.name,CATALOG:.status.catalogSource | grep -i flagger +``` + +Catalog `system` means the package is expected to be absent from the platform catalog; look for it under Installed Plugins. For Flagger, no manual installation is needed. + +### Cause 5: The package is a cluster plugin, not an Operator + +```bash +kubectl get moduleplugins | grep -i kyverno +kubectl get artifact -A | grep -i kyverno +``` + +If the ModulePlugin is missing, push the package: + +```bash +violet push kyverno.ALL.v4.2.3.tgz --platform-address --platform-username --platform-password '' +``` + +Then open Cluster Plugins, select the cluster that should run Kyverno in the cluster selector, and install it there. The Upload Packages page (App Market Management, `上架软件包`) with the same cluster selector shows whether the package is offered to that cluster. Check the result on `global`: + +```bash +kubectl get minfo | grep -i kyverno +``` + +### Cause 6: Chart-type packages show no present or success status + +```bash +kubectl get artifactversion -A | grep -vi succe +kubectl -n cpaas-system get artifactversion -o yaml | grep -n -E 'present|tag|type' +kubectl -n cpaas-system get artifact -o yaml | grep -n 'type:' +``` + +`type: chart` with `spec.present: true` and the expected version is the expected state. Pushing again does not add a status. Look for the chart under Catalog > Helm Charts, not under Cluster Plugins or OperatorHub. + +### Cause 7: ACP 4.0.3 ships plugins inside the extension package + +Unpack the 4.0.3 extension package and push the ASM plugin from it with `violet`: + +```bash +tar -tf | grep -i asm +``` + +The `asm-global` plugin from that package is shown on `global` only (Cause 1). + +### Cause 8: Operators shipped with the release show as ready and have no Pods until they are deployed + +```bash +kubectl get artifactversion -A | grep -iE 'envoy-gateway-operator|ingress-nginx-operator' +kubectl get packagemanifest -A | grep -iE 'envoy-gateway|ingress-nginx' +``` + +When both are present, deploy the operator you need from OperatorHub. Pods appear only after the operator is installed. + +## Diagnostic Steps + +```bash +kubectl get moduleplugins +kubectl get moduleplugin -o yaml | grep -n -A8 affinity +kubectl get packagemanifest -A -o custom-columns=NAME:.metadata.name,CATALOG:.status.catalogSource +kubectl get artifact,artifactversion -A | grep +``` + +- An empty plugin page on a non-global cluster for GitOps or `asm-global`: Cause 1. +- A push without `--clusters`, or a question of how to push a cluster plugin to a cluster other than `global`: Cause 2. +- A `clusterAffinity` (`matchLabels`, or a `matchExpressions` list such as `cluster-type`) that the target cluster does not satisfy: Cause 3. +- Catalog source `system`: Cause 4. +- A `type=plugin` ModulePlugin missing from OperatorHub: Cause 5. +- A chart-type package with `spec.present: true` and no status: Cause 6. +- A 4.0.3 plugin with no standalone package: Cause 7. +- Operators listed as `ready` with no Pods: Cause 8. diff --git a/docs/en/solutions/Replacing_or_rejoining_an_ACP_master_node_same-IP_replacement_etcd-join_extra_masters_and_control-plane_requests.md b/docs/en/solutions/Replacing_or_rejoining_an_ACP_master_node_same-IP_replacement_etcd-join_extra_masters_and_control-plane_requests.md new file mode 100644 index 000000000..555468d17 --- /dev/null +++ b/docs/en/solutions/Replacing_or_rejoining_an_ACP_master_node_same-IP_replacement_etcd-join_extra_masters_and_control-plane_requests.md @@ -0,0 +1,151 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Replacing or rejoining an ACP master node: same-IP replacement, etcd-join, extra masters, and control-plane requests + +## Issue + +- A master has to be replaced (virtual machine to physical machine, or the reverse), and the question is whether to delete it and add it again. +- After a master was removed and added back, kube-apiserver restarts and nothing listens on `127.0.0.1:2379`. +- An extra master is requested so that external clients can call `http://:6443` and master CPU is spread out, or apiserver is to run on two workers. +- During a drill, a newly added machine is expected to become a master after a master fails. +- Master component CPU and memory requests look too small. + +Related articles: "ACP control-plane limits: apiserver address change, snapshot.db scope, total master loss, x86 to ARM, and planned shutdown" covers changes that have no supported procedure. + +## Root Cause + +### Cause 1: Replace one master and keep the same node IP + +When the new machine keeps the old master IP, the path for a business cluster is to delete that master in the platform UI and add it again, one node at a time. After each node, etcd membership and the Kube-OVN `OVN_Northbound` and `OVN_Southbound` databases must list the new master and must not list the removed one. + +The Global Cluster Master Node Reset and Replacement operation guide is validated for v3.18.x, v4.0.x, v4.1.x, and v4.2.x, not for v4.3.x; on 4.3 use it as a reference only after a test cluster has been through it. Keeping the IP is a precondition in both paths. Adding a spare node now so that it can take over a master role later is not possible, because the node role is chosen when the node joins and cannot be changed afterwards. + +### Cause 2: After rejoin, nothing listens on 127.0.0.1:2379 because etcd-join did not finish + +The master was removed and added back. kube-apiserver is already generated and keeps exiting, and nothing listens on `127.0.0.1:2379`, so kube-apiserver cannot reach etcd. While the node was out, CoreDNS P99 latency on another master could stay high. The kubeadm etcd join phase did not run; `kubeadm join phase etcd-join --config=/etc/kubernetes/kubeadm-config.yaml --v=5` on the cleaned node completes it. This is not a WAL corruption and not a reason to delete `/var/lib/etcd`. + +### Cause 3: Another master only to expose its :6443 or spread load + +Adding control-plane nodes is supported, and the count should stay odd (3 or 5). Publishing `http://:6443` to external clients, or running apiserver alone on former workers to spread load, is not an accepted layout and does not meet the need to spread pressure. + +### Cause 4: A machine added in an expansion plan stays a compute node + +In a drill that adds a device and then fails a master, the new device is expanded as a compute node, not imaged as a management node. The failed master is recovered first with the replacement procedure (Cause 1) until the cluster is back to three masters; after that a further master or worker failure still leaves the platform running. No step turns the new device into a master. + +### Cause 5: apiserver and etcd requests are adjusted in their static Pod YAML + +Master system-component requests are changed by editing the static Pod YAML of the components, including kube-apiserver and etcd. There is no fixed target value; use the size planned for the cluster. etcd Pods that are OOM-killed (with etcd sync then failing) are addressed by increasing the etcd component's resources. + +## Resolution + +### Cause 1: Replace one master and keep the same node IP + +Continue only when the replacement machine uses the same IP as the master being removed, you are on 3.18, 4.0, 4.1, or 4.2 (or on 4.3 after a test cluster completed the procedure), and you replace one master at a time. Masters that also run workloads or pin log and monitoring Pods need extra planning; confirm the node is a clean master before the delete. + +> **Warning:** Deleting a master in the platform UI removes a control-plane node and its etcd member. Do it only when the replacement keeps the same IP, `endpoint health` shows the other members healthy, and only one master at a time. The delete cannot be undone; the rollback is to add a machine back at that IP and wait until `member list` and both OVN cluster statuses agree. + +Wait until the delete finishes before the add. After adding the new machine in the UI with the same IP, confirm membership before touching the next master: + +```bash +kubectl -n kube-system exec etcd- -- etcdctl \ + --endpoints=https://127.0.0.1:2379 \ + --cacert=/etc/kubernetes/pki/etcd/ca.crt \ + --cert=/etc/kubernetes/pki/etcd/server.crt \ + --key=/etc/kubernetes/pki/etcd/server.key \ + member list -w table +``` + +The removed master's peer URL must be absent and the new master's present. On Kube-OVN clusters, check the OVN databases in Deployment `ovn-central` in `kube-system`: + +```bash +kubectl -n kube-system exec deploy/ovn-central -c ovn-central -- ovn-nbctl cluster/status OVN_Northbound +kubectl -n kube-system exec deploy/ovn-central -c ovn-central -- ovn-sbctl cluster/status OVN_Southbound +``` + +Both must show the new master and not the removed one. If `-c ovn-central` fails, read the container name from `kubectl -n kube-system get deploy ovn-central -o yaml`. Then check etcd health: + +```bash +kubectl -n kube-system exec etcd- -- etcdctl \ + --endpoints=https://127.0.0.1:2379 \ + --cacert=/etc/kubernetes/pki/etcd/ca.crt \ + --cert=/etc/kubernetes/pki/etcd/server.crt \ + --key=/etc/kubernetes/pki/etcd/server.key \ + endpoint health --cluster +``` + +Proceed to the next master only when that command reports every member healthy and the two OVN statuses match the new set. If the replacement uses a different IP, this procedure does not apply; update the VIP forwarding policy to the new IP before clients use it, and follow the platform procedure for replacing masters with new IPs. + +### Cause 2: After rejoin, nothing listens on 127.0.0.1:2379 because etcd-join did not finish + +Apply only when this master was removed and is being added back, kube-apiserver is restarting, and nothing listens on `127.0.0.1:2379`. Confirm the other members are healthy first: + +```bash +ss -lnt | grep 2379 || true +ls -l /etc/kubernetes/kubeadm-config.yaml +``` + +> **Warning:** `kubeadm join phase etcd-join` adds this node's etcd member and starts it. Run it only on the cleaned node that is missing from `member list` or has no listener on `127.0.0.1:2379` after a rejoin, in a maintenance window, with the other members healthy. If the phase errors, stop and collect `member list` from a healthy master; do not wipe `/var/lib/etcd` unless the WAL is corrupt (`walpb: crc mismatch`). + +```bash +sudo kubeadm join phase etcd-join --config=/etc/kubernetes/kubeadm-config.yaml --v=5 +``` + +Afterwards `ss -lnt | grep 2379` shows a listener and kube-apiserver stops restarting. Check membership with the `member list` command from Cause 1 before the next master change. + +### Cause 3: Another master only to expose its :6443 or spread load + +If a real extra master is needed for quorum, keep the control-plane count odd (3 or 5), add it through the normal master-add path with its own IP, and check membership as in Cause 1. Do not build an external-endpoint or worker-hosted apiserver layout by hand. + +### Cause 4: A machine added in an expansion plan stays a compute node + +Treat the new device as a worker. If a master has already failed, restore that master with Cause 1 until `member list` shows three healthy members, then continue the drill. + +```bash +kubectl get nodes -l node-role.kubernetes.io/control-plane= -o wide +kubectl get nodes -o wide +``` + +### Cause 5: apiserver and etcd requests are adjusted in their static Pod YAML + +```bash +sudo ls -l /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/manifests/etcd.yaml +sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /var/tmp/kube-apiserver.yaml.before +sudo cp /etc/kubernetes/manifests/etcd.yaml /var/tmp/etcd.yaml.before +``` + +> **Warning:** Saving `/etc/kubernetes/manifests/kube-apiserver.yaml` or `etcd.yaml` restarts that static Pod on this node. One master at a time, in a maintenance window, and change only `resources.requests`. A request larger than the node's allocatable CPU or memory leaves the static Pod Pending. Keep the backups outside `/etc/kubernetes/manifests/`. Rollback: copy the backups back. + +Edit `resources.requests`, save, and wait until the Pod is Ready before the next master: + +```bash +kubectl -n kube-system get pods -l 'component in (kube-apiserver,etcd)' -o wide +``` + +## Diagnostic Steps + +```bash +kubectl -n kube-system get pods -l component=etcd -o wide +kubectl get nodes -l node-role.kubernetes.io/control-plane= -o wide +kubectl get machine -A +``` + +- Same IP, one master, version 3.18 to 4.2 (or 4.3 after a test run): Cause 1. +- Spare node added ahead of time so it can later become a master: not possible; the role is fixed at join (Cause 1). +- Rejoin, kube-apiserver restarting, nothing on `127.0.0.1:2379`: Cause 2. +- Request for an extra endpoint or worker-hosted apiservers: Cause 3. +- New device during a drill, expected to become a master: Cause 4. +- apiserver or etcd `requests` too small, or etcd OOM-killed: Cause 5. + +A `member list` that still contains a node you already deleted is an unfinished Cause 1 check, not a reason to wipe `/var/lib/etcd` on the surviving members. diff --git a/docs/en/solutions/base_AppRelease_stays_failed_because_the_upgrade-sysctl-config_BroadcastJob_cannot_finish_on_a_large_cluster_on_ACP.md b/docs/en/solutions/base_AppRelease_stays_failed_because_the_upgrade-sysctl-config_BroadcastJob_cannot_finish_on_a_large_cluster_on_ACP.md index 4d8a54d75..4327fe42a 100644 --- a/docs/en/solutions/base_AppRelease_stays_failed_because_the_upgrade-sysctl-config_BroadcastJob_cannot_finish_on_a_large_cluster_on_ACP.md +++ b/docs/en/solutions/base_AppRelease_stays_failed_because_the_upgrade-sysctl-config_BroadcastJob_cannot_finish_on_a_large_cluster_on_ACP.md @@ -17,10 +17,20 @@ kubectl -n cpaas-system get ar base kubectl -n cpaas-system get broadcastjob upgrade-sysctl-config ``` +The same AppRelease can also stay in `syncing` on a cluster of any size, for example when the pre-upgrade check of a business cluster shows its base component (Cluster Base, AppRelease `base`) in `syncing` and it never finishes, or a business cluster ARS stays `syncing` for a long time. `kubectl get bcj -n cpaas-system` then shows a BroadcastJob (`broadcastjobs.operator.alauda.io`, short name `bcj`) whose `SUCCEEDED` count stays below `DESIRED`, or with `FAILED` greater than `0`. Adding a dummy key to the ARS values to force a new sync does not help. + ## Root Cause The `base` AppRelease includes the BroadcastJob `upgrade-sysctl-config`, which runs a Pod on every node. The AppRelease has a timeout (`spec.timeout`) within which its resources must become ready. On a large cluster the BroadcastJob cannot finish on all nodes within that timeout. The sentry component that watches the release then judges the operation as timed out and runs it again. The new run starts over and times out the same way, so the `base` AppRelease never becomes healthy. +### A Pod of upgrade-sysctl-config was Evicted + +`kubectl get job,bcj -n cpaas-system` shows `upgrade-sysctl-config` with, for example, `DESIRED 5`, `ACTIVE 0`, `SUCCEEDED 4`, `FAILED 1`, and one `upgrade-sysctl-config-` Pod in `cpaas-system` is `Evicted` while the others are `Completed`. Because of the failed Pod the BroadcastJob never counts all nodes as succeeded, and the base component stays in `syncing`. Ordinary Jobs in the same list (for example `keep-marketplace-resources`, `Complete`) are not involved. Deleting the Evicted Pod lets the base component recover. + +### update-containerd-config does not finish + +The `sentry` log ties the long `syncing` state of a business cluster ARS to BroadcastJob `update-containerd-config`, and some of its jobs are stuck. `update-containerd-config` synchronizes and updates the containerd configuration on the nodes and is triggered whenever the business cluster ARS is updated. Raising the BroadcastJob `parallelism` to `10` lets every job finish with `success`, and the ARS returns to normal. Why individual jobs get stuck is not known. + ## Resolution Use one of the two options. @@ -60,6 +70,58 @@ kubectl -n cpaas-system get ar base -w kubectl -n cpaas-system scale deploy/cluster-transformer --replicas= ``` +The two options above apply when the BroadcastJob is only slow. The following sections apply to the other two causes in Root Cause. + +### Evicted Pod in upgrade-sysctl-config: delete the Evicted Pod + +1. Confirm the signature and read the eviction message, which names the node resource that ran short: + + ```bash + kubectl get job,bcj -n cpaas-system + kubectl -n cpaas-system get pods -o wide | grep upgrade-sysctl-config + kubectl -n cpaas-system get pod -o jsonpath='{.status.phase} {.status.reason}: {.status.message}{"\n"}' + ``` + +2. Check that the node no longer reports that pressure; a node still under memory or disk pressure has to be cleared first, otherwise a new Pod there can be evicted again: + + ```bash + kubectl describe node | grep -E 'MemoryPressure|DiskPressure|PIDPressure' + ``` + +3. Save the Evicted Pod, because its status is the only record of the eviction: + + ```bash + kubectl -n cpaas-system get pod -o yaml > "$HOME/.yaml" + ``` + +> **Warning:** Deleting the Pod object removes its eviction status and message from the cluster; the Pod is not running, so no workload stops. Precondition: only when `upgrade-sysctl-config` (or the BroadcastJob the base component waits on) shows `FAILED` greater than `0`, the failed Pod is `Evicted`, the other Pods of that BroadcastJob are `Completed`, and the YAML above is saved. Rollback: none for the Pod object; the saved YAML keeps the eviction details. + +4. Delete the Evicted Pod and watch the base component: + + ```bash + kubectl -n cpaas-system delete pod + kubectl get job,bcj -n cpaas-system + kubectl get ars -A | grep -i base + ``` + +Confirm that the component shows synced on the cluster page before you start a business cluster upgrade. If it stays in `syncing`, collect the BroadcastJob YAML and the `sentry` log before trying anything else. + +### update-containerd-config does not finish: raise its parallelism + +```bash +kubectl get broadcastjob -A | grep update-containerd-config +kubectl -n get broadcastjob update-containerd-config -o yaml > "$HOME/update-containerd-config.yaml" +``` + +> **Warning:** A higher `parallelism` updates the containerd configuration on more nodes at the same time. Precondition: only when the ARS stays `syncing` and the `sentry` log points to an incomplete `update-containerd-config` BroadcastJob. Rollback: set `parallelism` back to the value in the export. + +```bash +kubectl -n patch broadcastjob update-containerd-config --type=merge -p '{"spec":{"parallelism":10}}' +kubectl -n get broadcastjob update-containerd-config -w +``` + +Wait until all jobs are `success`, then check that the ARS leaves `syncing`. + ## Diagnostic Steps 1. Check the state of the `base` AppRelease and its conditions: @@ -83,3 +145,11 @@ kubectl -n cpaas-system get ar base -w kubectl -n cpaas-system get pods | grep upgrade-sysctl-config | grep -v -E 'Running|Completed' kubectl -n cpaas-system describe pod ``` + +4. List every BroadcastJob and read the `sentry` log, because the BroadcastJob that holds the AppRelease differs between cases. `FAILED` greater than `0` with an `Evicted` Pod is the Evicted-Pod cause. An `update-containerd-config` that stays below `DESIRED` with stuck jobs, named in the `sentry` log, is the `update-containerd-config` cause. If every BroadcastJob and Job is complete and the `sentry` log shows `Watching for changes to Job ` followed by `Jobs active: 0`, the chart is waiting on a hook Job instead: + + ```bash + kubectl get bcj -A + kubectl get pods -A | grep -E 'upgrade-sysctl-config|update-containerd-config' + kubectl -n cpaas-system logs --tail=500 | grep -E 'update-containerd-config|upgrade-sysctl-config|Watching for changes to Job|Jobs active' + ``` diff --git a/docs/en/solutions/etcd_and_kube-apiserver_do_not_start_on_a_master_after_runc_or_containerd_is_replaced_on_ACP.md b/docs/en/solutions/etcd_and_kube-apiserver_do_not_start_on_a_master_after_runc_or_containerd_is_replaced_on_ACP.md new file mode 100644 index 000000000..9c4a94ccf --- /dev/null +++ b/docs/en/solutions/etcd_and_kube-apiserver_do_not_start_on_a_master_after_runc_or_containerd_is_replaced_on_ACP.md @@ -0,0 +1,190 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.1" + - "4.2" + - "4.3" +--- + +# etcd and kube-apiserver do not start on a master after runc or containerd is replaced on ACP + +## Issue + +After a kernel upgrade, or after Docker (`docker-ce`) was installed on a master, the static Pods `etcd-` and `kube-apiserver-` in `kube-system` stay in `CrashLoopBackOff` or never start, and every Pod on that node fails to start. Typical signs: + +- kubelet repeats `Failed to delete cgroup paths`. +- `crictl ps -a | grep etcd` shows etcd `Exited` with `"reason": "StartError"` and `"exitCode": 128`, and `crictl logs` prints nothing because the container never started. +- containerd logs a `shim-run-v2` error, and etcd and kube-apiserver stay down even after `/usr/bin` was recopied from a healthy node. +- Copying `containerd` and `runc` into `/usr/bin` and restarting kubelet does not help while an older `runc` under `/usr/local/bin` is still the one found first on `PATH`. +- The command line stops working after Docker was installed, and after Docker is uninstalled the scheduler still does not start. +- calico-node restarts with a port conflict, and two calico containers cannot be deleted. + +Related articles: "Control-plane Pods do not start with mkdir /var/log/pods: file exists after containerd directories are lost on ACP" covers a master where kubelet and containerd run but no container exists. "Container runtime conflicts on ACP nodes: Docker, podman, docker-runc, and overwritten runtime binaries" covers the same overwrite on worker nodes. + +## Root Cause + +### Cause 1: /usr/local/bin/runc is an old build after a kernel upgrade + +After a kernel upgrade (for example on Kylin), etcd and kube-apiserver on the upgraded master enter `CrashLoopBackOff` and kubelet cannot create the sandbox. `which -a runc` lists `/usr/local/bin/runc` before `/usr/bin/runc`. The first binary is `runc 1.0.0-rc3`, a build shipped with the OS image, which does not match the new kernel's IPC namespace handling, so sandbox creation fails. The working binary is the platform's `runc` under `/usr/bin` (for example `1.1.12`). Recopying `/usr/bin` and restarting kubelet and containerd does not help until `/usr/local/bin/runc` is moved aside and the runtime is restarted. + +### Cause 2: Installing docker-ce replaced runc and containerd + +`yum install docker-ce` replaced the `runc` and `containerd` binaries. After a reboot no Pod starts, and kubelet repeats `Failed to delete cgroup paths` from `pod_container_manager_linux.go:210`. On a containerd cluster the overwrite can show up only after a reboot: `crictl inspect` of the etcd container shows `"state": "CONTAINER_EXITED"`, `"exitCode": 128`, `"reason": "StartError"`, and: + +```text +failed to create containerd task: failed to create shim task: OCI runtime create failed: container_linux.go:311: creating new parent process caused "container_linux.go:1572: running lstat on namespace path \"/proc//ns/ipc\" caused \"lstat /proc//ns/ipc: no such file or directory\"": unknown +``` + +`systemctl status docker` shows `docker.service` `active (running)` on the master. If several masters are affected (for example one failed and a second was rebooted), etcd loses quorum. Restoring an etcd backup does not help; restoring the overwritten binaries from a healthy node brings etcd back and the nodes to `Ready`. + +Restoring the binaries without removing Docker is enough for etcd and kube-apiserver, which use the host network, but a running `dockerd` on a platform node is not supported: it rewrites `docker0`, iptables, NAT, and `FORWARD` rules, which breaks Pod-network traffic on the node (for example Pods that cannot reach the `kubernetes` Service on `443`) while host-network static Pods keep working. + +### Cause 3: /usr/local/bin still has Docker runtime binaries + +Docker installed on the containerd node left `runc`, `containerd`, or `containerd-shim*` copies in `/usr/local/bin`, which are found first on `PATH`. containerd logs a `shim-run-v2` error, and etcd and kube-apiserver stay down after `/usr/bin` was recopied. After `systemctl stop kubelet` and `systemctl stop containerd`, many `/usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id ...` processes can remain. Moving the leftovers aside, copying the containerd binaries again, and restarting containerd and kubelet is the recovery; a reboot may also be needed. + +### Cause 4: After Docker is uninstalled the scheduler stays down until a reboot + +The cluster is abnormal, `kubectl` cannot run, and Docker is installed on a node. After the Docker package is uninstalled the scheduler still cannot start; rebooting that node brings the scheduler and the cluster back. A reboot alone is enough only when the Docker packages did not own the platform runtime files. + +## Resolution + +Before any step: if the API server still answers, cordon and drain the node (`kubectl drain --ignore-daemonsets --delete-emptydir-data`). When etcd or kube-apiserver on this master is down and `kubectl` does not answer, skip the drain and work from a console session. Static Pods are not evicted by a drain either way. + +### Cause 1: /usr/local/bin/runc is an old build after a kernel upgrade + +Apply only when etcd and kube-apiserver are in `CrashLoopBackOff` after a kernel upgrade, `/usr/local/bin/runc --version` prints `1.0.0-rc3` (or another build older than the healthy node's), and `/usr/bin/runc` matches the healthy node: + +```bash +which -a runc +/usr/local/bin/runc --version +/usr/bin/runc --version +``` + +> **Warning:** Moving `/usr/local/bin/runc` changes which runc creates the next sandbox. New containers wait until containerd is restarted. Rollback: `sudo mv /usr/local/bin/runc.old /usr/local/bin/runc` and restart containerd again. Keep the old file. + +```bash +sudo mv /usr/local/bin/runc /usr/local/bin/runc.old +hash -r +command -v runc +runc --version +sudo systemctl restart containerd +sudo systemctl restart kubelet +``` + +`runc --version` must now match `/usr/bin/runc`. etcd and kube-apiserver should leave `CrashLoopBackOff` without a manifest edit and without deleting `/var/lib/etcd`. + +### Cause 2: Installing docker-ce replaced runc and containerd + +Compare the node with a healthy master of the same cluster: + +```bash +which -a runc containerd +runc --version +containerd --version +ls -l "$(command -v runc)" "$(command -v containerd)" +rpm -q docker-ce || true +rpm -qf "$(command -v runc)" "$(command -v containerd)" +``` + +If `rpm -qf` shows that a Docker package (for example `docker-ce` or `containerd.io`) owns `runc` or `containerd` and you also intend to remove Docker, remove it before the copy: removing the package deletes the files it owns, including freshly restored ones. + +Copy `runc` and `containerd` from the healthy node to `/var/tmp/runc.from-healthy` and `/var/tmp/containerd.from-healthy` on this node. If the healthy node's `ls -l` shows other differences, also copy `ctd-decoder`, `containerd-shim-runc-v2`, `ctr`, `containerd-shim`, `crictl`, `containerd-shim-runc-v1`, `critest`, `containerd-stress`, `kubelet`, and `kubectl` into `/usr/bin`. + +> **Warning:** Replacing `runc` and `containerd` and restarting containerd stops new container creation on this node until the units are back. Running containers survive the stop only when the containerd unit has `KillMode=process` (check `systemctl show containerd -p KillMode`); with `control-group` or `mixed`, stopping containerd stops every container on the node, including static Pods that are still up. Apply only when the cgroup-path log or the `StartError` is confirmed and the healthy-node binaries differ. Stop kubelet and containerd before the copy; overwriting a running `containerd` fails with `Text file busy`. Match owner and mode to the healthy node. Rollback: stop kubelet and containerd, copy `/var/tmp/runc.before-restore` and `/var/tmp/containerd.before-restore` back, and start containerd and kubelet. This does not touch etcd data. + +```bash +RUNC="$(command -v runc)"; CONTAINERD="$(command -v containerd)" +sudo cp -a "$RUNC" /var/tmp/runc.before-restore +sudo cp -a "$CONTAINERD" /var/tmp/containerd.before-restore +sudo systemctl stop kubelet containerd +sudo cp -a /var/tmp/runc.from-healthy "$RUNC" +sudo cp -a /var/tmp/containerd.from-healthy "$CONTAINERD" +sudo systemctl start containerd +sudo systemctl start kubelet +``` + +If `which` on the healthy node and on this node disagree, restore onto the healthy node's path and move any extra copy under `/usr/local/bin` aside (Cause 3). Pods, including the static Pods, should start once kubelet can create cgroups again. If Pods on this node then cannot reach the `kubernetes` Service, stop and remove Docker, or take the node out of the cluster if Docker must stay. + +### Cause 3: /usr/local/bin still has Docker runtime binaries + +Apply only when the runtime is containerd, Docker was installed on the node, containerd logs `shim-run-v2`, and `/usr/local/bin` holds `runc`, `containerd`, or `containerd-shim` files that a healthy node does not have: + +```bash +rpm -qa | grep docker || true +ls -l /usr/local/bin | grep -E 'runc|containerd|shim' || true +which -a runc containerd +ps -ef | grep '[c]ontainerd-shim-runc-v2' | wc -l +``` + +> **Warning:** Moving the `/usr/local/bin` binaries changes which shim the next container uses. Move them only when they are confirmed Docker leftovers and `/usr/bin` already matches a healthy node. New containers wait until containerd and kubelet are restarted. Rollback: `sudo mv /var/tmp/usr-local-bin-docker/ /usr/local/bin/` for each moved file. + +```bash +sudo mkdir -p /var/tmp/usr-local-bin-docker +for f in runc containerd containerd-shim containerd-shim-runc-v1 containerd-shim-runc-v2; do + if [ -e "/usr/local/bin/$f" ]; then sudo mv "/usr/local/bin/$f" /var/tmp/usr-local-bin-docker/; fi +done +hash -r +sudo systemctl restart containerd +sudo systemctl restart kubelet +``` + +Copy the healthy `/usr/bin` set again if `which` still disagrees with the healthy node. Leave `/usr/local/bin` without a `runc` unless a unit, script, or containerd setting calls that path by full name; in that case copy `/usr/bin/runc` there instead. etcd and kube-apiserver should start without deleting the WAL. + +If calico-node then restarts with a port conflict and `crictl ps -a | grep calico` shows two containers that cannot be deleted, stop kubelet, reboot the host, delete the stale calico container, and start kubelet. + +> **Warning:** Stopping kubelet and rebooting stops Pods on this node until the host is back and kubelet is started. Only when the duplicate calico containers remain after the binary restore and calico-node is in a port conflict. Rollback of the kubelet stop before the reboot: `sudo systemctl start kubelet`. + +```bash +sudo systemctl stop kubelet +sudo shutdown -r now +# after the host is back +sudo crictl ps -a | grep calico +sudo crictl rm +sudo systemctl start kubelet +``` + +### Cause 4: After Docker is uninstalled the scheduler stays down until a reboot + +List the Docker packages and check whether they own the platform runtime files: + +```bash +rpm -qa | grep docker || true +rpm -qf /usr/bin/containerd /usr/bin/runc +``` + +If a Docker package (for example `containerd.io` or `docker-ce`) owns them, removing it deletes those files; restore them from a healthy node (Cause 2) before the reboot. + +> **Warning:** Removing Docker and rebooting stops containers on this node until the host is back. Only when Docker was installed on a containerd node and the cluster command line is already dead. Reinstalling Docker is not a rollback, because Docker caused the fault. If etcd is still down after the reboot and `/usr/local/bin` holds runtime binaries, use Cause 3. + +Remove the listed Docker packages with the package manager, then reboot: + +```bash +sudo shutdown -r now +``` + +After the reboot, `kubectl get pods -n kube-system -l component=kube-scheduler` should show the scheduler Running. + +## Diagnostic Steps + +```bash +which -a runc containerd +runc --version +containerd --version +sudo journalctl -u kubelet --since "1 hour ago" | grep -F "Failed to delete cgroup paths" || true +systemctl is-active docker 2>/dev/null || true +sudo crictl ps -a | grep etcd +sudo crictl inspect | grep -E '"reason"|"exitCode"|"message"' +kubectl -n kube-system get pods -l 'component in (etcd,kube-apiserver)' -o wide +``` + +- `/usr/local/bin/runc` is `1.0.0-rc3` (or older than the healthy node's) after a kernel upgrade, static Pods in `CrashLoopBackOff`: Cause 1. +- `Failed to delete cgroup paths`, or etcd `StartError` / `lstat /proc//ns/ipc: no such file or directory` with `docker.service` running on a containerd node: Cause 2. `rpm -q docker-ce` returning a package supports it. +- `shim-run-v2` and Docker leftovers in `/usr/local/bin`: Cause 3. +- Docker installed, command line dead, scheduler down after the uninstall: Cause 4. +- `walpb: crc mismatch` with binaries that match a healthy node: a corrupt etcd member, not a binary restore. + +A node that never had Docker installed and whose `which -a runc` lists only `/usr/bin/runc` at the healthy version is not covered here. diff --git a/docs/en/solutions/etcd_quota_stays_full_from_a_large_live_key_set_Pod_bursts_or_a_database_over_2_GB_on_ACP.md b/docs/en/solutions/etcd_quota_stays_full_from_a_large_live_key_set_Pod_bursts_or_a_database_over_2_GB_on_ACP.md new file mode 100644 index 000000000..ed5b2697f --- /dev/null +++ b/docs/en/solutions/etcd_quota_stays_full_from_a_large_live_key_set_Pod_bursts_or_a_database_over_2_GB_on_ACP.md @@ -0,0 +1,156 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "2.6" + - "3.16" + - "4.1" + - "4.2" + - "4.3" +--- + +# etcd quota stays full from a large live key set, Pod bursts, or a database over 2 GB on ACP + +## Issue + +- etcd DB SIZE stays above about 1 GB after compaction, close to the default 2 GiB quota, and the key count is large (for example tens of thousands of ReplicaSets). +- A workload creates and deletes many Pods in bursts; during a load test etcd returns `Error from server : etcdserver: mvcc:database space exceeded` after about 1,000 Pods, every Pod create and delete fails, and deleted Pods stay in `Terminated`. +- kube-apiserver (for example on a standby `global` cluster) restarts again and again, its log shows `etcdserver: leader changed`, and every member's database is over 2 GB. + +On ACP, etcd runs as static Pods `etcd-` in `kube-system` (label `component=etcd`). The manifest is `/etc/kubernetes/manifests/etcd.yaml` and the data directory is `/var/lib/etcd`. + +Related articles: "etcd reports database space exceeded, or compaction frees only tens of megabytes on ACP" covers the snapshot, compaction, defrag, and `alarm disarm` steps used below. + +## Root Cause + +### Cause 1: DB SIZE stays above 1 GB after compaction because the live key set is large + +`endpoint status` shows DB SIZE around 1.9 GB on every member, and a prefix count shows a large live set (for example 52126 replicasets, 14579 services, 12352 secrets, 9761 pods, 7522 configmaps, 7262 deployments). Compaction drops the file to about 1.2 GB, which is still a capacity risk next to the default 2 GiB quota. A ten-minute watch shows `/registry/minions/` and `/registry/leases/` as the most frequently changing keys, so no single client loop is the source. + +ReplicaSets often dominate: about 5000 Deployments with 5 to 10 kept revisions each produce tens of thousands of ReplicaSets, because old ReplicaSets are kept up to the Deployment's `spec.revisionHistoryLimit` (Kubernetes default 10). Other large prefixes can be `rolebindings`, `aquasecurity.github.io` objects, `cert-manager.io`, and `events`. + +kube-apiserver compacts etcd about every 5 minutes. If kube-apiserver has been restarting continuously, that compaction does not run and revisions accumulate. + +Once compaction has done what it can, raise `quota-backend-bytes` to 8 GiB (`8589934592`) and keep that value permanently. + +### Cause 2: Pod create and delete bursts fill the quota + +A workload starts many Pods per task and deletes them when the task ends. On a v3.16.2 cluster (etcd 3.5.15), `endpoint status` can show DB SIZE 4.6 GB and `du -sh /var/lib/etcd/` 4.8G while a key count under `/registry` finds only about 2700 keys in total. A file that large over so few live keys is revision history from the Pod churn, not live data. The temporary relief is `--auto-compaction-retention=2h` together with `--quota-backend-bytes=8589934592`, followed by compaction and defrag; restore the previous parameters afterwards. + +### Cause 3: kube-apiserver restarts on etcdserver: leader changed while every member's database is over 2 GB + +kube-apiserver restarts frequently and its log before each restart shows etcd leader changes. The etcd log repeats `"msg":"apply request took too long"` with `"expected-duration":"100ms"`; read-only range requests on `/registry/events/` (`limit:500`) take 100 to 236 ms, some requests take 1 to 3 seconds (`"took":"3.427755923s"`) and end with `"error":"etcdserver: leader changed"`, and `/readyz` fails with `[-]linearizable_read failed: etcdserver: leader changed` and `"status-code":503`. The three members' databases are 2.3G, 2.4G, and 2.5G. After an etcd compaction, kube-apiserver stops restarting. + +## Resolution + +Run the diagnostic commands first and apply only the cause whose signature you confirmed. Take a snapshot from a healthy member before any change, as described in the related article. The helper below runs `etcdctl` inside an etcd Pod with the ACP certificate paths: + +```bash +etcdctl_on() { kubectl -n kube-system exec "etcd-$1" -- etcdctl \ + --endpoints=https://127.0.0.1:2379 \ + --cacert=/etc/kubernetes/pki/etcd/ca.crt \ + --cert=/etc/kubernetes/pki/etcd/server.crt \ + --key=/etc/kubernetes/pki/etcd/server.key "${@:2}"; } +etcdctl_on snapshot save /var/lib/etcd/etcd-before-change.db +``` + +Copy the snapshot from `/var/lib/etcd` on that node to `/var/tmp` before you continue. + +### Cause 1: DB SIZE stays above 1 GB after compaction because the live key set is large + +Compact first (compact to the current revision, then defrag each member, leader last; no `alarm disarm` unless `NOSPACE` is set). Confirm kube-apiserver is not crash-looping, otherwise the periodic compaction stalls again: + +```bash +kubectl -n kube-system get pods -l component=kube-apiserver -o wide +``` + +Then raise the quota only when DB SIZE is still above about 1 GB after a successful compaction, the prefix count shows a large live set, and the filesystem that holds `/var/lib/etcd` has room for an 8 GiB file. Set it where etcd's arguments come from on this cluster. + +When the control plane is generated from a `kubeadmConfigSpec`, set the value there, as a quoted string: + +```yaml +kubeadmConfigSpec: + clusterConfiguration: + etcd: + local: + extraArgs: + quota-backend-bytes: "8589934592" +``` + +When kubelet runs etcd from `/etc/kubernetes/manifests/etcd.yaml`, add the flag next to the other etcd arguments on each master, one master at a time. If both the manifest and `kubeadmConfigSpec` exist, set the same value in both so they do not disagree. + +```yaml +# in the etcd container command list of /etc/kubernetes/manifests/etcd.yaml +- --quota-backend-bytes=8589934592 +``` + +> **Warning:** Raising the quota does not shrink etcd; it lets the member grow up to 8 GiB. Saving `etcd.yaml`, or applying the kubeadm extra argument, restarts that etcd member. One master at a time, after compaction, in a maintenance window, and only with free space for an 8 GiB `snap/db`. Rollback: set the previous value again only after DB SIZE is under that value. Without the flag, etcd uses 2147483648 (2 GiB); putting that back while the file is larger turns `NOSPACE` on again. + +After the Pod is recreated, its arguments must contain `--quota-backend-bytes=8589934592`: + +```bash +kubectl -n kube-system get pod etcd- -o yaml | grep quota-backend-bytes +``` + +To reduce the number of ReplicaSets, lower `spec.revisionHistoryLimit` on Deployments that do not need many rollback revisions. + +### Cause 2: Pod create and delete bursts fill the quota + +Raise the quota as in Cause 1 so that writes can resume, and add `--auto-compaction-retention=2h` temporarily. + +> **Warning:** Saving `etcd.yaml` restarts this member. One master at a time, after a snapshot, and only while `mvcc: database space exceeded` is confirmed and Pod churn is the source. A `2h` retention discards revisions faster than before, so clients with old resourceVersions see compaction errors sooner. Rollback: remove `--auto-compaction-retention=2h` once DB SIZE is back under the quota you will keep, and save again. + +```yaml +# temporary, next to the other etcd arguments in /etc/kubernetes/manifests/etcd.yaml +- --auto-compaction-retention=2h +- --quota-backend-bytes=8589934592 +``` + +When the file is under the new quota, run `alarm disarm`, then compact and defrag each member. Restore the previous parameters when DB SIZE allows the previous quota. + +### Cause 3: kube-apiserver restarts on etcdserver: leader changed while every member's database is over 2 GB + +Apply only when `endpoint status` shows every member above about 2 GB, kube-apiserver's pre-restart log shows `etcdserver: leader changed`, and the etcd log has `apply request took too long` on large range reads: + +```bash +kubectl -n kube-system logs etcd- --tail=500 | grep -E 'changed leader|leader changed|linearizable_read' || true +``` + +Take the snapshot, compact to the current revision (no `alarm disarm` unless `NOSPACE` is set), and defrag each member, leader last. kube-apiserver stops restarting after the compaction. Leader changes while the database is small are not fixed by compaction; check etcd disk latency and packet loss between the masters instead. + +## Diagnostic Steps + +```bash +kubectl -n kube-system get pods -l component=etcd -o wide +etcdctl_on endpoint status -w table +etcdctl_on alarm list +``` + +- A large live prefix count, usually `replicasets`: Cause 1. +- A load test that creates and deletes Pods until `database space exceeded`, with a multi-gigabyte file over a few thousand keys: Cause 2. +- Every member above 2 GB while kube-apiserver restarts on `leader changed`: Cause 3. +- `NOSPACE` without one of these patterns, or a DB SIZE that barely moved after compaction: see the related article. + +Count keys by prefix: + +```bash +etcdctl_on get / --prefix --keys-only \ + | awk -F '/' '{ h[$3]++ } END {for (k in h) print h[k], k}' | sort -nr | head -30 +``` + +Watch which keys change for ten minutes, to rule out a single client writing in a loop: + +```bash +kubectl -n kube-system exec etcd- -- sh -c \ + 'export ETCDCTL_API=3; timeout 600 etcdctl \ + --endpoints=https://127.0.0.1:2379 \ + --cacert=/etc/kubernetes/pki/etcd/ca.crt \ + --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \ + --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \ + watch --prefix / -w fields' > /tmp/etcd-watch.txt +grep -w Key /tmp/etcd-watch.txt | awk '{print $NF}' | tr -d '"' | sort | uniq -c | sort -nr | head -20 +``` + +Lease and node-status keys at the top (for example about 300 changes per lease and 120 per `/registry/minions/` in ten minutes) are normal heartbeat traffic. diff --git a/docs/en/solutions/etcd_reports_database_space_exceeded_or_compaction_frees_only_tens_of_megabytes_on_ACP.md b/docs/en/solutions/etcd_reports_database_space_exceeded_or_compaction_frees_only_tens_of_megabytes_on_ACP.md new file mode 100644 index 000000000..090134142 --- /dev/null +++ b/docs/en/solutions/etcd_reports_database_space_exceeded_or_compaction_frees_only_tens_of_megabytes_on_ACP.md @@ -0,0 +1,146 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "2.6" + - "3.16" + - "4.1" + - "4.2" + - "4.3" +--- + +# etcd reports database space exceeded, or compaction frees only tens of megabytes on ACP + +## Issue + +The cluster reports `etcdserver: no space`, `Error from server : etcdserver: mvcc:database space exceeded`, or `etcdserver: mvcc: database space exceeded`. Pods cannot be created or deleted and stay in `Terminated`; `kubectl` writes fail. The platform raises an etcd db size alarm, or an inspection reports the etcd database over its threshold (for example about 800 MB, 1 GB, 1.4 GB, or 1.5 GB). Sometimes compaction has already been run and the file on disk barely shrinks. + +On ACP, etcd runs as static Pods `etcd-` in `kube-system` (label `component=etcd`). The manifest is `/etc/kubernetes/manifests/etcd.yaml` and the data directory is `/var/lib/etcd`. + +Related articles: "etcd quota stays full from a large live key set, Pod bursts, or a database over 2 GB on ACP" covers a database that stays large after compaction. + +## Root Cause + +### Cause 1: etcdserver: mvcc: database space exceeded + +The database file reached the backend quota and etcd raised the `NOSPACE` alarm. `etcdctl alarm list` shows `NOSPACE`, and in `endpoint status -w table` the ERRORS column reads `memberID: alarm:NOSPACE` on every member, even when only one member's file is large (for example 2.2 GB on a follower and 547 MB on the others). While the alarm is set, etcd rejects writes, so object deletes fail as well as creates. + +The repair is `etcdctl compact`, then `defrag` on each member, then `alarm disarm`. Compaction drops historical revisions, defrag returns the freed pages to the filesystem, and disarm clears the alarm once the file is under the quota. After this sequence the members can drop to around 400 MB with an empty ERRORS column. + +### Cause 2: Compaction frees only tens of megabytes and snap/db stays about 1.4G + +An inspection reports the etcd database usage (`ETCD数据库使用量`) above the warning threshold, a key count does not show one runaway prefix, and compaction frees only about 50 MB. On disk, `member` is about 1.7G, `snap` about 1.4G, `wal` about 367M, and `snap/db` about 1.4G. Endpoint status is healthy, DB SIZE matches that file, and no leader change is in progress. + +The inspection series is `cluster.kube.etcd.db.size.mb.per.instance`, query `etcd_mvcc_db_total_size_in_bytes{namespace="kube-system"} / 1024 / 1024`, from inspection item `cluster-etcd-database-usage` (`inspectionitems.ait.alauda.io`). It measures the on-disk MVCC file, including pages that compaction already marked free. Compaction alone does not shrink the file; defrag rewrites `snap/db` without the free pages. + +## Resolution + +Run the diagnostic commands first and apply only the cause whose signature you confirmed. Define a helper for `etcdctl` inside an etcd Pod (certificate paths as on ACP; check them first with `ls /etc/kubernetes/pki/etcd/ca.crt /etc/kubernetes/pki/etcd/server.crt /etc/kubernetes/pki/etcd/server.key` on a master, and do not use the commands as written if the paths differ): + +```bash +etcdctl_on() { kubectl -n kube-system exec "etcd-$1" -- etcdctl \ + --endpoints=https://127.0.0.1:2379 \ + --cacert=/etc/kubernetes/pki/etcd/ca.crt \ + --cert=/etc/kubernetes/pki/etcd/server.crt \ + --key=/etc/kubernetes/pki/etcd/server.key "${@:2}"; } +``` + +### Take a snapshot first + +Take a snapshot from a healthy member before any compaction or defrag. This is also the command-line form of a manual etcd backup. Find the `hostPath` whose `volumeMounts.mountPath` is `/var/lib/etcd` in `kubectl -n kube-system get pod etcd- -o yaml`; `snapshot save` writes inside the container, and the file lands on that hostPath on the node. The etcd image usually has no `tar`, so `kubectl cp` cannot copy the file out. Leave enough free space for another full copy of the database. + +```bash +etcdctl_on snapshot save /var/lib/etcd/etcd-before-compact.db +``` + +On that node, move the file out of the data directory: + +> **Warning:** The `rm -f` deletes only `/var/lib/etcd/etcd-before-compact.db` (or the same file name on the hostPath), and only after `ls` shows a non-empty copy under `/var/tmp`. Never delete anything inside `member/`. If the copy is missing, skip the `rm` and copy again. + +```bash +sudo cp /var/lib/etcd/etcd-before-compact.db /var/tmp/etcd-before-compact.db +sudo ls -l /var/tmp/etcd-before-compact.db +sudo rm -f /var/lib/etcd/etcd-before-compact.db +``` + +When the etcd Pod cannot be exec'd, run `etcdctl` on the master itself with `ETCDCTL_API=3`, the member endpoints on port 2379, and the certificates under `/etc/kubernetes/pki/etcd/` (`peer.crt`/`peer.key` or `healthcheck-client.crt`/`healthcheck-client.key`). + +### Cause 1: etcdserver: mvcc: database space exceeded + +Apply only when `alarm list` prints `NOSPACE` or the member log contains `database space exceeded`, and the snapshot exists. + +Read the current revision. The table form does not show it on every `etcdctl` build; the JSON form prints it in the response header: + +```bash +etcdctl_on endpoint status -w table +etcdctl_on endpoint status --write-out=json | grep -o '"revision":[0-9]*' +``` + +> **Warning:** `etcdctl compact ` drops every revision older than that number; clients that still hold those resourceVersions get a compaction error. Compaction is cluster-wide and cannot be undone. Run it once, with the current revision, after the snapshot exists. A wrong revision is not a reason to wipe `/var/lib/etcd`. + +```bash +etcdctl_on compact +``` + +Defragment one member at a time, the current leader (`IS LEADER` true in `endpoint status`) last, and wait until `endpoint health` of that member is true before the next. On a file of several gigabytes add `--command-timeout=300s`; on a file of about 1 GB, `--command-timeout=120s` is enough. Each member returns `Finished defragmenting etcd member[...]`. + +> **Warning:** `defrag` blocks the member while it rewrites `snap/db`; expect an API latency spike and a possible leader change. Maintenance window, snapshot saved, compact already run, one member at a time. If a member does not come back, stop. Defrag has no data undo. A snapshot restore is a rollback only for the whole cluster (every member restored from the same file after quorum is lost); one member cannot be restored from it into a running cluster. + +```bash +etcdctl_on defrag --command-timeout=300s +``` + +> **Warning:** `alarm disarm` lifts the read-only `NOSPACE` condition for the whole cluster. Run it only when `endpoint status` DB SIZE is below the quota after compact and defrag; disarming while the file is still over quota lets the alarm return. There is no separate re-arm step. + +```bash +etcdctl_on alarm disarm +etcdctl_on alarm list +``` + +`alarm list` should be empty and the console message should clear; writes, including Pod deletes, work again. Run `endpoint status -w table` against all members afterwards. If one member answers `context deadline exceeded`, check that member's health and etcd log before closing. + +When compaction cannot bring the file under the default 2 GiB quota because the live key set is large, raise `--quota-backend-bytes` to `8589934592` first (see the related article on the etcd quota), wait for the member to come back, `alarm disarm`, compact to the current revision, then defrag each member with `--command-timeout=300s`. + +### Cause 2: Compaction frees only tens of megabytes and snap/db stays about 1.4G + +An inspection that flags the size while DB SIZE is only slightly above 500 MB needs no immediate action: keep monitoring and compact once the database approaches 1 GB. An inspection that reports the threshold reached is answered with the snapshot and compaction steps of Cause 1, without `alarm disarm` unless `NOSPACE` is set. + +When compaction has already run, DB SIZE dropped by only tens of megabytes, `snap/db` is still about 1.4G, and `endpoint status` shows no alarm and a stable leader, defrag each member, leader last. The default 5-second `etcdctl` client timeout can expire before defrag returns on a 1 GB file, so add `--command-timeout=120s` (a client wait, not an etcd setting). Changing `quota-backend-bytes` is not needed for this signature. + +> **Warning:** `defrag` rewrites this member's `snap/db` and can move the leader. Maintenance window, snapshot saved, one member at a time, and only for the "compaction freed tens of MB, file still about 1.4G" signature. If the member does not return, stop; the snapshot is for a whole-cluster restore only. + +```bash +etcdctl_on defrag --command-timeout=120s +``` + +## Diagnostic Steps + +```bash +kubectl -n kube-system get pods -l component=etcd -o wide +etcdctl_on endpoint status -w table +etcdctl_on alarm list +``` + +- `NOSPACE` in `alarm list`, or `alarm:NOSPACE` in the ERRORS column: Cause 1. +- A healthy endpoint whose DB SIZE barely moved after compact: Cause 2. +- A large live key count, usually `replicasets`, Pod create/delete bursts, or every member above 2 GB with kube-apiserver restarting on `leader changed`: see the related etcd quota article. + +Count keys by prefix to see whether one resource type dominates: + +```bash +etcdctl_on get / --prefix --keys-only \ + | awk -F '/' '{ h[$3]++ } END {for (k in h) print h[k], k}' | sort -nr | head -30 +``` + +On the node, `du -sh /var/lib/etcd/member /var/lib/etcd/member/snap /var/lib/etcd/member/wal /var/lib/etcd/member/snap/db` separates a large WAL from a large `snap/db`. + +The inspection item behind the MB threshold: + +```bash +kubectl get inspectionitems.ait.alauda.io -A | grep etcd +kubectl get inspectionitems.ait.alauda.io cluster-etcd-database-usage -o yaml | grep indicator +``` + +etcd backup files that fill `/` do not move `etcd_mvcc_db_total_size_in_bytes`; that is a disk-space problem, not this metric. diff --git a/docs/en/solutions/kube-apiserver_latency_alerts_noisy_apiserver_logs_and_controller-manager_or_scheduler_health_alerts_on_ACP.md b/docs/en/solutions/kube-apiserver_latency_alerts_noisy_apiserver_logs_and_controller-manager_or_scheduler_health_alerts_on_ACP.md new file mode 100644 index 000000000..19ad3270a --- /dev/null +++ b/docs/en/solutions/kube-apiserver_latency_alerts_noisy_apiserver_logs_and_controller-manager_or_scheduler_health_alerts_on_ACP.md @@ -0,0 +1,199 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.1" + - "4.2" + - "4.3" +--- + +# kube-apiserver latency alerts, noisy apiserver logs, and controller-manager or scheduler health alerts on ACP + +## Issue + +- Alerts say kube-apiserver write latency is over 0.50 seconds, or the read success rate is under 0.80 or 0.90, while the static Pods are Running. `kubectl` is slow only for some calls, often Pod creates. +- An apiserver error-rate alert arrives, or the apiserver log is full of `invalid bearer token` and OpenAPI download errors such as `v1.enhancement.cluster.alauda.io ... resource not found`, and a control-plane restart is being considered. +- `cluster-enhancer-apiserver` uses a lot of memory and its resources need to change. +- The monitoring page shows kube-controller-manager as down, but `telnet 127.0.0.1 10257` on the node succeeds and the Pod is healthy. On a managed cluster the platform raises `集群内KubeScheduler的健康状态 != 1 且持续 1 分钟` and `集群内ControllerManager的健康状态 != 1 且持续 1 分钟` ("KubeScheduler / ControllerManager health status in the cluster != 1 for 1 minute", policy `cpaas-cluster-rules`), while both Pods are `1/1 Running` with no restarts. +- `kubectl get componentstatuses` shows `controller-manager` and `scheduler` as `Unhealthy`. +- An inspection reports apiserver errors tied to an already-compacted etcd revision and a connection reset. + +Check admission webhooks before scaling kube-apiserver or compacting etcd. + +## Root Cause + +### Cause 1: An admission webhook has no backend (dial tcp :443: connect: connection refused) + +kube-apiserver logs `dial tcp :443: connect: connection refused` while calling a mutating admission webhook (for example `chaosblade-operator-arm64.chaosblade.svc`), and `POST pods` p99 is about 1.25 seconds. The webhook Service still exists, but its Endpoints and EndpointSlice have no addresses: the operator Deployment is `READY 0/0`. The `MutatingWebhookConfiguration` still matches Pod creates with `failurePolicy: Ignore` and `timeoutSeconds: 10`, so each create waits on the dead backend and then continues. Deleting that MutatingWebhookConfiguration stops the alert. The same applies to any webhook configuration whose Service has no endpoints. + +### Cause 2: invalid bearer token and v1.enhancement.cluster.alauda.io resource not found + +The apiserver log contains, in volume: + +- `Unable to authenticate the request` with `err="invalid bearer token"` +- `Unhandled Error` with `loading OpenAPI spec for "v1.enhancement.cluster.alauda.io" failed` and `failed to download v1.enhancement.cluster.alauda.io: resource not found` + +The first line comes from the nevermore log agent; platform components call kube-apiserver with tokens and other methods, so these lines can appear while the platform works normally. The second comes from `cluster-enhancer-apiserver`. While kube-apiserver serves normally, these lines do not affect nevermore, `cluster-enhancer-apiserver`, or kube-apiserver and can be ignored. Expanding the audit policy to identify the client is not needed when node time is consistent across clusters and the platform works. + +The ignore conclusion covers the OpenAPI line only while the aggregated API behind it is up: APIService `v1.enhancement.cluster.alauda.io` reports `Available=True` and its Service has Ready endpoints. When the APIService reports `Available=False` (for example `MissingEndpoints` or `FailedDiscoveryCheck`), requests to that API group fail, and the line is a symptom of the missing backend, for example because the backend Pods have no node left to run on. + +### Cause 3: cluster-enhancer-apiserver memory is high and a resource change needs a restart + +`cluster-enhancer-apiserver` serves the `EtcdBackupConfiguration` API, certificate monitoring, etcd backup storage, and Prometheus metrics for that API. `cluster-enhancer-controller` cleans expired cert-manager certificates, monitoring dashboard configuration, and event records. Restarting `cluster-enhancer-apiserver` for a resource change does not affect workloads that are already running. + +### Cause 4: kube-controller-manager or kube-scheduler listens only on 127.0.0.1 + +Monitoring probes kube-controller-manager on `:10257` (and kube-scheduler on `:10259`), but the process listens only on `127.0.0.1`, or a firewall change blocks the node address: + +- Firewall: `telnet 127.0.0.1 10257` connects and `telnet 10257` is refused because the planned iptables forwarding rules were applied incorrectly. The fix restores the rules as planned and aligns the Prometheus Service, ServiceMonitor, and target port with the port that answers. +- Bind address: `netstat -lntp` shows kube-controller-manager on `127.0.0.1:10257` and kube-scheduler on `127.0.0.1:10259`. Forwarding cannot make `:10257` connect until the bind address changes. A burst of `pods is forbidden: User "system:kube-scheduler" cannot list resource "pods"` lines before `successfully acquired lease kube-system/kube-scheduler` in the scheduler log is not the cause. Scheduling and control are not affected. + +### Cause 5: kubectl get componentstatuses shows controller-manager and scheduler Unhealthy + +`kubectl get componentstatuses` prints `controller-manager` and `scheduler` as `Unhealthy` with `dial tcp 127.0.0.1:10257: connect: connection refused` (and the same for `127.0.0.1:10259`), warns that `v1 ComponentStatus` is deprecated in v1.19+, and shows `etcd-0` `Healthy`. This deprecated probe output has no impact on the service and needs no remediation. + +### Cause 6: apiserver errors from an already-compacted revision, with no leader change + +The requested historical revision was already removed by etcd auto-compaction, and the connection was reset. etcd has not restarted and is not changing leader. These errors have no impact and can be ignored. A cluster that also logs `leader changed` is not covered by this conclusion. + +## Resolution + +### Cause 1: An admission webhook has no backend + +Apply only when the apiserver log names this webhook, its Service has empty endpoints, and the operator has no ready Pods: + +```bash +kubectl get mutatingwebhookconfiguration -o yaml > /tmp/.yaml +kubectl get endpoints,endpointslice -n +kubectl get deploy,pods -n +``` + +> **Warning:** Deleting a MutatingWebhookConfiguration stops that mutation for every matched resource. Do it only when the log shows `connection refused` to that webhook and its Service has no endpoints; never for a webhook that still has ready Pods. With `failurePolicy: Ignore`, creates were already allowed after the timeout, so the delete removes the wait rather than changing allow versus deny. Rollback: `kubectl apply -f /tmp/.yaml`. + +```bash +kubectl delete mutatingwebhookconfiguration +``` + +Watch `POST pods` latency afterwards, and repeat the endpoint check for every other webhook before changing etcd. + +### Cause 2: invalid bearer token and v1.enhancement.cluster.alauda.io resource not found + +Check the APIService first; `readyz` reports kube-apiserver itself and does not show whether an aggregated API has a backend: + +```bash +kubectl get apiservice v1.enhancement.cluster.alauda.io +kubectl get apiservice v1.enhancement.cluster.alauda.io -o jsonpath='{.spec.service.namespace}/{.spec.service.name}{"\n"}{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}' +kubectl get endpoints -n +kubectl get --raw='/readyz?verbose' +``` + +- `AVAILABLE True`, endpoints present, and `readyz` succeeds: no change. Do not restart the control plane, rotate tokens, or delete `cluster-enhancer-apiserver` because of these lines. +- `AVAILABLE False`, or a Service without endpoint addresses: find out why the backend Pods are missing or not Ready (for example after worker nodes were removed), and fix that: + + ```bash + kubectl get nodes -o wide + kubectl get pods -A -o wide | grep cluster-enhancer-apiserver + kubectl describe pod -n + ``` + +For `invalid bearer token`, compare the clock on every node of every cluster before closing: + +```bash +date -u '+%F %T' +timedatectl | grep -E 'synchronized|NTP' +``` + +### Cause 3: cluster-enhancer-apiserver memory is high and a resource change needs a restart + +Change the resources where the platform owns that Deployment; a hand edit of the Pod template is reverted when a controller reconciles it. The restart does not disturb running workloads. + +```bash +kubectl get pods -A -o wide | grep cluster-enhancer-apiserver +``` + +### Cause 4: kube-controller-manager or kube-scheduler listens only on 127.0.0.1 + +Confirm the split between localhost and the node IP, and save iptables before editing: + +```bash +sudo ss -lntp | grep -E ':10257|:10259' +sudo iptables-save > /tmp/iptables-before-kcm.rules +kubectl get svc,servicemonitor -A | grep -i -E 'controller-manager|scheduler' +``` + +**Firewall:** restore the forwarding rules from the approved firewall plan (the exact rules are site-specific), then point the Service and ServiceMonitor at the port that answers. + +> **Warning:** Changing filter or nat rules can black-hole node traffic unrelated to port 10257. Edit only the rules the firewall plan uses for these ports, and only after localhost works while the node IP does not. Rollback: `sudo iptables-restore < /tmp/iptables-before-kcm.rules`. + +**Bind address on a cluster reconciled from `kubeadmConfigSpec`:** set the listen address there: + +```yaml +kubeadmConfigSpec: + clusterConfiguration: + controllerManager: + extraArgs: + bind-address: "::" + scheduler: + extraArgs: + bind-address: "::" +``` + +**Bind address on a cluster whose control plane runs from static manifests:** change `--bind-address` from `127.0.0.1` to the node IP that monitoring scrapes. + +> **Warning:** Changing `bind-address` restarts kube-controller-manager and kube-scheduler on each control-plane node; edit one master at a time, in a maintenance window, and only when `ss` shows the ports bound to `127.0.0.1`. The metrics and health ports then answer on the node address, so keep the firewall plan in front of them. Keep backups outside `/etc/kubernetes/manifests/`; a copy left in that directory is started as a second static Pod. Rollback: restore the backups (or set `bind-address` back to `127.0.0.1`) and the previous ServiceMonitor. + +```bash +sudo cp /etc/kubernetes/manifests/kube-controller-manager.yaml /var/tmp/kube-controller-manager.yaml.before +sudo cp /etc/kubernetes/manifests/kube-scheduler.yaml /var/tmp/kube-scheduler.yaml.before +sudo sed -i 's/--bind-address=127.0.0.1/--bind-address=/' /etc/kubernetes/manifests/kube-controller-manager.yaml +sudo sed -i 's/--bind-address=127.0.0.1/--bind-address=/' /etc/kubernetes/manifests/kube-scheduler.yaml +kubectl get pods -n kube-system -l 'component in (kube-controller-manager,kube-scheduler)' -o wide +``` + +After the Pods are Ready, `telnet 10257` from the Prometheus node should connect and the health alerts should clear. + +### Cause 5: kubectl get componentstatuses shows controller-manager and scheduler Unhealthy + +When that output is the only complaint and the Pods are Running, no change is needed: no firewall edit, no bind-address change, and no restart. If the monitoring page is also red, use Cause 4. + +```bash +kubectl get componentstatuses +kubectl -n kube-system get pods -l 'component in (kube-controller-manager,kube-scheduler,etcd)' -o wide +``` + +### Cause 6: apiserver errors from an already-compacted revision, with no leader change + +```bash +kubectl -n kube-system get pods -l component=etcd +kubectl -n kube-system logs etcd- --tail=200 | grep -F 'leader changed' || true +``` + +With a compacted-revision error plus a connection reset, no etcd restart, and no leader changes, ignore the lines. A `NOSPACE` alarm or leader changes need the etcd quota or disk investigation instead. + +## Diagnostic Steps + +Webhooks first, then the log signatures, then the controller-manager port: + +```bash +kubectl get mutatingwebhookconfiguration -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.webhooks[*].clientConfig.service.namespace}{"/"}{.webhooks[*].clientConfig.service.name}{"\tfail="}{.webhooks[*].failurePolicy}{"\ttimeout="}{.webhooks[*].timeoutSeconds}{"\n"}{end}' +kubectl get validatingwebhookconfiguration -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' +kubectl -n kube-system logs kube-apiserver- --tail=200 | grep -E 'connection refused|invalid bearer token|enhancement.cluster.alauda.io' || true +kubectl get apiservice | grep -v 'True' || true +``` + +For each webhook Service namespace printed above: + +```bash +kubectl get endpoints,pods,deploy -n +``` + +- Empty endpoints plus `connection refused`: Cause 1. +- The two log lines with a healthy `readyz` and APIService `v1.enhancement.cluster.alauda.io` `AVAILABLE True`: Cause 2. The same OpenAPI line with that APIService `False`: start from the missing backend Pods. +- High memory on `cluster-enhancer-apiserver`: Cause 3. +- `10257` open on `127.0.0.1` and refused on the node IP, or the KubeScheduler / ControllerManager health alerts with ports bound to `127.0.0.1`: Cause 4. +- `kubectl get componentstatuses` Unhealthy with the Pods Running and no monitoring failure: Cause 5. +- Compacted-revision errors with a quiet etcd: Cause 6. + +Read-success alerts that are mostly HTTP 403 are authorization failures, not a webhook timeout. Capture one request's user and verb before deleting any webhook. diff --git a/docs/en/solutions/sentry_at_full_CPU_or_a_redeployed_component_fails_because_of_Failed_or_leftover_ResourcePatches_on_ACP.md b/docs/en/solutions/sentry_at_full_CPU_or_a_redeployed_component_fails_because_of_Failed_or_leftover_ResourcePatches_on_ACP.md new file mode 100644 index 000000000..4a1fbe8ae --- /dev/null +++ b/docs/en/solutions/sentry_at_full_CPU_or_a_redeployed_component_fails_because_of_Failed_or_leftover_ResourcePatches_on_ACP.md @@ -0,0 +1,140 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# sentry at full CPU or a redeployed component fails because of Failed or leftover ResourcePatches on ACP + +## Issue + +- One of the two `sentry` Pods is almost idle and the other runs at about 100% of its CPU limit (around 2000m). The spikes can come and go within minutes. +- The platform cert-manager component keeps upgrading and never finishes. +- After ClickHouse was installed no ClickHouse instance appears, and redeploying the ClickHouse plugin fails. +- `olm-registry-custom`, `olm-registry-platform`, and `olm-registry-system` are all `0/1` `CrashLoopBackOff` a few minutes after they were created. + +`sentry` (Deployment `sentry` in `cpaas-system` on each cluster) installs, upgrades, and removes plugin charts and records the result on the `ars` (AppRelease) objects. A ResourcePatch (`rpch`) records a change made to a platform-managed resource; `sentry` applies it on top of the chart every time the release is reconciled. A ResourcePatch is not removed when its plugin is uninstalled, and it is applied again when the plugin is reinstalled. + +Related articles: "Plugin install or uninstall stuck, or Marketplace and OperatorHub empty: sentry OOMKilled or cluster-transformer errors". + +## Root Cause + +### Cause 1: One sentry Pod at 100% CPU because Failed ResourcePatches are retried in a tight loop + +The busy `sentry` Pod logs `ResourcePatch Reconcile` every few seconds for the same objects, with errors such as: + +```text +Apply ResourcePatch rp-4mcvk for apps/v1.Deployment cert-manager/cert-manager-cainjector in Release cpaas-system/cert-manager error: error in add for path: '/spec/template/spec/containers/1': Unable to access invalid index: 1: invalid index referenced +``` + +The idle Pod writes no log lines. Three ResourcePatches are in `Failed`, targeting release `cpaas-system/cert-manager` and the `cert-manager-cainjector`, `cert-manager`, and `cert-manager-webhook` Deployments. Each tries to patch `/spec/template/spec/containers/1`, a second container that does not exist in those Deployments, so `sentry` retries them continuously. Deleting those ResourcePatches brings the busy Pod down from about 2000m to about 456m CPU. + +### Cause 2: cert-manager keeps upgrading because the platform cert-manager was replaced with community images + +The cert-manager component stays in upgrading. `kubectl get rpch -A | grep cert-manager` shows the ResourcePatches for the `cert-manager-cainjector`, `cert-manager-webhook`, and `cert-manager` Deployments and the `cert-manager-webhook` and `cert-manager` Services in `Failed` (the RBAC and ServiceAccount ones `Active`), and the `cert-manager-controller` container runs `jetstack/cert-manager-controller:v1.16.1` with `--leader-election-namespace=kube-system` and `--max-concurrent-challenges=60`. + +The cert-manager deployment script of the Kubeflow 1.10 community release, run when Kubeflow was deployed, replaced the platform cert-manager images with community images. That conflicts with the platform component, so its upgrade stays stuck. Restore every cert-manager component to the platform's own images and do not modify platform components. Restoring the images does not affect platform certificates. The Kubeflow certificates issued by the community controller (in `ai-platform`, `kserve`, and `kubeflow`, some managed by Argo CD) stay, but the platform cert-manager manages only platform certificates and will not renew them; the team that deployed Kubeflow has to plan their renewal. + +### Cause 3: A leftover ResourcePatch breaks a redeployed component + +A component that was removed and deployed again fails, and a ResourcePatch for it from the earlier deployment is still in the cluster. + +- ClickHouse: the ClickHouse plugin deployment fails because of a leftover ClickHouse ResourcePatch; after it is deleted, the plugin deploys normally. Separately, the ClickHouse operator (for example `clickhouse-operator.v4.2.1` in `cpaas-logging`, `Succeeded`) does not create an instance by itself; the instance has to be created manually. +- OLM: all three `olm-registry-*` Pods are `0/1` `CrashLoopBackOff` with the same restart count, and ResourcePatches related to them exist. After the patches that are no longer needed are deleted, the OLM components run normally. + +## Resolution + +### Cause 1: One sentry Pod at 100% CPU because Failed ResourcePatches are retried in a tight loop + +```bash +kubectl -n cpaas-system top pod | grep sentry +kubectl -n cpaas-system logs --tail=300 | grep -E 'ResourcePatch|invalid index' +kubectl get rpch -A | grep -i failed +``` + +Back up every ResourcePatch the log names: + +```bash +mkdir -p "$HOME/rpch-backup" +for rp in ; do kubectl get rpch "$rp" -o yaml > "$HOME/rpch-backup/$rp.yaml"; done +``` + +> **Warning:** Deleting a ResourcePatch drops the customization it carries from that component. A `Failed` patch is not being applied, so nothing that currently runs changes. Precondition: only for ResourcePatches in `Failed` whose `invalid index referenced` errors repeat in the `sentry` log, with the backups written and the owner of the customization informed. Rollback: `kubectl apply -f "$HOME/rpch-backup/.yaml"` (remove `metadata.resourceVersion` and `metadata.uid` first). + +```bash +kubectl delete rpch +kubectl -n cpaas-system top pod | grep sentry +``` + +If `kubectl get rpch -A` shows a namespace for these objects, add `-n ` to the `get` and `delete` commands. + +When the spikes are short and gone before you can look, check for the same retry pattern after the fact: `Failed` ResourcePatches, and the same object names repeating in the `sentry` log. Raising the `sentry` CPU limit does not stop a retry loop. + +### Cause 2: cert-manager keeps upgrading because the platform cert-manager was replaced with community images + +Confirm the community images and record which certificates depend on them: + +```bash +kubectl get ars -A | grep -i cert +kubectl get rpch -A | grep cert-manager +kubectl get deploy -A | grep cert-manager +kubectl -n get deploy -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.template.spec.containers[*].image}{"\n"}{end}' +kubectl get certs -A +kubectl -n get deploy -o yaml > "$HOME/cert-manager-deploy.before.yaml" +``` + +Images such as `jetstack/cert-manager-controller:` instead of images from the platform registry are this cause. Get the platform images and container settings for your platform version from a cluster on the same version where cert-manager is healthy (same command), or from Alauda support. + +> **Warning:** Restoring the platform cert-manager images changes the controller that issues certificates; the Kubeflow certificates created by the community controller stay but are no longer renewed by the platform cert-manager. Precondition: only when the cert-manager Deployments run community images after a Kubeflow community deployment script and the cert-manager upgrade is stuck; agree with the team that owns Kubeflow how its certificates will be renewed first. Rollback: `kubectl apply -f "$HOME/cert-manager-deploy.before.yaml"`, which returns the conflicting state. + +```bash +kubectl -n set image deploy/ =/: +kubectl get ars -A | grep -i cert +``` + +Repeat for each cert-manager Deployment, and bring the container arguments back to the platform values if they differ (the community controller runs with `--leader-election-namespace=kube-system`). Do not run third-party scripts that modify platform components. + +### Cause 3: A leftover ResourcePatch breaks a redeployed component + +Find the ResourcePatches for the component, back them up, and confirm with their owner that they are no longer needed: + +```bash +kubectl get rpch -A +kubectl get rpch -A -o yaml | grep -n -i -E '|release:' +mkdir -p "$HOME/rpch-backup" +kubectl get rpch -o yaml > "$HOME/rpch-backup/.yaml" +``` + +> **Warning:** Deleting a ResourcePatch removes the customization it would apply to the component when it is deployed again. Precondition: only when the redeployed component (the ClickHouse plugin, or all `olm-registry-*` Pods in `CrashLoopBackOff`) fails, the ResourcePatch belongs to that component's earlier deployment, its owner confirmed it is not needed, and the backup is written. Rollback: `kubectl apply -f "$HOME/rpch-backup/.yaml"`. + +```bash +kubectl delete rpch +``` + +Redeploy the ClickHouse plugin, or watch the OLM registry Pods recover (`kubectl -n cpaas-system get pods -w | grep olm-registry`). For ClickHouse, create the ClickHouse instance manually after the plugin is deployed: + +```bash +kubectl get csv -n cpaas-logging | grep -i clickhouse +``` + +## Diagnostic Steps + +```bash +kubectl -n cpaas-system get pods | grep sentry +kubectl -n cpaas-system top pod | grep sentry +kubectl get rpch -A +kubectl get rpch -A | grep -i failed +kubectl -n cpaas-system logs --tail=300 | grep -E 'ResourcePatch|invalid index' +``` + +- One `sentry` Pod near its CPU limit logging `ResourcePatch Reconcile` with `invalid index referenced`, and `Failed` ResourcePatches: Cause 1. +- A stuck cert-manager upgrade with `jetstack/...` images: Cause 2. +- A redeployed ClickHouse plugin or OLM registry failing while an old ResourcePatch for it exists: Cause 3. +- `sentry` restarting with `OOMKilled`: see the related article on stuck installs and uninstalls. diff --git a/docs/en/solutions/violet_push_fails_to_parse_a_package_or_a_new_operator_version_is_skipped_or_missing_on_ACP.md b/docs/en/solutions/violet_push_fails_to_parse_a_package_or_a_new_operator_version_is_skipped_or_missing_on_ACP.md new file mode 100644 index 000000000..32b63907f --- /dev/null +++ b/docs/en/solutions/violet_push_fails_to_parse_a_package_or_a_new_operator_version_is_skipped_or_missing_on_ACP.md @@ -0,0 +1,106 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "4.0" + - "4.2" + - "4.3" +--- + +# violet push fails to parse a package, or a new operator version is skipped or missing on ACP + +## Issue + +- A push fails immediately with `failed to parse package and manifest for package .tgz: unexpected EOF`. +- GPU Operator and SR-IOV Network uploads fail while every other required plugin from the same set (for example ACP 4.3.2) uploads. +- A new operator version was pushed (for example Tekton 4.14, or `tektoncd-operator` 4.15.1), but the page and `PackageManifests` still show only the old version, and `violet` printed `WARN artifactversion operatorhub-.v already exist, skip it`. +- An operator package synchronized from `global` is visible as a software package but not in OperatorHub on a business cluster. + +Related articles: "violet push succeeds but the plugin is not published: Artifact, ModuleConfig, or chart missing on ACP" covers successful pushes whose objects never appear. A pushed Helm chart that reports `artifact already exist, will create artifactversion ... type=chart` is covered in "Pushed Helm chart version not shown because violet reused an Artifact of another chart repository on ACP". + +## Root Cause + +### Cause 1: failed to parse package and manifest, unexpected EOF + +`violet` cannot read the archive (for example `workbench-operator.stable.amd64.v0.2.1.tgz`): the archive was corrupted during upload. Matching MD5 sums across several downloads do not rule this out. Uploading the package again clears the error. + +### Cause 2: GPU Operator or SR-IOV package is missing required fields + +On ACP 4.3.2, the GPU Operator and SR-IOV Network marketplace packages were missing required fields that belong in `moduleplugin.yaml`; fetching the marketplace package again leaves the same gap. `gpu-operator` was re-released, and version `v26.3.2-default.1.gcdaaba55` is the build to select. The `sriov-network-plugin` package was still in testing and not released for production use; it is not the production plugin, and its official release has to be awaited. + +### Cause 3: New version pushed, ArtifactVersion is Failed or skipped as already existing + +`violet` prints `WARN artifactversion operatorhub-.v already exist, skip it`, and the new version is missing from the page and from `kubectl get packagemanifest`. The ArtifactVersion shows `phase=Failed`, `reason=Error`, with a bundle pull error such as `failed to do request: Head "http:///v2//-bundle/manifests/": dial tcp :80: connect: no route to host` or `... not found`. + +An earlier push left the ArtifactVersion in the cluster, but the OLM registry could not pull the bundle image it points to; the `olm-registry-platform` log repeats `failed to add bundle` for it. A plain re-push sees the existing ArtifactVersion and skips it, and restarting `olm`, `packageserver`, and `catalog-operator` does not help. Pushing again with `violet push --force` overwrites the existing objects, and the new version appears. A version that does not show up after release, with no error text, is recovered the same way. An operator package synchronized from `global` that is missing from OperatorHub on a business cluster is restored by uploading the package again; no underlying cause is known for that form. + +## Resolution + +### Cause 1: failed to parse package and manifest, unexpected EOF + +Test the file on disk before another push; `tar` reporting an unexpected end of archive means the file is not a complete package: + +```bash +tar -tzf .tgz >/dev/null +``` + +Obtain the package again from the distribution source and push that copy: + +```bash +violet push --platform-address= --platform-username= --platform-password='' .tgz +``` + +`violet` should report a successful push and no `unexpected EOF`. + +### Cause 2: GPU Operator or SR-IOV package is missing required fields + +Select the re-released GPU Operator build `v26.3.2-default.1.gcdaaba55`. For SR-IOV Network, wait for the officially released plugin; the testing package is not for production use. + +```bash +kubectl get moduleplugins | grep -iE 'gpu|sriov' +``` + +### Cause 3: New version pushed, ArtifactVersion is Failed or skipped as already existing + +On the cluster that should offer the version, read the ArtifactVersion phase and message: + +```bash +kubectl get packagemanifest,artifact,artifactversion -A | grep +kubectl -n cpaas-system get artifactversion operatorhub-.v -o yaml +kubectl -n cpaas-system logs deploy/olm-registry-platform --tail=300 | grep -i 'failed to add bundle' +``` + +> **Warning:** `--force` overwrites the package, Artifact, and ArtifactVersion already on the platform for this package version and architecture; any earlier content stored under the same name is replaced. Precondition: only when the version is missing from `PackageManifests`, and either `violet` printed `already exist, skip it` for that ArtifactVersion or the ArtifactVersion is `Failed` with a bundle pull error. Rollback: there is no undo of the overwrite; push the previous tarball again with `--force` if you need the old content back. + +```bash +violet push --platform-address= --platform-username= --platform-password='' --clusters --force .tgz +``` + +Check that the version is listed: + +```bash +kubectl get packagemanifest -A | grep +kubectl -n cpaas-system get artifactversion | grep +``` + +If the ArtifactVersion still fails with `no route to host`, the cluster cannot reach the registry host in that message. Fix that network path or push to a registry the cluster can reach; another `--force` does not change the address. + +For an operator package that is visible as a software package but missing from OperatorHub on a business cluster, upload the package again for that cluster. + +## Diagnostic Steps + +```bash +violet version +tar -tzf .tgz >/dev/null +kubectl get moduleplugins | grep +kubectl -n cpaas-system get artifactversion | grep +``` + +- `unexpected EOF` from `violet` or from `tar`: Cause 1. +- A GPU or SR-IOV upload error that names missing fields, while other plugins succeed: Cause 2. +- An ArtifactVersion in `Failed` with a bundle pull error, or `artifactversion operatorhub-... already exist, skip it`: Cause 3. +- The same `already exist` message with `type=chart`: see the related Helm chart article. +- No `artifact` / `artifactversion` at all after a successful push: see the related article on plugins that are not published. diff --git a/docs/en/solutions/violet_push_succeeds_but_the_plugin_is_not_published_Artifact_ModuleConfig_or_chart_missing_on_ACP.md b/docs/en/solutions/violet_push_succeeds_but_the_plugin_is_not_published_Artifact_ModuleConfig_or_chart_missing_on_ACP.md new file mode 100644 index 000000000..d32a2aee6 --- /dev/null +++ b/docs/en/solutions/violet_push_succeeds_but_the_plugin_is_not_published_Artifact_ModuleConfig_or_chart_missing_on_ACP.md @@ -0,0 +1,169 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "4.0" + - "4.2" + - "4.3" +--- + +# violet push succeeds but the plugin is not published: Artifact, ModuleConfig, or chart missing on ACP + +## Issue + +`violet push` finishes without an error, yet the plugin cannot be used: + +- Cluster Plugins shows the package as not published (`未上架`), or the Publish Software Package page lists it with no version. OperatorHub can show the same package as not onboarded. +- The uploaded packages menu lists the package with the right upload time while Cluster Plugins shows nothing (for example Alauda Build of NPU Operator and Alauda Build of Node Feature Discovery). +- The marketplace keeps showing the old version after a new one was pushed (for example `hyperflux` 1.4.2 on ACP 4.2.3). +- `kubectl get moduleplugins` lists the plugin, but `kubectl get moduleconfigs` returns nothing for it and the page never offers it (for example Machine Configuration). +- An ARM plugin (for example the DCS provider on ACP 4.2.4) shows `State: Absent` and no version in Cluster Plugins, with the banner "This plugin has not been uploaded to the current cluster", and cannot be deployed. +- The upload command for the HAMi plugin completed, the platform registry is Harbor, and the target cluster has no HAMi plugin; Harbor instances in other regions also lack its charts. + +Related articles: "violet push fails to parse a package, or a new operator version is skipped or missing on ACP" covers push errors and missing versions. "Pushed package not listed where expected: global-only plugins, cluster affinity, catalog source, or plugin type on ACP" covers packages that exist but are shown only on `global`, only in another catalog, or without a status by design. + +## Root Cause + +### Cause 1: Artifact and ArtifactVersion were never created because cluster-transformer stopped processing the ModulePlugin + +`violet push` reports success, the module-plugin object (`mplugin`) exists, often with no `status` field at all, while `artifact` and `artifactversion` for the package do not. `cluster-transformer` has not written those objects, so the plugin page has nothing to render. The upload itself succeeded; restarting Deployment `cluster-transformer` in `cpaas-system` makes the objects appear. Pushing the same tarball again, restarting the OLM catalog Pods, `catalog-operator`, or `marketplace-apiserver`, or re-pushing with a newer `violet` does not help. Why `cluster-transformer` stops processing the ModulePlugin is not known. + +Variants of the same signature: + +- `violet push` prints `already exists, skip push` for the images and `moduleplugin has been updated successfully cluster=global type=plugin`; `kubectl get artifact -A | grep ` returns nothing; the `marketplace-apiserver` log only shows `"OnModulePluginChanged events" logger="storage-provider" eventType="ADDED" name=""`. +- A ModuleConfig exists without a `status`, and none of the `cluster-transformer`, catalog, OLM, or ModuleInfo logs mention the plugin (for example `asm-global` 4.3.4, ARM build, on ACP 4.3.1). +- Only the `moduleplugin` exists for a new version (for example `hyperflux` 1.4.2 pushed with `violet` 4.2.11). Deleting the old version from the uploaded packages page leaves no artifacts at all; restarting `cluster-transformer` and then pushing the plugin again makes the new version display. +- After a package was first pushed to a control-plane node address by mistake and then to the platform access address, the UI still shows it as not uploaded although `apollo`, `cluster-transformer`, and `cluster-transformer-apiservice` are `1/1 Running`. Restarting those Pods brings the plugins to listed (`已上架`). + +### Cause 2: ModulePlugin exists but no ModuleConfig was generated + +`kubectl get moduleplugins` returns the plugin with the pushed chart version (for example `machine-config` `v4.0.18`, chart `ait/chart-machine-config`), `kubectl get moduleconfigs` returns nothing for it, and a repeated `violet push ... --clusters=global --force` changes nothing. This is the same `cluster-transformer` stall as Cause 1, one object later. Restarting `cluster-transformer` produces the ModuleConfig and the plugin can be installed. + +### Cause 3: sentry cannot pull the plugin chart because the registry's anonymous access was removed + +`violet` reports success (for example `found image .../ait/chart-cluster-api-provider-kubeadm:v1.0.8 already exists, skip push` and `moduleplugin cluster-api-provider-kubeadm has been created successfully cluster=global type=plugin`), but Marketplace > Cluster Plugins on `global` shows the plugin with `State: Absent`, an empty Version, and the banner "This plugin has not been uploaded to the current cluster. To install it, please complete the upload process first." No `Artifact`, `ArtifactVersion`, or PackageManifest exists, and the ModulePlugin YAML carries an error that `sentry` failed to pull the plugin chart. + +At the same time the `catalog-operator` log can show `failed to list bundles: rpc error: code = Unavailable ... dial tcp :50051: connect: connection refused`, `failed to populate resolver cache from source platform/cpaas-system`, and `ResolutionFailed` / `CatalogSourcesUnhealthy`; restarting the OLM catalog Pod does not change the plugin state. The ModulePlugin error is the one that decides this cause. + +The anonymous user was deleted from the platform image registry (and the registry credentials were changed), so `sentry` can no longer pull the chart and the plugin never moves from not listed to listed. Re-downloading and re-pushing the package does not help. Adding the anonymous user back to the registry restores the chart pull. + +### Cause 4: The Harbor registry has no mlops project, so the HAMi images and charts were never stored + +The HAMi and DCGM exporter images and charts are pushed under the `mlops` repository path. AI components are not part of the standard product, so Harbor does not contain the `mlops` project by default, and without it the push of those artifacts fails even though the command reports success. After the `mlops` project is created and the package is pushed again, the Harbor of the `global` cluster's region holds `mlops/dcgm-exporter`, `mlops/chart-dcgm-exporter`, `mlops/hami-webui-fe-oss`, `mlops/hami-webui-be-oss`, `mlops/chart-hami-webui`, `mlops/hami`, and `mlops/chart-hami`. When several regions each have their own Harbor, the other Harbor instances still lack them: push once against each Harbor (creating `mlops` wherever it is missing), or use Harbor replication. + +## Resolution + +### Cause 1: Artifact and ArtifactVersion were never created because cluster-transformer stopped processing the ModulePlugin + +Confirm the push reported success, `mplugin` exists, and `artifact` / `artifactversion` for the package do not. Skip this restart when the plugin is global-only and the push targeted another cluster. + +```bash +kubectl get mplugin -A +kubectl get artifact,artifactversion -A | grep -i +kubectl get mplugin -o jsonpath='{.status}{"\n"}' +kubectl -n cpaas-system get deploy cluster-transformer +``` + +> **Warning:** Restarting `cluster-transformer` pauses plugin synchronization until the new Pod is Ready; in-flight plugin installs wait. Precondition: only when `violet push` succeeded, `mplugin` for the package exists, and `artifact` / `artifactversion` for it are missing (or the publish page shows the package with no version). Rollback: none needed. If the page is still empty after the new Pod is Ready, collect the controller log and the `mplugin` YAML before repeating the restart. + +```bash +kubectl -n cpaas-system rollout restart deployment/cluster-transformer +kubectl -n cpaas-system rollout status deployment/cluster-transformer +kubectl get artifact,artifactversion -A | grep -i +``` + +The plugin page should show the package as published, with a version. If a new version still does not show after the restart, push the same package once more: + +```bash +violet push --platform-address= --platform-username= --platform-password='' .tgz +``` + +If the UI still shows the plugin as not uploaded while `artifact` / `artifactversion` now exist, restart `apollo` and `cluster-transformer-apiservice` as well. + +> **Warning:** Restarting `apollo` briefly interrupts the console views it serves, and restarting `cluster-transformer-apiservice` briefly interrupts the plugin API. Precondition: only after the `cluster-transformer` restart, when `artifact` / `artifactversion` exist and the UI still shows the plugin as not uploaded. Rollback: none needed; the Deployments recreate the Pods. + +```bash +kubectl -n cpaas-system get pods | grep -E 'apollo|transformer' +kubectl -n cpaas-system delete pod +``` + +If `artifact` is still missing after the restart, check the package (see the related article on parse errors) or the chart pull (Cause 3) instead of restarting again. + +### Cause 2: ModulePlugin exists but no ModuleConfig was generated + +Run on `global`: + +```bash +kubectl get moduleplugins -o yaml +kubectl get moduleconfigs | grep +``` + +When the ModulePlugin shows the chart version you pushed and no ModuleConfig exists for it, restart `cluster-transformer` with the Warning and commands in Cause 1 (the precondition extends to this signature). Then check again: + +```bash +kubectl get moduleconfigs | grep +``` + +The ModuleConfig appears and the plugin becomes installable from Cluster Plugins. + +### Cause 3: sentry cannot pull the plugin chart because the registry's anonymous access was removed + +Read the ModulePlugin and the `sentry` log for the chart pull error: + +```bash +kubectl get moduleplugin -o yaml | grep -n -i -B2 -A4 'pull\|error' +kubectl -n cpaas-system logs --tail=300 | grep -i '' +kubectl -n cpaas-system logs deploy/catalog-operator --tail=200 | grep -E 'failed to list bundles|resolver cache|CatalogSourcesUnhealthy' +``` + +Check whether the platform image registry still accepts anonymous access. A request without credentials that used to succeed and now returns `401` means anonymous access is gone: + +```bash +curl -k -s -o /dev/null -w '%{http_code}\n' https:///v2//tags/list +``` + +Restore the anonymous user on the platform image registry, configured as before the change (compare with a healthy environment of the same version). If the registry credentials were also changed, make sure the credentials the platform uses for the registry match the new ones. Then check that the ModulePlugin error clears and that `artifact` / `artifactversion` appear: + +```bash +kubectl get artifact,artifactversion -A | grep -i '' +``` + +### Cause 4: The Harbor registry has no mlops project, so the HAMi images and charts were never stored + +Check whether the project exists in each Harbor the clusters pull from: + +```bash +curl -k -s -u ':' "https:///api/v2.0/projects?name=mlops" +curl -k -s -u ':' "https:///api/v2.0/projects/mlops/repositories?page_size=50" | grep -o '"name":"[^"]*"' +``` + +An empty project list is this cause. Create the `mlops` project in that Harbor (Harbor UI > Projects > New Project, or the API below), then push the HAMi package again: + +```bash +curl -k -s -u ':' -H 'Content-Type: application/json' -X POST "https:///api/v2.0/projects" -d '{"project_name":"mlops"}' +violet push --platform-address= --platform-username= --platform-password='' .tgz +``` + +Check that the `mlops` repositories listed in the Root Cause now exist. For Harbor instances in other regions, repeat the project creation and the push against each Harbor, or replicate the `mlops` project with a Harbor replication rule. + +## Diagnostic Steps + +Separate a controller stall from a bad file before restarting anything: + +```bash +violet version +kubectl get mplugin -A +kubectl get artifact,artifactversion -A +kubectl get moduleconfigs | grep +kubectl get moduleplugin -o yaml | grep -n -i 'failed to pull' +tar -tzf .tgz >/dev/null +``` + +- Push success, `mplugin` present (often with no `status`), `artifact` / `artifactversion` absent: Cause 1. +- ModulePlugin present, no ModuleConfig: Cause 2. +- A ModulePlugin error that `sentry` failed to pull the chart: Cause 3. +- HAMi (or another AI component) missing after a successful push to a Harbor registry without an `mlops` project: Cause 4. +- `unexpected EOF` from `violet` or `tar`, or an ArtifactVersion that is `Failed` or skipped as already existing: see the related article on parse errors and missing versions. From d9af042752efe8c555bd091436522ceb8d642c33 Mon Sep 17 00:00:00 2001 From: Komh Date: Thu, 12 Nov 2026 21:31:42 +0800 Subject: [PATCH 06/25] docs(solutions): add 23 and update 2 troubleshooting articles --- ...about_120_seconds_proxy_timeouts_on_ACP.md | 183 ++++++++++++++ ..._through_the_overwrite_ConfigMap_on_ACP.md | 208 ++++++++++++++++ ...lua_or_access_logs_lack_trace_id_on_ACP.md | 117 +++++++++ ...with_network_interface_not_found_on_ACP.md | 62 +++++ ...n-default_was_created_as_overlay_on_ACP.md | 74 ++++++ ..._Service_is_a_node_tunl0_address_on_ACP.md | 77 ++++++ ...es_data_hosts_from_cpaas-coredns_on_ACP.md | 64 ++++- ...when_a_CoreDNS_node_is_shut_down_on_ACP.md | 118 +++++++++ ...se_of_UDP_6081_TX_offload_or_MTU_on_ACP.md | 212 ++++++++++++++++ ..._lost_tunnel_packets_or_iptables_on_ACP.md | 201 +++++++++++++++ ...gateway_UI_plugins_are_installed_on_ACP.md | 70 ++++++ ...onses_60_second_cuts_504_and_SSE_bursts.md | 132 ++++++++++ ...Ingress_or_rejects_Ingress_saves_on_ACP.md | 229 ++++++++++++++++++ ...ert_or_CoreDNS_upstream_timeouts_on_ACP.md | 153 ++++++++++++ ..._or_logical_switch_port_holds_it_on_ACP.md | 147 +++++++++++ ...idge_network_InitOVSBridgeFailed_on_ACP.md | 192 +++++++++++++++ ...ops_tunnel_traffic_between_nodes_on_ACP.md | 95 ++++++++ ...oviderNetwork_or_underlay_subnet_on_ACP.md | 104 ++++++++ ...s_10254_or_control_plane_traffic_on_ACP.md | 203 ++++++++++++++++ ...r_marketplace-apiserver_restarts_on_ACP.md | 119 +++++++++ ...ts_caused_by_node_search_domains_on_ACP.md | 193 +++++++++++++++ ...led_or_old_cluster_state_remains_on_ACP.md | 119 +++++++++ ...se_global_cannot_reach_its_ports_on_ACP.md | 174 +++++++++++++ .../acp/NodeLocal_DNSCache_S2_Workarounds.md | 94 ++++++- ...me_clock_skew_or_hostname_change_on_ACP.md | 168 +++++++++++++ 25 files changed, 3506 insertions(+), 2 deletions(-) create mode 100644 docs/en/solutions/ALB_cuts_long_HTTP_or_TCP_connections_at_about_120_seconds_proxy_timeouts_on_ACP.md create mode 100644 docs/en/solutions/ALB_nginx_settings_are_reverted_after_a_restart_or_upgrade_change_them_through_the_overwrite_ConfigMap_on_ACP.md create mode 100644 docs/en/solutions/ALB_overwrite_ConfigMap_edits_do_not_apply_nginx_fails_with_init_by_lua_or_access_logs_lack_trace_id_on_ACP.md create mode 100644 docs/en/solutions/Adding_a_node_to_a_Kube-OVN_underlay_cluster_in_the_web_UI_fails_with_network_interface_not_found_on_ACP.md create mode 100644 docs/en/solutions/All_charts_stay_ChartDownloading_on_a_new_Kube-OVN_cluster_because_ovn-default_was_created_as_overlay_on_ACP.md create mode 100644 docs/en/solutions/An_unknown_source_IP_in_the_Calico_Pod_CIDR_that_is_not_a_Pod_or_Service_is_a_node_tunl0_address_on_ACP.md create mode 100644 docs/en/solutions/CoreDNS_fails_after_a_node_restart_or_lookups_time_out_when_a_CoreDNS_node_is_shut_down_on_ACP.md create mode 100644 docs/en/solutions/Cross_node_Pod_traffic_fails_on_the_Kube-OVN_Geneve_overlay_because_of_UDP_6081_TX_offload_or_MTU_on_ACP.md create mode 100644 docs/en/solutions/Cross_node_traffic_fails_in_the_host_network_path_default_route_firewall_lost_tunnel_packets_or_iptables_on_ACP.md create mode 100644 docs/en/solutions/Envoy_Gateway_console_pages_differ_between_two_environments_because_different_gateway_UI_plugins_are_installed_on_ACP.md create mode 100644 docs/en/solutions/Finding_which_hop_cuts_a_request_through_ALB_on_ACP_truncated_large_responses_60_second_cuts_504_and_SSE_bursts.md create mode 100644 docs/en/solutions/Ingress_NGINX_controller_serves_the_fake_certificate_ignores_an_Ingress_or_rejects_Ingress_saves_on_ACP.md create mode 100644 docs/en/solutions/Intermittent_DNS_failures_in_Pods_from_a_new_node_NIC_flapping_the_KubeDNS_alert_or_CoreDNS_upstream_timeouts_on_ACP.md create mode 100644 docs/en/solutions/Kube-OVN_fixed_IP_rejected_or_no_address_allocated_because_a_stale_IP_object_or_logical_switch_port_holds_it_on_ACP.md create mode 100644 docs/en/solutions/Kube-OVN_underlay_Pods_fail_with_resolve_MAC_address_timeout_or_bridge_network_InitOVSBridgeFailed_on_ACP.md create mode 100644 docs/en/solutions/New_node_cannot_reach_CoreDNS_or_the_API_server_because_a_firewall_drops_tunnel_traffic_between_nodes_on_ACP.md create mode 100644 docs/en/solutions/Node_API_and_VIP_access_lost_after_creating_a_Kube-OVN_ProviderNetwork_or_underlay_subnet_on_ACP.md create mode 100644 docs/en/solutions/Node_firewall_or_iptables_rules_block_kubelet_10250_metrics_10254_or_control_plane_traffic_on_ACP.md create mode 100644 docs/en/solutions/OperatorHub_instances_or_pages_fail_from_a_broken_11780_VIP_mapping_or_marketplace-apiserver_restarts_on_ACP.md create mode 100644 docs/en/solutions/Platform_UI_stuck_loading_or_DevOps_errors_from_Service_name_timeouts_caused_by_node_search_domains_on_ACP.md create mode 100644 docs/en/solutions/Pods_fail_on_one_Kube-OVN_node_after_Docker_or_a_second_runtime_is_installed_or_old_cluster_state_remains_on_ACP.md create mode 100644 docs/en/solutions/Workload_cluster_abnormal_or_monitoring_empty_in_the_console_because_global_cannot_reach_its_ports_on_ACP.md create mode 100644 docs/en/solutions/kube-ovn-cni_fails_on_one_node_after_a_NIC_rename_clock_skew_or_hostname_change_on_ACP.md diff --git a/docs/en/solutions/ALB_cuts_long_HTTP_or_TCP_connections_at_about_120_seconds_proxy_timeouts_on_ACP.md b/docs/en/solutions/ALB_cuts_long_HTTP_or_TCP_connections_at_about_120_seconds_proxy_timeouts_on_ACP.md new file mode 100644 index 000000000..48d491dd6 --- /dev/null +++ b/docs/en/solutions/ALB_cuts_long_HTTP_or_TCP_connections_at_about_120_seconds_proxy_timeouts_on_ACP.md @@ -0,0 +1,183 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.10.x + - 3.12.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# ALB cuts long HTTP or TCP connections at about 120 seconds: proxy timeouts on ACP + +Related articles: [Finding which hop cuts a request through ALB on ACP: truncated large responses, 60-second cuts, 504 and SSE bursts](Finding_which_hop_cuts_a_request_through_ALB_on_ACP_truncated_large_responses_60_second_cuts_504_and_SSE_bursts.md); [ALB nginx settings are reverted after a restart or upgrade; change them through the overwrite ConfigMap on ACP](ALB_nginx_settings_are_reverted_after_a_restart_or_upgrade_change_them_through_the_overwrite_ConfigMap_on_ACP.md). + +## Issue + +A call through an Alauda Load Balancer (ALB) is cut off at about two minutes, while the same call inside the cluster succeeds. Typical reports: + +- A long HTTP API call, a long-running task or a large-model call (streaming or not) through the ALB entry times out after about 2 minutes; the same call to the Service, the Pod or a NodePort returns after more than 2 minutes. A task that passes in a test environment times out in production. +- A frontend or rule timeout field (`spec.config.timeout`) is saved and then ignored. +- A database or other TCP service behind an ALB port drops connections after about two minutes, with `curl: (56) Recv failure: Connection reset by peer` or `curl: (8) Weird server reply` against the ALB port. The ALB `nginx` log shows `connection timed out (110: Operation timed out) while proxying connection, client: ..., server: 0.0.0.0:, upstream: ":"`, with a session time of about `120.001`. +- A TCP load test of a model service gets about half as many successful requests through the ALB as against the Pod. +- Someone asks whether `nginx.ingress.kubernetes.io/proxy-connect-timeout`, `proxy-send-timeout` and `proxy-read-timeout` apply to ALB and to an individual Ingress, or has set them and sees no change. + +The usual request is to raise the timeouts to 600 seconds (10 minutes), or longer for slow generation. + +## Root Cause + +`proxy_read_timeout` in nginx is an idle timer: it restarts whenever the upstream sends a byte, and it is not a cap measured from the TCP handshake. `proxy_send_timeout` is the idle timer in the other direction, and `proxy_connect_timeout` bounds only the TCP handshake to the upstream. ALB long connections behave the same: the timeout is counted from the last communication, not from connection creation. A handler that stays silent until the final body therefore looks exactly like a timer that started at connect time; a handler that streams tokens does not, as long as each gap stays inside the idle limit. + +### Cause 1: the ALB HTTP idle timers are still at the default 120 seconds + +The ConfigMap with the same name as the ALB in `cpaas-system` shows the defaults: + +```nginx +proxy_connect_timeout 5s; +proxy_send_timeout 120s; +proxy_read_timeout 120s; +proxy_buffering off; +keepalive_timeout 65; +proxy_next_upstream_tries 5; +``` + +Long-running calls, including large-model generation, that are silent for longer than 120 seconds are cut. An external load balancer in front of the ALB can also be the hop that closes first; its timer must be higher than the ALB timer or the client still dies at the outer limit. + +### Cause 2: per-frontend or per-rule timeout fields are not honored on the running version + +The Frontend and Rule fields `spec.config.timeout.proxy_connect_timeout_ms`, `proxy_read_timeout_ms` and `proxy_send_timeout_ms` are not supported by ALB on ACP 4.0.2: they can be saved, but the running proxy does not change. They are supported from the 4.0.4 patch release, and ALB on 4.3.2 supports them on a single forwarding rule. The ALB image moves with the platform and cannot be upgraded on its own. On a version without the fields, only the ALB-wide timeout can be changed; each ALB still has its own configuration, so two ALBs can have different values. + +### Cause 3: TCP or UDP listeners are cut by the stream proxy_timeout + +The ALB keeps separate timeouts for HTTP and for TCP/UDP forwarding, both 2 minutes by default. In the nginx stream configuration the idle timer is `proxy_timeout` in the `stream-tcp` and `stream-udp` sections. Raising the HTTP `proxy_*_timeout` values does not change it. A TCP port (for example a StarRocks query port forwarded to the FE Pod port 9030) drops connections after 120 seconds even when users estimate "about one minute"; the ALB log value decides. A long HTTP API published through an ALB TCP port is cut at 2 minutes even when the `http` block timeout is above 3 minutes, because `stream-tcp` still has `proxy_timeout 120s`. A TCP load test whose P99 latency (for example about 252 seconds) exceeds `proxy_timeout 120s` loses about half its requests. + +### Cause 4: the nginx.ingress.kubernetes.io proxy timeout annotations on an Ingress + +The annotations `nginx.ingress.kubernetes.io/proxy-connect-timeout`, `proxy-send-timeout` and `proxy-read-timeout` can be applied to ALB and to an individual Ingress; they correspond to the `proxy_*_timeout_ms` fields. Adding or removing them affects only that Ingress. + +On ACP 3.12 and 3.18, however, the ALB Ingress did not support these annotations: with them already set, the rendered ALB configuration did not change. On those versions, use the ALB-wide overwrite (Cause 1). + +## Resolution + +### Cause 1: the ALB HTTP idle timers are still at the default 120 seconds + +Decide the idle budget the application needs (600 seconds is the usual request), apply it on the ALB, then raise every device in front of the ALB so that the outer timer is strictly larger. + +- **ACP 4.0.4 and later**: set milliseconds on the Frontend (`frontends.crd.alauda.io`, `crd.alauda.io/v1`, usually named `-`) or on the Rule. 600000 ms is 10 minutes. The change is applied through Lua; nginx does not reload. + + ```yaml + apiVersion: crd.alauda.io/v1 + kind: Frontend + metadata: + name: - + namespace: cpaas-system + spec: + config: + timeout: + proxy_connect_timeout_ms: 600000 + proxy_read_timeout_ms: 600000 + proxy_send_timeout_ms: 600000 + ``` + + ```bash + kubectl -n cpaas-system patch frontend - --type merge -p '{"spec":{"config":{"timeout":{"proxy_read_timeout_ms":600000,"proxy_send_timeout_ms":600000}}}}' + ``` + +- **ACP 3.10 to 3.18, or when one value for the whole ALB is acceptable**: set the directives in the `http_server` section of the ALB overwrite ConfigMap (setup in the related article on the overwrite ConfigMap), keep the indentation of the copied block, and trigger the `alb2` update. `proxy_connect_timeout` can stay at `5s`. + + ```yaml + data: + http_server: |2 + # keep the existing lines of the copied block, and change or add: + proxy_send_timeout 600s; + proxy_read_timeout 600s; + ``` + + ```bash + kubectl -n cpaas-system annotate alb2 force-reload="$(date +%s)" --overwrite + kubectl -n cpaas-system exec -c nginx -- sh -c 'grep -R -n -E "proxy_(connect|send|read)_timeout" /alb/tweak | head' + ``` + + > **Warning:** These directives apply to every frontend on this ALB, and idle connections are held for the new duration, consuming connections, file descriptors and memory. Export the overwrite ConfigMap before editing. To roll back, restore the previous `proxy_*_timeout` values and trigger another `force-reload`. + +There is no separate connection-lifetime setting; lengthen the idle timers if the goal is "count from the last byte". + +### Cause 2: per-frontend or per-rule timeout fields are not honored on the running version + +1. Read the running ALB image and the platform version. On a version without the fields, stop retrying them. +2. Either upgrade the platform to a version that supports them (4.0.4 or later) and then use the Frontend or Rule fields, or set one ALB-wide value in the overwrite ConfigMap (Cause 1) until then. + + > **Warning:** A platform upgrade restarts the ALB and other platform components. Do it in a maintenance window, export the `alb2` and the overwrite ConfigMap first, and keep the previous version available as the rollback. Do not pull a newer `alb-nginx` image into the cluster by hand. + +### Cause 3: TCP or UDP listeners are cut by the stream proxy_timeout + +1. Find the stream sections and current values: + + ```bash + kubectl -n cpaas-system get configmap -o jsonpath='{.data}' | python3 -c 'import json,sys; print(list(json.load(sys.stdin)))' + kubectl -n cpaas-system get configmap -o yaml | grep -n -E 'proxy_timeout|proxy_connect_timeout' + ``` + +2. Raise `proxy_timeout` in `stream-tcp` (and `stream-udp` if used) in the overwrite ConfigMap, for example: + + ```nginx + proxy_next_upstream_tries 5; + proxy_connect_timeout 5s; + proxy_timeout 360s; + tcp_nodelay on; + ``` + + Values that fixed the examples above: 240s for the StarRocks connections, 360s for the API cut at 2 minutes, and 5 minutes for the large-model TCP load test. On versions that honor the Frontend timeout fields (Cause 2), a timeout set directly on a port's Frontend is used as an L4 timeout for that port only. + + > **Warning:** The change reloads nginx on every replica of this ALB, and longer idle timers keep idle TCP connections open longer. Keep a backup of the overwrite ConfigMap; if the ALB Pods are restarted to apply it, do it in a maintenance window. To roll back, restore the previous values and trigger the `alb2` update again. + +3. Trigger the `alb2` update and confirm the value in `/alb/tweak/stream-tcp.conf` and `/alb/tweak/stream-udp.conf`. Check that the value you wrote matches the request. + +### Cause 4: the nginx.ingress.kubernetes.io proxy timeout annotations on an Ingress + +Put the annotations on the Ingress whose rules need a different timer, in the ingress-nginx format (seconds), and confirm with a timed request: + +```yaml +metadata: + annotations: + nginx.ingress.kubernetes.io/proxy-connect-timeout: "600" + nginx.ingress.kubernetes.io/proxy-send-timeout: "600" + nginx.ingress.kubernetes.io/proxy-read-timeout: "600" +``` + +Only when the annotations are already set, the cluster is 3.12 or 3.18, and the timed request or the rendered configuration shows no change, move the timeout to the ALB-wide overwrite ConfigMap (Cause 1), with its warning. + +## Diagnostic Steps + +1. Show the image and the configured timers before changing them: + + ```bash + kubectl -n cpaas-system get pods -l service_name=alb2- -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.containers[*].image}{"\n"}{end}' + kubectl -n cpaas-system get alb2 -o jsonpath='{.spec.config.overwrite.configmap}{"\n"}' + kubectl get frontends.crd.alauda.io -n cpaas-system -o yaml | grep -n -A6 timeout + ``` + +2. Time the same path from outside, from the Service and from the Pod. A 120-second cliff only on the ALB URL is Cause 1 (HTTP) or Cause 3 (TCP port). The same cliff on the Pod is not the ALB: + + ```bash + curl -k -o /dev/null -sS -w 'code=%{http_code} start=%{time_starttransfer} total=%{time_total}\n' --max-time 700 "https://app.example.com/" + kubectl -n exec -- wget -S -O /dev/null "http://..svc:/" + ``` + +3. For a TCP listener, compare the ALB port with the NodePort of the same Service, and read the ALB log for stream timeouts: + + ```bash + nc -vz -w 3 192.0.2.10 + kubectl -n cpaas-system logs -c nginx --since=30m | grep -iE 'timed out|upstream' | tail + ``` + +4. For the Ingress annotations, read what the Ingress carries, then time a request through the ALB as in step 2: + + ```bash + kubectl -n get ingress -o jsonpath='{.metadata.annotations}{"\n"}' + ``` diff --git a/docs/en/solutions/ALB_nginx_settings_are_reverted_after_a_restart_or_upgrade_change_them_through_the_overwrite_ConfigMap_on_ACP.md b/docs/en/solutions/ALB_nginx_settings_are_reverted_after_a_restart_or_upgrade_change_them_through_the_overwrite_ConfigMap_on_ACP.md new file mode 100644 index 000000000..fe527dd92 --- /dev/null +++ b/docs/en/solutions/ALB_nginx_settings_are_reverted_after_a_restart_or_upgrade_change_them_through_the_overwrite_ConfigMap_on_ACP.md @@ -0,0 +1,208 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.10.x + - 3.14.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# ALB nginx settings are reverted after a restart or upgrade; change them through the overwrite ConfigMap on ACP + +Related articles: [ALB overwrite ConfigMap edits do not apply, nginx fails with init_by_lua, or access logs lack trace_id on ACP](ALB_overwrite_ConfigMap_edits_do_not_apply_nginx_fails_with_init_by_lua_or_access_logs_lack_trace_id_on_ACP.md). + +## Issue + +Timeouts, access-log format, request-header buffers and similar nginx settings on an Alauda Load Balancer (ALB) do not survive a restart or a platform upgrade. The data plane is the `ALB2` object (`alaudaloadbalancer2.crd.alauda.io`, short name `alb2`), usually in `cpaas-system`. Typical reports: + +- The ALB ConfigMap is restored every time it is edited. Where should ALB settings be configured? +- `proxy_read_timeout` was changed and goes back to `120s` after a restart. +- File downloads end with `gateway timeout` after a little over a minute, and a download timeout changed by hand is reset shortly afterwards. +- On ALB 3.14, edits to the Pod spec, container files or probes are reverted. +- A previously working nginx customization stopped working after an upgrade. +- After an upgrade to 4.3.3, the load balancer address and the domain stop answering, while `telnet` to the container port works. + +## Root Cause + +### Cause 1: the rendered ALB ConfigMap is not the source of truth + +The ConfigMap with the same name as the ALB in `cpaas-system` (next to `-port-info`) is rendered and managed by the ALB. After an edit, a restart or an upgrade the controller writes it again, so a direct change to `proxy_read_timeout` or any other key is lost. The durable place for nginx settings is an overwrite ConfigMap referenced from `spec.config.overwrite.configmap` on the `ALB2`. Setting up the reference is a one-time step; later changes go into the overwrite ConfigMap, followed by an update of the `ALB2` (see the related article). + +ALB configuration compatibility is not carried over automatically across a platform upgrade (for example from 3.8 to 3.12): customized parameters have to be checked and adapted by hand after the upgrade. + +### Cause 2: proxy_read_timeout was changed in the grpc_server section and returns to 120s + +Two mistakes together: the edit was made on the rendered ConfigMap (Cause 1), and it was made in the `grpc_server` section, which does not control these connection timeouts. Layer-7 timeouts are `proxy_read_timeout` and `proxy_send_timeout` in the `http` / `http_server` sections; the Layer-4 idle timeout is `proxy_timeout` in `stream-tcp`. + +### Cause 3: on ALB 3.14, Pod YAML or files edited by hand are reverted by the operator + +On platform 3.14 the ALB is owned by an operator (`alb-operator`). Changes to the Pod template, container files or probes are restored, and that release does not expose those fields on the CRD. Later 3.18 builds publish more ALB metrics; they do not make Pod edits durable on 3.14. + +### Cause 4: custom nginx settings stop working after an upgrade although the overwrite is still referenced + +The ALB overwrite from the previous version was not updated during the upgrade, so the upgraded ALB kept rendering from the old copy, which no longer matches the new template. The overwrite procedure itself is still valid; rebuilding the copy from the new version's template fixes it. + +### Cause 5: after an upgrade to 4.3.3, Pod annotations added on the ALB Deployment are removed + +When an ALB is set up for the MetalLB underlay scheme (MetalLB with ALB providing independent load balancing for underlay business traffic), the Pod annotations that scheme needs may have been added directly to the ALB Deployment. The upgrade to 4.3.3 regenerates the Deployment and drops those additions. Access through the load balancer address and the domain then fails, while the ALB itself is running and `telnet` to the container port works. + +## Resolution + +### Cause 1: the rendered ALB ConfigMap is not the source of truth + +1. Back up the live ConfigMap, the overwrite ConfigMap if it already exists, and the `alb2` object, before an upgrade and before every edit: + + ```bash + kubectl -n cpaas-system get configmap -o yaml > /tmp/-cm.backup.yaml + kubectl -n cpaas-system get configmap -cm-update -o yaml > /tmp/-cm-update.backup.yaml + kubectl -n cpaas-system get alb2 -o yaml > /tmp/-alb2.backup.yaml + ``` + +2. If `spec.config.overwrite` already exists, do not create a second copy: edit the ConfigMap it references and go to step 5. Otherwise create the overwrite ConfigMap once, from a full copy of the live `data` keys. Do not `kubectl apply` the backup itself; that object is owned by the ALB. + + ```bash + kubectl -n cpaas-system get configmap -o json | python3 -c ' + import json, sys + cm = json.load(sys.stdin) + out = { + "apiVersion": "v1", + "kind": "ConfigMap", + "metadata": { + "name": cm["metadata"]["name"] + "-cm-update", + "namespace": cm["metadata"]["namespace"], + }, + "data": cm.get("data") or {}, + } + json.dump(out, sys.stdout, indent=2) + print() + ' > /tmp/-cm-update.json + kubectl apply -f /tmp/-cm-update.json + ``` + + The same copy can be made in the console: **Cluster Management > Resource Management**, search ConfigMap for the ALB name, copy the content of `` into **Create Resource Object**, rename it `-cm-update`, and keep only `name` and `namespace` under `metadata` (remove `creationTimestamp`, `labels` such as `alb.cpaas.io/managed-by: alb-operator`, `managedFields`, `ownerReferences` and `resourceVersion`). + +3. Point the `ALB2` at that object. On 4.1, `spec.config.overwrite.configmap` of the `alb2` (`crd.alauda.io/v2beta1`) is a list of `name` entries, each a `namespace/name` string: + + > **Warning:** Pointing the ALB at a different ConfigMap for the first time can make the ALB remount it and restart, which interrupts its listeners; nginx takes the new configuration at once, and a faulty configuration breaks it at once. Check the content before saving the reference, do it in a maintenance window with more than one ALB replica, and keep the `alb2` backup. To roll back, remove the `overwrite` entry (apply the saved `alb2` YAML). + + ```bash + kubectl -n cpaas-system patch alb2 --type merge -p \ + '{"spec":{"config":{"overwrite":{"configmap":[{"name":"cpaas-system/-cm-update"}]}}}}' + ``` + +4. From now on, edit only the overwrite ConfigMap. Typical keys inside `data.http` are the nginx `http` context: `log_format`, `large_client_header_buffers`, `client_max_body_size` and the `proxy_*_timeout` directives. Leave every other line in place. For long calls (large-model inference, downloads), put `proxy_connect_timeout`, `proxy_send_timeout` and `proxy_read_timeout` here; this applies to the whole ALB, not to one frontend. Per-frontend timeouts are fields of the `Frontend` resource instead. On 4.2 the same kinds of setting are also available from the Ingress and ALB resources. + +5. After each edit, update the `alb2` so the controller regenerates its ConfigMap. The `force-reload` annotation is a hot reload and does not delete the Pod: + + > **Warning:** A bad `data.http` makes every replica of this ALB fail its reload or crash, which drops traffic for every Ingress and rule on it. Change one setting at a time. To roll back, re-apply the overwrite backup and set the annotation again. + + ```bash + kubectl -n cpaas-system annotate alb2 force-reload="$(date +%s)" --overwrite + ``` + +6. Confirm the rendered files in the ALB Pod under `/alb/tweak` (`http.conf`, `stream-tcp.conf`, `stream-udp.conf`) instead of assuming that the ConfigMap and the running file match: + + ```bash + kubectl -n cpaas-system exec -c nginx -- sh -c 'grep -R -n -E "proxy_(connect|send|read)_timeout|proxy_timeout" /alb/tweak | head' + ``` + + Alternatively, update the overwrite ConfigMap and restart the ALB Pods one at a time in a maintenance window, then read the files in the new Pod. + + > **Warning:** Restarting the ALB Pods drops in-flight connections on each replica and, for a single-replica ALB, interrupts all its listeners until the Pod is Ready. Wait until each replacement is Ready before the next. + + ```bash + kubectl -n cpaas-system get pods -o wide | grep + kubectl -n cpaas-system delete pod + kubectl -n cpaas-system exec -- cat /alb/tweak/stream-tcp.conf /alb/tweak/stream-udp.conf + ``` + +### Cause 2: proxy_read_timeout was changed in the grpc_server section and returns to 120s + +Set up the overwrite ConfigMap as in Cause 1 (or edit the one already referenced), and set `proxy_read_timeout` / `proxy_send_timeout` in the `http_server` section, and `proxy_timeout` in `stream-tcp` for TCP ports. Then trigger the `alb2` update and check `/alb/tweak` in the nginx container. + +```bash +kubectl -n cpaas-system get alb2 -o yaml | grep -A8 overwrite +kubectl -n cpaas-system get cm -cm-update -o yaml | grep -n -E '^ [a-z_-]+:|proxy_(read|send)_timeout|proxy_timeout' +``` + +### Cause 3: on ALB 3.14, Pod YAML or files edited by hand are reverted by the operator + +Stop editing the Deployment, the Pod and files inside the container. Put supported settings on the `ALB2` (or on rule and Ingress objects), and nginx settings in the overwrite ConfigMap. Probe and sidecar changes that are not CRD fields cannot be kept on 3.14. Do not scale `alb-operator` to zero to make a Pod edit stick; that is too risky for production. + +```bash +kubectl -n cpaas-system get deploy alb-operator +kubectl -n cpaas-system get alb2 -o yaml +``` + +### Cause 4: custom nginx settings stop working after an upgrade although the overwrite is still referenced + +1. Before an upgrade, export the overwrite ConfigMap and the `alb2` (Cause 1, step 1). +2. After the upgrade, rebuild the overwrite ConfigMap from the new version's rendered ConfigMap and re-apply only the custom lines, so the copy matches the new template. Then trigger the `alb2` update. + + > **Warning:** Replacing the overwrite content reloads nginx on every replica of this ALB, and a mistake stops the reload for all its listeners. Keep backups of the old overwrite ConfigMap and the new rendered ConfigMap, and write down the list of custom lines. To roll back, re-apply the old overwrite backup and change the `alb2` annotation again. + +3. Some settings are fields of the `alb2` resource rather than nginx text. For example, forwarded headers are enabled with a `config` block under `spec`, leaving the other spec fields (`enablePortProject`, `networkMode`, `nodeSelector`) unchanged, in a maintenance window: + + ```yaml + spec: + config: + proxy_header: + use_forward_headers: "true" + ``` + + Verify inside the ALB container: + + ```bash + kubectl -n cpaas-system exec -- sh -c 'cat /etc/alb2/nginx/policy.new | grep forwarded' + ``` + +### Cause 5: after an upgrade to 4.3.3, Pod annotations added on the ALB Deployment are removed + +Put the Pod annotations on the `alb2` resource instead of the Deployment, with the `alb.cpaas.io/pod-annotations` annotation, so they persist at the `alb2` level and are rendered into the Pods. Replace the example key and value with the annotations the scheme needs: + +> **Warning:** The ALB Pods roll to pick up the new annotations, and wrong network annotations can leave them without the intended address. Copy the annotations from the pre-upgrade Deployment YAML (or the scheme documentation) and save the `alb2` YAML. To roll back, run `kubectl annotate alb2 -n cpaas-system alb.cpaas.io/pod-annotations-`. + +```bash +kubectl annotate alb2 -n cpaas-system \ + 'alb.cpaas.io/pod-annotations={"example.com/key":"value"}' \ + --overwrite +``` + +Also check the external load balancer's forwarding to the ALB. + +## Diagnostic Steps + +1. Identify which ALB owns the traffic and whether an overwrite ConfigMap is already referenced. If the jsonpath prints nothing, edits to the rendered ConfigMap will be lost (Cause 1): + + ```bash + kubectl get alb2 -A + kubectl -n cpaas-system get alb2 -o jsonpath='{.spec.config.overwrite.configmap}{"\n"}' + kubectl -n cpaas-system get configmap -cm-update + ``` + +2. For a timeout that returns to `120s`, find which section carries the edited line. A hit only under `grpc_server` is Cause 2: + + ```bash + kubectl -n cpaas-system get cm -o yaml | grep -n -E '^ [a-z_-]+:|proxy_(read|send)_timeout|proxy_timeout' + ``` + +3. On 3.14, an existing `alb-operator` Deployment is the reconciler that undoes Pod edits (Cause 3): + + ```bash + kubectl get deploy -n cpaas-system alb-operator + ``` + +4. After an upgrade, diff the rendered `data.http` against the backup (Cause 4), and compare the Pod annotations with the Deployment YAML saved before the upgrade (Cause 5): + + ```bash + kubectl -n cpaas-system get configmap -o jsonpath='{.data.http}' > /tmp/live-http.conf + diff -u /tmp/expected-http.conf /tmp/live-http.conf + kubectl -n cpaas-system get pod -o jsonpath='{.metadata.annotations}{"\n"}' + kubectl -n cpaas-system get alb2 -o jsonpath='{.metadata.annotations}{"\n"}' + ``` diff --git a/docs/en/solutions/ALB_overwrite_ConfigMap_edits_do_not_apply_nginx_fails_with_init_by_lua_or_access_logs_lack_trace_id_on_ACP.md b/docs/en/solutions/ALB_overwrite_ConfigMap_edits_do_not_apply_nginx_fails_with_init_by_lua_or_access_logs_lack_trace_id_on_ACP.md new file mode 100644 index 000000000..045c37708 --- /dev/null +++ b/docs/en/solutions/ALB_overwrite_ConfigMap_edits_do_not_apply_nginx_fails_with_init_by_lua_or_access_logs_lack_trace_id_on_ACP.md @@ -0,0 +1,117 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.10.x + - 3.14.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# ALB overwrite ConfigMap edits do not apply, nginx fails with init_by_lua, or access logs lack trace_id on ACP + +Related articles: [ALB nginx settings are reverted after a restart or upgrade; change them through the overwrite ConfigMap on ACP](ALB_nginx_settings_are_reverted_after_a_restart_or_upgrade_change_them_through_the_overwrite_ConfigMap_on_ACP.md). + +## Issue + +An Alauda Load Balancer (ALB) already uses an overwrite ConfigMap (referenced from `spec.config.overwrite.configmap` of the `alb2`, setup described in the related article), and: + +- a timeout or another key was changed in the overwrite ConfigMap, the ALB Pod was restarted, and the running nginx still has the old value; +- before a production change, someone asks whether editing the overwrite ConfigMap restarts the ALB and how much a timeout change affects the business; +- after the overwrite ConfigMap was created, the ALB Pod crashes in Lua before it listens, with `init_by_lua` errors and the `balancer` module failing to load; +- the access logs do not contain a trace ID, so a failed call cannot be tied to the caller's `X-Trace-Id`. + +## Root Cause + +### Cause 1: edits to the overwrite ConfigMap are not synced until the alb2 resource changes + +When an overwrite ConfigMap is set, edits to it are not synced on their own: the ConfigMap with the same name as the ALB is regenerated only when the `alb2` resource itself changes. A Pod restart is not such a change. + +A second problem can hide behind the first. After the `alb2` was updated, a value can still be missing because the overwrite ConfigMap itself is wrong, for example `proxy_connect_timeout s60;` instead of `proxy_connect_timeout 60s;` in the `stream-tcp` section (the unit must follow the number), or lines in the last section of the YAML that need more indentation. On 3.14 the new values can appear only after a delay, although a mounted ConfigMap normally reloads quickly. + +Editing only the content of the ConfigMap that is already referenced has small impact: normally none, and with many slow connections the ALB uses somewhat more memory. Adding an annotation to the `alb2` to trigger the update applies the new value without restarting the ALB Pod. + +### Cause 2: init_by_lua error, cannot load the balancer module + +The overwrite ConfigMap was cloned from the live object, but part of `data.http` was dropped (for example while adding `escape=json` to the log format), so Lua initialization no longer sees the balancer setup, `init_by_lua` fails and the `balancer` module does not load. + +### Cause 3: the access logs do not contain trace_id + +The default `log_format http` has no `trace_id` variable. This is a missing overwrite entry, not a dropped request. + +## Resolution + +### Cause 1: edits to the overwrite ConfigMap are not synced until the alb2 resource changes + +1. After every edit of the overwrite ConfigMap, change the `alb2` resource so that the controller regenerates the ALB ConfigMap. Any annotation change works, for example: + + ```bash + kubectl -n cpaas-system annotate alb2 force-reload="$(date +%s)" --overwrite + ``` + + In the console the same step is **Resource Management**, resource type `alb2`, version `v2beta1`: edit the ALB and add an annotation. + +2. Compare the rendered ConfigMap with the overwrite copy, then check the running files inside the ALB Pod: + + ```bash + kubectl -n cpaas-system get configmap -cm-update -o yaml | grep -n -E 'proxy_(connect_)?timeout|proxy_(send|read)_timeout' + kubectl -n cpaas-system get configmap -o yaml | grep -n -E 'proxy_(connect|send|read)_timeout' + kubectl -n cpaas-system exec -c nginx -- sh -c 'grep -R -n -E "proxy_(connect|send|read)_timeout|proxy_timeout" /alb/tweak 2>/dev/null | head' + ``` + +3. If the new value is still absent, the overwrite ConfigMap itself is wrong. Look for a malformed value such as `s60` instead of `60s`, a misplaced entry, or indentation errors in the last block, fix it, and trigger the `alb2` update again. A correct `stream-tcp` section looks like this: + + ```nginx + proxy_connect_timeout 60s; + proxy_next_upstream_tries 5; + proxy_timeout 600s; + tcp_nodelay on; + ``` + + > **Warning:** A broken overwrite ConfigMap makes every replica of this ALB fail its reload, which drops traffic for every Ingress and rule on it. Back up the overwrite ConfigMap before each edit. To roll back, re-apply the backup and change the `alb2` annotation again. + +### Cause 2: init_by_lua error, cannot load the balancer module + +Rebuild the overwrite ConfigMap from a fresh full copy of the live object's `data` (procedure in the related article), then change only `log_format`. Replacing `log_format http` with `log_format http escape=json` is enough for JSON access logs. Keep the `access_log` line pointed at that format, for example `access_log /dev/stdout http buffer=16k flush=1s;`. Then trigger the `alb2` update. + +> **Warning:** Reloading after a truncated `http` block keeps the ALB down until the full `data.http` is restored. Keep a backup of the overwrite ConfigMap. To roll back, `kubectl apply` the backup and set the `force-reload` annotation again. + +### Cause 3: the access logs do not contain trace_id + +Add the map and the log variables to the overwrite ConfigMap `data.http`, preserving the rest of the content, then trigger the `alb2` update. When the client sends `X-Trace-Id`, that value is logged; otherwise nginx's `$request_id` is used. + +```nginx +map $http_x_trace_id $trace_id { + default $http_x_trace_id; + "" $request_id; +} +``` + +Inside `log_format http`, add `trace_id="$trace_id"` and `req_id="$request_id"`. To forward the inbound header to the upstream, add `proxy_set_header X-Trace-Id $http_x_trace_id;` in the `http_server` section of the same ConfigMap. Do not create a second log format name; the data plane reads the format named `http`. + +## Diagnostic Steps + +1. Confirm that an overwrite ConfigMap is referenced. If not, start with the related article: + + ```bash + kubectl -n cpaas-system get alb2 -o jsonpath='{.spec.config.overwrite.configmap}{"\n"}' + ``` + +2. If the overwrite is set and a restart did not apply an edit, compare the overwrite ConfigMap, the rendered ConfigMap and `/alb/tweak` in the Pod (Cause 1). A value missing from the rendered ConfigMap means the `alb2` was not updated; a malformed value in the overwrite copy is the second part of Cause 1. + +3. For a Lua crash, read the previous container log before editing anything else. `init_by_lua` and `balancer` errors are Cause 2: + + ```bash + kubectl -n cpaas-system logs -c nginx --previous | grep -E 'init_by_lua|balancer' + ``` + +4. After a successful reload, confirm the running files rather than the ConfigMap alone: + + ```bash + kubectl -n cpaas-system exec -c nginx -- sh -c 'find / -path "*tweak*http.conf" 2>/dev/null | head' + ``` diff --git a/docs/en/solutions/Adding_a_node_to_a_Kube-OVN_underlay_cluster_in_the_web_UI_fails_with_network_interface_not_found_on_ACP.md b/docs/en/solutions/Adding_a_node_to_a_Kube-OVN_underlay_cluster_in_the_web_UI_fails_with_network_interface_not_found_on_ACP.md new file mode 100644 index 000000000..54a6c2fe2 --- /dev/null +++ b/docs/en/solutions/Adding_a_node_to_a_Kube-OVN_underlay_cluster_in_the_web_UI_fails_with_network_interface_not_found_on_ACP.md @@ -0,0 +1,62 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x +--- + +# Adding a node to a Kube-OVN underlay cluster in the web UI fails with network interface not found on ACP + +## Issue + +In the web UI, adding a node to a business cluster that uses a Kube-OVN underlay network fails with an error that the network interface (`网卡`, NIC) was not found. Only the node IP, the username and the password were entered. + +- The add-node page seems to have no place to enter the NIC, and the NIC of this node cannot be added on the network provider page either. +- The cluster nodes do not all use the same NIC name. The bridge network status lists the nodes being added as `InitOVSBridgeFailed` with `failed to create and configure external bridge br-provider: nic does not exist`, and the node that has `` as `InitOVSBridgeSucceeded`. + +## Root Cause + +### Cause 1: the NIC setting of each node row in the add-node form was left collapsed and empty + +On ACP 4.0, the add-node form of an underlay cluster has a network interface setting for every node, but it is folded into the node row. It opens from the expand arrow (`>`) at the very start of the row, to the left of the node type selector (for example `计算节点`, compute node) and the `IPv4 地址` (IPv4 address) field. When the row is not expanded, no interface is specified for the node, and node addition fails with the interface-not-found error. + +The `nic does not exist` condition on the new nodes is consistent with this: with no per-node entry, the bridge network looks for its default interface, which those machines do not have. + +## Resolution + +### Cause 1: the NIC setting of each node row in the add-node form was left collapsed and empty + +1. On the node to be added, read the name of the NIC that carries the underlay network. A VLAN interface prints as `@`; enter the part before `@`. + + ```bash + ip -br link + ip -br addr + ``` + +2. In the add-node form, click the `>` arrow at the start of the node row to expand it. +3. Fill in the network interface of that node with the NIC name from step 1, then submit. The node is then added. +4. Check that the bridge network now reports the node as ready: + + ```bash + kubectl get provider-networks -o jsonpath='{range .status.conditions[*]}{.node}{" "}{.reason}{" "}{.message}{"\n"}{end}' + ``` + + `InitOVSBridgeSucceeded` for the new node means the bridge was created on the NIC you entered. + +For a node that is already in the cluster and was never given a matching NIC, set the per-node NIC in the ProviderNetwork itself (`customInterfaces`, or `excludeNodes` for a node without an underlay NIC), as described in [Kube-OVN underlay Pods fail with resolve MAC address timeout or bridge network InitOVSBridgeFailed on ACP](Kube-OVN_underlay_Pods_fail_with_resolve_MAC_address_timeout_or_bridge_network_InitOVSBridgeFailed_on_ACP.md). + +## Diagnostic Steps + +Compare what the bridge network expects with what the new machine has: + +```bash +kubectl get provider-networks -o jsonpath='default={.spec.defaultInterface}{"\n"}custom={.spec.customInterfaces}{"\n"}exclude={.spec.excludeNodes}{"\n"}' +kubectl get provider-networks -o jsonpath='{range .status.conditions[*]}{.node}{" "}{.reason}{"\n"}{end}' | grep -v Succeeded +ip -br link +``` + +- A node being added through the web UI whose NIC name differs from the default interface, together with a NIC-not-found error on the add, points to the collapsed NIC field (Cause 1). +- If the NIC that the bridge network names is missing on a node that has been in the cluster for a while, or it vanished after a hardware or IaaS change, the node row is not the issue; fix the ProviderNetwork or restore the NIC on the host (see the article linked above). +- If `kube-ovn-cni` on the node fails with `failed to initialize config` or `failed to parse config` for an old interface name, the tunnel interface is wrong, not the bridge NIC; see [kube-ovn-cni fails on one node after a NIC rename, clock skew or hostname change on ACP](kube-ovn-cni_fails_on_one_node_after_a_NIC_rename_clock_skew_or_hostname_change_on_ACP.md). diff --git a/docs/en/solutions/All_charts_stay_ChartDownloading_on_a_new_Kube-OVN_cluster_because_ovn-default_was_created_as_overlay_on_ACP.md b/docs/en/solutions/All_charts_stay_ChartDownloading_on_a_new_Kube-OVN_cluster_because_ovn-default_was_created_as_overlay_on_ACP.md new file mode 100644 index 000000000..6daf6bbf4 --- /dev/null +++ b/docs/en/solutions/All_charts_stay_ChartDownloading_on_a_new_Kube-OVN_cluster_because_ovn-default_was_created_as_overlay_on_ACP.md @@ -0,0 +1,74 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# All charts stay ChartDownloading on a new Kube-OVN cluster because ovn-default was created as overlay on ACP + +Related articles: [Node, API and VIP access lost after creating a Kube-OVN ProviderNetwork or underlay subnet on ACP](Node_API_and_VIP_access_lost_after_creating_a_Kube-OVN_ProviderNetwork_or_underlay_subnet_on_ACP.md); [Kube-OVN underlay Pods fail with resolve MAC address timeout or bridge network InitOVSBridgeFailed on ACP](Kube-OVN_underlay_Pods_fail_with_resolve_MAC_address_timeout_or_bridge_network_InitOVSBridgeFailed_on_ACP.md). + +## Issue + +On a newly added cluster that was planned with a Kube-OVN underlay network, CoreDNS and the other platform components are never installed. `kubectl get ars -A` shows every AppRelease in `cpaas-system` (`alauda-alb2`, `cni-kube-ovn`, `coredns`, `prometheus`, `region` and others) in `ChartDownloading`, and `kubectl get rpch` returns `No resources found`. Describing one AppRelease shows: + +```text +ChartDownloadFailed [1/1] failed to download chart acp/chart-csi-driver-nfs-v3.18.3, err: failed to do request: Head "http://:60080/v2/acp/chart-csi-driver-nfs/manifests/v3.18.3": dial tcp :60080: connect: connection timed out +``` + +At the same time: + +- `telnet` and `curl -v http://:60080/v2` from the cluster nodes succeed (the answer is `Client sent an HTTP request to an HTTPS server.`), and every Pod is `Running`. +- Restarting `sentry` has no effect. Its log has no download error, only `too many processors, process later ... MaxConcurrentProcessors=10 CurrentProcessors=10`. +- OVN logs show nothing abnormal. + +## Root Cause + +### Cause 1: ovn-default was created as an overlay subnet although the cluster was planned as underlay + +The `ovn-default` subnet was meant to be underlay but was configured as overlay: `kubectl get subnet` shows `ovn-default` with `NAT` `true` and no VLAN. In this network design the Pods could not reach the registry on that path, so every chart download from inside the cluster timed out, while the nodes, which use the host network, could reach it. From inside the `sentry` Pod: + +```text +curl: (28) Failed to connect to port 60080 after 130152 ms: Could not connect to server +``` + +A subnet that references a VLAN is an underlay subnet; one without a VLAN is an overlay subnet. + +## Resolution + +### Cause 1: ovn-default was created as an overlay subnet although the cluster was planned as underlay + +1. Confirm the signature: the node `curl` answers, the `curl` from `sentry` fails, and `ovn-default` has no VLAN although the cluster was planned as underlay (see Diagnostic Steps). + +2. Redeploy the cluster with `ovn-default` as an underlay subnet (with the planned VLAN, gateway and NIC). + + > **Warning:** Redeploying deletes the cluster and everything running on it, and it cannot be undone. Only when the cluster is newly added and has no business workloads yet, and the signature in step 1 is confirmed. If the `curl` from `sentry` works, or `ovn-default` already has the intended type, this is not the fault: look at DNS resolution of the registry address and the `sentry` component instead, which also leave charts in `ChartDownloading`. + +3. After the redeploy, test from a Pod on the underlay subnet to the registry and to the global cluster's platform address. A firewall between the underlay subnet and the global cluster can still block that path; open it on the network side. + +## Diagnostic Steps + +1. List the AppReleases stuck in download and read the error: + + ```bash + kubectl get ars -A | grep -i chartdownloading + kubectl -n cpaas-system describe ars | grep -A2 ChartDownloadFailed + ``` + +2. Compare the node path with the Pod path to the registry. The node should answer; a timeout from `sentry` matches this issue: + + ```bash + curl -v --max-time 10 http://:60080/v2/ + kubectl -n cpaas-system get pod | grep sentry + kubectl -n cpaas-system exec -- curl -v --max-time 10 http://:60080/v2/ + ``` + +3. Check the type of `ovn-default`. An empty `vlan` with `nat=true` on a cluster planned as underlay confirms the cause: + + ```bash + kubectl get subnet ovn-default -o jsonpath='vlan={.spec.vlan} provider={.spec.provider} nat={.spec.natOutgoing}{"\n"}' + ``` diff --git a/docs/en/solutions/An_unknown_source_IP_in_the_Calico_Pod_CIDR_that_is_not_a_Pod_or_Service_is_a_node_tunl0_address_on_ACP.md b/docs/en/solutions/An_unknown_source_IP_in_the_Calico_Pod_CIDR_that_is_not_a_Pod_or_Service_is_a_node_tunl0_address_on_ACP.md new file mode 100644 index 000000000..fd2d7d7cc --- /dev/null +++ b/docs/en/solutions/An_unknown_source_IP_in_the_Calico_Pod_CIDR_that_is_not_a_Pod_or_Service_is_a_node_tunl0_address_on_ACP.md @@ -0,0 +1,77 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# An unknown source IP in the Calico Pod CIDR that is not a Pod or Service is a node tunl0 address on ACP + +## Issue + +On a cluster that uses Calico in IPIP mode: + +- The access log of an application Pod keeps showing one client address sending requests to paths the application does not have, and every request returns `404` in rapid succession. Examples: `"POST /seeyon/rest/authentication/ucpcLogin HTTP/1.1" 404`, `"GET /index.php?option=com_joomlapicasa2&controller=../../../../etc/passwd%00 HTTP/1.1" 404`, `"GET /geoserver/ows?service=WFS&version=1.0.0&request=GetCapabilities HTTP/1.1" 404`, `"GET /actuator/httptrace HTTP/1.1" 404`. +- The question is which Pod has this IP. `kubectl get pods -A -o wide` and `kubectl get svc -A -o wide` do not find the address, although it is inside the Pod CIDR and answers `ping`. +- The address may end in `.0`, which looks like a network address and not a host. +- The requests may stop on their own before anyone can capture them. + +## Root Cause + +### Cause 1: the address is the Calico IPIP tunnel address (tunl0) of a node, so the sender is a process on that node + +Calico allocates each node a block of the IP pool (block size from the `IPPool` `blockSize`, for example in `default-ipv4-ippool`) and gives the node's `tunl0` interface an address from its own block. That address is in the Pod CIDR but belongs to no Pod or Service, which is why the usual lookups return nothing. + +A process running on the host itself that connects to a Pod on another node goes out through `tunl0`, and the kernel uses the `tunl0` address as the source. The application therefore logs the node's tunnel address as the client. Because blocks start on block boundaries, the first address of a block can end in `.0` and still be a valid tunnel address. + +The request paths above match a vulnerability scan. When the node that owns the address runs no application Pods (for example a master with scheduling disabled) and platform audit and container logs do not line up with the requests, the requests did not come from a Pod; security or scanning software installed on that node is the likely origin. Kubernetes, Pod IP allocation and the application itself are not involved. + +## Resolution + +### Cause 1: the address is the Calico IPIP tunnel address (tunl0) of a node, so the sender is a process on that node + +1. Map the address to a node. The node annotation holds each node's IPIP tunnel address, and the IPAM block shows which node owns the range: + + ```bash + kubectl get nodes -o custom-columns='NAME:.metadata.name,TUNL0:.metadata.annotations.projectcalico\.org/IPv4IPIPTunnelAddr' + kubectl get ippools.crd.projectcalico.org -o yaml | grep -E 'name:|cidr|blockSize|ipipMode' + kubectl get ipamblocks.crd.projectcalico.org -o yaml | grep -B 15 -A 30 '' + ``` + + On the node that owns the block, confirm the interface address: + + ```bash + ip -br addr show tunl0 + ``` + +2. Treat the node, not the cluster, as the sender. Check what runs on that node and whether any Pod there could be the client: + + ```bash + kubectl get pod -A -o wide --field-selector spec.nodeName= + kubectl get events -A -o wide --sort-by=.lastTimestamp | grep + ``` + +3. While the requests are arriving, find the local process. Run these on the node that owns the address: + + ```bash + ss -tnp | grep ':' + tcpdump -nn -i tunl0 host and port -c 20 + ``` + + `ss -p` prints the process name and PID of the connection. A host process (not a container) as the owner confirms this cause. + +4. If the requests have already stopped, read the node's OS logs and the scan records of any security or vulnerability-scanning software installed on the node for the same time window, and ask its owner whether a scan ran. + +## Diagnostic Steps + +Rule out cluster objects first, then look the address up in Calico IPAM: + +```bash +kubectl get pods -A -o wide | grep -w +kubectl get svc -A -o wide | grep -w +kubectl get ipamblocks.crd.projectcalico.org -o yaml | grep -B 15 -A 30 '' +kubectl get nodes -o custom-columns='NAME:.metadata.name,TUNL0:.metadata.annotations.projectcalico\.org/IPv4IPIPTunnelAddr' | grep -w +``` + +No Pod or Service, an address inside a block whose `affinity` is `host:`, and that node's `tunl0` carrying the address is Cause 1. `calicoctl` is not needed for these lookups; the IPAM blocks are custom resources. `arp -n` on the application node shows nothing for this address, because the traffic arrives through the IPIP tunnel rather than on the local segment. diff --git a/docs/en/solutions/CoreDNS_Host_Alias_records_do_not_take_effect_because_a_ResourcePatch_removes_data_hosts_from_cpaas-coredns_on_ACP.md b/docs/en/solutions/CoreDNS_Host_Alias_records_do_not_take_effect_because_a_ResourcePatch_removes_data_hosts_from_cpaas-coredns_on_ACP.md index 39fd934a2..bcde85d99 100644 --- a/docs/en/solutions/CoreDNS_Host_Alias_records_do_not_take_effect_because_a_ResourcePatch_removes_data_hosts_from_cpaas-coredns_on_ACP.md +++ b/docs/en/solutions/CoreDNS_Host_Alias_records_do_not_take_effect_because_a_ResourcePatch_removes_data_hosts_from_cpaas-coredns_on_ACP.md @@ -3,7 +3,9 @@ kind: - Troubleshooting products: - Alauda Container Platform -ProductsVersion: [] +ProductsVersion: + - 4.2.x + - 4.3.x --- # CoreDNS Host Alias records do not take effect because a ResourcePatch removes data hosts from cpaas-coredns on ACP @@ -38,6 +40,22 @@ As a result: - new Host Alias entries never reach CoreDNS; - after a restart, CoreDNS starts with no static records, and because of `fallthrough` the names are resolved by the upstream DNS instead, which can return a different, stale address. +### Other reasons a static record does not reach CoreDNS + +The same symptom (a name added for CoreDNS never resolves in Pods) has two more causes, even when no ResourcePatch removes `/data/hosts`. + +**A direct edit of `cpaas-coredns` is converted into a ResourcePatch.** `cpaas-coredns` is a chart-rendered platform resource. The `sentry` component watches UPDATE and DELETE of objects that carry the `helm.sh/chart-name` label, through the `ValidatingWebhookConfiguration` named `sentry` (its webhook `vsentry.kb.io` calls the sentry Service `/validate`). A `kubectl edit` or `kubectl patch` of the ConfigMap returns: + +```text +admission webhook "vsentry.kb.io" denied the request: The operation is denied, but ResourcePatch is created. Please wait sentry to apply the ResourcePatch. +``` + +The edit is not rejected for good: sentry applies the ResourcePatch it created. Whichever copy sentry applied last is the one CoreDNS serves, so a ResourcePatch can keep an older hosts list after the plugin page was saved, or the opposite, and the **Host Alias** list and the ConfigMap disagree. + +**A hosts entry added by hand is in the wrong format or is overridden.** The Corefile loads the `hosts` plugin from a file (`hosts /etc/coredns/hosts`), and that file is the `hosts` data key of the same ConfigMap. A mapping written as a line inside the Corefile `hosts` block, or as a single-line data field `hosts: `, is not what the `hosts` key needs; it must be a multiline block (`hosts: |` followed by one ` ` line per entry). Even with the format corrected and CoreDNS restarted, a hand edit can still be overridden by the plugin-managed copy (the same mechanism as above); the entry then takes effect only when it is entered under **Host Alias**. + +**Host Alias** is also the way to publish additional names to Pods when the node `/etc/resolv.conf` cannot take more nameservers. + ## Resolution 1. Back up the ConfigMap and the ResourcePatch: @@ -82,6 +100,44 @@ As a result: Do not add a `remove /data/hosts` operation to a ResourcePatch for CoreDNS again. Keep static records in **Host Alias** and limit the ResourcePatch to the Corefile changes that cannot be made in the plugin. +### If a direct edit was denied by vsentry.kb.io + +The edit was converted into a ResourcePatch for sentry to apply; modification is not prohibited. Do not send a second patch. Wait, then check that the ResourcePatch was applied and that the ConfigMap contains the change: + +```bash +kubectl get validatingwebhookconfiguration sentry -o yaml +kubectl get rpch -A +kubectl get rpch -o yaml +kubectl -n kube-system get cm cpaas-coredns -o yaml +``` + +When the **Host Alias** list and the ConfigMap disagree, pick one writer and keep to it: + +- **ConfigMap edits** (when other custom lines must stay in the ConfigMap): keep editing `cpaas-coredns` and let sentry apply the ResourcePatch. +- **Host Alias only**: back up the ConfigMap and the ResourcePatch as in step 1, delete the ResourcePatch that overrides the plugin, and enter the IP and the domains (separated by spaces) under **Host Alias**. + + > **Warning:** Deleting the ResourcePatch removes every ConfigMap customization that is not on the plugin page, including a hand-written `forward` block. Confirm with the application owners which lines must be re-entered first. To roll back, apply the saved ResourcePatch and ConfigMap. + +### If a hand-added hosts entry is ignored after a restart + +1. Enter the record under **Host Alias** in **Alauda Container Platform Cluster DNS** (fields **IP** and **Domains**). On ACP 4.x this setting is reloaded without a CoreDNS restart. Then query again from the workload Pod: + + ```bash + kubectl -n exec -- nslookup + ``` + +2. If the record must stay in the ConfigMap instead, keep it in one place only (not both the Corefile and the `hosts` key), and write the `hosts` key as a multiline block: + + ```yaml + data: + hosts: | + 192.0.2.10 app.example.com + ``` + + > **Warning:** Editing `cpaas-coredns` changes DNS for every Pod in the cluster, and a malformed Corefile or hosts file can stop CoreDNS from loading. Back up the ConfigMap first (step 1). Sentry turns the write into a ResourcePatch, as described above. To roll back, run `kubectl apply -f cpaas-coredns.before.yaml`. + + If the ConfigMap edit still does not show up in answers, move the entry to **Host Alias**. + ## Diagnostic Steps 1. Check whether the ConfigMap carries the static records. A `Corefile` that references `/etc/coredns/hosts` but no `hosts` key matches this issue: @@ -112,3 +168,9 @@ Do not add a `remove /data/hosts` operation to a ResourcePatch for CoreDNS again kubectl -n exec -- cat /etc/resolv.conf kubectl -n exec -- nslookup ``` + +5. Tell the causes apart: + + - `cpaas-coredns` has no `hosts` key and a ResourcePatch on it has `op: remove` on `/data/hosts`: the ResourcePatch removes the hosts data (this article's main cause). + - A direct edit returned `admission webhook "vsentry.kb.io" denied the request`, or the **Host Alias** list and the ConfigMap `hosts` entries differ: the edit was converted into a ResourcePatch. + - A hosts line was added to `cpaas-coredns` by hand and a CoreDNS restart did not help: check the format of the `hosts` key and move the entry to **Host Alias**. diff --git a/docs/en/solutions/CoreDNS_fails_after_a_node_restart_or_lookups_time_out_when_a_CoreDNS_node_is_shut_down_on_ACP.md b/docs/en/solutions/CoreDNS_fails_after_a_node_restart_or_lookups_time_out_when_a_CoreDNS_node_is_shut_down_on_ACP.md new file mode 100644 index 000000000..4287e60c8 --- /dev/null +++ b/docs/en/solutions/CoreDNS_fails_after_a_node_restart_or_lookups_time_out_when_a_CoreDNS_node_is_shut_down_on_ACP.md @@ -0,0 +1,118 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# CoreDNS fails after a node restart or lookups time out when a CoreDNS node is shut down on ACP + +Related articles: [Intermittent DNS failures in Pods from a new node, NIC flapping, the KubeDNS alert or CoreDNS upstream timeouts on ACP](Intermittent_DNS_failures_in_Pods_from_a_new_node_NIC_flapping_the_KubeDNS_alert_or_CoreDNS_upstream_timeouts_on_ACP.md). + +## Issue + +Cluster DNS is down or degraded after a change to the nodes, and the cause is outside the Corefile. Typical reports: + +- After the nodes restarted, CoreDNS and other basic components in the global cluster are all abnormal. +- A node that ran CoreDNS was shut down. The CoreDNS Pod restarted and application lookups timed out for a while. The question is why, and which platform components a node restart can affect. +- CoreDNS was scaled from 2 to 4 replicas with `kubectl scale`, and some time later it is back to 2. + +CoreDNS on ACP 4.x is the `coredns` Deployment in `kube-system` (Pod label `k8s-app=kube-dns`), configured by the ConfigMap `cpaas-coredns`. The CoreDNS Pods use the `/etc/resolv.conf` of the node they run on as their upstream list, and `9153` is their metrics port. + +## Root Cause + +### Cause 1: the node /etc/resolv.conf is empty after a restart, so CoreDNS has no upstream + +The default Corefile forwards with `forward . /etc/resolv.conf`. When `/etc/resolv.conf` on the node exists but is empty (no `nameserver` line), CoreDNS has no upstream and fails to start; CoreDNS normally logs `no nameservers found` in this case. Components that depend on DNS fail with it. + +The file can be reset to empty when the node restarts by tooling outside the platform, for example on the IaaS side (cloud-init, DHCP, or image tooling). Restoring the file contents returns cluster DNS to normal. + +### Cause 2: a node running CoreDNS is shut down and lookups time out until its endpoint is removed + +When a node goes down abruptly, kubelet cannot report that it is leaving. The endpoint controller removes that node's Pods from the `kube-dns` endpoints only after the node is marked `NotReady`, so there is a delay. During it, DNS caching and connection reuse in the clients keep sending queries to the offline replica until the cache expires or the connection closes, and lookups time out. + +Platform components are deliberately not pinned to specific nodes or to the masters: a pinned Pod cannot fail over when its node fails, and master resources differ between clusters, so heavy components there could destabilize the masters. The components whose node going down affects the whole cluster are: + +| Release | Component (Deployment) | Effect when its node goes down | +| --- | --- | --- | +| 3.8 | `cpaas-system/lcn-ccm-alb` | business traffic entering and leaving the cluster | +| 3.16 | the `*-alb`, `*-clb`, `*-lb` Deployments in `cpaas-system` | business traffic entering and leaving the cluster | +| 3.8, 3.16 | `cpaas-system/coredns` | in-cluster name resolution | + +On 4.x CoreDNS is `kube-system/coredns`. `raven` and `calico-kube-controllers` affect only changes such as Pod creation and deletion; running workloads are not affected. When every business team uses its own ALB on hosts it manages, CoreDNS is effectively the only shared component to plan around. + +### Cause 3: CoreDNS replica changes made with kubectl are reverted + +The CoreDNS Deployment is managed by an upper-layer platform controller, which overwrites direct edits to the Deployment. A replica count set with `kubectl scale deploy` therefore returns to the earlier value after a while. A replica count changed through the platform UI persists. + +## Resolution + +### Cause 1: the node /etc/resolv.conf is empty after a restart, so CoreDNS has no upstream + +Only when `/etc/resolv.conf` on the node running the CoreDNS Pod is empty (no `nameserver` line). + +1. Restore the file contents, from your own backup or from a healthy node of the same cluster with the same resolvers. +2. Recreate the CoreDNS Pods. A Pod's resolver file is written when the Pod is created, so a container restart alone does not pick up the restored file. + +> **Warning:** Writing the wrong resolvers into `/etc/resolv.conf` breaks external name resolution for the node and for CoreDNS, and recreating the CoreDNS Pods briefly delays cluster lookups. Make sure the nameservers you restore are the ones the cluster used before (compare with a healthy node). To roll back, restore the previous file and recreate the Pods again. + +```bash +cat /etc/resolv.conf +cp -a /etc/resolv.conf /root/resolv.conf.empty +scp :/etc/resolv.conf /etc/resolv.conf +kubectl -n kube-system delete pod -l k8s-app=kube-dns +kubectl -n kube-system get pod -l k8s-app=kube-dns -o wide -w +``` + +3. Ask the owner of the node image or IaaS why the file is emptied on restart; otherwise the next reboot repeats it. + +### Cause 2: a node running CoreDNS is shut down and lookups time out until its endpoint is removed + +No configuration change fixes a past event; the timeouts end once the endpoint is removed or the node returns. For planned work: + +1. Use a change process: report node restarts or decommissioning to the cluster administrator and announce them to the other business teams on the cluster beforehand. Keep the platform components unpinned so they can fail over. +2. Drain the node before a planned shutdown. Eviction removes the CoreDNS Pod from the endpoints before the node goes away, instead of waiting for `NotReady`. + +> **Warning:** Draining evicts every non-DaemonSet Pod on the node, and Pods with no other replica are unavailable until they start elsewhere. Make sure at least one other CoreDNS replica is Ready on another node. To roll back, run `kubectl uncordon `; evicted Pods are not moved back automatically. + +```bash +kubectl -n kube-system get pod -l k8s-app=kube-dns -o wide +kubectl drain --ignore-daemonsets --delete-emptydir-data +``` + +### Cause 3: CoreDNS replica changes made with kubectl are reverted + +Set the replica count in the platform UI rather than with `kubectl scale`, then confirm that the count holds: + +```bash +kubectl -n kube-system get deploy coredns -o jsonpath='{.spec.replicas}{"\n"}' +``` + +## Diagnostic Steps + +1. Check CoreDNS state and the previous container log: + + ```bash + kubectl -n kube-system get pod -l k8s-app=kube-dns -o wide + kubectl -n kube-system describe pod -l k8s-app=kube-dns | grep -iE 'Reason|Restart|Ready' + kubectl -n kube-system logs -l k8s-app=kube-dns --previous --tail=50 + ``` + +2. On the node of a failing CoreDNS Pod, read the resolver file. No `nameserver` line is Cause 1: + + ```bash + cat /etc/resolv.conf + grep -c '^nameserver' /etc/resolv.conf + ``` + +3. Compare the time of the lookup failures with node events. A CoreDNS restart and lookup timeouts right after its node was shut down is Cause 2: + + ```bash + kubectl get events -A --field-selector reason=NodeNotReady --sort-by=.lastTimestamp + kubectl -n kube-system get endpointslice -l kubernetes.io/service-name=kube-dns -o wide + ``` + +4. A replica count that returns to the old value after `kubectl scale` is Cause 3. diff --git a/docs/en/solutions/Cross_node_Pod_traffic_fails_on_the_Kube-OVN_Geneve_overlay_because_of_UDP_6081_TX_offload_or_MTU_on_ACP.md b/docs/en/solutions/Cross_node_Pod_traffic_fails_on_the_Kube-OVN_Geneve_overlay_because_of_UDP_6081_TX_offload_or_MTU_on_ACP.md new file mode 100644 index 000000000..fe772f761 --- /dev/null +++ b/docs/en/solutions/Cross_node_Pod_traffic_fails_on_the_Kube-OVN_Geneve_overlay_because_of_UDP_6081_TX_offload_or_MTU_on_ACP.md @@ -0,0 +1,212 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.10.x + - 3.16.x + - 4.0.x + - 4.3.x +--- + +# Cross-node Pod traffic fails on the Kube-OVN Geneve overlay because of UDP 6081, TX offload or MTU on ACP + +## Issue + +Pods on one node cannot reach Pods on other nodes, while the hosts can reach each other and Services sometimes still answer. Typical reports: + +- Pod IP and Service access fail on and off from a few nodes, with no clear pattern. +- A new node joins, and its Pods (or CoreDNS on it) cannot reach the API server Service. `kube-ovn-pinger` on the new node logs `failed to parse config` and times out against the API. +- NodePort access through a second node is unstable, and a capture of Geneve traffic shows `IP/TCP Checksum Bad`. +- Pods fail only on physical machines while VM nodes are fine. +- After the node operating system was upgraded (for example to Kylin), some Pods fail to start because they cannot resolve or connect to Service names. +- Engineers ask how to see and change the Kube-OVN MTU and why the standard value is `1400`. +- After an MTU change, operations become slow; after the change is rolled back, containers still cannot reach external Redis or MySQL, or `ovn0` stays at `1350` instead of returning to `1400`. + +All of these involve the overlay path between nodes: Geneve on UDP `6081`, the offload settings of the encapsulation NIC, and the MTU. + +## Root Cause + +### Cause 1: the new node receives no Geneve traffic on UDP 6081 + +Kube-OVN carries Pod traffic between nodes in Geneve tunnels on UDP `6081`. When a firewall, security group or switch ACL blocks that port, `tcpdump` on the new node shows no UDP `6081` from the existing nodes, the tunnel never comes up, the new node's OVN components stay abnormal, and `kube-ovn-pinger` logs `failed to parse config` because it cannot reach the API through the overlay. + +### Cause 2: tx-checksum-ip-generic is on and Geneve packets are damaged + +With checksum offload on, the kernel leaves the checksum of encapsulated packets for the NIC to fill in. When `ethtool -k ` shows `tx-checksum-ip-generic: on` (not `off [fixed]`) and the NIC fills it in wrongly, cross-node encapsulated traffic fails or the encapsulation NIC counts checksum errors, while tiny pings still pass. In extreme cases the offload also makes connection tracking drop connections, which shows up as loss and retransmissions. + +Situations fixed by turning this offload off: + +- Pod IPs and Services intermittently unreachable from only some nodes. +- NodePort access unstable when a node forwards to another node. +- Pods in a new cluster unable to reach the `kubernetes` Service or any node while node-to-node traffic works. +- On a global cluster, `kubectl` on two of three masters unable to reach the API server, Harbor (sharing the global ALB VIP) unreachable, `console-portal` returning `500`, and `apollo` logging `Get router manager error {"error": "cluster not found"}` and `net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)`. All OVN Pods were `Running`, and restarting OVN and platform components did not help. + +Two cautions: + +- A flag that is already `off [fixed]` rules this cause out (look at Cause 1), and a flag that is `on` while the `genev_sys_6081` counters do not move is an idle tunnel interface, not a checksum fault. +- `IP/TCP Checksum Bad` on inner packets in a capture on the sending node is not proof of corruption by itself: with the offload on, `tcpdump` sees the packet before the NIC fills in the checksum. Only a capture on the receiving side is evidence. + +### Cause 3: transmit offload on a physical or Kylin node rewrites encapsulated packets + +On some physical NICs and on Kylin and other domestic operating systems, a NIC validation defect is incompatible with OVN encapsulation and modifies transmitted packets. UDP `6081` can already be open, and the node still cannot use the Pod network. Turning all transmit offload off (`ethtool -K tx off`) recovers it. This is broader than Cause 2, which turns off only `tx-checksum-ip-generic`. + +Situations fixed this way: + +- On ACP 4.3.2 ARM, Pods on a new physical machine cannot complete cross-node requests, while the masters are virtual machines. In a cluster mixing VMs and physical machines, Pods fail only on the physical machines. +- On a new Kylin node, Pods cannot reach Pods on other nodes while Services still work. A capture shows Geneve-encapsulated ICMP requests and replies between nodes with no kernel drops. `ethtool -K tx off` on the cluster bond fixes it. +- On a new node, `cert-manager-cainjector`, `kube-ovn-pinger`, `sentry` and other Pods are `CrashLoopBackOff`, and `coredns` is `Running` but `0/1`, logging `plugin/kubernetes: waiting for Kubernetes API before starting server`, `plugin/ready: Plugins not ready: "kubernetes"` and `starting server with unsynced Kubernetes API`. From the node, the API server Service on `443` answers; from inside the `coredns` network namespace it hangs. `ethtool -k ` shows `tx-checksumming: on`. +- On a new workload cluster of arm64 physical machines with bonded NICs, every `kube-ovn-pinger` Pod is `CrashLoopBackOff` with `failed to get self pod ... context deadline exceeded`. The physical NICs under the bond compute wrong checksums for Geneve packets; `tx off` is needed on every slave NIC of the bond, and the permanent fix is a NIC firmware update. The step-by-step procedure is in [kube-ovn-pinger CrashLoopBackOff context deadline exceeded on bonded NICs due to Geneve TX checksum offload on ACP](kube-ovn-pinger_CrashLoopBackOff_context_deadline_exceeded_on_bonded_NICs_due_to_Geneve_TX_checksum_offload_on_ACP.md). +- After the node OS was upgraded to Kylin, some Pods fail to start because they cannot resolve or connect to Service names. With older kernels and some domestic NICs, checksum offload makes the NIC compute wrong checksums, and the receiving NIC drops the packets as invalid. There are two fixes: upgrade the kernel to `4.19.90-25.16.v2101` or later, or turn checksum offload off with `ethtool -K tx off`. + +### Cause 4: which MTU Kube-OVN uses, and why the standard value is 1400 + +By default Kube-OVN reads the MTU of the host NIC and sets the Pod and OVS interface MTU from it, reduced in overlay networks by the encapsulation overhead: + +| Encapsulation | Pod MTU | +|---|---| +| Geneve over IPv4 | physical MTU - 100 | +| Geneve over IPv6 | physical MTU - 120 | +| VXLAN over IPv4 | physical MTU - 50 | +| VXLAN over IPv6 | physical MTU - 70 | + +The standard value of `1400` is the usual `1500` physical MTU minus `100` bytes for Geneve, which avoids fragmentation. Underlay Pods are not encapsulated, so their MTU follows the physical NIC (`1500` on a `1500` NIC). On ACP 4.x the Kube-OVN MTU can be customized, globally for overlay networks or per subnet, with the product procedure **Networking > How To > Kube-OVN > Configure MTU**. ACP 3.16 has no supported way to change the default OVN MTU. + +### Cause 5: after an MTU change is rolled back, ovn0 and some Pod interfaces keep the old MTU + +When the OVN MTU is changed (for example from `1400` to `1350`) and then rolled back, Pods created while the MTU was `1350` keep `1350` on their interfaces. In OVS an internal port such as `ovn0` cannot have a larger MTU than the other interfaces attached to the same bridge, so those Pods also hold `ovn0` at `1350`. Workloads on those Pods can fail to connect to external databases such as Redis or MySQL. This is default behavior: once every Pod with the old MTU is restarted, all interfaces become `1400` and `ovn0` and the other internal ports follow. + +## Resolution + +### Cause 1: the new node receives no Geneve traffic on UDP 6081 + +1. Open UDP `6081` from every existing node's encapsulation address to the new node, and the reverse path. Check the host firewall and any security group or switch ACL between the nodes. Restrict the source to the node CIDR. + + > **Warning:** The rule opens the Geneve port on the node. Limit the source to the cluster node prefix, apply it in a maintenance window, and keep console access. To roll back, delete the same rule (`iptables -D INPUT ...`). A network ACL outside the host has to be rolled back on that device. + + ```bash + iptables -I INPUT -p udp -s --dport 6081 -j ACCEPT + iptables -I OUTPUT -p udp -d --dport 6081 -j ACCEPT + ip6tables -I INPUT -p udp -s --dport 6081 -j ACCEPT + ``` + +2. Persist the rule with the mechanism the node already uses (`iptables-save`, `firewall-cmd --permanent --add-rich-rule`, or the cloud security group). +3. After packets show up in `tcpdump`, delete the unready `kube-ovn-cni` and `kube-ovn-pinger` Pods on that node only, so they rebuild the tunnel. + +### Cause 2: tx-checksum-ip-generic is on and Geneve packets are damaged + +1. Disable the generic IP checksum offload on the encapsulation NIC. Skip this when the driver already reports `off [fixed]` or when error counters are not increasing. + + > **Warning:** Disabling offload raises CPU use on that NIC and can lower throughput. Apply it to the encapsulation interface only, in a maintenance window. To roll back, run `ethtool -K tx-checksum-ip-generic on` if the driver allows it. + + ```bash + ethtool -K tx-checksum-ip-generic off + ethtool -k | grep tx-checksum-ip-generic + ``` + +2. The setting does not survive a reboot. To keep it, add a udev rule (replace `` in both places), reboot in a window, and confirm the flag afterwards. + + > **Warning:** A udev rule runs on every add event for that interface. A wrong interface name turns offload off on another NIC or does nothing. To roll back, delete the rule file and reload udev. + + ```bash + cat > /etc/udev/rules.d/99-ovn-fix-checksum.rules <<'RULE' + ACTION=="add", SUBSYSTEM=="net", KERNEL=="", RUN+="/sbin/ethtool -K tx-checksum-ip-generic off" + RULE + udevadm control --reload-rules && udevadm trigger + ethtool -k | grep tx-checksum-ip-generic + ``` + +### Cause 3: transmit offload on a physical or Kylin node rewrites encapsulated packets + +1. Only when cross-node Pod traffic fails on a physical node, a newly added Kylin node, or a node whose OS has the NIC validation defect, and UDP `6081` is open (otherwise it is Cause 1), turn transmit offload off on the encapsulation NIC. + + > **Warning:** `tx off` disables transmit offload on that NIC, raises CPU use and can lower throughput. Apply it to the encapsulation NIC only, in a maintenance window. To roll back, run `ethtool -K tx on` if the driver accepts it. A healthy VM node does not need this. + + ```bash + ethtool -k | grep -E '^tx|tx-checksum|tx-udp' + ethtool -K tx off + ethtool -k | grep -E 'tx-checksumming|tcp-segmentation-offload' + ``` + + Expected after the change: `tx-checksumming: off`, `tx-checksum-ip-generic: off`, `tcp-segmentation-offload: off`. + +2. On a bond, apply `tx off` to the bond and to every slave NIC (`cat /proc/net/bonding/` lists them). To keep the setting on nodes that use network-scripts files, add it to each NIC's file (keep a copy first; to roll back, restore it): + + ```bash + cp -a /etc/sysconfig/network-scripts/ifcfg- /root/ifcfg-.before + echo 'ETHTOOL_OPTS="-K tx off"' >> /etc/sysconfig/network-scripts/ifcfg- + ``` + +3. Crash-looping Pods retry on their own once the path works; `kubectl -n kube-system delete pod -l app=kube-ovn-pinger` only skips the back-off wait. Ask the NIC vendor for a firmware update that fixes the offloaded checksum. + +4. For the Kylin kernel case, the alternative to `tx off` is a kernel at `4.19.90-25.16.v2101` or later. + + > **Warning:** A kernel upgrade needs a drain and a reboot of each node. Upgrade one node at a time and keep the previous kernel in the boot menu as the rollback. + + ```bash + uname -r + kubectl drain --ignore-daemonsets --delete-emptydir-data + # install the vendor kernel package and reboot the node + kubectl uncordon + ``` + +### Cause 4: which MTU Kube-OVN uses, and why the standard value is 1400 + +Read the MTU in effect before changing anything. On ACP 4.x, follow the product procedure **Configure MTU**; on ACP 3.16 keep the default. + +```bash +kubectl -n kube-system exec -- ovs-vsctl list interface | grep -E "name|mtu" +kubectl -n exec -- ip link show eth0 +ip link show +``` + +`ip link show eth0` inside the Pod is the value the Pod really uses. + +### Cause 5: after an MTU change is rolled back, ovn0 and some Pod interfaces keep the old MTU + +1. After the MTU setting itself is back at `1400`, find the interfaces that still carry the old value. The list also prints `lo` and the physical NICs, whose MTU is legitimately different; look at the Pod veths and `ovn0`. + + ```bash + ip -o link show | awk '{for(i=1;i<=NF;i++) if($i=="mtu") print $2, $(i+1)}' | grep -v ' 1400$' + ip link show ovn0 + kubectl get pod -A -o wide --field-selector spec.nodeName= + ``` + +2. Restart the Pods whose `eth0` still reports the old MTU. When no Pod interface on the bridge is below `1400` any more, `ovn0` follows. + + > **Warning:** Restarting the Pods interrupts those workloads; roll them in a window, spare replicas first. Only after the MTU setting itself has been restored; otherwise the new Pods get the wrong value again. + + ```bash + kubectl -n exec -- ip link show eth0 + kubectl -n rollout restart deploy/ + ip link show ovn0 + ``` + +## Diagnostic Steps + +Compare a failing node with a healthy one. Check that Geneve arrives, then read the offload flags on the encapsulation NIC (often a bond): + +```bash +tcpdump -ni any -c 20 'udp and port 6081' +iptables -S | grep 6081 || true +ethtool -k | grep -E '^tx-checksumming|tx-checksum-ip-generic|tcp-segmentation-offload' +ip -s link show genev_sys_6081 +netstat -us | grep -i InCsumErrors +kubectl -n kube-system exec -- ovs-vsctl show | grep -C5 remote_ip +``` + +Test from a Pod network namespace as well as from the host. The host path working while the Pod path fails is typical for Causes 2 and 3: + +```bash +crictl ps -a | grep coredns +crictl inspect | grep -i '"pid"' +nsenter -n -t -- timeout 5 bash -c '/443' && echo open || echo blocked +timeout 5 bash -c '/443' && echo open || echo blocked +kubectl -n exec -- curl -k -m 5 https://:443/healthz +``` + +- No UDP `6081` from the other nodes is Cause 1. +- `tx-checksum-ip-generic: on` with failing cross-node traffic or climbing errors is Cause 2. +- A physical or newly added Kylin node that works only after `tx off`, Pods that cannot resolve Service names after an OS upgrade to Kylin, or bonded physical NICs where API clients crash-loop with `context deadline exceeded` to the `kubernetes` Service, is Cause 3. `InCsumErrors` increasing while the Pods fail supports it. Rule out a node firewall first: if the node itself cannot reach the API server on `6443`, the problem is in the host network, not in the overlay. +- Interfaces still at an old MTU after a rollback, with `ovn0` below `1400`, are Cause 5. diff --git a/docs/en/solutions/Cross_node_traffic_fails_in_the_host_network_path_default_route_firewall_lost_tunnel_packets_or_iptables_on_ACP.md b/docs/en/solutions/Cross_node_traffic_fails_in_the_host_network_path_default_route_firewall_lost_tunnel_packets_or_iptables_on_ACP.md new file mode 100644 index 000000000..44b9ea13a --- /dev/null +++ b/docs/en/solutions/Cross_node_traffic_fails_in_the_host_network_path_default_route_firewall_lost_tunnel_packets_or_iptables_on_ACP.md @@ -0,0 +1,201 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# Cross-node traffic fails in the host network path: default route, firewall, lost tunnel packets or iptables on ACP + +## Issue + +The CNI Pods are `Running`, yet traffic between nodes breaks. Typical reports: + +- Nodes added to a cluster through their out-of-band address become abnormal. +- An interface that the server can reach cannot be reached from inside a container on some nodes. +- Hundreds of Pods are down, and `kube-ovn-pinger` can reach its own node but no other node. +- Monitoring is abnormal on a new cluster because one `vmselect` replica has no network. +- After someone ran `iptables-restore` on a node, the workload cluster shows as abnormal in the platform, its VIP answers only some of the time, and restarting containerd does not help. On the global cluster, `erebus` logs `cluster "" is temporarily not accessible`. +- Cross-node communication fails and `ping` returns `Destination Host Prohibited`. + +Each cause below is found by testing the path between two specific nodes: their routes, the firewalls between them, and whether a packet sent by one ever arrives at the other. + +## Root Cause + +### Cause 1: nodes joined through the out-of-band IP have their only default route on the out-of-band NIC + +Hosts with an out-of-band and an in-band address (for example `bond1` and `bond0`) were added to the cluster using the out-of-band IP. On normal nodes the default route prefers `bond0` (in-band, metric `300`) with `bond1` (out-of-band, metric `301`) as backup. On the abnormal nodes the only default route is via `bond1`, and the `bond0` default route is missing. Kubernetes control-plane traffic depends on the in-band interface, so the nodes lose communication with the control plane. Restoring the default route on the joined nodes brings the node status back, but Pods still fail to start; the routing has to be correct before the node is added. + +### Cause 2: the firewall between two groups of nodes is not fully open + +The `kube-ovn-pinger` results on an affected node show the pattern: `ping pod` to most other pinger Pods reports `count: 3, loss count 3, average rtt 0.00ms`, with only a few at `loss count 0`; `ping node` reports `loss count 3` to most nodes and `loss count 0` only to a small group, which is the other nodes of the affected group plus the masters; DNS lookups of `kubernetes.default` and an external name work; `ping external address` reports `loss count 3`. The affected nodes and the other nodes belong to two different network zones or systems, and the network policies between them are not fully open for node and Pod traffic. + +### Cause 3: replies never reach the requesting node because a conflicting subnet takes the return path + +From the `kube-ovn-pinger` Pod, a request to the local node works and requests to other nodes fail. A capture on `genev_sys_6081` while a `curl` runs in the pinger's network namespace shows, on port `6443`, the requester sending repeated `[SYN]` packets and `[TCP Retransmission]` of them, and the target nodes answering with `[SYN, ACK]` (some marked `[TCP Retransmission]` and `[TCP Out-Of-Order]`) that the requester never receives. A subnet that conflicts with a cluster range sends the replies elsewhere. Removing the conflicting subnet fixes it. When the nodes are VMs, also check whether the virtualization routing mode or the physical hosts under the VMs changed. + +### Cause 4: IPIP packets from one node never arrive at one peer's NIC + +On a Calico cluster in IPIP mode, every node has a `tunl0` interface (` mtu 1480`) with a `/32` address. From one master, a ping to a worker's `tunl0` address gets `100% packet loss`, while another worker reaches it. `tcpdump` on the master shows echo requests leaving on `tunl0` with no replies; a capture of IPIP (`ip proto 4`) on `eth0` shows the worker sending IPIP packets to the master (including TCP `[S.]` replies from port `6443`), and the master's `eth0` receives none of them, with no RX or TX errors or drops. The packets are lost in the underlying network between those two hosts, outside the cluster. Pods on the affected node (for example a `vmselect` replica) lose connectivity. + +### Cause 5: iptables-restore overwrote the node rules and the workload cluster VIP flaps + +`iptables-restore` run on a node of a workload cluster overwrites the node's iptables rules. The cluster's self-built VIP (served by keepalived, the `alive` container) then answers only some of the time, and the platform marks the cluster abnormal. Restarting containerd does not help. On the global cluster both `erebus` Pods in `cpaas-system` are `Running` with no restarts, and their log repeats: + +```text +proxy.go:158] cluster "" is temporarily not accessible +Generate new reverse proxy for: 600:https://
:6443 +``` + +Restarting the keepalived container restores VIP connectivity, and the workload cluster returns to normal. + +The signature of this cause is the VIP itself failing part of the time. The same `temporarily not accessible` text, returned to UI requests and by `apollo` as `410` with reason `Gone`, can also appear while the VIP answers every time; then the problem is on the platform side, not in the node network, and the steps below do not apply. + +### Cause 6: Destination Host Prohibited, a host FORWARD rule rejects cross-node traffic + +`calico-node`, `kube-proxy` and the other node Pods are `Running`, but a cross-node `ping` gets `Destination Host Prohibited` for every packet. `iptables -L FORWARD -n --line-numbers` on the affected node shows, after the `DOCKER-USER`, `DOCKER-ISOLATION-STAGE-1`, `ACCEPT` and `DOCKER` rules, a rule `REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited`, and the command may also print `# Warning: iptables-legacy tables present, use iptables-legacy to see them`. That host firewall rule rejects the forwarded cross-node traffic. + +## Resolution + +### Cause 1: nodes joined through the out-of-band IP have their only default route on the out-of-band NIC + +1. Compare the routing table with a normal node: + + ```bash + ip route show default + ip -br addr show bond0 bond1 + ``` + +2. Remove the node, correct the default route, and join it again. Restoring the route alone brings the node status back but does not make the Pods start. + + > **Warning:** Changing the default route can cut the SSH session that uses it; work from a console. Deleting a node evicts and removes all its Pods, and the node must be joined again; drain it first. Only when the node's default route is on the out-of-band NIC and normal nodes prefer the in-band NIC. Save `ip route` before the change; to roll back, add the saved route again. + + ```bash + kubectl drain --ignore-daemonsets --delete-emptydir-data + kubectl delete node + ip route del default via dev bond1 + ip route add default via dev bond0 metric 300 + ``` + +3. Make the route persistent in the node's network configuration, with a lower metric for the in-band NIC than for the out-of-band one, confirm `ip route show default` matches a normal node, and add the node again with its in-band address. + +### Cause 2: the firewall between two groups of nodes is not fully open + +1. Use the pinger results to map which nodes cannot reach which: + + ```bash + kubectl -n kube-system get pod -l app=kube-ovn-pinger -o wide | grep + kubectl -n kube-system logs --tail=500 | grep -E 'ping (pod|node|external)' | grep -v 'loss count 0' | head -50 + ``` + +2. Group the failing targets by node. A clean split along a system or zone boundary, with only the masters reachable across it, is this cause. Have the network team open the policies between the two groups for node and Pod traffic. + +### Cause 3: replies never reach the requesting node because a conflicting subnet takes the return path + +1. Capture on the Geneve interface in one window, and run the `curl` from the pinger's network namespace in another, on the same node: + + ```bash + tcpdump -i genev_sys_6081 -n -v -w genev_sys_6081_traffic.pcap + ``` + +2. Check the route back to the source on the target node and on the requester, and compare it with the cluster subnets: + + ```bash + ip route get + ip route + kubectl get subnet -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidrBlock' + ``` + + A reply route that points somewhere other than the path back to the requester, through a range that overlaps a cluster subnet, identifies the conflicting subnet. + +3. Remove the conflicting subnet or route. + + > **Warning:** Deleting a subnet or a route that overlaps a cluster range changes where that whole range is routed. Confirm which object is the conflicting one and that nothing still needs it. Save it first (`kubectl get subnet -o yaml`, or `ip route` on the node) so it can be recreated. + +### Cause 4: IPIP packets from one node never arrive at one peer's NIC + +1. Prove the loss on the wire. Run the `ip proto 4` capture on both nodes at the same time: + + ```bash + ip a | grep tunl0 + ping -c 5 + tcpdump -i any -nnve icmp and host + tcpdump -i eth0 -nnve ip proto 4 and host and host + ip -s link show eth0 + ``` + + Packets leaving the sender's NIC that never appear on the receiver's NIC, with clean NIC counters, are this cause. + +2. Hand the captures to the IaaS or network team. Until the underlying network is repaired, take the affected node out of service (replace it if needed). + + > **Warning:** Draining the node moves or stops every Pod on it. To roll back, uncordon it once the underlying network is repaired. + + ```bash + kubectl drain --ignore-daemonsets --delete-emptydir-data + kubectl uncordon + ``` + +### Cause 5: iptables-restore overwrote the node rules and the workload cluster VIP flaps + +1. On the global cluster, confirm that `erebus` sees the cluster as unreachable, then test the workload cluster's VIP repeatedly: + + ```bash + kubectl -n cpaas-system get pod -o wide | grep erebus + kubectl -n cpaas-system logs --tail=500 | grep -E 'temporarily not accessible|Generate new reverse proxy' + for i in $(seq 1 30); do curl -k -s -o /dev/null -m 2 -w '%{http_code}\n' https://:6443/healthz; sleep 1; done | sort | uniq -c + ``` + + Restarting `erebus` can confirm that the status persists; the fault is on the VIP side. + +2. Restart the keepalived container (`alive`) on the workload cluster masters, one master at a time. + + > **Warning:** Only when the `healthz` loop above shows failures or timeouts. Restarting keepalived moves or briefly drops the VIP, so API access through the VIP is interrupted for a moment. Test the VIP between masters. There is nothing to roll back; the container comes back with its own configuration. + + ```bash + kubectl get pod -A -o wide | grep -i alive + crictl ps | grep -i alive || docker ps | grep -i alive + ``` + + If `alive` is a Pod, delete that Pod on the master so it is recreated. Otherwise restart the container with the node's runtime (`docker restart `, or `crictl stop ` for a kubelet-managed container, which kubelet then starts again). + +3. Save the node's rules (`iptables-save`) before anyone runs `iptables-restore` on a cluster node again. + +### Cause 6: Destination Host Prohibited, a host FORWARD rule rejects cross-node traffic + +Only when a cross-node ping returns `Destination Host Prohibited` and the `FORWARD` chain holds a `REJECT ... reject-with icmp-host-prohibited` rule that healthy nodes do not have, delete that rule by its line number. + +> **Warning:** Changing the host firewall opens forwarded traffic that the rule used to reject; agree it with whoever owns the node security policy. Save the rules first. To roll back, run `iptables-restore < /root/iptables.before-forward`. The line number changes when rules are added or removed, so read it again right before the delete. + +```bash +iptables-save > /root/iptables.before-forward +iptables -L FORWARD -n --line-numbers +iptables -D FORWARD +ping -c 3 +``` + +Find where the host loads the rule from (for example a saved iptables service ruleset) so that a reboot does not bring it back. + +## Diagnostic Steps + +Pick one failing pair of nodes and test the path between them in both directions: + +```bash +kubectl get nodes -o wide +ip route show default +ip route get +kubectl -n kube-system logs --tail=200 | grep 'loss count' | grep -v 'loss count 0' | head +tcpdump -ni host -c 50 +iptables -L FORWARD -n --line-numbers | grep -E 'REJECT|DROP' +``` + +- A default route on a different NIC than on healthy nodes, on nodes joined through another address, is Cause 1. +- Pinger losses that split the cluster into two groups, with masters reachable from both, are Cause 2. +- Requests that leave, answers that are sent, and answers that never arrive at the requester, together with an overlapping subnet, are Cause 3. +- Encapsulated packets that leave one NIC and never reach the other, with clean counters, are Cause 4. +- `temporarily not accessible` in `erebus` with a VIP that answers only some of the time, after `iptables-restore` on a node, is Cause 5. A VIP loop that returns only `200` while the platform still reports `temporarily not accessible` does not match Cause 5. +- `Destination Host Prohibited` with a `REJECT` rule in the `FORWARD` chain is Cause 6. +- If the cluster VIP is a domain name that a node or a Pod cannot resolve, see [Platform console does not open because the cluster VIP domain name cannot be resolved inside the cluster on ACP](Platform_console_does_not_open_because_the_cluster_VIP_domain_name_cannot_be_resolved_inside_the_cluster_on_ACP.md). +- If Geneve packets arrive but are damaged, see [Cross-node Pod traffic fails on the Kube-OVN Geneve overlay because of UDP 6081, TX offload or MTU on ACP](Cross_node_Pod_traffic_fails_on_the_Kube-OVN_Geneve_overlay_because_of_UDP_6081_TX_offload_or_MTU_on_ACP.md). diff --git a/docs/en/solutions/Envoy_Gateway_console_pages_differ_between_two_environments_because_different_gateway_UI_plugins_are_installed_on_ACP.md b/docs/en/solutions/Envoy_Gateway_console_pages_differ_between_two_environments_because_different_gateway_UI_plugins_are_installed_on_ACP.md new file mode 100644 index 000000000..8f4537e11 --- /dev/null +++ b/docs/en/solutions/Envoy_Gateway_console_pages_differ_between_two_environments_because_different_gateway_UI_plugins_are_installed_on_ACP.md @@ -0,0 +1,70 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x +--- + +# Envoy Gateway console pages differ between two environments because different gateway UI plugins are installed on ACP + +## Issue + +The Envoy Gateway pages in the web console look different in two environments, for example development and production: + +- In one environment the route page is under **Network > Route**. It offers only the `HTTPRoute`, `TCPRoute` and `UDPRoute` route types, and it shows no routes. +- In the other environment the route page is under **Network > Gateway > Route**, next to **Gateway** and **Policy** sub-menus. It offers `HTTPRoute`, `TCPRoute`, `UDPRoute`, `TLSRoute` and `GRPCRoute`, and lists routes with their gateway and listener. +- Both environments run the same ACP version (for example 4.2.5), both have the `TLSRoute` and `GRPCRoute` CRDs, and both use the same network mode. + +## Root Cause + +### Cause 1: different gateway UI plugins are installed in the two environments + +The page layout and the route types come from the console plugin that serves the gateway pages, not from the cluster CRDs. **Administrator > Marketplace > Cluster Plugins** shows different plugins in the two environments, for example: + +- **Alauda Container Platform GatewayAPI Plugin** ("User Interface and extended API server for the GatewayAPI"), `v4.3.4`, Installed, in one; +- **Alauda Container Platform Ingress Gateway**, `v4.2.9`, Installed, in the other. + +The GatewayAPI Plugin is the GatewayAPI UI of later ACP versions, opened at **Container Platform > Network > Gateway**, which matches the layout with **Gateway**, **Route** and **Policy** sub-menus. The plugin difference causes the page differences. + +## Resolution + +### Cause 1: different gateway UI plugins are installed in the two environments + +Only when the ACP version and the route CRDs are the same in both environments, and the installed gateway plugins differ. + +The difference is only in the console pages; the Gateway API resources are the same. If both environments must show the same pages, install the same plugin at the same version in both. On ACP 4.2, [How to Install the GatewayAPI UI Plugin for Shared Gateways in ACP 4.2](How_to_Install_GatewayAPI_UI_Which_Could_ShareGateway_IN_ACP_4.2.md) installs the GatewayAPI Plugin on the `global` cluster from **Administrator > Marketplace > Cluster Plugins**, after the plugin package is obtained from the platform maintenance team and uploaded with `violet`. That guide first hides the built-in GatewayAPI page with the `gatewayapi` feature gate: + +> **Warning:** Setting the `gatewayapi` feature gate to `false` hides the built-in GatewayAPI page for every user of the platform until the plugin page is installed. It does not delete Gateway API resources. Have the plugin package for this version on hand, schedule the change with the console users, and record the current value. To roll back, patch `spec.enabled` back to `true`. + +```bash +kubectl get alaudafeaturegates.alauda.io gatewayapi -n cpaas-system -o jsonpath='{.spec.enabled}{"\n"}' +kubectl patch alaudafeaturegates.alauda.io gatewayapi -n cpaas-system --type=merge -p '{"spec":{"enabled":false}}' +``` + +Then follow the guide's upload and install steps, and open **Container Platform > Network > Gateway** in both environments. + +The feature gate step belongs to this ACP 4.2 plugin installation. It is not a way to hide the **Gateway** menu entry on ACP 4.3, where that entry is provided by the `gatewayapi-plugin` cluster plugin pre-installed on the `global` cluster. + +## Diagnostic Steps + +1. Compare the versions and the CRDs first. If they differ, the page difference can come from them instead: + + ```bash + kubectl get crd | grep -E 'tlsroutes|grpcroutes|httproutes' + kubectl get gatewayclass + ``` + +2. Compare the installed plugins in **Administrator > Marketplace > Cluster Plugins** in both environments, or on the `global` cluster. One environment listing the GatewayAPI Plugin and the other the Ingress Gateway plugin, or different versions, is Cause 1: + + ```bash + kubectl get moduleinfo | grep -i -E 'gateway' + kubectl get alaudafeaturegates.alauda.io gatewayapi -n cpaas-system -o jsonpath='{.spec.enabled}{"\n"}' + ``` + +3. A route that exists in the cluster and is missing from one page is not proof that the route is broken. Read it from the cluster: + + ```bash + kubectl get httproute,tlsroute,grpcroute -A + ``` diff --git a/docs/en/solutions/Finding_which_hop_cuts_a_request_through_ALB_on_ACP_truncated_large_responses_60_second_cuts_504_and_SSE_bursts.md b/docs/en/solutions/Finding_which_hop_cuts_a_request_through_ALB_on_ACP_truncated_large_responses_60_second_cuts_504_and_SSE_bursts.md new file mode 100644 index 000000000..85410415b --- /dev/null +++ b/docs/en/solutions/Finding_which_hop_cuts_a_request_through_ALB_on_ACP_truncated_large_responses_60_second_cuts_504_and_SSE_bursts.md @@ -0,0 +1,132 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.10.x + - 3.12.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Finding which hop cuts a request through ALB on ACP: truncated large responses, 60-second cuts, 504 and SSE bursts + +Related articles: [ALB cuts long HTTP or TCP connections at about 120 seconds: proxy timeouts on ACP](ALB_cuts_long_HTTP_or_TCP_connections_at_about_120_seconds_proxy_timeouts_on_ACP.md). + +## Issue + +A request that passes through an Alauda Load Balancer (ALB) and one or more devices in front of it fails, and it is not clear which hop is responsible. Typical reports: + +- A large API response (for example JSON larger than about 1 MB) returns only about half its body through the platform entry address or domain, and the backend logs "broken pipe". A smaller response from the same service is complete, `curl` on the server returns the full body, and the same API works in another environment. The request is to check the ALB for a response-size limit, proxy buffers or timeouts. +- A request is cut at exactly 60 seconds, and the ALB has no 60-second timeout. +- A client receives `504` and asks whether the ALB timed out and which value is best practice. +- Server-sent events (SSE) or chunked output arrive in one burst; a team asks whether `proxy_buffering` or `proxy_cache` can be turned off for one domain. + +The effective timeout of a chain is the smallest one along it, and the hop that closes first is often in front of the ALB. + +## Root Cause + +### Cause 1: a large response is truncated because an HAProxy VIP in front of the ALB resets the connection + +The ALB `nginx` log shows, while the failure is reproduced: + +```text +writev() failed (104: Connection reset by peer) while reading upstream, client: , ... request: "GET /? HTTP/1.0" +``` + +The access-log line for the same request records status `200 200` and a request time of about `0.242` seconds. The platform entry address is a VIP served by HAProxy on two nodes in front of the ALB. The HAProxy layer closed the connection on its side while the ALB was still writing the large response, the ALB write failed with `Connection reset by peer`, and the client received a truncated body. The backend's "broken pipe" is consistent with the ALB giving up the upstream connection after its client side was reset. + +Two facts place the fault in front of the ALB: the client address in the error line is an HAProxy node, and the same request sent directly to the ALB address returns the full response. The ALB is not truncating. ALB timer differences between environments (for example `proxy_read_timeout` 300s against 120s) do not explain it either, because no timer is reached by a request that finishes in a quarter of a second. + +### Cause 2: requests are cut at exactly 60 seconds by a device in front of the ALB + +The ALB `nginx` log shows `epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while sending request to upstream`, with many `client closed connection while waiting for request` lines around the same time. The ALB ConfigMap has no 60-second value (`proxy_connect_timeout 5s;`, `proxy_send_timeout 120s;`, `proxy_read_timeout 120s;`, `keepalive_timeout 65;`). The ALB logged the error passively when it tried to answer: the requester side had already closed. In a chain such as domain mapping, WAF, cloud ELB and then the ALB, the 60-second limit belongs to a device in front of the ALB (a load balancer idle timeout, a WAF or a proxy). + +### Cause 3: a 504 Gateway Timeout, and whether the ALB is the hop that timed out + +An ALB timeout shows as HTTP `504`, usually with the response headers `X-Error: 504` and `Via: alb`, and `upstream timed out` in the ALB error log, close to the configured timeout after the request started. A cloud load balancer or WAF in front can return its own `504` first. The common ALB default is 120 seconds; the effective value depends on the platform version, the ALB type and any load balancer in front, so read it from the configuration loaded in the environment. + +### Cause 4: SSE or chunked output arrives in one burst + +The ALB template already sets `proxy_buffering off`, and `proxy_cache` is unset, so its default `off` applies; `proxy_request_buffering` is also off by default. These ALB parameters are global only; they cannot be set per domain or per Ingress rule, and they do not need to be. When output still arrives in one burst, an outer nginx in front of the ALB is buffering, or the backend buffers the stream before it writes to the socket. + +## Resolution + +### Cause 1: a large response is truncated because an HAProxy VIP in front of the ALB resets the connection + +Only when the ALB log shows the `writev() failed (104: Connection reset by peer)` line with an HAProxy node as the client, and a direct request to the ALB returns the full body: + +1. Hand the evidence to the owner of the HAProxy layer: the ALB error line with its timestamp and client address, the `200 200` access-log line with its request time, and the two `curl` results (through the VIP and direct to the ALB) from the Diagnostic Steps. +2. Have that owner find the matching session in the HAProxy log at the same timestamp and review why HAProxy closed it. The session termination state in the HAProxy log shows which side ended the session and in which phase. +3. If the working environment has a similar layer in front of its ALB, compare that layer's configuration as well. +4. After the HAProxy configuration is changed, repeat the large request through the VIP and compare the downloaded size with the direct request. + +Raising ALB timeouts or changing ALB proxy buffers is not the fix for this signature, because the ALB returns the complete response when called directly. + +### Cause 2: requests are cut at exactly 60 seconds by a device in front of the ALB + +Do not change the ALB. Walk the chain in front of it and find the device with a 60-second limit. Have that device's owner raise it above the request duration, and keep the ALB timer at or above it. + +### Cause 3: a 504 Gateway Timeout, and whether the ALB is the hop that timed out + +There is no universal value. Size the timeout to the longest normal request plus a margin: ordinary HTTP can keep the default, and long queries, large transfers or model inference may be evaluated for 300 to 600 seconds. Avoid unlimited increases: slow connections hold more ALB connections, file descriptors and memory, backend failures surface later, a slow backend or an exhausted connection pool is not fixed, and a shorter timer in front still wins. How to change the ALB timers is in the related article; how to order the timeouts of an external load balancer and the ALB is in [Diagnosing HTTP 504 timeouts behind an external load balancer on ACP](Diagnosing_HTTP_504_timeouts_behind_an_external_load_balancer_on_ACP.md). + +```bash +curl -sk -o /dev/null -D - -w 'total=%{time_total}\n' https://app.example.com/ | grep -iE '^HTTP|x-error|^via|total=' +kubectl -n cpaas-system logs -c nginx --since=30m | grep 'upstream timed out' | tail +``` + +### Cause 4: SSE or chunked output arrives in one burst + +Do not add another `proxy_buffering off` on the ALB or a per-domain buffering setting. Call the ALB VIP or node port directly, bypassing the outer nginx, with the same stream URL: + +- If the direct call streams and the outer URL does not, disable buffering and lengthen the read timeout on that outer nginx. +- If both arrive in one burst, the backend is buffering the stream before it writes to the socket; fix it in the application. + +```bash +kubectl -n cpaas-system get configmap -o yaml | grep -n -E 'proxy_buffering|proxy_cache' +curl -sk -N --resolve app.example.com:443:192.0.2.20 https://app.example.com/ +``` + +## Diagnostic Steps + +1. Search the ALB log for the close signatures. Each points to a different cause: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep + kubectl -n cpaas-system logs -c nginx --since=1h | grep -E 'Connection reset by peer|writev\(\) failed|prematurely closed|upstream timed out' | tail + ``` + + - `writev() failed (104: Connection reset by peer) while reading upstream`: the ALB's client reset the connection while the response was being written (Cause 1). + - `client prematurely closed connection` (status `499`) at a fixed duration such as 60 seconds: a device in front closed on a timer (Cause 2). + - `upstream timed out`: an ALB proxy timer expired (Cause 3, and the related article). + +2. Compare the client address of the error line with the addresses of the nodes in front of the ALB, and check which process listens and which node holds the VIP: + + ```bash + ip -o addr show | grep '' + ss -ltnp | grep -i haproxy + ``` + +3. Send the same request through the entry address and directly to the ALB, and compare the downloaded size and time. `--resolve` keeps the host name and TLS SNI the same in both requests. In this example `198.51.100.10` is the entry VIP and `192.0.2.20` is the ALB address: + + ```bash + curl -sk -o /tmp/via-vip.json -w 'code=%{http_code} size=%{size_download} total=%{time_total}\n' \ + --resolve app.example.com:443:198.51.100.10 'https://app.example.com/?' + curl -sk -o /tmp/direct-alb.json -w 'code=%{http_code} size=%{size_download} total=%{time_total}\n' \ + --resolve app.example.com:443:192.0.2.20 'https://app.example.com/?' + ``` + + A smaller `size` through the VIP with a complete `size` direct to the ALB is Cause 1. If the direct request is also truncated, the cut is at the ALB or behind it; check the ALB timers and the backend. When `total` through the entry address matches a front device's timeout and the direct call lasts longer, raise that device (Cause 2). + +4. To rule out ALB timers, read them in the ConfigMap with the same name as the ALB. A request time far below every value printed means no ALB timer was involved: + + ```bash + kubectl -n cpaas-system get configmap -o yaml | grep -n -E 'proxy_(connect|send|read)_timeout|proxy_timeout|proxy_buffering' + ``` + +5. For a `504`, a response with `X-Error: 504` and `Via: alb` and an `upstream timed out` line in the ALB log at the same time put the timeout on the ALB; a `504` without `Via: alb` came from a hop in front (Cause 3). diff --git a/docs/en/solutions/Ingress_NGINX_controller_serves_the_fake_certificate_ignores_an_Ingress_or_rejects_Ingress_saves_on_ACP.md b/docs/en/solutions/Ingress_NGINX_controller_serves_the_fake_certificate_ignores_an_Ingress_or_rejects_Ingress_saves_on_ACP.md new file mode 100644 index 000000000..d6d246883 --- /dev/null +++ b/docs/en/solutions/Ingress_NGINX_controller_serves_the_fake_certificate_ignores_an_Ingress_or_rejects_Ingress_saves_on_ACP.md @@ -0,0 +1,229 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# Ingress NGINX controller serves the fake certificate, ignores an Ingress, or rejects Ingress saves on ACP + +## Issue + +These problems concern the Alauda build of Ingress NGINX Controller, deployed by `ingress-nginx-operator` from an `IngressNginx` resource (`ingressnginxes.ingress-nginx.alauda.io`), with IngressClass `ingress-nginx-operator-`. Typical reports: + +- A certificate was attached to a domain on the platform domain page, and services using that domain show a browser warning with the certificate `Kubernetes Ingress Controller Fake Certificate`, while another service whose Secret is in its own namespace works. +- An Ingress with its own TLS certificate still gets the controller's default certificate. +- Applications in one project cannot use an Ingress NGINX gateway from another project. +- An IngressClass limited with **Ingress Scope** is still selectable in other projects. +- After an Ingress was created, requests return `404` and the controller has no access log for them. +- Requests hang for about two minutes before they succeed, and the error log shows `connect() failed (110: Operation timed out) while connecting to upstream`. +- Saving an Ingress fails on ACP 4.0 when the Higress gateway is used. +- A GitLab Ingress with a `configuration-snippet` is rejected with `failed calling webhook "inject-label.cpaas.io": webhook returned response.patchType but not response.patch`. +- A gray-release (traffic-split) Ingress that repeats the host and path of the primary Ingress is rejected by the admission webhook of the platform component `metis`. + +## Root Cause + +### Cause 1: the domain default certificate option is used with Ingress NGINX + +A wildcard domain created under **Platform Management > Network Management > Domain** keeps its certificate Secret in `cpaas-system`. When an Ingress of class `ingress-nginx-operator-` selects the "domain default certificate" option, the generated YAML has `spec.tls: []` and only the annotation `alb.networking.cpaas.io/tls: =cpaas-system/`. That option, and the platform-assigned certificate in general, is designed for the ALB load balancer. Ingress NGINX lets an Ingress reference only a TLS Secret in the same namespace, so the binding fails and the controller serves its fake certificate (CN `Kubernetes Ingress Controller Fake Certificate`, O `Acme Co`). Plain `http://` access shows no certificate, which can hide the problem; the certificates themselves can be identical and valid. + +### Cause 2: several Ingress objects share one host and only one has TLS + +The Ingress with the dedicated certificate is correct (for example two host rules and a `tls` section listing both hosts with one `secretName`), but other Ingress objects use the same host names for other paths and are HTTP-only, without TLS. That conflicting configuration for the same host makes the controller's default certificate win for the domain. + +### Cause 3: the IngressNginx instance is scoped to other namespaces + +By default an Ingress NGINX instance handles Ingress resources in all namespaces that either name its IngressClass or name no IngressClass. `.spec.controller.scope.namespaceSelector` restricts it, for example `namespaceSelector: 'cpaas.io/project=demo'` limits it to namespaces labeled `cpaas.io/project=demo`. Ingress objects from other projects are then ignored. + +### Cause 4: the console lists a scoped IngressClass in every project + +The console does not filter the IngressClass list by project, so a class limited with **Ingress Scope** appears everywhere. The scope is enforced at runtime: Ingress objects from projects outside the scope are not served. + +### Cause 5: an invalid Ingress makes requests return 404 with no access log + +An invalid Ingress was added, for example one with `nginx.ingress.kubernetes.io/configuration-snippet: | gzip off;` next to `nginx.ingress.kubernetes.io/backend-protocol: HTTP` and `nginx.ingress.kubernetes.io/proxy-body-size: "0"`. Snippet annotations are disabled by default in newer Ingress NGINX versions (Cause 8). Removing the invalid Ingress is the fix. + +### Cause 6: requests wait about 131 seconds for a stale upstream, then succeed + +The controller error log shows: + +```text +[error] ... connect() failed (110: Operation timed out) while connecting to upstream ... request: "GET HTTP/1.1", upstream: "http://:80" +``` + +The access log line for the same request lists two upstreams, the old one on port 80 and the expected one on port 8080, with statuses `502, 200` and upstream times of about `131.015, 0.003`. + +An Endpoints object was applied by hand for a Service that has a selector (it carries `kubectl.kubernetes.io/last-applied-configuration`), and a Pod of another service had been associated with the Service by mistake, which produced an EndpointSlice owned by the Service with a `ready: true` endpoint whose `targetRef` is a Pod of a different service. `kube-controller-manager` logs: + +```text +"Error mirroring EndpointSlices for Endpoints, retrying" logger="endpointslice-mirroring-controller" key="/" err="error(s) deleting 1/1 EndpointSlices for / Endpoints, including: endpointslices.discovery.k8s.io \"-\" not found" +``` + +The hand-made Endpoints disrupt the native controller's takeover; its cleanup gets stuck on a slice that no longer exists, the stale slice stays, and the controller keeps trying the old upstream first. + +### Cause 7: saving an Ingress fails on 4.0 when the Higress gateway is used + +On ACP 4.0, the Ingress check webhook `check-ingress` of the `metis` Deployment in `cpaas-system` interferes with saving Ingress resources used by Higress. The same operation works on 3.16. + +### Cause 8: an Ingress with a configuration-snippet is rejected with an inject-label.cpaas.io webhook error + +Newer Ingress NGINX versions disable snippet annotations by default. A Helm-deployed GitLab whose `gitlab-webservice-default` Ingress gains a multi-line `nginx.ingress.kubernetes.io/configuration-snippet` (for WebSocket) is not served, although the controller log shows `"Found valid IngressClass"` for it, and admission fails with `failed calling webhook "inject-label.cpaas.io": webhook returned response.patchType but not response.patch`. + +### Cause 9: a gray-release Ingress with the same host and path as the primary Ingress is rejected by metis + +The `check-ingress` webhook of `metis` enforces a duplicate host and path check on Ingress resources. A gray-release Ingress (one that carries the ingress-nginx gray-release annotations, for example routing requests with the header `runenv: gray-ack` to the new version) shares host and path with the primary Ingress by design, so the check rejects it. + +From ACP 4.3, the `metis` Deployment carries `--disable-webhook=check-ingress` automatically. + +## Resolution + +### Cause 1: the domain default certificate option is used with Ingress NGINX + +Create the certificate Secret in the business namespace and reference it from the Ingress `spec.tls`; do not use the domain default certificate option with Ingress NGINX. + +```bash +kubectl -n create secret tls --cert= --key= +``` + +```yaml +spec: + ingressClassName: ingress-nginx-operator- + tls: + - hosts: + - app.example.com + secretName: +``` + +### Cause 2: several Ingress objects share one host and only one has TLS + +Change every Ingress that uses the domain to HTTPS and add the same TLS certificate to each of them. Find them with: + +```bash +kubectl get ingress -A -o json | jq -r '.items[] | select([.spec.rules[]?.host] | index("app.example.com")) | .metadata.namespace + "/" + .metadata.name + " tls=" + ((.spec.tls // []) | length | tostring)' +``` + +### Cause 3: the IngressNginx instance is scoped to other namespaces + +Check the scope of the instance. To serve Ingress objects from all projects, use an instance without `controller.scope.namespaceSelector`; otherwise adjust the selector to include the other project's namespaces. + +```bash +kubectl -n get ingressnginxes.ingress-nginx.alauda.io -o jsonpath='{.spec.controller.scope}{"\n"}' +``` + +```yaml +apiVersion: ingress-nginx.alauda.io/v1 +kind: IngressNginx +metadata: + name: +spec: + controller: + nodeSelector: + kubernetes.io/os: linux + replicaCount: 1 +``` + +### Cause 4: the console lists a scoped IngressClass in every project + +No change is needed for isolation; it works at runtime. Selecting the class from a project outside the scope does not get that Ingress served. + +### Cause 5: an invalid Ingress makes requests return 404 with no access log + +Find the invalid Ingress in the controller log or with `kubectl describe`, and remove it (or fix it, for example as in Cause 8 for snippets). + +> **Warning:** Deleting the Ingress removes its hosts from the controller. Do it only when the controller log or events name this Ingress as invalid, and save its YAML first. To roll back, apply the saved YAML after fixing it. + +```bash +kubectl -n logs deploy/-ingress-nginx-controller --since=30m | grep -iE 'error|invalid' | tail +kubectl -n get ingress -o yaml > /tmp/.yaml +kubectl -n delete ingress +``` + +### Cause 6: requests wait about 131 seconds for a stale upstream, then succeed + +Only when the access log shows the two-upstream `502, 200` pattern and an EndpointSlice of the Service lists an address that should not be there, delete that residual EndpointSlice. The requests then stop trying the old upstream. Avoid hand-made Endpoints for Services that have a selector. + +> **Warning:** Only delete a slice that contains the stale address. Deleting the wrong slice removes healthy endpoints until the controller recreates them. Save the slice YAML first; to roll back, apply it. + +```bash +kubectl -n get endpointslices -o wide +kubectl -n get endpoints -o yaml +kubectl -n kube-system logs --since=1h | grep -i 'mirroring EndpointSlices' +kubectl -n get endpointslices -o yaml > /tmp/.yaml +kubectl -n delete endpointslices +``` + +### Cause 7: saving an Ingress fails on 4.0 when the Higress gateway is used + +Add `check-ingress` to the `--disable-webhook` startup argument of the `metis` Deployment in `cpaas-system`, as in Cause 9. + +### Cause 8: an Ingress with a configuration-snippet is rejected with an inject-label.cpaas.io webhook error + +Enable snippet annotations and raise the allowed annotation risk level in the controller's global configuration (`spec.controller.config` of the IngressNginx resource): + +```yaml +spec: + controller: + config: + allow-snippet-annotations: "true" + annotations-risk-level: "Critical" +``` + +> **Warning:** Snippet annotations let anyone who can edit an Ingress inject raw nginx configuration into the shared controller, which affects every host it serves. Do this only when Ingress write access is limited to trusted users. To roll back, remove both keys from `spec.controller.config`. + +### Cause 9: a gray-release Ingress with the same host and path as the primary Ingress is rejected by metis + +Long-term fix: upgrade to ACP 4.3, where the argument is set automatically. On an earlier release, add `check-ingress` to the `--disable-webhook` argument of the `metis` Deployment. If the container already has a `--disable-webhook` argument, add `check-ingress` to it rather than adding a second flag. Only the Ingress duplicate host and path check is disabled. + +```bash +kubectl -n cpaas-system get deploy metis -o jsonpath='{.spec.template.spec.containers[*].args}{"\n"}' +``` + +> **Warning:** Only when the error comes from the `metis` webhook and, for a gray release, the rejected Ingress intentionally shares host and path with another Ingress. Impact: the platform's Ingress check is turned off for every Ingress in the cluster, so accidental duplicate hosts and paths are no longer caught at admission, and the edit rolls the `metis` Pods. Save the Deployment YAML first. To roll back, remove `check-ingress` from `--disable-webhook` again and wait for the rollout. + +```bash +kubectl -n cpaas-system get deploy metis -o yaml > /tmp/metis-deploy.before.yaml +kubectl -n cpaas-system edit deployments.apps metis +kubectl -n cpaas-system rollout status deploy/metis +``` + +```yaml +args: + - --disable-webhook=check-ingress +``` + +Then create or save the Ingress again. After an upgrade, read the arguments again to confirm the setting. + +## Diagnostic Steps + +1. Which certificate is served, and where the Ingress takes it from. `CN=Kubernetes Ingress Controller Fake Certificate` with an empty `spec.tls` is Cause 1; a dedicated `spec.tls` on one Ingress and HTTP-only Ingress objects for the same host is Cause 2: + + ```bash + openssl s_client -connect 192.0.2.10:443 -servername app.example.com /dev/null | openssl x509 -noout -subject -issuer + kubectl -n get ingress -o jsonpath='{.spec.tls}{" "}{.metadata.annotations}{"\n"}' + ``` + +2. Instances, their scope and classes (Causes 3 and 4): + + ```bash + kubectl get ingressnginxes.ingress-nginx.alauda.io -A + kubectl get ingressclass + ``` + +3. Upstream attempts per request (Cause 6). Two upstream addresses and two statuses in one access-log line mean a retry after a dead endpoint: + + ```bash + kubectl -n logs deploy/-ingress-nginx-controller --since=30m | grep 'Operation timed out' | tail -n 3 + kubectl -n get endpointslices -l kubernetes.io/service-name= -o yaml | grep -n -E 'name:|addresses' + ``` + +4. Admission failures on save show the webhook name in the API error (Causes 7, 8 and 9). List the webhooks and the `metis` arguments, and for a gray release confirm the duplicate host and path: + + ```bash + kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations | grep -iE 'ingress|inject-label|metis' + kubectl -n cpaas-system get deploy metis -o jsonpath='{.spec.template.spec.containers[*].args}{"\n"}' | grep -o 'disable-webhook=[^"]*' + kubectl -n get ingress -o custom-columns='NAME:.metadata.name,HOSTS:.spec.rules[*].host,PATHS:.spec.rules[*].http.paths[*].path' + ``` diff --git a/docs/en/solutions/Intermittent_DNS_failures_in_Pods_from_a_new_node_NIC_flapping_the_KubeDNS_alert_or_CoreDNS_upstream_timeouts_on_ACP.md b/docs/en/solutions/Intermittent_DNS_failures_in_Pods_from_a_new_node_NIC_flapping_the_KubeDNS_alert_or_CoreDNS_upstream_timeouts_on_ACP.md new file mode 100644 index 000000000..6ad66e2a1 --- /dev/null +++ b/docs/en/solutions/Intermittent_DNS_failures_in_Pods_from_a_new_node_NIC_flapping_the_KubeDNS_alert_or_CoreDNS_upstream_timeouts_on_ACP.md @@ -0,0 +1,153 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# Intermittent DNS failures in Pods from a new node, NIC flapping, the KubeDNS alert or CoreDNS upstream timeouts on ACP + +Related articles: [CoreDNS fails after a node restart or lookups time out when a CoreDNS node is shut down on ACP](CoreDNS_fails_after_a_node_restart_or_lookups_time_out_when_a_CoreDNS_node_is_shut_down_on_ACP.md). + +## Issue + +Cluster DNS comes and goes, and the cause is outside the Corefile. Typical reports: + +- From a node that was just added, access to a Service by name works one moment and fails the next. Older nodes are fine. +- Many applications log `java.net.UnknownHostException` at the same time, together with lost heartbeats (`DiscoveryClient was unable to send heartbeat`), ZooKeeper `SessionExpiredException` and database `Communications link failure`. Node monitoring has gaps in the same window. +- An alert says `KubeDNS(:9153)` is unavailable, and it clears by itself almost immediately. +- On a global cluster whose CNI is flannel with galaxy, every console page returns `504` or an internal server error page (`服务器内部错误`), `GET /console-platform/api/v1/token/login` returns `500`, and one `tke-platform-api` replica in `cpaas-system` is in `CrashLoopBackOff`. + +CoreDNS on ACP 4.x is the `coredns` Deployment in `kube-system` (Pod label `k8s-app=kube-dns`). The CoreDNS Pods listen on `1053` inside the Pod network, use the node `/etc/resolv.conf` as their upstream list, and expose metrics on `9153`. + +## Root Cause + +### Cause 1: Service names fail intermittently only from a newly added node + +After a node is added, lookups of Service names from that node fail intermittently while older nodes are fine. CoreDNS has not yet recognized the newly added node; restarting CoreDNS restores normal access. If the new node cannot open `1053` to every CoreDNS Pod, the cause is a firewall between the node subnets instead (see Diagnostic Steps). + +### Cause 2: physical NICs flapping cause UnknownHostException bursts with connection drops + +When the physical NICs of the cluster machines go up and down, each flap interrupts DNS queries and every other connection that crosses the NIC. Several systems lose name resolution and connections in the same minutes, monitoring for the affected hosts shows gaps in CPU, memory and Pod count, and nodes drop out of the cluster count. The CNI and the operating system show nothing abnormal. + +A long node `search` list makes each interruption worse: with six suffixes on the node, several of them cluster-style, every short name is tried with each suffix in turn. Removing the `search` entries from the node `/etc/resolv.conf` stops the DNS failures, but connection drops continue until the NICs or links are repaired. + +### Cause 3: the KubeDNS :9153 unavailable alert clears by itself + +The alert names the CoreDNS metrics target (`:9153`). By the time anyone looks, it has cleared and CoreDNS is serving. A short network fluctuation can make a single query or scrape time out (the CoreDNS log for the window can show a request to port `53` timing out). A custom alert rule whose trigger timing does not match the platform's own alert behavior fires on a single missed scrape and reports every such blip. + +### Cause 4: CoreDNS cannot reach its upstream on a flannel and galaxy cluster + +One `tke-platform-api` replica in `cpaas-system` is `CrashLoopBackOff` (`0/1`, `Exit Code: 2`, more than 1,800 restarts) with events `Readiness probe failed: Get "https://:9443/healthz/ping": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)`. The CoreDNS log repeats: + +```text +[ERROR] plugin/errors: 2 HINFO: read udp :->:53: i/o timeout +``` + +The `HINFO` query is the probe CoreDNS sends at startup to detect forwarding loops, so its timeout means the CoreDNS Pod cannot reach the upstream through the Pod network. Node load is normal. The cluster network is flannel with galaxy (no `ovn` or `calico` Pods), and the `flannel` and `galaxy-daemonset` Pods in `kube-system` all show `1/1 Running`. Restarting the flannel and galaxy Pods restores the Pod network and platform access; what was wrong in the Pod network before the restart was not determined. + +## Resolution + +### Cause 1: Service names fail intermittently only from a newly added node + +Only when the failures started with the new node, other nodes resolve normally, and the `1053` check in Diagnostic Steps passes. Restart CoreDNS. + +> **Warning:** A rollout restart replaces the CoreDNS Pods one by one; lookups can stall briefly while a Pod is not Ready. Make sure more than one CoreDNS replica is Ready. No rollback is needed; the Deployment spec does not change. + +```bash +kubectl -n kube-system rollout restart deploy/coredns +kubectl -n kube-system rollout status deploy/coredns +kubectl exec -n -- nslookup ..svc.cluster.local +``` + +Run the last command several times; every attempt should answer. + +### Cause 2: physical NICs flapping cause UnknownHostException bursts with connection drops + +Only when kernel logs or the switch show the node NICs going up and down in the failure window. + +1. Repair the flapping NICs or links on the underlay. Redundant (dual-link) uplinks work as a stopgap until the repair. +2. As a DNS-side mitigation, comment out the `search` line in the node `/etc/resolv.conf` on all nodes, then recreate the affected Pods so their resolver file no longer carries those suffixes. If NetworkManager or DHCP writes the file, remove the search domains at that source too. + + > **Warning:** Removing the node `search` line changes how short names resolve for host processes and for every Pod created afterwards; names that relied on a site suffix stop expanding. Back up the file on each node and confirm the applications use full names for site domains. To roll back, restore the backup and recreate the Pods. + + ```bash + cp -a /etc/resolv.conf /root/resolv.conf.before + sed -i 's/^search /# search /' /etc/resolv.conf + grep -n search /etc/resolv.conf + ``` + +### Cause 3: the KubeDNS :9153 unavailable alert clears by itself + +If the CoreDNS Pods did not restart and the CoreDNS logs show no sustained errors in the window, treat the alert as a transient and give the custom rule a one-minute threshold, so that it fires only when the target stays down for one minute. In a Prometheus rule that is the `for` field. + +> **Warning:** A longer threshold delays real alerts by the same amount. Save the rule first and agree on the threshold with the alert owner. To roll back, apply the saved rule. + +```bash +kubectl get prometheusrule -A | grep -iE 'dns|coredns' +kubectl -n get prometheusrule -o yaml > /tmp/.before.yaml +kubectl -n edit prometheusrule +``` + +```yaml +- alert: + expr: + for: 1m +``` + +If the alert keeps firing with a one-minute threshold, it is no longer a timing artifact: collect CoreDNS logs and node network counters for the window and check Cause 2. + +### Cause 4: CoreDNS cannot reach its upstream on a flannel and galaxy cluster + +Only when the CoreDNS log shows the upstream `i/o timeout`, node resources are normal, and the cluster CNI is flannel with galaxy. Restart the flannel and galaxy Pods, then check CoreDNS and `tke-platform-api` again. + +> **Warning:** Restarting the CNI Pods interrupts Pod networking on each node while its Pods are recreated, and new Pods cannot get networking meanwhile. Keep console or SSH access to the masters that does not depend on the platform UI, and use a window in which a short Pod network interruption is acceptable. No rollback is needed; the DaemonSets recreate the Pods. + +```bash +kubectl get po -A -o wide | grep -E 'ovn|calico' +kubectl -n kube-system get ds | grep -E 'flannel|galaxy' +kubectl -n kube-system rollout restart ds/ ds/ +kubectl -n kube-system rollout status ds/ +kubectl -n kube-system rollout status ds/ +kubectl -n kube-system logs --tail=20 +kubectl -n cpaas-system get pod | grep tke-platform-api +``` + +After the restart the platform is accessible again. + +## Diagnostic Steps + +1. For a new node, query a Service FQDN repeatedly from a Pod on that node, and check that the node reaches every CoreDNS Pod on `1053`. Failures only from that node with `1053` open are Cause 1. `1053` closed to some replicas points at a firewall between node subnets; see [New node cannot reach CoreDNS or the API server because a firewall drops tunnel traffic between nodes on ACP](New_node_cannot_reach_CoreDNS_or_the_API_server_because_a_firewall_drops_tunnel_traffic_between_nodes_on_ACP.md). + + ```bash + for i in $(seq 1 20); do kubectl exec -n -- nslookup ..svc.cluster.local >/dev/null 2>&1 && echo ok || echo fail; done + kubectl -n kube-system get pod -l k8s-app=kube-dns -o wide + nc -vz -w 3 1053 + ``` + +2. For bursts of `UnknownHostException` with connection drops, look for link flaps and test Pod-to-Pod across nodes. A capture on the business node narrows which hop drops (Cause 2): + + ```bash + journalctl -k --since "" --until "" | grep -iE 'link (is )?(up|down)' + ip -s link show + kubectl exec -n -- ping -c 5 + tcpdump -i host and port -C 100 -W 5 -w /tmp/capture.pcap + ``` + +3. For the `:9153` alert, compare the alert time with CoreDNS restarts and logs, and read the rule that fired (Cause 3): + + ```bash + kubectl -n kube-system get pod -l k8s-app=kube-dns -o wide + kubectl -n kube-system logs -l k8s-app=kube-dns --since=1h | grep -iE 'timeout|error' + ``` + +4. For console errors with a crash-looping `tke-platform-api`, read the CoreDNS log and identify the CNI. `HINFO ... i/o timeout` on a flannel and galaxy cluster is Cause 4: + + ```bash + kubectl -n kube-system logs -l k8s-app=kube-dns --tail=50 | grep -E 'HINFO|i/o timeout' + kubectl -n kube-system get pod -o wide | grep -E 'flannel|galaxy|ovn|calico' + kubectl top nodes + ``` diff --git a/docs/en/solutions/Kube-OVN_fixed_IP_rejected_or_no_address_allocated_because_a_stale_IP_object_or_logical_switch_port_holds_it_on_ACP.md b/docs/en/solutions/Kube-OVN_fixed_IP_rejected_or_no_address_allocated_because_a_stale_IP_object_or_logical_switch_port_holds_it_on_ACP.md new file mode 100644 index 000000000..69f75a70b --- /dev/null +++ b/docs/en/solutions/Kube-OVN_fixed_IP_rejected_or_no_address_allocated_because_a_stale_IP_object_or_logical_switch_port_holds_it_on_ACP.md @@ -0,0 +1,147 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.16.x + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# Kube-OVN fixed IP rejected or no address allocated because a stale IP object or logical switch port holds it on ACP + +## Issue + +One of the following happens on a Kube-OVN cluster: + +- A business Pod with a fixed IP stays in `ContainerCreating`, and its sandbox event says `request ip return 500 no address allocated to pod`. The used-IP page lists the address as used by two containers, one of which no longer exists. +- When a container is created with a fixed IP, the console says the fixed IP cannot be used, and the address turns out to belong to a CronJob Pod that finished long ago. +- In a namespace bound to two underlay subnets, Pods take addresses from the older subnet until it is nearly full and then stay stuck with `NoAvailableAddress`, while the newer subnet has more than a thousand free addresses: + + ```text + FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "": plugin type="kube-ovn" failed (add): RPC failed; request ip return 500 no address allocated to pod / provider ovn, please see kube-ovn-controller logs to find errors + ``` + +In each case the address is held by Kube-OVN bookkeeping, not by a running Pod: an `ips.kubeovn.io` object (IP object) or an OVN logical switch port left behind by a deleted Pod, or a u2o interconnection IP object that was never created for an old subnet. If `kubectl get pod -A -o wide | grep -w
` returns a running Pod, the address is really in use and this article does not apply. + +## Root Cause + +### Cause 1: a deleted Pod's IP object and logical switch port remain on the fixed address + +A defect in the automatic garbage collection (GC) of some newer Kube-OVN versions leaves the IP object and the logical switch port of a deleted Pod on its fixed address. Environments upgraded from older versions are more likely to hit it; later versions improve the GC logic. + +The new Pod then fails with `request ip return 500 no address allocated to pod`, and the controller logs report the address pool as exhausted. The used-IP view first lists only one container on the address; after `kube-ovn-controller` restarts, it lists two, one of which no longer exists. The OVN southbound database is healthy. Deleting the stale IP object lets the new Pod start; the used-IP page shows two users until the stale logical switch port is also deleted and `kube-ovn-controller` is restarted. + +### Cause 2: the console rejects the fixed IP because IP objects of finished CronJob Pods remain + +`kubectl get pod -A -o wide | grep -w ` returns nothing, but `kubectl get ip | grep ` returns an IP object named after a historical CronJob Pod (`.`). One address can carry many such objects (dozens from the same CronJob) next to the valid object of the Pod that really uses it. + +On Kube-OVN `v1.12.35`, these objects are residue from before the current controller started; cleanup of recent CronJob Pods is normal. That version does not clean up existing residue: its cleanup deletes the OVN port first and then the IP object, and if the IP object delete fails it only logs the error and still releases the IPAM address without requeuing the object. The periodic GC iterates only existing ports and does not check IP object owners. That is why the objects persist and why several of them can record one address. The residue affects only the console, which reads IP objects rather than subnet status. + +### Cause 3: NoAvailableAddress in a namespace bound to two underlay subnets because the older subnet has no u2o IP object + +A namespace is bound to an older underlay subnet created before an upgrade (for example a `/23` with 508 used and 1 available) and a newer one (for example a `/22` with 1 used and 1020 available). Pods take addresses from the older subnet first, and once it runs out the remaining Pods stay stuck instead of using the newer one; their events repeat `AcquireAddressFailed: NoAvailableAddress`. + +`kubectl get ip | grep u2o` shows a u2o interconnection IP object for the newer subnet and none for the older one. The older subnet was carried over from an earlier version, and its u2o IP object was not created automatically during the upgrade. Its u2o address usage records are incomplete, and when the subnet is close to exhaustion the allocation state becomes inconsistent. + +This is different from a subnet whose u2o address is the default `.2`, which has no IP object by design. Here a sibling subnet of the same cluster has one, the subnet without it predates an upgrade, and allocation fails as it fills. + +## Resolution + +### Cause 1: a deleted Pod's IP object and logical switch port remain on the fixed address + +1. Prove that the Pod named by the IP object and the port is gone: + + ```bash + kubectl get ip -o wide | grep + kubectl get pod -A -o wide | grep -w + kubectl ko nbctl list logical_switch_port | grep -B3 -A8 + kubectl ko sb status + ``` + +2. Delete the stale IP object and logical switch port, then restart `kube-ovn-controller`. The logical switch port of a Pod is named `.`. + + > **Warning:** Deleting an IP object frees the address for the next allocation, and `lsp-del` removes the port from the OVN northbound database at once. If the Pod named in them is in fact running, it loses its network and the address can be handed out twice. Only when the Pod named in the IP object and the port does not exist. Save both objects first. There is no undo other than recreating the Pod. Restarting `kube-ovn-controller` pauses allocation cluster-wide until the new leader is ready. + + ```bash + kubectl get ip -o yaml > /root/ip-.yaml + kubectl delete ip + kubectl ko nbctl list logical_switch_port . > /root/lsp-.txt + kubectl ko nbctl lsp-del . + kubectl -n kube-system delete pod -l app=kube-ovn-controller + ``` + + After the delete the new Pod starts, and after the controller restart the used-IP page shows one user again. A Kube-OVN release with the improved GC prevents a recurrence. + +### Cause 2: the console rejects the fixed IP because IP objects of finished CronJob Pods remain + +1. List every IP object whose Pod does not exist. Node IP objects, u2o interconnection objects and VIP objects have no Pod by design; they are printed for review and must be left alone. + + ```bash + kubectl get ip -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.namespace}{"\t"}{.spec.podName}{"\t"}{.spec.ipAddress}{"\n"}{end}' > /root/ip-objects.txt + while IFS=$'\t' read -r name ns pod addr; do + [ -z "$ns" ] && { echo "review (no namespace): $name $addr"; continue; } + kubectl -n "$ns" get pod "$pod" >/dev/null 2>&1 || echo "orphan: $name $ns/$pod $addr" + done < /root/ip-objects.txt + kubectl get pod -A -o wide | grep -w + ``` + +2. Delete the orphan objects after review. + + > **Warning:** Deleting an IP object whose Pod is still running, or is being created at that moment, lets the address be allocated twice. Only delete objects printed as `orphan` whose Pod is confirmed absent, and keep `/root/ip-objects.txt` plus a YAML copy of each object. The delete is irreversible; recreating the Pod is the only way to get a valid object back. + + ```bash + kubectl get ip -o yaml > /root/ip-.yaml + kubectl delete ip + ``` + +3. Retry the fixed IP in the console. A CronJob whose Pods are cleaned up normally on the current version needs no change. + +### Cause 3: NoAvailableAddress in a namespace bound to two underlay subnets because the older subnet has no u2o IP object + +1. Confirm the pattern: both subnets list the namespace, the older one is nearly full, and only the newer one has a u2o IP object. + + ```bash + kubectl get subnet -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidrBlock,VLAN:.spec.vlan,NS:.spec.namespaces,U2O:.spec.u2oInterconnection,U2OIP:.status.u2oInterconnectionIP,USED:.status.v4usingIPs,AVAIL:.status.v4availableIPs' + kubectl get ip | grep u2o-interconnection + kubectl -n kube-system logs deploy/kube-ovn-controller --tail=3000 > /root/kube-ovn-controller.log + ``` + +2. Export the existing object as a template, and read the values for the older subnet: its u2o address from the Subnet status, its UID for the owner reference, and the MAC of its u2o router port from the northbound database. + + ```bash + kubectl get ip u2o-interconnection.ovn-cluster. -o yaml > /root/u2o-ip-template.yaml + kubectl get subnet -o jsonpath='{.status.u2oInterconnectionIP} {.metadata.uid}{"\n"}' + kubectl ko nbctl show ovn-cluster | grep -B2 -A2 '' + ``` + +3. In a copy of the template, change every field that names the newer subnet, its address, its MAC or its UID to the older subnet's values, and remove `metadata.uid`, `metadata.resourceVersion`, `metadata.creationTimestamp` and `status`. Use the template's name pattern (`u2o-interconnection.ovn-cluster.`). Then create the object and check its ownership and the subnet counters. + + > **Warning:** An IP object with a wrong address or MAC records a false allocation in the subnet, and other Pods can then collide with the real u2o address or be refused a free one. Only when the sibling subnet's object exists and this subnet's is missing. Keep the template and the new file. To roll back, delete the object you created. + + ```bash + cp /root/u2o-ip-template.yaml /root/u2o-ip-.yaml + vi /root/u2o-ip-.yaml + kubectl apply -f /root/u2o-ip-.yaml + kubectl get ip u2o-interconnection.ovn-cluster. -o yaml | grep -A6 ownerReferences + kubectl get subnet -o jsonpath='{.status.u2oInterconnectionIP} {.status.v4usingIPs} {.status.v4availableIPs}{"\n"}' + ``` + +4. Recreate one stuck Pod and watch its events. If Pods still do not fall back to the newer subnet when the older one is full, pin the workload to the newer subnet with the Pod annotation `ovn.kubernetes.io/logical_switch: `. + +## Diagnostic Steps + +```bash +kubectl describe pod -n | grep -A3 -E 'FailedCreatePodSandBox|AcquireAddressFailed' +kubectl get ip -o wide | grep
+kubectl get pod -A -o wide | grep -w
+kubectl ko nbctl list logical_switch_port | grep -B3 -A8
+kubectl get ip | grep u2o-interconnection +``` + +- An IP object or logical switch port for the address whose Pod does not exist, with `no address allocated to pod` on the new Pod, is Cause 1. +- A console rejection of a fixed IP whose IP object is named after a finished CronJob Pod, often with several objects for one address, is Cause 2. +- `NoAvailableAddress` in a namespace bound to two underlay subnets, with a u2o IP object for only one of them, is Cause 3. +- If `kubectl get pod -A -o wide | grep -w
` returns a running Pod, the address is really in use by that Pod. diff --git a/docs/en/solutions/Kube-OVN_underlay_Pods_fail_with_resolve_MAC_address_timeout_or_bridge_network_InitOVSBridgeFailed_on_ACP.md b/docs/en/solutions/Kube-OVN_underlay_Pods_fail_with_resolve_MAC_address_timeout_or_bridge_network_InitOVSBridgeFailed_on_ACP.md new file mode 100644 index 000000000..1380b6adb --- /dev/null +++ b/docs/en/solutions/Kube-OVN_underlay_Pods_fail_with_resolve_MAC_address_timeout_or_bridge_network_InitOVSBridgeFailed_on_ACP.md @@ -0,0 +1,192 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# Kube-OVN underlay Pods fail with resolve MAC address timeout or bridge network InitOVSBridgeFailed on ACP + +Related articles: [Node, API and VIP access lost after creating a Kube-OVN ProviderNetwork or underlay subnet on ACP](Node_API_and_VIP_access_lost_after_creating_a_Kube-OVN_ProviderNetwork_or_underlay_subnet_on_ACP.md); [All charts stay ChartDownloading on a new Kube-OVN cluster because ovn-default was created as overlay on ACP](All_charts_stay_ChartDownloading_on_a_new_Kube-OVN_cluster_because_ovn-default_was_created_as_overlay_on_ACP.md); [Adding a node to a Kube-OVN underlay cluster in the web UI fails with network interface not found on ACP](Adding_a_node_to_a_Kube-OVN_underlay_cluster_in_the_web_UI_fails_with_network_interface_not_found_on_ACP.md). + +## Issue + +Pods on a Kube-OVN underlay subnet cannot be created, on all nodes or only on some, or the bridge network itself reports a failure. Typical signs: + +- Sandbox creation fails with an event like this, for CoreDNS right after a console deployment, or for platform Pods such as `kube-ovn-pinger`, `zeus` and `nevermore` on some nodes, or for a fixed-IP business Pod: + + ```text + Warning FailedCreatePodSandBox ... plugin type="kube-ovn" failed (add): RPC failed; request ip return 500 configure nic eth0 for pod timeout: read packet ...: is not ready for interface eth0 after 200 checks: resolve MAC address of timeout + ``` + + On a freshly deployed cluster this can look as if the CoreDNS address were already in use. + +- The bridge network (ProviderNetwork) shows status Error with `Ready: false`, and the per-node condition reads: + + ```text + Ready False InitOVSBridgeFailed failed to create and configure external bridge br-: nic does not exist + ``` + +- The underlay network breaks right after a ProviderNetwork was renamed (for example while configuring an underlay sub-interface for MetalLB), and does not recover. +- After a power cycle a VLAN sub-interface such as `bond1.131` reappears on the node, and containers on the underlay subnet stop working. + +## Root Cause + +### Cause 1: the ProviderNetwork is bound to the wrong NIC after deployment + +A cluster deployed from the console may bind the ProviderNetwork to a NIC other than the intended underlay NIC, for example when the deployment form did not allow choosing it. `br-provider` is then created on the host-network NIC. `ip r` on the node shows the default route and the host subnet on `br-provider`, while the underlay NIC holds only a kernel link route for its own subnet. Pods on the underlay subnet then try to resolve the gateway through the wrong NIC. In the `resolve MAC address of timeout` error, `` is in the host network subnet, not the underlay gateway, while the node itself can ping the underlay gateway on the intended NIC. + +### Cause 2: the bridge NIC does not exist on some nodes + +A ProviderNetwork names one `defaultInterface` for all nodes, plus optional per-node `customInterfaces` and `excludeNodes`. When a node does not have the configured NIC: + +- the bridge cannot be created there and the node condition shows `InitOVSBridgeFailed ... nic does not exist`, while nodes that have the NIC show `InitOVSBridgeSucceeded`; +- if the ProviderNetwork does not list the NIC that some nodes actually use for the underlay (for example `p40p2`), ARP for the gateway cannot complete on those nodes and sandbox creation fails with `resolve MAC address of timeout`; +- if a NIC disappears from the host (for example after a hardware or IaaS change), the ProviderNetwork shows every node under `Not Ready Nodes` and `ip a` on the nodes lists the other NICs but not the configured one. + +### Cause 3: renaming a ProviderNetwork leaves a stale OVN localnet port + +Kube-OVN does not handle a ProviderNetwork rename. The OVN logical switch port of type `localnet` for the underlay subnet (named `localnet.`) is left behind still pointing at the old provider, and the underlay network stays disconnected. Changing only the ProviderNetwork's NIC while keeping the ProviderNetwork and VLAN names does not trigger this. + +When one underlay NIC carries several VLAN subnets through sub-interfaces, give each VLAN its own ProviderNetwork. + +### Cause 4: the node OS recreates a VLAN sub-interface that conflicts with the underlay + +The underlay NIC (for example `bond1`) is attached to `ovs-system` by the ProviderNetwork. If the node OS also creates a VLAN sub-interface on that NIC (for example `bond1.131`), the sub-interface conflicts with the underlay subnet of the same VLAN, and underlay Pods time out resolving the gateway MAC, although a test VLAN interface on the node reaches the gateway. `ip a | grep bond1` shows the physical NICs enslaved to `bond1`, `bond1` with `master ovs-system`, and a leftover `bond1.131@bond1` in state `DOWN`. + +The sub-interface can come back without any script creating it. On CentOS 7, the network service (`/etc/init.d/network`) recreates it every time `network` restarts or the node boots, even with kubelet and containerd stopped. A likely explanation is that the service implicitly creates the VLAN sub-interface when the `8021q` module is loaded, `bond1` exists and VLAN 131 context is present on the system (for example an address of that VLAN's segment on `bond0`). + +## Resolution + +### Cause 1: the ProviderNetwork is bound to the wrong NIC after deployment + +Point the ProviderNetwork at the NIC that reaches the underlay gateway. + +> **Warning:** Changing `defaultInterface` re-creates the bridge on each node and interrupts underlay traffic. Save the object first and restore it to roll back. Do not select the NIC that carries the node IP; that takes node access with it. + +```bash +kubectl get provider-networks -o yaml > /root/pn-.before.yaml +kubectl edit provider-networks +``` + +After the correct NIC is set, IP allocation works again. If traffic between the host subnet and the underlay subnet is still blocked, a firewall between them has to be opened on the network side. + +If the underlay subnet has only a few usable addresses (for example about 20), it is too small for the platform components. Deploy a standard overlay cluster first and add the underlay subnet afterwards. + +### Cause 2: the bridge NIC does not exist on some nodes + +1. On every failing node, read the real NIC name: + + ```bash + ip -br link + kubectl get provider-networks -o yaml + ``` + +2. Make the ProviderNetwork match the hardware: list nodes whose underlay NIC has another name under `customInterfaces`, and put nodes that have no underlay NIC under `excludeNodes`. + + > **Warning:** Editing the ProviderNetwork reconfigures the bridge on the listed nodes and briefly interrupts underlay traffic there. Save the object first; to roll back, apply the saved copy. + + ```yaml + apiVersion: kubeovn.io/v1 + kind: ProviderNetwork + metadata: + name: + spec: + defaultInterface: + customInterfaces: + - interface: + nodes: + - + excludeNodes: + - + ``` + + For example, when only one node has the configured NIC and the others have no underlay NIC, list the others under `excludeNodes`; the bridge network then becomes available (`可用`, Available). + +3. If a NIC has vanished from a node, restore it on the host first; the ProviderNetwork cannot use an interface the kernel does not show. +4. Recreate the failed Pods, keep underlay Pods off excluded nodes with a node selector, and check gateway ARP on the underlay NIC: + + ```bash + arping -c 3 -I + ``` + +### Cause 3: renaming a ProviderNetwork leaves a stale OVN localnet port + +Only when the underlay broke right after a ProviderNetwork rename: + +1. List the `localnet` ports and find the one that still points at the old ProviderNetwork. The `network_name` option of a `localnet` port is the provider it maps to; a value that names the old ProviderNetwork marks the stale port. + + ```bash + kubectl get provider-networks,vlan,subnet + kubectl ko nbctl show | grep -B2 -A3 localnet | tee /root/localnet.before + kubectl ko nbctl lsp-get-options localnet. + kubectl ko vsctl get open . external-ids:ovn-bridge-mappings + ``` + +2. Delete the stale port and restart `kube-ovn-controller` so that it creates the port again for the current provider. + + > **Warning:** `lsp-del` removes the port from the OVN northbound database at once, and the underlay subnet has no path to the physical network until `kube-ovn-controller` creates it again. Delete only the `localnet` port confirmed in step 1, and keep the saved listing. The delete cannot be undone except by the controller recreating the port. Restarting `kube-ovn-controller` pauses IP allocation until the new leader is ready. + + ```bash + kubectl ko nbctl lsp-del localnet. + kubectl -n kube-system rollout restart deploy/kube-ovn-controller + kubectl -n kube-system rollout status deploy/kube-ovn-controller + kubectl ko nbctl lsp-get-options localnet. + ``` + +3. Keep ProviderNetwork and VLAN names stable afterwards. To use a different interface, change only the NIC. + +### Cause 4: the node OS recreates a VLAN sub-interface that conflicts with the underlay + +1. When a Pod is already stuck, delete the leftover sub-interface. This needs root; without `sudo` the command returns `RTNETLINK answers: Operation not permitted`. + + > **Warning:** Only when the interface is a leftover VLAN sub-interface on the NIC that the ProviderNetwork has attached to `ovs-system`, and nothing on the host uses it. Deleting a sub-interface that carries a host address cuts that address. To roll back, recreate it with `ip link add link name . type vlan id `. + + ```bash + ip a | grep + sudo ip link del . + ``` + + The stuck Pod then starts. + +2. Remove the source of the interface from the OS network configuration if you can find it. If you cannot, add a udev rule that deletes the interface whenever it appears (replace `bond1.131` with your conflicting interface): + + > **Warning:** The rule deletes the named interface every time it appears, including when someone creates it on purpose. Name the interface exactly. To roll back, delete the rule file and reload udev. + + ```bash + cat > /etc/udev/rules.d/99-disable-bond1.131.rules <<'RULE' + SUBSYSTEM=="net", KERNEL=="bond1.131", ENV{SYSTEMD_ALIAS}="*", RUN+="/bin/sh -c '/sbin/ip link set bond1.131 down; /sbin/ip link delete bond1.131'" + RULE + udevadm control --reload-rules + udevadm trigger + ``` + +3. Confirm the rule: restart the network service, or reboot once in a maintenance window, and check that the interface does not come back. + + > **Warning:** Restarting the network service briefly drops every host interface and can cut the session you are using. Run it from a console, in a maintenance window, one node at a time. + + ```bash + systemctl restart network + ip link show . + ``` + +## Diagnostic Steps + +```bash +kubectl get provider-networks -o yaml +kubectl get vlan,subnet +ip -br addr +ip route +ip -d link show | grep -A1 -E 'vlan|br-' +kubectl get provider-networks -o jsonpath='{range .status.conditions[*]}{.node}{" "}{.type}{" "}{.status}{" "}{.reason}{" "}{.message}{"\n"}{end}' +kubectl ko nbctl show | grep -B2 -A3 localnet +``` + +- `br-provider` holding the node address and default route right after deployment, with `resolve MAC address of timeout` for a gateway in the host network subnet, is Cause 1. +- A bridge network that fails only on nodes missing the interface, a node condition `InitOVSBridgeFailed` with `nic does not exist`, or gateway ARP failing only on nodes whose underlay NIC is not listed, is Cause 2. +- An underlay failure that started right after renaming a ProviderNetwork, with a `localnet` port whose `network_name` is the old name, is Cause 3. +- An unexpected VLAN sub-interface after a reboot, or a `DOWN` sub-interface such as `bond1.131@bond1` on a NIC whose master is `ovs-system` while underlay Pods time out resolving the gateway MAC, is Cause 4. +- If the underlay gateway does not answer ARP from any NIC of the node, the fault is in the switch or gateway configuration, not in the ProviderNetwork. diff --git a/docs/en/solutions/New_node_cannot_reach_CoreDNS_or_the_API_server_because_a_firewall_drops_tunnel_traffic_between_nodes_on_ACP.md b/docs/en/solutions/New_node_cannot_reach_CoreDNS_or_the_API_server_because_a_firewall_drops_tunnel_traffic_between_nodes_on_ACP.md new file mode 100644 index 000000000..26be10de6 --- /dev/null +++ b/docs/en/solutions/New_node_cannot_reach_CoreDNS_or_the_API_server_because_a_firewall_drops_tunnel_traffic_between_nodes_on_ACP.md @@ -0,0 +1,95 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# New node cannot reach CoreDNS or the API server because a firewall drops tunnel traffic between nodes on ACP + +Related articles: [Workload cluster abnormal or monitoring empty in the console because global cannot reach its ports on ACP](Workload_cluster_abnormal_or_monitoring_empty_in_the_console_because_global_cannot_reach_its_ports_on_ACP.md). + +## Issue + +Pods on one node, often a newly added one, cannot reach cluster services, while the other nodes are fine: + +- A worker that was just added cannot open CoreDNS port `53`, while an older worker can. +- The log component status is abnormal because the log collection Pod (`nevermore`) fails to start on one node only, while it runs on the other nodes. This can show up right after the log storage is switched (for example from Elasticsearch to ClickHouse) and the log collector is reinstalled, but the switch is not the cause. + +## Root Cause + +### Cause 1: a firewall between node subnets drops Calico IPIP traffic + +Both CoreDNS replicas run in one node subnet, and the new node is in another. From the new node, Pod port `1053` on one replica answers and the other replica does not. On a Calico cluster in IPIP mode, Pod-to-Pod traffic between nodes is carried as IP protocol `4` (IPIP). The firewall between the subnets does not allow IPIP, so the tunnel is dropped even where the TCP port looks open. Allowing IPIP restores communication. + +A Kube-OVN cluster uses a different encapsulation (Geneve on UDP `6081`), so the rule it needs is different (Cause 2). + +### Cause 2: a firewall blocks the Kube-OVN network and the kubernetes Service path from one node + +The firewall on the path from the node restricts two things: the Kube-OVN network between that node and the other nodes, and access to the API server behind the `kubernetes` Service on `443`. A Pod on that node cannot reach the API server through the Service, and Pod traffic to other nodes is dropped, so DaemonSet Pods such as the log collector fail on that node only. Opening that access returns the log component to normal. + +## Resolution + +### Cause 1: a firewall between node subnets drops Calico IPIP traffic + +Only when the CNI is Calico in IPIP mode, the new node is in a different subnet from the CoreDNS Pods, those Pods are Ready, and `1053` to the replicas is only partly open. + +1. On the firewall between the node subnets, permit IP protocol `4` (IPIP) between the cluster node prefixes. Opening TCP `53` alone does not carry the tunnel. + + > **Warning:** An IPIP permission lets every encapsulated Pod flow cross, not only DNS. Make sure both CIDRs are cluster node subnets. To roll back, delete the same firewall entry. A short loss of cross-subnet Pod traffic is possible while the device commits the policy. + +2. If the filtering device is host `iptables` rather than a middlebox, the equivalent on each node is below. Persist it with the mechanism the node already uses (`iptables-save` or `firewall-cmd`); to roll back, run the same commands with `-D` instead of `-I`. + + ```bash + iptables -I INPUT -p 4 -s -j ACCEPT + iptables -I OUTPUT -p 4 -d -j ACCEPT + ``` + +3. The new worker should now open `1053` on every CoreDNS Pod IP and `53` on the `kube-dns` ClusterIP. + +### Cause 2: a firewall blocks the Kube-OVN network and the kubernetes Service path from one node + +Only when the failing Pod is on a single node, Pods on that node cannot reach the `kubernetes` Service on `443`, and the Kube-OVN tunnel between that node and the others fails while the other nodes are fine. + +1. On the firewall of that node, and on any device between it and the other nodes, open the Kube-OVN tunnel traffic between all cluster nodes (Geneve UDP `6081` by default) and the API server path behind the `kubernetes` Service (the endpoint addresses and port listed in its EndpointSlice, `6443` on the control-plane nodes). + + > **Warning:** Opening ports on a node firewall widens what can reach that node. Limit the source to the cluster node prefix and save the current rule set first. To roll back, remove the rules you added or restore the saved set. A firewall reload can briefly drop existing flows on that node. + + If the node uses `firewalld`, one way that keeps the source scoped is to trust the node prefix: + + ```bash + firewall-cmd --list-all > /root/firewalld.before.txt + firewall-cmd --permanent --zone=trusted --add-source= + firewall-cmd --reload + ``` + +2. Recreate the collector Pod on that node and confirm it stays `Running`: + + ```bash + kubectl get pod -A -o wide | grep nevermore | grep + kubectl -n delete pod + ``` + +## Diagnostic Steps + +1. For a new node and CoreDNS, compare the node subnet with the subnet of the nodes running CoreDNS, and test `1053` to every replica. Partial `1053` reachability across that boundary on Calico IPIP is Cause 1: + + ```bash + kubectl -n kube-system get pod -l k8s-app=kube-dns -o wide + nc -vz -w 3 1053 + ``` + +2. For a DaemonSet Pod failing on one node only, test that node's path to the API server and to the other nodes. A timeout here while the same test passes from a healthy node is Cause 2. `kube-ovn-pinger` on the failing node logs the node-to-node and Pod-to-Pod checks that fail: + + ```bash + kubectl get endpointslice -n default -l kubernetes.io/service-name=kubernetes + nc -vz -w 3 6443 + kubectl -n kube-system get pod -l app=kube-ovn-pinger -o wide | grep + kubectl -n kube-system logs --tail=50 + firewall-cmd --list-all + ``` + +3. If `1053` is reachable from the new node and lookups still fail only there, see [Intermittent DNS failures in Pods from a new node, NIC flapping, the KubeDNS alert or CoreDNS upstream timeouts on ACP](Intermittent_DNS_failures_in_Pods_from_a_new_node_NIC_flapping_the_KubeDNS_alert_or_CoreDNS_upstream_timeouts_on_ACP.md). If the firewall is on the node itself and blocks kubelet or metrics ports, see [Node firewall or iptables rules block kubelet 10250, metrics 10254 or control-plane traffic on ACP](Node_firewall_or_iptables_rules_block_kubelet_10250_metrics_10254_or_control_plane_traffic_on_ACP.md). diff --git a/docs/en/solutions/Node_API_and_VIP_access_lost_after_creating_a_Kube-OVN_ProviderNetwork_or_underlay_subnet_on_ACP.md b/docs/en/solutions/Node_API_and_VIP_access_lost_after_creating_a_Kube-OVN_ProviderNetwork_or_underlay_subnet_on_ACP.md new file mode 100644 index 000000000..e8ab76347 --- /dev/null +++ b/docs/en/solutions/Node_API_and_VIP_access_lost_after_creating_a_Kube-OVN_ProviderNetwork_or_underlay_subnet_on_ACP.md @@ -0,0 +1,104 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# Node, API and VIP access lost after creating a Kube-OVN ProviderNetwork or underlay subnet on ACP + +Related articles: [Kube-OVN underlay Pods fail with resolve MAC address timeout or bridge network InitOVSBridgeFailed on ACP](Kube-OVN_underlay_Pods_fail_with_resolve_MAC_address_timeout_or_bridge_network_InitOVSBridgeFailed_on_ACP.md); [All charts stay ChartDownloading on a new Kube-OVN cluster because ovn-default was created as overlay on ACP](All_charts_stay_ChartDownloading_on_a_new_Kube-OVN_cluster_because_ovn-default_was_created_as_overlay_on_ACP.md). + +## Issue + +One of the following happens right after an underlay network is set up with Kube-OVN: + +- A ProviderNetwork (bridge network) was just created and the cluster becomes unavailable: the login page opens but login fails, the cluster VIP does not answer `ping`, and a new interface `br-` appears on every node with the same addresses as the node NIC. +- A new cluster is being created with Kube-OVN, transport mode `Underlay`, a custom default subnet and a VLAN ID. As soon as creation starts, the masters can no longer be logged into, and the switch reports MAC or IP conflicts such as `ARP_USER_DUPLICATE_IPADDR_DETECT` (`user IP address conflict`). +- Before setting up an underlay network on bonded NICs, an engineer asks how the bond is configured and what the platform needs. + +All three come from the choice of the node NIC that the ProviderNetwork uses, or from addresses that collide with the node network. + +## Root Cause + +### Cause 1: the ProviderNetwork defaultInterface is the NIC that carries the node IP + +When a ProviderNetwork is created, Kube-OVN moves the NIC named in `defaultInterface` (or `customInterfaces`) into the OVS bridge `br-`. The node addresses and all routes of that NIC move to the bridge. If that NIC is the one that carries the node IP (for example `defaultInterface: bond0` when `bond0` holds the node address), node, API server and VIP access break on every node. The underlay NIC must be a separate NIC, not the one that carries the node IP. + +The resource is named `provider-networks` (`provider-networks.kubeovn.io`, cluster-scoped). `kubectl delete ProviderNetwork ` fails with `the server doesn't have a resource type "ProviderNetwork"`. + +### Cause 2: the underlay subnet chosen at cluster creation overlaps the node network + +When a cluster is created with an underlay default subnet that lies in the same segment as the node IPs, platform component Pods receive addresses from that subnet as soon as creation starts. Those addresses collide with node addresses in the same VLAN: the switch sees one IP on several MACs (`ARP_USER_DUPLICATE_IPADDR_DETECT`), and the masters become unreachable. Stopping the creation or scale-out task restores node access. + +### Cause 3: NIC bonding for an underlay network is done on the node, not by the platform + +An underlay network needs the gateway address, the subnet, the VLAN ID and the NIC. The platform does not configure bonds. Node-level NIC configuration, including bond, trunk and VLAN, is done in the node operating system. The platform then creates a bridge on top of that interface. If the bond is missing on some nodes, or it is the NIC that holds the node IP, the bridge either fails on those nodes or takes node access with it (Cause 1). + +## Resolution + +### Cause 1: the ProviderNetwork defaultInterface is the NIC that carries the node IP + +1. From a session that still reaches the API (a node console, or any master that still answers), save and delete the ProviderNetwork. + + > **Warning:** Deleting a ProviderNetwork removes the bridge from every node, and underlay Pods that use it lose their network. Here that is the intended effect, because node access is already broken. There is no undo; to roll back, recreate the object from the saved YAML (with a corrected NIC). + + ```bash + kubectl get provider-networks + kubectl get provider-networks -o yaml > /root/pn-.yaml + kubectl delete provider-networks + ip -br addr show br- || true + ``` + + After the delete, the node addresses return to the NIC and node, API and VIP access come back. + +2. Choose a `defaultInterface` from `ip -br addr` that does not hold the node IP, and create the ProviderNetwork again with it. + +### Cause 2: the underlay subnet chosen at cluster creation overlaps the node network + +1. Stop the cluster creation or scale-out task that started handing out addresses. Node access comes back once it stops. +2. Plan an underlay subnet that does not include any node address, and recreate the cluster with it. Exclude every node, gateway and VIP address that must stay in the same VLAN. + + ```bash + kubectl get nodes -o wide + kubectl get subnet -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidrBlock,VLAN:.spec.vlan,EXCLUDE:.spec.excludeIps' + ``` + +### Cause 3: NIC bonding for an underlay network is done on the node, not by the platform + +1. Configure the bond (and any trunk or VLAN) in the operating system of every underlay node. NetworkManager is the preferred tool for node network management, because the platform's functional tests run with it. +2. Check that the bond exists with the same name on all those nodes, and that it is not the NIC that carries the node IP: + + ```bash + ip -d link show + cat /proc/net/bonding/ + ip -br addr show + ``` + +3. Give the platform the bond name, whether the switch port is a trunk or an access (VLAN) port, the VLAN ID, the subnet and the gateway. Nodes whose underlay NIC has another name are listed under `customInterfaces` of the ProviderNetwork; nodes without an underlay NIC go under `excludeNodes` (see the related article on `InitOVSBridgeFailed`). + +## Diagnostic Steps + +1. Check where the node address sits. If it is on `br-` and no longer on the physical NIC or bond, Cause 1 applies: + + ```bash + ip -br addr + ip route + kubectl get provider-networks -o yaml | grep -E 'name:|defaultInterface|interface:' + ``` + +2. Compare the underlay subnet with the node addresses. A subnet CIDR that contains node IPs, together with duplicate-IP alarms on the switch, is Cause 2: + + ```bash + kubectl get nodes -o wide + kubectl get subnet -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidrBlock,VLAN:.spec.vlan' + ``` + +3. For a bond, confirm that it exists and has the expected members on every underlay node before the bridge network is created (Cause 3): + + ```bash + ip -d link show | grep -A1 -E 'bond|vlan' + ``` diff --git a/docs/en/solutions/Node_firewall_or_iptables_rules_block_kubelet_10250_metrics_10254_or_control_plane_traffic_on_ACP.md b/docs/en/solutions/Node_firewall_or_iptables_rules_block_kubelet_10250_metrics_10254_or_control_plane_traffic_on_ACP.md new file mode 100644 index 000000000..d9568a023 --- /dev/null +++ b/docs/en/solutions/Node_firewall_or_iptables_rules_block_kubelet_10250_metrics_10254_or_control_plane_traffic_on_ACP.md @@ -0,0 +1,203 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# Node firewall or iptables rules block kubelet 10250, metrics 10254 or control-plane traffic on ACP + +## Issue + +A firewall or extra iptables rules on the cluster nodes themselves (not a firewall between sites) break part of the cluster. Typical reports: + +- Web CLI to a node fails, and logs of Pods on that node cannot be viewed in the platform. The node is still `Ready` and scheduling works. The Web CLI session shows the banner `远程终端连接已断开` (remote terminal disconnected) and: + + ```text + warning: couldn't attach to pod/nsenter-, falling back to streaming logs: Internal error occurred: error sending request: Post "https://:10250/attach/cpaas-system/nsenter-/nsenter?input=1&output=1&tty=1": dial tcp :10250: connect: no route to host + Error from server: Get "https://:10250/containerLogs/cpaas-system/nsenter-/nsenter": dial tcp :10250: connect: no route to host + ``` + +- Container services on a two-node deployment look hung with no error, one of the nodes is not displayed, and container logs cannot be viewed. +- `kube-ovn-pinger` does not start on some nodes of a new cluster, and `kubectl` on the master cannot fetch logs from those nodes, while the nodes show `Ready`. +- Platform inspection, check "Status of monitor related component", returns Warning with `Availability of service(cpaas-system/alb2-cpaas-system) for monitoring component to get Metrics is not working properly.`, and the same for the ingress-nginx controller `-metrics` Services. No functional impact is visible. +- The production API server is down, and traffic from one master to the other masters does not pass. +- Alerts fire after iptables rules were added to the `raw` table to stop outside access to the nodes, and they do not clear at once after the rules are removed. + +## Root Cause + +### Cause 1: dial tcp :10250: connect: no route to host + +Exec (Web CLI) and log retrieval are API server to kubelet connections on `10250`. From the global cluster and from other nodes, TCP to `10250` on the affected node fails (`telnet: Unable to connect to remote host: No route to host`), while `telnet 10250` on the node itself connects. In the other direction the node reaches the API server on `6443`, so kubelet keeps reporting status and the node stays `Ready`. + +The block can be: + +- iptables rules on the node that reject all external access to `10250`. `no route to host` while the host is up is what a `REJECT` with `icmp-host-prohibited` returns, which is the reject type firewalld uses by default. +- a network policy outside the node that has not been fully opened from the masters to the nodes. Then `telnet 10250` from the master stays at `Trying ...` instead of failing fast, and a `kube-ovn-pinger` on a working node reports `loss count 3` for pings to the pinger Pods and the node addresses of the affected nodes. + +### Cause 2: kube-ovn-pinger still cannot reach the API server Service after 10250 is open + +After `10250` is opened, the logs become readable and show `kube-ovn-pinger` failing to connect to the API server Service, while TCP to the API VIP on `6443` works from the same nodes. Restarting the Kube-OVN `ovs-ovn`, `kube-ovn-cni` and `kube-ovn-pinger` Pods on those nodes restores them; why the Service path failed before the restart was not determined. + +A `curl` from a container on such a node that returns `curl: (60) SSL certificate problem: unable to get local issuer certificate` for `https://:443` or `https://:6443` shows that the path is reachable: the error comes after the TCP and TLS exchange started, and only means `curl` has no CA bundle for the cluster certificate. + +### Cause 3: monitoring cannot scrape metrics on 10254 because the node firewall does not allow it + +In the workload cluster VMUI, the job `-ingress-nginx-controller-metrics` shows `up` at `0`. The controller Pods are `1/1 Running`, and the `*-controller-metrics` Services are `ClusterIP` on `10254/TCP`. From another node, port `80` of the Pod IP returns nginx `404 Not Found`, but `curl :10254/metrics` returns `curl: (7) Failed to connect to port 10254 after 0 ms: Could not connect to server`. `curl 127.0.0.1:10254/metrics` inside the Pod returns data, and so does the same request from the Pod's own node. + +On the node, `systemctl status firewalld` is `active (running)` and `firewall-cmd --list-all` shows zone `public (active)` with no `10254` in its port list. The node firewall does not allow `10254/tcp`, so the monitoring component cannot reach the metrics endpoints from outside the node. (The `alb2-cpaas-system` warning in the same inspection can appear while its VMUI job still returns data.) + +### Cause 4: a firewall enabled on the masters blocks master-to-master traffic and the API server goes down + +A firewall enabled on the masters blocks the traffic between them, the API server becomes unavailable, and traffic from one master to the others does not pass. + +### Cause 5: iptables raw table rules raise alerts and the cluster recovers slowly after their removal + +Rules in the `raw` table `PREROUTING` chain see every packet entering the node before connection tracking and before the kube-proxy and CNI chains, including Service, tunnel and Pod traffic. Rules added there to block outside access to in-cluster services also disturb the cluster, and high-load alerts fire. After the rules are removed, the cluster returns to normal gradually rather than at once. + +Restricting all ports on all nodes with iptables without affecting the cluster has no product solution. To restrict access to workload services, use **Network > Network Policies** in the Container Platform view; network policies cover the ports that workloads expose, not every port on the node. + +## Resolution + +### Cause 1: dial tcp :10250: connect: no route to host + +Only when `10250` fails from other nodes, answers locally, and the node is `Ready`. + +- **Block outside the node** (network policy): open `10250` from the control-plane nodes to every node. +- **Block on the node**: find the rule that rejects `10250`, or rejects everything after an allow list, and remove only that rule by line number. + + > **Warning:** Flushing a whole table (`iptables -F`) also removes the kube-proxy and CNI chains until those components re-sync, which interrupts Service and Pod traffic on the node. Deleting the wrong rule can open the node wider than intended. Save the rules first and identify the rule by line number. To roll back, run `iptables-restore < /root/iptables.before`. + + ```bash + iptables-save > /root/iptables.before + iptables -L INPUT -n --line-numbers | grep -E 'REJECT|DROP|10250' + iptables -D INPUT + ``` + +- **firewalld on the node**: add the port, scoped to the cluster node range. + + > **Warning:** A rich rule with a source range that is too wide exposes the kubelet API on that node. Save the current configuration first. To roll back, run the same command with `--remove-rich-rule` and reload. + + ```bash + firewall-cmd --state + firewall-cmd --list-all > /root/firewalld.before.txt + firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="" port port="10250" protocol="tcp" accept' + firewall-cmd --reload + ``` + +Persist an iptables change with the mechanism the node already uses, then test from a master and from the global cluster: + +```bash +nc -vz -w 3 10250 +kubectl logs -n --tail=5 +``` + +### Cause 2: kube-ovn-pinger still cannot reach the API server Service after 10250 is open + +Only after Cause 1 is fixed and `kube-ovn-pinger` on the same nodes still logs API server Service connection failures while the VIP on `6443` answers. Restart the three Kube-OVN Pods on each affected node. + +> **Warning:** Restarting `ovs-ovn` and `kube-ovn-cni` on a node interrupts Pod networking on that node until they are Ready again, and new Pods cannot get networking meanwhile. Do one node at a time. No rollback is needed; the DaemonSets recreate the Pods. + +```bash +kubectl -n kube-system get pod -o wide | grep -E 'ovs-ovn|kube-ovn-cni|kube-ovn-pinger' | grep +kubectl -n kube-system delete pod +kubectl -n kube-system get pod -o wide | grep +``` + +### Cause 3: monitoring cannot scrape metrics on 10254 because the node firewall does not allow it + +Only when `10254/metrics` answers inside the Pod and on its node, and fails from another node. Allow `10254` on the node firewall from the addresses the monitoring component scrapes from. + +> **Warning:** Opening a port on the node firewall widens what can reach the node. Limit the source to the cluster range and save the current configuration. To roll back, remove the rule (`--remove-rich-rule` with the same text) and reload. + +```bash +firewall-cmd --list-all > /root/firewalld.before.txt +firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="" port port="10254" protocol="tcp" accept' +firewall-cmd --reload +curl -s http://:10254/metrics | head -5 +``` + +If the site's rule is simply to open the port in the active zone, the plain form is `firewall-cmd --permanent --zone=public --add-port=10254/tcp` followed by `firewall-cmd --reload` (roll back with `--remove-port=10254/tcp`). Afterwards, `up` for the `-controller-metrics` job in VMUI should return to `1`; re-run the inspection. + +### Cause 4: a firewall enabled on the masters blocks master-to-master traffic and the API server goes down + +Only when the firewall on the masters is active and master-to-master traffic fails. Restore the traffic between the masters: either return the firewall to its previous state, or allow the control-plane ports between the master addresses. In general Kubernetes terms these are `6443` (API server), `2379` and `2380` (etcd) and `10250` (kubelet), plus the CNI traffic between nodes; the full list for ACP nodes is the communication matrix in the installation documentation. + +> **Warning:** Rules on a broken control plane are added node by node; keep console or out-of-band access to each master and agree with the security owner on the option used. To roll back, remove the added rules. + +```bash +systemctl is-active firewalld +firewall-cmd --list-all +for p in 6443 2379 2380 10250; do nc -vz -w 3 $p; done +firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="" accept' +firewall-cmd --reload +``` + +If the site decides to turn the firewall off instead: + +> **Warning:** Stopping and disabling firewalld removes every rule it enforced on that master, now and after reboot. Only when the security owner accepts an unfiltered node and `firewall-cmd --list-all` was saved. To roll back, run `systemctl enable --now firewalld`, which re-applies the saved permanent configuration. + +```bash +systemctl stop firewalld +systemctl disable firewalld +``` + +Then check etcd and the API server: + +```bash +kubectl -n kube-system get pod -l component=etcd -o wide +kubectl get --raw='/readyz?verbose' | tail -5 +``` + +### Cause 5: iptables raw table rules raise alerts and the cluster recovers slowly after their removal + +Only when the alerts started with rules added to the `raw` table. Remove those rules; expect recovery to be gradual. + +> **Warning:** Removing the rules restores outside access that the security team wanted to block. Save the table and agree with the security owner. To roll back, run `iptables-restore -T raw < /root/iptables-raw.before` (or re-add the rules), which brings the alerts back. + +```bash +iptables-save -t raw > /root/iptables-raw.before +iptables -t raw -S +iptables -t raw -D PREROUTING +``` + +Use network policies to restrict access to workload services instead. + +## Diagnostic Steps + +1. Test the port from several directions. Failing from other nodes while answering locally means a filter on the path or on the node: + + ```bash + nc -vz -w 3 10250 + ssh 'curl -sk https://127.0.0.1:10250/healthz; echo' + kubectl get node + ``` + +2. List what filters the node. Look for rules that are not from kube-proxy (`KUBE-`), Calico (`cali-`) or Kube-OVN (`ovn`): + + ```bash + systemctl is-active firewalld + iptables-save | grep -vE 'KUBE-|cali-|ovn' | grep -E 'REJECT|DROP' + iptables -t raw -S + ``` + +3. For a metrics alert, compare local and remote scrapes. Run the second `curl` from a different node and again on the Pod's own node. `Could not connect to server` from the other node, data on the own node, and no `10254` in `firewall-cmd --list-all` is Cause 3: + + ```bash + kubectl -n exec -- curl -s 127.0.0.1:10254/metrics | head -3 + curl -s --max-time 3 http://:10254/metrics | head -3 + firewall-cmd --list-all | grep 10254 + ``` + +4. Map the findings: + + - Logs and Web CLI fail with `no route to host` on `10250`, node `Ready`: Cause 1. + - `10250` open, `kube-ovn-pinger` still cannot reach the API server Service: Cause 2. + - `kube-ovn-pinger` `CrashLoopBackOff` on every node of a new cluster with `failed to get self pod ... context deadline exceeded`, no rejecting rule, bonded physical NICs: not a firewall; see [Cross-node Pod traffic fails on the Kube-OVN Geneve overlay because of UDP 6081, TX offload or MTU on ACP](Cross_node_Pod_traffic_fails_on_the_Kube-OVN_Geneve_overlay_because_of_UDP_6081_TX_offload_or_MTU_on_ACP.md). + - Metrics target down, `10254` answers locally only: Cause 3. + - API server down, master-to-master traffic blocked, firewall active: Cause 4. + - Alerts began with `raw` table rules: Cause 5. + - The global cluster cannot reach a workload cluster's VIP ports: see [Workload cluster abnormal or monitoring empty in the console because global cannot reach its ports on ACP](Workload_cluster_abnormal_or_monitoring_empty_in_the_console_because_global_cannot_reach_its_ports_on_ACP.md). diff --git a/docs/en/solutions/OperatorHub_instances_or_pages_fail_from_a_broken_11780_VIP_mapping_or_marketplace-apiserver_restarts_on_ACP.md b/docs/en/solutions/OperatorHub_instances_or_pages_fail_from_a_broken_11780_VIP_mapping_or_marketplace-apiserver_restarts_on_ACP.md new file mode 100644 index 000000000..2280d48fa --- /dev/null +++ b/docs/en/solutions/OperatorHub_instances_or_pages_fail_from_a_broken_11780_VIP_mapping_or_marketplace-apiserver_restarts_on_ACP.md @@ -0,0 +1,119 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# OperatorHub instances or pages fail from a broken 11780 VIP mapping or marketplace-apiserver restarts on ACP + +## Issue + +An instance of an Operator-based tool cannot be created on a workload cluster although the Operator looks healthy, or the OperatorHub page itself cannot be opened. Typical reports: + +- GitLab (or Nexus, or another Operator on the same cluster) cannot create an instance. In OperatorHub the creation page shows `Resource is being initialized, please try again later` and stays there. +- The Operator Pods on the workload cluster (for example `gitlab-ce-operator`, `nexus-ce-operator-controller-manager`) are `Running`, the Operator log shows nothing wrong, and its CRDs and resources look normal. +- On the global cluster the OLM-related Pods are all `Running`, but their logs keep reporting that the `console-plugins` of that workload cluster cannot be retrieved, with `502 Bad Gateway`. +- OperatorHub, the cluster plugin pages and the Tekton pages cannot be opened at all, and the marketplace apiserver in the backend has an abnormal status. + +## Root Cause + +### Cause 1: marketplace-api on global gets 502 from one workload cluster because the 11780 VIP port mapping is broken + +The instance form depends on the global cluster reaching the workload cluster. `marketplace-api` on the global cluster calls the workload cluster's VIP on port `11780`, which the workload cluster ALB serves. When the port mapping for `11780` on the workload cluster VIP (on a firewall NAT, load balancer or VIP gateway) is not connected, global cannot reach the workload cluster on that port, and `marketplace-api` logs: + +```text +collect console-plugins in cluster failed. err: invalid status code: 502. response status: 502 Bad Gateway, response body: X-Error: 502 +``` + +OperatorHub resource initialization then fails, and no instance can be created for any Operator on that cluster. Restoring the `11780` VIP port mapping makes instance creation work again. + +### Cause 2: marketplace-apiserver is killed by its liveness probe before its cache loads + +Several console pages that depend on the marketplace fail together: OperatorHub, the cluster plugin list, Tekton and others. `marketplace-apiserver` loads a cache of the cluster's CRD and APIService resources at startup. In an environment with a large number of those resources, the load has not finished when the liveness probe starts to count failures, so the container is killed and started again before it can serve. Raising `livenessProbe.initialDelaySeconds` of `marketplace-apiserver` to `180` lets the cache finish loading, and the pages recover. + +## Resolution + +### Cause 1: marketplace-api on global gets 502 from one workload cluster because the 11780 VIP port mapping is broken + +Only when `marketplace-api` (or the OLM Pods) on global log `502` for that one workload cluster, and a connection from the global nodes to the workload VIP on `11780` fails or does not reach the workload ALB. + +1. Restore the port mapping of the workload cluster VIP for `11780` on the device that does the mapping, so that it forwards to the workload cluster ALB that serves `11780`. This is usually done by the network team. + + > **Warning:** Changing NAT or load-balancer mappings on a shared device can affect other services that use it. Confirm the backends are the workload cluster ALB nodes for that cluster, save the device configuration, and use a change window agreed with the network owner. To roll back, restore the saved mapping. + +2. Verify from more than one global node. The first two commands should connect and get an HTTP response from the ALB instead of a timeout, and the `marketplace-api` log should stop reporting `502` for that cluster: + + ```bash + nc -vz -w 3 11780 + curl -sv --max-time 5 http://:11780/ -o /dev/null + kubectl logs -n cpaas-system deploy/marketplace-api --since=10m | grep -E '502|console-plugins' + ``` + +3. Create the instance again in OperatorHub. + +### Cause 2: marketplace-apiserver is killed by its liveness probe before its cache loads + +Only when the `marketplace-apiserver` Pod keeps being restarted by its liveness probe (or never becomes ready) and the cluster holds a large number of CRDs or APIServices. A `502` toward one workload cluster with a healthy `marketplace-apiserver` is Cause 1 instead. + +1. Find the workload, save it, and read the current probe delay per container: + + ```bash + kubectl get deploy -A | grep -i marketplace + kubectl -n get deploy -o yaml > /tmp/marketplace-apiserver.before.yaml + kubectl -n get deploy -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" "}{.livenessProbe.initialDelaySeconds}{"\n"}{end}' + ``` + +2. Raise the initial delay to `180` seconds. + + > **Warning:** Changing the probe rolls the `marketplace-apiserver` Pod, so OperatorHub and cluster plugin pages stay unavailable until the new Pod is ready. A longer initial delay also means a hung container is detected later after each start. To roll back, patch `initialDelaySeconds` back to the saved value, or run `kubectl apply -f /tmp/marketplace-apiserver.before.yaml`. + + ```bash + kubectl -n patch deploy -p '{"spec":{"template":{"spec":{"containers":[{"name":"","livenessProbe":{"initialDelaySeconds":180}}]}}}}' + kubectl -n rollout status deploy/ + ``` + + If the patch is denied by `vsentry.kb.io` with a message that a `ResourcePatch` was created, the change is applied through the platform's `sentry` component; wait and check the Deployment again instead of patching a second time. + +3. Reload OperatorHub, the cluster plugin page and the Tekton pages. This is a temporary change: note it and check the value again after a platform upgrade. + +## Diagnostic Steps + +1. On the workload cluster, rule out the Operator. Running Pods with a clean log mean the failure is earlier in the path: + + ```bash + kubectl get pod -A | grep -E 'gitlab-ce-operator|nexus-ce-operator' + kubectl -n logs --tail=50 + ``` + +2. On the global cluster, check the marketplace and OLM Pods and look for `502` toward one cluster: + + ```bash + kubectl get pod -A | grep -E 'market|olm' + kubectl logs -n cpaas-system deploy/marketplace-api --tail=200 | grep -E '502|console-plugins' + ``` + +3. From several global nodes, test the workload cluster VIP on `6443` and `11780`. `6443` answering while `11780` fails points at the `11780` mapping (Cause 1): + + ```bash + for p in 6443 11780; do nc -vz -w 3 $p; done + ``` + +4. When whole pages fail rather than one instance form, check the `marketplace-apiserver` Pod for liveness-probe restarts and count the API resources it has to load. Restarts with `Liveness probe failed` events in a cluster with many CRDs or APIServices is Cause 2: + + ```bash + kubectl get pod -A | grep -i marketplace + kubectl -n describe pod | grep -iE 'Liveness|Restart Count|Killing' + kubectl -n logs --previous --tail=50 + kubectl get crd --no-headers | wc -l + kubectl get apiservice --no-headers | wc -l + ``` + +5. Look-alikes: + + - `11780` accepts connections but the load balancer forwards it to the wrong backends: fix the load-balancer backends; the symptom is the same `502`. + - `marketplace-api` gets `502` for a cluster whose ALB Pods are not Ready: fix the workload cluster ALB first. + - Global cannot reach the workload VIP on `6443` either: see [Workload cluster abnormal or monitoring empty in the console because global cannot reach its ports on ACP](Workload_cluster_abnormal_or_monitoring_empty_in_the_console_because_global_cannot_reach_its_ports_on_ACP.md). diff --git a/docs/en/solutions/Platform_UI_stuck_loading_or_DevOps_errors_from_Service_name_timeouts_caused_by_node_search_domains_on_ACP.md b/docs/en/solutions/Platform_UI_stuck_loading_or_DevOps_errors_from_Service_name_timeouts_caused_by_node_search_domains_on_ACP.md new file mode 100644 index 000000000..fab7e82d2 --- /dev/null +++ b/docs/en/solutions/Platform_UI_stuck_loading_or_DevOps_errors_from_Service_name_timeouts_caused_by_node_search_domains_on_ACP.md @@ -0,0 +1,193 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda DevOps +ProductsVersion: + - 4.3.x +--- + +# Platform UI stuck loading or DevOps errors from Service name timeouts caused by node search domains on ACP + +## Issue + +The platform opens slowly or not at all, while the platform Pods all look `Running`. Typical reports: + +- After login, the page keeps loading and never enters the platform. In browser developer tools, `GET /console/api/v1/envs` returns `500 Internal Server Error` with: + + ```text + {"status": "Failure", "message": "failed to reach validation endpoint: Get \"https://erebus.cpaas-system.svc.cluster.local/kubernetes/global/api\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "reason": "InternalError", "code": 500} + ``` + + The `erebus` Pod is `Running` and its log shows only `Starting https proxy server on: :9443` and repeated `Generate new reverse proxy for: ...` lines. +- The platform cannot be opened right after a 4.3 deployment. Requests to `erebus` time out, and the `apollo` log shows connection timeouts to `erebus` or failures resolving a Service. +- The global `apollo` log repeats `List clusters error, sleep 15s`, `Get cluster resource error`, `List log-agent feature error` or `Use global as log-center cluster is empty`, with requests to `https://kubernetes.default.svc.cluster.local/...` or `https://erebus.cpaas-system.svc.cluster.local/...` failing with `net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)` or `context deadline exceeded`. +- Platform access is very slow, and the image selection list loads for a long time or shows nothing when creating a Deployment. +- The DevOps UI shows an error, or GitLab or Harbor integration fails, with `dial tcp: lookup katanomi-plugin.cpaas-system.svc.cluster.local: i/o timeout` (or the same for `katanomi-plugin.katanomi-operator.svc.cluster.local`) in a `500` `InternalError` response. When integrating an external GitLab, the name that fails is the in-cluster `katanomi-plugin` Service, not the GitLab host. +- The image repository dropdown keeps loading for one project; the request `repositories?itemsPerPage=100&page=1` returns `500`. +- On a business cluster, `alertmanager` fails to start and nothing listens on `9093`. +- Every page takes more than a minute to display data, with no specific error. + +Restarting `erebus`, `apollo` or every Pod in `cpaas-system` alone changes nothing. + +## Root Cause + +### Cause 1: extra search domains in the node /etc/resolv.conf + +For Pods with `dnsPolicy: ClusterFirst`, the kubelet appends the node `search` domains to the Pod search list after the cluster suffixes, with `options ndots:5`. A name such as `erebus.cpaas-system.svc.cluster.local` has four dots, so the resolver tries it with every search suffix before the absolute name. Site suffixes (for example `localdomain` or `novalocal`, often written by NetworkManager) are not cluster zones, so CoreDNS forwards those tries to the upstream resolver. When that resolver is slow or drops them, each try waits for its timeout, the calling component's own deadline expires first, and the error looks like an `erebus` or Service failure. The CoreDNS log shows it directly: + +```text +[ERROR] plugin/errors: 2 erebus.cpaas-system.svc.cluster.local.localdomain. A: read udp :->:53: i/o timeout +``` + +A container's `/etc/resolv.conf` is written when the Pod is created. Fixing or rebooting the node does not change running Pods; they keep the old search list until they are recreated. That is why restarting containers or the host alone does not help. + +This cause covers the login page that keeps loading, the 4.3 platform that does not open, the slow image selection page, the DevOps UI error, the `katanomi-plugin` timeouts during GitLab integration, and the `alertmanager` that does not start. For comparison, a healthy node has no `search` line, for example a single nameserver with `options timeout:1` and `options single-request-reopen`. + +### Cause 2: an in-cluster name is still forwarded upstream after the node fix + +After the node `search` conflict is fixed and the Pods are recreated, component logs can still report resolution timeouts for `katanomi-plugin.cpaas-system`. The Pods use `ClusterFirst` and no host network, and manual lookups from the container succeed, but the CoreDNS log shows the name being forwarded to the external upstream and timing out, so the `kubernetes` plugin is not answering it. Why it is forwarded was not determined. A static CoreDNS hosts record that maps `katanomi-plugin.cpaas-system` to its Service ClusterIP stops the timeouts, the external forwarding and the UI errors. + +### Cause 3: katanomi-controller CPU saturation makes the image dropdown time out + +The image dropdown fails for one project only, with `dial tcp: lookup katanomi-plugin.cpaas-system.svc.cluster.local: i/o timeout`, while `katanomi-api`, `katanomi-controller`, `katanomi-plugin` and `katanomi-operator` are all `1/1 Running` and the Harbor instance is associated with the project. Along the path: + +- the `nginx` container log of the `global-alb2` Pod shows `repositories?itemsPerPage=100&page=N` requests to `katanomi-api` returning `500` after about 10 to 16 seconds, and one `499`, while other pages return `200` in 1 to 2 seconds; +- the `katanomi-api` log shows `"list integrationclass error"` when calling `erebus`. + +A large number of concurrent repository requests drives `katanomi-controller` to its CPU limit, and the UI requests behind the dropdown time out under that load. Raising the controller CPU to 2 cores makes the service stable. + +### Cause 4: the image list is empty when creating a Deployment and a CoreDNS restart restores it + +The image list is not displayed when creating a Deployment, and nothing else is found. Restarting CoreDNS on the cluster brings the list back; the reason was not identified. + +### Cause 5: the whole platform is slow because of the users' VPN path + +Every page loads slowly and data takes more than a minute to show, with no failing request. When the delay follows the users' VPN path, the platform is not at fault, and the fix belongs to whoever runs the VPN. + +## Resolution + +### Cause 1: extra search domains in the node /etc/resolv.conf + +Only when the node `/etc/resolv.conf` has a `search` (or `domain`) line with site suffixes and the platform errors are Service-name timeouts. + +1. Comment out the extra `search` entries on every node of the affected cluster (global and workload clusters, if both have them). Keep the `nameserver` lines. If NetworkManager or DHCP writes the file, remove the search domains at that source as well, or the next restart puts them back. + + > **Warning:** Removing the node search domains changes how host processes resolve short names; anything on the host that relied on a site suffix must use full names afterwards. Back up the file on each node. To roll back, restore the backup and recreate the Pods again. + + ```bash + cp -a /etc/resolv.conf /root/resolv.conf.before + grep -nE '^(search|domain)' /etc/resolv.conf + sed -i -E 's/^(search|domain)([[:space:]])/# \1\2/' /etc/resolv.conf + cat /etc/resolv.conf + ``` + +2. Recreate the Pods so that the kubelet writes a new Pod `/etc/resolv.conf`. The smallest set that recovers the platform is CoreDNS plus `apollo`, `erebus`, the frontend and the katanomi Pods; recreating every Pod in `cpaas-system` also works. + + > **Warning:** Deleting these Pods makes the console and DevOps functions unavailable until the replacements are Ready, and a CoreDNS restart briefly delays lookups. Fix the node files first. No rollback is needed; the controllers recreate the Pods. + + ```bash + kubectl -n kube-system rollout restart deploy/coredns + kubectl -n cpaas-system get pod -o name | grep -E 'apollo|erebus|frontend|katanomi' | xargs kubectl -n cpaas-system delete + ``` + +3. Recreate business Pods that show the same timeouts (for example a business-cluster `alertmanager`) the same way. Then confirm a platform Pod has no site suffix: + + ```bash + kubectl -n cpaas-system exec deploy/apollo -- cat /etc/resolv.conf + ``` + +### Cause 2: an in-cluster name is still forwarded upstream after the node fix + +Only when the CoreDNS log shows `katanomi-plugin.cpaas-system` (or another platform Service name) being forwarded to the upstream after Cause 1 is fixed. + +1. Read the Service ClusterIP: + + ```bash + kubectl -n cpaas-system get svc katanomi-plugin -o jsonpath='{.spec.clusterIP}{"\n"}' + ``` + +2. Add a static record ` katanomi-plugin.cpaas-system` under **Host Alias** in the cluster plugin **Alauda Container Platform Cluster DNS** (**Administrator > Marketplace > Cluster Plugins**), then restart CoreDNS and watch its log. See [CoreDNS Host Alias records do not take effect because a ResourcePatch removes data hosts from cpaas-coredns on ACP](CoreDNS_Host_Alias_records_do_not_take_effect_because_a_ResourcePatch_removes_data_hosts_from_cpaas-coredns_on_ACP.md) if the record does not take effect. + + > **Warning:** A hosts record pins the ClusterIP. If the Service is deleted and recreated with a new ClusterIP, the record keeps answering the old address and the component breaks again. Note that the entry exists so it can be removed later. To roll back, delete the record and restart CoreDNS. + + ```bash + kubectl -n kube-system rollout restart deploy/coredns + kubectl -n kube-system logs -l k8s-app=kube-dns --tail=100 | grep katanomi-plugin + ``` + +### Cause 3: katanomi-controller CPU saturation makes the image dropdown time out + +Only when `katanomi-controller` CPU sits at its limit while the dropdown fails. Raise the controller CPU to 2 cores in the katanomi instance resource (`katanomi-global`, under **Operators > katanomi-operator** in the console), under `helmValues` `controller.resources`. + +> **Warning:** Changing the resources rolls `katanomi-controller`, and DevOps operations pause until it is Ready. Make sure the node has 2 free cores, and save the instance YAML. To roll back, apply the saved YAML. + +```bash +kubectl -n cpaas-system top pod | grep katanomi +kubectl api-resources | grep -i katanomi +kubectl get -A | grep katanomi-global +kubectl get -n -o yaml > /tmp/katanomi.before.yaml +kubectl edit -n +``` + +```yaml +helmValues: + controller: + resources: + limits: + cpu: "2" +``` + +### Cause 4: the image list is empty when creating a Deployment and a CoreDNS restart restores it + +Only after the checks for Causes 1 to 3 found nothing. Restart CoreDNS and reload the Deployment creation page. + +> **Warning:** A rollout restart replaces the CoreDNS Pods one at a time, so cluster lookups can stall for a moment. Make sure at least two CoreDNS replicas are Ready. No rollback is needed. + +```bash +kubectl -n kube-system rollout restart deploy/coredns +kubectl -n kube-system rollout status deploy/coredns +``` + +### Cause 5: the whole platform is slow because of the users' VPN path + +Do not change nodes or Pods for this signature. Time the platform once from a client that goes through the VPN and once from a host that reaches the platform without it (for example a global cluster node), and hand the comparison to the VPN owner: + +```bash +curl -sk -o /dev/null -w 'connect=%{time_connect}s tls=%{time_appconnect}s first_byte=%{time_starttransfer}s total=%{time_total}s\n' https:/// +``` + +A large gap in `connect` or `first_byte` between the two, with fast times from the host, places the delay on the VPN path. + +## Diagnostic Steps + +1. In the browser developer tools, find the request that hangs. A `500` with `failed to reach validation endpoint` on `erebus`, or a timeout on an `erebus` URL, points to name resolution inside the platform Pods. If no request fails and every request is uniformly slow, time the platform from a host that does not use the users' VPN first (Cause 5). + +2. Read the node and Pod resolver files. A site suffix in either is Cause 1: + + ```bash + cat /etc/resolv.conf + kubectl -n cpaas-system exec deploy/apollo -- cat /etc/resolv.conf + kubectl -n cpaas-system logs deploy/apollo --tail=100 | grep -iE 'erebus|timeout|resolve' + ``` + +3. Time a lookup of the platform Service from inside a platform Pod whose image has `sh` and `nslookup`. Seconds for the name without a trailing dot and milliseconds for the absolute name confirm the search expansion cost: + + ```bash + kubectl -n cpaas-system exec deploy/apollo -- sh -c 'time nslookup erebus.cpaas-system.svc.cluster.local' + kubectl -n cpaas-system exec deploy/apollo -- sh -c 'time nslookup erebus.cpaas-system.svc.cluster.local.' + ``` + +4. For DevOps errors, check katanomi and its endpoints, then CoreDNS for forwarded platform names (Cause 2) and controller CPU (Cause 3): + + ```bash + kubectl get pod -A | grep katanomi + kubectl get ep -A | grep katanomi + kubectl -n kube-system logs -l k8s-app=kube-dns --tail=200 | grep -iE 'cpaas-system|i/o timeout' + kubectl -n cpaas-system top pod | grep katanomi + kubectl -n cpaas-system logs -c nginx | grep 'repositories?itemsPerPage' + kubectl -n cpaas-system logs deploy/katanomi-api --tail=200 | grep 'list integrationclass error' + ``` + + A CoreDNS log line whose name ends in a site suffix (for example `.localdomain.`) with `i/o timeout` is Cause 1. A platform Service name forwarded upstream with no suffix after the node fix is Cause 2. Slow `500` or `499` answers from `katanomi-api` in the ALB log, with `katanomi-controller` at its CPU limit, is Cause 3. diff --git a/docs/en/solutions/Pods_fail_on_one_Kube-OVN_node_after_Docker_or_a_second_runtime_is_installed_or_old_cluster_state_remains_on_ACP.md b/docs/en/solutions/Pods_fail_on_one_Kube-OVN_node_after_Docker_or_a_second_runtime_is_installed_or_old_cluster_state_remains_on_ACP.md new file mode 100644 index 000000000..50e7c4934 --- /dev/null +++ b/docs/en/solutions/Pods_fail_on_one_Kube-OVN_node_after_Docker_or_a_second_runtime_is_installed_or_old_cluster_state_remains_on_ACP.md @@ -0,0 +1,119 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.12.x + - 3.18.x + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# Pods fail on one Kube-OVN node after Docker or a second runtime is installed or old cluster state remains on ACP + +Related articles: [kube-ovn-cni fails on one node after a NIC rename, clock skew or hostname change on ACP](kube-ovn-cni_fails_on_one_node_after_a_NIC_rename_clock_skew_or_hostname_change_on_ACP.md). + +## Issue + +Pods on one node of a Kube-OVN cluster cannot be created or cannot reach the API server, although the CNI looks healthy elsewhere. Typical reports: + +- Pods on one node fail with `request ip return 500 configure nic failed no packets received from gateway` after someone installed Docker or a second containerd on the host. +- A platform component such as `zeus` panics because it times out connecting to the API server. From that node's Pods, `curl` to the `kubernetes` Service on `443` times out, while the API server node on `6443` answers: + + ```text + curl: (28) Failed to connect to port 443 after 130856 ms: Couldn't connect to server + curl: (28) Failed to connect to port 443 after 5002 ms: Timeout was reached + ``` + +- On a newly added node, `kube-ovn-pinger` is in `CrashLoopBackOff` (`Exit Code: 1`, hundreds of restarts), other platform Pods crash-loop, and the kubelet log repeats `"Unable to authenticate the request due to an error" err="[invalid bearer token, invalid signature, no keys found]"`. It looks as if the Kube-OVN components cannot connect to the API server. +- The node reaches a destination under the host policy, but Pods on it cannot. + +## Root Cause + +### Cause 1: Docker or a second containerd replaced or disturbed the platform runtime + +The event text looks like a Kube-OVN gateway failure (`no packets received from gateway`), but the runtime is what broke: + +- Docker installed by hand overwrites containerd-related binaries. Pod sandboxes cannot be created until those binaries are restored. On a single-node ACP 4.0.2 cluster, an extra containerd installed on the node broke the runtime the same way; recreating `ovn0` and clearing the OVN database did not help, and replacing the runtime files did. +- When Docker is started next to the platform's containerd and left running, it rewrites `docker0`, iptables NAT and `FORWARD` rules, and Pod forwarding on that node breaks. Pods then reach the API server node on `6443` but not the `kubernetes` Service on `443`, and components such as `zeus` panic because they cannot create their metrics handler. + +A platform node cannot keep a second runtime that rewrites its iptables. + +### Cause 2: the node kept state from a deleted cluster, or already had a runtime or another vendor's CNI + +- The machine was previously a node of a cluster that was removed, and the node cleanup was never run, so container services and iptables rules are still present. The node reaches destinations under the host policy, but Pods cannot. +- A newly added node already had containerd installed and another vendor's network plugin deployed. On it, `kube-ovn-pinger` (`pinger` container) is `CrashLoopBackOff` with `Exit Code: 1`, `zeus` and other platform Pods crash-loop, and the kubelet log repeats `invalid bearer token, invalid signature, no keys found`. The containerd log has `an error occurred when try to find sandbox: not found` and image pulls fail with `dial tcp :80: i/o timeout`. Such a node does not meet the platform's node software and directory requirements, which ask for conflicting packages to be removed and leftover runtime and kubelet directories to be deleted before the node is added. The exact failure mechanism on such a node was not determined. + +## Resolution + +### Cause 1: Docker or a second containerd replaced or disturbed the platform runtime + +1. Confirm the signature from the `kube-ovn-pinger` Pod on that node, and check whether Docker is running: + + ```bash + kubectl -n kube-system exec -- curl -vk --connect-timeout 5 --max-time 10 https://:443/healthz + kubectl -n kube-system exec -- curl -k -m 5 https://:6443/healthz + systemctl is-active docker containerd + ip link show docker0 + iptables -t nat -S | grep -i docker | head + rpm -qf /usr/bin/containerd /usr/bin/runc + ``` + +2. Restore the platform runtime. Keep this order: stop kubelet, containerd and Docker; uninstall the Docker packages; copy the platform binaries; start containerd and kubelet. Removing a Docker package deletes the files it owns, which can include `/usr/bin/containerd` and `/usr/bin/runc`, so uninstalling after the copy deletes the restored binaries again. A running `containerd` cannot be overwritten in place (`Text file busy`). + + The binaries to copy back from a healthy node of the same ACP version and architecture are `containerd`, `containerd-shim`, `containerd-shim-runc-v1`, `containerd-shim-runc-v2`, `containerd-stress`, `ctr`, `crictl`, `critest`, `runc`, `ctd-decoder`, `kubelet` and `kubectl`. + + > **Warning:** Stopping Docker and replacing `containerd` and `kubelet` stops every container that runs under `dockerd`, stops all container creation and probes on the node, and can leave it `NotReady`. Drain the node first if the API answers. Containers under the platform containerd keep running under `containerd-shim` only when the containerd unit has `KillMode=process` (check `systemctl show containerd -p KillMode`); with `control-group` or `mixed`, stopping containerd stops every Pod on the node. Work from a console, copy the current binaries aside first, and only copy from a node on the same ACP build and architecture. To roll back, restore the saved binaries and start the previous runtime. + + ```bash + mkdir -p /root/runtime-before && cp -a /usr/bin/containerd* /usr/bin/ctr /usr/bin/runc /usr/bin/kubelet /root/runtime-before/ 2>/dev/null + systemctl stop kubelet containerd + systemctl disable --now docker docker.socket || true + # uninstall the Docker packages reported by rpm -qf / rpm -qa | grep docker + # copy the binaries listed above from the healthy node to the same paths + which -a runc containerd + systemctl start containerd + systemctl start kubelet + ``` + +3. If `which -a runc containerd` still lists a copy under `/usr/local/bin` after the restore, Pods keep failing until that leftover is moved aside. +4. When Docker must stay on the machine for other applications, take the node out of the cluster instead: cut its access to the API server and delete the Node object. + +### Cause 2: the node kept state from a deleted cluster, or already had a runtime or another vendor's CNI + +- **Node from a deleted cluster.** Run the platform's node cleanup script for a removed node: when a node is deleted from the cluster in the web UI, the dialog offers **Download Cleanup Script**; run it on the node after the deletion. Then add the node again if it should be part of the cluster, and retest from a Pod. + + > **Warning:** The cleanup removes CNI state and rules from the node and is not fully reversible. Do not run it on a node that is still a healthy member of the cluster. A broad `iptables -F` or `ip link delete` on OVS bridges is not a substitute: it blackholes the node. Save the rules first. + + ```bash + iptables-save > /root/iptables.before-node-cleanup + crictl ps + systemctl list-units --type=service --all | grep -Ei 'docker|kubelet|containerd' + ``` + +- **New node with a pre-installed runtime or another vendor's CNI.** Remove the node from the cluster, uninstall packages that conflict with the platform components, delete the directories listed in the node software and directory requirements, and add the node again. + + > **Warning:** The commands below destroy every container image, container, Pod log and kubelet client certificate on the host, and they are irreversible. Only on a node that has already been removed from the cluster (`kubectl get node ` returns `NotFound`), with its runtime stopped, and only when the `invalid bearer token` or pre-installed runtime signature is confirmed. Do not run them on a member node. + + ```bash + kubectl get node + systemctl stop kubelet containerd docker 2>/dev/null || true + rm -rf /var/lib/docker /var/lib/nerdctl /opt/nerdctl/ /var/lib/containerd /var/log/pods /var/lib/kubelet/pki + ``` + +## Diagnostic Steps + +```bash +kubectl -n kube-system get pod -o wide | grep -E 'kube-ovn-cni|ovs-ovn|kube-ovn-pinger' | grep +kubectl -n kube-system describe pod | grep -A3 FailedCreatePodSandBox +crictl version +systemctl is-active containerd kubelet docker || true +journalctl -u kubelet --since '-30m' | grep -E 'invalid bearer token|Unable to authenticate' | tail -5 +journalctl -u containerd --since '-30m' | grep -E 'find sandbox|i/o timeout' | tail -5 +``` + +- `docker` active on a containerd node, or runtime binaries that changed after a Docker install, is Cause 1. On that node, a Pod reaching the API server node on `6443` but not the `kubernetes` Service on `443` points to the same cause. +- `invalid bearer token` in the kubelet log of a new node that already had containerd or another vendor's CNI, or a node that came from a deleted cluster, is Cause 2. +- If `kube-ovn-cni` itself fails with `failed to parse config` or `failed to initialize config` for an old NIC name, see the related article. diff --git a/docs/en/solutions/Workload_cluster_abnormal_or_monitoring_empty_in_the_console_because_global_cannot_reach_its_ports_on_ACP.md b/docs/en/solutions/Workload_cluster_abnormal_or_monitoring_empty_in_the_console_because_global_cannot_reach_its_ports_on_ACP.md new file mode 100644 index 000000000..668f57779 --- /dev/null +++ b/docs/en/solutions/Workload_cluster_abnormal_or_monitoring_empty_in_the_console_because_global_cannot_reach_its_ports_on_ACP.md @@ -0,0 +1,174 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# Workload cluster abnormal or monitoring empty in the console because global cannot reach its ports on ACP + +Related articles: [New node cannot reach CoreDNS or the API server because a firewall drops tunnel traffic between nodes on ACP](New_node_cannot_reach_CoreDNS_or_the_API_server_because_a_firewall_drops_tunnel_traffic_between_nodes_on_ACP.md). + +## Issue + +The global console shows a workload cluster as abnormal, the container overview does not open, Grafana opened from the platform UI returns an error, or monitoring and logs stay empty after a cluster is imported. After the global worker nodes are migrated, GitOps sync or business-cluster monitoring fails. Platform Inspection cannot connect to Prometheus. Typical error text: + +- Console request for a workload cluster: + + ```text + "status": "Failure", "code": 500, Get "https://erebus.cpaas-system.svc.cluster.local/kubernetes//version?timeout=10s": net/http: request canceled (Client.Timeout exceeded while awaiting headers) + ``` + +- Grafana opened from **Operations Center > Monitoring > Grafana**: `"reason": "Unknown"`, `"code": 500`, `Post "http://
:11780/clusters//grafana/login": dial tcp
:11780: i/o timeout`. +- A GitOps `ApplicationSet` shows Sync Failed while health is Healthy: `ComparisonError: Failed to load live state: failed to get cluster info for "https://:6443": error synchronizing cache state : Get "https://:6443/version?timeout=32s": dial tcp :6443: i/o timeout`. +- Every business cluster overview page shows the banner `监控组件异常,只能获取部分监控数据,请修复。` ("Monitoring component abnormal, only partial monitoring data can be obtained, please fix.") while the cluster status is normal. + +What the global cluster has to reach on a workload cluster: + +| Port | What uses it | +| --- | --- | +| `6443` | Kubernetes API. `erebus` in `cpaas-system` proxies the console to this port; Argo CD uses it for GitOps. | +| `11780` | Monitoring. `apollo` queries `http://:11780/clusters//prometheus/api/v1/query`; Grafana uses the same path. | +| `11781` | Logs, once log data is sunk to Elasticsearch on the workload cluster. | +| `16443` | A workload VIP port used as the API server address on some sites (Cause 4). | +| `2379` | etcd client. Required between active and standby in a disaster-recovery failover, together with `80` and `443`. | + +The workload cluster also has to reach the global cluster's image registry. For an imported external cluster to show logs and monitoring, open `6443`, `11780` and `11781` (once logs are sunk to that cluster) from every global node to the imported cluster's control plane (its VIP or the control-plane node addresses), and open access from the imported cluster to the global registry. + +## Root Cause + +### Cause 1: erebus times out on 6443 or apollo on 11780 because the path from some global nodes is closed + +`erebus` logs `timeout or abort` while calling `https://:6443`, and `apollo` logs `context deadline exceeded` for `http://:11780/clusters//prometheus/api/v1/query`. Every node in the workload cluster can be `Ready`, and `cpaas-system` and `kube-system` Pods on both clusters can look healthy. + +The path from the global nodes to that VIP is only partly open. A check from one global node, or only from the first master, is not enough: each global node needs `6443` and `11780` to the workload VIP. `11781` is the log path on the same VIP; closing it alone does not blank the container overview. Grafana from the platform UI uses `11780` as well, so an unopened `11780` to the address in the Grafana error makes monitoring look abnormal even when the monitoring components are healthy. Restarting `apollo` does not open a port that the firewall drops. + +### Cause 2: an imported cluster's monitoring endpoint fell back to the master internal IP + +A cluster was imported with a public VIP, and monitoring stays empty. From the `apollo` container, TCP to the workload node on `11780` fails. `kubectl get nodes -o wide` on the workload cluster shows no public `ExternalIP` on the masters, and nodes labeled `cpaas.io/system-alb` lack it too. The Endpoints object `cluster-proxy-` then contains the master internal IP instead of the VIP that global can route to. + +Two gaps stack: the firewall from global to the workload control plane does not allow `6443`, `11780` and `11781`, and the monitoring component still targets the fallen-back internal address. + +### Cause 3: Pods on newly migrated global workers cannot reach the business VIP + +After the global cluster's worker nodes are migrated, GitOps applications fail with `ComparisonError: Failed to load live state ... dial tcp :6443: i/o timeout` (or `Failed to load target state: failed to get cluster version for cluster "https://:6443" ... (retried 3 times).`), or monitoring disappears for every business cluster with the banner above, and the `apollo` log shows `Prometheus query error`. + +The Argo CD `argocd-gitops-application-controller-0` / `-1` Pods, or the global `apollo` Pods, run on the new global workers (role ``). From a new worker, `curl -v -k https://:6443` and `telnet 6443` hang at `Trying...`; from a global master the same `curl` returns `403 Forbidden` (`User "system:anonymous" cannot get path "/"`), which proves the path from the master is open. The firewall toward the business clusters allows only the older global node addresses. Platform Pods are not pinned to masters, so after the migration they run on workers that have no path: `6443` for Argo CD, `11780` and `11781` for `apollo`. + +### Cause 4: global cannot reach the workload cluster VIP on 16443 + +The workload cluster is reported abnormal with the `erebus ... version?timeout=10s ... Client.Timeout exceeded` error. On the global cluster, `kubectl get minfo -A | grep | grep -vi run` lists that cluster's ModuleInfo objects (for example `logcenter`, `velero`, `kubernetes`, `prometheus`, `cosi`) as `Failed`. Restarting `erebus`, `apollo` and `archon` changes nothing, and the workload nodes and `kube-system` are healthy. + +The workload nodes use `:16443` as the API server address (the `server` in `/etc/kubernetes/kubelet.conf`). From a workload node, `curl -k https://:16443` returns `403 Forbidden` for `system:anonymous`; from the global master, `curl -kv https://:16443` stops at `Trying ...` with no response. A network firewall blocks global's access to that VIP port. + +### Cause 5: Platform Inspection cannot reach Prometheus because the prdb domain resolves to an external load balancer + +Platform Inspection reports that Prometheus cannot be connected for a cluster, although Prometheus is healthy: the `prometheus` ModuleInfo is `Running`, `kubectl get ars -A | grep prome` shows `Synced` and `Ready`, the Prometheus, Alertmanager, Thanos query and Grafana Pods in `cpaas-system` are `Running`, and `sentry` logs no errors. `courier` connects using the address stored in `prdb`. That address is a domain name, and the domain resolves to the address of an external load balancer (for example an F5) that `courier` cannot reach. This was seen on ACP 3.14. + +## Resolution + +### Cause 1: erebus times out on 6443 or apollo on 11780 because the path from some global nodes is closed + +1. Confirm the gap from more than one global node: + + ```bash + nc -vz -w 3 6443 + nc -vz -w 3 11780 + nc -vz -w 3 11781 + ``` + +2. Open `6443` and `11780` from every global node to the workload VIP, and `11781` as well when log views must work, using the firewall's own syntax. + + > **Warning:** Opening these ports to a source wider than the global node prefix exposes the workload API and monitoring listener. Limit the source to the global node addresses and keep the previous rule set. To roll back, delete the rule you added. + +3. After `nc` succeeds from every global node, open the container overview and a monitoring page as the acceptance test. + +### Cause 2: an imported cluster's monitoring endpoint fell back to the master internal IP + +Only when the `cluster-proxy-` addresses are internal master IPs and the import VIP is a different, reachable address. + +1. Open `6443`, `11780` and `11781` from global to the workload control-plane addresses, including the public VIP, as in Cause 1. +2. On the global cluster, edit the monitoring ModuleInfo (`minfo`) of that workload cluster and set its access address under `spec` to the VIP that global can reach. Copy the address field from the monitoring `minfo` of a healthy imported cluster rather than guessing the key. + + ```bash + kubectl get endpoints -A | grep cluster-proxy- + kubectl get minfo -A | grep -i + kubectl get minfo -n -o yaml > /tmp/minfo.before.yaml + ``` + + > **Warning:** Saving a ModuleInfo rolls the monitoring stack for that cluster, and scrapes pause until the new Pods are Ready. Change only the address that points at the VIP. To roll back, run `kubectl apply -f /tmp/minfo.before.yaml`. + + ```bash + kubectl edit minfo -n + ``` + +3. After about five minutes, TCP `11780` from `apollo` to the VIP should succeed and the monitoring page should fill in. Also confirm that the workload cluster can reach the global registry on the registry port; a closed registry path fails image pulls independently of the monitoring page. + +### Cause 3: Pods on newly migrated global workers cannot reach the business VIP + +Only when the failing Pod runs on a new global worker and the same `nc` succeeds from a global master. + +1. Request the permanent fix: a firewall rule from every global worker node to each business cluster VIP on `6443` (GitOps) and on `11780` and `11781` (monitoring). +2. As a temporary measure, delete the affected Pods so they are scheduled again, and check where the replacements land; the scheduler can pick another new worker. + + > **Warning:** Deleting `application-controller` pauses GitOps reconciliation, and deleting `apollo` blanks monitoring pages, until the new Pod is Ready. Make sure no sync you depend on is in progress. No rollback is needed; the controller recreates the Pod. + + ```bash + kubectl get nodes + kubectl get pod -A -o wide | grep -E 'argocd-gitops-application-controller|apollo' + kubectl -n delete pod + kubectl get pod -A -o wide | grep -E 'argocd-gitops-application-controller|apollo' + ``` + +### Cause 4: global cannot reach the workload cluster VIP on 16443 + +Only when the port fails from global and answers from inside the workload cluster. Have the network team allow the global network segment to the workload cluster segment on TCP `6443`, `11780`, `15012` and `16443`. Restarting global components does not substitute for it. + +```bash +grep server /etc/kubernetes/kubelet.conf +curl -k https://:16443 +for p in 6443 11780 15012 16443; do nc -vz -w 3 $p; done +kubectl get minfo -A | grep | grep -vi run +``` + +Run the first two on a workload node (`403 Forbidden` for `system:anonymous` is the healthy answer) and the rest on a global master. After the firewall change every port reports connected, the ModuleInfo objects leave `Failed`, and the clusters show Normal. + +### Cause 5: Platform Inspection cannot reach Prometheus because the prdb domain resolves to an external load balancer + +Only when the domain in `prdb` resolves to an address `courier` cannot reach. Add a CoreDNS static record that maps that domain to the global cluster's actual API server VIP (**Host Alias** in the cluster plugin **Alauda Container Platform Cluster DNS**), then rerun the inspection. + +```bash +kubectl get prdb -A -o yaml | grep -iE 'https?://' +kubectl get pod -A | grep -E 'courier|warlock' +kubectl -n exec -- nslookup +kubectl get minfo -A | grep prome +``` + +After the record points the domain at the global API server VIP, the inspection completes without the Prometheus fault. + +## Diagnostic Steps + +1. From a global node that is not the only master you usually test, probe the workload VIP. A timeout on `6443` or `11780` is Cause 1. A timeout only on `11781` blocks sunk logs and does not explain an `erebus` API timeout: + + ```bash + nc -vz -w 3 6443 + nc -vz -w 3 11780 + nc -vz -w 3 11781 + kubectl -n cpaas-system logs deploy/erebus --tail=50 | grep -E 'timeout or abort|6443' + kubectl -n cpaas-system logs deploy/apollo --tail=50 | grep -E '11780|context deadline exceeded' + ``` + +2. On the workload cluster, see whether the proxy target drifted to an internal IP. Internal addresses in `cluster-proxy-` while the cluster was imported with a public VIP is Cause 2: + + ```bash + kubectl get nodes -o wide + kubectl get nodes -l cpaas.io/system-alb -o yaml | grep -A3 addresses + kubectl get endpoints -A | grep cluster-proxy + ``` + +3. Compare `nc` to the business VIP from a new global worker and from a master (Cause 3), and from global versus inside the workload cluster (Cause 4). Resolve the `prdb` domain from `courier` for inspection failures (Cause 5). + +4. If TCP to `11780` and `11781` succeeds and the monitoring page is still empty, the port is not the problem; check the HTTPS answer on `https://:11780` and the `apollo` and `courier` logs for cache or `cluster not found` errors. diff --git a/docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md b/docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md index 177349739..febc31a95 100644 --- a/docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md +++ b/docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md @@ -10,11 +10,13 @@ id: KB261000048 # Temporary Workarounds for Common NodeLocal DNSCache Issues in Field Environments -This article provides temporary workarounds for three common NodeLocal DNSCache issues in field environments: +This article provides temporary workarounds for five common NodeLocal DNSCache issues in field environments: - DNS resolution is affected when the `node-cache` Pod on a node is unavailable. - Health check port `8080` conflicts. - External monitoring systems or dashboards cannot collect NodeLocal DNSCache metrics. +- A custom domain added as a hosts record in CoreDNS is not resolved through NodeLocal DNSCache. +- `node-local-dns` Pods remain after the plugin is uninstalled. These workarounds are temporary. Manual changes may be overwritten after plugin upgrade, plugin reinstall, platform reconciliation, chart re-rendering, or node rebuild. Perform the change in a maintenance window. @@ -154,6 +156,39 @@ ss -ltnp | grep ':8080' The expected result is that `18080` is listened on by NodeLocal DNSCache, and `8080` is no longer listened on by NodeLocal DNSCache. +### Reverse case: a host process already holds port 8080 + +**Symptom:** While NodeLocal DNSCache is deployed, the `node-local-dns` Pod on one node does not start, and its log shows: + +```text +listen tcp 169.254.20.10:8080: bind: address already in use +``` + +**Cause:** A process on that node already listens on port `8080` on all addresses (`0.0.0.0:8080`, shown by `lsof` as `TCP *:webcache (LISTEN)`). A wildcard listener holds the port for every local address, so the cache cannot bind its health address on `8080` and exits on that node only. The listener can be an ordinary host service, for example a Java service started from a user's home directory under a non-root account; `/proc//cgroup` shows whether it is a container workload. + +**Resolution:** Identify the process and its owner first: + +```bash +kubectl -n kube-system logs --previous | grep 'address already in use' +ss -ltnp | grep ':8080' +lsof -i :8080 +ls -l /proc//exe /proc//cwd +cat /proc//cgroup +systemctl status +``` + +Then either move the NodeLocal DNSCache health port as described above, or, if the owner of the host service agrees, stop that service. If `systemctl status ` names a unit, stop the unit (`systemctl stop `), because a supervised service starts again after a plain `kill`; otherwise use `kill `. + +> **Warning:** Stopping the process stops the host service that owns port `8080` on this node, and its in-flight work is lost. Do it only after the service owner confirms it can stop on this node or move to another port or host. To roll back, start the service again with its own start method; `node-local-dns` then fails to bind again on this node. + +Confirm that the port is free (the `ss` output is empty), then recreate the cache Pod on that node: + +```bash +ss -ltnp | grep ':8080' +kubectl -n kube-system delete pod +kubectl -n kube-system get pod -o wide | grep node-local-dns +``` + ## Issue 3: NodeLocal DNSCache metrics cannot be collected externally **Symptom:** External monitoring systems or dashboards cannot access NodeLocal DNSCache metrics. @@ -202,3 +237,60 @@ Confirm that the Corefile is updated, and verify that metrics can be accessed fr ```bash kubectl -n "$NS" get cm "$CM" -o yaml | grep 'prometheus' ``` + +## Issue 4: A CoreDNS hosts record is not resolved through NodeLocal DNSCache + +**Symptom:** A domain was added as a hosts record in CoreDNS (for example under **Host Alias**). In a test Pod, `nslookup ` returns the address, but plain `nslookup ` fails, and the Pod's `/etc/resolv.conf` points at the NodeLocal DNSCache address (for example `169.254.20.10`). When the same record is added to the upstream DNS instead, both forms resolve. + +**Cause:** NodeLocal DNSCache reads its own Corefile from its ConfigMap, not from `cpaas-coredns`. Its Corefile has server blocks only for the cluster zones, such as `cluster.local:53` and `in-addr.arpa:53`, each with `forward . { force_tcp }`, plus a catch-all `.:53` block. Only the zones that have their own block are forwarded to CoreDNS. Any other name, including a custom domain, falls into `.:53` and goes to the node's upstream DNS without passing through CoreDNS, so a record that exists only in CoreDNS is never seen on the default path. A record in the upstream DNS is visible on both paths because both end at that upstream. + +**Resolution:** Add a server block for the custom domain to the NodeLocal DNSCache ConfigMap, so that queries for that zone are forwarded to CoreDNS. Copy the existing `in-addr.arpa:53` block exactly and change only the zone name. Keep the `bind` line and the `forward` target as they appear in the live ConfigMap, including any `__PILLAR__` token. For example: + +```text +example.internal:53 { + errors + cache 30 + reload + bind 169.254.20.10 __PILLAR__DNS__SERVER__ + forward . { + force_tcp + } + prometheus 169.254.20.10:9253 +} +``` + +Save the ConfigMap before editing: + +```bash +kubectl -n "$NS" get cm "$CM" -o jsonpath='{.data.Corefile}{"\n"}' +kubectl -n "$NS" get cm "$CM" -o yaml > /tmp/nodelocaldns.before.yaml +``` + +> **Warning:** A syntax error in this Corefile stops the cache from loading the new configuration, and every Pod that uses NodeLocal DNSCache on the affected nodes can lose DNS. Do this only when the record already answers `nslookup `. To roll back, run `kubectl apply -f /tmp/nodelocaldns.before.yaml`. + +```bash +kubectl -n "$NS" edit cm "$CM" +``` + +The `reload` plugin picks up the new Corefile after the kubelet refreshes the mounted ConfigMap, which can take about a minute. If the new zone is still not served after that, restart the DaemonSet; DNS from Pods on a node can pause while its cache Pod restarts. + +```bash +kubectl -n "$NS" rollout restart ds "$DS" +kubectl -n "$NS" rollout status ds "$DS" +kubectl exec -n -- nslookup +``` + +The last command, without an explicit server, is the acceptance test. + +## Issue 5: `node-local-dns` Pods remain after the plugin is uninstalled + +**Symptom:** The NodeLocal DNSCache plugin was uninstalled and the page reports the uninstall as successful, but `kubectl -n kube-system get ds,pod | grep node-local-dns` still lists the DaemonSet and its Pods. + +**Cause:** This is expected product behavior. The `node-local-dns` DaemonSet and its Pods remaining after the plugin is uninstalled does not mean that the uninstall failed. + +**Resolution:** No recovery action is required. To confirm that what remains is the retained DaemonSet: + +```bash +kubectl -n kube-system get ds -o wide | grep -iE 'node-?local' +kubectl -n kube-system get pod -o wide | grep node-local-dns +``` diff --git a/docs/en/solutions/kube-ovn-cni_fails_on_one_node_after_a_NIC_rename_clock_skew_or_hostname_change_on_ACP.md b/docs/en/solutions/kube-ovn-cni_fails_on_one_node_after_a_NIC_rename_clock_skew_or_hostname_change_on_ACP.md new file mode 100644 index 000000000..ebd9063eb --- /dev/null +++ b/docs/en/solutions/kube-ovn-cni_fails_on_one_node_after_a_NIC_rename_clock_skew_or_hostname_change_on_ACP.md @@ -0,0 +1,168 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.12.x + - 3.18.x + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# kube-ovn-cni fails on one node after a NIC rename, clock skew or hostname change on ACP + +Related articles: [Pods fail on one Kube-OVN node after Docker or a second runtime is installed or old cluster state remains on ACP](Pods_fail_on_one_Kube-OVN_node_after_Docker_or_a_second_runtime_is_installed_or_old_cluster_state_remains_on_ACP.md). + +## Issue + +`kube-ovn-cni` keeps restarting on one node or a group of nodes, or Pods there cannot be created, and the network plugin itself is not what broke. Typical reports: + +- After a master VM was migrated, `kube-ovn-cni` on it does not start. It is in `BackOff`, and the `cni-server` log ends with `"failed to parse config" err="network interface not exist"`. +- After an IaaS upgrade renamed the NICs of every worker (IPs unchanged), many Pods are `Unknown`, `kube-ovn-cni` logs `iface has no valid IP address`, and restarting `ovs-ovn` does not help. +- Adding a node fails with `Check network interface`, or a node whose NIC was renamed never brings OVN up. +- A node that was powered off came back with a wrong clock, and `kube-ovn-cni` and `ovs-ovn` there cannot create Pods. +- A node (for example a GPU node) was removed from the cluster, its hostname was changed, and after it rejoined, the component Pods on it are abnormal. + +Each cause below is a property of the host: its interface names, its clock, or its identity in the OVN database. + +## Root Cause + +### Cause 1: Check network interface, the configured interface name is not on the node + +The cluster was created with one interface name, but some machines use another name, so those nodes never finish joining. Separately, renaming a NIC on a node that is already in the cluster leaves OVN unable to start, because restarting `ovs-ovn` does not re-read host NIC configuration. At cluster creation the platform can take a different interface name per node. For a node already in the cluster, check Cause 2 first; re-adding the node after the name is corrected is the fallback. + +### Cause 2: failed to parse config or failed to initialize config for the old NIC after a VM migration or IaaS upgrade + +`kube-ovn-cni` reads the tunnel interface of a node from the Node annotation `ovn.kubernetes.io/tunnel_interface`. The per-node annotation overrides the DaemonSet `--iface` argument. When the NIC is renamed on the host side (VM migration, IaaS upgrade) and the annotation still holds the old name, `kube-ovn-cni` cannot build its configuration and `cni-server` keeps restarting. + +After a VM migration (Kube-OVN `v1.9.25`), the events show `NodeNotReady`, `SandboxChanged`, `Error: services have not yet been read at least once, cannot construct envvars`, repeated restarts of the `cni-server` container and `Back-off restarting failed container`. The `cni-server` log reads: + +```text +Find node tunnel interface name: +network interface not exist +failed to find iface , network interface not exist +"failed to parse config" err="network interface not exist" +``` + +After an IaaS upgrade that renamed the NICs but kept the IPs (Kube-OVN `v1.15.17`), the old name may still exist without the node IP: + +```text +iface has no valid IP address +"failed to initialize config" err="iface has no valid IP address" +``` + +Restarting `ovs-ovn` does not help in either case. Setting the annotation to the current interface name does. + +### Cause 3: the node clock is far from the cluster after a power cycle + +After a node is shut down and powered on again, its clock can be far from the rest of the cluster. `kube-ovn-cni` and `ovs-ovn` on that node then cannot create Pods, a large set of Pods stays abnormal, and new Pods cannot be created there. A clock hours ahead of the API server also makes the API server reject that node's clients. The platform requires node clocks to stay within 10 seconds of each other. + +### Cause 4: the node rejoined with a new hostname and OVN still holds its old chassis + +When a node is removed, renamed and joined again (seen on ACP 3.12), the OVN southbound database still holds a chassis record for the old hostname. The component Pods on the node are abnormal until that stale chassis is destroyed. A node that moves between clusters or gets a new management IP can leave the same stale row. + +## Resolution + +### Cause 1: Check network interface, the configured interface name is not on the node + +- For a cluster that has not finished creating, recreate it and set each node's interface to the name shown by `ip -br link` on that machine. Do not keep a single global name that only some nodes have. +- For a node already in the cluster whose interface was renamed, try Cause 2 first. If the `kube-ovn-cni` log shows no Cause 2 signature, or the annotation did not recover the node, remove the node, fix the host interface name (or the per-node interface setting) so it matches what Kube-OVN was given, and add the node again through the platform node workflow. + + > **Warning:** `kubectl drain` evicts workloads and `kubectl delete node` removes the node from scheduling. Pods with local storage stay stranded. Add the node back before leaving the maintenance window. Precondition: no Cause 2 signature, or the Cause 2 annotation did not recover the node. With a Cause 2 signature, do not drain or delete the node; annotate it instead. + + ```bash + kubectl drain --ignore-daemonsets --delete-emptydir-data + kubectl delete node + ``` + +### Cause 2: failed to parse config or failed to initialize config for the old NIC after a VM migration or IaaS upgrade + +1. Confirm the signature, and compare the annotation with the interfaces the node really has. `ip link` prints a VLAN interface as `@` (for example `bond0.1341@bond0`); compare the part before `@` with the annotation. + + ```bash + kubectl -n kube-system logs -c cni-server --tail=100 | grep -E 'tunnel interface|not exist|no valid IP|failed to (parse|initialize) config' + kubectl get node -o jsonpath='{.metadata.annotations.ovn\.kubernetes\.io/tunnel_interface}{"\n"}' + kubectl -n kube-system get ds kube-ovn-cni -o yaml | grep -- '--iface' + ip -br addr + ip route show default + kubectl get node -o yaml > /root/node-.before.yaml + ``` + +2. Set the annotation to the NIC that now holds the node IP. + + > **Warning:** The annotation decides which NIC Kube-OVN uses for the tunnel on this node; a wrong name leaves the node without a working tunnel. Only when the log names the old NIC (`not exist` or `has no valid IP address`), `ip -br addr` shows the node IP on another NIC, and the NIC changed on the host side. To roll back, restore the annotation from the saved node YAML. + + ```bash + kubectl annotate node --overwrite ovn.kubernetes.io/tunnel_interface= + kubectl -n kube-system get pod -o wide | grep -E 'kube-ovn-cni|ovs-ovn' | grep + ``` + + After a VM migration, `kube-ovn-cni` starts after the annotation update; no Pod deletion, drain or node re-add is needed. Annotate every renamed node. + +3. If `kube-ovn-cni` stays in back-off (as after the IaaS upgrade), restart it on that node. + + > **Warning:** Deleting the `kube-ovn-cni` Pod stops sandbox creation on that node until the new Pod is Ready. Only after the annotation names the NIC that holds the node IP. The DaemonSet recreates the Pod; there is nothing to roll back. + + ```bash + kubectl -n kube-system delete pod -l app=kube-ovn-cni --field-selector spec.nodeName= + ``` + +### Cause 3: the node clock is far from the cluster after a power cycle + +1. Compare the clock with a healthy node, and set it from the same time source the healthy nodes use: + + ```bash + timedatectl + chronyc tracking || true + ``` + +2. Delete the `kube-ovn-cni` and `ovs-ovn` Pods on that node so the DaemonSets recreate them. + + > **Warning:** Deleting `ovs-ovn` drops forwarding on that node until the new Pod is Ready. Confirm the clock is correct first. Stepping the clock by hours can expire TLS sessions on the node; that is part of the repair, and local clients reconnect. There is no rollback of a deleted Pod; the DaemonSet creates another. + + ```bash + kubectl -n kube-system delete pod + ``` + +3. If a TopoLVM volume-prepare Pod in `nativestor-system` is still not `Completed` after the network Pods are Ready, delete that prepare Pod too; it is recreated and reaches `Completed`. Skip this when that namespace has no stuck prepare Pod. + + > **Warning:** Deleting the TopoLVM prepare Pod retries volume-group setup and can disrupt a volume that was mid-prepare. + + ```bash + kubectl -n nativestor-system delete pod + ``` + +### Cause 4: the node rejoined with a new hostname and OVN still holds its old chassis + +Find the chassis whose hostname is the node's old name, destroy only that row, and restart the OVN Pods on the node. + +> **Warning:** Destroying a chassis takes effect immediately. Destroying the chassis of a live node cuts that node's Pod traffic until its `ovn-controller` registers again. Match the old hostname exactly and save the list first. There is no undo other than the node registering again. + +```bash +kubectl ko sbctl --columns=_uuid,name,hostname list chassis | tee /root/chassis.before +kubectl ko sbctl destroy chassis +kubectl -n kube-system delete pod +kubectl ko sbctl --columns=name,hostname list chassis | grep -A1 +``` + +## Diagnostic Steps + +Look at the host before touching Kube-OVN objects: + +```bash +kubectl get nodes -o wide +kubectl -n kube-system get pod -o wide | grep -E 'kube-ovn-cni|ovs-ovn|kube-ovn-pinger' | grep +kubectl -n kube-system describe pod | tail -20 +kubectl -n kube-system logs -c cni-server --tail=100 +ip -br link +timedatectl +kubectl ko sbctl --columns=name,hostname list chassis +``` + +- A node-add check that names an interface the machine does not have is Cause 1. +- `network interface not exist` (`failed to parse config`) or `iface has no valid IP address` (`failed to initialize config`) after a VM migration or IaaS upgrade is Cause 2. +- A clock that differs from healthy nodes by more than a few seconds (often hours) after power-on is Cause 3. +- A rejoined node whose old hostname is still listed in the chassis table is Cause 4. +- If Pods fail on the node with `no packets received from gateway` after Docker or another runtime was installed, or the node used to belong to another cluster, see the related article. From c16f3fd575ec68ce54dded0fee660c5752ff756b Mon Sep 17 00:00:00 2001 From: Komh Date: Thu, 12 Nov 2026 21:31:50 +0800 Subject: [PATCH 07/25] docs(solutions): add 21 and update 1 troubleshooting articles --- ...r_a_newly_uploaded_chart_version_on_ACP.md | 117 ++++++++++ ...ransparent_huge_pages_low_CPU_or_on_NFS.md | 210 +++++++++++++++++ ...too_many_clients_already_SQLSTATE_53300.md | 112 +++++++++ ...ce_status_or_Pods_cycling_after_an_edit.md | 145 ++++++++++++ ...observice_exhausts_database_connections.md | 6 +- ...the_project_account_or_node_credentials.md | 221 ++++++++++++++++++ ...ecause_Trivy_cannot_reach_mirror.gcr.io.md | 97 ++++++++ ...wn_authority_or_certificate_has_expired.md | 128 ++++++++++ ...nse_to_HTTPS_client_or_port_443_refused.md | 201 ++++++++++++++++ ..._then_endpoint_certificate_verification.md | 125 ++++++++++ ...Core_or_a_shared_integration_credential.md | 191 +++++++++++++++ ...ot_mount_its_volume_or_has_corrupt_data.md | 204 ++++++++++++++++ ...peration_log_rotation_and_tag_retention.md | 200 ++++++++++++++++ ...of_space_because_Harbor_storage_is_full.md | 202 ++++++++++++++++ ...ry_cannot_be_selected_in_Catalog_on_ACP.md | 93 ++++++++ ...rules_blocked_ports_or_the_network_path.md | 203 ++++++++++++++++ ...t_scope_auth.yaml_targets_or_node_login.md | 133 +++++++++++ ..._cluster_mode_reported_as_empty_secrets.md | 116 +++++++++ ...scope_or_authentication_required_on_ACP.md | 111 +++++++++ ..._password_after_a_Redis_password_change.md | 162 +++++++++++++ ...address_family_unsupported_or_OOMKilled.md | 102 ++++++++ ...ce_change_on_its_Deployment_is_reverted.md | 146 ++++++++++++ 22 files changed, 3223 insertions(+), 2 deletions(-) create mode 100644 docs/en/solutions/Catalog_does_not_list_a_Helm_chart_that_is_already_in_Harbor_or_a_newly_uploaded_chart_version_on_ACP.md create mode 100644 docs/en/solutions/Harbor_PostgreSQL_database_Pod_never_starts_after_an_operator_upgrade_with_transparent_huge_pages_low_CPU_or_on_NFS.md create mode 100644 docs/en/solutions/Harbor_PostgreSQL_rejects_sessions_with_too_many_clients_already_SQLSTATE_53300.md create mode 100644 docs/en/solutions/Harbor_core_service_unavailable_or_internal_server_error_stuck_instance_status_or_Pods_cycling_after_an_edit.md create mode 100644 docs/en/solutions/Harbor_image_pull_returns_401_because_of_the_pull_Secret_the_project_account_or_node_credentials.md create mode 100644 docs/en/solutions/Harbor_image_scan_fails_because_Trivy_cannot_reach_mirror.gcr.io.md create mode 100644 docs/en/solutions/Harbor_or_registry_image_pull_fails_with_x509_certificate_signed_by_unknown_authority_or_certificate_has_expired.md create mode 100644 docs/en/solutions/Harbor_push_or_pull_fails_because_the_client_uses_the_wrong_scheme_HTTP_response_to_HTTPS_client_or_port_443_refused.md create mode 100644 docs/en/solutions/Harbor_push_returns_HTTP_500_on_an_S3_or_Huawei_OBS_bucket_empty_access_keys_then_endpoint_certificate_verification.md create mode 100644 docs/en/solutions/Harbor_returns_401_because_of_time_skew_the_node_auth_container_Harbor_Core_or_a_shared_integration_credential.md create mode 100644 docs/en/solutions/Harbor_returns_500_when_its_PostgreSQL_loses_the_Kubernetes_API_cannot_mount_its_volume_or_has_corrupt_data.md create mode 100644 docs/en/solutions/Harbor_storage_cleanup_scheduled_garbage_collection_of_untagged_artifacts_operation_log_rotation_and_tag_retention.md create mode 100644 docs/en/solutions/Harbor_unreachable_or_its_PostgreSQL_out_of_space_because_Harbor_storage_is_full.md create mode 100644 docs/en/solutions/Helm_chart_template_upload_fails_or_the_template_directory_cannot_be_selected_in_Catalog_on_ACP.md create mode 100644 docs/en/solutions/Image_registry_or_Harbor_unreachable_or_slow_because_of_firewall_rules_blocked_ports_or_the_network_path.md create mode 100644 docs/en/solutions/Platform_registry_pull_fails_with_authorization_failed_or_insufficient_scope_auth.yaml_targets_or_node_login.md create mode 100644 docs/en/solutions/Redis_for_Harbor_unavailable_no_Data_Services_license_or_Redis_in_cluster_mode_reported_as_empty_secrets.md create mode 100644 docs/en/solutions/Registry_push_fails_with_insufficient_scope_or_authentication_required_on_ACP.md create mode 100644 docs/en/solutions/harbor-core_cannot_connect_to_Redis_NOAUTH_from_Sentinel_or_ERR_invalid_password_after_a_Redis_password_change.md create mode 100644 docs/en/solutions/harbor-nginx_or_portal_does_not_start_or_keeps_restarting_IP_address_family_unsupported_or_OOMKilled.md create mode 100644 docs/en/solutions/harbor-registry_is_OOMKilled_with_exit_code_137_or_a_resource_change_on_its_Deployment_is_reverted.md diff --git a/docs/en/solutions/Catalog_does_not_list_a_Helm_chart_that_is_already_in_Harbor_or_a_newly_uploaded_chart_version_on_ACP.md b/docs/en/solutions/Catalog_does_not_list_a_Helm_chart_that_is_already_in_Harbor_or_a_newly_uploaded_chart_version_on_ACP.md new file mode 100644 index 000000000..a398182fc --- /dev/null +++ b/docs/en/solutions/Catalog_does_not_list_a_Helm_chart_that_is_already_in_Harbor_or_a_newly_uploaded_chart_version_on_ACP.md @@ -0,0 +1,117 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" +--- + +# Catalog does not list a Helm chart that is already in Harbor, or a newly uploaded chart version, on ACP + +## Issue + +A Helm chart is in Harbor, or an upload reported success, but Catalog, the version dropdown, or template selection does not show it: + +- `helm push` printed `Pushed: //:`, Harbor shows the chart, but the template repository and Catalog stay on the old list or show nothing. +- A newly added chart repository lists none of the charts already stored in it. The repository detail page says `无应用模板` ("no application templates") or `暂时没有找到符合过滤条件的应用模板` ("no application templates match the filter"). +- A new chart package was uploaded (for example with `violet push`), but the template version dropdown still offers only the previous version, and the repository (for example `public-charts`) shows last sync time `-`. + +Restarting OLM Pods and refreshing the browser does not make the chart appear. `helm show chart` against the Harbor URL proves only that the registry has the package, not that Catalog has indexed it. + +Related articles: [Helm chart template upload fails, or the template directory cannot be selected, in Catalog on ACP](Helm_chart_template_upload_fails_or_the_template_directory_cannot_be_selected_in_Catalog_on_ACP.md). + +## Root Cause + +The Catalog UI lists a chart only when the platform has registered it: a matching `Artifact` and `ArtifactVersion` exist in the `cpaas-system` namespace of the `global` cluster. A chart that only exists in the registry is not listed. + +### Cause 1: ACP 4.2 does not synchronize OCI charts automatically + +ACP 4.2 does not support automatic chart synchronization. An OCI chart is not imported from Harbor because the registry has the tag; the template repository shows it after the chart is uploaded through the platform UI (or the Catalog API). + +### Cause 2: helm push to Harbor bypasses the platform + +A version pushed with `helm push` (or any client that talks only to the registry) goes straight to Harbor, so Catalog does not learn about the tag. ChartRepo synchronization runs when a template is uploaded through Catalog, and the Catalog page has no manual synchronization button. Pushing the same tag again with `helm push` does not help. + +### Cause 3: an added OCI chart repository does not import the charts already in it + +After an OCI chart repository is added, the `ChartRepo` reaches phase `Synced`, and `captain-chartmuseum` logs only `Rebuilding index for tenant`, `Fetching chart list from storage`, and `No change detected between cache and storage`, but no `Artifact`/`ArtifactVersion` pair exists for the charts. The platform has no function that copies charts already stored in a newly added OCI repository into Catalog; a synced phase does not create those objects. The result is the same whether the repository address is the Harbor host only or the host plus the project path. + +### Cause 4: the chart repository did not resynchronize after an accepted upload + +The upload succeeded, the architecture of the package matches the cluster, but the repository index did not refresh: last sync time is `-`, and the template list and the version selector of the create form still offer only the previous version. Setting the repository status to `Pending` triggers a resynchronization, after which the new version is available. + +## Resolution + +### Cause 1 and Cause 2 + +Upload the chart through the platform: + +1. Open Catalog → Helm Charts, choose Add Template, and select a template repository of type Chart or OCI Chart that you can manage. +2. Upload the `.tgz`. For an OCI Chart repository, also set the template directory (the chart storage directory). +3. When the row leaves `Uploading` and shows upload successful, the chart is listed, and the upload also triggers ChartRepo synchronization so versions already stored in Harbor appear. + +The same upload through the Catalog API, with an API token from the user profile (Add API Token): + +```bash +curl -k --request POST \ + --url "https:///catalog/v1/chartrepos/cpaas-system//charts" \ + --header "Authorization: Bearer " \ + --data-binary @"-.tgz" +``` + +`` is the ChartRepo name in `cpaas-system`, not the Harbor project name. Use a repository whose `status.phase` is already synced; fix a failed repository first. + +### Cause 3 + +Adding the repository does not import existing charts. Upload each chart that should appear through Catalog → Helm Charts or the Catalog API, as in Cause 1, so the platform registers it. Confirm the state before and after: + +```bash +kubectl get chartrepos.app.alauda.io -A +kubectl get chartrepo -n cpaas-system -o jsonpath='type={.spec.type} phase={.status.phase} reason={.status.reason}{"\n"}' +kubectl get artifact,artifactversion -n cpaas-system +``` + +### Cause 4 + +1. Rule out a reused Artifact first: if the `violet push` output says `artifact already exist, will create artifactversion`, compare the `cpaas.io/chart-purpose` label of the existing `Artifact` with that of the new `ArtifactVersion`. A mismatch means an `Artifact` of the same name from another chart repository hides the version; setting `Pending` does not fix that. + +2. If the upload itself keeps failing, set `--platform-address` to the platform access address only, without a port or a path such as `/public-charts`: + + ```bash + violet push .tgz --platform-address "https://" --platform-username "" --platform-password "" + ``` + +3. Once the upload has succeeded and the version is still missing, set the ChartRepo status to `Pending`. + + > **Warning:** Only when the upload succeeded, the new version is missing from the dropdown and template selection, and the architecture matches. Impact: synchronization of that ChartRepo is re-queued, Catalog can show it as syncing until the phase settles, and a wrong name resyncs the wrong repository. Precondition: the current object is saved and its phase printed by the first two commands. Rollback: patch `status.phase` back to the printed value. The patch does not delete charts or Harbor blobs. + + ```bash + kubectl get chartrepo -n cpaas-system -o yaml > chartrepo-status.yaml + kubectl get chartrepo -n cpaas-system -o jsonpath='{.status.phase}{"\n"}' + kubectl patch chartrepo -n cpaas-system --subresource=status --type=merge -p '{"status":{"phase":"Pending"}}' + kubectl get chartrepo -n cpaas-system -o jsonpath='{.status.phase}{"\n"}{.status.reason}{"\n"}' + ``` + + If the status patch is rejected, run `kubectl edit chartrepo -n cpaas-system --subresource=status` and set the phase to `Pending`. + +4. After the repository leaves `Pending`, the uploaded version is available in the application templates. + +## Diagnostic Steps + +```bash +kubectl get chartrepo -n cpaas-system -o custom-columns=NAME:.metadata.name,TYPE:.spec.type,URL:.spec.url,PHASE:.status.phase,REASON:.status.reason +kubectl get artifact,artifactversion -n cpaas-system +helm show chart --version +``` + +| Check | Conclusion | +| --- | --- | +| ACP 4.2, OCI chart present in Harbor, never uploaded in the UI | Cause 1 | +| The version reached Harbor only through `helm push`; the Catalog list is unchanged | Cause 2 | +| OCI repository phase `Synced`, `captain-chartmuseum` only logs `No change detected between cache and storage`, page says `无应用模板`, no `Artifact` pair in `cpaas-system` on `global` | Cause 3 | +| Upload succeeded, version missing from the dropdown, last sync time `-`, architecture matches | Cause 4 | +| `violet push` says `artifact already exist`, and the `cpaas.io/chart-purpose` labels differ | Reused Artifact (Cause 4, step 1) | +| The chart is hidden only in an ARM cluster namespace (`共 0 个应用`, "0 applications in total") | The chart's architecture label, not synchronization | + +On ACP 4.2, Cause 1 and Cause 2 can both apply: there is no background synchronization, and a Catalog upload is also the event that registers the chart. A synced phase together with a missing `Artifact` pair is Cause 3, not Cause 4. Restarting OLM Pods does not fix any of these causes. diff --git a/docs/en/solutions/Harbor_PostgreSQL_database_Pod_never_starts_after_an_operator_upgrade_with_transparent_huge_pages_low_CPU_or_on_NFS.md b/docs/en/solutions/Harbor_PostgreSQL_database_Pod_never_starts_after_an_operator_upgrade_with_transparent_huge_pages_low_CPU_or_on_NFS.md new file mode 100644 index 000000000..d3401fe39 --- /dev/null +++ b/docs/en/solutions/Harbor_PostgreSQL_database_Pod_never_starts_after_an_operator_upgrade_with_transparent_huge_pages_low_CPU_or_on_NFS.md @@ -0,0 +1,210 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Harbor PostgreSQL database Pod never starts after an operator upgrade, with transparent huge pages, low CPU, or on NFS + +## Issue + +Harbor is unreachable, the login page reports that the core service is unavailable, or `harbor-core` is in `CrashLoopBackOff` because it cannot connect to PostgreSQL. `harbor-jobservice` and `harbor-notary-server` crash for the same database outage. The database itself does not come up, and one of the following is seen: + +- The PostgreSQL log contains `unrecognized configuration parameter "wal_keep_segments"`, or no Harbor Pod can start right after the PostgreSQL operator upgraded itself. +- The database Service answers `connection refused` on port 5432 and the operator never creates the `pg` Pod. +- The `pg` Pod stays `Pending`. +- The built-in `harbor-database` Pod takes a very long time to become Ready on NFS storage, logs `FATAL: the database system is starting up` until it times out, or has `FailedMount` events on `database-data`. + +Related articles: [Harbor PostgreSQL rejects sessions with too many clients already (SQLSTATE 53300)](Harbor_PostgreSQL_rejects_sessions_with_too_many_clients_already_SQLSTATE_53300.md) covers a database that is up but has no free sessions. + +## Root Cause + +### Cause 1: a newer PostgreSQL operator upgraded a PostgreSQL 12 cluster + +A newer PostgreSQL operator was installed, or upgraded itself automatically through its Subscription, on a cluster still running PostgreSQL 12. That operator no longer supports PostgreSQL 12 and starts an automatic upgrade. The existing configuration still contains `wal_keep_segments`, which the newer server refuses, so startup aborts: + +```text +unrecognized configuration parameter "wal_keep_segments" +``` + +`harbor-core`, `harbor-jobservice`, and `harbor-notary-server` then crash-loop because the database never finishes startup. The log line is not always captured; an instance that became abnormal right after an automatic operator upgrade is the same cause. Rolling the operator back to the previous version restores the instance. Before the newer operator is installed again, PostgreSQL 12 must be upgraded to PostgreSQL 14 or 16. + +### Cause 2: transparent huge pages are enabled on the node + +Connections to `acid-..svc:5432` fail with `connection refused`, and the `pg` Pod is not created. With transparent huge pages enabled on the node, the PostgreSQL cluster fails during startup (its initialization uses `/dev/shm`) and the operator does not create the Pod. Disabling transparent huge pages at the operating-system level lets the cluster start. The fix is the huge-page setting, not a resize of `/dev/shm`. + +### Cause 3: the pg Pod is Pending because its node has too little CPU + +`harbor-core` logs that it cannot connect to `pg`, and the `pg` Pod has been `Pending` the whole time with `Insufficient cpu`. When the PostgreSQL volume is node-local (for example TopoLVM), the PV is bound to one node and the Pod cannot move. If that node's CPU was reduced at the infrastructure level (for example a VM resized to 1 CPU, leaving about 200m allocatable), the Pod cannot be scheduled and nothing listens on port 5432. + +### Cause 4: the built-in harbor-database starts very slowly on NFS storage + +The database PVC uses an NFS StorageClass (for example `nfs-client`). A very large number of files on NFS can make Harbor start abnormally or extremely slowly. The database logs `FATAL: the database system is starting up` until startup times out, or the Pod has: + +```text +FailedMount: Unable to attach or mount volumes: unmounted volumes=[database-data] ... timed out waiting for the condition +``` + +The `FailedMount` timeout on `database-data` is a typical NFS server-side timeout. The hang clears after the Harbor operator, the Harbor workloads, and the CSI Pods on the Harbor nodes are restarted and Pods stuck in `Terminating` are force-deleted. Where startup stays slow, a new Harbor instance on local `hostpath` or CephFS storage is the recommended layout. + +## Resolution + +### Cause 1 + +1. Confirm the log line and the operator version, and check whether the operator upgraded itself. A recent install plan for a newer `postgres-operator` CSV with `installPlanApproval: Automatic` matches an automatic upgrade. + + ```bash + PG_NS= + PG_POD= + kubectl -n "$PG_NS" logs "$PG_POD" --tail=100 | grep wal_keep_segments + kubectl get csv -A | grep postgres-operator + kubectl get installplan -A | grep -i postgres + kubectl get subscription -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,APPROVAL:.spec.installPlanApproval,CURRENT:.status.currentCSV,INSTALLED:.status.installedCSV' | grep -i postgres + ``` + +2. Uninstall the newer PostgreSQL operator and install the previous version that was running this PostgreSQL 12 cluster. If a Subscription keeps reinstalling the newer CSV, remove that Subscription and the newer CSV, then install the previous version from OperatorHub. + + > **Warning:** Only when the PostgreSQL log contains `unrecognized configuration parameter "wal_keep_segments"` after a newer operator was installed against PostgreSQL 12, or the instance became abnormal right after an automatic operator upgrade shown in step 1. Impact: while the operator is removed, no PostgreSQL instance on the cluster is reconciled, not only Harbor's. Precondition: a maintenance window, the previous operator version available in OperatorHub, and every `postgresql` resource and its PVCs left in place (if the uninstall offers to delete operands, leave that off). Rollback: reinstall the newer operator, which is valid only after PostgreSQL 12 has been upgraded to 14 or 16. + + ```bash + OP_NS= + kubectl -n "$OP_NS" delete subscription + kubectl -n "$OP_NS" delete csv + ``` + +3. After installing the previous version, wait until the cluster is Running and the `wal_keep_segments` line is gone: + + ```bash + kubectl get postgresql -A + kubectl -n "$PG_NS" get pods -o wide | grep acid- + kubectl -n "$PG_NS" logs "$PG_POD" --tail=40 + ``` + +4. Plan the upgrade of PostgreSQL 12 to 14 or 16 before installing the newer operator again. To prevent another automatic upgrade, set the Subscription to manual approval. + +### Cause 2 + +1. On the node that should run the `pg` Pod (the node that owns the volume, if the volume is node-bound), read and save the current setting: + + ```bash + cat /sys/kernel/mm/transparent_hugepage/enabled + cat /sys/kernel/mm/transparent_hugepage/defrag + ``` + + Continue only when `enabled` shows `[always]` and the `pg` Pod is absent or the database never opens port 5432. + +2. Disable transparent huge pages and keep the setting across reboots. + + > **Warning:** Only when transparent huge pages are `[always]` and the PostgreSQL cluster fails to start as described. Impact: the change applies to every process on the node, not only PostgreSQL. Precondition: the previous `enabled` and `defrag` values are saved; use a maintenance window if latency-sensitive workloads share the node. Rollback: write the saved values back to both files and disable the unit below. + + ```bash + echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled + echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag + sudo tee /etc/systemd/system/disable-thp.service >/dev/null <<'EOF' + [Unit] + Description=Disable transparent huge pages for PostgreSQL + After=network.target + + [Service] + Type=oneshot + ExecStart=/bin/sh -c 'echo never > /sys/kernel/mm/transparent_hugepage/enabled; echo never > /sys/kernel/mm/transparent_hugepage/defrag' + RemainAfterExit=yes + + [Install] + WantedBy=multi-user.target + EOF + sudo systemctl daemon-reload + sudo systemctl enable --now disable-thp.service + ``` + +3. Watch the operator create the `pg` Pod and the cluster reach `Running`. Harbor reconnects once port 5432 accepts sessions. + +### Cause 3 + +1. Confirm the Pending reason and the node the volume is bound to. Adding CPUs to a different node does not help a node-local volume that is already bound. + + ```bash + kubectl -n "$PG_NS" describe pod "$PG_POD" + kubectl describe node + ``` + + On the node: `lscpu` and `grep -c processor /proc/cpuinfo`. + +2. Give the VM more CPUs at the infrastructure level, then reboot the node. + + > **Warning:** Only when the `pg` Pod is `Pending` with `Insufficient cpu` and its PV is bound to this node. Impact: rebooting stops every Pod on the node, including non-Harbor workloads. Precondition: the new CPU allocation is already applied to the VM, and a maintenance window covers the other Pods on the node. A reboot cannot be undone; reducing the CPUs again returns the Pod to Pending. + + ```bash + sudo shutdown -r now + ``` + +3. When the node is Ready and `lscpu` shows the new CPUs, confirm the `pg` Pod is Running, then restart the Harbor Deployments that are still crash-looping so they connect again. + + > **Warning:** Only when the `pg` Pod is Running and `harbor-core` or `harbor-jobservice` is still in `CrashLoopBackOff`. Impact: in-flight core and jobservice work is dropped until the new Pods are Ready. The Deployments recreate the Pods; nothing needs to be rolled back. + + ```bash + H_NS= + CORE=$(kubectl -n "$H_NS" get deploy -o name | grep harbor-core | head -1) + JOB=$(kubectl -n "$H_NS" get deploy -o name | grep harbor-jobservice | head -1) + kubectl -n "$H_NS" rollout restart "$CORE" "$JOB" + kubectl -n "$H_NS" rollout status "$CORE" + ``` + +### Cause 4 + +1. Rule out Causes 1 to 3 and a full NFS export (a database that cannot write to a full export also fails to start; see [Harbor unreachable or its PostgreSQL out of space because Harbor storage is full](Harbor_unreachable_or_its_PostgreSQL_out_of_space_because_Harbor_storage_is_full.md)). Then confirm the volume is on NFS: + + ```bash + H_NS= + kubectl -n "$H_NS" get pods -o wide | grep harbor-database + kubectl -n "$H_NS" describe pod | sed -n '/Events:/,$p' + kubectl -n "$H_NS" logs --tail=100 + kubectl -n "$H_NS" get pvc -o wide | grep -i database + kubectl get storageclass -o jsonpath='{.provisioner}{"\n"}' + ``` + +2. Restart the Harbor operator, the Harbor Deployments, and the CSI Pods on the Harbor nodes. + + > **Warning:** Only when the volume is NFS and the database is stuck in `the database system is starting up` or has a `FailedMount` timeout on `database-data`. Impact: Harbor reconciliation pauses until the operator Pod is Ready; restarting CSI node Pods interrupts mount and unmount on those nodes until CSI is Ready again. Precondition: delete only CSI Pods that run on the Harbor nodes. The controllers recreate all of these Pods; no data is changed. + + ```bash + kubectl get deploy -A | grep -i harbor | grep -i operator + kubectl -n rollout restart deploy/ + kubectl -n "$H_NS" get deploy -o name | grep harbor + kubectl -n "$H_NS" rollout restart deploy/ + kubectl get pods -A -o wide | grep -i csi | grep + kubectl -n delete pod + ``` + +3. If a Pod stays `Terminating`, force-delete that Pod only. + + > **Warning:** Only for a Pod that has stayed `Terminating` after a normal delete. Impact: `--force --grace-period=0` skips graceful shutdown, and a volume can stay mounted on the node until CSI releases it. The controller recreates a managed Pod; the stuck process cannot be recovered. + + ```bash + kubectl -n "$H_NS" delete pod --force --grace-period=0 + kubectl -n "$H_NS" get pods -o wide | grep -E 'database|harbor' + ``` + +4. If the NFS export holds a very large number of files and startup stays slow, deploy a new Harbor instance on local `hostpath` or CephFS storage, and compare its startup time before removing the old instance. + +## Diagnostic Steps + +```bash +kubectl get postgresql -A +kubectl get pods -A -o wide | grep -E 'acid-|harbor-database|harbor-pg' +``` + +| What you see | Cause | +| --- | --- | +| `unrecognized configuration parameter "wal_keep_segments"`, or the instance broke right after an automatic operator CSV upgrade | Cause 1 | +| `connection refused` on port 5432, `pg` Pod missing, `/sys/kernel/mm/transparent_hugepage/enabled` shows `[always]` | Cause 2 | +| `pg` Pod `Pending` with `Insufficient cpu`, volume bound to that node | Cause 3 | +| Built-in `harbor-database` slow to become Ready, `FATAL: the database system is starting up`, `FailedMount` timeout on `database-data`, NFS StorageClass | Cause 4 | +| Database Pod Running, `too many clients already (SQLSTATE 53300)` | SQLSTATE 53300 article above | +| Patroni `DCS is not accessible`, core `read-only transaction (SQLSTATE 25006)`, or `MountDevice failed ... globalmount: mkdir ... file exists` | [Harbor returns 500 when its PostgreSQL loses the Kubernetes API, cannot mount its volume, or has corrupt data](Harbor_returns_500_when_its_PostgreSQL_loses_the_Kubernetes_API_cannot_mount_its_volume_or_has_corrupt_data.md) | + +The Harbor PostgreSQL resource is `acid.zalan.do/v1` `postgresql`; its Pods and Service use the `acid-` prefix, parameters live under `spec.postgresql.parameters`, and the instance size under `spec.resources`. An install that uses the built-in `harbor-database` workload keeps the database Pod and its PVC in the Harbor namespace. diff --git a/docs/en/solutions/Harbor_PostgreSQL_rejects_sessions_with_too_many_clients_already_SQLSTATE_53300.md b/docs/en/solutions/Harbor_PostgreSQL_rejects_sessions_with_too_many_clients_already_SQLSTATE_53300.md new file mode 100644 index 000000000..c124746a2 --- /dev/null +++ b/docs/en/solutions/Harbor_PostgreSQL_rejects_sessions_with_too_many_clients_already_SQLSTATE_53300.md @@ -0,0 +1,112 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Harbor PostgreSQL rejects sessions with too many clients already (SQLSTATE 53300) + +## Issue + +Harbor garbage collection (GC) fails as soon as it starts, or `harbor-core` cannot open a database session. The GC job log or the core log shows: + +```text +failed to execute GC job at mark phase ... too many clients already (SQLSTATE 53300) +``` + +The error is raised while dialing the Harbor PostgreSQL Service, for example `acid-..svc:5432`. The database Pod is Running and the Service has endpoints, so the database looks healthy. + +Related articles: [Harbor PostgreSQL database Pod never starts after an operator upgrade, with transparent huge pages, low CPU, or on NFS](Harbor_PostgreSQL_database_Pod_never_starts_after_an_operator_upgrade_with_transparent_huge_pages_low_CPU_or_on_NFS.md) covers a database that does not start at all. + +## Root Cause + +### Cause 1: the PostgreSQL instance has no free client slots + +`SQLSTATE 53300` means PostgreSQL refused a new session because `max_connections` is reached. Harbor GC then cannot open a session during the mark phase. A Running database Pod and a healthy Endpoint do not rule this out. + +A small instance (1 CPU / 2 GiB) with default connection settings can run out of slots under Harbor's load. Raising the instance to 2 CPU / 4 GiB and setting `max_connections`, `shared_buffers`, `work_mem`, and `effective_cache_size` clears the GC failure. + +If `harbor-notary-server` is crash-looping with `transport: x509: certificate has expired or is not yet valid`, the sessions are being used up by Notary reconnects. Fix the Notary certificate first (see [Harbor notary-server fails with an expired x509 certificate on ACP](Harbor_notary-server_fails_with_an_expired_x509_certificate_on_ACP.md)); the parameter change below is for a database that rejects sessions while Notary is not in that restart loop. + +GC that fails at the mark phase with `write: broken pipe` on port 5432 and no `SQLSTATE 53300` is a different problem: the database already reset sessions held by jobservice. See [Harbor garbage collection fails with write broken pipe to PostgreSQL when jobservice exhausts database connections](Harbor_garbage_collection_fails_with_write_broken_pipe_to_PostgreSQL_when_jobservice_exhausts_database_connections.md). + +## Resolution + +### Cause 1 + +1. Read the live limit, the session count, and the current instance size, and keep a copy of the current values: + + ```bash + PG_NS= + PG_POD= + kubectl -n "$PG_NS" exec "$PG_POD" -- psql -U postgres -c "SHOW max_connections;" + kubectl -n "$PG_NS" exec "$PG_POD" -- psql -U postgres -c \ + "SELECT state, count(*) FROM pg_stat_activity GROUP BY 1;" + kubectl -n "$PG_NS" get postgresql -o yaml > "$HOME/-before.yaml" + kubectl -n "$PG_NS" get postgresql -o yaml | sed -n '/resources:/,/volume:/p' + ``` + +2. Raise the instance size and the connection parameters. The values below recovered a 1 CPU / 2 GiB instance whose sessions were at the limit. On a larger instance, set `max_connections` above the current `SHOW max_connections` value and raise memory with it, keeping `shared_buffers` within the instance memory. + + > **Warning:** Only when the error is `too many clients already (SQLSTATE 53300)`, the session count is at `max_connections`, and Notary is not restarting on an expired certificate. Impact: patching `spec.resources` and `spec.postgresql.parameters` makes the PostgreSQL operator restart the database Pods; Harbor cannot connect until they are Ready. Precondition: a maintenance window and the copy saved in step 1. Rollback: patch the previous `resources` and `parameters` values back. + + ```bash + kubectl -n "$PG_NS" patch postgresql --type=merge -p '{ + "spec": { + "resources": { + "requests": {"cpu": "2", "memory": "4Gi"}, + "limits": {"cpu": "2", "memory": "4Gi"} + }, + "postgresql": { + "parameters": { + "max_connections": "200", + "shared_buffers": "1GB", + "work_mem": "64MB", + "effective_cache_size": "2GB" + } + } + } + }' + kubectl -n "$PG_NS" get postgresql -o jsonpath='{.status.PostgresClusterStatus}{"\n"}' + ``` + +3. When `PostgresClusterStatus` is `Running` and `SHOW max_connections` returns the new value, run GC again. + +## Diagnostic Steps + +1. Read the log of the failed GC run through the Harbor API: + + ```bash + curl -sk -u 'admin:' 'https:///api/v2.0/system/gc?page=1&page_size=10' + curl -sk -u 'admin:' 'https:///api/v2.0/system/gc//log' + ``` + + Expected for this cause: `too many clients already (SQLSTATE 53300)`. + +2. Compare the session count with the limit (step 1 of the Resolution). Expected: the total is at `max_connections`. If it is well below the limit, this is not the cause. + +3. Check Notary: + + ```bash + kubectl -n logs deploy/ --tail=100 | grep -i x509 + ``` + + Expected for this cause: no `certificate has expired` lines. + +4. If `psql` inside the Pod asks for a password, the admin password is in the Secret `postgres..credentials.postgresql.acid.zalan.do`, key `password`: + + ```bash + kubectl -n "$PG_NS" get secret "postgres..credentials.postgresql.acid.zalan.do" \ + -o jsonpath='{.data.password}' | base64 -d; echo + ``` + +| What you see | Where to go | +| --- | --- | +| `SQLSTATE 53300`, database Pod Running, Notary healthy | Cause 1 here | +| GC fails at the mark phase with `write: broken pipe` on port 5432, no `SQLSTATE 53300` | Harbor GC broken pipe article above: restart `harbor-jobservice` | +| Notary log has `certificate has expired`, sessions climb after each Notary restart | Notary certificate article above | +| Database Pod missing, Pending, or not starting | Harbor PostgreSQL database Pod never starts article above | diff --git a/docs/en/solutions/Harbor_core_service_unavailable_or_internal_server_error_stuck_instance_status_or_Pods_cycling_after_an_edit.md b/docs/en/solutions/Harbor_core_service_unavailable_or_internal_server_error_stuck_instance_status_or_Pods_cycling_after_an_edit.md new file mode 100644 index 000000000..573787fdd --- /dev/null +++ b/docs/en/solutions/Harbor_core_service_unavailable_or_internal_server_error_stuck_instance_status_or_Pods_cycling_after_an_edit.md @@ -0,0 +1,145 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "3.12" + - "4.2" + - "4.3" +--- + +# Harbor core service unavailable or internal server error, stuck instance status, or Pods cycling after an edit + +## Issue + +The Harbor login page or portal reports that the core service is unavailable (`核心服务不可用`, "core service unavailable"), or the web UI shows `internal server error`. Image pulls that used to work, including pulls from a public `library` project, start returning `unauthorized`. Harbor core, jobservice, and Notary can all look failed at once. Related signs: + +- `harbor-notary-server` restarts and logs `transport: x509: certificate has expired or is not yet valid`. +- A restart of `harbor-core` brings the UI back for a while, then the outage returns. +- The instance page stays on `Please wait for several minutes for Harbor deployment to complete` although the Pods are Running. +- After the Harbor resource is edited, every Harbor Pod is deleted and recreated again and again. + +Related articles: [harbor-nginx or portal does not start or keeps restarting: IP address family unsupported or OOMKilled](harbor-nginx_or_portal_does_not_start_or_keeps_restarting_IP_address_family_unsupported_or_OOMKilled.md). + +## Root Cause + +### Cause 1: the Notary TLS certificates expired + +`harbor-notary-server` logs `transport: x509: certificate has expired or is not yet valid`, usually with `Failed to dial -harbor-notary-signer:7899` and `Trust not fully operational`. This cause, including why it makes the core service unavailable, is covered in [Harbor notary-server fails with an expired x509 certificate on ACP](Harbor_notary-server_fails_with_an_expired_x509_certificate_on_ACP.md). + +### Cause 2: harbor-core needs a restart (internal server error, no other signature) + +The UI shows `internal server error` or `core service unavailable`, pulls fail with `unauthorized`, Notary does not log an expired certificate, and PostgreSQL has free sessions. A restart of `harbor-core` clears the error and restores pulls. The underlying fault has not been identified. The same restart also recovers a portal that shows `核心服务不可用` right after a container-platform expansion. If the outage returns within hours, the cause is Cause 1, not this one. + +### Cause 3: the instance status stays on "Please wait for several minutes for Harbor deployment to complete" + +The Harbor Pods are Running and the portal answers, but the instance status text stays on the starting message. Restarting the Harbor core process resumes status synchronization. The underlying cause has not been identified. Resetting unrelated instance fields (for example an OIDC job field) does not clear the message. + +### Cause 4: every Harbor Pod cycles after the Harbor resource was edited, and the registry PVC is larger than the declared size + +After an edit of the Harbor resource (such as the Cause 1 update), all Harbor Pods are deleted and restarted repeatedly instead of rolling once. This happens when the registry PVC was expanded beyond the size declared in the Harbor resource, for example a `-harbor-registry` PVC of `500Gi` while the resource still declares `200Gi`. Kubernetes does not allow a PVC request to be reduced, so the reconcile cannot apply the declared size. The size in `helmValues` must be set to the PVC's actual size. + +## Resolution + +### Cause 1 + +When `harbor-notary-server` logs `transport: x509: certificate has expired or is not yet valid`, follow the Resolution of [Harbor notary-server fails with an expired x509 certificate on ACP](Harbor_notary-server_fails_with_an_expired_x509_certificate_on_ACP.md). + +### Cause 2 + +Restart only the core Deployment. + +> **Warning:** Only when the UI shows `internal server error` or `core service unavailable`, the Notary log has no `certificate has expired`, and PostgreSQL has free sessions (see Diagnostic Steps). Impact: in-flight API calls, logins, and pushes fail until the new Pod is Ready. The Deployment recreates the Pod; nothing needs to be rolled back. If the same error returns within hours, stop restarting core and apply Cause 1. + +```bash +NS= +CORE=$(kubectl -n "$NS" get deploy -o name | grep harbor-core | head -1) +kubectl -n "$NS" rollout restart "$CORE" +kubectl -n "$NS" rollout status "$CORE" +``` + +### Cause 3 + +Restart the core Deployment and check the instance status. + +> **Warning:** Only when the Harbor Pods are Running and the status is still `Please wait for several minutes for Harbor deployment to complete`, and no operator upgrade is in progress. Impact: the Harbor API is interrupted for one rollout. The Deployment recreates the Pod; nothing needs to be rolled back. + +```bash +NS= +NAME= +CORE=$(kubectl -n "$NS" get deploy -o name | grep harbor-core | head -1) +kubectl -n "$NS" rollout restart "$CORE" +kubectl -n "$NS" rollout status "$CORE" +kubectl -n "$NS" get harbor "$NAME" -o jsonpath='{.status.conditions}{"\n"}' +``` + +### Cause 4 + +1. Compare the registry PVC with the size the Harbor resource declares: + + ```bash + NS= + NAME= + kubectl -n "$NS" get pvc | grep registry + kubectl -n "$NS" get pvc -o jsonpath='{.spec.resources.requests.storage}{" "}{.status.capacity.storage}{"\n"}' + kubectl -n "$NS" get harbor "$NAME" -o yaml | grep -n -i -E 'registryStorageSize|persistentVolumeClaim|size:' + ``` + +2. Set the declared registry size to the PVC's actual size. Component volumes are under `spec.helmValues.persistence.persistentVolumeClaim`, where the registry entry is `registry.size`; confirm that key on the live object before patching. + + > **Warning:** Only when the registry PVC is larger than the declared size and the Pods cycle after an edit of the Harbor resource. Impact: the operator reconciles again and Harbor components restart once more. Precondition: save the resource with the first command below, and use the PVC capacity exactly as printed (for example `500Gi`), never a smaller value. Rollback: apply the saved file, which returns the size mismatch. + + ```bash + kubectl -n "$NS" get harbor "$NAME" -o yaml > "$HOME/harbor-$NAME-before-size.yaml" + kubectl -n "$NS" patch harbor "$NAME" --type=merge -p \ + '{"spec":{"helmValues":{"persistence":{"persistentVolumeClaim":{"registry":{"size":""}}}}}}' + kubectl -n "$NS" get pods -w + ``` + +3. If the Pods still cycle with matching sizes, read the log of the operator that manages the instance: + + ```bash + kubectl get deploy -A | grep -E 'harbor-operator|devops-tools-operator' + kubectl -n logs deploy/ --tail=200 + ``` + +## Diagnostic Steps + +A core restart fixes Cause 2 and Cause 3 only. When Cause 1 is active it is a temporary pause. + +```bash +NS= +kubectl get harbor -A +kubectl -n "$NS" get pods -o wide +kubectl -n "$NS" get events --sort-by=.lastTimestamp | tail -30 +``` + +1. Read the Notary log first: + + ```bash + NOTARY=$(kubectl -n "$NS" get pods -o name | grep harbor-notary-server | head -1) + kubectl -n "$NS" logs "$NOTARY" --tail=80 + ``` + + Cause 1 is confirmed when the log contains `transport: x509: certificate has expired or is not yet valid`, or `Failed to dial` a Service ending in `harbor-notary-signer:7899` with the same text. Optionally decode the certificate the Notary Pod mounts and read `notAfter`; do not patch the Harbor resource for a certificate that is still valid: + + ```bash + kubectl -n "$NS" get secret -o name | grep -i notary + kubectl -n "$NS" get secret -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject + ``` + +2. Check the PostgreSQL sessions against the limit: + + ```bash + kubectl -n exec -- psql -U postgres -c "SHOW max_connections;" + kubectl -n exec -- psql -U postgres -c \ + "SELECT state, count(*) FROM pg_stat_activity GROUP BY 1;" + ``` + + Sessions at the limit while Notary restarts on the x509 error: Cause 1. Sessions well below the limit and no `idle in transaction` pile-up, with no Notary error: Cause 2. Sessions at the limit without a Notary error: see [Harbor PostgreSQL rejects sessions with too many clients already (SQLSTATE 53300)](Harbor_PostgreSQL_rejects_sessions_with_too_many_clients_already_SQLSTATE_53300.md). If the database Pod is Pending or not Running, see [Harbor PostgreSQL database Pod never starts after an operator upgrade, with transparent huge pages, low CPU, or on NFS](Harbor_PostgreSQL_database_Pod_never_starts_after_an_operator_upgrade_with_transparent_huge_pages_low_CPU_or_on_NFS.md). + +3. Cause 3 is the starting message on the instance while all Pods are Running. Ignore it when Notary or nginx is in `CrashLoopBackOff`. + +4. Cause 4 is all Harbor Pods cycling after an edit of the Harbor resource, with a registry PVC larger than the declared size (step 1 of Cause 4). diff --git a/docs/en/solutions/Harbor_garbage_collection_fails_with_write_broken_pipe_to_PostgreSQL_when_jobservice_exhausts_database_connections.md b/docs/en/solutions/Harbor_garbage_collection_fails_with_write_broken_pipe_to_PostgreSQL_when_jobservice_exhausts_database_connections.md index 361d7273b..622e52de0 100644 --- a/docs/en/solutions/Harbor_garbage_collection_fails_with_write_broken_pipe_to_PostgreSQL_when_jobservice_exhausts_database_connections.md +++ b/docs/en/solutions/Harbor_garbage_collection_fails_with_write_broken_pipe_to_PostgreSQL_when_jobservice_exhausts_database_connections.md @@ -48,7 +48,9 @@ Restarting `harbor-jobservice` closes the connections it holds and GC can run ag kubectl -n rollout status deployment/ ``` -3. Run GC manually from **Administration > Clean Up** (**Garbage Collection**) in the Harbor portal and confirm that it finishes as `Success`. The scheduled runs then succeed again. +3. Run GC manually from **Administration > Clean Up** (**Garbage Collection**) in the Harbor portal and confirm that it finishes as `Success`. The scheduled runs then succeed again. Use **Dry Run** first to see what a real run would delete. + + > **Warning:** A GC run that is not a dry run permanently deletes blobs that no artifact references, and with **Allow garbage collection on untagged artifacts** enabled it also deletes untagged artifacts. Use the same options as the scheduled job unless the registry owner has agreed to other values. Deleted blobs cannot be recovered from Harbor. ## Diagnostic Steps @@ -59,7 +61,7 @@ Restarting `harbor-jobservice` closes the connections it holds and GC can run ag curl -sk -u 'admin:' 'https:///api/v2.0/system/gc//log' ``` - Expected for this cause: `failed to execute GC job at mark phase` with `write: broken pipe` towards port 5432. A different error in the GC log is a different problem. + Expected for this cause: `failed to execute GC job at mark phase` with `write: broken pipe` towards port 5432. A different error in the GC log is a different problem. If the GC log shows `too many clients already (SQLSTATE 53300)` instead, the database refused a new session and restarting jobservice is not the fix; see [Harbor PostgreSQL rejects sessions with too many clients already (SQLSTATE 53300)](Harbor_PostgreSQL_rejects_sessions_with_too_many_clients_already_SQLSTATE_53300.md). 2. Check the database Pod and its log: diff --git a/docs/en/solutions/Harbor_image_pull_returns_401_because_of_the_pull_Secret_the_project_account_or_node_credentials.md b/docs/en/solutions/Harbor_image_pull_returns_401_because_of_the_pull_Secret_the_project_account_or_node_credentials.md new file mode 100644 index 000000000..729909755 --- /dev/null +++ b/docs/en/solutions/Harbor_image_pull_returns_401_because_of_the_pull_Secret_the_project_account_or_node_credentials.md @@ -0,0 +1,221 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Harbor image pull returns 401 because of the pull Secret, the project account, or node credentials + +## Issue + +The registry answers HTTP, but `docker login`, a push, or an image pull is rejected with `401 Unauthorized`, `403 Forbidden`, or `authorization failed: no basic auth credentials`, and the problem is the credential itself: the workload's pull Secret, the project account in it, or what is stored on the node. + +| Signature | Cause | +| --- | --- | +| `authorization failed: no basic auth credentials` | Cause 1 | +| `FailedToRetrieveImagePullSecret` or `Unable to retrieve some image pull secrets (global-registry-auth)` | Cause 2 | +| Private Harbor project: the credential of an authorized (project-scoped) account fails; admin or a public project succeeds | Cause 3 | +| Pull from the global built-in registry as `:11443//...` fails; the same image without `:11443` pulls | Cause 4 | +| The ServiceAccount lists `imagePullSecrets`, the running Pod spec has none, and the pull fails | Cause 5 | +| After a node or host reboot, platform components on that node cannot pull, while a manual `crictl pull` on the node works | Cause 6 | + +An anonymous `GET /v2/` that returns 401 only proves the port is open; do not rotate a password because of it. A successful login does not grant push or pull on every Harbor project. A Secret of type `kubernetes.io/dockerconfigjson` is namespace-scoped, and the kubelet matches its `auths` key to the registry host string in the image reference, including the port. + +A 401 that also hits a public project, follows a clock fault, is limited to one node, comes with a Harbor web UI `internal server error`, or comes from a pipeline or UI upload through a toolchain integration is covered in [Harbor returns 401 because of time skew, the node auth container, Harbor Core, or a shared integration credential](Harbor_returns_401_because_of_time_skew_the_node_auth_container_Harbor_Core_or_a_shared_integration_credential.md). + +## Root Cause + +### Cause 1: no credential matches the registry host + +The kubelet or CRI presents no Authorization header that matches this registry host. Usually the workload was never bound to a Harbor image credential. The same error appears when a Secret exists but its `auths` key is `192.0.2.10` while the image is `192.0.2.10:5000/team/app:1.0`: to the kubelet those are different registries. + +### Cause 2: the named pull Secret does not exist in the namespace + +The Pod or its ServiceAccount names a Secret that is absent from the workload namespace, or the object is not of type `kubernetes.io/dockerconfigjson`. After a platform upgrade the event can name `global-registry-auth`, the platform registry pull Secret, which is not created automatically in every new namespace. + +### Cause 3: the project-scoped account lacks permission on the private project + +On a private Harbor project, the image credential of a project member fails with `authentication failed`, while the same image pulls when the project is made public or when the admin credential is used. The blobs are present. The workload Secret holds the wrong user, or that user lacks pull (or push) permission on this project. Switching the workload to admin removes the permission boundary; it is a test, not the fix. + +### Cause 4: the repository used through port 11443 is outside the registry auth configuration + +Nodes pull from the global cluster's built-in registry with addresses like `:11443//:`. These pulls worked before and now fail, while the same image pulls without the port. The business cluster's registry configuration lists only the domain, without `11443`. When the repository used through `11443` is not in the registry auth configuration and anonymous access to it is not allowed, the node has no way to authenticate. + +### Cause 5: the Pod started without the ServiceAccount's imagePullSecrets + +Kubernetes copies a ServiceAccount's `imagePullSecrets` into the Pod spec only when the Pod is created. A Pod created while the ServiceAccount lacked the right Secret keeps running without it, even after the ServiceAccount is corrected. The pull fails with `ErrImagePull` and `ImagePullBackOff`, and the event text can be misleading (for example `server gave HTTP response to HTTPS client`). This can happen when `katanomi-controller` rewrites the namespace's `default` ServiceAccount, changing its `imagePullSecrets` list, around the time a Pod is created; it is a known product issue. + +### Cause 6: wrong registry credentials stored on the host after a reboot + +After a physical machine or node went down and restarted, some platform components on it cannot download images, while `crictl pull` of the image and `nerdctl login` to the registry VIP on the same node work. The registry credentials stored on the host for the registry domain are wrong. The kubelet reads node-level Docker config files (for example `/var/lib/kubelet/config.json` and `/root/.docker/config.json`) as registry credentials, and `nerdctl login` / `nerdctl logout` write the `.docker/config.json` of the user who runs them. Logging in to the registry domain with the password recorded in `/etc/kubernetes/registry/auth.yaml` on the global cluster makes the component pulls work; after a logout and a restart of the components, the issue does not return. + +## Resolution + +### Cause 1 + +1. Create the pull Secret in the workload namespace. `--docker-server` must be the host:port written in the image reference. + + ```bash + kubectl create secret docker-registry \ + --namespace= \ + --docker-server=: \ + --docker-username= \ + --docker-password="$HARBOR_PASSWORD" + kubectl get secret -n -o jsonpath='{.type}{"\n"}' + ``` + +2. Record the current `imagePullSecrets` of the Pod template, then attach the new Secret. + + > **Warning:** The patch below changes the Pod template and rolls the Deployment. A JSON merge patch would replace the whole `imagePullSecrets` array, so include every Secret the Pod still needs. Rollback: patch the array back to the recorded list and delete the added Secret if it should not remain. + + ```bash + kubectl get deploy/ -n -o jsonpath='{.spec.template.spec.imagePullSecrets}{"\n"}' + kubectl patch deploy/ -n --type=strategic \ + -p '{"spec":{"template":{"spec":{"imagePullSecrets":[{"name":""}]}}}}' + ``` + + To combine several registries into one Secret, see [Merge Multiple Image Pull Secrets Into a Single Kubernetes Docker Config Secret](Merge_Multiple_Image_Pull_Secrets_Into_a_Single_Kubernetes_Docker_Config_Secret.md). + +3. Verify with the runtime on a node, not only with Docker: + + ```bash + crictl pull ://: + ``` + +### Cause 2 + +1. Find which namespaces have the named Secret, and copy it into the workload namespace without decoding it. + + > **Warning:** Copying `global-registry-auth` lets Pods in the destination namespace pull whatever it authorizes, usually the platform registry. Precondition: that namespace is allowed to pull platform images; back up the destination object if one exists. Rollback: `kubectl delete secret global-registry-auth -n `, or re-apply the previous object. + + ```bash + kubectl get secret -A --field-selector metadata.name=global-registry-auth + kubectl get secret global-registry-auth -n -o json \ + | python3 -c 'import json,sys; o=json.load(sys.stdin); m=o["metadata"]; + [m.pop(k, None) for k in ("namespace","resourceVersion","uid","creationTimestamp","managedFields")]; + o.pop("status", None); json.dump(o, sys.stdout)' \ + | kubectl apply -n -f - + ``` + +2. For any other missing name, create it with the Cause 1 command instead of copying a platform Secret. Confirm the type is `kubernetes.io/dockerconfigjson` and that the Pod or its ServiceAccount lists that name. + +### Cause 3 + +1. Test as the project member: + + ```bash + printf '%s' "$HARBOR_PASSWORD" | docker login : --username --password-stdin + docker pull ://: + ``` + + If this fails, give that user or robot account pull permission on the private project (push permission for a push) in Harbor. Do not leave the project public and do not put the admin password in the workload Secret. + +2. Create a docker-registry Secret from the project member's username and password in the workload namespace, and point the workload at it (Pod template patch as in Cause 1): + + ```bash + kubectl create secret docker-registry \ + --namespace= \ + --docker-server=: \ + --docker-username= \ + --docker-password="$HARBOR_PASSWORD" + ``` + +### Cause 4 + +1. Confirm the split on a failing node: the TCP connection to `11443` opens, the pull with the port is refused, and the pull without the port succeeds. + + ```bash + nc -vz -w 3 11443 + crictl pull :11443//: + crictl pull //: + ``` + +2. Compare the registry entries in the business cluster's registry configuration with the address the workloads use (domain only versus domain with `11443`). + +3. Pulls through `11443` recover when anonymous access is enabled for that registry. This is a temporary measure. The lasting fix is to put that repository and address inside the registry auth configuration, or to point workloads at an address that is already covered; agree on it with the registry owner. + + > **Warning:** Enabling anonymous access lets any client that can reach the registry pull from it without credentials. Precondition: the split above is confirmed, the registry owner accepts unauthenticated pulls for the duration, and the images are not sensitive. Rollback: disable anonymous access again once the repository is covered by the auth configuration, then test a pull through `11443`. + +### Cause 5 + +1. Compare the ServiceAccount with the running Pod, and see which manager last wrote the ServiceAccount: + + ```bash + kubectl get sa -n -o jsonpath='{.imagePullSecrets[*].name}{"\n"}' + kubectl get pod -n -o jsonpath='{.spec.serviceAccountName}{" "}{.spec.imagePullSecrets}{"\n"}' + kubectl get sa -n --show-managed-fields -o yaml | grep -E 'manager:|time:' + ``` + +2. When the ServiceAccount now holds the correct Secret and the Pod spec lacks it, recreate the Pods so admission copies the current list. + + > **Warning:** Impact: the Pods are replaced; a single-replica workload is unavailable until the new Pod is Ready. Precondition: re-check right before that the ServiceAccount lists the correct Secret, because the controller may rewrite it. If the new Pod also lacks the Secret, the ServiceAccount changed again in between. + + ```bash + kubectl rollout restart deploy/ -n + kubectl get pod -n -l -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.imagePullSecrets}{"\n"}{end}' + ``` + +3. A Pod-template `imagePullSecrets` entry (Cause 1) does not depend on what the ServiceAccount holds at Pod creation, so it protects the workload from later rewrites. + +### Cause 6 + +1. On the affected node, list which registry hosts have stored credentials, without printing the secrets: + + ```bash + jq -r '.auths | keys[]' /var/lib/kubelet/config.json 2>/dev/null + jq -r '.auths | keys[]' /root/.docker/config.json 2>/dev/null + ``` + +2. Read the registry password from `/etc/kubernetes/registry/auth.yaml` on a control-plane node of the global cluster. On the affected node, back up the Docker config file and log in to the registry domain with it, then check the failing component Pods: + + ```bash + cp -a /root/.docker/config.json /root/docker-config.json.bak 2>/dev/null + printf '%s' "$REGISTRY_PASSWORD" | nerdctl login --username --password-stdin + kubectl get pods -A -o wide --field-selector spec.nodeName= | grep -E 'ImagePull|ErrImage' + ``` + +3. Log out of the registry domain and restart the platform component Pods that had failed. + + > **Warning:** Impact: `nerdctl logout` removes the stored credential for that host from the node's Docker config file, and the restarted Pods are interrupted until they are Ready. Precondition: the login in step 2 made the component pulls succeed, and `/root/docker-config.json.bak` exists. Rollback: copy the backup file back to `/root/.docker/config.json`. + + ```bash + nerdctl logout + kubectl delete pod -n + kubectl get pods -A -o wide --field-selector spec.nodeName= | grep -E 'ImagePull|ErrImage' + ``` + + The recreated Pods pull their images and stay `Running`. + +## Diagnostic Steps + +1. Save the first event, not the final `ImagePullBackOff` line, and compare the Pod's image and Secrets with its ServiceAccount: + + ```bash + kubectl describe pod -n + kubectl get pod -n -o jsonpath='{.spec.containers[*].image}{"\n"}{.spec.imagePullSecrets[*].name}{"\n"}' + kubectl get sa -n -o jsonpath='{.imagePullSecrets[*].name}{"\n"}' + ``` + + `FailedToRetrieveImagePullSecret` selects Cause 2. `no basic auth credentials` selects Cause 1: compare the image host:port with the Secret's registry key. A ServiceAccount that lists Secrets while the Pod spec has none selects Cause 5, whatever the pull error says. + +2. From the failing node, separate reachability from authentication: + + ```bash + curl -sS -o /dev/null -D - --connect-timeout 10 https://:/v2/ + printf '%s' "$HARBOR_PASSWORD" | docker login : --username --password-stdin + crictl pull ://: + ``` + + A TLS error or `server gave HTTP response to HTTPS client` happens before authentication; see [Harbor push or pull fails because the client uses the wrong scheme: HTTP response to HTTPS client, or port 443 refused](Harbor_push_or_pull_fails_because_the_client_uses_the_wrong_scheme_HTTP_response_to_HTTPS_client_or_port_443_refused.md) or [Harbor or registry image pull fails with x509 certificate signed by unknown authority or certificate has expired](Harbor_or_registry_image_pull_fails_with_x509_certificate_signed_by_unknown_authority_or_certificate_has_expired.md). + +3. On a private project, if the member's credential fails and admin or a public project works, select Cause 3. If a pull from a public project also returns 401, check clocks first (see the related article under Issue). + +4. If the same image pulls without `:11443` and fails with it, select Cause 4. + +5. If only platform components on a node that was just rebooted fail, and `crictl pull` on that node succeeds, select Cause 6. diff --git a/docs/en/solutions/Harbor_image_scan_fails_because_Trivy_cannot_reach_mirror.gcr.io.md b/docs/en/solutions/Harbor_image_scan_fails_because_Trivy_cannot_reach_mirror.gcr.io.md new file mode 100644 index 000000000..342ebd1be --- /dev/null +++ b/docs/en/solutions/Harbor_image_scan_fails_because_Trivy_cannot_reach_mirror.gcr.io.md @@ -0,0 +1,97 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Harbor image scan fails because Trivy cannot reach mirror.gcr.io + +## Issue + +A Harbor vulnerability scan fails and the UI shows no scan result, while push and pull of the same tag succeed. The Trivy Pod log shows that it cannot update the vulnerability database from `mirror.gcr.io` (a timeout, a connection reset or refusal, or a TLS error naming that host). The cluster is typically in an intranet that blocks outbound access. + +## Root Cause + +### Cause 1: Trivy cannot download its vulnerability database from mirror.gcr.io + +Harbor 2.14 (for example `v2.14.1`) uses the online Trivy vulnerability database by default; the scanner image does not carry a current database. On each update Trivy connects to `mirror.gcr.io` on port 80 or 443. If that connection times out or is reset, the database refresh fails and the scan returns an error. The image layers in the registry are fine, and opening the Harbor UI or restarting Harbor Core does not fetch the database. + +An intranet cluster that blocks those ports cannot scan until egress to `mirror.gcr.io` is allowed or an offline database is loaded. + +## Resolution + +### Cause 1 + +Choose one of the two repairs. Prefer egress when the security policy allows it; use the offline database when the cluster must stay without that path. + +**Option A: allow egress to mirror.gcr.io** + +1. Confirm the block from the Trivy Pod: + + ```bash + kubectl get pods -A | grep -i trivy + kubectl exec -n -- wget -S -O /dev/null --timeout=10 https://mirror.gcr.io/ 2>&1 | head -n 20 + ``` + + If `wget` is not in the image, run the same request with `curl` from a debug Pod on the same network. A timeout confirms the block. + +2. Allow `mirror.gcr.io` on ports 80 and 443 from the Trivy Pod in the firewall or proxy. + +3. Restart the Trivy Pod so it retries the download. + + > **Warning:** Only after egress is in place, otherwise the new Pod fails the same way. Impact: scans in progress are aborted; no images are deleted. The controller recreates the Pod; to undo the firewall change, restore the previous egress rule. + + ```bash + kubectl delete pod -n + ``` + +4. Trigger the scan again in Harbor. A completed scan with a vulnerability report means the database downloaded. + +**Option B: use an offline vulnerability database** + +Follow "Update the Trivy Vulnerability Database in Offline Environments" in the Alauda Build of Harbor documentation for the installed Harbor version. In outline: download the Trivy database on a machine that can reach `mirror.gcr.io`, place it in the volume the Trivy Pod mounts, and turn off online database updates through the `Harbor` resource so the scanner does not try `mirror.gcr.io` again. + +1. Save the Harbor resource and find the volume the Trivy Pod mounts: + + ```bash + kubectl get harbor -n -o yaml > /tmp/harbor-before.yaml + kubectl get harbor -n -o jsonpath='{.spec.helmValues.trivy}{"\n"}' + kubectl get pod -n -o jsonpath='{range .spec.volumes[*]}{.name}{" "}{.persistentVolumeClaim.claimName}{"\n"}{end}' + ``` + +2. Change the Trivy update settings on the `Harbor` resource as the documentation describes. Do not edit the generated Trivy Deployment: the operator reverts it on the next reconcile. + + > **Warning:** Impact: editing the Harbor resource rolls Harbor components, which interrupts login and pushes until they are Ready. Copying a database file into the volume overwrites the previous one. Precondition: a maintenance window, `/tmp/harbor-before.yaml` saved, and a copy of the old database file if you may need it. Rollback: apply the saved file and restore the old database file. + + ```bash + kubectl edit harbor -n + ``` + +3. After the database is on the volume and online updates are off, restart the Trivy Pod and run a scan. If the log still dials `mirror.gcr.io`, the update setting did not reach the running scanner. + +## Diagnostic Steps + +1. Separate a scan failure from a pull failure. If `crictl pull` of the same tag works, credentials and the manifest are not the problem. + +2. Read the Trivy log for the database host: + + ```bash + kubectl logs -n --tail=100 + ``` + + A timeout, connection refusal, or TLS error naming `mirror.gcr.io` selects this article. An error naming the Harbor Core URL is a scanner-to-core problem, not the public database. + +3. Test egress from the Trivy Pod (Option A, step 1). Success from a workstation and failure from the Pod means the cluster egress policy is the block. + +4. Check the Harbor version and the current Trivy settings: + + ```bash + kubectl get harbor -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,VERSION:.spec.version + kubectl get harbor -n -o jsonpath='{.spec.helmValues.trivy}{"\n"}' + ``` + +5. After either repair, scan a second small image and confirm the UI stores a result. A single cached report does not prove that the update path works. diff --git a/docs/en/solutions/Harbor_or_registry_image_pull_fails_with_x509_certificate_signed_by_unknown_authority_or_certificate_has_expired.md b/docs/en/solutions/Harbor_or_registry_image_pull_fails_with_x509_certificate_signed_by_unknown_authority_or_certificate_has_expired.md new file mode 100644 index 000000000..abd56c666 --- /dev/null +++ b/docs/en/solutions/Harbor_or_registry_image_pull_fails_with_x509_certificate_signed_by_unknown_authority_or_certificate_has_expired.md @@ -0,0 +1,128 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Harbor or registry image pull fails with x509 certificate signed by unknown authority or certificate has expired + +## Issue + +An image pull or push to Harbor or another HTTPS registry completes the TCP connection but fails the TLS check, before any permission check. The client (`crictl`, the kubelet, Docker, Buildah, Skopeo) reports one of: + +```text +x509: certificate signed by unknown authority +x509: certificate has expired or is not yet valid +``` + +The failure can affect every node right after a containerd configuration change, or only one node, where `crictl pull` fails while `nerdctl pull` of the same image succeeds. + +Related articles: [Harbor push or pull fails because the client uses the wrong scheme: HTTP response to HTTPS client, or port 443 refused](Harbor_push_or_pull_fails_because_the_client_uses_the_wrong_scheme_HTTP_response_to_HTTPS_client_or_port_443_refused.md). + +## Root Cause + +### Cause 1: the client does not trust the CA that signed the registry certificate + +The HTTPS listener presents a certificate that the client does not trust. On a node this is usually one of: + +- A private CA that was never added to containerd's host configuration under `/etc/containerd/certs.d/:/`. +- A regenerated `/etc/containerd/config.toml` that dropped `config_path`. Running `containerd config default` and writing the result over `/etc/containerd/config.toml` makes the next pull from the platform registry fail with this error, because the host configuration is no longer loaded. +- A hand-added Harbor block in `config.toml` that replaced the trust configuration of the platform registry. +- One node that lacks the `/etc/containerd/certs.d/:/hosts.toml` that the other nodes have. `crictl pull` on that node reports x509 while `nerdctl pull` of the same image succeeds. + +`registry.alauda.cn:60080` is the in-cluster registry name used by ACP workloads; it is not an insecure HTTP endpoint. + +### Cause 2: the certificate is outside its validity window, or the node clock is + +The certificate presented for the registry hostname has expired or is not yet valid, or the node clock is outside the certificate's validity window. Putting the expired certificate into the trust store does not help, and skipping verification hides the expiry and leaves every pull on that node unprotected. + +## Resolution + +### Cause 1 + +1. Put the CA that signed the registry certificate into the containerd host directory and name it in `hosts.toml`. The directory name must match the image reference, including a non-default port. + + ```toml + server = "https://" + + [host."https://"] + capabilities = ["pull", "resolve", "push"] + ca = "/etc/containerd/certs.d//ca.crt" + ``` + + When `hosts.toml` exists, containerd does not load a `*.crt` file from that directory unless `hosts.toml` names it with `ca`. A `hosts.toml` without a `ca` line clears the error only if the node already trusts the CA in its system trust store. + +2. If only one node fails and other nodes pull, copy the `/etc/containerd/certs.d/:/` directory from a healthy node to the failing node, then retry `crictl pull`. containerd 1.5 and later read a new or changed `hosts.toml` on the next pull without a restart, as long as `config_path` is already loaded. + +3. If `config_path` is missing because `config.toml` was replaced or hand-edited, restore it from a healthy node of the same platform version, including any platform registry host entries. The `config_path` line for containerd 1.x is under `[plugins."io.containerd.grpc.v1.cri".registry]`; for containerd 2.x under `[plugins."io.containerd.cri.v1.images".registry]`. Do not regenerate the file with `containerd config default`. Then restart containerd. + + > **Warning:** Impact: restarting containerd interrupts the CRI socket on the node; the kubelet cannot start, stop, or probe containers meanwhile and the node can report `NotReady`. With the standard unit (`KillMode=process`, check with `systemctl show containerd -p KillMode`) running containers keep running; with `control-group` or `mixed` every container on the node stops. Precondition: copy `/etc/containerd/config.toml` and `/etc/containerd/certs.d` aside, and drain the node in a maintenance window. Rollback: restore the copies and restart containerd again. + + ```bash + kubectl drain --ignore-daemonsets --delete-emptydir-data + systemctl restart containerd + kubectl uncordon + ``` + +4. For a Docker client, add the CA as `/etc/docker/certs.d//ca.crt` instead of adding the host to `insecure-registries`. + +### Cause 2 + +1. Compare node time with the certificate dates before replacing anything: + + ```bash + date -u + echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -dates -subject -ext subjectAltName + ``` + +2. If `notBefore` is in the future or `notAfter` is in the past only because the node clock is wrong, correct NTP first. See [Configuring chrony NTP on Cluster Nodes](Configuring_chrony_NTP_on_Cluster_Nodes.md). + + > **Warning:** Stepping the clock on a control-plane node can stall etcd and invalidate in-flight tokens. Precondition: a maintenance window; correct one control-plane node at a time. ACP etcd runs as the static Pod `etcd-` in `kube-system` (label `component=etcd`); before the next node, confirm that Pod is `Running` and that its log has no `clock difference against peer` line. The clock cannot be rolled back to the wrong time; point chrony at the previous source if the new source is wrong. + + ```bash + kubectl get pods -n kube-system -l component=etcd -o wide + kubectl -n kube-system logs -c etcd etcd- --tail=50 | grep -i "clock difference" + ``` + +3. If the certificate is expired against a correct clock, replace the certificate served for that hostname, with a new certificate for the same name prepared in advance. When the registry hostname is the platform access address, the certificate is the platform access certificate; replace it with the procedure in the ACP documentation. Confirm the `openssl` command from step 1 shows the new dates, then retry the pull. Do not add the expired certificate to `ca.crt`, and do not use `skip_verify` as the lasting fix. + + If an intermediate CA in the chain is what expired, see [Diagnosing x509 certificate has expired caused by an expired CA in the chain](Diagnosing_x509_certificate_has_expired_caused_by_an_expired_CA_in_the_chain.md). + +## Diagnostic Steps + +1. Inspect the certificate the port presents: + + ```bash + echo | openssl s_client -connect :443 -servername -showcerts + ``` + + `Verify return code: 19`, `20`, or `21` (self-signed certificate in the chain, unknown local issuer, or first certificate not verifiable) points to Cause 1; dates outside the current time point to Cause 2. + +2. See what containerd uses for this host and compare with a node that can pull: + + ```bash + grep -n "config_path\|certs.d" /etc/containerd/config.toml + containerd config dump | grep -n config_path + ls -l /etc/containerd/certs.d/:/ + sed -n '1,80p' /etc/containerd/certs.d/:/hosts.toml + ``` + + Expected on a healthy node: `config_path = "/etc/containerd/certs.d"` and a `hosts.toml` with a `ca` line for a private CA. + +3. Compare runtimes on the failing node: + + ```bash + crictl pull ://: + nerdctl pull ://: + ``` + + `crictl` failing with x509 while `nerdctl` succeeds, on a node without the registry's `hosts.toml`, is the single-node form of Cause 1. + +4. If the error is `server gave HTTP response to HTTPS client` or `connection refused` on 443 instead of x509, see the related article under Issue. diff --git a/docs/en/solutions/Harbor_push_or_pull_fails_because_the_client_uses_the_wrong_scheme_HTTP_response_to_HTTPS_client_or_port_443_refused.md b/docs/en/solutions/Harbor_push_or_pull_fails_because_the_client_uses_the_wrong_scheme_HTTP_response_to_HTTPS_client_or_port_443_refused.md new file mode 100644 index 000000000..481291ffb --- /dev/null +++ b/docs/en/solutions/Harbor_push_or_pull_fails_because_the_client_uses_the_wrong_scheme_HTTP_response_to_HTTPS_client_or_port_443_refused.md @@ -0,0 +1,201 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Harbor push or pull fails because the client uses the wrong scheme: HTTP response to HTTPS client, or port 443 refused + +## Issue + +`docker login`, `docker push`, `crictl pull`, Buildah, Skopeo, Jenkins, or a pipeline reaches a Harbor or registry address and fails before any repository permission check. A request such as "images cannot be imported into the image registry" often lands here: run `docker login :` on the host that pushes and read the first error. + +| Client error | Cause | +| --- | --- | +| `Error response from daemon: Get "https://:/v2/": http: server gave HTTP response to HTTPS client` | Cause 1 | +| After the cluster comes up, most Pods cannot pull from an HTTP registry, and `/etc/containerd/certs.d` has no directory for that registry | Cause 1 | +| Login, or Skopeo authentication, fails although the registry port is open; the token realm points at a different scheme or host | Cause 2 | +| `dial tcp :443: connect: connection refused` while the builder always uses HTTPS | Cause 3 | + +The kubelet and `crictl` use containerd. A `daemon.json` change does not affect them, and `docker login` does not test the runtime that starts Pods. For x509 errors, see [Harbor or registry image pull fails with x509 certificate signed by unknown authority or certificate has expired](Harbor_or_registry_image_pull_fails_with_x509_certificate_signed_by_unknown_authority_or_certificate_has_expired.md). + +## Root Cause + +### Cause 1: the registry serves HTTP and the client uses HTTPS + +containerd, Docker, Buildah, and Skopeo open an HTTPS connection when the image reference has no scheme. If Harbor is published as HTTP, the TCP connection succeeds and the client aborts as soon as it reads a clear-text HTTP response. Adding `skip_verify` does not change the scheme. containerd can try HTTPS first even when an HTTP fallback exists, and a Docker pull succeeds only when the reference includes the HTTP port that Harbor actually listens on. A `daemon.json` edit takes effect only after `dockerd` restarts, has no effect when it names a different host or port than the image reference, and never applies to containerd. + +The same missing host configuration shows up right after a cluster starts: most Pods cannot pull, and `/etc/containerd/certs.d` has no directory for the registry those images use. Adding the registry host configuration restores the pulls. Nodes can also differ: one control-plane node pulls the HTTP registry while the others cannot, because their containerd host configuration differs. Pods that stay in `CrashLoopBackOff` after pulls work are application faults, not a second registry fault. + +### Cause 2: the Harbor external URL scheme does not match the client + +Harbor builds its token and redirect responses from its external URL. When the external URL is `http` while containerd is configured for `https` (or the reverse), or a redirect proxy in front of Harbor rewrites the host, the client follows the token realm to a URL it did not authenticate against, and the failure looks like a bad password. The same mistake leaves the UI unreachable after a new Harbor instance is deployed with an access address that clients do not use, while the Pods stay `Running`. + +### Cause 3: nothing listens on port 443 + +After a platform upgrade, Jenkins and BuildKit can keep requesting `https://:443` for a local Harbor that does not listen there. The build log shows `connect: connection refused` while loading base-image metadata. This is not a certificate or permission problem. + +containerd can report the same line, `failed to do request: Head "https:///v2/.../manifests/": dial tcp :443: connect: connection refused`, when the requested tag does not exist in the registry. Check that the tag exists before changing the builder or the Harbor listener. + +## Resolution + +### Cause 1 + +1. On every node that pulls, create a host directory that matches the image reference exactly, including a non-default port. For an image `192.0.2.10:5000/team/app:1.0` the file is `/etc/containerd/certs.d/192.0.2.10:5000/hosts.toml`: + + ```toml + server = "http://192.0.2.10:5000" + + [host."http://192.0.2.10:5000"] + capabilities = ["pull", "resolve", "push"] + ``` + +2. Check that `/etc/containerd/config.toml` loads that directory: + + ```bash + grep -n "config_path" /etc/containerd/config.toml + ``` + + containerd 1.x (`io.containerd.grpc.v1.cri` present): + + ```toml + [plugins."io.containerd.grpc.v1.cri".registry] + config_path = "/etc/containerd/certs.d" + ``` + + containerd 2.x (`version = 3` or `io.containerd.cri.v1.images` present): + + ```toml + [plugins."io.containerd.cri.v1.images".registry] + config_path = "/etc/containerd/certs.d" + ``` + + Change `config.toml` only to add a missing `config_path` that a healthy node of the same platform version already has. Do not regenerate it with `containerd config default`, and do not add `registry.mirrors` or `configs` blocks for the registry: a hand-added Harbor mirror block can break pulls from the platform registry. For extra insecure registries, the platform procedure is "Configure trust for insecure image registries" in the ACP documentation. + +3. Retry `crictl pull` on the node. When `config_path` was already loaded and only a `hosts.toml` was added or changed, containerd 1.5 and later read it on the next pull without a restart. A `config.toml` change needs a containerd restart. + + > **Warning:** Only when `config.toml` was changed. Impact: restarting containerd interrupts the CRI socket on the node; the kubelet cannot start, stop, or probe containers meanwhile and the node can report `NotReady`. With the standard unit (`KillMode=process`, check with `systemctl show containerd -p KillMode`) running containers keep running; with `control-group` or `mixed` every container on the node stops. Precondition: drain the node, one node at a time, in a maintenance window, after copying `/etc/containerd/config.toml` and `/etc/containerd/certs.d` aside. Rollback: restore the copies and restart containerd again. + + ```bash + kubectl drain --ignore-daemonsets --delete-emptydir-data + systemctl restart containerd + kubectl uncordon + ``` + +4. Docker clients ignore `hosts.toml`. Put the same host:port in `insecure-registries` in `/etc/docker/daemon.json` and use that host:port in the image reference; a reference without the port still uses 443. + + ```json + { + "insecure-registries": ["192.0.2.10:5000"] + } + ``` + + > **Warning:** `insecure-registries` turns off TLS verification for that host:port and allows clear-text pulls; limit it to the HTTP registry. Impact: restarting Docker stops containers that run under `dockerd`. Precondition: a copy of `daemon.json` and a maintenance window. Rollback: restore the copy and restart Docker. This does not change kubelet pulls. + + ```bash + cp -a /etc/docker/daemon.json /root/daemon.json.bak + systemctl restart docker + ``` + + After the restart, the first `docker login` can return `unauthorized: authentication required`; entering the credentials again returns `Login Succeeded`, and `docker tag` and `docker push` then complete. Retry the login once before treating it as a credential fault. + +5. Skopeo and Buildah default to HTTPS. For an HTTP Harbor, pass the TLS flags. If a pipeline task hides the CLI, set the task's TLS parameter so it adds them. + + > **Warning:** These flags disable certificate checks for that command. Use them for an HTTP registry only, not as a default for an HTTPS registry. Rollback: run the command without the flags. + + ```bash + skopeo copy --src-tls-verify=false --dest-tls-verify=false \ + docker://192.0.2.10:5000/team/app:1.0 docker://192.0.2.10:5000/team/app:1.0 + buildah pull --tls-verify=false docker://192.0.2.10:5000/team/app:1.0 + buildah push --tls-verify=false docker://192.0.2.10:5000/team/app:1.0 + ``` + +### Cause 2 + +1. Read and save the Harbor resource: + + ```bash + kubectl get harbor -A + kubectl get harbor -n -o yaml > /tmp/harbor-before.yaml + ``` + +2. Set the external URL to the scheme, host, and port that clients actually open. If a redirect proxy rewrites the host, use the URL the client uses, not an internal Service URL. After a reinstall, set the access address back to the address clients have always used. + + > **Warning:** Saving the Harbor resource makes the operator roll Harbor components; login, push, and pull pause until the Pods are Ready. Precondition: a maintenance window and `/tmp/harbor-before.yaml`. Rollback: apply that file. + + ```bash + kubectl edit harbor -n + ``` + +3. Point containerd at the same URL in `hosts.toml`. Prefer a `ca` entry with the registry CA (see the x509 article linked under Issue). Use `skip_verify = true` only when the CA is not available and `openssl s_client` from the node has shown which certificate the port presents. + + > **Warning:** `skip_verify = true` accepts any certificate for that host. Precondition: copy `/etc/containerd/certs.d` aside first. Rollback: restore the copy. + + ```toml + server = "https://:443" + + [host."https://:443"] + capabilities = ["pull", "resolve", "push"] + skip_verify = true + ``` + +### Cause 3 + +1. Confirm what is listening: + + ```bash + curl -sv --connect-timeout 10 https:///v2/ + curl -sv --connect-timeout 10 http://:/v2/ + ``` + +2. If 443 is refused and the HTTP port returns `401 Unauthorized` (the anonymous probe reached the registry), point Jenkins or BuildKit at `http://:` and allow HTTP for that host in `buildkitd.toml`: + + ```toml + [registry.":"] + http = true + ``` + + Or publish Harbor on 443. Do not add a TLS skip flag: nothing presents a certificate on the refused port. + +3. Retry the build with the same base-image reference. + +## Diagnostic Steps + +1. Record the exact image reference, the client, the node, and the first error line. Keep the port: `registry.example:5000` and `registry.example` are different registry keys. + +2. From the failing node, probe both schemes without `-k`: + + ```bash + curl -sv --connect-timeout 10 https://:/v2/ + curl -sv --connect-timeout 10 http://:/v2/ + ``` + + `401 Unauthorized` on one scheme means that scheme reached a registry; it is not evidence of a wrong password. `server gave HTTP response to HTTPS client` selects Cause 1. `connection refused` on 443 selects Cause 3. An `x509` line belongs to the x509 article. + +3. See what containerd uses for this host, and compare with a node that can pull: + + ```bash + containerd --version + grep -n "config_path\|certs.d" /etc/containerd/config.toml + sed -n '1,80p' /etc/containerd/certs.d/:/hosts.toml + containerd config dump | grep -n config_path + ``` + +4. Separate runtime pulls from Docker pulls: + + ```bash + crictl pull ://: + docker pull ://: + ``` + + `crictl pull` failing while `docker pull` works means the node runtime configuration is wrong: fix `hosts.toml`, not `daemon.json`. + +5. For a pipeline Skopeo or Buildah push, read the task log for the registry URL it contacted. An authentication error that names a different host than the image reference selects Cause 2, even when TLS succeeded. + +6. If a 401 remains after the scheme is correct, see [Harbor image pull returns 401 because of the pull Secret, the project account, or node credentials](Harbor_image_pull_returns_401_because_of_the_pull_Secret_the_project_account_or_node_credentials.md). diff --git a/docs/en/solutions/Harbor_push_returns_HTTP_500_on_an_S3_or_Huawei_OBS_bucket_empty_access_keys_then_endpoint_certificate_verification.md b/docs/en/solutions/Harbor_push_returns_HTTP_500_on_an_S3_or_Huawei_OBS_bucket_empty_access_keys_then_endpoint_certificate_verification.md new file mode 100644 index 000000000..3a7fa86a2 --- /dev/null +++ b/docs/en/solutions/Harbor_push_returns_HTTP_500_on_an_S3_or_Huawei_OBS_bucket_empty_access_keys_then_endpoint_certificate_verification.md @@ -0,0 +1,125 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Harbor push returns HTTP 500 on an S3 or Huawei OBS bucket: empty access keys, then endpoint certificate verification + +## Issue + +The Harbor registry stores images in an S3-compatible bucket (for example a Huawei OBS public bucket), and pushes fail: + +- `docker push` or `podman push` to Harbor fails with an HTTP `500` status. +- Image synchronization (replication) tasks fail at the push step with a `500` error. +- The bucket is public, so the access key and secret key were left empty. +- After the access key and secret key are filled in, the registry reports a certificate verification failure when it connects to S3. + +The two causes below are usually met in sequence: clear Cause 1 first; Cause 2 is the error that can appear once Cause 1 is fixed. A push that returns 500 on NFS or NAS storage, without an S3 backend, is a different problem. + +## Root Cause + +### Cause 1: the S3 access key and secret key are empty + +The S3 credential Secret referenced by the Harbor resource contains the keys `REGISTRY_STORAGE_S3_ACCESSKEY` and `REGISTRY_STORAGE_S3_SECRETKEY`, but both values are empty, because the bucket is public. A public bucket can be public for reads (GET) only, while a push writes objects, so pushes fail with a `500`. Filling in a valid access key and secret key fixes it: even for a public bucket, the registry needs keys that can write to it. + +### Cause 2: the registry fails certificate verification against the S3 endpoint + +With the keys in place, the registry fails certificate verification when it connects to the S3 endpoint over HTTPS. When the endpoint also serves plain HTTP (the Huawei OBS console lists `HTTPS/HTTP` as the supported protocols of the Endpoint and the bucket access domain), changing the registry's S3 endpoint (`regionendpoint`) from `https://` to `http://` restores image push and synchronization. + +## Resolution + +### Cause 1 + +1. Find the Secret the Harbor resource uses for S3 and check whether its values are empty, without printing them: + + ```bash + kubectl -n get harbor -o yaml | grep -n -E 'imageChartStorage|type: s3|existingSecret|bucket|regionendpoint' + kubectl -n get secret -o jsonpath='{.data.REGISTRY_STORAGE_S3_ACCESSKEY}' | wc -c + kubectl -n get secret -o jsonpath='{.data.REGISTRY_STORAGE_S3_SECRETKEY}' | wc -c + ``` + + `0` means the value is empty or missing. If the live Secret uses different key names, compare it with a Harbor whose S3 pushes succeed and set the keys under the names that Secret already has. + +2. Write an access key and secret key with write access to the bucket into the Secret. + + > **Warning:** Only when a key value is empty. Impact: the registry presents new credentials to S3; a wrong key repeats the 500. Precondition: an access key and secret key that can write to the bucket. Rollback: `kubectl apply -f /tmp/s3-secret-before.yaml`. The 500 itself did not delete any blob. + + ```bash + kubectl -n get secret -o yaml > /tmp/s3-secret-before.yaml + kubectl -n create secret generic \ + --from-literal=REGISTRY_STORAGE_S3_ACCESSKEY='' \ + --from-literal=REGISTRY_STORAGE_S3_SECRETKEY='' \ + --dry-run=client -o yaml | kubectl apply -f - + ``` + +3. Restart the registry so it reads the new values. + + > **Warning:** Impact: pushes and pulls are interrupted until the new Pods are Ready. The Deployment recreates the Pods on the same spec; nothing needs to be rolled back. + + ```bash + kubectl -n get deploy -o name | grep harbor-registry + kubectl -n rollout restart deployment + kubectl -n rollout status deployment + ``` + +4. Retry the push or the replication task. If the 500 is gone and the new error is certificate verification, continue with Cause 2. + +### Cause 2 + +1. Confirm that the endpoint serves plain HTTP (see Diagnostic Steps, step 3, or the provider console), then save the Harbor resource: + + ```bash + kubectl -n get harbor -o yaml > /tmp/harbor-s3-endpoint.yaml + grep -n regionendpoint /tmp/harbor-s3-endpoint.yaml + ``` + +2. Change only the scheme of `regionendpoint` to `http://`, at the path where `grep` found it, keeping the host and any port. + + > **Warning:** Only when certificate verification is the remaining error, the keys are present, and the endpoint serves plain HTTP. Impact: the access key and the image layers travel without TLS, readable by anyone on that network path; the operator reconciles and the registry restarts. Rollback: `kubectl apply -f /tmp/harbor-s3-endpoint.yaml`. If the provider does not serve plain HTTP, this repair does not apply. + + ```bash + kubectl -n edit harbor + ``` + + ```yaml + spec: + helmValues: + persistence: + imageChartStorage: + type: s3 + s3: + regionendpoint: http:// + ``` + +3. If the registry Pods do not pick up the change, restart them as in Cause 1, step 3. Then retry the push and the replication task. + +## Diagnostic Steps + +1. Confirm the registry backend is S3 and read the registry log around a failed push: + + ```bash + kubectl -n get harbor -o yaml | grep -n -E 'type: s3|existingSecret|regionendpoint' + kubectl -n logs deploy/ --all-containers --tail=300 | grep -i -E 's3|certificate|x509|denied|500' + ``` + + The registry reports a certificate verification failure with an `x509:` prefix. + +2. Push returns 500 and either S3 key in the Secret has length 0: Cause 1. + +3. Keys present and the registry log shows certificate verification failing against the S3 endpoint: Cause 2. Check which protocols the endpoint serves, from a node the registry routes through: + + ```bash + curl -sS -o /dev/null -w '%{http_code}\n' https:// + curl -sS -o /dev/null -w '%{http_code}\n' http:// + ``` + + A TLS error on the first command and an HTTP status on the second match Cause 2. + +4. After either fix, retry the failed replication task or push one small tag, then pull it back. + +To move an existing registry from a PVC to S3 storage, see [How to migrate harbor registry pvc storage to s3](How_to_migrate_harbor_registry_pvc_storage_to_s3.md). diff --git a/docs/en/solutions/Harbor_returns_401_because_of_time_skew_the_node_auth_container_Harbor_Core_or_a_shared_integration_credential.md b/docs/en/solutions/Harbor_returns_401_because_of_time_skew_the_node_auth_container_Harbor_Core_or_a_shared_integration_credential.md new file mode 100644 index 000000000..929144804 --- /dev/null +++ b/docs/en/solutions/Harbor_returns_401_because_of_time_skew_the_node_auth_container_Harbor_Core_or_a_shared_integration_credential.md @@ -0,0 +1,191 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Harbor returns 401 because of time skew, the node auth container, Harbor Core, or a shared integration credential + +## Issue + +Pulls or pushes that worked before now fail with `401 Unauthorized` or `unauthorized`, and the workload's credentials did not change. Common forms: the rejection also hits a public project such as `library`, only one node is affected, the Harbor web UI shows `internal server error`, only the pipeline fails, or an image upload from the platform UI through a toolchain integration fails with: + +```text +upload file error: writing blob: initiating layer upload to /v2///blobs/uploads/ in :: unauthorized: unauthorized to access repository: /, action: push +``` + +while the same account pushes from the command line. + +| Signature | Cause | +| --- | --- | +| `401 Unauthorized` on pull and push, including a pull from a public project, and node clocks differ; or a push that recovers after time is synchronized | Cause 1 | +| Pipeline log says image registry credentials were not found, and the imageRepository integration is abnormal | Cause 2 | +| UI image upload through a toolchain integration returns `500` with `unauthorized to access repository: ..., action: push`, and the integration's credential is also used by another integration | Cause 3 | +| `401` on one node only; other nodes pull the same reference | Cause 4 | +| Harbor web UI keeps showing `internal server error`, and pulls that worked before (even from public `library`) now say unauthorized | Cause 5 | + +A public Harbor project such as `library` lets anyone pull, but a push to it still needs an account with push permission. A refused push to `library` is not, on its own, a sign of clock skew or of a Harbor Core problem; only a refused pull from it is. + +For a missing or wrong pull Secret, a project-scoped account, port `11443`, or credentials stored on the node, see [Harbor image pull returns 401 because of the pull Secret, the project account, or node credentials](Harbor_image_pull_returns_401_because_of_the_pull_Secret_the_project_account_or_node_credentials.md). + +## Root Cause + +### Cause 1: control-plane clocks are skewed + +Kubernetes tokens carried by the DevOps operator fall outside their validity window because the control-plane clock is wrong. Pulls and pushes fail with `401 Unauthorized`, or the pipeline keeps asking for credentials, even against a public project. etcd on ACP runs as the static Pod `etcd-` in `kube-system` (label `component=etcd`); its log shows `rafthttp: the clock difference against peer ... is too high` when members disagree by more than one second. Restarting Jenkins or the operator before the clock is corrected does not produce a usable token. The same clock fault can stop image pushes without any `401` message; synchronizing time restores them. + +A Docker message `Stored credentials invalid or expired` is the local credential store rejecting a saved entry; it is not proof that the registry certificate expired. + +### Cause 2: the global and workload clusters share one Harbor integration credential + +The pipeline `image-build` stage returns `unauthorized`, reports that image registry credentials were not found, and the imageRepository integration status is abnormal. Assigning the project again makes pulls work once, then the next run fails. When the global cluster and a workload cluster share one Harbor integration credential, the two conflict, and `katanomi` on the global cluster cannot hand the pipeline a usable registry credential. Sharing one credential between integrations is not supported. + +### Cause 3: one credential is shared by several toolchain integrations + +The UI upload goes to the image registry of a toolchain integration (request path `/files/v1alpha1/integrations//namespaces//projects//upload`). The response is `StatusCode: 500` with `reason: InternalError` and the `action: push` message; the `via: alb` and `x-alb-err-reason: BackendError : read byte data from backend` headers only show the ALB passing on the backend's `500`. + +A toolchain integration card does not support sharing a single credential. When one credential is shared across integrations, and the registry address in its data is not the address of the integration that uploads, the upload fails with the permission error above, although the account itself can push from the command line and `harbor-core` logs no rejection of that account. + +### Cause 4: the node-local auth container on one node + +Platform components return `401` only on one node. The image is absent on that node, the containerd configuration matches a healthy node, and a manual pull succeeds after the node-local registry `auth` container is restarted. Other nodes are unaffected, so this is not a Harbor project permission and not a cluster-wide Secret. + +### Cause 5: Harbor Core is unstable + +The Harbor web page frequently shows `internal server error`, and pipelines that used to build fail at the image pull with `unauthorized`, even for an image in the public `library` project. Credentials and project permissions did not change. Restarting Harbor Core brings back both the web page and the pulls. The reason Core became unstable has not been identified. + +## Resolution + +### Cause 1 + +1. On each control-plane node, check time synchronization, and read the etcd clock warning: + + ```bash + timedatectl status + chronyc tracking + chronyc sources + kubectl get pods -n kube-system -l component=etcd -o wide + kubectl -n kube-system logs -c etcd etcd- --tail=80 | grep -F "clock difference against peer" + ``` + +2. Correct the clock. + + > **Warning:** `chronyc makestep` jumps the clock. On a control-plane node that can stall etcd and invalidate in-flight tokens. Precondition: a maintenance window, and `chronyc sources` shows a reachable NTP server. Step one control-plane node at a time, and confirm `etcd-` is `Running` and the clock-difference warning has stopped before the next node. The clock cannot be rolled back to the wrong time; if the new source is wrong, restore the previous NTP server in the chrony configuration. + + ```bash + chronyc makestep + ``` + +3. After time is correct, restart the DevOps tools operator so it requests a new token. The Deployment name varies; select it from the list. + + > **Warning:** Only after the clocks agree. Impact: DevOps tool reconciliation pauses until the new Pod is Ready. The Deployment recreates the Pod; nothing needs to be rolled back. + + ```bash + kubectl get deploy -A | grep -Ei 'devops.*tool|tools-operator' + kubectl rollout restart deploy/ -n + kubectl rollout status deploy/ -n + ``` + +4. Retry the pipeline pull and push. A pull from a public project must succeed without a new password. If Docker still says `Stored credentials invalid or expired`, clear only that host and log in again: + + ```bash + docker logout : + printf '%s' "$HARBOR_PASSWORD" | docker login : --username --password-stdin + ``` + +### Cause 2 + +1. Find the global `katanomi` Deployment and record its replica count: + + ```bash + kubectl get deploy -A | grep -i katanomi + kubectl get deploy -n -o jsonpath='{.spec.replicas}{"\n"}' + ``` + +2. Scale it to zero while the credentials are split. + + > **Warning:** Scaling the global `katanomi` to 0 stops integration and pipeline reconciliation on the global cluster until replicas return. Precondition: a maintenance window and the recorded replica count. Rollback: `kubectl scale deploy/ -n --replicas=`. + + ```bash + kubectl scale deploy/ -n --replicas=0 + ``` + +3. Create a distinct Harbor integration credential for the global cluster and another for the workload cluster, then scale `katanomi` back to the recorded replica count. Re-run the pipeline stage. Reassigning the project without splitting the credential only clears the symptom until the next run. + +### Cause 3 + +1. Confirm the account is not the fault: log in and push to the `:` named in the error with the account the credential is supposed to hold. + + ```bash + printf '%s' "$HARBOR_PASSWORD" | docker login : --username --password-stdin + docker push ://: + ``` + + If this push is refused too, the account lacks push permission on the project; see [Registry push fails with insufficient_scope or authentication required on ACP](Registry_push_fails_with_insufficient_scope_or_authentication_required_on_ACP.md). + +2. If it succeeds, open the integration on the toolchain integration page, note the credential bound to it, and check whether another integration card uses the same credential. Read the registry addresses in that credential without printing the password (for a `kubernetes.io/dockerconfigjson` credential): + + ```bash + kubectl get secret -A | grep + kubectl get secret -n -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq -r '.auths | keys[]' + ``` + + The cause is confirmed when the credential is shared and a registry address in it differs from the integration's access address and from `:` in the error. + +3. Create a separate credential for this integration with the integration's registry address and an account that can push to the project, and bind it to this integration card. Do not change the address of the shared credential in place: the other integrations still use it. + + > **Warning:** Impact: from that moment, everything that goes through this integration (UI uploads, pipelines, and projects that use it) uses the new credential. Precondition: the command-line push in step 1 succeeded with the account in the new credential, and the name of the currently bound credential is recorded. Rollback: bind the recorded credential to the integration again. + +4. Retry the UI upload. It completes with no `500` and no `action: push` message. + +### Cause 4 + +On the failing node, restart the registry `auth` container. The kubelet recreates a container that belongs to a Pod. + +> **Warning:** Only when the failure is limited to this node and the container is the registry auth helper, not an unrelated workload. Impact: image pulls that depend on it on this node are interrupted until the kubelet recreates the container. If it does not come back, delete the owning Pod so its controller recreates it. Do not `crictl rm` the container; that is hard to reverse if no controller owns it. + +```bash +crictl ps --name auth +crictl stop +crictl ps --name auth +crictl pull ://: +``` + +### Cause 5 + +1. Check Harbor Core for restarts and errors at the time the UI shows `internal server error`: + + ```bash + kubectl get pods -A | grep -i harbor + kubectl -n logs --tail=200 + ``` + +2. Restart Harbor Core. + + > **Warning:** Only when the Harbor UI itself returns `internal server error` and unauthorized pulls affect projects whose permissions did not change (including public `library`). Impact: the Harbor UI and API, and registry logins and pulls that go through Core, are interrupted until the new Pod is Ready. The Deployment recreates the Pod; nothing needs to be rolled back. If the errors return, collect the Core log instead of restarting repeatedly. + + ```bash + kubectl -n get deploy | grep -i core + kubectl -n rollout restart deploy/ + kubectl -n rollout status deploy/ + ``` + +3. Reload the Harbor UI and rerun the failed pull. + +## Diagnostic Steps + +1. Test the same pull with a known-good account and check the workload's pull Secret first (see the pull Secret article linked under Issue). A missing Secret, a wrong account, or a Pod without `imagePullSecrets` is that article. + +2. If a pull from a public project also returns 401, check clocks before rotating passwords (Cause 1, step 1). Expected for Cause 1: a large `chronyc tracking` offset and the etcd `clock difference against peer` warning. If clocks agree and the Harbor web UI itself shows `internal server error`, select Cause 5. + +3. For a pipeline-only failure, read the `image-build` log. `credentials were not found` together with an abnormal imageRepository integration selects Cause 2. + +4. For a UI upload that returns `500` with `action: push`, push from the command line with the same account to the host:port in the error. If that works, compare the host:port in the error with the integration's address and check whether the credential is shared; that selects Cause 3. Credentials are matched by host and port, so an account checked on a different port of the same host proves nothing about the port in the error. + +5. A 401 on one node while a neighboring node pulls the same reference selects Cause 4. diff --git a/docs/en/solutions/Harbor_returns_500_when_its_PostgreSQL_loses_the_Kubernetes_API_cannot_mount_its_volume_or_has_corrupt_data.md b/docs/en/solutions/Harbor_returns_500_when_its_PostgreSQL_loses_the_Kubernetes_API_cannot_mount_its_volume_or_has_corrupt_data.md new file mode 100644 index 000000000..abc712f08 --- /dev/null +++ b/docs/en/solutions/Harbor_returns_500_when_its_PostgreSQL_loses_the_Kubernetes_API_cannot_mount_its_volume_or_has_corrupt_data.md @@ -0,0 +1,204 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "3.10" + - "4.2" + - "4.3" +--- + +# Harbor returns 500 when its PostgreSQL loses the Kubernetes API, cannot mount its volume, or has corrupt data + +## Issue + +Harbor returns HTTP 500, an image push returns 500 and keeps retrying, or image pulls fail because Harbor is unavailable. The client shows: + +```text +received unexpected HTTP status: 500 Internal Server Error +Retrying in 5 seconds +``` + +Behind this, Harbor core cannot use its PostgreSQL database. One of the following is seen in `harbor-core`, in the `postgres` container of a Harbor PostgreSQL Pod, or in the Pod events: + +```text +failed to update blob: sha256:... status to StatusNone, error:ERROR: cannot execute UPDATE in a read-only transaction (SQLSTATE 25006) +[FATAL] [/core/main.go:188]: failed to migrate: pq: cannot execute CREATE TABLE in a read-only transaction ... +ERROR: Error communicating with DCS +INFO: DCS is not accessible +MountVolume.MountDevice failed for volume "" : ... failed to create dir ".../globalmount": mkdir ... file exists +FATAL: catalog is missing 1 attribute(s) for relid 2662 +``` + +The registry log may show nothing obvious, and the registry storage (for example a highly available NAS) may have enough free space, so storage looks like the suspect even when it is not. + +## Root Cause + +### Cause 1: Patroni cannot reach the Kubernetes API (DCS), so PostgreSQL serves read-only transactions + +The Harbor PostgreSQL instance runs Patroni, which stores its cluster state in the Kubernetes API (its DCS, distributed configuration store). When Patroni's requests to the API time out, its `postgres` container logs: + +```text +ERROR: ObjectCache.run ApiException() +ERROR: get_cluster +patroni.utils.RetryFailedError: 'Exceeded retry deadline' +ERROR: Error communicating with DCS +INFO: DCS is not accessible +WARNING: Loop time exceeded, rescheduling immediately. +``` + +While Patroni cannot confirm leadership in the DCS, it keeps or demotes PostgreSQL to read-only. `harbor-core` then logs `cannot execute UPDATE in a read-only transaction (SQLSTATE 25006)`, or exits at startup with `failed to migrate: pq: cannot execute CREATE TABLE in a read-only transaction`, and Harbor returns 500. An image push may need several attempts before one succeeds. + +The DCS errors may be on only one Pod of the instance: another Pod can still log `no action. i am the leader with the lock`. Read every Pod of the instance. Restarting `harbor-core` alone does not clear the 500. Restarting the Harbor PostgreSQL Pods, and then `harbor-core`, restores pushes. + +The same DCS errors can also appear without any `read-only` line, with Harbor Pods Running and the portal reporting that the core service is unavailable. Restarting the Harbor PostgreSQL instance is the recovery in that case as well. + +### Cause 2: a stale CSI global mount keeps the PostgreSQL Pod in ContainerCreating + +After the Harbor `pg` Pod is deleted or restarted, it stays in `ContainerCreating`, and its events show: + +```text +MountVolume.MountDevice failed for volume "" : kubernetes.io/csi: attacher.MountDevice failed to create dir "/var/lib/kubelet/plugins/kubernetes.io/csi/pv//globalmount": mkdir ... file exists +Unable to attach or mount volumes: unmounted volumes=[pgdata] ... timed out waiting for the condition +``` + +The CSI global mount directory of the `pgdata` volume already exists on the node as a stale mount, so the kubelet cannot stage the volume. The PVCs are `Bound` and the storage backend (for example Ceph) can report healthy. Without the database the other Harbor components fail in turn: + +```text +harbor-core Init:0/1, then Init:Error +harbor-jobservice failed to load rest config ... connect: connection refused +harbor-notary-server Failed to Ping DB, sleep for 1 second. / Failed to connect DB after 30 seconds, time out. +harbor migrator failed to connect to host=-pgpool2 ... dial error +``` + +Not every Harbor install has a `pgpool` Pod; where it exists it reaches PostgreSQL through a pgpool Service (port 9999). + +### Cause 3: the PostgreSQL data is corrupt after the underlying storage was damaged + +When the storage under the Harbor database (for example a Ceph cluster) is damaged, the database files can be corrupt. Notary reports that it cannot connect to the database, and the `postgres` container logs: + +```text +ERROR: Exception when changing replication slots +psycopg2.errors.WrongObjectType: cannot open relation "pg_replication_slots" +ERROR libpq: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL: catalog is missing 1 attribute(s) for relid 2662 +WARNING [notary_signer]: default timeout +``` + +In standard PostgreSQL, OID 2662 is the `pg_class` OID index, so the damage is in the system catalog, not in a Harbor table. The Patroni leader can still look normal (`no action. i am the leader with the lock`), so the Pod list and the leader lock do not reveal this cause. Corruption of the system catalog cannot be repaired in place. + +## Resolution + +### Cause 1 + +1. Confirm the DCS errors on every Harbor PostgreSQL Pod and the `read-only` errors in core. The Patroni log is in the `postgres` container. Pod names vary between installs (for example `-harbor--pg-0` or `acid--0`). + + ```bash + PG_NS= + for P in $(kubectl -n "$PG_NS" get pods -o name | grep -E -- '-pg-[0-9]+$|acid-'); do + echo "== $P" + kubectl -n "$PG_NS" logs "$P" -c postgres --tail=200 \ + | grep -E 'DCS is not accessible|Error communicating with DCS|get_cluster|Exceeded retry deadline|leader with the lock' + done + NS= + CORE=$(kubectl -n "$NS" get pods -o name | grep -- -core | head -1) + kubectl -n "$NS" logs "$CORE" --tail=100 | grep -E 'read-only|SQLSTATE 25006|failed to migrate' + kubectl -n "$NS" get harbor -o jsonpath='{.spec.helmValues.database}{"\n"}' + ``` + + Continue only when a PostgreSQL Pod logs `Error communicating with DCS` or `DCS is not accessible`, or core logs `read-only` while the database host Harbor uses is the PostgreSQL Service name. If the database host is a single Pod IP and that Pod reports `t` for `SELECT pg_is_in_recovery();`, Harbor is pointed at a replica: fix the database address instead of restarting. + +2. Restart the PostgreSQL Pods that log the DCS errors. + + > **Warning:** Only when Cause 1 is confirmed in step 1. Impact: deleting a PostgreSQL Pod restarts the database; Harbor cannot connect until the Pod is Ready, and restarting the primary fails over or recovers in place depending on the number of replicas. Precondition: a maintenance window. The StatefulSet recreates the Pod and the volume is kept; in-flight transactions are not recovered. + + ```bash + kubectl -n "$PG_NS" delete pod + kubectl -n "$PG_NS" get pods -o wide -w + ``` + + When more than one Pod logs the DCS errors, delete them one at a time and wait for each to be Running before the next. If a `pgpool` Pod belongs to this database and is not Ready after PostgreSQL is up, delete that Pod too so it reconnects. + +3. If Harbor still returns 500 or core still logs `read-only`, restart `harbor-core`. + + > **Warning:** Only when PostgreSQL is accepting connections again. Impact: in-flight Harbor API calls are dropped until the new core Pod is Ready. The Deployment recreates the Pod; nothing needs to be rolled back. + + ```bash + CORE=$(kubectl -n "$NS" get deploy -o name | grep -- -core | head -1) + kubectl -n "$NS" rollout restart "$CORE" + kubectl -n "$NS" rollout status "$CORE" + ``` + +4. Push the image again. It completes without `500 Internal Server Error`. + +### Cause 2 + +1. Find the node and the volume the `pg` Pod cannot mount. The path to clean is the one printed in the `MountDevice failed` event. + + ```bash + PG_NS= + kubectl -n "$PG_NS" describe pod | sed -n '/Events:/,$p' + kubectl -n "$PG_NS" get pod -o jsonpath='{.spec.nodeName}{"\n"}' + kubectl -n "$PG_NS" get pvc -o wide + ``` + +2. On that node, inspect the global mount directory named in the event. A stale mount is one that `findmnt` still shows, or where `stat` or `ls` returns an error, while no running Pod uses the volume. + + ```bash + GM=/var/lib/kubelet/plugins/kubernetes.io/csi/pv//globalmount + findmnt "$GM" + stat "$GM" + ls -la "$GM" + mount | grep -F + ``` + +3. Clean up the stale mount. + + > **Warning:** Only when the `pg` Pod does not start, its events show `MountDevice failed ... globalmount: mkdir ... file exists` for this volume, and `findmnt` shows a stale mount for it on this node. Impact: `umount` detaches that mount. Precondition: the `pg` Pod is not running, no other Pod on the node uses the volume, and the path is not the PostgreSQL data directory (a directory that contains `PG_VERSION` is data, not a leftover mount point). Rollback: the kubelet mounts a healthy volume again when the Pod starts; do not remount it by hand. `rmdir` is only for an empty leftover directory and cannot be undone. + + ```bash + sudo umount "$GM" + # Only if the path is no longer a mount point and does not contain PG_VERSION: + sudo rmdir "$GM" + ``` + +4. Wait for the `pg` Pod to become Ready. If a `pgpool` Pod exists and is still not Ready, delete it so it reconnects. Core, jobservice, and Notary recover once the database is reachable. + +### Cause 3 + +1. Confirm the corruption in the database log and save the Harbor resource: + + ```bash + NS= + kubectl -n "$NS" logs -c postgres --tail=200 \ + | grep -E 'catalog is missing|WrongObjectType|cannot open relation|PQconnectPoll' + kubectl -n "$NS" get harbor -o yaml > "$HOME/harbor--before-redeploy.yaml" + ``` + +2. Look for a backup of the Harbor database (a scheduled dump or a volume snapshot). If one exists, restore it instead of redeploying onto an empty database. See [How to perform disaster recovery for Harbor](How_to_perform_disaster_recovery_for_harbor.md). + +3. Without a backup, redeploy Harbor through the operator against a new PostgreSQL instance. Do not reuse the corrupt data directory. + + > **Warning:** Only when the database log shows the corruption signature (`catalog is missing ... attribute(s) for relid`, `cannot open relation "pg_replication_slots"`) and no restorable backup exists. Impact: the registry metadata in the corrupt database (projects, repositories, artifact records, users, and settings) is lost; image blobs on the registry storage are neither repaired nor deleted by the redeploy. Precondition: a maintenance window and the Harbor YAML saved in step 1. Rollback: re-apply the saved YAML only if you have a good database to point it at; pointing it back at the corrupt data returns the same failure. + +4. After the new instance is Ready, configure regular backups for the Harbor database. + +## Diagnostic Steps + +```bash +kubectl get pods -A -o wide | grep -E 'harbor|acid-|pgpool' +NS= +CORE=$(kubectl -n "$NS" get pods -o name | grep -- -core | head -1) +kubectl -n "$NS" logs "$CORE" --tail=80 +kubectl -n "$NS" get harbor -o jsonpath='{.spec.helmValues.database}{"\n"}' +``` + +Harbor workload names differ between installs (`-harbor-core`, `harbor--core`); `grep -- -core` matches both. The last command shows the database host Harbor uses: a Service name with `SQLSTATE 25006` in core is Cause 1; a single Pod IP that is in recovery is a wrong database address, not this article. + +| What you see | Cause | +| --- | --- | +| Patroni `Error communicating with DCS` / `DCS is not accessible`; core `read-only transaction (SQLSTATE 25006)` or `failed to migrate`; push 500 and `Retrying` | Cause 1 | +| `MountDevice failed ... globalmount: mkdir ... file exists`; `pg` Pod `ContainerCreating`; Notary `Failed to Ping DB` | Cause 2 | +| `catalog is missing 1 attribute(s) for relid 2662`; `cannot open relation "pg_replication_slots"`; the storage was damaged | Cause 3 | +| `too many clients already (SQLSTATE 53300)` | [Harbor PostgreSQL rejects sessions with too many clients already (SQLSTATE 53300)](Harbor_PostgreSQL_rejects_sessions_with_too_many_clients_already_SQLSTATE_53300.md) | +| The database Pod does not start | [Harbor PostgreSQL database Pod never starts after an operator upgrade, with transparent huge pages, low CPU, or on NFS](Harbor_PostgreSQL_database_Pod_never_starts_after_an_operator_upgrade_with_transparent_huge_pages_low_CPU_or_on_NFS.md) | diff --git a/docs/en/solutions/Harbor_storage_cleanup_scheduled_garbage_collection_of_untagged_artifacts_operation_log_rotation_and_tag_retention.md b/docs/en/solutions/Harbor_storage_cleanup_scheduled_garbage_collection_of_untagged_artifacts_operation_log_rotation_and_tag_retention.md new file mode 100644 index 000000000..113e72528 --- /dev/null +++ b/docs/en/solutions/Harbor_storage_cleanup_scheduled_garbage_collection_of_untagged_artifacts_operation_log_rotation_and_tag_retention.md @@ -0,0 +1,200 @@ +--- +kind: + - How To +products: + - Alauda DevOps +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Harbor storage cleanup: scheduled garbage collection of untagged artifacts, operation-log rotation, and tag retention + +## Issue + +Harbor storage keeps growing and a cleanup policy is needed, or an image is gone and the question is whether Harbor removed it. Typical questions: + +- How do I delete untagged images in Harbor on a schedule? +- Can Harbor keep only one month of operation (audit) logs? +- Do images in Harbor have a retention period, or expire? How do I list every Harbor instance in the cluster? +- A business system restarted and a specific image version can no longer be found in Harbor. +- A retention policy that keeps only the 10 most recently pulled tags has to be set up in many projects. Is there a global or batch way? + +All of this is Harbor's own Clean Up service (System Management, Clean Up; English builds label the area Administration) and per-project Tag Retention. + +## Root Cause + +### Cause 1: untagged artifacts are removed only by garbage collection with the untagged option enabled + +Untagged artifacts stay in storage until a garbage collection (GC) run with "Allow garbage collection on untagged artifacts" enabled removes them. The Garbage Collection tab of Clean Up holds the schedule (for example `Weekly`), that option, and the buttons GC Now and Dry Run. Artifacts pushed within the last two hours (the default window) are not collected, so a GC run right after a push can free less than expected. + +### Cause 2: operation-log retention is set by Log Rotation, in days + +Harbor operation logs are purged by the Log Rotation tab of the Clean Up service. The retention period is set in days, as an integer greater than 0 and less than 10000. The purged operations are Create, Delete, and Pull. Without a saved schedule, nothing is purged and the Log Rotation history stays empty. + +### Cause 3: Harbor keeps images until they are deleted; node copies are removed by kubelet image garbage collection + +Harbor keeps stored images permanently by default. An image leaves Harbor only when someone deletes it, or when a cleanup policy configured in Harbor (a Tag Retention rule followed by GC) removes it. A restart of a business system does not delete images from Harbor. + +The copies of an image on cluster nodes are a separate store. The kubelet garbage-collects node images according to `/var/lib/kubelet/config.yaml`, for example: + +- `imageGCHighThresholdPercent: 85`: cleanup starts at 85% disk usage. +- `imageGCLowThresholdPercent: 80`: cleanup stops at 80% disk usage. +- `imageMinimumGCAge: 2m0s`: minimum image age before it can be collected. +- `imageMaximumGCAge: 0s`: age-based cleanup is disabled. + +The Pod's `imagePullPolicy` decides whether Harbor is contacted at start: `Always` pulls from Harbor on every start; `IfNotPresent` uses the node copy when it exists. A tag missing from Harbor therefore shows up only when a Pod starts on a node without a copy, or when the policy is `Always`. + +### Cause 4: Tag Retention is per project; there is no global or batch setting + +Tag Retention in Harbor is a per-project policy. Harbor has no global, one-time configuration that applies a retention policy to all projects, and no product-level batch feature. Each project carries its own policy, created by hand or by a script that calls the Harbor API for each project. + +## Resolution + +### Cause 1 + +1. Open Harbor as a system administrator: System Management (Administration), Clean Up, Garbage Collection. +2. Enable "Allow garbage collection on untagged artifacts". +3. Click Dry Run and read its log; it reports what a real run would delete. +4. Set the schedule, for example `Weekly`, and save. Use GC Now only for an immediate run. + +> **Warning:** Garbage collection with the untagged option permanently deletes untagged artifacts and the blobs that only they reference. Impact: anything that pulls those artifacts by digest fails afterwards. Precondition: the Dry Run log has been reviewed and no workload depends on untagged digests. Rollback: none; deleted artifacts must be pushed again. + +Check the saved schedule and recent runs (read-only): + +```bash +curl -sk -u 'admin:' 'https:///api/v2.0/system/gc/schedule' +curl -sk -u 'admin:' 'https:///api/v2.0/system/gc?page=1&page_size=10' +``` + +The schedule response shows the schedule type and `delete_untagged` set to `true` in its parameters. + +### Cause 2 + +1. System Management (Administration), Clean Up, Log Rotation. +2. Set the retention period in days, an integer from 1 to 9999; `30` keeps about one month. +3. Keep the operations to purge (Create, Delete, Pull) selected as required. +4. Set a schedule and save. + +> **Warning:** Log Rotation permanently deletes operation-log entries older than the retention period. Impact: the audit history for those operations is gone. Precondition: export the logs first if an audit requirement keeps them longer. Rollback: none. + +```bash +curl -sk -u 'admin:' 'https:///api/v2.0/system/purgeaudit/schedule' +``` + +After the first scheduled run, the Log Rotation history lists a finished job. + +### Cause 3 + +1. List every Harbor instance in the cluster: + + ```bash + kubectl get harbor -A + ``` + +2. Check whether the tag is still in Harbor. A `404` means Harbor no longer has it. A nested repository name is URL-encoded twice (`a/b` becomes `a%252Fb`). + + ```bash + curl -sk -o /dev/null -w '%{http_code}\n' -u 'admin:' \ + 'https:///api/v2.0/projects//repositories//artifacts/' + ``` + +3. Check the policies that can remove it: the project's Tag Retention rules (Projects, the project, Policy, Tag Retention) and the GC schedule. The project metadata carries a `retention_id` when a retention policy exists. + + ```bash + curl -sk -u 'admin:' 'https:///api/v2.0/projects/' | grep -o '"retention_id": *"[0-9]*"' + curl -sk -u 'admin:' 'https:///api/v2.0/retentions/' + ``` + +4. Read the project's operation log for delete operations on that repository (the Logs tab of the project). Log Rotation may already have purged old entries. + + ```bash + curl -sk -u 'admin:' \ + 'https:///api/v2.0/projects//logs?page=1&page_size=100' | grep -i -B2 -A6 delete + ``` + +5. On the node, read the kubelet image GC settings, whether the node still holds the image, and the Pod's pull policy: + + ```bash + grep -i -E 'imageGC|imageMinimumGCAge|imageMaximumGCAge' /var/lib/kubelet/config.yaml + crictl images | grep + kubectl -n get pod -o jsonpath='{range .spec.containers[*]}{.image}{" "}{.imagePullPolicy}{"\n"}{end}' + ``` + + A node copy removed by the kubelet is expected behavior at its thresholds, not a loss from Harbor. If the tag is gone from Harbor, a retention rule, a GC run, or a delete in the project log is where to look. For the node side, see [Container image garbage collection on ACP nodes](Container_image_garbage_collection_on_ACP_nodes.md). + +### Cause 4 + +For one project, in the UI: Projects, the project, Policy, Tag Retention, Add Rule. Choose the rule that retains the most recently pulled artifacts, set the count to `10`, set the repository and tag filters (`**` matches everything), then set a schedule. + +For many projects, script it against the Harbor v2.0 API. One way: create the policy once in the UI on a reference project, export it, and post a copy to every project that has no retention policy yet. Test on one project before running it on all. + +1. Export the reference policy (`` is the reference project's `retention_id`): + + ```bash + HARBOR='https://' + AUTH='admin:' + curl -sk -u "$AUTH" "$HARBOR/api/v2.0/retentions/" \ + | jq 'del(.id) | .rules |= map(del(.id))' > /tmp/retention-template.json + cat /tmp/retention-template.json + ``` + +2. List the projects without a retention policy (repeat with `page=2` and so on for more than 100 projects), and remove from the file every project that must not get the policy: + + ```bash + curl -sk -u "$AUTH" "$HARBOR/api/v2.0/projects?page=1&page_size=100" \ + | jq -r '.[] | select(.metadata.retention_id == null) | "\(.project_id) \(.name)"' > /tmp/retention-targets.txt + cat /tmp/retention-targets.txt + ``` + +3. Create the policy in each listed project. Harbor accepts a create only for a project that has no policy yet; a `201` means it was created. + + > **Warning:** A retention policy removes, on every scheduled run, all tags in its scope that the rule does not retain, here every tag outside the 10 most recently pulled in each repository. Impact: workloads that pull a removed tag fail, and the space returns only after the next GC. Precondition: `/tmp/retention-targets.txt` holds only the intended projects, and a dry run on each project (step 4) is reviewed before its schedule first fires. Rollback: deleting or disabling the policy stops future runs; removed tags must be pushed again. + + ```bash + while read -r pid name; do + jq --argjson pid "$pid" '.scope = {"level": "project", "ref": $pid}' /tmp/retention-template.json \ + | curl -sk -u "$AUTH" -H 'Content-Type: application/json' -X POST \ + -o /dev/null -w "$name %{http_code}\n" -d @- "$HARBOR/api/v2.0/retentions" + done < /tmp/retention-targets.txt + ``` + +4. Dry-run each new policy and read the result; with `dry_run` set to `true` nothing is deleted. The project's Tag Retention page shows the same results per repository. + + ```bash + curl -sk -u "$AUTH" "$HARBOR/api/v2.0/projects/" | jq -r '.metadata.retention_id' + curl -sk -u "$AUTH" -H 'Content-Type: application/json' -X POST \ + -d '{"dry_run": true}' "$HARBOR/api/v2.0/retentions//executions" + curl -sk -u "$AUTH" "$HARBOR/api/v2.0/retentions//executions" + ``` + +## Diagnostic Steps + +1. Read the current Clean Up state before changing anything: + + ```bash + curl -sk -u 'admin:' 'https:///api/v2.0/system/gc/schedule' + curl -sk -u 'admin:' 'https:///api/v2.0/system/purgeaudit/schedule' + ``` + +2. Untagged artifacts are not removed: check that the GC schedule has `delete_untagged` set to `true`, and remember that artifacts pushed in the last two hours are skipped (Cause 1). + +3. Operation logs grow without limit and the Log Rotation history is empty: no Log Rotation schedule is saved (Cause 2). + +4. Scheduled or manual GC runs end in `Error`, for example with `0 blob(s) and 0 manifest(s) deleted, 0 space freed`: a schedule whose runs fail frees no space. Read the job log of one failed run; the jobservice Deployment log can show only `reap outdated job stats` lines while the job log holds the error. + + ```bash + curl -sk -u 'admin:' 'https:///api/v2.0/system/gc//log' + ``` + + - `failed to execute GC job at mark phase` with `write: broken pipe` to `:5432`: see [Harbor garbage collection fails with write broken pipe to PostgreSQL when jobservice exhausts database connections](Harbor_garbage_collection_fails_with_write_broken_pipe_to_PostgreSQL_when_jobservice_exhausts_database_connections.md). + - `too many clients already (SQLSTATE 53300)`: see [Harbor PostgreSQL rejects sessions with too many clients already (SQLSTATE 53300)](Harbor_PostgreSQL_rejects_sessions_with_too_many_clients_already_SQLSTATE_53300.md). + +5. A tag reported missing, or a question about image expiry: Cause 3. Check the tag in Harbor first, then the policies and the project log, then the node copy and the pull policy. + +6. One retention policy needed in many projects: Cause 4. List which projects already have a policy before scripting: + + ```bash + curl -sk -u 'admin:' 'https:///api/v2.0/projects?page=1&page_size=100' \ + | jq -r '.[] | "\(.name) \(.metadata.retention_id // "none")"' + ``` diff --git a/docs/en/solutions/Harbor_unreachable_or_its_PostgreSQL_out_of_space_because_Harbor_storage_is_full.md b/docs/en/solutions/Harbor_unreachable_or_its_PostgreSQL_out_of_space_because_Harbor_storage_is_full.md new file mode 100644 index 000000000..9de4d3555 --- /dev/null +++ b/docs/en/solutions/Harbor_unreachable_or_its_PostgreSQL_out_of_space_because_Harbor_storage_is_full.md @@ -0,0 +1,202 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Harbor unreachable or its PostgreSQL out of space because Harbor storage is full + +## Issue + +Image push to Harbor does not succeed, no image can be selected when an application is deployed, and a Harbor PVC shows 100 percent usage. Variants of the same capacity problem: + +- Harbor is unreachable. Redis writes fail, Redis keeps re-running leader election, and Harbor cannot connect to Redis. Every Harbor component can be Running while pushes fail, and a restart shows Redis cannot write. +- An expansion of the Harbor PVC is rejected because the StorageClass does not support expansion. The PVC object can show a small requested size while the mount is already at 100 percent. +- The PostgreSQL database used by Harbor has run out of storage. +- While the full volume is being fixed, Pods on one TopoLVM data node cannot create containers and their events repeat: + + ```text + Failed to create pod sandbox: rpc error: code = NotFound desc = failed to create containerd container: failed to create snapshot: missing parent "k8s.io//sha256:" bucket: not found + ``` + +On the `harbor-core` side, a Redis that cannot persist shows up as: + +```text +failed to ping redis://...: MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. +failed to initialize cache: retry timeout +``` + +## Root Cause + +### Cause 1: the NFS export behind Harbor is full + +Harbor Redis, and sometimes the other Harbor volumes, are on an NFS StorageClass. When the NFS export fills up, Redis cannot persist writes, leader election runs again and again, and Harbor loses Redis and becomes unreachable. + +An NFS-backed volume does not take its limit from the PVC size: the Pod sees the free space of the export. Growing `spec.resources.requests.storage` does not add blocks, and an NFS StorageClass commonly does not support expansion at all. The capacity is the export. + +A common NFS class is the NFS client provisioner (provisioner `fuseim.pri/ifs`, often named `nfs-client` and set as the default class): + +```yaml +kind: StorageClass +metadata: + name: nfs-client + annotations: + storageclass.kubernetes.io/is-default-class: "true" +provisioner: fuseim.pri/ifs +reclaimPolicy: Delete +parameters: + archiveOnDelete: "false" +``` + +This provisioner creates one directory per PVC on a single shared export. Every PVC on the class, from Harbor or any other workload, draws on the same free space, so another workload can fill the export that Harbor Redis writes to. As the default class, it also receives every PVC that names no StorageClass. With `reclaimPolicy: Delete` and `archiveOnDelete: "false"`, deleting a PVC deletes its directory on the export, so deleting and recreating the Redis PVC discards the Redis data. + +### Cause 2: a TopoLVM PVC shared by the Harbor components is full + +When all Harbor components, PostgreSQL included, store their data on one TopoLVM PVC, that PVC can run out of space. The database can no longer write (PostgreSQL usually reports `No space left on device`), and image uploads fail. Unlike NFS, a TopoLVM volume is a logical volume of the PVC's size on one node, so expanding the PVC adds space, as long as the volume group on that node has free capacity. + +### Cause 3: corrupt containerd data on a TopoLVM data node blocks the recovery + +The containerd data on one node is corrupt: the snapshot metadata points to a parent snapshot that no longer exists. No Pod on that node can create a container, and every Pod there fails with `FailedCreatePodSandBox ... missing parent ... bucket: not found`, while Pods on other nodes start normally. When the node is a TopoLVM data node, its TopoLVM Pods (for example `topolvm-prepare-vg-` in `nativestor-system`) cannot start either, TopoLVM stays abnormal on that node, and the expansion of a full Harbor PVC (Cause 2) has to wait. Resetting the containerd data on that node makes container creation work again. This is a container runtime fault on the node, not a Harbor fault. + +## Resolution + +### Cause 1 + +1. Identify the export and confirm it is the full filesystem: + + ```bash + NS= + kubectl -n "$NS" get pvc -o custom-columns='NAME:.metadata.name,SC:.spec.storageClassName,VOLUME:.spec.volumeName' + kubectl get sc -o yaml + kubectl get pv -o jsonpath='{.spec.nfs.server}{" "}{.spec.nfs.path}{"\n"}' + REDIS=$(kubectl -n "$NS" get pods -o name | grep redis | head -1) + kubectl -n "$NS" exec "$REDIS" -- df -h + kubectl -n "$NS" logs "$REDIS" --tail=100 + ``` + + If `.spec.nfs` is empty, the server and path are in the CSI `volumeAttributes` of the PV. + +2. Expand the export on the NFS server (or the storage array behind it) until `df` in the Pod shows free space. No Harbor resource change and no PVC resize is needed. + +3. If the export cannot grow, remove only files on it that the storage owner confirms are unused. + + > **Warning:** Only when `df` on this export is full, Redis shows write failures or repeated leader election, and the file is confirmed unused. Impact: deletion cannot be undone. Precondition: the PV server and path match this export, and a copy or snapshot exists of anything Harbor might still reference. Rollback: only by restoring that copy. + + ```bash + # On the NFS server, export path taken from the PV: + du -xhd1 | sort -h + rm -f / + ``` + +4. If the Redis Pod is still in `CrashLoopBackOff` after `df` shows free space, restart only that Pod. + + > **Warning:** Only when the export is no longer full and Redis is still crash-looping. Impact: Redis restarts and drops its in-memory data for one start. The controller recreates the Pod and the volume is kept; Redis rebuilds its cache. + + ```bash + kubectl -n "$NS" delete "$REDIS" + ``` + +### Cause 2 + +1. Confirm which PVC is full, that it is TopoLVM, that its StorageClass allows expansion, and that the node holding the volume has free capacity: + + ```bash + NS= + kubectl -n "$NS" get pvc -o custom-columns='NAME:.metadata.name,SC:.spec.storageClassName,REQ:.spec.resources.requests.storage,CAP:.status.capacity.storage' + kubectl -n "$NS" exec -- df -h + kubectl get sc -o jsonpath='{.provisioner}{" "}{.allowVolumeExpansion}{"\n"}' + kubectl get pv -o jsonpath='{.spec.nodeAffinity}{"\n"}' + kubectl get node -o yaml | grep -i capacity + ``` + + On ACP the default TopoLVM class `topolvm-hdd` uses provisioner `topolvm.cybozu.com`. The PV `nodeAffinity` names the node that holds the logical volume; its TopoLVM capacity annotation shows what the volume group can still give. Check that the TopoLVM Pods on that node are Running (`kubectl -n nativestor-system get pods -o wide`); if they are not, fix Cause 3 first. + +2. Expand the PVC. + + > **Warning:** Only when `df` shows the TopoLVM PVC full, `allowVolumeExpansion` is `true`, and the node has more free capacity than the increase. Impact: the TopoLVM CSI driver extends the logical volume and the filesystem using space from that node's volume group. This cannot be undone: Kubernetes does not allow a PVC request to be reduced. + + ```bash + kubectl -n "$NS" patch pvc --type=merge \ + -p '{"spec":{"resources":{"requests":{"storage":""}}}}' + kubectl -n "$NS" get pvc -o jsonpath='{.status.capacity.storage}{"\n"}' + kubectl -n "$NS" exec -- df -h + ``` + + If `status.capacity` has grown but `df` has not, read the PVC conditions with `kubectl -n "$NS" describe pvc `. + +3. If the Harbor resource declares a size for this PVC, set it to the new size. A PVC larger than the declared size makes every Harbor Pod cycle on the next edit of the Harbor resource; see [Harbor core service unavailable or internal server error, stuck instance status, or Pods cycling after an edit](Harbor_core_service_unavailable_or_internal_server_error_stuck_instance_status_or_Pods_cycling_after_an_edit.md) (Cause 4). + +4. Verify that image push works and that images can be selected in application deployment. + +### Cause 3 + +1. Confirm that the error is limited to one node: + + ```bash + kubectl -n nativestor-system get pods -o wide + kubectl get pods -A -o wide --field-selector spec.nodeName= | grep -v -E 'Running|Completed' + kubectl get events -A --field-selector reason=FailedCreatePodSandBox | grep -E 'missing parent|bucket: not found' + ``` + + On the node, find the containerd root and check the runtime log: + + ```bash + sudo containerd config dump | grep -E '^(root|state) ' + sudo journalctl -u containerd --since "1 hour ago" | grep -E 'missing parent|bucket: not found' | tail -20 + ``` + +2. Drain the node. + + > **Warning:** Only when Pods on this node fail with `failed to create snapshot: missing parent ... bucket: not found` and Pods on other nodes start normally. Impact: the node's Pods are evicted; Pods whose TopoLVM volumes are on this node stay Pending until it is back. Precondition: a maintenance window for the workloads on this node. Rollback: `kubectl uncordon `. + + ```bash + kubectl cordon + kubectl drain --ignore-daemonsets --delete-emptydir-data + ``` + +3. Reset the containerd data by moving the data directory aside (the default root is `/var/lib/containerd`; use the `root` value printed in step 1 if it differs). + + > **Warning:** Only after step 2. Impact: every container on the node stops, and afterwards the node has no local images and pulls every image again, including the sandbox (pause), TopoLVM, and CSI images. Precondition: the registries those images come from are reachable from the node; images stored only in the Harbor being repaired cannot be pulled until Harbor works again. Rollback: stop `kubelet` and `containerd`, move the saved directory back, and start both services, which also brings back the corrupt state. + + ```bash + sudo systemctl disable --now kubelet + sudo systemctl disable --now containerd + sudo reboot + # after the reboot both services stay stopped: + sudo mv /var/lib/containerd /var/lib/containerd.bak-$(date +%Y%m%d%H%M) + sudo systemctl enable --now containerd + sudo systemctl enable --now kubelet + ``` + +4. Bring the node back and check TopoLVM: + + ```bash + kubectl get node + kubectl uncordon + kubectl -n nativestor-system get pods -o wide | grep + ``` + + The Pods on the node, including the TopoLVM Pods, reach Running with no new `FailedCreatePodSandBox` events. Then expand the full Harbor PVC (Cause 2). Remove the saved `/var/lib/containerd.bak-` directory only after the node has run normally for a while; that removal cannot be undone. + +## Diagnostic Steps + +Read the StorageClass of the full volume first: NFS is Cause 1, TopoLVM is Cause 2. + +```bash +NS= +kubectl -n "$NS" get pvc -o custom-columns='NAME:.metadata.name,SC:.spec.storageClassName,REQ:.spec.resources.requests.storage,CAP:.status.capacity.storage' +kubectl get pods -A -o wide | grep -E 'harbor|redis' +kubectl -n "$NS" exec -- df -h +kubectl get sc -o jsonpath='{.provisioner}{" "}{.allowVolumeExpansion}{"\n"}' +``` + +| What you see | Cause | +| --- | --- | +| Redis write failures or repeated leader election, `MISCONF ... not able to persist on disk` in core, NFS export at 100 percent; or a PVC expand rejected on an NFS class | Cause 1: on NFS, `df -h ` on the server is the reading that matters, not the PVC capacity | +| `df` in the PostgreSQL Pod full on a TopoLVM PVC | Cause 2: the PVC size is the ceiling, expand it | +| Pods on one node fail with `FailedCreatePodSandBox ... missing parent ... bucket: not found` | Cause 3: fix the node before expanding | +| `too many clients already` or `wal_keep_segments` in the database log | [Harbor PostgreSQL rejects sessions with too many clients already (SQLSTATE 53300)](Harbor_PostgreSQL_rejects_sessions_with_too_many_clients_already_SQLSTATE_53300.md) or [Harbor PostgreSQL database Pod never starts after an operator upgrade, with transparent huge pages, low CPU, or on NFS](Harbor_PostgreSQL_database_Pod_never_starts_after_an_operator_upgrade_with_transparent_huge_pages_low_CPU_or_on_NFS.md) | diff --git a/docs/en/solutions/Helm_chart_template_upload_fails_or_the_template_directory_cannot_be_selected_in_Catalog_on_ACP.md b/docs/en/solutions/Helm_chart_template_upload_fails_or_the_template_directory_cannot_be_selected_in_Catalog_on_ACP.md new file mode 100644 index 000000000..aae24e8ea --- /dev/null +++ b/docs/en/solutions/Helm_chart_template_upload_fails_or_the_template_directory_cannot_be_selected_in_Catalog_on_ACP.md @@ -0,0 +1,93 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" +--- + +# Helm chart template upload fails, or the template directory cannot be selected, in Catalog on ACP + +## Issue + +Uploading a Helm chart so that it appears in Catalog does not work: + +- A plain chart package (`.tgz`) uploaded with `violet` fails, and the chart never appears under Catalog → Helm Charts. +- The template upload does not become usable, and the `captain-controller` log shows that it cannot connect to port 80 of the Harbor VIP. +- The template repository has no template directory to select, or the upload dialog cannot select one. + +Related articles: [Catalog does not list a Helm chart that is already in Harbor, or a newly uploaded chart version, on ACP](Catalog_does_not_list_a_Helm_chart_that_is_already_in_Harbor_or_a_newly_uploaded_chart_version_on_ACP.md). + +## Root Cause + +### Cause 1: a plain chart is uploaded with violet + +A plain Helm chart uploaded with `violet` can fail and never reach Catalog → Helm Charts. `violet push` also cannot target a chosen template repository: charts pushed that way go to the default `public-charts` repository only. The working path for a plain chart is a template repository of type Chart or OCI Chart and the upload in Catalog → Helm Charts. + +### Cause 2: captain-controller cannot reach the Harbor VIP on port 80 + +The forwarding policy of the VIP in front of Harbor does not forward port 80, so `captain-controller` cannot connect to Harbor and the template upload fails. Opening port 80 on the VIP forwarding policy and then restarting `captain-controller-manager` makes the upload work. + +### Cause 3: the template directory is not created or the repository address hides it + +A template directory is created by typing its name in the template-directory field of the upload dialog; that creates the directory under the chart repository. For an OCI repository, an address that is only the Harbor host lets the upload dialog offer the Harbor projects (for example `library`) as directories, while an address that includes a project path prevents that selection. + +## Resolution + +### Cause 1 + +1. A platform administrator adds a template repository of type Chart or OCI Chart and grants Management on it to the user or project. +2. Open Catalog → Helm Charts, choose Add Template, and associate that template repository. +3. For OCI Chart, fill in the chart storage directory. +4. Upload Template and confirm. View and deploy the chart from Catalog → Helm Charts, not from the plugin list. + +### Cause 2 + +1. Confirm the log line: + + ```bash + kubectl get deploy -A | grep captain-controller + kubectl -n logs deploy/captain-controller-manager --tail=200 + ``` + +2. From a global cluster node, check port 80 of the Harbor VIP. Any HTTP status (Harbor answers `401` without credentials) means the port is open; a timeout or refusal matches this cause. + + ```bash + curl -sS -o /dev/null -w '%{http_code}\n' --connect-timeout 5 http://:80/v2/ + ``` + +3. Open TCP port 80 on the forwarding policy that publishes the Harbor VIP, and run the check again. + +4. Restart `captain-controller-manager`. + + > **Warning:** Only after the log shows the port 80 failure and the port has been opened. Impact: HelmRequest reconciliation pauses until the new Pod is Ready. Precondition: note the current image with the first command. Rollback: `kubectl -n rollout undo deployment/captain-controller-manager`. The restart does not delete charts. + + ```bash + kubectl -n get deploy captain-controller-manager -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' + kubectl -n rollout restart deployment/captain-controller-manager + kubectl -n rollout status deployment/captain-controller-manager + ``` + +5. Retest the template upload. If it succeeds and the chart is hidden only in an ARM cluster namespace, the chart's architecture label is the remaining filter, not the port. + +### Cause 3 + +1. Type the directory name in the template-directory field; this creates the directory under the chart repository. +2. If the repository is OCI and its address includes a project path (`library` or another project name), change the address to the Harbor host only and open the upload dialog again. The Harbor projects are then offered as directories. +3. If the address contains a Harbor web UI path such as `/harbor/projects//repositories`, remove that path first. + +A bare host address does not by itself list charts that were pushed straight to Harbor; see the related article for that case. + +## Diagnostic Steps + +```bash +kubectl get chartrepo -n cpaas-system -o custom-columns=NAME:.metadata.name,TYPE:.spec.type,URL:.spec.url,PHASE:.status.phase,REASON:.status.reason +``` + +| Check | Conclusion | +| --- | --- | +| The attempted upload used `violet`, and the package is a plain chart `.tgz` | Cause 1 | +| `captain-controller` log cannot connect to the Harbor VIP on port 80; `curl` to port 80 times out or is refused | Cause 2 | +| No template directory to select, or an OCI repository whose `URL` includes a project path | Cause 3 | +| Upload succeeds but the chart or version is not listed | Related article above | diff --git a/docs/en/solutions/Image_registry_or_Harbor_unreachable_or_slow_because_of_firewall_rules_blocked_ports_or_the_network_path.md b/docs/en/solutions/Image_registry_or_Harbor_unreachable_or_slow_because_of_firewall_rules_blocked_ports_or_the_network_path.md new file mode 100644 index 000000000..5025e669f --- /dev/null +++ b/docs/en/solutions/Image_registry_or_Harbor_unreachable_or_slow_because_of_firewall_rules_blocked_ports_or_the_network_path.md @@ -0,0 +1,203 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.1" + - "4.2" + - "4.3" +--- + +# Image registry or Harbor unreachable or slow because of firewall rules, blocked ports, or the network path + +## Issue + +Image pulls or pushes never get an HTTP answer, or crawl, and the cause lies between the client and the registry: + +| Signature | Cause | +| --- | --- | +| Creating a business cluster gets stuck with an image registry access error; a packet capture shows the node connecting to port `443` of the global VIP instead of `11443` | Cause 1 | +| `ImagePullBackOff` with `rpc error: code = DeadlineExceeded ... Head "http:///v2///manifests/": dial tcp :80: i/o timeout` on every node of a newly built cluster; restriction rules in the node firewall | Cause 2 | +| `Failed to pull image "nginx:1.7.9" ... Head "https://registry-1.docker.io/v2/library/nginx/manifests/1.7.9": dial tcp ...:443: i/o timeout` in a cluster without Internet egress | Cause 3 | +| Pulls from the platform registry and pushes to Harbor work but are very slow; `ss -ti` shows a high `rcv_ooopack` count | Cause 4 | +| `FATA[0000] failed to copy: read tcp ...: read: connection reset by peer` for a large image whose layers stay at `downloading` with 0.0 B, on nodes in another data center; the same pull works on Harbor's own network | Cause 5 | +| Right after the runtime was switched from Docker to containerd, `Head "http:///v2/...": dial tcp :80: i/o timeout` although the registry is HTTPS | Cause 6 | + +When there is no error text, first ask: does the same image pull from a node of another cluster, and which network devices sit between the failing nodes and Harbor? If only one node times out and its neighbors pull the same reference, see [Intermittent image pull io timeout from a single ACP node](Intermittent_image_pull_io_timeout_from_a_single_ACP_node.md). + +The platform image registry runs in the `global` cluster and listens on port `11443`. Business clusters do not always pull from it on `11443`: an Ingress in the `global` cluster can expose the same registry under the same domain on another port (for example 80), and the business cluster then pulls platform images through that port. Compare the address with the image registry configured for platform components in the business cluster and with the private registry address in the business cluster overview. + +## Root Cause + +### Cause 1: port 443 from the business cluster to the global VIP is blocked + +The firewall between the business cluster and the global cluster allows only `11443`, but the access path also uses port `443` on the global VIP, so registry access fails during cluster creation. Port `443` from the business cluster to the global VIP must also be allowed. + +### Cause 2: firewall rules on the nodes, or in front of Harbor, drop the traffic + +Restriction rules in the node firewall drop traffic to the Harbor address, so the scheduling nodes of a newly built or rebuilt cluster cannot reach Harbor at all. Deleting those restriction rules restores image downloads. A firewall in front of Harbor has the same effect: the allowance must cover the path from every cluster that pulls to the address in Harbor's external URL. + +### Cause 3: the cluster cannot reach the public registry the image names + +An image reference without a registry host, such as `nginx:1.7.9`, resolves to `docker.io/library/nginx:1.7.9`. Without access to Docker Hub every such pull times out. Helm charts that reference Docker Hub images fail the same way. + +### Cause 4: TCP packet reordering on a multipath physical network + +When the physical network forwards one flow over several paths, for example through a redundant ESXi uplink, segments arrive out of order and throughput collapses. TCP statistics show many out-of-order segments (for example `rcv_ooopack=1519`). Removing the redundant link improves pull and push throughput significantly. + +### Cause 5: a WAF on the path resets large pulls from an external data center + +Nodes in an external data center that reach Harbor through a jump host, WAF, or firewall fail to pull a large image (for example about 20 GiB): the layers stay at `downloading` with 0.0 B, then the pull fails with `connection reset by peer`. The same pull succeeds from the global cluster and from other clusters on Harbor's network. The WAF intercepts the transfer and resets the connection. Adjusting the WAF fixes it; no Harbor setting is involved. + +### Cause 6: after a Docker-to-containerd switch, the node has no valid registry login + +After the container runtime of business nodes is switched from Docker to containerd, a manual pull on a node sends a plain HTTP request to port 80, which the registry does not serve, and times out. Logging the node in to the registry again with `nerdctl -n k8s.io login :11443 --insecure-registry` makes the pulls work. This is not a firewall rule: the fault follows the runtime switch. The underlying mechanism has not been identified. + +## Resolution + +### Cause 1 + +1. From a business cluster node, while the failing operation runs, confirm the destination port: + + ```bash + tcpdump -nn -i any host and tcp + nc -vz -w 3 443 + nc -vz -w 3 11443 + ``` + +2. If SYNs to `443` get no answer while `11443` connects, have the network team allow TCP `443` from every business cluster node to the global VIP, in addition to `11443`. If the business cluster pulls through a global Ingress port, allow that port too. Retry the cluster creation step. + +### Cause 2 + +1. Test from the node and list the rules that match the Harbor address: + + ```bash + nc -vz -w 3 + iptables-save > /root/iptables-before.rules + iptables-save | grep -nE '|DROP|REJECT' + firewall-cmd --state 2>/dev/null && firewall-cmd --list-all + ``` + +2. Remove the restriction rule that drops or rejects traffic to the Harbor address or port. + + > **Warning:** Only when `nc` times out from the node and a matching drop or reject rule is found. Impact: the node's security posture changes, and deleting the wrong line can cut traffic that Kubernetes, the CNI, or kube-proxy rely on. Precondition: `/root/iptables-before.rules` exists, the rule is a site restriction rule (not in a `KUBE-*`, OVN, or other CNI chain), `` is the table that holds it (`filter` by default, `raw` for `PREROUTING` rules), and the security owner approves. Rollback: `iptables-restore < /root/iptables-before.rules`. + + ```bash + iptables -t
-L -n --line-numbers | grep '' + iptables -t
-D + nc -vz -w 3 + crictl pull ://: + ``` + + If firewalld or a host security agent manages the rule, remove it there as well, or it returns after a reload. + +3. For a firewall in front of Harbor (including rules in the `raw` table `PREROUTING` chain of the Harbor host), open the path to the address and port of Harbor's external URL. Image pulls by the kubelet and containerd leave from the node IPs, so allow the node IPs of every cluster that pulls. Service IPs never arrive at Harbor as a source. Pods that call Harbor themselves, such as build Pods, arrive with the node IP when their subnet translates outgoing traffic (Kube-OVN `natOutgoing: true`) and with the Pod IP otherwise: + + ```bash + kubectl get subnet -o custom-columns=NAME:.metadata.name,CIDR:.spec.cidrBlock,NAT:.spec.natOutgoing + ``` + +### Cause 3 + +Open egress to the public registry, or bring the image into Harbor. To bring it in, save the image on a machine that can download it, then load, tag, and push it on a host that reaches Harbor: + +```bash +# On the connected machine: +docker pull docker.io/library/: +docker save docker.io/library/: -o -.tar +# On a host that reaches Harbor: +docker load -i -.tar +docker tag docker.io/library/: ://: +docker push ://: +``` + +Change the workload, or the Helm chart values, to the Harbor reference, and give the workload a pull Secret for that project (see [Harbor image pull returns 401 because of the pull Secret, the project account, or node credentials](Harbor_image_pull_returns_401_because_of_the_pull_Secret_the_project_account_or_node_credentials.md)). + +### Cause 4 + +1. Measure reordering on a live transfer: + + ```bash + ss -ti dst | grep -oE 'rcv_ooopack:[0-9]+|retrans:[0-9/]+|reordering:[0-9]+' + nstat -az | grep -iE 'TCPOFOQueue|TCPSACKReorder|TCPTSReorder' + ``` + + A `rcv_ooopack` count that keeps growing during one pull points at the physical path. + +2. Have the virtualization team remove the redundant link that splits the flow, or change the uplink teaming so one flow stays on one physical path. + + > **Warning:** Impact: changing ESXi uplinks affects every VM on that host and can drop their network briefly. Precondition: a maintenance window, the current vSwitch and teaming configuration recorded, and the remaining uplink confirmed healthy. Rollback: re-add the link with the recorded configuration. + +3. Repeat the same pull and compare elapsed time and `rcv_ooopack`. + +### Cause 5 + +1. Run the same pull on a node in the external data center and on a node of the global cluster: + + ```bash + crictl pull ://: + ``` + + Layers that never leave 0.0 B before `read: connection reset by peer`, on the remote node only, confirm the cause. + +2. Have the network team adjust the WAF that intercepts the transfer, then retry the pull from the remote node. + +3. As an interim workaround while the path is broken, copy the image onto the hosts that run the workload and set the image pull policy to `IfNotPresent`. + + > **Warning:** Impact: with `IfNotPresent` those hosts do not refresh the tag until the policy changes or the local image is removed, so a later push of the same tag is not picked up and a registry outage is hidden. Precondition: the image is already on the hosts that will run the workload; use it only until the WAF change is verified. Rollback: set the policy back to `Always`. + + ```bash + kubectl patch deploy -n --type=json \ + -p '[{"op":"replace","path":"/spec/template/spec/containers/0/imagePullPolicy","value":"IfNotPresent"}]' + ``` + +### Cause 6 + +1. Confirm the node was just switched to containerd, the failing request goes to `http://...:80`, and the registry answers on its HTTPS port: + + ```bash + kubectl get node -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}{"\n"}' + crictl pull //: + nc -vz -w 3 11443 + ``` + +2. Log the node in to the registry again. + + > **Warning:** Only when the runtime switch preceded the failure and the pull goes to port 80. Impact: `nerdctl login` stores the registry credential, base64-encoded, in `/root/.docker/config.json` on this node, and the kubelet then sends it for pulls from that registry; `--insecure-registry` skips certificate verification for this login. Precondition: the account is one approved for node pulls, and the existing config file is copied aside. Rollback: `nerdctl -n k8s.io logout :11443`, or restore the copied file. + + ```bash + cp -a /root/.docker/config.json /root/docker-config.json.bak 2>/dev/null + nerdctl -n k8s.io login :11443 --insecure-registry + crictl pull //: + ``` + +3. Repeat on every switched node. If the pull still goes to port 80, compare `/etc/containerd/certs.d/` with a node that pulls (see [Harbor push or pull fails because the client uses the wrong scheme: HTTP response to HTTPS client, or port 443 refused](Harbor_push_or_pull_fails_because_the_client_uses_the_wrong_scheme_HTTP_response_to_HTTPS_client_or_port_443_refused.md)). + +## Diagnostic Steps + +1. Record the image reference, the node, and the first error line. `DeadlineExceeded` or `i/o timeout` means the TCP connection never completed; a `401`, `x509`, or `HTTP response to HTTPS client` line is not a network-path problem. + +2. Check whether every node fails or only one: + + ```bash + kubectl get pods -A -o wide | grep -E 'ImagePull|ErrImage' + ``` + + All nodes of a new or rebuilt cluster point to Cause 1 or Cause 2. + +3. From a failing node, test each port the image reference and the platform use: + + ```bash + nc -vz -w 3 + curl -sv --connect-timeout 10 http://:/v2/ + curl -sv --connect-timeout 10 https:///v2/ + ``` + + A timeout on `443` toward the global VIP selects Cause 1. A timeout toward Harbor with matching drop or reject rules on the node selects Cause 2. A timeout toward a public registry selects Cause 3. + +4. If transfers complete but slowly, read `ss -ti` during a pull. A high `rcv_ooopack` selects Cause 4; low reordering with slow transfers needs a bandwidth test on the path. + +5. A connection reset on a large pull from an external data center, with the same pull succeeding next to Harbor, selects Cause 5. + +6. Before removing firewall rules for a port 80 timeout, ask whether the node runtime was just switched from Docker to containerd. A switch, an `http://...:80` request for an HTTPS registry, and no matching drop rule select Cause 6. diff --git a/docs/en/solutions/Platform_registry_pull_fails_with_authorization_failed_or_insufficient_scope_auth.yaml_targets_or_node_login.md b/docs/en/solutions/Platform_registry_pull_fails_with_authorization_failed_or_insufficient_scope_auth.yaml_targets_or_node_login.md new file mode 100644 index 000000000..c6dc8ab17 --- /dev/null +++ b/docs/en/solutions/Platform_registry_pull_fails_with_authorization_failed_or_insufficient_scope_auth.yaml_targets_or_node_login.md @@ -0,0 +1,133 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Platform registry pull fails with authorization failed or insufficient_scope: auth.yaml targets or node login + +## Issue + +Nodes reach the platform image registry (the built-in registry of the global cluster, often addressed as `:11443`), but pulls are denied: + +- After the core environment is upgraded, custom-built images stored under the `/solution` path fail to pull. +- Plugin images newly added to the built-in registry on port `11443` fail to pull in Pods and with `crictl pull` on the node, with `authorization failed`. +- Images uploaded to the platform registry cannot be pulled in a business cluster, although they are present in the registry storage: + + ```text + PullImage from image service failed ... pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed + ``` + +Related articles: [Registry push fails with insufficient_scope or authentication required on ACP](Registry_push_fails_with_insufficient_scope_or_authentication_required_on_ACP.md). + +## Root Cause + +The platform registry authorizes requests with `/etc/kubernetes/registry/auth.yaml` on the global control-plane nodes. Each user entry grants `actions` (`pull`, `push`, `delete`) on one or more `target` patterns (regular expressions when `useRegexp: true`). A node that presents no credential is handled by the `_anonymous` entry, which by default grants `pull` only on a few repositories (for example `tkestack/.*` and `windows/.*`). + +### Cause 1: the user's pull target does not include the solution repository + +The user's `pull` entry lists the platform prefixes only, for example `target: (acp|ait|devops|asm|alauda|3rdparty|ops|middleware|automation|frontend|developer\-platform)/.*` with `useRegexp: true`, and the `_anonymous` entry covers only `tkestack/.*` and `windows/.*`. Neither matches `solution/...`, so pulls of custom-built images under `/solution` are denied. Adding `solution` to the user's `target` and restarting the registry fixes it. + +### Cause 2: the repository of newly added images has no anonymous pull entry + +Node pulls without an image pull secret go through the `_anonymous` entry. When new images are added to a repository that no anonymous `target` covers (for example `3rdparty`), the pulls fail with `authorization failed`. Adding an anonymous pull target for that repository in `auth.yaml` is the fix. + +### Cause 3: the business-cluster node has no valid credential for the repository + +On a business-cluster node, `crictl pull :11443//:` returns `insufficient_scope: authorization failed`, while DNS resolution and TCP to port `11443` work and the image is present in the registry storage (MinIO under `/cpaas/minio` on each global control-plane node). Adding an anonymous pull entry to `auth.yaml` does not help. Logging the node in to the registry again with `nerdctl -n k8s.io login :11443 --insecure-registry` makes the pulls work. The underlying mechanism has not been identified. + +## Resolution + +Before any edit of `auth.yaml`: the file exists on every global control-plane node, and all copies must stay identical. Compare them first: + +```bash +md5sum /etc/kubernetes/registry/auth.yaml +``` + +### Cause 1 + +1. Find the entry of the affected user: + + ```bash + grep -n -B3 -A8 'useRegexp' /etc/kubernetes/registry/auth.yaml + ``` + +2. Add `solution` to the user's `target` alternation, in the same form as the other prefixes, for example `(acp|ait|...|developer\-platform|solution)/.*`, then restart the registry so it reads the file. + + > **Warning:** Only when the failing images are under `/solution` and no `target` of the user or of `_anonymous` matches `solution/`. Impact: a wrong edit can deny pulls of platform images to every cluster that uses this registry, and the restart interrupts pushes and pulls until the registry is Ready. Precondition: a copy of the file on each node, only the `target` of the affected user changes, and every global control-plane node gets the same change. Rollback: restore the copies and restart the registry again. + + ```bash + cp -a /etc/kubernetes/registry/auth.yaml /root/auth.yaml.bak + vi /etc/kubernetes/registry/auth.yaml + kubectl get pods -A | grep -i registry + kubectl rollout restart deploy/ -n + kubectl rollout status deploy/ -n + ``` + +3. Retry the pull of a custom-built image under `/solution`. A pull from another repository is not the check. + +### Cause 2 + +1. Read the `_anonymous` entry on each global control-plane node: + + ```bash + grep -n -A12 '_anonymous' /etc/kubernetes/registry/auth.yaml + ``` + +2. Add an anonymous pull target for the failing repository in the same format as the existing ones (for example a target `3rdparty/.*` with action `pull` next to `tkestack/.*`), change nothing else, and restart the registry as in Cause 1. + + > **Warning:** Only when the failing repository is not covered by any anonymous target and the pulls are made without an image pull secret. Impact: anyone who can reach the registry can then pull every image in that repository without credentials; a YAML error in the file can stop the registry from authenticating anyone. Precondition: the images in that repository may be readable by anyone who reaches the registry, a copy of the file is kept on each node, and every node gets the same edit. Rollback: restore the copies and restart the registry. + +3. Retry `crictl pull` of one of the new images on a node. If it still fails with `insufficient_scope`, continue with Cause 3. + +### Cause 3 + +1. Confirm the image is in the registry storage, on a global control-plane node: + + ```bash + du -sh /cpaas/minio + ls /cpaas/minio/*/registry/docker/registry/v2/repositories// + ``` + +2. From the failing node, confirm name resolution and the port, and reproduce the error: + + ```bash + nslookup + nc -vz -w 3 11443 + crictl pull :11443//: + ``` + +3. Log the node in to the registry again. + + > **Warning:** Only when the image is present, the port answers, and the pull returns `insufficient_scope: authorization failed`. Impact: `nerdctl login` stores the registry credential, base64-encoded, in the Docker config file of the user who runs it (`/root/.docker/config.json` for root), and the kubelet then sends it for pulls from that registry; `--insecure-registry` skips certificate verification for this login. Precondition: the account is one the registry owner approves for node pulls, and the existing config file is copied aside. Rollback: `nerdctl -n k8s.io logout :11443`, or restore the copied file. + + ```bash + cp -a /root/.docker/config.json /root/docker-config.json.bak 2>/dev/null + nerdctl -n k8s.io login :11443 --insecure-registry + crictl pull :11443//: + ``` + +4. Repeat on every node that must pull the image. If you added an anonymous entry to `auth.yaml` while testing and it is not part of the intended configuration, remove it again on every global control-plane node. + +## Diagnostic Steps + +1. Separate a permission denial from a missing tag: list the tags with an account that can read the repository. An empty list means the image is not there. + +2. Read the entries that could match the failing repository: + + ```bash + grep -n -B2 -A12 -E '_anonymous|useRegexp' /etc/kubernetes/registry/auth.yaml | grep -v -i password + ``` + + - Custom-built images under `/solution`, with no `target` matching `solution/`: Cause 1. + - A repository with newly added images, pulled without an image pull secret, with no anonymous `target` covering it: Cause 2. + +3. `insufficient_scope: authorization failed` on a business-cluster node, with the image present under `/cpaas/minio` and port `11443` reachable: Cause 3. Try the node login before editing `auth.yaml`. + +4. A `401` from Harbor rather than from the platform registry is covered in [Harbor image pull returns 401 because of the pull Secret, the project account, or node credentials](Harbor_image_pull_returns_401_because_of_the_pull_Secret_the_project_account_or_node_credentials.md). diff --git a/docs/en/solutions/Redis_for_Harbor_unavailable_no_Data_Services_license_or_Redis_in_cluster_mode_reported_as_empty_secrets.md b/docs/en/solutions/Redis_for_Harbor_unavailable_no_Data_Services_license_or_Redis_in_cluster_mode_reported_as_empty_secrets.md new file mode 100644 index 000000000..7f7b8b8a5 --- /dev/null +++ b/docs/en/solutions/Redis_for_Harbor_unavailable_no_Data_Services_license_or_Redis_in_cluster_mode_reported_as_empty_secrets.md @@ -0,0 +1,116 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps + - Alauda Application Services +ProductsVersion: [] +--- + +# Redis for Harbor unavailable: no Data Services license, or Redis in cluster mode reported as empty secrets + +## Issue + +Harbor cannot be deployed because its Redis is not available or not accepted: + +- While preparing Redis for a Harbor v4 deployment, the Select Instance Type page blocks creation of a `Redis` instance with `请在 Data Services 视图使用此功能` ("use this function in the Data Services view"), and the create action is disabled. The list also shows `ActiveRedis`, `ActiveRedisConnection`, `DistributedRedisCluster`, and `RedisClusterBackup`. Opening Data Services then shows `无法访问此页面` ("the page cannot be accessed"), states that the Data Services product function cannot be used, and points to `许可证管理 / 平台许可证` (License Management / Platform License) with an `立即导入` (Import now) action. +- Deploying Harbor reports that the secrets are empty, although the Secrets named by the deployment do contain data. + +Related articles: [harbor-core cannot connect to Redis: NOAUTH from Sentinel, or ERR invalid password after a Redis password change](harbor-core_cannot_connect_to_Redis_NOAUTH_from_Sentinel_or_ERR_invalid_password_after_a_Redis_password_change.md). + +## Root Cause + +### Cause 1: Data Services is not licensed + +Redis provisioning for Harbor v4 from the platform goes through Data Services ("v4" is the ACP packaging name, not the Helm chart or Harbor version). When Data Services has not been purchased, its license is absent, the Data Services view cannot be opened, and a Redis instance cannot be created there. This is a licensing condition, not a fault. Harbor can instead be deployed against Redis and PostgreSQL that you provide, passed to Harbor through credential Secrets. + +### Cause 2: the Redis instance runs in cluster mode + +When the Redis instance configured for Harbor runs in cluster mode, deploying Harbor reports that the secrets are empty, even though the Secrets contain data. The error comes from the unsupported Redis mode, not from missing Secret content. **Harbor does not support Redis cluster mode**; it needs a Redis that is not in cluster mode (Sentinel or standalone). + +## Resolution + +### Cause 1 + +1. Check `许可证管理 / 平台许可证` (Platform License) for a Data Services license. If Data Services is purchased later, import the license from that page; Redis can then be created from the Data Services view. + +2. Without the license, provide Redis and PostgreSQL yourself and give Harbor their connection details in Opaque Secrets in the Harbor namespace. Follow the "Harbor credentials" page of the Alauda Build of Harbor installation documentation for the Harbor version you deploy: it lists the Secret keys, the supported PostgreSQL versions, and how the Secrets are referenced from the `Harbor` resource. The Harbor version decides which PostgreSQL version is required (for example, Harbor 2.12.z requires PostgreSQL 14.x; Harbor 2.14.z supports PostgreSQL 15.x and 16.x). + + Redis in Sentinel mode uses the keys `password`, `address`, and `masterName`: + + ```bash + NS= + kubectl -n "$NS" create secret generic harbor-redis \ + --from-literal=password='' \ + --from-literal=address='192.0.2.11:26379,192.0.2.12:26379,192.0.2.13:26379' \ + --from-literal=masterName='mymaster' + ``` + + Redis in standalone mode uses `host`, `port`, and `password` instead. + + PostgreSQL uses `host`, `port`, `username`, `password`, `database`, and `sslmode`. The database must already exist and be empty; `harbor_db` below is only an example name and must be the database you created: + + ```bash + kubectl -n "$NS" create secret generic harbor-database \ + --from-literal=host='192.0.2.20' \ + --from-literal=port='5432' \ + --from-literal=username='postgres' \ + --from-literal=password='' \ + --from-literal=database='harbor_db' \ + --from-literal=sslmode='require' + ``` + +3. Use a Redis without an independent Sentinel password: Harbor does not send the Sentinel password and fails with `NOAUTH` (see the related article). + +### Cause 2 + +1. Confirm that the Secrets are not empty and that Redis runs in cluster mode: + + ```bash + NS= + kubectl -n "$NS" get secret -o go-template='{{range $k, $v := .data}}{{$k}} {{len $v}}{{"\n"}}{{end}}' + kubectl get redis -A + kubectl -n get redis -o yaml | grep -n -i -E 'arch|mode|cluster' + ``` + + The cause is confirmed when every key has a non-zero length and the Redis instance is in cluster mode. + +2. Create a Redis instance that is not in cluster mode (Sentinel or standalone) in Data Services, and copy its credential Secret into the Harbor namespace if it lives elsewhere: + + ```bash + kubectl get secret -n -l middleware.instance/type=Redis + ``` + +3. Point the `Harbor` resource at the new instance. + + > **Warning:** Only when the deployment reports empty secrets, the Secrets are not empty, and Redis is in cluster mode. Impact: the operator restarts Harbor components, and cache, sessions, and queued jobs on the old Redis are not carried over. Precondition: the new Redis instance is Ready and its Secret is in the Harbor namespace; save the current resource with the first command below. Rollback: apply the saved file, which returns the unsupported mode and the same error. Do not delete the cluster-mode instance until Harbor works on the new one; its deletion cannot be undone. + + ```bash + NAME= + kubectl -n "$NS" get harbor "$NAME" -o yaml > "$HOME/harbor-$NAME-before.yaml" + kubectl -n "$NS" patch harbor "$NAME" --type=merge -p '{ + "spec": {"helmValues": {"redis": {"type": "external", "external": { + "addr": ":26379,:26379,:26379", + "existingSecret": "", + "existingSecretKey": "password", + "sentinelMasterSet": "mymaster" + }}}} + }' + ``` + + For a standalone replacement, use its `host:port` as `addr` and leave `sentinelMasterSet` unset. + +## Diagnostic Steps + +1. For Cause 1 the evidence is in the UI: the text `请在 Data Services 视图使用此功能` when creating Redis, and `无法访问此页面` pointing to the platform license page when Data Services is opened. Check the Platform License page before investigating a fault. + +2. For Cause 2, list the key sizes of every Secret the deployment references (command in Cause 2, step 1). Expected: non-zero sizes. Then check the Redis architecture. Expected: cluster mode. + +3. Once Harbor is deployed, read the core log for Redis errors: + + ```bash + CORE=$(kubectl -n get pods -o name | grep harbor-core | head -1) + kubectl -n logs "$CORE" --tail=200 | grep -iE 'redis|sentinel|NOAUTH|invalid password' + ``` + + Expected: no Redis errors. `NOAUTH` or `ERR invalid password` is covered by the related article. diff --git a/docs/en/solutions/Registry_push_fails_with_insufficient_scope_or_authentication_required_on_ACP.md b/docs/en/solutions/Registry_push_fails_with_insufficient_scope_or_authentication_required_on_ACP.md new file mode 100644 index 000000000..b8fcdd3fa --- /dev/null +++ b/docs/en/solutions/Registry_push_fails_with_insufficient_scope_or_authentication_required_on_ACP.md @@ -0,0 +1,111 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda DevOps +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Registry push fails with insufficient_scope or authentication required on ACP + +## Issue + +Login reaches the registry, but the account is not allowed to push, or the push carries no credential: + +- `nerdctl` or `ctr` push fails, after the registry address itself is correct, with: + + ```text + push access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed + ``` + + The account is the registry user entered when the platform was deployed (for example `cpaas`). A push as `admin` is not rejected. +- A pipeline Buildah task pulls the base image, then the push fails with: + + ```text + Error: pushing image "//:" to "docker:////:": trying to reuse blob sha256: at destination: checking whether a blob sha256: exists in //: authentication required + ``` +- A new global cluster cannot reuse a Harbor that was deployed with the previous platform version. + +A `docker login` or push that fails with `http: server gave HTTP response to HTTPS client` never reaches the account check; see [Harbor push or pull fails because the client uses the wrong scheme: HTTP response to HTTPS client, or port 443 refused](Harbor_push_or_pull_fails_because_the_client_uses_the_wrong_scheme_HTTP_response_to_HTTPS_client_or_port_443_refused.md). For pull denials from the platform registry, see [Platform registry pull fails with authorization failed or insufficient_scope: auth.yaml targets or node login](Platform_registry_pull_fails_with_authorization_failed_or_insufficient_scope_auth.yaml_targets_or_node_login.md). + +## Root Cause + +### Cause 1: the platform-deployment registry user has only pull permission + +In `/etc/kubernetes/registry/auth.yaml` on the global control-plane nodes, the registry user entered during platform deployment has only `pull` on target `.*`, while `admin` has `pull`, `push`, and `delete` on `.*`. A push with that user fails with `insufficient_scope: authorization failed`. Pushing with an account that has push permission (`admin`) is the fix. + +### Cause 2: the Buildah task has no registry credentials bound for the push + +In an ACP UI pipeline that uses the built-in Buildah task, the base-image pull succeeds and the push fails with `authentication required` on the blob check, because the task has no registry credentials for the push. The task has an optional `registry-config` workspace that provides a `config.json` at the workspace root, which Buildah uses to authenticate to the registry. Binding that workspace to the docker-credentials Secret of the target Harbor integration supplies the credentials. + +### Cause 3: the Harbor integration on the new global cluster has wrong credentials + +When a new global cluster is set up to reuse the Harbor deployed with the previous platform version, the integration does not work if the credentials configured for the existing Harbor are wrong. Removing the integration and adding it again with the correct credentials restores access. + +## Resolution + +### Cause 1 + +1. With the registry address in `/etc/containerd/certs.d` already correct, read the actions granted to the account you pushed with (password lines filtered out): + + ```bash + grep -n -B2 -A8 '' /etc/kubernetes/registry/auth.yaml | grep -v -i password + ``` + + Only `pull` for that user, and `pull`, `push`, `delete` for `admin`, confirms the cause. + +2. Push as `admin`: + + ```bash + printf '%s' "$REGISTRY_ADMIN_PASSWORD" | nerdctl login : --username admin --password-stdin + nerdctl push ://: + ``` + + Do not put the `admin` password into a workload image pull Secret. For pipelines, use an account that has push permission on the target project. + +### Cause 2 + +1. Open the pipeline stage that runs the Buildah task and check its settings: `skipPush` must be `false`, and the `source` workspace is bound to the build workspace. + +2. Find the docker-credentials Secret of the Harbor integration for the target registry. In the UI it appears as ` (harbor) - docker-credentials`; the Secret is named `docker-credentials-`. Print its type and key names only: + + ```bash + kubectl get secret -n | grep docker-credentials + kubectl get secret -n \ + -o go-template='{{.type}}{{"\n"}}{{range $k, $v := .data}}{{$k}}{{"\n"}}{{end}}' + ``` + +3. Bind the task's `registry-config` workspace to that Secret and run the pipeline again. + +4. If the push still fails with `authentication required`, confirm that the account in the credential can push to the Harbor project (log in and push with it from the command line). + +### Cause 3 + +1. Confirm a login with the credentials you intend to use: + + ```bash + printf '%s' "$HARBOR_PASSWORD" | docker login : --username --password-stdin + ``` + + A failed login means the credentials are still wrong; do not re-add the integration until it succeeds. + +2. On the new global cluster, remove the Harbor integration and add it again with those credentials, from the same platform integration screen used to add it. + + > **Warning:** Impact: while the integration is removed, pipelines and projects that use it cannot reach Harbor. Precondition: the login in step 1 succeeded, and the integration settings (name, address, assigned projects) are recorded so they can be entered again. Rollback: add the integration again with the recorded settings. + +## Diagnostic Steps + +1. Separate a permission denial from a missing tag or a scheme error. An `http: server gave HTTP response to HTTPS client` or x509 error happens before the account check. + +2. `insufficient_scope: authorization failed` on push with the platform-deployment user selects Cause 1. Check the user's actions in `auth.yaml` for `push`. + +3. A Buildah task whose pull works and whose push fails with `authentication required` on the blob check selects Cause 2. Check the task's `registry-config` workspace binding first. + +4. A new global cluster that cannot open an older Harbor selects Cause 3. Test the configured username and password with `docker login` before changing anything else. + +5. A UI upload through a toolchain integration that fails with `unauthorized to access repository: ..., action: push` while the same account pushes from the command line is covered in [Harbor returns 401 because of time skew, the node auth container, Harbor Core, or a shared integration credential](Harbor_returns_401_because_of_time_skew_the_node_auth_container_Harbor_Core_or_a_shared_integration_credential.md). diff --git a/docs/en/solutions/harbor-core_cannot_connect_to_Redis_NOAUTH_from_Sentinel_or_ERR_invalid_password_after_a_Redis_password_change.md b/docs/en/solutions/harbor-core_cannot_connect_to_Redis_NOAUTH_from_Sentinel_or_ERR_invalid_password_after_a_Redis_password_change.md new file mode 100644 index 000000000..fc4e8f79a --- /dev/null +++ b/docs/en/solutions/harbor-core_cannot_connect_to_Redis_NOAUTH_from_Sentinel_or_ERR_invalid_password_after_a_Redis_password_change.md @@ -0,0 +1,162 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.4" +--- + +# harbor-core cannot connect to Redis: NOAUTH from Sentinel, or ERR invalid password after a Redis password change + +## Issue + +`harbor-core` keeps restarting, or the Harbor login page reports that the core service is unavailable, and the core log shows that it cannot connect to Redis. Two signatures: + +- A Harbor that uses Redis in Sentinel mode never comes up, or creating a standalone (single-instance) Harbor fails, and the core log has: + + ```text + GetMasterAddrByName master="mymaster" failed: NOAUTH Authentication required + redis: all sentinels specified in configuration are unreachable + ``` + +- After the password of an external Redis was changed by editing the Redis Secret, `harbor-core` no longer starts, although `redis-cli` with the new password answers `PONG` and restarting `harbor-core` does not help: + + ```text + [/lib/cache/cache.go:126]: failed to ping redis://:xxxxx@:6379/0?idle_timeout_seconds=30 : ERR invalid password + [FATAL] [/core/main.go:161]: failed to initialize cache: retry timeout: ERR invalid password + ``` + +Related articles: [Redis for Harbor unavailable: no Data Services license, or Redis in cluster mode reported as empty secrets](Redis_for_Harbor_unavailable_no_Data_Services_license_or_Redis_in_cluster_mode_reported_as_empty_secrets.md). + +## Root Cause + +### Cause 1: the Redis instance has an independent Sentinel password + +Harbor connects to Redis in Sentinel mode through `spec.helmValues.redis.external` on the `Harbor` resource: `addr` lists the Sentinel endpoints on port 26379 (for example `rfs--.rfs--hl.:26379` for ordinals 0, 1, and 2), `existingSecret` names the Redis credential Secret, `existingSecretKey` is `password`, and `sentinelMasterSet` is `mymaster`. + +When the Redis instance is created in Data Services with both `Set Redis Password` and `Set Sentinel Password` enabled, its generated credential Secret (for example `redis--default-credential`) carries both a `password` and a `sentinelPassword` key. Harbor reads only the key named in `existingSecretKey` and never sends the Sentinel password. The Sentinel nodes reject Harbor's master lookup with `NOAUTH`, every Sentinel counts as unreachable, and `harbor-core` cannot find the Redis master. + +**Harbor does not support a Redis Sentinel independent password.** The Redis instance used by Harbor must have `Set Sentinel Password` disabled. The same limitation makes creation of a standalone Harbor fail. A manual `redis-cli` that succeeds with the Sentinel password does not rule this out, because Harbor does not send that password. + +### Cause 2: the operator did not carry a changed external Redis password into harbor-core + +With an external Redis (`redis.type: external`, `existingSecret: harbor-redis`, `existingSecretKey: password`), changing the password by editing the `harbor-redis` Secret does not update `REDIS_URL` in the Harbor core ConfigMap. `harbor-core` keeps authenticating with the old password, and restarting it re-reads the same stale ConfigMap. Triggering a Harbor instance update through the operator regenerates the configuration and `harbor-core` recovers. + +## Resolution + +### Cause 1 + +1. Confirm the cause (see Diagnostic Steps): `NOAUTH Authentication required` from `GetMasterAddrByName` in core, and `Set Sentinel Password` enabled on the Redis instance (or a `sentinelPassword` key in its credential Secret). + +2. In Data Services, open the Redis instance, go to its connection (access) settings, keep `Set Redis Password` enabled, and turn off `Set Sentinel Password`. + + > **Warning:** Only when Cause 1 is confirmed. Impact: the Sentinel endpoints of this Redis instance stop requiring the Sentinel password; the Redis data password stays in force. Other clients of this instance that send the Sentinel password must be checked. Rollback: enable `Set Sentinel Password` again, which returns Harbor to the `NOAUTH` failure. + +3. Watch `harbor-core` until the `NOAUTH` lines stop and the Pod becomes Ready: + + ```bash + NS= + kubectl -n "$NS" get pods -w | grep harbor-core + CORE=$(kubectl -n "$NS" get pods -o name | grep harbor-core | head -1) + kubectl -n "$NS" logs "$CORE" --tail=50 | grep -E 'NOAUTH|sentinels' || echo "no Sentinel auth errors" + ``` + +4. If the option cannot be turned off on the existing instance, recreate the Redis instance with `Set Sentinel Password` disabled. + + > **Warning:** Only when Cause 1 is confirmed and the Sentinel password cannot be disabled on the existing instance. Impact: deleting the Redis instance deletes its data; Harbor keeps cache, session, and job-queue data in Redis, so logged-in sessions end and queued jobs are lost. Precondition: the instance is used only by this Harbor (or Harbor is not yet in service), and a maintenance window. Deletion cannot be undone. + + Find the credential Secret Data Services generated for the new instance, and copy it into the Harbor namespace if it lives elsewhere: + + ```bash + kubectl get secret -n -l middleware.instance/type=Redis + ``` + +5. If the new instance's Sentinel addresses or Secret name differ from what the `Harbor` resource references, update the `Harbor` resource. + + > **Warning:** Only when the replacement Redis has a different address or Secret name. Impact: the operator restarts Harbor components; login, push, and pull fail until the Pods are Ready. Precondition: save the current resource with the first command below. Rollback: apply the saved file. + + ```bash + NS= + NAME= + kubectl -n "$NS" get harbor "$NAME" -o yaml > "$HOME/harbor-$NAME-before.yaml" + kubectl -n "$NS" patch harbor "$NAME" --type=merge -p '{ + "spec": {"helmValues": {"redis": {"type": "external", "external": { + "addr": "rfs--0.rfs--hl.:26379,rfs--1.rfs--hl.:26379,rfs--2.rfs--hl.:26379", + "existingSecret": "", + "existingSecretKey": "password", + "sentinelMasterSet": "mymaster" + }}}} + }' + ``` + +### Cause 2 + +1. Test the new password from a host that can reach Redis, then compare the password in the Secret with the one in the core ConfigMap without printing either: + + ```bash + redis-cli -u "redis://:@:6379/0" PING + NS= + kubectl -n "$NS" get secret harbor-redis -o jsonpath='{.data.password}' | base64 -d | sha256sum + CM=$(kubectl -n "$NS" get cm -o name | grep -- '-core$' | head -1) + kubectl -n "$NS" get "$CM" -o yaml | grep REDIS_URL \ + | sed -E 's#.*://[^:]*:([^@]*)@.*#\1#' | sort -u \ + | while read -r p; do printf '%s' "$p" | sha256sum; done + ``` + + The cause is confirmed when `PING` returns `PONG`, core logs `ERR invalid password`, and the ConfigMap digest differs from the Secret digest. A password with special characters can appear URL-encoded in `REDIS_URL`; decode it before comparing if the digests differ only for that reason. + +2. Trigger a Harbor instance update through the operator: open the instance in the Harbor operator's update view and submit it, or make a change to the `Harbor` resource that the operator reconciles. + + > **Warning:** Only when the cause is confirmed in step 1. Impact: the operator regenerates the Harbor configuration and restarts components; login, push, and pull fail until the Pods are Ready. Precondition: a short maintenance window and the `Harbor` resource saved. Rollback: apply the saved file (the stale password in the ConfigMap is not a state to return to). + + ```bash + NAME= + kubectl -n "$NS" get harbor "$NAME" -o yaml > "$HOME/harbor-$NAME-before-redis.yaml" + kubectl -n "$NS" patch harbor "$NAME" --type=merge \ + -p '{"spec":{"helmValues":{"core":{"resources":{"requests":{"cpu":""}}}}}}' + kubectl -n "$NS" get pods -w | grep harbor-core + ``` + +3. Run the digest comparison again: the two values match and the `ERR invalid password` lines stop. + +## Diagnostic Steps + +1. Read the core log: + + ```bash + NS= + kubectl get harbor -A + CORE=$(kubectl -n "$NS" get pods -o name | grep harbor-core | head -1) + kubectl -n "$NS" logs "$CORE" --tail=200 | grep -iE 'redis|sentinel|NOAUTH|invalid password' + ``` + + `NOAUTH Authentication required` from `GetMasterAddrByName` points to Cause 1; `ERR invalid password` with `redis.type: external` after a password change points to Cause 2. + +2. For Cause 1, check what Harbor is configured with and which keys the credential Secret carries (key names only): + + ```bash + kubectl -n "$NS" get harbor -o jsonpath='{.spec.helmValues.redis}{"\n"}' + kubectl -n "$NS" get secret \ + -o go-template='{{range $k, $v := .data}}{{$k}}{{"\n"}}{{end}}' + ``` + + Expected for Cause 1: a `sentinelPassword` key next to `password`, and `existingSecretKey: password` on the Harbor resource. + +3. Query Sentinel directly on port 26379 (add `-c ` if the Pod has more than one container): + + ```bash + kubectl -n exec rfs--0 -- \ + redis-cli -p 26379 SENTINEL get-master-addr-by-name mymaster + ``` + + `NOAUTH Authentication required` confirms that Sentinel requires its own password; after the option is disabled the command returns the master address and port. `redis-cli` defaults to port 6379, the data port, where a `SENTINEL` command returns `unknown command`. + +4. `redis: all sentinels specified in configuration are unreachable` alone is not specific to Cause 1. If `GetMasterAddrByName` fails with `dial tcp: lookup rfs--.rfs--hl. ...: no such host` instead of `NOAUTH`, the Sentinel Pods do not exist yet. Check them before changing any password setting: + + ```bash + kubectl -n get pods -o wide | grep -E 'rfs-|rfr-' + kubectl -n describe pod rfs--0 | sed -n '/Events:/,$p' + ``` + + `Init:ImagePullBackOff` or `Init:0/1` on the `rfs-` Pods with `Failed to pull image ... i/o timeout` is an image-pull problem on the node, not a password problem. diff --git a/docs/en/solutions/harbor-nginx_or_portal_does_not_start_or_keeps_restarting_IP_address_family_unsupported_or_OOMKilled.md b/docs/en/solutions/harbor-nginx_or_portal_does_not_start_or_keeps_restarting_IP_address_family_unsupported_or_OOMKilled.md new file mode 100644 index 000000000..3b482fdd1 --- /dev/null +++ b/docs/en/solutions/harbor-nginx_or_portal_does_not_start_or_keeps_restarting_IP_address_family_unsupported_or_OOMKilled.md @@ -0,0 +1,102 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "3.12" + - "4.2" + - "4.3" +--- + +# harbor-nginx or portal does not start or keeps restarting: IP address family unsupported or OOMKilled + +## Issue + +The Harbor web entry point is down or unstable because the `harbor-nginx` Pod (and sometimes the portal Pod) does not become Ready or keeps restarting: + +- `harbor-nginx` and the portal Pod never become Ready, and their logs contain `IP address family was unsupported`. +- `harbor-nginx` keeps restarting, and `kubectl describe pod` shows `Last State: Terminated, Reason: OOMKilled`. The Harbor service looks unstable even though `harbor-core` is up. + +Related articles: [Harbor core service unavailable or internal server error, stuck instance status, or Pods cycling after an edit](Harbor_core_service_unavailable_or_internal_server_error_stuck_instance_status_or_Pods_cycling_after_an_edit.md). + +## Root Cause + +### Cause 1: Harbor requires IPv6 on a cluster where IPv6 is disabled + +The Harbor resource is configured with IPv6 enabled, so nginx and the portal try to listen on IPv6. On a cluster where IPv6 is disabled, the listen fails with `IP address family was unsupported` and the Pods never become Ready. Setting the Harbor resource to IPv4 only, under `spec.helmValues.ipFamily`, fixes it. + +### Cause 2: the harbor-nginx memory limit is too low + +`harbor-nginx` exceeds its memory limit and is OOMKilled. A change made directly on the `harbor-nginx` Deployment is overwritten by the operator's sync unless the Deployment carries the annotation `skip-sync: "true"`. Adding that annotation and raising the memory limit to twice the original value stops the restarts. + +## Resolution + +### Cause 1 + +1. Confirm the error in the nginx and portal logs, and that the nodes are IPv4-only (see Diagnostic Steps). Save the current `ipFamily` block if one is set: + + ```bash + NS= + NAME= + kubectl -n "$NS" get harbor "$NAME" -o jsonpath='{.spec.helmValues.ipFamily}{"\n"}' + ``` + +2. Set IPv4 only. + + > **Warning:** Only when the nginx and portal logs say the IP address family is unsupported and the cluster is IPv4-only. Impact: the operator reconciles and restarts Harbor components. Precondition: a short maintenance window and the previous `ipFamily` values saved. Rollback: patch the previous values back; on an IPv4-only cluster they are the failing configuration, so roll back only when moving to a cluster that provides IPv6. + + ```bash + kubectl -n "$NS" patch harbor "$NAME" --type=merge -p \ + '{"spec":{"helmValues":{"ipFamily":{"ipv6":{"enabled":false},"ipv4":{"enabled":true}}}}}' + kubectl -n "$NS" get pods | grep -E 'nginx|portal' + ``` + +3. Wait until the nginx and portal Pods are Ready and check that their logs no longer contain the address-family error. + +### Cause 2 + +1. Confirm OOMKilled and read the current memory limit: + + ```bash + NS= + kubectl -n "$NS" describe pod | grep -A3 'Last State' + DEP=$(kubectl -n "$NS" get deploy -o name | grep harbor-nginx | head -1) + kubectl -n "$NS" get "$DEP" -o jsonpath='{.spec.template.spec.containers[0].resources.limits.memory}{"\n"}' + ``` + +2. Mark the Deployment so the operator does not overwrite it, and double the memory limit (`512Mi` becomes `1Gi`, `1Gi` becomes `2Gi`). + + > **Warning:** Only when `harbor-nginx` is OOMKilled. Impact: `skip-sync: "true"` stops the operator from syncing this Deployment, so later changes from the Harbor resource no longer reach it until the annotation is removed; the Pod restarts once with the new limit and may use twice the previous memory on its node. Precondition: the node has that much allocatable memory. Rollback: set the limit back to the printed value and remove the annotation (`kubectl -n "$NS" annotate "$DEP" skip-sync-`); removing only the annotation lets the operator restore the old limit and the OOMKilled loop. + + ```bash + kubectl -n "$NS" annotate "$DEP" skip-sync="true" --overwrite + kubectl -n "$NS" set resources "$DEP" --limits=memory= + kubectl -n "$NS" rollout status "$DEP" + ``` + +3. Watch the Pod. Another OOMKilled means doubling was not enough for that workload. + +For other Harbor components, such as `harbor-registry`, the limit is set on the Harbor resource instead; see [harbor-registry is OOMKilled with exit code 137, or a resource change on its Deployment is reverted](harbor-registry_is_OOMKilled_with_exit_code_137_or_a_resource_change_on_its_Deployment_is_reverted.md). + +## Diagnostic Steps + +```bash +NS= +kubectl -n "$NS" get pods -o wide | grep -E 'nginx|portal' +NGINX=$(kubectl -n "$NS" get pods -o name | grep harbor-nginx | head -1) +kubectl -n "$NS" logs "$NGINX" --tail=80 +kubectl -n "$NS" describe "$NGINX" | grep -A3 'Last State' +``` + +| What you see | Cause | +| --- | --- | +| `IP address family was unsupported` in nginx or portal logs, Pods never Ready | Cause 1 | +| `harbor-nginx` restarts, `Last State` `OOMKilled` | Cause 2 | +| `harbor-notary-server` restarts with `certificate has expired`, or the portal says the core service is unavailable | Related article above | + +To confirm the cluster is IPv4-only for Cause 1, check the node addresses; only IPv4 entries are expected: + +```bash +kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.addresses[*].address}{"\n"}{end}' +``` diff --git a/docs/en/solutions/harbor-registry_is_OOMKilled_with_exit_code_137_or_a_resource_change_on_its_Deployment_is_reverted.md b/docs/en/solutions/harbor-registry_is_OOMKilled_with_exit_code_137_or_a_resource_change_on_its_Deployment_is_reverted.md new file mode 100644 index 000000000..824f14cc7 --- /dev/null +++ b/docs/en/solutions/harbor-registry_is_OOMKilled_with_exit_code_137_or_a_resource_change_on_its_Deployment_is_reverted.md @@ -0,0 +1,146 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "3.14" + - "3.18" +--- + +# harbor-registry is OOMKilled with exit code 137, or a resource change on its Deployment is reverted + +## Issue + +The Harbor `registry` component keeps restarting and its log shows no obvious error, or it has hit its CPU or memory limit and the image registry is unavailable. Typical signs: + +- The `harbor-registry` Pod's `RESTARTS` count grows, it may alternate between `Running` and `CrashLoopBackOff`, and its last state shows exit code `137`. This can start after the node container runtime was switched from Docker to containerd. +- An engineer raises the limits on the `harbor-registry` Deployment, and the change is reverted shortly afterwards. +- Clicking Update on the Harbor instance reports that some operators are not ready because they were never deployed. After the Redis and PostgreSQL operators are installed, Update asks for a storage class that a local-PV instance does not have. + +## Root Cause + +### Cause 1: the registry container is OOMKilled at its memory limit + +The kernel kills the `registry` container when it reaches its memory limit. The container's last state shows reason `OOMKilled` and exit code `137` (`128 + 9`, the process ended by `SIGKILL`), and the registry log has no error because the process was killed from outside. Raising the resources of the Harbor instance stops the restarts. Because the operator owns the Deployment, the change must be made on the Harbor instance, not on the Deployment (see Cause 2). + +### Cause 2: the Harbor operator owns the component resources and reverts Deployment edits + +Harbor component resources are controlled by the `Harbor` custom resource. The Harbor operator reconciles the Deployments from it, so a manual edit of the Deployment limits is written back. With `spec.resourceAssign.type: GlobalAssign`, the operator derives per-component values from one global setting (for example cpu `"16"` and memory `16Gi`) and writes them to `status.helmValues`; those status values are what the Deployments get. The API group can be `operator.devops.alauda.io/v1alpha1` or `operator.alaudadevops.io/v1alpha1` depending on the Harbor line; `kubectl get harbor` works for both. + +Two ways give a component its own size: + +- **The Update form** on the instance details page, which switches from global to per-component assignment. It requires the Redis and PostgreSQL operators to be installed. On platform 3.18 it also moves Harbor's PostgreSQL and Redis to Data Services-managed instances on TopoLVM storage, which is why it asks for a storage class that an instance with `persistence.type: LocalPath` does not have. That is an upgrade path, not only a limit change. +- **Component resources in the Harbor resource** under `spec.helmValues`. For the registry container the path is `spec.helmValues.registry.registry.resources`: the outer `registry` is the Harbor registry component, the inner `registry` is its registry container, next to `controller` (the `registryctl` container). Other components, such as `core`, sit at the same level as the outer `registry`. This path works on a `LocalPath` instance without a storage migration. A common mistake is `spec.helmValues.registry.resources`, which the operator does not use for the registry container. + +## Resolution + +### Cause 1 + +1. Confirm the OOM kill and read the current values (Diagnostic Steps, steps 1 and 2). + +2. Raise the memory of the container whose last state is `OOMKilled` on the Harbor resource: `registry` at `spec.helmValues.registry.registry.resources`, or `registryctl` at `spec.helmValues.registry.controller.resources`. Choose a limit above the current one in `status.helmValues` and above the usage seen just before the kills, and keep the request no higher than a node can schedule. You can also update the instance resources from the UI. + + > **Warning:** Only when the `OOMKilled` / `137` signature is confirmed. Impact: the operator reconciles the Harbor Deployments; `harbor-registry` Pods roll, and pushes and pulls are interrupted until they are Ready. Precondition: the backup below is saved and the new limit fits on the nodes. Rollback: `kubectl apply -f /tmp/harbor-before-oom.yaml`. + + ```bash + kubectl -n get harbor -o yaml > /tmp/harbor-before-oom.yaml + kubectl -n edit harbor + ``` + + Merge the block into the existing `spec.helmValues`; do not replace keys such as `externalURL`, `expose`, or `persistence`: + + ```yaml + spec: + helmValues: + registry: + registry: + resources: + limits: + memory: + requests: + memory: + ``` + +3. Confirm the value reached the operator and the Deployment, and that the restart count stops growing under normal load: + + ```bash + kubectl -n get harbor -o jsonpath='{.status.helmValues.registry.registry.resources}{"\n"}' + kubectl -n get deploy -o jsonpath='{.spec.template.spec.containers[*].resources}{"\n"}' + kubectl -n rollout status deployment + kubectl get pods -A | grep harbor-registry + ``` + + If the new Pod is `Pending`, read its events for `Insufficient memory` and lower the request or free a node. If the Deployment does not change, the edit missed the field the operator reads. + +If all Harbor Pods start cycling after the edit, check whether the registry PVC is larger than the size declared in the Harbor resource; see [Harbor core service unavailable or internal server error, stuck instance status, or Pods cycling after an edit](Harbor_core_service_unavailable_or_internal_server_error_stuck_instance_status_or_Pods_cycling_after_an_edit.md) (Cause 4). + +### Cause 2 + +1. Save the resource and read how resources are assigned: + + ```bash + kubectl get harbor -A + kubectl -n get harbor -o yaml > /tmp/harbor-resources.yaml + grep -n -E 'resourceAssign|GlobalAssign|currentVersion|LocalPath|persistence' /tmp/harbor-resources.yaml + kubectl -n get harbor -o jsonpath='{.status.helmValues.registry}{"\n"}' + ``` + +2. Choose the path: + + - Use the Update form only when the move of PostgreSQL and Redis to Data Services instances on TopoLVM is the change you intend. If Update says operators were never deployed, install the Redis and PostgreSQL operators first. + - Otherwise, especially on a `LocalPath` instance, add component resources to the Harbor resource. Size them for your workload, comparing with the current value in `status.helmValues`. Example values for the registry container: + + > **Warning:** Impact: the operator reconciles the Harbor Deployments; `harbor-registry` Pods roll, and a limit that is still too small leaves the registry unavailable. Precondition: `/tmp/harbor-resources.yaml` is a complete copy of the object. Rollback: `kubectl apply -f /tmp/harbor-resources.yaml`. + + ```bash + kubectl -n edit harbor + ``` + + ```yaml + spec: + helmValues: + registry: + registry: + resources: + limits: + cpu: 300m + memory: 800Mi + requests: + cpu: 150m + memory: 400Mi + ``` + + Leave other components out unless you mean to resize them, or add them at the same level as `registry` (for example `spec.helmValues.core.resources`). + +3. Confirm with the commands in Cause 1, step 3. Do not edit the Deployment again: that is the change the operator reverts. When `harbor-registry` is Ready, pull one known tag. + +## Diagnostic Steps + +1. Read the last termination state of each container in the registry Pod, its events, and the previous log: + + ```bash + kubectl -n get pod -o jsonpath='{range .status.containerStatuses[*]}{.name}{" restarts="}{.restartCount}{" reason="}{.lastState.terminated.reason}{" exit="}{.lastState.terminated.exitCode}{"\n"}{end}' + kubectl -n get events --field-selector involvedObject.name= + kubectl -n logs -c registry --previous --tail=50 + ``` + + `OOMKilled` with exit code `137` is Cause 1. Exit code `137` without `OOMKilled` is a SIGKILL from elsewhere: check the events for a failing liveness probe before changing memory. A storage, configuration, or permission error at the end of the previous log points at the registry storage instead. + +2. Compare memory use with the limit, and look for the kernel OOM line on the node that ran the Pod. Usage that climbs to the limit before each restart confirms Cause 1 and gives the floor for the new limit: + + ```bash + kubectl top pod -n --containers + journalctl -k --since "1 hour ago" | grep -iE 'out of memory|oom-kill|killed process' + ``` + +3. Check whether the operator owns the values: + + ```bash + kubectl -n describe deploy | grep -A8 -E 'Limits|Requests' + kubectl -n get harbor -o yaml | grep -n -E 'resourceAssign|GlobalAssign|LocalPath' + ``` + + `GlobalAssign` and a reverted Deployment edit are Cause 2. + +4. If the restarting component is `harbor-nginx` rather than `harbor-registry`, see [harbor-nginx or portal does not start or keeps restarting: IP address family unsupported or OOMKilled](harbor-nginx_or_portal_does_not_start_or_keeps_restarting_IP_address_family_unsupported_or_OOMKilled.md). If nodes cannot pull images after a Docker-to-containerd switch (`dial tcp :80: i/o timeout`), see [Image registry or Harbor unreachable or slow because of firewall rules, blocked ports, or the network path](Image_registry_or_Harbor_unreachable_or_slow_because_of_firewall_rules_blocked_ports_or_the_network_path.md) (Cause 6). From 6a0ee19856bdcd765d2ebce8d711fa320935daf3 Mon Sep 17 00:00:00 2001 From: Komh Date: Thu, 12 Nov 2026 21:31:59 +0800 Subject: [PATCH 08/25] docs(solutions): add 24 and update 1 troubleshooting articles --- ...because_erebus_or_frontend_is_unhealthy.md | 131 +++++++++ ...he_standby_global_cluster_cannot_log_in.md | 126 +++++++++ ...ror_or_empty_GitOps_project_list_on_ACP.md | 148 ++++++++++ ..._an_ambient_mesh_or_dev_shm_full_on_ACP.md | 144 ++++++++++ ..._hangs_or_the_Nexus_connection_is_reset.md | 152 +++++++++++ ...ty_and_the_Dex_deployment_layout_on_ACP.md | 157 +++++++++++ ...1_after_the_Dex_permanent_token_changes.md | 92 +++++++ ...tlabOfficial_edits_or_cannot_be_deleted.md | 235 ++++++++++++++++ ...icate_verify_invalid_client_LDAP_groups.md | 131 +++++++++ ...rojects_API_500_or_an_extra_bot_project.md | 142 ++++++++++ ...nregistered_redirect_uri_console-portal.md | 163 +++++++++++ ...in_timeout_or_push_asks_for_credentials.md | 93 +++++++ ...one_node_or_a_buildah_certificate_error.md | 148 ++++++++++ ...ration_repositories_request_returns_500.md | 94 +++++++ ...though_build_node_resources_were_raised.md | 89 ++++++ ...s_republished_to_the_private_repository.md | 107 ++++++++ ...in_shows_Invalid_parameter_redirect_uri.md | 137 ++++++++++ ...riaMetrics_or_shows_too_many_namespaces.md | 255 ++++++++++++++++++ ..._HTTPS_Required_after_generate-settings.md | 95 +++++++ ..._to_the_wrong_Dex_issuer_address_on_ACP.md | 96 +++++++ ...tes_because_the_Maven_stage_runs_mvn_-U.md | 48 ++++ ...ll_Connector_placeholder_or_PodSecurity.md | 167 ++++++++++++ ...piserver_OIDC_for_an_external_UI_on_ACP.md | 89 ++++++ ...credentials_and_Connector_tokens_on_ACP.md | 155 +++++++++++ .../nacos/How_to_Deploy_Nacos_2.2.md | 11 + 25 files changed, 3205 insertions(+) create mode 100644 docs/en/solutions/ACP_login_loops_or_returns_to_the_login_page_because_erebus_or_frontend_is_unhealthy.md create mode 100644 docs/en/solutions/ACP_login_shows_an_internal_error_from_a_bad_Dex_cache_or_the_standby_global_cluster_cannot_log_in.md create mode 100644 docs/en/solutions/Argo_CD_Application_sync_not_permitted_repository_Not_Found_InvalidSpecError_or_empty_GitOps_project_list_on_ACP.md create mode 100644 docs/en/solutions/Argo_CD_components_fail_Dex_image_pull_when_air-gapped_Redis_timeouts_in_an_ambient_mesh_or_dev_shm_full_on_ACP.md create mode 100644 docs/en/solutions/DevOps_Node.js_or_apt_downloads_fail_gnutls_handshake_failed_npm_install_hangs_or_the_Nexus_connection_is_reset.md create mode 100644 docs/en/solutions/Dex_login_options_CORS_SSO_user_identity_and_the_Dex_deployment_layout_on_ACP.md create mode 100644 docs/en/solutions/Dex_update_fails_on_ACP_3.10_or_account_login_returns_401_after_the_Dex_permanent_token_changes.md create mode 100644 docs/en/solutions/GitLab_instance_upgrade_fails_breaks_after_a_Data_Services_move_reverts_GitlabOfficial_edits_or_cannot_be_deleted.md create mode 100644 docs/en/solutions/GitLab_or_Argo_CD_login_through_the_platform_Dex_fails_host_required_certificate_verify_invalid_client_LDAP_groups.md create mode 100644 docs/en/solutions/GitLab_toolchain_integration_fails_token-gitlab_not_found_Internal_error_projects_API_500_or_an_extra_bot_project.md create mode 100644 docs/en/solutions/Harbor_SSO_through_the_platform_Dex_fails_wrong_issuer_OAuth2Client_name_or_Unregistered_redirect_uri_console-portal.md create mode 100644 docs/en/solutions/Harbor_toolchain_integration_fails_images_not_selectable_katanomi-plugin_timeout_or_push_asks_for_credentials.md create mode 100644 docs/en/solutions/Image_build_fails_with_PodSecurity_privileged_a_RUN_step_failing_on_one_node_or_a_buildah_certificate_error.md create mode 100644 docs/en/solutions/Image_tag_cannot_be_selected_in_Update_Image_Version_because_the_integration_repositories_request_returns_500.md create mode 100644 docs/en/solutions/Jenkins_Maven_build_fails_with_java.lang.OutOfMemoryError_Java_heap_space_although_build_node_resources_were_raised.md create mode 100644 docs/en/solutions/Jenkins_Maven_build_packages_an_old_dependency_jar_after_the_same_version_is_republished_to_the_private_repository.md create mode 100644 docs/en/solutions/Keycloak_shows_HTTPS_required_the_CR_warns_about_hostname-strict_or_login_shows_Invalid_parameter_redirect_uri.md create mode 100644 docs/en/solutions/Kiali_will_not_open_or_log_in_shows_Core_istiod_Not_found_cannot_reach_VictoriaMetrics_or_shows_too_many_namespaces.md create mode 100644 docs/en/solutions/Maven_stage_misses_Nexus_mirrorOf_stays_central_after_mavenMirrorURL_or_501_HTTPS_Required_after_generate-settings.md create mode 100644 docs/en/solutions/OIDC_clients_and_pages_are_sent_to_the_wrong_Dex_issuer_address_on_ACP.md create mode 100644 docs/en/solutions/Pipeline_takes_more_than_40_minutes_because_the_Maven_stage_runs_mvn_-U.md create mode 100644 docs/en/solutions/Pipeline_task_Pod_does_not_start_connector-csi_OOMKilled_controller_image_pull_Connector_placeholder_or_PodSecurity.md create mode 100644 docs/en/solutions/Toolchain_SSO_on_a_domain_name_after_an_IP-based_install_and_kube-apiserver_OIDC_for_an_external_UI_on_ACP.md create mode 100644 docs/en/solutions/auth-controller2_webhook_certificate_missing_or_log_burst_monitoring_query_credentials_and_Connector_tokens_on_ACP.md diff --git a/docs/en/solutions/ACP_login_loops_or_returns_to_the_login_page_because_erebus_or_frontend_is_unhealthy.md b/docs/en/solutions/ACP_login_loops_or_returns_to_the_login_page_because_erebus_or_frontend_is_unhealthy.md new file mode 100644 index 000000000..053b83a21 --- /dev/null +++ b/docs/en/solutions/ACP_login_loops_or_returns_to_the_login_page_because_erebus_or_frontend_is_unhealthy.md @@ -0,0 +1,131 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.0" + - "4.2" +--- + +# ACP login loops or returns to the login page because erebus or frontend is unhealthy + +## Issue + +After the username and password are accepted, the browser returns to the login page, or it bounces between the platform and the SSO page until login fails. Every account is affected, local and LDAP alike, and the platform cannot be accessed at all. It often happens on the first morning of the work week. The frontend log names the global erebus API as abnormal, typically with: + +```text +oidc verify token from acp ... "error": "failed to send request: Get \"https://erebus/kubernetes/global/api\": dial tcp :443: connect: cannot assign requested address" +http: superfluous response.WriteHeader call from .../alauda-backend/pkg/server/options.(*ResponseWriterDelegator).WriteHeader (metrics.go:319) +``` + +Related articles: "ACP login shows an internal error from a bad Dex cache, or the standby global cluster cannot log in" covers an internal error on the login page and standby clusters. An `invalid authorization code` or `Unregistered redirect_uri` page is a different failure. + +## Root Cause + +### Cause 1: erebus is unhealthy and the console cannot finish login + +The erebus log reports that it cannot find cluster-related information, or the erebus Pod is otherwise abnormal, and the frontend log says the global erebus API is unhealthy. Restarting the erebus Pod restores login. + +A variant: erebus keeps looking up a cluster that has already been deleted from the platform, and its log repeats `websocket.go:514] Websocket Ping failed: write tcp :->:443: write: connection timed out` together with `handler.go:66] Handle request error: client rate limiter Wait returned an error: context canceled`. The source address belongs to the `archon` Pod in `cpaas-system`. Apollo looks healthy, and restarting Apollo does not help. erebus appears to hang while retrying the deleted cluster; the exact mechanism is not confirmed. Restarting erebus restores login for all users as a temporary workaround. + +### Cause 2: local ports are exhausted for the connections to erebus + +Login fails, and the erebus log says no port is available because short-lived TCP connections are exhausted. The frontend log shows the `connect: cannot assign requested address` line from the Issue section, and the erebus log is a continuous stream of `proxy.go:150] Generate new reverse proxy for: 600:https://:6443` mixed with `log.go:245] http: proxy error: context canceled` and `timeout.go:142] post-timeout activity` for GET requests such as `/api/v1/namespaces/cpaas-system/configmaps`. Restarting both frontend and erebus restores login. + +A related trace, before a restart, shows erebus logging `asyncopy.go:16] async copy data error: read tcp :9443->:: read: connection reset by peer`, and the `archon-api` container of the `archon` Pod logging `websocket.go:500] Websocket Ping failed: write tcp :->:443: write: connection timed out`. The Service still lists the erebus Pod as an endpoint, but the Pod is not reading. Another Pod on the same node that leaks connections can exhaust the node's ephemeral ports. One candidate is courier-api: when Operations Center log queries fail with `dial tcp :9200: connect: cannot assign requested address`, the courier-api network namespace can hold tens of thousands of established connections to the Elasticsearch endpoint on port 9200. This link was not proven, so collect evidence before restarting if it happens again. + +Clusters removed only from the platform UI can keep printing errors such as `cluster.go:284] cluster: failed to get cluster "" resource. err: clusters.clusterregistry.k8s.io "" not found` and `proxy.go:128] Resolve cluster error: ... not found`. Deleting a cluster in the UI drops platform management of that cluster and leaves its components in place. + +### Cause 3: a frontend socket leak turns a CAS retry loop into a dead console + +When the CAS session is not logged in, platform login retries on its own, over and over, and after enough retries the console stops accepting logins. ALB entries for console APIs point at frontend, and frontend logs that it cannot connect to erebus, with the `connect: cannot assign requested address` and `superfluous response.WriteHeader` lines. Yet the `erebus` Service in `cpaas-system` (ports `443/TCP,6060/TCP`) has both Pods `1/1 Running` with no restarts, and `curl` to erebus succeeds. The cause is a socket leak in frontend, seen on v3.18.2. Restarting frontend restores login. A product fix is expected in a version after 4.4. + +## Resolution + +### Cause 1: erebus is unhealthy and the console cannot finish login + +1. Confirm that local and LDAP logins fail the same way and that erebus is the abnormal component: + + ```bash + kubectl get deploy,po -n cpaas-system | grep -i erebus + kubectl logs -n cpaas-system --tail=200 | grep -E 'Websocket Ping failed|rate limiter|not found' + kubectl get pods -A -o wide | grep + kubectl get svc -A | grep + ``` + +2. Restart erebus. + + > **Warning:** Restarting erebus drops console calls that go through that API until the new Pod is Ready. Precondition: only when this cause is confirmed. The restart does not change configuration. Rollback: wait for the new Pod; there is no object to restore. + + ```bash + kubectl -n cpaas-system rollout restart deploy/ + kubectl -n cpaas-system rollout status deploy/ + ``` + +3. Retry a local user and an LDAP user. When erebus was retrying a deleted cluster, also remove the leftover components of that cluster (see Cause 2), because the restart is only a temporary workaround. + +### Cause 2: local ports are exhausted for the connections to erebus + +1. Confirm the port or timeout signature. If the outage is still in progress, collect the logs before restarting anything: + + ```bash + kubectl get deploy,po -A | grep -Ei 'erebus|archon|courier-api' + kubectl logs -n cpaas-system --tail=200 | grep -E 'connection reset by peer|no port|Generate new reverse proxy|post-timeout' + kubectl logs -n cpaas-system -c archon-api --tail=200 | grep -F 'connection timed out' + ``` + +2. To see which Pod is exhausting ports, count its connections inside its network namespace on the node where it runs (the example uses courier-api; the same commands work for frontend or erebus): + + ```bash + crictl ps -a | grep courier + crictl inspect | grep -m1 '"pid"' + nsenter -n -t ss -s + nsenter -n -t ss -ant | grep ':9200' | grep -c ESTAB + cat /proc/sys/net/ipv4/ip_local_port_range + ``` + + Tens of thousands of `ESTAB` connections to one endpoint, close to the size of `ip_local_port_range`, mean that Pod has used up the ephemeral ports. + +3. Restart erebus, and frontend when the log says no port is available. + + > **Warning:** Restarting erebus and frontend interrupts login for the length of the rollout. Precondition: only when the port-exhaustion or erebus/archon timeout lines above are present. Rollback: none beyond the new Pods becoming Ready. + + ```bash + kubectl -n cpaas-system rollout restart deploy/ + kubectl -n cpaas-system rollout restart deploy/frontend + ``` + +4. When clusters deleted in the UI are still emitting `clusters.clusterregistry.k8s.io "" not found` errors, remove the platform components that remain on the deleted cluster by following the product procedure for removing a cluster, then restart courier-api on the global cluster. Restarting courier-api drops notification and related API calls until the new Pod is Ready. + +### Cause 3: a frontend socket leak turns a CAS retry loop into a dead console + +1. Confirm that frontend cannot reach erebus while erebus itself is healthy: + + ```bash + kubectl logs -n cpaas-system deploy/frontend --tail=200 | grep -E 'cannot assign requested address|superfluous' + kubectl get pods -n cpaas-system | grep erebus + curl -k -sS -o /dev/null -w '%{http_code}\n' https:/// + ``` + +2. Restart frontend, not erebus. + + > **Warning:** Restarting `cpaas-system/frontend` makes the console unavailable until the rollout finishes; business workloads are not restarted. Precondition: only when this cause is confirmed. Rollback: wait for the rollout. The leak can return until the product fix is installed. + + ```bash + kubectl -n cpaas-system rollout restart deploy/frontend + kubectl -n cpaas-system rollout status deploy/frontend + ``` + +## Diagnostic Steps + +```bash +kubectl get deploy,po -n cpaas-system | grep -Ei 'erebus|frontend|apollo|archon|courier' +kubectl logs -n cpaas-system deploy/frontend --tail=80 +``` + +- Redirect loop or return to the login page for both local and LDAP users; erebus cannot find cluster information, retries a deleted cluster with `Websocket Ping failed ... connection timed out`, or its Pod is abnormal: Cause 1. +- erebus says no port is available or streams `Generate new reverse proxy for`, or erebus and archon log connection resets and write timeouts: Cause 2. Collect logs and `ss -s` output before restarting if the fault is still happening. +- CAS retries for a long time, frontend logs `cannot assign requested address` toward erebus, the erebus Pods are healthy, and `curl` to erebus works: Cause 3. Restart frontend. +- `kubectl get cls` on the global cluster shows a cluster abnormal: check whether the cluster kubeconfig Secrets have expired before restarting erebus or frontend. diff --git a/docs/en/solutions/ACP_login_shows_an_internal_error_from_a_bad_Dex_cache_or_the_standby_global_cluster_cannot_log_in.md b/docs/en/solutions/ACP_login_shows_an_internal_error_from_a_bad_Dex_cache_or_the_standby_global_cluster_cannot_log_in.md new file mode 100644 index 000000000..3bf47f63a --- /dev/null +++ b/docs/en/solutions/ACP_login_shows_an_internal_error_from_a_bad_Dex_cache_or_the_standby_global_cluster_cannot_log_in.md @@ -0,0 +1,126 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.0" + - "4.2" +--- + +# ACP login shows an internal error from a bad Dex cache, or the standby global cluster cannot log in + +## Issue + +One of these is reported: + +- The platform login page shows an internal error, and the Dex log repeats `ERROR error handling request {"error": "Internal error occurred: login error."}`. A variant shows `no matches for kind "Secret" in version "v1"` in the Dex log. +- On a primary/standby deployment the standby console does not log in. The browser developer tools show no error, and restarts of frontend, Apollo, and erebus do not help. + +Related articles: "ACP login loops or returns to the login page because erebus or frontend is unhealthy" covers redirect loops. An `invalid authorization code` or `Unregistered redirect_uri` page is a different failure. + +## Root Cause + +### Cause 1: Dex answers login from erroneous cached content + +The Dex log shows the captcha check passing (`check captcha verifyResult:{NeedReturnCaptcha:false IsLoginError:false CaptchaNotMatch:false CaptchaExpired:false}, err:`) and then, repeatedly: + +```text +ERROR decrypt password The encryption timestamp has expired +ERROR error handling request {"error": "Internal error occurred: login error."} +ERROR Failed to get auth request: authrequests.dex.coreos.com "" not found +Failed to get auth request: authrequests.dex.coreos.com "undefined" not found +WARN database inconsistent, refresh token missing: +``` + +or, in the variant, for each login: + +```text +error handling request {"error": "Internal error occurred: no matches for kind \"Secret\" in version \"v1\""} +decrypt password failed to login user: no matches for kind "Secret" in version "v1" +error handling request {"error": "Internal error occurred: login error."} +Failed to get auth request: authrequests.dex.coreos.com "" not found +``` + +Dex holds erroneous cached content and serves it on the login request; the exact mechanism is not confirmed. Restarting the Dex Pod on the global cluster drops the cache, and login works again. No ConfigMap edit is needed. Periodic `garbage collection run, delete auth requests=...` lines are normal background output. + +### Cause 2: the standby cluster token in `tke-cluster-global` does not match the current `k8sadmin` token + +On the standby global cluster, `cpaas-system` holds several rotated `kubernetes.io/service-account-token` Secrets named `k8sadmin-`, labeled `cpaas.io/managed-by: k8sadmin-token-rotator` and annotated with `cpaas.io/cluster-token-expires-at`. The Secret `tke-cluster-global` in `cpaas-system` (label `platform.tkestack.io/clustercredential: cc-global`) still carries a much older creation time, and its token no longer matches the newest `k8sadmin-` token. A request to `/kubernetes/global/api/v1/namespaces` with the `tke-cluster-global` token returns 403 `forbidden: User "system:anonymous" cannot get path ...`, which shows that the token is not accepted as any identity. Copying the current `k8sadmin` token into `tke-cluster-global` restores login. + +The Dex signing key `signingkeies.dex.coreos.com/openid-connect-keys` in `cpaas-system` has the same `signingKeyPub` and `verificationKeys` on both sides in this case, so the signing key is not the mismatch. + +## Resolution + +### Cause 1: Dex answers login from erroneous cached content + +1. Confirm the Dex log lines. On 4.x, Dex runs inside the `apollo` Deployment in `cpaas-system`; on a cluster that still has a `dex` Deployment, read and restart that one instead. + + ```bash + kubectl get deploy -n cpaas-system | grep -Ei 'apollo|dex' + kubectl logs -n cpaas-system deploy/apollo --tail=300 | grep -E 'login error|encryption timestamp|no matches for kind|Failed to get auth request|refresh token missing' + ``` + +2. Restart the workload that runs Dex. + + > **Warning:** Restarting Apollo (or Dex) interrupts logins until the Pod is Ready. Precondition: only when the page is an internal error and the Dex log shows the lines above. Rollback: wait for the rollout; the cache is discarded by design. + + ```bash + kubectl -n cpaas-system rollout restart deploy/apollo + kubectl -n cpaas-system rollout status deploy/apollo + ``` + +3. Log in again, and keep watching the Dex log for a recurrence. + +### Cause 2: the standby cluster token in `tke-cluster-global` does not match the current `k8sadmin` token + +Apply this only when restarts of frontend, Apollo, and erebus did not help, the Dex public keys match on both sides, and the `tke-cluster-global` token is rejected while a current `k8sadmin-` token exists. + +1. Compare the Secrets on the standby global cluster: + + ```bash + kubectl get signingkeies.dex.coreos.com openid-connect-keys -n cpaas-system -o yaml + kubectl -n cpaas-system get secret --sort-by=.metadata.creationTimestamp | grep k8sadmin + kubectl -n cpaas-system get secret tke-cluster-global -o yaml | grep -E 'creationTimestamp|clustercredential' + ``` + +2. Decode the `tke-cluster-global` token and test it against the platform proxy. List the keys first and use the key that holds the token: + + ```bash + kubectl -n cpaas-system get secret tke-cluster-global -o go-template='{{range $k,$v := .data}}{{$k}}{{"\n"}}{{end}}' + TOKEN=$(kubectl -n cpaas-system get secret tke-cluster-global -o jsonpath='{.data.}' | base64 -d) + curl -k -sS -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer ${TOKEN}" \ + https:///kubernetes/global/api/v1/namespaces + ``` + + A `403` that names `system:anonymous`, or a `401`, means the token is not valid. Compare it with the `token` key of the newest `k8sadmin-` Secret: + + ```bash + kubectl -n cpaas-system get secret -o jsonpath='{.data.token}' | base64 -d; echo + ``` + +3. Replace the token in `tke-cluster-global`. + + > **Warning:** Replacing the token in `tke-cluster-global` changes the credential that the platform uses for the global cluster; a wrong value keeps login broken. Precondition: only when the decoded tokens differ and the other checks above have ruled out erebus and the Dex signing key. Rollback: `kubectl apply -f /tmp/tke-cluster-global.bak.yaml`. + + ```bash + kubectl -n cpaas-system get secret tke-cluster-global -o yaml > /tmp/tke-cluster-global.bak.yaml + kubectl -n cpaas-system edit secret tke-cluster-global + ``` + + Set its token to the token from the newest `k8sadmin-` Secret, keeping the encoding the key already uses (base64 under `data`), and save. + +4. Log in to the standby console again. The portal loads normally. + +## Diagnostic Steps + +```bash +kubectl get deploy,po -n cpaas-system | grep -Ei 'apollo|dex|frontend|erebus' +kubectl -n cpaas-system get secret | grep -E 'tke-cluster-global|k8sadmin' +kubectl get signingkeies.dex.coreos.com openid-connect-keys -n cpaas-system +``` + +- Login internal error, and the Dex log shows `Internal error occurred: login error.` with `The encryption timestamp has expired` and `Failed to get auth request`: Cause 1. +- Login internal error, and the Dex log shows `decrypt password failed to login user: no matches for kind "Secret" in version "v1"` after a passing captcha check: Cause 1. +- Standby login still fails after frontend, Apollo, and erebus restarts, and the `tke-cluster-global` token is rejected while a newer `k8sadmin-` exists: Cause 2. diff --git a/docs/en/solutions/Argo_CD_Application_sync_not_permitted_repository_Not_Found_InvalidSpecError_or_empty_GitOps_project_list_on_ACP.md b/docs/en/solutions/Argo_CD_Application_sync_not_permitted_repository_Not_Found_InvalidSpecError_or_empty_GitOps_project_list_on_ACP.md new file mode 100644 index 000000000..9d81ff7f3 --- /dev/null +++ b/docs/en/solutions/Argo_CD_Application_sync_not_permitted_repository_Not_Found_InvalidSpecError_or_empty_GitOps_project_list_on_ACP.md @@ -0,0 +1,148 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Argo CD Application sync not permitted, repository Not Found, InvalidSpecError, or empty GitOps project list on ACP + +## Issue + +Argo CD on ACP is reconciled by `argocd-operator` from an `ArgoCD` resource (`argoproj.io/v1beta1`), and the GitOps pages of the platform work on top of it. One of these is reported: + +- An Application fails with `SyncError Failed sync attempt to : one or more synchronization tasks are not valid resource : is not permitted in project `, for example `resource operator.alauda.io:ResourceSyncer is not permitted in project `. Adding a cluster resource allow list in the Argo CD UI does not stick (the project page then shows "The cluster resource allow list is empty"), and `kubectl edit appproject -n argocd` reverts after saving. +- On **GitOps Applications > Create** with the source **Platform Integration**, the required **Integration Project Name** dropdown has nothing to select, and the page reports an error. +- A repository shows CONNECTION STATUS `Failed`, and removing it returns `Unable to disconnect repository: Not Found`, while every Argo CD and GitOps Pod is `Running`. +- An Application applied as YAML does not appear in Argo CD. Recreating it in the UI fails with `Unable to create application: application spec for is invalid: InvalidSpecError: application repo /helm-charts is not permitted in project ''`. + +Related articles: "Argo CD components fail: Dex image pull when air-gapped, Redis timeouts in an ambient mesh, or dev shm full on ACP". + +## Root Cause + +### Cause 1: the AppProject is synchronized from a platform project and its cluster resource allow list is reset + +An AppProject with the same name as a platform project is synchronized automatically from the platform project by gitops-controller. It carries the label `cpaas.io/project: ` and the annotation `gitops.app.cpaas.io/last-applied-configuration`. On every reconcile, the gitops-controller log prints `Reconcile project "/"`, "The sourceRepos and destinations after merged", and `"Updating project" project=""`, and applies a patch that contains `"spec":{"clusterResourceWhitelist":null}`. UI saves (the `PUT .../api/v1/projects/` request returns `200 OK`), `kubectl edit`, and `kubectl patch` all revert. This is by design: it prevents a project from creating cluster-scoped resources and escalating privileges. A platform-synchronized AppProject does not support changes to its cluster resource allow list. Projects created by hand in Argo CD accept changes, and the `default` AppProject allows group `'*'` kind `'*'`, destinations `'*'`, and sourceRepos `'*'`. + +### Cause 2: the API component behind the GitOps pages is not running + +The GitOps application creation form loads, but the **Integration Project Name** list stays empty because the API component behind the GitOps pages has not started, for example because of insufficient resources. Once its resources are adjusted, it starts and the page works again. + +### Cause 3: the business cluster that the repository targets has no Argo CD CRDs + +All GitOps components on the cluster that runs Argo CD are `Running` (`argocd-gitops-application-controller`, `argocd-gitops-repo-server`, `argocd-gitops-server`, `argocd-gitops-redis-ha-*`, `gitops-api`, `gitops-controller`, `argocd-operator-controller-manager`). The business cluster that the repository and its applications target has no Argo CD CRD, because the Argo CD operator is not installed there. Installing the Argo CD operator on the business cluster creates the CRDs. + +### Cause 4: the Application `repoURL` lacks the `oci://` prefix that the project uses + +The Application exists in the cluster, but Argo CD rejects its spec, so it does not appear in the UI. The AppProject lists the Helm chart repository as `oci:///helm-charts`, while the Application uses `repoURL: /helm-charts` without the scheme. Because the address in the Application differs from the one registered for the project, Argo CD reports the repository as not permitted, even when the project's **SOURCE REPOSITORIES** is `*`. + +## Resolution + +### Cause 1: the AppProject is synchronized from a platform project and its cluster resource allow list is reset + +1. Create a project by hand in the Argo CD UI, with a name that is not a platform project name, and add to its **CLUSTER RESOURCE ALLOW LIST** only the cluster-scoped kinds that the Application needs (for the example above: group `operator.alauda.io`, kind `ResourceSyncer`). The allow list of a manually created project is saved and kept. + +2. Move the Application to that project. Moving it to the `default` AppProject also makes the sync work, as a quick workaround. + + > **Warning:** Moving an Application to another AppProject changes which destinations, source repositories, and resource kinds it may manage; `default` is permissive. Precondition: confirm that the target project allows only what this Application needs. Rollback: patch `spec.project` back to the original value. + + ```bash + kubectl -n argocd patch applications.argoproj.io --type=merge \ + -p='{"spec":{"project":""}}' + ``` + +### Cause 2: the API component behind the GitOps pages is not running + +1. Find the Pod and the reason it does not run (Diagnostic Steps, step 3). + +2. If it is `Pending` with `Insufficient cpu` or `Insufficient memory`, free capacity on the nodes or lower its requests; if it is `OOMKilled`, raise its memory limit. Change the resources in the object that owns the workload. Direct edits to Deployments and StatefulSets that the operator manages are reverted. For the Argo CD components, the upstream `ArgoCD` resource has a `resources` field per component (for example `spec.controller.resources`, `spec.repo.resources`, `spec.server.resources`); for the operator itself, see "Configure Argo CD Operator container resources via its CSV on ACP". + + > **Warning:** Changing the resources rolls the component's Pods; the GitOps pages are unavailable until the new Pod is Ready. Precondition: only when the API Pod is not Running because of resources and the nodes have capacity for the new requests. Record the current values. Rollback: set the previous values again. + + ```bash + kubectl -n edit argocd + ``` + +3. The component starts, and the **Integration Project Name** dropdown is populated again. + +### Cause 3: the business cluster that the repository targets has no Argo CD CRDs + +On the business cluster that the applications deploy to, install the Argo CD operator from OperatorHub, as on the cluster that runs Argo CD. Then confirm that the `argoproj.io` CRDs exist (Diagnostic Steps, step 4) and check the repository connection again in the Argo CD UI. + +### Cause 4: the Application `repoURL` lacks the `oci://` prefix that the project uses + +Set `spec.source.repoURL` of the Application to exactly the address registered in the project, including `oci://`: + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: + namespace: +spec: + project: + source: + repoURL: oci:///helm-charts + chart: + targetRevision: + destination: + name: + namespace: +``` + +```bash +kubectl apply -f .yaml +kubectl -n get applications.argoproj.io -o jsonpath='{.status.conditions}{"\n"}' +``` + +The `InvalidSpecError` condition is gone and the Application appears in the UI. + +## Diagnostic Steps + +1. Identify the instance and the component state: + + ```bash + kubectl get argocd -A + kubectl -n get pods + ``` + +2. For a sync that is not permitted, check whether the AppProject is platform-synchronized and whether gitops-controller resets it: + + ```bash + kubectl -n argocd get appproject -o jsonpath='{.spec.clusterResourceWhitelist}{"\n"}{.metadata.labels.cpaas\.io/project}{"\n"}' + kubectl -n argocd get appproject -o yaml | grep -c 'gitops.app.cpaas.io/last-applied-configuration' + kubectl get pods -A | grep gitops-controller + kubectl -n logs --tail=500 | grep -E 'Reconcile project|Updating project|clusterResourceWhitelist' + ``` + + A project named after a platform project, with the `cpaas.io/project` label and the `last-applied-configuration` annotation, an empty whitelist that returns after each edit, and `"Updating project"` lines with `"clusterResourceWhitelist":null`, is Cause 1. + +3. For an empty **Integration Project Name** list, find the API Pod and why it is not running: + + ```bash + kubectl get pods -A | grep -E 'argocd-api|gitops-api' + kubectl -n describe pod | grep -iE 'insufficient|oomkilled|reason' + ``` + + A `Pending` Pod with `Insufficient cpu` or `Insufficient memory`, or a container that ends `OOMKilled`, is Cause 2. + +4. For `Unable to disconnect repository: Not Found`, check the Argo CD CRDs on the business cluster that the applications target: + + ```bash + kubectl get crd | grep argoproj.io + ``` + + No output on the business cluster is Cause 3. + +5. For an Application that exists but does not appear in the UI, compare its `repoURL` with the project and the registered repositories: + + ```bash + kubectl -n get applications.argoproj.io -o jsonpath='{.spec.source.repoURL}{"\n"}{.status.conditions}{"\n"}' + kubectl -n get appproject -o jsonpath='{.spec.sourceRepos}{"\n"}' + kubectl -n get secret -l argocd.argoproj.io/secret-type=repository -o jsonpath='{range .items[*]}{.data.url}{"\n"}{end}' | while read u; do echo "$u" | base64 -d; echo; done + ``` + + `InvalidSpecError ... is not permitted in project` with a `repoURL` that differs from the registered repository only by the `oci://` prefix is Cause 4. diff --git a/docs/en/solutions/Argo_CD_components_fail_Dex_image_pull_when_air-gapped_Redis_timeouts_in_an_ambient_mesh_or_dev_shm_full_on_ACP.md b/docs/en/solutions/Argo_CD_components_fail_Dex_image_pull_when_air-gapped_Redis_timeouts_in_an_ambient_mesh_or_dev_shm_full_on_ACP.md new file mode 100644 index 000000000..b8bd9e610 --- /dev/null +++ b/docs/en/solutions/Argo_CD_components_fail_Dex_image_pull_when_air-gapped_Redis_timeouts_in_an_ambient_mesh_or_dev_shm_full_on_ACP.md @@ -0,0 +1,144 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Argo CD components fail: Dex image pull when air-gapped, Redis timeouts in an ambient mesh, or dev shm full on ACP + +## Issue + +Argo CD on ACP is reconciled by `argocd-operator` from an `ArgoCD` resource (`argoproj.io/v1beta1`); direct edits to the Deployments and StatefulSets it owns are reverted. One of these is reported: + +- In an air-gapped cluster, `argocd-dex-server` cannot pull `ghcr.io/dexidp/dex@sha256:...`, and SSO (for example through a Keycloak OIDC connector) fails with `i/o timeout`. +- After the Dex image is mirrored, the Dex Pod still fails with `failed to pull and unpack image "/3rdparty/dex:2.43.0": failed to resolve reference ... x509: certificate signed by unknown authority`, or the `argocd-dex-server` Deployment returns to the `ghcr.io` image after it was edited. +- Every Argo CD component fails to reach Redis with `connection reset by peer`, `EOF`, `context deadline exceeded`, or `i/o timeout`; `application-controller` logs `Failed to save cluster info: i/o timeout` and `Failed to cache app resources: i/o timeout`; SSO is broken too. +- The Argo CD UI shows `failed to write manifest: write /dev/shm/...: no space left on device`. + +Related articles: "Argo CD Application sync not permitted, repository Not Found, InvalidSpecError, or empty GitOps project list on ACP". + +## Root Cause + +### Cause 1: the Dex image is referenced from `ghcr.io`, or the mirrored image is set in the wrong place or with the wrong tag + +An instance with `spec.sso.provider: dex` references the external image `ghcr.io/dexidp/dex@sha256:b08a58c9731c693b8db02154d7afda798e1888dc76db30d34c4a0d0b8a26d913` (Dex `v2.43.0`, linux/amd64). Without internet access no Dex Pod runs. `argocd-server` is always started with the `--dex-server` flag, so a running Dex Pod is required. Two mistakes keep Dex failing after the image is mirrored: + +- **Tag without `v`:** the image is pushed as `3rdparty/dex:v2.43.0`, but the Pod references `3rdparty/dex:2.43.0`. The pull fails with `failed to resolve reference ... x509: certificate signed by unknown authority`. Although the message names a certificate, correcting the tag to `v2.43.0` lets the Pod start. +- **Image edited in the Deployment:** the operator reconciles the `argocd-dex-server` Deployment back to the `ghcr.io` image. The image must be set in the Dex section of the `ArgoCD` resource (`spec.sso.dex.image` and `spec.sso.dex.version`). + +### Cause 2: the Argo CD namespace is in an Istio ambient mesh with STRICT mTLS + +The Argo CD namespace carries the label `istio.io/dataplane-mode=ambient`, and a global `PeerAuthentication` in `istio-system` sets mTLS to `STRICT`. ztunnel enforces mTLS on the node, without sidecars, and silently blocks the plain TCP traffic between Argo CD components. This surfaces as timeouts across Redis, Dex, and repo-server. + +The Redis TLS setup looks like the cause but is not. The operator starts Redis with TLS when a Redis TLS Secret exists and configures components with `--redis-use-tls`; the resource has no option to disable Redis TLS, and `spec.redis.disableTLSVerification` only skips certificate verification. Deleting `argocd-repo-server-tls`, the Argo CD server Deployment, or the Redis StatefulSet to force reconfiguration leaves Redis stuck on `MountVolume.SetUp failed for volume "redis-tls" : secret "-redis-tls" not found` and does not address the mesh. + +### Cause 3: `/dev/shm` of the repo server is too small + +The repo server writes generated manifests to `/dev/shm`. The default shared memory is too small for the manifests of the instance. + +## Resolution + +### Cause 1: the Dex image is referenced from `ghcr.io`, or the mirrored image is set in the wrong place or with the wrong tag + +1. On a master node of the global cluster, load the image and push it to the platform registry. Use `nerdctl` there, because Docker is not running on the masters (for example on ACP 4.2.1). The loaded image has no name (``), so tag it by image ID: + + ```bash + nerdctl -n k8s.io load -i dex.tar + nerdctl -n k8s.io images | grep '' + cat /etc/kubernetes/registry/auth.yaml # platform registry credentials + nerdctl -n k8s.io login + nerdctl -n k8s.io tag /3rdparty/dex:v2.43.0 + nerdctl -n k8s.io push /3rdparty/dex:v2.43.0 + ``` + +2. Set the image in the `ArgoCD` resource, not in the Deployment. The tag must be exactly the pushed tag, `v2.43.0`: + + ```yaml + spec: + sso: + provider: dex + dex: + image: /3rdparty/dex + version: v2.43.0 + ``` + + In the instance form these are the **Image** and **Version** fields of the Dex section. + + ```bash + kubectl -n edit argocd + ``` + +3. The Dex Pod starts and SSO works. + +### Cause 2: the Argo CD namespace is in an Istio ambient mesh with STRICT mTLS + +1. Take the Argo CD namespace out of the ambient mesh. + + > **Warning:** Removing the label takes every Pod in the namespace out of the Istio ambient mesh: traffic to and from them is no longer mTLS-protected by ztunnel, and mesh policies stop applying to them. Precondition: only when the label and the STRICT `PeerAuthentication` are confirmed (Diagnostic Steps, step 2) and no workload in the namespace depends on the mesh. Rollback: `kubectl label ns istio.io/dataplane-mode=ambient`. + + ```bash + kubectl label ns istio.io/dataplane-mode- + ``` + +2. Keep Argo CD in a namespace without the ambient label; the operator does not support opting individual Argo CD Pods out of the mesh. + +3. If Redis TLS Secrets or Argo CD workloads were deleted during troubleshooting, let the operator recreate them and check that the Redis Pod mounts its TLS volume. + +### Cause 3: `/dev/shm` of the repo server is too small + +1. Mount a memory-backed `emptyDir` at `/dev/shm` for the repo server through the `ArgoCD` resource, leaving the rest of the configuration unchanged. + + > **Warning:** The change rolls the repo server; manifest generation and syncs pause until the new Pod is Ready. An `emptyDir` with `medium: Memory` counts against the Pod's memory, so a large `sizeLimit` can push the repo server into OOM. Precondition: memory headroom in the repo server limit. Rollback: remove the volume and the mount again. + + ```yaml + spec: + repo: + volumeMounts: + - mountPath: /dev/shm + name: shm-volume + volumes: + - name: shm-volume + emptyDir: + medium: Memory + sizeLimit: 1Gi # 512Mi or 1Gi recommended + ``` + + ```bash + kubectl -n edit argocd + ``` + +2. `df -h /dev/shm` in the `argocd-repo-server` container shows a `tmpfs` mount with free space, and the error does not return. + +## Diagnostic Steps + +1. For SSO failures, check the Dex image and pull events: + + ```bash + kubectl -n get deploy -dex-server -o jsonpath='{.spec.template.spec.containers[*].image}{"\n"}' + kubectl -n get events --field-selector reason=Failed | grep -i dex + kubectl -n get argocd -o jsonpath='{.spec.sso.dex.image}:{.spec.sso.dex.version}{"\n"}' + ``` + + An image that still starts with `ghcr.io` is Cause 1. For an `x509: certificate signed by unknown authority` pull error on the mirrored image, first compare the referenced tag with the pushed tag (`v2.43.0`). + +2. For timeouts across components, check the mesh before touching Redis TLS: + + ```bash + kubectl get ns --show-labels + kubectl get peerauthentication -A + kubectl -n logs -application-controller-0 --tail=50 | grep -i timeout + ``` + + `istio.io/dataplane-mode=ambient` on the namespace plus a `STRICT` `PeerAuthentication` is Cause 2. + +3. For the manifest write error, check the repo server mount: + + ```bash + kubectl -n exec deploy/-repo-server -c argocd-repo-server -- df -h /dev/shm + ``` + + A full `/dev/shm` (`Use%` near `100%`) is Cause 3. diff --git a/docs/en/solutions/DevOps_Node.js_or_apt_downloads_fail_gnutls_handshake_failed_npm_install_hangs_or_the_Nexus_connection_is_reset.md b/docs/en/solutions/DevOps_Node.js_or_apt_downloads_fail_gnutls_handshake_failed_npm_install_hangs_or_the_Nexus_connection_is_reset.md new file mode 100644 index 000000000..27b831ac2 --- /dev/null +++ b/docs/en/solutions/DevOps_Node.js_or_apt_downloads_fail_gnutls_handshake_failed_npm_install_hangs_or_the_Nexus_connection_is_reset.md @@ -0,0 +1,152 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.0" + - "4.3" +--- + +# DevOps Node.js or apt downloads fail: gnutls_handshake failed, npm install hangs, or the Nexus connection is reset + +## Issue + +A Node.js stage, or a Maven stage that installs operating system packages, fails in a DevOps pipeline while downloading dependencies. One of these symptoms is shown: + +- npm cannot download a front-end dependency that lives on GitHub. The stage dies while running `git`, and the log contains `gnutls_handshake() failed: Error in the pull function.` and `error code: 128`. A workstation using the same Nexus npm proxy still installs. +- The Node.js stage hangs while pulling dependencies until the stage times out, although the same dependencies can be pulled from outside the cluster. The log stops after a line such as `npm warn deprecated wrench@1.3.9: wrench.js is deprecated! ...`. +- `apt update` in a build command fails with `Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)` and `The repository 'http://archive.ubuntu.com/ubuntu jammy InRelease' is not signed.` +- A custom build image whose Yarn registry points at a private Nexus npm repository over HTTPS keeps timing out on package pulls. + +Related articles: "Maven stage misses Nexus: mirrorOf stays central after mavenMirrorURL, or 501 HTTPS Required after generate-settings" covers Maven mirror settings. + +## Root Cause + +### Cause 1: `gnutls_handshake() failed`: the git subprocess that npm starts does not use npm's proxy + +npm starts `/usr/bin/git ls-remote -h -t https://github.com//.git` for dependencies declared as GitHub URLs (directly, or inside a parent `package.json`), and git exits `128` during the TLS handshake. The stage has an npm proxy and a Nexus npm repository, but those settings never reach git, because the git subprocess does not load npm's proxy configuration. The cluster has no direct path to GitHub, so the Git URLs fail. The Node.js dependency cache is not involved. + +### Cause 2: npm or `apt` downloads time out because the build Pod cannot reach the external network + +The build Pod, or the nodes of the cluster where the build runs, have no connectivity to the external repository. Reachability from a workstation does not matter; the path from inside the cluster does. + +- For `apt update`, the Pod requests the external Ubuntu repositories, cannot reach them, and receives something other than the signed `InRelease` file, which produces the `NOSPLIT` error for `jammy`, `jammy-updates`, and `jammy-backports`. +- For npm, the registry may redirect tarball downloads to an external host, or a dependency is fetched from GitHub. A deprecation warning such as `wrench@1.3.9` is not the error: it is the last line printed before npm tries to fetch a dependency that it cannot reach. Unlike Cause 1, no `git ls-remote` or `gnutls_handshake()` line is printed. + +### Cause 3: `curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104` between the nodes and the private Nexus + +Package pulls from the private Nexus npm registry time out. A `curl` from a cluster node to one tarball in that repository negotiates TLS and verifies the certificate (`SSL certificate verify ok.`), and right after the GET is sent it fails with `OpenSSL SSL_read: Connection reset by peer, errno 104`, `Failed receiving HTTP2 data`, `OpenSSL SSL_write: Broken pipe, errno 32`, and finally `curl: (56)`. The timeouts are a network fault on the path between the nodes and Nexus. Unlike Cause 2, the target is the site's own Nexus, and TCP and TLS succeed before the reset. Changing the Yarn registry, the lock file, or the build image does not change a reset on this path. + +## Resolution + +### Cause 1: `gnutls_handshake() failed`: the git subprocess that npm starts does not use npm's proxy + +Apply this only when the stage log shows `/usr/bin/git ls-remote` to a GitHub URL together with `gnutls_handshake() failed: Error in the pull function.` + +1. Put the proxy in the environment of that stage's shell, before `npm install`. Registry-only npm configuration can stay as it is; it is not sufficient for the git child process. Replace `192.0.2.10:3128` with the real proxy, and keep the in-cluster Nexus Service in `NO_PROXY`. + + > **Warning:** The proxy then applies to the rest of that shell, including fetches that are not GitHub. Precondition: only when the handshake error above is in the log and the proxy is one the site already trusts for GitHub. Rollback: remove the export lines. + + ```bash + export http_proxy="http://192.0.2.10:3128" + export https_proxy="http://192.0.2.10:3128" + export HTTP_PROXY="http://192.0.2.10:3128" + export HTTPS_PROXY="http://192.0.2.10:3128" + export no_proxy="nexus-nxrm-ha.devops-system.svc.cluster.local,localhost,127.0.0.1" + export NO_PROXY="nexus-nxrm-ha.devops-system.svc.cluster.local,localhost,127.0.0.1" + npm install + ``` + +2. Alternatively, if the manifest pins the dependency to a GitHub URL, switch it to a version that the npm repository already serves, so that git is no longer invoked. + + > **Warning:** This edits application source and selects a different artifact. Precondition: only when the version exists in the npm repository selected for the pipeline. Rollback: revert the manifest in Git. + + ```json + { + "dependencies": { + "": "" + } + } + ``` + +3. As a temporary bridge while the Git URL is removed or the proxy export is added, the installed `node_modules` tree can be copied to a PVC that is mounted on the next runs, with the npm prefix set to it: + + > **Warning:** Later builds reuse whatever is on that volume, including packages that were replaced upstream. Precondition: only when Cause 1 is confirmed and a trusted `node_modules` tree already exists. Rollback: detach the PVC. + + ```bash + npm config set prefix "/workspace/tmp/node_modules" + ``` + +### Cause 2: npm or `apt` downloads time out because the build Pod cannot reach the external network + +Apply this only when the checks in Diagnostic Steps fail from a Pod in the build namespace and the log has no `git ls-remote` (that is Cause 1). + +1. Run the front-end build in the pipeline's Node.js stage, not in an ad hoc shell step or a Dockerfile. +2. Give the cluster network access to the external dependency repository, including the host that the registry redirects tarballs to, and GitHub when a dependency is fetched from there. If that path cannot be opened, switch the dependency source to a mirror that the cluster can reach, such as an npm proxy repository inside the cluster (for example the Nexus npm proxy selected as the dependency repository). If the site uses an HTTP forward proxy, export it in the stage as shown for Cause 1. +3. Stop running `apt` at build time. Install the packages into the Java build image on a machine that has network access, and use that image for the Maven stage. For a build that needed the Chinese locale: + + ```dockerfile + FROM + USER root + RUN apt-get update \ + && apt-get install -y --no-install-recommends language-pack-zh-hans \ + && rm -rf /var/lib/apt/lists/* + ENV LC_ALL=zh_CN.UTF-8 + USER + ``` + + Read `` with `docker inspect --format '{{.Config.User}}' `; if it prints nothing, the image runs as root and the last `USER` line can be dropped. Check the image before pushing it: `docker run --rm --entrypoint /bin/sh -c 'locale -a | grep -i zh_CN'` should list `zh_CN.utf8`. Push the image, point the build node or task at it, and remove the `apt` commands from the stage, so that the command is only the Maven part, for example `mvn -DskipTests package -B -T 4`. + +### Cause 3: `curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104` between the nodes and the private Nexus + +1. Give the `curl` output from Diagnostic Steps to the team that owns the network path between the nodes and Nexus (firewalls, load balancers, or proxies in front of Nexus), and have the reset removed there. +2. After the network change, run the same `curl` again. A complete download (HTTP `200` and the expected size) is the condition for running the build again. + +## Diagnostic Steps + +Cause 1, in the Node.js stage log: + +```text +/usr/bin/git ls-remote -h -t https://github.com//.git +gnutls_handshake() failed: Error in the pull function. +error code: 128 +``` + +In a debug Pod that uses the build image, check whether git and npm disagree: + +```bash +npm config get registry +npm config get proxy +command -v git && git --version +``` + +Cause 2, from a Pod in the build namespace that uses the build image: + +```bash +kubectl run netcheck -n --rm -it --restart=Never --image= -- sh +# inside the shell +npm config get registry +npm view dist.tarball +curl -sS -o /dev/null -w '%{http_code}\n' --max-time 10 "$(npm config get registry)" +curl -sS -o /dev/null -w '%{http_code}\n' --max-time 10 +curl -sS -o /dev/null -w '%{http_code}\n' --max-time 10 https://github.com +``` + +A timeout here, while the same URL works from a workstation, is Cause 2. A tarball URL on a host other than the registry is a redirect that also needs a network path. For `apt`, list the sources and test one of them: + +```bash +cat /etc/apt/sources.list /etc/apt/sources.list.d/* 2>/dev/null | grep -v '^#' +curl -sS --max-time 10 http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease | head -3 +``` + +A signed file starts with `-----BEGIN PGP SIGNED MESSAGE-----`. A timeout, or an HTML page from a gateway, matches the `NOSPLIT` error. + +Cause 3: read the registry URL in the build image with `yarn config get registry` (it ends with `/`). Then, from a cluster node, request one tarball that the build fails to pull, over HTTP/2 and over HTTP/1.1: + +```bash +curl -v -o /dev/null /-/-.tgz +curl -v --http1.1 -o /dev/null /-/-.tgz +``` + +`SSL certificate verify ok.`, then `Connection reset by peer, errno 104` and `curl: (56)`, is Cause 3. The second command shows whether the reset also happens over HTTP/1.1; give both outputs to the network team. diff --git a/docs/en/solutions/Dex_login_options_CORS_SSO_user_identity_and_the_Dex_deployment_layout_on_ACP.md b/docs/en/solutions/Dex_login_options_CORS_SSO_user_identity_and_the_Dex_deployment_layout_on_ACP.md new file mode 100644 index 000000000..ebda2bbf6 --- /dev/null +++ b/docs/en/solutions/Dex_login_options_CORS_SSO_user_identity_and_the_Dex_deployment_layout_on_ACP.md @@ -0,0 +1,157 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.10" + - "3.10.2" + - "3.16" + - "4.0" + - "4.2" + - "4.2.1" + - "4.3" + - "4.3.2" + - "4.3.3" +--- + +# Dex login options, CORS, SSO user identity, and the Dex deployment layout on ACP + +## Issue + +One of these is reported about the platform Dex: + +- After SSO authentication completes and the user selects the platform, the browser stops on the ACP login page and the user has to click the login button again. Or a security drill needs username/password login turned off without removing OIDC. +- A security scan reports that the Dex SSO API reflects any `Origin`: `POST /dex/api/v1/authorize/sso` with `Origin: https://` returns `Access-Control-Allow-Origin: https://` and `Access-Control-Allow-Credentials: true`. +- A custom tool reached through the platform SSO redirect can see that a user arrived, but cannot tell which user it was. +- After a fresh install, or after the global cluster moves from 3.16 to 4.0, there is no `dex` Deployment in `cpaas-system`. + +The Dex configuration is held in the ConfigMap `dex-configmap` in `cpaas-system` on the global cluster. Related articles: "OIDC clients and pages are sent to the wrong Dex issuer address on ACP" and "Dex update fails on ACP 3.10 or account login returns 401 after the Dex permanent token changes". + +## Root Cause + +### Cause 1: SSO users must click the login button again because local login is still enabled + +Local user authentication (the Dex password database) and OIDC authentication are both enabled, so the login page offers two methods and waits for a choice. Setting `enablePasswordDB` to `false` in `dex-configmap` turns local user authentication off and leaves OIDC as the only login method, which gives direct access. + +### Cause 2: the Dex SSO API reflects any Origin because `allowedOrigins` is not set + +`allowedOrigins` is not configured under the `web` section of the Dex configuration in `dex-configmap`, so the request `Origin` is echoed back. Adding `allowedOrigins` with the platform's real access URLs restricts CORS to those addresses. + +### Cause 3: a custom tool behind SSO cannot see the current user + +A tool that calls Harbor (or a similar API) after the platform SSO redirect has no check of the user who clicked it, so one user can act on another user's project. With the UI redirect kept, the page carries `id_token` only when the frontend is started with `--enable-redirect-with-token=true`. Integrating the tool as an OIDC client of the platform avoids depending on that redirect flag. + +### Cause 4: ACP 4.x has no standalone Dex Deployment + +On 3.x, Dex runs as its own Pod. From 4.0 onward, Dex runs inside the `apollo` Deployment in `cpaas-system`. Nothing failed to install, and no separate Dex chart or plugin is required. OIDC is still configured in `dex-configmap`. + +## Resolution + +### Cause 1: SSO users must click the login button again because local login is still enabled + +1. On the global cluster, find the key in the Dex configuration: + + ```bash + kubectl -n cpaas-system get cm dex-configmap -o yaml | grep -n enablePasswordDB + ``` + +2. Set `enablePasswordDB: false`. + + > **Warning:** `enablePasswordDB: false` removes local account/password login from the platform login page, including local accounts such as `admin`; only OIDC users can log in afterwards. Restarting Dex briefly interrupts logins. Precondition: only when OIDC login already works end to end and at least one platform administrator can log in through OIDC. Back up the ConfigMap first. Rollback: `kubectl apply -f /tmp/dex-configmap.bak.yaml` and restart Dex again. + + ```bash + kubectl -n cpaas-system get cm dex-configmap -o yaml > /tmp/dex-configmap.bak.yaml + kubectl -n cpaas-system edit cm dex-configmap + ``` + + Set `enablePasswordDB: false` in the Dex configuration held by the ConfigMap, keeping the existing indentation. + +3. Upstream Dex reads its configuration at startup, so if the login page still offers local login after the edit, restart the workload that runs Dex (`apollo` on 4.x, the `dex` Deployment on 3.x): + + ```bash + kubectl -n cpaas-system get deploy | grep -Ei 'apollo|dex' + kubectl -n cpaas-system rollout restart deploy/ + kubectl -n cpaas-system rollout status deploy/ + ``` + +4. Check `enablePasswordDB` again after each platform upgrade. + +Editing `dex-configmap` creates an `rpch` object that stores the override. To turn password login back on, delete that `rpch`. + +> **Warning:** Deleting the `rpch` removes the stored `dex-configmap` override, including `enablePasswordDB: false`. Precondition: only when password login should return and this `rpch` is the object the edit created. Rollback: edit `dex-configmap` again and set `enablePasswordDB: false`, which creates a new `rpch`. + +```bash +kubectl get rpch -A +kubectl get rpch -n -o yaml > /tmp/rpch.bak.yaml +kubectl delete rpch -n +``` + +When a drill requires the whole login page to be unreachable, scaling the `frontend` Deployment to zero does that; the page then returns an HTTP 500-class error, and OIDC login is down as well. + +> **Warning:** `frontend` at 0 replicas removes the console until it is scaled back; business workloads keep running. Precondition: only when the drill requires the login page to be unreachable and a maintenance window covers the outage. Record the current replica count. Rollback: scale back to the recorded count. + +```bash +kubectl -n cpaas-system get deploy frontend -o jsonpath='{.spec.replicas}{"\n"}' +kubectl scale deployment -n cpaas-system frontend --replicas=0 +# restore: +kubectl scale deployment -n cpaas-system frontend --replicas= +``` + +### Cause 2: the Dex SSO API reflects any Origin because `allowedOrigins` is not set + +1. Reproduce the finding: + + ```bash + curl -k -sS -o /dev/null -D - -X POST -H 'Origin: https://untrusted.example.com' \ + https:///dex/api/v1/authorize/sso | grep -i '^access-control-allow' + ``` + +2. Add `allowedOrigins` under the existing `web` section of the Dex configuration. + + > **Warning:** Once `allowedOrigins` is set, browsers on origins that are not in the list lose cross-origin access to Dex; an access address left out of the list can break login pages served from that address. Precondition: only when the response above echoes the untrusted Origin and you have the complete list of platform access addresses, including additional access addresses. Back up the ConfigMap first. Rollback: `kubectl apply -f /tmp/dex-configmap.bak.yaml` and restart Dex. + + ```bash + kubectl -n cpaas-system get cm dex-configmap -o yaml > /tmp/dex-configmap.bak.yaml + kubectl -n cpaas-system edit cm dex-configmap + ``` + + ```yaml + web: + allowedOrigins: + - https:// + - https:// + ``` + + Keep the other keys already under `web`. + +3. Restart Dex as in Cause 1 if the response does not change, then run the `curl` again. The untrusted Origin no longer appears in `Access-Control-Allow-Origin`. + +### Cause 3: a custom tool behind SSO cannot see the current user + +Register the tool as an OIDC client of the platform, with the platform as the OIDC server, and have the tool check the user from the token it receives. Alternatively, start the frontend with `--enable-redirect-with-token=true` so that the redirect carries `id_token`, which contains part of the logged-in user's identity, and use it for the tool's own permission check. The OIDC client integration is the recommended option because it does not depend on the redirect flag. + +### Cause 4: ACP 4.x has no standalone Dex Deployment + +Read the embedded configuration and follow the product OIDC documentation: + +```bash +kubectl get deploy -n cpaas-system apollo -o yaml | grep dex +kubectl get configmap dex-configmap -n cpaas-system -o yaml +``` + +The absence of a `dex` Deployment on 4.0 and later is expected. On 3.x, a missing Dex Deployment is not this cause. + +## Diagnostic Steps + +```bash +kubectl -n cpaas-system get cm dex-configmap -o yaml | grep -n -E 'enablePasswordDB|allowedOrigins|issuer' +kubectl -n cpaas-system get deploy | grep -Ei 'apollo|dex|frontend' +kubectl get rpch -A +``` + +- After SSO the ACP login page asks for another click, and `enablePasswordDB` is not `false`: Cause 1. +- Password login must be disabled for a drill, or an `rpch` was left behind by a `dex-configmap` edit: Cause 1. +- `Access-Control-Allow-Origin` echoes an arbitrary Origin on `/dex/api/v1/authorize/sso` with credentials allowed, and `web.allowedOrigins` is absent: Cause 2. +- A custom UI behind SSO has no user identity: Cause 3. +- No `dex` Deployment in `cpaas-system` on 4.x while `apollo` is running: Cause 4. diff --git a/docs/en/solutions/Dex_update_fails_on_ACP_3.10_or_account_login_returns_401_after_the_Dex_permanent_token_changes.md b/docs/en/solutions/Dex_update_fails_on_ACP_3.10_or_account_login_returns_401_after_the_Dex_permanent_token_changes.md new file mode 100644 index 000000000..5dc59536b --- /dev/null +++ b/docs/en/solutions/Dex_update_fails_on_ACP_3.10_or_account_login_returns_401_after_the_Dex_permanent_token_changes.md @@ -0,0 +1,92 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.10" + - "3.10.2" + - "3.16" + - "4.0" + - "4.2" + - "4.2.1" + - "4.3" + - "4.3.2" + - "4.3.3" +--- + +# Dex update fails on ACP 3.10 or account login returns 401 after the Dex permanent token changes + +## Issue + +One of these is reported: + +- On ACP 3.10, the dex component update fails, which raises the question of whether the platform upgrade can continue. +- Right after someone updates the permanent token in the Dex ConfigMap (`dex-configmap` in `cpaas-system`), account login returns HTTP 401. Restarting the container runtime on the node does not clear it. + +Related articles: "OIDC clients and pages are sent to the wrong Dex issuer address on ACP" and "Dex login options, CORS, SSO user identity, and the Dex deployment layout on ACP". + +## Root Cause + +### Cause 1: on 3.10 the dex update fails while the `ui-*` ConfigMaps keep `helm.sh/resource-policy` + +The ConfigMaps `ui-logos`, `ui-login-logo`, and `ui-login-bg` in `cpaas-system` carry the annotation `helm.sh/resource-policy`, and the dex update fails. This is a defect in that version, fixed in 3.10.2. Removing the annotation lets the update proceed on a 3.10 build that does not have the fix. + +### Cause 2: login returns 401 after the permanent token in `dex-configmap` changes + +Account login returns 401 immediately after the permanent token in `dex-configmap` is updated. Restarting the container runtime on the node does not clear the 401; restarting `kube-apiserver` on all clusters does. No deeper root cause was established, so treat the restart as the change that clears the symptom, not as proof of a token-format problem. + +## Resolution + +### Cause 1: on 3.10 the dex update fails while the `ui-*` ConfigMaps keep `helm.sh/resource-policy` + +Prefer upgrading to 3.10.2, which contains the fix. On a 3.10 cluster that must finish the update first: + +1. Confirm the annotation on the three ConfigMaps: + + ```bash + kubectl get cm -n cpaas-system ui-logos ui-login-logo ui-login-bg \ + -o custom-columns=NAME:.metadata.name,ANN:.metadata.annotations + ``` + +2. Remove the annotation. + + > **Warning:** Without `helm.sh/resource-policy`, the next dex Helm release may replace `ui-logos`, `ui-login-logo`, and `ui-login-bg`, so logos and login backgrounds that exist only in those objects can be replaced. Precondition: only when the dex update on 3.10 fails and the three objects show `helm.sh/resource-policy`. Back them up first. Rollback: `kubectl apply -f /tmp/dex-ui-cm.bak.yaml` restores the ConfigMaps and the annotation. + + ```bash + kubectl -n cpaas-system get cm ui-logos ui-login-logo ui-login-bg -o yaml > /tmp/dex-ui-cm.bak.yaml + kubectl annotate -n cpaas-system cm ui-logos ui-login-logo ui-login-bg helm.sh/resource-policy- + ``` + +3. Retry the dex update. + +### Cause 2: login returns 401 after the permanent token in `dex-configmap` changes + +Apply this only when the 401 started right after the permanent token inside `cpaas-system/dex-configmap` was changed, and a container-runtime restart on the node left the 401 in place. + +1. Restart `kube-apiserver` on each control-plane node, one node at a time, by moving its static Pod manifest out and back. + + > **Warning:** Restarting `kube-apiserver` drops API requests on that control-plane node until the static Pod is Ready again. Do one control-plane node at a time and wait for it before the next one. Precondition: only when this cause is confirmed and the other control-plane nodes are Ready. Rollback: if the node stays down with the manifest moved aside, move `/tmp/kube-apiserver.yaml` back to `/etc/kubernetes/manifests/kube-apiserver.yaml`. + + ```bash + sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml + sleep 20 + sudo mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml + kubectl get nodes + kubectl -n kube-system get pods -o wide | grep kube-apiserver + ``` + +2. Repeat on the remaining control-plane nodes and on every other cluster that shows the 401. + +3. Log in again. Account login succeeds after `kube-apiserver` has been restarted on all clusters. + +## Diagnostic Steps + +```bash +kubectl get cm -n cpaas-system ui-logos ui-login-logo ui-login-bg \ + -o custom-columns=NAME:.metadata.name,ANN:.metadata.annotations +kubectl -n cpaas-system get cm dex-configmap -o yaml > /tmp/dex-configmap.now.yaml +``` + +- The dex update fails on 3.10, and the three `ui-*` ConfigMaps have `helm.sh/resource-policy`: Cause 1. +- HTTP 401 on account login starts immediately after the permanent token in `dex-configmap` is updated: Cause 2. diff --git a/docs/en/solutions/GitLab_instance_upgrade_fails_breaks_after_a_Data_Services_move_reverts_GitlabOfficial_edits_or_cannot_be_deleted.md b/docs/en/solutions/GitLab_instance_upgrade_fails_breaks_after_a_Data_Services_move_reverts_GitlabOfficial_edits_or_cannot_be_deleted.md new file mode 100644 index 000000000..e8be7317b --- /dev/null +++ b/docs/en/solutions/GitLab_instance_upgrade_fails_breaks_after_a_Data_Services_move_reverts_GitlabOfficial_edits_or_cannot_be_deleted.md @@ -0,0 +1,235 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "3.14" + - "3.20" + - "4.2" + - "4.3" +--- + +# GitLab instance upgrade fails, breaks after a Data Services move, reverts GitlabOfficial edits, or cannot be deleted + +## Issue + +A GitLab instance managed by the GitLab Operator shows one of these problems when it is upgraded or reconfigured: + +- After a platform upgrade (for example 4.2.1 to 4.3.3), the GitLab Operator is upgraded and **Upgrade** is clicked on the instance; the instance shows Deploy Failed with: + + ```text + failed upgrade (pre-upgrade hooks failed: resource Job//-gitlab-official-upgrade-check not ready. status: Failed, message: Job Failed. failed: 3/1) and failed rollback: original object ConfigMap with the name "gitlab-webservice-puma" not found + ``` + +- During the GitLab 14 upgrade procedure, the task-runner Deployment stays `Pending` after it is patched with the backup PVC, with no logs and no events; later the backup fails with permission errors. +- After GitLab's Redis and PostgreSQL are moved to Data Services management, some GitLab services fail to start, and the PostgreSQL instance in Data Services shows status Unknown. +- Edits to the GitLab custom resource deployed by the operator keep being reverted. +- Deleting the GitLab instance has no effect (seen on 3.14.1), and a DevOps Tools Operator deployment fails at the same time. +- During the GitLab upgrade the `task-runner` image is missing, and the default registry does not have it either. + +## Root Cause + +### Cause 1: an intermediate GitLab version was skipped + +GitLab supports upgrades only through specific intermediate versions, and the operator enforces this in the `upgrade-check` pre-upgrade hook. When only the latest GitLab Operator is applied (for example operator 17.11.1 before and 18.8.1 after), the instance tries to go straight from GitLab 18.2.8 (chart `9.2.8-beta.1.g17490f65`) to 18.8.11 (chart `9.8.7-beta.1`), and the hook log shows: + +```text +MIN_VERSION=18.5 +CHART_MIN_VERSION=9.5 +GITLAB_VERSION=18.8.11 +It seems you are upgrading the GitLab Helm Chart from 9.2.8-beta.1.g17490f65 (GitLab 18.2.8) to 9.8.7-beta.1 (GitLab 18.8.11). +It is required to upgrade to the latest 9.5.x version first before proceeding. +``` + +followed by `exit 1`. The upgrade path for the operator and the instance is, in order, using the latest patch version of each minor release: `17.8.z` → `17.11.z` → `18.2.z` (requires the PostgreSQL 16 upgrade) → `18.5.z` → `18.8.z`. The `failed rollback ... ConfigMap "gitlab-webservice-puma" not found` part of the message is the Helm rollback after the failed hook; the hook failure is the cause. + +### Cause 2: the GitLab 14 task-runner is `Pending` because `backup-pvc` is not bound + +The upgrade procedure from GitLab 14.0.12 patches the `-task-runner` Deployment with the annotation `skip-sync: "true"` and replaces `/spec/template/spec/volumes/1` with a volume `task-runner-tmp` that uses the PVC `backup-pvc`. When no PV is bound to `backup-pvc`, the new Pod (image `gitlab-task-runner-ce:14.0.12`) stays `Pending` with `Node: `, condition `PodScheduled False`, and `Events: `. After a PV is bound, the backup can still fail when the backup directory was created and given permissions on one node but the PV, PVC, and Pod end up on another node. + +### Cause 3: the Data Services PostgreSQL shows Unknown after the move + +The PostgreSQL instance in Data Services (for example `-database`, PostgreSQL 12) shows status Unknown with: + +```text +Please manually upgrade the PostgreSQL version.The current major version 12 is less than the minimal required version 14 +``` + +The exact cause of the failing GitLab services was not confirmed. Restarting the PostgreSQL database and then the failing GitLab Pods brings the components back. The message itself states that the managed instance is below the minimal major version that Data Services requires (14), so the instance still needs a major version upgrade. + +### Cause 4: edits were written under `status` instead of `spec` + +`status` is written by the operator and reflects the observed state, so any value placed there is overwritten. Parameters set under `spec` (for `GitlabOfficial`, `spec.helmValues`) take effect and stay. + +### Cause 5: deleting the instance has no effect while the GitLab CSV is still installed + +Deleting the GitLab instance directly does nothing. After the ClusterServiceVersion (CSV) that belongs to GitLab is deleted, the instance uninstall proceeds normally. Why the remaining CSV blocks the uninstall was not determined. + +### Cause 6: the `task-runner` image tag does not resolve to the platform registry + +When the instance is updated according to the upgrade documentation with `task-runner: enabled: "true"` and no explicit image, the resulting task-runner Deployment does not reference the image in the platform's built-in registry, so the image cannot be pulled. Setting the actual image tag restores normal operation. + +## Resolution + +### Cause 1: an intermediate GitLab version was skipped + +Download and apply each operator version on the path, and upgrade the instance after each one, before moving to the next: `17.8.z` → `17.11.z` → `18.2.z` → `18.5.z` → `18.8.z`. Perform the PostgreSQL 16 upgrade before the instance goes to 18.2.z. After installing the intermediate operator, check the instance status (Diagnostic Steps, step 1) before continuing. + +> **Warning:** Each instance upgrade runs database migrations and restarts GitLab; GitLab is unavailable during each step. Precondition: a full GitLab backup (database together with `-rails-secret`) and a maintenance window that covers all steps. Rollback: restore the backup into an instance of the original version. + +### Cause 2: the GitLab 14 task-runner is `Pending` because `backup-pvc` is not bound + +1. Create a PV that matches the PVC and pins it to one node. A `local` PV with `nodeAffinity` makes the scheduler place the task-runner Pod on the node that holds the backup directory: + + ```yaml + apiVersion: v1 + kind: PersistentVolume + metadata: + name: gitlab-backup-pv + spec: + capacity: + storage: # at least the request of backup-pvc + accessModes: + - ReadWriteOnce # same as backup-pvc + persistentVolumeReclaimPolicy: Retain + storageClassName: # same as backup-pvc; "" if it has none + claimRef: + namespace: + name: backup-pvc + local: + path: + nodeAffinity: + required: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/hostname + operator: In + values: + - + ``` + + ```bash + kubectl apply -f gitlab-backup-pv.yaml + kubectl -n get pvc backup-pvc + kubectl -n get pod -o wide | grep task-runner + ``` + +2. On ``, create `` and give write access to the user that the task-runner container runs as (Diagnostic Steps, step 3 shows `runAsUser` and `fsGroup`). The backup succeeds once the permissions are granted on the node where the task-runner Pod actually runs. + +### Cause 3: the Data Services PostgreSQL shows Unknown after the move + +> **Warning:** Restarting PostgreSQL interrupts GitLab and every other client of the instance until it is back. Precondition: only when the Data Services PostgreSQL shows Unknown with the version message and GitLab Pods fail, and a recent backup exists. A restart changes no data. + +```bash +kubectl -n get pods | grep -database +kubectl -n delete pod -database-0 +kubectl -n get pods -w +kubectl -n get pods | grep -vE 'Running|Completed' +kubectl -n delete pod +``` + +The GitLab components return to normal. Then plan the PostgreSQL major version upgrade that the status message asks for, following the Data Services documentation. + +### Cause 4: edits were written under `status` instead of `spec` + +```bash +kubectl -n edit gitlabofficial +``` + +Put the parameters under `spec.helmValues` (for example `spec.helmValues.gitlab.webservice.resources` or `spec.helmValues.global.hosts`), save, and confirm that the operator applies them. Remove anything you added under `status`. + +### Cause 5: deleting the instance has no effect while the GitLab CSV is still installed + +> **Warning:** Deleting the CSV removes the GitLab operator Deployment that OLM created from it; no GitLab instance in the cluster is reconciled until the operator is installed again. Precondition: only when deleting the instance has had no effect (Diagnostic Steps, step 6) and you intend to remove GitLab. Save the CSV YAML first. Rollback: install the GitLab operator again from OperatorHub. + +```bash +kubectl get csv -A | grep -i gitlab +kubectl -n get csv -o yaml > -before.yaml +kubectl -n delete csv +``` + +Then delete the instance again and confirm that its resources are removed. + +### Cause 6: the `task-runner` image tag does not resolve to the platform registry + +Read the registry and path that the other GitLab components of the instance use, and give the task-runner the same registry with the explicit tag of the task-runner image that matches the GitLab version (for GitLab 14.0.12, `gitlab-task-runner-ce:14.0.12`). In the GitLab chart values the fields are `gitlab.task-runner.image.repository` and `gitlab.task-runner.image.tag`; in the `GitlabOfficial` resource they sit under `spec.helmValues`: + +```yaml +spec: + helmValues: + gitlab: + task-runner: + enabled: "true" + image: + repository: //gitlab-task-runner-ce + tag: +``` + +```bash +kubectl -n edit gitlabofficial +``` + +Set the image on the instance rather than on the Deployment; a change made only in the Deployment can be reverted by the operator (Cause 4). + +## Diagnostic Steps + +1. Read the instance status and the failed hook Job: + + ```bash + kubectl -n get gitlabofficial -o jsonpath='{.status}{"\n"}' + kubectl -n get jobs | grep upgrade-check + kubectl -n logs job/-gitlab-official-upgrade-check --tail=50 + ``` + + `It is required to upgrade to the latest 9.5.x version first` (or another `MIN_VERSION`) is Cause 1. + +2. Compare the operator version with the instance version: + + ```bash + kubectl get csv -A | grep -i gitlab + kubectl -n get gitlabofficial -o yaml | grep -iE 'version' | head + ``` + +3. For a `Pending` task-runner, check the Pod and the PVC: + + ```bash + kubectl -n describe pod | grep -E 'Status:|Node:|PodScheduled|ClaimName' + kubectl -n get pvc backup-pvc + kubectl -n get deploy -task-runner -o jsonpath='{.spec.template.spec.securityContext}{"\n"}' + ``` + + A PVC in `Pending` with no bound volume is Cause 2. After the PV is bound, compare the node of the Pod with the node where the directory permissions were set. + +4. For failures after a Data Services move, check the PostgreSQL instance status: + + ```bash + kubectl get postgresqls.acid.zalan.do -A | grep + kubectl -n get postgresqls.acid.zalan.do -database -o jsonpath='{.status}{"\n"}' + ``` + + `The current major version 12 is less than the minimal required version 14` is Cause 3. A replica whose log shows `system ID is invalid`, or an operator status of `could not sync cluster ... no switchover candidate found`, is a PostgreSQL replica problem that is not covered here. + +5. For reverted edits, check where the changed values are: + + ```bash + kubectl -n get gitlabofficial -o yaml | grep -n -E '^spec:|^status:' + ``` + + Values that you added below the `status:` line are Cause 4. + +6. For an instance whose deletion has no effect, check whether the GitLab operator CSV is still present: + + ```bash + kubectl get csv -A | grep -i gitlab + kubectl get gitlabofficial -A + ``` + +7. For a task-runner that cannot pull its image, compare its image with the other components: + + ```bash + kubectl -n get deploy -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.template.spec.containers[*].image}{"\n"}{end}' + kubectl -n get events --field-selector reason=Failed | grep -i task-runner + ``` + + A task-runner image that is not in the platform registry used by the other components, or has no tag, is Cause 6. diff --git a/docs/en/solutions/GitLab_or_Argo_CD_login_through_the_platform_Dex_fails_host_required_certificate_verify_invalid_client_LDAP_groups.md b/docs/en/solutions/GitLab_or_Argo_CD_login_through_the_platform_Dex_fails_host_required_certificate_verify_invalid_client_LDAP_groups.md new file mode 100644 index 000000000..2e7fcb9d1 --- /dev/null +++ b/docs/en/solutions/GitLab_or_Argo_CD_login_through_the_platform_Dex_fails_host_required_certificate_verify_invalid_client_LDAP_groups.md @@ -0,0 +1,131 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps + - Alauda Container Platform +ProductsVersion: + - "4.1" + - "4.1.3" + - "4.2" + - "4.3" + - "4.3.2" +--- + +# GitLab or Argo CD login through the platform Dex fails: host required, certificate verify, invalid_client, LDAP groups + +## Issue + +Login to GitLab or Argo CD through the platform Dex fails. One of these symptoms is shown: + +- The GitLab sign-in page reports `Could not authenticate you from Dex because "'host' required.".` +- The GitLab sign-in page reports `Could not authenticate you from Dex because "Ssl connect returned=1 errno=0 peeraddr=:443 state=error: certificate verify failed (self-signed certificate in certificate chain)".` +- Argo CD opened from the platform fails `LOG IN VIA OIDC` with `failed to get token: oauth2: "invalid_client" "Invalid client credentials."`, while the Argo CD admin password still works; **Create GitOps Application** is a blank page, and the browser console shows `ERROR TypeError: Cannot read properties of undefined (reading 'ecmp')`. +- LDAP users do not receive the Argo CD groups that local users with the same kind of platform role receive. + +Related articles: "Harbor SSO through the platform Dex fails: wrong issuer, OAuth2Client name, or Unregistered redirect_uri console-portal" and "Toolchain SSO on a domain name after an IP-based install, and kube-apiserver OIDC for an external UI on ACP". + +## Root Cause + +### Cause 1: the GitLab `OAuth2Client` redirect has no scheme + +For an Alauda Build of GitLab instance (installed from OperatorHub) whose SSO provider is the platform Dex (`key: provider`, `secret: gitlab-sso-config`, `syncProfileFromProvider: - dex`), `'host' required` is returned when the redirect URLs on the Dex `OAuth2Client` for GitLab do not start with `https://`. + +### Cause 2: GitLab does not trust the platform (Dex) certificate + +`certificate verify failed (self-signed certificate in certificate chain)` is returned when the platform certificate chain is self-signed and GitLab has no CA to trust it. Creating a Secret with that CA and referencing it under `certificates.customCAs` on the GitLab instance fixes it. After Causes 1 and 2 are fixed, the Dex login reaches the GitLab profile page. + +### Cause 3: GitOps 4.3.0 on ACP 4.1.3 uses a Dex client that does not exist + +GitOps plugin 4.3.0 installed on the global cluster of ACP 4.1.3 configures Argo CD with `oidcConfig` `issuer: /dex`, `clientID: alauda-argocd-gitops`, and `clientSecret: $argocd-gitops-sso:clientSecret`. Dex (`dex-configmap` in `cpaas-system`) has no client `alauda-argocd-gitops`, and the Secret `argocd-gitops-sso` decodes to a different secret, so token exchange fails with `invalid_client`. ACP 4.1.3 is not adapted for GitOps 4.3.0. GitOps plugin 4.2.0 with argocd-operator 4.2.0 makes SSO work but leaves the blank page on 4.1.3; GitOps plugin 4.1.8 with argocd-operator 4.2.0 clears both. + +### Cause 4: Argo CD does not map LDAP usernames that contain uppercase letters + +Argo CD uses the platform Dex, and the platform uses LDAP. A local user with a platform role gets a group such as `platform-admin-system` or a project administrator group in Argo CD. An LDAP user with the same kind of role does not get the group when the LDAP username contains uppercase characters, because the sync ignores them, so group-based rights (for example creating a repository) cannot be granted. LDAP usernames in lowercase get the groups. There is no setting that stores such a user in lowercase. + +## Resolution + +### Cause 1: the GitLab `OAuth2Client` redirect has no scheme + +1. Back up the clients and read their redirects: + + ```bash + kubectl get oauth2clients.dex.coreos.com -A -o yaml > /tmp/oauth2clients.bak.yaml + kubectl get oauth2clients.dex.coreos.com -A -o yaml | grep -n -i redirect + ``` + +2. Prefix each GitLab redirect with `https://` (`https:///`), keeping the host and path. + + > **Warning:** Changing redirect URLs breaks callbacks that still use the old string. Precondition: only when the error is `'host' required` and a redirect value has no `https://` prefix. Rollback: `kubectl apply -f /tmp/oauth2clients.bak.yaml`. + + ```bash + kubectl edit oauth2clients.dex.coreos.com -n cpaas-system + ``` + +### Cause 2: GitLab does not trust the platform (Dex) certificate + +1. Export the chain that the platform presents and keep the CA certificate (the last certificate in the chain): + + ```bash + openssl s_client -connect :443 -showcerts /dev/null \ + | awk '/BEGIN CERTIFICATE/{n++} n{print > ("/tmp/dex-chain-" n ".pem")}' + ls /tmp/dex-chain-*.pem + openssl x509 -in /tmp/dex-chain-.pem -noout -subject -issuer + ``` + +2. Create the Secret in the GitLab instance namespace. The upstream GitLab chart expects one PEM CA per key, with a `.crt` key name: + + ```bash + kubectl create secret generic dex-tls -n --from-file=dex-ca.crt=/tmp/dex-chain-.pem + ``` + +3. Reference the Secret on the GitLab instance. + + > **Warning:** Editing the GitLab instance makes the operator roll the GitLab Pods, so GitLab is unavailable during the rollout. Precondition: only when the sign-in error is `certificate verify failed` against the platform address and the instance has no `customCAs` entry for that CA. Rollback: `kubectl apply -f /tmp/gitlab-instance.bak.yaml`. + + ```bash + kubectl api-resources | grep -i gitlab + kubectl get -n -o yaml > /tmp/gitlab-instance.bak.yaml + kubectl edit -n + ``` + + ```yaml + certificates: + customCAs: + - secret: dex-tls + ``` + +4. Retry the SSO login after the GitLab Pods are Ready. + +### Cause 3: GitOps 4.3.0 on ACP 4.1.3 uses a Dex client that does not exist + +1. Confirm the mismatch: + + ```bash + kubectl -n cpaas-system get cm dex-configmap -o yaml | grep -n -E 'staticClients|id:' -A2 + kubectl get oauth2clients.dex.coreos.com -A | grep -i argocd + kubectl get secret -A | grep argocd-gitops-sso + kubectl get argocd -A -o yaml | grep -n -A12 oidcConfig + ``` + + No Dex client with id `alauda-argocd-gitops` while Argo CD asks for it is this cause. + +2. Deploy GitOps plugin 4.1.8 with argocd-operator 4.2.0 on ACP 4.1.3. That combination makes both SSO and the GitOps application pages work. Hand-editing the Argo CD client id and secret can make SSO pass, but does not fix a one-click install on this ACP version. + +### Cause 4: Argo CD does not map LDAP usernames that contain uppercase letters + +Use lowercase LDAP usernames for accounts that must receive Argo CD groups from platform roles. The groups that local users already show (`platform-admin-system` and the project administrator groups) confirm that the mapping works for lowercase names. + +## Diagnostic Steps + +```bash +curl -k -sS "https:///dex/.well-known/openid-configuration" +kubectl get oauth2clients.dex.coreos.com -n cpaas-system -o custom-columns=NAME:.metadata.name,ID:.id +kubectl get deploy -n cpaas-system | grep -Ei 'argo|gitops|apollo' +kubectl get csv -A | grep -Ei 'gitops|argocd' +``` + +- `'host' required` on the GitLab sign-in page, and a GitLab redirect without `https://`: Cause 1. +- `certificate verify failed (self-signed certificate in certificate chain)` on the GitLab sign-in page: Cause 2. +- `invalid_client` on Argo CD OIDC login, or `Cannot read properties of undefined (reading 'ecmp')` on Create GitOps Application, on ACP 4.1.3: Cause 3. +- LDAP users missing Argo CD groups that local users have: Cause 4. diff --git a/docs/en/solutions/GitLab_toolchain_integration_fails_token-gitlab_not_found_Internal_error_projects_API_500_or_an_extra_bot_project.md b/docs/en/solutions/GitLab_toolchain_integration_fails_token-gitlab_not_found_Internal_error_projects_API_500_or_an_extra_bot_project.md new file mode 100644 index 000000000..a925bc5ce --- /dev/null +++ b/docs/en/solutions/GitLab_toolchain_integration_fails_token-gitlab_not_found_Internal_error_projects_API_500_or_an_extra_bot_project.md @@ -0,0 +1,142 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.2.2" + - "4.3" +--- + +# GitLab toolchain integration fails: token-gitlab not found, Internal error, projects API 500, or an extra bot project + +## Issue + +A GitLab toolchain integration of DevOps v3 is configured and does not work in pipelines. One of these symptoms is shown: + +- Creating a CI pipeline says the GitLab token cannot be found, or updating a build returns `Secret "token-gitlab" not found` (`"reason": "NotFound"`, `"code": 404`), while the token page does not show the token and the toolchain integration page shows no error. +- A GitLab 17 (or later) integration reports an error and cannot be used. +- After the GitLab credential is updated, requests intermittently return `Internal error`. +- `GET /integrations/v1alpha1/namespaces//integrations//projects/` returns `500 (Internal Server Error)` on every call. +- After GitLab projects are assigned, the code management list shows an extra entry named like `project__bot_` that does not exist as a project in GitLab. + +These integrations are served by katanomi (`katanomi-api` and `katanomi-plugin` in `cpaas-system`). Each platform project is expected to have a namespace of the same name in the cluster. Related articles: "Harbor toolchain integration fails: images not selectable, katanomi-plugin timeout, or push asks for credentials" covers the Harbor integration. + +## Root Cause + +### Cause 1: `Secret "token-gitlab" not found` because the integration uses the external GitLab address + +The GitLab integration was registered with an external network address, while the platform's internal authentication goes through the internal network address. The front end and the back end validate against different addresses, so pipeline creation cannot find the token and pipeline execution fails. The error body is `"status": "Failure"`, `"reason": "NotFound"`, `"details": {"name": "token-gitlab", "kind": "Secret"}`, `"code": 404`. + +### Cause 2: GitLab 17 or later integration error because incremental job logging needs object storage + +GitLab 17 and later turn on incremental job logging, which uploads archived job logs incrementally to object storage. When the GitLab instance has no object storage configured, the DevOps integration fails. Turning off **Turn on incremental logging** in GitLab fixes the integration. + +### Cause 3: intermittent `Internal error` after the GitLab credential is updated + +After the credential is updated, requests sometimes work and sometimes return `Internal error`. The error comes from GitLab, not from the platform. The old GitLab credentials left behind by the update cause it; deleting them recovers the integration. + +### Cause 4: the integration projects API returns 500 because node DNS is broken + +The DNS server configured on the node is malfunctioning. katanomi cannot resolve the GitLab address, and the projects API returns 500 on every call. + +### Cause 5: an extra `project__bot_` entry with subtype `GitUser` + +When an external GitLab is integrated with root permissions and projects are assigned, the code management list of the project can show an extra entry named like `project_62_bot_`, labelled as platform-assigned, whose code repository tab is empty. GitLab creates a bot user of that name when a project access token is created for project ID ``. The platform syncs GitLab user (personal) namespaces as well as groups, so the bot user's personal namespace is synced as a tool project with `subType: GitUser`. The entry does not affect usage. + +## Resolution + +### Cause 1: `Secret "token-gitlab" not found` because the integration uses the external GitLab address + +1. Compare the integration's address with the address the cluster uses to reach GitLab: + + ```bash + kubectl get pods -A | grep katanomi-api + kubectl -n exec deploy/ -- getent hosts + ``` + +2. Create a new GitLab integration with the internal network address, assign it to the project, and point the pipeline at it. Keep the external-address integration until the new one works, then remove it. + +3. Create the pipeline again. Pipeline creation finds the token and the pipeline runs. + +### Cause 2: GitLab 17 or later integration error because incremental job logging needs object storage + +> **Warning:** Turning the option off changes how every project on that GitLab stores CI job logs. Precondition: only when GitLab is 17 or later, has no object storage configured, and the integration fails. Note the current settings first. Rollback: turn the option on again after object storage is configured. + +1. In GitLab, open **Admin Area > Settings > CI/CD** (`/admin/application_settings/ci_cd`) and expand **Job logs** (the last section). +2. Under **Incremental logging configuration**, clear **Turn on incremental logging** and save. GitLab answers "Application settings saved successfully". +3. Retry the integration. Its status in DevOps becomes healthy. + +### Cause 3: intermittent `Internal error` after the GitLab credential is updated + +1. Test the token that GitLab sees. `` is the token saved in the integration's Secret, base64-decoded: + + ```bash + curl -v "https:///api/v4/version" -H "PRIVATE-TOKEN: " + ``` + + A JSON version answer means the token and connectivity are fine. + +2. Delete the old GitLab credentials that the update left behind. + + > **Warning:** Deleting a credential that a pipeline still references breaks that pipeline. Precondition: only when `Internal error` started after a credential update and the new credential works with the `curl` check. Confirm which pipelines use the old credential first. Rollback: recreate the credential with its previous token. + +3. The error no longer returns. + +### Cause 4: the integration projects API returns 500 because node DNS is broken + +1. Resolve the GitLab host from the node and from a Pod: + + ```bash + # on the node + getent hosts + cat /etc/resolv.conf + # from a Pod + kubectl -n exec deploy/ -- getent hosts + ``` + + A failed or wrong answer is this cause. + +2. Repair the DNS server that the node uses. As a temporary workaround, add a hosts entry for GitLab on the node (`192.0.2.20` stands for the real GitLab address). + + > **Warning:** A hosts entry pins the name on that node only and goes stale if GitLab moves. Precondition: only when the node's DNS server fails for the GitLab name. Back up `/etc/hosts`. Rollback: remove the line. Remove the entry after the DNS service is repaired. + + ```bash + cp /etc/hosts /tmp/hosts.bak + echo '192.0.2.20 ' >> /etc/hosts + ``` + +3. The projects API returns data instead of 500. + +### Cause 5: an extra `project__bot_` entry with subtype `GitUser` + +Confirm that the entry is a GitLab bot user, not a project: + +```bash +kubectl get projects.meta.katanomi.dev -A -o yaml | grep -n -B4 -A8 '_bot_' +``` + +The object shows `subType: GitUser`. In GitLab, open `/projects/` for the ID in the name; it redirects to the project whose access token created the bot. No action is required. The DevOps v4 GitLab integration, available from ACP 4.1, is the alternative when this listing is not wanted. + +## Diagnostic Steps + +```bash +kubectl get pods -A | grep -E 'katanomi-api|katanomi-plugin' +kubectl -n logs deploy/ --tail=100 | grep -Ei 'gitlab|timeout|500' +``` + +Capture the failing request in the browser developer tools; the path and the status code identify the cause. + +- Token not found at pipeline creation, and the integration uses an external address: Cause 1. +- GitLab 17 or later without object storage: Cause 2. +- Intermittent `Internal error` after a credential update: Cause 3. +- `500` on the integration projects API, and DNS resolution of the GitLab host fails on the node: Cause 4. +- An extra `project__bot_` tool project with subtype `GitUser`: Cause 5. + +When katanomi logs domain resolution timeouts for a platform Service name such as `katanomi-plugin.cpaas-system`, rather than for the GitLab host, the fault is cluster DNS, not GitLab. Check the node `/etc/resolv.conf` for a `search` line with site domains, and the CoreDNS log for that Service name being forwarded upstream: + +```bash +kubectl get ep -A | grep katanomi +kubectl -n kube-system logs -l k8s-app=kube-dns --tail=200 | grep -iE 'katanomi|i/o timeout' +cat /etc/resolv.conf +``` diff --git a/docs/en/solutions/Harbor_SSO_through_the_platform_Dex_fails_wrong_issuer_OAuth2Client_name_or_Unregistered_redirect_uri_console-portal.md b/docs/en/solutions/Harbor_SSO_through_the_platform_Dex_fails_wrong_issuer_OAuth2Client_name_or_Unregistered_redirect_uri_console-portal.md new file mode 100644 index 000000000..aa1e4c511 --- /dev/null +++ b/docs/en/solutions/Harbor_SSO_through_the_platform_Dex_fails_wrong_issuer_OAuth2Client_name_or_Unregistered_redirect_uri_console-portal.md @@ -0,0 +1,163 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps + - Alauda Container Platform +ProductsVersion: + - "4.1" + - "4.1.7" + - "4.2" + - "4.3" + - "4.3.2" +--- + +# Harbor SSO through the platform Dex fails: wrong issuer, OAuth2Client name, or Unregistered redirect_uri console-portal + +## Issue + +Single sign-on from Alauda Build of Harbor through the platform Dex does not work. One of these symptoms is shown: + +- SSO was turned on in the Harbor instance and has no effect, or an already installed Harbor should be switched to SSO. +- The Harbor login page shows the OIDC button, and clicking it opens the ACP login page even though the user is already logged in to ACP, so Harbor single sign-on never completes. +- After a platform upgrade, clicking the OIDC login on Harbor ends on `The authorization code is invalid`. In the browser developer tools the Dex `authorize` request returns HTTP 400 with: + + ```text + status: Failure + reason: "ErrReqInvalid" + message: "Unregistered redirect_uri (\"https:///console-portal/\")" + ``` + + The `authorize` URL carries `client_id=alauda-auth` (the ACP console client) and a callback under `/console-portal/` on the Harbor host, not the Harbor client id and `/c/oidc/callback`. + +Related articles: "GitLab or Argo CD login through the platform Dex fails: host required, certificate verify, invalid_client, LDAP groups" and "OIDC clients and pages are sent to the wrong Dex issuer address on ACP". + +## Root Cause + +### Cause 1: Harbor's issuer is the Harbor address plus `/dex` + +The issuer in the Harbor SSO settings was set to the Harbor access address plus `/dex`. The issuer must be the platform access address plus `/dex`, whether or not platform SSO is otherwise used by end users. + +### Cause 2: OIDC is not enabled on an already installed Harbor + +A Harbor that is already deployed can be switched to SSO without a reinstall. OIDC is enabled with `helmValues.oidc.enable: true` on the instance, and the remaining OIDC fields follow the Configure SSO section of the Alauda Build of Harbor deployment documentation, with the issuer from Cause 1. + +### Cause 3: the Harbor `OAuth2Client` name is not derived from its id, or its redirect is mis-indented + +The Dex `OAuth2Client` for Harbor (`apiVersion: dex.coreos.com/v1`, namespace `cpaas-system`) must have a `metadata.name` computed from its `id`. In upstream Dex Kubernetes storage, the object name is the lowercase base32 encoding of the client id bytes followed by the FNV-64 hash of empty input, with `=` padding removed; Dex finds a client by computing that name from `client_id`. For `id: harbor-dex` the name is `nbqxeytpoiwwizlyzpzjzzeeeirsk`. A manifest copied from another client without recomputing the name (for example the name for `gitlab-dex`, `m5uxi3dbmiwwizlyzpzjzzeeeirsk`) is not found by Dex. A `redirectURIs` entry for `/c/oidc/callback` that is indented incorrectly is not a valid list item either. Either error sends the browser to the ACP login page instead of completing Harbor SSO. + +### Cause 4: the Harbor domain reaches the platform access address, so the login runs the console flow + +The login is not running Harbor's OIDC flow at all. Requests for the Harbor domain reach the platform access address instead of the Ingress that serves Harbor, so the platform answers them, and the browser goes into the ACP console login with `client_id=alauda-auth` and a `/console-portal/` callback on the Harbor host. That redirect is not registered on `alauda-auth`, so Dex rejects it with `Unregistered redirect_uri`. The Harbor OIDC values, the Harbor `OAuth2Client` (with `redirectURIs` of `/c/oidc/callback`), and the Harbor Ingress can all be correct. Correcting the address that the Harbor domain is forwarded to restores Harbor SSO; nothing in Harbor, its `OAuth2Client`, or `alauda-auth` needs to change. + +## Resolution + +### Cause 1: Harbor's issuer is the Harbor address plus `/dex` + +Set the Harbor SSO issuer to `https:///dex`, not the Harbor address with `/dex` appended. Save the instance and retry the OIDC login. + +### Cause 2: OIDC is not enabled on an already installed Harbor + +Enable OIDC on the Harbor instance values: + +```yaml +helmValues: + oidc: + enable: true +``` + +Then complete the issuer, client id, client secret, and redirect URLs from the Configure SSO section of the Alauda Build of Harbor documentation, using the issuer from Cause 1. + +### Cause 3: the Harbor `OAuth2Client` name is not derived from its id, or its redirect is mis-indented + +1. List the clients with their ids and compute the name the Harbor client must have: + + ```bash + kubectl get oauth2clients.dex.coreos.com -n cpaas-system -o custom-columns=NAME:.metadata.name,ID:.id + python3 -c 'import base64,struct,sys; print(base64.b32encode(sys.argv[1].encode()+struct.pack(">Q",0xcbf29ce484222325)).decode().lower().rstrip("="))' harbor-dex + ``` + + Use the id that your Harbor configuration sends as its client id. This cause applies only when the `NAME` of the Harbor client differs from the computed value, or its `redirectURIs` is not a proper YAML list. + +2. Build the corrected object from the existing one: + + ```bash + kubectl get oauth2clients.dex.coreos.com -n cpaas-system -o yaml > /tmp/harbor-oauth2client.bak.yaml + cp /tmp/harbor-oauth2client.bak.yaml /tmp/harbor-oauth2client.yaml + vi /tmp/harbor-oauth2client.yaml + ``` + + In the copy, set `metadata.name` to the computed value, delete `metadata.resourceVersion`, `metadata.uid`, and `metadata.creationTimestamp`, and keep `id`, `name`, the secret, and every other field. Write the Harbor callback as a list item: + + ```yaml + redirectURIs: + - https:///c/oidc/callback + ``` + +3. Replace the client. + + > **Warning:** Deleting the old `OAuth2Client` removes that client from Dex; anything that logs in through it fails until the new object exists. Precondition: only when the name check above confirms this cause and the backup is saved. Rollback: `kubectl delete -f /tmp/harbor-oauth2client.yaml`, then apply the backup after removing `resourceVersion` and `uid` from it. + + ```bash + kubectl apply -f /tmp/harbor-oauth2client.yaml + kubectl delete oauth2clients.dex.coreos.com -n cpaas-system + kubectl get oauth2clients.dex.coreos.com -n cpaas-system -o custom-columns=NAME:.metadata.name,ID:.id + ``` + +4. Make Harbor pick up the client again: remove the OIDC section from the Harbor instance, wait until the instance is ready, and add the same OIDC section back. + + > **Warning:** Each save of the Harbor instance makes the operator reconcile Harbor; OIDC login is off while the section is removed, and Harbor can be briefly unavailable. Precondition: only after the corrected `OAuth2Client` exists, with the instance YAML backed up. Rollback: `kubectl apply -f /tmp/harbor-instance.bak.yaml`. + + ```bash + kubectl api-resources | grep -i harbor + kubectl get -n -o yaml > /tmp/harbor-instance.bak.yaml + kubectl edit -n + ``` + +5. Click the OIDC button on the Harbor login page. The Harbor `oidc-onboard` page opens with the dialog "You must create a Harbor username the first time when authenticating via a third party(OIDC)". Set the Harbor username there once. + +### Cause 4: the Harbor domain reaches the platform access address, so the login runs the console flow + +Apply this only when the failing `authorize` request carries `client_id=alauda-auth` and a `redirect_uri` under `/console-portal/` on the Harbor host (see Diagnostic Steps). + +> **Warning:** Changing where the Harbor domain is forwarded moves every Harbor client (browsers, `docker login`, image pulls from nodes) to the new target; a wrong target makes Harbor unreachable. Precondition: only when Diagnostic Steps confirm that the Harbor domain reaches the platform access address and that the Harbor Ingress address serves the Harbor portal. Plan a maintenance window if image pulls depend on Harbor. Record the current value first. Rollback: restore the previous forwarding address. + +1. Change whichever of the DNS record, external load balancer, or hosts entry sends the Harbor domain to the platform access address, so that it points at the Harbor Ingress address (`status.loadBalancer.ingress` of the Harbor Ingress). + +2. Confirm the result and retry Harbor's OIDC button: + + ```bash + getent hosts + curl -k -sS -o /dev/null -w '%{http_code} %{redirect_url}\n' https:/// + ``` + + The request returns the Harbor portal and does not redirect to the ACP login. Harbor SSO works again. + +## Diagnostic Steps + +1. Capture the failing `authorize` request in the browser developer tools and note `client_id` and `redirect_uri`. `client_id=alauda-auth` with `https:///console-portal/` is Cause 4. The Harbor client id with `/c/oidc/callback` points to Causes 1 to 3. + +2. Check the issuer and client settings of the Harbor instance and the Harbor client in Dex: + + ```bash + curl -k -sS "https:///dex/.well-known/openid-configuration" + kubectl api-resources | grep -i harbor + kubectl get -n -o yaml | grep -n -A8 -i oidc + kubectl get oauth2clients.dex.coreos.com -n cpaas-system -o custom-columns=NAME:.metadata.name,ID:.id + ``` + + - The issuer ends in the Harbor host plus `/dex`: Cause 1. No OIDC values at all: Cause 2. + - The Harbor client's name is not the encoding of its id (for `harbor-dex`, `nbqxeytpoiwwizlyzpzjzzeeeirsk`), or its `redirectURIs` is mis-indented: Cause 3. + +3. For Cause 4, compare where the Harbor domain resolves with the address the Harbor Ingress is served on, then compare the two paths without changing anything: + + ```bash + kubectl get ingress -n -o wide + kubectl get ingress -n -o jsonpath='{.status.loadBalancer.ingress}{"\n"}' + getent hosts + curl -k -sS -o /dev/null -w '%{http_code} %{redirect_url}\n' https:/// + curl -k -sS -o /dev/null -w '%{http_code} %{redirect_url}\n' \ + --resolve :443: https:/// + ``` + + The domain resolving to the platform access address instead of the Ingress address, the first request redirecting toward the ACP login, and the second request returning the Harbor portal confirm Cause 4. A successful ping to the domain does not rule it out. diff --git a/docs/en/solutions/Harbor_toolchain_integration_fails_images_not_selectable_katanomi-plugin_timeout_or_push_asks_for_credentials.md b/docs/en/solutions/Harbor_toolchain_integration_fails_images_not_selectable_katanomi-plugin_timeout_or_push_asks_for_credentials.md new file mode 100644 index 000000000..e3c227723 --- /dev/null +++ b/docs/en/solutions/Harbor_toolchain_integration_fails_images_not_selectable_katanomi-plugin_timeout_or_push_asks_for_credentials.md @@ -0,0 +1,93 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.3" +--- + +# Harbor toolchain integration fails: images not selectable, katanomi-plugin timeout, or push asks for credentials + +## Issue + +A Harbor toolchain integration of DevOps v3 is configured and does not work. One of these symptoms is shown: + +- Harbor images cannot be selected from the dropdown when a service is created in one environment (for example production), while another environment (for example test) works. +- The Harbor integration fails with `katanomi-plugin.cpaas-system.svc.cluster.local i/o timeout`. +- A pipeline image push fails saying that Harbor credentials are required, and at the same time the Harbor integration in the toolchain shows an error. + +These integrations are served by katanomi (`katanomi-api` and `katanomi-plugin` in `cpaas-system`). Related articles: "GitLab toolchain integration fails: token-gitlab not found, Internal error, projects API 500, or an extra bot project" covers the GitLab integration; "Image tag cannot be selected in Update Image Version because the integration repositories request returns 500" covers a tag dropdown that fails with `500` and `Client.Timeout`. + +## Root Cause + +### Cause 1: the platform project is not assigned in the Harbor toolchain project + +Image tags appear in the service creation dropdown only after the platform project has been assigned to the Harbor toolchain project. When one environment has the assignment and the other does not, only the first one lists images. + +### Cause 2: the full `svc.cluster.local` name of `katanomi-plugin` does not resolve + +The integration URLs are generated with the `.cluster.local` suffix. On the affected cluster (Kubernetes `v1.33.7`), `http://katanomi-plugin.cpaas-system.svc` and `http://katanomi-plugin.cpaas-system` answered from a Pod, while the names ending in `.svc.cluster` and `.svc.cluster.local` did not resolve. Harbor did not accept a change that drops the suffix. Setting `ndots: 1` in the `dnsConfig` of the `katanomi-api` Deployment restores the Harbor integration as a workaround. Why the full name did not resolve on that cluster was not determined, so cluster DNS still needs investigation. + +### Cause 3: a firewall blocks katanomi from reaching Harbor + +The pipeline push and the toolchain integration fail together, while all Pods on the global cluster are Running. A connectivity test from katanomi to Harbor fails: katanomi cannot reach Harbor at all, because a firewall in front of the Harbor address (for example on the VIP side) blocks that traffic. The push cannot obtain the integration's credentials, so it reports that credentials are required, although the credentials themselves are correct. Removing the matching firewall rules restores both the integration and the push. + +## Resolution + +### Cause 1: the platform project is not assigned in the Harbor toolchain project + +Open the Harbor toolchain integration in the failing environment and assign the platform project to the Harbor project that holds the images. Projects are assigned individually. Then open service creation again. The image and tag dropdowns list the Harbor images. + +### Cause 2: the full `svc.cluster.local` name of `katanomi-plugin` does not resolve + +1. Test the names from a Pod: + + ```bash + kubectl -n exec -- curl -sv -o /dev/null http://katanomi-plugin.cpaas-system.svc + kubectl -n exec -- curl -sv -o /dev/null http://katanomi-plugin.cpaas-system.svc.cluster.local + ``` + + The short name answering and the full name timing out is this cause. + +2. Set `ndots: 1` on `katanomi-api`. + + > **Warning:** Patching `katanomi-api` restarts it. If the katanomi operator renders this Deployment, the patch can be reverted on its next reconcile. Precondition: only when the Harbor integration fails with this exact timeout and the short Service name resolves. Save the Deployment first. Rollback: remove the `dnsConfig` again with the second command below. + + ```bash + kubectl -n cpaas-system get deploy katanomi-api -o yaml > /tmp/katanomi-api.yaml + kubectl -n cpaas-system patch deploy katanomi-api --type='merge' -p '{"spec":{"template":{"spec":{"dnsConfig":{"options":[{"name":"ndots","value":"1"}]}}}}}' + kubectl -n cpaas-system rollout status deploy/katanomi-api + # rollback: + # kubectl -n cpaas-system patch deploy katanomi-api --type=json -p '[{"op":"remove","path":"/spec/template/spec/dnsConfig"}]' + ``` + +3. The Harbor integration completes. This is a workaround; keep investigating why the `.cluster.local` names do not resolve in cluster DNS. + +### Cause 3: a firewall blocks katanomi from reaching Harbor + +1. Check for unhealthy Pods, then test Harbor from inside a katanomi Pod, not from a node: + + ```bash + kubectl get po -A | grep -vi running + kubectl -n cpaas-system exec deploy/katanomi-api -- curl -skv -o /dev/null --max-time 10 https:///api/v2.0/ping + ``` + + If the image has no `curl`, run the same request from another Pod on the same cluster. A timeout or `No route to host` while Harbor itself is up is this cause. A fast HTTP answer means the path is open and the error is elsewhere (Cause 2 for a `katanomi-plugin` timeout). + +2. Trace the path from the cluster to the Harbor address, including any VIP in front of Harbor, and have the network owner remove the rule that drops this traffic. Change only the rules that match this traffic. + +3. The Harbor integration is healthy again, and the image push completes without a credentials error. + +## Diagnostic Steps + +```bash +kubectl get pods -A | grep -E 'katanomi-api|katanomi-plugin' +kubectl -n cpaas-system logs deploy/katanomi-api --tail=100 | grep -Ei 'harbor|timeout|500' +``` + +- Harbor images missing in one environment only, with no error status in the browser: Cause 1. +- `katanomi-plugin.cpaas-system.svc.cluster.local i/o timeout`, and the short Service name resolves: Cause 2. +- The image push asks for Harbor credentials, the Harbor integration shows an error, and katanomi cannot connect to Harbor: Cause 3. +- The tag dropdown request `repositories?itemsPerPage=100&page=1` returns `500` with `Client.Timeout ... while reading body`: not one of these causes; see "Image tag cannot be selected in Update Image Version because the integration repositories request returns 500". +- Harbor integration health-check warnings that appear only during a network change and disappear when the network is stable need no action. diff --git a/docs/en/solutions/Image_build_fails_with_PodSecurity_privileged_a_RUN_step_failing_on_one_node_or_a_buildah_certificate_error.md b/docs/en/solutions/Image_build_fails_with_PodSecurity_privileged_a_RUN_step_failing_on_one_node_or_a_buildah_certificate_error.md new file mode 100644 index 000000000..b5c7c7534 --- /dev/null +++ b/docs/en/solutions/Image_build_fails_with_PodSecurity_privileged_a_RUN_step_failing_on_one_node_or_a_buildah_certificate_error.md @@ -0,0 +1,148 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.0" + - "4.3" +--- + +# Image build fails with PodSecurity privileged, a RUN step failing on one node, or a buildah certificate error + +## Issue + +The image build task of a pipeline fails while the clone and compile tasks pass. One of these symptoms is shown: + +- The BuildKit task Pod is never created, and the TaskRun reports: + + ```text + pods "-build-image-buildkit-v1-pod" is forbidden: violates PodSecurity "baseline:latest": privileged (containers "step-prepare", "step-build", "step-push" must not set securityContext.privileged=true) + ``` + + After the task's privilege setting is turned off, the Pod starts but the `build` step fails with `failed to read dockerfile: failed to mount /tmp/buildkit-mount: [...]: operation not permitted`. +- A Dockerfile `RUN` step, for example `RUN chmod +x ./apps/start.sh && chown -R : /data`, fails with `did not complete successfully: exit code: 1`, but only for build Pods scheduled on one particular node. Restarting that node does not help. +- A buildah build or push to Harbor fails certificate verification, although the registry certificate is issued by a public CA that browsers trust. The pipeline runs when TLS verification is disabled. + +## Root Cause + +### Cause 1: BuildKit needs a privileged container and the namespace enforces `baseline` or `restricted` + +From ACP 4.0 onward, the BuildKit build task requests privileged containers for its steps (`step-prepare`, `step-build`, `step-push`). A namespace that enforces the Pod Security Standard `baseline` rejects `securityContext.privileged=true`, so the Pod is rejected at admission. A namespace that enforces `restricted` rejects the same steps. + +Setting the task's `privilege` field to `false` is not a workaround. The Pod is admitted, but BuildKit then cannot mount its snapshot (`failed to mount /tmp/buildkit-mount: [{Type:bind Source:/var/lib/buildkit/runc-native/snapshots/snapshots/1 Target: Options:[rbind ro]}]: operation not permitted`), even for a simple `FROM` / `COPY` / `CMD` Dockerfile. BuildKit pipelines can run only in a namespace whose Pod Security level allows privileged containers. + +### Cause 2: containerd's `runc` runtime on one node lacks `SystemdCgroup = true` + +The build log on the failing node contains a cgroup error such as: + +```text +stat /sys/fs/cgroup/cpuset/kubepods-burstable-pod.slice:cri-containerd:/buildkit/...: no such file or directory +``` + +kubelet on the node uses `cgroupDriver: systemd`, but `/etc/containerd/config.toml` on that node has no `SystemdCgroup = true` under `plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options` (it may be set only for another runtime, such as `nvidia`). containerd and runc then use the cgroupfs driver for ordinary containers while kubelet uses systemd. The failing node shows flat cgroup paths such as `/sys/fs/cgroup/cpuset/kubepods-burstable-pod.slice:cri-containerd:`, while a healthy node shows the systemd hierarchy `/sys/fs/cgroup//kubepods.slice/...`. With that mismatch, BuildKit's nested container cannot create its sub-cgroup under the cpuset hierarchy, and the `RUN` step fails. Other workloads, such as Maven Pods on the same node, can still complete. + +### Cause 3: the buildah task Pod carries no CA certificates + +The buildah Pod of the Buildah task does not include any CA certificates, so it cannot verify the Harbor registry certificate, even one issued by a public CA. To keep TLS verification on, the CA chain has to be given to the buildah container through the Buildah task's `cert-dir` workspace. + +## Resolution + +### Cause 1: BuildKit needs a privileged container and the namespace enforces `baseline` or `restricted` + +1. Read the namespace's Pod Security labels and confirm that the rejected containers are the BuildKit steps: + + ```bash + kubectl get ns --show-labels | tr ',' '\n' | grep pod-security + kubectl get taskrun -n -o jsonpath='{.status.conditions[0].message}{"\n"}' + ``` + +2. Leave the task's `privilege` field at its default (`true`). If it was changed to `false`, set it back; otherwise the `operation not permitted` mount error stays. + +3. Allow privileged containers in the namespace, from the namespace settings in the console or with the label. + + > **Warning:** The `privileged` level lets every Pod in the namespace run privileged containers, mount host paths, and run as root. Precondition: only when the TaskRun is rejected with `violates PodSecurity ... privileged` for the BuildKit steps and the namespace is used for pipeline runs. Record the current label values first. Rollback: set `pod-security.kubernetes.io/enforce` back to the recorded value. Prefer a namespace dedicated to pipeline runs, so that application namespaces keep `baseline` or `restricted`. + + ```bash + kubectl label ns pod-security.kubernetes.io/enforce=privileged --overwrite + ``` + +4. Run the pipeline again. The BuildKit Pod is created and the `build` step reads the Dockerfile. + +When application namespaces must stay at `restricted`, run the BuildKit pipelines in a separate namespace used only for pipeline runs, label only that namespace `privileged`, and leave the application namespaces unchanged. + +### Cause 2: containerd's `runc` runtime on one node lacks `SystemdCgroup = true` + +Apply this only when all failing builds are on one node and the error is the cgroup `stat ... no such file or directory`. + +1. Compare the failing node with a node where builds pass: + + ```bash + kubectl get pod -A -o wide | grep image-build + # on both nodes + grep -n -A2 -E 'runtimes\.(runc|nvidia)\.options' /etc/containerd/config.toml + grep -i cgroupDriver /var/lib/kubelet/config.yaml + ls /sys/fs/cgroup/cpuset/ | grep -m3 kubepods + ``` + + `cgroupDriver: systemd` on both nodes, `SystemdCgroup = true` under `runtimes.runc.options` only on the healthy node, and flat `kubepods-burstable-pod...slice:cri-containerd:...` directories on the failing node confirm this cause. + +2. Add the section on the failing node so that it matches the healthy node: + + ```toml + [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] + SystemdCgroup = true + ``` + +3. containerd reads `config.toml` at start, so restart containerd. Containers already running on the node were created under the old driver, so drain the node first to have its Pods recreated under the new one. + + > **Warning:** Draining evicts every Pod on the node, and restarting containerd interrupts container operations on that node until it is back. Precondition: only when builds fail on this node only, the cgroup `stat` error is confirmed, and the healthy node's `runc` section has `SystemdCgroup = true`. Back up the file first. Rollback: restore the backup, restart containerd, and uncordon the node. + + ```bash + kubectl drain --ignore-daemonsets --delete-emptydir-data + # on the node + cp /etc/containerd/config.toml /etc/containerd/config.toml.bak + vi /etc/containerd/config.toml + systemctl restart containerd + systemctl is-active containerd + # from a client + kubectl uncordon + ``` + +4. Run the pipeline again on that node. The image-build Pod completes there, and the cgroup paths are under `/sys/fs/cgroup//kubepods.slice/`. + +### Cause 3: the buildah task Pod carries no CA certificates + +1. Get the CA chain that issued the registry certificate (root and intermediate) in PEM form, and store it in the pipeline namespace: + + ```bash + openssl s_client -connect :443 -servername -showcerts /dev/null | grep -E 's:|i:' + kubectl -n create configmap registry-ca --from-file=ca.crt= + ``` + +2. Bind that ConfigMap to the Buildah task's `cert-dir` workspace (in the workspace section of the pipeline, next to `source`). In PipelineRun YAML: + + ```yaml + workspaces: + - name: cert-dir + configMap: + name: registry-ca + ``` + + Use the workspace name that your pipeline maps to the task's `cert-dir` workspace. + +3. Turn TLS verification back on and run the pipeline. The build and the push to Harbor complete with verification enabled. If verification is still refused, compare the issuer printed by `openssl s_client` with the certificates in the ConfigMap; the chain must contain the issuer of the server certificate. + +## Diagnostic Steps + +```bash +kubectl get taskrun -n | tail -10 +kubectl get pod -A -o wide | grep image-build +kubectl logs -n --all-containers --tail=60 +``` + +- TaskRun message `violates PodSecurity "baseline:latest": privileged` (or `restricted:latest` listing `privileged` for the BuildKit steps), or `failed to mount /tmp/buildkit-mount... operation not permitted` after privilege was turned off: Cause 1. +- `RUN` step `exit code: 1` with a cgroup `stat ... no such file or directory`, and every failure on the same node: Cause 2. +- A certificate verification error from buildah to a registry whose certificate is publicly trusted, and the run passes with TLS verification off: Cause 3. + +A TaskRun Pod rejected by `restricted:latest` for Tekton containers such as `prepare`, `place-scripts`, or `step-clone` (not BuildKit steps) is covered in "Pipeline task Pod does not start: connector-csi OOMKilled, controller image pull, Connector placeholder, or PodSecurity". A rejection that names the `katanomi-setting-init` init container is covered in "Tekton pipeline Pod rejected by Pod Security admission because of the katanomi-setting-init init container". diff --git a/docs/en/solutions/Image_tag_cannot_be_selected_in_Update_Image_Version_because_the_integration_repositories_request_returns_500.md b/docs/en/solutions/Image_tag_cannot_be_selected_in_Update_Image_Version_because_the_integration_repositories_request_returns_500.md new file mode 100644 index 000000000..38caab447 --- /dev/null +++ b/docs/en/solutions/Image_tag_cannot_be_selected_in_Update_Image_Version_because_the_integration_repositories_request_returns_500.md @@ -0,0 +1,94 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "3.10" +--- + +# Image tag cannot be selected in Update Image Version because the integration repositories request returns 500 + +## Issue + +In the container platform, the image tag cannot be selected in the **Update Image Version** dialog of a Deployment container. The tag dropdown stays empty or fails to load, although the repository in Harbor has tags. The failure is intermittent: the same dialog sometimes works and sometimes does not. The browser developer tools show the failing request returning `500` after about 10 seconds with this body: + +```json +{ + "metadata": {}, + "status": "Failure", + "message": "Internal error occurred: unexpected error when reading response body. Please retry. Original error: net/http: request canceled (Client.Timeout or context cancellation while reading body)", + "reason": "InternalError", + "code": 500 +} +``` + +The `harbor-core` log repeats `[panic.go:969] Handler crashed with error net/http: abort Handler`, while every Harbor component is Running and none restarts often. + +## Root Cause + +### Cause 1: katanomi has run for a very long time without a restart, and its repository list requests to Harbor time out + +The dialog reads registry data through the DevOps v3 toolchain Harbor integration, which is served by katanomi (`katanomi-api`, `katanomi-plugin`, and `katanomi-controller` in `cpaas-system`, and `katanomi-operator` in `operators`). + +The dialog first lists registry projects (`GET .../projects?includeType=ImageRegistry&itemsPerPage=100&page=1`), and that call returns `200`. It then lists the repositories of the selected Harbor project: + +```text +GET /integrations/v1alpha1/namespaces//integrations//projects//repositories?itemsPerPage=100&page=1 +``` + +That call fails with the `500` above after about 10 seconds. When the katanomi Pods have been running for a very long time (for example about two years without a restart), they accumulate cached state, and with that state the repository list requests to Harbor time out. The integration API returns the timeout to the browser as `InternalError`. The `abort Handler` panic in `harbor-core` accompanies these timeouts and is consistent with Harbor's handler being aborted because the caller gave up the request, which is why Harbor itself looks healthy. + +Restarting the katanomi components, including the `katanomi-operator` Pod, reloads that state, and tag selection works again. + +## Resolution + +### Cause 1: katanomi has run for a very long time without a restart, and its repository list requests to Harbor time out + +1. Capture the requests in the browser developer tools (Network tab) while opening the tag dropdown. This cause needs both: `projects?includeType=ImageRegistry...` returns `200`, and `repositories?itemsPerPage=100&page=1` returns `500` with `Client.Timeout or context cancellation while reading body`. An empty `200` answer is not this cause. + +2. Confirm that Harbor has the repositories and answers directly, so that the fault is on the integration side: + + ```bash + curl -sk -u ':' --max-time 30 \ + -w '\ncode=%{http_code} time=%{time_total}\n' \ + 'https:///api/v2.0/projects//repositories?page=1&page_size=100' + kubectl -n get pods + kubectl -n logs --tail=300 | grep -c 'abort Handler' + ``` + +3. Read the katanomi Pod ages and restart counts: + + ```bash + kubectl get pods -A -o wide | grep katanomi + ``` + + Running katanomi Pods that are months or years old, with Harbor answering on its own, match this cause. A katanomi Pod in `CrashLoopBackOff` is a different fault. + +4. Save the logs, then restart the katanomi Pods. + + > **Warning:** Deleting the katanomi Pods makes DevOps pages, toolchain integrations, image and tag selectors, and pipeline triggers unavailable on that cluster until the new Pods are Ready. Running TaskRun Pods are not deleted. Precondition: only when the `repositories` call returns `500` with `Client.Timeout or context cancellation while reading body`, Harbor answers the same listing directly, and the katanomi Pods are Running. Do it in a quiet period. Rollback: none needed; the owners recreate the Pods from an unchanged spec. + + ```bash + mkdir -p /tmp/katanomi-logs + for p in $(kubectl -n cpaas-system get pods -o name | grep katanomi); do + kubectl -n cpaas-system logs "$p" --all-containers --tail=500 > "/tmp/katanomi-logs/$(basename "$p").log" + done + kubectl -n cpaas-system delete pod + kubectl -n operators delete pod + kubectl get pods -A | grep katanomi + ``` + +5. Open the **Update Image Version** dialog again, several times, because the fault was intermittent. The `repositories` request returns `200` well within 10 seconds and the tags are listed. If the `500` returns soon after the restart, keep the saved katanomi logs and the `harbor-core` log for the support team. + +## Diagnostic Steps + +```bash +kubectl get pods -A -o wide | grep katanomi +kubectl -n get pods +kubectl -n logs --tail=300 | grep 'abort Handler' +``` + +- Browser: `repositories?itemsPerPage=100&page=1` returns `500` with `Client.Timeout or context cancellation while reading body`, Harbor is healthy, and the katanomi Pods are very old: Cause 1. +- The image dropdown is empty in one environment only, with no `500`, and the platform project is not assigned to the Harbor project in the toolchain integration: see "Harbor toolchain integration fails: images not selectable, katanomi-plugin timeout, or push asks for credentials". +- A tag that cannot be selected in an artifact promotion form, not in the Update Image Version dialog, is a different problem. diff --git a/docs/en/solutions/Jenkins_Maven_build_fails_with_java.lang.OutOfMemoryError_Java_heap_space_although_build_node_resources_were_raised.md b/docs/en/solutions/Jenkins_Maven_build_fails_with_java.lang.OutOfMemoryError_Java_heap_space_although_build_node_resources_were_raised.md new file mode 100644 index 000000000..772b54ecc --- /dev/null +++ b/docs/en/solutions/Jenkins_Maven_build_fails_with_java.lang.OutOfMemoryError_Java_heap_space_although_build_node_resources_were_raised.md @@ -0,0 +1,89 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.0" + - "4.3" +--- + +# Jenkins Maven build fails with java.lang.OutOfMemoryError Java heap space although build node resources were raised + +## Issue + +A Java back-end project cannot be deployed by a Jenkins pipeline because the build reports that the heap is full, while front-end pipelines on the same platform deploy normally. Raising the build node resources did not help. Typically: + +- The `mvn-build` stage runs for about 15 minutes and then fails; the later `image-build` and `deploy` stages do not run. +- The stage log contains `The system is out of resources.` and `java.lang.OutOfMemoryError: Java heap space`, with a stack trace through `lombok` and `com.sun.tools.javac`, so the failure happens while the code is compiled. +- The build node resources were already raised, for example to request CPU `2` / memory `2Gi` and limit CPU `3` / memory `3Gi` on the `java` build node, and the error did not change. + +The platform default `java` build node uses the image `devops/builder-java:8`, the labels `java`, `java-openjdk8`, `maven`, `maven3.5`, and mounts the Maven cache PVC `maven-cache` at `/root/.m2/repository`. Build node resources are edited under **Toolchain Management** > **Instance Management** > the Jenkins instance > **Build Nodes** > edit the node. A project can have its own auto-assigned Jenkins instance, so the instance that runs the pipeline is not necessarily the one that was edited. + +## Root Cause + +### Cause 1: the Maven JVM heap is too small + +The Maven JVM does not have enough heap to compile and package the project. Neither Jenkins nor the size of the build container causes the failure, and a larger container limit does not, by itself, give Maven a larger heap. The JVM picks its default maximum heap when it starts, and an explicit `-Xmx` in `MAVEN_OPTS` overrides that default. Starting Maven with an explicit heap lets the pipeline deploy; in the reported case the build time also dropped from about 15 minutes to about 3 minutes. + +Raising the build node resources first does not help for two reasons: the heap does not change, and the edit may be made on a Jenkins instance that the pipeline does not use. + +## Resolution + +### Cause 1: the Maven JVM heap is too small + +Apply this only when the Maven stage log contains `java.lang.OutOfMemoryError: Java heap space`. A Pod that ends `OOMKilled` without that line has a container limit problem (see Diagnostic Steps). + +1. Set the Maven JVM heap on the packaging command in the pipeline's Maven stage: + + ```bash + MAVEN_OPTS="-Xmx2048m -Xms1024m -XX:MaxPermSize=512m" mvn clean package + ``` + + `-XX:MaxPermSize` applies only to Java 7 and earlier. A Java 8 JVM, such as the one in `builder-java:8`, prints a warning that it ignores the option and then runs. Java 17 and later JVMs can refuse to start with `Unrecognized VM option 'MaxPermSize=512m'`; on those build nodes (for example `java21`), drop that option and keep `-Xmx2048m -Xms1024m`. + +2. Keep `-Xmx` below the memory limit of the build container, with room for non-heap memory (metaspace, threads, and the compiler's native memory). A 2 GiB heap fits a `3Gi` limit. Against a node that still has a lower default limit, for example the `java21` build node default of request CPU `500m` / memory `750Mi` and limit CPU `1` / memory `2500Mi`, a 2 GiB heap leaves little room, so raise the limit as well. + + > **Warning:** Higher requests and limits on a build node apply to every pipeline that runs on that node, and the requests count against the namespace quota. The agent Pod can stay Pending if no node or quota can satisfy them. Precondition: you confirmed in Diagnostic Steps which Jenkins instance and build node the pipeline actually uses. Rollback: set the previous request and limit values back on the build node. + + Edit the build node on the Jenkins instance that the pipeline uses: **Toolchain Management** > **Instance Management** > that Jenkins instance > **Build Nodes** > the node (for example `java`) > change **Resource Requests** and **Resource Limits**. Two to three times the original values is a reasonable guideline. + +3. On a custom build node, `MAVEN_OPTS` can also be set in the node's **Environment Variables** field, so that every Maven run on it gets the heap. A per-project alternative is the standard Maven file `.mvn/jvm.config` in the repository. + +4. Run the pipeline again. The `mvn` stage finishes without `OutOfMemoryError`, followed by the image build and deploy stages. + +## Diagnostic Steps + +1. Confirm the error text in the exported pipeline log: + + ```bash + grep -nE 'OutOfMemoryError|The system is out of resources|GC overhead limit exceeded' + ``` + +2. While the pipeline runs again, find the agent Pod on the pipeline's cluster and check that it uses the instance and build node you edited: + + ```bash + kubectl get pod -A | grep java + kubectl describe pod -n + ``` + + The output shows the image (for example `devops/builder-java:8`) and the `Limits` and `Requests` of each container. If those values are not the ones you set, the pipeline runs on a different Jenkins instance or build node; edit the build node of the instance that serves the project. + +3. Check whether the JVM ran out of heap or the container was killed: + + ```bash + kubectl get pod -n -o jsonpath='{range .status.containerStatuses[*]}{.name}{" "}{.state.terminated.reason}{" "}{.lastState.terminated.reason}{"\n"}{end}' + ``` + + `OOMKilled` means the container limit is too small for the heap plus non-heap memory. `java.lang.OutOfMemoryError: Java heap space` in the log, without `OOMKilled`, is Cause 1. + +4. To see the default heap of the JVM in that container and what Maven receives, add these lines before `mvn` in the stage once: + + ```bash + echo "MAVEN_OPTS=${MAVEN_OPTS-unset}" + java -XX:+PrintFlagsFinal -version 2>/dev/null | grep -E ' MaxHeapSize ' + ``` + + A `MaxHeapSize` far below the size the project needs to compile, with `MAVEN_OPTS` unset, matches Cause 1. + +A Maven stage that fails on dependency resolution, or a green build whose image contains an old copy of a republished dependency, is a different problem; the latter is covered in "Jenkins Maven build packages an old dependency jar after the same version is republished to the private repository". diff --git a/docs/en/solutions/Jenkins_Maven_build_packages_an_old_dependency_jar_after_the_same_version_is_republished_to_the_private_repository.md b/docs/en/solutions/Jenkins_Maven_build_packages_an_old_dependency_jar_after_the_same_version_is_republished_to_the_private_repository.md new file mode 100644 index 000000000..011e0ec9b --- /dev/null +++ b/docs/en/solutions/Jenkins_Maven_build_packages_an_old_dependency_jar_after_the_same_version_is_republished_to_the_private_repository.md @@ -0,0 +1,107 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: [] +--- + +# Jenkins Maven build packages an old dependency jar after the same version is republished to the private repository + +## Issue + +A custom library is published to the private Maven repository again without raising its version number (a release version, not `-SNAPSHOT`). The Jenkins pipeline runs green and builds a new service image, but the library jar in the application's `lib` directory inside the image is still the earlier upload. The build did not fetch the republished jar from the repository, and the question is whether Jenkins caches the jar and how to replace the cached copy. + +Jenkins Java build nodes can keep the Maven local repository on persistent storage. The platform default `java` build node mounts the Maven cache PVC `maven-cache` at `/root/.m2/repository`, so artifacts downloaded by one build stay available to later builds. + +## Root Cause + +### Cause 1: the old copy of the republished release version is still in the persistent Maven cache + +The earlier jar is still in the Maven cache that the build node mounts. Because the republished jar kept the same version, the build reuses the cached copy and does not download the new one. Removing the old version directory from the cache makes later builds package the new upload. + +This matches standard Maven resolution. Maven looks for a release version in the local repository first, and once that version is present it is resolved locally. The `mvn --help` text for `-U, --update-snapshots` reads "Forces a check for missing releases and updated snapshots on remote repositories", so `-U` refreshes SNAPSHOT versions and fetches releases that are absent locally; a release that is already in the cache is used as it is. With a cache that persists across builds, the first copy of a release version stays in use until it is removed from the cache. + +Signature: the Maven stage log has no `Downloading from` / `Downloaded from` line for the republished artifact, and the jar in the cache directory is older than, and has a different checksum from, the one the repository serves now. + +## Resolution + +### Cause 1: the old copy of the republished release version is still in the persistent Maven cache + +Apply this only when Diagnostic Steps confirm all three: the repository now serves the new jar, the cached jar differs from it, and the build log shows no download of that artifact. + +1. Find the directory that holds the Maven cache. On the default `java` build node it is the volume behind PVC `maven-cache`, seen in the agent Pod at `/root/.m2/repository`. Locate its backing path: + + ```bash + kubectl get pvc -A | grep maven-cache + kubectl get pvc -n maven-cache -o jsonpath='{.spec.volumeName}{"\n"}' + kubectl get pv -o yaml + ``` + + In the PV, read `spec.hostPath.path`, `spec.local.path` together with `spec.nodeAffinity`, `spec.nfs`, or the CSI driver attributes to find the node or server and the directory. That directory is the repository root. A custom build node can mount a different claim; read its name from the agent Pod volumes (Diagnostic Steps, step 1). + +2. On that node or storage server, go to the version directory of the republished artifact. The group path is the `groupId` with `.` replaced by `/`: + + ```bash + REPO= + GAV_DIR="$REPO///" + ls -l --time-style=long-iso "$GAV_DIR" + ``` + + The jar date should match the earlier upload. + +3. Move the version directory out of the repository. + + > **Warning:** Removing the version directory makes the next build download that artifact again. If the private repository is unreachable from the build, or the version was removed from it, that build fails on dependency resolution. A build that runs on the same cache while you move the directory can fail or package a partial file. Precondition: only when Cause 1 is confirmed by Diagnostic Steps, the repository already serves the new jar, and no pipeline that uses this cache is running. Rollback: move the directory back (command below). Deleting the backup later is irreversible. + + ```bash + mkdir -p /root/maven-cache-backup + mv "$GAV_DIR" /root/maven-cache-backup/- + # rollback: + # mv /root/maven-cache-backup/- "$GAV_DIR" + ``` + +4. Run the pipeline again. The Maven stage log shows `Downloading from :` and `Downloaded from :` for `-.jar`, and the jar in the new image has the checksum of the new upload (Diagnostic Steps, step 4). Then remove the backup directory. + +Each further republish under the same version leaves the cache in the same state, so the cleanup has to be repeated. In general Maven practice, a library that changes without a version bump is published as a `-SNAPSHOT` version and the consuming build runs with `-U`, or each change gets a new release version. Running `-U` on every build adds update checks and time to the Maven stage. + +## Diagnostic Steps + +1. Confirm which cache the build uses. While the pipeline runs, describe the Java agent Pod and read the volume mounted at `/root/.m2/repository` and its claim name: + + ```bash + kubectl get pod -A | grep java + kubectl describe pod -n + ``` + +2. Check whether the build downloaded the artifact or resolved it locally. In the exported Maven stage log: + + ```bash + grep -nE 'Download(ing|ed) from .*-' + ``` + + No match means the jar came from the local repository, which is the Cause 1 path. + +3. Compare the cached jar with what the repository serves now. In the cache, or inside a running agent Pod at `/root/.m2/repository`: + + ```bash + ls -l --time-style=long-iso "$GAV_DIR" + sha1sum "$GAV_DIR/-.jar" + cat "$GAV_DIR/_remote.repositories" + ``` + + `_remote.repositories` names the repository id the jar was downloaded from. Ask that repository for the checksum of the current upload, using the URL from the build's `settings.xml`: + + ```bash + curl -sS -u "////-.jar.sha1"; echo + ``` + + Different checksums, with the cached file dated at the earlier upload, confirm Cause 1. If the repository still returns the checksum of the old jar, the republish has not reached the URL that the build resolves from (for example a group or proxy repository in front of the hosted one). Clearing the build cache would then download the same old file again, so fix the repository first. + +4. Check the jar that ended up in the image: + + ```bash + docker run --rm --entrypoint sh -c 'sha1sum /-.jar' + ``` + + If that checksum already equals the repository's new checksum, the stale jar is not coming from the Maven cache, and Cause 1 does not apply. diff --git a/docs/en/solutions/Keycloak_shows_HTTPS_required_the_CR_warns_about_hostname-strict_or_login_shows_Invalid_parameter_redirect_uri.md b/docs/en/solutions/Keycloak_shows_HTTPS_required_the_CR_warns_about_hostname-strict_or_login_shows_Invalid_parameter_redirect_uri.md new file mode 100644 index 000000000..ba9a58106 --- /dev/null +++ b/docs/en/solutions/Keycloak_shows_HTTPS_required_the_CR_warns_about_hostname-strict_or_login_shows_Invalid_parameter_redirect_uri.md @@ -0,0 +1,137 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Application Services +ProductsVersion: + - "4.3" +--- + +# Keycloak shows HTTPS required, the CR warns about hostname-strict, or login shows Invalid parameter redirect_uri + +## Issue + +A Keycloak 26.4 instance, managed by the Keycloak operator through a `Keycloak` custom resource, is running, but: + +- The admin console is opened through a NodePort address over plain HTTP, and the login page (`realms/master`, client `security-admin-console`) shows `We are sorry...` `HTTPS required`. +- The `Keycloak` CR is `Ready` (`status: "True"`), but its `HasErrors` condition says `warning: You need to specify these fields as the first-class citizen of the CR: hostname-strict`. +- An application or connector that signs in through Keycloak gets the realm login page with `We are sorry...` `Invalid parameter: redirect_uri`. + +## Root Cause + +### Cause 1: `HTTPS required`: the realm's `sslRequired` setting rejects the plain HTTP request + +Every Keycloak realm has an `sslRequired` setting with the values `none`, `external` (recommended for production), and `all`. With `external`, Keycloak requires HTTPS for requests from external addresses; with `all`, it requires HTTPS for every request. A request that the setting covers and that arrives over plain HTTP gets the `HTTPS required` page. Access through a NodePort over plain HTTP falls under that rule. Keycloak should be accessed over HTTPS. + +### Cause 2: `hostname-strict` is passed as a generic option instead of a first-class CR field + +The `hostname-strict` option was passed as a generic option of the instance (for example under `additionalOptions`) instead of through the dedicated fields of the CR. The operator expects hostname settings in `spec.hostname`; `spec.hostname.strict` is the entry point for `hostname-strict`. The CR stays `Ready`; the condition is a warning. + +### Cause 3: `Invalid parameter: redirect_uri`: the client's redirect URI is not registered + +Keycloak returns this page when it rejects the `redirect_uri` that the client sends in the authorization request. In standard Keycloak behavior that happens when the value is not covered by the **Valid redirect URIs** of the client named in `client_id`. Keycloak compares the URI exactly, except that a registered entry can end with a `*` wildcard. + +## Resolution + +### Cause 1: `HTTPS required`: the realm's `sslRequired` setting rejects the plain HTTP request + +1. Access Keycloak over HTTPS, for example through an Ingress with TLS or the instance's HTTPS endpoint, instead of an HTTP NodePort. + +2. For realms that you create, set `sslRequired` when the realm is imported through a `KeycloakRealmImport`: + + ```yaml + apiVersion: k8s.keycloak.org/v2alpha1 + kind: KeycloakRealmImport + metadata: + name: -import + namespace: + spec: + keycloakCRName: + realm: + realm: + enabled: true + sslRequired: external + ``` + + ```bash + kubectl api-resources | grep -i keycloak + kubectl apply -f -import.yaml + kubectl -n get keycloakrealmimports + ``` + + Use the API version that `kubectl api-resources` lists for `KeycloakRealmImport` in your cluster. `external` is the value recommended for production. + +### Cause 2: `hostname-strict` is passed as a generic option instead of a first-class CR field + +Move the hostname options into `spec.hostname` and remove `hostname-strict` from the generic options. + +> **Warning:** Changing the hostname settings restarts Keycloak and changes the URLs that Keycloak issues in redirects and tokens; clients that use a different address can stop working. Precondition: the external hostname that clients use is known and resolves to the Keycloak entry point. Save the CR first. Rollback: `kubectl apply -f keycloak-before.yaml`. + +```bash +kubectl -n get keycloak -o yaml > keycloak-before.yaml +kubectl -n edit keycloak +``` + +```yaml +spec: + hostname: + hostname: + strict: true + strictBackchannel: false # depending on the scenario +``` + +After saving, check that the `HasErrors` warning is gone (Diagnostic Steps, step 1). + +### Cause 3: `Invalid parameter: redirect_uri`: the client's redirect URI is not registered + +1. Take the `client_id` and the decoded `redirect_uri` from the failing request and compare them with the client registered in the realm (Diagnostic Steps, step 4). + +2. Make them match: either add the exact callback address that the application sends to the client's **Valid redirect URIs** in the realm, or change the application's callback address to one that is already registered. + + > **Warning:** Adding broad wildcard redirect URIs lets tokens be sent to any matching address. Precondition: register only the callback addresses that the application actually uses. Rollback: remove the added entry from the client. + +3. Retry the sign-in. + +## Diagnostic Steps + +1. Read the CR conditions: + + ```bash + kubectl -n get keycloak -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" "}{.message}{"\n"}{end}' + ``` + + `HasErrors` with `first-class citizen of the CR: hostname-strict` is Cause 2. + +2. Check how Keycloak is exposed and with which scheme the console is opened: + + ```bash + kubectl -n get svc,ingress + curl -sk -o /dev/null -w '%{http_code}\n' "https:///realms/master/protocol/openid-connect/auth?client_id=security-admin-console" + ``` + + A console opened through an `http://` NodePort address that shows `HTTPS required` is Cause 1. + +3. Read the `sslRequired` value of a realm from inside the Keycloak Pod, after logging in with an admin account: + + ```bash + kubectl -n exec -it -- sh -c \ + '/opt/keycloak/bin/kcadm.sh config credentials --config /tmp/kcadm.config --server http://localhost:8080 --realm master --user && + /opt/keycloak/bin/kcadm.sh get realms/ --config /tmp/kcadm.config --fields sslRequired' + ``` + + If the instance serves only TLS, use `--server https://localhost:8443` with `--truststore` or the instance certificate instead. + +4. For `Invalid parameter: redirect_uri`, copy the full URL of the failing login page from the browser address bar, decode it, and read the parameters: + + ```bash + python3 -c 'import sys, urllib.parse as u; q = u.parse_qs(u.urlsplit(sys.argv[1]).query); print(q.get("client_id"), q.get("redirect_uri"))' '' + ``` + + Then list the redirect URIs registered for that client, with the same `kcadm.sh` session as in step 3: + + ```bash + kubectl -n exec -it -- \ + /opt/keycloak/bin/kcadm.sh get clients -r -q clientId= --fields clientId,redirectUris --config /tmp/kcadm.config + ``` + + A `redirect_uri` that does not appear in `redirectUris` (and is not covered by a trailing `*` entry) is Cause 3. diff --git a/docs/en/solutions/Kiali_will_not_open_or_log_in_shows_Core_istiod_Not_found_cannot_reach_VictoriaMetrics_or_shows_too_many_namespaces.md b/docs/en/solutions/Kiali_will_not_open_or_log_in_shows_Core_istiod_Not_found_cannot_reach_VictoriaMetrics_or_shows_too_many_namespaces.md new file mode 100644 index 000000000..7c6f8ba53 --- /dev/null +++ b/docs/en/solutions/Kiali_will_not_open_or_log_in_shows_Core_istiod_Not_found_cannot_reach_VictoriaMetrics_or_shows_too_many_namespaces.md @@ -0,0 +1,255 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Service Mesh +ProductsVersion: + - "4.1" + - "4.2" + - "4.3" +--- + +# Kiali will not open or log in, shows Core istiod Not found, cannot reach VictoriaMetrics, or shows too many namespaces + +## Issue + +Kiali (Alauda Build of Kiali) on Service Mesh v2 shows one of these problems: + +- Kiali access fails and its log shows nothing useful. OpenID login goes to a wrong address: behind the business-cluster ALB, `GET .../kiali/api/auth/info` returns an authorization address built from the business cluster domain instead of the ACP platform domain. +- The page shows `Cannot start authentication because it is not possible to use OpenId's authorization code flow. Check Kiali logs for more details.` at `/kiali/api/auth/openid_redirect`. +- Kiali loads, but its metrics are empty because the Kiali server cannot reach the cluster's VictoriaMetrics. +- The Overview page shows the cluster badge `Kubernetes` in error, Cluster Status `Core istiod Not found`, `Control planes (0)`, and `Data planes (0)`. +- A user who has access to only one namespace should see only that namespace in Kiali. + +A failure after the ACP login with `Authentication openid_error: Unable to create a Kubernetes client from the auth token: illegal base64 data at input byte ` is covered in "Kiali OpenID login fails with illegal base64 data at input byte because api_proxy_ca_data is malformed". + +## Root Cause + +### Cause 1: OpenID login uses the business cluster domain because `web_fqdn` is not set and the ALB sets `X-Forwarded-Host` + +The authorization endpoint that Kiali returns from `/api/auth/info` is built from `conf.Server.WebFQDN` (`GuessKialiURL` in the kiali-server source). When that value is empty, Kiali falls back in order to the `X-Forwarded-Host` request header, then the request URL host name, then the request `Host`. With `web_fqdn` not configured, the business-cluster ALB sets `X-Forwarded-Host` to the business cluster domain, so the authorization address is wrong. The ALB log on the business cluster shows Kiali requests (rule `istio-system/kiali`, upstream port `20001`) with the business cluster domain in `host`, while the same requests through the global cluster ALB under `/clusters//kiali/` carry the ACP platform domain. + +Switching the Kiali CR to `spec.auth.strategy: anonymous` loads the console through the ACP platform domain at `/clusters//kiali/console/overview`, which isolates the fault to the OpenID address. OpenID access is restored by setting `web_fqdn` and the related server settings in the Kiali CR, turning off automatic ingress generation, and creating the Ingress `kiali` in `istio-system` by hand with an annotation that clears `X-Forwarded-Host`. + +### Cause 2: OpenID discovery times out because of a node DNS search domain + +The Kiali log shows: + +```text +WRN Error when fetching OpenID provider's metadata: Get "https:///dex/.well-known/openid-configuration": context deadline exceeded (Client.Timeout exceeded while awaiting headers) +WRN Could not read the session: session [not found]: cookie [kiali-token-Kubernetes] does not exist in request +``` + +The Kiali Pod runs on a node that has an extra DNS search domain configured. Pods with the default DNS policy inherit the node's search domains, and that search domain affects name resolution, so Kiali's request to the Dex discovery endpoint times out. Removing the search domain from the node makes the Kiali page accessible. + +### Cause 3: Kiali cannot reach VictoriaMetrics through the business-cluster ALB + +Kiali opens, but a network problem prevents the Kiali server from reaching the cluster's VictoriaMetrics through the business-cluster ALB. Pointing the Kiali monitoring address at the ACP platform domain address is a workaround; the network path itself still needs a fix. + +### Cause 4: `Core istiod Not found` because Istio is not deployed + +The Kiali log shows: + +```text +WRN Cannot resolve local cluster name. Err: no istiod deployment found, no clusterName determined. Falling back to [Kubernetes] +``` + +Istio is not deployed in the cluster, so Kiali has no control plane to show. + +### Cause 5: namespace visibility follows ACP RBAC by default + +Kiali with `auth.strategy: openid` and `auth.openid.disable_rbac: false` enforces ACP RBAC, so no extra synchronization is needed. Namespaces that the user has no permission for may still be listed, but no data is displayed for them. A user given a RoleBinding in only one namespace sees only that namespace in Kiali. + +## Resolution + +### Cause 1: OpenID login uses the business cluster domain because `web_fqdn` is not set and the ALB sets `X-Forwarded-Host` + +1. Confirm the host that the ALB passes to Kiali on the business cluster, and the current Kiali settings: + + ```bash + kubectl logs -l service_name=alb2-cpaas-system --tail 200 -n cpaas-system | grep '//kiali/' + kubectl get kiali -n istio-system + kubectl get kiali -n istio-system -o jsonpath='{.spec.auth}{"\n"}{.spec.server}{"\n"}{.spec.deployment.ingress}{"\n"}' + ``` + +2. Optionally isolate the fault by switching to `anonymous` for a short test. + + > **Warning:** With `strategy: anonymous`, anyone who can reach the URL uses Kiali without logging in. Precondition: only as a short isolation test for Cause 1, with the Kiali CR backed up. Rollback: patch `strategy` back to `openid` (first command of the next step). + + ```bash + kubectl get kiali -n istio-system -o yaml > /tmp/kiali-cr.bak.yaml + kubectl patch kiali -n istio-system --type=merge -p '{"spec":{"auth":{"strategy":"anonymous"}}}' + ``` + +3. Set `spec.server.web_fqdn` to the ACP platform domain that the browser uses, and turn off automatic ingress generation with `spec.deployment.ingress.enabled: false`. Align the related server settings (`spec.server.web_root`, `spec.server.web_port`, `spec.server.web_schema`) with the platform URL in the browser. + + > **Warning:** The Kiali operator reconciles, Kiali restarts, and the generated ingress is removed, so Kiali is unreachable until the Ingress in the next step exists. Precondition: only when Cause 1 is confirmed (wrong host in `/api/auth/info` or in the ALB log, and `web_fqdn` empty) and `/tmp/kiali-cr.bak.yaml` exists. Rollback: restore the previous `spec.server` and `spec.deployment.ingress` values from the backup, and delete the manual Ingress. + + ```bash + kubectl patch kiali -n istio-system --type=merge -p '{"spec":{"auth":{"strategy":"openid"}}}' + kubectl patch kiali -n istio-system --type=merge \ + -p '{"spec":{"server":{"web_fqdn":""},"deployment":{"ingress":{"enabled":false}}}}' + ``` + +4. Create the Ingress, clearing `X-Forwarded-Host`: + + ```yaml + apiVersion: networking.k8s.io/v1 + kind: Ingress + metadata: + name: kiali + namespace: istio-system + annotations: + alb.ingress.cpaas.io/rewrite-request: '{"headers_var": {"X-Forwarded-Host": ""}}' + nginx.ingress.kubernetes.io/backend-protocol: HTTP + nginx.ingress.kubernetes.io/secure-backends: "false" + spec: + ingressClassName: cpaas-system + rules: + - http: + paths: + - path: /clusters//kiali + pathType: Prefix + backend: + service: + name: kiali + port: + number: 20001 + ``` + + ```bash + kubectl apply -f kiali-ingress.yaml + kubectl get ingress -n istio-system kiali + ``` + +5. Log in through the ACP platform domain. OpenID login succeeds. + +### Cause 2: OpenID discovery times out because of a node DNS search domain + +1. Confirm the Dex discovery timeout and find the node that runs Kiali; compare its resolver configuration with a node where Kiali works: + + ```bash + kubectl -n istio-system logs deploy/kiali --tail=300 | grep -i -E 'openid|context deadline' + kubectl -n istio-system get pod -o wide | grep kiali + kubectl -n istio-system exec deploy/kiali -- cat /etc/resolv.conf + # on the Kiali node and on a healthy node + cat /etc/resolv.conf + ``` + + An extra `search` domain on the Kiali node that the healthy nodes do not have matches this cause. + +2. Remove the extra search domain from the node. + + > **Warning:** Changing the node resolver changes name resolution for the node and for Pods created on it afterwards. Precondition: only when Cause 2 is confirmed. Back up the file first. If the file is generated by a network manager, remove the search domain in that manager's configuration instead, or it comes back. Rollback: `cp /etc/resolv.conf.bak /etc/resolv.conf` (or restore the network manager setting). + + ```bash + cp /etc/resolv.conf /etc/resolv.conf.bak + vi /etc/resolv.conf + ``` + +3. A Pod's `/etc/resolv.conf` is written when the Pod is created, so recreate the Kiali Pod if it still times out: + + ```bash + kubectl -n istio-system delete pod + ``` + + The Kiali page becomes accessible. + +### Cause 3: Kiali cannot reach VictoriaMetrics through the business-cluster ALB + +1. Read where Kiali points for metrics and confirm the failure in its log: + + ```bash + kubectl get kiali -n istio-system -o jsonpath='{.spec.external_services.prometheus}{"\n"}' + kubectl -n istio-system logs deploy/kiali --tail=200 | grep -i -E 'prometheus|metrics|timeout' + ``` + +2. As a workaround, point Kiali at the VictoriaMetrics address on the ACP platform domain. + + > **Warning:** Kiali restarts and queries metrics through the platform domain, so the platform entry carries Kiali's metric queries. Precondition: only when Cause 3 is confirmed. Back up the CR first. Rollback: `kubectl apply -f /tmp/kiali-cr-monitoring.bak.yaml`. + + ```bash + kubectl get kiali -n istio-system -o yaml > /tmp/kiali-cr-monitoring.bak.yaml + kubectl patch kiali -n istio-system --type=merge \ + -p '{"spec":{"external_services":{"prometheus":{"url":""}}}}' + ``` + +3. Fix the network path from Kiali to the business-cluster ALB separately. + +### Cause 4: `Core istiod Not found` because Istio is not deployed + +```bash +kubectl get istio,istiorevision -A +kubectl get deploy -A | grep istiod +``` + +No `Istio` object and no `istiod` Deployment confirm this cause. Deploy Istio by following the Service Mesh v2 installation documentation. Kiali shows the control plane after `istiod` is running. + +### Cause 5: namespace visibility follows ACP RBAC by default + +1. Confirm the Kiali auth settings. A working layout on the `global` cluster: + + ```yaml + auth: + strategy: openid + openid: + api_proxy: /kubernetes/global + api_proxy_ca_data: + client_id: alauda-auth + disable_rbac: false + insecure_skip_verify_tls: true + issuer_uri: /dex + username_claim: email + ``` + + ```bash + kubectl get kiali -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" "}{.spec.auth.strategy}{" disable_rbac="}{.spec.auth.openid.disable_rbac}{"\n"}{end}' + ``` + +2. With `disable_rbac: false`, grant the user access only to the target namespace, for example with a RoleBinding. With `username_claim: email`, the subject name is the user's email. Bind the namespace role that the user should have: + + ```yaml + apiVersion: rbac.authorization.k8s.io/v1 + kind: RoleBinding + metadata: + name: -kiali-view + namespace: + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: User + name: + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: + ``` + + ```bash + kubectl apply -f -kiali-rolebinding.yaml + ``` + +3. The user sees only that namespace in Kiali. + +## Diagnostic Steps + +1. Read the Kiali log first; most of these causes have a distinct line: + + ```bash + kubectl -n istio-system logs deploy/kiali --tail=300 | grep -E 'WRN|ERR' + ``` + +2. Check the auth and server settings and how Kiali is exposed: + + ```bash + kubectl get kiali -A -o yaml | grep -n -E 'strategy|web_fqdn|web_root|ingress|disable_rbac|prometheus' + kubectl get ingress -n istio-system + ``` + +| What you see | Cause | +| --- | --- | +| `/api/auth/info` or the ALB log carries the business cluster domain, and `web_fqdn` is empty | 1 | +| `Cannot start authentication ... authorization code flow`, log `context deadline exceeded` on `/dex/.well-known/openid-configuration` | 2 | +| Kiali up, metrics missing, Kiali cannot reach VictoriaMetrics through the business-cluster ALB | 3 | +| `Core istiod Not found`, `Control planes (0)`, log `no istiod deployment found ... Falling back to [Kubernetes]` | 4 | +| A user should see only their own namespaces | 5 | +| After the ACP login, `openid_error: ... illegal base64 data at input byte ` | see "Kiali OpenID login fails with illegal base64 data at input byte because api_proxy_ca_data is malformed" | diff --git a/docs/en/solutions/Maven_stage_misses_Nexus_mirrorOf_stays_central_after_mavenMirrorURL_or_501_HTTPS_Required_after_generate-settings.md b/docs/en/solutions/Maven_stage_misses_Nexus_mirrorOf_stays_central_after_mavenMirrorURL_or_501_HTTPS_Required_after_generate-settings.md new file mode 100644 index 000000000..c47edf162 --- /dev/null +++ b/docs/en/solutions/Maven_stage_misses_Nexus_mirrorOf_stays_central_after_mavenMirrorURL_or_501_HTTPS_Required_after_generate-settings.md @@ -0,0 +1,95 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.0" + - "4.3" +--- + +# Maven stage misses Nexus: mirrorOf stays central after mavenMirrorURL, or 501 HTTPS Required after generate-settings + +## Issue + +The Maven stage of a DevOps (Tekton) pipeline does not use the built-in Nexus as intended. One of these symptoms is shown: + +- The standard Maven task is given `mavenMirrorURL` for the built-in Nexus, but the generated settings keep `central`. Changing that parameter so that `mirrorOf` becomes `*` is not possible. +- Connectors created from the `maven-settings` workspace do not put dependency downloads on Nexus. +- The `generate-settings` step succeeds, Nexus was not selected for the run, and the Maven stage then fails with `Non-resolvable import POM` and `501 HTTPS Required`. The transfer names `https://maven.aliyun.com/repository/spring` and `https://repo.maven.apache.org/maven2/`. + +Maven settings travel in the `maven-settings` workspace. The built-in Nexus is the Service `nexus-nxrm-ha` in the namespace `devops-system`. Related articles: "DevOps Node.js or apt downloads fail: gnutls_handshake failed, npm install hangs, or the Nexus connection is reset" covers npm and network failures. + +## Root Cause + +### Cause 1: `mavenMirrorURL` always generates `mirrorOf` `central` + +The `mavenMirrorURL` parameter of the Maven task always generates a mirror with `mirrorOf` `central`. No value of that parameter yields `*`. Artifacts requested from any other repository id skip the mirror. The workspace Maven connector mounts `settings.xml` and is limited to proxy and authentication unless `useAsMirror=true` is set and the connector URL is a repository that can serve as a mirror. + +### Cause 2: `501 HTTPS Required` after `generate-settings` when Nexus is not set + +When Nexus is not specified, the pipeline generates default Maven settings, and `generate-settings` still succeeds. With those defaults the Maven stage fails to resolve an import POM, for example: + +```text +Non-resolvable import POM: org.springframework.boot:spring-boot-dependencies:pom:2.1.11.RELEASE +... could not be transferred from https://maven.aliyun.com/repository/spring ... +501 HTTPS Required ... https://repo.maven.apache.org/maven2/ +``` + +Specifying Maven settings, so that the generated default configuration is not used, clears the failure. This is not Cause 1: `mavenMirrorURL` is not involved, and the failure comes from the generated defaults, not from a mirror left at `central`. + +## Resolution + +### Cause 1: `mavenMirrorURL` always generates `mirrorOf` `central` + +Apply this only when the settings rendered by the task contain `central`, or the connector is mounted and traffic still misses Nexus. + +1. Stop using `mavenMirrorURL` for a full mirror. Put a `settings.xml` into the `maven-settings` workspace, or change the Task template so that it uses that file. To make the connector itself the mirror, set `useAsMirror=true` and aim the connector at the mirror repository URL. + + > **Warning:** A `mirrorOf` of `*` sends every repository id to one Nexus URL. Precondition: only when Nexus already proxies those artifacts, you have a copy of the old settings, and the rendered settings show `central` (or a connector that is not a mirror). Rollback: put the old settings back and run the pipeline again. + + ```xml + + + + internal-nexus + http://nexus-nxrm-ha.devops-system.svc.cluster.local/repository/maven-public/ + * + + + + ``` + + `maven-public` stands for the repository path on your Nexus; replace it with the real one. + +2. Run the pipeline again. The downloads appear in Nexus. + +### Cause 2: `501 HTTPS Required` after `generate-settings` when Nexus is not set + +Apply this only when `generate-settings` succeeded, Nexus was not selected on that run, and the Maven log contains `Non-resolvable import POM`, `https://maven.aliyun.com/repository/spring`, and `501 HTTPS Required`. + +1. Specify Maven settings so that the pipeline does not apply the generated defaults. Place the file in the `maven-settings` workspace, and point it at a repository that already contains the artifacts this build resolves. The mirror block shown for Cause 1 is one way to do it. + + > **Warning:** `*` sends every repository id, including plugin repositories, to that one URL. Precondition: only when the log signature above is in this run, Nexus was not selected, and that URL already proxies or hosts the artifacts the build needs. Keep the previous settings file. Rollback: restore that file and run the pipeline again. If the rendered `` is already `central` because `mavenMirrorURL` was set, use Cause 1 instead of adding a second mirror. + +2. Run the pipeline again. The Maven stage no longer prints `501 HTTPS Required` or the Aliyun spring repository URL. If `` later shows `central` and downloads still miss Nexus, that is Cause 1. + +## Diagnostic Steps + +From the Maven step, find the settings file and read the effective settings: + +```bash +find /workspace /root /home -name settings.xml -print +mvn help:effective-settings -N +``` + +- `central` after `mavenMirrorURL` was set, and downloads that never appear in Nexus: Cause 1. `*` already present, with failures after that, is a Nexus content or network problem, not this parameter. +- `generate-settings` is green, the Maven stage is red, Nexus was left unset, and the log shows the lines below: Cause 2. + + ```text + Non-resolvable import POM + https://maven.aliyun.com/repository/spring + 501 HTTPS Required + ``` + +A Maven stage that ends in `taskruntimeout`, or a JAR that is produced but omits files with Chinese names, is not a mirror problem. diff --git a/docs/en/solutions/OIDC_clients_and_pages_are_sent_to_the_wrong_Dex_issuer_address_on_ACP.md b/docs/en/solutions/OIDC_clients_and_pages_are_sent_to_the_wrong_Dex_issuer_address_on_ACP.md new file mode 100644 index 000000000..4118517e5 --- /dev/null +++ b/docs/en/solutions/OIDC_clients_and_pages_are_sent_to_the_wrong_Dex_issuer_address_on_ACP.md @@ -0,0 +1,96 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.10" + - "3.10.2" + - "3.16" + - "4.0" + - "4.2" + - "4.2.1" + - "4.3" + - "4.3.2" + - "4.3.3" +--- + +# OIDC clients and pages are sent to the wrong Dex issuer address on ACP + +## Issue + +The platform Dex advertises an issuer address that does not match how users or OIDC clients reach the platform. One of these is reported: + +- Users open the platform through one address (for example a floating IP in front of the HA VIP), and Service Mesh v2 pages such as Kiali or Jaeger, each with its own access address, redirect to the VIP for Dex. +- SonarQube reports `Issuer URL in provider metadata doesn't match the issuer URI specified in plugin configuration`, and other OIDC clients (GitLab, Harbor, Nexus) fail the same way when they are configured with `https:///dex`. The Dex discovery document on the DNS name still returns `"issuer": "https:///dex"`. + +Related articles: "Dex login options, CORS, SSO user identity, and the Dex deployment layout on ACP" and "Dex update fails on ACP 3.10 or account login returns 401 after the Dex permanent token changes". + +## Root Cause + +### Cause 1: Dex does not switch the issuer when more than one access address is in use + +Dex publishes one issuer and does not change it dynamically according to the address the browser used. Registering extra callback URLs does not make the issuer follow the browser host. When the console and the Mesh pages are opened on different addresses, the pages that use Dex are sent to the issuer address. The supported layout is one domain name for access. + +### Cause 2: Dex still advertises the IP issuer after the platform moved to a DNS name + +The platform was installed with an IP address as its platform access address. A DNS name and a new certificate were added later by rotating the TLS certificates of the platform access addresses. That rotation lets browsers reach the DNS name; it does not change the Dex issuer. The current product does not support converting an IP-based platform access address to a DNS name in place, so there is no in-place fix on the existing platform. The way to get a DNS issuer is to redeploy the platform with the domain name as the access address. + +On ACP 4.3.2 the default platform access address cannot be changed after installation, and neither can the internal registry address; additional access addresses can still be added. + +## Resolution + +### Cause 1: Dex does not switch the issuer when more than one access address is in use + +Publish one domain name and open the platform, Service Mesh, Kiali, and Jaeger on that domain. + +Only when a product page must stay up without Dex, and its owners accept an unauthenticated UI, these per-product changes remove the dependency on the issuer: + +> **Warning:** Kiali with `spec.auth.strategy: anonymous` no longer asks who the user is; anyone who can open the Kiali address can use it. Precondition: only when Kiali's OpenID redirect to the VIP is the confirmed problem and a single domain is not available. Rollback: `kubectl apply -f /tmp/kiali.bak.yaml`. + +```bash +kubectl get kiali -A +kubectl get kiali -n -o yaml > /tmp/kiali.bak.yaml +kubectl patch kiali -n --type=merge -p '{"spec":{"auth":{"strategy":"anonymous"}}}' +``` + +> **Warning:** Removing `spec.additionalContainers` removes the `oauth2-proxy` container from the OpenTelemetryCollector, and pointing the Ingress at `jaeger-collector-extension:16686` serves Jaeger without that proxy, unauthenticated. Precondition: only when Jaeger is the page that bounces to the VIP for Dex. Rollback: `kubectl apply -f /tmp/otelcol.bak.yaml` and restore the previous Ingress backend. + +```bash +kubectl get opentelemetrycollector -A +kubectl get opentelemetrycollector -n -o yaml > /tmp/otelcol.bak.yaml +kubectl patch opentelemetrycollector -n --type=json \ + -p='[{"op":"remove","path":"/spec/additionalContainers"}]' +``` + +Then change the backend of the Jaeger Ingress to the Service `jaeger-collector-extension`, port `16686`. + +### Cause 2: Dex still advertises the IP issuer after the platform moved to a DNS name + +1. Confirm the issuer that OIDC clients receive on the DNS name, and the certificate that name presents: + + ```bash + curl -k -sS https:///dex/.well-known/openid-configuration | jq -r .issuer + openssl s_client -connect :443 -servername /dev/null \ + | openssl x509 -noout -subject -ext subjectAltName + ``` + + `issuer` is `https:///dex`: the platform was installed on the IP, and the issuer cannot be changed in place. + +2. Plan a redeployment of the platform with the domain name as the platform access address, and configure the OIDC clients with `https:///dex` after that. Redeploying is not realistic once user clusters already run workloads. Until then, OIDC clients that require an exact issuer match can only use the IP issuer that the platform already advertises. + +3. If the certificate subject or SAN does not contain the DNS name, also replace the platform access certificate with one that contains it. A browser warning such as `Not secure` on the DNS name with a certificate CN for a different name is a separate certificate problem. + +Adding a domain only for toolchain callbacks, without changing the issuer, is covered in "Toolchain SSO on a domain name after an IP-based install, and kube-apiserver OIDC for an external UI on ACP". + +## Diagnostic Steps + +```bash +curl -k -sS https:///dex/.well-known/openid-configuration | jq -r .issuer +kubectl -n cpaas-system get cm dex-configmap -o yaml | grep -n issuer +kubectl get oauth2clients.dex.coreos.com -n cpaas-system +``` + +- Several browser hosts are in use, and the console or Mesh pages jump to the VIP for Dex: Cause 1. +- An OIDC client reports `Issuer URL in provider metadata doesn't match`, and the discovery `issuer` on the DNS name is still the install-time IP: Cause 2. +- `Unregistered redirect_uri` for `/console-portal/` on a toolchain host, with `client_id=alauda-auth`: not an issuer problem; see "Harbor SSO through the platform Dex fails: wrong issuer, OAuth2Client name, or Unregistered redirect_uri console-portal". diff --git a/docs/en/solutions/Pipeline_takes_more_than_40_minutes_because_the_Maven_stage_runs_mvn_-U.md b/docs/en/solutions/Pipeline_takes_more_than_40_minutes_because_the_Maven_stage_runs_mvn_-U.md new file mode 100644 index 000000000..9633b35a6 --- /dev/null +++ b/docs/en/solutions/Pipeline_takes_more_than_40_minutes_because_the_Maven_stage_runs_mvn_-U.md @@ -0,0 +1,48 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.3" +--- + +# Pipeline takes more than 40 minutes because the Maven stage runs mvn -U + +## Issue + +A pipeline needs more than 40 minutes to finish, and releases slip because of the wait. The run does complete: it is not stuck in Pending, and it does not fail with `index-pack` or `OOMKilled`. Per-stage timing shows that the Maven stage is the long part, and the Maven command in that stage includes `-U` (for example `mvn -U clean package`). + +## Root Cause + +### Cause 1: the Maven stage forces dependency updates with `-U` + +`-U` (`--update-snapshots`) makes Maven check the remote repositories for updated snapshots and for missing releases on every run. When the Maven stage is the stage that dominates the run and its command contains `-U`, that forced update check is the extra time in the stage, and removing it is the change to make. Removing `-U` is the optimization direction that matches the stage timing; measure the result on your own pipeline. + +## Resolution + +### Cause 1: the Maven stage forces dependency updates with `-U` + +1. Time the stages before changing the command. Continue only when Maven is the long stage and the command contains `-U`: + + ```bash + kubectl get pipelinerun -n + kubectl logs -n | grep -E 'mvn |BUILD ' + ``` + +2. Remove `-U` from the Maven invocation in the stage script, so that the stage uses the dependencies already resolved instead of forcing an update on every run. + + > **Warning:** Without `-U`, a newly published snapshot is not picked up until the local repository is refreshed or `-U` is added back for one run. Precondition: only when stage timing shows the Maven stage takes tens of minutes and the command includes `-U`. Rollback: restore `-U` on that Maven step. Leave the Maven cache volume in place. + +3. Compare the duration of the next run's Maven stage with the previous run. If Maven is still the long stage after `-U` is gone, the force update was not the only cost. Measure the remaining Maven log (download and compile phases) instead of changing clone memory or Pod scheduling. + +## Diagnostic Steps + +```bash +kubectl get pipelinerun -n +kubectl logs -n | grep -E 'mvn |BUILD |Downloading|Downloaded' +``` + +- A completed run whose Maven stage dominates the total time, with `mvn -U` in the command and many `Downloading` lines for metadata on every run: Cause 1. +- A run that never starts is a scheduling problem (Pod Pending), not Maven. +- A Node.js task killed for memory, or a clone failing in `index-pack`, is not related to the Maven `-U` flag. diff --git a/docs/en/solutions/Pipeline_task_Pod_does_not_start_connector-csi_OOMKilled_controller_image_pull_Connector_placeholder_or_PodSecurity.md b/docs/en/solutions/Pipeline_task_Pod_does_not_start_connector-csi_OOMKilled_controller_image_pull_Connector_placeholder_or_PodSecurity.md new file mode 100644 index 000000000..f4ca938f2 --- /dev/null +++ b/docs/en/solutions/Pipeline_task_Pod_does_not_start_connector-csi_OOMKilled_controller_image_pull_Connector_placeholder_or_PodSecurity.md @@ -0,0 +1,167 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.2.0" + - "4.3" + - "4.3.2" +--- + +# Pipeline task Pod does not start: connector-csi OOMKilled, controller image pull, Connector placeholder, or PodSecurity + +## Issue + +A DevOps v4 (Tekton) pipeline is created or run, and a task Pod does not start. One of these symptoms is shown: + +- The gitclone Pod stays unstarted while its volume is being mounted. +- The continuous-build API returns HTTP 500, and `katanomi-controller` cannot pull its own image. +- A buildah Pod stays in `Init:0/3`, and kubelet repeats: + + ```text + Warning FailedMount ... kubelet MountVolume.SetUp failed for volume "ws-" : rpc error: code = Unknown desc = failed to get Connector /{selectedConnectors()}: Connector.connectors.alauda.io "{selectedConnectors()}" not found + ``` + +- The TaskRun Pod is rejected at admission: + + ```text + pods "" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (containers "prepare", "place-scripts", "step-clone" ...) + ``` + + followed by the `capabilities.drop`, `runAsNonRoot`, and `seccompProfile` requirements. Namespaces are created with `pod-security.kubernetes.io/enforce: restricted` by default, and relaxing an application namespace to `privileged` is not acceptable because it places no constraint on user deployments. + +A clone that starts and then fails with `fatal: could not read Username` is a missing Git credential, not a mount failure. A step that fails inside Maven, the JDK, or Node.js after the Pod is Running is a build-environment problem. + +## Root Cause + +### Cause 1: gitclone Pod does not start because `connector-csi` is `OOMKilled` + +The gitclone Pod waits for a volume provided by the `connector-csi` component. The `connector-csi` Pod log shows a timeout fetching a ConfigMap, which looks like an API problem, but the Pod's container status shows that it was `OOMKilled` and restarted. The mount handler does not stay alive long enough to finish the volume setup, so the gitclone Pod never leaves the mount phase. Raising the `connector-csi` memory limit lets the mount complete. A ConfigMap RBAC error without `OOMKilled` is not this cause. + +### Cause 2: continuous build returns HTTP 500 because `katanomi-controller` cannot pull its image + +The controller Pod is not Ready because its image pull fails, so the API it serves returns HTTP 500. Once the pull can succeed, recreating the Pod clears the 500. A 500 while the controller Pod is Running and its image is pulled is a different fault. + +### Cause 3: the buildah Pod volume carries the `{selectedConnectors()}` placeholder + +A pipeline created in the UI uses the buildah task to build and push to Harbor. The Pod's CSI volume (driver `connectors.alauda.io`, attributes `configuration.names: registry-config`, `connectors: '{selectedConnectors()}'`) still carries the placeholder instead of the Connector selected in the UI, so the CSI driver looks up a Connector that does not exist. This is a known defect in Alauda DevOps Pipelines `tektoncd-operator.v4.8.0` (seen with ACP 4.2.0 and `connectors-operator.v1.9.0`), fixed in Alauda DevOps Pipelines 4.10. + +### Cause 4: the namespace enforces `restricted` and the Tekton containers do not meet it + +The namespace where the pipeline runs enforces the `restricted` Pod Security Standard, and the Tekton init containers (`prepare`, `place-scripts`) and step containers (`step-*`) do not set the required securityContext fields. The Pod is rejected at admission and never exists. + +The Tekton `set-security-context` feature flag is not sufficient on its own. In upstream Tekton the flag only gives a restricted-compatible securityContext to the containers that Tekton injects (init containers such as `prepare` and `place-scripts`, and the affinity assistant). `step-*` containers keep the securityContext that the Task defines, and a BuildKit step that requests `privileged: true` is still rejected by `restricted`. BuildKit build tasks require privileged containers; with privilege turned off the build fails with `operation not permitted`. + +## Resolution + +### Cause 1: gitclone Pod does not start because `connector-csi` is `OOMKilled` + +1. Confirm `OOMKilled` on `connector-csi` and find its owner: + + ```bash + kubectl get pods -A | grep connector-csi + kubectl -n get pod -o jsonpath='{.metadata.ownerReferences[0].kind}{" "}{.metadata.ownerReferences[0].name}{"\n"}{.status.containerStatuses[*].name}{" "}{.status.containerStatuses[*].lastState.terminated.reason}{"\n"}' + ``` + + `OOMKilled` in the output is required before the change. Note the owner kind, its name, and the container name, and record the current limits. + +2. Raise that container's memory limit on the owning workload. Doubling the current limit is a starting point. Keep the request less than or equal to the new limit. + + > **Warning:** `kubectl set resources` rolls the `connector-csi` Pods. New gitclone mounts fail until the new Pod is Ready; task Pods that already mounted are not affected. Precondition: only when the last terminated reason is `OOMKilled`. Rollback: set the memory limit back to the recorded value with the same command. If a controller reverts the change, the limit is owned by an operator resource or plugin configuration; set it there. + + ```bash + kubectl -n set resources / -c --limits=memory= --requests=memory= + kubectl -n rollout status / + ``` + + For example, when the old limit was `256Mi`: `--limits=memory=512Mi --requests=memory=256Mi`. + +3. Watch the new Pod. Another `OOMKilled` means the limit is still too low; raise it again. The gitclone Pod passes the mount and starts its container. + +### Cause 2: continuous build returns HTTP 500 because `katanomi-controller` cannot pull its image + +1. Find the controller Pod and read why the pull failed: + + ```bash + kubectl get pods -A | grep katanomi-controller + kubectl -n describe pod | sed -n '/Events/,$p' + ``` + + `ImagePullBackOff` or `ErrImagePull` on `katanomi-controller`, together with HTTP 500 from continuous build, is this cause. + +2. Fix the reason for the pull failure first (registry reachability, scheme, or pull rate), or the replacement Pod hits the same error. + +3. Recreate the Pod. + + > **Warning:** Deleting the `katanomi-controller` Pod stops continuous-build reconciliation until the replacement is Ready. Precondition: only when the Pod event is an image-pull failure and the registry path is restored. Rollback: none needed; the owner recreates the Pod from an unchanged spec. + + ```bash + kubectl -n delete pod + kubectl -n rollout status deploy/katanomi-controller + ``` + + If the owner is not a Deployment of that name, wait on the owner from `ownerReferences` instead. The new Pod is Running and continuous-build calls no longer return 500. + +### Cause 3: the buildah Pod volume carries the `{selectedConnectors()}` placeholder + +1. Confirm the placeholder in the Pod spec and record the operator versions: + + ```bash + kubectl -n describe pod | sed -n '/Events/,$p' + kubectl -n get pod -o yaml | grep -n 'selectedConnectors' + kubectl get csv -A | grep -Ei 'tektoncd-operator|connectors-operator' + ``` + + The literal `{selectedConnectors()}` in the volume attributes, with `tektoncd-operator` older than 4.10, is this cause. + +2. Download the Alauda DevOps Pipelines 4.10 package from the Alauda application marketplace (for example `tektoncd-operator.latest.amd64.v4.10.0.tgz`, listed as compatible with ACP 4.0 to 4.3), publish it to the cluster, and upgrade. + + > **Warning:** Upgrading Alauda DevOps Pipelines rolls the Tekton controllers and webhooks; PipelineRuns that start during the rollout can fail admission or wait. Precondition: only when the FailedMount names `{selectedConnectors()}` and the installed `tektoncd-operator` is older than 4.10. Upgrade in a quiet window and note the current CSV. Treat the upgrade as one-way: going back requires reinstalling the previous version. + +3. Run the pipeline again. The buildah Pod leaves `Init`, and the CSI volume names the real Connector. + +### Cause 4: the namespace enforces `restricted` and the Tekton containers do not meet it + +1. Read the namespace labels and the containers named in the admission error: + + ```bash + kubectl get ns --show-labels | tr ',' '\n' | grep pod-security + kubectl get taskrun -n -o jsonpath='{.status.conditions[0].message}{"\n"}' + ``` + +2. Run pipelines in a namespace that is used only for pipeline runs, and change only that namespace's container security policy (Pod Security Admission level) to privileged, from the namespace settings in the console or with the label. Leave the application namespaces at `restricted`, so user deployments stay constrained. + + > **Warning:** The `privileged` level lets any Pod in that namespace run privileged containers, mount host paths, and run as root. Precondition: only when the error is `violates PodSecurity "restricted:latest"` for Tekton containers such as `prepare`, `place-scripts`, or `step-*`, and the namespace is used for pipeline runs. Record the current labels. Rollback: set `pod-security.kubernetes.io/enforce` back to the recorded value. + + ```bash + kubectl label ns pod-security.kubernetes.io/enforce=privileged --overwrite + kubectl get ns -L pod-security.kubernetes.io/enforce + ``` + + The expected result is `privileged` on the pipeline namespace only and `restricted` on the application namespaces. + +3. Run the pipeline again in the pipeline namespace. The TaskRun Pod is created. If a deploy step then fails with `Forbidden` on an application namespace, grant the pipeline's ServiceAccount the permissions it needs in that namespace. + +To check whether the Tekton flag is set and which containers the error lists: + +```bash +kubectl get tektonpipelines.operator.tekton.dev -o yaml | grep -n 'set-security-context' +kubectl get cm -A | grep feature-flags +``` + +`step-*` containers in the message depend on the securityContext the Task defines, not on the flag. `privileged` listed for BuildKit steps means the namespace that runs them needs the privileged level. + +## Diagnostic Steps + +```bash +kubectl get pods -A | grep -Ei 'connector-csi|gitclone|katanomi-controller' +kubectl get events -n --sort-by=.lastTimestamp | grep -Ei 'OOMKilled|FailedMount|Forbidden|Failed' | tail -40 +``` + +- gitclone stuck mounting, and `connector-csi` last state `OOMKilled`: Cause 1. +- HTTP 500 from continuous build, and `katanomi-controller` in `ImagePullBackOff`: Cause 2. +- `FailedMount` naming `{selectedConnectors()}`: Cause 3. +- `violates PodSecurity "restricted:latest"` for `prepare`, `place-scripts`, or `step-*`: Cause 4. + +A Pending gitclone Pod whose event is `0/N nodes are available` is a scheduling problem, not CSI. A rejection that names the `katanomi-setting-init` init container is covered in "Tekton pipeline Pod rejected by Pod Security admission because of the katanomi-setting-init init container". BuildKit steps rejected for `privileged` are covered in "Image build fails with PodSecurity privileged, a RUN step failing on one node, or a buildah certificate error". diff --git a/docs/en/solutions/Toolchain_SSO_on_a_domain_name_after_an_IP-based_install_and_kube-apiserver_OIDC_for_an_external_UI_on_ACP.md b/docs/en/solutions/Toolchain_SSO_on_a_domain_name_after_an_IP-based_install_and_kube-apiserver_OIDC_for_an_external_UI_on_ACP.md new file mode 100644 index 000000000..1b4ae3f1e --- /dev/null +++ b/docs/en/solutions/Toolchain_SSO_on_a_domain_name_after_an_IP-based_install_and_kube-apiserver_OIDC_for_an_external_UI_on_ACP.md @@ -0,0 +1,89 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.1" + - "4.2" + - "4.3" + - "4.3.2" +--- + +# Toolchain SSO on a domain name after an IP-based install, and kube-apiserver OIDC for an external UI on ACP + +## Issue + +One of these is reported: + +- The platform was installed and is opened by IP address, and toolchain SSO (Harbor, GitLab, and similar) should use a domain name instead. Adding the DNS name alone does not switch SSO. +- Headlamp, or a similar cluster UI, needs kube-apiserver OIDC flags, and the concern is that the API would then accept only OIDC, which would break the ACP console, in-cluster components, and kubectl, and that an upgrade would remove the flags. + +Related articles: "OIDC clients and pages are sent to the wrong Dex issuer address on ACP", "Harbor SSO through the platform Dex fails: wrong issuer, OAuth2Client name, or Unregistered redirect_uri console-portal", and "GitLab or Argo CD login through the platform Dex fails: host required, certificate verify, invalid_client, LDAP groups". + +## Root Cause + +### Cause 1: a domain for toolchain SSO needs an access address, a certificate, and updated client callbacks + +Publishing a DNS name does not change anything on the platform by itself. For toolchain SSO to work on the domain, the domain has to be added as a platform access address, a certificate for that name has to be installed, and each toolchain OAuth2 client callback has to use the domain. A reinstall is normally not needed for this. These steps do not change the default access URL, and they do not change the issuer that Dex publishes. On ACP 4.3.2 the default platform access address and the internal registry address cannot be edited after installation; only additional access addresses can be added. Redeploying with the domain as the default is the only way to change the default itself, and it is not realistic once user clusters already run workloads. + +### Cause 2: kube-apiserver OIDC flags add OIDC authentication without turning off ServiceAccount or certificate authentication + +Adding the OIDC flags to the kube-apiserver static Pod manifest on every control-plane node enables OIDC login for the external UI. ServiceAccount and client-certificate authentication keep working, so the ACP console, in-cluster components, and kubectl are not affected. A platform upgrade does not revert this manifest edit, and the standard product supports it. Passing the same parameters through installer `extraArgs` has not been shown to work; edit the manifest instead. + +This is not how a third-party application obtains a token for the ACP API proxy: the platform API proxy accepts tokens from the `alauda-auth` client only. + +## Resolution + +### Cause 1: a domain for toolchain SSO needs an access address, a certificate, and updated client callbacks + +1. Publish DNS for the platform name. +2. Add the name as an additional platform access address. +3. Install a platform access certificate that contains the DNS name. +4. Edit each toolchain OAuth2 client and set its callback (redirect) URLs to `https:///...`: + + ```bash + kubectl get oauth2clients.dex.coreos.com -n cpaas-system -o yaml > /tmp/oauth2clients.bak.yaml + kubectl edit oauth2clients.dex.coreos.com -n cpaas-system + ``` + + > **Warning:** Changing redirect URLs breaks callbacks that still use the old string. Precondition: only after the domain resolves and serves a certificate that contains the name. Rollback: `kubectl apply -f /tmp/oauth2clients.bak.yaml`. + +If an OIDC client then reports that the issuer in the provider metadata does not match the issuer it was configured with, the discovery document still advertises the install-time IP; see "OIDC clients and pages are sent to the wrong Dex issuer address on ACP". + +### Cause 2: kube-apiserver OIDC flags add OIDC authentication without turning off ServiceAccount or certificate authentication + +Apply this only when an external UI must talk to kube-apiserver with OIDC, and a maintenance window covers a rolling kube-apiserver restart. + +1. On one control-plane node, back up the static Pod manifest and add the OIDC arguments, keeping every existing argument: + + ```yaml + - --authorization-mode=Node,RBAC + - --oidc-issuer-url=https:// + - --oidc-username-claim=email + - --oidc-client-id= + ``` + + > **Warning:** kubelet restarts kube-apiserver when the manifest is saved; API calls on that node fail until the Pod is Ready. Precondition: only when the issuer and client id are the identity provider the UI should trust, and the other control-plane nodes are Ready. Rollback: copy `/tmp/kube-apiserver.yaml.bak` back to `/etc/kubernetes/manifests/kube-apiserver.yaml`. An upgrade does not revert this file, so removing the flags is a manual restore from the backup. + + ```bash + sudo cp -a /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml.bak + sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml + kubectl get nodes + ``` + +2. Wait until the kube-apiserver Pod on that node is Ready, then repeat on the next control-plane node, one node at a time. + +3. After every control plane is Ready, confirm that an in-cluster ServiceAccount call and a kubectl call still succeed, then test the OIDC UI. + +## Diagnostic Steps + +```bash +curl -k -sS https:///dex/.well-known/openid-configuration | jq -r .issuer +openssl s_client -connect :443 -servername /dev/null | openssl x509 -noout -subject -ext subjectAltName +kubectl get oauth2clients.dex.coreos.com -n cpaas-system -o yaml | grep -n -i redirect +kubectl -n kube-system get pods -o wide | grep kube-apiserver +``` + +- The toolchains should use a new domain, the platform was installed on an IP, and the client callbacks or the certificate do not contain the domain: Cause 1. +- An external UI needs kube-apiserver OIDC, and the console must keep working: Cause 2. After the change, the kube-apiserver Pods are Running on every control-plane node and both kubectl and the OIDC UI work. diff --git a/docs/en/solutions/auth-controller2_webhook_certificate_missing_or_log_burst_monitoring_query_credentials_and_Connector_tokens_on_ACP.md b/docs/en/solutions/auth-controller2_webhook_certificate_missing_or_log_burst_monitoring_query_credentials_and_Connector_tokens_on_ACP.md new file mode 100644 index 000000000..11a41eb47 --- /dev/null +++ b/docs/en/solutions/auth-controller2_webhook_certificate_missing_or_log_burst_monitoring_query_credentials_and_Connector_tokens_on_ACP.md @@ -0,0 +1,155 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.2" + - "4.3.3" +--- + +# auth-controller2 webhook certificate missing or log burst, monitoring query credentials, and Connector tokens on ACP + +## Issue + +One of these is reported on the platform: + +- `auth-controller2` never starts. The Pod event is `MountVolume.SetUp failed for volume "cert": secret "auth-controller2-webhook-cert" not found`, and an upgrade of `3rdparty/chart-cert-manager` fails while creating a ResourceQuota at `metis-webhook.cpaas-system.svc:443/check-resourcequota` with `x509: certificate signed by unknown authority`. +- One daily `log-platform` index is suddenly many times larger than its neighbours (about a terabyte or more), and the question is which component produced it, or whether `auth-controller2` logs surge and the controller will crash while administrators are adding permissions for users. +- An external caller needs a credential for a Prometheus or VictoriaMetrics query, or the API token page shows `Permissions required. Please contact your administrator.` +- `kube-prometheus-exporter-node` uses a different username on some clusters. +- A Kubernetes-type Connector shows authentication failed. + +## Root Cause + +### Cause 1: `auth-controller2-webhook-cert` is not issued because a second cert-manager produces conflicting CertificateRequests + +The certificate volume of `auth-controller2` is not optional (`optional: false`), so the Pod cannot start without the Secret `auth-controller2-webhook-cert` in `cpaas-system`. When a non-platform cert-manager and the platform cert-manager are both present, more than one `CertificateRequest` exists for the certificate, and admission to the `metis-validation` webhook fails with `x509: certificate signed by unknown authority` (also logged as `tls: failed to verify certificate: x509: certificate signed by unknown authority`). The webhook Secret comes back after `metis-validation` is set to ignore failures, the extra `CertificateRequest` objects are removed (one is kept), and cert-manager is restarted. This is not the case where the metis Pod is down and does not answer. + +### Cause 2: a one-time `auth-controller2` log burst while user permissions are being added + +A daily `log-platform` index can grow far beyond its neighbours when `auth-controller2` writes a burst of log lines while operations staff add permissions for users. In one such case the large day held about 62 million documents and about 2.1 TB, against about 18 to 39 million documents and 8 to 23 GB on neighbouring days. A product-log query on the two `auth-controller2` Pods returned about 13 million entries, about 10 million of them in a single interval, with the surrounding intervals near zero. During the burst, container `auth-controller2-manager` rose from about 750 Mi to about 1.1 Gi and returned to baseline, far below its 6144 Mi limit, so the Pod was not near an OOM kill; a "utilization from requests" figure above 1000 % only reflects its small memory request. No crash occurred, the burst did not repeat, and later daily indices returned to normal size. Why the permission changes produced that many lines was not determined. + +### Cause 3: external Prometheus or VictoriaMetrics queries need the matching credential + +Platform API queries use a platform API token. Native Prometheus queries use the basic-auth Secret `kube-prometheus-prometheus-basic-auth` in `cpaas-system`. Native VictoriaMetrics queries use `victoriametrics-basic-auth`. The API token page hides token creation from users who lack that page permission, which produces the `Permissions required` message; the monitoring endpoints below are an alternative for a monitoring caller that reads every cluster. + +### Cause 4: the `kube-prometheus-exporter-node` credential changed in 4.2 + +`kube-prometheus-exporter-node` does not use the same username and password on every cluster. Before 4.2 it uses a fixed credential. From 4.2 the username is `monitor@cpaas.io` and the password is a random string. The difference is a security policy change in 4.2, not a cluster that lost its configuration. + +### Cause 5: a Kubernetes Connector uses an endpoint that does not match its token + +The Connector object is created and then shows authentication failed. The usual mismatches are: the endpoint is a cluster API server (`https://192.0.2.10:6443`) but the token is a platform API token, or the endpoint is the ACP platform address but the token is a cluster ServiceAccount token; the Secret that was inspected is not the Secret named on the Connector; or `secretRef` is not under `spec.auth` with `bearerTokenAuth`. A token created with `kubectl create token` is valid for one hour unless the API server lifetime flags say otherwise. + +## Resolution + +### Cause 1: `auth-controller2-webhook-cert` is not issued because a second cert-manager produces conflicting CertificateRequests + +1. Confirm that the Secret is absent and the Pod event names it: + + ```bash + kubectl -n cpaas-system get secret auth-controller2-webhook-cert + kubectl -n cpaas-system describe pod + kubectl get validatingwebhookconfiguration,mutatingwebhookconfiguration -o name | grep metis-validation + kubectl get certificaterequest.cert-manager.io -A + ``` + +2. Set `failurePolicy: Ignore` on the `metis-validation` webhook entries (at least the one whose service is `metis-webhook` and path `/check-resourcequota`). + + > **Warning:** `failurePolicy: Ignore` lets requests through while the webhook certificate is untrusted, including requests the webhook would have rejected. Precondition: only when `auth-controller2-webhook-cert` is missing and admission to `metis-webhook.cpaas-system.svc:443/check-resourcequota` fails with `x509: certificate signed by unknown authority`. Rollback: `kubectl apply -f /tmp/metis-validation.bak.yaml`. + + ```bash + kubectl get validatingwebhookconfiguration metis-validation -o yaml > /tmp/metis-validation.bak.yaml + kubectl edit validatingwebhookconfiguration metis-validation + ``` + +3. Keep one `CertificateRequest`, delete the extra ones created by the non-platform cert-manager, and restart cert-manager. + + > **Warning:** Deleting a `CertificateRequest` drops that issuance. Precondition: only when both cert-managers have produced multiple `CertificateRequest` objects for this certificate and the Secret is still missing. Save the YAML first. Re-applying a deleted `CertificateRequest` retries issuance. + + ```bash + kubectl get certificaterequest.cert-manager.io -A -o yaml > /tmp/cr.bak.yaml + kubectl delete certificaterequest.cert-manager.io -n + kubectl get deploy -A | grep cert-manager + kubectl -n rollout restart deploy/ + kubectl -n cpaas-system get secret auth-controller2-webhook-cert + ``` + +4. `auth-controller2` starts after the Secret exists. Once the webhook certificate verifies (the `x509` error is gone), restore the previous `failurePolicy` of `metis-validation` from `/tmp/metis-validation.bak.yaml`. + +Do not install a second cert-manager next to the platform cert-manager. + +### Cause 2: a one-time `auth-controller2` log burst while user permissions are being added + +1. Find the source of the volume before treating it as business logs. In the Operations Center, open **Logs > Query**, select product logs, narrow the time range to the large day, filter by the `auth-controller2` Pod names, and look at the histogram for a single spike. + +2. Check the Pod's memory in monitoring for the same interval, and its current state: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep auth-controller2 + kubectl -n cpaas-system get pod -o jsonpath='{range .status.containerStatuses[*]}{.name}{" restarts="}{.restartCount}{" lastState="}{.lastState}{"\n"}{end}' + kubectl -n cpaas-system top pod --containers + ``` + + A single spike at the time permissions were being granted, memory back at baseline, and no restart or `OOMKilled` last state need no action. + +3. If the burst repeats, collect a sample of the repeated lines while it happens, together with the permission operations being performed, for the support team: + + ```bash + kubectl -n cpaas-system logs -c auth-controller2-manager --since=10m > /tmp/auth-controller2-burst.log + wc -l /tmp/auth-controller2-burst.log + ``` + +### Cause 3: external Prometheus or VictoriaMetrics queries need the matching credential + +Platform API, with a platform API token: + +```text +https:///v2/metrics//prometheus/query +``` + +Native Prometheus, with the username and password from the Secret: + +```bash +kubectl get secret -n cpaas-system kube-prometheus-prometheus-basic-auth -o jsonpath='{.data.username}' | base64 -d; echo +kubectl get secret -n cpaas-system kube-prometheus-prometheus-basic-auth -o jsonpath='{.data.password}' | base64 -d; echo +``` + +```text +https:///clusters//prometheus-0/api/v1/query +``` + +VictoriaMetrics uses the Secret `victoriametrics-basic-auth` and paths of the form `/clusters/global/vmselect/api/v1/query` and `/api/v1/query_range` under the same prefix. Replacing `vmselect` in that path with `prometheus-0` gives the native Prometheus form; the query APIs are the same. + +### Cause 4: the `kube-prometheus-exporter-node` credential changed in 4.2 + +Read the credential that the cluster actually uses and compare it with the cluster version. On 4.2 and later, expect username `monitor@cpaas.io` and a random password. No reset is needed to make clusters of different versions "match". + +### Cause 5: a Kubernetes Connector uses an endpoint that does not match its token + +1. Match the endpoint and the token. A cluster API server address takes a bearer token from that cluster. An ACP platform address takes a platform API token. The Secret name and namespace on the Connector must be the Secret you filled in. + +2. To use a ServiceAccount token for a cluster API server endpoint, create one for the ServiceAccount the Connector should act as, store it in the Secret key `token`, and reference the Secret under `spec.auth.bearerTokenAuth.secretRef` (name and namespace). + + > **Warning:** Anyone who can read the Connector Secret can call the API as that ServiceAccount until the token expires (one hour by default). Precondition: only when the Connector endpoint and this ServiceAccount are the pair that should authenticate. Rollback: replace the `token` value or delete the key; a token already issued stays valid until it expires. The API server flags `--service-account-extend-token-expiration` and `--service-account-max-token-expiration` affect the lifetime of tokens issued later. + + ```bash + kubectl -n create token + ``` + +3. After the endpoint, the token type, and `secretRef` agree, the Connector authenticates and can be used from a pipeline Task. + +## Diagnostic Steps + +```bash +kubectl -n cpaas-system get pods | grep auth-controller +kubectl -n cpaas-system get secret auth-controller2-webhook-cert +kubectl -n cpaas-system get secret kube-prometheus-prometheus-basic-auth +``` + +- Event `secret "auth-controller2-webhook-cert" not found`, a `metis-validation` `x509` error, a second cert-manager, and more than one `CertificateRequest`: Cause 1. If the metis Pod itself is not Running, the webhook does not answer at all, which is a different problem. +- One daily `log-platform` index many times larger than its neighbours, and a product-log query showing a single `auth-controller2` spike at the time user permissions were added, with no restart: Cause 2. +- An external metrics query needs a credential: Cause 3. Exporter username differs by cluster version: Cause 4. +- Connector authentication failed: Cause 5. diff --git a/docs/en/solutions/ecosystem/nacos/How_to_Deploy_Nacos_2.2.md b/docs/en/solutions/ecosystem/nacos/How_to_Deploy_Nacos_2.2.md index 1a54336bb..08040e4a8 100644 --- a/docs/en/solutions/ecosystem/nacos/How_to_Deploy_Nacos_2.2.md +++ b/docs/en/solutions/ecosystem/nacos/How_to_Deploy_Nacos_2.2.md @@ -288,3 +288,14 @@ TopoLVM volumes are local to one node, so a Nacos Pod cannot move to another nod ``` 5. Repeat steps 3 and 4 for the next member until every member runs on a new node. Then remove the taints from the old nodes or take the nodes out of service. + +### Q10. Can workloads in another cluster reach this Nacos through its `svc.cluster.local` names? + +No. The in-cluster names in `NACOS_SERVERS` (`nacos-server-.nacos-server..svc.cluster.local:8848`) resolve only inside the cluster where Nacos runs, and the multi-cluster Headless Service domain names (`.amcs-..svc.cluster.local`) are not a supported way to reach Nacos. Cross-cluster access to Nacos is supported only by exposing the Nacos service outside its cluster through `NodePort` or `LoadBalancer`, as described under Architecture Overview, and pointing the clients in the other cluster at that address. Expose the gRPC port as well, and keep it at exactly the HTTP port plus 1000 on the outside (for example `8848` and `9848`). + +```bash +kubectl -n get svc -o wide +kubectl -n get sts -o jsonpath='{range .spec.template.spec.containers[0].env[*]}{.name}{"="}{.value}{"\n"}{end}' | grep -E 'NACOS_SERVERS|MODE|PREFER_HOST_MODE' +``` + +Only `ClusterIP` or headless Services in the output means Nacos is not yet reachable from another cluster. From 4dd3406bb3a525c5645756a9e57decda7be697fa Mon Sep 17 00:00:00 2001 From: Komh Date: Thu, 12 Nov 2026 21:32:07 +0800 Subject: [PATCH 09/25] docs(solutions): add 20 and update 0 troubleshooting articles --- ...g_missing_tables_or_broken_parts_on_ACP.md | 263 ++++++++++++++++++ ...pikes_or_a_hand-built_ClickHouse_on_ACP.md | 165 +++++++++++ ...ils_or_ClickHouse_does_not_start_on_ACP.md | 167 +++++++++++ ...after_editing_the_index_template_on_ACP.md | 89 ++++++ ...or_yellow_with_unassigned_shards_on_ACP.md | 254 +++++++++++++++++ ...es_logs_after_the_log_disk_fills_on_ACP.md | 156 +++++++++++ ...imits_log_backlog_or_WebCLI_role_on_ACP.md | 175 ++++++++++++ ...zer_errors_or_too_few_partitions_on_ACP.md | 207 ++++++++++++++ ...ed_log_entries_or_a_stuck_offset_on_ACP.md | 149 ++++++++++ ...ranges_or_an_empty_real-time_log_on_ACP.md | 192 +++++++++++++ ...eywords_clock_skew_or_page_scope_on_ACP.md | 177 ++++++++++++ ...sk_CPU_or_memory_or_is_OOMKilled_on_ACP.md | 244 ++++++++++++++++ ..._while_the_log_Kafka_is_degraded_on_ACP.md | 214 ++++++++++++++ ...lled_or_Prometheus_stuck_in_Init_on_ACP.md | 161 +++++++++++ ...nal_Kafka_or_OpenTelemetry_fails_on_ACP.md | 240 ++++++++++++++++ ..._conflict_or_deleted_StatefulSet_on_ACP.md | 246 ++++++++++++++++ ...or_callback-secret_mount_failure_on_ACP.md | 207 ++++++++++++++ ...illed_after_a_disk_queue_backlog_on_ACP.md | 160 +++++++++++ ...rt_or_ignore_new_resource_values_on_ACP.md | 249 +++++++++++++++++ ...e_log_storage_install_or_upgrade_on_ACP.md | 225 +++++++++++++++ 20 files changed, 3940 insertions(+) create mode 100644 docs/en/solutions/ClickHouse_log_queries_fail_after_the_log_storage_had_been_working_missing_tables_or_broken_parts_on_ACP.md create mode 100644 docs/en/solutions/ClickHouse_log_storage_logs_RaftInstance_errors_with_pod_restarts_merge_CPU_spikes_or_a_hand-built_ClickHouse_on_ACP.md create mode 100644 docs/en/solutions/ClickHouse_log_storage_plugin_install_fails_or_ClickHouse_does_not_start_on_ACP.md create mode 100644 docs/en/solutions/Elasticsearch_log_index_replica_count_reverts_after_editing_the_index_template_on_ACP.md create mode 100644 docs/en/solutions/Elasticsearch_log_indices_red_or_yellow_with_unassigned_shards_on_ACP.md create mode 100644 docs/en/solutions/Elasticsearch_log_storage_Pending_or_every_cluster_loses_logs_after_the_log_disk_fills_on_ACP.md create mode 100644 docs/en/solutions/Inspection_faults_x86-64-v2_CPU_error_discover-device_crash_loop_small_ClickHouse_limits_log_backlog_or_WebCLI_role_on_ACP.md create mode 100644 docs/en/solutions/Log_Kafka_lag_from_Elasticsearch_write_hot_spots_breaker_trips_IK_analyzer_errors_or_too_few_partitions_on_ACP.md create mode 100644 docs/en/solutions/Log_Kafka_lag_on_a_single_partition_from_oversized_log_entries_or_a_stuck_offset_on_ACP.md create mode 100644 docs/en/solutions/Log_query_shows_no_logs_HTTP_500_slow_long_ranges_or_an_empty_real-time_log_on_ACP.md create mode 100644 docs/en/solutions/Log_search_misses_entries_IK_analyzer_tokens_uppercase_keywords_clock_skew_or_page_scope_on_ACP.md create mode 100644 docs/en/solutions/Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md create mode 100644 docs/en/solutions/Logs_audit_and_events_stop_while_the_log_Kafka_is_degraded_on_ACP.md create mode 100644 docs/en/solutions/Monitoring_faults_a_second_Prometheus_missing_vmagent-agent_vmalert_OOMKilled_or_Prometheus_stuck_in_Init_on_ACP.md create mode 100644 docs/en/solutions/Vector_log_forwarding_to_external_Kafka_or_OpenTelemetry_fails_on_ACP.md create mode 100644 docs/en/solutions/cpaas-kafka_broker_fails_to_start_corrupt_segment_no_space_broker_id_conflict_or_deleted_StatefulSet_on_ACP.md create mode 100644 docs/en/solutions/nevermore_OOMKilled_cache_written_to_var_cpaas_or_callback-secret_mount_failure_on_ACP.md create mode 100644 docs/en/solutions/nevermore_Pending_Error_with_filebeat_lock_or_OOMKilled_after_a_disk_queue_backlog_on_ACP.md create mode 100644 docs/en/solutions/nevermore_and_ClickHouse_log_components_alert_on_CPU_or_memory_restart_or_ignore_new_resource_values_on_ACP.md create mode 100644 docs/en/solutions/razor_does_not_become_ready_after_a_ClickHouse_log_storage_install_or_upgrade_on_ACP.md diff --git a/docs/en/solutions/ClickHouse_log_queries_fail_after_the_log_storage_had_been_working_missing_tables_or_broken_parts_on_ACP.md b/docs/en/solutions/ClickHouse_log_queries_fail_after_the_log_storage_had_been_working_missing_tables_or_broken_parts_on_ACP.md new file mode 100644 index 000000000..caf0ad569 --- /dev/null +++ b/docs/en/solutions/ClickHouse_log_queries_fail_after_the_log_storage_had_been_working_missing_tables_or_broken_parts_on_ACP.md @@ -0,0 +1,263 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.2.x + - 4.3.x +--- + +# ClickHouse log queries fail after the log storage had been working: missing tables or broken parts on ACP + +## Issue + +The ClickHouse log storage was working, and now log queries fail or return nothing. Typical reports: + +- "The last 12 hours of logs cannot be queried, but older logs still can." +- "After the ClickHouse cluster came back from exit code 137, nothing can be found." +- "`razor-0` is abnormal on the global cluster." +- "Audit and Operations Center logs are not displayed." + +The log store runs in `cpaas-system` as pods `chi-cpaas-clickhouse-replicated---0`, created from the ClickHouseInstallation `cpaas-clickhouse`, with database `observability`. `razor` is the gateway in front of it. + +The 3-replica layout uses `ReplicatedMergeTree` tables, coordinated by ClickHouse Keeper inside the ClickHouse container. There is no separate ZooKeeper Deployment, although the configuration refers to ZooKeeper. Replication is asynchronous and multi-master, so every replica accepts writes and syncs to the others. PVC usage that differs between replicas by tens of gigabytes is normal, because merges, replication lag, temporary parts and TTL cleanup differ per replica. If the same query returns the same rows on each replica, the data is consistent. A size difference is a fault only together with other symptoms. + +The shared tables include `audit`, `event`, `log_kubernetes`, `log_platform`, `log_system` and `log_workload`. Inserts go through Distributed tables such as `audit_all` and `event_all`. `SHOW TABLES FROM observability` on a healthy replica lists them all. + +Installs that never worked are covered in [ClickHouse log storage plugin install fails or ClickHouse does not start on ACP](./ClickHouse_log_storage_plugin_install_fails_or_ClickHouse_does_not_start_on_ACP.md). Out-of-memory kills and capacity problems are covered in [Logging ClickHouse runs out of disk, CPU or memory, or is OOMKilled on ACP](./Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md). + +## Root Cause + +### Cause 1: Part of the queries fail with UNKNOWN_TABLE because a replica lost its schema but is still registered + +Logs from the last hours fail to load while older ones work, and the error is `UNKNOWN_TABLE`. One replica (for example `-1-0-0`) restarted and lost its local `observability` schema. Its registration in Keeper was never removed. The replica remains an empty shell that still receives traffic, so every query routed to it fails (about one query in six in a 3-replica layout). + +### Cause 2: After recovery from exit code 137, logs cannot be found and table log_kubernetes is missing on some replicas + +The ClickHouse cluster failed with exit code 137. After its resources were raised (to 16 CPU / 16Gi) it came back, but audit, product logs and Kubernetes logs return nothing. The ClickHouse log says table `log_kubernetes` cannot be found. Tables are missing on some replicas (for example `0-0-0` and `0-2-0`) and complete on another (`0-1-0`). + +On an affected replica, `SHOW TABLES FROM observability` lists the Distributed table `log_kubernetes_all` but not the local table `log_kubernetes`. The client may also print `Effective user of the process (clickhouse) does not match the owner of the data (root).` Why the tables went missing has not been determined. Reinstalling ClickHouse with an emptied data directory restores the tables and the queries. The data directory must be cleared before the reinstall. + +### Cause 3: razor fails to initialize because ClickHouse refuses to load observability.log_workload with too many suspicious broken parts + +`razor-0` is abnormal in the console. Its events show initialization failing because ClickHouse does not load table `observability.log_workload`. The ClickHouse log reports suspicious broken parts (for example about 20 parts, 8.11 GiB in total). That is more than ClickHouse handles automatically: `merge_tree/max_suspicious_broken_parts_bytes` defaults to 1 GiB. Disk space is sufficient. + +Parts are typically broken by an abnormal node or ClickHouse restart, an interrupted merge, or a transient storage I/O error. Raising the threshold lets the table load and set the broken parts aside. Afterwards the setting can go back to the default without data loss in the loaded table. Rows in the parts that were set aside can be lost. + +### Cause 4: The ClickHouse disk filled, vector inserts failed, and after space returned ClickHouse is missing metadata files + +The log nodes ran out of disk. The `vector` container in `razor` reported, for the `event`, `audit` and `log_system` sinks: + +```text +Code: 243. DB::Exception: Cannot reserve 1.00 MiB, not enough space. (NOT_ENOUGH_SPACE) +``` + +`nevermore` then repeated `write error: failed to write to file: write /cpaas/log/logs/filebeat--1.ndjson: no space left on device`, and logs stopped arriving. Disk usage later dropped, but the components did not recover by themselves, and a `razor` restart failed: + +- The `razor` container logged `http: proxy error: dial tcp [::1]:8081: connect: connection refused` and `Remove table ttl error ... Code: 159. DB::Exception: Timeout exceeded ... (TIMEOUT_EXCEEDED)`. +- The previous `vector` log showed `vector_buffers::variants::disk_v2::writer: Last written record was unable to be deserialized. Corruption likely.` and repeated `Events dropped intentionally ... reason="Source send cancelled."`. It ended with `run "/usr/local/bin/vector" error: signal: killed`. +- ClickHouse was unhealthy, with metadata files missing that could not be recovered. + +At that point both the ClickHouse data and the `vector` disk buffer are corrupted. The recovery is a reinstall with both data directories cleared on every log node. + +### Cause 5: The clickhouse container fails its liveness probe on 8123 for days, and clickhouse-operator is in CrashLoopBackOff + +Audit logs and Operations Center logs are empty, and the monitoring dashboard shows a log collection rate of 0. `chi-cpaas-clickhouse-replicated-0-0-0` is `Running` with thousands of restarts. Its events repeat: + +```text +Container clickhouse failed liveness probe, will be restarted +Liveness probe failed: Get "http://:8123/ping": dial tcp :8123: connect: connection refused +``` + +On the global cluster, `clickhouse-operator` in `ck-system` is in `CrashLoopBackOff`, and the data disk has free space. The ClickHouse database has been broken for a long time. + +A liveness-probe kill can also show as exit code `137` with `Reason: Error`. That signature is this cause when the events show the `8123` probe failing and the operator is crash-looping. When the reason is `OOMKilled`, or there are no probe failures, see the capacity article instead. The recommended fix is to uninstall the ClickHouse log plugin, empty the logging ClickHouse data directory on the ClickHouse nodes, and redeploy. + +## Resolution + +Open a client on a ClickHouse pod for the SQL below: + +```bash +kubectl -n cpaas-system get pods | grep chi-cpaas-clickhouse +kubectl -n cpaas-system exec -it -c clickhouse -- clickhouse-client +``` + +### Cause 1: Part of the queries fail with UNKNOWN_TABLE because a replica lost its schema but is still registered + +1. Compare the schema on each replica: + + ```sql + SELECT name FROM system.tables WHERE database = 'observability' ORDER BY name; + SELECT database, table, replica_name, is_readonly, active_replicas, total_replicas + FROM system.replicas WHERE database = 'observability'; + ``` + + The ghost is the replica that lists no `observability` tables while the other replicas still count it in `total_replicas`. + +2. Remove the ghost entry from Keeper. + + > **Warning:** `SYSTEM DROP REPLICA` removes that replica's metadata from Keeper for the named table. If you name a replica that is actually healthy, it stops replicating and must be re-created. Precondition: only when `UNKNOWN_TABLE` appears on a share of the queries and exactly one replica has no `observability` tables. Run the statement on a healthy replica, naming the empty one. Rollback: none for the Keeper entry; the replica registers again when its tables are created. + + ```sql + SYSTEM DROP REPLICA '' FROM TABLE observability.log_workload; + ``` + + Repeat it for each replicated table listed in `system.replicas`. + +3. Rebuild the tables on the empty replica. On a healthy replica, list the tables and print each definition: + + ```sql + SHOW TABLES FROM observability; + SHOW CREATE TABLE observability.
; + ``` + + Run each `CREATE TABLE` statement on the empty replica. If the database is gone, run `CREATE DATABASE IF NOT EXISTS observability` first. Create the replicated tables first and the Distributed tables after. The replicated tables then fetch their parts from the other replicas. + +4. Query the same time range several times. The `UNKNOWN_TABLE` errors should stop completely. + +### Cause 2: After recovery from exit code 137, logs cannot be found and table log_kubernetes is missing on some replicas + +1. Compare the table count per replica: + + ```sql + SELECT hostName(), count() FROM clusterAllReplicas('replicated', system.tables) + WHERE database = 'observability' GROUP BY 1; + ``` + + Fewer tables on some replicas, together with `log_kubernetes` not found in the log, matches this cause. + +2. Reinstall with an empty data directory. + + > **Warning:** Emptying the ClickHouse data directory and reinstalling deletes all stored logs, audit records and events. Precondition: only when this cause is confirmed and losing the stored history is accepted. The ClickHouse pods must be stopped (the plugin uninstalled) before the delete. Rollback: none. + + Uninstall the ClickHouse log storage plugin. Then, on every ClickHouse node, empty the data directory (`/cpaas/data/clickhouse` by default): + + ```bash + cd /cpaas/data/clickhouse && rm -rf ./* + ``` + +3. Reinstall the plugin with resources large enough to avoid another exit code 137. Size them from the observed usage and the free memory on the nodes. + +### Cause 3: razor fails to initialize because ClickHouse refuses to load observability.log_workload with too many suspicious broken parts + +1. Confirm the signature: + + ```bash + kubectl -n cpaas-system describe pod razor-0 | sed -n '/Events:/,$p' + kubectl -n cpaas-system logs -c clickhouse --tail=200 | grep -i 'suspicious' + ``` + +2. Raise the threshold. + + > **Warning:** A higher `max_suspicious_broken_parts_bytes` lets ClickHouse load the table and set the broken parts aside, so rows in those parts can be lost. The edit restarts all ClickHouse pods. Precondition: only when the log shows `log_workload` refused because the suspicious broken parts exceed the limit, disk space is sufficient, and the owner accepts possible log loss (and a reinstall if this does not work). Rollback: remove the setting again (step 4). + + ```bash + kubectl -n cpaas-system get clickhouseinstallations.clickhouse.altinity.com cpaas-clickhouse -o yaml > /tmp/chi.before.yaml + kubectl -n cpaas-system edit clickhouseinstallations.clickhouse.altinity.com cpaas-clickhouse + ``` + + Under `spec.configuration.settings`, add a value above the broken size (10 GiB here): + + ```yaml + merge_tree/max_suspicious_broken_parts_bytes: "10737418240" + ``` + +3. After the pods restart, check the setting and the parts that were set aside: + + ```sql + SELECT name, value, changed FROM system.merge_tree_settings WHERE name LIKE '%suspicious%'; + SELECT database, table, name, reason FROM system.detached_parts WHERE database = 'observability'; + ``` + +4. When `razor` is Ready and the tables answer, restore the default. The edit created a generated resource patch (`rpch`). Deleting that patch returns the setting to 1 GiB, and the pods rebuild once more: + + ```bash + kubectl get rpch -A | grep -i clickhouse + kubectl get rpch -o yaml > /tmp/clickhouse-rpch.yaml + kubectl delete rpch + ``` + + Confirm that `log_workload` still loads. + +### Cause 4: The ClickHouse disk filled, vector inserts failed, and after space returned ClickHouse is missing metadata files + +1. Confirm the signature: + + ```bash + df -hT /cpaas + kubectl -n cpaas-system logs razor-0 -c vector --tail=100 | grep -iE 'NOT_ENOUGH_SPACE|insert|disk|space' + kubectl -n cpaas-system logs razor-0 -c vector -p --tail=100 | grep -E 'Corruption likely|signal: killed' + kubectl -n cpaas-system logs -c clickhouse --tail=100 | grep -iE 'metadata|No such file' + ``` + +2. Free the disk permanently first. Otherwise the same sequence repeats. If `du` on the log node shows the Kafka directory (`/cpaas/data/kafka0`) as the large one, see [Logs, audit and events stop while the log Kafka is degraded on ACP](./Logs_audit_and_events_stop_while_the_log_Kafka_is_degraded_on_ACP.md). + +3. Reinstall with both data directories emptied. + + > **Warning:** Reinstalling ClickHouse and emptying its data directories removes all stored logs. Precondition: only when the disk is no longer full, `razor` still cannot start, and ClickHouse reports missing metadata files (or the `vector` disk buffer reports `Corruption likely`). Confirm the paths against the pod volume mounts on each log node. Rollback: none. + + Uninstall the ClickHouse plugin. Then, on every log node: + + ```bash + cd /cpaas/data/clickhouse && rm -rf ./* + cd /cpaas/data/vector && rm -rf ./* + ``` + +4. Reinstall the plugin, and give the `vector` container in `razor` enough memory. `razor` and log query should come back. + +This every-node cleanup applies only when ClickHouse is missing metadata files and `razor` cannot start. A single filled node, with read-only audit tables while the other replicas are healthy, is handled differently: see the capacity article. + +### Cause 5: The clickhouse container fails its liveness probe on 8123 for days, and clickhouse-operator is in CrashLoopBackOff + +1. Confirm the signature: + + ```bash + kubectl get pod -A | grep -E 'nevermore|razor|clickhouse' + kubectl -n cpaas-system describe pod chi-cpaas-clickhouse-replicated-0-0-0 | sed -n '/Events:/,$p' + kubectl -n ck-system get pods + ``` + +2. Find the host path that backs the logging ClickHouse data volume. The default is `/cpaas/data/clickhouse`. Some installations use `/cpaas/ck`, which on other clusters belongs to the MySQL slow-query ClickHouse (`chi-pxc-ck-slowsql-cluster`). Check which pod owns the path before you empty it: + + ```bash + kubectl -n cpaas-system get pod chi-cpaas-clickhouse-replicated-0-0-0 -o jsonpath='{range .spec.volumes[*]}{.name}{" "}{.hostPath.path}{.persistentVolumeClaim.claimName}{"\n"}{end}' + findmnt -T + ``` + +3. Uninstall, empty the data directory, and redeploy. + + > **Warning:** Uninstalling the ClickHouse log plugin and emptying its data directory deletes the stored logs on those nodes. Precondition: only when the liveness probe on 8123 has failed for a long time, the restarts number in the thousands, free space is not the problem, and step 2 shows that the path belongs to the logging ClickHouse pods and not to `chi-pxc-ck-slowsql-cluster`. Rollback: none. + + Uninstall the ClickHouse log plugin, empty the data directory found in step 2 on each ClickHouse node, and redeploy the plugin. + +4. Check that `clickhouse-operator` is `Running` and that the log collection rate on the dashboard is above 0. + +## Diagnostic Steps + +1. Read the error the console or `razor` reports: + + ```bash + kubectl -n cpaas-system get pods | grep -Ei 'chi-|razor' + kubectl -n cpaas-system logs razor-0 -c razor --tail=50 + kubectl -n cpaas-system describe pod razor-0 | sed -n '/Events:/,$p' + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | `UNKNOWN_TABLE` on some of the requests | Cause 1 | + | A table not found on some replicas after a recovery from exit code 137 | Cause 2 | + | `log_workload` refused because of suspicious broken parts | Cause 3 | + | `NOT_ENOUGH_SPACE` from `vector`, then missing metadata files or a corrupted `vector` buffer after the disk was freed | Cause 4 | + | Thousands of liveness-probe restarts on 8123, with `clickhouse-operator` crash-looping | Cause 5 | + +3. Exit code 137 without liveness-probe failures on `8123`, or `OOMKilled`, is a memory problem. See [Logging ClickHouse runs out of disk, CPU or memory, or is OOMKilled on ACP](./Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md). + +## Related Information + +- [ClickHouse log storage plugin install fails or ClickHouse does not start on ACP](./ClickHouse_log_storage_plugin_install_fails_or_ClickHouse_does_not_start_on_ACP.md) +- [razor does not become ready after a ClickHouse log storage install or upgrade on ACP](./razor_does_not_become_ready_after_a_ClickHouse_log_storage_install_or_upgrade_on_ACP.md) +- [Logging ClickHouse runs out of disk, CPU or memory, or is OOMKilled on ACP](./Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md) diff --git a/docs/en/solutions/ClickHouse_log_storage_logs_RaftInstance_errors_with_pod_restarts_merge_CPU_spikes_or_a_hand-built_ClickHouse_on_ACP.md b/docs/en/solutions/ClickHouse_log_storage_logs_RaftInstance_errors_with_pod_restarts_merge_CPU_spikes_or_a_hand-built_ClickHouse_on_ACP.md new file mode 100644 index 000000000..779e976e5 --- /dev/null +++ b/docs/en/solutions/ClickHouse_log_storage_logs_RaftInstance_errors_with_pod_restarts_merge_CPU_spikes_or_a_hand-built_ClickHouse_on_ACP.md @@ -0,0 +1,165 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# ClickHouse log storage logs RaftInstance errors with pod restarts, merge CPU spikes, or a hand-built ClickHouse on ACP + +## Issue + +The ClickHouse log storage (Alauda Container Platform Log Storage for ClickHouse) runs in `cpaas-system` as pods `chi-cpaas-clickhouse-replicated---0` with container `clickhouse`. ClickHouse Keeper runs inside the same container, and its Raft traffic between replicas uses port `9444`. The `clickhouse` container log contains `RaftInstance` errors such as: + +```text + RaftInstance: session 13 failed to read rpc header from socket ... due to error 2, End of file +``` + +Typical questions are: + +- "ClickHouse keeps printing error logs but nothing seems broken. Is this normal?" +- "ClickHouse keeps restarting after the platform upgrade." +- "A `chi-cpaas-clickhouse-replicated` pod has high CPU and its log says `failed to read rpc header from socket`." +- "We deployed our own ClickHouse, switched the log collector to it, and log query is unavailable." + +Sometimes this line is the only symptom: no restarts, and log query and ingestion work. In that case it is benign reconnect noise (Cause 4). The line does not identify a fault by itself, because it also appears next to unrelated faults. Use the other signatures below to find the real cause. + +## Root Cause + +### Cause 1: After an upgrade, the pods restart repeatedly with Election timeout and pre-vote failures while CPU sits at its limit + +After the platform was upgraded (for example from 4.1.3 to 4.2.1), the logging ClickHouse on the `global` cluster kept restarting on all three replicas, with limits of 2 cores / 4Gi per pod. The previous `clickhouse` container log repeats: + +```text + RaftInstance: Election timeout, initiate leader election +RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 16750, live 0, dead 1), at least 2 nodes should respond +failed to send prevote request: peer 3 (chi-cpaas-clickhouse-replicated-0-2.cpaas-system.svc.cluster.local:9444) is busy +peer (1) response error: failed to connect to peer 1, chi-cpaas-clickhouse-replicated-0-0.cpaas-system.svc.cluster.local:9444 +``` + +The `log/Cpaas Log Center for Clickhouse` dashboard shows CPU at 100% of the 2-core limit again and again, and memory at about 20% or less. ClickHouse needs more CPU during startup after the upgrade than the limit allows. As a result, Raft leader election between the replicas fails and the pods restart. The exact internal mechanism has not been determined. Temporarily raising the limits on the cluster plugin (to 16 cores / 16Gi) lets the pods start, and CPU then settles at about 2 cores. Once started, the pods keep running when the limits are set back to 2 cores / 4Gi. + +### Cause 2: CPU of a ClickHouse pod spikes while ClickHouse merges data parts + +A `chi-cpaas-clickhouse-replicated` pod shows high CPU, and its log shows `failed to read rpc header from socket`. The ClickHouse log shows automatic merges of data parts. The scheduled merge tasks run with high concurrency, so CPU spikes for a short time. This is known behavior. It does not affect the logging component, and no remediation is needed. A later release is expected to optimize merge CPU usage. + +### Cause 3: Log query is unavailable because ClickHouse was deployed by hand instead of through the cluster plugin + +A ClickHouse instance was deployed on a cluster by hand, and the log collector was switched to it. Afterwards log query is unavailable. The ClickHouse operator pod and `chi-cpaas-clickhouse-replicated-0-0-0` in `cpaas-system` are `Running`, and the ClickHouse log repeats `RaftInstance: session NNNN failed to read rpc header from socket ... due to error 2, End of file, ref count 1`. The platform log query works only with ClickHouse installed as the standard cluster plugin (Alauda Container Platform Log Storage for ClickHouse). After the plugin was redeployed, log query verification succeeded. + +### Cause 4: RaftInstance failed to read rpc header ... End of file repeats while ClickHouse works normally + +```text + RaftInstance: session NNNN failed to read rpc header from socket : due to error 2, End of file, and reset + RaftInstance: session NNNN failed to read header ... End of file +``` + +The ClickHouse log keeps printing these lines, often at a steady interval (for example every 10 seconds on one replica). The pods do not restart, and log query and ingestion work. The error count can look alarming in a log viewer or a log-keyword alert. + +ClickHouse Keeper keeps long-lived TCP connections between the replicas for Raft (port `9444`). When the far side closes one of these connections, the reading end gets end-of-file. Short-lived connections, such as those opened by readiness probes, end the same way. So do idle connections that a firewall or NAT device between the nodes reclaims. Keeper logs this line each time, then reconnects automatically and rebuilds the Raft session. The message is benign communication noise. It is not a Keeper failure and not a sign that the log store is short of resources, and it needs no action. + +## Resolution + +### Cause 1: After an upgrade, the pods restart repeatedly with Election timeout and pre-vote failures while CPU sits at its limit + +1. Confirm the signature: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep chi-cpaas-clickhouse-replicated + kubectl -n cpaas-system logs -c clickhouse --previous | grep -E 'Election timeout|pre-vote|prevote|failed to connect to peer' | tail -10 + kubectl -n cpaas-system top pod | grep chi-cpaas-clickhouse-replicated + ``` + + Election timeouts and pre-vote failures, with CPU pinned at the limit on every replica, match this cause. Memory far below its limit rules out an out-of-memory kill. + +2. Raise the limits temporarily. + + > **Warning:** Changing the ClickHouse resources on the cluster plugin rolls the ClickHouse pods. Log queries and inserts pause on each pod while it restarts, and `vector` buffers the data in the meantime. Precondition: only when this cause is confirmed, and the nodes have the CPU and memory for the temporary limit. Rollback: set the previous limits on the plugin again. + + Raise the ClickHouse CPU and memory limits on the Log Storage for ClickHouse cluster plugin (Marketplace, Cluster Plugins, select the cluster, update the plugin). Do not change them on the StatefulSets, because the plugin writes the old values back. + +3. Wait until all replicas are `Running` and CPU has settled. If the nodes cannot keep the larger reservation, set the limits back to the previous values. After startup completes, the pods can keep running at 2 cores / 4Gi. + +### Cause 2: CPU of a ClickHouse pod spikes while ClickHouse merges data parts + +Confirm that the spike is merge work and that logging still works: + +```bash +kubectl -n cpaas-system exec -c clickhouse -- \ + clickhouse-client --query "SELECT database, table, elapsed, progress, num_parts FROM system.merges" +kubectl -n cpaas-system top pod | grep chi-cpaas-clickhouse-replicated +``` + +Rows in `system.merges` during the spike, with log query and ingestion working, match this cause. No change is needed. If the pod restarts or queries fail at the same time, use Cause 1, or see [Logging ClickHouse runs out of disk, CPU or memory, or is OOMKilled on ACP](./Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md). + +### Cause 3: Log query is unavailable because ClickHouse was deployed by hand instead of through the cluster plugin + +1. Check whether the ClickHouse log storage is installed as the cluster plugin. On the global cluster: + + ```bash + kubectl get minfo | grep -i clickhouse | grep + ``` + + In the console, also open Marketplace, Cluster Plugins, select the cluster, and look for Alauda Container Platform Log Storage for ClickHouse. A ClickHouse that runs in the cluster with no matching plugin entry matches this cause. + +2. Replace it with the plugin. + + > **Warning:** Removing the hand-built ClickHouse deletes everything it stored. Logs written to it are lost unless they are exported first, and log query stays unavailable until the plugin is installed. Precondition: only when this cause is confirmed and the owner accepts the loss of the hand-built instance's data. Rollback: none for the removed data. + + Remove the hand-built ClickHouse with the same method that created it. Then install Alauda Container Platform Log Storage for ClickHouse on the cluster from Marketplace, Cluster Plugins. Choose the plugin version that supports the current platform version. + +3. Point the log collector at the plugin storage, and query the logs of a pod that is logging now. + +### Cause 4: RaftInstance failed to read rpc header ... End of file repeats while ClickHouse works normally + +1. Confirm that nothing else is wrong: + + ```bash + kubectl -n cpaas-system get pods | grep chi-cpaas-clickhouse-replicated + kubectl -n cpaas-system get pod -o jsonpath='{range .status.containerStatuses[*]}{.name}{" restarts="}{.restartCount}{" last="}{.lastState.terminated.reason}{"\n"}{end}' + kubectl -n cpaas-system logs -c clickhouse --since=1h \ + | grep -v 'RaftInstance' | grep -E '||MEMORY_LIMIT_EXCEEDED|Election timeout|pre-vote|Address already in use' | tail -20 + kubectl -n cpaas-system exec -c clickhouse -- \ + clickhouse-client --query "SELECT name FROM system.zookeeper WHERE path = '/'" + ``` + +2. This cause is confirmed when all of the following hold: + - The restart counts do not grow. + - The second listing is empty, meaning there is no other `` or `` line. + - The Keeper query answers with a list of znodes. + - A console log search returns current entries. + + Ignore the line. No configuration change, restart, or resource change is needed. + +3. If the restart count grows or log query fails, the line is only a bystander. Use Causes 1 to 3, or the articles under Related Information. `MEMORY_LIMIT_EXCEEDED` or CPU and memory alerts are covered in [nevermore and ClickHouse log components alert on CPU or memory, restart, or ignore new resource values on ACP](./nevermore_and_ClickHouse_log_components_alert_on_CPU_or_memory_restart_or_ignore_new_resource_values_on_ACP.md). + +## Diagnostic Steps + +1. Read the pod state and restart counts: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep -E 'clickhouse|razor' + kubectl -n cpaas-system get pod -o jsonpath='{range .status.containerStatuses[*]}{.name}{" restarts="}{.restartCount}{" exit="}{.lastState.terminated.exitCode}{"\n"}{end}' + ``` + +2. Read the previous `clickhouse` log and match it to a cause: + + | What you see | Cause | + |---|---| + | `Election timeout` and `at least 2 nodes should respond`, with CPU at the limit after an upgrade | Cause 1 | + | `failed to read rpc header from socket` with a short CPU spike and active rows in `system.merges` | Cause 2 | + | A ClickHouse that is not installed as the Log Storage for ClickHouse cluster plugin | Cause 3 | + | Only `failed to read rpc header ... End of file`, no restarts, and working log query | Cause 4, no action needed | + +3. Exit code `137` points to a memory limit or a failed liveness probe, not to Raft. See [Logging ClickHouse runs out of disk, CPU or memory, or is OOMKilled on ACP](./Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md) and [ClickHouse log queries fail after the log storage had been working: missing tables or broken parts on ACP](./ClickHouse_log_queries_fail_after_the_log_storage_had_been_working_missing_tables_or_broken_parts_on_ACP.md). + +## Related Information + +- [ClickHouse log storage plugin install fails or ClickHouse does not start on ACP](./ClickHouse_log_storage_plugin_install_fails_or_ClickHouse_does_not_start_on_ACP.md), for first installs that never become ready, including Keeper port errors. +- [Logging ClickHouse runs out of disk, CPU or memory, or is OOMKilled on ACP](./Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md) +- [ClickHouse log queries fail after the log storage had been working: missing tables or broken parts on ACP](./ClickHouse_log_queries_fail_after_the_log_storage_had_been_working_missing_tables_or_broken_parts_on_ACP.md) diff --git a/docs/en/solutions/ClickHouse_log_storage_plugin_install_fails_or_ClickHouse_does_not_start_on_ACP.md b/docs/en/solutions/ClickHouse_log_storage_plugin_install_fails_or_ClickHouse_does_not_start_on_ACP.md new file mode 100644 index 000000000..67a5e7427 --- /dev/null +++ b/docs/en/solutions/ClickHouse_log_storage_plugin_install_fails_or_ClickHouse_does_not_start_on_ACP.md @@ -0,0 +1,167 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# ClickHouse log storage plugin install fails or ClickHouse does not start on ACP + +## Issue + +Alauda Container Platform Log Storage for ClickHouse (module `logclickhouse`) is installed or reinstalled on a cluster and never reaches a working state. Typical reports: + +- "After a redeploy the ClickHouse cluster fails with a port `already in use`, but nothing on the node uses that port." +- "The pre-upgrade inspection says the logging ClickHouse is abnormal." +- "The ClickHouse storage plugin failed while waiting for resources to be ready." +- "`logclickhouse` cannot be installed, and the plugin shows `安装失败` (installation failed)." +- "The logging ClickHouse is not running, and the log only says it cannot find `storage_configuration.xml`." + +The pods in `cpaas-system` are `chi-cpaas-clickhouse-replicated---0` (ClickHouseInstallation `cpaas-clickhouse`) and `razor-` (StatefulSet `razor`). The operator is `clickhouse-operator`. Some background on the install: + +- `vector` is a container inside the `razor` pod, not a separate workload. Deploying ClickHouse log storage deploys `razor` and `vector` automatically. +- ClickHouse Keeper runs inside the ClickHouse container. In the pod, `clickhouse-server` listens on 8123, 9000, 9004, 9005, 9009, 9181 and 9444. +- The default data paths are `/cpaas/data/clickhouse` for ClickHouse and `/cpaas/data/vector` for the `vector` buffer. On nodes with an immutable OS they are under `/var/cpaas` instead. + +This article covers the cases where ClickHouse itself does not come up. When ClickHouse runs but `razor` does not become ready, see [razor does not become ready after a ClickHouse log storage install or upgrade on ACP](./razor_does_not_become_ready_after_a_ClickHouse_log_storage_install_or_upgrade_on_ACP.md). An install with HTTPS S3 storage signed by a private CA is covered in [ClickHouse log storage fails on HTTPS S3 with a self-signed CA without the clickhouse-ca-certificates Secret on ACP](./ClickHouse_log_storage_fails_on_HTTPS_S3_with_a_self-signed_CA_without_the_clickhouse-ca-certificates_Secret_on_ACP.md). + +## Root Cause + +### Cause 1: ClickHouse reports a port already in use although no host process holds it + +The logging ClickHouse fails after a redeploy and reports a port as `already in use`. The failure can first show up as an abnormal component in a pre-upgrade inspection. The ClickHouse log shows: + +```text + Context: Effective user of the process (clickhouse) does not match the owner of the data (root). +Application: Listen [0.0.0.0]:9181 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:9181 +Application: Listen [0.0.0.0]:9009 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:9009 +RaftInstance: deny, req term 6904, my term 6904, req log idx 77357199, my log idx 77357150 +``` + +Nothing listens on that port on the node. The cluster had been deployed more than once, and the PVCs of an earlier deployment were never deleted. State files left in those old volumes make the new cluster believe the port is taken. Inside the pod, `netstat -tulpn` shows `clickhouse-server` itself on 9181 and 9009. Uninstalling the logging component, deleting the old PVCs and deploying again fixes it. Deleting the PVCs is the step that cannot be skipped. + +A variant logs `Unable to resolve placement (cpaas-clickhouse-keeper)` beside the same `Address already in use` lines for 9181 and 9009, and `RaftInstance: peer (2) response error: failed to connect to peer 2`. The Keeper quorum never forms, ClickHouse is not ready, and Vector and razor fail with connection errors. That placement failure has not been fully diagnosed. The install succeeded after the leftover contents of `/cpaas/data` on the ClickHouse nodes were deleted before the install. Using Elasticsearch log storage instead is the alternative when this does not help. + +Lines such as `RaftInstance: session ... failed to read rpc header from socket ... End of file` about every 10 seconds, or `CgroupsReader: Cannot find 'kernel' in '/sys/fs/cgroup/memory.stat'`, also appear on clusters that work. They are not the failure. + +### Cause 2: The install fails and the operator ArtifactVersions stay empty because CPU passthrough is off + +Deploying the cluster plugin on a workload cluster ends in `安装失败` (installation failed) with: + +```text +failed to install chart for release "logclickhouse-operators": execution error at (install-operators/templates/subscriptions.yml:68:26): PackageManifest "cpaas-logging/clickhouse-operator" not exists. +``` + +The operator packages had been uploaded. The control-plane logs repeat `"wait csv" err="packagemanifests.packages.operators.coreos.com \"\" not found"`. `kubectl get artifactversion -A` lists `clickhouse-operator.` and the other operators in `cpaas-system` with empty `VERSION`, `PHASE` and `REASON` columns. + +The node VMs did not have CPU passthrough enabled. ClickHouse binaries require the SSE 4.2 instruction set, and a virtual CPU model that hides host CPU flags does not expose it. Without it the operator artifacts never became ready, and no PackageManifest was produced. After passthrough was enabled, the install succeeded. + +### Cause 3: ClickHouse does not start and its log stops at Can not find storage_configuration.xml in configmap, skip + +```text +[INFO] Try to parse HOST and DOMAIN ... [INFO] Try to generate keeper settings ... [INFO] Can not find storage_configuration.xml in configmap, skip +``` + +These lines are normal initialization messages, not the error. The resources configured for the ClickHouse component were too small for it to start. Raising them on the plugin let ClickHouse start normally. + +## Resolution + +### Cause 1: ClickHouse reports a port already in use although no host process holds it + +1. Confirm the signature: + + ```bash + kubectl -n cpaas-system get pod | grep -E 'click|raz' + kubectl -n cpaas-system logs -c clickhouse --tail=50 | grep -Ei 'already in use|placement' + ss -ltnp | grep -E ':(8123|9000|9009|9181|9444)\b' + kubectl -n cpaas-system get pvc | grep -Ei 'clickhouse|razor' + ``` + + Run `ss` on the node. This cause matches when the pod log says `already in use`, `ss` shows no listener on the node, and some PVCs are older than the current install. To see that ClickHouse itself holds the port inside the pod: + + ```bash + kubectl -n cpaas-system exec -c clickhouse -- netstat -tulpn + ``` + +2. Uninstall the logging component, then delete the leftover PVCs. + + > **Warning:** Deleting the old ClickHouse PVCs destroys the log data in them, and log search stays empty until the new cluster ingests again. Precondition: only when this cause is confirmed, the logging component is already uninstalled, and every PVC you delete belongs to an earlier ClickHouse deployment. Rollback: none; local volumes are gone together with their data. + + ```bash + kubectl -n cpaas-system delete pvc + ``` + + Repeat this for each leftover PVC, and confirm that the PVs are released or deleted. If the PVs use node-local storage and keep their files, also empty the host path printed in the PV's `spec.local.path`. + +3. Deploy the logging component again. + +4. For the `Unable to resolve placement (cpaas-clickhouse-keeper)` variant without leftover PVCs, list what is left under `/cpaas/data` on each ClickHouse node (`du -sh /cpaas/data/*`). + + > **Warning:** Deleting the contents of `/cpaas/data` removes every log storage data directory on that node, including Elasticsearch, Kafka or ZooKeeper data if those also run there. Precondition: only when the logging component is uninstalled, the placement error persists, and the directories hold nothing that must be kept. Rollback: none. + + Delete the leftover contents, then install again. If the placement error persists, use Elasticsearch log storage on that cluster instead. + +### Cause 2: The install fails and the operator ArtifactVersions stay empty because CPU passthrough is off + +1. Confirm the signature: + + ```bash + kubectl get artifactversion -A + kubectl get packagemanifest -A | grep -i clickhouse + grep -o -w -E 'sse4_2|avx|avx2' /proc/cpuinfo | sort | uniq -c + ``` + + Run the `grep` on a node VM. Empty `VERSION`, `PHASE` and `REASON` columns on the operator ArtifactVersions, no `clickhouse-operator` PackageManifest, and missing CPU flags all match this cause. + +2. Enable CPU passthrough on the hypervisor. + + > **Warning:** Changing the VM CPU mode needs a VM restart, which stops every pod on that node. Precondition: the node is cordoned and drained (`kubectl cordon ` and `kubectl drain --ignore-daemonsets --delete-emptydir-data`), and the hypervisor allows host passthrough for that VM. Rollback: switch the CPU mode back and restart again. + + Set the CPU mode of the node VMs to host passthrough, restart them one at a time, and run `kubectl uncordon ` when each node is back. + +3. Retry the plugin install. + +### Cause 3: ClickHouse does not start and its log stops at Can not find storage_configuration.xml in configmap, skip + +1. Check the current resources and events: + + ```bash + kubectl -n cpaas-system describe pod | grep -E 'Limits|Requests|OOMKilled|Events' -A3 + kubectl get moduleinfo -o yaml | grep -n -A6 resources + ``` + +2. Raise the resources. + + > **Warning:** Changing the resources restarts the ClickHouse pods. Precondition: the nodes have free CPU and memory for the new values. Rollback: set the previous values on the plugin. + + Raise the ClickHouse CPU and memory on the log storage plugin (Marketplace, Cluster Plugins, update the plugin). Size them from the node capacity. No single value is prescribed. + +## Diagnostic Steps + +1. Read the plugin and pod state: + + ```bash + kubectl get ars -A | grep -Ei 'click|log' + kubectl get artifactversion -A | grep -i clickhouse + kubectl -n cpaas-system get pods | grep -Ei 'click|razor' + kubectl -n ck-system get pods 2>/dev/null + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | `already in use` with no host listener, or `Unable to resolve placement (cpaas-clickhouse-keeper)` | Cause 1 | + | `PackageManifest "cpaas-logging/clickhouse-operator" not exists`, or empty operator ArtifactVersions on VMs without CPU passthrough | Cause 2 | + | A log that stops at the `storage_configuration.xml` line, with small resource limits | Cause 3 | + +3. If ClickHouse runs and `razor` stays not ready, the failure is in the `razor` initialization. See [razor does not become ready after a ClickHouse log storage install or upgrade on ACP](./razor_does_not_become_ready_after_a_ClickHouse_log_storage_install_or_upgrade_on_ACP.md). + +## Related Information + +- Related articles: [razor does not become ready after a ClickHouse log storage install or upgrade on ACP](./razor_does_not_become_ready_after_a_ClickHouse_log_storage_install_or_upgrade_on_ACP.md) covers the other first-install and upgrade failures of the same plugin. +- [ClickHouse log storage logs RaftInstance errors with pod restarts, merge CPU spikes, or a hand-built ClickHouse on ACP](./ClickHouse_log_storage_logs_RaftInstance_errors_with_pod_restarts_merge_CPU_spikes_or_a_hand-built_ClickHouse_on_ACP.md) +- [ClickHouse log queries fail after the log storage had been working: missing tables or broken parts on ACP](./ClickHouse_log_queries_fail_after_the_log_storage_had_been_working_missing_tables_or_broken_parts_on_ACP.md) diff --git a/docs/en/solutions/Elasticsearch_log_index_replica_count_reverts_after_editing_the_index_template_on_ACP.md b/docs/en/solutions/Elasticsearch_log_index_replica_count_reverts_after_editing_the_index_template_on_ACP.md new file mode 100644 index 000000000..9eded792b --- /dev/null +++ b/docs/en/solutions/Elasticsearch_log_index_replica_count_reverts_after_editing_the_index_template_on_ACP.md @@ -0,0 +1,89 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Elasticsearch log index replica count reverts after editing the index template on ACP + +## Issue + +The replica count of the platform log indices needs to change, for example to give each shard a copy or to remove replicas on a single-node cluster. The indices are the platform log indices in the Elasticsearch log storage (Alauda Container Platform Log Storage for Elasticsearch, `cpaas-elasticsearch` in `cpaas-system`), such as `log-workload-`. + +`number_of_replicas` was changed in the index template directly in Elasticsearch, through the `_template` API or a tool such as Kibana. Shortly afterwards the old value is back, and new daily indices keep the old replica count. + +## Root Cause + +The platform manages the settings of the log index templates through ConfigMap `aiops-logging-config` in namespace `cpaas-system`, on the cluster that runs the log storage. The platform writes the templates from that ConfigMap, so a change made only in the Elasticsearch template is reverted. Changing the replica count is supported, but it has to be made in the ConfigMap. + +Each key of the ConfigMap holds one template. The `log.json` key holds the template for `"index_patterns": "log-*"` (`"order": 1`). Its `settings` look like this: + +```json +"index.lifecycle.name": "alauda-log", +"number_of_shards": 1, +"number_of_replicas": 1, +"refresh_interval": "10s" +``` + +The field to change is `"number_of_replicas"` in the `settings` of the key whose `index_patterns` matches the indices you want to change. + +## Resolution + +1. On the cluster that runs the Elasticsearch log storage, find the ConfigMap and save a copy: + + ```bash + kubectl get cm -A | grep aiops + kubectl -n cpaas-system get cm aiops-logging-config -o yaml > /tmp/aiops-logging-config.before.yaml + ``` + + The listing shows three ConfigMaps in `cpaas-system`: `aiops-logging-config`, `aiops-monitor-config` and `aiops-nf-migrate-config`. Only `aiops-logging-config` holds the log index templates. + +2. Edit `"number_of_replicas"` in the `settings` of the key whose `index_patterns` matches the indices (`log.json` for `log-*`). Change only `number_of_replicas`, and keep the other settings of the key as they are. + + > **Warning:** The replica count decides how many copies of each shard exist. With `0`, every shard has a single copy, so losing one data node loses that node's data. A higher value needs proportionally more disk on the data nodes and starts replica recovery traffic for each new index. A replica can never be placed on the node that holds its primary, so a single-node Elasticsearch stays `yellow` with any value above `0`. Precondition: the number of data nodes is larger than the new replica count, there is disk for the extra copies, and the backup from step 1 is saved. Rollback: `kubectl apply -f /tmp/aiops-logging-config.before.yaml`. + + ```bash + kubectl -n cpaas-system edit cm aiops-logging-config + ``` + +3. Check that the template now has the new value and that the value is not reverted: + + ```bash + ES_USER=$(kubectl -n cpaas-system get secret elasticsearch-basic-auth -o jsonpath='{.data.username}' | base64 -d) + ES_PASS=$(kubectl -n cpaas-system get secret elasticsearch-basic-auth -o jsonpath='{.data.password}' | base64 -d) + kubectl -n cpaas-system exec cpaas-elasticsearch-0 -c cpaas-elasticsearch -- \ + curl -sS -u "${ES_USER}:${ES_PASS}" 'http://cpaas-elasticsearch:9200/_template?pretty' \ + | grep -n -E '"log-\*"|number_of_replicas' + ``` + +4. A template applies only to indices created after the change, so the new count shows from the next daily index onward. `number_of_replicas` is a dynamic Elasticsearch index setting. Indices that already exist keep their count unless you change it on them with the `_settings` API: + + ```bash + kubectl -n cpaas-system exec cpaas-elasticsearch-0 -c cpaas-elasticsearch -- \ + curl -sS -u "${ES_USER}:${ES_PASS}" -X PUT -H 'Content-Type: application/json' \ + 'http://cpaas-elasticsearch:9200//_settings' -d '{"index":{"number_of_replicas":}}' + ``` + +## Diagnostic Steps + +1. Read the replica count the indices actually have (with `ES_USER` and `ES_PASS` loaded as in Resolution step 3): + + ```bash + kubectl -n cpaas-system exec cpaas-elasticsearch-0 -c cpaas-elasticsearch -- \ + curl -sS -u "${ES_USER}:${ES_PASS}" 'http://cpaas-elasticsearch:9200/_cat/indices/log-*?v&h=index,health,pri,rep&s=index' | tail -5 + ``` + +2. Compare it with `number_of_replicas` in `aiops-logging-config`: + + ```bash + kubectl -n cpaas-system get cm aiops-logging-config -o jsonpath='{.data.log\.json}' | grep -n -E 'index_patterns|number_of_replicas|number_of_shards' + ``` + + If the template value matches the ConfigMap and not the value you set in Elasticsearch, the platform has restored the ConfigMap value. Change the ConfigMap as described above. + +## Related Information + +- [Elasticsearch log indices red or yellow with unassigned shards on ACP](./Elasticsearch_log_indices_red_or_yellow_with_unassigned_shards_on_ACP.md), for a single-node Elasticsearch that stays `yellow` because its indices have replicas. +- [Log Kafka lag from Elasticsearch write hot spots, breaker trips, IK analyzer errors, or too few partitions on ACP](./Log_Kafka_lag_from_Elasticsearch_write_hot_spots_breaker_trips_IK_analyzer_errors_or_too_few_partitions_on_ACP.md), for `number_of_shards` in the same `log.json` settings. diff --git a/docs/en/solutions/Elasticsearch_log_indices_red_or_yellow_with_unassigned_shards_on_ACP.md b/docs/en/solutions/Elasticsearch_log_indices_red_or_yellow_with_unassigned_shards_on_ACP.md new file mode 100644 index 000000000..962c9cea3 --- /dev/null +++ b/docs/en/solutions/Elasticsearch_log_indices_red_or_yellow_with_unassigned_shards_on_ACP.md @@ -0,0 +1,254 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Elasticsearch log indices red or yellow with unassigned shards on ACP + +## Issue + +Platform logs or events cannot be viewed, or the newest log is hours old. Typical reports: + +- "System logs cannot be viewed. The newest log is from last night." +- "The event query returns no data." +- "The ES cluster state alarm fired and we cannot find why." +- "A newly deployed ES alerts with status 2." +- "The log ES is down. How do we recover it?" + +On the Elasticsearch log storage (Alauda Container Platform Log Storage for Elasticsearch, `cpaas-elasticsearch` in `cpaas-system`), `_cluster/health` or `_cat/indices` shows `red` or `yellow` indices and unassigned shards. `lanaya` may report that there is no space to allocate primary shards, and Kafka backs up behind it. + +`lanaya` consumes the log topics in order. It takes the next message only after the current one is persisted. A single index that cannot accept writes therefore stalls the whole consumer, and Kafka lag grows from that moment. + +The built-in alert rule for this state is `ES 集群的状态 > 0 且持续 5 分钟` ("ES cluster status above 0 for 5 minutes", severity critical) in the `cpaas-log-rules` policy. Status 1 is `yellow` and status 2 is `red`. + +When an Elasticsearch pod is `Pending`, every business cluster loses logs at once, or CPU is high right after the disk was freed, see [Elasticsearch log storage Pending or every cluster loses logs after the log disk fills on ACP](./Elasticsearch_log_storage_Pending_or_every_cluster_loses_logs_after_the_log_disk_fills_on_ACP.md). + +## Root Cause + +### Cause 1: lanaya reports no space to allocate primary shards, and the current index is red + +The data path (`/cpaas`) is above about 80% or full. The current day's index cannot allocate its primary shard, so it is `red`. `lanaya` cannot persist the batch it holds and logs bulk failures such as: + +```text +"type":"unavailable_shards_exception","reason":"[event-][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[event-][0]] containing [2] requests]" +elasticsearch/elasticsearch.go:166] Database bulk error {"bulkSize": 100, ... "error": "bulk response error"} +``` + +`_cluster/allocation/explain` for the unassigned primary returns `"reason" : "ALLOCATION_FAILED"` after 5 attempts. It shows `failed to create shard, failure FileSystemException[... No space left on device]` and `"allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes that hold an in-sync shard copy"`. The Kafka dashboard can show the same full disk first: the consume rate drops to 0, and lag on every `alauda_log` partition climbs. + +Deleting the largest recent indices frees the disk. Shard allocation then has to be triggered again, with a reroute retry or an Elasticsearch restart, before every index recovers. A single project index of about 200 GB per day is beyond what a small-scale deployment is sized for: [Elasticsearch Scale Switching](./Elasticsearch_Scale_Switching.md) rates small scale at up to 12,000 entries per second and 300 GB of total logs. If the disk fills again the next day, the store is too small for the volume. Expand the disk, reduce the applications' log output, or shorten retention. + +### Cause 2: The disk was expanded, and Elasticsearch still reports insufficient space on that node + +One data node filled. After its disk was expanded, the cluster still reports insufficient disk on that node, and shard allocation stays failed. Elasticsearch stops retrying a shard after repeated allocation failures (`max_retries`), so the shards wait even though space now exists. `POST _cluster/reroute?retry_failed=true` reallocates them. The cluster turns `green` and the Kafka backlog drains. + +### Cause 3: A single-node Elasticsearch has indices with replicas + +The state alarm fires, the pods are `Running`, and the node disks are fine. `_cluster/health` on the single-node Elasticsearch returns `"status" : "yellow"`, `"number_of_nodes" : 1`, many `unassigned_shards`, and `"unassigned_primary_shards" : 0`. `_cat/indices` lists indices such as `audit-*` and `event-*` as `yellow` with `pri 1 rep 1`. + +`GET _cluster/allocation/explain` shows replica shards with reason `INDEX_CREATED`, `"can_allocate":"no"`, and the `same_shard` decider answering `a copy of this shard is already allocated to this node`. A single node cannot host a replica of its own primary, so those indices stay `yellow` forever. Setting `number_of_replicas` to 0 on all indices lets health turn `green` gradually. For high availability, run at least three Elasticsearch nodes instead. + +### Cause 4: An earlier cleanup left historical indices behind + +On a multi-node Elasticsearch with the same alarm, abnormal indices are present. Deleting only the visibly abnormal ones does not make the cluster `green`. An earlier cleanup of historical indices was incomplete, and older indices are still there. A complete cleanup of everything outside the retention window recovers the cluster. + +### Cause 5: Every index turns red after a reinstall following an Elasticsearch major version upgrade + +Elasticsearch was reinstalled after its disk filled, and application logs still do not appear. All indices are `red`. The suspected reason, which has not been confirmed, is that `nevermore` uploaded cache from before the platform upgrade that is not compatible with the new index version (Elasticsearch moved from v7 to v8). Cleaning the Elasticsearch data again fixes it. Afterwards every index holds the current day's logs and is `green`, apart from a few pipeline indices. + +### Cause 6: The cluster is red with one unassigned primary shard in a single daily index, and the disk has room + +Disk usage on the Elasticsearch nodes is fine. `_cluster/health` returns `red` with `unassigned_shards` at 1. `_cat/shards` shows primary shard `0` of one daily project index (named like `log-project--`) as `UNASSIGNED`, while every other primary is `STARTED`. One unassigned primary is enough to make the whole cluster `red`. Why that shard could not be assigned is not known. Deleting that one index brings the cluster back to `green` with 0 unassigned shards. The log data in that index is lost. + +## Resolution + +Load the credentials and a helper for the API calls below: + +```bash +ES_USER=$(kubectl -n cpaas-system get secret elasticsearch-basic-auth -o jsonpath='{.data.username}' | base64 -d) +ES_PASS=$(kubectl -n cpaas-system get secret elasticsearch-basic-auth -o jsonpath='{.data.password}' | base64 -d) +es() { kubectl -n cpaas-system exec cpaas-elasticsearch-0 -c cpaas-elasticsearch -- \ + curl -sS -u "${ES_USER}:${ES_PASS}" "$@"; } +``` + +### Cause 1: lanaya reports no space to allocate primary shards, and the current index is red + +1. Confirm the signature: + + ```bash + df -hT /cpaas + es 'http://cpaas-elasticsearch:9200/_cat/indices?v&health=red' + es 'http://cpaas-elasticsearch:9200/_cat/indices?v&h=index,health,store.size&s=store.size:desc' | head -15 + es 'http://cpaas-elasticsearch:9200/_cluster/allocation/explain?pretty' | grep -E 'reason|No space|explanation' + kubectl -n cpaas-system logs deploy/lanaya --tail=100 | grep -iE 'disk|space|shard|unavailable_shards' + ``` + + Run `df` on the Elasticsearch nodes. + +2. Free space by deleting the largest historical indices. + + > **Warning:** Deleting an index permanently removes every log in it. Precondition: only indices you have chosen by exact name from the size listing, outside the period that must be kept. Never use wildcards. Rollback: none without a snapshot. + + ```bash + es -X DELETE 'http://cpaas-elasticsearch:9200/' + ``` + +3. Let the red index allocate. Try the reroute retry in Cause 2 first. A restart of the Elasticsearch pods also triggers allocation: + + > **Warning:** Restarting the Elasticsearch pods one at a time degrades log search and ingestion while each pod is down. Precondition: only when this cause is confirmed and `df` now shows free space below the watermark. Rollback: none needed; the StatefulSet recreates the pods. + + ```bash + for p in $(kubectl -n cpaas-system get pod -l service_name=cpaas-elasticsearch -o name); do + kubectl -n cpaas-system delete "$p"; sleep 15 + kubectl -n cpaas-system wait --for=condition=Ready "$p" --timeout=15m + done + ``` + +4. If the disk refills within a day, expand `/cpaas`, shorten the retention in the log storage settings, or reduce the applications' log output. + +### Cause 2: The disk was expanded, and Elasticsearch still reports insufficient space on that node + +1. Confirm the signature: + + ```bash + es 'http://cpaas-elasticsearch:9200/_cat/allocation?v' + es 'http://cpaas-elasticsearch:9200/_cluster/allocation/explain?pretty' | head -40 + ``` + + `max_retries`, or a stale disk message for a node that now has room, matches this cause. + +2. Retry the failed allocations. + + > **Warning:** A reroute retry starts moving and recovering shards at once, which adds disk and network load. Precondition: only when the node's disk was already expanded or cleaned and `df` confirms the free space. Rollback: none; recoveries that started run to completion. + + ```bash + es -X POST 'http://cpaas-elasticsearch:9200/_cluster/reroute?retry_failed=true' + es 'http://cpaas-elasticsearch:9200/_cluster/health?pretty' + ``` + +3. Watch the Kafka lag fall afterwards. + +### Cause 3: A single-node Elasticsearch has indices with replicas + +1. Confirm the signature: + + ```bash + es 'http://cpaas-elasticsearch:9200/_cat/nodes?v' + es 'http://cpaas-elasticsearch:9200/_cluster/allocation/explain?pretty' | grep -E 'index|primary|explanation' | head + ``` + +2. Set the replicas to 0. + + > **Warning:** With `number_of_replicas` at 0, every shard exists once. If more data nodes join later, the indices have no copy until the value is raised again. Precondition: only when the cluster really has one data node and the unassigned shards are replicas. Rollback: set `number_of_replicas` back to 1 after a second data node joins. + + ```bash + es -X PUT -H 'Content-Type: application/json' \ + 'http://cpaas-elasticsearch:9200/*/_settings' \ + -d '{"index":{"number_of_replicas":0}}' + ``` + +3. Health should turn `green` over the next minutes. New daily indices are created from the platform index template, which sets one replica. Recheck the next day, and change the template value as described in [Elasticsearch log index replica count reverts after editing the index template on ACP](./Elasticsearch_log_index_replica_count_reverts_after_editing_the_index_template_on_ACP.md). + +### Cause 4: An earlier cleanup left historical indices behind + +1. List every index that is not `green` and every index older than the retention window: + + ```bash + es 'http://cpaas-elasticsearch:9200/_cat/indices?v&s=index' | grep -v green + ``` + +2. Delete each old index by exact name, with the command and warning in Cause 1. Repeat until `_cat/indices` holds only indices inside the window. + +### Cause 5: Every index turns red after a reinstall following an Elasticsearch major version upgrade + +1. Confirm the signature: + + ```bash + es 'http://cpaas-elasticsearch:9200/_cat/indices?v&h=index,health,docs.count,creation.date.string&s=index' + ``` + + Every index is `red` right after Elasticsearch was reinstalled on the new major version. + +2. Clean the data again. + + > **Warning:** Cleaning the Elasticsearch data removes every index, including the red ones. Precondition: only when this cause is confirmed and Elasticsearch holds no history that must be kept. Rollback: none. + + Record the replica count and scale the Elasticsearch StatefulSet to 0: + + ```bash + kubectl -n cpaas-system get sts cpaas-elasticsearch + kubectl -n cpaas-system scale sts cpaas-elasticsearch --replicas=0 + ``` + + On each Elasticsearch node, empty the Elasticsearch data directory (`/cpaas/data/elasticsearch` by default): + + ```bash + cd /cpaas/data/elasticsearch && rm -rf ./* + ``` + + Scale the StatefulSet back: + + ```bash + kubectl -n cpaas-system scale sts cpaas-elasticsearch --replicas= + ``` + +3. Check `_cat/indices` again. The indices should hold the current day's logs and be `green`. + +### Cause 6: The cluster is red with one unassigned primary shard in a single daily index, and the disk has room + +1. Confirm the signature: the disk has room, exactly one primary is unassigned, and it belongs to one index. + + ```bash + df -hT /cpaas + es 'http://cpaas-elasticsearch:9200/_cluster/health?pretty' | grep -E '"status"|unassigned' + es 'http://cpaas-elasticsearch:9200/_cat/shards?v&h=index,shard,prirep,state,unassigned.reason' | grep UNASSIGNED + es 'http://cpaas-elasticsearch:9200/_cluster/allocation/explain?pretty' | grep -E '"index"|"shard"|"reason"|explanation' + ``` + + Save the allocation explanation before you act. If it says allocation failed for lack of space on a node that now has room, use Cause 2 instead. If `df` is above the watermark, use Cause 1. + +2. Delete that index. + + > **Warning:** Deleting the index permanently removes every log stored in it, which for a daily project index is one day of that project's logs. Precondition: only when the cluster is `red` because of an unassigned primary of this one index, `df` shows free space, the other primaries are `STARTED`, and the owner accepts losing that index. Take the exact name from `_cat/shards`, without wildcards. Rollback: none without a snapshot. + + ```bash + es -X DELETE 'http://cpaas-elasticsearch:9200/' + es 'http://cpaas-elasticsearch:9200/_cluster/health?pretty' | grep -E '"status"|unassigned' + ``` + + Health should return to `green` with `unassigned_shards` at 0. + +## Diagnostic Steps + +1. Check disk and health together: + + ```bash + df -hT / /cpaas + es 'http://cpaas-elasticsearch:9200/_cluster/health?pretty' + es 'http://cpaas-elasticsearch:9200/_cluster/allocation/explain?pretty' | head -40 + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | A red current index, with `unavailable_shards_exception` in `lanaya` or `No space left on device` in allocation explain, or a consume rate at 0 with `/cpaas` near full | Cause 1 | + | Space available, and allocation still failed on one node | Cause 2 | + | One node with unassigned replicas and the `same_shard` decider answering no | Cause 3 | + | Old indices beyond the retention window still present | Cause 4 | + | All indices red right after a reinstall on a new major version | Cause 5 | + | A red cluster with enough disk and a single unassigned primary in one index | Cause 6 | + +## Related Information + +- Related articles: [Elasticsearch log storage Pending or every cluster loses logs after the log disk fills on ACP](./Elasticsearch_log_storage_Pending_or_every_cluster_loses_logs_after_the_log_disk_fills_on_ACP.md) covers a `Pending` Elasticsearch pod, a full global log storage, and high CPU after the disk was freed. +- [Elasticsearch log index replica count reverts after editing the index template on ACP](./Elasticsearch_log_index_replica_count_reverts_after_editing_the_index_template_on_ACP.md) +- [Elasticsearch Scale Switching](./Elasticsearch_Scale_Switching.md) diff --git a/docs/en/solutions/Elasticsearch_log_storage_Pending_or_every_cluster_loses_logs_after_the_log_disk_fills_on_ACP.md b/docs/en/solutions/Elasticsearch_log_storage_Pending_or_every_cluster_loses_logs_after_the_log_disk_fills_on_ACP.md new file mode 100644 index 000000000..9667af713 --- /dev/null +++ b/docs/en/solutions/Elasticsearch_log_storage_Pending_or_every_cluster_loses_logs_after_the_log_disk_fills_on_ACP.md @@ -0,0 +1,156 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Elasticsearch log storage Pending or every cluster loses logs after the log disk fills on ACP + +## Issue + +The Elasticsearch log storage (Alauda Container Platform Log Storage for Elasticsearch, `cpaas-elasticsearch` in `cpaas-system`) has filled its disk, and one of the following follows: + +- Event queries return no data, and an Elasticsearch pod is `Pending`. Only one of three pods may be `Running`. +- Log queries fail for every business cluster at once, because those clusters store their logs on the global cluster. +- Right after the disk was freed, Elasticsearch CPU is at its limit, and Kafka lag falls only slowly. + +When the pods run but indices are `red` or `yellow` with unassigned shards, see [Elasticsearch log indices red or yellow with unassigned shards on ACP](./Elasticsearch_log_indices_red_or_yellow_with_unassigned_shards_on_ACP.md). + +A global-cluster node that hosts Elasticsearch log storage can be powered off for maintenance. While it is down, only log query is affected. Drain it first (`kubectl cordon ` and `kubectl drain --ignore-daemonsets --delete-emptydir-data`), then shut it down. After it is back, run `kubectl uncordon ` and wait for the cluster to return to `green`. + +## Root Cause + +### Cause 1: /cpaas is not a separate disk, so a full log store fills / and the Elasticsearch pod is evicted or Pending + +Event queries return no data, and the Elasticsearch pod is `Pending`. Its events read: + +```text +FailedScheduling: 0/23 nodes are available: 2 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }, 21 node(s) didn't match PersistentVolume's node affinity. preemption: 0/23 nodes are available: 23 Preemption is not helpful for scheduling. +``` + +The pod's local PV pins it to the tainted node, so no other node can take it. `lsblk` on the storage node shows the root logical volume on the first disk and a second large disk with no mount point. No dedicated disk was mounted on `/cpaas`, so Elasticsearch data was written to the root filesystem, which filled and triggered eviction. Cleaning the Elasticsearch data brings `/` back below the eviction threshold. The `DiskPressure` taint is then removed, and the pod schedules again. + +On a small global cluster where Elasticsearch runs on the three control-plane nodes, the same full disk shows as `FailedScheduling: 0/3 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }, 2 node(s) had volume node affinity conflict.` Log query and audit show no recent data. Cleaning the node directory restores Elasticsearch. + +### Cause 2: The global log storage is full and every business cluster that stores logs there has no data + +The region or business clusters use the global cluster as their log storage. `/cpaas` on several global storage nodes is at 100%: `df -h /cpaas` shows almost nothing available, and `du` shows `/cpaas/data/elasticsearch` and `/cpaas/data/kafka0` as the large directories. Every Elasticsearch component on the global cluster has failed, so log queries fail for every business cluster at once. + +There are two immediate choices: expand `/cpaas` and recover, or clear all log data and lose the history. The lasting fix is log offloading. Deploy log storage on each business cluster and repoint that cluster's collector to it. After the business clusters store their own logs, the global storage can be repaired without affecting them. + +### Cause 3: Elasticsearch CPU is high right after a full disk was freed + +Elasticsearch and `razor` restarted recently, and Elasticsearch CPU is at its limit (for example one pod at 5.78 cores, 96.4% of the limit). Monitoring shows Elasticsearch storage at 99% to 100% until a moment when it drops to about 50%. `alauda_log` lag on some partitions is very large and starts to fall slowly from that moment. + +The disk had been full. Elasticsearch stayed up but could not write, and Kafka backed up. A shorter log retention then deleted old indices and freed the disk, so Elasticsearch began indexing the whole backlog at once. Raising the Elasticsearch CPU limit (to 6 cores) lets it work through the backlog, after which CPU returns to normal. Restarts of unrelated components in the same listing, such as `courier`, are not part of this cause. + +## Resolution + +### Cause 1: /cpaas is not a separate disk, so a full log store fills / and the Elasticsearch pod is evicted or Pending + +1. Confirm the signature: + + ```bash + kubectl -n cpaas-system describe pod | sed -n '/Events:/,$p' + kubectl describe node | grep -E 'Taints|DiskPressure' + kubectl -n cpaas-system get pods -o wide | grep elasticsearch + ``` + + On the storage node: + + ```bash + lsblk + findmnt /cpaas || echo "/cpaas is not a separate mount" + df -hT / /cpaas + ``` + +2. Free the space. + + > **Warning:** Cleaning the Elasticsearch data removes the stored logs on that node. Precondition: only when this cause is confirmed, the disk-pressure taint blocks the pod, and the stored history on that node is not needed. Rollback: none. + + Record the replica count and scale the Elasticsearch StatefulSet to 0. Empty the Elasticsearch data directory on the node (`/cpaas/data/elasticsearch`), then scale back: + + ```bash + kubectl -n cpaas-system get sts cpaas-elasticsearch + kubectl -n cpaas-system scale sts cpaas-elasticsearch --replicas=0 + ``` + + On the node: + + ```bash + cd /cpaas/data/elasticsearch && rm -rf ./* + ``` + + ```bash + kubectl -n cpaas-system scale sts cpaas-elasticsearch --replicas= + ``` + +3. When `DiskPressure` clears, the taint is removed and the pod schedules. + +4. Give `/cpaas` its own disk before the store fills `/` again. Mounting a new disk on a directory that already holds files hides those files and does not free their space on `/`. Move or delete them before you mount. + +### Cause 2: The global log storage is full and every business cluster that stores logs there has no data + +1. Confirm that the business clusters store their logs on the global cluster, and that the global storage nodes are full: + + ```bash + df -h /cpaas + du -sh /cpaas/data/* + kubectl -n cpaas-system get pods | grep -E 'elasticsearch|kafka|zookeeper' + ``` + +2. Offload each business cluster: + 1. Prepare 3 to 5 log storage nodes in the business cluster. + 2. Mount a separate disk for the Elasticsearch data directory on each of them. + 3. Deploy the Elasticsearch log storage plugin on the business cluster from the cluster plugins. + 4. Change the storage cluster setting of that cluster's log collection plugin to the new storage. + + > **Warning:** Repointing a business cluster's log collector to a new storage leaves its history on the old store, and the new storage starts empty. Precondition: the new storage is deployed and healthy. Rollback: set the collector's storage cluster back to global. + +3. After every business cluster stores its own logs, repair the global storage. Either expand `/cpaas`, or clear its log data with the scale-to-0 and directory cleanup shown in Cause 1. + +### Cause 3: Elasticsearch CPU is high right after a full disk was freed + +1. Check whether the storage graph dropped sharply and whether retention was just shortened. If both are true, the CPU is the backlog being indexed. + + ```bash + kubectl -n cpaas-system get pod | grep -E 'cpaas-kafka|cpaas-zookeeper|cpaas-elasticsearch|lanaya|razor' + kubectl -n cpaas-system top pod | grep cpaas-elasticsearch + ``` + + Recent restarts of the Elasticsearch and `razor` pods, a storage graph that dropped from about 100%, and lag that starts falling at the same moment match this cause. + +2. Raise the CPU limit. + + > **Warning:** Raising the Elasticsearch CPU limit on the log storage plugin rolls every Elasticsearch pod, and ingestion pauses on each pod while it restarts. Precondition: only when the storage graph shows the disk was just freed, lag is falling, and CPU is at its limit. Rollback: set the previous CPU limit back on the plugin. + + Raise the Elasticsearch CPU on the log storage plugin (6 cores was enough here). Watch the lag fall. CPU returns to normal (below about one core per pod) once the backlog is indexed. + +## Diagnostic Steps + +1. Read the pod state and the disk: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep -E 'elasticsearch|kafka|lanaya' + df -hT / /cpaas + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | `/cpaas` is not a separate mount, or a `Pending` Elasticsearch pod on a node with the `disk-pressure` taint | Cause 1 | + | Every business cluster failing at once, with its collector pointed at the global log storage | Cause 2 | + | High Elasticsearch CPU right after the storage graph dropped from a full disk | Cause 3 | + +## Related Information + +- Related articles: [Elasticsearch log indices red or yellow with unassigned shards on ACP](./Elasticsearch_log_indices_red_or_yellow_with_unassigned_shards_on_ACP.md) covers shards that stay unassigned after space was freed, single-node replicas, and indices that stay red. +- [Logs, audit and events stop while the log Kafka is degraded on ACP](./Logs_audit_and_events_stop_while_the_log_Kafka_is_degraded_on_ACP.md) diff --git a/docs/en/solutions/Inspection_faults_x86-64-v2_CPU_error_discover-device_crash_loop_small_ClickHouse_limits_log_backlog_or_WebCLI_role_on_ACP.md b/docs/en/solutions/Inspection_faults_x86-64-v2_CPU_error_discover-device_crash_loop_small_ClickHouse_limits_log_backlog_or_WebCLI_role_on_ACP.md new file mode 100644 index 000000000..3d000fe50 --- /dev/null +++ b/docs/en/solutions/Inspection_faults_x86-64-v2_CPU_error_discover-device_crash_loop_small_ClickHouse_limits_log_backlog_or_WebCLI_role_on_ACP.md @@ -0,0 +1,175 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.16.x + - 4.2.x + - 4.3.x +--- + +# Inspection faults: x86-64-v2 CPU error, discover-device crash loop, small ClickHouse limits, log backlog, or WebCLI role on ACP + +## Issue + +A cluster inspection (for example the inspection taken before an upgrade) reports a component as abnormal, and the finding is a real fault rather than a false positive. One of these is on the report or in the pod log: + +- `olm-registry-platform` crashes with `Fatal glibc error: CPU does not support x86-64-v2`. +- Pods named `discover-device` in `nativestor-system` are in `CrashLoopBackOff` with very high restart counts. +- ClickHouse CPU on the global or a region cluster is above the 80% inspection threshold, and the log storage plugin is abnormal. +- Log search in the console returns nothing, while many business clusters write their logs into the global log storage. +- A custom platform-inspector role can do anything in WebCLI, including node login, after an upgrade from 3.16. + +For monitoring rows (a second Prometheus, a missing `vmagent-agent`, an OOMKilled `vmalert`, or a Prometheus stuck in `Init`), see [Monitoring faults: a second Prometheus, missing vmagent-agent, vmalert OOMKilled, or Prometheus stuck in Init on ACP](./Monitoring_faults_a_second_Prometheus_missing_vmagent-agent_vmalert_OOMKilled_or_Prometheus_stuck_in_Init_on_ACP.md). + +## Root Cause + +### Cause 1: Fatal glibc error: CPU does not support x86-64-v2 + +`olm-registry-platform` exits as soon as it starts, with `Fatal glibc error: CPU does not support x86-64-v2`. `/proc/cpuinfo` on that virtual machine has `sse4_2` but not `popcnt`. The image requires the x86-64-v2 instruction set level, so the guest CPU model must expose `popcnt` inside the VM. This appeared in a 4.2 inspection taken before a move to 4.3. + +### Cause 2: discover-device in nativestor-system is in CrashLoopBackOff and TopoLVM is not in use + +`discover-device` pods in `nativestor-system` are `CrashLoopBackOff` with restart counts in the tens of thousands. The workload is the device discovery used by TopoLVM local storage. On a cluster that does not use local storage, the DaemonSet has nothing to do and can be deleted. When TopoLVM volumes are in use, keep it; the crash then needs a storage investigation, not a delete. + +### Cause 3: ClickHouse is still at 2 CPU and 4Gi while the log storage plugin is abnormal + +The inspection says ClickHouse CPU on the global cluster and on a region cluster is above 80%, and the log storage plugin Alauda Container Platform Log Storage for ClickHouse is abnormal. The live limits of that ClickHouse are 2 CPU and 4Gi, which is too small for its load. The fix that matches this saturation is 4 CPU and 8Gi for both requests and limits. + +When `chi-cpaas-clickhouse` is close to 100% CPU and memory and several pods on that cluster are failing, adding resources to the cluster hosts resolves the pod errors. etcd leader changes and controller restarts flagged in the same report can be ignored once a current check shows that etcd no longer changes leader. + +### Cause 4: The platform cannot query logs because the global log Kafka is behind the business clusters + +Log search in the console returns nothing. Many business clusters write their logs into the global log storage at the same time. The global log storage consumes more slowly than the writers produce, so records pile up in the global log Kafka. After the backlog drains, log search works again. Kafka pod memory above 80% also appears on inspections by itself. The user-visible fault here is the lag, not the memory. + +### Cause 5: A custom platform-inspector role can operate WebCLI because its YAML contains acp-podsexec and acp-rootexec + +A role cloned from the platform auditor before an upgrade from 3.16 can suddenly use WebCLI for full operations, including node login. On 4.2, the built-in platform auditor has query-only WebCLI: no create, no delete, and no direct node login. The custom role YAML contains `acp-podsexec` and `acp-rootexec`. Removing those two entries, and leaving `get`, `list` and `watch`, restores query-only access. + +## Resolution + +### Cause 1: Fatal glibc error: CPU does not support x86-64-v2 + +1. Confirm the error: + + ```bash + kubectl get pods -A -o wide | grep olm-registry-platform + kubectl logs -n --tail=30 | grep x86-64-v2 + ``` + +2. On the virtual machine that runs the pod (a host shell, or `kubectl debug` on that node), check the CPU flags: + + ```bash + grep -m1 -o -w -E 'popcnt|sse4_2' /proc/cpuinfo || true + grep -c popcnt /proc/cpuinfo + ``` + + `sse4_2` without `popcnt` matches this cause. + +3. Change the guest CPU model. + + > **Warning:** Changing the guest CPU model usually needs a VM power cycle, which stops every pod on that node. Precondition: the node is cordoned and drained (`kubectl cordon ` and `kubectl drain --ignore-daemonsets --delete-emptydir-data`). Rollback: set the previous CPU model and power-cycle again. + + In the hypervisor, set a guest CPU model that exposes x86-64-v2, including `popcnt`. Power the VM off and on if the hypervisor applies the model only at boot. No image tag or Kubernetes flag bypasses this check. + +4. After boot, run `kubectl uncordon `. `grep popcnt /proc/cpuinfo` should print a match, and `olm-registry-platform` should stay up. + +### Cause 2: discover-device in nativestor-system is in CrashLoopBackOff and TopoLVM is not in use + +1. Confirm that the cluster has no TopoLVM volumes: + + ```bash + kubectl -n nativestor-system get ds,pods | grep discover-device + kubectl get sc + kubectl get pvc -A | grep -i topolvm || true + ``` + +2. Delete the DaemonSet. + + > **Warning:** Deleting the `discover-device` DaemonSet stops TopoLVM device discovery on every node. Precondition: the checks above show no TopoLVM volumes, nobody expects `nativestor` devices, and the DaemonSet YAML is saved. Skip this step when any PVC is TopoLVM-backed. Rollback: `kubectl apply -f /tmp/discover-device.before.yaml`, or reinstall the TopoLVM plugin if that YAML is incomplete. + + ```bash + kubectl -n nativestor-system get ds discover-device -o yaml > /tmp/discover-device.before.yaml + kubectl -n nativestor-system delete ds discover-device + ``` + +### Cause 3: ClickHouse is still at 2 CPU and 4Gi while the log storage plugin is abnormal + +1. Read the live requests and limits: + + ```bash + kubectl -n cpaas-system get pods | grep -i clickhouse + kubectl -n cpaas-system get pod -o jsonpath='{range .spec.containers[*]}{.name}{" req="}{.resources.requests.cpu}{"/"}{.resources.requests.memory}{" lim="}{.resources.limits.cpu}{"/"}{.resources.limits.memory}{"\n"}{end}' + ``` + +2. Raise the resources on the plugin. + + > **Warning:** Raising the limits restarts the ClickHouse pods, and log search and ingestion pause until they are Ready. Precondition: the live limits are 2 CPU and 4Gi, the log storage plugin is the abnormal one, and the nodes have the extra CPU and memory. Rollback: set the requests and limits back to 2 CPU and 4Gi on the plugin. + + In the console, open Alauda Container Platform Log Storage for ClickHouse, set CPU to 4 and memory to 8Gi for both requests and limits, and save. + +3. After the pods are Ready, watch the utilization and the plugin status. If the pods keep the old values, see [nevermore and ClickHouse log components alert on CPU or memory, restart, or ignore new resource values on ACP](./nevermore_and_ClickHouse_log_components_alert_on_CPU_or_memory_restart_or_ignore_new_resource_values_on_ACP.md). + +### Cause 4: The platform cannot query logs because the global log Kafka is behind the business clusters + +1. Confirm that the global log Kafka is lagging. Read the consumer group lag on the Grafana Kafka dashboards, or describe the groups inside `cpaas-kafka-0` with the listener and client properties that this cluster's brokers expose: + + ```bash + kubectl -n cpaas-system exec cpaas-kafka-0 -- \ + kafka-consumer-groups.sh --bootstrap-server \ + --command-config --describe --all-groups + ``` + +2. Let the backlog drain, and check that log search returns current entries once the lag is low. If the lag keeps growing, the consumer side is too slow for the writers. See [Log Kafka lag from Elasticsearch write hot spots, breaker trips, IK analyzer errors, or too few partitions on ACP](./Log_Kafka_lag_from_Elasticsearch_write_hot_spots_breaker_trips_IK_analyzer_errors_or_too_few_partitions_on_ACP.md). To reduce the load on the global cluster for good, store the business clusters' logs in log storage on those clusters. + +### Cause 5: A custom platform-inspector role can operate WebCLI because its YAML contains acp-podsexec and acp-rootexec + +1. Export the role and look for the two entries: + + ```bash + kubectl get -o yaml > .yaml + cp .yaml .before.yaml + grep -n -e acp-podsexec -e acp-rootexec .yaml + ``` + + Use the object kind that `kubectl get` shows for that role name. + +2. Remove the rules or resource names `acp-podsexec` and `acp-rootexec` from `.yaml`, and keep `get`, `list` and `watch`. The 4.2 platform auditor, which has query verbs only, is the reference. + +3. Apply the trimmed role. + + > **Warning:** Applying the trimmed role removes pod exec and node login from everyone bound to it, including any emergency use of that role. Precondition: you saved the previous YAML, and the extra verbs are the WebCLI access you want to remove. Rollback: `kubectl apply -f .before.yaml`. + + ```bash + kubectl apply -f .yaml + ``` + +## Diagnostic Steps + +1. Check the CPU flags, only on the node that runs `olm-registry-platform`: + + ```bash + kubectl logs -n --tail=50 | grep x86-64-v2 + grep -c popcnt /proc/cpuinfo + ``` + + The glibc line and a count of 0 are Cause 1. + +2. Check `nativestor-system` and the ClickHouse limits: + + ```bash + kubectl -n nativestor-system get pods + kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{range .spec.containers[*]}{.name}{"="}{.resources.limits.cpu}{"/"}{.resources.limits.memory}{" "}{end}{"\n"}{end}' | grep -i clickhouse + ``` + + A crash-looping `discover-device` with no TopoLVM volumes is Cause 2. Limits of `2` and `4Gi` with an abnormal log storage plugin are Cause 3. + +3. For empty log search, check the global Kafka lag before you treat it as a collector problem (Cause 4). Also check whether `nevermore` is `OOMKilled` (`kubectl get pods -A | grep nevermore`, then `kubectl describe pod` for `Reason: OOMKilled`). That is a collection problem: see [nevermore OOMKilled, cache written to /var/cpaas, or callback-secret mount failure on ACP](./nevermore_OOMKilled_cache_written_to_var_cpaas_or_callback-secret_mount_failure_on_ACP.md). + +4. For WebCLI access on a custom inspector, open the role YAML and search for `acp-podsexec` (Cause 5). + +## Related Information + +- Related articles: [Monitoring faults: a second Prometheus, missing vmagent-agent, vmalert OOMKilled, or Prometheus stuck in Init on ACP](./Monitoring_faults_a_second_Prometheus_missing_vmagent-agent_vmalert_OOMKilled_or_Prometheus_stuck_in_Init_on_ACP.md) covers the monitoring rows of the same inspection. +- [Logging ClickHouse runs out of disk, CPU or memory, or is OOMKilled on ACP](./Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md) diff --git a/docs/en/solutions/Log_Kafka_lag_from_Elasticsearch_write_hot_spots_breaker_trips_IK_analyzer_errors_or_too_few_partitions_on_ACP.md b/docs/en/solutions/Log_Kafka_lag_from_Elasticsearch_write_hot_spots_breaker_trips_IK_analyzer_errors_or_too_few_partitions_on_ACP.md new file mode 100644 index 000000000..40e6d2284 --- /dev/null +++ b/docs/en/solutions/Log_Kafka_lag_from_Elasticsearch_write_hot_spots_breaker_trips_IK_analyzer_errors_or_too_few_partitions_on_ACP.md @@ -0,0 +1,207 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.8.x + - 3.14.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Log Kafka lag from Elasticsearch write hot spots, breaker trips, IK analyzer errors, or too few partitions on ACP + +## Issue + +Logs are late or missing, and the platform log Kafka is backed up. Unlike a plain capacity shortage, the lag has a recognizable shape: + +- Two Elasticsearch nodes sit at nearly 100% CPU while a third one idles. +- `lanaya` reports a `Data too large` circuit-breaker message. +- `lanaya` loops on `bulk response error`, and the Elasticsearch log shows IK analyzer offset errors. +- The lag is spread evenly over every partition of a topic that has too few partitions for the clusters it serves, while Elasticsearch has headroom. + +The application log topic is `ALAUDA_LOG_TOPIC`, and its platform consumer group is `alauda_log`. `lanaya` (Deployment in `cpaas-system`) consumes it and bulk-writes to Elasticsearch, taking the next batch of a partition only after the current one is persisted. + +When the lag sits on a single partition, see [Log Kafka lag on a single partition from oversized log entries or a stuck offset on ACP](./Log_Kafka_lag_on_a_single_partition_from_oversized_log_entries_or_a_stuck_offset_on_ACP.md). + +## Root Cause + +### Cause 1: Application log indices have one primary shard, so two nodes carry all writes + +The daily application index (`log-workload-`) is created with one primary shard. It therefore lands on two Elasticsearch nodes, one for the primary and one for the replica. Those two nodes sit at about 96% CPU while the third idles, and Kafka backs up. Scaling Elasticsearch out does not spread the load. The index can stay `yellow` (1 primary, 1 replica, hundreds of millions of documents), with one shard stuck initializing while the index keeps growing. + +The shard count comes from the `log.json` index template (`"index_patterns": "log-*"`, `"index.lifecycle.name": "alauda-log"`). The platform manages that template through key `log.json` of ConfigMap `aiops-logging-config` in `cpaas-system`. Changing `"number_of_shards"` from 1 to `3`, while keeping `"number_of_replicas": 1` and `"refresh_interval": "10s"`, spreads the writes, and log queries return to normal. + +When `lanaya` already has as many replicas as the topic has partitions and the daily index still has a single shard on one hot node, the backlog can also clear on its own once the Elasticsearch load drops. Check the `log.json` template in that case as well. + +### Cause 2: lanaya writes fail with Data too large from the parent circuit breaker + +`lanaya` logs a write error such as: + +```text +Data too large, data for [] would be [8958381455/8.3gb], which is larger than the limit of [8958371430/8.3gb], usages [request=0/0b, fielddata=7652620260/7.1gb, ... +``` + +The Elasticsearch pods are healthy. Fielddata holds most of the breaker budget (7.1 GB of 8.3 GB here), so each bulk write trips the parent breaker, and the displayed logs fall behind. Raising `indices.breaker.total.limit` to `92%` and `indices.breaker.fielddata.limit` to `85%` restores writes. Treat this as a temporary fix. Adding heap is the lasting one. + +### Cause 3: lanaya repeats bulk response error because the Elasticsearch IK analyzer rejects documents + +`lanaya` cannot consume and loops on: + +```text +elasticsearch/elasticsearch.go:166] Database bulk error {"bulkSize": 10000, ... "error": "bulk response error"} +consumer/consumer.go:107] Failed to insert message ... "error": "bulk response error" +consumer/consumer.go:48] Failed to consume message, try to reconnect {"category": "log", ... "error": "bulk response error"} +``` + +The platform Elasticsearch log (for example Elasticsearch 6.8.23 on ACP 3.14.1) repeats `failed to execute bulk item` for `log-platform-` with: + +```text +java.lang.IllegalArgumentException: startOffset must be non-negative, and endOffset must be >= startOffset, and offsets must not go backwards startOffset=846,endOffset=847,lastStartOffset=847 for field 'log_data' +``` + +A line such as `(index) index {[log-platform-][_doc][...], source[n/a, actual length: [2.7kb], max length: 2kb]}` may be the first error found. It only shows a truncated document in the log message and is not the cause. Without the Elasticsearch hotfix, the platform is affected by a known IK analyzer issue that produces these offset errors on `log_data`. Replacing the Elasticsearch image with the fixed one lets `lanaya` consume normally again. + +### Cause 4: The topic has too few partitions for the clusters it serves + +An Elasticsearch with 3 master and 8 data nodes stores logs for 15 clusters. Elasticsearch runs at about 60% memory and 40% CPU, the indices are normal, and the cluster is still behind. `_cluster/health` is `green` with 0 unassigned shards, and every log component is `Running` without restarts. The `lanaya` logs show delays (`dly`) of tens of hours on the log partitions. + +Consumption parallelism is too low for the volume. Expanding the log topic from 30 to 60 partitions makes the platform backlog, and the backlog of external consumer groups, fall steadily. `lanaya` replicas beyond the partition count add nothing, and adding replicas does not help while Elasticsearch is the limit. A `razor` line such as `Validate token error ... the server has asked for the client to provide credentials` in the same period is not related. + +## Resolution + +Load the Elasticsearch credentials once for the API calls below: + +```bash +ES_USER=$(kubectl -n cpaas-system get secret elasticsearch-basic-auth -o jsonpath='{.data.username}' | base64 -d) +ES_PASS=$(kubectl -n cpaas-system get secret elasticsearch-basic-auth -o jsonpath='{.data.password}' | base64 -d) +es() { kubectl -n cpaas-system exec cpaas-elasticsearch-0 -c cpaas-elasticsearch -- \ + curl -sS -u "${ES_USER}:${ES_PASS}" "$@"; } +``` + +### Cause 1: Application log indices have one primary shard, so two nodes carry all writes + +1. Confirm the signature: + + ```bash + es 'http://cpaas-elasticsearch:9200/_cat/shards/log-workload-*?v&s=index' | tail -20 + es 'http://cpaas-elasticsearch:9200/_cat/nodes?v&h=name,cpu,heap.percent' + es 'http://cpaas-elasticsearch:9200/_template?pretty' | grep -n -E '"log-\*"|number_of_shards|number_of_replicas|alauda-log' + ``` + + One primary (`p`) and one replica (`r`) per daily application index, with two nodes near 96% CPU, matches this cause. + +2. Change the shard count in the ConfigMap. The platform reverts direct template edits, so edit it here. + + > **Warning:** Changing `number_of_shards` in the `log.json` template affects every index created from it afterwards. More shards per daily index means more shards in the cluster in total. Precondition: only when the daily application index has one primary and two nodes carry the write load. Rollback: `kubectl apply -f /tmp/aiops-logging-config.before.yaml`; indices that already exist keep the shard count they were created with. + + ```bash + kubectl -n cpaas-system get cm aiops-logging-config -o yaml > /tmp/aiops-logging-config.before.yaml + kubectl -n cpaas-system edit cm aiops-logging-config + ``` + + In key `log.json`, set `"number_of_shards": 3`, and keep `"number_of_replicas": 1` and `"refresh_interval": "10s"`. + +3. Existing indices keep their shard count, so the load spreads from the next daily index onward. + +### Cause 2: lanaya writes fail with Data too large from the parent circuit breaker + +1. Confirm the signature: + + ```bash + kubectl -n cpaas-system logs deploy/lanaya --tail=200 | grep 'Data too large' + es 'http://cpaas-elasticsearch:9200/_nodes/stats/breaker?pretty' | grep -E '"(parent|fielddata)"|limit_size|estimated_size' + ``` + +2. Raise the breaker limits. + + > **Warning:** Raising the breaker limits lets Elasticsearch use more heap before it rejects requests. If the heap is really exhausted, the node can hit a Java `OutOfMemoryError` instead of rejecting the write. Precondition: only when `lanaya` logs `Data too large` and fielddata dominates the usage. Save the current settings first. Rollback: set both keys to `null` to return to the defaults. + + ```bash + es 'http://cpaas-elasticsearch:9200/_cluster/settings?pretty' > /tmp/es-cluster-settings.before.json + es -X PUT -H 'Content-Type: application/json' \ + 'http://cpaas-elasticsearch:9200/_cluster/settings' \ + -d '{"persistent":{"indices.breaker.total.limit":"92%","indices.breaker.fielddata.limit":"85%"}}' + ``` + +3. Writes should resume immediately. If the breaker keeps tripping, add heap by raising the Elasticsearch memory limit on the log storage plugin. + +### Cause 3: lanaya repeats bulk response error because the Elasticsearch IK analyzer rejects documents + +1. Confirm the signature: + + ```bash + kubectl -n cpaas-system logs deploy/lanaya --tail=500 | grep -E 'bulk response error' | tail -5 + for p in $(kubectl -n cpaas-system get pod -l service_name=cpaas-elasticsearch -o name); do + kubectl -n cpaas-system logs "$p" -c cpaas-elasticsearch --tail=2000 | grep -m3 'startOffset must be non-negative' + done + kubectl -n cpaas-system get sts cpaas-elasticsearch -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' + ``` + + `startOffset must be non-negative ... for field 'log_data'` in the Elasticsearch log, together with `bulk response error` in `lanaya`, matches this cause. + +2. Ask Alauda support for the Elasticsearch hotfix image that matches the platform version. Compare its tag with the image printed above to see whether the hotfix is already applied. + +3. Apply the hotfix image. + + > **Warning:** Replacing the Elasticsearch image rolls every Elasticsearch pod, and search and ingestion pause on each pod while it restarts. Precondition: only when the offset error is confirmed, the hotfix image is the one provided for this platform version, and you saved the StatefulSet (and the `rpch` that pins the image, if one exists). Rollback: apply the saved StatefulSet or `rpch` to return to the previous image. + + ```bash + kubectl -n cpaas-system get sts cpaas-elasticsearch -o yaml > /tmp/cpaas-elasticsearch.sts.before.yaml + kubectl get rpch -A | grep -i elasticsearch + ``` + + Apply the image the way support specifies for that version. Then confirm that `lanaya` stops logging `bulk response error` and that the consume rate rises. + +### Cause 4: The topic has too few partitions for the clusters it serves + +1. Read the current partition count. Use the listener and client configuration that this cluster's brokers expose: + + ```bash + kubectl -n cpaas-system exec cpaas-kafka-0 -- \ + kafka-topics.sh --bootstrap-server \ + --command-config --describe --topic ALAUDA_LOG_TOPIC | head -3 + ``` + +2. Add partitions. + + > **Warning:** A partition count can only be increased. New partitions change which partition a key maps to, and they use more broker disk and file handles. Precondition: only when Elasticsearch has headroom (Causes 1 to 3 are ruled out) and the lag persists across many partitions. Rollback: none; a topic cannot shrink its partition count. + + ```bash + kubectl -n cpaas-system exec cpaas-kafka-0 -- \ + kafka-topics.sh --bootstrap-server \ + --command-config --alter --topic ALAUDA_LOG_TOPIC --partitions 60 + ``` + +3. Watch the platform and external consumer groups in the describe output. Both backlogs should fall steadily. Scale `lanaya` only up to the partition count. + +## Diagnostic Steps + +1. Read per-partition lag from the `log/Cpaas Log Stack for Elasticsearch` dashboard or with `kafka-consumer-groups.sh --describe --group alauda_log`. + +2. Check the consumer and the nodes: + + ```bash + kubectl -n cpaas-system logs deploy/lanaya --tail=300 | grep -Ei 'Data too large|bulk response error|Failed to insert' + es 'http://cpaas-elasticsearch:9200/_cat/nodes?v&h=name,cpu,heap.percent' + es 'http://cpaas-elasticsearch:9200/_cat/indices?v' | grep -v green + ``` + +3. Match the signature: + + | What you see | Cause | + |---|---| + | Two nodes near 100% CPU and a third idle, with today's `log-workload-*` index at one primary | Cause 1 | + | `Data too large` with fielddata holding most of the breaker | Cause 2 | + | `bulk response error` in `lanaya` and `startOffset must be non-negative` in the Elasticsearch log | Cause 3 | + | Lag spread evenly over every partition, Elasticsearch well below its limits, and all indices `green` | Cause 4 | + +## Related Information + +- Related articles: [Log Kafka lag on a single partition from oversized log entries or a stuck offset on ACP](./Log_Kafka_lag_on_a_single_partition_from_oversized_log_entries_or_a_stuck_offset_on_ACP.md) covers lag that sits on one partition. +- [Elasticsearch log index replica count reverts after editing the index template on ACP](./Elasticsearch_log_index_replica_count_reverts_after_editing_the_index_template_on_ACP.md) +- [Log search misses entries: IK analyzer tokens, uppercase keywords, clock skew, or page scope on ACP](./Log_search_misses_entries_IK_analyzer_tokens_uppercase_keywords_clock_skew_or_page_scope_on_ACP.md) diff --git a/docs/en/solutions/Log_Kafka_lag_on_a_single_partition_from_oversized_log_entries_or_a_stuck_offset_on_ACP.md b/docs/en/solutions/Log_Kafka_lag_on_a_single_partition_from_oversized_log_entries_or_a_stuck_offset_on_ACP.md new file mode 100644 index 000000000..9d80cd201 --- /dev/null +++ b/docs/en/solutions/Log_Kafka_lag_on_a_single_partition_from_oversized_log_entries_or_a_stuck_offset_on_ACP.md @@ -0,0 +1,149 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.8.x + - 3.14.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Log Kafka lag on a single partition from oversized log entries or a stuck offset on ACP + +## Issue + +Logs are late or missing, and the lag of the platform log Kafka sits on **one** partition of `ALAUDA_LOG_TOPIC`, while the other partitions are at 1 or 2. Typical reports: + +- "Kafka is backed up and log query fails." +- "One partition has all the lag." +- "One partition does not consume logs, and `lanaya` shows many errors." +- The alert `Kafka 消费队列滞后增长速率 > 0 且持续 1 分钟` ("Kafka consumer lag growth rate above 0 for 1 minute") fires again and again. + +The application log topic is `ALAUDA_LOG_TOPIC`, and its platform consumer group is `alauda_log`. `lanaya` (Deployment in `cpaas-system`) consumes the topic and bulk-writes to Elasticsearch. It takes the next batch of a partition only after the current batch is persisted, so one bulk that keeps failing stops that partition. The `lanaya` bulk size is set with the `-es-bulk-size` flag (default 10000). The Elasticsearch settings are in ConfigMap `es-config` in `cpaas-system`. + +Per-partition lag is shown on the Grafana dashboards `log/Cpaas Log Stack for Elasticsearch` (Lag detail by `alauda_log/`) and `Platform Middleware / Kafka Cluster Status`. You can also read it from `kafka-consumer-groups.sh --describe`. + +When the lag is spread over many partitions, or two Elasticsearch nodes carry all the load, see [Log Kafka lag from Elasticsearch write hot spots, breaker trips, IK analyzer errors, or too few partitions on ACP](./Log_Kafka_lag_from_Elasticsearch_write_hot_spots_breaker_trips_IK_analyzer_errors_or_too_few_partitions_on_ACP.md). + +## Root Cause + +### Cause 1: Lag sits on one partition because a few log entries are too large + +All the lag is on the log topic, on one partition. On the `log/Cpaas Log Stack for Elasticsearch` dashboard the production and consumption rates match, while the lag of one partition rises in a straight line past 40,000. Every other partition stays at 1 or 2. The broker logs show only routine segment deletion. Scaling `lanaya` out to more replicas has no effect. + +The `lanaya` log for that partition shows repeated `Try to record message` entries and Kafka reader entries with a very large `"Content-Length"` (about 10 MB). Filtered by the consumer id, the log shows a loop: + +```text +Try to do bulk {"from": 701, "to": 1402, "batch": 701} +Do bulk error ... "error": "[413 Request Entity Too Large] " +Failed to insert message, abort +Try to do consume again +``` + +Elasticsearch returns `413` for the bulk, because some application log entries are individually huge. Raising `http.max_content_length` to `2000mb` in ConfigMap `es-config`, and setting `-es-bulk-size=100` on `lanaya`, makes the partition move again. A small bulk size lowers throughput, so the lasting fix is to stop the application from writing oversized entries, and then return `-es-bulk-size` to its default. + +### Cause 2: One partition stops consuming at a fixed offset, and the stuck record is skipped with kafka-delete-records.sh + +One partition of the log topic does not consume at all. Its `CURRENT-OFFSET` stays at one fixed offset, while `lanaya` logs many errors. If the `lanaya` lines for that partition show the `413` loop, use Cause 1 first. No more specific root cause is known. The remediation is to delete the records of that partition up to its latest offset with `kafka-delete-records.sh`, so that the consumer moves past the record it cannot process. + +## Resolution + +The Kafka commands below run inside `cpaas-kafka-0`. Use the listener and client configuration that this cluster's brokers expose. On some clusters the plaintext `localhost:9092` listener times out, while the TLS listener works with the client properties file mounted in the pod. Do not mix the two. If a script is not on the `PATH`, it is in `/opt/kafka/bin/`. + +### Cause 1: Lag sits on one partition because a few log entries are too large + +1. Find the `lanaya` lines for the lagging partition, then the bulk loop for its consumer: + + ```bash + for p in $(kubectl -n cpaas-system get pods -o name | grep lanaya); do + echo "== $p" + kubectl -n cpaas-system logs "$p" --tail 10000 | grep '"ptn": ""' | grep log | tail -3 + done + kubectl -n cpaas-system logs --tail 10000 | grep '' | grep -E 'Try to do bulk|413|abort' | tail -10 + ``` + + Replace `` with the lagging partition, and `` with the id shown on those lines (for example `log-01-04i`). A `413 Request Entity Too Large` loop for that consumer matches this cause. + +2. Save the current configuration: + + ```bash + kubectl -n cpaas-system get cm es-config -o yaml > /tmp/es-config.before.yaml + kubectl -n cpaas-system get deploy lanaya -o yaml > /tmp/lanaya.before.yaml + ``` + +3. Raise the request size limit and lower the bulk size. + + > **Warning:** A larger `http.max_content_length` lets one request use more heap, and a smaller `-es-bulk-size` lowers `lanaya` throughput on every partition. The `lanaya` edit restarts its pods. `http.max_content_length` is a static Elasticsearch node setting, so it takes effect only after the Elasticsearch pods restart. Restart them one at a time. Elasticsearch does not accept a value above about 2 GB, so do not go past `2000mb`. Precondition: only when this cause is confirmed (lag on one partition and `413` in `lanaya` for it), and the Elasticsearch heap is about half of its memory limit. Rollback: apply `/tmp/es-config.before.yaml` and `/tmp/lanaya.before.yaml`, then restart Elasticsearch one pod at a time. + + ```bash + kubectl -n cpaas-system edit cm es-config + kubectl -n cpaas-system edit deploy lanaya + ``` + + Set `http.max_content_length: 2000mb` in `es-config`, and add `-es-bulk-size=100` to the `lanaya` arguments. Restart the Elasticsearch pods one at a time: + + ```bash + for p in $(kubectl -n cpaas-system get pod -l service_name=cpaas-elasticsearch -o name); do + kubectl -n cpaas-system delete "$p"; sleep 15 + kubectl -n cpaas-system wait --for=condition=Ready "$p" --timeout=15m + done + ``` + +4. After the partition catches up and the application no longer writes oversized entries, remove `-es-bulk-size` again. + +### Cause 2: One partition stops consuming at a fixed offset, and the stuck record is skipped with kafka-delete-records.sh + +1. Read the stuck offset and the log-end offset of each partition: + + ```bash + kubectl -n cpaas-system exec cpaas-kafka-0 -- \ + kafka-consumer-groups.sh --bootstrap-server \ + --command-config --describe --group alauda_log + ``` + + A partition whose `CURRENT-OFFSET` does not move between runs, while the other partitions advance, matches this cause. + +2. Delete the records of that partition up to the log-end offset. + + > **Warning:** `kafka-delete-records.sh` permanently deletes every record in that partition below the offset you give. With the log-end offset, the whole unconsumed range of that partition never reaches the log store. Precondition: only when one partition is stuck at a fixed offset while the others consume, Cause 1 is ruled out, and losing those records is accepted. Rollback: none. + + ```bash + kubectl -n cpaas-system exec -i cpaas-kafka-0 -- sh -c 'cat > /tmp/delete-records.json' <<'EOF' + {"partitions": [{"topic": "ALAUDA_LOG_TOPIC", "partition": , "offset": }], "version": 1} + EOF + kubectl -n cpaas-system exec cpaas-kafka-0 -- \ + kafka-delete-records.sh --bootstrap-server \ + --command-config --offset-json-file /tmp/delete-records.json + ``` + + Use the topic and partition that the describe output named. + +3. Run the describe again and confirm that `CURRENT-OFFSET` moves. + +## Diagnostic Steps + +1. Read per-partition lag from the dashboard, or with the group describe in Cause 2. + +2. Check the consumer: + + ```bash + kubectl -n cpaas-system logs deploy/lanaya --tail=300 | grep -Ei '413|Failed to insert|abort' + ``` + +3. Match the signature: + + | What you see | Cause | + |---|---| + | Lag on one partition, and `lanaya` repeats `413 Request Entity Too Large` for it | Cause 1 | + | Lag on one partition whose `CURRENT-OFFSET` stays at one fixed offset, without the `413` loop | Cause 2 | + | Lag on many partitions, or two Elasticsearch nodes near 100% CPU | See the related article | + +## Related Information + +- Related articles: [Log Kafka lag from Elasticsearch write hot spots, breaker trips, IK analyzer errors, or too few partitions on ACP](./Log_Kafka_lag_from_Elasticsearch_write_hot_spots_breaker_trips_IK_analyzer_errors_or_too_few_partitions_on_ACP.md) covers lag spread over many partitions and lag caused by the Elasticsearch side. +- [Logs, audit and events stop while the log Kafka is degraded on ACP](./Logs_audit_and_events_stop_while_the_log_Kafka_is_degraded_on_ACP.md) diff --git a/docs/en/solutions/Log_query_shows_no_logs_HTTP_500_slow_long_ranges_or_an_empty_real-time_log_on_ACP.md b/docs/en/solutions/Log_query_shows_no_logs_HTTP_500_slow_long_ranges_or_an_empty_real-time_log_on_ACP.md new file mode 100644 index 000000000..c1bbb943f --- /dev/null +++ b/docs/en/solutions/Log_query_shows_no_logs_HTTP_500_slow_long_ranges_or_an_empty_real-time_log_on_ACP.md @@ -0,0 +1,192 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Log query shows no logs, HTTP 500, slow long ranges, or an empty real-time log on ACP + +## Issue + +The log store holds the data, but the console does not show it. Typical reports: + +- "No logs in Operations Center, although `razor`, ClickHouse and `nevermore` are normal and their logs show no errors." +- "ClickHouse runs normally, and the ClickHouse database contains the data." +- "The real-time log is empty, and the raw logs are garbled." +- "A keyword is found over the last 6 hours but shows no data over the last day." +- "A filter that matches 59 lines still takes two minutes." +- "Cluster logs are empty after we moved to ClickHouse, and the browser shows HTTP 500." + +The failure is on the query or display path between the console and the storage, not in collection. Open the browser developer tools on the failing page first. The status and body of the log query request decide which cause applies. + +Keyword, time-window and page-scope mistakes are covered in [Log search misses entries: IK analyzer tokens, uppercase keywords, clock skew, or page scope on ACP](./Log_search_misses_entries_IK_analyzer_tokens_uppercase_keywords_clock_skew_or_page_scope_on_ACP.md). + +## Root Cause + +### Cause 1: Operations Center shows no logs while the whole pipeline is healthy, because the clock of the computer running the browser is wrong + +After log center, ClickHouse and the log collector were installed (ACP 4.3.3), Operations Center shows no logs, even with product logs and Kubernetes logs both selected in the collector. `nevermore` uploads normally, and the `vector`, ClickHouse, `courier-api` and `alb` logs show no errors. A manual query in ClickHouse finds current rows. Synchronizing the node clocks and reinstalling the components with their data cleared does not help. + +The system time of the computer that opens the console is wrong, and that alone is enough to leave Operations Center empty. The log pipeline itself works. + +### Cause 2: The real-time log view stays empty because a reverse proxy in front of the platform does not forward WebSocket + +The pod's real-time log in the console is empty or never loads, and raw logs may look garbled. The backend pod returns the data without error. Two breaks can be on this path: + +- A node clock is out of sync, which makes log API authentication fail, and the platform gateway blocks the WebSocket requests. Calibrating the time fixes this part. +- An Nginx reverse proxy was added in front of the platform VIP without WebSocket forwarding settings, so the `wss` handshake never passes. Going to the platform VIP directly restores the live log. + +### Cause 3: A query over a long time range returns no data or takes minutes, because Elasticsearch or the storage is overloaded + +A keyword finds matches over the last 6 hours but shows no data over the last day. The global log storage had 3 log nodes with a 1 core / 2Gi limit, and one project kept logs for 30 days. A 3-day keyword query on one namespace matched about 14 million entries. The page also states that only the first 10,000 matched entries can be viewed and asks for a narrower filter. When the range holds a large log volume, filtering for one entry overwhelms Elasticsearch and the query fails. + +On ClickHouse log storage, a filtered query that matched only about 59 lines took almost 2 minutes, and the query memory chart spiked to 6 to 7 GiB again and again. The storage was NAS. The log component capacity planning documentation gives the benchmark disk as 6000 IOPS and 250 MB/s read/write on a dedicated SSD mount. Storage below that needs a larger-scale configuration with more CPU and memory. An `fio` test against those numbers showed that the disk was too slow. + +When every Elasticsearch query is slow and the Elasticsearch logs report errors, the component lacks the capacity for the query load. Raising the Elasticsearch memory limit triggers a rolling update, after which index queries return to normal and the errors stop. + +### Cause 4: Queries return HTTP 500 and global apollo logs cache error after the store was switched to ClickHouse + +Elasticsearch was replaced by ClickHouse on a workload cluster. ClickHouse holds the log rows, but the console shows `共 0 条日志` ("0 logs in total"). In the browser developer tools, requests to `logs?action=aggregation&type=workload...` fail with: + +```text +code: 500, message: "unknown action: aggregation", reason: "Unknown", status: "Failure" +``` + +`apollo` on the global cluster logs `cache error`. Global `apollo` reaches the workload cluster's ClickHouse through that cluster's load balancer `/clickhouse` endpoint. A known issue keeps `apollo` on a stale cache of the cluster's data-access information after the switch. Restarting `apollo` on the global cluster makes the logs queryable. + +The ClickHouse replicas may also log `RaftInstance: session ... failed to read rpc header from socket ... End of file` repeatedly, and Vector may log ClickHouse health-check failures. Neither causes the 500. + +### Cause 5: The global cluster cannot reach the business cluster apiserver on 6443 + +The business cluster's `nevermore` is Ready, `razor` logs authentication failures, and the global `cluster-transformer` cannot reach the business cluster's API VIP on port 6443. Queries for that cluster return nothing. Opening that path from all global nodes, not only the first control-plane node, restores queries. + +Port 11781 is another port that centralized logging uses between the global and business clusters. A closed 11781 can appear beside the 6443 break without being its cause. Do not disable 11781 on a centralized-logging design that still needs it. Port 11780 on the log cluster's load balancer is the log sink port from platform 4.0 onward, so log shipping connections land on it. A connection count on 11780 that rises from almost 0 to several thousand after an upgrade to 4.0 is expected. + +## Resolution + +### Cause 1: Operations Center shows no logs while the whole pipeline is healthy, because the clock of the computer running the browser is wrong + +1. Compare the clock of the computer that runs the browser with a cluster node. On a node: + + ```bash + date -u + ``` + + On a Windows client, `w32tm /stripchart /computer: /samples:3` shows the offset from a time server. A client clock that is off while the nodes agree matches this cause. + +2. Correct the client's date, time and time zone, or enable automatic time synchronization on it (on Windows, `w32tm /resync` after a time source is configured). + +3. Reload the console and query again. Opening the console from another computer with a correct clock is a quick confirmation. Check this before you reinstall any log component. + +### Cause 2: The real-time log view stays empty because a reverse proxy in front of the platform does not forward WebSocket + +1. Check the node clocks first with `date -u` on each node, and fix any skew. + +2. Open the browser developer tools on the pod log page and look at the `wss` request. A handshake that fails only through the proxy, and works against the platform VIP directly, matches this cause. + +3. Add WebSocket forwarding to the Nginx `location` that proxies the platform, for example: + + ```nginx + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + proxy_read_timeout 3600s; + ``` + + Reload Nginx and retry. Until the proxy is fixed, use the platform VIP directly. + +### Cause 3: A query over a long time range returns no data or takes minutes, because Elasticsearch or the storage is overloaded + +1. Narrow the time range, and add exact conditions (cluster, namespace, pod) before the keyword. + +2. If even small result sets take minutes, measure the log storage disk on the log storage node, against the filesystem of the data directory: + + ```bash + fio --name=randwrite --directory= --rw=randwrite --bs=4k --size=1G \ + --ioengine=libaio --direct=1 --iodepth=32 --runtime=60 --time_based --group_reporting + ``` + + Remove the test file afterwards. Compare the result with the documented benchmark of 6000 IOPS and 250 MB/s on a dedicated SSD. A disk below the requirement has to be replaced, or the log storage sized up with more CPU and memory. Storing each cluster's logs in that cluster also spreads the query load. + +3. When every Elasticsearch query is slow and the Elasticsearch log shows errors, compare its memory with the limit: + + ```bash + kubectl -n cpaas-system top pod | grep cpaas-elasticsearch + kubectl -n cpaas-system logs cpaas-elasticsearch-0 -c cpaas-elasticsearch --tail=100 | grep -E 'WARN|ERROR' + ``` + + > **Warning:** Raising the Elasticsearch memory limit rolls every Elasticsearch pod, and search and ingestion pause on each pod while it restarts. Precondition: only when queries are slow across the board, Elasticsearch memory is near its limit, and the nodes can give the new limit. Rollback: set the previous limit back on the log storage plugin. + + Change the memory limit on the log storage plugin, and let the rolling update finish before you test again. + +### Cause 4: Queries return HTTP 500 and global apollo logs cache error after the store was switched to ClickHouse + +1. On the global cluster, check `apollo`: + + ```bash + kubectl -n cpaas-system get pods | grep apollo + kubectl -n cpaas-system logs deploy/apollo --tail=100 | grep -i 'cache error' + ``` + +2. Restart `apollo`. If `get pods` shows a different owner kind, use that kind in the restart. + + > **Warning:** Restarting `apollo` on the global cluster briefly interrupts the console API calls it serves. Precondition: only when ClickHouse on the workload cluster holds current rows, the browser shows HTTP 500 with `unknown action: aggregation` on log queries, and global `apollo` logs `cache error`. Rollback: none needed; the rollout does not change the spec. + + ```bash + kubectl -n cpaas-system rollout restart deploy/apollo + kubectl -n cpaas-system rollout status deploy/apollo + ``` + +### Cause 5: The global cluster cannot reach the business cluster apiserver on 6443 + +1. From each global node, test the business cluster API VIP (the address in that cluster's kubeconfig): + + ```bash + nc -vz 6443 + nc -vz 11781 + ``` + +2. Restore the 6443 path (security group, route or VIP listener). Queries should return without a pod restart. + +3. Leave 11781 enabled when centralized logging is in use. Disable the 11781 load-balancer health check only after you have confirmed that nothing listens on it and the design does not need the port. + +## Diagnostic Steps + +1. Open the browser developer tools on the failing page and read the log query request: + + | What you see | Cause | + |---|---| + | No HTTP error, current rows in ClickHouse or Elasticsearch, every component healthy, and still no logs | Cause 1, compare the browser computer's clock with a node | + | A `wss` request that fails while `kubectl logs` works | Cause 2 | + | A narrow range works and a wide one fails, or small results take minutes | Cause 3 | + | HTTP 500 with `unknown action: aggregation` | Cause 4, read global `apollo` for `cache error` | + | Queries empty for one business cluster only, with `razor` authentication failures there | Cause 5, test 6443 from the global nodes | + +2. To confirm that the store holds the rows, query ClickHouse directly from a replica pod: + + ```bash + kubectl -n cpaas-system exec -it chi-cpaas-clickhouse-replicated-0-0-0 -c clickhouse -- clickhouse-client + ``` + + ```sql + SELECT timestamp, log_data FROM observability.log_workload_all + WHERE match(namespace, '') AND match(workload, '') + AND timestamp > now() - INTERVAL 1 HOUR + ORDER BY timestamp DESC LIMIT 20; + ``` + + Rows here with nothing on the page point to the display path (Causes 1, 2 and 4). + +3. A 401 or 403 on the query request is an authentication problem and not covered here. + +## Related Information + +- [Log search misses entries: IK analyzer tokens, uppercase keywords, clock skew, or page scope on ACP](./Log_search_misses_entries_IK_analyzer_tokens_uppercase_keywords_clock_skew_or_page_scope_on_ACP.md) +- [Logging ClickHouse runs out of disk, CPU or memory, or is OOMKilled on ACP](./Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md) diff --git a/docs/en/solutions/Log_search_misses_entries_IK_analyzer_tokens_uppercase_keywords_clock_skew_or_page_scope_on_ACP.md b/docs/en/solutions/Log_search_misses_entries_IK_analyzer_tokens_uppercase_keywords_clock_skew_or_page_scope_on_ACP.md new file mode 100644 index 000000000..d80927db1 --- /dev/null +++ b/docs/en/solutions/Log_search_misses_entries_IK_analyzer_tokens_uppercase_keywords_clock_skew_or_page_scope_on_ACP.md @@ -0,0 +1,177 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.16.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x + - 4.4.x +--- + +# Log search misses entries: IK analyzer tokens, uppercase keywords, clock skew, or page scope on ACP + +## Issue + +Elasticsearch or ClickHouse log storage is up, the indices or tables exist, and the console still shows nothing, the wrong time window, or lines that do not match the keyword. Typical reports: + +- "A Chinese sentence I copied from the log cannot be found." +- "Searching `ERROR` in uppercase shows unrelated logs, but lowercase works." +- "The project filter also shows another project." +- "Logs from `cpaas-system` cannot be seen in the console." +- "Only the last 30 minutes are shown." + +Use this article when a keyword that you can see in `kubectl logs` does not match, when two nodes disagree on the date, or when the page shows a different scope from the line you are looking for. If logs cannot be viewed at all, first check the per-node agent: an `OOMKilled` or restarting `nevermore` is a collection problem, not a query problem. If the Kafka consumer lag is hours, fix the lag first; a deployment name that appears only after you widen the time range is a sign of lag, not of a query bug. + +## Root Cause + +### Cause 1: On Elasticsearch storage, the IK analyzer lowercases text and splits long Chinese strings + +With Elasticsearch and the IK analyzer, the console search is a token query, not a `grep` of the raw line. + +- **Uppercase does not match.** The tokens are lowercase, so searching the lowercase form returns the line. Wrapping the original text in quotes does not turn it into an exact-string query. For example, logs that contain `CN=cluster-operator,O=io.strimzi` are found by `cn` and not by `CN` (`共 0 条日志`, "0 logs in total"). +- **A Chinese compound fails when the dictionary did not emit that compound.** `资产` ("asset") and `管理` ("management") match, but `资产管理` ("asset management") does not. Two-character pieces match more often than a long phrase, so split the phrase (for example search `没有` ("not") instead of `没有执行` ("not executed")). +- **Chinese quotation marks, URLs and `/` are split as well**, so the original string misses. An unquoted search that returns nothing and a quoted search that returns inaccurate rows are the same keyword-matching limitation. +- **A long alphanumeric ID can lose its digits at query time, depending on the IK build.** For example, `RF20260928140102047100190` matched only `RF` on one 4.2 environment and matched exactly on another. With one IK build (8.19.10), the query-time `ik_smart` analyzer drops the numeric part, so the query becomes `rf`. Another build (8.19.14) keeps the whole string. The index-time `ik_max_word` analyzer keeps the full string and its fragments on both. On the affected build, quoted searches returned hundreds of thousands of entries that merely share the `RF` prefix. The unquoted lowercase ID returned only exact matches, and the unquoted uppercase ID returned nothing. +- **A Chinese sentence copied from a displayed line returns nothing, while showing all logs displays it.** Log queries follow the Elasticsearch tokenization, so some Chinese phrases can be found and others cannot. This behavior cannot be changed while the log storage is Elasticsearch. ClickHouse log storage supports full-text search and does not use IK. The log content field accepts `AND` or `OR` combinations, for example `"the connection is unavailable" AND "rpc error"`, but not both operators in one query. + +No index setting is known that disables IK in place. + +### Cause 2: Node clocks disagree, so the selected range is empty + +`nevermore` stamps records with one node's clock, and the store is queried with another. When those clocks are far apart (for example months), searching either window returns nothing, because the pipeline's "now" and the application's "now" are not the same range. + +On an Elasticsearch cluster, the same skew creates indices whose names and `@timestamp` values do not line up. The cluster stays `yellow`. Deleting the data does not help, because the next index is wrong again until the clocks match. An existing yellow index cannot be repaired into a correct time range. + +The clock that matters can also be the client's. When node clocks agree and the pipeline is healthy, a wrong system time on the computer that runs the browser leaves Operations Center empty. See [Log query shows no logs, HTTP 500, slow long ranges, or an empty real-time log on ACP](./Log_query_shows_no_logs_HTTP_500_slow_long_ranges_or_an_empty_real-time_log_on_ACP.md). + +### Cause 3: The page has a different scope from the line you are searching + +Several "no data" reports are about the console scope, not a broken indexer: + +- **Product logs.** Namespace `cpaas-system` holds platform product logs. Observability, Logs lists application logs only. With product-log collection enabled, those lines are under Platform Management, Operations Center, Logs. +- **Default time window.** Observability, Logs opens on the last 30 minutes, which hides every earlier line. Change the time range in the upper-right corner. The lines were not removed by retention. +- **Pod filter.** A pod chosen in the query filter applies only after the confirm button in that filter is clicked. Without it, results come from other pods. +- **Wrong project.** A project named `default` reached through a non-standard entry path may not exist, so a namespace's logs look empty there. Switch to the project that owns the namespace and select the namespace there. +- **One cluster at a time.** An administrator query covers one cluster at a time. There is no cross-cluster aggregate. +- **Export limit.** Export stops at one million rows. Narrow the time range and the workload, export in batches, and merge the files outside the platform. To export one project's logs from the built-in Elasticsearch, filter Operations Center, Logs by that project and export the result. +- **Missing component on 4.0.** A data-service component such as RabbitMQ is missing from the component dropdown on 4.0 when its pod has no Project label, because the query joins pod, namespace and project by that label. This is fixed in 4.1. On 4.0, add the Project label that the other visible components carry. +- **ALB history.** An ALB pod in `cpaas-system` shows only its real-time log under the cluster's pods. Its history is under Platform Management, Operations Center, Logs, with Log Type `Product Logs`, a wider time range, and a filter for the ALB pod. The page shows only the first 10,000 matched logs. +- **ALB access logs.** Inbound ALB access logs are product logs. Set the log type to Product Logs, filter by the load balancer's component name as the component dropdown lists it (for example `alb2-cpaas-system`), and enter `client_addr` in Log Content. Each match is a JSON access record with fields such as `client_addr`, `host`, `request`, `status` and `upstream_addr`. Without product-log collection, read the recent records from the `nginx` container of the ALB pod (`kubectl logs -c nginx`). +- **GBK bytes.** The raw-log view shows GBK bytes, while the live tail, which is decoded as UTF-8, looks fine. Forcing the raw response to UTF-8 fixes the display. The file on disk is not corrupt. +- **NUL bytes.** A line full of `0x00` bytes renders as empty or as `NUL`, depending on the viewer. The zeros come from the application's write or its log rotation, not from the store. +- **Garbled live view before 4.0.** Chinese characters that are intermittently garbled only in the pod's real-time log view come from one UTF-8 character being split across two transmissions. This is fixed in platform 4.0. + +### Cause 4: On ClickHouse storage, an uppercase keyword returns unrelated lines, or a project filter also returns a project with a longer name + +Two behaviors of a ClickHouse-backed query page: + +- Searching the log content for an uppercase keyword such as `ERROR` returns lines that do not contain it (`[WARN ]` lines, retry messages, stack-trace lines), while the same keyword in lowercase works. A direct ClickHouse query with `log_data ILIKE '%ERROR%'` for the same pod and time range returns only correct lines, so the stored data is right. This is a product bug in the log query parsing. It affects v4.3.1 to v4.3.5 and v4.4.0 to v4.4.1, and it is fixed in v4.4.2 and later and in v4.5. +- Filtering by project `` also returns logs of a project named `-`. The project keyword is matched fuzzily by design. + +## Resolution + +### Cause 1: On Elasticsearch storage, the IK analyzer lowercases text and splits long Chinese strings + +1. Search with the lowercase English token, or with the short Chinese token that IK emits. Quotes around the original uppercase or compound string do not change the analyzer. For a long alphanumeric ID on an IK build that drops the digits, use the unquoted lowercase ID. + +2. To see which tokens a build produces, run `_analyze` with `ik_smart` against a log index: + + ```bash + ES_USER=$(kubectl -n cpaas-system get secret elasticsearch-basic-auth -o jsonpath='{.data.username}' | base64 -d) + ES_PASS=$(kubectl -n cpaas-system get secret elasticsearch-basic-auth -o jsonpath='{.data.password}' | base64 -d) + kubectl -n cpaas-system exec cpaas-elasticsearch-0 -c cpaas-elasticsearch -- \ + curl -sS -u "${ES_USER}:${ES_PASS}" -H 'Content-Type: application/json' \ + 'http://cpaas-elasticsearch:9200//_analyze?pretty' \ + -d '{"analyzer": "ik_smart", "text": ""}' + ``` + +3. If the requirement is an exact raw substring, move that cluster's log storage to ClickHouse. Confirm which store the cluster has before you treat this as a bug: + + ```bash + kubectl -n cpaas-system get pods | grep -Ei 'elastic|clickhouse' + ``` + +### Cause 2: Node clocks disagree, so the selected range is empty + +1. On each node that runs `nevermore` or the log store, compare the clocks: + + ```bash + date -u + ``` + + A gap of minutes is enough to push a short window off the record. A gap of months means the log center cannot be the source of truth until NTP is fixed. + +2. Fix time synchronization. + + > **Warning:** Stepping a node clock affects certificates, leases and audit order, not only log search. Precondition: the skew is confirmed with `date -u` on the `nevermore` node and on the ClickHouse or Elasticsearch node, and you have a maintenance window for chrony or the hypervisor clock. Rollback: restore the previous NTP configuration. Do not set the clock of a single control-plane node by hand. + +3. If the business requires clocks that differ, use the workload's live log (`kubectl logs`), and do not expect the log center to reconcile the ranges. Yellow Elasticsearch indices created by the skew stay yellow, and deleting them without fixing NTP recreates them. + +### Cause 3: The page has a different scope from the line you are searching + +Use the entry and filter that match the line, as listed in Root Cause: + +- For `cpaas-system` product logs, use Platform Management, Operations Center, Logs. +- Widen the upper-right time range before you investigate Kafka or the store. +- Click the confirm button after you choose a pod. +- Export in windows that stay under one million rows. +- For GBK, set the raw-log request charset to UTF-8. +- For NUL bytes, count them in the file inside the container: + + ```bash + kubectl exec -n -c -- \ + python3 -c 'import pathlib; b=pathlib.Path("").read_bytes(); print(b.count(b"\x00"))' + ``` + + A large count means the application or its log rotation wrote the zeros. Fix that writer, because reindexing stores the same bytes. + +### Cause 4: On ClickHouse storage, an uppercase keyword returns unrelated lines, or a project filter also returns a project with a longer name + +1. Confirm that the stored rows are correct by querying ClickHouse directly from a replica pod: + + ```bash + kubectl -n cpaas-system exec -it chi-cpaas-clickhouse-replicated-0-0-0 -c clickhouse -- clickhouse-client + ``` + + ```sql + SELECT timestamp, log_data + FROM observability.log_workload_all + WHERE match(namespace, '') AND match(workload, '') + AND timestamp > now() - INTERVAL 1 HOUR + AND log_data ILIKE '%ERROR%' + ORDER BY timestamp DESC + LIMIT 20; + ``` + + If the database name differs on this version, list it with `SHOW DATABASES`. + +2. Correct rows here, and unrelated rows on the page for the same uppercase keyword, match the parsing bug. Until the platform is upgraded to v4.4.2 or later (or v4.5), search with the lowercase form. + +3. To trace one page request, open the browser console (F12) and run the search. Copy the `request-id` from the response headers of the search request, and look it up in the `razor-api` logs on the global cluster: + + ```bash + kubectl -n cpaas-system get pods | grep razor-api + kubectl -n cpaas-system logs | grep '' + ``` + +4. For the project filter, nothing needs fixing. When two project names share a prefix, narrow the result with the namespace or workload filter. + +## Diagnostic Steps + +1. Decide whether the record exists. A green Elasticsearch `_cat/indices`, or a ClickHouse table with today's rows, means the indexer ran, and this article applies. An empty index together with Kafka lag means a collection or consumption problem. + +2. Repeat the query with the lowercase or short token (Cause 1) and with a wide time range. A page left on the default last 30 minutes is Cause 3, not a missing index. If the wide range works and the node clocks differ, it is Cause 2. + +3. On ClickHouse storage, an uppercase keyword that returns unrelated lines while a direct `ILIKE` query returns the right rows is Cause 4. So is a project filter that also returns a project whose name starts with the same text. + +4. A single namespace or component missing while everything else is searchable is Cause 3. For one business cluster that returns nothing, test the 6443 path from the global cluster as described in [Log query shows no logs, HTTP 500, slow long ranges, or an empty real-time log on ACP](./Log_query_shows_no_logs_HTTP_500_slow_long_ranges_or_an_empty_real-time_log_on_ACP.md). + +## Related Information + +- [Log query shows no logs, HTTP 500, slow long ranges, or an empty real-time log on ACP](./Log_query_shows_no_logs_HTTP_500_slow_long_ranges_or_an_empty_real-time_log_on_ACP.md) +- [Log Kafka lag from Elasticsearch write hot spots, breaker trips, IK analyzer errors, or too few partitions on ACP](./Log_Kafka_lag_from_Elasticsearch_write_hot_spots_breaker_trips_IK_analyzer_errors_or_too_few_partitions_on_ACP.md) diff --git a/docs/en/solutions/Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md b/docs/en/solutions/Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md new file mode 100644 index 000000000..b73ca87d3 --- /dev/null +++ b/docs/en/solutions/Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md @@ -0,0 +1,244 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.10.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Logging ClickHouse runs out of disk, CPU or memory, or is OOMKilled on ACP + +## Issue + +The log store is ClickHouse, not Elasticsearch, and one of these happens: + +- A log query returns a memory error, or the console shows no logs. +- A node hits 100% disk, and Kube-OVN, backup and the log pods fail together. +- ClickHouse pods restart with exit code `137`, or with `OOMKilled` in their events. + +Two different ClickHouse installations appear in these problems, and cleaning one does not fix the other: + +- The **logging ClickHouse** holds application, audit and platform logs (pods `chi-cpaas-clickhouse-replicated---0` in `cpaas-system`). +- The **MySQL slow-query ClickHouse** is `chi-pxc-ck-slowsql-cluster` (StatefulSet `chi-pxc-ck-slowsql-cluster-0-0`, pod `chi-pxc-ck-slowsql-cluster-0-0-0`). Its data is often on `/cpaas/ck`. + +A third kind is a ClickHouse that a team runs with its own ClickHouseInstallation in its own namespace (container `clickhouse-pod`). Cause 6 covers it. + +## Root Cause + +### Cause 1: ClickHouse slow-query logs filled the node disk + +`du` on the affected node stops in the slow-query volume, not in Elasticsearch. These files grow even when no MySQL pod is present, because the slow-query ClickHouse keeps its own log tables, including `system.query_log`. Other platform pods then fail because the node cannot write. + +A `pxc` `chi-ck` pod in `cpaas-system` that is abnormal and does not start is this default slow-query deployment as well. When nobody uses it, stop it by setting its replicas to 0. To remove the feature completely, uninstall its operator (its CSV and Subscription go away with it), set the ClickHouse replica count to 0, and delete the PVC. The operator does not recreate the instance after that. + +### Cause 2: The logging ClickHouse is out of CPU or memory + +Both CPU and memory of the logging ClickHouse are at 100%, and the console cannot query. Scaling out the cluster restores queries. A smaller example is one ClickHouse node at 2 CPU / 4Gi receiving logs from two clusters: query memory runs out, and raising that node's resources stops the overflow. Neither situation involves a corrupt table, so add capacity before you consider a reinstall. + +Raising memory does not always help. In one cluster, a log query kept exceeding the memory limit after the plugin memory was raised from 4G to 10G. The error stopped only after the environment was scaled out. A `MEMORY_LIMIT_EXCEEDED` with a `maximum` of about 3.6 GiB at a 4Gi limit is covered in [nevermore and ClickHouse log components alert on CPU or memory, restart, or ignore new resource values on ACP](./nevermore_and_ClickHouse_log_components_alert_on_CPU_or_memory_restart_or_ignore_new_resource_values_on_ACP.md). + +### Cause 3: One logging node filled, and after expansion the audit tables are read-only + +The pod restart loop says the disk is full. After expansion the pod is `Running`, and the console works for a while if you widen the time range. Then queries fail again. The database log on that node shows errors, some audit tables are read-only (`is_readonly = 1`), and the three nodes do not hold the same amount of data. + +A likely way to get there is an earlier uninstall and reinstall in which only some of the nodes had their data cleaned. Expansion alone does not make the replicas match. The repair is to remove the bad node from the ClickHouse cluster, join it again, and reinstall ClickHouse. + +A size difference between replicas is not this cause by itself. Replicas of the 3-replica logging ClickHouse normally differ by tens of gigabytes. + +### Cause 4: System component logs dominate the ClickHouse volume + +Most of the stored volume comes from `cpaas-system` workloads. Their log level can be lowered directly on the component. The collector has no switch that drops one namespace's debug lines, so the volume has to be cut at the source or kept for fewer days. Do not change the log level of the MetalLB components without first confirming the change with product support. + +### Cause 5: Logging ClickHouse pods exit with code 137 and the plugin memory limit is too low + +All three logging ClickHouse pods restart repeatedly, with container exit code `137`. Raising the ClickHouse plugin memory limit stops the restarts. CPU and memory utilization can stay high afterwards. + +Exit code `137` means only that the container received SIGKILL. A kernel out-of-memory kill is normally reported as `Reason: OOMKilled`. With `Reason: Error`, the kubelet may have killed the container after failed liveness probes. If the events repeat `Liveness probe failed` on `:8123/ping` with `connection refused` for days, and `clickhouse-operator` in `ck-system` is in `CrashLoopBackOff`, the database is broken rather than short of memory. See [ClickHouse log queries fail after the log storage had been working: missing tables or broken parts on ACP](./ClickHouse_log_queries_fail_after_the_log_storage_had_been_working_missing_tables_or_broken_parts_on_ACP.md) (Cause 5). The same article covers tables that are still missing after the restarts stop (Cause 2). + +### Cause 6: A ClickHouse managed by its own ClickHouseInstallation restarts with OOMKilled + +The container of a ClickHouse that is not the logging plugin keeps restarting, and its events show OOM. The instance is a `ClickHouseInstallation` in its own namespace, for example with pod template `clickhouse:21.8` and container `clickhouse-pod`. The fix is made on the ClickHouseInstallation, because the operator rebuilds the StatefulSet from it. Raise the memory requests and limits of `clickhouse-pod` (to 8Gi in a confirmed fix). Set `max_server_memory_usage` below the limit (6Gi, `6442450944`), so that ClickHouse refuses work before the kernel kills it. + +## Resolution + +### Cause 1: ClickHouse slow-query logs filled the node disk + +1. Confirm which volume is full before you delete anything: + + ```bash + df -hT / /cpaas /cpaas/ck + kubectl -n cpaas-system get sts,pod,pvc | grep -i slowsql + ``` + +2. If the slow-query instance is in use, find the largest tables and truncate only the log table at the top of that list (usually `system.query_log`): + + ```bash + kubectl -n cpaas-system exec chi-pxc-ck-slowsql-cluster-0-0-0 -c clickhouse-pod -- clickhouse-client --query \ + "SELECT database, table, formatReadableSize(sum(bytes_on_disk)) AS size FROM system.parts WHERE active GROUP BY database, table ORDER BY sum(bytes_on_disk) DESC LIMIT 10" + ``` + + > **Warning:** `TRUNCATE` permanently removes all rows of that table. Precondition: only for a ClickHouse system log table (such as `system.query_log`) that the list above shows as the largest. Never truncate tables of the `observability` database. Rollback: none. + + ```bash + kubectl -n cpaas-system exec chi-pxc-ck-slowsql-cluster-0-0-0 -c clickhouse-pod -- clickhouse-client --query "TRUNCATE TABLE system.query_log" + ``` + + Do not delete files under `/cpaas/ck` by hand while the slow-query pod runs. + +3. If the slow-query instance is not used, stop it instead. + + > **Warning:** Setting the replica count to 0 and deleting the PVC removes the slow-query history and stops slow-query collection. MySQL pods are not affected. Uninstalling the operator also removes its CSV and Subscription, so reinstall the operator if another ClickHouse still needs it. Precondition: the pod you stop is `chi-pxc-ck-slowsql-cluster-0-0-0`, not the logging cluster, and nobody uses the slow-query console. Rollback: scale the replicas back to the previous count. Data on the deleted PVC does not return. + + ```bash + kubectl -n cpaas-system scale sts/chi-pxc-ck-slowsql-cluster-0-0 --replicas=0 + kubectl -n cpaas-system delete pvc + ``` + + If an operator scales the StatefulSet back up, set the replicas to 0 on the ClickHouseInstallation that owns it, uninstall that operator, and delete the PVC again. + +### Cause 2: The logging ClickHouse is out of CPU or memory + +1. Check usage on the logging pods. Ignore pods that match `slowsql`. + + ```bash + kubectl -n cpaas-system top pod | grep -i clickhouse + kubectl -n cpaas-system get pods -o wide | grep -i clickhouse + ``` + +2. Add capacity. + + > **Warning:** Raising the resources of the logging ClickHouse, or scaling it, restarts replicas, and queries fail over while they roll. Precondition: `top` shows CPU or memory near 100% on the logging pods (not on `chi-pxc-ck-slowsql-cluster`), and the nodes have the extra CPU and memory. Rollback: set the previous values on the plugin again. + + Set the new CPU and memory on the Log Storage for ClickHouse cluster plugin, so the plugin does not write the old values back. No single size is prescribed. Size from `top` and the free memory on the nodes. A node that ingests logs from two clusters needs a larger size even when average CPU looks moderate. When more memory does not stop query memory errors, scale the environment out instead. + +### Cause 3: One logging node filled, and after expansion the audit tables are read-only + +1. Expand the disk or the PVC so the pod stays `Running`. A wider time range in the console only proves that some parts are readable. + +2. On each replica, compare the read-only flags. Newer images use `clickhouse client`, older images `clickhouse-client`. + + ```bash + kubectl -n cpaas-system exec -c clickhouse -- clickhouse-client --query \ + "SELECT database, table, is_readonly, absolute_delay FROM system.replicas ORDER BY is_readonly DESC" + ``` + +3. Rejoin the bad node. + + > **Warning:** Removing the bad logging node and reinstalling ClickHouse drops parts that existed only on that node, and log search is down during the reinstall. Precondition: the disk is already large enough for the pod to run, `system.replicas` shows read-only audit tables on that node, console queries still fail after expansion, and you have a maintenance window. A replica whose size differs from the other two is not enough by itself. Rollback: none for parts that were only on the removed node. + + Take the abnormal node out of the ClickHouse cluster, join it again, and reinstall the log storage plugin through the original install path. Do not empty the data on the healthy replicas to force them to match, and do not uninstall again without removing the bad node. + +### Cause 4: System component logs dominate the ClickHouse volume + +1. Rank namespaces and components by log volume in the console. + +2. Lower the log level on the `cpaas-system` Deployments that dominate. Use the setting that each component already documents: a flag such as `--v`, an environment variable such as `LOG_LEVEL`, or a configuration file. Leave the MetalLB components at their shipped level. + +3. If the volume cannot be cut at the source, shorten the log retention in the log storage settings. + +### Cause 5: Logging ClickHouse pods exit with code 137 and the plugin memory limit is too low + +1. Read the last termination and the probe events: + + ```bash + kubectl -n cpaas-system get pod -o jsonpath='{range .status.containerStatuses[*]}{.name}{" exit="}{.lastState.terminated.exitCode}{" reason="}{.lastState.terminated.reason}{"\n"}{end}' + kubectl -n cpaas-system describe pod | grep -i 'liveness probe failed' + kubectl -n ck-system get pods + ``` + + If the liveness probe on `8123` has failed for days and `clickhouse-operator` is in `CrashLoopBackOff`, stop here and use the article in Cause 5 of Root Cause. + +2. Raise the memory limit. + + > **Warning:** Raising memory on the logging ClickHouse rolls its pods, and queries and inserts pause on each pod while it restarts. Precondition: the pods are the logging cluster, not `chi-pxc-ck-slowsql-cluster`, the last exit code is 137, and the nodes have free memory for the new limit. Rollback: set the previous memory limit again. + + Set the new memory limit on the ClickHouse log storage plugin, sized from the usage that led to exit code 137 and from the free memory on the node. If the plugin form is not available, the StatefulSet can be changed directly, but the plugin can write it back: + + ```bash + kubectl -n cpaas-system set resources sts/ \ + -c --limits=memory= + ``` + + Read the container name with `kubectl -n cpaas-system get pod -o jsonpath='{.spec.containers[*].name}'`. + +3. Watch that the restarts stop. + +### Cause 6: A ClickHouse managed by its own ClickHouseInstallation restarts with OOMKilled + +1. Confirm the kill reason and the current memory, and save the ClickHouseInstallation: + + ```bash + kubectl -n get chi + kubectl -n get pod -o jsonpath='{range .status.containerStatuses[*]}{.name}{" exit="}{.lastState.terminated.exitCode}{" reason="}{.lastState.terminated.reason}{"\n"}{end}' + kubectl -n get pod -o jsonpath='{range .spec.containers[*]}{.name}{" req="}{.resources.requests.memory}{" lim="}{.resources.limits.memory}{"\n"}{end}' + kubectl -n get chi -o yaml > /tmp/.before.yaml + ``` + + Only `reason=OOMKilled` on the ClickHouse container (or OOM in the pod events) matches this cause. `Error` with a failing probe on `8123` does not. + +2. Edit the ClickHouseInstallation. + + > **Warning:** Editing the ClickHouseInstallation makes the operator roll the ClickHouse pods of that instance, which stops serving queries and inserts while each pod restarts. Precondition: only when `OOMKilled` is confirmed, the backup above is saved, and the node has the new memory free. Rollback: `kubectl apply -f /tmp/.before.yaml` returns the previous resources and settings, and the pods roll again. + + ```bash + kubectl -n edit chi + ``` + + Change only these three things: `max_server_memory_usage` under `spec.configuration.settings`, and the memory requests and limits of the ClickHouse container in the pod template. Use this instance's pod template name and container name: + + ```yaml + spec: + configuration: + settings: + max_server_memory_usage: 6442450944 # 6Gi, below the 8Gi limit + templates: + podTemplates: + - name: clickhouse:21.8 + spec: + containers: + - name: clickhouse-pod + resources: + requests: + memory: 8Gi + limits: + memory: 8Gi + ``` + + 8Gi and 6Gi are example values. Size both from the instance's usage, and keep `max_server_memory_usage` below the limit. + +3. Watch that the restarts stop. + +## Diagnostic Steps + +1. Separate the ClickHouse installations: + + ```bash + kubectl get chi -A + kubectl -n cpaas-system get pods,sts,pvc | grep -Ei 'clickhouse|slowsql|chi-' + du -sh /cpaas/ck 2>/dev/null + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | `/cpaas/ck` full with `slowsql` pods, or an abnormal `pxc` `chi-ck` pod that nobody uses | Cause 1 | + | Logging pods at 100% CPU or memory | Cause 2 | + | One logging node with read-only audit tables after expansion | Cause 3 | + | `cpaas-system` components dominate the log volume in the console | Cause 4 | + | Logging ClickHouse pods whose last exit code is 137, without days of failed liveness probes on `8123` | Cause 5 | + | A ClickHouse outside the logging plugin with container `clickhouse-pod`, last state `OOMKilled` | Cause 6, change the ClickHouseInstallation and not the StatefulSet | + +3. The monitoring graphs of the node can stay low while a container is killed. `dmesg` reports `Memory cgroup out of memory` when a container hits its own limit, regardless of node memory. + +## Related Information + +- [ClickHouse log queries fail after the log storage had been working: missing tables or broken parts on ACP](./ClickHouse_log_queries_fail_after_the_log_storage_had_been_working_missing_tables_or_broken_parts_on_ACP.md) +- [nevermore and ClickHouse log components alert on CPU or memory, restart, or ignore new resource values on ACP](./nevermore_and_ClickHouse_log_components_alert_on_CPU_or_memory_restart_or_ignore_new_resource_values_on_ACP.md) +- [ClickHouse log storage logs RaftInstance errors with pod restarts, merge CPU spikes, or a hand-built ClickHouse on ACP](./ClickHouse_log_storage_logs_RaftInstance_errors_with_pod_restarts_merge_CPU_spikes_or_a_hand-built_ClickHouse_on_ACP.md) diff --git a/docs/en/solutions/Logs_audit_and_events_stop_while_the_log_Kafka_is_degraded_on_ACP.md b/docs/en/solutions/Logs_audit_and_events_stop_while_the_log_Kafka_is_degraded_on_ACP.md new file mode 100644 index 000000000..26ad2a65b --- /dev/null +++ b/docs/en/solutions/Logs_audit_and_events_stop_while_the_log_Kafka_is_degraded_on_ACP.md @@ -0,0 +1,214 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Logs, audit and events stop while the log Kafka is degraded on ACP + +## Issue + +Platform logs, audit records and events stop arriving, and the log storage Kafka is part of the reason. Typical reports: + +- "The Events tab in the application view is empty, and pods cannot be debugged." +- "Logs, audit and events stop six days ago." +- "One of three Kafka nodes failed, log collection stopped, and `razor` memory spiked." +- "The log disk was full, we deleted the data, and the observability page still shows no logs." + +The broker is the log storage Kafka in `cpaas-system` (StatefulSet `cpaas-kafka`, pods `cpaas-kafka-0..2`, with `cpaas-zookeeper`). Its data is on node-local storage, typically `/cpaas/data/kafka0`, plus `/cpaas/data/kafka` and `/cpaas/data/zookeeper`. `lanaya` consumes the log topics into Elasticsearch. `razor` writes audit, events and logs into Kafka on clusters where it is deployed. + +When a broker does not start at all, see [cpaas-kafka broker fails to start: corrupt segment, no space, broker id conflict, or deleted StatefulSet on ACP](./cpaas-kafka_broker_fails_to_start_corrupt_segment_no_space_broker_id_conflict_or_deleted_StatefulSet_on_ACP.md). + +## Root Cause + +### Cause 1: cpaas-kafka-0 is abnormal with a large kafka0 directory, and logs, audit and events stop + +The application Events tab is empty, and a pod that cannot start cannot be debugged from the console. `kubectl describe pod` and `kubectl logs` from the backend still show the events and logs. Logs, audit records and events in platform management reach back only to a date several days earlier. The log storage plugin is stuck in `部署中` ("deploying"). + +`cpaas-kafka-0` is `0/1 Running`, while the other brokers, ZooKeeper, Elasticsearch and `razor` are `1/1`. The Grafana dashboard Platform Middleware / Log Center Plugin Status shows only 2 brokers up, and consumer group `alauda_audit` far behind on every `ALAUDA_AUDIT_TOPIC` partition. `du -sh *` under `/cpaas/data/` on the log storage node shows `kafka0` at tens of gigabytes. + +After the owner confirmed that the accumulated data could be discarded, the consumers and Kafka were stopped and the `kafka0` directory was emptied. The log storage plugin then recovered, and the Events page showed events again. + +A large `kafka0` alone is not this cause. When every broker is `Running` and the lag is still moving, it is consumption lag, which drains without deleting segments once `lanaya` and Elasticsearch catch up. + +### Cause 2: One broker fails and every write fails, and razor is OOMKilled, because each partition needs two in-sync replicas + +On a three-broker cluster, one broker failed. Log collection stopped, `razor` memory climbed, and all `razor` pods restarted with last state `Terminated`, `OOMKilled`, exit code `137`. Before the restart, `razor` repeated: + +```text +server/server.go:78 | server | Failed to write kafka {"category": "logs", "error": "context deadline exceeded"} +``` + +It did not move to the healthy brokers. Its `--kafka-endpoint` listed all three broker addresses on port `9092`, so the client configuration is not the fault. + +The broker configuration has `default.replication.factor=2` and `min.insync.replicas=2`. `kafka-topics.sh --describe` on `ALAUDA_LOG_TOPIC` shows replication factor 2 and `min.insync.replicas: 2`, and the producers use `acks=all`. Every partition has one of its two replicas on the failed broker, so every ISR drops to 1, below the minimum, and every write is refused. `razor` buffers what it cannot send until it is OOMKilled. When the broker comes back, memory and collection recover. + +For `acks=all` producers, a topic tolerates one broker down only when the replication factor minus `min.insync.replicas` is at least 1. For example, factor 3 with minimum 2, or factor 2 with minimum 1. + +### Cause 3: After a full log disk was cleaned, the observability page still shows no logs until Kafka and ZooKeeper start again in order + +The log disk filled, and data was deleted to free it. The observability page still shows no logs. Deleting the Kafka and ZooKeeper data does not help, and neither does deleting the Elasticsearch, Kafka and ZooKeeper data and then the Elasticsearch log indices. Logs become visible again after `nevermore` is restarted and the Kafka and ZooKeeper data is deleted once more. + +The failure is tied to the startup order of ZooKeeper, Kafka and Elasticsearch. Start ZooKeeper first, then Kafka once ZooKeeper is fully ready, then Elasticsearch once Kafka is fully ready. It is not known which of the last steps made the difference, so the Resolution repeats all of them in that order. + +## Resolution + +The Kafka commands below run inside `cpaas-kafka-0`. Use the listener and client properties file that this cluster's brokers expose. If a script is not on the `PATH`, it is in `/opt/kafka/bin/`. + +### Cause 1: cpaas-kafka-0 is abnormal with a large kafka0 directory, and logs, audit and events stop + +1. Confirm the signature. On the log storage node: + + ```bash + cd /cpaas/data && du -sh * + ``` + + On the cluster: + + ```bash + kubectl get pod -A -owide | grep -E 'kafka|razor|zookeeper|elas' + ``` + + In Grafana, Platform Middleware / Log Center Plugin Status, check how many brokers are up and the lag per consumer group. + +2. Record the replica counts, then stop the consumers and Kafka: + + ```bash + kubectl -n cpaas-system get deploy lanaya + kubectl -n cpaas-system get sts cpaas-kafka + kubectl -n cpaas-system scale deploy lanaya --replicas=0 + kubectl -n cpaas-system scale sts cpaas-kafka --replicas=0 + ``` + + If the cluster also runs `razor` against this Kafka, stop it as well. + +3. Empty `kafka0`. + + > **Warning:** Emptying `kafka0` discards every log, audit and event record still in Kafka, and those records never reach the store. Precondition: only when `cpaas-kafka-0` is abnormal, the directory is large, the owner has confirmed that the accumulated data can be discarded, and the consumers and Kafka are stopped. Rollback: none. + + On each log node: + + ```bash + cd /cpaas/data/kafka0 && rm -rf ./* + ``` + +4. Start Kafka, then the consumers, with their previous replica counts: + + ```bash + kubectl -n cpaas-system scale sts cpaas-kafka --replicas= + kubectl -n cpaas-system scale deploy lanaya --replicas= + ``` + + When the plugin is healthy, open an application's Events tab. + +### Cause 2: One broker fails and every write fails, and razor is OOMKilled, because each partition needs two in-sync replicas + +1. Confirm the signature: + + ```bash + kubectl -n cpaas-system exec cpaas-kafka-0 -- \ + kafka-topics.sh --bootstrap-server \ + --command-config --describe --topic ALAUDA_LOG_TOPIC + kubectl -n cpaas-system logs -p | grep 'Failed to write kafka' + kubectl -n cpaas-system describe pod | grep -E 'OOMKilled|Restart Count' + ``` + + `ReplicationFactor: 2` with `min.insync.replicas=2`, and an `Isr` of one broker for every partition while one broker is down, match this cause. + +2. Bring the failed broker back first. That alone restores collection. + +3. To tolerate one broker down in the future, change the topic configuration together with product support, in a maintenance window. Read the current topic configuration: + + ```bash + kubectl -n cpaas-system exec cpaas-kafka-0 -- \ + kafka-configs.sh --bootstrap-server \ + --command-config --describe --entity-type topics --entity-name ALAUDA_LOG_TOPIC + ``` + + > **Warning:** Lowering `min.insync.replicas` to 1 lets a write succeed with a single copy, so losing that broker before replication catches up loses those records. Raising the replication factor instead needs a partition reassignment and more disk on every broker. Precondition: only when this cause is confirmed and the change is agreed. Rollback: `kafka-configs.sh ... --alter --entity-type topics --entity-name ALAUDA_LOG_TOPIC --delete-config min.insync.replicas` returns the topic to the broker default. + + Check `ALAUDA_AUDIT_TOPIC` and `ALAUDA_EVENT_TOPIC` the same way, because `razor` also writes audit records and events. + +### Cause 3: After a full log disk was cleaned, the observability page still shows no logs until Kafka and ZooKeeper start again in order + +1. Record the replica counts, confirm that the disk has room, and check whether the log storage workloads came up together after the cleanup: + + ```bash + df -hT /cpaas /cpaas/data + kubectl -n cpaas-system get sts cpaas-zookeeper cpaas-kafka cpaas-elasticsearch + kubectl -n cpaas-system get pods -o wide | grep -E 'cpaas-zookeeper|cpaas-kafka|cpaas-elasticsearch|nevermore' + ``` + +2. Restart the collector. + + > **Warning:** Restarting DaemonSet `nevermore` pauses log, audit and event collection on each node while its pod restarts. Precondition: only when logs are still missing after the disk cleanup and the log storage pods are `Running`. Rollback: none needed; the DaemonSet recreates the pods. + + ```bash + kubectl -n cpaas-system rollout restart ds/nevermore + kubectl -n cpaas-system rollout status ds/nevermore + ``` + +3. If logs are still missing, stop Elasticsearch, Kafka and ZooKeeper: + + ```bash + kubectl -n cpaas-system scale sts cpaas-elasticsearch cpaas-kafka cpaas-zookeeper --replicas=0 + kubectl -n cpaas-system get pods | grep -E 'cpaas-zookeeper|cpaas-kafka|cpaas-elasticsearch' + ``` + +4. Empty the Kafka and ZooKeeper directories. + + > **Warning:** Emptying `/cpaas/data/zookeeper`, `/cpaas/data/kafka` and `/cpaas/data/kafka0` deletes every log, audit and event record still in Kafka and the ZooKeeper state of the log cluster, and those records never reach storage. Elasticsearch indices are not touched. Precondition: only when the log disk was already cleaned, logs are still not visible after the `nevermore` restart, and all three StatefulSets are at 0 replicas. Rollback: none. + + On each Kafka node, empty the directories and keep them: + + ```bash + cd /cpaas/data/zookeeper && rm -rf ./* + cd /cpaas/data/kafka && rm -rf ./* + cd /cpaas/data/kafka0 && rm -rf ./* + ``` + +5. Start ZooKeeper, then Kafka, then Elasticsearch, and wait for each to be fully ready before the next. Use the replica counts you recorded: + + ```bash + kubectl -n cpaas-system scale sts cpaas-zookeeper --replicas= + kubectl -n cpaas-system rollout status sts/cpaas-zookeeper + kubectl -n cpaas-system scale sts cpaas-kafka --replicas= + kubectl -n cpaas-system rollout status sts/cpaas-kafka + kubectl -n cpaas-system scale sts cpaas-elasticsearch --replicas= + kubectl -n cpaas-system rollout status sts/cpaas-elasticsearch + ``` + + If `kubectl get deploy,sts` shows Elasticsearch as a Deployment on this cluster, scale `deploy/cpaas-elasticsearch` instead. + +6. When all pods are `Running`, query current logs on the observability page. + +## Diagnostic Steps + +1. Read the broker and consumer state: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep -E 'kafka|zookeeper|razor|lanaya' + kubectl -n cpaas-system logs -p cpaas-kafka-0 | tail -30 + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | `cpaas-kafka-0` at `0/1` with a very large `kafka0` and old data, and the plugin stuck deploying | Cause 1 | + | Every broker `Running` and lag still moving | Consumption lag, do not empty `kafka0` | + | All brokers but one up, writes refused, and `razor` `OOMKilled` | Cause 2, read the topic describe for the factor and `min.insync.replicas` | + | Every log pod `Running` after a full-disk cleanup, and still no logs on the observability page | Cause 3 | + +## Related Information + +- Related articles: [cpaas-kafka broker fails to start: corrupt segment, no space, broker id conflict, or deleted StatefulSet on ACP](./cpaas-kafka_broker_fails_to_start_corrupt_segment_no_space_broker_id_conflict_or_deleted_StatefulSet_on_ACP.md) covers brokers that do not start. +- [Log Kafka lag on a single partition from oversized log entries or a stuck offset on ACP](./Log_Kafka_lag_on_a_single_partition_from_oversized_log_entries_or_a_stuck_offset_on_ACP.md) +- [Elasticsearch log storage Pending or every cluster loses logs after the log disk fills on ACP](./Elasticsearch_log_storage_Pending_or_every_cluster_loses_logs_after_the_log_disk_fills_on_ACP.md) diff --git a/docs/en/solutions/Monitoring_faults_a_second_Prometheus_missing_vmagent-agent_vmalert_OOMKilled_or_Prometheus_stuck_in_Init_on_ACP.md b/docs/en/solutions/Monitoring_faults_a_second_Prometheus_missing_vmagent-agent_vmalert_OOMKilled_or_Prometheus_stuck_in_Init_on_ACP.md new file mode 100644 index 000000000..7d9e2a6c7 --- /dev/null +++ b/docs/en/solutions/Monitoring_faults_a_second_Prometheus_missing_vmagent-agent_vmalert_OOMKilled_or_Prometheus_stuck_in_Init_on_ACP.md @@ -0,0 +1,161 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.16.x + - 4.2.x + - 4.3.x +--- + +# Monitoring faults: a second Prometheus, missing vmagent-agent, vmalert OOMKilled, or Prometheus stuck in Init on ACP + +## Issue + +A cluster inspection reports monitoring items as abnormal (the report may only say `集群巡检项异常`, "cluster inspection item abnormal", without naming a row), or platform monitoring data looks wrong, and the cause is a real fault rather than a false positive. One of these is in the report, the events, or a pod log: + +- Events or the API server log contain `apiserver was unable to write a JSON response: http: Handler timeout` for `GET /apis/monitoring.coreos.com/v1/prometheusrules`. An application team has deployed its own Prometheus in the cluster. +- The VictoriaMetrics row on a global cluster (including a standby global cluster) says that the `vmagent-agent` Deployment does not exist. +- `vmalert` or another `vm*` container in `cpaas-system` restarts with `OOMKilled`, and monitoring for many clusters drops in and out. +- The cluster monitoring component is abnormal and cannot start (`集群监控组件异常 无法启动`). One Prometheus replica stays in `Init`, and its events say that the data volume cannot be mounted. + +When the report names no specific row, run the first check in Diagnostic Steps before you ask for more detail. It is read-only and separates these causes from each other. + +## Root Cause + +### Cause 1: A second Prometheus makes the API server time out on prometheusrules + +Events and API server logs show `apiserver was unable to write a JSON response: http: Handler timeout` while serving `GET /apis/monitoring.coreos.com/v1/prometheusrules`. A Prometheus installed by an application team runs in the cluster next to the platform monitoring stack. That second Prometheus is what makes platform monitoring and the inspection data wrong. Removing it restores platform monitoring. The platform monitoring release in `cpaas-system` is not the one to remove. + +### Cause 2: deployments.apps "vmagent-agent" not found + +The VictoriaMetrics AppRelease in `cpaas-system` stays `NotReady` with the event `deployments.apps "vmagent-agent" not found`. `kubectl get deploy -n cpaas-system vmagent-agent` confirms that the Deployment is gone. The VMAgent custom resource named `agent` can still exist, with status `expanding`. The `vm-operator` logs show no error or warning, the audit logs show no delete of that Deployment, and no resource patch (`rpch`) explains it. Updating the `victoriametrics` AppRelease, so that the controller renders the chart again, brings the Deployment back. Why the Deployment disappeared is not known. + +### Cause 3: A VictoriaMetrics monitoring container is OOMKilled because its memory limit is too low + +A global VictoriaMetrics monitoring container restarts with `OOMKilled` (for example `vmalert-alert` in `cpaas-system`, with both pods abnormal). Several business clusters store their metrics on the global cluster, and the memory limit of that component is too low for the load. Raising the memory limit on the monitoring component resolves it. This is not Cause 2: here the Deployment exists and is being killed. + +### Cause 4: One Prometheus replica is stuck in Init because its data volume cannot be mounted on a failing storage backend + +The monitoring component is reported abnormal and cannot start. The stack is Prometheus (not VictoriaMetrics). One Prometheus replica stays in `Init`, and the pod events show that the Prometheus database volume cannot be mounted. The PVC is on Ceph storage, and the storage side has a disk problem. After the storage problem is repaired, the volume mounts and monitoring returns to normal, with no change to the monitoring plugin. This is not an OOM restart: the container never starts, so there is no `OOMKilled` last state. + +## Resolution + +### Cause 1: A second Prometheus makes the API server time out on prometheusrules + +1. Find the Prometheus instances that are not the platform release: + + ```bash + kubectl get prometheus -A + kubectl get deploy,sts -A | grep -i prometheus + ``` + +2. Remove the application team's Prometheus. + + > **Warning:** Uninstalling that Prometheus stops the application team's metrics, rules and any dashboard that reads it. Precondition: the `Handler timeout` line is present, and you have identified the object by namespace and name as not belonging to the platform monitoring in `cpaas-system`. Rollback: reinstall it from the team's manifest. Leave the platform monitoring plugin in place. + + Uninstall it with the same method that installed it: the team's Helm release, or `kubectl delete` of their `Prometheus` and its operator. + +3. Check that `GET /apis/monitoring.coreos.com/v1/prometheusrules` answers again and that the inspection items that depend on platform monitoring clear. + +### Cause 2: deployments.apps "vmagent-agent" not found + +1. Confirm that the Deployment is really absent: + + ```bash + kubectl -n cpaas-system get deploy vmagent-agent + kubectl -n cpaas-system describe ars victoriametrics + kubectl -n cpaas-system get vmagent,ars | grep -i agent + ``` + +2. Save the AppRelease, then update it. + + > **Warning:** Updating the `victoriametrics` AppRelease renders the chart again and can roll the VictoriaMetrics pods, and scrapes pause during that rollout. Precondition: the Deployment is missing, and you saved the AppRelease YAML on the global cluster where the event appears. Rollback: `kubectl apply -f /tmp/victoriametrics-ars.before.yaml`. Update the AppRelease; do not delete it. + + ```bash + kubectl -n cpaas-system get ars victoriametrics -o yaml > /tmp/victoriametrics-ars.before.yaml + ``` + + Run the VictoriaMetrics plugin update in the console, so that this object is rewritten and the chart renders again. + +3. Check that `vmagent-agent` exists and that the AppRelease becomes Ready: + + ```bash + kubectl -n cpaas-system get deploy vmagent-agent + kubectl -n cpaas-system get ars victoriametrics + ``` + +### Cause 3: A VictoriaMetrics monitoring container is OOMKilled because its memory limit is too low + +1. Confirm that the container's last state is `OOMKilled`: + + ```bash + kubectl -n cpaas-system get pods | grep -E 'vmagent|vmalert|vmstorage|vminsert' + kubectl -n cpaas-system get pod -o jsonpath='{range .status.containerStatuses[*]}{.name}{" last="}{.lastState.terminated.reason}{"\n"}{end}' + kubectl -n cpaas-system get pod -o jsonpath='{range .spec.containers[*]}{.name}{" lim="}{.resources.limits.memory}{"\n"}{end}' + ``` + +2. Raise the memory limit. + + > **Warning:** Raising the limit rolls the pods of that component, and alerting or scraping pauses briefly while they restart. Precondition: `OOMKilled` is confirmed, and the node has free memory for the new limit. Rollback: set the previous limit on the plugin again. + + Raise the memory limit of the OOMKilled component on the monitoring plugin that owns the pod, so that the plugin does not write the old value back. Size it from the usage that led to the kill and from the free memory on the node. + +3. Watch that the restarts stop. + +### Cause 4: One Prometheus replica is stuck in Init because its data volume cannot be mounted on a failing storage backend + +1. Find out which monitoring stack the cluster uses and which pod does not start: + + ```bash + kubectl -n cpaas-system get pods | grep -E 'prometheus|vmstorage|vmselect|vmagent|vmalert' + kubectl -n cpaas-system describe pod | sed -n '/Events/,$p' + ``` + +2. From the events, take the PVC behind the volume that fails to mount, and find its StorageClass and provisioner: + + ```bash + kubectl -n cpaas-system get pod -o jsonpath='{range .spec.volumes[*]}{.name}{" "}{.persistentVolumeClaim.claimName}{"\n"}{end}' + kubectl -n cpaas-system get pvc -o wide + kubectl get sc -o jsonpath='{.provisioner}{"\n"}' + ``` + +3. Repair the storage backend, not the monitoring component. For Rook Ceph, check the cluster health and the OSDs first: + + ```bash + kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s + kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd tree + ``` + + A down OSD or a failed disk is a storage repair. Follow the procedures for that backend. + +4. After the storage side is healthy, watch the Prometheus pod leave `Init`. No change to the monitoring plugin or its resources is needed. + +## Diagnostic Steps + +1. Read the lines that separate the causes, before you uninstall or update anything: + + ```bash + kubectl get events -A --sort-by=.lastTimestamp | grep -i 'Handler timeout' || true + kubectl -n kube-system logs kube-apiserver- --tail=100 | grep -F 'prometheusrules' || true + kubectl get prometheus -A + kubectl -n cpaas-system get deploy vmagent-agent + kubectl -n cpaas-system get pods | grep -E 'prometheus|vmalert|vmagent|vmstorage' + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | `http: Handler timeout` on the `prometheusrules` path, and a Prometheus that is not the platform release | Cause 1 | + | `deployments.apps "vmagent-agent" not found` | Cause 2 | + | `OOMKilled` on a `vm*` container | Cause 3 | + | A Prometheus pod in `Init` whose events report a volume mount failure | Cause 4, check the storage backend of its PVC | + +3. A red monitoring row with healthy pods and only a slow scrape or query is usually a false positive, not one of these causes. + +## Related Information + +- Related articles: [Inspection faults: x86-64-v2 CPU error, discover-device crash loop, small ClickHouse limits, log backlog, or WebCLI role on ACP](./Inspection_faults_x86-64-v2_CPU_error_discover-device_crash_loop_small_ClickHouse_limits_log_backlog_or_WebCLI_role_on_ACP.md) covers the other real faults that a cluster inspection reports. +- [vmagent fails to scrape node-exporter 9100 metrics because its basicAuth does not match node-exporter on ACP](./vmagent_fails_to_scrape_node-exporter_9100_metrics_because_its_basicAuth_does_not_match_node-exporter_on_ACP.md) diff --git a/docs/en/solutions/Vector_log_forwarding_to_external_Kafka_or_OpenTelemetry_fails_on_ACP.md b/docs/en/solutions/Vector_log_forwarding_to_external_Kafka_or_OpenTelemetry_fails_on_ACP.md new file mode 100644 index 000000000..92def0847 --- /dev/null +++ b/docs/en/solutions/Vector_log_forwarding_to_external_Kafka_or_OpenTelemetry_fails_on_ACP.md @@ -0,0 +1,240 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# Vector log forwarding to external Kafka or OpenTelemetry fails on ACP + +## Issue + +Vector forwards platform logs to a target outside the platform, such as an external Kafka or an OpenTelemetry (OTLP) endpoint, and one of the following happens: + +- The Vector Kafka sink fails SASL authentication (for example the broker answers with a password or credentials error), although the user, the password and the permissions are correct. +- Vector exits immediately with status 78 and `No provider for SASL mechanism GSSAPI` when the destination Kafka uses Kerberos. +- The external Kafka topic is auto-created but stays empty. +- The platform log page goes blank right after forwarding to Kafka was configured, while the external Kafka receives logs. +- Platform log search fails after compute nodes were added, on a cluster that uses Vector forwarding. +- Ingress logs need to be forwarded to an OTLP endpoint. + +The in-cluster log pipeline can be healthy while the forwarding path is not. Vector runs as the `vector` container of the `razor` pod in `cpaas-system`. Its configuration is in Secret `cpaas-vector` (key `vector.yaml`), and changes to it may be carried by a resource patch (`rpch`). + +## Root Cause + +### Cause 1: The Vector Kafka sink fails SASL authentication because the sink enables TLS and the broker does not + +Vector forwards logs to a Kafka with authentication enabled, and authentication fails. The Kafka user has full permissions, the username and password are correct, and TLS is not enabled on the Kafka listener. The sink has `sasl.enabled: true` with `mechanism: PLAIN`. It also has `tls.enabled: true` with `verify_certificate: false`, `verify_hostname: false`, and `librdkafka_options` `"enable.ssl.certificate.verification": "false"`. + +The sink speaks TLS to a plaintext listener, so the SASL exchange never completes, and the failure looks like an authentication error. Setting `tls.enabled: false` in the sink fixes it, and forwarding returns to normal. + +### Cause 2: The Vector librdkafka build has no GSSAPI, so a Kerberos Kafka sink exits with status 78 + +Vector loads its configuration and exits with status 78. The error is: + +```text +Sink "output_logs_kafka": creating kafka producer failed: Client creation error: No provider for SASL mechanism GSSAPI: recompile librdkafka with libsasl2 or openssl support. Current build options: PLAIN SASL_SCRAM OAUTHBEARER. +``` + +A warning that transform `dispatcher_unmatched` has no consumers may appear first, and it is not the failure. The librdkafka in the Vector image is built with `PLAIN`, `SASL_SCRAM` and `OAUTHBEARER` only. The SASL mechanism `GSSAPI` (Kerberos) is not in this build, so the Kafka producer is not created. No setting enables GSSAPI in this image. + +### Cause 3: The external Kafka topic is created and stays empty because the Vector condition matches only one cluster + +The log storage is centralized ClickHouse on the global cluster. Platform log queries work, `razor` is `Running`, and Vector forwards to an external Kafka. Kafka auto-created the topic, which proves connectivity, but the topic holds no logs. The forwarding configuration's `condition` filters events by `.cluster`, and as written it does not cover the clusters whose logs are wanted. Rewriting it as an OR over all those cluster names, `(.cluster == "" || .cluster == "")`, delivers logs from all of them. + +### Cause 4: The platform log page is empty after ClickHouse-to-Kafka forwarding was configured, while the external Kafka receives logs + +Logs arrive continuously in the external Kafka, so collection and forwarding work, but the platform log page shows nothing from ClickHouse. `razor` logs no error, ClickHouse runs well below its limits, and restarting `razor` changes nothing. + +The applied Vector configuration kept the `clickhouse` sink (`endpoint: "http://cpaas-clickhouse:8123"`, `database: observability`, `table: log_kubernetes_all`, `auth.strategy: "basic"`, `skip_unknown_fields: true`, a disk buffer). It added an `output_logs_kafka` sink of `type: kafka`. A special character somewhere in that configuration is suspected but has not been confirmed. Deleting the `rpch` that carried the configuration restores the default, and logs display again. Re-applying the configuration carefully keeps them displaying. + +### Cause 5: Log search fails after new nodes until Vector forwarding is disabled and enabled again + +Log search from the platform fails after compute nodes are added (for example 10 nodes) on a cluster that uses Vector forwarding. The forwarding path is what broke. Disabling forwarding and then enabling it again makes queries return. No deeper defect has been identified. + +### Cause 6: Forwarding ingress logs to an OpenTelemetry endpoint + +Vector can forward ingress logs to an OTLP endpoint. The working setup reshapes each event into OTLP JSON (`resourceLogs`) with a `remap` transform. It sends the result through an `opentelemetry` sink over HTTP POST to the endpoint's `/v1/logs` path, with JSON encoding, `content-type: application/json`, and the TLS CA file. A wrong path on the endpoint side makes the delivery fail. + +## Resolution + +### Cause 1: The Vector Kafka sink fails SASL authentication because the sink enables TLS and the broker does not + +1. Find out which security protocol the broker exposes on the port the sink uses. Ask the Kafka owner, or read the broker's `listeners` and `listener.security.protocol.map`. Also note which SASL mechanism the listener enables (`PLAIN`, `SCRAM-SHA-256` or `SCRAM-SHA-512`). + +2. Read the sink: + + ```bash + kubectl -n cpaas-system get secret cpaas-vector -o jsonpath='{.data.vector\.yaml}' | base64 -d | grep -n -A20 'type: kafka' + ``` + + `tls.enabled: true` against a `SASL_PLAINTEXT` listener matches this cause. The sink's `sasl.mechanism` must also be the mechanism that the listener enables, in the same spelling. + +3. Correct the sink in the Vector configuration that holds it. Save the current configuration first. + + > **Warning:** Changing the Vector configuration restarts the forwarding pipeline, and a wrong sink can stop forwarding or the platform's own ClickHouse sink. Precondition: the listener protocol and mechanism are confirmed. Rollback: re-apply the saved configuration. + + ```yaml + sinks: + : + type: kafka + inputs: + - + bootstrap_servers: : + topic: + encoding: + codec: json + sasl: + enabled: true + mechanism: PLAIN + username: + password: + tls: + enabled: false + ``` + + Remove the `librdkafka_options` SSL entries that were only there for the TLS attempt. Validate the file with `vector validate` in the `vector` container before you apply it. + +4. Check that the topic receives records. + +### Cause 2: The Vector librdkafka build has no GSSAPI, so a Kerberos Kafka sink exits with status 78 + +1. Confirm the signature: + + ```bash + kubectl -n cpaas-system logs -c vector --tail=30 + ``` + + The `GSSAPI` line and exit status 78 match this cause. + +2. This Vector image cannot use Kerberos to authenticate to Kafka. Use one of the mechanisms the build supports (`PLAIN`, `SCRAM`, or `OAUTHBEARER`) on a listener of the destination Kafka, and configure the sink with it. + +### Cause 3: The external Kafka topic is created and stays empty because the Vector condition matches only one cluster + +1. Find the `condition` in the Kafka forwarding configuration: + + ```bash + kubectl get rpch -A | grep -Ei 'vector|razor|clickhouse' + kubectl -n cpaas-system get secret cpaas-vector -o jsonpath='{.data.vector\.yaml}' | base64 -d | grep -n -B2 -A4 'condition' + ``` + +2. Widen the condition. + + > **Warning:** Widening the condition sends more clusters' logs to the external Kafka, which adds traffic and storage on the external side. Nothing changes in the platform storage. Precondition: only when the topic exists, stays empty, and the condition names fewer clusters than you intend to forward. Rollback: restore the saved configuration and restart `razor`. + + Write the condition as one expression that names every cluster to forward: + + ```yaml + condition: '(.cluster == "" || .cluster == "")' + ``` + + Apply the configuration the same way it was created, then restart `razor`. + +3. Read a few records from the topic. + +### Cause 4: The platform log page is empty after ClickHouse-to-Kafka forwarding was configured, while the external Kafka receives logs + +1. Save the forwarding patch: + + ```bash + kubectl get rpch -A | grep -Ei 'vector|razor|clickhouse' + kubectl get rpch -o yaml > /tmp/forwarding-rpch.yaml + ``` + +2. Delete it. + + > **Warning:** Deleting the forwarding `rpch` stops forwarding to the external Kafka until you apply it again, and returns the Vector pipeline to the default. Precondition: only when the external Kafka receives logs, the platform page is empty, and `razor` and ClickHouse are healthy. Rollback: `kubectl apply -f /tmp/forwarding-rpch.yaml`. + + ```bash + kubectl delete rpch + ``` + +3. Confirm that the platform page shows current logs. + +4. Apply the forwarding configuration again. Keep the `clickhouse` sink next to the new `kafka` sink, and check the file for stray quotes, non-ASCII punctuation or other special characters. Confirm that both the page and the external topic receive logs. + +### Cause 5: Log search fails after new nodes until Vector forwarding is disabled and enabled again + +> **Warning:** Turning Vector forwarding off stops the copy to the external target until it is turned on again, so external consumers see a gap for that interval. Precondition: only when search failed after compute nodes were added and forwarding is in use. Rollback: enable forwarding again, which is also the second step of the fix. + +Disable Vector forwarding with the same switch that enabled it, then enable it again. Query a workload that is logging now. + +### Cause 6: Forwarding ingress logs to an OpenTelemetry endpoint + +Add a transform and a sink to the Vector configuration. `filter_workload_ingress` is the input name of the ingress filter in this example. Use the name that your configuration defines. + +```yaml +transforms: + remap_otlp_ingress: + type: remap + inputs: + - filter_workload_ingress + source: | + ts = to_unix_timestamp(timestamp(.timestamp) ?? now(), unit: "nanoseconds") + body = string(.message) ?? encode_json(.) + . = { + "resourceLogs": [{ + "resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "acp-ingress"}}]}, + "scopeLogs": [{ + "scope": {"name": "vector"}, + "logRecords": [{"timeUnixNano": ts, "body": {"stringValue": body}}] + }] + }] + } +sinks: + output_logs_ingress: + type: opentelemetry + inputs: + - remap_otlp_ingress + protocol: + type: http + uri: https:///v1/logs + method: post + encoding: + codec: json + framing: + method: newline_delimited + batch: + max_events: 1 + request: + headers: + content-type: application/json + tls: + ca_file: /var/ca-bundle.crt + buffer: + type: disk + max_size: +``` + +Set `max_size` to the disk buffer size in bytes (1 GiB is 1073741824). The `protocol` block takes the options of Vector's `http` sink. Check the key placement against the Vector version in the `vector` container (`vector --version`), and run `vector validate` on the file before you apply it. Make sure the URI ends in `/v1/logs`. + +## Diagnostic Steps + +1. Find the forwarding configuration and the Vector state: + + ```bash + kubectl get secret -A | grep -E 'cpaas-vector' + kubectl get rpch -A | grep -Ei 'vector|razor|clickhouse' + kubectl -n cpaas-system get pods | grep -Ei 'razor' + kubectl -n cpaas-system logs -c vector --tail=50 + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | SASL authentication fails with correct credentials, and the sink sets `tls.enabled: true` against a broker without TLS | Cause 1 | + | Exit status 78 and `No provider for SASL mechanism GSSAPI` | Cause 2 | + | An external topic that exists and stays empty | Cause 3 | + | An empty platform page right after Kafka forwarding was added, while the external topic fills | Cause 4 | + | Search that broke only after new nodes were added, on a cluster with forwarding | Cause 5 | + | OTLP deliveries fail | Cause 6, check the URI path and the CA file | + +3. If `ALAUDA_LOG_TOPIC` lag is growing and forwarding is not configured, the forwarding path is not involved. See [Log Kafka lag from Elasticsearch write hot spots, breaker trips, IK analyzer errors, or too few partitions on ACP](./Log_Kafka_lag_from_Elasticsearch_write_hot_spots_breaker_trips_IK_analyzer_errors_or_too_few_partitions_on_ACP.md). + +## Related Information + +- [Log query shows no logs, HTTP 500, slow long ranges, or an empty real-time log on ACP](./Log_query_shows_no_logs_HTTP_500_slow_long_ranges_or_an_empty_real-time_log_on_ACP.md) diff --git a/docs/en/solutions/cpaas-kafka_broker_fails_to_start_corrupt_segment_no_space_broker_id_conflict_or_deleted_StatefulSet_on_ACP.md b/docs/en/solutions/cpaas-kafka_broker_fails_to_start_corrupt_segment_no_space_broker_id_conflict_or_deleted_StatefulSet_on_ACP.md new file mode 100644 index 000000000..4ee8ce8cc --- /dev/null +++ b/docs/en/solutions/cpaas-kafka_broker_fails_to_start_corrupt_segment_no_space_broker_id_conflict_or_deleted_StatefulSet_on_ACP.md @@ -0,0 +1,246 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# cpaas-kafka broker fails to start: corrupt segment, no space, broker id conflict, or deleted StatefulSet on ACP + +## Issue + +A broker of the log storage Kafka does not start, or the Kafka is missing altogether. Typical reports: + +- "The ES log component keeps showing Installing (`正在安装`)." +- "Kafka for logging filled its PVC and still fails after the PVC was expanded." +- "Kafka cannot write to its data disk: `No space left on device`." +- "One Kafka pod crashes while Elasticsearch log storage is being deployed." +- "Kafka cannot start again after a reinstall: broker id conflict." +- "The `cpaas-kafka` StatefulSet was deleted." + +The broker is the log storage Kafka in `cpaas-system` (StatefulSet `cpaas-kafka`, pods `cpaas-kafka-0..2`, with `cpaas-zookeeper`), not a Data Services Kafka. Its data is on node-local storage, typically `/cpaas/data/kafka0` (seen inside the pod as `/kafka0`), plus `/cpaas/data/kafka` and `/cpaas/data/zookeeper`. + +When the brokers run but logs, audit and events stop, see [Logs, audit and events stop while the log Kafka is degraded on ACP](./Logs_audit_and_events_stop_while_the_log_Kafka_is_degraded_on_ACP.md). + +## Root Cause + +### Cause 1: No space left on device under /kafka0, and expanding the PVC does not help + +```text +ERROR Error while loading log dir /kafka0 (org.apache.kafka.storage.internals.log.LogDirFailureChannel) +java.nio.file.FileSystemException: /kafka0/ALAUDA_EVENT_TOPIC-2/00000000000000000007.snapshot: No space left on device +``` + +Kafka filled its volume and crashed. What fixes it depends on what backs the PV: + +- **Node-local PV** (`spec.local` or `spec.hostPath`). By default, `cpaas-kafka` uses PVs backed by the node's local storage, so the real capacity is the node's local filesystem. Expanding the PVC (for example from 30Gi to 60Gi) changes the number but not the space, and the pod still cannot create files in `/kafka0`. Patching `volumeClaimTemplates` is rejected, because that StatefulSet field is immutable. Extending the logical volume behind the path on the node fixes Kafka. +- **PV from a StorageClass that supports expansion.** Expanding the PVC adds real space, and Kafka recovers. + +After space is added, ZooKeeper may log `ERROR Unsuccessful handshake with session 0x0 (org.apache.zookeeper.server.NettyServerCnxnFactory)`. This message is known, and it can be ignored while ZooKeeper is `Running`. + +### Cause 2: Kafka exits at startup with a fault in an unsafe memory access operation in LogSegment.recover + +The previous log of the broker ends with: + +```text +ERROR Exiting Kafka due to fatal exception during startup. (kafka.Kafka$) +java.lang.InternalError: a fault occurred in an unsafe memory access operation + at org.apache.kafka.storage.internals.log.LogSegment.recover(LogSegment.java:498) + at kafka.log.LogLoader.recoverSegment(LogLoader.scala:371) +``` + +Kafka cannot read its on-disk segments while it recovers them at startup, so the broker never starts. The pod is `CrashLoopBackOff` with thousands of restarts, while the other brokers, Elasticsearch and `lanaya` are `Running`. `kubectl get ars -A` shows every chart `Synced`/`Ready` except `logcenter`, which is `Synced` with health `WaitReady`. That keeps the Elasticsearch log component in Installing. The fix is to stop Kafka and ZooKeeper, clear the contents of their data directories on all three log nodes, and scale both back to 3 replicas. + +### Cause 3: One Kafka pod crashes at startup with a broker id conflict + +One `cpaas-kafka` pod crash-loops while its peers run, and its broker id is the same as another broker's, or conflicts with data left from an earlier installation. Upstream Kafka reports these states with messages such as `InconsistentBrokerIdException: Configured broker.id ... doesn't match stored broker.id ... in meta.properties`, or with a broker that cannot register because its id is already registered in ZooKeeper. Two variants exist: + +- While Elasticsearch was being deployed as the log storage, one Kafka node failed to start because its broker id duplicated the id of another Kafka node. Giving each Kafka node a different broker id and restarting Kafka fixed it. +- After the log storage was reinstalled, Kafka could not start because of a broker id conflict. The data of the previous installation had not been cleaned before the reinstall. Clearing the `zookeeper`, `kafka` and `kafka0` directories under `/cpaas/data` on the three Kafka nodes fixes it. + +### Cause 4: The cpaas-kafka StatefulSet was deleted + +The log storage Kafka StatefulSet is gone. A resource patch (`rpch`) for Kafka may exist that only adds configuration, and it can be left alone. Kafka can be recreated from the StatefulSet YAML of a healthy environment with the same platform and plugin version. After that, stop `lanaya`, Kafka and ZooKeeper, clean their directories, and start them in the order ZooKeeper, Kafka, `lanaya`. Log queries then work again. + +## Resolution + +### Cause 1: No space left on device under /kafka0, and expanding the PVC does not help + +1. Find the PV behind the broker and what backs it: + + ```bash + kubectl get pod -A | grep -E 'zook|kafka' + kubectl -n cpaas-system get pvc | grep kafka + kubectl get pv -o jsonpath='{.spec.local.path}{.spec.hostPath.path}{"\n"}{.spec.nodeAffinity}{"\n"}{.spec.storageClassName}{"\n"}' + ``` + +2. If the PV has a `local` or `hostPath` path, find the filesystem behind that path on the node the PV is pinned to: + + ```bash + df -hT + lvs + vgs + ``` + +3. Add space. + - For a node-local PV, extend the logical volume. + + > **Warning:** Extending a logical volume and its filesystem changes the node's disk layout. An online extension is normally safe, but a wrong logical volume or volume group name can consume space that another workload needs. Precondition: only when the Kafka log shows `No space left on device` under `/kafka0`, `df` on the node shows the logical volume behind the PV path full, and the volume group has free extents (`vgs`). Rollback: shrinking is not supported online for XFS and is risky for ext4, so treat the extension as permanent. + + ```bash + lvextend -r -L + /dev// + df -hT + ``` + + `-r` grows the filesystem together with the logical volume. + - For a PV from an expandable StorageClass, expand the PVC instead: + + ```bash + kubectl -n cpaas-system patch pvc -p '{"spec":{"resources":{"requests":{"storage":""}}}}' + ``` + +4. Restart the broker and check a current log query: + + ```bash + kubectl -n cpaas-system delete pod + ``` + + A ZooKeeper `Unsuccessful handshake with session 0x0` line after recovery needs no action. + +### Cause 2: Kafka exits at startup with a fault in an unsafe memory access operation in LogSegment.recover + +1. Confirm the signature: + + ```bash + kubectl get po -A | grep -E 'kafka|lanaya|elastic' + kubectl get ars -A + kubectl -n cpaas-system logs -p cpaas-kafka-0 | grep -B2 -A6 'unsafe memory access' + ``` + +2. Stop Kafka and ZooKeeper: + + ```bash + kubectl -n cpaas-system scale sts cpaas-kafka cpaas-zookeeper --replicas=0 + ``` + +3. Empty their data directories. + + > **Warning:** Clearing the Kafka and ZooKeeper data directories deletes every unconsumed log, audit and event record, and the ZooKeeper state of the log cluster. Precondition: only when the previous Kafka log shows the `LogSegment.recover` fault above and both StatefulSets are at 0 replicas. Rollback: none. + + On each of the three log nodes, empty the directories but keep them. The `cd ... &&` form stops before `rm` if a directory is missing: + + ```bash + cd /cpaas/data/kafka && rm -rf ./* + cd /cpaas/data/kafka0 && rm -rf ./* + cd /cpaas/data/zookeeper && rm -rf ./* + ``` + +4. Start both again and watch: + + ```bash + kubectl -n cpaas-system scale sts cpaas-kafka cpaas-zookeeper --replicas=3 + kubectl -n cpaas-system get po -w | grep -E 'kafka|zookeeper' + ``` + +### Cause 3: One Kafka pod crashes at startup with a broker id conflict + +1. Find the Kafka nodes. In the console, open Marketplace, Cluster Plugins, select the cluster, and open the log storage plugin. Its details list the three Kafka nodes. + +2. Compare the broker ids. The stored id is the `broker.id` line of `meta.properties` in the data directory (`/kafka0` inside the pod, `/cpaas/data/kafka0` on the node): + + ```bash + kubectl -n cpaas-system get pods -o wide | grep -E 'cpaas-kafka|cpaas-zookeeper' + kubectl -n cpaas-system logs -p | grep -i -E 'broker.?id' | tail -20 + for p in cpaas-kafka-0 cpaas-kafka-1 cpaas-kafka-2; do + echo "== $p"; kubectl -n cpaas-system exec "$p" -- cat /kafka0/meta.properties 2>/dev/null + done + ``` + + A crashing pod cannot be opened with `exec`. Read its file on the node with `cat /cpaas/data/kafka0/meta.properties`. + +3. When two live brokers have the same id and the data on each node belongs to the current installation, make the ids distinct and restart Kafka. Compare with a healthy installation of the same version before you edit anything. + +4. When the log storage was reinstalled on nodes that still hold the previous installation's data, clear that data. First stop both StatefulSets: + + ```bash + kubectl -n cpaas-system scale sts cpaas-kafka cpaas-zookeeper --replicas=0 + kubectl -n cpaas-system get pods | grep -E 'cpaas-kafka|cpaas-zookeeper' + ``` + + > **Warning:** Emptying `/cpaas/data/zookeeper`, `/cpaas/data/kafka` and `/cpaas/data/kafka0` deletes every record still in Kafka and the ZooKeeper state of the log cluster. Unconsumed logs, audit records and events are lost. Elasticsearch indices are not touched. Precondition: only when the broker id conflict appeared after a reinstall, the data under these directories is left from the previous installation, and both StatefulSets are at 0 replicas. Rollback: none. + + On each of the three Kafka nodes, delete the contents and keep the directories: + + ```bash + cd /cpaas/data/zookeeper && rm -rf ./* + cd /cpaas/data/kafka && rm -rf ./* + cd /cpaas/data/kafka0 && rm -rf ./* + ``` + + Then start both again and watch until every pod is `Running`: + + ```bash + kubectl -n cpaas-system scale sts cpaas-kafka cpaas-zookeeper --replicas=3 + kubectl -n cpaas-system get pods -w | grep -E 'cpaas-kafka|cpaas-zookeeper' + ``` + +### Cause 4: The cpaas-kafka StatefulSet was deleted + +1. Confirm that the StatefulSet is missing, and list the resource patches: + + ```bash + kubectl -n cpaas-system get sts | grep -E 'kafka|zookeeper' + kubectl get rpch -A + ``` + +2. On a healthy environment with the same platform and plugin version, export the StatefulSet: + + ```bash + kubectl -n cpaas-system get sts cpaas-kafka -o yaml > /tmp/cpaas-kafka.sts.yaml + ``` + + Remove `metadata.uid`, `metadata.resourceVersion`, `metadata.creationTimestamp`, `metadata.managedFields` and `status`. Replace node names, addresses and storage values that belong to the other environment. + +3. Apply it on the affected cluster: + + ```bash + kubectl -n cpaas-system apply -f /tmp/cpaas-kafka.sts.yaml + ``` + +4. Clean and restart the pipeline in order. + + > **Warning:** Cleaning the Kafka and ZooKeeper directories drops every record that was in Kafka. Precondition: Kafka was recreated from the exported YAML, and `lanaya`, Kafka and ZooKeeper are stopped. Rollback: none. + + Stop `lanaya`, Kafka and ZooKeeper, and empty the Kafka and ZooKeeper data directories as in Cause 2. Start ZooKeeper, then Kafka, then `lanaya`, and wait for each to be Ready before the next. Keep watching the Kafka pods afterwards, and run a log query. + +## Diagnostic Steps + +1. Read the broker state and the last fatal line: + + ```bash + kubectl -n cpaas-system get sts | grep -E 'kafka|zookeeper' + kubectl -n cpaas-system get pods -o wide | grep -E 'kafka|zookeeper' + kubectl -n cpaas-system logs -p cpaas-kafka-0 | tail -30 + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | `No space left on device` under `/kafka0` | Cause 1, check what backs the PV and not only the PVC size | + | `unsafe memory access` in `LogSegment.recover` | Cause 2 | + | One broker crash-looping with "broker id" in its previous log, two brokers with the same `broker.id` in `meta.properties`, or a conflict right after a reinstall | Cause 3 | + | No `cpaas-kafka` StatefulSet | Cause 4 | + +3. A Kafka in KRaft mode (broker and controller pods, data under `/var/lib/kafka/data`, `No space left on device` on `__cluster_metadata`) is not this log storage Kafka. Check its disk the same way: find the PV behind the data directory, and add space to the volume that actually backs it. + +## Related Information + +- Related articles: [Logs, audit and events stop while the log Kafka is degraded on ACP](./Logs_audit_and_events_stop_while_the_log_Kafka_is_degraded_on_ACP.md) covers a large `kafka0` backlog, one broker down with `min.insync.replicas=2`, and logs that stay missing after a disk cleanup. +- [Kafka Scale Switching](./Kafka_Scale_Switching.md) diff --git a/docs/en/solutions/nevermore_OOMKilled_cache_written_to_var_cpaas_or_callback-secret_mount_failure_on_ACP.md b/docs/en/solutions/nevermore_OOMKilled_cache_written_to_var_cpaas_or_callback-secret_mount_failure_on_ACP.md new file mode 100644 index 000000000..19af11b4e --- /dev/null +++ b/docs/en/solutions/nevermore_OOMKilled_cache_written_to_var_cpaas_or_callback-secret_mount_failure_on_ACP.md @@ -0,0 +1,207 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# nevermore OOMKilled, cache written to /var/cpaas, or callback-secret mount failure on ACP + +## Issue + +Application pods are logging and `kubectl logs` shows the lines, but the log console is empty or hours behind. On the nodes, one of the following is seen: + +- The `nevermore` DaemonSet in `cpaas-system` is restarting, and its main container or its `audit` container shows `OOMKilled`. On the node, `dmesg -T | grep -i -E 'out of memory|killed process'` shows `Memory cgroup out of memory` every few minutes, while the cluster monitoring shows node CPU and memory at only 20% to 30%. +- `/var` is at 100% because files appeared under `/var/cpaas`. +- On a fresh install, the `nevermore` pods sit in `Init:0/1` or `ContainerCreating` with `secret "log-collector-bearer-auth" not found`. + +`nevermore` is the per-node log agent. It reads container logs, Kubernetes audit records and events, and produces them into the platform Kafka, from which they are written into Elasticsearch or ClickHouse. Its local queue is under `/cpaas` (`/cpaas/log`, `/cpaas/audit`, `/cpaas/event`), and its own log is `/cpaas/log/logs/nevermore`. + +A `Memory cgroup out of memory` kill means that a container reached its **own** memory limit. That happens regardless of how much memory the node has free, which is why node-level monitoring can look low at the same time. + +When `nevermore` is `Pending`, fails on `filebeat.lock`, or is `OOMKilled` with a large disk queue after the log store was down, see [nevermore Pending, Error with filebeat.lock, or OOMKilled after a disk queue backlog on ACP](./nevermore_Pending_Error_with_filebeat_lock_or_OOMKilled_after_a_disk_queue_backlog_on_ACP.md). + +## Root Cause + +### Cause 1: The nevermore main container or its audit container is OOMKilled by its memory limit + +`kubectl describe` on a `nevermore-*` pod shows `Reason: OOMKilled` and exit code 137, either on the main container or on the container named `audit`. The container names differ between versions (for example `nevermore-audit`, `nevermore-event` and `nevermore-log` on older releases), so read them from the DaemonSet. While the container is down, the agent stops shipping, and the records wait in its on-disk queue under `/cpaas`. + +Consumer lag is the gap between what was produced and what was consumed, so a stopped producer does not raise it. The lag on `ALAUDA_LOG_TOPIC` climbs when the restarted agent flushes its queue faster than the consumer reads it. It falls once the container stays up and the consumer catches up. + +The node's log volume exceeds what the current limit allows. The main container stops restarting after its resources are raised (to about 2 CPU and 2Gi). An `audit` container OOM is cleared the same way, by raising that container's resources. + +One workload can be the whole cause. On one cluster, the `log` container of `nevermore` ran at its 2000m CPU limit, and its memory spiked toward the 4096Mi limit. The other containers stayed low, and the cache under `/cpaas/log/data` was small. The real-time application log showed about 600 thousand entries per minute, nearly all from one test workload running an `echo` loop with no resource limits. Stopping that application ended the OOM, and the next day the `log` container ran at about 19m CPU and 552Mi. + +A full node disk can also push the agent over its limit. Free the disk before you only add memory, or the same pod is OOMKilled again. + +### Cause 2: spec.valuesOverride on the logagent ModuleInfo points the cache at /var/cpaas + +`df` shows `/var` at 100%, and `du -sh /var/cpaas` shows several gigabytes, while `/cpaas` itself may still have room. The installed log agent module object (kind ModuleInfo, short name `minfo`, on the global cluster) has a `spec.valuesOverride` that changed the `nevermore` local cache directory. The product default is `/cpaas/`. After the override, `event`, `log` and `audit` are written onto the smaller `/var` partition. + +Deleting that `spec.valuesOverride` puts the cache back on `/cpaas/`. Files that are already under `/var/cpaas` are not moved by the revert. + +### Cause 3: FailedMount for volume callback-secret, secret log-collector-bearer-auth not found + +The pod event is: + +```text +MountVolume.SetUp failed for volume "callback-secret": secret "log-collector-bearer-auth" not found +``` + +The agent cannot start because the Secret that the callback volume mounts was never created. For example, on a 4.3 business cluster every `nevermore` pod sits at `0/4` `Init:0/1`. `kubectl get secret -A | grep log` lists `log-callback-secret` and `log-center-secret` in `cpaas-system`, but no `log-collector-bearer-auth`. + +Reinstalling the Log Collector does not create the Secret. The Alauda Container Platform Log Essentials plugin (`log-essentials`) creates it. Install Log Essentials on the cluster before the Log Collector, then restart Deployment `cluster-transformer` in `cpaas-system` so that it regenerates the Secret. + +## Resolution + +### Cause 1: The nevermore main container or its audit container is OOMKilled by its memory limit + +1. List the containers and the last termination: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep nevermore + kubectl -n cpaas-system get ds nevermore -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" "}{.resources.limits.memory}{"\n"}{end}' + kubectl -n cpaas-system describe pod | grep -E 'OOMKilled|Exit Code|Limits|Requests' + ``` + +2. Before raising limits, check whether one workload floods its output or the disk is full: + + ```bash + kubectl -n cpaas-system top pod --containers | grep nevermore + du -sh /cpaas/log/data/* 2>/dev/null + df -hT /cpaas + ``` + + Run `du` and `df` on the node. If only the `log` container is hot and the cache is small, find the workload with the largest real-time log volume in the console, and stop or fix it. Free a full disk first. + +3. Raise the resources of the container that was OOMKilled. + + > **Warning:** Raising CPU and memory on DaemonSet `nevermore` rolls the agent on every node, and collection pauses on a node while its pod restarts. Precondition: the node has the extra memory free, and you noted the previous requests and limits. A module controller can revert a direct DaemonSet edit. If the pod spec snaps back, set the same numbers in the log agent ModuleInfo `spec.valuesOverride`, under the resources key that object already uses. Rollback: `kubectl set resources` with the previous values, or re-apply the saved ModuleInfo. + + ```bash + kubectl -n cpaas-system set resources ds/nevermore \ + -c --requests=cpu=2,memory=2Gi --limits=cpu=2,memory=2Gi + ``` + + Repeat the command with `-c` for the `audit` container if that is the one with `OOMKilled`. Size the `audit` container above the usage that killed it, instead of copying 2Gi onto every container. + +4. Watch that `RESTARTS` stops increasing, then confirm that the Kafka lag is falling: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep nevermore + ``` + +### Cause 2: spec.valuesOverride on the logagent ModuleInfo points the cache at /var/cpaas + +1. On the global cluster, save the ModuleInfo: + + ```bash + kubectl get minfo | grep logagent | grep + kubectl get minfo -o yaml > /tmp/logagent-minfo.before.yaml + kubectl get minfo -o jsonpath='{.spec.valuesOverride}' + ``` + +2. Remove the cache-path override. + + > **Warning:** Clearing the whole `spec.valuesOverride` drops every override on that module, including resource overrides from Cause 1, and `nevermore` rolls back to the default cache directory `/cpaas/`. Precondition: `/cpaas` has free space, you saved the YAML, and you read the override so that other keys can be put back. Rollback: `kubectl apply -f /tmp/logagent-minfo.before.yaml`. If other keys must stay, delete only the key that sets the cache path. + + ```bash + kubectl edit minfo + ``` + + Wait until DaemonSet `nevermore` is Ready, then check on the node that the default directories exist: + + ```bash + ls -ld /cpaas/event /cpaas/log /cpaas/audit + ``` + +3. If `/cpaas/event`, `/cpaas/log` or `/cpaas/audit` does not exist yet and the only copy is under `/var/cpaas`, move that directory once. Skip the move when `/cpaas/` already exists, because that tree is the live queue. + + > **Warning:** The directories under `/var/cpaas` hold unshipped cache, and a later delete of the source is irreversible. Precondition: `nevermore` has been reverted to `/cpaas/`, and `du` showed these directories under `/var/cpaas`, not `/var/log` or `/var/lib`. Rollback: move the directory back, only if `nevermore` has not started a new queue at the destination. + + ```bash + mkdir -p /cpaas + for d in event log audit; do + if [ -d "/var/cpaas/$d" ] && [ ! -e "/cpaas/$d" ]; then + mv "/var/cpaas/$d" "/cpaas/$d" + fi + done + ``` + +4. Free what remains under `/var/cpaas`. + + > **Warning:** `rm -rf` under `/var/cpaas` deletes log cache that has not been shipped, so search has a gap for those records. Precondition: the path that `du` printed is under `/var/cpaas`, and `nevermore` is already writing the restored `/cpaas/` tree. When `/var` is at 100% and pods cannot start, deleting the `logs` directory first gives temporary relief before the revert takes effect. Rollback: none. + + ```bash + du -sh /var/cpaas /var/cpaas/* + rm -rf /var/cpaas//logs + ``` + +### Cause 3: FailedMount for volume callback-secret, secret log-collector-bearer-auth not found + +1. Confirm the signature: + + ```bash + kubectl get secret -A | grep log + kubectl -n cpaas-system get pods | grep nevermore + kubectl -n cpaas-system describe pod | grep -A2 FailedMount + ``` + +2. Install the `log-essentials` plugin from the cluster plugin page, where the log collector was installed. Reinstalling only the Log Collector does not create the Secret. + +3. Restart the transformer so that it regenerates the Secret. + + > **Warning:** Restarting `cluster-transformer` pauses plugin status updates during the rollout. It does not delete log data. Precondition: `log-essentials` is installed or installing. Rollback: none needed; a restart has no spec change to undo. + + ```bash + kubectl -n cpaas-system rollout restart deployment/cluster-transformer + kubectl -n cpaas-system rollout status deployment/cluster-transformer + kubectl -n cpaas-system get secret log-collector-bearer-auth + kubectl -n cpaas-system get pods | grep nevermore + ``` + + The Secret must exist, and the `nevermore` pods must leave `Init` and `ContainerCreating`. + +4. If Log Essentials was already installed and `log-collector-bearer-auth` is still missing after the transformer rollout, this procedure does not apply. That situation needs a different investigation, for example on a standby global cluster. + +## Diagnostic Steps + +1. Compare the source with the agent before you look at Kafka: + + ```bash + kubectl logs -n -c --tail=20 + kubectl -n cpaas-system get pods -o wide | grep nevermore + ``` + + If the line is absent from the source, this article does not apply. + +2. On the node that reports OOM kills, read the kernel log and the container that was killed: + + ```bash + dmesg -T | grep -i -E 'out of memory|killed process' | tail -5 + kubectl -n cpaas-system describe pod | grep -E 'OOMKilled|Exit Code|Limits|Requests' + ``` + + `Memory cgroup out of memory` naming a `nevermore` process, together with `OOMKilled` on a `nevermore` container, is Cause 1. + +3. For the cache path, run `df -hT /var /cpaas` and `du -sh /var/cpaas` on the node, then read the `logagent` ModuleInfo `spec.valuesOverride` on the global cluster. A cache path under `/var/cpaas` is Cause 2. + +4. The `callback-secret` event with pods at `Init:0/1`, on a cluster without Log Essentials, is Cause 3. + +5. When the pod is up, read the agent log for enqueue or connection errors: + + ```bash + kubectl -n cpaas-system logs -c --tail=100 + ``` + +## Related Information + +- [nevermore Pending, Error with filebeat.lock, or OOMKilled after a disk queue backlog on ACP](./nevermore_Pending_Error_with_filebeat_lock_or_OOMKilled_after_a_disk_queue_backlog_on_ACP.md) +- [nevermore and ClickHouse log components alert on CPU or memory, restart, or ignore new resource values on ACP](./nevermore_and_ClickHouse_log_components_alert_on_CPU_or_memory_restart_or_ignore_new_resource_values_on_ACP.md) diff --git a/docs/en/solutions/nevermore_Pending_Error_with_filebeat_lock_or_OOMKilled_after_a_disk_queue_backlog_on_ACP.md b/docs/en/solutions/nevermore_Pending_Error_with_filebeat_lock_or_OOMKilled_after_a_disk_queue_backlog_on_ACP.md new file mode 100644 index 000000000..4a3c8a434 --- /dev/null +++ b/docs/en/solutions/nevermore_Pending_Error_with_filebeat_lock_or_OOMKilled_after_a_disk_queue_backlog_on_ACP.md @@ -0,0 +1,160 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.14.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# nevermore Pending, Error with filebeat.lock, or OOMKilled after a disk queue backlog on ACP + +## Issue + +The `nevermore` log agent does not run on some nodes. Typical reports: + +- "The log collection component is abnormal", and `nevermore` pods are `Pending` for days. +- "`nevermore` is stuck": one pod is `3/4 Error` while the others are `4/4 Running`. +- A `nevermore` pod cannot start and its status is `OOMKilled` after the log store was down. + +`nevermore` is the per-node DaemonSet in `cpaas-system` that collects container logs, Kubernetes audit records and events. Its local queue is under `/cpaas` (`/cpaas/log`, `/cpaas/audit`, `/cpaas/event`), with segment files `*.seg` under, for example, `/cpaas/log/data/diskqueue`. Its own log is `/cpaas/log/logs/nevermore`. The container names differ between releases (for example `audit` and `log`, or `nevermore-audit`, `nevermore-event` and `nevermore-log`). Read them from the DaemonSet before you pass `-c`: + +```bash +kubectl -n cpaas-system get ds nevermore -o jsonpath='{.spec.template.spec.containers[*].name}' +``` + +When `nevermore` is `OOMKilled` without a queue backlog, writes its cache under `/var/cpaas`, or waits for the `log-collector-bearer-auth` Secret, see [nevermore OOMKilled, cache written to /var/cpaas, or callback-secret mount failure on ACP](./nevermore_OOMKilled_cache_written_to_var_cpaas_or_callback-secret_mount_failure_on_ACP.md). + +## Root Cause + +### Cause 1: nevermore cannot start because the node has too little memory + +`kubectl get ars -A` shows `logagent` as `Synced`, with health `WaitReady` and message `chart synced`. One or more `nevermore` pods are `0/4` `Pending`, and their events read: + +```text +Warning FailedScheduling default-scheduler 0/59 nodes are available: 1 Insufficient memory. preemption: 0/59 nodes are available: 59 No preemption victims found for incoming pod.. +``` + +A DaemonSet pod is bound to one node, so `1 Insufficient memory` refers to that node. `kubectl logs` on the pending pod prints only `Defaulted container ...`, because a pod that never started has no container log. The node is a small machine (for example 2 CPU / 4 GB) that cannot fit the `nevermore` requests next to its other pods. Giving the machine more memory lets the pod schedule. + +### Cause 2: nevermore is OOMKilled because its disk queue filled while Elasticsearch was down + +The `nevermore` pod cannot start, and its status is `OOMKilled`. Elasticsearch had been abnormal, so `nevermore` could not ship, and the collected data piled up in the local disk queue. `/cpaas/log/data/diskqueue` is large. Repairing Elasticsearch and then deleting the `*.seg` files under `/cpaas/log/data/diskqueue` returns `nevermore` to normal. Raising the limits alone does not empty this queue. + +The same cache family also has `/cpaas/audit/data/diskqueue` and `/cpaas/event/data/diskqueue`. Clean those only when `du` shows that they are large, and only the files inside `diskqueue`. Do not delete `/cpaas/log`, `/cpaas/audit` or `/cpaas/event` themselves. Cleaning `diskqueue` while Elasticsearch is still down frees disk space only until the queue refills. + +### Cause 3: An old nevermore log container stays on the node and blocks the new one + +One `nevermore` pod is `3/4 Error` with a few restarts, while the others are `4/4 Running`. `kubectl logs -p ` defaults to the audit container and answers `previous terminated container "audit" in pod "" not found`. The failing container is the log container, and its previous log is: + +```text +Exiting: /cpaas/log/data/filebeat.lock: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data) +``` + +On its node, `crictl ps -a | grep never` shows the new pod's log container `Exited` with several restarts. It also shows the log container of an old `nevermore` pod, created months earlier, still `Running`, while the old pod's other containers are `Exited`. The old container holds `/cpaas/log/data/filebeat.lock` and prevents the new one from starting. Deleting the old container fails, and killing its process by PID does not remove it either. Rebooting the node clears it, and `nevermore` recovers. Why the container could not be removed is not known. + +## Resolution + +### Cause 1: nevermore cannot start because the node has too little memory + +1. Confirm the signature: + + ```bash + kubectl get ars -A | grep logagent + kubectl get pod -A | grep nevermore | grep -v Run + kubectl -n cpaas-system describe pod | sed -n '/Events:/,$p' + kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.capacity.cpu,MEM:.status.capacity.memory + kubectl describe node | sed -n '/Allocated resources/,/Events/p' + ``` + + `0/N nodes are available: 1 Insufficient memory` on a `Pending` `nevermore` pod, on a node too small for its requests, matches this cause. + +2. Add memory (and CPU) to the node. For a VM, this usually needs the VM to be stopped. + + > **Warning:** Resizing the node VM restarts it, which stops every pod on that node. Precondition: the node is cordoned and drained (`kubectl cordon ` and `kubectl drain --ignore-daemonsets --delete-emptydir-data`). Rollback: resize the VM back; run `kubectl uncordon ` when it is up. + +3. After the node is back, run `kubectl uncordon `. The DaemonSet pod then schedules. + +### Cause 2: nevermore is OOMKilled because its disk queue filled while Elasticsearch was down + +1. Repair Elasticsearch first, or the queue refills. See [Elasticsearch log indices red or yellow with unassigned shards on ACP](./Elasticsearch_log_indices_red_or_yellow_with_unassigned_shards_on_ACP.md). + +2. On the node, check the queue size: + + ```bash + du -sh /cpaas/log/data/diskqueue /cpaas/audit/data/diskqueue /cpaas/event/data/diskqueue + ls /cpaas/log/data/diskqueue | head + ``` + +3. Delete the queued segment files and restart the pod. + + > **Warning:** Deleting `*.seg` files removes queued log records that were never shipped, so search has a permanent gap for that window. Precondition: only when `nevermore` is `OOMKilled`, the disk queue is large, and Elasticsearch has already been repaired. Do not delete the parent `/cpaas/log` tree. Rollback: none. + + ```bash + find /cpaas/log/data/diskqueue -name '*.seg' -delete + kubectl -n cpaas-system delete pod + ``` + + If `du` also showed a large audit or event `diskqueue`, the same gap applies when you delete the segment files there. + +### Cause 3: An old nevermore log container stays on the node and blocks the new one + +1. Confirm the signature: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep never + kubectl -n cpaas-system logs -c -p | grep 'filebeat.lock' + ``` + + On the node: + + ```bash + crictl ps -a | grep -i nevermore + crictl inspect | grep -E '"pid"|"state"' + ``` + + `data path already locked by another beat` in the new log container, together with a `Running` log container whose pod ID belongs to an old `nevermore` pod, matches this cause. + +2. Try to remove the old container with `crictl rm -f `. If that fails and killing its PID does not remove it, reboot the node. + + > **Warning:** Rebooting the node stops every pod on it. Precondition: only when an old `nevermore` log container remains, `crictl rm -f` and killing its PID both failed, and the node is cordoned and drained. Rollback: none needed; uncordon the node after the reboot. + + ```bash + kubectl cordon + kubectl drain --ignore-daemonsets --delete-emptydir-data + ``` + + On the node: + + ```bash + systemctl reboot + ``` + +3. After the node is back, run `kubectl uncordon `, and check that `nevermore` is `Running`. + +## Diagnostic Steps + +1. Read the pod state and the node: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep nevermore + kubectl -n cpaas-system describe pod | grep -E 'State|Reason|Exit Code|Limits' -A2 + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | `Pending` with `Insufficient memory` on a small node | Cause 1 | + | `OOMKilled` with a large `/cpaas/log/data/diskqueue` after Elasticsearch trouble | Cause 2, repair Elasticsearch before deleting `*.seg` | + | `filebeat.lock: data path already locked by another beat`, with an old container that cannot be removed | Cause 3 | + | `OOMKilled` with no queue backlog | See the related article on limit-driven OOM | + +## Related Information + +- [nevermore OOMKilled, cache written to /var/cpaas, or callback-secret mount failure on ACP](./nevermore_OOMKilled_cache_written_to_var_cpaas_or_callback-secret_mount_failure_on_ACP.md) +- [nevermore and ClickHouse log components alert on CPU or memory, restart, or ignore new resource values on ACP](./nevermore_and_ClickHouse_log_components_alert_on_CPU_or_memory_restart_or_ignore_new_resource_values_on_ACP.md) diff --git a/docs/en/solutions/nevermore_and_ClickHouse_log_components_alert_on_CPU_or_memory_restart_or_ignore_new_resource_values_on_ACP.md b/docs/en/solutions/nevermore_and_ClickHouse_log_components_alert_on_CPU_or_memory_restart_or_ignore_new_resource_values_on_ACP.md new file mode 100644 index 000000000..e829a6466 --- /dev/null +++ b/docs/en/solutions/nevermore_and_ClickHouse_log_components_alert_on_CPU_or_memory_restart_or_ignore_new_resource_values_on_ACP.md @@ -0,0 +1,249 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.3.x +--- + +# nevermore and ClickHouse log components alert on CPU or memory, restart, or ignore new resource values on ACP + +## Issue + +One of the following is reported for the log components: + +- One `nevermore` pod restarted abnormally. Only the log container on that node has high memory, and `nevermore` on every other node is fine. +- On a cluster with dozens of nodes, `nevermore` CPU on one node keeps alerting while the other nodes are normal. The node's CPU is high only at intervals. +- ClickHouse log storage keeps raising high CPU and memory alarms, and adding resources does not stop them. +- The ClickHouse log shows `Memory limit (total) exceeded` with `MEMORY_LIMIT_EXCEEDED`. Log queries may fail at the same time. +- On the global cluster, the ClickHouse StatefulSet runs with values different from the plugin configuration, and restarting `clickhouse-operator` changes nothing. +- The ClickHouse CPU alarm fires on a regular schedule (for example about once an hour, for about 10 minutes), or node load on the management cluster rises at the same time. +- After an upgrade, a new ClickHouse limit set in the console does not take effect. +- "What are the `nevermore` and `zeus` services for? They take 10 cores on a 64-core server. Can that be reduced?" + +`nevermore` is the node log collector and runs one pod per node. Its local buffer on the node is `/cpaas/log/data/diskqueue`. ClickHouse is the log store of Alauda Container Platform Log Storage for ClickHouse, managed by `clickhouse-operator` through a ClickHouseInstallation (`chi`). The ClickHouse log often also contains `RaftInstance ... failed to read rpc header from socket ... End of file` every few seconds. That line is benign and unrelated to CPU or memory. + +## Root Cause + +### Cause 1: The nevermore log container is OOMKilled on one node after a short memory surge + +Pod OOM events rise on one node (for example 6 in 30 minutes) and stay at 0 on the others. The `nevermore` log container on that node has a 4G limit. Its usual memory use is a few percent to just over 10%, but a short surge passes 4G and the container is killed. After the log container memory was raised to 5G, it recovered and usage returned below 10%. + +### Cause 2: A nevermore CPU alert on one node comes from bursts of business logs + +Business logs on that node are written in concentrated bursts. Log file rotation, compression and batch sending then push `nevermore` CPU up for a while, and the alert fires. If the node's local log directory or disk is slow, `nevermore` builds a backlog and processes it all at once, which makes the spike sharper. `/cpaas/log/data/diskqueue` grows during the alert. With a 2-core CPU limit on the log container, the recommendation is 4 cores. The queued data is compressed, so the business log that caused the burst cannot always be identified. Such an alert can clear without further change. + +### Cause 3: ClickHouse CPU and memory alarms persist after expansion (known issue) + +ClickHouse log storage keeps raising CPU and memory alarms, and more resources do not help. This is a known issue in the log component. It is fixed in `log-4.3.2` (ACP 4.3.2) and not in 4.3.1. + +### Cause 4: MEMORY_LIMIT_EXCEEDED in MergeTreeBackgroundExecutor or in log queries + +The ClickHouse log shows: + +```text +Memory limit (total) exceeded: would use 3.64 GiB (attempt to allocate chunk of 4361312 bytes), maximum: 3.60 GiB +``` + +with `MEMORY_LIMIT_EXCEEDED` on `MergeTreeBackgroundExecutor`. The `maximum` is close to the 4Gi container limit: background merges, and queries, need more memory than the ClickHouse limit allows. The console can then fail to show logs, and the `vector` sinks retry their inserts. Raising ClickHouse from 2 CPU / 4Gi to 4 CPU / 8Gi removes the error. An `Address already in use` line for `0.0.0.0:9004` in the same log was not related. + +### Cause 5: The ClickHouse StatefulSet does not follow the configured resources + +The plugin and the `chi` both say 2 CPU / 4G, and no resource patch (`rpch`) exists. However, the StatefulSet runs two replicas at 3 CPU / 6G and one at 4 CPU / 6G. Restarting `clickhouse-operator` does not update the StatefulSet or its pods. Saving the resources on the plugin again triggers the update, and the StatefulSet returns to the configured values. + +When the global cluster nodes are too small for ClickHouse (8 CPU per node, with ClickHouse alone driving node CPU high), the lasting change is to move ClickHouse off the global cluster. Store the global cluster's audit data in a business cluster's log storage instead. + +### Cause 6: Periodic CPU spikes from MergeTree background merges + +The CPU alarm repeats at regular intervals, or CPU spikes about every hour and raises node load on the management cluster. ClickHouse does not restart, and its log during a spike looks the same as at other times. The pod graph shows CPU at 90% to 100% at regular intervals, often as two adjacent peaks at the limit, with 10% to 20% in between. On the node, `top` puts `clickhouse-serv` far above every other process during a spike. + +The MergeTree background merge periodically combines small data parts of a partition into larger ones and applies TTL deletes. With high merge-thread concurrency, it takes a lot of CPU for a short time. Raising the limit does not remove the spike: after a 4-core limit was raised to 6 cores, the merges used all 6. This is a known issue, and an optimization has been requested. It does not affect normal ClickHouse operation. A slow `/cpaas` disk makes merges last longer. + +### Cause 7: A resource change in the console does not apply because a resource patch already exists + +An earlier backend change to the ClickHouse resources created a resource patch (`rpch`). While that `rpch` exists, resource changes made in the console do not reach the ClickHouse StatefulSet. The patch targets the ClickHouseInstallation, not the StatefulSet, and appears like this: + +```text +rp- Active cpaas-system/logclickhouse clickhouse.altinity.com/v1 ClickHouseInstallation cpaas-system cpaas-clickhouse +``` + +This is the opposite situation to Cause 5, where no `rpch` exists. + +### Cause 8: nevermore and zeus appear to take 10 CPU cores + +`nevermore` is the log collection component. `zeus` reads the certificate information of the Kubernetes components on each node, generates Secrets from it, and creates the matching Certificate resources so that those certificates can be monitored. + +The "10 cores" is the sum of the containers' CPU **limits**, not their requests and not their actual usage. Pod scheduling counts only requests against the node's allocatable CPU, and the sum of limits on a node may exceed the node's physical CPU. The figure therefore takes nothing away from scheduling capacity. + +## Resolution + +### Cause 1: The nevermore log container is OOMKilled on one node after a short memory surge + +1. Read the containers and their limits: + + ```bash + kubectl -n cpaas-system get ds nevermore -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" "}{.resources.limits}{"\n"}{end}' + ``` + +2. Raise the log container memory limit. + + > **Warning:** Changing the DaemonSet restarts `nevermore` on every node, one at a time, and log collection on a node pauses while its pod restarts. Precondition: the OOM events are on the `nevermore` log container, and each node has the extra memory free. Rollback: set the previous limit with the same command. + + ```bash + kubectl -n cpaas-system set resources ds/nevermore -c --limits=memory=5Gi + ``` + +3. If the log collection plugin renders `nevermore` with its own resource values, set the same limit there. After the next plugin update, check that the DaemonSet still has the new limit. + +### Cause 2: A nevermore CPU alert on one node comes from bursts of business logs + +1. Confirm that the CPU on that node's `nevermore` is intermittent and lines up with business log bursts or `diskqueue` growth (Diagnostic Steps 1 and 2). + +2. Raise the log container CPU limit so the peaks are absorbed. A CPU limit does not reserve resources up front. + + > **Warning:** This restarts `nevermore` on every node, as in Cause 1. Precondition: the nodes have CPU headroom at peak. Rollback: set the CPU limit back to 2. + + ```bash + kubectl -n cpaas-system set resources ds/nevermore -c --limits=cpu=4 + ``` + +3. Compare the node's business log volume with other nodes, and check the write speed of the local log disk. A slow disk keeps the backlog pattern even with a higher limit. + +### Cause 3: ClickHouse CPU and memory alarms persist after expansion (known issue) + +Upgrade the log components to `log-4.3.2` (ACP 4.3.2), using the package provided by Alauda support. More CPU and memory do not fix this cause. A ClickHouse that is still at 2 CPU / 4Gi and logs `MEMORY_LIMIT_EXCEEDED` needs Cause 4 first. + +### Cause 4: MEMORY_LIMIT_EXCEEDED in MergeTreeBackgroundExecutor or in log queries + +1. Read the live requests and limits of every replica: + + ```bash + kubectl -n cpaas-system get pods | grep chi-cpaas-clickhouse + kubectl -n cpaas-system get pod -o jsonpath='{range .spec.containers[*]}{.name}{" req="}{.resources.requests.cpu}{"/"}{.resources.requests.memory}{" lim="}{.resources.limits.cpu}{"/"}{.resources.limits.memory}{"\n"}{end}' + ``` + + Continue only when the `maximum` in the error is close to the current memory limit (for example 3.60 GiB at 4Gi). + +2. Raise the resources. + + > **Warning:** Saving new resources on the plugin rolls the ClickHouse pods, and log writes and queries are degraded until every replica is Ready again. Precondition: the nodes that run ClickHouse have 4 CPU and 8Gi free for each replica. Rollback: set the plugin back to the previous values and save. + + In the console, open the Alauda Container Platform Log Storage for ClickHouse plugin on the affected cluster. Set CPU to 4 and memory to 8Gi, and save. + +3. When the pods are Ready, read the limits again. If they still show the old values, check for a resource patch or a StatefulSet that does not follow the plugin (Causes 5 and 7). + +### Cause 5: The ClickHouse StatefulSet does not follow the configured resources + +1. Confirm the drift (Diagnostic Step 4): the StatefulSet differs from both the plugin and the `chi`, and no `rpch` exists for it. + +2. Re-apply the resources from the plugin. + + > **Warning:** Re-applying the resources rolls the ClickHouse pods, and log writes and queries are degraded until all replicas are Ready. Precondition: the drift is confirmed. Rollback: none needed; this applies the values the plugin already holds. + + Open Alauda Container Platform Log Storage for ClickHouse in the console, change the CPU and memory values and save, so that an update is triggered. Check that the StatefulSet now matches. Restarting `clickhouse-operator` alone does not do this. + +3. If the global nodes are too small to host ClickHouse, move it off the global cluster. + + > **Warning:** Uninstalling ClickHouse log storage from the global cluster deletes the logs stored there, which is irreversible. Precondition: a business cluster's log storage is set up to receive the global cluster's audit data, and the owner accepts losing the stored global logs. Rollback: reinstall the plugin on the global cluster; the old data does not return. + + Uninstall the plugin from the global cluster in the console. Then confirm that new global audit records arrive in the business cluster's log storage. + +### Cause 6: Periodic CPU spikes from MergeTree background merges + +1. Confirm that the spikes line up with merges (Diagnostic Step 5). + +2. Make sure the CPU limit leaves the node headroom during a spike. Raising the limit does not remove the spike. + +3. To stop frequent alerts in the meantime, lengthen the trigger duration of the ClickHouse CPU alert rule beyond the length of one merge spike. Edit the rule in the alert policy that holds it, and note the old duration so that you can set it back. + +4. If the log components are older than `log-4.3.2` and the alarms persist after expansion, also apply Cause 3. + +### Cause 7: A resource change in the console does not apply because a resource patch already exists + +1. Confirm that a patch exists for ClickHouse: + + ```bash + kubectl get rpch -A | grep -i clickhouse + ``` + +2. Set the resources on the StatefulSet. + + > **Warning:** Editing the StatefulSet rolls the ClickHouse pods, and log writes and queries are degraded until all replicas are Ready. Precondition: the `rpch` exists, you saved the StatefulSet YAML, and the nodes have the capacity. Rollback: `kubectl apply -f /tmp/clickhouse-sts.before.yaml`. + + ```bash + kubectl -n cpaas-system get sts -o yaml > /tmp/clickhouse-sts.before.yaml + kubectl -n cpaas-system edit sts + ``` + + Set the container resources there. Console changes stay blocked while the `rpch` exists. + +### Cause 8: nevermore and zeus appear to take 10 CPU cores + +No reduction is needed. The 10 cores are the combined CPU limits of `nevermore` and `zeus`. Limits do not reserve CPU and do not reduce what the scheduler can place on the node. The only condition is that the actual CPU usage of all containers on the node stays below the node's total CPU. Show the three numbers side by side: requests, limits and live usage (Diagnostic Step 6). + +## Diagnostic Steps + +1. For `nevermore`, find the node, the container and the reason: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep nevermore + kubectl -n cpaas-system get pod -o jsonpath='{range .status.containerStatuses[*]}{.name}{" restarts="}{.restartCount}{" last="}{.lastState.terminated.reason}{"\n"}{end}' + kubectl -n cpaas-system top pod --containers | grep nevermore + ``` + + `OOMKilled` on the log container of one node is Cause 1. CPU near the 2-core limit on one node, without restarts, is Cause 2. + +2. On the alerting node, watch the disk queue during an alert: + + ```bash + du -sh /cpaas/log/data/diskqueue + top -b -n 1 | head -20 + ``` + + Growth during the alert and a quiet queue outside it support Cause 2. + +3. Check the ClickHouse version and log lines: + + ```bash + kubectl -n cpaas-system logs -c clickhouse --tail=500 | grep -E 'MEMORY_LIMIT_EXCEEDED|Memory limit' + ``` + + Alarms that stay after expansion on log components older than 4.3.2 are Cause 3. `MEMORY_LIMIT_EXCEEDED` with a `maximum` near the limit is Cause 4. + +4. Compare the plugin, the `chi` and the StatefulSet: + + ```bash + kubectl -n cpaas-system get chi cpaas-clickhouse -o yaml | grep -n -A6 resources + kubectl -n cpaas-system get sts | grep -i clickhouse + kubectl -n cpaas-system get sts -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" "}{.resources}{"\n"}{end}' + kubectl get rpch -A 2>/dev/null | grep -i clickhouse || true + ``` + + `chi` values that differ from the StatefulSet with no `rpch` are Cause 5. A console change that does not apply while an `rpch` for ClickHouse exists is Cause 7. + +5. During a periodic CPU spike, list the running merges: + + ```bash + kubectl -n cpaas-system exec -c clickhouse -- clickhouse-client \ + --query "SELECT database, table, elapsed, progress, num_parts FROM system.merges" + ``` + + Active merges during each spike, with an unchanged log and no restart, are Cause 6. + +6. For CPU attributed to `nevermore` and `zeus`, compare requests, limits and usage on one node: + + ```bash + kubectl get pods -A -o wide --field-selector spec.nodeName= | grep -E 'nevermore|zeus' + kubectl -n get pod -o jsonpath='{range .spec.containers[*]}{.name}{" req="}{.resources.requests.cpu}{" lim="}{.resources.limits.cpu}{"\n"}{end}' + kubectl -n top pod --containers + kubectl describe node | sed -n '/Allocated resources/,/Events/p' + ``` + + A large total under `lim=` with small requests and low `top` values is Cause 8. In `Allocated resources`, the `Limits` column may be above 100%. The `Requests` column is what scheduling uses. + +## Related Information + +- [ClickHouse log storage logs RaftInstance errors with pod restarts, merge CPU spikes, or a hand-built ClickHouse on ACP](./ClickHouse_log_storage_logs_RaftInstance_errors_with_pod_restarts_merge_CPU_spikes_or_a_hand-built_ClickHouse_on_ACP.md), for the benign `RaftInstance ... End of file` line. +- [Logging ClickHouse runs out of disk, CPU or memory, or is OOMKilled on ACP](./Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md) +- [nevermore OOMKilled, cache written to /var/cpaas, or callback-secret mount failure on ACP](./nevermore_OOMKilled_cache_written_to_var_cpaas_or_callback-secret_mount_failure_on_ACP.md) diff --git a/docs/en/solutions/razor_does_not_become_ready_after_a_ClickHouse_log_storage_install_or_upgrade_on_ACP.md b/docs/en/solutions/razor_does_not_become_ready_after_a_ClickHouse_log_storage_install_or_upgrade_on_ACP.md new file mode 100644 index 000000000..bb0337245 --- /dev/null +++ b/docs/en/solutions/razor_does_not_become_ready_after_a_ClickHouse_log_storage_install_or_upgrade_on_ACP.md @@ -0,0 +1,225 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# razor does not become ready after a ClickHouse log storage install or upgrade on ACP + +## Issue + +Alauda Container Platform Log Storage for ClickHouse is installed, reinstalled or upgraded. ClickHouse itself runs, but `razor` never becomes ready. `razor` is the gateway in front of the logging ClickHouse, StatefulSet `razor` in `cpaas-system`. `vector` is a container inside the `razor` pod. Typical reports: + +- "`razor` will not start after the install." +- "The `vector` container fails with a read-only file system." +- "`razor` stays in `Error` after the platform upgrade." +- "`razor-0` is `1/2 Running` and restarting on a new deploy." + +The `razor` log may end with: + +```text +Try to do recover from dirty +Failed to prepare {"error": "migration aborted after recoverd"} +``` + +The default data paths are `/cpaas/data/clickhouse` for ClickHouse and `/cpaas/data/vector` for the `vector` buffer. On an immutable OS the writable root for platform data is `/var/cpaas` instead of `/cpaas`. + +When ClickHouse itself never starts, see [ClickHouse log storage plugin install fails or ClickHouse does not start on ACP](./ClickHouse_log_storage_plugin_install_fails_or_ClickHouse_does_not_start_on_ACP.md). + +## Root Cause + +### Cause 1: On an immutable OS, razor fails with a read-only file system or never starts because the vector path lacks /var + +Two variants share this cause: + +- The install from the console wrote the ClickHouse path as `/var/cpaas/data/clickhouse`, but passed `/cpaas/data/vector` to `vector`, without the `/var` prefix. `razor` cannot start. Editing the ClickHouse ModuleInfo in place to `/var/cpaas/data/vector` does not refresh the path of the existing `razor` PVC and PV. Recreating the ModuleInfo with the corrected path does. +- After a disk replacement moved the writable path from `/cpaas` to `/var/cpaas`, the `vector` container fails with `Configuration error` on every sink (`audit`, `event`, `log kubernetes`, `log platform`, `log system`, `log workload`): + + ```text + error occurred when building buffer: failed to build individual stage 0: failed to load/create ledger: ledger I/O error: Read-only file system (os error 30) + run "/usr/local/bin/vector" error: exit status 78 + ``` + + `mount | grep cpaas` on the host shows the disks mounted `rw` on `/var/cpaas`, so the path `vector` was given is not the writable one. The `Transform "dispatcher.unmatched" has no consumers` warning in the same log is not the failure. + +### Cause 2: razor stays in Error after an upgrade because migration_versions_new has a dirty record + +After a platform upgrade, the ClickHouse component is installed and `razor` never starts. Its log shows the migration check, an automatic recovery attempt, and the abort: + +```text +clickhouse/store_setup.go:94 | server | Try to do recover from dirty +server/options.go:169 | server | Failed to prepare {"error": "migration aborted after recoverd"} +``` + +The latest record in the ClickHouse table `migration_versions_new` is marked dirty. `razor` cannot recover from it. Emptying the ClickHouse data directory with `razor` and every ClickHouse replica stopped, then starting them again, lets `razor` start. That discards all stored ClickHouse data. + +### Cause 3: razor fails with migration aborted after recoverd on a new deploy because one replica is missing tables + +While the ClickHouse log storage is being deployed, `razor-0` is `1/2 Running` and restarting. Its log shows `curVersion:10 dirty:false` and `run migration error:no change` against one replica (`chi-cpaas-clickhouse-replicated-0-0:8123`). Against another replica (`chi-cpaas-clickhouse-replicated-0-1:8123`) it shows: + +```text +Try to do recover from dirty +Failed to prepare {"error": "migration aborted after recoverd"} +Failed to apply to server {"error": "migration aborted after recoverd"} +``` + +A later log shows the migration statement and the real error: + +```text +CREATE TABLE IF NOT EXISTS audit_all ON CLUSTER replicated AS audit ENGINE = Distributed(replicated, observability, audit, rand()); +Code: 60. DB::Exception: Table observability.audit does not exist. (UNKNOWN_TABLE) +``` + +ClickHouse had been uninstalled and reinstalled with new PVCs and PVs. The database and table initialization had not completed on one replica, so tables of the `observability` database (such as `audit`) were missing there. The migration against that replica failed and then aborted. This is a known ClickHouse log storage issue. The fix is to create the missing tables on that replica from the definitions on a healthy replica. Unlike Cause 2, the migration record here is not dirty. + +## Resolution + +### Cause 1: On an immutable OS, razor fails with a read-only file system or never starts because the vector path lacks /var + +1. Confirm the signature. On the global cluster: + + ```bash + kubectl get moduleinfo -A | grep -i clickhouse + kubectl get moduleinfo -o yaml | grep -n -E '/cpaas/data|/var/cpaas' + ``` + + On the affected cluster: + + ```bash + kubectl -n cpaas-system get pv | grep razor + kubectl -n cpaas-system logs razor-1 -c vector --tail=50 | grep -E 'os error 30|exit status' + ``` + + On the node, run `mount | grep cpaas`. This cause matches when the ClickHouse path is under `/var/cpaas` and the `vector` path under `/cpaas` in the same ModuleInfo. It also matches when `vector` reports `Read-only file system (os error 30)` while the host mount is `rw`. + +2. Save the ModuleInfo and prepare a corrected copy: + + ```bash + kubectl get moduleinfo -o yaml > /tmp/clickhouse-minfo.backup.yaml + cp /tmp/clickhouse-minfo.backup.yaml /tmp/clickhouse-minfo.new.yaml + ``` + + In `/tmp/clickhouse-minfo.new.yaml`, change the `vector` path to `/var/cpaas/data/vector`. Remove `metadata.uid`, `metadata.resourceVersion`, `metadata.creationTimestamp`, `metadata.managedFields` and `status`. + +3. Recreate the plugin from the corrected copy. + + > **Warning:** Uninstalling ClickHouse from the console removes the log storage and its PVCs, and log search is empty until the recreated plugin ingests again. Precondition: only when this cause is confirmed and the ModuleInfo backup from step 2 is saved on the global cluster. Rollback: apply the saved backup unchanged to return to the previous state. + + Uninstall ClickHouse from the console and wait until the ModuleInfo is gone. Then: + + ```bash + kubectl apply -f /tmp/clickhouse-minfo.new.yaml + kubectl -n cpaas-system get pv | grep razor + ``` + +4. Check that the new `razor` PV points under `/var/cpaas`. An in-place edit of the old ModuleInfo is not enough, because it leaves the `razor` PVC and PV on the old path. + +### Cause 2: razor stays in Error after an upgrade because migration_versions_new has a dirty record + +1. Confirm the signature: + + ```bash + kubectl -n cpaas-system logs -c razor --tail=40 | grep -E 'recover from dirty|migration aborted' + kubectl -n cpaas-system exec -c clickhouse -- clickhouse-client --query \ + "SELECT version, dirty, sequence FROM migration_versions_new ORDER BY sequence DESC LIMIT 1" + ``` + + `dirty` = 1 on the latest row matches this cause. + +2. Stop `razor` and every ClickHouse replica. Record the current `razor` replica count first. + + > **Warning:** Emptying the ClickHouse data directory permanently removes every log, audit and event row stored in this ClickHouse. Precondition: only when this cause is confirmed (`Try to do recover from dirty` followed by `migration aborted after recoverd`, and the latest `migration_versions_new` row is dirty), `razor` and all ClickHouse StatefulSets are at 0 replicas, and the data loss is accepted. Rollback: none. + + ```bash + kubectl -n cpaas-system get sts razor + kubectl -n cpaas-system scale sts razor --replicas=0 + kubectl -n cpaas-system scale sts chi-cpaas-clickhouse-replicated-0-0 --replicas=0 + kubectl -n cpaas-system scale sts chi-cpaas-clickhouse-replicated-0-1 --replicas=0 + kubectl -n cpaas-system scale sts chi-cpaas-clickhouse-replicated-0-2 --replicas=0 + ``` + +3. On every node that runs a logging ClickHouse pod, empty the logging data directory. This is not the MySQL slow-query store, which some clusters keep at `/cpaas/ck`. + + ```bash + cd /cpaas/data/clickhouse && rm -rf ./* + ``` + +4. Start ClickHouse again, then `razor` with its previous replica count: + + ```bash + kubectl -n cpaas-system scale sts chi-cpaas-clickhouse-replicated-0-0 --replicas=1 + kubectl -n cpaas-system scale sts chi-cpaas-clickhouse-replicated-0-1 --replicas=1 + kubectl -n cpaas-system scale sts chi-cpaas-clickhouse-replicated-0-2 --replicas=1 + kubectl -n cpaas-system scale sts razor --replicas= + ``` + +### Cause 3: razor fails with migration aborted after recoverd on a new deploy because one replica is missing tables + +1. Compare the `observability` tables on each replica: + + ```bash + for p in chi-cpaas-clickhouse-replicated-0-0-0 chi-cpaas-clickhouse-replicated-0-1-0 chi-cpaas-clickhouse-replicated-0-2-0; do + echo "== $p" + kubectl -n cpaas-system exec "$p" -c clickhouse -- clickhouse-client --query \ + "SELECT name FROM system.tables WHERE database = 'observability' ORDER BY name" + done + kubectl -n cpaas-system logs razor-0 -c razor --tail=200 | grep -E 'UNKNOWN_TABLE|does not exist' + ``` + + A replica that lacks tables the others have (for example `audit`), with `UNKNOWN_TABLE` in the `razor` log for that replica, matches this cause. + +2. Export the table definitions from a healthy replica: + + ```bash + kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -c clickhouse -- clickhouse-client --format TSVRaw --query \ + "SELECT name AS table_name, create_table_query FROM system.tables WHERE database = 'observability'" > /tmp/observability-ddl.tsv + ``` + +3. Create the missing tables on the affected replica. + + > **Warning:** Creating tables by hand on a replica changes its schema outside the `razor` migration. A statement for a table that already exists on that replica fails, and a wrong statement can register a table that does not match its peers. Precondition: only when this cause is confirmed, the statements come from a healthy replica of the same installation, and you run only the statements for tables missing on the target replica. Rollback: `DROP TABLE observability.
` on that replica removes a table created by mistake. + + Open a client on the replica that is missing tables: + + ```bash + kubectl -n cpaas-system exec -it -c clickhouse -- clickhouse-client + ``` + + Run the `create_table_query` of each missing table. Run the base tables first and the `*_all` Distributed tables that reference them afterwards. + +4. Repeat the comparison from step 1. Then restart `razor-0` so that it runs the migration again, and check that it becomes Ready: + + ```bash + kubectl -n cpaas-system delete pod razor-0 + kubectl -n cpaas-system get pods | grep razor + ``` + +## Diagnostic Steps + +1. Read the plugin and pod state: + + ```bash + kubectl get ars -A | grep -Ei 'click|log' + kubectl -n cpaas-system get pods | grep -Ei 'click|razor' + kubectl -n cpaas-system logs razor-0 -c razor --tail=100 + kubectl -n cpaas-system logs razor-0 -c vector --tail=100 + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | A mix of `/var/cpaas` and `/cpaas` paths in the ModuleInfo on an immutable OS, or `Read-only file system (os error 30)` in the `vector` container | Cause 1 | + | `razor` in `Error` after an upgrade, `Try to do recover from dirty`, and the latest `migration_versions_new` row has `dirty` = 1 | Cause 2 | + | `migration aborted after recoverd` on a new deploy, `UNKNOWN_TABLE` for an `observability` table, and one replica missing tables the others have | Cause 3 | + +3. The `RaftInstance ... End of file` lines that ClickHouse prints repeatedly also appear on correctly deployed clusters. They do not identify any of these causes. + +## Related Information + +- Related articles: [ClickHouse log storage plugin install fails or ClickHouse does not start on ACP](./ClickHouse_log_storage_plugin_install_fails_or_ClickHouse_does_not_start_on_ACP.md) covers the first-install failures where ClickHouse itself does not come up. +- [ClickHouse log queries fail after the log storage had been working: missing tables or broken parts on ACP](./ClickHouse_log_queries_fail_after_the_log_storage_had_been_working_missing_tables_or_broken_parts_on_ACP.md) From fb550bc36393533d01d4495898c2178aebd7eac0 Mon Sep 17 00:00:00 2001 From: Komh Date: Thu, 12 Nov 2026 21:32:15 +0800 Subject: [PATCH 10/25] docs(solutions): add 28 and update 1 troubleshooting articles --- ...apollo_cannot_reach_the_receiver_on_ACP.md | 151 ++++++++++++ ...are_rejected_or_cannot_be_edited_on_ACP.md | 135 +++++++++++ ..._delete_hangs_because_of_captain_on_ACP.md | 220 ++++++++++++++++++ ...er_Cluster_Enhancer_is_installed_on_ACP.md | 62 +++++ ..._sentry_is_OOM_hung_or_CPU_bound_on_ACP.md | 187 +++++++++++++++ ...lobal_cannot_reach_an_API_server_on_ACP.md | 146 ++++++++++++ ...he_marketplace_backends_are_slow_on_ACP.md | 189 +++++++++++++++ ...load_cluster_ALB_or_a_credential_on_ACP.md | 143 ++++++++++++ ...e_plugin_or_product_entry_is_off_on_ACP.md | 134 +++++++++++ ...rors_after_a_host_network_change_on_ACP.md | 141 +++++++++++ ...ter_installation_or_a_VIP_change_on_ACP.md | 197 ++++++++++++++++ ...rier_on_the_global_cluster_fails_on_ACP.md | 195 ++++++++++++++++ ...olicies_and_notification_routing_on_ACP.md | 208 +++++++++++++++++ ...supported_chart_types_or_no_data_on_ACP.md | 170 ++++++++++++++ ...eded_quota_and_no_built_in_alert_on_ACP.md | 116 +++++++++ ...ns_with_errcode_300001_or_310000_on_ACP.md | 136 +++++++++++ ...cpc-exec_feature_gate_is_enabled_on_ACP.md | 17 +- ...nly_Email_is_offered_as_a_method_on_ACP.md | 113 +++++++++ ..._zero_while_the_backend_has_data_on_ACP.md | 151 ++++++++++++ ..._built_in_dashboards_are_missing_on_ACP.md | 151 ++++++++++++ ...80_on_the_cluster_VIP_is_blocked_on_ACP.md | 100 ++++++++ ...ollo_holds_a_stale_cluster_cache_on_ACP.md | 172 ++++++++++++++ ..._an_empty_mountpoint_or_no_value_on_ACP.md | 95 ++++++++ ...ob_instance_or_the_workload_name_on_ACP.md | 115 +++++++++ ...cer_DNS_or_access_address_change_on_ACP.md | 141 +++++++++++ ...mal_while_their_Pods_are_Running_on_ACP.md | 96 ++++++++ ...r_or_console_file_transfer_fails_on_ACP.md | 204 ++++++++++++++++ ...or_miss_a_required_custom_header_on_ACP.md | 161 +++++++++++++ ...ithout_the_notification_template_on_ACP.md | 163 +++++++++++++ 29 files changed, 4208 insertions(+), 1 deletion(-) create mode 100644 docs/en/solutions/Alert_notifications_are_not_delivered_because_courier-api_or_apollo_cannot_reach_the_receiver_on_ACP.md create mode 100644 docs/en/solutions/Alert_rules_do_not_fire_are_rejected_or_cannot_be_edited_on_ACP.md create mode 100644 docs/en/solutions/Application_updates_are_slow_status_stays_Failed_or_Processing_or_delete_hangs_because_of_captain_on_ACP.md create mode 100644 docs/en/solutions/Cluster_Overview_page_shows_no_monitoring_charts_after_Cluster_Enhancer_is_installed_on_ACP.md create mode 100644 docs/en/solutions/Cluster_plugins_stay_Deploying_or_AppReleases_stall_because_sentry_is_OOM_hung_or_CPU_bound_on_ACP.md create mode 100644 docs/en/solutions/Console_is_inaccessible_or_a_business_cluster_shows_0_nodes_because_global_cannot_reach_an_API_server_on_ACP.md create mode 100644 docs/en/solutions/Console_is_slow_or_shows_data_loading_errors_because_calls_through_erebus_or_the_marketplace_backends_are_slow_on_ACP.md create mode 100644 docs/en/solutions/Console_menu_bar_or_Data_Services_navigation_does_not_load_because_of_a_workload_cluster_ALB_or_a_credential_on_ACP.md create mode 100644 docs/en/solutions/Console_menu_or_feature_is_missing_because_a_feature_gate_plugin_or_product_entry_is_off_on_ACP.md create mode 100644 docs/en/solutions/Console_returns_500_or_data_loading_errors_after_a_host_network_change_on_ACP.md create mode 100644 docs/en/solutions/Console_returns_502_or_does_not_open_after_installation_or_a_VIP_change_on_ACP.md create mode 100644 docs/en/solutions/Console_returns_HTTP_500_or_a_blank_page_because_apollo_or_courier_on_the_global_cluster_fails_on_ACP.md create mode 100644 docs/en/solutions/Create_and_manage_alert_policies_and_notification_routing_on_ACP.md create mode 100644 docs/en/solutions/Custom_or_imported_monitoring_dashboards_show_parse_errors_unsupported_chart_types_or_no_data_on_ACP.md create mode 100644 docs/en/solutions/Deployment_rollout_stalls_with_FailedCreate_exceeded_quota_and_no_built_in_alert_on_ACP.md create mode 100644 docs/en/solutions/DingTalk_robot_rejects_alert_notifications_with_errcode_300001_or_310000_on_ACP.md create mode 100644 docs/en/solutions/Email_alert_notifications_fail_with_535_or_never_arrive_or_only_Email_is_offered_as_a_method_on_ACP.md create mode 100644 docs/en/solutions/Monitoring_dashboard_panels_are_blank_or_drop_to_zero_while_the_backend_has_data_on_ACP.md create mode 100644 docs/en/solutions/Monitoring_page_freezes_the_browser_or_built_in_dashboards_are_missing_on_ACP.md create mode 100644 docs/en/solutions/Monitoring_pages_return_502_or_context_deadline_exceeded_because_port_11780_on_the_cluster_VIP_is_blocked_on_ACP.md create mode 100644 docs/en/solutions/Monitoring_pages_stay_empty_because_courier-api_or_apollo_holds_a_stale_cluster_cache_on_ACP.md create mode 100644 docs/en/solutions/Node_disk_alert_notification_shows_an_empty_mountpoint_or_no_value_on_ACP.md create mode 100644 docs/en/solutions/Notification_template_cannot_show_metric_labels_such_as_job_instance_or_the_workload_name_on_ACP.md create mode 100644 docs/en/solutions/Platform_access_fails_or_navigation_entries_disappear_after_a_load_balancer_DNS_or_access_address_change_on_ACP.md create mode 100644 docs/en/solutions/Platform_health_page_shows_components_abnormal_while_their_Pods_are_Running_on_ACP.md create mode 100644 docs/en/solutions/Web_terminal_exec_window_debug_container_or_console_file_transfer_fails_on_ACP.md create mode 100644 docs/en/solutions/Webhook_alert_notifications_fail_with_invalid_header_field_value_or_miss_a_required_custom_header_on_ACP.md create mode 100644 docs/en/solutions/Webhook_alert_notifications_rejected_as_InvalidJson_or_sent_without_the_notification_template_on_ACP.md diff --git a/docs/en/solutions/Alert_notifications_are_not_delivered_because_courier-api_or_apollo_cannot_reach_the_receiver_on_ACP.md b/docs/en/solutions/Alert_notifications_are_not_delivered_because_courier-api_or_apollo_cannot_reach_the_receiver_on_ACP.md new file mode 100644 index 000000000..c1c4b8599 --- /dev/null +++ b/docs/en/solutions/Alert_notifications_are_not_delivered_because_courier-api_or_apollo_cannot_reach_the_receiver_on_ACP.md @@ -0,0 +1,151 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.0" + - "4.2" + - "4.3" +--- + +# Alert notifications are not delivered because courier-api or apollo cannot reach the receiver on ACP + +## Issue + +The real-time alert page shows a firing alert, but the mailbox, Feishu bot, DingTalk group, or external monitoring center receives nothing. A test send times out, a webhook POST fails with `tls: failed to verify certificate`, or webhook messages to a receiver addressed by host name only start working after DNS entries are added. A related question is which component, and which nodes, need network access to the mail server or the webhook endpoint for an egress allow-list. + +On Alauda Container Platform: + +- **Mail** notifications are sent by `apollo`. Its Pod is not pinned to a node and can move after a restart, so the mail server must be reachable from **all nodes of the `global` cluster**. +- **DingTalk, Feishu, and webhook** notifications are sent by `courier-api`, which can be scheduled onto any node of the cluster it runs on (normally `global`). A firewall rule that allows only one node is not enough. + +If the receiver answers with an HTTP error such as `errcode` or `InvalidJson`, the request already left the cluster; see [Webhook alert notifications rejected as InvalidJson or sent without the notification template on ACP](Webhook_alert_notifications_rejected_as_InvalidJson_or_sent_without_the_notification_template_on_ACP.md) or [DingTalk robot rejects alert notifications with errcode 300001 or 310000 on ACP](DingTalk_robot_rejects_alert_notifications_with_errcode_300001_or_310000_on_ACP.md). Email failures with `535` or emails marked as sent that never arrive are in [Email alert notifications fail with 535 or never arrive, or only Email is offered as a method on ACP](Email_alert_notifications_fail_with_535_or_never_arrive_or_only_Email_is_offered_as_a_method_on_ACP.md). A missing custom header or `invalid header field value` is in [Webhook alert notifications fail with invalid header field value or miss a required custom header on ACP](Webhook_alert_notifications_fail_with_invalid_header_field_value_or_miss_a_required_custom_header_on_ACP.md). + +## Root Cause + +### Cause 1: the sender cannot reach the SMTP or webhook server and times out + +A test send to Feishu, DingTalk, a webhook, or email times out, for example a `courier-api` connect timeout to the Feishu bot webhook on port 443 that still fails after retries. Email fails the same way when the `global` cluster cannot open the configured SMTP server. Connectivity from a laptop, or from a single node, does not prove the path: the sending Pod can run on any node, so every node needs the route, and an allow-list that names a Pod does not match this traffic. + +For QQ Mail, the working settings are SMTP over SSL on port 465 or 587, the full mailbox address as the user name, and the generated authorization code, not the account password, as the password. + +### Cause 2: tls: failed to verify certificate because the certificate does not name the webhook host + +The webhook POST fails and the notification shows `status: False`, `type: Ready`, `phase: Failed` with a message of this shape: + +```text +tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, 192.0.2.50, not 192.0.2.4 +``` + +The certificate presented by the webhook endpoint does not list the host used in the webhook URL. On platform v3.18.2, HTTPS certificate verification for webhook notifications cannot be skipped. + +### Cause 3: the webhook host name does not resolve inside the cluster + +Webhook-forwarded alerts to an external system are not delivered, and they are delivered as soon as host entries for the receiver are added to CoreDNS on the `global` cluster. `courier-api` runs as a Pod and resolves the webhook host through cluster DNS (CoreDNS). A name that is known only to a resolver that CoreDNS does not forward to, or only to `/etc/hosts` on a workstation, is not resolvable from the Pod. + +## Resolution + +### Cause 1: the sender cannot reach the SMTP or webhook server and times out + +1. Find the sending Pods and their nodes, then read the send log. Do not restart `courier-api` to "refresh" a timeout; the next schedule lands on the same blocked path. + + ```bash + kubectl get pods -A -o wide | grep -E 'courier-api|apollo' + kubectl logs -n cpaas-system deploy/courier-api --tail=200 | grep -i -E 'Failed to send|timeout|dial|refused|errcode' + kubectl logs -n cpaas-system --tail=200 | grep -i -E 'timeout|dial|refused' + ``` + + Use the namespace from the first command if the Pods are not in `cpaas-system`. For a mail failure, read both logs. + +2. From the node that hosts the Pod (or from a debug Pod with `hostNetwork` on that node), open the same host and port that the log names, using the real webhook URL or SMTP host: + + ```bash + curl -sv --max-time 10 "https:///" + ``` + +3. A timeout here, matching the timeout in the log, is the firewall or the route. Allow the webhook port (443 for a Feishu or DingTalk robot) and the SMTP port from every node of the cluster that runs the sender, or from the NAT address those nodes share. Allow by node address, not by Pod; the Pod name changes on every reschedule. + +4. Send one test notification. Success is a send log line without a timeout and a message at the receiver. An HTTP response with `errcode` means the path works and the payload is wrong. + +### Cause 2: tls: failed to verify certificate because the certificate does not name the webhook host + +1. Confirm the error before changing a URL or a certificate: + + ```bash + kubectl logs -n cpaas-system deploy/courier-api --tail=300 | grep -F 'failed to verify certificate' + ``` + +2. Use one of the two recoveries: + - Issue a certificate on the webhook endpoint that includes the host used in the URL. This is done on the webhook server, not in the cluster. + - Change the webhook address to a host that the certificate already lists (the names in the `x509` message). + + > **Warning:** Changing the webhook address stops posts to the previous endpoint as soon as the next alert fires. Do this only when the log says the certificate is valid for names other than the host in the current URL. Write down the current URL first; to roll back, put it back. + +### Cause 3: the webhook host name does not resolve inside the cluster + +1. Check whether the webhook host resolves from a Pod on the `global` cluster: + + ```bash + kubectl logs -n cpaas-system deploy/courier-api --tail=300 | grep -i -E 'lookup|no such host' + kubectl run dns-check -n cpaas-system --rm -it --restart=Never --image= -- nslookup + ``` + + `NXDOMAIN` or no answer from the debug Pod, while the same host resolves from the site's own resolver, matches this cause. + +2. Preferably, make the name resolvable on the DNS servers that the nodes use, so that CoreDNS forwards it. If that is not possible, add a static record to cluster DNS. The persistent way is a **Host Alias** entry in the **Alauda Container Platform Cluster DNS** plugin, as described in [Platform console does not open because the cluster VIP domain name cannot be resolved inside the cluster on ACP](Platform_console_does_not_open_because_the_cluster_VIP_domain_name_cannot_be_resolved_inside_the_cluster_on_ACP.md). + +3. To add the record directly to the Corefile, find the ConfigMap that the `coredns` Deployment mounts (for example `kube-system/cpaas-coredns`, data key `Corefile`) and export it: + + ```bash + kubectl get deploy -n kube-system coredns -o jsonpath='{.spec.template.spec.volumes}{"\n"}' + kubectl get cm -n kube-system cpaas-coredns -o yaml > /tmp/cpaas-coredns.yaml + ``` + + > **Warning:** The Corefile drives DNS for every Pod in the cluster. A syntax error makes CoreDNS fail to parse it and crash-loop, which breaks name resolution cluster-wide. Do this only when the webhook host does not resolve from a Pod and the site's DNS cannot serve it. A manual edit can be overwritten when the Cluster DNS plugin is reconciled. To roll back, run `kubectl apply -f /tmp/cpaas-coredns.yaml` and confirm the `coredns` Pods are `Running` and `Ready`. + + ```bash + kubectl edit cm -n kube-system cpaas-coredns + ``` + + Add a `hosts` block to the existing server block, before the `forward` line, and keep `fallthrough` so that every other name still resolves: + + ```text + hosts { + 192.0.2.10 + fallthrough + } + ``` + + If the Corefile has the `reload` plugin, CoreDNS picks up the change on its own; otherwise run `kubectl rollout restart deploy -n kube-system coredns`. + +4. Repeat the `nslookup` from the debug Pod, then trigger one alert. Success is a `courier-api` send for that webhook with phase `Done` and the message at the receiver. + +## Diagnostic Steps + +1. Locate the sender. For mail it is `apollo`; for DingTalk, Feishu, and webhooks it is `courier-api`. List where it runs and where it can run: + + ```bash + kubectl get pods -A -o wide | grep -E 'courier-api|apollo' + kubectl get deploy -A | grep courier-api + kubectl get deploy -n courier-api -o jsonpath='{.spec.template.spec.nodeSelector}{"\n"}{.spec.template.spec.affinity}{"\n"}' + ``` + + The destination port is the one in the webhook URL (443 for `https://`). + +2. Read the `courier-api` log, and the `apollo` log for mail, for one failed send: + - `dial` or `i/o timeout`: Cause 1. + - `failed to verify certificate`: Cause 2. + - `lookup` or `no such host`, or a host name that a debug Pod cannot resolve: Cause 3. + - `errcode` or `InvalidJson`: the path works; use the payload articles linked in the Issue section. + - `invalid header field value`: nothing was sent; use the header article. + +3. If a webhook receiver gets some alerts and not others, read the `webhook-business` log. A render failure with invalid JSON means the alert was discarded before sending, which is a template problem, not the network: + + ```bash + kubectl get pods -A | grep webhook-business + kubectl logs -n --tail=300 | grep -i -E 'json|render|invalid' + ``` + +4. After a network or DNS change, repeat the test from a Pod on a different node. One successful node does not cover the scheduler. diff --git a/docs/en/solutions/Alert_rules_do_not_fire_are_rejected_or_cannot_be_edited_on_ACP.md b/docs/en/solutions/Alert_rules_do_not_fire_are_rejected_or_cannot_be_edited_on_ACP.md new file mode 100644 index 000000000..5268c26c3 --- /dev/null +++ b/docs/en/solutions/Alert_rules_do_not_fire_are_rejected_or_cannot_be_edited_on_ACP.md @@ -0,0 +1,135 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Alert rules do not fire, are rejected, or cannot be edited on ACP + +## Issue + +The metric or the log line is there, but the real-time alert page stays empty, or creating the alert policy is rejected, or a change to a built-in policy does not last. Typical symptoms: + +- A log keyword alert chart goes from "no data" to `0` after the first matching line, and the policy does not fire. +- Creating the alert policy fails with `parse error: expected type instant vector in aggregation expression, got range vector`. +- A custom metric rule never appears on the real-time page, and the `VMRule` fails validation with an annotation template parsing error and an undefined variable. +- Rules exported from a Prometheus cluster do not load on a VictoriaMetrics cluster. +- A built-in (platform) alert policy is edited or deleted, and the change is reverted. The question is whether built-in policies can be modified, deleted, or extended. + +On Alauda Container Platform, platform rules are `AlertRule` objects (`ait.alauda.io/v1`) shipped by the `cpaas-monitor` module plugin (chart `chart-cpaas-monitor`). The `warlock` controller reconciles each `AlertRule` into a `PrometheusRule`, a `VMRule` on VictoriaMetrics clusters, and the Alertmanager configuration. + +## Root Cause + +### Cause 1: the first keyword match only creates the workload_log_keyword_count series + +A log keyword alert evaluates a rate over `workload_log_keyword_count`. Before the keyword has ever matched, there is no series at all; the value is not `0`. The first match only creates the series. A later match in a following window produces the increase that can satisfy `>= 1`. This is expected, not a missed evaluation. + +### Cause 2: a range vector is passed straight into an aggregation + +The mutating webhook `prometheusrules.mutating.alauda.io` rejects the object with `parse error: expected type instant vector in aggregation expression, got range vector`. The expression passes a range vector such as `metric[5m]` directly into `sum`, `avg`, `max`, `min`, or `count`, which require an instant vector. + +### Cause 3: an annotation template references an undefined variable + +The expression is true in vmui, but even a test rule such as CPU above 1 percent produces no real-time alert, and the controller log reports an annotation template parsing error with an undefined variable. The annotation template is evaluated when the `VMRule` is admitted, so the whole rule stays inactive. Plugin status and metric queries can both look healthy. + +### Cause 4: built-in alert rules cannot be deleted, and direct edits are reverted + +The rules the platform ships cannot be deleted, and direct modifications can be reverted by the platform: the `warlock` controller re-renders the generated `PrometheusRule` and `VMRule` objects from the `AlertRule`, and a module upgrade of `cpaas-monitor` can restore the chart copy of the `AlertRule` itself. Changing or extending a built-in policy in place is therefore not a supported way to get a lasting rule. Adding a label from metrics into a built-in alert expression is not an in-product configuration (stated for v4.2.4). + +An existing alert policy also refuses a change of compute component, and a template policy accepts only one compute component, because the rule set is bound to that resource type. + +Some conditions have no built-in rule at all: there is no alert for maintenance or license expiry, and on 4.0 `cpaas-cluster-rules` covers controller and scheduler health but no default policy covers the APIServer restart count or etcd read/write latency, although the dashboard already has metrics such as `etcd_disk_wal_fsync_duration_seconds_bucket`, `etcd_server_proposals_pending`, `etcd_server_proposals_failed_total`, and `grpc_server_handled_total{grpc_code!="OK"}`. These need a custom policy. + +### Cause 5: VictoriaMetrics evaluates VMRule, not PrometheusRule + +On a VictoriaMetrics cluster (for example ACP 4.3.2), the evaluator reads `VMRule` objects, not `PrometheusRule`, and a `VMRule` needs the `victoriametrics` label before `vmalert` selects it. A `PrometheusRule` exported from a Prometheus cluster (for example 3.10) is therefore not evaluated. + +## Resolution + +### Cause 1: the first keyword match only creates the workload_log_keyword_count series + +Do not lower the threshold and do not restart Alertmanager. Emit the keyword again after the series exists, and evaluate this shape (replace the matchers): + +```promql +round(sum(rate(workload_log_keyword_count{cluster_name="",kind=~"(?i:(Deployment))",name="",namespace="",query=""}[300s])) * 300) >= 1 +``` + +The notification does not contain the log line; it carries `query`, `cluster`, `namespace`, `name`, and `kind`. Retrieve the line from the log query pages or APIs. If the keyword never appears in the log query, the problem is log collection, not this rule. + +### Cause 2: a range vector is passed straight into an aggregation + +Wrap the range in `rate()` or `increase()` before aggregating, or drop the range selector if the metric is already an instant vector. Rejected: + +```promql +sum by (namespace) (container_cpu_usage_seconds_total[5m]) +``` + +Accepted: + +```promql +sum by (namespace) (rate(container_cpu_usage_seconds_total[5m])) +``` + +Run the expression in vmui before creating the policy. vmui is at `/clusters//vmselect/vmui/`; after login, remove a trailing `prometheus/` from the server URL in vmui and save, or the query page uses the wrong API. + +### Cause 3: an annotation template references an undefined variable + +1. Find the rule and the evaluator Pods: + + ```bash + kubectl get vmrule -A + kubectl get vmrule -n -o yaml + kubectl get pods -n cpaas-system | grep -E 'vmalert|vmoperator|victoria' + ``` + +2. Search the operator or `vmalert` Pod log for `template` and `undefined`. +3. In the custom annotation, remove or correct the unknown variable. Only labels, annotations, and built-in alert fields (`.labels.*`, `.annotations.*`, `.status`, `.startsAt`, `.endsAt`) can be referenced. +4. Save, confirm that the `VMRule` no longer reports a validation error, and check the real-time page again. + +### Cause 4: built-in alert rules cannot be deleted, and direct edits are reverted + +1. Do not edit or delete built-in rules. To alert on something else, or with a different threshold, create a custom alert policy (from the alert policy page, or as a separate `AlertRule`) next to the built-in ones, and copy the built-in expression into it if useful. +2. To stop a built-in rule from notifying, silence its alert policy from the console (**Set Silence**) instead of editing its `expr`. +3. For a different compute component, create a new policy; existing policies and template policies stay bound to their component. +4. Inventory rules from `PrometheusRule`, not only from `AlertRule`; `kubectl get alertrules.ait.alauda.io -A` misses policies that exist only as rendered objects: + + ```bash + kubectl get prometheusrules.monitoring.coreos.com -A + kubectl get prometheusrules.monitoring.coreos.com -n -o yaml + kubectl get vmrule -A + ``` + +Wrapped metric names such as `cpaas_advanced_kube_workload_status_replicas_available` (available replicas of a workload) and `cpaas_advanced_kube_pod_container_status_ready` (`0` not ready, `1` ready) are recording rules. Read their native expression from the recording rule in the `PrometheusRule` instead of guessing. + +### Cause 5: VictoriaMetrics evaluates VMRule, not PrometheusRule + +1. Export the old rules and look at an existing `VMRule` on the target cluster: + + ```bash + kubectl get prometheusrule -A -o yaml > /tmp/prometheusrule-export.yaml + kubectl get vmrule -A + kubectl get vmrule -n cpaas-system -o jsonpath='{.items[0].apiVersion}{"\n"}' + ``` + +2. Convert each group to a `VMRule`, copy `apiVersion` from the existing `VMRule`, and set the `victoriametrics` label as it appears on the existing `VMRule` objects. +3. Apply, then run every expression in vmui and confirm it returns data. An expression that worked on Prometheus can be empty on VictoriaMetrics if a metric or label name changed. + +## Diagnostic Steps + +1. Find the object. Use the `prometheusrules.monitoring.coreos.com` list, and `vmrule` on a VictoriaMetrics cluster. + +2. A create or update that returns the range-vector parse error: Cause 2. Expected after the fix: the policy is created. + +3. The object exists, vmui shows the expression is true, and the real-time page is empty: read the `VMRule` and the operator log for an annotation template error (Cause 3). + +4. For a log keyword policy, query `workload_log_keyword_count` with the same matchers. No series: the keyword has not matched yet, or logs are not collected. One series that just appeared with the chart at `0`: Cause 1. Wait for a second match. + +5. Rules carried over from a Prometheus cluster: compare `kubectl get prometheusrule -A` with `kubectl get vmrule -A`. A rule that exists only as a `PrometheusRule`, or a `VMRule` without the `victoriametrics` label, is Cause 5. + +6. A firing rule whose notification never arrives is not a rule problem; see [Alert notifications are not delivered because courier-api or apollo cannot reach the receiver on ACP](Alert_notifications_are_not_delivered_because_courier-api_or_apollo_cannot_reach_the_receiver_on_ACP.md). diff --git a/docs/en/solutions/Application_updates_are_slow_status_stays_Failed_or_Processing_or_delete_hangs_because_of_captain_on_ACP.md b/docs/en/solutions/Application_updates_are_slow_status_stays_Failed_or_Processing_or_delete_hangs_because_of_captain_on_ACP.md new file mode 100644 index 000000000..e60953923 --- /dev/null +++ b/docs/en/solutions/Application_updates_are_slow_status_stays_Failed_or_Processing_or_delete_hangs_because_of_captain_on_ACP.md @@ -0,0 +1,220 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Application updates are slow, status stays Failed or Processing, or delete hangs because of captain on ACP + +## Issue + +Native applications and Helm chart applications are reconciled by `captain` (`captain-controller-manager` and `captain-chartmuseum` in `cpaas-system`) through `HelmRequest` objects. When it is blocked, the console is open but: + +- After a native application's template is updated, the Pods start rolling only after a long time (about 10 minutes, or never). +- A native application shows `失败(0/1)` or `失败(0/0)` (Failed) while its containers are `Running`. +- A pipeline-published native application stays on `处理中(1/1)` (Processing). +- Deleting a Helm application stays on `删除中` (Deleting) with `处理中(0/0)` for hours. +- `captain` crash-loops and application updates fail. + +The `captain` log often contains `Unauthorized, requeuing`, `failed to list *v1.HelmRequest: Unauthorized`, or a `nil pointer dereference` panic in the `chartrepo` controller. + +## Root Cause + +### Cause 1: one HelmRequest worker per cluster is blocked by failing chart applications + +`captain` has one worker per cluster for `HelmRequest`. Failed HelmRequests keep retrying with back-off and occupy that worker, so new native application updates wait in the queue behind them. The `manager` container of `captain-controller-manager` repeats, for many different HelmRequests: + +```text +helmrequest.go:31] Start sync helmrequest: / +clusterwatch.go:262] error syncing '/': Unauthorized, requeuing +``` + +Chart applications left abnormal from earlier failures cause this: updates that normally take about 3 minutes take about 10 minutes or never take effect. Restarting `captain-controller-manager` releases the stuck applications, but later updates stay slow until the failing chart applications are fixed or cleaned up. + +### Cause 2: the native application status is out of sync with its workloads + +The native application status combines `assemblyPhase` with whether the workload is running. `assemblyPhase` changes only when the application is created or updated through the console or the API. When several people trigger updates, those updates can conflict and `assemblyPhase` falls out of sync with the running Pods, which shows as `失败` (Failed) while the services work, sometimes with a tooltip from an old failed update. Triggering an update of the application from the console (**Update**, then **Confirm** with no change) restores the status. When the workloads are healthy, this status can be ignored. + +### Cause 3: a pipeline-published application keeps spec.assemblyPhase Pending + +The Application stays `处理中(1/1)` while its Deployment is `Running` and its events are clean, and a hand-created application of the same shape is normal. The Application (`app.k8s.io/v1beta1`) has `spec.assemblyPhase: Pending`. Setting it to `Succeeded` is the recommended fix. When that edit has no effect and the captain controller logs `failed to list *v1.HelmRequest: Unauthorized` (and ChartRepo objects log `not ready for resync`), the controller cannot list or watch HelmRequests, and restarting it lets the release finish. + +### Cause 4: captain cannot authenticate to the chart repository + +The application stays `失败(0/1)` while its workload is `Running`. The captain log shows the HelmRequest failing to fetch the chart: + +```text +get chart {"name": "/", "version": ""} +"reason": "FailedSync", "message": "Unauthorized" +``` + +The ChartRepo keeps logging `reconcile start`, and under **App Store Management** > **Template Repositories** the repository shows `同步中` (Syncing). The repository rejected the configured credentials. + +### Cause 5: captain does not reload its ServiceAccount token + +A chart that changes RBAC fails an update for lack of permission, and a later update as administrator still shows `Failed`. When `captain` reloads its ServiceAccount token, the reload does not work, and Helm updates start failing after some time. This is a product defect fixed in later versions; restarting `captain` is the workaround. + +### Cause 6: a finalizer blocks the delete of a Helm application + +The application list shows `删除中` with `处理中(0/0)`, the workloads are gone (`kubectl get deployment,pod -n ` returns `No resources found`), and the Application is in phase `Pending` with no events. `captain-controller-manager` repeats `error syncing ... Unauthorized, requeuing` and `Failed to watch *v1.HelmRequest: failed to list *v1.HelmRequest: Unauthorized`. Restarting it stops those errors but does not finish the delete; the finalizer on the Application is still set. + +### Cause 7: captain crash-loops while sorting a chart repository index + +The previous `captain` log shows, while the `chartrepo` controller reconciles a ChartRepo in `cpaas-system`: + +```text +Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference +"controller": "chartrepo", "controllerGroup": "app.alauda.io", "controllerKind": "ChartRepo" +``` + +The stack runs through `repo.ChartVersions.Less`, `IndexFile.SortEntries`, `loadIndex`, and `(*ChartRepoReconciler).syncCharts`: `captain` panics while sorting that repository's index entries. Removing the repository from the platform returns `captain` to `Running`. + +## Resolution + +### Cause 1: one HelmRequest worker per cluster is blocked by failing chart applications + +1. Confirm that failing HelmRequests occupy the worker, and list abnormal applications: + + ```bash + kubectl logs -n cpaas-system deploy/captain-controller-manager -c manager --tail=500 | grep -E 'Start sync helmrequest|Unauthorized, requeuing' | sort | uniq -c | sort -rn | head + kubectl get helmrequest -A + kubectl get application -A | grep -v Running + ``` + +2. Fix or clean up the chart applications whose HelmRequests keep failing. This is the lasting step. + +3. To release updates that are already stuck, restart `captain-controller-manager`. + + > **Warning:** HelmRequest processing pauses until the new Pod is `Ready`. A restart alone leaves later updates slow while failing HelmRequests remain. The Pod is recreated from the same spec. + + ```bash + kubectl delete pod -n cpaas-system + ``` + +4. Once the failing applications are fixed or removed, native updates take effect within a few minutes. Also check the workload's own rollout strategy, for example `maxUnavailable: 0` with a readiness probe that does not pass, before attributing a slow rollout to `captain`. + +### Cause 2: the native application status is out of sync with its workloads + +Confirm that the containers are `Running`, then open **Update** on the application and click **Confirm** without changes. If the status stays failed, read the `captain` log before repeating. + +### Cause 3: a pipeline-published application keeps spec.assemblyPhase Pending + +1. Export the Application and set the field: + + > **Warning:** Setting `spec.assemblyPhase` by hand tells the console that the application finished assembling, and a wrong value can hide a real failure. Do this only when the workloads are `Running` and the events are clean. To roll back, apply the export. + + ```bash + kubectl get application -n -o yaml > "$HOME/-application.yaml" + kubectl patch application -n --type=merge -p '{"spec":{"assemblyPhase":"Succeeded"}}' + ``` + +2. If the edit has no effect, check the captain controller log and restart it: + + ```bash + kubectl get pod -n cpaas-system | grep captain + kubectl logs -n cpaas-system --tail=200 | grep -E 'failed to list \*v1.HelmRequest|not ready for resync' + ``` + + > **Warning:** Application releases pause until the new Pod is `Ready`. The Pod is recreated from the same spec. + + ```bash + kubectl delete pod -n cpaas-system + ``` + +### Cause 4: captain cannot authenticate to the chart repository + +1. Confirm the chart fetch failure: + + ```bash + kubectl logs -n cpaas-system deploy/captain-controller-manager --tail=300 | grep -E 'get chart|FailedSync|Unauthorized' + ``` + +2. Open **App Store Management** > **Template Repositories**, select the repository, and correct the user name and password in its update dialog. + +3. Restart the captain Pods. + + > **Warning:** Chart serving and HelmRequest processing pause until the Pods are `Ready`. Do this only after the credentials have been corrected. + + ```bash + kubectl delete pod -n cpaas-system + ``` + +4. The repository leaves `同步中` and the application leaves `失败`. If `Unauthorized` remains with correct credentials, test the repository login from outside the platform. + +### Cause 5: captain does not reload its ServiceAccount token + +> **Warning:** Helm updates pause until the new Pod is `Ready`. The Pod is recreated from the same spec. + +```bash +kubectl get pods -n cpaas-system | grep captain +kubectl delete pod -n cpaas-system +``` + +Retry the Helm update after the new Pod is `Running`. Upgrade to a version that contains the fix. + +### Cause 6: a finalizer blocks the delete of a Helm application + +1. Export the Application and check `captain`: + + ```bash + kubectl get application -n -o yaml > "$HOME/-application.yaml" + kubectl logs -n cpaas-system deploy/captain-controller-manager --tail=200 | grep -E 'Unauthorized|error syncing' + ``` + +2. Restart `captain-controller-manager`, then remove the `finalizers` field from the Application. + + > **Warning:** Removing finalizers deletes the object without controller cleanup, and orphaned releases can remain. Do this only when the delete is stuck, the workloads are already gone, and a restart of `captain-controller-manager` did not finish it. This cannot be rolled back. If the finalizer is `chart.app.alauda.io` and the matching `helmrequests.app.alauda.io` object does not exist, this cause does not apply. + + ```bash + kubectl delete pod -n cpaas-system + kubectl edit application -n + ``` + +### Cause 7: captain crash-loops while sorting a chart repository index + +1. Find the repository named in the panic: + + ```bash + kubectl get pods -n cpaas-system | grep captain + kubectl logs -n cpaas-system --previous --tail=300 | grep -E 'Observed a panic|ChartRepo|SortEntries|loadIndex' + kubectl get chartrepo -n cpaas-system + ``` + +2. Save it and remove it from the platform chart repository list. + + > **Warning:** Applications that pull charts from this repository can no longer update. Do this only when `captain` crash-loops with the `chartrepo` nil pointer panic and the log names this ChartRepo. Applying the export restores it, and brings the crash loop back until its index is fixed. + + ```bash + kubectl get chartrepo -n cpaas-system -o yaml > "$HOME/-chartrepo.yaml" + ``` + +3. `captain` stays `Running`. + +## Diagnostic Steps + +```bash +kubectl get application -n -o yaml +kubectl get pods -n cpaas-system | grep captain +kubectl get helmrequest -A +kubectl logs -n cpaas-system deploy/captain-controller-manager --tail=500 \ + | grep -E 'Unauthorized|FailedSync|Observed a panic' | sort | uniq -c | sort -rn | head +``` + +```text +native updates slow, many HelmRequests Unauthorized, requeuing, abnormal chart apps -> Cause 1 +失败 status, containers Running, several people updating -> Cause 2 +pipeline application 处理中, spec.assemblyPhase Pending -> Cause 3 +失败(0/1), get chart FailedSync Unauthorized, repository 同步中 -> Cause 4 +Helm update stays Failed after a permission error, admin retry fails -> Cause 5 +Helm delete stuck, workloads gone, error syncing Unauthorized -> Cause 6 +captain crash-loop, chartrepo panic in SortEntries -> Cause 7 +``` diff --git a/docs/en/solutions/Cluster_Overview_page_shows_no_monitoring_charts_after_Cluster_Enhancer_is_installed_on_ACP.md b/docs/en/solutions/Cluster_Overview_page_shows_no_monitoring_charts_after_Cluster_Enhancer_is_installed_on_ACP.md new file mode 100644 index 000000000..a80315278 --- /dev/null +++ b/docs/en/solutions/Cluster_Overview_page_shows_no_monitoring_charts_after_Cluster_Enhancer_is_installed_on_ACP.md @@ -0,0 +1,62 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Cluster Overview page shows no monitoring charts after Cluster Enhancer is installed on ACP + +## Issue + +Under **Cluster Management** > **Clusters**, clicking a cluster opens its **Overview** (`概览`) page, and there are no monitoring charts below the cluster information. The chart area is missing entirely; this is not a set of charts that keep loading or report an error. The question that usually follows is whether the charts only appear once the Cluster Enhancer plugin is installed. + +This article applies when the **Alauda Container Platform Cluster Enhancer** plugin is installed on that cluster. In **App Marketplace Management** > **Cluster Plugins** the plugin is listed as installed, with a description saying that it enhances the cluster by backing up etcd data, monitoring certificates, and cleaning up events. + +If Cluster Enhancer is **not** installed and the cluster or host dashboards are missing from **Operations Center** > **Monitoring**, see Cause 3 of [Monitoring page freezes the browser or built-in dashboards are missing on ACP](Monitoring_page_freezes_the_browser_or_built_in_dashboards_are_missing_on_ACP.md). Charts that are present but spin or stay empty are covered in [Monitoring pages return 502 or context deadline exceeded because port 11780 on the cluster VIP is blocked on ACP](Monitoring_pages_return_502_or_context_deadline_exceeded_because_port_11780_on_the_cluster_VIP_is_blocked_on_ACP.md). + +## Root Cause + +### Cause 1: the Overview monitoring panels are removed when Cluster Enhancer is installed + +This is expected behavior, not a collection or query fault. Once Cluster Enhancer is installed on a cluster, the monitoring panels on that cluster's **Overview** page are removed. Cluster, node, container, and Pod monitoring for that cluster is shown instead in **Operations Center** > **Monitoring** > **Monitoring Dashboards**, in the built-in dashboards of the `kubernetes` group. + +## Resolution + +### Cause 1: the Overview monitoring panels are removed when Cluster Enhancer is installed + +There is nothing to repair. Use the built-in dashboards instead: + +1. Open **Operations Center** > **Monitoring** > **Monitoring Dashboards**. +2. Select the cluster and open the `kubernetes` group. +3. Use its built-in dashboards: + + | Dashboard | Name in the Chinese UI | + | --- | --- | + | Cluster Status | `集群状态` | + | Container Monitoring | `容器监控` | + | Node Monitoring | `节点监控` | + | Pod Monitoring | `容器组监控` | + +Because installing Cluster Enhancer is what removes the Overview panels, installing or reinstalling it does not bring them back. There is no setting that restores monitoring panels on the Overview page. + +If those `kubernetes` dashboards are themselves empty, treat it as a separate fault: see [Monitoring dashboard panels are blank or drop to zero while the backend has data on ACP](Monitoring_dashboard_panels_are_blank_or_drop_to_zero_while_the_backend_has_data_on_ACP.md) when the API returns data, or the port 11780 article above when the browser developer tools show HTTP 500, 502, or `context deadline exceeded`. + +## Diagnostic Steps + +1. Confirm which page is empty. The symptom here is the **Overview** page under **Cluster Management** > **Clusters**, with no chart area at all. + +2. On that cluster, list the dashboards that Cluster Enhancer synchronizes: + + ```bash + kubectl get monitordashboards.ait.alauda.io -n cpaas-system -l helm.sh/chart-name=cluster-enhancer + ``` + + Expected for this article: objects are listed, so the plugin is installed and its dashboards are present. If nothing is listed, the plugin is not installed or has not synchronized; use the "built-in dashboards are missing" article linked in the Issue section. + +3. Open the `kubernetes` group under **Operations Center** > **Monitoring** > **Monitoring Dashboards** for the same cluster. Expected: **Cluster Status** and **Node Monitoring** show data, which confirms that monitoring works and only the Overview panels were removed. To see the dashboard objects behind the group: + + ```bash + kubectl get monitordashboards.ait.alauda.io -A | grep -i kubernetes + ``` diff --git a/docs/en/solutions/Cluster_plugins_stay_Deploying_or_AppReleases_stall_because_sentry_is_OOM_hung_or_CPU_bound_on_ACP.md b/docs/en/solutions/Cluster_plugins_stay_Deploying_or_AppReleases_stall_because_sentry_is_OOM_hung_or_CPU_bound_on_ACP.md new file mode 100644 index 000000000..5b152e97e --- /dev/null +++ b/docs/en/solutions/Cluster_plugins_stay_Deploying_or_AppReleases_stall_because_sentry_is_OOM_hung_or_CPU_bound_on_ACP.md @@ -0,0 +1,187 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.16" + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Cluster plugins stay Deploying or AppReleases stall because sentry is OOM, hung, or CPU-bound on ACP + +## Issue + +`sentry` is the component in `cpaas-system` of each cluster that installs, upgrades, and uninstalls plugin charts; it records the results in AppReleases (`ars`) and the chart sync records (`rels`), and applies ResourcePatches (`rpch`). When it is unhealthy, the console is up but: + +- CoreDNS, log collection, Prometheus, and other cluster plugins stay **Deploying**. +- A `sentry` Pod sits near 100% of its CPU limit for days, and a critical alert fires on `sentry` CPU usage. +- Pods fail to deploy across the cluster, and several AppReleases are `Failed`, `WaitReady`, or empty, while `sentry` Pods are `Running`. +- A ResourcePatch stays `pending` while `sentry` reports OOM. +- The `base` or `region` AppRelease of a workload cluster stays syncing, and the BroadcastJob `update-containerd-config` succeeded count rises and falls back (for example 50, 51, 52, then 50 again). + +Related article: [Platform health page shows components abnormal while their Pods are Running on ACP](Platform_health_page_shows_components_abnormal_while_their_Pods_are_Running_on_ACP.md). + +## Root Cause + +### Cause 1: sentry is OOM-killed, so AppRelease status does not sync + +`sentry` is not running its sync because it is OOM-killed, so plugin rows stay **Deploying** and AppRelease status does not sync. A higher memory limit lets it complete. + +### Cause 2: a large number of failed rels keeps sentry CPU high + +One `sentry` replica stays at about its CPU limit while the other is near 0, and the log keeps processing AppReleases with repeated `warn="set cocurrentProcessors failed, process later"` and `info="beginning wait for 45 resources with timeout of 30s"`, although `kubectl get ars -A -w` shows every AppRelease `Synced`/`Ready` and rarely updating. A CPU profile shows most of the time in Go garbage collection, with `(*AppReleaseReconciler).Process` behind it. The cluster holds a large number of failed `rels` (chart sync records), for example left from a long abnormal period while the cluster was being imported. Raising the CPU or memory limit, or moving the Pod, does not help; deleting the failed `rels` does. + +### Cause 3: sentry is hung while its Pods report Running + +Pods fail across the cluster, for example the kubelet journal repeats `Error syncing pod, skipping ... CrashLoopBackOff` for platform Pods, and `kubectl get ars -A` shows some entries `Failed` (for example `cni-kube-ovn`), `WaitReady` (`logcenter`), or with no SYNC, HEALTH, or MESSAGE value (`monitor`). `sentry` itself shows `Synced`/`Ready`, and all `sentry-*` and `sentry-service-*` Pods are `1/1 Running` with 0 restarts, but `sentry` is hung and does not synchronize. + +### Cause 4: a ResourcePatch stays pending because sentry is OOM + +The `sentry` Pods watch and enforce ResourcePatch (`rpch`) objects. While they are OOM-killed, nothing processes a new `rpch`, so it stays `pending` when a resource is changed (seen on 3.12.3). + +### Cause 5: a BroadcastJob hook cannot finish within the AppRelease timeout + +After all nodes were restarted, the `region` AppRelease on a workload cluster is upgraded again. Its chart `alauda-cluster-base` has the BroadcastJob `update-containerd-config` as a `helm.sh/hook: post-upgrade` hook with `parallelism: 1`, so it processes one node at a time. On a cluster with many nodes (for example 68), it cannot finish within the AppRelease `timeout` (240 seconds), so the upgrade is retried, the BroadcastJob is recreated, and its succeeded count resets. Deleting the BroadcastJob makes it reappear, and editing the AppRelease `timeout` is reverted automatically. On the `global` cluster the `minfo` of that cluster shows the condition `CheckStatus-region` with `status: Processing`, `reason: Syncing`, and `message: '[1/2] chart ait/chart-alauda-cluster-base- Upgrading'`. This was seen on 3.16. The same mechanism with the `upgrade-sysctl-config` BroadcastJob of the `base` AppRelease is described in [base AppRelease stays failed because the upgrade-sysctl-config BroadcastJob cannot finish on a large cluster on ACP](base_AppRelease_stays_failed_because_the_upgrade-sysctl-config_BroadcastJob_cannot_finish_on_a_large_cluster_on_ACP.md). + +## Resolution + +### Cause 1: sentry is OOM-killed, so AppRelease status does not sync + +1. Confirm the OOM: + + ```bash + kubectl get ars -A + kubectl get pods -n cpaas-system | grep sentry + kubectl describe pod -n cpaas-system | grep -E 'OOMKilled|Last State' + ``` + +2. Export the `sentry` Deployment and raise its memory limit above the usage that was OOM-killed. + + > **Warning:** A higher limit lets `sentry` use more node memory. Do this only when the Pod's last state is `OOMKilled`. To roll back, apply the export. + + ```bash + kubectl get deploy -n cpaas-system sentry -o yaml > "$HOME/sentry-deploy.yaml" + kubectl set resources deploy -n cpaas-system sentry --limits=memory= + ``` + +3. Plugin rows leave **Deploying** once `sentry` stays `Running` and AppRelease status syncs. + +### Cause 2: a large number of failed rels keeps sentry CPU high + +1. Confirm the signature: + + ```bash + kubectl top pod -n cpaas-system | grep sentry + kubectl logs -n cpaas-system --tail=300 | grep -E 'cocurrentProcessors|beginning wait for' + kubectl get rels -A | wc -l + kubectl get rels -A | grep -i fail | head + ``` + + If `rels` is not a known resource type on the cluster, find the kind of the chart sync records with `kubectl api-resources` first. + +2. Back up and delete the failed `rels`. + + > **Warning:** Deleted records cannot be restored; `sentry` creates new ones as it reconciles. Delete only objects that are already failed, and leave successful ones in place. + + ```bash + kubectl get rels -A -o yaml > "$HOME/rels-backup.yaml" + kubectl delete rels -n + ``` + +3. `sentry` CPU returns to normal once the failed records are gone. + +### Cause 3: sentry is hung while its Pods report Running + +1. Confirm the signature: + + ```bash + journalctl -u kubelet --since "30 min ago" | grep -E 'Error syncing pod|CrashLoopBackOff' | tail + kubectl get ars -A + kubectl get pods -n cpaas-system | grep sentry + ``` + +2. Recreate all `sentry` and `sentry-service` Pods. + + > **Warning:** AppRelease reconciliation pauses until the new Pods are `Ready`. Do this only when several AppReleases are `Failed`, `WaitReady`, or empty, Pods fail across the cluster, and the `sentry` Pods are `Running` with no restarts. The Pods are recreated from the same specs. + + ```bash + kubectl delete pod -n cpaas-system + kubectl get ars -A + ``` + +3. Every AppRelease returns to `Synced`/`Ready`/`chart synced` with a recent UPDATE time. If one stays `Failed`, read its message instead of restarting `sentry` again. + +### Cause 4: a ResourcePatch stays pending because sentry is OOM + +1. Confirm the OOM, find the pending object, and save the current state: + + ```bash + kubectl get deploy -A | grep -E 'sentry|cluster-transformer' + kubectl describe pod -n cpaas-system | grep -E 'OOMKilled|Last State' + kubectl get rpch -A | grep -i pending + kubectl get rpch -o yaml > "$HOME/-rpch.yaml" + kubectl get deploy -n cpaas-system sentry sentry-service cluster-transformer -o yaml > "$HOME/sentry-deploys.yaml" + ``` + +2. Delete the pending `rpch`, stop the components, raise the `sentry` limit, and start them again with their previous replica counts. + + > **Warning:** Deleting the `rpch` drops the patch it would have applied. With the three Deployments at 0 replicas, plugin and AppRelease reconciliation stops. Do this only when `sentry` is OOM-killed and an `rpch` is stuck `pending`, in a maintenance window. To roll back, apply the Deployment export and, if the patch is still wanted, the `rpch` export once `sentry` is healthy. + + ```bash + kubectl delete rpch + kubectl scale deploy -n cpaas-system sentry sentry-service cluster-transformer --replicas=0 + kubectl set resources deploy -n cpaas-system sentry --limits=memory= + kubectl scale deploy -n cpaas-system sentry --replicas= + kubectl scale deploy -n cpaas-system sentry-service --replicas= + kubectl scale deploy -n cpaas-system cluster-transformer --replicas= + ``` + +3. `sentry` stays `Running`, and a new resource change does not leave its `rpch` pending. + +### Cause 5: a BroadcastJob hook cannot finish within the AppRelease timeout + +1. Confirm the loop: + + ```bash + kubectl get broadcastjobs.operator.alauda.io -A + kubectl describe broadcastjobs.operator.alauda.io -n cpaas-system update-containerd-config | sed -n '/Events/,$p' + kubectl get broadcastjobs.operator.alauda.io -n cpaas-system update-containerd-config -o yaml | grep -n -E 'parallelism|helm.sh/hook|ttlSecondsAfterFinished|restartLimit' + kubectl get ars -n cpaas-system region -o yaml | grep -n timeout + ``` + + Expected: SUCCEEDED cycles below DESIRED, and the event shows thousands of `SuccessfulCreate` for `update-containerd-config-` Pods. + +2. Raise the parallelism so that the hook finishes within the timeout. + + > **Warning:** This BroadcastJob updates the containerd configuration on every node it runs on; with `parallelism: 10` up to ten nodes are processed at the same time. Do this only on the signature above. To roll back, set `parallelism` back to `1`; work already done on nodes is not undone. + + ```bash + kubectl patch broadcastjobs.operator.alauda.io -n cpaas-system update-containerd-config --type=merge -p '{"spec":{"parallelism":10}}' + kubectl get broadcastjobs.operator.alauda.io -n cpaas-system update-containerd-config -w + ``` + +3. SUCCEEDED reaches DESIRED, and the `region` AppRelease and the `CheckStatus-region` condition return to normal. + +## Diagnostic Steps + +```bash +kubectl get pods -n cpaas-system | grep sentry +kubectl get ars -A | grep -viE 'Ready|Succeeded|Running' || true +kubectl get broadcastjobs.operator.alauda.io -A +kubectl get rpch -A +``` + +```text +plugins stuck Deploying, sentry OOMKilled -> Cause 1 +sentry CPU high for days, cocurrentProcessors warnings, many failed rels -> Cause 2 +Pods fail cluster-wide, ars Failed or WaitReady, sentry Running with 0 restarts -> Cause 3 +rpch pending after a resource change, sentry OOMKilled -> Cause 4 +update-containerd-config SUCCEEDED cycles, CheckStatus-region Processing -> Cause 5 +``` diff --git a/docs/en/solutions/Console_is_inaccessible_or_a_business_cluster_shows_0_nodes_because_global_cannot_reach_an_API_server_on_ACP.md b/docs/en/solutions/Console_is_inaccessible_or_a_business_cluster_shows_0_nodes_because_global_cannot_reach_an_API_server_on_ACP.md new file mode 100644 index 000000000..6f306c3d9 --- /dev/null +++ b/docs/en/solutions/Console_is_inaccessible_or_a_business_cluster_shows_0_nodes_because_global_cannot_reach_an_API_server_on_ACP.md @@ -0,0 +1,146 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Console is inaccessible or a business cluster shows 0 nodes because global cannot reach an API server on ACP + +## Issue + +The console worked before, and now one of the following happens: + +- A business cluster row shows 0 nodes, 0 containers, and 0 storage (for example after a `global` migration), while `kubectl get nodes` inside the business cluster shows `Ready`. +- The page says the system cannot be accessed (`系统无法访问`), and platform component logs time out against the in-cluster API Service. +- The console is slow, one `apollo` replica is unhealthy, and renewing credentials or restarting `apollo` does not help. +- Login fails while a burst of alerts fires and the control-plane nodes run at 100% CPU. + +Related article: [Console returns 502 or does not open after installation or a VIP change on ACP](Console_returns_502_or_does_not_open_after_installation_or_a_VIP_change_on_ACP.md). If only monitoring pages time out on port 11780, see [Monitoring pages return 502 or context deadline exceeded because port 11780 on the cluster VIP is blocked on ACP](Monitoring_pages_return_502_or_context_deadline_exceeded_because_port_11780_on_the_cluster_VIP_is_blocked_on_ACP.md). + +## Root Cause + +### Cause 1: the global cluster cannot open the business API server on 6443 + +The cluster's `CheckStatus` is `Failed` with a message such as `Get "https://:6443/api/v1/nodes?labelSelector=kubernetes.io%2Fos%3Dlinux&timeout=2m0s": context deadline exceeded`. From the `global` side, TCP to the business VIP on 6443 fails or times out, for example because a switch no longer forwards from `global` to that VIP. The business nodes and containers are not at fault. If the path was restored but the cluster sync controllers were not restarted, the 0 counts can remain. + +A failed `global` control-plane node looks similar from the browser: the whole platform is down and `apollo` cannot list API groups for the cluster `global`, because the `global` API endpoint does not answer. + +### Cause 2: one control-plane host is abnormal and components lose their API watches + +Abnormal platform components repeat client-go errors such as: + +```text +watch of *v1.Pod ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding +"Objects listed" error:Get "https://:443/api/v1/pods?resourceVersion=...": dial tcp :443: i/o timeout +``` + +Restarting the API server does not help. One master host is abnormal, and restarting that host restores the platform. The exact host fault was not identified. + +### Cause 3: a global node is NotReady + +The UI loads slowly and one `apollo` replica is unhealthy. Regenerating credentials and restarting that Pod do not help, because the node it runs on is `NotReady` or broken (for example after another container runtime was installed on it). Repairing the node restores platform access. + +### Cause 4: control-plane nodes run at 100% CPU + +The console cannot be logged into, a large set of alerts fires, and two or more control-plane nodes are at 100% CPU, shared between host agent processes and containers. A batch restart of the Pods that burn CPU brings it down, and restarting `apollo` then restores login. + +## Resolution + +### Cause 1: the global cluster cannot open the business API server on 6443 + +1. From a `global` node: + + ```bash + curl -k --max-time 5 https://:6443/readyz + kubectl get cls -A + kubectl describe cls + ``` + + `readyz` must succeed before any controller restart helps. Fix the network path, the VIP, or the failed control-plane node first. + +2. Once 6443 answers and the console still shows 0 nodes, restart the `global` cluster components whose logs show the `context deadline exceeded` timeouts to that business cluster (the cluster synchronization components). Restarting `cluster-api-provider-alauda-manager` alone may not be enough. + + > **Warning:** Restarting these controllers pauses cluster synchronization; it does not delete clusters. Do not restart them while 6443 is still closed; the new Pods only repeat the timeout. + + ```bash + for p in $(kubectl get pods -n cpaas-system -o name); do kubectl logs -n cpaas-system "$p" --all-containers --tail=200 2>/dev/null | grep -q ":6443.*context deadline exceeded" && echo "$p"; done + kubectl delete pod -n cpaas-system + ``` + +### Cause 2: one control-plane host is abnormal and components lose their API watches + +1. Find the master that is not healthy: + + ```bash + kubectl get po -A | grep -vi run + kubectl get nodes -o wide + kubectl get pods -n kube-system -l component=kube-apiserver -o wide + kubectl get pods -n kube-system -l component=etcd -o wide + ``` + + On each master: + + ```bash + uptime + dmesg -T | tail -50 + crictl ps | grep -E 'kube-apiserver|etcd' + ``` + +2. Restart the abnormal master host. + + > **Warning:** The API server and etcd member on that host stop until it boots. Do this only when the component logs show `http2: client connection lost` or `dial tcp :443: i/o timeout` and this master is the abnormal one. Confirm the other masters are healthy so that etcd keeps quorum, and keep console access to the host. A reboot cannot be rolled back. + +3. Platform pages load once the node is `Ready` and the reflector errors stop. + +### Cause 3: a global node is NotReady + +```bash +kubectl get nodes +kubectl describe node | sed -n '/Conditions/,$p' +``` + +Repair or remove the bad node. Do not keep rotating credentials once the certificate is known to be valid and only the `apollo` replica on that node is stuck. Console latency drops when every `global` node used by platform Pods is `Ready`. + +### Cause 4: control-plane nodes run at 100% CPU + +1. Confirm the saturation: + + ```bash + kubectl top nodes + ps aux --sort=-%cpu | head + ``` + +2. Restart the Pods that burn CPU on the saturated nodes, then `apollo` if login still fails. + + > **Warning:** Restarting Pods interrupts the workloads they serve, including platform components. Do this only when the masters are at 100% CPU and the login failure matches that window, preferably in a maintenance window. The Pods return from the same specs. + + ```bash + kubectl get pods -A --field-selector spec.nodeName= --no-headers + kubectl delete pod -n + kubectl delete pod -n cpaas-system + ``` + +3. Login succeeds once CPU is no longer pinned and `apollo` is `Running`. + +## Diagnostic Steps + +```bash +curl -k --max-time 5 https://:6443/readyz +kubectl get nodes +kubectl top nodes +kubectl get po -A | grep -vi run +``` + +```text +business kubectl works, global curl to :6443 fails, CheckStatus Failed -> Cause 1 +http2: client connection lost, dial tcp :443 i/o timeout, one master abnormal -> Cause 2 +slow UI, one global node NotReady, credentials already valid -> Cause 3 +login fails, master CPU at 100% -> Cause 4 +``` diff --git a/docs/en/solutions/Console_is_slow_or_shows_data_loading_errors_because_calls_through_erebus_or_the_marketplace_backends_are_slow_on_ACP.md b/docs/en/solutions/Console_is_slow_or_shows_data_loading_errors_because_calls_through_erebus_or_the_marketplace_backends_are_slow_on_ACP.md new file mode 100644 index 000000000..26e4fe8bb --- /dev/null +++ b/docs/en/solutions/Console_is_slow_or_shows_data_loading_errors_because_calls_through_erebus_or_the_marketplace_backends_are_slow_on_ACP.md @@ -0,0 +1,189 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Console is slow or shows data loading errors because calls through erebus or the marketplace backends are slow on ACP + +## Issue + +The platform is slow and data on the pages takes more than a minute to appear, or a data loading error keeps appearing and clears when the user clicks **Reload**. Related reports: + +- OperatorHub, the cluster plugin page (`集群插件`), and Tekton pages all fail to open at the same time. +- Every page is slow, with occasional no-permission errors and 502. +- The left navigation menu takes a long time to load. +- In the browser developer tools the `envs` API or other calls that go through `erebus` time out. + +`erebus` is the console backend in `cpaas-system` on the `global` cluster that proxies calls toward a target cluster, so a slow path through it makes many pages slow at once. + +Start by timing the slow request from a host inside the platform network, so that a slow client path, such as a VPN, is ruled out before any platform change. + +## Root Cause + +### Cause 1: the user's network path is slow while the platform is healthy + +Every page takes more than a minute for the user, but the same requests answer quickly from inside the platform network. The delay is on the client path, for example the VPN the user goes through to reach the platform, and no platform component is at fault. + +### Cause 2: the envs API to erebus times out and the erebus log has no error + +Browser tools show the `envs` API to `erebus` timing out, and the `erebus` Pod log on the `global` cluster has no clear exception. Restarting the `erebus` Pod restores the connection; no more specific fault was identified. + +### Cause 3: a data loading error clears on reload and erebus logs context canceled + +`erebus` logs `context canceled`, and the path from the node that runs `erebus` to the platform VIP answers normally. The failed call is the client connection dropping; after the client connects again, loading succeeds. No change is needed. + +### Cause 4: a Layer-4 load balancer with an X-Forwarded-For profile on port 6443 delays calls through erebus + +Any page can be slow, intermittently but often; some show a no-permission error and some requests return 502 from `erebus`. No container is abnormal, and `kubectl` on a workload-cluster master answers quickly while the UI is slow. A normal request takes seconds instead of milliseconds during problem periods. + +The path client > global VIP > `erebus` > workload-cluster VIP > API server reproduces the delay; the path client > workload-cluster VIP > API server does not, nor does a direct `curl` from the global nodes to the workload API server. Captures on the global nodes show many TCP retransmissions toward the workload-cluster VIP on 6443, and the load balancer capture shows retransmissions, port-reuse warnings, and duplicate ACKs ending in an RST from the client side. The F5 virtual server for API server port 6443 is a Performance (Layer 4) server with the `fastL4` profile, and its HTTP Client profile is set to `X-Forwarded-For`. After XFF was disabled on that virtual server, the problem no longer reproduced; the exact mechanism is not confirmed. Host and VM NIC queue tuning improves stability somewhat but does not remove the delay. + +### Cause 5: a decommissioned cluster is still registered + +The left navigation menu loads very slowly. A cluster that was decommissioned is still registered on the platform, and platform components keep trying to reach its VIP. + +### Cause 6: marketplace-apiserver is killed before its cache finishes loading + +OperatorHub, the cluster plugin page, and Tekton and other product pages fail at the same time, and `marketplace-apiserver` keeps restarting. On a cluster with a large number of CRDs or APIServices, `marketplace-apiserver` does not finish loading its cache before the liveness probe fails, so the Pod is restarted before it ever becomes healthy. Raising `livenessProbe.initialDelaySeconds` to `180` restores the service. + +## Resolution + +### Cause 1: the user's network path is slow while the platform is healthy + +1. In the browser developer tools (**Network**), right-click the slow request and choose **Copy as cURL**. +2. Run it from the user's machine and from a host inside the platform network (for example a `global` control-plane node or a jump host), with timing output appended: + + ```bash + curl -k -s -o /dev/null -w 'connect=%{time_connect}s first-byte=%{time_starttransfer}s total=%{time_total}s\n' https:/// + ``` + +3. A fast answer inside the network and tens of seconds from the user's machine places the delay on the client path. A long **Initial connection** or **Content download** phase in the browser **Timing** tab points the same way. Hand the path to the team that runs it; no platform change is needed. + +### Cause 2: the envs API to erebus times out and the erebus log has no error + +1. Read the `erebus` log before deleting the Pod: + + ```bash + kubectl get pods -n cpaas-system --no-headers | awk '/^erebus-/ {print}' + kubectl logs -n cpaas-system --tail=200 + ``` + +2. Restart `erebus`. + + > **Warning:** Console calls proxied by `erebus` fail until the new Pod is `Ready`. Do this only when the `envs` API to `erebus` times out and the log shows no separate fault such as clock skew, an expired platform credential, or a down business API server. The Pod is recreated from the same spec. + + ```bash + kubectl delete pod -n cpaas-system + ``` + +3. If the `envs` API still times out after the new Pod is `Running`, the fault is elsewhere. + +### Cause 3: a data loading error clears on reload and erebus logs context canceled + +Confirm the log and reload the page so that the client opens a new connection: + +```bash +kubectl logs -n cpaas-system --tail=200 | grep -F 'context canceled' +``` + +If the platform VIP does not answer from the node that runs `erebus`, the problem is reachability, not this cause. + +### Cause 4: a Layer-4 load balancer with an X-Forwarded-For profile on port 6443 delays calls through erebus + +1. From each `global` node, time the workload API server through its VIP, then reproduce the slow page and compare: + + ```bash + time curl -k -s -o /dev/null -w '%{http_code}\n' https://:6443/readyz + kubectl get pods -n cpaas-system -o wide | grep erebus + ``` + +2. On the `global` node that runs `erebus`, capture traffic toward the workload-cluster VIP while the problem reproduces: + + ```bash + tcpdump -i any -nn host and port 6443 -w /tmp/erebus-6443.pcap + ``` + + Retransmissions, port-reuse warnings, and duplicate ACKs that end in a client-side RST, only on the path through the load balancer, match this cause. + +3. On the load balancer, open the virtual server that publishes 6443 and disable the `X-Forwarded-For` HTTP Client profile. + + > **Warning:** Changing the virtual server profile affects every connection through it, not only ACP. Do this only when the virtual server is Layer 4 (`fastL4`) for API server port 6443, its HTTP Client profile is `X-Forwarded-For`, and the captures show the retransmissions above. Use a change window with the network team. To roll back, set the profile back. + +4. Watch whether the slow requests, no-permission errors, and 502 return. + +### Cause 5: a decommissioned cluster is still registered + +1. List the clusters the platform knows about and test each VIP from a `global` node: + + ```bash + kubectl get cls -A + curl -k --max-time 5 https://:6443/readyz + ``` + +2. Delete the decommissioned cluster from the platform cluster list. + + > **Warning:** Removing a cluster from the platform ends its management from the console, and it cannot be undone from the cluster list; to manage it again it must be added back. Do this only for a cluster that is decommissioned and whose VIP no longer answers. + +3. The left navigation loads normally once components stop retrying that VIP. + +### Cause 6: marketplace-apiserver is killed before its cache finishes loading + +1. Check whether the Pod is restarted by its liveness probe: + + ```bash + kubectl get deploy -A | grep marketplace-apiserver + kubectl describe pod -n | grep -E 'Restart Count|Last State|Liveness|Unhealthy|Killing' + kubectl logs -n --previous --tail=200 + kubectl get crd --no-headers | wc -l + kubectl get apiservice --no-headers | wc -l + kubectl get deploy -n marketplace-apiserver -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" "}{.livenessProbe}{"\n"}{end}' + ``` + + The signature is a growing restart count with `Liveness probe failed` events while the previous log shows the server still starting, on a cluster with many CRDs or APIServices. A Pod that crashes on its own error before the probe fires is not this cause. + +2. Export the Deployment and raise the initial delay of the liveness probe. + + > **Warning:** The patch rolls the Pod, so OperatorHub, cluster plugin, and Tekton pages stay unavailable until it is `Ready`. A longer initial delay also means a process that really hangs at startup is restarted later. To roll back, patch `initialDelaySeconds` back to the value in the export. This change can be reverted by a platform or plugin upgrade; check it again afterwards. + + ```bash + kubectl get deploy -n marketplace-apiserver -o yaml > /tmp/marketplace-apiserver.bak.yaml + kubectl patch deploy -n marketplace-apiserver --type=json -p '[{"op":"add","path":"/spec/template/spec/containers/0/livenessProbe/initialDelaySeconds","value":180}]' + kubectl rollout status deploy -n marketplace-apiserver + ``` + + The path uses container index `0`; if the `jsonpath` output lists the probe on another container, use that index. + +3. Once the Pod stays `Running` without liveness restarts, open OperatorHub, the cluster plugin page, and a Tekton page again. + +## Diagnostic Steps + +1. In the browser **Network** panel, note whether the failing call is the `envs` API or another call proxied by `erebus`, and whether it times out or returns 500. Time the same call from inside the platform network (Cause 1); if only the user's side is slow, stop there. + +2. Read the `erebus` and `apollo` logs on the `global` cluster: + + ```bash + kubectl get pods -n cpaas-system -o wide | grep -E 'erebus|apollo' + kubectl logs -n cpaas-system --since=30m --tail=200 + kubectl logs -n cpaas-system --since=30m --tail=200 | grep -i erebus + ``` + +3. Match the result: + + ```text + every page over a minute from the user's side, same call fast inside the network -> Cause 1 + envs API timeout, erebus log has no error -> Cause 2 + data loading error, reload works, erebus logs context canceled -> Cause 3 + all pages slow, 502 from erebus, retransmissions through the load balancer on 6443 -> Cause 4 + left menu slow, a decommissioned cluster still listed -> Cause 5 + OperatorHub, cluster plugins and Tekton fail together, marketplace-apiserver liveness restarts -> Cause 6 + ``` + +4. If the problem started right after a change on the hosts (DNS, NetworkManager, CNI, kernel parameters), see [Console returns 500 or data loading errors after a host network change on ACP](Console_returns_500_or_data_loading_errors_after_a_host_network_change_on_ACP.md). diff --git a/docs/en/solutions/Console_menu_bar_or_Data_Services_navigation_does_not_load_because_of_a_workload_cluster_ALB_or_a_credential_on_ACP.md b/docs/en/solutions/Console_menu_bar_or_Data_Services_navigation_does_not_load_because_of_a_workload_cluster_ALB_or_a_credential_on_ACP.md new file mode 100644 index 000000000..b80a1a3d2 --- /dev/null +++ b/docs/en/solutions/Console_menu_bar_or_Data_Services_navigation_does_not_load_because_of_a_workload_cluster_ALB_or_a_credential_on_ACP.md @@ -0,0 +1,143 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Console menu bar or Data Services navigation does not load because of a workload cluster ALB or a credential on ACP + +## Issue + +The console opens, but part of it does not load: + +- The left menu bar of platform management stays on a loading placeholder. In the browser developer tools (**Network**), only the `global-info` request stays pending. +- After middleware operators were installed, the **Data Services** left navigation shows no middleware, and the `console-plugins` response contains only `global` data. +- Platform pages freeze and only part of the interface loads. + +Related article: [Console menu or feature is missing because a feature gate, plugin, or product entry is off on ACP](Console_menu_or_feature_is_missing_because_a_feature_gate_plugin_or_product_entry_is_off_on_ACP.md). + +## Root Cause + +### Cause 1: the menu bar waits on a workload cluster whose ALB failed because a node disk is full + +`marketplace-api` on the `global` cluster collects console plugins from every cluster through that cluster's ALB, and repeats for the affected workload clusters: + +```text +consoleplugins.go:64] collect console-plugins in cluster failed. err: invalid status code: 502. response status: 502 Bad Gateway, response body: X-Error: 502 +``` + +The global `apollo` log shows cluster cache errors such as `List log-agent feature error(... Client.Timeout exceeded while awaiting headers)` and `Get monitoring feature error(... context deadline exceeded)`. On the affected workload cluster, the `cpaas-system-*` ALB Pods are `0/2` in `Error` or `Pending`, because node storage ran out. The frontend Pods there are `Running`, and restarting `apollo` and `marketplace-api` does not help. + +### Cause 2: the workload cluster ALB runs in container mode and its LoadBalancer Service stays pending + +The same `collect console-plugins ... 502` line appears, but the ALB Pods on the workload cluster are `Running`. `kubectl get alb2 -A` shows `cpaas-system` with STATE `Progressing`, REASON `wait lb svc ready`, MODE `container`, and the `LoadBalancer` Service `cpaas-system-lb-` (ports `11782` and `11780`) has EXTERNAL-IP ``. This was seen on 4.2.1 on a workload cluster deployed with the cluster-api-provider-hcs procedure, which has no ALB option: the ALB chart set container network mode, which gets no external address there. + +### Cause 3: an expired GitLab model repository token freezes the pages + +Platform pages freeze and only part of the UI loads when the token for the GitLab model repository has expired. The token is stored in the Secret `aml-image-builder-secret` in the namespace `kubeflow-admin-cpaas-io`. + +## Resolution + +### Cause 1: the menu bar waits on a workload cluster whose ALB failed because a node disk is full + +1. Find the workload cluster and its ALB state: + + ```bash + # global cluster + kubectl logs -n cpaas-system deploy/marketplace-api --tail=200 | grep 'collect console-plugins' + # affected workload cluster + kubectl get po -n cpaas-system -o wide | grep cpaas-system- + kubectl describe pod -n cpaas-system | sed -n '/Events/,$p' + ``` + +2. On the node where the ALB Pods are `Error` or `Pending`, check the disk and free space: + + ```bash + df -h + ``` + + > **Warning:** Deleting files is irreversible; copy anything you still need first, and leave live etcd data and container runtime storage in place. Do this only when `global-info` stays pending, `marketplace-api` logs 502 toward that cluster, and the node filesystem is full. + +3. Once the ALB Pods are `2/2 Running`, reload the console. The menu bar loads when `global-info` completes and `marketplace-api` stops logging 502 for that cluster. + +### Cause 2: the workload cluster ALB runs in container mode and its LoadBalancer Service stays pending + +1. On the workload cluster: + + ```bash + kubectl get alb2 -A + kubectl get svc -n cpaas-system | grep LoadBalancer + kubectl get nodes -l cpaas-system-alb + ``` + +2. Switch the ALB to host network mode. + + > **Warning:** In host network mode the ALB Pods run on the host network of the nodes labelled `cpaas-system-alb`, and their ports must be free there. Traffic through this ALB is interrupted while the Pods roll. Do this only when the ALB2 is `Progressing` with `wait lb svc ready`, mode `container`, and the LoadBalancer Service has no external IP. Export the object first; applying the export is the rollback. + + ```bash + kubectl get alb2 -n cpaas-system cpaas-system -o yaml > "$HOME/alb2-cpaas-system.yaml" + kubectl explain alb2.spec --recursive | grep -iE 'networkMode|nodeSelector' + kubectl edit alb2 -n cpaas-system cpaas-system + ``` + + Set `networkMode: host` and the node selector `cpaas-system-alb: ""` at the paths that `kubectl explain` shows. + +3. Make it permanent: on the `global` cluster, add an override to the workload cluster's ALB `minfo`: + + ```bash + kubectl get minfo | grep -i alb | grep + kubectl edit minfo + ``` + + ```yaml + spec: + valuesOverride: + _global: + global: + cluster: + type: DCS + ``` + +4. The ALB2 reaches a ready state, `marketplace-api` stops logging 502 for that cluster, and **Data Services** lists the middleware. + +### Cause 3: an expired GitLab model repository token freezes the pages + +1. Back up the Secret: + + ```bash + kubectl get secret -n kubeflow-admin-cpaas-io aml-image-builder-secret -o yaml > "$HOME/aml-image-builder-secret.yaml" + chmod 600 "$HOME/aml-image-builder-secret.yaml" + ``` + +2. Replace the token value with a current GitLab model repository token. Change only the key that stores the token. + + > **Warning:** The Secret holds a repository credential; keep the backup private. To roll back, apply the backup. + + ```bash + kubectl edit secret -n kubeflow-admin-cpaas-io aml-image-builder-secret + ``` + +3. Reload the platform pages. + +## Diagnostic Steps + +1. In the browser **Network** panel, check whether `global-info` stays pending (Cause 1) or the `console-plugins` response holds only `global` (Cause 2). + +2. On the `global` cluster, read `marketplace-api`. Expected for Causes 1 and 2: `collect console-plugins in cluster failed ... 502`. + +3. On that workload cluster: + + ```text + cpaas-system-* ALB Pods 0/2 Error or Pending, node disk full -> Cause 1 + alb2 Progressing, wait lb svc ready, MODE container, LB Service EXTERNAL-IP -> Cause 2 + ``` + +4. Pages freeze with an incomplete UI and the model repository token has expired: Cause 3. diff --git a/docs/en/solutions/Console_menu_or_feature_is_missing_because_a_feature_gate_plugin_or_product_entry_is_off_on_ACP.md b/docs/en/solutions/Console_menu_or_feature_is_missing_because_a_feature_gate_plugin_or_product_entry_is_off_on_ACP.md new file mode 100644 index 000000000..9126e2804 --- /dev/null +++ b/docs/en/solutions/Console_menu_or_feature_is_missing_because_a_feature_gate_plugin_or_product_entry_is_off_on_ACP.md @@ -0,0 +1,134 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Console menu or feature is missing because a feature gate, plugin, or product entry is off on ACP + +## Issue + +The console opens, but a menu, a feature, or a product entry is missing or shows an error, typically after an upgrade: + +- Namespace-scoped CRD management, available on 3.18, is not visible on 4.x. +- The namespace audit switch is greyed out. +- The UI cannot be switched to Chinese after an upgrade to 4.2.4. +- Load balancers are no longer listed in platform management after an upgrade. +- The data services entry is missing from the portal after an upgrade to 4.0.10. +- Clicking a cluster name in **Alerts** > **Real-time Alerts** shows `缺少翻译文件:插件console-core-service缺少zh语言翻译文件:plugin_console-core-service.json` (missing translation file: plugin console-core-service lacks the zh translation file). +- The **EXEC** entry is missing from Pod and application pages; see [EXEC and Debug entries missing from Pod and application pages until the cpc-exec feature gate is enabled on ACP](EXEC_and_Debug_entries_missing_from_Pod_and_application_pages_until_the_cpc-exec_feature_gate_is_enabled_on_ACP.md). + +Feature gates are managed at `/console-platform/feature-gate`, and are `AlaudaFeatureGate` objects (`alaudafeaturegates.alauda.io`) in `cpaas-system`. + +Related article: [Console menu bar or Data Services navigation does not load because of a workload cluster ALB or a credential on ACP](Console_menu_bar_or_Data_Services_navigation_does_not_load_because_of_a_workload_cluster_ALB_or_a_credential_on_ACP.md). + +## Root Cause + +### Cause 1: namespace-scoped CRD management is behind the namespace-resource-manage gate + +The feature was not removed in 4.x. It is off by default behind the feature gate `namespace-resource-manage`, which a search for an obvious name on the feature gate page does not find. + +### Cause 2: the namespace audit switch is greyed out because namespaceaudits is disabled + +The availability of the namespace audit switch is controlled by the `namespaceaudits` entry of a feature gate resource, not by the switch itself. While that entry is disabled, the switch stays grey. + +### Cause 3: the Chinese language pack plugin is not deployed + +After the upgrade to 4.2.4, Chinese cannot be selected until the Chinese language pack plugin is deployed. + +### Cause 4: the alb2 feature gate is off + +After an upgrade, load balancers are missing from platform management because the `alb2` `AlaudaFeatureGate` in `cpaas-system` is not enabled. + +### Cause 5: the data services ProductEntry has packType AdminPage + +`kubectl get ProductEntry console-dataservices -o yaml` shows an `alauda.io/v1alpha1` `ProductEntry` (owned by the Helm release `application-services-core` in `cpaas-system`) with `spec.entrypoint: /console-dataservices` and `spec.packType: AdminPage`. With `packType: AdminPage`, the entry is not shown on the portal. + +### Cause 6: the new web console plugin has no Chinese translation file + +In the browser developer tools on the real-time alert page, `GET /console-i18n/zh/plugin__console-core-service.json` returns `404 Not Found` with header `X-Alb-Err-Reason: BackendError ...`, and other plugin translation requests such as `plugin__web-cli-service` can fail too. The new web console plugins (`console-core`) insert jump links into the existing console pages but do not ship the matching Chinese translation file. The Chinese language pack is upgraded separately, and its latest version contains the file. + +## Resolution + +### Cause 1: namespace-scoped CRD management is behind the namespace-resource-manage gate + +Open `/console-platform/feature-gate`, search for `namespace-resource-manage`, and enable it. Namespace-scoped CRD management is shown again. + +### Cause 2: the namespace audit switch is greyed out because namespaceaudits is disabled + +1. Find the feature gate resource kind: + + ```bash + kubectl api-resources | grep -i featuregate + ``` + +2. In **Resource Management**, open the feature gate object that contains `namespaceaudits` and set its `enabled` field to `true`. + + > **Warning:** This turns on namespace audit for the platform. Do it only when the switch is greyed out and this entry is disabled. To roll back, set `enabled` back to `false`. + +### Cause 3: the Chinese language pack plugin is not deployed + +Deploy the Chinese language pack plugin from the cluster plugins page, then switch the UI language again. No feature gate change is needed. + +### Cause 4: the alb2 feature gate is off + +1. Read the gate: + + ```bash + kubectl get alaudafeaturegates -n cpaas-system alb2 -o yaml + ``` + +2. Enable it and set the annotation `cpaas.io/ignoreme` to `true`. + + > **Warning:** This enables load balancer management in the console. Apply it only when load balancers are missing after an upgrade and `spec.enabled` is not already `true`. To roll back, set `spec.enabled` to `false` and remove the annotation. + + ```bash + kubectl patch alaudafeaturegates alb2 -n cpaas-system --type=merge -p '{"spec":{"enabled":true},"metadata":{"annotations":{"cpaas.io/ignoreme":"true"}}}' + ``` + +3. Reload platform management. Load balancers are listed again. + +### Cause 5: the data services ProductEntry has packType AdminPage + +> **Warning:** Changing `packType` changes how the portal classifies the entry. Do it only when the entry is missing and `spec.packType` is `AdminPage`. Keep a copy first; to roll back, set `spec.packType` back to `AdminPage`. + +```bash +kubectl get ProductEntry console-dataservices -o jsonpath='{.spec.packType}{"\n"}' +kubectl get ProductEntry console-dataservices -o yaml > "$HOME/console-dataservices.yaml" +kubectl patch ProductEntry console-dataservices --type=merge -p '{"spec":{"packType":"Buildin"}}' +``` + +The portal shows the data services entry once `spec.packType` is `Buildin`. + +### Cause 6: the new web console plugin has no Chinese translation file + +1. Confirm the signature in the browser developer tools: the request for `/console-i18n/zh/plugin__console-core-service.json` returns 404. +2. Upgrade the Chinese language pack plugin to its latest version, which contains the translation file. +3. If the new web console is not used and the language pack cannot be upgraded yet, uninstalling the new web console plugins also removes the error. + + > **Warning:** Uninstalling those plugins removes the new web console jump links and any page that depends on them. Remove only the plugins added for the new web console, not other console plugins. To roll back, install them again. + +## Diagnostic Steps + +```bash +kubectl get alaudafeaturegates -n cpaas-system | grep -E 'alb2|exec|namespace' +kubectl get ProductEntry console-dataservices -o yaml +``` + +```text +CRD management missing on 4.x, namespace-resource-manage off -> Cause 1 +namespace audit switch greyed out -> Cause 2 +cannot switch to Chinese after 4.2.4 -> Cause 3 +load balancers missing after an upgrade, alb2 disabled -> Cause 4 +data services entry missing, packType AdminPage -> Cause 5 +plugin_console-core-service.json missing, /console-i18n/zh/... 404 -> Cause 6 +EXEC missing, cpc-exec disabled -> the cpc-exec article linked above +``` diff --git a/docs/en/solutions/Console_returns_500_or_data_loading_errors_after_a_host_network_change_on_ACP.md b/docs/en/solutions/Console_returns_500_or_data_loading_errors_after_a_host_network_change_on_ACP.md new file mode 100644 index 000000000..bac589b37 --- /dev/null +++ b/docs/en/solutions/Console_returns_500_or_data_loading_errors_after_a_host_network_change_on_ACP.md @@ -0,0 +1,141 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Console returns 500 or data loading errors after a host network change on ACP + +## Issue + +One of the following starts right after a change on the cluster hosts, not on ACP objects: + +- The management console returns `x-error 500` after node DNS was changed and NetworkManager was restarted. +- Switching between platform sub-pages sometimes returns a 500 internal server error, more often in an incognito window. +- The native application page keeps showing `数据加载错误` (data loading error), the external address pool page shows `数据加载错误 重试` (data loading error, retry), the cluster shows no subnet information, and later the whole platform is unusable. + +Platform Pods can all be `Running` while their network path is broken. The common check is to compare what works from the node with what works from inside a Pod on that node, and then to find the host change. + +If nothing changed on the hosts, and pages are slow or fail together, see [Console is slow or shows data loading errors because calls through erebus or the marketplace backends are slow on ACP](Console_is_slow_or_shows_data_loading_errors_because_calls_through_erebus_or_the_marketplace_backends_are_slow_on_ACP.md). + +## Root Cause + +### Cause 1: NetworkManager was restarted after a DNS change and Pods time out to the API Service + +Node DNS was changed with `nmcli conn mod "" ipv4.dns ""` and NetworkManager was restarted. Afterwards a platform component logs: + +```text +Error: Get "https://:443/version": dial tcp :443: i/o timeout +``` + +The same request succeeds from the node and fails from inside the Pod. From inside the Pod public name servers cannot be reached either, which first looks like a CoreDNS problem. The fault is NetworkManager running and being restarted on the node; stopping the NetworkManager service returns the environment to normal. How the restart breaks Pod traffic is not established. + +### Cause 2: a Multus CNI configuration breaks the frontend Pod's network to dex + +Inside the `frontend` container, a request to the dex Service on port 5556 times out: + +```text +curl: (28) Failed to connect to port 5556: Connection timed out +``` + +On the host, `/etc/cni/net.d` contains a Multus configuration that broke the container networking of the `frontend` Pod, so it cannot reach dex and pages return 500. Inside the container the network looks normally configured: `ip a` shows `eth0@if` with a Pod-network address and `route -n` shows the default route through `eth0`. Removing Multus and restarting the affected Pods restores the connection. + +### Cause 3: the host IPv6 kernel parameters were changed + +When the IPv6 kernel parameters on the hosts are modified, many pages show `数据加载错误` (data loading error) and the platform degrades until it is unusable. Developer tools on the native application page can look clean, with only a `heartbeat?action=false` request returning 200. Restoring the previous IPv6 parameters returns the pages to normal. + +## Resolution + +### Cause 1: NetworkManager was restarted after a DNS change and Pods time out to the API Service + +1. List what is not `Running`, and compare the node with the Pod: + + ```bash + kubectl get pod -A -o wide | grep -vi running + kubectl logs -n --tail=100 | grep -E 'i/o timeout|/version' + curl -k --max-time 5 https://:443/version + systemctl is-active NetworkManager + systemctl is-enabled NetworkManager + ``` + + Run the `curl` on the node, then the same request from inside the failing Pod with `kubectl exec` if its image has `curl`. Success on the node and a timeout in the Pod, right after a NetworkManager restart, matches this cause. + +2. Stop NetworkManager on the affected nodes. + + > **Warning:** With NetworkManager stopped, nothing reapplies the host's connection profiles until it is started again; addresses already configured stay in place. Do this only on nodes where the 500 followed a NetworkManager restart and Pods time out to the API Service while the node reaches it. Keep out-of-band console access to the node. To roll back, run `systemctl start NetworkManager`, which brings the fault back. + + ```bash + systemctl stop NetworkManager + kubectl get pod -A -o wide | grep -vi running + ``` + +3. Decide whether NetworkManager should also stay disabled at boot according to how the hosts' networking is managed. Keep the intended DNS server in the node configuration. + +### Cause 2: a Multus CNI configuration breaks the frontend Pod's network to dex + +1. From the `frontend` Pod, test dex and read the paired host interface: + + ```bash + kubectl get pod -A -o wide | grep -E 'frontend|dex' + kubectl get svc -A | grep dex + kubectl exec -n cpaas-system -- curl -k -I --max-time 5 https://:5556/ + kubectl exec -n cpaas-system -- cat /sys/class/net/eth0/iflink + ``` + +2. On the node that runs the Pod, list the CNI configuration and look up the interface index printed by `iflink`: + + ```bash + ls -l /etc/cni/net.d/ + ip -o link | awk -F': ' '{print $1": "$2}' | grep '^:' + ``` + + A Multus file in `/etc/cni/net.d` together with a dex timeout from `frontend` matches this cause. + +3. Uninstall Multus the way it was installed, make sure its configuration file is gone from `/etc/cni/net.d` on every node, and restart the affected Pods. + + > **Warning:** Uninstalling Multus removes the secondary networks it provides; every Pod attached to a `NetworkAttachmentDefinition` loses those interfaces when it is recreated. Do this only when no workload on the cluster needs Multus, in a maintenance window. To roll back, install Multus again the same way. + + ```bash + kubectl delete pod -n cpaas-system + kubectl exec -n cpaas-system -- curl -k -I --max-time 5 https://:5556/ + ``` + +### Cause 3: the host IPv6 kernel parameters were changed + +1. Compare the IPv6 parameters on an affected node with a node that was not changed, and look for recent edits: + + ```bash + sysctl -a 2>/dev/null | grep -E '^net\.ipv6\.conf\.(all|default)\.' + ls -lt /etc/sysctl.conf /etc/sysctl.d/ + grep -rn ipv6 /etc/sysctl.conf /etc/sysctl.d/ + ``` + +2. Set the changed parameters back to their previous values, and restore them in the file that was edited so that they survive a reboot. + + > **Warning:** IPv6 kernel parameters affect host and Pod networking on that node immediately. Do this only for values that differ from the healthy nodes or from the values before the change, and record the current values first. To roll back, set the recorded values again. + + ```bash + sysctl -w = + ``` + +## Diagnostic Steps + +1. Find out what changed on the hosts just before the errors started: DNS, NetworkManager, CNI files, or kernel parameters. + + ```bash + journalctl -u NetworkManager --since "-24h" --no-pager | tail -50 + ls -lt /etc/cni/net.d/ + ls -lt /etc/sysctl.conf /etc/sysctl.d/ + ``` + +2. Test the same target from the node and from inside a platform Pod on that node. Expected for this article: the node works and the Pod fails. + +3. Match the signature: + + ```text + x-error 500 after an nmcli DNS change and a NetworkManager restart, Pod i/o timeout to :443 -> Cause 1 + intermittent 500 on sub-page switch, frontend cannot reach dex :5556, Multus in /etc/cni/net.d -> Cause 2 + 数据加载错误 on many pages, no subnet information, host IPv6 parameters changed -> Cause 3 + ``` diff --git a/docs/en/solutions/Console_returns_502_or_does_not_open_after_installation_or_a_VIP_change_on_ACP.md b/docs/en/solutions/Console_returns_502_or_does_not_open_after_installation_or_a_VIP_change_on_ACP.md new file mode 100644 index 000000000..ead230101 --- /dev/null +++ b/docs/en/solutions/Console_returns_502_or_does_not_open_after_installation_or_a_VIP_change_on_ACP.md @@ -0,0 +1,197 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Console returns 502 or does not open after installation or a VIP change on ACP + +## Issue + +The console host does not answer, the browser receives HTTP 502 from ALB, or the installation wizard never leaves its first step. Typical situations: + +- After a new ACP 4.4 installation, the platform opens with an nginx `502`, although every Pod is `Running`. +- A new `global` installation stays on the repository initialization step (`初始化仓库配置`). +- An EIP is bound to the cluster, but the management page does not load through it. +- A business cluster reports an `alive` error, and the console cannot operate it. +- Web CLI drops a few seconds after it opens, and real-time log follow does the same. + +In the browser developer tools there is no HTTP response, a 502, or a long pending request: a reachability problem. An HTTP 500 with a JSON body from `apollo` is a different problem; see [Console returns HTTP 500 or a blank page because apollo or courier on the global cluster fails on ACP](Console_returns_HTTP_500_or_a_blank_page_because_apollo_or_courier_on_the_global_cluster_fails_on_ACP.md). + +Related article: [Console is inaccessible or a business cluster shows 0 nodes because global cannot reach an API server on ACP](Console_is_inaccessible_or_a_business_cluster_shows_0_nodes_because_global_cannot_reach_an_API_server_on_ACP.md). + +## Root Cause + +### Cause 1: ACP 4.4 was installed without all Required Aligned Extensions + +From ACP 4.4 the web console is not part of the Core Package. The Required Aligned Extensions must be copied into the `plugins/` directory of the extracted Core Package before the installer runs: + +- Alauda Container Platform Web Console +- Alauda Container Platform Web Console Central +- Alauda Container Platform Marketplace Web Console +- Alauda Container Platform Helm Application Catalog +- Alauda Container Platform Observability Essentials +- Alauda Container Platform Essentials +- Alauda Container Platform Cluster Essentials +- Alauda Container Platform GatewayAPI Plugin + +If they are missing, the console Pods, Services, and Endpoints are never created and ALB returns 502. Leaving out a single one gives the same 502: every Pod is `Running` and the `alb2-controller` log can even show 200 responses from the login chain. Copying additional marketplace packages beyond this set into `plugins/` makes `setup.sh` fail. + +### Cause 2: kubelet does not start because required kernel parameters are missing + +A new `global` installation stays on the repository initialization step. The `cpaas-install` logs are quiet, the `cls` object reports errors, `kubelet` is not active, and its journal says required kernel parameters are missing. No container can start until `kubelet` runs. + +### Cause 3: the alive VIP mode is not arp + +`alive` holds the cluster VIP. When the VIP mode file `/etc/kubernetes/alive/vipmode` on the nodes is not `arp`, an EIP bound to the cluster does not reach the management page. + +### Cause 4: alive crash-loops because alive.yaml names a NIC that no longer exists + +The business cluster's nodes are `Ready`, but `kubectl describe cls` shows that port 6443 on the business cluster fails, `cluster-api-provider-alauda-manager` logs an error loading the cluster, and the `alive` Pod is `CrashLoopBackOff` with a log of mostly `logrotate.state` and VIP or IPVS state. The network interface name changed (for example from `ens192` to `eth0`), but `alive.yaml` still names the old one, so `alive` cannot attach the VIP and the `global` cluster cannot reach the business API. + +### Cause 5: an external stream proxy closes long-lived connections after 5 seconds + +Web CLI opens (status 101) and dies a few seconds later. The ALB rule `cpaas-system/web-cli` logs `/auth/v1/pod/shell` with a `request_time` of a few seconds, and `web-terminal` logs `EOF`. Bypassing the external nginx makes the session stable. On that nginx, the `stream` listener that forwards 6443 has `proxy_timeout 5s`, while 443 uses `3m`. + +## Resolution + +### Cause 1: ACP 4.4 was installed without all Required Aligned Extensions + +1. Check whether the console has Pods, Services, and Endpoints. Empty results on a new 4.4 installation match this cause: + + ```bash + kubectl get pods,svc,endpoints -A | grep -Ei 'frontend|console|alb' + ``` + +2. Compare the packages that were copied into the Core Package `plugins/` directory with the eight Required Aligned Extensions: + + ```bash + ls /plugins/ + ``` + +3. To rule out a self-built VIP, test it from each control-plane node and compare: TLS handshake, status code, and agreement between nodes. A TLS handshake EOF points at the VIP (loopback access, ARP, routing, or the load balancer health checks), not at the ALB certificates: + + ```bash + curl -vk --connect-timeout 5 --resolve :443: https:/// + ``` + +4. Install again with the complete set. + + > **Warning:** The recovery runs `cleanup.sh` on every node and installs again, which tears down the failed installation. Never run it against a cluster that already serves workloads, and copy off any data you need first. A wiped installation can only be restored from backup. + + Copy exactly the eight Required Aligned Extensions (and no other marketplace packages) into `plugins/`, run `cleanup.sh` on each node, then run `setup.sh` as documented. ALB stops returning 502 once the console Service has Endpoints. + +If DNS problems stopped Kubernetes from coming up during the same installation, fix DNS and finish the cluster installation first; that is a separate, earlier fault. + +### Cause 2: kubelet does not start because required kernel parameters are missing + +1. On the node, check `kubelet` and the kernel parameters: + + ```bash + systemctl status kubelet --no-pager + journalctl -u kubelet -n 100 --no-pager + sysctl vm.overcommit_memory vm.panic_on_oom kernel.panic kernel.panic_on_oops kernel.keys.root_maxkeys kernel.keys.root_maxbytes + ``` + +2. Apply the parameters and restart `kubelet`. + + > **Warning:** `kernel.panic=10` and `kernel.panic_on_oops=1` make the node reboot itself after a kernel panic. Apply them only on nodes that will run ACP, and keep console access in case a key is rejected. To roll back, remove `/etc/sysctl.d/acp.conf`, restore the previous values, and run `sysctl` again. + + ```bash + cat >/etc/sysctl.d/acp.conf <<'EOF' + vm.overcommit_memory = 1 + vm.panic_on_oom = 0 + kernel.panic = 10 + kernel.panic_on_oops = 1 + kernel.keys.root_maxkeys = 1000000 + kernel.keys.root_maxbytes = 25000000 + EOF + sysctl -p /etc/sysctl.d/acp.conf + systemctl restart kubelet + systemctl is-active kubelet + ``` + +3. Once `kubelet` stays active, the wizard can continue. If `kubelet` is active and the wizard is still stuck, go back to the `cls` error. + +### Cause 3: the alive VIP mode is not arp + +On each node where `alive` runs: + +> **Warning:** Changing the VIP mode and restarting keepalived moves or drops the management VIP for a few seconds, and open console sessions break. Do this in a maintenance window. To roll back, write the previous mode back to the file and restart keepalived again. + +```bash +cat /etc/kubernetes/alive/vipmode +echo arp > /etc/kubernetes/alive/vipmode +kubectl get pods -n kube-system --no-headers | awk '/alive/ {print $1}' +kubectl exec -n kube-system -- pkill keepalived +``` + +The management page answers on the EIP once `alive` is `Running` again. + +### Cause 4: alive crash-loops because alive.yaml names a NIC that no longer exists + +1. Compare the live interfaces with the one in the `alive` configuration: + + ```bash + ip -br link + kubectl logs -n kube-system --tail=100 + ls /etc/kubernetes/alive + ``` + +2. Back up `alive.yaml` and change the interface to the one that `ip -br link` shows as `UP`. + + > **Warning:** A wrong interface name leaves the VIP unbound and the business cluster unreachable from `global`. Restoring the backup is the rollback. + + ```bash + cp -a /etc/kubernetes/alive/alive.yaml /root/alive.yaml.bak + vi /etc/kubernetes/alive/alive.yaml + ``` + +3. Restart `cluster-api-provider-alauda-manager` on the `global` cluster so that it reloads cluster access. + + > **Warning:** The restart briefly pauses cluster reconciliation on the `global` cluster. + + ```bash + kubectl get deploy -A | grep cluster-api-provider-alauda + kubectl rollout restart deployment -n cluster-api-provider-alauda-manager + kubectl get pods -n kube-system | grep alive + ``` + + `alive` leaves `CrashLoopBackOff`, and `kubectl describe cls` stops reporting 6443 failures. + +### Cause 5: an external stream proxy closes long-lived connections after 5 seconds + +On the external nginx that publishes the VIP, set the `stream` server that listens on 6443 to the same order of timeout as 443, then reload nginx: + +> **Warning:** A longer `proxy_timeout` keeps idle TCP connections open longer on that proxy; it does not change ACP. To roll back, restore the previous value, which brings the disconnects back. + +```nginx +# stream server that listens on 6443 +proxy_timeout 3m; +``` + +Confirm that `/auth/v1/pod/shell` stays on 101 with a growing `request_time`, and that `web-terminal` no longer logs an immediate `EOF`. + +## Diagnostic Steps + +```bash +curl -k --max-time 5 https://:6443/readyz +kubectl get pods -n kube-system | grep -E 'alive|kube-apiserver' +kubectl get pods -n cpaas-system | grep -E 'alb|frontend|apollo' +kubectl get pods,svc,endpoints -A | grep -Ei 'frontend|console' +``` + +```text +ALB 502, no console Endpoints on a new 4.4 installation, or one extension missing -> Cause 1 +kubelet inactive, installation stuck on repository initialization -> Cause 2 +EIP silent, vipmode is not arp -> Cause 3 +alive CrashLoopBackOff, alive.yaml names a NIC that does not exist -> Cause 4 +Web CLI status 101 then EOF, only through the external nginx -> Cause 5 +``` diff --git a/docs/en/solutions/Console_returns_HTTP_500_or_a_blank_page_because_apollo_or_courier_on_the_global_cluster_fails_on_ACP.md b/docs/en/solutions/Console_returns_HTTP_500_or_a_blank_page_because_apollo_or_courier_on_the_global_cluster_fails_on_ACP.md new file mode 100644 index 000000000..9b17e06b3 --- /dev/null +++ b/docs/en/solutions/Console_returns_HTTP_500_or_a_blank_page_because_apollo_or_courier_on_the_global_cluster_fails_on_ACP.md @@ -0,0 +1,195 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.14" + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Console returns HTTP 500 or a blank page because apollo or courier on the global cluster fails on ACP + +## Issue + +The browser reaches the platform host, but a page stays blank, spins, or returns HTTP 500. Typical reports are "apollo reports errors and the platform returns 500", a monitoring home page that spins for one cluster, or Operations Center saying that monitoring is not installed while the monitoring Pods are `Running`. + +Start from the failing request, not the screenshot. In the browser developer tools (**Network**), sort by status and note the first failing path and its status code: + +| Failing request | First component to check | +| --- | --- | +| Broad `/platform/...` 500, or the document itself errors after login | `apollo` and other `cpaas-system` Pods on the `global` cluster | +| `/platform/monitoring.alauda.io/v1beta1/clusters//metrics` returns 500, Grafana says `cluster not found` | the `apollo` cluster cache | +| Operations Center says monitoring is not installed, or vmui returns 403 | `courier` and `courier-api` | +| The document request never completes, or HTTP 502 from the load balancer | the entry path, not this article | + +`apollo`, `courier`, `courier-api`, `frontend`, `archon`, and `erebus` are console backends in `cpaas-system` on the `global` cluster. `apollo` aggregates the platform API, `erebus` proxies calls toward a target cluster, `courier` and `courier-api` feed the monitoring and inspection views, `frontend` serves the shell, and `archon` serves file download from containers. + +## Root Cause + +### Cause 1: apollo crash-loops with apiserver 401 because node clocks differ + +`apollo` is `CrashLoopBackOff` and its log shows cache calls returning 401 and an API server authentication failure. The node clocks are skewed, for example after nodes were powered off without stopping `kubelet` and the container runtime, or after a single master was rebooted. Token authentication between platform components fails when the clocks differ. On a 3.14 `global` cluster the same skew after one master reboot showed as `katanomi` not starting, `archon` and other components returning 401, and the platform saying the license could not be loaded. Recreating `apollo` without fixing the clocks clears the symptom only until the next authentication check. + +### Cause 2: apollo does not have the cluster in its cache + +The monitoring home page of a cluster spins, the metrics API returns 500, and the global `apollo` log contains `get cluster from cache error`, `cluster not found`, or `New manager error`. The business cluster's Prometheus answers. This happens for a newly created cluster, or when `apollo` Pods have run for a long time without picking up a new cluster. Restarting the global `apollo` Pods reloads the cluster list. Restarting `erebus` or other controllers does not. + +### Cause 3: apollo receives HTML instead of JSON from the metrics backend + +Monitoring charts stay empty after Prometheus was installed or reinstalled, while Prometheus itself returns `up`. `apollo` repeats `bad_response: invalid character '<' looking for beginning of value`: it received an HTML or XML body instead of JSON. A user-created Prometheus Deployment in `cpaas-system` is not necessarily the cause; restarting `apollo` restores the charts. + +### Cause 4: courier holds a stale view and the UI says monitoring is not installed + +Marketplace, the monitoring Pods, and the AppReleases are healthy, but Operations Center says the monitoring component is not installed and vmui returns 403. Restarting `courier` and `courier-api` reloads the view. + +### Cause 5: courier panics in the leftover resource-statistics job + +`courier` keeps restarting with a panic in the resource usage statistics path. Older versions ran that scheduled job; the current version no longer uses the feature, but the code that schedules it is still present. Scheduling the job for a date that never occurs stops the restarts. + +### Cause 6: the global control-plane disk is full of etcd backups + +The browser reports that the site cannot be reached, and `kubectl get pods -n cpaas-system` shows Pods that are not `Running`. The global control-plane filesystem is at 100% because etcd backups under `/cpaas-backup` filled it. This is a node disk fault that shows as a dead console. + +## Resolution + +### Cause 1: apollo crash-loops with apiserver 401 because node clocks differ + +1. Compare the clocks of all nodes. After a single master reboot, that node is the likely outlier: + + ```bash + date -u + timedatectl status + ``` + +2. Synchronize every control-plane and worker node to the same NTP source. + + > **Warning:** Stepping a node clock affects TLS, leases, and log order. Prefer slewing through the NTP service; if the clock must be stepped, do it in a maintenance window and expect brief API server 401 errors until `apollo` is restarted. + +3. Restart `apollo` only, and confirm the new Pod stays `Running` and the 500 is gone: + + ```bash + kubectl get pods -n cpaas-system --no-headers | awk '$1 ~ /^apollo-/ {print $1}' + kubectl delete pod -n cpaas-system + ``` + +### Cause 2: apollo does not have the cluster in its cache + +1. Confirm the log signature: + + ```bash + kubectl logs -n cpaas-system --tail=200 | grep -E 'from cache error|not found|New manager error' + ``` + +2. Restart every global `apollo` Pod. + + > **Warning:** Platform API calls, including monitoring and other aggregated pages, return errors until the new Pods are `Ready`. Do this when users can retry. The Pods are recreated from the same spec. If the new Pods log the same `cluster not found` after several minutes, the cluster object is missing or the API watch is failing; do not keep deleting Pods. + + ```bash + kubectl delete pod -n cpaas-system + ``` + +The monitoring-specific variants of this cache problem are covered in more detail in [Monitoring pages stay empty because courier-api or apollo holds a stale cluster cache on ACP](Monitoring_pages_stay_empty_because_courier-api_or_apollo_holds_a_stale_cluster_cache_on_ACP.md). + +### Cause 3: apollo receives HTML instead of JSON from the metrics backend + +1. Confirm the log line: + + ```bash + kubectl logs -n cpaas-system --tail=200 | grep bad_response + ``` + +2. Restart `apollo`. + + > **Warning:** Platform APIs return errors until the Pod is `Ready`. Do not delete a user's Prometheus just because it is in `cpaas-system`. + + ```bash + kubectl delete pod -n cpaas-system + ``` + +3. If `bad_response` continues, the upstream still returns a non-JSON body; check it before restarting `apollo` again. + +### Cause 4: courier holds a stale view and the UI says monitoring is not installed + +> **Warning:** Restarting these Deployments interrupts the monitoring and inspection views until the new Pods are `Ready`. The monitoring store keeps running. + +```bash +kubectl get deploy -n cpaas-system | grep courier +kubectl rollout restart deployment -n cpaas-system +kubectl rollout restart deployment -n cpaas-system +kubectl rollout status deployment -n cpaas-system +``` + +### Cause 5: courier panics in the leftover resource-statistics job + +1. Export the Deployment: + + ```bash + kubectl get deploy -n cpaas-system -o name | grep courier + kubectl get deploy -n cpaas-system -o yaml > "$HOME/courier-deploy.yaml" + ``` + +2. Add the following argument to the `command` of the `courier` container. It schedules the job for 31 February, which never occurs. + + > **Warning:** Editing the command rolls the Deployment and interrupts health and monitoring views until the new Pod is `Ready`. Do this only when the panic is in the resource-statistics job. To roll back, apply the export, which re-enables the job and can bring the panic back. + + ```yaml + - --record-job-spec=CRON_TZ=Asia/Shanghai 0 0 31 2 * + ``` + + ```bash + kubectl edit deploy -n cpaas-system + ``` + +3. `courier` stays `Running` after the rollout. If the argument disappears and the panic returns, the owning release is still rendering the old command. + +### Cause 6: the global control-plane disk is full of etcd backups + +1. Find what fills the disk: + + ```bash + df -h + du -sh /cpaas-backup/* 2>/dev/null | sort -h + ``` + +2. Remove old backups. + + > **Warning:** Files under `/cpaas-backup` are etcd backups and deleting them is irreversible. Copy the newest successful backup to another disk first, and never touch the live etcd data directory. If the disk is full for another reason, removing backups is not the fix. + + ```bash + rm -f /cpaas-backup/ + df -h + ``` + +3. Watch the `cpaas-system` Pods return to `Running`, then load the console again. + +## Diagnostic Steps + +1. In the browser **Network** panel, reproduce the problem once and note the first failing path and status. + +2. On the `global` cluster, list the platform Pods that are not `Running` and read the matching log: + + ```bash + kubectl get pods -n cpaas-system -o wide + kubectl get pods -n cpaas-system | grep -viE 'Running|Completed' + kubectl logs -n cpaas-system --since=30m --tail=200 + ``` + +3. Match the log text: + + ```text + 401 and CrashLoopBackOff on apollo -> Cause 1 (node time) + archon 401, license cannot be loaded after one master reboot -> Cause 1 + get cluster from cache error -> Cause 2 + bad_response: invalid character '<' -> Cause 3 + vmui 403, UI says monitoring is not installed -> Cause 4 + courier panic in the resource-statistics job -> Cause 5 + filesystem 100% and /cpaas-backup is large -> Cause 6 + ``` + +4. If only one feature path fails (logs, a registry, DevOps, a data-service instance) and `apollo` is healthy, the problem is in that product, not in the platform backends. diff --git a/docs/en/solutions/Create_and_manage_alert_policies_and_notification_routing_on_ACP.md b/docs/en/solutions/Create_and_manage_alert_policies_and_notification_routing_on_ACP.md new file mode 100644 index 000000000..1ac1abd7e --- /dev/null +++ b/docs/en/solutions/Create_and_manage_alert_policies_and_notification_routing_on_ACP.md @@ -0,0 +1,208 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.16" + - "4.0" + - "4.3" +--- + +# Create and manage alert policies and notification routing on ACP + +## Issue + +Common questions when setting up alerting on Alauda Container Platform: + +- How do I get an alert to the operations staff when a resource is abnormal or reaches a threshold? +- Can one alert policy apply to all clusters? +- How do I create alert policies for many namespaces in batch? +- Every alert rule uses `cpaas-admin-notification`; can I change the notification policy of all of them at once? +- How do I raise a threshold, for example make the node zombie-process alert fire only above 7? +- How do I alert on my own metric, such as JVM heap usage above 92% for 3 minutes? +- How often are alert rules evaluated? +- Is there a disk usage alert for master nodes? +- How do I list the existing alert policies and their notification channels, or get a table of the built-in alerts? +- Can a failed backup raise an alert? + +A rule that does not fire or is rejected is covered in [Alert rules do not fire, are rejected, or cannot be edited on ACP](Alert_rules_do_not_fire_are_rejected_or_cannot_be_edited_on_ACP.md). A notification that is configured and does not arrive is covered in [Alert notifications are not delivered because courier-api or apollo cannot reach the receiver on ACP](Alert_notifications_are_not_delivered_because_courier-api_or_apollo_cannot_reach_the_receiver_on_ACP.md). + +## Root Cause + +### Cause 1: notifying people needs a template, a notification policy, and an alert policy that references it + +Alerts reach people only when three pieces exist and are connected: a notification template (what the message says), a notification policy (who receives it and by which method, through notification contact groups), and an alert policy whose **Notification Policy** field references that notification policy. Built-in policies already reference one (for example `cpaas-admin-notification`); a new policy needs one selected. + +### Cause 2: alert policies are per cluster + +An alert policy belongs to one cluster, so a single policy cannot apply to all clusters. To avoid repeating the setup, create an alert template once and create the policy from it in each cluster. + +### Cause 3: there is no batch creation of alert policies + +The platform has no built-in function to create alert policies in batch, for example one per namespace. The options are creating each policy from an alert template, or creating the underlying `PrometheusRule` resources directly (creating an alert policy creates a `PrometheusRule`), with the target fields replaced in each copy. + +### Cause 4: the notification policy on alert policies cannot be changed in bulk + +There is no bulk change of the notification policy bound to alert policies. The default rules come with `cpaas-admin-notification` by design, and each policy has to be changed individually. + +### Cause 5: a threshold is part of the rule's trigger condition + +A threshold, for example of the node zombie-process rule, is changed by editing the rule's **Trigger Condition** (operator, value, and duration) inside the alert policy. The built-in node rule "Number of zombie processes on the node" in `cpaas-node-rules` can be changed to `> 7` for 1 minute this way, and the rule stays enabled. + +### Cause 6: a custom metric alert is a custom alert rule in a resource alert policy + +A custom metric such as JVM heap usage is alerted on with a rule of type **Custom Alert** (`自定义告警`) inside a resource alert policy (`资源类告警`) whose resource type is compute component (`计算组件`). On ACP 3.12 the rule takes a metric name (up to 64 characters), a PromQL expression (up to 512 characters), an optional unit and legend, a trigger condition (operator, threshold with up to 17 integer digits and 4 decimal places, and duration), and a level. After several compute components are selected, some built-in metric alerts are not supported; configuring per component supports all alert metrics. + +### Cause 7: metrics are scraped every 60 seconds and rules evaluated every 30 seconds by default + +By default metrics are scraped every 60 seconds and alert rules are evaluated every 30 seconds. The rule's duration (for example 1 minute) is counted on top of those. + +### Cause 8: node disk usage is covered by the built-in node policy + +The default node policy includes the rule `节点磁盘空间使用率 > 80% 且持续 1 分钟` (node disk space usage above 80% for 1 minute), type metric alert, level critical, enabled by default, together with the node CPU, memory, health, NodeExporter, and OOM rules. The platform inspection also has a node disk usage item, but it reflects only the moment the inspection runs. + +### Cause 9: alert policies are PrometheusRules, notification policies are notifications.ait.alauda.io + +Alert policies are stored as `prometheusrules` in each cluster, and each references its notification policies in the annotation `alert.cpaas.io/notifications`. Notification policies are `notifications.ait.alauda.io` resources on the `global` cluster; their YAML lists the bound contact groups and methods. The product documentation has no ready-made table of built-in alerts, but one can be generated from the rule labels. + +### Cause 10: backup task failures cannot be alerted on + +Application backup results are not exposed as Prometheus metrics (there is no `ServiceMonitor` for them), so no alert rule can watch them. etcd backup monitoring is different and can be configured through Prometheus. + +## Resolution + +### Cause 1: notifying people needs a template, a notification policy, and an alert policy that references it + +1. **Platform Management** > **Operations Center** > **Notification**. On **Notification Templates**, create a template of type **Alert Message**, or reuse one. +2. On **Notification Contact Groups**, create the group of receivers. +3. On **Notification Policies**, click **Create Notification Policy** and select the contact group, the methods, and the template. +4. **Operations Center** > **Alerts** > **Alert Policies**, select the cluster, and click **Create Alert Policy**, or open an existing policy with **Actions** > **Update**. +5. Under **Policy Configuration**, select the notification policy in **Notification Policy**, set **Alert Sending Interval**, and save. + +### Cause 2: alert policies are per cluster + +1. **Operations Center** > **Alerts** > **Alert Templates** > **Create Alert Template**. Choose the resource type (for example cluster) and add the rules once. +2. In each target cluster under **Alert Policies**, open the dropdown next to **Create Alert Policy** and choose **Create Alert Policy from Template**. +3. Select the template and fill in the name and display name. For a cluster template the resource is set to the current cluster. Click **Create**. + +### Cause 3: there is no batch creation of alert policies + +For a few targets, use templates (Cause 2). For many namespaces, create one policy in the console, export its `PrometheusRule`, and generate copies: + +```bash +kubectl get prometheusrules.monitoring.coreos.com -n -o yaml > /tmp/policy-template.yaml +grep -n -E 'name:|alert.cpaas.io/namespace|namespace=~|alert_involved_object_namespace|resourceVersion|uid:|creationTimestamp' /tmp/policy-template.yaml +``` + +The namespace appears in the annotation `alert.cpaas.io/namespace`, in the `namespace=~"..."` matchers of every `expr`, and in the `alert_involved_object_namespace` label of every rule. In each copy, remove `uid`, `resourceVersion`, `creationTimestamp`, and `managedFields`, give it a unique `metadata.name`, and replace the namespace in all three places. Validate, then apply: + +```bash +kubectl apply --dry-run=server -f /tmp/policy-.yaml +kubectl apply -f /tmp/policy-.yaml +``` + +Open one generated policy in the console and check that its rules and notification policy display correctly before creating the rest. + +### Cause 4: the notification policy on alert policies cannot be changed in bulk + +Change it on each alert policy: **Alert Policies** > **Actions** > **Update** > **Notification Policy**. To see which policies reference a notification policy first: + +```bash +kubectl get prometheusrules.monitoring.coreos.com -A -o yaml | grep -n 'alert.cpaas.io/notifications' +``` + +### Cause 5: a threshold is part of the rule's trigger condition + +1. **Platform Management**, select the cluster, **Operations Center** > **Alerts** > **Alert Policies**. +2. Open the policy (for example `cpaas-node-rules`) and click edit on the rule. +3. In **Edit Alert Rule**, change the value (and, if needed, the operator and duration) in **Trigger Condition**, then save the rule and the policy. + +Built-in rules can be restored from the platform's own copy, for example by an upgrade, so re-check custom thresholds on built-in policies after an upgrade. + +### Cause 6: a custom metric alert is a custom alert rule in a resource alert policy + +1. **Alert Policies** > **Create Alert Policy**. Alert type resource alert, resource type compute component, component type Deployment. Optionally choose namespaces and resource names; with none selected, any matches, up to 30 can be selected, and new namespaces are included automatically. +2. Under the alert rules, click **Add Alert Rule** and set the alert type to **Custom Alert**. +3. Enter the metric name and the expression. Scope the expression to the workload and test it in the metrics UI first. In a project or namespace view the expression must carry the namespace label, or the preview stays empty: + + ```promql + max by (pod) (jvm_memory_used_bytes{area="heap",namespace="",pod=~"-.*"} / jvm_memory_max_bytes{area="heap",namespace="",pod=~"-.*"}) + ``` + +4. Set the trigger condition to `>=` `0.92` (use `>` if `>=` is not offered) with a duration of 3 minutes, and choose the level. +5. Bind the notification policy and save. + +### Cause 7: metrics are scraped every 60 seconds and rules evaluated every 30 seconds by default + +A condition becomes visible after the next scrape (up to 60 seconds), is noticed at the next evaluation (up to 30 seconds), and fires after the rule's duration has also passed. Read the live values before relying on them: + +```bash +kubectl get vmagent -A -o jsonpath='{range .items[*]}{.metadata.name}{" scrapeInterval="}{.spec.scrapeInterval}{"\n"}{end}' +kubectl get vmalert -A -o jsonpath='{range .items[*]}{.metadata.name}{" evaluationInterval="}{.spec.evaluationInterval}{"\n"}{end}' +``` + +On a Prometheus cluster, read `spec.scrapeInterval` and `spec.evaluationInterval` from `kubectl get prometheus -A -o yaml`. An empty value means the component default. + +### Cause 8: node disk usage is covered by the built-in node policy + +Confirm that the disk rule is present and enabled: + +```bash +kubectl get prometheusrules.monitoring.coreos.com -A | grep node-rules +kubectl get prometheusrules.monitoring.coreos.com -n cpaas-system cpaas-node-rules -o yaml | grep -n -A3 node.disk.space.utilization +``` + +For another threshold or scope, create a custom policy: **Create Alert Policy**, alert type resource alert, resource type node, resource name all nodes, then add a rule with the disk metric and the threshold. If a root disk above 90% is neither alerted nor reported by inspection, contact technical support. + +### Cause 9: alert policies are PrometheusRules, notification policies are notifications.ait.alauda.io + +On each cluster, list the alert policies and their notification policies: + +```bash +kubectl get prometheusrules -A +kubectl get prometheusrules -n -o yaml | grep alert.cpaas.io/notifications +``` + +On the `global` cluster, list the notification policies and read their contact groups and methods: + +```bash +kubectl get notifications.ait.alauda.io -A +kubectl get notifications.ait.alauda.io -n -o yaml | grep -i group -C 5 +kubectl get notifications.ait.alauda.io -n -o yaml | grep -i methods -C 5 +``` + +For a table of built-in alerts (policy, alert, indicator, severity, resource kind, expression; a missing label prints an empty column): + +```bash +kubectl get prometheusrules -n cpaas-system -o json | jq -r ' + .items[] | .metadata.name as $p | .spec.groups[]?.rules[]? + | select(.alert != null) + | [$p, .alert, (.labels.alert_indicator // ""), (.labels.severity // ""), (.labels.alert_involved_object_kind // ""), (.expr | gsub("\n"; " "))] + | @tsv' +``` + +Run it on each cluster type you need (`global` and a business cluster); the built-in set differs. + +### Cause 10: backup task failures cannot be alerted on + +No alert can be configured for a failed application backup. Check results directly from the `status` of the backup objects, or look for the backup files in the target directory: + +```bash +kubectl get backups -A +``` + +If `backups` is ambiguous on the cluster, list the kinds with `kubectl api-resources | grep -i backup` and use the full name. + +## Diagnostic Steps + +1. The alert fires in alert history and nobody is told: the alert policy has no notification policy, or the notification policy has no contact group or template (Cause 1). + +2. One policy for all clusters, batch creation, or a bulk change of the notification policy: not available as a single action (Causes 2, 3, 4). + +3. A different threshold: Cause 5. A metric the platform does not ship: Cause 6. + +4. "The alert came late": add the scrape interval, the evaluation interval, and the rule's duration (Cause 7) before suspecting the evaluator. + +5. "Is there an alert for X?": search the existing policies first (Cause 9). Node disk usage is built in (Cause 8). Backup failures are not alertable (Cause 10). diff --git a/docs/en/solutions/Custom_or_imported_monitoring_dashboards_show_parse_errors_unsupported_chart_types_or_no_data_on_ACP.md b/docs/en/solutions/Custom_or_imported_monitoring_dashboards_show_parse_errors_unsupported_chart_types_or_no_data_on_ACP.md new file mode 100644 index 000000000..d2568549c --- /dev/null +++ b/docs/en/solutions/Custom_or_imported_monitoring_dashboards_show_parse_errors_unsupported_chart_types_or_no_data_on_ACP.md @@ -0,0 +1,170 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.16 + - 4.0 + - 4.1 + - 4.2 +--- + +# Custom or imported monitoring dashboards show parse errors, unsupported chart types, or no data on ACP + +## Issue + +A dashboard that was built in, or imported into, the platform monitoring dashboards (**Operations Center** > **Monitoring** > **Monitoring Dashboards**) does not behave like it did in Grafana. Typical symptoms: + +- A panel shows `加载失败,请 刷新` (load failed, please refresh) with the error `update matchers error: 1:59: parse error: missing unit character in duration`. The query uses `[$__range]`. +- After a Grafana dashboard JSON was imported, some panels show `不支持的图表类型` (unsupported chart type) instead of a chart. +- A standard Grafana dashboard JSON (for example one supplied by a hardware or software vendor) was imported, and some panels show no data although the same expressions return data on the Prometheus page. This is sometimes described as the import "escaping" or changing characters of the original JSON. +- A user wants to create a new folder for dashboards, or wants application monitoring to show a custom dashboard selected by application labels. +- An external dashboard JSON was imported into the custom monitoring panel of an application (the **Monitoring** tab of a Deployment), and the panels show `加载失败,请 刷新`. In the browser developer tools a `monitordashboards` request fails with `mysql_up should has selector namespace=...`. +- There is no Envoy Gateway dashboard under **Monitoring Dashboards** comparable to the ALB dashboards, and 4xx/5xx response-code data is needed for alerting. +- In the monitoring dashboards the **HTTPRoute status** and **Rule Status** panels show no data while every other panel is normal. + +Built-in dashboards that freeze the browser, stay blank, or are missing are a different problem; see [Monitoring page freezes the browser or built-in dashboards are missing on ACP](Monitoring_page_freezes_the_browser_or_built_in_dashboards_are_missing_on_ACP.md) and [Monitoring dashboard panels are blank or drop to zero while the backend has data on ACP](Monitoring_dashboard_panels_are_blank_or_drop_to_zero_while_the_backend_has_data_on_ACP.md). + +## Root Cause + +### Cause 1: Grafana built-in variables such as $__range are not supported + +A query such as `sum(last_over_time(api_requests_total{appId="$appId"}[$__range]))` fails with `parse error: missing unit character in duration`. The platform monitoring dashboard does not support Grafana built-in variables such as `$__range`, and it does not support making the range selector inside a query follow a variable. The placeholder is sent unchanged, so the range selector is not a valid PromQL duration. + +### Cause 2: the imported panel type is not supported + +Some Grafana panel types are not supported by the platform dashboard. An imported panel of such a type shows `不支持的图表类型` (unsupported chart type). + +### Cause 3: imported queries keep Grafana template-variable selectors + +An imported query can keep label selectors that reference Grafana template variables, for example `DCGM_FI_DEV_POWER_USAGE{instance=~"$instance", gpu=~"$gpu"}` in the NVIDIA DCGM Exporter dashboard. If those variables are not defined on the platform dashboard, they do not resolve and the panel returns no data. + +### Cause 4: dashboard folders and label-based application dashboards + +Whether a new dashboard folder can be created depends on the version. On ACP 4.0.3 new monitoring folders cannot be created. On versions whose **Create Dashboard** form has a required **Folder** (`文件夹`) field, limited to 32 characters, typing a new name into that field creates the folder. + +Showing a custom dashboard in application monitoring based on application labels is not supported. + +### Cause 5: importing external JSON into an application's custom monitoring panel is not supported + +On the **Monitoring** tab of a Deployment, the custom panel accepts charts built in the console. Importing an external dashboard JSON into it, or changing its panels through YAML, is not a supported feature. Imported panels fail with `加载失败,请 刷新`; in the browser developer tools the `variables?expr=label_values...` and `metrics?start=...` requests fail, and a `monitordashboards` request returns `code: 500`, `reason: "Unknown"` with a message such as `mysql_up should has selector namespace=...`. The message shows that queries on this panel must carry a `namespace=` selector, which the imported query does not have. + +Importing a Grafana JSON as a dashboard under **Operations Center** > **Monitoring** > **Monitoring Dashboards** is a different feature and is not affected. + +### Cause 6: the Envoy Gateway dashboards have not been added + +The Envoy Gateway dashboards are not listed under **Monitoring Dashboards** until they are added to the cluster. They are not installed with Envoy Gateway itself. + +### Cause 7: HTTPRoute status and Rule Status panels are empty because the cluster has no HTTPRoute + +Both panels are built from HTTPRoute resources. When the cluster has no HTTPRoute, they have nothing to show. This is expected and is not a fault. + +### Cause 8: a dashboard variable expression is malformed + +An imported dashboard can carry a variable (`变量`, Variables) whose own expression is invalid, for example with an extra closing parenthesis. The panels that use the variable then show no data, while the panel expressions return data on the Prometheus page when real values are filled in. This differs from Cause 3: here the variable is defined on the dashboard, but its query cannot be parsed. + +## Resolution + +### Cause 1: Grafana built-in variables such as $__range are not supported + +Remove the variable from the range selector. Put a fixed duration in the query, or use a query that does not need the range, and select the window with the time range picker at the top of the dashboard (for example `近 7 天`, last 7 days): + +```promql +sum(last_over_time(api_requests_total{appId="$appId"}[7d])) +``` + +A simple check that the time picker drives the panel is a query such as `up{service=~".*"}`, which renders data across the selected range. + +### Cause 2: the imported panel type is not supported + +Edit each affected panel and change its type to a line chart (`趋势图`). No setting renders the unsupported Grafana panel type. + +### Cause 3: imported queries keep Grafana template-variable selectors + +1. Edit the panel query and remove the selectors that reference Grafana template variables (`$instance`, `$gpu`, and similar), or replace them with fixed values or with variables defined on the platform dashboard. +2. Run the edited expression in vmui or the Prometheus UI before saving it, for example: + + ```promql + DCGM_FI_DEV_POWER_USAGE + ``` + + With the legend `GPU {{.gpu}}`, this draws the power usage of each GPU. + +### Cause 4: dashboard folders and label-based application dashboards + +If the **Create Dashboard** form has the **Folder** field, type the new folder name (up to 32 characters) there when creating the dashboard. On a version without that field, new folders cannot be created. Label-based custom dashboards in application monitoring are not available; there is no configuration for them. + +### Cause 5: importing external JSON into an application's custom monitoring panel is not supported + +1. Build each chart with **Add Chart** (`添加图表`) on the application's custom panel instead of importing a JSON file. +2. Test each expression in vmui or the Prometheus UI first, restricted to the application's namespace: + + ```promql + mysql_up{namespace=""} + ``` + +There is no bulk-import path for application custom panels. + +### Cause 6: the Envoy Gateway dashboards have not been added + +1. Follow [How to Add Dashboards for EnvoyGateway](How_to_add_Dashboard_for_EnvoyGateway.md) (ACP 4.2.x and later) on the cluster where Envoy Gateway is installed. It applies a manifest, `envoy-gateway-dashboard.yaml`, that creates the `MonitorDashboard` resources `envoygateway-adminspace-control-panel-dashboard` and `envoygateway-adminspace-data-panel-dashboard` and the `PodMonitor` resources `envoy-gateway-controlplane-monitoring` and `envoy-gateway-proxy-monitoring` in `cpaas-system`. +2. Verify the objects: + + ```bash + kubectl -n cpaas-system get monitordashboard envoygateway-adminspace-control-panel-dashboard envoygateway-adminspace-data-panel-dashboard + kubectl -n cpaas-system get podmonitor envoy-gateway-controlplane-monitoring envoy-gateway-proxy-monitoring + ``` + +3. Open the dashboards in the `container-platform` folder. The data-plane dashboard has a **Response Codes (Route)** panel, which reads `envoy_cluster_upstream_rq_xx` grouped by `envoy_response_code_class`. For an alert on the 5xx responses of one gateway, use an expression such as: + + ```promql + sum by (envoy_cluster_name) (rate(envoy_cluster_upstream_rq_xx{gateway_namespace="", gateway="", envoy_response_code_class="5"}[2m])) + ``` + + Use `envoy_response_code_class="4"` for 4xx. + +To remove the dashboards and PodMonitors again, run `kubectl delete -f envoy-gateway-dashboard.yaml` with the same manifest. + +### Cause 7: HTTPRoute status and Rule Status panels are empty because the cluster has no HTTPRoute + +Confirm that the cluster has no HTTPRoute: + +```bash +kubectl get httproute -A +``` + +`No resources found` means the empty panels are expected and nothing needs to be fixed. The panels show data once HTTPRoute resources exist. + +### Cause 8: a dashboard variable expression is malformed + +1. Export the dashboard object and keep the export as a copy to compare against: + + ```bash + kubectl get monitordashboards.ait.alauda.io -A | grep -i + kubectl get monitordashboards.ait.alauda.io -n -o yaml > /tmp/.yaml + grep -n -E 'label_values|expr|query' /tmp/.yaml + ``` + +2. For every variable used by the empty panels, run its PromQL part in the Prometheus page or vmui: for a `label_values(,
" +``` + +> **Warning:** A shorter TTL makes ClickHouse delete slow-query rows older than the new period during background merges; deleted history cannot be recovered. Only when the space is confirmed to be in the slow-query tables listed in step 1 and older history is no longer needed. Rolling back to the previous TTL only keeps future data longer. MySQL PXC is not affected. + +```bash +kubectl exec -n chi-pxc-ck-slowsql-cluster-0-0-0 -- clickhouse-client --query \ + "ALTER TABLE .
MODIFY TTL + INTERVAL DAY" +``` + +3. Repeat step 1 over the following days. Space is released as old parts are merged away, not at the moment of the change. + +## Diagnostic Steps + +1. Separate an OOM from a pre-check finding. OOM is a restarting Pod; the pre-check finding is a Service type, even when the Pod is healthy. + +```bash +kubectl get pods -n cpaas-system | grep -E 'pxc-ck|ckslow' +kubectl get svc clickhouse-pxc-ck -n cpaas-system +kubectl describe pod -n | grep -E 'OOMKilled|Limits|Restart' +``` + +2. Read the limit actually enforced. Cause 1 applies when the OOM-killed container shows `2Gi` or `2G`. With any other limit, 4Gi is not the fix; size from actual usage and free node memory. + +3. Before an uninstall, list the operator subscription and the PVCs so you delete the slow-query volume and not a MySQL PXC volume: + +```bash +kubectl get csv,sub,pvc -n cpaas-system | grep -Ei 'clickhouse|pxc-ck|slowsql' +``` + +High memory without `OOMKilled` while `kubectl get mysql -A` is empty is Cause 4: scale to 0, do not delete the PVC. + +4. If the OOM-killed ClickHouse Pod runs in an application namespace and is not the slow-query store, the slow-query causes do not apply. Confirm `OOMKilled` or exit code `137` (a container whose last state is `Error` without `OOMKilled` was not killed for memory, and more memory is not its fix), then raise the memory above the current limit on its `ClickHouseInstallation` as in the Cause 1 resolution, steps 3 and 4. Do not scale it to 0 or delete its PVC for an OOM. + +```bash +kubectl get pod -n \ + -o jsonpath='{range .status.containerStatuses[*]}{.name}{" reason="}{.lastState.terminated.reason}{" exit="}{.lastState.terminated.exitCode}{"\n"}{end}' +``` + +5. Disk growth without OOM or high memory, with `value.bin` files taking the space, is Cause 5: + +```bash +kubectl exec -n chi-pxc-ck-slowsql-cluster-0-0-0 -- sh -c 'find /var/lib/clickhouse -name value.bin -size +100M | head' +``` diff --git a/docs/en/solutions/PostgreSQL_instance_abnormal_disk_full_WAL_buildup_replica_cannot_rejoin_or_writes_fail_on_ACP.md b/docs/en/solutions/PostgreSQL_instance_abnormal_disk_full_WAL_buildup_replica_cannot_rejoin_or_writes_fail_on_ACP.md new file mode 100644 index 000000000..81f6608e6 --- /dev/null +++ b/docs/en/solutions/PostgreSQL_instance_abnormal_disk_full_WAL_buildup_replica_cannot_rejoin_or_writes_fail_on_ACP.md @@ -0,0 +1,230 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Application Services +ProductsVersion: + - "3.18" + - "4.1" + - "4.3" +--- + +# PostgreSQL instance abnormal: disk full, WAL buildup, replica cannot rejoin, or writes fail on ACP + +## Issue + +A Data Services PostgreSQL instance is reported abnormal. The custom resource is `postgresql` (`acid.zalan.do/v1`), and the members are Patroni-managed Pods `-0` and `-1`. One of these is visible: + +- The PostgreSQL log says the disk is full: `OSError: [Errno 28] No space left on device: '/home/postgres/pgdata/pgroot/data/recovery.conf'`, followed by `WARNING: Postgresql is not running.` (Cause 1) +- The primary cannot serve requests and its data volume is full of WAL; storage dashboards can show the nodes at 99% disk usage although the business data is only megabytes (Cause 2). +- One member Pod is `0/1 Running` while the other is `1/1`, and `patronictl list` shows that member as `Replica` in state `starting`, with an empty `TL` and `Lag in MB` of `unknown`, while the other member is the running `Leader`. It looks like split-brain (Cause 3). +- Containers of the instance do not start, with `SMT Alignment Error: requested 1 cpus not multiple cpus per core = 2` (Cause 4). +- SQL statements fail and the application cannot write data (Cause 5). +- After its host node hung and was rebooted, one replica does not start while the primary runs normally; its log shows that `pg_controldata` failed and `/home/postgres/pgdata/pgroot/data/global/pg_control` could not be read (Cause 6). +- A replica is `stopped` in `patronictl list` and its log repeats `data dir for the cluster is not empty, but system ID is invalid; consider doing reinitialize`. Data Services can show the instance as `Error` while its Pods are `Running`, or the upgrade checklist reports PostgreSQL abnormal (Cause 7). + +## Root Cause + +### Cause 1: The data volume is full + +The PV that backs the PostgreSQL data directory has no free space. The `postgres` container of the member logs: + +```text +INFO: Lock owner: None; I am -1 +INFO: starting as a secondary +ERROR: Exception during execution of long running task restarting after failure +... + in write_recovery_conf +OSError: [Errno 28] No space left on device: '/home/postgres/pgdata/pgroot/data/recovery.conf' +WARNING: Postgresql is not running. +``` + +followed by `Database cluster state: shutting down`. Inside the Pod, `df -h` shows the data volume on `/home/postgres/pgdata` at 100% (for example Size `25G`, Used `25G`, Avail `104K`), while `sv status` still lists `cron`, `patroni`, and `pgqd` as `run`. Patroni cannot write `recovery.conf`, so PostgreSQL on that member does not start. Expand the PV, or delete data that is no longer needed. If the space is taken by WAL rather than by tables, see Cause 2. + +### Cause 2: WAL fills the primary because replication has been broken for a long time + +When primary-replica replication fails for a long time, the primary keeps every WAL segment the replica has not received. The WAL fills the primary's disk and the primary can no longer serve. Slow disks make it worse, because the replica falls behind faster than it can replay. Recovery: expand the primary PVC, rebuild the replica, and let the primary drop the old WAL once the rebuilt replica is streaming. + +On TopoLVM storage, the reported disk usage is the capacity already allocated to volumes, not what the data inside them uses. Cleaning other data (for example Elasticsearch indices) does not lower that figure. + +### Cause 3: Replica stays starting with Lag unknown after the WAL it needs is lost + +`patronictl list` shows the replica `starting` with no timeline and `Lag in MB` `unknown`, and the leader `running` on its timeline. The WAL the replica needs to catch up is no longer available (for example the WAL archive was lost), so it cannot heal on its own. `patronictl reinit` of that member from the leader recovers it. + +### Cause 4: SMT Alignment Error: requested 1 cpus not multiple cpus per core = 2 + +The nodes have NUMA enabled, and the kubelet CPU Manager and Topology Manager policies require SMT alignment. Under that policy an exclusive CPU request must be a multiple of the CPUs per physical core (2 on these nodes). The PostgreSQL container requests 1 CPU, and the kubelet rejects it at start. Setting the CPU resources to an even value fixes it. The rule applies to any Data Services Pod scheduled on those nodes; see the ACP 4.3 documentation on CPU Manager and Topology Manager policies. + +### Cause 5: SQL writes fail because the primary/standby switchover is stuck + +SQL statements against the instance fail and no data can be written because a primary/standby switchover is stuck. Restarting the PostgreSQL Pod of the instance restores writes. What leaves the switchover stuck has not been identified. + +### Cause 6: Replica cannot start because global/pg_control cannot be read after a node crash + +The host node became unresponsive (for example because of an operating system kernel bug). After the node was fixed and rebooted, the primary ran normally and the replica did not. The replica's log shows `pg_controldata` failing because `global/pg_control` cannot be read: the control file of that member was damaged by the crash. Resynchronizing the replica from the primary, by deleting the replica's PVC and Pod, returns it to normal. + +### Cause 7: Replica stopped with "data dir for the cluster is not empty, but system ID is invalid" + +The replica's `postgres` container repeats `INFO: data dir for the cluster is not empty, but system ID is invalid; consider doing reinitialize`. `patronictl list` shows it as `Replica`, state `stopped`, `Lag in MB` `unknown`, while the leader is `running` and logs `no action. I am (-0), the leader with the lock`. Patroni does not start PostgreSQL on a non-empty data directory whose system identifier it cannot read, so the replica never rejoins. The instance page can show `could not sync statefulsets: could not recreate pods: skipping switchover: no switchover candidate found`. Bootstrap lines such as `chmod: changing permissions of '/var/log/pg_log': Operation not permitted` and `Disabling 'bypass_api_service'` are not related. + +`patronictl reinit` of the replica can return `Success` while the member stays in `creating replica` with Lag `unknown`. Scaling the instance from 2 members to 1 and back to 2, which recreates the replica from the leader, restores the cluster. + +## Resolution + +### Cause 1: The data volume is full + +```bash +kubectl get pvc -n +kubectl exec -n -0 -c postgres -- df -h /home/postgres/pgdata +kubectl get sc -o jsonpath='{.allowVolumeExpansion}{"\n"}' +``` + +> **Warning:** A PVC can be expanded but never shrunk. Precondition: `allowVolumeExpansion` is `true`, the storage backend has the capacity, and you expand the PVC of the member whose disk is full. There is no rollback for the size; a smaller volume needs a new instance and a data migration. + +Raise the volume size of the instance and of the full claim to the same value, so the operator does not work against the new size: + +```bash +kubectl patch postgresql -n --type=merge -p '{"spec":{"volume":{"size":""}}}' +kubectl patch pvc -n -p '{"spec":{"resources":{"requests":{"storage":""}}}}' +kubectl get pvc -n -w +``` + +If the data owner prefers to free space instead, delete data that is no longer needed through SQL. + +### Cause 2: WAL fills the primary because replication has been broken for a long time + +Find the leader, how far the replica is behind, and how much of the disk is WAL: + +```bash +kubectl exec -n -0 -c postgres -- patronictl list +kubectl exec -n -c postgres -- du -sh /home/postgres/pgdata/pgroot/data/pg_wal +kubectl exec -n -c postgres -- psql -U postgres -c 'select slot_name, active, restart_lsn from pg_replication_slots' +``` + +A large `pg_wal`, a replica far behind or not streaming, and an inactive replication slot for that replica fit this cause. Expand the leader PVC first (Cause 1) so the leader can serve while the replica is rebuilt. Then rebuild the replica by deleting its PVC and then its Pod; the StatefulSet creates both again and the member clones itself from the leader. + +> **Warning:** Deleting the replica PVC destroys that member's copy of the database. The new member copies the full data set from the leader, which loads the leader's disk and network until it finishes. Precondition: `patronictl list` shows the Pod you delete as `Replica`, not `Leader`; the leader is `running`; a backup of the leader exists. The PVC delete is irreversible. Run it only against the replica. + +```bash +kubectl get pod -n -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}{"\n"}' +kubectl delete pvc -n --wait=false +kubectl delete pod -n +kubectl get pod,pvc -n -w +``` + +When `patronictl list` shows the rebuilt member `streaming` (or `running` with a small lag), the leader can remove the old WAL. Check `du -sh` on `pg_wal` again; the space is freed after the next checkpoints. + +### Cause 3: Replica stays starting with Lag unknown after the WAL it needs is lost + +```bash +kubectl exec -n -0 -c postgres -- patronictl list +kubectl logs -n -c postgres --tail=100 +``` + +Reinitialize only the member that shows `Replica` / `starting` / `unknown`, while the other member is `Leader` / `running`. + +> **Warning:** `patronictl reinit` erases the data directory of the named member and copies it again from the leader. Anything that existed only on that member is lost. Precondition: the named member is the replica, the leader is running and serving, and a backup of the leader exists. Irreversible for the member's local data. Name only the replica member. + +```bash +kubectl exec -it -n -c postgres -- patronictl reinit +``` + +The command can run from any member Pod, including the `0/1` replica itself; Patroni asks for confirmation. Watch `patronictl list` until the member shows the leader's timeline and a numeric lag. If it stays in `creating replica` with Lag `unknown`, or its log says `system ID is invalid`, continue with Cause 7. + +### Cause 4: SMT Alignment Error: requested 1 cpus not multiple cpus per core = 2 + +```bash +kubectl get pod -n -o jsonpath='{.status.reason}{" "}{.status.message}{"\n"}' +kubectl get pod -n -o jsonpath='{range .spec.containers[*]}{.name}{" "}{.resources}{"\n"}{end}' +``` + +On the node named in the Pod spec: + +```bash +grep -i -E 'cpuManagerPolicy|cpuManagerPolicyOptions|full-pcpus-only|topologyManagerPolicy' -A2 /var/lib/kubelet/config.yaml +``` + +> **Warning:** Changing the instance CPU recreates its Pods; each member restarts once and the leader can fail over. Precondition: the nodes have enough free CPU for the even value, and the application owner accepts a short disconnect. Rollback is setting the previous value again, which brings the error back on these nodes. + +Set the CPU request and limit of the instance to an even number (for example `2`) with the instance Update action in Data Services. + +### Cause 5: SQL writes fail because the primary/standby switchover is stuck + +```bash +kubectl exec -n -0 -c postgres -- patronictl list +kubectl logs -n -0 -c postgres --tail=100 +kubectl exec -n -c postgres -- psql -U postgres -c 'select pg_is_in_recovery()' +``` + +No member `running` as `Leader`, a pending switchover, or a leader that still answers `t` to `pg_is_in_recovery()` while writes fail fits this cause. + +> **Warning:** Deleting a PostgreSQL Pod restarts that member and clients see a disconnect; if it holds the leader lock, Patroni elects a leader again. Precondition: this cause is confirmed with `patronictl list`. Delete Pods only, never PVCs and never the `postgresql` resource. No rollback is needed; the StatefulSet recreates the Pod on its existing PVC. + +```bash +kubectl delete pod -n +kubectl exec -n -0 -c postgres -- patronictl list +``` + +Retry a write from the application. If writes still fail, collect `patronictl list` and the `postgres` logs of both members before any further restart. + +### Cause 6: Replica cannot start because global/pg_control cannot be read after a node crash + +```bash +kubectl exec -n -c postgres -- patronictl list +kubectl logs -n -c postgres --tail=200 | grep -i -E 'pg_control|pg_controldata' +kubectl get pod -n -o jsonpath='{.spec.nodeName}{" "}{.spec.volumes[*].persistentVolumeClaim.claimName}{"\n"}' +``` + +The node that hung must be healthy again (`Ready`, the kernel issue fixed), and the primary must be `Leader` and `running`. + +> **Warning:** Deleting the replica's PVC permanently removes its data directory, including the damaged `pg_control`. The rebuilt member copies the whole database from the primary, which loads the primary until the copy finishes. Precondition: the `pg_control` read failure is confirmed on a member that `patronictl list` shows as the replica, never on the leader; the leader is `running` and a backup of it exists. Irreversible for the replica's local data. + +```bash +kubectl delete pvc -n --wait=false +kubectl delete pod -n +kubectl get pod,pvc -n -w +``` + +When the new member shows `streaming` with the leader's timeline, the replica is resynchronized. If the leader itself reports the `pg_control` error, this procedure does not apply; collect both logs and contact support. + +### Cause 7: Replica stopped with "data dir for the cluster is not empty, but system ID is invalid" + +```bash +kubectl logs -n -c postgres --tail=200 | grep 'system ID is invalid' +kubectl exec -n -c postgres -- patronictl list +kubectl get postgresql -n -o yaml > -before.yaml +``` + +Continue only when the member that logs the message is `Replica` / `stopped` and the other is `Leader` / `running`. First run the Cause 3 `patronictl reinit` against that replica. If it stays in `creating replica` with Lag `unknown`, scale the instance to 1 member and back to 2, with the instance Update action in Data Services or on the resource. + +> **Warning:** Scaling to 1 member removes the highest-ordinal Pod (`-1`) and leaves the instance without a replica until it is scaled back; if that Pod held the leader lock, Patroni fails over. Precondition: `patronictl list` shows `-1` as the `stopped` replica and `-0` as `Leader` / `running`, and a backup of the leader exists. Never scale to 0. Rollback is setting `numberOfInstances` back to its previous value. If the broken replica is not the highest-ordinal Pod, scaling down removes the wrong member; use the Cause 6 replica rebuild instead. + +```bash +kubectl patch postgresql -n --type=merge -p '{"spec":{"numberOfInstances":1}}' +kubectl get pod,pvc -n -w +kubectl patch postgresql -n --type=merge -p '{"spec":{"numberOfInstances":2}}' +kubectl exec -n -0 -c postgres -- patronictl list +``` + +Patch back to 2 only after `-1` is gone. If the new `-1` mounts its old PVC again and returns with the same log, use the Cause 6 replica PVC procedure (replica only). The member is healthy when it shows `streaming` or `running` with the leader's timeline and a numeric lag; then rerun the upgrade checklist. + +## Diagnostic Steps + +```bash +kubectl get postgresql,pods,pvc -n -o wide +kubectl exec -n -0 -c postgres -- patronictl list +kubectl exec -n -0 -c postgres -- df -h /home/postgres/pgdata +kubectl get events -n --sort-by=.lastTimestamp | tail -n 30 +``` + +| Signature | Cause | +| --- | --- | +| `No space left on device` on `recovery.conf`, `df` at 100%, no WAL backlog | Cause 1 | +| `pg_wal` large on the leader, replica far behind or not streaming | Cause 2 | +| Replica `starting`, empty TL, `Lag in MB` `unknown`, leader `running` | Cause 3 | +| `SMT Alignment Error: requested 1 cpus not multiple cpus per core = 2` | Cause 4 | +| Writes fail, switchover stuck in `patronictl list` | Cause 5 | +| Replica log: `pg_controldata` fails, `global/pg_control` unreadable, after its node crashed | Cause 6 | +| Replica `stopped`, Lag `unknown`, log `system ID is invalid; consider doing reinitialize` | Cause 7 | + +If a member's container is not running, `patronictl list` still works from the other member. If both members are down because both disks are full, expand both PVCs before any rebuild. diff --git a/docs/en/solutions/Shutting_down_or_rebooting_ACP_nodes_for_maintenance.md b/docs/en/solutions/Shutting_down_or_rebooting_ACP_nodes_for_maintenance.md new file mode 100644 index 000000000..d486044f3 --- /dev/null +++ b/docs/en/solutions/Shutting_down_or_rebooting_ACP_nodes_for_maintenance.md @@ -0,0 +1,228 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - 3.x + - 4.x +--- + +# Shutting down or rebooting ACP nodes for maintenance + +## Issue + +Typical questions before planned work on nodes: + +- "We need to reboot some worker nodes. Can we just reboot the servers?" +- "A server's RAID card or memory failed and must be shut down for replacement. What must the platform side do?" +- "One global (management) server needs a memory replacement. What is the shutdown plan, how do we back up etcd, and what do we check after the restart?" +- "A node of the logging cluster hosts Kafka, ZooKeeper and Elasticsearch and will be shut down for repair. Do the logging components need any preparation?" +- "We will upgrade the kernel, or the OS (for example Kylin SP2 to SP3). What must be done before the reboot?" +- "The IaaS will migrate all VMs for a hardware expansion, with short interruptions on every node. Will the cluster be affected?" +- "After patching we rebooted the nodes; the Pods on each node were evicted and recreated elsewhere, so replicas are now unevenly spread. Does the platform taint the node during a reboot, and can the eviction time be controlled?" +- "All three masters of a test environment restarted at the same time. Nodes look normal; is anything else needed?" + +These procedures take nodes down one at a time while the rest of the cluster keeps running. Shutting down every node of the platform at once (a planned power outage of the whole site) is not covered here. + +Related articles: *Node capacity, pod limit or data disk after resizing or replacing node hardware on ACP* (capacity that does not update after a resize, `maxPods`, and replacing a data disk), and *Safely resizing CPU and memory on Alauda Container Platform nodes*. + +## Root Cause + +These are planned operations, not faults. The procedures exist for three reasons: + +- **Pods move only if they are evicted first.** If a node is rebooted without preparation, its Pods stay bound to it and are unavailable until it returns. Pods pinned to the node (`nodeSelector`, local volumes) never move and are unavailable while it is down. +- **Control-plane and global nodes hold etcd and platform components.** Losing one member of a 3-node etcd keeps quorum, but the data has to be protected against a second failure during the work. +- **Node failures are handled with taints.** About 40 seconds after a node stops reporting, kube-controller-manager adds the `node.kubernetes.io/not-ready` or `node.kubernetes.io/unreachable` taint with effect `NoExecute`. The platform does not add any taint of its own during a reboot. + +### Cause 1: A worker node is rebooted or shut down + +The node's business Pods have to be evicted first so they run elsewhere during the work. Without preparation, Pods are deleted when their toleration for the `NoExecute` taint expires (300 seconds by default in upstream Kubernetes, unless the Pod sets its own) and replaced on other nodes; they are not moved back afterwards, which leaves replicas unevenly spread. Replacement Pods that cannot be placed show `FailedScheduling ... node(s) had untolerated taint {node.kubernetes.io/unreachable: }`. + +### Cause 2: A control-plane node or a global-cluster node is shut down + +etcd runs on the control-plane nodes. With 3 control-plane nodes, one can be down while etcd keeps quorum; an etcd backup protects against a second failure. While a global node is down, platform pages, log and monitoring collection, and components fixed to that node can be affected; business services already running in business clusters are generally not affected. + +### Cause 3: A node of the logging cluster (Kafka, ZooKeeper, Elasticsearch) is shut down + +The logging components on that node have to be stopped before the node goes down, whether the Elasticsearch instance on it is a data node or a master node. + +### Cause 4: Kernel or OS upgrade with reboots + +Each node is rebooted, so the worker procedure applies node by node, after an etcd backup. There is no platform-specific OS upgrade procedure; the OS upgrade is carried out by the user, and the target OS version must be one the platform supports. + +### Cause 5: IaaS hardware expansion or VM migration + +Whether preparation is needed depends on whether the cluster runs business workloads at all. Decide this from what is deployed, not from the cluster's name or type: a new business cluster can still hold only platform base components. + +## Resolution + +### Cause 1: A worker node is rebooted or shut down + +Work on **one node at a time**; start the next node only when the previous one is `Ready` and its Pods are running. + +1. Check that the remaining nodes can absorb the node's Pods: + +```bash +kubectl describe nodes | grep -A8 'Allocated resources' +kubectl get pdb -A +``` + +2. Mark the node unschedulable and evict its Pods. In the console this is the node's **Evict Pods** action; from the CLI: + +> **Warning:** Eviction restarts every non-DaemonSet Pod of the node on other nodes, and `--delete-emptydir-data` deletes their `emptyDir` data. Single-replica services are briefly unavailable, and Pods pinned to this node stay down until it returns. Precondition: the other nodes have room, and PodDisruptionBudgets allow the eviction. Roll back with `kubectl uncordon `; evicted Pods do not move back by themselves. + +```bash +kubectl cordon +kubectl drain --ignore-daemonsets --delete-emptydir-data --timeout=10m +``` + +3. Reboot, or shut down and do the hardware work (RAID controller, memory, VM specification): + +```bash +systemctl reboot # or: shutdown -h now +``` + +4. When the node is back and `Ready`, re-enable scheduling and check for Pods that are not running: + +```bash +kubectl get node +kubectl uncordon +kubectl get pods -A -o wide | grep -viE 'Running|Completed' +``` + +Platform components on a worker need no extra handling; they recover by themselves after the restart. Rebooting directly without cordon and eviction is not recommended on clusters that carry business workloads. + +**Keeping Pods on a node through a short reboot.** To keep Pods bound to the node for up to 15 minutes while it reboots, so that only their restart count increases, set both tolerations in each workload's Pod template: + +```yaml +spec: + tolerations: + - key: "node.kubernetes.io/not-ready" + operator: "Exists" + effect: "NoExecute" + tolerationSeconds: 900 + - key: "node.kubernetes.io/unreachable" + operator: "Exists" + effect: "NoExecute" + tolerationSeconds: 900 +``` + +> **Warning:** Changing the Pod template rolls out new Pods for the workload. A longer toleration also delays replacement when a node really fails: its Pods stay unavailable for up to `tolerationSeconds` before they are recreated elsewhere. Agree the value with the application owner and apply it in a change window. Roll back by removing the tolerations or restoring the previous value. + +```bash +kubectl get pod -n -o jsonpath='{range .spec.tolerations[*]}{.key}{" "}{.effect}{" "}{.tolerationSeconds}{"\n"}{end}' +``` + +On clusters whose node configuration is managed by Machine Configuration, a node left cordoned also keeps its machine configuration pool from reporting all machines ready: a node counts as ready only when it is `NodeReady=True` and not unschedulable. For example, a cordoned master with its configuration applied shows `MachineCount = 3`, `UpdatedMachineCount = 3`, `ReadyMachineCount = 2`, with no errors in the controller logs. That is expected; uncordon the node when its maintenance is over: + +```bash +kubectl get nodes | grep SchedulingDisabled +kubectl uncordon +``` + +### Cause 2: A control-plane node or a global-cluster node is shut down + +1. Confirm that the cluster has 3 control-plane nodes, and shut down only one at a time. +2. Back up etcd first. Use the platform's etcd backup feature if it is configured (see the etcd backup documentation for your ACP version), and take a manual snapshot from the etcd static Pod. Read the certificate paths and data directory from the Pod spec instead of guessing them: + +```bash +kubectl -n kube-system get pods -l component=etcd -o wide +kubectl -n kube-system get pod etcd- -o yaml | grep -E -- '--(cert-file|key-file|trusted-ca-file|data-dir)=' +kubectl -n kube-system exec etcd- -- etcdctl \ + --endpoints=https://127.0.0.1:2379 \ + --cacert= --cert= --key= \ + snapshot save /snapshot-before-maintenance.db +``` + +Copy the snapshot file off the node (it is under the data directory on the host) and confirm its size is not zero. + +3. Shut the node down normally: + +```bash +shutdown -h now +``` + +4. After the repair and power-on, check that the node is `Ready` and the platform Pods on it are healthy, then handle any component that does not recover. On a node that runs Ceph, also check that Ceph has returned to normal. + +```bash +kubectl get nodes +kubectl -n kube-system get pods -o wide | grep +kubectl get pods -A -o wide | grep | grep -viE 'Running|Completed' +``` + +### Cause 3: A node of the logging cluster (Kafka, ZooKeeper, Elasticsearch) is shut down + +Stop the logging components first, then shut the node down for repair. Cordon and drain are the worker procedure for business workloads, not part of this step. To confirm the Elasticsearch role or cluster health before the shutdown, use read-only checks only. After the repair and power-on, confirm the node is `Ready` and the logging components on it are running again: + +```bash +kubectl get node +kubectl get pods -A -o wide | grep | grep -viE 'Running|Completed' +``` + +### Cause 4: Kernel or OS upgrade with reboots + +Back up etcd first (Cause 2). Then: + +**Business can be stopped:** + +1. Stop the business containers. +2. Upgrade and reboot the compute nodes one by one, confirming each node is normal before the next. +3. Upgrade the control-plane nodes one by one. +4. Start the business. + +**Business cannot be stopped:** + +1. For each compute node: cordon it, evict the business Pods (Cause 1), and confirm the business has recovered on other nodes. +2. Upgrade the kernel and reboot. Confirm the node has rejoined normally, then uncordon it. +3. Repeat for the other compute nodes, then upgrade the control-plane nodes one by one. + +Schedule an OS version upgrade in a coordinated maintenance window, with the same node-by-node approach. + +### Cause 5: IaaS hardware expansion or VM migration + +Ask the cluster owner, and list what runs outside the platform's own namespaces (read-only): + +```bash +kubectl get ns +kubectl get deploy,sts -A +``` + +- **No business workloads** (only platform base components): there is no significant risk. The expansion and VM migration can go ahead directly, and the hosts can simply be rebooted. Wait a while after the reboot, then list the Pods that are not running: + +```bash +kubectl get po -A | grep -vi run +``` + +- **Business workloads**: migrate or restart nodes one at a time with the Cause 1 procedure, for example shutting each node down, giving it a new VM specification or moving it to another resource pool, and restarting it, with each node `Ready` before the next. + +After any VM migration, check the container runtime on each node; a migration can corrupt the runtime data directory and runtime binaries. + +## Diagnostic Steps + +Before maintenance: + +```bash +kubectl get nodes -o wide +kubectl get pods -A -o wide --field-selector spec.nodeName= +kubectl get pdb -A +kubectl -n kube-system get pods -l component=etcd -o wide +``` + +Pods with a local PV, or a `nodeSelector` or `nodeName` for this node, will not move; plan their downtime. Fewer than 3 etcd members means no control-plane node can be shut down safely. + +After maintenance: + +```bash +kubectl get node +kubectl get pods -A | grep -vi run +``` + +After an unplanned restart of all masters at once, the same check is the first step. Periodic job Pods in `cpaas-system` (names such as `--`) in state `0/1 Completed` with 0 restarts are normal; with no remaining alerts, no further action is needed. + +If a node gets stuck during `shutdown`, power it on again, check that the system time is right with `date`, and collect the shutdown logs for analysis: + +```bash +journalctl --since "" > ~/shutdown-log.log +ls -lh ~/shutdown-log.log +``` From a495c2c9a818c9f42caec05008e3645040c76690 Mon Sep 17 00:00:00 2001 From: Komh Date: Wed, 18 Nov 2026 17:40:46 +0800 Subject: [PATCH 14/25] docs(solutions): add 14 and update 0 troubleshooting articles --- ...r_init.sh_or_a_cloned_node_fails_at_yum.md | 97 +++++++ ...s_IO_error_or_missing_log_secret_on_ACP.md | 176 +++++++++++++ ...lled_or_etcd-backup-default_not_created.md | 94 +++++++ ...01_architecture_or_search_domain_on_ACP.md | 240 ++++++++++++++++++ ...aising_DisabledCopiedCSVs_events_on_ACP.md | 108 ++++++++ ...the_NFS_plugin_or_node_isolation_on_ACP.md | 156 ++++++++++++ ...init_script_DefaultTasksMax_root_access.md | 182 +++++++++++++ ..._blank_after_a_Pipelines_upgrade_on_ACP.md | 107 ++++++++ ...architectures_can_run_in_an_ACP_cluster.md | 115 +++++++++ ...y_standby_global_pair_replicates_on_ACP.md | 152 +++++++++++ ...rt_on_the_standby_global_cluster_on_ACP.md | 168 ++++++++++++ ...te_copies_or_retention_to_change_on_ACP.md | 172 +++++++++++++ ..._502_or_invalid_reference_format_on_ACP.md | 220 ++++++++++++++++ ..._or_violet_or_kubectl_is_missing_on_ACP.md | 124 +++++++++ 14 files changed, 2111 insertions(+) create mode 100644 docs/en/solutions/Adding_a_node_to_an_ACP_cluster_fails_on_UseDNS_or_UsePAM_the_node_does_not_boot_after_init.sh_or_a_cloned_node_fails_at_yum.md create mode 100644 docs/en/solutions/Cluster_plugin_stuck_at_wait_resources_ready_Pod_Pending_cpaas_IO_error_or_missing_log_secret_on_ACP.md create mode 100644 docs/en/solutions/No_default_etcd_backup_on_ACP_Cluster_Enhancer_not_installed_or_etcd-backup-default_not_created.md create mode 100644 docs/en/solutions/Operator_or_plugin_ImagePullBackOff_or_exec_format_error_registry_address_401_architecture_or_search_domain_on_ACP.md create mode 100644 docs/en/solutions/Operator_stays_Processing_base-operator_logs_certwatch_errors_or_a_CSV_keeps_raising_DisabledCopiedCSVs_events_on_ACP.md create mode 100644 docs/en/solutions/Pre-upgrade_checklist_reports_a_CIDR_mismatch_a_missing_is-global_label_the_NFS_plugin_or_node_isolation_on_ACP.md create mode 100644 docs/en/solutions/Preparing_a_host_before_adding_it_to_an_ACP_cluster_node_check_items_disks_init_script_DefaultTasksMax_root_access.md create mode 100644 docs/en/solutions/TektonHub_cannot_be_viewed_or_the_pipeline_task_page_is_blank_after_a_Pipelines_upgrade_on_ACP.md create mode 100644 docs/en/solutions/Which_operating_systems_kernels_and_CPU_architectures_can_run_in_an_ACP_cluster.md create mode 100644 docs/en/solutions/etcd-sync_check_reports_localMissedKeys_or_localSurplusKeys_or_what_a_primary_standby_global_pair_replicates_on_ACP.md create mode 100644 docs/en/solutions/etcd_Synchronizer_plugin_install_stuck_or_etcd-sync_does_not_start_on_the_standby_global_cluster_on_ACP.md create mode 100644 docs/en/solutions/etcd_backup_dates_missing_in_the_console_backup_tab_gone_duplicate_copies_or_retention_to_change_on_ACP.md create mode 100644 docs/en/solutions/violet_push_fails_with_no_package_found_code_000_HTTP_500_or_502_or_invalid_reference_format_on_ACP.md create mode 100644 docs/en/solutions/violet_push_goes_to_the_wrong_global_cluster_a_wrong_package_cannot_be_deleted_or_violet_or_kubectl_is_missing_on_ACP.md diff --git a/docs/en/solutions/Adding_a_node_to_an_ACP_cluster_fails_on_UseDNS_or_UsePAM_the_node_does_not_boot_after_init.sh_or_a_cloned_node_fails_at_yum.md b/docs/en/solutions/Adding_a_node_to_an_ACP_cluster_fails_on_UseDNS_or_UsePAM_the_node_does_not_boot_after_init.sh_or_a_cloned_node_fails_at_yum.md new file mode 100644 index 000000000..024fc0a22 --- /dev/null +++ b/docs/en/solutions/Adding_a_node_to_an_ACP_cluster_fails_on_UseDNS_or_UsePAM_the_node_does_not_boot_after_init.sh_or_a_cloned_node_fails_at_yum.md @@ -0,0 +1,97 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Adding a node to an ACP cluster fails on UseDNS or UsePAM, the node does not boot after init.sh, or a cloned node fails at yum + +## Issue + +- The availability check of a new node fails with `节点可用性检查未通过,无法添加节点到集群下` (the node availability check did not pass; the node cannot be added to the cluster) together with `检查sshd配置:/etc/ssh/sshd_config文件中 UseDNS 参数被注释或值不为 no,请调整` (sshd check: the `UseDNS` parameter in `/etc/ssh/sshd_config` is commented out or not `no`; adjust it). +- Adding a node succeeds only sometimes. The node's secure log shows most platform logins as `Accepted password for ... ssh2` and some as `Failed password for ... ssh2` followed by `Connection closed by authenticating user ... [preauth]`. +- A new workload-cluster node runs the initialization script and then does not boot after a reboot. Without the script it reboots normally, and `lsblk` and `/etc/fstab` look fine. +- Adding a node that was cloned from an existing node fails at the task `[core/bootstrap-os : Installing basic packages using yum]` with `"msg": "MODULE FAILURE"` and `Install failed!`, and the traceback ends with: + + ```text + shutil.Error: Destination path '/etc/yum.repos.d/temp/temp_repo_0.repo' already exists + ``` + +Related articles: "Preparing a host before adding it to an ACP cluster: node check items, disks, init script, DefaultTasksMax, root access" covers what to prepare; "Which operating systems, kernels, and CPU architectures can run in an ACP cluster" covers an add that fails with an unsupported operating system. + +## Root Cause + +### Cause 1: UseDNS or UsePAM in sshd_config is commented out or not no + +The platform adds nodes through automated SSH logins. The availability check requires `UseDNS no` and inspects `UsePAM`. With `UseDNS` commented out or not `no`, the check fails and the node is not added; after `UseDNS no` is set, the node is added. + +Intermittent `Failed password` entries for the platform's logins, while most logins of the same user succeed, stop after `UsePAM` is set to `no`, and the checks then pass every time. Because a commented or overridden line in `sshd_config` is not what sshd actually uses, check the effective values with `sshd -T`, not only the file. + +### Cause 2: the cgroup nokmem step of the init script stops the node from booting + +The `config cgroup nokmem` step of the initialization script writes kernel parameters into the grub configuration. On some hosts those parameters stop the node from booting. Commenting out that section of the script before running it lets the node be initialized and rebooted normally. + +### Cause 3: a cloned node carries the yum repository backup of the original node + +The yum task backs up the existing repository files by moving them into `/etc/yum.repos.d/temp/`. A machine cloned from an existing node (even after a cleanup script from another environment was run on it) still holds state from the original node, including `/etc/yum.repos.d/temp/temp_repo_0.repo`. The move then fails because the destination exists, and the install stops. The output can also show `setlocale: LC_ALL: cannot change locale (C.UTF-8)` warnings, which are not the failure. A freshly installed machine is added normally. + +## Resolution + +### Cause 1: UseDNS or UsePAM in sshd_config is commented out or not no + +1. Read the file and the effective values: + + ```bash + sudo grep -n -E '^\s*#?\s*(UsePAM|UseDNS)' /etc/ssh/sshd_config + sudo sshd -T | grep -i -E '^(usepam|usedns) ' + ``` + +2. Set `UseDNS no` and `UsePAM no` explicitly in `/etc/ssh/sshd_config`. + + > **Warning:** A syntax error in `sshd_config` can stop sshd from accepting new logins after the restart. Keep the current session open, back the file up first, and validate it before restarting. Rollback: restore the backup and restart sshd. + + ```bash + sudo cp -a /etc/ssh/sshd_config /root/sshd_config.bak + sudo vi /etc/ssh/sshd_config + sudo sshd -t && sudo systemctl restart sshd + sudo sshd -T | grep -i -E '^(usepam|usedns) ' + ``` + + Expected: `usedns no` and `usepam no`. + +3. Run the availability check and add the node again. + +### Cause 2: the cgroup nokmem step of the init script stops the node from booting + +1. Before running the init script on such a node, find its `config cgroup nokmem` section: + + ```bash + grep -n -i 'nokmem' init.sh + ``` + +2. Comment out that section, then run the script with `bash`. +3. For a node that already fails to boot, compare its kernel command line and grub configuration with a node that boots, and remove the parameters that the step added: + + ```bash + cat /proc/cmdline # on a node that boots, for comparison + ``` + +### Cause 3: a cloned node carries the yum repository backup of the original node + +1. Check the node for the leftover backup: + + ```bash + ls -la /etc/yum.repos.d/ /etc/yum.repos.d/temp/ 2>/dev/null + ``` + +2. Replace the clone with a freshly installed machine and add that machine instead. + +## Diagnostic Steps + +1. Read the failed item of the availability check in the console. + - `UseDNS 参数被注释或值不为 no`: Cause 1. + - Intermittent add failures: read the node's secure log (`/var/log/secure`, or `/var/log/auth.log` on Ubuntu) for `Failed password` from the platform's address next to `Accepted password` for the same user: Cause 1. +2. A node that boots without the init script and does not boot after it: check whether the script contains a `nokmem` step (`grep -n -i nokmem init.sh`): Cause 2. +3. `Installing basic packages using yum` fails with `temp_repo_0.repo' already exists`, and `ls /etc/yum.repos.d/temp/` lists files on a cloned machine: Cause 3. diff --git a/docs/en/solutions/Cluster_plugin_stuck_at_wait_resources_ready_Pod_Pending_cpaas_IO_error_or_missing_log_secret_on_ACP.md b/docs/en/solutions/Cluster_plugin_stuck_at_wait_resources_ready_Pod_Pending_cpaas_IO_error_or_missing_log_secret_on_ACP.md new file mode 100644 index 000000000..3328b0b2c --- /dev/null +++ b/docs/en/solutions/Cluster_plugin_stuck_at_wait_resources_ready_Pod_Pending_cpaas_IO_error_or_missing_log_secret_on_ACP.md @@ -0,0 +1,176 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.2" + - "4.3" +--- + +# Cluster plugin stuck at wait resources ready: Pod Pending, /cpaas I/O error, or missing log secret on ACP + +## Issue + +The plugin chart is applied, and the install or upgrade then waits for the plugin's workloads to become ready and never gets there: + +- The multus plugin installation hangs in Deploying. `kubectl describe ars -n cpaas-system multus` shows Reason `NotReady` with `failed to wait chart alauda-multus-plugin-v3.18.3 state, err: timed out waiting for the condition`. +- The Elasticsearch log storage plugin upgrade stays at the `CheckStatus-logcenter` step, and the progress log repeats `WaitReady: wait resources ready`. +- After the 4.3.2 logging packages (`log-agent`, `log-api`, `log-clickhouse`) were pushed, the upgrade button of the logging cluster plugins stays greyed out. Once it appears, the "Alauda Container Platform Log Collector" v4.3.2 upgrade on a business cluster fails (`安装失败`, installation failed) with `wait resources ready`. + +In each case one workload Pod of the plugin cannot become Ready, and the AppRelease (`ars`) waits for it until it times out. A plugin Pod that cannot pull its image is a different problem; see the related article below. + +Related articles: "Operator or plugin ImagePullBackOff or exec format error: registry address, 401, architecture, or search domain on ACP" covers image pull failures; "Cluster plugin install or upgrade stuck in chart sync: sentry, StatefulSet Forbidden, hook Job, or missing ModuleConfig" covers failures before the workloads are created. + +## Root Cause + +### Cause 1: a DaemonSet Pod of the plugin is Pending with Insufficient cpu on one node + +The chart waits for every Pod of its DaemonSets. When one node lacks the CPU that a DaemonSet Pod requests, that Pod stays `Pending` with `Insufficient cpu`, the DaemonSet has one Pod fewer ready than desired (for example `delete-multus-cfg-ds` in `kube-system` at 10/11), and the AppRelease stays in `WaitReady` until it times out. Once CPU is freed on that node, the Pod is scheduled and the installation completes. + +### Cause 2: the log storage Pods on one node fail because /cpaas returns Input/output error + +The log storage components (`cpaas-elasticsearch-`, `cpaas-kafka-`, `cpaas-zookeeper-` in `cpaas-system`) keep their data under `/cpaas` on the node. When the filesystem behind `/cpaas` on one node fails, every log storage Pod scheduled there is `0/1`, and the upgrade never passes `CheckStatus-logcenter` (`prepare-logcenter-upgrade` has already completed). On that node, `ls` in `/cpaas` or `/cpaas/data` returns `ls: cannot open directory .: Input/output error` (or `cannot access 'data': Input/output error`). + +When the I/O error is a transient fault of an LVM-based mount, rebooting the host restores `/cpaas`, and the log storage Pods recover on their own. When the underlying block device or its paths (for example a SAN or multipath device) are failing, a reboot does not help until the storage is repaired. + +### Cause 3: nevermore cannot start, secret log-collector-bearer-auth not found, Log Essentials on global was not upgraded first + +The Log Collector upgrade on the business cluster fails with `wait resources ready`, and the `nevermore` Pods in `cpaas-system` stay `0/4` `Init:0/1` with: + +```text +Warning FailedMount MountVolume.SetUp failed for volume "callback-secret" : secret "log-collector-bearer-auth" not found +``` + +`kubectl get secret -A | grep log-collector` returns nothing on the business cluster. + +The Secret `log-collector-bearer-auth` is distributed to business clusters by the "Alauda Container Platform Log Essentials" plugin on `global`. When Log Essentials on `global` is still on the previous release line (for example v4.2.9) while the business cluster plugins are upgraded to v4.3.2, the Secret does not exist yet. After Log Essentials is upgraded, `log-collector-bearer-auth` and `log-database-bearer-auth` are created in `cpaas-system` on the business cluster, and Log Collector and Log Storage for ClickHouse there finish their upgrade. + +A greyed-out upgrade button after the packages were pushed is cleared by restarting `cluster-transformer` on `global`. + +## Resolution + +### Cause 1: a DaemonSet Pod of the plugin is Pending with Insufficient cpu on one node + +1. Find the Pending Pod and the node it cannot fit on: + + ```bash + kubectl get ars -A + kubectl describe ars -n cpaas-system multus + kubectl get ds -A | grep multus + kubectl get pods -A --field-selector status.phase=Pending -o wide | grep multus + kubectl -n kube-system describe pod | grep -A5 Events + ``` + +2. Check the allocation of the node named by the scheduler event, or of the node missing from the `-o wide` list: + + ```bash + kubectl describe node | grep -A10 'Allocated resources' + ``` + +3. Free CPU requests on that node: move or scale down workloads that do not need to run there, or lower requests that are larger than needed. The Pending Pod schedules by itself. +4. Confirm that the DaemonSet is complete and the AppRelease leaves `WaitReady`: + + ```bash + kubectl get ds -n kube-system delete-multus-cfg-ds + kubectl get ars -n cpaas-system multus + ``` + +### Cause 2: the log storage Pods on one node fail because /cpaas returns Input/output error + +1. Find the node that runs the failing Pods: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep -E 'cpaas-elasticsearch|cpaas-kafka|cpaas-zookeeper' + ``` + +2. On that node, confirm the I/O error and look at the storage stack under the mount: + + ```bash + ls /cpaas /cpaas/data + findmnt /cpaas /cpaas/data + lsblk + lvs + dmesg -T | grep -iE 'i/o error|blk_update_request|xfs|ext4|dm-' | tail -30 + ``` + + If the device is a multipath device, also check its paths with `multipath -ll`. Failed paths, or kernel messages that another process (for example a partitioning tool) has the device open exclusively for writing, point at the storage itself; involve the storage owner before rebooting. + +3. Do not delete or clean `/cpaas/data` while it returns I/O errors; the data cannot be read in that state and may be needed after the storage is repaired. + +4. Reboot the host. + + > **Warning:** Rebooting the host stops every Pod on it, including other log storage replicas and platform components scheduled there, until the node is back. Precondition: only when `/cpaas` on that node returns `Input/output error`, the failing log storage Pods are on that node, and the other nodes can carry the load meanwhile. Drain the node first where possible. Rollback: none for the reboot itself. If `/cpaas` still fails after the reboot, the disk, its paths, or the LVM volume is damaged; keep the `dmesg` output and involve the storage owner before touching the volume. + + ```bash + kubectl cordon + kubectl drain --ignore-daemonsets --delete-emptydir-data + systemctl reboot + ``` + +5. After the node is back, confirm the mount and return the node to scheduling: + + ```bash + ls /cpaas + kubectl uncordon + kubectl -n cpaas-system get pods -o wide | grep -E 'cpaas-elasticsearch|cpaas-kafka|cpaas-zookeeper' + ``` + + The log storage Pods recover by themselves, and the upgrade passes `CheckStatus-logcenter`. + +### Cause 3: nevermore cannot start, secret log-collector-bearer-auth not found, Log Essentials on global was not upgraded first + +1. On the business cluster, confirm the mount failure and the missing Secret: + + ```bash + kubectl get pod -A | grep nevermore + kubectl -n cpaas-system describe pod | grep -A3 FailedMount + kubectl get secret -A | grep log-collector + ``` + +2. On `global`, compare the version of Log Essentials with the logging plugins of the business cluster: + + ```bash + kubectl get minfo | grep -i log + ``` + +3. Upgrade "Alauda Container Platform Log Essentials" on `global` to the same release line (for example v4.3.2) from Cluster Plugins first. +4. Check that the Secrets arrived on the business cluster, then retry the business cluster plugin upgrade: + + ```bash + kubectl get secret -n cpaas-system | grep -E 'log-collector-bearer-auth|log-database-bearer-auth' + kubectl get pod -A | grep nevermore + ``` + +5. If the upgrade button of a pushed version stays greyed out, restart `cluster-transformer` on `global`. + + > **Warning:** Restarting `cluster-transformer` pauses plugin synchronization until the new Pod is Ready. Precondition: only when the new plugin versions were pushed and the upgrade button stays greyed out. Rollback: none needed; the Deployment recreates the Pod. + + ```bash + kubectl -n cpaas-system rollout restart deployment/cluster-transformer + kubectl -n cpaas-system rollout status deployment/cluster-transformer + ``` + +## Diagnostic Steps + +1. List AppReleases that are not settled and read their reason: + + ```bash + kubectl get ars -A | grep -v -E 'Synced|Ready' + kubectl describe ars -n cpaas-system | grep -E 'Reason|Message' + ``` + + Expected for this article: Reason `NotReady` with `timed out waiting for the condition`, or a progress log that repeats `wait resources ready`. + +2. Find the plugin Pod that is not Ready and read its events: + + ```bash + kubectl get pods -A -o wide | grep -v -E 'Running|Completed' + kubectl -n describe pod | grep -A10 Events + ``` + + - `Pending` with `Insufficient cpu`: Cause 1. + - Several log storage Pods failing on one node whose `/cpaas` returns `Input/output error`: Cause 2. + - `nevermore` in `Init` with `secret "log-collector-bearer-auth" not found`: Cause 3. + - `ImagePullBackOff` or `ErrImagePull`: not this article; see the image pull article named above. diff --git a/docs/en/solutions/No_default_etcd_backup_on_ACP_Cluster_Enhancer_not_installed_or_etcd-backup-default_not_created.md b/docs/en/solutions/No_default_etcd_backup_on_ACP_Cluster_Enhancer_not_installed_or_etcd-backup-default_not_created.md new file mode 100644 index 000000000..f4f5c4eb6 --- /dev/null +++ b/docs/en/solutions/No_default_etcd_backup_on_ACP_Cluster_Enhancer_not_installed_or_etcd-backup-default_not_created.md @@ -0,0 +1,94 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" + - "4.3" +--- + +# No default etcd backup on ACP: Cluster Enhancer not installed or etcd-backup-default not created + +## Issue + +- An inspection finds no etcd backup on a cluster. After the Cluster Enhancer plugin is installed, `kubectl get ebc -A` is empty and `kubectl get acj,bcj -A | grep -i etcd` shows no backup job. +- Creating the default backup by hand fails: `kubectl create -f ebc.yaml` for `etcd-backup-default` returns `Error from server (MethodNotAllowed): error when creating "ebc.yaml": the server does not allow this method on the requested resource` (HTTP 405). +- On a 4.2.1 `global` cluster the etcd backup feature is missing, and the monitoring dashboard list shows only `container-platform`, `log`, and `monitor`, without `cluster` and `kubernetes`. +- After the platform moved from 3.16 to 4.2, it is unclear whether the etcd backup policy reached the workload clusters, including workload clusters still on 3.16. + +Related articles: "etcd backup dates missing in the console, backup tab gone, duplicate copies, or retention to change on ACP" covers an existing backup configuration; "Scheduled etcd backups stop after an upgrade because the etcd-backup-default AdvancedCronJob stops triggering on ACP" covers backups that stop. + +## Root Cause + +### Cause 1: Cluster Enhancer is installed but does not create the default backup until create-etcd-backup-default is true + +The default etcd backup (`EtcdBackupConfiguration`, short name `ebc`, named `etcd-backup-default`) is created by the Cluster Enhancer API Deployment (its name contains `cluster-enhancer-api`) only when its argument `--create-etcd-backup-default` is `true`. Without it, no `ebc` and no backup jobs exist, and the object cannot be created by hand because the API does not allow `create` (`MethodNotAllowed`). Setting the argument to `true` creates the object. For example, on a 4.3 cluster with the `Alauda Container Platform Cluster Enhancer` plugin `v4.3.0`, `kubectl get ebc -A` then lists `etcd-backup-default` with schedule `0 0 * * *`, PAUSED `false`, local path `/cpaas/backup`, TTL `604800`, and 0 records. When the default backup path is also missing, the argument `--default-etcd-backup-path=` (for example `/var/cpaas/backup`) is added next to it. + +### Cause 2: the Cluster Enhancer plugin is not installed on the cluster + +etcd backup rules come from the `Alauda Container Platform Cluster Enhancer` cluster plugin, which backs up etcd data, monitors certificates, cleans up events, and provides the `cluster` and `kubernetes` monitoring dashboards. A cluster without the plugin has neither the etcd backup feature nor those two dashboards. After the plugin is installed (for example version `v4.2.6` on 4.2.1), the dashboard list shows `container-platform`, `cluster`, `log`, `monitor`, and `kubernetes`, and etcd backup is available through `EtcdBackupConfiguration`. + +The policy is per cluster: deploying the etcd backup plugin on the global cluster after an upgrade from 3.16 to 4.2 does not push the policy onto every workload cluster. Backup rules exist on the clusters where Cluster Enhancer is deployed after the 4.2 upgrade, and workload clusters left on 3.16 keep their 3.16 backup rules. + +## Resolution + +### Cause 1: Cluster Enhancer is installed but does not create the default backup until create-etcd-backup-default is true + +1. Confirm that the plugin is installed, no `ebc` exists, and find the Deployment: + + ```bash + kubectl get mplugin cluster-enhancer -o yaml | grep -iE 'version|status' | head + kubectl get ebc -A + kubectl get acj,bcj -A | grep -i etcd || true + kubectl get deploy -A | grep cluster-enhancer-api + kubectl -n get deploy -o jsonpath='{.spec.template.spec.containers[*].args}{"\n"}' + ``` + +2. Save the Deployment: + + ```bash + kubectl -n get deploy -o yaml > /tmp/cluster-enhancer-api.yaml + ``` + +3. Set `--create-etcd-backup-default=true` in the container arguments, and add `--default-etcd-backup-path=` if no default backup path is configured. + + > **Warning:** Editing the Deployment restarts the Cluster Enhancer API Pods; backup and enhancement API calls pause until the new Pods are Ready. Precondition: only when the default etcd backup objects are absent and the argument is not already `true`. Rollback: `kubectl apply -f /tmp/cluster-enhancer-api.yaml`. + + ```bash + kubectl -n edit deploy + kubectl -n rollout status deploy + ``` + +4. Check that the default backup now exists: + + ```bash + kubectl get ebc -A + ``` + + Expected: `etcd-backup-default` with a schedule, PAUSED `false`, a local path, and a TTL. + +### Cause 2: the Cluster Enhancer plugin is not installed on the cluster + +1. Check whether the plugin and a backup configuration exist on the cluster in question: + + ```bash + kubectl get deploy -A | grep cluster-enhancer + kubectl get etcdbackupconfiguration -A + ``` + +2. Install the `Alauda Container Platform Cluster Enhancer` cluster plugin from **Administrator → Marketplace → Cluster Plugins** for that cluster. +3. Check the dashboard list (it should now include `cluster` and `kubernetes`) and run `kubectl get etcdbackupconfiguration -A` again. If no default object appears, continue with Cause 1. +4. To change a backup rule, edit `EtcdBackupConfiguration` on the cluster whose rule you mean to change. A workload cluster still on 3.16 keeps its existing 3.16 backup configuration. + +## Diagnostic Steps + +```bash +kubectl get deploy -A | grep cluster-enhancer +kubectl get ebc -A +kubectl get acj,bcj -A | grep -i etcd +``` + +- Cluster Enhancer Deployments present, `ebc` empty, or `MethodNotAllowed` (405) when creating an `ebc` by hand: Cause 1. Expected: the `cluster-enhancer-api` arguments do not contain `--create-etcd-backup-default=true`. +- No Cluster Enhancer Deployment on the cluster, no etcd backup, and no `cluster` or `kubernetes` dashboards: Cause 2. +- An `ebc` exists and backups ran before but stopped: see "Scheduled etcd backups stop after an upgrade because the etcd-backup-default AdvancedCronJob stops triggering on ACP". diff --git a/docs/en/solutions/Operator_or_plugin_ImagePullBackOff_or_exec_format_error_registry_address_401_architecture_or_search_domain_on_ACP.md b/docs/en/solutions/Operator_or_plugin_ImagePullBackOff_or_exec_format_error_registry_address_401_architecture_or_search_domain_on_ACP.md new file mode 100644 index 000000000..c4f54b4c5 --- /dev/null +++ b/docs/en/solutions/Operator_or_plugin_ImagePullBackOff_or_exec_format_error_registry_address_401_architecture_or_search_domain_on_ACP.md @@ -0,0 +1,240 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.1" + - "4.2" +--- + +# Operator or plugin ImagePullBackOff or exec format error: registry address, 401, architecture, or search domain on ACP + +## Issue + +- An operator deployed from OperatorHub (for example `knative-operator` on `global`, installed for all namespaces) stays in Deploying (`部署中`), and its Pod events say the image cannot be pulled. +- The Knative Operator from the marketplace cannot be installed into a namespace because its image cannot be downloaded. Editing the image on the Deployment works for a while and is then changed back. +- Installing the ClickHouse plugin on `global` times out and the console shows the installation as failed; `cpaas-elasticsearch-0` is in `ImagePullBackOff`. +- Cluster functional components stay in upgrading, and the `connectors-operator` Pod on an ARM business cluster crashes with `exec /manager: exec format error`. +- After the global cluster recovers from a failure, platform component Pods in a workload cluster stay in `ImagePullBackOff`; a manual pull from the platform registry fails with `received unexpected HTTP status: 500 Internal Server Error`. + +A plugin that waits for a Pod that is Pending or failing on its node for reasons other than the image is covered in "Cluster plugin stuck at wait resources ready: Pod Pending, /cpaas I/O error, or missing log secret on ACP". + +## Root Cause + +### Cause 1: the registry address was changed and the running Pod still pulls from the old address + +The Pod event is `BackOff: Back-off pulling image "/:"`, repeated a very large number of times, and the registry in the event is not the one the platform is configured with now. After the image registry address of the platform is changed, an operator Pod that was created before the change (for example `knative-operator` in namespace `operators`) still references the old address, so every pull fails and the operator stays in Deploying. Recreating the Pod makes it pull from the current address. + +### Cause 2: the image points to a registry domain the cluster cannot reach, and Deployment edits are reverted + +The events of the operator Pods (for example `knative-operator-` and `operator-webhook-`) show: + +```text +Failed to pull image "/:": rpc error: code = DeadlineExceeded desc = failed to pull and unpack image ... failed to resolve reference ... dial tcp ...:443: i/o timeout +``` + +followed by `Back-off pulling image` and `Error: ImagePullBackOff`. + +The operator's image references use a registry domain that the cluster cannot resolve or reach. Normally Metis, the platform webhook controller that passes values from upper-level resources down to workloads, replaces such image addresses with the platform registry address; when it does not, the Pods keep the unreachable domain. The same repository and tag pull fine from the platform registry address of the cluster (Clusters → `` → Overview → Basic Info → Image Repository Address, for example an `IP:11443` address). Editing the image on the Deployment does not last, because the operator Deployment is owned by its ClusterServiceVersion (CSV) and OLM writes the CSV's spec back. Changing the image address in the CSV propagates to the Deployments, and the operator starts. + +For Knative, the components from the platform marketplace are supported; open-source Knative Operators that are not from the product are not. + +### Cause 3: 401 Unauthorized from the registry token service, a platform image cannot be pulled + +A platform Pod (for example `cpaas-elasticsearch-0` in `cpaas-system`, image `ait/elasticsearch-basic-http:`) cannot pull its image: + +```text +failed to authorize: failed to fetch oauth token: unexpected status from GET request to https:///auth/token?scope=repository%3Aait%2Felasticsearch-basic-http%3Apull&service=token-service: 401 Unauthorized +``` + +then `ErrImagePull`, `ImagePullBackOff`, and `Back-off pulling image`; a plugin install that needs the Pod times out. The registry token service refused the pull with `401`. Pulling the image manually on the node and then recreating the Pod recovers it. + +### Cause 4: exec format error, the operator package that was pushed has no image for the cluster's architecture + +`kubectl logs` of the operator Pod returns `exec /manager: exec format error`. The image itself was pulled successfully, the `manager` container restarts, and the event is `Back-off restarting failed container manager`. The cluster runs on another architecture than the image (for example `global` is x86 and the business cluster is ARM). + +The operator package that was pushed contained only the x86 image, so the ARM nodes run an x86 binary. Pushing the multi-architecture (hybrid) package of the same operator version with `violet push --force` overwrites the package, the business cluster pulls the ARM image, and the Pod reaches `Running`. Triggering the upgrade of the stuck components from OperatorHub then clears the upgrading state. + +### Cause 5: pulls return 500 because a node search domain makes registry name lookups time out + +Platform component Pods on a workload cluster stay in `ImagePullBackOff`. A manual pull on a node fails with: + +```text +Error response from daemon: received unexpected HTTP status: 500 Internal Server Error +``` + +and `curl -kv` against the platform registry, and against its backend address, also returns `500`. + +The node `/etc/resolv.conf` (written by NetworkManager) carries an extra search domain such as `linklocal`. Pods with `dnsPolicy: ClusterFirst` inherit the node search domains after the cluster suffixes, with `options ndots:5`; the `alb2` container of the global ALB, for example, shows `search cpaas-system.svc.cluster.local svc.cluster.local cluster.local linklocal`. Every lookup of the registry domain and of in-cluster Service names is then also tried with `.linklocal` appended, CoreDNS forwards those names upstream, and the lookups time out. The CoreDNS log shows: + +```text +[ERROR] plugin/errors: 2 .linklocal. A: read udp :->:53: i/o timeout +``` + +for names such as the platform registry domain, `registry.kube-system.svc.cluster.local.linklocal.`, and `erebus.cpaas-system.svc.cluster.local.linklocal.`. The component that serves the registry cannot resolve what it needs in time and answers `500`. Removing the search domain from the node `/etc/resolv.conf` and recreating the ALB and CoreDNS Pods restores the pulls. + +## Resolution + +### Cause 1: the registry address was changed and the running Pod still pulls from the old address + +1. Compare the registry in the event with the current Image Repository Address of the cluster: + + ```bash + kubectl get pods -A | grep -i + kubectl -n describe pod | grep -E 'Image:|Back-off pulling' + ``` + +2. Recreate the Pod. + + > **Warning:** Deleting the operator Pod restarts the operator; its reconciliation pauses until the new Pod is Running. Precondition: only when the registry address was changed and the Pod event still pulls from the old address. Rollback: none needed; the Deployment recreates the Pod. + + ```bash + kubectl -n delete pod + kubectl -n get pod -w | grep + ``` + +3. If the new Pod still names the old registry, the address is stored in the workload spec; continue with Cause 2. + +### Cause 2: the image points to a registry domain the cluster cannot reach, and Deployment edits are reverted + +1. Read the image of the Deployment and find the CSV: + + ```bash + kubectl get po -A | grep + kubectl -n get deploy -o jsonpath='{.spec.template.spec.containers[*].image}{"\n"}' + kubectl get csv -A | grep + kubectl get po -A | grep metis + ``` + +2. On a node, test both addresses. The domain failing and the platform registry address working is this cause: + + ```bash + crictl pull /: + crictl pull /: + ``` + +3. Save the CSV: + + ```bash + kubectl -n get csv -o yaml > "$HOME/.before.yaml" + ``` + +4. Change the image in the CSV, not on the Deployment. + + > **Warning:** Saving the CSV makes OLM roll the operator Deployments with the new image; the operator restarts. Precondition: only when the Pods are in `ImagePullBackOff` for an image whose registry domain the cluster cannot reach, and the same repository and tag pull from the platform registry address. Rollback: `kubectl apply -f "$HOME/.before.yaml"`. + + ```bash + kubectl -n edit csv + ``` + + Replace `` with the platform registry address in every image under `.spec.install.spec.deployments[].spec.template.spec.containers[].image` (and `relatedImages` if present), and save. + +5. Check the Pods: + + ```bash + kubectl get po -A | grep + ``` + +If other operators from the marketplace show the same unreplaced domain, collect the Metis Pod log (`kubectl -n logs --tail=300`) for Alauda technical support before editing more CSVs. + +### Cause 3: 401 Unauthorized from the registry token service, a platform image cannot be pulled + +1. On the node that runs the Pod, pull the image the event names: + + ```bash + kubectl -n cpaas-system get pod cpaas-elasticsearch-0 -o wide + kubectl -n cpaas-system describe pod cpaas-elasticsearch-0 | grep -E 'Image:|Failed' + crictl pull /ait/elasticsearch-basic-http: + ``` + + If the plain pull also returns `401`, pull with the platform registry credentials: + + ```bash + crictl pull --creds ':' /ait/elasticsearch-basic-http: + ``` + +2. Recreate the Pod. + + > **Warning:** Deleting the Pod restarts that Elasticsearch replica; it rejoins after it starts. Precondition: only after the image is present on that node (`crictl images | grep elasticsearch-basic-http`) and the Pod is in `ImagePullBackOff` with the `401 Unauthorized` token error. Rollback: none needed; the StatefulSet recreates the Pod. + + ```bash + kubectl -n cpaas-system delete pod cpaas-elasticsearch-0 + kubectl -n cpaas-system get pod cpaas-elasticsearch-0 -w + ``` + +3. Retry the plugin install. + +### Cause 4: exec format error, the operator package that was pushed has no image for the cluster's architecture + +1. Compare the node architecture with the crash: + + ```bash + kubectl get nodes -o custom-columns=NAME:.metadata.name,ARCH:.status.nodeInfo.architecture + kubectl -n get pods + kubectl -n logs --previous --tail=20 + ``` + +2. Push the multi-architecture package of the same version. + + > **Warning:** `--force` overwrites the package, Artifact, and ArtifactVersion already on the platform for this operator version. Precondition: only when the operator Pod logs `exec format error` on nodes of one architecture and the package you push is the multi-architecture (hybrid) build of the same version. Rollback: none for the overwritten content; push the previous file again with `--force` if needed. + + ```bash + violet push .tgz --platform-address --platform-username --platform-password '' --clusters --force + ``` + +3. Trigger the upgrade of the stuck components from OperatorHub on that cluster, and check that the Pod pulls the ARM image and runs: + + ```bash + kubectl -n get pods -w + ``` + +### Cause 5: pulls return 500 because a node search domain makes registry name lookups time out + +1. Confirm the search domain in a platform Pod and the timeouts in CoreDNS (on the global cluster for the global ALB): + + ```bash + kubectl -n cpaas-system get pods | grep alb + kubectl -n cpaas-system exec -c alb2 -- cat /etc/resolv.conf + kubectl -n kube-system logs -l k8s-app=kube-dns --tail=200 | grep -i 'i/o timeout' + ``` + +2. On each node whose `/etc/resolv.conf` carries the extra search domain, back the file up and comment out the `search` entry. + + > **Warning:** Removing the node search domain changes how host processes resolve short names; anything on the host that relied on that suffix must use full names afterwards. Precondition: only when Pod `resolv.conf` files carry the site suffix and CoreDNS logs `i/o timeout` for names with that suffix appended. Rollback: restore the backup file and recreate the Pods again. + + ```bash + cp -a /etc/resolv.conf /root/resolv.conf.before + grep -nE '^(search|domain)' /etc/resolv.conf + sed -i -E 's/^(search|domain)([[:space:]])/# \1\2/' /etc/resolv.conf + cat /etc/resolv.conf + ``` + + When NetworkManager writes the file, also remove the search domain from the NetworkManager connection, or the next NetworkManager restart puts it back. + +3. Recreate the ALB and CoreDNS Pods so that their `/etc/resolv.conf` is written again. + + > **Warning:** Restarting the ALB interrupts the traffic it carries, including console access, until the new Pods are Ready, and a CoreDNS restart briefly delays lookups. Precondition: the node files are fixed first. Rollback: none needed; the controllers recreate the Pods. + + ```bash + kubectl -n cpaas-system delete pod + kubectl -n kube-system rollout restart deploy/coredns + ``` + +4. Pull the image again on a workload cluster node and watch the Pods leave `ImagePullBackOff`. + +For the same search-domain mechanism behind console and DevOps timeouts, see "Platform UI stuck loading or DevOps errors from Service name timeouts caused by node search domains on ACP". + +## Diagnostic Steps + +```bash +kubectl get pods -A | grep -E 'ImagePullBackOff|ErrImagePull|CrashLoopBackOff' +kubectl -n describe pod | grep -E 'Image:|Failed|Back-off' +kubectl -n logs --previous --tail=20 +``` + +- A back-off on an image from a registry address that was changed: Cause 1. +- `i/o timeout` or a resolution failure for a build registry domain, with Deployment edits reverted: Cause 2. +- `failed to fetch oauth token ... 401 Unauthorized`: Cause 3. +- A successful pull followed by `exec format error` on nodes of one architecture: Cause 4. +- `received unexpected HTTP status: 500` from the platform registry, and CoreDNS `i/o timeout` errors for names ending in an extra suffix such as `.linklocal.`: Cause 5. diff --git a/docs/en/solutions/Operator_stays_Processing_base-operator_logs_certwatch_errors_or_a_CSV_keeps_raising_DisabledCopiedCSVs_events_on_ACP.md b/docs/en/solutions/Operator_stays_Processing_base-operator_logs_certwatch_errors_or_a_CSV_keeps_raising_DisabledCopiedCSVs_events_on_ACP.md new file mode 100644 index 000000000..2717d6841 --- /dev/null +++ b/docs/en/solutions/Operator_stays_Processing_base-operator_logs_certwatch_errors_or_a_CSV_keeps_raising_DisabledCopiedCSVs_events_on_ACP.md @@ -0,0 +1,108 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Operator stays Processing, base-operator logs certwatch errors, or a CSV keeps raising DisabledCopiedCSVs events on ACP + +## Issue + +One of the following is seen on a cluster whose operators otherwise run normally: + +- The cluster functional-component page shows `redis-operator` as Processing, and it stays there. The operator Pod is not crash-looping. A managed instance under that operator is paused, and the ModuleInfo (`minfo`) of the component reports Processing. +- `base-operator` logs keep printing certificate reread errors such as `error re-reading certificate` with `tls: private key does not match public key`, and reconciliation reports `ProductBase Reconcile error: ProductBase need retry`. `base-operator` is otherwise healthy. +- A ClusterServiceVersion (CSV) keeps raising the same warning event, for example `DisabledCopiedCSVs: CSV copying disabled for operators/katanomi-operator.v3.20.36` hundreds of times over a few weeks, and the question is whether it is a fault. + +`redis-operator` that stays Processing during a platform upgrade because its Subscription waits for manual approval is a different cause; see "ACP upgrade stuck on an operator, OLM catalog, or operator-managed component". + +## Root Cause + +### Cause 1: redis-operator stays Processing because a managed instance is paused + +When an instance managed by an operator is paused, the platform's operator component logic reports the cluster functional component as Processing on its `minfo`. This is the expected display for a paused instance, not an operator failure. While the instance stays paused, the component stays Processing; when the instance is resumed, `minfo` and the functional-component page follow. + +### Cause 2: error re-reading certificate, tls: private key does not match public key + +`base-operator` watches its certificate files with certwatch, which relies on file system notifications. During certificate rotation `tls.crt` is updated first and `tls.key` after it. A reread that runs in this gap sees a certificate and key that do not belong together, or a file that is not fully written yet, and logs one of: + +```text +error re-reading certificate ... tls: private key does not match public key +error re-reading certificate ... tls: failed to find any PEM data in key input +error re-reading certificate ... tls: failed to find any PEM data in certificate input +``` + +When `tls.key` has finished updating, the pair matches again. `base-operator` and `prdb` stay healthy through these lines, the errors do not affect use of the product, and a `ProductBase need retry` from the same gap clears with the file update. + +### Cause 3: repeated DisabledCopiedCSVs warning events on katanomi or Knative operator CSVs + +The event list of the `operators` namespace shows, for example: + +```text +DisabledCopiedCSVs: CSV copying disabled for operators/knative-operator.v3.20.10 +DisabledCopiedCSVs: CSV copying disabled for operators/katanomi-operator.v3.20.36 +``` + +with counts in the hundreds over about two weeks. OLM has CSV copying turned off (`spec.features.disableCopiedCSVs: true` on the `OLMConfig` named `cluster`), and the event reports that state for each cluster-wide operator CSV. It is informational and not an error; the operators work normally and the event count simply grows. + +## Resolution + +### Cause 1: redis-operator stays Processing because a managed instance is paused + +1. List ModuleInfo and the instances the operator manages, and match a Processing `minfo` to a paused instance: + + ```bash + kubectl get moduleinfo -A + kubectl get pods -A | grep redis-operator + kubectl api-resources | grep -i redis + kubectl get -A + ``` + +2. If the instance is meant to stay paused, no action is needed; Processing stays with it. +3. If the instance should run again, resume it through the same control that paused it (for example in the instance's page in the console). After the instance is no longer paused, `minfo` leaves Processing and the functional-component page follows. + +If every managed instance is running and `minfo` is still Processing, this cause does not apply; check the CSV phase (`kubectl get csv -A | grep -i redis`) and the operator Pod status. + +### Cause 2: error re-reading certificate, tls: private key does not match public key + +1. Confirm the log lines and that the components are otherwise Ready: + + ```bash + kubectl get deploy,pods -A | grep -E 'base-operator|prdb' + kubectl logs -n deploy/base-operator --tail=200 + ``` + +2. If the lines match this cause and `base-operator` and `prdb` are healthy, leave them alone. They stop after `tls.key` is written; no recovery operation is required. + +A certificate error that continues after rotation has finished, or a `base-operator` Pod that is not Ready, is not this gap. Collect the steady-state log before changing any certificate files. + +### Cause 3: repeated DisabledCopiedCSVs warning events on katanomi or Knative operator CSVs + +1. Read the events and the OLM setting that produces them: + + ```bash + kubectl -n operators get events --field-selector reason=DisabledCopiedCSVs + kubectl get events -A --field-selector involvedObject.kind=ClusterServiceVersion | grep -iE 'katanomi|knative' + kubectl get olmconfig cluster -o jsonpath='{.spec.features.disableCopiedCSVs}{"\n"}' + ``` + +2. An event that says CSV copying is disabled for that CSV, with `disableCopiedCSVs` set to `true`, is expected. No action is required. + +For what the setting controls and how to change it, see "Controlling Copied CSVs on ACP with the OLMConfig disableCopiedCSVs feature". + +## Diagnostic Steps + +```bash +kubectl get moduleinfo -A +kubectl get csv -A | grep -iE 'redis|katanomi|knative' +kubectl get pods -A | grep -E 'redis-operator|base-operator|prdb' +kubectl get olmconfig cluster -o jsonpath='{.spec.features.disableCopiedCSVs}{"\n"}' +``` + +- Processing `minfo`, a paused managed instance, and the operator Pod `Running`: Cause 1. +- `error re-reading certificate` naming a key/certificate mismatch or missing PEM data, while `base-operator` and `prdb` are healthy and the lines stop after rotation: Cause 2. +- A recurring `DisabledCopiedCSVs` warning on katanomi or Knative operator CSVs with `disableCopiedCSVs` printed as `true`: Cause 3. diff --git a/docs/en/solutions/Pre-upgrade_checklist_reports_a_CIDR_mismatch_a_missing_is-global_label_the_NFS_plugin_or_node_isolation_on_ACP.md b/docs/en/solutions/Pre-upgrade_checklist_reports_a_CIDR_mismatch_a_missing_is-global_label_the_NFS_plugin_or_node_isolation_on_ACP.md new file mode 100644 index 000000000..1f22a379a --- /dev/null +++ b/docs/en/solutions/Pre-upgrade_checklist_reports_a_CIDR_mismatch_a_missing_is-global_label_the_NFS_plugin_or_node_isolation_on_ACP.md @@ -0,0 +1,156 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.1" + - "4.2" +--- + +# Pre-upgrade checklist reports a CIDR mismatch, a missing is-global label, the NFS plugin, or node isolation on ACP + +## Issue + +The pre-upgrade checklist prints one of the following lines before an ACP upgrade, and it is not clear whether the line is a real blocker or what to change: + +- `集群的 cluster cidr不一致,cluster资源上的为:,kubeadm config上的为: cluster-cidr` (the cluster CIDR is inconsistent: the Cluster resource has ``, the kubeadm config has `cluster-cidr`). On the cluster, `kubectl get cm -n kube-system kubeadm-config -o yaml | grep -iE "cluster-cidr"` prints only the parameter name, not a CIDR. +- `ERROR: global集群不存在is-global的label` (the global cluster has no `is-global` label). +- `ERROR: 集群 已部署 NFS 插件,请确认 NFS 的协议版本,如果为 NFSv3 则需要手动添加 nolock 参数或者升级后将 NFS 插件的 readOnlyRootFilesystem 修改为 false` (the NFS plugin is deployed on cluster ``; confirm the NFS protocol version; for NFSv3 add the `nolock` option, or after the upgrade set `readOnlyRootFilesystem` of the NFS plugin to `false`). +- `升级前,请确认 global 集群中的节点是否已 启用 global Cluster Platform Node Isolation 。如需解决方案,请联系技术支持。` (before upgrading, confirm whether the global cluster nodes have global Cluster Platform Node Isolation enabled; contact technical support for a solution). + +## Root Cause + +### Cause 1: CIDR mismatch line that shows the parameter name cluster-cidr instead of a value + +The checklist compares the CIDRs on the platform's Cluster resource with the values in the `kube-system/kubeadm-config` ConfigMap. Newer kubeadm configurations (the `v1beta4` configuration API) store the controller-manager flags under `controllerManager.extraArgs` as a list of `name`/`value` pairs: + +```yaml +controllerManager: + extraArgs: + - name: allocate-node-cidrs + value: "false" + - name: cluster-cidr + value: + - name: node-cidr-mask-size + value: "24" +``` + +The CIDR is on the `value:` line after `- name: cluster-cidr`. Older configurations write `cluster-cidr: ` on one line. With the list layout, a one-line match for `cluster-cidr` returns only the name line, so the checklist prints `cluster-cidr` where a CIDR is expected. When the YAML is read in full and the value matches the Cluster resource, the line is a parsing artifact and nothing has to change. + +The same comparison answers other CIDR lines of the checklist: compare the cluster overview page with `kubeadm-config` and, on Kube-OVN clusters, with the `join` subnet. When the CIDR shown on the cluster's CIDR page is the one in use, the CIDR item can be ignored. + +### Cause 2: the global cluster has no is-global label + +The cluster label check applies to upgrades to 3.8 or later. It requires the label `is-global=true` on both global cluster resources: `cluster global`, and the `cluster.v1alpha1.clusterregistry.k8s.io` object named `global` in `cpaas-system`. When either object lacks the label, the checklist stops with this error. + +### Cause 3: the NFS plugin is deployed on the cluster + +The checklist flags every cluster that has the NFS plugin deployed, because an NFSv3 export needs either the `nolock` mount option or `readOnlyRootFilesystem: false` on the NFS plugin after the upgrade. The line is printed once per cluster, for example for both `global` and a business cluster. The check does not test whether the plugin is actually in use on that cluster, so it also appears for a cluster where the plugin is installed but unused. + +### Cause 4: Confirm whether global Cluster Platform Node Isolation is enabled + +The item refers to the node isolation policy of the global cluster. It is a confirmation prompt: when the policy is not enabled, the item does not apply. + +The same pre-upgrade step can also ask to upload the TopoLVM plugin before upgrading. That is the Alauda Build of TopoLVM entry for the matching node architecture in the App Marketplace (for a 4.2.1 target: channel `alpha`, version `v4.2.4`), which is offered for x86, ARM, and heterogeneous architectures. + +## Resolution + +### Cause 1: CIDR mismatch line that shows the parameter name cluster-cidr instead of a value + +Compare the two sources for each cluster the checklist names, one cluster at a time. + +1. In the console, open Cluster Management, select the cluster, and record the pod CIDR, the service CIDR, and on Kube-OVN clusters the join CIDR from the cluster overview page. +2. On a control-plane node of that cluster, read the values in use. `-A1` prints the `value:` line that follows each `name:` line; `podSubnet` and `serviceSubnet` are printed on their own lines: + + ```bash + kubectl get cm -n kube-system kubeadm-config -o yaml | grep -iE -A1 "cluster-cidr|subnet" + ``` + + On a Kube-OVN cluster also read the join subnet: + + ```bash + kubectl get subnet join -o jsonpath='{.spec.cidrBlock}{"\n"}' + ``` + +3. If every value matches the overview page, the checklist line is a parsing artifact. Ignore it and keep the comparison with the upgrade plan. + +If a value really differs from what the cluster is running, this cause does not apply. Send the comparison and the checklist output to Alauda technical support before the upgrade window. + +### Cause 2: the global cluster has no is-global label + +1. On a control-plane node of the global cluster, check the label on both objects: + + ```bash + kubectl get cluster global --show-labels | grep is-global + kubectl get cluster.v1alpha1.clusterregistry.k8s.io -n cpaas-system global --show-labels | grep is-global + ``` + +2. If either command prints nothing, add the label to that object: + + ```bash + kubectl label cluster global is-global=true + kubectl label cluster.v1alpha1.clusterregistry.k8s.io -n cpaas-system global is-global=true + ``` + +3. Run the checklist again. + +Add the label only to the global cluster's objects. A label added by mistake to another cluster is removed with `kubectl label cluster is-global-`. + +### Cause 3: the NFS plugin is deployed on the cluster + +1. On a node of each named cluster that mounts the export, read the NFS protocol version. `vers=` in the mount options is the version in use. Also list the StorageClasses, PVs, and NFS Pods: + + ```bash + grep -E ' nfs4? ' /proc/mounts + kubectl get sc -o custom-columns=NAME:.metadata.name,PROVISIONER:.provisioner,MOUNTOPTIONS:.mountOptions + kubectl get pv -o custom-columns=NAME:.metadata.name,SC:.spec.storageClassName,CLAIM:.spec.claimRef.name + kubectl get pods -A | grep -i nfs + ``` + + If `vers=4` or later is in use, the NFSv3 condition does not apply. + +2. **The cluster does not use the NFS plugin.** Uninstall the NFS plugin from that cluster through the platform's cluster plugin management before the upgrade. + + > **Warning:** Only when no PV on that cluster uses the NFS StorageClass and no workload mounts the NFS export, as shown by the `kubectl get pv` and `kubectl get sc` output above. Impact: new PVCs on the NFS StorageClass can no longer be provisioned on that cluster. Save the PV list first. Rollback: install the NFS plugin again from cluster plugin management with the same export settings. + +3. **The cluster uses the NFS plugin over NFSv3.** Add `nolock` to the mount options of the NFS StorageClass. Back the StorageClass up first: + + ```bash + kubectl get sc -o yaml > .yaml + ``` + + > **Warning:** With `nolock`, NFS file locks are local to each client node, so two nodes writing to the same file are not protected from each other by NFS locking. Precondition: the export is confirmed to be NFSv3 and the backup file exists. Rollback: remove `nolock` again with `kubectl apply -f .yaml`. A StorageClass change applies to PVs provisioned after it; each existing PV keeps the options in its own `spec.mountOptions`. + + If the StorageClass already has a `mountOptions` list, append to it: + + ```bash + kubectl patch sc --type=json -p '[{"op":"add","path":"/mountOptions/-","value":"nolock"}]' + ``` + + If it has no `mountOptions` field yet, create the list: + + ```bash + kubectl patch sc --type=merge -p '{"mountOptions":["nolock"]}' + ``` + +The checklist also offers setting `readOnlyRootFilesystem` to `false` on the NFS plugin after the upgrade as the alternative. This article covers the `nolock` path only. + +### Cause 4: Confirm whether global Cluster Platform Node Isolation is enabled + +1. Open the node isolation policy setting of the global cluster in the console. +2. If the policy is not enabled, ignore this checklist item. +3. If it is enabled, contact Alauda technical support for the solution before the upgrade, as the item instructs. + +For the TopoLVM upload, read the node architecture and pick the TopoLVM package for it (for example `topolvm-operator.alpha.arm64.v4.2.4.tgz` when the nodes are ARM): + +```bash +kubectl get nodes -o custom-columns=NAME:.metadata.name,ARCH:.status.nodeInfo.architecture +``` + +## Diagnostic Steps + +1. A CIDR line that shows a parameter name (`cluster-cidr`) or an empty value where a CIDR is expected: read `kubeadm-config` with `grep -A1` and compare with the overview page and, on Kube-OVN, the `join` subnet. Expected for Cause 1: the `value:` line carries the same CIDR as the overview page. +2. `global集群不存在is-global的label`: the two `--show-labels` commands of Cause 2. Expected for Cause 2: at least one of them prints nothing. +3. `已部署 NFS 插件`: read `vers=` from `/proc/mounts` and list the PVs on the NFS StorageClass. No PVs: uninstall the unused plugin. `vers=3` with PVs in use: add `nolock`. `vers=4` or later: the NFSv3 condition does not apply. +4. `global Cluster Platform Node Isolation`: check the node isolation policy of the global cluster. Not enabled: ignore. Enabled: contact technical support. diff --git a/docs/en/solutions/Preparing_a_host_before_adding_it_to_an_ACP_cluster_node_check_items_disks_init_script_DefaultTasksMax_root_access.md b/docs/en/solutions/Preparing_a_host_before_adding_it_to_an_ACP_cluster_node_check_items_disks_init_script_DefaultTasksMax_root_access.md new file mode 100644 index 000000000..319555c41 --- /dev/null +++ b/docs/en/solutions/Preparing_a_host_before_adding_it_to_an_ACP_cluster_node_check_items_disks_init_script_DefaultTasksMax_root_access.md @@ -0,0 +1,182 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - "4.1" + - "4.2" + - "4.3" +--- + +# Preparing a host before adding it to an ACP cluster: node check items, disks, init script, DefaultTasksMax, root access + +## Issue + +Before deploying the platform or adding a node, the following questions come up: + +- Which operating-system and kernel parameters must be changed on the host, and why? +- How much disk does a new worker node need, and which mount points? +- What has to be run on a new server before it is added to an existing cluster? +- Why did the initialization script comment out `UsePAM` in `/etc/ssh/sshd_config`? +- What should `DefaultTasksMax` be, and which ports does a business cluster need? +- Can the deployment use a non-root user with limited rights? + +Skipping these steps shows up later as node availability check failures, kernels the CNI cannot work with, or installs that stop on a missing package. + +Related articles: "Adding a node to an ACP cluster fails on UseDNS or UsePAM, the node does not boot after init.sh, or a cloned node fails at yum" covers failures during the add; "Which operating systems, kernels, and CPU architectures can run in an ACP cluster" covers OS, kernel, and architecture support. + +## Root Cause + +### Cause 1: the node check has mandatory items that depend on the platform version + +The requirements are version-specific and are published in the product documentation under **Installation → Node Preprocessing → Node Check** (on a running platform, open **Help Documentation** in the top-right corner). For ACP 4.1.6 the check splits configuration into items that must pass and scenario-specific checks. The mandatory items include: + +- Kernel command line: `transparent_hugepage=never`; on CentOS 7.x also `cgroup.memory=nokmem`. +- Kernel modules: the `ip_vs` series and `nf_conntrack`; with Kube-OVN as CNI also `geneve` and `openvswitch`. +- AppArmor / SELinux, the host firewall, and swap. +- SSH: the user's permissions, and `UseDNS` and `UsePAM` in `sshd_config`. + +### Cause 2: the initialization script comments out UsePAM + +The platform drives nodes through automated SSH logins and remote commands. The initialization script comments out `UsePAM` (which was `yes`) so that PAM modules do not interfere with those automated logins and commands. The node check inspects `UsePAM` and `UseDNS`, and warns with `/etc/ssh/sshd_config文件中 UsePAM 参数被注释或值不为 no,请调整` (the `UsePAM` parameter in `/etc/ssh/sshd_config` is commented out or not `no`; adjust it). That warning is non-blocking: node preprocessing sets it to prevent SSH timeouts during the add, and it is not a problem once the node has joined. + +### Cause 3: worker disk layout follows usage, not the existing nodes + +For ACP 4.3.1 worker nodes, the recommended starting point is: + +| Mount point | Minimum | +|---|---| +| `/cpaas` | 100 GB | +| `/var/lib/containerd` | 200 GB | + +The real need depends on the number of Pods, image sizes and how many versions are kept, temporary container data, log volume, and local storage. Disks can start at the initial size and be expanded as usage grows. A new node does not have to copy the disks of the existing nodes. A node that will later host log or monitoring storage needs `/cpaas` on its own mount. + +### Cause 4: a new server must be initialized before it is added + +To add a server to a business cluster (for example on ACP 4.2.1): check the node against the deployment documentation of that version, run `init.sh` from the installer package of the same version on the node, then add the node through the platform. The `init.sh` kept on the first control-plane node of `global` (under `/root`) can also be copied to the new node and run there. + +### Cause 5: DefaultTasksMax must be infinity, and the business cluster ports + +For a production plan on Kylin (kernel `4.19.90-89.26.v2401`), two check items apply: `DefaultTasksMax` in `/etc/systemd/system.conf` must be `infinity`, and a 4.x business cluster needs ports 6443 and 11780 open for this purpose; 11781 is not needed. A change to `/etc/systemd/system.conf` takes effect only after `systemctl daemon-reexec`; `daemon-reload` rereads unit files but not the manager configuration. After `daemon-reexec`, only units started afterwards get the new default. + +### Cause 6: the deployment needs root or sudo on the nodes + +Deployment creates directories such as `/var/lib/containerd/`, `/var/lib/kubelet/`, `/var/lib/etcd/`, and `/cpaas/`, and runs partitioning commands. A non-root user with a restricted set of rights is not supported; the deployment needs `root` or a user with sudo rights. + +## Resolution + +### Cause 1: the node check has mandatory items that depend on the platform version + +Apply the items from the Node Check page of your version. The commands below cover the items listed above. + +> **Warning:** Changing the kernel command line requires a reboot, and a typo in the boot entry can leave the host unbootable. Do it before the host joins, or drain it first. Rollback: remove the arguments with the same tool (`grubby --update-kernel=ALL --remove-args=...`) and reboot. Disabling swap and the firewall changes host behavior for every process; follow your site's security approval for the firewall item. + +```bash +# kernel command line (distributions that use grubby, such as CentOS and Kylin) +cat /proc/cmdline +sudo grubby --update-kernel=ALL --args="transparent_hugepage=never" +# CentOS 7.x only +sudo grubby --update-kernel=ALL --args="cgroup.memory=nokmem" + +# kernel modules, loaded now and at boot +sudo modprobe -a ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack +printf '%s\n' ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack | sudo tee /etc/modules-load.d/acp.conf +# Kube-OVN clusters only +sudo modprobe -a geneve openvswitch +printf '%s\n' geneve openvswitch | sudo tee -a /etc/modules-load.d/acp.conf + +# swap +sudo swapoff -a +``` + +Comment out the swap line in `/etc/fstab` so swap stays off after a reboot. Reboot once after the command-line change and confirm with `cat /proc/cmdline`. + +### Cause 2: the initialization script comments out UsePAM + +The change is intentional; keep it. If the availability check warns about `UsePAM` or `UseDNS`, set both explicitly to `no`, then check the values sshd actually uses: + +```bash +sudo grep -n -E '^\s*#?\s*(UsePAM|UseDNS)' /etc/ssh/sshd_config +# after setting "UsePAM no" and "UseDNS no": +sudo sshd -t && echo "sshd_config syntax OK" +sudo systemctl restart sshd +sudo sshd -T | grep -i -E '^(usepam|usedns) ' +``` + +If a site baseline requires PAM for SSH logins, raise it with Alauda technical support before re-enabling `UsePAM`. + +### Cause 3: worker disk layout follows usage, not the existing nodes + +Create the mounts before the node is added, sized from the table and the expected workload: + +```bash +lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE +df -h /cpaas /var/lib/containerd +findmnt /cpaas /var/lib/containerd +``` + +Plan a separate `/cpaas` volume on any node that may later become a log or monitoring storage node. + +### Cause 4: a new server must be initialized before it is added + +1. Run the checks of your version on the new server. +2. Run the installer's `init.sh` of the same version: + + ```bash + # on the new server, from the extracted installer package of the same version + sudo bash ./init.sh + ``` + + The script path inside the package can differ between versions; use the one named in that version's deployment documentation. Start the scripts with `bash`, not `sh`; an add-node script run with `sh` fails on its syntax. + +3. Add the node from the cluster's node page, and check afterwards that its monitoring metrics are collected. + +### Cause 5: DefaultTasksMax must be infinity, and the business cluster ports + +> **Warning:** `systemctl daemon-reexec` restarts the systemd manager in place; running services keep running, but a broken `/etc/systemd/system.conf` affects every unit started afterwards. Apply it before the node joins, after backing up the file. Rollback: restore the backup and run `systemctl daemon-reexec` again. + +```bash +sudo cp /etc/systemd/system.conf /root/system.conf.bak +if grep -qE '^\s*#?\s*DefaultTasksMax=' /etc/systemd/system.conf; then + sudo sed -ri 's/^\s*#?\s*DefaultTasksMax=.*/DefaultTasksMax=infinity/' /etc/systemd/system.conf +else + echo 'DefaultTasksMax=infinity' | sudo tee -a /etc/systemd/system.conf +fi +sudo systemctl daemon-reexec +systemctl show --property=DefaultTasksMax +``` + +The expected output is `DefaultTasksMax=infinity`. Units that are already running keep the old value until they restart or the node reboots; because node preprocessing happens before installation, containerd and kubelet start with the new value. + +Open ports 6443 and 11780 for the business cluster as listed above. + +### Cause 6: the deployment needs root or sudo on the nodes + +Give the deployment user root, or sudo without a password prompt (the availability check runs sudo non-interactively): + +```bash +id +sudo -l -U +``` + +## Diagnostic Steps + +Run on the host before adding it, and compare with a node already in the cluster: + +```bash +cat /etc/os-release; uname -r +cat /proc/cmdline +cat /sys/kernel/mm/transparent_hugepage/enabled +lsmod | grep -E 'ip_vs|nf_conntrack|geneve|openvswitch' +swapon --show +getenforce 2>/dev/null || true +systemctl is-active firewalld 2>/dev/null || true +sudo sshd -T | grep -i -E '^(usepam|usedns) ' +df -h /cpaas /var/lib/containerd +hostname +systemctl show --property=DefaultTasksMax +rpm -qa | grep -i -E 'docker|podman' || true +``` + +Expected results: `[never]` in the transparent hugepage file, the listed modules loaded, an empty `swapon --show`, `usepam no` and `usedns no`, `DefaultTasksMax=infinity`, a lowercase hostname, and no preinstalled Docker or podman packages. A missing module or an active `UsePAM` appears in the availability check as a failure or warning. diff --git a/docs/en/solutions/TektonHub_cannot_be_viewed_or_the_pipeline_task_page_is_blank_after_a_Pipelines_upgrade_on_ACP.md b/docs/en/solutions/TektonHub_cannot_be_viewed_or_the_pipeline_task_page_is_blank_after_a_Pipelines_upgrade_on_ACP.md new file mode 100644 index 000000000..cd89db40c --- /dev/null +++ b/docs/en/solutions/TektonHub_cannot_be_viewed_or_the_pipeline_task_page_is_blank_after_a_Pipelines_upgrade_on_ACP.md @@ -0,0 +1,107 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "3.14" + - "3.16" + - "4.2" + - "4.3" +--- + +# TektonHub cannot be viewed, or the pipeline task page is blank after a Pipelines upgrade on ACP + +## Issue + +- TektonHub cannot be opened, and uninstalling the Tekton operator and installing it again fails. +- After a Pipelines upgrade the pipeline task page is blank, and `kubectl get tektonhub.operator.tekton.dev -A` returns `No resources found`, although `TektonConfig.spec.profile` is already `all`. + +## Root Cause + +### Cause 1: TektonHub cannot be viewed and the operator reinstall fails because the Triggers webhook Role is missing + +The Tekton Triggers webhook runs with its own ServiceAccount, which is granted its permissions by a `Role` and `RoleBinding`. When those two objects are missing (for example after an uninstall that removed them), the webhook cannot work, the operator install does not finish, and TektonHub stays unreachable. Restoring the `Role` and `RoleBinding` and restarting the webhook brings the operator install and TektonHub back. + +### Cause 2: pipeline task page blank after a Pipelines upgrade because AUTOINSTALL_TEKTONHUB is false + +The task page reads tasks from TektonHub. ConfigMap `tekton-config-defaults` in namespace `tekton-operator` has `AUTOINSTALL_TEKTONHUB` set to `"false"`. With that default the operator does not create the TektonHub custom resource, even with profile `all`, so no Hub exists and the page is blank. The value is injected into the operator's environment and is not reloaded while the Pod runs, so the operator Pod has to be recreated after the change; deleting the Pod is preferred over `rollout restart`. + +This applies to Tekton operator versions that still ship the built-in TektonHub. Check that the installed Pipelines version provides TektonHub before turning the flag on. + +## Resolution + +### Cause 1: TektonHub cannot be viewed and the operator reinstall fails because the Triggers webhook Role is missing + +1. Find the Triggers webhook and confirm that its `Role` and `RoleBinding` are absent: + + ```bash + kubectl get deploy,pods -A | grep -iE 'triggers-webhook|tekton-triggers' + kubectl get role,rolebinding -A | grep -i trigger + kubectl -n logs deploy/ --tail=80 + ``` + + A missing Role for the webhook ServiceAccount, or a `forbidden` error in that log, matches this cause. + +2. Export the same `Role` and `RoleBinding` from a healthy cluster that runs the same Tekton operator version, or take them from the Triggers release manifests of that version, and apply them here. + + > **Warning:** Applying a Role and RoleBinding grants the verbs in that file to the webhook ServiceAccount. A copy from a different operator version can be too wide or too narrow. Precondition: only when the Tekton operator reinstall fails after an uninstall and these Triggers webhook objects are missing. Keep the YAML you applied. Rollback: delete the Role and RoleBinding you added. + + ```bash + kubectl apply -f + kubectl apply -f + ``` + +3. Restart the webhook. + + > **Warning:** Restarting the Triggers webhook drops admission for trigger resources until the new Pod is Ready; creating or changing EventListeners and other trigger objects fails in that window. Precondition: only after the Role and RoleBinding from step 2 are in place. Rollback: none for the restart; delete the objects from step 2 if they were the wrong ones. + + ```bash + kubectl -n rollout restart deploy/ + kubectl -n rollout status deploy/ + ``` + +4. Retry the operator install. TektonHub should open, and the webhook Pod should stay Ready. + +### Cause 2: pipeline task page blank after a Pipelines upgrade because AUTOINSTALL_TEKTONHUB is false + +1. Confirm that no TektonHub exists, the profile is `all`, and the flag is `false`: + + ```bash + kubectl get tektonhub.operator.tekton.dev -A + kubectl get tektonconfig -o jsonpath='{range .items[*]}{.metadata.name}{" profile="}{.spec.profile}{"\n"}{end}' + kubectl -n tekton-operator get configmap tekton-config-defaults -o jsonpath='{.data.AUTOINSTALL_TEKTONHUB}{"\n"}' + ``` + +2. Save the ConfigMap, set the flag, and recreate the operator Pod. + + > **Warning:** Setting `AUTOINSTALL_TEKTONHUB` to `true` makes the operator create TektonHub (`tektonhub/hub`, with its Deployments in `tekton-pipelines`) and pull its images; deleting the operator Pod pauses Tekton component reconciliation until the new Pod is Ready. Precondition: only when no TektonHub exists, `TektonConfig.spec.profile` is `all`, and `AUTOINSTALL_TEKTONHUB` is `false`. Rollback: apply the saved ConfigMap and delete the operator Pod again; if `tektonhub/hub` remains, delete it with `kubectl delete tektonhub hub`, which removes the Hub UI. + + ```bash + kubectl -n tekton-operator get configmap tekton-config-defaults -o yaml > /tmp/tekton-config-defaults.yaml + kubectl -n tekton-operator patch configmap tekton-config-defaults --type merge -p '{"data":{"AUTOINSTALL_TEKTONHUB":"true"}}' + kubectl -n tekton-operator delete pod -l name=tekton-operator + ``` + +3. Wait for TektonHub and check its resources: + + ```bash + kubectl wait --for=condition=Ready tektonhub/hub --timeout=600s + kubectl get tektonhub.operator.tekton.dev -A + kubectl -n tekton-pipelines get svc,deploy,pod | grep tekton-hub + ``` + +4. Reload the pipeline task page. + +## Diagnostic Steps + +```bash +kubectl get tektonhub.operator.tekton.dev -A +kubectl get csv -A | grep -i tekton +kubectl get role,rolebinding -A | grep -i trigger +kubectl -n tekton-operator get configmap tekton-config-defaults -o jsonpath='{.data.AUTOINSTALL_TEKTONHUB}{"\n"}' +``` + +- The Tekton operator reinstall fails, and no Role or RoleBinding exists for the Triggers webhook: Cause 1. +- `No resources found` for TektonHub, profile `all`, and `AUTOINSTALL_TEKTONHUB` printed as `false`: Cause 2. +- A TektonHub object exists and its Pods are Running while the page is still blank: neither cause; check the Hub API Pod log before installing anything. diff --git a/docs/en/solutions/Which_operating_systems_kernels_and_CPU_architectures_can_run_in_an_ACP_cluster.md b/docs/en/solutions/Which_operating_systems_kernels_and_CPU_architectures_can_run_in_an_ACP_cluster.md new file mode 100644 index 000000000..59eff3ae1 --- /dev/null +++ b/docs/en/solutions/Which_operating_systems_kernels_and_CPU_architectures_can_run_in_an_ACP_cluster.md @@ -0,0 +1,115 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - "3.10" + - "3.14" + - "3.16" + - "4.1" + - "4.2" + - "4.3" +--- + +# Which operating systems, kernels, and CPU architectures can run in an ACP cluster + +## Issue + +Before adding a node, upgrading a node kernel, or mixing node types, the following questions come up: + +- Is a newer Kylin kernel (for example for a security patch) supported? A newer openEuler release? Rocky Linux 10.1? An Ubuntu HWE kernel? +- Can CentOS and Kylin nodes, or Kylin and Ubuntu nodes, share one cluster, with some nodes running only ALB? +- Can x86 and ARM nodes share one 4.x cluster? Does a Hygon C86 cluster accept x86 nodes? Can an ARM global cluster manage Intel x86 servers? Are Kunpeng 920 (aarch64) hosts supported? +- The kernel of a new node changed after a reboot and its NIC stopped working. Did the platform initialization upgrade the kernel? + +Related articles: "Preparing a host before adding it to an ACP cluster: node check items, disks, init script, DefaultTasksMax, root access"; "Shutting down or rebooting ACP nodes for maintenance" covers rebooting nodes one by one for a kernel or OS upgrade. + +## Root Cause + +### Cause 1: only the OS releases and kernels in the version's baseline are supported + +Each platform release documents its own verified OS and kernel list in the node preprocessing section of its installation documentation. A node outside that list can fail the availability check or run with components that do not work on it. + +Known baselines and answers: + +- **3.14.2**, on both ARM (Kunpeng 920) and x86: Kylin Linux Advanced Server V10, V10-SP1, and V10-SP2, with the official built-in kernels of those releases only. The kernel `A.B.C` must be `4.19.90`; the suffix after `-` may differ, so the full build string does not have to match a tested kernel. Tested kernels include `4.19.90-11.ky10`, `4.19.90-23.8.v2101.ky10`, and `4.19.90-24.4.v2101.ky10` on both architectures, and `4.19.90-17.ky10.aarch64` on ARM. SP1 `4.19.90-23.72` and SP2 `4.19.90-25.63` fit the rule when each is the official built-in kernel of its release. V10-SP3 is not in the 3.14 baseline, so an SP3 kernel such as `4.19.90-52.64` is not supported on 3.14.2 even though its `A.B.C` is `4.19.90`. A vendor-fixed or already-installed kernel that is not an official built-in kernel of V10, V10-SP1, or V10-SP2 (for example `4.19.90-52.22.v2207.ky10` on aarch64) is outside the tested list, and stable operation is not guaranteed. +- **3.14.1**: Kylin V10 / SP1 / SP2, Ubuntu 20.04 / 22.04, and CentOS 7.6–7.9 with a kernel from `3.10.0-1127` up to but not including 3.11. Oracle Linux 9.7 is not adapted (`kube-ovn-cni` crashes on its ipset), and Ubuntu 24.04 is not verified. +- **4.2.3** supports a broader OS and kernel list, which adds Kylin V10-SP3 and MicroOS 5.5, among others. +- **Ubuntu**: the stock kernels `5.4.0-135-generic` (Ubuntu 20.04) and `5.15.0-56-generic` (Ubuntu 22.04) are the baseline; HWE kernels such as `6.8.0-106-generic` are not supported (on 4.1.2, such a server was reinstalled with a 5.15 kernel). +- **Rocky Linux 10.1** is not on the supported OS list of ACP 4.3. The availability check shows `检查操作系统:未支持的操作系统: Rocky Linux 10.1` (OS check: unsupported operating system), `EnsurePrepareNode` passes, and the add stops at `EnsureContainerRuntime` with `FailedInit`. +- **openEuler 24.03 (LTS-SP4)** is reported as an unsupported version by the 4.x availability check. +- On ACP 4.2.1, a failure prompt shown while adding a node whose OS version is listed as supported is non-blocking; the deployment can be continued past it. + +### Cause 2: mixed operating systems or CPU architectures in one cluster + +- Nodes of different operating systems can share a cluster as long as every node has the same CPU architecture and an OS within the version's baseline; the platform does not require one OS for all nodes. On 3.16 and 3.10, CentOS and Kylin nodes can share a cluster. One cluster may contain both Kylin and Ubuntu nodes, and dedicating some Kylin nodes to ALB only is feasible. +- On ACP 4.x, x86 and ARM nodes in one cluster need the multi-architecture (heterogeneous) images, uploaded once from the multi-architecture product package of the matching version. The 3.x-era step of setting the label `cpaas.io/node-arch-constraint` to `""` on the cluster's `clusters.platform.tkestack.io` resource on global is no longer needed on 4.x; the label is not present by default. An x86-only installation rejects ARM nodes with `硬件架构不匹配` (hardware architecture mismatch). +- A Hygon C86 cluster supports adding x86 nodes. +- A platform whose global cluster runs on ARM, with a business cluster of ARM and C86 servers, can manage Intel x86 servers by putting them in a separately created cluster instead of adding them to the existing mixed business cluster; the x86 package is compatible for managing those servers. +- Hosts with the aarch64 instruction set on Kunpeng 920 are supported in principle; no validated configuration or version is stated for that answer, and it says nothing about mixing x86 and aarch64 in one cluster. + +### Cause 3: the kernel of a new node changed after a reboot because of automatic OS updates, not the platform + +The platform initialization script does not upgrade the node kernel. When a node comes up on a newer kernel after a reboot (for example `5.15.0-187` instead of `5.15.0-119` on Ubuntu) and its NIC then fails on a kernel compatibility problem, the operating system's automatic updates are the likely source: the Unattended Upgrades service installs new kernel packages, and the next reboot boots them. + +## Resolution + +### Cause 1: only the OS releases and kernels in the version's baseline are supported + +1. Before adding a node or upgrading its kernel, read the OS and kernel on every node and compare them with the supported list of your platform version: + + ```bash + kubectl get nodes -o custom-columns=NAME:.metadata.name,OS:.status.nodeInfo.osImage,KERNEL:.status.nodeInfo.kernelVersion + cat /etc/os-release + uname -r + ``` + +2. On 3.14.2, an official built-in kernel of Kylin V10 / SP1 / SP2 is within the baseline when the `A.B.C` in `uname -r` is `4.19.90`. A kernel from another `A.B.C`, any SP3 kernel, or a vendor-fixed kernel outside the tested list is not. +3. For an OS that is not in the list of your version (for example Rocky Linux 10.1 on 4.3), reinstall the host with a supported OS before adding it; the add cannot be completed on it. +4. Reboot nodes one at a time for a kernel upgrade, as described in "Shutting down or rebooting ACP nodes for maintenance". + +### Cause 2: mixed operating systems or CPU architectures in one cluster + +1. Read the architecture of each host; `uname -m` prints `x86_64` on Hygon C86 and Intel x86 hosts and `aarch64` on Kunpeng hosts, and that value decides which product package a node needs: + + ```bash + uname -m + kubectl get nodes -o custom-columns=NAME:.metadata.name,ARCH:.status.nodeInfo.architecture + ``` + +2. For mixed CPU architectures on 4.x, install or upgrade from the multi-architecture product package of your version and upload its images; there is no label to change. To confirm the label is absent on an existing 4.x cluster, run on the global cluster: + + ```bash + kubectl get clusters.platform.tkestack.io -o jsonpath='{.metadata.labels}' | grep -o 'cpaas.io/node-arch-constraint[^,]*' || echo "no node-arch-constraint label" + ``` + +3. For mixed operating systems, check that every node's OS and kernel are in the version's baseline (Cause 1). + +### Cause 3: the kernel of a new node changed after a reboot because of automatic OS updates, not the platform + +1. Confirm the source of the kernel change on the node: + + ```bash + uname -r + ls /boot | grep -E 'vmlinuz|initrd' + systemctl status unattended-upgrades --no-pager + cat /etc/apt/apt.conf.d/20auto-upgrades + grep -h -E 'Start-Date|linux-image' /var/log/apt/history.log 2>/dev/null | tail -n 20 + sudo zgrep -h -i 'linux-image' /var/log/unattended-upgrades/unattended-upgrades.log* 2>/dev/null | tail -n 20 + ``` + + `APT::Periodic::Unattended-Upgrade "1";` in `20auto-upgrades` means automatic upgrades are enabled, and a `linux-image` package in the logs with a date matching the change confirms the source. + +2. Hand the NIC recovery and the kernel analysis to the owners of the OS and the infrastructure. Whatever way a kernel got onto a node, it still has to be in the version's baseline (Cause 1). + +## Diagnostic Steps + +```bash +kubectl get nodes -o custom-columns=NAME:.metadata.name,OS:.status.nodeInfo.osImage,KERNEL:.status.nodeInfo.kernelVersion,ARCH:.status.nodeInfo.architecture +cat /etc/os-release; uname -r; uname -m +``` + +- The availability check reports `未支持的操作系统` (unsupported operating system), or the planned OS or kernel is not in the supported list of the version: Cause 1. +- A question about mixing operating systems or architectures, or an add rejected with `硬件架构不匹配`: Cause 2. +- A kernel in `uname -r` that differs from the one the node had at initialization, with no platform action in between, and Unattended Upgrades enabled: Cause 3. diff --git a/docs/en/solutions/etcd-sync_check_reports_localMissedKeys_or_localSurplusKeys_or_what_a_primary_standby_global_pair_replicates_on_ACP.md b/docs/en/solutions/etcd-sync_check_reports_localMissedKeys_or_localSurplusKeys_or_what_a_primary_standby_global_pair_replicates_on_ACP.md new file mode 100644 index 000000000..54c5b8b5d --- /dev/null +++ b/docs/en/solutions/etcd-sync_check_reports_localMissedKeys_or_localSurplusKeys_or_what_a_primary_standby_global_pair_replicates_on_ACP.md @@ -0,0 +1,152 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" + - "4.3" +--- + +# etcd-sync check reports localMissedKeys or localSurplusKeys, or what a primary/standby global pair replicates on ACP + +## Issue + +On the standby global cluster of a primary/standby (disaster recovery) setup, the sync check of the etcd Synchronizer (`curl "http:///check"`) or a daily inspection lists keys under `LOCAL ETCD missed keys` / `localMissedKeys` or `LOCAL ETCD surplus keys` / `localSurplusKeys`, and it is not clear whether they need action. Examples: + +- Right after disaster recovery is set up: `localMissedKeys [0]` and `localSurplusKeys [2]`, the two keys being `/registry/auth.alauda.io/disabletokens/` and `/registry/monitoring.coreos.com/servicemonitors/cpaas-system/etcd-sync-monitor`. +- A daily inspection with `localMissedKeys [4]` and `localSurplusKeys [88]`. +- `/registry/secrets/cpaas-logging/clickhouse-operator` stays in the missed list after the `etcd-sync` Pod is restarted several times. +- Dozens of missed keys under `/registry/ait.alauda.io/notifications/...`, or a question about which objects the pair replicates, or whether the nodes of the two clusters need full network connectivity. + +Missed keys exist on the primary and not on the standby. Surplus keys exist only on the standby. + +Related articles: "etcd Synchronizer plugin install stuck or etcd-sync does not start on the standby global cluster on ACP"; "etcd-sync Pod stuck in ContainerCreating with failed to sync secret cache on the standby global cluster on ACP". + +## Root Cause + +### Cause 1: platform surplus keys and rotated platform objects are expected differences + +Some keys differ between the clusters by design and need no action: + +- `/registry/monitoring.coreos.com/servicemonitors/cpaas-system/etcd-sync-monitor`: the ServiceMonitor of the etcd Synchronizer, which is installed only on the standby, so the primary has no such key. +- `/registry/auth.alauda.io/disabletokens/`: created when an account logs out, to ban that user's unexpired tokens. +- dex `authrequests` and `refreshtokens`, and `auth.alauda.io` sessions and groups. +- Rotated certificate and token Secrets in platform namespaces, for example `cpaas-system/dex.tls-*`, `etcd-ca-*`, `base-operator.tls-*`, `argocd-manager-token-*`, and operator service certificates such as `/-service-cert`. +- Platform alert and monitoring rules (`ait.alauda.io/alertrules/cpaas-system/cpaas-*-rules`, `monitoring.coreos.com/prometheusrules/cpaas-system/cpaas-*-rules`, `operator.victoriametrics.com/vmrules/cpaas-system/cpaas-*-rules`), and the etcd-sync monitor dashboard. +- Inspection results (`ait.alauda.io/inspections/inspection--` and `inspectiondetails/...`), a `cluster.alauda.io/moduleinfoes/` entry, and an empty `productentries` key. + +With `localMissedKeys [0]`, the synchronization is normal; surplus keys from the list above do not affect business workloads or platform use. A daily inspection whose 4 missed and 88 surplus keys all fall into these classes needs no handling either. + +### Cause 2: a few missed keys clear after the etcd-sync leader Pod is restarted + +Missed keys are often caused by garbage collection during synchronization, when objects arrive in an order that leaves their owner references unresolved. Restarting the current etcd-sync leader Pod runs a full sync again and clears them. + +### Cause 3: the clickhouse-operator Secret stays missed and a restart does not create it + +`localMissedKeys [1]`: `/registry/secrets/cpaas-logging/clickhouse-operator`, a Secret with an `ownerReferences` entry of kind `ClusterServiceVersion` (for example `clickhouse-operator.v4.2.1`). Restarting the `etcd-sync` Pod more than once leaves the key missing. The standby copy has to be created by hand, with its owner reference pointing to the CSV on the standby. + +### Cause 4: plugins, feature gates, component YAML edits, OperatorHub installs, and customized keys are not in the sync set + +From 4.2 on, the sync program runs on the standby cluster and pulls from the primary: a full sync when it starts, then incremental sync. Only the keys listed in the ConfigMap that the sync workload mounts are replicated. Installed plugins, feature-gate settings, YAML edits of platform components, and OperatorHub installs are not replicated; each of those changes has to be applied on both clusters. Customized keys are also outside the set, for example custom notification objects (`/registry/ait.alauda.io/notificationgroups//...`, `notifications/...`, `notificationtemplates/...`). Their absence on the standby is expected, not a failed sync. + +Losing one primary etcd member does not stop synchronization. Losing two or more interrupts it until the primary recovers; the sync Pod restart then runs a full sync again. + +### Cause 5: which network the primary and standby global clusters need + +The two clusters do not need full node-to-node connectivity. The reference design: + +- Both global clusters are installed with the same platform access domain. Normally it resolves to the primary cluster's VIP; on failover it is re-resolved to the standby cluster's VIP. +- Each cluster has its own VIP, and each VIP forwards TCP `443` (and `80` when the platform is accessed over HTTP), `6443`, `2379` (unless the etcd endpoints are reached directly), and `11443` to that cluster's own control-plane nodes. +- Those ports are allowed in both directions between the primary and standby networks, because the roles reverse after failover. +- etcd data is copied from the primary to the standby by the etcd Synchronizer. + +## Resolution + +### Cause 1: platform surplus keys and rotated platform objects are expected differences + +Compare each listed key with the classes above. If `localMissedKeys` is `0` and every surplus key belongs to those classes, no action is needed. Do not delete these keys from the standby. Only a difference in a key that the business itself writes needs to be reviewed with the operations team before anything is deleted. + +### Cause 2: a few missed keys clear after the etcd-sync leader Pod is restarted + +1. Find the current leader Pod: + + ```bash + kubectl get po -n cpaas-system -l app=etcd-sync + kubectl get lease -n cpaas-system etcd-sync-mirror -o jsonpath='{.spec.holderIdentity}{"\n"}' + ``` + + On versions without this Lease, restart one of the `etcd-sync` Pods. + +2. Restart it. + + > **Warning:** Deleting the leader Pod restarts the sync process; synchronization pauses until a Pod takes over and runs a full sync. Precondition: only when the missed list is small and does not name a key covered by Cause 3. Rollback: none needed; the Deployment recreates the Pod. + + ```bash + kubectl delete pod -n cpaas-system + ``` + +3. Run the check again. A business key that is still missing after the restart is Cause 3 or Cause 4, not a reason for further restarts. + +### Cause 3: the clickhouse-operator Secret stays missed and a restart does not create it + +1. On the primary, export the Secret. On the standby, read the uid of the ClickHouse operator CSV: + + ```bash + # primary + kubectl -n cpaas-logging get secret clickhouse-operator -o yaml > /tmp/clickhouse-operator.secret.yaml + # standby + kubectl -n cpaas-logging get csv | grep -i clickhouse + kubectl -n cpaas-logging get csv -o jsonpath='{.metadata.uid}{"\n"}' + ``` + +2. In the saved file, remove `metadata.creationTimestamp`, `metadata.resourceVersion`, `metadata.uid`, and `metadata.managedFields`, and set the `uid` under `ownerReferences` (kind `ClusterServiceVersion`) to the standby CSV uid just printed. Use the CSV name that `get csv` shows on the standby. +3. Apply it on the standby. + + > **Warning:** This creates or overwrites `clickhouse-operator` in `cpaas-logging` on the standby. Precondition: only when that name is the missed key and an `etcd-sync` restart did not create it. A wrong owner uid leaves the operator unable to adopt the Secret. Rollback: `kubectl -n cpaas-logging delete secret clickhouse-operator` if the apply created it; if a Secret existed before, save it first and re-apply that copy. + + ```bash + kubectl apply -f /tmp/clickhouse-operator.secret.yaml + ``` + + Expected: `secret/clickhouse-operator created`. Run the check again; the key is no longer listed. + +### Cause 4: plugins, feature gates, component YAML edits, OperatorHub installs, and customized keys are not in the sync set + +1. Read the sync scope from the ConfigMap that the sync workload mounts, on the cluster that runs the sync program: + + ```bash + kubectl get deploy -n cpaas-system | grep -E 'etcd-sync|etcd-mirror' + kubectl -n cpaas-system get deploy -o jsonpath='{.spec.template.spec.volumes}{"\n"}' + kubectl -n cpaas-system get configmap -o yaml + ``` + +2. Apply plugin installs, feature-gate changes, platform component YAML edits, and OperatorHub installs on the other global cluster as well. +3. Handle missing customized keys with the team that owns the customization; they are not synchronized automatically. +4. Before a planned switchover, take an etcd backup on both the active and the standby cluster. + +### Cause 5: which network the primary and standby global clusters need + +From a host outside each cluster, check that the VIP forwards the ports to its own control-plane nodes, once for the primary VIP and once for the standby VIP: + +```bash +for p in 443 6443 2379 11443; do nc -vz -w 3 $p; done +``` + +Also check the same ports from the standby network to the primary and back. + +## Diagnostic Steps + +Run the sync check on a control-plane node of the standby global cluster. The ClusterIP of the `etcd-sync-monitor` Service is wrapped in brackets when it is IPv6: + +```bash +mirror_svc=$(kubectl get svc -n cpaas-system etcd-sync-monitor -o jsonpath='{.spec.clusterIP}') +if [[ $mirror_svc =~ ^[0-9a-fA-F:]+$ ]]; then mirror_host="[$mirror_svc]"; else mirror_host="$mirror_svc"; fi +curl -g "http://${mirror_host}/check" +``` + +- `localMissedKeys [0]` and only platform surplus keys such as `etcd-sync-monitor` and `disabletokens`: Cause 1, normal. +- A few missed keys of platform objects: Cause 2. +- `clickhouse-operator` still missed after a restart: Cause 3. +- Missing plugins, feature gates, or customized keys such as `ait.alauda.io/notifications`: Cause 4. +- A question about network connectivity between the two clusters: Cause 5. diff --git a/docs/en/solutions/etcd_Synchronizer_plugin_install_stuck_or_etcd-sync_does_not_start_on_the_standby_global_cluster_on_ACP.md b/docs/en/solutions/etcd_Synchronizer_plugin_install_stuck_or_etcd-sync_does_not_start_on_the_standby_global_cluster_on_ACP.md new file mode 100644 index 000000000..6970559ac --- /dev/null +++ b/docs/en/solutions/etcd_Synchronizer_plugin_install_stuck_or_etcd-sync_does_not_start_on_the_standby_global_cluster_on_ACP.md @@ -0,0 +1,168 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.2" +--- + +# etcd Synchronizer plugin install stuck or etcd-sync does not start on the standby global cluster on ACP + +## Issue + +The etcd Synchronizer cluster plugin of a primary/standby (disaster recovery) global setup does not come up on the standby global cluster: + +- Installing or reinstalling the plugin (Administrator → Marketplace → Cluster Plugins → `global`) stays in the installing state, and `kubectl get ars -A` shows: + + ```text + cpaas-system etcd-sync ChartValuesRenderFailed failed to render values for chart ait/chart-etcd-sync-v4.2.1, err: run script error: exit status 2, stderr: base64: invalid input... + ``` + +- During a disaster-recovery upgrade (for example 4.0.9 to 4.2.4) the sync plugin is deployed on the standby global cluster and never becomes Ready. +- `etcd-sync` fails to deploy while the historical primary cluster still runs an `etcd-mirror` Deployment. +- The `ars` of the plugin on the standby times out calling the etcd sync API. +- The plugin has to be removed or installed again with `kubectl`. + +Related articles: "etcd-sync check reports localMissedKeys or localSurplusKeys, or what a primary/standby global pair replicates on ACP" covers a running synchronizer; "etcd-sync Pod stuck in ContainerCreating with failed to sync secret cache on the standby global cluster on ACP" covers `invalid padding on input`. + +## Root Cause + +### Cause 1: ChartValuesRenderFailed with base64: invalid input because the token was entered in plaintext + +The installation takes a token for the active global cluster as a parameter. The chart's values-rendering script expects that token base64-encoded and decodes it. A plaintext token makes the script fail with `base64: invalid input`, the AppRelease stays in `ChartValuesRenderFailed`, and the installation never reaches the point of creating Pods. Replacing the token with its base64-encoded form lets the installation complete. + +On plugin versions that read the token from the `etcd-sync-active-cluster-token` Secret instead of a parameter, the Secret is created from the plaintext token as the installation documentation describes. + +### Cause 2: Secret etcd-ca is missing in cpaas-system, the sync plugin does not start + +The sync plugin needs the Secret `etcd-ca` in `cpaas-system`. When it is missing on the primary and the standby (for example after a 4.0.9 to 4.2.4 disaster-recovery upgrade), the plugin deployed on the standby does not become Ready. Creating the Secret on both clusters from the etcd CA certificate and key, then redeploying the plugin, brings it up. + +### Cause 3: a leftover etcd-mirror Deployment on the historical primary blocks etcd-sync + +An `etcd-mirror` Deployment from an earlier synchronization setup is still running on the cluster that used to be the primary. It is not the current sync plugin, and while it exists, `etcd-sync` fails to deploy. Deleting the leftover Deployment clears the install. + +### Cause 4: the plugin was installed while the platform domain still resolved to the standby + +The standby cluster is installed while the platform domain temporarily points at the standby VIP, and the domain has to point back at the primary VIP before the etcd Synchronizer is installed. When the plugin is installed while the domain still resolves to the standby, the standby's `ars` times out calling the etcd sync API. After the domain resolves to the primary again and the plugin is redeployed, the timeout clears. + +### Cause 5: the plugin is installed and uninstalled through its ModuleInfo + +The etcd Synchronizer, like other cluster plugins, is represented by a ModuleInfo (`minfo`). Deleting that object uninstalls the plugin, and creating it again installs it. + +## Resolution + +### Cause 1: ChartValuesRenderFailed with base64: invalid input because the token was entered in plaintext + +1. Confirm the render failure: + + ```bash + kubectl get ars -n cpaas-system etcd-sync + kubectl describe ars -n cpaas-system etcd-sync | grep -iE 'reason|message|base64' + ``` + +2. Encode the token without a trailing newline: + + ```bash + echo -n '' | base64 -w0; echo + ``` + +3. Enter the encoded value as the token in the plugin's installation parameters and install again. +4. Check that the AppRelease leaves the failed state and the Pods run: + + ```bash + kubectl get ars -n cpaas-system etcd-sync + kubectl get po -n cpaas-system -l app=etcd-sync + ``` + +### Cause 2: Secret etcd-ca is missing in cpaas-system, the sync plugin does not start + +1. On the primary and on the standby, check the Secret and the CA files on a control-plane node: + + ```bash + kubectl get secret -n cpaas-system etcd-ca + sudo ls -l /etc/kubernetes/pki/etcd/ca.crt /etc/kubernetes/pki/etcd/ca.key + ``` + +2. On each cluster where it is missing, create it from that cluster's etcd CA. + + > **Warning:** The Secret stores the etcd CA private key in `cpaas-system`. Precondition: only when `etcd-ca` is missing and the plugin does not start; if the Secret exists, do not recreate it. Rollback: `kubectl delete secret -n cpaas-system etcd-ca` for a Secret this command just created; do not delete one that other components already use. + + ```bash + kubectl create secret tls etcd-ca \ + --cert=/etc/kubernetes/pki/etcd/ca.crt \ + --key=/etc/kubernetes/pki/etcd/ca.key \ + -n cpaas-system + ``` + +3. Redeploy the sync plugin after both clusters have the Secret. + +### Cause 3: a leftover etcd-mirror Deployment on the historical primary blocks etcd-sync + +1. On the historical primary cluster, not on the standby, find the leftover: + + ```bash + kubectl get deploy -A | grep etcd-mirror + ``` + +2. Save and delete it. + + > **Warning:** Deleting `etcd-mirror` stops that old sync process. Precondition: only the leftover Deployment on the historical primary, only while the `etcd-sync` install is failing. Do not delete an `etcd-sync` Deployment in this step. Rollback: `kubectl apply -f /tmp/etcd-mirror.yaml`. + + ```bash + kubectl -n get deploy etcd-mirror -o yaml > /tmp/etcd-mirror.yaml + kubectl -n delete deploy etcd-mirror + ``` + +3. Retry the etcd-sync install. + +### Cause 4: the plugin was installed while the platform domain still resolved to the standby + +1. Check where the platform domain resolves, from a standby node and from inside the cluster: + + ```bash + getent hosts + kubectl exec -n cpaas-system deployments/sentry -- nslookup + kubectl get ars -A | grep -i etcd + ``` + +2. Point the domain at the primary global cluster's VIP. +3. Redeploy the etcd Synchronizer on the standby global cluster, as the installation documentation describes, and check that the `ars` timeout is gone. + +### Cause 5: the plugin is installed and uninstalled through its ModuleInfo + +1. Find the object: + + ```bash + kubectl get minfo -A | grep -i etcd-sync + ``` + +2. Save it and delete it to uninstall. + + > **Warning:** Uninstalling the etcd Synchronizer stops disaster-recovery synchronization. Uninstalling it while the standby is missing data that the primary holds can make owner references resolve incorrectly and lead to workload-cluster node `Machine` objects being deleted. Precondition: only when the sync check reports no missed or surplus keys that are unexplained, and the name is the etcd-sync plugin. Rollback: `kubectl apply -f /tmp/etcd-sync-minfo.yaml`. + + ```bash + kubectl get minfo -o yaml > /tmp/etcd-sync-minfo.yaml + kubectl delete minfo + ``` + + Add `-n ` to both commands if the list printed a namespace. + +3. To install it again, apply a ModuleInfo for `etcd-sync` taken from the saved object (remove `metadata.resourceVersion`, `metadata.uid`, and `status` first), not a hand-written manifest. + +## Diagnostic Steps + +```bash +kubectl get ars -A | grep -i etcd +kubectl get pods -A | grep -E 'etcd-sync|etcd-mirror' +kubectl get secret -n cpaas-system etcd-ca +kubectl get minfo -A | grep -i etcd +kubectl logs -n cpaas-system --tail=200 +``` + +- `ChartValuesRenderFailed` with `base64: invalid input`: Cause 1; no Pods exist yet. +- Plugin Pods not Ready and `etcd-ca` reported `NotFound`: Cause 2. +- Install fails and the historical primary still lists an `etcd-mirror` Deployment: Cause 3. +- `ars` times out on the etcd sync API, and the platform domain resolved to the standby when the plugin was installed: Cause 4. +- Removing or reinstalling the plugin from `kubectl`: Cause 5. diff --git a/docs/en/solutions/etcd_backup_dates_missing_in_the_console_backup_tab_gone_duplicate_copies_or_retention_to_change_on_ACP.md b/docs/en/solutions/etcd_backup_dates_missing_in_the_console_backup_tab_gone_duplicate_copies_or_retention_to_change_on_ACP.md new file mode 100644 index 000000000..ca2dd9371 --- /dev/null +++ b/docs/en/solutions/etcd_backup_dates_missing_in_the_console_backup_tab_gone_duplicate_copies_or_retention_to_change_on_ACP.md @@ -0,0 +1,172 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "4.0" + - "4.1" + - "4.2" +--- + +# etcd backup dates missing in the console, backup tab gone, duplicate copies, or retention to change on ACP + +## Issue + +- The etcd backup task looks like it skipped dates: every control-plane node has a schedule such as `0 18 * * *`, and some dates are absent from the console. +- The Administrator cluster-management page has no backup-management tab, so the backup policy and a restore cannot be started from that screen. +- Every day a new backup is written to `/cpaas-backup`, and it is not clear how long snapshots are kept or how to shorten the retention. A directory `backup_` sits next to `cpaas-backup` on `/`. +- The backup retention on a standby global cluster has to change without switching the platform domain to it. +- Each etcd backup run writes two copies. + +Related articles: "No default etcd backup on ACP: Cluster Enhancer not installed or etcd-backup-default not created"; "Scheduled etcd backups stop after an upgrade because the etcd-backup-default AdvancedCronJob stops triggering on ACP". + +## Root Cause + +### Cause 1: repeated label updates on one EtcdBackupRecord collapse the console list, the files are on disk + +The scheduled backups run (the `etcd-backup-default` job logs show the execution), and the `snapshot-etcd-*.db` files for the missing dates exist in the backup directory (for example `/mnt/cpaas-backup/` or `/cpaas/backup`). The `EtcdBackupRecord` objects and the audit log show the same resource being label-updated repeatedly, so the console lists one record of that name. The snapshots are not missing, and no backup has to be regenerated. + +### Cause 2: on 4.1 the backup-management tab was removed, the backup configuration still exists + +On 4.1 the backup-management page of cluster management has been removed. The backup itself keeps running: `cluster-enhancer-apiserver` Pods are `1/1 Running`, and the `EtcdBackupConfiguration` `etcd-backup-default` exists, is not paused, and keeps creating records. Records are viewed from the etcd backup monitoring dashboard (Operations Center → Monitoring → Dashboards → Switch → Cluster → etcd backup), and policy changes are made on the `EtcdBackupConfiguration` object. A restore screen has never been provided in the console; restoring etcd is not started from the UI. + +### Cause 3: retention is spec.localStorage.ttl, and backup_ is an upgrade copy + +Daily backups to `/cpaas-backup` are the normal behavior. Snapshot retention is `spec.localStorage.ttl` on the `EtcdBackupConfiguration` named `etcd-backup-default` (`apiVersion: enhancement.cluster.alauda.io/v1`), in seconds, for example `ttl: "7776000"` (90 days) with `localStorage.path: /cpaas-backup` and `schedule: 0 0 * * *`. The same object also carries the annotations `life` and `path`; retention is changed through `spec.localStorage.ttl`, not through the `life` annotation. The change applies without a component restart, but snapshots already past the new retention are removed on the next scheduled backup, not at the moment of the edit. In the console, the object is found under Resource Management; searching Custom Resource Definitions for `EtcdBackupConfiguration` returns nothing. + +`cpaas-backup` on `/` is the etcd backup data directory and has to be kept. `backup_` (for example `backup_` plus a date stamp) is written by a platform upgrade and exists only on clusters that were upgraded. It holds an `etcd` and a `kubernetes` subdirectory (with `admin.conf`, `kubeadm-config.yaml`, `encryption-provider.conf`, `kubelet.conf`, and the `manifests`, `pki`, `audit`, and `patches` directories). That upgrade copy can be deleted. + +### Cause 4: on a standby global cluster, retention is the LIFE variable of the etcd backup CronJob + +When the platform domain is not switched to the standby global cluster, its console backup page is not the path. The retention period there is the `LIFE` environment variable, in seconds, of the etcd backup CronJob (for example container `etcd-backup` running `/backup-local.sh` with `NAME=etcd-backup-default` and `LIFE="1296000"`, 15 days). + +### Cause 5: on 4.2.4 the etcd-backup-upgrade-k8s bcj writes a second copy + +Each run produces two copies because a `bcj` named `etcd-backup-upgrade-k8s` sits beside the regular backup configuration and fires too. Removing that bcj's annotations, labels, and owner references stops the second copy. Why both objects fire is not established, so treat this as a workaround. + +## Resolution + +### Cause 1: repeated label updates on one EtcdBackupRecord collapse the console list, the files are on disk + +Look on the control-plane nodes before scheduling another backup: + +```bash +kubectl get cronjob -A | grep etcd-backup +kubectl get etcdbackuprecord -A +kubectl get etcdbackupconfiguration -A -o yaml | grep -A3 localStorage +sudo ls -l /mnt/cpaas-backup/snapshot-etcd-*.db /cpaas/backup/snapshot-etcd-*.db /cpaas-backup/snapshot-etcd-*.db 2>/dev/null +``` + +Check the path printed on the configuration object as well. If the dated `snapshot-etcd-*.db` files exist, the single console row is a display effect of repeated label updates on one `EtcdBackupRecord`. Leave the files. If no file exists on any path, this cause does not apply. + +### Cause 2: on 4.1 the backup-management tab was removed, the backup configuration still exists + +1. Confirm that the backup is still in place: + + ```bash + kubectl get pods -A | grep cluster-enhancer-apiserver + kubectl get etcdbackupconfiguration -A -o yaml + ``` + +2. Read records from the etcd backup monitoring dashboard. +3. To change the policy, save the object, edit the saved copy, and apply it. + + > **Warning:** Applying a changed `EtcdBackupConfiguration` changes the schedule, path, or retention of the platform etcd backup; a bad path or a much smaller `ttl` can remove snapshots you still need on the next run. Precondition: only when you intend that policy change, and only after the backup copy exists. Rollback: apply the saved copy again before the next scheduled run. + + ```bash + kubectl get etcdbackupconfiguration -A -o yaml > /tmp/etcd-backup-default.yaml + cp /tmp/etcd-backup-default.yaml /tmp/etcd-backup-default.new.yaml + vi /tmp/etcd-backup-default.new.yaml + kubectl apply -f /tmp/etcd-backup-default.new.yaml + ``` + +### Cause 3: retention is spec.localStorage.ttl, and backup_ is an upgrade copy + +1. Read the current retention, path, and schedule. An empty NAMESPACE column means the object is cluster-scoped; drop `-n` then: + + ```bash + kubectl get etcdbackupconfiguration -A + kubectl get etcdbackupconfiguration etcd-backup-default -n -o yaml > /tmp/etcd-backup-default.yaml + grep -n -A3 'localStorage' /tmp/etcd-backup-default.yaml + ``` + +2. Set the retention you want, in seconds, as a quoted string like the existing value (`604800` is 7 days, `2592000` is 30 days). + + > **Warning:** Lowering `ttl` makes the next scheduled backup delete every snapshot older than the new value. Precondition: only when that retention is intended, after saving the object. Rollback: `kubectl apply -f /tmp/etcd-backup-default.yaml` before the next scheduled run; snapshots already removed by a run do not come back. Leave the `life` annotation alone; it is not the setting. + + ```bash + kubectl patch etcdbackupconfiguration etcd-backup-default -n --type=merge \ + -p '{"spec":{"localStorage":{"ttl":"604800"}}}' + ``` + + Old snapshots are removed by the next scheduled backup. + +3. To free the upgrade copy on `/`, check what it is first: + + ```bash + sudo ls -ld /cpaas-backup /backup_* 2>/dev/null || true + sudo ls /backup_ + ``` + + > **Warning:** Deleting `backup_` removes the configuration and etcd copy that the upgrade took, and cannot be undone. Precondition: only that directory, after `ls` shows it is the upgrade backup (`etcd` and `kubernetes` subdirectories) and not `cpaas-backup`, and only when the upgrade has completed. Rollback: none, unless you archived the directory elsewhere first. + + ```bash + sudo rm -rf /backup_ + ``` + +### Cause 4: on a standby global cluster, retention is the LIFE variable of the etcd backup CronJob + +1. Find the CronJob and save it: + + ```bash + kubectl get cronjob -A | grep -i etcd-backup + kubectl -n get cronjob -o yaml > /tmp/etcd-backup-cronjob.yaml + grep -n -A1 'name: LIFE' /tmp/etcd-backup-cronjob.yaml + ``` + +2. Set `LIFE` to the retention you want, in seconds. + + > **Warning:** A smaller `LIFE` removes older standby snapshots on the next run. Precondition: only on the CronJob whose container is `etcd-backup`, and only with the value you intend. Rollback: `kubectl apply -f /tmp/etcd-backup-cronjob.yaml`. + + ```bash + kubectl -n set env cronjob/ -c etcd-backup LIFE=1296000 + ``` + +### Cause 5: on 4.2.4 the etcd-backup-upgrade-k8s bcj writes a second copy + +1. Confirm the extra object: + + ```bash + kubectl get bcj -A | grep etcd-backup + ``` + +2. Save it, then clear its annotations, labels, and owner references. + + > **Warning:** This detaches `etcd-backup-upgrade-k8s` from its owner; the duplicate stops only as long as no controller writes those fields back. It does not delete snapshot files already written. Precondition: only when `etcd-backup-upgrade-k8s` exists and each run writes two copies. Rollback: `kubectl apply -f /tmp/etcd-backup-upgrade-k8s.yaml`. + + ```bash + kubectl -n get bcj etcd-backup-upgrade-k8s -o yaml > /tmp/etcd-backup-upgrade-k8s.yaml + kubectl -n annotate bcj etcd-backup-upgrade-k8s --all- + kubectl -n label bcj etcd-backup-upgrade-k8s --all- + kubectl -n patch bcj etcd-backup-upgrade-k8s --type=json -p='[{"op":"remove","path":"/metadata/ownerReferences"}]' + ``` + + If `ownerReferences` is already absent, skip the patch. The next backup run should write one copy. + +## Diagnostic Steps + +```bash +kubectl get etcdbackupconfiguration -A +kubectl get etcdbackuprecord -A +kubectl get cronjob -A | grep etcd-backup +kubectl get bcj -A | grep etcd-backup +``` + +- Dated snapshot files exist on disk while the console shows one `EtcdBackupRecord`: Cause 1. +- 4.1, no backup-management tab, `etcd-backup-default` present: Cause 2. +- A retention change, or `backup_` next to `cpaas-backup`: Cause 3. Expected: `spec.localStorage.ttl` in the YAML, and `etcd` and `kubernetes` subdirectories in `backup_`. +- Retention on a standby global cluster without a domain switch: Cause 4. +- Two copies per run and `etcd-backup-upgrade-k8s` in the `bcj` list: Cause 5. +- No `EtcdBackupConfiguration` at all: see the related article on the missing default backup. diff --git a/docs/en/solutions/violet_push_fails_with_no_package_found_code_000_HTTP_500_or_502_or_invalid_reference_format_on_ACP.md b/docs/en/solutions/violet_push_fails_with_no_package_found_code_000_HTTP_500_or_502_or_invalid_reference_format_on_ACP.md new file mode 100644 index 000000000..3dadf7a1b --- /dev/null +++ b/docs/en/solutions/violet_push_fails_with_no_package_found_code_000_HTTP_500_or_502_or_invalid_reference_format_on_ACP.md @@ -0,0 +1,220 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# violet push fails with no package found, code 000, HTTP 500 or 502, or invalid reference format on ACP + +## Issue + +Uploading an extension package or operator with `violet push` stops with an error: + +- `Error: no package found in ./plugins/.gz`, although the archive opens fine in an archive tool (for example `edp-keycloak-operator-v1.23.0.gz` downloaded from Alauda Marketplace), or `Error: no package found in installer-core-v4.3.2-hybrid.tar`. +- The push to the platform repository returns `code 000`, and afterwards ModuleInfo reports `moduleconfig not found`, although the previous version pushed to the same repository worked. +- A batch `violet push` of every extension package before an upgrade succeeds for most packages and fails for some with `500 Internal Server Error`. +- `violet push` skips the images that already exist and then repeats `copy error: writing blob: uploading layer chunked: received unexpected HTTP status: 502 Bad Gateway`. +- `violet push ... --dest-repo "https://:"` fails with `invalid destination name docker://https://:/...: invalid reference format`, and `docker login` to the platform registry on the same host fails with an X509 error. +- Pushing a chart package to the platform's built-in public chart repository with the platform address, username, and password fails with an invalid username or password. + +Related articles: "violet push goes to the wrong global cluster, a wrong package cannot be deleted, or violet or kubectl is missing on ACP" covers pushes that complete in the wrong place or cannot be undone; "violet push fails to parse a package, or a new operator version is skipped or missing on ACP" covers `unexpected EOF`; "violet push succeeds but the plugin is not published: Artifact, ModuleConfig, or chart missing on ACP" covers pushes that report success. + +## Root Cause + +### Cause 1: Error: no package found, the file is not in the violet package layout + +`violet push` accepts only packages in the Violet package layout. A package produced by the standard Violet process (`violet create`, then `violet package `) is named `PLUGIN_NAME.BUNDLE_VERSION.tgz`, for example `edp-keycloak-operator.v1.23.0.tgz`, and pushes successfully. A `.gz` downloaded from Alauda Marketplace that differs in structure gives `no package found`; renaming or repacking it as `.tgz` or `.tar.gz` gives the same error. The error also appears when the file is not a plugin package at all: `installer-core-v4.3.2-hybrid.tar` is an installer archive, so `violet` finds no package metadata in it. + +### Cause 2: code 000, the platform repository could not be reached, and ModuleInfo reports moduleconfig not found + +`code 000` means that no HTTP response was received: the host running `violet` could not reach the repository. The plugin artifact was never stored, so the ModuleConfig for that version was never created, and ModuleInfo cannot find it. Once the repository is reachable again, the push succeeds and the ModuleConfig appears. + +A host that cannot resolve the registry name fails the same way before any HTTP exchange, with an error such as `dial tcp: lookup : no such host`. + +### Cause 3: some packages of a batch push fail with 500 Internal Server Error + +Most images print `found image ... already exists, skip push`; the failing ones print: + +```text +copy error: copying image 2/2 from manifest list: writing blob: uploading layer chunked: received unexpected HTTP status: 500 Internal Server Error +push artifact .v to cluster global failed. error: copying image 2/2 from manifest list: ... +failed to push images via skopeo mode for package /.tgz +``` + +The same error can name `copying image 1/2`. After a failure, `violet push` continues with the remaining packages. The registry rejected individual layer uploads; pushing only the failed packages again after the run has finished completes them. + +### Cause 4: 502 Bad Gateway while uploading layers, the platform registry is OOMKilled and restarting + +The push skips the images that already exist and fails on the first layer it has to upload, again and again. The registry Pod that receives the push keeps restarting: its `registry` container shows `Last State: Terminated`, `Reason: OOMKilled`, `Exit Code: 137` (for example with limits `cpu: 1`, `memory: 2Gi`). Each restart cuts off the layer upload in progress, and the gateway in front of the registry answers `502`. After the registry's CPU and memory limits are raised, it stops restarting and the push completes. + +### Cause 5: invalid reference format because --dest-repo has https://, and docker login fails with X509 + +```text +ERRO invalid destination name docker://https://:/:: invalid reference format +WARN failed to push images via skopeo mode for package .tgz, error: invalid reference format +Error: failed to push one or more packages. +``` + +Two separate problems on the host that runs the push: + +- `violet` builds the destination image name from `--dest-repo`. An image reference cannot contain a URL scheme, so `--dest-repo https://:` makes every image fail with `invalid reference format`. Passwords that contain shell special characters also break when they are not single-quoted. +- `docker login` to the platform registry fails with an X509 error when the registry address is not configured in Docker's `daemon.json` on that host. + +### Cause 6: a chart push to the built-in public chart repository is rejected as invalid username or password + +Pushing the raw chart to the built-in `public chart` repository with the platform credentials is rejected as an invalid username or password. Packaging the chart with `violet` and pushing it with `violet push` succeeds. + +## Resolution + +### Cause 1: Error: no package found, the file is not in the violet package layout + +1. If you already have a `.tgz` named `PLUGIN_NAME.BUNDLE_VERSION.tgz` (for example from Alauda technical support), push that file: + + ```bash + violet push --platform-address= --platform-username= --platform-password='' edp-keycloak-operator.v1.23.0.tgz + ``` + +2. Otherwise build the package with the standard process and push the generated `.tgz`: + + ```bash + violet create + violet package + ls *.tgz + ``` + +Repacking a Marketplace `.gz` by hand does not help. Do not push installer archives such as `installer-core--.tar` with `violet push`; they are not plugin packages. + +### Cause 2: code 000, the platform repository could not be reached, and ModuleInfo reports moduleconfig not found + +1. From the host that runs `violet`, check name resolution and reachability of the repository. Any HTTP status means the host is reachable; `000` or a connection error means it is not: + + ```bash + getent hosts + curl -k -s -o /dev/null -w '%{http_code}\n' https://:/v2/ + ``` + +2. Restore access to the repository: the network path, DNS (or an `/etc/hosts` entry on the push host), the proxy, or the repository service itself. +3. Push the same package again, then check that the ModuleConfig exists on `global`: + + ```bash + kubectl get moduleconfigs | grep + kubectl get moduleinfo | grep + ``` + + `moduleconfig not found` clears once the version has actually been pushed. + +### Cause 3: some packages of a batch push fail with 500 Internal Server Error + +1. Let the running push finish completely. +2. Collect the failed packages from its output (the `push artifact to cluster failed` and `failed to push images via skopeo mode for package ` lines). +3. Push only those: + + ```bash + mkdir -p ./retry + cp .tgz .tgz ./retry/ + violet push ./retry --platform-address= --platform-username= --platform-password='' + ``` + +If the same packages fail again with `500`, check the registry Pod as in Cause 4. + +### Cause 4: 502 Bad Gateway while uploading layers, the platform registry is OOMKilled and restarting + +1. Find the registry Pod on the cluster that the push targets and read why it restarted: + + ```bash + kubectl get pods -A | grep -i registry + kubectl -n describe pod | grep -E -A6 'Last State|Limits|Requests|Controlled By' + ``` + + `Reason: OOMKilled` with `Exit Code: 137` on the `registry` container, while the push repeats `502 Bad Gateway`, is this cause. + +2. Save the workload before changing it: + + ```bash + kubectl -n get deploy -o yaml > "$HOME/.before.yaml" + ``` + +3. Raise the registry CPU and memory limits. + + > **Warning:** Changing the registry resources restarts the registry Pod; image pulls and pushes against it fail until the new Pod is Ready, and Pods scheduled in that window can be in `ImagePullBackOff` for a while. Precondition: only when the `registry` container's last state is `OOMKilled` (exit code `137`) and the push fails with `502 Bad Gateway`; do it when no push or large rollout is running. Rollback: `kubectl apply -f "$HOME/.before.yaml"`. + + ```bash + kubectl -n set resources deploy -c registry --limits=cpu=,memory= + kubectl -n rollout status deploy + ``` + + Size the memory limit above the peak the container reached before the OOM. + +4. After a few minutes, check that the Deployment still carries the new limits. If they went back, the values are managed by the component that deploys the registry; collect the Deployment's labels and annotations and ask Alauda technical support where to set them. +5. When the registry stays Running, push the package again; images that already exist are skipped. + +### Cause 5: invalid reference format because --dest-repo has https://, and docker login fails with X509 + +1. Give `--dest-repo` as `host:port` with no scheme, and single-quote the passwords: + + ```bash + violet push .tgz \ + --platform-address \ + --platform-username \ + --platform-password '' \ + --dest-repo ":" \ + --username "" \ + --password '' \ + --debug + ``` + + The output should end with `successfully create artifact resources for package `. If an earlier failed attempt left an artifact behind, the push may need `--force`; read the Warning about `--force` in the related article on wrong packages before using it. + +2. For the X509 error on `docker login`, check the Docker daemon configuration on the push host: + + ```bash + cat /etc/docker/daemon.json + docker login : + ``` + + For a registry whose certificate the host does not trust, the Docker setting that takes a registry address is `insecure-registries`: + + ```json + { + "insecure-registries": [":"] + } + ``` + + > **Warning:** Docker reads `daemon.json` only when it starts. Restarting Docker stops the containers on this host unless `live-restore` is enabled, and an invalid `daemon.json` keeps Docker from starting. `insecure-registries` also turns off TLS verification for that registry from this host. Precondition: only on the host that runs `violet`, when `docker login` to the platform registry fails with an X509 error. Back the file up and merge the key into the existing content instead of replacing it. Rollback: restore the backup and restart Docker again. + + ```bash + cp /etc/docker/daemon.json /etc/docker/daemon.json.bak + vi /etc/docker/daemon.json + systemctl restart docker + docker login : + ``` + +### Cause 6: a chart push to the built-in public chart repository is rejected as invalid username or password + +Package the chart with the packaging flow of the `violet` version that matches the platform, then push the resulting package with `violet push` instead of pushing the raw chart with the platform credentials: + +```bash +violet push --platform-address= --platform-username= --platform-password='' .tgz +``` + +## Diagnostic Steps + +```bash +violet version +getent hosts +curl -k -s -o /dev/null -w '%{http_code}\n' https://:/v2/ +kubectl get pods -A | grep -i registry +``` + +- `no package found` on a Marketplace `.gz` or an installer `.tar`: Cause 1. +- `code 000`, or `lookup ... no such host`, then `moduleconfig not found`: Cause 2. Expected: the `curl` above prints `000` or a connection error. +- `500 Internal Server Error` for some packages of a batch push while the registry Pod is stable: Cause 3. +- `502 Bad Gateway` while uploading layers, with the registry Pod `OOMKilled`: Cause 4. +- `invalid reference format` with `docker://https://` in the destination name, or an X509 error on `docker login`: Cause 5. +- A raw chart push to `public chart` rejected as invalid credentials: Cause 6. diff --git a/docs/en/solutions/violet_push_goes_to_the_wrong_global_cluster_a_wrong_package_cannot_be_deleted_or_violet_or_kubectl_is_missing_on_ACP.md b/docs/en/solutions/violet_push_goes_to_the_wrong_global_cluster_a_wrong_package_cannot_be_deleted_or_violet_or_kubectl_is_missing_on_ACP.md new file mode 100644 index 000000000..64dd09d4c --- /dev/null +++ b/docs/en/solutions/violet_push_goes_to_the_wrong_global_cluster_a_wrong_package_cannot_be_deleted_or_violet_or_kubectl_is_missing_on_ACP.md @@ -0,0 +1,124 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# violet push goes to the wrong global cluster, a wrong package cannot be deleted, or violet or kubectl is missing on ACP + +## Issue + +- Cluster plugins pushed for the standby global cluster still show as not listed (`未上架`) there (for example Log Essentials and Log Storage for Elasticsearch), although `violet push` reported no error. +- A package was uploaded by mistake (wrong architecture or wrong build), and the console says it cannot be deleted. +- Deploying an immutable upload package fails because `packtool` cannot run `kubectl`. +- `violet` is not available on the operator workstation, and the question is whether a container of the platform Core already has it, or where a specific version can be downloaded. On ACP 4.2.1 the console has no upload (`上架`) button like the one in 3.18. + +Related articles: "violet push fails with no package found, code 000, HTTP 500 or 502, or invalid reference format on ACP" covers pushes that stop with an error; "Pushed package not listed where expected: global-only plugins, cluster affinity, catalog source, or plugin type on ACP" covers packages that were registered correctly but are not offered on a cluster. + +## Root Cause + +### Cause 1: the platform address given to violet resolves to the other global cluster + +In a primary/standby global setup both clusters share one platform domain, which normally resolves to the primary. When `violet push` is given that domain as `--platform-address` on a host where it resolves to the other global cluster, the packages are registered there, the push reports success, and the intended standby cluster shows nothing new. Running the push with `--platform-address` set to the standby cluster's VIP registers the packages on the standby. + +### Cause 2: a wrongly uploaded package cannot be deleted from the console + +The console refuses to delete the package. `violet push --force` overwrites and updates an existing package with the same image and architecture on the platform, so pushing the correct package for the same image and architecture (for example ARM) with `--force` replaces the wrong upload. + +### Cause 3: packtool fails because kubectl is not in PATH + +Deploying an immutable upload package runs `packtool`, which calls `kubectl` during execution. The failure is not a push to the image registry: `kubectl` is not available through `$PATH` on the node where `packtool` runs. After `kubectl` is installed on that node, the deployment completes. + +### Cause 4: violet is not part of the platform Core and is downloaded separately + +The `violet` tool is not built into the platform Core package, so it cannot be copied out of a running Core container. On ACP 4.2.1 the console has no 3.18-style upload button; packages are uploaded with `violet`. `violet` is downloaded from the Alauda customer portal, under the command-line tools of the matching ACP release. + +## Resolution + +### Cause 1: the platform address given to violet resolves to the other global cluster + +1. On the host that runs `violet`, check where the platform domain resolves, and compare it with the standby cluster's VIP: + + ```bash + getent hosts + ``` + +2. If the domain resolves to the other global cluster, run the push with the standby cluster's VIP as the platform address: + + ```bash + violet push .tgz --platform-address "https://" --platform-username --platform-password '' + ``` + +3. On the standby global cluster, check that the packages are registered: + + ```bash + kubectl get artifact,artifactversion -A | grep -i + kubectl get moduleplugins | grep -i + ``` + +If the push still does not register on the standby cluster, follow the Upload Packages procedure in the documentation for your platform version step by step for the standby cluster. + +### Cause 2: a wrongly uploaded package cannot be deleted from the console + +1. Get the correct package for the same image and architecture as the wrong upload. +2. Push it with `--force`. + + > **Warning:** `--force` overwrites the existing package with the same image and architecture on the platform. The previous upload is replaced, not kept. Precondition: only when the package already on the platform is the wrong one, the console refuses to delete it, and the file you push is the correct package for the same image and architecture. Rollback: none for the overwritten content; push the earlier file again with `--force` if you need it back. + + ```bash + violet push --platform-address= --platform-username= --platform-password='' --force .tgz + ``` + +3. Check that the console now shows the correct version and architecture for the package. + +### Cause 3: packtool fails because kubectl is not in PATH + +1. On the node where `packtool` runs, check for `kubectl`: + + ```bash + command -v kubectl || echo "kubectl not in PATH: $PATH" + ``` + +2. Install a `kubectl` that matches the cluster version (for example copied from a control-plane node) into a directory on `$PATH`, and check that it reaches the cluster: + + ```bash + install -m 0755 /usr/local/bin/kubectl + kubectl version + ``` + +3. Run the immutable upload package deployment again. + +### Cause 4: violet is not part of the platform Core and is downloaded separately + +1. Some platform builds serve `violet` themselves. Check whether your platform answers on this path before looking elsewhere: + + ```bash + curl -k -I https:///platform/tools/download/violet/violet_linux_amd64 + ``` + +2. Otherwise download it from the Alauda customer portal: Products (`产品`) → Product Download (`产品下载`) → the release line in the left sidebar (for example v4.0 → v4.0.9, or v4.2 → v4.2.1) → on the Alauda Container Platform page, under Type (`类型`), select Command-line tools (`命令行工具`) and choose the architecture: x86, ARM, or Hybrid (`异构`). The other types on that page are Core Package (`核心包`) and Extensions (`扩展`). +3. Use the `violet` of the same version as the platform, and check it: + + ```bash + violet version + ``` + +## Diagnostic Steps + +```bash +getent hosts +command -v kubectl +violet version +kubectl get artifact,artifactversion -A | grep -i +``` + +- The push reports success, nothing appears on the standby global cluster, and the platform domain resolves to the other global cluster from the push host: Cause 1. +- A wrong upload that the console will not delete: Cause 2. +- `packtool` stops on a missing `kubectl`: Cause 3. Expected: `command -v kubectl` prints nothing on that node. +- No `violet` binary on the workstation, or no upload button in the 4.2 console: Cause 4. From 88dbd1fb6e7b5e1750fb1b57acbc5666b8f89e3c Mon Sep 17 00:00:00 2001 From: Komh Date: Wed, 18 Nov 2026 17:40:56 +0800 Subject: [PATCH 15/25] docs(solutions): add 27 and update 0 troubleshooting articles --- ...0_access_to_the_platform_console_on_ACP.md | 131 +++++++++++ ...l_or_fetch_policy_fail_log_lines_on_ACP.md | 116 ++++++++++ ...r-rate_panel_exceeds_100_percent_on_ACP.md | 94 ++++++++ ...m_ALB_ports_and_access_addresses_on_ACP.md | 162 ++++++++++++++ ...od_ping_Redis_Sentinel_or_Harbor_on_ACP.md | 154 +++++++++++++ ...ch_reserved_IPs_or_deleted_raven_on_ACP.md | 106 +++++++++ ...ystemd-udevd_loops_on_calico_tmp_on_ACP.md | 159 +++++++++++++ ...ss_instead_of_the_real_client_IP_on_ACP.md | 136 +++++++++++ ...r_install_HMAC-SM3_and_migration_on_ACP.md | 133 +++++++++++ ...ble_TLS-RSA_ciphers_or_Wasm_x509_on_ACP.md | 134 +++++++++++ ..._client_IP_health_checks_and_503_on_ACP.md | 181 +++++++++++++++ ...ing_nameserver_or_a_new_upstream_on_ACP.md | 182 +++++++++++++++ ...Conflict_or_no_address_allocated_on_ACP.md | 171 ++++++++++++++ ...several_subnets_or_leftover_port_on_ACP.md | 127 +++++++++++ ...duplicate_entry_or_RollingUpdate_on_ACP.md | 150 +++++++++++++ ...l_and_an_underlay_global_cluster_on_ACP.md | 169 ++++++++++++++ ...espace_binding_or_uppercase_IPv6_on_ACP.md | 124 ++++++++++ ...ender_that_never_joins_the_group_on_ACP.md | 113 ++++++++++ ...address_outside_the_node_segment_on_ACP.md | 127 +++++++++++ ..._or_several_addresses_on_one_NIC_on_ACP.md | 84 +++++++ ...ty_podCIDR_on_a_Kube-OVN_cluster_on_ACP.md | 79 +++++++ ...el_OS_hypervisor_or_network_path_on_ACP.md | 145 ++++++++++++ ...wall_BGP_sessions_or_IPv6_routes_on_ACP.md | 211 ++++++++++++++++++ ...S_server_or_debug_container_ping_on_ACP.md | 135 +++++++++++ ...dresses_search_domains_and_ndots_on_ACP.md | 131 +++++++++++ ...weighted_routes_and_route_export_on_ACP.md | 196 ++++++++++++++++ ...rrupted_image_clock_skew_or_x509_on_ACP.md | 144 ++++++++++++ 27 files changed, 3794 insertions(+) create mode 100644 docs/en/solutions/ALB_TLS_1.0_and_1.1_and_HTTP_port_80_access_to_the_platform_console_on_ACP.md create mode 100644 docs/en/solutions/ALB_alb_lua_shared_dict_free_or_alb_loop_time_alerts_and_internal_dsl_is_null_or_fetch_policy_fail_log_lines_on_ACP.md create mode 100644 docs/en/solutions/ALB_error-rate_alert_fires_or_the_ALB_error-rate_panel_exceeds_100_percent_on_ACP.md create mode 100644 docs/en/solutions/ALB_placement_cordoned_nodes_tolerate-all_node_selectors_cpaas-system_ALB_ports_and_access_addresses_on_ACP.md create mode 100644 docs/en/solutions/Application_breaks_after_a_NetworkPolicy_change_Ingress_Pod-to-Pod_ping_Redis_Sentinel_or_Harbor_on_ACP.md create mode 100644 docs/en/solutions/Calico_Pods_get_no_address_disabled_default_IPPool_Raven_fixed_IP_mismatch_reserved_IPs_or_deleted_raven_on_ACP.md create mode 100644 docs/en/solutions/Calico_ipipMode_reverts_Felix_metrics_off_bandwidth_annotation_breaks_Pods_systemd-udevd_loops_on_calico_tmp_on_ACP.md create mode 100644 docs/en/solutions/Envoy_Gateway_access_log_shows_a_Kube-OVN_join_address_instead_of_the_real_client_IP_on_ACP.md create mode 100644 docs/en/solutions/Envoy_Gateway_limits_and_questions_64_listeners_access_logs_operator_install_HMAC-SM3_and_migration_on_ACP.md create mode 100644 docs/en/solutions/Envoy_Gateway_setup_fails_Backend_is_disabled_platform_certificate_not_selectable_TLS-RSA_ciphers_or_Wasm_x509_on_ACP.md create mode 100644 docs/en/solutions/External_load_balancer_in_front_of_ALB_stalls_pool_members_client_IP_health_checks_and_503_on_ACP.md create mode 100644 docs/en/solutions/Host_DNS_changes_do_not_reach_CoreDNS_systemd-resolved_kubelet_resolvConf_missing_nameserver_or_a_new_upstream_on_ACP.md create mode 100644 docs/en/solutions/Kube-OVN_Pods_fail_with_AddressOutOfRange_AddressConflict_or_no_address_allocated_on_ACP.md create mode 100644 docs/en/solutions/Kube-OVN_fixed_IP_NoAvailableAddress_spaces_in_the_list_ip_pool_not_reserved_several_subnets_or_leftover_port_on_ACP.md create mode 100644 docs/en/solutions/Kube-OVN_fixed_IP_occupied_or_not_released_held_by_another_Pod_node_down_duplicate_entry_or_RollingUpdate_on_ACP.md create mode 100644 docs/en/solutions/Kube-OVN_subnet_planning_CIDR_changes_overlaps_subnet_type_per-namespace_IPPool_and_an_underlay_global_cluster_on_ACP.md create mode 100644 docs/en/solutions/Kube-OVN_subnet_stuck_Terminating_or_Pods_use_the_wrong_subnet_finalizers_namespace_binding_or_uppercase_IPv6_on_ACP.md create mode 100644 docs/en/solutions/Kube-OVN_underlay_multicast_not_delivered_DDS_IGMP_membership_limit_or_a_sender_that_never_joins_the_group_on_ACP.md create mode 100644 docs/en/solutions/LoadBalancer_VIP_unreachable_after_project_isolation_or_with_a_MetalLB_address_outside_the_node_segment_on_ACP.md create mode 100644 docs/en/solutions/Multus_secondary_networks_unexpected_end_of_JSON_input_duplicated_addresses_or_several_addresses_on_one_NIC_on_ACP.md create mode 100644 docs/en/solutions/Node_events_show_CIDRNotAvailable_and_some_nodes_have_an_empty_podCIDR_on_a_Kube-OVN_cluster_on_ACP.md create mode 100644 docs/en/solutions/Pod_or_Service_traffic_fails_on_some_nodes_because_of_the_node_kernel_OS_hypervisor_or_network_path_on_ACP.md create mode 100644 docs/en/solutions/Pod_traffic_fails_on_Calico_because_of_an_IPPool_destination_MTU_firewall_BGP_sessions_or_IPv6_routes_on_ACP.md create mode 100644 docs/en/solutions/Pods_cannot_resolve_intranet_or_external_names_nameserver_order_internal_DNS_server_or_debug_container_ping_on_ACP.md create mode 100644 docs/en/solutions/Pods_fail_to_resolve_Service_names_or_resolve_them_to_external_addresses_search_domains_and_ndots_on_ACP.md create mode 100644 docs/en/solutions/Traefik_gateway_X-Forwarded-For_missing_ports_80_and_443_underscore_headers_weighted_routes_and_route_export_on_ACP.md create mode 100644 docs/en/solutions/calico-node_fails_to_start_or_Pods_hang_in_ContainerCreating_dummy_module_corrupted_image_clock_skew_or_x509_on_ACP.md diff --git a/docs/en/solutions/ALB_TLS_1.0_and_1.1_and_HTTP_port_80_access_to_the_platform_console_on_ACP.md b/docs/en/solutions/ALB_TLS_1.0_and_1.1_and_HTTP_port_80_access_to_the_platform_console_on_ACP.md new file mode 100644 index 000000000..d7e133cf0 --- /dev/null +++ b/docs/en/solutions/ALB_TLS_1.0_and_1.1_and_HTTP_port_80_access_to_the_platform_console_on_ACP.md @@ -0,0 +1,131 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "4.1" + - "4.2" + - "4.3" +--- + +# ALB TLS 1.0 and 1.1, and HTTP port 80 access to the platform console on ACP + +## Issue + +- A compliance scan requires TLS 1.0 and TLS 1.1 to be disabled, and the question is where the ALB's nginx configuration can be checked. +- The platform console can only be opened over HTTPS on 443; `http://` on port 80 does not work, and the administrator wants HTTP access back, or asks whether port 80 can still be opened after the platform is deployed. +- A script calls a platform API such as `/apis/platform.tkestack.io/v1/clusters` over HTTP and gets `404` on one platform while the same call works on another. + +These are settings of the ALB data plane and of the platform configuration, not of a single rule. + +## Root Cause + +### Cause 1: ssl_protocols in the ALB ConfigMap enables TLSv1 and TLSv1.1 + +The protocols the ALB accepts are set by `ssl_protocols` in the ConfigMap that has the same name as the ALB (namespace `cpaas-system` of the cluster where the ALB runs), for example: + +```nginx +ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; +``` + +`TLSv1` and `TLSv1.1` in that line mean TLS 1.0 and TLS 1.1 are enabled. + +### Cause 2: forceRedirectHttps: true and httpPort: 0 in the prdb base + +Whether the console is served on port 80 is decided by the `http` section of the `base` ProductBase (`kubectl get prdb base -o yaml`). With `forceRedirectHttps: true`, `httpPort: 0` and `httpsPort: 443`, only HTTPS is served; the platform ALB `global-alb2` follows it (its Deployment carries the environment variable `DEFAULT-SSL-STRATEGY` with value `Always`), and there is no `global-alb2-00080` frontend. Changing these two fields after deployment opens port 80. + +This concerns the platform console only. Do not set `forceRedirectHttps` to `false` to change redirects of business listeners. The platform console protocol is also unrelated to the access protocol of a business Harbor instance. + +### Cause 3: 404 with X-ALB-ERR-REASON: InvalidUpstream : no rule match on port 80 of global-alb2 + +`curl -v http:///apis/platform.tkestack.io/v1/clusters` returns `HTTP/1.1 404 Not Found` with `X-ALB-ERR-REASON: InvalidUpstream : no rule match` and `Via: alb`, and the `global-alb2` log shows the request with `404` and `no rule match` for the port 80 policy. On that platform the `erebus` rules for `/api`, `/apis`, `/kubernetes/` and `/tunnel` exist only on port 443 (`global-alb2-00443-*`) because port 80 is not enabled, so HTTP requests match no rule. A platform where the call works has the same rules on both `global-alb2-00080-*` and `global-alb2-00443-*`. + +## Resolution + +### Cause 1: ssl_protocols in the ALB ConfigMap enables TLSv1 and TLSv1.1 + +Remove `TLSv1` and `TLSv1.1` from `ssl_protocols` and restart the ALB Pods. Make the edit in the overwrite ConfigMap so the next reconcile does not restore the old line (see "ALB nginx settings are reverted after a restart or upgrade; change them through the overwrite ConfigMap on ACP"). Keep the newer protocols the line already lists: + +```nginx +ssl_protocols TLSv1.2 TLSv1.3; +``` + +> **Warning:** Clients that only speak TLS 1.0 or 1.1 can no longer connect to any HTTPS port of this ALB, and each restart drops connections on that replica. Check client compatibility and back up the overwrite ConfigMap first. Restoring the previous line and restarting again is the rollback. + +```bash +kubectl -n cpaas-system get pods -o wide | grep +kubectl -n cpaas-system delete pod +``` + +Delete one ALB Pod at a time and wait for its replacement to be Ready before the next. + +### Cause 2: forceRedirectHttps: true and httpPort: 0 in the prdb base + +On the global cluster: + +1. Choose the HTTP port (for example `80`) and add a forwarding rule on the platform address (VIP or external load balancer) that sends that port to the three global masters. +2. Back up the ProductBase and the platform ALB: + + ```bash + kubectl get prdb base -o yaml > /tmp/prdb-base.backup.yaml + kubectl -n cpaas-system get deploy global-alb2 -o yaml > /tmp/global-alb2-deploy.backup.yaml + ``` + +3. Edit the ProductBase and set `spec.http.httpPort: 80` and `spec.http.forceRedirectHttps: false`, then wait until every `ars` in `cpaas-system` is ready. + + > **Warning:** The platform releases reconcile, the console then also answers in clear text on port 80, and the console can be briefly unreachable while `global-alb2` restarts. Do it in a window and only where HTTP access to the console is acceptable on that network. Roll back with `kubectl apply -f /tmp/prdb-base.backup.yaml` and wait for the `ars` again. + + ```bash + kubectl edit prdb base + kubectl get ars -n cpaas-system + ``` + +4. Check that the port 80 frontend exists and that `DEFAULT-SSL-STRATEGY` changed. If not, restart the `global-alb2` Pods one at a time. + + ```bash + kubectl get ft -n cpaas-system | grep global-alb2 + kubectl -n cpaas-system get deploy global-alb2 -o yaml | grep -n -A1 DEFAULT-SSL-STRATEGY + kubectl -n cpaas-system delete pod + ``` + +5. Log in over both `http://` and `https://`. + +### Cause 3: 404 with X-ALB-ERR-REASON: InvalidUpstream : no rule match on port 80 of global-alb2 + +Send the request over HTTPS on port 443, where the `erebus` rules exist. If HTTP on port 80 is required for the platform, enable it with Cause 2, which creates the `global-alb2-00080` frontend and its rules. + +```bash +curl -k -sS -o /dev/null -w '%{http_code}\n' -H 'Authorization: Bearer ' https:///apis/platform.tkestack.io/v1/clusters +``` + +## Diagnostic Steps + +1. Read the enabled protocols and test the handshake: + + ```bash + kubectl -n cpaas-system get configmap -o yaml | grep -C 10 ssl_protocols + openssl s_client -connect 192.0.2.10:443 -servername app.example.com -tls1_1 /apis/platform.tkestack.io/v1/clusters 2>&1 | grep -iE '^< HTTP|x-alb-err-reason|^< via' + kubectl get rule -A | grep erebus + ``` + + `no rule match` and only `global-alb2-00443-*` rules for `erebus`: Cause 3. diff --git a/docs/en/solutions/ALB_alb_lua_shared_dict_free_or_alb_loop_time_alerts_and_internal_dsl_is_null_or_fetch_policy_fail_log_lines_on_ACP.md b/docs/en/solutions/ALB_alb_lua_shared_dict_free_or_alb_loop_time_alerts_and_internal_dsl_is_null_or_fetch_policy_fail_log_lines_on_ACP.md new file mode 100644 index 000000000..3074b7791 --- /dev/null +++ b/docs/en/solutions/ALB_alb_lua_shared_dict_free_or_alb_loop_time_alerts_and_internal_dsl_is_null_or_fetch_policy_fail_log_lines_on_ACP.md @@ -0,0 +1,116 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# ALB alb_lua_shared_dict_free or alb_loop_time alerts and internal_dsl is null or fetch policy fail log lines on ACP + +## Issue + +An ALB (`global-alb2` or a business ALB) raises a resource or loop-time alert, or its log repeats a line that looks like an error, while traffic is served: + +- The alert `alb_lua_shared_dict_free < 10%` for 30 seconds fires, and the ALB log repeats `prometheus.lua:982: log_error(): Error getting 'nginx_http_upstream_request_duration_seconds_bucket...`. +- The alert `alb_loop_time > 1000ms` for 30 seconds fires from the policy `cpaas-alb-rules`. +- The ALB log repeats `worker.lua:57: init_http_rule_dsl(): internal_dsl is null`. +- The ALB log repeats `fetch policy fail`. + +Related articles: "ALB error-rate alert fires or the ALB error-rate panel exceeds 100 percent on ACP"; "ALB nginx settings are reverted after a restart or upgrade; change them through the overwrite ConfigMap on ACP". + +## Root Cause + +### Cause 1: alb_lua_shared_dict_free alert + +The alert reports that free space in an ALB Lua shared dictionary is low. The dictionaries are declared in the ALB ConfigMap, for example `lua_shared_dict http_raw` and `lua_shared_dict prometheus_metrics`. A free-space curve that stays flat for days points at a dictionary sized close to its steady-state use rather than at a leak. + +On ALB2 v4.1.2 the alert on `global-alb2` is a known defect. Its signature: in the 7-day Lua shared dictionary panel, `balancer_ewma`, `http_certs_cache`, `http_alb_cache`, `http_backend_cache`, `http_raw` and `http_ipc_shared_dict` stay at about 96 to 99 percent free while `prometheus_metrics` is near or below 10 percent, and the log repeats: + +```text +[error] ... [lua] prometheus.lua:982: log_error(): Error getting 'nginx_http_upstream_request_duration_seconds_bucket{port="443",rule="global-alb2-00443-",upstream_ip="... +``` + +with an `upstream_ip` label that holds a very long, repeated list of the same healthy backend addresses. Abnormal monitoring metrics keep occupying the dictionary, some ALB metrics go missing or become discontinuous, and errors appear in the log. Business forwarding is not affected. The defect is fixed in the latest 4.2 release. + +### Cause 2: alb_loop_time > 1000ms for 30 seconds + +`alb_loop_time` does not measure business request latency. It is the total time of one ALB processing loop: fetching and processing Ingress, Service, Endpoint and related resources, generating the nginx configuration, reloading nginx, and updating rule status. The alert fires when that loop takes longer than the threshold for 30 seconds; slow API server responses or slow writes of the configuration file can stretch it. The alert does not affect business traffic by itself. + +When the alert fires intermittently and the ALB logs for the window were not retained, the cause cannot be determined afterwards: low node CPU, memory and load in that window do not prove a disk or API server cause. + +### Cause 3: worker.lua:57: init_http_rule_dsl(): internal_dsl is null + +A new or idle replica logs this line next to certificate-cache and policy-cache lines when the frontend has an empty policy or no backend Service. It is noise and does not indicate a failure. + +### Cause 4: fetch policy fail in the ALB log + +Inside the ALB, a Go process periodically writes the `policy` file and Lua periodically reads it; Lua can start reading while the file is still being written, and logs `fetch policy fail`. Traffic is served normally. This was seen with ALB image `alb2:v3.16.6` and is optimized in later versions. + +## Resolution + +### Cause 1: alb_lua_shared_dict_free alert + +On ALB2 v4.1.2, upgrade to a release with the fix. Until then, a rolling restart of the ALB releases the metrics memory. + +> **Warning:** A rolling restart drops in-flight connections on each replica in turn. Only when the image is v4.1.2 and the signature above matches, with more than one replica Ready. Do not use it to clear an `error-rate` or `alb_loop_time` alert. There is nothing to roll back. + +```bash +kubectl -n cpaas-system rollout restart deploy/ +``` + +On other versions, check the `alb2` and `nginx` container logs for errors first. If they are clean and the metric has been flat for days, raise both dictionaries to `50m` in the overwrite ConfigMap (`data.http`) and watch whether the alert clears. Edit the existing `lua_shared_dict` lines in place; nginx refuses a configuration that declares the same dictionary twice. + +```nginx +lua_shared_dict http_raw 50m; +lua_shared_dict prometheus_metrics 50m; +``` + +> **Warning:** Applying the overwrite ConfigMap reloads nginx on every replica of the ALB, and a syntax error stops the reload for all its listeners. Each dictionary is allocated per Pod, so memory use grows. Back up the overwrite ConfigMap and check headroom under the `nginx` container limit; restoring the previous values is the rollback. The procedure is in the related overwrite ConfigMap article. + +### Cause 2: alb_loop_time > 1000ms for 30 seconds + +Do not delete or restart the ALB Pod to clear the alert. If it fired once, recovered within about 30 seconds, users were not blocked, and the Pods did not restart, record it and leave the policy. For intermittent firing, make sure the next occurrence leaves evidence: save the `alb2` log of every replica for the window before Pods roll, keep the platform log collection for ALB components long enough, and compare slow loops with what changed at that moment (bulk Ingress or rule edits, certificate updates, API server latency, disk latency on the node). + +```bash +for p in $(kubectl -n cpaas-system get pods -l service_name=alb2- -o name); do + kubectl -n cpaas-system logs "$p" -c alb2 --since=1h > "/tmp/$(basename "$p")-alb2.log" +done +kubectl -n cpaas-system logs --all-containers --since=3h | grep -E 'reload:|reload timeout|generate conf|reload load balancer' +``` + +### Cause 3: worker.lua:57: init_http_rule_dsl(): internal_dsl is null + +No change. Attach a Service to the frontend when it should carry traffic. If requests are failing on a replica that serves traffic, look elsewhere; this line is not the failure. + +### Cause 4: fetch policy fail in the ALB log + +If the services behind the ALB answer normally, ignore the line. Upgrading to a release with the optimized policy publishing removes it. Do not edit the policy file inside the container. + +## Diagnostic Steps + +1. Read the image tag, the declared dictionaries and the metrics error count: + + ```bash + kubectl -n cpaas-system get pods -o jsonpath='{.spec.containers[*].image}{"\n"}' + kubectl -n cpaas-system get configmap -o yaml | grep -n lua_shared_dict + kubectl -n cpaas-system logs -c nginx --since=1h | grep -c 'prometheus.lua:982' + ``` + + A v4.1.2 image, `prometheus_metrics` as the only low dictionary, and `prometheus.lua:982` errors: Cause 1 defect. To identify the repeated `upstream_ip` addresses: `kubectl get po -A -o wide | grep -E "|"`. + +2. For `alb_loop_time`, open the policy's monitoring graph for the window and save the `alb2` logs as in Cause 2 before any restart. + +3. Check the noise lines: + + ```bash + kubectl -n cpaas-system logs -c nginx --since=10m | grep -F internal_dsl + kubectl -n cpaas-system logs -c nginx --since=1h | grep -c 'fetch policy fail' + ``` + + With traffic served normally: Cause 3 or Cause 4. diff --git a/docs/en/solutions/ALB_error-rate_alert_fires_or_the_ALB_error-rate_panel_exceeds_100_percent_on_ACP.md b/docs/en/solutions/ALB_error-rate_alert_fires_or_the_ALB_error-rate_panel_exceeds_100_percent_on_ACP.md new file mode 100644 index 000000000..50ae10db8 --- /dev/null +++ b/docs/en/solutions/ALB_error-rate_alert_fires_or_the_ALB_error-rate_panel_exceeds_100_percent_on_ACP.md @@ -0,0 +1,94 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# ALB error-rate alert fires or the ALB error-rate panel exceeds 100 percent on ACP + +## Issue + +`global-alb2` or a business ALB raises `error-rate` alerts while the platform or the service works, or the ALB monitoring dashboard shows an error rate above 100 percent: + +- The alert `error-rate > 3%` or `error-rate > 5%` for 30 seconds (warning level) fires from the load balancer routing rule alert policy `cpaas-alb-router-rules` for routes such as `ingress/cpaas-system/dex/...` and `ingress/cpaas-system/courier-api/...`. +- Many platform Ingress objects in `cpaas-system` (`dex`, `frontend`, `apollo`, `courier-api` and others) cross the threshold at the same time. +- The ALB monitoring panel shows an error rate above 100 percent, and the access log is dominated by status `499` and `404`. + +`global-alb2` in `cpaas-system` is the platform ingress data plane. Its Pods run the containers `nginx` (access log) and `alb2` (configuration loop log). + +Related articles: "ALB alb_lua_shared_dict_free or alb_loop_time alerts and internal_dsl is null or fetch policy fail log lines on ACP". + +## Root Cause + +### Cause 1: error-rate crosses the threshold on many platform routes with no matching errors in the access log + +`cpaas-alb-router-rules` evaluates the error ratio per route over a sliding window (5 minutes) and fires when it stays above the threshold for 30 seconds; the 30 seconds is the hold time, not the window. Depending on the version, the expression counts 5xx only or 4xx and 5xx, so read the expression before interpreting a value. + +When many `cpaas-system` Ingresses cross the line together (for example `dex` and `courier-api` at `11.11 %` while `archon`, `base-api`, `erebus`, `frontend` and `razor` are at `0.00 %`), the `global-alb2` Pods are `2/2 Running` with no restarts, and the `nginx` access log for those rules shows `"status": "200"` and `"upstream_status": "200"` with no error wave in the window, there is no ALB defect behind the alert. On ACP 4.1.5 this signature was closed as needing no action, with the metric to be improved in a later release. + +A route with very little traffic reaches a high ratio from a single failed request: one request in the window that returns HTTP 500 is 100 percent. When one route dominates, the errors come from that route's backend; an access-log line whose `upstream_status` equals `status` means the backend produced the code and the ALB passed it through. + +### Cause 2: error-rate panel above 100 percent with many 499 and 404 responses + +`499` is the nginx code for a client that closed the connection before the response was sent. When the access log for the window is dominated by `499` (with `404`), clients disconnected before the server answered; no ALB or backend fault was found. A panel value above 100 percent from that mix is not the alert expression computing a real backend failure rate. + +## Resolution + +### Cause 1: error-rate crosses the threshold on many platform routes with no matching errors in the access log + +1. Read the alert expression to see which status codes it counts: + + ```bash + kubectl get prometheusrules.monitoring.coreos.com -A -o yaml | grep -n 'nginx_http_status' + ``` + +2. Check the ALB Pods and the access log for the alert window: + + ```bash + kubectl -n cpaas-system get pods -l service_name=alb2-global-alb2 -o wide + kubectl -n cpaas-system logs -c nginx --since=30m | grep dex | tail + kubectl -n cpaas-system logs -c nginx --since=30m | grep -E ' (401|403|404|499|500|502|503|504) ' | tail + ``` + +3. If many platform routes fired together, the Pods are healthy, and the log has no matching error wave, record the alert as not caused by a route or ALB fault. Do not raise the threshold and do not restart `global-alb2` for this signature. + +4. If one route dominates, find its backend and read the backend log for the same minute. Fix sustained backend errors on the backend; raising the threshold does not fix them. + + ```bash + kubectl get rule -A | grep + kubectl get rule -n -o yaml | grep -A10 service + kubectl logs -n deploy/ --tail=200 + ``` + +### Cause 2: error-rate panel above 100 percent with many 499 and 404 responses + +Count the status codes in the window. When `499` dominates, look at the clients: timeouts shorter than the backend response time, load-test tools that abort requests, or callers that cancel on navigation. Fix the client timeout or the slow backend; the ALB needs no change, and neither a threshold change nor an ALB restart helps. Treat a `404` share separately: it means requests matched no rule or the backend returned `404`. + +```bash +kubectl -n cpaas-system logs -c nginx --since=30m | grep -oE '" [1-5][0-9]{2} ' | sort | uniq -c | sort -rn | head +``` + +## Diagnostic Steps + +1. Identify the policy and the routes. `cpaas-alb-router-rules` is the per-route `error-rate` policy; an `alb_loop_time` alert from `cpaas-alb-rules` is covered in the related article. + +2. Compare the alert with the `nginx` access log of every ALB replica for the same window: + + ```bash + kubectl -n cpaas-system logs deploy/global-alb2 -c nginx --since=30m | grep -E ' (401|403|404|499|500|502|503|504) ' | tail + ``` + + The `-c nginx` flag is needed because the Pod has two containers. + +3. Map the result: + + - Many `cpaas-system` Ingresses fire together, Pods healthy, no matching error codes in the access log: Cause 1, no action. + - One route dominates: Cause 1, follow that backend. + - A status count led by `499` (with `404`) and a panel above 100 percent: Cause 2. diff --git a/docs/en/solutions/ALB_placement_cordoned_nodes_tolerate-all_node_selectors_cpaas-system_ALB_ports_and_access_addresses_on_ACP.md b/docs/en/solutions/ALB_placement_cordoned_nodes_tolerate-all_node_selectors_cpaas-system_ALB_ports_and_access_addresses_on_ACP.md new file mode 100644 index 000000000..6d593906b --- /dev/null +++ b/docs/en/solutions/ALB_placement_cordoned_nodes_tolerate-all_node_selectors_cpaas-system_ALB_ports_and_access_addresses_on_ACP.md @@ -0,0 +1,162 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.1" + - "4.2" +--- + +# ALB placement: cordoned nodes, tolerate-all, node selectors, cpaas-system ALB ports and access addresses on ACP + +## Issue + +Questions and failures about where an Alauda Load Balancer (ALB) runs and how it is reached: + +- The masters were cordoned with `kubectl cordon`, and the ALB Pods (or other `cpaas-system` Pods) can no longer be scheduled there, so they fail to restart during an upgrade or after a VM restart. Tolerations added to the ALB Deployment disappear right after saving. +- How to move an ALB to other nodes, and how to find the port mappings of an ALB before its VIP is replaced. +- Business ports are wanted on the `cpaas-system` ALB on the masters, or a new ALB should take port 443 on the global masters. +- The access IP of a container-network ALB was changed, or a second one added, and the address does not answer. +- Whether several business ALBs in one cluster conflict when they all use 80, 443 or the same ports, and whether more ALBs can be added. +- Edits to the `alb2` resource of the `cpaas-system` ALB (for example `networkMode`) are reverted, and the monitoring dashboard of the business cluster does not display correctly. + +Every ALB Deployment is generated from its `alb2` resource (`crd.alauda.io/v2beta1`), so changes go on the `alb2` object. On a business cluster, the ALB named `cpaas-system` in namespace `cpaas-system` is the platform's own ALB. + +## Root Cause + +### Cause 1: ALB Pods cannot be scheduled onto cordoned nodes, and Deployment edits are reverted + +`kubectl cordon` marks a node `unschedulable: true`, represented as the taint `node.kubernetes.io/unschedulable:NoSchedule`. The ALB Pods do not tolerate it, and tolerations typed into the generated Deployment are reverted by the controller that owns it. The annotation `tolerate-all: "true"` on the `alb2` resource makes the ALB Pods tolerate all taints, including the cordon taint; the change shows on the Pods, not in the Deployment YAML. An annotation named `tolerate` has no effect. + +### Cause 2: the ALB has to run on different nodes + +The ALB node selector decides which nodes run ALB Pods. Moving the ALB also moves the address that the VIP or external load balancer must forward to; for the `cpaas-system` ALB, the platform VIP forwarding must change too. A single-instance ALB reached through a node's own IP needs its **Access Address** changed; a high-availability ALB behind a VIP keeps the VIP and changes the VIP's backend members. Updating the load balancer does not affect short connections but interrupts long connections in progress. + +### Cause 3: the cpaas-system ALB is reserved for platform traffic + +The `cpaas-system` ALB is a platform-exclusive resource that forwards only platform traffic and cannot be delegated to business users. A platform upgrade syncs its configuration and overwrites custom changes, so ports added to it are lost and can conflict with platform traffic. On the global cluster, port 443 of the three masters is used for platform access and must not be changed, so a new ALB must not be scheduled onto those nodes. + +### Cause 4: a container-network ALB does not take a changed or added access IP + +For a container-network ALB, the access address comes from its `LoadBalancer` Service (`-lb-` in `cpaas-system`). The existing address cannot be edited, and added addresses do not take effect: `telnet ` connects while `telnet ` hangs at `Trying ...`, and the Service `EXTERNAL-IP` lists only the original address. Changing or adding access addresses is supported only for host-network ALBs, which accept several access addresses (domain names or IPs), each a separate entry; a domain entered there must resolve to the load balancer's VIP. + +### Cause 5: several business ALBs using the same ports do not conflict + +Host-network ALB instances of one cluster cannot be deployed on the same node (the console does not allow it), so each instance serves its ports on different nodes. Container-network ALBs do not occupy node ports. More ALB instances can be added on demand as long as the cluster has the resources. + +### Cause 6: the cpaas-system alb2 is rendered by a chart and edits are reconciled back + +The `alb2` of the `cpaas-system` ALB is rendered by the chart release `alauda-alb2` (chart `acp/chart-alauda-alb2`), so a direct change such as `networkMode` is reconciled back after saving. The values must be set through the chart's `valuesOverride`. A business cluster whose `cpaas-system` ALB was not in host network mode also had a monitoring dashboard that did not display correctly; it displayed normally after the same `valuesOverride` was applied. The link between the two was not confirmed. + +## Resolution + +### Cause 1: ALB Pods cannot be scheduled onto cordoned nodes, and Deployment edits are reverted + +Add the annotation to the `alb2` resource, then check the Pods. + +> **Warning:** With `tolerate-all`, taints and the unschedulable mark no longer keep ALB Pods off any node; only the node selector limits placement. The change rolls the ALB Pods. Only when the node selector already restricts the ALB to the intended nodes and more than one replica is Ready. Roll back with `kubectl -n cpaas-system annotate alb2 tolerate-all-`. + +```bash +kubectl -n cpaas-system get alb2 +kubectl -n cpaas-system annotate alb2 tolerate-all="true" --overwrite +kubectl -n cpaas-system get pods -l service_name=alb2- -o jsonpath='{range .items[*]}{.metadata.name} {.spec.tolerations}{"\n"}{end}' +``` + +A toleration with `operator: Exists` and no key confirms it. The same annotation lets the `cpaas-system` ALB return to cordoned masters by itself after a VM restart or an upgrade. Where the annotation is not wanted, `uncordon` the masters during the upgrade, wait for the Pods to start, and `cordon` them again. + +### Cause 2: the ALB has to run on different nodes + +Change the node selector under `spec.config` of the `alb2` (console: **Platform Management > Network Management > Load Balancers**, open the ALB, **Update**), and update the VIP or external load balancer pool in the same change: + +```yaml +spec: + config: + nodeSelector: + ingress: "true" +``` + +> **Warning:** When the Pods move, the old nodes stop answering on the ALB ports and long connections are interrupted. The new nodes must carry the label and already be members of the VIP or load balancer pool (or clients must be ready for the new **Access Address**). Do it in a maintenance window. Restoring the previous selector is the rollback. + +```bash +kubectl label node = +kubectl -n cpaas-system get alb2 -o yaml | grep -n -A3 nodeSelector +kubectl get ft -A +``` + +The ports configured on an ALB are Frontend (`ft`) resources; `kubectl get ft -A` lists the port mappings before a move or a VIP replacement. + +### Cause 3: the cpaas-system ALB is reserved for platform traffic + +Create a separate ALB for the business component, on the same master nodes and behind the same VIP if needed, with high-numbered ports the platform ALB does not use. Leave the `cpaas-system` ALB unchanged. On the global cluster, do not schedule new ALBs onto the three masters. + +### Cause 4: a container-network ALB does not take a changed or added access IP + +To keep a given VIP, use an ALB in host network mode: create it with host networking and enter the address (and any additional ones, each as a separate entry) in the access address field (**Network Configuration**, network mode host network, **Add** under the access address list). A container-network ALB cannot be switched in place; plan a new ALB with the same listeners and rules. + +### Cause 5: several business ALBs using the same ports do not conflict + +No change is needed. Give each host-network ALB its own nodes through its node selector, and point each VIP or DNS name at the nodes of its own ALB. Check capacity before adding another ALB: + +```bash +kubectl get alb2 -A +kubectl describe node | grep -A8 'Allocated resources' +``` + +### Cause 6: the cpaas-system alb2 is rendered by a chart and edits are reconciled back + +On the global cluster, edit the ALB `minfo` of the target business cluster and set the values under `spec.valuesOverride`, keeping the `version` the `minfo` already shows. The same `valuesOverride` can be set on the ALB's `ars` (AppRelease `alauda-alb2`). + +```yaml +spec: + valuesOverride: + acp/chart-alauda-alb2: + networkMode: host + nodeSelector: + cpaas-system-alb: "" + version: +``` + +> **Warning:** Changing `networkMode` or `nodeSelector` redeploys the `cpaas-system` ALB, interrupting platform traffic through it while its Pods move; host network mode needs its ports free on the selected nodes. The nodes must carry the `cpaas-system-alb` label. Save the `minfo` YAML; restoring the saved `valuesOverride` is the rollback. + +```bash +kubectl get minfo | grep -i alb +kubectl get minfo -o yaml > /tmp/.yaml +kubectl edit minfo +kubectl get node -l cpaas-system-alb +``` + +## Diagnostic Steps + +1. Why ALB Pods are Pending: + + ```bash + kubectl -n cpaas-system describe pod | grep -A5 Events + kubectl get nodes -o custom-columns=NAME:.metadata.name,UNSCHEDULABLE:.spec.unschedulable,TAINTS:.spec.taints + ``` + + `untolerated taint {node.kubernetes.io/unschedulable: }`: Cause 1. + +2. Network mode and access address of an ALB; for a container-network ALB, the `EXTERNAL-IP` of its Service is the address that works (Cause 4): + + ```bash + kubectl -n cpaas-system get pods -l service_name=alb2- -o jsonpath='{range .items[*]}{.metadata.name} hostNetwork={.spec.hostNetwork}{"\n"}{end}' + kubectl -n cpaas-system get svc | grep -lb + ``` + +3. Ports and placement of every ALB (Causes 2 and 5): + + ```bash + kubectl get ft -A + kubectl get pods -A -o wide | grep -E 'alb' + ``` + +4. A reverted change on the `cpaas-system` ALB (Cause 6): + + ```bash + kubectl get minfo -o jsonpath='{.spec.valuesOverride}{"\n"}' + kubectl -n cpaas-system get alb2 cpaas-system -o yaml | grep -n -E 'networkMode|nodeSelector' + ``` + +For external load balancer pool members that must match ALB placement, see "External load balancer in front of ALB: stalls, pool members, client IP, health checks and 503 on ACP". diff --git a/docs/en/solutions/Application_breaks_after_a_NetworkPolicy_change_Ingress_Pod-to-Pod_ping_Redis_Sentinel_or_Harbor_on_ACP.md b/docs/en/solutions/Application_breaks_after_a_NetworkPolicy_change_Ingress_Pod-to-Pod_ping_Redis_Sentinel_or_Harbor_on_ACP.md new file mode 100644 index 000000000..089f5de73 --- /dev/null +++ b/docs/en/solutions/Application_breaks_after_a_NetworkPolicy_change_Ingress_Pod-to-Pod_ping_Redis_Sentinel_or_Harbor_on_ACP.md @@ -0,0 +1,154 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Application breaks after a NetworkPolicy change: Ingress, Pod-to-Pod ping, Redis Sentinel or Harbor on ACP + +## Issue + +An application that worked, or a new one, fails, and a network policy is involved: + +- An Ingress created on the platform for a Service port (for example `8080`) cannot be reached. +- Two Pods in the same namespace cannot ping each other in either direction: `2 packets transmitted, 0 received, 100% packet loss`. +- A Redis instance in Sentinel mode does not start after it is created. The `rfr-` StatefulSet Pods are in `CrashLoopBackOff`, and their logs show every call to the Kubernetes API server timing out: + - `Get ".../configmaps/rfr-acl-": context deadline exceeded`, `ERROR load acl failed` + - `Get ".../pods/rfs--0": dial tcp ...:443: i/o timeout` + - `level=fatal msg="Error getting password from secret rfr-acl--operator-secret ...: context deadline exceeded"` + - `ERROR Access enable nodeport service access failed`, `ERROR Access load cached config failed` + + Events show `FailedPreStopHook`, `Back-off restarting failed container redis-exporter` and `Startup probe failed: dial tcp ...: connect: connection refused`. +- Harbor is inaccessible, and the `harbor-nginx` Pod stays `Running` with `0/1` Ready. + +In each case a policy was the cause, and correcting or removing it restored service. + +## Root Cause + +### Cause 1: a NetworkPolicy blocks the ingress controller from the backend + +A `NetworkPolicy` that selects the application Pods does not allow traffic from the ingress controller. Policies match what arrives at the Pod: the source is the ingress controller (a Pod, or the node when the controller runs on the host network), and the port is the container port behind the Service (`targetPort`), which can differ from the Service port. + +### Cause 2: a policy with Egress blocks Pod-to-Pod traffic in the namespace + +A network policy in the namespace selects the Pods with `Egress` in `policyTypes` and has no rule that allows traffic to the other Pods, so outbound traffic from those Pods is dropped. + +### Cause 3: the Redis Sentinel Pods cannot reach the Kubernetes API server + +At startup the Redis and Sentinel containers read their ACL ConfigMap, password Secret and Pod objects from the API server (Service port `443`). The required network path had not been opened, so every request timed out and the containers exited. Adjusting the Sentinel resource limits is not the fix. The block can be a Kubernetes `NetworkPolicy` or a rule on the network side, so check both. + +### Cause 4: harbor-nginx is 0/1 Ready after the cluster network policies were modified + +The cluster network policies were changed just before Harbor stopped serving, and restoring the previous policy configuration brought `harbor-nginx` back to Ready. + +## Resolution + +Save every policy before changing any of them, so each step can be undone: + +```bash +kubectl get networkpolicy -A -o yaml > /tmp/all-netpol.before.yaml +kubectl get adminnetworkpolicy -o yaml > /tmp/all-anp.before.yaml +``` + +### Cause 1: a NetworkPolicy blocks the ingress controller from the backend + +Find the container port and the controller's source, then add an ingress allow for that pair to the policy: + +```bash +kubectl -n get svc -o jsonpath='{.spec.ports}{"\n"}' +kubectl get pod -A -o wide | grep -iE 'alb|ingress' +kubectl -n get networkpolicy -o yaml +``` + +```yaml + ingress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: + ports: + - protocol: TCP + port: +``` + +If the controller Pods use the host network, a namespace selector does not match them; allow the node addresses, or on Kube-OVN the join subnet, with `ipBlock`. + +> **Warning:** Editing the policy changes who can reach every Pod it selects. Roll back with `kubectl apply -f /tmp/all-netpol.before.yaml`. + +```bash +kubectl -n edit networkpolicy +curl -sv --max-time 5 http:/// -o /dev/null +``` + +### Cause 2: a policy with Egress blocks Pod-to-Pod traffic in the namespace + +Delete the policy, or keep it and add an egress rule `to: - podSelector: {}`, which allows traffic to every Pod in the same namespace. + +> **Warning:** Deleting the policy removes all the isolation it gave to the Pods it selected. Get agreement from the namespace owner. Roll back with `kubectl apply -f /tmp/all-netpol.before.yaml`. + +```bash +kubectl -n delete networkpolicy +kubectl -n exec -- ping -c 3 +``` + +### Cause 3: the Redis Sentinel Pods cannot reach the Kubernetes API server + +Only when the Redis or Sentinel logs show `context deadline exceeded` or `dial tcp ...:443: i/o timeout` on API server requests. Open the path from the instance Pods to the API server: + +- If a `NetworkPolicy` with `Egress` selects those Pods, add an egress allow to the API server. On Kube-OVN the rule has to name the backend endpoints of the `kubernetes` Service (the addresses in its EndpointSlice, port `6443`), not only the ClusterIP; the rule shape is in "Operator manager pod stuck in CrashLoopBackOff when a NetworkPolicy denies egress to the apiserver on ACP". +- If no policy selects the Pods, the block is on the network path from the Pod network to the control-plane nodes; have it opened there. + +> **Warning:** An egress allow changes what the selected Pods can reach; limit it to the instance Pods and the API server endpoints. Roll back by deleting the allow policy you added. + +```bash +kubectl -n logs --all-containers --tail=50 | grep -E 'deadline|i/o timeout|failed' +kubectl -n get networkpolicy +kubectl get endpointslice -n default -l kubernetes.io/service-name=kubernetes +kubectl -n apply -f allow-apiserver-egress.yaml +kubectl -n get pod -w | grep -E 'rfr-|rfs-' +``` + +### Cause 4: harbor-nginx is 0/1 Ready after the cluster network policies were modified + +Put back the configuration from before the change, then reintroduce the intended change one rule at a time while watching `harbor-nginx` readiness. + +> **Warning:** Restoring the old policies removes whatever the change was meant to add. You need a copy of the configuration from before the change (an export, Git, or the previous console values). Re-apply the current copy saved above to roll back. + +```bash +kubectl apply -f .yaml +kubectl get pod -A | grep harbor-nginx +``` + +## Diagnostic Steps + +1. List every policy that can select the failing Pod and its peer, cluster-scoped ones included: + + ```bash + kubectl -n get pod --show-labels + kubectl -n get networkpolicy + kubectl get adminnetworkpolicy + kubectl get baselineadminnetworkpolicy + ``` + +2. Read why a Pod is not Ready; probe or upstream timeouts that began with a policy change point to that change: + + ```bash + kubectl -n describe pod | grep -A10 -iE 'Readiness|Events' + ``` + +3. Test the exact path with the Pod IP and the container port: + + ```bash + kubectl -n exec -- sh -c 'nc -vz -w 3 ' + ``` + +- Ingress to the Service fails and a policy selects the backend: Cause 1. +- Same-namespace Pods cannot ping and a policy has `Egress`: Cause 2. +- Redis Sentinel Pods crash-loop with API server calls on `443` timing out: Cause 3. +- `harbor-nginx` `0/1` right after cluster policy changes: Cause 4. + +If LoadBalancer VIPs (not application Pods) fail after isolation between projects is enabled, see "LoadBalancer VIP unreachable after project isolation or with a MetalLB address outside the node segment on ACP". diff --git a/docs/en/solutions/Calico_Pods_get_no_address_disabled_default_IPPool_Raven_fixed_IP_mismatch_reserved_IPs_or_deleted_raven_on_ACP.md b/docs/en/solutions/Calico_Pods_get_no_address_disabled_default_IPPool_Raven_fixed_IP_mismatch_reserved_IPs_or_deleted_raven_on_ACP.md new file mode 100644 index 000000000..74e72ac71 --- /dev/null +++ b/docs/en/solutions/Calico_Pods_get_no_address_disabled_default_IPPool_Raven_fixed_IP_mismatch_reserved_IPs_or_deleted_raven_on_ACP.md @@ -0,0 +1,106 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "3.18" + - "4.0" + - "4.2" + - "4.3" +--- + +# Calico Pods get no address: disabled default IPPool, Raven fixed IP mismatch, reserved IPs or deleted raven on ACP + +## Issue + +On a Calico cluster, Pods cannot get an address, or a subnet question comes up: + +- A Pod (for example a Kafka instance Pod) stays in `Init` because Calico allocates nothing from the default IPPool `default-ipv4-ippool`. +- A fixed-IP Deployment loops on one address because the Raven IP status and Calico IPAM disagree. +- Someone wants to reserve part of a Calico subnet so it is never allocated. +- The `raven` Deployment was deleted by mistake. + +On Calico clusters, `raven` is the platform component that converts platform `Subnet` resources into Calico `IPPool` objects and maintains the platform IP objects. + +Related articles: "Kube-OVN Pods fail with AddressOutOfRange, AddressConflict or no address allocated on ACP"; "Multus secondary networks: unexpected end of JSON input, duplicated addresses or several addresses on one NIC on ACP". + +## Root Cause + +### Cause 1: the default Calico IPPool is disabled + +`kubectl get ippools.crd.projectcalico.org default-ipv4-ippool -o yaml` shows `disabled: true` in the spec (next to `blockSize: 26`, `ipipMode: Always`, `natOutgoing: true`, `vxlanMode: Never`). Calico allocates nothing from a disabled pool, so Pods that depend on it wait in `Init`. + +### Cause 2: the Raven fixed IP stays empty while Calico IPAM still holds the address + +A Calico overlay Deployment with more pinned addresses than ready replicas leaves one Pod requesting an address that another Pod already has. Raven does not write the occupancy back and does not repair an existing IP object, so the CNI keeps selecting the address that Calico IPAM considers taken. A restart of the controller does not clear it; the repair is in a later 4.2.x release. + +### Cause 3: a Calico subnet cannot reserve part of its addresses + +There is no setting for reserved IPs on a Calico subnet. Excluding part of a Calico container subnet from allocation cannot be done by configuration; this is a product limitation. + +### Cause 4: the raven Deployment was deleted + +`raven` exports Calico subnet metrics such as `subnet_available_ip_count` and `subnet_used_ip_count` on port `10660` at `/metrics`, and it creates and deletes Calico `IPPool` objects to match the platform `Subnet` resources. While it is missing, those metrics stop and subnet-to-IPPool synchronization stops, which affects Pod creation and deletion. The platform records manual changes to `raven` (for example its resource limits) as a ResourcePatch (`rpch`) object, and a leftover `rpch` keeps the platform from restoring the default Deployment. + +## Resolution + +### Cause 1: the default Calico IPPool is disabled + +Remove the `disabled` field. + +> **Warning:** Re-enabling the pool lets every new Pod draw addresses from it again; if it was disabled on purpose to drain it, this undoes that plan. Only when no other pool is meant to serve these Pods. Save the object first; setting `disabled: true` again is the rollback. + +```bash +kubectl get ippools.crd.projectcalico.org -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidr,DISABLED:.spec.disabled' +kubectl get ippools.crd.projectcalico.org default-ipv4-ippool -o yaml > /root/ippool-default.before.yaml +kubectl patch ippools.crd.projectcalico.org default-ipv4-ippool --type=json -p='[{"op":"remove","path":"/spec/disabled"}]' +``` + +If a platform `Subnet` with the same name manages the pool, check that object as well, so the field is not set back. + +### Cause 2: the Raven fixed IP stays empty while Calico IPAM still holds the address + +Until the release with the fix, stop requesting the address that Calico still holds: remove it from the workload's fixed-IP list, or scale the Deployment down so only Pods with a consistent address remain. + +> **Warning:** Editing the fixed-IP list recreates Pods and drops the addresses you remove. Do not delete Calico IPAM blocks by hand; that can free an address a live Pod uses. Restoring the saved Deployment is the rollback and brings the conflict back. + +```bash +kubectl get ip -A | grep +kubectl get deploy -n -o yaml > /root/deploy.before.yaml +kubectl edit deploy -n +``` + +An IP object whose `containerID`, `nodeName` and `podName` are all empty, with `resource already exists` from the CNI, is a different case that a `raven` restart does fix; see "Kube-OVN fixed IP occupied or not released: held by another Pod, node down, duplicate entry or RollingUpdate on ACP". + +### Cause 3: a Calico subnet cannot reserve part of its addresses + +No configuration exists. Keep addresses that must not be allocated outside the Calico subnet's CIDR when planning it, or raise the requirement through the product channel. + +### Cause 4: the raven Deployment was deleted + +Delete the `rpch` objects that belong to `raven`; the platform then recreates the Deployment with its default specification. + +> **Warning:** Deleting the `rpch` drops every manual change recorded for `raven`, such as raised limits. Save it first and reapply the change to the recreated Deployment if you still need it. + +```bash +kubectl get rpch -A | grep raven +kubectl -n get rpch -o yaml > /root/raven-rpch.before.yaml +kubectl -n delete rpch +kubectl -n kube-system get deploy raven +``` + +## Diagnostic Steps + +```bash +kubectl get ippools.crd.projectcalico.org -o yaml | grep -E 'name:|cidr:|disabled:' +kubectl -n kube-system get deploy raven +kubectl get ip -A | grep
+kubectl -n kube-system get ds calico-node -o wide +``` + +- A pool (normally `default-ipv4-ippool`) with `disabled: true` and Pods waiting in `Init`: Cause 1. +- A fixed-IP Deployment whose Raven IP object and Calico allocation disagree on one address: Cause 2. Compare the two before deleting anything. +- A request to reserve addresses in a Calico subnet: Cause 3. +- No `raven` Deployment in `kube-system`: Cause 4. diff --git a/docs/en/solutions/Calico_ipipMode_reverts_Felix_metrics_off_bandwidth_annotation_breaks_Pods_systemd-udevd_loops_on_calico_tmp_on_ACP.md b/docs/en/solutions/Calico_ipipMode_reverts_Felix_metrics_off_bandwidth_annotation_breaks_Pods_systemd-udevd_loops_on_calico_tmp_on_ACP.md new file mode 100644 index 000000000..d6528892b --- /dev/null +++ b/docs/en/solutions/Calico_ipipMode_reverts_Felix_metrics_off_bandwidth_annotation_breaks_Pods_systemd-udevd_loops_on_calico_tmp_on_ACP.md @@ -0,0 +1,159 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Calico ipipMode reverts, Felix metrics off, bandwidth annotation breaks Pods, systemd-udevd loops on calico_tmp on ACP + +## Issue + +On a cluster that uses Calico as the CNI, a configuration change does not behave as expected: + +- `ipipMode` on the `IPPool` `default-ipv4-ippool` is changed from `Always` to `CrossSubnet` and flips back to `Always`. +- Felix monitoring shows no data and the Felix metrics port is not listening. +- New workloads cannot get an address because the default Calico BGP subnet has no free addresses, and other subnets are being considered. +- A Deployment stops starting Pods after a Calico bandwidth annotation was added to it. +- Node load is high because `systemd-udevd` keeps trying to configure the interfaces `calico_tmp_A` and `calico_tmp_B`. + +Related articles: "Pod traffic fails on Calico because of an IPPool destination, MTU, firewall, BGP sessions or IPv6 routes on ACP". + +## Root Cause + +### Cause 1: ipipMode CrossSubnet reverts to Always until the Subnet and the IPPool match + +The platform keeps a `Subnet` object and a Calico `IPPool` object with the same name (for example `default-ipv4-ippool`). The two have to stay consistent; otherwise the controller can overwrite one with the other, and the console can show a value different from the dataplane. Changing only the `IPPool` to `ipipMode: CrossSubnet` while the `Subnet` still says `Always` is the inconsistency that gets overwritten. The exact revert path was not confirmed. + +### Cause 2: Felix Prometheus metrics are off because prometheusMetricsEnabled is not set + +The Felix configuration has no `prometheusMetricsEnabled` setting, which means Felix metrics are disabled, so the metrics port is not listening and monitoring has nothing to scrape. Felix serves metrics on TCP `9091` when enabled; the endpoint is protected only by a self-signed certificate and has no authentication. + +### Cause 3: the default Calico BGP subnet has no free addresses + +New workloads cannot get an address from the default subnet because it is full. Placing business workloads on other subnets that still have free addresses restores allocation. Traffic between those subnets works without extra network policy. + +### Cause 4: Pods fail after a Calico bandwidth annotation was added + +A bandwidth annotation from the open-source Calico procedure (for example `kubernetes.io/ingress-bandwidth`) was added to the Pod template. Rate limiting did not work, and when a Pod was rebuilt the qdisc rules of the previous Pod had not been cleaned up, so the new Pod failed on a conflict between several qdiscs on one interface. The platform currently provides no supported method for Calico Pod bandwidth limiting. + +### Cause 5: systemd-udevd loops on calico_tmp_A and calico_tmp_B + +Calico XDP creates the temporary interfaces `calico_tmp_A` and `calico_tmp_B` and destroys them immediately. The host's `systemd-udevd` tries to configure each one, fails because it is already gone, and retries, which drives the node load up. XDP is an acceleration feature and disabling it has no functional impact. It is on by default in current versions and cannot be turned off at deployment time. + +## Resolution + +### Cause 1: ipipMode CrossSubnet reverts to Always until the Subnet and the IPPool match + +Set the same values on both objects in one change. + +> **Warning:** Changing `ipipMode` reprograms encapsulation and can reset cross-subnet tunnels while Felix applies the pool. Only when an `ipipMode: CrossSubnet` edit returns to `Always`. Save both objects first; restoring them is the rollback. + +```bash +kubectl get subnet default-ipv4-ippool -o yaml > /root/subnet-default-ipv4-ippool.before.yaml +kubectl get ippools.crd.projectcalico.org default-ipv4-ippool -o yaml > /root/ippool-default-ipv4-ippool.before.yaml +kubectl edit ippools.crd.projectcalico.org default-ipv4-ippool +kubectl edit subnet default-ipv4-ippool +``` + +Set these values where the fields exist: + +```yaml +ipipMode: CrossSubnet +vxlanMode: Never +``` + +If the `Subnet` uses a different key for the IPIP mode, change that key so its value matches the `IPPool`; do not add a parallel key. + +### Cause 2: Felix Prometheus metrics are off because prometheusMetricsEnabled is not set + +Enable Felix metrics on the `calico-node` DaemonSet. + +> **Warning:** The DaemonSet rolls and `calico-node` restarts on every node, which can briefly disturb container networking. Enabling the variable exposes the unauthenticated metrics endpoint on TCP `9091` on every node, so enable it only where a scraper needs it. Roll back by setting the variable to `false`, as described in "Disable Calico Node Metrics Port". + +```bash +kubectl -n kube-system get ds calico-node -o jsonpath='{.spec.template.spec.containers[*].name}{"\n"}' +kubectl -n kube-system set env ds/calico-node -c calico-node FELIX_PROMETHEUSMETRICSENABLED=true +kubectl -n kube-system rollout status ds/calico-node +ss -lnt | grep 9091 || true +``` + +### Cause 3: the default Calico BGP subnet has no free addresses + +Place new business workloads on a subnet that still has free addresses, using the subnet assignment the cluster already uses for business workloads, then confirm the new Pod address falls in that CIDR. Check the pools before moving any system namespace; this change concerns business workloads only, not `kube-system` or `cpaas-system`. + +```bash +kubectl get ippools.crd.projectcalico.org -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidr,DISABLED:.spec.disabled' +kubectl get subnet +kubectl -n get pod -o wide +``` + +### Cause 4: Pods fail after a Calico bandwidth annotation was added + +Remove the bandwidth annotations from the Pod template and let the Pods be recreated. + +> **Warning:** Editing the Pod template rolls the Deployment. Putting the annotation back brings the qdisc conflict back. + +```bash +kubectl -n get deploy -o jsonpath='{.spec.template.metadata.annotations}{"\n"}' +kubectl -n patch deploy --type=json -p='[{"op":"remove","path":"/spec/template/metadata/annotations/kubernetes.io~1ingress-bandwidth"}]' +``` + +`~1` encodes the `/` in the key. Repeat the patch for each bandwidth key the template has (for example `kubernetes.io/egress-bandwidth`). If Pods still fail afterwards, restarting kubelet and the container runtime on the node is the next step to have the stale qdiscs cleaned up; that restart affects every Pod on the node. + +### Cause 5: systemd-udevd loops on calico_tmp_A and calico_tmp_B + +Disable XDP on `calico-node`, then check that the temporary interfaces stop appearing. Analyzing `systemd-udevd` on the host is the alternative. + +> **Warning:** The change restarts every `calico-node` Pod, which affects workloads; do it in a maintenance window. Roll back by restoring the saved DaemonSet or unsetting the variable. + +```bash +kubectl -n kube-system get ds calico-node -o yaml > /root/calico-node.before-xdp.yaml +kubectl -n kube-system set env ds/calico-node -c calico-node FELIX_XDPENABLED=false +kubectl -n kube-system rollout status ds/calico-node +ip link show | grep calico_tmp || true +``` + +## Diagnostic Steps + +1. Compare the `Subnet` and the `IPPool` of the same name: + + ```bash + kubectl get subnet default-ipv4-ippool -o yaml | grep -iE 'ipip|vxlan' + kubectl get ippools.crd.projectcalico.org default-ipv4-ippool -o yaml | grep -iE 'ipipMode|vxlanMode' + ``` + + Different values: Cause 1. + +2. Check whether Felix metrics are enabled: + + ```bash + kubectl -n kube-system get ds calico-node -o yaml | grep -i -A1 PROMETHEUS + ss -lnt | grep 9091 || echo "9091 not listening" + ``` + + No `FELIX_PROMETHEUSMETRICSENABLED=true` and nothing listening on `9091`: Cause 2. + +3. Allocation failures with the default subnet full: Cause 3. + +4. Pod failures that start with a bandwidth annotation, with several qdiscs on the interface: + + ```bash + tc qdisc show | head + ``` + + Cause 4. + +5. `systemd-udevd` busy on the temporary interfaces: + + ```bash + journalctl -u systemd-udevd --since '-10 min' | grep calico_tmp | tail + ``` + + Cause 5. diff --git a/docs/en/solutions/Envoy_Gateway_access_log_shows_a_Kube-OVN_join_address_instead_of_the_real_client_IP_on_ACP.md b/docs/en/solutions/Envoy_Gateway_access_log_shows_a_Kube-OVN_join_address_instead_of_the_real_client_IP_on_ACP.md new file mode 100644 index 000000000..43223b665 --- /dev/null +++ b/docs/en/solutions/Envoy_Gateway_access_log_shows_a_Kube-OVN_join_address_instead_of_the_real_client_IP_on_ACP.md @@ -0,0 +1,136 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.3" +--- + +# Envoy Gateway access log shows a Kube-OVN join address instead of the real client IP on ACP + +## Issue + +The caller cannot be found in the Envoy Gateway access log, or the forwarded address is wrong: + +- Which field in the gateway log is the client address, and can the log format be changed? +- The client field never changes across browser requests: it stays an address from the Kube-OVN `join` subnet, not the browser host's address. `x-forwarded-for` and `downstream_remote_address` hold that same address. The browser name resolves straight to the Envoy Gateway LoadBalancer IP from a MetalLB pool, with no proxy in front, and the cluster CNI is Kube-OVN. +- After `externalTrafficPolicy` was changed to `Cluster`, `X-Forwarded-For` carries only OVN addresses, and the site must keep `Cluster` and still pass the external client address. +- How is X-Forwarded-For (XFF) handling turned on for the gateway? + +`downstream_remote_address` is the access-log field for the client IP: the address Envoy accepted the connection from. There is no console control for the gateway log format; a defined text format can be set with `EnvoyProxy` telemetry (see "Envoy Gateway limits and questions: 64 listeners, access logs, operator install, HMAC-SM3 and migration on ACP", Cause 2). + +## Root Cause + +### Cause 1: Service forwarding replaces the client address before Envoy sees the connection + +The client address is replaced on the Service path before the packet reaches Envoy, which matches source NAT applied by the Kubernetes Service forwarding on Kube-OVN; the exact mechanism was not confirmed. A `ClientTrafficPolicy` that targets the Gateway with `clientIPDetection.xForwardedFor.trustedCIDRs: [0.0.0.0/0, ::/0]` does not change it, because MetalLB inserts neither `X-Forwarded-For` nor PROXY protocol, and an XFF policy cannot recover an address that was never in the connection or in a header. Setting `externalTrafficPolicy` to `Local` on the gateway LoadBalancer Service made the real client address appear in the log. + +### Cause 2: Cluster must stay, so the original address has to come from a front proxy + +With `externalTrafficPolicy: Cluster`, Service forwarding may source-NAT the connection, and Envoy cannot read the original client from it. The options are: + +- use `Local` on the Envoy Gateway Service and land VIP traffic only on nodes that run a local Envoy Pod; or +- keep `Cluster` and have a front load balancer or proxy pass `X-Forwarded-For` or PROXY protocol. + +### Cause 3: XFF trust is a ClientTrafficPolicy and only helps when a proxy added the header + +XFF handling is a `ClientTrafficPolicy` (`gateway.envoyproxy.io/v1alpha1`) that targets the Gateway and sets `spec.clientIPDetection.xForwardedFor`, with exactly one of two fields (`kubectl explain`: "Only one of NumTrustedHops and TrustedCIDRs must be set."): + +- `numTrustedHops`: the client IP is taken from the Nth address from the right end of `X-Forwarded-For`; set it to the number of trusted proxies in front of the gateway (`0` for none, `1` for one load balancer, `2` for two layers). +- `trustedCIDRs`: when the connection's remote address is in a trusted CIDR, XFF entries are evaluated right to left and the first untrusted address is the client; if all are trusted, the leftmost entry is used. + +Trusting every CIDR does not fix Cause 1, because no proxy appended the real client there. + +## Resolution + +Read the live Service policy and the logged address before changing either: + +```bash +kubectl get svc -A -l gateway.envoyproxy.io/owning-gateway-name= \ + -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,POLICY:.spec.externalTrafficPolicy' +kubectl logs -n -c envoy --tail=20 +kubectl get subnet join -o jsonpath='{.spec.cidrBlock}{"\n"}' +``` + +### Cause 1: Service forwarding replaces the client address before Envoy sees the connection + +Set `externalTrafficPolicy: Local` on the `EnvoyProxy` the Gateway references (`spec.infrastructure.parametersRef`). A direct edit of the generated Service is reconciled away. + +> **Warning:** With `Local`, VIP packets that arrive on a node without a ready Envoy Pod are dropped, so clients can lose the VIP unless Envoy Pods run on the nodes that receive it. Only when `downstream_remote_address` is a join-subnet address and an XFF policy did not change it. Save the object; replacing the value back with `Cluster` is the rollback. + +```bash +kubectl get envoyproxy -n -o yaml > /tmp/envoyproxy-before.yaml +kubectl patch envoyproxy -n --type='json' -p='[ + {"op": "replace", "path": "/spec/provider/kubernetes/envoyService/externalTrafficPolicy", "value": "Local"} +]' +``` + +If the path does not exist yet, create it with a merge patch: + +```bash +kubectl patch envoyproxy -n --type=merge -p '{"spec":{"provider":{"type":"Kubernetes","kubernetes":{"envoyService":{"externalTrafficPolicy":"Local"}}}}}' +``` + +Scope the MetalLB `L2Advertisement` to the gateway nodes so the VIP is not announced from a node without an Envoy Pod, then send one request and confirm `downstream_remote_address` is the client: + +```bash +kubectl get pods -A -o wide -l gateway.envoyproxy.io/owning-gateway-name= +kubectl get l2advertisements.metallb.io -A -o yaml | grep -n -A6 nodeSelectors +``` + +### Cause 2: Cluster must stay, so the original address has to come from a front proxy + +Configure the front load balancer or proxy to append `X-Forwarded-For` or to use PROXY protocol, then apply Cause 3 so Envoy trusts that hop. If no front proxy exists, `Local` with VIP placement on Envoy nodes (Cause 1) is the option that preserves the address. + +### Cause 3: XFF trust is a ClientTrafficPolicy and only helps when a proxy added the header + +Set either `numTrustedHops` or `trustedCIDRs`, never both, in one `ClientTrafficPolicy` for the Gateway. For one load balancer in front: + +```yaml +apiVersion: gateway.envoyproxy.io/v1alpha1 +kind: ClientTrafficPolicy +metadata: + name: enable-xff + namespace: +spec: + targetRefs: + - group: gateway.networking.k8s.io + kind: Gateway + name: + clientIPDetection: + xForwardedFor: + numTrustedHops: 1 +``` + +For a first test, `trustedCIDRs` with `0.0.0.0/0` and `::/0` trusts every source; for production, list only the proxy ranges (for example private ranges used by internal proxies). + +> **Warning:** Trusting `0.0.0.0/0` and `::/0` lets any client set `X-Forwarded-For`, and allowlists or audit logs that use the detected client IP will follow it. Roll back by deleting the policy or removing `clientIPDetection`. + +```bash +kubectl explain clienttrafficpolicies.gateway.envoyproxy.io.spec.clientIPDetection.xForwardedFor +kubectl apply -f enable-xff.yaml +kubectl get clienttrafficpolicy -A +``` + +## Diagnostic Steps + +1. Read one access line and the Service policy together: + + ```bash + kubectl logs -n -c envoy --tail=20 + kubectl get svc -A -l gateway.envoyproxy.io/owning-gateway-name= \ + -o custom-columns='NAME:.metadata.name,POLICY:.spec.externalTrafficPolicy,IP:.status.loadBalancer.ingress[*].ip' + ``` + +2. Check for an existing XFF policy: + + ```bash + kubectl get clienttrafficpolicy -n -o yaml + ``` + +- `downstream_remote_address` inside the `join` subnet CIDR and the policy not `Local`, or unchanged with an XFF policy present: Cause 1. +- The policy is `Cluster` and must stay: Cause 2. +- A front proxy appends `X-Forwarded-For` and no `ClientTrafficPolicy` trusts that hop: Cause 3. + +3. After a `Local` change, `curl` the VIP from a node that does not run Envoy. If it fails while an Envoy node succeeds, the address is now correct and the failure is the reachability side effect of `Local`; restrict the VIP announcement to Envoy nodes. diff --git a/docs/en/solutions/Envoy_Gateway_limits_and_questions_64_listeners_access_logs_operator_install_HMAC-SM3_and_migration_on_ACP.md b/docs/en/solutions/Envoy_Gateway_limits_and_questions_64_listeners_access_logs_operator_install_HMAC-SM3_and_migration_on_ACP.md new file mode 100644 index 000000000..759d8af0a --- /dev/null +++ b/docs/en/solutions/Envoy_Gateway_limits_and_questions_64_listeners_access_logs_operator_install_HMAC-SM3_and_migration_on_ACP.md @@ -0,0 +1,133 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.2" + - "4.3" +--- + +# Envoy Gateway limits and questions: 64 listeners, access logs, operator install, HMAC-SM3 and migration on ACP + +## Issue + +Questions about what Envoy Gateway on ACP can do: + +- A single Gateway needs more than 64 listeners; how can the cap be raised? +- Gateway access logs cannot be found: Administrator > Operations Center > Logs > Query returns `共 0 条日志` (0 logs) for the Envoy Gateway Pod while the log collector already collects Kubernetes logs. How are access logs enabled, or put in a defined format? +- Does installing the Envoy Gateway Operator take over existing Ingress or ALB traffic? +- The Gateway must verify HMAC-SM3 signatures. +- After an upgrade to 4.2.2, how can every ALB entry be moved to Envoy Gateway? +- ingress-nginx is going out of maintenance; what is the migration plan? + +Related articles: "Envoy Gateway setup fails: Backend is disabled, platform certificate not selectable, TLS-RSA ciphers or Wasm x509 on ACP". + +## Root Cause + +### Cause 1: one Gateway is limited to 64 listeners + +The product cap is 64 listeners on a single Gateway. It is not a field on `EnvoyGatewayCtl` or on the Gateway and cannot be changed in the current product. + +### Cause 2: access logs are on the Envoy data-plane stdout and are not shown in the platform log query + +The Envoy data-plane Pod (named `envoy---`, container `envoy`, next to `shutdown-manager`) writes access logs as JSON lines to standard output, with fields such as `"method"`, `"protocol"`, `"response_code"`, `"response_code_details"` and `"route_name"`. The data-plane Pod can run in the operator namespace `envoy-gateway-operator`. Gateway logs cannot be displayed or queried in **Operations Center > Logs > Query**: the Pod can be selected as a filter, but the query returns no data. This is a product limitation, with log query support for the gateway component planned for a future version. `kubectl logs` on the data-plane Pod is the way to read them. + +When the Pod prints no access lines, or a defined text format is needed, access logging is configured with `spec.telemetry.accessLog` on the `EnvoyProxy` that the Gateway references. + +### Cause 3: installing the operator does not intercept Ingress or ALB traffic + +Installing the Envoy Gateway Operator alone changes no traffic. Traffic moves only when an `EnvoyGatewayCtl`, a `Gateway` and its data-plane Service exist and clients are pointed at it. + +### Cause 4: HMAC-SM3 signature verification is not an Envoy Gateway capability + +The installed Envoy Gateway features cannot verify HMAC-SM3 signatures, and there is no policy field for it. The external authorization guide does not add it either. + +### Cause 5: there is no whole-cluster procedure to replace ALB with Envoy Gateway + +No supported procedure exists to switch all ALB entries of a cluster to Envoy Gateway (asked after a 4.2.2 upgrade). Exposing an individual service through a new Gateway and HTTPRoute is a separate task, not a replacement plan for every ALB. + +### Cause 6: the ingress-nginx to Gateway API migration is documentation-based + +The recommended replacement for ingress-nginx is Gateway API with Envoy Gateway, and standard support covers it with documentation. A plan customized for one environment needs a feature request that includes all of the cluster's Ingress resources. + +## Resolution + +### Cause 1: one Gateway is limited to 64 listeners + +Count the listeners before assuming the cap is the failure. A higher number is a product requirement. + +```bash +kubectl get gateway -n -o jsonpath='{.spec.listeners[*].name}{"\n"}' | wc -w +``` + +### Cause 2: access logs are on the Envoy data-plane stdout and are not shown in the platform log query + +Read the data-plane Pod, not the operator or controller Pod: + +```bash +kubectl get pods -A -o wide -l gateway.envoyproxy.io/owning-gateway-name= +kubectl logs -n -c envoy --tail=100 +``` + +A permission error for a developer account is an RBAC question about who may read Pod logs, not a missing log switch. + +When the lines are missing or a defined format is required, add telemetry to the `EnvoyProxy` named by the Gateway's `spec.infrastructure.parametersRef`. Confirm the field path with `kubectl explain envoyproxy.spec.telemetry.accessLog` and follow it if it differs. Request headers can be added as `%REQ(HEADER_NAME)%` and response headers as `%RESP(HEADER_NAME)%`. + +```yaml +spec: + telemetry: + accessLog: + settings: + - format: + type: Text + text: '[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%"' + sinks: + - type: File + file: + path: /dev/stdout +``` + +> **Warning:** Changing `EnvoyProxy` telemetry rolls the data-plane Deployment and closes connections on the replaced Pods. Merge the block into the existing object; a file with only `telemetry` drops `provider` and other settings. Save the object first; applying the saved copy is the rollback. + +```bash +kubectl get envoyproxy -n -o yaml > /tmp/envoyproxy-before.yaml +kubectl edit envoyproxy -n +kubectl logs -n -c envoy --tail=50 +``` + +### Cause 3: installing the operator does not intercept Ingress or ALB traffic + +No traffic change is needed to answer it. Empty results below mean no data plane exists yet and existing Ingress and ALB listeners keep their traffic: + +```bash +kubectl get envoygatewayctl -A +kubectl get gateway -A +``` + +### Cause 4: HMAC-SM3 signature verification is not an Envoy Gateway capability + +Raise a product requirement that names the signature algorithm and the Gateway that needs it. + +### Cause 5: there is no whole-cluster procedure to replace ALB with Envoy Gateway + +Plan the move per service: create a Gateway and HTTPRoute for each service and move its clients, keeping the ALB entry until the new one is verified. + +### Cause 6: the ingress-nginx to Gateway API migration is documentation-based + +Follow "How to Migrate Ingress to Gateway API with Envoy Gateway on ACP" (ACP 4.3.x and later). It maps one ingress controller entry to one `Gateway` and one `Ingress` to one `HTTPRoute`, and converts annotations to filters and policies. For a plan made for one environment, export the Ingress resources and IngressClasses and attach them to a feature request, after removing anything the site does not allow to leave the environment: + +```bash +kubectl get ingressclass -o yaml > /tmp/ingressclass-all.yaml +kubectl get ingress -A -o yaml > /tmp/ingress-all.yaml +kubectl get ingress -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,CLASS:.spec.ingressClassName,HOSTS:.spec.rules[*].host' +``` + +## Diagnostic Steps + +- Listener count of 64 and an extra listener rejected: Cause 1. +- An empty result in the platform log query for the gateway Pod: Cause 2; run `kubectl logs` on the data-plane Pod. Lines present there confirm logging works; none means the telemetry block is needed. +- No `EnvoyGatewayCtl` and no `Gateway` in the cluster: Cause 3. +- An explicit HMAC-SM3 requirement and no matching field in `kubectl explain securitypolicy.spec`: Cause 4. +- A request for a whole-cluster ALB replacement: Cause 5. A migration plan from ingress-nginx: Cause 6. diff --git a/docs/en/solutions/Envoy_Gateway_setup_fails_Backend_is_disabled_platform_certificate_not_selectable_TLS-RSA_ciphers_or_Wasm_x509_on_ACP.md b/docs/en/solutions/Envoy_Gateway_setup_fails_Backend_is_disabled_platform_certificate_not_selectable_TLS-RSA_ciphers_or_Wasm_x509_on_ACP.md new file mode 100644 index 000000000..990ebf978 --- /dev/null +++ b/docs/en/solutions/Envoy_Gateway_setup_fails_Backend_is_disabled_platform_certificate_not_selectable_TLS-RSA_ciphers_or_Wasm_x509_on_ACP.md @@ -0,0 +1,134 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.2" + - "4.3" +--- + +# Envoy Gateway setup fails: Backend is disabled, platform certificate not selectable, TLS-RSA ciphers or Wasm x509 on ACP + +## Issue + +Envoy Gateway is installed, and a route, listener or policy does not work: + +- HTTPRoute status says `ResolvedRefs: UnsupportedValue: Failed to process route rule 0 backendRef 0: Backend is disabled in Envoy Gateway configuration.` A custom `Backend` was created so a virtual machine or another non-Service endpoint can be a route target. +- A certificate created in platform certificate management (Networking / Certificates) can be selected for an ALB but not for an Envoy Gateway HTTPS listener, even after it was assigned to all projects or namespaces were authorized. +- The platform certificate exists as a Secret in `cpaas-system` and the Gateway UI does not list it. +- Client TLS fails because the client offers only TLS-RSA cipher suites. +- An `EnvoyExtensionPolicy` Wasm module fails with `could not fetch Wasm OCI image: could not fetch manifest` and `x509: certificate signed by unknown authority`. + +Related articles: "Envoy Gateway limits and questions: 64 listeners, access logs, operator install, HMAC-SM3 and migration on ACP". + +## Root Cause + +### Cause 1: Backend is disabled in Envoy Gateway configuration + +The custom `Backend` kind (`gateway.envoyproxy.io`, kind `Backend`) is not enabled on the `EnvoyGatewayCtl` that owns this Gateway's GatewayClass; creating a `Backend` object does not turn the feature on. The switch is `spec.config.envoyGateway.extensionApis.enableBackend`. A flag on a different `EnvoyGatewayCtl` does not help. When the flag is already `true`, the error can persist until the controller is restarted, or until another configuration error on that `EnvoyGatewayCtl` is corrected. + +### Cause 2: the Gateway can reference a cpaas-system Secret only through a ReferenceGrant + +The Envoy Gateway certificate picker lists TLS Secrets in the Gateway namespace only. The platform certificate is synchronized to `cpaas-system` as a Secret of type `kubernetes.io/tls`. `Gateway.spec.listeners[].tls.certificateRefs` can point at a Secret in another namespace only when a `ReferenceGrant` in that namespace allows the Gateway namespace; the grant is per namespace, not per Secret name. + +### Cause 3: Networking / Certificates does not attach certificates to Envoy Gateway + +Certificates created in platform certificate management apply to ALB; they are not associated with Envoy Gateway or Ingress. A Gateway HTTPS listener references a Kubernetes Secret through `tls.certificateRefs`, and by default that Secret must be in the Gateway's namespace, so assigning a platform certificate to all projects does not make it selectable. **Configuration / Secrets** is the supported way to supply the certificate. This is a product scope limit, not a failed sync. + +### Cause 4: TLS-RSA cipher suites are disabled by default + +Envoy Gateway does not offer TLS-RSA key-exchange suites such as `TLS_RSA_WITH_AES_128_GCM_SHA256` by default, so a client that offers only those suites fails the handshake. Adding TLS-RSA suites to a `ClientTrafficPolicy` was only partly accepted: some suites were rejected by Envoy and others had no effect. No working product configuration enables them. + +### Cause 5: the Wasm OCI fetch does not accept a private CA + +The Wasm module is an OCI image in a registry whose certificate is signed by a CA that Envoy Gateway does not trust. Envoy Gateway does not support a custom CA for this fetch. + +## Resolution + +### Cause 1: Backend is disabled in Envoy Gateway configuration + +Find the `EnvoyGatewayCtl` whose GatewayClass the Gateway uses, enable the flag there, and restart the Envoy Gateway controller. + +> **Warning:** Restarting the `envoy-gateway` controller Deployment pauses reconciliation until the new Pod is Ready; data-plane traffic is served by the Envoy Pods and continues. Only when the HTTPRoute shows the Backend-disabled message. Save the object; setting `enableBackend` back and restarting again is the rollback. + +```bash +kubectl get envoygatewayctl -A +kubectl get gateway -n -o jsonpath='{.spec.gatewayClassName}{"\n"}' +kubectl get envoygatewayctl -n -o yaml > /tmp/envoygatewayctl-before.yaml +kubectl patch envoygatewayctl -n --type merge -p '{"spec":{"config":{"envoyGateway":{"extensionApis":{"enableBackend":true}}}}}' +kubectl get deploy -A -o name | grep envoy-gateway +kubectl rollout restart deploy/ -n +kubectl rollout status deploy/ -n +``` + +Do not edit the ConfigMap or Deployment generated by `EnvoyGatewayCtl` directly. If the error remains after the restart, compare the live `EnvoyGatewayCtl` with the intended configuration and correct the field that does not match. Check that the `Backend` object and the `backendRef` group and kind match the installed CRD. + +### Cause 2: the Gateway can reference a cpaas-system Secret only through a ReferenceGrant + +Only when the Secret is in `cpaas-system` with type `kubernetes.io/tls`. Create the grant in `cpaas-system`, using the `ReferenceGrant` version that `kubectl api-resources | grep referencegrant` prints. + +> **Warning:** The grant lets every Gateway in `` reference any Secret in `cpaas-system`. Roll back with `kubectl delete referencegrant allow--secrets -n cpaas-system`; listeners that reference `cpaas-system` Secrets then fail their reference check. + +```bash +kubectl get secret -n cpaas-system -o jsonpath='{.type}{"\n"}' +kubectl apply -f - <<'EOF' +apiVersion: gateway.networking.k8s.io/v1beta1 +kind: ReferenceGrant +metadata: + name: allow--secrets + namespace: cpaas-system +spec: + from: + - group: gateway.networking.k8s.io + kind: Gateway + namespace: + to: + - group: "" + kind: Secret +EOF +``` + +On the listener, set `tls.certificateRefs` to the Secret name with namespace `cpaas-system`. Edit the exported Gateway and apply the whole file; a listeners-only manifest drops the other listeners and `infrastructure.parametersRef`. The UI still lists only Secrets of the Gateway namespace, so this reference stays in YAML. Keeping the Secret name unchanged means a certificate renewal needs no edit. + +### Cause 3: Networking / Certificates does not attach certificates to Envoy Gateway + +Create a TLS Secret in the Gateway namespace (**Configuration / Secrets**, or the command below) and reference it from the listener's `tls.certificateRefs`. If the needed Secret is already the platform copy in `cpaas-system`, use Cause 2 instead of creating a second copy. + +```bash +kubectl create secret tls -n --key= --cert= +kubectl get secret -n -o jsonpath='{.type}{"\n"}' +``` + +### Cause 4: TLS-RSA cipher suites are disabled by default + +Move the client to cipher suites that Envoy offers. RSA key exchange may also fail security compliance requirements. + +### Cause 5: the Wasm OCI fetch does not accept a private CA + +Host the Wasm OCI image in a registry served over HTTP or with a publicly trusted certificate, and point the `EnvoyExtensionPolicy` image reference at it. + +## Diagnostic Steps + +1. Read the route and Gateway status: + + ```bash + kubectl get httproute -n -o yaml + kubectl get gateway -n -o yaml + kubectl get backend -A + ``` + + `Backend is disabled in Envoy Gateway configuration`: Cause 1. A missing `Backend` CRD is a different failure. + +2. For certificates, compare namespaces: + + ```bash + kubectl get secret -n --field-selector type=kubernetes.io/tls + kubectl get secret -n cpaas-system --field-selector type=kubernetes.io/tls + kubectl get referencegrant -n cpaas-system + ``` + + Secret only in `cpaas-system` and no `ReferenceGrant`: Cause 2. Certificate only under Networking / Certificates: Cause 3. + +3. A client cipher list with only TLS-RSA suites: Cause 4. `EnvoyExtensionPolicy` status with `certificate signed by unknown authority`: Cause 5. diff --git a/docs/en/solutions/External_load_balancer_in_front_of_ALB_stalls_pool_members_client_IP_health_checks_and_503_on_ACP.md b/docs/en/solutions/External_load_balancer_in_front_of_ALB_stalls_pool_members_client_IP_health_checks_and_503_on_ACP.md new file mode 100644 index 000000000..29550ffd6 --- /dev/null +++ b/docs/en/solutions/External_load_balancer_in_front_of_ALB_stalls_pool_members_client_IP_health_checks_and_503_on_ACP.md @@ -0,0 +1,181 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.1" + - "4.2" + - "4.3" +--- + +# External load balancer in front of ALB: stalls, pool members, client IP, health checks and 503 on ACP + +## Issue + +The path is client, then an external load balancer (an F5 or a cloud load balancer) or the platform VIP, then the ALB (`global-alb2` for the platform, or a business ALB), then the Pods. Typical reports: + +- The console is often slow through the F5 address: a page or API call takes about 19 seconds, especially on first open, while the ALB shows the request finishing quickly. +- A business service behind the ALB fails intermittently, and the ALB logs show no errors. +- The console does not load: the browser shows `ERR_CONNECTION_CLOSED`, and `curl` to the platform domain fails with `curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL`, while a request straight to a master on 443 works. +- The platform audit log records node IPs instead of the real client; on a global cluster without an external load balancer, `login` events show master IPs in `SourceIPs`. +- Which nodes belong in the F5 pool, and whether the health check can probe a `/healthz` endpoint. +- A port added on an ALB is unreachable through the F5, or a new load balancer returns `503 Service Unavailable` through the external load balancer while the ALB answers inside the cluster. +- The health check of a VIP shows only one of the three global masters up for port `80`. + +The ALB access log in the `nginx` container is the dividing line: a request that never shows up there was lost in front of the ALB. + +## Root Cause + +### Cause 1: requests stall for about 19 seconds only through the F5 VIP + +In the browser timing of a slow request, Stalled is about 19 seconds of the total while `Waiting for server response` is tens of milliseconds. The `global-alb2` access log shows the same requests with `status: 200` and `request_time` well under a second; the protocol is HTTP/2, and going to the ALB address directly does not reproduce it. A client capture with the filter `tcp.analysis.retransmission` shows repeated `[TCP Retransmission]` to port 443 lasting about 19 seconds, only toward that F5 VIP. The F5 virtual server for 443 had X-Forwarded-For (XFF) insertion enabled; disabling it stopped the stalls. + +### Cause 2: the F5 pool contains a node that does not run this ALB + +The pool covered four nodes while the ALB Pods ran on three. The fourth node ran another ALB that also listened on 443 but had no rule for this service. Because the port was open, the health check marked it healthy and part of the traffic went there. + +### Cause 3: the console does not load and global-alb2 logs no request + +`curl -kv https://:443` connects, sends the TLS `Client hello`, then fails with `SSL_ERROR_SYSCALL`, and `global-alb2` logs nothing for browser or `curl` requests through the domain, although the domain resolves correctly. A request straight to a master on 443 completes the TLS handshake, returns `HTTP/2 302` with `location: /console-platform/portal` and `via: alb`, and is logged. The fault is in the F5 and network in front of the platform. + +### Cause 4: a Layer-4 load balancer hides the client IP from the audit log + +A TCP (Layer-4) load balancer on ports 443, 6443 and 11443 does not pass the client address, so the audit records show the backend node IPs as the source. + +### Cause 5: the health check needs a target + +No `/healthz`-style endpoint is provided for the external load balancer to probe on the platform port 443; a TCP check on the port is the supported method. + +### Cause 6: the F5 port for a new ALB listener is wrong + +The port configured on the F5 for the ALB did not match the port the ALB listens on, so the F5 did not forward it. + +### Cause 7: 503 from the external load balancer because the firewall allows only the MetalLB address + +The external load balancer returns `HTTP/1.1 503 Service Unavailable` with `This server is temporarily unable to service requests.`, while from inside the cluster, with the domain pointed at the ALB's MetalLB address, `curl -v http:///` returns `200` with `Via: alb`. The firewall between the external load balancer and the cluster allowed traffic only to the MetalLB address. That address is not bound to a server: MetalLB announces it by ARP from one of the nodes and it can move, so traffic to the node IPs that actually carry it was blocked. + +### Cause 8: login events record master IPs because the self-built global VIP runs in IPVS mode + +Without an external load balancer, the global VIP is the self-built VIP (alive). In its default mode alive forwards with IPVS, which does not guarantee that the source IP is passed through, so a `login` audit event (`RequestURI` `/api/v2/token/callback?...`) can list a master address in `SourceIPs`, while requests through the ALB keep the client address. + +### Cause 9: a VIP health check shows one member up because only one ALB listens on that port + +A VIP can carry several ports with different owners: `80` and `443` for HTTP and HTTPS access, `11443` for the platform image registry, `6443` for the Kubernetes API server, and NodePorts of deployed tools. When port `80` on the global VIP is not used for platform access but is served only by a host-network business ALB that is bound to one master by its node selector, the health check for `80` shows only that master up. That is expected, not a fault. + +## Resolution + +### Cause 1: requests stall for about 19 seconds only through the F5 VIP + +Disable XFF on the F5 virtual server for port 443 of the platform. Until the F5 owner changes it, accessing the platform through the ALB address directly is a workaround. Compare with an F5 configuration that does not stall before changing other settings. + +### Cause 2: the F5 pool contains a node that does not run this ALB + +Keep the pool to the nodes that actually run this ALB's Pods, and keep it in step with the ALB node selector (shown on **Network Management > Load Balancers > Update**) when nodes are added or relabeled. + +```bash +kubectl -n cpaas-system get pods -l service_name=alb2- -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName +kubectl get nodes -l = +``` + +### Cause 3: the console does not load and global-alb2 logs no request + +Prove the split with the commands in Diagnostic Steps and hand it to the F5 and network teams; no platform change is required. + +### Cause 4: a Layer-4 load balancer hides the client IP from the audit log + +Change the front listener for port 443 to HTTP(S) mode with XFF enabled; the ALB processes XFF correctly. If the load balancer terminates HTTPS, configure the platform's `dex.tls` certificate on it. Keep 6443 and 11443 as TCP. After the switch, re-run the timing and retransmission checks, because Cause 1 involved XFF on a 443 virtual server. + +### Cause 5: the health check needs a target + +Configure a TCP check on 443, pointed only at the pool members of Cause 2. + +### Cause 6: the F5 port for a new ALB listener is wrong + +Read the listener ports, test them on the ALB nodes, and have the F5 owner correct the virtual server or pool port: + +```bash +kubectl -n cpaas-system get ft | grep +nc -vz -w 3 +``` + +### Cause 7: 503 from the external load balancer because the firewall allows only the MetalLB address + +Open the firewall from the external load balancer to the IP addresses of all cluster nodes that can announce the MetalLB address (the **Available Nodes** of the external address pool), not only to the MetalLB address. + +### Cause 8: login events record master IPs because the self-built global VIP runs in IPVS mode + +Switch alive to ARP mode. + +> **Warning:** Redeploying alive moves the global VIP, briefly interrupting the console, API calls and image pulls through it, and the registry static Pod is recreated on each master. Do it in a maintenance window, with copies of `/etc/kubernetes/alive/alive.yaml`, `/etc/kubernetes/alive/vipmode` (if present) and `/etc/kubernetes/manifests/registry.yaml` from every master. Roll back by restoring the annotation to its previous value and the saved files. + +1. Only on a cluster upgraded from 4.0: on every node that runs alive, save and remove the leftover mode file. + + ```bash + cp /etc/kubernetes/alive/vipmode /root/alive-vipmode.backup + rm /etc/kubernetes/alive/vipmode + ``` + +2. On the global cluster, annotate the cluster; alive is redeployed and `alive.yaml` gains `vip_mode: arp`. + + ```bash + kubectl annotate clusters.platform.tkestack.io cpaas.io/alive-vip-mode=arp + grep -n vip_mode /etc/kubernetes/alive/alive.yaml + ``` + +3. When the global cluster uses its built-in registry behind the same VIP, edit `/etc/kubernetes/manifests/registry.yaml` on all three masters and change `--server-bind-address=$(MY_POD_IPS)` to `--server-bind-address=0.0.0.0`; the kubelet recreates the registry Pod. + +Then log in from a known client and check that the new `login` event records that address. + +### Cause 9: a VIP health check shows one member up because only one ALB listens on that port + +No change is needed when the port belongs to a single-node host-network ALB. Identify which ALB serves the port and where it runs; if the port should be served on all members, move or scale that ALB (or change the VIP pool to its nodes). + +```bash +kubectl get pods -A -o wide | grep alb +kubectl get ft -A | grep -E '\b80\b|00080' +ss -lntp | grep ':80 ' +``` + +## Diagnostic Steps + +1. Separate the load balancer from the platform: from the same client, call the domain and a master directly while watching the ALB log: + + ```bash + curl -k -o /dev/null -sS -w 'code=%{http_code} connect=%{time_connect} total=%{time_total}\n' https://platform.example.com/ + curl -k -o /dev/null -sS -w 'code=%{http_code} total=%{time_total}\n' --resolve platform.example.com:443: https://platform.example.com/ + kubectl -n cpaas-system logs -f -l service_name=alb2-global-alb2 -c nginx --max-log-requests=6 | grep --line-buffered '' + ``` + + A slow or failing domain call that never reaches the log, with a fast direct call, is in front of the ALB (Causes 1 and 3). + +2. For Cause 3, the decisive pair is: + + ```bash + curl -kv https://platform.example.com:443 2>&1 | grep -E 'Client hello|SSL_ERROR_SYSCALL|HTTP/' + curl -kv https://:443 2>&1 | grep -E 'subject:|HTTP/|location:|via:' + ``` + +3. For Cause 1, capture on the client and count retransmissions per target: + + ```bash + tcpdump -nn -i host 192.0.2.10 and tcp port 443 -w /tmp/f5-vip.pcap + tshark -r /tmp/f5-vip.pcap -Y tcp.analysis.retransmission | wc -l + ``` + +4. For Causes 2 and 9, compare pool members with ALB placement, and run `ss -lntp | grep -E ':(80|443) '` on each member; a listener that belongs to a different ALB is Cause 2, a port served by one single-node ALB is Cause 9. + +5. For Cause 8, confirm the recorded addresses are masters, not Pods or Services, and read the alive mode: + + ```bash + kubectl get pod -A -o wide | grep + kubectl get clusters.platform.tkestack.io -o jsonpath='{.metadata.annotations.cpaas\.io/alive-vip-mode}{"\n"}' + grep -n -E 'vip_mode|ipvs' /etc/kubernetes/alive/alive.yaml + ``` + +6. For a `503` the ALB did not generate (Cause 7), test the rule from inside the cluster with the domain pinned to the ALB address; `Via: alb` and the expected status put the fault in front of the ALB: + + ```bash + curl -sv --resolve app.example.com:80:192.0.2.40 http://app.example.com/ 2>&1 | grep -E '^< HTTP|^< Via' + ``` diff --git a/docs/en/solutions/Host_DNS_changes_do_not_reach_CoreDNS_systemd-resolved_kubelet_resolvConf_missing_nameserver_or_a_new_upstream_on_ACP.md b/docs/en/solutions/Host_DNS_changes_do_not_reach_CoreDNS_systemd-resolved_kubelet_resolvConf_missing_nameserver_or_a_new_upstream_on_ACP.md new file mode 100644 index 000000000..cc40e5a29 --- /dev/null +++ b/docs/en/solutions/Host_DNS_changes_do_not_reach_CoreDNS_systemd-resolved_kubelet_resolvConf_missing_nameserver_or_a_new_upstream_on_ACP.md @@ -0,0 +1,182 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Host DNS changes do not reach CoreDNS: systemd-resolved, kubelet resolvConf, missing nameserver or a new upstream on ACP + +## Issue + +The DNS servers of the hosts were changed and Pods still resolve through the old servers, or the question is where CoreDNS gets its upstream servers. Typical reports: + +- "The Corefile says `forward . /etc/resolv.conf`. Which file is that, and how does it get into CoreDNS? Is `/run/systemd/resolve/resolv.conf` used by default?" +- DNS was changed on an Ubuntu node, the `/etc/resolv.conf` inside the CoreDNS Pod still shows the old servers, and manual edits to the host file are reverted. +- Production Pods need to reach a public domain whose DNS answers change at random. +- An external DNS server has to be added to CoreDNS, and the standard steps are requested. +- A new global installation hangs at the Install Kubernetes step and the CNI plugin is never created; after the installation finally completes, the platform page keeps loading, `apollo` logs timeouts, and CoreDNS logs that no nameserver is configured. + +The platform CoreDNS runs as Deployment `coredns` in `kube-system` with the ConfigMap `cpaas-coredns`. Its default server block forwards with: + +```text +forward . /etc/resolv.conf { + max_concurrent 1000 + max_fails 0 + policy sequential +} +``` + +For a Pod with `dnsPolicy: ClusterFirst`, a name is looked up in the Pod's own `/etc/hosts`, then in the `hosts` entries of the CoreDNS ConfigMap (cluster Service names are answered by the `kubernetes` plugin), and finally by the servers in the `/etc/resolv.conf` that the CoreDNS Pod received from the node it runs on. + +## Root Cause + +### Cause 1: on Ubuntu nodes, kubelet resolvConf is /run/systemd/resolve/resolv.conf and systemd-resolved reverts edits + +CoreDNS does not read the node file directly. The kubelet writes each Pod's `/etc/resolv.conf` when the Pod is created; for a Pod with `dnsPolicy: Default`, which is how CoreDNS resolves its upstreams, the kubelet copies the file named by `resolvConf` in `/var/lib/kubelet/config.yaml`. (`clusterDNS` in the same file is the `kube-dns` address given to `ClusterFirst` Pods.) + +In a cluster with mixed nodes, CentOS nodes had `resolvConf: /etc/resolv.conf` and Ubuntu nodes had `resolvConf: /run/systemd/resolve/resolv.conf`. On Ubuntu that file is generated by `systemd-resolved`, which rewrites it and reverts manual DNS changes, so the host change never reached CoreDNS. `systemd-resolved` is not supported with Kubernetes on these nodes: disable it and use a static resolver file. A running Pod also keeps its old file until it is recreated. + +### Cause 2: a public domain whose addresses change at random + +A fixed hosts record would go stale for such a domain. CoreDNS forwards names it does not own to the servers in its resolver file, which comes from the nodes it runs on (Cause 1). Once the host resolver answers the domain, Pods resolve it through CoreDNS and get the current answers, and the containers stay on cluster DNS. + +### Cause 3: Install Kubernetes stalls and the platform page keeps loading, CoreDNS reports a missing nameserver + +On a new ACP 4.3.2 deployment on ARM, the installer stopped at Install Kubernetes and the CNI plugin was not created. Inside the installer container `minialauda-control-plane`, CoreDNS was abnormal and logged a missing nameserver: `/etc/resolv.conf` on the installer node had no usable `nameserver` line. The same gap then appeared on the new global cluster: the masters' `/etc/resolv.conf` had no usable `nameserver`, CoreDNS stayed abnormal, the platform page kept loading, and restarting `apollo`, `archon` and `erebus` did not help. + +### Cause 4: an external DNS server is added through the node resolver file + +Because the default server block uses `forward . /etc/resolv.conf`, the upstream servers of CoreDNS are the `nameserver` lines of `/etc/resolv.conf` on the nodes that run CoreDNS. Adding an external DNS server therefore means adding it to that file on every node and recreating the CoreDNS Pods so the kubelet writes the new list into them. With `policy sequential`, the servers are tried in the order they are listed. The kubelet copies at most three `nameserver` entries into a Pod's resolver file, and the node scale-out precheck accepts only two. + +## Resolution + +### Cause 1: on Ubuntu nodes, kubelet resolvConf is /run/systemd/resolve/resolv.conf and systemd-resolved reverts edits + +Only when `resolvConf` points at `/run/systemd/resolve/resolv.conf` and `systemd-resolved` is active on the node. On each Ubuntu node, back up the resolver content, stop and disable `systemd-resolved`, replace the `/etc/resolv.conf` symlink with a regular file, and put the new servers in it. On VMs, check whether cloud-init manages the file. + +> **Warning:** Until the new file is in place, host processes on the node cannot resolve names, and a wrong server breaks external lookups for the node and, through CoreDNS, for Pods. Work one node at a time, and confirm each new server answers `dig @ ` first. Roll back with `cp -a /root/resolv.conf.before /etc/resolv.conf` (or `ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf`) and `systemctl enable --now systemd-resolved`. + +```bash +grep -n -E 'resolvConf|clusterDNS' /var/lib/kubelet/config.yaml +systemctl is-active systemd-resolved +ls -l /etc/resolv.conf +cat /run/systemd/resolve/resolv.conf > /root/resolv.conf.before +systemctl disable --now systemd-resolved +rm -f /etc/resolv.conf +cp /root/resolv.conf.before /etc/resolv.conf +vi /etc/resolv.conf +grep -rn -iE 'manage_resolv_conf|resolv' /etc/cloud/cloud.cfg /etc/cloud/cloud.cfg.d/ 2>/dev/null +``` + +`/run` is a tmpfs, so with `systemd-resolved` disabled nothing recreates `/run/systemd/resolve/resolv.conf` after a reboot, while the kubelet still needs the file it names. Either recreate that file from the node initialization script with the same content as `/etc/resolv.conf`, or point the kubelet at `/etc/resolv.conf` and restart it: + +> **Warning:** The node stops reporting status briefly while the kubelet restarts; running containers keep running. Save the configuration first; restoring it and restarting again is the rollback. + +```bash +cp -a /var/lib/kubelet/config.yaml /root/kubelet-config.before.yaml +sed -i 's#^resolvConf: /run/systemd/resolve/resolv.conf#resolvConf: /etc/resolv.conf#' /var/lib/kubelet/config.yaml +systemctl restart kubelet +``` + +Then recreate the CoreDNS Pods so they receive the new file: + +> **Warning:** Lookups are briefly delayed while each replica is replaced. Only after the node files are corrected on every node that can run CoreDNS, with more than one replica Ready. + +```bash +kubectl -n kube-system rollout restart deploy/coredns +kubectl -n kube-system rollout status deploy/coredns +kubectl -n kube-system exec -- cat /etc/resolv.conf +``` + +If `exec` is not possible, `crictl inspectp | grep -A8 -i dnsconfig` on the node lists the servers the kubelet gave the Pod. Update the node initialization script so new Ubuntu nodes are built the same way. + +### Cause 2: a public domain whose addresses change at random + +Make the resolver on every node that may run CoreDNS answer the domain, restart CoreDNS as in Cause 1, and test from a business Pod: + +```bash +kubectl -n kube-system get pod -l k8s-app=kube-dns -o wide +dig +short +kubectl exec -n -- nslookup +``` + +A name with a stable address can instead be added as a CoreDNS Host Alias; see "CoreDNS Host Alias records do not take effect because a ResourcePatch removes data hosts from cpaas-coredns on ACP". + +### Cause 3: Install Kubernetes stalls and the platform page keeps loading, CoreDNS reports a missing nameserver + +During installation, fix `/etc/resolv.conf` on the node that runs the installer, then restart the installer container. + +> **Warning:** While `minialauda-control-plane` restarts, the installation progress page stops updating. A wrong nameserver breaks name resolution for every process on the node. Back up the file and confirm the nameserver answers from the node first; `cp -a /root/resolv.conf.before /etc/resolv.conf` is the rollback. + +```bash +cp -a /etc/resolv.conf /root/resolv.conf.before +vi /etc/resolv.conf +sudo nerdctl exec minialauda-control-plane kubectl --kubeconfig /etc/kubernetes/admin.conf -n kube-system get pod -o wide | grep coredns +sudo nerdctl restart minialauda-control-plane +``` + +After installation, if the page keeps loading and the global CoreDNS log still reports the missing nameserver, correct `/etc/resolv.conf` on every master the same way, and recreate the CoreDNS Pods if they do not become Ready (a Pod's resolver file is written when it is created). Restarting `apollo`, `archon` or `erebus` is not the fix. + +```bash +kubectl -n kube-system logs -l k8s-app=kube-dns --tail=20 +for n in ; do ssh $n 'hostname; grep ^nameserver /etc/resolv.conf'; done +``` + +### Cause 4: an external DNS server is added through the node resolver file + +1. Confirm that the Corefile forwards to the node file, and that the new server answers from a node: + + ```bash + kubectl -n kube-system get cm cpaas-coredns -o jsonpath='{.data.Corefile}{"\n"}' | grep -A5 forward + dig @ +short + ``` + +2. On every cluster node, add the server as a `nameserver` line in `/etc/resolv.conf`, in the position it should be tried. + + > **Warning:** A wrong or unreachable server delays or breaks external lookups for the node and, through CoreDNS, for all Pods. Back up the file on each node; restoring it and restarting CoreDNS again is the rollback. + + ```bash + cp -a /etc/resolv.conf /root/resolv.conf.before + vi /etc/resolv.conf + ``` + +3. Restart CoreDNS so it loads the new upstream list (Warning in Cause 1), then test from a Pod: + + ```bash + kubectl -n kube-system rollout restart deploy/coredns + kubectl -n kube-system rollout status deploy/coredns + kubectl exec -n -- nslookup + ``` + +If the server should answer only some zones, or the node file must not change, forward those zones from the Corefile instead; see "Pods cannot resolve intranet or external names: nameserver order, internal DNS server or debug container ping on ACP". + +## Diagnostic Steps + +1. Read which file the kubelet hands to `dnsPolicy: Default` Pods, and the CoreDNS DNS policy: + + ```bash + grep -n -E 'resolvConf|clusterDNS' /var/lib/kubelet/config.yaml + kubectl -n kube-system get deploy coredns -o jsonpath='{.spec.template.spec.dnsPolicy}{"\n"}' + ``` + +2. On Ubuntu nodes, check whether `systemd-resolved` owns the file: + + ```bash + systemctl is-active systemd-resolved + ls -l /etc/resolv.conf /run/systemd/resolve/resolv.conf + ``` + +3. Compare the node file with what CoreDNS received; different servers after a host change mean CoreDNS was not recreated or the kubelet copies a different file: + + ```bash + kubectl -n kube-system exec -- cat /etc/resolv.conf + ``` + +- Ubuntu node, `resolvConf` is `/run/systemd/resolve/resolv.conf`, host edits reverted: Cause 1. +- A public domain with changing answers: Cause 2. +- New installation stalls or the page keeps loading after installation, and CoreDNS reports a missing nameserver: Cause 3. If the node file exists but is empty after a node restart, see "CoreDNS fails after a node restart or lookups time out when a CoreDNS node is shut down on ACP". +- A request to add an upstream DNS server: Cause 4. diff --git a/docs/en/solutions/Kube-OVN_Pods_fail_with_AddressOutOfRange_AddressConflict_or_no_address_allocated_on_ACP.md b/docs/en/solutions/Kube-OVN_Pods_fail_with_AddressOutOfRange_AddressConflict_or_no_address_allocated_on_ACP.md new file mode 100644 index 000000000..485d61a9d --- /dev/null +++ b/docs/en/solutions/Kube-OVN_Pods_fail_with_AddressOutOfRange_AddressConflict_or_no_address_allocated_on_ACP.md @@ -0,0 +1,171 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "3.18" + - "4.0" + - "4.2" + - "4.3" +--- + +# Kube-OVN Pods fail with AddressOutOfRange, AddressConflict or no address allocated on ACP + +## Issue + +On a Kube-OVN cluster, a Pod stays in `ContainerCreating` or comes up with the wrong address. Typical reports: + +- The Pod event is `AcquireAddressFailed` with `AddressOutOfRange`, or a pinned address flaps `AddressConflict` after a fast delete. +- Pods in some namespaces fail while other namespaces work, with `request ip return 500 no address allocated to pod / provider ovn, please see kube-ovn-controller logs to find errors`. +- `arping` from a Pod gets a reply from an external host, or an external host on the underlay VLAN stops answering while a Pod holds the same address. +- In the console's fixed-IP dropdown, the subnet's `.2` address is offered although it is the u2o interconnection address. +- On ACP 3.18.2 the console does not allow adding a second IPv4 CIDR to an existing Subnet. +- The question is whether the fixed IP field takes a range such as `..`. + +Restarting `ovs-ovn` fixes none of these. + +Related articles: "Calico Pods get no address: disabled default IPPool, Raven fixed IP mismatch, reserved IPs or deleted raven on ACP"; "Multus secondary networks: unexpected end of JSON input, duplicated addresses or several addresses on one NIC on ACP"; "Kube-OVN fixed IP rejected or no address allocated because a stale IP object or logical switch port holds it on ACP". + +## Root Cause + +### Cause 1: on ACP 3.18.2, Subnet spec.cidrBlock holds one IPv4 CIDR + +A single-stack Subnet stores one CIDR in `spec.cidrBlock`; a dual-stack Subnet stores one IPv4 and one IPv6 CIDR. The console can edit that value but cannot append another IPv4 block. Editing the original CIDR in place strands the Pods that already use it. Likewise, `spec.protocol` cannot be changed after creation, so a Subnet created as `IPv4` cannot serve a workload that asks for an IPv6 address. + +### Cause 2: AddressOutOfRange in an underlay fixed-IP Deployment + +Two problems appear together: + +- The Subnet's NIC binding is wrong, so allocation is refused until the VLAN or ProviderNetwork points at the right interface. +- The number of pinned addresses equals `spec.replicas` while `strategy.type` is `RollingUpdate`. The new Pod asks for an address the old Pod has not released, and `kube-ovn-controller` still holds the old IP object. + +The same race makes a pinned address on a Deployment flap `AddressConflict` after a fast delete, because the ReplicaSet starts the replacement immediately. A StatefulSet waits until the previous ordinal is gone. + +### Cause 3: an external device or a previous platform still uses addresses in the underlay CIDR + +Dynamic allocation hands out any address of the underlay CIDR that is not listed in `spec.excludeIps`. When an external host, the gateway, or an old platform sharing the VLAN uses such an address, a Pod can be given it: `arping` from the Pod gets a reply from the external host, ICMP to that host stops returning, and OVN logs `MAC_Binding` conflicts for one `logical_port` and IP, sometimes with `OVNSB commit failed`. An address that is "usually free" is not free. + +### Cause 4: the u2o interconnection address is offered as a free static IP + +With `u2oInterconnection` enabled, the subnet's second address (`.2`) is by default the u2o interconnection address. Kube-OVN creates a `u2o-interconnection..` IP object only when the u2o address cannot use `.2` (for example because `.2` is in `excludeIps`); when `.2` is used, there is no IP object for it, which is expected. The defect is that the console then offers `.2` as a free fixed IP, and a workload pinned to it collides with the interconnection hop. It is fixed in versions after 4.2.3. + +### Cause 5: failed to patch resource ... spec.initContainers[n].lifecycle: Forbidden + +Pods in a few namespaces fail with `FailedCreatePodSandBox ... plugin type="kube-ovn" failed (add): RPC failed; request ip return 500 no address allocated to pod / provider ovn`, and `kube-ovn-controller` may also re-elect its leader. The controller log shows its annotation patch (`ovn.kubernetes.io/allocated`, `ovn.kubernetes.io/ip_address`, `ovn.kubernetes.io/logical_switch` and related keys) rejected: + +```text +failed to patch resource with json merge patch ... Pod "" is invalid: [spec.initContainers[1].lifecycle: Forbidden: may not be set for init containers without restartPolicy=Always, spec.initContainers[1].readinessProbe: Forbidden: ..., spec: Forbidden: pod updates may not change fields other than ...] +``` + +The diff in the message shows `RestartPolicy: &"Always"` changed to `nil` on the Istio native sidecar init container. The affected namespaces ran workloads into which a monitoring operator (`datakit-operator`) injected annotations alongside the Istio sidecar. The spec submitted with the annotation patch had `restartPolicy` cleared from the sidecar init container, so the API server rejected the update and no IP annotation was written. How the injector altered the spec was not determined. Restarting the injecting operator restored allocation. + +### Cause 6: the fixed IP field does not accept a range + +The fixed IP field accepts individual addresses only. Several addresses can be configured, one entry each; range notation is not accepted. This is a product limitation. + +## Resolution + +### Cause 1: on ACP 3.18.2, Subnet spec.cidrBlock holds one IPv4 CIDR + +Leave the existing `cidrBlock` unchanged. Create a second Subnet for the new prefix, bind the same namespaces with `spec.namespaces` (or set `ovn.kubernetes.io/logical_switch` on the workloads that should move), and migrate by recreating those Pods. Use the same create-and-migrate path for a protocol mismatch. + +> **Warning:** Recreating a Pod moves it to the new Subnet and drops its connections. Keep the old Subnet until every migrated workload is Ready. + +```bash +kubectl get subnet -o yaml > /root/subnet.before.yaml +kubectl apply -f - <<'EOF' +apiVersion: kubeovn.io/v1 +kind: Subnet +metadata: + name: +spec: + protocol: IPv4 + cidrBlock: 203.0.113.0/24 + gateway: 203.0.113.1 + excludeIps: + - 203.0.113.1 + namespaces: + - +EOF +``` + +Replace the documentation prefix with the real one. On releases newer than 3.18.2, check whether multiple IPv4 CIDRs are accepted before assuming this limit. + +### Cause 2: AddressOutOfRange in an underlay fixed-IP Deployment + +1. Fix the NIC binding of the underlay Subnet first if it is wrong. +2. Switch the Deployment to `strategy.type: Recreate`, or pin more addresses than `replicas` so a rolling surge has a free one. For a workload that must keep stable addresses across deletes, use a StatefulSet. +3. If an IP object is still owned by a deleted Pod, delete it only after the old Pod is gone and no running Pod uses the address. + +> **Warning:** `Recreate` stops every replica before starting new ones. Deleting an IP object frees the address for anyone; if a live Pod still uses it, you create a duplicate. Save the IP object first. Setting `RollingUpdate` back is the rollback, and brings the race back if the pin list is still as long as `replicas`. + +```bash +kubectl get pod -n -o jsonpath='{.metadata.annotations.ovn\.kubernetes\.io/ip_address}{"\n"}' +kubectl get ip -o wide | grep
+kubectl get ip -o yaml > /root/ip-object.before.yaml +kubectl delete ip +kubectl patch deploy -n --type=merge -p '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}' +``` + +### Cause 3: an external device or a previous platform still uses addresses in the underlay CIDR + +Add the external hosts, the gateway, and every address still used by a previous platform or planned for another purpose to `spec.excludeIps`, then delete the Pods that hold a conflicting address so they are allocated again outside the list. + +> **Warning:** Deleting the Pods restarts them and, if they are not pinned, gives them new addresses. `excludeIps` does not take back an address that is already allocated; the Pod delete does. Save the Subnet and restore `excludeIps` if you excluded too much. + +```bash +kubectl get subnet -o yaml > /root/subnet.before.yaml +kubectl patch subnet --type=json -p='[{"op":"add","path":"/spec/excludeIps/-","value":"198.51.100.12"}]' +kubectl delete pod -n +``` + +Repeat the patch for each address. If `excludeIps` does not exist yet, create the array with a merge patch instead of `add` on `/-`. + +### Cause 4: the u2o interconnection address is offered as a free static IP + +Add the u2o address and the gateway to `excludeIps` so the console cannot offer them. No Pod restart is needed; the exclusion applies to the next allocation. Upgrading past 4.2.3 removes the need for this workaround. + +```bash +kubectl get subnets.kubeovn.io -o jsonpath='{.status.u2oInterconnectionIP}{"\n"}{.spec.excludeIps}{"\n"}' +kubectl patch subnet --type=json -p='[{"op":"add","path":"/spec/excludeIps/-","value":""}]' +``` + +### Cause 5: failed to patch resource ... spec.initContainers[n].lifecycle: Forbidden + +Find the operator that injects into the affected namespaces, restart it, and recreate the failed Pods. Compare a failing Pod with one from a working namespace: look for the injector's annotations and for `restartPolicy: Always` on the sidecar init container. + +> **Warning:** While the injecting operator restarts, Pods created in that window miss its injection. Recreating the failed Pods restarts those workloads. Only when the controller log shows the init-container `Forbidden` rejection. + +```bash +kubectl -n kube-system logs deploy/kube-ovn-controller --tail=2000 | grep -A3 'failed to patch resource' +kubectl -n get pod -o yaml | grep -n -A3 -E 'initContainers|restartPolicy' +kubectl get deploy -A | grep -i datakit +kubectl -n rollout restart deploy/ +kubectl -n delete pod +``` + +### Cause 6: the fixed IP field does not accept a range + +Enter each address of the range as a separate entry, then check the addresses the workload actually carries: + +```bash +kubectl -n get deploy -o jsonpath='{.spec.template.metadata.annotations}{"\n"}' +kubectl -n get pod -o wide +``` + +## Diagnostic Steps + +```bash +kubectl describe pod -n +kubectl get subnet -o yaml +kubectl get ip -o wide +kubectl -n kube-system logs deploy/kube-ovn-controller --tail=500 | grep -E 'AddressOutOfRange|AddressConflict|has been assigned to|failed to patch' +``` + +- `AddressOutOfRange`: the requested address is outside `cidrBlock`, or the NIC binding made the Subnet unusable, or the pin list is as long as `replicas` (Cause 2). A missing second CIDR on 3.18.2 is Cause 1. +- `AddressConflict`: an IP object or a live owner already has the address (Cause 2). If no running Pod holds it, see "Kube-OVN fixed IP rejected or no address allocated because a stale IP object or logical switch port holds it on ACP". +- An external host answering for a Pod address, or `MAC_Binding` conflicts: Cause 3. +- `.2` offered as a fixed IP while `status.u2oInterconnectionIP` is `.2`: Cause 4. +- `failed to patch resource` with init-container `Forbidden` text: Cause 5. diff --git a/docs/en/solutions/Kube-OVN_fixed_IP_NoAvailableAddress_spaces_in_the_list_ip_pool_not_reserved_several_subnets_or_leftover_port_on_ACP.md b/docs/en/solutions/Kube-OVN_fixed_IP_NoAvailableAddress_spaces_in_the_list_ip_pool_not_reserved_several_subnets_or_leftover_port_on_ACP.md new file mode 100644 index 000000000..0579678a6 --- /dev/null +++ b/docs/en/solutions/Kube-OVN_fixed_IP_NoAvailableAddress_spaces_in_the_list_ip_pool_not_reserved_several_subnets_or_leftover_port_on_ACP.md @@ -0,0 +1,127 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.16" + - "4.0" + - "4.2" + - "4.3" +--- + +# Kube-OVN fixed IP NoAvailableAddress: spaces in the list, ip_pool not reserved, several subnets or leftover port on ACP + +## Issue + +A Kube-OVN workload with fixed IPs, or in a namespace bound to several subnets, cannot get an address although free addresses exist. Typical reports: + +- A new fixed-IP Deployment on ACP 3.16.2 fails with `AcquireAddressFailed: NoAvailableAddress` even though the address is free. +- Addresses named in `ovn.kubernetes.io/ip_pool` are taken by other Pods before the fixed-IP workload starts. +- A restarted StatefulSet Pod gets no IP in a namespace bound to two subnets, while the other subnet has plenty of free addresses. +- After a restart, a StatefulSet Pod gets no IP and the controller log says `port already exists but in switch `. +- A Pod was given an address from the subnet's reserved range, and another workload's fixed-IP Pod then failed. + +Related articles: "Kube-OVN fixed IP occupied or not released: held by another Pod, node down, duplicate entry or RollingUpdate on ACP". + +## Root Cause + +### Cause 1: spaces in the fixed IP list + +On ACP 3.16.2 with Kube-OVN 1.12.35, fixed-IP parsing is not consistent across the create and update paths. A list with spaces after the commas (`192.0.2.10, 192.0.2.11`) fails on some paths with `AcquireAddressFailed: NoAvailableAddress` and works on others. The address is inside the subnet, not in `excludeIps`, and no Pod uses it. Restarting `kube-ovn-controller`, waiting, or restarting the workload several times can clear the error temporarily. + +### Cause 2: ovn.kubernetes.io/ip_pool does not reserve addresses that no Pod is using + +An address listed in `ovn.kubernetes.io/ip_pool` and not currently held by a Pod is still free for dynamic allocation, so another Pod can take it first. Listing those addresses in the Subnet's `spec.excludeIps` stops dynamic allocation from using them, while the fixed-IP mechanism can still assign them. + +### Cause 3: a namespace bound to several overlay subnets + +A Kube-OVN overlay namespace can be bound to more than one Subnet. A Pod that does not select a Subnet may get an address from any of them; `ovn.kubernetes.io/logical_switch` selects exactly one. The CIDRs must not overlap. + +On ACP 3.16.2 (Kube-OVN v1.12.35) a confirmed defect makes a restarted StatefulSet Pod fail to get an IP when all three conditions hold: the namespace is bound to two or more subnets, the Pod template does not name a subnet, and the Pod's original subnet has no free address at that moment. The Pod does not fall back to the other subnet. The event is `AcquireAddressFailed` `NoAvailableAddress`, and the controller log repeats `allocate v4 , v6 , mac for /-0 from subnet ` followed by `error syncing '/-0': NoAvailableAddress, requeuing`. + +### Cause 4: port already exists but in switch + +A StatefulSet Pod keeps its name across restarts. In a namespace bound to several subnets, the first chosen subnet had no free address, and OVN had not removed the old Pod's logical switch port in the other subnet, so the new port with the same name was refused. The controller log shows an allocation from one subnet followed by `port already exists but in switch `. This is an OVN cleanup issue and is rarely seen. + +### Cause 5: a Pod received an address from the subnet's reserved range + +After an image update a workload restarted and was given an address inside the reserved range of its subnet, which was another workload's fixed IP, so the fixed-IP Pod could not start. This is an IPAM defect in the Kube-OVN component of that release, fixed in ACP 3.18 and later. + +## Resolution + +### Cause 1: spaces in the fixed IP list + +Write the list comma-separated with no blanks (`192.0.2.10,192.0.2.11`). Restart `kube-ovn-controller` only as a temporary recovery when the addresses are free and the list still contains spaces. + +> **Warning:** Restarting `kube-ovn-controller` pauses address allocation in the cluster until the new Pod is Ready. It does not rewrite the annotation, so the next create can fail again while spaces remain. Save the Deployment first. + +```bash +kubectl get deploy -n -o yaml > /root/deploy-ip.before.yaml +kubectl edit deploy -n +kubectl -n kube-system rollout restart deploy/kube-ovn-controller +``` + +### Cause 2: ovn.kubernetes.io/ip_pool does not reserve addresses that no Pod is using + +Add each pool address to the Subnet's `spec.excludeIps` before dynamic Pods can take it. + +> **Warning:** `excludeIps` changes which addresses dynamic Pods may receive; an address already allocated is not taken back. Save the Subnet; restoring `excludeIps` is the rollback. + +```bash +kubectl get subnet -o yaml > /root/subnet-reserve.before.yaml +kubectl patch subnet --type=json -p='[{"op":"add","path":"/spec/excludeIps/-","value":"192.0.2.10"}]' +``` + +If `excludeIps` does not exist yet, create the array with a merge patch instead of `add` on `/-`. + +### Cause 3: a namespace bound to several overlay subnets + +Pin workloads that must stay on one prefix, and every StatefulSet, to one Subnet with `ovn.kubernetes.io/logical_switch` in the Pod template (or give them fixed IPs). If a Pod is already stuck, free addresses in its original subnet, or restart `kube-ovn-controller` as a temporary fix. + +> **Warning:** Changing the StatefulSet Pod template restarts its Pods one ordinal at a time. Restarting `kube-ovn-controller` pauses allocation cluster-wide. Removing the annotation is the rollback and brings the defect back. + +```bash +kubectl get subnet -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidrBlock,NS:.spec.namespaces,AVAIL:.status.v4availableIPs' +kubectl patch sts -n --type=merge -p '{"spec":{"template":{"metadata":{"annotations":{"ovn.kubernetes.io/logical_switch":""}}}}}' +``` + +### Cause 4: port already exists but in switch + +Restart the StatefulSet Pod. If the error returns, delete only the leftover logical switch port named after the Pod, then restart the Pod. Pinning the StatefulSet to one subnet (Cause 3) prevents it. + +> **Warning:** `lsp-del` removes the port from the OVN northbound database; deleting a port of a running Pod cuts its network until it is recreated. Only when the Pod named in the port is not running. Record the `list` output first. + +```bash +kubectl ko nbctl list logical_switch_port | grep -B5 -A5 +kubectl ko nbctl lsp-del +kubectl -n delete pod +``` + +### Cause 5: a Pod received an address from the subnet's reserved range + +Upgrade to ACP 3.18 or later, which contains the fix. To unblock the fixed-IP Pod before that, recreate the Pod that took the reserved address so it is allocated again; on an affected version the wrong allocation can recur. Collect the `ovs-ovn` logs from all nodes and `kubectl get pod -A -o wide` first if the allocation needs to be traced. + +> **Warning:** Deleting the Pod that holds the address interrupts that workload and gives it a new address. Confirm it is not the workload that is supposed to own the address. + +```bash +kubectl get subnet -o jsonpath='{.spec.excludeIps}{"\n"}' +kubectl get pod -A -o wide | grep +kubectl -n delete pod +``` + +## Diagnostic Steps + +```bash +kubectl describe pod -n +kubectl get subnet -o yaml +kubectl get ns -o yaml | grep -i ovn +kubectl -n kube-system logs deploy/kube-ovn-controller --tail=2000 | grep -E 'NoAvailableAddress|port already exists|allocate v4' +``` + +- `NoAvailableAddress` with spaces in the fixed-IP list: Cause 1. +- An `ip_pool` address held by a Pod of another workload that never had it pinned: Cause 2. +- A StatefulSet in a multi-subnet namespace failing with `NoAvailableAddress` only when its original subnet is full: Cause 3. +- `port already exists but in switch`: Cause 4. +- An address inside `excludeIps` assigned to a dynamic Pod: Cause 5. diff --git a/docs/en/solutions/Kube-OVN_fixed_IP_occupied_or_not_released_held_by_another_Pod_node_down_duplicate_entry_or_RollingUpdate_on_ACP.md b/docs/en/solutions/Kube-OVN_fixed_IP_occupied_or_not_released_held_by_another_Pod_node_down_duplicate_entry_or_RollingUpdate_on_ACP.md new file mode 100644 index 000000000..f83cdd885 --- /dev/null +++ b/docs/en/solutions/Kube-OVN_fixed_IP_occupied_or_not_released_held_by_another_Pod_node_down_duplicate_entry_or_RollingUpdate_on_ACP.md @@ -0,0 +1,150 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.16" + - "4.0" + - "4.2" + - "4.3" +--- + +# Kube-OVN fixed IP occupied or not released: held by another Pod, node down, duplicate entry or RollingUpdate on ACP + +## Issue + +A workload with a fixed IP cannot start because its address is taken. Typical reports: + +- The console says the IP is in use and cannot be released, or the event is `FailedCreatePodSandBox ... plugin type="kube-ovn" failed (add): RPC failed; request ip return 500 no address allocated to pod / provider ovn`. +- A fixed IP looks stuck and is reassigned only after about 40 seconds. +- After a node dropped off the network, the replacement Pod cannot take the same fixed address. +- Two applications in one namespace list the same IP. +- A new Pod fails with `AddressOutOfRange` and its annotations show the same address twice. +- A pipeline release fails because the new Pod wants an address the old Pod still holds, or a third Pod appears that never starts after a namespace quota was raised. +- On a Calico cluster, a second replica reports `failed to allocate ... from IPAM: resource already exists` while the IP object's `containerID`, `nodeName` and `podName` are empty. + +Related articles: "Kube-OVN fixed IP NoAvailableAddress: spaces in the list, ip_pool not reserved, several subnets or leftover port on ACP"; "Kube-OVN fixed IP rejected or no address allocated because a stale IP object or logical switch port holds it on ACP" (when no running Pod holds the address). + +## Root Cause + +### Cause 1: another running application holds the address + +`kubectl get ip` and `kubectl get pods -A -o wide | grep ` show a live owner: a Pod of another application, possibly in another namespace and running for days, already uses the address pinned with `ovn.kubernetes.io/ip_pool` or `ovn.kubernetes.io/ip_address`. The address is not leaked; it is in use. + +A fixed IP can also only look stuck. The `kube-ovn-controller` log shows the new Pod failing for a while with `static address for / has been assigned to /` and `acquire address from ippool ... failed`, then `release v4 from subnet for /, add ip to released list`, and later `allocate v4 ... for /`. The new Pod gets the same IP and MAC about 40 seconds after the first failure. + +### Cause 2: the node is down and the terminating Pod still holds the fixed IP + +When the node running a fixed-IP Pod loses contact, the Pod stays `Terminating` and is not deleted, so its fixed-IP lease is not released. The replacement on another node fails network setup with an address conflict. + +### Cause 3: two applications in the same namespace are configured with the same IP + +Both workloads pin the same address, so whichever starts second fails. + +### Cause 4: Calico IP object with empty containerID, nodeName and podName + +On a Calico cluster, a second replica fails at the CNI with `failed to allocate ... from IPAM: resource already exists`. `kubectl get ip
` shows empty `containerID`, `nodeName` and `podName` while `workload` still names the application, and a live Pod already uses the address. The `raven` logs in `kube-system` show nothing. The platform IP status and Calico IPAM are out of step. + +### Cause 5: AddressOutOfRange after a duplicate IP assignment + +A new Pod fails with `AddressOutOfRange`, and its annotations show the same address assigned twice. Removing the duplicate assignment lets the Pod be created. + +### Cause 6: RollingUpdate creates the new Pod while the fixed IP is still taken + +With `strategy.type: RollingUpdate`, the new Pod is created before the old one is deleted, so the fixed address is still in use; a single-instance service with one fixed IP fails the same way on every update. A stalled rollout causes the same shortage later: a Deployment with `replicas: 2`, the default `maxSurge: 25%` and two addresses in `ovn.kubernetes.io/ip_pool` was restarted while the namespace quota was full, so the surge Pod could not be created and the rollout never finished. Deleting Pods by hand does not cancel the rollout. When the quota was raised, the rollout resumed and created a third Pod (`deployment.kubernetes.io/max-replicas: "3"` on the ReplicaSet) that could never start because both pinned addresses were in use. + +## Resolution + +### Cause 1: another running application holds the address + +Stop the application that holds the address, or change this workload's fixed IP to a free address. When the log shows the release-and-reallocate sequence above, wait for the reallocation before treating the address as leaked. + +> **Warning:** Changing `ovn.kubernetes.io/ip_pool` in the Pod template rolls the Deployment onto a different address; anything that reaches it by the old address breaks. Pick an address inside the namespace's subnet, outside `excludeIps`, with no IP object. Save the Deployment; restoring it is the rollback. + +```bash +kubectl get pods -A -o wide | grep +kubectl get ip -o wide | grep +kubectl get deploy -n -o yaml > /root/deploy-.before.yaml +kubectl -n patch deploy --type=merge -p '{"spec":{"template":{"metadata":{"annotations":{"ovn.kubernetes.io/ip_pool":""}}}}}' +kubectl -n kube-system logs deploy/kube-ovn-controller --tail=2000 | grep | grep -E 'has been assigned to|acquire address|release v4|allocate v4' +``` + +### Cause 2: the node is down and the terminating Pod still holds the fixed IP + +Force-delete the stuck Pod so the lease is released. For the longer term, pin more addresses than replicas and spread the replicas with Pod anti-affinity, so one node loss cannot use up the pins. + +> **Warning:** A force delete skips the grace period and `preStop`. Only when the node is confirmed down and unreachable; otherwise the old process may still use the address and you create a duplicate. The delete cannot be undone. + +```bash +kubectl get node +kubectl get pod -n -o wide +kubectl delete pod -n --force --grace-period=0 +``` + +### Cause 3: two applications in the same namespace are configured with the same IP + +Change one of the two settings so they no longer overlap, then restart both applications. + +```bash +kubectl get deploy,sts -n -o yaml | grep -n -E 'ip_address|ip_pool' +``` + +### Cause 4: Calico IP object with empty containerID, nodeName and podName + +Restart the `raven` Pods in `kube-system`, then restart the application. + +> **Warning:** Restarting `raven` pauses IP status updates until the new Pod is Ready, and restarting the application drops its connections. Only when the IP object has the empty fields, a live Pod uses the address, and the CNI says `resource already exists`. + +```bash +kubectl get ip
-o yaml +kubectl -n kube-system get pod -o wide | grep raven +kubectl -n kube-system delete pod +kubectl -n rollout restart deploy/ +``` + +### Cause 5: AddressOutOfRange after a duplicate IP assignment + +Inspect the annotations, keep one assignment of the duplicated address, and recreate the Pod. + +```bash +kubectl get pod -n -o jsonpath='{.metadata.annotations}{"\n"}' +``` + +### Cause 6: RollingUpdate creates the new Pod while the fixed IP is still taken + +Switch the workload to `Recreate`. If the service does not need a fixed IP, removing the fixed-IP setting also works. + +> **Warning:** `Recreate` stops the current replicas before starting new ones, so requests fail during the gap. Setting `RollingUpdate` back is the rollback and brings the failure back while the pin list is only as long as `replicas`. + +```bash +kubectl patch deploy -n --type=merge -p '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}' +``` + +For a stalled rollout that left an extra Pod, scale the Deployment to `0` and back to clear the rollout state. To prevent it, pin at least `replicas + maxSurge` addresses (three for two replicas at `25%`), or set `maxSurge: 0` and accept one fewer replica during updates. + +> **Warning:** Scaling to `0` stops every replica until the new Pods are Ready. Run it in a window. + +```bash +kubectl -n get rs -o yaml | grep -E 'desired-replicas|max-replicas' +kubectl -n scale deploy --replicas=0 +kubectl -n scale deploy --replicas= +kubectl -n patch deploy --type=merge -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":0,"maxUnavailable":1}}}}' +``` + +## Diagnostic Steps + +```bash +kubectl get ip -o wide +kubectl get pod -A -o wide | grep
+kubectl describe pod -n +kubectl get events -n --field-selector involvedObject.name= +``` + +- A running Pod of another workload on the address, or `no address allocated to pod` with the pinned address owned elsewhere: Cause 1. A release-then-reallocate sequence in the controller log: Cause 1, wait. +- The previous owner is `Terminating` on a `NotReady` node: Cause 2. +- Two workloads in the namespace with the same address in their annotations: Cause 3. +- A Calico IP object with empty `containerID`, `nodeName` and `podName` plus `resource already exists`: Cause 4. +- `AddressOutOfRange` with a duplicated address in the annotations: Cause 5. +- A failure during an update with `RollingUpdate`, or an extra Pod after a quota change: Cause 6. diff --git a/docs/en/solutions/Kube-OVN_subnet_planning_CIDR_changes_overlaps_subnet_type_per-namespace_IPPool_and_an_underlay_global_cluster_on_ACP.md b/docs/en/solutions/Kube-OVN_subnet_planning_CIDR_changes_overlaps_subnet_type_per-namespace_IPPool_and_an_underlay_global_cluster_on_ACP.md new file mode 100644 index 000000000..d2b333a06 --- /dev/null +++ b/docs/en/solutions/Kube-OVN_subnet_planning_CIDR_changes_overlaps_subnet_type_per-namespace_IPPool_and_an_underlay_global_cluster_on_ACP.md @@ -0,0 +1,169 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.2" + - "4.3" +--- + +# Kube-OVN subnet planning: CIDR changes, overlaps, subnet type, per-namespace IPPool and an underlay global cluster on ACP + +## Issue + +Planning questions and errors around Kube-OVN subnets: + +- The Service address pool is used up and no Service can be created, or someone wants to change the Pod, Service or join CIDR of a running cluster, or move a cluster from Calico to Kube-OVN, and asks whether a redeploy is needed. The join CIDR given at global cluster deployment was wrong. +- Creating a new subnet fails with a CIDR conflict. +- Overlay subnets were carved from the same block as an underlay subnet, and the question is whether that is safe. +- How to tell underlay subnets from overlay subnets in `kubectl get subnet`, and why the used-IP list of an underlay subnet is not sorted. +- A checklist inspection says the join address shown for the cluster in the console does not match the join address used in the backend. +- Someone wants to give one IP range of a subnet to project A and another range to project B. +- Someone asks whether the global cluster can be deployed with Kube-OVN underlay instead of overlay. +- Someone wants a `kubectl` command that prints the cluster's subnet CIDRs. + +Related articles: "Kube-OVN subnet stuck Terminating or Pods use the wrong subnet: finalizers, namespace binding or uppercase IPv6 on ACP". + +## Root Cause + +### Cause 1: there is no supported procedure to change the Pod, Service or join CIDR after deployment + +There is no ready-made, supported procedure to change the Pod subnet, the Service subnet or the join subnet of a deployed cluster (confirmed for ACP 4.3.1 global and business clusters). A change on a running cluster needs a case-by-case evaluation by the vendor's R&D, or the cluster is redeployed with the new ranges. A wrong global join CIDR on ACP 4.2.0 was handled by redeploying. There is likewise no verified method to enlarge or add a Service CIDR. For a Calico-to-Kube-OVN move, "How to Migrate CNI from Calico to Kube-OVN on ACP" covers ACP 3.18 business clusters only, not the global cluster. An overlay range causes no problem as long as it does not conflict with the production network or with the cluster CIDRs. + +### Cause 2: a new subnet's CIDR overlaps the join subnet + +A new subnet must avoid the CIDRs of all existing subnets, including `join`, which is easy to forget because no workload uses it directly. A CIDR inside the `join` range is rejected with a conflict error. + +### Cause 3: overlay subnets taken from the same block as an underlay subnet + +Splitting one block into disjoint pieces (for example two `/24` blocks for the default and `join` overlay subnets and the remaining `/23` as an underlay subnet) does not by itself break the cluster, because the ranges do not overlap. The general recommendation still stands: keep overlay ranges out of physical-network address space, because an overlapping or misconfigured subnet can confuse routing, for example by sending traffic meant for Pods to physical nodes. + +### Cause 4: an underlay subnet is the one with a VLAN + +A subnet that references a `vlan` is an underlay subnet; a subnet without one is an overlay subnet. The console lists the used addresses of an underlay subnet in no particular order; that is the current design. + +### Cause 5: the checklist join CIDR mismatch comes from the cluster's kube-ovn.cpaas.io/join-cidr annotation + +The console reads the join CIDR from the annotation `kube-ovn.cpaas.io/join-cidr` on the `Cluster` resource (`kubectl get cls `, `platform.tkestack.io/v1`, next to `cpaas.io/network-type: kube-ovn` and `kube-ovn.cpaas.io/transmit-type: overlay`). When that annotation does not hold the join CIDR the cluster really uses, the checklist reports a mismatch. + +### Cause 6: a subnet cannot be divided between projects; an IPPool binds ranges to namespaces + +The platform does not support dividing one subnet between projects. The Kube-OVN `IPPool` (`kubeovn.io/v1`) binds ranges to namespaces instead: + +- `spec.subnet` names the owning subnet; `spec.ips` accepts ``, `` and `..`, IPv4 and IPv6. +- `spec.namespaces` optionally binds namespaces; Pods in a bound namespace get addresses only from the bound pool. Several pools can be bound to one namespace. +- Pools in one subnet must not overlap. Random allocation from the subnet excludes all pool ranges. `spec.ips` can be updated at any time and takes effect immediately. + +### Cause 7: the global cluster cannot be deployed with Kube-OVN underlay + +The global cluster deployment page of the installer (port `8080`) offers only the Kube-OVN overlay configuration, and no parameter switches the global cluster to underlay at deployment time. + +### Cause 8: subnet CIDRs can be read with kubectl + +On Calico the CIDRs are in the Calico `IPPool` objects (`crd.projectcalico.org/v1`, `spec.cidr`). On Kube-OVN, `kubectl get subnet` prints a CIDR column for every subnet together with provider, VPC, protocol, NAT, default, gateway type, used and available counts, and excluded IPs. + +## Resolution + +### Cause 1: there is no supported procedure to change the Pod, Service or join CIDR after deployment + +Plan the Pod, Service and join ranges before deployment. For a running cluster, request an evaluation for that cluster or redeploy with the new ranges. For an exhausted Service CIDR, free addresses by deleting Services that are no longer used. + +> **Warning:** Deleting a Service removes its ClusterIP and DNS name immediately. Confirm with the owner that it is unused (no endpoints is a hint, not proof) and export it first; re-creating it from the export restores the object, possibly with a different ClusterIP. + +```bash +kubectl get svc -A --no-headers | wc -l +kubectl get endpoints -A | awk '$3==""' +kubectl -n get svc -o yaml > /root/svc-.yaml +kubectl -n delete svc +``` + +### Cause 2: a new subnet's CIDR overlaps the join subnet + +List all subnet CIDRs, including `join`, and choose a range that overlaps none of them: + +```bash +kubectl get subnet -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidrBlock,VLAN:.spec.vlan' +``` + +### Cause 3: overlay subnets taken from the same block as an underlay subnet + +Check that no two subnets overlap and that no overlay range is used on the physical network; if so, no change is needed. For new plans, use ranges that are not routed on the physical network for overlay subnets. + +### Cause 4: an underlay subnet is the one with a VLAN + +```bash +kubectl get subnet -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidrBlock,VLAN:.spec.vlan,PROVIDER:.spec.provider' +kubectl get vlan +``` + +An empty VLAN column means overlay. To read the used addresses of a subnet in order: + +```bash +kubectl get ip -o custom-columns='IP:.spec.ipAddress,SUBNET:.spec.subnet,POD:.spec.podName' | grep | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n +``` + +### Cause 5: the checklist join CIDR mismatch comes from the cluster's kube-ovn.cpaas.io/join-cidr annotation + +Read the real join CIDR on the business cluster, compare it with the annotation on the global cluster, and set the annotation to the real value. Then rerun the checklist. + +> **Warning:** The annotation is platform metadata; a wrong value misleads the console and the checklist. Record the current value; annotating it back is the rollback. + +```bash +# on the business cluster +kubectl get subnet join -o jsonpath='{.spec.cidrBlock}{"\n"}' +# where the cluster resource lives +kubectl get cls -o jsonpath='{.metadata.annotations.kube-ovn\.cpaas\.io/join-cidr}{"\n"}' +kubectl annotate cls kube-ovn.cpaas.io/join-cidr= --overwrite +``` + +### Cause 6: a subnet cannot be divided between projects; an IPPool binds ranges to namespaces + +Use one `IPPool` per namespace when a range must be reserved: + +```yaml +apiVersion: kubeovn.io/v1 +kind: IPPool +metadata: + name: +spec: + subnet: + ips: + - 198.51.100.50..198.51.100.150 + namespaces: + - +``` + +Use your own addresses; pools in one subnet must not overlap. + +### Cause 7: the global cluster cannot be deployed with Kube-OVN underlay + +Deploy the global cluster with the overlay configuration the page offers. + +### Cause 8: subnet CIDRs can be read with kubectl + +```bash +# Calico cluster +kubectl get ippools.crd.projectcalico.org -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidr,BLOCK:.spec.blockSize,IPIP:.spec.ipipMode,VXLAN:.spec.vxlanMode' +# Kube-OVN cluster +kubectl get subnet +``` + +The fully qualified name avoids picking up Kube-OVN's own `ippools.kubeovn.io` on a cluster that has both CRDs. + +## Diagnostic Steps + +Start from the subnet list; it answers the type question, shows overlaps and shows bound namespaces: + +```bash +kubectl get subnet -o wide +kubectl get subnet -o yaml +kubectl get ippool +``` + +- A question about changing the Pod, Service or join range, or a full Service CIDR: Cause 1. +- A creation error naming a CIDR conflict: Cause 2. Disjoint ranges cut from one block: Cause 3. +- The VLAN column decides underlay or overlay: Cause 4. +- A checklist join-CIDR mismatch: Cause 5. +- A request to divide a subnet between projects: Cause 6. An underlay global cluster: Cause 7. A command for the CIDRs: Cause 8. diff --git a/docs/en/solutions/Kube-OVN_subnet_stuck_Terminating_or_Pods_use_the_wrong_subnet_finalizers_namespace_binding_or_uppercase_IPv6_on_ACP.md b/docs/en/solutions/Kube-OVN_subnet_stuck_Terminating_or_Pods_use_the_wrong_subnet_finalizers_namespace_binding_or_uppercase_IPv6_on_ACP.md new file mode 100644 index 000000000..1bc82e5e7 --- /dev/null +++ b/docs/en/solutions/Kube-OVN_subnet_stuck_Terminating_or_Pods_use_the_wrong_subnet_finalizers_namespace_binding_or_uppercase_IPv6_on_ACP.md @@ -0,0 +1,124 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.2" + - "4.3" +--- + +# Kube-OVN subnet stuck Terminating or Pods use the wrong subnet: finalizers, namespace binding or uppercase IPv6 on ACP + +## Issue + +- A subnet stays in deleting (`Terminating`) forever, and `kubectl describe subnet` still lists `Finalizers: kubeovn.io/kube-ovn-controller`. +- After namespaces were bound to `ovn-default`, Pods in other namespaces stop getting addresses; or Pod restarts in a namespace fail with `AddressOutOfRange` because the namespace was never bound to the subnet it should use. +- The MetalLB address pool and BGP peer pages say MetalLB is not installed although it was never uninstalled, containers cannot resolve the registry domain, and the default subnet `ovn-default` is stuck in deleting. +- A namespace was moved from an overlay to an underlay subnet, and its Pods (and the fixed-IP picker) still use the overlay range. + +Related articles: "Kube-OVN subnet planning: CIDR changes, overlaps, subnet type, per-namespace IPPool and an underlay global cluster on ACP". + +## Root Cause + +### Cause 1: the subnet is stuck in Terminating because its finalizer was never removed + +Kubernetes removes a deleted object only after every entry in `metadata.finalizers` is gone. `kube-ovn-controller` did not complete its cleanup, so `kubeovn.io/kube-ovn-controller` stayed and the subnet never disappeared. `kubectl describe subnet` shows `Deletion Timestamp` set, `Finalizers: kubeovn.io/kube-ovn-controller`, and conditions `Validated` and `Ready` still `True`. + +### Cause 2: namespace binding decides which subnet a Pod uses + +Once namespaces are listed on `ovn-default`, its addresses (other than reserved IPs) are allocated only to Pods of those namespaces, so Pods elsewhere cannot get an address from it. Keep `ovn-default` unbound and give business workloads their own subnets; assigning a business subnet to the project when it is created is the way to keep business workloads off `ovn-default`. + +In the other direction, a namespace that was never bound to its intended subnet keeps getting addresses from the default subnet: `kubectl describe ns` shows `ovn.kubernetes.io/logical_switch: ovn-default` and the `ovn-default` CIDR in `ovn.kubernetes.io/cidr`, and restarts of Pods with addresses from the intended subnet fail with `AddressOutOfRange`. + +### Cause 3: ovn-default was deleted and is stuck in deleting + +Someone tried to delete `ovn-default` (and another subnet), and the deletion never completed. A half-deleted default subnet breaks the cluster: every `ars` is abnormal, containers cannot resolve the private registry domain, and the MetalLB pages report that MetalLB is not installed. Without an etcd backup, rolling back is not possible. Stopping `kube-ovn-controller`, removing the finalizer so the deletion completes, and starting the controller again brings `ovn-default` back as the default subnet with its original CIDR; other deleted subnets do not come back. Leftover interfaces from a previously tested CNI (for example `cilium_host@cilium_net`) on a node are a separate cleanup. + +### Cause 4: v6 ip address can not contain upper case, so the subnet fails validation + +A namespace was assigned to an underlay dual-stack subnet, but restarted Pods still received overlay addresses and the Pod events were clean. The `kube-ovn-controller` log shows: + +```text +subnet U2O interconnection ip , v6 ip address can not contain upper case +failed to validate subnet +``` + +and the Subnet has a `Warning` event `ValidateLogicalSwitchFailed`. The subnet failed validation and was not usable, so the namespace kept using the overlay subnet. + +## Resolution + +### Cause 1: the subnet is stuck in Terminating because its finalizer was never removed + +Remove the finalizers of a subnet that is really meant to be deleted, after its Pods and IP objects are gone. + +> **Warning:** Removing finalizers skips the controller's cleanup and is irreversible: the object is deleted as soon as the list is empty. Keep the YAML so the subnet can be recreated if needed. + +```bash +kubectl get subnet -o yaml > /root/subnet-.before.yaml +kubectl get subnet -o jsonpath='{.metadata.deletionTimestamp}{"\n"}{.metadata.finalizers}{"\n"}' +kubectl get ip | grep || true +kubectl patch subnet --type=merge -p '{"metadata":{"finalizers":null}}' +``` + +### Cause 2: namespace binding decides which subnet a Pod uses + +Leave `ovn-default` unbound. Bind each business namespace to its own subnet; add a namespace that is missing from its intended subnet. + +> **Warning:** Changing `spec.namespaces` changes where new Pods of those namespaces get addresses; running Pods keep theirs until recreated. Save the subnet first; restoring `spec.namespaces` is the rollback. + +```bash +kubectl get subnet -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidrBlock,NAMESPACES:.spec.namespaces' +kubectl describe ns | grep ovn.kubernetes.io +kubectl get subnet -o yaml > /root/subnet-.before.yaml +kubectl patch subnet --type=json -p='[{"op":"add","path":"/spec/namespaces/-","value":""}]' +``` + +If `spec.namespaces` does not exist yet, use a merge patch that sets the whole list. + +### Cause 3: ovn-default was deleted and is stuck in deleting + +Only when `ovn-default` itself has a `deletionTimestamp` and there is no etcd backup to roll back to: let the deletion complete while `kube-ovn-controller` is stopped, then start the controller so it recreates the default subnet. + +> **Warning:** While `kube-ovn-controller` is at `0` replicas, no Pod in the cluster gets an address. Removing the finalizers is irreversible. Work in a maintenance window from a session that does not depend on Pod networking, and save the Subnet YAML and the replica count. If the subnet does not come back, recreate it from the saved YAML after removing `status`, `metadata.uid`, `metadata.resourceVersion`, `metadata.deletionTimestamp` and `metadata.finalizers`. + +```bash +kubectl get subnet ovn-default -o yaml > /root/subnet-ovn-default.before.yaml +kubectl -n kube-system get deploy kube-ovn-controller -o jsonpath='{.spec.replicas}{"\n"}' +kubectl -n kube-system scale deploy kube-ovn-controller --replicas=0 +kubectl patch subnet ovn-default --type=merge -p '{"metadata":{"finalizers":null}}' +kubectl get subnet ovn-default -w +kubectl -n kube-system scale deploy kube-ovn-controller --replicas= +kubectl get subnet ovn-default -o jsonpath='{.spec.cidrBlock} {.spec.default}{"\n"}' +``` + +Stop the watch when the subnet is gone. Treat other subnets stuck in deleting the same way only if they are meant to be deleted. Check nodes for leftovers of other CNIs: `ip -br link | grep -i cilium`. + +### Cause 4: v6 ip address can not contain upper case, so the subnet fails validation + +Rewrite the IPv6 address the controller rejects in lowercase on the Subnet, then recreate the Pods that received overlay addresses. + +> **Warning:** Editing the Subnet affects allocation for every bound namespace. Change only the letter case of the address named in the log. Save the Subnet first; restoring it is the rollback. + +```bash +kubectl -n kube-system logs deploy/kube-ovn-controller --tail=2000 | grep -E 'can not contain upper case|failed to validate subnet' +kubectl get subnet -o yaml > /root/subnet-.before-lowercase.yaml +kubectl edit subnet +kubectl -n delete pod +``` + +In a mixed overlay and underlay deployment where an overlay subnet's IPv6 address also contains uppercase letters, leave that one unchanged as long as Kube-OVN reports no errors and it works, because changing it involves Service CIDR changes. + +## Diagnostic Steps + +```bash +kubectl get subnet +kubectl get subnet -o jsonpath='{.metadata.deletionTimestamp}{"\n"}{.metadata.finalizers}{"\n"}' +kubectl get ns -o yaml | grep -i ovn +kubectl get events -A --field-selector reason=ValidateLogicalSwitchFailed +``` + +- A subnet with a `deletionTimestamp` and `kubeovn.io/kube-ovn-controller` in `finalizers`: Cause 1 (Cause 3 if it is `ovn-default`, together with broken MetalLB pages and registry resolution). +- `AddressOutOfRange` on restart with the namespace annotation still `ovn.kubernetes.io/logical_switch: ovn-default` and the namespace absent from the intended subnet, or Pods elsewhere failing after namespaces were bound to `ovn-default`: Cause 2. +- `can not contain upper case` or `ValidateLogicalSwitchFailed`: Cause 4. diff --git a/docs/en/solutions/Kube-OVN_underlay_multicast_not_delivered_DDS_IGMP_membership_limit_or_a_sender_that_never_joins_the_group_on_ACP.md b/docs/en/solutions/Kube-OVN_underlay_multicast_not_delivered_DDS_IGMP_membership_limit_or_a_sender_that_never_joins_the_group_on_ACP.md new file mode 100644 index 000000000..d8f4553cf --- /dev/null +++ b/docs/en/solutions/Kube-OVN_underlay_multicast_not_delivered_DDS_IGMP_membership_limit_or_a_sender_that_never_joins_the_group_on_ACP.md @@ -0,0 +1,113 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Kube-OVN underlay multicast not delivered: DDS, IGMP membership limit or a sender that never joins the group on ACP + +## Issue + +Pods on a Kube-OVN underlay subnet send or receive multicast, and the data does not arrive. Unicast between the same endpoints works. Typical reports: + +- DDS traffic from one underlay Pod to another is missing or arrives only after several minutes, on the same node or across nodes. The same receiver accepts DDS immediately when the sender is a physical host using that sender's address. +- An application cannot join one more multicast group, and its multicast send or receive breaks; joining works again after another group is dropped. +- A Pod sends to a multicast group. The packets can be captured on the host's physical NIC but not on the switch, and receivers that joined the group get nothing. `ping -c 3 ` from the Pod shows `100% packet loss`. + +## Root Cause + +### Cause 1: underlay DDS depends on platform multicast support that is enabled separately + +On ACP 4.2 with business Pods on a Kube-OVN underlay network, DDS can fail or arrive late between Pods while unicast is normal, across several applications, while a physical sender on the same address works. This is the platform multicast path, not IPAM and not a single application's socket. Enabling multicast for this scenario is a separate request to the platform vendor; there is no in-cluster switch for it. + +### Cause 2: net.ipv4.igmp_max_memberships is 20, so a new group cannot be joined + +`net.ipv4.igmp_max_memberships` defaults to `20` and applies per network namespace. Once the Pod has joined 20 groups, the next `setsockopt` join fails and the application's multicast traffic for that group breaks. Leaving one group allows the new join; raising the sysctl keeps the cap from being hit again. + +### Cause 3: the sending application never joins the target group + +The application sends UDP to the target group but never joins it with `setsockopt(IP_ADD_MEMBERSHIP)`. The kernel therefore sends no IGMP report for that group, and OVN records no `IGMP_Group` entry for it. With IGMP snooping, OVN forwards ordinary multicast only to ports that have joined the group, so the packets match no forwarding flow and are dropped before they reach the bridge port and the switch. A group the Pod did join (for example the group it receives from) works, because it has an `IGMP_Group` entry. + +The evidence on the Pod and the node: + +- `ip maddr` in the container lists the joined groups on `eth0` and has no entry for the target group. +- `tcpdump -i eth0 -nnve igmp` on the node shows the gateway's `igmp query v3`, and the Pod's port answers with `igmp v3 report, 1 group record(s) [gaddr is_ex, 0 source(s)]` only for the joined group, never for the target group. +- The OVN `IGMP_Group` table has an entry for the joined group and none for the target group. + +This is an application-side problem; the network configuration is working as designed. + +## Resolution + +### Cause 1: underlay DDS depends on platform multicast support that is enabled separately + +Rule out Cause 3 first. If unicast works, the application joins its groups, and DDS still fails between Pods while a physical sender on the same address succeeds, request multicast enablement for the cluster. Locate where the packets stop and attach the result: + +```bash +kubectl exec -n -- ip maddr +kubectl exec -n -- ip route +tcpdump -ni -c 20 host +``` + +On the receiver node, a packet on the Pod veth means the CNI delivered it. A packet on the physical NIC and none on the veth means the underlay path did not forward the group. A packet on the veth and no group membership in the receiver means the application did not join. + +### Cause 2: net.ipv4.igmp_max_memberships is 20, so a new group cannot be joined + +Only when a new join fails and the sysctl in the Pod prints `20`: leave a group the application owner confirms is no longer needed, or raise the limit. + +> **Warning:** A higher limit uses more kernel memory for membership state in that network namespace. A value written with `sysctl -w` in a running Pod is lost when the Pod is recreated. Writing back the previous value (normally `20`) is the rollback. Dropping a group stops reception for it immediately; rejoining reverses it. + +```bash +kubectl exec -n -- sysctl net.ipv4.igmp_max_memberships +kubectl exec -n -- sysctl -w net.ipv4.igmp_max_memberships=64 +kubectl ko sbctl list Multicast_Group +``` + +If `kubectl ko` is not available on the operator node, take the `kubectl-ko` helper from the `kube-ovn-pinger` image the cluster already runs. To keep a higher limit across restarts, set the sysctl in the Pod spec only if the kubelet already allows that key; otherwise reapply it after each recreate. + +### Cause 3: the sending application never joins the target group + +Have the application join the target group with `setsockopt(IP_ADD_MEMBERSHIP)` before sending, so the kernel sends an IGMP report and OVN creates the `IGMP_Group` entry. Then check again on the node: + +```bash +kubectl exec -n -- ip maddr +tcpdump -i -nnve igmp +kubectl ko sbctl list IGMP_Group +``` + +The target group should now appear in `ip maddr`, in an IGMP report from the Pod's port, and in `IGMP_Group`. + +## Diagnostic Steps + +1. Confirm unicast and read the group memberships of sender and receiver: + + ```bash + kubectl get pod -n -o wide + kubectl exec -n -- ping -c 3 + kubectl exec -n -- ip maddr + kubectl exec -n -- ip maddr + ``` + +2. Watch IGMP on the node of the sending Pod and read OVN's group table: + + ```bash + tcpdump -i -nnve igmp + kubectl ko sbctl list IGMP_Group + ``` + + No membership, no IGMP report and no `IGMP_Group` entry for the group the Pod sends to: Cause 3. + +3. Check the membership cap: + + ```bash + kubectl exec -n -- sysctl net.ipv4.igmp_max_memberships + ``` + + A join error at 20 memberships: Cause 2. + +4. Unicast works, groups are joined, and DDS from a Pod sender fails while a physical sender on the same address succeeds: Cause 1. + +If unicast to the underlay gateway already fails, the problem is not multicast; see "Kube-OVN underlay Pods fail with resolve MAC address timeout or bridge network InitOVSBridgeFailed on ACP". diff --git a/docs/en/solutions/LoadBalancer_VIP_unreachable_after_project_isolation_or_with_a_MetalLB_address_outside_the_node_segment_on_ACP.md b/docs/en/solutions/LoadBalancer_VIP_unreachable_after_project_isolation_or_with_a_MetalLB_address_outside_the_node_segment_on_ACP.md new file mode 100644 index 000000000..15b0c7d40 --- /dev/null +++ b/docs/en/solutions/LoadBalancer_VIP_unreachable_after_project_isolation_or_with_a_MetalLB_address_outside_the_node_segment_on_ACP.md @@ -0,0 +1,127 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" + - "4.3" +--- + +# LoadBalancer VIP unreachable after project isolation or with a MetalLB address outside the node segment on ACP + +## Issue + +The external IP that MetalLB assigned to a `LoadBalancer` Service does not answer on its port: + +- After the cluster network policy "Isolation between Projects" was turned on, every VIP exposed through MetalLB `LoadBalancer` Services became unreachable. Turning the isolation off brings them back. +- A tool exposed by MetalLB (for example Harbor) cannot be reached from the global cluster. The integration fails with `katanomi-plugin to harbor domain address request timeout`. The domain resolves, but from a global master and from inside the `katanomi-plugin` container network namespace: + - `ping ` returns `Destination Port Unreachable`, `4 packets transmitted, 0 received, +4 errors, 100% packet loss`; + - `curl -v https://:443 --insecure` and `telnet 443` hang at `Trying ...`. + +## Root Cause + +### Cause 1: with isolation between projects, LoadBalancer traffic arrives from the join subnet + +On a Kube-OVN cluster, traffic that arrives for a `LoadBalancer` Service is SNATed to the node's `ovn0` address, not to the node IP. `ovn0` sits in the Kube-OVN join subnet (the `Subnet` named `join`), so the backend Pod sees a source address from the join subnet. The isolation policy does not allow that subnet, so the traffic is dropped. + +The cluster network policy introduced in ACP 4.2 is based on `AdminNetworkPolicy` / `ClusterNetworkPolicy`, supports CIDR matching, and can be configured on the console page; the older cluster network policy lacks this. Kube-OVN does not support skipping ACL checks for specific subnets, so the fix is to allow the join subnet explicitly while keeping isolation on. + +### Cause 2: the MetalLB address is in a segment the nodes are not in + +In MetalLB layer 2 mode, a node answers ARP for the VIP on its own network, so the external address pool must be Layer 2 reachable from the nodes that announce it. When the nodes have their addresses in one `/24` and the MetalLB address is in a different `/24` that the nodes are not attached to, no node can answer for the VIP and it is unreachable from other networks. This is not a firewall problem; compare the VIP with the node segment before requesting firewall changes. + +## Resolution + +### Cause 1: with isolation between projects, LoadBalancer traffic arrives from the join subnet + +Only when the VIPs fail with isolation on and work with it off, on Kube-OVN. Read the join subnet CIDR: + +```bash +kubectl get subnet join -o jsonpath='{.spec.cidrBlock}{"\n"}' +ip -4 addr show ovn0 +``` + +On ACP 4.2 and later, keep isolation enabled and allow the join subnet on the cluster network policy page: + +1. Go to **Networking > Cluster Network Policies > Config Network Policy**. +2. With **Isolation between Projects** enabled, add an **Access Using IP Addresses** entry. +3. Set **Source IP Segment** to the join subnet CIDR and **Target** to the project whose LoadBalancer VIPs must be reachable. + +For a namespace that keeps its own `NetworkPolicy`, the equivalent ingress rule is: + +```yaml +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: allow-join-subnet + namespace: +spec: + podSelector: {} + policyTypes: + - Ingress + ingress: + - from: + - ipBlock: + cidr: +``` + +> **Warning:** The join subnet is the source of all node-to-Pod traffic on Kube-OVN, so allowing it lets every node, and anything SNATed through a node, reach the selected Pods. Narrow `podSelector` and `ports` to the LoadBalancer backends where required. Save the current policies first. Roll back with `kubectl -n delete networkpolicy allow-join-subnet`, or by removing the entry on the policy page. + +```bash +kubectl get networkpolicy -A -o yaml > /tmp/all-netpol.before.yaml +kubectl -n apply -f allow-join-subnet.yaml +curl -sv --max-time 5 http://:/ -o /dev/null +``` + +### Cause 2: the MetalLB address is in a segment the nodes are not in + +Change the address pool to a free address in the nodes' segment, then update everything that refers to the old address (DNS or CoreDNS host records, integrations, client configuration). + +> **Warning:** The Service gets a new external IP; clients and records that use the old address stop working until updated. Only when the VIP is outside every subnet configured on the announcing nodes, and the new address is not used by any other host. Save the pool and Service YAML; restoring them is the rollback. + +```bash +kubectl get ipaddresspools.metallb.io -A -o yaml > /tmp/metallb-pools.before.yaml +kubectl -n edit ipaddresspools.metallb.io +kubectl -n get svc -o wide +curl -kv --max-time 5 https://:443/ -o /dev/null +``` + +Test the new address from a global node before re-running the integration. If the integration uses a domain name held in a CoreDNS host record, point that record at the new address; see "CoreDNS Host Alias records do not take effect because a ResourcePatch removes data hosts from cpaas-coredns on ACP". + +## Diagnostic Steps + +1. Confirm the failing endpoint is the MetalLB external IP and Service port, and read the MetalLB logs: + + ```bash + kubectl get svc -A -o wide | grep LoadBalancer + kubectl get pod -A | grep -i metallb + kubectl -n logs --tail=50 + ``` + +2. Check whether isolation between projects is enabled and whether the VIPs failed when it was turned on: + + ```bash + kubectl get adminnetworkpolicy + kubectl get networkpolicy -A + ``` + + Yes to both: Cause 1. + +3. Compare the VIP with the node addresses: + + ```bash + kubectl get nodes -o wide + ip -4 addr show + ip -4 route + ``` + + A VIP in no node subnet: Cause 2. To see the failure from the calling component, repeat the test inside its network namespace on a global master: + + ```bash + crictl ps | grep katanomi-plugin + crictl inspect | grep -i '"pid"' + nsenter -t -n curl -v --max-time 10 https://:443 --insecure + ``` + +If an application Pod (not the VIP) broke after a policy change, see "Application breaks after a NetworkPolicy change: Ingress, Pod-to-Pod ping, Redis Sentinel or Harbor on ACP". diff --git a/docs/en/solutions/Multus_secondary_networks_unexpected_end_of_JSON_input_duplicated_addresses_or_several_addresses_on_one_NIC_on_ACP.md b/docs/en/solutions/Multus_secondary_networks_unexpected_end_of_JSON_input_duplicated_addresses_or_several_addresses_on_one_NIC_on_ACP.md new file mode 100644 index 000000000..20dda4b62 --- /dev/null +++ b/docs/en/solutions/Multus_secondary_networks_unexpected_end_of_JSON_input_duplicated_addresses_or_several_addresses_on_one_NIC_on_ACP.md @@ -0,0 +1,84 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "3.18" + - "4.0" + - "4.2" + - "4.3" +--- + +# Multus secondary networks: unexpected end of JSON input, duplicated addresses or several addresses on one NIC on ACP + +## Issue + +A Pod with secondary networks attached through Multus (`k8s.v1.cni.cncf.io/networks` and `NetworkAttachmentDefinition`) does not get the NICs or addresses that were intended: + +- The Pod is scheduled, then sandbox creation fails, and Multus reports `failed to unmarshal Spec.Config: unexpected end of JSON input` for `multus-cni-network`. +- The Pod should get three VLAN addresses and instead shows two identical ones. +- A multi-homed Pod should receive several addresses from the same subnet on one NIC, and no annotation makes Kube-OVN do it. + +Related articles: "Kube-OVN Pods fail with AddressOutOfRange, AddressConflict or no address allocated on ACP"; "Calico Pods get no address: disabled default IPPool, Raven fixed IP mismatch, reserved IPs or deleted raven on ACP". + +## Root Cause + +### Cause 1: failed to unmarshal Spec.Config: unexpected end of JSON input + +`GetCNIConfig` cannot parse the NetworkAttachmentDefinition: its `spec.config` (or the annotation that carries it) is truncated or is not valid JSON. Every new Pod that references the attachment fails the same way; Pods that already started keep their NIC. + +### Cause 2: k8s.v1.cni.cncf.io/networks lists the same attachment twice + +The annotation repeats one attachment and omits another, for example `default/vlan161,default/vlan161` instead of `default/vlan161,default/vlan162`. IPAM does what the annotation asks, so the Pod gets two addresses from the same network. + +### Cause 3: one NIC cannot be given several addresses in the same subnet automatically + +Kube-OVN and OVN cannot allocate several addresses from one subnet onto a single NIC of a Pod, and there is no annotation that turns it on. This is a current product gap, not a misconfigured Subnet. + +## Resolution + +### Cause 1: failed to unmarshal Spec.Config: unexpected end of JSON input + +Replace `spec.config` with a complete JSON document, validate it before updating the object, then delete the failed Pod so Multus reads the new config. + +> **Warning:** The replacement must be the full JSON string; there is no partial update. Save the object first and re-apply the saved copy to roll back. + +```bash +kubectl get network-attachment-definitions.k8s.cni.cncf.io -A +kubectl get network-attachment-definitions.k8s.cni.cncf.io -n -o yaml > /root/nad.before.yaml +python3 -m json.tool < /tmp/nad-config.json > /dev/null && echo valid +kubectl edit network-attachment-definitions.k8s.cni.cncf.io -n +kubectl -n delete pod +``` + +For a safe way to rewrite the field, see "Updating a NetworkAttachmentDefinition's spec.config JSON Safely". + +### Cause 2: k8s.v1.cni.cncf.io/networks lists the same attachment twice + +Set the annotation to the distinct attachments, in the order of the addresses you intend, and let the Pod be recreated. + +> **Warning:** Patching the Pod template rolls the workload. + +```bash +kubectl patch deploy -n --type=merge -p '{"spec":{"template":{"metadata":{"annotations":{"k8s.v1.cni.cncf.io/networks":"default/vlan160,default/vlan161,default/vlan162"}}}}}' +``` + +Use your own NetworkAttachmentDefinition names; the example shows each VLAN listed once. + +### Cause 3: one NIC cannot be given several addresses in the same subnet automatically + +Give each address its own NIC and its own NetworkAttachmentDefinition, or use one address. If a single NIC must carry several addresses from one subnet, raise a product requirement; there is no cluster-side switch. + +## Diagnostic Steps + +```bash +kubectl -n describe pod | grep -A5 -E 'FailedCreatePodSandBox|multus' +kubectl -n get pod -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/networks}{"\n"}' +kubectl -n exec -- ip -br addr +``` + +- `failed to unmarshal Spec.Config: unexpected end of JSON input`: Cause 1. +- The same attachment name twice in the annotation and two identical addresses in the Pod: Cause 2. +- A request for several same-subnet addresses on one NIC: Cause 3. diff --git a/docs/en/solutions/Node_events_show_CIDRNotAvailable_and_some_nodes_have_an_empty_podCIDR_on_a_Kube-OVN_cluster_on_ACP.md b/docs/en/solutions/Node_events_show_CIDRNotAvailable_and_some_nodes_have_an_empty_podCIDR_on_a_Kube-OVN_cluster_on_ACP.md new file mode 100644 index 000000000..e4e1e1a14 --- /dev/null +++ b/docs/en/solutions/Node_events_show_CIDRNotAvailable_and_some_nodes_have_an_empty_podCIDR_on_a_Kube-OVN_cluster_on_ACP.md @@ -0,0 +1,79 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Node events show CIDRNotAvailable and some nodes have an empty podCIDR on a Kube-OVN cluster on ACP + +## Issue + +On a cluster whose CNI is Kube-OVN, node events with reason `CIDRNotAvailable` appear, and the following command returns a value on some nodes and nothing on others: + +```bash +kubectl get node -o jsonpath='{.spec.podCIDR}{"\n"}{.spec.podCIDRs}{"\n"}' +``` + +The question is whether these events and the empty fields affect the cluster network. + +Related articles: "Kube-OVN Pods fail with AddressOutOfRange, AddressConflict or no address allocated on ACP". + +## Root Cause + +### Cause 1: CIDRNotAvailable comes from Kubernetes node CIDR allocation, which Kube-OVN does not use + +In Kubernetes, `spec.podCIDR` and `spec.podCIDRs` on a Node are filled by the node IPAM controller of kube-controller-manager when node CIDR allocation is enabled (`--allocate-node-cidrs` with a `--cluster-cidr`). When that controller cannot assign a range to a node, it records a `CIDRNotAvailable` event for the node and leaves the field empty. + +Kube-OVN does not take Pod addresses from the Node `podCIDR`. It allocates them from its own `Subnet` resources (the default subnet, or the subnet bound to the project or namespace). For that reason the `CIDRNotAvailable` events have no impact on the Kube-OVN cluster network itself. Why the controller could not allocate a range in a particular cluster is a separate question and does not change this conclusion. + +## Resolution + +### Cause 1: CIDRNotAvailable comes from Kubernetes node CIDR allocation, which Kube-OVN does not use + +No change is required for the Kube-OVN network. Confirm that Pod networking is healthy on the nodes with an empty `podCIDR`: + +1. Pods on such a node have addresses and are `Running`: + + ```bash + kubectl get pods -A -o wide --field-selector spec.nodeName= + ``` + +2. The Kube-OVN subnets are ready and have free addresses: + + ```bash + kubectl get subnet + ``` + +3. `kube-ovn-controller` reports no allocation failures: + + ```bash + kubectl -n kube-system logs deploy/kube-ovn-controller --tail=500 | grep -iE 'no address allocated|NoAvailableAddress' + ``` + +If all three are clean, the events can be ignored for the network. If Pods do fail to get addresses, the cause is in Kube-OVN allocation, not in the Node `podCIDR`; see the related article. + +## Diagnostic Steps + +1. Compare the nodes that have a `podCIDR` with those that do not: + + ```bash + kubectl get nodes -o custom-columns='NAME:.metadata.name,PODCIDR:.spec.podCIDR,PODCIDRS:.spec.podCIDRs' + kubectl get events -A --field-selector reason=CIDRNotAvailable + ``` + +2. Check whether kube-controller-manager allocates node CIDRs, and from which range: + + ```bash + kubectl -n kube-system get pods -l component=kube-controller-manager -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.containers[*].command}{"\n"}{end}' | tr ',' '\n' | grep -E 'allocate-node-cidrs|cluster-cidr|node-cidr-mask-size' + ``` + +3. Confirm the cluster CNI is Kube-OVN and Pod addresses come from its subnets: + + ```bash + kubectl get subnet -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidrBlock,NS:.spec.namespaces' + kubectl get pods -A -o wide | head + ``` + +Pod addresses inside the Kube-OVN subnet CIDRs on nodes with and without a `podCIDR` confirm that the events do not affect the cluster network. diff --git a/docs/en/solutions/Pod_or_Service_traffic_fails_on_some_nodes_because_of_the_node_kernel_OS_hypervisor_or_network_path_on_ACP.md b/docs/en/solutions/Pod_or_Service_traffic_fails_on_some_nodes_because_of_the_node_kernel_OS_hypervisor_or_network_path_on_ACP.md new file mode 100644 index 000000000..d21a208ae --- /dev/null +++ b/docs/en/solutions/Pod_or_Service_traffic_fails_on_some_nodes_because_of_the_node_kernel_OS_hypervisor_or_network_path_on_ACP.md @@ -0,0 +1,145 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.18" + - "4.0" + - "4.2" + - "4.3" +--- + +# Pod or Service traffic fails on some nodes because of the node kernel, OS, hypervisor or network path on ACP + +## Issue + +The report names the network plugin (Kube-OVN or Calico), but the CNI Pods are healthy or recover by themselves. Typical descriptions: + +- An Istio sidecar fails its startup probe because DNS lookups of `istiod` time out: `lookup istiod.istio-system.svc on :53: read udp :->:53: i/o timeout`. +- A load test fails a few times a day with `Internal Server Error` while no Pod restarts, and log collection drops to zero for a few seconds. +- One Pod of a Deployment cannot reach a ClusterIP (`curl -v` stays at `Trying :...`) while its sibling can. +- Kube-OVN components alert and restart together while the cluster looks fine, or the whole business cluster misbehaves for hours. +- Pods fail with `FailedCreatePodSandBox ... failed to reserve sandbox name`, and the kernel log repeats `unregister_netdevice: waiting for lo to become free`. +- A platform component cannot connect to a Service even though a test from inside the container works. +- Pods on one node cannot reach the apiserver Service while the host can, or newly added nodes behind an ALB return `504`. + +Each cause below lies in the node operating system, the kernel, the virtualization layer or the physical network, not in the CNI configuration. + +Related articles: "calico-node fails to start or Pods hang in ContainerCreating: dummy module, corrupted image, clock skew or x509 on ACP". + +## Root Cause + +### Cause 1: Kylin V10 kernel nf_conntrack defect drops NAT replies + +Kylin V10 kernels older than `4.19.90-52.33.v2207.ky10` have a connection-tracking defect, confirmed by the OS vendor, that breaks reverse NAT translation. Replies that need reverse NAT, such as DNS replies and Service traffic, are dropped by the kernel. It has appeared on `4.19.90-52.22.v2207.ky10.x86_64` and `4.19.90-24.4.v2101.ky10.aarch64`, in three forms: + +- Intermittent Istio sidecar startup failures for some microservices, with `istiod` lookups timing out and `citadelclient failed to sign CSR` and `sds failed to warm certificate` following. `nslookup` of the same name from another container on the node works. Restarting OVS on the node clears it only temporarily. +- Under heavy load, readiness probes fail for a moment and Pods are removed from the Service Endpoints, so the external load balancer has no backend. No Pod restarts. +- One Pod cannot reach a ClusterIP while a sibling can, with DNS, routes, NetworkPolicy, Endpoints and the OVN load balancer all correct. Restarting the Pod helps temporarily; after the kernel upgrade it stayed normal. + +### Cause 2: the node intermittently loses the apiserver VIP + +The node log shows lease renewals failing with `dial tcp :6443: connect: no route to host`. The VIP and its backends are healthy; the node itself cannot reach the VIP, goes `NotReady`, and the OVN components on it restart as a consequence. Continuous ping probes from nodes to the VIP show rounds of heavy or total packet loss and RTT spikes of seconds, aligned with `NodeNotReady` events. Uneven traffic graphs and gaps in node monitoring are side effects. The fault is in the underlying network between the nodes and the VIP. + +### Cause 3: unregister_netdevice: waiting for lo to become free + +A kernel network-namespace leak keeps the loopback device of old Pod sandboxes busy, so those sandboxes are never torn down. On the affected nodes: + +- Platform Pods show `FailedCreatePodSandBox: ... context deadline exceeded`, then repeat `failed to reserve sandbox name "___0": name ... is reserved for ""`. +- Fixed-IP workloads fail with `AcquireAddressFailed` / `NoAvailableAddress`, and `kube-ovn-controller` logs `static address has been assigned to /`, because the old Pods holding the fixed IPs cannot be removed. +- `dmesg` keeps printing `unregister_netdevice: waiting for lo to become free. Usage count = `. + +The OS vendor (Kylin) confirmed it as a known kernel issue. Restarting `ovn-controller` does not help; nodes without the message are unaffected. + +### Cause 4: the VM source/destination check is enabled on the virtualization platform + +The hypervisor drops traffic whose source or destination is a Pod address when the source/destination check (anti-spoofing) is enabled on the node VMs. A platform component that needs a Service fails to start while a test from inside its container succeeds, and cleaning the node or restarting kubelet and the container runtime does not help. + +### Cause 5: one node uses a different iptables backend or has leftover legacy rules + +- A node whose iptables runs in nf_tables mode (for example TencentOS `3.1`, kernel `5.4`, iptables `1.8.5`) joined a cluster whose other nodes use legacy mode (TencentOS `2.4`, kernel `4.19`, iptables `1.4.21`). nf_tables and legacy rule tables are separate, so the rules written by Calico and kube-proxy do not apply to that node's traffic: its host reaches the apiserver Service, every Pod on it times out, and the Calico Pods on it fail to start. Flushing iptables and restarting `calico-node` and `kube-proxy` does not help. Node admission does not check OS or iptables consistency. +- On TencentOS `3.1` with the ACP `3.12` Calico, after an upgrade one Pod (for example `calico-kube-controllers`) cannot be reached from its own node: `ip r get ` returns `local ... dev lo`. Leftover `iptables-legacy` rules that this `calico-node` version does not clean up are the cause; later versions clean them up. +- Newly added TencentOS Server 3 nodes could not carry Calico container traffic at all (ALB `504`, NodePort, Pod IP and Service access to their Pods all failing) while `tunl0` and `cali*` routes looked normal. The mechanism was not determined; replacing them with nodes on the OS version the rest of the cluster runs restored service. + +## Resolution + +### Cause 1: Kylin V10 kernel nf_conntrack defect drops NAT replies + +Upgrade the kernel on all nodes to `4.19.90-52.33.v2207.ky10` or later; the recommended stable version is `kernel-4.19.90-52.44.v2207.ky10`. Until then, restarting OVS on an affected node (deleting its `ovs-ovn` Pod) is a temporary mitigation. Collect `/var/log/kube-ovn` and `/var/log/openvswitch` from the node before restarting if the cause is not yet confirmed. + +> **Warning:** A kernel upgrade needs a reboot. Drain each node first, upgrade one node at a time, and keep the previous kernel in the boot menu as the rollback. Deleting the `ovs-ovn` Pod interrupts Pod traffic on that node until the new Pod is Ready. + +```bash +uname -r +kubectl drain --ignore-daemonsets --delete-emptydir-data +# install the vendor kernel package, reboot the node, then: +kubectl uncordon +kubectl -n kube-system delete pod +``` + +### Cause 2: the node intermittently loses the apiserver VIP + +Hand the path between the nodes and the VIP to the network or IaaS team with evidence. Run a continuous probe from several nodes and keep it until the next alert: + +```bash +journalctl -u kubelet --since '-2h' | grep -E 'no route to host|lease' | tail +mkdir -p /var/log/network_check +while true; do ping -c 50 -i 0.2 -q | tail -2 | sed "s/^/$(date '+%F %T') /"; done >> /var/log/network_check/vip_ping.log +``` + +Stop the loop with `Ctrl-C`. Rounds with non-zero loss, matched against `kubectl get events -A --field-selector reason=NodeNotReady`, are the evidence. See also "Node NotReady caused by network, storage or host-level faults on ACP". + +### Cause 3: unregister_netdevice: waiting for lo to become free + +Upgrade the kernel to the vendor version that fixes the leak, only on nodes where the `waiting for lo` message repeats. + +> **Warning:** The kernel upgrade needs a drain and a reboot of each node. Keep the old kernel as the boot fallback. + +```bash +dmesg -T | grep 'waiting for lo to become free' | tail -3 +uname -r +``` + +### Cause 4: the VM source/destination check is enabled on the virtualization platform + +Disable the source/destination check for every VM that is a cluster node, on the virtualization platform, then retry the component. No cluster change is needed. + +### Cause 5: one node uses a different iptables backend or has leftover legacy rules + +Keep the same OS version, kernel and iptables backend on all nodes of a cluster; replace or reinstall the odd node to match the others. + +For the TencentOS `3.1` variant on ACP `3.12`, flush the leftover legacy rules from inside the `calico-node` Pod on that node. Upgrading `calico-node` to a version that cleans them up is the permanent fix. + +> **Warning:** The flush removes every rule in the `iptables-legacy` filter, raw, mangle and nat tables of the host. Felix rewrites its own rules, but other software on the node that relies on legacy rules loses them. Only when the node matches this variant and `iptables-legacy-save` shows rules. Save them first; restoring the file brings the fault back. + +```bash +kubectl -n kube-system exec -c calico-node -- iptables-legacy-save > /root/iptables-legacy.before +kubectl -n kube-system exec -c calico-node -- sh -c 'iptables-legacy -F; iptables-legacy -F -t raw; iptables-legacy -F -t mangle; iptables-legacy -F -t nat' +curl -m 5 : +``` + +When new nodes on a different OS version cannot carry Pod traffic and no cause is found, replace them with nodes on the OS version the rest of the cluster runs. + +## Diagnostic Steps + +Before changing the CNI, check whether the fault follows a node property: + +```bash +kubectl get nodes -o wide +uname -r; grep -E '^(NAME|VERSION)=' /etc/os-release +dmesg -T | grep -E 'nf_conntrack|waiting for lo' | tail +journalctl -u kubelet --since '-1h' | grep -E 'no route to host|timeout' | tail +iptables --version +iptables-legacy-save 2>/dev/null | wc -l +iptables-nft-save 2>/dev/null | wc -l +``` + +- A Kylin V10 kernel older than `4.19.90-52.33` with DNS timeouts to `istiod`, Endpoint flaps under load, or one Pod that cannot reach a correct ClusterIP: Cause 1. +- `no route to host` toward the VIP in the kubelet log, with `NodeNotReady` events: Cause 2. +- `waiting for lo to become free` together with `failed to reserve sandbox name`: Cause 3. +- Pod traffic failing only on VMs whose source/destination check is on: Cause 4. +- `iptables v1.8.x (nf_tables)` on the failing node and `(legacy)` on the others, or a TencentOS `3.1` node whose `calico-node` still sees `iptables-legacy` rules: Cause 5. + +For Pods that cannot reach the apiserver Service because a host `FORWARD` rule or firewall drops cross-node traffic, see "Cross-node traffic fails in the host network path: default route, firewall, lost tunnel packets or iptables on ACP". diff --git a/docs/en/solutions/Pod_traffic_fails_on_Calico_because_of_an_IPPool_destination_MTU_firewall_BGP_sessions_or_IPv6_routes_on_ACP.md b/docs/en/solutions/Pod_traffic_fails_on_Calico_because_of_an_IPPool_destination_MTU_firewall_BGP_sessions_or_IPv6_routes_on_ACP.md new file mode 100644 index 000000000..fcac6cafe --- /dev/null +++ b/docs/en/solutions/Pod_traffic_fails_on_Calico_because_of_an_IPPool_destination_MTU_firewall_BGP_sessions_or_IPv6_routes_on_ACP.md @@ -0,0 +1,211 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Pod traffic fails on Calico because of an IPPool destination, MTU, firewall, BGP sessions or IPv6 routes on ACP + +## Issue + +On a cluster that uses Calico as the CNI, Pod traffic fails or times out while the node itself is fine. Typical reports: + +- `telnet ` from the node prints `Connected to ...`, but the same command from inside a Pod (or after `nsenter -t -n`) stays at `Trying ...`. +- Pods on some nodes cannot reach a VM outside the cluster while Pods on other nodes can, and the nodes differ in `net.ipv4.conf.all.rp_filter`. +- One business container cannot open an external port, while an agent container in the same Pod, other Pods on the node, and the host all can. +- Business nginx times out toward a master NodePort after the MTU of the physical NIC was changed. +- A Pod-to-Pod call times out, and a capture shows the request already inside the destination Pod. +- ClusterIP `telnet` fails right after a node gained an IPv6 address and IPv6 routes. +- After a node was removed and a new node with the same IP was added, Pods on the new node cannot resolve names, and `ping` from the node to a CoreDNS Pod IP returns `Destination Net Unreachable`. +- In a BGP cluster with IPIP and VXLAN off, traffic from containers to outside services arrives from the node address, and the question is whether Calico encapsulates it. + +Related articles: "Calico ipipMode reverts, Felix metrics off, bandwidth annotation breaks Pods, systemd-udevd loops on calico_tmp on ACP". + +## Root Cause + +### Cause 1: the external destination lies inside a Calico IPPool CIDR + +Calico treats every address inside the CIDR of any `IPPool` as a cluster address. Its `cali-nat-outgoing` chain masquerades Pod traffic only when the destination is outside all pools: + +```text +MASQUERADE all -- 0.0.0.0/0 0.0.0.0/0 /* cali:... */ match-set cali40masq-ipam-pools src ! match-set cali40all-ipam-pools dst +``` + +When the external host's address falls inside a pool CIDR, Pod traffic to it is not source-NATed and is routed as if it were in-cluster traffic (over IPIP when `ipipMode: Always`). The request leaves with the Pod address, and the remote host cannot answer it. The node uses its own routing table, so the same connection works from the node. + +The pool can be the active pool (for example `default-ipv4-ippool` with a `/16` that also covers a range used outside the cluster) or a pool that is disabled and unused: a disabled pool still counts. A difference in `rp_filter` between nodes looks like the cause and is not. + +### Cause 2: a non-Calico firewall rule on the node blocks one container + +The business container hangs on an external address. The host, an agent container in the same Pod, and other Pods on the node connect. `net.ipv4.ip_forward` is `1`, and the container's default route is the normal Calico route `default via 169.254.1.1 dev eth0`. The node has firewall rules that are not written by Calico or kube-proxy, and they conflict with the container's traffic. Correcting those rules restores the connection. + +### Cause 3: the physical NIC MTU was lowered and running Pods kept the old veth MTU + +With the IPIP overlay, the tunnel adds 20 bytes to every packet. When the physical NIC MTU was reduced (for example to 1496) and running Pods were not recreated, their veths stayed at 1480, so encapsulated packets exceeded the NIC MTU and were fragmented or dropped. The symptom is intermittent timeouts, for example business nginx failing toward a master NodePort, while small API checks on the master, host load and kube-proxy logs look normal. + +### Cause 4: the callee Pod ACKs at once and the application does not read + +A caller times out toward one replica. A capture shows the request in the callee Pod's kernel within about a millisecond and an immediate TCP ACK, but the application thread does not read the request for tens of seconds. When it resumes it sends a burst, and the caller has already reset. Other replicas of the same workload are clean. This is an application stall, not a Calico drop; changing IP pools, BGP or MTU does not shorten it. + +### Cause 5: ClusterIP fails after an IPv6 address and IPv6 routes were added to a node + +On a dual-stack Calico cluster, a node was given an IPv6 address and IPv6 routes, and ClusterIP access from or through that node then failed. The mechanism was not determined. Recreating that node's `calico-node` and `kube-proxy` Pods restored ClusterIP access. + +### Cause 6: BGP: Unexpected connect from unknown address, after a node was re-added with the same IP + +After a node is removed and a new node with the same IP joins, an existing node's BIRD can fail to create a BGP session for the new node and reject its connections. There are then no routes to the Pods on the new node: from the new node, `ping` to the CoreDNS Service address works while `ping` to a CoreDNS Pod IP fails with `Destination Net Unreachable`, and its Pods cannot resolve names. TCP `179` from the new node is refused on the affected peers, whose `calico-node` log repeats: + +```text +bird: BGP: Unexpected connect from unknown address (port ) +``` + +`calicoctl node status` can still show every peer `up`, but `birdcl show protocols` on the affected peer lists `Mesh_*` sessions to the other nodes and no entry for the new node. This is a low-probability timing defect in Calico BGP peering. Restarting `calico-node` on the new node does not help; restarting it on the affected peer does. + +### Cause 7: container egress leaves with the node address because the IPPool has natOutgoing: true + +In BGP mode with `ipipMode` and `vxlanMode` off, nothing is encapsulated. When the pool has `natOutgoing: true`, Calico source-NATs container traffic that leaves the cluster pools to the node address, so external services see the node, and return traffic comes back to the node and is translated back there. This is the expected behavior of that setting. + +## Resolution + +### Cause 1: the external destination lies inside a Calico IPPool CIDR + +1. List the pools and check which CIDR contains the destination: + + ```bash + kubectl get ippools.crd.projectcalico.org -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidr,DISABLED:.spec.disabled,NAT:.spec.natOutgoing' + ``` + +2. If the destination is inside the pool the cluster actually uses, reach the external host through an address outside every pool CIDR (for example, have the network team map the service to such an address), then test again from the Pod. Do not shrink or delete a pool that holds Pod addresses. + +3. If the destination is inside a disabled pool that no Pod uses, back the pool up and delete it, so traffic to that range is NATed like other external traffic. Leave `rp_filter` as it is. + + > **Warning:** Deleting an `IPPool` that still has allocations strands those Pods. Only when no Pod address lies in its CIDR. Roll back by applying the saved YAML. If a platform `Subnet` with the same name manages the pool, handle that object too, or the pool comes back. + + ```bash + kubectl get pod -A -o wide | grep '' || true + kubectl get ippools.crd.projectcalico.org -o yaml > /root/ippool-.yaml + kubectl delete ippools.crd.projectcalico.org + ``` + +### Cause 2: a non-Calico firewall rule on the node blocks one container + +Compare the node's firewall with a node where every container connects, and remove or correct the rules that are neither Calico's (`cali-*`) nor kube-proxy's (`KUBE-*`) and that affect the container's egress. + +> **Warning:** Editing the filter table can cut SSH and API access. Work from a console, save the rules first, and change only rules that a healthy node does not have. Roll back with `iptables-restore < /root/iptables.before`. + +```bash +iptables-save > /root/iptables.before +iptables -S +sysctl net.ipv4.ip_forward +``` + +Retry from the business container after the rules match a healthy node. + +### Cause 3: the physical NIC MTU was lowered and running Pods kept the old veth MTU + +Put the physical NIC MTU back to the value it had before the change, so that the Pod MTU plus the 20-byte IPIP header fits again, and persist it in the node network configuration so a reboot does not reapply the lower value. + +> **Warning:** Changing the NIC MTU disturbs large in-flight packets on that port. Only when the NIC MTU was reduced and running Pods still show the old veth MTU. Record the current value; setting it again is the rollback (and brings the fragmentation back). + +```bash +ip link show +ip -d link show | grep -E 'mtu|veth' | head +ip link set dev mtu +``` + +### Cause 4: the callee Pod ACKs at once and the application does not read + +No CNI change. Capture on the callee node long enough to see whether the application reads the payload. An immediate kernel ACK followed by tens of seconds of silence from the process, with other replicas healthy, goes to the application owner. + +```bash +tcpdump -ni any -c 50 host and port +``` + +### Cause 5: ClusterIP fails after an IPv6 address and IPv6 routes were added to a node + +Delete the `calico-node` and `kube-proxy` Pods on that node so the DaemonSets recreate them. + +> **Warning:** Until the new Pods are Ready the node's routes and Service rules are reprogrammed and in-flight connections on it reset. Only when ClusterIP access broke after the node received IPv6 addresses and routes. Restart one node at a time and confirm ClusterIP from a Pod on that node before the next. Nothing needs to be rolled back. + +```bash +kubectl -n kube-system get pod -o wide | grep -E 'calico-node|kube-proxy' | grep +kubectl -n kube-system delete pod +kubectl -n kube-system rollout status ds/calico-node +``` + +### Cause 6: BGP: Unexpected connect from unknown address, after a node was re-added with the same IP + +1. Find the peers that have no session to the re-added node: + + ```bash + kubectl -n kube-system logs -c calico-node --tail=500 | grep 'Unexpected connect from unknown address' + kubectl -n kube-system exec -c calico-node -- birdcl -s /var/run/calico/bird.ctl show protocols + ``` + +2. Restart `calico-node` on each affected peer. + + > **Warning:** Restarting `calico-node` on a peer withdraws and re-learns its BGP routes, so Pods on it can briefly lose cross-node traffic. Only on peers whose log shows the message above or whose protocol list lacks the new node. Restart one peer at a time; the DaemonSet recreates the Pod, so there is nothing to roll back. + + ```bash + kubectl -n kube-system delete pod + kubectl -n kube-system exec -c calico-node -- birdcl -s /var/run/calico/bird.ctl show protocols | grep -i mesh + ``` + +3. If it recurs, check session establishment with `calicoctl get bgpsessions -o wide`. If it is empty, do a rolling restart of every `calico-node` Pod except the one on the re-added node, one at a time. + +### Cause 7: container egress leaves with the node address because the IPPool has natOutgoing: true + +No fault to repair. Confirm the setting: + +```bash +kubectl get ippools.crd.projectcalico.org -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidr,NAT:.spec.natOutgoing,IPIP:.spec.ipipMode,VXLAN:.spec.vxlanMode' +``` + +Showing Pod addresses to the outside instead is a design change, not a fix: with NAT off, the external network has to route the Pod CIDR back to the nodes. + +## Diagnostic Steps + +1. Compare the node and the Pod toward the same destination, and check whether the destination is inside a pool: + + ```bash + telnet + nsenter -t -n telnet + iptables -t nat -L cali-nat-outgoing -n + kubectl get ippools.crd.projectcalico.org -o yaml | grep -E 'name:|cidr:|disabled:|natOutgoing:' + ``` + + The node connects, the Pod stays at `Trying`, and the destination is inside a pool CIDR: Cause 1. + +2. Check whether only one container fails. Run the same test from another Pod on the node; if only one container fails, compare `iptables -S` with a healthy node: Cause 2. + +3. Check MTU and fragmentation: + + ```bash + kubectl exec -n -- ip route + ip link show + ping -M do -s 1400 -c 2 || true + ``` + + Timeouts that start when the NIC MTU was lowered while Pod veths kept the old MTU: Cause 3. + +4. An immediate ACK in the callee Pod followed by a silent application: Cause 4. A ClusterIP failure that starts with an IPv6 route change on dual-stack: Cause 5. + +5. For a re-added node, check BGP on the peers: + + ```bash + kubectl -n kube-system logs -c calico-node --tail=500 | grep -E 'Unexpected connect|BGP' + ss -anp | grep ':179' + ``` + + `Unexpected connect from unknown address `: Cause 6. + +6. External services see the node address and the pool has `natOutgoing: true`: Cause 7, expected behavior. + +If cross-node `ping` returns `Destination Host Prohibited`, see "Cross-node traffic fails in the host network path: default route, firewall, lost tunnel packets or iptables on ACP". diff --git a/docs/en/solutions/Pods_cannot_resolve_intranet_or_external_names_nameserver_order_internal_DNS_server_or_debug_container_ping_on_ACP.md b/docs/en/solutions/Pods_cannot_resolve_intranet_or_external_names_nameserver_order_internal_DNS_server_or_debug_container_ping_on_ACP.md new file mode 100644 index 000000000..ce3996900 --- /dev/null +++ b/docs/en/solutions/Pods_cannot_resolve_intranet_or_external_names_nameserver_order_internal_DNS_server_or_debug_container_ping_on_ACP.md @@ -0,0 +1,135 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Pods cannot resolve intranet or external names: nameserver order, internal DNS server or debug container ping on ACP + +## Issue + +Pods fail to resolve names outside the cluster. Typical reports: + +- External names from Pods succeed and then fail. The node `/etc/resolv.conf` lists several `nameserver` lines, mixing an intranet DNS that does not recurse to the public Internet with one that does. +- The node scale-out precheck stops and says `/etc/resolv.conf` may contain only two `nameserver` entries, while the site needs intranet and public resolution. +- Applications must reach external systems by domain name, and those names are served only by an internal DNS server: `nslookup ` from a Pod returns `NXDOMAIN` or times out. +- `ping ` in a container debug session reports that the name cannot be resolved, while `ping
` of the same host works. + +Related articles: "Pods fail to resolve Service names or resolve them to external addresses: search domains and ndots on ACP"; "Host DNS changes do not reach CoreDNS: systemd-resolved, kubelet resolvConf, missing nameserver or a new upstream on ACP". + +## Root Cause + +### Cause 1: a list of nameservers is not split-horizon + +A list of nameservers in `/etc/resolv.conf` is not "ask the next server when this one says no such name". A resolver stops at the first definitive `NXDOMAIN`. Putting an intranet server that does not recurse in front of a public server therefore fails public names, and the reverse order fails intranet names. CoreDNS inherits the same list through `forward . /etc/resolv.conf`. ACP nodes also do not accept more than two `nameserver` lines: the scale-out precheck refuses a longer list, and that limit is not bypassed by a sysctl or a kubelet flag. + +### Cause 2: external system names exist only on an internal DNS server + +The lookup path is: business Pod, then `kube-dns` (CoreDNS), then `forward`, then the nameservers in `/etc/resolv.conf` of the node that runs the CoreDNS Pod. If neither the Corefile `forward` target nor those node nameservers is the internal DNS server, the internal zone cannot be answered. Either pointing the CoreDNS `forward` at the internal DNS server, or adding the server to the node resolver file on every node and restarting CoreDNS, makes the names resolvable. + +### Cause 3: ping by name fails in a musl-based debug container + +The container debug image is built on the musl C library. Its `ping` requests IPv6 by default and fails at once when the name has no `AAAA` record; a glibc-based `ping` falls back to IPv4. Cluster DNS is not failing. This is a property of the base image, not a CoreDNS setting. + +## Resolution + +### Cause 1: a list of nameservers is not split-horizon + +Give the nodes one resolver address that forwards by zone (intranet zones to the intranet servers, everything else to the public servers); publish it twice if a second address is needed for redundancy, and point every node's `/etc/resolv.conf` at those addresses only. + +When only Pods need the split and the node file must keep two lines, put per-zone `forward` stanzas in the Corefile instead. Example shape with documentation addresses: + +```text +intranet.example.com:1053 { + forward . 192.0.2.53 192.0.2.54 +} +.:1053 { + errors + kubernetes cluster.local in-addr.arpa ip6.arpa { + fallthrough in-addr.arpa ip6.arpa + } + forward . /etc/resolv.conf +} +``` + +> **Warning:** A broken zone stanza black-holes that zone for every Pod. Confirm each upstream answers a known name with `dig` from a node before adding it, and save the ConfigMap. The platform turns the ConfigMap edit into a ResourcePatch; wait until it is applied before testing. Roll back with `kubectl apply -f /tmp/cpaas-coredns.before.yaml`. + +```bash +kubectl -n kube-system get cm cpaas-coredns -o yaml > /tmp/cpaas-coredns.before.yaml +kubectl -n kube-system edit cm cpaas-coredns +``` + +Static names that are not a whole zone belong in a CoreDNS Host Alias (an IP, then domains separated by spaces), not in a third `nameserver` line. + +### Cause 2: external system names exist only on an internal DNS server + +Only when the internal DNS server answers the name (`dig @ ` from a node) and the Pod does not. Use one of the two approaches. + +**Corefile forward.** When the internal server should answer every non-cluster name, replace the target of the default `forward`; when it should answer only some zones, add a per-zone stanza as in Cause 1 and keep `forward . /etc/resolv.conf`: + +```text +.:1053 { + # other plugins unchanged + forward . 192.0.2.53 +} +``` + +> **Warning:** Changing the default `forward` sends every external lookup in the cluster to the new server; if it does not recurse for public names, public lookups fail for all Pods. Save the ConfigMap and confirm with `dig` that the server answers both internal names and the public names the cluster needs. Wait for the ResourcePatch to be applied. Roll back with `kubectl apply -f /tmp/cpaas-coredns.before.yaml`. + +```bash +kubectl -n kube-system get cm cpaas-coredns -o yaml > /tmp/cpaas-coredns.before.yaml +kubectl -n kube-system edit cm cpaas-coredns +kubectl exec -n -- nslookup +``` + +**Node resolver.** Add the server as a `nameserver` line in `/etc/resolv.conf` on every node, then restart CoreDNS so its Pods receive the new list. The two-line limit and the first-`NXDOMAIN` behavior of Cause 1 apply to this file. The full procedure is Cause 4 of "Host DNS changes do not reach CoreDNS: systemd-resolved, kubelet resolvConf, missing nameserver or a new upstream on ACP". + +> **Warning:** Restarting CoreDNS briefly delays lookups cluster-wide. Back up `/etc/resolv.conf` on each node; restoring it and restarting CoreDNS again is the rollback. + +```bash +cp -a /etc/resolv.conf /root/resolv.conf.before +kubectl -n kube-system rollout restart deploy/coredns +kubectl -n kube-system rollout status deploy/coredns +``` + +Adding the server to `spec.dnsConfig.nameservers` of individual workloads is a third possibility that affects only those workloads; a hand edit of the file inside a running container is lost on the next recreate. + +### Cause 3: ping by name fails in a musl-based debug container + +No cluster change fixes this. Test name resolution directly, and reachability by address: + +```bash +nslookup +ping -c 3 +``` + +A glibc-based image resolves the IPv4 address and pings it. + +## Diagnostic Steps + +1. Ask each node nameserver for the failing name; a definitive `NXDOMAIN` from the first one explains a failure even when the second one knows the name: + + ```bash + cat /etc/resolv.conf + dig @ +short + dig @ +short + kubectl -n kube-system get pod -l k8s-app=kube-dns -o wide + ``` + +2. Read the Corefile forward targets: + + ```bash + kubectl -n kube-system get cm cpaas-coredns -o jsonpath='{.data.Corefile}{"\n"}' | grep -B2 -A5 forward + ``` + +3. Map the result: + + - Public names flap, the node has several nameservers, or the precheck says only two are allowed: Cause 1. + - The name is answered by the internal DNS server and returns `NXDOMAIN` or a timeout from the Pod: Cause 2. + - `ping` by name fails only in the debug container while `nslookup` of the same name works: Cause 3. + +If `nslookup` to `kube-dns` times out instead of returning `NXDOMAIN`, check NetworkPolicies and the Pod's route to the CoreDNS ClusterIP before changing resolvers. diff --git a/docs/en/solutions/Pods_fail_to_resolve_Service_names_or_resolve_them_to_external_addresses_search_domains_and_ndots_on_ACP.md b/docs/en/solutions/Pods_fail_to_resolve_Service_names_or_resolve_them_to_external_addresses_search_domains_and_ndots_on_ACP.md new file mode 100644 index 000000000..49c05e08a --- /dev/null +++ b/docs/en/solutions/Pods_fail_to_resolve_Service_names_or_resolve_them_to_external_addresses_search_domains_and_ndots_on_ACP.md @@ -0,0 +1,131 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Pods fail to resolve Service names or resolve them to external addresses: search domains and ndots on ACP + +## Issue + +A Pod fails names that cluster DNS answers when asked directly, resolves a Service to the wrong address, or spends a long time resolving. Typical reports: + +- Application logs show `Name or service not known`, `UnknownHostException` or `SearchDomainUnknownHostException` for a name like `.` (one dot) or a short Service name, while `..svc.cluster.local` returns `NOERROR`. +- An in-cluster Service such as `mysql` starts returning an external address after someone pointed containers at an external resolver; the node resolves public names, the Pod does not unless a public resolver is written into the container. +- A task hangs for a long time at one stage with the default `dnsPolicy: ClusterFirst`, and does not hang with `dnsPolicy: Default`. + +A Pod's `/etc/resolv.conf` is generated by the kubelet. With `dnsPolicy: ClusterFirst`, `nameserver` is the `kube-dns` ClusterIP, `options ndots:5` is set, and `search` lists `.svc.cluster.local`, `svc.cluster.local`, `cluster.local`, plus the search or domain suffix of the node's own `/etc/resolv.conf`. + +Related articles: "Pods cannot resolve intranet or external names: nameserver order, internal DNS server or debug container ping on ACP". + +## Root Cause + +### Cause 1: an expanded name returns NXDOMAIN while the FQDN returns NOERROR + +With `ndots:5`, a name with fewer than five dots is tried with each search suffix before it is tried as an absolute name. When the node's `/etc/resolv.conf` has a company `domain` or `search` line, the kubelet copies it into the Pod search list. The CoreDNS log then shows `NOERROR` for `..svc.cluster.local` and `NXDOMAIN` for expansions such as the name with the namespace suffix appended twice or with the company domain appended. Resolvers such as Netty fail the whole lookup when one expanded query returns `NXDOMAIN`, even though another expansion is the real Service. CoreDNS itself answers the FQDN quickly. + +A readiness failure of one CoreDNS replica on `http://:8181/ready` (`i/o timeout`) in the same window is a separate fault and does not explain these `NXDOMAIN` answers. + +### Cause 2: a Service name resolves to an external address after external DNS was added to the container + +Writing nameservers into a container's `/etc/resolv.conf` bypasses `kube-dns`. The remaining search list, or a search domain inherited from the node, then lets a short name (for example `mysql`) match an external zone. CoreDNS learns the node's resolvers only through `forward . /etc/resolv.conf` on the node where the CoreDNS Pod runs; a nameserver inside one container is not cluster DNS, and it is lost when the Pod is recreated. + +If only some nodes have a route to the public resolver, CoreDNS Pods on the other nodes time out on external names while in-cluster names still work. + +### Cause 3: the task hangs with dnsPolicy ClusterFirst and not with dnsPolicy Default + +With `ClusterFirst`, every external name with fewer than five dots is first tried with each search suffix, often for both A and AAAA records, and each attempt is a round trip. An application that resolves many external names, or retries after slow answers, accumulates that delay. `dnsPolicy: Default` copies the node file, which has no cluster search list and the default `ndots:1`, so the expansions disappear. Lowering `ndots` to `1` on the workload reduces resolution time while keeping cluster DNS. + +## Resolution + +### Cause 1: an expanded name returns NXDOMAIN while the FQDN returns NOERROR + +Only when CoreDNS shows `NOERROR` for the FQDN and `NXDOMAIN` for an expansion that repeats the namespace or appends the company domain. Apply both changes: + +1. Point the application at the short Service name or at the full FQDN `..svc.cluster.local` (a trailing dot forces an absolute query). +2. Remove the company `domain` / `search` suffix from `/etc/resolv.conf` on every node, keep the `nameserver` lines, and recreate the application Pods. + +> **Warning:** Editing the node file changes resolution for host processes and for every new Pod. Copy the file on each node first; restoring it and recreating the Pods is the rollback. In-cluster Service names keep working through `kube-dns`. + +```bash +cp -a /etc/resolv.conf /root/resolv.conf.before +vi /etc/resolv.conf # delete only the domain or search token with the company suffix +kubectl delete pod -n +kubectl exec -n -- cat /etc/resolv.conf +``` + +New Pods should list `search .svc.cluster.local svc.cluster.local cluster.local` without the company domain. See also "Platform UI stuck loading or DevOps errors from Service name timeouts caused by node search domains on ACP". + +### Cause 2: a Service name resolves to an external address after external DNS was added to the container + +Fix the nodes, not the container. Remove the extra search domain from `/etc/resolv.conf` on all nodes; on the nodes that run CoreDNS, put the resolver that answers public names first. Restart CoreDNS so it re-reads the file, then recreate the test Pod. + +> **Warning:** Cluster lookups stall briefly while CoreDNS restarts, and recreated application Pods drop their connections. Confirm the first nameserver answers every zone you need and save `/etc/resolv.conf` first; restoring it and restarting CoreDNS again is the rollback. + +```bash +cp -a /etc/resolv.conf /root/resolv.conf.before +kubectl -n kube-system rollout restart deploy/coredns +kubectl -n kube-system rollout status deploy/coredns +kubectl delete pod -n +kubectl exec -n -- nslookup kubernetes.default +kubectl exec -n -- nslookup +``` + +If public resolution works only on some nodes, keep `/etc/resolv.conf` identical on all nodes and schedule the Pods that need the public network onto the nodes that have the route (taints on the other nodes, plus a matching toleration and node affinity on those workloads). For one external name with a fixed address, add a CoreDNS Host Alias instead of a container entry; see "CoreDNS Host Alias records do not take effect because a ResourcePatch removes data hosts from cpaas-coredns on ACP". + +### Cause 3: the task hangs with dnsPolicy ClusterFirst and not with dnsPolicy Default + +Keep `dnsPolicy: ClusterFirst` and set `ndots` to `1` in the workload's `dnsConfig`: + +```yaml +spec: + template: + spec: + dnsPolicy: ClusterFirst + dnsConfig: + options: + - name: ndots + value: "1" +``` + +> **Warning:** Patching the Pod template rolls the workload. With `ndots:1`, a name such as `.` is tried as an absolute name first, so the application should use short names or full FQDNs. Save the Deployment; applying the saved copy is the rollback. + +```bash +kubectl -n get deployment -o yaml > /tmp/.before.yaml +kubectl -n patch deployment --type=strategic \ + -p '{"spec":{"template":{"spec":{"dnsConfig":{"options":[{"name":"ndots","value":"1"}]}}}}}' +kubectl -n rollout status deployment +kubectl exec -n -- cat /etc/resolv.conf +``` + +The new Pod shows `options ndots:1` with the same `nameserver` and `search` lines. The same change on a platform component is described in "ArgoCD repo-server DNS lookups leak upstream because of pod ndots:5 search expansion". + +## Diagnostic Steps + +1. Compare the node file, the Pod file, and one absolute query: + + ```bash + cat /etc/resolv.conf + kubectl exec -n -- cat /etc/resolv.conf + kubectl exec -n -- nslookup + kubectl exec -n -- nslookup ..svc.cluster.local + ``` + +2. Turn on CoreDNS query logging if needed (add `log` to the server block of `cpaas-coredns`; it reloads without a restart) and read the answers: + + ```bash + kubectl -n kube-system edit cm cpaas-coredns + kubectl -n kube-system logs -l k8s-app=kube-dns --tail=50 + ``` + +3. Map the result: + + - FQDN `NOERROR`, expanded name `NXDOMAIN`, company domain in the Pod search list: Cause 1. + - Short Service name returns an off-cluster address and the container's `/etc/resolv.conf` was edited: Cause 2. + - Hang with `ClusterFirst`, none with `dnsPolicy: Default`, `options ndots:5` in the Pod: Cause 3. + +Read the cause from the CoreDNS log rather than from a monitoring chart before changing resolvers. diff --git a/docs/en/solutions/Traefik_gateway_X-Forwarded-For_missing_ports_80_and_443_underscore_headers_weighted_routes_and_route_export_on_ACP.md b/docs/en/solutions/Traefik_gateway_X-Forwarded-For_missing_ports_80_and_443_underscore_headers_weighted_routes_and_route_export_on_ACP.md new file mode 100644 index 000000000..f26e8813b --- /dev/null +++ b/docs/en/solutions/Traefik_gateway_X-Forwarded-For_missing_ports_80_and_443_underscore_headers_weighted_routes_and_route_export_on_ACP.md @@ -0,0 +1,196 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" +--- + +# Traefik gateway: X-Forwarded-For missing, ports 80 and 443, underscore headers, weighted routes and route export on ACP + +## Issue + +The customized Traefik gateway (`traefik.traefik.alauda.io`), delivered for specific projects, is in use, and one of these comes up: + +- Traefik cannot get the client IP from the request headers. The access log for one entrypoint shows `ClientHost` `127.0.0.6` and `request_X-Real-Ip` `127.0.0.6` and no `X-Forwarded-For` value, although the client sent one. The response carries `server: istio-envoy`. +- Does Traefik drop request headers whose names contain an underscore? +- Can Traefik's default ports 80 and 443 be closed, or opened again? +- Where is a `TraefikService` maintained? The console has no page for it, and the goal is a gray release (weighted split between a stable and a new version) without the service mesh module. +- Can routes be exported and imported between environments by copying the IngressRoute YAML from the console? + +## Root Cause + +### Cause 1: the entrypoint has no forwardedHeaders trust + +Traefik trusts `X-Forwarded-*` headers only from addresses listed in `forwardedHeaders.trustedIPs`, or from every source when `forwardedHeaders.insecure` is true, and the setting is per entrypoint. Requests reached Traefik through an Istio sidecar, so the source Traefik saw was `127.0.0.6` (the address the sidecar uses for inbound traffic to the application container). The custom entrypoint that received them (for example `mytcp` on ACP 4.2.2) had no `forwardedHeaders` setting, so the client's `X-Forwarded-For` did not appear in the log, while other ports could keep it. + +### Cause 2: Traefik accepts header names with underscores by default + +Traefik supports request headers whose names contain underscores by default; no extra configuration is needed. This differs from Envoy Gateway, which rejects such headers by default. + +### Cause 3: ports 80 and 443 are the web and websecure entries of the instance + +Ports 80 and 443 are the `web` and `websecure` entries under `spec.ports` of the Traefik instance, and their `expose.default` field controls whether they are published. + +### Cause 4: TraefikService has no UI, and a weighted IngressRoute does the gray release + +Traefik is a project-specific customization, and the console does not support `TraefikService`. A gray release without the service mesh module is done with an `IngressRoute` whose route lists two backend Services with weights. + +### Cause 5: copying the IngressRoute YAML is a valid way to move routes + +Opening a route under **Resource Management > IngressRoute**, copying its YAML, cleaning it up and creating it in another environment is a viable way to export and import Traefik routes, provided the target environment is compatible and already holds everything the route depends on. + +## Resolution + +### Cause 1: the entrypoint has no forwardedHeaders trust + +Only when the access log of a specific entrypoint lacks the `X-Forwarded-For` value the request carried. Add `forwardedHeaders.insecure: true` to that port of the Traefik instance, at the same level as `expose`, `exposedPort` and `port`: + +```yaml +spec: + ports: + mytcp: + expose: + default: true + exposedPort: 9000 + forwardedHeaders: + insecure: true + port: 19090 +``` + +Keep your own entry name and ports; only add the `forwardedHeaders` block. Traefik also accepts `forwardedHeaders.trustedIPs` with a list of trusted source CIDRs instead of `insecure`. + +> **Warning:** `insecure: true` makes the entrypoint trust `X-Forwarded-For` from any source, so a client that reaches the port directly can set any address, and backends or allowlists that read the header will believe it. The Traefik Pods can roll, dropping connections. Use a maintenance window and save the instance first; removing `forwardedHeaders` or applying the saved file is the rollback. + +```bash +kubectl get traefik.traefik.alauda.io -n -o yaml > /tmp/traefik-before.yaml +kubectl edit traefik.traefik.alauda.io -n +kubectl rollout status deploy/ -n +kubectl logs -n deploy/ --since=5m | grep '' +``` + +The new log entry should show the `X-Forwarded-For` value. Removing the Istio sidecar from the Traefik Pod, so that Traefik is the only proxy on the hop, is a separate option; the `forwardedHeaders` change alone is enough. + +### Cause 2: Traefik accepts header names with underscores by default + +No change is needed. If such a header really goes missing behind Traefik, test the hop (Diagnostic Steps) before changing any setting. + +### Cause 3: ports 80 and 443 are the web and websecure entries of the instance + +Set `spec.ports.web.expose.default` and `spec.ports.websecure.expose.default` to `false` to close them, or `true` to open them again. Example that keeps a custom port and closes both: + +```yaml +spec: + ports: + mytcp: + expose: + default: true + exposedPort: 9000 + port: 9000 + web: + expose: + default: false + websecure: + expose: + default: false +``` + +Leave any `forwardedHeaders` settings that the entries already have. + +> **Warning:** Clients that use ports 80 or 443 on this instance lose access as soon as the change applies. Confirm no route still needs `web` or `websecure`, and save the instance; setting `expose.default` back to `true` is the rollback. + +```bash +kubectl edit traefik.traefik.alauda.io -n +kubectl get svc -n -o custom-columns='NAME:.metadata.name,TYPE:.spec.type,PORTS:.spec.ports[*].port' +``` + +### Cause 4: TraefikService has no UI, and a weighted IngressRoute does the gray release + +Give the stable (v1) and gray (v2) versions their own Services, then create a weighted `IngressRoute`. This example sends 90 percent of `app.example.com` to `app-v1` and 10 percent to `app-v2` on entrypoint `web` (use `websecure` for HTTPS, or another exposed entrypoint if `web` is closed): + +```yaml +apiVersion: traefik.io/v1alpha1 +kind: IngressRoute +metadata: + name: app-ingress-gray + namespace: +spec: + entryPoints: + - web + routes: + - match: Host(`app.example.com`) + kind: Rule + services: + - name: app-v1 + kind: Service + port: 80 + weight: 90 + - name: app-v2 + kind: Service + port: 80 + weight: 10 +``` + +```bash +kubectl get svc app-v1 app-v2 -n +kubectl apply -f /tmp/app-ingress-gray.yaml +kubectl get ingressroute app-ingress-gray -n -o yaml +``` + +Change the two `weight` values and apply again to shift traffic to v2. + +### Cause 5: copying the IngressRoute YAML is a valid way to move routes + +1. Check that the target environment is ready: + - Traefik is installed with the matching IngressRoute CRD, and the `apiVersion` is compatible (for example `traefik.io/v1alpha1`). + - The backend Services, Middlewares, TLS Secrets and TLSOption/TLSStore resources that the route references exist there. + - Environment-specific settings are prepared: entrypoints, domain DNS, the VIP or LoadBalancer Service, and the namespaces the Traefik instance watches. + + ```bash + kubectl get crd | grep -i traefik + kubectl get traefik.traefik.alauda.io -A + kubectl get svc,secret -n + ``` + +2. Export the route, remove the server-generated fields (`metadata.uid`, `metadata.resourceVersion`, `metadata.creationTimestamp`, `metadata.generation`, `metadata.managedFields` and `status`), and adjust the domain, certificate, Services and entrypoints for the target. + + ```bash + kubectl get ingressroute -n -o yaml > /tmp/.yaml + vi /tmp/.yaml + ``` + +3. Create it in the target environment and test the domain: + + ```bash + kubectl apply -f /tmp/.yaml + kubectl get ingressroute -n + ``` + +## Diagnostic Steps + +1. Find the instance, its Pods and the entrypoint that handles the request: + + ```bash + kubectl get traefik.traefik.alauda.io -A + kubectl get traefik.traefik.alauda.io -n -o yaml | sed -n '/^ ports:/,/^ [a-z]/p' + kubectl get pods -n -o wide + ``` + +2. For a missing client address, send a request with a known header and read its log line: + + ```bash + curl -v -H 'X-Forwarded-For: 203.0.113.10' http://:/ + kubectl logs -n | grep '' + kubectl get pod -n -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{end}' + ``` + + `ClientHost` and `request_X-Real-Ip` of `127.0.0.6`, no `X-Forwarded-For` value, and no `forwardedHeaders` on that port: Cause 1. An `istio-proxy` container means a sidecar sits in front of Traefik. + +3. For an underscore header, send it through Traefik to a backend that logs headers and compare with a request straight to the Service: + + ```bash + curl -v -H 'x_custom_header: test' http://:/ + ``` + +4. For ports 80 and 443, read `spec.ports.web` and `spec.ports.websecure` and the ports on the Traefik Service. For gray release or route export, confirm the CRD and the Services with `kubectl get ingressroute -A` and `kubectl get svc -n `. diff --git a/docs/en/solutions/calico-node_fails_to_start_or_Pods_hang_in_ContainerCreating_dummy_module_corrupted_image_clock_skew_or_x509_on_ACP.md b/docs/en/solutions/calico-node_fails_to_start_or_Pods_hang_in_ContainerCreating_dummy_module_corrupted_image_clock_skew_or_x509_on_ACP.md new file mode 100644 index 000000000..2b7715b65 --- /dev/null +++ b/docs/en/solutions/calico-node_fails_to_start_or_Pods_hang_in_ContainerCreating_dummy_module_corrupted_image_clock_skew_or_x509_on_ACP.md @@ -0,0 +1,144 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.18" + - "4.0" + - "4.2" + - "4.3" +--- + +# calico-node fails to start or Pods hang in ContainerCreating: dummy module, corrupted image, clock skew or x509 on ACP + +## Issue + +On a Calico cluster, `calico-node` does not start on a node, or Pods on a node stay in `ContainerCreating` with Calico errors. Typical reports: + +- `calico-node` fails with `failed to connect kubernetes svc:443`. +- `calico-node` does not start on a node right after a VM live migration, and container DNS and logins fail. +- Pods stay in `ContainerCreating`, the kubelet log shows Calico errors while setting up the Pod network, and after a restart `calico-node` fails with `x509: certificate has expired or is not yet valid: current time ... is before ...`. +- After every node of the cluster was restarted, the Pods on one node stay in `ContainerCreating` with `plugin type="calico" failed (add): error getting ClusterInformation: ... x509: certificate signed by unknown authority`. + +Each cause is a property of the node (kernel modules, local image store, clock), not of the Calico configuration. + +Related articles: "Pod or Service traffic fails on some nodes because of the node kernel, OS, hypervisor or network path on ACP". + +## Root Cause + +### Cause 1: calico-node: failed to connect kubernetes svc:443, and the dummy kernel module is missing + +`calico-node` fails to start with `failed to connect kubernetes svc:443`. The node has no dummy interface, and `lsmod` shows no `dummy` module because the module file is missing from `/lib/modules/$(uname -r)`. + +### Cause 2: calico-node image layers are corrupted after a VM live migration + +After the node VM was live-migrated, the local image layers of the `calico-node` image were corrupted, so the container could not start and Pod networking on the node failed. + +### Cause 3: node clock out of sync, install-cni fails x509 verification + +Pods stay in `ContainerCreating`, and restarting Calico makes it worse: `calico-node` no longer starts. Its `install-cni` container keeps exiting `1`, and its log shows the CNI binaries written to `/host/opt/cni/bin` followed by: + +```text +[ERROR] cni-installer/token_watch.go 108: Unable to create token for CNI kubeconfig error=Post ".../api/v1/namespaces/kube-system/serviceaccounts/calico-cni-plugin/token": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time is before +[FATAL] cni-installer/install.go 499: ... +``` + +The node clock was behind the start of the certificate's validity, so `install-cni` failed TLS verification when requesting the `calico-cni-plugin` token, and the Pod network could not be set up. + +### Cause 4: x509: certificate signed by unknown authority from the Calico plugin after a full cluster restart + +After every node was restarted, the Pods on one node stayed in `ContainerCreating` with: + +```text +Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "": plugin type="calico" failed (add): error getting ClusterInformation: Get "https://:443/apis/crd.projectcalico.org/v1/clusterinformations/default": tls: failed to verify certificate: x509: certificate signed by unknown authority +``` + +followed by `Pod sandbox changed, it will be killed and re-created.` Every `calico-node` Pod was `1/1 Running` with routine logs, restarting `calico-node` did not help, and the node clock was correct. The Calico CNI plugin on that node could not verify the API server certificate; the exact mechanism was not determined. Restarting the node restored its Pods. This differs from Cause 3, where the message says the certificate `is not yet valid`. + +## Resolution + +### Cause 1: calico-node: failed to connect kubernetes svc:443, and the dummy kernel module is missing + +Copy the `dummy` module from a node that runs exactly the same kernel version, rebuild the module index, load it, and recreate `calico-node`. + +> **Warning:** A module built for another kernel can fail to load or destabilize the host. Compare `uname -r` on both nodes first. Roll back by removing the copied file and running `depmod -a` again. + +```bash +uname -r +lsmod | grep dummy || modinfo dummy +# on a healthy node with the same kernel: +find /lib/modules/$(uname -r) -name 'dummy.ko*' +# copy that file to the same path on the affected node, then on the affected node: +depmod -a +modprobe dummy +kubectl -n kube-system delete pod +``` + +### Cause 2: calico-node image layers are corrupted after a VM live migration + +Stop kubelet, delete the local `calico-node` image, and start kubelet so a clean copy is pulled. + +> **Warning:** With kubelet stopped, the node does not manage Pods. The pull after the delete fails if the node cannot reach the registry, so check registry access first. The delete cannot be undone; the pull replaces the image. + +```bash +systemctl stop kubelet +crictl images | grep calico/node +crictl rmi +systemctl start kubelet +``` + +On a Docker node use `docker images` and `docker rmi -f` instead of `crictl`. + +### Cause 3: node clock out of sync, install-cni fails x509 verification + +Compare the clock with a healthy node and synchronize it from the site time source; restarting Calico before that does not help. Then retry the Pods. + +```bash +date -u; timedatectl +chronyc tracking || true +chronyc makestep +``` + +### Cause 4: x509: certificate signed by unknown authority from the Calico plugin after a full cluster restart + +Rule out the clock first (Cause 3), then restart the affected node. + +> **Warning:** Restarting the node stops every Pod on it. Drain it first and restart one node at a time; uncordon it when it is back. Nothing needs to be rolled back. + +```bash +kubectl drain --ignore-daemonsets --delete-emptydir-data +# on the node: +systemctl reboot +# from a master, once the node is Ready: +kubectl uncordon +``` + +## Diagnostic Steps + +1. Read the Pod events and the kubelet log on the affected node: + + ```bash + kubectl describe pod -n | grep -A3 FailedCreatePodSandBox + journalctl -u kubelet --since '-30m' | grep -i calico | tail -n 20 + kubectl get pods -A -o wide | grep -E 'calico|ContainerCreating' + ``` + +2. Read the `install-cni` log when `calico-node` is not starting: + + ```bash + kubectl -n kube-system logs -c install-cni --previous | grep -E 'x509|is before|token' + ``` + +3. Check the node properties: + + ```bash + lsmod | grep dummy + date -u; timedatectl + ``` + +- `failed to connect kubernetes svc:443` with no `dummy` module: Cause 1. +- A failure that starts right after a VM live migration: Cause 2. +- `x509: certificate has expired or is not yet valid: current time ... is before ...` from `install-cni`, with a node clock behind the other nodes: Cause 3. +- `x509: certificate signed by unknown authority` from the Calico plugin after a full cluster restart, with a correct clock: Cause 4. From f62daf83e8f181c9e248b929e12e93b2f78d4002 Mon Sep 17 00:00:00 2001 From: Komh Date: Wed, 18 Nov 2026 17:41:01 +0800 Subject: [PATCH 16/25] docs(solutions): add 6 and update 2 troubleshooting articles --- ..._delete_hangs_because_of_captain_on_ACP.md | 109 ++++++++++ ...r_a_newly_uploaded_chart_version_on_ACP.md | 43 +++- ...wrong_externalURL_ALB_and_Envoy_Gateway.md | 189 ++++++++++++++++ ...ted_with_too_large_on_a_NodePort_Harbor.md | 118 ++++++++++ ...awaiting_response_headers_or_408_on_ACP.md | 191 +++++++++++++++++ ..._or_the_browser_is_older_than_Chrome_98.md | 105 +++++++++ ...ils_or_creates_the_wrong_objects_on_ACP.md | 201 ++++++++++++++++++ ...times_out_with_proxyconnect_i_o_timeout.md | 65 ++++++ 8 files changed, 1020 insertions(+), 1 deletion(-) create mode 100644 docs/en/solutions/Harbor_access_address_and_exposure_HA_needs_Ingress_and_a_domain_wrong_externalURL_ALB_and_Envoy_Gateway.md create mode 100644 docs/en/solutions/Harbor_image_push_fails_while_harbor-nginx_restarts_or_is_rejected_with_too_large_on_a_NodePort_Harbor.md create mode 100644 docs/en/solutions/Harbor_push_of_large_image_layers_fails_with_unexpected_EOF_timeout_awaiting_response_headers_or_408_on_ACP.md create mode 100644 docs/en/solutions/Harbor_web_page_keeps_loading_because_chartmuseum_cannot_reach_its_NAS_storage_or_the_browser_is_older_than_Chrome_98.md create mode 100644 docs/en/solutions/Helm_chart_deployment_from_Catalog_fails_or_creates_the_wrong_objects_on_ACP.md create mode 100644 docs/en/solutions/docker_push_to_Harbor_from_Docker_Desktop_times_out_with_proxyconnect_i_o_timeout.md diff --git a/docs/en/solutions/Application_updates_are_slow_status_stays_Failed_or_Processing_or_delete_hangs_because_of_captain_on_ACP.md b/docs/en/solutions/Application_updates_are_slow_status_stays_Failed_or_Processing_or_delete_hangs_because_of_captain_on_ACP.md index e60953923..f0421382e 100644 --- a/docs/en/solutions/Application_updates_are_slow_status_stays_Failed_or_Processing_or_delete_hangs_because_of_captain_on_ACP.md +++ b/docs/en/solutions/Application_updates_are_slow_status_stays_Failed_or_Processing_or_delete_hangs_because_of_captain_on_ACP.md @@ -4,6 +4,7 @@ kind: products: - Alauda Container Platform ProductsVersion: + - "3.12" - "3.16" - "3.18" - "4.0" @@ -24,6 +25,11 @@ Native applications and Helm chart applications are reconciled by `captain` (`ca - A pipeline-published native application stays on `处理中(1/1)` (Processing). - Deleting a Helm application stays on `删除中` (Deleting) with `处理中(0/0)` for hours. - `captain` crash-loops and application updates fail. +- A Helm application stays `删除中` (Deleting) while its workloads are still `Running`, so a redeploy cannot start. +- The native application page shows no compute components (`0/0`) while its Deployments are `Running`. +- An application exported to a chart and imported again stays processing (`0/0`) with no error and no event. +- A manual edit of an object rendered by the chart disappears at the next sync. +- The application history shows a new revision that nobody created. The `captain` log often contains `Unauthorized, requeuing`, `failed to list *v1.HelmRequest: Unauthorized`, or a `nil pointer dereference` panic in the `chartrepo` controller. @@ -78,6 +84,32 @@ Observed a panic in reconciler: runtime error: invalid memory address or nil poi The stack runs through `repo.ChartVersions.Less`, `IndexFile.SortEntries`, `loadIndex`, and `(*ChartRepoReconciler).syncCharts`: `captain` panics while sorting that repository's index entries. Removing the repository from the platform returns `captain` to `Running`. +### Cause 8: an expired captain certificate leaves a delete stuck while the workloads still run + +The Application keeps phase `Succeeded` with `metadata.deletionTimestamp` set, and the matching HelmRequest is `phase: Synced` with a `deletionTimestamp`. The workloads are still `Running` because the uninstall never ran. `captain` logs `failed to list *v1.HelmRequest: Unauthorized` and `failed to watch *v1.HelmRequest: failed to list *v1.HelmRequest: Unauthorized`. The certificate `captain` uses expired and was not rotated. This is a product defect fixed in ACP 4.2.5; a `captain` that still holds the expired certificate, including on 4.2.5, needs a restart of `captain-controller-manager` and `metis`, after which the delete finishes. Unlike Cause 6, the workloads still exist and the finalizer is not removed by hand. + +### Cause 9: no compute components because metis cannot impersonate groups + +The console asks `metis` (Deployment `metis` in `cpaas-system`) to draw native-application components. When the ClusterRole bound to `system:serviceaccount:cpaas-system:metis` allows impersonating `users` but not `groups`, the page shows `0/0` while the Deployments are `Running`, and the application has an event: + +```text +Failed: groups "" is forbidden: User "system:serviceaccount:cpaas-system:metis" cannot impersonate resource "groups" in API group "" at the cluster scope +``` + +This gap exists on ACP 4.2.4. Adding the missing impersonation rule is the fix. + +### Cause 10: a re-imported chart application stays processing because the Harbor registry is unhealthy + +After an application is exported to a chart and imported again, it stays processing (`0/0`), `spec.assemblyPhase` stays `pending`, and there are no events. The ChartRepo can still be `Synced` while the Harbor `registry` component is unhealthy: the status sync of the chart-backed application times out talking to that registry. Repairing the Harbor registry ends the processing state; setting `spec.assemblyPhase` by hand (Cause 3) is not the repair. + +### Cause 11: manual edits are reverted when the HelmRequest syncs + +Objects changed with `kubectl edit` or in the console are put back to the rendered chart at the next HelmRequest sync. Persistent changes belong in HelmRequest `spec.values` or the Application annotation `app.cpaas.io/chart.values`. + +### Cause 12: a captain restart re-applies applications and adds a revision nobody made + +On ACP 3.12, when `captain` restarts, its check of whether a HelmRequest changed is faulty: it resyncs and re-applies the application's resources, which writes a new revision to the application history without any user action. The Kubernetes audit log shows a `patch` of the application's objects (for example a Service) with `fieldManager=manager`, user agent `Go-http-client/2.0`, by a platform service account such as `system:serviceaccount:kube-system:admin`, setting `app.kubernetes.io/managed-by: Helm`; the objects carry `meta.helm.sh/release-name`. This is a product defect fixed in ACP 3.15 and later. + ## Resolution ### Cause 1: one HelmRequest worker per cluster is blocked by failing chart applications @@ -199,6 +231,78 @@ Retry the Helm update after the new Pod is `Running`. Upgrade to a version that 3. `captain` stays `Running`. +### Cause 8: an expired captain certificate leaves a delete stuck while the workloads still run + +1. Confirm both the `deletionTimestamp` and the `Unauthorized` list/watch error (Diagnostic Steps), and record the images: + + ```bash + kubectl -n cpaas-system get deploy captain-controller-manager metis \ + -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.template.spec.containers[0].image}{"\n"}{end}' + ``` + +2. Restart both components. Below 4.2.5, also upgrade to 4.2.5 or later to get the fix. + + > **Warning:** Only when the `Unauthorized` error and a `deletionTimestamp` are both present. Impact: HelmRequest reconciliation and native-application status pause until both Deployments are `Ready`. The restart reloads access state and does not delete the Application. Rollback: `kubectl -n cpaas-system rollout undo deployment/` for each. + + ```bash + kubectl -n cpaas-system rollout restart deployment/captain-controller-manager deployment/metis + kubectl -n cpaas-system rollout status deployment/captain-controller-manager + kubectl -n cpaas-system rollout status deployment/metis + ``` + +3. The uninstall runs and the Application leaves the console. + +### Cause 9: no compute components because metis cannot impersonate groups + +1. Check the permission and find the ClusterRole bound to metis. `no` plus the event text confirms this cause: + + ```bash + kubectl auth can-i impersonate groups --as=system:serviceaccount:cpaas-system:metis + kubectl get clusterrolebinding -o jsonpath='{range .items[*]}{.roleRef.name}{"\t"}{range .subjects[*]}{.namespace}{"/"}{.name}{" "}{end}{"\n"}{end}' | grep cpaas-system/metis + ``` + +2. Add this rule to that ClusterRole: + + > **Warning:** The rule lets metis act as the groups named in user tokens. Add only this rule, to the ClusterRole already bound to metis; do not grant `impersonate` on `*`. Precondition: `kubectl get clusterrole -o yaml > metis-role.yaml`. Rollback: `kubectl apply -f metis-role.yaml`. + + ```yaml + - apiGroups: [""] + resources: ["groups"] + verbs: ["impersonate"] + ``` + +3. Run the `auth can-i` command again until it prints `yes`, then reopen the application; the Running Deployments are listed as compute components. + +### Cause 10: a re-imported chart application stays processing because the Harbor registry is unhealthy + +Repair the Harbor `registry` component until its Pods are `Ready` and Harbor reports the registry healthy, then retry the import or the HelmRequest sync. A ChartRepo in `Synced` is not evidence that the registry is healthy. + +```bash +kubectl get application -n -o jsonpath='{.spec.assemblyPhase}{"\n"}' +kubectl get pods -A | grep harbor-registry +``` + +### Cause 11: manual edits are reverted when the HelmRequest syncs + +Put the change in HelmRequest `spec.values` (or the `app.cpaas.io/chart.values` annotation of the Application) so the next sync renders it. Repeating the edit on the Deployment does not stick. + +```bash +kubectl get helmrequest -n -o jsonpath='{.spec.values}{"\n"}' +kubectl edit helmrequest -n +``` + +### Cause 12: a captain restart re-applies applications and adds a revision nobody made + +1. Compare the time of the unexplained revision with the start time of the `captain` Pod: + + ```bash + kubectl get pods -n cpaas-system | grep captain + kubectl get pod -n cpaas-system -o jsonpath='{.status.startTime}{" "}{.status.containerStatuses[*].lastState}{"\n"}' + ``` + +2. When they match and the audit log shows the `patch` described in Root Cause, the revision re-applied the application's own chart and needs no rollback. Hand edits made outside the chart are lost (Cause 11). +3. Upgrade to ACP 3.15 or later to stop restarts from creating revisions. + ## Diagnostic Steps ```bash @@ -217,4 +321,9 @@ pipeline application 处理中, spec.assemblyPhase Pending Helm update stays Failed after a permission error, admin retry fails -> Cause 5 Helm delete stuck, workloads gone, error syncing Unauthorized -> Cause 6 captain crash-loop, chartrepo panic in SortEntries -> Cause 7 +Deleting, workloads still Running, deletionTimestamp, list HelmRequest Unauthorized -> Cause 8 +0/0 components, event: metis cannot impersonate resource "groups" -> Cause 9 +re-imported chart app processing, assemblyPhase pending, Harbor registry down -> Cause 10 +objects revert after a sync -> Cause 11 +unexplained revision at a captain restart, audit patch by fieldManager=manager -> Cause 12 ``` diff --git a/docs/en/solutions/Catalog_does_not_list_a_Helm_chart_that_is_already_in_Harbor_or_a_newly_uploaded_chart_version_on_ACP.md b/docs/en/solutions/Catalog_does_not_list_a_Helm_chart_that_is_already_in_Harbor_or_a_newly_uploaded_chart_version_on_ACP.md index a398182fc..7a1409e28 100644 --- a/docs/en/solutions/Catalog_does_not_list_a_Helm_chart_that_is_already_in_Harbor_or_a_newly_uploaded_chart_version_on_ACP.md +++ b/docs/en/solutions/Catalog_does_not_list_a_Helm_chart_that_is_already_in_Harbor_or_a_newly_uploaded_chart_version_on_ACP.md @@ -4,6 +4,7 @@ kind: products: - Alauda Container Platform ProductsVersion: + - "4.0" - "4.2" --- @@ -16,6 +17,7 @@ A Helm chart is in Harbor, or an upload reported success, but Catalog, the versi - `helm push` printed `Pushed: //:`, Harbor shows the chart, but the template repository and Catalog stay on the old list or show nothing. - A newly added chart repository lists none of the charts already stored in it. The repository detail page says `无应用模板` ("no application templates") or `暂时没有找到符合过滤条件的应用模板` ("no application templates match the filter"). - A new chart package was uploaded (for example with `violet push`), but the template version dropdown still offers only the previous version, and the repository (for example `public-charts`) shows last sync time `-`. +- Upload reported `上传成功` (upload successful) and Harbor has the tag, but with a namespace of an ARM cluster selected, Catalog shows `共 0 个应用` (0 applications), `Helm Charts (0个)`, and `无应用列表` (no application list), while x86 clusters list the chart. Or the chart details open but the deploy form has no version to select. Or, after a version was deleted and the same version uploaded again, the whole chart disappears from Catalog, and creating it again reports that the version already exists. Restarting OLM Pods and refreshing the browser does not make the chart appear. `helm show chart` against the Harbor URL proves only that the registry has the package, not that Catalog has indexed it. @@ -41,6 +43,18 @@ After an OCI chart repository is added, the `ChartRepo` reaches phase `Synced`, The upload succeeded, the architecture of the package matches the cluster, but the repository index did not refresh: last sync time is `-`, and the template list and the version selector of the create form still offer only the previous version. Setting the repository status to `Pending` triggers a resynchronization, after which the new version is available. +### Cause 5: the chart's hardware architecture does not include the cluster's architecture + +Application templates support two hardware architectures, x86 and ARM, and default to x86. A cluster does not display chart packages whose architecture does not match its own: an x86-only template is listed on x86 clusters (and on C86 clusters), not on ARM clusters. Charts uploaded to the platform chart repository are marked x86 only by default. The chart details page under Platform Management shows the value as Hardware Architecture `x86`. + +The value is the `cpaas.io/arch` label of the platform `Chart` resource (`app.alauda.io/v1beta1`, in `cpaas-system` on the `global` cluster), taken from the package's `Chart.yaml`: `cpaas.io/arch: amd64` is stored as x86, `cpaas.io/arch: amd64.arm64` as x86 and ARM. Node architectures are reported as `amd64` / `arm64`; the template values are `x86` / `arm`. The mismatch shows up three ways: + +- The chart is not listed for a namespace of an ARM cluster. +- The chart details load on a cluster of another architecture, but the version picker is empty. On an ARM cluster, changing the version's label from `x86` to `arm` makes the versions selectable. +- A new upload of a package whose `cpaas.io/arch` is `amd64` only writes the label back to x86, even if the chart had been set to x86 and ARM. The chart then looks deleted from ARM clusters, although `kubectl get chart -A` still lists it (named `--.`) with the re-uploaded version. Deleting the version is not what removed it; the label written by the new package is. + +Restarting `captain` components does not make the chart appear. + ## Resolution ### Cause 1 and Cause 2 @@ -97,6 +111,31 @@ kubectl get artifact,artifactversion -n cpaas-system 4. After the repository leaves `Pending`, the uploaded version is available in the application templates. +### Cause 5 + +1. On the `global` cluster, confirm the chart resource exists and read its architecture; on the affected cluster, read the node architecture: + + ```bash + kubectl get chart -A | grep + kubectl get chart -n cpaas-system -o jsonpath='{.metadata.labels.cpaas\.io/arch}{"\n"}' + kubectl get nodes -o custom-columns=NAME:.metadata.name,ARCH:.status.nodeInfo.architecture + ``` + + A label without `arm64` on a chart that an ARM cluster cannot see confirms this cause. + +2. Change the hardware architecture in the UI: Platform Management > App Marketplace Management > Chart Repositories > `` > the chart, and edit Hardware Architecture to include ARM (x86 and ARM). On releases that name the menu App Store Management > Template Repository, the same control is on the application template details page. Keep x86 when x86 or C86 clusters must still see the chart. For an empty version picker on an ARM-only cluster, set the version's architecture to `arm`. + + > **Warning:** Changing the hardware architecture changes which clusters list this chart; a cluster whose architecture you remove stops showing it. Precondition: note the current value on the details page. Rollback: set that value again. No chart object or Harbor blob is deleted. + +3. Reopen Catalog with a namespace of the affected cluster; the chart and its versions are listed. + +4. Make it persistent: the next upload of a package with `cpaas.io/arch: amd64` overwrites the UI change. Set the annotation in `Chart.yaml`, repackage, and upload: + + ```yaml + annotations: + cpaas.io/arch: amd64.arm64 + ``` + ## Diagnostic Steps ```bash @@ -112,6 +151,8 @@ helm show chart --version | OCI repository phase `Synced`, `captain-chartmuseum` only logs `No change detected between cache and storage`, page says `无应用模板`, no `Artifact` pair in `cpaas-system` on `global` | Cause 3 | | Upload succeeded, version missing from the dropdown, last sync time `-`, architecture matches | Cause 4 | | `violet push` says `artifact already exist`, and the `cpaas.io/chart-purpose` labels differ | Reused Artifact (Cause 4, step 1) | -| The chart is hidden only in an ARM cluster namespace (`共 0 个应用`, "0 applications in total") | The chart's architecture label, not synchronization | +| The chart is hidden only in an ARM cluster namespace (`共 0 个应用`, "0 applications in total") | The chart's architecture label, not synchronization (Cause 5) | +| Chart details load, the version picker is empty, and the node architecture and the `x86` / `arm` label disagree | Cause 5 | +| The chart vanished from Catalog after a re-upload, `kubectl get chart -A` still lists it, its `cpaas.io/arch` label has no `arm64` | Cause 5 | On ACP 4.2, Cause 1 and Cause 2 can both apply: there is no background synchronization, and a Catalog upload is also the event that registers the chart. A synced phase together with a missing `Artifact` pair is Cause 3, not Cause 4. Restarting OLM Pods does not fix any of these causes. diff --git a/docs/en/solutions/Harbor_access_address_and_exposure_HA_needs_Ingress_and_a_domain_wrong_externalURL_ALB_and_Envoy_Gateway.md b/docs/en/solutions/Harbor_access_address_and_exposure_HA_needs_Ingress_and_a_domain_wrong_externalURL_ALB_and_Envoy_Gateway.md new file mode 100644 index 000000000..7dc133862 --- /dev/null +++ b/docs/en/solutions/Harbor_access_address_and_exposure_HA_needs_Ingress_and_a_domain_wrong_externalURL_ALB_and_Envoy_Gateway.md @@ -0,0 +1,189 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Harbor access address and exposure: HA needs Ingress and a domain, wrong externalURL, ALB, and Envoy Gateway + +## Issue + +The Harbor instance installs, but the way it is reached is wrong or unclear: + +- The high-availability creation form demands a domain name. No domain is available, and NodePort, a raw IP, or DNS to a VIP that load-balances onto node NodePorts was expected to provide high availability. +- The Harbor Pods are `Running`, but the UI does not open on the name clients use. +- The Harbor toolchain integration and the clients use an `IP:Port` address (for example a hardware load balancer in front of a NodePort) and have to move to a domain name. +- After a platform upgrade, Harbor has to be served through ALB, and it is not clear which `expose` type and fields that needs. +- The Harbor entry has to move from ALB Ingress to Envoy Gateway. + +## Root Cause + +### Cause 1: high availability requires Ingress and a domain name; NodePort does not provide it + +The high-availability template of Alauda Build of Harbor uses an Ingress and a domain name, with the domain resolving to a VIP. NodePort exposure, including a YAML deployment whose expose type is `nodePort`, is the non-HA (single-point) form. A domain whose DNS points at a VIP that load-balances onto node NodePorts is not the supported HA exposure, and IP-only access is not supported for an HA instance. A single-point instance can still use `IP:NodePort` as its one access address. + +The domain has to exist before the HA deployment. A temporary domain is enough to finish the deployment and can be replaced later (Cause 2). + +The documented baseline for the HA template is: more than two cluster nodes, highly available Redis and PostgreSQL, and a highly available load balancer (with ALB, a VIP is part of it). Node-local storage is not the HA storage form; the StorageClass or PVC must be `ReadWriteMany`. + +### Cause 2: the UI does not open because the access address on the Harbor resource is wrong + +The Pods are `Running`, but the access address stored on the `Harbor` resource is not the name clients resolve, or does not resolve to the instance's Ingress. Harbor builds its links and token realm from that address, so the UI does not open on the client's name. The repair is to edit the access address on the Harbor resource, not the NodePort Service. + +The address is `spec.helmValues.externalURL`, and the Ingress host is `spec.helmValues.expose.ingress.hosts.core`, with `spec.helmValues.expose.type: ingress`. Some examples, such as the [Harbor disaster recovery procedure](How_to_perform_disaster_recovery_for_harbor.md), place `externalURL` directly under `spec`. Edit the field that exists on the live object, and keep the host identical in `externalURL` and in the Ingress host. + +### Cause 3: moving the toolchain integration and clients from `IP:Port` to a domain + +Under standard usage Harbor has a single access address, either `IP:NodePort` or a domain name, and a single expose type; the portal, push, and pull must all use that address. A second, hand-written Ingress next to a NodePort exposure is not a supported way to add a domain. Moving to a domain therefore means switching the instance to Ingress access with that domain (Causes 1 and 2), after which the old `IP:Port` entry stops serving Harbor. Everything that names the old address (the toolchain integration, image references, pull Secrets) has to move with it. + +### Cause 4: serving Harbor through ALB is the `ingress` expose type + +In the Alauda Build of Harbor 2.14 deployment documentation for the 4.3 platform, the complete example that combines high availability, a storage class, and Ingress network access is the ALB form. ALB access is the `ingress` expose type. Under `spec.helmValues` the example sets: + +- `expose.type: ingress` +- `expose.tls.enabled: false` +- `expose.ingress.hosts.core`: the Harbor domain, `harbor.example.com` in the example +- `externalURL`: the access address on the same domain, `http://harbor.example.com` in the example +- `existingSecretAdminPassword: harbor-password` and `existingSecretAdminPasswordKey: password` +- `persistence.enabled: true` + +The host in `externalURL` and in `expose.ingress.hosts.core` is the same value, the same rule as Cause 2. + +### Cause 5: Envoy Gateway exposure needs Harbor 2.14 + +Alauda Build of Harbor 2.14 supports Envoy Gateway network access: its documentation describes an Envoy Gateway template (`spec.helmValues.expose.type: route`) and a high-availability Envoy Gateway template. An older instance (for example 2.6.4) cannot switch its entry from ALB to Envoy Gateway directly; upgrade it to 2.14 first. + +## Resolution + +### Cause 1: high availability requires Ingress and a domain name; NodePort does not provide it + +1. Use the high-availability template with an Ingress and a domain name. If the formal domain is not ready, use a temporary domain that you control and that resolves to the VIP. +2. Check the baseline before deploying: more than two nodes, HA Redis and PostgreSQL, an HA load balancer, and `ReadWriteMany` storage. +3. After the temporary name works, replace it with the formal one using the edit in Cause 2. That edit is how the domain is changed; it is not a reinstall. + +A NodePort instance, with or without DNS and a VIP in front of it, stays the non-HA form. + +### Cause 2: the UI does not open because the access address on the Harbor resource is wrong + +1. Save the resource and find where the address is stored: + + ```bash + kubectl -n get harbor -o yaml > /tmp/harbor-before.yaml + grep -n -E 'externalURL|type: |core:' /tmp/harbor-before.yaml + nslookup + ``` + + `nslookup` must return the address of the instance's Ingress (the load balancer or VIP) before you continue. + +2. Set the live `externalURL` and `spec.helmValues.expose.ingress.hosts.core` to the same host, with `https://` on the URL when TLS is enabled, and `spec.helmValues.expose.type` to `ingress`. + + > **Warning:** Changing the access address makes Harbor emit links and token realms for the new host. Impact: browsers and `docker login` that still use the old host fail until they switch. Precondition: a maintenance window, `/tmp/harbor-before.yaml` saved, and the new name resolving to this instance's Ingress. Rollback: `kubectl apply -f /tmp/harbor-before.yaml`. + + ```bash + kubectl -n edit harbor + ``` + +3. After the operator reconciles, open the UI on that host and log in. + +### Cause 3: moving the toolchain integration and clients from `IP:Port` to a domain + +1. Record what uses the old address before the window: + + ```bash + kubectl -n get harbor -o yaml > /tmp/harbor-before.yaml + grep -nE 'externalURL|type: nodePort|type: ingress|hosts:|core:' /tmp/harbor-before.yaml + kubectl -n get svc,ingress + kubectl get secret -A --field-selector type=kubernetes.io/dockerconfigjson -o name + nslookup + ``` + + The domain must resolve to the load balancer address that will front the Harbor Ingress. The Secret list shows the pull Secrets that may need to be recreated for the new host. + +2. Switch the instance to Ingress access on the domain: the HA template from Cause 1 when the instance must be highly available, otherwise the edit in Cause 2 on the existing instance (setting `expose.type` from `nodePort` to `ingress`). Use the backup, Warning, and rollback of Cause 2. + +3. Edit the Harbor tool integration so that its access address and its API address both use the new domain. + +4. Change image references and pull Secrets that name the old `IP:Port` to the domain, in the same window: once `expose.type` is `ingress`, the old address no longer serves Harbor. Test `docker login ` and one pull. + +### Cause 4: serving Harbor through ALB is the `ingress` expose type + +1. Read the current expose block and confirm that the domain resolves to the ALB address (the VIP when ALB is the HA load balancer): + + ```bash + kubectl -n get harbor -o yaml > /tmp/harbor-before.yaml + grep -n -E 'externalURL|expose:|type: |hosts:|core:|tls:' /tmp/harbor-before.yaml + kubectl get ingressclass + nslookup + ``` + +2. Merge this block into the existing `spec.helmValues`, keeping the database, Redis, and persistence keys the live object already has: + + ```yaml + spec: + helmValues: + expose: + type: ingress + tls: + enabled: false + ingress: + hosts: + core: + externalURL: http:// + existingSecretAdminPassword: harbor-password + existingSecretAdminPasswordKey: password + persistence: + enabled: true + ``` + + `harbor-password` is the Secret name of the example; use the admin-password Secret the live object already names, or create it before a new Harbor resource. With TLS enabled, `externalURL` starts with `https://`. + + > **Warning:** Changing `expose.type` on a running instance (for example from `nodePort` to `ingress`) replaces the way Harbor is reached. Impact: the old `IP:Port` entry stops serving Harbor, and clients, pull Secrets, and the toolchain integration that use it fail until they move to the domain. Precondition: the domain resolves to the ALB address, `/tmp/harbor-before.yaml` is saved, and the change is made in a maintenance window. Rollback: `kubectl apply -f /tmp/harbor-before.yaml`. + + ```bash + kubectl -n edit harbor + ``` + +3. After the operator reconciles, an Ingress for the domain exists in the instance namespace. Open the UI on the domain, run `docker login `, and pull one known tag. Move the integration and pull Secrets as in Cause 3. + +### Cause 5: Envoy Gateway exposure needs Harbor 2.14 + +1. Read the Harbor version: + + ```bash + kubectl get harbor -A + kubectl get csv -A | grep -i harbor + ``` + +2. Below 2.14, upgrade Harbor to 2.14 first, following the upgrade documentation for your Harbor version. +3. On 2.14, follow the Envoy Gateway network access section of the Alauda Build of Harbor 2.14 deployment documentation. Its template sets `spec.helmValues.expose.type: route`; take the other fields from that document. +4. For a high-availability instance, use the HA Envoy Gateway template from the same documentation and test it on the target cluster before the production change. + +Changing `expose.type` replaces the Harbor entry; use the backup, Warning, and client checks of Cause 4. + +## Diagnostic Steps + +1. Read the expose block and the access address on the live object: + + ```bash + kubectl get harbor -A + kubectl -n get harbor -o yaml | grep -n -E 'externalURL|expose:|type: |core:' + kubectl -n get svc,ingress + ``` + +2. Match the request: + + | Situation | Cause | + | --- | --- | + | New HA instance, the form does not continue without a domain | Cause 1 | + | Instance `Running`, only the URL does not open; the host in `externalURL` differs from what clients resolve | Cause 2 | + | Healthy instance, integration and clients must move from `IP:Port` to a domain | Cause 3 | + | Which `expose` values serve Harbor through ALB | Cause 4 | + | Move the entry from ALB Ingress to Envoy Gateway | Cause 5; check the version first | + +3. Before any edit, confirm where `externalURL` sits on the live YAML (`spec.helmValues` or `spec`) and that `nslookup ` returns the load balancer address. + +An instance that never finishes installing (PostgreSQL or Redis failures) is not covered here; see [harbor-core cannot connect to Redis](harbor-core_cannot_connect_to_Redis_NOAUTH_from_Sentinel_or_ERR_invalid_password_after_a_Redis_password_change.md) and [Harbor core service unavailable or internal server error, stuck instance status, or Pods cycling after an edit](Harbor_core_service_unavailable_or_internal_server_error_stuck_instance_status_or_Pods_cycling_after_an_edit.md). For a toolchain integration that fails after the address is correct, see [Harbor toolchain integration fails](Harbor_toolchain_integration_fails_images_not_selectable_katanomi-plugin_timeout_or_push_asks_for_credentials.md). diff --git a/docs/en/solutions/Harbor_image_push_fails_while_harbor-nginx_restarts_or_is_rejected_with_too_large_on_a_NodePort_Harbor.md b/docs/en/solutions/Harbor_image_push_fails_while_harbor-nginx_restarts_or_is_rejected_with_too_large_on_a_NodePort_Harbor.md new file mode 100644 index 000000000..0d903ff35 --- /dev/null +++ b/docs/en/solutions/Harbor_image_push_fails_while_harbor-nginx_restarts_or_is_rejected_with_too_large_on_a_NodePort_Harbor.md @@ -0,0 +1,118 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Harbor image push fails while harbor-nginx restarts, or is rejected with too large on a NodePort Harbor + +## Issue + +An image push to Harbor fails, and Harbor looks abnormal, in one of two ways: + +- The push fails during the upload of many layers at once, and the Harbor nginx Pod (`-nginx`, also seen as `harbor-nginx`) has restarted during the push. Its events show `Liveness probe failed`, or the Pod keeps restarting because it does not have enough CPU. Every upload in progress is dropped at each restart. +- The push fails with a response body that says `too large`, and the question is how to raise the Harbor upload size limit. Harbor is exposed by NodePort, the push target is a node address, and the push comes from Docker on a machine outside the platform. + +Related articles: [Harbor push of large image layers fails with unexpected EOF, timeout awaiting response headers, or 408 on ACP](Harbor_push_of_large_image_layers_fails_with_unexpected_EOF_timeout_awaiting_response_headers_or_408_on_ACP.md) and [docker push to Harbor from Docker Desktop times out with proxyconnect i/o timeout](docker_push_to_Harbor_from_Docker_Desktop_times_out_with_proxyconnect_i_o_timeout.md). + +## Root Cause + +### Cause 1: the Harbor nginx Pod is killed during the push because it lacks memory or CPU + +The client pushes many blobs concurrently. nginx proxies them to Harbor Core, and Core proxies them to Harbor Registry. When the upload initialization requests (`POST /v2///blobs/uploads/`) take several seconds each, the nginx workers are blocked and cannot answer the liveness probe, which has a 1-second timeout. Kubernetes kills and restarts the nginx container repeatedly, which drops every upload in progress. nginx memory use above its request (for example 184 Mi used against a 128 Mi request and a 256 Mi limit) goes with this pattern. Increasing the memory of the nginx component restores image upload. + +The same failure appears when the nginx Pod restarts because it does not have enough CPU. Increasing the nginx CPU and memory in the Harbor instance makes image push work again. + +### Cause 2: `too large` from an old Docker client pushing to a node address + +The push target is a Kubernetes node address because Harbor is exposed by NodePort, and no load balancer or gateway sits in the path. The push comes from Docker on an external machine. The Docker client version on that machine is too old, and the `too large` error comes from that client, not from a Harbor size limit. The same images (about 300 MB, and about 1 GB) push successfully with containerd (`ctr`) from a platform node, with no Harbor limit changed. + +When a Registry Gateway is in the push path, `too large` is that gateway's size policy instead; see [Restricting Docker image push size and tag count using the Registry Gateway plugin](Restricting_Docker_Image_Push_Size_and_Tag_Count_using_the_Registry_Gateway_Plugin.md). Do not disable the gateway to get a large image through. + +## Resolution + +### Cause 1: the Harbor nginx Pod is killed during the push because it lacks memory or CPU + +1. Confirm the restarts and their reason, and compare CPU and memory use with the request and limit: + + ```bash + kubectl get pods -n | grep -i nginx + kubectl describe pod -n | grep -iE 'Restart Count|Last State|Reason|Liveness|Requests|Limits|cpu|memory' + kubectl top pod -n + ``` + + Continue only when the restarts line up with the failed pushes and the events show `Liveness probe failed`, or the CPU use is at the limit at the restart times. + +2. Raise the nginx resources on the Harbor resource that the operator reconciles, not on the generated Deployment. Raise memory for a liveness-probe kill; raise CPU and memory together when CPU is short. In the Harbor chart values the nginx resources are under `nginx.resources`; check what `spec.helmValues` already contains before adding them, and size the values above what `kubectl top` showed. Example: + + ```yaml + spec: + helmValues: + nginx: + resources: + requests: + cpu: + memory: 256Mi + limits: + cpu: + memory: 512Mi + ``` + + > **Warning:** Impact: saving the Harbor resource rolls the Harbor Pods it changes, and pushes and pulls through nginx are interrupted until the new nginx Pod is `Ready`. Precondition: the restart signature in step 1 is confirmed, a maintenance window, and the current resource saved by the first command below. Rollback: `kubectl apply -f /tmp/harbor-before-nginx.yaml`. + + ```bash + kubectl get harbor -n -o yaml > /tmp/harbor-before-nginx.yaml + kubectl edit harbor -n + kubectl get deploy -n -nginx -o jsonpath='{.spec.template.spec.containers[0].resources}{"\n"}' + ``` + +3. The last command must show the new values. If the Deployment does not change, the operator did not take the field; the Deployment-level alternative is described in [harbor-nginx or portal does not start or keeps restarting](harbor-nginx_or_portal_does_not_start_or_keeps_restarting_IP_address_family_unsupported_or_OOMKilled.md) (Cause 2). + +4. Repeat the push that failed and confirm that the nginx restart count no longer grows. + +### Cause 2: `too large` from an old Docker client pushing to a node address + +1. Confirm that no load balancer or Registry Gateway is in front of Harbor, that the push target is a node address and NodePort, and that the push comes from a machine outside the platform. Read the client version: + + ```bash + docker version + ``` + +2. Upgrade Docker on that machine. Compare its version with a machine that can push the same image to choose the target version. + +3. Until the client is upgraded, push the image from a platform node that runs containerd: + + ```bash + docker save ://: -o /tmp/-.tar + # copy the tar to the platform node, then on that node: + ctr -n k8s.io image import /tmp/-.tar + ctr -n k8s.io image push -u :'' ://: + ``` + + Add `--plain-http` to `ctr image push` for an HTTP Harbor, or `--skip-verify` for an HTTPS Harbor whose CA the node does not trust; `ctr` does not read `/etc/containerd/certs.d` by default. No Harbor upload size limit needs to change. + +## Diagnostic Steps + +1. Check whether nginx restarted during the push: + + ```bash + kubectl get pods -n -o wide | grep -iE 'nginx|core|registry' + kubectl get events -n --field-selector involvedObject.name= + ``` + + A restart count that grows at the time of the failed push, with `Liveness probe failed` events or CPU at its limit, is Cause 1, whatever the layer size. `Last State: Terminated, Reason: OOMKilled` without a push in progress is covered by [harbor-nginx or portal does not start or keeps restarting](harbor-nginx_or_portal_does_not_start_or_keeps_restarting_IP_address_family_unsupported_or_OOMKilled.md). + +2. For `too large`, establish the push path: + + | What you see | Conclusion | + | --- | --- | + | A Registry Gateway is in the path | The gateway's size policy; follow the Registry Gateway article above | + | Target is `:`, no load balancer, push from an external machine's Docker, the same image pushes with `ctr` from a platform node | Cause 2 | + +3. A push that fails with `unexpected EOF`, `timeout awaiting response headers`, or an nginx `408` without nginx restarting belongs to the first related article above. diff --git a/docs/en/solutions/Harbor_push_of_large_image_layers_fails_with_unexpected_EOF_timeout_awaiting_response_headers_or_408_on_ACP.md b/docs/en/solutions/Harbor_push_of_large_image_layers_fails_with_unexpected_EOF_timeout_awaiting_response_headers_or_408_on_ACP.md new file mode 100644 index 000000000..55c5de2e5 --- /dev/null +++ b/docs/en/solutions/Harbor_push_of_large_image_layers_fails_with_unexpected_EOF_timeout_awaiting_response_headers_or_408_on_ACP.md @@ -0,0 +1,191 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Harbor push of large image layers fails with unexpected EOF, timeout awaiting response headers, or 408 on ACP + +## Issue + +Small layers of an image push to Harbor (or report `Layer already exists`), but a large layer gets part way through the blob upload and then the connection closes or the commit never returns. Typical client output: + +- `unexpected EOF` once a layer passes about 150 MB. +- `failed commit on ref "layer-sha256:": failed to do request: Put ".../v2///blobs/uploads/...": EOF`, or `failed commit on ref` with `unexpected status` from the blob `PUT`, typically for multi-gigabyte layers. +- `timeout awaiting response headers` while a large layer is being committed. +- Images larger than about 1 GB fail during transfer to a Harbor exposed by NodePort. + +`failed commit on ref` with `unexpected status` from the blob `PUT` means the registry reports that the upload did not finish. Find out where the connection was cut (client path, node agent, NodePort) before raising any Harbor limit. + +Related articles: [Harbor image push fails while harbor-nginx restarts, or is rejected with too large on a NodePort Harbor](Harbor_image_push_fails_while_harbor-nginx_restarts_or_is_rejected_with_too_large_on_a_NodePort_Harbor.md) and [docker push to Harbor from Docker Desktop times out with proxyconnect i/o timeout](docker_push_to_Harbor_from_Docker_Desktop_times_out_with_proxyconnect_i_o_timeout.md). + +## Root Cause + +### Cause 1: a security agent on the Harbor nodes cuts uploads at about 150 MB + +`unexpected EOF` appears while the blob upload is still below the size of images that otherwise push, commonly at about 150 MB. The registry logs show that the client connection dropped, and the upload URL then returns 404 or 500 on retry. Moving the client to NodePort does not help, because the connection is closed on the Harbor node and not by an ingress timeout. A host security agent such as Trend Micro Deep Security Agent (`dsa_agent` / `ds_agent`) on the nodes running Harbor, with a policy that inspects the upload, closes it. Relaxing that policy lets the same layer commit. + +### Cause 2: the VPN or workstation path drops long uploads + +Pushes from a workstation (for example Docker Desktop on Windows or macOS) through a VPN fail on large layers: smaller layers report `Layer already exists` or push, multi-gigabyte layers stall in `Pushing` and the final `PUT` ends with `unexpected EOF`, `EOF`, or `failed commit on ref ... unexpected status`. The `registry` container of the Harbor registry Pod logs: + +```text +level=error msg="client disconnected during blob PUT" ... error="unexpected EOF" contentLength=241553920 copied=6859330 +msg="response completed with error" ... http.request.method=PUT ... http.response.status=500 +http: superfluous response.WriteHeader call +``` + +The `500` on the upload URL follows the client disconnect; it is not a registry storage error. Harbor itself is healthy: pushing the same image from a node inside the cluster, or on the Harbor server side, succeeds. The connection is dropped on the client path, most often by a rate or connection-time limit of the VPN, or by a timeout of the local Docker client. Giving Docker Desktop more memory does not help. The exact mechanism differs per network and is usually not visible from Harbor. + +### Cause 3: `timeout awaiting response headers` on a large layer commit + +The client sends the layer and then waits for the registry's response until its header timeout expires. Repeating the same push from a Linux Docker host on a network that reaches Harbor directly succeeds, so the failure belongs to the original client path. A wrong image name, node time skew, and the volume type of `harbor-registry` do not explain this pattern. A Linux retry that succeeds is a client-path recovery, not evidence of a corrupt blob store. + +### Cause 4: images larger than about 1 GB fail against a NodePort Harbor with `408` and `upload resumed at wrong offset` + +Harbor is exposed through NodePort and the client pushes to `:`. The `nginx` container of `harbor-nginx` logs the blob uploads ending with `408` after roughly 750 seconds: + +```text +"PUT /v2///blobs/uploads/?... HTTP/1.1" 408 0 ... "docker/ ..." +``` + +The `harbor-registry` log then rejects the resumed upload: + +```text +level=error msg="upload resumed at wrong offset: 626907975 != 0" +msg="response completed with error" err.code="blob upload invalid" err.message="blob upload invalid" http.response.status=404 +``` + +Concurrent large-layer uploads with the default MTU make the blob uploads time out through the NodePort, and the retried uploads then fail because the registry expects the upload to start over. Limiting the Docker client to one concurrent upload and setting its MTU to 1440 lets the large image push. Why exactly these two settings help is not established. + +## Resolution + +### Cause 1: a security agent on the Harbor nodes cuts uploads at about 150 MB + +1. On each node that runs Harbor Registry, check for the agent. A hit here plus a repeatable close near 150 MB is the signature: + + ```bash + ps -ef | grep -E 'dsa_agent|ds_agent' | grep -v grep + systemctl status ds_agent --no-pager + ``` + + If no agent is installed, this cause does not apply. + +2. In the security policy bound to those nodes, disable the intrusion-prevention or firewall rule that terminates the long upload, or exclude the Harbor registry port from that inspection. + + > **Warning:** Relaxing the agent policy stops that control from inspecting traffic on the Harbor nodes, which is a lasting security change if it stays. Impact: less inspection on those nodes. Precondition: a maintenance window, the policy name and previous setting recorded, and only the registry port excluded. Rollback: restore the recorded policy. + +3. Push one image with a layer larger than 150 MB from the same client that failed. If it commits, keep the port exclusion, or restore the policy and run Harbor on nodes the agent does not inspect. Do not delete partial blobs or run garbage collection as part of this test; an interrupted upload does not corrupt existing manifests. + +### Cause 2: the VPN or workstation path drops long uploads + +1. Confirm that the registry saw the client go away rather than a storage failure: + + ```bash + kubectl get pods -n | grep registry + kubectl logs -n -c registry --since=2h | grep -E 'client disconnected during blob PUT|response completed with error' + ``` + +2. Retry the identical push from a host that reaches Harbor directly, without the VPN. If it succeeds, Harbor needs no change; take the VPN rate or connection-time limit to whoever operates that network. + +3. To get the image in without the VPN path, save it, copy it to a node inside the cluster, and push it from there: + + ```bash + docker save ://: -o /tmp/-.tar + # copy the tar to a cluster node, then on that node: + ctr -n k8s.io image import /tmp/-.tar + ctr -n k8s.io image push -u :'' ://: + ``` + + Add `--plain-http` to `ctr image push` for an HTTP Harbor, or `--skip-verify` for an HTTPS Harbor whose CA the node does not trust; `ctr` does not read `/etc/containerd/certs.d` by default. Compare the size of the tar file at both ends of the copy; an interrupted copy produces a defective image. + +4. Lowering the client MTU (see Cause 4) can be tried as a mitigation on the VPN path; whether it helps depends on the VPN. + +### Cause 3: `timeout awaiting response headers` on a large layer commit + +1. Retry the push from a Linux Docker client on a network that reaches Harbor directly: + + ```bash + docker push ://: + ``` + +2. While it runs, confirm that the registry is not stuck on storage: + + ```bash + kubectl get pods -A | grep harbor-registry + kubectl describe pod -n + ``` + + If the events or volume status show the disk is full, free space or expand the volume before another multi-gigabyte attempt. Do not wipe the registry volume because a push timed out. + +### Cause 4: images larger than about 1 GB fail against a NodePort Harbor with `408` and `upload resumed at wrong offset` + +1. Confirm the signature in the Harbor logs for the time of the failed push. If `registry` is not the container name, list them with `kubectl get pod -n -o jsonpath='{.spec.containers[*].name}'`. + + ```bash + kubectl get deploy -n | grep -E 'nginx|registry' + kubectl logs -n deploy/ -c nginx --since=2h | grep -E 'PUT /v2/.*/blobs/uploads/.* 408 ' + kubectl logs -n -c registry --since=2h | grep -E 'upload resumed at wrong offset|blob upload invalid' + ``` + +2. Merge two settings into the client's existing Docker Engine configuration, keeping unrelated keys. The result must contain `"max-concurrent-uploads": 1` and `"mtu": 1440`, for example: + + ```json + { + "insecure-registries": ["192.0.2.10:"], + "max-concurrent-uploads": 1, + "mtu": 1440 + } + ``` + + On Docker Desktop the same JSON is edited under **Settings > Docker Engine**; **Apply & restart** restarts the engine. On a Linux host, edit `/etc/docker/daemon.json` and restart Docker. + + > **Warning:** Only when the NodePort signature in step 1 is confirmed. Impact: restarting Docker stops the containers that run under that `dockerd`; kubelet image pulls are not affected. Precondition: `/etc/docker/daemon.json` is copied aside. Rollback: restore the copy and restart Docker again. + + ```bash + cp -a /etc/docker/daemon.json /root/daemon.json.bak + # edit /etc/docker/daemon.json so it contains max-concurrent-uploads 1 and mtu 1440, then: + systemctl restart docker + docker push ://: + ``` + +## Diagnostic Steps + +1. Record the layer size, the bytes sent at failure, the elapsed time, the client OS and network path (VPN or direct), the destination host and port, and whether a small push succeeds. Do not loop a multi-gigabyte push while collecting this. + +2. Read the Harbor components and the registry log for the same timestamp: + + ```bash + kubectl get pods -n -o wide | grep -iE 'nginx|core|registry' + kubectl logs -n -c registry --tail=200 + ``` + + A client disconnect followed by 404 or 500 on the upload URL means the connection was cut outside the registry. A storage I/O or no-space error is a registry volume fault, not one of these causes. + +3. Classify: + + | What you see | Cause | + | --- | --- | + | `unexpected EOF` near 150 MB, `dsa_agent` / `ds_agent` on the Harbor nodes, NodePort fails the same way as the ingress | Cause 1 | + | `client disconnected during blob PUT` in the registry log; the push fails only from a VPN or workstation and succeeds from a cluster node | Cause 2 | + | `timeout awaiting response headers` on commit; a Linux client on a direct network succeeds | Cause 3 | + | NodePort Harbor, images over about 1 GB, nginx `408` on the blob `PUT`, then `upload resumed at wrong offset` | Cause 4 | + | `harbor-nginx` restarted during the push, or the body says `too large` | First related article above | + | `proxyconnect tcp: ... i/o timeout` before any layer uploads | Second related article above | + +4. After the fix, confirm the manifest reads back, using the scheme this Harbor serves: + + ```bash + curl -sS -o /dev/null -D - -u : \ + https://:/v2///manifests/ + ``` + + An HTTP `200`, and a later pull of the same tag, confirm the push. A `401` here is a credential problem, not a short upload. + +A pull-side `unexpected EOF` on a single node is a different problem; see [Intermittent image pull io timeout from a single ACP node](Intermittent_image_pull_io_timeout_from_a_single_ACP_node.md). A push that never starts a blob and fails with `x509` or `HTTP response to HTTPS client` is covered in [Harbor push or pull fails because the client uses the wrong scheme](Harbor_push_or_pull_fails_because_the_client_uses_the_wrong_scheme_HTTP_response_to_HTTPS_client_or_port_443_refused.md). diff --git a/docs/en/solutions/Harbor_web_page_keeps_loading_because_chartmuseum_cannot_reach_its_NAS_storage_or_the_browser_is_older_than_Chrome_98.md b/docs/en/solutions/Harbor_web_page_keeps_loading_because_chartmuseum_cannot_reach_its_NAS_storage_or_the_browser_is_older_than_Chrome_98.md new file mode 100644 index 000000000..71315845d --- /dev/null +++ b/docs/en/solutions/Harbor_web_page_keeps_loading_because_chartmuseum_cannot_reach_its_NAS_storage_or_the_browser_is_older_than_Chrome_98.md @@ -0,0 +1,105 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Harbor web page keeps loading because chartmuseum cannot reach its NAS storage, or the browser is older than Chrome 98 + +## Issue + +The Harbor web UI shows a loading spinner that never finishes, while every Harbor Pod is `Running` and none is in `CrashLoopBackOff`. Harbor looks "slow" rather than down. There are two variants: + +- Login works, but opening a project page keeps loading for a very long time. +- The Harbor page stays on loading and never redirects to the login page. The API requests the page sends, as seen in the browser developer tools, return no obvious error. + +## Root Cause + +### Cause 1: chartmuseum requests to NAS storage time out from a node with a network fault + +The `harbor-chartmuseum` Pod is `Running` on a node that has a network fault, and its requests to the NAS (NFS) storage behind its volume time out. A chartmuseum Pod that cannot reach its storage holds the project page in a loading state while the rest of Harbor stays healthy. Moving the Pod to a node with a working network restores the page. + +This applies only to Harbor deployments that still include the chartmuseum component with its storage on NAS (NFS). Harbor releases that no longer ship chartmuseum are not affected by this cause. + +### Cause 2: the browser is older than Chrome 98 + +The Harbor UI requires Chrome 98 or later. In an older browser the page cannot render: it keeps loading, never reaches the login page, and the API calls behind it succeed. Updating the browser restores access; no change on the cluster is needed. + +## Resolution + +### Cause 1: chartmuseum requests to NAS storage time out from a node with a network fault + +1. Identify the node that runs chartmuseum and confirm the NAS timeouts from that node (see Diagnostic Steps). Report the node network fault to whoever owns the node network so it gets repaired; the Pod restart below restores the page, it does not repair the node. + +2. If the node's network is still faulty, cordon it so the replacement Pod is scheduled on a healthy node. + + > **Warning:** Only when the chartmuseum Pod's node is the one with the confirmed network fault. Impact: no new Pods are scheduled on that node; running Pods stay. Precondition: other nodes have capacity for the chartmuseum Pod and can reach the NAS. Rollback: `kubectl uncordon ` once the network is repaired. + + ```bash + kubectl cordon + ``` + +3. Delete the chartmuseum Pod so its controller recreates it on another node. + + > **Warning:** Only when chartmuseum runs on a node whose NAS requests time out, as confirmed in Diagnostic Steps. Impact: chart requests are interrupted until the replacement Pod is `Ready`. Precondition: the volume is NAS/NFS storage that other nodes can mount. The deleted Pod is not restored; the replacement created by the controller is the recovery. + + ```bash + NS= + CM=$(kubectl -n "$NS" get pods -o name | grep chartmuseum | head -1) + kubectl -n "$NS" delete "$CM" + kubectl -n "$NS" get pods -o wide | grep chartmuseum + ``` + +4. Open the project page again. It loads normally once the new chartmuseum Pod is `Ready` on a healthy node. + +### Cause 2: the browser is older than Chrome 98 + +1. Check the browser version before changing anything in Harbor: open `chrome://version` and read the major version. +2. If it is lower than 98, update the browser to Chrome 98 or later (or open the same Harbor URL in such a browser). +3. Reload the Harbor URL. The login page appears. Do not restart Harbor Pods or edit the Harbor resource for this cause. + +## Diagnostic Steps + +1. List the Harbor components and follow chartmuseum to its node: + + ```bash + NS= + kubectl -n "$NS" get pods -o wide + CM=$(kubectl -n "$NS" get pods -o name | grep chartmuseum | head -1) + kubectl -n "$NS" logs "$CM" --tail=100 + kubectl -n "$NS" get "$CM" -o jsonpath='{.spec.nodeName}{"\n"}' + ``` + + All Pods `Running` is expected for both causes; it does not rule either out. No chartmuseum Pod at all rules out Cause 1. + +2. Check whether the chartmuseum volume is NAS/NFS and whether a filesystem call from the Pod hangs. A `df` that does not return within the timeout means the mount is not answering: + + ```bash + kubectl -n "$NS" get pvc | grep -i chartmuseum + kubectl get pv -o jsonpath='{.spec.nfs.server}{" "}{.spec.nfs.path}{"\n"}' + kubectl -n "$NS" exec "$CM" -- timeout 10 df -h || echo "df did not complete: storage not responding" + ``` + +3. On the node that runs chartmuseum, look for NFS timeouts and test reachability of the NAS server. Repeat the `ping` from another node for comparison: + + ```bash + dmesg -T | grep -iE 'nfs.*(not responding|timed out)' | tail -20 + ping -c 3 + ``` + + NFS `not responding` / `timed out` messages on that node only, while other nodes reach the NAS, point to Cause 1. + +4. Tell the two causes apart: + + | What you see | Cause | + | --- | --- | + | Login works, the project page keeps loading, NAS timeouts only from the chartmuseum node | Cause 1 | + | The page never reaches login, API requests show no error, the browser is older than Chrome 98 | Cause 2 | + | The NFS export is reachable but has no free space | [Harbor unreachable or its PostgreSQL out of space because Harbor storage is full](Harbor_unreachable_or_its_PostgreSQL_out_of_space_because_Harbor_storage_is_full.md) | + | The portal says the core service is unavailable instead of loading slowly | [Harbor core service unavailable or internal server error, stuck instance status, or Pods cycling after an edit](Harbor_core_service_unavailable_or_internal_server_error_stuck_instance_status_or_Pods_cycling_after_an_edit.md) | + + For slow `harbor-database` startup on NFS storage, see [Harbor PostgreSQL database Pod never starts after an operator upgrade with transparent huge pages, low CPU, or on NFS](Harbor_PostgreSQL_database_Pod_never_starts_after_an_operator_upgrade_with_transparent_huge_pages_low_CPU_or_on_NFS.md). diff --git a/docs/en/solutions/Helm_chart_deployment_from_Catalog_fails_or_creates_the_wrong_objects_on_ACP.md b/docs/en/solutions/Helm_chart_deployment_from_Catalog_fails_or_creates_the_wrong_objects_on_ACP.md new file mode 100644 index 000000000..72c62dd58 --- /dev/null +++ b/docs/en/solutions/Helm_chart_deployment_from_Catalog_fails_or_creates_the_wrong_objects_on_ACP.md @@ -0,0 +1,201 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.2" +--- + +# Helm chart deployment from Catalog fails or creates the wrong objects on ACP + +## Issue + +A custom Helm chart deployed from Catalog (template deployment, a `HelmRequest` reconciled by `captain`) fails, or deploys a different set of objects than expected: + +- A `range` that should create several Deployments creates only the last one; the earlier names show up and then are not found. +- The same package installs with `helm install`, but the platform rejects it: a permission error creating `apps/v1` Deployments, or a refusal to create resources outside the target namespace. +- `captain-controller-manager` fails the release with `artifacts.app.alauda.io "" not found` (for example `"higress-core"`). +- The deployment log shows `ytt: Error: Compiling YAML template 'template.yaml'`, `Expected ytt-formatted string (use '#@' for annotations or code, '#!' for comments)` and `error: no objects passed to apply`. +- The deploy fails with `rendered manifests contain a resource that already exists` and `cannot be imported into the current release`, naming a cluster-scoped object of another release. +- A release installed with `helm install` never appears as a platform application, or a later HelmRequest for it fails. +- The first import of a template reports success and creates no resources, although Harbor has the chart; a second import creates them. + +A chart that Catalog does not list at all, or lists without a selectable version, is covered in [Catalog does not list a Helm chart that is already in Harbor, or a newly uploaded chart version, on ACP](Catalog_does_not_list_a_Helm_chart_that_is_already_in_Harbor_or_a_newly_uploaded_chart_version_on_ACP.md). + +## Root Cause + +### Cause 1: `range` emits several Deployments as one YAML document + +The template walks a list with `range` and prints a Deployment each time, without a `---` document separator inside the loop. Helm parses the output as a single Deployment, and only the last iteration is stored. The earlier names may still appear in the UI, but no such Deployment exists. + +### Cause 2: the chart renders namespaced objects outside the target namespace + +HelmRequest `spec.namespace` is the namespace the release is installed into, and the chart has to render that same namespace. A chart that hard-codes `metadata.namespace` to another namespace fails on the platform even though `helm install` of it succeeds: a normal user gets a permission error creating Deployments, and a platform administrator is told that resources are not allowed in a namespace other than the target namespace. + +The restriction is about namespaced objects. Catalog does not filter cluster-scoped objects that the chart defines, such as CRDs, StorageClasses, PersistentVolumes, and ClusterRoles; they are created. Whether they are created successfully depends on the template content, the permissions in the cluster, and the storage backend. + +### Cause 3: `artifacts.app.alauda.io "" not found` + +`captain` looks the chart up as an `Artifact`. The named chart (for example a dependency such as `higress-core`) is not in the template repository the deploy resolves from. Uploading the parent chart does not create the Artifact of that chart. The same log can also contain an `Unauthorized` HelmRequest error, which is a different fault; see [Application updates are slow, status stays Failed or Processing, or delete hangs because of captain on ACP](Application_updates_are_slow_status_stays_Failed_or_Processing_or_delete_hangs_because_of_captain_on_ACP.md). + +### Cause 4: the chart's `template.yaml` is not valid ytt + +The chart's own `template.yaml` is processed by ytt, and ytt rejects plain `#` comment lines: it accepts `#@` for annotations or code and `#!` for comments. Example: + +```text +ytt: Error: Compiling YAML template 'template.yaml': + Failed to parse line 113: '# ytt if/end: only wraps the immediately following node': + Expected ytt-formatted string (use '#@' for annotations or code, '#!' for comments) +error: no objects passed to apply +``` + +Here line 113 is a plain comment placed right before `#@ if/end data.values.certmanagerEnabled:`. ytt fails on it, produces no objects, and the apply step has nothing to apply. The failure is in the chart content, not in the platform's Helm; the bundled Helm version (`v3.12.1` on ACP 4.0, older than some charts require) is not what produces this error. + +### Cause 5: a cluster-scoped object already belongs to another release + +```text +rendered manifests contain a resource that already exists +ClusterRole "" in namespace "" +cannot be imported into the current release +release-name must equal "": current value is "" +release-namespace must equal "": current value is "" +``` + +A cluster-scoped object rendered by the chart already exists, and its Helm ownership annotations point to a different release in a different namespace, typically an earlier install of the same operator chart under another release name. Helm refuses to take over an object owned by another release. + +### Cause 6: a release installed with `helm install` is not a platform application + +Catalog creates a HelmRequest (`chart`, `version`, `clusterName`, `namespace`, `releaseName`, `values`, `valuesFrom`), and `captain` installs the release and writes `status.phase` and `status.reason`. A release created only with `helm install` on a server has no HelmRequest, so the platform does not show it as an application, and a later HelmRequest for the same release conflicts with it. + +### Cause 7: the first template import reports success and creates no resources + +The first import reports success and Harbor has the chart, but no resources are created; importing the same template again creates them. This is a product defect fixed in ACP 4.2.5. + +## Resolution + +### Cause 1: `range` emits several Deployments as one YAML document + +Put a document separator inside the loop so each iteration is its own object: + +```yaml +{{- range .Values.deployments }} +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ .name }} + namespace: {{ $.Release.Namespace }} +spec: + selector: + matchLabels: + app: {{ .name }} + template: + metadata: + labels: + app: {{ .name }} + spec: + containers: + - name: app + image: {{ .image }} +{{- end }} +``` + +`helm template ./chart --namespace ` must print one `---` document per Deployment. Package the chart as a new version, upload it through Catalog, and redeploy. + +### Cause 2: the chart renders namespaced objects outside the target namespace + +Render namespaced objects into the release namespace (`{{ .Release.Namespace }}`, as in Cause 1) and remove hard-coded namespaces. Check before uploading: + +```bash +helm template ./chart --namespace | grep -n '^ namespace:' +``` + +Every namespaced object must show ``. Upload the new version and redeploy into that namespace. + +### Cause 3: `artifacts.app.alauda.io "" not found` + +1. Upload the missing chart package into the same template repository the parent chart is resolved from (Catalog > Helm Charts > Add Template, or the Catalog API `POST /catalog/v1/chartrepos/cpaas-system//charts`). +2. Retry the HelmRequest once the Artifact exists: + + ```bash + kubectl get artifact -A | grep + kubectl get helmrequest -n -o jsonpath='{.spec.dependencies}{"\n"}{.status.phase}{" "}{.status.reason}{"\n"}' + ``` + +If the HelmRequest lists names in `spec.dependencies`, those HelmRequests must be in the same namespace and `Synced` first. The package has to exist even when `spec.dependencies` is empty. + +### Cause 4: the chart's `template.yaml` is not valid ytt + +Fix the line the error names: turn the plain comment into a ytt comment (or remove it) and keep the `#@` annotation that follows: + +```yaml +#! ytt if/end: only wraps the immediately following node +#@ if/end data.values.certmanagerEnabled: +- --cert-dir=/var/run/gpustack/certs +``` + +Check every other plain `#` line in files that ytt processes, then repackage, upload the new version, and redeploy. With the ytt CLI, `ytt -f template.yaml` (plus the data values the template expects) reproduces the parse error offline. + +### Cause 5: a cluster-scoped object already belongs to another release + +1. Read the ownership of the object named in the error and confirm it matches the `current value` in the message: + + ```bash + kubectl get clusterrole \ + -o jsonpath='{.metadata.annotations.meta\.helm\.sh/release-name}{" "}{.metadata.annotations.meta\.helm\.sh/release-namespace}{"\n"}' + helm list -A | grep + kubectl get helmrequest -A | grep + ``` + +2. When the owning release is no longer needed (or agrees to give up the object), delete the object and deploy the chart again. + + > **Warning:** Deleting a ClusterRole removes those permissions from every subject bound to it; workloads of `` lose that access at once. Only when the `cannot be imported into the current release` error is confirmed for this object. Precondition: the object is saved by the first command. Rollback: before the new deploy, `kubectl apply -f .yaml`; after the new release has created its own copy, the new release has to be removed first. + + ```bash + kubectl get clusterrole -o yaml > .yaml + kubectl delete clusterrole + ``` + +3. If the next attempt names another object, repeat the ownership check for that object instead of deleting objects blindly. + +### Cause 6: a release installed with `helm install` is not a platform application + +Install charts that must appear as applications through Catalog. For a release that already exists, either keep managing it with Helm and create no HelmRequest with the same release name, or uninstall it in a window and let the HelmRequest install it. + +> **Warning:** `helm uninstall` deletes the workloads of that release and cannot be undone. Only when a HelmRequest will reinstall the same chart and you have its package and values. Rollback: `helm install` the same package again. + +```bash +helm list -n +kubectl get helmrequest -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,CHART:.spec.chart,REL:.spec.releaseName,PHASE:.status.phase,REASON:.status.reason +helm uninstall -n +``` + +### Cause 7: the first template import reports success and creates no resources + +Import the template again; the second import creates the resources. Upgrade to ACP 4.2.5 or later for the fix. If a first import still creates nothing on 4.2.5 or later, or the application exists but stays processing with `spec.assemblyPhase` `pending`, use [Application updates are slow, status stays Failed or Processing, or delete hangs because of captain on ACP](Application_updates_are_slow_status_stays_Failed_or_Processing_or_delete_hangs_because_of_captain_on_ACP.md) instead. + +```bash +kubectl get application -n -o jsonpath='phase={.status.phase} assembly={.spec.assemblyPhase}{"\n"}' +``` + +## Diagnostic Steps + +```bash +helm template ./chart --namespace +kubectl -n cpaas-system logs deploy/captain-controller-manager --tail=200 | grep -E 'artifacts.app.alauda.io|Unauthorized' +kubectl get helmrequest -n -o jsonpath='{.status.phase}{" "}{.status.reason}{"\n"}' +``` + +The HelmRequest `status.reason` carries the failure text of a Catalog deployment. + +```text +helm template prints several Deployments as one document, the UI keeps only the last -> Cause 1 +rendered namespace differs from ; helm install works, platform refuses -> Cause 2 +captain log: artifacts.app.alauda.io "" not found -> Cause 3 +ytt: Error ... Expected ytt-formatted string, then error: no objects passed to apply -> Cause 4 +rendered manifests contain a resource that already exists ... cannot be imported -> Cause 5 +release only in helm list, no HelmRequest, or a new HelmRequest fights it -> Cause 6 +first import said success, Harbor has the chart, nothing created -> Cause 7 +``` diff --git a/docs/en/solutions/docker_push_to_Harbor_from_Docker_Desktop_times_out_with_proxyconnect_i_o_timeout.md b/docs/en/solutions/docker_push_to_Harbor_from_Docker_Desktop_times_out_with_proxyconnect_i_o_timeout.md new file mode 100644 index 000000000..8e8a4f82c --- /dev/null +++ b/docs/en/solutions/docker_push_to_Harbor_from_Docker_Desktop_times_out_with_proxyconnect_i_o_timeout.md @@ -0,0 +1,65 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.3" +--- + +# docker push to Harbor from Docker Desktop times out with proxyconnect i/o timeout + +## Issue + +`docker push` from a workstation running Docker Desktop (for example on macOS) to the platform Harbor times out. The error names a proxy rather than Harbor: + +```text +proxyconnect tcp: dial tcp :3128: i/o timeout +``` + +The proxy on port `3128` is Docker Desktop's own HTTP proxy (`http.docker.internal:3128`). `docker info` shows an `HTTP Proxy` / `HTTPS Proxy`, and its `No Proxy` list contains only entries such as `hubproxy.docker.internal`, not the Harbor address. + +Related articles: [Harbor push of large image layers fails with unexpected EOF, timeout awaiting response headers, or 408 on ACP](Harbor_push_of_large_image_layers_fails_with_unexpected_EOF_timeout_awaiting_response_headers_or_408_on_ACP.md) and [Harbor image push fails while harbor-nginx restarts, or is rejected with too large on a NodePort Harbor](Harbor_image_push_fails_while_harbor-nginx_restarts_or_is_rejected_with_too_large_on_a_NodePort_Harbor.md). + +## Root Cause + +### Cause 1: Docker Desktop sends the registry traffic to its proxy + +Docker Desktop is configured with an HTTP/HTTPS proxy, and the Harbor address is not in its `No Proxy` (bypass) list. The engine therefore sends the push to the proxy instead of connecting to Harbor directly. The proxy cannot reach the internal Harbor address, so the connection times out before any layer is uploaded. Adding the Harbor address to the `No Proxy` list in Docker Desktop stops the timeout. + +## Resolution + +### Cause 1: Docker Desktop sends the registry traffic to its proxy + +1. In Docker Desktop, open the proxy settings (**Settings > Resources > Proxies** in current releases) and add the Harbor host or IP address, with and without the port as your clients use it, to the bypass / `No Proxy` list. Keep the existing entries. +2. Apply the change. Docker Desktop restarts its engine, which stops running containers on that workstation until it is back. +3. Check that the address is now bypassed and retry: + + ```bash + docker info | grep -iE 'HTTP Proxy|HTTPS Proxy|No Proxy' + docker login : + docker push ://: + ``` + +On a Linux host with `dockerd`, the same rule applies: the daemon's proxy configuration must list the Harbor address in `NO_PROXY` / `no-proxy`. + +If the timeout is gone but large layers now stop in `Pushing` and the push ends with `unexpected EOF`, the proxy is fixed and the remaining failure is on the network path; continue with Cause 2 of [Harbor push of large image layers fails with unexpected EOF, timeout awaiting response headers, or 408 on ACP](Harbor_push_of_large_image_layers_fails_with_unexpected_EOF_timeout_awaiting_response_headers_or_408_on_ACP.md). Increasing the Docker Desktop memory does not fix that. + +## Diagnostic Steps + +1. Read the engine's proxy settings and the shell's proxy environment: + + ```bash + docker info | grep -iE 'HTTP Proxy|HTTPS Proxy|No Proxy' + env | grep -iE 'http_proxy|https_proxy|no_proxy' + ``` + + An `HTTP Proxy` / `HTTPS Proxy` entry, with a `No Proxy` list that does not contain the Harbor address, matches this cause. + +2. Test Harbor directly, bypassing any proxy: + + ```bash + curl -vk --noproxy '*' --connect-timeout 5 https://:/v2/ + ``` + + A response (`401` is normal without credentials) proves that Harbor is reachable directly and only the proxy path fails. A timeout here as well means the workstation cannot reach Harbor at all; check the network path and firewall instead, see [Image registry or Harbor unreachable or slow because of firewall rules, blocked ports, or the network path](Image_registry_or_Harbor_unreachable_or_slow_because_of_firewall_rules_blocked_ports_or_the_network_path.md). From f5e8bf7d7233e1243b64ea01bc0f4fe18e4e77d0 Mon Sep 17 00:00:00 2001 From: Komh Date: Wed, 18 Nov 2026 17:41:07 +0800 Subject: [PATCH 17/25] docs(solutions): add 16 and update 1 troubleshooting articles --- ...e_login_page_shows_the_previous_account.md | 94 ++++++++ ...ation_page_after_a_template_is_selected.md | 84 +++++++ ...sen_password_and_password_policy_limits.md | 135 +++++++++++ ...the_access_address_does_not_work_on_ACP.md | 203 ++++++++++++++++ ...rojects_API_500_or_an_extra_bot_project.md | 13 + ..._next_to_istio-proxy_on_Service_Mesh_v2.md | 163 +++++++++++++ ...Secret_certificate_alerts_remain_on_ACP.md | 118 +++++++++ ...sticsearch_to_OpenSearch_with_Jaeger_v2.md | 205 ++++++++++++++++ ...s_resources_or_system_namespaces_on_ACP.md | 141 +++++++++++ ...ccount_has_no_ClusterRoleBinding_on_ACP.md | 73 ++++++ ...ng_or_TektonHub_cannot_be_viewed_on_ACP.md | 227 ++++++++++++++++++ ..._pods_that_still_have_sidecar_injection.md | 168 +++++++++++++ ...ata_or_jaeger-prod-query_crashes_on_ACP.md | 177 ++++++++++++++ ...hile_a_direct_role_binding_works_on_ACP.md | 135 +++++++++++ ...a_or_an_otel_instance_cannot_be_deleted.md | 181 ++++++++++++++ ...does_not_start_or_fails_requests_on_ACP.md | 144 +++++++++++ ...after_ac_login_or_ac_login_fails_on_ACP.md | 113 +++++++++ 17 files changed, 2374 insertions(+) create mode 100644 docs/en/solutions/ACP_console_session_survives_cookie_deletion_or_a_browser_close_or_the_login_page_shows_the_previous_account.md create mode 100644 docs/en/solutions/Alauda_AI_fine-tuning_has_no_task_template_or_shows_a_blank_configuration_page_after_a_template_is_selected.md create mode 100644 docs/en/solutions/Forgotten_local_admin_password_on_ACP_reset_the_password_hash_set_a_chosen_password_and_password_policy_limits.md create mode 100644 docs/en/solutions/GitLab_login_fails_returns_422_or_the_access_address_does_not_work_on_ACP.md create mode 100644 docs/en/solutions/Java_auto-instrumentation_init_container_is_missing_or_the_Java_agent_lands_next_to_istio-proxy_on_Service_Mesh_v2.md create mode 100644 docs/en/solutions/Leftover_Service_Mesh_v1_recreates_istio-system_against_Mesh_v2_or_Istio_Secret_certificate_alerts_remain_on_ACP.md create mode 100644 docs/en/solutions/Moving_Service_Mesh_v2_tracing_storage_from_Elasticsearch_to_OpenSearch_with_Jaeger_v2.md create mode 100644 docs/en/solutions/Namespace_quota_exceeds_the_project_quota_a_project_shows_the_whole_clusters_resources_or_system_namespaces_on_ACP.md create mode 100644 docs/en/solutions/OpenTelemetry_Collector_logs_cannot_list_resources_pods_because_its_ServiceAccount_has_no_ClusterRoleBinding_on_ACP.md create mode 100644 docs/en/solutions/Pipeline_task_list_is_empty_tasks_are_missing_or_TektonHub_cannot_be_viewed_on_ACP.md create mode 100644 docs/en/solutions/Service_Mesh_delete_is_stuck_on_a_failed_to_get_cluster_webhook_denial_or_on_pods_that_still_have_sidecar_injection.md create mode 100644 docs/en/solutions/Service_Mesh_tracing_Kiali_trace_links_return_404_a_workload_has_no_data_or_jaeger-prod-query_crashes_on_ACP.md create mode 100644 docs/en/solutions/User_group_member_gets_a_data_error_or_no_permissions_while_a_direct_role_binding_works_on_ACP.md create mode 100644 docs/en/solutions/asm-otel_collectors_keep_restarting_Service_Mesh_JVM_monitoring_has_no_data_or_an_otel_instance_cannot_be_deleted.md create mode 100644 docs/en/solutions/devops-controller_or_devops-apiserver_does_not_start_or_fails_requests_on_ACP.md create mode 100644 docs/en/solutions/kubectl_returns_the_server_has_asked_for_the_client_to_provide_credentials_after_ac_login_or_ac_login_fails_on_ACP.md diff --git a/docs/en/solutions/ACP_console_session_survives_cookie_deletion_or_a_browser_close_or_the_login_page_shows_the_previous_account.md b/docs/en/solutions/ACP_console_session_survives_cookie_deletion_or_a_browser_close_or_the_login_page_shows_the_previous_account.md new file mode 100644 index 000000000..3ffd40e0a --- /dev/null +++ b/docs/en/solutions/ACP_console_session_survives_cookie_deletion_or_a_browser_close_or_the_login_page_shows_the_previous_account.md @@ -0,0 +1,94 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.2" +--- + +# ACP console session survives cookie deletion or a browser close, or the login page shows the previous account + +## Issue + +The platform web console keeps, loses, or reuses a login in a way the user does not expect. One of these is seen: + +- After logging in and deleting the site's cookies, a page refresh keeps the account logged in; the user is not logged out. This is often found during a security assessment. +- With "end the session when all product tabs are closed" enabled, closing the browser and opening the platform again goes straight in without a login. +- Users can log in only after clearing the browser cache and restarting the browser. +- The login page says the license code has expired, although the license is still valid. +- After a direct local Dex login URL is opened, the form accepts account B but the session shown is still account A. +- The Service Mesh page says that no authorization token was provided, while the console login itself works. + +Related articles: "Forgotten local admin password on ACP: reset the password hash, set a chosen password, and password policy limits" covers local passwords. "kubectl returns the server has asked for the client to provide credentials after ac login, or ac login fails, on ACP" covers the CLI. + +## Root Cause + +### Cause 1: login session data is also kept in the browser's local storage + +The console stores data in the browser's Local Storage (keys such as `meepo-project` and `theme-mode`) as well as in cookies, and part of the login state is kept in Local Storage. Deleting only the cookies leaves that state, and the next page load restores the login. Clearing both cookies and Local Storage for the platform site logs the user out. + +### Cause 2: the session ends about 30 seconds after the last product tab closes + +On 4.2, Access Control offers "end the session when all product tabs are closed". The browser does not clear the session at the instant it closes; the front end calls the logout API through its multi-tab mechanism, and that call is delayed. The session therefore stays valid for about 30 seconds after the last product tab closes. Reopening within that window does not ask for a login; after it, a login is required. This is the designed behavior. + +### Cause 3: the login page sat idle until its session cookie expired + +The login page was left open with no activity until its session cookie expired, so submitting the stale form fails. Reopening the platform access URL, or clearing the cache, loads a fresh form. This is expected. + +### Cause 4: the license-expired message is stale page cache + +A license-expired message that does not appear in an incognito window comes from cached page content, not from the license. Retyping the platform address does not clear it. + +### Cause 5: a direct Dex login URL reuses the previous account's cached token + +On 4.2, account A logs in and the browser is closed without logging out. A direct local Dex URL (path `/console-dex/login/local` with `client_id=alauda-auth` and `redirect_uri` pointing at `/console-portal`) is then opened and account B is entered. The browser still holds A's token, and that URL uses the cached token instead of B's credentials, so the session is A's. Whether it reproduces depends on the browser. + +### Cause 6: the token held by a long-open page has expired + +On 3.18 the Service Mesh page reports that no authorization token was provided after the browser session has been open long enough for its token to expire. The page has nothing valid to send. A fresh login gets a new token. + +## Resolution + +### Cause 1: login session data is also kept in the browser's local storage + +1. Open the browser developer tools (F12) on the platform page, go to **Application**, and check both **Cookies** and **Local Storage** for the platform address. +2. Clear both for the platform site, or use **Clear site data**, then refresh. The user is logged out. +3. For a normal logout, use the console's logout action, which ends the session on the server instead of only removing browser data. + +### Cause 2: the session ends about 30 seconds after the last product tab closes + +No change is needed. Wait more than about 30 seconds after the last product tab closes before expecting a new login. This is not an idle-timeout setting. + +### Cause 3: the login page sat idle until its session cookie expired + +Open the platform access URL again, or clear the site cache, and log in. Do not treat this as a wrong password or a Dex fault when a fresh page accepts the same user. + +### Cause 4: the license-expired message is stale page cache + +Check the license validity date in the platform first. If it is still valid, open an incognito window, or clear the site cache, and log in. If a wrong-password error follows, it is a password problem, not the license. + +### Cause 5: a direct Dex login URL reuses the previous account's cached token + +Log out, or clear the site's cookies and Local Storage, before using `/console-dex/login/local` to switch accounts. No platform object needs to change. + +### Cause 6: the token held by a long-open page has expired + +Log out and log in again, or open the platform in an incognito window. The page then loads. No mesh configuration change is needed. + +## Diagnostic Steps + +1. On the platform page, open the developer tools (F12): + - **Application** > **Cookies** and **Local Storage**: note which entries remain for the platform address. + - **Network**: refresh, select a platform API request, and check whether it still carries `Cookie` or `Authorization`. +2. Open the same address in an incognito window and compare the behavior. + +| What you see | Cause | +| --- | --- | +| Cookies deleted, refresh keeps the login, Local Storage still has entries; clearing it logs out | 1 | +| Session survives a browser close for less than about 30 seconds with tab-close logout enabled | 2 | +| Login works only after a cache clear, the form was left idle | 3 | +| License-expired text, license still valid, incognito window is clean | 4 | +| Direct Dex URL logs in as the previous user | 5 | +| Service Mesh page says no authorization token, a fresh login fixes it | 6 | diff --git a/docs/en/solutions/Alauda_AI_fine-tuning_has_no_task_template_or_shows_a_blank_configuration_page_after_a_template_is_selected.md b/docs/en/solutions/Alauda_AI_fine-tuning_has_no_task_template_or_shows_a_blank_configuration_page_after_a_template_is_selected.md new file mode 100644 index 000000000..1aded30c1 --- /dev/null +++ b/docs/en/solutions/Alauda_AI_fine-tuning_has_no_task_template_or_shows_a_blank_configuration_page_after_a_template_is_selected.md @@ -0,0 +1,84 @@ +--- +kind: + - Troubleshooting +products: + - Alauda AI +ProductsVersion: [] +--- + +# Alauda AI fine-tuning has no task template or shows a blank configuration page after a template is selected + +## Issue + +- In **Model Optimization** > **Fine-Tuning**, the task template list is empty, so a fine-tuning task cannot be created. The expectation was that a task template is available right after the initial deployment. +- A template is selected in the **Create Fine-tuning Task** dialog, and the task configuration page that follows is completely blank. +- The question is what goes in the dataset text input column (`数据集文本输入列`), and whether more than one column can be selected. + +Fine-tuning and training are optional features. They need the fine-tuning plugins (Volcano, Kubeflow Training Operator, MLFlow, and the others listed in "Enable Fine-Tuning and Training Features" of the Alauda AI documentation) and `tuneModels: true` under `spec.values.experimentalFeatures` of the `AmlCluster` resource `default`. If the Fine-Tuning menu itself is missing, start with that guide. + +## Root Cause + +### Cause 1: no template has been uploaded, or the template lacks the label the fine-tuning entry filters on + +Task templates are not created by the deployment. The example templates are shipped in the `alaudadockerhub/training-templates` image and have to be extracted and uploaded to **Task Template**, as described in "Enable Fine-Tuning and Training Features". In addition, a task template is shown at the fine-tuning entry only when it carries the label that the fine-tuning entry filters on; a template uploaded without it does not appear. + +### Cause 2: an old template with missing fields is still in use after a Volcano upgrade conflict + +A conflict during the Volcano component upgrade left the old component in place. The fine-tuning task was therefore created from the old task template, which lacks fields that the current configuration form needs, and the form renders as a blank page. Uploading the current template again makes the page display normally. + +### Cause 3: the text input column takes text columns only + +The dataset text input column must contain only text columns and must not include the label column. More than one column can be selected. + +## Resolution + +### Cause 1: no template has been uploaded, or the template lacks the label the fine-tuning entry filters on + +1. Check whether template projects exist in the GitLab that Alauda AI uses. The built-in templates are GitLab projects under `/aml-templates/`, for example `/aml-templates/finetune-text-classification`: + + ```bash + GITLAB=https:// + TOKEN= + curl -s -H "PRIVATE-TOKEN: ${TOKEN}" "${GITLAB}/api/v4/projects?search=aml-templates&simple=false" \ + | jq '.[] | {id, path_with_namespace, topics}' + curl -s -H "PRIVATE-TOKEN: ${TOKEN}" "${GITLAB}/api/v4/projects?search=finetune&simple=false" \ + | jq '.[] | {id, path_with_namespace, topics}' + ``` + +2. If no template project exists, extract the example templates from the `alaudadockerhub/training-templates` image and upload them to **Task Template**, following "Enable Fine-Tuning and Training Features" for your Alauda AI version. Make sure all Bash scripts in the template (especially `run.sh`) are executable before uploading, as "Developing Custom Fine-Tuning Templates" requires. +3. If templates exist but are not listed, compare their labels and topics with a template that is listed (in another namespace or environment), and add the label the fine-tuning entry expects, as the Alauda AI fine-tuning template documentation of your version describes. +4. Reopen **Create Fine-tuning Task** and check the list. + +### Cause 2: an old template with missing fields is still in use after a Volcano upgrade conflict + +1. Check for an old Volcano component next to the new one: + + ```bash + kubectl get pods -A | grep -i volcano + kubectl get deploy -A | grep -i volcano + kubectl get apprelease -A | grep -i volcano + kubectl get minfo | grep | grep -i volcano + ``` + + Two generations of Volcano workloads, or an AppRelease of the previous version that is still present, matches this cause. + +2. Compare the template in use with the template package of the installed Alauda AI version; the old template lacks fields the new one has. +3. Upload the current template again to the same **Task Template** location. Reopen **Create Fine-tuning Task**, select the template, and confirm that the configuration form renders. + +### Cause 3: the text input column takes text columns only + +Select one or more text columns as the text input column and leave the label column out. + +## Diagnostic Steps + +1. Confirm the feature is on: + + ```bash + kubectl get amlcluster default -o jsonpath='{.spec.values.experimentalFeatures}{"\n"}' + ``` + + The output should include `tuneModels: true`. + +2. Open the browser developer tools (F12) on the fine-tuning page. An empty template list returned with HTTP `200` points at missing or unlabeled templates (Cause 1). A `500` from `aml-api-deploy-svc` points at the backend, not at templates. +3. For a blank form, note which template was selected and compare its files with the current template package (Cause 2). +4. A question about which dataset columns to select is Cause 3. diff --git a/docs/en/solutions/Forgotten_local_admin_password_on_ACP_reset_the_password_hash_set_a_chosen_password_and_password_policy_limits.md b/docs/en/solutions/Forgotten_local_admin_password_on_ACP_reset_the_password_hash_set_a_chosen_password_and_password_policy_limits.md new file mode 100644 index 000000000..ef690489f --- /dev/null +++ b/docs/en/solutions/Forgotten_local_admin_password_on_ACP_reset_the_password_hash_set_a_chosen_password_and_password_policy_limits.md @@ -0,0 +1,135 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.6" + - "3.12" + - "4.2" + - "4.3" +--- + +# Forgotten local admin password on ACP: reset the password hash, set a chosen password, and password policy limits + +## Issue + +One of the following is needed for a local (not LDAP) platform account: + +- The console password set at installation, or the platform `admin` (for example `admin@cpaas.io`) password, was forgotten and must be reset, and user management cannot show the old password. +- A backend password change must set a specific new password instead of the default. +- Users are being copied to a new platform from a backup, and the question is whether their passwords come along. +- A security review asks which algorithm protects local user passwords, or an audit list asks for maximum password age, password history, and lockout after failed logins. + +Related articles: "kubectl returns the server has asked for the client to provide credentials after ac login, or ac login fails, on ACP" covers the CLI and API tokens. "ACP console session survives cookie deletion or a browser close, or the login page shows the previous account" covers browser sessions. An LDAP user's password is changed in the directory, not on the platform. + +## Root Cause + +### Cause 1: the local credential is the `hash` of a `password` object + +A local account's credential is the `hash` field of a `password` object in `cpaas-system` on the `global` cluster. The platform cannot display the original password. Replacing the hash with the known default hash makes the password `P@ssw0rd`; the user then sets a new password from the profile page. Sometimes the first login after the patch is still rejected until Dex reloads. + +### Cause 2: local passwords are stored as PBKDF2 hashes, so a plaintext value cannot be written + +To set a chosen password from the backend, a plaintext password cannot be put into the `password` object. A hash of the chosen password has to be generated by the platform itself, for example by creating a temporary local account with that password and reusing its hash. + +### Cause 3: a User backup does not contain the local password + +An `auth.alauda.io/v1` User object has no password field. Exports such as `users.yaml` and `userbindings.json` restore identities and role bindings, not local passwords. There is no backup and restore for local passwords; each one has to be set again on the new platform. + +### Cause 4: local user passwords use pbkdf2 + +Local user passwords are stored with the pbkdf2 algorithm; the default hash of Cause 1 is a pbkdf2 hash of `P@ssw0rd`. The RSA public key that the console front end uses to encrypt the password in the login request is a different topic. When the question does not say which one is meant, ask. + +### Cause 5: the user password settings do not cover every audit item, and lockout applies to local users + +The platform user password settings do not provide a maximum password age of three months, a reuse limit of five generations, or lockout after five failures. A configured lockout ("lock the account for five minutes after five wrong passwords") applies to local users; LDAP users are not locked by it, so a test with an LDAP account shows no lockout. + +## Resolution + +### Cause 1: the local credential is the `hash` of a `password` object + +1. On the `global` cluster, find the `password` object of the account and back it up: + + ```bash + kubectl get password -n cpaas-system -o=custom-columns=Name:.metadata.name,Username:.username,Email:.email + kubectl get user -n cpaas-system | grep admin + kubectl get password -n cpaas-system -o yaml > /tmp/password.bak.yaml + ``` + + Continue only when the row's username is the local account to reset. + +2. Replace the hash with the default hash. + + > **Warning:** The account's password becomes `P@ssw0rd` until it is changed; anyone who can open the console can log in as that user in the meantime. Only for a local account whose password is forgotten. Rollback: `kubectl apply -f /tmp/password.bak.yaml` restores the previous hash (restart Dex again if it was restarted). + + ```bash + kubectl patch password -n cpaas-system --type='json' -p='[{"op": "replace", "path": "/hash", "value":"UEJLREYyOjgrNDgzZ0QyQmJObXZXbWp6eVBVQUNNdXMyVTRqS2ZmRFlKdnZ0ZXRUMGc9OjFiTHhEMDVGeDJwWEthekF3bzBWb0E9PQ=="}]' + ``` + +3. Log in with `P@ssw0rd`. If the password is still rejected, restart Dex once and retry. On 4.x Dex runs inside `apollo`; on a cluster with a separate `dex` Deployment, restart that one. + + > **Warning:** Login is interrupted until the Pod is Ready. Only when the patch succeeded and the first login was rejected. Rollback: not needed. + + ```bash + kubectl get deploy -n cpaas-system | grep -Ei 'apollo|^dex' + kubectl -n cpaas-system rollout restart deploy/ + ``` + +4. Open the personal information page from the upper right corner and set a new password with the pencil icon next to the password. + +### Cause 2: local passwords are stored as PBKDF2 hashes, so a plaintext value cannot be written + +1. On a platform that is accessible, create a temporary local account with the desired password and read its hash: + + ```bash + kubectl get password -n cpaas-system -o=custom-columns=Name:.metadata.name,Username:.username,Email:.email + kubectl get password -n cpaas-system -o jsonpath='{.hash}{"\n"}' + ``` + +2. Write that hash into the target account. + + > **Warning:** The target account's password becomes the temporary account's password. Only for a local account, with its `password` object identified as in Cause 1 and backed up. Rollback: `kubectl apply -f /tmp/password.bak.yaml`. + + ```bash + kubectl get password -n cpaas-system -o yaml > /tmp/password.bak.yaml + kubectl patch password -n cpaas-system --type='json' -p='[{"op": "replace", "path": "/hash", "value":""}]' + ``` + +3. Log in with the chosen password (restart Dex as in Cause 1 if it is rejected), then delete the temporary account. This method has not been verified as widely as the default-hash reset in Cause 1; if it fails, use Cause 1 and change the password from the profile page. + +### Cause 3: a User backup does not contain the local password + +Export identities and role bindings the usual way if that is how users are recreated, and plan to set every local password again on the new platform. Copying `password` hashes forward is not a supported restore. + +### Cause 4: local user passwords use pbkdf2 + +Answer that local user passwords use pbkdf2. No change is needed. + +### Cause 5: the user password settings do not cover every audit item, and lockout applies to local users + +1. Read the policy actually configured on the `global` cluster for your version: + + ```bash + kubectl get userpolicies -A + kubectl get userpolicies -A -o yaml | grep -A40 'name: user-security-policy' + ``` + +2. Report to the audit that a three-month maximum age, a five-generation reuse limit, and lockout after five failures are not provided by the user password settings, unless the policy object of your version shows them. +3. Test a configured lockout with a local user. An LDAP user not being locked is expected and is not a fault. + +## Diagnostic Steps + +```bash +kubectl get password -n cpaas-system -o=custom-columns=Name:.metadata.name,Username:.username,Email:.email +kubectl get user -n cpaas-system | grep +``` + +| What you see | Cause | +| --- | --- | +| Local account, password forgotten, console login rejected | 1 | +| Reset must set a specific password instead of `P@ssw0rd` | 2 | +| Users restored from a backup cannot log in with their old passwords | 3 | +| Question about the algorithm that protects local passwords | 4 | +| Audit asks for password age, history, or lockout; LDAP user not locked | 5 | +| The account is an LDAP user | Reset the password in the directory | diff --git a/docs/en/solutions/GitLab_login_fails_returns_422_or_the_access_address_does_not_work_on_ACP.md b/docs/en/solutions/GitLab_login_fails_returns_422_or_the_access_address_does_not_work_on_ACP.md new file mode 100644 index 000000000..f0e638c2b --- /dev/null +++ b/docs/en/solutions/GitLab_login_fails_returns_422_or_the_access_address_does_not_work_on_ACP.md @@ -0,0 +1,203 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "3.10" + - "4.2" + - "4.3" +--- + +# GitLab login fails, returns 422, or the access address does not work on ACP + +## Issue + +A GitLab instance deployed on the platform shows a normal status and every component is Running, but users cannot get in or the address does not work. Typical reports: + +- Login says `Invalid login or password`, or authentication failed, although the password in the Secret is correct. On a new instance, for example one deployed for a migration, the GitLab log shows `Failed Login: username=root`. +- The administrator can log in, but one user suddenly cannot. +- Login shows `422: The change you requested was rejected. Make sure you have access to the thing you tried to change.` +- After HTTPS is enabled, the instance update fails with `Ingress.networking.k8s.io "-webservice-default" is invalid: spec.rules[0].host: Invalid value: "": must be a DNS name, not an IP address` and `failed rollback: invalid/unknown release server-side apply method`. +- The internal domain is unstable and GitLab should be switched to NodePort, or someone asks whether the Ingress rules in `cpaas-system` can be used to reach GitLab or Harbor. +- The GitLab v4 plugin cannot be switched to Chinese. + +The root account password comes from the Secret referenced by `spec.helmValues.global.initialRootPassword` of the `GitlabOfficial` resource (default name `gitlab-root-password`, key `password`). The access address comes from `spec.helmValues.global.hosts` and the Ingress or NodePort settings of the same resource. + +Related articles: "GitLab toolchain integration fails: token-gitlab not found, Internal error, projects API 500, or an extra bot project" covers a toolchain integration that reports `Your account has been blocked`. + +## Root Cause + +### Cause 1: the configured root password is weak + +GitLab rejects weak root passwords, so login fails although the Secret matches what the user types. Passwords that fail include a word, a symbol, and a year (such as `Password@2024`), anything containing `gitlab`, `root`, or `admin`, and anything that does not meet the complexity rule. The product documentation requires the root password to contain letters, numbers, and special characters, be at least 8 characters long, and not be one of GitLab's blocked weak passwords. Login works after GitLab is redeployed with a complex password. This is the most frequent cause. + +### Cause 2: one user's account is not in an active state + +GitLab works and the administrator logs in, but the error for the affected user indicates the account is not approved or not active. Resetting the user's password refreshes the account state. + +### Cause 3: the client uses a different address than `external_url` + +GitLab is configured for HTTPS with Ingress, and `external_url` is an HTTPS domain plus the Ingress controller's NodePort. Opening GitLab through another address, for example plain HTTP, makes form submissions fail with 422. + +### Cause 4: HTTPS was enabled while the instance used an IP address + +With HTTPS enabled on an instance still configured for IP access, the rendered Ingress uses the IP as `spec.rules[0].host`, which Kubernetes rejects, so both the upgrade and the Helm rollback fail. HTTPS requires access through a domain name. + +### Cause 5: the access method is set in the instance, not in platform Ingress rules + +GitLab offers Ingress with a domain, or NodePort with HTTP and SSH ports. Switching is done in the instance: on the toolchain instance update page, or in the `GitlabOfficial` resource. The access method can be changed after creation; the storage settings (for example node storage with path `/cpaas/devops/gitlab`) cannot. The Ingress rules in `cpaas-system` belong to the operator and cannot be used for GitLab or Harbor domain access. NodePort is not available in the high availability template, which requires Ingress with a domain. + +### Cause 6: the language is a per-user setting + +The GitLab v4 plugin supports Chinese. The management view does not offer a language choice for users; each user sets the language in the user view. + +## Resolution + +### Cause 1: the configured root password is weak + +Choose a password with upper- and lower-case letters, numbers, and special characters that avoids `admin`, `gitlab`, `root`, the user name, and word-plus-year patterns. Redeploy GitLab with it. If the first deployment already initialized the database, rebuild the GitLab database as well. + +> **Warning:** Deleting the `GitlabOfficial` resource and dropping the GitLab database removes all GitLab data: projects, users, and settings. Only for a new instance with no data to keep (or with a backup), where root login fails and the configured password is weak. The database drop is irreversible; the saved YAML restores only the configuration. + +```bash +# 1. Save the instance definition +kubectl -n get gitlabofficial -o yaml > gitlabofficial-.yaml +# 2. Write the new password into the account Secret +kubectl -n create secret generic gitlab-root-password \ + --from-literal=password='' --dry-run=client -o yaml | kubectl apply -f - +# 3. Delete the instance +kubectl -n delete gitlabofficial +# 4. Recreate the GitLab database empty +kubectl -n exec -it -- psql -U postgres \ + -c 'DROP DATABASE IF EXISTS WITH (FORCE);' \ + -c 'CREATE DATABASE OWNER ;' +``` + +Remove `metadata.resourceVersion`, `uid`, `creationTimestamp`, `generation`, `managedFields`, and the `status` block from the saved file, then apply it: + +```bash +kubectl apply -f gitlabofficial-.yaml +``` + +Give each toolchain instance its own PostgreSQL database instead of sharing one across GitLab, SonarQube, and others. GitLab must not use Harbor's PostgreSQL instance; give it a separate instance before the redeploy and update `spec.helmValues.global.psql` and the database Secret. + +### Cause 2: one user's account is not in an active state + +As an administrator, open **Admin Area > Users**, select the user, and set a new password. If the user is shown as pending approval or blocked, approve or unblock the account on the same page. + +### Cause 3: the client uses a different address than `external_url` + +```bash +kubectl -n get gitlabofficial -o jsonpath='{.spec.helmValues.global.hosts}{"\n"}' +kubectl -n get ingress +``` + +Open `https://:`. If the domain is not in DNS, add it to the client `hosts` file, for example `192.0.2.10 gitlab.example.com`, where `192.0.2.10` is a node or VIP that serves the Ingress controller. + +### Cause 4: HTTPS was enabled while the instance used an IP address + +Edit the `GitlabOfficial` resource, replace the IP with a domain in every host field, enable Ingress, then enable TLS: + +```yaml +spec: + helmValues: + gitlab: + webservice: + ingress: + enabled: true + global: + hosts: + domain: gitlab.example.com + gitlab: + hostnameOverride: gitlab.example.com + https: true + name: gitlab.example.com + ssh: gitlab.example.com + ingress: + class: "" + enabled: true + tls: + enabled: true + secretName: +``` + +The TLS Secret is of type `kubernetes.io/tls` in the instance namespace. The instance recovers once the access address is a domain. + +### Cause 5: the access method is set in the instance, not in platform Ingress rules + +On the toolchain instance update page, change the network access mode from `Ingress` to `Node Port`, and the service access address to the node IP and NodePort that clients will use. + +> **Warning:** Switching the access method can restart GitLab components and changes the URL used by users, Git remotes, and CI runners; clones and webhooks that use the old address stop working. Use a maintenance window and update the clients. Rollback: switch back to `Ingress` with the previous address. + +The same change in YAML (`kubectl edit gitlabofficial `), using the NodePort layout from the product documentation: + +```yaml +spec: + helmValues: + gitlab: + gitlab-shell: + service: + nodePort: 30000 # SSH port + type: NodePort + webservice: + ingress: + enabled: false + global: + appConfig: + gitlabPort: 30010 # HTTP port + gitlabNodePort: + http: 30010 + ssh: 30000 + hosts: + domain: 192.0.2.10 # node IP + gitlab: + https: false + name: 192.0.2.10 + ssh: 192.0.2.10 + ingress: + enabled: false + shell: + port: 30000 +``` + +Make sure the ports are free on the nodes. For domain access, use the Cause 4 layout with `https: false` and `tls.enabled: false`, and make the domain resolve to the Ingress entry point. + +### Cause 6: the language is a per-user setting + +1. Open GitLab in the user view as the user who wants Chinese. +2. Open the avatar menu > **Preferences** > **Localization**, and choose `简体中文` (Chinese, Simplified) in **Language**. +3. Select **Save changes** and reload. Each user repeats this for their own account. + +To check the stored value through the API: + +```bash +curl -s -H "PRIVATE-TOKEN: " "https:///api/v4/user/preferences" +``` + +## Diagnostic Steps + +1. Confirm all components are Running: + + ```bash + kubectl -n get pods + ``` + +2. Read the configured root password and check it against Cause 1, and look for failed logins: + + ```bash + SECRET=$(kubectl -n get gitlabofficial -o jsonpath='{.spec.helmValues.global.initialRootPassword.secret}') + kubectl -n get secret "$SECRET" -o jsonpath='{.data.password}' | base64 -d; echo + kubectl -n logs deploy/-webservice-default -c webservice --tail=500 | grep -i 'Failed Login' + ``` + +3. If root works and one user fails, go to Cause 2. +4. Compare the browser address with `spec.helmValues.global.hosts` and the Ingress NodePort (Cause 3). +5. For a failed update after enabling HTTPS, check the host fields: + + ```bash + kubectl -n get gitlabofficial -o yaml | grep -n -A3 'hosts:' + ``` + + An IP in `hosts.gitlab.name` with `https: true` is Cause 4. +6. A missing Chinese option in the management view is Cause 6. diff --git a/docs/en/solutions/GitLab_toolchain_integration_fails_token-gitlab_not_found_Internal_error_projects_API_500_or_an_extra_bot_project.md b/docs/en/solutions/GitLab_toolchain_integration_fails_token-gitlab_not_found_Internal_error_projects_API_500_or_an_extra_bot_project.md index a925bc5ce..fdef6d634 100644 --- a/docs/en/solutions/GitLab_toolchain_integration_fails_token-gitlab_not_found_Internal_error_projects_API_500_or_an_extra_bot_project.md +++ b/docs/en/solutions/GitLab_toolchain_integration_fails_token-gitlab_not_found_Internal_error_projects_API_500_or_an_extra_bot_project.md @@ -19,6 +19,7 @@ A GitLab toolchain integration of DevOps v3 is configured and does not work in p - After the GitLab credential is updated, requests intermittently return `Internal error`. - `GET /integrations/v1alpha1/namespaces//integrations//projects/` returns `500 (Internal Server Error)` on every call. - After GitLab projects are assigned, the code management list shows an extra entry named like `project__bot_` that does not exist as a project in GitLab. +- The integration instance shows an abnormal status with `gitlab returned error: GET .../api/v4/todos: 403 {message: 403 Forbidden - Your account has been blocked.}`. These integrations are served by katanomi (`katanomi-api` and `katanomi-plugin` in `cpaas-system`). Each platform project is expected to have a namespace of the same name in the cluster. Related articles: "Harbor toolchain integration fails: images not selectable, katanomi-plugin timeout, or push asks for credentials" covers the Harbor integration. @@ -44,6 +45,10 @@ The DNS server configured on the node is malfunctioning. katanomi cannot resolve When an external GitLab is integrated with root permissions and projects are assigned, the code management list of the project can show an extra entry named like `project_62_bot_`, labelled as platform-assigned, whose code repository tab is empty. GitLab creates a bot user of that name when a project access token is created for project ID ``. The platform syncs GitLab user (personal) namespaces as well as groups, so the bot user's personal namespace is synced as a tool project with `subType: GitUser`. The entry does not affect usage. +### Cause 6: `403 Forbidden - Your account has been blocked` because the credential password no longer matches the GitLab account + +The password stored in the integration's credential no longer matches the GitLab account's current password, for example because the password was changed in GitLab. GitLab rejects the integration's API requests with `403 Forbidden - Your account has been blocked.`, and logging in to the GitLab web console with the same account and password returns `Invalid login or password.` + ## Resolution ### Cause 1: `Secret "token-gitlab" not found` because the integration uses the external GitLab address @@ -118,6 +123,13 @@ kubectl get projects.meta.katanomi.dev -A -o yaml | grep -n -B4 -A8 '_bot_' The object shows `subType: GitUser`. In GitLab, open `/projects/` for the ID in the name; it redirects to the project whose access token created the bot. No action is required. The DevOps v4 GitLab integration, available from ACP 4.1, is the alternative when this listing is not wanted. +### Cause 6: `403 Forbidden - Your account has been blocked` because the credential password no longer matches the GitLab account + +1. Log in to the GitLab web console with the account and password stored in the integration's credential. `Invalid login or password.` confirms the mismatch. +2. Find out the account's current password from its owner, or reset it in GitLab. +3. Update the credential used by the integration with the current password, then check the integration status again. It returns to normal once GitLab accepts the credential. +4. If GitLab still answers `Your account has been blocked`, a GitLab administrator checks the account in **Admin Area > Users** and unblocks it. + ## Diagnostic Steps ```bash @@ -132,6 +144,7 @@ Capture the failing request in the browser developer tools; the path and the sta - Intermittent `Internal error` after a credential update: Cause 3. - `500` on the integration projects API, and DNS resolution of the GitLab host fails on the node: Cause 4. - An extra `project__bot_` tool project with subtype `GitUser`: Cause 5. +- `403 Forbidden - Your account has been blocked` from the integration, and the same account gets `Invalid login or password.` on the GitLab login page: Cause 6. When katanomi logs domain resolution timeouts for a platform Service name such as `katanomi-plugin.cpaas-system`, rather than for the GitLab host, the fault is cluster DNS, not GitLab. Check the node `/etc/resolv.conf` for a `search` line with site domains, and the CoreDNS log for that Service name being forwarded upstream: diff --git a/docs/en/solutions/Java_auto-instrumentation_init_container_is_missing_or_the_Java_agent_lands_next_to_istio-proxy_on_Service_Mesh_v2.md b/docs/en/solutions/Java_auto-instrumentation_init_container_is_missing_or_the_Java_agent_lands_next_to_istio-proxy_on_Service_Mesh_v2.md new file mode 100644 index 000000000..54f4dedde --- /dev/null +++ b/docs/en/solutions/Java_auto-instrumentation_init_container_is_missing_or_the_Java_agent_lands_next_to_istio-proxy_on_Service_Mesh_v2.md @@ -0,0 +1,163 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Service Mesh +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Java auto-instrumentation init container is missing or the Java agent lands next to istio-proxy on Service Mesh v2 + +## Issue + +Non-intrusive (automatic) tracing is configured for a Java application, but Jaeger shows no traces from that Pod. One of these is seen: + +- The Pod has no `opentelemetry-auto-instrumentation-java` init container. +- Both the Istio sidecar and the OpenTelemetry Java agent are injected, but `JAVA_TOOL_OPTIONS` is set on `istio-proxy` instead of the business container. +- Business logs carry no TraceID, so a log line cannot be matched to a trace in the Jaeger UI. +- The question is whether the platform's OpenTelemetry Operator can auto-instrument Nginx. + +The Alauda Build of OpenTelemetry v2 operator injects the agent through a pod-mutating admission webhook when the pod template carries an `instrumentation.opentelemetry.io/inject-java` annotation that points at an `Instrumentation` resource. + +Related articles: "asm-otel collectors keep restarting, Service Mesh JVM monitoring has no data, or an otel instance cannot be deleted" covers an agent that is injected but cannot reach the Collector, and missing JVM metrics. + +## Root Cause + +### Cause 1: the OpenTelemetry v2 integration was not completed in order + +The integration has an order: install Jaeger and the OpenTelemetry Collector, connect them to Elasticsearch or OpenSearch, create an `Instrumentation`, and only then annotate the Java workload. When any earlier step is missing, the webhook does not generate the `opentelemetry-auto-instrumentation-java` init container. Completing the whole sequence resolves it. + +### Cause 2: on Mesh v2 with Kubernetes earlier than 1.33, the agent targets the sidecar + +On Mesh v2 and Kubernetes versions earlier than 1.33, the business container that should receive the Java agent has to be named with an annotation; otherwise injection can apply to the Istio sidecar. From Kubernetes 1.33, sidecar init containers are supported and this does not happen. On Mesh v1, simultaneous injection of the Java agent and the Istio sidecar is supported. + +### Cause 3: the TraceID is created by the Java agent and has to be written into the log pattern + +Jaeger does not generate TraceIDs. The OpenTelemetry Java agent generates and propagates them. The Istio sidecar can generate a TraceID, but propagation across calls must then be handled by the application. The TraceID appears in business logs only when the log pattern prints it. + +### Cause 4: Nginx auto-instrumentation is enabled in `opentelemetry-operator2.v0.157.0-r1` + +The operator build `opentelemetry-operator2.v0.157.0-r1` sets `ENABLE_NGINX_AUTO_INSTRUMENTATION` to `"true"` (next to `ENABLE_GO_AUTO_INSTRUMENTATION: "true"`). The implementation is the upstream OpenTelemetry Operator Nginx auto-instrumentation of v0.157.0. + +## Resolution + +### Cause 1: the OpenTelemetry v2 integration was not completed in order + +1. Check each step and complete the first one that is missing: + 1. The OpenTelemetry v2 operator is installed and its webhook is registered. + 2. Jaeger and the Collector are deployed and connected to Elasticsearch or OpenSearch. Jaeger v2 is deployed by the OpenTelemetry v2 operator; a separate Alauda Build of Jaeger v2 install is not needed. + 3. An `Instrumentation` exists in the application namespace, or the annotation references one in another namespace as `/`. + 4. The pod template carries the injection annotation. + + ```bash + kubectl get csv -A | grep -i opentelemetry + kubectl get mutatingwebhookconfiguration | grep -i opentelemetry + kubectl get opentelemetrycollector -A + kubectl get pods -A | grep -i -E 'jaeger|otel|collector' + kubectl get instrumentation -n + ``` + +2. Create the `Instrumentation`, pointing at the Collector Service. Use the `.` form; a full name with a wrong cluster domain does not resolve. + + ```yaml + apiVersion: opentelemetry.io/v1alpha1 + kind: Instrumentation + metadata: + name: java-instrumentation + namespace: + spec: + exporter: + endpoint: http://.:4318 + propagators: + - tracecontext + - baggage + ``` + + ```bash + kubectl apply -f java-instrumentation.yaml + ``` + +3. Annotate the workload. + + > **Warning:** Patching the pod template restarts the workload's Pods. Only after steps 1.1 to 1.3 are in place. Rollback: set the annotation to `null` with the same patch, which restarts the Pods without the agent. + + ```bash + kubectl -n patch deployment --type=merge \ + -p '{"spec":{"template":{"metadata":{"annotations":{"instrumentation.opentelemetry.io/inject-java":"java-instrumentation"}}}}}' + kubectl -n get pod -o jsonpath='{.spec.initContainers[*].name}{"\n"}' + ``` + + The new Pod should list `opentelemetry-auto-instrumentation-java` among its init containers. + +### Cause 2: on Mesh v2 with Kubernetes earlier than 1.33, the agent targets the sidecar + +1. Check the Kubernetes version and the container names: + + ```bash + kubectl version + kubectl -n get deploy -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\n"}{end}' + ``` + +2. Below 1.33, name the business container with `instrumentation.opentelemetry.io/container-names`. + + > **Warning:** The Pods restart with the agent attached only to the named container. Only on Mesh v2 with Kubernetes earlier than 1.33 and both the sidecar and the agent injected. Rollback: set `instrumentation.opentelemetry.io/container-names` to `null`. + + ```bash + kubectl -n patch deployment --type=merge \ + -p '{"spec":{"template":{"metadata":{"annotations":{"instrumentation.opentelemetry.io/inject-java":"java-instrumentation","instrumentation.opentelemetry.io/container-names":""}}}}}' + ``` + + On Kubernetes 1.33 and later, and on Mesh v1, this annotation is not needed. + +### Cause 3: the TraceID is created by the Java agent and has to be written into the log pattern + +After the agent is injected, add the TraceID to the business log pattern. For Spring Boot logging: + +```properties +logging.pattern.level = trace_id=%mdc{trace_id} +``` + +Search the TraceID from a log line in the Jaeger UI to open the full call chain. For the agent's overhead, use the OpenTelemetry Java agent performance test report as the reference; no single figure applies to every application. + +### Cause 4: Nginx auto-instrumentation is enabled in `opentelemetry-operator2.v0.157.0-r1` + +1. Confirm the switch on the installed operator: + + ```bash + kubectl get csv -A | grep opentelemetry-operator2 + kubectl get csv -n -o yaml | grep -A1 -E 'ENABLE_NGINX_AUTO_INSTRUMENTATION|ENABLE_GO_AUTO_INSTRUMENTATION' + ``` + +2. Follow the upstream Nginx auto-instrumentation document for v0.157.0: the `Instrumentation` carries an `nginx` section (optionally `configFile` and `attrs`), and the workload opts in with `instrumentation.opentelemetry.io/inject-nginx: "true"`. Check the supported Nginx versions and the expected configuration layout (`/etc/nginx/nginx.conf` with a `conf.d` include in the `http` block, unless `configFile` says otherwise) against the image first. + + > **Warning:** The Pods restart with the Nginx module injected. Only when an `Instrumentation` with an `nginx` section exists in the namespace and the image meets the upstream requirements. Rollback: set the annotation to `null`. + + ```bash + kubectl -n patch deployment --type=merge \ + -p '{"spec":{"template":{"metadata":{"annotations":{"instrumentation.opentelemetry.io/inject-nginx":"true"}}}}}' + ``` + +## Diagnostic Steps + +1. Look at what was injected into a running Pod: + + ```bash + kubectl -n get pod -o jsonpath='{.metadata.annotations}{"\n"}{.spec.initContainers[*].name}{"\n"}' + kubectl -n get pod -o jsonpath='{range .spec.containers[*]}{.name}{" "}{.env[?(@.name=="JAVA_TOOL_OPTIONS")].value}{"\n"}{end}' + ``` + +2. Read the operator log for the Pod's admission: + + ```bash + kubectl -n logs --tail=200 | grep -i -E 'inject|instrumentation' + ``` + +| What you see | Cause | +| --- | --- | +| No `opentelemetry-auto-instrumentation-java` init container; Collector, Jaeger, storage, or `Instrumentation` missing | 1 | +| `JAVA_TOOL_OPTIONS` on `istio-proxy`, Kubernetes earlier than 1.33 | 2 | +| Agent injected, logs have no TraceID | 3 | +| Nginx auto-instrumentation wanted on `opentelemetry-operator2.v0.157.0-r1` | 4 | +| Agent injected into the business container, init container `terminated`, application log shows `Failed to export ... timeout` | The asm-otel article, Cause 4 | diff --git a/docs/en/solutions/Leftover_Service_Mesh_v1_recreates_istio-system_against_Mesh_v2_or_Istio_Secret_certificate_alerts_remain_on_ACP.md b/docs/en/solutions/Leftover_Service_Mesh_v1_recreates_istio-system_against_Mesh_v2_or_Istio_Secret_certificate_alerts_remain_on_ACP.md new file mode 100644 index 000000000..deb786cb8 --- /dev/null +++ b/docs/en/solutions/Leftover_Service_Mesh_v1_recreates_istio-system_against_Mesh_v2_or_Istio_Secret_certificate_alerts_remain_on_ACP.md @@ -0,0 +1,118 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Service Mesh +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Leftover Service Mesh v1 recreates istio-system against Mesh v2 or Istio Secret certificate alerts remain on ACP + +## Issue + +After Service Mesh v1 (ASM v1) was removed, or while Mesh v2 is being installed on a cluster that used to run v1, one of these is seen: + +- Creating Istio-type resources for Mesh v2 fails. Namespace `istio-system` and ServiceAccount `istio-reader-service-account` reappear on the business cluster on their own after they are deleted. +- Certificate monitoring keeps reporting that Istio Secrets are expired or about to expire, although ASM v1 has been uninstalled. + +Mesh v1 and Mesh v2 must not be deployed on one cluster at the same time; two Istio installs cannot coexist. Mesh v2 support starts with platform 4.1.1, ASM v1 is no longer maintained, and Service Mesh v1 is removed in platform 4.4. + +Related articles: "Service Mesh delete is stuck on a failed to get cluster webhook denial or on pods that still have sidecar injection" covers a delete that does not finish. "Platform certificate expiry alert for a cert-manager serving certificate that renews by itself on ACP" covers Istio CA alerts of a mesh that is still installed. + +## Root Cause + +### Cause 1: Mesh v1 `servicemesh` and `servicemeshgroup` are still on `global` + +The `global` cluster still holds Mesh v1 `servicemesh` and `servicemeshgroup` objects. `global-asm-controller` sees them and keeps creating `istio-system` and related objects on the business cluster. Those objects collide with the Mesh v2 install. + +### Cause 2: Istio Secrets left behind by the ASM v1 uninstall + +The Secrets named in the alert are leftovers of the ASM v1 uninstall, not the certificate of a live mesh that failed to renew. Nothing renews them any more, so the alert keeps firing until they are removed. + +## Resolution + +### Cause 1: Mesh v1 `servicemesh` and `servicemeshgroup` are still on `global` + +1. On `global`, confirm the v1 objects and find the controller; on the business cluster, back up the Mesh v2 objects: + + ```bash + # global cluster + kubectl get servicemesh,servicemeshgroup -A + kubectl get servicemesh,servicemeshgroup -A -o yaml > /tmp/mesh-v1-leftovers.bak.yaml + kubectl get deploy -A | grep global-asm-controller + # business cluster + kubectl get istio,istiocni -A -o yaml > /tmp/mesh-v2.bak.yaml + ``` + +2. Delete the v1 objects on `global`. + + > **Warning:** Mesh v1 reconciliation stops for every business cluster these objects point at. Only when the v1 objects exist and `istio-system` is being recreated against a v2 install, and the backup exists. Rollback: `kubectl apply -f /tmp/mesh-v1-leftovers.bak.yaml` if v1 is still needed and v2 is not installed. If the delete is denied with `failed to get cluster`, follow the stuck-delete article first. + + ```bash + kubectl delete servicemesh -n + kubectl delete servicemeshgroup -n + ``` + +3. On the business cluster, delete the conflicting Mesh v2 resources and `istio-system`. + + > **Warning:** This removes the mesh control plane, gateways, and sidecar injection on that cluster; meshed calls fail until Mesh v2 is installed again. Only after step 2 is done and `/tmp/mesh-v2.bak.yaml` exists; use a maintenance window. Rollback: install Mesh v2 again; use the backup only as a reference for its settings. + + ```bash + kubectl delete istio -n + kubectl delete istiocni + kubectl delete namespace istio-system + ``` + +4. Install Mesh v2 again only after `kubectl get servicemesh,servicemeshgroup -A` on `global` shows nothing for this cluster and `istio-system` is no longer recreated. + +### Cause 2: Istio Secrets left behind by the ASM v1 uninstall + +1. List the Secrets the alert names and confirm that no mesh still uses them: + + ```bash + kubectl get secret -A | grep -Ei 'istio|cacert' + kubectl get servicemesh,istio,istiocni -A + kubectl get pods -A | grep -Ei 'istiod|istio-operator|asm-controller' + ``` + + A Secret referenced by a running Mesh v2 control plane, or by a mesh that is still installed, is not a leftover; leave it. + +2. Back up and delete each leftover Secret. + + > **Warning:** Deleting a Secret that a live control plane uses breaks that mesh's certificates. Only when ASM v1 is uninstalled, no v1 `servicemesh` remains, and no running `istiod` mounts the Secret. The backup contains a private key; store it securely. Rollback: `kubectl apply -f /tmp/istio-secret-.bak.yaml`. + + ```bash + kubectl get secret -n -o yaml > /tmp/istio-secret-.bak.yaml + kubectl delete secret -n + ``` + +3. Repeat for each Secret the alert names. The certificate alerts stop after the leftovers are gone. + +## Diagnostic Steps + +1. On `global`: + + ```bash + kubectl get servicemesh,servicemeshgroup -A + kubectl get deploy -A | grep -E 'global-asm-controller|asm-controller' + ``` + +2. On the business cluster: + + ```bash + kubectl get istio,istiocni -A + kubectl get ns istio-system + kubectl get sa -n istio-system istio-reader-service-account + kubectl get pods -n istio-system + ``` + +| What you see | Cause | +| --- | --- | +| `istio-system` or `istio-reader-service-account` returns after deletion, `global` still has `servicemesh` or `servicemeshgroup`, Mesh v2 resources fail to create | 1 | +| Certificate alert on Istio Secrets after ASM v1 is uninstalled, no running `istiod` uses them | 2 | +| Alert on `asm-istio-cacerts-` for a mesh that is still installed | Not a leftover; see the certificate expiry article | diff --git a/docs/en/solutions/Moving_Service_Mesh_v2_tracing_storage_from_Elasticsearch_to_OpenSearch_with_Jaeger_v2.md b/docs/en/solutions/Moving_Service_Mesh_v2_tracing_storage_from_Elasticsearch_to_OpenSearch_with_Jaeger_v2.md new file mode 100644 index 000000000..70d0065d1 --- /dev/null +++ b/docs/en/solutions/Moving_Service_Mesh_v2_tracing_storage_from_Elasticsearch_to_OpenSearch_with_Jaeger_v2.md @@ -0,0 +1,205 @@ +--- +kind: + - How To +products: + - Alauda Service Mesh +ProductsVersion: + - "4.2" + - "4.3" + - "4.4" +--- + +# Moving Service Mesh v2 tracing storage from Elasticsearch to OpenSearch with Jaeger v2 + +## Issue + +A Service Mesh v2 tracing pipeline stores traces in Elasticsearch and has to move to OpenSearch. The questions are whether the move affects business traffic, which component versions fit together, and what the steps are. Related questions: + +- Is Alauda Build of Jaeger v2 now deployed independently of ASM, and how does the mesh send traces to it? +- Which Elasticsearch version does the tracing feature support, for example whether an existing ES7 can be used? +- Must the host that runs the Jaeger v2 installation have `jq`? + +The layout described for Cause 1 is that of ACP 4.2.2: an OpenTelemetry Collector sends traces to a Jaeger v2 service in `jaeger-system` that writes to Elasticsearch. The Alauda Build of OpenTelemetry v2 operator (adapted to ACP 4.2) deploys Jaeger v2; OpenSearch Operator v2.8 from the App Marketplace deploys OpenSearch 3.x. + +## Root Cause + +### Cause 1: the tracing pipeline uses the legacy mesh integration with Elasticsearch storage + +Moving to OpenSearch means deploying a Jaeger v2 instance integrated with OpenSearch and pointing the OpenTelemetry Collector at it. Switching the backend does not affect business traffic; a small amount of trace data can be lost during the switch. + +### Cause 2: Jaeger v2 is the standalone product Alauda Distributed Tracing + +Alauda Build of Jaeger v2 is installed and run as the standalone product Alauda Distributed Tracing; it is no longer part of ASM. ASM only exports Istio spans to the deployed Jaeger v2 and OpenTelemetry Collector. One tracing stack can serve both the mesh and other workloads in the cluster; its default namespace is `jaeger-system`. The old stack, Alauda Build of Jaeger (Jaeger 1.60.0) plus Alauda Build of OpenTelemetry, is deprecated in ACP 4.4 and will be removed; the old Distributed Tracing pages are removed in ACP 4.4, with ACP 4.3 as the last version that contains them. + +### Cause 3: Elasticsearch 8.x is the supported version + +The Alauda Distributed Tracing documentation (Configuration > Storage Backend > Elasticsearch) lists Elasticsearch 8.x as supported. Jaeger detects the version from the Elasticsearch root endpoint and then uses the matching index mappings and REST APIs; the version can also be set explicitly with the `version` property of the Elasticsearch storage configuration. ES7 is not listed as supported. + +### Cause 4: the Jaeger v2 installation depends on `jq` + +The Jaeger v2 installation depends on `jq`, so the host that runs the installation must have it. + +## Resolution + +### Cause 1: the tracing pipeline uses the legacy mesh integration with Elasticsearch storage + +Install OpenSearch Operator v2.8 from the App Marketplace and create an OpenSearch 3.x cluster. Back up the Collector configuration: + +```bash +kubectl get opentelemetrycollector -A +kubectl -n get opentelemetrycollector -o yaml > /tmp/otel-collector.bak.yaml +kubectl -n jaeger-system get all +``` + +Then choose one of the three paths. + +#### Path A: historical traces are not needed + +1. Deploy a new Jaeger v2 service integrated with OpenSearch, following the Jaeger v2 deployment procedure of the OpenTelemetry v2 operator. Set `JAEGER_INSTANCE_NAME` to a value other than `jaeger`, for example `jaeger-opensearch`, so it does not collide with the old instance. +2. Point the Collector exporter at the new service; copy TLS and other settings from the exporter that points at the old Jaeger. + + ```yaml + spec: + config: + exporters: + otlp/jaeger-opensearch: + endpoint: ..svc:4317 + service: + pipelines: + traces: + exporters: + - otlp/jaeger-opensearch + ``` + + > **Warning:** The Collector restarts and spans in flight can be lost. Only when the new Jaeger v2 is Running and connected to OpenSearch, and the backup exists. Rollback: `kubectl apply -f /tmp/otel-collector.bak.yaml`. + + ```bash + kubectl -n edit opentelemetrycollector + kubectl -n get pods -w + ``` + +3. Confirm new traces in the new Jaeger UI, then uninstall the old Jaeger v2 service in `jaeger-system` with the uninstall step of the procedure that deployed it. + + > **Warning:** Traces stored through the old Jaeger can no longer be queried once it is removed. Only after new traces are confirmed and only when historical traces are not needed. Rollback: reinstall the old Jaeger against Elasticsearch to read old traces. + +#### Path B: historical traces must be kept for a period + +Deploy the new Jaeger v2 as in Path A. Add a second traces pipeline so data goes to both services; keep the receivers and processors of the existing pipeline: + +```yaml +spec: + config: + exporters: + otlp/jaeger: + endpoint: .jaeger-system.svc:4317 + otlp/jaeger-opensearch: + endpoint: ..svc:4317 + service: + pipelines: + traces: + receivers: [otlp] + processors: [batch] + exporters: [otlp/jaeger] + traces/opensearch: + receivers: [otlp] + processors: [batch] + exporters: [otlp/jaeger-opensearch] +``` + +Apply it with `kubectl edit` under the same Warning as Path A step 2. After the retention period, remove the `traces` pipeline and the `otlp/jaeger` exporter, and uninstall the old service under the Warning of Path A step 3. + +#### Path C: test environment where a short trace gap is acceptable + +Uninstall the old Jaeger v2 first, then deploy the new one with `JAEGER_INSTANCE_NAME=jaeger`, integrated with OpenSearch. Because the name stays `jaeger`, the Collector exporter can stay unchanged; confirm the Service it points to exists after the new instance is Ready. + +> **Warning:** No traces are stored between the uninstall and the new instance becoming Ready, and old traces are no longer reachable. Test environments only. Rollback: reinstall Jaeger v2 against Elasticsearch to read old traces. + +```bash +kubectl -n jaeger-system get opentelemetrycollector,deploy,svc +kubectl -n jaeger-system get svc -o custom-columns='NAME:.metadata.name,PORTS:.spec.ports[*].port' +``` + +### Cause 2: Jaeger v2 is the standalone product Alauda Distributed Tracing + +1. For a new deployment, follow the current Alauda Distributed Tracing installation guide for the backend you use (Elasticsearch or OpenSearch) instead of the deprecated tracing documents; existing deployments follow its migration guide. The order is: + 1. Cluster plugin Alauda Build of Jaeger v2. + 2. OpenTelemetry v2 Operator. + 3. Deploy Jaeger v2. + 4. Deploy the Collector. + + ```bash + kubectl get pods -n jaeger-system + kubectl get svc -n jaeger-system | grep -E 'otel-collector|jaeger' + ``` + +2. To send mesh traces, add an `extensionProviders` entry on the `Istio` resource that points at the Collector, and reference it from a `Telemetry` resource. The provider name is an example and the sampling percentage is a site choice. If the cluster DNS domain is not `cluster.local`, use the cluster's domain in the service name. + + > **Warning:** istiod pushes new configuration to every proxy, and a `Telemetry` in `istio-system` changes tracing for the whole mesh. Copy existing `extensionProviders` entries into the patch, because a merge patch replaces the whole list. Rollback: `kubectl apply -f /tmp/istio-tracing.bak.yaml` (remove `metadata.resourceVersion` on conflict) and `kubectl delete telemetry -n istio-system mesh-default`. + + ```bash + kubectl get istio -o yaml > /tmp/istio-tracing.bak.yaml + kubectl get istio -o jsonpath='{.spec.values.meshConfig.extensionProviders}{"\n"}' + kubectl patch istio --type=merge -p '{"spec":{"values":{"meshConfig":{"extensionProviders":[{"name":"otel-tracing","opentelemetry":{"service":"otel-collector.jaeger-system.svc.cluster.local","port":4317}}]}}}}' + ``` + + ```yaml + apiVersion: telemetry.istio.io/v1 + kind: Telemetry + metadata: + name: mesh-default + namespace: istio-system + spec: + tracing: + - providers: + - name: otel-tracing + randomSamplingPercentage: 10.0 + ``` + + ```bash + kubectl apply -f mesh-default-telemetry.yaml + ``` + +3. To remove the mesh integration later, remove the `Istio` and `Telemetry` configuration first; leave Jaeger and the Collector in place while other workloads still use tracing. + +### Cause 3: Elasticsearch 8.x is the supported version + +Use Elasticsearch 8.x. If only ES7 is available, it can be tested, but it is not a supported combination. Read the version Jaeger will detect: + +```bash +curl -s -k -u ':' https://:9200/ | grep '"number"' +``` + +### Cause 4: the Jaeger v2 installation depends on `jq` + +Install `jq` on the host that runs the installation, then rerun the step: + +```bash +command -v jq || echo 'jq is missing' +yum install -y jq # or: apt-get install -y jq; use an internal mirror on an offline host +jq --version +``` + +## Diagnostic Steps + +1. Confirm the layout before starting: + + ```bash + kubectl get csv -A | grep -i -E 'opentelemetry|opensearch' + kubectl get pods -n jaeger-system + kubectl -n get opentelemetrycollector -o jsonpath='{.spec.config}{"\n"}' + ``` + +2. After the switch, check that the Collector exports without errors and the new Jaeger receives data: + + ```bash + kubectl -n logs --tail=200 | grep -i -E 'error|export' + kubectl -n logs --tail=200 | grep -i -E 'opensearch|error' + ``` + +| What you see | Next step | +| --- | --- | +| Collector logs export errors to the new endpoint | Check the Service name and port in the exporter | +| New Jaeger logs OpenSearch connection errors | Check the OpenSearch address and credentials given to Jaeger | +| No mesh spans in the standalone Jaeger v2 | Cause 2: check `extensionProviders` on `Istio` and the `Telemetry` provider name | +| Elasticsearch root endpoint reports a 7.x `number` | Cause 3 | +| Installation stops because `jq` is not found | Cause 4 | diff --git a/docs/en/solutions/Namespace_quota_exceeds_the_project_quota_a_project_shows_the_whole_clusters_resources_or_system_namespaces_on_ACP.md b/docs/en/solutions/Namespace_quota_exceeds_the_project_quota_a_project_shows_the_whole_clusters_resources_or_system_namespaces_on_ACP.md new file mode 100644 index 000000000..be4fa27d5 --- /dev/null +++ b/docs/en/solutions/Namespace_quota_exceeds_the_project_quota_a_project_shows_the_whole_clusters_resources_or_system_namespaces_on_ACP.md @@ -0,0 +1,141 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.1" + - "4.1.4" + - "4.2" + - "4.3" +--- + +# Namespace quota exceeds the project quota, a project shows the whole cluster's resources, or system namespaces on ACP + +## Issue + +The project view shows quota, resources, or namespaces that do not match what the project should own. Typical questions: + +- "The project has a resource quota, but a namespace created in it has no limit." The namespace quota can then exceed the project quota, and the console does not force a quota when the namespace is created. +- A namespace quota change is rejected. +- "The `cpaas-system` quota is allocated unreasonably and cannot be managed." `cpaas-system` appears with a project and quota figures. +- "How do I give a user `kube-system` access?" The project namespace list shows `命名空间 (0)` (Namespaces (0)) with `无可用命名空间,请联系管理员创建命名空间或添加已有命名空间权限` (no namespace available, ask an administrator), and a chart deployed from Platform Management > App Store Management > Template Repository offers only `default`. +- "How do I dedicate a few nodes to one project?" After nodes are dedicated, a namespace with no project quota still shows the whole cluster's resources. +- "Which namespaces on this cluster are built in by the platform?" A security scan flags `acp-storage` and `argo-rollouts` and asks whether they can be removed. + +## Root Cause + +### Cause 1: namespace quota fields are optional, and a namespace can be created without a quota limit + +Project quota (`projectquota.auth.alauda.io`) is enforced only when a namespace `ResourceQuota` is created or updated: the sum of the same keys across the project's namespaces must stay within the project's hard limits. On the namespace creation page the resource quota is optional; a resource type with no quota is unbounded, so the namespace can consume all of that resource allocated to the project. A key that is not written into the namespace `ResourceQuota` is not limited by the project quota. Separately, creating a namespace in a project that has a quota can intermittently produce a namespace with no quota limit at all; this is a known product defect with a fix planned for 4.3, and no workaround is available for it. + +### Cause 2: a namespace quota change that would exceed the project quota is rejected + +Namespace quotas are carved out of the project quota. A change that would push the project total above its hard limit is rejected; the project quota has to be raised first. + +### Cause 3: the `cpaas-system` project shown from the cluster view is display only + +There is no real `cpaas-system` project. When `cpaas-system` is opened from a cluster's container groups, the project shown is for display only, and its overview and resource limits can be ignored. + +### Cause 4: `kube-system` and `cpaas-system` are not offered as project namespaces by design + +System namespaces such as `kube-system` and `cpaas-system` are not selectable in the project namespace list or the template deployment form. Making them selectable is a feature request, not a setting. + +### Cause 5: dedicated nodes are a Node Isolation Policy, and resource figures follow quota, not nodes + +Nodes are dedicated to a project with a Node Isolation Policy (Platform Management > Security Management) with **Project Exclusive** (`项目独占`) on: only the projects in the policy may use the nodes. Resource pages on the container platform show resources by project and namespace, not by node. A project without a quota can use the whole cluster, so its namespace shows the whole cluster's totals even when its Pods can land only on the dedicated nodes. Project exclusivity and project quota can be used together. This was confirmed on 4.1.4. + +### Cause 6: platform namespaces depend on which components are installed + +There is no single official list of platform namespaces. Common ones: + +- Always present: `cpaas-system` (platform components) and `kube-system` (Kubernetes system components), plus the Kubernetes defaults `default`, `kube-public`, and `kube-node-lease`. +- Present when the related component is deployed: `argocd`, `cert-manager` (certificate management), `katanomi-system` (pipeline katanomi components), `kubevirt` (virtualization), `rook-ceph` (Ceph storage), `nativestor-system` (TopoLVM local storage), `operators` (pipeline-related operators such as `devops-tool-operator-controller-manager`, `harbor-operator-controller-manager`, `katanomi-api`), `rds-system` and `redis` (data services), `argo-rollouts` (default namespace of Alauda Build of Argo Rollouts), and `acp-storage` (default namespace of Alauda Build of Local Storage and of Alauda Container Platform Storage Essentials). + +These are system-component namespaces, not business namespaces to delete. + +## Resolution + +### Cause 1: namespace quota fields are optional, and a namespace can be created without a quota limit + +1. Find the project's namespaces that have no `ResourceQuota`, and compare the keys of an existing quota with the project's: + + ```bash + PROJECT= + for ns in $(kubectl get namespace -l "cpaas.io/project=${PROJECT}" -o name | cut -d/ -f2); do + printf '%s ' "$ns"; kubectl get resourcequota -n "$ns" --no-headers 2>/dev/null | wc -l + done + kubectl get projectquota -l "cpaas.io/project=${PROJECT}" -o yaml + kubectl get resourcequota -n -o yaml + ``` + + `0` means no `ResourceQuota`; a quota whose `hard` lacks `requests.cpu`, `limits.cpu`, `requests.memory`, or `limits.memory` leaves those resources unbounded. + +2. Set the missing values in the console as a normal namespace quota update (Project Management > Namespaces > the namespace > resource quota). The platform rejects a value above what the project has left. When creating namespaces, fill in requests and limits instead of leaving them unlimited. +3. For the intermittent defect where a namespace in a quota-limited project is created with no quota limit, upgrade to 4.3, where the fix is planned. + +### Cause 2: a namespace quota change that would exceed the project quota is rejected + +Raise the project quota on the project details page first, or reduce other namespaces' quotas, then change the namespace quota: + +```bash +kubectl get projectquota -l "cpaas.io/project=" -o yaml +kubectl get resourcequota -n -o yaml +``` + +### Cause 3: the `cpaas-system` project shown from the cluster view is display only + +No action is needed. + +### Cause 4: `kube-system` and `cpaas-system` are not offered as project namespaces by design + +No setting changes this. Submit a feature request if it is needed. + +### Cause 5: dedicated nodes are a Node Isolation Policy, and resource figures follow quota, not nodes + +1. Go to Platform Management > Security Management > Node Isolation Policy > **Create**. Enter a name, turn on **Project Exclusive** (`项目独占`), select the project and the nodes, read the confirmation dialog, and create the policy. + + > **Warning:** Existing Pods of the project are moved onto the policy's nodes only when they are rebuilt; with Project Exclusive on, Pods of other projects already on those nodes are not evicted automatically. Only when these nodes should serve this project alone; plan the rescheduling in a maintenance window. Rollback: delete the policy, or turn Project Exclusive off. + +2. Find workloads of other projects still running on a dedicated node and restart them so they are scheduled elsewhere: + + ```bash + kubectl get pods -A -o wide --field-selector spec.nodeName= + ``` + + > **Warning:** A restart recreates all Pods of the workload; a single-replica workload is briefly unavailable. Only for other projects' workloads still on the dedicated nodes. Rollback: none needed. + + ```bash + kubectl rollout restart deployment/ -n + ``` + +3. To see the project's own capacity, set a project quota sized to the total resources of the dedicated nodes, then view it under Project Management > Namespaces. Node status is checked in the cluster view. + +### Cause 6: platform namespaces depend on which components are installed + +For a namespace not in the list, look at what runs in it and how it is labeled: + +```bash +kubectl get pods -n +kubectl get namespace --show-labels +kubectl get deploy,sts,ds -n +``` + +Pods named after platform components (for example `topolvm-*` or `katanomi-*`) identify a platform namespace. A `cpaas.io/project` label that names a business project identifies a project namespace. Answer a scan finding on `acp-storage` or `argo-rollouts` by showing the owning component running there. + +## Diagnostic Steps + +```bash +kubectl get namespace -l "cpaas.io/project=" +kubectl get resourcequota,limitrange -n +kubectl get projectquota -l "cpaas.io/project=" +``` + +| What you see | Cause | +| --- | --- | +| Namespace in a quota-limited project has no `ResourceQuota`, or its quota lacks CPU or memory keys | 1 | +| Namespace quota update rejected as exceeding the project quota | 2 | +| `cpaas-system` shows a project and quota that cannot be managed | 3 | +| `kube-system` or `cpaas-system` missing from the project namespace list or the chart form | 4 | +| Nodes needed for one project only, or a namespace shows the whole cluster's resources | 5 | +| Which namespaces belong to the platform, or a scan flags `acp-storage` or `argo-rollouts` | 6 | diff --git a/docs/en/solutions/OpenTelemetry_Collector_logs_cannot_list_resources_pods_because_its_ServiceAccount_has_no_ClusterRoleBinding_on_ACP.md b/docs/en/solutions/OpenTelemetry_Collector_logs_cannot_list_resources_pods_because_its_ServiceAccount_has_no_ClusterRoleBinding_on_ACP.md new file mode 100644 index 000000000..fdadf662a --- /dev/null +++ b/docs/en/solutions/OpenTelemetry_Collector_logs_cannot_list_resources_pods_because_its_ServiceAccount_has_no_ClusterRoleBinding_on_ACP.md @@ -0,0 +1,73 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# OpenTelemetry Collector logs cannot list resources pods because its ServiceAccount has no ClusterRoleBinding on ACP + +## Issue + +The OpenTelemetry Collector log repeats an RBAC error that engineers usually quote as `cannot list resources "pod" in API group`. The full Kubernetes form is: + +```text +pods is forbidden: User "system:serviceaccount::otel-collector" cannot list resource "pods" in API group "" at the cluster scope +``` + +The Collector Pod is Running, but processors that look up Kubernetes metadata (for example Pod attributes on spans) fail. The Collector was deployed by following a tracing deployment document, often into an application namespace instead of the namespace that document uses. + +Related articles: "asm-otel collectors keep restarting, Service Mesh JVM monitoring has no data, or an otel instance cannot be deleted" covers the Service Mesh collectors and a Java agent that cannot reach the Collector. + +## Root Cause + +The Collector runs as a ServiceAccount that the operator creates in the namespace of the `OpenTelemetryCollector` resource, for example `system:serviceaccount::otel-collector`. A Kubernetes ClusterRoleBinding names one ServiceAccount in one namespace. The deprecated deployment document grants cluster-wide read access only to `system:serviceaccount:cpaas-system:otel-collector` (ClusterRoleBinding `otel-collector:cpaas-system:cluster-admin`). When the Collector is deployed in any other namespace, the ServiceAccount actually in use has no binding and cannot list Pods across the cluster. + +The old tracing deployment documents are deprecated: the old Distributed Tracing pages are removed in ACP 4.4, and ACP 4.3 is the last version that contains them. Current deployments follow the Alauda Distributed Tracing documentation. + +## Resolution + +1. Redeploy the tracing stack by following the current Alauda Distributed Tracing documentation, for example "Installing Alauda Distributed Tracing (OpenSearch)" for distributed-tracing 2.1, instead of the deprecated document. This is the confirmed fix: the current procedure creates the Collector together with the permissions it needs. + + > **Warning:** Removing the old Collector stops trace and metric collection until the new Collector is Running; spans sent in that window are lost. Back up the old resource first with `kubectl -n get opentelemetrycollector -o yaml > /tmp/otel-collector.bak.yaml`. Rollback: apply the backup. + +2. If the existing Collector has to stay where it is until the redeployment, the binding can be pointed at the ServiceAccount that is really used. Grant only what the Collector needs; the example ClusterRole below allows reading Pods, Namespaces, and ReplicaSets, which is what Kubernetes attribute lookups usually need: + + ```bash + kubectl create clusterrole otel-collector-read --verb=get,list,watch --resource=pods,namespaces,replicasets.apps + kubectl create clusterrolebinding otel-collector-read: \ + --clusterrole=otel-collector-read --serviceaccount=:otel-collector + ``` + + Rollback: `kubectl delete clusterrolebinding otel-collector-read:` and `kubectl delete clusterrole otel-collector-read`. + +3. After the change the Collector log should no longer show the `cannot list resource "pods"` error. + +## Diagnostic Steps + +1. Read the full error and the ServiceAccount the Collector Pod runs as: + + ```bash + kubectl logs -n --tail=80 + kubectl get pod -n -o jsonpath='{.spec.serviceAccountName}{"\n"}' + kubectl get sa -n otel-collector -o jsonpath='{.metadata.ownerReferences}{"\n"}' + ``` + + The ServiceAccount is typically `otel-collector`, owned by the `OpenTelemetryCollector` resource. + +2. Test the permission of exactly that ServiceAccount: + + ```bash + kubectl auth can-i list pods -A --as=system:serviceaccount::otel-collector + ``` + + `no` confirms the missing permission. + +3. Find which subject the existing binding names: + + ```bash + kubectl get clusterrolebinding -o yaml | grep -B 8 -A 12 'name: otel-collector' + ``` + + A binding whose subject is ServiceAccount `otel-collector` in namespace `cpaas-system`, while the Collector runs in another namespace, matches this article. diff --git a/docs/en/solutions/Pipeline_task_list_is_empty_tasks_are_missing_or_TektonHub_cannot_be_viewed_on_ACP.md b/docs/en/solutions/Pipeline_task_list_is_empty_tasks_are_missing_or_TektonHub_cannot_be_viewed_on_ACP.md new file mode 100644 index 000000000..a1d64d4ab --- /dev/null +++ b/docs/en/solutions/Pipeline_task_list_is_empty_tasks_are_missing_or_TektonHub_cannot_be_viewed_on_ACP.md @@ -0,0 +1,227 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.3" + - "4.3.2" +--- + +# Pipeline task list is empty, tasks are missing, or TektonHub cannot be viewed on ACP + +## Issue + +One of the following happens in the pipeline (Tekton) console: + +- On the graphical pipeline creation page, **Add Task** opens an empty task list. A task the user expects, such as `npm install`, appears to be missing, but in fact no Hub task is listed at all. The Pipeline Tasks page also lists nothing, so a new pipeline cannot be composed, even though Tekton Pipelines itself reports Ready. +- TektonHub cannot be opened, and reinstalling the Tekton operator fails. +- After the pipeline component is upgraded from 4.12.0 to 4.15.1, every Task disappears from the page, although the upgrade did not delete Task resources. +- Every CICD task is gone, a release cannot be started, and `devops-api` is unhealthy. +- A custom or private catalog added to Hub never shows its tasks, or edits to the `tekton-hub-api` ConfigMap are reverted. + +The task list reads its data from TektonHub (or, from pipeline version 4.14, from ArtifactHub Shim). The versions 4.12, 4.14 and 4.15.1 in this article are versions of the pipeline (Tekton operator) component, not of the platform; confirm them with `TektonConfig` before changing anything. + +## Root Cause + +### Cause 1: TektonHub is not installed + +The Tasks page and the **Add Task** dialog have no Hub to read, so the list is empty while Tekton Pipelines is Ready. Restarting the pipeline controllers does not fill the list. On platform 4.3.2, TektonHub was simply not installed; installing a TektonHub instance from the application marketplace restored the task list. + +On the 4.12 operator line, the operator creates TektonHub automatically only when both conditions hold: `TektonConfig.spec.profile` is `all`, and `AUTOINSTALL_TEKTONHUB` in ConfigMap `tekton-operator/tekton-config-defaults` is `"true"`. A stock install leaves the flag `"false"`, so Hub never appears. From pipeline version 4.14 the built-in Tekton Hub is no longer bundled, and 4.15.1 cannot re-enable it through this preset; that is Cause 3. + +### Cause 2: the Triggers webhook Role and RoleBinding are missing after an operator uninstall + +After the Tekton operator was uninstalled, the reinstall failed because the Triggers webhook `Role` and `RoleBinding` were gone. Without them the webhook cannot start, Hub stays unreachable, and the operator install does not finish. + +### Cause 3: from pipeline 4.14, Tekton Hub is no longer bundled + +The upgrade to 4.15.1 does not delete Task resources. From 4.14, the built-in Tekton Hub is no longer shipped, and catalog access moves to the standalone ArtifactHub Shim. Tasks stay off the page until the `artifacthub-shim` plugin is installed, and existing custom catalogs have to be migrated to it. If the Git host that serves a catalog repository no longer answers, that catalog stays empty even with `artifacthub-shim` running. + +### Cause 4: `devops-api` is wedged + +The task list used for release is empty and a release cannot be started because `devops-api` is unhealthy. A slow etcd can leave the component wedged. Restarting `devops-api` brings the CICD tasks back; it does not repair the underlying etcd performance problem. + +### Cause 5: the operator owns `tekton-hub-api` and overwrites direct edits + +When Hub is deployed through `TektonConfig`, the operator owns the `tekton-hub-api` ConfigMap and reverts direct edits, including its `CATALOGS` value. On `tektoncd-operator` release 4.12 the supported path is to override `CATALOGS` through `TektonConfig.spec.hub.options.configMaps`. A private catalog repository needs both `url` and `sshurl`, plus the fixed-name SSH Secret `tekton-hub-api-ssh-crds` (keys `id_rsa`, `id_rsa.pub`, `known_hosts`) in the namespace where Hub runs. Hub rereads catalogs after `CATALOG_REFRESH_INTERVAL` or when the `tekton-hub-api` Pod restarts. Hub itself must be enabled first (Cause 1). + +## Resolution + +### Cause 1: TektonHub is not installed + +1. Confirm that no TektonHub exists and check the pipeline version: + + ```bash + kubectl get tektonhub.operator.tekton.dev -A + kubectl get tektonconfig config -o jsonpath='{.status.version}{"\n"}{.spec.profile}{"\n"}' + kubectl -n tekton-operator get cm tekton-config-defaults -o jsonpath='{.data.AUTOINSTALL_TEKTONHUB}{"\n"}' + ``` + + No TektonHub row means this cause. If the pipeline version is 4.14 or later, go to Cause 3 instead. + +2. Install a TektonHub instance from the application marketplace on the cluster that runs the pipelines. This is the fix that restored the list on platform 4.3.2. Then reopen **Add Task**. + +3. On the 4.12 operator line, the auto-install switch is an alternative to step 2. + + > **Warning:** Setting the flag and restarting `tekton-operator` pulls Hub images and creates Hub API and database Pods; setting `spec.profile` to `all` installs every component of that profile, not only Hub. Only when the list is empty, no TektonHub exists, and the flag is `"false"`. Rollback: restore the saved ConfigMap and the previous `spec.profile`, then restart the operator. Turning the flag off does not delete a TektonHub that was already created. + + ```bash + kubectl -n tekton-operator get cm tekton-config-defaults -o yaml > /tmp/tekton-config-defaults.yaml + kubectl -n tekton-operator patch cm tekton-config-defaults --type=merge -p '{"data":{"AUTOINSTALL_TEKTONHUB":"true"}}' + # only when spec.profile is not already "all" + kubectl patch tektonconfig config --type=merge -p '{"spec":{"profile":"all"}}' + kubectl -n tekton-operator rollout restart deploy/tekton-operator + kubectl -n tekton-operator rollout status deploy/tekton-operator + kubectl get tektonhub.operator.tekton.dev -A + ``` + + Confirm the operator Deployment name with `kubectl -n tekton-operator get deploy` first. + +4. If a TektonHub object exists and the list is still empty, check that the Hub API and database Pods are Ready and that the catalog has synchronized: + + ```bash + kubectl get tektonhub.operator.tekton.dev -A -o jsonpath='{range .items[*]}{.metadata.name}{" "}{range .status.conditions[*]}{.type}{"="}{.status}{" "}{end}{"\n"}{end}' + kubectl get pods -A | grep -Ei 'tekton-hub|hub-api|hub-db' + kubectl logs -n --tail=100 | grep -Ei 'catalog|sync|error' + ``` + + Repair a Pod that is not Ready, or the catalog error in the API log, before installing a second Hub. + +### Cause 2: the Triggers webhook Role and RoleBinding are missing after an operator uninstall + +1. Confirm that the Role and RoleBinding are gone and that the webhook log shows a forbidden error: + + ```bash + kubectl get deploy,pods -A | grep -i 'triggers-webhook\|tekton-triggers' + kubectl get role,rolebinding -A | grep -i trigger + kubectl logs -n deploy/ --tail=80 + ``` + +2. Restore both objects from a healthy cluster on the same operator version, or from that version's Triggers static manifests (installer sets named like `trigger-main-static`). Do not bind the webhook ServiceAccount to `cluster-admin` instead. If `TektonConfig` is Not Ready because an installer set is stuck terminating, follow "TektonConfig stuck Not Ready after Tekton operator upgrade on ACP" instead. + + ```bash + kubectl apply -f /tmp/triggers-webhook-role.yaml + kubectl apply -f /tmp/triggers-webhook-rolebinding.yaml + ``` + +3. Restart the webhook. + + > **Warning:** Admission for trigger resources is unavailable for a few seconds; EventListener and trigger creates fail until the Pod is Ready. Only after the restored Role and RoleBinding come from the same operator version. Rollback: delete the applied objects if they were wrong; a restart alone does not remove them. + + ```bash + kubectl -n rollout restart deploy/ + kubectl -n rollout status deploy/ + ``` + +4. Retry the operator install. If the task list is then empty, continue with Cause 1 or Cause 3. + +### Cause 3: from pipeline 4.14, Tekton Hub is no longer bundled + +1. Confirm the version and whether ArtifactHub Shim runs: + + ```bash + kubectl get tektonconfig config -o jsonpath='{.status.version}{"\n"}' + kubectl get pods -A | grep -i artifacthub + ``` + +2. On 4.14 or later with no ArtifactHub Shim workload, install the `artifacthub-shim` plugin through the same plugin installation path used for other DevOps plugins. The Cause 1 flag does not bring the built-in Hub back on these versions. +3. Migrate existing custom catalogs to ArtifactHub Shim, following its documentation for the cross-namespace global `repository` ConfigMap. Copy the shape from a cluster where the task list already loads instead of inventing keys. +4. If the list is still empty, check that the Git host named in the repository configuration answers, and point the configuration at a reachable host: + + ```bash + kubectl get configmap -A | grep -i repository + kubectl exec -n -- getent hosts + ``` + +### Cause 4: `devops-api` is wedged + +1. Confirm `devops-api` is not Ready or its log shows it is stuck. A healthy `devops-api` with an empty list is Cause 1 or Cause 3. + + ```bash + kubectl get deploy,pods -A | grep devops-api + kubectl -n logs deploy/ --tail=80 + ``` + +2. Restart it. + + > **Warning:** The CICD task list and release API are unavailable until the new Pod is Ready. Only when every CICD task is gone and `devops-api` is unhealthy. If API calls are still slow from an etcd problem, wait until they answer, or the component wedges again. Rollback: none; a restart changes no configuration. + + ```bash + kubectl -n rollout restart deploy/ + kubectl -n rollout status deploy/ + ``` + + If the list empties again soon after, the etcd performance problem is still open; repeating the restart is not a fix. + +### Cause 5: the operator owns `tekton-hub-api` and overwrites direct edits + +1. Read the current list; the override replaces the whole `CATALOGS` value: + + ```bash + kubectl -n get cm tekton-hub-api -o jsonpath='{.data.CATALOGS}{"\n"}' + kubectl get tektonconfig config -o yaml > /tmp/tektonconfig.yaml + ``` + +2. Put the full list, with your entries appended, under the override in `TektonConfig` `config` (example for `tektoncd-operator` release 4.12; keep your own existing entries): + + ```yaml + spec: + hub: + options: + disabled: false + configMaps: + tekton-hub-api: + data: + CATALOGS: | + - name: catalog + org: tektoncd + type: community + provider: github + url: file:////mnt/git/catalog.git + revision: master + - name: private-catalog + org: example + type: private + provider: gitlab + url: https:///devops/private-tekton-catalog.git + sshurl: git@:devops/private-tekton-catalog.git + revision: main + ``` + + > **Warning:** The operator rolls `tekton-hub-api`, so the Hub UI and task list are briefly unavailable; a malformed `CATALOGS` value can leave Hub with no catalogs. Rollback: `kubectl apply -f /tmp/tektonconfig.yaml`, or remove `spec.hub.options.configMaps`. + + ```bash + kubectl edit tektonconfig config + ``` + +3. For a private repository, create the SSH Secret in the Hub namespace: + + ```bash + kubectl -n create secret generic tekton-hub-api-ssh-crds \ + --from-file=id_rsa=/path/to/id_rsa \ + --from-file=id_rsa.pub=/path/to/id_rsa.pub \ + --from-file=known_hosts=/path/to/known_hosts + ``` + +4. Wait for `CATALOG_REFRESH_INTERVAL`, or restart the `tekton-hub-api` Pod. The catalog's tasks should appear in the Hub list. + +## Diagnostic Steps + +```bash +kubectl get tektonhub.operator.tekton.dev -A +kubectl get tektonconfig config -o jsonpath='{.status.version}{"\n"}{.spec.profile}{"\n"}' +kubectl -n tekton-operator get cm tekton-config-defaults -o jsonpath='{.data.AUTOINSTALL_TEKTONHUB}{"\n"}' +kubectl get role,rolebinding -A | grep -i 'triggers-webhook\|tekton-triggers' +kubectl get pods -A | grep -Ei 'artifacthub|devops-api' +``` + +| What you see | Cause | +| --- | --- | +| Empty list, no TektonHub object, pipeline version below 4.14 | 1 | +| Operator reinstall fails, Triggers webhook Role or RoleBinding missing, forbidden in the webhook log | 2 | +| Empty list after an upgrade to 4.15.1, or on 4.14, no ArtifactHub Shim workload | 3 | +| Every CICD task gone, `devops-api` not Ready | 4 | +| Custom catalog never appears, or a `tekton-hub-api` edit is reverted | 5 | + +A stage list that is empty in a DevOps v4 pipeline while TektonHub is installed and Ready, with no API error, is not explained by this article. diff --git a/docs/en/solutions/Service_Mesh_delete_is_stuck_on_a_failed_to_get_cluster_webhook_denial_or_on_pods_that_still_have_sidecar_injection.md b/docs/en/solutions/Service_Mesh_delete_is_stuck_on_a_failed_to_get_cluster_webhook_denial_or_on_pods_that_still_have_sidecar_injection.md new file mode 100644 index 000000000..6ed86c627 --- /dev/null +++ b/docs/en/solutions/Service_Mesh_delete_is_stuck_on_a_failed_to_get_cluster_webhook_denial_or_on_pods_that_still_have_sidecar_injection.md @@ -0,0 +1,168 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Service Mesh +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Service Mesh delete is stuck on a failed to get cluster webhook denial or on pods that still have sidecar injection + +## Issue + +A service mesh, or its gateway, will not uninstall and the console stays on deleting. One of these is seen: + +- The business cluster that the mesh belonged to no longer exists. Deleting the `servicemesh` object with `kubectl` is denied: + + ```text + admission webhook "mservicemesh.alauda.io" denied the request: failed to get cluster "": clusters.clusterregistry.k8s.io "" not found + ``` + + The `servicemesh` can still show `Deployed`, and removing its `finalizers` is denied with the same message. +- The ASM delete does not complete, and the `asm-operator` log reports that pods outside the mesh workloads being removed still have sidecar injection enabled. After that is fixed, `istio-operator` still does not go away. + +`servicemesh` objects live on the `global` cluster as `asm.alauda.io/v1alpha1` in `cpaas-system`, and the object name is the business cluster name. An unused mesh whose ASM process consumes a lot of CPU is removed through this same uninstall path. + +Related articles: "Leftover Service Mesh v1 recreates istio-system against Mesh v2 or Istio Secret certificate alerts remain on ACP" covers leftovers after an uninstall. + +## Root Cause + +### Cause 1: the `mservicemesh.alauda.io` webhook cannot find the deleted cluster + +The mutating webhook configuration `servicemesh-v1alpha1-mutation` (webhook `mservicemesh.alauda.io`, served by `asm-controller`) looks up the business cluster record for every change to a `servicemesh`. When that record is gone, the lookup fails and the webhook denies the delete and any finalizer patch, so the object can never be removed while the webhook is in place. + +### Cause 2: other workloads still have sidecar injection enabled + +ASM is deleted only after sidecar injection is disabled on every pod that still has it. A workload outside the mesh, for example `ingress-nginx`, can carry the sidecar injection annotation on its pod template; `asm-operator` then waits indefinitely. After injection is turned off, a leftover `istio-operator` object can still hold the delete and has to be force-removed. + +## Resolution + +### Cause 1: the `mservicemesh.alauda.io` webhook cannot find the deleted cluster + +Run these steps on the `global` cluster. + +1. If the cluster record still exists, this cause does not apply; remove the meshed workloads and delete normally: + + ```bash + kubectl get clusters.clusterregistry.k8s.io + kubectl get servicemesh -n cpaas-system -o yaml > /tmp/servicemesh-.bak.yaml + kubectl -n cpaas-system delete servicemesh --wait + ``` + +2. When the record is `not found`, back up the webhook configuration and the mesh objects, and inspect the webhook: + + ```bash + kubectl get mutatingwebhookconfiguration servicemesh-v1alpha1-mutation -o yaml > /tmp/servicemesh-v1alpha1-mutation.bak.yaml + kubectl get servicemesh -A -o yaml > /tmp/servicemesh-all.bak.yaml + kubectl get mutatingwebhookconfiguration servicemesh-v1alpha1-mutation \ + -o jsonpath='{range .webhooks[*]}{.name}{" "}{.clientConfig.service.name}{" "}{.failurePolicy}{"\n"}{end}' + ``` + +3. Take the webhook out of the path, using one of the two methods below. + + > **Warning:** While the webhook is bypassed, ServiceMesh changes on this `global` cluster are not validated by it. Only when the delete fails because the cluster record is missing and the backup files exist; use a maintenance window. Rollback: `kubectl apply -f /tmp/servicemesh-v1alpha1-mutation.bak.yaml` (remove `metadata.resourceVersion`, `uid`, `managedFields`, and `status` from the file if apply complains). + + - Method A: set `failurePolicy` of the `asm-controller` webhook to `Ignore`. This resolved a delete for a mesh whose cluster no longer existed. Kubernetes applies `failurePolicy` only when the webhook call itself fails; if the delete still returns the `failed to get cluster` denial after this change, use Method B. + + ```bash + kubectl patch mutatingwebhookconfiguration servicemesh-v1alpha1-mutation --type=json \ + -p '[{"op":"replace","path":"/webhooks/0/failurePolicy","value":"Ignore"}]' + ``` + + Use the index of the `mservicemesh.alauda.io` entry from the listing in step 2 if it is not `0`. + + - Method B: delete the webhook configuration. + + ```bash + kubectl delete mutatingwebhookconfiguration servicemesh-v1alpha1-mutation + ``` + +4. Delete the mesh object. + + > **Warning:** The ServiceMesh object and the gateways and injection it owns are removed. Because the cluster record is gone, the object cannot be recreated until that record exists again; treat this delete as irreversible. + + ```bash + kubectl get servicemesh -A + kubectl delete servicemesh -n cpaas-system + ``` + +5. Restore the webhook immediately: + + ```bash + kubectl apply -f /tmp/servicemesh-v1alpha1-mutation.bak.yaml + kubectl get mutatingwebhookconfiguration servicemesh-v1alpha1-mutation -o jsonpath='{.webhooks[*].failurePolicy}{"\n"}' + ``` + + A gateway that was stuck in deleting leaves that state once the webhook no longer denies the request. + +### Cause 2: other workloads still have sidecar injection enabled + +1. Read the `asm-operator` log and find the workloads it names: + + ```bash + kubectl get pods -A | grep asm-operator + kubectl -n istio-system logs --tail=200 | grep -i -E 'sidecar|inject' + kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" "}{.spec.containers[*].name}{"\n"}{end}' | grep istio-proxy + kubectl -n get deploy -o jsonpath='{.spec.template.metadata.annotations}{"\n"}' + ``` + +2. Set the Istio injection annotation `sidecar.istio.io/inject` to `"false"` on each named workload. Use `daemonset` instead of `deployment` where the workload is a DaemonSet. + + > **Warning:** The workload rolls and its new pods start without `istio-proxy`; ingress traffic through `ingress-nginx` can drop briefly. Only for workloads the `asm-operator` log names, and only when the mesh is really being removed. Rollback: set the annotation back to `"true"` (re-injection works only while the mesh is still installed). + + ```bash + kubectl -n patch deployment --type=merge \ + -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}}}}}' + kubectl -n rollout status deployment + ``` + +3. Retry the ASM delete. If it stops on `istio-operator`, find which object is left: + + ```bash + kubectl get ns | grep istio-operator + kubectl get pods -A | grep istio-operator + kubectl get istiooperator -A + kubectl get istiooperator -n -o jsonpath='{.metadata.deletionTimestamp}{" "}{.metadata.finalizers}{"\n"}' + ``` + +4. Force-remove the remaining object. + + > **Warning:** A force delete skips the controller's cleanup, so resources it would have removed can remain and need manual cleanup. Only after injection is disabled on the named workloads and the remaining `istio-operator` object has a `deletionTimestamp` or is stuck in `Terminating`. Rollback: none; keep the backup as a reference. + + ```bash + kubectl get istiooperator -n -o yaml > /tmp/istiooperator.bak.yaml + # stuck IstioOperator resource + kubectl patch istiooperator -n --type=merge -p '{"metadata":{"finalizers":null}}' + # Pod stuck in Terminating + kubectl delete pod -n --grace-period=0 --force + ``` + + Run only the line that matches the object left. The ASM delete should then complete. If the `servicemesh` on `global` is then denied with `failed to get cluster`, continue with Cause 1. + +## Diagnostic Steps + +1. On `global`, check the mesh object, the cluster record, and the webhook: + + ```bash + kubectl -n cpaas-system get servicemesh + kubectl get clusters.clusterregistry.k8s.io + kubectl get mutatingwebhookconfiguration servicemesh-v1alpha1-mutation + ``` + +2. On the business cluster, check what blocks `asm-operator`: + + ```bash + kubectl -n istio-system logs --tail=200 | grep -i -E 'sidecar|inject' + kubectl get ns,istiooperator -A | grep -i istio-operator + ``` + +| What you see | Cause | +| --- | --- | +| Delete denied by `mservicemesh.alauda.io` with `failed to get cluster` and the cluster record `not found` | 1 | +| ASM delete hangs, `asm-operator` log names pods that are still injected (for example `ingress-nginx`), or `istio-operator` stays | 2 | +| `istio-system` keeps returning, or Istio Secret certificate alerts after Mesh v1 is gone | The leftover Mesh v1 article | diff --git a/docs/en/solutions/Service_Mesh_tracing_Kiali_trace_links_return_404_a_workload_has_no_data_or_jaeger-prod-query_crashes_on_ACP.md b/docs/en/solutions/Service_Mesh_tracing_Kiali_trace_links_return_404_a_workload_has_no_data_or_jaeger-prod-query_crashes_on_ACP.md new file mode 100644 index 000000000..a68814a39 --- /dev/null +++ b/docs/en/solutions/Service_Mesh_tracing_Kiali_trace_links_return_404_a_workload_has_no_data_or_jaeger-prod-query_crashes_on_ACP.md @@ -0,0 +1,177 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Service Mesh +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Service Mesh tracing: Kiali trace links return 404, a workload has no data, or jaeger-prod-query crashes on ACP + +## Issue + +One of these tracing symptoms appears on a cluster where the mesh is installed and running: + +- In ASM v2, every link from Kiali to the details of a trace opens a 404 page. +- Kiali pages for one workload show no data: Outbound Metrics show `No data available` for Request duration, Request size, Response size, gRPC received, and gRPC sent; only the TCP charts have data, and the Traces tab is empty. +- The container platform Observability tracing page shows no data and the error `There was an error quering for traces:Unknown Error` (the page text has that spelling). +- The call chain keeps showing an `unknown` request, and nobody can tell who sends it. +- After ASM v1 is deployed, Jaeger does not start and `jaeger-prod-query` in `istio-system` keeps crashing. +- After the platform log storage was switched from Elasticsearch to ClickHouse, `jaeger-prod-es-index-cleaner` in `istio-system` keeps reporting `failed to query indices and unmarshall response body`. + +## Root Cause + +### Cause 1: Kiali `external_url` for tracing is not set + +Kiali builds trace detail links from `spec.external_services.tracing.external_url` of the Kiali custom resource. The documented tracing block (`provider: jaeger`, `use_grpc: true`, `internal_url: "http://jaeger-prod-query.istio-system:16685/jaeger"`) shows `external_url` commented out as optional. Left unset, every trace detail link returns 404. The value has the form `/clusters//istio/jaeger`. + +### Cause 2: `service.istio.io/canonical-name` does not match the Deployment name + +The sidecar is injected and the Service port name (for example `http-8040-8040`) lets Istio detect HTTP, yet Kiali shows no request metrics. The Pod label `service.istio.io/canonical-name` is set to the Service name instead of the Deployment name, so telemetry is reported under a name Kiali does not look up for this workload. In the Jaeger UI, selecting the service `.` returns the traces that Kiali does not show. + +### Cause 3: the container platform tracing page is a separate product from Service Mesh v2 tracing + +This is a product boundary, not a fault. Mesh v1 bundles Jaeger v1 with a custom tracing page; on 4.x a separate tracing feature for containers was derived from that page and released as a standalone UI plugin with its own backend and indices. Mesh v2 integrates Jaeger in the open-source way, and its traces are read in the Jaeger UI or in Kiali. When the container tracing backend is not deployed, the page's API calls `GET /clusters//acp/jaeger/api/services` and `.../api/spans` return `404` with response headers `via: alb` and `x-alb-err-reason: BackendError`, although the mesh Jaeger UI (instance `jaeger-prod` in `istio-system`) lists services and the `istio-tracing--jaeger-span-` indices contain documents. This was confirmed on ACP 4.2.2. + +### Cause 4: a Prometheus scrape appears as an `unknown` caller + +The ztunnel or `istio-proxy` log of the destination workload shows an inbound `access connection complete` entry about once a minute from the same source address. That address belongs to a Prometheus instance scraping the mesh workloads. Those scrapes appear in the call chain as `unknown`. This is expected behavior. + +### Cause 5: the ACP 4.3.0 `asm-operator` generates a 44-byte `cookie_secret` + +The `jaeger-prod-query` Pod has two containers, `jaeger-query` and `oauth2-proxy`. The `oauth2-proxy` container exits with: + +```text +invalid configuration: cookie_secret must be 16, 24, or 32 bytes to create an AES cipher, but is 44 bytes +``` + +Secret `oauth2proxy-secret` in `istio-system` is generated by ASM (Helm release `asm`, owned by `Asm/asm`). The ACP 4.3.0 `asm-operator` writes a 44-byte value, which `oauth2-proxy` rejects. This is a known issue of that package and is fixed in the ACP v4.3.1 `asm-operator` package. A plain `kubectl logs | grep error` shows nothing because the error is in the `oauth2-proxy` container. + +### Cause 6: ASM v1 tracing supports only Elasticsearch + +`jaeger-es-index-cleaner` is a Job that removes old Elasticsearch tracing indices (traces are kept for 7 days by default). ASM v1 tracing supports only Elasticsearch as storage, not ClickHouse. After Elasticsearch is removed, the cleaner still calls the old address, receives a portal HTML page, and cannot parse it. Log queries on ClickHouse are unaffected. ASM v2 has no `jaeger-es-index-cleaner`. + +## Resolution + +### Cause 1: Kiali `external_url` for tracing is not set + +1. Confirm the field is empty: + + ```bash + kubectl get kiali -A + kubectl get kiali -n -o jsonpath='{.spec.external_services.tracing}{"\n"}' + ``` + +2. Set it. + + > **Warning:** The Kiali operator restarts Kiali, so the Kiali UI is briefly unavailable. Only when trace links return 404 and `external_url` is empty. Rollback: `kubectl apply -f /tmp/kiali.bak.yaml`. + + ```bash + kubectl get kiali -n -o yaml > /tmp/kiali.bak.yaml + kubectl patch kiali -n --type=merge \ + -p '{"spec":{"external_services":{"tracing":{"external_url":"/clusters//istio/jaeger"}}}}' + ``` + + `` is the platform address used in the browser and `` is the cluster that runs Jaeger. After Kiali is Ready, trace links should open the Jaeger trace. + +### Cause 2: `service.istio.io/canonical-name` does not match the Deployment name + +1. Compare the label with the Deployment name and check the sidecar and port name: + + ```bash + kubectl -n get pod -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{end}' + kubectl -n get pod -o jsonpath='{.metadata.labels.service\.istio\.io/canonical-name}{" "}{.metadata.labels.service\.istio\.io/canonical-revision}{"\n"}' + kubectl -n get svc -o jsonpath='{range .spec.ports[*]}{.name}{" "}{.port}{"\n"}{end}' + ``` + +2. Set the label on the Pod template; a label on a running Pod is lost when it is recreated. + + > **Warning:** The workload rolls, and its telemetry is reported under the new name from then on. Only when the label differs from the Deployment name and Jaeger shows traces under `.`. Rollback: patch the previous value back. + + ```bash + kubectl -n patch deployment --type=merge \ + -p '{"spec":{"template":{"metadata":{"labels":{"service.istio.io/canonical-name":""}}}}}' + kubectl -n rollout status deployment + ``` + + If Kiali still reports `Pod has no Istio sidecar`, injection is the problem, not the label. + +### Cause 3: the container platform tracing page is a separate product from Service Mesh v2 tracing + +1. Confirm that Mesh v2 traces are stored and readable: + + ```bash + kubectl get jaeger -n istio-system + kubectl get pods -n istio-system | grep -E 'jaeger|kiali' + ``` + + Open the mesh Jaeger UI or the Kiali Traces tab and check that traces are listed. + +2. In the browser developer tools on the tracing page, a `404` on `/acp/jaeger/api/services` or `/acp/jaeger/api/spans` with `x-alb-err-reason: BackendError` matches this cause. +3. Read Mesh v2 traces in the Jaeger UI or Kiali. No mesh change makes the container platform tracing page show Mesh v2 traces. If the Jaeger UI is also empty, this cause does not apply. + +### Cause 4: a Prometheus scrape appears as an `unknown` caller + +1. Map the recurring source address to a Pod: + + ```bash + kubectl logs -n --tail=1000 | grep 'access connection complete' | grep '' + kubectl logs -n -c istio-proxy --tail=1000 + kubectl get pod -A -o wide | grep '' + ``` + + Use the ztunnel log in ambient mode and the `istio-proxy` log in sidecar mode. + +2. An address that resolves to a Prometheus Pod (or another scraper) is expected traffic. Stopping that scrape removes the `unknown` entry; no mesh object needs to change. + +### Cause 5: the ACP 4.3.0 `asm-operator` generates a 44-byte `cookie_secret` + +1. Confirm the signature: + + ```bash + kubectl logs -n istio-system -c oauth2-proxy --tail=50 + kubectl get secret -n istio-system oauth2proxy-secret -o jsonpath='{.data.OAUTH2_PROXY_COOKIE_SECRET}' | base64 -d | wc -c + ``` + + A length of `44` with the log line above matches this cause. + +2. Replace the `asm-operator` package and redeploy the mesh. Do not hand-edit the generated Secret. + + > **Warning:** Uninstalling the mesh removes all mesh components from the managed clusters; injection, gateways, and tracing stop until ASM v1 is deployed again. Only when the platform is 4.3.0, the log shows the 44-byte error, and a maintenance window is open. Rollback: not useful; the 4.3.0 package brings the crash back. + + 1. Uninstall the mesh. If the delete gets stuck, see "Service Mesh delete is stuck on a failed to get cluster webhook denial or on pods that still have sidecar injection". + 2. On the `global` cluster, go to **Marketplace** > **Upload Packages** > **Operator**, delete the 4.3.0 `asm-operator` package, and upload the ACP v4.3.1 `asm-operator` package. + 3. Deploy ASM v1 again. + +3. All Pods in `istio-system` should be Running, with `jaeger-prod-query` at `2/2` and no restarts. + +### Cause 6: ASM v1 tracing supports only Elasticsearch + +1. Confirm the error and the address the cleaner calls: + + ```bash + kubectl get cronjob,job -n istio-system | grep es-index-cleaner + kubectl logs -n istio-system --tail=50 + kubectl get cronjob -n istio-system -o jsonpath='{.spec.jobTemplate.spec.template.spec.containers[*].args}{"\n"}' + ``` + +2. ClickHouse cannot replace Elasticsearch for ASM v1 tracing. Either keep an Elasticsearch backend for ASM v1 tracing, or move the mesh to ASM v2, where the cleaner does not exist and tracing uses the separate stack described in "Moving Service Mesh v2 tracing storage from Elasticsearch to OpenSearch with Jaeger v2". Moving editions means uninstalling ASM v1 and installing ASM v2; evaluate the differences in sidecar injection, gateway deployment, and the effect on Alauda AI if it uses the mesh before choosing it. + +## Diagnostic Steps + +```bash +kubectl get kiali -A +kubectl get pods -n istio-system | grep -E 'jaeger|kiali' +kubectl logs -n istio-system -c oauth2-proxy --tail=20 +``` + +| What you see | Cause | +| --- | --- | +| Trace link from Kiali opens 404, `external_url` empty | 1 | +| One workload: no request metrics, Traces tab empty, Jaeger has traces under `.`, canonical-name is the Service name | 2 | +| Platform tracing page `Unknown Error`, `/acp/jaeger/api/*` returns 404 with `x-alb-err-reason: BackendError`, Jaeger UI has traces | 3 | +| `unknown` request about once a minute, source address is a Prometheus Pod | 4 | +| `jaeger-prod-query` crash loop, `oauth2-proxy` logs `cookie_secret ... but is 44 bytes`, platform 4.3.0 | 5 | +| `jaeger-prod-es-index-cleaner` logs `failed to query indices and unmarshall response body`, log storage moved to ClickHouse | 6 | diff --git a/docs/en/solutions/User_group_member_gets_a_data_error_or_no_permissions_while_a_direct_role_binding_works_on_ACP.md b/docs/en/solutions/User_group_member_gets_a_data_error_or_no_permissions_while_a_direct_role_binding_works_on_ACP.md new file mode 100644 index 000000000..c242b0384 --- /dev/null +++ b/docs/en/solutions/User_group_member_gets_a_data_error_or_no_permissions_while_a_direct_role_binding_works_on_ACP.md @@ -0,0 +1,135 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.2.6" +--- + +# User group member gets a data error or no permissions while a direct role binding works on ACP + +## Issue + +A user (local, synchronized from LDAP, or from a third-party identity source) is a member of a user group, and the group is bound to a platform or namespace role, including platform administrator. The user can log in but cannot do what a user with the same role bound directly can do: + +- Batch delete of Pods on the container platform shows a data error and deletes nothing; the same batch delete succeeds for a local platform administrator. +- Creating a namespace, or opening one, reports insufficient permissions. Assigning the role directly to the account makes it work. +- "The user group is authorized, but the user has no permissions after logging in": any role granted through that group, including a custom namespace role, is missing. +- Users added to a group with a role association cannot open the related namespaces, while a newly created group with a role association works. +- Pipeline (Tekton) pages do not load or query for a user whose permissions come only from a group, while every other feature works for that user. + +The failure is authorization through the group, not a bad password. + +## Root Cause + +### Cause 1: metis does not apply group authorization to batch delete + +`auth-controller2` logs show the batch-delete request intercepted by a webhook, and metis validation rejects it. Metis lacks group authorization: a permission that exists on the group is not applied when metis evaluates the user. A directly bound role does not take this path. This is a platform defect; the fix is planned for 4.2.6. + +### Cause 2: group role permissions do not take effect for namespace creation on 3.18 and 4.1.0 + +On 3.18, and on 4.1.0, permissions granted through a user group did not take effect for namespace creation. On 4.1.0 this affects LDAP users and local users in a local group alike. It is fixed starting in 4.2. + +### Cause 3: the group's name does not start with `g-` + +Authorization discards every permission of a group whose Kubernetes object name lacks the `g-` prefix. A group created in the web console gets a generated `g-` name; a group without the prefix was created another way, for example by applying an object with a hand-chosen name. The role content is irrelevant. + +### Cause 4: the group was never actually associated with the role + +A component problem while the group and its role association were first created left the role unbound, although the console showed the association. In the browser, `selfsubjectaccessreviews` returns `201` while the namespace page's requests return `403` (the namespace itself, `resourcequota`, `pod`, `workload`, `application`, `configmaps?labelSelector=features.cpaas.io%2Ftype%3DCustomResourceLimitation`, the `features` queries for `prometheus` and `victoriametrics`, and `log-agent`), and `custom-navconfig-acp` returns `404`. A new group with the role associated works, which shows the original association was never stored. The exact mechanism was not confirmed. + +### Cause 5: the pipeline Hub API of Tekton operator 4.10 does not honor group bindings + +The group's UserBindings are `Success` and synced to all clusters, and the user has every other feature, but the pipeline pages fail. The pipeline Hub API in Tekton operator (Alauda DevOps Pipelines) 4.10 does not support authorization through user-group bindings, so permissions granted only through a group are ignored by the pipeline pages. Directly bound roles are honored. Group authorization for pipelines needs Alauda DevOps Pipelines 4.14 or later together with Alauda Artifact Hub Shim v1.0.0, which are compatible only with platform v4.4 and v4.3.3. + +## Resolution + +### Cause 1: metis does not apply group authorization to batch delete + +1. Upgrade to the release that contains the metis group-authorization fix (planned for 4.2.6); check the release notes of the target version. +2. Until then, bind the role directly to the affected user in Platform Management > User Management, the same way the working local account is bound, and retry the batch delete. Rollback: remove that direct binding; the group binding is unchanged. + +### Cause 2: group role permissions do not take effect for namespace creation on 3.18 and 4.1.0 + +1. Upgrade to 4.2 or later, then create the namespace again as the group user without a direct binding. +2. While the upgrade is scheduled, a direct binding of the same role to the user also makes namespace creation succeed. Rollback: delete that direct binding. + +### Cause 3: the group's name does not start with `g-` + +An object name cannot be renamed, so the fix is a new group. + +1. Export the old group and its bindings: + + ```bash + kubectl get groups.auth.alauda.io + kubectl get groups.auth.alauda.io -o yaml > old-group.before.yaml + kubectl get userbinding -o yaml > userbindings.before.yaml + ``` + +2. In Platform Management > User Management, create a new user group in the web console and confirm its name starts with `g-`. +3. Import the affected users, and bind the role the old group had, with the same project or namespace scope. +4. Log in as an affected user and confirm the permissions. +5. Remove the old group. + + > **Warning:** Deleting the old group removes it and its bindings for every member; members not moved to the new group keep nothing from it (they had nothing in effect while this cause applied). Only after the new `g-` group is verified. Rollback: `kubectl apply -f old-group.before.yaml` and recreate its bindings from `userbindings.before.yaml`; a restored group still lacks the prefix. + +### Cause 4: the group was never actually associated with the role + +1. Save the original group's bindings so a recurrence can be compared: + + ```bash + kubectl get userbinding -o yaml | grep -n + kubectl get userbinding -o yaml > userbinding.before.yaml + ``` + +2. Create a new user group, associate the role the original group should have had, add the users, and confirm access. +3. Retire the original group after the new one is verified. If the problem reappears, collect the full `auth-controller2` log and the UserBinding YAML: + + ```bash + kubectl get pods -A | grep auth-controller2 + kubectl logs -n > auth-controller2.log + ``` + +### Cause 5: the pipeline Hub API of Tekton operator 4.10 does not honor group bindings + +1. Workaround: grant the role directly to each user who needs the pipeline pages, instead of only through the group. Rollback: remove the direct binding. +2. Permanent fix: upgrade Alauda DevOps Pipelines to 4.14 or later and deploy Alauda Artifact Hub Shim v1.0.0. Because these require platform v4.4 or v4.3.3, upgrade the platform first if it is on another version. + +## Diagnostic Steps + +1. Compare the group user with a directly bound user: + + ```bash + kubectl auth can-i create namespaces --as= + kubectl auth can-i create namespaces --as= + kubectl auth can-i get pods -n --as= + ``` + + `auth can-i` answers Kubernetes RBAC only. A `yes` together with a console data error on batch delete still fits Cause 1, because metis rejects the call after RBAC. + +2. Check the group name and its bindings: + + ```bash + kubectl get groups.auth.alauda.io + kubectl get userbinding | grep + kubectl get userbinding -o jsonpath='{.status}{"\n"}' + ``` + +3. For batch delete, read the intercepting component: + + ```bash + kubectl logs -n --tail=400 + ``` + +| What you see | Cause | +| --- | --- | +| Batch delete data error, local user works, `auth-controller2` log shows the metis webhook, 4.x before 4.2.6 | 1 | +| Namespace creation denied for group users, direct role works, platform 3.18 or 4.1 | 2 | +| No permission at all through one group, group name lacks the `g-` prefix | 3 | +| Group name has `g-`, namespace page requests return `403` while `selfsubjectaccessreviews` returns `201`, a new group works | 4 | +| Only the pipeline pages fail for a group-only user, group UserBindings `Success`, Tekton operator 4.10 | 5 | diff --git a/docs/en/solutions/asm-otel_collectors_keep_restarting_Service_Mesh_JVM_monitoring_has_no_data_or_an_otel_instance_cannot_be_deleted.md b/docs/en/solutions/asm-otel_collectors_keep_restarting_Service_Mesh_JVM_monitoring_has_no_data_or_an_otel_instance_cannot_be_deleted.md new file mode 100644 index 000000000..e41340791 --- /dev/null +++ b/docs/en/solutions/asm-otel_collectors_keep_restarting_Service_Mesh_JVM_monitoring_has_no_data_or_an_otel_instance_cannot_be_deleted.md @@ -0,0 +1,181 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Service Mesh +ProductsVersion: + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# asm-otel collectors keep restarting, Service Mesh JVM monitoring has no data, or an otel instance cannot be deleted + +## Issue + +One of the following is seen: + +- The platform components `asm-otel-backend-collector` and `asm-otel-collector` keep restarting and cannot be restarted cleanly. The image field shown for them in the console keeps changing. +- Service Mesh JVM monitoring, or the `OpenTelemetry Java JVM` dashboard, shows no data for a Java application. +- While OpenTelemetry is being uninstalled so that OpenTelemetry v2 can be installed, the otel instance is deleted but stays in place, and the OpenTelemetry v2 installation is blocked. + +Related articles: "OpenTelemetry Collector logs cannot list resources pods because its ServiceAccount has no ClusterRoleBinding on ACP" covers a Collector that cannot list Pods. "Java auto-instrumentation init container is missing or the Java agent lands next to istio-proxy on Service Mesh v2" covers a pod where the Java agent was never injected. + +## Root Cause + +### Cause 1: a leftover 3.x `opentelemetry-operator` in `cpaas-system` fights the asm-otel collectors + +A Deployment named `opentelemetry-operator` runs in `cpaas-system`. That Deployment belongs only to 3.x environments. It conflicts with the current `asm-otel-*` components and keeps changing them, so both collectors restart. The Mesh v2 OpenTelemetry operator that serves tracing (for example `opentelemetry-operator-controller-manager` in `istio-system`) is not this leftover. Removing the old Deployment returns the collectors to normal; other objects left by that old operator should be cleaned up so they do not cause a later fault. + +### Cause 2: `asm-otel` was OOMKilled + +JVM metrics are not collected because the `asm-otel` collection component was OOMKilled. Raising its CPU and memory allocation lets it run and collect again. + +### Cause 3: the otel instance still carries a finalizer + +The OpenTelemetry Operator was uninstalled before the otel instance was removed, so nothing clears the instance's `finalizers` and the delete never completes. Removing the finalizer lets the delete finish, and the OpenTelemetry v2 installation is no longer blocked. + +### Cause 4: the Java agent cannot resolve the exporter endpoint in the `Instrumentation` + +The Java agent is injected (the auto-instrumentation init container ran and is `terminated`), but the exporter endpoint in the `Instrumentation` resource uses a full service name that does not resolve from the application Pod, for example a mistyped or wrong cluster domain in `http://otel-collector..svc.cluster.local:4318`. The agent's metric export times out, so no JVM series reach the Collector. The application container log shows: + +```text +ERROR io.opentelemetry.exporter.internal.http.HttpExporter - Failed to export metrics. The request could not be executed. Full error message: timeout +java.io.InterruptedIOException: timeout +``` + +The documented endpoint form `http://.:4318` resolves through the Pod's DNS search path. + +## Resolution + +### Cause 1: a leftover 3.x `opentelemetry-operator` in `cpaas-system` fights the asm-otel collectors + +1. Confirm that both collectors restart and that an `opentelemetry-operator` Deployment runs in `cpaas-system`: + + ```bash + kubectl get pod -A | grep -i opentelemetry-operator + kubectl get deploy,pods -n cpaas-system | grep -E 'opentelemetry-operator|asm-otel' + ``` + + Two operators, an old `opentelemetry-operator-` in `cpaas-system` (typically more than a year old) and `opentelemetry-operator-controller-manager-` in `istio-system`, match this cause. + +2. Back up and delete the old Deployment. + + > **Warning:** This stops the 3.x operator in `cpaas-system`. Only when both `asm-otel-backend-collector` and `asm-otel-collector` restart, the image field keeps changing, and this Deployment is in `cpaas-system`. Rollback: `kubectl apply -f /tmp/opentelemetry-operator-deploy.bak.yaml`. + + ```bash + kubectl get deploy -n cpaas-system opentelemetry-operator -o yaml > /tmp/opentelemetry-operator-deploy.bak.yaml + kubectl delete deploy -n cpaas-system opentelemetry-operator + kubectl get pods -A | grep -E 'asm-otel|opentelemetry-operator' + ``` + + The collectors should stay Running and the image field should stop changing. + +3. Clean up the other objects of the old operator that still exist. List them first: + + ```bash + kubectl get mutatingwebhookconfiguration,validatingwebhookconfiguration | grep opentelemetry-operator + kubectl get certificate,issuer,svc,role,rolebinding,sa,servicemonitor -n cpaas-system | grep opentelemetry-operator + kubectl get clusterrole,clusterrolebinding | grep opentelemetry-operator + ``` + + > **Warning:** These webhook, RBAC, certificate, and Service objects are the old operator's admission and identity; cluster-scoped deletes affect every namespace. Only after the Deployment in step 2 is gone, and only for objects with these exact names. Back up each object with `kubectl get [-n cpaas-system] -o yaml > /tmp/.bak.yaml` first. Rollback: `kubectl apply` of the backup (remove `resourceVersion`, `uid`, and `managedFields` from a webhook backup if apply complains). + + ```bash + kubectl delete mutatingwebhookconfiguration opentelemetry-operator-mutation + kubectl delete validatingwebhookconfiguration opentelemetry-operator-validation + kubectl delete certificate opentelemetry-operator-serving-cert -n cpaas-system + kubectl delete issuer opentelemetry-operator-selfsigned-issuer -n cpaas-system + kubectl delete svc opentelemetry-operator-webhook opentelemetry-operator -n cpaas-system + kubectl delete role,rolebinding opentelemetry-operator-leader-election -n cpaas-system + kubectl delete clusterrole,clusterrolebinding opentelemetry-operator-manager + kubectl delete sa opentelemetry-operator -n cpaas-system + kubectl delete servicemonitor opentelemetry-operator -n cpaas-system + ``` + + Run a line only for an object the listing showed. + +### Cause 2: `asm-otel` was OOMKilled + +1. Confirm the last state is OOMKilled: + + ```bash + kubectl get pods -A | grep asm-otel + kubectl describe pod -n | grep -A8 'Last State' + ``` + +2. Raise CPU and memory. On a ServiceMesh install, component sizes are set under `spec.componentConfig` of the ServiceMesh resource (entries `asmOtelCollector` and `asmOtelCollectorLB` for `asm-otel-collector` and `asm-otel-backend-collector`). Change them there so the next reconcile does not restore the old limits. + + > **Warning:** A higher request can leave the Pod Pending until a node has the capacity; a limit that is still too low leaves it OOMKilled. Only when JVM monitoring is empty and the Pod's last state is OOMKilled. Rollback: apply the backup. + + ```bash + kubectl get servicemesh -A -o yaml | grep -n -i -E 'asmOtel' + kubectl get servicemesh -n -o yaml > /tmp/servicemesh-otel.bak.yaml + kubectl edit servicemesh -n + ``` + +3. The new Pod should not be OOMKilled and its log should progress; JVM series are then collected again. + +### Cause 3: the otel instance still carries a finalizer + +1. Confirm the instance has a `deletionTimestamp` and a finalizer, and that the old operator is already gone: + + ```bash + kubectl get opentelemetrycollector,instrumentation -A + kubectl get -n -o jsonpath='{.metadata.deletionTimestamp}{" "}{.metadata.finalizers}{"\n"}' + kubectl get csv -A | grep -i opentelemetry + ``` + +2. Remove the finalizer. + + > **Warning:** The instance is deleted without the operator's cleanup; objects it created that are not garbage-collected through owner references stay and must be removed by hand. Only when the instance has a `deletionTimestamp`, a finalizer is set, and the old OpenTelemetry Operator is uninstalled. Rollback: the delete is irreversible; recreate the instance from the backup after removing `metadata.finalizers`, `deletionTimestamp`, `resourceVersion`, and `uid`. + + ```bash + kubectl get -n -o yaml > /tmp/otel-instance.bak.yaml + kubectl patch -n --type=merge -p '{"metadata":{"finalizers":null}}' + ``` + + `` is the kind of the stuck instance, for example `opentelemetrycollector`. Then continue with the OpenTelemetry v2 installation. + +### Cause 4: the Java agent cannot resolve the exporter endpoint in the `Instrumentation` + +1. Read the endpoint and test it from the application Pod (or any Pod in the same namespace that has `curl`): + + ```bash + kubectl get instrumentation -n -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.exporter.endpoint}{"\n"}{end}' + kubectl -n logs -c --tail=200 | grep -i 'Failed to export' + kubectl -n exec -- curl -s -o /dev/null -w '%{http_code}\n' http://.:4318 + ``` + + `could not resolve` for the configured name and an HTTP code (for example `404`) for the short name means the Collector port is reachable and only the name is wrong. + +2. Change the endpoint to `http://.:4318`. + + > **Warning:** The new endpoint applies to Pods injected after the change; restart the instrumented workloads so they pick it up, which recreates their Pods. Rollback: restore the previous endpoint value and restart again. + + ```bash + kubectl -n patch instrumentation --type=merge \ + -p '{"spec":{"exporter":{"endpoint":"http://.:4318"}}}' + kubectl -n rollout restart deployment + ``` + +3. Also check that the workload uses an `Instrumentation` it can find: declare one in each application namespace, or reference one in another namespace with the annotation value `/`, for example `instrumentation.opentelemetry.io/inject-java: "/"`. If the Collector has no logs pipeline, setting the environment variable `OTEL_LOGS_EXPORTER=none` in the `Instrumentation` (`spec.env`) stops the agent from exporting logs to it. + +4. After the restart, the `Failed to export metrics` errors should stop and the JVM dashboard should start receiving series. + +## Diagnostic Steps + +```bash +kubectl get pods -A | grep -E 'asm-otel|opentelemetry-operator' +kubectl get deploy -n cpaas-system | grep opentelemetry-operator +kubectl get instrumentation -A +``` + +| What you see | Cause | +| --- | --- | +| Both asm-otel collectors restart, image field keeps changing, `opentelemetry-operator` Deployment in `cpaas-system` next to the one in `istio-system` | 1 | +| JVM monitoring empty, `asm-otel` last state OOMKilled | 2 | +| otel instance stays after delete, `finalizers` set, operator already uninstalled | 3 | +| JVM dashboard empty, agent injected, application log shows `Failed to export metrics ... timeout`, configured endpoint does not resolve | 4 | diff --git a/docs/en/solutions/devops-controller_or_devops-apiserver_does_not_start_or_fails_requests_on_ACP.md b/docs/en/solutions/devops-controller_or_devops-apiserver_does_not_start_or_fails_requests_on_ACP.md new file mode 100644 index 000000000..faae8e5ad --- /dev/null +++ b/docs/en/solutions/devops-controller_or_devops-apiserver_does_not_start_or_fails_requests_on_ACP.md @@ -0,0 +1,144 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "3.12" + - "4.3" +--- + +# devops-controller or devops-apiserver does not start or fails requests on ACP + +## Issue + +The DevOps API components in `cpaas-system` on the `global` cluster are down or fail requests, and pipeline pages that depend on them stay empty or stale. One of these signatures is shown: + +- The `devops-controller` Pods are not Ready. The controller log repeats `Failed to watch v1alpha1.CodeRepository`, and the integrated GitLab answers `Internal server error` on `GET /api/v4/projects?membership=true&per_page=100`. +- On a standby (backup) data center, `devops-apiserver` and `devops-controller` are both down, and the controller log says it cannot get a ConfigMap because it has no permission. +- `devops-apiserver` crash-loops with `panic: unable to create REST storage for a resource due to context deadline exceeded, will die`, after normal INFO lines such as `cacher (*devops.JenkinsBinding): initialized`. +- Updating a pipeline fails with HTTP 500 and a message like `Get "https://[::1]:443/apis/devops.alauda.io/v1alpha1/clusterpipelinetasktemplates/?resourceVersion=0": x509: certificate has expired or is not yet valid`, while the platform certificate pages show nothing relevant as expired. + +A GitLab integration whose projects API returns 500 while these components are healthy is covered in "GitLab toolchain integration fails: token-gitlab not found, Internal error, projects API 500, or an extra bot project". An empty task list with these components Ready is covered in "Pipeline task list is empty, tasks are missing, or TektonHub cannot be viewed on ACP". + +## Root Cause + +### Cause 1: `Failed to watch v1alpha1.CodeRepository` because the integrated GitLab returns Internal server error + +`devops-controller` watches `CodeRepository` objects and calls each integrated GitLab to list projects. When one GitLab answers `Internal server error` on `/api/v4/projects`, the watch fails and the controller Pods do not become Ready. The broken dependency is that GitLab integration; deleting the abnormal integration lets `devops-controller` start again. Restarting the controller does not clear it. + +### Cause 2: the DevOps API ServiceAccount lacks permission to read ConfigMaps + +On a standby data center the ServiceAccount used by the DevOps API components had fewer permissions than on the primary. The controller cannot get the ConfigMap it needs, and both `devops-apiserver` and `devops-controller` stay down. Granting the ServiceAccount the missing permission lets both recover. + +### Cause 3: etcd does not answer in time while `devops-apiserver` builds its storage + +At startup `devops-apiserver` connects to etcd and creates REST storage for its resources. If storage creation does not finish in time, the process panics from `registry.RESTInPeace` and the Pod restarts. On ACP 3.12 this was caused by an abnormal control-plane node; after the workload moved off that node the panics stopped. The first lines of the log are ordinary INFO messages, so the panic is visible only in the full log of a failed start. + +### Cause 4: the DevOps API uses a serving certificate that has expired + +The 500 response comes from a call to the aggregated `devops.alauda.io/v1alpha1` API. When no certificate in the platform views matches the expiry time in the message, the most likely explanation is that the devops component on the `global` cluster is still presenting a serving certificate that has expired. The exact mechanism is not confirmed; the recommended action is to restart `devops-apiserver` and `devops-controller`. + +## Resolution + +### Cause 1: `Failed to watch v1alpha1.CodeRepository` because the integrated GitLab returns Internal server error + +1. Confirm the log signature. A controller that is down for an image pull is not this cause. + + ```bash + kubectl -n cpaas-system get pods | grep devops-controller + kubectl -n cpaas-system logs deploy/devops-controller --tail=80 | grep -E 'CodeRepository|Internal server error' + ``` + +2. Identify the GitLab integration whose host appears in that log, and check that its projects API really fails: + + ```bash + curl -sk -H "PRIVATE-TOKEN: " "https:///api/v4/projects?membership=true&per_page=100" | head -c 300 + ``` + +3. Delete that integration from the console page where it was created (the toolchain integration list), after recording its settings. + + > **Warning:** Deleting the GitLab integration removes that code-host binding; repositories, triggers, and pipelines that use it stop resolving until it is created again. Only when `devops-controller` is not Ready, the log shows `Failed to watch v1alpha1.CodeRepository`, and this GitLab returns `Internal server error` on `/api/v4/projects`. Rollback: recreate the integration with the recorded settings once the GitLab projects API returns a project list again. Do not delete a healthy integration. + +4. Watch the controller recover: + + ```bash + kubectl -n cpaas-system rollout status deploy/devops-controller + ``` + + If the Pods stay down and the log no longer mentions GitLab, follow the new log. + +### Cause 2: the DevOps API ServiceAccount lacks permission to read ConfigMaps + +1. Find the ServiceAccount and test the permission named in the log: + + ```bash + kubectl -n cpaas-system get deploy devops-controller -o jsonpath='{.spec.template.spec.serviceAccountName}{"\n"}' + kubectl auth can-i get configmaps -n cpaas-system --as=system:serviceaccount:cpaas-system: + kubectl get clusterrolebinding -o wide | grep + ``` + + `no` for that ServiceAccount is this cause. Compare its bindings with the same ServiceAccount on the primary data center. + +2. Restore the binding the primary cluster has for this ServiceAccount. If that binding is not available, binding the ServiceAccount to `cluster-admin` also lets the components recover. + + > **Warning:** A `cluster-admin` binding gives every Pod that runs as this ServiceAccount full control of the cluster. Only when the log shows the ConfigMap permission error and `kubectl auth can-i` returns `no`, and only when the primary's exact binding cannot be copied. Rollback: `kubectl delete clusterrolebinding `. + + ```bash + kubectl create clusterrolebinding --clusterrole=cluster-admin --serviceaccount=cpaas-system: + kubectl -n cpaas-system rollout restart deploy/devops-controller + ``` + +3. `devops-apiserver` and `devops-controller` should become Ready and the permission error should leave the log. + +### Cause 3: etcd does not answer in time while `devops-apiserver` builds its storage + +1. Read the log of the failed start and check the control plane: + + ```bash + kubectl -n cpaas-system logs deploy/devops-apiserver --previous | grep -E 'panic|deadline' + kubectl get nodes -l node-role.kubernetes.io/control-plane= + kubectl -n kube-system get pods -l component=etcd -o wide + ``` + + The panic together with a NotReady control-plane node or an etcd member that is not Running is this cause. + +2. Restarting `devops-apiserver` does not help while etcd is slow. Repair the control-plane node, or move off it, following the cluster's control-plane maintenance procedure. `devops-apiserver` should then start without the panic; if the panic returns, the control-plane problem is still open. + +### Cause 4: the DevOps API uses a serving certificate that has expired + +1. Only when the 500 body contains `x509: certificate has expired or is not yet valid` for a `devops.alauda.io` path. Check the APIService and how long the Pods have run: + + ```bash + kubectl get apiservice | grep devops.alauda.io + kubectl -n cpaas-system get pods -o wide | grep -E 'devops-apiserver|devops-controller' + kubectl -n cpaas-system logs deploy/devops-apiserver --tail=100 | grep -i x509 + ``` + +2. Restart both components on the `global` cluster. + + > **Warning:** The DevOps API is unavailable until the new Pods are Ready; pipeline pages and API calls fail in that window. Rollback: none; a restart changes no configuration. + + ```bash + kubectl -n cpaas-system rollout restart deploy/devops-apiserver deploy/devops-controller + kubectl -n cpaas-system rollout status deploy/devops-apiserver + kubectl -n cpaas-system rollout status deploy/devops-controller + ``` + +3. Save the pipeline again. If the x509 error returns after the restart, the certificate in its Secret is itself expired; collect the `devops-apiserver` log and `kubectl get apiservice -o yaml` for the product team. + +## Diagnostic Steps + +```bash +kubectl -n cpaas-system get deploy,pods | grep -E 'devops-controller|devops-apiserver' +kubectl -n cpaas-system logs deploy/devops-controller --tail=80 +kubectl -n cpaas-system logs deploy/devops-apiserver --previous --tail=200 +``` + +| What you see | Cause | +| --- | --- | +| `Failed to watch v1alpha1.CodeRepository` plus GitLab `Internal server error` on `/api/v4/projects` | 1 | +| Controller log: cannot get a ConfigMap, no permission; `kubectl auth can-i` returns `no` | 2 | +| `devops-apiserver` panics with `unable to create REST storage ... context deadline exceeded` | 3 | +| Pipeline update returns 500 with `x509: certificate has expired or is not yet valid` on a `devops.alauda.io` path | 4 | +| GitLab webhook delivery fails while `devops-controller` is Ready | Not this article | diff --git a/docs/en/solutions/kubectl_returns_the_server_has_asked_for_the_client_to_provide_credentials_after_ac_login_or_ac_login_fails_on_ACP.md b/docs/en/solutions/kubectl_returns_the_server_has_asked_for_the_client_to_provide_credentials_after_ac_login_or_ac_login_fails_on_ACP.md new file mode 100644 index 000000000..92aa30665 --- /dev/null +++ b/docs/en/solutions/kubectl_returns_the_server_has_asked_for_the_client_to_provide_credentials_after_ac_login_or_ac_login_fails_on_ACP.md @@ -0,0 +1,113 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" +--- + +# kubectl returns the server has asked for the client to provide credentials after ac login, or ac login fails, on ACP + +## Issue + +One of the following happens with the `ac` CLI or with platform API tokens: + +- The day after `ac login`, `kubectl` fails while the console still works: + + ```text + error: You must be logged in to the server (the server has asked for the client to provide credentials) + ``` + +- `ac login` returns HTTP 400, while `curl` against the platform address returns 302. +- The question is whether `ac login --idp` works with an Azure AD account. +- On ACP 4.2, a user token is needed from the command line or an API instead of the console. + +Related articles: "Forgotten local admin password on ACP: reset the password hash, set a chosen password, and password policy limits" covers local passwords. "ACP console session survives cookie deletion or a browser close, or the login page shows the previous account" covers browser sessions. + +## Root Cause + +### Cause 1: the kubeconfig written by `ac login` carries a one-day token + +`ac login`, as described in the Configure ACP CLI documentation, writes a kubeconfig whose token is valid for one day. After it expires, every `kubectl` call is rejected with the credentials error, while the console session is unaffected. + +### Cause 2: the installed `ac` binary is too old + +An old `ac` build sends a login request the platform rejects with 400, although the platform itself answers (`curl` returns 302). Reinstalling `ac` makes login succeed. + +### Cause 3: `ac login --idp` supports Windows AD, not Azure AD + +`ac login --idp` works with a Windows AD identity provider. Azure AD is not supported by that command. + +### Cause 4: on ACP 4.2 a user token is created with an API call and shown only once + +On ACP 4.2 a user token can be created with `POST /api/v1/users/{userId}/tokens`, authenticated, with a body that includes the token name and the expiration time. The response contains the token string, which cannot be read again after creation. + +## Resolution + +### Cause 1: the kubeconfig written by `ac login` carries a one-day token + +1. Create a long-lived token on the user's personal information page in the console. +2. In the kubeconfig written by `ac login`, replace the token of that user: + + ```yaml + users: + - name: + user: + token: + ``` + +3. Confirm that a call succeeds: + + ```bash + kubectl get --raw=/readyz + ``` + + It should print `ok` instead of the credentials error. + +### Cause 2: the installed `ac` binary is too old + +1. Compare the CLI and the platform response: + + ```bash + ac version + curl -k -sS -o /dev/null -w '%{http_code}\n' https:/// + ``` + + A 302 from `curl` with a 400 from `ac login` matches this cause. + +2. Download and install `ac` again as described in the Configure ACP CLI documentation for this platform version, then log in again. + +### Cause 3: `ac login --idp` supports Windows AD, not Azure AD + +Use `ac login --idp` for Windows AD identity providers. Azure AD users log in through the console. + +### Cause 4: on ACP 4.2 a user token is created with an API call and shown only once + +Send an authenticated `POST` to `/api/v1/users//tokens` with a JSON body that contains the token name and the expiration time. Take the exact field names from the API reference of your platform version. + +```bash +curl -k -sS -X POST \ + -H "Authorization: Bearer ${TOKEN}" \ + -H 'Content-Type: application/json' \ + --data '' \ + "https:///api/v1/users//tokens" +``` + +Copy the token string out of the response and store it securely; it cannot be retrieved later. Put it in a kubeconfig as in Cause 1 if `kubectl` should use it. + +## Diagnostic Steps + +```bash +kubectl config view --minify -o jsonpath='{.users[0].name}{"\n"}' +kubectl get --raw=/readyz +ac version +``` + +| What you see | Cause | +| --- | --- | +| `kubectl` credentials error about a day after `ac login`, console still works | 1 | +| `ac login` returns 400, `curl` to the platform returns 302 | 2 | +| Azure AD account cannot log in with `ac login --idp` | 3 | +| A user token is needed without the console on ACP 4.2 | 4 | +| The credentials error comes from a node's `/root/.kube/config` or `admin.conf`, not from `ac login` | Not this article | From fc204909da084d3773d71f48417621bcc5d1c686 Mon Sep 17 00:00:00 2001 From: Komh Date: Wed, 18 Nov 2026 17:41:13 +0800 Subject: [PATCH 18/25] docs(solutions): add 11 and update 2 troubleshooting articles --- ...Kafka_address_and_archive_limits_on_ACP.md | 137 ++++++++++ ...ts_log_node_fails_or_is_cordoned_on_ACP.md | 114 +++++++++ ...y_heap_OOM_or_small-scale_layout_on_ACP.md | 191 ++++++++++++++ ...-upgrade_indices_or_is_OOMKilled_on_ACP.md | 126 +++++++++ ...tion_time_while_monitoring_works_on_ACP.md | 115 +++++++++ ...ormal_pipelines_after_switchover_on_ACP.md | 136 ++++++++++ ...afka_2_operator_is_not_installed_on_ACP.md | 59 +++++ ...ention_or_undeletable_.nfs_files_on_ACP.md | 157 ++++++++++++ ...ressure_Ceph_write_errors_or_OOM_on_ACP.md | 231 +++++++++++++++++ ...t_or_KAFKA_HEAP_OPTS_not_applied_on_ACP.md | 160 ++++++++++++ ...r_Elasticsearch_on_the_root_disk_on_ACP.md | 87 +++++++ ...s_or_leftover_Elasticsearch_data_on_ACP.md | 55 +++- ...Killed_or_stale_ClickHouse_token_on_ACP.md | 242 ++++++++++++++++++ 13 files changed, 1809 insertions(+), 1 deletion(-) create mode 100644 docs/en/solutions/Archive_Elasticsearch_logs_to_NFS_logarchive_mount_log_center_Kafka_address_and_archive_limits_on_ACP.md create mode 100644 docs/en/solutions/Elasticsearch_Kafka_or_ZooKeeper_log_pod_stays_Pending_after_its_log_node_fails_or_is_cordoned_on_ACP.md create mode 100644 docs/en/solutions/Elasticsearch_log_storage_high_CPU_or_memory_too_many_shards_master_memory_heap_OOM_or_small-scale_layout_on_ACP.md create mode 100644 docs/en/solutions/Elasticsearch_log_storage_rejects_new_indices_at_the_shard_limit_keeps_pre-upgrade_indices_or_is_OOMKilled_on_ACP.md create mode 100644 docs/en/solutions/Inspection_reports_the_monitoring_component_abnormal_or_a_slow_collection_time_while_monitoring_works_on_ACP.md create mode 100644 docs/en/solutions/Inspection_reports_zero_free_inodes_missing_event_data_expired_dex.tls_or_abnormal_pipelines_after_switchover_on_ACP.md create mode 100644 docs/en/solutions/Kafka_2.x_instance_has_a_broker_offline_because_the_Alauda_Build_of_Kafka_2_operator_is_not_installed_on_ACP.md create mode 100644 docs/en/solutions/Log_archive_output_platform_logs_with_workload_logs_flat_S3_objects_file_retention_or_undeletable_.nfs_files_on_ACP.md create mode 100644 docs/en/solutions/Platform_log_Kafka_brokers_fail_no_leader_election_exit_137_Pending_on_DiskPressure_Ceph_write_errors_or_OOM_on_ACP.md create mode 100644 docs/en/solutions/cpaas-kafka_retention_hours_memory_limit_above_80_percent_or_KAFKA_HEAP_OPTS_not_applied_on_ACP.md create mode 100644 docs/en/solutions/razor_crash-loops_or_audit_logs_stop_TOO_MANY_PARTS_BufferTooSmall_vector_OOMKilled_or_stale_ClickHouse_token_on_ACP.md diff --git a/docs/en/solutions/Archive_Elasticsearch_logs_to_NFS_logarchive_mount_log_center_Kafka_address_and_archive_limits_on_ACP.md b/docs/en/solutions/Archive_Elasticsearch_logs_to_NFS_logarchive_mount_log_center_Kafka_address_and_archive_limits_on_ACP.md new file mode 100644 index 000000000..159a0920d --- /dev/null +++ b/docs/en/solutions/Archive_Elasticsearch_logs_to_NFS_logarchive_mount_log_center_Kafka_address_and_archive_limits_on_ACP.md @@ -0,0 +1,137 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.14.x + - 3.16.x + - 4.2.x + - 4.3.x +--- + +# Archive Elasticsearch logs to NFS: logarchive mount, log center Kafka address, and archive limits on ACP + +## Issue + +Platform logs stored in Elasticsearch should also be kept on an external NAS or NFS share, and the archive deployment or its result raises questions: + +- The archive manifest has `hostPath` `/cpaas/data/logarchive`. Must the NFS export be mounted on the node first, or should that path be replaced with the NFS server's directory? +- Which Kafka goes into the archive form field `Kafka 服务地址` (Kafka service address): the Kafka that came with the log storage, or a new Kafka from the Data Services catalog? +- Local Elasticsearch should keep about 7 days and older logs should move to the NAS. How are logs older than 7 days read afterwards? +- The archive copies the logs of every project and uses too much space; can it archive only selected projects? +- Logs must be kept 180 days, but the platform allows at most 30 days, and a 180-day lifecycle set in Kibana is reset. +- With ClickHouse log storage on 4.2 or 4.3, can logs be archived or cold-backed up to NFS or NAS? + +The archive is the platform documentation's log archiving solution: "Transfer to external NFS" on 3.14, and "How to archive logs to third-party storage" (Observability, Logs, Practical Guides) on 4.x, which writes to an external NFS or S3 through a log sync workload created with `kubectl`. What a running archive writes (scopes, S3 layout, file retention, files that cannot be deleted) is in [Log archive output: platform logs with workload logs, flat S3 objects, file retention, or undeletable .nfs files on ACP](Log_archive_output_platform_logs_with_workload_logs_flat_S3_objects_file_retention_or_undeletable_.nfs_files_on_ACP.md). + +## Root Cause + +### Cause 1: /cpaas/data/logarchive is the host directory, and the NFS export must be mounted on it before the pod runs there + +The log sync workload mounts a `hostPath` volume `data` with `path: /cpaas/data/logarchive` and `type: DirectoryOrCreate`. The path stays as it is in the manifest; it is not rewritten to the NFS server's export. Because `DirectoryOrCreate` creates a local directory when nothing is mounted there, the NFS export has to be mounted on `/cpaas/data/logarchive` on the node before the pod starts on that node, and the pod has to be pinned to that node. Otherwise the archive writes to the node's local disk. The working sequence: label the node, schedule the archive workload onto it, create the directory, mount NFS on it. Whether NFS v3 needs `nolock` was not part of the verified setup. + +### Cause 2: the Kafka service address is the log center Kafka, not a new Data Services Kafka + +The log collection plugins write the collected logs into the log center Kafka, and the archive component reads them from there. The archive form's `Kafka 服务地址` (Kafka service address) is therefore the platform's built-in log center Kafka. A Kafka instance created from the Data Services catalog receives no logs, so an archive pointed at it stays empty. + +### Cause 3: the archive is a real-time copy, not a move after N days + +The archive writes a copy of each new log to the NFS share as it is produced. It does not move logs to the NAS after they have been kept locally for some days, and no supported option tiers Elasticsearch logs by age. Log queries within the configured retention are not affected. Logs older than the retention are read from the archive path, not through the log query page; archived files can be read directly after any compressed file is decompressed. + +### Cause 4: the archive is cluster-wide and cannot be limited to selected projects + +The archive works at cluster level and archives the logs of every project of the cluster (verified on 3.16.2). Archiving only some projects is not supported, so the archive storage has to be sized for all projects. + +### Cause 5: logs must be kept longer than the platform retention allows + +The platform caps log retention (30 days for a project policy and in 4.x consoles), and a lifecycle edited in Kibana is overwritten by the platform policy. On 3.16 the Elasticsearch retention can be set to 180 days, but adding disks to keep 180 days in place is not recommended, and later versions cap log retention at 30 days. For Elasticsearch, the long-term copy is the log archive to NFS. For ClickHouse log storage in 4.x, the logging plugin allows at most 30 days of log retention, events and audit records have their own, longer maximum (read it on the plugin form), and no supported way to keep logs longer is known. + +### Cause 6: ClickHouse log storage on 4.2 and 4.3 has no log archive to NFS or NAS + +The log archiving in the documentation applies to Elasticsearch storage. ClickHouse log storage on 4.2 and 4.3 has no log archive, so the archive guide does not apply to it and backing up ClickHouse logs to a NAS is not supported. What ClickHouse logging offers instead is S3 integration with hot/cold separation, available from the 4.3 logging component. A 4.2 platform gets it by upgrading the platform to 4.2.5 and then the logging component to 4.3. Hot/cold separation tiers data inside the retention the plugin allows; it is not a way past the 30-day log cap. + +## Resolution + +### Cause 1: /cpaas/data/logarchive is the host directory, and the NFS export must be mounted on it before the pod runs there + +1. Read the two images the documented manifest needs, on the global cluster: + + ```bash + # alpine image + kubectl get daemonset nevermore -n cpaas-system -o jsonpath='{.spec.template.spec.initContainers[0].image}' + # log sync image: the 3.14 document reads it from lanaya, the 4.x guide from razor + kubectl get deployment lanaya -n cpaas-system -o jsonpath='{.spec.template.spec.containers[0].image}' + kubectl get deployment razor -n cpaas-system -o jsonpath='{.spec.template.spec.containers[0].image}' + ``` + +2. On the node that will run the archive, create the directory and mount the NFS export on it. + + > **Warning:** Mounting NFS on `/cpaas/data/logarchive` hides any files already in that directory while the mount is in place. Precondition: the directory is empty or its contents were copied aside, and `:` is the archive target. Rollback: `umount /cpaas/data/logarchive`. Persist the mount the way this node persists its other NFS mounts, so a reboot does not leave the archive writing to local disk. + + ```bash + mkdir -p /cpaas/data/logarchive + mount -t nfs : /cpaas/data/logarchive + df -hT /cpaas/data/logarchive + ``` + + `df` must show an `nfs` type, not the root disk. + +3. Pin the archive workload to that node, either with a node label that the workload's node selector uses or with the node name if the manifest takes one: + + ```bash + kubectl label node =true + ``` + +4. Apply the manifest from the documentation, keeping `/cpaas/data/logarchive` unchanged. + +### Cause 2: the Kafka service address is the log center Kafka, not a new Data Services Kafka + +Read the log center Kafka service and use the broker address the log pipeline already produces to: + +```bash +kubectl -n cpaas-system get svc | grep -i kafka +kubectl -n cpaas-system get pods | grep -i kafka +``` + +### Cause 3: the archive is a real-time copy, not a move after N days + +Set the local retention to the number of days that must stay searchable, and deploy the archive as in Causes 1 and 2 so every new log is also copied. Read older logs on the share, for example: + +```bash +ls -lt /cpaas/data/logarchive | head +find /cpaas/data/logarchive -name '*.gz' -mtime +7 | head +zcat .gz | less +``` + +List the directory first; use the decompressor that matches the files if they are not gzip. + +### Cause 4: the archive is cluster-wide and cannot be limited to selected projects + +No configuration narrows it. Size the archive storage for the logs of all projects of the cluster. + +### Cause 5: logs must be kept longer than the platform retention allows + +On Elasticsearch, keep the platform retention within its cap and deploy the archive to NFS for the long-term copy; do not rely on Kibana lifecycle edits. On ClickHouse, 30 days is the maximum log retention in the 4.x logging plugin; a longer retention has to be raised as a feature request. + +### Cause 6: ClickHouse log storage on 4.2 and 4.3 has no log archive to NFS or NAS + +Check the documentation of the installed logging component; offer ClickHouse archiving only if that version lists it. To move retained ClickHouse logs off local disk, use S3 hot/cold storage from the 4.3 logging component. On 4.2: + +1. Upgrade the platform to 4.2.5. +2. Upgrade the logging component to 4.3. +3. Configure S3 as described in "How to Use S3 Storage with ClickHouse" in the logging documentation. + +## Diagnostic Steps + +1. On the archive node, confirm the mount and the pod placement: + + ```bash + findmnt /cpaas/data/logarchive + kubectl get pods -A -o wide | grep -iE 'log-export|archive' + ``` + + `findmnt` must show `nfs`, and the archive pod's node must be the node with the mount. A local file system type means the archive writes to local disk (Cause 1). + +2. If the archive stays empty while Elasticsearch has recent logs, compare the configured Kafka address with the brokers listed in Cause 2. The archive only receives what the collectors write to the log center Kafka. +3. Which log store runs: `kubectl get pods -A | grep -iE 'clickhouse|cpaas-elasticsearch'`. A ClickHouse store asking for archive or NAS backup is Cause 6, and a request to keep logs longer than 30 days is Cause 5. For Elasticsearch, tiering by age is Cause 3 and selected projects is Cause 4. diff --git a/docs/en/solutions/Elasticsearch_Kafka_or_ZooKeeper_log_pod_stays_Pending_after_its_log_node_fails_or_is_cordoned_on_ACP.md b/docs/en/solutions/Elasticsearch_Kafka_or_ZooKeeper_log_pod_stays_Pending_after_its_log_node_fails_or_is_cordoned_on_ACP.md new file mode 100644 index 000000000..87b00a397 --- /dev/null +++ b/docs/en/solutions/Elasticsearch_Kafka_or_ZooKeeper_log_pod_stays_Pending_after_its_log_node_fails_or_is_cordoned_on_ACP.md @@ -0,0 +1,114 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# Elasticsearch, Kafka, or ZooKeeper log pod stays Pending after its log node fails or is cordoned on ACP + +## Issue + +The log storage plugin `Alauda Container Platform Log Storage for Elasticsearch` uses local storage, and a log pod does not move: + +- A node that hosts Elasticsearch, Kafka, and ZooKeeper failed and was taken offline. `cpaas-elasticsearch-0`, `cpaas-kafka-1`, and `cpaas-zookeeper-2` (for example) are `Pending` with no node assigned, or stay `Terminating`, while the other two replicas of each are `Running`. The question is how to move them to another node. +- Replacing the failed node with another node in the plugin's log node list does not take effect: the plugin shows `安装失败` (installation failed), and the Kafka node list cannot be changed. +- A node that hosts Elasticsearch must be shut down for maintenance. +- After a resource change, a scale to 0 and back, or a PVC resize, an Elasticsearch, Kafka, or ZooKeeper pod stays `Pending` on a log node that was made unschedulable. +- A new cluster should reserve 3 nodes for logging only, and the question is which access mode the local log disks should use. + +## Root Cause + +### Cause 1: log pods are pinned to their node by local persistent volumes + +With LocalVolume storage the plugin creates local PVs whose node affinity names one node, and the data sits under `/cpaas/data` on that node. Each Elasticsearch, Kafka, and ZooKeeper pod therefore belongs to its node: when the node fails, the pod does not move to another node, it waits for its node to return. Local volumes are single-node read-write (ReadWriteOnce), which is the correct mode for local disks; block storage such as TopoLVM or Ceph RBD also gives single-node access. + +With three replicas, log collection continues on the two remaining replicas. In one case, Kafka production and consumption stayed matched (both dropped from about 800 to about 550 messages per minute) and the Elasticsearch write queue stayed near zero, so the cluster ran on two nodes until the failed node was repaired and rejoined; the Elasticsearch pod then returned to `Running` on it. Elasticsearch was `red` afterwards with unassigned shards (8 shards, 98.5% active), and deleting expired historical indices returned it to normal. + +During a planned outage of a data node in a large-scale layout, Elasticsearch allocates the replicas of that node's shards elsewhere and writes nothing new to it; when the node returns, shards are allocated back. For a data node down about 6 hours on a lightly loaded cluster, the impact on logging and on the global cluster was small. + +### Cause 2: a recreated log pod stays Pending on a cordoned log node + +A cordoned node accepts no new pods; pods that keep running there were scheduled before the cordon. Cordoning the log nodes after Elasticsearch is deployed is the way to keep them for logging only. If a log pod is then deleted or recreated (a resource change, a scale to 0 and back, a PVC resize), its local PV ties it to the cordoned node, so it stays `Pending` until the node is uncordoned. This is the expected side effect of that setup. + +## Resolution + +### Cause 1: log pods are pinned to their node by local persistent volumes + +1. Confirm that the pods wait for their node: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep -E 'cpaas-elasticsearch|cpaas-kafka|cpaas-zookeeper' + kubectl get pv -o custom-columns=NAME:.metadata.name,CLAIM:.spec.claimRef.name,NODE:.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0] | grep -iE 'elasticsearch|kafka|zookeeper' + ``` + +2. Bring the failed node back; the pods follow the node, not the scheduler. Do not replace the node in the plugin's log node list to move the pods: that does not take effect and leaves the plugin in a failed state. +3. While the node is down, check that the remaining replicas keep up: on the Kafka monitoring dashboard, production and consumption rates should match and the Elasticsearch write queue should stay near zero, and log queries should stay current. +4. After the node is `Ready` and the pods are `Running`, check Elasticsearch health: + + ```bash + ES_SVC=$(kubectl -n cpaas-system get svc cpaas-elasticsearch -o jsonpath='{.spec.clusterIP}') + ES_USER=$(kubectl -n cpaas-system get secret elasticsearch-basic-auth -o jsonpath='{.data.username}' | base64 -d) + ES_PASS=$(kubectl -n cpaas-system get secret elasticsearch-basic-auth -o jsonpath='{.data.password}' | base64 -d) + curl -s -u "$ES_USER:$ES_PASS" "http://$ES_SVC:9200/_cluster/health?pretty" + curl -s -u "$ES_USER:$ES_PASS" "http://$ES_SVC:9200/_cat/shards?v" | grep UNASSIGNED + ``` + + If it is `red` and the unassigned shards belong to expired historical indices, delete those indices by name, as described in [Elasticsearch log indices red or yellow with unassigned shards on ACP](Elasticsearch_log_indices_red_or_yellow_with_unassigned_shards_on_ACP.md). + +5. For a planned shutdown of a node that also hosts Kafka and ZooKeeper, or whose Elasticsearch role is not known, stop the logging components first and do not drain; see [Shutting down or rebooting ACP nodes for maintenance](Shutting_down_or_rebooting_ACP_nodes_for_maintenance.md). The cordon and drain below applies only to a confirmed Elasticsearch data node in a large-scale layout with enough remaining data nodes: + + ```bash + curl -s -u "$ES_USER:$ES_PASS" "http://$ES_SVC:9200/_cluster/health?pretty" + kubectl cordon + ``` + + > **Warning:** Draining evicts every pod on the node; the Elasticsearch data pod on it stays `Pending` until the node returns, and the cluster runs with one data node fewer (yellow while replicas are reallocated). Precondition: the cluster is green, the layout is large scale with enough remaining data nodes, and the maintenance window is agreed. Rollback: `kubectl uncordon ` after the node is back; Elasticsearch allocates shards back. + + ```bash + kubectl drain --ignore-daemonsets --delete-emptydir-data + ``` + + Power off, do the maintenance, start the node, run `kubectl uncordon `, and wait for health to return to green. + +### Cause 2: a recreated log pod stays Pending on a cordoned log node + +To reserve nodes for logging, deploy Elasticsearch on them first and then cordon them: + +```bash +kubectl -n cpaas-system get pods -o wide | grep -E 'cpaas-elasticsearch|cpaas-kafka|cpaas-zookeeper' +kubectl cordon +``` + +Any later step that recreates a log pod on those nodes needs an uncordon first and a cordon afterwards. + +> **Warning:** While the log nodes are uncordoned, the scheduler can place other workloads on them. Precondition: the change that recreates the log pods is ready to run right away. Rollback: cordon the nodes again as soon as the log pods are `Running`; pods that landed there in the meantime must be moved by their owners. + +```bash +kubectl uncordon +# run the change that recreates the log pods, then wait until each is Running on its node +kubectl -n cpaas-system get pods -o wide | grep -E 'cpaas-elasticsearch|cpaas-kafka|cpaas-zookeeper' +kubectl cordon +``` + +## Diagnostic Steps + +1. A `Pending` log pod after its node went down, with its PV bound to that node, is Cause 1. +2. A `Pending` log pod whose PV node is `Ready` but `SchedulingDisabled` is Cause 2: + + ```bash + kubectl get nodes | grep SchedulingDisabled + kubectl -n cpaas-system describe pod | sed -n '/Events/,$p' + ``` + +3. The access mode of the existing log volumes: + + ```bash + kubectl get pvc -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,MODES:.spec.accessModes,SC:.spec.storageClassName | grep -E 'elasticsearch|kafka|zookeeper' + ``` + +4. A log pod that is `Pending` because of `node.kubernetes.io/disk-pressure`, not a down or cordoned node, is a full disk; see [Elasticsearch log storage Pending or every cluster loses logs after the log disk fills on ACP](Elasticsearch_log_storage_Pending_or_every_cluster_loses_logs_after_the_log_disk_fills_on_ACP.md). diff --git a/docs/en/solutions/Elasticsearch_log_storage_high_CPU_or_memory_too_many_shards_master_memory_heap_OOM_or_small-scale_layout_on_ACP.md b/docs/en/solutions/Elasticsearch_log_storage_high_CPU_or_memory_too_many_shards_master_memory_heap_OOM_or_small-scale_layout_on_ACP.md new file mode 100644 index 000000000..9c76c6783 --- /dev/null +++ b/docs/en/solutions/Elasticsearch_log_storage_high_CPU_or_memory_too_many_shards_master_memory_heap_OOM_or_small-scale_layout_on_ACP.md @@ -0,0 +1,191 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.14.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Elasticsearch log storage high CPU or memory: too many shards, master memory, heap OOM, or small-scale layout on ACP + +## Issue + +The platform log store on Elasticsearch alarms or degrades while its pods stay `Running`: + +- Inspection of the global cluster reports "Elasticsearch cluster status > 0", "Es cpu utilization > 90" (CPU of the containers under `cpaas-elasticsearch` above 90% for 1 minute), and "Kafka consumer lag growth rate > 0" together, while all `cpaas-elasticsearch-*`, `cpaas-kafka-*`, `lanaya-*`, and `razor-*` pods are `1/1 Running`. +- One `cpaas-elasticsearch-master` pod stays above 90% memory for days while the other two masters are around 50% and 70%, with no errors in its log. +- Data-node memory was raised, but CPU on the master pod stays high. +- A data pod restarts with `java.lang.OutOfMemoryError: Java heap space`, or `_cat/indices` returns `circuit_breaking_exception` (parent breaker) while the cluster is yellow with many unassigned shards. +- Elasticsearch pods are above 80% of their resources, a write queue builds, and Kafka lag keeps rising, although another cluster with more daily logs is fine. + +Elasticsearch runs in `cpaas-system` as StatefulSet `cpaas-elasticsearch` (data nodes) and, in the large-scale layout only, `cpaas-elasticsearch-master` (dedicated masters that manage cluster metadata and shard allocation and hold no shards). The plugin `Alauda Container Platform Log Storage for Elasticsearch` keeps its settings in the `logcenter` ModuleInfo on the global cluster: `spec.config.components.elasticsearch.resources` (data nodes), `masterResources` (masters), and `type`, which is `single`, `normal` (small scale: every pod is both master and data), or `big` (large scale: dedicated master and data pods). + +Index creation rejected at the shard limit, pre-upgrade indices, and an OOM from the first start are in [Elasticsearch log storage rejects new indices at the shard limit, keeps pre-upgrade indices, or is OOMKilled on ACP](Elasticsearch_log_storage_rejects_new_indices_at_the_shard_limit_keeps_pre-upgrade_indices_or_is_OOMKilled_on_ACP.md). + +## Root Cause + +### Cause 1: more than 2,000 shards on 2 CPU / 4Gi data nodes saturate CPU and leave shards unassigned + +The data pods run at about 89% to 97% CPU, the cluster status row is above 0, and `_cat/shards` lists `UNASSIGNED` shards across `event-*`, `audit-*`, `meter-detail-*`, and `log-workload-*`. More than 2,000 shards on data nodes sized 2 CPU / 4Gi is more than the nodes can serve, so CPU saturates and shards cannot be allocated. The Kafka lag alarm is the downstream effect. + +### Cause 2: one dedicated master stays above 90% memory with about 2,000 shards and 180-day audit retention + +Memory is uneven across the dedicated masters while data-node usage is even. The cluster state that the elected master holds grows with the number of indices and shards; with about 2,000 shards and audit indices kept for 180 days, the master's memory stays high. Deleting half of the audit indices was the relief applied; watch the result before deleting more. + +### Cause 3: master CPU stays high because the elected master runs out of JVM heap + +CPU on a master pod stays high after data-node memory was raised, and health is not green. The single elected master does all shard management and metadata work, and its JVM heap is exhausted; the CPU is the JVM fighting for heap, not query load. Giving the master a larger heap (Java options on the master configuration) ended the OOM and the cluster recovered. + +### Cause 4: a data node heap far below its memory limit, or a parent circuit breaker trip + +A data pod logs `java.lang.OutOfMemoryError: Java heap space` and restarts. The heap of the running process can be far smaller than the container allows (one node with a `12c24g` limit ran with `-Xmx1g`). Setting `ES_JAVA_OPTS` to `-Xms12g -Xmx12g`, about half the memory limit, stopped the crash. The pod spec is not proof of the heap in use; read the running process. + +`circuit_breaking_exception` from the parent breaker, with the cluster yellow, initializing shards, and many unassigned shards, is the same memory pressure. Raising the Elasticsearch limit to 8 CPU / 16Gi and retrying the failed allocations recovered that cluster. + +### Cause 5: the small-scale layout, where every pod is both master and data, saturates under load + +With `type: normal`, every pod serves reads and writes, holds data, and holds the cluster metadata. The two roles compete for CPU and memory, which leads to GC pauses and heartbeat timeouts. This shows up as pods above 80% of their resources with a moderate daily volume (1,111 indices on a 3 x 16G small-scale cluster), or as a write queue and rising Kafka lag on a small-scale cluster that receives the logs of more than ten business clusters (Elasticsearch at 8 CPU / 16G and 60% to 70% used, Kafka at 2 CPU / 4G, production steady while consumption comes only in bursts). Moving to the large-scale layout with dedicated masters (3 masters with 3 or 5 data pods) resolved both. After the switch, master and data roles are no longer combined in one pod; a master pod and a data pod may still run on the same host, and the switch does not affect business workloads. + +## Resolution + +Set the connection variables on a node of the cluster that runs Elasticsearch: + +```bash +ES_SVC=$(kubectl -n cpaas-system get svc cpaas-elasticsearch -o jsonpath='{.spec.clusterIP}') +ES_USER=$(kubectl -n cpaas-system get secret elasticsearch-basic-auth -o jsonpath='{.data.username}' | base64 -d) +ES_PASS=$(kubectl -n cpaas-system get secret elasticsearch-basic-auth -o jsonpath='{.data.password}' | base64 -d) +curl -s -u "$ES_USER:$ES_PASS" "http://$ES_SVC:9200/" +``` + +The shell then holds the password; do not share the transcript. Find the ModuleInfo with `kubectl get moduleinfo -A | grep logcenter | grep ` and save it before any change: `kubectl get moduleinfo -o yaml > /tmp/logcenter-minfo.before.yaml`. + +### Cause 1: more than 2,000 shards on 2 CPU / 4Gi data nodes saturate CPU and leave shards unassigned + +Only when `active_shards` is above about 2,000, `UNASSIGNED` shards exist, and the data pods are limited to 2 CPU / 4Gi. Raising them to 4 CPU / 8Gi cleared the alarms and let reallocation start. + +> **Warning:** Changing the Elasticsearch resources rolls the `cpaas-elasticsearch` pods one by one; log search degrades and Kafka lag grows until they are Ready. Precondition: the signature above is confirmed and each data node has 4 CPU and 8Gi free. Rollback: `kubectl apply -f /tmp/logcenter-minfo.before.yaml`, or set 2 CPU / 4Gi again in the plugin form. + +Update the plugin in the console, or: + +```bash +kubectl edit moduleinfo +``` + +Set `spec.config.components.elasticsearch.resources.limits` to `cpu: "4"` and `memory: 8Gi`, then watch the unassigned count fall: + +```bash +curl -s -u "$ES_USER:$ES_PASS" "http://$ES_SVC:9200/_cluster/health?pretty" +``` + +### Cause 2: one dedicated master stays above 90% memory with about 2,000 shards and 180-day audit retention + +Only when the high memory is on a master pod, the shard count is around 2,000, and audit indices cover a long retention. List them oldest first: + +```bash +curl -s -u "$ES_USER:$ES_PASS" "http://$ES_SVC:9200/_cat/indices/audit-*?v&h=index,pri,rep,store.size&s=index" +``` + +> **Warning:** Deleting an index is irreversible; the audit records in it are gone, including for compliance review. Precondition: the owner approved the date range, any required audit export was taken, and the names come from the listing above. Rollback: none without a snapshot. + +```bash +curl -s -u "$ES_USER:$ES_PASS" -X DELETE "http://$ES_SVC:9200/," +``` + +Then shorten the audit retention in the log storage settings; otherwise new daily indices bring the shard count back. Watch master memory afterwards. + +### Cause 3: master CPU stays high because the elected master runs out of JVM heap + +Confirm heap exhaustion on the elected master: + +```bash +curl -s -u "$ES_USER:$ES_PASS" "http://$ES_SVC:9200/_cat/master?v" +curl -s -u "$ES_USER:$ES_PASS" "http://$ES_SVC:9200/_cat/nodes?v&h=name,node.role,master,heap.percent,heap.max,cpu" +kubectl -n cpaas-system logs --tail=500 | grep -iE 'OutOfMemoryError|java heap' +kubectl -n cpaas-system get sts cpaas-elasticsearch-master -o yaml | grep -n -iE 'java|Xmx|Xms' +``` + +Give the masters a larger heap through their Java options, sized from the master memory limit (`masterResources`); raise that limit too if the heap would not fit in about half of it. + +> **Warning:** Changing the master Java options or `masterResources` restarts the master pods one at a time; while the elected master restarts, index creation and shard allocation pause. Precondition: you saved the ModuleInfo and the master StatefulSet YAML, and the nodes have the extra memory. Rollback: apply the saved objects. + +Afterwards `_cat/nodes` shows a larger `heap.max` on the masters, and master CPU drops once health is green. + +### Cause 4: a data node heap far below its memory limit, or a parent circuit breaker trip + +1. Read the heap of the running process and the memory limit: + + ```bash + kubectl -n cpaas-system exec cpaas-elasticsearch-0 -c cpaas-elasticsearch -- ps -ef | grep -o -- '-Xm[sx][^ ]*' + kubectl -n cpaas-system get sts cpaas-elasticsearch -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" "}{.resources.limits}{"\n"}{end}' + ``` + +2. If the heap is far below half the limit, set it to about half. + + > **Warning:** Editing `ES_JAVA_OPTS` rolls Elasticsearch; search and ingest stop on each pod while it restarts. A heap larger than about half the limit, or larger than the node can give, gets the pod `OOMKilled` (exit 137) instead. Precondition: you saved the StatefulSet YAML, and the new `-Xmx` is about half the memory limit (12g on 24Gi, 24g on 48Gi). Rollback: `kubectl apply -f` the saved YAML. + + ```bash + kubectl -n cpaas-system get sts cpaas-elasticsearch -o yaml > /tmp/cpaas-elasticsearch-sts.before.yaml + kubectl -n cpaas-system edit sts cpaas-elasticsearch + ``` + + Set `ES_JAVA_OPTS` to `-Xmsg -Xmxg`. Do not set the heap through a resource patch (`rpch`) override; one such override started a pod with `-Xmx0m`. If a plugin update reverts the StatefulSet, make the same change where the plugin keeps its Elasticsearch settings. + +3. For the parent circuit breaker, raise the data-node limit (8 CPU / 16Gi in the recovered case) the same way as Cause 1, then retry failed allocations. + + > **Warning:** Retrying allocation starts shard copies; on a disk that is over the watermark this fills the node again. Precondition: health is yellow because of allocation, not red, and `df` on the data nodes has room. Rollback: none for copies already started. + + ```bash + curl -s -u "$ES_USER:$ES_PASS" -X POST "http://$ES_SVC:9200/_cluster/reroute?retry_failed=true" + ``` + +### Cause 5: the small-scale layout, where every pod is both master and data, saturates under load + +Confirm the layout and the write pressure: + +```bash +kubectl get moduleinfo -o jsonpath='{.spec.config.components.elasticsearch.type}{"\n"}' +curl -s -u "$ES_USER:$ES_PASS" "http://$ES_SVC:9200/_cat/indices?h=index" | wc -l +curl -s -u "$ES_USER:$ES_PASS" "http://$ES_SVC:9200/_cat/thread_pool/write?v&h=node_name,active,queue,rejected" +``` + +Switch to large scale: + +- On 3.x consoles, update the Elasticsearch log storage plugin in Cluster Plugins and change the scale to large. +- From 4.0, edit the `logcenter` ModuleInfo as in [Elasticsearch Scale Switching](Elasticsearch_Scale_Switching.md): add data nodes in `spec.config.components.elasticsearch.k8sNodes`, master nodes in `masterK8sNodes` (3 masters with 5 data nodes was the recommendation), set data-node `resources.limits` to at least the large-scale default of 8 CPU / 16Gi, and change `type` from `normal` to `big`. Node lists and limits can also be changed later in the plugin form; `type` is changed only in the ModuleInfo. + +> **Warning:** The switch restarts the Elasticsearch components; logs cannot be queried until the new pods are up. Business workloads are not affected and stored logs are kept. Precondition: the new master and data nodes have the resources and `/cpaas` disk for the large-scale layout, you saved the ModuleInfo, and a gap in log search is acceptable. Rollback: no return path from `big` to `normal` is documented; plan the switch as a forward change. + +```bash +kubectl edit moduleinfo +``` + +Confirm the plugin shows as installed and logs can be queried. While the switch is being planned, raising Kafka from 2 CPU / 4G to 4 CPU / 8G and `lanaya` from 2 to 4 or 6 replicas eased the write queue. + +## Diagnostic Steps + +1. Health, roles, heap, and shards: + + ```bash + curl -s -u "$ES_USER:$ES_PASS" "http://$ES_SVC:9200/_cluster/health?pretty" + curl -s -u "$ES_USER:$ES_PASS" "http://$ES_SVC:9200/_cat/nodes?v&h=name,node.role,master,cpu,heap.percent,ram.percent" + curl -s -u "$ES_USER:$ES_PASS" "http://$ES_SVC:9200/_cat/shards?v" | grep UNASSIGNED | head -20 + kubectl -n cpaas-system get pods -o wide | grep -E 'elasticsearch|kafka|lanaya|razor' + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | `active_shards` above about 2,000, `UNASSIGNED` shards, data pods at a 2-core limit | 1 | + | One master with much higher `ram.percent` than the others, long audit retention | 2 | + | Master at high CPU, elected master `heap.percent` near 100, health not green | 3 | + | `Java heap space` on a data pod with a small `-Xmx`, or `circuit_breaking_exception` | 4 | + | Every node shows both master and data roles, cluster saturated | 5 | + +3. If plain HTTP to the Service is refused, read the address the platform uses (`kubectl -n cpaas-system get feature log-center -o jsonpath='{.spec.accessInfo.elasticsearch.address}'`) and call it with `curl -k`. diff --git a/docs/en/solutions/Elasticsearch_log_storage_rejects_new_indices_at_the_shard_limit_keeps_pre-upgrade_indices_or_is_OOMKilled_on_ACP.md b/docs/en/solutions/Elasticsearch_log_storage_rejects_new_indices_at_the_shard_limit_keeps_pre-upgrade_indices_or_is_OOMKilled_on_ACP.md new file mode 100644 index 000000000..7150a9312 --- /dev/null +++ b/docs/en/solutions/Elasticsearch_log_storage_rejects_new_indices_at_the_shard_limit_keeps_pre-upgrade_indices_or_is_OOMKilled_on_ACP.md @@ -0,0 +1,126 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.14.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Elasticsearch log storage rejects new indices at the shard limit, keeps pre-upgrade indices, or is OOMKilled on ACP + +## Issue + +One of these is seen on the platform log store (`Alauda Container Platform Log Storage for Elasticsearch`, StatefulSet `cpaas-elasticsearch` in `cpaas-system`): + +- The day after an upgrade, `lanaya` and other containers keep restarting and the console no longer shows events. Creating an index returns `400`: + + ```text + "illegal_argument_exception","reason":"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [6446]/[5000] maximum shards open;" + ``` + +- After an upgrade, Elasticsearch still holds old indices that were not refreshed to 7.x. +- Deploying the Elasticsearch log storage plugin on a business cluster fails, and the Elasticsearch container is `OOMKilled` again and again from the first start. + +High CPU or memory with too many shards, master memory, heap OOM, and small-scale role contention are in [Elasticsearch log storage high CPU or memory: too many shards, master memory, heap OOM, or small-scale layout on ACP](Elasticsearch_log_storage_high_CPU_or_memory_too_many_shards_master_memory_heap_OOM_or_small-scale_layout_on_ACP.md). + +## Root Cause + +### Cause 1: the cluster is at its maximum number of open shards, so no new index can be created + +Elasticsearch limits the open shards of a cluster to `cluster.max_shards_per_node` (community default 1000) times the number of data nodes. At 6446 open shards against a limit of 5000, every new index is rejected, including the new day's event index, so events disappear from the console and `lanaya` restarts. Raising the cap does not improve performance; deleting historical and unused indices freed capacity, after which events were displayed again and `lanaya` returned to normal. + +The same upgrade changed the Elasticsearch workload to a StatefulSet. A resource patch (`rpch`) that had held the earlier resource limits no longer applied, so those limits had to be set again on the StatefulSet. + +### Cause 2: indices created before the upgrade remain on the older version + +Elasticsearch does not rewrite an existing index to the new version during an upgrade. The indices in question were created before the upgrade, and the log retention in use was no more than 3 days, so they were outside the retention the cluster keeps. Deleting them returned the service to normal. + +### Cause 3: the memory in the plugin form was entered with the wrong unit + +The container got far less memory than intended because the log node resources were entered as request `2` cores / `4Mi` and limit `8` cores / `16Mi`, where `Gi` was meant. The container is killed for memory from its first start. After the unit was corrected, the deployment completed. A log delay that appeared afterwards on that cluster was a separate resource bottleneck in `lanaya` and Elasticsearch; the final values there were 12 CPU / 16G for Elasticsearch and 3 CPU / 4G for `lanaya`. + +## Resolution + +Set the connection variables on a node of the cluster that runs Elasticsearch: + +```bash +ES_SVC=$(kubectl -n cpaas-system get svc cpaas-elasticsearch -o jsonpath='{.spec.clusterIP}') +ES_USER=$(kubectl -n cpaas-system get secret elasticsearch-basic-auth -o jsonpath='{.data.username}' | base64 -d) +ES_PASS=$(kubectl -n cpaas-system get secret elasticsearch-basic-auth -o jsonpath='{.data.password}' | base64 -d) +``` + +### Cause 1: the cluster is at its maximum number of open shards, so no new index can be created + +1. Read the shard count, the limit, and the oldest indices: + + ```bash + curl -s -u "$ES_USER:$ES_PASS" "http://$ES_SVC:9200/_cluster/health?filter_path=active_shards,unassigned_shards,number_of_data_nodes" + curl -s -u "$ES_USER:$ES_PASS" "http://$ES_SVC:9200/_cluster/settings?include_defaults=true&filter_path=**.max_shards_per_node" + curl -s -u "$ES_USER:$ES_PASS" "http://$ES_SVC:9200/_cat/indices?v&h=index,pri,rep,creation.date.string,store.size&s=creation.date" | head -50 + ``` + +2. To confirm that index creation is rejected, create a throwaway index. A `400` with `maximum shards open` confirms this cause; if the request succeeds, delete the test index right away. + + ```bash + curl -s -u "$ES_USER:$ES_PASS" -X PUT "http://$ES_SVC:9200/test_index_0" + curl -s -u "$ES_USER:$ES_PASS" -X DELETE "http://$ES_SVC:9200/test_index_0" + ``` + +3. Delete historical or unused indices by name. + + > **Warning:** Deleting indices removes those logs, events, or audit records permanently. Precondition: the indices are older than the retention you keep, or belong to index families nobody queries, and the owner agreed. Rollback: none without a snapshot. + + ```bash + curl -s -u "$ES_USER:$ES_PASS" -X DELETE "http://$ES_SVC:9200/," + ``` + + Delete until the open-shard count is well below the limit, then check that the new day's `event-*` index exists and `lanaya` stops restarting. Shorten the retention afterwards, or the count climbs back. + +4. If the earlier resource limits must be restored after the upgrade, set them on the StatefulSet. + + > **Warning:** Changing the StatefulSet resources rolls the Elasticsearch pods. Precondition: you saved the StatefulSet YAML and the nodes have the capacity. Rollback: `kubectl apply -f /tmp/cpaas-elasticsearch-sts.before.yaml`. + + ```bash + kubectl -n cpaas-system get sts cpaas-elasticsearch -o yaml > /tmp/cpaas-elasticsearch-sts.before.yaml + kubectl -n cpaas-system set resources sts/cpaas-elasticsearch -c cpaas-elasticsearch --limits=cpu=,memory= + ``` + + If the limits snap back after a plugin update or reconcile, set the same values in the plugin form or in the `logcenter` ModuleInfo (`spec.config.components.elasticsearch.resources`). + +### Cause 2: indices created before the upgrade remain on the older version + +List indices with the version they were created with and their creation date: + +```bash +curl -s -u "$ES_USER:$ES_PASS" "http://$ES_SVC:9200/_all/_settings/index.version.created" \ + | jq -r 'to_entries[] | "\(.key) \(.value.settings.index.version.created)"' | sort -k2 +curl -s -u "$ES_USER:$ES_PASS" "http://$ES_SVC:9200/_cat/indices?v&h=index,creation.date.string,store.size&s=creation.date" +``` + +> **Warning:** Deleting the old indices removes those logs permanently. Precondition: each name is an index created before the upgrade and its dates are outside the current retention. Rollback: none. + +```bash +curl -s -u "$ES_USER:$ES_PASS" -X DELETE "http://$ES_SVC:9200/," +``` + +### Cause 3: the memory in the plugin form was entered with the wrong unit + +Compare the memory limit on the workload with what you meant to enter: + +```bash +kubectl -n cpaas-system get pods | grep cpaas-elasticsearch +kubectl -n cpaas-system get sts cpaas-elasticsearch -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" "}{.resources.limits.memory}{"\n"}{end}' +``` + +A value in `Mi` where `Gi` was meant confirms it. Update the plugin and enter the memory with the correct unit; the deployment completes once the containers get the intended memory. If logs are delayed afterwards, check `lanaya` and Elasticsearch resources separately. + +## Diagnostic Steps + +1. Index creation fails or new daily indices are missing: compare `active_shards` with `max_shards_per_node` times `number_of_data_nodes`. At or above the limit is Cause 1. +2. Old indices after an upgrade: an `index.version.created` from the previous major version on indices outside the retention is Cause 2. +3. An Elasticsearch container that is `OOMKilled` from its first start, with a memory limit in `Mi`, is Cause 3. A container with a correct limit that runs out of heap later is the heap OOM in the high CPU or memory article linked above. diff --git a/docs/en/solutions/Inspection_reports_the_monitoring_component_abnormal_or_a_slow_collection_time_while_monitoring_works_on_ACP.md b/docs/en/solutions/Inspection_reports_the_monitoring_component_abnormal_or_a_slow_collection_time_while_monitoring_works_on_ACP.md new file mode 100644 index 000000000..9b1372087 --- /dev/null +++ b/docs/en/solutions/Inspection_reports_the_monitoring_component_abnormal_or_a_slow_collection_time_while_monitoring_works_on_ACP.md @@ -0,0 +1,115 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# Inspection reports the monitoring component abnormal or a slow collection time while monitoring works on ACP + +## Issue + +The platform inspection report marks a monitoring row as failed, but the cluster still serves metrics and the monitoring page works: + +- The basic inspection says the monitoring component or monitoring service is abnormal, while the Prometheus or VictoriaMetrics pods, the AppReleases, and the monitoring page are healthy. Prometheus returns `context deadline exceeded`, or `courier` logs a timeout while querying a metric. +- Pulling node-exporter metrics of a node times out. +- "asm-otel alert collection takes too long", or the average time to scrape `cpaas-system/cluster-enhancer-apiserver` (for example 20.28s against 10.00s) or to run monitoring queries (5.48s against 5.00s) is over the warning threshold. + +Before treating a red row as a false positive, rule out a real monitoring fault. If the API server log shows `apiserver was unable to write a JSON response: http: Handler timeout` on `GET /apis/monitoring.coreos.com/v1/prometheusrules` and an application team runs its own Prometheus in the cluster, the inspection data is genuinely wrong; see [Monitoring faults: a second Prometheus, missing vmagent-agent, vmalert OOMKilled, or Prometheus stuck in Init on ACP](Monitoring_faults_a_second_Prometheus_missing_vmagent-agent_vmalert_OOMKilled_or_Prometheus_stuck_in_Init_on_ACP.md). + +An inspection that fails with `bad_response: invalid character '<' looking for beginning of value`, with `courier` logging `Unauthorized` when it lists `clusters`, is Cause 2 of [Monitoring pages stay empty because courier-api or apollo holds a stale cluster cache on ACP](Monitoring_pages_stay_empty_because_courier-api_or_apollo_holds_a_stale_cluster_cache_on_ACP.md). + +## Root Cause + +### Cause 1: courier times out querying a monitoring target + +The inspection runs its monitoring checks through `courier`. When a query to the monitoring backend takes longer than the query timeout configured in the `monitoring` Feature, `courier` gives up with a timeout (`context deadline exceeded`), and the row says the monitoring component is abnormal although every monitoring pod and the monitoring page are fine. `ContainerStatusUnknown` pods in the same report are historical leftovers, not the failed hop. + +The item can be defined as `cpaas_cluster_prometheus_up` while the query `courier` actually runs differs: one run queried `kafka_consumergroup_lag_sum` on a business cluster that had no log storage plugin, so that series was absent, and the remaining failure was still the timeout. Raising the `monitoring` Feature timeout to 20 cleared the row. Restarting `apollo` and `courier` did not. When the row only matters as an upgrade gate, and `apollo` and the collectors log nothing, it can be recorded as a false positive; it did not block an upgrade. + +### Cause 2: node-exporter metrics take longer than the scrape timeout + +A manual request to the node-exporter metrics endpoint of the named node takes longer than the scrape limit (about 16s in one case), while the ServiceMonitor for node-exporter still has the default `scrapeTimeout` of `10s`. The node is up; the scrape is cut off. Apply this only after measuring the URL on the node the row names: another row, "average collection time 41s over 10s" for a business sample, was not fixed this way. + +### Cause 3: a collection or query time slightly above its threshold while the component is healthy + +The row names a collection or query time just above the threshold (10s for collection). For asm-otel, the pods were running without restarts, the logs had no related errors, and the time was only slightly above 10s; the warning had no impact. Rows for `cluster-enhancer-apiserver` at 20.28s and query time at 5.48s were assessed as metric request timeouts without a further cause. `cluster-enhancer-apiserver` is also the ServiceMonitor that collects `certificate_expires_time`; if that series count keeps growing, investigate it before treating a slow scrape there as harmless. + +## Resolution + +### Cause 1: courier times out querying a monitoring target + +Only when `courier` logs a timeout on a metric query, or Prometheus returns `context deadline exceeded`, and the monitoring pods and page are otherwise healthy. + +1. Save the Feature and find the timeout field: + + ```bash + kubectl get feature monitoring -o yaml > /tmp/feature-monitoring.before.yaml + kubectl get feature monitoring -o yaml | grep -n -i timeout + ``` + +2. Raise the timeout to 20 (`20s` if the field is a duration). Leave every other field as it is. + + > **Warning:** Editing `feature monitoring` changes how the console queries metrics; a wrong field can blank monitoring pages for every cluster that uses it. Precondition: the saved YAML is on disk, the timeout is confirmed in the `courier` log, and you change only the timeout that is below 20. Rollback: `kubectl apply -f /tmp/feature-monitoring.before.yaml`. + + ```bash + kubectl edit feature monitoring + ``` + +3. Re-run the inspection. If the row stays and `courier` now logs `Unauthorized` with an HTML body, follow the `bad_response` article linked in the Issue. + +### Cause 2: node-exporter metrics take longer than the scrape timeout + +1. Time the node-exporter metrics URL of the named node from a host that can reach it, and read the current timeout: + + ```bash + kubectl -n cpaas-system get servicemonitor | grep -i node + kubectl -n cpaas-system get servicemonitor -o yaml | grep -n -i -e scrapeTimeout -e interval + time curl -s -o /dev/null http://:/metrics + ``` + +2. If the measured time is above `scrapeTimeout`, raise it. + + > **Warning:** A longer scrape holds a scraper worker for up to 20s per node. Precondition: the measured time on that node is above the current timeout, and you saved the ServiceMonitor. Rollback: `kubectl apply -f /tmp/node-exporter-sm.before.yaml`. If a plugin sync restores `10s`, set the value through the monitoring plugin that owns the ServiceMonitor. + + ```bash + kubectl -n cpaas-system get servicemonitor -o yaml > /tmp/node-exporter-sm.before.yaml + kubectl -n cpaas-system edit servicemonitor + ``` + + Set `spec.endpoints[].scrapeTimeout` to `20s`; the generated Prometheus configuration then shows `scrape_timeout: 20s`. Where node scraping is defined only by a `VMNodeScrape` or `VMServiceScrape`, set that object's timeout to `20s` instead. + +### Cause 3: a collection or query time slightly above its threshold while the component is healthy + +Check the named component: + +```bash +kubectl get pods -A | grep asm-otel +kubectl -n logs --tail=200 | grep -iE 'error|timeout' +``` + +No restarts, no errors, and a time just above the threshold: close the row as no impact. A time well above the threshold on node-exporter is Cause 2. + +## Diagnostic Steps + +1. Separate a query timeout from an authorization failure: + + ```bash + kubectl -n cpaas-system logs deploy/courier --tail=200 | grep -Ei 'timeout|unauthorized|deadline|bad_response' + kubectl -n cpaas-system get pods | grep -Ei 'prometheus|vmagent|vmselect|vmstorage' + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | `context deadline exceeded` or a `courier` query timeout, monitoring pods healthy | 1 | + | `Unauthorized` on `clusters` plus `bad_response: invalid character '<'` | the `bad_response` article linked in the Issue | + | The named node's metrics URL answers slower than `scrapeTimeout` | 2 | + | A collection time just over the threshold on a healthy component | 3 | + +3. Both the monitoring pipeline row and the log pipeline row abnormal right after a primary/standby switchover, with healthy pods, is a different false positive: see [Inspection reports zero free inodes, missing event data, expired dex.tls, or abnormal pipelines after switchover on ACP](Inspection_reports_zero_free_inodes_missing_event_data_expired_dex.tls_or_abnormal_pipelines_after_switchover_on_ACP.md). diff --git a/docs/en/solutions/Inspection_reports_zero_free_inodes_missing_event_data_expired_dex.tls_or_abnormal_pipelines_after_switchover_on_ACP.md b/docs/en/solutions/Inspection_reports_zero_free_inodes_missing_event_data_expired_dex.tls_or_abnormal_pipelines_after_switchover_on_ACP.md new file mode 100644 index 000000000..6e5d2ad1f --- /dev/null +++ b/docs/en/solutions/Inspection_reports_zero_free_inodes_missing_event_data_expired_dex.tls_or_abnormal_pipelines_after_switchover_on_ACP.md @@ -0,0 +1,136 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# Inspection reports zero free inodes, missing event data, expired dex.tls, or abnormal pipelines after switchover on ACP + +## Issue + +The platform inspection report marks a node, the log pipeline, a certificate, or both pipeline rows as failed, but the cluster still serves logs, metrics, and logins. Typical report text: + +- `节点内剩余inode数为0个,低于预警阈值(5000个)` (remaining inodes on the node are 0, below the warning threshold of 5000), or the installation self-test warns about inodes while the nodes show 99% free inodes. +- A node has not sent `event` data (often `节点未发送 event 类型数据`, node has not sent event-type data), or `audit` data, for 15 minutes. Or the `日志链路组件状态` (log pipeline component status) row names `razor` with `日志未发送` (logs not sent), and the nodes it names are missing from the node list of the application log query page. +- `dex.tls` is reported expired, or another certificate has fewer than 30 days left or is named as `packageserver-service-cert` or `redis-operator-service-cert`, and a direct read of the Secret does not match. +- Inserting `audits` data into Elasticsearch is reported slow on a cluster that does not run `lanaya`. +- After a primary/standby switchover of the global cluster, both the monitoring pipeline row and the log pipeline row are abnormal (`监控组件(global/alert)健康状态不正常` and `监控组件(global/monitor)健康状态不正常`, monitoring component health abnormal), and the platform health page shows `平台监控功能故障,可能会影响集群 (global) 的功能健康检查结果的准确性,请尽快修复。` (platform monitoring is faulty, which may affect the accuracy of the health check results of cluster global), while the AppReleases and pods are all normal. + +Monitoring rows caused by query timeouts or slow scrapes are in [Inspection reports the monitoring component abnormal or a slow collection time while monitoring works on ACP](Inspection_reports_the_monitoring_component_abnormal_or_a_slow_collection_time_while_monitoring_works_on_ACP.md). + +## Root Cause + +### Cause 1: the inode count is 0 because /boot/efi (or another file system without inodes) is counted + +`df -i /boot/efi` is the empty pool, while `df -i /` and `df -i /var` have free inodes. The row is computed by the indicator `node.disk.inode.free` in the ConfigMap `indicators-standard-node` in `cpaas-system`, which reads `node_filesystem_files_free` with `device!~"rootfs|tmpfs"` and no mount-point filter, so the 0 of `/boot/efi` becomes the node's value. This is a known defect, and it does not mean the node cannot create files. The installation self-test has the same shape: it scans every device without excluding file system types, so file systems that have no inode counts report 0 remaining and fall below 5000. A later version is expected to exclude them; until then the warning can be ignored or the indicator corrected. A root or `/var` file system whose `df -i` is really exhausted is a real problem and not this cause. + +### Cause 2: the node produced no event, audit, or log data for 15 minutes + +The rule fires when no data of that type was sent from the node for 15 consecutive minutes. It means nothing of that type was produced on the node during the window, not that collection failed, and the same line can appear on every run. A "logs not sent" row naming `razor` came from nodes where system logs were visible and application logs were not, because nothing on those nodes wrote application logs; a node appears in the application log node list only after it has produced application logs. The log pipeline item checks the health of `nevermore`, `kafka`, and `lanaya`; a delay in those components is the other thing to rule out. + +### Cause 3: dex.tls is reported expired after a virtual machine restart while its dates are valid + +`openssl` on `cpaas-system/dex.tls` shows a validity period that still contains the current time. The report started after the virtual machine hosting the platform was restarted, earlier inspections had not flagged it, and later inspections returned to normal without a certificate replacement. If the dates really are in the past, this cause does not apply. + +### Cause 4: an audits-to-Elasticsearch latency row on a cluster without lanaya + +The cluster has no `lanaya` workload, so the row has nothing real to measure; it is a false alarm. Where the monitoring page or log search was really broken at the same time, the cause was elsewhere (for example a VictoriaMetrics OOM or Kafka lag). + +### Cause 5: a certificate row does not match the Secret it names + +The row says a certificate expires within 30 days, or names `packageserver-service-cert`, `redis-operator-service-cert`, or a short-lived `ca.crt`, while the Secret shows a much longer lifetime. Some certificates in these rows are renewed automatically by the platform 15 days before expiry, so the row needs no action. Replacing `dex.tls` or deleting a Certificate is wrong when the file is not the object the row measured. + +### Cause 6: warlock was not restarted after a primary/standby switchover + +The platform health page lists `vmselect-cluster` as `故障` (faulty) with 3/3 replicas and a reason such as `Service health metric is false (Error: health metric with zero value: cpaas_module_health{container="warlock", ... module="monitor", namespace="cpaas-system", ... service="warlock", ...} => 0 ...)`, while `courier` and `courier-api` are healthy. The `cpaas_module_health` value exported by `warlock` is stuck at 0 because `warlock` was not restarted after the switchover. In the switchover procedure this restart follows the restart of VictoriaMetrics and `cluster-transformer` on both global clusters, and it applies to every business cluster that sends monitoring data to the global VictoriaMetrics. + +## Resolution + +### Cause 1: the inode count is 0 because /boot/efi (or another file system without inodes) is counted + +Only when `df -i /` has free inodes and `/boot/efi` (or another file system with no inode count) is the 0. Edit the indicator on every cluster that shows the row. + +```bash +df -i / /var /boot/efi +kubectl -n cpaas-system get configmap indicators-standard-node -o yaml | grep -n -A3 'node.disk.inode.free' +``` + +> **Warning:** Excluding the wrong mount hides a real inode alert. Precondition: `df -i /` shows free inodes, and you saved the ConfigMap. Rollback: `kubectl apply -f /tmp/inode-configmap.before.yaml`. Exclude only `/boot/efi`; do not disable the whole inode item. + +```bash +kubectl -n cpaas-system get configmap indicators-standard-node -o yaml > /tmp/inode-configmap.before.yaml +kubectl -n cpaas-system edit configmap indicators-standard-node +``` + +In `node.disk.inode.free`, append `mountpoint!~\"/boot/efi\"` to the label set of `node_filesystem_files_free` when the expression has no `mountpoint` matcher. If it already has a positive `mountpoint=~` list containing `/boot/efi`, remove `/boot/efi` from that list instead. After the edit the query reads (quotes escaped as inside the ConfigMap; keep the other labels, such as the `vmcluster` value, as they are): + +```text +node_filesystem_files_free{vmcluster=\"region\",node_name=~\"{{.name}}\",device!~\"rootfs|tmpfs\",mountpoint!~\"/boot/efi\"} +``` + +The collector picks the change up in about 15 minutes. To apply it sooner, restart `courier`: + +> **Warning:** Restarting `courier` drops in-flight inspection queries until the new pod is Ready. Precondition: the ConfigMap edit is saved. Rollback: none needed; the Deployment recreates the pod. + +```bash +kubectl -n cpaas-system rollout restart deploy/courier +``` + +If the only goal is to stop the warning, the threshold can be changed in the inspection configuration; a lower threshold also hides a real exhaustion. + +### Cause 2: the node produced no event, audit, or log data for 15 minutes + +No object has to change. Treat the line as an ignorable prompt when the node is `Ready` and the 15-minute gap is the only finding. To stop it, disable the `日志链路组件状态` (log pipeline component status) item in the inspection configuration (`运维中心 → 巡检 → 巡检配置`, Operations Center → Inspection → Inspection Configuration) and watch `nevermore`, `kafka`, and `lanaya` through real-time alerts instead. Keep the item enabled if you want a real log pipeline failure to show up in the inspection; it covers more than the event sentence. + +### Cause 3: dex.tls is reported expired after a virtual machine restart while its dates are valid + +No replacement and no delete. After confirming the dates (Diagnostic Steps), wait for the next inspection. If the dates are past, follow [Replace the platform access certificate in dex.tls on ACP](Replace_the_platform_access_certificate_in_dex.tls_on_ACP.md) instead of waiting. + +### Cause 4: an audits-to-Elasticsearch latency row on a cluster without lanaya + +No change. Confirm that `lanaya` is absent and close the row as a false alarm. + +### Cause 5: a certificate row does not match the Secret it names + +Read the named Secret first (Diagnostic Steps). When it is still well inside its dates, the row does not match the file. For a certificate the platform renews automatically 15 days before expiry, check after that point that the dates moved forward. Renew only when `openssl` shows that named certificate inside the warning window, and only for that object: see [Platform certificate expiry alert for a cert-manager serving certificate that renews by itself on ACP](Platform_certificate_expiry_alert_for_a_cert_manager_serving_certificate_that_renews_by_itself_on_ACP.md), [Operator service certificates such as packageserver-service-cert about to expire or not renewed on ACP](Operator_service_certificates_such_as_packageserver-service-cert_about_to_expire_or_not_renewed_on_ACP.md), or [Expired platform component certificate Secret such as hco-webhook-service-cert not reissued on ACP](Expired_platform_component_certificate_Secret_such_as_hco-webhook-service-cert_not_reissued_on_ACP.md). + +### Cause 6: warlock was not restarted after a primary/standby switchover + +Only when a switchover happened, both pipeline rows went abnormal after it, and the pods and AppReleases are healthy. + +> **Warning:** Restarting `warlock` interrupts what it serves until the new pod is Ready. Precondition: the switchover is complete and VictoriaMetrics (`alertmanager`, `vmselect`, `vminsert`) and `cluster-transformer` were already restarted on the primary and standby global clusters. Rollback: none needed; the Deployment recreates the pod. + +Run on each affected cluster, including every business cluster that sends monitoring data to the global VictoriaMetrics: + +```bash +kubectl -n cpaas-system get pods -l service_name=warlock +kubectl delete po -n cpaas-system -l service_name=warlock +kubectl -n cpaas-system get pods -l service_name=warlock +``` + +If the earlier switchover steps were skipped, first run `kubectl delete po -n cpaas-system -l 'service_name in (alertmanager,vmselect,vminsert)'` and `kubectl delete po -n cpaas-system -l service_name=cluster-transformer` on both global clusters. Re-run the inspection, and make the `warlock` restart part of the switchover procedure. + +## Diagnostic Steps + +1. Inode row: compare mounts before editing anything. `/boot/efi` at 0 with `/` healthy is Cause 1. To list every mount that reports no inodes at all: `df -i -T | awk 'NR==1 || $3==0'`. `/` or `/var` at 0 is a real exhaustion, not this article. +2. Event or logs-not-sent row: "15 minutes" and the data type, repeating on every run, is Cause 2. On the log query page, switch to system logs for the named node; system logs present and no application pods on the node confirms it. If system logs are also missing, check the pipeline: + + ```bash + kubectl get pods -A -o wide | grep -E 'nevermore|lanaya|razor|kafka' | grep -v Running + ``` + +3. Certificate rows: + + ```bash + kubectl -n cpaas-system get secret dex.tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates -subject + kubectl -n get secret -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates + ``` + + Valid `dex.tls` dates after a VM restart is Cause 3. For a key named `ca.crt`, decode `.data['ca.crt']` the same way. A long lifetime against a "less than 30 days" row is Cause 5. A date that has passed is a real expiry. +4. Audits latency: `kubectl get pods -A | grep lanaya` returning nothing is Cause 4. +5. Both pipeline rows abnormal right after a switchover, `kubectl get ars -A` and the pods healthy, and the health page reason `health metric with zero value: cpaas_module_health{container="warlock", ...} => 0`, is Cause 6. diff --git a/docs/en/solutions/Kafka_2.x_instance_has_a_broker_offline_because_the_Alauda_Build_of_Kafka_2_operator_is_not_installed_on_ACP.md b/docs/en/solutions/Kafka_2.x_instance_has_a_broker_offline_because_the_Alauda_Build_of_Kafka_2_operator_is_not_installed_on_ACP.md new file mode 100644 index 000000000..50c607d5e --- /dev/null +++ b/docs/en/solutions/Kafka_2.x_instance_has_a_broker_offline_because_the_Alauda_Build_of_Kafka_2_operator_is_not_installed_on_ACP.md @@ -0,0 +1,59 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Application Services +ProductsVersion: [] +--- + +# Kafka 2.x instance has a broker offline because the Alauda Build of Kafka 2 operator is not installed on ACP + +## Issue + +One broker of a Data Services Kafka instance is reported offline ("a Kafka broker node is not online"). The instance runs Kafka 2.x (for example 2.7). The other brokers may still be running. + +This is an application-facing Kafka instance, not the platform log Kafka (`cpaas-kafka-*` in `cpaas-system`). For the platform log Kafka, see [Platform log Kafka brokers fail: no leader election, exit 137, Pending on DiskPressure, Ceph write errors, or OOM on ACP](Platform_log_Kafka_brokers_fail_no_leader_election_exit_137_Pending_on_DiskPressure_Ceph_write_errors_or_OOM_on_ACP.md). + +## Root Cause + +A Kafka 2.x instance is managed by the **Alauda Build of Kafka 2** operator. When that operator is not installed on the cluster, nothing reconciles the instance, so a broker that goes down is not brought back. + +Operator version `v4.0.3` of Alauda Build of Kafka 2 supports platform versions `v4.0` and `v4.1`, and Kafka `2.5`, `2.7`, and `2.8`. + +## Resolution + +1. Confirm the Kafka version of the instance and that the operator is missing (see Diagnostic Steps). +2. In the application marketplace (OperatorHub), check the Alauda Build of Kafka 2 operator versions and pick the one whose supported platform and Kafka versions match this cluster and this instance. +3. If the operator package is not yet on the platform, push it with the `violet` tool version that matches the operator package, not the `violet` version shipped with the platform. +4. Install the Alauda Build of Kafka 2 operator and wait until it reports success. +5. Restart the Kafka instance so the operator takes it over and brings the offline broker back. + + > **Warning:** Restarting the Kafka instance interrupts producers and consumers of that instance while its brokers restart. Precondition: the operator is installed and healthy, and the instance's applications can tolerate the interruption (do it in a maintenance window). Rollback: none needed for the restart itself; the operator recreates the broker pods. + + ```bash + kubectl -n get pods -o wide | grep + kubectl -n delete pod + ``` + + Restart the brokers one at a time and wait for each to be `Running` and ready before the next. + +6. Check that every broker of the instance is online. + +## Diagnostic Steps + +1. Find the instance, its namespace, and the broker that is offline: + + ```bash + kubectl get pods -A -o wide | grep -iE 'kafka|zookeeper' + kubectl -n logs -p --tail=50 + ``` + +2. Check whether the Kafka 2 operator is installed (no matching ClusterServiceVersion, or no operator pod, means it is missing): + + ```bash + kubectl get csv -A | grep -i kafka + kubectl get pods -A | grep -i kafka | grep -i operator + ``` + +3. Read the Kafka version of the instance from its custom resource or broker image, and compare it with the Kafka versions the available operator version supports. +4. If the operator is installed and healthy and a broker is still offline, the cause is elsewhere: check the broker pod events, its node, and its volume (for example a node that is `NotReady` or a volume that is full). diff --git a/docs/en/solutions/Log_archive_output_platform_logs_with_workload_logs_flat_S3_objects_file_retention_or_undeletable_.nfs_files_on_ACP.md b/docs/en/solutions/Log_archive_output_platform_logs_with_workload_logs_flat_S3_objects_file_retention_or_undeletable_.nfs_files_on_ACP.md new file mode 100644 index 000000000..78d20c683 --- /dev/null +++ b/docs/en/solutions/Log_archive_output_platform_logs_with_workload_logs_flat_S3_objects_file_retention_or_undeletable_.nfs_files_on_ACP.md @@ -0,0 +1,157 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Log archive output: platform logs with workload logs, flat S3 objects, file retention, or undeletable .nfs files on ACP + +## Issue + +The log archive (log sync to external NFS or S3) is running, and its output is not what the owner expected: + +- Only application (workload) logs are enabled, but system component logs from `cpaas-system` and `kube-system` are archived as well. +- After switching the archive from NFS to S3, the bucket has no directory hierarchy; only the log files appear. +- Business logs must be kept for 3 years, but the archive holds only about half a year; the oldest daily directory is about 180 days old. +- `log-exporter-config` says `path: /cpaas/data/logarchive`, but `ls /cpaas/data/logarchive` on the host says `No such file or directory`. +- Does the hostPath archive directory grow without limit? +- Are audit logs archived? +- Deleting old archived logs fails, and files whose names start with `.nfs` appear in the archive directory; their content looks like the consumed Kafka records. + +Deploying the archive (the `/cpaas/data/logarchive` mount, the Kafka address, real-time copy versus move after N days) is in [Archive Elasticsearch logs to NFS: logarchive mount, log center Kafka address, and archive limits on ACP](Archive_Elasticsearch_logs_to_NFS_logarchive_mount_log_center_Kafka_address_and_archive_limits_on_ACP.md). + +The documented archive ("How to archive logs to third-party storage") consists of three objects in `cpaas-system`: + +- Secret `log-export-s3-secret` with `access_key_id` and `secret_access_key` (S3 output only). +- ConfigMap `log-exporter-config` with the file `export.yml`: `scopes` (`system`, `workload`, `kubernetes`, `platform`, each `true` or `false`) and `output` (`type` `local` or `s3`, `path: /cpaas/data/logarchive`, `layout: TimePrefixed`, `max_size`, `compression` such as `gzip`, `file_type: txt`, and for S3 the endpoint, bucket, `s3forcepathstyle`, and `insecure`). +- Deployment `log-exporter` (also seen as `log-export`): an alpine init container that runs `chown`, and a `razor` container that runs `/razor consumer ... --database-type=file --export-config=/etc/log-export/export.yml` against the log center Kafka. + +## Root Cause + +### Cause 1: on Elasticsearch, the Workload scope also exports platform logs, and the Platform scope has no effect + +The archiver selects each log item by its index prefix: `log-system` follows `scopes.system`, `log-kubernetes` follows `scopes.kubernetes`, `log-platform` follows `scopes.platform`, and every other value falls to the default branch, which follows `scopes.workload`. With Elasticsearch log storage, the index prefix is empty for both application (workload) and product (platform) logs, so both fall to the default branch: with Workload on, platform logs are archived too; with Workload off, neither is; the Platform switch changes nothing. System and Kubernetes logs are not affected. No setting separates them. + +### Cause 2: S3 objects have no directory hierarchy + +This is expected. The date-time / project / cluster directories exist only in the temporary local path; the uploaded objects are the log files themselves. + +### Cause 3: archive files older than 180 days are deleted by a cron job on the host, not by the platform + +The platform does not limit how long archive files are kept. A scheduled task on the host that mounts the archive volume ran a cleanup script that took a target directory (`TARGET_DIR="${1:-.}"`) and a day threshold (`DAYS_THRESHOLD="${2:-180}"`), listed subdirectories with `find "$TARGET_DIR" -maxdepth 1 -type d`, compared their modification time (`stat -c %Y`), and ran `rm -rf` on those older than the threshold. With the default of 180, about half a year of daily directories remained. + +### Cause 4: the output path in log-exporter-config is a path inside the pod + +`output.path: /cpaas/data/logarchive` is where the archive container writes. What backs it is the volume mounted at that path in the Deployment. When that volume is a PVC on an NFS StorageClass, the files are on the NFS share of that PV, not in a host directory of that name. Only in the hostPath layout is the same path also the host directory. + +### Cause 5: the hostPath archive directory is not cleaned by the platform + +Nothing in the archive caps or cleans the hostPath directory; it grows until something removes old files. A scheduled cleanup on the node is the way to bound it. + +### Cause 6: audit logs are not an archive scope + +The archive covers log data only: system, application (workload), Kubernetes, and product (platform) logs. Audit data is not archived. For a 6-month log backup requirement, the archive is the recommended path rather than a 180-day retention in the log Elasticsearch, which uses a large amount of storage and affects Elasticsearch performance. + +### Cause 7: archived files cannot be deleted, and .nfs files appear in the archive directory + +The log archive does not create `.nfs` files itself. They are the NFS client's protection for open files: when a file that a process still holds open is deleted on an NFS mount, the client renames it to `.nfs` instead of removing it, and removes it when the last process closes it. Archived files that cannot be deleted, with `.nfs` files left in their place, come from a known defect in the log archiving solution that keeps files open; it is fixed in versions after 3.18.3. + +## Resolution + +### Cause 1: on Elasticsearch, the Workload scope also exports platform logs, and the Platform scope has no effect + +Plan the archive storage for workload plus platform logs when Workload is on, and do not rely on the Platform switch. Check the live scopes: + +```bash +kubectl -n cpaas-system get configmap log-exporter-config -o jsonpath='{.data.export\.yml}' +``` + +### Cause 2: S3 objects have no directory hierarchy + +No change. Find files by object name and time in the bucket. + +### Cause 3: archive files older than 180 days are deleted by a cron job on the host, not by the platform + +Find the scheduled task on the node that mounts the archive volume and read its threshold: + +```bash +crontab -l +grep -r -l -iE 'rm -rf|find ' /etc/cron.d/ 2>/dev/null +cat +``` + +> **Warning:** The script runs `rm -rf` on archive directories; a wrong directory argument or a threshold in the wrong unit deletes archive data permanently. Precondition: you saved the current crontab, the first argument is the archive directory, and the new threshold is in days (about 1095 for 3 years). Rollback: restore the saved crontab line; deleted directories do not return. + +```bash +crontab -l > /tmp/crontab.before +crontab -e +``` + +Change the day threshold argument of the cleanup line to the number of days that must be kept, and size the share for that period (a 1.7T share was 54% used with about 180 days on it). + +### Cause 4: the output path in log-exporter-config is a path inside the pod + +Read the volume mounted at that path, then where that volume lives: + +```bash +kubectl -n cpaas-system get deploy | grep -E 'log-export' +kubectl -n cpaas-system get deploy -o jsonpath='{range .spec.template.spec.volumes[*]}{.name}{" pvc="}{.persistentVolumeClaim.claimName}{" hostPath="}{.hostPath.path}{"\n"}{end}' +kubectl -n cpaas-system get pvc -o jsonpath='{.spec.volumeName}{"\n"}' +df -h | grep +``` + +For a PVC the files are on the PV's share; for hostPath they are on the node that runs the pod. + +### Cause 5: the hostPath archive directory is not cleaned by the platform + +Add a scheduled cleanup on each node that runs the archive pod. + +> **Warning:** The cleanup deletes archive files permanently. Precondition: `` is the hostPath archive directory (check it with the Cause 4 commands), `` is at least the period that must be kept, and you ran the `find` without `-delete` first. Rollback: remove the crontab line; deleted files do not return. + +```bash +find -type f -mtime + | head +( crontab -l; echo "0 2 * * * find -type f -mtime + -delete" ) | crontab - +``` + +### Cause 6: audit logs are not an archive scope + +Audit records are not archived. Keep the needed audit days in the log storage retention (read the maximum on the plugin form) and size the disk for it. For logs, deploy the archive with the scopes that are required. + +### Cause 7: archived files cannot be deleted, and .nfs files appear in the archive directory + +1. Confirm which process holds the files (on the node that mounts the share): + + ```bash + ls -la /.nfs* | head + lsof +D 2>/dev/null | head + kubectl -n cpaas-system get deploy,pods | grep -E 'log-export' + ``` + +2. Do not delete the `.nfs` files by hand; the NFS client removes each one when the process that holds it closes it. +3. Upgrade the log archiving solution to a version after 3.18.3, which fixes the defect. + +## Diagnostic Steps + +1. Read the archive configuration and workload: + + ```bash + kubectl -n cpaas-system get configmap log-exporter-config -o yaml + kubectl -n cpaas-system get deploy,pods | grep -E 'log-export' + kubectl -n cpaas-system logs deploy/ --tail=100 + ``` + +2. Store type: `kubectl get pods -A | grep -iE 'cpaas-elasticsearch|clickhouse'`. Platform logs in the archive with only Workload on, on Elasticsearch, is Cause 1. +3. Archive directories disappearing at a fixed age that matches a host cron script threshold is Cause 3: + + ```bash + ls | head + ``` + +4. `No such file or directory` on the host for the ConfigMap path is Cause 4. +5. `.nfs` files next to archived files that cannot be deleted, on an NFS-backed archive, is Cause 7. diff --git a/docs/en/solutions/Platform_log_Kafka_brokers_fail_no_leader_election_exit_137_Pending_on_DiskPressure_Ceph_write_errors_or_OOM_on_ACP.md b/docs/en/solutions/Platform_log_Kafka_brokers_fail_no_leader_election_exit_137_Pending_on_DiskPressure_Ceph_write_errors_or_OOM_on_ACP.md new file mode 100644 index 000000000..acb548515 --- /dev/null +++ b/docs/en/solutions/Platform_log_Kafka_brokers_fail_no_leader_election_exit_137_Pending_on_DiskPressure_Ceph_write_errors_or_OOM_on_ACP.md @@ -0,0 +1,231 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.14.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Platform log Kafka brokers fail: no leader election, exit 137, Pending on DiskPressure, Ceph write errors, or OOM on ACP + +## Issue + +The platform log Kafka (`cpaas-kafka-*` with `cpaas-zookeeper-*` in `cpaas-system`) fails as a broker, not as a slow consumer. Typical reports: + +- `lanaya` cannot connect to Kafka, and the broker log says a leader election failed. +- `cpaas-kafka-1` and `cpaas-kafka-2` are in `Error` with tens of thousands of restarts and exit code `137`; adding replicas does not help. +- A `cpaas-kafka` pod is `Pending` and log delivery has stopped. +- Observability shows no logs; the logging pods start healthy and later go abnormal, and the Kafka log shows data write failures on a Ceph-backed volume. +- The broker log shows `java.lang.OutOfMemoryError: Java heap space` or `Direct buffer memory`, logs pile up, and `razor` logs `Send kafka msg error` with `context deadline exceeded` or `kafka write errors`. + +This article is only for the platform log pipeline. A Data Services Kafka instance used by applications is a different workload; do not apply the wipe, `Pending`, or metadata branches here to it. + +Related articles: [cpaas-kafka broker fails to start: corrupt segment, no space, broker id conflict, or deleted StatefulSet on ACP](cpaas-kafka_broker_fails_to_start_corrupt_segment_no_space_broker_id_conflict_or_deleted_StatefulSet_on_ACP.md), [Kafka and ZooKeeper down from storage stalls, a full node disk, or a probe kill during log replay on ACP](Kafka_and_ZooKeeper_down_from_storage_stalls_a_full_node_disk_or_a_probe_kill_during_log_replay_on_ACP.md), [Logs, audit, and events stop while the log Kafka is degraded on ACP](Logs_audit_and_events_stop_while_the_log_Kafka_is_degraded_on_ACP.md), and [cpaas-kafka retention hours, memory limit above 80 percent, or KAFKA_HEAP_OPTS not applied on ACP](cpaas-kafka_retention_hours_memory_limit_above_80_percent_or_KAFKA_HEAP_OPTS_not_applied_on_ACP.md). + +## Root Cause + +### Cause 1: the brokers cannot elect a leader because the Kafka metadata is corrupt + +The brokers are up enough to log, and the log says a leader election failed because metadata is corrupt. A normal restart does not elect a leader. Clearing the Kafka data directory restores elections. Lag and a large on-disk queue of the `nevermore` log agent under `/cpaas` (the `audit`, `event`, and `log` trees and their `diskqueue` directories) can remain afterwards; clearing those trees on every node and restarting `nevermore` drains it, at the cost of every record that had not been shipped yet. + +### Cause 2: cpaas-kafka pods restart with exit code 137 and extra replicas still crash + +Exit code `137` means the container received `SIGKILL`: either the kernel OOM killer (`OOMKilled` in the container's last state) or the kubelet after a failed liveness probe (a `Liveness probe failed` event). A failing or full disk does not produce 137 directly, but it can slow startup until the probe kills the container. When the loop persists after disk and memory are checked and scaling out has failed, the recovery that works is a full rebuild of the log cluster: stop the writers and the stores, clear the Elasticsearch, Kafka, and ZooKeeper data directories, and start the stores in order. Every stored log and the ZooKeeper state of the log cluster are lost. + +A broker that is killed by its probe while it is still loading segments is the narrower replay case in the Kafka and ZooKeeper article linked above; that recovery does not delete Elasticsearch data. + +### Cause 3: the Kafka pod is Pending because its node has DiskPressure + +Disk usage crossed the kubelet eviction threshold, the Kafka pod was evicted, and it stays `Pending`, so log delivery stops. Expanding or freeing the disk and then deleting the pod lets it schedule again. The same `Pending` state clears after `NotReady` nodes recover; restarting pods does not fix a `NotReady` node. + +### Cause 4: Kafka data writes fail on a Ceph-backed volume + +Kafka logs from before a restart show data write failures, and the Kafka PVC is on Ceph. Expanding that Kafka volume brings Kafka back, and log display returns (one case: ACP 4.2.1 on x86 with MicroOS nodes). + +### Cause 5: cpaas-kafka runs out of JVM heap or direct buffer memory + +```text +java.lang.OutOfMemoryError: Java heap space +ERROR kafka.network.Processor - Processor got uncaught exception. +java.lang.OutOfMemoryError: Direct buffer memory +``` + +`ReplicaFetcherThread` also logs `Error sending fetch request ... java.lang.OutOfMemoryError: Java heap space`. Log volume grew beyond what the Kafka configuration can carry, the broker JVM ran out of memory, and Kafka stopped processing log data. `nevermore` builds up a local backlog meanwhile; when Kafka recovers it sends the backlog in a burst, and `apollo` on the global cluster can hit OOM and restart, which partly affects the console. + +Raising only the container resources does not stop the OOM; `lanaya` can then also go OOM. Raising the broker heap (`KAFKA_HEAP_OPTS`) stops it. Check the running JVM afterwards: on some images the start script sets its own heap, and the environment value never reaches the broker (see the retention and heap article linked above). + +## Resolution + +Some of the workloads below are Deployments on one version and StatefulSets on another. If `kubectl scale` rejects the kind, use the kind printed by `kubectl -n cpaas-system get deploy,sts`. + +### Cause 1: the brokers cannot elect a leader because the Kafka metadata is corrupt + +1. Stop the writer and the brokers: + + ```bash + kubectl -n cpaas-system scale deploy/lanaya --replicas=0 + kubectl -n cpaas-system scale sts/cpaas-kafka --replicas=0 + ``` + +2. Clear the Kafka data directory on each broker node. + + > **Warning:** Clearing the Kafka data directory destroys every record on that broker; consumers lose the backlog. Precondition: only when the broker log shows a failed leader election or corrupt metadata, a normal restart did not elect a leader, both workloads above are at 0, `lsof` shows no Kafka process in the directory, and the path is the Kafka volume (often `/cpaas/data/kafka0`), not ZooKeeper. Rollback: none. + + Delete the contents, not the mount directory itself: + + ```bash + rm -rf /* + ``` + +3. Scale `cpaas-kafka` back to its previous replica count, wait until a leader is elected, then scale `lanaya` back. +4. Only if lag remains and the `nevermore` queue trees under `/cpaas` are large, clear them and restart `nevermore`. + + > **Warning:** Clearing `/cpaas/audit`, `/cpaas/event`, and `/cpaas/log` drops every record that never reached Kafka, and also removes the agent's own logs under `/cpaas/log`. Precondition: Kafka elects leaders again, and `du` shows these agent trees, not the Elasticsearch data directory. Rollback: none; `nevermore` recreates empty queues. + + ```bash + du -sh /cpaas/audit /cpaas/event /cpaas/log + rm -rf /cpaas/audit/* /cpaas/event/* /cpaas/log/* + kubectl -n cpaas-system rollout restart ds/nevermore + ``` + +### Cause 2: cpaas-kafka pods restart with exit code 137 and extra replicas still crash + +1. Find who sends the kill before you delete anything: + + ```bash + kubectl -n cpaas-system describe pod cpaas-kafka-1 | grep -E -A3 'Last State|Liveness probe failed' + df -h /cpaas /cpaas/data + dmesg -T | grep -iE 'oom|i/o error' | tail + ``` + +2. Record the replica counts: + + ```bash + kubectl -n cpaas-system get deploy,sts | grep -E 'lanaya|devops-controller|courier|razor|cpaas-kibana|cpaas-elasticsearch|cpaas-kafka|cpaas-zookeeper' + ``` + +3. Stop the writers and the stores (skip names that do not exist, such as `cpaas-elasticsearch-master` on a small-scale layout). + + > **Warning:** This stops log collection, log search, Kibana, and the inspection queries served by `courier` for the whole cluster, and the next step deletes every stored log, every unconsumed Kafka record, and the ZooKeeper state of the log cluster. Precondition: exit 137 persists after the disk and memory checks, scaling out already failed, and the owner accepts a full rebuild of the log cluster. Rollback: none for the data. + + ```bash + kubectl -n cpaas-system scale deploy/lanaya deploy/devops-controller deploy/courier deploy/razor deploy/cpaas-kibana --replicas=0 + kubectl -n cpaas-system scale sts/cpaas-elasticsearch sts/cpaas-elasticsearch-master sts/cpaas-kafka sts/cpaas-zookeeper --replicas=0 + ``` + +4. When every pod is gone, on each data node record owner and mode, then clear the contents of each data directory (take the paths from the pods' volume mounts; never a directory that also holds etcd or application data): + + ```bash + stat -c '%U:%G %a %n' + rm -rf /* /* /* + ``` + + Recreate any directory you removed with the same owner and mode. + +5. Start back to the recorded counts, waiting for each to be Ready: ZooKeeper, Kafka, Elasticsearch master, Elasticsearch data, then `cpaas-kibana`, `razor`, `courier`, `lanaya`, and `devops-controller`. The order of the stateless components after the stores has not been shown to matter. +6. Query a pod that is logging now; older records were deleted on purpose. + +### Cause 3: the Kafka pod is Pending because its node has DiskPressure + +```bash +kubectl describe node | grep -E 'DiskPressure|MemoryPressure|Ready' +kubectl -n cpaas-system describe pod cpaas-kafka-0 | sed -n '/Events:/,$p' +``` + +Free or extend the filesystem that `df` shows full. When `DiskPressure` is `False` and the node is `Ready`: + +> **Warning:** Deleting the Kafka pod interrupts produce and consume until the StatefulSet recreates it. Precondition: the node condition that kept it `Pending` is gone. Rollback: none needed; the StatefulSet recreates the pod. + +```bash +kubectl -n cpaas-system delete pod cpaas-kafka-0 +``` + +Delete Elasticsearch pods the same way only if they are also stuck on that condition. Do not clear data directories for a `Pending` pod. If the node is `NotReady`, repair the node; the pods become Ready once it is. + +### Cause 4: Kafka data writes fail on a Ceph-backed volume + +```bash +kubectl -n cpaas-system get pvc | grep -i kafka +kubectl -n cpaas-system logs cpaas-kafka-0 --tail=80 +kubectl -n cpaas-system exec cpaas-kafka-0 -- df -h +``` + +A hostPath or node partition such as `/cpaas/data/kafka0` is not this procedure; grow that filesystem on the node instead. + +> **Warning:** A PVC expansion generally cannot be shrunk afterwards; topic data is not deleted by it. Precondition: only when the broker log shows write failures and the Kafka PVC is Ceph-backed. Pick `` larger than the current capacity. Rollback: none (no shrink). If the StorageClass does not allow expansion, expand the volume from the storage side, and restart the broker pod only if it does not see the new space. + +```bash +kubectl -n cpaas-system patch pvc --type merge \ + -p '{"spec":{"resources":{"requests":{"storage":""}}}}' +``` + +When the PVC and `cpaas-kafka` pods are healthy, query a pod that is logging now. If the `logcenter` ModuleInfo declares a Kafka storage size (`spec.config.components.kafka.storageSize`), set it to the same value. + +### Cause 5: cpaas-kafka runs out of JVM heap or direct buffer memory + +1. Confirm the error and read the current settings: + + ```bash + kubectl -n cpaas-system logs cpaas-kafka-0 --tail=200 | grep -E 'OutOfMemoryError|Direct buffer memory' + kubectl -n cpaas-system get sts cpaas-kafka -o yaml | grep -E -A2 'KAFKA_HEAP_OPTS|resources:' + kubectl -n cpaas-system exec cpaas-kafka-0 -- ps -ef | grep '[j]ava' | grep -o -- '-Xm[sx][^ ]*' + ``` + +2. Raise the Kafka resources through the platform chart override on the global cluster, so a reconcile does not revert them. The platform applies the change in about 5 to 10 minutes. + + > **Warning:** Editing `prdb base` changes platform-level chart overrides on the global cluster, and the new Kafka resources roll the `cpaas-kafka` pods; produce and consume pause on each broker while it restarts. Precondition: only when the broker log shows the `OutOfMemoryError` lines above, you saved the object, and the nodes have the memory for the new limit. Rollback: `kubectl apply -f /tmp/prdb-base.before.yaml`. + + ```bash + kubectl get prdb base -o yaml > /tmp/prdb-base.before.yaml + kubectl edit prdb base + ``` + + One working override (keep any keys already under `valuesOverride`): + + ```yaml + spec: + valuesOverride: + ait/chart-kafka-zookeeper: + cpaasKafka: + resources: + requests: + cpu: 200m + memory: 256Mi + limits: + cpu: "2" + memory: 4Gi + kafka: + jvm_size: -Xmx1g -Xms512m + ``` + +3. If the OOM continues, raise the broker heap through the same override path. Keep the heap well below the container memory limit, because direct buffers and the page cache live outside the heap. +4. Confirm that the running broker JVM uses the new heap: the `-Xmx` of the Java process that loads `kafka_server_jaas.conf` must match. If it stays at `-Xmx1G` while the environment shows the new value, the start script overrides it; follow the heap cause in the retention and heap article linked above. +5. Watch `sum(kafka_consumergroup_lag{})` until it trends down, and watch `lanaya` and `apollo` for OOM restarts while the `nevermore` backlog drains. + +## Diagnostic Steps + +1. Broker state and the last fatal log: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep -E 'kafka|zookeeper|lanaya|razor' + kubectl -n cpaas-system logs -p cpaas-kafka-0 --tail=50 + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | Leader election fails, metadata corrupt | 1 | + | Exit code `137` in a tight loop, scale-out does not help | 2 | + | Pod `Pending`, node `DiskPressure` or `NotReady` | 3 | + | Write failures in the broker log, Ceph-backed Kafka PVC | 4 | + | `OutOfMemoryError: Java heap space` or `Direct buffer memory` | 5 | + +3. Distinguish the two kinds of exit 137: `Reason: OOMKilled` in `Last State` is memory; a `Liveness probe failed` event with `Reason: Error` is the probe. +4. After any wipe, check a pod that is logging right now, not a historical time range. diff --git a/docs/en/solutions/cpaas-kafka_retention_hours_memory_limit_above_80_percent_or_KAFKA_HEAP_OPTS_not_applied_on_ACP.md b/docs/en/solutions/cpaas-kafka_retention_hours_memory_limit_above_80_percent_or_KAFKA_HEAP_OPTS_not_applied_on_ACP.md new file mode 100644 index 000000000..fc9a0bd0e --- /dev/null +++ b/docs/en/solutions/cpaas-kafka_retention_hours_memory_limit_above_80_percent_or_KAFKA_HEAP_OPTS_not_applied_on_ACP.md @@ -0,0 +1,160 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# cpaas-kafka retention hours, memory limit above 80 percent, or KAFKA_HEAP_OPTS not applied on ACP + +## Issue + +Questions and problems with the Kafka of the platform log storage plugin (StatefulSet `cpaas-kafka` in `cpaas-system`): + +- How long does the log storage Kafka keep messages, and how is the retention (`log.retention.hours`) changed? +- The log nodes have a single disk, and during business peaks Kafka data could fill `/`. Can Kafka keep less? +- A rule requires Kafka memory usage at or below 80%, and the `cpaas-kafka` pods run at 80.70% to 82.00% of their container memory limit. +- `KAFKA_HEAP_OPTS` on the StatefulSet says `-Xms6g -Xmx6g` (or `2g`), but the broker Java process still runs with `-Xmx1G -Xms1G`. Logs pile up, and `razor` logs `Send kafka msg error {"topic": "ALAUDA_LOG_TOPIC", ..., "error": "context deadline exceeded"}` and `"error": "kafka write errors (1/1)"`, also for `ALAUDA_AUDIT_TOPIC`. + +This Kafka is not a Data Services instance and is not listed on the Data Services page. Its broker settings are environment variables on the container, for example: + +| Variable | Value seen | +| --- | --- | +| `KAFKA_LOG_RETENTION_HOURS` | `"48"` | +| `KAFKA_LOG_ROLL_HOURS` | `"12"` | +| `KAFKA_LOG_SEGMENT_DELETE_DELAY_MS` | `"0"` | +| `KAFKA_LOG_DIRS` | `/kafka0` | +| `KAFKA_HEAP_OPTS` | `-Xmx1g -Xms512m` | + +Container resources were limits `cpu: "2"` / `memory: 4Gi` and requests `cpu: 200m` / `memory: 256Mi` in one environment. The plugin keeps Kafka settings in the `logcenter` ModuleInfo (`spec.config.components.kafka`). The default Kafka backlog alerts are in the alert policy `cpaas-log-rules` on the global cluster. + +## Root Cause + +### Cause 1: Kafka keeps log data for 48 hours, set by KAFKA_LOG_RETENTION_HOURS + +By default the log storage Kafka keeps data for 2 days: `KAFKA_LOG_RETENTION_HOURS: "48"` in the `cpaas-kafka` environment. Where the variable is absent from the StatefulSet, the default is expected to be the same. Retention is changed by adding or editing that variable, which restarts the Kafka pods. + +### Cause 2: Kafka data fills a single disk during peaks, and retention is the trade-off + +When the Kafka log directory sits on the same file system as `/`, peak log volume can fill `/` and make the node abnormal. A lower retention reduces that risk but shortens the time available to clear a backlog: with 24 hours, any log data not written to the log store within 24 hours during a backlog is lost. Kafka removes segments only once they pass the retention age; a consumer that catches up does not free disk by itself. + +### Cause 3: cpaas-kafka container memory is above an 80% requirement + +With a `4Gi` limit and about 3.2Gi used, container memory usage is 80.70% to 82.00%. With no backlog, this is normal: Kafka relies on the page cache, which is counted in container memory. To meet a rule of 80% or less, raise the container memory limit. + +### Cause 4: KAFKA_HEAP_OPTS does not reach the broker JVM because the start script sets its own heap + +The container starts the broker through `/opt/kafka/bin/start-kafka.sh`, which calls `/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties`. In the affected image, `kafka-server-start.sh` contains an unconditional `export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"`, which overwrites the value from the environment. The StatefulSet and `printenv` show the configured heap, while the broker JVM (`java -Xmx1G -Xms1G -Djava.security.auth.login.config=/opt/kafka/conf/kafka_server_jaas.conf -server -XX:+UseG1GC ...`) runs with 1G. Threads such as `kafka-scheduler`, `data-plane-kafka`, and `ExpirationReaper` belong to that same broker JVM; `kafka_exporter` in the same pod is a separate, non-Java process. + +With a 1G heap under a high log volume, the brokers cannot keep up: replica fetchers log `Error sending fetch request ... java.io.IOException: Connection to was disconnected before the response was read`, the controller repeats `Processing automatic preferred replica leader election`, and `razor` writes time out. Mounting a corrected `kafka-server-start.sh` from a ConfigMap and restarting Kafka makes the heap setting take effect. Large on-disk queues on some nodes also made Kafka fail while loading them at startup and needed cleanup. If the backlog persists after the heap is right, the remaining limit is capacity: Kafka and Elasticsearch sharing the same nodes could not sustain the log volume, and log collection for the noisiest namespace was disabled temporarily. + +## Resolution + +Save the StatefulSet before any change: + +```bash +kubectl -n cpaas-system get sts cpaas-kafka -o yaml > /tmp/cpaas-kafka-sts.before.yaml +``` + +### Cause 1: Kafka keeps log data for 48 hours, set by KAFKA_LOG_RETENTION_HOURS + +```bash +kubectl -n cpaas-system get sts cpaas-kafka -o jsonpath='{range .spec.template.spec.containers[*].env[*]}{.name}{"="}{.value}{"\n"}{end}' | grep -E '^KAFKA_LOG_(RETENTION|ROLL|DIRS|SEGMENT)' +``` + +> **Warning:** Changing the StatefulSet environment restarts the `cpaas-kafka` pods one by one; ingestion slows and lag rises until they are Ready. A shorter retention deletes older segments, including data not consumed yet. Precondition: consumer lag is near zero, and the change was tried on a test cluster of the same version. Rollback: `kubectl apply -f /tmp/cpaas-kafka-sts.before.yaml`; deleted segments do not return. + +```bash +kubectl -n cpaas-system set env sts/cpaas-kafka KAFKA_LOG_RETENTION_HOURS= +kubectl -n cpaas-system rollout status sts/cpaas-kafka +``` + +Editing the YAML (`kubectl -n cpaas-system edit sts cpaas-kafka`, or the StatefulSet's YAML in the console) is the same change: add or modify `name: KAFKA_LOG_RETENTION_HOURS` with a quoted value such as `"48"`. If a plugin update restores the old value, read the environment again and repeat. + +### Cause 2: Kafka data fills a single disk during peaks, and retention is the trade-off + +1. Size retention as the hours of peak log volume that fit on the Kafka disk with headroom, but no shorter than the longest backlog you must survive. Set it as in Cause 1. +2. Watch the Kafka backlog alerts in `cpaas-log-rules` and set thresholds that warn before data reaches the retention age. +3. Plan a separate disk for log data if `/` is the only disk. + +For a log node whose disk is already full of Elasticsearch and Kafka data, free Elasticsearch space by deleting old indices as in [/cpaas/data full with ZooKeeper down or Elasticsearch on the root disk on ACP](cpaas_data_full_with_ZooKeeper_down_or_Elasticsearch_on_the_root_disk_on_ACP.md), and let the shorter Kafka retention remove old segments. + +### Cause 3: cpaas-kafka container memory is above an 80% requirement + +```bash +kubectl -n cpaas-system top pod | grep cpaas-kafka +kubectl -n cpaas-system get sts cpaas-kafka -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" "}{.resources}{"\n"}{end}' +``` + +Pick a limit that puts usage below 80% (usage divided by 0.8, rounded up; about 3.2Gi needs more than 4Gi). Page cache can grow toward the new limit, so check again after a few days. + +> **Warning:** Changing the resources restarts the `cpaas-kafka` pods one by one. Precondition: the Kafka nodes have the extra memory free and consumer lag is near zero. Rollback: `kubectl apply -f /tmp/cpaas-kafka-sts.before.yaml`. + +```bash +kubectl -n cpaas-system edit sts cpaas-kafka +kubectl -n cpaas-system rollout status sts/cpaas-kafka +``` + +Set `resources.limits.memory` of the Kafka container. If a plugin update restores the old limit, set the same value in the log storage plugin configuration. + +### Cause 4: KAFKA_HEAP_OPTS does not reach the broker JVM because the start script sets its own heap + +1. Compare the three values and read the script: + + ```bash + kubectl -n cpaas-system get sts cpaas-kafka -o jsonpath='{range .spec.template.spec.containers[*].env[*]}{.name}={.value}{"\n"}{end}' | grep HEAP + kubectl -n cpaas-system exec cpaas-kafka-0 -- printenv KAFKA_HEAP_OPTS + kubectl -n cpaas-system exec cpaas-kafka-0 -- sh -c "ps -ef | grep '[k]afka_server_jaas' | grep -o -- '-Xm[sx][^ ]*'" + kubectl -n cpaas-system exec cpaas-kafka-0 -- grep -n 'KAFKA_HEAP_OPTS' /opt/kafka/bin/kafka-server-start.sh + ``` + + The environment showing the configured heap, the broker process showing `-Xmx1G`, and an unconditional `export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"` in the script confirm this cause. + +2. Copy the script out and correct it so the environment value wins, for example by wrapping the line in the upstream Kafka form `if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"; fi`: + + ```bash + kubectl -n cpaas-system exec cpaas-kafka-0 -- cat /opt/kafka/bin/kafka-server-start.sh > /tmp/kafka-server-start.sh + vi /tmp/kafka-server-start.sh + kubectl -n cpaas-system create configmap cpaas-kafka-server-start --from-file=kafka-server-start.sh=/tmp/kafka-server-start.sh + ``` + +3. Mount it over the original and restart Kafka. + + > **Warning:** Changing the StatefulSet restarts the `cpaas-kafka` pods; log produce and consume pause on each broker while it restarts. Set `KAFKA_HEAP_OPTS` well below the container memory limit, or the pod is `OOMKilled`. Precondition: Cause 4 is confirmed and the StatefulSet YAML is saved. Rollback: `kubectl apply -f /tmp/cpaas-kafka-sts.before.yaml`, then delete the ConfigMap. A plugin update can replace the StatefulSet; check the mount after every update. + + ```bash + kubectl -n cpaas-system edit sts cpaas-kafka + ``` + + Add a volume from ConfigMap `cpaas-kafka-server-start` with `defaultMode: 0755`, and in the Kafka container a `volumeMount` with `mountPath: /opt/kafka/bin/kafka-server-start.sh` and `subPath: kafka-server-start.sh`. Then: + + ```bash + kubectl -n cpaas-system rollout status sts/cpaas-kafka + kubectl -n cpaas-system exec cpaas-kafka-0 -- sh -c "ps -ef | grep '[k]afka_server_jaas' | grep -o -- '-Xm[sx][^ ]*'" + ``` + + The broker process must now show the configured heap. + +4. If a broker fails while loading a large on-disk queue at startup, clear that queue before restarting it; see Cause 1 of [Platform log Kafka brokers fail: no leader election, exit 137, Pending on DiskPressure, Ceph write errors, or OOM on ACP](Platform_log_Kafka_brokers_fail_no_leader_election_exit_137_Pending_on_DiskPressure_Ceph_write_errors_or_OOM_on_ACP.md). +5. If the backlog persists with the right heap, add capacity: move Kafka or Elasticsearch to separate nodes, or reduce the log volume (for example by disabling log collection for a noisy namespace) until capacity is added. + +## Diagnostic Steps + +1. Confirm it is the platform log Kafka: `kubectl -n cpaas-system get sts,pods | grep cpaas-kafka`. +2. Retention and disk: + + ```bash + lsblk + kubectl -n cpaas-system exec cpaas-kafka-0 -- df -h /kafka0 + du -sh /cpaas/data/kafka* + ``` + + Kafka data on the same disk as `/`, or near the disk's size, is Cause 2. +3. `kubectl top` usage divided by the limit above 80%, with no lag, is Cause 3. If consumer lag is also growing, fix the backlog first. +4. Heap: when the broker JVM's `-Xmx` differs from `KAFKA_HEAP_OPTS` in the environment, read the start script (Cause 4). The Java process that loads `kafka_server_jaas.conf` is the broker; `kafka_exporter` is not a Java process. +5. Backlog alerts: in the global cluster's alert policies, check that `cpaas-log-rules` exists and its Kafka rules are normal. A firing lag rule means data older than `KAFKA_LOG_RETENTION_HOURS` in the backlog is at risk. diff --git a/docs/en/solutions/cpaas_data_full_with_ZooKeeper_down_or_Elasticsearch_on_the_root_disk_on_ACP.md b/docs/en/solutions/cpaas_data_full_with_ZooKeeper_down_or_Elasticsearch_on_the_root_disk_on_ACP.md index b06a2d2fa..7332a3213 100644 --- a/docs/en/solutions/cpaas_data_full_with_ZooKeeper_down_or_Elasticsearch_on_the_root_disk_on_ACP.md +++ b/docs/en/solutions/cpaas_data_full_with_ZooKeeper_down_or_Elasticsearch_on_the_root_disk_on_ACP.md @@ -22,6 +22,10 @@ The log storage on Alauda Container Platform is abnormal because `/cpaas` is ful - A production pod (for example Redis) cannot be scheduled and the event mentions `disk-pressure`. - Many pods, including `nevermore`, keep restarting, and `/` is above 85%. - Local disk on the global masters is high because Elasticsearch and MinIO both store data there. +- After space was freed, Elasticsearch shard recovery is very slow (`"decision" : "THROTTLE"`, `reached the limit of incoming shard recoveries [2]`), or Kafka still exits with `kafka.zookeeper.ZooKeeperClientTimeoutException`. +- The Elasticsearch log storage plugin keeps restarting with `/cpaas` at 100%, and the stored logs do not need to be kept. +- `/cpaas` is full of data that is not the running log store, such as Prometheus monitoring data or ClickHouse log data from an earlier installation. +- The platform cannot be opened after the log disk filled: `kubectl get ars -A` shows `auth-manager` as `Failed`, and `apollo` stays abnormal after the disk was freed. Logging components on ACP 4.x are in `cpaas-system` (StatefulSet `cpaas-elasticsearch`, DaemonSet `nevermore`). Audit indices or leftover Elasticsearch data filling `/cpaas`, and index deletes that free nothing, are covered in [/cpaas filled by Elasticsearch audit indices or leftover Elasticsearch data on ACP](./cpaas_filled_by_Elasticsearch_audit_indices_or_leftover_Elasticsearch_data_on_ACP.md), together with how to read the Elasticsearch credentials. @@ -64,6 +68,22 @@ The same root cause shows up as `nevermore` and other pods restarting while `/` Both components store data on the local disk of the global masters. There is no procedure to move Elasticsearch onto other global nodes; that relocation is a feature request. What can be done now is index cleanup or a larger disk. +### Cause 5: after space is freed, shard recovery is throttled and Kafka still cannot reach ZooKeeper + +The components do not all come back by themselves after the disk is cleaned. Elasticsearch recovers shards at most 2 incoming and 2 outgoing at a time per node by default, so the allocation explain shows `THROTTLE` and recovery of a large store is slow; raising concurrency to 10 and recovery bandwidth to 400MB for the duration, then reverting, speeds it up. Kafka lost its ZooKeeper connection while the disk was full and keeps failing with `Fatal error during KafkaServer startup` and `ZooKeeperClientTimeoutException`; restarting the ZooKeeper pods first and then the Kafka pods restores it. + +### Cause 6: /cpaas is 100% full and the stored logs can be discarded + +The log storage plugin keeps restarting because Elasticsearch, Kafka, and ZooKeeper under `/cpaas/data` cannot write, and the owner accepts losing all stored logs. Stopping the log components, clearing the Elasticsearch, Kafka, and ZooKeeper data on the data nodes, and starting the stores in order brings the plugin back. This is the only full-disk case in which ZooKeeper data is cleared. In a narrower variant only the ZooKeeper and Kafka data were cleared; the Elasticsearch indices came back yellow and recovered once their shards were allocated again. Check with `du` which directory actually holds the bytes first: in one case `/cpaas/data/elasticsearch/data` was only 157G of a full 1.8T. + +### Cause 7: /cpaas is filled by data that is not the current log store + +`du` names a directory that does not belong to the running Elasticsearch: Prometheus monitoring data (a `monitoring` directory) on the same disk as the log storage, or ClickHouse log data under `/cpaas/data` from an earlier ClickHouse log storage installation that the console no longer lists. Removing that data frees the space. `/cpaas/audit`, `/cpaas/event`, and `/cpaas/log` are not this cause; they are the live queue of the `nevermore` log agent. + +### Cause 8: the console is unreachable after the log disk filled, and apollo stays abnormal + +`auth-manager` is `Failed`, and `apollo`, Kafka, and ZooKeeper pods are abnormal. Kafka and ZooKeeper failed because `/cpaas` on the log storage node was full. After the disk was freed, `apollo` stayed abnormal and recovered only after the Secret `global-kubeconfig` was backed up and deleted and `apollo` was restarted. Why that Secret blocked `apollo` is not known. + ## Resolution Start with `df` and `du`, so that you do not delete Elasticsearch data to fix a log collector cache, or the reverse: @@ -161,6 +181,70 @@ du -x -d 1 -h /cpaas Clean up indices as described in the audit-indices article, or expand the disk. Moving Elasticsearch to other global nodes has no supported procedure. +### Cause 5: after space is freed, shard recovery is throttled and Kafka still cannot reach ZooKeeper + +1. Check why shards wait (set `ES_USER` and `ES_PASS` as in the audit-indices article, and `ES_HOST` and `ES_PORT` as in Diagnostic Steps): + +```bash +curl -s -u "$ES_USER:$ES_PASS" "http://$ES_HOST:$ES_PORT/_cluster/allocation/explain?pretty" | grep -E '"decision"|reached the limit' +``` + +2. Raise the recovery limits for the duration, and reset them when health is green: + +```bash +curl -s -u "$ES_USER:$ES_PASS" -X PUT "http://$ES_HOST:$ES_PORT/_cluster/settings" -H 'Content-Type: application/json' \ + -d '{"transient":{"cluster.routing.allocation.node_concurrent_recoveries":10,"indices.recovery.max_bytes_per_sec":"400mb"}}' +# when health is green: +curl -s -u "$ES_USER:$ES_PASS" -X PUT "http://$ES_HOST:$ES_PORT/_cluster/settings" -H 'Content-Type: application/json' \ + -d '{"transient":{"cluster.routing.allocation.node_concurrent_recoveries":null,"indices.recovery.max_bytes_per_sec":null}}' +``` + +3. If Kafka still fails on ZooKeeper, restart ZooKeeper first, then Kafka. + +> **Warning:** Restarting ZooKeeper and Kafka pauses log ingestion until both are Ready. Precondition: `/cpaas` has free space and the previous Kafka log shows `ZooKeeperClientTimeoutException`. Rollback: none needed; the StatefulSets recreate the pods. + +```bash +kubectl -n cpaas-system logs cpaas-kafka-0 -p --tail 50 | grep -E 'Fatal error during KafkaServer startup|ZooKeeperClientTimeoutException' +kubectl -n cpaas-system delete pods -l service_name=cpaas-zookeeper +kubectl -n cpaas-system rollout status sts/cpaas-zookeeper +kubectl -n cpaas-system delete pods -l service_name=cpaas-kafka +kubectl -n cpaas-system rollout status sts/cpaas-kafka +``` + +### Cause 6: /cpaas is 100% full and the stored logs can be discarded + +Only when `du` shows the log store holds the bytes and the owner accepts losing every stored log and unconsumed Kafka record. Follow the stop, clear, and start sequence in Cause 2 of [Platform log Kafka brokers fail: no leader election, exit 137, Pending on DiskPressure, Ceph write errors, or OOM on ACP](Platform_log_Kafka_brokers_fail_no_leader_election_exit_137_Pending_on_DiskPressure_Ceph_write_errors_or_OOM_on_ACP.md), with the data directories under `/cpaas/data` (for example `elasticsearch`, `kafka0`, and `zookeeper`). Record owner and mode with `stat` before clearing, and give recreated directories the same. If the log nodes are cordoned to keep them for logging, uncordon them before scaling up and cordon them again once the pods are Running. For the narrower variant, clear only the Kafka and ZooKeeper directories, then retry allocation (`_cluster/reroute?retry_failed=true`) until the indices are green. + +### Cause 7: /cpaas is filled by data that is not the current log store + +```bash +du -sh /cpaas/* /cpaas/data/* 2>/dev/null | sort -h | tail -20 +kubectl get pods -A -o wide | grep | grep -Ei 'prometheus|clickhouse|elasticsearch' +``` + +> **Warning:** Uninstalling Prometheus monitoring and clearing its data deletes the stored metrics and stops collection until monitoring is installed again. Precondition: the owner accepted removing Prometheus monitoring from this cluster, and the directory is its data directory on each node. Rollback: reinstall the monitoring plugin; old metrics do not return. + +Uninstall the Prometheus monitoring plugin, then on each node run `rm -rf /cpaas//*`. + +> **Warning:** Deleting old ClickHouse data removes those logs permanently. Precondition: no ClickHouse pod runs on the node (`kubectl get pods -A -o wide | grep -i clickhouse`), and the owner accepts the loss. Rollback: none. + +Remove the leftover ClickHouse data with `rm -rf /cpaas/data/`. Clean it before reinstalling ClickHouse log storage on that node. + +### Cause 8: the console is unreachable after the log disk filled, and apollo stays abnormal + +Recover the disk first. Then, only when `apollo` is still abnormal after Kafka and ZooKeeper are Running: + +> **Warning:** Deleting `global-kubeconfig` removes a Secret that `apollo` reads; what recreates it is not documented. Precondition: the backup file is written and readable, the disk is no longer full, and Kafka and ZooKeeper are Running. Rollback: `kubectl apply -f /tmp/global-kubeconfig.bak.yaml`. + +```bash +kubectl get secret -A | grep global-kubeconfig +kubectl -n get secret global-kubeconfig -o yaml > /tmp/global-kubeconfig.bak.yaml +kubectl -n delete secret global-kubeconfig +kubectl -n rollout restart deploy/ +``` + +If `apollo` stays abnormal, restore the backup and collect its log. To keep the log disk from filling again, set a project-level log retention and watch index sizes. + ## Diagnostic Steps 1. Split `/cpaas` from `/var/cpaas` with `df` and `du`. A full `/var` with a small `/cpaas` is the log collector cache, not an Elasticsearch problem. @@ -175,3 +259,6 @@ ES_HOST=$(kubectl get svc -n cpaas-system cpaas-elasticsearch -o jsonpath='{.spe ES_PORT=$(kubectl get svc -n cpaas-system cpaas-elasticsearch -o jsonpath='{.spec.ports[0].port}') curl -u "$ES_USER:$ES_PASS" "http://$ES_HOST:$ES_PORT/_cat/indices?v&s=store.size:desc" | head -n 20 ``` + +7. After cleanup, `THROTTLE` in the allocation explain or `ZooKeeperClientTimeoutException` in the previous Kafka log: Cause 5. A full log store whose data may be discarded: Cause 6. The largest directory is not Elasticsearch, Kafka, ZooKeeper, or the `nevermore` queue: Cause 7. `auth-manager` `Failed` and `apollo` abnormal after the disk is free: Cause 8. +8. If `ls /cpaas` or `ls /cpaas/data` returns `Input/output error` instead of listing files, the disk is not full but the file system or the block device under it has failed. Check `dmesg` and the device (`lsblk`, and `multipath -ll` on multipath storage) before freeing space or deleting data. diff --git a/docs/en/solutions/cpaas_filled_by_Elasticsearch_audit_indices_or_leftover_Elasticsearch_data_on_ACP.md b/docs/en/solutions/cpaas_filled_by_Elasticsearch_audit_indices_or_leftover_Elasticsearch_data_on_ACP.md index 0dda0567d..c0b63c8ac 100644 --- a/docs/en/solutions/cpaas_filled_by_Elasticsearch_audit_indices_or_leftover_Elasticsearch_data_on_ACP.md +++ b/docs/en/solutions/cpaas_filled_by_Elasticsearch_audit_indices_or_leftover_Elasticsearch_data_on_ACP.md @@ -5,6 +5,7 @@ products: - Alauda Container Platform ProductsVersion: - 3.12.x + - 3.18.x - 4.0.x - 4.1.x - 4.2.x @@ -21,6 +22,8 @@ A node alarm on Alauda Container Platform names `/cpaas`, or `df` shows `/cpaas` - Application-log retention in the log center was shortened, and the disk did not shrink. - Elasticsearch log search fails, `curl` against the Elasticsearch API returns 503, or a delete of old indices returns success and the disk does not move. - `/cpaas/data/elasticsearch` is very large although the cluster now uses an external Elasticsearch. +- Logs were moved to a new disk mounted at the old path, and the root disk stays full although `du` finds nothing large. +- Elasticsearch keeps restarting with a CPU alarm after an application upgrade, and the system log indices are the large ones. Do not treat every directory under `/cpaas` as disposable log cache: ZooKeeper data can live on the same filesystem. When `/cpaas/data` is full and ZooKeeper does not start, when the console shows no logs while `kubectl logs` works, or when Elasticsearch shares the root disk, see [/cpaas/data full with ZooKeeper down or Elasticsearch on the root disk on ACP](./cpaas_data_full_with_ZooKeeper_down_or_Elasticsearch_on_the_root_disk_on_ACP.md). @@ -46,6 +49,14 @@ There are two recoveries: expand the disk, wait until health is not red, clear t After a switch to an external Elasticsearch, `/cpaas/data/elasticsearch` on a global node can still hold the old indices. For example, `du -sh *` shows `nodes` at 114G; under `nodes/0`, `indices` is 114G, `node.lock` 0, and `_state` 64K, and all index directories are dated long before the switch. +### Cause 4: the old log files are hidden under the mount of the new log disk + +The logs were migrated to another disk that is mounted on the old path (for example `/cpaas/data`), and the root file system stays full. The files that were written to that path before the new disk was mounted are still on the root file system, but the mount hides them: `du` on the mounted path only sees the new disk, so nothing large shows up. They can be removed only by unmounting the new disk or by reaching the underlying file system through a second mount point. In the recovery that cleared this, the log files on the new disk were removed as well, and the log history was not kept. + +### Cause 5: system log collection left on after an application upgrade floods Elasticsearch + +After an application upgrade, system log collection stayed enabled, the system log indices grew quickly, and Elasticsearch kept restarting with a CPU alarm. Disabling system log collection and deleting those indices stopped the restarts and the alarm. There is no per-project collection quota and no switch to collect only some log levels; what exists is a retention policy per project and the retention of `log-workload` in the log policy settings. + ## Resolution Start with `df` and `du`, so that you do not delete Elasticsearch data to fix a log collector cache, or the reverse: @@ -156,6 +167,48 @@ rm -rf /cpaas/data/elasticsearch/nodes/0/indices/* df -h /cpaas ``` +### Cause 4: the old log files are hidden under the mount of the new log disk + +1. Confirm that the root file system is full while `du` cannot account for it, and find the mount that sits on the old path: + +```bash +df -hT / /cpaas /cpaas/data +du -x -d 1 -h / 2>/dev/null | sort -h | tail +findmnt -T /cpaas/data -o TARGET,SOURCE,FSTYPE +``` + +2. Stop the log pods that hold the mount (scale the log storage workloads to 0 as in Cause 2 and record their replica counts), then reach the root file system through a second mount point and list what lies under the old path: + +```bash +mkdir -p /mnt/rootfs +mount --bind / /mnt/rootfs +du -sh /mnt/rootfs/cpaas/data/* 2>/dev/null +``` + +> **Warning:** Deleting the hidden tree removes the old log files permanently. Precondition: the path is under `/mnt/rootfs` (the root file system seen without the new disk), not the live mount of the new log disk, and `lsof` shows no process using it. Rollback: none. + +```bash +rm -rf /mnt/rootfs/cpaas/data/ +umount /mnt/rootfs +df -h / +``` + +3. Scale the log storage workloads back to their recorded replica counts. + +### Cause 5: system log collection left on after an application upgrade floods Elasticsearch + +1. Confirm that system log indices dominate (credentials as in Cause 1): + +```bash +kubectl -n cpaas-system exec cpaas-elasticsearch-0 -c cpaas-elasticsearch -- \ + curl -sS -u "${ES_USER}:${ES_PASS}" \ + 'http://cpaas-elasticsearch:9200/_cat/indices/log-system-*?v&h=index,docs.count,store.size&s=store.size:desc' +``` + +2. Disable system log collection in the log collector settings. +3. Delete the large system log indices by name, with the same warning and command as Cause 1. +4. To keep the remaining volume down, shorten the retention of `log-workload` in the log policy settings or set a retention for the noisy project. + ## Diagnostic Steps 1. Split `/cpaas` from `/var/cpaas` with `df` and `du` as at the start of the Resolution. @@ -167,7 +220,7 @@ kubectl -n cpaas-system exec cpaas-elasticsearch-0 -c cpaas-elasticsearch -- \ curl -sS -u "${ES_USER}:${ES_PASS}" 'http://cpaas-elasticsearch:9200/_cluster/health?pretty' ``` -3. Health green and the large indices are audit: Cause 1. Health red, 503, or a delete that frees nothing: Cause 2. A large Elasticsearch directory that `lsof` shows no process using: Cause 3. +3. Health green and the large indices are audit: Cause 1. Health red, 503, or a delete that frees nothing: Cause 2. A large Elasticsearch directory that `lsof` shows no process using: Cause 3. A full root file system that `du` cannot account for, with a log disk mounted on the old path: Cause 4. Large `log-system-*` indices after an application upgrade: Cause 5. 4. To see which components produce the log volume, list indices by size from a master node through the Service address: ```bash diff --git a/docs/en/solutions/razor_crash-loops_or_audit_logs_stop_TOO_MANY_PARTS_BufferTooSmall_vector_OOMKilled_or_stale_ClickHouse_token_on_ACP.md b/docs/en/solutions/razor_crash-loops_or_audit_logs_stop_TOO_MANY_PARTS_BufferTooSmall_vector_OOMKilled_or_stale_ClickHouse_token_on_ACP.md new file mode 100644 index 000000000..5fd4ffaa9 --- /dev/null +++ b/docs/en/solutions/razor_crash-loops_or_audit_logs_stop_TOO_MANY_PARTS_BufferTooSmall_vector_OOMKilled_or_stale_ClickHouse_token_on_ACP.md @@ -0,0 +1,242 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# razor crash-loops or audit logs stop: TOO_MANY_PARTS, BufferTooSmall, vector OOMKilled, or stale ClickHouse token on ACP + +## Issue + +The log storage is ClickHouse, and one of these is seen on the cluster that runs it: + +- Audit search is empty, or has nothing for the last day, while application logs or events still arrive. The `vector` container of `razor` in `cpaas-system` logs `Too many partitions for single INSERT block (more than 100)` or `Code: 252 ... (TOO_MANY_PARTS)`. +- `razor-0` is `1/2` or `CrashLoopBackOff`, and `vector` logs `failed to encode record: BufferTooSmall`. +- `razor` is `CrashLoopBackOff`, its probes fail with `dial tcp :8081: connect: connection refused`, and the log query page fails with a ClickHouse `500` error while the ClickHouse pods log no errors. +- `razor` logs `Failed to create project table` and ClickHouse reports `SYNTAX_ERROR` at `'-'`. +- After an upgrade from 4.3.1 to 4.3.2, ClickHouse looks empty although no table was dropped. +- Logs, events, and audit all stop on the same day while every pod is `Running`, and `nevermore` logs timeouts to `razor`. +- ClickHouse log storage was just deployed, the log module shows nothing, and ClickHouse has no new rows. + +`razor` runs two containers: `razor` and `vector`. Vector writes to ClickHouse and keeps a persistent disk buffer. Readiness, liveness, and startup probes use TCP port `8081`, which is served only while `vector` is running, so any `vector` crash makes the pod fail its probes. `razor` is a StatefulSet on most installs; check with `kubectl -n cpaas-system get deploy,sts | grep razor`. + +## Root Cause + +### Cause 1: inserts into observability.audit_all exceed max_partitions_per_insert_block + +The ClickHouse tables are partitioned by day, and ClickHouse rejects an INSERT that touches more than `max_partitions_per_insert_block` partitions (default 100). A replay of historical audit data, or a very large Vector batch, puts more than 100 days into one INSERT. Vector treats the rejection as transient and retries the same batch, the disk buffer fills (about 2.5 GB in one case), and real-time audit stops. A restart alone does not clear it, because the buffer is on disk. `event_all` can stay current, which shows that ClickHouse, the network, and `razor` are reachable and only the audit sink is stuck. + +The same limit can stop logs, events, and audit together: no data after a given date in ClickHouse, all pods `Running`, `vector` logging on the `clickhouse` sink with `component_id=audit`: + +```text +Code: 252. DB::Exception: Too many partitions for single INSERT block (more than 100). The limit is controlled by 'max_partitions_per_insert_block' setting. ... (TOO_MANY_PARTS) +``` + +and the `http` source (`path=/v4/callback/logs`) reporting `component_events_dropped` with `reason="Source send cancelled."`. Reducing the Vector batch size (or raising the partition limit for the duration of the replay) lets inserts succeed again. `system.errors` counts the `TOO_MANY_PARTS` failures (tens of thousands in one case). + +### Cause 2: vector exits on BufferTooSmall, and the razor container then sits at its CPU limit + +The `razor` container is running and `vector` is not. Vector logs `Disk buffer writer has encountered an unrecoverable error` and `failed to encode record: BufferTooSmall`, then stops, and the probes on `8081` are refused. PVC `data-razor-0` is not full and the node filesystem has space. `razor` sent Vector a batch that was too large to encode. When the crash loop slows, the `razor` container CPU can sit at 100% of a 1-core limit; restarting does not change it, and raising the limit to 2 cores leaves it running at about 1.5 cores with no errors. + +### Cause 3: a project name containing a hyphen makes ClickHouse reject the project table + +`razor` creates a per-project table for a project-level retention policy. When the project name contains a hyphen, the statement (` ON CLUSTER replicated`) fails with `SYNTAX_ERROR` at `'-'`, and `razor` logs `Failed to create project table` repeatedly. The shared tables in database `observability` (`audit`, `event`, `log_kubernetes`, `log_platform`, `log_system`, `log_workload`, `meter_detail`, `meter_summary`, `schema_migrations`) are not affected. + +### Cause 4: after the upgrade from 4.3.1 to 4.3.2, razor and nevermore keep a stale ClickHouse token + +ClickHouse looks empty after the upgrade, but no rows were deleted. `razor` and `nevermore` keep ClickHouse authentication that the upgrade did not refresh. This is a known defect on that upgrade path. Forcing the token in Secret `log-database-bearer-auth` to expire and restarting `razor-api` and `nevermore` makes the data visible again. + +Log queries belong on the primary global cluster; a standby global cluster whose log feature points at a domain name is not a log query endpoint. + +### Cause 5: nevermore upload requests to razor time out although razor is reachable + +`nevermore` logs: + +```text +Failed to publish events ... Post "http://razor:8080/v4/callback/logs": context deadline exceeded (Client.Timeout exceeded while awaiting headers) +``` + +A manual request to the same URL from a `nevermore` pod returns `Unauthorized with error:authorization header is empty`, so the endpoint answers. Each upload batch is so large that the request times out. Lowering the upload `batch_size` from 1000 to 100 restores the connection. This can come together with Cause 1. + +### Cause 6: nevermore requests to the Kubernetes API time out, and then vector is OOMKilled + +On a newly deployed ClickHouse log storage, the log module shows nothing, a direct ClickHouse query finds no new rows, and the query page returns no error, so the break is before ClickHouse. `nevermore` requests to the Kubernetes API are timing out. Restarting `nevermore` clears that, and the next hop then fails: the `vector` container in `razor` is `OOMKilled`. Both fixes are needed: restart `nevermore`, then raise the `vector` memory limit. + +### Cause 7: vector is OOMKilled at its memory limit, razor crash-loops, and log queries return a ClickHouse 500 error + +The `vector` container reaches its memory limit (`memory: 1Gi`, with a request of `256Mi`, in one case) and is killed. `kubectl describe pod` shows `State: Waiting`, `Reason: CrashLoopBackOff`, `Last State: Terminated`, `Reason: OOMKilled`, and a growing restart count. Nothing listens on `8081`, so `razor` logs `dial tcp :8081: connect: connection refused`, the pod never becomes ready, and the log query page fails with a ClickHouse `500` error. The ClickHouse pods themselves are healthy and log no errors. Raising the `vector` memory limit restores log queries. + +## Resolution + +### Cause 1: inserts into observability.audit_all exceed max_partitions_per_insert_block + +1. Confirm the error and read the Vector configuration: + + ```bash + kubectl -n cpaas-system logs razor-0 -c vector --tail=1500 | grep -iE 'audit|Code:|TOO_MANY|partition|dropped' -C3 + kubectl -n cpaas-system get secret cpaas-vector -o yaml > /tmp/cpaas-vector.before.yaml + kubectl -n cpaas-system get secret cpaas-vector -o jsonpath='{.data.vector\.yaml}' | base64 -d > /tmp/vector.yaml + grep -n max_events /tmp/vector.yaml + ``` + + Repeat the log command for `razor-1` if it exists. + +2. Edit `/tmp/vector.yaml` so the audit sink has `sinks.audit.batch.max_events: 50`. If the file has only one `batch.max_events` (for example `10000000`), set it to `99` until the backlog is written, then restore the saved value. + + > **Warning:** Patching Secret `cpaas-vector` and restarting `razor` reloads Vector; ingestion through `razor` pauses until the pod is Ready. Precondition: only when `Too many partitions for single INSERT block (more than 100)` or `TOO_MANY_PARTS` is confirmed. Rollback: `kubectl apply -f /tmp/cpaas-vector.before.yaml` and restart `razor` again. + + ```bash + kubectl -n cpaas-system patch secret cpaas-vector --type merge \ + -p "{\"data\":{\"vector.yaml\":\"$(base64 -w0 /tmp/vector.yaml)\"}}" + kubectl -n cpaas-system rollout restart sts/razor + kubectl -n cpaas-system rollout status sts/razor + ``` + +3. If the stuck batch still fails, raise the ClickHouse `max_partitions_per_insert_block` setting above 100 while the backlog drains, and set it back once `audit_all` receives current timestamps. Read the current value first: + + ```bash + kubectl -n cpaas-system exec -c -- \ + clickhouse-client --query "SELECT value FROM system.settings WHERE name = 'max_partitions_per_insert_block'" + ``` + +4. Query audit for the current hour. A gap matching the time the sink was stuck can remain. + +### Cause 2: vector exits on BufferTooSmall, and the razor container then sits at its CPU limit + +1. Confirm which container stops and why: + + ```bash + kubectl -n cpaas-system get pod razor-0 -o jsonpath='{range .status.containerStatuses[*]}{.name}{" restarts="}{.restartCount}{" last="}{.lastState.terminated.reason}{"\n"}{end}' + kubectl -n cpaas-system logs razor-0 -c vector --previous --tail=80 + ``` + +2. If `razor` also logs `Failed to create project table`, fix Cause 3 first. +3. If the `razor` container stays at its CPU limit, raise it. + + > **Warning:** Changing the `razor` container resources rolls the workload; ingestion and queries pause during the roll. Precondition: only when `BufferTooSmall` is confirmed and the `razor` container is at its CPU limit, and the node has a free core. Rollback: set the previous limit again. If a plugin update puts the old value back, set it again the same way. + + ```bash + kubectl -n cpaas-system set resources sts/razor -c razor --limits=cpu=2 + ``` + +### Cause 3: a project name containing a hyphen makes ClickHouse reject the project table + +```bash +kubectl -n cpaas-system logs -c razor --tail=80 | grep -E 'Failed to create project table|SYNTAX_ERROR' +``` + +> **Warning:** Deleting the retention policy removes that project's retention rule; its logs follow the default retention afterwards. Precondition: only when the failing statement names a project that contains a hyphen. Rollback: none for the deleted policy. + +Delete that project's log retention policy in the console. The `Failed to create project table` line stops. If CPU stays high afterwards, apply Cause 2. + +### Cause 4: after the upgrade from 4.3.1 to 4.3.2, razor and nevermore keep a stale ClickHouse token + +> **Warning:** Expiring the token and restarting `razor-api` and `nevermore` pauses log queries and collection during the restarts. Precondition: only when ClickHouse looks empty after the upgrade from 4.3.1 to 4.3.2. Rollback: restore the saved Secret and restart the same two workloads. + +```bash +kubectl -n cpaas-system get secret log-database-bearer-auth -o yaml > /tmp/log-database-bearer-auth.before.yaml +kubectl -n cpaas-system annotate secret log-database-bearer-auth \ + razor-api/token-expires-at="2000-01-01T00:00:00Z" --overwrite +kubectl -n cpaas-system rollout restart deploy/razor-api +kubectl -n cpaas-system rollout restart ds/nevermore +kubectl -n cpaas-system rollout status deploy/razor-api +kubectl -n cpaas-system rollout status ds/nevermore +``` + +Refresh the log page after both are Ready. + +### Cause 5: nevermore upload requests to razor time out although razor is reachable + +```bash +kubectl -n cpaas-system logs --all-containers --tail=200 | grep -iE 'razor|timeout|Failed to publish' +kubectl -n cpaas-system exec -c log -- curl -sS http://razor:8080/v4/callback/logs +kubectl -n cpaas-system get cm nevermore-config -o yaml > /tmp/nevermore-config.before.yaml +grep -n batch_size /tmp/nevermore-config.before.yaml +``` + +An `Unauthorized` answer means `razor` is reachable. If the container has no `curl`, run the request from another pod in `cpaas-system`. + +> **Warning:** Editing `nevermore-config` and restarting `nevermore` pauses collection on each node while its pod restarts; a batch that is too small slows shipping on busy nodes. Precondition: only when `nevermore` logs upload timeouts and a manual request to `razor` is answered. Rollback: `kubectl apply -f /tmp/nevermore-config.before.yaml` and restart `nevermore` again. + +```bash +kubectl -n cpaas-system edit cm nevermore-config +kubectl -n cpaas-system rollout restart ds/nevermore +``` + +Change the upload `batch_size` from `1000` to `100`, then watch the timeouts stop and current rows arrive in ClickHouse. + +### Cause 6: nevermore requests to the Kubernetes API time out, and then vector is OOMKilled + +1. Confirm that nothing new reaches ClickHouse (use `DESCRIBE TABLE observability.log_workload` if the time column has another name): + + ```bash + kubectl -n cpaas-system exec -c -- \ + clickhouse-client --query "SELECT max(timestamp) FROM observability.log_workload" + kubectl -n cpaas-system logs --all-containers --tail=300 | grep -iE 'timeout|deadline|apiserver|443' + ``` + +2. Restart `nevermore`. + + > **Warning:** Restarting `nevermore` pauses log, audit, and event collection on each node while its pod restarts. Precondition: only when ClickHouse has no new rows and `nevermore` logs Kubernetes API timeouts. Rollback: none needed; the DaemonSet recreates the pods. + + ```bash + kubectl -n cpaas-system rollout restart ds/nevermore + kubectl -n cpaas-system rollout status ds/nevermore + ``` + +3. If `vector` is then `OOMKilled`, raise its memory as in Cause 7. + +### Cause 7: vector is OOMKilled at its memory limit, razor crash-loops, and log queries return a ClickHouse 500 error + +1. Read the last termination reason and the current limit of each container: + + ```bash + for p in $(kubectl -n cpaas-system get pod -o name | grep razor); do + echo "== $p" + kubectl -n cpaas-system get "$p" -o jsonpath='{range .status.containerStatuses[*]}{.name}{" restarts="}{.restartCount}{" last="}{.lastState.terminated.reason}{"\n"}{end}' + done + kubectl -n cpaas-system get sts razor -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" "}{.resources.limits.memory}{"\n"}{end}' + ``` + +2. Raise the memory limit of the `vector` container. + + > **Warning:** Editing the `razor` workload restarts its pods; log ingestion and queries pause until they are Ready. Precondition: only when the `vector` container's last state is `OOMKilled`, and the node has the memory for the new limit. Rollback: set the previous memory limit again. + + ```bash + kubectl -n cpaas-system edit sts razor + ``` + + In the `vector` container, raise `resources.limits.memory` (one recovery used `4Gi`; size it from observed usage and the node's free memory). In that recovery the edit generated a resource patch (`rpch`) that kept the new value through later reconciles; check that the value is still in place after the next plugin update. + +3. When `vector` stops restarting and the pod is Ready, run the log query again. + +## Diagnostic Steps + +1. Identify which container fails and what it logs: + + ```bash + kubectl -n cpaas-system get pods | grep -E 'razor|nevermore' + kubectl -n cpaas-system describe pod razor-0 | sed -n '/Containers:/,/Conditions:/p' + kubectl -n cpaas-system logs razor-0 -c vector --previous --tail=60 + kubectl -n cpaas-system logs razor-0 -c razor --tail=40 + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | `TOO_MANY_PARTS` or `more than 100` on the ClickHouse sink | 1 | + | `BufferTooSmall` in the previous `vector` log | 2 | + | `Failed to create project table` with `SYNTAX_ERROR` at `'-'` | 3 | + | ClickHouse looks empty right after the 4.3.1 to 4.3.2 upgrade | 4 | + | `nevermore` times out on `/v4/callback/logs`, manual request answered | 5 | + | No new ClickHouse rows and `nevermore` times out against the Kubernetes API | 6 | + | `vector` last state `OOMKilled`, `8081` refused, query page shows ClickHouse `500` | 7 | + +3. A `connection refused` on `8081` only says that `vector` is down; always read the previous `vector` log and the last termination reason before changing anything. A ClickHouse pod that is itself `OOMKilled` or exits with `137` is a ClickHouse capacity problem, see [Logging ClickHouse runs out of disk, CPU or memory, or is OOMKilled on ACP](Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md). + +Related articles: [razor does not become ready after a ClickHouse log storage install or upgrade on ACP](razor_does_not_become_ready_after_a_ClickHouse_log_storage_install_or_upgrade_on_ACP.md), [nevermore OOMKilled, cache written to /var/cpaas, or callback-secret mount failure on ACP](nevermore_OOMKilled_cache_written_to_var_cpaas_or_callback-secret_mount_failure_on_ACP.md). From b6fd65cb0aaa570a429213ce5781fe04bf2fc735 Mon Sep 17 00:00:00 2001 From: Komh Date: Wed, 18 Nov 2026 17:41:20 +0800 Subject: [PATCH 19/25] docs(solutions): add 16 and update 0 troubleshooting articles --- ...d_platform_alerts_raised_with_it_on_ACP.md | 89 ++++++++ ...ress_from_cpaas-alb-router-rules_on_ACP.md | 131 +++++++++++ ...ng_interval_does_not_take_effect_on_ACP.md | 85 +++++++ ...nfo_of_a_deleted_cluster_remains_on_ACP.md | 66 ++++++ ...ess_certificates_or_fail_to_load_on_ACP.md | 176 +++++++++++++++ ...or_misses_menus_after_an_upgrade_on_ACP.md | 163 ++++++++++++++ ...or_empty_Pod_logs_in_one_browser_on_ACP.md | 113 ++++++++++ ...le_file_upload_or_download_fails_on_ACP.md | 186 ++++++++++++++++ ...ress_credentials_and_vmui_errors_on_ACP.md | 177 +++++++++++++++ ...e_or_rules_and_alerts_do_not_carry_over.md | 196 +++++++++++++++++ ...ile_the_target_Pod_looks_healthy_on_ACP.md | 208 ++++++++++++++++++ ...ube-state_Prometheus_vm-operator_on_ACP.md | 184 ++++++++++++++++ ...ervice_mkdirOnNode_or_alertmanager-pv-0.md | 157 +++++++++++++ ...es_ModuleInfo_AppRelease_or_Pods_on_ACP.md | 182 +++++++++++++++ ...hing_between_firing_and_resolved_on_ACP.md | 68 ++++++ ...c_StatefulSet_partition_reloader_on_ACP.md | 165 ++++++++++++++ 16 files changed, 2346 insertions(+) create mode 100644 docs/en/solutions/ALB_alb_loop_time_alert_from_cpaas-alb-rules_and_platform_alerts_raised_with_it_on_ACP.md create mode 100644 docs/en/solutions/ALB_error-rate_alert_on_a_route_or_Ingress_from_cpaas-alb-router-rules_on_ACP.md create mode 100644 docs/en/solutions/Alert_repeat_interval_cannot_be_set_to_4_hours_or_the_global_sending_interval_does_not_take_effect_on_ACP.md create mode 100644 docs/en/solutions/AppRelease_abnormal_because_a_ModuleInfo_of_a_deleted_cluster_remains_on_ACP.md create mode 100644 docs/en/solutions/Certificate_and_restart_alerts_that_persist_after_an_upgrade_match_business_certificates_or_fail_to_load_on_ACP.md create mode 100644 docs/en/solutions/Console_returns_404_shows_old_pages_or_misses_menus_after_an_upgrade_on_ACP.md create mode 100644 docs/en/solutions/Console_white_page_clipped_menu_dead_Create_button_or_empty_Pod_logs_in_one_browser_on_ACP.md create mode 100644 docs/en/solutions/Debug_container_blocked_or_left_running_and_console_file_upload_or_download_fails_on_ACP.md create mode 100644 docs/en/solutions/External_Grafana_or_scripts_cannot_query_platform_metrics_vmselect_address_credentials_and_vmui_errors_on_ACP.md create mode 100644 docs/en/solutions/Grafana_returns_HTTP_500_after_a_password_change_is_missing_after_an_upgrade_or_rules_and_alerts_do_not_carry_over.md create mode 100644 docs/en/solutions/Metrics_Service_availability_alert_is_not_1_while_the_target_Pod_looks_healthy_on_ACP.md create mode 100644 docs/en/solutions/Monitoring_components_OOMKilled_or_near_their_limits_vmagent_vmstorage_kube-state_Prometheus_vm-operator_on_ACP.md create mode 100644 docs/en/solutions/Monitoring_plugin_install_or_redeploy_fails_ResourcePatch_conflict_IPv6_Service_mkdirOnNode_or_alertmanager-pv-0.md create mode 100644 docs/en/solutions/Monitoring_plugin_uninstall_or_Prometheus_to_VictoriaMetrics_switch_leaves_ModuleInfo_AppRelease_or_Pods_on_ACP.md create mode 100644 docs/en/solutions/Node_zombie_process_alert_keeps_switching_between_firing_and_resolved_on_ACP.md create mode 100644 docs/en/solutions/Workload_changes_reverted_not_applied_or_restarting_Pods_OAM_Helm_template_sync_StatefulSet_partition_reloader_on_ACP.md diff --git a/docs/en/solutions/ALB_alb_loop_time_alert_from_cpaas-alb-rules_and_platform_alerts_raised_with_it_on_ACP.md b/docs/en/solutions/ALB_alb_loop_time_alert_from_cpaas-alb-rules_and_platform_alerts_raised_with_it_on_ACP.md new file mode 100644 index 000000000..72d2d1da5 --- /dev/null +++ b/docs/en/solutions/ALB_alb_loop_time_alert_from_cpaas-alb-rules_and_platform_alerts_raised_with_it_on_ACP.md @@ -0,0 +1,89 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.1.x + - 4.3.x +--- + +# ALB alb_loop_time alert from cpaas-alb-rules and platform alerts raised with it on ACP + +## Issue + +The real-time alert page shows policy `cpaas-alb-rules` with `alb_loop_time` above 1000 ms (`alb_loop_time超过1000.00ms`), for example a trigger value of `5534.00`, often recovering by itself. A general request to troubleshoot the platform alerts (`告警问题排查处理`) frequently bundles it with other alerts from the same period: the ALB `error-rate` alert, compute component restarts, compute component CPU above 90%, Elasticsearch disk usage above 70%, and node disk usage. In one such set, Elasticsearch disk usage was above 70% on only one `global` node while the others stayed low. + +Related article: [ALB error-rate alert on a route or Ingress from cpaas-alb-router-rules on ACP](ALB_error-rate_alert_on_a_route_or_Ingress_from_cpaas-alb-router-rules_on_ACP.md). + +## Root Cause + +### Cause 1: alb_loop_time is the duration of one ALB control-plane loop + +`alb_loop_time` measures one loop of the ALB control plane, not business request latency. A loop fetches resources, generates the NGINX configuration, reloads it (including writing `nginx.conf` to disk) and updates rule status. A loop longer than 1 second is typically caused by a slow kube-apiserver request; slow disk on the node can also stretch the reload step. The alert is warning level (`alb_loop_time > 1000ms` for 30 seconds). A spike can occur on a node with very low CPU, memory and load, and the specific cause of a single spike is not always identifiable afterwards, especially when the ALB logs of that window were not kept. + +A single alert row that recovered within about 30 seconds after it appeared, with users able to reach the service and no ALB Pod restart, can be recorded and ignored. A row that stays up for minutes is not covered by that rule. + +### Cause 2: Elasticsearch disk usage above 70% on one global node because a disk is mounted on the wrong path + +Only one `global` node reports high Elasticsearch disk usage, and the other Elasticsearch nodes are low. On that global master node a disk is mounted on the wrong path, which makes its usage uneven with the other nodes. This is not an ALB fault, and no ALB change fixes it. + +## Resolution + +### Cause 1: alb_loop_time is the duration of one ALB control-plane loop + +1. Read the alert value and how long the row stayed, and check the ALB Pods and their node: + + ```bash + kubectl get pods -n cpaas-system -o wide | grep -E 'alb2|alb' + ``` + +2. If the row fired once, recovered within about 30 seconds, users were not blocked and the ALB Pods did not restart, record it and leave the policy. The "about 30 seconds" is how fast the row cleared, not the `for 30 seconds` clause of the rule. + +3. If it repeats or stays up, save the `alb2` container log of every replica before the Pods roll, and check the API server latency and the disk latency of the node in the same window: + + ```bash + kubectl logs -n cpaas-system -c alb2 --since=1h > /tmp/-alb2.log + ``` + +Do not delete the ALB Pod to clear a single short spike. + +### Cause 2: Elasticsearch disk usage above 70% on one global node because a disk is mounted on the wrong path + +1. Locate the Elasticsearch Pods and their nodes, then compare the block devices and filesystems on each node: + + ```bash + kubectl get pods -A -o wide | grep -i elas + lsblk + df -Th | less + ``` + + On the high node, `lsblk` and `df -Th` show the data disk mounted on a different path from the other nodes. + +2. Migrate the data and remount the disk on the correct path in a maintenance window. + + > **Warning:** Data on that path is unavailable during the move, and a mistake in the copy or the mount can lose it. Precondition: one node is high, the others are low, `lsblk` and `df -Th` show the wrong mount, a verified copy of the data exists, and `/etc/fstab` is saved. Rollback: restore the saved `/etc/fstab` entry, remount, and copy the data back; data removed without a copy cannot be recovered. + +3. Afterwards `df -Th` on that node shows the disk on the correct path, and Elasticsearch usage is in line with the other nodes. + +## Diagnostic Steps + +1. Read the policy name. `cpaas-alb-rules` and `alb_loop_time` are Cause 1. `cpaas-alb-router-rules` and `error-rate` are the related article. + +2. One firing that recovered within about 30 seconds, no user-facing failure, ALB Pods with no new restarts: record and leave the warning. + +3. When several platform alerts arrive together, handle each separately: + - ALB `error-rate`: the 4xx/5xx ratio of ALB requests over the last 5 minutes passed the threshold, typically a backend that was stopped while its rule still received requests, or a backend that returns 4xx/5xx itself. Check the ALB and backend logs; see the related article. + - `alb_loop_time > 1000ms`: Cause 1. + - Compute component restarts: read the Pod events and the previous container log for the last restart reason, such as OOM: + + ```bash + kubectl describe pod -n | grep -A5 'Last State' + kubectl logs -n --previous --tail=100 + ``` + + If both are normal, the alert can be ignored. + - Compute component CPU above 90%: raise the CPU limit of that StatefulSet or Deployment, for example to double its current value. Changing the limit restarts its Pods; keep the previous value to roll back. + - Elasticsearch disk above 70% on every node: check and expand the disks. On only one node: Cause 2. + - Node disk usage: on the node, find the full filesystem with `df -Th`, and clean it up or expand it. diff --git a/docs/en/solutions/ALB_error-rate_alert_on_a_route_or_Ingress_from_cpaas-alb-router-rules_on_ACP.md b/docs/en/solutions/ALB_error-rate_alert_on_a_route_or_Ingress_from_cpaas-alb-router-rules_on_ACP.md new file mode 100644 index 000000000..f339f00cf --- /dev/null +++ b/docs/en/solutions/ALB_error-rate_alert_on_a_route_or_Ingress_from_cpaas-alb-router-rules_on_ACP.md @@ -0,0 +1,131 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.1.x + - 4.3.x +--- + +# ALB error-rate alert on a route or Ingress from cpaas-alb-router-rules on ACP + +## Issue + +The real-time alert page shows an `error-rate` alert from policy `cpaas-alb-router-rules` (also seen as `cpaas-lb-router-rules`, or called `alb-error-rate`). The text is `error-rate > 3%` sustained for 30 seconds (`error-rate > 3% 且持续 30 秒`), `error-rate > 5%` for 30 seconds, or `error-rate` at 100 percent. The alert names an ALB such as `global-alb2` in `cpaas-system`, and an Ingress or route such as `ingress/cpaas-system/courier-api/global-alb2`. It often clears after a few minutes. + +The ALB Pods run two containers: `nginx` (access log) and `alb2` (configuration-loop log). The `alb_loop_time` alert from `cpaas-alb-rules` and other platform alerts that arrive together with these are covered in the related article. + +Related article: [ALB alb_loop_time alert from cpaas-alb-rules and platform alerts raised with it on ACP](ALB_alb_loop_time_alert_from_cpaas-alb-rules_and_platform_alerts_raised_with_it_on_ACP.md). + +## Root Cause + +### Cause 1: the alert is the error ratio of one route, and the threshold and counted statuses differ by policy + +`cpaas-alb-router-rules` watches the error ratio of each ALB route or Ingress over a 5-minute window. The "30 seconds" is how long the threshold must hold before firing, not the ratio window. The threshold and the statuses counted differ between installations: + +- the threshold can be `0.03` (label `alert_indicator_threshold: "0.03"`) or 5% (the 4.1.5 `global-alb2` policy); +- the 4.1.5 expression counts all 4xx and 5xx (`status=~"[4-5].*"`); other descriptions count 5xx only. + +If 4xx is counted, a wave of `499` or `404` in the access log raises the ratio. 5xx responses are either passed through from the backend, or generated by ALB (`502`, `504`) when the backend is unreachable. + +A route whose 5-minute window contains a single request that the backend answered with HTTP 500 computes an error rate of 100 percent and fires once. For example, `ingress/cpaas-system/razor-api/global-alb2` showed one 100% point that did not recur, and the `razor-api` log showed sporadic `500` responses to `GET /platform/events.alauda.io/v1/...` requests. That sporadic window needs no change. + +Many `cpaas-system` Ingresses firing at once (`dex`, `frontend`, `apollo`, `courier-api`, peaks about 7% to 17%) with no matching 4xx/5xx wave in the ALB `nginx` access log is not a backend fault: no action, no threshold change and no `global-alb2` restart. + +### Cause 2: global-alb2 error-rate > 5% from occasional courier-api 500 responses on 4.1.5 + +On ACP 4.1.5 the alert is `global-alb2` `error-rate > 5%` for 30 seconds (`error-rate > 5%且持续30s`). Per rule, only `ingress/cpaas-system/courier-api/global-alb2` is elevated (maximum 14.29%), while `razor`, `frontend`, `erebus`, `dex`, `base-api` and `archon` stay at 0%. The expression is: + +```promql +sum(rate(nginx_http_status{namespace=~"cpaas-system",pod=~"global-alb2-.*",status=~"[4-5].*",vmcluster="global"}[5m])) / (sum(irate(nginx_http_status{namespace=~"cpaas-system",pod=~"global-alb2-.*",vmcluster="global"}[5m])) + 0.01) +``` + +The 5xx come from one `GET` rule, `global-alb2-00443-`, which matches `STARTS_WITH /platform/monitoring.alauda.io/v1beta1` and routes to Service `courier-api` port 8080. The `courier-api` log shows `500` for `GET /platform/monitoring.alauda.io/v1beta1/projects//clusters//namespaces//variables?regex=...&expr=query_result(kube_namespace_labels{})...`, and the same request in a browser returns `"message": "kube_namespace_labels should has selector namespace="`, `"code": 500`. A namespace-level monitoring dashboard asks for the global metric `kube_namespace_labels{}` without a namespace selector, and the permission check of that API rejects it; a query with `kube_namespace_labels{namespace=""}` returns `200 OK`. These are normal requests, but this policy counts all 4xx and 5xx, so the occasional 500s push the ratio over 5%. + +### Cause 3: error-rate spikes while the logging component is small or being scaled + +`error-rate` (from `cpaas-alb-router-rules`, `source_type: "ingress"`) and `alb_loop_time` (from `cpaas-alb-rules`, for example a trigger value of `5534.00` that recovered within 30 seconds) both move, and the error-rate breakdown concentrates on the logging ingress `ingress/cpaas-system/cpaas-clickhouse/global-alb2` (maximum 61.54% over three days). At the same time `cpaas-statefulset-rules` reports CPU above 90% on the ClickHouse Pods `chi-cpaas-clickhouse-replicated--`. The logging component's resource specification is too small; after it was scaled up, the related error rate stopped rising. The remaining fluctuation is within the supported range. + +## Resolution + +### Cause 1: the alert is the error ratio of one route, and the threshold and counted statuses differ by policy + +1. Read the threshold from the alert text or the `alert_indicator_threshold` label, and the counted statuses from the `status=~` matcher in the rule expression: + + ```bash + kubectl get prometheusrules.monitoring.coreos.com -A -o yaml | grep -n 'nginx_http_status' + ``` + +2. Find the backend from the rule or Ingress name in the alert, and read the ALB access log and the backend log for the same minute: + + ```bash + kubectl get rule -A | grep + kubectl get rule -n -o yaml | grep -A10 service + kubectl logs -n cpaas-system -c nginx --tail=500 + kubectl logs -n deploy/ --tail=200 + ``` + +3. Decide: + - one request and one HTTP 500 in the window: sporadic, no change; + - a sustained 500 in the backend log: the alert reports that backend; fix the backend, and do not raise the threshold; + - ALB lines with `502` or `504`: ALB cannot reach the backend; check the backend Pods and Service endpoints; + - many platform Ingresses firing with no status wave in the `nginx` log: no action. + +### Cause 2: global-alb2 error-rate > 5% from occasional courier-api 500 responses on 4.1.5 + +1. Break the 5xx down by rule in vmui, then read the rule and the backend log: + + ```promql + sort_desc(sum by(rule,method)(nginx_http_status{status=~"[5].*",svc_name="courier-api"})) + ``` + + ```bash + kubectl get rule -A | grep courier-api + kubectl get rule -n cpaas-system -o yaml + kubectl logs -n cpaas-system | grep 500 + ``` + + A rule on `/platform/monitoring.alauda.io/v1beta1`, and `500` lines for `.../variables?...expr=query_result(kube_namespace_labels{})` with the `should has selector namespace` body, is this signature. + +2. Raise the threshold on that alert policy in the console. + + > **Warning:** A higher error-rate threshold also hides later 4xx and 5xx spikes up to the new value, including a backend that is really failing. Precondition: the window is occasional `courier-api` 500 responses on normal requests, as above, and the current threshold is written down. Rollback: set the previous threshold on the same policy. + +Do not raise it when the `nginx` access log has no matching errors (no-action case in Cause 1), or when a backend keeps returning 500. A namespace-level chart that cannot see global series is this permission check, not a missing scrape: query with a namespace matcher, or use the global view. + +### Cause 3: error-rate spikes while the logging component is small or being scaled + +1. Compare the size and load of the logging Pods with the error-rate window: + + ```bash + kubectl get pods -n cpaas-system -o wide | grep -i clickhouse + kubectl top pod -n cpaas-system + ``` + +2. Scale the logging component up through the workload or plugin that owns the `chi-cpaas-clickhouse-replicated-*` Pods. + + > **Warning:** Resizing ClickHouse restarts its replicas; log queries and writes through the affected replica pause until it is Ready. Precondition: its CPU is at the alert threshold during the error-rate window. Rollback: set the previous resources on the same object. + +3. After the scale-up the related error rate stops rising. If the remaining movement stays inside the supported range, the `error-rate` and `alb_loop_time` thresholds can be raised under the same warning as Cause 2. Do not raise them while a backend still returns 500. + +## Diagnostic Steps + +1. Read the policy name: `cpaas-alb-router-rules` or `cpaas-lb-router-rules` with `error-rate` is this article; `cpaas-alb-rules` with `alb_loop_time` is the related article. + +2. Read the threshold and `status=~` matcher from the live rule (Cause 1) instead of assuming 3%, 5% or 5xx only. + +3. Break down the error ratio by Ingress: + + ```promql + sort_desc(sum by (rule) (rate(nginx_http_status{status=~"[4-5].*"}[5m]))) + ``` + + Expected: one rule dominates. A single 100% sample with one 500 in its backend log is Cause 1's sporadic window; `courier-api` with `kube_namespace_labels` 500s is Cause 2; `cpaas-clickhouse` together with ClickHouse CPU alerts is Cause 3. + +4. Check that the ALB Pods did not restart and that users can reach the service: + + ```bash + kubectl get pods -n cpaas-system -o wide | grep -E 'alb2|alb' + ``` diff --git a/docs/en/solutions/Alert_repeat_interval_cannot_be_set_to_4_hours_or_the_global_sending_interval_does_not_take_effect_on_ACP.md b/docs/en/solutions/Alert_repeat_interval_cannot_be_set_to_4_hours_or_the_global_sending_interval_does_not_take_effect_on_ACP.md new file mode 100644 index 000000000..54aebf191 --- /dev/null +++ b/docs/en/solutions/Alert_repeat_interval_cannot_be_set_to_4_hours_or_the_global_sending_interval_does_not_take_effect_on_ACP.md @@ -0,0 +1,85 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.3.x +--- + +# Alert repeat interval cannot be set to 4 hours, or the global sending interval does not take effect on ACP + +## Issue + +One of these is reported about how often alert notifications repeat: + +- The repeat interval should be 4 hours. On an alert policy, **Policy Configuration** (`策略配置`) > **Alert Sending Interval** (`告警发送间隔`) offers **Global** (`全局`) or **Custom** (`自定义`). With Custom, each level (`灾难告警` disaster, `严重告警` critical, `警告告警` warning, `提示告警` info) has a dropdown with 3 hours and 6 hours, and no 4-hour choice. +- The interval should be changed for all alerts at once. +- The sending interval is changed in the UI (for example to 3 hours, or the global interval), and the notifications keep the old rhythm; the change is visible only after editing the configuration from the backend. The VictoriaMetrics Alertmanager configuration is Secret `vmalertmanager-alertmanager` in `cpaas-system`. + +Related article: [Node zombie process alert keeps switching between firing and resolved on ACP](Node_zombie_process_alert_keeps_switching_between_firing_and_resolved_on_ACP.md). + +## Root Cause + +### Cause 1: 4 hours is not one of the offered repeat intervals + +The per-policy dropdown offers 3 hours and 6 hours; a 4-hour repeat interval is not supported. The interval for all alerts is set globally on **Operations Center > Alerts > Alert Settings** (`运维中心 / 告警 / 告警设置`), section **Alert Sending Interval (Global)** (`告警发送间隔 (全局)`). Example values there: disaster 5 minutes, critical 15 minutes, warning 1 hour, info 2 hours. Policies whose sending interval is **Global** use these values; policies set to **Custom** keep their own per-level values. The same page has a separate monitoring-component exception alert section (`监控组件异常告警`) with its own check interval (5 minutes) and notification interval (1 day). + +### Cause 2: the UI interval is not written through to the Alertmanager configuration + +On ACP 4.0 (for example 4.0.11), a sending interval changed in the UI is not written to the `alertmanager.yaml` key of Secret `vmalertmanager-alertmanager`, so Alertmanager keeps the old configuration. This is a known product issue, fixed in ACP 4.2, where the manual procedure below is no longer needed. + +The `warlock` Deployment in `cpaas-system` writes that Secret back. A manual edit therefore has to be made while `warlock` is scaled to 0, and the checksum annotation on ConfigMap `alert-repeat-config` (key `alertmanager.yamMD5`, with that spelling) has to match the file. Any later change of the sending interval in the UI rewrites the Secret and undoes the manual change, so the procedure has to be repeated after each such UI change. + +## Resolution + +### Cause 1: 4 hours is not one of the offered repeat intervals + +Pick 3 hours or 6 hours on each alert policy. To change the interval for all alerts, open **Operations Center > Alerts > Alert Settings**, click **Update** in **Alert Sending Interval (Global)**, set each level, and make sure the policies that should follow it have their sending interval set to **Global**. If the saved global value does not show on the Alertmanager status page, see Cause 2. The backend procedure in Cause 2 does not add a 4-hour choice. + +### Cause 2: the UI interval is not written through to the Alertmanager configuration + +1. Read the current state and keep copies: + + ```bash + kubectl get deploy -n cpaas-system warlock -o jsonpath='{.spec.replicas}{"\n"}' + kubectl get secret -n cpaas-system vmalertmanager-alertmanager -o go-template='{{index .data "alertmanager.yaml"}}' | base64 -d > /tmp/alertmanager.yaml + cp /tmp/alertmanager.yaml /tmp/alertmanager.yaml.bak + kubectl get cm -n cpaas-system alert-repeat-config -o yaml > /tmp/alert-repeat-config.yaml + ``` + +2. Edit `/tmp/alertmanager.yaml` and set the value that should be live, for example `group_interval: 1m`. + +3. Apply it with `warlock` stopped. + + > **Warning:** With `warlock` at 0 replicas the platform does not reconcile alert configuration until it is scaled back, and patching `vmalertmanager-alertmanager` changes the configuration every notification uses. Precondition: the UI value is missing from the Alertmanager status page, `/tmp/alertmanager.yaml.bak` and `/tmp/alert-repeat-config.yaml` exist, and the `warlock` replica count is written down. Rollback: patch the Secret from the backup file with the same command, `kubectl apply -f /tmp/alert-repeat-config.yaml`, and scale `warlock` back. + + ```bash + kubectl scale deploy -n cpaas-system warlock --replicas=0 + kubectl patch secret vmalertmanager-alertmanager -n cpaas-system --type=merge \ + -p "{\"data\":{\"alertmanager.yaml\":\"$(base64 < /tmp/alertmanager.yaml | tr -d '\n')\"}}" + md5sum /tmp/alertmanager.yaml | awk '{print $1}' + kubectl edit cm -n cpaas-system alert-repeat-config + kubectl scale deploy -n cpaas-system warlock --replicas= + ``` + + In `kubectl edit`, set the annotation `alertmanager.yamMD5` to the hash printed by `md5sum` (the hash only, not the file name). + +4. Wait about five minutes, open the Alertmanager status page, and confirm that the value matches the file. If the old value is still shown, the Secret key or the annotation was not the one `warlock` reads; do not patch other keys. + +5. After any later change of the sending interval in the UI, check the status page again and repeat the procedure if the value was reverted. On ACP 4.2 and later, set the interval in the UI only. + +## Diagnostic Steps + +1. A request for 4 hours, with the dropdown showing 3 hours and 6 hours: Cause 1. + +2. Decode the live configuration and compare it with the UI value: + + ```bash + kubectl -n cpaas-system get secret vmalertmanager-alertmanager -o jsonpath='{.data.alertmanager\.yaml}' | base64 -d | grep -n -E 'group_interval|repeat_interval' + ``` + + Expected for Cause 2: the interval saved in the UI does not appear in the file or on the Alertmanager status page. + +3. If notifications for one alert alternate between firing and resolved, the interval is not the problem; see the related article. diff --git a/docs/en/solutions/AppRelease_abnormal_because_a_ModuleInfo_of_a_deleted_cluster_remains_on_ACP.md b/docs/en/solutions/AppRelease_abnormal_because_a_ModuleInfo_of_a_deleted_cluster_remains_on_ACP.md new file mode 100644 index 000000000..62127e142 --- /dev/null +++ b/docs/en/solutions/AppRelease_abnormal_because_a_ModuleInfo_of_a_deleted_cluster_remains_on_ACP.md @@ -0,0 +1,66 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# AppRelease abnormal because a ModuleInfo of a deleted cluster remains on ACP + +## Issue + +An AppRelease (`ars`) is reported abnormal. The cluster it relates to has already been deleted from the platform, but a `ModuleInfo` (`minfo`) for that cluster is still present on the global cluster, for example the `clickhouse-updater` module of the deleted cluster. + +Related article: [Monitoring plugin uninstall or Prometheus to VictoriaMetrics switch leaves ModuleInfo, AppRelease, or Pods on ACP](Monitoring_plugin_uninstall_or_Prometheus_to_VictoriaMetrics_switch_leaves_ModuleInfo_AppRelease_or_Pods_on_ACP.md). + +## Root Cause + +### Cause 1: a ModuleInfo of the deleted cluster was left behind + +A `ModuleInfo` describes a module installed on one cluster; the label `cpaas.io/cluster-name` names that cluster. When the cluster was deleted, the `clickhouse-updater` `minfo` of that cluster was not removed with it. That leftover `minfo` for a cluster that no longer exists makes the AppRelease abnormal. + +## Resolution + +### Cause 1: a ModuleInfo of the deleted cluster was left behind + +1. On the global cluster, confirm that the cluster no longer exists and list the `minfo` objects that still point at it: + + ```bash + kubectl get cls | grep + kubectl get minfo -A -l cpaas.io/cluster-name= + kubectl get minfo -A | grep + ``` + +2. Back up each leftover `minfo` of the deleted cluster, then delete it. + + > **Warning:** Deleting a `ModuleInfo` removes the platform's record of that module installation. Precondition: the cluster named on the `minfo` no longer exists, and the backup file is saved. Do not delete a `minfo` whose cluster still exists. Rollback: `kubectl apply -f /tmp/.yaml`. + + ```bash + kubectl get minfo -o yaml > /tmp/.yaml + kubectl delete minfo + ``` + +3. Check the AppReleases again; the abnormal status clears. + + ```bash + kubectl get ars -A | grep -v -E 'Ready|Synced' + ``` + +## Diagnostic Steps + +1. List the clusters the platform knows (`kubectl get cls`) and compare them with the clusters named on `ModuleInfo` objects: + + ```bash + kubectl get minfo -A -o custom-columns='NAME:.metadata.name,CLUSTER:.metadata.labels.cpaas\.io/cluster-name,MODULE:.metadata.labels.cpaas\.io/module-name' + ``` + + Expected for this cause: one or more `minfo` objects (for example module `clickhouse-updater`) whose cluster is no longer in the cluster list. + +2. List AppReleases that are not healthy: + + ```bash + kubectl get ars -A | grep -v -E 'Ready|Synced' + ``` + + An abnormal AppRelease that relates to the deleted cluster, with its `minfo` still present, is this cause. An abnormal AppRelease for a cluster that still exists is not. diff --git a/docs/en/solutions/Certificate_and_restart_alerts_that_persist_after_an_upgrade_match_business_certificates_or_fail_to_load_on_ACP.md b/docs/en/solutions/Certificate_and_restart_alerts_that_persist_after_an_upgrade_match_business_certificates_or_fail_to_load_on_ACP.md new file mode 100644 index 000000000..006d5a49f --- /dev/null +++ b/docs/en/solutions/Certificate_and_restart_alerts_that_persist_after_an_upgrade_match_business_certificates_or_fail_to_load_on_ACP.md @@ -0,0 +1,176 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Certificate and restart alerts that persist after an upgrade, match business certificates, or fail to load on ACP + +## Issue + +After a platform upgrade, or around certificate monitoring in general, alerts do not behave as expected: + +- Kubernetes certificate alerts keep firing after an upgrade for certificate files with hundreds of negative days of validity, or for certificate Secrets that were already deleted. +- The global cluster keeps reporting `sentry-serving-cert` below `10.00d` for a workload cluster whose machines are powered off. +- Certificate expiry warnings fire 30 days and 10 days ahead (`平台组件证书的到期时间即将到期(小于10天)`), more often than the site wants. +- During a cluster upgrade, restart alerts name business containers that did not restart. +- Platform certificate alerts fire for business (workload) certificates, and the question is whether replacing VictoriaMetrics with Prometheus avoids that. +- The rule `平台组件证书已到期 <= 0d 且持续 1 分钟` (platform component certificate expired) is firing, and its chart shows only `加载失败,请 刷新` (failed to load, please refresh), so nobody can tell which certificate expired. + +## Root Cause + +### Cause 1: cluster-enhancer keeps data for certificates that no longer exist after an upgrade + +The certificate expiry data shows negative validity for files that are no longer in use, for example `/var/lib/kubelet/pki/kubelet.crt` at about `-952.1 d` on several nodes and `/etc/kubernetes/super-admin.conf` at `-16.86 d`, while current certificates such as `kubelet-client-current.pem` and `apiserver-loopback-client` are positive. Deleted certificate Secrets still appear in the real-time alerts. After the upgrade, `cluster-enhancer-controller` and `cluster-enhancer-apiserver` were out of sync and kept data for certificates and Secrets that no longer exist. + +When the VictoriaMetrics UI returns no `certificate_expires_time` series for the deleted object and the alert still shows it, the stale state is in `courier` instead; that variant is Cause 3 of [Monitoring pages stay empty because courier-api or apollo holds a stale cluster cache on ACP](Monitoring_pages_stay_empty_because_courier-api_or_apollo_holds_a_stale_cluster_cache_on_ACP.md). + +### Cause 2: sentry-serving-cert below 10.00d is reported for a powered-off workload cluster + +The global cluster reports `sentry-serving-cert` below `10.00d` for a workload cluster whose nodes are all shut down, and the cluster must be kept. On v4.1.2, with full VictoriaMetrics on both sides, restarting `courier` and `courier-api` on the global cluster cleared it. Why a powered-off cluster was still reported is not known. + +### Cause 3: the expiry warning thresholds do not match the site's requirement + +The certificate policy warns 30 days and 10 days before expiry. That frequency is a configuration choice, not a fault. Sites that want fewer warnings silence the 30-day rule and move the 10-day threshold (for example to 3 days, a site requirement and not a product default). + +### Cause 4: kube-state-metrics restarts split the restart-count series and fake an increase + +The rule `计算组件下容器组最近5分钟的重启次数 > 0 且持续 1 分钟` (Pod restarts in the last 5 minutes > 0, level `提示` info) fires several times within minutes during a cluster upgrade. It queries `cpaas_advanced_kube_pod_container_status_restarts_total` with an expression of this shape: + +```promql +sum by (kind, name, namespace, pod) (round((increase(cpaas_advanced_kube_pod_container_status_restarts_total{kind=~"Deployment",name=~"",namespace=~""}[5m]) + abs(increase(cpaas_advanced_kube_pod_container_status_restarts_total{...}[5m]))) / 2)) +``` + +When kube-state-metrics restarts, the same container gets two series that differ only in `instance` (two kube-state-metrics endpoints on port `8080`), with the same `uid` and value `1`; the first ends where the second begins. `increase(...[5m])` counts the new series' `1` as an increase from 0 and fires, although `restartCount` did not change. The expression has not been changed for this. + +### Cause 5: the certificate alert expression includes business certificates, whatever the backend + +The platform certificate alert expression matches workload certificates as well, which produces platform alerts for business certificates. The expression, not the VictoriaMetrics backend, selects them, so switching to Prometheus monitors the same certificates. + +### Cause 6: the certificate-expired rule chart fails to load because the vmselect query times out + +In a cluster-level policy `证书告警策略` (certificate alert policy), the rule `平台组件证书已到期 <= 0d 且持续 1 分钟` (level `灾难` disaster) is firing, while the 30-day and 10-day rules in the same policy are normal. The rule's expression covers every certificate in the cluster, and its query exceeds the `vmselect` query time limits, so the chart shows `加载失败,请 刷新`. A similar 4.1 symptom is Cause 3 of [Certificates tab missing, platform domain certificate not listed, or certificate alert shows no data on ACP](Certificates_tab_missing_platform_domain_certificate_not_listed_or_certificate_alert_shows_no_data_on_ACP.md). + +## Resolution + +### Cause 1: cluster-enhancer keeps data for certificates that no longer exist after an upgrade + +1. Confirm that the files with negative validity are not the certificates the node uses; a certificate that really expired needs renewal, not a restart: + + ```bash + openssl x509 -noout -enddate -in /var/lib/kubelet/pki/kubelet-client-current.pem + ls -l /var/lib/kubelet/pki/ + kubectl get secret -n + ``` + +2. Run the alert expression in the VictoriaMetrics UI. A `certificate_expires_time` series still returned for the deleted object is this cause. + +3. In the affected cluster, restart both cluster-enhancer components. + + > **Warning:** Restarting `cluster-enhancer-controller` and `cluster-enhancer-apiserver` drops their endpoints for one start; certificate monitoring for that cluster pauses and a metrics-availability alert can fire once. Precondition: the alerts name certificates or Secrets that no longer exist, and the series is still returned. Rollback is the replacement Pods becoming Ready. + + ```bash + kubectl get pods -A | grep cluster-enhancer + kubectl delete pod -n + ``` + +The Kubernetes certificate alerts, including the rows for deleted Secrets, return to normal. + +### Cause 2: sentry-serving-cert below 10.00d is reported for a powered-off workload cluster + +Keep the workload cluster, and on the global cluster restart `courier` and `courier-api`: + +> **Warning:** Restarting both pauses notification sends and the active-alert scrape until the new Pods are Ready. Precondition: the row is `sentry-serving-cert` below `10.00d` for a workload cluster whose nodes are powered off, and you are on the global cluster. Rollback is the new Pods becoming Ready. + +```bash +kubectl get pods -n cpaas-system | grep courier +kubectl delete pod -n cpaas-system +``` + +The row clears. If it returns, the cause is still open; do not delete the workload cluster for it. + +### Cause 3: the expiry warning thresholds do not match the site's requirement + +Find the certificate policy by its rule text, then, on that policy under **Operations Center > Alerts > Alert Policies**, silence the 30-day rule and change the nearer threshold to the value the site requires: + +```bash +kubectl get alertrule -A | grep -i certificate +``` + +These changes reduce frequency only; they do not reduce query load. + +### Cause 4: kube-state-metrics restarts split the restart-count series and fake an increase + +1. Compare `restartCount` with the alert: + + ```bash + kubectl get pod -n -o jsonpath='{.status.containerStatuses[*].restartCount}{"\n"}' + ``` + +2. Query the raw series in the VictoriaMetrics UI over the alert window: + + ```promql + cpaas_advanced_kube_pod_container_status_restarts_total{namespace="",pod=""} + ``` + + Two series with the same `uid` and value, one ending where the other starts, with different `instance` values, is this cause. + +3. Treat a restart alert in that kube-state-metrics window as a false positive when `restartCount` did not increase. No change is needed on the workload. + +### Cause 5: the certificate alert expression includes business certificates, whatever the backend + +Do not replace VictoriaMetrics with Prometheus to fix this. Find which certificate the alert names: + +```bash +kubectl get secret -A --field-selector type=kubernetes.io/tls | grep +``` + +A business certificate that is really near expiry is for its owner to renew. To stop the cluster-level policy from alerting on business certificates, disable it and rely on the general certificate policy from the `global` cluster, as in Cause 6. Built-in rules cannot be edited in place. + +### Cause 6: the certificate-expired rule chart fails to load because the vmselect query times out + +1. Read the current `vmselect` arguments and log on the cluster whose chart fails: + + ```bash + kubectl get vmclusters -A + kubectl get vmclusters -n cpaas-system -o jsonpath='{.spec.vmselect.extraArgs}{"\n"}' + kubectl logs -n cpaas-system vmselect-cluster-0 --tail=200 | grep -i -E 'timeout|deadline|maxQueryDuration|maxQueueDuration' + ``` + +2. Raise `search.maxQueryDuration` and `search.maxQueueDuration` (VictoriaMetrics `vmselect` flags; upstream defaults 30 seconds and 10 seconds) under `spec.vmselect.extraArgs`. + + > **Warning:** The operator rolls the `vmselect-cluster` Pods; queries and the monitoring and alert pages stall until they are Ready. Longer limits also let heavy queries hold `vmselect` CPU and memory longer. Precondition: the chart shows `加载失败,请 刷新` and the query times out, and the object is exported to `/tmp/vmcluster.yaml`. Rollback: `kubectl apply -f /tmp/vmcluster.yaml`. A plugin reconcile or upgrade can write the plugin value back. + + ```bash + kubectl get vmclusters -n cpaas-system -o yaml > /tmp/vmcluster.yaml + kubectl patch vmclusters -n cpaas-system --type=merge \ + -p '{"spec":{"vmselect":{"extraArgs":{"search.maxQueryDuration":"","search.maxQueueDuration":""}}}}' + ``` + +3. Reload the rule chart and read which certificate is at or below 0 days. Renew a certificate that has really expired. + +4. Disable the cluster-wide certificate policy from its action menu under **Operations Center > Alerts > Alert Policies**, and make sure the general certificate policy of the `global` cluster covers the platform certificates. + +## Diagnostic Steps + +1. After an upgrade, certificate alerts that name files with large negative validity or Secrets that do not exist: check the object and the series (Cause 1): + + ```bash + kubectl get secret -n + kubectl get pods -A | grep cluster-enhancer + ``` + + Expected for Cause 1: the Secret is `NotFound` and the VictoriaMetrics UI still returns its series. + +2. `sentry-serving-cert` below `10.00d` for a cluster whose nodes are all powered off: Cause 2. + +3. 30-day and 10-day warnings that are too frequent for the site: Cause 3. + +4. A restart alert during an upgrade with unchanged `restartCount` and two series that differ only in `instance`: Cause 4. + +5. A certificate alert that names a business TLS Secret: Cause 5. A certificate rule chart with `加载失败,请 刷新`: Cause 6. diff --git a/docs/en/solutions/Console_returns_404_shows_old_pages_or_misses_menus_after_an_upgrade_on_ACP.md b/docs/en/solutions/Console_returns_404_shows_old_pages_or_misses_menus_after_an_upgrade_on_ACP.md new file mode 100644 index 000000000..f12ea865b --- /dev/null +++ b/docs/en/solutions/Console_returns_404_shows_old_pages_or_misses_menus_after_an_upgrade_on_ACP.md @@ -0,0 +1,163 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Console returns 404, shows old pages, or misses menus after an upgrade on ACP + +## Issue + +After a platform upgrade, or on a particular version or package, the console shell does not match what is expected, in every browser: + +- requests to the base platform services return HTTP 404 while the API Pods are Running (seen on 4.0.9); +- pages mix old and new views after an upgrade from 4.0 to 4.3.2, and the `frontend` image is back on the 4.0 tag; +- the login page is blank right after images were updated as a vulnerability fix; +- the DevOps entry is missing from Platform Management and Container Management; +- Backup Management has no etcd backup page on 4.1, the etcd monitor panel is empty on 4.2 and later, or License Management is missing; +- on bare metal, the network page cannot create MetalLB, and the Kube-OVN subnet, external address pool and VLAN sections are missing; +- a ConfigMap value entered with line breaks becomes one line in YAML, and the mounted file has Windows-style line breaks; +- new jobs are not visible in the job list after an upgrade on 3.18. + +Related article: [Console white page, clipped menu, dead Create button, or empty Pod logs in one browser on ACP](Console_white_page_clipped_menu_dead_Create_button_or_empty_Pod_logs_in_one_browser_on_ACP.md). + +## Root Cause + +### Cause 1: sentry restored the frontend image to the pre-upgrade tag + +After an upgrade from 4.0 to 4.3.2 the `frontend` image was back on the 4.0 tag; the audit log showed that `sentry` had restored it. Until the image matches the platform, pages mix old static files with the new API. + +### Cause 2: the frontend image configuration does not match the upgraded platform + +After an upgrade to 4.0.9, base platform services returned HTTP 404 because the `frontend` image configuration was wrong. Pods are Running while the image they run is not the one for the upgraded platform. + +### Cause 3: images updated as a vulnerability fix break the login page + +After images were replaced as a vulnerability fix, the login page is blank and nobody can log in. Reverting those image updates restores login. + +### Cause 4: DevOps is missing on the global cluster or has the wrong version there + +The DevOps entry is read from the global cluster. DevOps installed only on a business cluster never shows the entry; it appears once DevOps is deployed on global and a Katanomi instance is created there. After global moves from 3.18 to 4.0 while `katanomi` stays on 3.18, the package for the new global version was never pushed, and both Platform Management and Container Management hide DevOps. + +### Cause 5: the menu was removed or not shipped in this version or package + +- 4.1 removed the etcd backup page from Backup Management. Backups are still visible through the API, and schedules are changed on the `etcdbackupconfiguration` object. +- From 4.2 the 4.0 etcd monitor panel is discontinued, so its charts stay empty even when etcd is healthy. +- The overseas 4.3.2 package omits License Management, including the menu entry and the path `/console-platform/manage-platform/platform-setting/license`; the China distribution has it. + +### Cause 6: the ProviderBaremetal adaptation is incomplete before 4.3.3 + +On bare metal, `Artifact` and `ArtifactVersion` for `metallb-operator` exist and marketplace logs are clean, but `console-plugins` returns no subnet payload and the cluster overview has no network-mode field. The `ProviderBaremetal` adaptation is incomplete in that build and is fixed in 4.3.3. + +### Cause 7: the ConfigMap form converts line breaks + +The ConfigMap form's character conversion turns a multi-line value into one line, and the mounted file then has Windows-style line breaks. The YAML editor keeps the breaks. The form behaves correctly from 4.1.4. + +### Cause 8: the 3.18 job list shows only the first 20 jobs + +When more than 20 jobs exist, the 3.18 list page stops at 20. The other jobs exist in the API and search finds them. 4.0 changed the list. + +## Resolution + +### Cause 1: sentry restored the frontend image to the pre-upgrade tag + +```bash +kubectl get deploy -A | grep frontend +kubectl get deploy -n \ + -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" "}{.image}{"\n"}{end}' +``` + +> **Warning:** Changing the `frontend` image rolls the console shell; users lose the page until the new Pod is Ready. Precondition: the running tag is older than the upgraded platform. Rollback: set the previous image again, which brings back the mismatched UI. + +Set the container image to the tag that belongs to the upgraded platform, then watch whether `sentry` reverts it. If it keeps writing the old tag back, the release that `sentry` reconciles still carries the old tag, and a one-off edit will not stick; check the audit log and that release before editing again. + +### Cause 2: the frontend image configuration does not match the upgraded platform + +```bash +kubectl get deploy -n cpaas-system -o custom-columns='NAME:.metadata.name,IMAGE:.spec.template.spec.containers[0].image' | grep frontend +``` + +> **Warning:** Setting a new `frontend` image rolls the console shell; users lose the page until the new Pod is Ready. Precondition: base services return 404 after an upgrade, and this image does not match that upgrade. Rollback: set the previous image. + +```bash +kubectl set image deployment/ -n cpaas-system = +kubectl rollout status deployment/ -n cpaas-system +``` + +Requests stop returning 404 once the running image matches. If `sentry` writes the old image back, see Cause 1. + +### Cause 3: images updated as a vulnerability fix break the login page + +> **Warning:** Reverting the vulnerability-fix images puts the previous tags back and re-opens what those updates closed. Precondition: the blank login page started with that image update. Keep the list of tags you revert. Re-applying the fix tags is the rollback, and it can blank the login page again; obtain a supported image set before re-applying the fix. + +Revert each image that was changed as part of the vulnerability fix to its previous tag, using the same method that applied it. Login returns. + +### Cause 4: DevOps is missing on the global cluster or has the wrong version there + +Install DevOps on the global cluster and create the Katanomi instance there, not only on the business cluster. After a global upgrade, push the DevOps package that belongs to the new global version and upgrade the component so the `katanomi` version matches. The Katanomi instance fields are described in Cause 2 of [DevOps v3 toolchain entry missing after an upgrade: Katanomi MonitorDashboard ownership conflict or missing instance](DevOps_v3_toolchain_entry_missing_after_an_upgrade_Katanomi_MonitorDashboard_ownership_conflict_or_missing_instance.md). + +### Cause 5: the menu was removed or not shipped in this version or package + +- etcd backups on 4.1: use the API, and edit the object when a schedule must change: + + ```bash + kubectl get etcdbackupconfiguration -A + kubectl edit etcdbackupconfiguration -n + ``` + +- etcd monitor panel on 4.2 and later: use the monitoring dashboard shipped with the running version instead of the 4.0 panel; do not repair the monitoring stack for it. +- License Management: check which distribution is installed. Getting the feature means installing a distribution that contains it; reinstalling the console does not add it. + +### Cause 6: the ProviderBaremetal adaptation is incomplete before 4.3.3 + +Confirm that the `metallb-operator` artifacts are present, so it is not a missing upload, then upgrade to 4.3.3 or later. The subnet, external address pool and VLAN sections and MetalLB creation return on the fixed build. + +```bash +kubectl get artifact,artifactversion -A | grep metallb +``` + +### Cause 7: the ConfigMap form converts line breaks + +Open the ConfigMap, switch to **YAML**, enter the value with real line breaks (for example as a `|` block), and apply. Mount the key again and confirm that the file has no Windows-style breaks: + +```bash +kubectl exec -n -- cat -A | head +``` + +`^M` before `$` at line ends shows Windows-style breaks. From 4.1.4 the form keeps the breaks. + +### Cause 8: the 3.18 job list shows only the first 20 jobs + +Search for the job by name before treating it as a failed create; search opens jobs beyond the first 20. Upgrading to 4.0 or later removes that list limit. + +## Diagnostic Steps + +1. Read the images and their tags, and compare them with the platform version: + + ```bash + kubectl get pods -n cpaas-system | grep -E 'frontend|archon' + kubectl get deploy -A -o wide | grep -E 'frontend|archon' + ``` + +2. Map the result: + + ```text + frontend tag older than the platform, sentry wrote it back -> Cause 1 + base services return 404 after an upgrade -> Cause 2 + blank login page right after a vulnerability image update -> Cause 3 + DevOps entry absent, katanomi old or only on a business cluster -> Cause 4 + no etcd backup page on 4.1, no License Management overseas -> Cause 5 + MetalLB and subnet pages missing on bare metal before 4.3.3 -> Cause 6 + ConfigMap value collapsed to one line after a form edit -> Cause 7 + new jobs missing from the 3.18 list, search finds them -> Cause 8 + only one browser affected -> related article + ``` + +3. A missing **EXEC** entry or another menu behind a feature gate is [Console menu or feature is missing because a feature gate, plugin, or product entry is off on ACP](Console_menu_or_feature_is_missing_because_a_feature_gate_plugin_or_product_entry_is_off_on_ACP.md). diff --git a/docs/en/solutions/Console_white_page_clipped_menu_dead_Create_button_or_empty_Pod_logs_in_one_browser_on_ACP.md b/docs/en/solutions/Console_white_page_clipped_menu_dead_Create_button_or_empty_Pod_logs_in_one_browser_on_ACP.md new file mode 100644 index 000000000..b0f0862eb --- /dev/null +++ b/docs/en/solutions/Console_white_page_clipped_menu_dead_Create_button_or_empty_Pod_logs_in_one_browser_on_ACP.md @@ -0,0 +1,113 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.16.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x + - 4.4.x +--- + +# Console white page, clipped menu, dead Create button, or empty Pod logs in one browser on ACP + +## Issue + +The platform API is up and the `cpaas-system` Pods are Running, but in one browser or on one PC the console misbehaves: + +- a white page, a partly drawn administrator menu, or the container platform UI does not open while another management UI on the same address does; +- **Create** for an Ingress or gateway does nothing, with no error in the browser console; +- the login page at `/console-portal` stays blank with `证书错误` (certificate error) in the address bar; +- EXEC and DEBUG open only a `文件传输` (File Transfer) button, or a `platform.cfc` download fails; +- every Pod log view is empty, and entering EXEC reports `连接失败` (connection failed); +- the console is very slow and needs manual refreshes in one browser; +- monitoring charts are blank on one PC only. + +Another browser, a private window, or another PC works. If every browser fails and the browser developer tools (F12) show HTTP 500 or 502, the problem is on the server side, for example [Console returns HTTP 500 or a blank page because apollo or courier on the global cluster fails on ACP](Console_returns_HTTP_500_or_a_blank_page_because_apollo_or_courier_on_the_global_cluster_fails_on_ACP.md). + +Related articles: [Console returns 404, shows old pages, or misses menus after an upgrade on ACP](Console_returns_404_shows_old_pages_or_misses_menus_after_an_upgrade_on_ACP.md), [Debug container blocked or left running, and console file upload or download fails on ACP](Debug_container_blocked_or_left_running_and_console_file_upload_or_download_fails_on_ACP.md). + +## Root Cause + +### Cause 1: stale browser cache or an expired idle session + +The browser holds cached static files or a session that expired while the tab was idle. Logging out and in restores a clipped menu; a private window loads the container platform at once; a different browser opens the page. There is no server-side fault. + +### Cause 2: the browser is older than the platform version supports + +The supported browser versions depend on the platform version; check the browser compatibility list in the documentation of the installed version. An older build causes client-side failures without a server error: + +- **Create** does nothing, also in a private window of the same old browser; +- a `platform.cfc` download fails on 4.3.1 in a browser below Chrome 145, while Edge succeeds; +- EXEC and DEBUG show only a **File Transfer** button in a UnionTech (UOS) browser 5.4.38 based on Chromium `83.0.4103.116`; +- the login page stays blank with `证书错误` for a `kube-ca`-issued certificate in an old browser, and displays in a newer version; +- the console is very slow in a Firefox build that is not in the validated set, while Google Chrome on the same client loads quickly; API timings in the browser trace are fast (the slowest XHR about 152 ms) and server-side checks are clean. + +### Cause 3: the PC clock is wrong + +Monitoring charts are blank on one PC while the same OS and browser draw them elsewhere, and installing another browser does not help. The query time range is likely built from the client clock. This is described in Cause 4 of [Monitoring dashboard panels are blank or drop to zero while the backend has data on ACP](Monitoring_dashboard_panels_are_blank_or_drop_to_zero_while_the_backend_has_data_on_ACP.md). + +### Cause 4: Pod logs are empty and EXEC fails because the browser is below the 4.1 requirement + +Every Pod log view is empty, including start-up and error logs, and EXEC reports a connection failure, although the application writes to standard output. On 4.1 the browser requirement includes Firefox 140 or later; a lower Firefox version produces exactly this, and both views work after the browser is upgraded. + +## Resolution + +### Cause 1: stale browser cache or an expired idle session + +1. Sign out and sign in again, then try a private window. +2. If the private window works, clear the site data for the platform address only. +3. If a second browser works and the first does not, reset or replace the broken browser profile. + +No platform Pod restart is needed when F12 shows no 5xx and `cpaas-system` is Running. + +### Cause 2: the browser is older than the platform version supports + +1. Read the browser version and its engine version on its **About** page and compare it with the compatibility list of the installed platform version. +2. Move the user to a current Google Chrome (or another browser from the list) and retry. +3. For a slow console, capture a Network trace in the slow browser and in Chrome on the same client; fast API timings and a fast Chrome load point at the browser, not the platform. + +If **Create** works but a new Layer 4 Ingress gets no address, check whether its host port (for example `32320`) is already used by another Layer 4 rule; a Layer 4 port cannot be shared, so no second address is allocated. That is not a browser fault. + +### Cause 3: the PC clock is wrong + +Compare the PC clock and time zone with a PC that renders the charts, correct them, and reload the monitoring view. Cluster NTP and the monitoring stack need no change. + +### Cause 4: Pod logs are empty and EXEC fails because the browser is below the 4.1 requirement + +1. Confirm that the application has standard output: + + ```bash + kubectl logs -n -c --tail=20 + ``` + +2. Check the browser version on its **About** page. On 4.1, Firefox below 140 matches this cause. +3. Upgrade the browser and reopen the page; logs and EXEC display normally. + +## Diagnostic Steps + +1. Confirm the server side is healthy: + + ```bash + kubectl get pods -n cpaas-system | grep -E 'frontend|archon|apollo' + ``` + + Expected: all Running. Then in the browser open F12 > **Network** and reload once. + +2. Map the result: + + ```text + private window or another browser works, no 5xx -> Cause 1 + Create is a dead click, platform.cfc fails, only File Transfer shown -> Cause 2 + blank login page with 证书错误 in an old browser only -> Cause 2 + console slow in one browser, fast in Chrome on the same client -> Cause 2 + charts blank on one PC only -> Cause 3 + all Pod logs empty, EXEC connection fails, kubectl logs has output -> Cause 4 + every browser fails, F12 shows 500 or 502 -> not this article + ``` + +3. Sign-in redirects and 401 from the identity provider are an authentication problem. A 403 with a clear `forbidden` message on one API is authorization, not a browser problem. diff --git a/docs/en/solutions/Debug_container_blocked_or_left_running_and_console_file_upload_or_download_fails_on_ACP.md b/docs/en/solutions/Debug_container_blocked_or_left_running_and_console_file_upload_or_download_fails_on_ACP.md new file mode 100644 index 000000000..b1f4c51eb --- /dev/null +++ b/docs/en/solutions/Debug_container_blocked_or_left_running_and_console_file_upload_or_download_fails_on_ACP.md @@ -0,0 +1,186 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x +--- + +# Debug container blocked or left running, and console file upload or download fails on ACP + +## Issue + +The console opens and the workload is healthy, but the debug function or the console file transfer of a container does not work: + +- Enabling debug shows `forbidden by security policy`. +- A debug container of an application has existed for days and has not exited. +- Exec into a debug container keeps loading and never opens, with no error. +- Uploading a large file through **File Transfer** fails, and the browser shows `413 too large`. +- Uploading a large file or a folder through the page, or downloading a file larger than 1G, fails. +- Downloading a file from a container fails, and the browser says to check the internet connection (`请检查互联网连接状况`). It fails for some containers only, or for every container. + +Console file transfer runs `tar` inside the target container through exec. An upload runs `kubectl exec -i -n -c -- tar -xmf - -C `; no local `tar` or `kubectl` is started. Web CLI failures, a debug image pull that fails, and a greyed-out **File Transfer** button are covered in [Web terminal, exec window, debug container, or console file transfer fails on ACP](Web_terminal_exec_window_debug_container_or_console_file_transfer_fails_on_ACP.md). + +## Root Cause + +### Cause 1: Pod Security Admission on the namespace blocks the debug container + +The namespace carries a Pod Security Admission (PSA) enforce level, and the debug container that the platform adds does not meet it, so its creation is rejected with `forbidden by security policy`. An unprivileged debug mode that fits the namespace PSA level, and copy-based debugging, are not available in these versions. + +### Cause 2: debug containers are not cleaned up automatically + +The debug function has no automatic cleanup. A debug container stays in the Pod until the business container is restarted. + +### Cause 3: host security software intercepts exec into the debug container + +Exec into the debug container never finishes loading and shows no error; the WebSocket (`wss`) connection is fine. An old version of host security software on the node intercepts exec and similar requests. + +### Cause 4: an nginx in front of the platform limits the request body + +Uploading through the platform domain returns `413 too large`, while uploading the same file through the global VIP address succeeds. An nginx operated by the site in front of the global VIP limits the request body size. The `archon-api` container of the `archon` Pod in `cpaas-system` logs the upload form fields (`form name: file`, `form name: dest`, `form name: size`) and no size error; other lines in that log (`Failed to get in-cluster client builder: Unauthorized`, `Get autoscaler with workload ... not found`, `find apiResource for kind error: ... skip`) are unrelated. + +### Cause 5: page transfers of large files and folders are limited + +The page exec channel has a size ceiling, and transfers through the page depend on several link-layer factors such as the browser and nginx; a download of more than 1G failed on 3.18. The page has also not been shown to upload a directory. Large transfers through the page are not recommended. + +### Cause 6: the archon image is older than the download path + +The console cannot download any file from any container, although the file exists. The `archon` image predates the download path, so restarting `archon` does not help. + +### Cause 7: the tar in the container image fails with "Function not implemented" + +Downloads fail for some containers only, including small files such as `/etc/adduser.conf`. The platform log for the transfer shows: + +```text +Handle request error: read source file error: tar: Removing leading '/' from member names +tar: /etc/adduser.conf: Cannot stat: Function not implemented +tar: Exiting with failure status due to previous errors +Websocket Ping failed: ... write: broken pipe +``` + +`kubectl cp` from the same container fails with the same `Cannot stat: Function not implemented`. Running `tar -cf - -- /etc/gai.conf >/dev/null` inside that container fails with exit code `2`, while in another container on the same kind of node (same `tar (GNU tar) 1.35` version) it succeeds with exit code `0`. The custom-built business image cannot execute the `tar` command that file transfer relies on. `Function not implemented` is the Linux `ENOSYS` error: a system call used by that image's `tar` or its libraries is not available to the container on that node. + +## Resolution + +### Cause 1: Pod Security Admission on the namespace blocks the debug container + +Confirm the PSA labels and the rejection: + +```bash +kubectl get ns -o jsonpath='{.metadata.labels}' | tr ',' '\n' | grep pod-security +kubectl get events -n --sort-by=.lastTimestamp | grep -i -E 'forbidden|PodSecurity' | tail +``` + +A `pod-security.kubernetes.io/enforce` level such as `restricted` or `baseline`, together with `forbidden by security policy`, is this cause. Debug works only in namespaces whose PSA level admits the debug container; if the namespace policy must stay as it is, debug is not available there. + +### Cause 2: debug containers are not cleaned up automatically + +1. Find the Pod that still carries the debug container: + + ```bash + kubectl get pod -n -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{end}{range .spec.ephemeralContainers[*]}{.name}{" (ephemeral)\n"}{end}' + ``` + +2. Restart the workload to remove it. + + > **Warning:** Restarting replaces the Pods; with one replica the service is unavailable until the new Pod is Ready. Precondition: the leftover debug container must go, and a restart is acceptable in this window. The restart cannot be undone, and the business container keeps running without it. + + ```bash + kubectl rollout restart deployment -n + kubectl rollout status deployment -n + ``` + +### Cause 3: host security software intercepts exec into the debug container + +1. Rule out the console path: if `kubectl exec` from a control-plane node into the same container also hangs, the request is stopped below the console. + + ```bash + kubectl exec -n -c -- sh -c 'echo ok' + kubectl get pod -n -o jsonpath='{.spec.nodeName}{"\n"}' + ``` + +2. On that node, check the host security agent's console or logs for blocked exec or process-creation events, and ask its owner to stop intercepting the exec requests (for example by upgrading the agent). Retry exec afterwards. + +### Cause 4: an nginx in front of the platform limits the request body + +1. Upload the same file through the platform domain and through the global VIP address. `413` on the domain and success on the VIP is this cause. + + ```bash + kubectl logs -n cpaas-system -c archon-api --tail=200 | grep -E 'form name|size' + ``` + +2. Raise the request-body limit on the proxy in front of the global VIP. For nginx that is `client_max_body_size`; set it above the largest file you need to transfer. + + > **Warning:** Reloading that nginx applies the change to all traffic through it, and a syntax error stops it from serving. A larger limit also lets clients send larger requests to every site behind it. Precondition: a copy of the old configuration. Rollback: restore the copy and reload. + + ```bash + grep -rn client_max_body_size /etc/nginx/ + nginx -t && nginx -s reload + ``` + +### Cause 5: page transfers of large files and folders are limited + +Copy large files with `kubectl cp` from a host that can reach the cluster API: + +```bash +kubectl cp /:/path/in/container/ ./ -c +kubectl cp ./ /:/path/in/container -c +ls -l ./ +``` + +For a directory, archive it first, copy the archive, and extract it inside the container. `kubectl cp` also uses `tar` inside the container (see Cause 7). + +### Cause 6: the archon image is older than the download path + +```bash +kubectl get pods -A | grep archon +kubectl get deploy -A -o wide | grep archon +kubectl logs -n --tail=100 +``` + +> **Warning:** Changing the `archon` image restarts console downloads and the exec helpers it serves until the new Pod is Ready. Precondition: the running image is older than the one shipped with the platform version; use that shipped image, not an arbitrary newer tag. Rollback: set the previous image. + +Set the `archon` image to the one that ships with the platform version, wait until the Pod is Ready, and download a small file from a container to confirm. + +### Cause 7: the tar in the container image fails with "Function not implemented" + +1. Reproduce inside the failing container: + + ```bash + kubectl exec -n -c -- tar --version + kubectl exec -n -c -- sh -c 'tar -cf - -- /etc/hostname >/dev/null; echo exit=$?' + ``` + + `Cannot stat: Function not implemented` and `exit=2` confirm the cause. Run the same in a container where downloads work for comparison. + +2. Fix the image: rebuild the business image with a `tar` (and libraries) that runs on the cluster nodes, and roll the workload to the new image. Neither the console nor `archon` needs a change. + +3. Until the image is fixed, a single file can be read without `tar`: + + ```bash + kubectl exec -n -c -- cat /path/in/container/ > ./ + ``` + +## Diagnostic Steps + +```bash +kubectl get ns --show-labels +kubectl get pod -n -o yaml | grep -n -E 'ephemeralContainers|container-debug' +kubectl get pods -n cpaas-system | grep archon +``` + +```text +debug says forbidden by security policy, namespace has a pod-security enforce label -> Cause 1 +debug container still present days later -> Cause 2 +debug exec keeps loading, kubectl exec from a node also hangs -> Cause 3 +upload returns 413 through the domain, succeeds through the VIP -> Cause 4 +large file (over 1G) or folder through the page fails, kubectl cp works -> Cause 5 +every download from every container fails, archon image older than the platform -> Cause 6 +download fails for some containers, tar in that container: Function not implemented -> Cause 7 +``` + +To tell Cause 7 from the others, run `kubectl cp` for the same file: it fails the same way only for Cause 7. diff --git a/docs/en/solutions/External_Grafana_or_scripts_cannot_query_platform_metrics_vmselect_address_credentials_and_vmui_errors_on_ACP.md b/docs/en/solutions/External_Grafana_or_scripts_cannot_query_platform_metrics_vmselect_address_credentials_and_vmui_errors_on_ACP.md new file mode 100644 index 000000000..824ae65cf --- /dev/null +++ b/docs/en/solutions/External_Grafana_or_scripts_cannot_query_platform_metrics_vmselect_address_credentials_and_vmui_errors_on_ACP.md @@ -0,0 +1,177 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.8.x + - 3.10.x + - 3.12.x + - 3.14.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# External Grafana or scripts cannot query platform metrics: vmselect address, credentials, and vmui errors on ACP + +## Issue + +An external Grafana, a self-built monitoring platform such as Nightingale, a big-data platform, or a `curl` script must read the metrics that the platform monitoring stores, and one of these happens: + +- Nobody can find the query address or the username and password. VMUI shows a query page but no connection string. +- `curl` to `vmselect` returns `200 OK` with the platform login page (`Product Login`, ``) instead of JSON. +- `curl` with basic auth returns `HTTP/1.1 200 OK`, `Content-Length: 0`, a `Gap-Auth: ` header and an empty body, or an export from `/select//prometheus/api/v1/export` returns nothing. +- vmui fails every query with `Error executing query: Unexpected token 'u', "unsupporte"... is not valid JSON. Please check your serverURL settings and confirm server availability.` or `SyntaxError: Unexpected token 'r', "remoteAddr"... is not valid JSON`. +- vmui opened through a NAT-mapped address redirects the browser to the platform's internal address, which the user's network cannot reach. +- The HAMi web UI stays empty although its Pod and Prometheus are Running. +- Someone asks for a login-free or NodePort address for Prometheus. + +On VictoriaMetrics clusters the query endpoint is `vmselect`, which accepts the Prometheus query API. On Prometheus clusters it is the Prometheus instance behind `/clusters//prometheus-0`. Both are protected by basic auth or the platform login. + +Related articles: [Grafana returns HTTP 500 after a password change, is missing after an upgrade, or rules and alerts do not carry over on ACP](Grafana_returns_HTTP_500_after_a_password_change_is_missing_after_an_upgrade_or_rules_and_alerts_do_not_carry_over.md). Credentials for the platform API and the 4.2 node-exporter change are in [auth-controller2 webhook certificate missing or log burst, monitoring query credentials, and Connector tokens on ACP](auth-controller2_webhook_certificate_missing_or_log_burst_monitoring_query_credentials_and_Connector_tokens_on_ACP.md). + +## Root Cause + +### Cause 1: the query address and the basic-auth Secret are on the monitoring feature, not on the VMUI page + +VMUI is a browser on top of `vmselect` and does not print a connection string. On the global cluster, the `feature` object named `monitoring` carries `spec.accessInfo.database`: + +- `instanceType: victoriametrics` (or Prometheus) +- `address`: the external form `https:///clusters//vmselect`; this is what an external Grafana uses +- `service`: the in-cluster form, for example `http://vmselect:8481/select/10001/prometheus` +- `basicAuth.secretName: victoriametrics-basic-auth` + +The credentials are in Secret `victoriametrics-basic-auth` in `cpaas-system`. Credentials copied from another system do not log in. A request without credentials is redirected to the platform login page, so it returns `200 OK` with HTML. The tenant segment in `/select//prometheus` must be the one the cluster writes to; a mistyped tenant (for example `10010` instead of `10001`) returns an empty result rather than an error. + +The built-in Prometheus of a Prometheus cluster is reachable at `https:///clusters//prometheus-0`, with the credentials in Secret `kube-prometheus-prometheus-basic-auth` in `cpaas-system` (user `monitor@cpaas.io`). + +### Cause 2: the vmui server URL ends in prometheus/ + +A vmui page that worked before now fails every query with `Unexpected token 'u', "unsupporte"... is not valid JSON`. The server URL in vmui settings ends in `prometheus/` (for example `/clusters/global/vmselect/prometheus/`), so queries go to a path that answers with a plain-text `unsupported path` body. The first visit to `/clusters//vmselect/vmui/?` can redirect to the Prometheus path and leave that suffix in the setting. The `"remoteAddr"... is not valid JSON` variant has the same cause. + +### Cause 3: vmui through a NAT-mapped address redirects to an unreachable internal address + +vmui login is SSO with the platform Dex as identity provider. The Dex callback address is fixed in the start-up arguments of the VictoriaMetrics auth proxy and is bound to the platform address given when the environment was created. On a platform deployed by IP, a browser that reaches the platform only through a NAT-mapped address follows the redirect to `https:///dex/auth?...client_id=alauda-auth&redirect_uri=...vmselect%2Foauth2%2Fcallback...` and fails to connect. The routes themselves are healthy: on global, ingress `cluster-proxy-` sends `/clusters/(.|$)(.*)` to service `cluster-proxy-` port `11780`, and on the workload cluster ingress `vmselect` sends `/clusters//vmselect($|/)(.*)` to service `vmselect` port `8481`. Opening `/clusters//vmselect/select/10001/vmui/` directly returns `unsupported path requested: "/select/10001/prometheus/select/10001/vmui/"`; that path is not the vmui entry. + +### Cause 4: the HAMi web UI has the wrong Prometheus URL or credentials + +`hamiwebui` and the Prometheus Pod are Running, and the in-cluster Prometheus URL returns data. The address, or the username and password, entered in the HAMi UI installation do not match that endpoint, so the HAMi page stays empty. + +### Cause 5: Prometheus has no login-free access + +Prometheus authentication is enforced by the platform and cannot be bypassed; there is no NodePort or login-free address. The platform Elasticsearch, which may seem to "open directly", has no web UI: what answers is its API. + +## Resolution + +### Cause 1: the query address and the basic-auth Secret are on the monitoring feature, not on the VMUI page + +1. On the global cluster, read the address and the Secret name: + + ```bash + kubectl get feature monitoring -o yaml + kubectl get feature monitoring -o jsonpath='{.spec.accessInfo.database.address}{"\n"}{.spec.accessInfo.database.service}{"\n"}' + ``` + +2. List the keys of the Secret, then decode them. If the key list is empty, the Secret name does not match this version; do not invent a password. + + ```bash + kubectl -n cpaas-system get secret victoriametrics-basic-auth -o go-template='{{range $k,$v := .data}}{{printf "%s\n" $k}}{{end}}' + kubectl -n cpaas-system get secret victoriametrics-basic-auth -o jsonpath='{.data.username}' | base64 -d; echo + kubectl -n cpaas-system get secret victoriametrics-basic-auth -o jsonpath='{.data.password}' | base64 -d; echo + ``` + +3. Take the tenant from the cluster's own `vmagent` remote-write URL instead of typing it: + + ```bash + kubectl -n cpaas-system get vmagent -o yaml | grep -n 'insert/' + ``` + +4. From the network where Grafana or the script runs, test before saving a data source. Use `-G` with `--data-urlencode` instead of hand-encoding `match%5B%5D`: + + ```bash + curl -k -s -u ':' -G "https:///clusters//vmselect/api/v1/query" --data-urlencode 'query=up' + curl -k -s -u ':' -G "https:///clusters//vmselect/federate" --data-urlencode 'match[]=up' + ``` + + Inside the cluster, the `service` address works the same way, for example `http://vmselect:8481/select//prometheus/api/v1/query`. For raw export: + + ```bash + curl -k -u ':' 'https:///select//prometheus/api/v1/export' \ + --data-urlencode 'match[]=node_cpu_seconds_total{mode="idle"}' \ + --data-urlencode 'start=' --data-urlencode 'end=' + ``` + +5. In Grafana, create a **Prometheus** data source with the `address` URL, enable basic auth with the decoded user and password, and save. The platform monitoring dashboards cannot be exported as Grafana JSON; build the Grafana dashboards from the metric names. + +6. For an external system that only needs business Pod CPU, memory and network I/O, query this API rather than the platform API, and exclude platform namespaces, for example: + + ```promql + sum by (namespace, pod) (rate(container_cpu_usage_seconds_total{namespace!~"cpaas-system|kube-system",container!="",container!="POD"}[5m])) + ``` + +On a Prometheus cluster, use the Prometheus path and its Secret: + +```bash +kubectl get secret -n cpaas-system kube-prometheus-prometheus-basic-auth -o jsonpath='{.data.password}' | base64 -d; echo +curl -k -u 'monitor@cpaas.io:' -G 'https:///clusters//prometheus-0/api/v1/query_range' \ + --data-urlencode 'query=node_cpu_seconds_total' --data-urlencode 'start=' \ + --data-urlencode 'end=' --data-urlencode 'step=1m' +``` + +On Windows, call `curl.exe` explicitly. In Windows PowerShell `curl` is an alias of `Invoke-WebRequest`, which rejects curl options with `ParameterBindingException` / `AmbiguousParameter`; that error is about the command, not the API. If the API answers from a cluster node but not from the Windows client, the client network policy is blocking it. + +### Cause 2: the vmui server URL ends in prometheus/ + +1. Open `https:///clusters//vmselect/vmui/?`. If the first visit redirects to the Prometheus path, change the path back. +2. Open **Settings** at the top right, remove the trailing `prometheus/` from the server URL, and click **Apply**. +3. Run the query again; it returns data. + +### Cause 3: vmui through a NAT-mapped address redirects to an unreachable internal address + +1. From the user's PC, confirm where the redirect goes: + + ```bash + curl -k -L 'https:///clusters//vmselect/vmui/?' + ``` + + A `Found` redirect to `https:///dex/auth?...` followed by `Failed to connect to port 443` is this cause. The same request from a global node returns the login page. + +2. Open vmui from a network that reaches the real platform address (VPN, cloud desktop, or the global VIP). The documented entry is `https:///clusters//vmselect-ui/vmui/`, then SSO login, then the vmui settings; for a VictoriaMetrics plugin that is not agent-only, `/clusters//vmselect/vmui/` also works. + +On a platform deployed by IP, vmui cannot work through a mapped address, and opening extra ports does not change that. A hosts-file entry that points a domain at the mapped address helps only on platforms deployed with a domain name, where the Dex callback is a resolvable domain. Only the global nodes need to reach the workload cluster on port `11780`. + +### Cause 4: the HAMi web UI has the wrong Prometheus URL or credentials + +Prove the in-cluster Prometheus URL with `curl` and the basic-auth pair from Cause 1, then enter exactly that URL and pair in the HAMi web UI installation parameters. No Prometheus restart is needed when the Pod is Running and `up` returns samples. + +### Cause 5: Prometheus has no login-free access + +There is no supported login-free or NodePort access to Prometheus. Use the platform address with the platform login, or the API with basic auth as in Cause 1. + +## Diagnostic Steps + +1. On the global cluster, print `feature/monitoring` and the Secret keys before asking anyone for a connection string (Cause 1): + + ```bash + kubectl get feature monitoring -o jsonpath='{.spec.accessInfo.database}{"\n"}' + ``` + + Expected: an `address`, a `service`, and `basicAuth.secretName`. + +2. Run the query with and without credentials: + + ```bash + curl -k -s -o /dev/null -w '%{http_code} %{content_type}\n' -G "https:///clusters//vmselect/api/v1/query" --data-urlencode 'query=up' + curl -k -s -u ':' -G "https:///clusters//vmselect/api/v1/query" --data-urlencode 'query=up' | head -c 200 + ``` + + Without credentials: `200 text/html` (login page). With credentials: JSON with `"status":"success"`. An empty body or empty result with credentials means a wrong tenant or path. + +3. vmui `is not valid JSON`: open vmui **Settings** and read the server URL; a trailing `prometheus/` is Cause 2. + +4. vmui redirects to an internal address only from the user's network: Cause 3. Compare the same `curl -L` from a global node. + +5. HAMi UI empty while its Pods are Running: test the URL and credentials it was installed with (Cause 4). diff --git a/docs/en/solutions/Grafana_returns_HTTP_500_after_a_password_change_is_missing_after_an_upgrade_or_rules_and_alerts_do_not_carry_over.md b/docs/en/solutions/Grafana_returns_HTTP_500_after_a_password_change_is_missing_after_an_upgrade_or_rules_and_alerts_do_not_carry_over.md new file mode 100644 index 000000000..9bf3709d5 --- /dev/null +++ b/docs/en/solutions/Grafana_returns_HTTP_500_after_a_password_change_is_missing_after_an_upgrade_or_rules_and_alerts_do_not_carry_over.md @@ -0,0 +1,196 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.8.x + - 3.10.x + - 3.12.x + - 3.14.x + - 3.16.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Grafana returns HTTP 500 after a password change, is missing after an upgrade, or rules and alerts do not carry over on ACP + +## Issue + +One of these is reported about Grafana or the monitoring stack around it: + +- The operations center returns HTTP 500 on Grafana after the Grafana admin password was changed in the Grafana UI. Logging in on the Grafana page itself with the new password works. +- Platform 3.18 has no Grafana at all, or the Grafana page disappeared after an upgrade to 4.0, and JVM, Tomcat, Istio or StarRocks charts are still needed. +- On 3.8.2, logging in to Grafana (v7.5.16) with `admin/admin` fails with `Invalid username or password`. +- A security scan reports Prometheus or component metrics reachable without authentication on 3.14, including `kube-ovn-monitor`, `kube-ovn-controller`, or `devops-api` on NodePort `30699`. +- DingTalk answers `errcode: 300001`, `errmsg: msgtype is null`, or Feishu and other webhooks return `InvalidJson` from `apollo`, for example `invalid character 'å' looking for beginning of value`. +- `PrometheusRule` objects exported from a 3.10 cluster do nothing on a 4.3 VictoriaMetrics cluster. +- There is no built-in alert for APIServer restarts or etcd WAL and proposal latency. + +When the chart includes it, Grafana runs in `cpaas-system` and the operations center signs in to it with a password stored in a Secret. + +Related article: [External Grafana or scripts cannot query platform metrics: vmselect address, credentials, and vmui errors on ACP](External_Grafana_or_scripts_cannot_query_platform_metrics_vmselect_address_credentials_and_vmui_errors_on_ACP.md). + +## Root Cause + +### Cause 1: the Grafana Secret still holds the old password + +The operations center signs in to Grafana server-side with the password from Grafana's Secret. A password changed only in the Grafana UI leaves the Secret on the old value, so the operations center session fails with HTTP 500 while a browser login with the new password works. + +### Cause 2: platform versions from 3.18 do not ship Grafana + +3.18 has Prometheus and no built-in Grafana, and later versions no longer provide Grafana dashboards. A Grafana page that is gone after an upgrade to 4.0 is expected. JVM and Tomcat charts exist only when something scrapes the application and a dashboard reads those metrics. + +### Cause 3: the 3.8.2 Grafana admin password is not admin/admin and cannot be changed + +On 3.8.2 `admin/admin` is rejected, and that version has no procedure for changing the Grafana admin password. The Grafana embedded under **Operations Center > Monitoring > Grafana** still shows its dashboards. + +### Cause 4: metrics ports answer without authentication on 3.14 + +On 3.14.x the metrics ports answer without authentication; builds from 3.18.0 close it. 3.14 has no supported switch to add authentication to those ports. `kube-ovn-monitor` and `kube-ovn-controller` in `kube-system` belong to the same finding, and a build without the `--enable-metrics` flag cannot turn their handler off. `devops-api` exposed as NodePort `30699` makes its metrics reachable from outside the node. + +### Cause 5: the webhook body is not the JSON the receiver requires + +DingTalk answers `msgtype is null` because the rendered body has no `msgtype`. Feishu or a generic webhook returns `InvalidJson` because the template is Markdown or HTML, or webhook template rendering is off. TCP reachability is fine; the HTTP status is an application error. These cases are covered in [DingTalk robot rejects alert notifications with errcode 300001 or 310000 on ACP](DingTalk_robot_rejects_alert_notifications_with_errcode_300001_or_310000_on_ACP.md) and [Webhook alert notifications rejected as InvalidJson or sent without the notification template on ACP](Webhook_alert_notifications_rejected_as_InvalidJson_or_sent_without_the_notification_template_on_ACP.md). + +### Cause 6: a PrometheusRule is not evaluated by VictoriaMetrics + +3.10 alert objects are `PrometheusRule`. A VictoriaMetrics cluster (for example 4.3.2) evaluates `VMRule`. Importing the old kind creates objects that `vmalert` never loads. + +### Cause 7: APIServer restarts and etcd latency have no dedicated default rule + +`cpaas-cluster-rules` covers broad control-plane, controller and scheduler symptoms, but does not ship a dedicated APIServer restart-count rule or an etcd WAL / proposal rule. The series are already collected: APIServer restarts, `etcd_disk_wal_fsync_duration_seconds_bucket`, `etcd_server_proposals_pending`, `etcd_server_proposals_failed_total`, and `grpc_server_handled_total{grpc_code!="OK"}`. + +## Resolution + +### Cause 1: the Grafana Secret still holds the old password + +1. Find the Grafana Secret and the key that holds the admin password (the name varies with the chart release; on 3.16 it is `grafana-basic-auth`): + + ```bash + kubectl -n cpaas-system get secret | grep -i grafana + kubectl -n cpaas-system get secret -o yaml > /tmp/grafana-secret.yaml + ``` + +2. Write the password that was set in the Grafana UI into that key and recreate the Grafana Pod, which does not reload the Secret by itself. + + > **Warning:** Replacing the password and restarting Grafana logs every operations-center session out of Grafana until the Pod is Ready, and writing the wrong key overwrites another credential in that Secret. Precondition: `/tmp/grafana-secret.yaml` is saved and you change only the password key. Rollback: `kubectl apply -f /tmp/grafana-secret.yaml` and restart Grafana again. + + ```bash + B64=$(printf '%s' '' | base64 -w0) + kubectl -n cpaas-system patch secret --type=merge -p "{\"data\":{\"\":\"${B64}\"}}" + kubectl -n cpaas-system delete pod + ``` + + On 3.16 the same is done with `kubectl edit secret -n cpaas-system grafana-basic-auth` after the `admin` password was changed in the Grafana UI. + +### Cause 2: platform versions from 3.18 do not ship Grafana + +- On 4.0 and later, import the existing Grafana dashboard JSON under **Operations Center > Monitoring > Monitoring Dashboards**; it is converted into a native monitoring dashboard. JVM metrics are viewed the same way, with queries on the exposed metric names. +- On 3.18, make the application expose JVM metrics, scrape them with a ServiceMonitor (see [How to Add Metrics for Monitoring Collection](How_to_Add_Metrics_for_Monitoring_Collection.md)), install your own Grafana, and point it at `https:///clusters//prometheus-0` with basic auth. Do not look for a hidden Grafana Deployment. + +Confirm that the process actually exposes the JVM metrics before debugging any dashboard. + +### Cause 3: the 3.8.2 Grafana admin password is not admin/admin and cannot be changed + +Leave the Grafana password unchanged on 3.8.2. Use the embedded Grafana under **Operations Center > Monitoring > Grafana**, or read data through the Prometheus `query_range` API with the `monitor@cpaas.io` credentials from Secret `kube-prometheus-prometheus-basic-auth`: + +```bash +kubectl get secret -n cpaas-system kube-prometheus-prometheus-basic-auth -o jsonpath='{.data.password}' | base64 -d; echo +``` + +The full `curl` and the Windows `curl.exe` note are in Cause 1 of the related article above. + +### Cause 4: metrics ports answer without authentication on 3.14 + +Upgrade to 3.18.0 or later; that is the fix for the Prometheus metrics ports and for kube-ovn. Do not add `--enable-metrics=false` to a kube-ovn build that does not already have the flag. + +If a port must be closed before the upgrade: + +> **Warning:** A host firewall drop on the Prometheus metrics port stops all scraping through it and the operations-center charts that read it; the rest of the platform keeps running. This is not authentication. Precondition: a monitoring blackout until the upgrade is accepted, and the current firewall rules are saved. Rollback: restore the saved rules. + +For `devops-api` on NodePort `30699`: + +> **Warning:** Changing `devops-api` to ClusterIP closes port `30699` on every node, including for any legitimate external scraper. Precondition: the Service name and namespace are confirmed. Rollback: set `spec.type` back to `NodePort` with `nodePort: 30699`. + +```bash +kubectl get svc -A | grep devops-api +kubectl -n patch svc devops-api --type=merge -p '{"spec":{"type":"ClusterIP"}}' +``` + +### Cause 5: the webhook body is not the JSON the receiver requires + +Enable webhook template rendering and replace the Markdown sample with a JSON body that the receiver's API documents. A DingTalk body that returned HTTP 200 and `errcode: 0`: + +```json +{"msgtype":"text","text":{"content":"alert fired"}} +``` + +Read what `apollo` rendered; if it is not JSON, the template is still wrong: + +```bash +kubectl -n cpaas-system logs deploy/apollo --tail=200 | grep -Ei 'webhook|InvalidJson|msgtype' +``` + +### Cause 6: a PrometheusRule is not evaluated by VictoriaMetrics + +1. Confirm the API group on the destination cluster and read the labels of a healthy existing `VMRule`, so the new objects carry the label `vmalert` selects: + + ```bash + kubectl api-resources | grep -i vmrule + kubectl get vmrule -A --show-labels | head + ``` + +2. Convert each exported rule to a `VMRule`: + + ```yaml + apiVersion: operator.victoriametrics.com/v1beta1 + kind: VMRule + metadata: + name: + namespace: + labels: + victoriametrics: "true" # use the label key and value of an existing healthy VMRule + spec: + groups: + - name: + rules: + - alert: + expr: + for: 5m + ``` + + > **Warning:** A `VMRule` with an expression that is true on empty or wrong labels pages at once. Precondition: run each expression in vmui and compare the firing set with the old cluster. Rollback: `kubectl delete vmrule -n `. + +### Cause 7: APIServer restarts and etcd latency have no dedicated default rule + +Create a custom alert rule from the expression you need, with your own threshold, for example: + +```promql +increase(etcd_server_proposals_failed_total[5m]) > 0 +``` + +Do not expect `cpaas-cluster-rules` to gain these rules by itself. + +## Diagnostic Steps + +1. HTTP 500 on Grafana only from the operations center, right after a password change in the Grafana UI: Cause 1. Compare the decoded Secret password with the one that logs in. + +2. Record the platform version. 3.18 or later with no Grafana Deployment is Cause 2; 3.8.2 with `admin/admin` rejected is Cause 3; an unauthenticated-metrics finding below 3.18.0 is Cause 4, and a `NodePort` type on `devops-api` is its NodePort variant: + + ```bash + kubectl get svc -A | grep -E 'devops-api|kube-ovn' + ``` + +3. A vendor error code from the webhook receiver (`msgtype is null`, `InvalidJson`): Cause 5. + +4. Show which rule kind the cluster evaluates; `PrometheusRule` objects on a VictoriaMetrics cluster with no matching `VMRule` are Cause 6: + + ```bash + kubectl get prometheusrule,vmrule -A + ``` + +5. No alert for APIServer restarts or etcd latency in `cpaas-cluster-rules`: Cause 7. Confirm the series exist in vmui first, for example `etcd_server_proposals_pending`. diff --git a/docs/en/solutions/Metrics_Service_availability_alert_is_not_1_while_the_target_Pod_looks_healthy_on_ACP.md b/docs/en/solutions/Metrics_Service_availability_alert_is_not_1_while_the_target_Pod_looks_healthy_on_ACP.md new file mode 100644 index 000000000..5f484f840 --- /dev/null +++ b/docs/en/solutions/Metrics_Service_availability_alert_is_not_1_while_the_target_Pod_looks_healthy_on_ACP.md @@ -0,0 +1,208 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Metrics Service availability alert is not 1 while the target Pod looks healthy on ACP + +## Issue + +The real-time alert page reports that the availability of a Metrics Service obtained by Prometheus is not 1 (`the availability of Metrics Service != 1`), or "Prometheus failed to fetch metrics" (`Prometheus获取metrics异常`), or a node kubelet health alert. The rule stays firing for one or three minutes and sometimes clears by itself, often over and over. The Service named in the alert and its Pods are `Running`. `curl` from inside the Pod returns metrics, or an anonymous request to the Service returns HTTP 403, which only shows that the process answers. In vmagent targets the endpoint is `down`, or `up` for that Service is `0`. + +The built-in rule on a cluster has this shape (the `vmcluster` label is the cluster name): + +```promql +max by (namespace, service) (up{vmcluster="",service!="kube-prometheus-exporter-dockerd",job!="apiserver",job!="kube-controller-manager",job!="kube-scheduler",job!="kubelet",job!="kube-dns",job!="kube-etcd"}) != 1 +``` + +The alert reports that a scrape failed. Each cause below is a different reason for the failed scrape. + +## Root Cause + +### Cause 1: the node firewall blocks the metrics port + +`curl` of `/metrics` inside the Pod succeeds. From another node the same endpoint IP and port fails with `No route to host` or `Failed to connect`, and vmagent lists the target as `down`. The host firewall zone on the node that hosts the endpoint does not allow the metrics port, for example `10254` of an ingress-nginx controller metrics Service. This is covered in detail in Cause 3 of [Node firewall or iptables rules block kubelet 10250, metrics 10254 or control-plane traffic on ACP](Node_firewall_or_iptables_rules_block_kubelet_10250_metrics_10254_or_control_plane_traffic_on_ACP.md). + +### Cause 2: calico-felix-metrics-svc is scraped while Felix metrics are disabled + +`calico-felix-metrics-svc` availability is `0`. The `calico-node` template sets `FELIX_PROMETHEUSMETRICSENABLED=false`, but the Service is still exposed on port `9091` with selector `k8s-app=calico-node`, so monitoring scrapes an endpoint that is turned off. After an upgrade to 4.3.2 (cpaas-calico chart `v4.3.30`) the same state was found with Endpoints on `:9091`; there the Service was confirmed as no longer used, left over from before the upgrade. + +### Cause 3: the metrics backend is OOMKilled, so the alert clears by itself + +Availability drops below 1 and later recovers with no scrape change. The backend named by the alert is restarting with `OOMKilled` (for example `clickhouse-operator`). Each restart takes the metrics endpoint away until the Pod is back. + +### Cause 4: cluster-enhancer-apiserver scrape duration reaches the scrape timeout + +The alert names `cpaas-system` / `cluster-enhancer-apiserver`, and `up` is `0` for minutes or hours, often on a cluster where the alert is sent again and again. The Pods are `Running` with normal CPU and memory. An anonymous `curl` to port `8443` returns HTTP 403 `forbidden`, which is the expected answer of this endpoint. `scrape_duration_seconds` for job `cluster-enhancer-apiserver` sits at or above the configured scrape timeout (for example `45.001` against a timeout of `45` with a sampling interval of `60`, the VictoriaMetrics plugin defaults). The scrape does not finish inside the timeout, so `up` is `0`. This was seen with image `cluster-enhancer-apiserver:v4.2.6`. + +### Cause 5: the alb image serves metrics over HTTP + +"Prometheus failed to fetch metrics" fires. On the targets page `serviceMonitor/cpaas-system/kube-prometheus-cpaas-components-1/1` is `0/5 up`, and the ALB targets (ports `1936` and `11782`, endpoint `https-metrics`) report: + +```text +Get "https://:/metrics": http: server gave HTTP response to HTTPS client +``` + +The `alb` image had been changed away from the version the platform ships, and that image serves its metrics over plain HTTP. + +### Cause 6: kubelet health flips because one container produces duplicate metric data + +The node kubelet health alert fires (rule `sum(up{job="kubelet"}) / count(up{job="kubelet"})!=1` for 1 minute), and `up{job="kubelet"}` for one node (`endpoint="https-metrics"`, port `10250`) flips between 0 and 1. Restarting kubelet and containerd does not help. Node events can repeat `Starting kubelet.`, `InvalidDiskCapacity` / `invalid capacity 0 on image filesystem`, and `NodeNotReady` followed by `NodeReady`. A request to the kubelet metrics endpoint fails as a whole because one container on that node produces a duplicate series: + +```text +An error has occurred while serving metrics: collected metric "kubelet_container_log_filesystem_used_bytes" { label:" > label:" > label:" > ... } was collected before with the same name and label values +``` + +When storage on the node had been cleaned by hand earlier, leftovers most likely made that container's metric conflict. + +## Resolution + +### Cause 1: the node firewall blocks the metrics port + +Run the checks in the linked article: the metrics path answers inside the Pod and on its own node, fails from another node, and `firewall-cmd --list-all` on the owning node does not list the port. + +> **Warning:** Opening the port on the node firewall exposes that metrics port to every source the zone allows. Precondition: the three checks match. Use the active zone if it is not `public`. Rollback: `firewall-cmd --zone=public --remove-port=10254/tcp --permanent` and `firewall-cmd --reload`. + +```bash +firewall-cmd --zone=public --add-port=10254/tcp --permanent +firewall-cmd --reload +``` + +After the next scrape the target leaves `down`. + +### Cause 2: calico-felix-metrics-svc is scraped while Felix metrics are disabled + +1. Confirm the mismatch: + + ```bash + kubectl get svc -A | grep calico-felix-metrics + kubectl -n kube-system get ds calico-node -o yaml | grep -n -A1 FELIX_PROMETHEUSMETRICSENABLED + ``` + +2. Where the Service is confirmed unused (as on 4.3.2 after the upgrade), back it up and delete it. + + > **Warning:** Deleting `calico-felix-metrics-svc` removes that scrape target. Precondition: Felix metrics are disabled, the alert names this Service, and nothing else uses it. On other versions confirm that it is unused first. Rollback: `kubectl apply -f /tmp/calico-felix-metrics-svc.yaml`. + + ```bash + kubectl -n kube-system get svc calico-felix-metrics-svc -o yaml > /tmp/calico-felix-metrics-svc.yaml + kubectl -n kube-system delete svc calico-felix-metrics-svc + ``` + +Felix metrics can be disabled on purpose, as in [Disable Calico Node Metrics Port](How_to_Disable_Calico_Node_Metrics_Port.md); in that case the Service is the leftover, not the setting. + +### Cause 3: the metrics backend is OOMKilled, so the alert clears by itself + +1. Confirm that the last termination is `OOMKilled` and that availability returns to 1 after each restart: + + ```bash + kubectl get pod -A | grep + kubectl -n describe pod | grep -E 'OOMKilled|Exit Code|Restart Count' + ``` + +2. Raise memory on the object that owns that workload, above the working set that was killed. + + > **Warning:** Raising memory rolls the backend; its control loop and metrics endpoint pause until the new Pod is Ready. Precondition: the export below is saved. Rollback: set the previous memory limit on the same object. + + ```bash + kubectl -n get deploy -o yaml > /tmp/backend-deploy.yaml + kubectl -n edit deploy + ``` + + If the edit is written back within a minute, an operator or plugin owns the size; make the same change on that owner instead. + +### Cause 4: cluster-enhancer-apiserver scrape duration reaches the scrape timeout + +1. In vmui on the affected cluster, compare scrape duration with the timeout: + + ```promql + scrape_duration_seconds{vmcluster="",job="cluster-enhancer-apiserver"} + ``` + +2. Read the current sampling interval and scrape timeout in the VictoriaMetrics plugin configuration (Marketplace > Cluster Plugins, update the VictoriaMetrics monitoring plugin of that cluster). They are stored as `spec.config.components.vmagent.scrapeInterval` and `scrapeTimeout` on its ModuleInfo, default `60` and `45`. + +3. When duration is at or above the timeout, set the sampling interval to `90` and the scrape timeout to `60` in the plugin configuration and save. The interval must be greater than the timeout. + + > **Warning:** A longer timeout and interval reduce scrape frequency for every target of that vmagent and let slow scrapes hold resources longer. Precondition: `up` is 0, the Pod is Running, and `scrape_duration_seconds` is at or above the current timeout. Rollback: set the interval back to `60` and the timeout back to `45` in the same form. + +4. After the next scrapes, `scrape_duration_seconds` stays below the new timeout, `up` returns to `1`, and the alert stops. + +If the Pod has stayed Running with `up` at 0 for hours and the duration is not pinned at the timeout, restarting the Pod also restored scraping: + +> **Warning:** Restarting `cluster-enhancer-apiserver` drops its endpoint for one start. Precondition: the duration query does not show the timeout case above. Rollback is the new Pod reaching Ready. + +```bash +kubectl -n cpaas-system get pod | grep cluster-enhancer-apiserver +kubectl -n cpaas-system delete pod +``` + +HTTP 403 from an anonymous `curl` is not the reason `up` is 0; do not restart for it. + +### Cause 5: the alb image serves metrics over HTTP + +1. Compare the `alb` image tag with the one the platform version ships (a healthy cluster of the same version is the easiest reference): + + ```bash + kubectl get deploy,ds -A -o wide | grep -i alb + ``` + +2. Set the image back. + + > **Warning:** Changing the `alb` image rolls the ALB Pods and can briefly interrupt traffic through them. Precondition: the running image differs from the shipped one and the alert started after that change. Rollback: `kubectl apply -f /tmp/alb-deploy.yaml`. + + ```bash + kubectl -n get deploy -o yaml > /tmp/alb-deploy.yaml + kubectl -n set image deploy/ = + ``` + +The scrape recovers and the alert clears. + +### Cause 6: kubelet health flips because one container produces duplicate metric data + +1. Fetch the kubelet metrics and read the error: + + ```bash + kubectl get --raw "/api/v1/nodes//proxy/metrics" > /tmp/kubelet-metrics.txt; echo "exit=$?" + ``` + + Or from the node, with the kubelet client certificate: + + ```bash + curl -sk https://:10250/metrics --cacert /etc/kubernetes/pki/ca.crt \ + --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key | head -3 + ``` + +2. Confirm that the Pod named in the duplicate-metric error runs on this node, then recreate it. + + > **Warning:** Recreating the Pod interrupts that workload until it is Running again. Precondition: the kubelet metrics error names that container's metric and the Pod is on the flapping node. Rollback is the controller recreate. + + ```bash + kubectl -n delete pod + ``` + +After the restart the duplicate data is gone and the kubelet health of the node recovers. + +## Diagnostic Steps + +1. Read the alert labels and query `up` for that namespace and Service. `up=0` with a Running Pod is this article. + + ```bash + kubectl get pod -A | grep -E 'vmagent|cluster-enhancer-apiserver|calico-node' + ``` + +2. In-Pod `curl` works and another node gets `No route to host`: Cause 1. + +3. `calico-felix-metrics-svc` at 0 with `FELIX_PROMETHEUSMETRICSENABLED=false`: Cause 2. + +4. The alert clears by itself and the backend's last state is `OOMKilled`: Cause 3. + +5. `cluster-enhancer-apiserver` Running, anonymous `curl` returns 403, and `scrape_duration_seconds` equals the scrape timeout: Cause 4. Expected after the fix: duration below the new timeout and `up` at `1`. + +6. Targets page shows `server gave HTTP response to HTTPS client` on ALB endpoints after an image change: Cause 5. + +7. Kubelet health flapping on one node, and the kubelet metrics endpoint returns `was collected before with the same name and label values`: Cause 6. diff --git a/docs/en/solutions/Monitoring_components_OOMKilled_or_near_their_limits_vmagent_vmstorage_kube-state_Prometheus_vm-operator_on_ACP.md b/docs/en/solutions/Monitoring_components_OOMKilled_or_near_their_limits_vmagent_vmstorage_kube-state_Prometheus_vm-operator_on_ACP.md new file mode 100644 index 000000000..d2019efa3 --- /dev/null +++ b/docs/en/solutions/Monitoring_components_OOMKilled_or_near_their_limits_vmagent_vmstorage_kube-state_Prometheus_vm-operator_on_ACP.md @@ -0,0 +1,184 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.12.x + - 3.14.x + - 3.16.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Monitoring components OOMKilled or near their limits: vmagent, vmstorage, kube-state, Prometheus, vm-operator on ACP + +## Issue + +A platform monitoring component in `cpaas-system` runs out of memory or stays near its limit: + +- `vmagent-agent` restarts with `OOMKilled` / exit code `137`, ends in `CrashLoopBackOff` (`1/2` Ready), or fails its readiness probe so charts have gaps. Critical `cpaas-deployment-rules` alerts fire for `vmagent-agent`, and some nodes or the whole cluster have no data. +- `vmstorage-cluster` sits above 90% CPU or memory on one or every replica, or restarts with `OOMKilled`, and the page says some monitoring cannot be displayed (`部分监控无法显示`). +- `kube-prometheus-exporter-kube-state` is `OOMKilled`, and the Deployment CPU and memory trend panels are empty, or an inspection flags it as abnormal. +- The `prometheus` container is `OOMKilled` and monitoring data stops. +- `vm-operator` restarts on OOM and cluster monitoring is abnormal. + +Sizes set directly on the live Deployment, StatefulSet or VictoriaMetrics object are often written back. The persistent place is the monitoring plugin configuration: Marketplace > Cluster Plugins > update the monitoring plugin > choose a custom scale and set the component quota. It is stored on the plugin's ModuleInfo as `spec.config.components..resources` (for example `vmagent`, `vmstorage`, `vmselect`, `kubeStateExporter`, `prometheus`). + +## Root Cause + +### Cause 1: vmagent is undersized for the scrape and remote-write load + +`kubectl describe pod` shows `Last State: Terminated`, `Reason: OOMKilled`, `Exit Code: 137`, often with thousands of restarts. The log shows a normal start (`kubernetes_sd_configs: added targets: `) and the kill a minute or two later, and after a restart: + +```text +queue/... size (... bytes) is bigger than writer offset (... bytes); this may be the case on unclean shutdown (OOM, kill -9, hardware reset) +``` + +That line is a consequence of the OOM, not a second fault. The agent cache, scraping, and temporary remote-write buffers together exceed the limit (for example about 1.2Gi of cache plus about 200MB of scraping against a 2Gi limit). Triggers seen: a global `vmagent` at 1 CPU / 1Gi; a monitoring configuration change in the plugin; bulk, high-frequency calls to the platform API for container metrics, which also restart `vmselect`; and failed or timed-out scrapes cached in memory. Restarts that look causeless are still OOM: 15s samples are aggregated over 60s, so charts and the 90% alert can miss the peak. When the agent is near its limit without being killed, readiness probe failures show up as chart gaps instead of restarts. + +A `vmagent` that restarts with `remote_write` `502 Bad Gateway` and is not `OOMKilled` is query overload from a very large `certificate_expires_time` series, and more memory does not fix it. Read the limit of the `vmagent` container, not of the `config-reloader` sidecar in the same Pod. + +### Cause 2: vmstorage memory or CPU limit is too small for the stored series + +Restart count is 0 and usage stays above 90%: the limit itself caps the component (for example memory request `1000Mi` / limit `2000Mi`, or a 4-core CPU limit on all three replicas). Tuning the VMCluster cache-hit setting and restarting does not lower CPU. + +When the limit is far below the working set, one replica OOM-restarts (`OOMKilled`, exit `137`, memory at 155.80% of its limit, hundreds of restarts). That case is misleading: `vmagent` logs that it cannot write, restarting `vmagent` brings data back for a while, and the gap returns. The real failure is in `vminsert` and `vmstorage`: + +```text +# vmagent +sending a block with size 258764 bytes to "2:secret-url" was rejected (skipping the block): status code 400; response body: duplicate sample for timestamp +# vminsert +cannot make a copy #2 out of 2 copies according to -replicationFactor=2 ... since a part of storage nodes is temporarily unavailable +cannot dial storageNode "vmstorage-cluster-0.vmstorage-cluster.cpaas-system:8400": ... connect: connection refused +``` + +`successfully dialed -storageNode` follows each time the replica returns, so the gap looks intermittent. + +### Cause 3: kube-state-metrics is sized by /pod_nanny + +The container command is `/pod_nanny` with `--container=exporter-kube-state`, `--cpu`, `--extra-cpu`, `--memory`, `--extra-memory`, `--threshold` and `--deployment=kube-prometheus-exporter-kube-state`. The addon resizer sets CPU from `--cpu` plus `--extra-cpu` per node and memory from `--memory` plus `--extra-memory` per node, so a patch to `resources` loses to the next nanny loop. Starting flags `--cpu=100m`, `--extra-cpu=5m`, `--memory=130Mi`, `--extra-memory=20Mi`, `--threshold=5` had already resized the Pod to `540m` / `1890Mi` and it still OOMed. On 3.12.2 the visible symptom was empty Deployment trend panels while kube-state restarted with `OOMKilled`. + +### Cause 4: the Prometheus container limit is too small + +The `prometheus` container is `OOMKilled` / exit `137` (for example `prometheus-kube-prometheus-0-0` at `Running (3/4)` with limits `cpu: 2` / `memory: 8000Mi`), and the business cluster's data stops, including in the global views. Doubling the limit let it stay up; 4 CPU / 16Gi stopped the restarts of an `8000Mi` instance. + +### Cause 5: vm-operator is OOMKilled + +After the cluster recovered from an outage, global monitoring dashboards were abnormal and `vm-operator` on the business cluster restarted with `OOMKilled`. Hosts near 100% memory at the same time are a separate problem to handle on the nodes. + +## Resolution + +### Cause 1: vmagent is undersized for the scrape and remote-write load + +1. Confirm `137` / `OOMKilled` and read the real `vmagent` container limit: + + ```bash + kubectl -n cpaas-system describe pod | grep -E 'Exit Code|Reason|OOMKilled' + kubectl -n cpaas-system get pod -o jsonpath='{range .spec.containers[*]}{.name}{" req="}{.resources.requests.memory}{" lim="}{.resources.limits.memory}{"\n"}{end}' + ``` + +2. Raise `vmagent` memory in the plugin configuration (custom scale). Sizes that stopped the restarts: `1000Mi` or 1Gi to 2Gi; 2Gi to 4Gi; `6000Mi`; 2 CPU / `2000Mi` where readiness failed above 80% memory. Under bulk API pulls, double CPU and memory of both `vmagent` and `vmselect`, then watch for another OOM. + +3. If the plugin cannot be updated, edit the VMAgent object as a stop-gap: + + > **Warning:** Editing VMAgent `agent` restarts `vmagent`; scraping and remote write pause until the new Pod is Ready, and a size the nodes cannot hold leaves it Pending. Precondition: the last termination is `OOMKilled`, or readiness failures coincide with memory near the limit. Rollback: `kubectl apply -f /tmp/vmagent-agent.yaml`. Carry the value into the plugin configuration afterwards, or a reconcile writes it back. + + ```bash + kubectl -n cpaas-system get vmagent agent -o yaml > /tmp/vmagent-agent.yaml + kubectl -n cpaas-system edit vmagent agent + ``` + +4. Confirm the running Pod shows the new limit and stops restarting. Do not delete the remote-write queue directory; `vmagent` repairs the queue on its next clean start. + +### Cause 2: vmstorage memory or CPU limit is too small for the stored series + +1. When `vmagent` restarts only help for a while, read `vmstorage` and `vminsert` first: + + ```bash + kubectl -n cpaas-system describe pod | grep -E 'Reason|Exit Code|Restart Count' + kubectl -n cpaas-system logs -c vmagent --tail=300 | grep -E 'duplicate sample for timestamp|was rejected' + kubectl -n cpaas-system logs --tail=300 | grep -E 'cannot make a copy|cannot dial storageNode|broken pipe' + ``` + +2. Raise the `vmstorage` limit in the plugin configuration, not with `kubectl edit sts vmstorage-cluster` (the plugin page stays read-only until the scale is switched to custom). Changes that cleared the alert: memory limit `2000Mi` to `4000Mi`; CPU limit 4 to 5 cores; CPU `1000m` to `2000m` where only gaps were seen. Without a better measurement, double the current limit and watch. + + > **Warning:** Changing `vmstorage` resources rolls the replicas one by one; writes and queries to a restarting replica fail over to the others, and with replication factor 2 a second failure during the roll can leave gaps. Rollback: set the previous values in the same form. + +3. Confirm the new limit on the Pods: + + ```bash + kubectl -n cpaas-system get pod | grep vmstorage + kubectl -n cpaas-system get pod -o jsonpath='{.spec.containers[0].resources}{"\n"}' + ``` + +An `OOMKilled` `vmstorage` is fixed by more `vmstorage` memory, not by another `vmagent` restart. + +### Cause 3: kube-state-metrics is sized by /pod_nanny + +1. Print the nanny flags: + + ```bash + kubectl -n cpaas-system get deploy kube-prometheus-exporter-kube-state -o yaml | grep -n -E 'pod_nanny|--cpu|--memory|extra-cpu|extra-memory|threshold' + ``` + +2. Raise both the base and the per-node extra so the result lands above the OOM working set (for example from `130Mi` / `20Mi` upwards). + + > **Warning:** Changing `--memory` or `--cpu` makes `/pod_nanny` resize and restart `exporter-kube-state`; kube-state series pause during the restart. Precondition: the four old flag values are noted. Rollback: set them back. If the old args return within a minute, the flags are chart-owned: set the Kube State Exporter quota in the plugin configuration instead. + + ```bash + kubectl -n cpaas-system edit deploy kube-prometheus-exporter-kube-state + ``` + +3. The limits on the new Pod follow the nanny formula, not numbers typed under `resources`. Empty Deployment trend panels return as soon as the exporter stays up. + +### Cause 4: the Prometheus container limit is too small + +1. Confirm the termination reason and rule out a corrupt WAL: + + ```bash + kubectl -n cpaas-system describe pod | grep -E 'Reason|Exit Code|OOMKilled' + kubectl -n cpaas-system logs -c prometheus --previous --tail=100 | grep -E 'invalid magic number|checksum mismatch|WAL' + ``` + + `invalid magic number` or `checksum mismatch` is a damaged segment that more memory does not repair. A long WAL replay with `Startup probe failed: HTTP probe failed with statuscode: 503` is covered in [Prometheus WAL replay slow or pod restarts before replay finishes on ACP](Prometheus_WAL_replay_slow_or_pod_restarts_before_replay_finishes_on_ACP.md). + +2. Double the Prometheus limit in the plugin configuration (for an `8000Mi` instance, 4 CPU / 16Gi stopped the restarts). + + > **Warning:** The new limit rolls the Prometheus Pod; queries stop until it is Ready, and a size that does not fit the node leaves it Pending. Precondition: the last termination is `OOMKilled` and the log shows no corrupt segment. Rollback: set the previous limit in the same form. + +### Cause 5: vm-operator is OOMKilled + +```bash +kubectl -n cpaas-system get pod | grep vm-operator +kubectl -n cpaas-system describe pod | grep -E 'Reason|Exit Code|OOMKilled' +``` + +> **Warning:** Raising `vm-operator` resources restarts the operator; VictoriaMetrics objects are not reconciled while it is down, while running `vmagent`, `vmselect` and `vmstorage` keep serving. Recreating the other monitoring Pods afterwards interrupts queries and scraping until each is Ready. Precondition: `vm-operator` is restarting with `OOMKilled`. Rollback: set the previous resources in the same form. + +Raise the `vm-operator` resources in the plugin configuration, then recreate the other monitoring Pods in `cpaas-system` one component at a time. Cluster monitoring displays normally afterwards. + +## Diagnostic Steps + +1. Describe the restarting Pod. `OOMKilled` on `vmagent`, with or without the queue-offset line, is Cause 1: + + ```bash + kubectl -n cpaas-system get pod | grep -E 'vmagent|vmstorage|vminsert|vmselect|kube-state|prometheus|vm-operator' + kubectl -n cpaas-system describe pod | grep -E 'Reason|Exit Code|Restart Count' + ``` + +2. High but stable usage on `vmstorage` with 0 restarts, or an `OOMKilled` `vmstorage` behind a `vmagent` that recovers only after restarts, is Cause 2. + +3. For kube-state, print the `/pod_nanny` flags before touching `resources` (Cause 3). + +4. When `vmagent` restarts with `502` on remote write and is not `OOMKilled`, count the certificate series in vmui before adding memory: + + ```promql + count({__name__="certificate_expires_time"}) + ``` + + A very large count is query overload, not this article. + +5. `prometheus` `OOMKilled` with no WAL checksum line is Cause 4. `vm-operator` `OOMKilled` is Cause 5. diff --git a/docs/en/solutions/Monitoring_plugin_install_or_redeploy_fails_ResourcePatch_conflict_IPv6_Service_mkdirOnNode_or_alertmanager-pv-0.md b/docs/en/solutions/Monitoring_plugin_install_or_redeploy_fails_ResourcePatch_conflict_IPv6_Service_mkdirOnNode_or_alertmanager-pv-0.md new file mode 100644 index 000000000..817784d46 --- /dev/null +++ b/docs/en/solutions/Monitoring_plugin_install_or_redeploy_fails_ResourcePatch_conflict_IPv6_Service_mkdirOnNode_or_alertmanager-pv-0.md @@ -0,0 +1,157 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Monitoring plugin install or redeploy fails: ResourcePatch conflict, IPv6 Service, mkdirOnNode, or alertmanager-pv-0 + +## Issue + +Installing, redeploying or running the platform monitoring plugin (Prometheus or VictoriaMetrics) fails in one of these ways: + +- A VictoriaMetrics redeploy fails because the new vmagent objects are not created; a VictoriaMetrics uninstall stays in Uninstalling; or the Prometheus plugin shows `部署中` (deploying) forever. +- The monitoring AppRelease fails with an error that requires an IPv6 address on an IPv4 single-stack cluster. +- VictoriaMetrics install fails in `mkdirOnNode` with an invalid Pod name that contains a space, such as `- -var-cpaas-monitoring-alertmanager`, and `metadata.name: Invalid value` against the RFC 1123 subdomain rule. +- A VictoriaMetrics install waits on an unbound PVC because PV `alertmanager-pv-0` was never created. +- On a standby global cluster with high-availability Prometheus, `prometheus-1` and `prometheus-2` are rebuilt again and again, and an alert reports Prometheus status 0. + +Related article: [Monitoring plugin uninstall or Prometheus to VictoriaMetrics switch leaves ModuleInfo, AppRelease, or Pods on ACP](Monitoring_plugin_uninstall_or_Prometheus_to_VictoriaMetrics_switch_leaves_ModuleInfo_AppRelease_or_Pods_on_ACP.md). + +## Root Cause + +### Cause 1: leftover ResourcePatch (rpch) records conflict with the monitoring objects + +Earlier changes to monitoring components left `rpch` records (sentry resource patches). They conflict with newly deployed objects, so a VictoriaMetrics redeploy does not create the new vmagent resources. The same records block other transitions: a VictoriaMetrics uninstall stays in Uninstalling while `Active` records of release `cpaas-system/victoriametrics` target VMAgent `agent`, VMCluster `cluster` and PersistentVolume `alertmanager-pv-0`. A Prometheus plugin stays in deploying when an `rpch` of release `cpaas-system/kube-prometheus` targets PersistentVolume `prometheus-pv-0` with a `jsonPatch` that adds `/metadata/deletionTimestamp` and `/metadata/deletionGracePeriodSeconds: 0`, so it keeps re-applying a delete to the PV. + +### Cause 2: a Kyverno policy writes IPv6 into Services on an IPv4 cluster + +The cluster has no IPv6. A Kyverno policy mutates `Service` objects and writes an IPv6 `ipFamilies` (or a similar IPv6 field), and the monitoring chart then fails to create its Service. + +### Cause 3: the LocalVolume storage path has a leading space + +The installer builds the temporary directory-init Pod name from the node name and the LocalVolume `storage.path`, and does not strip spaces. A value of `" /var/cpaas/monitoring"` instead of `/var/cpaas/monitoring` produces an invalid name. + +### Cause 4: an old Prometheus PVC still holds the alertmanager-pv name + +An old Prometheus PVC is still bound to a PV that uses the same name pattern, `alertmanager-pv-`, that the VictoriaMetrics deployment wants to create, so `alertmanager-pv-0` is not created and the new PVC stays unbound. + +### Cause 5: the standby global etcd sync overwrites prometheus-1 and prometheus-2 + +In a primary/standby global setup, `etcd-mirror` Pods in `kube-system` sync etcd from the primary to the standby. ConfigMap `etcd-sync-ignore-text` in `kube-system` has ignore entries only for `prometheus-0` (`/registry/secrets/cpaas-system/prometheus-kube-prometheus-0`, `...-0-tls-assets`, `...-0-web-config`, `/registry/monitoring.coreos.com/prometheuses/cpaas-system/kube-prometheus-0`), so the sync keeps overwriting the objects of the other two replicas. This is a known issue. + +## Resolution + +### Cause 1: leftover ResourcePatch (rpch) records conflict with the monitoring objects + +1. Check for Pods stuck in `Terminating`, then list and back up the monitoring `rpch` records on the business cluster: + + ```bash + kubectl get pods -A | grep -E 'vm|prometheus' | grep Terminating + kubectl get rpch -A | grep -Ei 'vm|prometheus' + kubectl get rpch -A -o yaml > /tmp/rpch-backup.yaml + kubectl get rpch -n -o yaml | grep -n -E 'release|kind|name:|deletionTimestamp' + ``` + +2. Delete only the records that belong to the monitoring release and target the objects that are not being created or deleted. + + > **Warning:** Deleting an `rpch` drops that sentry patch; an intentional customization stored only in it is gone on the next reconcile. Precondition: the deploy, redeploy or uninstall is not progressing, and the records belong to VictoriaMetrics or Prometheus monitoring objects. Rollback: `kubectl apply -f /tmp/rpch-backup.yaml`. + + ```bash + kubectl delete rpch -n + ``` + +3. The components are created and reach Running, the uninstall completes, or the plugin changes to `部署成功` (deployed). Retry the redeploy if it does not continue by itself. + +### Cause 2: a Kyverno policy writes IPv6 into Services on an IPv4 cluster + +1. Find the policy whose mutation writes IPv6 into Services: + + ```bash + kubectl get policy,clusterpolicy -A + kubectl -n cpaas-system get svc -o yaml | grep -n -i ipfamil + ``` + +2. Remove that policy, or rewrite its mutation so it no longer injects IPv6. + + > **Warning:** Deleting a Kyverno policy stops that control for every object it matched, not only the monitoring Service. Precondition: the policy YAML is saved and its mutation is the IPv6 field the AppRelease rejected. Rollback: `kubectl apply -f /tmp/kyverno-policy-backup.yaml`. + + ```bash + kubectl get clusterpolicy -o yaml > /tmp/kyverno-policy-backup.yaml + kubectl delete clusterpolicy + ``` + + Use `policy -n ` for a namespaced policy. The next reconcile creates the Service with IPv4 only and the AppRelease becomes Ready. + +### Cause 3: the LocalVolume storage path has a leading space + +The init Pod was rejected by the API, so there is nothing to delete. Correct the LocalVolume storage path in the VictoriaMetrics install parameters (`spec.config.storage.path` on the ModuleInfo) so it has no leading or trailing space, then install again: + +```text +storage.path: /var/cpaas/monitoring +``` + +The default path is `/cpaas/monitoring`; on Alauda OS it is `/var/cpaas/monitoring`. + +### Cause 4: an old Prometheus PVC still holds the alertmanager-pv name + +```bash +kubectl get pvc -A | grep -Ei 'alertmanager|prome' +kubectl get pv | grep alertmanager-pv +``` + +> **Warning:** Deleting the old Prometheus PVC also deletes its PV, and the old Alertmanager data on it is gone. Precondition: the PV `alertmanager-pv-` is bound to a PVC of the earlier Prometheus deployment and the VictoriaMetrics PVC waits for that name. Copy the data off first if it is needed. There is no rollback after the PV is deleted. + +```bash +kubectl -n delete pvc +``` + +The VictoriaMetrics deployment then creates `alertmanager-pv-` itself and the PVC binds. + +### Cause 5: the standby global etcd sync overwrites prometheus-1 and prometheus-2 + +1. Check the ignore list: + + ```bash + kubectl get cm -n kube-system etcd-sync-ignore-text -o yaml | grep prome + ``` + +2. Recommended: reinstall Prometheus on that global cluster as single-node instead of high-availability. + + > **Warning:** Reinstalling Prometheus stops monitoring on that cluster until the new install is Ready, and history is not carried over unless the volumes are reused. Precondition: the rebuilt Pods are `prometheus-1` and `prometheus-2` on the standby global and the ignore list lacks them. Rollback: reinstall in the previous mode. + +3. Alternative: add ignore entries for `prometheus-1` and `prometheus-2` under `data.ignore-equal.txt`, copying the format of the existing `prometheus-0` entries. + + > **Warning:** Editing `etcd-sync-ignore-text` creates an `rpch`, which may block later upgrades. Precondition: same signature as above, and a saved copy. Rollback: `kubectl apply -f /tmp/etcd-sync-ignore-text.yaml`, then remove the generated `rpch`. + + ```bash + kubectl get cm -n kube-system etcd-sync-ignore-text -o yaml > /tmp/etcd-sync-ignore-text.yaml + kubectl edit cm -n kube-system etcd-sync-ignore-text + ``` + +## Diagnostic Steps + +1. Read the module and release state: + + ```bash + kubectl get minfo -A | grep -Ei 'prometheus|victoriametrics' + kubectl -n cpaas-system get ars + kubectl get rpch -A | grep -Ei 'vm|prometheus' + ``` + +2. A redeploy, uninstall or "deploying" state that does not progress, with monitoring `rpch` records present: Cause 1. A record whose `jsonPatch` adds `deletionTimestamp` to a PV is the deploying variant. + +3. An AppRelease error about IPv6 on a single-stack cluster: Cause 2. + +4. `mkdirOnNode` and an RFC 1123 name with a space in it: Cause 3. Expected after the fix: the install proceeds past `mkdirOnNode`. + +5. A PVC waiting for `alertmanager-pv-0` while an old Prometheus PVC holds an `alertmanager-pv-` volume: Cause 4. + +6. `prometheus-1` and `prometheus-2` rebuilt in a loop only on the standby global, with only `prometheus-0` in `etcd-sync-ignore-text`: Cause 5. diff --git a/docs/en/solutions/Monitoring_plugin_uninstall_or_Prometheus_to_VictoriaMetrics_switch_leaves_ModuleInfo_AppRelease_or_Pods_on_ACP.md b/docs/en/solutions/Monitoring_plugin_uninstall_or_Prometheus_to_VictoriaMetrics_switch_leaves_ModuleInfo_AppRelease_or_Pods_on_ACP.md new file mode 100644 index 000000000..7853fc0f2 --- /dev/null +++ b/docs/en/solutions/Monitoring_plugin_uninstall_or_Prometheus_to_VictoriaMetrics_switch_leaves_ModuleInfo_AppRelease_or_Pods_on_ACP.md @@ -0,0 +1,182 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Monitoring plugin uninstall or Prometheus to VictoriaMetrics switch leaves ModuleInfo, AppRelease, or Pods on ACP + +## Issue + +A monitoring plugin is being uninstalled, or a cluster is being moved from Prometheus to VictoriaMetrics, and the platform does not reach a clean state: + +- The Prometheus uninstall never finishes: the global `ModuleInfo` (`minfo`) stays `deleting`, while `prometheus-operator` on the workload cluster is already gone. +- The functional-component page still says Prometheus Operator is abnormal after the cluster moved to VictoriaMetrics, and the namespace shown on the `minfo` contains no operator objects. +- VictoriaMetrics is installed and Running, but a Prometheus AppRelease (`ars`) is still `Syncing`, and Prometheus Pods such as `kube-prometheus-grafana`, `kube-prometheus-thanos-query`, `prometheus-operator` or `cpaas-monitor-prometheus-adapter` remain. +- All VictoriaMetrics Pods are Running, but the VictoriaMetrics `minfo` on global stays `Processing`. +- Installing VictoriaMetrics fails on a taint that says Prometheus is already running, although Marketplace lists Prometheus as not installed; or two VictoriaMetrics instances run on one cluster. + +Plugin state is the global `ModuleInfo` plus the workload-cluster `AppRelease` in `cpaas-system`; `sentry` applies the releases, and `cluster-transformer` recreates a `minfo` that is deleted while it is running. + +Related articles: [Monitoring plugin install or redeploy fails: ResourcePatch conflict, IPv6 Service, mkdirOnNode, or alertmanager-pv-0](Monitoring_plugin_install_or_redeploy_fails_ResourcePatch_conflict_IPv6_Service_mkdirOnNode_or_alertmanager-pv-0.md), [AppRelease abnormal because a ModuleInfo of a deleted cluster remains on ACP](AppRelease_abnormal_because_a_ModuleInfo_of_a_deleted_cluster_remains_on_ACP.md). + +## Root Cause + +### Cause 1: the post-delete job times out on a very large set of PrometheusRules + +The uninstall job and the chart cleanup script walk every `PrometheusRule`. With a very large set the job hits its deadline and retries forever, so the `minfo` stays `deleting`. Pods of the old release can keep running after the AppRelease is gone, and if the `Alertmanager` and `Prometheus` custom resources were not deleted, a `prometheus-operator` that comes back adopts them again and replica counts look wrong. + +### Cause 2: the Prometheus Operator plugin is still installed after the move to VictoriaMetrics + +The `minfo` still points at the old operator namespace, but that namespace has no Prometheus Operator Pod, CSV or PackageManifest (seen on 3.18.2 with VictoriaMetrics live). Restarting `sentry`, editing the `minfo` version, or patching an unrelated AppRelease does not clear it. Deleting the `minfo` while `cluster-transformer` runs makes it come back; deleting it with `cluster-transformer` stopped fails the `vmoduleinfo.kb.io` webhook. Removing the Prometheus Operator plugin itself clears the row. + +### Cause 3: a previous Prometheus uninstall left its AppRelease behind its finalizers + +The VictoriaMetrics `minfo` is `Running`, global has no Prometheus `minfo`, and on the workload cluster the Prometheus `ars` is still `Syncing` or `Ready` with a status saying the kube-prometheus chart is Installing. The earlier uninstall did not finish, and the `ars` stays until its `finalizers` are removed. + +### Cause 4: a leftover Prometheus monitor AppRelease keeps the VictoriaMetrics minfo in Processing + +A Prometheus uninstall got stuck during an earlier upgrade; its `minfo` was deleted by hand but its AppReleases were not. VictoriaMetrics was then deployed on top of that state, so its `minfo` never leaves `Processing`, and a normal VictoriaMetrics uninstall cannot complete either. + +### Cause 5: Prometheus and VictoriaMetrics cannot run side by side + +The platform Prometheus and VictoriaMetrics share the same node-exporter, so they are not allowed to coexist and the install conflict is expected. After an upgrade from 3.18 to 4.1, the Prometheus Pods were still running while Marketplace listed Prometheus as not installed. Two platform VictoriaMetrics instances on one 4.2 global cluster made the data services page show wrong data. A second, self-installed open-source VictoriaMetrics next to the platform one uses the same upstream CRDs and may cause data conflicts; it is not recommended. + +## Resolution + +### Cause 1: the post-delete job times out on a very large set of PrometheusRules + +1. Count the rules and read the module state: + + ```bash + kubectl get prometheusrule -A --no-headers | wc -l + kubectl get moduleinfo -A | grep -i prometheus + kubectl -n cpaas-system get pod | grep -Ei 'prometheus|alertmanager|sentry' + ``` + +2. Back up the rules off the cluster, then delete them per namespace that holds a large share. + + > **Warning:** Deleting `PrometheusRule` objects removes those alerts until they are restored, and the backup file is the only rollback. Precondition: `/tmp/prometheusrule-backup.yaml` is copied off the cluster, and missing alerts are acceptable for the window. Do not delete `cpaas-cluster-rules` unless it is part of the blocking set and is in the backup. Rollback: `kubectl apply -f /tmp/prometheusrule-backup.yaml`. + + ```bash + kubectl get prometheusrule -A -o yaml > /tmp/prometheusrule-backup.yaml + kubectl delete prometheusrule -n --all + ``` + +3. Let `sentry` retry the post-delete job. If it does not retry, recreate the `sentry` Pod. + + > **Warning:** Recreating `sentry` pauses module installs and uninstalls platform-wide until it is Ready. Precondition: the rule backlog is already deleted, or the job times out again. Rollback is the controller starting a new Pod. + + ```bash + kubectl get pod -A | grep sentry + kubectl -n delete pod + ``` + +4. Remove leftover operator objects so a later install does not adopt them. + + > **Warning:** Deleting the `Prometheus` and `Alertmanager` resources tears down their StatefulSets and stops scraping and notification. Precondition: this is the uninstall you intended, and the data on the PVC is expendable or copied. Rollback: reinstall the plugin, which creates new resources. + + ```bash + kubectl -n cpaas-system get prometheus,alertmanager + kubectl -n cpaas-system delete prometheus + kubectl -n cpaas-system delete alertmanager + ``` + +### Cause 2: the Prometheus Operator plugin is still installed after the move to VictoriaMetrics + +```bash +kubectl get minfo -A | grep -i prometheus +kubectl get pod,csv,packagemanifest -A | grep -i prometheus +kubectl -n cpaas-system get pod | grep -Ei 'cluster-transformer|sentry' +``` + +> **Warning:** Removing the Prometheus Operator plugin deletes any operands it still has. Precondition: VictoriaMetrics already serves the cluster, and any `PrometheusRule` you still need is backed up (Cause 1). Rollback: install the plugin again from the catalog. + +Uninstall the Prometheus Operator plugin through the platform with `cluster-transformer` left running; do not delete the `minfo` by hand. If a VictoriaMetrics install in the same window fails on a `webhook.cert-manager.io` `context deadline exceeded` timeout, repair the cert-manager webhook before retrying. + +### Cause 3: a previous Prometheus uninstall left its AppRelease behind its finalizers + +1. Confirm VictoriaMetrics is the live stack and the Prometheus `ars` is the leftover: + + ```bash + kubectl get minfo -A | grep -Ei 'prometheus|victoriametrics' + kubectl -n cpaas-system get ars + kubectl get pods -A -o wide | grep prometheus | grep -iv exporter + ``` + +2. Save the `ars`, clear its finalizers and delete it. + + > **Warning:** This drops the leftover Prometheus release, including its Grafana, Thanos query, operator and adapter, and any scraping that still depended on it. Precondition: the VictoriaMetrics `minfo` is `Running`, no Prometheus `minfo` exists, and the `ars` shows the old kube-prometheus chart Installing. Rollback after a successful delete: install the Prometheus plugin again; old Prometheus data is reattached only if its PVC still exists. + + ```bash + kubectl -n cpaas-system get ars -o yaml > /tmp/prometheus-ars.yaml + kubectl -n cpaas-system patch ars --type=json -p '[{"op":"remove","path":"/metadata/finalizers"}]' + kubectl -n cpaas-system delete ars --wait=false + ``` + +3. Delete the leftover Prometheus Pods and Jobs from the list (for example `cpaas-monitor-prometheus-adapter`, `kube-prometheus-grafana`, `kube-prometheus-thanos-query`, `prometheus-operator`). Skip every name that belongs to the VictoriaMetrics release. + + > **Warning:** Deleting them stops the old Prometheus query path and adapter. Precondition: each name is in the leftover list and vmui already returns samples. Rollback: reinstall Prometheus. + + ```bash + kubectl -n cpaas-system delete pod + ``` + +### Cause 4: a leftover Prometheus monitor AppRelease keeps the VictoriaMetrics minfo in Processing + +1. Identify the leftover monitor `ars` created by the earlier Prometheus install: + + ```bash + kubectl get minfo -A | grep -Ei 'victoriametrics|prometheus' + kubectl -n cpaas-system get ars + kubectl -n cpaas-system get ars -o yaml | grep -n -i -E 'chart|prometheus' + ``` + +2. Save and delete the VictoriaMetrics `minfo` (on global), the VictoriaMetrics `ars` and the leftover monitor `ars` (on the workload cluster), then deploy VictoriaMetrics again. + + > **Warning:** This removes VictoriaMetrics monitoring of that cluster until it is redeployed; data on a retained volume is reattached only if the same PVC is reused. Precondition: the VictoriaMetrics `minfo` is stuck in `Processing`, a Prometheus monitor `ars` predates it, and a normal uninstall cannot complete. Rollback: redeploy the plugin; the saved YAML is for reference, not for reapplying the stuck state. + + ```bash + kubectl get minfo -o yaml > /tmp/vm-minfo.yaml + kubectl -n cpaas-system get ars -o yaml > /tmp/vm-ars.yaml + kubectl -n cpaas-system get ars -o yaml > /tmp/monitor-ars.yaml + kubectl delete minfo + kubectl -n cpaas-system delete ars + ``` + + If an object hangs on finalizers, handle it as in Cause 3. + +### Cause 5: Prometheus and VictoriaMetrics cannot run side by side + +```bash +kubectl get minfo -A | grep -Ei 'prometheus|victoriametrics' +kubectl get ds -A | grep -i exporter-node +``` + +Run one platform monitoring solution per cluster. Switch by uninstalling the current plugin through the platform before installing the other; clean up leftovers of an incomplete switch with Causes 3 and 4 rather than deleting the Prometheus DaemonSet and Pods by hand, which left the overview without cluster metrics. For two platform VictoriaMetrics instances on global, uninstall the duplicate. For a business-owned second VictoriaMetrics, do not install it on a cluster that already runs the platform one. + +## Diagnostic Steps + +1. Read the module phase on global and the AppReleases on the workload cluster before restarting anything: + + ```bash + kubectl get minfo -A + kubectl -n cpaas-system get ars + ``` + +2. `deleting` plus a large `PrometheusRule` count: Cause 1. + +3. An abnormal Prometheus Operator row with an empty namespace while VictoriaMetrics is live: Cause 2. + +4. VictoriaMetrics `minfo` `Running`, no Prometheus `minfo`, and a Prometheus `ars` still Installing: Cause 3. + +5. VictoriaMetrics `minfo` stuck `Processing` with a leftover Prometheus monitor `ars`: Cause 4. + +6. A taint that says Prometheus is running, or two VictoriaMetrics installs: Cause 5. Expected after cleanup: exactly one monitoring `minfo` per cluster, `Running`. diff --git a/docs/en/solutions/Node_zombie_process_alert_keeps_switching_between_firing_and_resolved_on_ACP.md b/docs/en/solutions/Node_zombie_process_alert_keeps_switching_between_firing_and_resolved_on_ACP.md new file mode 100644 index 000000000..c7f5cd9fb --- /dev/null +++ b/docs/en/solutions/Node_zombie_process_alert_keeps_switching_between_firing_and_resolved_on_ACP.md @@ -0,0 +1,68 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x +--- + +# Node zombie process alert keeps switching between firing and resolved on ACP + +## Issue + +Notifications for the node zombie process alert keep arriving as firing (`告警中`) and then resolved (`已恢复`), again and again, although the zombie process count seems unchanged. The rule is in the `uni-node-rules` alert policy: `节点的僵尸进程数 > 10 且持续 1 分钟` (number of zombie processes on the node > 10 for 1 minute). The chart of the rule in the policy may show `无数据,请确认表达式涉及到的监控指标是否存在` (no data, check whether the metrics in the expression exist), which suggests a monitoring problem. Changing the alert sending interval does not stop the alternation. + +Related article: [Alert repeat interval cannot be set to 4 hours, or the global sending interval does not take effect on ACP](Alert_repeat_interval_cannot_be_set_to_4_hours_or_the_global_sending_interval_does_not_take_effect_on_ACP.md). + +## Root Cause + +### Cause 1: the zombie process count on the node really crosses the threshold back and forth + +The rule expression is `node_processes_state{state="Z"}>10`. Zombie processes on the node appear and disappear intermittently, so the count goes above 10 for more than a minute, falls below it, and rises again. Monitoring collects the metric normally, and each crossing correctly produces a firing and then a resolved notification. The "no data" message on the policy chart does not mean that the metric is missing: the same expression returns data in vmui. + +## Resolution + +### Cause 1: the zombie process count on the node really crosses the threshold back and forth + +1. Confirm that the metric is collected and see how it moves. In vmui on that cluster, run the rule expression and the raw series over the last hours: + + ```promql + node_processes_state{state="Z"} > 10 + node_processes_state{state="Z"} + ``` + + Values above 10 on the node named in the alert (for example 18 or 128), alternating with lower values, confirm this cause. + +2. Confirm that the notifications match those crossings. The `courier-api` log in `cpaas-system` records each firing and resolved event of the alert: + + ```bash + kubectl get pods -n cpaas-system | grep courier-api + kubectl logs -n cpaas-system --tail 10000 | grep '' + ``` + + Expected: entries for alert `节点的僵尸进程数` with `alert_resource: uni-node-rules`, `state: "Z"`, a current value such as `"11"`, and `status: "resolved"` alternating with firing entries at the times the notifications arrived. + +3. Find out why zombie processes keep appearing on the node. A zombie is a child process that has exited but has not been reaped by its parent; on the node, list them with their parents: + + ```bash + ps -eo pid,ppid,stat,comm | awk '$3 ~ /^Z/' + ``` + + The parent process (`ppid`) is the one that does not reap its children. Exec probes are a common source; see [Defunct Processes From Exec Probes Driving Worker Node Pressure](Defunct_Processes_From_Exec_Probes_Driving_Worker_Node_Pressure.md). Fix or restart the owning workload. + +4. If the alternation is expected for that node and the alert should be quieter, change the rule's trigger condition (operator, value and duration) inside the alert policy, as described in [Create and manage alert policies and notification routing on ACP](Create_and_manage_alert_policies_and_notification_routing_on_ACP.md). + +## Diagnostic Steps + +1. Run `node_processes_state{state="Z"}` in vmui for the node in the alert. Data present and crossing 10 repeatedly: Cause 1. No data at all in vmui: a collection problem, not this article. + +2. Compare the notification times with the `courier-api` log entries for the alert (Resolution step 2). Matching firing/resolved pairs show that the alerting chain works as designed. + +3. On the node, count zombies a few times over several minutes: + + ```bash + for i in 1 2 3 4 5; do ps -eo stat | grep -c '^Z'; sleep 60; done + ``` + + Expected for this cause: counts that move above and below 10. diff --git a/docs/en/solutions/Workload_changes_reverted_not_applied_or_restarting_Pods_OAM_Helm_template_sync_StatefulSet_partition_reloader_on_ACP.md b/docs/en/solutions/Workload_changes_reverted_not_applied_or_restarting_Pods_OAM_Helm_template_sync_StatefulSet_partition_reloader_on_ACP.md new file mode 100644 index 000000000..727a05f45 --- /dev/null +++ b/docs/en/solutions/Workload_changes_reverted_not_applied_or_restarting_Pods_OAM_Helm_template_sync_StatefulSet_partition_reloader_on_ACP.md @@ -0,0 +1,165 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.12.x + - 3.14.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Workload changes reverted, not applied, or restarting Pods: OAM, Helm template sync, StatefulSet partition, reloader on ACP + +## Issue + +A workload is changed and the cluster does something else: + +- A manual edit of a Deployment that an OAM application created is reverted. +- A manual edit of the `resources` of a Deployment in an OAM application without `apply-once` is not reverted, while the controller log keeps printing `patching object`. +- A native application or a Helm-deployed application (for example Redis) is updated without anyone touching it: its history shows a new revision, its Pods restart several times, and manual configuration changes are lost. +- An application deployed with Helm is now maintained by editing its Deployments in the UI, and the question is whether it must be detached from its template. +- After a parameter change in the console, a StatefulSet Pod does not restart. +- Updating a ConfigMap makes a Deployment restart by itself. + +Each case is a controller doing what it is configured to do, or a known product defect. + +## Root Cause + +### Cause 1: the OAM application restores the Deployment it rendered + +An OAM application with a custom template manages the service lifecycle and restores the Deployment it rendered to the template's desired state (seen on 4.0.2). An in-place edit of a ComponentDefinition does not reach existing applications, which keep the version they reference, and there is no dry-run interface to preview the render of a new template. + +### Cause 2: OAM synchronizes only the fields the template defines + +The application controller re-applies the template periodically; its log repeats `"patching object" name="" resource="apps/v1, Kind=Deployment"` for the Service, PersistentVolumeClaim, ConfigMap, Deployment and `StartStopScalerTrait` of the application, for example every five minutes. Fields that the template does not define, such as `resources` in this case, are outside the synchronized state, so a manual value stays. + +### Cause 3: a captain restart re-synchronizes Helm-based applications and redeploys them + +When `captain` (captain-controller) restarts, a defect in its check for whether a HelmRequest changed triggers a resynchronization of the downstream Helm templates, which re-applies the application's resources. The result is a new application revision without user action, Pods that restart, and manual changes that are reverted. The Kubernetes audit log then shows `patch` requests on the application's resources, for example `/api/v1/namespaces//services/?fieldManager=manager`, by user `system:serviceaccount:kube-system:admin` with user agent `Go-http-client/2.0`, setting `app.kubernetes.io/managed-by: Helm` on objects annotated with `meta.helm.sh/release-name`. A Redis StatefulSet hit this way can also leave a deleted Pod outside the StatefulSet's management, so the new Pod cannot start. The defect is in ACP 3.12 and 3.14 and is fixed in 3.15 and later. + +### Cause 4: a Helm-deployed application maintained in the UI must be detached from its template + +As long as the application is still attached to its Helm template, the Helm-managed state (the values) can be applied again, for example when `captain` on the global cluster restarts (Cause 3), and UI edits are overwritten. + +### Cause 5: rollingUpdate.partition is higher than the Pod ordinal + +With `updateStrategy.type: RollingUpdate` and `rollingUpdate.partition: `, only Pods whose ordinal is greater than or equal to the partition are updated when the template changes. A StatefulSet with one replica (ordinal 0) and `partition: 4` replaces no Pod after a change, for example to a ConfigMap script mounted as `/scripts/start.sh` with `subPath: start.sh`. + +### Cause 6: the workload carries reloader.stakater.com/auto: "true" + +That annotation makes the workload restart whenever its ConfigMap changes, so it reads the new configuration. Some platform component charts set it in their packages; on business workloads it was added by the workload owner. There is no global switch. + +## Resolution + +### Cause 1: the OAM application restores the Deployment it rendered + +1. Export the application, then add the `apply-once` policy so the rendered Deployments can be changed while the template keeps publishing: + + ```bash + kubectl get applications.core.oam.dev -n -o yaml > "$HOME/-oam.yaml" + kubectl edit applications.core.oam.dev -n + ``` + + ```yaml + spec: + policies: + - name: apply-once + type: apply-once + properties: + enable: true + ``` + + > **Warning:** With `apply-once` the application no longer restores drift; manual edits stay until the next publish changes those fields. Precondition: direct edits to the rendered workloads are intended. Rollback: apply the exported YAML, which removes the policy and brings back the revert. + +2. For a template change, keep the existing ComponentDefinition and publish a new versioned one instead of editing it in place: keep `web-service-v1`, publish `web-service-v2` with the new parameters (for example CPU and memory defaults `*"500m"` and `*"512Mi"`), then change the component `type` in the Application from `web-service-v1` to `web-service-v2` and apply. The Deployment is re-rendered; the Application does not have to be deleted. + +Detaching the application from its template also stops the revert, but gives up the template. + +### Cause 2: OAM synchronizes only the fields the template defines + +Expected behavior. Check which fields the template defines: + +```bash +kubectl get pods -A | grep -i vela +kubectl logs -n --tail=200 | grep -F 'patching object' +kubectl get applications.core.oam.dev -n -o yaml | grep -n -A5 'resources' +``` + +To have the template enforce `resources`, define them in the OAM template. + +### Cause 3: a captain restart re-synchronizes Helm-based applications and redeploys them + +1. Match the unexpected revision with a `captain` restart and with the audit log entries described above: + + ```bash + kubectl get pods -n cpaas-system | grep captain + kubectl get helmrequest -A | grep + ``` + +2. Stop template sync for the affected application by annotating its HelmRequest (in the console: Platform Management > Cluster > Resource Management, search for the `helmrequest`, edit its YAML; or with `kubectl` on the cluster that holds it): + + ```bash + kubectl get helmrequest -n -o yaml > "$HOME/-hr.yaml" + kubectl annotate helmrequest -n captain-no-sync="true" + ``` + + > **Warning:** `captain-no-sync: "true"` stops captain from synchronizing this HelmRequest, so later template or values changes are not deployed until it is removed. Precondition: the syncs of this HelmRequest restart the application. Rollback: `kubectl annotate helmrequest -n captain-no-sync-`; the next sync can redeploy the application and restart its Pods. + +3. Long term: upgrade to 3.15 or later, where the defect is fixed. For Redis, prefer Operator mode over Helm mode, and keep the main configuration of business charts in values so a resync does not lose it. + +### Cause 4: a Helm-deployed application maintained in the UI must be detached from its template + +Use the application's detach-from-template (`脱离模板`) action before maintaining its Deployments and StatefulSets directly in the UI. Afterwards Helm values are no longer applied to it. + +### Cause 5: rollingUpdate.partition is higher than the Pod ordinal + +```bash +kubectl get sts -n -o jsonpath='{.spec.replicas}{" "}{.spec.updateStrategy}{"\n"}' +kubectl get sts -n -o jsonpath='{.status.currentRevision}{" "}{.status.updateRevision}{"\n"}' +``` + +A partition greater than or equal to the replica count, with `currentRevision` different from `updateRevision`, means no Pod moves to the new template. + +> **Warning:** Setting the partition to 0 rolls every Pod of the StatefulSet to the new template, one at a time from the highest ordinal. Precondition: the partition was not left in place on purpose for a staged rollout, and the restarts are acceptable. Rollback: set the partition back; Pods already replaced stay on the new revision. + +```bash +kubectl patch sts -n --type merge -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":0}}}}' +kubectl rollout status sts -n +``` + +### Cause 6: the workload carries reloader.stakater.com/auto: "true" + +```bash +kubectl get deployment,sts,ds -A -o yaml | grep -n reloader.stakater.com +``` + +Remove the annotation from each business workload that should not restart on ConfigMap changes; leave the platform components as their charts set them. + +> **Warning:** Without the annotation, a ConfigMap change no longer restarts the workload, and an application that reads its configuration only at start keeps the old values until it is restarted another way. Rollback: add the annotation again. + +```bash +kubectl annotate deployment -n reloader.stakater.com/auto- +``` + +## Diagnostic Steps + +```bash +kubectl get deploy,sts -n -o yaml | grep -n -E 'ownerReferences|reloader|partition|captain|app.oam.dev|meta.helm.sh' +kubectl get helmrequest -A | grep +kubectl get applications.core.oam.dev -A | grep +``` + +```text +edit of an OAM-rendered Deployment reverted -> Cause 1 +edit of resources not reverted, controller logs patching object -> Cause 2 +new revision or Pod restarts with no user action, right after captain restart -> Cause 3 +Helm-deployed application to be maintained in the UI -> Cause 4 +StatefulSet change did not roll the Pod, partition >= replicas -> Cause 5 +ConfigMap update restarts the workload, reloader annotation present -> Cause 6 +``` + +For Cause 3, the audit log entries by `system:serviceaccount:kube-system:admin` with `fieldManager=manager` at the time of the new revision, and no user request, confirm the resync. From 499d4d16ae503325471b9f08f840b51c443b146f Mon Sep 17 00:00:00 2001 From: Komh Date: Wed, 18 Nov 2026 17:41:25 +0800 Subject: [PATCH 20/25] docs(solutions): add 6 and update 2 troubleshooting articles --- ...-design_short-lived_certificates_on_ACP.md | 177 +++++++++++++ ...luster-enhancer-apiserver_metric_on_ACP.md | 182 +++++++++++++ ...L_PROTOCOL_ERROR_or_TLS_alert_80_on_ACP.md | 186 +++++++++++++ ...wn_authority_or_certificate_has_expired.md | 8 + ..._exit_status_32_or_access_denied_on_ACP.md | 245 ++++++++++++++++++ ...f_the_volume_directory_is_denied_on_ACP.md | 124 +++++++++ ...or_upstream_load_balancer_update_on_ACP.md | 143 ++++++++++ ...nager_or_scheduler_health_alerts_on_ACP.md | 34 ++- 8 files changed, 1098 insertions(+), 1 deletion(-) create mode 100644 docs/en/solutions/Certificate_alerts_from_a_custom_VMRule_ait.alauda.io_webhook_denials_or_by-design_short-lived_certificates_on_ACP.md create mode 100644 docs/en/solutions/Certificate_page_empty_or_stale_because_of_clock_drift_or_a_stale_cluster-enhancer-apiserver_metric_on_ACP.md create mode 100644 docs/en/solutions/HTTPS_to_a_published_service_fails_with_plain_HTTP_sent_to_HTTPS_port_ERR_SSL_PROTOCOL_ERROR_or_TLS_alert_80_on_ACP.md create mode 100644 docs/en/solutions/NFS_PVC_stays_Pending_csi-nfs-controller_mount_fails_with_rpc.statd_exit_status_32_or_access_denied_on_ACP.md create mode 100644 docs/en/solutions/NFS_PVC_stays_Pending_csi-nfs-controller_mount_times_out_or_mkdir_of_the_volume_directory_is_denied_on_ACP.md create mode 100644 docs/en/solutions/Old_business_TLS_certificate_still_served_after_an_Ingress_ALB_rule_or_upstream_load_balancer_update_on_ACP.md diff --git a/docs/en/solutions/Certificate_alerts_from_a_custom_VMRule_ait.alauda.io_webhook_denials_or_by-design_short-lived_certificates_on_ACP.md b/docs/en/solutions/Certificate_alerts_from_a_custom_VMRule_ait.alauda.io_webhook_denials_or_by-design_short-lived_certificates_on_ACP.md new file mode 100644 index 000000000..c568cd038 --- /dev/null +++ b/docs/en/solutions/Certificate_alerts_from_a_custom_VMRule_ait.alauda.io_webhook_denials_or_by-design_short-lived_certificates_on_ACP.md @@ -0,0 +1,177 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.0" + - "4.2" + - "4.3" +--- + +# Certificate alerts from a custom VMRule, ait.alauda.io webhook denials, or by-design short-lived certificates on ACP + +## Issue + +A certificate-related alert keeps firing although no platform certificate is actually expired. One of the following is seen: + +- The alert `平台组件证书已到期 <= 0d 且持续 1 分钟` ("platform component certificates have expired <= 0d and persisted for 1 minute") stays active after a minor upgrade. `openssl` shows the platform certificates are valid, AlertRule `cpaas-certificates-rule` has already been deleted, and restarting the monitoring components does not stop the alert. +- kube-apiserver logs cert-manager webhook denials that reference `{cpaas-ca ClusterIssuer ait.alauda.io}`, or the custom alert "apiserver read request success rate < 0.9" fires on a workload cluster and most failed reads are `403`. +- A cluster with KubeVirt virtualization always has one certificate with one day of validity left, so the "about to expire" alert never clears. +- An external monitor such as Zabbix reports Pod `node-local-certs-exporter-` in `cpaas-system` as not healthy (or failed after a scale-out), and `kubectl get pod -A` cannot find the Pod. + +Certificate expiry metrics (`certificate_expires_time`) are produced by `cluster-enhancer-apiserver`; the certificate page and alert status are served through `courier` on the `global` cluster. + +Related articles: *Certificate page empty or stale because of clock drift or a stale cluster-enhancer-apiserver metric on ACP* (the page is empty, or still shows a certificate that was renewed or removed). For a certificate that really is about to expire, see [Operator service certificates such as packageserver-service-cert about to expire or not renewed on ACP](Operator_service_certificates_such_as_packageserver-service-cert_about_to_expire_or_not_renewed_on_ACP.md). On ACP 4.1, an alert detail that shows no data because `cpaas-certificates-rule` queries time out is Cause 3 of [Certificates tab missing, platform domain certificate not listed, or certificate alert shows no data on ACP](Certificates_tab_missing_platform_domain_certificate_not_listed_or_certificate_alert_shows_no_data_on_ACP.md). + +## Root Cause + +### Cause 1: a custom VMRule still emits cpaas-certificates-rule after the AlertRule is gone + +On a VictoriaMetrics cluster, `vmalert` evaluates the rules from `VMRule` objects, which are rendered into the rule-file ConfigMaps mounted into the `vmalert` Pod. When a custom `VMRule` (one that is not generated from the platform `AlertRule`) carries the `cpaas-certificates-rule` rules, deleting AlertRule `cpaas-certificates-rule` does not remove them: the rule-file ConfigMap still contains the rules, `vmalert` keeps producing the alert, and `courier-api` keeps logging it as coming from `vmalert`. Restarting `enhancer-api`, the monitoring controller, `vmalert`, `alertmanager`, or `courier-api` does not change this. Deleting the custom `VMRule` clears the alert. + +The same symptom also appears when the AlertRule delete itself never took effect, for example because the cluster was broken and its monitoring plugin could not reconcile the change. In that case no `rpch` object exists for the change (`kubectl get rpch -A`). Either way, the fix is to remove the `VMRule` that still carries the rule. + +### Cause 2: cert-manager webhook denials for {cpaas-ca ClusterIssuer ait.alauda.io} count as 403 read failures + +Some platform certificates are deliberately not issued by cert-manager, but the platform still uses cert-manager's certificate expiry tracking for them. To do that, it registers a non-existent API group, `ait.alauda.io`, as the issuer group. Requests for that group are rejected by the cert-manager webhook, so kube-apiserver logs denials that reference `{cpaas-ca ClusterIssuer ait.alauda.io}`. + +The rejections are `403` responses and count as failed reads. A custom success-rate alert such as: + +```text +sum(irate(apiserver_request_total{code=~"20.*",verb=~"GET|LIST"}[2m]))/sum(irate(apiserver_request_total{verb=~"GET|LIST"}[2m]))<0.9 +``` + +then oscillates around 0.9 on a workload cluster. On the affected versions this behavior is expected and cannot be avoided; when cert-manager runs normally and no certificate is actually expired, the log lines can be ignored. + +### Cause 3: cdi-uploadserver-client-cert on a KubeVirt cluster is valid for one day by design + +On a cluster with KubeVirt virtualization, `cdi-uploadserver-client-cert` has a one-day lifetime and is renewed daily. It is the client TLS certificate for VM image upload, used only for mutual TLS between `cdi-uploadproxy` (or the upload client) and `cdi-uploadserver` when an ISO or qemu image is uploaded into a DataVolume. That traffic is short-lived, so the short lifetime is intended. The certificate is active and not deprecated. The platform cannot silence the alert for one individual certificate, so the "about to expire" alert keeps firing for it. + +### Cause 4: the node-local-certs-exporter Pod is short-lived and no longer exists + +`node-local-certs-exporter-` belongs to Cluster Enhancer and inspects node certificates. It runs one check, and once it has collected the certificate information it is cleaned up automatically. An external monitor that discovers Pods through kube-apiserver can catch it while it runs, or in a final failed state, and then keep alerting on a Pod that no longer exists. Its absence is normal. Why a single run ended in a failed state cannot be determined after the Pod has been cleaned up. + +## Resolution + +### Cause 1: a custom VMRule still emits cpaas-certificates-rule after the AlertRule is gone + +1. Confirm that the platform certificates are not expired (see Diagnostic Steps) and that AlertRule `cpaas-certificates-rule` is gone: + + ```bash + kubectl get alertrule -A | grep cpaas-certificates-rule + ``` + +2. Find the object that still carries the rule. Name matches are not enough, because a custom `VMRule` can have any name; search the `vmalert` rule-file ConfigMaps and the `VMRule` contents: + + ```bash + kubectl get vmrule -A + for i in 0 1 2; do kubectl get cm -n cpaas-system vm-alert-rulefiles-$i -o yaml | grep -B5 -A10 "certificate_expires_time"; done + kubectl get vmrule -A -o yaml | grep -B5 -A10 "certificate_expires_time" + kubectl get rpch -A + ``` + +3. Back up and delete that `VMRule`. + + > **Warning:** Deleting the custom `VMRule` stops every alert it generates, including alerts you may still want from it. Do this only when `openssl` shows the platform certificates are valid and this `VMRule` is still the source of `cpaas-certificates-rule` after the AlertRule is gone. Roll back with `kubectl apply -f` of the backup, which restores the same alerts. + + ```bash + kubectl get vmrule -n -o yaml > "$HOME/.yaml" + kubectl delete vmrule -n + ``` + +4. Repeat step 2. The rule-file ConfigMaps should no longer contain the rule, and the alert should resolve. + +### Cause 2: cert-manager webhook denials for {cpaas-ca ClusterIssuer ait.alauda.io} count as 403 read failures + +1. Check that cert-manager is running and that no certificate is actually failing: + + ```bash + kubectl get pods -A | grep cert-manager + kubectl get certificate -A | grep -v True + ``` + + If cert-manager is healthy and the second command lists only the header, the log lines need no fix. + +2. If a custom read-success-rate alert fires because of these 403s, edit that existing alert policy (do not create a second one) so that 4xx responses are excluded from the denominator: + + ```text + sum(irate(apiserver_request_total{code=~"20.*",verb=~"GET|LIST"}[2m]))/sum(irate(apiserver_request_total{code!~"4..",verb=~"GET|LIST"}[2m]))<0.9 + ``` + + This is a workaround: it also hides 4xx responses from other clients. For other sources of apiserver error-rate alerts, see [kube-apiserver latency alerts, noisy apiserver logs, and controller-manager or scheduler health alerts on ACP](kube-apiserver_latency_alerts_noisy_apiserver_logs_and_controller-manager_or_scheduler_health_alerts_on_ACP.md). + +### Cause 3: cdi-uploadserver-client-cert on a KubeVirt cluster is valid for one day by design + +1. Confirm the alerting certificate is `cdi-uploadserver-client-cert`: + + ```bash + kubectl get secret -A | grep cdi-uploadserver-client-cert + ``` + +2. Leave the certificate as it is and handle the alert on the policy side. Under **Operation and Maintenance Center** > **Alerts** > **Alert Policies**, find the certificate expiry policy and adjust it, for example by turning off the 30-day and 10-day pre-expiry rules and keeping the already-expired rule, or silence the policy. Changing the alert level of the built-in policy from the console may not be possible. + +3. Alternatively, lengthen the certificate lifetime through the HyperConverged (HCO) object with a CDI JSON patch annotation that sets `/spec/certConfig/client`. + + > **Warning:** HCO passes the patch to CDI, which reissues the upload client certificate with the new lifetime (ten years below). Invalid JSON in the annotation can stop HCO from reconciling CDI. Do this only when the alerting certificate is `cdi-uploadserver-client-cert`. Save the HCO object first. Roll back by removing the annotation: `kubectl annotate hco -n kubevirt kubevirt-hyperconverged containerizeddataimporter.kubevirt.io/jsonpatch-`. + + ```bash + kubectl get hco -n kubevirt kubevirt-hyperconverged -o yaml > "$HOME/kubevirt-hyperconverged.yaml" + kubectl annotate hco -n kubevirt kubevirt-hyperconverged --overwrite \ + containerizeddataimporter.kubevirt.io/jsonpatch='[{"op": "add", "path": "/spec/certConfig/client", "value": {"duration": "87600h0m0s", "renewBefore": "720h0m0s"}}]' + kubectl get secret -n kubevirt cdi-uploadserver-client-cert -o yaml | grep -E 'certConfig|certificate-not-after' + ``` + + The Secret should then carry the annotation `operator.cdi.kubevirt.io/certConfig` with `"lifetime":"87600h0m0s"`. + +### Cause 4: the node-local-certs-exporter Pod is short-lived and no longer exists + +1. Confirm the Pod is gone: + + ```bash + kubectl get pod -A -o wide | grep certs + ``` + + If needed, use the kube-apiserver audit log to confirm the Pod was created and deleted as part of a normal run. + +2. No repair is required. Tune the external monitor so that it does not alert on this short-lived Pod. + +If the Pod exists and stays in `Error` with `cert-tool export` in its log, that is a different fault: see Cause 3 of [Node kubectl fails with You must be logged in to the server or an alert names root kube config on ACP](Node_kubectl_fails_with_You_must_be_logged_in_to_the_server_or_an_alert_names_root_kube_config_on_ACP.md). + +## Diagnostic Steps + +1. Print the real certificate the alert names, to tell whether the certificate or the alert is wrong: + + ```bash + kubectl get secret -n -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject + ``` + + A `notAfter` in the future means the certificate is valid and the alert source has to be found. + +2. Check the components and rule objects that produce the alert: + + ```bash + kubectl get po -A | grep -E 'cluster-enhancer|courier|vmalert' + kubectl get vmrule -A | grep -i cert + kubectl get alertrule -A | grep cpaas-certificates-rule + ``` + +3. Match the signature: + + | What you see | Cause | + | --- | --- | + | AlertRule `cpaas-certificates-rule` absent, rule still in `vm-alert-rulefiles-*` or a `VMRule` | 1 | + | kube-apiserver log mentions `ait.alauda.io`; failed reads are mostly `403` | 2 | + | Alerting certificate is `cdi-uploadserver-client-cert` with one day left | 3 | + | External monitor alerts on a `node-local-certs-exporter-` Pod that does not exist | 4 | + | Certificate page empty, or still shows a renewed or removed certificate | *Certificate page empty or stale because of clock drift or a stale cluster-enhancer-apiserver metric on ACP* | + + To see which response codes drive a read-failure alert, query in the monitoring UI: + + ```text + sum by (code) (rate(apiserver_request_total{verb=~"GET|LIST",code!~"2.."}[5m])) + ``` + + A result dominated by `code="403"` together with `ait.alauda.io` in the kube-apiserver log matches Cause 2. diff --git a/docs/en/solutions/Certificate_page_empty_or_stale_because_of_clock_drift_or_a_stale_cluster-enhancer-apiserver_metric_on_ACP.md b/docs/en/solutions/Certificate_page_empty_or_stale_because_of_clock_drift_or_a_stale_cluster-enhancer-apiserver_metric_on_ACP.md new file mode 100644 index 000000000..c3667aa6e --- /dev/null +++ b/docs/en/solutions/Certificate_page_empty_or_stale_because_of_clock_drift_or_a_stale_cluster-enhancer-apiserver_metric_on_ACP.md @@ -0,0 +1,182 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.0" + - "4.2" + - "4.3" +--- + +# Certificate page empty or stale because of clock drift or a stale cluster-enhancer-apiserver metric on ACP + +## Issue + +The platform certificate page or a certificate expiry alert does not match the certificates that actually exist: + +- The certificate management page is empty although certificate metrics exist, and the Prometheus UI of the `global` cluster warns `Detected seconds time difference between your browser and the server`. +- "Platform component certificates have expired" stays active after the certificate was renewed or removed, while VictoriaMetrics counts 0 expired certificates, or the alert data loads very slowly. +- After an upgrade, every cluster reports `alive-admin` certificates (`certmanager.-etc-kubernetes-alive-admin.conf`) as expired, with a value of about `-739,800` days. +- A Secret was rebuilt with a new certificate, and the page still shows the old date or "expired". +- `平台Kubernetes证书已到期 <= 0d 且持续 1 分钟` ("platform Kubernetes certificate has expired <= 0d for 1 minute", rule group `cpaas-cluster-rules`) names `/etc/kubernetes/kubelet.conf` or `/etc/kubernetes/admin.conf` on nodes where no matching file seems to exist. + +Certificate expiry metrics (`certificate_expires_time`) are exposed by `cluster-enhancer-apiserver` (Cluster Enhancer); node certificate files are read on the nodes; the page and alert status are served through `courier` on the `global` cluster, which refreshes its stored state about every 4 hours. + +Related articles: *Certificate alerts from a custom VMRule, ait.alauda.io webhook denials, or by-design short-lived certificates on ACP* (a custom `VMRule`, `ait.alauda.io` webhook denials, a one-day `cdi-uploadserver-client-cert`, or a missing `node-local-certs-exporter` Pod). [Monitoring pages stay empty because courier-api or apollo holds a stale cluster cache on ACP](Monitoring_pages_stay_empty_because_courier-api_or_apollo_holds_a_stale_cluster_cache_on_ACP.md) (Cause 3) covers an alert that remains after the expired certificate object was deleted. + +## Root Cause + +### Cause 1: browser time and server time differ, so the certificate page shows no data + +The certificate data exists, but the page renders nothing. Querying the metrics directly (for example `curl :19890/metrics | grep cert` from the `cluster-console` terminal of the `global` cluster) returns `certmanager_certificate_expiration_timestamp_seconds` for files such as `/etc/kubernetes/admin.conf`, `/etc/kubernetes/pki/apiserver.crt`, and `/etc/kubernetes/pki/etcd/server.crt`, together with `platform_scrape_collector_success{collector="cert"} 1`. The Prometheus UI of the `global` cluster shows: + +```text +Warning: Error fetching server time: Detected 305.3659999370575 seconds time difference between your browser and the server. Prometheus relies on accurate time and time drift might cause unexpected query results. +``` + +The clock of the machine running the browser and the node clock differ by several minutes (after time-zone conversion). Queries evaluated against the wrong time range return nothing, so the page is empty. Synchronizing node time makes the page fill again. + +### Cause 2: cluster-enhancer-apiserver is hung or cannot parse a certificate, so the expiry metric is stale + +The certificate is valid or already gone, and the "expired" alert does not clear because `cluster-enhancer-apiserver` keeps exporting a stale or invalid `certificate_expires_time`. Three forms are known; restarting `cluster-enhancer-apiserver` (together with `cluster-enhancer-controller` in the hung form) clears the alert in each: + +- **Empty alive certificate after an upgrade.** The expression `(max by(secret_namespace,name) (certificate_expires_time{secret_namespace!=""}) - time()) / (60*60*24)<=0` returns series `certmanager.-etc-kubernetes-alive-admin.conf` in `secret_namespace="cpaas-system"` with `check_time`, `not_after`, and `not_before` all `0001-01-01T00:00:00Z` and the value `-62135596800` (the zero time). The `cluster-enhancer-apiserver` log has `failed to parse certificate from secret cpaas-system/certmanager.-etc-kubernetes-alive-admin.conf, error: PEM is empty` for each control-plane node, sometimes with watch errors ending in `net/http: TLS handshake timeout`, while its Pods are `1/1 Running`. +- **Zero expiry on a region cluster.** The alert detail of `cpaas-certificates-rule` lists `certmanager.-etc-kubernetes-alive-admin.conf` entries at about `-739781 d` (the distance back to year 1, the same zero time), while other certificates such as `cluster-enhancer-apiserver-cert` are positive. `/etc/kubernetes/alive/admin.conf` on the alerting nodes has no `client-certificate-data`. Why that field was missing has not been determined. +- **Hung after certificates were removed.** Expired certificates were removed (for example `argocd-gitops-ca` and `argocd-gitops-tls` in `argocd`, by uninstalling the Argo CD GitOps plugin); VictoriaMetrics then counts 0 expired, but the alert stays and the alert data loads very slowly. `kubectl logs -f` of `cluster-enhancer-apiserver` stops at startup lines such as `Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file` and prints nothing new. + +If alert data loads slowly because `vmselect` is overloaded or `OOMKilled`, not because `cluster-enhancer-apiserver` is hung, the cause is query load, not this one. + +### Cause 3: the certificate was reissued, or its file was deleted, and the page still shows the previous state + +The status the page and alert show is refreshed about every 4 hours. After a Secret is deleted and recreated with a new certificate, the old date or "expired" remains until that refresh, or until `courier` on the `global` cluster is restarted. A variant after a renewal: the browser console reports `found duplicate series for the match group` with one `phase="Expired"` and one `phase="Normal"` series for the same certificate and `many-to-many matching not allowed`; a `courier` restart clears it. + +Node-file alerts follow the same rule. When `/etc/kubernetes/kubelet.conf` shows a negative number of days (for example about `-158 d`), the node certificate really has expired. When `/etc/kubernetes/admin.conf` shows a large negative value (for example about `-745 d`) but the file has been deleted, the collector cannot read a new file and keeps reporting the last state. In both situations, reissue the certificates and then wait four hours or restart `courier`. Stale node-file rows after an upgrade (for example `/var/lib/kubelet/pki/kubelet.crt` or `/etc/kubernetes/super-admin.conf`) have also been cleared by restarting `cluster-enhancer-controller` and `cluster-enhancer-apiserver` instead, so if a `courier` restart does not help, apply the Cause 2 restart. + +## Resolution + +### Cause 1: browser time and server time differ, so the certificate page shows no data + +1. From the `cluster-console` terminal of the `global` cluster, confirm the data exists and read the server time; compare it with the clock of the machine running the browser (convert the time zone first): + + ```bash + curl -s :19890/metrics | grep cert | head + date + ``` + +2. If the metrics are present and the clocks differ by minutes, compare node time with a reference clock and with the browser host, and fix time synchronization on whichever clock is wrong. On nodes that use chrony: + + ```bash + date -u + timedatectl status + chronyc tracking 2>/dev/null || ntpq -p + ``` + + See [Configuring chrony NTP on Cluster Nodes](Configuring_chrony_NTP_on_Cluster_Nodes.md) for the node time source. + +3. Correct the node clock. + + > **Warning:** Stepping the clock of a running node can disturb certificate validity checks, leases, and etcd on that node. Do this only when the offset is confirmed against a reference clock and the Prometheus UI shows the time-difference warning. Prefer letting chrony slew the clock; a large step belongs in a maintenance window, one control-plane node at a time. There is no rollback to the previous wrong time; if the new time source is wrong, point chrony back to the previous source. + + ```bash + systemctl restart chronyd + chronyc makestep + ``` + +4. Reload the certificate page; it should list the certificates again. + +### Cause 2: cluster-enhancer-apiserver is hung or cannot parse a certificate, so the expiry metric is stale + +1. Confirm the certificate is valid or removed (see Diagnostic Steps), then check `cluster-enhancer-apiserver` on the cluster that raised the alert: + + ```bash + kubectl get po -A | grep cluster-enhancer-apiserver + kubectl logs -n --since=1h | tail -50 + ``` + + No log lines in the last hour (the log stops at startup lines such as `Caches are synced for client-ca::...`), or `failed to parse certificate from secret ... error: PEM is empty`, matches this cause. For a removed certificate, check that its Secret is really gone: + + ```bash + kubectl get secret -n | grep + ``` + +2. Restart `cluster-enhancer-apiserver`. + + > **Warning:** Restarting `cluster-enhancer-apiserver` pauses certificate metric collection for that cluster until the new Pod is Running. Do this only when `openssl` or the certificate page already shows the alerting certificate as valid or removed, and the component is silent or logs `PEM is empty`. The Deployment recreates the Pod; there is nothing else to roll back. + + ```bash + kubectl delete pod -n + kubectl get po -n | grep cluster-enhancer-apiserver + ``` + +3. When `cluster-enhancer-apiserver` was hung (no new log lines), restart both Cluster Enhancer Pods in `cpaas-system` on that cluster. The same Warning applies; the controller is also paused until its Pod is Running. + + ```bash + kubectl get pod -n cpaas-system | grep enhancer | awk '{print $1}' | xargs kubectl delete pod -n cpaas-system + kubectl get pod -n cpaas-system | grep enhancer + ``` + +4. After one scrape interval, re-run the alert expression in the VictoriaMetrics UI. It should return no series for the fixed certificate. + +### Cause 3: the certificate was reissued, or its file was deleted, and the page still shows the previous state + +1. Print the real certificate: + + ```bash + kubectl get secret -n -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates + ``` + + For a node-file alert, print the file the alert names on that node: + + ```bash + ls -l /etc/kubernetes/kubelet.conf /etc/kubernetes/admin.conf + grep client-certificate-data /etc/kubernetes/kubelet.conf | awk '{print $2}' | base64 -d | openssl x509 -noout -dates + ``` + + If the kubeconfig has `client-certificate: ` instead of `client-certificate-data`, run `openssl x509 -noout -dates -in ` on that file. + + If the certificate in the file is expired, it is a real expiry; reissue it with the certificate renewal procedure for your ACP version before continuing. If the file no longer exists, the alert is the last stored state. + +2. When the real certificate is new (or back in place), wait about 4 hours, or restart `courier` on the `global` cluster. + + > **Warning:** Deleting the `courier` Pod fails in-flight monitoring and inspection calls and pauses notification sending until the replacement is Ready. Do this only after the certificate is confirmed valid and only the displayed state is old. The controller recreates the Pod; there is nothing to roll back. + + ```bash + kubectl -n cpaas-system get pods | grep courier + kubectl -n cpaas-system delete pod + ``` + +3. If the stale row remains and the VictoriaMetrics UI still returns the old `certificate_expires_time` series, apply the Cause 2 restart. + +## Diagnostic Steps + +1. Print the real certificate the alert or page names, to tell whether the certificate or the display is wrong: + + ```bash + kubectl get secret -n -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject + ``` + +2. Check the components that produce the status: + + ```bash + kubectl get po -A | grep -E 'cluster-enhancer|courier|vm' + ``` + +3. Query the series in the VictoriaMetrics UI: + + ```text + (max by(secret_namespace,name) (certificate_expires_time{secret_namespace!=""}) - time()) / (60*60*24)<=0 + ``` + +4. Match the signature: + + | What you see | Cause | + | --- | --- | + | Page empty, metrics present, Prometheus warns `Detected seconds time difference between your browser and the server` | 1 | + | Series with `not_after="0001-01-01T00:00:00Z"` (about `-739,800 d`), or `PEM is empty` in the `cluster-enhancer-apiserver` log | 2 | + | Certificate removed, VictoriaMetrics counts 0 expired, `cluster-enhancer-apiserver` log silent | 2 | + | Secret rebuilt or node file deleted, no stale series in VictoriaMetrics, page shows the previous state | 3 | + | Node file such as `kubelet.conf` contains a certificate that is really expired | 3 (real expiry: reissue first) | diff --git a/docs/en/solutions/HTTPS_to_a_published_service_fails_with_plain_HTTP_sent_to_HTTPS_port_ERR_SSL_PROTOCOL_ERROR_or_TLS_alert_80_on_ACP.md b/docs/en/solutions/HTTPS_to_a_published_service_fails_with_plain_HTTP_sent_to_HTTPS_port_ERR_SSL_PROTOCOL_ERROR_or_TLS_alert_80_on_ACP.md new file mode 100644 index 000000000..6320433dd --- /dev/null +++ b/docs/en/solutions/HTTPS_to_a_published_service_fails_with_plain_HTTP_sent_to_HTTPS_port_ERR_SSL_PROTOCOL_ERROR_or_TLS_alert_80_on_ACP.md @@ -0,0 +1,186 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "3.18" + - "4.0" + - "4.2" + - "4.3" +--- + +# HTTPS to a published service fails with plain HTTP sent to HTTPS port, ERR_SSL_PROTOCOL_ERROR, or TLS alert 80 on ACP + +## Issue + +HTTPS access to a service published through ALB or an Ingress fails during or right after the TLS handshake. One of the following is seen: + +- The browser or client gets `400 Bad Request` with the body `The plain HTTP request was sent to HTTPS port`, typically when traffic arrives through an external load balancer. +- The browser reports `ERR_SSL_PROTOCOL_ERROR` after the service was switched from HTTP to HTTPS. +- The handshake fails with TLS alert 80, even with certificate verification disabled: + + ```text + curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error + ``` + + `openssl s_client` shows `SSL alert number 80`. This can follow an application migration in which the Ingress was set up again with the application's previous self-signed certificate. + +A self-signed certificate that the client does not trust gives a verification error, not an alert from the server; `curl -k` still failing means the server side aborted the handshake. + +Related articles: *Old business TLS certificate still served after an Ingress, ALB rule, or upstream load balancer update on ACP* (the handshake works but the previous certificate is presented). [TLS passthrough breaks behind an L7 load balancer on ACP ALB ingress](TLS_passthrough_breaks_behind_an_L7_load_balancer_on_ACP_ALB_ingress.md) explains TCP passthrough versus TLS termination on ALB frontends. + +## Root Cause + +### Cause 1: an external load balancer sends decrypted HTTP to the ALB HTTPS port + +The ALB listener on 443 is HTTPS. Opening that ALB entry directly, bypassing the external load balancer, reaches the service. The external load balancer terminates TLS on 443 and forwards the decrypted HTTP request to the ALB port that still expects TLS, and the ALB answers `400 The plain HTTP request was sent to HTTPS port`. + +A port 80 to 443 redirect does not repair this forward. The fix is on the external load balancer, and it depends on where TLS should terminate: + +- TCP passthrough to ALB 443 makes the ALB certificate the one clients see. +- Sending the decrypted HTTP to an HTTP listener on the ALB keeps TLS termination on the external load balancer, so the certificate clients see is the external load balancer's certificate. + +### Cause 2: ERR_SSL_PROTOCOL_ERROR because the published Service port reaches a container port that speaks plain HTTP + +The application inside the Pod terminates TLS itself (for example nginx with `listen 443 ssl`, certificate and key files present), but the Service port that clients use still targets the container's HTTP port. For example, the original Service port 80 targets container port 80, and the added Service port 443 also targets container port 80 instead of 443. The client starts a TLS handshake against a plain HTTP listener and the browser reports `ERR_SSL_PROTOCOL_ERROR`. + +This cause applies only when the Pod terminates TLS and the Service `targetPort` is the HTTP container port. If the ALB log says no certificate is bound for the domain, or the ALB rule form rejects the domain because it is not registered under the platform domain names, that is a different problem. + +### Cause 3: ALB returns TLS alert 80 and logs cert_tool.lua: no cert found + +All replicas of the ALB (for example `common-lb` or `global-alb2` in `cpaas-system`) fail the handshake with `tlsv1 alert internal error` / SSL alert number 80. The ALB `nginx` container logs: + +```text +cert_tool.lua: no cert found for / +SSL_do_handshake() failed ... cert cb error +``` + +`` is the frontend port. The same replicas can also log `policy_fetch.lua` with `empty policy` or `invalid policy data`, and `/etc/alb2/nginx/policy.new` can be seen at 0 bytes or at a partial size while it is rewritten; when parsing fails, the running ALB keeps the previous active policy, so those lines alone do not explain a lasting outage. The direct failure is the runtime certificate lookup miss for the host in the ALB's in-memory certificate cache. A rolling restart of the ALB replicas makes them serve the expected certificate again. If a fresh replica immediately logs `no cert found` again, the certificate binding itself is missing and a restart does not help. + +### Cause 4: the Ingress names the intended certificate, but the controller has not refreshed it + +The Ingress for the host already references the intended TLS Secret (for example the application's previous self-signed certificate, reused after a migration), the Secret exists and parses, and `curl -k` still fails with `tlsv1 alert internal error`. Setting the same certificate on the Ingress again does not help. Switching the Ingress to a different certificate (for example the platform certificate) and then back to the intended certificate makes the controller pick it up, and access recovers. The certificate had not been refreshed by the controller; the exact mechanism has not been determined. + +## Resolution + +### Cause 1: an external load balancer sends decrypted HTTP to the ALB HTTPS port + +1. Confirm the ALB and the application work when requested directly: + + ```bash + curl -vk --resolve :443: https:/// + ``` + + A response from the application means the ALB listener and the Pods are fine. + +2. Change the forwarding on the external load balancer so that it no longer sends decrypted HTTP to ALB 443: + - When the ALB should be the only certificate presenter, pass TCP through to the ALB 443 listener. + - When the external load balancer should keep terminating TLS, forward the decrypted HTTP to an HTTP listener on the ALB. The certificate to maintain is then the one on the external load balancer. + +3. Retry through the public name. The `400` should be gone. + +### Cause 2: ERR_SSL_PROTOCOL_ERROR because the published Service port reaches a container port that speaks plain HTTP + +1. Compare the Service ports with the port where the process enables TLS: + + ```bash + kubectl get svc -n -o yaml + kubectl exec -n -- sh -c 'ss -lntp || netstat -lntp' + ``` + +2. Point the HTTPS Service port at the container port that terminates TLS. + + > **Warning:** Changing the Service `targetPort` drops the current mapping for that port (and its NodePort or load balancer port). Clients using that port fail until the new target answers. Do this only when the client error is `ERR_SSL_PROTOCOL_ERROR` and the Service's HTTPS port targets the container's HTTP port. Save the Service first; roll back with `kubectl apply -f` of the saved file. + + ```bash + kubectl get svc -n -o yaml > "$HOME/.yaml" + kubectl edit svc -n + ``` + + For example, set `targetPort: 443` on the Service port that clients use for HTTPS. + +3. A new HTTPS request should complete the handshake. + +### Cause 3: ALB returns TLS alert 80 and logs cert_tool.lua: no cert found + +1. Find the ALB workload and confirm the log signature on its replicas: + + ```bash + kubectl get pods -A | grep -E 'common-lb|global-alb2' + kubectl logs -n cpaas-system -c nginx --tail=100 | grep -E 'cert_tool|alert|policy_fetch' + ``` + + Make sure the Pods are the platform ALB in `cpaas-system`, not another ingress controller that the `grep` also matched. If the leaf is simply expired, replace the certificate instead of restarting. Do not hand-edit `/etc/alb2/nginx/policy.new`. + +2. Restart the ALB replicas one at a time. + + > **Warning:** A rolling restart closes the HTTPS connections handled by each replica while it is recreated; in-flight requests on that replica fail and must be retried. With only one ready replica the ALB is unavailable during the restart, so use a maintenance window. Do this only after `cert_tool.lua: no cert found` is confirmed and clients see TLS alert 80. The in-memory certificate cache is rebuilt by the restart; there is nothing to roll back to. + + ```bash + kubectl rollout restart deployment/ -n cpaas-system + kubectl rollout status deployment/ -n cpaas-system + ``` + + If the ALB is not a Deployment, restart the workload kind that `kubectl get pods` shows, one replica at a time. + +3. Check that each replica serves the expected certificate: + + ```bash + echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -dates -subject -issuer + ``` + + If a new replica logs `no cert found` again right away, bind the certificate to the host on the ALB rule or Ingress; a restart will not fix a missing binding. + +### Cause 4: the Ingress names the intended certificate, but the controller has not refreshed it + +1. Confirm that the Ingress references the Secret and that the certificate in it is valid for the host: + + ```bash + kubectl get ingress -A -o yaml | grep -F '' + kubectl get ingress -n -o yaml + kubectl get secret -n -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject -issuer + ``` + +2. Check whether the server presents any certificate: + + ```bash + echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -dates -subject -issuer + ``` + + No certificate printed, together with the TLS alert, matches this cause. + +3. In the Ingress, switch the TLS certificate to a different valid certificate (for example the platform certificate), save, then switch it back to the intended certificate and save again. + + > **Warning:** While the other certificate is selected, clients of this host receive that certificate and may see a name mismatch. Do this only when the intended Secret exists and parses and the host is already failing with TLS alert 80. Save the Ingress first (`kubectl get ingress -n -o yaml > "$HOME/.yaml"`); roll back with `kubectl apply -f` of that file. + +4. Retry `curl https:///` (with `-k` for a self-signed certificate) and confirm the handshake completes. + +## Diagnostic Steps + +1. Read what the client actually receives: + + ```bash + curl -vk https:/// + echo | openssl s_client -connect :443 -servername + ``` + +2. Compare with a direct request to the ALB, bypassing any external load balancer: + + ```bash + curl -vk --resolve :443: https:/// + ``` + +3. Match the signature: + + | What you see | Cause | + | --- | --- | + | `400` with `The plain HTTP request was sent to HTTPS port` through the external load balancer; the direct ALB request works | 1 | + | `ERR_SSL_PROTOCOL_ERROR`; the Service HTTPS port targets the container's HTTP port | 2 | + | TLS alert 80 on every ALB replica; ALB log `cert_tool.lua: no cert found for /` | 3 | + | TLS alert 80; the Ingress already references a valid Secret for the host; set up again after a migration | 4 | + | Handshake succeeds but the old certificate is presented | *Old business TLS certificate still served after an Ingress, ALB rule, or upstream load balancer update on ACP* | diff --git a/docs/en/solutions/Harbor_or_registry_image_pull_fails_with_x509_certificate_signed_by_unknown_authority_or_certificate_has_expired.md b/docs/en/solutions/Harbor_or_registry_image_pull_fails_with_x509_certificate_signed_by_unknown_authority_or_certificate_has_expired.md index abd56c666..cf85b6d47 100644 --- a/docs/en/solutions/Harbor_or_registry_image_pull_fails_with_x509_certificate_signed_by_unknown_authority_or_certificate_has_expired.md +++ b/docs/en/solutions/Harbor_or_registry_image_pull_fails_with_x509_certificate_signed_by_unknown_authority_or_certificate_has_expired.md @@ -125,4 +125,12 @@ The certificate presented for the registry hostname has expired or is not yet va `crictl` failing with x509 while `nerdctl` succeeds, on a node without the registry's `hosts.toml`, is the single-node form of Cause 1. + The two tools read different trust locations: `nerdctl` reads both `/etc/containerd/certs.d` and `/etc/docker/certs.d` by default, while CRI (`crictl` and the kubelet) reads only the `config_path` set in `/etc/containerd/config.toml`. A CA file under `/etc/docker/certs.d/:/` can therefore explain why `nerdctl` pulls while `crictl` does not: + + ```bash + ls -la /etc/containerd/certs.d/:/ /etc/docker/certs.d/:/ + ``` + + Before adding a host file on that one node, also compare `sha256sum /usr/bin/containerd` with a healthy node of the same platform version. A different checksum with Docker installed on the node means the runtime binaries were overwritten; see [Container runtime conflicts on ACP nodes: Docker, podman, docker-runc, and overwritten runtime binaries](Container_runtime_conflicts_on_ACP_nodes_Docker_podman_docker-runc_and_overwritten_runtime_binaries.md) and restore the binaries instead. + 4. If the error is `server gave HTTP response to HTTPS client` or `connection refused` on 443 instead of x509, see the related article under Issue. diff --git a/docs/en/solutions/NFS_PVC_stays_Pending_csi-nfs-controller_mount_fails_with_rpc.statd_exit_status_32_or_access_denied_on_ACP.md b/docs/en/solutions/NFS_PVC_stays_Pending_csi-nfs-controller_mount_fails_with_rpc.statd_exit_status_32_or_access_denied_on_ACP.md new file mode 100644 index 000000000..25a4c23f3 --- /dev/null +++ b/docs/en/solutions/NFS_PVC_stays_Pending_csi-nfs-controller_mount_fails_with_rpc.statd_exit_status_32_or_access_denied_on_ACP.md @@ -0,0 +1,245 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# NFS PVC stays Pending: csi-nfs-controller mount fails with rpc.statd, exit status 32, or access denied on ACP + +## Issue + +A PVC on an NFS StorageClass (provisioner `nfs.csi.k8s.io`) stays `Pending`, or the console keeps showing it as `处理中` (Processing), and no PV is created. `kubectl describe pvc` shows: + +```text +ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'nfs.csi.k8s.io' or manually by the system administrator +Provisioning External provisioner is provisioning volume for claim "/" +ProvisioningFailed failed to provision volume with StorageClass "": rpc error: code = Internal desc = failed to mount nfs server: ... +``` + +The NFS plugin Pods are Running (`csi-nfs-controller` 4/4, `csi-nfs-node` 3/3), and often `showmount -e` lists the export and a manual NFS mount on a node works. The `nfs` container log of `csi-nfs-controller` shows the mount failing with one of: + +- `rpc.statd is not running but is required for remote locking`, `cannot create /run/rpc.statd.lock`, `Mount failed: exit status 32` +- `mount.nfs: access denied by server while mounting :` + +Sometimes the PVC is created only after the NFS version in the StorageClass is changed, or after `nolock` is added to its `mountOptions`. + +At `CreateVolume`, `csi-nfs-controller` mounts the export itself, on the node where it runs, with the StorageClass `server`, `share`, and `mountOptions`, and then creates a per-volume directory. A mount that works by hand from another node, or with other options, does not prove that this mount works. + +Related articles: *NFS PVC stays Pending: csi-nfs-controller mount times out or mkdir of the volume directory is denied on ACP* (the mount times out, or succeeds and `mkdir` fails). For the StorageClass fields see [Dynamic NFS storage provisioning for Tekton CICD on ACP](Dynamic_NFS_storage_provisioning_for_Tekton_CICD_on_ACP.md). + +## Root Cause + +`access denied by server while mounting` has four causes here (Causes 1, 2, 3, and 4); the table in Diagnostic Steps separates them. + +### Cause 1: the NFSv3 mount needs rpc.statd, and it is not running on the node + +An NFSv3 mount without `nolock` needs `rpc.statd` for remote locking. The CSI container cannot start it itself, because its `/run` is read-only: + +```text +/usr/sbin/start-statd: 10: cannot create /run/rpc.statd.lock: Read-only file system +mount.nfs: rpc.statd is not running but is required for remote locking. +mount.nfs: Either use '-o nolock' to keep locks local, or start statd. +Mount failed: exit status 32 +GRPC error: rpc error: code = Internal desc = failed to mount nfs server: rpc error: code = Internal desc = mount failed: exit status 32 +``` + +The node that hosts `csi-nfs-controller` has `rpc-statd` stopped, so the mount fails. The same root problem has three other forms: + +- An NFSv3 StorageClass whose `mountOptions` omit `nolock`: the manual mount with `nolock` works, the PVC never gets a PV, and adding `nolock` to the class restores provisioning. +- A NAS that must stay on NFSv3 (for example a Huawei NAS through the default NFS CSI plugin): NFS 4.0 and 4.1 work through the plugin and NFS 3.0 fails, while a manual v3 mount on the node succeeds. Installing the NFS client utilities and running `rpcbind` and `rpc-statd` on the nodes fixes it. +- The provisioner reports `failed to get nfs info from nfs host` and `access denied by server while mounting`, the server is reachable, a manual `mount -t nfs -o hard,nfsvers=3,nolock` succeeds, and `rpc-statd.service` is inactive. Restarting and enabling `rpc-statd.service`, then connecting the StorageClass again, restores PVC creation. + +### Cause 2: the StorageClass NFS version is one the server does not serve + +The `nfsvers` in the StorageClass `mountOptions` is used by `csi-nfs-controller` for its own mount. If the server does not serve that version, every `CreateVolume` fails, and changing the version on the StorageClass makes the next PVC succeed. Examples: + +- The server does not support NFSv4 at all and the class uses v4. +- The server runs v4.1 and the storage was integrated as v3 (`nfsvers=3`, `nolock`). The controller reports `access denied by server while mounting`, and a manual v3 mount shows: + + ```text + mount -t nfs -o hard,nfsvers=3,nolock : /test -v + mount.nfs: portmap query retrying: RPC: Program not registered + mount.nfs: mount(2): Permission denied + mount.nfs: access denied by server while mounting : + ``` + + The same mount without the v3 option succeeds, and `showmount -e` still lists the export. Integrating the NFS storage again with version v4.1 selected makes provisioning and mounting work. + +The platform supports NFS protocol versions `v3`, `v4.0`, and `v4.1`. + +### Cause 3: the StorageClass share path has no leading / + +The `share` parameter of the StorageClass is written as a relative name (for example `sfs-example-0001`) instead of the export path (`/sfs-example-0001`). `csi-nfs-controller` mounts exactly that string, so the server refuses an export it does not have: + +```text +failed to provision volume with StorageClass "": rpc error: code = Internal desc = failed to mount nfs server: rpc error: code = Internal desc = mount failed: exit status 32 +Mounting arguments: -t nfs -o hard,nfsvers=3,nolock :sfs-example-0001 /tmp/pvc- +Output: mount.nfs: access denied by server while mounting :sfs-example-0001 +``` + +`rpc-statd` can be active on every node, and a manual mount of `:/sfs-example-0001` (with the slash) succeeds on the controller node. On some NAS devices no error text is shown at all; the PVC simply stays Pending until `share` is corrected. + +### Cause 4: the NAS authorizes only some nodes, not the node running csi-nfs-controller + +Mount permission on the NAS was granted to the worker nodes, not to the control-plane nodes (or to some other subset). Dynamic provisioning is done by `csi-nfs-controller`, which mounts the export from whichever node it runs on. By default it is not pinned to a fixed node (its `nodeSelector` is `kubernetes.io/os: linux`), so it can land on a node that has no mount permission on the NAS, and PVC creation fails with a permission error right after the StorageClass is created. + +## Resolution + +### Cause 1: the NFSv3 mount needs rpc.statd, and it is not running on the node + +Use one of two repairs: run `rpc-statd` on the nodes, or add `nolock` to the StorageClass. Use the service when the NAS has to stay on NFSv3 with locking; use `nolock` when the error suggests it or the service cannot be enabled. + +**Run rpc-statd on the nodes.** Run this on every node that can run `csi-nfs-controller` or a Pod that uses the volume. If `rpc.statd` is not installed, install the node OS package that provides `rpcbind` and `rpc-statd` (the NFS client utilities) first. + +> **Warning:** `rpcbind` and `rpc-statd` listen on the node and take part in NFSv3 locking; restarting `rpc-statd` briefly interrupts NFSv3 lock recovery on that node. Do this only when a Cause 1 signature is confirmed and the unit is inactive. Roll back with `sudo systemctl disable --now rpc-statd rpcbind`. + +```bash +systemctl status rpc-statd rpcbind +sudo systemctl enable --now rpcbind +sudo systemctl enable --now rpc-statd +rpcinfo -p 127.0.0.1 +``` + +`rpcinfo` should list program `100000` (portmapper) and program `100024` (status). In the `failed to get nfs info from nfs host` form, also connect the StorageClass to the NFS storage again after the service is running. + +**Or add nolock to the StorageClass.** Kubernetes stores mount options without the `-o` flag. + +```bash +kubectl edit storageclass +``` + +```yaml +mountOptions: + - hard + - nfsvers=3 + - nolock +``` + +`mountOptions` on an existing class can be changed; `parameters` cannot. Existing PVs keep the options they were provisioned with. + +**Then recreate the Pending claim.** + +> **Warning:** Deleting a PVC whose PV uses reclaim policy `Delete` makes the provisioner remove the backend directory. Do this only when the claim is still Pending, `spec.volumeName` is empty, and no application data was written. A directory the provisioner has already removed cannot be restored; if `spec.volumeName` is set, stop and check the reclaim policy first. + +```bash +kubectl -n get pvc -o jsonpath='{.spec.volumeName}{"\n"}' +kubectl -n get pvc -o yaml > "$HOME/.yaml" +kubectl -n delete pvc +``` + +Create it again from your manifest (remove `metadata.uid`, `metadata.resourceVersion`, and `status` if you reuse the saved copy). + +### Cause 2: the StorageClass NFS version is one the server does not serve + +1. From the node that runs `csi-nfs-controller`, try the class's version and another version: + + ```bash + kubectl get storageclass -o jsonpath='{.mountOptions}{"\n"}' + sudo mount -t nfs -o nfsvers=4.1 : /mnt && sudo umount /mnt + sudo mount -t nfs -o nfsvers=3,nolock : /mnt && sudo umount /mnt + rpcinfo -p + ``` + + `portmap query retrying: RPC: Program not registered` on the v3 mount means the server does not offer NFSv3 to this client. A server that serves only NFSv4 may not answer `rpcinfo` at all. On the NFS server, `nfsstat -s` shows which versions it serves. + +2. Set the version that mounted in the StorageClass `mountOptions` (for a server without NFSv4, `nfsvers=3` plus `nolock` as in Cause 1), or integrate the NFS storage again with that version selected in the **NFS Protocol Version** field of the StorageClass form. Then create the PVC again. + + ```bash + kubectl edit storageclass + ``` + +### Cause 3: the StorageClass share path has no leading / + +1. Read the class and the controller's mount arguments: + + ```bash + kubectl get storageclass -o jsonpath='{.parameters.server}{" "}{.parameters.share}{"\n"}' + kubectl -n cpaas-system logs deploy/csi-nfs-controller -c nfs --tail=500 | grep -E 'Mounting arguments|access denied' + showmount -e + ``` + + A `share` without a leading `/`, or mount arguments with no `/` after the colon, matches this cause. `showmount -e` lists the export paths exactly as the server publishes them (NFSv3 servers; an NFSv4-only server may not answer). + +2. StorageClass `parameters` cannot be updated in place, so recreate the class under the same name with the corrected `share`. + + > **Warning:** While the class is deleted, new PVCs that name it cannot be provisioned. Deleting a StorageClass does not remove existing PVs or PVCs. Do this only when the Cause 3 signature is confirmed. Back up the class first; roll back with `kubectl apply -f /tmp/.yaml`. + + ```bash + kubectl get storageclass -o yaml > /tmp/.yaml + cp /tmp/.yaml /tmp/-fixed.yaml + vi /tmp/-fixed.yaml + kubectl delete storageclass + kubectl apply -f /tmp/-fixed.yaml + ``` + + In the copy, set `share: /` and remove `metadata.uid`, `metadata.resourceVersion`, `metadata.creationTimestamp`, and `metadata.managedFields`. + +3. If a claim still has no PV after a few minutes, recreate it as in Cause 1 (only while it holds no data). + +### Cause 4: the NAS authorizes only some nodes, not the node running csi-nfs-controller + +1. Find the node that runs the controller and compare it with the NAS client allowlist: + + ```bash + kubectl get pods -A -o wide | grep csi-nfs-controller + ``` + +2. Grant NAS mount permission (the client allowlist or export rule) to every node in the cluster. This is the recommended fix, because `csi-nfs-controller` can be rescheduled to any node, and Pods mount NFS through `csi-nfs-node` on whichever node they run. + +3. If new nodes should not need NAS permission, pin `csi-nfs-controller` to a labeled set of nodes that the NAS authorizes. Restarting or recreating the `csi-nfs-controller` Pod does not affect existing PVCs or PVs. + + > **Warning:** Saving the Deployment restarts `csi-nfs-controller`; provisioning pauses until it is Ready, and stays stopped if no labeled node is schedulable. A plugin reinstall or upgrade can drop the edit. Do this only when the Cause 4 signature is confirmed and every labeled node can mount the export. Back up the Deployment first; roll back with `kubectl -n cpaas-system apply -f /tmp/csi-nfs-controller.yaml`, or set the `nodeSelector` back to `kubernetes.io/os: linux`. + + ```bash + kubectl label node storage.example/nas-access=allowed + kubectl -n cpaas-system get deploy csi-nfs-controller -o yaml > /tmp/csi-nfs-controller.yaml + kubectl -n cpaas-system edit deploy csi-nfs-controller + ``` + + ```yaml + spec: + template: + spec: + nodeSelector: + storage.example/nas-access: allowed + ``` + + `storage.example/nas-access` is a placeholder label key; every node that runs NFS workloads still needs NAS permission. + +## Diagnostic Steps + +1. Read the claim's events, the class, and the controller log: + + ```bash + kubectl -n describe pvc + kubectl get storageclass -o yaml + kubectl -n cpaas-system get pods -o wide | grep csi-nfs + kubectl -n cpaas-system logs deploy/csi-nfs-controller -c nfs --tail=200 + ``` + +2. On the node that runs `csi-nfs-controller`, repeat the controller's mount by hand with the class's `server`, `share`, and options: + + ```bash + systemctl status rpc-statd rpcbind + sudo mount -t nfs -v -o hard,nfsvers=3,nolock : /mnt && sudo umount /mnt + ``` + +3. Match the signature: + + | Controller log or manual test | Cause | + | --- | --- | + | `rpc.statd is not running`, `cannot create /run/rpc.statd.lock`, `exit status 32` | 1 | + | `access denied by server`; a manual mount with `nolock` works; `rpc-statd` inactive | 1 | + | `portmap query retrying: RPC: Program not registered` with `nfsvers=3`; a mount without `nfsvers=3` works | 2 | + | Mounting arguments `:` with no `/` after the colon | 3 | + | Manual mount denied from the controller node, allowed from a node the NAS authorizes | 4 | + | `timeout after 110s`, or `failed to make subdirectory ... permission denied` | *NFS PVC stays Pending: csi-nfs-controller mount times out or mkdir of the volume directory is denied on ACP* | + +A host mount that works does not by itself clear a Pending claim: the controller uses host networking and its own node, StorageClass options, and `share` string, so a failure that appears only in the controller log still blocks provisioning. diff --git a/docs/en/solutions/NFS_PVC_stays_Pending_csi-nfs-controller_mount_times_out_or_mkdir_of_the_volume_directory_is_denied_on_ACP.md b/docs/en/solutions/NFS_PVC_stays_Pending_csi-nfs-controller_mount_times_out_or_mkdir_of_the_volume_directory_is_denied_on_ACP.md new file mode 100644 index 000000000..1487aab98 --- /dev/null +++ b/docs/en/solutions/NFS_PVC_stays_Pending_csi-nfs-controller_mount_times_out_or_mkdir_of_the_volume_directory_is_denied_on_ACP.md @@ -0,0 +1,124 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# NFS PVC stays Pending: csi-nfs-controller mount times out or mkdir of the volume directory is denied on ACP + +## Issue + +A PVC on an NFS StorageClass (provisioner `nfs.csi.k8s.io`) stays `Pending`, or the console keeps showing it as `处理中` (Processing) or creating, and no PV is created. The NFS plugin Pods are Running. The PVC events or the `nfs` container log of `csi-nfs-controller` show one of: + +```text +failed to mount nfs server: rpc error: code = Internal desc = mount volume : to /tmp/pvc- timeout after 110s +``` + +```text +GRPC error: rpc error: code = Internal desc = failed to make subdirectory: mkdir /tmp/pvc-/pvc-: permission denied +``` + +A typical case is an application that creates its own claims, for example a Harbor deployment whose PostgreSQL PVCs (`pgdata-acid-postgresql-0`, `pgdata-acid-postgresql-1`) are never provisioned, or a new RWX PVC created right after an NFS StorageClass was added. + +At `CreateVolume`, `csi-nfs-controller` mounts the export on the node where it runs, under `/tmp/pvc-`, creates a per-volume directory in it, and unmounts it again. These two failures sit on either side of that mount. + +Related articles: *NFS PVC stays Pending: csi-nfs-controller mount fails with rpc.statd, exit status 32, or access denied on ACP* (the mount itself is refused). + +## Root Cause + +### Cause 1: the provisioner mounts the export but cannot create the volume directory in it + +The mount succeeds, then the provisioner fails to create the per-volume directory because the NFS server has not granted the required permissions on the export directory. The driver log shows the export mounted on `/tmp/pvc-`, then: + +```text +GRPC error: rpc error: code = Internal desc = failed to make subdirectory: mkdir /tmp/pvc-/pvc-: permission denied +``` + +and then the temporary mount being unmounted again (`unmount volume ... successfully`). The claim stays Pending. Once the server-side permissions are opened, the PVC is created. A PVC that stays in "creating" right after a new StorageClass was created can also come from an NFS server configuration that is not finished (export not defined, clients not allowed, or service not started). + +### Cause 2: the cluster nodes cannot reach the NFS server + +The provisioner cannot complete the mount at all. The PVC events cycle between `Provisioning` (`External provisioner is provisioning volume for claim ...`) and `ProvisioningFailed` with `mount volume : to /tmp/pvc- timeout after 110s`, and the provisioner log counts repeated failures (for example `failures=9`). Network connectivity from the cluster nodes to the NFS storage is missing, for example because of routing, a firewall, or a storage-side client allowlist. For example, on ACP 4.3.0 the PostgreSQL PVCs that a Harbor deployment creates in `cpaas-solution` stay Pending this way when the nodes cannot reach the NFS NAS. When the cluster nodes cannot reach the NFS server address at all, opening network access from the nodes to the server is the whole fix; the PVC is then created and mounted normally. + +## Resolution + +### Cause 1: the provisioner mounts the export but cannot create the volume directory in it + +1. Search the controller log for the exact text: + + ```bash + kubectl -n cpaas-system logs deploy/csi-nfs-controller -c nfs --tail=500 | grep -E 'failed to make subdirectory|permission denied' + ``` + +2. From the node that hosts `csi-nfs-controller`, mount with the StorageClass options and try to create a directory: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep csi-nfs-controller + sudo mount -t nfs -o hard,nfsvers=3,nolock : /mnt + sudo mkdir /mnt/provision-test && sudo rmdir /mnt/provision-test + sudo umount /mnt + ``` + + Use the `nfsvers` and options from the StorageClass. If the mount succeeds and `mkdir` fails, this cause is confirmed. + +3. Grant the permissions on the NFS server side. The provisioner creates directories as root, so on a Linux `nfsd` an export with `root_squash` over a root-owned directory blocks it; compare the export with one that provisions correctly (`exportfs -v`). On a storage appliance, set the equivalent permission for the export on the array. If the server configuration is incomplete, finish it first (export defined, clients allowed, service started). + + > **Warning:** Widening export permissions lets the allowed clients create and change files in the export. Do this only when the mount succeeds and the `mkdir` test fails. Back up the export configuration first; roll back by restoring `/etc/exports` from the backup and running `sudo exportfs -ra` (on an appliance, revert the export setting). + + ```bash + sudo cp -a /etc/exports /etc/exports.bak + sudo vi /etc/exports + sudo exportfs -ra + sudo exportfs -v + ``` + +4. Repeat the `mkdir` test, then create the PVC again. The claim should become `Bound`. + +### Cause 2: the cluster nodes cannot reach the NFS server + +1. From the node that hosts `csi-nfs-controller` (and from the other nodes), test the path to the server: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep csi-nfs-controller + rpcinfo -p + timeout 5 bash -c '/2049' && echo reachable + ``` + + A timeout or no route confirms this cause. + +2. Fix the network path between the cluster nodes and the NFS server: routing, firewalls in between, and the storage-side client allowlist. Allow NFS (TCP 2049) and, for NFSv3, the RPC services the server registers (portmapper on 111 and the ports `rpcinfo -p` lists). + +3. When the connectivity test succeeds from every node, create the claim again. If only one node cannot reach the server while others can, investigate that node's routes and source address. + +## Diagnostic Steps + +1. Read the claim's events and the controller log: + + ```bash + kubectl -n describe pvc + kubectl -n cpaas-system get pods -o wide | grep csi-nfs + kubectl -n cpaas-system logs deploy/csi-nfs-controller -c nfs --tail=200 + ``` + +2. On the node that runs `csi-nfs-controller`, repeat the controller's mount by hand: + + ```bash + sudo mount -t nfs -v -o hard,nfsvers=3,nolock : /mnt + ``` + +3. Match the signature: + + | Controller log or manual test | Cause | + | --- | --- | + | Mount succeeds, then `failed to make subdirectory ... permission denied`; manual `mkdir` in the export fails | 1 | + | `timeout after 110s`; manual mount or `/dev/tcp//2049` hangs from the controller node | 2 | + | `rpc.statd`, `exit status 32`, or `access denied by server while mounting` | *NFS PVC stays Pending: csi-nfs-controller mount fails with rpc.statd, exit status 32, or access denied on ACP* | + +The controller Pod uses host networking and the node's route table, so a test from a different node, or from inside an application Pod, does not prove the controller's path works. diff --git a/docs/en/solutions/Old_business_TLS_certificate_still_served_after_an_Ingress_ALB_rule_or_upstream_load_balancer_update_on_ACP.md b/docs/en/solutions/Old_business_TLS_certificate_still_served_after_an_Ingress_ALB_rule_or_upstream_load_balancer_update_on_ACP.md new file mode 100644 index 000000000..58e96b340 --- /dev/null +++ b/docs/en/solutions/Old_business_TLS_certificate_still_served_after_an_Ingress_ALB_rule_or_upstream_load_balancer_update_on_ACP.md @@ -0,0 +1,143 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "3.18" + - "4.0" + - "4.2" + - "4.3" +--- + +# Old business TLS certificate still served after an Ingress, ALB rule, or upstream load balancer update on ACP + +## Issue + +A business certificate, a new wildcard certificate, or an Ingress TLS Secret was updated for a published service, and clients still receive the previous certificate. An incognito browser window, or `openssl s_client`, still shows the old `notBefore` / `notAfter` dates and issuer. A typical report is "the domain certificate replacement does not take effect". + +This article covers the certificate that a published service presents through ALB (and, where present, an Ingress or an upstream load balancer). The platform access certificate (`dex.tls`) and platform component Secrets are covered elsewhere: see [Replace the platform access certificate in dex.tls on ACP](Replace_the_platform_access_certificate_in_dex.tls_on_ACP.md) and [Update a platform domain certificate stored in a cpaas-system Secret on ACP](Update_a_platform_domain_certificate_stored_in_a_cpaas-system_Secret_on_ACP.md). An ingress-nginx controller that serves a renewed certificate only after a restart is Cause 2 of [Business TLS certificate not renewed, not reloaded by ingress-nginx, or rejected with a format error on import on ACP](Business_TLS_certificate_not_renewed_not_reloaded_by_ingress-nginx_or_rejected_with_a_format_error_on_import_on_ACP.md). + +Related articles: *HTTPS to a published service fails with plain HTTP sent to HTTPS port, ERR_SSL_PROTOCOL_ERROR, or TLS alert 80 on ACP* (handshake and protocol errors instead of an old certificate). + +## Root Cause + +Three places can each present a certificate for the same hostname: the ALB rule for that host and port, an Ingress for that host, and an upstream load balancer in front of the cluster. Clients see whichever certificate is presented by the hop that terminates TLS for them. Updating the certificate in one place does not update the others. + +### Cause 1: the Ingress certificate changed and the ALB rule still selects the old one + +The business certificate lives in a Secret (sometimes with a copy in a ConfigMap), and only the Ingress was updated. The certificate selected on the ALB rule for that host was not updated with it, so the ALB keeps serving the old leaf. An Ingress annotation of the form `alb.networking.cpaas.io/tls: =/` and the ALB rule select a Secret; replacing a ConfigMap that holds a copy, or creating a new Secret that the rule does not reference, does not change what the ALB serves. + +A hostname that matches no rule certificate falls through to the default certificate of the HTTPS port. Such names keep the default certificate even after the rule for other hosts is updated; for them, the port default certificate is what has to change. + +### Cause 2: the ALB serves the new certificate and the upstream load balancer still serves the old one + +A new wildcard certificate was selected for the service, there is no second Ingress for that host, and `openssl s_client` against the ALB node shows the new leaf. Users who come through an upstream load balancer (for example an F5 appliance) still get the previous certificate, because that device terminates TLS and presents its own certificate configuration. + +Two designs are valid, and they have different fixes: + +- The upstream device should keep terminating TLS: its certificate has to be updated. +- Only the ALB certificate should be presented: the upstream device should stop terminating TLS and pass TCP through to ALB port 443. + +Forwarding decrypted HTTP from the upstream device to the ALB HTTPS port meets neither goal (see the related article on `The plain HTTP request was sent to HTTPS port`). + +### Cause 3: the certificate was replaced on the ALB and the Ingress certificate was left unchanged + +The service is published with both an Ingress and an ALB rule. The certificate selected on the ALB was replaced, but the Ingress TLS Secret was not, so clients that reach the service through the Ingress keep receiving the previous leaf. Replacing the Ingress certificate as well makes the new domain certificate take effect. + +If the direct ALB check still shows the old leaf, this is Cause 1, not Cause 3. + +## Resolution + +### Cause 1: the Ingress certificate changed and the ALB rule still selects the old one + +1. Read the annotation, the leaf the ALB serves, and the certificate in the Secret: + + ```bash + kubectl get ingress -n -o yaml + echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -dates -subject -issuer + kubectl get secret -n -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject -issuer + ``` + +2. When the Secret holds the new certificate and the ALB still serves the old leaf, open the ALB rule for that host and port and select the same Secret that the Ingress annotation names. This is a normal rule update; it does not need an ALB restart. + +3. If the hostname does not match a rule certificate and the HTTPS port default is the old leaf, update the default certificate of that port instead. + +4. Repeat the `openssl s_client` check against the ALB node and in an incognito browser window. Both should show the new leaf. + +### Cause 2: the ALB serves the new certificate and the upstream load balancer still serves the old one + +1. Compare the ALB node with the public name users type: + + ```bash + echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -dates -subject -issuer + echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -dates -subject -issuer + kubectl get ingress -A -o yaml | grep -F '' + ``` + +2. If the ALB node shows the new certificate, the public name shows the old one, and no other Ingress serves the host: + - When the upstream device should remain the TLS terminator, replace the certificate on the upstream device. + - When only the ALB certificate should be presented, remove the certificate from the upstream device and configure it to pass TCP through to ALB port 443, instead of updating its certificate in place. + +3. If the ALB node itself is still old, go back to Cause 1. If clients reach an Ingress whose Secret was not replaced, use Cause 3. + +### Cause 3: the certificate was replaced on the ALB and the Ingress certificate was left unchanged + +1. Compare the Ingress TLS Secret with the leaf the ALB already serves: + + ```bash + kubectl get ingress -A -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.spec.tls}{"\n"}{end}' + kubectl get secret -n -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject -issuer + ``` + +2. Replace the Ingress TLS Secret with the new certificate and key. + + > **Warning:** Applying a new TLS Secret changes the certificate that the Ingress presents for this host. A mismatched key or a not-yet-valid certificate breaks HTTPS for the host until the previous Secret is restored. Do this only when the service has both an Ingress and an ALB rule, the ALB certificate was already replaced, and this Ingress Secret still holds the old leaf. Back up the Secret first; roll back with `kubectl apply -f` of the backup. + + ```bash + umask 077 + kubectl get secret -n -o yaml > "$HOME/.yaml" + openssl x509 -noout -modulus -in tls.crt | openssl md5 + openssl rsa -noout -modulus -in tls.key | openssl md5 + kubectl create secret tls --cert=tls.crt --key=tls.key -n \ + --dry-run=client -o yaml | kubectl apply -f - + ``` + + The two `md5` values must match before you apply (for an RSA key; for an EC key compare `openssl x509 -noout -pubkey` with `openssl pkey -pubout`). + +3. Check the public name again: + + ```bash + echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -dates -subject -issuer + ``` + + If the Ingress Secret is already new and the ALB leaf is old, use Cause 1 instead. + +## Diagnostic Steps + +Collect three leaves before changing a rule or an upstream device: the Secret the Ingress names, the ALB node, and the public name. + +```bash +kubectl get ingress -n -o yaml +kubectl get secret -n -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject -issuer +echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -dates -subject -issuer +echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -dates -subject -issuer +``` + +| Secret | ALB node | Public name | Cause | +| --- | --- | --- | --- | +| new | old | old | 1 (rule certificate, or the HTTPS port default for an unmatched name) | +| new | new | old, no other Ingress for the host | 2 | +| Ingress Secret old, ALB rule certificate already replaced | new | old | 3 | + +If the client gets no certificate at all, a handshake alert, or an HTTP error instead of an old leaf, see the related article named under Issue. diff --git a/docs/en/solutions/kube-apiserver_latency_alerts_noisy_apiserver_logs_and_controller-manager_or_scheduler_health_alerts_on_ACP.md b/docs/en/solutions/kube-apiserver_latency_alerts_noisy_apiserver_logs_and_controller-manager_or_scheduler_health_alerts_on_ACP.md index 19ad3270a..baf647e05 100644 --- a/docs/en/solutions/kube-apiserver_latency_alerts_noisy_apiserver_logs_and_controller-manager_or_scheduler_health_alerts_on_ACP.md +++ b/docs/en/solutions/kube-apiserver_latency_alerts_noisy_apiserver_logs_and_controller-manager_or_scheduler_health_alerts_on_ACP.md @@ -19,6 +19,7 @@ ProductsVersion: - The monitoring page shows kube-controller-manager as down, but `telnet 127.0.0.1 10257` on the node succeeds and the Pod is healthy. On a managed cluster the platform raises `集群内KubeScheduler的健康状态 != 1 且持续 1 分钟` and `集群内ControllerManager的健康状态 != 1 且持续 1 分钟` ("KubeScheduler / ControllerManager health status in the cluster != 1 for 1 minute", policy `cpaas-cluster-rules`), while both Pods are `1/1 Running` with no restarts. - `kubectl get componentstatuses` shows `controller-manager` and `scheduler` as `Unhealthy`. - An inspection reports apiserver errors tied to an already-compacted etcd revision and a connection reset. +- The apiserver error-rate alert rises and stays high after worker nodes were removed from the cluster, and the failed requests are mostly `404` on `nodes`. Check admission webhooks before scaling kube-apiserver or compacting etcd. @@ -58,6 +59,10 @@ Monitoring probes kube-controller-manager on `:10257` (and kube-schedul The requested historical revision was already removed by etcd auto-compaction, and the connection was reset. etcd has not restarted and is not changing leader. These errors have no impact and can be ignored. A cluster that also logs `leader changed` is not covered by this conclusion. +### Cause 7: removed worker machines are still running and keep requesting their deleted Node objects + +Worker nodes were removed from the cluster, so their Node objects no longer exist, but the machines themselves were left running. Each of them keeps sending `GET` requests to kube-apiserver for its own Node object, and every such request returns `404`. The constant 404 stream raises the apiserver error rate even though the cluster itself is healthy. The 404 rate jumps when the nodes are removed and then stays flat; nearly all of it is `verb=GET` on resource `nodes`, while `PATCH` on `nodes/status` stays at zero and other resources show almost no errors. Shutting down the removed machines brings the error rate back to normal. + ## Resolution ### Cause 1: An admission webhook has no backend @@ -172,6 +177,32 @@ kubectl -n kube-system logs etcd- --tail=200 | grep -F 'leader change With a compacted-revision error plus a connection reset, no etcd restart, and no leader changes, ignore the lines. A `NOSPACE` alarm or leader changes need the etcd quota or disk investigation instead. +### Cause 7: removed worker machines are still running and keep requesting their deleted Node objects + +1. Break the error rate down by code, then by resource, in the monitoring UI: + + ```text + sum by (code) (apiserver_request_count_rate_5m_sum_by_instance_and_code) + topk(20, sum by (resource, code) (apiserver_request_count_rate_5m_sum_by_resource_and_code{code=~"(4|5)..",code!="429"})) + sum by (verb, subresource) (rate(apiserver_request_total{job="apiserver",resource="nodes",code="404"}[5m])) + ``` + + This cause matches when `code="404", resource="nodes"` dominates, almost all of it is `verb="GET"`, and the rise started when the nodes were removed. + +2. Confirm the machines that send these requests are no longer cluster members: their names are absent from `kubectl get nodes`, and the machines are still powered on. + +3. Shut down the removed machines. + + > **Warning:** Shutting a machine down stops everything still running on it. Do this only for machines whose Node objects were deliberately removed from the cluster and that carry nothing else in use. Roll back by powering the machine on again; it will resume the 404 requests until it is shut down or rejoined. + + If a machine has to stay powered on for another purpose, stop and disable the kubelet on it instead, so that it no longer contacts the cluster: + + ```bash + systemctl disable --now kubelet + ``` + +4. Re-run the queries in step 1 after a few minutes. The `404` rate on `nodes` should drop to near zero and the error-rate alert should clear. + ## Diagnostic Steps Webhooks first, then the log signatures, then the controller-manager port: @@ -195,5 +226,6 @@ kubectl get endpoints,pods,deploy -n - `10257` open on `127.0.0.1` and refused on the node IP, or the KubeScheduler / ControllerManager health alerts with ports bound to `127.0.0.1`: Cause 4. - `kubectl get componentstatuses` Unhealthy with the Pods Running and no monitoring failure: Cause 5. - Compacted-revision errors with a quiet etcd: Cause 6. +- The error rate is mostly `404` from `GET` on `nodes`, starting when worker nodes were removed: Cause 7. -Read-success alerts that are mostly HTTP 403 are authorization failures, not a webhook timeout. Capture one request's user and verb before deleting any webhook. +Read-success alerts that are mostly HTTP 403 are authorization failures, not a webhook timeout. Capture one request's user and verb before deleting any webhook. When the 403s come with cert-manager webhook denials for `{cpaas-ca ClusterIssuer ait.alauda.io}` in the kube-apiserver log, they are expected; see *Certificate alerts from a custom VMRule, ait.alauda.io webhook denials, or by-design short-lived certificates on ACP* (Cause 2). From 7245c3affad326a22c60316d526223eaf20f8c53 Mon Sep 17 00:00:00 2001 From: Komh Date: Wed, 18 Nov 2026 17:41:30 +0800 Subject: [PATCH 21/25] docs(solutions): add 8 and update 0 troubleshooting articles --- ...ice_class_does_not_become_an_OSD_on_ACP.md | 214 ++++++++++++++++++ ..._to_move_to_another_device_class_on_ACP.md | 107 +++++++++ ...use_rook-ceph-tools_never_starts_on_ACP.md | 156 +++++++++++++ ...se_the_deployment_created_no_OSD_on_ACP.md | 135 +++++++++++ ...VM_and_thin-pool_mappings_remain_on_ACP.md | 125 ++++++++++ ...nd_file-handle_alerts_fire_on_ACP_nodes.md | 148 ++++++++++++ ...isappeared_or_was_decommissioned_on_ACP.md | 196 ++++++++++++++++ ...fails_to_create_the_volume_group_on_ACP.md | 196 ++++++++++++++++ 8 files changed, 1277 insertions(+) create mode 100644 docs/en/solutions/A_raw_disk_added_to_a_Ceph_device_class_does_not_become_an_OSD_on_ACP.md create mode 100644 docs/en/solutions/Ceph_OSD_crash-loops_with_not_a_block_device_or_an_OSD_has_to_move_to_another_device_class_on_ACP.md create mode 100644 docs/en/solutions/Ceph_operator_install_returns_404_or_the_storage_page_is_empty_because_rook-ceph-tools_never_starts_on_ACP.md create mode 100644 docs/en/solutions/Ceph_storage_pool_cannot_be_created_or_never_finishes_because_the_deployment_created_no_OSD_on_ACP.md create mode 100644 docs/en/solutions/Former_TopoLVM_data_disk_stays_unavailable_because_old_LVM_and_thin-pool_mappings_remain_on_ACP.md create mode 100644 docs/en/solutions/Inspection_reports_remaining_inodes_as_0_or_inode_and_file-handle_alerts_fire_on_ACP_nodes.md create mode 100644 docs/en/solutions/TopoLVM_device_unavailable_or_PVC_unusable_because_the_backing_disk_failed_disappeared_or_was_decommissioned_on_ACP.md create mode 100644 docs/en/solutions/TopoLVM_does_not_offer_a_disk_or_topolvm-prepare-vg_fails_to_create_the_volume_group_on_ACP.md diff --git a/docs/en/solutions/A_raw_disk_added_to_a_Ceph_device_class_does_not_become_an_OSD_on_ACP.md b/docs/en/solutions/A_raw_disk_added_to_a_Ceph_device_class_does_not_become_an_OSD_on_ACP.md new file mode 100644 index 000000000..f215775e9 --- /dev/null +++ b/docs/en/solutions/A_raw_disk_added_to_a_Ceph_device_class_does_not_become_an_OSD_on_ACP.md @@ -0,0 +1,214 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# A raw disk added to a Ceph device class does not become an OSD on ACP + +## Issue + +A bare disk is attached to a node for Ceph distributed storage on Alauda Container Platform, but it never becomes an OSD in the device class the pool uses: + +- The raw disks are attached and Ceph storage has to be expanded (routine expansion). +- A disk was added to the device class from the page and the configuration was accepted. `CephCluster` lists the disk under `spec.storage.nodes`, but nothing happens: `ceph osd tree` shows no new OSD and the capacity does not grow. +- The prepare log shows `skipping device "": ["Has GPT headers"]`. +- The disk was already wiped with `dd`, `wipefs`, or `sgdisk`, and Ceph still does not use it. +- Rook-Ceph raises an abnormal alert, `osd-prepare` is never created for the node, and the node has a taint. + +Work in the `rook-ceph` namespace (`CephCluster` `cephclusters.ceph.rook.io`, operator Deployment `rook-ceph-operator`, toolbox Deployment `rook-ceph-tools`). Ceph commands run in the toolbox unless the step says to run on the node. + +Related articles: [Ceph storage pool cannot be created or never finishes because the deployment created no OSD on ACP](./Ceph_storage_pool_cannot_be_created_or_never_finishes_because_the_deployment_created_no_OSD_on_ACP.md), [Ceph OSD crash-loops with not a block device, or an OSD has to move to another device class on ACP](./Ceph_OSD_crash-loops_with_not_a_block_device_or_an_OSD_has_to_move_to_another_device_class_on_ACP.md), [Ceph device class page rejects a disk, or Ceph does not show the new capacity on ACP](./Ceph_device_class_page_rejects_a_disk_or_Ceph_does_not_show_the_new_capacity_on_ACP.md) (the page offers no device, `至少需要 3 个节点`, USB devices, a grown virtual disk). + +## Root Cause + +### Cause 1: skipping device "": ["Has GPT headers"] + +Rook does not create an OSD on a disk that still has a GPT (or other) signature. A replacement disk and a disk that once belonged to another Ceph cluster both look like this. The disk is not bad; it is not empty enough for prepare. + +### Cause 2: the raw disk is on the node but not in the device class + +The disk is visible in `lsblk` with no filesystem mounted, but `ceph osd tree` has no OSD for it. A pool only places data on its CRUSH device class. Until the disk is added to that class from the Device Class page (or an equivalent `CephCluster` `spec.storage` entry), capacity does not grow. This path assumes healthy, unused disks; damaged disks have to be repaired or replaced first. + +On ACP 4.3.1 the page that adds disks to an existing device class sometimes does not list disks that are already members, and submitting the form can drop those disks from `spec.storage`. + +### Cause 3: the whole disk still carries an LVM2_member label, so no osd-prepare pod starts on the node + +`CephCluster` has the new device, other nodes have `osd-prepare` pods in `Completed`, and the node with the new disk has no prepare pod at all. `lsblk -f` shows the whole disk with `FSTYPE` `LVM2_member`, no partitions, and no mount. `pvs`, `lvs`, and `dmsetup ls` can be empty: no active volume group uses the disk, but the LVM label is still on it. `sgdisk --zap-all` and `fdisk` clear only partition tables, not this label. As long as `blkid` reports a filesystem type for the whole disk, the disk is treated as occupied. + +### Cause 4: disk still occupied after dd, wipefs, and sgdisk + +The disk was cleaned with `dd if=/dev/zero of=/dev/ bs=4k count=512`, `wipefs -af /dev/`, and `sgdisk --zap-all /dev/`, `blkid -p` no longer reports a type, and Ceph still does not use it. Those commands clear signatures and the partition table (the `dd` overwrites about the first 2 MiB) but do not release leftover OSD occupancy of the device on the node. Restarting the node released it. + +### Cause 5: osd-prepare not created because the node is tainted + +Operator logs suggest the affected node may have been deleted, the `osd-prepare` object is not created, and `kubectl describe node` shows a taint. That taint blocks `osd-prepare` on the node. + +## Resolution + +### Cause 1: skipping device "": ["Has GPT headers"] + +1. On the node, confirm the device. It must be the skipped disk, not the OS disk and not a device that backs an `up` OSD: + +```bash +lsblk -o NAME,SIZE,TYPE,FSTYPE,LABEL,MOUNTPOINT,MODEL,SERIAL /dev/ +wipefs /dev/ +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s +``` + +2. If `lsblk` shows a leftover `ceph-*` device-mapper node on this disk only, remove it first. Skip this if there is none. + +> **Warning:** `dmsetup remove` detaches the mapping immediately; if it is an active OSD, that OSD crashes and its PGs go degraded. Precondition: `lsblk` shows the mapping only on the skipped disk, and `ceph osd tree` lists no `up` OSD on it. Rollback: none. + +```bash +dmsetup remove /dev/mapper/ +``` + +3. Wipe the disk. + +> **Warning:** This destroys every signature, the partition table, and the first 100 MiB of `/dev/`. Irreversible. Precondition: a second person has checked the device name against `lsblk`, the disk is not mounted or in `findmnt`, and it is not the OS disk or a live OSD disk. Rollback: none. + +```bash +wipefs -af /dev/ +sgdisk --zap-all /dev/ +dd if=/dev/zero of=/dev/ bs=1M count=100 oflag=direct,dsync +``` + +4. Restart the operator so prepare runs again. + +> **Warning:** Restarting `rook-ceph-operator` restarts only the operator pod; OSD pods keep running, but prepare and reconcile pause until the new pod is Ready. Precondition: the wipe finished on the intended disk. Rollback: none needed; the Deployment spec is unchanged. + +```bash +kubectl -n rook-ceph rollout restart deploy/rook-ceph-operator +kubectl -n rook-ceph rollout status deploy/rook-ceph-operator +``` + +5. Add the disk to the device class (Cause 2). `ceph osd tree` then shows a new OSD `up` and `in`. + +### Cause 2: the raw disk is on the node but not in the device class + +This is the normal expansion path; wiping is not part of it. If a disk added here never becomes an OSD, read the prepare log and go to the matching cause. + +1. Check each disk's health first. I/O or medium errors, or a failed health check, are not this cause; plan a repair or replacement with the storage owner. + +```bash +dmesg -T | grep -i -E '|I/O error|medium error' | tail -n 20 +smartctl -H /dev/ +``` + +2. Save the cluster object: + +```bash +kubectl -n rook-ceph get cephcluster -o yaml > cephcluster-before.yaml +``` + +3. In the console, open **Storage Management** > **Distributed Storage** > the cluster > **Device Class**, open the class the pool uses, choose **Add Device**, select the node, and enter the disk name (`sde`, `nvme0n1`, or whatever `lsblk` showed). Two disks on one node are two names on that node, for example `nvme0n1` and `nvme1n1`. On ACP 4.0.3 the page path was Storage Management, Distributed Storage, Access Storage. Create a new class the same way when the pool is to use a separate class. + +4. After the operator prepares the disk, check from the toolbox: + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd tree +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd df +``` + +Adding disks starts backfill, and available-capacity numbers settle only after backfill finishes. + +5. Diff `spec.storage.nodes` against `cephcluster-before.yaml`; every previous device must still be present. If the ACP 4.3.1 page dropped existing disks, put only the missing device entries back with `kubectl -n rook-ceph edit cephcluster `. Do not `kubectl apply` the whole backup over a cluster that has moved on. See also [Ceph OSDs nearfull, backfill blocked, or OSD usage uneven on ACP](./Ceph_OSDs_nearfull_backfill_blocked_or_OSD_usage_uneven_on_ACP.md), Cause 2. + +### Cause 3: the whole disk still carries an LVM2_member label, so no osd-prepare pod starts on the node + +1. Confirm the label and that no prepare pod exists for the node: + +```bash +lsblk -f /dev/ +blkid -p /dev/; echo exit:$? +pvs; lvs; dmsetup ls +kubectl -n rook-ceph get pods -o wide | grep -i prepare +``` + +2. Clear the label and refresh udev. + +> **Warning:** Impact: `wipefs -af` removes every signature on `/dev/` and `dd` zeroes its first 100 MiB. Irreversible. Precondition: only when `lsblk -f` shows `LVM2_member` on this whole disk, `pvs`, `lvs`, and `dmsetup ls` show no volume group or mapping you still use on it, it is not mounted, and it is not the OS disk. Rollback: none. + +```bash +wipefs -af /dev/ +dd if=/dev/zero of=/dev/ bs=1M count=100 oflag=direct conv=fsync +udevadm trigger --action=change /sys/class/block/ +udevadm settle +blkid -p /dev/; echo exit:$? +lsblk -f /dev/ +``` + +`blkid -p` printing nothing with `exit:2` and `lsblk -f` showing no `FSTYPE` confirm the disk is clean. The operator then prepares it; restart `rook-ceph-operator` as in Cause 1 if no prepare pod appears, and check `ceph osd tree`. + +### Cause 4: disk still occupied after dd, wipefs, and sgdisk + +Use this only when the wipe already ran, `blkid -p` reports nothing (Cause 3 is excluded), and Ceph still does not use the disk. + +```bash +lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT /dev/ +findmnt | grep || true +``` + +> **Warning:** Restarting the node stops every OSD and any MON on it; PGs peer again, and a second storage-node failure during the reboot can drop a replica set. Precondition: the disk is still occupied after the wipe, other replicas are `up`, and you are in a maintenance window. Rollback: the node comes back on its own; a reboot does not restore a wiped disk. + +Restart the node with your node maintenance procedure. When it is `Ready` and the device is free in `lsblk`, add it with Cause 2. + +### Cause 5: osd-prepare not created because the node is tainted + +```bash +kubectl describe node | grep -i -A3 Taints +kubectl -n rook-ceph logs deploy/rook-ceph-operator --tail=200 | grep -i prepare +``` + +> **Warning:** Removing the taint allows pods to schedule on ``, including workloads the taint was holding off. Precondition: you saved the full taint key, value, and effect, and this node is supposed to host OSDs. Rollback: `kubectl taint node =:`. + +```bash +kubectl taint node - +``` + +Then restart `rook-ceph-operator` as in Cause 1 (same Warning) and check that prepare is created: + +```bash +kubectl -n rook-ceph rollout restart deploy/rook-ceph-operator +kubectl -n rook-ceph get pods -o wide | grep -i prepare +``` + +## Diagnostic Steps + +1. List the prepare pods with their nodes, and read the skip reason: + +```bash +kubectl -n rook-ceph get pods -o wide | grep -i prepare +kubectl -n rook-ceph logs | grep -E 'skipping device|Has GPT|' +``` + +Make sure the prepare pod you read runs on the node that holds the disk. + +2. On the node, separate an empty disk from one that still has a signature, a partition, or a mount: + +```bash +lsblk -o NAME,SIZE,TYPE,FSTYPE,LABEL,MOUNTPOINT +blkid -p /dev/; echo exit:$? +findmnt +``` + +A disk with partitions or a mounted filesystem (for example a container runtime disk) is in use; do not wipe it, choose a free disk instead. + +3. Check OSDs and classes: + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd tree +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd crush class ls +``` + +- `Has GPT headers` in the prepare log: Cause 1. +- Clean disk not yet in the device class: Cause 2. +- No prepare pod on the node, `FSTYPE` `LVM2_member` on the whole disk: Cause 3. +- Wipe done, `blkid -p` empty, disk still not used: Cause 4. +- No prepare pod and a taint on the node: Cause 5. diff --git a/docs/en/solutions/Ceph_OSD_crash-loops_with_not_a_block_device_or_an_OSD_has_to_move_to_another_device_class_on_ACP.md b/docs/en/solutions/Ceph_OSD_crash-loops_with_not_a_block_device_or_an_OSD_has_to_move_to_another_device_class_on_ACP.md new file mode 100644 index 000000000..5220e9539 --- /dev/null +++ b/docs/en/solutions/Ceph_OSD_crash-loops_with_not_a_block_device_or_an_OSD_has_to_move_to_another_device_class_on_ACP.md @@ -0,0 +1,107 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# Ceph OSD crash-loops with not a block device, or an OSD has to move to another device class on ACP + +## Issue + +An existing Ceph OSD on Alauda Container Platform is not in the device class where it is needed: + +- An OSD pod is `Init:CrashLoopBackOff` and its log contains `RuntimeError: Error: ["/dev/"] not a block device`. `ceph osd tree` shows that OSD `down` with `REWEIGHT` `0`, and the storage view shows the same disk assigned to TopoLVM. +- An OSD has to move from one device class to another (for example from a CephFS class to an object-storage class). Adding `spec.storage.allowDeviceClassUpdate` to `CephCluster` and editing the OSD `deviceClass` there returns `unknown field "spec.storage.allowDeviceClassUpdate"`. + +Work in the `rook-ceph` namespace; Ceph commands run in `deploy/rook-ceph-tools`. + +Related articles: [A raw disk added to a Ceph device class does not become an OSD on ACP](./A_raw_disk_added_to_a_Ceph_device_class_does_not_become_an_OSD_on_ACP.md), [Ceph storage pool cannot be created or never finishes because the deployment created no OSD on ACP](./Ceph_storage_pool_cannot_be_created_or_never_finishes_because_the_deployment_created_no_OSD_on_ACP.md). + +## Root Cause + +### Cause 1: RuntimeError: Error: ["/dev/"] not a block device + +The disk that the OSD was prepared on has been taken by TopoLVM. Ceph cannot activate an OSD on a path that is no longer the block device it prepared, so the OSD init container fails on every start. The other OSDs stay `up`. + +### Cause 2: unknown field "spec.storage.allowDeviceClassUpdate" + +The `CephCluster` CRD of the installed version does not have `spec.storage.allowDeviceClassUpdate` (confirmed with Ceph 17.2.7, Quincy). The field is rejected and removed, and the OSD stays in the old class: `ceph osd tree` still shows the previous class name. The device class of an existing OSD is a property of the CRUSH map. + +## Resolution + +### Cause 1: RuntimeError: Error: ["/dev/"] not a block device + +Use this only when the OSD log contains `not a block device` and the storage view shows that disk assigned to TopoLVM. Give Ceph a different disk: retire the broken OSD and add a replacement disk. Do not take the disk back from TopoLVM while TopoLVM volumes live on it. + +1. Confirm the signature and the OSD id: + +```bash +kubectl -n rook-ceph get pods -o wide | grep rook-ceph-osd- +kubectl -n rook-ceph logs --all-containers | grep -F 'not a block device' +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd tree +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s +kubectl -n rook-ceph get cephcluster -o yaml > cephcluster-before.yaml +``` + +2. Retire the OSD. + +> **Warning:** `ceph osd out`, `ceph osd crush remove`, `ceph auth del`, and `ceph osd rm` retire this OSD id. Impact: any PG whose only remaining copy was on it loses that data. Precondition: the log signature and the TopoLVM assignment are both confirmed, `ceph pg dump` shows another `up` copy for the PGs that were on this OSD, and `cephcluster-before.yaml` is saved. Rollback: none for `osd rm`; add the replacement disk and let backfill restore redundancy. + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd out osd. +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd crush remove osd. +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph auth del osd. +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd rm osd. +``` + +3. Add a replacement disk to the device class (Cause 2 of [A raw disk added to a Ceph device class does not become an OSD on ACP](./A_raw_disk_added_to_a_Ceph_device_class_does_not_become_an_OSD_on_ACP.md)). It has to be a block device that the storage view does not list under TopoLVM. Remove the old device entry from `spec.storage.nodes` of `CephCluster` so the operator does not try to prepare it again. + +### Cause 2: unknown field "spec.storage.allowDeviceClassUpdate" + +Do not set `allowDeviceClassUpdate` on this CRD. Change the class in the CRUSH map. + +1. Record the current class: + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd crush get-device-class osd. +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd tree +``` + +2. Move the OSD. + +> **Warning:** Removing and setting the device class makes CRUSH remap PGs; the cluster backfills, client IO rises, and a failure during backfill has less spare redundancy. Precondition: `ceph -s` is healthy enough to backfill (no full OSDs), and you saved the old class name. Rollback: run the same two commands with the saved class name. The class change itself does not delete data. + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd crush rm-device-class osd. +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd crush set-device-class osd. +``` + +3. Watch `ceph -s` until all PGs return to `active+clean`. The target pool must use a CRUSH rule that selects ``, or the OSD does not receive that pool's data: + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd crush rule dump +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool ls detail +``` + +## Diagnostic Steps + +```bash +kubectl -n rook-ceph get pods -o wide | grep rook-ceph-osd- +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd tree +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd crush class ls +``` + +On the node of the failing OSD: + +```bash +lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT /dev/ +``` + +- `Init:CrashLoopBackOff`, `not a block device` in the log, and the disk listed under TopoLVM (or `lsblk` shows TopoLVM logical volumes on it): Cause 1. +- `unknown field "spec.storage.allowDeviceClassUpdate"` when editing `CephCluster`, and `ceph osd tree` still shows the old class: Cause 2. diff --git a/docs/en/solutions/Ceph_operator_install_returns_404_or_the_storage_page_is_empty_because_rook-ceph-tools_never_starts_on_ACP.md b/docs/en/solutions/Ceph_operator_install_returns_404_or_the_storage_page_is_empty_because_rook-ceph-tools_never_starts_on_ACP.md new file mode 100644 index 000000000..6ca197257 --- /dev/null +++ b/docs/en/solutions/Ceph_operator_install_returns_404_or_the_storage_page_is_empty_because_rook-ceph-tools_never_starts_on_ACP.md @@ -0,0 +1,156 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.0.x + - 4.3.x +--- + +# Ceph operator install returns 404, or the storage page is empty because rook-ceph-tools never starts on ACP + +## Issue + +Ceph distributed storage on Alauda Container Platform shows "no Ceph operator" or an empty distributed-storage page, in one of these shapes: + +- Installing the Ceph operator returns 404, the operator catalog page has nothing to install, and `kubectl get packagemanifests -A` prints no rows. +- A `CephCluster` exists and the operator is up, but the storage page does not list the cluster and the device class is empty. `rook-ceph-tools` is not Running, and its container spec has `tty: true`. +- Storage-pool installation runs for a long time, and `ceph-cluster` stays `HEALTH_WARN` with `TOO_FEW_OSDS` (`OSD count 0 < osd_pool_default_size 3`), often together with `PG_AVAILABILITY` (`Reduced data availability: 2 pgs inactive`) and `MDS_SLOW_METADATA_IO`, while the tools pod has `tty: true`. +- On ACP 3.18 the `rook-ceph-operator` Deployment disappears or comes back, and the pod fails at startup with `C | rookcmd: failed to get pod: pods "rook-ceph-operator-" is forbidden: User "system:serviceaccount:rook-ceph:rook-ceph-system" cannot get resource "pods" in API group "" in the namespace "rook-ceph"`. Creating the operator from the Distributed Storage page fails with the same error. + +Environment: Rook namespace `rook-ceph`, toolbox Deployment `rook-ceph-tools`, operator Deployment `rook-ceph-operator`. On ACP 4.3 the OLM components `packageserver`, `catalog-operator`, and `olm-operator` run in `cpaas-system`. On ACP 4.3.4 the CoreDNS Corefile is ConfigMap `kube-system/cpaas-coredns`, data key `Corefile`. + +Related articles: [Ceph storage pool cannot be created or never finishes because the deployment created no OSD on ACP](./Ceph_storage_pool_cannot_be_created_or_never_finishes_because_the_deployment_created_no_OSD_on_ACP.md) (the tools pod is Running and still no OSD exists), [A raw disk added to a Ceph device class does not become an OSD on ACP](./A_raw_disk_added_to_a_Ceph_device_class_does_not_become_an_OSD_on_ACP.md). + +## Root Cause + +### Cause 1: packagemanifests empty because packageserver cannot resolve a name it needs + +`packageserver` logs show errors, and from inside that pod a DNS name it needs does not resolve. Without packagemanifests, the catalog has no Ceph operator to install and the install API answers 404. The operator Deployment is absent because OLM never got a package, not because `rook-ceph-operator` crashed. Adding the missing resolution entries to CoreDNS let the operator install. The name that fails depends on the environment. + +### Cause 2: rook-ceph-tools pod rejected because tty is true + +The tools container spec contains `tty: true`. A security policy or a host security component on the cluster refuses or terminates a container with a TTY, so `rook-ceph-tools` never becomes Ready. The distributed-storage page loads the cluster through the tools pod. While it is down, the page shows no cluster and the device class looks empty even when `CephCluster` is fine and OSD pods are Running, and storage-pool installation cannot finish, leaving the OSD count at 0. Redeploying the `CephCluster` does not help, and neither does uninstalling the operator (its objects in `app-storage` and `rook-ceph` remain). + +The tools Deployment is rendered from the operator's CSV. Changing `tty` in the CSV prevents the next reconcile from putting `tty: true` back, but a CSV edit alone may not roll out to the running Deployment; patching the live Deployment is what brings the pod up. Check both objects. + +### Cause 3: rook-ceph-system cannot get pods on ACP 3.18 + +The operator is OLM-managed: its Deployment carries `olm.managed: "true"` and `olm.owner.kind: ClusterServiceVersion`, with Subscription `rook-operator` in `rook-ceph`. The Deployment and the RBAC it depends on went missing, so the recreated pod runs without permission to get pods. Adding roles and service accounts by hand starts the pod, but the platform still does not see the Ceph cluster. Deleting the Subscription and the CSV and redeploying `ceph-operator` restores it. Why the Deployment and its RBAC went missing is not known. + +## Resolution + +### Cause 1: packagemanifests empty because packageserver cannot resolve a name it needs + +1. Confirm the failure is DNS inside `packageserver`, and save CoreDNS before editing it. If `cpaas-coredns` does not exist on your version, list `kubectl -n kube-system get configmap` and use the ConfigMap that holds the `Corefile` key. + +```bash +kubectl get packagemanifests -A +kubectl -n cpaas-system logs deploy/packageserver --tail=200 +kubectl -n cpaas-system exec deploy/packageserver -- cat /etc/resolv.conf +kubectl -n kube-system get configmap cpaas-coredns -o yaml > cpaas-coredns.backup.yaml +``` + +2. Add a `hosts` entry for the name that failed, with an address that is valid on this cluster. + +> **Warning:** A bad Corefile makes CoreDNS fail and every in-cluster lookup fail, including Kubernetes service DNS. Precondition: `cpaas-coredns.backup.yaml` exists and has been read back, and you only add a `hosts` record for the name `packageserver` could not resolve. Rollback: `kubectl -n kube-system apply -f cpaas-coredns.backup.yaml` and wait until the CoreDNS pods are Ready. + +```bash +kubectl -n kube-system edit configmap cpaas-coredns +``` + +Inside the `Corefile` value, a block of this shape is enough (replace the documentation address and the name): + +```text +hosts { + 192.0.2.10 + fallthrough +} +``` + +Keep the existing `forward` and zone blocks. CoreDNS reloads the ConfigMap. + +3. If `packagemanifests` is still empty after lookups succeed, restart `packageserver`. + +> **Warning:** Restarting `packageserver` interrupts operator catalog reads for about a minute. It does not delete Subscriptions or Ceph data. Precondition: CoreDNS pods are Ready and the name resolves. Rollback: none needed. + +```bash +kubectl -n cpaas-system rollout restart deploy/packageserver +kubectl get packagemanifests -A +``` + +When the Ceph package is listed, install it again from the console. + +### Cause 2: rook-ceph-tools pod rejected because tty is true + +1. Show which container has `tty` set, and read the rejection: + +```bash +kubectl -n rook-ceph get deploy rook-ceph-tools -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" tty="}{.tty}{"\n"}{end}' +kubectl -n rook-ceph describe deploy rook-ceph-tools +kubectl -n rook-ceph get pods -l app=rook-ceph-tools -o wide +kubectl -n rook-ceph get csv | grep rook-ceph +kubectl -n rook-ceph get csv -o yaml | grep -n -5 tty +``` + +2. Set the tools container `tty` to `false` in the CSV (the container under the CSV's `rook-ceph-tools` Deployment spec), so reconciles stop rendering `tty: true`: + +```bash +kubectl -n rook-ceph get csv -o yaml > rook-ceph-csv-before.yaml +kubectl -n rook-ceph edit csv +``` + +3. If the Deployment still shows `tty: true`, patch it. + +> **Warning:** Patching `rook-ceph-tools` recreates the toolbox pod; `ceph` commands from the toolbox fail until the new pod is Ready, while OSDs and client IO keep running. Precondition: container index 0 is the one whose `tty` printed `true` (change the index if not). Rollback: set `tty` back with the same patch only if the security policy allows it. + +```bash +kubectl -n rook-ceph patch deploy rook-ceph-tools --type=json \ + -p='[{"op":"replace","path":"/spec/template/spec/containers/0/tty","value":false}]' +kubectl -n rook-ceph rollout status deploy/rook-ceph-tools +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s +``` + +The storage page lists the cluster again once `ceph -s` works from that pod, and pool installation proceeds past the `OSD count 0` warning that the missing tools pod caused. If `MDS_SLOW_METADATA_IO` remains after the tools pod is up, it is a different fault. + +### Cause 3: rook-ceph-system cannot get pods on ACP 3.18 + +Use this only when the operator log contains `cannot get resource "pods"` for `system:serviceaccount:rook-ceph:rook-ceph-system` in `rook-ceph`. + +```bash +kubectl get subs -A | grep -i -E 'rook|ceph' +kubectl get csv -A | grep -i ceph +kubectl -n rook-ceph get deploy rook-ceph-operator -o jsonpath='{.metadata.labels}{"\n"}' +kubectl -n rook-ceph logs deploy/rook-ceph-operator --tail=50 +``` + +> **Warning:** Deleting the Subscription and the CSV makes OLM remove the operator Deployment and the RBAC it owns; no Rook reconcile happens until the operator is redeployed. Running MON, OSD, and RGW pods are not deleted by this step, and the `CephCluster` resource stays. Precondition: the forbidden error above is in the log, the names come from the `get subs` and `get csv` output for the Ceph operator in `rook-ceph`, and both objects are saved. Rollback: reinstall the operator from the Distributed Storage page or the operator catalog. + +```bash +kubectl -n rook-ceph get subs rook-operator -o yaml > rook-operator-subs-before.yaml +kubectl -n rook-ceph get csv -o yaml > rook-ceph-csv-before.yaml +kubectl -n rook-ceph delete subs rook-operator +kubectl -n rook-ceph delete csv +``` + +Then redeploy `ceph-operator` from the Distributed Storage page (or the catalog path that installed it). When the new pod is Running, confirm that the platform lists the Ceph cluster again: + +```bash +kubectl -n rook-ceph get pods | grep operator +kubectl -n rook-ceph get cephcluster +``` + +## Diagnostic Steps + +```bash +kubectl get packagemanifests -A +kubectl -n rook-ceph get deploy,pods | grep -E 'operator|tools' +kubectl -n rook-ceph get cephcluster +``` + +- No packagemanifests and no `rook-ceph-operator`: Cause 1. Read the `packageserver` logs in `cpaas-system` for lookup errors, then compare Services and Endpoints for the name that failed (`kubectl -n cpaas-system get svc,endpoints`). +- Operator and `CephCluster` present, tools pod not Ready, `tty=true`, or pool installation on `HEALTH_WARN` with `TOO_FEW_OSDS` while the tools pod has `tty: true`: Cause 2. `kubectl -n rook-ceph describe pod -l app=rook-ceph-tools` shows the policy or security-component rejection. +- Operator pod forbidden to get pods as `rook-ceph-system` on ACP 3.18 (`rookcmd: failed to get pod ... is forbidden`): Cause 3. A hand-added Role that only makes the pod start is not sufficient. +- Tools pod Running and still no OSD, or an empty device class: a disk-prepare or device-name problem; see the related article on deployments that created no OSD. diff --git a/docs/en/solutions/Ceph_storage_pool_cannot_be_created_or_never_finishes_because_the_deployment_created_no_OSD_on_ACP.md b/docs/en/solutions/Ceph_storage_pool_cannot_be_created_or_never_finishes_because_the_deployment_created_no_OSD_on_ACP.md new file mode 100644 index 000000000..fc4699316 --- /dev/null +++ b/docs/en/solutions/Ceph_storage_pool_cannot_be_created_or_never_finishes_because_the_deployment_created_no_OSD_on_ACP.md @@ -0,0 +1,135 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# Ceph storage pool cannot be created or never finishes because the deployment created no OSD on ACP + +## Issue + +A new Ceph distributed storage cluster on Alauda Container Platform is deployed, but no OSD is created, so the storage pool cannot be built on it: + +- In step 3 of the distributed storage wizard (create storage pool), the device class dropdown shows `无数据` ("no data") for the class created in step 2. +- Creating a block storage pool does not complete, and some deployment options show no output. +- `CephCluster` is `Ready` with `HEALTH_WARN`, the `rook-ceph-osd-prepare-*` pods are `Completed`, and there is no `rook-ceph-osd-` pod. `kubectl describe cephcluster` lists versions only for `mgr` and `mon`, with no OSD entry. + +With no OSD there is no CRUSH device class, so the pool form has nothing to offer, and a replicated pool cannot place its copies. + +Work in the `rook-ceph` namespace (`CephCluster`, operator Deployment `rook-ceph-operator`, toolbox Deployment `rook-ceph-tools`). + +Related articles: [A raw disk added to a Ceph device class does not become an OSD on ACP](./A_raw_disk_added_to_a_Ceph_device_class_does_not_become_an_OSD_on_ACP.md) (disks added to an existing cluster), [Ceph OSD crash-loops with not a block device, or an OSD has to move to another device class on ACP](./Ceph_OSD_crash-loops_with_not_a_block_device_or_an_OSD_has_to_move_to_another_device_class_on_ACP.md), [Ceph operator install returns 404, or the storage page is empty because rook-ceph-tools never starts on ACP](./Ceph_operator_install_returns_404_or_the_storage_page_is_empty_because_rook-ceph-tools_never_starts_on_ACP.md) (the tools pod is not Running). + +## Root Cause + +### Cause 1: leftover data on the disks, so prepare completes without creating an OSD + +The disks selected for the cluster still carry data from an earlier use. Prepare runs, skips them, and finishes as `Completed` without creating an OSD. This is confirmed on ACP 4.3.3 (Ceph 19.2.5), where cleaning the disks resolved it. A `ReconcileFailed` event `failed to add finalizer ... the object has been modified` next to `ReconcileSucceeded` can appear at the same time; it is not the cause. + +### Cause 2: the device name was entered with extra quotes + +The disk name was typed with extra quotes during deployment, so `CephCluster` records it literally, for example: + +```yaml +storage: + useAllDevices: false + nodes: + - name: + devices: + - name: '"vde"' +``` + +The device name is `"vde"` (with the double quotes), which matches no device on the node. Storage initialization fails because no OSD is created on `vde`. + +## Resolution + +### Cause 1: leftover data on the disks, so prepare completes without creating an OSD + +1. Confirm that no OSD was created and the tools pod is Running: + +```bash +kubectl -n rook-ceph get pods | grep -E 'rook-ceph-osd-|rook-ceph-tools' +kubectl -n rook-ceph describe cephcluster | grep -A12 -i versions +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s +kubectl -n rook-ceph logs | grep -i -E 'skipping|available' +``` + +Only `rook-ceph-osd-prepare-*` pods in `Completed`, no `rook-ceph-osd-` pod, and no OSD in `ceph -s` match this cause. If the tools pod is not Running, use the related article on the tools pod instead. + +2. On each disk node, confirm the target is an unused data disk, and find the CSI pod running on that node: + +```bash +lsblk -o NAME,SIZE,TYPE,FSTYPE,LABEL,MOUNTPOINT /dev/ +kubectl -n rook-ceph get pods -o wide | grep -i csi +``` + +3. Zap the disk from inside that CSI pod. + +> **Warning:** `ceph-bluestore-tool zap-device` erases BlueStore labels and signatures on `/dev/`. Irreversible. Precondition: only when step 1 matches, `lsblk` shows this is the disk you intend to give Ceph, it is not mounted, it is not the OS disk, and it does not back an `up` OSD. Rollback: none. + +```bash +kubectl -n rook-ceph exec -- ceph-bluestore-tool zap-device --dev /dev/ +``` + +If `exec` reports that a container must be chosen, rerun with `-c `. The signature wipe on the node in Cause 1 of [A raw disk added to a Ceph device class does not become an OSD on ACP](./A_raw_disk_added_to_a_Ceph_device_class_does_not_become_an_OSD_on_ACP.md) is the alternative when the tool is not available in the pod. + +4. If no OSD pod appears after the disks are clean, restart the operator so prepare runs again. + +> **Warning:** Restarting `rook-ceph-operator` restarts only the operator pod; prepare and reconcile pause until it is Ready. Precondition: the disks are clean. Rollback: none needed. + +```bash +kubectl -n rook-ceph rollout restart deploy/rook-ceph-operator +kubectl -n rook-ceph get pods | grep -E 'rook-ceph-osd-' +``` + +5. When OSD pods are Running, the device class appears in the pool wizard; create the pool again. + +### Cause 2: the device name was entered with extra quotes + +1. Look at the device names exactly as stored: + +```bash +kubectl -n rook-ceph get cephcluster -o jsonpath='{range .spec.storage.nodes[*]}{.name}{": "}{.devices}{"\n"}{end}' +``` + +A name such as `"\"vde\""` or one that contains `'` or `"` is this cause. + +> **Warning:** Editing `spec.storage` makes the operator prepare the devices named there. A wrong name can point prepare at a different disk. Precondition: you saved the object, and the corrected name is the free data disk that `lsblk` shows on that node. Rollback: apply the saved device entries with `kubectl -n rook-ceph edit cephcluster `. + +2. Correct the names: + +```bash +kubectl -n rook-ceph get cephcluster -o yaml > cephcluster-before.yaml +kubectl -n rook-ceph edit cephcluster +``` + +Under `spec.storage.nodes[].devices`, set each name to the bare device name, for example `- name: vde`, on every node that has the wrong value. + +3. Watch prepare create the OSDs and the pool finish: + +```bash +kubectl -n rook-ceph get pods | grep -E 'rook-ceph-osd-' +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd tree +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s +``` + +When entering disk names in the wizard, type the bare name as `lsblk` shows it (`vde`), without quotes. The `?` help icon next to each field on the create page describes what the field expects. + +## Diagnostic Steps + +```bash +kubectl -n rook-ceph get cephcluster +kubectl -n rook-ceph get pods -o wide | grep -E 'osd|tools' +kubectl -n rook-ceph get cephcluster -o yaml | grep -n -A6 'nodes:' +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s +``` + +- Prepare pods `Completed`, no OSD pod, device names correct, and `lsblk -f` or the prepare log shows the disks still hold data: Cause 1. +- Device names in `spec.storage.nodes[].devices` wrapped in extra quotes: Cause 2. +- `rook-ceph-tools` not Running, or `tty: true` on its container: see the related article on the tools pod. +- An existing cluster where a newly added disk does not become an OSD: see the related article on raw disks added to a device class. diff --git a/docs/en/solutions/Former_TopoLVM_data_disk_stays_unavailable_because_old_LVM_and_thin-pool_mappings_remain_on_ACP.md b/docs/en/solutions/Former_TopoLVM_data_disk_stays_unavailable_because_old_LVM_and_thin-pool_mappings_remain_on_ACP.md new file mode 100644 index 000000000..f008e8baf --- /dev/null +++ b/docs/en/solutions/Former_TopoLVM_data_disk_stays_unavailable_because_old_LVM_and_thin-pool_mappings_remain_on_ACP.md @@ -0,0 +1,125 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.16.x + - 3.18.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Former TopoLVM data disk stays unavailable because old LVM and thin-pool mappings remain on ACP + +## Issue + +After a cluster cleanup, or after the platform was deleted and reinstalled, the old TopoLVM data disk is not raw. Creating local storage reports the disk as unavailable, `lsblk -f` shows `LVM2_member`, and `dmsetup ls` still lists thin-pool mappings (`*_tmeta`, `*_tdata`, and the pool). `lsblk` on the data disk still shows the old TopoLVM devices, named after the old volume group: + +```text +vdc 500G disk +├─topolvm---topolvm--thin--pool_tmeta 228M lvm +│ └─topolvm---topolvm--thin--pool 449.6G lvm +│ ├─topolvm--- 5G lvm +│ └─topolvm--- 5G lvm +└─topolvm---topolvm--thin--pool_tdata 449.6G lvm + └─topolvm---topolvm--thin--pool 449.6G lvm +``` + +The 5G entries are the logical volumes that backed the old PVCs. + +Related articles: [TopoLVM does not offer a disk or topolvm-prepare-vg fails to create the volume group on ACP](./TopoLVM_does_not_offer_a_disk_or_topolvm-prepare-vg_fails_to_create_the_volume_group_on_ACP.md) (old signatures while TopoLVM is still installed, a blank disk, a disk missing from `TopolvmCluster`, or a multipath device), [TopoLVM device unavailable or PVC unusable because the backing disk failed, disappeared, or was decommissioned on ACP](./TopoLVM_device_unavailable_or_PVC_unusable_because_the_backing_disk_failed_disappeared_or_was_decommissioned_on_ACP.md). + +## Root Cause + +### Cause 1: LVM2_member and thin-pool dm mappings remain after the cluster is gone + +The volume group, its thin pool, and the logical volumes of the old installation are still on the disk, and the kernel still holds their device-mapper mappings. They keep the disk occupied, so it is not offered as a raw disk. The TopoLVM operator that created them is already gone, so nothing deletes the volume group for you. `wipefs` does not remove device-mapper mappings that are still open, so wiping alone fails or leaves the disk busy. + +## Resolution + +### Cause 1: LVM2_member and thin-pool dm mappings remain after the cluster is gone + +Use this only when the signature is on the former data disk, the path is not the disk that holds `/`, and `findmnt` and `pvs` show no consumer you still need. If `wipefs -n` is empty and `dmsetup ls` has no mappings for this disk, stop: this article does not apply (see the related article on disks TopoLVM does not offer). + +The same LV, VG, PV, and `dmsetup` steps also apply on a cluster where TopoLVM is still installed, when `wipefs -a` reports the device busy on a node whose pods rebuild forever. There they apply only after that node has been removed from `TopolvmCluster` and `lvs` shows no logical volume that backs a bound PV. + +1. List the volume group and mappings: + +```bash +lsblk -f /dev/ +dmsetup ls +lvs -a -o lv_name,vg_name,lv_attr,pool_lv +pvs +vgs +findmnt -n -o SOURCE / +``` + +2. Remove the logical volumes, the volume group, and the physical-volume label, in that order. + +> **Warning:** Impact: `lvremove`, `vgremove`, and `pvremove` delete the logical volumes, the volume group, and the physical-volume label on `/dev/`; data on those volumes is destroyed. Precondition: TopoLVM has already been removed (or the node has been removed from `TopolvmCluster` as described above), `lsblk -f` shows `LVM2_member` on this data disk, and it is not the OS disk and not a volume another host still mounts. Rollback: none. + +```bash +lvremove / +vgremove +pvremove /dev/ +``` + +Remove every logical volume in that volume group, including the thin pool, before `vgremove`. If `vgremove` says the group is busy, run step 4 and retry. + +3. Clear the remaining signatures. + +> **Warning:** Impact: `dd` overwrites the start of `/dev/`, `wipefs -af` removes every filesystem, RAID, and partition signature, and `sgdisk --zap-all` destroys the GPT and the protective MBR. Precondition: step 2 has finished, or has failed only because leftover mappings are still open, and `wipefs -n` still lists a signature. Rollback: none. If `sgdisk` is not installed, run the `dd` and `wipefs` lines. + +```bash +dd if=/dev/zero of=/dev/ bs=4k count=512 +wipefs -af /dev/ +sgdisk --zap-all /dev/ +``` + +4. Remove leftover device-mapper mappings of that volume group. + +> **Warning:** Impact: `dmsetup remove` deletes the named device-mapper mapping; a mapping that still holds a thin pool drops that pool. Precondition: `dmsetup ls` still lists mappings for the volume group you just removed (`*_tmeta`, `*_tdata`, and the pool), and those names belong to this disk, not to another volume group. Rollback: none. + +```bash +dmsetup ls +dmsetup remove +``` + +Repeat `dmsetup remove` for each leftover mapping of that volume group. A hyphen in a volume-group or logical-volume name is written as two hyphens in the device-mapper name (`topolvm---...`). When `dmsetup ls` no longer lists those mappings and `wipefs -n /dev/` prints nothing, the disk is raw again and can be used to create local storage. + +5. After a platform reinstall, if creation still reports the disk unavailable while `topolvm---...` nodes remain under `/dev/mapper`, remove those mapper devices and reboot the node; the reboot is part of this recovery. List them by the old volume-group prefix first: + +```bash +ls -l /dev/mapper | grep topolvm +lsblk /dev/ +``` + +> **Warning:** Impact: removing a device-mapper node drops that mapping, and a reboot interrupts every pod on the node. Precondition: the names belong to the removed TopoLVM volumes, `dmsetup ls` does not show a volume group another workload still mounts, and steps 2 to 4 are done or those groups are already gone. Drain or plan downtime for the node's workloads first. Rollback: none for the mapping; the node comes back from the reboot on its own. + +```bash +dmsetup remove +reboot +``` + +After the node is back, confirm the disk is raw and create local storage on it again: + +```bash +lsblk -f /dev/ +wipefs -n /dev/ +``` + +## Diagnostic Steps + +```bash +lsblk -f /dev/ +lsblk /dev/ +dmsetup ls +pvs; vgs; lvs -a +kubectl get topolvmclusters.topolvm.cybozu.com -A +``` + +- TopoLVM already removed or the platform reinstalled, `LVM2_member` on the disk, and thin-pool mappings in `dmsetup ls` or `topolvm---...` children under the disk in `lsblk`: this article. +- The same signatures while TopoLVM is installed and the node is still in `TopolvmCluster`: see the related article on disks TopoLVM does not offer, and check `lvs` against bound PVs before removing anything. +- `wipefs -n` empty and no mappings: the disk is already raw; the problem is elsewhere (devices file, `TopolvmCluster` entry, or a blank disk). diff --git a/docs/en/solutions/Inspection_reports_remaining_inodes_as_0_or_inode_and_file-handle_alerts_fire_on_ACP_nodes.md b/docs/en/solutions/Inspection_reports_remaining_inodes_as_0_or_inode_and_file-handle_alerts_fire_on_ACP_nodes.md new file mode 100644 index 000000000..bf07222c6 --- /dev/null +++ b/docs/en/solutions/Inspection_reports_remaining_inodes_as_0_or_inode_and_file-handle_alerts_fire_on_ACP_nodes.md @@ -0,0 +1,148 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Inspection reports remaining inodes as 0, or inode and file-handle alerts fire on ACP nodes + +## Issue + +On Alauda Container Platform one of the following is reported for some or all nodes of a cluster: + +- The platform inspection reports that the remaining inode count on the node is 0 (`节点内剩余inode数为0`, "the remaining inode count on the node is 0"), although the root filesystem still has inodes. +- An inode usage alert fires on some nodes. +- A file-handle usage alert fires, but the node looks fine. + +Only one of the three shapes below is a node that really cannot create files. `df -h` free bytes say nothing about inodes; compare `df -i` per mount with the number the alert or the inspection shows before cleaning anything. + +## Root Cause + +### Cause 1: df -i IUse% near 100% on the root or /var filesystem + +`df -h` still has free bytes, `df -i` on `/` or `/var` shows almost no free inodes, and the node condition mentions `nodefs.inodesFree` or `imagefs.inodesFree`. This is real inode exhaustion, caused by many tiny files, often under `/var/log`, a container log directory, or an overlay upper directory. When the kubelet `inodesFree` eviction threshold is crossed, the node also reports `DiskPressure=True`. + +### Cause 2: inspection remaining inode is 0, while df -i on / still has free inodes + +`df -i /boot/efi` is the exhausted number; `df -i /` and `df -i /var` are not. The inspection indicator `node.disk.inode.free`, defined in ConfigMap `indicators-standard-node` in `cpaas-system`, reads `node_filesystem_files_free` with `device!~"rootfs|tmpfs"` and no mount-point filter. The `/boot/efi` partition is typically vfat and reports no inode pool, so its 0 becomes the node's value. The alert is a false positive: `/boot/efi` is normally unused, the node can still create files, and scheduling is not affected. + +### Cause 3: inode or file-handle alert fires, but the node still has plenty of both + +`df -i` on the alarmed mount and `/proc/sys/fs/file-nr` show ample headroom, and kubelet reports no pressure condition. The inode and file-handle thresholds of the monitoring rules are low for that environment. No remediation is required on the nodes. + +## Resolution + +### Cause 1: df -i IUse% near 100% on the root or /var filesystem + +1. Find the directory with the most files. `du -x` stays on one filesystem; a `find` loop over `/*` also walks `/proc` and floods the output with errors such as `Too many levels of symbolic links`. + +```bash +df -i +du -x --inodes -d 1 | sort -n | tail +``` + +Repeat `du` on the largest directory until you reach the one where small files accumulate. + +2. Remove files you are allowed to lose (rotated logs, expired cache). Do not delete `/boot/efi` to fix an inode alarm. + +> **Warning:** Deleting a large set of small files is irreversible. Precondition: the directory is not a live etcd, Elasticsearch, or containerd data directory, and you have listed a sample of the names. Rollback: none. + +```bash +find -xdev -type f -name '' -print | head +find -xdev -type f -name '' -delete +df -i +``` + +Truncating a file frees bytes but not its inode, and a deleted file that a process still holds open keeps its inode until it is closed. When `df -i` is the number at 100%, the count drops only after closed files are removed. For container logs under `/var/log/pods`, let the runtime rotate them instead of removing directories it still has open. + +### Cause 2: inspection remaining inode is 0, while df -i on / still has free inodes + +1. Confirm the false positive. If `/` or `/var` is actually out of inodes, use Cause 1 instead. + +```bash +df -i / /var /boot/efi +``` + +2. On a control-plane node of the affected cluster, look at the indicator: + +```bash +kubectl -n cpaas-system get configmap indicators-standard-node -o yaml | grep -n -A3 'node.disk.inode.free' +``` + +If the indicator is not there, find the ConfigMap that holds the inode query (a search for `boot/efi` alone can miss it, because the query has no mount-point filter): + +```bash +kubectl get configmap -A -o json \ + | jq -r '.items[] | select((.data // {}) | tostring | test("node.disk.inode.free|node_filesystem_files_free")) | "\(.metadata.namespace)/\(.metadata.name)"' +``` + +3. Exclude `/boot/efi` from the query. + +> **Warning:** A bad edit can hide real filesystem alerts, not only `/boot/efi`. Precondition: `df -i /` shows free inodes, and the ConfigMap is saved. Rollback: `kubectl apply -f /tmp/indicators-standard-node.before.yaml`. Do not disable the whole inode item; that also hides a real `nodefs.inodesFree` problem. + +```bash +kubectl -n cpaas-system get configmap indicators-standard-node -o yaml > /tmp/indicators-standard-node.before.yaml +kubectl -n cpaas-system edit configmap indicators-standard-node +``` + +Search for `node.disk.inode.free` and change its `query` value from: + +```text +node_filesystem_files_free{node_name=~\"{{.name}}\",device!~\"rootfs|tmpfs\"} +``` + +to: + +```text +node_filesystem_files_free{node_name=~\"{{.name}}\",device!~\"rootfs|tmpfs\",mountpoint!~\"/boot/efi\"} +``` + +Keep the escaped quotes as they are inside the ConfigMap. If the query on your cluster has a positive mount-point list (`mountpoint=~...`) that contains `/boot/efi`, remove `/boot/efi` from that list instead. Make the change on every cluster that reports the row. + +4. Collection refreshes about every 15 minutes. To apply it sooner, restart `courier` in `cpaas-system`, then re-run the inspection and confirm the row is gone. + +> **Warning:** Restarting `courier` drops inspection and cluster-list calls until the new pod is Ready. Precondition: the ConfigMap edit is saved. Rollback: none needed; the controller recreates the pod. + +```bash +kubectl -n cpaas-system get deploy,pod | grep courier +kubectl -n cpaas-system rollout restart deploy/courier +``` + +If `courier` is not a Deployment, delete the pod that `kubectl -n cpaas-system get pod | grep courier` prints, so that its controller recreates it. + +`indicators-standard-node` is a platform object. Re-check the query after a platform upgrade, and include the change in your pre-upgrade review of platform customizations. + +### Cause 3: inode or file-handle alert fires, but the node still has plenty of both + +Confirm the headroom before you treat the alert as noise: + +```bash +df -i +cat /proc/sys/fs/file-nr +cat /proc/sys/fs/file-max +``` + +`file-nr` prints allocated handles, unused handles, and the maximum. If the allocated number is far below the maximum and `IUse%` is far below 100%, the alert threshold, not the node, is the issue, and no handling is required. If the alert is too noisy for the environment, review the threshold of that alert rule. When `IUse%` really is high, use Cause 1. + +## Diagnostic Steps + +1. Print inode usage for every mount the alert could mean, including `/boot/efi`: + +```bash +df -i / /var /var/lib /cpaas /boot/efi +kubectl describe node | grep -E 'DiskPressure|inodesFree' +``` + +2. Classify: + +- `/` or `/var` with no free inodes, possibly `DiskPressure`: Cause 1. Rank directories with `du -x --inodes -d 1`. +- Only `/boot/efi` shows 0 free inodes, and the inspection reports remaining inodes as 0: Cause 2. +- An inode or file-handle alert while `df -i` and `/proc/sys/fs/file-nr` show plenty of headroom and the node has no pressure condition: Cause 3. + +A node that is `DiskPressure` because bytes, not inodes, ran out is a different problem; use `df -h` and `du -x -d 1` to find the directory that filled the mount. diff --git a/docs/en/solutions/TopoLVM_device_unavailable_or_PVC_unusable_because_the_backing_disk_failed_disappeared_or_was_decommissioned_on_ACP.md b/docs/en/solutions/TopoLVM_device_unavailable_or_PVC_unusable_because_the_backing_disk_failed_disappeared_or_was_decommissioned_on_ACP.md new file mode 100644 index 000000000..944ff474d --- /dev/null +++ b/docs/en/solutions/TopoLVM_device_unavailable_or_PVC_unusable_because_the_backing_disk_failed_disappeared_or_was_decommissioned_on_ACP.md @@ -0,0 +1,196 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# TopoLVM device unavailable or PVC unusable because the backing disk failed, disappeared, or was decommissioned on ACP + +## Issue + +On **Storage Management > Local Storage > topolvm > Storage Devices** a TopoLVM device (for example `/dev/sdc`) is listed as unavailable (`不可用`, "Unavailable"), and one of the following is reported: + +- After the servers were powered off and on again, TopoLVM on a node does not start, and the workloads whose PVCs live on that node (for example a registry or a Git service) do not start either. +- A disk on a TopoLVM node is damaged, and a PVC on it can no longer be recognized or mounted. The question is whether rebuilding the PVC helps. +- `discover-device` fails with `failed to list all devices: exit status 127` on some nodes, although `lsblk`, `blkid`, and the `lvm2` packages are installed. +- A disk that TopoLVM used has been taken offline for good, and its entries still show in TopoLVM. + +TopoLVM volumes are LVM logical volumes on the node's own disk. When that disk is gone, the logical volumes on it are gone too; TopoLVM does not keep a second copy. Recovering the data means restoring it from the application's own backup or replica. + +Operands run in `nativestor-system`; the cluster object is `topolvmclusters.topolvm.cybozu.com/topolvm`; the per-node lvmd settings are ConfigMap `lvmdconfig-`. + +Related articles: [TopoLVM does not offer a disk or topolvm-prepare-vg fails to create the volume group on ACP](./TopoLVM_does_not_offer_a_disk_or_topolvm-prepare-vg_fails_to_create_the_volume_group_on_ACP.md) (the disk is present but still carries old signatures, or a new disk has to be added), [TopoLVM lvmd exits with should have at least one device-class or Error: not found on ACP](./TopoLVM_lvmd_exits_with_should_have_at_least_one_device-class_or_Error_not_found_on_ACP.md) (the disk is present but the volume group named in `lvmd.yaml` is missing or named differently), [TopoLVM device class or storage devices show Unavailable or Unknown on ACP](./TopoLVM_device_class_or_storage_devices_show_Unavailable_or_Unknown_on_ACP.md). + +## Root Cause + +### Cause 1: the device is unavailable after a power cycle and lsblk no longer lists the disk + +`lsblk` on the node lists the other disks (the operating-system disk and its partitions) but not the disk that the TopoLVM device names. The disk dropped off the node after the power cycle, so the device class on that node has no volume group, TopoLVM on that node does not start, and the pods whose PVCs live there stay down. This is a hardware or virtualization-layer fault of the disk, and the disk may have to be replaced. + +### Cause 2: a damaged disk leaves a PVC on it unusable + +A disk on a TopoLVM node is damaged, and the PV and PVC on it can no longer be recognized or mounted. Rebuilding the PVC creates a new, empty volume; it does not recover what was on the damaged disk. + +### Cause 3: discover fails with exit status 127 and the GPT main header of the disk is corrupted + +`discover-device` logs `failed to list all devices: exit status 127` on some nodes while the device tools are installed, and `topolvm-controller` repeats `probe failed` with `csidrivers.storage.k8s.io "topolvm.cybozu.com" not found`. A GPT check of a disk on those nodes reports a damaged primary partition table: + +```text +Invalid CRC on main header data; loaded backup partition table. +Main and backup partition tables differ! +One or more CRCs don't match. You should repair the disk! +Main header: ERROR +Backup header: OK +Main partition table: ERROR +Backup partition table: OK +Invalid partition data! +``` + +The failure was resolved by repairing that disk and rebuilding TopoLVM. The same `exit status 127` is also the signature of a CPU without `x86-64-v2` support, see [TopoLVM operator or discover-device crashes with CPU does not support x86-64-v2 on ACP](./TopoLVM_operator_or_discover-device_crashes_with_CPU_does_not_support_x86-64-v2_on_ACP.md); check the CPU flags as well. + +### Cause 4: a disk taken offline for good still appears in TopolvmCluster and in the lvmd ConfigMap + +A disk was physically taken offline, but TopoLVM still lists it. The device is still in the node's `devices` list under `spec.storage.deviceClasses` of `TopolvmCluster`, and it is still an entry of `deviceStates` in `status.json` of ConfigMap `lvmdconfig-`, for example: + +```json +"deviceStates": [{"name":"/dev/vdb","state":"Online"},{"name":"/dev/vdc","state":"Online"}] +``` + +Those leftover entries keep the node and the device class abnormal on the Local Storage page. + +## Resolution + +### Cause 1: the device is unavailable after a power cycle and lsblk no longer lists the disk + +1. Confirm, from the node and from the kernel log, that the disk is gone rather than renamed: + +```bash +lsblk +dmesg -T | grep -i -E 'sd[a-z]|nvme|I/O error' | tail -50 +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm \ + -o jsonpath='{.status.nodeStorageState}' | jq . +``` + +2. Have the hardware or virtualization team check the disk and, if needed, replace it. +3. A new disk does not carry the old volume group or its logical volumes. Restore the PVCs that lived there from the application's backup or replicas. Remove the old device entries as in Cause 4, and add the new raw disk with **Add Storage Device** (see the related article on adding a disk). + +### Cause 2: a damaged disk leaves a PVC on it unusable + +Rebuild the PVC only when the data on it is not needed or can be restored. First record which PV and node the claim uses: + +```bash +kubectl get pvc -n -o jsonpath='{.spec.volumeName}{"\n"}' +kubectl get pv -o jsonpath='{.spec.csi.driver}{" "}{.spec.nodeAffinity}{"\n"}' +kubectl get pod -n -o wide | grep +kubectl get pvc -n -o yaml > /tmp/.before.yaml +``` + +> **Warning:** Impact: deleting the PVC and PV drops the claim and whatever data was still on that volume; the restarted pod gets a new, empty volume. Precondition: only when the disk under this PV is confirmed damaged and the data is not needed or is backed up elsewhere. Rollback: none; restore from the application's backup. + +```bash +kubectl delete pvc -n +kubectl delete pv +kubectl delete pod -n +kubectl get pvc -n +``` + +The new claim is provisioned on a node whose device class is healthy. If the pod's owner does not create its PVC from a template (a StatefulSet `volumeClaimTemplates` does), recreate the PVC from the saved manifest, without its `spec.volumeName`, before you restart the pod. + +### Cause 3: discover fails with exit status 127 and the GPT main header of the disk is corrupted + +1. Confirm with a read-only check. `sgdisk -v` (verify) and `gdisk -l` only read the disk: + +```bash +lsblk -f +sgdisk -v /dev/ +gdisk -l /dev/ +``` + +Look for `Invalid CRC on main header data`, `Main and backup partition tables differ!`, `Main header: ERROR`, or `Main partition table: ERROR`, with `Backup header: OK`. Do not use `sgdisk -e` as a check: it writes to the disk (it moves the backup GPT structures to the end of the disk and writes the partition table back). + +2. Repair the disk, choosing by what it holds: + - The disk is a TopoLVM data disk and holds nothing you need: return it to a raw disk by removing its signatures and partition table, as in Cause 1 of [TopoLVM does not offer a disk or topolvm-prepare-vg fails to create the volume group on ACP](./TopoLVM_does_not_offer_a_disk_or_topolvm-prepare-vg_fails_to_create_the_volume_group_on_ACP.md). + - The partitions are still needed: rebuild the main header from the intact backup header. Agree this with whoever owns the disk first. + +> **Warning:** Impact: `sgdisk -e` (and the recovery menu of `gdisk`) rewrites the GPT headers and partition table on `/dev/`; a wrong disk, or a backup table that is itself stale, writes a bad partition table and can make every partition on the disk unreachable. Precondition: only when this cause is confirmed with the read-only check above, the disk is the right one, the GPT has been saved with `sgdisk --backup`, and the disk owner has agreed. Rollback: `sgdisk --load-backup=/root/-gpt.bak /dev/` restores the saved headers and table. + +```bash +sgdisk --backup=/root/-gpt.bak /dev/ +sgdisk -e /dev/ +sgdisk -v /dev/ +``` + +3. Rebuild the TopoLVM cluster on the repaired disks afterwards, and confirm `discover-device` no longer logs `exit status 127`. + +### Cause 4: a disk taken offline for good still appears in TopolvmCluster and in the lvmd ConfigMap + +Volumes that lived on that disk are already lost. The supported procedure is "Scenario 1: Remove a volume group device from a device-class volume group" in the product documentation topic "Remove Disks, Device-Class Volume Groups, or Nodes from LVM Storage". It applies when the node is still reachable, the device class provision type is `Thick`, and the volume group still has at least one healthy disk. If the failed disk was the only disk of that volume group, so `vgs` no longer lists the group, use Scenario 2 of the same topic (remove the device-class volume group); if the node itself is retired, use Scenario 3 (see [Removing a TopoLVM storage node or releasing its disks on ACP](./Removing_a_TopoLVM_storage_node_or_releasing_its_disks_on_ACP.md)). + +1. Save both objects and identify the logical volumes that used the failed disk: + +```bash +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm -o yaml > /tmp/topolvmcluster.before.yaml +kubectl -n nativestor-system get cm lvmdconfig- -o yaml > /tmp/lvmdconfig.before.yaml +kubectl -n nativestor-system scale deployment topolvm-operator --replicas=0 +vgs +lvs -a -o +devices 2>/dev/null | egrep "/dev/|unknown" | awk '{print $1}' +``` + +2. Each name the `lvs` line prints is a logical volume on the failed disk; its PV has that name as `spec.csi.volumeHandle`: + +```bash +kubectl get pv -o json | jq -r --arg HANDLE ' + .items[] | select(.spec.csi.volumeHandle == $HANDLE) + | [.metadata.name, .spec.claimRef.namespace, .spec.claimRef.name] | @tsv' +``` + +3. Stop the workloads that use those PVCs. + +> **Warning:** Impact: these commands delete the PVC, the PV, the `logicalvolumes.topolvm.cybozu.com` object, and the logical volume, and drop the missing disk from the volume group. Data on those volumes is not recoverable. Precondition: only when the disk is confirmed gone for good, the workloads are stopped, and the names come from the `lvs` and `jq` output above. Rollback: none. + +```bash +kubectl delete pvc -n +kubectl delete pv +kubectl delete logicalvolumes.topolvm.cybozu.com +lvremove / +vgreduce --removemissing +``` + +4. Remove the device entries, then start the operator. + +> **Warning:** Impact: while `topolvm-operator` is at 0 replicas no device class is reconciled on any node, and removing the entries drops TopoLVM's record of that disk. Existing mounts on other disks keep working. Precondition: only when the disk is physically offline for good, both objects are saved, and you remove only the failed device on the target node. Rollback: while the operator is still at 0, apply the saved `TopolvmCluster` and ConfigMap. + +```bash +kubectl -n nativestor-system edit topolvmclusters.topolvm.cybozu.com topolvm +kubectl -n nativestor-system edit cm lvmdconfig- +kubectl -n nativestor-system scale deployment topolvm-operator --replicas=1 +``` + +In `TopolvmCluster`, delete the failed device from the target node's `devices` list. In `status.json` of `lvmdconfig-`, delete that device's element from `deviceStates` and keep the others, for example leave only `{"name":"/dev/vdc","state":"Online"}` when `/dev/vdb` is the device that is gone. + +5. Confirm the device is gone from status. Every node then shows Normal (`正常`) on the Local Storage view and the device class shows Available (`可用`): + +```bash +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm -o jsonpath='{.status.nodeStorageState}' | jq +``` + +## Diagnostic Steps + +```bash +lsblk +kubectl -n nativestor-system get pods -o wide +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm -o yaml +kubectl -n nativestor-system get cm -o name | grep -i lvm +kubectl logs -n nativestor-system --tail=50 +``` + +- A device on the Storage Devices tab that `lsblk` on its node no longer shows after a power cycle: Cause 1. +- A PVC that cannot be mounted because its disk is damaged (`dmesg` shows I/O errors for that disk): Cause 2. +- `failed to list all devices: exit status 127`, CPU flags complete, and `Main header: ERROR` from `sgdisk -v` or `gdisk -l`: Cause 3. +- A disk that is gone for good while `TopolvmCluster` `devices` and `deviceStates` in `lvmdconfig-` still list it: Cause 4. +- Data inside a container is missing while the disk is healthy: check that the PVC is still bound to the original PV (`kubectl get pvc -n -o jsonpath='{.spec.volumeName}'`) and that `lvs -a` on the PV's node still lists the logical volume named by the PV's `spec.csi.volumeHandle`. A recreated PVC or PV is a new, empty volume. For what deletes TopoLVM data, see [Removing a TopoLVM storage node or releasing its disks on ACP](./Removing_a_TopoLVM_storage_node_or_releasing_its_disks_on_ACP.md). diff --git a/docs/en/solutions/TopoLVM_does_not_offer_a_disk_or_topolvm-prepare-vg_fails_to_create_the_volume_group_on_ACP.md b/docs/en/solutions/TopoLVM_does_not_offer_a_disk_or_topolvm-prepare-vg_fails_to_create_the_volume_group_on_ACP.md new file mode 100644 index 000000000..ba8e27289 --- /dev/null +++ b/docs/en/solutions/TopoLVM_does_not_offer_a_disk_or_topolvm-prepare-vg_fails_to_create_the_volume_group_on_ACP.md @@ -0,0 +1,196 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.16.x + - 3.18.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# TopoLVM does not offer a disk or topolvm-prepare-vg fails to create the volume group on ACP + +## Issue + +A data disk cannot be used by TopoLVM on Alauda Container Platform: + +- TopoLVM deployment reports an error on a data disk, TopoLVM cannot be created because the disk shows as unavailable, or a bare disk is missing from the addable device list. +- `pvcreate` reports `Device /dev/ excluded by a filter.`, or a device class reports `disk may has filesystem or is not raw disk please check`. +- The `topolvm-prepare-vg` log shows `Cannot use /dev/: device is partitioned` followed by `create vg - retry failed err:exit status 5`. +- A newly added disk stays "device unavailable" while the TopoLVM pod for that node keeps being recreated. +- A device class stays Offline or shows `0 GiB`, and `topolvm-prepare-vg` logs `device:/dev/mapper/mpatha is not available`. +- Adding a disk for expansion finishes with no extra capacity. + +The operands live in `nativestor-system`. The cluster object is `topolvmclusters.topolvm.cybozu.com/topolvm` (`TopolvmCluster`). Preparation is the Job `topolvm-prepare-vg`; discovery is the `discover-device` pod. + +TopoLVM needs a whole disk of type `disk`. A partition such as `/dev/sdb1` is rejected. TopoLVM does not require a particular RAID level: a hardware RAID volume (for example RAID 5) qualifies when the operating system sees it as a whole disk with no partitions, no filesystem, and no RAID member signature. `/dev/sda` is often the operating-system disk; never wipe a disk because its name matches an example. + +Related articles: [Former TopoLVM data disk stays unavailable because old LVM and thin-pool mappings remain on ACP](./Former_TopoLVM_data_disk_stays_unavailable_because_old_LVM_and_thin-pool_mappings_remain_on_ACP.md), [TopoLVM device class capacity does not change after the underlying disk is expanded on ACP](./TopoLVM_device_class_capacity_does_not_change_after_the_underlying_disk_is_expanded_on_ACP.md) (a disk TopoLVM already uses was grown in place), [TopoLVM lvmd exits with should have at least one device-class or Error: not found on ACP](./TopoLVM_lvmd_exits_with_should_have_at_least_one_device-class_or_Error_not_found_on_ACP.md). + +## Root Cause + +### Cause 1: Device excluded by a filter, or the disk still has a filesystem, partition, or LVM signature + +The disk still carries an old signature: `wipefs -n` or `lsblk -f` shows `LVM2_member`, a partition table, a filesystem, or a foreign member such as `jmicron_raid_member`. TopoLVM does not offer such a disk, `pvcreate` refuses it, and pool expansion does nothing until those signatures are removed. A partition table on a whole disk shows up as the `pvcreate` that prepare runs on the host failing: + +```text +prepare-vg: start provision with node status +nsenter -u -i -n -m -p -t 1 lvm pvcreate /dev/nvme0n1 +Cannot use /dev/nvme0n1: device is partitioned +prepare-vg: create vg - retry failed err:exit status 5 +``` + +The message `disk may has filesystem or is not raw disk please check` is also written when the device is already a physical volume of a TopoLVM volume group whose name differs from the one TopoLVM recorded. Then `pvs` shows the device in a VG named `-` and `lvs` shows live logical volumes. That is not a dirty disk, and wiping it destroys the PVC data (see the related article on lvmd). Run `pvs` and `lvs` before any wipe. + +If `wipefs -n` is empty and `pvcreate` still prints `excluded by a filter`, the LVM devices file is the next check, not a format. + +### Cause 2: the disk is not in TopolvmCluster.spec.storage.deviceClasses + +`TopolvmCluster` lists other devices on that `nodeName` and not the target path, and the `discover-device` logs have no line for it. A disk that was wiped but never added is not used. The UI **Add Storage Device** action is the normal path once an initial device class exists. On a cluster whose Local Storage page shows a healthy TopoLVM install but refuses the first node and disk, writing the node and device into `TopolvmCluster` by hand makes later UI adds work. + +### Cause 3: prepare-vg: device:/dev/mapper/mpatha is not available + +The prepare log shows `sys: device:dm-12 is available` and, on the next line, `prepare-vg: device:/dev/mapper/mpatha is not available`. Inventory and prepare disagree about one multipath device (`/dev/mapper/mpath*`, type `mpath`), for example an iSCSI LUN. The device class becomes Offline with `0 GiB`, and the prepare Job can show Succeeded while a new Job is created about every 30 seconds. `wipefs -n` is empty; this is not a dirty disk. The TopoLVM operator of ACP 3.16 (v3.16.6) does not support multipath devices. `/dev/dm-N` is not a stable name to configure instead. + +### Cause 4: newly added node disk is blank, the device stays unavailable and the pod is recreated repeatedly + +A disk was added to a node for TopoLVM. The device stays unavailable and the TopoLVM pod for that node keeps being recreated. `lsblk -f`, `blkid`, and `wipefs -n` show no filesystem, partition table, or LVM signature: the disk is blank, not dirty. Formatting it on the host made the device available and stopped the pod recreation. The exact filesystem type that was written is not known. + +A different shape with the same symptom: `topolvm-prepare-vg` and `topolvm-node` pods rebuild forever while `pvs`, `vgs`, and `lvs` already show the complete PV, VG, and LV and the logs are quiet. That disk is not blank, so this cause does not apply. + +## Resolution + +Identify the device with the commands in Diagnostic Steps before any wipe. Stop if the path is mounted, is the parent of `/`, `/boot`, or `/var`, or is already a PV in `pvs` that an application still uses. + +### Cause 1: Device excluded by a filter, or the disk still has a filesystem, partition, or LVM signature + +> **Warning:** Impact: `wipefs -af` deletes every filesystem, RAID, LVM, and partition signature on that path, and `sgdisk --zap-all` destroys the GPT and the protective MBR; the data is not recoverable from the node. Precondition: only when `wipefs -n` lists at least one leftover signature; `lsblk`, `findmnt`, `blkid`, and `pvs` show the disk is unused; it is not the disk that holds `/`; and no bound TopoLVM PV points at it. If `wipefs -n` prints nothing, there is nothing to wipe: go to the devices-file check below, or to Cause 4. Rollback: none; restore only from a backup taken before the wipe. + +```bash +lsblk /dev/ +wipefs -af /dev/ +sgdisk --zap-all /dev/ +wipefs -n /dev/ +lsblk /dev/ +``` + +The last `lsblk` must show the disk as `disk` with no child partitions and no mountpoint, and `wipefs -n` must print nothing. If `sgdisk` is not installed and `wipefs -n` is already empty, the `wipefs -af` step alone is enough. Then add the device (Cause 2). If `topolvm-prepare-vg` already failed on the old signature, delete that Job so the operator starts a new one; deleting the Job does not delete a volume group: + +```bash +kubectl delete job -n nativestor-system +``` + +If `pvcreate` still prints `excluded by a filter` after the wipe, the LVM devices file may be omitting the disk: + +```bash +lvmconfig devices/use_devicesfile +lvmdevices +``` + +> **Warning:** `lvmdevices --adddev` allows LVM to claim `/dev/`. Precondition: `use_devicesfile` is 1 and this is the TopoLVM data disk, not a disk you meant to keep out of LVM. Rollback: `lvmdevices --deldev /dev/`. + +```bash +lvmdevices --adddev /dev/ +``` + +Do not change the global `filter` in `/etc/lvm/lvm.conf` for this. If the disk still shows old TopoLVM thin-pool mappings in `dmsetup ls`, `wipefs` cannot remove them; see the related article on leftover mappings. + +### Cause 2: the disk is not in TopolvmCluster.spec.storage.deviceClasses + +Prefer **Storage Management > Local Storage > Storage Devices > Add Storage Device** and select the clean whole disk. If the page will not offer the node, edit the object. + +> **Warning:** A wrong `devices[].name` makes prepare-vg operate on that path, and `useAllDevices: true` can pull in the OS disk. Precondition: Cause 1 is clear, you saved the YAML, and the path is the intended whole disk. Rollback: remove the device entry you added and save again. + +```bash +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm -o yaml > /tmp/topolvmcluster.before.yaml +kubectl -n nativestor-system edit topolvmclusters.topolvm.cybozu.com topolvm +``` + +Add the node and device under `spec.storage.deviceClasses` (names are examples; keep `useAllDevices`, `useAllNodes`, and `useLoop` false unless the cluster already sets them): + +```yaml +spec: + storage: + useAllDevices: false + useAllNodes: false + useLoop: false + deviceClasses: + - nodeName: + classes: + - className: hdd + default: true + devices: + - name: /dev/ + type: disk +``` + +`type` must be `disk`. Wait until `status.nodeStorageState` shows the class Ready and the UI capacity includes the new size; a Job phase of Succeeded is not that check. After a device is added the capacity can take about 10 minutes to appear, for each extra raw disk. + +### Cause 3: prepare-vg: device:/dev/mapper/mpatha is not available + +1. Do not wipe the LUN and do not switch the class to `/dev/dm-N`. +2. To stop the 30-second Job recreation, remove the `mpath` device from `spec.storage.deviceClasses` with the same edit as Cause 2. That leaves the class without this LUN; it does not delete data on the LUN. +3. Give TopoLVM a non-multipath whole disk, or upgrade to an ACP version whose TopoLVM release notes list multipath support and add the LUN with the path that version's device list offers. + +### Cause 4: newly added node disk is blank, the device stays unavailable and the pod is recreated repeatedly + +Decide by what the disk holds, not by the error alone: + +- `wipefs -n` or `lsblk -f` lists old signatures: Cause 1. +- `wipefs -n` prints nothing and `pvcreate` prints `excluded by a filter`: neither wipe nor format; check the LVM devices file as in Cause 1. +- `wipefs -n` and `blkid` print nothing, there is no filter error, and the disk is the new, empty data disk: this cause. +- `pvs`, `vgs`, and `lvs` already show the complete TopoLVM PV, VG, and LV: neither wipe nor format here. Removing the node from `TopolvmCluster`, wiping the disk, and enrolling the node again is the recovery for that shape, and only when `lvs` shows no logical volume that backs a bound PV (see [Removing a TopoLVM storage node or releasing its disks on ACP](./Removing_a_TopoLVM_storage_node_or_releasing_its_disks_on_ACP.md)). + +Take the format from a disk that already works in the same device class on another node: + +```bash +# on a node where the same device class is Ready +lsblk -f /dev/ +blkid /dev/ || true +# on the new node +lsblk -o NAME,PATH,TYPE,SIZE,FSTYPE,MOUNTPOINT,PKNAME /dev/ +wipefs -n /dev/ || true +findmnt -n -o SOURCE / +``` + +> **Warning:** Impact: formatting writes a new signature to the whole disk and destroys anything on it. Precondition: the device is the newly added data disk, not the disk that holds `/`, `/boot`, or `/var`, not mounted, not in `pvs`, and `wipefs -n` shows no signatures. Rollback: none. If you cannot tell which format the working disk uses, do not guess; collect the output above and contact support. + +After the disk matches the working disk, delete a failed `topolvm-prepare-vg` Job so the operator creates a new one, and confirm the class is Ready in `status.nodeStorageState` and the pod on that node stops being recreated. + +## Diagnostic Steps + +1. Confirm the operands are up, then read device-class status. A missing `TopolvmCluster` is a controller problem, not a reason to wipe. + +```bash +kubectl get deploy,ds,job,pod -n nativestor-system -o wide +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm \ + -o jsonpath='{.status.nodeStorageState}' | jq . +``` + +2. On the node, prove identity and ownership: + +```bash +lsblk -o NAME,KNAME,PATH,TYPE,SIZE,FSTYPE,MOUNTPOINT,PKNAME /dev/ +blkid /dev/ || true +wipefs -n /dev/ || true +findmnt -n -o SOURCE / +pvs; vgs; lvs +multipath -ll +``` + +3. Read the prepare log: + +```bash +kubectl logs -n nativestor-system job/ --all-containers --tail=200 +``` + +- `excluded by a filter`, `device is partitioned`, or `disk may has filesystem` with a signature in `wipefs -n` and no TopoLVM VG in `pvs`: Cause 1. +- `disk may has filesystem or is not raw disk please check` while `pvs` shows the device in a TopoLVM `-` VG with live LVs: a VG-name mismatch, see the related lvmd article. +- Clean disk not listed in `TopolvmCluster`: Cause 2. +- `device:/dev/mapper/mpath* is not available` and a new prepare Job every 30 seconds: Cause 3. +- Blank disk, no filter error, pod recreated repeatedly: Cause 4. +- A pod mount error `target device is already formatted with different filesystem` is a signature on the logical volume, not on the disk; see [TopoLVM PVC rejects an access mode, fails to mount, or cannot be scheduled on ACP](./TopoLVM_PVC_rejects_an_access_mode_fails_to_mount_or_cannot_be_scheduled_on_ACP.md), Cause 2. +- In-use disk grown in place, so the Storage Device size is larger than `pvs`: see the related capacity article. From 4b15be30ce2f04bddf8caa51e1600c169a83af7c Mon Sep 17 00:00:00 2001 From: Komh Date: Wed, 18 Nov 2026 17:41:37 +0800 Subject: [PATCH 22/25] docs(solutions): add 12 and update 0 troubleshooting articles --- ..._no_data_or_Deployment_GPU_chart_on_ACP.md | 143 +++++++++ ...w_broker_Pod_is_blocked_by_quota_on_ACP.md | 159 ++++++++++ ...ization_NodePort_and_reverse_DNS_on_ACP.md | 283 ++++++++++++++++++ ...or_the_instance_stays_processing_on_ACP.md | 154 ++++++++++ ...or_the_cluster_operator_restarts_on_ACP.md | 174 +++++++++++ ...or_ZooKeeper_mode_blocks_upgrade_on_ACP.md | 191 ++++++++++++ ...pported_or_uppercase_topic_names_on_ACP.md | 187 ++++++++++++ ...unpublished_or_failing_to_deploy_on_ACP.md | 175 +++++++++++ ...ng_or_cni_plugin_not_initialized_on_ACP.md | 143 +++++++++ ...n_Input_output_error_or_df_hangs_on_ACP.md | 173 +++++++++++ ...ule_or_a_read-only_cpuset_cgroup_on_ACP.md | 136 +++++++++ ...mmit_Pod_count_and_node_capacity_on_ACP.md | 202 +++++++++++++ 12 files changed, 2120 insertions(+) create mode 100644 docs/en/solutions/HAMi_GPU_usage_by_API_or_PromQL_copied_vGPU_dashboard_shows_no_data_or_Deployment_GPU_chart_on_ACP.md create mode 100644 docs/en/solutions/Kafka_broker_memory_looks_high_or_a_memory_change_or_new_broker_Pod_is_blocked_by_quota_on_ACP.md create mode 100644 docs/en/solutions/Kafka_client_connection_failures_timeouts_TLS_certificates_authorization_NodePort_and_reverse_DNS_on_ACP.md create mode 100644 docs/en/solutions/Kafka_config_change_does_not_roll_brokers_a_broker_restart_hangs_or_the_instance_stays_processing_on_ACP.md create mode 100644 docs/en/solutions/Kafka_instance_creation_times_out_on_the_entity-operator_shows_failed_or_the_cluster_operator_restarts_on_ACP.md create mode 100644 docs/en/solutions/Kafka_operator_mismatch_wrong_create_page_version_Kafka_2x_instance_skipped_or_ZooKeeper_mode_blocks_upgrade_on_ACP.md create mode 100644 docs/en/solutions/Kafka_user_secret_missing_user_stuck_in_Creating_ACL_not_supported_or_uppercase_topic_names_on_ACP.md create mode 100644 docs/en/solutions/No_Kafka_in_Data_Services_or_Alauda_Build_of_Kafka_2_unpublished_or_failing_to_deploy_on_ACP.md create mode 100644 docs/en/solutions/Node_NotReady_with_containerd_not_running_or_cni_plugin_not_initialized_on_ACP.md create mode 100644 docs/en/solutions/Node_runtime_or_kubelet_directories_return_Input_output_error_or_df_hangs_on_ACP.md create mode 100644 docs/en/solutions/Node_unstable_from_a_GPU_NVRM_log_flood_a_faulty_memory_module_or_a_read-only_cpuset_cgroup_on_ACP.md create mode 100644 docs/en/solutions/Project_quota_and_namespace_ResourceQuota_aggregate_limits_overcommit_Pod_count_and_node_capacity_on_ACP.md diff --git a/docs/en/solutions/HAMi_GPU_usage_by_API_or_PromQL_copied_vGPU_dashboard_shows_no_data_or_Deployment_GPU_chart_on_ACP.md b/docs/en/solutions/HAMi_GPU_usage_by_API_or_PromQL_copied_vGPU_dashboard_shows_no_data_or_Deployment_GPU_chart_on_ACP.md new file mode 100644 index 000000000..7f1fd5107 --- /dev/null +++ b/docs/en/solutions/HAMi_GPU_usage_by_API_or_PromQL_copied_vGPU_dashboard_shows_no_data_or_Deployment_GPU_chart_on_ACP.md @@ -0,0 +1,143 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# HAMi GPU usage by API or PromQL, copied vGPU dashboard shows no data, or Deployment GPU chart on ACP + +## Issue + +GPU usage data from HAMi is needed outside HAMi WebUI: + +- Code has to start a Pod, delete a Pod, and query the GPU card usage of a cluster, and no HAMi or GPU usage API can be found. +- The `hami-vgpu-metrics-dashboard` that HAMi deploys was copied into the Container Platform as a custom namespace-level monitoring dashboard, and it shows no data. Panels such as GPU Total, XID Error, and GPU temp(DCGM) fail to load, and gauges such as Average GPU Temp and GPU power usage show "No data" (`无数据`). The query behind a panel returns HTTP `500` with the header `x-alb-err-reason: BackendError : read 463 byte data from backend` and a message such as `DCGM_FI_DEV_POWER_USAGE should has selector namespace=`. +- A user wants a GPU usage chart for their own Deployment. The `监控` (Monitoring) tab, under `自定义面板` (Custom Dashboard), shows `当前无数据,请 添加图表` (no data, please add a chart), and HAMi metrics are not in the built-in metric list. +- GPU memory per namespace queried from VictoriaMetrics looks much higher than what `nvidia-smi` shows on the node, or it is unclear which HAMi metric gives allocated versus used GPU memory and cores. + +A copied dashboard that fails for reasons other than the namespace selector is covered in [Custom or imported monitoring dashboards show parse errors, unsupported chart types, or no data on ACP](Custom_or_imported_monitoring_dashboards_show_parse_errors_unsupported_chart_types_or_no_data_on_ACP.md). + +## Root Cause + +### Cause 1: There is no dedicated platform API for HAMi GPU usage + +An exporter collects HAMi and GPU metrics into the monitoring storage (Prometheus or VictoriaMetrics), and they can only be read with PromQL. The platform monitoring metrics API takes a PromQL expression, and it also serves CPU and memory for a group of Pods. Creating and deleting Pods uses the standard Kubernetes API through the platform. + +The expressions already exist in the platform dashboards: in `HAMi/hami-vgpu-metrics-dashboard`, the panel menu entry "Query expression" (`查询表达式`) shows each panel's query, for example the `nodeGPUOverview` GPU count, the `vGPUCorePercentage` per-card usage, and the `nodeGPUList(DCGM)` per-node distribution. + +When the GPU cluster's monitoring plugin, Alauda Container Platform Monitoring for VictoriaMetrics, is set to "Deploy agent only: Yes" (`仅部署代理: 是`), the GPU cluster runs only the agent and the data is stored on the storage cluster named in that plugin. Queries must go to that storage cluster, and the `vmcluster` label in the expressions carries the GPU cluster name. + +### Cause 2: A copied hami-vgpu-metrics-dashboard shows no data because card-level metrics have no namespace label + +In the Container Platform view, a dashboard expression must be restricted to a namespace, or it is rejected with ` should has selector namespace=` (for example for `DCGM_FI_DEV_POWER_USAGE` and for `sum by (deviceuuid, nodename) (vGPUCorePercentage{nodename=~""})`). The copy was made visible there with the labels `cpaas.io/dashboard.folder: container-platform` and `cpaas.io/dashboard.tag.overview: "true"`. The card-level and node-level series in that dashboard, the DCGM metrics and HAMi series such as `vGPUCorePercentage`, carry no `namespace` label, so the restriction can never match. Adding labels to the dashboard or to the queries does not help. This dashboard cannot be migrated into the Container Platform view; HAMi WebUI remains the place for vGPU monitoring. Per-container HAMi series are different (Cause 3). + +### Cause 3: A Deployment GPU chart needs a custom chart, and HAMi metrics need the native PromQL mode + +GPU usage of a workload is shown by adding a chart to the Deployment's `自定义面板` (Custom Dashboard). The built-in metrics (`内置指标`) cover physical GPUs only: + +- `workload.gpu.utilization`: physical GPU utilization of the workload +- `workload.gpu.memory.utilization`: physical GPU memory utilization of the workload +- `workload.pod.gpu.memory.utilization`: physical GPU memory utilization of the workload's Pods + +HAMi metrics are not built-in metrics. The platform monitoring dashboards have per-container HAMi panels, such as `HAMi-Memory desc of container`, `HAMi-GPU util desc of container`, `HAMi-Host memory usage`, and `HAMi-Host core util`. Their expressions carry `namespace` and `podname` labels and can be reused in a Deployment chart through the `原生` (native) PromQL method. + +### Cause 4: Allocated and used GPU are different HAMi metrics + +HAMi exports what the scheduler allocated separately from what is actually used: + +| Metric | Meaning | +|---|---| +| `hami_vgpu_memory_allocated_bytes` | GPU memory allocated by the scheduler | +| `hami_vgpu_memory_used_byte` | GPU memory actually used | +| `hami_host_gpu_utilization_ratio` | Actual GPU core utilization | +| `hami_vgpu_core_allocated_ratio` | GPU cores allocated | + +An allocation-based figure is not the same as the memory a process holds, so it can be much higher than what `nvidia-smi` lists. Compare `nvidia-smi` with the used metric, not the allocated one. + +## Resolution + +### Cause 1: There is no dedicated platform API for HAMi GPU usage + +1. Find the storage cluster. Open the cluster plugin Alauda Container Platform Monitoring for VictoriaMetrics of the GPU cluster. If "Deploy agent only" is Yes, use the storage cluster named there; otherwise use the GPU cluster itself. +2. Copy the expression from a panel of `HAMi/hami-vgpu-metrics-dashboard` ("Query expression"). Expressions have this shape: + + ```text + count(DCGM_FI_DEV_DEC_UTIL{UUID=~"(|)",node_name=~"(|)",vmcluster=""}) + sum by (deviceuuid, nodename) (vGPUCorePercentage{nodename=~"(|)",vmcluster=""}) + ``` + +3. Create a token in the user profile under **API Tokens** > **Add API Token** (`API 令牌` > `添加 API 令牌`), then call the metrics API of the storage cluster. A successful call returns JSON with `query` and `result`. + + ```bash + TOKEN='' + curl -k -H "Authorization: Bearer ${TOKEN}" --get \ + --data-urlencode 'expr=count(DCGM_FI_DEV_DEC_UTIL{vmcluster=""})' \ + "https:///platform/monitoring.alauda.io/v1beta1/clusters//metrics" + ``` + +To try expressions interactively, open vmui of the storage cluster at `https:///clusters//vmselect/vmui/`. In the vmui settings, remove `/prometheus` from the server URL and apply, or queries do not run. + +For Pod-group CPU and memory, copy the expressions from the matching existing dashboards in the same way. + +Pods are created with the standard Kubernetes `POST` to `https:///kubernetes//api/v1/namespaces//pods` (Pod manifest as body) and deleted with `DELETE` on `.../pods/`. + +> **Warning:** The `DELETE` call removes the Pod immediately. A Pod owned by a Deployment or another controller is recreated; a bare Pod is gone. Save the manifest of a bare Pod first; roll back by creating it again from that manifest. + +```bash +curl -k -X DELETE -H "Authorization: Bearer ${TOKEN}" \ + "https:///kubernetes//api/v1/namespaces//pods/" +``` + +### Cause 2: A copied hami-vgpu-metrics-dashboard shows no data because card-level metrics have no namespace label + +Do not migrate this dashboard into the Container Platform view; use HAMi WebUI for vGPU monitoring. To confirm, run a panel expression without the namespace filter in vmui or through the metrics API and list the labels on the result: no `namespace` label confirms this cause. For a per-workload view, use Cause 3. + +### Cause 3: A Deployment GPU chart needs a custom chart, and HAMi metrics need the native PromQL mode + +1. Open the Deployment, go to `监控` (Monitoring), and switch to `自定义面板` (Custom Dashboard). +2. Click `添加图表` (Add Chart). For physical GPU usage, choose a built-in metric: `workload.gpu.utilization`, `workload.gpu.memory.utilization`, or `workload.pod.gpu.memory.utilization`. +3. For HAMi vGPU usage, copy the expression of a per-container HAMi panel ("Query expression"), add the chart with the `原生` (native) method, and paste the expression restricted to the workload's namespace: + + ```text + sum by (podname) (Device_memory_desc_of_container{namespace=""}) + ``` + + This draws one series per Pod of the namespace. To show one workload only, add `podname=~"-.*"`. If a pasted expression returns `should has selector namespace=`, it is a card-level series (Cause 2). + +### Cause 4: Allocated and used GPU are different HAMi metrics + +Query the metric that matches the question in vmui of the storage cluster (or through the metrics API): + +```text +sum by (namespace) (hami_vgpu_memory_allocated_bytes) / 1024^3 +sum by (namespace) (hami_vgpu_memory_used_byte) / 1024^3 +hami_host_gpu_utilization_ratio +hami_vgpu_core_allocated_ratio +``` + +If an aggregated expression returns `[]`, run the bare metric (for example `hami_vgpu_memory_allocated_bytes`) first and read the labels it actually carries, then aggregate by a label that exists. For a trend instead of the current value, use the vmui **Graph** view with a time range such as the last 30 minutes; the **Table** view shows instant values. + +## Diagnostic Steps + +1. Confirm which cluster stores the monitoring data: + + ```bash + kubectl get pods -A | grep -E 'vmselect|vmagent|vminsert|vmstorage' + ``` + + On the GPU cluster, `vmagent` without `vmselect` matches agent-only mode: query the storage cluster. + +2. Test an expression through the metrics API. A JSON body with `result` means the expression and storage cluster are right. + +3. For a copied dashboard with no data, open the browser developer tools on a failing panel. HTTP `500`, `x-alb-err-reason: BackendError`, and `should has selector namespace=` in the body: Cause 2. + +4. Run the panel expression in vmui and look at the result labels. No `namespace` label on a HAMi or DCGM series confirms that the Container Platform view cannot filter it. A per-container series with `namespace` and `podname` labels (for example `Device_memory_desc_of_container`) can be charted as in Cause 3. + +5. When comparing with `nvidia-smi`, run it on the GPU node itself. Run from the platform node terminal after `chroot /host`, `nvidia-smi` can fail with `Failed to initialize NVML: Unknown Error` even though it works when logged in to the node directly; that failure says nothing about the metrics. diff --git a/docs/en/solutions/Kafka_broker_memory_looks_high_or_a_memory_change_or_new_broker_Pod_is_blocked_by_quota_on_ACP.md b/docs/en/solutions/Kafka_broker_memory_looks_high_or_a_memory_change_or_new_broker_Pod_is_blocked_by_quota_on_ACP.md new file mode 100644 index 000000000..4f799919c --- /dev/null +++ b/docs/en/solutions/Kafka_broker_memory_looks_high_or_a_memory_change_or_new_broker_Pod_is_blocked_by_quota_on_ACP.md @@ -0,0 +1,159 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda Application Services +ProductsVersion: + - "3.12" + - "3.14" + - "3.16" + - "3.18" + - "4.0" + - "4.1" + - "4.2" +--- + +# Kafka broker memory looks high, or a memory change or new broker Pod is blocked by quota on ACP + +## Issue + +On a running Data Services Kafka instance: + +- An inspection reports that a Kafka broker pod uses more than 80 percent of its memory, and someone asks whether that is a fault. +- Kafka memory usage climbs to about 90 percent no matter how much memory it is given. +- Updating the node spec (memory) of the instance fails, and rolling the change back fails too. The `RdsKafka` events show `ErrorKafkaStatus: Error while waiting for restarted pod -kafka-0 to become ready`, and the StatefulSet reports: + + ```text + pods "-kafka-0" is forbidden: exceeded quota: default, requested: limits.memory=8Gi,requests.memory=8Gi, used: limits.memory=28800Mi,requests.memory=28800Mi, limited: limits.memory=32Gi,requests.memory=32Gi + ``` + +- The instance shows abnormal, its `Kafka` resource is `NotReady` with `Exceeded timeout of 300000ms while waiting for Pods resource -kafka-0 in namespace to be ready`, and clients log `Timed out waiting for a node assignment. Call: fetchMetadata`. + +Related articles: [Kafka config change does not roll brokers, a broker restart hangs, or the instance stays processing on ACP](Kafka_config_change_does_not_roll_brokers_a_broker_restart_hangs_or_the_instance_stays_processing_on_ACP.md) covers rollouts and restarts that stall for reasons other than quota. Create-time failures are covered in [Kafka instance creation times out on the entity-operator, shows failed, or the cluster operator restarts on ACP](Kafka_instance_creation_times_out_on_the_entity-operator_shows_failed_or_the_cluster_operator_restarts_on_ACP.md). + +## Root Cause + +### Cause 1: Inspection flags broker memory above 80 percent + +A routine inspection reports broker memory above 80 percent of the limit (for example 87 percent). The pod is not restarting and is not `OOMKilled`. That level is normal for Kafka, not a fault, and no leak is involved. If the inspection threshold must be met, the broker memory limit is raised. + +### Cause 2: Memory usage reaches about 90 percent whatever the limit is (page cache) + +After the memory is expanded, usage again climbs to about 90 percent, with no message backlog behind it. This is expected: Kafka relies on the operating system page cache for throughput, and that cache grows into whatever memory the container is given, so the percentage stays high after every increase. No action is needed as long as the pods are not restarting because of OOM. + +### Cause 3: Spec update and rollback both fail with exceeded quota + +A memory change makes the operator stop a broker (`Killing: Stopping container kafka`) and recreate it with the new request. The StatefulSet `-kafka` then reports `FailedCreate` with the quota error, because the namespace `ResourceQuota` `default` is nearly full: in the example, 28800Mi of 32Gi is used by the other pods, so neither the new 8Gi request nor the old 4Gi request (more than the 3968Mi left) can be admitted for the pod being recreated. That is why the rollback fails the same way. The project quota on that cluster can also be fully allocated, which prevents raising the namespace quota. + +### Cause 4: The namespace LimitRange and ResourceQuota leave the broker pods without resources + +The cluster operator log repeats `Exceeded timeout of 300000ms while waiting for Pods resource -kafka- ... to be ready`, `createOrUpdate failed io.strimzi.operator.common.TimeoutException`, and `Failed to reconcile`. Clients log `Error connecting to node -kafka-2.-kafka-brokers..svc:9091` and `Timed out waiting for a node assignment. Call: fetchMetadata`. The broker pods of the Kafka `StrimziPodSet` did not come up because the namespace container limits (`LimitRange`) and `ResourceQuota` did not leave them enough resources to start (for example Kafka 3.8.0 in ZooKeeper mode with 3 brokers at CPU `500m` and memory `10Gi`, limits equal to requests). + +## Resolution + +### Cause 1: Inspection flags broker memory above 80 percent + +```bash +kubectl -n top pod -kafka-0 --containers +kubectl -n describe pod -kafka-0 | grep -E 'Last State|Reason|Restart Count' +kubectl -n get kafka -o jsonpath='{.spec.kafka.resources}{"\n"}{.metadata.ownerReferences}{"\n"}' +``` + +If the pod is not `OOMKilled` and not restarting, the usage is normal. To bring it under the inspection threshold, raise the broker memory. When `ownerReferences` names an `RdsKafka`, change the memory on the Data Services instance (its update page or its spec) so a direct edit of `Kafka` is not reverted. + +> **Warning:** Raising the broker memory rolls the broker pods one by one. Do it in a maintenance window, only when the flagged pod is a broker and it is not crash-looping. First confirm that the namespace quota has room for the larger request (Cause 3). Roll back by restoring the previous memory values. + +```bash +kubectl -n get kafka -o yaml > kafka.bak.yaml +kubectl -n edit kafka # only when the Kafka resource has no RdsKafka owner +``` + +Under `spec.kafka.resources`, raise `requests.memory` and `limits.memory` above the current limit and leave CPU unchanged. If usage climbs back to the same percentage afterwards, that is Cause 2. Growth in `-entity-operator` is an entity-operator issue, covered in the creation article linked in the Issue section. + +### Cause 2: Memory usage reaches about 90 percent whatever the limit is (page cache) + +```bash +kubectl -n get pods | grep -kafka +kubectl -n describe pod -kafka-0 | grep -E 'Last State|Reason|Restart Count' +kubectl -n top pod -kafka-0 --containers +``` + +If the restart count is not growing and the last state is not `OOMKilled`, the usage is page cache and needs no action; more memory does not lower the percentage. If the broker restarts with `OOMKilled`, the limit really is too small: raise it as in Cause 1. + +### Cause 3: Spec update and rollback both fail with exceeded quota + +```bash +kubectl -n get events --sort-by=.lastTimestamp | grep -E 'FailedCreate|ErrorKafkaStatus' | tail -20 +kubectl -n describe sts -kafka | grep -A3 FailedCreate +kubectl -n describe resourcequota default +``` + +The `FailedCreate` text names the quota, the request, the usage, and the limit. A rollback does not help when the hard limit minus `used` is below the old request as well. Check the project quota of that cluster on the project page. + +> **Warning:** Raising the project quota, and then the namespace quota `default`, lets the project and namespace schedule more CPU, memory, or storage than before. Do it only when `FailedCreate` names `exceeded quota` for a broker pod of this instance. Note the current values first; roll back by setting them again once the instance is stable. + +1. Raise the project quota for that cluster if it is fully allocated. +2. Make sure the hard limit minus `used` of the namespace quota `default` covers the broker request; raise it within the project quota if not. +3. Watch the brokers roll: + + ```bash + kubectl -n describe resourcequota default + kubectl -n rollout status sts/-kafka + kubectl -n get pods | grep -E '-(kafka|zookeeper)' + ``` + +The StatefulSet then completes its rolling update and all brokers run at the new spec. + +### Cause 4: The namespace LimitRange and ResourceQuota leave the broker pods without resources + +```bash +kubectl -n get kafka -o jsonpath='{.status.conditions}{"\n"}' +kubectl -n get strimzipodset +kubectl -n get pods | grep -kafka +kubectl -n get events --sort-by=.lastTimestamp | tail -40 +kubectl -n describe limitrange +kubectl -n describe resourcequota +``` + +Compare the broker requests and limits with the `LimitRange` maximums and with the quota hard limits minus `used`. A missing broker pod, an `exceeded quota` or LimitRange `forbidden` message, or a broker killed for lack of memory points at this cause. With `StrimziPodSet` the cluster operator creates the pods itself, so a rejected create shows up in the operator log and the namespace events, not as a StatefulSet `FailedCreate`. + +> **Warning:** Raising the namespace `LimitRange` and `ResourceQuota` lets the namespace consume more CPU and memory than before. Do it only when the events or the operator log name the quota or limit for the broker pods of this instance. Save both objects first and roll back by re-applying them once the instance is stable. + +```bash +kubectl -n get limitrange -o yaml > limitrange.bak.yaml +kubectl -n get resourcequota -o yaml > resourcequota.bak.yaml +kubectl -n edit limitrange +kubectl -n edit resourcequota +``` + +Raise the container maximums so they cover the broker request and limit, and the quota hard values so they cover current usage plus every broker pod of the instance. The same limits can be changed from the namespace settings in the console. Then watch the brokers start and the condition become `Ready`: + +```bash +kubectl -n get pods -w | grep -kafka +kubectl -n get kafka +``` + +## Diagnostic Steps + +1. Identify which pod the memory figure belongs to: + + ```bash + kubectl -n top pod --containers | grep + ``` + + A broker container: Cause 1 or Cause 2. `-entity-operator`: not this article. + +2. Memory around 90 percent that stays there after an increase, no `OOMKilled`, no backlog: Cause 2. + +3. A memory change that fails, a rollback that fails too, and `FailedCreate` with `exceeded quota: default`: Cause 3. + +4. `Kafka` condition `NotReady` with `Exceeded timeout of 300000ms while waiting for Pods resource -kafka-`, broker pods not running, and a `LimitRange` or `ResourceQuota` in the namespace: Cause 4. The same `Timed out waiting for a node assignment` text from a CLI tool while the `Kafka` resource is `Ready` and every broker pod is Running is a different problem (often a SASL listener used without client credentials). + +5. Quota pressure at a glance: + + ```bash + kubectl -n get resourcequota -o custom-columns='NAME:.metadata.name,HARD:.status.hard,USED:.status.used' + ``` + + Usage close to hard on memory or CPU confirms there is no room for a recreated broker. diff --git a/docs/en/solutions/Kafka_client_connection_failures_timeouts_TLS_certificates_authorization_NodePort_and_reverse_DNS_on_ACP.md b/docs/en/solutions/Kafka_client_connection_failures_timeouts_TLS_certificates_authorization_NodePort_and_reverse_DNS_on_ACP.md new file mode 100644 index 000000000..95e751e26 --- /dev/null +++ b/docs/en/solutions/Kafka_client_connection_failures_timeouts_TLS_certificates_authorization_NodePort_and_reverse_DNS_on_ACP.md @@ -0,0 +1,283 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda Application Services +ProductsVersion: + - "3.10" + - "3.12" + - "3.14" + - "3.16" + - "3.18" + - "4.0" + - "4.1" + - "4.2" +--- + +# Kafka client connection failures: timeouts, TLS certificates, authorization, NodePort, and reverse DNS on ACP + +## Issue + +Applications cannot reach a Data Services Kafka instance, or need to know how to connect: + +- New connections time out from outside the cluster on the external listener, while the broker pods are Running. +- The broker returns `TopicAuthorizationException: Not authorized to access topics: []` and a Spring application logs `Send failed`. Clients that used to connect without a user are rejected after authentication was turned on. +- A Kafka CLI tool run inside a broker pod fails with `TimeoutException: Timed out waiting for a node assignment. Call: listTopics`, and the broker logs `Failed authentication with /
(Unexpected Kafka request of type METADATA during SASL handshake.)`. +- A TLS client fails with `ERR_TLS_CERT_ALTNAME_INVALID`: `Failed to connect to seed broker, trying another broker from the list: Connection error: Hostname/IP does not match certificate's altnames: IP: is not in the cert's list:`. +- `openssl` on the client certificate shows it is past `notAfter`. +- Exporting the broker certificate through the console file download saves a `.crt` of 0 bytes. +- A new Kafka instance also creates a NodePort Service that is not wanted. +- Questions: can the connection address shown in Data Services change or be pinned, and can the Kafka Pod IPs be fixed? + +A user whose secret does not exist, a user stuck in `Creating`, or `Simple authorization ACL rules are configured but not supported in the Kafka cluster configuration.` is covered in [Kafka user secret missing, user stuck in Creating, ACL not supported, or uppercase topic names on ACP](Kafka_user_secret_missing_user_stuck_in_Creating_ACL_not_supported_or_uppercase_topic_names_on_ACP.md). An application outside the cluster that reports `No route to host` while curl from inside the application pod to Kafka succeeds is a network path problem, not a certificate fault. + +Listener ports of a Data Services Kafka: `9092` username/password inside the cluster, `9093` TLS inside the cluster, `9094` the external listener published through NodePort. + +## Root Cause + +### Cause 1: Kafka 2.7.0 blocks SASL_SSL threads on reverse DNS + +New connections to a Kafka 2.7.0 broker on the external SASL_SSL listener (port 9094 behind a NodePort) time out, both by Pod IP and by NodePort. One broker accumulates many `CLOSE_WAIT` sockets (around 200) and its listen socket receive queue backs up. Inside the pod, reverse lookups of client addresses hang: CoreDNS forwards those PTR queries to an upstream resolver that the pod cannot reach, and the SASL_SSL accept thread stalls in the lookup. These are the upstream Kafka defects KAFKA-13576 and KAFKA-8562. + +### Cause 2: The application client certificate is expired + +Kafka-related application pods fail because the client certificate they present is past its end date. Replacing that client certificate restores them. The operator's clients CA is a different object: on ACP 3.10.2 it is issued for about one year, and the operator renews it near expiry, which restarts brokers (see [Kafka Restarts After Client CA Certificate Expiry on ACP 3.10.2](ecosystem/kafka/Kafka_Certificate_Expiry_Restart_on_ACP_3102.md)). + +### Cause 3: Authentication and authorization are on and the client has no matching user, ACL, or SASL settings + +With authentication enabled on a listener, a client needs a user, and with `authorization.type: simple` that user needs ACL entries for the topic and for the consumer group. A user without the group ACL still fails as a consumer. + +A client that connects to a SASL listener without SASL settings (for example `kafka-topics.sh --bootstrap-server localhost:9092` with no client properties) sends ordinary Kafka requests where the broker expects a SASL handshake. The broker logs `Unexpected Kafka request of type METADATA during SASL handshake` and closes the connection, and the client keeps retrying until `Timed out waiting for a node assignment`. The source address in that broker line can be a node address (for example a Kube-OVN join subnet address when traffic arrives through a NodePort); it identifies the client, not a failing broker. + +### Cause 4: The console file download writes an empty crt + +The certificate path inside the broker pod is correct, but the console exec download saves a 0-byte `.crt`. + +### Cause 5: External access is enabled, so a NodePort Service is created + +A new Kafka gets a Service of type NodePort because the instance was created with external access on, and the external listener (named `external`, Service `-kafka-external-bootstrap`) is present in the Kafka resource. No safe way is known to drop that Service from an existing instance without recreating it. + +### Cause 6: The dialed broker IP is not in the certificate SAN + +The client performs TLS hostname verification and dials an IP that is not in the server certificate's subject alternative names. An empty list after `is not in the cert's list:` means the certificate carries no SAN at all. The failure is in the client that verifies the name, not in a broker refusing TCP. + +### Cause 7: Does the Kafka connection address in Data Services change, and can it be fixed? + +The question comes up before applications are pointed at a new instance. + +### Cause 8: Can the IP addresses of the Kafka Pods be fixed? + +The request is to give broker Pods fixed IPs, for example from a Kube-OVN underlay IP pool, so that clients or firewalls can use them. + +## Resolution + +### Cause 1: Kafka 2.7.0 blocks SASL_SSL threads on reverse DNS + +Apply this only when the broker is 2.7.0, the timeout is on 9094 or its NodePort, `CLOSE_WAIT` climbs on a broker, and PTR lookups from inside the pod are slow. + +Add host records for the client source addresses to the CoreDNS `hosts` plugin so the reverse lookup answers immediately: + +```bash +kubectl -n kube-system get cm coredns -o yaml > coredns.bak.yaml +kubectl -n kube-system edit cm coredns +``` + +```text +hosts { + 192.0.2.10 client-a.example.internal + fallthrough +} +``` + +Use the real client source addresses. CoreDNS reloads the Corefile on its own when the `reload` plugin is enabled; otherwise restart it: + +> **Warning:** Restarting CoreDNS briefly affects name resolution in the cluster. Do it one replica at a time with a rollout, and only after the ConfigMap change is saved. Roll back by re-applying `coredns.bak.yaml`. + +```bash +kubectl -n kube-system rollout restart deploy/coredns +kubectl -n exec -kafka-0 -- getent hosts 192.0.2.10 +``` + +The lookup should return immediately; then retry the external client. Moving the cluster off 2.7.0 also avoids the defect: 2.7.2, 2.8.1, and 3.0.0 do not block this way. A version change rolls the brokers; do it in a window. + +### Cause 2: The application client certificate is expired + +```bash +openssl x509 -in client.crt -noout -dates +``` + +Replace the expired certificate in the application key store or trust store and roll the application pods. + +To inspect the clients CA of the instance: + +```bash +kubectl -n get secret -clients-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > clients-ca.crt +openssl x509 -in clients-ca.crt -noout -dates +``` + +To lengthen later issuances, set both CA lifetimes on the `Kafka` resource (`3650` is a supported multi-year value) and renew: + +```yaml +spec: + clusterCa: + validityDays: 3650 + clientsCa: + validityDays: 3650 +``` + +> **Warning:** Renewing the clients CA with `strimzi.io/force-renew` restarts brokers and invalidates client certificates signed by the previous CA. Use it only when `openssl -dates` shows the CA itself is expired or close to expiry, in a maintenance window, and give clients the new CA material afterwards. An expired application certificate alone is replaced without this step. + +```bash +kubectl -n annotate secret -clients-ca-cert strimzi.io/force-renew=true --overwrite +``` + +### Cause 3: Authentication and authorization are on and the client has no matching user, ACL, or SASL settings + +Create a user bound to the instance and grant topic and group operations. For a Data Services instance the object is `RdsKafkaUser` (`middleware.alauda.io/v1`, label `middleware.alauda.io/cluster: `); for a Strimzi `Kafka` created in the management view it is `KafkaUser` (label `strimzi.io/cluster: `). The listener must have authentication (`scram-sha-512` or `tls`) and `spec.kafka.authorization.type` must be `simple`. + +```yaml +apiVersion: kafka.strimzi.io/v1beta1 +kind: KafkaUser +metadata: + name: + namespace: + labels: + strimzi.io/cluster: +spec: + authentication: + type: scram-sha-512 + authorization: + type: simple + acls: + - {host: "*", operation: Write, resource: {type: topic, name: , patternType: literal}} + - {host: "*", operation: Read, resource: {type: topic, name: , patternType: literal}} + - {host: "*", operation: Describe, resource: {type: topic, name: , patternType: literal}} + - {host: "*", operation: Read, resource: {type: group, name: , patternType: literal}} +``` + +```bash +kubectl -n apply -f kafkauser.yaml +kubectl -n get kafkauser +kubectl -n get secret -o jsonpath='{.data.password}' | base64 -d; echo +kubectl -n get svc -kafka-bootstrap -kafka-external-bootstrap +``` + +Password clients use `SASL_PLAINTEXT` on 9092 (or `SASL_SSL` on a TLS listener) with `sasl.mechanism=SCRAM-SHA-512`. TLS clients also need the cluster CA in their trust store. + +For CLI tools inside a broker pod, write a client file and pass it with `--command-config`: + +```bash +kubectl -n exec -i -kafka-0 -- sh -c 'cat > /tmp/client.properties' <<'EOF' +security.protocol=SASL_PLAINTEXT +sasl.mechanism=SCRAM-SHA-512 +sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="" password=""; +EOF +kubectl -n exec -kafka-0 -- /opt/kafka/bin/kafka-topics.sh \ + --bootstrap-server localhost:9092 --command-config /tmp/client.properties --describe --under-replicated-partitions +``` + +An empty result means no under-replicated partitions. + +### Cause 4: The console file download writes an empty crt + +Read the Secret, or copy the file with `kubectl cp`, instead of the console download: + +```bash +kubectl -n get secret -cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > cluster-ca.crt +kubectl -n cp -kafka-0: ./exported.crt +ls -l cluster-ca.crt exported.crt +``` + +Both files should be non-empty. + +### Cause 5: External access is enabled, so a NodePort Service is created + +For an instance that is not created yet, turn external access off in the create form; the `external` listener is then absent and no NodePort Service is created. For an existing instance, deleting the Service by hand is not a verified fix: the operator can recreate it while the external listener remains. + +### Cause 6: The dialed broker IP is not in the certificate SAN + +Confirm the address is missing from the SAN, using the port the application uses: + +```bash +openssl s_client -connect 192.0.2.10:9094 -servername 192.0.2.10 /dev/null | openssl x509 -noout -ext subjectAltName +``` + +No `X509v3 Subject Alternative Name` block means the certificate has no SAN. The fix is to turn off hostname verification on the failing client. + +> **Warning:** Turning off hostname verification makes the client accept any certificate signed by the trusted CA, whatever name it was issued for. Apply it only when the error is `Hostname/IP does not match certificate's altnames` and the dialed address is not in the SAN. Keep a copy of the previous client configuration; roll back by removing the setting. + +Java client (`ssl.endpoint.identification.algorithm` empty): + +```properties +ssl.endpoint.identification.algorithm= +``` + +Spring: `spring.kafka.properties.ssl.endpoint.identification.algorithm=`. librdkafka-based clients: `ssl.endpoint.identification.algorithm=none`. + +The wording `ERR_TLS_CERT_ALTNAME_INVALID` and `Failed to connect to seed broker` comes from a Node.js client such as KafkaJS, which does not read Java properties. There, skip the name check with `checkServerIdentity` while keeping the CA check (general Node.js behavior): + +```javascript +const fs = require('fs'); +const { Kafka } = require('kafkajs'); +const kafka = new Kafka({ + clientId: '', + brokers: ['192.0.2.10:9094'], + ssl: { + ca: [fs.readFileSync('/path/to/ca.crt', 'utf-8')], + checkServerIdentity: () => undefined, + }, +}); +``` + +### Cause 7: Does the Kafka connection address in Data Services change, and can it be fixed? + +The address is already fixed: it comes from the Kubernetes Services of the instance and stays the same as long as those Services are not modified. Changing it is not recommended. For external access, NodePort gives a node IP plus node port that does not change after the instance is created. + +```bash +kubectl -n get svc -l strimzi.io/cluster= +kubectl -n get svc -kafka-external-bootstrap -o jsonpath='{.spec.ports[*].nodePort}{"\n"}' +``` + +Inside the cluster, use the bootstrap Service `-kafka-bootstrap..svc` or the per-broker names `-kafka-.-kafka-brokers..svc`. + +### Cause 8: Can the IP addresses of the Kafka Pods be fixed? + +Not through a Data Services instance at present: `RdsKafka` does not support setting Pod `annotations`, which is how a Kube-OVN IP pool is normally attached, so a fixed IP pool cannot be assigned to the broker Pods. Pod IPs usually change when a Pod is recreated or rescheduled, so clients should not depend on them. Use the Service addresses from Cause 7 inside the cluster and NodePort from outside. + +## Diagnostic Steps + +1. Classify the client error: + + - Timeout only from outside, Kafka 2.7.0, `CLOSE_WAIT` on a broker: Cause 1. + - `certificate has expired`, or `openssl -dates` in the past: Cause 2. + - `TopicAuthorizationException`, or `Unexpected Kafka request of type METADATA during SASL handshake` on the broker with a client timeout: Cause 3. + - Empty file from the console download: Cause 4. + - `ERR_TLS_CERT_ALTNAME_INVALID`, `Hostname/IP does not match certificate's altnames`: Cause 6. Note the client language from the error format. + +2. Listener authentication and ports: + + ```bash + kubectl -n get kafka -o jsonpath='{.spec.kafka.listeners}{"\n"}{.spec.kafka.version}{"\n"}' + kubectl -n get svc -l strimzi.io/cluster= + ``` + + `"authentication":{"type":"scram-sha-512"}` on a listener means every client of that port, CLI tools included, needs SASL settings. + +3. Accept queue and reverse lookup (Cause 1). The broker image may lack `ss`; use `/proc/net/tcp` then (state `08` is `CLOSE_WAIT`): + + ```bash + kubectl -n exec -kafka-0 -- sh -c 'ss -ant 2>/dev/null | grep -c CLOSE-WAIT || awk "\$4==\"08\"" /proc/net/tcp /proc/net/tcp6 | wc -l' + kubectl -n exec -kafka-0 -- sh -c 'time getent hosts ' + ``` + + A multi-second PTR lookup is Cause 1. An immediate return rules it out. + +4. Find where a source address in a broker log comes from before blaming a broker: + + ```bash + kubectl get pod -A -o wide | grep
; kubectl get node -o wide | grep
+ kubectl get ip -o custom-columns='IP:.spec.ipAddress,SUBNET:.spec.subnet,POD:.spec.podName' | grep
+ ``` + +5. A timeout combined with ZooKeeper down or `FailedAttachVolume` is not a client problem; see [Kafka and ZooKeeper down from storage stalls, a full node disk, or a probe kill during log replay on ACP](Kafka_and_ZooKeeper_down_from_storage_stalls_a_full_node_disk_or_a_probe_kill_during_log_replay_on_ACP.md). diff --git a/docs/en/solutions/Kafka_config_change_does_not_roll_brokers_a_broker_restart_hangs_or_the_instance_stays_processing_on_ACP.md b/docs/en/solutions/Kafka_config_change_does_not_roll_brokers_a_broker_restart_hangs_or_the_instance_stays_processing_on_ACP.md new file mode 100644 index 000000000..e103ba024 --- /dev/null +++ b/docs/en/solutions/Kafka_config_change_does_not_roll_brokers_a_broker_restart_hangs_or_the_instance_stays_processing_on_ACP.md @@ -0,0 +1,154 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda Application Services +ProductsVersion: + - "3.12" + - "3.14" + - "3.16" + - "3.18" + - "4.0" + - "4.1" + - "4.2" +--- + +# Kafka config change does not roll brokers, a broker restart hangs, or the instance stays processing on ACP + +## Issue + +On a running Data Services Kafka instance: + +- A parameter change saved on the Data Services page does not restart the broker pods, so the new configuration never takes effect. Restarting the cluster operator lets stop, start, and restart work again for a while. The operator log shows: + + ```text + Failed to reconcile ... KafkaRoller$UnforceableProblem: Pod -kafka-2 is currently not rollable + Could not roll pod 2 due to ...ForceableProblem: Error getting broker config, retrying after at least 250ms + / will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 2 is restarted. + ``` + +- A broker pod does not become ready after a restart, the instance reports `Error while trying to restart pod -kafka- to become ready`, and the operator shows `.status.phase (Failed) != Active` for the `RdsKafka`. +- After a node change, the instance stays in processing (`处理中`) although the Kafka pods have already moved to the new node. + +Related articles: [Kafka broker memory looks high, or a memory change or new broker Pod is blocked by quota on ACP](Kafka_broker_memory_looks_high_or_a_memory_change_or_new_broker_Pod_is_blocked_by_quota_on_ACP.md) covers rollouts that fail with `exceeded quota`. A cluster operator stuck on `Reconciliation is in progress` with no ISR line is covered in [Kafka instance creation times out on the entity-operator, shows failed, or the cluster operator restarts on ACP](Kafka_instance_creation_times_out_on_the_entity-operator_shows_failed_or_the_cluster_operator_restarts_on_ACP.md). A broker that is down because its disk is full is covered in [Kafka broker PVC full or still too small after expansion on ACP](Kafka_broker_PVC_full_or_still_too_small_after_expansion_on_ACP.md). + +## Root Cause + +### Cause 1: Rolling restart blocked because ISR is at min.insync.replicas + +The operator will not restart a broker if that would drop any partition below `min.insync.replicas`. When some partitions already have only one replica in ISR, and that replica is on the broker to be rolled, the `KafkaRoller` marks the pod `not rollable` and keeps retrying; `KafkaAvailability` lists the protected partitions with `will be underreplicated (|ISR|=1 and min.insync.replicas=1)`. `kafka-topics.sh --describe --topic __consumer_offsets` then shows partitions whose ISR is shorter than the replica list, for example `Replicas: 2,0,1 Isr: 2` and `Replicas: 1,0,2 Isr: 2,1`, while most partitions have all three in ISR. This was seen with Strimzi cluster operator 0.25.0 (the Kafka 2.7 and 2.8 era); the published procedure for this block is written for Kafka 2.7.0 and 2.8.2 on ACP 3.x (see [Resolve Kafka ISR Synchronization Issues That Block Broker Restart](ecosystem/kafka/Kafka_ISR_Synchronization_Issues.md)). Restarting the operator does not remove the ISR gap; restoring ISR does. + +### Cause 2: A broker does not become ready after a restart and the topic-store changelog has no leader + +On ACP 3.16.2 the instance logs repeat: + +```text +Cannot get current offset of topic __strimzi-topic-operator-kstreams-topic-store-changelog partition 0: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes +``` + +The stuck restart is attributed to that operator-internal changelog partition having no leader. Deleting the stuck broker pod so it starts again restores service. + +### Cause 3: The datapool keeps the old node after Kafka has moved + +After a node change, Kafka is rescheduled onto the new node, but the datapool pod is not: its node selector still names the old node, so it cannot schedule on the remaining nodes and the instance stays in processing. The datapool belongs to the MinIO component, and Kafka's scheduling change does not update MinIO. + +## Resolution + +### Cause 1: Rolling restart blocked because ISR is at min.insync.replicas + +1. Confirm the log lines and the ISR gap: + + ```bash + kubectl get pods -A -l strimzi.io/kind=cluster-operator + kubectl -n logs deploy/ --tail=500 | grep -E 'not rollable|will be underreplicated|Error getting broker config|Failed to reconcile' + kubectl -n exec -kafka-0 -- /opt/kafka/bin/kafka-topics.sh \ + --bootstrap-server localhost:9092 --describe --topic __consumer_offsets + ``` + + Use the topic named in the operator log if it is not `__consumer_offsets`. If the listener on `localhost:9092` requires SASL, add `--command-config ` to every Kafka command here. Take the broker ids from the `Replicas` line of a partition that lists every replica (ids such as `0`, `1`, `2`, or larger ids such as `1100`). + +2. Generate a reassignment plan. Create `/tmp/topic-generate.json`: + + ```json + {"topics": [{"topic": "__consumer_offsets"}], "version": 1} + ``` + + ```bash + kubectl -n cp /tmp/topic-generate.json -kafka-0:/tmp/topic-generate.json + kubectl -n exec -kafka-0 -- /opt/kafka/bin/kafka-reassign-partitions.sh \ + --bootstrap-server localhost:9092 \ + --topics-to-move-json-file /tmp/topic-generate.json \ + --broker-list ",," --generate + ``` + + Save the "Current partition replica assignment" from the output for rollback, and save the "Proposed partition reassignment configuration" as `/tmp/partition-replica-reassignment.json`. + +3. Execute it. + +> **Warning:** Partition reassignment moves replicas and adds disk and network load until it finishes. Run it only when the operator log says the broker `is currently not rollable` with `will be underreplicated`, and describe shows ISR shorter than Replicas on the topic you pass in. Use a maintenance window. Roll back by executing the saved current assignment. The alternative in the same procedure, force-deleting the stuck broker pod (`kubectl delete pod --force --grace-period=0`), can lose the latest records when ISR has a single replica; use it only when the business accepts that loss, which cannot be rolled back. + + ```bash + kubectl -n cp /tmp/partition-replica-reassignment.json -kafka-0:/tmp/partition-replica-reassignment.json + kubectl -n exec -kafka-0 -- /opt/kafka/bin/kafka-reassign-partitions.sh \ + --bootstrap-server localhost:9092 \ + --reassignment-json-file /tmp/partition-replica-reassignment.json --execute + kubectl -n exec -kafka-0 -- /opt/kafka/bin/kafka-reassign-partitions.sh \ + --bootstrap-server localhost:9092 \ + --reassignment-json-file /tmp/partition-replica-reassignment.json --verify + ``` + +4. When describe shows ISR equal to Replicas, apply the parameter change again; the brokers can roll. + +### Cause 2: A broker does not become ready after a restart and the topic-store changelog has no leader + +```bash +kubectl get rdskafka -A +kubectl -n get pods | grep +kubectl -n logs deploy/-entity-operator --all-containers --tail=300 | grep -i 'topic-store-changelog' +kubectl -n logs -kafka- --tail=300 +kubectl -n logs deploy/ --tail=500 | grep -E 'not rollable|will be underreplicated' +``` + +The changelog error can appear in the entity-operator, broker, or exporter logs of the instance. If the operator log has `not rollable` with `will be underreplicated`, use Cause 1 instead: deleting the pod there can lose records. + +> **Warning:** Deleting the stuck broker pod restarts that broker; partitions it leads move to other brokers and clients reconnect. Do it only when the instance reports `Error while trying to restart pod -kafka- to become ready`, the logs show no leader for `__strimzi-topic-operator-kstreams-topic-store-changelog`, the other brokers are Running, and the operator log has no `not rollable` line. The pod is recreated with the same PVC, so there is nothing to roll back. Add `--force --grace-period=0` only if the pod stays `Terminating`. + +```bash +kubectl -n delete pod -kafka- +kubectl -n get pods -w | grep -kafka- +kubectl get rdskafka -A +``` + +The pod should become `1/1 Running` and the `RdsKafka` phase should return to `Active`. + +### Cause 3: The datapool keeps the old node after Kafka has moved + +```bash +kubectl get pods -A -o wide | grep -i datapool +kubectl -n describe pod | grep -E 'Node-Selectors|Status:|Controlled By|Warning' +kubectl -n get pod -o jsonpath='{.metadata.ownerReferences}{"\n"}' +``` + +A Pending datapool pod whose node selector names a node Kafka has already left confirms this cause. Change the scheduling node of the MinIO component separately, to the node Kafka now runs on, with the same scheduling setting used for Kafka. + +> **Warning:** Changing the MinIO scheduling node reschedules the datapool pod. Do it only when that pod is Pending, its node selector names a node Kafka has left, and the target node is Ready. Roll back by pointing MinIO back at the previous node if that node is still schedulable. + +After the datapool pod is Running on the new node, the instance leaves processing. + +## Diagnostic Steps + +1. A console parameter change that does not roll brokers: read the operator log. `is currently not rollable` with `will be underreplicated (|ISR|=1 and min.insync.replicas=1)` and an `Isr` list shorter than `Replicas` in describe is Cause 1. Only `Reconciliation is in progress`, with no ISR line, is the stuck-operator case in the creation article. + +2. List only the partitions with a gap: + + ```bash + kubectl -n exec -kafka-0 -- /opt/kafka/bin/kafka-topics.sh \ + --bootstrap-server localhost:9092 --describe --under-replicated-partitions + ``` + + Empty output means no partition is under-replicated, so an ISR block is unlikely. A `Timed out waiting for a node assignment` from this command while every broker pod is Running usually means the listener needs SASL settings (`--command-config`). + +3. `Error while trying to restart pod -kafka- to become ready` with no leader for `__strimzi-topic-operator-kstreams-topic-store-changelog`, and no `not rollable` line: Cause 2. + +4. Instance in processing after a node change: compare the nodes of the Kafka pods and the datapool pod. Kafka moved, datapool Pending with the old node selector: Cause 3. diff --git a/docs/en/solutions/Kafka_instance_creation_times_out_on_the_entity-operator_shows_failed_or_the_cluster_operator_restarts_on_ACP.md b/docs/en/solutions/Kafka_instance_creation_times_out_on_the_entity-operator_shows_failed_or_the_cluster_operator_restarts_on_ACP.md new file mode 100644 index 000000000..09c0d3b74 --- /dev/null +++ b/docs/en/solutions/Kafka_instance_creation_times_out_on_the_entity-operator_shows_failed_or_the_cluster_operator_restarts_on_ACP.md @@ -0,0 +1,174 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda Application Services +ProductsVersion: + - "3.6" + - "3.12" + - "3.14" + - "3.16" + - "3.18" + - "4.0" + - "4.1" + - "4.2" +--- + +# Kafka instance creation times out on the entity-operator, shows failed, or the cluster operator restarts on ACP + +## Issue + +A Data Services Kafka instance does not come up, or the operator behind it is unhealthy: + +- The create fails, or the new instance sits failed, with: + + ```text + Exceeded timeout of 300000ms while waiting for Deployment resource -entity-operator in namespace to be ready + ``` + +- The Kafka service looks deployed, yet the instance still shows a creation failure in the console. +- The Kafka cluster operator pod restarts continuously (for example after an upgrade to ACP 3.14), so the console cannot create an instance or pick a namespace. +- A parameter change saved in the console never rolls the broker pods until the cluster operator pod is deleted. +- On a running instance, the memory of the entity-operator pod keeps rising and a resource increase has to be justified. + +Related articles: [No Kafka in Data Services, or Alauda Build of Kafka 2 unpublished or failing to deploy on ACP](No_Kafka_in_Data_Services_or_Alauda_Build_of_Kafka_2_unpublished_or_failing_to_deploy_on_ACP.md) covers the operator install and marketplace side. A ZooKeeper or broker pod named in the 300000 ms timeout is covered in [Kafka and ZooKeeper down from storage stalls, a full node disk, or a probe kill during log replay on ACP](Kafka_and_ZooKeeper_down_from_storage_stalls_a_full_node_disk_or_a_probe_kill_during_log_replay_on_ACP.md) and in [Kafka broker memory looks high, or a memory change or new broker Pod is blocked by quota on ACP](Kafka_broker_memory_looks_high_or_a_memory_change_or_new_broker_Pod_is_blocked_by_quota_on_ACP.md). A rollout blocked by `is currently not rollable` is covered in [Kafka config change does not roll brokers, a broker restart hangs, or the instance stays processing on ACP](Kafka_config_change_does_not_roll_brokers_a_broker_restart_hangs_or_the_instance_stays_processing_on_ACP.md). + +## Root Cause + +### Cause 1: The entity-operator Deployment never becomes Ready + +The create error names Deployment `-entity-operator` and the 300000 ms wait. On ACP 3.6 and 3.6.1 the containers of that Deployment inherit a very small namespace default limit and never become Ready (see [Kafka Entity Operator Fails to Start Because Container Limits Are Too Small](ecosystem/kafka/Kafka_Entity_Operator_Startup_Failure_Due_to_Container_Limits.md)). Later versions set explicit resources on the entity operator; the same timeout then means the pod is Pending, failing to pull its image, or crash-looping for another reason. The 300000 ms text only says that the Deployment did not become Ready in time. + +### Cause 2: The namespace quota blocks the Kafka exporter + +The Kafka brokers are running, but the instance shows creation failed. The namespace `ResourceQuota` is at its hard limit, so the Kafka exporter pod cannot be created. Raising the quota lets the exporter start, and the instance returns to normal. + +### Cause 3: The cluster operator is crash-looping or not reconciling + +Two different states: + +- On ACP 3.14 the operator pod restarts continuously. The startup failure for that upgrade is a null pointer when an `RdsTopic` exists and the matching `KafkaTopic` does not (see [Kafka Operator Startup Exception After ACP 3.14 Upgrade](ecosystem/kafka/Kafka_Operator_Startup_Exception_After_314_Upgrade.md)). +- The operator logs `Reconciliation is in progress` after an API server outage and then never moves instances; a parameter edit in the console then waits until the operator pod is deleted. This was fixed in the ACP 3.15 operator line; deleting the operator pod is the recovery on earlier builds (see [Kafka Operator Is Blocked by an Abnormal Instance](ecosystem/kafka/Kafka_Operator_Blocked_By_Abnormal_Instance.md)). + +`Reconciliation is in progress` alone does not prove a stuck operator. When the same log also has `KafkaRoller$UnforceableProblem: Pod -kafka- is currently not rollable` and `will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker is restarted`, the operator is deliberately holding the roll because of ISR, which is a different problem. + +### Cause 4: Entity operator memory keeps rising on a running instance + +The entity-operator pod of a running instance shows memory usage climbing steadily, for example from 67 percent to 92 percent of its limit, with `strimzi-kafka-operator.v4.2.0`, an entity operator sized at 1 CPU and 1000 MiB, two users, and one topic. No memory leak is known in that Kafka version and no root cause was established. Raising the entity operator to 2 CPU and 2 GiB and continuing to observe was sufficient: the instance kept operating normally. + +## Resolution + +### Cause 1: The entity-operator Deployment never becomes Ready + +```bash +kubectl -n get deploy -entity-operator +kubectl -n describe pod -l strimzi.io/name=-entity-operator +kubectl -n get limitrange -o yaml +``` + +When the described containers show a tiny CPU or memory limit inherited from the namespace `LimitRange` on ACP 3.6 or 3.6.1 (or the pod is clearly throttled or `OOMKilled` on that limit), raise the namespace default limit to a size the entity operator can start with, then recreate the pod. Deleting the pod only restarts topic and user reconciliation; brokers are not affected. + +```bash +kubectl -n delete pod -l strimzi.io/name=-entity-operator +kubectl -n rollout status deploy/-entity-operator +``` + +On later platforms, follow the `describe` output instead: image pull failures, unschedulable pods, and crashes in the `topic-operator` or `user-operator` container are ordinary workload faults. + +### Cause 2: The namespace quota blocks the Kafka exporter + +```bash +kubectl -n get pods | grep +kubectl -n get rs | grep -kafka-exporter +kubectl -n describe rs -kafka-exporter- +kubectl -n describe resourcequota +``` + +The exporter ReplicaSet event names the exceeded quota (`exceeded quota: , requested: ..., used: ..., limited: ...`). The exporter is normally named `-kafka-exporter`. + +> **Warning:** Raising a namespace `ResourceQuota` lets that namespace consume more CPU, memory, or pods than before. Do it only when the exporter event says the quota is exceeded and the instance is failed because that pod is not running. Save the quota first; roll back by re-applying the saved file. + +```bash +kubectl -n get resourcequota -o yaml > resourcequota.bak.yaml +kubectl -n edit resourcequota +``` + +Raise the exhausted hard value so it covers current usage plus the exporter request. On a project with a project quota, the namespace quota cannot exceed what the project has left; raise the project quota first if needed. After the exporter is Running, the instance status returns to normal. + +### Cause 3: The cluster operator is crash-looping or not reconciling + +```bash +kubectl get pods -A -l strimzi.io/kind=cluster-operator +kubectl -n logs deploy/ --tail=200 +kubectl -n logs deploy/ --tail=500 | grep -E 'not rollable|will be underreplicated|Error getting broker config' +``` + +If the ISR lines are present, stop here and use the config-rollout article linked in the Issue section. + +For the ACP 3.14 null pointer on an `RdsTopic` without a `KafkaTopic`: + +> **Warning:** Clearing finalizers and deleting an `RdsTopic` drops that custom resource. Do it only when the operator log names that `RdsTopic` and the failure is the startup null pointer with no matching `KafkaTopic`. Back it up first and roll back by re-creating it from the backup. On versions affected by the topic deletion defect, change `spec.topicName` to an unused name before the delete if it still points at a live topic (see [Avoid Kafka Topic Data Loss When Deleting an Abnormal Topic CR](ecosystem/kafka/Kafka_Topic_Delete_Data_Loss_Issue.md)). + +```bash +kubectl -n get rdstopic -o yaml > rdstopic.bak.yaml +kubectl -n patch rdstopic --type merge -p '{"metadata":{"finalizers":[]}}' +kubectl -n delete rdstopic +``` + +For reconciliation stuck after an API server outage, or a console parameter change that never rolls the brokers with no ISR line in the log, restart only the cluster operator. This does not restart brokers; reconciliation resumes when the new pod is up. + +```bash +kubectl delete pods -A -l strimzi.io/kind=cluster-operator +kubectl get kafka -A +``` + +Wait until each `Kafka` returns to its previous ready state. + +### Cause 4: Entity operator memory keeps rising on a running instance + +```bash +kubectl get csv -A | grep -i strimzi +kubectl -n top pod --containers | grep entity-operator +kubectl -n get kafka -o jsonpath='{.spec.entityOperator}{"\n"}' +kubectl -n get kafka -o jsonpath='{.metadata.ownerReferences}{"\n"}' +``` + +Raise the entity operator to 2 CPU and 2 GiB. On a Strimzi `Kafka` the resources are set per container, under `spec.entityOperator.topicOperator.resources` and `spec.entityOperator.userOperator.resources`; give the larger limit to the container that `kubectl top` shows growing. If `ownerReferences` shows the `Kafka` is owned by an `RdsKafka`, make the change on that instance (its update page or its spec) so the operator does not revert a direct edit. + +> **Warning:** Changing the entity operator resources recreates the entity-operator pod. Topic and user reconciliation pauses until the new pod is Ready; brokers are not restarted. Do it only when the growing pod is the entity operator. Roll back by restoring the previous `resources` values. + +```bash +kubectl -n patch kafka --type merge -p '{"spec":{"entityOperator":{"":{"resources":{"requests":{"cpu":"1","memory":"2Gi"},"limits":{"cpu":"2","memory":"2Gi"}}}}}}' +kubectl -n rollout status deploy/-entity-operator +``` + +If usage keeps climbing toward the new limit with an unchanged number of topics and users, collect the operator logs and the CSV version for escalation. Broker memory above an inspection threshold is covered in the broker memory article linked in the Issue section. + +## Diagnostic Steps + +1. Read the resource named in the 300000 ms sentence. A Deployment ending in `-entity-operator` is Cause 1. A ZooKeeper or broker pod, or a `Service`, is a different problem (see the related articles). + +2. Operator and instance health: + + ```bash + kubectl get pods -A -l strimzi.io/kind=cluster-operator + kubectl get kafka,rdskafka -A + kubectl -n get deploy,sts,po | grep + ``` + + A cluster operator pod with a growing restart count is Cause 3. + +3. Instance failed, brokers Running, exporter pod absent, and an `exceeded quota` event on its ReplicaSet: Cause 2. + +4. A console parameter change that does not roll brokers: check the operator log for `not rollable`. Present means an ISR block; absent with `Reconciliation is in progress` after an API server outage means Cause 3. + +5. Memory growth on a running instance: `kubectl -n top pod --containers`. Growth in `-entity-operator` is Cause 4; growth in a broker is not. + +6. Version skew between operator and instance (an upgraded operator leaving the instance on an old version, a Kafka 2.x instance that ignores the console): + + ```bash + kubectl -n get kafka -o yaml | grep -n -i version + ``` + + See [Kafka operator mismatch: wrong create page version, Kafka 2.x instance skipped, or ZooKeeper mode blocks upgrade on ACP](Kafka_operator_mismatch_wrong_create_page_version_Kafka_2x_instance_skipped_or_ZooKeeper_mode_blocks_upgrade_on_ACP.md). diff --git a/docs/en/solutions/Kafka_operator_mismatch_wrong_create_page_version_Kafka_2x_instance_skipped_or_ZooKeeper_mode_blocks_upgrade_on_ACP.md b/docs/en/solutions/Kafka_operator_mismatch_wrong_create_page_version_Kafka_2x_instance_skipped_or_ZooKeeper_mode_blocks_upgrade_on_ACP.md new file mode 100644 index 000000000..05c0c1506 --- /dev/null +++ b/docs/en/solutions/Kafka_operator_mismatch_wrong_create_page_version_Kafka_2x_instance_skipped_or_ZooKeeper_mode_blocks_upgrade_on_ACP.md @@ -0,0 +1,191 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda Application Services +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Kafka operator mismatch: wrong create page version, Kafka 2.x instance skipped, or ZooKeeper mode blocks upgrade on ACP + +## Issue + +One of these appears after a platform upgrade or after the Kafka operator version was changed: + +- The Kafka component is reported abnormal, with no error text, on a platform 4.2 cluster that still runs Kafka operator 4.1.0. +- The Data Services **Create Kafka Instance** page offers only `Kafka 4.1` or `Kafka 4.2`, although the operator was switched to a version that should offer `Kafka 3.8`. Reinstalling the operator does not change the list, and in some environments no Kafka instance can be created at all. +- A Kafka 2.x instance does not react to the console: a stop leaves it in "Stopping" with every pod still `Running`, configuration changes never reach the pods, or a broker is reported offline and nothing repairs it. `kubectl get event` in the namespace shows nothing. The cluster operator log repeats one of: + + ```text + skip reconcile (Paused), kafka version is 2.8.2 + skip reconcile , kafka version is 2.7.0 + io.vertx.core.impl.NoStackTraceThrowable: skip reconcile, not match + ``` + +- During a business-cluster upgrade, the upgrade log of the kafka functional component says the Kafka instance operator version does not match. The marketplace already shows the operator at 4.2 and the instance has not been updated. +- Questions: can the 4.3 Kafka Operator be used on a 4.1 platform, and on 4.4.0 and later are Data Services Essentials and RDS Framework still needed for Kafka? + +Operator install and publish failures (no Kafka menu, Alauda Build of Kafka 2 unpublished or failing to deploy on 4.1) are covered in [No Kafka in Data Services, or Alauda Build of Kafka 2 unpublished or failing to deploy on ACP](No_Kafka_in_Data_Services_or_Alauda_Build_of_Kafka_2_unpublished_or_failing_to_deploy_on_ACP.md). + +## Environment + +- Kafka operator package: Alauda Streaming Service for Kafka, Subscription `strimzi-kafka-operator`, CSV `strimzi-kafka-operator.v`, channel `stable`, Deployment `strimzi-cluster-operator`. It is usually installed in `kafka-system`; on clusters upgraded from 3.x it can be in another namespace such as `operators`. Find it with `kubectl get csv -A | grep -i kafka`. +- Kafka 2.x instances are served by a separate operator, Alauda Build of Kafka 2. +- The Kafka versions offered on the create page come from `ImageVersion` objects (`imageversion.middleware.alauda.io`), one per Kafka operator artifact, for example `kafka-artifact-v4.1.2` or `kafka-artifact-v4.2.0`. + +## Root Cause + +### Cause 1: Kafka operator 4.1.0 on platform 4.2 leaves the Kafka component abnormal + +`strimzi-kafka-operator.v4.1.0` is not compatible with platform 4.2. Replacing it with operator 4.1.2, the latest release of that line at the time, fixes the component: OperatorHub then shows `strimzi-kafka-operator.v4.1.2` with status `Succeeded`, and a Kafka 3.8 instance (ZooKeeper mode) runs with its broker pods `Running (1/1)` and its PVCs `Bound`. + +### Cause 2: A leftover ImageVersion from another operator version controls the create page + +`kubectl get imageversion` lists two Kafka artifacts, for example `kafka-artifact-v4.1.2` and `kafka-artifact-v4.2.0`, or `kafka-artifact-v4.1.0` and `kafka-artifact-v4.3.2`. + +The Kafka operator was moved to a different version (for example 4.2.0 to 4.1.2 on platform 4.2.4 to get Kafka 3.8, or a 4.3 operator was installed, uninstalled, and replaced by 4.1.0). The new operator installed cleanly (`Succeeded`, `InstallSucceeded`, "install strategy completed with no errors"), but the `ImageVersion` of the previous operator stayed in the cluster. The higher version takes priority, so the create page keeps offering the Kafka versions of the removed operator. When a lower Kafka Operator version is installed after a higher one, the leftover resources of the higher version must be cleaned up by hand. + +### Cause 3: No Kafka 2 operator reconciles the Kafka 2.x instance + +After an upgrade to platform 4.0 (for example from 3.16 to 4.0.9), only Alauda Streaming Service for Kafka is installed. Alauda Build of Kafka 2, the operator for Kafka 2.x instances, was not published or installed. The installed operator skips every Kafka 2.x instance on each timer reconciliation (`skip reconcile ... kafka version is 2.`, `skip reconcile, not match`) and then logs it as `reconciled`. Nothing manages the instance any more: a stop from the console never reaches the pods, configuration changes are not applied, and a broker that drops out is not repaired by an operator. Alauda Build of Kafka 2 operator `v4.0.3` supports platform `v4.0` and `v4.1` and Kafka `2.5`, `2.7`, and `2.8`. + +A `tlsSidecar ... has been deprecated` warning in the same log is unrelated and can be ignored. + +### Cause 4: Kafka operator upgraded to 4.2 while instances still run in ZooKeeper mode + +The product document "Migrate from ZooKeeper to KRaft mode" states that the v4.2.x Kafka operator supports only KRaft mode and has removed ZooKeeper mode. Before upgrading to Alauda Streaming Service for Kafka v4.2.x, every instance must be migrated to KRaft mode with the v4.1.x operator; the v4.2.x operator cannot perform that migration. Upgrading the operator to 4.2 while a ZooKeeper-mode instance exists leaves that instance behind the operator, and the cluster upgrade reports the version mismatch. + +### Cause 5: Can the 4.3 Kafka Operator be used on a 4.1 platform? + +The 4.3 operator is documented as compatible with 4.0, 4.1, and 4.2, but it gives a usable create page only when the conditions in the Resolution are met. + +### Cause 6: On 4.4.0 and later, are Data Services Essentials and RDS Framework still needed? + +The question is whether only Kafka 2 and ClickHouse Operator need to be deployed on v4.4.0 and later, without the web console plugins Alauda Container Platform Data Services Essentials and Alauda Container Platform Data Services RDS Framework. + +## Resolution + +### Cause 1: Kafka operator 4.1.0 on platform 4.2 leaves the Kafka component abnormal + +```bash +kubectl get csv -A | grep strimzi-kafka-operator +kubectl -n get subscription strimzi-kafka-operator -o jsonpath='{.status.installedCSV}{"\n"}' +``` + +When the CSV is `strimzi-kafka-operator.v4.1.0` and the platform is 4.2, replace it with operator 4.1.2 in **Administrator > App Marketplace Management > OperatorHub** (Alauda Streaming Service for Kafka, channel `stable`). If 4.1.2 is not offered, upload that operator package to the platform first with `violet push`. + +> **Warning:** Replacing the operator restarts `strimzi-cluster-operator`. Reconciliation of every Kafka instance on the cluster pauses until the new CSV is `Succeeded`, and the new operator can then roll broker pods if it changes their images or configuration. Do it only when the installed CSV is 4.1.0 on a platform 4.2 cluster. Roll back by installing 4.1.0 again from OperatorHub. + +```bash +kubectl get csv -n -w +kubectl get kafka -A +kubectl -n get pods,pvc | grep +``` + +If an instance still does not recover and its data is not needed, it can be recreated after the 4.1.2 operator is `Succeeded`. + +> **Warning:** Recreating an instance deletes its topics and messages; this cannot be undone. Do it only for an instance whose data is not needed. + +### Cause 2: A leftover ImageVersion from another operator version controls the create page + +```bash +kubectl get imageversion +kubectl get csv -A | grep strimzi-kafka-operator +``` + +The `ImageVersion` to remove is the Kafka artifact whose version does not match the installed CSV, for example `kafka-artifact-v4.2.0` while the CSV is `strimzi-kafka-operator.v4.1.2`. + +> **Warning:** Deleting an `ImageVersion` removes the Kafka versions it provides from the create page. Do it only when two `kafka-artifact-*` objects exist, the one you delete belongs to an operator version that is no longer installed, and no instance is meant to use the versions it offers. Export it first; roll back by re-creating it from that file after removing `metadata.resourceVersion`, `metadata.uid`, and `metadata.creationTimestamp`. + +```bash +kubectl get imageversion kafka-artifact-v4.2.0 -o yaml > kafka-artifact-v4.2.0.yaml +kubectl delete imageversion kafka-artifact-v4.2.0 +kubectl get imageversion +``` + +Refresh the create page. It then offers `Kafka 3.8`, and with the 4.1.x operator `ZooKeeper 模式` (ZooKeeper mode) is available. When a 4.3 operator had been installed first, clean up all leftover resources of that higher-version package, including `kafka-artifact-v4.3.2`. + +### Cause 3: No Kafka 2 operator reconciles the Kafka 2.x instance + +1. Confirm the instance version, the skip message, and that no Kafka 2 operator is installed: + + ```bash + kubectl get kafka -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,VERSION:.spec.kafka.version + kubectl get csv -A | grep -i kafka + kubectl -n logs deploy/strimzi-cluster-operator -c strimzi-cluster-operator --tail=500 | grep 'skip reconcile' + ``` + + A Kafka `2.` instance, `skip reconcile ... kafka version is 2.` for it, and only `strimzi-kafka-operator.v` (Alauda Streaming Service for Kafka) in the CSV list confirm this cause. + +2. Upload the Alauda Build of Kafka 2 operator package to the platform with `violet push`. Use the `violet` version that matches the operator package, not simply the `violet` binary that came with the platform. Use a package version that supports both the platform and the Kafka version, for example `v4.0.3` for platform 4.0 or 4.1 with Kafka 2.5, 2.7, or 2.8. + +3. Publish it and install it from **OperatorHub** (Alauda Build of Kafka 2, channel `stable`). If the package is uploaded but OperatorHub still shows it as not published, or the deployment fails on 4.1, see the article on Alauda Build of Kafka 2 publishing linked in the Issue section. Do not uninstall Alauda Streaming Service for Kafka for this. + +4. Restart the Kafka 2.x instance from the console (or repeat the stop that did nothing) so the Kafka 2 operator takes it over. + +> **Warning:** Restarting the instance rolls its broker and ZooKeeper pods; clients reconnect and partitions led by a restarting broker move. Do it only after the Alauda Build of Kafka 2 CSV is `Succeeded`, in a maintenance window. There is nothing to roll back; if the operator does not take the instance over, collect the operator logs before further changes. + +5. Check that the Kafka 2 CSV is listed and that `skip reconcile` no longer appears for the instance in the Alauda Streaming Service for Kafka log. + +Alauda Build of Kafka 2 is supported only on platform v4.0 and v4.1 (see Cause 6). + +### Cause 4: Kafka operator upgraded to 4.2 while instances still run in ZooKeeper mode + +```bash +kubectl get kafka -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,VERSION:.spec.kafka.version,METADATA:.status.kafkaMetadataState,OPERATOR:.status.operatorLastSuccessfulVersion +kubectl get csv -A | grep strimzi-kafka-operator +``` + +Any instance whose `METADATA` is `ZooKeeper` blocks the v4.2.x operator. The supported order is: keep the v4.1.x operator, migrate every instance to KRaft mode following "Migrate from ZooKeeper to KRaft mode", and only then upgrade to v4.2.x. + +When the operator has already been upgraded and the cluster upgrade is waiting on this mismatch, a possible workaround is to uninstall the Kafka operator temporarily so the cluster upgrade can continue. This workaround is not a verified procedure; prefer the supported order above whenever the upgrade can wait. + +> **Warning:** While the Kafka operator is uninstalled, nothing reconciles any Kafka instance on the cluster: configuration changes, stop and start, users, and topics are not applied, and a deleted or evicted broker pod is not recreated by the operator. Do it only when the cluster upgrade is blocked by this mismatch and at least one instance is still in ZooKeeper mode. Reinstall the operator version that matches the instances (v4.1.x for ZooKeeper-mode instances) from OperatorHub as soon as the cluster upgrade finishes; that reinstall is the rollback. + +Uninstall it from **OperatorHub** (Alauda Streaming Service for Kafka), or remove its Subscription and CSV: + +```bash +kubectl -n get subscription strimzi-kafka-operator -o yaml > strimzi-kafka-operator-sub.yaml +kubectl -n delete subscription strimzi-kafka-operator +kubectl -n delete csv +``` + +### Cause 5: Can the 4.3 Kafka Operator be used on a 4.1 platform? + +Only when both conditions of the official upgrade documentation are met. First, v4.2.x and later support only KRaft mode, so every instance must already run in KRaft mode; ZooKeeper-mode instances must be migrated with the v4.1.x operator first (Cause 4). Second, Kafka 4.2.0 requires Alauda Container Platform Data Services RDS Framework 4.2.2 or later. If the 4.3 operator is later removed in favor of a lower one, clean up its leftover resources as in Cause 2. + +### Cause 6: On 4.4.0 and later, are Data Services Essentials and RDS Framework still needed? + +Yes for the Kafka Operator: on v4.4.0 and later it depends on Alauda Container Platform Data Services Essentials and Alauda Container Platform Data Services RDS Framework, and both must be installed to support upgrades and troubleshooting. Kafka 2 Operator does not apply there: it is supported only on v4.0 and v4.1. ClickHouse Operator does not depend on these two plugins and can be deployed without them. + +```bash +kubectl get csv -A | grep -Ei 'kafka|rds-operator|clickhouse' +``` + +## Diagnostic Steps + +1. Record the platform version, the installed Kafka operator CSVs, and the Kafka artifacts: + + ```bash + kubectl get csv -A | grep -i kafka + kubectl get imageversion + kubectl -n get operatorcondition + ``` + + An `operatorcondition` with `CatalogSourcesUnhealthy` `Status: False` and `Reason: AllCatalogSourcesHealthy` is healthy; it does not explain a wrong version list. + +2. The create page offers a Kafka version of a different operator version, and two `kafka-artifact-*` objects exist: Cause 2. + +3. CSV `strimzi-kafka-operator.v4.1.0` on a platform 4.2 cluster with the component reported abnormal: Cause 1. + +4. A Kafka 2.x instance ignores console actions or keeps a broker down, the operator log says `skip reconcile ... kafka version is 2.`, and no Alauda Build of Kafka 2 CSV exists: Cause 3. Before treating a broker as offline, check its pod and the cluster state; all broker pods `1/1 Running` with the `Kafka` resource `Ready` does not by itself prove a broker failure. + +5. Operator at 4.2 or later, the instance not updated, and `status.kafkaMetadataState` is `ZooKeeper`: Cause 4. A direct platform hop from 4.0 to 4.2 with the operator stuck has the same KRaft requirement. + +6. Several middleware operators abnormal together, the operator crash-looping, or no Kafka menu at all: see the operator install article linked in the Issue section, or [Kafka instance creation times out on the entity-operator, shows failed, or the cluster operator restarts on ACP](Kafka_instance_creation_times_out_on_the_entity-operator_shows_failed_or_the_cluster_operator_restarts_on_ACP.md). diff --git a/docs/en/solutions/Kafka_user_secret_missing_user_stuck_in_Creating_ACL_not_supported_or_uppercase_topic_names_on_ACP.md b/docs/en/solutions/Kafka_user_secret_missing_user_stuck_in_Creating_ACL_not_supported_or_uppercase_topic_names_on_ACP.md new file mode 100644 index 000000000..d788e5988 --- /dev/null +++ b/docs/en/solutions/Kafka_user_secret_missing_user_stuck_in_Creating_ACL_not_supported_or_uppercase_topic_names_on_ACP.md @@ -0,0 +1,187 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda Application Services +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" +--- + +# Kafka user secret missing, user stuck in Creating, ACL not supported, or uppercase topic names on ACP + +## Issue + +A Kafka user was created from Data Services, but opening its secret (`保密字典`, Secret) says the resource does not exist. The user shows one of these: + +- `Simple authorization ACL rules are configured but not supported in the Kafka cluster configuration.` The same is often reported as "works on Kafka 2.8, fails on Kafka 2.7". +- `status.phase` stays `Creating` with `Events: `, on an instance that uses `SCRAM-SHA-512` and where user creation used to work. Other Kafka instances in the same cluster create users normally, and other recently created users of the same instance also have an empty `READY` column in `kubectl get kafkauser`. + +Also covered: client applications fail with a permission error after Kafka users and topics were migrated, and the topic creation form rejects uppercase letters in the topic name. + +A client whose user and secret exist but which gets `TopicAuthorizationException: Not authorized to access topics` needs the ACL and listener setup in [Kafka client connection failures: timeouts, TLS certificates, authorization, NodePort, and reverse DNS on ACP](Kafka_client_connection_failures_timeouts_TLS_certificates_authorization_NodePort_and_reverse_DNS_on_ACP.md). + +## Environment + +- A Data Services Kafka user is an `RdsKafkaUser` (`middleware.alauda.io/v1`, label `middleware.alauda.io/cluster: `). The operator creates a Strimzi `KafkaUser` (`kafka.strimzi.io/v1beta2`, label `strimzi.io/cluster: `) with the same name and spec. +- The `KafkaUser` is reconciled by the `user-operator` container of the Deployment `-entity-operator`, which also runs `topic-operator` and `tls-sidecar`. +- The password of a SCRAM user is kept in a Secret named `-password`, which the `RdsKafkaUser` references. + +## Root Cause + +### Cause 1: Simple authorization ACL rules are configured but not supported in the Kafka cluster configuration + +The user carries `authorization.type: simple` with ACL rules, but the instance was created without an authentication method, so authorization is not enabled in the cluster configuration. The user is not provisioned and its secret is never created. The Kafka version is not the cause: a Kafka 2.8 instance created the same way shows the same error. To use ACLs, `authorization.type: simple` must be enabled on the instance before topic or group ACLs are configured. + +### Cause 2: A user with the same name already exists + +The name entered in the Data Services form is already taken, for example by a `KafkaUser` created earlier directly through the Kafka Operator. Creating a user with the same name from Data Services fails. + +### Cause 3: The user-operator is hung while its Deployment still reports Ready + +The `user-operator` container of `-entity-operator` stops reconciling users, while the Deployment still shows `READY 1/1` and the pod is Running. Its log repeats: + +```text +WARNING: A connection to https:/// was leaked. Did you forget to close a response body? +WARN WatcherWebSocketListener:76 - Exec Failure java.io.FileNotFoundException /home/strimzi/.kube/config (No such file or directory) +``` + +Every new user of that instance stays in `Creating` and gets no secret. Authorization being disabled and a duplicate name are ruled out. Restarting the entity operator restores user creation. + +### Cause 4: Client permission error after migrating users and topics: the `-password` secret is missing + +The `-password` secret that the `RdsKafkaUser` references was not migrated, so client authentication fails. + +### Cause 5: Can a Kafka topic name contain uppercase letters? + +Yes. Kafka topic names allow uppercase letters, and so does the topic resource; only the creation form rejects them. + +## Resolution + +### Cause 1: Simple authorization ACL rules are configured but not supported in the Kafka cluster configuration + +```bash +kubectl -n get kafkauser -o jsonpath='{.spec.authorization}{"\n"}{.status.conditions}{"\n"}' +kubectl -n get kafka -o jsonpath='{.spec.kafka.authorization}{"\n"}{.spec.kafka.listeners}{"\n"}' +``` + +An empty `spec.kafka.authorization` together with the error confirms this cause. Choose one fix: + +- If the user does not need ACLs, remove `spec.authorization` from the `RdsKafkaUser`. The user is then provisioned and its secret appears. +- If ACLs are required, enable authorization on the instance through the platform, so that the cluster runs with `authorization.type: simple` and an authentication method on its listeners, and then configure the topic and group ACLs. + +Edit the `RdsKafkaUser`, not the `KafkaUser`: the operator writes the `RdsKafkaUser` spec over the `KafkaUser`. + +```bash +kubectl -n edit rdskafkauser +kubectl -n get rdskafkauser,kafkauser +``` + +### Cause 2: A user with the same name already exists + +```bash +kubectl -n get kafkauser,rdskafkauser +kubectl -n get kafkauser -o jsonpath='{.metadata.creationTimestamp}{"\n"}' +``` + +A `KafkaUser` older than the Data Services request was created earlier, outside this request. Create the Data Services user with a name that is not used in that namespace. + +### Cause 3: The user-operator is hung while its Deployment still reports Ready + +1. Rule out Cause 1 and Cause 2 with the commands above, then confirm the signature: + + ```bash + kubectl -n describe rdskafkauser + kubectl -n logs deploy/-entity-operator -c user-operator --tail=300 | grep -E 'Exec Failure|was leaked' + kubectl -n get deploy -entity-operator + ``` + + Repeated `Exec Failure java.io.FileNotFoundException /home/strimzi/.kube/config` lines, with the Deployment at `1/1`, confirm this cause. A `READY 1/1` Deployment does not prove the user-operator works. + +2. Restart the entity operator. + +> **Warning:** Restarting the entity operator restarts the `topic-operator`, `user-operator`, and `tls-sidecar` containers; topic and user reconciliation pauses until the new pod is Ready. Brokers are not restarted and clients are not affected. Do it only when users stay in `Creating` and the `user-operator` log shows the `Exec Failure ... /home/strimzi/.kube/config` lines. No rollback is needed; the Deployment recreates the pod with the same spec. + + ```bash + kubectl -n rollout restart deploy/-entity-operator + kubectl -n rollout status deploy/-entity-operator + ``` + +3. Check every user of the instance, not only the one reported. Each user that was stuck in `Creating` should become `Active`, and its secret should open: + + ```bash + kubectl -n get rdskafkauser + kubectl -n get kafkauser + kubectl -n get secret | grep + ``` + +If the `Exec Failure` lines come back later, collect the user-operator log and the operator version before restarting again. + +### Cause 4: Client permission error after migrating users and topics: the `-password` secret is missing + +```bash +kubectl -n get rdskafkauser -o jsonpath='{.spec.authentication}{"\n"}' +kubectl -n get secret -password +``` + +`Error from server (NotFound)` for `-password` confirms this cause. Reset the password of the user on its Data Services user page; the reset recreates the `-password` secret and restores access. + +> **Warning:** A reset gives the user a new password. Every client that authenticates as this user must be updated, and clients that keep the old password fail. Do it only when the `-password` secret of that user is missing. The old password cannot be restored, so plan the client updates first. + +```bash +kubectl -n get secret -password +kubectl -n get secret -o jsonpath='{.data.sasl\.jaas\.config}' | base64 -d +``` + +When users are migrated again, migrate each `-password` secret together with its `RdsKafkaUser`. + +### Cause 5: Can a Kafka topic name contain uppercase letters? + +Create the topic from YAML instead of the form. A Kubernetes object name cannot contain uppercase letters, so keep `metadata.name` lowercase and put the real topic name in `spec.topicName`. Start from an existing topic of the same instance so labels and fields match your operator version: + +```bash +kubectl -n get rdstopic -o yaml > new-topic.yaml +``` + +In `new-topic.yaml`, delete `metadata.resourceVersion`, `metadata.uid`, `metadata.creationTimestamp`, `metadata.generation`, `metadata.managedFields`, and the `status` block, then change only the two names: + +```yaml +metadata: + name: +spec: + topicName: +``` + +```bash +kubectl -n create -f new-topic.yaml +kubectl -n get rdstopic -o jsonpath='{.spec.topicName}{"\n"}' +``` + +Do not reuse a `spec.topicName` that another topic resource already uses; on some versions deleting the duplicate resource can clear the existing topic (see [Avoid Kafka Topic Data Loss When Deleting an Abnormal Topic CR](ecosystem/kafka/Kafka_Topic_Delete_Data_Loss_Issue.md)). + +## Diagnostic Steps + +1. Read the user objects: + + ```bash + kubectl -n get rdskafkauser -o yaml + kubectl -n get kafkauser -o yaml + kubectl -n get secret | grep + ``` + +2. Match the signature: + + - `Simple authorization ACL rules are configured but not supported in the Kafka cluster configuration.`: Cause 1. + - A `KafkaUser` with that name existed before the Data Services request: Cause 2. + - `phase: Creating` with no events, `spec.kafka.authorization` is `{"type":"simple"}`, no older user of that name, and `Exec Failure java.io.FileNotFoundException /home/strimzi/.kube/config` in the `user-operator` log: Cause 3. + - The user is `Active` but `-password` is missing after a migration: Cause 4. + - The user and its secrets exist and the client gets `TopicAuthorizationException`: the client connection article linked above. + +3. Check that the entity operator is running at all. A Deployment that is not Ready, or whose memory keeps rising, is covered in [Kafka instance creation times out on the entity-operator, shows failed, or the cluster operator restarts on ACP](Kafka_instance_creation_times_out_on_the_entity-operator_shows_failed_or_the_cluster_operator_restarts_on_ACP.md). + + ```bash + kubectl -n get deploy -entity-operator + kubectl -n top pod --containers | grep entity-operator + ``` diff --git a/docs/en/solutions/No_Kafka_in_Data_Services_or_Alauda_Build_of_Kafka_2_unpublished_or_failing_to_deploy_on_ACP.md b/docs/en/solutions/No_Kafka_in_Data_Services_or_Alauda_Build_of_Kafka_2_unpublished_or_failing_to_deploy_on_ACP.md new file mode 100644 index 000000000..f4b3bb745 --- /dev/null +++ b/docs/en/solutions/No_Kafka_in_Data_Services_or_Alauda_Build_of_Kafka_2_unpublished_or_failing_to_deploy_on_ACP.md @@ -0,0 +1,175 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda Application Services +ProductsVersion: + - "3.6" + - "3.12" + - "3.14" + - "3.16" + - "3.18" + - "4.0" + - "4.1" + - "4.2" +--- + +# No Kafka in Data Services, or Alauda Build of Kafka 2 unpublished or failing to deploy on ACP + +## Issue + +Kafka cannot be offered to business namespaces because its operator is missing or unhealthy: + +- The Data Services menu lists other services (for example MySQL, Redis, PostgreSQL) but no Kafka, so there is no **Create Kafka Instance** button. `kubectl get po -A | grep kafka` lists only `cpaas-kafka-0/1/2` in `cpaas-system`. Often the request started as "connect a Kafka web UI such as kafka-ui to the platform Kafka". +- OperatorHub shows Alauda Build of Kafka 2 as not published (`未上架`), or its install fails on ACP 4.1. +- After the global cluster was upgraded (for example from 3.18 to 4.0.12), a business cluster no longer lists MySQL or Kafka, while they still appear on the global cluster. +- Kafka, MySQL, and RabbitMQ operators all show abnormal at the same time. + +Related articles: [Kafka instance creation times out on the entity-operator, shows failed, or the cluster operator restarts on ACP](Kafka_instance_creation_times_out_on_the_entity-operator_shows_failed_or_the_cluster_operator_restarts_on_ACP.md) covers create failures once the operator is installed. A Kafka 2.x instance that the installed operator skips is covered in [Kafka operator mismatch: wrong create page version, Kafka 2.x instance skipped, or ZooKeeper mode blocks upgrade on ACP](Kafka_operator_mismatch_wrong_create_page_version_Kafka_2x_instance_skipped_or_ZooKeeper_mode_blocks_upgrade_on_ACP.md). + +## Root Cause + +### Cause 1: Data Services has no Kafka menu because the Kafka operator is not deployed + +The `cpaas-kafka-*` pods in `cpaas-system` are the platform's own Kafka, used by platform components, not a business instance. The Data Services Kafka feature switch is off and the Kafka operator (Alauda Streaming Service for Kafka) is not installed, so Data Services offers no Kafka. A web UI or business traffic should use a Data Services Kafka instance; the platform Kafka is not reconfigured for that. + +### Cause 2: Alauda Build of Kafka 2 fails to deploy on ACP 4.1 because Kafka Operator 3.x earlier than 4.2 is not deployed first + +On ACP 4.1 (for example 4.1.3), installing Alauda Build of Kafka 2 fails at the operator deployment. On platform 4.1, Kafka Operator 3.x must be deployed before Kafka Operator 2.x, and that 3.x operator must be a version earlier than 4.2: Kafka Operator 3.x 4.2 and later no longer supports Kafka 2.x. + +### Cause 3: The Alauda Build of Kafka 2 package has not been pushed with the matching violet + +OperatorHub shows Alauda Build of Kafka 2 (for example `4.0.3` on a 4.0.9 cluster) as not published, and no `kafka-operator-2x` `artifactversion` exists yet. The operator package has to be uploaded with `violet push`, using the `violet` version that matches the package rather than simply the `violet` that came with the platform. + +### Cause 4: The Kafka 2 bundle artifact name does not match OperatorHub + +The package is uploaded, `artifactversion` for `kafka-operator-2x` is `Present` and `Success`, and `packagemanifest` `strimzi-kafka-operator-2x` exists, but the marketplace still shows Alauda Build of Kafka 2 as not published. OperatorHub does not accept an artifact named `kafka-operator-2x`; the name needs the `operatorhub-` prefix. + +### Cause 5: Global and business clusters are on incompatible versions + +The global cluster is 4.0.12 after an upgrade from 3.18 (Data Services Essentials `v4.0.23` on the global side) while the business cluster is still on the previous version. MySQL and Kafka disappear from the business cluster view. Errors in other products (for example service mesh) in the same window come from the same version skew. + +### Cause 6: MySQL (MGR) operator 4.4.4 with MySQL 8.0 makes several middleware operators abnormal + +Kafka, MySQL, and RabbitMQ operators show abnormal together. MGR Operator 4.4.4 does not support MySQL 8.0. The Kafka operator status clears after the MySQL operator and Data Services Essentials are moved back to the 4.3 line, not after a Kafka restart. + +## Resolution + +### Cause 1: Data Services has no Kafka menu because the Kafka operator is not deployed + +```bash +kubectl get po -A | grep -i kafka +kubectl get csv -A | grep -i kafka +kubectl get packagemanifest -A | grep -i kafka +``` + +If the only Kafka pods are `cpaas-kafka-*` in `cpaas-system` and no Kafka operator CSV exists: + +1. Enable the Kafka feature switch for Data Services. +2. Install Alauda Streaming Service for Kafka from **OperatorHub**: target cluster, channel `stable`, namespace `kafka-system`, and the operator version that matches your platform (for example `4.1.0`). On clusters that still need Kafka 2.x, respect the version limit in Cause 2. +3. Confirm the operator is running, then reload Data Services. The Kafka menu and **Create Kafka Instance** appear. + + ```bash + kubectl -n kafka-system get pods + kubectl get csv -A | grep -i kafka + ``` + +4. Create the business instance. For an SSL client such as kafka-ui, the create form offers in-cluster access with link encryption (Plain/SCRAM-SHA-512, or TLS with SCRAM-SHA-512 or TLS authentication) and out-of-cluster `NodePort` access with TLS. Deploy the community web UI separately and point it at the instance. Client certificates and users are covered in [Kafka client connection failures: timeouts, TLS certificates, authorization, NodePort, and reverse DNS on ACP](Kafka_client_connection_failures_timeouts_TLS_certificates_authorization_NodePort_and_reverse_DNS_on_ACP.md). + +### Cause 2: Alauda Build of Kafka 2 fails to deploy on ACP 4.1 because Kafka Operator 3.x earlier than 4.2 is not deployed first + +```bash +kubectl get packagemanifest,artifact,artifactversion -A | grep kafka +kubectl get csv -A | grep -i kafka +``` + +If `artifactversion` `kafka-operator-2x.*` is `Present` and `Success` and the only problem is that the marketplace shows the package as not published, use Cause 4. If the Kafka Operator 2.x deployment fails and no Kafka Operator 3.x is installed, or the installed 3.x is 4.2 or later: + +1. Deploy Kafka Operator 3.x at a version earlier than 4.2 from the marketplace. +2. Deploy Kafka Operator 2.x (Alauda Build of Kafka 2) after the 3.x operator is running. +3. Confirm both CSVs are `Succeeded` with `kubectl get csv -A | grep -i kafka`. + +Do not upgrade the 3.x operator to 4.2 or later on a cluster that still needs Kafka 2.x. + +### Cause 3: The Alauda Build of Kafka 2 package has not been pushed with the matching violet + +1. Confirm the package objects are missing: + + ```bash + kubectl get packagemanifest,artifact,artifactversion -A | grep kafka + ``` + +2. Push the Alauda Build of Kafka 2 package with the `violet` version that matches it (check with `violet version`), choosing a package version that supports the platform and the Kafka version (`v4.0.3` supports platform 4.0 and 4.1 with Kafka 2.5, 2.7, and 2.8): + + ```bash + violet push .tgz \ + --platform-address "https://" \ + --platform-username "" \ + --platform-password "" \ + --clusters "" + ``` + +3. Publish it and install it from **OperatorHub** (channel `stable`). If `artifactversion` is now `Present`/`Success` and the package is still not published, continue with Cause 4. + +### Cause 4: The Kafka 2 bundle artifact name does not match OperatorHub + +> **Warning:** Deleting the `artifactversion` and the old `artifact` removes the objects OperatorHub is currently ignoring. Do it only when `kafka-operator-2x` is `Present` and `Success` but Alauda Build of Kafka 2 still shows as not published, and only after both objects are saved to YAML. Roll back by re-applying the backup files. + +```bash +kubectl get packagemanifest,artifact,artifactversion -A | grep kafka +kubectl -n cpaas-system get artifactversion -o yaml > artifactversion.bak.yaml +kubectl -n cpaas-system get artifact kafka-operator-2x -o yaml > artifact.bak.yaml +kubectl -n cpaas-system delete artifactversion +``` + +Use the `artifactversion` name in your cluster (for the 4.0.3 bundle it is `kafka-operator-2x.v4.0.3`). Copy `artifact.bak.yaml` to `operatorhub-kafka-operator-2x.yaml` and edit it: + +- Remove `resourceVersion`, `uid`, `creationTimestamp`, `generation`, `managedFields`, and the whole `status` block. +- Set `metadata.name` to `operatorhub-kafka-operator-2x`. +- Set the `spec.artifactVersionSelector.matchLabels` key `cpaas.io/artifact-version` to `operatorhub-kafka-operator-2x`. +- Keep the existing `registry`, `repository`, and `type: bundle` values. + +```bash +kubectl -n cpaas-system apply -f operatorhub-kafka-operator-2x.yaml +kubectl -n cpaas-system delete artifact kafka-operator-2x +``` + +Publish the bundle again the same way it was first uploaded. Confirm that `kubectl -n cpaas-system get artifact operatorhub-kafka-operator-2x -o yaml` reports a synced `status` and that OperatorHub lists the Kafka 2 operator as available. + +### Cause 5: Global and business clusters are on incompatible versions + +Upgrade the business cluster, and the middleware components on it, to versions compatible with the global cluster. No configuration change brings the menus back while the business cluster stays on the old version. Data Services Essentials `v4.0.23` is the global version in this situation, not a downgrade target. + +### Cause 6: MySQL (MGR) operator 4.4.4 with MySQL 8.0 makes several middleware operators abnormal + +> **Warning:** Moving the MySQL operator to 4.3.2 and Data Services Essentials to 4.3.3 downgrades those two components and restarts their workloads. Do it only when Kafka, MySQL, and RabbitMQ operators went abnormal together and the installed MGR or MySQL operator is 4.4.4 with MySQL 8.0. Returning to 4.4.4 brings the incompatibility back. + +Install MySQL operator 4.3.2 and Data Services Essentials 4.3.3 through the same marketplace path used for 4.4.4. The Kafka operator status recovers with them. + +```bash +kubectl get csv -A | grep -Ei 'mysql|kafka|rabbit' +``` + +## Diagnostic Steps + +1. Is any business Kafka operator installed? + + ```bash + kubectl get csv -A | grep -i kafka + kubectl get po -A | grep -i kafka + ``` + + No Kafka operator CSV and only `cpaas-kafka-*` pods in `cpaas-system`: Cause 1. + +2. Marketplace state of the Kafka 2 package: + + ```bash + kubectl get packagemanifest,artifact,artifactversion -A | grep kafka + ``` + + No `kafka-operator-2x` objects at all: Cause 3. `Present` plus `Success` with a name that lacks `operatorhub-`: Cause 4. The package is published but its deployment fails on 4.1: Cause 2. + +3. Menus missing only on business clusters right after a global upgrade: compare the cluster versions on the cluster list. Different minor versions: Cause 5. + +4. Several middleware operators abnormal on the same day as a MySQL operator change: compare the CSV versions with Cause 6 before restarting Kafka. diff --git a/docs/en/solutions/Node_NotReady_with_containerd_not_running_or_cni_plugin_not_initialized_on_ACP.md b/docs/en/solutions/Node_NotReady_with_containerd_not_running_or_cni_plugin_not_initialized_on_ACP.md new file mode 100644 index 000000000..a6f1d2bc3 --- /dev/null +++ b/docs/en/solutions/Node_NotReady_with_containerd_not_running_or_cni_plugin_not_initialized_on_ACP.md @@ -0,0 +1,143 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.x + - 4.x +--- + +# Node NotReady with containerd not running or "cni plugin not initialized" on ACP + +## Issue + +Typical reports: + +- After the cluster was restarted, some nodes stay `NotReady` and cannot be connected. +- In a newly deployed environment, some masters of a workload cluster are `NotReady`; `kubectl describe node` says the CNI is broken, but the OVN containers are Running: + + ```text + Ready False KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized + ``` + +- During an upgrade a node becomes `NotReady` and platform Pods on it (for example `zeus` in `cpaas-system`) fail to start, while `calico-node` is running and logs no errors. The kubelet log repeats: + + ```text + "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" + ``` + +- After a GPU node was rebooted, many Pods on it cannot start, and the containerd log shows the `loopback` CNI plugin crashing. + +The kubelet is running and reports to the API server, but it reports the node as not ready because the container runtime is missing or has not loaded the CNI configuration. Memory, disk, and PID pressure conditions are normal, and the other nodes of the cluster are `Ready`. + +If the runtime runs but hangs (`PLEG is not healthy`, `container runtime is down`), see [Node NotReady on ACP with kubelet "PLEG is not healthy"](Node_NotReady_with_PLEG_is_not_healthy.md). If the kubelet cannot reach the API server at all, see [Node NotReady caused by network, storage or host-level faults on ACP](Node_NotReady_caused_by_network_storage_or_host_level_faults_on_ACP.md). CoreDNS problems right after a node restart are covered in [CoreDNS fails after a node restart or lookups time out when a CoreDNS node is shut down on ACP](CoreDNS_fails_after_a_node_restart_or_lookups_time_out_when_a_CoreDNS_node_is_shut_down_on_ACP.md). + +## Root Cause + +### Cause 1: containerd did not start after a node or cluster restart + +After a cluster or host restart, `systemctl status containerd` on the `NotReady` nodes shows the service `inactive (dead)` or `failed`, and `crictl ps` cannot connect to the runtime socket. Without containerd the kubelet cannot run any Pod, so the node cannot become `Ready`. Why containerd did not start is not known; starting it by hand recovers the node. + +### Cause 2: `cni plugin not initialized` while the kube-ovn CNI Pod is Running + +The node condition shows `NetworkPluginNotReady ... cni plugin not initialized`. At the same time `NetworkUnavailable` is `False` with reason `JoinSubnetGatewayReachable`; the `kube-ovn-cni-` Pod in `kube-system` on that node has its `cni-server` container `Running` with 0 restarts and its init containers `install-cni` and `hostpath-init` completed; and containerd is running. This happened on two of three control-plane nodes of a newly deployed 4.2 workload cluster (Kubernetes `v1.33.6`). containerd had not picked up the CNI configuration that kube-ovn installs, although both looked healthy. Restarting containerd makes the nodes `Ready`; the underlying cause is not confirmed. + +### Cause 3: A corrupted plugin binary in `/opt/cni/bin` (`portmap`, `loopback`) + +A Pod network is set up by running every plugin in the CNI chain, so one damaged binary breaks it even when the main network plugin is healthy. Two forms: + +- `network plugin is not ready: cni config uninitialized` on a Docker-runtime node with Calico during an upgrade. All Calico Pods are Running, the `calico-node` log on the node has no errors, and the Calico configuration file and binaries are present. Restarting `calico-node` and the kubelet does not help. The kubelet also logs `"Error syncing pod, skipping"` for `cpaas-system/zeus-*` Pods and `KillPodSandbox` failures with `networkPlugin cni failed to teardown pod ... network: cni config uninitialized`. `ls -l /opt/cni/bin/` shows `portmap` at 1572864 bytes on the affected node and about 3.6 MB on a healthy node, while the other binaries match. +- Pods cannot start after a reboot of a (GPU) node, and the kubelet and containerd logs show the `loopback` plugin binary crashing during Pod network setup. The `md5sum` of `/opt/cni/bin/loopback` differs from every healthy node. + +How the binaries were damaged is not known: around an upgrade for `portmap`, around a reboot for `loopback`. + +## Resolution + +### Cause 1: containerd did not start after a node or cluster restart + +```bash +systemctl start containerd +systemctl status containerd --no-pager | head -n 5 +systemctl is-enabled containerd +kubectl get node +``` + +The node should return to `Ready`. If `systemctl is-enabled containerd` does not print `enabled`, compare it with a healthy node of the same cluster and enable it the same way (`systemctl enable containerd`); otherwise the node stays `NotReady` again after the next reboot. If containerd fails to start, read `journalctl -u containerd -b --no-pager | tail -n 50` for the reason. + +### Cause 2: `cni plugin not initialized` while the kube-ovn CNI Pod is Running + +Restart containerd on each affected node, one node at a time. + +> **Warning:** Restarting containerd briefly interrupts container operations on the node: for a few seconds no Pods are created, stopped, or probed. Running containers keep running in their shims. Do it only when the node condition shows `cni plugin not initialized` and the `kube-ovn-cni` Pod on that node is Running (step 2 of Diagnostic Steps). No rollback is needed. + +```bash +systemctl restart containerd +systemctl status containerd --no-pager | head -n 5 +kubectl get node +kubectl describe node | grep -E 'Ready|NetworkPluginNotReady' +``` + +If the condition stays, check that the CNI configuration and binaries exist on the node (step 3 of Diagnostic Steps) and compare them with a `Ready` node. + +### Cause 3: A corrupted plugin binary in `/opt/cni/bin` (`portmap`, `loopback`) + +Replace the damaged binary with the copy from a healthy node of the same cluster. The node returns to `Ready` (`portmap`), or the Pods start normally (`loopback`). + +> **Warning:** A binary from a node with a different architecture or CNI version breaks Pod networking on this node. Do it only when the binary's size or checksum differs from healthy nodes (step 4 of Diagnostic Steps), and take the copy from a healthy node of the same cluster, OS, and architecture. Keep the old file to roll back. + +```bash +cp -a /opt/cni/bin/portmap /root/portmap.corrupt.$(date +%s) +scp :/opt/cni/bin/portmap /opt/cni/bin/portmap +chmod 755 /opt/cni/bin/portmap +md5sum /opt/cni/bin/portmap # must match the healthy node +kubectl get node +``` + +Use the same steps for `loopback` or any other binary that differs. Compare all binaries at once, because more than one can be damaged. + +## Diagnostic Steps + +1. Read the node condition and the runtime state: + + ```bash + kubectl get nodes -o wide + kubectl describe node | sed -n '/Conditions:/,/Addresses:/p' + systemctl status containerd kubelet --no-pager | grep -E '●|Active' + journalctl -u kubelet --since "-30min" --no-pager | grep -iE 'NetworkPluginNotReady|cni|runtime' | tail -n 20 + ``` + + `containerd` not active: Cause 1. `cni plugin not initialized` with containerd active: Cause 2. `cni config uninitialized` with a healthy Calico, or a plugin binary crashing: Cause 3. + +2. Check the kube-ovn CNI Pod on the node: + + ```bash + kubectl -n kube-system get pods -o wide --field-selector spec.nodeName= | grep kube-ovn-cni + crictl ps -a | grep -E 'cni-server|install-cni|hostpath-init' + ``` + + Expected for Cause 2: `cni-server` Running, `install-cni` and `hostpath-init` Exited (completed). + +3. Check the CNI configuration and plugin binaries containerd loads, and what it reports about loading them: + + ```bash + ls -l /etc/cni/net.d/ /opt/cni/bin/ + crictl info | grep -iE -A3 'lastCNILoadStatus|NetworkReady' + journalctl -u containerd --since "-30min" --no-pager | grep -i cni | tail -n 20 + ``` + + Compare the file lists with a `Ready` node of the same cluster. + +4. Compare every CNI plugin binary with a healthy node (Cause 3). A size or checksum that differs is the damaged one: + + ```bash + md5sum /opt/cni/bin/* > /tmp/cni-$(hostname).md5 + ssh 'md5sum /opt/cni/bin/*' > /tmp/cni-healthy.md5 + diff /tmp/cni-healthy.md5 /tmp/cni-$(hostname).md5 + grep -o '"type": *"[a-z0-9-]*"' /etc/cni/net.d/*.conflist # plugins in the chain + journalctl -u kubelet --since "-30min" --no-pager | grep -E 'cni config uninitialized|loopback|portmap' | tail -n 10 + ``` + + No `diff` output means the binaries match the healthy node. + +5. Runtime binaries that were replaced or conflict (Docker on a containerd node, overwritten `runc`) are covered in [Container runtime conflicts on ACP nodes: Docker, podman, docker-runc, and overwritten runtime binaries](Container_runtime_conflicts_on_ACP_nodes_Docker_podman_docker-runc_and_overwritten_runtime_binaries.md). diff --git a/docs/en/solutions/Node_runtime_or_kubelet_directories_return_Input_output_error_or_df_hangs_on_ACP.md b/docs/en/solutions/Node_runtime_or_kubelet_directories_return_Input_output_error_or_df_hangs_on_ACP.md new file mode 100644 index 000000000..34d955860 --- /dev/null +++ b/docs/en/solutions/Node_runtime_or_kubelet_directories_return_Input_output_error_or_df_hangs_on_ACP.md @@ -0,0 +1,173 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.x + - 4.x +--- + +# Node runtime or kubelet directories return Input/output error, or df hangs, on ACP + +## Issue + +Typical reports: + +- All services scheduled to one node are abnormal, cannot be deleted, and cannot be exec'd into, and it comes back after the node is rebooted. +- `ls` under `/var/lib` prints `ls: cannot access 'containerd': Input/output error` (Chinese locale: `ls: 无法访问 'containerd': 输入/输出错误`), and the entry is listed as `d?????????`. The same can happen for `/cpaas`, which holds platform component data. +- The kubelet on one worker will not start: `failed to run Kubelet: unable to determine image API version: rpc error ... /var/lib/docker/image: input/output error`. +- After a storage failure the platform cannot select clusters or namespaces, and kube-apiserver on a master refuses connections on `6443`. +- The platform Grafana page returns `login grafana with unexpected response` (code 500) while some nodes are `NotReady` with disk problems. +- A container keeps failing with `Back-off restarting failed container`, although the same image works elsewhere. +- The platform suddenly raises many alerts, all about DaemonSets, and `df -h` on one node hangs. + +These are faults in the host disks, below Kubernetes. Related article: [Node unstable from a GPU NVRM log flood, a faulty memory module, or a read-only cpuset cgroup on ACP](Node_unstable_from_a_GPU_NVRM_log_flood_a_faulty_memory_module_or_a_read-only_cpuset_cgroup_on_ACP.md) covers GPU, memory, and cgroup faults. Storage that stalls rather than fails is covered in [Node NotReady caused by network, storage or host-level faults on ACP](Node_NotReady_caused_by_network_storage_or_host_level_faults_on_ACP.md) and [Node NotReady with high load average and D-state processes on ACP](NotReady_Node_with_High_Load_and_Many_D_State_Processes.md). + +## Root Cause + +### Cause 1: I/O errors on the disk that holds the runtime, kubelet, or platform data + +`ls /var/lib/containerd` (or `/var/lib/docker`, `/var/lib/kubelet`, `/cpaas`) returns `Input/output error`. The data disk under that directory is damaged and can no longer be read or written reliably; restarts and Pod evictions only move the symptom. Stopping containerd and cleaning up its processes helps for a while, but the fault returns. + +Errors on the node: + +```text +ls: cannot access 'containerd': Input/output error +d????????? ? ? ? ? ? containerd +FATA[0000] execing command in container: ... failed to load task: no running task found: task not found: not found +Fail to write "stderr" log to log file "/var/log/pods///2.log" error="write ...: input/output error" +"command failed" err="failed to run Kubelet: unable to determine image API version: rpc error ... /var/lib/docker/image: input/output error" +FATA[0000] failed to try resolving symlinks in path "/var/log/pods/kube-system_kube-apiserver-_/kube-apiserver/1.log": lstat /var/lib/containerd: input/output error +``` + +Kernel messages with them: + +```text +VFS: Open an exclusive opened block device for write vdb [ sgdisk]. +blk_update_request: I/O error, dev sdb, sector +sd 0:0:2:0: timing out command, waited 180s +XFS (dm-1): metadata I/O error in "xlog_iodone" ... error 5 +XFS (dm-1): Log I/O Error Detected. Shutting down filesystem +xfs_imap_to_bp: xfs_trans_read_buf() returned error -5 +``` + +Variants: + +- **containerd data disk on a worker.** Every Pod on the node is abnormal, cannot be deleted, and `crictl exec` fails with `no running task found`. When `/var/log/pods` is a symlink into `/var/lib/containerd`, container log writes fail too. The kernel can repeat `VFS: Open an exclusive opened block device for write [ sgdisk]` about every 10 seconds, meaning an `sgdisk` process keeps opening the in-use disk for writing; what starts it is not known. CNI teardown errors such as `dial unix /run/openvswitch/kube-ovn-daemon.sock: connect: connection refused` at the same time are side effects. +- **kubelet will not start on a Docker worker.** `systemctl status kubelet` shows `activating (auto-restart) (Result: exit-code)` with `status=1/FAILURE`. `mount | grep sdb` returns nothing because the disk is used through LVM; `lsblk` shows the `sdb` → `dm-*` → mount point chain. +- **`/var/lib/containerd` on a master.** kube-apiserver cannot run, `curl -k https://:6443` returns `Connection refused`, and the platform cannot select the cluster or its namespaces. +- **Nodes `NotReady` with faulty disks.** Platform Pods on them crash-loop, for example `kube-prometheus-grafana-*` in `cpaas-system`, which makes the Grafana page return `login grafana with unexpected response`. +- **Several directories on several nodes.** `df -h /var/lib/containerd` returns `Input/output error`, and `dmesg` shows command timeouts, `I/O error, dev sdc`, then `xfs_do_force_shutdown(0x2)`. An XFS filesystem in that state stays shut down until it is unmounted and mounted again, even after the disk is readable. +- **Formatted or broken data disks.** A custom container that mounts a file-server path keeps failing with `BackOff` because some nodes had a data disk formatted and another node had a broken disk. + +### Cause 2: The root filesystem hangs + +On the node, `df -h` hangs, and `strace df -h` shows it stuck on one mount point; `cd /; ls` hangs too. DaemonSet Pods on that node fail to start, so the platform raises a burst of DaemonSet alerts at once. The root disk mount is faulty and every process that touches it blocks; `df` can be stuck on a directory where no disk is actually mounted. A reboot restores the node. + +## Resolution + +### Cause 1: I/O errors on the disk that holds the runtime, kubelet, or platform data + +1. Move workloads off the node with `kubectl cordon ` and `kubectl drain --ignore-daemonsets --delete-emptydir-data`. +2. Give the disk errors (and any repeating `sgdisk` line with its process) to the IaaS or hardware owner to repair or replace the disk. Containers on the node cannot run normally until the disk is fixed. Stopping the runtime and cleaning up its processes is only temporary relief. +3. If the disk was replaced with an empty one, or `lsblk -f` shows a disk was formatted (its data is gone), rebuild the node data directories as in [Node capacity, pod limit or data disk after resizing or replacing node hardware on ACP](Node_capacity_pod_limit_or_data_disk_after_resizing_or_replacing_node_hardware_on_ACP.md). +4. If the same disks were repaired, cleaned, and re-added, remount the affected directories so a shut-down XFS filesystem comes back. The order below is general Linux and Kubernetes practice. containerd normally uses `KillMode=process`, so stopping it leaves `containerd-shim` processes and their containers running and still holding the mounts; stop the Pods while containerd still runs. + +> **Warning:** `crictl stopp` stops every Pod sandbox on the node, including DaemonSet and static Pods; on a control-plane node that takes its kube-apiserver and etcd member offline until the kubelet starts again. Do it only when the infrastructure side confirms the disk is readable again (no new `I/O error` lines in `dmesg`), the node is drained, and, on a master, the other masters are healthy and etcd is backed up. If the mount fails, keep the services stopped and return the disk to its owner. + +```bash +systemctl show containerd -p KillMode +systemctl stop kubelet +crictl stopp $(crictl pods -q) +systemctl stop containerd +ps -eo pid,cmd | grep -E 'containerd-shim' | grep -v grep # must print nothing +umount /var/lib/containerd /cpaas +``` + +If `umount` reports the target is busy, find and stop what still uses it: + +```bash +fuser -vm /var/lib/containerd /cpaas +lsof +f -- /var/lib/containerd +``` + +Then mount and start again: + +```bash +mount -a +findmnt /var/lib/containerd; findmnt /cpaas +ls /var/lib/containerd > /dev/null && echo readable +systemctl start containerd kubelet +kubectl uncordon +``` + +5. For a master whose `/var/lib/containerd` returns I/O errors, reboot it after the storage owner has repaired the underlying storage. + +> **Warning:** Rebooting a control-plane node takes its kube-apiserver and etcd member offline until it is back. Do it only when `ls /var/lib/containerd` on that master returns `Input/output error`, the storage owner has confirmed the repair, and the other masters are healthy. A reboot cannot be undone. + +```bash +systemctl reboot +# afterwards, from another master: +curl -k --max-time 5 https://:6443/healthz +kubectl -n kube-system get pods -l component=kube-apiserver -o wide +``` + +After the host-side repair, platform Pods such as Grafana return to normal on their own; check them with `kubectl -n cpaas-system get pods -o wide | grep `. + +### Cause 2: The root filesystem hangs + +> **Warning:** A forced reboot kills every Pod on the node, and blocked filesystem writes may be lost. Do it only when `df` or `ls` on `/` hang and `strace` confirms the process is blocked on the filesystem. A reboot cannot be undone. + +```bash +systemctl reboot -f # use the IaaS console reset if the shell cannot run it +``` + +After the reboot, check for a leftover taint that keeps Pods `Pending` (for example one `alb` replica). Remove only a taint that is not one of the automatic `node.kubernetes.io/*` taints and is not part of your node-role design: + +```bash +kubectl describe node | grep -A3 Taints +kubectl taint nodes :- +``` + +Have the IaaS owner check the root disk and its storage path so the hang does not recur. + +## Diagnostic Steps + +1. Disk errors and the device behind the data directories (Cause 1): + + ```bash + dmesg -T | grep -iE 'I/O error|blk_update_request|Buffer I/O|EXT4-fs error|XFS .*error|sgdisk|exclusive opened' | tail -n 30 + findmnt -T /var/lib/containerd; findmnt -T /var/lib/docker; findmnt -T /var/lib/kubelet; findmnt -T /cpaas + lsblk -f + ls /var/lib/containerd > /dev/null && echo readable + ls -ld /var/log/pods + ``` + + Expected on a healthy node: no I/O error lines and `readable`. `findmnt` and `lsblk` show the device even behind LVM or multipath. A symlinked `/var/log/pods` shares the same disk. + +2. A repeating `sgdisk` line: + + ```bash + dmesg -T | grep -c 'exclusive opened block device' + ps -ef | grep [s]gdisk + ``` + + Record the process and give it to the IaaS owner with the disk errors. + +3. On a master, an apiserver that refuses `6443` while `ls /var/lib/containerd` fails is the master variant of Cause 1: + + ```bash + curl -k --max-time 5 https://:6443 + crictl ps -a | grep kube-apiserver + ``` + +4. Root filesystem hang (Cause 2). Run each command with a timeout so the shell itself does not hang: + + ```bash + timeout 10 df -h || echo "df hung" + timeout 10 strace -f df -h 2>&1 | tail -n 5 + timeout 10 ls / > /dev/null || echo "ls / hung" + ``` + +5. Many Pods on one node failing while the kernel log is clean points elsewhere: a hung runtime is covered in [Node NotReady on ACP with kubelet "PLEG is not healthy"](Node_NotReady_with_PLEG_is_not_healthy.md), and missing containerd directories in [Control-plane Pods do not start with mkdir /var/log/pods: file exists after containerd directories are lost on ACP](Control-plane_Pods_do_not_start_with_mkdir_var_log_pods_file_exists_after_containerd_directories_are_lost_on_ACP.md). diff --git a/docs/en/solutions/Node_unstable_from_a_GPU_NVRM_log_flood_a_faulty_memory_module_or_a_read-only_cpuset_cgroup_on_ACP.md b/docs/en/solutions/Node_unstable_from_a_GPU_NVRM_log_flood_a_faulty_memory_module_or_a_read-only_cpuset_cgroup_on_ACP.md new file mode 100644 index 000000000..703423ce3 --- /dev/null +++ b/docs/en/solutions/Node_unstable_from_a_GPU_NVRM_log_flood_a_faulty_memory_module_or_a_read-only_cpuset_cgroup_on_ACP.md @@ -0,0 +1,136 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.x + - 4.x +--- + +# Node unstable from a GPU NVRM log flood, a faulty memory module, or a read-only cpuset cgroup on ACP + +## Issue + +Typical reports: + +- A node is abnormal, services cannot be deployed to it, and `journalctl -u kubelet` prints nothing. The node is `NotReady` with `PLEG is not healthy`, and a reboot does not fix it. +- A memory fault on a node caused a service disruption. +- The business network on one worker (for example an ARM node) is unreachable although the switch configuration is fine, and the kube-ovn Pods fail with `/sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice: no such file or directory`. + +These are faults in the GPU, memory, or OS configuration of the host. Kubernetes only shows their effect; each cause gives the host-side signature that identifies it. Disk I/O errors and a hanging root filesystem are covered in [Node runtime or kubelet directories return Input/output error, or df hangs, on ACP](Node_runtime_or_kubelet_directories_return_Input_output_error_or_df_hangs_on_ACP.md). A PLEG problem without a GPU log flood is covered in [Node NotReady on ACP with kubelet "PLEG is not healthy"](Node_NotReady_with_PLEG_is_not_healthy.md). + +## Root Cause + +### Cause 1: A GPU that failed to initialise floods the kernel log + +On a GPU node, `journalctl -fu kubelet` and `journalctl -fu docker` print nothing. `systemctl status systemd-journald` shows a continuous stream of `systemd-journald[]: Missed kernel messages`, with journald using several GiB of memory. `dmesg -T` is full of `NVRM` errors. The node looks like a PLEG or runtime problem: + +```text +Ready False KubeletNotReady PLEG is not healthy: pleg was last seen active 38m36.485032165s ago; threshold is 3m0s +"PLEG: Write status" err="rpc error: code = DeadlineExceeded desc = operation timeout: context deadline exceeded" +``` + +while `MemoryPressure`, `DiskPressure`, `PIDPressure`, `KernelDeadlock`, and `NetworkUnavailable` are all `False`. The kernel messages identify the card: + +```text +NVRM: gpuHandleSanityCheckRegReadError_GM107: Possible bad register read: addr: 0x110100, regvalue: 0xbadf5620, error code: Unknown SYS_PRI_ERROR_CODE +NVRM: kflcnWaitForHalt_TU102: Timeout waiting for Falcon to halt +NVRM: GSP failed to halt with GFW_BOOT: (progress 0xff) +NVRM: failed to wait for GFW boot complete: 0x65 VBIOS version +NVRM: (the GPU may be in a bad state and may need to be reset) +NVRM: RmInitAdapter: Cannot initialize GSP firmware RM +NVRM: GPU <0000:BB:DD.F>: RmInitAdapter failed! (0x62:0x65:2015) +NVRM: rm_init_adapter failed, device minor number 0 +``` + +One GPU card failed to initialise. Its very frequent `NVRM` messages overflow the kernel log buffer, journald drops the journal, runtime, and kubelet logs, and the node becomes unstable. + +### Cause 2: Faulty memory module + +Services on one node are disrupted at a specific moment and may recover on their own. The hardware log (BMC, iLO, iDRAC, or IaaS events) or the kernel log reports memory errors for that host. A failing memory module (DIMM) crashes or stalls whatever was using the bad memory. Taking the faulty memory offline or replacing it during a planned shutdown makes the node healthy again. + +### Cause 3: `/sys/fs/cgroup/cpuset` is read-only, so kube-ovn Pods cannot start + +On one node `/sys/fs/cgroup/cpuset` has become read-only, so the cpuset path for Pods cannot be used. The OVN Pods fail with: + +```text +/sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice: no such file or directory +``` + +and the node's Pod network fails, although the physical network is fine. Why the cgroup filesystem became read-only is not known; a reboot restores it. + +## Resolution + +### Cause 1: A GPU that failed to initialise floods the kernel log + +1. Raise the GPU errors with the hardware vendor before continuing Kubernetes-level troubleshooting. +2. Identify the faulty card from the PCI address in the `RmInitAdapter failed!` line, or with `nvidia-smi`. +3. Reset that card with a PCIe Function Level Reset, which recovers the card and the node. + +> **Warning:** Resetting a GPU kills every process using it. Do it only when no workload uses the card (cordon the node and drain or stop the GPU Pods) and the PCI address is confirmed as the faulty card. The reset itself cannot be undone; if it does not recover the card, the card needs vendor service. + +```bash +kubectl cordon +nvidia-smi -q | grep -E 'Bus Id|Product Name' +# by PCI address (domain:bus:device.function, from dmesg or nvidia-smi): +echo 1 > /sys/bus/pci/devices/<0000:BB:DD.F>/reset +# or with the NVIDIA tool: +nvidia-smi --gpu-reset -i +``` + +4. Confirm that `dmesg` stops showing new `NVRM` errors and that the journal shows kubelet and runtime logs again, then `kubectl uncordon `. + +### Cause 2: Faulty memory module + +Take the faulty memory offline or replace it in a planned shutdown. Follow [Shutting down or rebooting ACP nodes for maintenance](Shutting_down_or_rebooting_ACP_nodes_for_maintenance.md): evict Pods, back up etcd if it is a control-plane node, shut down, repair, boot, and check that the platform and applications recover. + +### Cause 3: `/sys/fs/cgroup/cpuset` is read-only, so kube-ovn Pods cannot start + +Reboot the host; this restores write access to `/sys/fs/cgroup/cpuset/`, and the OVN components and business network recover. Ask the OS owner to check the cgroup filesystem afterwards. + +> **Warning:** A reboot restarts every Pod on the node. Do it only when `/sys/fs/cgroup/cpuset` is confirmed read-only on that node and the kube-ovn Pods fail with the `kubepods-burstable.slice: no such file or directory` error. Drain the node first if it still runs workloads. A reboot cannot be undone. + +```bash +kubectl drain --ignore-daemonsets --delete-emptydir-data --timeout=10m +systemctl reboot +# after boot: +mount | grep 'cgroup/cpuset' +kubectl -n kube-system get pods -o wide --field-selector spec.nodeName= | grep -E 'kube-ovn|ovs' +kubectl uncordon +``` + +The cpuset mount should show `rw`, and the kube-ovn Pods on the node should be Running. + +## Diagnostic Steps + +1. GPU log flood (Cause 1): + + ```bash + kubectl describe node | grep -E 'Ready|PLEG' + systemctl status systemd-journald --no-pager | grep -iE 'Missed .* kernel messages|Memory' + journalctl -u systemd-journald --since "-1h" --no-pager | grep -iE 'lost|missed|overrun' + dmesg -T | grep -c NVRM + dmesg -T | grep -E 'RmInitAdapter failed|rm_init_adapter failed|may need to be reset' | tail -n 10 + ``` + + A large `NVRM` count together with `Missed ... kernel messages` is Cause 1. The PCI address in `RmInitAdapter failed!` is the card to reset. `PLEG is not healthy` with a clean kernel log is not this cause. + +2. Memory errors (Cause 2): + + ```bash + dmesg -T | grep -iE 'EDAC|Hardware Error|mce|memory error' | tail -n 20 + ras-mc-ctl --errors 2>/dev/null | tail -n 20 + ``` + + Also check the server BMC event log or the IaaS host events for the same time. Corrected or uncorrected memory errors for that host point at Cause 2. + +3. cpuset cgroup state (Cause 3): + + ```bash + mount | grep 'cgroup/cpuset' # "ro" in the options means read-only + ls /sys/fs/cgroup/cpuset/kubepods.slice/ 2>&1 | head + kubectl -n kube-system get pods -o wide --field-selector spec.nodeName= | grep kube-ovn + ``` + + `ro` on the cpuset mount with failing kube-ovn Pods is Cause 3. A node whose kube-ovn CNI fails after a NIC rename or a clock change is covered in [kube-ovn-cni fails on one node after a NIC rename, clock skew or hostname change on ACP](kube-ovn-cni_fails_on_one_node_after_a_NIC_rename_clock_skew_or_hostname_change_on_ACP.md). diff --git a/docs/en/solutions/Project_quota_and_namespace_ResourceQuota_aggregate_limits_overcommit_Pod_count_and_node_capacity_on_ACP.md b/docs/en/solutions/Project_quota_and_namespace_ResourceQuota_aggregate_limits_overcommit_Pod_count_and_node_capacity_on_ACP.md new file mode 100644 index 000000000..6717a711f --- /dev/null +++ b/docs/en/solutions/Project_quota_and_namespace_ResourceQuota_aggregate_limits_overcommit_Pod_count_and_node_capacity_on_ACP.md @@ -0,0 +1,202 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.16" + - "4.1" +--- + +# Project quota and namespace ResourceQuota: aggregate limits, overcommit, Pod count, and node capacity on ACP + +## Issue + +Quota behavior looks wrong, or the quota model needs explaining: + +- A project has CPU and memory quota on a cluster. Two namespaces are created in it without quota numbers, and their combined usage is expected to stay inside the project quota. +- A project has a storage quota, the namespace was created without one, and the Container Platform view of the namespace shows no storage quota. +- A namespace cannot hold more than 1000 Pods, and the console offers no field to raise that number. +- The namespace Pod limit "does not take effect": the limit is 20, yet more than 30 Pods exist in the namespace, many of them `Completed`, and the quota shows something like `pods: 4/20`. +- Cluster resources are almost fully allocated to projects while real utilization is below 10 percent. Can projects be overcommitted? +- A node has 96 physical CPU cores but `Allocatable` shows 92. How do project quotas map to physical node resources, and why do the CPU and memory totals on the project page not match real usage? +- How are business Pods kept off the nodes that host logging and monitoring components? +- A namespace has more than one quota object, for example the platform quota `default` and a hand-made `-quota`, with different values for the same resource. Which one applies? + +## Root Cause + +### Cause 1: Namespaces without their own quota do not share the project quota as a combined limit + +The project quota is a configuration boundary for allocation. Resource limits are enforced per namespace, by the namespace `ResourceQuota`. Namespaces cannot be left without quota numbers and still have their combined usage bounded by the project quota. + +### Cause 2: A namespace without its own storage quota shows none + +When only the project has a storage quota and none is allocated to the namespace, the Container Platform view shows no storage quota for the namespace. This is expected. The namespace storage limit is then determined by the maximum available storage. + +### Cause 3: The 1000 Pods per namespace cap is `spec.hard.pods` of the namespace ResourceQuota + +On ACP 3.16 the cap of 1000 Pods per namespace is the `spec.hard.pods` value of the namespace's `ResourceQuota`. The console does not expose that value, but it can be changed on the `ResourceQuota` directly. + +### Cause 4: The Pod quota does not count Completed Pods + +The `pods` entry of a `ResourceQuota` counts only Pods in a non-terminal state. A Pod whose `.status.phase` is `Succeeded` or `Failed` is terminal, and `Completed` Pods are in that state. On ACP 4.1.2, for example, the quota `default` shows `pods: 4/20` while 31 Pods exist. Native `ResourceQuota` cannot be made to count completed Pods; this is Kubernetes design (see the Kubernetes `ResourceQuota` object count documentation). + +### Cause 5: Project quotas may add up to more than the cluster has + +The platform supports overcommit between projects. With 200 CPU cores in the cluster, projects A, B, and C can each be allocated 80 cores, 240 in total. Inside one project the namespace quotas are bounded by the project quota: with namespaces `ns1` and `ns2` in project A, `ns1` can have 30 cores and `ns2` 50 cores, as long as `ns1 + ns2 <= 80`. + +### Cause 6: Project quotas are administrator limits, not node capacity, and Allocatable excludes reserved resources + +- Project quotas are based on Kubernetes `ResourceQuota`, which limits `requests.cpu`, `requests.memory`, `limits.cpu`, `limits.memory`, Pod and PVC counts, and extended resources. The effective values are those in `status.hard`. +- A `ResourceQuota` is an upper limit set by an administrator. It is not calculated from physical resources and is not bound to nodes. The platform does not keep the sum of project quotas within the schedulable capacity (that is the overcommit in Cause 5). +- Pods are scheduled by their `requests` against node `Allocatable`. Business capacity is the total `Allocatable` of the worker nodes that run business workloads, minus system components, DaemonSets, a reserve for node failures, and a reserve for rolling upgrades. +- `Allocatable` lower than `Capacity` (92 of 96 cores) is the resource reserved for the system and Kubernetes components through the kubelet `kubeReserved`, `systemReserved`, or `reservedSystemCPUs` settings. +- The CPU and memory totals on the project page are a resource overview. They do not combine actual usage and, with many projects, cannot be calculated accurately. + +### Cause 7: Keeping business Pods off logging and monitoring nodes needs taints and tolerations + +Without taints, any Pod can be scheduled on the logging and monitoring nodes. Node labels, `NoSchedule` taints, tolerations, and node affinity make those nodes a separate infrastructure pool. Components that already tolerate the taint keep running there (for example, ClickHouse carries an `infra` toleration, as do most platform components); ordinary business Pods are no longer scheduled there. Components deployed later, such as MetalLB, must be checked for the toleration separately. + +### Cause 8: Several ResourceQuota objects in one namespace all apply + +Kubernetes evaluates every `ResourceQuota` in the namespace. Creating or scaling a Pod must fit all of them, so for each resource the effective limit is the smallest hard value among the quotas that list it. Resources listed in only one quota are limited by that quota alone. The platform-managed quota is the one named `default`. For example, with `default` at `pods: 1k`, `requests.cpu: 128` and a second quota `-quota` at `pods: 10k`, `cpu: 120`, the namespace can run at most 1000 Pods and CPU requests up to 120 cores. In a `ResourceQuota`, `cpu` and `requests.cpu` (and `memory` and `requests.memory`) both limit the sum of requests, so they count as the same resource when comparing quotas. + +## Resolution + +### Cause 1: Namespaces without their own quota do not share the project quota as a combined limit + +Give each namespace its own quota, with the namespace values adding up to no more than the project quota (Cause 5). + +```bash +kubectl get resourcequota -n -o yaml +kubectl get resourcequota -n -o yaml +``` + +### Cause 2: A namespace without its own storage quota shows none + +No action is needed. To confirm, look for a storage entry (`requests.storage` or a per-StorageClass entry) in the namespace quota: + +```bash +kubectl get resourcequota -n -o yaml | grep -i storage +``` + +No output means the namespace has no storage quota of its own. + +### Cause 3: The 1000 Pods per namespace cap is `spec.hard.pods` of the namespace ResourceQuota + +Find the quota object (on ACP 4.1 the namespace quota is normally named `default`) and save it: + +```bash +kubectl get resourcequota -n +kubectl get resourcequota -n -o yaml > "$HOME/resourcequota-.yaml" +``` + +> **Warning:** This changes how many non-terminal Pods the namespace may run. A value below the current count blocks new Pods until existing ones end; a higher value lets the namespace create more Pods on the cluster. Do it only when the cap comes from `spec.hard.pods` of this `ResourceQuota`. Roll back with `kubectl apply -f` of the saved file or by patching the previous value back. + +```bash +kubectl patch resourcequota -n --type=merge -p '{"spec":{"hard":{"pods":""}}}' +kubectl get resourcequota -n -o jsonpath='{.spec.hard.pods}{"\t"}{.status.hard.pods}{"\n"}' +``` + +Both values should show the new limit. It is not known whether a later quota save in the console keeps this value; read `spec.hard.pods` again after any console edit of that quota. + +### Cause 4: The Pod quota does not count Completed Pods + +No fix exists in native `ResourceQuota`. Confirm the numbers agree: + +```bash +kubectl get resourcequota -n +kubectl get pods -n --field-selector=status.phase!=Succeeded,status.phase!=Failed --no-headers | wc -l +kubectl get pods -n --field-selector=status.phase=Succeeded --no-headers | wc -l +``` + +The first count matches the used `pods` value of the quota; the second is the `Completed` Pods it does not count. + +### Cause 5: Project quotas may add up to more than the cluster has + +Overcommit between projects is allowed. Within one project, keep the sum of namespace quotas at or below the project quota. + +```bash +kubectl get resourcequota -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,LIMITS_CPU:.status.hard.limits\.cpu,USED_LIMITS_CPU:.status.used.limits\.cpu' +``` + +Pods are still placed only where node `Allocatable` has room for their `requests` (Cause 6). + +### Cause 6: Project quotas are administrator limits, not node capacity, and Allocatable excludes reserved resources + +Plan project quotas from business capacity as defined above, and read the node reserve from the kubelet configuration: + +```bash +kubectl get nodes -o custom-columns='NAME:.metadata.name,CPU_CAPACITY:.status.capacity.cpu,CPU_ALLOCATABLE:.status.allocatable.cpu,MEM_CAPACITY:.status.capacity.memory,MEM_ALLOCATABLE:.status.allocatable.memory' +kubectl get --raw "/api/v1/nodes//proxy/configz" | jq '.kubeletconfig | {kubeReserved, systemReserved, reservedSystemCPUs}' +``` + +The difference between capacity and allocatable should match the reserved values (plus any eviction threshold for memory). Use monitoring, not the project page totals, for real CPU and memory usage. + +### Cause 7: Keeping business Pods off logging and monitoring nodes needs taints and tolerations + +Before tainting, read the tolerations of every Pod that must stay on those nodes, and use the key those tolerations already match (`infra` for ClickHouse): + +```bash +kubectl get pods -A -o wide --field-selector spec.nodeName= +kubectl get pod -n -o jsonpath='{.spec.tolerations}{"\n"}' +``` + +> **Warning:** A `NoSchedule` taint stops new Pods without a matching toleration from being scheduled on the node, including platform add-ons deployed later that lack it. Running Pods are not evicted. Taint only after confirming that every component that must run on these nodes tolerates it. Roll back by removing the taint with the trailing `-` command. + +```bash +kubectl label nodes = +kubectl taint nodes =:NoSchedule +# rollback +kubectl taint nodes =:NoSchedule- +``` + +Afterwards, check that the logging and monitoring Pods still schedule on those nodes, and check later components such as MetalLB for the toleration. + +### Cause 8: Several ResourceQuota objects in one namespace all apply + +List what each quota limits and compare the hard values per resource: + +```bash +kubectl get resourcequota -n +kubectl get resourcequota -n -o custom-columns='NAME:.metadata.name,HARD:.status.hard,USED:.status.used' +``` + +For each resource, the smallest hard value is the one that blocks a new Pod. To make one value authoritative, change the quota that is stricter, or remove the extra quota object and keep `default`. + +> **Warning:** Deleting a `ResourceQuota` removes the limits it alone enforces. Delete only a quota that was added by hand, never `default`, and save it first (`kubectl get resourcequota -n -o yaml > .yaml`). Roll back with `kubectl apply -f .yaml`. + +After an edit, compare `spec.hard` with `status.hard`: the quota controller updates `status.hard` shortly after `spec.hard` changes, and admission uses the new value once it is there. + +## Diagnostic Steps + +1. Decide whether the question is about enforcement or display. List the quota objects and the project quota: + + ```bash + kubectl get resourcequota -A + kubectl get projectquotas.auth.alauda.io -A + ``` + + Namespaces with no `ResourceQuota` that are expected to share the project limit: Cause 1. A namespace quota without a storage entry: Cause 2. More than one `ResourceQuota` in a namespace: Cause 8. + +2. For a Pod count question, read `spec.hard.pods` and `status.used.pods`: + + ```bash + kubectl get resourcequota -n -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.hard.pods}{"\t"}{.status.used.pods}{"\n"}{end}' + kubectl get pods -n --no-headers | awk '{print $3}' | sort | uniq -c + ``` + + A cap at 1000 that the console cannot change: Cause 3. More Pods listed than `status.used.pods`, the extra ones `Completed`: Cause 4. + +3. For allocation questions, compare the sum of project quotas with cluster capacity (Cause 5), and node `Capacity` with `Allocatable` and the kubelet reserve (Cause 6). + +4. For placement of business Pods, list node taints: + + ```bash + kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints' + ``` + + No taint on the logging and monitoring nodes: Cause 7. + +5. A rollout that stalls with `FailedCreate ... exceeded quota` is covered in [Deployment rollout stalls with FailedCreate exceeded quota and no built-in alert on ACP](Deployment_rollout_stalls_with_FailedCreate_exceeded_quota_and_no_built_in_alert_on_ACP.md). From 4dc175f84161db1cd5f48843c63129a09ca43f36 Mon Sep 17 00:00:00 2001 From: Komh Date: Wed, 18 Nov 2026 17:41:41 +0800 Subject: [PATCH 23/25] docs(solutions): add 3 and update 1 troubleshooting articles --- ...s_off_on_Kube-OVN_genev_sys_6081_on_ACP.md | 100 +++++++++++++++++ ...an_still_be_assigned_to_its_Pods_on_ACP.md | 71 ++++++++++++ ...ollo_holds_a_stale_cluster_cache_on_ACP.md | 6 +- ...enabling_etcd_sync_for_global_DR_on_ACP.md | 104 ++++++++++++++++++ 4 files changed, 279 insertions(+), 2 deletions(-) create mode 100644 docs/en/solutions/Cross_node_Pod_throughput_far_below_node_bandwidth_because_TX_offload_is_off_on_Kube-OVN_genev_sys_6081_on_ACP.md create mode 100644 docs/en/solutions/Fixed_IPs_outside_the_Kube-OVN_IPPool_bound_to_a_namespace_can_still_be_assigned_to_its_Pods_on_ACP.md create mode 100644 docs/en/solutions/Which_address_goes_into_Active_Global_Cluster_ETCD_Endpoints_when_enabling_etcd_sync_for_global_DR_on_ACP.md diff --git a/docs/en/solutions/Cross_node_Pod_throughput_far_below_node_bandwidth_because_TX_offload_is_off_on_Kube-OVN_genev_sys_6081_on_ACP.md b/docs/en/solutions/Cross_node_Pod_throughput_far_below_node_bandwidth_because_TX_offload_is_off_on_Kube-OVN_genev_sys_6081_on_ACP.md new file mode 100644 index 000000000..3b21ff9b6 --- /dev/null +++ b/docs/en/solutions/Cross_node_Pod_throughput_far_below_node_bandwidth_because_TX_offload_is_off_on_Kube-OVN_genev_sys_6081_on_ACP.md @@ -0,0 +1,100 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Cross-node Pod throughput far below node bandwidth because TX offload is off on Kube-OVN genev_sys_6081 on ACP + +## Issue + +On a cluster that uses the Kube-OVN Geneve overlay, an `iperf` test between two nodes reaches close to the line rate of the NIC, but the same test between two Pods on those nodes reaches much less. For example, on nodes with 25 Gbit/s NICs (ARM Kunpeng servers): + +- node to node, across nodes: about 23.5 Gbit/s +- Pod to Pod, across nodes: about 9.1 Gbit/s, a loss of more than 60% + +Connectivity itself is fine: cross-node Pod traffic works, it is just slow. On the nodes, `ethtool -k genev_sys_6081` shows transmit offload switched off on the Geneve tunnel interface, with TCP segmentation offload requested but not active: + +```text +tx-checksumming: off + tx-checksum-ip-generic: off +tcp-segmentation-offload: off + tx-tcp-segmentation: off [requested on] + tx-tcp-ecn-segmentation: off [requested on] + tx-tcp-mangleid-segmentation: off [requested on] + tx-tcp6-segmentation: off [requested on] +generic-segmentation-offload: on +generic-receive-offload: on +``` + +If cross-node Pod traffic fails rather than being slow, this article does not apply; see [Cross-node Pod traffic fails on the Kube-OVN Geneve overlay because of UDP 6081, TX offload or MTU on ACP](Cross_node_Pod_traffic_fails_on_the_Kube-OVN_Geneve_overlay_because_of_UDP_6081_TX_offload_or_MTU_on_ACP.md), where turning offload **off** is the fix. + +## Root Cause + +### Cause 1: a Kube-OVN defect leaves TX offload disabled on genev_sys_6081 + +Pod traffic between nodes is encapsulated in Geneve and leaves the node through the OVS tunnel interface `genev_sys_6081`. A Kube-OVN defect leaves TX checksum offload disabled on that interface. Because TCP segmentation offload depends on TX checksum offload, the kernel also turns TSO off, which is why the TSO features read `off [requested on]`. Every encapsulated packet then has to be segmented and checksummed in software, and cross-node Pod throughput falls far below what the node itself reaches over the same NIC. + +The setting does not persist. TX offload is disabled on the interface again after the node reboots or after the `kube-ovn-cni` Pod on that node restarts. The defect is fixed in a later OVN version; until the cluster runs a fixed version, the workaround below has to be reapplied after every such event. + +## Resolution + +### Cause 1: a Kube-OVN defect leaves TX offload disabled on genev_sys_6081 + +1. On each node, confirm the symptom before changing anything. Continue only if `tx-checksumming` is `off` and the TSO features show `off [requested on]`: + + ```bash + ethtool -k genev_sys_6081 | grep -E '^tx-checksumming|tx-checksum-ip-generic|^tcp-segmentation-offload|tx-tcp' + ``` + +2. Turn TX offload on for the Geneve tunnel interface, on every node of the cluster (the physical nodes that carry Pod traffic): + + > **Warning:** This changes the offload settings of the tunnel interface that carries all cross-node Pod traffic of the node. Apply it only on nodes where `ethtool -k genev_sys_6081` shows the signature above, and test one node first. Do not apply it on a node where transmit offload was switched off on purpose to work around a NIC checksum problem (see the related article in the Issue section). If cross-node Pod traffic from the node starts failing or checksum errors rise after the change, roll back with `ethtool -K genev_sys_6081 tx off`. + + ```bash + ethtool -K genev_sys_6081 tx on + ``` + +3. Verify the result on the node. Expected: `tx-checksumming: on` and `tcp-segmentation-offload: on`: + + ```bash + ethtool -k genev_sys_6081 | grep -E '^tx-checksumming|^tcp-segmentation-offload' + ``` + +4. Repeat the cross-node Pod `iperf` test (see Diagnostic Steps). Pod-to-Pod throughput should now be close to node-to-node throughput. + +5. Reapply step 2 after every reboot of a node and after every restart of the `kube-ovn-cni` Pod on a node, until the cluster runs an OVN version that contains the fix. After such an event, repeat step 1 to check whether the interface has fallen back to `tx-checksumming: off`. + +## Diagnostic Steps + +1. Measure node-to-node throughput across two nodes, then Pod-to-Pod throughput between Pods scheduled on the same two nodes. Run `iperf3 -s` on the server side and the client against it: + + ```bash + # on node B, or in a Pod on node B + iperf3 -s + # on node A, or in a Pod on node A + iperf3 -c -t 30 -P 4 + ``` + + Expected for this article: the node-to-node result is close to the NIC line rate, and the Pod-to-Pod result is much lower, with no connection errors. + +2. Read the offload settings of the Geneve tunnel interface on both nodes: + + ```bash + ethtool -k genev_sys_6081 + ``` + + Expected for this article: `tx-checksumming: off`, `tcp-segmentation-offload: off`, and `tx-tcp-segmentation: off [requested on]`. If `tx-checksumming` is already `on`, this cause does not apply and the throughput gap has another reason. + +3. Check whether the offload state lines up with recent node reboots or `kube-ovn-cni` restarts. A node where TX offload was turned on earlier and is `off` again after a restart matches this cause: + + ```bash + kubectl -n kube-system get pod -o wide | grep kube-ovn-cni + uptime + ``` + + The `AGE` and restart count of the `kube-ovn-cni` Pod on the node, and the node uptime, show when the setting was last reset. + +4. Rule out a connectivity fault. Cross-node Pod traffic that times out or loses packets, rather than being slow, belongs to the related article on Geneve failures, not to this one. diff --git a/docs/en/solutions/Fixed_IPs_outside_the_Kube-OVN_IPPool_bound_to_a_namespace_can_still_be_assigned_to_its_Pods_on_ACP.md b/docs/en/solutions/Fixed_IPs_outside_the_Kube-OVN_IPPool_bound_to_a_namespace_can_still_be_assigned_to_its_Pods_on_ACP.md new file mode 100644 index 000000000..de93963aa --- /dev/null +++ b/docs/en/solutions/Fixed_IPs_outside_the_Kube-OVN_IPPool_bound_to_a_namespace_can_still_be_assigned_to_its_Pods_on_ACP.md @@ -0,0 +1,71 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Fixed IPs outside the Kube-OVN IPPool bound to a namespace can still be assigned to its Pods on ACP + +## Issue + +A Kube-OVN `IPPool` is bound to a namespace through `spec.namespaces`, with the intention that Pods in that namespace only use addresses from the pool. When a workload in that namespace is configured with a fixed IP, addresses outside the bound pool can still be selected and assigned, including addresses intended for other namespaces. This raises the concern that a fixed IP chosen this way conflicts with an address planned for another namespace and causes a production outage. + +## Root Cause + +### Cause 1: the namespace binding of an IPPool applies only to automatic IP allocation + +Binding an IPPool to a namespace restricts only the addresses that Kube-OVN allocates **automatically** to Pods of that namespace: a Pod without a fixed IP gets its address from the bound pool instead of from the rest of the subnet. + +An IP that is specified manually for a Pod or workload (a fixed IP) is not limited by the namespace binding. Kube-OVN accepts a manually specified address from outside the bound pool, so the binding does not stop a fixed IP from taking an address that belongs to another namespace's pool. This is current product behavior, not a misconfiguration. + +## Resolution + +### Cause 1: the namespace binding of an IPPool applies only to automatic IP allocation + +There is no setting that makes the namespace binding apply to manually specified IPs. Keep the namespace's addresses inside its pool with one of these approaches: + +1. Let Pods of the namespace get their IPs automatically. Do not configure fixed IPs for workloads that do not need them; their addresses then always come from the bound pool. + +2. When a workload does need a fixed IP, pick it only from the pool bound to its namespace. Read the pool's ranges first: + + ```bash + kubectl get ippool -o jsonpath='{.spec.subnet}{"\n"}{.spec.ips}{"\n"}{.spec.namespaces}{"\n"}' + ``` + + Check that the address is not already used before you assign it: + + ```bash + kubectl get ip -o wide | grep -w + kubectl get pod -A -o wide | grep -w + ``` + + Both commands should return nothing for a free address. + +3. Treat fixed IPs as a reviewed setting. Because the binding does not check them, the person who configures a fixed IP has to make sure it lies inside the namespace's pool. + +## Diagnostic Steps + +1. List the IPPools, their subnets and the namespaces they are bound to: + + ```bash + kubectl get ippool + kubectl get ippool -o yaml + ``` + +2. List the Pods of the namespace with their IPs and compare them with the pool's `spec.ips`: + + ```bash + kubectl -n get pod -o wide + ``` + + Expected for this article: Pods without a fixed IP have addresses inside the bound pool. A Pod whose address lies outside the pool is one that was given its IP manually. + +3. Find the workloads of the namespace that request a fixed IP. Fixed IPs are set through Kube-OVN annotations in the Pod template, such as `ovn.kubernetes.io/ip_address` (one address) or `ovn.kubernetes.io/ip_pool` when it holds a list of addresses. An `ovn.kubernetes.io/ip_pool` value that is the name of an IPPool is not a fixed IP; addresses are then allocated automatically from that pool: + + ```bash + kubectl -n get deploy,statefulset -o yaml | grep -E 'ovn.kubernetes.io/(ip_address|ip_pool)' + ``` + + Every fixed address found here has to be checked against the bound pool by hand, as described in the Resolution. diff --git a/docs/en/solutions/Monitoring_pages_stay_empty_because_courier-api_or_apollo_holds_a_stale_cluster_cache_on_ACP.md b/docs/en/solutions/Monitoring_pages_stay_empty_because_courier-api_or_apollo_holds_a_stale_cluster_cache_on_ACP.md index ea12fbac2..ad3f27700 100644 --- a/docs/en/solutions/Monitoring_pages_stay_empty_because_courier-api_or_apollo_holds_a_stale_cluster_cache_on_ACP.md +++ b/docs/en/solutions/Monitoring_pages_stay_empty_because_courier-api_or_apollo_holds_a_stale_cluster_cache_on_ACP.md @@ -19,6 +19,7 @@ ProductsVersion: The monitoring page of a workload cluster, the cluster overview, or Grafana shows no data, or the overview shows the banner that the monitoring component is abnormal. Prometheus or VictoriaMetrics on that workload cluster is healthy, vmui inside the cluster has series, and port 11780 on the cluster VIP answers. Typical situations: - A newly created or newly attached cluster never loads monitoring, even with ports 11780 and 11781 open. +- After Prometheus was reinstalled on a workload cluster, none of that cluster's monitoring data is displayed, model inference monitoring included, while monitoring of the `global` cluster is normal. - In the browser developer tools (F12, **Network**) the failing request returns HTTP 500 with `"message": "get cluster from cache error"`. - An inspection or monitoring API call fails with `bad_response: invalid character '<' looking for beginning of value`. - After a disaster-recovery switchover of the `global` cluster, every business cluster's Prometheus is shown as not deployed although it is running. @@ -32,7 +33,7 @@ Related article: [Monitoring pages return 502 or context deadline exceeded becau ### Cause 1: courier-api has a stale cluster list -`courier-api` logs a cache error, vmui already has series, the VictoriaMetrics Pods are `Running` and not CPU-bound, and a newly built or newly attached cluster does not load in the UI. `courier-api` has not picked up the new cluster. A `config-reloader` line such as `Post "http://localhost:8429/-/reload": dial tcp [::1]:8429: connect: connection refused` can appear at the same time; it is a missed reload, not the reason the cluster list is stale. +`courier-api` logs a cache error, vmui already has series, the VictoriaMetrics Pods are `Running` and not CPU-bound, and a newly built or newly attached cluster does not load in the UI. `courier-api` has not picked up the new cluster. The same applies after the Prometheus of a workload cluster is reinstalled: the cluster's monitoring stays empty until `courier-api` is restarted, while `global` monitoring is unaffected. Why `courier-api` does not pick up the reinstalled backend on its own is not known. A `config-reloader` line such as `Post "http://localhost:8429/-/reload": dial tcp [::1]:8429: connect: connection refused` can appear at the same time; it is a missed reload, not the reason the cluster list is stale. ### Cause 2: the response body is HTML, not JSON @@ -66,7 +67,7 @@ On a standby `global` cluster that is not serving traffic, an abnormal monitorin ### Cause 1: courier-api has a stale cluster list -> **Warning:** Recreating `courier-api` fails in-flight monitoring and inspection calls until the new Pod is `Ready`. Do this only when the `courier-api` log shows the cache error, preferably in a quiet period if this `global` cluster serves many workload clusters. The Deployment recreates the Pod; nothing needs rolling back if the new Pod is healthy. +> **Warning:** Recreating `courier-api` fails in-flight monitoring and inspection calls until the new Pod is `Ready`. Do this only when the `courier-api` log shows the cache error, or when a workload cluster's monitoring stays empty after its Prometheus was reinstalled and the Prometheus Pods there are `Running`, preferably in a quiet period if this `global` cluster serves many workload clusters. The Deployment recreates the Pod; nothing needs rolling back if the new Pod is healthy. ```bash kubectl -n cpaas-system get pod -o wide | grep courier-api @@ -166,6 +167,7 @@ kubectl -n cpaas-system delete pod 3. Match the log or F12 response to a cause: - `cache error` in `courier-api`: Cause 1. + - The cluster's Prometheus was reinstalled shortly before its monitoring went empty, and `global` monitoring is normal: Cause 1. - `Unauthorized` on `clusters` in `courier`, or `bad_response: invalid character '<'`: Cause 2. - `get cluster from cache error` or `cluster not found` in `apollo` or in the F12 response: Cause 4, even when port 11780 is open. diff --git a/docs/en/solutions/Which_address_goes_into_Active_Global_Cluster_ETCD_Endpoints_when_enabling_etcd_sync_for_global_DR_on_ACP.md b/docs/en/solutions/Which_address_goes_into_Active_Global_Cluster_ETCD_Endpoints_when_enabling_etcd_sync_for_global_DR_on_ACP.md new file mode 100644 index 000000000..8417ec7d0 --- /dev/null +++ b/docs/en/solutions/Which_address_goes_into_Active_Global_Cluster_ETCD_Endpoints_when_enabling_etcd_sync_for_global_DR_on_ACP.md @@ -0,0 +1,104 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - 4.3.x +--- + +# Which address goes into Active Global Cluster ETCD Endpoints when enabling etcd sync for global DR on ACP + +## Issue + +While deploying a standby `global` cluster for disaster recovery (DR), the last installation step is to enable etcd synchronization. On the standby cluster's Web Console, under **Marketplace** > **Cluster Plugins** with the `global` cluster selected, the **etcd Synchronizer** plugin is installed with these parameters: + +- **Active Global Cluster VIP** +- **Active Global Cluster ETCD Endpoints** +- **Standby Cluster ETCD Endpoints** +- **Active Global Cluster Token Secret** +- **Data Check Interval** +- **Print detail logs** + +The installation guide only says to "configure **Active Global Cluster ETCD Endpoints** correctly" when port `2379` is not forwarded through a load balancer, and it is not obvious which address that field expects: the active cluster or the standby cluster, a VIP or the individual control plane nodes. + +## Root Cause + +### Cause 1: the field names the etcd of the active cluster, reached through its VIP on port 2379 + +The etcd Synchronizer runs on the **standby** `global` cluster. It reads etcd data from the **active** (primary) `global` cluster on port `2379` and writes it into the standby cluster's local etcd. **Active Global Cluster ETCD Endpoints** tells it where to read from, so it must point at the active cluster, not at the standby. The value is the VIP of the active `global` cluster followed by port `2379`, for example `https://:2379`. + +**Standby Cluster ETCD Endpoints** is the local side and normally keeps its default value. + +## Resolution + +### Cause 1: the field names the etcd of the active cluster, reached through its VIP on port 2379 + +1. On the **active** `global` cluster, read the bearer token that the standby cluster uses to access the active cluster's API server: + + ```bash + kubectl -n cpaas-system get secret k8sadmin -o jsonpath='{.data.token}' | base64 -d + ``` + +2. On the **standby** `global` cluster, store that token in the `etcd-sync-active-cluster-token` Secret, under the data key `token`: + + ```bash + ACTIVE_CLUSTER_TOKEN='' + kubectl -n cpaas-system create secret generic etcd-sync-active-cluster-token \ + --from-literal=token="${ACTIVE_CLUSTER_TOKEN}" \ + --dry-run=client -o yaml | kubectl apply -f - + ``` + +3. Make sure the standby cluster's control plane nodes can reach port `2379` on the active cluster's VIP over TCP (see Diagnostic Steps). If a load balancer is used for this port, it must forward in TCP mode; L7 forwarding is not supported. + +4. Open the **standby** cluster's Web Console through its own VIP, switch to the **Administrator** view, go to **Marketplace** > **Cluster Plugins**, select the `global` cluster, and install the **etcd Synchronizer** plugin with: + + | Parameter | Value | + | --- | --- | + | Active Global Cluster VIP | VIP of the active `global` cluster | + | Active Global Cluster ETCD Endpoints | VIP of the active `global` cluster with port `2379`, for example `https://:2379` | + | Standby Cluster ETCD Endpoints | Default value, unless the local etcd is exposed through a different endpoint | + | Active Global Cluster Token Secret | `etcd-sync-active-cluster-token` | + | Data Check Interval | Default value | + | Print detail logs | Disabled, unless troubleshooting | + +5. During installation the `etcd-sync-bootstrap` Job runs before the `etcd-sync` Deployment starts. Verify the Job and the resources it prepares, on the standby cluster: + + ```bash + kubectl get job -n cpaas-system etcd-sync-bootstrap + kubectl logs -n cpaas-system job/etcd-sync-bootstrap + kubectl get secret -n cpaas-system remote-etcd-ca + kubectl get issuer -n cpaas-system remote-etcd-issuer + kubectl get certificate -n cpaas-system remote-etcd-client + kubectl get secret -n cpaas-system remote-etcd-client + ``` + +6. Verify that the sync Pods run and that the leader has started syncing: + + ```bash + kubectl get po -n cpaas-system -l app=etcd-sync + leader_pod=$(kubectl get lease -n cpaas-system etcd-sync-mirror -o jsonpath='{.spec.holderIdentity}') + kubectl logs -n cpaas-system "$leader_pod" | grep -E "Acquired leader lease|Start Sync update" + ``` + +## Diagnostic Steps + +1. From a control plane node of the **standby** cluster, check that port `2379` on the active cluster's VIP accepts TCP connections: + + ```bash + nc -vz -w 3 2379 + ``` + + Expected: the connection succeeds. A timeout or `Connection refused` means the standby cluster cannot reach the active etcd through that address, and the network path or load balancer forwarding has to be fixed before the plugin can sync. + +2. Check the values the plugin was installed with. **Active Global Cluster VIP** and **Active Global Cluster ETCD Endpoints** must both refer to the **active** cluster; **Standby Cluster ETCD Endpoints** refers to the standby cluster. + +3. Check that the token Secret exists on the standby cluster and holds the key `token`: + + ```bash + kubectl -n cpaas-system get secret etcd-sync-active-cluster-token -o jsonpath='{.data}' | grep -o '"token"' + ``` + + Expected output: `"token"`. + +4. Check the leader's log on the standby cluster. Expected: `Acquired leader lease` followed by `Start Sync update`. If the sync Pods do not start at all, see [etcd-sync Pod stuck in ContainerCreating with failed to sync secret cache on the standby global cluster on ACP](etcd-sync_Pod_stuck_in_ContainerCreating_with_failed_to_sync_secret_cache_on_the_standby_global_cluster_on_ACP.md). From 9d81df37deccb0a52fe3ff3b27546cdbd25a7f31 Mon Sep 17 00:00:00 2001 From: Komh Date: Wed, 18 Nov 2026 17:41:46 +0800 Subject: [PATCH 24/25] docs(solutions): add 6 and update 1 troubleshooting articles --- ...n_s3_GetBucketVersioning_is_not_granted.md | 71 +++++++++++ ...rojects_API_500_or_an_extra_bot_project.md | 13 ++ ...s_CR_with_an_empty_spec.agent.namespace.md | 90 ++++++++++++++ ..._operator_order_and_Kafka_2.x_instances.md | 86 +++++++++++++ ...s_in_Catalog_after_upgrading_to_ACP_4.3.md | 91 ++++++++++++++ ...OutOfMemoryError_Java_heap_space_on_ACP.md | 115 ++++++++++++++++++ ...14_instance_to_PostgreSQL_16_on_ACP_4.2.md | 55 +++++++++ 7 files changed, 521 insertions(+) create mode 100644 docs/en/solutions/Data_Services_backup_to_external_S3_storage_fails_with_access_denied_when_s3_GetBucketVersioning_is_not_granted.md create mode 100644 docs/en/solutions/Jenkins_upgrade_stuck_admission_webhook_vjenkins.kb.io_rejects_a_Jenkins_CR_with_an_empty_spec.agent.namespace.md create mode 100644 docs/en/solutions/Planning_an_ACP_upgrade_to_4.2.6_the_4.0.12_starting_version_PostgreSQL_operator_order_and_Kafka_2.x_instances.md create mode 100644 docs/en/solutions/Project_administrators_lose_permission_to_manage_template_repositories_in_Catalog_after_upgrading_to_ACP_4.3.md create mode 100644 docs/en/solutions/Tekton_pipeline_Maven_task_fails_with_java.lang.OutOfMemoryError_Java_heap_space_on_ACP.md create mode 100644 docs/en/solutions/Upgrading_a_Data_Services_PostgreSQL_14_instance_to_PostgreSQL_16_on_ACP_4.2.md diff --git a/docs/en/solutions/Data_Services_backup_to_external_S3_storage_fails_with_access_denied_when_s3_GetBucketVersioning_is_not_granted.md b/docs/en/solutions/Data_Services_backup_to_external_S3_storage_fails_with_access_denied_when_s3_GetBucketVersioning_is_not_granted.md new file mode 100644 index 000000000..d1d4de59a --- /dev/null +++ b/docs/en/solutions/Data_Services_backup_to_external_S3_storage_fails_with_access_denied_when_s3_GetBucketVersioning_is_not_granted.md @@ -0,0 +1,71 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Application Services +ProductsVersion: [] +--- + +# Data Services backup to external S3 storage fails with access denied when s3:GetBucketVersioning is not granted + +## Issue + +An external S3-compatible object storage is configured as a backup storage for Data Services, and the connection to it fails. The connection error names a missing permission on the bucket, in the form: + +```text +s3:GetBucketVersioning on +``` + +and the RDS operator log reports `access denied` for the same storage. + +## Root Cause + +### Cause 1: the S3 user lacks s3:GetBucketVersioning on the backup bucket + +When the platform checks the backup storage, it does more than log in: it calls the S3 API `GetBucketVersioning` on the configured bucket. The object storage authorizes every S3 action separately, against the user or bucket policy of the access key. A policy that grants object read and write, or bucket listing, but not `s3:GetBucketVersioning` on that bucket makes the storage answer `AccessDenied` to that call, and the platform reports the storage connection as failed. + +The fix is on the object storage side: grant `s3:GetBucketVersioning` on the bucket to the user whose access key the platform uses. No change is needed on the platform. + +## Resolution + +### Cause 1 + +1. Identify the bucket and the access key configured for the backup storage on the platform, and the user (or sub-user) on the object storage that owns that access key. + +2. On the object storage, add `s3:GetBucketVersioning` on that bucket to the user's policy or to the bucket policy. S3-compatible storage products expose this in their own console as a bucket or user permission; where AWS-style JSON policies are supported, the statement looks like this: + + ```json + { + "Effect": "Allow", + "Principal": {"AWS": [""]}, + "Action": ["s3:GetBucketVersioning"], + "Resource": ["arn:aws:s3:::"] + } + ``` + + The resource is the bucket itself (`arn:aws:s3:::`), not the objects in it (`arn:aws:s3:::/*`); a grant on objects only does not cover this bucket-level action. Keep the permissions the user already has for reading and writing backup objects. + +3. Test the storage connection again from the platform, or run the backup again. The connection succeeds and the `access denied` messages stop in the RDS operator log. + +## Diagnostic Steps + +1. From a machine that reaches the object storage, call the same API with the same credentials (AWS CLI shown; any S3 client works): + + ```bash + export AWS_ACCESS_KEY_ID= + export AWS_SECRET_ACCESS_KEY= + aws s3api get-bucket-versioning --bucket --endpoint-url https:// --region + ``` + + - `An error occurred (AccessDenied) when calling the GetBucketVersioning operation` confirms this cause. + - After the fix, the call returns `{}` (versioning never configured) or a JSON object with a `Status` field. Either is a pass; versioning does not have to be enabled. + +2. For comparison, list the bucket with the same credentials: + + ```bash + aws s3 ls s3:// --endpoint-url https:// --region + ``` + + A listing that succeeds while step 1 returns `AccessDenied` shows that the credentials are valid and only the bucket-level permission is missing. + +3. Rule out look-alikes in the same connection check: `NoSuchBucket` from `GetBucketVersioning` means the bucket name on the platform does not match a bucket on the storage (check the spelling), and `InvalidAccessKeyId` or `SignatureDoesNotMatch` means the key pair itself is wrong. Neither is fixed by granting permissions. diff --git a/docs/en/solutions/GitLab_toolchain_integration_fails_token-gitlab_not_found_Internal_error_projects_API_500_or_an_extra_bot_project.md b/docs/en/solutions/GitLab_toolchain_integration_fails_token-gitlab_not_found_Internal_error_projects_API_500_or_an_extra_bot_project.md index fdef6d634..13e15e89f 100644 --- a/docs/en/solutions/GitLab_toolchain_integration_fails_token-gitlab_not_found_Internal_error_projects_API_500_or_an_extra_bot_project.md +++ b/docs/en/solutions/GitLab_toolchain_integration_fails_token-gitlab_not_found_Internal_error_projects_API_500_or_an_extra_bot_project.md @@ -20,6 +20,7 @@ A GitLab toolchain integration of DevOps v3 is configured and does not work in p - `GET /integrations/v1alpha1/namespaces//integrations//projects/` returns `500 (Internal Server Error)` on every call. - After GitLab projects are assigned, the code management list shows an extra entry named like `project__bot_` that does not exist as a project in GitLab. - The integration instance shows an abnormal status with `gitlab returned error: GET .../api/v4/todos: 403 {message: 403 Forbidden - Your account has been blocked.}`. +- A Jenkins pipeline fails to pull code from GitLab with `ERROR: Error cloning remote repo 'origin'`, `git fetch ... returned status code 128`, `remote: HTTP Basic: Access denied`, and `fatal: Authentication failed for '.git/'`, after the API address of the GitLab integration was changed in its integration card. These integrations are served by katanomi (`katanomi-api` and `katanomi-plugin` in `cpaas-system`). Each platform project is expected to have a namespace of the same name in the cluster. Related articles: "Harbor toolchain integration fails: images not selectable, katanomi-plugin timeout, or push asks for credentials" covers the Harbor integration. @@ -49,6 +50,10 @@ When an external GitLab is integrated with root permissions and projects are ass The password stored in the integration's credential no longer matches the GitLab account's current password, for example because the password was changed in GitLab. GitLab rejects the integration's API requests with `403 Forbidden - Your account has been blocked.`, and logging in to the GitLab web console with the same account and password returns `Invalid login or password.` +### Cause 7: `HTTP Basic: Access denied` on clone after the integration's API address was changed away from the GitLab domain name + +The GitLab service is addressed by a domain name. When the API address of an existing GitLab integration is changed through the integration card to a different address, the platform does not recognize the GitLab service correctly under the new address. Code pulls from Jenkins then fail authentication with `remote: HTTP Basic: Access denied` and `fatal: Authentication failed`. Changing the integration's API address back to the GitLab domain name restores the pulls; the repository's clone URL (`spec.httpCloneUrl` of its `GitRepository` object, `meta.katanomi.dev/v1alpha1`) then uses the domain-based address again. This differs from Cause 1, where a new integration is created rather than the address of an existing one being edited. + ## Resolution ### Cause 1: `Secret "token-gitlab" not found` because the integration uses the external GitLab address @@ -130,6 +135,13 @@ The object shows `subType: GitUser`. In GitLab, open `/projects/` for the ID 3. Update the credential used by the integration with the current password, then check the integration status again. It returns to normal once GitLab accepts the credential. 4. If GitLab still answers `Your account has been blocked`, a GitLab administrator checks the account in **Admin Area > Users** and unblocks it. +### Cause 7: `HTTP Basic: Access denied` on clone after the integration's API address was changed away from the GitLab domain name + +1. In the failed Jenkins job log, note the host in the `git fetch ... -- https:////` line, and compare it with the domain name under which GitLab is served. +2. Open the GitLab integration and check its API address in the integration card. If it was changed from the GitLab domain name to another address, change it back to the domain name and save. +3. Check the repository's clone URL in the platform: the `GitRepository` object of the repository shows `spec.httpCloneUrl` starting with `https:///` again. +4. Run the pipeline again. The clone stage completes. + ## Diagnostic Steps ```bash @@ -145,6 +157,7 @@ Capture the failing request in the browser developer tools; the path and the sta - `500` on the integration projects API, and DNS resolution of the GitLab host fails on the node: Cause 4. - An extra `project__bot_` tool project with subtype `GitUser`: Cause 5. - `403 Forbidden - Your account has been blocked` from the integration, and the same account gets `Invalid login or password.` on the GitLab login page: Cause 6. +- Jenkins clone fails with `HTTP Basic: Access denied`, and the integration's API address is no longer the GitLab domain name: Cause 7. To check the token separately from the address, run `git ls-remote https://oauth2:@//.git` from a machine that reaches GitLab; a list of refs means the token is valid and the address is the problem. When katanomi logs domain resolution timeouts for a platform Service name such as `katanomi-plugin.cpaas-system`, rather than for the GitLab host, the fault is cluster DNS, not GitLab. Check the node `/etc/resolv.conf` for a `search` line with site domains, and the CoreDNS log for that Service name being forwarded upstream: diff --git a/docs/en/solutions/Jenkins_upgrade_stuck_admission_webhook_vjenkins.kb.io_rejects_a_Jenkins_CR_with_an_empty_spec.agent.namespace.md b/docs/en/solutions/Jenkins_upgrade_stuck_admission_webhook_vjenkins.kb.io_rejects_a_Jenkins_CR_with_an_empty_spec.agent.namespace.md new file mode 100644 index 000000000..cc591a464 --- /dev/null +++ b/docs/en/solutions/Jenkins_upgrade_stuck_admission_webhook_vjenkins.kb.io_rejects_a_Jenkins_CR_with_an_empty_spec.agent.namespace.md @@ -0,0 +1,90 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - 4.0.x +--- + +# Jenkins upgrade stuck: admission webhook vjenkins.kb.io rejects a Jenkins CR with an empty spec.agent.namespace + +## Issue + +During a toolchain upgrade (for example while the platform moves from 3.16 to 4.0), the Jenkins instance upgrade does not finish. The Jenkins operator controller log repeats a reconcile error of this form: + +```text +"msg"="Reconciler error" "error"="admission webhook \"vjenkins.kb.io\" denied the request: Jenkins.operator.devops.alauda.io \"\" is invalid: spec.agent.namespace: Invalid value: \"\": a lowercase RFC 1123 ... +``` + +The Jenkins Deployment itself still shows `1/1` ready, so Jenkins keeps serving; only the operator cannot apply the upgrade to it. + +## Root Cause + +### Cause 1: the Jenkins CR created by the older version has no spec.agent.namespace + +The new Jenkins operator installs a validating admission webhook, `vjenkins.kb.io`, that requires `spec.agent.namespace` of every `Jenkins` resource (`jenkins.operator.devops.alauda.io`) to be a valid lowercase RFC 1123 name. A Jenkins CR created by the older version never had this field, so it is empty. + +The webhook validates every update of the CR, not only updates made by users. When the operator reconciles the instance it writes its own management information back to the CR; that update is rejected because the field is empty, the reconcile fails, and the upgrade of the instance stops. The running Deployment is not touched, which is why Jenkins stays `1/1`. + +Setting `spec.agent.namespace` on the CR to the namespace where the build agents run makes the CR valid, and the operator can reconcile again. + +## Resolution + +### Cause 1 + +1. Find the Jenkins CR and confirm the field is empty (Diagnostic Steps, step 2). + +2. Decide the agent namespace. It is the namespace in which this Jenkins instance creates its build agent Pods. When the build agents already run in the same namespace as the Jenkins instance (for example `operators`), use that namespace. + +3. Set the field: + + ```bash + kubectl patch jenkins.operator.devops.alauda.io -n --type merge \ + -p '{"spec":{"agent":{"namespace":""}}}' + ``` + + For an instance named `devops-jenkins` in `operators` whose build agents run in `operators`: + + ```bash + kubectl patch jenkins.operator.devops.alauda.io devops-jenkins -n operators --type merge \ + -p '{"spec":{"agent":{"namespace":"operators"}}}' + ``` + + Once the CR is valid, the operator resumes the upgrade that was blocked, so the Jenkins instance can be updated and restarted by the operator right after the patch. Run it at a time when a Jenkins restart is acceptable. If the namespace is wrong, patch the field again with the correct namespace; an empty value is rejected by the webhook and is not a way back. + +4. Watch the operator log again (Diagnostic Steps, step 1). The `vjenkins.kb.io` denial no longer appears, and the toolchain upgrade continues. + +## Diagnostic Steps + +1. Find the Jenkins operator Pod and look for the webhook denial: + + ```bash + kubectl get pods -A | grep -i jenkins + kubectl logs -n --tail=200 | grep -i 'vjenkins.kb.io' + ``` + + A line containing `denied the request` and `spec.agent.namespace: Invalid value: ""` confirms this cause. + +2. Show the agent namespace of every Jenkins CR: + + ```bash + kubectl get jenkins.operator.devops.alauda.io -A \ + -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,AGENT_NS:.spec.agent.namespace' + ``` + + `` in `AGENT_NS` marks a CR that the webhook rejects. + +3. Confirm the webhook is installed: + + ```bash + kubectl get validatingwebhookconfiguration -o yaml | grep -n 'vjenkins.kb.io' + ``` + +4. Check that Jenkins itself is still running, which separates this cause from a Jenkins instance that is down: + + ```bash + kubectl get deploy -n + ``` + + The Jenkins Deployment shows `1/1`. If it is not ready, look at the Jenkins Pod events and logs first; this cause leaves the running instance untouched. diff --git a/docs/en/solutions/Planning_an_ACP_upgrade_to_4.2.6_the_4.0.12_starting_version_PostgreSQL_operator_order_and_Kafka_2.x_instances.md b/docs/en/solutions/Planning_an_ACP_upgrade_to_4.2.6_the_4.0.12_starting_version_PostgreSQL_operator_order_and_Kafka_2.x_instances.md new file mode 100644 index 000000000..3a3e05b0d --- /dev/null +++ b/docs/en/solutions/Planning_an_ACP_upgrade_to_4.2.6_the_4.0.12_starting_version_PostgreSQL_operator_order_and_Kafka_2.x_instances.md @@ -0,0 +1,86 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda Application Services +ProductsVersion: + - 4.0.x + - 4.2.x +--- + +# Planning an ACP upgrade to 4.2.6: the 4.0.12 starting version, PostgreSQL operator order and Kafka 2.x instances + +## Issue + +A platform is upgraded in several hops, for example from ACP 3.18.1 to 4.0 and then to 4.2.6, and the plan is blocked by one of these questions: + +- The 4.2 documentation says the supported path is `4.0` → `4.2`, but the published upgrade path for 4.2.6 shows 4.0.12 as its starting version. Can a cluster on an earlier 4.0 patch, such as 4.0.10, go straight to 4.2.6? +- After the platform reaches 4.0.12 the PostgreSQL operator is on its 4.0 line (for example 4.0.25). The compatibility lists seem to block both orders: the PostgreSQL operator 4.0 line is not listed for ACP 4.2, the 4.1 line is not listed for ACP 4.0, and the operator itself is upgraded one minor line at a time (4.0 → 4.1 → 4.2 → 4.3). Which goes first: the platform upgrade to 4.2.6, or the operator upgrade to 4.1? +- Kafka instances still run Kafka 2.x (for example 2.5), and the Kafka operator that comes with the 4.2.6 platform is on the 4.3 line (for example 4.3.3 or 4.3.5). Does that operator support Kafka 2.x? + +## Root Cause + +### Cause 1: the published upgrade path for 4.2.6 starts from 4.0.12 + +The documentation describes supported upgrades between minor versions (`4.0` → `4.2`). The recommended upgrade path published for a target release is more specific: it names the patch version that the upgrade to that release starts from. For 4.2.6 that starting version is 4.0.12. A cluster on an earlier 4.0 patch is first upgraded to 4.0.12 and then to 4.2.6; from 3.18.1 the path is 3.18.1 → 4.0.12 → 4.2.6. A plan made for an earlier 4.2 patch (for example 3.18.1 → 4.0.10 → 4.2.5) is adjusted when the target changes to 4.2.6. + +### Cause 2: upgrade the PostgreSQL operator to 4.1 before the platform upgrade + +With the constraints above, neither order keeps every component inside its compatibility list for the whole upgrade. The recommended order is to upgrade the PostgreSQL operator from 4.0 to 4.1 first, while the platform is still on 4.0.12, and to upgrade the platform to 4.2.6 afterwards. Upgrading the operator first has the smaller impact. + +### Cause 3: Kafka 2.x is not managed by the 4.2 and later Kafka operator lines + +The Kafka operator lines from 4.2 on no longer manage Kafka 2.x instances, so the 4.3 operator that comes with the 4.2.6 platform does not cover an existing Kafka 2.5 instance. On ACP 4.2, Kafka 2.x instances run when the Kafka operator 4.1 line (`kafka-operator` 4.1) is deployed together with the Kafka 2 operator (`kafka-operator-2x`). This combination has been verified only to the point that the Kafka 2.x Pods run; other functions were not tested on ACP 4.2. + +Both of these operators are past the end of their support: they receive no further support and no vulnerability fixes. Running Kafka 2.x on ACP 4.2 this way is a stopgap until the instances are moved to a Kafka version that the current operator line supports. + +## Resolution + +### Cause 1 + +1. Check the current platform version and compare it with the starting version in the recommended upgrade path published for the target release. +2. If the cluster is on a 4.0 patch earlier than 4.0.12, upgrade it to 4.0.12 first, following the ACP 4.0 upgrade documentation, and validate it. +3. Then upgrade from 4.0.12 to 4.2.6, following the ACP 4.2 upgrade documentation. + +> **Warning:** Each platform upgrade restarts platform components and upgrades every cluster, `global` first. Precondition: the pre-upgrade checks of the target version pass. Rollback: follow the recovery guidance in the upgrade documentation of the target version, and take the backups it requires before you start. + +### Cause 2 + +1. On ACP 4.0.12, back up every PostgreSQL instance and confirm that all instances are `Running` (Diagnostic Steps, step 2). +2. Upgrade the PostgreSQL operator from the 4.0 line to the 4.1 line in OperatorHub. + + > **Warning:** While the operator is replaced, no PostgreSQL instance is reconciled, and the new operator reconciles every instance when it starts. OLM does not downgrade an operator in place, so the way back is restoring the instance backups taken in step 1. Do this in a maintenance window, and only when every instance is healthy beforehand. + +3. Wait until the 4.1 operator is `Succeeded` and every instance is `Running` again. +4. Upgrade the platform from 4.0.12 to 4.2.6 (Cause 1). +5. Continue the operator upgrade one minor line at a time (4.1 → 4.2 → 4.3) as the target platform version requires, checking the instances after each step. + +### Cause 3 + +1. Before the platform upgrade, list the Kafka instances and their versions (Diagnostic Steps, step 3). If any instance runs Kafka 2.x, do not let the Kafka operator move to the 4.2 or 4.3 line on that cluster while the instance exists. +2. To keep Kafka 2.x instances running on ACP 4.2, deploy the Kafka operator 4.1 line together with the Kafka 2 operator (`kafka-operator-2x`) instead of the 4.3 operator. Expect only the Pods to be covered by testing on ACP 4.2, and no support or vulnerability fixes for either operator. +3. Plan the move of every Kafka 2.x instance to a Kafka version supported by a current operator line, following the product upgrade documentation of the Kafka operator (moving to the 4.2 line also requires KRaft mode; see the product document "Migrate from ZooKeeper to KRaft mode"). + +## Diagnostic Steps + +1. Platform version: check the current platform version in the platform console and compare it with the starting version of the published upgrade path for the target release. + +2. PostgreSQL operator and instances: + + ```bash + kubectl get csv -A | grep -i postgres + kubectl get postgresql -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,VER:.spec.postgresql.version,STATUS:.status.PostgresClusterStatus + ``` + + The CSV version shows the operator line (`4.0.x`, `4.1.x`, ...). Every instance should show `Running` before and after each operator step. + +3. Kafka operators and instance versions: + + ```bash + kubectl get csv -A | grep -i kafka + kubectl get rdskafka -A -o yaml | grep -nE '^\s+(name|namespace|version):' + ``` + + A `version` of `2.x` on any instance means the cluster needs the Kafka 4.1 operator plus the Kafka 2 operator on ACP 4.2 (Cause 3). A Kafka CSV on the `4.2.x` or `4.3.x` line with such an instance present is the combination to avoid. diff --git a/docs/en/solutions/Project_administrators_lose_permission_to_manage_template_repositories_in_Catalog_after_upgrading_to_ACP_4.3.md b/docs/en/solutions/Project_administrators_lose_permission_to_manage_template_repositories_in_Catalog_after_upgrading_to_ACP_4.3.md new file mode 100644 index 000000000..a15de99e6 --- /dev/null +++ b/docs/en/solutions/Project_administrators_lose_permission_to_manage_template_repositories_in_Catalog_after_upgrading_to_ACP_4.3.md @@ -0,0 +1,91 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.3.x +--- + +# Project administrators lose permission to manage template repositories in Catalog after upgrading to ACP 4.3 + +## Issue + +After the platform is upgraded to ACP 4.3, users who hold the project administrator role can no longer manage template repositories in **Catalog** > **Helm Charts**. Before the upgrade, the same users could upload new chart versions to the template repositories assigned to their project. After the upgrade, the users have no permission to manage the template repositories. + +## Root Cause + +### Cause 1: the acp-catalog rule of the project administrator RoleTemplate is read-only after the upgrade + +Platform roles are defined by `RoleTemplate` objects on the `global` cluster. Each entry in `spec.rules` points at a `FunctionResource` through `functionResourceRef` and lists the verbs the role gets on that function. The built-in project administrator role is backed by the RoleTemplate `legacy-project-admin-system`. + +ACP 4.3 changed the permissions for template repositories. After the upgrade, the `legacy-project-admin-system` rule with `functionResourceRef: acp-catalog` grants only `get`, `list`, and `watch`, which are read-only verbs, while the neighbouring management rules of the same template (for example `acp-batchaction`, `acp-imagewhitelist`, and `acp-autoscalers`) grant `"*"`. Every user bound to the project administrator role therefore loses write access to template repositories. Changing the verbs of the `acp-catalog` rule in that RoleTemplate restores the permission. + +## Resolution + +### Cause 1 + +Run these steps against the `global` cluster with a platform administrator account. + +1. Save the current RoleTemplate so that you can roll back: + + ```bash + kubectl get roletemplate legacy-project-admin-system -o yaml > legacy-project-admin-system.bak.yaml + ``` + +2. Confirm the signature of this cause (see Diagnostic Steps, step 1): the `acp-catalog` rule lists only `get`, `list`, and `watch`. + +3. Grant management verbs on the `acp-catalog` rule. + + > **Warning:** This changes a built-in role. Every user bound to the project administrator role, in every project, gets the new permission on template repositories as soon as the platform regenerates the role. Do it only when step 2 confirmed that the `acp-catalog` rule is read-only and project administrators are expected to manage template repositories. To roll back, edit the RoleTemplate again and set the `acp-catalog` verbs back to the values saved in `legacy-project-admin-system.bak.yaml`. + + ```bash + kubectl edit roletemplate legacy-project-admin-system + ``` + + Find the rule for `acp-catalog` and change its verbs, for example to the same value that the neighbouring management rules use: + + ```yaml + spec: + rules: + - functionResourceRef: acp-catalog + verbs: + - "*" + ``` + + Leave all other rules unchanged. + +4. Ask an affected user to sign in again and repeat the template repository action, for example **Catalog** > **Helm Charts** > **Add Template**, select a repository assigned to the project, then **Upload Template**. The template status changes from `Uploading` to `Upload Successful`. + +5. The change is made on a built-in role. After every later platform upgrade, run Diagnostic Steps, step 1 again to check that the `acp-catalog` rule still has the verbs you set. + +## Diagnostic Steps + +1. On the `global` cluster, list the rules of the project administrator RoleTemplate: + + ```bash + kubectl get roletemplate legacy-project-admin-system \ + -o jsonpath='{range .spec.rules[*]}{.functionResourceRef}{" "}{.verbs}{"\n"}{end}' + ``` + + Output that matches this cause: + + ```text + acp-catalog ["get","list","watch"] + acp-batchaction ["*"] + acp-imagewhitelist ["*"] + acp-autoscalers ["*"] + ... + ``` + + If `acp-catalog` already has `"*"` or other write verbs, the role is not the problem; continue with step 3. + +2. To see which resources the `acp-catalog` function covers, read the FunctionResource: + + ```bash + kubectl get functionresource acp-catalog -o yaml + ``` + +3. Rule out the look-alike: template repositories are added by platform administrators, and a project can upload templates only to a Chart or OCI Chart repository that is assigned to it with **Management** permission. In the console, check that the repository the user is trying to manage is assigned to the user's project with that permission. A repository that is not assigned to the project, or is assigned without **Management**, is refused for every project user, whatever the role grants. + +4. Check that the affected user actually holds the project administrator role in that project (platform **Role Management** or the user's role list). A user with a custom role is governed by that role's RoleTemplate, not by `legacy-project-admin-system`; inspect its `acp-catalog` rule with the command in step 1, replacing the template name. diff --git a/docs/en/solutions/Tekton_pipeline_Maven_task_fails_with_java.lang.OutOfMemoryError_Java_heap_space_on_ACP.md b/docs/en/solutions/Tekton_pipeline_Maven_task_fails_with_java.lang.OutOfMemoryError_Java_heap_space_on_ACP.md new file mode 100644 index 000000000..445270353 --- /dev/null +++ b/docs/en/solutions/Tekton_pipeline_Maven_task_fails_with_java.lang.OutOfMemoryError_Java_heap_space_on_ACP.md @@ -0,0 +1,115 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - 4.4.x +--- + +# Tekton pipeline Maven task fails with java.lang.OutOfMemoryError Java heap space on ACP + +## Issue + +A Tekton pipeline fails in its Maven build task. The task log ends with: + +```text +[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin::compile (default-compile) on project : Fatal error compiling: java.lang.OutOfMemoryError: Java heap space -> [Help 1] +``` + +The question that follows is how to give the pipeline's Maven task more resources. The Pod of the task is not necessarily `OOMKilled`; the Java process inside it runs out of heap. + +The same error in a Jenkins pipeline is covered in "Jenkins Maven build fails with java.lang.OutOfMemoryError Java heap space although build node resources were raised". + +## Root Cause + +### Cause 1: the Maven JVM heap is too small for the build + +`java.lang.OutOfMemoryError: Java heap space` is raised by the JVM when the heap reaches its maximum size (`-Xmx`). The compiler runs inside the Maven JVM, so a large project can exhaust the heap during `compile` even though the container still has free memory. Raising only the container memory does not raise the heap if the JVM keeps the same `-Xmx`; the heap limit has to be raised, and the container limit has to leave room for it. + +Two places control this in a Tekton pipeline: + +- The `mavenOpts` parameter of the Maven task. Its value is appended after the task's default JVM options, so an `-Xmx` set there takes effect. The Maven step has a memory limit of `2Gi`, so the heap plus metaspace set through `mavenOpts` must fit inside `2Gi`. +- The `taskRunSpecs` of the PipelineRun. An entry for the Maven task can set larger compute resources for that task and add JVM options through an environment variable, for builds that need more than the default step limit allows. + +## Resolution + +### Cause 1 + +Use option A when a heap that fits in `2Gi` is enough. Use option B when the build needs more memory than that. + +**Option A: set the heap through the `mavenOpts` task parameter** + +1. Edit the pipeline and, on the Maven task, set the `mavenOpts` parameter, for example to a 1 GiB heap: + + ```yaml + params: + - name: mavenOpts + value: "-Xmx1g" + ``` + + Put JVM options only in `mavenOpts`. Do not add them to the `goals` list; entries in `goals` are passed to Maven as goals and arguments, not to the JVM. + +2. Keep the heap plus metaspace below the `2Gi` step limit, with room for the JVM's other native memory. TaskRuns that do not set `mavenOpts` keep the previous default behaviour, so only runs started after the change use the new heap. + +3. Run the pipeline again. + +**Option B: give the Maven task more resources with `taskRunSpecs`** + +1. Open the failed run and choose **Re-run Pipeline**, then add a `taskRunSpecs` entry to the PipelineRun YAML. Set `pipelineTaskName` to the name of the Maven task as it is written in your pipeline (`maven` in this example): + + ```yaml + spec: + taskRunSpecs: + - pipelineTaskName: maven + computeResources: + requests: + cpu: "4" + memory: 8Gi + limits: + cpu: "4" + memory: 16Gi + podTemplate: + env: + - name: JAVA_TOOL_OPTIONS + value: "-Xms4g -Xmx12g -XX:MaxMetaspaceSize=2g" + ``` + + The values are an example: keep `-Xmx` plus `MaxMetaspaceSize` below the memory limit, with headroom for non-heap memory. + + > **Warning:** The larger requests count against the namespace ResourceQuota and must fit on a node. If they cannot be satisfied, the task Pod stays `Pending` and the run does not start the Maven task. Apply this only after Diagnostic Steps confirmed `java.lang.OutOfMemoryError: Java heap space`. To roll back, re-run without the `taskRunSpecs` entry. + +2. To make the change permanent, add the same `taskRunSpecs` entry wherever the PipelineRun is created for this pipeline (for example its trigger template), instead of adding it on every re-run. + +3. Run the pipeline. The Maven task completes without `OutOfMemoryError`. + +## Diagnostic Steps + +1. Confirm the error type in the failed task's log. `java.lang.OutOfMemoryError: Java heap space` is this cause. `GC overhead limit exceeded` is also heap exhaustion and has the same fix. + +2. Check whether the task container was killed instead: + + ```bash + kubectl get pod -n -l tekton.dev/pipelineRun= \ + -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.containerStatuses[*].state.terminated.reason}{"\n"}{end}' + ``` + + `OOMKilled` on the Maven task Pod means the container limit is smaller than the heap plus non-heap memory: lower `-Xmx` or raise the limit (option B). `Error` together with the `Java heap space` message in the log is Cause 1 with a heap that is too small. + +3. Check the memory limits the Maven task Pod actually received, after a change: + + ```bash + kubectl get pod -n \ + -o jsonpath='{range .spec.containers[*]}{.name}{" limits="}{.resources.limits.memory}{" requests="}{.resources.requests.memory}{"\n"}{end}' + ``` + + With option B, the Maven step container shows the limit set in `taskRunSpecs`. If it still shows the old value, the `pipelineTaskName` in the entry does not match the task name in the pipeline. + +4. Check which JVM options reach Maven: + + ```bash + kubectl get pod -n \ + -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{range .env[*]}{" "}{.name}={.value}{"\n"}{end}{end}' + ``` + + With option B, `JAVA_TOOL_OPTIONS` is listed on the Maven step container, and the task log contains `Picked up JAVA_TOOL_OPTIONS: ...` from the JVM. When the same option appears more than once, the JVM uses the last one; options given on the command line, such as those from `mavenOpts`, come after `JAVA_TOOL_OPTIONS`. If the heap does not change with option B, set the same `-Xmx` through `mavenOpts` as well, keeping it below the container limit shown in step 3. diff --git a/docs/en/solutions/Upgrading_a_Data_Services_PostgreSQL_14_instance_to_PostgreSQL_16_on_ACP_4.2.md b/docs/en/solutions/Upgrading_a_Data_Services_PostgreSQL_14_instance_to_PostgreSQL_16_on_ACP_4.2.md new file mode 100644 index 000000000..c5baecb93 --- /dev/null +++ b/docs/en/solutions/Upgrading_a_Data_Services_PostgreSQL_14_instance_to_PostgreSQL_16_on_ACP_4.2.md @@ -0,0 +1,55 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Application Services +ProductsVersion: + - 4.2.x +--- + +# Upgrading a Data Services PostgreSQL 14 instance to PostgreSQL 16 on ACP 4.2 + +## Issue + +On ACP 4.2 (for example 4.2.3), Data Services offers PostgreSQL 14 and PostgreSQL 16 for new instances. An existing instance runs PostgreSQL 14 and has to move to PostgreSQL 16. The upgrade documentation only says that the PostgreSQL server version has to be upgraded manually, and the backup and restore documentation describes restoring into a new instance; neither gives the steps for moving an existing instance from 14 to 16. The question is whether updating the instance and selecting the new version is enough. + +## Root Cause + +### Cause 1: the major version is changed on the instance update page + +No separate migration procedure is needed for this path. On ACP 4.2, the update page of a PostgreSQL instance has a database version field that offers `PostgreSQL 14` and `PostgreSQL 16`. Updating the instance and switching that field from `PostgreSQL 14` to `PostgreSQL 16` upgrades the existing instance to 16. + +A PostgreSQL major version change rewrites the instance's system catalogs for the new version and requires the database server to restart, so it is a planned maintenance operation even though it is started from one form field. + +## Resolution + +### Cause 1 + +1. Check that the instance is healthy before the upgrade (Diagnostic Steps, step 1). Do not start a major version upgrade on an instance that is not `Running`. + +2. Take a backup of the instance and confirm that it completed. + + > **Warning:** A major version upgrade cannot be undone by selecting `PostgreSQL 14` again: a data directory upgraded to 16 cannot be opened by PostgreSQL 14. The instance is unavailable to clients while the server restarts on the new version. Precondition: a completed backup of the instance, a maintenance window, and applications that are tested against PostgreSQL 16. Rollback: restore the backup into a PostgreSQL 14 instance and point the applications at it. + +3. In the console, open the PostgreSQL instance, choose **Update**, and change the database version from `PostgreSQL 14` to `PostgreSQL 16`. Leave the other fields (replicas, CPU, memory, storage class) as they are, then confirm. + +4. Wait until the instance returns to `Running`, then verify the server version (Diagnostic Steps, step 2) and test the applications that use it. + +## Diagnostic Steps + +1. Check the current major version and status of the instances: + + ```bash + kubectl get postgresql -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,VER:.spec.postgresql.version,STATUS:.status.PostgresClusterStatus + ``` + + Before the upgrade the instance shows `VER` `14` and `STATUS` `Running`. After the update is submitted, `VER` shows `16`; wait until `STATUS` is `Running` again. A status such as `SyncFailed` or `UpdateFailed` means the change was not applied; inspect `kubectl describe postgresql -n ` before retrying. + +2. Confirm the version that the server actually runs: + + ```bash + kubectl exec -n -0 -- psql -U postgres -c 'SHOW server_version;' + ``` + + The output starts with `16`. A value starting with `14` means the server has not restarted on the new version yet. From beae376adbc5f6543c5abb91db50a71c2fbd4fc3 Mon Sep 17 00:00:00 2001 From: Komh Date: Wed, 18 Nov 2026 17:41:50 +0800 Subject: [PATCH 25/25] docs(solutions): add 1 and update 0 troubleshooting articles --- ...n_password_equal_to_the_username_on_ACP.md | 116 ++++++++++++++++++ 1 file changed, 116 insertions(+) create mode 100644 docs/en/solutions/Security_scan_reports_a_weak_Grafana_admin_password_equal_to_the_username_on_ACP.md diff --git a/docs/en/solutions/Security_scan_reports_a_weak_Grafana_admin_password_equal_to_the_username_on_ACP.md b/docs/en/solutions/Security_scan_reports_a_weak_Grafana_admin_password_equal_to_the_username_on_ACP.md new file mode 100644 index 000000000..927fdadf1 --- /dev/null +++ b/docs/en/solutions/Security_scan_reports_a_weak_Grafana_admin_password_equal_to_the_username_on_ACP.md @@ -0,0 +1,116 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Security scan reports a weak Grafana admin password equal to the username on ACP + +## Issue + +A security scan of a cluster's monitoring endpoint reports a weak password on the Grafana login page, for example on a URL of the form: + +```text +http://:11780/clusters//grafana/login +``` + +The finding says that the Grafana `admin` user accepts the password `admin`, the same as the username. Operators then ask whether Grafana has a password of its own, separate from the platform login, and where to change it. + +## Root Cause + +### Cause 1: the platform Grafana admin password is still `admin` + +The platform Grafana runs in the `cpaas-system` namespace of each cluster (Pods named `kube-prometheus-grafana-`). It has its own `admin` account, separate from platform users, and that account's password is stored in Grafana itself. When the password has not been changed, it is `admin`, the same as the username, and the scanner reports it as a weak credential. + +The platform also keeps the Grafana credentials in the Secret `kube-prometheus-grafana-basic-auth` in `cpaas-system`. The Secret has the keys `username`, `password`, `guestUsername` and `guestPassword`. In this state, `username` and `password` both decode to `admin`. The password in Grafana and the `password` value in this Secret must match. So a password change has to be made in both places, and the `prometheus` and `warlock` components of that cluster are restarted afterwards. + +## Resolution + +### Cause 1 + +Run the steps on the cluster named in the scanned URL (`/clusters//grafana`), with `kubectl` access to that cluster. Choose the new password first; `` below stands for it. + +1. Find the Grafana Pod: + + ```bash + kubectl -n cpaas-system get pod | grep kube-prometheus-grafana + ``` + +2. Back up the credentials Secret and decode its current values: + + ```bash + kubectl -n cpaas-system get secret kube-prometheus-grafana-basic-auth -o yaml > kube-prometheus-grafana-basic-auth.bak.yaml + kubectl -n cpaas-system get secret kube-prometheus-grafana-basic-auth -o jsonpath='{.data.username}' | base64 -d; echo + kubectl -n cpaas-system get secret kube-prometheus-grafana-basic-auth -o jsonpath='{.data.password}' | base64 -d; echo + ``` + + `admin` and `admin` confirm the finding. + +3. Reset the Grafana admin password inside the Grafana Pod. + + > **Warning:** Impact: from this step until the Secret is updated in step 4, the password in Grafana and the copy in the Secret differ. Platform features that log in to Grafana with the Secret can fail during that time. Precondition: the backup from step 2 exists, and you run step 4 right after this one. Rollback: run the same command again with the old password (the value decoded in step 2), then restore the Secret with `kubectl apply -f kube-prometheus-grafana-basic-auth.bak.yaml`. Rolling back brings the weak password back. + + ```bash + kubectl -n cpaas-system exec -it -- \ + grafana-cli --homepath=/usr/share/grafana admin reset-admin-password '' + ``` + + If the Pod has more than one container, add `-c ` (list the containers with `kubectl -n cpaas-system get pod -o jsonpath='{.spec.containers[*].name}'`). + +4. Write the same password into the Secret. Change only the `password` key, and leave `username`, `guestUsername` and `guestPassword` unchanged: + + ```bash + NEW_B64=$(echo -n '' | base64 -w0) + kubectl -n cpaas-system patch secret kube-prometheus-grafana-basic-auth \ + --type merge -p "{\"data\":{\"password\":\"${NEW_B64}\"}}" + kubectl -n cpaas-system get secret kube-prometheus-grafana-basic-auth -o jsonpath='{.data.password}' | base64 -d; echo + ``` + + The last command must print the new password exactly. + +5. Restart `prometheus` and `warlock` on the same cluster. + + > **Warning:** Impact: while the Pods restart, metric queries, monitoring charts and alert evaluation on this cluster are briefly unavailable, and a short gap can appear in the collected data. Precondition: steps 3 and 4 are done and the Secret prints the new password. Rollback: none needed for a restart, because the controllers recreate the Pods. Choose a low-traffic time. + + ```bash + kubectl get pods -A | grep -E 'warlock|prometheus' + kubectl -n delete pod + kubectl -n delete pod + kubectl get pods -A | grep -E 'warlock|prometheus' + ``` + + Delete the Pods of the `warlock` and Prometheus workloads that the first command prints. Do not delete the Grafana Pod in this step. Wait until the recreated Pods are `Running` and ready. + +6. Verify and rescan. In the commands below, `` is the scanned URL without the trailing `/login`, for example `http://:11780/clusters//grafana`: + + ```bash + curl -sk -u 'admin:admin' -o /dev/null -w '%{http_code}\n' '/api/user' + curl -sk -u 'admin:' -o /dev/null -w '%{http_code}\n' '/api/user' + ``` + + Expect `401` for `admin:admin` and `200` for the new password. Then open the cluster monitoring pages and Grafana from the platform console, check that they still load, and run the security scan again. + +## Diagnostic Steps + +1. Confirm that the platform Grafana runs on the scanned cluster: + + ```bash + kubectl -n cpaas-system get pod | grep kube-prometheus-grafana + kubectl -n cpaas-system get secret | grep grafana + ``` + + Expected: a `Running` `kube-prometheus-grafana-` Pod and the Secret `kube-prometheus-grafana-basic-auth`. If there is no Grafana Pod, the scanned address belongs to a different Grafana, not to the platform one in `cpaas-system`. If the Secret has another name, use the name printed here in the Resolution steps. + +2. Confirm the weak credential from a host that can reach the scanned URL: + + ```bash + curl -sk -u 'admin:admin' -o /dev/null -w '%{http_code}\n' '/api/user' + ``` + + Expected: `200` while the weak password is still active, and `401` after the fix. + +3. After the change, check that Grafana and the Secret hold the same password. Decode the Secret as in step 2 of the Resolution, then test that value against Grafana with the second `curl` in step 6. A `401` for the Secret's value means the two differ. Repeat step 4 with the password that was set in Grafana, then repeat step 5. + +4. If the console reports `dial tcp
:11780: i/o timeout` when it opens Grafana, the problem is network reachability of port `11780`, not the password. See [Workload cluster abnormal or monitoring empty in the console because global cannot reach its ports on ACP](Workload_cluster_abnormal_or_monitoring_empty_in_the_console_because_global_cannot_reach_its_ports_on_ACP.md).