diff --git a/docs/en/solutions/ACP_console_session_survives_cookie_deletion_or_a_browser_close_or_the_login_page_shows_the_previous_account.md b/docs/en/solutions/ACP_console_session_survives_cookie_deletion_or_a_browser_close_or_the_login_page_shows_the_previous_account.md new file mode 100644 index 000000000..3ffd40e0a --- /dev/null +++ b/docs/en/solutions/ACP_console_session_survives_cookie_deletion_or_a_browser_close_or_the_login_page_shows_the_previous_account.md @@ -0,0 +1,94 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.2" +--- + +# ACP console session survives cookie deletion or a browser close, or the login page shows the previous account + +## Issue + +The platform web console keeps, loses, or reuses a login in a way the user does not expect. One of these is seen: + +- After logging in and deleting the site's cookies, a page refresh keeps the account logged in; the user is not logged out. This is often found during a security assessment. +- With "end the session when all product tabs are closed" enabled, closing the browser and opening the platform again goes straight in without a login. +- Users can log in only after clearing the browser cache and restarting the browser. +- The login page says the license code has expired, although the license is still valid. +- After a direct local Dex login URL is opened, the form accepts account B but the session shown is still account A. +- The Service Mesh page says that no authorization token was provided, while the console login itself works. + +Related articles: "Forgotten local admin password on ACP: reset the password hash, set a chosen password, and password policy limits" covers local passwords. "kubectl returns the server has asked for the client to provide credentials after ac login, or ac login fails, on ACP" covers the CLI. + +## Root Cause + +### Cause 1: login session data is also kept in the browser's local storage + +The console stores data in the browser's Local Storage (keys such as `meepo-project` and `theme-mode`) as well as in cookies, and part of the login state is kept in Local Storage. Deleting only the cookies leaves that state, and the next page load restores the login. Clearing both cookies and Local Storage for the platform site logs the user out. + +### Cause 2: the session ends about 30 seconds after the last product tab closes + +On 4.2, Access Control offers "end the session when all product tabs are closed". The browser does not clear the session at the instant it closes; the front end calls the logout API through its multi-tab mechanism, and that call is delayed. The session therefore stays valid for about 30 seconds after the last product tab closes. Reopening within that window does not ask for a login; after it, a login is required. This is the designed behavior. + +### Cause 3: the login page sat idle until its session cookie expired + +The login page was left open with no activity until its session cookie expired, so submitting the stale form fails. Reopening the platform access URL, or clearing the cache, loads a fresh form. This is expected. + +### Cause 4: the license-expired message is stale page cache + +A license-expired message that does not appear in an incognito window comes from cached page content, not from the license. Retyping the platform address does not clear it. + +### Cause 5: a direct Dex login URL reuses the previous account's cached token + +On 4.2, account A logs in and the browser is closed without logging out. A direct local Dex URL (path `/console-dex/login/local` with `client_id=alauda-auth` and `redirect_uri` pointing at `/console-portal`) is then opened and account B is entered. The browser still holds A's token, and that URL uses the cached token instead of B's credentials, so the session is A's. Whether it reproduces depends on the browser. + +### Cause 6: the token held by a long-open page has expired + +On 3.18 the Service Mesh page reports that no authorization token was provided after the browser session has been open long enough for its token to expire. The page has nothing valid to send. A fresh login gets a new token. + +## Resolution + +### Cause 1: login session data is also kept in the browser's local storage + +1. Open the browser developer tools (F12) on the platform page, go to **Application**, and check both **Cookies** and **Local Storage** for the platform address. +2. Clear both for the platform site, or use **Clear site data**, then refresh. The user is logged out. +3. For a normal logout, use the console's logout action, which ends the session on the server instead of only removing browser data. + +### Cause 2: the session ends about 30 seconds after the last product tab closes + +No change is needed. Wait more than about 30 seconds after the last product tab closes before expecting a new login. This is not an idle-timeout setting. + +### Cause 3: the login page sat idle until its session cookie expired + +Open the platform access URL again, or clear the site cache, and log in. Do not treat this as a wrong password or a Dex fault when a fresh page accepts the same user. + +### Cause 4: the license-expired message is stale page cache + +Check the license validity date in the platform first. If it is still valid, open an incognito window, or clear the site cache, and log in. If a wrong-password error follows, it is a password problem, not the license. + +### Cause 5: a direct Dex login URL reuses the previous account's cached token + +Log out, or clear the site's cookies and Local Storage, before using `/console-dex/login/local` to switch accounts. No platform object needs to change. + +### Cause 6: the token held by a long-open page has expired + +Log out and log in again, or open the platform in an incognito window. The page then loads. No mesh configuration change is needed. + +## Diagnostic Steps + +1. On the platform page, open the developer tools (F12): + - **Application** > **Cookies** and **Local Storage**: note which entries remain for the platform address. + - **Network**: refresh, select a platform API request, and check whether it still carries `Cookie` or `Authorization`. +2. Open the same address in an incognito window and compare the behavior. + +| What you see | Cause | +| --- | --- | +| Cookies deleted, refresh keeps the login, Local Storage still has entries; clearing it logs out | 1 | +| Session survives a browser close for less than about 30 seconds with tab-close logout enabled | 2 | +| Login works only after a cache clear, the form was left idle | 3 | +| License-expired text, license still valid, incognito window is clean | 4 | +| Direct Dex URL logs in as the previous user | 5 | +| Service Mesh page says no authorization token, a fresh login fixes it | 6 | diff --git a/docs/en/solutions/ACP_control-plane_limits_apiserver_address_change_snapshot_db_scope_total_master_loss_x86_to_ARM_and_planned_shutdown.md b/docs/en/solutions/ACP_control-plane_limits_apiserver_address_change_snapshot_db_scope_total_master_loss_x86_to_ARM_and_planned_shutdown.md new file mode 100644 index 000000000..4a334a394 --- /dev/null +++ b/docs/en/solutions/ACP_control-plane_limits_apiserver_address_change_snapshot_db_scope_total_master_loss_x86_to_ARM_and_planned_shutdown.md @@ -0,0 +1,127 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# ACP control-plane limits: apiserver address change, snapshot.db scope, total master loss, x86 to ARM, and planned shutdown + +## Issue + +- A business-cluster apiserver VIP or URL has changed and the ACP console still uses the old address. +- A namespace was deleted and the only copy left is an etcd `snapshot.db` or `snapshot-etcd` file. +- All masters are lost and the only copy is the platform etcd backup; a version-specific command list for a full restore is requested. +- The `global` cluster runs on x86 and has to move to ARM. +- The whole cluster has a scheduled shutdown. + +These are procedure and product-limit questions, not a single-member etcd repair and not a database-full alarm. + +Related articles: "Replacing or rejoining an ACP master node: same-IP replacement, etcd-join, extra masters, and control-plane requests". + +## Root Cause + +### Cause 1: A business cluster apiserver address change has no standard platform procedure + +On platform versions 3.12, 3.18, and 4.2 there is no standard procedure to point the console or the `global` cluster at a new business-cluster apiserver VIP or address. + +### Cause 2: A deleted namespace cannot be restored on its own from snapshot.db + +A single namespace cannot be extracted from `snapshot.db` (running `strings` on the file does not produce the objects). The supported path is a restore of the whole cluster from an etcd backup, which returns every namespace to that backup's point in time. There is no reference procedure for starting a separate etcd just to read one namespace. + +### Cause 3: All masters lost with only an etcd backup has no supported restore + +There is no supported procedure to rebuild a cluster from the etcd backup alone after all masters are lost, on any version including 4.3, with or without bare-metal masters. An etcd snapshot stores cluster data but not the certificates and keys the cluster needs. The backup job can also save node files (`/etc/kubernetes/pki`, manifests, kubelet configuration, container runtime units), but those are version-specific; a cluster rebuilt from them is not a standard deployment and is likely to fail a later upgrade. A `snapshot-etcd` file is the platform backup, not a namespace extract and not a rebuild kit. + +### Cause 4: An x86 global cluster cannot be moved to ARM by replacing servers or importing etcd + +Replacing the servers does not change the cluster's CPU architecture, and redeploying an ARM `global` and importing the existing etcd data does not let it take over the existing business clusters. An ARM `global` is a new deployment plus a business migration. + +### Cause 5: A scheduled shutdown of the whole cluster + +A planned outage needs an etcd backup first and a normal `shutdown` rather than a power cut. Power the nodes off together when possible; otherwise shut down workers first and masters after. After power-on, platform components recover on their own. + +## Resolution + +### Cause 1: A business cluster apiserver address change has no standard platform procedure + +Do not create a procedure by rewriting kube-apiserver manifests or restoring etcd. Collect the old and new URLs and contact Alauda support for a version-specific procedure. The following prints the server URL a kubeconfig uses; it does not update the platform: + +```bash +kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}{"\n"}' +``` + +### Cause 2: A deleted namespace cannot be restored on its own from snapshot.db + +Restoring the namespace means restoring the whole cluster from the etcd backup, which is a planned-downtime decision for every namespace, not a partial extract. `file` only identifies the blob: + +```bash +file snapshot.db +``` + +### Cause 3: All masters lost with only an etcd backup has no supported restore + +Plan for this before it happens: + +1. Take full server or VM backups on a schedule, in addition to the etcd backup. +2. Protect master disks (for example with RAID) and repair failed disks promptly. +3. Run critical workloads on more than one independent cluster. + +Check how many etcd members are still running: + +```bash +kubectl -n kube-system get pods -l component=etcd -o wide +``` + +Zero running members with only `snapshot-etcd` or `snapshot.db` left is this limit. With one or more healthy members, the remaining members are repaired or replaced one at a time instead. + +### Cause 4: An x86 global cluster cannot be moved to ARM by replacing servers or importing etcd + +Plan a new ARM deployment and a migration of the business workloads. `uname -m` only confirms the current architecture: + +```bash +uname -m +kubectl get nodes -o wide +``` + +### Cause 5: A scheduled shutdown of the whole cluster + +Take the etcd backup before any power-off: run the backup on the platform's etcd backup task and confirm that a current backup record exists, or take a snapshot with `etcdctl snapshot save` from a healthy member. + +```bash +kubectl get etcdbackupconfiguration -A +``` + +> **Warning:** `shutdown -h now` powers this node off. The etcd backup must already exist. Run it on workers before masters when the nodes cannot go down together. Rollback is powering the node on again. A node that does not return after power-on is a new fault, not a step of this shutdown. + +```bash +sudo shutdown -h now +``` + +After power-on, wait for the platform components to recover and check: + +```bash +kubectl get nodes +kubectl get pods -A | grep -v -E 'Running|Completed' +``` + +## Diagnostic Steps + +```bash +kubectl -n kube-system get pods -l component=etcd -o wide +kubectl get nodes -l node-role.kubernetes.io/control-plane= -o wide +kubectl get nodes -o wide +``` + +- Address change of an existing business cluster on 3.12, 3.18, or 4.2: Cause 1. +- `snapshot.db` in hand and one missing namespace: Cause 2. +- Every master lost and the only copy is an etcd backup: Cause 3. +- x86 `global` to ARM by server swap or etcd import: Cause 4. +- Planned power-off of the whole cluster: Cause 5. diff --git a/docs/en/solutions/ACP_login_loops_or_returns_to_the_login_page_because_erebus_or_frontend_is_unhealthy.md b/docs/en/solutions/ACP_login_loops_or_returns_to_the_login_page_because_erebus_or_frontend_is_unhealthy.md new file mode 100644 index 000000000..053b83a21 --- /dev/null +++ b/docs/en/solutions/ACP_login_loops_or_returns_to_the_login_page_because_erebus_or_frontend_is_unhealthy.md @@ -0,0 +1,131 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.0" + - "4.2" +--- + +# ACP login loops or returns to the login page because erebus or frontend is unhealthy + +## Issue + +After the username and password are accepted, the browser returns to the login page, or it bounces between the platform and the SSO page until login fails. Every account is affected, local and LDAP alike, and the platform cannot be accessed at all. It often happens on the first morning of the work week. The frontend log names the global erebus API as abnormal, typically with: + +```text +oidc verify token from acp ... "error": "failed to send request: Get \"https://erebus/kubernetes/global/api\": dial tcp :443: connect: cannot assign requested address" +http: superfluous response.WriteHeader call from .../alauda-backend/pkg/server/options.(*ResponseWriterDelegator).WriteHeader (metrics.go:319) +``` + +Related articles: "ACP login shows an internal error from a bad Dex cache, or the standby global cluster cannot log in" covers an internal error on the login page and standby clusters. An `invalid authorization code` or `Unregistered redirect_uri` page is a different failure. + +## Root Cause + +### Cause 1: erebus is unhealthy and the console cannot finish login + +The erebus log reports that it cannot find cluster-related information, or the erebus Pod is otherwise abnormal, and the frontend log says the global erebus API is unhealthy. Restarting the erebus Pod restores login. + +A variant: erebus keeps looking up a cluster that has already been deleted from the platform, and its log repeats `websocket.go:514] Websocket Ping failed: write tcp :->:443: write: connection timed out` together with `handler.go:66] Handle request error: client rate limiter Wait returned an error: context canceled`. The source address belongs to the `archon` Pod in `cpaas-system`. Apollo looks healthy, and restarting Apollo does not help. erebus appears to hang while retrying the deleted cluster; the exact mechanism is not confirmed. Restarting erebus restores login for all users as a temporary workaround. + +### Cause 2: local ports are exhausted for the connections to erebus + +Login fails, and the erebus log says no port is available because short-lived TCP connections are exhausted. The frontend log shows the `connect: cannot assign requested address` line from the Issue section, and the erebus log is a continuous stream of `proxy.go:150] Generate new reverse proxy for: 600:https://:6443` mixed with `log.go:245] http: proxy error: context canceled` and `timeout.go:142] post-timeout activity` for GET requests such as `/api/v1/namespaces/cpaas-system/configmaps`. Restarting both frontend and erebus restores login. + +A related trace, before a restart, shows erebus logging `asyncopy.go:16] async copy data error: read tcp :9443->:: read: connection reset by peer`, and the `archon-api` container of the `archon` Pod logging `websocket.go:500] Websocket Ping failed: write tcp :->:443: write: connection timed out`. The Service still lists the erebus Pod as an endpoint, but the Pod is not reading. Another Pod on the same node that leaks connections can exhaust the node's ephemeral ports. One candidate is courier-api: when Operations Center log queries fail with `dial tcp :9200: connect: cannot assign requested address`, the courier-api network namespace can hold tens of thousands of established connections to the Elasticsearch endpoint on port 9200. This link was not proven, so collect evidence before restarting if it happens again. + +Clusters removed only from the platform UI can keep printing errors such as `cluster.go:284] cluster: failed to get cluster "" resource. err: clusters.clusterregistry.k8s.io "" not found` and `proxy.go:128] Resolve cluster error: ... not found`. Deleting a cluster in the UI drops platform management of that cluster and leaves its components in place. + +### Cause 3: a frontend socket leak turns a CAS retry loop into a dead console + +When the CAS session is not logged in, platform login retries on its own, over and over, and after enough retries the console stops accepting logins. ALB entries for console APIs point at frontend, and frontend logs that it cannot connect to erebus, with the `connect: cannot assign requested address` and `superfluous response.WriteHeader` lines. Yet the `erebus` Service in `cpaas-system` (ports `443/TCP,6060/TCP`) has both Pods `1/1 Running` with no restarts, and `curl` to erebus succeeds. The cause is a socket leak in frontend, seen on v3.18.2. Restarting frontend restores login. A product fix is expected in a version after 4.4. + +## Resolution + +### Cause 1: erebus is unhealthy and the console cannot finish login + +1. Confirm that local and LDAP logins fail the same way and that erebus is the abnormal component: + + ```bash + kubectl get deploy,po -n cpaas-system | grep -i erebus + kubectl logs -n cpaas-system --tail=200 | grep -E 'Websocket Ping failed|rate limiter|not found' + kubectl get pods -A -o wide | grep + kubectl get svc -A | grep + ``` + +2. Restart erebus. + + > **Warning:** Restarting erebus drops console calls that go through that API until the new Pod is Ready. Precondition: only when this cause is confirmed. The restart does not change configuration. Rollback: wait for the new Pod; there is no object to restore. + + ```bash + kubectl -n cpaas-system rollout restart deploy/ + kubectl -n cpaas-system rollout status deploy/ + ``` + +3. Retry a local user and an LDAP user. When erebus was retrying a deleted cluster, also remove the leftover components of that cluster (see Cause 2), because the restart is only a temporary workaround. + +### Cause 2: local ports are exhausted for the connections to erebus + +1. Confirm the port or timeout signature. If the outage is still in progress, collect the logs before restarting anything: + + ```bash + kubectl get deploy,po -A | grep -Ei 'erebus|archon|courier-api' + kubectl logs -n cpaas-system --tail=200 | grep -E 'connection reset by peer|no port|Generate new reverse proxy|post-timeout' + kubectl logs -n cpaas-system -c archon-api --tail=200 | grep -F 'connection timed out' + ``` + +2. To see which Pod is exhausting ports, count its connections inside its network namespace on the node where it runs (the example uses courier-api; the same commands work for frontend or erebus): + + ```bash + crictl ps -a | grep courier + crictl inspect | grep -m1 '"pid"' + nsenter -n -t ss -s + nsenter -n -t ss -ant | grep ':9200' | grep -c ESTAB + cat /proc/sys/net/ipv4/ip_local_port_range + ``` + + Tens of thousands of `ESTAB` connections to one endpoint, close to the size of `ip_local_port_range`, mean that Pod has used up the ephemeral ports. + +3. Restart erebus, and frontend when the log says no port is available. + + > **Warning:** Restarting erebus and frontend interrupts login for the length of the rollout. Precondition: only when the port-exhaustion or erebus/archon timeout lines above are present. Rollback: none beyond the new Pods becoming Ready. + + ```bash + kubectl -n cpaas-system rollout restart deploy/ + kubectl -n cpaas-system rollout restart deploy/frontend + ``` + +4. When clusters deleted in the UI are still emitting `clusters.clusterregistry.k8s.io "" not found` errors, remove the platform components that remain on the deleted cluster by following the product procedure for removing a cluster, then restart courier-api on the global cluster. Restarting courier-api drops notification and related API calls until the new Pod is Ready. + +### Cause 3: a frontend socket leak turns a CAS retry loop into a dead console + +1. Confirm that frontend cannot reach erebus while erebus itself is healthy: + + ```bash + kubectl logs -n cpaas-system deploy/frontend --tail=200 | grep -E 'cannot assign requested address|superfluous' + kubectl get pods -n cpaas-system | grep erebus + curl -k -sS -o /dev/null -w '%{http_code}\n' https:/// + ``` + +2. Restart frontend, not erebus. + + > **Warning:** Restarting `cpaas-system/frontend` makes the console unavailable until the rollout finishes; business workloads are not restarted. Precondition: only when this cause is confirmed. Rollback: wait for the rollout. The leak can return until the product fix is installed. + + ```bash + kubectl -n cpaas-system rollout restart deploy/frontend + kubectl -n cpaas-system rollout status deploy/frontend + ``` + +## Diagnostic Steps + +```bash +kubectl get deploy,po -n cpaas-system | grep -Ei 'erebus|frontend|apollo|archon|courier' +kubectl logs -n cpaas-system deploy/frontend --tail=80 +``` + +- Redirect loop or return to the login page for both local and LDAP users; erebus cannot find cluster information, retries a deleted cluster with `Websocket Ping failed ... connection timed out`, or its Pod is abnormal: Cause 1. +- erebus says no port is available or streams `Generate new reverse proxy for`, or erebus and archon log connection resets and write timeouts: Cause 2. Collect logs and `ss -s` output before restarting if the fault is still happening. +- CAS retries for a long time, frontend logs `cannot assign requested address` toward erebus, the erebus Pods are healthy, and `curl` to erebus works: Cause 3. Restart frontend. +- `kubectl get cls` on the global cluster shows a cluster abnormal: check whether the cluster kubeconfig Secrets have expired before restarting erebus or frontend. diff --git a/docs/en/solutions/ACP_login_shows_an_internal_error_from_a_bad_Dex_cache_or_the_standby_global_cluster_cannot_log_in.md b/docs/en/solutions/ACP_login_shows_an_internal_error_from_a_bad_Dex_cache_or_the_standby_global_cluster_cannot_log_in.md new file mode 100644 index 000000000..3bf47f63a --- /dev/null +++ b/docs/en/solutions/ACP_login_shows_an_internal_error_from_a_bad_Dex_cache_or_the_standby_global_cluster_cannot_log_in.md @@ -0,0 +1,126 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.0" + - "4.2" +--- + +# ACP login shows an internal error from a bad Dex cache, or the standby global cluster cannot log in + +## Issue + +One of these is reported: + +- The platform login page shows an internal error, and the Dex log repeats `ERROR error handling request {"error": "Internal error occurred: login error."}`. A variant shows `no matches for kind "Secret" in version "v1"` in the Dex log. +- On a primary/standby deployment the standby console does not log in. The browser developer tools show no error, and restarts of frontend, Apollo, and erebus do not help. + +Related articles: "ACP login loops or returns to the login page because erebus or frontend is unhealthy" covers redirect loops. An `invalid authorization code` or `Unregistered redirect_uri` page is a different failure. + +## Root Cause + +### Cause 1: Dex answers login from erroneous cached content + +The Dex log shows the captcha check passing (`check captcha verifyResult:{NeedReturnCaptcha:false IsLoginError:false CaptchaNotMatch:false CaptchaExpired:false}, err:`) and then, repeatedly: + +```text +ERROR decrypt password The encryption timestamp has expired +ERROR error handling request {"error": "Internal error occurred: login error."} +ERROR Failed to get auth request: authrequests.dex.coreos.com "" not found +Failed to get auth request: authrequests.dex.coreos.com "undefined" not found +WARN database inconsistent, refresh token missing: +``` + +or, in the variant, for each login: + +```text +error handling request {"error": "Internal error occurred: no matches for kind \"Secret\" in version \"v1\""} +decrypt password failed to login user: no matches for kind "Secret" in version "v1" +error handling request {"error": "Internal error occurred: login error."} +Failed to get auth request: authrequests.dex.coreos.com "" not found +``` + +Dex holds erroneous cached content and serves it on the login request; the exact mechanism is not confirmed. Restarting the Dex Pod on the global cluster drops the cache, and login works again. No ConfigMap edit is needed. Periodic `garbage collection run, delete auth requests=...` lines are normal background output. + +### Cause 2: the standby cluster token in `tke-cluster-global` does not match the current `k8sadmin` token + +On the standby global cluster, `cpaas-system` holds several rotated `kubernetes.io/service-account-token` Secrets named `k8sadmin-`, labeled `cpaas.io/managed-by: k8sadmin-token-rotator` and annotated with `cpaas.io/cluster-token-expires-at`. The Secret `tke-cluster-global` in `cpaas-system` (label `platform.tkestack.io/clustercredential: cc-global`) still carries a much older creation time, and its token no longer matches the newest `k8sadmin-` token. A request to `/kubernetes/global/api/v1/namespaces` with the `tke-cluster-global` token returns 403 `forbidden: User "system:anonymous" cannot get path ...`, which shows that the token is not accepted as any identity. Copying the current `k8sadmin` token into `tke-cluster-global` restores login. + +The Dex signing key `signingkeies.dex.coreos.com/openid-connect-keys` in `cpaas-system` has the same `signingKeyPub` and `verificationKeys` on both sides in this case, so the signing key is not the mismatch. + +## Resolution + +### Cause 1: Dex answers login from erroneous cached content + +1. Confirm the Dex log lines. On 4.x, Dex runs inside the `apollo` Deployment in `cpaas-system`; on a cluster that still has a `dex` Deployment, read and restart that one instead. + + ```bash + kubectl get deploy -n cpaas-system | grep -Ei 'apollo|dex' + kubectl logs -n cpaas-system deploy/apollo --tail=300 | grep -E 'login error|encryption timestamp|no matches for kind|Failed to get auth request|refresh token missing' + ``` + +2. Restart the workload that runs Dex. + + > **Warning:** Restarting Apollo (or Dex) interrupts logins until the Pod is Ready. Precondition: only when the page is an internal error and the Dex log shows the lines above. Rollback: wait for the rollout; the cache is discarded by design. + + ```bash + kubectl -n cpaas-system rollout restart deploy/apollo + kubectl -n cpaas-system rollout status deploy/apollo + ``` + +3. Log in again, and keep watching the Dex log for a recurrence. + +### Cause 2: the standby cluster token in `tke-cluster-global` does not match the current `k8sadmin` token + +Apply this only when restarts of frontend, Apollo, and erebus did not help, the Dex public keys match on both sides, and the `tke-cluster-global` token is rejected while a current `k8sadmin-` token exists. + +1. Compare the Secrets on the standby global cluster: + + ```bash + kubectl get signingkeies.dex.coreos.com openid-connect-keys -n cpaas-system -o yaml + kubectl -n cpaas-system get secret --sort-by=.metadata.creationTimestamp | grep k8sadmin + kubectl -n cpaas-system get secret tke-cluster-global -o yaml | grep -E 'creationTimestamp|clustercredential' + ``` + +2. Decode the `tke-cluster-global` token and test it against the platform proxy. List the keys first and use the key that holds the token: + + ```bash + kubectl -n cpaas-system get secret tke-cluster-global -o go-template='{{range $k,$v := .data}}{{$k}}{{"\n"}}{{end}}' + TOKEN=$(kubectl -n cpaas-system get secret tke-cluster-global -o jsonpath='{.data.}' | base64 -d) + curl -k -sS -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer ${TOKEN}" \ + https:///kubernetes/global/api/v1/namespaces + ``` + + A `403` that names `system:anonymous`, or a `401`, means the token is not valid. Compare it with the `token` key of the newest `k8sadmin-` Secret: + + ```bash + kubectl -n cpaas-system get secret -o jsonpath='{.data.token}' | base64 -d; echo + ``` + +3. Replace the token in `tke-cluster-global`. + + > **Warning:** Replacing the token in `tke-cluster-global` changes the credential that the platform uses for the global cluster; a wrong value keeps login broken. Precondition: only when the decoded tokens differ and the other checks above have ruled out erebus and the Dex signing key. Rollback: `kubectl apply -f /tmp/tke-cluster-global.bak.yaml`. + + ```bash + kubectl -n cpaas-system get secret tke-cluster-global -o yaml > /tmp/tke-cluster-global.bak.yaml + kubectl -n cpaas-system edit secret tke-cluster-global + ``` + + Set its token to the token from the newest `k8sadmin-` Secret, keeping the encoding the key already uses (base64 under `data`), and save. + +4. Log in to the standby console again. The portal loads normally. + +## Diagnostic Steps + +```bash +kubectl get deploy,po -n cpaas-system | grep -Ei 'apollo|dex|frontend|erebus' +kubectl -n cpaas-system get secret | grep -E 'tke-cluster-global|k8sadmin' +kubectl get signingkeies.dex.coreos.com openid-connect-keys -n cpaas-system +``` + +- Login internal error, and the Dex log shows `Internal error occurred: login error.` with `The encryption timestamp has expired` and `Failed to get auth request`: Cause 1. +- Login internal error, and the Dex log shows `decrypt password failed to login user: no matches for kind "Secret" in version "v1"` after a passing captcha check: Cause 1. +- Standby login still fails after frontend, Apollo, and erebus restarts, and the `tke-cluster-global` token is rejected while a newer `k8sadmin-` exists: Cause 2. diff --git a/docs/en/solutions/ACP_upgrade_stuck_on_an_operator_OLM_catalog_or_operator-managed_component.md b/docs/en/solutions/ACP_upgrade_stuck_on_an_operator_OLM_catalog_or_operator-managed_component.md new file mode 100644 index 000000000..5f85fff72 --- /dev/null +++ b/docs/en/solutions/ACP_upgrade_stuck_on_an_operator_OLM_catalog_or_operator-managed_component.md @@ -0,0 +1,261 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.16" + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# ACP upgrade stuck on an operator, OLM catalog, or operator-managed component + +## Issue + +The platform upgrade waits on an operator-managed component and does not progress. One of the following is visible: + +- `olm-registry-custom`, `olm-registry-platform`, or `olm-registry-system` is in `CrashLoopBackOff` with `unable to load in-cluster config`. +- The ARS (AppRelease) stays unready after an operator upgrade, and the `sentry` log shows a version-check Job that still checks the previous operator version. +- Argo CD `minfo` (ModuleInfo) is Failed during the upgrade to 4.0.9 and the Argo CD wait Job has no logs. +- The upgrade page stays in the Katanomi phase. +- The ASM (service mesh) upgrade fails with `bundles outdated`. +- The platform upgrade is stuck at the Tekton component. +- `redis-operator` stays in processing on the upgrade page. +- `rds-operator-updater` is stuck, or several components wait for the `rds-operator` upgrade. +- Argo CD is not ready after `bash upgrade.sh` has completed. + +## Root Cause + +### Cause 1: olm-registry Pods crash with unable to load in-cluster config + +The three `olm-registry-*` Pods crash with `unable to load in-cluster config`, and `kubectl get rpch` shows a ResourcePatch (`rpch`) that targets ConfigMap `olm-registry-config` in `cpaas-system`. The local override of that ConfigMap breaks the registry Pods. + +### Cause 2: The version-check Job still validates the previous operator version + +After an operator upgrade the ARS stays unready, and the `sentry` log shows the Job that checks the operator version and status still pinned to the previous version (for example the Job checks `v4.2.0` while `argocd-operator` is already `v4.2.1`). Editing the Job in place does not change what its Pod runs. + +### Cause 3: Argo CD minfo is Failed because the installed operator package is the wrong build + +During the upgrade to 4.0.9, the Argo CD `minfo` is Failed, the ARS error names GitOps Operator `v4.1.8`, and the Argo CD wait Job has no logs. The cluster has Argo CD Operator `4.2.1` loaded, while that upgrade requires package `4.2.0`. The version pair is specific to the 4.0.9 upgrade; for another target, compare the version named in the ARS error with the version shipped in the upgrade package. + +### Cause 4: Katanomi update is stuck on a historical Helm release + +The upgrade remains in the Katanomi phase. `helm ls -A` lists release `katanomi-global` in `cpaas-system` from the previous product generation, and the `katanomis.operators.katanomi.dev` object still points to a CatalogSource named `stable` that `kubectl get catalogsource -A` does not show. + +### Cause 5: The mesh upgrade reports bundles outdated + +The ASM upgrade fails with `bundles outdated`, although the mesh images are present on both the `global` cluster and the business cluster. Restarting `catalog-operator`, `packageserver`, `olm-operator`, and `olm-registry-system` unblocks OLM. Which of the four processes held stale data is not known. + +### Cause 6: The v3 tekton-operator and the v4 Pipeline operator both manage the same Tekton instance + +On a 4.1.2 to 4.2.1 platform upgrade the upgrade stays on the Tekton component. Both the v3 Tekton operator (`tekton-operator`) and the v4 Pipeline operator are installed and reconcile the same Tekton instance. v3 Tekton and v4 Pipeline cannot coexist; uninstalling the v3 `tekton-operator`, without changing the instance itself, clears the block. + +### Cause 7: redis-operator stays processing because its Subscription requires manual approval + +During a 4.1.3 to 4.2.1 upgrade, after the plugin packages were uploaded and `upgrade.sh` was run, `redis-operator` stays in processing. Its OperatorHub details page (Alauda Cache Service for Redis OSS) shows the old installed CSV (for example `redis-operator.v4.1.1` in `redis-system`, status `Succeeded`, channel `alpha`) and a banner that a new version is detected. The Subscription's upgrade strategy is manual, so the platform upgrade does not upgrade the operator; the new version waits for confirmation. + +### Cause 8: rds-operator or rds-operator-updater blocks the operators that follow it + +- `rds-operator-updater` is stuck (for example on a 4.1.3 to 4.1.4 upgrade). Deleting its Subscription lets the upgrade continue. +- On a workload-cluster upgrade, several components wait for the `rds-operator` upgrade. Uninstalling `rds-operator` and deleting its `minfo` and Subscription lets the upgrade continue. Why the `rds-operator` upgrade blocks the others is not known. + +If the next operators then stall because their Pods cannot be created on a node, that is a separate node-level problem. + +### Cause 9: Argo CD is not ready after upgrade.sh completes + +`bash upgrade.sh` finished, but Argo CD is not ready. This state can be ignored at this point; start the `global` cluster upgrade directly. The cause of the not-ready state has not been determined, so check Argo CD again after the global upgrade. + +## Resolution + +Match one signature before changing the cluster, and save every object before deleting it. + +### Cause 1: olm-registry Pods crash with unable to load in-cluster config + +> **Warning:** Deleting this `rpch` removes the local override of ConfigMap `olm-registry-config`; images or endpoints that the override forced fall back to the chart defaults. Precondition: only when the CrashLoop log is `unable to load in-cluster config` and the `rpch` targets `cpaas-system/olm-registry-config`. Rollback: `kubectl apply -f /tmp/rpch-olm.bak.yaml`. + +```bash +kubectl get rpch -o yaml > /tmp/rpch-olm.bak.yaml +kubectl get rpch | grep olm +kubectl delete rpch +kubectl get pods -n cpaas-system | grep olm-registry +kubectl delete pod -n cpaas-system +``` + +The replacement Pods should load the in-cluster config. + +### Cause 2: The version-check Job still validates the previous operator version + +> **Warning:** Deleting the version-check Job stops its Pod. Precondition: only when the `sentry` log quotes the old version inside that Job and an in-place edit of the Job did not change the check. Rollback: the controller recreates the Job. If the new Job still pins the old version, the installed package is wrong; use Cause 3. + +```bash +kubectl get jobs -A +kubectl delete job -n +``` + +Watch the ARS. Ready means this Job was the blocker. + +### Cause 3: Argo CD minfo is Failed because the installed operator package is the wrong build + +For the 4.0.9 upgrade, upload Argo CD Operator package `4.2.0` with `violet`, remove package `4.2.1`, and let the Argo CD `minfo` reconcile. For another target version, upload the operator package version that the upgrade bundle requires and remove the other build that the ARS error shows being installed. + +### Cause 4: Katanomi update is stuck on a historical Helm release + +> **Warning:** This deletes Helm release `katanomi-global` and its objects and removes the CatalogSource `stable` reference from the Katanomi object. Precondition: only when the upgrade is stuck in the Katanomi phase, `helm ls -n cpaas-system` shows `katanomi-global`, and no CatalogSource named `stable` exists. Save the Helm manifest first; the historical release is not reinstalled afterwards. Rollback of the Katanomi object: `kubectl apply -f /tmp/katanomi.bak.yaml`. + +```bash +helm get all -n cpaas-system katanomi-global > /tmp/katanomi-global.helm.txt +kubectl get katanomis.operators.katanomi.dev -A -o yaml > /tmp/katanomi.bak.yaml +helm delete -n cpaas-system katanomi-global +``` + +Edit the `katanomis.operators.katanomi.dev` object: remove the spec fields that name CatalogSource `stable`, and set an annotation so the object changes (for example `cpaas.io/reconcile-at` with the current UTC time). Then restart the catalog Pods: + +```bash +kubectl get pods -n cpaas-system | grep -E 'packageserver|catalog' +kubectl delete pod -n cpaas-system +``` + +Katanomi enqueues its own upgrade after those Pods are Ready. + +### Cause 5: The mesh upgrade reports bundles outdated + +Confirm that the mesh images are on the `global` cluster and the business cluster, then restart the four OLM processes. + +> **Warning:** Catalog resolution pauses until the replacement Pods are Ready, and in-flight InstallPlans retry. Precondition: only when the upgrade reports `bundles outdated` and the images are already present on both clusters. Rollback: none needed; the controllers recreate the Pods. + +```bash +kubectl get pods -n cpaas-system | grep -E 'catalog-operator|packageserver|olm-operator|olm-registry-system' +kubectl delete pod -n cpaas-system +``` + +### Cause 6: The v3 tekton-operator and the v4 Pipeline operator both manage the same Tekton instance + +List both operators and the instance they share: + +```bash +kubectl get csv -A | grep -Ei 'tekton|pipeline' +kubectl get subscriptions.operators.coreos.com -A | grep -Ei 'tekton|pipeline' +kubectl get deploy -A | grep -Ei 'tekton-operator|pipeline' +kubectl api-resources | grep -i tekton +``` + +For upstream-based Tekton operators the shared instance is the `TektonConfig` object (`operator.tekton.dev`); read it so you know what has to stay. + +> **Warning:** The v3 `tekton-operator` stops reconciling Tekton and the v4 Pipeline operator becomes the only manager of the instance. Precondition: only when both operators are installed and the upgrade is stuck at the Tekton component. Keep the Tekton instance, its CRDs, and its namespaces; remove the v3 operator only. Rollback: `kubectl apply -f` the saved objects, which installs the v3 operator again and brings back the conflict. + +When the v3 operator is installed through OLM (the commands above list a Subscription and CSV for it), remove those two objects: + +```bash +kubectl get subscriptions.operators.coreos.com -n -o yaml > /tmp/sub-tekton-operator.bak.yaml +kubectl get csv -n -o yaml > /tmp/csv-tekton-operator.bak.yaml +kubectl delete subscriptions.operators.coreos.com -n +kubectl delete csv -n +``` + +Deleting a CSV leaves the CRDs and custom resources in place. When the v3 operator was not installed through OLM, uninstall it the way it was installed, again keeping the instance. The Tekton stage moves once only the v4 Pipeline operator manages the instance. + +### Cause 7: redis-operator stays processing because its Subscription requires manual approval + +In OperatorHub, search for `redis-operator` and click **Confirm** on the new-version banner of its details page. To upgrade every operator that waits in the same way at once, use the batch upgrade under Cluster Management > Cluster > Feature Components. + +The same state from the command line: + +```bash +kubectl get subscriptions.operators.coreos.com -n redis-system -o yaml | grep -n -E 'installPlanApproval|currentCSV|installedCSV|state' +kubectl get installplan -n redis-system +``` + +`installPlanApproval: Manual` and an InstallPlan with `APPROVED` `false` are the signature. Approving that InstallPlan performs the same upgrade: + +> **Warning:** The Redis operator is upgraded to the new CSV, and the instances it manages may be reconciled by the new operator version. Precondition: only when `redis-operator` is stuck in processing, its Subscription is `Manual`, and the InstallPlan names the target version. Check the Redis instance prerequisites for the operator upgrade first. Rollback: treat this as one-way; OLM does not downgrade an approved InstallPlan. + +```bash +kubectl patch installplan -n redis-system --type=merge -p '{"spec":{"approved":true}}' +kubectl get csv -n redis-system +``` + +When the new CSV is `Succeeded`, check that the upgrade page has moved past `redis-operator`. + +### Cause 8: rds-operator or rds-operator-updater blocks the operators that follow it + +```bash +kubectl get subscriptions.operators.coreos.com -A | grep -i rds +kubectl get csv -A | grep -i rds +kubectl get minfo | grep -i rds +``` + +> **Warning:** Removing the Subscription stops OLM from managing that operator's upgrades. Uninstalling `rds-operator` stops reconciliation of the RDS-managed instances (failover, scaling, backup) until it is installed again; running database Pods are not deleted. Precondition: only when the upgrade is stuck on `rds-operator-updater` or on components that wait for `rds-operator`. Back up each object first. Rollback: `kubectl apply -f` the backups, or install `rds-operator` again from OperatorHub after the platform upgrade. + +For a stuck `rds-operator-updater`, delete its Subscription: + +```bash +kubectl get subscriptions.operators.coreos.com -n -o yaml > /tmp/sub-rds-operator-updater.bak.yaml +kubectl delete subscriptions.operators.coreos.com -n +``` + +When components wait for the `rds-operator` upgrade, uninstall `rds-operator` (from OperatorHub, or by deleting its Subscription and CSV), then delete its `minfo` on the `global` cluster: + +```bash +kubectl get subscriptions.operators.coreos.com -n -o yaml > /tmp/sub-rds-operator.bak.yaml +kubectl get csv -n -o yaml > /tmp/csv-rds-operator.bak.yaml +kubectl delete subscriptions.operators.coreos.com -n +kubectl delete csv -n +kubectl get minfo -o yaml > /tmp/minfo-rds-operator.bak.yaml +kubectl delete minfo +``` + +If RDS-managed instances run on the cluster, install the `rds-operator` version that matches the new platform version once the upgrade is complete. + +### Cause 9: Argo CD is not ready after upgrade.sh completes + +No change is needed at this point. Start the `global` cluster upgrade from the upgrade page, then check Argo CD again: + +```bash +kubectl get minfo | grep -i argo +kubectl get pods -A | grep -i argocd | grep -Ev 'Running|Completed' +``` + +If the Argo CD `minfo` is Failed after the global upgrade and the ARS error names an operator build other than the one in the upgrade package, use Cause 3. + +## Diagnostic Steps + +1. Note the component the upgrade page waits on, and whether it is installed through OLM (a CatalogSource, an operator package, or a CSV). +2. Read the OLM and operator objects on the stuck cluster: + + ```bash + kubectl get pods -n cpaas-system | grep -E 'olm-registry|olm-operator|catalog-operator|packageserver' + kubectl get catalogsource -A + kubectl get csv -A | grep -Ev 'Succeeded' + kubectl get rpch | grep -i olm + kubectl get jobs -A | grep -Ei 'check|wait' + helm ls -A | grep -i katanomi + ``` + +3. Match the first finding: + +| What you see | Cause | +| --- | --- | +| `unable to load in-cluster config` in an olm-registry Pod, and an `rpch` targets `olm-registry-config` | 1 | +| ARS unready after an operator upgrade; the version-check Job still quotes the previous version | 2 | +| Argo CD `minfo` Failed, empty wait Job, an operator build other than the one in the upgrade package is loaded | 3 | +| Katanomi phase, Helm release `katanomi-global`, CatalogSource `stable` missing | 4 | +| `bundles outdated` with the mesh images already present | 5 | +| Stuck at Tekton, both the v3 `tekton-operator` and the v4 Pipeline operator installed | 6 | +| `redis-operator` processing, new-version banner in OperatorHub, Subscription `installPlanApproval: Manual` | 7 | +| `rds-operator-updater` stuck, or several components waiting for `rds-operator` | 8 | +| Argo CD not ready right after `bash upgrade.sh`, before the global upgrade | 9 | + +4. After the change, the operator's CSV should reach `Succeeded`, the ARS should become Ready, and the upgrade page should move to the next stage. + +A message such as `redis-operator is not ready for version ` while another component (for example CoreDNS) is being upgraded, with the installed Redis operator already on a newer version, is not Cause 7. Compare the installed CSVs, the `minfo` versions, and `installedCSV` against `currentCSV` on the Subscription before acting: + +```bash +kubectl get csv -A | grep -i redis-operator +kubectl get minfo -A | grep -iE 'coredns|redis-operator' +kubectl get subscriptions.operators.coreos.com -A -o yaml | grep -n -E 'name:|installPlanApproval|currentCSV|installedCSV|state' +``` diff --git a/docs/en/solutions/ALB_TLS_1.0_and_1.1_and_HTTP_port_80_access_to_the_platform_console_on_ACP.md b/docs/en/solutions/ALB_TLS_1.0_and_1.1_and_HTTP_port_80_access_to_the_platform_console_on_ACP.md new file mode 100644 index 000000000..d7e133cf0 --- /dev/null +++ b/docs/en/solutions/ALB_TLS_1.0_and_1.1_and_HTTP_port_80_access_to_the_platform_console_on_ACP.md @@ -0,0 +1,131 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "4.1" + - "4.2" + - "4.3" +--- + +# ALB TLS 1.0 and 1.1, and HTTP port 80 access to the platform console on ACP + +## Issue + +- A compliance scan requires TLS 1.0 and TLS 1.1 to be disabled, and the question is where the ALB's nginx configuration can be checked. +- The platform console can only be opened over HTTPS on 443; `http://` on port 80 does not work, and the administrator wants HTTP access back, or asks whether port 80 can still be opened after the platform is deployed. +- A script calls a platform API such as `/apis/platform.tkestack.io/v1/clusters` over HTTP and gets `404` on one platform while the same call works on another. + +These are settings of the ALB data plane and of the platform configuration, not of a single rule. + +## Root Cause + +### Cause 1: ssl_protocols in the ALB ConfigMap enables TLSv1 and TLSv1.1 + +The protocols the ALB accepts are set by `ssl_protocols` in the ConfigMap that has the same name as the ALB (namespace `cpaas-system` of the cluster where the ALB runs), for example: + +```nginx +ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; +``` + +`TLSv1` and `TLSv1.1` in that line mean TLS 1.0 and TLS 1.1 are enabled. + +### Cause 2: forceRedirectHttps: true and httpPort: 0 in the prdb base + +Whether the console is served on port 80 is decided by the `http` section of the `base` ProductBase (`kubectl get prdb base -o yaml`). With `forceRedirectHttps: true`, `httpPort: 0` and `httpsPort: 443`, only HTTPS is served; the platform ALB `global-alb2` follows it (its Deployment carries the environment variable `DEFAULT-SSL-STRATEGY` with value `Always`), and there is no `global-alb2-00080` frontend. Changing these two fields after deployment opens port 80. + +This concerns the platform console only. Do not set `forceRedirectHttps` to `false` to change redirects of business listeners. The platform console protocol is also unrelated to the access protocol of a business Harbor instance. + +### Cause 3: 404 with X-ALB-ERR-REASON: InvalidUpstream : no rule match on port 80 of global-alb2 + +`curl -v http:///apis/platform.tkestack.io/v1/clusters` returns `HTTP/1.1 404 Not Found` with `X-ALB-ERR-REASON: InvalidUpstream : no rule match` and `Via: alb`, and the `global-alb2` log shows the request with `404` and `no rule match` for the port 80 policy. On that platform the `erebus` rules for `/api`, `/apis`, `/kubernetes/` and `/tunnel` exist only on port 443 (`global-alb2-00443-*`) because port 80 is not enabled, so HTTP requests match no rule. A platform where the call works has the same rules on both `global-alb2-00080-*` and `global-alb2-00443-*`. + +## Resolution + +### Cause 1: ssl_protocols in the ALB ConfigMap enables TLSv1 and TLSv1.1 + +Remove `TLSv1` and `TLSv1.1` from `ssl_protocols` and restart the ALB Pods. Make the edit in the overwrite ConfigMap so the next reconcile does not restore the old line (see "ALB nginx settings are reverted after a restart or upgrade; change them through the overwrite ConfigMap on ACP"). Keep the newer protocols the line already lists: + +```nginx +ssl_protocols TLSv1.2 TLSv1.3; +``` + +> **Warning:** Clients that only speak TLS 1.0 or 1.1 can no longer connect to any HTTPS port of this ALB, and each restart drops connections on that replica. Check client compatibility and back up the overwrite ConfigMap first. Restoring the previous line and restarting again is the rollback. + +```bash +kubectl -n cpaas-system get pods -o wide | grep +kubectl -n cpaas-system delete pod +``` + +Delete one ALB Pod at a time and wait for its replacement to be Ready before the next. + +### Cause 2: forceRedirectHttps: true and httpPort: 0 in the prdb base + +On the global cluster: + +1. Choose the HTTP port (for example `80`) and add a forwarding rule on the platform address (VIP or external load balancer) that sends that port to the three global masters. +2. Back up the ProductBase and the platform ALB: + + ```bash + kubectl get prdb base -o yaml > /tmp/prdb-base.backup.yaml + kubectl -n cpaas-system get deploy global-alb2 -o yaml > /tmp/global-alb2-deploy.backup.yaml + ``` + +3. Edit the ProductBase and set `spec.http.httpPort: 80` and `spec.http.forceRedirectHttps: false`, then wait until every `ars` in `cpaas-system` is ready. + + > **Warning:** The platform releases reconcile, the console then also answers in clear text on port 80, and the console can be briefly unreachable while `global-alb2` restarts. Do it in a window and only where HTTP access to the console is acceptable on that network. Roll back with `kubectl apply -f /tmp/prdb-base.backup.yaml` and wait for the `ars` again. + + ```bash + kubectl edit prdb base + kubectl get ars -n cpaas-system + ``` + +4. Check that the port 80 frontend exists and that `DEFAULT-SSL-STRATEGY` changed. If not, restart the `global-alb2` Pods one at a time. + + ```bash + kubectl get ft -n cpaas-system | grep global-alb2 + kubectl -n cpaas-system get deploy global-alb2 -o yaml | grep -n -A1 DEFAULT-SSL-STRATEGY + kubectl -n cpaas-system delete pod + ``` + +5. Log in over both `http://` and `https://`. + +### Cause 3: 404 with X-ALB-ERR-REASON: InvalidUpstream : no rule match on port 80 of global-alb2 + +Send the request over HTTPS on port 443, where the `erebus` rules exist. If HTTP on port 80 is required for the platform, enable it with Cause 2, which creates the `global-alb2-00080` frontend and its rules. + +```bash +curl -k -sS -o /dev/null -w '%{http_code}\n' -H 'Authorization: Bearer ' https:///apis/platform.tkestack.io/v1/clusters +``` + +## Diagnostic Steps + +1. Read the enabled protocols and test the handshake: + + ```bash + kubectl -n cpaas-system get configmap -o yaml | grep -C 10 ssl_protocols + openssl s_client -connect 192.0.2.10:443 -servername app.example.com -tls1_1 /apis/platform.tkestack.io/v1/clusters 2>&1 | grep -iE '^< HTTP|x-alb-err-reason|^< via' + kubectl get rule -A | grep erebus + ``` + + `no rule match` and only `global-alb2-00443-*` rules for `erebus`: Cause 3. diff --git a/docs/en/solutions/ALB_alb_loop_time_alert_from_cpaas-alb-rules_and_platform_alerts_raised_with_it_on_ACP.md b/docs/en/solutions/ALB_alb_loop_time_alert_from_cpaas-alb-rules_and_platform_alerts_raised_with_it_on_ACP.md new file mode 100644 index 000000000..72d2d1da5 --- /dev/null +++ b/docs/en/solutions/ALB_alb_loop_time_alert_from_cpaas-alb-rules_and_platform_alerts_raised_with_it_on_ACP.md @@ -0,0 +1,89 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.1.x + - 4.3.x +--- + +# ALB alb_loop_time alert from cpaas-alb-rules and platform alerts raised with it on ACP + +## Issue + +The real-time alert page shows policy `cpaas-alb-rules` with `alb_loop_time` above 1000 ms (`alb_loop_time超过1000.00ms`), for example a trigger value of `5534.00`, often recovering by itself. A general request to troubleshoot the platform alerts (`告警问题排查处理`) frequently bundles it with other alerts from the same period: the ALB `error-rate` alert, compute component restarts, compute component CPU above 90%, Elasticsearch disk usage above 70%, and node disk usage. In one such set, Elasticsearch disk usage was above 70% on only one `global` node while the others stayed low. + +Related article: [ALB error-rate alert on a route or Ingress from cpaas-alb-router-rules on ACP](ALB_error-rate_alert_on_a_route_or_Ingress_from_cpaas-alb-router-rules_on_ACP.md). + +## Root Cause + +### Cause 1: alb_loop_time is the duration of one ALB control-plane loop + +`alb_loop_time` measures one loop of the ALB control plane, not business request latency. A loop fetches resources, generates the NGINX configuration, reloads it (including writing `nginx.conf` to disk) and updates rule status. A loop longer than 1 second is typically caused by a slow kube-apiserver request; slow disk on the node can also stretch the reload step. The alert is warning level (`alb_loop_time > 1000ms` for 30 seconds). A spike can occur on a node with very low CPU, memory and load, and the specific cause of a single spike is not always identifiable afterwards, especially when the ALB logs of that window were not kept. + +A single alert row that recovered within about 30 seconds after it appeared, with users able to reach the service and no ALB Pod restart, can be recorded and ignored. A row that stays up for minutes is not covered by that rule. + +### Cause 2: Elasticsearch disk usage above 70% on one global node because a disk is mounted on the wrong path + +Only one `global` node reports high Elasticsearch disk usage, and the other Elasticsearch nodes are low. On that global master node a disk is mounted on the wrong path, which makes its usage uneven with the other nodes. This is not an ALB fault, and no ALB change fixes it. + +## Resolution + +### Cause 1: alb_loop_time is the duration of one ALB control-plane loop + +1. Read the alert value and how long the row stayed, and check the ALB Pods and their node: + + ```bash + kubectl get pods -n cpaas-system -o wide | grep -E 'alb2|alb' + ``` + +2. If the row fired once, recovered within about 30 seconds, users were not blocked and the ALB Pods did not restart, record it and leave the policy. The "about 30 seconds" is how fast the row cleared, not the `for 30 seconds` clause of the rule. + +3. If it repeats or stays up, save the `alb2` container log of every replica before the Pods roll, and check the API server latency and the disk latency of the node in the same window: + + ```bash + kubectl logs -n cpaas-system -c alb2 --since=1h > /tmp/-alb2.log + ``` + +Do not delete the ALB Pod to clear a single short spike. + +### Cause 2: Elasticsearch disk usage above 70% on one global node because a disk is mounted on the wrong path + +1. Locate the Elasticsearch Pods and their nodes, then compare the block devices and filesystems on each node: + + ```bash + kubectl get pods -A -o wide | grep -i elas + lsblk + df -Th | less + ``` + + On the high node, `lsblk` and `df -Th` show the data disk mounted on a different path from the other nodes. + +2. Migrate the data and remount the disk on the correct path in a maintenance window. + + > **Warning:** Data on that path is unavailable during the move, and a mistake in the copy or the mount can lose it. Precondition: one node is high, the others are low, `lsblk` and `df -Th` show the wrong mount, a verified copy of the data exists, and `/etc/fstab` is saved. Rollback: restore the saved `/etc/fstab` entry, remount, and copy the data back; data removed without a copy cannot be recovered. + +3. Afterwards `df -Th` on that node shows the disk on the correct path, and Elasticsearch usage is in line with the other nodes. + +## Diagnostic Steps + +1. Read the policy name. `cpaas-alb-rules` and `alb_loop_time` are Cause 1. `cpaas-alb-router-rules` and `error-rate` are the related article. + +2. One firing that recovered within about 30 seconds, no user-facing failure, ALB Pods with no new restarts: record and leave the warning. + +3. When several platform alerts arrive together, handle each separately: + - ALB `error-rate`: the 4xx/5xx ratio of ALB requests over the last 5 minutes passed the threshold, typically a backend that was stopped while its rule still received requests, or a backend that returns 4xx/5xx itself. Check the ALB and backend logs; see the related article. + - `alb_loop_time > 1000ms`: Cause 1. + - Compute component restarts: read the Pod events and the previous container log for the last restart reason, such as OOM: + + ```bash + kubectl describe pod -n | grep -A5 'Last State' + kubectl logs -n --previous --tail=100 + ``` + + If both are normal, the alert can be ignored. + - Compute component CPU above 90%: raise the CPU limit of that StatefulSet or Deployment, for example to double its current value. Changing the limit restarts its Pods; keep the previous value to roll back. + - Elasticsearch disk above 70% on every node: check and expand the disks. On only one node: Cause 2. + - Node disk usage: on the node, find the full filesystem with `df -Th`, and clean it up or expand it. diff --git a/docs/en/solutions/ALB_alb_lua_shared_dict_free_or_alb_loop_time_alerts_and_internal_dsl_is_null_or_fetch_policy_fail_log_lines_on_ACP.md b/docs/en/solutions/ALB_alb_lua_shared_dict_free_or_alb_loop_time_alerts_and_internal_dsl_is_null_or_fetch_policy_fail_log_lines_on_ACP.md new file mode 100644 index 000000000..3074b7791 --- /dev/null +++ b/docs/en/solutions/ALB_alb_lua_shared_dict_free_or_alb_loop_time_alerts_and_internal_dsl_is_null_or_fetch_policy_fail_log_lines_on_ACP.md @@ -0,0 +1,116 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# ALB alb_lua_shared_dict_free or alb_loop_time alerts and internal_dsl is null or fetch policy fail log lines on ACP + +## Issue + +An ALB (`global-alb2` or a business ALB) raises a resource or loop-time alert, or its log repeats a line that looks like an error, while traffic is served: + +- The alert `alb_lua_shared_dict_free < 10%` for 30 seconds fires, and the ALB log repeats `prometheus.lua:982: log_error(): Error getting 'nginx_http_upstream_request_duration_seconds_bucket...`. +- The alert `alb_loop_time > 1000ms` for 30 seconds fires from the policy `cpaas-alb-rules`. +- The ALB log repeats `worker.lua:57: init_http_rule_dsl(): internal_dsl is null`. +- The ALB log repeats `fetch policy fail`. + +Related articles: "ALB error-rate alert fires or the ALB error-rate panel exceeds 100 percent on ACP"; "ALB nginx settings are reverted after a restart or upgrade; change them through the overwrite ConfigMap on ACP". + +## Root Cause + +### Cause 1: alb_lua_shared_dict_free alert + +The alert reports that free space in an ALB Lua shared dictionary is low. The dictionaries are declared in the ALB ConfigMap, for example `lua_shared_dict http_raw` and `lua_shared_dict prometheus_metrics`. A free-space curve that stays flat for days points at a dictionary sized close to its steady-state use rather than at a leak. + +On ALB2 v4.1.2 the alert on `global-alb2` is a known defect. Its signature: in the 7-day Lua shared dictionary panel, `balancer_ewma`, `http_certs_cache`, `http_alb_cache`, `http_backend_cache`, `http_raw` and `http_ipc_shared_dict` stay at about 96 to 99 percent free while `prometheus_metrics` is near or below 10 percent, and the log repeats: + +```text +[error] ... [lua] prometheus.lua:982: log_error(): Error getting 'nginx_http_upstream_request_duration_seconds_bucket{port="443",rule="global-alb2-00443-",upstream_ip="... +``` + +with an `upstream_ip` label that holds a very long, repeated list of the same healthy backend addresses. Abnormal monitoring metrics keep occupying the dictionary, some ALB metrics go missing or become discontinuous, and errors appear in the log. Business forwarding is not affected. The defect is fixed in the latest 4.2 release. + +### Cause 2: alb_loop_time > 1000ms for 30 seconds + +`alb_loop_time` does not measure business request latency. It is the total time of one ALB processing loop: fetching and processing Ingress, Service, Endpoint and related resources, generating the nginx configuration, reloading nginx, and updating rule status. The alert fires when that loop takes longer than the threshold for 30 seconds; slow API server responses or slow writes of the configuration file can stretch it. The alert does not affect business traffic by itself. + +When the alert fires intermittently and the ALB logs for the window were not retained, the cause cannot be determined afterwards: low node CPU, memory and load in that window do not prove a disk or API server cause. + +### Cause 3: worker.lua:57: init_http_rule_dsl(): internal_dsl is null + +A new or idle replica logs this line next to certificate-cache and policy-cache lines when the frontend has an empty policy or no backend Service. It is noise and does not indicate a failure. + +### Cause 4: fetch policy fail in the ALB log + +Inside the ALB, a Go process periodically writes the `policy` file and Lua periodically reads it; Lua can start reading while the file is still being written, and logs `fetch policy fail`. Traffic is served normally. This was seen with ALB image `alb2:v3.16.6` and is optimized in later versions. + +## Resolution + +### Cause 1: alb_lua_shared_dict_free alert + +On ALB2 v4.1.2, upgrade to a release with the fix. Until then, a rolling restart of the ALB releases the metrics memory. + +> **Warning:** A rolling restart drops in-flight connections on each replica in turn. Only when the image is v4.1.2 and the signature above matches, with more than one replica Ready. Do not use it to clear an `error-rate` or `alb_loop_time` alert. There is nothing to roll back. + +```bash +kubectl -n cpaas-system rollout restart deploy/ +``` + +On other versions, check the `alb2` and `nginx` container logs for errors first. If they are clean and the metric has been flat for days, raise both dictionaries to `50m` in the overwrite ConfigMap (`data.http`) and watch whether the alert clears. Edit the existing `lua_shared_dict` lines in place; nginx refuses a configuration that declares the same dictionary twice. + +```nginx +lua_shared_dict http_raw 50m; +lua_shared_dict prometheus_metrics 50m; +``` + +> **Warning:** Applying the overwrite ConfigMap reloads nginx on every replica of the ALB, and a syntax error stops the reload for all its listeners. Each dictionary is allocated per Pod, so memory use grows. Back up the overwrite ConfigMap and check headroom under the `nginx` container limit; restoring the previous values is the rollback. The procedure is in the related overwrite ConfigMap article. + +### Cause 2: alb_loop_time > 1000ms for 30 seconds + +Do not delete or restart the ALB Pod to clear the alert. If it fired once, recovered within about 30 seconds, users were not blocked, and the Pods did not restart, record it and leave the policy. For intermittent firing, make sure the next occurrence leaves evidence: save the `alb2` log of every replica for the window before Pods roll, keep the platform log collection for ALB components long enough, and compare slow loops with what changed at that moment (bulk Ingress or rule edits, certificate updates, API server latency, disk latency on the node). + +```bash +for p in $(kubectl -n cpaas-system get pods -l service_name=alb2- -o name); do + kubectl -n cpaas-system logs "$p" -c alb2 --since=1h > "/tmp/$(basename "$p")-alb2.log" +done +kubectl -n cpaas-system logs --all-containers --since=3h | grep -E 'reload:|reload timeout|generate conf|reload load balancer' +``` + +### Cause 3: worker.lua:57: init_http_rule_dsl(): internal_dsl is null + +No change. Attach a Service to the frontend when it should carry traffic. If requests are failing on a replica that serves traffic, look elsewhere; this line is not the failure. + +### Cause 4: fetch policy fail in the ALB log + +If the services behind the ALB answer normally, ignore the line. Upgrading to a release with the optimized policy publishing removes it. Do not edit the policy file inside the container. + +## Diagnostic Steps + +1. Read the image tag, the declared dictionaries and the metrics error count: + + ```bash + kubectl -n cpaas-system get pods -o jsonpath='{.spec.containers[*].image}{"\n"}' + kubectl -n cpaas-system get configmap -o yaml | grep -n lua_shared_dict + kubectl -n cpaas-system logs -c nginx --since=1h | grep -c 'prometheus.lua:982' + ``` + + A v4.1.2 image, `prometheus_metrics` as the only low dictionary, and `prometheus.lua:982` errors: Cause 1 defect. To identify the repeated `upstream_ip` addresses: `kubectl get po -A -o wide | grep -E "|"`. + +2. For `alb_loop_time`, open the policy's monitoring graph for the window and save the `alb2` logs as in Cause 2 before any restart. + +3. Check the noise lines: + + ```bash + kubectl -n cpaas-system logs -c nginx --since=10m | grep -F internal_dsl + kubectl -n cpaas-system logs -c nginx --since=1h | grep -c 'fetch policy fail' + ``` + + With traffic served normally: Cause 3 or Cause 4. diff --git a/docs/en/solutions/ALB_cuts_long_HTTP_or_TCP_connections_at_about_120_seconds_proxy_timeouts_on_ACP.md b/docs/en/solutions/ALB_cuts_long_HTTP_or_TCP_connections_at_about_120_seconds_proxy_timeouts_on_ACP.md new file mode 100644 index 000000000..48d491dd6 --- /dev/null +++ b/docs/en/solutions/ALB_cuts_long_HTTP_or_TCP_connections_at_about_120_seconds_proxy_timeouts_on_ACP.md @@ -0,0 +1,183 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.10.x + - 3.12.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# ALB cuts long HTTP or TCP connections at about 120 seconds: proxy timeouts on ACP + +Related articles: [Finding which hop cuts a request through ALB on ACP: truncated large responses, 60-second cuts, 504 and SSE bursts](Finding_which_hop_cuts_a_request_through_ALB_on_ACP_truncated_large_responses_60_second_cuts_504_and_SSE_bursts.md); [ALB nginx settings are reverted after a restart or upgrade; change them through the overwrite ConfigMap on ACP](ALB_nginx_settings_are_reverted_after_a_restart_or_upgrade_change_them_through_the_overwrite_ConfigMap_on_ACP.md). + +## Issue + +A call through an Alauda Load Balancer (ALB) is cut off at about two minutes, while the same call inside the cluster succeeds. Typical reports: + +- A long HTTP API call, a long-running task or a large-model call (streaming or not) through the ALB entry times out after about 2 minutes; the same call to the Service, the Pod or a NodePort returns after more than 2 minutes. A task that passes in a test environment times out in production. +- A frontend or rule timeout field (`spec.config.timeout`) is saved and then ignored. +- A database or other TCP service behind an ALB port drops connections after about two minutes, with `curl: (56) Recv failure: Connection reset by peer` or `curl: (8) Weird server reply` against the ALB port. The ALB `nginx` log shows `connection timed out (110: Operation timed out) while proxying connection, client: ..., server: 0.0.0.0:, upstream: ":"`, with a session time of about `120.001`. +- A TCP load test of a model service gets about half as many successful requests through the ALB as against the Pod. +- Someone asks whether `nginx.ingress.kubernetes.io/proxy-connect-timeout`, `proxy-send-timeout` and `proxy-read-timeout` apply to ALB and to an individual Ingress, or has set them and sees no change. + +The usual request is to raise the timeouts to 600 seconds (10 minutes), or longer for slow generation. + +## Root Cause + +`proxy_read_timeout` in nginx is an idle timer: it restarts whenever the upstream sends a byte, and it is not a cap measured from the TCP handshake. `proxy_send_timeout` is the idle timer in the other direction, and `proxy_connect_timeout` bounds only the TCP handshake to the upstream. ALB long connections behave the same: the timeout is counted from the last communication, not from connection creation. A handler that stays silent until the final body therefore looks exactly like a timer that started at connect time; a handler that streams tokens does not, as long as each gap stays inside the idle limit. + +### Cause 1: the ALB HTTP idle timers are still at the default 120 seconds + +The ConfigMap with the same name as the ALB in `cpaas-system` shows the defaults: + +```nginx +proxy_connect_timeout 5s; +proxy_send_timeout 120s; +proxy_read_timeout 120s; +proxy_buffering off; +keepalive_timeout 65; +proxy_next_upstream_tries 5; +``` + +Long-running calls, including large-model generation, that are silent for longer than 120 seconds are cut. An external load balancer in front of the ALB can also be the hop that closes first; its timer must be higher than the ALB timer or the client still dies at the outer limit. + +### Cause 2: per-frontend or per-rule timeout fields are not honored on the running version + +The Frontend and Rule fields `spec.config.timeout.proxy_connect_timeout_ms`, `proxy_read_timeout_ms` and `proxy_send_timeout_ms` are not supported by ALB on ACP 4.0.2: they can be saved, but the running proxy does not change. They are supported from the 4.0.4 patch release, and ALB on 4.3.2 supports them on a single forwarding rule. The ALB image moves with the platform and cannot be upgraded on its own. On a version without the fields, only the ALB-wide timeout can be changed; each ALB still has its own configuration, so two ALBs can have different values. + +### Cause 3: TCP or UDP listeners are cut by the stream proxy_timeout + +The ALB keeps separate timeouts for HTTP and for TCP/UDP forwarding, both 2 minutes by default. In the nginx stream configuration the idle timer is `proxy_timeout` in the `stream-tcp` and `stream-udp` sections. Raising the HTTP `proxy_*_timeout` values does not change it. A TCP port (for example a StarRocks query port forwarded to the FE Pod port 9030) drops connections after 120 seconds even when users estimate "about one minute"; the ALB log value decides. A long HTTP API published through an ALB TCP port is cut at 2 minutes even when the `http` block timeout is above 3 minutes, because `stream-tcp` still has `proxy_timeout 120s`. A TCP load test whose P99 latency (for example about 252 seconds) exceeds `proxy_timeout 120s` loses about half its requests. + +### Cause 4: the nginx.ingress.kubernetes.io proxy timeout annotations on an Ingress + +The annotations `nginx.ingress.kubernetes.io/proxy-connect-timeout`, `proxy-send-timeout` and `proxy-read-timeout` can be applied to ALB and to an individual Ingress; they correspond to the `proxy_*_timeout_ms` fields. Adding or removing them affects only that Ingress. + +On ACP 3.12 and 3.18, however, the ALB Ingress did not support these annotations: with them already set, the rendered ALB configuration did not change. On those versions, use the ALB-wide overwrite (Cause 1). + +## Resolution + +### Cause 1: the ALB HTTP idle timers are still at the default 120 seconds + +Decide the idle budget the application needs (600 seconds is the usual request), apply it on the ALB, then raise every device in front of the ALB so that the outer timer is strictly larger. + +- **ACP 4.0.4 and later**: set milliseconds on the Frontend (`frontends.crd.alauda.io`, `crd.alauda.io/v1`, usually named `-`) or on the Rule. 600000 ms is 10 minutes. The change is applied through Lua; nginx does not reload. + + ```yaml + apiVersion: crd.alauda.io/v1 + kind: Frontend + metadata: + name: - + namespace: cpaas-system + spec: + config: + timeout: + proxy_connect_timeout_ms: 600000 + proxy_read_timeout_ms: 600000 + proxy_send_timeout_ms: 600000 + ``` + + ```bash + kubectl -n cpaas-system patch frontend - --type merge -p '{"spec":{"config":{"timeout":{"proxy_read_timeout_ms":600000,"proxy_send_timeout_ms":600000}}}}' + ``` + +- **ACP 3.10 to 3.18, or when one value for the whole ALB is acceptable**: set the directives in the `http_server` section of the ALB overwrite ConfigMap (setup in the related article on the overwrite ConfigMap), keep the indentation of the copied block, and trigger the `alb2` update. `proxy_connect_timeout` can stay at `5s`. + + ```yaml + data: + http_server: |2 + # keep the existing lines of the copied block, and change or add: + proxy_send_timeout 600s; + proxy_read_timeout 600s; + ``` + + ```bash + kubectl -n cpaas-system annotate alb2 force-reload="$(date +%s)" --overwrite + kubectl -n cpaas-system exec -c nginx -- sh -c 'grep -R -n -E "proxy_(connect|send|read)_timeout" /alb/tweak | head' + ``` + + > **Warning:** These directives apply to every frontend on this ALB, and idle connections are held for the new duration, consuming connections, file descriptors and memory. Export the overwrite ConfigMap before editing. To roll back, restore the previous `proxy_*_timeout` values and trigger another `force-reload`. + +There is no separate connection-lifetime setting; lengthen the idle timers if the goal is "count from the last byte". + +### Cause 2: per-frontend or per-rule timeout fields are not honored on the running version + +1. Read the running ALB image and the platform version. On a version without the fields, stop retrying them. +2. Either upgrade the platform to a version that supports them (4.0.4 or later) and then use the Frontend or Rule fields, or set one ALB-wide value in the overwrite ConfigMap (Cause 1) until then. + + > **Warning:** A platform upgrade restarts the ALB and other platform components. Do it in a maintenance window, export the `alb2` and the overwrite ConfigMap first, and keep the previous version available as the rollback. Do not pull a newer `alb-nginx` image into the cluster by hand. + +### Cause 3: TCP or UDP listeners are cut by the stream proxy_timeout + +1. Find the stream sections and current values: + + ```bash + kubectl -n cpaas-system get configmap -o jsonpath='{.data}' | python3 -c 'import json,sys; print(list(json.load(sys.stdin)))' + kubectl -n cpaas-system get configmap -o yaml | grep -n -E 'proxy_timeout|proxy_connect_timeout' + ``` + +2. Raise `proxy_timeout` in `stream-tcp` (and `stream-udp` if used) in the overwrite ConfigMap, for example: + + ```nginx + proxy_next_upstream_tries 5; + proxy_connect_timeout 5s; + proxy_timeout 360s; + tcp_nodelay on; + ``` + + Values that fixed the examples above: 240s for the StarRocks connections, 360s for the API cut at 2 minutes, and 5 minutes for the large-model TCP load test. On versions that honor the Frontend timeout fields (Cause 2), a timeout set directly on a port's Frontend is used as an L4 timeout for that port only. + + > **Warning:** The change reloads nginx on every replica of this ALB, and longer idle timers keep idle TCP connections open longer. Keep a backup of the overwrite ConfigMap; if the ALB Pods are restarted to apply it, do it in a maintenance window. To roll back, restore the previous values and trigger the `alb2` update again. + +3. Trigger the `alb2` update and confirm the value in `/alb/tweak/stream-tcp.conf` and `/alb/tweak/stream-udp.conf`. Check that the value you wrote matches the request. + +### Cause 4: the nginx.ingress.kubernetes.io proxy timeout annotations on an Ingress + +Put the annotations on the Ingress whose rules need a different timer, in the ingress-nginx format (seconds), and confirm with a timed request: + +```yaml +metadata: + annotations: + nginx.ingress.kubernetes.io/proxy-connect-timeout: "600" + nginx.ingress.kubernetes.io/proxy-send-timeout: "600" + nginx.ingress.kubernetes.io/proxy-read-timeout: "600" +``` + +Only when the annotations are already set, the cluster is 3.12 or 3.18, and the timed request or the rendered configuration shows no change, move the timeout to the ALB-wide overwrite ConfigMap (Cause 1), with its warning. + +## Diagnostic Steps + +1. Show the image and the configured timers before changing them: + + ```bash + kubectl -n cpaas-system get pods -l service_name=alb2- -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.containers[*].image}{"\n"}{end}' + kubectl -n cpaas-system get alb2 -o jsonpath='{.spec.config.overwrite.configmap}{"\n"}' + kubectl get frontends.crd.alauda.io -n cpaas-system -o yaml | grep -n -A6 timeout + ``` + +2. Time the same path from outside, from the Service and from the Pod. A 120-second cliff only on the ALB URL is Cause 1 (HTTP) or Cause 3 (TCP port). The same cliff on the Pod is not the ALB: + + ```bash + curl -k -o /dev/null -sS -w 'code=%{http_code} start=%{time_starttransfer} total=%{time_total}\n' --max-time 700 "https://app.example.com/" + kubectl -n exec -- wget -S -O /dev/null "http://..svc:/" + ``` + +3. For a TCP listener, compare the ALB port with the NodePort of the same Service, and read the ALB log for stream timeouts: + + ```bash + nc -vz -w 3 192.0.2.10 + kubectl -n cpaas-system logs -c nginx --since=30m | grep -iE 'timed out|upstream' | tail + ``` + +4. For the Ingress annotations, read what the Ingress carries, then time a request through the ALB as in step 2: + + ```bash + kubectl -n get ingress -o jsonpath='{.metadata.annotations}{"\n"}' + ``` diff --git a/docs/en/solutions/ALB_error-rate_alert_fires_or_the_ALB_error-rate_panel_exceeds_100_percent_on_ACP.md b/docs/en/solutions/ALB_error-rate_alert_fires_or_the_ALB_error-rate_panel_exceeds_100_percent_on_ACP.md new file mode 100644 index 000000000..50ae10db8 --- /dev/null +++ b/docs/en/solutions/ALB_error-rate_alert_fires_or_the_ALB_error-rate_panel_exceeds_100_percent_on_ACP.md @@ -0,0 +1,94 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# ALB error-rate alert fires or the ALB error-rate panel exceeds 100 percent on ACP + +## Issue + +`global-alb2` or a business ALB raises `error-rate` alerts while the platform or the service works, or the ALB monitoring dashboard shows an error rate above 100 percent: + +- The alert `error-rate > 3%` or `error-rate > 5%` for 30 seconds (warning level) fires from the load balancer routing rule alert policy `cpaas-alb-router-rules` for routes such as `ingress/cpaas-system/dex/...` and `ingress/cpaas-system/courier-api/...`. +- Many platform Ingress objects in `cpaas-system` (`dex`, `frontend`, `apollo`, `courier-api` and others) cross the threshold at the same time. +- The ALB monitoring panel shows an error rate above 100 percent, and the access log is dominated by status `499` and `404`. + +`global-alb2` in `cpaas-system` is the platform ingress data plane. Its Pods run the containers `nginx` (access log) and `alb2` (configuration loop log). + +Related articles: "ALB alb_lua_shared_dict_free or alb_loop_time alerts and internal_dsl is null or fetch policy fail log lines on ACP". + +## Root Cause + +### Cause 1: error-rate crosses the threshold on many platform routes with no matching errors in the access log + +`cpaas-alb-router-rules` evaluates the error ratio per route over a sliding window (5 minutes) and fires when it stays above the threshold for 30 seconds; the 30 seconds is the hold time, not the window. Depending on the version, the expression counts 5xx only or 4xx and 5xx, so read the expression before interpreting a value. + +When many `cpaas-system` Ingresses cross the line together (for example `dex` and `courier-api` at `11.11 %` while `archon`, `base-api`, `erebus`, `frontend` and `razor` are at `0.00 %`), the `global-alb2` Pods are `2/2 Running` with no restarts, and the `nginx` access log for those rules shows `"status": "200"` and `"upstream_status": "200"` with no error wave in the window, there is no ALB defect behind the alert. On ACP 4.1.5 this signature was closed as needing no action, with the metric to be improved in a later release. + +A route with very little traffic reaches a high ratio from a single failed request: one request in the window that returns HTTP 500 is 100 percent. When one route dominates, the errors come from that route's backend; an access-log line whose `upstream_status` equals `status` means the backend produced the code and the ALB passed it through. + +### Cause 2: error-rate panel above 100 percent with many 499 and 404 responses + +`499` is the nginx code for a client that closed the connection before the response was sent. When the access log for the window is dominated by `499` (with `404`), clients disconnected before the server answered; no ALB or backend fault was found. A panel value above 100 percent from that mix is not the alert expression computing a real backend failure rate. + +## Resolution + +### Cause 1: error-rate crosses the threshold on many platform routes with no matching errors in the access log + +1. Read the alert expression to see which status codes it counts: + + ```bash + kubectl get prometheusrules.monitoring.coreos.com -A -o yaml | grep -n 'nginx_http_status' + ``` + +2. Check the ALB Pods and the access log for the alert window: + + ```bash + kubectl -n cpaas-system get pods -l service_name=alb2-global-alb2 -o wide + kubectl -n cpaas-system logs -c nginx --since=30m | grep dex | tail + kubectl -n cpaas-system logs -c nginx --since=30m | grep -E ' (401|403|404|499|500|502|503|504) ' | tail + ``` + +3. If many platform routes fired together, the Pods are healthy, and the log has no matching error wave, record the alert as not caused by a route or ALB fault. Do not raise the threshold and do not restart `global-alb2` for this signature. + +4. If one route dominates, find its backend and read the backend log for the same minute. Fix sustained backend errors on the backend; raising the threshold does not fix them. + + ```bash + kubectl get rule -A | grep + kubectl get rule -n -o yaml | grep -A10 service + kubectl logs -n deploy/ --tail=200 + ``` + +### Cause 2: error-rate panel above 100 percent with many 499 and 404 responses + +Count the status codes in the window. When `499` dominates, look at the clients: timeouts shorter than the backend response time, load-test tools that abort requests, or callers that cancel on navigation. Fix the client timeout or the slow backend; the ALB needs no change, and neither a threshold change nor an ALB restart helps. Treat a `404` share separately: it means requests matched no rule or the backend returned `404`. + +```bash +kubectl -n cpaas-system logs -c nginx --since=30m | grep -oE '" [1-5][0-9]{2} ' | sort | uniq -c | sort -rn | head +``` + +## Diagnostic Steps + +1. Identify the policy and the routes. `cpaas-alb-router-rules` is the per-route `error-rate` policy; an `alb_loop_time` alert from `cpaas-alb-rules` is covered in the related article. + +2. Compare the alert with the `nginx` access log of every ALB replica for the same window: + + ```bash + kubectl -n cpaas-system logs deploy/global-alb2 -c nginx --since=30m | grep -E ' (401|403|404|499|500|502|503|504) ' | tail + ``` + + The `-c nginx` flag is needed because the Pod has two containers. + +3. Map the result: + + - Many `cpaas-system` Ingresses fire together, Pods healthy, no matching error codes in the access log: Cause 1, no action. + - One route dominates: Cause 1, follow that backend. + - A status count led by `499` (with `404`) and a panel above 100 percent: Cause 2. diff --git a/docs/en/solutions/ALB_error-rate_alert_on_a_route_or_Ingress_from_cpaas-alb-router-rules_on_ACP.md b/docs/en/solutions/ALB_error-rate_alert_on_a_route_or_Ingress_from_cpaas-alb-router-rules_on_ACP.md new file mode 100644 index 000000000..f339f00cf --- /dev/null +++ b/docs/en/solutions/ALB_error-rate_alert_on_a_route_or_Ingress_from_cpaas-alb-router-rules_on_ACP.md @@ -0,0 +1,131 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.1.x + - 4.3.x +--- + +# ALB error-rate alert on a route or Ingress from cpaas-alb-router-rules on ACP + +## Issue + +The real-time alert page shows an `error-rate` alert from policy `cpaas-alb-router-rules` (also seen as `cpaas-lb-router-rules`, or called `alb-error-rate`). The text is `error-rate > 3%` sustained for 30 seconds (`error-rate > 3% 且持续 30 秒`), `error-rate > 5%` for 30 seconds, or `error-rate` at 100 percent. The alert names an ALB such as `global-alb2` in `cpaas-system`, and an Ingress or route such as `ingress/cpaas-system/courier-api/global-alb2`. It often clears after a few minutes. + +The ALB Pods run two containers: `nginx` (access log) and `alb2` (configuration-loop log). The `alb_loop_time` alert from `cpaas-alb-rules` and other platform alerts that arrive together with these are covered in the related article. + +Related article: [ALB alb_loop_time alert from cpaas-alb-rules and platform alerts raised with it on ACP](ALB_alb_loop_time_alert_from_cpaas-alb-rules_and_platform_alerts_raised_with_it_on_ACP.md). + +## Root Cause + +### Cause 1: the alert is the error ratio of one route, and the threshold and counted statuses differ by policy + +`cpaas-alb-router-rules` watches the error ratio of each ALB route or Ingress over a 5-minute window. The "30 seconds" is how long the threshold must hold before firing, not the ratio window. The threshold and the statuses counted differ between installations: + +- the threshold can be `0.03` (label `alert_indicator_threshold: "0.03"`) or 5% (the 4.1.5 `global-alb2` policy); +- the 4.1.5 expression counts all 4xx and 5xx (`status=~"[4-5].*"`); other descriptions count 5xx only. + +If 4xx is counted, a wave of `499` or `404` in the access log raises the ratio. 5xx responses are either passed through from the backend, or generated by ALB (`502`, `504`) when the backend is unreachable. + +A route whose 5-minute window contains a single request that the backend answered with HTTP 500 computes an error rate of 100 percent and fires once. For example, `ingress/cpaas-system/razor-api/global-alb2` showed one 100% point that did not recur, and the `razor-api` log showed sporadic `500` responses to `GET /platform/events.alauda.io/v1/...` requests. That sporadic window needs no change. + +Many `cpaas-system` Ingresses firing at once (`dex`, `frontend`, `apollo`, `courier-api`, peaks about 7% to 17%) with no matching 4xx/5xx wave in the ALB `nginx` access log is not a backend fault: no action, no threshold change and no `global-alb2` restart. + +### Cause 2: global-alb2 error-rate > 5% from occasional courier-api 500 responses on 4.1.5 + +On ACP 4.1.5 the alert is `global-alb2` `error-rate > 5%` for 30 seconds (`error-rate > 5%且持续30s`). Per rule, only `ingress/cpaas-system/courier-api/global-alb2` is elevated (maximum 14.29%), while `razor`, `frontend`, `erebus`, `dex`, `base-api` and `archon` stay at 0%. The expression is: + +```promql +sum(rate(nginx_http_status{namespace=~"cpaas-system",pod=~"global-alb2-.*",status=~"[4-5].*",vmcluster="global"}[5m])) / (sum(irate(nginx_http_status{namespace=~"cpaas-system",pod=~"global-alb2-.*",vmcluster="global"}[5m])) + 0.01) +``` + +The 5xx come from one `GET` rule, `global-alb2-00443-`, which matches `STARTS_WITH /platform/monitoring.alauda.io/v1beta1` and routes to Service `courier-api` port 8080. The `courier-api` log shows `500` for `GET /platform/monitoring.alauda.io/v1beta1/projects//clusters//namespaces//variables?regex=...&expr=query_result(kube_namespace_labels{})...`, and the same request in a browser returns `"message": "kube_namespace_labels should has selector namespace="`, `"code": 500`. A namespace-level monitoring dashboard asks for the global metric `kube_namespace_labels{}` without a namespace selector, and the permission check of that API rejects it; a query with `kube_namespace_labels{namespace=""}` returns `200 OK`. These are normal requests, but this policy counts all 4xx and 5xx, so the occasional 500s push the ratio over 5%. + +### Cause 3: error-rate spikes while the logging component is small or being scaled + +`error-rate` (from `cpaas-alb-router-rules`, `source_type: "ingress"`) and `alb_loop_time` (from `cpaas-alb-rules`, for example a trigger value of `5534.00` that recovered within 30 seconds) both move, and the error-rate breakdown concentrates on the logging ingress `ingress/cpaas-system/cpaas-clickhouse/global-alb2` (maximum 61.54% over three days). At the same time `cpaas-statefulset-rules` reports CPU above 90% on the ClickHouse Pods `chi-cpaas-clickhouse-replicated--`. The logging component's resource specification is too small; after it was scaled up, the related error rate stopped rising. The remaining fluctuation is within the supported range. + +## Resolution + +### Cause 1: the alert is the error ratio of one route, and the threshold and counted statuses differ by policy + +1. Read the threshold from the alert text or the `alert_indicator_threshold` label, and the counted statuses from the `status=~` matcher in the rule expression: + + ```bash + kubectl get prometheusrules.monitoring.coreos.com -A -o yaml | grep -n 'nginx_http_status' + ``` + +2. Find the backend from the rule or Ingress name in the alert, and read the ALB access log and the backend log for the same minute: + + ```bash + kubectl get rule -A | grep + kubectl get rule -n -o yaml | grep -A10 service + kubectl logs -n cpaas-system -c nginx --tail=500 + kubectl logs -n deploy/ --tail=200 + ``` + +3. Decide: + - one request and one HTTP 500 in the window: sporadic, no change; + - a sustained 500 in the backend log: the alert reports that backend; fix the backend, and do not raise the threshold; + - ALB lines with `502` or `504`: ALB cannot reach the backend; check the backend Pods and Service endpoints; + - many platform Ingresses firing with no status wave in the `nginx` log: no action. + +### Cause 2: global-alb2 error-rate > 5% from occasional courier-api 500 responses on 4.1.5 + +1. Break the 5xx down by rule in vmui, then read the rule and the backend log: + + ```promql + sort_desc(sum by(rule,method)(nginx_http_status{status=~"[5].*",svc_name="courier-api"})) + ``` + + ```bash + kubectl get rule -A | grep courier-api + kubectl get rule -n cpaas-system -o yaml + kubectl logs -n cpaas-system | grep 500 + ``` + + A rule on `/platform/monitoring.alauda.io/v1beta1`, and `500` lines for `.../variables?...expr=query_result(kube_namespace_labels{})` with the `should has selector namespace` body, is this signature. + +2. Raise the threshold on that alert policy in the console. + + > **Warning:** A higher error-rate threshold also hides later 4xx and 5xx spikes up to the new value, including a backend that is really failing. Precondition: the window is occasional `courier-api` 500 responses on normal requests, as above, and the current threshold is written down. Rollback: set the previous threshold on the same policy. + +Do not raise it when the `nginx` access log has no matching errors (no-action case in Cause 1), or when a backend keeps returning 500. A namespace-level chart that cannot see global series is this permission check, not a missing scrape: query with a namespace matcher, or use the global view. + +### Cause 3: error-rate spikes while the logging component is small or being scaled + +1. Compare the size and load of the logging Pods with the error-rate window: + + ```bash + kubectl get pods -n cpaas-system -o wide | grep -i clickhouse + kubectl top pod -n cpaas-system + ``` + +2. Scale the logging component up through the workload or plugin that owns the `chi-cpaas-clickhouse-replicated-*` Pods. + + > **Warning:** Resizing ClickHouse restarts its replicas; log queries and writes through the affected replica pause until it is Ready. Precondition: its CPU is at the alert threshold during the error-rate window. Rollback: set the previous resources on the same object. + +3. After the scale-up the related error rate stops rising. If the remaining movement stays inside the supported range, the `error-rate` and `alb_loop_time` thresholds can be raised under the same warning as Cause 2. Do not raise them while a backend still returns 500. + +## Diagnostic Steps + +1. Read the policy name: `cpaas-alb-router-rules` or `cpaas-lb-router-rules` with `error-rate` is this article; `cpaas-alb-rules` with `alb_loop_time` is the related article. + +2. Read the threshold and `status=~` matcher from the live rule (Cause 1) instead of assuming 3%, 5% or 5xx only. + +3. Break down the error ratio by Ingress: + + ```promql + sort_desc(sum by (rule) (rate(nginx_http_status{status=~"[4-5].*"}[5m]))) + ``` + + Expected: one rule dominates. A single 100% sample with one 500 in its backend log is Cause 1's sporadic window; `courier-api` with `kube_namespace_labels` 500s is Cause 2; `cpaas-clickhouse` together with ClickHouse CPU alerts is Cause 3. + +4. Check that the ALB Pods did not restart and that users can reach the service: + + ```bash + kubectl get pods -n cpaas-system -o wide | grep -E 'alb2|alb' + ``` diff --git a/docs/en/solutions/ALB_load_balancer_Pods_restart_during_an_ACP_upgrade_and_the_restart_cannot_be_deferred.md b/docs/en/solutions/ALB_load_balancer_Pods_restart_during_an_ACP_upgrade_and_the_restart_cannot_be_deferred.md new file mode 100644 index 000000000..24d86e825 --- /dev/null +++ b/docs/en/solutions/ALB_load_balancer_Pods_restart_during_an_ACP_upgrade_and_the_restart_cannot_be_deferred.md @@ -0,0 +1,81 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# ALB load balancer Pods restart during an ACP upgrade and the restart cannot be deferred + +## Issue + +During an ACP upgrade the LB (ALB) Pods restart automatically, and business traffic through the load balancer is interrupted while they restart. Application owners ask: + +- Is the LB restart during the upgrade necessary? +- Can the upgrade skip the restart, so that the LB Pods are restarted by hand later, in a business downtime window? + +The question usually comes up after a test upgrade or an earlier upgrade window, once the impact on business continuity has been noticed. + +## Root Cause + +### Cause 1: The LB restart is part of the ACP upgrade and cannot be avoided + +- The LB Pod restart, and the network interruption it causes during the ACP upgrade, cannot be avoided. +- The restart cannot be skipped during the upgrade and replaced by a manual restart later. +- The upgrade has to be performed inside a maintenance window coordinated with the business owners. + +This restart comes from upgrading the cluster that runs the LB. A different case is a business-cluster ALB that rolls while only `global` is upgraded, because its Deployment no longer matches the HelmRequest on `global`. That drift-induced restart can be removed before the next global upgrade (see "Local overrides reverted or blocking an ACP upgrade: ResourcePatch, ALB ConfigMap, chart and feature gate changes", Cause 8), but removing it does not remove the restart described here. + +## Resolution + +### Cause 1: The LB restart is part of the ACP upgrade and cannot be avoided + +Treat the LB interruption as an expected part of the upgrade and plan for it: + +1. Agree on a maintenance window with the business owners before the upgrade. Tell them that traffic through the LB is interrupted while its Pods restart, for every domain and port served by those ALBs. +2. Inventory the ALB instances on the cluster being upgraded and the Pods behind each one, so the owners of every entry point can be notified. +3. Back up each `alb2` object and any overwrite ConfigMap before the upgrade, so custom nginx settings stay recoverable. +4. Agree how the interruption will be measured (for example with the probe loop in Diagnostic Steps), so the window can be closed on evidence. +5. Run the upgrade inside that window. There is no supported option to skip the LB restart and do it by hand afterwards. +6. After the upgrade, confirm that every ALB Pod is Running and Ready on the new image, and that custom settings are still rendered, before reopening business traffic. + +Read-only commands for the inventory and the post-upgrade check: + +```bash +kubectl get alb2.v2beta1.crd.alauda.io -A +kubectl -n get alb2.v2beta1.crd.alauda.io -o yaml > -alb2.yaml +kubectl -n get deploy,pods -o wide | grep +kubectl -n get pods -l service_name=alb2- \ + -o custom-columns=NAME:.metadata.name,START:.status.startTime,IMAGES:.spec.containers[*].image +``` + +If an ALB port conflict exists before the window, clear it first; otherwise the restart that the upgrade performs fails. + +## Diagnostic Steps + +1. Identify which cluster is being upgraded and on which cluster the restarted LB runs. +2. If the LB runs on the cluster being upgraded, this article applies: the restart is expected and cannot be deferred. +3. If the LB runs on a business cluster that was not being upgraded (only `global` moved), compare its Deployment with the HelmRequest on `global`; that is the drift case, not this one: + + ```bash + kubectl get helmrequests.app.alauda.io -A + ``` + +4. If the LB did not come back after the upgrade restarted it, check for an ALB port conflict. + +During the upgrade, watch the ALB Pods restart and note when each one is Ready again: + +```bash +kubectl -n get pods -l service_name=alb2- -w +``` + +Measure the interruption from a client outside the cluster with a probe against one business URL: + +```bash +while true; do + printf '%s ' "$(date +%T)" + curl -sk -o /dev/null -m 3 -w '%{http_code}\n' https://app.example.com/ || echo fail + sleep 1 +done +``` diff --git a/docs/en/solutions/ALB_nginx_settings_are_reverted_after_a_restart_or_upgrade_change_them_through_the_overwrite_ConfigMap_on_ACP.md b/docs/en/solutions/ALB_nginx_settings_are_reverted_after_a_restart_or_upgrade_change_them_through_the_overwrite_ConfigMap_on_ACP.md new file mode 100644 index 000000000..fe527dd92 --- /dev/null +++ b/docs/en/solutions/ALB_nginx_settings_are_reverted_after_a_restart_or_upgrade_change_them_through_the_overwrite_ConfigMap_on_ACP.md @@ -0,0 +1,208 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.10.x + - 3.14.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# ALB nginx settings are reverted after a restart or upgrade; change them through the overwrite ConfigMap on ACP + +Related articles: [ALB overwrite ConfigMap edits do not apply, nginx fails with init_by_lua, or access logs lack trace_id on ACP](ALB_overwrite_ConfigMap_edits_do_not_apply_nginx_fails_with_init_by_lua_or_access_logs_lack_trace_id_on_ACP.md). + +## Issue + +Timeouts, access-log format, request-header buffers and similar nginx settings on an Alauda Load Balancer (ALB) do not survive a restart or a platform upgrade. The data plane is the `ALB2` object (`alaudaloadbalancer2.crd.alauda.io`, short name `alb2`), usually in `cpaas-system`. Typical reports: + +- The ALB ConfigMap is restored every time it is edited. Where should ALB settings be configured? +- `proxy_read_timeout` was changed and goes back to `120s` after a restart. +- File downloads end with `gateway timeout` after a little over a minute, and a download timeout changed by hand is reset shortly afterwards. +- On ALB 3.14, edits to the Pod spec, container files or probes are reverted. +- A previously working nginx customization stopped working after an upgrade. +- After an upgrade to 4.3.3, the load balancer address and the domain stop answering, while `telnet` to the container port works. + +## Root Cause + +### Cause 1: the rendered ALB ConfigMap is not the source of truth + +The ConfigMap with the same name as the ALB in `cpaas-system` (next to `-port-info`) is rendered and managed by the ALB. After an edit, a restart or an upgrade the controller writes it again, so a direct change to `proxy_read_timeout` or any other key is lost. The durable place for nginx settings is an overwrite ConfigMap referenced from `spec.config.overwrite.configmap` on the `ALB2`. Setting up the reference is a one-time step; later changes go into the overwrite ConfigMap, followed by an update of the `ALB2` (see the related article). + +ALB configuration compatibility is not carried over automatically across a platform upgrade (for example from 3.8 to 3.12): customized parameters have to be checked and adapted by hand after the upgrade. + +### Cause 2: proxy_read_timeout was changed in the grpc_server section and returns to 120s + +Two mistakes together: the edit was made on the rendered ConfigMap (Cause 1), and it was made in the `grpc_server` section, which does not control these connection timeouts. Layer-7 timeouts are `proxy_read_timeout` and `proxy_send_timeout` in the `http` / `http_server` sections; the Layer-4 idle timeout is `proxy_timeout` in `stream-tcp`. + +### Cause 3: on ALB 3.14, Pod YAML or files edited by hand are reverted by the operator + +On platform 3.14 the ALB is owned by an operator (`alb-operator`). Changes to the Pod template, container files or probes are restored, and that release does not expose those fields on the CRD. Later 3.18 builds publish more ALB metrics; they do not make Pod edits durable on 3.14. + +### Cause 4: custom nginx settings stop working after an upgrade although the overwrite is still referenced + +The ALB overwrite from the previous version was not updated during the upgrade, so the upgraded ALB kept rendering from the old copy, which no longer matches the new template. The overwrite procedure itself is still valid; rebuilding the copy from the new version's template fixes it. + +### Cause 5: after an upgrade to 4.3.3, Pod annotations added on the ALB Deployment are removed + +When an ALB is set up for the MetalLB underlay scheme (MetalLB with ALB providing independent load balancing for underlay business traffic), the Pod annotations that scheme needs may have been added directly to the ALB Deployment. The upgrade to 4.3.3 regenerates the Deployment and drops those additions. Access through the load balancer address and the domain then fails, while the ALB itself is running and `telnet` to the container port works. + +## Resolution + +### Cause 1: the rendered ALB ConfigMap is not the source of truth + +1. Back up the live ConfigMap, the overwrite ConfigMap if it already exists, and the `alb2` object, before an upgrade and before every edit: + + ```bash + kubectl -n cpaas-system get configmap -o yaml > /tmp/-cm.backup.yaml + kubectl -n cpaas-system get configmap -cm-update -o yaml > /tmp/-cm-update.backup.yaml + kubectl -n cpaas-system get alb2 -o yaml > /tmp/-alb2.backup.yaml + ``` + +2. If `spec.config.overwrite` already exists, do not create a second copy: edit the ConfigMap it references and go to step 5. Otherwise create the overwrite ConfigMap once, from a full copy of the live `data` keys. Do not `kubectl apply` the backup itself; that object is owned by the ALB. + + ```bash + kubectl -n cpaas-system get configmap -o json | python3 -c ' + import json, sys + cm = json.load(sys.stdin) + out = { + "apiVersion": "v1", + "kind": "ConfigMap", + "metadata": { + "name": cm["metadata"]["name"] + "-cm-update", + "namespace": cm["metadata"]["namespace"], + }, + "data": cm.get("data") or {}, + } + json.dump(out, sys.stdout, indent=2) + print() + ' > /tmp/-cm-update.json + kubectl apply -f /tmp/-cm-update.json + ``` + + The same copy can be made in the console: **Cluster Management > Resource Management**, search ConfigMap for the ALB name, copy the content of `` into **Create Resource Object**, rename it `-cm-update`, and keep only `name` and `namespace` under `metadata` (remove `creationTimestamp`, `labels` such as `alb.cpaas.io/managed-by: alb-operator`, `managedFields`, `ownerReferences` and `resourceVersion`). + +3. Point the `ALB2` at that object. On 4.1, `spec.config.overwrite.configmap` of the `alb2` (`crd.alauda.io/v2beta1`) is a list of `name` entries, each a `namespace/name` string: + + > **Warning:** Pointing the ALB at a different ConfigMap for the first time can make the ALB remount it and restart, which interrupts its listeners; nginx takes the new configuration at once, and a faulty configuration breaks it at once. Check the content before saving the reference, do it in a maintenance window with more than one ALB replica, and keep the `alb2` backup. To roll back, remove the `overwrite` entry (apply the saved `alb2` YAML). + + ```bash + kubectl -n cpaas-system patch alb2 --type merge -p \ + '{"spec":{"config":{"overwrite":{"configmap":[{"name":"cpaas-system/-cm-update"}]}}}}' + ``` + +4. From now on, edit only the overwrite ConfigMap. Typical keys inside `data.http` are the nginx `http` context: `log_format`, `large_client_header_buffers`, `client_max_body_size` and the `proxy_*_timeout` directives. Leave every other line in place. For long calls (large-model inference, downloads), put `proxy_connect_timeout`, `proxy_send_timeout` and `proxy_read_timeout` here; this applies to the whole ALB, not to one frontend. Per-frontend timeouts are fields of the `Frontend` resource instead. On 4.2 the same kinds of setting are also available from the Ingress and ALB resources. + +5. After each edit, update the `alb2` so the controller regenerates its ConfigMap. The `force-reload` annotation is a hot reload and does not delete the Pod: + + > **Warning:** A bad `data.http` makes every replica of this ALB fail its reload or crash, which drops traffic for every Ingress and rule on it. Change one setting at a time. To roll back, re-apply the overwrite backup and set the annotation again. + + ```bash + kubectl -n cpaas-system annotate alb2 force-reload="$(date +%s)" --overwrite + ``` + +6. Confirm the rendered files in the ALB Pod under `/alb/tweak` (`http.conf`, `stream-tcp.conf`, `stream-udp.conf`) instead of assuming that the ConfigMap and the running file match: + + ```bash + kubectl -n cpaas-system exec -c nginx -- sh -c 'grep -R -n -E "proxy_(connect|send|read)_timeout|proxy_timeout" /alb/tweak | head' + ``` + + Alternatively, update the overwrite ConfigMap and restart the ALB Pods one at a time in a maintenance window, then read the files in the new Pod. + + > **Warning:** Restarting the ALB Pods drops in-flight connections on each replica and, for a single-replica ALB, interrupts all its listeners until the Pod is Ready. Wait until each replacement is Ready before the next. + + ```bash + kubectl -n cpaas-system get pods -o wide | grep + kubectl -n cpaas-system delete pod + kubectl -n cpaas-system exec -- cat /alb/tweak/stream-tcp.conf /alb/tweak/stream-udp.conf + ``` + +### Cause 2: proxy_read_timeout was changed in the grpc_server section and returns to 120s + +Set up the overwrite ConfigMap as in Cause 1 (or edit the one already referenced), and set `proxy_read_timeout` / `proxy_send_timeout` in the `http_server` section, and `proxy_timeout` in `stream-tcp` for TCP ports. Then trigger the `alb2` update and check `/alb/tweak` in the nginx container. + +```bash +kubectl -n cpaas-system get alb2 -o yaml | grep -A8 overwrite +kubectl -n cpaas-system get cm -cm-update -o yaml | grep -n -E '^ [a-z_-]+:|proxy_(read|send)_timeout|proxy_timeout' +``` + +### Cause 3: on ALB 3.14, Pod YAML or files edited by hand are reverted by the operator + +Stop editing the Deployment, the Pod and files inside the container. Put supported settings on the `ALB2` (or on rule and Ingress objects), and nginx settings in the overwrite ConfigMap. Probe and sidecar changes that are not CRD fields cannot be kept on 3.14. Do not scale `alb-operator` to zero to make a Pod edit stick; that is too risky for production. + +```bash +kubectl -n cpaas-system get deploy alb-operator +kubectl -n cpaas-system get alb2 -o yaml +``` + +### Cause 4: custom nginx settings stop working after an upgrade although the overwrite is still referenced + +1. Before an upgrade, export the overwrite ConfigMap and the `alb2` (Cause 1, step 1). +2. After the upgrade, rebuild the overwrite ConfigMap from the new version's rendered ConfigMap and re-apply only the custom lines, so the copy matches the new template. Then trigger the `alb2` update. + + > **Warning:** Replacing the overwrite content reloads nginx on every replica of this ALB, and a mistake stops the reload for all its listeners. Keep backups of the old overwrite ConfigMap and the new rendered ConfigMap, and write down the list of custom lines. To roll back, re-apply the old overwrite backup and change the `alb2` annotation again. + +3. Some settings are fields of the `alb2` resource rather than nginx text. For example, forwarded headers are enabled with a `config` block under `spec`, leaving the other spec fields (`enablePortProject`, `networkMode`, `nodeSelector`) unchanged, in a maintenance window: + + ```yaml + spec: + config: + proxy_header: + use_forward_headers: "true" + ``` + + Verify inside the ALB container: + + ```bash + kubectl -n cpaas-system exec -- sh -c 'cat /etc/alb2/nginx/policy.new | grep forwarded' + ``` + +### Cause 5: after an upgrade to 4.3.3, Pod annotations added on the ALB Deployment are removed + +Put the Pod annotations on the `alb2` resource instead of the Deployment, with the `alb.cpaas.io/pod-annotations` annotation, so they persist at the `alb2` level and are rendered into the Pods. Replace the example key and value with the annotations the scheme needs: + +> **Warning:** The ALB Pods roll to pick up the new annotations, and wrong network annotations can leave them without the intended address. Copy the annotations from the pre-upgrade Deployment YAML (or the scheme documentation) and save the `alb2` YAML. To roll back, run `kubectl annotate alb2 -n cpaas-system alb.cpaas.io/pod-annotations-`. + +```bash +kubectl annotate alb2 -n cpaas-system \ + 'alb.cpaas.io/pod-annotations={"example.com/key":"value"}' \ + --overwrite +``` + +Also check the external load balancer's forwarding to the ALB. + +## Diagnostic Steps + +1. Identify which ALB owns the traffic and whether an overwrite ConfigMap is already referenced. If the jsonpath prints nothing, edits to the rendered ConfigMap will be lost (Cause 1): + + ```bash + kubectl get alb2 -A + kubectl -n cpaas-system get alb2 -o jsonpath='{.spec.config.overwrite.configmap}{"\n"}' + kubectl -n cpaas-system get configmap -cm-update + ``` + +2. For a timeout that returns to `120s`, find which section carries the edited line. A hit only under `grpc_server` is Cause 2: + + ```bash + kubectl -n cpaas-system get cm -o yaml | grep -n -E '^ [a-z_-]+:|proxy_(read|send)_timeout|proxy_timeout' + ``` + +3. On 3.14, an existing `alb-operator` Deployment is the reconciler that undoes Pod edits (Cause 3): + + ```bash + kubectl get deploy -n cpaas-system alb-operator + ``` + +4. After an upgrade, diff the rendered `data.http` against the backup (Cause 4), and compare the Pod annotations with the Deployment YAML saved before the upgrade (Cause 5): + + ```bash + kubectl -n cpaas-system get configmap -o jsonpath='{.data.http}' > /tmp/live-http.conf + diff -u /tmp/expected-http.conf /tmp/live-http.conf + kubectl -n cpaas-system get pod -o jsonpath='{.metadata.annotations}{"\n"}' + kubectl -n cpaas-system get alb2 -o jsonpath='{.metadata.annotations}{"\n"}' + ``` diff --git a/docs/en/solutions/ALB_overwrite_ConfigMap_edits_do_not_apply_nginx_fails_with_init_by_lua_or_access_logs_lack_trace_id_on_ACP.md b/docs/en/solutions/ALB_overwrite_ConfigMap_edits_do_not_apply_nginx_fails_with_init_by_lua_or_access_logs_lack_trace_id_on_ACP.md new file mode 100644 index 000000000..045c37708 --- /dev/null +++ b/docs/en/solutions/ALB_overwrite_ConfigMap_edits_do_not_apply_nginx_fails_with_init_by_lua_or_access_logs_lack_trace_id_on_ACP.md @@ -0,0 +1,117 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.10.x + - 3.14.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# ALB overwrite ConfigMap edits do not apply, nginx fails with init_by_lua, or access logs lack trace_id on ACP + +Related articles: [ALB nginx settings are reverted after a restart or upgrade; change them through the overwrite ConfigMap on ACP](ALB_nginx_settings_are_reverted_after_a_restart_or_upgrade_change_them_through_the_overwrite_ConfigMap_on_ACP.md). + +## Issue + +An Alauda Load Balancer (ALB) already uses an overwrite ConfigMap (referenced from `spec.config.overwrite.configmap` of the `alb2`, setup described in the related article), and: + +- a timeout or another key was changed in the overwrite ConfigMap, the ALB Pod was restarted, and the running nginx still has the old value; +- before a production change, someone asks whether editing the overwrite ConfigMap restarts the ALB and how much a timeout change affects the business; +- after the overwrite ConfigMap was created, the ALB Pod crashes in Lua before it listens, with `init_by_lua` errors and the `balancer` module failing to load; +- the access logs do not contain a trace ID, so a failed call cannot be tied to the caller's `X-Trace-Id`. + +## Root Cause + +### Cause 1: edits to the overwrite ConfigMap are not synced until the alb2 resource changes + +When an overwrite ConfigMap is set, edits to it are not synced on their own: the ConfigMap with the same name as the ALB is regenerated only when the `alb2` resource itself changes. A Pod restart is not such a change. + +A second problem can hide behind the first. After the `alb2` was updated, a value can still be missing because the overwrite ConfigMap itself is wrong, for example `proxy_connect_timeout s60;` instead of `proxy_connect_timeout 60s;` in the `stream-tcp` section (the unit must follow the number), or lines in the last section of the YAML that need more indentation. On 3.14 the new values can appear only after a delay, although a mounted ConfigMap normally reloads quickly. + +Editing only the content of the ConfigMap that is already referenced has small impact: normally none, and with many slow connections the ALB uses somewhat more memory. Adding an annotation to the `alb2` to trigger the update applies the new value without restarting the ALB Pod. + +### Cause 2: init_by_lua error, cannot load the balancer module + +The overwrite ConfigMap was cloned from the live object, but part of `data.http` was dropped (for example while adding `escape=json` to the log format), so Lua initialization no longer sees the balancer setup, `init_by_lua` fails and the `balancer` module does not load. + +### Cause 3: the access logs do not contain trace_id + +The default `log_format http` has no `trace_id` variable. This is a missing overwrite entry, not a dropped request. + +## Resolution + +### Cause 1: edits to the overwrite ConfigMap are not synced until the alb2 resource changes + +1. After every edit of the overwrite ConfigMap, change the `alb2` resource so that the controller regenerates the ALB ConfigMap. Any annotation change works, for example: + + ```bash + kubectl -n cpaas-system annotate alb2 force-reload="$(date +%s)" --overwrite + ``` + + In the console the same step is **Resource Management**, resource type `alb2`, version `v2beta1`: edit the ALB and add an annotation. + +2. Compare the rendered ConfigMap with the overwrite copy, then check the running files inside the ALB Pod: + + ```bash + kubectl -n cpaas-system get configmap -cm-update -o yaml | grep -n -E 'proxy_(connect_)?timeout|proxy_(send|read)_timeout' + kubectl -n cpaas-system get configmap -o yaml | grep -n -E 'proxy_(connect|send|read)_timeout' + kubectl -n cpaas-system exec -c nginx -- sh -c 'grep -R -n -E "proxy_(connect|send|read)_timeout|proxy_timeout" /alb/tweak 2>/dev/null | head' + ``` + +3. If the new value is still absent, the overwrite ConfigMap itself is wrong. Look for a malformed value such as `s60` instead of `60s`, a misplaced entry, or indentation errors in the last block, fix it, and trigger the `alb2` update again. A correct `stream-tcp` section looks like this: + + ```nginx + proxy_connect_timeout 60s; + proxy_next_upstream_tries 5; + proxy_timeout 600s; + tcp_nodelay on; + ``` + + > **Warning:** A broken overwrite ConfigMap makes every replica of this ALB fail its reload, which drops traffic for every Ingress and rule on it. Back up the overwrite ConfigMap before each edit. To roll back, re-apply the backup and change the `alb2` annotation again. + +### Cause 2: init_by_lua error, cannot load the balancer module + +Rebuild the overwrite ConfigMap from a fresh full copy of the live object's `data` (procedure in the related article), then change only `log_format`. Replacing `log_format http` with `log_format http escape=json` is enough for JSON access logs. Keep the `access_log` line pointed at that format, for example `access_log /dev/stdout http buffer=16k flush=1s;`. Then trigger the `alb2` update. + +> **Warning:** Reloading after a truncated `http` block keeps the ALB down until the full `data.http` is restored. Keep a backup of the overwrite ConfigMap. To roll back, `kubectl apply` the backup and set the `force-reload` annotation again. + +### Cause 3: the access logs do not contain trace_id + +Add the map and the log variables to the overwrite ConfigMap `data.http`, preserving the rest of the content, then trigger the `alb2` update. When the client sends `X-Trace-Id`, that value is logged; otherwise nginx's `$request_id` is used. + +```nginx +map $http_x_trace_id $trace_id { + default $http_x_trace_id; + "" $request_id; +} +``` + +Inside `log_format http`, add `trace_id="$trace_id"` and `req_id="$request_id"`. To forward the inbound header to the upstream, add `proxy_set_header X-Trace-Id $http_x_trace_id;` in the `http_server` section of the same ConfigMap. Do not create a second log format name; the data plane reads the format named `http`. + +## Diagnostic Steps + +1. Confirm that an overwrite ConfigMap is referenced. If not, start with the related article: + + ```bash + kubectl -n cpaas-system get alb2 -o jsonpath='{.spec.config.overwrite.configmap}{"\n"}' + ``` + +2. If the overwrite is set and a restart did not apply an edit, compare the overwrite ConfigMap, the rendered ConfigMap and `/alb/tweak` in the Pod (Cause 1). A value missing from the rendered ConfigMap means the `alb2` was not updated; a malformed value in the overwrite copy is the second part of Cause 1. + +3. For a Lua crash, read the previous container log before editing anything else. `init_by_lua` and `balancer` errors are Cause 2: + + ```bash + kubectl -n cpaas-system logs -c nginx --previous | grep -E 'init_by_lua|balancer' + ``` + +4. After a successful reload, confirm the running files rather than the ConfigMap alone: + + ```bash + kubectl -n cpaas-system exec -c nginx -- sh -c 'find / -path "*tweak*http.conf" 2>/dev/null | head' + ``` diff --git a/docs/en/solutions/ALB_placement_cordoned_nodes_tolerate-all_node_selectors_cpaas-system_ALB_ports_and_access_addresses_on_ACP.md b/docs/en/solutions/ALB_placement_cordoned_nodes_tolerate-all_node_selectors_cpaas-system_ALB_ports_and_access_addresses_on_ACP.md new file mode 100644 index 000000000..6d593906b --- /dev/null +++ b/docs/en/solutions/ALB_placement_cordoned_nodes_tolerate-all_node_selectors_cpaas-system_ALB_ports_and_access_addresses_on_ACP.md @@ -0,0 +1,162 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.1" + - "4.2" +--- + +# ALB placement: cordoned nodes, tolerate-all, node selectors, cpaas-system ALB ports and access addresses on ACP + +## Issue + +Questions and failures about where an Alauda Load Balancer (ALB) runs and how it is reached: + +- The masters were cordoned with `kubectl cordon`, and the ALB Pods (or other `cpaas-system` Pods) can no longer be scheduled there, so they fail to restart during an upgrade or after a VM restart. Tolerations added to the ALB Deployment disappear right after saving. +- How to move an ALB to other nodes, and how to find the port mappings of an ALB before its VIP is replaced. +- Business ports are wanted on the `cpaas-system` ALB on the masters, or a new ALB should take port 443 on the global masters. +- The access IP of a container-network ALB was changed, or a second one added, and the address does not answer. +- Whether several business ALBs in one cluster conflict when they all use 80, 443 or the same ports, and whether more ALBs can be added. +- Edits to the `alb2` resource of the `cpaas-system` ALB (for example `networkMode`) are reverted, and the monitoring dashboard of the business cluster does not display correctly. + +Every ALB Deployment is generated from its `alb2` resource (`crd.alauda.io/v2beta1`), so changes go on the `alb2` object. On a business cluster, the ALB named `cpaas-system` in namespace `cpaas-system` is the platform's own ALB. + +## Root Cause + +### Cause 1: ALB Pods cannot be scheduled onto cordoned nodes, and Deployment edits are reverted + +`kubectl cordon` marks a node `unschedulable: true`, represented as the taint `node.kubernetes.io/unschedulable:NoSchedule`. The ALB Pods do not tolerate it, and tolerations typed into the generated Deployment are reverted by the controller that owns it. The annotation `tolerate-all: "true"` on the `alb2` resource makes the ALB Pods tolerate all taints, including the cordon taint; the change shows on the Pods, not in the Deployment YAML. An annotation named `tolerate` has no effect. + +### Cause 2: the ALB has to run on different nodes + +The ALB node selector decides which nodes run ALB Pods. Moving the ALB also moves the address that the VIP or external load balancer must forward to; for the `cpaas-system` ALB, the platform VIP forwarding must change too. A single-instance ALB reached through a node's own IP needs its **Access Address** changed; a high-availability ALB behind a VIP keeps the VIP and changes the VIP's backend members. Updating the load balancer does not affect short connections but interrupts long connections in progress. + +### Cause 3: the cpaas-system ALB is reserved for platform traffic + +The `cpaas-system` ALB is a platform-exclusive resource that forwards only platform traffic and cannot be delegated to business users. A platform upgrade syncs its configuration and overwrites custom changes, so ports added to it are lost and can conflict with platform traffic. On the global cluster, port 443 of the three masters is used for platform access and must not be changed, so a new ALB must not be scheduled onto those nodes. + +### Cause 4: a container-network ALB does not take a changed or added access IP + +For a container-network ALB, the access address comes from its `LoadBalancer` Service (`-lb-` in `cpaas-system`). The existing address cannot be edited, and added addresses do not take effect: `telnet ` connects while `telnet ` hangs at `Trying ...`, and the Service `EXTERNAL-IP` lists only the original address. Changing or adding access addresses is supported only for host-network ALBs, which accept several access addresses (domain names or IPs), each a separate entry; a domain entered there must resolve to the load balancer's VIP. + +### Cause 5: several business ALBs using the same ports do not conflict + +Host-network ALB instances of one cluster cannot be deployed on the same node (the console does not allow it), so each instance serves its ports on different nodes. Container-network ALBs do not occupy node ports. More ALB instances can be added on demand as long as the cluster has the resources. + +### Cause 6: the cpaas-system alb2 is rendered by a chart and edits are reconciled back + +The `alb2` of the `cpaas-system` ALB is rendered by the chart release `alauda-alb2` (chart `acp/chart-alauda-alb2`), so a direct change such as `networkMode` is reconciled back after saving. The values must be set through the chart's `valuesOverride`. A business cluster whose `cpaas-system` ALB was not in host network mode also had a monitoring dashboard that did not display correctly; it displayed normally after the same `valuesOverride` was applied. The link between the two was not confirmed. + +## Resolution + +### Cause 1: ALB Pods cannot be scheduled onto cordoned nodes, and Deployment edits are reverted + +Add the annotation to the `alb2` resource, then check the Pods. + +> **Warning:** With `tolerate-all`, taints and the unschedulable mark no longer keep ALB Pods off any node; only the node selector limits placement. The change rolls the ALB Pods. Only when the node selector already restricts the ALB to the intended nodes and more than one replica is Ready. Roll back with `kubectl -n cpaas-system annotate alb2 tolerate-all-`. + +```bash +kubectl -n cpaas-system get alb2 +kubectl -n cpaas-system annotate alb2 tolerate-all="true" --overwrite +kubectl -n cpaas-system get pods -l service_name=alb2- -o jsonpath='{range .items[*]}{.metadata.name} {.spec.tolerations}{"\n"}{end}' +``` + +A toleration with `operator: Exists` and no key confirms it. The same annotation lets the `cpaas-system` ALB return to cordoned masters by itself after a VM restart or an upgrade. Where the annotation is not wanted, `uncordon` the masters during the upgrade, wait for the Pods to start, and `cordon` them again. + +### Cause 2: the ALB has to run on different nodes + +Change the node selector under `spec.config` of the `alb2` (console: **Platform Management > Network Management > Load Balancers**, open the ALB, **Update**), and update the VIP or external load balancer pool in the same change: + +```yaml +spec: + config: + nodeSelector: + ingress: "true" +``` + +> **Warning:** When the Pods move, the old nodes stop answering on the ALB ports and long connections are interrupted. The new nodes must carry the label and already be members of the VIP or load balancer pool (or clients must be ready for the new **Access Address**). Do it in a maintenance window. Restoring the previous selector is the rollback. + +```bash +kubectl label node = +kubectl -n cpaas-system get alb2 -o yaml | grep -n -A3 nodeSelector +kubectl get ft -A +``` + +The ports configured on an ALB are Frontend (`ft`) resources; `kubectl get ft -A` lists the port mappings before a move or a VIP replacement. + +### Cause 3: the cpaas-system ALB is reserved for platform traffic + +Create a separate ALB for the business component, on the same master nodes and behind the same VIP if needed, with high-numbered ports the platform ALB does not use. Leave the `cpaas-system` ALB unchanged. On the global cluster, do not schedule new ALBs onto the three masters. + +### Cause 4: a container-network ALB does not take a changed or added access IP + +To keep a given VIP, use an ALB in host network mode: create it with host networking and enter the address (and any additional ones, each as a separate entry) in the access address field (**Network Configuration**, network mode host network, **Add** under the access address list). A container-network ALB cannot be switched in place; plan a new ALB with the same listeners and rules. + +### Cause 5: several business ALBs using the same ports do not conflict + +No change is needed. Give each host-network ALB its own nodes through its node selector, and point each VIP or DNS name at the nodes of its own ALB. Check capacity before adding another ALB: + +```bash +kubectl get alb2 -A +kubectl describe node | grep -A8 'Allocated resources' +``` + +### Cause 6: the cpaas-system alb2 is rendered by a chart and edits are reconciled back + +On the global cluster, edit the ALB `minfo` of the target business cluster and set the values under `spec.valuesOverride`, keeping the `version` the `minfo` already shows. The same `valuesOverride` can be set on the ALB's `ars` (AppRelease `alauda-alb2`). + +```yaml +spec: + valuesOverride: + acp/chart-alauda-alb2: + networkMode: host + nodeSelector: + cpaas-system-alb: "" + version: +``` + +> **Warning:** Changing `networkMode` or `nodeSelector` redeploys the `cpaas-system` ALB, interrupting platform traffic through it while its Pods move; host network mode needs its ports free on the selected nodes. The nodes must carry the `cpaas-system-alb` label. Save the `minfo` YAML; restoring the saved `valuesOverride` is the rollback. + +```bash +kubectl get minfo | grep -i alb +kubectl get minfo -o yaml > /tmp/.yaml +kubectl edit minfo +kubectl get node -l cpaas-system-alb +``` + +## Diagnostic Steps + +1. Why ALB Pods are Pending: + + ```bash + kubectl -n cpaas-system describe pod | grep -A5 Events + kubectl get nodes -o custom-columns=NAME:.metadata.name,UNSCHEDULABLE:.spec.unschedulable,TAINTS:.spec.taints + ``` + + `untolerated taint {node.kubernetes.io/unschedulable: }`: Cause 1. + +2. Network mode and access address of an ALB; for a container-network ALB, the `EXTERNAL-IP` of its Service is the address that works (Cause 4): + + ```bash + kubectl -n cpaas-system get pods -l service_name=alb2- -o jsonpath='{range .items[*]}{.metadata.name} hostNetwork={.spec.hostNetwork}{"\n"}{end}' + kubectl -n cpaas-system get svc | grep -lb + ``` + +3. Ports and placement of every ALB (Causes 2 and 5): + + ```bash + kubectl get ft -A + kubectl get pods -A -o wide | grep -E 'alb' + ``` + +4. A reverted change on the `cpaas-system` ALB (Cause 6): + + ```bash + kubectl get minfo -o jsonpath='{.spec.valuesOverride}{"\n"}' + kubectl -n cpaas-system get alb2 cpaas-system -o yaml | grep -n -E 'networkMode|nodeSelector' + ``` + +For external load balancer pool members that must match ALB placement, see "External load balancer in front of ALB: stalls, pool members, client IP, health checks and 503 on ACP". diff --git a/docs/en/solutions/ALB_pod_fails_to_start_after_rollback_with_serviceaccount_token_no_such_file_or_directory_on_ACP.md b/docs/en/solutions/ALB_pod_fails_to_start_after_rollback_with_serviceaccount_token_no_such_file_or_directory_on_ACP.md new file mode 100644 index 000000000..1070ee8a1 --- /dev/null +++ b/docs/en/solutions/ALB_pod_fails_to_start_after_rollback_with_serviceaccount_token_no_such_file_or_directory_on_ACP.md @@ -0,0 +1,73 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.16.x +--- + +# ALB pod fails to start after rollback with serviceaccount token no such file or directory on ACP + +## Issue + +After an ALB upgrade caused application errors, the ALB was rolled back urgently: the ALB operator version was set back to the version shipped with ACP 3.16, and the Deployment of the ALB instance was deleted so that the operator would recreate it. The recreated ALB pod does not start. Its log shows: + +```text +driver: use InClusterConfig +"lifecycle: get rest cfg fail" err="open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" +"run fail" err="open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" +``` + +The startup configuration printed by the pod shows `"K8s": {"Mode": "incluster", ...}`. The pod exits and restarts in a loop, and the traffic served by that ALB is not restored. + +## Root Cause + +The rolled-back ALB runs in in-cluster mode: it builds its Kubernetes client from the ServiceAccount token at the standard path `/var/run/secrets/kubernetes.io/serviceaccount/token`. + +That file only exists when Kubernetes mounts the ServiceAccount token into the pod. The ServiceAccount used by the ALB in `cpaas-system` has `automountServiceAccountToken: false`, and the pod created after the rollback does not mount the token itself, so the file is missing. Without the token the ALB cannot create its REST client and exits. + +## Resolution + +1. Find the ServiceAccount used by the ALB Deployment: + + ```bash + kubectl -n cpaas-system get deploy -o jsonpath='{.spec.template.spec.serviceAccountName}{"\n"}' + ``` + +2. Enable automatic token mounting on that ServiceAccount: + + ```bash + kubectl -n cpaas-system patch sa -p '{"automountServiceAccountToken": true}' + ``` + + To revert this change later, apply the same patch with `false`. + +3. Restart the ALB pods so that they are recreated with the token mounted. + + > **Warning:** Deleting an ALB pod interrupts the traffic it serves until the new pod is ready. Do this only after the `serviceaccount/token: no such file or directory` error is confirmed in the pod log. If the ALB has several replicas, delete one pod at a time and wait for it to become `Running` before the next. Pods are recreated by the Deployment, so no rollback is needed for this step. + + ```bash + kubectl -n cpaas-system get pods -o wide | grep + kubectl -n cpaas-system delete pod + ``` + +4. Confirm the new pod is `Running`, its log no longer shows the token error, and the applications behind the ALB respond again. + +## Diagnostic Steps + +1. Read the log of the failing ALB pod. The `get rest cfg fail` and `serviceaccount/token: no such file or directory` lines identify this issue: + + ```bash + kubectl -n cpaas-system logs --previous | grep -E 'InClusterConfig|rest cfg|serviceaccount/token' + ``` + +2. Check whether the token is mounted. Expected for this issue: the ServiceAccount prints `false`, and the pod has no volume mounted at `/var/run/secrets/kubernetes.io/serviceaccount`: + + ```bash + kubectl -n cpaas-system get sa -o jsonpath='{.automountServiceAccountToken}{"\n"}' + kubectl -n cpaas-system get pod \ + -o jsonpath='{.spec.automountServiceAccountToken}{"\n"}{range .spec.containers[*].volumeMounts[*]}{.mountPath}{"\n"}{end}' + ``` + +3. After the fix, the pod spec lists a mount at `/var/run/secrets/kubernetes.io/serviceaccount` and the log no longer reports the token error. diff --git a/docs/en/solutions/A_raw_disk_added_to_a_Ceph_device_class_does_not_become_an_OSD_on_ACP.md b/docs/en/solutions/A_raw_disk_added_to_a_Ceph_device_class_does_not_become_an_OSD_on_ACP.md new file mode 100644 index 000000000..f215775e9 --- /dev/null +++ b/docs/en/solutions/A_raw_disk_added_to_a_Ceph_device_class_does_not_become_an_OSD_on_ACP.md @@ -0,0 +1,214 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# A raw disk added to a Ceph device class does not become an OSD on ACP + +## Issue + +A bare disk is attached to a node for Ceph distributed storage on Alauda Container Platform, but it never becomes an OSD in the device class the pool uses: + +- The raw disks are attached and Ceph storage has to be expanded (routine expansion). +- A disk was added to the device class from the page and the configuration was accepted. `CephCluster` lists the disk under `spec.storage.nodes`, but nothing happens: `ceph osd tree` shows no new OSD and the capacity does not grow. +- The prepare log shows `skipping device "": ["Has GPT headers"]`. +- The disk was already wiped with `dd`, `wipefs`, or `sgdisk`, and Ceph still does not use it. +- Rook-Ceph raises an abnormal alert, `osd-prepare` is never created for the node, and the node has a taint. + +Work in the `rook-ceph` namespace (`CephCluster` `cephclusters.ceph.rook.io`, operator Deployment `rook-ceph-operator`, toolbox Deployment `rook-ceph-tools`). Ceph commands run in the toolbox unless the step says to run on the node. + +Related articles: [Ceph storage pool cannot be created or never finishes because the deployment created no OSD on ACP](./Ceph_storage_pool_cannot_be_created_or_never_finishes_because_the_deployment_created_no_OSD_on_ACP.md), [Ceph OSD crash-loops with not a block device, or an OSD has to move to another device class on ACP](./Ceph_OSD_crash-loops_with_not_a_block_device_or_an_OSD_has_to_move_to_another_device_class_on_ACP.md), [Ceph device class page rejects a disk, or Ceph does not show the new capacity on ACP](./Ceph_device_class_page_rejects_a_disk_or_Ceph_does_not_show_the_new_capacity_on_ACP.md) (the page offers no device, `至少需要 3 个节点`, USB devices, a grown virtual disk). + +## Root Cause + +### Cause 1: skipping device "": ["Has GPT headers"] + +Rook does not create an OSD on a disk that still has a GPT (or other) signature. A replacement disk and a disk that once belonged to another Ceph cluster both look like this. The disk is not bad; it is not empty enough for prepare. + +### Cause 2: the raw disk is on the node but not in the device class + +The disk is visible in `lsblk` with no filesystem mounted, but `ceph osd tree` has no OSD for it. A pool only places data on its CRUSH device class. Until the disk is added to that class from the Device Class page (or an equivalent `CephCluster` `spec.storage` entry), capacity does not grow. This path assumes healthy, unused disks; damaged disks have to be repaired or replaced first. + +On ACP 4.3.1 the page that adds disks to an existing device class sometimes does not list disks that are already members, and submitting the form can drop those disks from `spec.storage`. + +### Cause 3: the whole disk still carries an LVM2_member label, so no osd-prepare pod starts on the node + +`CephCluster` has the new device, other nodes have `osd-prepare` pods in `Completed`, and the node with the new disk has no prepare pod at all. `lsblk -f` shows the whole disk with `FSTYPE` `LVM2_member`, no partitions, and no mount. `pvs`, `lvs`, and `dmsetup ls` can be empty: no active volume group uses the disk, but the LVM label is still on it. `sgdisk --zap-all` and `fdisk` clear only partition tables, not this label. As long as `blkid` reports a filesystem type for the whole disk, the disk is treated as occupied. + +### Cause 4: disk still occupied after dd, wipefs, and sgdisk + +The disk was cleaned with `dd if=/dev/zero of=/dev/ bs=4k count=512`, `wipefs -af /dev/`, and `sgdisk --zap-all /dev/`, `blkid -p` no longer reports a type, and Ceph still does not use it. Those commands clear signatures and the partition table (the `dd` overwrites about the first 2 MiB) but do not release leftover OSD occupancy of the device on the node. Restarting the node released it. + +### Cause 5: osd-prepare not created because the node is tainted + +Operator logs suggest the affected node may have been deleted, the `osd-prepare` object is not created, and `kubectl describe node` shows a taint. That taint blocks `osd-prepare` on the node. + +## Resolution + +### Cause 1: skipping device "": ["Has GPT headers"] + +1. On the node, confirm the device. It must be the skipped disk, not the OS disk and not a device that backs an `up` OSD: + +```bash +lsblk -o NAME,SIZE,TYPE,FSTYPE,LABEL,MOUNTPOINT,MODEL,SERIAL /dev/ +wipefs /dev/ +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s +``` + +2. If `lsblk` shows a leftover `ceph-*` device-mapper node on this disk only, remove it first. Skip this if there is none. + +> **Warning:** `dmsetup remove` detaches the mapping immediately; if it is an active OSD, that OSD crashes and its PGs go degraded. Precondition: `lsblk` shows the mapping only on the skipped disk, and `ceph osd tree` lists no `up` OSD on it. Rollback: none. + +```bash +dmsetup remove /dev/mapper/ +``` + +3. Wipe the disk. + +> **Warning:** This destroys every signature, the partition table, and the first 100 MiB of `/dev/`. Irreversible. Precondition: a second person has checked the device name against `lsblk`, the disk is not mounted or in `findmnt`, and it is not the OS disk or a live OSD disk. Rollback: none. + +```bash +wipefs -af /dev/ +sgdisk --zap-all /dev/ +dd if=/dev/zero of=/dev/ bs=1M count=100 oflag=direct,dsync +``` + +4. Restart the operator so prepare runs again. + +> **Warning:** Restarting `rook-ceph-operator` restarts only the operator pod; OSD pods keep running, but prepare and reconcile pause until the new pod is Ready. Precondition: the wipe finished on the intended disk. Rollback: none needed; the Deployment spec is unchanged. + +```bash +kubectl -n rook-ceph rollout restart deploy/rook-ceph-operator +kubectl -n rook-ceph rollout status deploy/rook-ceph-operator +``` + +5. Add the disk to the device class (Cause 2). `ceph osd tree` then shows a new OSD `up` and `in`. + +### Cause 2: the raw disk is on the node but not in the device class + +This is the normal expansion path; wiping is not part of it. If a disk added here never becomes an OSD, read the prepare log and go to the matching cause. + +1. Check each disk's health first. I/O or medium errors, or a failed health check, are not this cause; plan a repair or replacement with the storage owner. + +```bash +dmesg -T | grep -i -E '|I/O error|medium error' | tail -n 20 +smartctl -H /dev/ +``` + +2. Save the cluster object: + +```bash +kubectl -n rook-ceph get cephcluster -o yaml > cephcluster-before.yaml +``` + +3. In the console, open **Storage Management** > **Distributed Storage** > the cluster > **Device Class**, open the class the pool uses, choose **Add Device**, select the node, and enter the disk name (`sde`, `nvme0n1`, or whatever `lsblk` showed). Two disks on one node are two names on that node, for example `nvme0n1` and `nvme1n1`. On ACP 4.0.3 the page path was Storage Management, Distributed Storage, Access Storage. Create a new class the same way when the pool is to use a separate class. + +4. After the operator prepares the disk, check from the toolbox: + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd tree +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd df +``` + +Adding disks starts backfill, and available-capacity numbers settle only after backfill finishes. + +5. Diff `spec.storage.nodes` against `cephcluster-before.yaml`; every previous device must still be present. If the ACP 4.3.1 page dropped existing disks, put only the missing device entries back with `kubectl -n rook-ceph edit cephcluster `. Do not `kubectl apply` the whole backup over a cluster that has moved on. See also [Ceph OSDs nearfull, backfill blocked, or OSD usage uneven on ACP](./Ceph_OSDs_nearfull_backfill_blocked_or_OSD_usage_uneven_on_ACP.md), Cause 2. + +### Cause 3: the whole disk still carries an LVM2_member label, so no osd-prepare pod starts on the node + +1. Confirm the label and that no prepare pod exists for the node: + +```bash +lsblk -f /dev/ +blkid -p /dev/; echo exit:$? +pvs; lvs; dmsetup ls +kubectl -n rook-ceph get pods -o wide | grep -i prepare +``` + +2. Clear the label and refresh udev. + +> **Warning:** Impact: `wipefs -af` removes every signature on `/dev/` and `dd` zeroes its first 100 MiB. Irreversible. Precondition: only when `lsblk -f` shows `LVM2_member` on this whole disk, `pvs`, `lvs`, and `dmsetup ls` show no volume group or mapping you still use on it, it is not mounted, and it is not the OS disk. Rollback: none. + +```bash +wipefs -af /dev/ +dd if=/dev/zero of=/dev/ bs=1M count=100 oflag=direct conv=fsync +udevadm trigger --action=change /sys/class/block/ +udevadm settle +blkid -p /dev/; echo exit:$? +lsblk -f /dev/ +``` + +`blkid -p` printing nothing with `exit:2` and `lsblk -f` showing no `FSTYPE` confirm the disk is clean. The operator then prepares it; restart `rook-ceph-operator` as in Cause 1 if no prepare pod appears, and check `ceph osd tree`. + +### Cause 4: disk still occupied after dd, wipefs, and sgdisk + +Use this only when the wipe already ran, `blkid -p` reports nothing (Cause 3 is excluded), and Ceph still does not use the disk. + +```bash +lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT /dev/ +findmnt | grep || true +``` + +> **Warning:** Restarting the node stops every OSD and any MON on it; PGs peer again, and a second storage-node failure during the reboot can drop a replica set. Precondition: the disk is still occupied after the wipe, other replicas are `up`, and you are in a maintenance window. Rollback: the node comes back on its own; a reboot does not restore a wiped disk. + +Restart the node with your node maintenance procedure. When it is `Ready` and the device is free in `lsblk`, add it with Cause 2. + +### Cause 5: osd-prepare not created because the node is tainted + +```bash +kubectl describe node | grep -i -A3 Taints +kubectl -n rook-ceph logs deploy/rook-ceph-operator --tail=200 | grep -i prepare +``` + +> **Warning:** Removing the taint allows pods to schedule on ``, including workloads the taint was holding off. Precondition: you saved the full taint key, value, and effect, and this node is supposed to host OSDs. Rollback: `kubectl taint node =:`. + +```bash +kubectl taint node - +``` + +Then restart `rook-ceph-operator` as in Cause 1 (same Warning) and check that prepare is created: + +```bash +kubectl -n rook-ceph rollout restart deploy/rook-ceph-operator +kubectl -n rook-ceph get pods -o wide | grep -i prepare +``` + +## Diagnostic Steps + +1. List the prepare pods with their nodes, and read the skip reason: + +```bash +kubectl -n rook-ceph get pods -o wide | grep -i prepare +kubectl -n rook-ceph logs | grep -E 'skipping device|Has GPT|' +``` + +Make sure the prepare pod you read runs on the node that holds the disk. + +2. On the node, separate an empty disk from one that still has a signature, a partition, or a mount: + +```bash +lsblk -o NAME,SIZE,TYPE,FSTYPE,LABEL,MOUNTPOINT +blkid -p /dev/; echo exit:$? +findmnt +``` + +A disk with partitions or a mounted filesystem (for example a container runtime disk) is in use; do not wipe it, choose a free disk instead. + +3. Check OSDs and classes: + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd tree +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd crush class ls +``` + +- `Has GPT headers` in the prepare log: Cause 1. +- Clean disk not yet in the device class: Cause 2. +- No prepare pod on the node, `FSTYPE` `LVM2_member` on the whole disk: Cause 3. +- Wipe done, `blkid -p` empty, disk still not used: Cause 4. +- No prepare pod and a taint on the node: Cause 5. diff --git a/docs/en/solutions/Adding_a_node_to_a_Kube-OVN_underlay_cluster_in_the_web_UI_fails_with_network_interface_not_found_on_ACP.md b/docs/en/solutions/Adding_a_node_to_a_Kube-OVN_underlay_cluster_in_the_web_UI_fails_with_network_interface_not_found_on_ACP.md new file mode 100644 index 000000000..54a6c2fe2 --- /dev/null +++ b/docs/en/solutions/Adding_a_node_to_a_Kube-OVN_underlay_cluster_in_the_web_UI_fails_with_network_interface_not_found_on_ACP.md @@ -0,0 +1,62 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x +--- + +# Adding a node to a Kube-OVN underlay cluster in the web UI fails with network interface not found on ACP + +## Issue + +In the web UI, adding a node to a business cluster that uses a Kube-OVN underlay network fails with an error that the network interface (`网卡`, NIC) was not found. Only the node IP, the username and the password were entered. + +- The add-node page seems to have no place to enter the NIC, and the NIC of this node cannot be added on the network provider page either. +- The cluster nodes do not all use the same NIC name. The bridge network status lists the nodes being added as `InitOVSBridgeFailed` with `failed to create and configure external bridge br-provider: nic does not exist`, and the node that has `` as `InitOVSBridgeSucceeded`. + +## Root Cause + +### Cause 1: the NIC setting of each node row in the add-node form was left collapsed and empty + +On ACP 4.0, the add-node form of an underlay cluster has a network interface setting for every node, but it is folded into the node row. It opens from the expand arrow (`>`) at the very start of the row, to the left of the node type selector (for example `计算节点`, compute node) and the `IPv4 地址` (IPv4 address) field. When the row is not expanded, no interface is specified for the node, and node addition fails with the interface-not-found error. + +The `nic does not exist` condition on the new nodes is consistent with this: with no per-node entry, the bridge network looks for its default interface, which those machines do not have. + +## Resolution + +### Cause 1: the NIC setting of each node row in the add-node form was left collapsed and empty + +1. On the node to be added, read the name of the NIC that carries the underlay network. A VLAN interface prints as `@`; enter the part before `@`. + + ```bash + ip -br link + ip -br addr + ``` + +2. In the add-node form, click the `>` arrow at the start of the node row to expand it. +3. Fill in the network interface of that node with the NIC name from step 1, then submit. The node is then added. +4. Check that the bridge network now reports the node as ready: + + ```bash + kubectl get provider-networks -o jsonpath='{range .status.conditions[*]}{.node}{" "}{.reason}{" "}{.message}{"\n"}{end}' + ``` + + `InitOVSBridgeSucceeded` for the new node means the bridge was created on the NIC you entered. + +For a node that is already in the cluster and was never given a matching NIC, set the per-node NIC in the ProviderNetwork itself (`customInterfaces`, or `excludeNodes` for a node without an underlay NIC), as described in [Kube-OVN underlay Pods fail with resolve MAC address timeout or bridge network InitOVSBridgeFailed on ACP](Kube-OVN_underlay_Pods_fail_with_resolve_MAC_address_timeout_or_bridge_network_InitOVSBridgeFailed_on_ACP.md). + +## Diagnostic Steps + +Compare what the bridge network expects with what the new machine has: + +```bash +kubectl get provider-networks -o jsonpath='default={.spec.defaultInterface}{"\n"}custom={.spec.customInterfaces}{"\n"}exclude={.spec.excludeNodes}{"\n"}' +kubectl get provider-networks -o jsonpath='{range .status.conditions[*]}{.node}{" "}{.reason}{"\n"}{end}' | grep -v Succeeded +ip -br link +``` + +- A node being added through the web UI whose NIC name differs from the default interface, together with a NIC-not-found error on the add, points to the collapsed NIC field (Cause 1). +- If the NIC that the bridge network names is missing on a node that has been in the cluster for a while, or it vanished after a hardware or IaaS change, the node row is not the issue; fix the ProviderNetwork or restore the NIC on the host (see the article linked above). +- If `kube-ovn-cni` on the node fails with `failed to initialize config` or `failed to parse config` for an old interface name, the tunnel interface is wrong, not the bridge NIC; see [kube-ovn-cni fails on one node after a NIC rename, clock skew or hostname change on ACP](kube-ovn-cni_fails_on_one_node_after_a_NIC_rename_clock_skew_or_hostname_change_on_ACP.md). diff --git a/docs/en/solutions/Adding_a_node_to_an_ACP_cluster_fails_on_UseDNS_or_UsePAM_the_node_does_not_boot_after_init.sh_or_a_cloned_node_fails_at_yum.md b/docs/en/solutions/Adding_a_node_to_an_ACP_cluster_fails_on_UseDNS_or_UsePAM_the_node_does_not_boot_after_init.sh_or_a_cloned_node_fails_at_yum.md new file mode 100644 index 000000000..024fc0a22 --- /dev/null +++ b/docs/en/solutions/Adding_a_node_to_an_ACP_cluster_fails_on_UseDNS_or_UsePAM_the_node_does_not_boot_after_init.sh_or_a_cloned_node_fails_at_yum.md @@ -0,0 +1,97 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Adding a node to an ACP cluster fails on UseDNS or UsePAM, the node does not boot after init.sh, or a cloned node fails at yum + +## Issue + +- The availability check of a new node fails with `节点可用性检查未通过,无法添加节点到集群下` (the node availability check did not pass; the node cannot be added to the cluster) together with `检查sshd配置:/etc/ssh/sshd_config文件中 UseDNS 参数被注释或值不为 no,请调整` (sshd check: the `UseDNS` parameter in `/etc/ssh/sshd_config` is commented out or not `no`; adjust it). +- Adding a node succeeds only sometimes. The node's secure log shows most platform logins as `Accepted password for ... ssh2` and some as `Failed password for ... ssh2` followed by `Connection closed by authenticating user ... [preauth]`. +- A new workload-cluster node runs the initialization script and then does not boot after a reboot. Without the script it reboots normally, and `lsblk` and `/etc/fstab` look fine. +- Adding a node that was cloned from an existing node fails at the task `[core/bootstrap-os : Installing basic packages using yum]` with `"msg": "MODULE FAILURE"` and `Install failed!`, and the traceback ends with: + + ```text + shutil.Error: Destination path '/etc/yum.repos.d/temp/temp_repo_0.repo' already exists + ``` + +Related articles: "Preparing a host before adding it to an ACP cluster: node check items, disks, init script, DefaultTasksMax, root access" covers what to prepare; "Which operating systems, kernels, and CPU architectures can run in an ACP cluster" covers an add that fails with an unsupported operating system. + +## Root Cause + +### Cause 1: UseDNS or UsePAM in sshd_config is commented out or not no + +The platform adds nodes through automated SSH logins. The availability check requires `UseDNS no` and inspects `UsePAM`. With `UseDNS` commented out or not `no`, the check fails and the node is not added; after `UseDNS no` is set, the node is added. + +Intermittent `Failed password` entries for the platform's logins, while most logins of the same user succeed, stop after `UsePAM` is set to `no`, and the checks then pass every time. Because a commented or overridden line in `sshd_config` is not what sshd actually uses, check the effective values with `sshd -T`, not only the file. + +### Cause 2: the cgroup nokmem step of the init script stops the node from booting + +The `config cgroup nokmem` step of the initialization script writes kernel parameters into the grub configuration. On some hosts those parameters stop the node from booting. Commenting out that section of the script before running it lets the node be initialized and rebooted normally. + +### Cause 3: a cloned node carries the yum repository backup of the original node + +The yum task backs up the existing repository files by moving them into `/etc/yum.repos.d/temp/`. A machine cloned from an existing node (even after a cleanup script from another environment was run on it) still holds state from the original node, including `/etc/yum.repos.d/temp/temp_repo_0.repo`. The move then fails because the destination exists, and the install stops. The output can also show `setlocale: LC_ALL: cannot change locale (C.UTF-8)` warnings, which are not the failure. A freshly installed machine is added normally. + +## Resolution + +### Cause 1: UseDNS or UsePAM in sshd_config is commented out or not no + +1. Read the file and the effective values: + + ```bash + sudo grep -n -E '^\s*#?\s*(UsePAM|UseDNS)' /etc/ssh/sshd_config + sudo sshd -T | grep -i -E '^(usepam|usedns) ' + ``` + +2. Set `UseDNS no` and `UsePAM no` explicitly in `/etc/ssh/sshd_config`. + + > **Warning:** A syntax error in `sshd_config` can stop sshd from accepting new logins after the restart. Keep the current session open, back the file up first, and validate it before restarting. Rollback: restore the backup and restart sshd. + + ```bash + sudo cp -a /etc/ssh/sshd_config /root/sshd_config.bak + sudo vi /etc/ssh/sshd_config + sudo sshd -t && sudo systemctl restart sshd + sudo sshd -T | grep -i -E '^(usepam|usedns) ' + ``` + + Expected: `usedns no` and `usepam no`. + +3. Run the availability check and add the node again. + +### Cause 2: the cgroup nokmem step of the init script stops the node from booting + +1. Before running the init script on such a node, find its `config cgroup nokmem` section: + + ```bash + grep -n -i 'nokmem' init.sh + ``` + +2. Comment out that section, then run the script with `bash`. +3. For a node that already fails to boot, compare its kernel command line and grub configuration with a node that boots, and remove the parameters that the step added: + + ```bash + cat /proc/cmdline # on a node that boots, for comparison + ``` + +### Cause 3: a cloned node carries the yum repository backup of the original node + +1. Check the node for the leftover backup: + + ```bash + ls -la /etc/yum.repos.d/ /etc/yum.repos.d/temp/ 2>/dev/null + ``` + +2. Replace the clone with a freshly installed machine and add that machine instead. + +## Diagnostic Steps + +1. Read the failed item of the availability check in the console. + - `UseDNS 参数被注释或值不为 no`: Cause 1. + - Intermittent add failures: read the node's secure log (`/var/log/secure`, or `/var/log/auth.log` on Ubuntu) for `Failed password` from the platform's address next to `Accepted password` for the same user: Cause 1. +2. A node that boots without the init script and does not boot after it: check whether the script contains a `nokmem` step (`grep -n -i nokmem init.sh`): Cause 2. +3. `Installing basic packages using yum` fails with `temp_repo_0.repo' already exists`, and `ls /etc/yum.repos.d/temp/` lists files on a cloned machine: Cause 3. diff --git a/docs/en/solutions/Alauda_AI_fine-tuning_has_no_task_template_or_shows_a_blank_configuration_page_after_a_template_is_selected.md b/docs/en/solutions/Alauda_AI_fine-tuning_has_no_task_template_or_shows_a_blank_configuration_page_after_a_template_is_selected.md new file mode 100644 index 000000000..1aded30c1 --- /dev/null +++ b/docs/en/solutions/Alauda_AI_fine-tuning_has_no_task_template_or_shows_a_blank_configuration_page_after_a_template_is_selected.md @@ -0,0 +1,84 @@ +--- +kind: + - Troubleshooting +products: + - Alauda AI +ProductsVersion: [] +--- + +# Alauda AI fine-tuning has no task template or shows a blank configuration page after a template is selected + +## Issue + +- In **Model Optimization** > **Fine-Tuning**, the task template list is empty, so a fine-tuning task cannot be created. The expectation was that a task template is available right after the initial deployment. +- A template is selected in the **Create Fine-tuning Task** dialog, and the task configuration page that follows is completely blank. +- The question is what goes in the dataset text input column (`数据集文本输入列`), and whether more than one column can be selected. + +Fine-tuning and training are optional features. They need the fine-tuning plugins (Volcano, Kubeflow Training Operator, MLFlow, and the others listed in "Enable Fine-Tuning and Training Features" of the Alauda AI documentation) and `tuneModels: true` under `spec.values.experimentalFeatures` of the `AmlCluster` resource `default`. If the Fine-Tuning menu itself is missing, start with that guide. + +## Root Cause + +### Cause 1: no template has been uploaded, or the template lacks the label the fine-tuning entry filters on + +Task templates are not created by the deployment. The example templates are shipped in the `alaudadockerhub/training-templates` image and have to be extracted and uploaded to **Task Template**, as described in "Enable Fine-Tuning and Training Features". In addition, a task template is shown at the fine-tuning entry only when it carries the label that the fine-tuning entry filters on; a template uploaded without it does not appear. + +### Cause 2: an old template with missing fields is still in use after a Volcano upgrade conflict + +A conflict during the Volcano component upgrade left the old component in place. The fine-tuning task was therefore created from the old task template, which lacks fields that the current configuration form needs, and the form renders as a blank page. Uploading the current template again makes the page display normally. + +### Cause 3: the text input column takes text columns only + +The dataset text input column must contain only text columns and must not include the label column. More than one column can be selected. + +## Resolution + +### Cause 1: no template has been uploaded, or the template lacks the label the fine-tuning entry filters on + +1. Check whether template projects exist in the GitLab that Alauda AI uses. The built-in templates are GitLab projects under `/aml-templates/`, for example `/aml-templates/finetune-text-classification`: + + ```bash + GITLAB=https:// + TOKEN= + curl -s -H "PRIVATE-TOKEN: ${TOKEN}" "${GITLAB}/api/v4/projects?search=aml-templates&simple=false" \ + | jq '.[] | {id, path_with_namespace, topics}' + curl -s -H "PRIVATE-TOKEN: ${TOKEN}" "${GITLAB}/api/v4/projects?search=finetune&simple=false" \ + | jq '.[] | {id, path_with_namespace, topics}' + ``` + +2. If no template project exists, extract the example templates from the `alaudadockerhub/training-templates` image and upload them to **Task Template**, following "Enable Fine-Tuning and Training Features" for your Alauda AI version. Make sure all Bash scripts in the template (especially `run.sh`) are executable before uploading, as "Developing Custom Fine-Tuning Templates" requires. +3. If templates exist but are not listed, compare their labels and topics with a template that is listed (in another namespace or environment), and add the label the fine-tuning entry expects, as the Alauda AI fine-tuning template documentation of your version describes. +4. Reopen **Create Fine-tuning Task** and check the list. + +### Cause 2: an old template with missing fields is still in use after a Volcano upgrade conflict + +1. Check for an old Volcano component next to the new one: + + ```bash + kubectl get pods -A | grep -i volcano + kubectl get deploy -A | grep -i volcano + kubectl get apprelease -A | grep -i volcano + kubectl get minfo | grep | grep -i volcano + ``` + + Two generations of Volcano workloads, or an AppRelease of the previous version that is still present, matches this cause. + +2. Compare the template in use with the template package of the installed Alauda AI version; the old template lacks fields the new one has. +3. Upload the current template again to the same **Task Template** location. Reopen **Create Fine-tuning Task**, select the template, and confirm that the configuration form renders. + +### Cause 3: the text input column takes text columns only + +Select one or more text columns as the text input column and leave the label column out. + +## Diagnostic Steps + +1. Confirm the feature is on: + + ```bash + kubectl get amlcluster default -o jsonpath='{.spec.values.experimentalFeatures}{"\n"}' + ``` + + The output should include `tuneModels: true`. + +2. Open the browser developer tools (F12) on the fine-tuning page. An empty template list returned with HTTP `200` points at missing or unlabeled templates (Cause 1). A `500` from `aml-api-deploy-svc` points at the backend, not at templates. +3. For a blank form, note which template was selected and compare its files with the current template package (Cause 2). +4. A question about which dataset columns to select is Cause 3. diff --git a/docs/en/solutions/Alert_notifications_are_not_delivered_because_courier-api_or_apollo_cannot_reach_the_receiver_on_ACP.md b/docs/en/solutions/Alert_notifications_are_not_delivered_because_courier-api_or_apollo_cannot_reach_the_receiver_on_ACP.md new file mode 100644 index 000000000..c1c4b8599 --- /dev/null +++ b/docs/en/solutions/Alert_notifications_are_not_delivered_because_courier-api_or_apollo_cannot_reach_the_receiver_on_ACP.md @@ -0,0 +1,151 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.0" + - "4.2" + - "4.3" +--- + +# Alert notifications are not delivered because courier-api or apollo cannot reach the receiver on ACP + +## Issue + +The real-time alert page shows a firing alert, but the mailbox, Feishu bot, DingTalk group, or external monitoring center receives nothing. A test send times out, a webhook POST fails with `tls: failed to verify certificate`, or webhook messages to a receiver addressed by host name only start working after DNS entries are added. A related question is which component, and which nodes, need network access to the mail server or the webhook endpoint for an egress allow-list. + +On Alauda Container Platform: + +- **Mail** notifications are sent by `apollo`. Its Pod is not pinned to a node and can move after a restart, so the mail server must be reachable from **all nodes of the `global` cluster**. +- **DingTalk, Feishu, and webhook** notifications are sent by `courier-api`, which can be scheduled onto any node of the cluster it runs on (normally `global`). A firewall rule that allows only one node is not enough. + +If the receiver answers with an HTTP error such as `errcode` or `InvalidJson`, the request already left the cluster; see [Webhook alert notifications rejected as InvalidJson or sent without the notification template on ACP](Webhook_alert_notifications_rejected_as_InvalidJson_or_sent_without_the_notification_template_on_ACP.md) or [DingTalk robot rejects alert notifications with errcode 300001 or 310000 on ACP](DingTalk_robot_rejects_alert_notifications_with_errcode_300001_or_310000_on_ACP.md). Email failures with `535` or emails marked as sent that never arrive are in [Email alert notifications fail with 535 or never arrive, or only Email is offered as a method on ACP](Email_alert_notifications_fail_with_535_or_never_arrive_or_only_Email_is_offered_as_a_method_on_ACP.md). A missing custom header or `invalid header field value` is in [Webhook alert notifications fail with invalid header field value or miss a required custom header on ACP](Webhook_alert_notifications_fail_with_invalid_header_field_value_or_miss_a_required_custom_header_on_ACP.md). + +## Root Cause + +### Cause 1: the sender cannot reach the SMTP or webhook server and times out + +A test send to Feishu, DingTalk, a webhook, or email times out, for example a `courier-api` connect timeout to the Feishu bot webhook on port 443 that still fails after retries. Email fails the same way when the `global` cluster cannot open the configured SMTP server. Connectivity from a laptop, or from a single node, does not prove the path: the sending Pod can run on any node, so every node needs the route, and an allow-list that names a Pod does not match this traffic. + +For QQ Mail, the working settings are SMTP over SSL on port 465 or 587, the full mailbox address as the user name, and the generated authorization code, not the account password, as the password. + +### Cause 2: tls: failed to verify certificate because the certificate does not name the webhook host + +The webhook POST fails and the notification shows `status: False`, `type: Ready`, `phase: Failed` with a message of this shape: + +```text +tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, 192.0.2.50, not 192.0.2.4 +``` + +The certificate presented by the webhook endpoint does not list the host used in the webhook URL. On platform v3.18.2, HTTPS certificate verification for webhook notifications cannot be skipped. + +### Cause 3: the webhook host name does not resolve inside the cluster + +Webhook-forwarded alerts to an external system are not delivered, and they are delivered as soon as host entries for the receiver are added to CoreDNS on the `global` cluster. `courier-api` runs as a Pod and resolves the webhook host through cluster DNS (CoreDNS). A name that is known only to a resolver that CoreDNS does not forward to, or only to `/etc/hosts` on a workstation, is not resolvable from the Pod. + +## Resolution + +### Cause 1: the sender cannot reach the SMTP or webhook server and times out + +1. Find the sending Pods and their nodes, then read the send log. Do not restart `courier-api` to "refresh" a timeout; the next schedule lands on the same blocked path. + + ```bash + kubectl get pods -A -o wide | grep -E 'courier-api|apollo' + kubectl logs -n cpaas-system deploy/courier-api --tail=200 | grep -i -E 'Failed to send|timeout|dial|refused|errcode' + kubectl logs -n cpaas-system --tail=200 | grep -i -E 'timeout|dial|refused' + ``` + + Use the namespace from the first command if the Pods are not in `cpaas-system`. For a mail failure, read both logs. + +2. From the node that hosts the Pod (or from a debug Pod with `hostNetwork` on that node), open the same host and port that the log names, using the real webhook URL or SMTP host: + + ```bash + curl -sv --max-time 10 "https:///" + ``` + +3. A timeout here, matching the timeout in the log, is the firewall or the route. Allow the webhook port (443 for a Feishu or DingTalk robot) and the SMTP port from every node of the cluster that runs the sender, or from the NAT address those nodes share. Allow by node address, not by Pod; the Pod name changes on every reschedule. + +4. Send one test notification. Success is a send log line without a timeout and a message at the receiver. An HTTP response with `errcode` means the path works and the payload is wrong. + +### Cause 2: tls: failed to verify certificate because the certificate does not name the webhook host + +1. Confirm the error before changing a URL or a certificate: + + ```bash + kubectl logs -n cpaas-system deploy/courier-api --tail=300 | grep -F 'failed to verify certificate' + ``` + +2. Use one of the two recoveries: + - Issue a certificate on the webhook endpoint that includes the host used in the URL. This is done on the webhook server, not in the cluster. + - Change the webhook address to a host that the certificate already lists (the names in the `x509` message). + + > **Warning:** Changing the webhook address stops posts to the previous endpoint as soon as the next alert fires. Do this only when the log says the certificate is valid for names other than the host in the current URL. Write down the current URL first; to roll back, put it back. + +### Cause 3: the webhook host name does not resolve inside the cluster + +1. Check whether the webhook host resolves from a Pod on the `global` cluster: + + ```bash + kubectl logs -n cpaas-system deploy/courier-api --tail=300 | grep -i -E 'lookup|no such host' + kubectl run dns-check -n cpaas-system --rm -it --restart=Never --image= -- nslookup + ``` + + `NXDOMAIN` or no answer from the debug Pod, while the same host resolves from the site's own resolver, matches this cause. + +2. Preferably, make the name resolvable on the DNS servers that the nodes use, so that CoreDNS forwards it. If that is not possible, add a static record to cluster DNS. The persistent way is a **Host Alias** entry in the **Alauda Container Platform Cluster DNS** plugin, as described in [Platform console does not open because the cluster VIP domain name cannot be resolved inside the cluster on ACP](Platform_console_does_not_open_because_the_cluster_VIP_domain_name_cannot_be_resolved_inside_the_cluster_on_ACP.md). + +3. To add the record directly to the Corefile, find the ConfigMap that the `coredns` Deployment mounts (for example `kube-system/cpaas-coredns`, data key `Corefile`) and export it: + + ```bash + kubectl get deploy -n kube-system coredns -o jsonpath='{.spec.template.spec.volumes}{"\n"}' + kubectl get cm -n kube-system cpaas-coredns -o yaml > /tmp/cpaas-coredns.yaml + ``` + + > **Warning:** The Corefile drives DNS for every Pod in the cluster. A syntax error makes CoreDNS fail to parse it and crash-loop, which breaks name resolution cluster-wide. Do this only when the webhook host does not resolve from a Pod and the site's DNS cannot serve it. A manual edit can be overwritten when the Cluster DNS plugin is reconciled. To roll back, run `kubectl apply -f /tmp/cpaas-coredns.yaml` and confirm the `coredns` Pods are `Running` and `Ready`. + + ```bash + kubectl edit cm -n kube-system cpaas-coredns + ``` + + Add a `hosts` block to the existing server block, before the `forward` line, and keep `fallthrough` so that every other name still resolves: + + ```text + hosts { + 192.0.2.10 + fallthrough + } + ``` + + If the Corefile has the `reload` plugin, CoreDNS picks up the change on its own; otherwise run `kubectl rollout restart deploy -n kube-system coredns`. + +4. Repeat the `nslookup` from the debug Pod, then trigger one alert. Success is a `courier-api` send for that webhook with phase `Done` and the message at the receiver. + +## Diagnostic Steps + +1. Locate the sender. For mail it is `apollo`; for DingTalk, Feishu, and webhooks it is `courier-api`. List where it runs and where it can run: + + ```bash + kubectl get pods -A -o wide | grep -E 'courier-api|apollo' + kubectl get deploy -A | grep courier-api + kubectl get deploy -n courier-api -o jsonpath='{.spec.template.spec.nodeSelector}{"\n"}{.spec.template.spec.affinity}{"\n"}' + ``` + + The destination port is the one in the webhook URL (443 for `https://`). + +2. Read the `courier-api` log, and the `apollo` log for mail, for one failed send: + - `dial` or `i/o timeout`: Cause 1. + - `failed to verify certificate`: Cause 2. + - `lookup` or `no such host`, or a host name that a debug Pod cannot resolve: Cause 3. + - `errcode` or `InvalidJson`: the path works; use the payload articles linked in the Issue section. + - `invalid header field value`: nothing was sent; use the header article. + +3. If a webhook receiver gets some alerts and not others, read the `webhook-business` log. A render failure with invalid JSON means the alert was discarded before sending, which is a template problem, not the network: + + ```bash + kubectl get pods -A | grep webhook-business + kubectl logs -n --tail=300 | grep -i -E 'json|render|invalid' + ``` + +4. After a network or DNS change, repeat the test from a Pod on a different node. One successful node does not cover the scheduler. diff --git a/docs/en/solutions/Alert_repeat_interval_cannot_be_set_to_4_hours_or_the_global_sending_interval_does_not_take_effect_on_ACP.md b/docs/en/solutions/Alert_repeat_interval_cannot_be_set_to_4_hours_or_the_global_sending_interval_does_not_take_effect_on_ACP.md new file mode 100644 index 000000000..54aebf191 --- /dev/null +++ b/docs/en/solutions/Alert_repeat_interval_cannot_be_set_to_4_hours_or_the_global_sending_interval_does_not_take_effect_on_ACP.md @@ -0,0 +1,85 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.3.x +--- + +# Alert repeat interval cannot be set to 4 hours, or the global sending interval does not take effect on ACP + +## Issue + +One of these is reported about how often alert notifications repeat: + +- The repeat interval should be 4 hours. On an alert policy, **Policy Configuration** (`策略配置`) > **Alert Sending Interval** (`告警发送间隔`) offers **Global** (`全局`) or **Custom** (`自定义`). With Custom, each level (`灾难告警` disaster, `严重告警` critical, `警告告警` warning, `提示告警` info) has a dropdown with 3 hours and 6 hours, and no 4-hour choice. +- The interval should be changed for all alerts at once. +- The sending interval is changed in the UI (for example to 3 hours, or the global interval), and the notifications keep the old rhythm; the change is visible only after editing the configuration from the backend. The VictoriaMetrics Alertmanager configuration is Secret `vmalertmanager-alertmanager` in `cpaas-system`. + +Related article: [Node zombie process alert keeps switching between firing and resolved on ACP](Node_zombie_process_alert_keeps_switching_between_firing_and_resolved_on_ACP.md). + +## Root Cause + +### Cause 1: 4 hours is not one of the offered repeat intervals + +The per-policy dropdown offers 3 hours and 6 hours; a 4-hour repeat interval is not supported. The interval for all alerts is set globally on **Operations Center > Alerts > Alert Settings** (`运维中心 / 告警 / 告警设置`), section **Alert Sending Interval (Global)** (`告警发送间隔 (全局)`). Example values there: disaster 5 minutes, critical 15 minutes, warning 1 hour, info 2 hours. Policies whose sending interval is **Global** use these values; policies set to **Custom** keep their own per-level values. The same page has a separate monitoring-component exception alert section (`监控组件异常告警`) with its own check interval (5 minutes) and notification interval (1 day). + +### Cause 2: the UI interval is not written through to the Alertmanager configuration + +On ACP 4.0 (for example 4.0.11), a sending interval changed in the UI is not written to the `alertmanager.yaml` key of Secret `vmalertmanager-alertmanager`, so Alertmanager keeps the old configuration. This is a known product issue, fixed in ACP 4.2, where the manual procedure below is no longer needed. + +The `warlock` Deployment in `cpaas-system` writes that Secret back. A manual edit therefore has to be made while `warlock` is scaled to 0, and the checksum annotation on ConfigMap `alert-repeat-config` (key `alertmanager.yamMD5`, with that spelling) has to match the file. Any later change of the sending interval in the UI rewrites the Secret and undoes the manual change, so the procedure has to be repeated after each such UI change. + +## Resolution + +### Cause 1: 4 hours is not one of the offered repeat intervals + +Pick 3 hours or 6 hours on each alert policy. To change the interval for all alerts, open **Operations Center > Alerts > Alert Settings**, click **Update** in **Alert Sending Interval (Global)**, set each level, and make sure the policies that should follow it have their sending interval set to **Global**. If the saved global value does not show on the Alertmanager status page, see Cause 2. The backend procedure in Cause 2 does not add a 4-hour choice. + +### Cause 2: the UI interval is not written through to the Alertmanager configuration + +1. Read the current state and keep copies: + + ```bash + kubectl get deploy -n cpaas-system warlock -o jsonpath='{.spec.replicas}{"\n"}' + kubectl get secret -n cpaas-system vmalertmanager-alertmanager -o go-template='{{index .data "alertmanager.yaml"}}' | base64 -d > /tmp/alertmanager.yaml + cp /tmp/alertmanager.yaml /tmp/alertmanager.yaml.bak + kubectl get cm -n cpaas-system alert-repeat-config -o yaml > /tmp/alert-repeat-config.yaml + ``` + +2. Edit `/tmp/alertmanager.yaml` and set the value that should be live, for example `group_interval: 1m`. + +3. Apply it with `warlock` stopped. + + > **Warning:** With `warlock` at 0 replicas the platform does not reconcile alert configuration until it is scaled back, and patching `vmalertmanager-alertmanager` changes the configuration every notification uses. Precondition: the UI value is missing from the Alertmanager status page, `/tmp/alertmanager.yaml.bak` and `/tmp/alert-repeat-config.yaml` exist, and the `warlock` replica count is written down. Rollback: patch the Secret from the backup file with the same command, `kubectl apply -f /tmp/alert-repeat-config.yaml`, and scale `warlock` back. + + ```bash + kubectl scale deploy -n cpaas-system warlock --replicas=0 + kubectl patch secret vmalertmanager-alertmanager -n cpaas-system --type=merge \ + -p "{\"data\":{\"alertmanager.yaml\":\"$(base64 < /tmp/alertmanager.yaml | tr -d '\n')\"}}" + md5sum /tmp/alertmanager.yaml | awk '{print $1}' + kubectl edit cm -n cpaas-system alert-repeat-config + kubectl scale deploy -n cpaas-system warlock --replicas= + ``` + + In `kubectl edit`, set the annotation `alertmanager.yamMD5` to the hash printed by `md5sum` (the hash only, not the file name). + +4. Wait about five minutes, open the Alertmanager status page, and confirm that the value matches the file. If the old value is still shown, the Secret key or the annotation was not the one `warlock` reads; do not patch other keys. + +5. After any later change of the sending interval in the UI, check the status page again and repeat the procedure if the value was reverted. On ACP 4.2 and later, set the interval in the UI only. + +## Diagnostic Steps + +1. A request for 4 hours, with the dropdown showing 3 hours and 6 hours: Cause 1. + +2. Decode the live configuration and compare it with the UI value: + + ```bash + kubectl -n cpaas-system get secret vmalertmanager-alertmanager -o jsonpath='{.data.alertmanager\.yaml}' | base64 -d | grep -n -E 'group_interval|repeat_interval' + ``` + + Expected for Cause 2: the interval saved in the UI does not appear in the file or on the Alertmanager status page. + +3. If notifications for one alert alternate between firing and resolved, the interval is not the problem; see the related article. diff --git a/docs/en/solutions/Alert_rules_do_not_fire_are_rejected_or_cannot_be_edited_on_ACP.md b/docs/en/solutions/Alert_rules_do_not_fire_are_rejected_or_cannot_be_edited_on_ACP.md new file mode 100644 index 000000000..5268c26c3 --- /dev/null +++ b/docs/en/solutions/Alert_rules_do_not_fire_are_rejected_or_cannot_be_edited_on_ACP.md @@ -0,0 +1,135 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Alert rules do not fire, are rejected, or cannot be edited on ACP + +## Issue + +The metric or the log line is there, but the real-time alert page stays empty, or creating the alert policy is rejected, or a change to a built-in policy does not last. Typical symptoms: + +- A log keyword alert chart goes from "no data" to `0` after the first matching line, and the policy does not fire. +- Creating the alert policy fails with `parse error: expected type instant vector in aggregation expression, got range vector`. +- A custom metric rule never appears on the real-time page, and the `VMRule` fails validation with an annotation template parsing error and an undefined variable. +- Rules exported from a Prometheus cluster do not load on a VictoriaMetrics cluster. +- A built-in (platform) alert policy is edited or deleted, and the change is reverted. The question is whether built-in policies can be modified, deleted, or extended. + +On Alauda Container Platform, platform rules are `AlertRule` objects (`ait.alauda.io/v1`) shipped by the `cpaas-monitor` module plugin (chart `chart-cpaas-monitor`). The `warlock` controller reconciles each `AlertRule` into a `PrometheusRule`, a `VMRule` on VictoriaMetrics clusters, and the Alertmanager configuration. + +## Root Cause + +### Cause 1: the first keyword match only creates the workload_log_keyword_count series + +A log keyword alert evaluates a rate over `workload_log_keyword_count`. Before the keyword has ever matched, there is no series at all; the value is not `0`. The first match only creates the series. A later match in a following window produces the increase that can satisfy `>= 1`. This is expected, not a missed evaluation. + +### Cause 2: a range vector is passed straight into an aggregation + +The mutating webhook `prometheusrules.mutating.alauda.io` rejects the object with `parse error: expected type instant vector in aggregation expression, got range vector`. The expression passes a range vector such as `metric[5m]` directly into `sum`, `avg`, `max`, `min`, or `count`, which require an instant vector. + +### Cause 3: an annotation template references an undefined variable + +The expression is true in vmui, but even a test rule such as CPU above 1 percent produces no real-time alert, and the controller log reports an annotation template parsing error with an undefined variable. The annotation template is evaluated when the `VMRule` is admitted, so the whole rule stays inactive. Plugin status and metric queries can both look healthy. + +### Cause 4: built-in alert rules cannot be deleted, and direct edits are reverted + +The rules the platform ships cannot be deleted, and direct modifications can be reverted by the platform: the `warlock` controller re-renders the generated `PrometheusRule` and `VMRule` objects from the `AlertRule`, and a module upgrade of `cpaas-monitor` can restore the chart copy of the `AlertRule` itself. Changing or extending a built-in policy in place is therefore not a supported way to get a lasting rule. Adding a label from metrics into a built-in alert expression is not an in-product configuration (stated for v4.2.4). + +An existing alert policy also refuses a change of compute component, and a template policy accepts only one compute component, because the rule set is bound to that resource type. + +Some conditions have no built-in rule at all: there is no alert for maintenance or license expiry, and on 4.0 `cpaas-cluster-rules` covers controller and scheduler health but no default policy covers the APIServer restart count or etcd read/write latency, although the dashboard already has metrics such as `etcd_disk_wal_fsync_duration_seconds_bucket`, `etcd_server_proposals_pending`, `etcd_server_proposals_failed_total`, and `grpc_server_handled_total{grpc_code!="OK"}`. These need a custom policy. + +### Cause 5: VictoriaMetrics evaluates VMRule, not PrometheusRule + +On a VictoriaMetrics cluster (for example ACP 4.3.2), the evaluator reads `VMRule` objects, not `PrometheusRule`, and a `VMRule` needs the `victoriametrics` label before `vmalert` selects it. A `PrometheusRule` exported from a Prometheus cluster (for example 3.10) is therefore not evaluated. + +## Resolution + +### Cause 1: the first keyword match only creates the workload_log_keyword_count series + +Do not lower the threshold and do not restart Alertmanager. Emit the keyword again after the series exists, and evaluate this shape (replace the matchers): + +```promql +round(sum(rate(workload_log_keyword_count{cluster_name="",kind=~"(?i:(Deployment))",name="",namespace="",query=""}[300s])) * 300) >= 1 +``` + +The notification does not contain the log line; it carries `query`, `cluster`, `namespace`, `name`, and `kind`. Retrieve the line from the log query pages or APIs. If the keyword never appears in the log query, the problem is log collection, not this rule. + +### Cause 2: a range vector is passed straight into an aggregation + +Wrap the range in `rate()` or `increase()` before aggregating, or drop the range selector if the metric is already an instant vector. Rejected: + +```promql +sum by (namespace) (container_cpu_usage_seconds_total[5m]) +``` + +Accepted: + +```promql +sum by (namespace) (rate(container_cpu_usage_seconds_total[5m])) +``` + +Run the expression in vmui before creating the policy. vmui is at `/clusters//vmselect/vmui/`; after login, remove a trailing `prometheus/` from the server URL in vmui and save, or the query page uses the wrong API. + +### Cause 3: an annotation template references an undefined variable + +1. Find the rule and the evaluator Pods: + + ```bash + kubectl get vmrule -A + kubectl get vmrule -n -o yaml + kubectl get pods -n cpaas-system | grep -E 'vmalert|vmoperator|victoria' + ``` + +2. Search the operator or `vmalert` Pod log for `template` and `undefined`. +3. In the custom annotation, remove or correct the unknown variable. Only labels, annotations, and built-in alert fields (`.labels.*`, `.annotations.*`, `.status`, `.startsAt`, `.endsAt`) can be referenced. +4. Save, confirm that the `VMRule` no longer reports a validation error, and check the real-time page again. + +### Cause 4: built-in alert rules cannot be deleted, and direct edits are reverted + +1. Do not edit or delete built-in rules. To alert on something else, or with a different threshold, create a custom alert policy (from the alert policy page, or as a separate `AlertRule`) next to the built-in ones, and copy the built-in expression into it if useful. +2. To stop a built-in rule from notifying, silence its alert policy from the console (**Set Silence**) instead of editing its `expr`. +3. For a different compute component, create a new policy; existing policies and template policies stay bound to their component. +4. Inventory rules from `PrometheusRule`, not only from `AlertRule`; `kubectl get alertrules.ait.alauda.io -A` misses policies that exist only as rendered objects: + + ```bash + kubectl get prometheusrules.monitoring.coreos.com -A + kubectl get prometheusrules.monitoring.coreos.com -n -o yaml + kubectl get vmrule -A + ``` + +Wrapped metric names such as `cpaas_advanced_kube_workload_status_replicas_available` (available replicas of a workload) and `cpaas_advanced_kube_pod_container_status_ready` (`0` not ready, `1` ready) are recording rules. Read their native expression from the recording rule in the `PrometheusRule` instead of guessing. + +### Cause 5: VictoriaMetrics evaluates VMRule, not PrometheusRule + +1. Export the old rules and look at an existing `VMRule` on the target cluster: + + ```bash + kubectl get prometheusrule -A -o yaml > /tmp/prometheusrule-export.yaml + kubectl get vmrule -A + kubectl get vmrule -n cpaas-system -o jsonpath='{.items[0].apiVersion}{"\n"}' + ``` + +2. Convert each group to a `VMRule`, copy `apiVersion` from the existing `VMRule`, and set the `victoriametrics` label as it appears on the existing `VMRule` objects. +3. Apply, then run every expression in vmui and confirm it returns data. An expression that worked on Prometheus can be empty on VictoriaMetrics if a metric or label name changed. + +## Diagnostic Steps + +1. Find the object. Use the `prometheusrules.monitoring.coreos.com` list, and `vmrule` on a VictoriaMetrics cluster. + +2. A create or update that returns the range-vector parse error: Cause 2. Expected after the fix: the policy is created. + +3. The object exists, vmui shows the expression is true, and the real-time page is empty: read the `VMRule` and the operator log for an annotation template error (Cause 3). + +4. For a log keyword policy, query `workload_log_keyword_count` with the same matchers. No series: the keyword has not matched yet, or logs are not collected. One series that just appeared with the chart at `0`: Cause 1. Wait for a second match. + +5. Rules carried over from a Prometheus cluster: compare `kubectl get prometheusrule -A` with `kubectl get vmrule -A`. A rule that exists only as a `PrometheusRule`, or a `VMRule` without the `victoriametrics` label, is Cause 5. + +6. A firing rule whose notification never arrives is not a rule problem; see [Alert notifications are not delivered because courier-api or apollo cannot reach the receiver on ACP](Alert_notifications_are_not_delivered_because_courier-api_or_apollo_cannot_reach_the_receiver_on_ACP.md). diff --git a/docs/en/solutions/All_charts_stay_ChartDownloading_on_a_new_Kube-OVN_cluster_because_ovn-default_was_created_as_overlay_on_ACP.md b/docs/en/solutions/All_charts_stay_ChartDownloading_on_a_new_Kube-OVN_cluster_because_ovn-default_was_created_as_overlay_on_ACP.md new file mode 100644 index 000000000..6daf6bbf4 --- /dev/null +++ b/docs/en/solutions/All_charts_stay_ChartDownloading_on_a_new_Kube-OVN_cluster_because_ovn-default_was_created_as_overlay_on_ACP.md @@ -0,0 +1,74 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# All charts stay ChartDownloading on a new Kube-OVN cluster because ovn-default was created as overlay on ACP + +Related articles: [Node, API and VIP access lost after creating a Kube-OVN ProviderNetwork or underlay subnet on ACP](Node_API_and_VIP_access_lost_after_creating_a_Kube-OVN_ProviderNetwork_or_underlay_subnet_on_ACP.md); [Kube-OVN underlay Pods fail with resolve MAC address timeout or bridge network InitOVSBridgeFailed on ACP](Kube-OVN_underlay_Pods_fail_with_resolve_MAC_address_timeout_or_bridge_network_InitOVSBridgeFailed_on_ACP.md). + +## Issue + +On a newly added cluster that was planned with a Kube-OVN underlay network, CoreDNS and the other platform components are never installed. `kubectl get ars -A` shows every AppRelease in `cpaas-system` (`alauda-alb2`, `cni-kube-ovn`, `coredns`, `prometheus`, `region` and others) in `ChartDownloading`, and `kubectl get rpch` returns `No resources found`. Describing one AppRelease shows: + +```text +ChartDownloadFailed [1/1] failed to download chart acp/chart-csi-driver-nfs-v3.18.3, err: failed to do request: Head "http://:60080/v2/acp/chart-csi-driver-nfs/manifests/v3.18.3": dial tcp :60080: connect: connection timed out +``` + +At the same time: + +- `telnet` and `curl -v http://:60080/v2` from the cluster nodes succeed (the answer is `Client sent an HTTP request to an HTTPS server.`), and every Pod is `Running`. +- Restarting `sentry` has no effect. Its log has no download error, only `too many processors, process later ... MaxConcurrentProcessors=10 CurrentProcessors=10`. +- OVN logs show nothing abnormal. + +## Root Cause + +### Cause 1: ovn-default was created as an overlay subnet although the cluster was planned as underlay + +The `ovn-default` subnet was meant to be underlay but was configured as overlay: `kubectl get subnet` shows `ovn-default` with `NAT` `true` and no VLAN. In this network design the Pods could not reach the registry on that path, so every chart download from inside the cluster timed out, while the nodes, which use the host network, could reach it. From inside the `sentry` Pod: + +```text +curl: (28) Failed to connect to port 60080 after 130152 ms: Could not connect to server +``` + +A subnet that references a VLAN is an underlay subnet; one without a VLAN is an overlay subnet. + +## Resolution + +### Cause 1: ovn-default was created as an overlay subnet although the cluster was planned as underlay + +1. Confirm the signature: the node `curl` answers, the `curl` from `sentry` fails, and `ovn-default` has no VLAN although the cluster was planned as underlay (see Diagnostic Steps). + +2. Redeploy the cluster with `ovn-default` as an underlay subnet (with the planned VLAN, gateway and NIC). + + > **Warning:** Redeploying deletes the cluster and everything running on it, and it cannot be undone. Only when the cluster is newly added and has no business workloads yet, and the signature in step 1 is confirmed. If the `curl` from `sentry` works, or `ovn-default` already has the intended type, this is not the fault: look at DNS resolution of the registry address and the `sentry` component instead, which also leave charts in `ChartDownloading`. + +3. After the redeploy, test from a Pod on the underlay subnet to the registry and to the global cluster's platform address. A firewall between the underlay subnet and the global cluster can still block that path; open it on the network side. + +## Diagnostic Steps + +1. List the AppReleases stuck in download and read the error: + + ```bash + kubectl get ars -A | grep -i chartdownloading + kubectl -n cpaas-system describe ars | grep -A2 ChartDownloadFailed + ``` + +2. Compare the node path with the Pod path to the registry. The node should answer; a timeout from `sentry` matches this issue: + + ```bash + curl -v --max-time 10 http://:60080/v2/ + kubectl -n cpaas-system get pod | grep sentry + kubectl -n cpaas-system exec -- curl -v --max-time 10 http://:60080/v2/ + ``` + +3. Check the type of `ovn-default`. An empty `vlan` with `nat=true` on a cluster planned as underlay confirms the cause: + + ```bash + kubectl get subnet ovn-default -o jsonpath='vlan={.spec.vlan} provider={.spec.provider} nat={.spec.natOutgoing}{"\n"}' + ``` diff --git a/docs/en/solutions/An_unknown_source_IP_in_the_Calico_Pod_CIDR_that_is_not_a_Pod_or_Service_is_a_node_tunl0_address_on_ACP.md b/docs/en/solutions/An_unknown_source_IP_in_the_Calico_Pod_CIDR_that_is_not_a_Pod_or_Service_is_a_node_tunl0_address_on_ACP.md new file mode 100644 index 000000000..fd2d7d7cc --- /dev/null +++ b/docs/en/solutions/An_unknown_source_IP_in_the_Calico_Pod_CIDR_that_is_not_a_Pod_or_Service_is_a_node_tunl0_address_on_ACP.md @@ -0,0 +1,77 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# An unknown source IP in the Calico Pod CIDR that is not a Pod or Service is a node tunl0 address on ACP + +## Issue + +On a cluster that uses Calico in IPIP mode: + +- The access log of an application Pod keeps showing one client address sending requests to paths the application does not have, and every request returns `404` in rapid succession. Examples: `"POST /seeyon/rest/authentication/ucpcLogin HTTP/1.1" 404`, `"GET /index.php?option=com_joomlapicasa2&controller=../../../../etc/passwd%00 HTTP/1.1" 404`, `"GET /geoserver/ows?service=WFS&version=1.0.0&request=GetCapabilities HTTP/1.1" 404`, `"GET /actuator/httptrace HTTP/1.1" 404`. +- The question is which Pod has this IP. `kubectl get pods -A -o wide` and `kubectl get svc -A -o wide` do not find the address, although it is inside the Pod CIDR and answers `ping`. +- The address may end in `.0`, which looks like a network address and not a host. +- The requests may stop on their own before anyone can capture them. + +## Root Cause + +### Cause 1: the address is the Calico IPIP tunnel address (tunl0) of a node, so the sender is a process on that node + +Calico allocates each node a block of the IP pool (block size from the `IPPool` `blockSize`, for example in `default-ipv4-ippool`) and gives the node's `tunl0` interface an address from its own block. That address is in the Pod CIDR but belongs to no Pod or Service, which is why the usual lookups return nothing. + +A process running on the host itself that connects to a Pod on another node goes out through `tunl0`, and the kernel uses the `tunl0` address as the source. The application therefore logs the node's tunnel address as the client. Because blocks start on block boundaries, the first address of a block can end in `.0` and still be a valid tunnel address. + +The request paths above match a vulnerability scan. When the node that owns the address runs no application Pods (for example a master with scheduling disabled) and platform audit and container logs do not line up with the requests, the requests did not come from a Pod; security or scanning software installed on that node is the likely origin. Kubernetes, Pod IP allocation and the application itself are not involved. + +## Resolution + +### Cause 1: the address is the Calico IPIP tunnel address (tunl0) of a node, so the sender is a process on that node + +1. Map the address to a node. The node annotation holds each node's IPIP tunnel address, and the IPAM block shows which node owns the range: + + ```bash + kubectl get nodes -o custom-columns='NAME:.metadata.name,TUNL0:.metadata.annotations.projectcalico\.org/IPv4IPIPTunnelAddr' + kubectl get ippools.crd.projectcalico.org -o yaml | grep -E 'name:|cidr|blockSize|ipipMode' + kubectl get ipamblocks.crd.projectcalico.org -o yaml | grep -B 15 -A 30 '' + ``` + + On the node that owns the block, confirm the interface address: + + ```bash + ip -br addr show tunl0 + ``` + +2. Treat the node, not the cluster, as the sender. Check what runs on that node and whether any Pod there could be the client: + + ```bash + kubectl get pod -A -o wide --field-selector spec.nodeName= + kubectl get events -A -o wide --sort-by=.lastTimestamp | grep + ``` + +3. While the requests are arriving, find the local process. Run these on the node that owns the address: + + ```bash + ss -tnp | grep ':' + tcpdump -nn -i tunl0 host and port -c 20 + ``` + + `ss -p` prints the process name and PID of the connection. A host process (not a container) as the owner confirms this cause. + +4. If the requests have already stopped, read the node's OS logs and the scan records of any security or vulnerability-scanning software installed on the node for the same time window, and ask its owner whether a scan ran. + +## Diagnostic Steps + +Rule out cluster objects first, then look the address up in Calico IPAM: + +```bash +kubectl get pods -A -o wide | grep -w +kubectl get svc -A -o wide | grep -w +kubectl get ipamblocks.crd.projectcalico.org -o yaml | grep -B 15 -A 30 '' +kubectl get nodes -o custom-columns='NAME:.metadata.name,TUNL0:.metadata.annotations.projectcalico\.org/IPv4IPIPTunnelAddr' | grep -w +``` + +No Pod or Service, an address inside a block whose `affinity` is `host:`, and that node's `tunl0` carrying the address is Cause 1. `calicoctl` is not needed for these lookups; the IPAM blocks are custom resources. `arp -n` on the application node shows nothing for this address, because the traffic arrives through the IPIP tunnel rather than on the local segment. diff --git a/docs/en/solutions/AppRelease_abnormal_because_a_ModuleInfo_of_a_deleted_cluster_remains_on_ACP.md b/docs/en/solutions/AppRelease_abnormal_because_a_ModuleInfo_of_a_deleted_cluster_remains_on_ACP.md new file mode 100644 index 000000000..62127e142 --- /dev/null +++ b/docs/en/solutions/AppRelease_abnormal_because_a_ModuleInfo_of_a_deleted_cluster_remains_on_ACP.md @@ -0,0 +1,66 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# AppRelease abnormal because a ModuleInfo of a deleted cluster remains on ACP + +## Issue + +An AppRelease (`ars`) is reported abnormal. The cluster it relates to has already been deleted from the platform, but a `ModuleInfo` (`minfo`) for that cluster is still present on the global cluster, for example the `clickhouse-updater` module of the deleted cluster. + +Related article: [Monitoring plugin uninstall or Prometheus to VictoriaMetrics switch leaves ModuleInfo, AppRelease, or Pods on ACP](Monitoring_plugin_uninstall_or_Prometheus_to_VictoriaMetrics_switch_leaves_ModuleInfo_AppRelease_or_Pods_on_ACP.md). + +## Root Cause + +### Cause 1: a ModuleInfo of the deleted cluster was left behind + +A `ModuleInfo` describes a module installed on one cluster; the label `cpaas.io/cluster-name` names that cluster. When the cluster was deleted, the `clickhouse-updater` `minfo` of that cluster was not removed with it. That leftover `minfo` for a cluster that no longer exists makes the AppRelease abnormal. + +## Resolution + +### Cause 1: a ModuleInfo of the deleted cluster was left behind + +1. On the global cluster, confirm that the cluster no longer exists and list the `minfo` objects that still point at it: + + ```bash + kubectl get cls | grep + kubectl get minfo -A -l cpaas.io/cluster-name= + kubectl get minfo -A | grep + ``` + +2. Back up each leftover `minfo` of the deleted cluster, then delete it. + + > **Warning:** Deleting a `ModuleInfo` removes the platform's record of that module installation. Precondition: the cluster named on the `minfo` no longer exists, and the backup file is saved. Do not delete a `minfo` whose cluster still exists. Rollback: `kubectl apply -f /tmp/.yaml`. + + ```bash + kubectl get minfo -o yaml > /tmp/.yaml + kubectl delete minfo + ``` + +3. Check the AppReleases again; the abnormal status clears. + + ```bash + kubectl get ars -A | grep -v -E 'Ready|Synced' + ``` + +## Diagnostic Steps + +1. List the clusters the platform knows (`kubectl get cls`) and compare them with the clusters named on `ModuleInfo` objects: + + ```bash + kubectl get minfo -A -o custom-columns='NAME:.metadata.name,CLUSTER:.metadata.labels.cpaas\.io/cluster-name,MODULE:.metadata.labels.cpaas\.io/module-name' + ``` + + Expected for this cause: one or more `minfo` objects (for example module `clickhouse-updater`) whose cluster is no longer in the cluster list. + +2. List AppReleases that are not healthy: + + ```bash + kubectl get ars -A | grep -v -E 'Ready|Synced' + ``` + + An abnormal AppRelease that relates to the deleted cluster, with its `minfo` still present, is this cause. An abnormal AppRelease for a cluster that still exists is not. diff --git a/docs/en/solutions/Application_breaks_after_a_NetworkPolicy_change_Ingress_Pod-to-Pod_ping_Redis_Sentinel_or_Harbor_on_ACP.md b/docs/en/solutions/Application_breaks_after_a_NetworkPolicy_change_Ingress_Pod-to-Pod_ping_Redis_Sentinel_or_Harbor_on_ACP.md new file mode 100644 index 000000000..089f5de73 --- /dev/null +++ b/docs/en/solutions/Application_breaks_after_a_NetworkPolicy_change_Ingress_Pod-to-Pod_ping_Redis_Sentinel_or_Harbor_on_ACP.md @@ -0,0 +1,154 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Application breaks after a NetworkPolicy change: Ingress, Pod-to-Pod ping, Redis Sentinel or Harbor on ACP + +## Issue + +An application that worked, or a new one, fails, and a network policy is involved: + +- An Ingress created on the platform for a Service port (for example `8080`) cannot be reached. +- Two Pods in the same namespace cannot ping each other in either direction: `2 packets transmitted, 0 received, 100% packet loss`. +- A Redis instance in Sentinel mode does not start after it is created. The `rfr-` StatefulSet Pods are in `CrashLoopBackOff`, and their logs show every call to the Kubernetes API server timing out: + - `Get ".../configmaps/rfr-acl-": context deadline exceeded`, `ERROR load acl failed` + - `Get ".../pods/rfs--0": dial tcp ...:443: i/o timeout` + - `level=fatal msg="Error getting password from secret rfr-acl--operator-secret ...: context deadline exceeded"` + - `ERROR Access enable nodeport service access failed`, `ERROR Access load cached config failed` + + Events show `FailedPreStopHook`, `Back-off restarting failed container redis-exporter` and `Startup probe failed: dial tcp ...: connect: connection refused`. +- Harbor is inaccessible, and the `harbor-nginx` Pod stays `Running` with `0/1` Ready. + +In each case a policy was the cause, and correcting or removing it restored service. + +## Root Cause + +### Cause 1: a NetworkPolicy blocks the ingress controller from the backend + +A `NetworkPolicy` that selects the application Pods does not allow traffic from the ingress controller. Policies match what arrives at the Pod: the source is the ingress controller (a Pod, or the node when the controller runs on the host network), and the port is the container port behind the Service (`targetPort`), which can differ from the Service port. + +### Cause 2: a policy with Egress blocks Pod-to-Pod traffic in the namespace + +A network policy in the namespace selects the Pods with `Egress` in `policyTypes` and has no rule that allows traffic to the other Pods, so outbound traffic from those Pods is dropped. + +### Cause 3: the Redis Sentinel Pods cannot reach the Kubernetes API server + +At startup the Redis and Sentinel containers read their ACL ConfigMap, password Secret and Pod objects from the API server (Service port `443`). The required network path had not been opened, so every request timed out and the containers exited. Adjusting the Sentinel resource limits is not the fix. The block can be a Kubernetes `NetworkPolicy` or a rule on the network side, so check both. + +### Cause 4: harbor-nginx is 0/1 Ready after the cluster network policies were modified + +The cluster network policies were changed just before Harbor stopped serving, and restoring the previous policy configuration brought `harbor-nginx` back to Ready. + +## Resolution + +Save every policy before changing any of them, so each step can be undone: + +```bash +kubectl get networkpolicy -A -o yaml > /tmp/all-netpol.before.yaml +kubectl get adminnetworkpolicy -o yaml > /tmp/all-anp.before.yaml +``` + +### Cause 1: a NetworkPolicy blocks the ingress controller from the backend + +Find the container port and the controller's source, then add an ingress allow for that pair to the policy: + +```bash +kubectl -n get svc -o jsonpath='{.spec.ports}{"\n"}' +kubectl get pod -A -o wide | grep -iE 'alb|ingress' +kubectl -n get networkpolicy -o yaml +``` + +```yaml + ingress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: + ports: + - protocol: TCP + port: +``` + +If the controller Pods use the host network, a namespace selector does not match them; allow the node addresses, or on Kube-OVN the join subnet, with `ipBlock`. + +> **Warning:** Editing the policy changes who can reach every Pod it selects. Roll back with `kubectl apply -f /tmp/all-netpol.before.yaml`. + +```bash +kubectl -n edit networkpolicy +curl -sv --max-time 5 http:/// -o /dev/null +``` + +### Cause 2: a policy with Egress blocks Pod-to-Pod traffic in the namespace + +Delete the policy, or keep it and add an egress rule `to: - podSelector: {}`, which allows traffic to every Pod in the same namespace. + +> **Warning:** Deleting the policy removes all the isolation it gave to the Pods it selected. Get agreement from the namespace owner. Roll back with `kubectl apply -f /tmp/all-netpol.before.yaml`. + +```bash +kubectl -n delete networkpolicy +kubectl -n exec -- ping -c 3 +``` + +### Cause 3: the Redis Sentinel Pods cannot reach the Kubernetes API server + +Only when the Redis or Sentinel logs show `context deadline exceeded` or `dial tcp ...:443: i/o timeout` on API server requests. Open the path from the instance Pods to the API server: + +- If a `NetworkPolicy` with `Egress` selects those Pods, add an egress allow to the API server. On Kube-OVN the rule has to name the backend endpoints of the `kubernetes` Service (the addresses in its EndpointSlice, port `6443`), not only the ClusterIP; the rule shape is in "Operator manager pod stuck in CrashLoopBackOff when a NetworkPolicy denies egress to the apiserver on ACP". +- If no policy selects the Pods, the block is on the network path from the Pod network to the control-plane nodes; have it opened there. + +> **Warning:** An egress allow changes what the selected Pods can reach; limit it to the instance Pods and the API server endpoints. Roll back by deleting the allow policy you added. + +```bash +kubectl -n logs --all-containers --tail=50 | grep -E 'deadline|i/o timeout|failed' +kubectl -n get networkpolicy +kubectl get endpointslice -n default -l kubernetes.io/service-name=kubernetes +kubectl -n apply -f allow-apiserver-egress.yaml +kubectl -n get pod -w | grep -E 'rfr-|rfs-' +``` + +### Cause 4: harbor-nginx is 0/1 Ready after the cluster network policies were modified + +Put back the configuration from before the change, then reintroduce the intended change one rule at a time while watching `harbor-nginx` readiness. + +> **Warning:** Restoring the old policies removes whatever the change was meant to add. You need a copy of the configuration from before the change (an export, Git, or the previous console values). Re-apply the current copy saved above to roll back. + +```bash +kubectl apply -f .yaml +kubectl get pod -A | grep harbor-nginx +``` + +## Diagnostic Steps + +1. List every policy that can select the failing Pod and its peer, cluster-scoped ones included: + + ```bash + kubectl -n get pod --show-labels + kubectl -n get networkpolicy + kubectl get adminnetworkpolicy + kubectl get baselineadminnetworkpolicy + ``` + +2. Read why a Pod is not Ready; probe or upstream timeouts that began with a policy change point to that change: + + ```bash + kubectl -n describe pod | grep -A10 -iE 'Readiness|Events' + ``` + +3. Test the exact path with the Pod IP and the container port: + + ```bash + kubectl -n exec -- sh -c 'nc -vz -w 3 ' + ``` + +- Ingress to the Service fails and a policy selects the backend: Cause 1. +- Same-namespace Pods cannot ping and a policy has `Egress`: Cause 2. +- Redis Sentinel Pods crash-loop with API server calls on `443` timing out: Cause 3. +- `harbor-nginx` `0/1` right after cluster policy changes: Cause 4. + +If LoadBalancer VIPs (not application Pods) fail after isolation between projects is enabled, see "LoadBalancer VIP unreachable after project isolation or with a MetalLB address outside the node segment on ACP". diff --git a/docs/en/solutions/Application_restore_fails_hangs_or_restored_workloads_do_not_run_on_another_cluster_on_ACP.md b/docs/en/solutions/Application_restore_fails_hangs_or_restored_workloads_do_not_run_on_another_cluster_on_ACP.md new file mode 100644 index 000000000..23a99e8a5 --- /dev/null +++ b/docs/en/solutions/Application_restore_fails_hangs_or_restored_workloads_do_not_run_on_another_cluster_on_ACP.md @@ -0,0 +1,185 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Application restore fails, hangs, or restored workloads do not run on another cluster on ACP + +## Issue + +An application (namespace) backup made with the platform backup and restore feature (Velero, objects `restores.velero.io` in `cpaas-system`) exists, and the restore on Alauda Container Platform fails, hangs, or finishes while the restored namespace does not work on the target: + +- The restore fails with `admission webhook "vconnector.kb.io" denied the request: Connector.connectors.alauda.io/ is invalid: spec.authSecretRef.Namespace: Invalid value: "": secretRef namespace must be empty or equal to connector namespace: `. +- In a disaster-recovery drill, a namespace backed up on one platform is restored to a separate DR platform and: + - restored StatefulSet or Deployment pods stay `Pending` because their `nodeSelector` still names the source host; + - images fail to pull because the two platforms use different Harbor registries; + - PVCs that used `local-path` on the source cannot be used on the DR cluster. +- A running restore task hangs and must be stopped by hand; it stays in progress with no error, even after the target namespace was deleted. +- Before relying on cross-platform restore: must the project match, and does a restore across two global platforms behave differently? + +A restore replays the namespace's own YAML as it was captured. Anything in that YAML that refers to the source environment (node labels, registry address, StorageClass, node-pinned PVs) is restored unchanged, and everything outside the namespace (project association, image pull authorization, load balancer rules, DNS) has to be prepared on the target. + +## Root Cause + +### Cause 1: a restored Connector points at a Secret in another namespace + +The `vconnector.kb.io` admission webhook validates `Connector` (`connectors.alauda.io`) objects and accepts `spec.authSecretRef.namespace` only when it is empty or equal to the Connector's own namespace. When the Connector is restored into a namespace different from the one named in `spec.authSecretRef`, the webhook rejects it and the restore fails. The problem is the namespace relationship between the Connector and its Secret, not the backup content. + +### Cause 2: restored pods are Pending because nodeSelector holds the source cluster's node value + +Restored workloads keep their `nodeSelector` (for example the original host IP). The restore does not rewrite scheduling fields, so on a cluster whose nodes do not carry that label value the pods cannot be scheduled: `0/ nodes are available: node(s) didn't match Pod's node affinity/selector`. + +### Cause 3: ErrImagePull / ImagePullBackOff because the target platform uses a different Harbor + +The restored YAML still names the source registry, and the target namespace has no pull authorization for it. + +### Cause 4: local-path PVCs are unusable on the DR cluster + +`local-path` writes to a directory on the host node. The DR cluster has no identical `local-path` StorageClass, so the restored PVCs cannot be provisioned. Local PV data does not move with the YAML, and a restored local PV keeps the source node in its `nodeAffinity`, so it cannot bind on the DR cluster. StorageClass project assignment is not related to this. + +### Cause 5: project and cross-platform expectations + +Projects must be the same on the source and the DR cluster. A restore between two independent global platforms is a migration of YAML resources: only the namespace's own YAML is restored, and business dependencies (project association, image pull secrets, load balancer rules, DNS resolution) are adapted by hand on the new platform. The target cluster must also be able to read the backup from object storage, for example by binding the same S3 storage on both platforms. + +### Cause 6: a restore task stays in progress and deleting the target namespace does not stop it + +A restore task is a `restores.velero.io` resource. Deleting the target namespace does not stop or terminate it, so the task keeps showing in progress without an error. + +## Resolution + +### Cause 1: a restored Connector points at a Secret in another namespace + +1. Compare the Connector's namespace with its Secret reference in the source: + +```bash +kubectl get connectors.connectors.alauda.io -A +kubectl -n get connectors.connectors.alauda.io -o yaml | grep -A3 authSecretRef +``` + +2. Bring the pair into the shape the webhook accepts: the referenced Secret is in the same namespace as the Connector, and `spec.authSecretRef.namespace` is empty or that same namespace after the restore. Adjust the Secret's namespace and the Connector configuration accordingly, then run the restore again. +3. Check the target: + +```bash +kubectl -n get connectors.connectors.alauda.io -o jsonpath='{.metadata.namespace} {.spec.authSecretRef}{"\n"}' +kubectl -n get secret +``` + +### Cause 2: restored pods are Pending because nodeSelector holds the source cluster's node value + +Either label target nodes with the value the workload asks for, or remove the `nodeSelector`: + +```bash +kubectl -n get statefulset -o jsonpath='{.spec.template.spec.nodeSelector}{"\n"}' +kubectl label node = +``` + +> **Warning:** Removing `nodeSelector` changes the pod template, so the workload rolls its pods and they may land on any node. Precondition: the restored pods are `Pending` because the selector names a source-cluster node, and the object is saved. Rollback: re-apply the saved YAML. + +```bash +kubectl -n get statefulset -o yaml > /tmp/-sts.yaml +kubectl -n patch statefulset --type=json \ + -p '[{"op":"remove","path":"/spec/template/spec/nodeSelector"}]' +``` + +Use `deployment` instead of `statefulset` for a Deployment. A pod bound to a local PV is also limited by the PV's `nodeAffinity` (Cause 4). + +### Cause 3: ErrImagePull / ImagePullBackOff because the target platform uses a different Harbor + +1. Read which registry the restored pod uses and whether it has a pull secret. For pods that pull after retries without a pull secret, this also shows the registry they really used: + +```bash +kubectl -n get pod -o jsonpath='{range .spec.containers[*]}{.image}{"\n"}{end}{.spec.imagePullSecrets}{"\n"}' +kubectl -n describe pod | grep -A5 -iE 'Failed|pull' +``` + +2. Choose one direction: + - The same images exist in the target platform's registry: before the restore, perform the pre-recovery resource creation step described in the platform backup and restore documentation. It replaces the image registry address in the restored YAML. Then restore again. + - Keep one shared Harbor: integrate that Harbor on both platforms (through toolchain management) and assign it to the project, then bring the recovered namespace under management with the import-namespace feature, which creates the registry authorization. + +### Cause 4: local-path PVCs are unusable on the DR cluster + +1. Create a `local-path` StorageClass on the DR cluster identical to the source one: + +```bash +kubectl get storageclass local-path -o yaml > /tmp/local-path-sc.yaml # on the source cluster +kubectl apply -f /tmp/local-path-sc.yaml # on the DR cluster, after removing uid, resourceVersion, creationTimestamp +``` + +2. Plan for empty volumes: the data in the source local PVs is not carried over. +3. For a restored local PV that names a source-cluster node, recreate the PV with the same name and a DR-cluster node in `nodeAffinity`. The API server rejects in-place changes to `spec.nodeAffinity` of an existing PV. The full procedure is in [Restoring local-volume workloads across ACP clusters with Velero](./Restoring_local_volume_workloads_across_ACP_clusters_with_Velero.md). Before deleting any restored PV, set its reclaim policy to `Retain` and save its YAML, so the provisioner does not remove the backing directory: + +```bash +kubectl patch pv -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' +kubectl get pv -o yaml > /tmp/.yaml +``` + +### Cause 5: project and cross-platform expectations + +1. Create the same project on the target platform and put the target namespace in it before restoring. +2. Prepare by hand what the namespace YAML does not carry: image registry access (Cause 3), StorageClasses (Cause 4), load balancer rules, and DNS records. +3. Make sure the target cluster can read the backup from the object storage that holds it. + +### Cause 6: a restore task stays in progress and deleting the target namespace does not stop it + +1. Confirm the task is still in progress and read what it last did: + +```bash +kubectl -n cpaas-system get restores.velero.io -o jsonpath='{.status.phase}{"\n"}' +kubectl -n cpaas-system describe restores.velero.io +kubectl -n cpaas-system logs deploy/velero --tail=500 | grep '' +``` + +> **Warning:** Deleting the `restores.velero.io` object abandons the task and removes its status. It cannot be resumed, and objects it already restored stay in the target namespace. Precondition: `status.phase` stays `InProgress` with no progress in the Velero log, the owner accepts abandoning this restore, and the object is saved. Rollback: none for this task; run a new restore from the same backup, which this delete does not remove. + +2. Stop the task by deleting the `restore` resource: + +```bash +kubectl -n cpaas-system get restores.velero.io -o yaml > /tmp/.yaml +kubectl -n cpaas-system delete restores.velero.io +kubectl -n cpaas-system get restores.velero.io +``` + +3. Clean up by hand what the task already restored. Velero labels restored objects with `velero.io/restore-name`; if the list is empty and the namespace still exists, compare the namespace with the backup by hand. + +```bash +kubectl -n get all,configmap,secret,pvc -l velero.io/restore-name= +``` + +> **Warning:** Deleting the restored objects removes them from the target namespace, and a PVC whose PV reclaim policy is `Delete` loses its volume data. Precondition: the `restores.velero.io` object is gone, and every listed object was created by this partial restore and holds nothing to keep. Rollback: run a new restore. + +```bash +kubectl -n delete +``` + +## Diagnostic Steps + +1. On the cluster that runs the restore, read the status and errors: + +```bash +kubectl -n cpaas-system get restores.velero.io +kubectl -n cpaas-system describe restores.velero.io +kubectl -n cpaas-system logs deploy/velero --tail=500 | grep -iE 'admission webhook|denied|' +``` + +2. For a finished restore whose workloads do not run, read the events in the restored namespace: + +```bash +kubectl -n get pods,pvc +kubectl -n get events --sort-by=.lastTimestamp | tail -40 +``` + +3. Match the signature: + - `vconnector.kb.io` and `secretRef namespace must be empty or equal to connector namespace`: Cause 1. + - `didn't match Pod's node affinity/selector` and a `nodeSelector` with a source node value: Cause 2. + - `ErrImagePull` / `ImagePullBackOff` on a source-registry address: Cause 3. + - PVC `Pending` on `local-path`, or `volume node affinity conflict`: Cause 4. + - Project or cross-platform questions: Cause 5. + - `status.phase` stays `InProgress` with no error, including after the target namespace was deleted: Cause 6. + - `status.phase` is `Completed` without errors but ConfigMaps or other objects are missing: see [ConfigMaps missing after a namespace restore that completed without errors on ACP](./ConfigMaps_missing_after_a_namespace_restore_that_completed_without_errors_on_ACP.md). + +## Related Information + +- [What platform backup, application snapshots, and volume snapshots cover on ACP](./What_platform_backup_application_snapshots_and_volume_snapshots_cover_on_ACP.md) diff --git a/docs/en/solutions/Application_updates_are_slow_status_stays_Failed_or_Processing_or_delete_hangs_because_of_captain_on_ACP.md b/docs/en/solutions/Application_updates_are_slow_status_stays_Failed_or_Processing_or_delete_hangs_because_of_captain_on_ACP.md new file mode 100644 index 000000000..f0421382e --- /dev/null +++ b/docs/en/solutions/Application_updates_are_slow_status_stays_Failed_or_Processing_or_delete_hangs_because_of_captain_on_ACP.md @@ -0,0 +1,329 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.16" + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Application updates are slow, status stays Failed or Processing, or delete hangs because of captain on ACP + +## Issue + +Native applications and Helm chart applications are reconciled by `captain` (`captain-controller-manager` and `captain-chartmuseum` in `cpaas-system`) through `HelmRequest` objects. When it is blocked, the console is open but: + +- After a native application's template is updated, the Pods start rolling only after a long time (about 10 minutes, or never). +- A native application shows `失败(0/1)` or `失败(0/0)` (Failed) while its containers are `Running`. +- A pipeline-published native application stays on `处理中(1/1)` (Processing). +- Deleting a Helm application stays on `删除中` (Deleting) with `处理中(0/0)` for hours. +- `captain` crash-loops and application updates fail. +- A Helm application stays `删除中` (Deleting) while its workloads are still `Running`, so a redeploy cannot start. +- The native application page shows no compute components (`0/0`) while its Deployments are `Running`. +- An application exported to a chart and imported again stays processing (`0/0`) with no error and no event. +- A manual edit of an object rendered by the chart disappears at the next sync. +- The application history shows a new revision that nobody created. + +The `captain` log often contains `Unauthorized, requeuing`, `failed to list *v1.HelmRequest: Unauthorized`, or a `nil pointer dereference` panic in the `chartrepo` controller. + +## Root Cause + +### Cause 1: one HelmRequest worker per cluster is blocked by failing chart applications + +`captain` has one worker per cluster for `HelmRequest`. Failed HelmRequests keep retrying with back-off and occupy that worker, so new native application updates wait in the queue behind them. The `manager` container of `captain-controller-manager` repeats, for many different HelmRequests: + +```text +helmrequest.go:31] Start sync helmrequest: / +clusterwatch.go:262] error syncing '/': Unauthorized, requeuing +``` + +Chart applications left abnormal from earlier failures cause this: updates that normally take about 3 minutes take about 10 minutes or never take effect. Restarting `captain-controller-manager` releases the stuck applications, but later updates stay slow until the failing chart applications are fixed or cleaned up. + +### Cause 2: the native application status is out of sync with its workloads + +The native application status combines `assemblyPhase` with whether the workload is running. `assemblyPhase` changes only when the application is created or updated through the console or the API. When several people trigger updates, those updates can conflict and `assemblyPhase` falls out of sync with the running Pods, which shows as `失败` (Failed) while the services work, sometimes with a tooltip from an old failed update. Triggering an update of the application from the console (**Update**, then **Confirm** with no change) restores the status. When the workloads are healthy, this status can be ignored. + +### Cause 3: a pipeline-published application keeps spec.assemblyPhase Pending + +The Application stays `处理中(1/1)` while its Deployment is `Running` and its events are clean, and a hand-created application of the same shape is normal. The Application (`app.k8s.io/v1beta1`) has `spec.assemblyPhase: Pending`. Setting it to `Succeeded` is the recommended fix. When that edit has no effect and the captain controller logs `failed to list *v1.HelmRequest: Unauthorized` (and ChartRepo objects log `not ready for resync`), the controller cannot list or watch HelmRequests, and restarting it lets the release finish. + +### Cause 4: captain cannot authenticate to the chart repository + +The application stays `失败(0/1)` while its workload is `Running`. The captain log shows the HelmRequest failing to fetch the chart: + +```text +get chart {"name": "/", "version": ""} +"reason": "FailedSync", "message": "Unauthorized" +``` + +The ChartRepo keeps logging `reconcile start`, and under **App Store Management** > **Template Repositories** the repository shows `同步中` (Syncing). The repository rejected the configured credentials. + +### Cause 5: captain does not reload its ServiceAccount token + +A chart that changes RBAC fails an update for lack of permission, and a later update as administrator still shows `Failed`. When `captain` reloads its ServiceAccount token, the reload does not work, and Helm updates start failing after some time. This is a product defect fixed in later versions; restarting `captain` is the workaround. + +### Cause 6: a finalizer blocks the delete of a Helm application + +The application list shows `删除中` with `处理中(0/0)`, the workloads are gone (`kubectl get deployment,pod -n ` returns `No resources found`), and the Application is in phase `Pending` with no events. `captain-controller-manager` repeats `error syncing ... Unauthorized, requeuing` and `Failed to watch *v1.HelmRequest: failed to list *v1.HelmRequest: Unauthorized`. Restarting it stops those errors but does not finish the delete; the finalizer on the Application is still set. + +### Cause 7: captain crash-loops while sorting a chart repository index + +The previous `captain` log shows, while the `chartrepo` controller reconciles a ChartRepo in `cpaas-system`: + +```text +Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference +"controller": "chartrepo", "controllerGroup": "app.alauda.io", "controllerKind": "ChartRepo" +``` + +The stack runs through `repo.ChartVersions.Less`, `IndexFile.SortEntries`, `loadIndex`, and `(*ChartRepoReconciler).syncCharts`: `captain` panics while sorting that repository's index entries. Removing the repository from the platform returns `captain` to `Running`. + +### Cause 8: an expired captain certificate leaves a delete stuck while the workloads still run + +The Application keeps phase `Succeeded` with `metadata.deletionTimestamp` set, and the matching HelmRequest is `phase: Synced` with a `deletionTimestamp`. The workloads are still `Running` because the uninstall never ran. `captain` logs `failed to list *v1.HelmRequest: Unauthorized` and `failed to watch *v1.HelmRequest: failed to list *v1.HelmRequest: Unauthorized`. The certificate `captain` uses expired and was not rotated. This is a product defect fixed in ACP 4.2.5; a `captain` that still holds the expired certificate, including on 4.2.5, needs a restart of `captain-controller-manager` and `metis`, after which the delete finishes. Unlike Cause 6, the workloads still exist and the finalizer is not removed by hand. + +### Cause 9: no compute components because metis cannot impersonate groups + +The console asks `metis` (Deployment `metis` in `cpaas-system`) to draw native-application components. When the ClusterRole bound to `system:serviceaccount:cpaas-system:metis` allows impersonating `users` but not `groups`, the page shows `0/0` while the Deployments are `Running`, and the application has an event: + +```text +Failed: groups "" is forbidden: User "system:serviceaccount:cpaas-system:metis" cannot impersonate resource "groups" in API group "" at the cluster scope +``` + +This gap exists on ACP 4.2.4. Adding the missing impersonation rule is the fix. + +### Cause 10: a re-imported chart application stays processing because the Harbor registry is unhealthy + +After an application is exported to a chart and imported again, it stays processing (`0/0`), `spec.assemblyPhase` stays `pending`, and there are no events. The ChartRepo can still be `Synced` while the Harbor `registry` component is unhealthy: the status sync of the chart-backed application times out talking to that registry. Repairing the Harbor registry ends the processing state; setting `spec.assemblyPhase` by hand (Cause 3) is not the repair. + +### Cause 11: manual edits are reverted when the HelmRequest syncs + +Objects changed with `kubectl edit` or in the console are put back to the rendered chart at the next HelmRequest sync. Persistent changes belong in HelmRequest `spec.values` or the Application annotation `app.cpaas.io/chart.values`. + +### Cause 12: a captain restart re-applies applications and adds a revision nobody made + +On ACP 3.12, when `captain` restarts, its check of whether a HelmRequest changed is faulty: it resyncs and re-applies the application's resources, which writes a new revision to the application history without any user action. The Kubernetes audit log shows a `patch` of the application's objects (for example a Service) with `fieldManager=manager`, user agent `Go-http-client/2.0`, by a platform service account such as `system:serviceaccount:kube-system:admin`, setting `app.kubernetes.io/managed-by: Helm`; the objects carry `meta.helm.sh/release-name`. This is a product defect fixed in ACP 3.15 and later. + +## Resolution + +### Cause 1: one HelmRequest worker per cluster is blocked by failing chart applications + +1. Confirm that failing HelmRequests occupy the worker, and list abnormal applications: + + ```bash + kubectl logs -n cpaas-system deploy/captain-controller-manager -c manager --tail=500 | grep -E 'Start sync helmrequest|Unauthorized, requeuing' | sort | uniq -c | sort -rn | head + kubectl get helmrequest -A + kubectl get application -A | grep -v Running + ``` + +2. Fix or clean up the chart applications whose HelmRequests keep failing. This is the lasting step. + +3. To release updates that are already stuck, restart `captain-controller-manager`. + + > **Warning:** HelmRequest processing pauses until the new Pod is `Ready`. A restart alone leaves later updates slow while failing HelmRequests remain. The Pod is recreated from the same spec. + + ```bash + kubectl delete pod -n cpaas-system + ``` + +4. Once the failing applications are fixed or removed, native updates take effect within a few minutes. Also check the workload's own rollout strategy, for example `maxUnavailable: 0` with a readiness probe that does not pass, before attributing a slow rollout to `captain`. + +### Cause 2: the native application status is out of sync with its workloads + +Confirm that the containers are `Running`, then open **Update** on the application and click **Confirm** without changes. If the status stays failed, read the `captain` log before repeating. + +### Cause 3: a pipeline-published application keeps spec.assemblyPhase Pending + +1. Export the Application and set the field: + + > **Warning:** Setting `spec.assemblyPhase` by hand tells the console that the application finished assembling, and a wrong value can hide a real failure. Do this only when the workloads are `Running` and the events are clean. To roll back, apply the export. + + ```bash + kubectl get application -n -o yaml > "$HOME/-application.yaml" + kubectl patch application -n --type=merge -p '{"spec":{"assemblyPhase":"Succeeded"}}' + ``` + +2. If the edit has no effect, check the captain controller log and restart it: + + ```bash + kubectl get pod -n cpaas-system | grep captain + kubectl logs -n cpaas-system --tail=200 | grep -E 'failed to list \*v1.HelmRequest|not ready for resync' + ``` + + > **Warning:** Application releases pause until the new Pod is `Ready`. The Pod is recreated from the same spec. + + ```bash + kubectl delete pod -n cpaas-system + ``` + +### Cause 4: captain cannot authenticate to the chart repository + +1. Confirm the chart fetch failure: + + ```bash + kubectl logs -n cpaas-system deploy/captain-controller-manager --tail=300 | grep -E 'get chart|FailedSync|Unauthorized' + ``` + +2. Open **App Store Management** > **Template Repositories**, select the repository, and correct the user name and password in its update dialog. + +3. Restart the captain Pods. + + > **Warning:** Chart serving and HelmRequest processing pause until the Pods are `Ready`. Do this only after the credentials have been corrected. + + ```bash + kubectl delete pod -n cpaas-system + ``` + +4. The repository leaves `同步中` and the application leaves `失败`. If `Unauthorized` remains with correct credentials, test the repository login from outside the platform. + +### Cause 5: captain does not reload its ServiceAccount token + +> **Warning:** Helm updates pause until the new Pod is `Ready`. The Pod is recreated from the same spec. + +```bash +kubectl get pods -n cpaas-system | grep captain +kubectl delete pod -n cpaas-system +``` + +Retry the Helm update after the new Pod is `Running`. Upgrade to a version that contains the fix. + +### Cause 6: a finalizer blocks the delete of a Helm application + +1. Export the Application and check `captain`: + + ```bash + kubectl get application -n -o yaml > "$HOME/-application.yaml" + kubectl logs -n cpaas-system deploy/captain-controller-manager --tail=200 | grep -E 'Unauthorized|error syncing' + ``` + +2. Restart `captain-controller-manager`, then remove the `finalizers` field from the Application. + + > **Warning:** Removing finalizers deletes the object without controller cleanup, and orphaned releases can remain. Do this only when the delete is stuck, the workloads are already gone, and a restart of `captain-controller-manager` did not finish it. This cannot be rolled back. If the finalizer is `chart.app.alauda.io` and the matching `helmrequests.app.alauda.io` object does not exist, this cause does not apply. + + ```bash + kubectl delete pod -n cpaas-system + kubectl edit application -n + ``` + +### Cause 7: captain crash-loops while sorting a chart repository index + +1. Find the repository named in the panic: + + ```bash + kubectl get pods -n cpaas-system | grep captain + kubectl logs -n cpaas-system --previous --tail=300 | grep -E 'Observed a panic|ChartRepo|SortEntries|loadIndex' + kubectl get chartrepo -n cpaas-system + ``` + +2. Save it and remove it from the platform chart repository list. + + > **Warning:** Applications that pull charts from this repository can no longer update. Do this only when `captain` crash-loops with the `chartrepo` nil pointer panic and the log names this ChartRepo. Applying the export restores it, and brings the crash loop back until its index is fixed. + + ```bash + kubectl get chartrepo -n cpaas-system -o yaml > "$HOME/-chartrepo.yaml" + ``` + +3. `captain` stays `Running`. + +### Cause 8: an expired captain certificate leaves a delete stuck while the workloads still run + +1. Confirm both the `deletionTimestamp` and the `Unauthorized` list/watch error (Diagnostic Steps), and record the images: + + ```bash + kubectl -n cpaas-system get deploy captain-controller-manager metis \ + -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.template.spec.containers[0].image}{"\n"}{end}' + ``` + +2. Restart both components. Below 4.2.5, also upgrade to 4.2.5 or later to get the fix. + + > **Warning:** Only when the `Unauthorized` error and a `deletionTimestamp` are both present. Impact: HelmRequest reconciliation and native-application status pause until both Deployments are `Ready`. The restart reloads access state and does not delete the Application. Rollback: `kubectl -n cpaas-system rollout undo deployment/` for each. + + ```bash + kubectl -n cpaas-system rollout restart deployment/captain-controller-manager deployment/metis + kubectl -n cpaas-system rollout status deployment/captain-controller-manager + kubectl -n cpaas-system rollout status deployment/metis + ``` + +3. The uninstall runs and the Application leaves the console. + +### Cause 9: no compute components because metis cannot impersonate groups + +1. Check the permission and find the ClusterRole bound to metis. `no` plus the event text confirms this cause: + + ```bash + kubectl auth can-i impersonate groups --as=system:serviceaccount:cpaas-system:metis + kubectl get clusterrolebinding -o jsonpath='{range .items[*]}{.roleRef.name}{"\t"}{range .subjects[*]}{.namespace}{"/"}{.name}{" "}{end}{"\n"}{end}' | grep cpaas-system/metis + ``` + +2. Add this rule to that ClusterRole: + + > **Warning:** The rule lets metis act as the groups named in user tokens. Add only this rule, to the ClusterRole already bound to metis; do not grant `impersonate` on `*`. Precondition: `kubectl get clusterrole -o yaml > metis-role.yaml`. Rollback: `kubectl apply -f metis-role.yaml`. + + ```yaml + - apiGroups: [""] + resources: ["groups"] + verbs: ["impersonate"] + ``` + +3. Run the `auth can-i` command again until it prints `yes`, then reopen the application; the Running Deployments are listed as compute components. + +### Cause 10: a re-imported chart application stays processing because the Harbor registry is unhealthy + +Repair the Harbor `registry` component until its Pods are `Ready` and Harbor reports the registry healthy, then retry the import or the HelmRequest sync. A ChartRepo in `Synced` is not evidence that the registry is healthy. + +```bash +kubectl get application -n -o jsonpath='{.spec.assemblyPhase}{"\n"}' +kubectl get pods -A | grep harbor-registry +``` + +### Cause 11: manual edits are reverted when the HelmRequest syncs + +Put the change in HelmRequest `spec.values` (or the `app.cpaas.io/chart.values` annotation of the Application) so the next sync renders it. Repeating the edit on the Deployment does not stick. + +```bash +kubectl get helmrequest -n -o jsonpath='{.spec.values}{"\n"}' +kubectl edit helmrequest -n +``` + +### Cause 12: a captain restart re-applies applications and adds a revision nobody made + +1. Compare the time of the unexplained revision with the start time of the `captain` Pod: + + ```bash + kubectl get pods -n cpaas-system | grep captain + kubectl get pod -n cpaas-system -o jsonpath='{.status.startTime}{" "}{.status.containerStatuses[*].lastState}{"\n"}' + ``` + +2. When they match and the audit log shows the `patch` described in Root Cause, the revision re-applied the application's own chart and needs no rollback. Hand edits made outside the chart are lost (Cause 11). +3. Upgrade to ACP 3.15 or later to stop restarts from creating revisions. + +## Diagnostic Steps + +```bash +kubectl get application -n -o yaml +kubectl get pods -n cpaas-system | grep captain +kubectl get helmrequest -A +kubectl logs -n cpaas-system deploy/captain-controller-manager --tail=500 \ + | grep -E 'Unauthorized|FailedSync|Observed a panic' | sort | uniq -c | sort -rn | head +``` + +```text +native updates slow, many HelmRequests Unauthorized, requeuing, abnormal chart apps -> Cause 1 +失败 status, containers Running, several people updating -> Cause 2 +pipeline application 处理中, spec.assemblyPhase Pending -> Cause 3 +失败(0/1), get chart FailedSync Unauthorized, repository 同步中 -> Cause 4 +Helm update stays Failed after a permission error, admin retry fails -> Cause 5 +Helm delete stuck, workloads gone, error syncing Unauthorized -> Cause 6 +captain crash-loop, chartrepo panic in SortEntries -> Cause 7 +Deleting, workloads still Running, deletionTimestamp, list HelmRequest Unauthorized -> Cause 8 +0/0 components, event: metis cannot impersonate resource "groups" -> Cause 9 +re-imported chart app processing, assemblyPhase pending, Harbor registry down -> Cause 10 +objects revert after a sync -> Cause 11 +unexplained revision at a captain restart, audit patch by fieldManager=manager -> Cause 12 +``` diff --git a/docs/en/solutions/Archive_Elasticsearch_logs_to_NFS_logarchive_mount_log_center_Kafka_address_and_archive_limits_on_ACP.md b/docs/en/solutions/Archive_Elasticsearch_logs_to_NFS_logarchive_mount_log_center_Kafka_address_and_archive_limits_on_ACP.md new file mode 100644 index 000000000..159a0920d --- /dev/null +++ b/docs/en/solutions/Archive_Elasticsearch_logs_to_NFS_logarchive_mount_log_center_Kafka_address_and_archive_limits_on_ACP.md @@ -0,0 +1,137 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.14.x + - 3.16.x + - 4.2.x + - 4.3.x +--- + +# Archive Elasticsearch logs to NFS: logarchive mount, log center Kafka address, and archive limits on ACP + +## Issue + +Platform logs stored in Elasticsearch should also be kept on an external NAS or NFS share, and the archive deployment or its result raises questions: + +- The archive manifest has `hostPath` `/cpaas/data/logarchive`. Must the NFS export be mounted on the node first, or should that path be replaced with the NFS server's directory? +- Which Kafka goes into the archive form field `Kafka 服务地址` (Kafka service address): the Kafka that came with the log storage, or a new Kafka from the Data Services catalog? +- Local Elasticsearch should keep about 7 days and older logs should move to the NAS. How are logs older than 7 days read afterwards? +- The archive copies the logs of every project and uses too much space; can it archive only selected projects? +- Logs must be kept 180 days, but the platform allows at most 30 days, and a 180-day lifecycle set in Kibana is reset. +- With ClickHouse log storage on 4.2 or 4.3, can logs be archived or cold-backed up to NFS or NAS? + +The archive is the platform documentation's log archiving solution: "Transfer to external NFS" on 3.14, and "How to archive logs to third-party storage" (Observability, Logs, Practical Guides) on 4.x, which writes to an external NFS or S3 through a log sync workload created with `kubectl`. What a running archive writes (scopes, S3 layout, file retention, files that cannot be deleted) is in [Log archive output: platform logs with workload logs, flat S3 objects, file retention, or undeletable .nfs files on ACP](Log_archive_output_platform_logs_with_workload_logs_flat_S3_objects_file_retention_or_undeletable_.nfs_files_on_ACP.md). + +## Root Cause + +### Cause 1: /cpaas/data/logarchive is the host directory, and the NFS export must be mounted on it before the pod runs there + +The log sync workload mounts a `hostPath` volume `data` with `path: /cpaas/data/logarchive` and `type: DirectoryOrCreate`. The path stays as it is in the manifest; it is not rewritten to the NFS server's export. Because `DirectoryOrCreate` creates a local directory when nothing is mounted there, the NFS export has to be mounted on `/cpaas/data/logarchive` on the node before the pod starts on that node, and the pod has to be pinned to that node. Otherwise the archive writes to the node's local disk. The working sequence: label the node, schedule the archive workload onto it, create the directory, mount NFS on it. Whether NFS v3 needs `nolock` was not part of the verified setup. + +### Cause 2: the Kafka service address is the log center Kafka, not a new Data Services Kafka + +The log collection plugins write the collected logs into the log center Kafka, and the archive component reads them from there. The archive form's `Kafka 服务地址` (Kafka service address) is therefore the platform's built-in log center Kafka. A Kafka instance created from the Data Services catalog receives no logs, so an archive pointed at it stays empty. + +### Cause 3: the archive is a real-time copy, not a move after N days + +The archive writes a copy of each new log to the NFS share as it is produced. It does not move logs to the NAS after they have been kept locally for some days, and no supported option tiers Elasticsearch logs by age. Log queries within the configured retention are not affected. Logs older than the retention are read from the archive path, not through the log query page; archived files can be read directly after any compressed file is decompressed. + +### Cause 4: the archive is cluster-wide and cannot be limited to selected projects + +The archive works at cluster level and archives the logs of every project of the cluster (verified on 3.16.2). Archiving only some projects is not supported, so the archive storage has to be sized for all projects. + +### Cause 5: logs must be kept longer than the platform retention allows + +The platform caps log retention (30 days for a project policy and in 4.x consoles), and a lifecycle edited in Kibana is overwritten by the platform policy. On 3.16 the Elasticsearch retention can be set to 180 days, but adding disks to keep 180 days in place is not recommended, and later versions cap log retention at 30 days. For Elasticsearch, the long-term copy is the log archive to NFS. For ClickHouse log storage in 4.x, the logging plugin allows at most 30 days of log retention, events and audit records have their own, longer maximum (read it on the plugin form), and no supported way to keep logs longer is known. + +### Cause 6: ClickHouse log storage on 4.2 and 4.3 has no log archive to NFS or NAS + +The log archiving in the documentation applies to Elasticsearch storage. ClickHouse log storage on 4.2 and 4.3 has no log archive, so the archive guide does not apply to it and backing up ClickHouse logs to a NAS is not supported. What ClickHouse logging offers instead is S3 integration with hot/cold separation, available from the 4.3 logging component. A 4.2 platform gets it by upgrading the platform to 4.2.5 and then the logging component to 4.3. Hot/cold separation tiers data inside the retention the plugin allows; it is not a way past the 30-day log cap. + +## Resolution + +### Cause 1: /cpaas/data/logarchive is the host directory, and the NFS export must be mounted on it before the pod runs there + +1. Read the two images the documented manifest needs, on the global cluster: + + ```bash + # alpine image + kubectl get daemonset nevermore -n cpaas-system -o jsonpath='{.spec.template.spec.initContainers[0].image}' + # log sync image: the 3.14 document reads it from lanaya, the 4.x guide from razor + kubectl get deployment lanaya -n cpaas-system -o jsonpath='{.spec.template.spec.containers[0].image}' + kubectl get deployment razor -n cpaas-system -o jsonpath='{.spec.template.spec.containers[0].image}' + ``` + +2. On the node that will run the archive, create the directory and mount the NFS export on it. + + > **Warning:** Mounting NFS on `/cpaas/data/logarchive` hides any files already in that directory while the mount is in place. Precondition: the directory is empty or its contents were copied aside, and `:` is the archive target. Rollback: `umount /cpaas/data/logarchive`. Persist the mount the way this node persists its other NFS mounts, so a reboot does not leave the archive writing to local disk. + + ```bash + mkdir -p /cpaas/data/logarchive + mount -t nfs : /cpaas/data/logarchive + df -hT /cpaas/data/logarchive + ``` + + `df` must show an `nfs` type, not the root disk. + +3. Pin the archive workload to that node, either with a node label that the workload's node selector uses or with the node name if the manifest takes one: + + ```bash + kubectl label node =true + ``` + +4. Apply the manifest from the documentation, keeping `/cpaas/data/logarchive` unchanged. + +### Cause 2: the Kafka service address is the log center Kafka, not a new Data Services Kafka + +Read the log center Kafka service and use the broker address the log pipeline already produces to: + +```bash +kubectl -n cpaas-system get svc | grep -i kafka +kubectl -n cpaas-system get pods | grep -i kafka +``` + +### Cause 3: the archive is a real-time copy, not a move after N days + +Set the local retention to the number of days that must stay searchable, and deploy the archive as in Causes 1 and 2 so every new log is also copied. Read older logs on the share, for example: + +```bash +ls -lt /cpaas/data/logarchive | head +find /cpaas/data/logarchive -name '*.gz' -mtime +7 | head +zcat .gz | less +``` + +List the directory first; use the decompressor that matches the files if they are not gzip. + +### Cause 4: the archive is cluster-wide and cannot be limited to selected projects + +No configuration narrows it. Size the archive storage for the logs of all projects of the cluster. + +### Cause 5: logs must be kept longer than the platform retention allows + +On Elasticsearch, keep the platform retention within its cap and deploy the archive to NFS for the long-term copy; do not rely on Kibana lifecycle edits. On ClickHouse, 30 days is the maximum log retention in the 4.x logging plugin; a longer retention has to be raised as a feature request. + +### Cause 6: ClickHouse log storage on 4.2 and 4.3 has no log archive to NFS or NAS + +Check the documentation of the installed logging component; offer ClickHouse archiving only if that version lists it. To move retained ClickHouse logs off local disk, use S3 hot/cold storage from the 4.3 logging component. On 4.2: + +1. Upgrade the platform to 4.2.5. +2. Upgrade the logging component to 4.3. +3. Configure S3 as described in "How to Use S3 Storage with ClickHouse" in the logging documentation. + +## Diagnostic Steps + +1. On the archive node, confirm the mount and the pod placement: + + ```bash + findmnt /cpaas/data/logarchive + kubectl get pods -A -o wide | grep -iE 'log-export|archive' + ``` + + `findmnt` must show `nfs`, and the archive pod's node must be the node with the mount. A local file system type means the archive writes to local disk (Cause 1). + +2. If the archive stays empty while Elasticsearch has recent logs, compare the configured Kafka address with the brokers listed in Cause 2. The archive only receives what the collectors write to the log center Kafka. +3. Which log store runs: `kubectl get pods -A | grep -iE 'clickhouse|cpaas-elasticsearch'`. A ClickHouse store asking for archive or NAS backup is Cause 6, and a request to keep logs longer than 30 days is Cause 5. For Elasticsearch, tiering by age is Cause 3 and selected projects is Cause 4. diff --git a/docs/en/solutions/Argo_CD_Application_sync_not_permitted_repository_Not_Found_InvalidSpecError_or_empty_GitOps_project_list_on_ACP.md b/docs/en/solutions/Argo_CD_Application_sync_not_permitted_repository_Not_Found_InvalidSpecError_or_empty_GitOps_project_list_on_ACP.md new file mode 100644 index 000000000..9d81ff7f3 --- /dev/null +++ b/docs/en/solutions/Argo_CD_Application_sync_not_permitted_repository_Not_Found_InvalidSpecError_or_empty_GitOps_project_list_on_ACP.md @@ -0,0 +1,148 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Argo CD Application sync not permitted, repository Not Found, InvalidSpecError, or empty GitOps project list on ACP + +## Issue + +Argo CD on ACP is reconciled by `argocd-operator` from an `ArgoCD` resource (`argoproj.io/v1beta1`), and the GitOps pages of the platform work on top of it. One of these is reported: + +- An Application fails with `SyncError Failed sync attempt to : one or more synchronization tasks are not valid resource : is not permitted in project `, for example `resource operator.alauda.io:ResourceSyncer is not permitted in project `. Adding a cluster resource allow list in the Argo CD UI does not stick (the project page then shows "The cluster resource allow list is empty"), and `kubectl edit appproject -n argocd` reverts after saving. +- On **GitOps Applications > Create** with the source **Platform Integration**, the required **Integration Project Name** dropdown has nothing to select, and the page reports an error. +- A repository shows CONNECTION STATUS `Failed`, and removing it returns `Unable to disconnect repository: Not Found`, while every Argo CD and GitOps Pod is `Running`. +- An Application applied as YAML does not appear in Argo CD. Recreating it in the UI fails with `Unable to create application: application spec for is invalid: InvalidSpecError: application repo /helm-charts is not permitted in project ''`. + +Related articles: "Argo CD components fail: Dex image pull when air-gapped, Redis timeouts in an ambient mesh, or dev shm full on ACP". + +## Root Cause + +### Cause 1: the AppProject is synchronized from a platform project and its cluster resource allow list is reset + +An AppProject with the same name as a platform project is synchronized automatically from the platform project by gitops-controller. It carries the label `cpaas.io/project: ` and the annotation `gitops.app.cpaas.io/last-applied-configuration`. On every reconcile, the gitops-controller log prints `Reconcile project "/"`, "The sourceRepos and destinations after merged", and `"Updating project" project=""`, and applies a patch that contains `"spec":{"clusterResourceWhitelist":null}`. UI saves (the `PUT .../api/v1/projects/` request returns `200 OK`), `kubectl edit`, and `kubectl patch` all revert. This is by design: it prevents a project from creating cluster-scoped resources and escalating privileges. A platform-synchronized AppProject does not support changes to its cluster resource allow list. Projects created by hand in Argo CD accept changes, and the `default` AppProject allows group `'*'` kind `'*'`, destinations `'*'`, and sourceRepos `'*'`. + +### Cause 2: the API component behind the GitOps pages is not running + +The GitOps application creation form loads, but the **Integration Project Name** list stays empty because the API component behind the GitOps pages has not started, for example because of insufficient resources. Once its resources are adjusted, it starts and the page works again. + +### Cause 3: the business cluster that the repository targets has no Argo CD CRDs + +All GitOps components on the cluster that runs Argo CD are `Running` (`argocd-gitops-application-controller`, `argocd-gitops-repo-server`, `argocd-gitops-server`, `argocd-gitops-redis-ha-*`, `gitops-api`, `gitops-controller`, `argocd-operator-controller-manager`). The business cluster that the repository and its applications target has no Argo CD CRD, because the Argo CD operator is not installed there. Installing the Argo CD operator on the business cluster creates the CRDs. + +### Cause 4: the Application `repoURL` lacks the `oci://` prefix that the project uses + +The Application exists in the cluster, but Argo CD rejects its spec, so it does not appear in the UI. The AppProject lists the Helm chart repository as `oci:///helm-charts`, while the Application uses `repoURL: /helm-charts` without the scheme. Because the address in the Application differs from the one registered for the project, Argo CD reports the repository as not permitted, even when the project's **SOURCE REPOSITORIES** is `*`. + +## Resolution + +### Cause 1: the AppProject is synchronized from a platform project and its cluster resource allow list is reset + +1. Create a project by hand in the Argo CD UI, with a name that is not a platform project name, and add to its **CLUSTER RESOURCE ALLOW LIST** only the cluster-scoped kinds that the Application needs (for the example above: group `operator.alauda.io`, kind `ResourceSyncer`). The allow list of a manually created project is saved and kept. + +2. Move the Application to that project. Moving it to the `default` AppProject also makes the sync work, as a quick workaround. + + > **Warning:** Moving an Application to another AppProject changes which destinations, source repositories, and resource kinds it may manage; `default` is permissive. Precondition: confirm that the target project allows only what this Application needs. Rollback: patch `spec.project` back to the original value. + + ```bash + kubectl -n argocd patch applications.argoproj.io --type=merge \ + -p='{"spec":{"project":""}}' + ``` + +### Cause 2: the API component behind the GitOps pages is not running + +1. Find the Pod and the reason it does not run (Diagnostic Steps, step 3). + +2. If it is `Pending` with `Insufficient cpu` or `Insufficient memory`, free capacity on the nodes or lower its requests; if it is `OOMKilled`, raise its memory limit. Change the resources in the object that owns the workload. Direct edits to Deployments and StatefulSets that the operator manages are reverted. For the Argo CD components, the upstream `ArgoCD` resource has a `resources` field per component (for example `spec.controller.resources`, `spec.repo.resources`, `spec.server.resources`); for the operator itself, see "Configure Argo CD Operator container resources via its CSV on ACP". + + > **Warning:** Changing the resources rolls the component's Pods; the GitOps pages are unavailable until the new Pod is Ready. Precondition: only when the API Pod is not Running because of resources and the nodes have capacity for the new requests. Record the current values. Rollback: set the previous values again. + + ```bash + kubectl -n edit argocd + ``` + +3. The component starts, and the **Integration Project Name** dropdown is populated again. + +### Cause 3: the business cluster that the repository targets has no Argo CD CRDs + +On the business cluster that the applications deploy to, install the Argo CD operator from OperatorHub, as on the cluster that runs Argo CD. Then confirm that the `argoproj.io` CRDs exist (Diagnostic Steps, step 4) and check the repository connection again in the Argo CD UI. + +### Cause 4: the Application `repoURL` lacks the `oci://` prefix that the project uses + +Set `spec.source.repoURL` of the Application to exactly the address registered in the project, including `oci://`: + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: + namespace: +spec: + project: + source: + repoURL: oci:///helm-charts + chart: + targetRevision: + destination: + name: + namespace: +``` + +```bash +kubectl apply -f .yaml +kubectl -n get applications.argoproj.io -o jsonpath='{.status.conditions}{"\n"}' +``` + +The `InvalidSpecError` condition is gone and the Application appears in the UI. + +## Diagnostic Steps + +1. Identify the instance and the component state: + + ```bash + kubectl get argocd -A + kubectl -n get pods + ``` + +2. For a sync that is not permitted, check whether the AppProject is platform-synchronized and whether gitops-controller resets it: + + ```bash + kubectl -n argocd get appproject -o jsonpath='{.spec.clusterResourceWhitelist}{"\n"}{.metadata.labels.cpaas\.io/project}{"\n"}' + kubectl -n argocd get appproject -o yaml | grep -c 'gitops.app.cpaas.io/last-applied-configuration' + kubectl get pods -A | grep gitops-controller + kubectl -n logs --tail=500 | grep -E 'Reconcile project|Updating project|clusterResourceWhitelist' + ``` + + A project named after a platform project, with the `cpaas.io/project` label and the `last-applied-configuration` annotation, an empty whitelist that returns after each edit, and `"Updating project"` lines with `"clusterResourceWhitelist":null`, is Cause 1. + +3. For an empty **Integration Project Name** list, find the API Pod and why it is not running: + + ```bash + kubectl get pods -A | grep -E 'argocd-api|gitops-api' + kubectl -n describe pod | grep -iE 'insufficient|oomkilled|reason' + ``` + + A `Pending` Pod with `Insufficient cpu` or `Insufficient memory`, or a container that ends `OOMKilled`, is Cause 2. + +4. For `Unable to disconnect repository: Not Found`, check the Argo CD CRDs on the business cluster that the applications target: + + ```bash + kubectl get crd | grep argoproj.io + ``` + + No output on the business cluster is Cause 3. + +5. For an Application that exists but does not appear in the UI, compare its `repoURL` with the project and the registered repositories: + + ```bash + kubectl -n get applications.argoproj.io -o jsonpath='{.spec.source.repoURL}{"\n"}{.status.conditions}{"\n"}' + kubectl -n get appproject -o jsonpath='{.spec.sourceRepos}{"\n"}' + kubectl -n get secret -l argocd.argoproj.io/secret-type=repository -o jsonpath='{range .items[*]}{.data.url}{"\n"}{end}' | while read u; do echo "$u" | base64 -d; echo; done + ``` + + `InvalidSpecError ... is not permitted in project` with a `repoURL` that differs from the registered repository only by the `oci://` prefix is Cause 4. diff --git a/docs/en/solutions/Argo_CD_components_fail_Dex_image_pull_when_air-gapped_Redis_timeouts_in_an_ambient_mesh_or_dev_shm_full_on_ACP.md b/docs/en/solutions/Argo_CD_components_fail_Dex_image_pull_when_air-gapped_Redis_timeouts_in_an_ambient_mesh_or_dev_shm_full_on_ACP.md new file mode 100644 index 000000000..b8bd9e610 --- /dev/null +++ b/docs/en/solutions/Argo_CD_components_fail_Dex_image_pull_when_air-gapped_Redis_timeouts_in_an_ambient_mesh_or_dev_shm_full_on_ACP.md @@ -0,0 +1,144 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Argo CD components fail: Dex image pull when air-gapped, Redis timeouts in an ambient mesh, or dev shm full on ACP + +## Issue + +Argo CD on ACP is reconciled by `argocd-operator` from an `ArgoCD` resource (`argoproj.io/v1beta1`); direct edits to the Deployments and StatefulSets it owns are reverted. One of these is reported: + +- In an air-gapped cluster, `argocd-dex-server` cannot pull `ghcr.io/dexidp/dex@sha256:...`, and SSO (for example through a Keycloak OIDC connector) fails with `i/o timeout`. +- After the Dex image is mirrored, the Dex Pod still fails with `failed to pull and unpack image "/3rdparty/dex:2.43.0": failed to resolve reference ... x509: certificate signed by unknown authority`, or the `argocd-dex-server` Deployment returns to the `ghcr.io` image after it was edited. +- Every Argo CD component fails to reach Redis with `connection reset by peer`, `EOF`, `context deadline exceeded`, or `i/o timeout`; `application-controller` logs `Failed to save cluster info: i/o timeout` and `Failed to cache app resources: i/o timeout`; SSO is broken too. +- The Argo CD UI shows `failed to write manifest: write /dev/shm/...: no space left on device`. + +Related articles: "Argo CD Application sync not permitted, repository Not Found, InvalidSpecError, or empty GitOps project list on ACP". + +## Root Cause + +### Cause 1: the Dex image is referenced from `ghcr.io`, or the mirrored image is set in the wrong place or with the wrong tag + +An instance with `spec.sso.provider: dex` references the external image `ghcr.io/dexidp/dex@sha256:b08a58c9731c693b8db02154d7afda798e1888dc76db30d34c4a0d0b8a26d913` (Dex `v2.43.0`, linux/amd64). Without internet access no Dex Pod runs. `argocd-server` is always started with the `--dex-server` flag, so a running Dex Pod is required. Two mistakes keep Dex failing after the image is mirrored: + +- **Tag without `v`:** the image is pushed as `3rdparty/dex:v2.43.0`, but the Pod references `3rdparty/dex:2.43.0`. The pull fails with `failed to resolve reference ... x509: certificate signed by unknown authority`. Although the message names a certificate, correcting the tag to `v2.43.0` lets the Pod start. +- **Image edited in the Deployment:** the operator reconciles the `argocd-dex-server` Deployment back to the `ghcr.io` image. The image must be set in the Dex section of the `ArgoCD` resource (`spec.sso.dex.image` and `spec.sso.dex.version`). + +### Cause 2: the Argo CD namespace is in an Istio ambient mesh with STRICT mTLS + +The Argo CD namespace carries the label `istio.io/dataplane-mode=ambient`, and a global `PeerAuthentication` in `istio-system` sets mTLS to `STRICT`. ztunnel enforces mTLS on the node, without sidecars, and silently blocks the plain TCP traffic between Argo CD components. This surfaces as timeouts across Redis, Dex, and repo-server. + +The Redis TLS setup looks like the cause but is not. The operator starts Redis with TLS when a Redis TLS Secret exists and configures components with `--redis-use-tls`; the resource has no option to disable Redis TLS, and `spec.redis.disableTLSVerification` only skips certificate verification. Deleting `argocd-repo-server-tls`, the Argo CD server Deployment, or the Redis StatefulSet to force reconfiguration leaves Redis stuck on `MountVolume.SetUp failed for volume "redis-tls" : secret "-redis-tls" not found` and does not address the mesh. + +### Cause 3: `/dev/shm` of the repo server is too small + +The repo server writes generated manifests to `/dev/shm`. The default shared memory is too small for the manifests of the instance. + +## Resolution + +### Cause 1: the Dex image is referenced from `ghcr.io`, or the mirrored image is set in the wrong place or with the wrong tag + +1. On a master node of the global cluster, load the image and push it to the platform registry. Use `nerdctl` there, because Docker is not running on the masters (for example on ACP 4.2.1). The loaded image has no name (``), so tag it by image ID: + + ```bash + nerdctl -n k8s.io load -i dex.tar + nerdctl -n k8s.io images | grep '' + cat /etc/kubernetes/registry/auth.yaml # platform registry credentials + nerdctl -n k8s.io login + nerdctl -n k8s.io tag /3rdparty/dex:v2.43.0 + nerdctl -n k8s.io push /3rdparty/dex:v2.43.0 + ``` + +2. Set the image in the `ArgoCD` resource, not in the Deployment. The tag must be exactly the pushed tag, `v2.43.0`: + + ```yaml + spec: + sso: + provider: dex + dex: + image: /3rdparty/dex + version: v2.43.0 + ``` + + In the instance form these are the **Image** and **Version** fields of the Dex section. + + ```bash + kubectl -n edit argocd + ``` + +3. The Dex Pod starts and SSO works. + +### Cause 2: the Argo CD namespace is in an Istio ambient mesh with STRICT mTLS + +1. Take the Argo CD namespace out of the ambient mesh. + + > **Warning:** Removing the label takes every Pod in the namespace out of the Istio ambient mesh: traffic to and from them is no longer mTLS-protected by ztunnel, and mesh policies stop applying to them. Precondition: only when the label and the STRICT `PeerAuthentication` are confirmed (Diagnostic Steps, step 2) and no workload in the namespace depends on the mesh. Rollback: `kubectl label ns istio.io/dataplane-mode=ambient`. + + ```bash + kubectl label ns istio.io/dataplane-mode- + ``` + +2. Keep Argo CD in a namespace without the ambient label; the operator does not support opting individual Argo CD Pods out of the mesh. + +3. If Redis TLS Secrets or Argo CD workloads were deleted during troubleshooting, let the operator recreate them and check that the Redis Pod mounts its TLS volume. + +### Cause 3: `/dev/shm` of the repo server is too small + +1. Mount a memory-backed `emptyDir` at `/dev/shm` for the repo server through the `ArgoCD` resource, leaving the rest of the configuration unchanged. + + > **Warning:** The change rolls the repo server; manifest generation and syncs pause until the new Pod is Ready. An `emptyDir` with `medium: Memory` counts against the Pod's memory, so a large `sizeLimit` can push the repo server into OOM. Precondition: memory headroom in the repo server limit. Rollback: remove the volume and the mount again. + + ```yaml + spec: + repo: + volumeMounts: + - mountPath: /dev/shm + name: shm-volume + volumes: + - name: shm-volume + emptyDir: + medium: Memory + sizeLimit: 1Gi # 512Mi or 1Gi recommended + ``` + + ```bash + kubectl -n edit argocd + ``` + +2. `df -h /dev/shm` in the `argocd-repo-server` container shows a `tmpfs` mount with free space, and the error does not return. + +## Diagnostic Steps + +1. For SSO failures, check the Dex image and pull events: + + ```bash + kubectl -n get deploy -dex-server -o jsonpath='{.spec.template.spec.containers[*].image}{"\n"}' + kubectl -n get events --field-selector reason=Failed | grep -i dex + kubectl -n get argocd -o jsonpath='{.spec.sso.dex.image}:{.spec.sso.dex.version}{"\n"}' + ``` + + An image that still starts with `ghcr.io` is Cause 1. For an `x509: certificate signed by unknown authority` pull error on the mirrored image, first compare the referenced tag with the pushed tag (`v2.43.0`). + +2. For timeouts across components, check the mesh before touching Redis TLS: + + ```bash + kubectl get ns --show-labels + kubectl get peerauthentication -A + kubectl -n logs -application-controller-0 --tail=50 | grep -i timeout + ``` + + `istio.io/dataplane-mode=ambient` on the namespace plus a `STRICT` `PeerAuthentication` is Cause 2. + +3. For the manifest write error, check the repo server mount: + + ```bash + kubectl -n exec deploy/-repo-server -c argocd-repo-server -- df -h /dev/shm + ``` + + A full `/dev/shm` (`Use%` near `100%`) is Cause 3. diff --git a/docs/en/solutions/Argo_Rollouts_Kubeflow_Base_or_a_wrong-architecture_cluster_plugin_fails_to_install_or_upgrade_on_ACP.md b/docs/en/solutions/Argo_Rollouts_Kubeflow_Base_or_a_wrong-architecture_cluster_plugin_fails_to_install_or_upgrade_on_ACP.md new file mode 100644 index 000000000..a174fc648 --- /dev/null +++ b/docs/en/solutions/Argo_Rollouts_Kubeflow_Base_or_a_wrong-architecture_cluster_plugin_fails_to_install_or_upgrade_on_ACP.md @@ -0,0 +1,199 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "4.1" + - "4.2" + - "4.3" +--- + +# Argo Rollouts, Kubeflow Base, or a wrong-architecture cluster plugin fails to install or upgrade on ACP + +## Issue + +A specific cluster plugin fails to install, upgrade, or uninstall: + +- An x86 plugin was installed on an ARM cluster, its init container reports an architecture mismatch, and its deletion is stuck. +- Argo Rollouts fails to deploy on ACP 4.2.2 because the init container `copy-gwapi-plugin` cannot pull `rollouts-plugin-trafficrouter-gatewayapi:v1`. +- Kubeflow Base ends in Install Failed (`安装失败`) with `namespaces "kubeflow" not found` or `illegal base64 data`, or its web UI answers `RBAC denied`. +- A Kubeflow Base upgrade fails on a cluster whose Istio uses revision-based injection, and the `kubeflow-istio-ingressgateway` Pod was never injected. + +Related articles: "Cluster plugin install or upgrade stuck in chart sync: sentry, StatefulSet Forbidden, hook Job, or missing ModuleConfig" covers failures that are not specific to one plugin. + +## Root Cause + +### Cause 1: An x86 plugin was installed on an ARM cluster, and its cleanup BroadcastJob fails too + +On an ARM cluster (for example ACP 4.3.1 ARM), the multus plugin init container reports an architecture mismatch, and after the plugin is deleted from the console it stays deleting because the cleanup BroadcastJob fails on the same mismatch. An x86 build of the plugin was deployed; its cleanup job uses the same wrong-architecture image, so the uninstall cannot finish. The known recovery is to redeploy the cluster and then deploy the plugin build for the cluster's architecture. + +### Cause 2: Argo Rollouts on 4.2.2 references the wrong init container image tag + +The Pod `argo-rollouts-` in namespace `argo-rollouts` repeats a `Warning` event for its init container `copy-gwapi-plugin`: + +```text +Failed to pull image "/acp/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi:v1": failed to pull and unpack image ... failed to resolve reference ... Head "https:///v2/acp/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi/manifests/v1": tls: failed to verify certificate: x509: certificate signed by unknown authority +``` + +This is a known issue in ACP 4.2.2: the init container of the argo-rollouts chart references the incorrect tag `v1`. The pull of that tag can surface as the `x509` error above; the cause is the tag. The workaround overrides the init container through `valuesOverride` on the argo-rollouts ModuleInfo of the affected cluster, pointing it to `rollouts-plugin-trafficrouter-gatewayapi:v0.6.0-6` in the cluster's registry. + +### Cause 3: Kubeflow Base install prerequisites are not met + +The Kubeflow Base cluster plugin (for example `v1.10.10`) ends in Install Failed with one of: + +```text +failed to create NetworkPolicy "aml-api-deploy/kubeflow": namespaces "kubeflow" not found +failed to create Secret "cpaas-https-certs/istio-system": Secret in version "v1" cannot be handled as a Secret: illegal base64 data at input byte 2272 +``` + +or the plugin installs and its web UI returns `RBAC denied`, also over NodePort and with `curl` without a token. + +Kubeflow Base needs Alauda Service Mesh (ASM) deployed in the same cluster, and its install form needs the istio ingress gateway HTTPS NodePort. Installing without a service mesh, creating a gateway by hand, or pointing `oidcRedirectURL` to that gateway's load balancer IP leads to these errors; creating the `kubeflow` namespace by hand only moves the failure to the Secret error. Kubeflow Base creates its own Service `kubeflow-istio-ingressgateway`, a NodePort Service (`80:30080/TCP,443:30443/TCP` with the NodePorts chosen in the form), so no manual gateway is needed. Other form rules: + +- `oidcRedirectURL` points to an existing cluster node or the cluster VIP. +- `pgStorageClass` must be a StorageClass in the same cluster. +- The form has no setting to keep components off GPU nodes; taint the GPU nodes instead. +- The plugin template supports only NodePort; LoadBalancer is not supported. +- The ASM console lists only `gatewaydeploys.asm.alauda.io` resources, so `kubeflow-istio-ingressgateway`, a plain Deployment created by the plugin, does not appear there. + +The `RBAC denied` web UI is a product limitation, not a fault: Kubeflow Base depends on both ASM and AML (Alauda AI). Deploying Kubeflow Base standalone, without AML, is not supported. + +### Cause 4: kubeflow-istio-ingressgateway has no istio.io/rev label in its Pod template + +The gateway Pod is not injected, so its image address is not rewritten, and the Kubeflow Base upgrade fails. The `kubeflow-istio-ingressgateway` Deployment is created by the Kubeflow chart (`helm.sh/chart-name: kfbase`, release `kfbase` in `cpaas-system`) and carries `istio.io/rev: ` (for example `1-22`) on the Deployment metadata. Its Pod template has `inject.istio.io/templates: gateway`, `sidecar.istio.io/inject: "true"`, and `app` / `istio` labels, but no `istio.io/rev` label. Service Mesh v1 Istio uses revision-based injection, so the Pod template needs `istio.io/rev: ` together with `sidecar.istio.io/inject: "true"`. Injection is what rewrites the gateway image address; adding the label to the Pod template is the fix. + +## Resolution + +### Cause 1: An x86 plugin was installed on an ARM cluster, and its cleanup BroadcastJob fails too + +Confirm the mismatch: + +```bash +kubectl get nodes -o custom-columns=NAME:.metadata.name,ARCH:.status.nodeInfo.architecture +kubectl get broadcastjob -A | grep -i +kubectl -n logs --all-containers --tail=50 +``` + +> **Warning:** Redeploying the cluster destroys every workload and all data on it. This cannot be undone. Precondition: only when an x86 build of the plugin was installed on an ARM cluster, the plugin is stuck deleting because its cleanup BroadcastJob fails on architecture, and the cluster's workloads and data are backed up or disposable. Rollback: none; restore workloads from backup after the redeploy. + +Redeploy the cluster, then deploy the plugin from a package built for the cluster's architecture. Before installing any plugin on an ARM or hybrid cluster, check that the package you push is the ARM or hybrid build. + +### Cause 2: Argo Rollouts on 4.2.2 references the wrong init container image tag + +Confirm the tag in the event, then, on `global`, find and save the argo-rollouts ModuleInfo of the affected cluster: + +```bash +kubectl -n argo-rollouts get pods +kubectl -n argo-rollouts describe pod | grep -E 'copy-gwapi-plugin|rollouts-plugin-trafficrouter-gatewayapi' +kubectl get moduleinfo | grep | grep argo-rollouts +kubectl get moduleinfo -o yaml > "$HOME/.before.yaml" +``` + +> **Warning:** `valuesOverride` changes the rendered argo-rollouts chart; the controller Pod is recreated with the overridden init container, and a wrong image or mount keeps it from starting. Precondition: only on ACP 4.2.2 when the argo-rollouts Pod fails to pull `rollouts-plugin-trafficrouter-gatewayapi:v1` for `copy-gwapi-plugin`. Rollback: remove the `valuesOverride` block again. + +```bash +kubectl edit moduleinfo +``` + +Add the override under `spec`, replacing `` with the image registry address of that cluster: + +```yaml +spec: + config: {} + valuesOverride: + acp/argoproj/chart/argo-rollouts: + controller: + initContainers: + - args: + - cp /bin/rollouts-plugin-trafficrouter-gatewayapi /plugins + command: + - /bin/sh + - -c + image: /acp/argoproj-labs/rollouts-plugin-trafficrouter-gatewayapi:v0.6.0-6 + name: copy-gwapi-plugin + resources: + requests: + cpu: 100m + memory: 64Mi + limits: + cpu: 200m + memory: 128Mi + ephemeral-storage: 500Mi + volumeMounts: + - mountPath: /plugins + name: gwapi-plugin +``` + +Check that the controller Pod starts: + +```bash +kubectl get pods -A | grep argo-rollouts +``` + +### Cause 3: Kubeflow Base install prerequisites are not met + +Check the prerequisites on the target cluster before reinstalling: + +```bash +kubectl get pods -A | grep -i istio +kubectl get gatewaydeploys.asm.alauda.io -A +kubectl get sc +kubectl get svc -A | grep -E '30443|30080' +kubectl get minfo | grep | grep -iE 'aml|asm|kubeflow' +``` + +Deploy Alauda Service Mesh in the same cluster first if it is missing. Pick two free NodePorts (no output from the `grep` above means 30080 and 30443 are free), choose a `pgStorageClass` from `kubectl get sc` on that cluster, and set `oidcRedirectURL` to an existing cluster node IP or the cluster VIP, not to a load balancer IP. + +> **Warning:** Deleting a gateway created by hand for Kubeflow removes any traffic routed through it. Uninstalling a failed Kubeflow Base removes what it partly deployed. Precondition: only when Kubeflow Base failed with the errors above and the gateway was created only for Kubeflow. Export it first with `kubectl -n get gatewaydeploys.asm.alauda.io -o yaml > "$HOME/.yaml"`. Rollback: `kubectl apply -f` the export. + +```bash +kubectl -n delete gatewaydeploys.asm.alauda.io +``` + +Uninstall the failed Kubeflow Base from Cluster Plugins, install it again with the values above, and check: + +```bash +kubectl get svc -A | grep kubeflow-istio-ingressgateway +kubectl -n kubeflow get deploy,pods +kubectl -n kubeflow-oauth2-proxy get deploy,pods +curl -k -s -o /dev/null -w '%{http_code}\n' https://:30443 +``` + +All `kubeflow` and `kubeflow-oauth2-proxy` Pods should reach `Running`. The UI is reached at `https://:`; keep the Service as NodePort. If the UI answers `RBAC denied` and AML is not installed on the cluster, install Kubeflow Base together with ASM and AML; standalone use is a product requirement, not a configuration change. + +### Cause 4: kubeflow-istio-ingressgateway has no istio.io/rev label in its Pod template + +```bash +kubectl get deploy -A | grep kubeflow-istio-ingressgateway +kubectl -n get deploy kubeflow-istio-ingressgateway -o jsonpath='{.metadata.labels.istio\.io/rev}{"\n"}{.spec.template.metadata.labels}{"\n"}' +kubectl -n get deploy kubeflow-istio-ingressgateway -o yaml > "$HOME/kubeflow-istio-ingressgateway.before.yaml" +``` + +A revision on the Deployment metadata and no `istio.io/rev` in the Pod template labels is this cause. + +> **Warning:** Changing the Pod template rolls the Kubeflow ingress gateway; Kubeflow UI traffic is interrupted until the injected Pod is Ready. Precondition: only when the mesh uses revision-based injection, the gateway Pod is not injected, and the Pod template lacks `istio.io/rev`; use the revision shown on the Deployment metadata. Rollback: `kubectl apply -f "$HOME/kubeflow-istio-ingressgateway.before.yaml"`. + +```bash +kubectl -n patch deploy kubeflow-istio-ingressgateway --type=merge -p '{"spec":{"template":{"metadata":{"labels":{"istio.io/rev":"","sidecar.istio.io/inject":"true"}}}}}' +kubectl -n get pods | grep kubeflow-istio-ingressgateway +``` + +Check that the new Pod runs the rewritten image, then retry the Kubeflow Base upgrade. + +## Diagnostic Steps + +```bash +kubectl get ars -A +kubectl get minfo | grep +kubectl get broadcastjob -A +kubectl get nodes -o custom-columns=NAME:.metadata.name,ARCH:.status.nodeInfo.architecture +``` + +- An architecture mismatch in both the plugin and its cleanup BroadcastJob: Cause 1. +- A pull failure for `rollouts-plugin-trafficrouter-gatewayapi:v1` on the `copy-gwapi-plugin` init container of Argo Rollouts on 4.2.2: Cause 2. +- Kubeflow Base failing on `namespaces "kubeflow" not found` or `illegal base64 data`, or answering `RBAC denied` without AML: Cause 3. +- A failed Kubeflow Base upgrade with an uninjected `kubeflow-istio-ingressgateway` whose Pod template has no `istio.io/rev`: Cause 4. + +A cluster plugin pushed with `violet push` is registered on `global` and offered to clusters by its affinity; look for it under the target cluster's Cluster Plugins rather than pushing it to each cluster. diff --git a/docs/en/solutions/Built-in_admin_username_cannot_be_renamed_disable_admin_after_creating_another_platform_administrator_on_ACP.md b/docs/en/solutions/Built-in_admin_username_cannot_be_renamed_disable_admin_after_creating_another_platform_administrator_on_ACP.md new file mode 100644 index 000000000..8e219ae03 --- /dev/null +++ b/docs/en/solutions/Built-in_admin_username_cannot_be_renamed_disable_admin_after_creating_another_platform_administrator_on_ACP.md @@ -0,0 +1,67 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x +--- + +# Built-in admin username cannot be renamed: disable admin after creating another platform administrator on ACP + +## Issue + +A security compliance assessment flags the platform's default `admin` account as a high-risk item (for example "the system has an admin account"), and asks for the default account to be renamed or removed. Administrators look for a way to change the login username of `admin`: + +- In the console, the user details and the personal center only allow changing the **display name** (and the password), not the username. +- The product documentation describes the platform administrator created at deployment as a system account that cannot be managed through user management, and has no procedure for renaming it. + +## Root Cause + +The login username of the local platform administrator created at deployment (`admin`) cannot be changed, either in the console or in the backend. The console only supports changing a user's display name. + +The `User` resource (`auth.alauda.io/v1`) of each user has a boolean `spec.is_disabled` field. Setting it to `true` on the `admin` user disables that account, which removes the default account from use without renaming it. + +## Resolution + +The supported workaround is to create a replacement platform administrator and then disable `admin`. + +1. Create a new local user with a non-default username in the console and grant it the platform administrator role. + +2. Log out, log in with the new user, and confirm it has full platform administration access. Keep this session open until step 4 is verified. + +3. On a control plane node of the `global` cluster, find the `User` resource of `admin`. Its resource name is a hash, not the username: + + ```bash + kubectl get users.auth.alauda.io | grep admin + kubectl get users.auth.alauda.io -o jsonpath='{.spec.username} is_admin={.spec.is_admin}{"\n"}' + ``` + + Expected: `admin is_admin=true`. + +4. Disable the account. + + > **Warning:** After this step nobody can log in to the console as `admin`. Run it only after step 2 confirms that the new platform administrator can log in and manage the platform, otherwise you can lock yourself out of the console. Check first that no script or integration still logs in as `admin`. To roll back, set `is_disabled` back to `false` (or remove it) on the same resource from a `global` control plane node with `kubectl`. + + ```bash + kubectl edit users.auth.alauda.io + ``` + + Add the field under `spec`: + + ```yaml + spec: + is_disabled: true + ``` + +5. Open a new private browser window and confirm that logging in as `admin` is refused while the new administrator still works. + +## Diagnostic Steps + +1. Confirm which account the assessment refers to: the local `admin` user created at deployment has `spec.username: admin` and `spec.is_admin: true`: + + ```bash + kubectl get users.auth.alauda.io -o custom-columns='NAME:.metadata.name,USERNAME:.spec.username,ADMIN:.spec.is_admin,DISABLED:.spec.is_disabled' + ``` + +2. After the change, the same command shows `DISABLED` = `true` for `admin` and the new administrator listed with `ADMIN` = `true`. diff --git a/docs/en/solutions/Business_Pods_recreated_on_many_nodes_after_kubelet_restarts_for_certificate_rotation_on_ACP.md b/docs/en/solutions/Business_Pods_recreated_on_many_nodes_after_kubelet_restarts_for_certificate_rotation_on_ACP.md new file mode 100644 index 000000000..76e0e2e28 --- /dev/null +++ b/docs/en/solutions/Business_Pods_recreated_on_many_nodes_after_kubelet_restarts_for_certificate_rotation_on_ACP.md @@ -0,0 +1,69 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.12.x + - 4.0.x + - 4.2.x + - 4.3.x + - 4.4.x +--- + +# Business Pods recreated on many nodes after kubelet restarts for certificate rotation on ACP + +## Issue + +Many business Pods are recreated at the same time, on more than one node and sometimes on more than one cluster. The kubelet logs on those nodes show that kubelet restarted because of certificate rotation. The Kubernetes Certificates Rotator plugin is installed, and the restart happened about 61 days after the kubelet certificate was issued. The recreated Pods do not mount any certificate volume. + +Related articles: "Recovering ACP Kubernetes Certificates with Kubernetes Certificates Rotator" covers restoring long-lived certificates after the plugin; "Log module and monitoring dashboard unavailable after a kube-apiserver internal certificate expires on ACP" covers certificates that expired after the plugin was uninstalled. + +## Root Cause + +### Cause 1: kubelet restarts on certificate-rotation day 61 and re-admits Pods against the current node isolation policy + +The Kubernetes Certificates Rotator plugin replaces the default long-lived Kubernetes certificates with short-lived ones (91 days for kubelet certificates), renews the kubelet certificate about 61 days after issue, and restarts kubelet so that it loads the new certificate. + +When kubelet restarts, it no longer has the admission state it held in memory, and it runs the admission checks, including node affinity, again for the Pods assigned to the node. Pods whose node affinity no longer matches the node are rejected, and their controllers create replacements. This happens when the saved Pods still require a previous node-group label while the node isolation policy in force has changed. The restart for certificate rotation is what triggers the recreation; the Pods are not missing a certificate. + +## Resolution + +### Cause 1: kubelet restarts on certificate-rotation day 61 and re-admits Pods against the current node isolation policy + +1. Confirm the signature: kubelet restarted because of certificate rotation, the rotation fell on day 61 of the kubelet certificate, and the recreated Pods fail node affinity against the current labels (Diagnostic Steps). + +2. Stop the periodic restarts: replace the short-lived certificates with 10-year certificates as described in the platform certificate documentation, then uninstall the Kubernetes Certificates Rotator plugin. The re-signing procedure is in "Recovering ACP Kubernetes Certificates with Kubernetes Certificates Rotator". + + > **Warning:** Uninstalling the plugin without long-lived certificates in place stops rotation of short-lived certificates, which then expire and break the control plane and kubelet. Uninstall only after the long-lived certificates are written and verified on every node, and only on a cluster that shows this kubelet-restart signature. Re-signing restarts the control-plane components and kubelet on each node; follow the backup and rollback steps of that procedure. + +3. Changing workload selectors or node labels would change where business Pods can be scheduled. That is a separate decision for the workload owner and the node isolation policy owner, not part of this fix. + +## Diagnostic Steps + +1. Check the kubelet restart and its reason on an affected node: + + ```bash + journalctl -u kubelet --since "" | grep -i -E 'certificate|rotat|Started|Stopping' | head -50 + openssl x509 -in /var/lib/kubelet/pki/kubelet.crt -noout -dates 2>/dev/null + openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates + ``` + + A restart at the time of the recreation, and a kubelet certificate whose `notBefore` matches that time with a validity of about 91 days, match this cause. + +2. Check that the plugin is installed: + + ```bash + kubectl get moduleinfo -l cpaas.io/module-name=cluster-cert-rotator + ``` + +3. Check why the old Pods were rejected: + + ```bash + kubectl get pods -A --field-selector status.phase=Failed -o wide | head + kubectl get pod -n -o jsonpath='{.status.reason}{"\t"}{.status.message}{"\n"}' + kubectl get pod -n -o jsonpath='{.spec.affinity.nodeAffinity}{"\n"}{.spec.nodeSelector}{"\n"}' + kubectl get node --show-labels + ``` + + A reason such as `NodeAffinity`, and a required node label that the node no longer carries, confirm that the recreation was caused by admission after the kubelet restart. diff --git a/docs/en/solutions/Business_TLS_certificate_not_renewed_not_reloaded_by_ingress-nginx_or_rejected_with_a_format_error_on_import_on_ACP.md b/docs/en/solutions/Business_TLS_certificate_not_renewed_not_reloaded_by_ingress-nginx_or_rejected_with_a_format_error_on_import_on_ACP.md new file mode 100644 index 000000000..e02873587 --- /dev/null +++ b/docs/en/solutions/Business_TLS_certificate_not_renewed_not_reloaded_by_ingress-nginx_or_rejected_with_a_format_error_on_import_on_ACP.md @@ -0,0 +1,125 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Business TLS certificate not renewed, not reloaded by ingress-nginx, or rejected with a format error on import on ACP + +## Issue + +One of the following happens with a business (application) TLS certificate: + +- A certificate expiry alert names a business certificate that the platform does not renew. +- The ingress-nginx gateway serves a renewed certificate only after it is restarted by hand, and a script that restarts it automatically after each certificate update is requested. +- A wildcard certificate has to be configured as a Secret in every project, and the question is whether it can be managed once for the whole platform. +- **Create Certificate** under **Network Management > Certificates** rejects a certificate that includes the intermediate chain: the public key field shows `格式错误` (format error). The question is which certificate format network management expects. + +Related articles: "Update a platform domain certificate stored in a cpaas-system Secret on ACP" covers platform domain certificates in `cpaas-system`; "Replace the platform access certificate in dex.tls on ACP" covers the platform access address. + +## Root Cause + +### Cause 1: the alerting certificate is a business certificate + +The certificate that did not renew is not a platform component certificate. Neither the platform cert-manager nor any platform operator renews it; the team that owns the business renews and reloads it. An ALB rule and the `alb.networking.cpaas.io/tls` annotation select a Secret, so replacing only a ConfigMap copy of a certificate does not change what the ALB serves. + +### Cause 2: ingress-nginx needs a restart to use a renewed certificate because its image is not the platform one + +When a certificate is replaced under **Platform Management > Network Management > Domain**, the Secret in `cpaas-system` is updated and the platform-provided ingress-nginx reloads it without a restart. An ingress-nginx image that is not provided by the platform may not hot-reload certificates, so it keeps serving the old certificate until it restarts. Custom ingress-nginx images are not supported, and no automatic restart script is provided. + +### Cause 3: ACP 4.3 has no platform-wide management of one wildcard certificate across projects + +ACP 4.3 has no feature that manages one certificate centrally for every project. Certificates are configured as Secrets in each project that needs them. The platform domain Secrets in `cpaas-system` are one per platform-managed domain; they are not a way to share a wildcard certificate with every project. + +### Cause 4: Create Certificate reports a format error because the imported file has Windows (\r\n) line endings + +Certificates under network management use the PEM x509 format. A chain with several `-----BEGIN CERTIFICATE-----` / `-----END CERTIFICATE-----` blocks (leaf plus intermediates) is valid input. On ACP 4.1.0, importing such a file with the `导入` (Import) button of the public key field fails the front-end format check when the file uses Windows `\r\n` line endings, because the import function of that version does not support them. Pasting the same content into the field works. + +## Resolution + +### Cause 1: the alerting certificate is a business certificate + +1. Confirm that the name is not a platform component or operator certificate (see "Platform certificate expiry alert for a cert-manager serving certificate that renews by itself on ACP"), and print the business certificate: + + ```bash + kubectl get secret -n -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject -issuer + ``` + +2. Hand the renewal to the business owner. A business certificate is updated under **Platform Management > Network Management > Certificates**. On a 3.x TKE-based platform the page is **Platform Management > Network Management > Domain Certificates**: select the certificate, open the three-dot menu on the right, and choose **Update**. +3. After the Secret is replaced, check that the ALB rule or Ingress references that Secret, and check the certificate served on the business hostname: + + ```bash + echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -dates -subject + ``` + +### Cause 2: ingress-nginx needs a restart to use a renewed certificate because its image is not the platform one + +1. Check which image the gateway runs and compare it with the image the platform ships: + + ```bash + kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.spec.containers[*].image}{"\n"}{end}' | grep -i ingress-nginx + ``` + +2. If the image is not the platform-provided one, switch to the platform ingress-nginx, so that certificate updates are reloaded automatically. +3. Until then, restart the gateway after each certificate change. + + > **Warning:** Restarting ingress-nginx interrupts the connections it is serving while its Pods are replaced. Do this only after the certificate Secret was updated and only for a gateway that runs a non-platform image. A rolling restart has nothing to roll back. + + ```bash + kubectl rollout restart deployment -n + kubectl rollout status deployment -n + ``` + +### Cause 3: ACP 4.3 has no platform-wide management of one wildcard certificate across projects + +Configure the wildcard certificate as a Secret in each project that needs it, and reference that Secret from the ALB rule or Ingress of the project. There is no central, platform-wide alternative in 4.3. + +### Cause 4: Create Certificate reports a format error because the imported file has Windows (\r\n) line endings + +1. Check the file for carriage returns and confirm that each block parses as PEM x509: + + ```bash + grep -c $'\r' .pem + openssl crl2pkcs7 -nocrl -certfile .pem | openssl pkcs7 -print_certs -noout + ``` + + A non-zero count means Windows line endings. + +2. Use one of these workarounds: + + - Paste the certificate content into the public key field instead of using `导入` (Import). + - After importing, make a small edit in the field (for example delete one `-` and type it back) so that the front-end validation runs again. + - Convert the file to Unix line endings before importing: + + ```bash + tr -d '\r' < .pem > -unix.pem + grep -c $'\r' -unix.pem + ``` + + The second command prints `0`. + +## Diagnostic Steps + +1. Identify the Secret and its owner: + + ```bash + kubectl get secret -n -o jsonpath='{.metadata.labels}{"\n"}{.metadata.annotations}{"\n"}{.metadata.ownerReferences}{"\n"}' + ``` + + A Secret in a business namespace without platform labels or an owning platform operator is a business certificate. + +2. Branch: + + - The alert names a business certificate: Cause 1. + - ingress-nginx serves the new certificate only after a restart: Cause 2. + - One wildcard certificate for every project is requested: Cause 3. + - Create Certificate shows `格式错误` for a chain, or the question is which format is accepted (PEM x509): Cause 4. diff --git a/docs/en/solutions/Calico_Pods_get_no_address_disabled_default_IPPool_Raven_fixed_IP_mismatch_reserved_IPs_or_deleted_raven_on_ACP.md b/docs/en/solutions/Calico_Pods_get_no_address_disabled_default_IPPool_Raven_fixed_IP_mismatch_reserved_IPs_or_deleted_raven_on_ACP.md new file mode 100644 index 000000000..74e72ac71 --- /dev/null +++ b/docs/en/solutions/Calico_Pods_get_no_address_disabled_default_IPPool_Raven_fixed_IP_mismatch_reserved_IPs_or_deleted_raven_on_ACP.md @@ -0,0 +1,106 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "3.18" + - "4.0" + - "4.2" + - "4.3" +--- + +# Calico Pods get no address: disabled default IPPool, Raven fixed IP mismatch, reserved IPs or deleted raven on ACP + +## Issue + +On a Calico cluster, Pods cannot get an address, or a subnet question comes up: + +- A Pod (for example a Kafka instance Pod) stays in `Init` because Calico allocates nothing from the default IPPool `default-ipv4-ippool`. +- A fixed-IP Deployment loops on one address because the Raven IP status and Calico IPAM disagree. +- Someone wants to reserve part of a Calico subnet so it is never allocated. +- The `raven` Deployment was deleted by mistake. + +On Calico clusters, `raven` is the platform component that converts platform `Subnet` resources into Calico `IPPool` objects and maintains the platform IP objects. + +Related articles: "Kube-OVN Pods fail with AddressOutOfRange, AddressConflict or no address allocated on ACP"; "Multus secondary networks: unexpected end of JSON input, duplicated addresses or several addresses on one NIC on ACP". + +## Root Cause + +### Cause 1: the default Calico IPPool is disabled + +`kubectl get ippools.crd.projectcalico.org default-ipv4-ippool -o yaml` shows `disabled: true` in the spec (next to `blockSize: 26`, `ipipMode: Always`, `natOutgoing: true`, `vxlanMode: Never`). Calico allocates nothing from a disabled pool, so Pods that depend on it wait in `Init`. + +### Cause 2: the Raven fixed IP stays empty while Calico IPAM still holds the address + +A Calico overlay Deployment with more pinned addresses than ready replicas leaves one Pod requesting an address that another Pod already has. Raven does not write the occupancy back and does not repair an existing IP object, so the CNI keeps selecting the address that Calico IPAM considers taken. A restart of the controller does not clear it; the repair is in a later 4.2.x release. + +### Cause 3: a Calico subnet cannot reserve part of its addresses + +There is no setting for reserved IPs on a Calico subnet. Excluding part of a Calico container subnet from allocation cannot be done by configuration; this is a product limitation. + +### Cause 4: the raven Deployment was deleted + +`raven` exports Calico subnet metrics such as `subnet_available_ip_count` and `subnet_used_ip_count` on port `10660` at `/metrics`, and it creates and deletes Calico `IPPool` objects to match the platform `Subnet` resources. While it is missing, those metrics stop and subnet-to-IPPool synchronization stops, which affects Pod creation and deletion. The platform records manual changes to `raven` (for example its resource limits) as a ResourcePatch (`rpch`) object, and a leftover `rpch` keeps the platform from restoring the default Deployment. + +## Resolution + +### Cause 1: the default Calico IPPool is disabled + +Remove the `disabled` field. + +> **Warning:** Re-enabling the pool lets every new Pod draw addresses from it again; if it was disabled on purpose to drain it, this undoes that plan. Only when no other pool is meant to serve these Pods. Save the object first; setting `disabled: true` again is the rollback. + +```bash +kubectl get ippools.crd.projectcalico.org -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidr,DISABLED:.spec.disabled' +kubectl get ippools.crd.projectcalico.org default-ipv4-ippool -o yaml > /root/ippool-default.before.yaml +kubectl patch ippools.crd.projectcalico.org default-ipv4-ippool --type=json -p='[{"op":"remove","path":"/spec/disabled"}]' +``` + +If a platform `Subnet` with the same name manages the pool, check that object as well, so the field is not set back. + +### Cause 2: the Raven fixed IP stays empty while Calico IPAM still holds the address + +Until the release with the fix, stop requesting the address that Calico still holds: remove it from the workload's fixed-IP list, or scale the Deployment down so only Pods with a consistent address remain. + +> **Warning:** Editing the fixed-IP list recreates Pods and drops the addresses you remove. Do not delete Calico IPAM blocks by hand; that can free an address a live Pod uses. Restoring the saved Deployment is the rollback and brings the conflict back. + +```bash +kubectl get ip -A | grep +kubectl get deploy -n -o yaml > /root/deploy.before.yaml +kubectl edit deploy -n +``` + +An IP object whose `containerID`, `nodeName` and `podName` are all empty, with `resource already exists` from the CNI, is a different case that a `raven` restart does fix; see "Kube-OVN fixed IP occupied or not released: held by another Pod, node down, duplicate entry or RollingUpdate on ACP". + +### Cause 3: a Calico subnet cannot reserve part of its addresses + +No configuration exists. Keep addresses that must not be allocated outside the Calico subnet's CIDR when planning it, or raise the requirement through the product channel. + +### Cause 4: the raven Deployment was deleted + +Delete the `rpch` objects that belong to `raven`; the platform then recreates the Deployment with its default specification. + +> **Warning:** Deleting the `rpch` drops every manual change recorded for `raven`, such as raised limits. Save it first and reapply the change to the recreated Deployment if you still need it. + +```bash +kubectl get rpch -A | grep raven +kubectl -n get rpch -o yaml > /root/raven-rpch.before.yaml +kubectl -n delete rpch +kubectl -n kube-system get deploy raven +``` + +## Diagnostic Steps + +```bash +kubectl get ippools.crd.projectcalico.org -o yaml | grep -E 'name:|cidr:|disabled:' +kubectl -n kube-system get deploy raven +kubectl get ip -A | grep
+kubectl -n kube-system get ds calico-node -o wide +``` + +- A pool (normally `default-ipv4-ippool`) with `disabled: true` and Pods waiting in `Init`: Cause 1. +- A fixed-IP Deployment whose Raven IP object and Calico allocation disagree on one address: Cause 2. Compare the two before deleting anything. +- A request to reserve addresses in a Calico subnet: Cause 3. +- No `raven` Deployment in `kube-system`: Cause 4. diff --git a/docs/en/solutions/Calico_ipipMode_reverts_Felix_metrics_off_bandwidth_annotation_breaks_Pods_systemd-udevd_loops_on_calico_tmp_on_ACP.md b/docs/en/solutions/Calico_ipipMode_reverts_Felix_metrics_off_bandwidth_annotation_breaks_Pods_systemd-udevd_loops_on_calico_tmp_on_ACP.md new file mode 100644 index 000000000..d6528892b --- /dev/null +++ b/docs/en/solutions/Calico_ipipMode_reverts_Felix_metrics_off_bandwidth_annotation_breaks_Pods_systemd-udevd_loops_on_calico_tmp_on_ACP.md @@ -0,0 +1,159 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Calico ipipMode reverts, Felix metrics off, bandwidth annotation breaks Pods, systemd-udevd loops on calico_tmp on ACP + +## Issue + +On a cluster that uses Calico as the CNI, a configuration change does not behave as expected: + +- `ipipMode` on the `IPPool` `default-ipv4-ippool` is changed from `Always` to `CrossSubnet` and flips back to `Always`. +- Felix monitoring shows no data and the Felix metrics port is not listening. +- New workloads cannot get an address because the default Calico BGP subnet has no free addresses, and other subnets are being considered. +- A Deployment stops starting Pods after a Calico bandwidth annotation was added to it. +- Node load is high because `systemd-udevd` keeps trying to configure the interfaces `calico_tmp_A` and `calico_tmp_B`. + +Related articles: "Pod traffic fails on Calico because of an IPPool destination, MTU, firewall, BGP sessions or IPv6 routes on ACP". + +## Root Cause + +### Cause 1: ipipMode CrossSubnet reverts to Always until the Subnet and the IPPool match + +The platform keeps a `Subnet` object and a Calico `IPPool` object with the same name (for example `default-ipv4-ippool`). The two have to stay consistent; otherwise the controller can overwrite one with the other, and the console can show a value different from the dataplane. Changing only the `IPPool` to `ipipMode: CrossSubnet` while the `Subnet` still says `Always` is the inconsistency that gets overwritten. The exact revert path was not confirmed. + +### Cause 2: Felix Prometheus metrics are off because prometheusMetricsEnabled is not set + +The Felix configuration has no `prometheusMetricsEnabled` setting, which means Felix metrics are disabled, so the metrics port is not listening and monitoring has nothing to scrape. Felix serves metrics on TCP `9091` when enabled; the endpoint is protected only by a self-signed certificate and has no authentication. + +### Cause 3: the default Calico BGP subnet has no free addresses + +New workloads cannot get an address from the default subnet because it is full. Placing business workloads on other subnets that still have free addresses restores allocation. Traffic between those subnets works without extra network policy. + +### Cause 4: Pods fail after a Calico bandwidth annotation was added + +A bandwidth annotation from the open-source Calico procedure (for example `kubernetes.io/ingress-bandwidth`) was added to the Pod template. Rate limiting did not work, and when a Pod was rebuilt the qdisc rules of the previous Pod had not been cleaned up, so the new Pod failed on a conflict between several qdiscs on one interface. The platform currently provides no supported method for Calico Pod bandwidth limiting. + +### Cause 5: systemd-udevd loops on calico_tmp_A and calico_tmp_B + +Calico XDP creates the temporary interfaces `calico_tmp_A` and `calico_tmp_B` and destroys them immediately. The host's `systemd-udevd` tries to configure each one, fails because it is already gone, and retries, which drives the node load up. XDP is an acceleration feature and disabling it has no functional impact. It is on by default in current versions and cannot be turned off at deployment time. + +## Resolution + +### Cause 1: ipipMode CrossSubnet reverts to Always until the Subnet and the IPPool match + +Set the same values on both objects in one change. + +> **Warning:** Changing `ipipMode` reprograms encapsulation and can reset cross-subnet tunnels while Felix applies the pool. Only when an `ipipMode: CrossSubnet` edit returns to `Always`. Save both objects first; restoring them is the rollback. + +```bash +kubectl get subnet default-ipv4-ippool -o yaml > /root/subnet-default-ipv4-ippool.before.yaml +kubectl get ippools.crd.projectcalico.org default-ipv4-ippool -o yaml > /root/ippool-default-ipv4-ippool.before.yaml +kubectl edit ippools.crd.projectcalico.org default-ipv4-ippool +kubectl edit subnet default-ipv4-ippool +``` + +Set these values where the fields exist: + +```yaml +ipipMode: CrossSubnet +vxlanMode: Never +``` + +If the `Subnet` uses a different key for the IPIP mode, change that key so its value matches the `IPPool`; do not add a parallel key. + +### Cause 2: Felix Prometheus metrics are off because prometheusMetricsEnabled is not set + +Enable Felix metrics on the `calico-node` DaemonSet. + +> **Warning:** The DaemonSet rolls and `calico-node` restarts on every node, which can briefly disturb container networking. Enabling the variable exposes the unauthenticated metrics endpoint on TCP `9091` on every node, so enable it only where a scraper needs it. Roll back by setting the variable to `false`, as described in "Disable Calico Node Metrics Port". + +```bash +kubectl -n kube-system get ds calico-node -o jsonpath='{.spec.template.spec.containers[*].name}{"\n"}' +kubectl -n kube-system set env ds/calico-node -c calico-node FELIX_PROMETHEUSMETRICSENABLED=true +kubectl -n kube-system rollout status ds/calico-node +ss -lnt | grep 9091 || true +``` + +### Cause 3: the default Calico BGP subnet has no free addresses + +Place new business workloads on a subnet that still has free addresses, using the subnet assignment the cluster already uses for business workloads, then confirm the new Pod address falls in that CIDR. Check the pools before moving any system namespace; this change concerns business workloads only, not `kube-system` or `cpaas-system`. + +```bash +kubectl get ippools.crd.projectcalico.org -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidr,DISABLED:.spec.disabled' +kubectl get subnet +kubectl -n get pod -o wide +``` + +### Cause 4: Pods fail after a Calico bandwidth annotation was added + +Remove the bandwidth annotations from the Pod template and let the Pods be recreated. + +> **Warning:** Editing the Pod template rolls the Deployment. Putting the annotation back brings the qdisc conflict back. + +```bash +kubectl -n get deploy -o jsonpath='{.spec.template.metadata.annotations}{"\n"}' +kubectl -n patch deploy --type=json -p='[{"op":"remove","path":"/spec/template/metadata/annotations/kubernetes.io~1ingress-bandwidth"}]' +``` + +`~1` encodes the `/` in the key. Repeat the patch for each bandwidth key the template has (for example `kubernetes.io/egress-bandwidth`). If Pods still fail afterwards, restarting kubelet and the container runtime on the node is the next step to have the stale qdiscs cleaned up; that restart affects every Pod on the node. + +### Cause 5: systemd-udevd loops on calico_tmp_A and calico_tmp_B + +Disable XDP on `calico-node`, then check that the temporary interfaces stop appearing. Analyzing `systemd-udevd` on the host is the alternative. + +> **Warning:** The change restarts every `calico-node` Pod, which affects workloads; do it in a maintenance window. Roll back by restoring the saved DaemonSet or unsetting the variable. + +```bash +kubectl -n kube-system get ds calico-node -o yaml > /root/calico-node.before-xdp.yaml +kubectl -n kube-system set env ds/calico-node -c calico-node FELIX_XDPENABLED=false +kubectl -n kube-system rollout status ds/calico-node +ip link show | grep calico_tmp || true +``` + +## Diagnostic Steps + +1. Compare the `Subnet` and the `IPPool` of the same name: + + ```bash + kubectl get subnet default-ipv4-ippool -o yaml | grep -iE 'ipip|vxlan' + kubectl get ippools.crd.projectcalico.org default-ipv4-ippool -o yaml | grep -iE 'ipipMode|vxlanMode' + ``` + + Different values: Cause 1. + +2. Check whether Felix metrics are enabled: + + ```bash + kubectl -n kube-system get ds calico-node -o yaml | grep -i -A1 PROMETHEUS + ss -lnt | grep 9091 || echo "9091 not listening" + ``` + + No `FELIX_PROMETHEUSMETRICSENABLED=true` and nothing listening on `9091`: Cause 2. + +3. Allocation failures with the default subnet full: Cause 3. + +4. Pod failures that start with a bandwidth annotation, with several qdiscs on the interface: + + ```bash + tc qdisc show | head + ``` + + Cause 4. + +5. `systemd-udevd` busy on the temporary interfaces: + + ```bash + journalctl -u systemd-udevd --since '-10 min' | grep calico_tmp | tail + ``` + + Cause 5. diff --git a/docs/en/solutions/Catalog_does_not_list_a_Helm_chart_that_is_already_in_Harbor_or_a_newly_uploaded_chart_version_on_ACP.md b/docs/en/solutions/Catalog_does_not_list_a_Helm_chart_that_is_already_in_Harbor_or_a_newly_uploaded_chart_version_on_ACP.md new file mode 100644 index 000000000..7a1409e28 --- /dev/null +++ b/docs/en/solutions/Catalog_does_not_list_a_Helm_chart_that_is_already_in_Harbor_or_a_newly_uploaded_chart_version_on_ACP.md @@ -0,0 +1,158 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.2" +--- + +# Catalog does not list a Helm chart that is already in Harbor, or a newly uploaded chart version, on ACP + +## Issue + +A Helm chart is in Harbor, or an upload reported success, but Catalog, the version dropdown, or template selection does not show it: + +- `helm push` printed `Pushed: //:`, Harbor shows the chart, but the template repository and Catalog stay on the old list or show nothing. +- A newly added chart repository lists none of the charts already stored in it. The repository detail page says `无应用模板` ("no application templates") or `暂时没有找到符合过滤条件的应用模板` ("no application templates match the filter"). +- A new chart package was uploaded (for example with `violet push`), but the template version dropdown still offers only the previous version, and the repository (for example `public-charts`) shows last sync time `-`. +- Upload reported `上传成功` (upload successful) and Harbor has the tag, but with a namespace of an ARM cluster selected, Catalog shows `共 0 个应用` (0 applications), `Helm Charts (0个)`, and `无应用列表` (no application list), while x86 clusters list the chart. Or the chart details open but the deploy form has no version to select. Or, after a version was deleted and the same version uploaded again, the whole chart disappears from Catalog, and creating it again reports that the version already exists. + +Restarting OLM Pods and refreshing the browser does not make the chart appear. `helm show chart` against the Harbor URL proves only that the registry has the package, not that Catalog has indexed it. + +Related articles: [Helm chart template upload fails, or the template directory cannot be selected, in Catalog on ACP](Helm_chart_template_upload_fails_or_the_template_directory_cannot_be_selected_in_Catalog_on_ACP.md). + +## Root Cause + +The Catalog UI lists a chart only when the platform has registered it: a matching `Artifact` and `ArtifactVersion` exist in the `cpaas-system` namespace of the `global` cluster. A chart that only exists in the registry is not listed. + +### Cause 1: ACP 4.2 does not synchronize OCI charts automatically + +ACP 4.2 does not support automatic chart synchronization. An OCI chart is not imported from Harbor because the registry has the tag; the template repository shows it after the chart is uploaded through the platform UI (or the Catalog API). + +### Cause 2: helm push to Harbor bypasses the platform + +A version pushed with `helm push` (or any client that talks only to the registry) goes straight to Harbor, so Catalog does not learn about the tag. ChartRepo synchronization runs when a template is uploaded through Catalog, and the Catalog page has no manual synchronization button. Pushing the same tag again with `helm push` does not help. + +### Cause 3: an added OCI chart repository does not import the charts already in it + +After an OCI chart repository is added, the `ChartRepo` reaches phase `Synced`, and `captain-chartmuseum` logs only `Rebuilding index for tenant`, `Fetching chart list from storage`, and `No change detected between cache and storage`, but no `Artifact`/`ArtifactVersion` pair exists for the charts. The platform has no function that copies charts already stored in a newly added OCI repository into Catalog; a synced phase does not create those objects. The result is the same whether the repository address is the Harbor host only or the host plus the project path. + +### Cause 4: the chart repository did not resynchronize after an accepted upload + +The upload succeeded, the architecture of the package matches the cluster, but the repository index did not refresh: last sync time is `-`, and the template list and the version selector of the create form still offer only the previous version. Setting the repository status to `Pending` triggers a resynchronization, after which the new version is available. + +### Cause 5: the chart's hardware architecture does not include the cluster's architecture + +Application templates support two hardware architectures, x86 and ARM, and default to x86. A cluster does not display chart packages whose architecture does not match its own: an x86-only template is listed on x86 clusters (and on C86 clusters), not on ARM clusters. Charts uploaded to the platform chart repository are marked x86 only by default. The chart details page under Platform Management shows the value as Hardware Architecture `x86`. + +The value is the `cpaas.io/arch` label of the platform `Chart` resource (`app.alauda.io/v1beta1`, in `cpaas-system` on the `global` cluster), taken from the package's `Chart.yaml`: `cpaas.io/arch: amd64` is stored as x86, `cpaas.io/arch: amd64.arm64` as x86 and ARM. Node architectures are reported as `amd64` / `arm64`; the template values are `x86` / `arm`. The mismatch shows up three ways: + +- The chart is not listed for a namespace of an ARM cluster. +- The chart details load on a cluster of another architecture, but the version picker is empty. On an ARM cluster, changing the version's label from `x86` to `arm` makes the versions selectable. +- A new upload of a package whose `cpaas.io/arch` is `amd64` only writes the label back to x86, even if the chart had been set to x86 and ARM. The chart then looks deleted from ARM clusters, although `kubectl get chart -A` still lists it (named `--.`) with the re-uploaded version. Deleting the version is not what removed it; the label written by the new package is. + +Restarting `captain` components does not make the chart appear. + +## Resolution + +### Cause 1 and Cause 2 + +Upload the chart through the platform: + +1. Open Catalog → Helm Charts, choose Add Template, and select a template repository of type Chart or OCI Chart that you can manage. +2. Upload the `.tgz`. For an OCI Chart repository, also set the template directory (the chart storage directory). +3. When the row leaves `Uploading` and shows upload successful, the chart is listed, and the upload also triggers ChartRepo synchronization so versions already stored in Harbor appear. + +The same upload through the Catalog API, with an API token from the user profile (Add API Token): + +```bash +curl -k --request POST \ + --url "https:///catalog/v1/chartrepos/cpaas-system//charts" \ + --header "Authorization: Bearer " \ + --data-binary @"-.tgz" +``` + +`` is the ChartRepo name in `cpaas-system`, not the Harbor project name. Use a repository whose `status.phase` is already synced; fix a failed repository first. + +### Cause 3 + +Adding the repository does not import existing charts. Upload each chart that should appear through Catalog → Helm Charts or the Catalog API, as in Cause 1, so the platform registers it. Confirm the state before and after: + +```bash +kubectl get chartrepos.app.alauda.io -A +kubectl get chartrepo -n cpaas-system -o jsonpath='type={.spec.type} phase={.status.phase} reason={.status.reason}{"\n"}' +kubectl get artifact,artifactversion -n cpaas-system +``` + +### Cause 4 + +1. Rule out a reused Artifact first: if the `violet push` output says `artifact already exist, will create artifactversion`, compare the `cpaas.io/chart-purpose` label of the existing `Artifact` with that of the new `ArtifactVersion`. A mismatch means an `Artifact` of the same name from another chart repository hides the version; setting `Pending` does not fix that. + +2. If the upload itself keeps failing, set `--platform-address` to the platform access address only, without a port or a path such as `/public-charts`: + + ```bash + violet push .tgz --platform-address "https://" --platform-username "" --platform-password "" + ``` + +3. Once the upload has succeeded and the version is still missing, set the ChartRepo status to `Pending`. + + > **Warning:** Only when the upload succeeded, the new version is missing from the dropdown and template selection, and the architecture matches. Impact: synchronization of that ChartRepo is re-queued, Catalog can show it as syncing until the phase settles, and a wrong name resyncs the wrong repository. Precondition: the current object is saved and its phase printed by the first two commands. Rollback: patch `status.phase` back to the printed value. The patch does not delete charts or Harbor blobs. + + ```bash + kubectl get chartrepo -n cpaas-system -o yaml > chartrepo-status.yaml + kubectl get chartrepo -n cpaas-system -o jsonpath='{.status.phase}{"\n"}' + kubectl patch chartrepo -n cpaas-system --subresource=status --type=merge -p '{"status":{"phase":"Pending"}}' + kubectl get chartrepo -n cpaas-system -o jsonpath='{.status.phase}{"\n"}{.status.reason}{"\n"}' + ``` + + If the status patch is rejected, run `kubectl edit chartrepo -n cpaas-system --subresource=status` and set the phase to `Pending`. + +4. After the repository leaves `Pending`, the uploaded version is available in the application templates. + +### Cause 5 + +1. On the `global` cluster, confirm the chart resource exists and read its architecture; on the affected cluster, read the node architecture: + + ```bash + kubectl get chart -A | grep + kubectl get chart -n cpaas-system -o jsonpath='{.metadata.labels.cpaas\.io/arch}{"\n"}' + kubectl get nodes -o custom-columns=NAME:.metadata.name,ARCH:.status.nodeInfo.architecture + ``` + + A label without `arm64` on a chart that an ARM cluster cannot see confirms this cause. + +2. Change the hardware architecture in the UI: Platform Management > App Marketplace Management > Chart Repositories > `` > the chart, and edit Hardware Architecture to include ARM (x86 and ARM). On releases that name the menu App Store Management > Template Repository, the same control is on the application template details page. Keep x86 when x86 or C86 clusters must still see the chart. For an empty version picker on an ARM-only cluster, set the version's architecture to `arm`. + + > **Warning:** Changing the hardware architecture changes which clusters list this chart; a cluster whose architecture you remove stops showing it. Precondition: note the current value on the details page. Rollback: set that value again. No chart object or Harbor blob is deleted. + +3. Reopen Catalog with a namespace of the affected cluster; the chart and its versions are listed. + +4. Make it persistent: the next upload of a package with `cpaas.io/arch: amd64` overwrites the UI change. Set the annotation in `Chart.yaml`, repackage, and upload: + + ```yaml + annotations: + cpaas.io/arch: amd64.arm64 + ``` + +## Diagnostic Steps + +```bash +kubectl get chartrepo -n cpaas-system -o custom-columns=NAME:.metadata.name,TYPE:.spec.type,URL:.spec.url,PHASE:.status.phase,REASON:.status.reason +kubectl get artifact,artifactversion -n cpaas-system +helm show chart --version +``` + +| Check | Conclusion | +| --- | --- | +| ACP 4.2, OCI chart present in Harbor, never uploaded in the UI | Cause 1 | +| The version reached Harbor only through `helm push`; the Catalog list is unchanged | Cause 2 | +| OCI repository phase `Synced`, `captain-chartmuseum` only logs `No change detected between cache and storage`, page says `无应用模板`, no `Artifact` pair in `cpaas-system` on `global` | Cause 3 | +| Upload succeeded, version missing from the dropdown, last sync time `-`, architecture matches | Cause 4 | +| `violet push` says `artifact already exist`, and the `cpaas.io/chart-purpose` labels differ | Reused Artifact (Cause 4, step 1) | +| The chart is hidden only in an ARM cluster namespace (`共 0 个应用`, "0 applications in total") | The chart's architecture label, not synchronization (Cause 5) | +| Chart details load, the version picker is empty, and the node architecture and the `x86` / `arm` label disagree | Cause 5 | +| The chart vanished from Catalog after a re-upload, `kubectl get chart -A` still lists it, its `cpaas.io/arch` label has no `arm64` | Cause 5 | + +On ACP 4.2, Cause 1 and Cause 2 can both apply: there is no background synchronization, and a Catalog upload is also the event that registers the chart. A synced phase together with a missing `Artifact` pair is Cause 3, not Cause 4. Restarting OLM Pods does not fix any of these causes. diff --git a/docs/en/solutions/CephObjectStore_stays_in_Deleting_with_failed_to_list_buckets_no_route_to_host_on_ACP.md b/docs/en/solutions/CephObjectStore_stays_in_Deleting_with_failed_to_list_buckets_no_route_to_host_on_ACP.md new file mode 100644 index 000000000..2108343cf --- /dev/null +++ b/docs/en/solutions/CephObjectStore_stays_in_Deleting_with_failed_to_list_buckets_no_route_to_host_on_ACP.md @@ -0,0 +1,117 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.4.1 +--- + +# CephObjectStore stays in Deleting with failed to list buckets no route to host on ACP + +## Issue + +A Ceph object storage pool on Alauda Container Platform keeps showing "Deleting" (`删除中`) and never goes away, while other object stores on the same cluster stay `Ready`: + +```text +kubectl -n rook-ceph get cephobjectstore +NAME PHASE ENDPOINT AGE + Ready http://rook-ceph-rgw-.rook-ceph.svc:7480 11d + Deleting http://rook-ceph-rgw-.rook-ceph.svc:7481 21h +``` + +The events of the stuck object store repeat every quarter of an hour or so: + +```text +Warning ReconcileFailed rook-ceph-object-controller failed to reconcile CephObjectStore "rook-ceph/". failed to get dependents of CephObjectStore "rook-ceph/": failed to list buckets in CephObjectStore "rook-ceph/": Get "http://rook-ceph-rgw-.rook-ceph.svc:7481/admin/bucket?": dial tcp :7481: connect: no route to host +``` + +`rook-ceph-object-controller` is one of the controllers inside the Rook operator. It has no pod of its own; its messages are in the log of Deployment `rook-ceph-operator` in `rook-ceph`. The RGW of each store runs as pods labelled `app=rook-ceph-rgw,rook_object_store=` behind the ClusterIP Service `rook-ceph-rgw-`, whose port is the gateway port of that store (for example `7480` for the first store and `7481` for the second). + +A whole `CephCluster` in `Deleting` is covered in [Removing Rook-Ceph fails in the UI, or the CephCluster stays in Deleting on ACP](./Removing_Rook-Ceph_fails_in_the_UI_or_the_CephCluster_stays_in_Deleting_on_ACP.md). + +## Root Cause + +### Cause 1: a host firewall blocks the RGW Service port, so the dependents check cannot list buckets + +Before Rook removes a `CephObjectStore`, the object controller checks whether the store still has dependents. Part of that check lists the buckets through the RGW admin API at `http://rook-ceph-rgw-.rook-ceph.svc:/admin/bucket`. When the operator cannot reach that endpoint, the check fails, the reconcile is retried later, and the object store stays in `Deleting`. + +On bare-metal clusters whose hosts run a firewall, the firewall can block the traffic to the stuck store's Service port (`7481` here) while the port of the working store (`7480`) is allowed. A firewall that rejects a packet with an ICMP "host prohibited" reply produces exactly the client-side error `connect: no route to host`, even though the Service and its pods exist. + +## Resolution + +### Cause 1: a host firewall blocks the RGW Service port, so the dependents check cannot list buckets + +Use this only when the events show `failed to list buckets` with `no route to host` (or a timeout) for the store's own RGW Service. + +1. Confirm that the RGW of the store is running and the Service has endpoints: + +```bash +kubectl -n rook-ceph describe cephobjectstore | sed -n '/Events/,$p' +kubectl -n rook-ceph get pod -l app=rook-ceph-rgw,rook_object_store= -o wide +kubectl -n rook-ceph get svc,endpoints rook-ceph-rgw- -o wide +kubectl -n rook-ceph get pod -l app=rook-ceph-operator -o wide +``` + +If the RGW pods are gone and the endpoints are empty, this procedure does not apply; read the operator log for the store first. + +2. Reproduce the failure from the operator and compare with the working store. Any HTTP status code (for example `200` or `403`) means the port is reachable; `No route to host` or a timeout reproduces the problem. + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-operator -- curl -sS -m 5 -o /dev/null -w '%{http_code}\n' http://rook-ceph-rgw-.rook-ceph.svc:/ +kubectl -n rook-ceph exec deploy/rook-ceph-operator -- curl -sS -m 5 -o /dev/null -w '%{http_code}\n' http://rook-ceph-rgw-.rook-ceph.svc:/ +kubectl -n rook-ceph get svc rook-ceph-rgw- -o jsonpath='{.spec.ports[*].port} {.spec.ports[*].targetPort}{"\n"}' +``` + +3. On the node that runs the operator pod and on the nodes that run the store's RGW pods, read the host firewall rules. Look for what lets the working store's port through and what rejects the stuck store's port: + +```bash +systemctl is-active firewalld +firewall-cmd --list-all +iptables -S | grep -iE 'reject|drop' +``` + +> **Warning:** Changing the host firewall changes what can reach the nodes, and a wrong reload can drop other allowed traffic. Precondition: the security owner approves the rule, and the current rules are saved (`firewall-cmd --list-all > firewall-before.txt` or `iptables-save > iptables-before.txt`). Rollback: remove the added rule (`firewall-cmd --permanent --remove-port=/tcp && firewall-cmd --reload`) or restore the saved rules with `iptables-restore < iptables-before.txt`. + +4. Allow the stuck store's Service port and its `targetPort` in the same way the working store is already allowed. With firewalld, for example: + +```bash +firewall-cmd --permanent --zone= --add-port=/tcp +firewall-cmd --reload +``` + +Repeat on every node from step 3, then run the `curl` from step 2 again until it returns an HTTP code. + +5. Wait for the next reconcile and check that the object store leaves `Deleting`: + +```bash +kubectl -n rook-ceph logs deploy/rook-ceph-operator --since=30m | grep -i '' +kubectl -n rook-ceph get cephobjectstore +``` + +6. Once the bucket list succeeds, the dependents check reports what still uses the store. If the event now names buckets or `CephObjectStoreUser` objects, those hold the deletion. List them before removing anything: + +```bash +kubectl -n rook-ceph get cephobjectstoreuser +kubectl get objectbucketclaim -A +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- radosgw-admin bucket list --rgw-realm= --rgw-zonegroup= --rgw-zone= +``` + +> **Warning:** Deleting the users, ObjectBucketClaims, or buckets of the store deletes the objects in them. Precondition: the owners have confirmed that the data of `` is no longer needed or has been copied elsewhere. Rollback: none. + +```bash +kubectl -n rook-ceph delete cephobjectstoreuser +``` + +## Diagnostic Steps + +```bash +kubectl -n rook-ceph get cephobjectstore,cephobjectstoreuser +kubectl -n rook-ceph describe cephobjectstore | sed -n '/Events/,$p' +kubectl -n rook-ceph get svc,endpoints | grep rgw +kubectl -n rook-ceph logs deploy/rook-ceph-operator --tail=200 | grep -i 'object' +``` + +- `ReconcileFailed` with `failed to list buckets` and `connect: no route to host` for the store's own Service, RGW pods running, and another store `Ready` on a different port: Cause 1; check the host firewall for that port. +- No endpoints and no RGW pod for the store: not this cause; read the operator log for the store. +- The event names remaining buckets or users instead of a connection error: the connectivity part is done; handle the dependents as in step 6. diff --git a/docs/en/solutions/Ceph_OSD_crash-loops_with_not_a_block_device_or_an_OSD_has_to_move_to_another_device_class_on_ACP.md b/docs/en/solutions/Ceph_OSD_crash-loops_with_not_a_block_device_or_an_OSD_has_to_move_to_another_device_class_on_ACP.md new file mode 100644 index 000000000..5220e9539 --- /dev/null +++ b/docs/en/solutions/Ceph_OSD_crash-loops_with_not_a_block_device_or_an_OSD_has_to_move_to_another_device_class_on_ACP.md @@ -0,0 +1,107 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# Ceph OSD crash-loops with not a block device, or an OSD has to move to another device class on ACP + +## Issue + +An existing Ceph OSD on Alauda Container Platform is not in the device class where it is needed: + +- An OSD pod is `Init:CrashLoopBackOff` and its log contains `RuntimeError: Error: ["/dev/"] not a block device`. `ceph osd tree` shows that OSD `down` with `REWEIGHT` `0`, and the storage view shows the same disk assigned to TopoLVM. +- An OSD has to move from one device class to another (for example from a CephFS class to an object-storage class). Adding `spec.storage.allowDeviceClassUpdate` to `CephCluster` and editing the OSD `deviceClass` there returns `unknown field "spec.storage.allowDeviceClassUpdate"`. + +Work in the `rook-ceph` namespace; Ceph commands run in `deploy/rook-ceph-tools`. + +Related articles: [A raw disk added to a Ceph device class does not become an OSD on ACP](./A_raw_disk_added_to_a_Ceph_device_class_does_not_become_an_OSD_on_ACP.md), [Ceph storage pool cannot be created or never finishes because the deployment created no OSD on ACP](./Ceph_storage_pool_cannot_be_created_or_never_finishes_because_the_deployment_created_no_OSD_on_ACP.md). + +## Root Cause + +### Cause 1: RuntimeError: Error: ["/dev/"] not a block device + +The disk that the OSD was prepared on has been taken by TopoLVM. Ceph cannot activate an OSD on a path that is no longer the block device it prepared, so the OSD init container fails on every start. The other OSDs stay `up`. + +### Cause 2: unknown field "spec.storage.allowDeviceClassUpdate" + +The `CephCluster` CRD of the installed version does not have `spec.storage.allowDeviceClassUpdate` (confirmed with Ceph 17.2.7, Quincy). The field is rejected and removed, and the OSD stays in the old class: `ceph osd tree` still shows the previous class name. The device class of an existing OSD is a property of the CRUSH map. + +## Resolution + +### Cause 1: RuntimeError: Error: ["/dev/"] not a block device + +Use this only when the OSD log contains `not a block device` and the storage view shows that disk assigned to TopoLVM. Give Ceph a different disk: retire the broken OSD and add a replacement disk. Do not take the disk back from TopoLVM while TopoLVM volumes live on it. + +1. Confirm the signature and the OSD id: + +```bash +kubectl -n rook-ceph get pods -o wide | grep rook-ceph-osd- +kubectl -n rook-ceph logs --all-containers | grep -F 'not a block device' +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd tree +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s +kubectl -n rook-ceph get cephcluster -o yaml > cephcluster-before.yaml +``` + +2. Retire the OSD. + +> **Warning:** `ceph osd out`, `ceph osd crush remove`, `ceph auth del`, and `ceph osd rm` retire this OSD id. Impact: any PG whose only remaining copy was on it loses that data. Precondition: the log signature and the TopoLVM assignment are both confirmed, `ceph pg dump` shows another `up` copy for the PGs that were on this OSD, and `cephcluster-before.yaml` is saved. Rollback: none for `osd rm`; add the replacement disk and let backfill restore redundancy. + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd out osd. +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd crush remove osd. +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph auth del osd. +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd rm osd. +``` + +3. Add a replacement disk to the device class (Cause 2 of [A raw disk added to a Ceph device class does not become an OSD on ACP](./A_raw_disk_added_to_a_Ceph_device_class_does_not_become_an_OSD_on_ACP.md)). It has to be a block device that the storage view does not list under TopoLVM. Remove the old device entry from `spec.storage.nodes` of `CephCluster` so the operator does not try to prepare it again. + +### Cause 2: unknown field "spec.storage.allowDeviceClassUpdate" + +Do not set `allowDeviceClassUpdate` on this CRD. Change the class in the CRUSH map. + +1. Record the current class: + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd crush get-device-class osd. +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd tree +``` + +2. Move the OSD. + +> **Warning:** Removing and setting the device class makes CRUSH remap PGs; the cluster backfills, client IO rises, and a failure during backfill has less spare redundancy. Precondition: `ceph -s` is healthy enough to backfill (no full OSDs), and you saved the old class name. Rollback: run the same two commands with the saved class name. The class change itself does not delete data. + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd crush rm-device-class osd. +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd crush set-device-class osd. +``` + +3. Watch `ceph -s` until all PGs return to `active+clean`. The target pool must use a CRUSH rule that selects ``, or the OSD does not receive that pool's data: + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd crush rule dump +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool ls detail +``` + +## Diagnostic Steps + +```bash +kubectl -n rook-ceph get pods -o wide | grep rook-ceph-osd- +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd tree +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd crush class ls +``` + +On the node of the failing OSD: + +```bash +lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT /dev/ +``` + +- `Init:CrashLoopBackOff`, `not a block device` in the log, and the disk listed under TopoLVM (or `lsblk` shows TopoLVM logical volumes on it): Cause 1. +- `unknown field "spec.storage.allowDeviceClassUpdate"` when editing `CephCluster`, and `ceph osd tree` still shows the old class: Cause 2. diff --git a/docs/en/solutions/Ceph_OSDs_nearfull_backfill_blocked_or_OSD_usage_uneven_on_ACP.md b/docs/en/solutions/Ceph_OSDs_nearfull_backfill_blocked_or_OSD_usage_uneven_on_ACP.md new file mode 100644 index 000000000..0fec07072 --- /dev/null +++ b/docs/en/solutions/Ceph_OSDs_nearfull_backfill_blocked_or_OSD_usage_uneven_on_ACP.md @@ -0,0 +1,223 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.2.x + - 4.3.x +--- + +# Ceph OSDs nearfull, backfill blocked, or OSD usage uneven on ACP + +## Issue + +Ceph distributed storage on Alauda Container Platform raises a capacity or distribution alert, and some OSDs really are fuller than others. Typical shapes: + +- `HEALTH_WARN` with `backfillfull` / `nearfull` OSDs and pools, OSD usage alerts above 75% or 80% for several OSDs of one device class and for the cluster, and an application on Ceph that fails when it restarts. +- `1 nearfull osd(s)` and `2 pool(s) nearfull`, with spare disks already in the servers. +- `PG_BACKFILL_FULL`, `backfill_toofull` PGs, and `nearfull` pools right after a disk was added on the Device Class page. +- A PG-distribution alert: one OSD's PG count differs from the average by more than 30%, the cluster is otherwise usable, and the disks are not the same size. +- A device-class usage alert says an OSD is over 75% while a Ceph upgrade is waiting. +- A small OSD sits near 80% while newly added large OSDs are at 8–10%, and `ceph osd reweight 2` fails with `Invalid command: 2.0 not in range [0.0, 1.0]`. +- An S3 client finds the bucket and then fails with `Unable to execute HTTP request: Read timed out` when setting a bucket policy or CORS rules. + +Environment: toolbox `deploy/rook-ceph-tools` and cluster object `CephCluster` in namespace `rook-ceph`. `ceph osd reweight` takes a value from 0.0 to 1.0. `ceph osd crush reweight` sets the size-based CRUSH weight, which Rook sets from the disk size when the OSD is created. + +## Root Cause + +### Cause 1: the whole device class is full + +`ceph osd df tree` shows the OSDs of one class at similar `%USE` with `VAR` near 1: the class is full, not skewed. `ceph health detail` lists `nearfull` and `backfillfull` OSDs and pools (for example `osd.2` at backfillfull and `osd.5` at nearfull, with pools such as `.mgr`, `cephfs-metadata`, and `cephfs-data0` unable to backfill). Workloads whose PVCs are on that class can fail when they restart. Reweighting moves data inside the class and frees nothing. + +### Cause 2: PG_BACKFILL_FULL after a device add dropped an existing disk from CephCluster + +On the ACP 4.3.1 Device Class page, the page did not show the disks a node already had. Submitting a new disk for that node (for example `sde` next to an existing `sdd`) rewrote `CephCluster` `spec.storage` without the old device. The old OSD kept running, but the CR no longer declared its disk. That OSD then sat near full (for example about 80% with a large omap object) and backfill could not finish. A deep scrub of the affected PGs found no object corruption. + +### Cause 3: ceph_osd_numpg differs from the average by more than 0.3 + +The alert compares `ceph_osd_numpg` with the per-job average and fires above `0.3`. With OSDs of different sizes, the PG counts are naturally skewed against a 30% threshold. Ceph usage itself is normal. This is not the same fault as a nearfull OSD. + +### Cause 4: device-class usage over 75% while a Ceph upgrade is waiting + +An alert says an OSD in a device class (for example `nvme`) is over 75% and advises freeing space or expanding the class. When this happened on ACP 3.18.2 before Ceph itself was upgraded, the Ceph service was disrupted and recovered once capacity was expanded. The upgrade was held until the service recovered. + +### Cause 5: one nearfull OSD and nearfull pools, with spare disks not yet added + +One device class crossed the nearfull threshold while the cluster as a whole was not full. For example, `ceph osd df` showed the six OSDs of a custom class far above the rest (one at `86.68` %USE with `VAR 3.72`), overall usage at `23.29`%, and `MIN/MAX VAR: 0/3.72`; `ceph osd tree` showed every OSD `up` with REWEIGHT `1.00000`. Spare disks were on the nodes but not yet OSDs. + +### Cause 6: a small OSD stays near full after large OSDs were added + +`ceph osd df` shows mixed sizes: small OSDs (for example 1.1 TiB, CRUSH WEIGHT `1.09059`) with one at about 80% (`VAR 3.91`), and new large OSDs (7.3 TiB, CRUSH WEIGHT `7.27689`) at 8–10%. The small OSD held a lot of data historically, and that usage does not come down by itself. The large OSDs already have a CRUSH weight that matches their size. `ceph osd reweight` accepts only 0 to 1, so setting the large OSDs to 2 is not valid: + +```text +Invalid command: 2.0 not in range [0.0, 1.0] +osd reweight : reweight osd to 0.0 < < 1.0 +Error EINVAL: invalid command +``` + +### Cause 7: S3 Read timed out because Ceph usage is at its storage limit + +The bucket exists, but later S3 calls (for example setting a public-read policy or CORS rules) fail with `Unable to execute HTTP request: Read timed out`. Ceph usage had reached the configured storage limit. It is not a bad access key. + +## Resolution + +Save the current values before changing anything: + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd dump | grep -E 'full_ratio|nearfull|backfillfull' +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph config get osd osd_max_backfills +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd df tree +``` + +### Cause 1: the whole device class is full + +1. Check whether the storage nodes have disks that are not yet OSDs (`lsblk` on each node), and how much each pool uses (`ceph df`). +2. Add capacity: add disks to that device class on the existing storage nodes, in a maintenance window, so the new OSDs join the class the full pools use. Or delete business data from the pools on that class. +3. If no disk can be added right away, raise the thresholds temporarily, following the product documentation topic "Setting Ceph OSD Full Thresholds". Raise `backfillfull` above the fullest OSD's usage so backfill can run. + +> **Warning:** Raising `nearfull`, `backfillfull`, and `full` lets Ceph fill disks closer to 100%; an OSD that reaches 100% stops IO and leaves no room to recover. Precondition: the current values are saved, the order `nearfull` < `backfillfull` < `full` is kept, and the increase is as small as possible. Rollback: set the saved values back once capacity is added or data is removed. This is not a substitute for adding disks. + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd set-nearfull-ratio 0.88 +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd set-backfillfull-ratio 0.92 +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd set-full-ratio 0.96 +``` + +To keep the values from being reconciled back, set them in the `CephCluster` CR (`spec.storage.nearFullRatio`, `backfillFullRatio`, `fullRatio`): + +```bash +kubectl patch cephcluster -n rook-ceph --type merge \ + -p '{"spec":{"storage":{"nearFullRatio":0.88,"backfillFullRatio":0.92,"fullRatio":0.96}}}' +``` + +4. Before restarting an affected workload, check that its PVC still exists: + +```bash +kubectl -n get pvc +kubectl -n rollout restart deploy/ +``` + +### Cause 2: PG_BACKFILL_FULL after a device add dropped an existing disk from CephCluster + +If the OSD is still `up` and you have the `CephCluster` YAML from before the submit, put the dropped device path back under that node's `spec.storage.nodes` with `kubectl -n rook-ceph edit cephcluster `. + +If the OSD has to be rebuilt instead: + +> **Warning:** Relaxing the full ratios, raising `osd_max_backfills`, or lowering a CRUSH weight lets Ceph place data closer to a full disk. Precondition: every value you change is saved, and you are in a maintenance window. Rollback: restore the saved values when `ceph -s` is `HEALTH_OK`, including `ceph osd crush reweight osd. `. + +1. Lower the stuck OSD's CRUSH weight (for example to `0.6`) and set temporary values only as high as backfill needs: + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd crush reweight osd. 0.6 +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph config set osd osd_max_backfills +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd set-backfillfull-ratio +``` + +> **Warning:** `ceph osd out`, `crush remove`, `auth del`, and `osd rm` drop this OSD id. PGs whose only remaining copy is on it are lost. Precondition: `ceph pg dump` shows another `up` copy for its PGs, and the `CephCluster` YAML is saved. Rollback: re-add the disk and let backfill copy data back; Ceph does not undo `osd rm`. + +2. Remove the OSD from the toolbox: + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd out osd. +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd crush remove osd. +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph auth del osd. +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd rm osd. +``` + +> **Warning:** `wipefs -af` and `sgdisk --zap-all` erase the disk. Run them on the node, not in the toolbox. Precondition: `lsblk` and the OSD metadata agree this is the disk you just removed, not the OS disk and not another OSD. Rollback: none. + +3. Wipe the disk on the node: + +```bash +wipefs -af /dev/ +sgdisk --zap-all /dev/ +``` + +4. Add the disk again through the device class, including every disk already declared for that node. After the new OSD is up, set its device class to the class the pool's CRUSH rule selects, using the id `ceph osd tree` shows: + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd crush set-device-class osd. +``` + +5. When backfill has caught up and `ceph -s` is `HEALTH_OK`, restore the saved ratios, `osd_max_backfills`, and CRUSH weight. + +### Cause 3: ceph_osd_numpg differs from the average by more than 0.3 + +1. Confirm the skew comes from disk size, not from a stuck balancer or a full OSD: + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd df +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph balancer status +``` + +2. The long-term fix is to use disks of the same size. To stop the unnecessary alert meanwhile, save the alert's current threshold and raise it above the skew `ceph osd df` shows; restore it once the disks match. Find the rule in the console alert editor or with: + +```bash +kubectl get prometheusrule -A -o json | grep -n ceph_osd_numpg +``` + +Changing the threshold only changes when the alert fires; it does not move PGs. + +### Cause 4: device-class usage over 75% while a Ceph upgrade is waiting + +Expand the device class, or free space in it, before starting the Ceph upgrade. After the expansion, check `ceph -s` and `ceph osd df tree` again, and start the upgrade only when Ceph is healthy. + +### Cause 5: one nearfull OSD and nearfull pools, with spare disks not yet added + +1. Find the device class of the nearfull OSDs. New disks relieve only the pools whose CRUSH rule selects the class the new OSDs join: + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd df tree +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd crush class ls +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool ls detail +``` + +2. Add the spare disks to that device class in a maintenance window (for example in the evening), then watch the rebalance with `ceph -s`. If client latency is already high, leave the recovery and backfill speeds as they are; the rebalance finishes on its own. +3. If `ceph -s` shows `noscrub,nodeep-scrub flag(s) set` during the rebalance, scrubbing is paused for the whole cluster. When no PG is `backfilling` or `backfill_wait`, clear the flags: + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd unset noscrub +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd unset nodeep-scrub +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail +``` + +If `nearfull` is still listed for the same OSD afterwards, the new disks did not join the full class, or that OSD needs the override in Cause 6. + +### Cause 6: a small OSD stays near full after large OSDs were added + +Lower the override weight of the hot small OSD. Start mild (for example `0.95`) off-hours; a stronger value such as `0.5` moves much more data. Values above 1 are rejected. + +> **Warning:** `ceph osd reweight` starts a rebalance away from that OSD, and client IO rises until PGs settle. Precondition: the hot OSD is the small historical one, the large OSDs are nearly empty, the current reweight (normally 1) is saved, and you are off-hours. Rollback: `ceph osd reweight `. + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd reweight 0.5 +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s +``` + +When `ceph osd df` shows the OSD well below the danger line and backfill is finished, restore the saved reweight. + +### Cause 7: S3 Read timed out because Ceph usage is at its storage limit + +Confirm that usage is at the configured limit (`ceph df`, `ceph health detail`). Raising the limit temporarily restores service; then add disks and clean up business data so usage stays below it. An `InvalidAccessKeyId` error is a credentials problem, not this cause. + +## Diagnostic Steps + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph health detail +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd df tree +kubectl -n rook-ceph get cephcluster -o yaml > cephcluster-now.yaml +``` + +Read `%USE` and `VAR` per device class; a low overall `%RAW USED` does not rule out one full class. + +- Every OSD of the class evenly high (`VAR` near 1): Cause 1. Add disks; do not reweight. +- `PG_BACKFILL_FULL` and a device path missing from `spec.storage.nodes` while the OSD still exists: Cause 2. +- PG-distribution alert above 0.3 with unequal disk sizes and normal usage: Cause 3. +- Usage alert over 75% before a Ceph upgrade: Cause 4. +- `nearfull osd` / `pools nearfull` with spare disks not yet in the device class: Cause 5. +- One small OSD near full, large OSDs nearly empty, or `Invalid command: 2.0 not in range [0.0, 1.0]`: Cause 6. +- S3 `Read timed out` after the bucket was found, with usage at the limit: Cause 7. diff --git a/docs/en/solutions/Ceph_device_class_page_rejects_a_disk_or_Ceph_does_not_show_the_new_capacity_on_ACP.md b/docs/en/solutions/Ceph_device_class_page_rejects_a_disk_or_Ceph_does_not_show_the_new_capacity_on_ACP.md new file mode 100644 index 000000000..c16901eda --- /dev/null +++ b/docs/en/solutions/Ceph_device_class_page_rejects_a_disk_or_Ceph_does_not_show_the_new_capacity_on_ACP.md @@ -0,0 +1,195 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# Ceph device class page rejects a disk, or Ceph does not show the new capacity on ACP + +## Issue + +A disk is on the node, but Ceph distributed storage on Alauda Container Platform will not take it, or Ceph does not show the size the operating system already shows: + +- Adding a storage node says no devices are available, although `dd if=/dev/zero of=/dev/ bs=4k count=512`, `wipefs -af`, and `sgdisk --zap-all` were already run. The data disk (for example `vdb`) is still missing from the list. +- A pool already exists on three nodes, a fourth node has one raw disk, and the page says `至少需要 3 个节点` ("at least 3 nodes are required"). The question is why expanding Ceph needs three nodes. +- A device class cannot be added for a device on a node that just joined, and the device is USB. +- A newly added disk is listed only on the `CephCluster` resource. +- A virtual disk under an existing OSD was enlarged; the OS shows the new size, but Ceph (or the device class capacity on the distributed storage page) still reports the old capacity, and the OSD cannot be deleted from the interface. +- Deploying a storage cluster stops with a message that the selected disks do not have sufficient identifying information and may lose identification after a reboot. The disks have names from custom udev rules such as `/dev/data1`. + +Environment: namespace `rook-ceph`, cluster object `CephCluster`, operator Deployment `rook-ceph-operator`, toolbox `deploy/rook-ceph-tools`. Adding a disk through the device class page assumes a healthy, unused disk; check for I/O or medium errors first (`dmesg`, `smartctl -H`). + +## Root Cause + +### Cause 1: no devices available after dd, wipefs, and sgdisk + +A disk that still holds an operating system or data is not listed, which is why wiping is the first check. Sometimes a leftover survives the short `dd`, `wipefs -af`, and `sgdisk --zap-all` sequence, and the disk still does not show up. Partitioning the disk with `fdisk` and then running the same wipe again makes it appear. A node restart is not needed for this. + +### Cause 2: at least 3 nodes are required when one new node's disk is added on the pool path + +The three-node check belongs to creating a storage pool. A replicated pool with `size 3` and failure domain `host` needs OSDs on at least three different hosts to place its three copies, so the create path rejects a submission with only one new node. Adding a disk to an existing pool is done from the device class section, on the device class that the existing pool uses; it does not require three new nodes. + +### Cause 3: the new device is USB + +Ceph storage on the platform does not accept USB devices, so the page cannot add a device class for one. Wiping does not make a USB device eligible. + +### Cause 4: the new disk is only in CephCluster, or an expanded disk keeps the old OSD size + +A disk added to existing Ceph storage can appear only in the `CephCluster` resource at first and in the other views after a wait. That is the normal add, not a reason to wipe again. + +When a disk under an existing OSD is enlarged in the virtualization platform, the OSD does not use the new space by itself. Restarting the OSD pod can make the expanded size display. If it does not, BlueStore itself has to be expanded onto the larger device before the device class capacity changes. Growing the disk under an existing OSD is not the recommended way to add capacity; adding a new OSD is. + +### Cause 5: selected disks do not have sufficient identifying information + +The deployment-phase disk check (confirmed on ACP 4.2.0) requires every selected disk to have a persistent `/dev/disk/by-id/` link. A disk without one is rejected. A name created by a custom udev rule (for example `/dev/data1`) is not a by-id link and does not satisfy the check. The check is about stable identification; filesystems, `/etc/fstab`, `UUID=`, or `LABEL` are not relevant for a Ceph raw disk. + +A disk is offered for selection when its type is `disk` or `mpath`, no filesystem is detected, and its capacity is larger than 10 GiB; `rbd`, `nbd`, and `dm-*` devices are not listed. Whole raw disks are recommended over partitions. + +## Resolution + +### Cause 1: no devices available after dd, wipefs, and sgdisk + +Use this only when the target is an unused data disk and the short wipe has already been run without the disk appearing. + +```bash +lsblk -o NAME,SIZE,TYPE,FSTYPE,LABEL,MOUNTPOINT /dev/ +``` + +> **Warning:** `fdisk` writes a partition table, and the following `dd`, `wipefs -af`, and `sgdisk --zap-all` destroy it and the start of the disk; any data on `/dev/` is gone. Precondition: this is the data disk the page refuses, not the OS disk, not mounted, and not backing an `up` OSD. Rollback: none. + +```bash +fdisk /dev/ +dd if=/dev/zero of=/dev/ bs=4k count=512 +wipefs -af /dev/ +sgdisk --zap-all /dev/ +``` + +Then add the storage node again from the page. + +### Cause 2: at least 3 nodes are required when one new node's disk is added on the pool path + +1. Leave the pool-create wizard. Find the device class the existing pool uses: + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool ls detail +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd crush rule dump +``` + +2. Save `CephCluster` before any submit on this page; on ACP 4.3.1 a submit could drop the disks a node already had from `spec.storage`: + +```bash +kubectl -n rook-ceph get cephcluster -o yaml > cephcluster-before.yaml +``` + +3. In the console, open **Storage Management** (`存储管理`) > **Distributed Storage** (`分布式存储`) > the cluster (for example `ceph-cluster`) > **Device Class** (`设备类`) tab. +4. Select the device class the pool uses (for example `hdd`) and click **Add Device** (`添加设备`). +5. In the **Input Device** (`输入设备`) dialog, pick the **Node** (`节点`) and the **Specified Disk** (`指定硬盘`), for example `vdb`. Use **Add Node** (`添加 节点`) for more nodes. +6. After the submit, check that the old disks are still under `spec.storage.nodes` and that the new OSD appears in `ceph osd tree`. The rebalance after the add generates backfill traffic; do it in a maintenance window. + +If the Add Device dialog of the device class itself shows `至少需要 3 个节点`, collect the dialog input and `cephcluster-before.yaml` and contact support. + +### Cause 3: the new device is USB + +```bash +lsblk -o NAME,SIZE,TRAN,TYPE,MODEL /dev/ +``` + +`TRAN` of `usb` is the signature. Use a non-USB disk; no restart or wipe changes the result. + +### Cause 4: the new disk is only in CephCluster, or an expanded disk keeps the old OSD size + +If the new disk is only in `CephCluster`, wait, then read `ceph osd tree`. If it never becomes an OSD, read the OSD prepare log. + +If the OS shows a larger size for a disk that backs an existing OSD and Ceph still reports the old capacity, restart that OSD first. + +> **Warning:** Restarting `rook-ceph-osd-` stops that OSD until the new pod is up, and its PGs peer again. Precondition: the OS size and the Ceph size disagree for this id, the other replicas are `up`, and you are not deleting the OSD. Rollback: none needed; if the new pod fails, read its log before any wipe. + +```bash +kubectl -n rook-ceph rollout restart deploy/rook-ceph-osd- +kubectl -n rook-ceph rollout status deploy/rook-ceph-osd- +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd df +``` + +If `ceph osd df` still shows the old `SIZE`, expand BlueStore with `ceph-bluestore-tool bluefs-bdev-expand`, which runs against the OSD data directory while the `ceph-osd` process is stopped. The steps below are general Rook practice: keep the OSD pod (its init containers prepare `/var/lib/ceph/osd/ceph-`) but replace the daemon with a sleep. Compare the sizes first, and check whether the OSD sits on an LVM volume: + +```bash +lsblk -o NAME,SIZE,TYPE /dev/ +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd metadata | grep -E 'bluestore_bdev_size|devices' +``` + +> **Warning:** The steps below stop `osd.` for the duration and pause the Rook operator; PGs on that OSD run degraded and no Rook reconcile happens meanwhile. Precondition: the OS shows the larger device, the restart did not change `ceph osd df`, every other OSD is `up`, no PG depends only on this OSD, and you are in a maintenance window. Rollback: `kubectl -n rook-ceph rollout undo deploy/rook-ceph-osd-`, then scale the operator back to 1 and unset `noout`. A completed expansion is not undone. + +```bash +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd set noout +kubectl -n rook-ceph scale deploy rook-ceph-operator --replicas=0 +kubectl -n rook-ceph get deploy rook-ceph-osd- -o yaml > rook-ceph-osd--before.yaml +kubectl -n rook-ceph edit deploy rook-ceph-osd- +``` + +In the editor, in the container named `osd`, set `command: ["sleep", "infinity"]`, delete its `args`, and delete its `livenessProbe` and `startupProbe` if present. Save, wait for the new pod, and confirm the data directory has a `block` link. If it is empty, roll back and contact support instead of continuing. + +```bash +kubectl -n rook-ceph rollout status deploy/rook-ceph-osd- +kubectl -n rook-ceph exec deploy/rook-ceph-osd- -c osd -- ls -l /var/lib/ceph/osd/ceph-/ +``` + +If `lsblk` showed a `ceph--*` logical volume on the disk, grow the physical volume and the logical volume on the node first; `bluefs-bdev-expand` can only use space the `block` device already has: + +```bash +pvresize /dev/ +lvextend -l +100%FREE /dev// +``` + +Run the expansion, then restore the daemon and the operator: + +```bash +kubectl -n rook-ceph exec -it deploy/rook-ceph-osd- -c osd -- ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph- +kubectl -n rook-ceph rollout undo deploy/rook-ceph-osd- +kubectl -n rook-ceph rollout status deploy/rook-ceph-osd- +kubectl -n rook-ceph scale deploy rook-ceph-operator --replicas=1 +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd unset noout +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd df +``` + +`SIZE` for the id and the device class capacity on the page now show the larger value. Repeat one OSD at a time, waiting for `active+clean` in between. For the next expansion, add new OSD disks instead of enlarging existing ones. + +### Cause 5: selected disks do not have sufficient identifying information + +1. On each storage node, check whether the disk has a by-id link and which serial or WWN it exposes: + +```bash +lsblk -o NAME,TYPE,SIZE,FSTYPE,SERIAL,WWN +ls -l /dev/disk/by-id/ | grep -w +``` + +A disk that is the target of some `/dev/disk/by-id/` link passes the check; a disk with no link fails it, whatever udev alias it has. + +2. Make the hypervisor or storage expose a stable serial or WWN for the virtual disk so that udev creates the by-id link. Check the `ls` output rather than assuming a hypervisor setting (such as disk UUIDs on ESXi) already does it. Then select the disk by its kernel name in the wizard. +3. Prepare each disk as a whole raw disk: not partitioned, no filesystem, not mounted, not in `/etc/fstab`. + +> **Warning:** `wipefs -af` and `sgdisk --zap-all` erase the signatures and partition table of `/dev/`. Precondition: `lsblk` confirms this is the data disk intended for Ceph, not mounted, and not the OS disk. Rollback: none. + +```bash +wipefs -af /dev/ +sgdisk --zap-all /dev/ +``` + +## Diagnostic Steps + +```bash +lsblk -o NAME,SIZE,TRAN,TYPE,FSTYPE,MOUNTPOINT +kubectl -n rook-ceph get cephcluster -o yaml | grep -n device +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd df +``` + +- No device available after the short wipe: Cause 1. +- `至少需要 3 个节点` while adding one disk on a new node to an existing pool: Cause 2. +- `lsblk` `TRAN` is `usb`: Cause 3. +- A disk only in `CephCluster`, or Ceph capacity stale after a virtualization grow: Cause 4. +- The wizard says the disks lack sufficient identifying information, and `ls -l /dev/disk/by-id/` has no link for the disk: Cause 5. +- `dmesg` shows I/O or medium errors, or `smartctl -H` fails: a disk health problem, not a page problem. diff --git a/docs/en/solutions/Ceph_operator_install_returns_404_or_the_storage_page_is_empty_because_rook-ceph-tools_never_starts_on_ACP.md b/docs/en/solutions/Ceph_operator_install_returns_404_or_the_storage_page_is_empty_because_rook-ceph-tools_never_starts_on_ACP.md new file mode 100644 index 000000000..6ca197257 --- /dev/null +++ b/docs/en/solutions/Ceph_operator_install_returns_404_or_the_storage_page_is_empty_because_rook-ceph-tools_never_starts_on_ACP.md @@ -0,0 +1,156 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.0.x + - 4.3.x +--- + +# Ceph operator install returns 404, or the storage page is empty because rook-ceph-tools never starts on ACP + +## Issue + +Ceph distributed storage on Alauda Container Platform shows "no Ceph operator" or an empty distributed-storage page, in one of these shapes: + +- Installing the Ceph operator returns 404, the operator catalog page has nothing to install, and `kubectl get packagemanifests -A` prints no rows. +- A `CephCluster` exists and the operator is up, but the storage page does not list the cluster and the device class is empty. `rook-ceph-tools` is not Running, and its container spec has `tty: true`. +- Storage-pool installation runs for a long time, and `ceph-cluster` stays `HEALTH_WARN` with `TOO_FEW_OSDS` (`OSD count 0 < osd_pool_default_size 3`), often together with `PG_AVAILABILITY` (`Reduced data availability: 2 pgs inactive`) and `MDS_SLOW_METADATA_IO`, while the tools pod has `tty: true`. +- On ACP 3.18 the `rook-ceph-operator` Deployment disappears or comes back, and the pod fails at startup with `C | rookcmd: failed to get pod: pods "rook-ceph-operator-" is forbidden: User "system:serviceaccount:rook-ceph:rook-ceph-system" cannot get resource "pods" in API group "" in the namespace "rook-ceph"`. Creating the operator from the Distributed Storage page fails with the same error. + +Environment: Rook namespace `rook-ceph`, toolbox Deployment `rook-ceph-tools`, operator Deployment `rook-ceph-operator`. On ACP 4.3 the OLM components `packageserver`, `catalog-operator`, and `olm-operator` run in `cpaas-system`. On ACP 4.3.4 the CoreDNS Corefile is ConfigMap `kube-system/cpaas-coredns`, data key `Corefile`. + +Related articles: [Ceph storage pool cannot be created or never finishes because the deployment created no OSD on ACP](./Ceph_storage_pool_cannot_be_created_or_never_finishes_because_the_deployment_created_no_OSD_on_ACP.md) (the tools pod is Running and still no OSD exists), [A raw disk added to a Ceph device class does not become an OSD on ACP](./A_raw_disk_added_to_a_Ceph_device_class_does_not_become_an_OSD_on_ACP.md). + +## Root Cause + +### Cause 1: packagemanifests empty because packageserver cannot resolve a name it needs + +`packageserver` logs show errors, and from inside that pod a DNS name it needs does not resolve. Without packagemanifests, the catalog has no Ceph operator to install and the install API answers 404. The operator Deployment is absent because OLM never got a package, not because `rook-ceph-operator` crashed. Adding the missing resolution entries to CoreDNS let the operator install. The name that fails depends on the environment. + +### Cause 2: rook-ceph-tools pod rejected because tty is true + +The tools container spec contains `tty: true`. A security policy or a host security component on the cluster refuses or terminates a container with a TTY, so `rook-ceph-tools` never becomes Ready. The distributed-storage page loads the cluster through the tools pod. While it is down, the page shows no cluster and the device class looks empty even when `CephCluster` is fine and OSD pods are Running, and storage-pool installation cannot finish, leaving the OSD count at 0. Redeploying the `CephCluster` does not help, and neither does uninstalling the operator (its objects in `app-storage` and `rook-ceph` remain). + +The tools Deployment is rendered from the operator's CSV. Changing `tty` in the CSV prevents the next reconcile from putting `tty: true` back, but a CSV edit alone may not roll out to the running Deployment; patching the live Deployment is what brings the pod up. Check both objects. + +### Cause 3: rook-ceph-system cannot get pods on ACP 3.18 + +The operator is OLM-managed: its Deployment carries `olm.managed: "true"` and `olm.owner.kind: ClusterServiceVersion`, with Subscription `rook-operator` in `rook-ceph`. The Deployment and the RBAC it depends on went missing, so the recreated pod runs without permission to get pods. Adding roles and service accounts by hand starts the pod, but the platform still does not see the Ceph cluster. Deleting the Subscription and the CSV and redeploying `ceph-operator` restores it. Why the Deployment and its RBAC went missing is not known. + +## Resolution + +### Cause 1: packagemanifests empty because packageserver cannot resolve a name it needs + +1. Confirm the failure is DNS inside `packageserver`, and save CoreDNS before editing it. If `cpaas-coredns` does not exist on your version, list `kubectl -n kube-system get configmap` and use the ConfigMap that holds the `Corefile` key. + +```bash +kubectl get packagemanifests -A +kubectl -n cpaas-system logs deploy/packageserver --tail=200 +kubectl -n cpaas-system exec deploy/packageserver -- cat /etc/resolv.conf +kubectl -n kube-system get configmap cpaas-coredns -o yaml > cpaas-coredns.backup.yaml +``` + +2. Add a `hosts` entry for the name that failed, with an address that is valid on this cluster. + +> **Warning:** A bad Corefile makes CoreDNS fail and every in-cluster lookup fail, including Kubernetes service DNS. Precondition: `cpaas-coredns.backup.yaml` exists and has been read back, and you only add a `hosts` record for the name `packageserver` could not resolve. Rollback: `kubectl -n kube-system apply -f cpaas-coredns.backup.yaml` and wait until the CoreDNS pods are Ready. + +```bash +kubectl -n kube-system edit configmap cpaas-coredns +``` + +Inside the `Corefile` value, a block of this shape is enough (replace the documentation address and the name): + +```text +hosts { + 192.0.2.10 + fallthrough +} +``` + +Keep the existing `forward` and zone blocks. CoreDNS reloads the ConfigMap. + +3. If `packagemanifests` is still empty after lookups succeed, restart `packageserver`. + +> **Warning:** Restarting `packageserver` interrupts operator catalog reads for about a minute. It does not delete Subscriptions or Ceph data. Precondition: CoreDNS pods are Ready and the name resolves. Rollback: none needed. + +```bash +kubectl -n cpaas-system rollout restart deploy/packageserver +kubectl get packagemanifests -A +``` + +When the Ceph package is listed, install it again from the console. + +### Cause 2: rook-ceph-tools pod rejected because tty is true + +1. Show which container has `tty` set, and read the rejection: + +```bash +kubectl -n rook-ceph get deploy rook-ceph-tools -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" tty="}{.tty}{"\n"}{end}' +kubectl -n rook-ceph describe deploy rook-ceph-tools +kubectl -n rook-ceph get pods -l app=rook-ceph-tools -o wide +kubectl -n rook-ceph get csv | grep rook-ceph +kubectl -n rook-ceph get csv -o yaml | grep -n -5 tty +``` + +2. Set the tools container `tty` to `false` in the CSV (the container under the CSV's `rook-ceph-tools` Deployment spec), so reconciles stop rendering `tty: true`: + +```bash +kubectl -n rook-ceph get csv -o yaml > rook-ceph-csv-before.yaml +kubectl -n rook-ceph edit csv +``` + +3. If the Deployment still shows `tty: true`, patch it. + +> **Warning:** Patching `rook-ceph-tools` recreates the toolbox pod; `ceph` commands from the toolbox fail until the new pod is Ready, while OSDs and client IO keep running. Precondition: container index 0 is the one whose `tty` printed `true` (change the index if not). Rollback: set `tty` back with the same patch only if the security policy allows it. + +```bash +kubectl -n rook-ceph patch deploy rook-ceph-tools --type=json \ + -p='[{"op":"replace","path":"/spec/template/spec/containers/0/tty","value":false}]' +kubectl -n rook-ceph rollout status deploy/rook-ceph-tools +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s +``` + +The storage page lists the cluster again once `ceph -s` works from that pod, and pool installation proceeds past the `OSD count 0` warning that the missing tools pod caused. If `MDS_SLOW_METADATA_IO` remains after the tools pod is up, it is a different fault. + +### Cause 3: rook-ceph-system cannot get pods on ACP 3.18 + +Use this only when the operator log contains `cannot get resource "pods"` for `system:serviceaccount:rook-ceph:rook-ceph-system` in `rook-ceph`. + +```bash +kubectl get subs -A | grep -i -E 'rook|ceph' +kubectl get csv -A | grep -i ceph +kubectl -n rook-ceph get deploy rook-ceph-operator -o jsonpath='{.metadata.labels}{"\n"}' +kubectl -n rook-ceph logs deploy/rook-ceph-operator --tail=50 +``` + +> **Warning:** Deleting the Subscription and the CSV makes OLM remove the operator Deployment and the RBAC it owns; no Rook reconcile happens until the operator is redeployed. Running MON, OSD, and RGW pods are not deleted by this step, and the `CephCluster` resource stays. Precondition: the forbidden error above is in the log, the names come from the `get subs` and `get csv` output for the Ceph operator in `rook-ceph`, and both objects are saved. Rollback: reinstall the operator from the Distributed Storage page or the operator catalog. + +```bash +kubectl -n rook-ceph get subs rook-operator -o yaml > rook-operator-subs-before.yaml +kubectl -n rook-ceph get csv -o yaml > rook-ceph-csv-before.yaml +kubectl -n rook-ceph delete subs rook-operator +kubectl -n rook-ceph delete csv +``` + +Then redeploy `ceph-operator` from the Distributed Storage page (or the catalog path that installed it). When the new pod is Running, confirm that the platform lists the Ceph cluster again: + +```bash +kubectl -n rook-ceph get pods | grep operator +kubectl -n rook-ceph get cephcluster +``` + +## Diagnostic Steps + +```bash +kubectl get packagemanifests -A +kubectl -n rook-ceph get deploy,pods | grep -E 'operator|tools' +kubectl -n rook-ceph get cephcluster +``` + +- No packagemanifests and no `rook-ceph-operator`: Cause 1. Read the `packageserver` logs in `cpaas-system` for lookup errors, then compare Services and Endpoints for the name that failed (`kubectl -n cpaas-system get svc,endpoints`). +- Operator and `CephCluster` present, tools pod not Ready, `tty=true`, or pool installation on `HEALTH_WARN` with `TOO_FEW_OSDS` while the tools pod has `tty: true`: Cause 2. `kubectl -n rook-ceph describe pod -l app=rook-ceph-tools` shows the policy or security-component rejection. +- Operator pod forbidden to get pods as `rook-ceph-system` on ACP 3.18 (`rookcmd: failed to get pod ... is forbidden`): Cause 3. A hand-added Role that only makes the pod start is not sufficient. +- Tools pod Running and still no OSD, or an empty device class: a disk-prepare or device-name problem; see the related article on deployments that created no OSD. diff --git a/docs/en/solutions/Ceph_storage_pool_cannot_be_created_or_never_finishes_because_the_deployment_created_no_OSD_on_ACP.md b/docs/en/solutions/Ceph_storage_pool_cannot_be_created_or_never_finishes_because_the_deployment_created_no_OSD_on_ACP.md new file mode 100644 index 000000000..fc4699316 --- /dev/null +++ b/docs/en/solutions/Ceph_storage_pool_cannot_be_created_or_never_finishes_because_the_deployment_created_no_OSD_on_ACP.md @@ -0,0 +1,135 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# Ceph storage pool cannot be created or never finishes because the deployment created no OSD on ACP + +## Issue + +A new Ceph distributed storage cluster on Alauda Container Platform is deployed, but no OSD is created, so the storage pool cannot be built on it: + +- In step 3 of the distributed storage wizard (create storage pool), the device class dropdown shows `无数据` ("no data") for the class created in step 2. +- Creating a block storage pool does not complete, and some deployment options show no output. +- `CephCluster` is `Ready` with `HEALTH_WARN`, the `rook-ceph-osd-prepare-*` pods are `Completed`, and there is no `rook-ceph-osd-` pod. `kubectl describe cephcluster` lists versions only for `mgr` and `mon`, with no OSD entry. + +With no OSD there is no CRUSH device class, so the pool form has nothing to offer, and a replicated pool cannot place its copies. + +Work in the `rook-ceph` namespace (`CephCluster`, operator Deployment `rook-ceph-operator`, toolbox Deployment `rook-ceph-tools`). + +Related articles: [A raw disk added to a Ceph device class does not become an OSD on ACP](./A_raw_disk_added_to_a_Ceph_device_class_does_not_become_an_OSD_on_ACP.md) (disks added to an existing cluster), [Ceph OSD crash-loops with not a block device, or an OSD has to move to another device class on ACP](./Ceph_OSD_crash-loops_with_not_a_block_device_or_an_OSD_has_to_move_to_another_device_class_on_ACP.md), [Ceph operator install returns 404, or the storage page is empty because rook-ceph-tools never starts on ACP](./Ceph_operator_install_returns_404_or_the_storage_page_is_empty_because_rook-ceph-tools_never_starts_on_ACP.md) (the tools pod is not Running). + +## Root Cause + +### Cause 1: leftover data on the disks, so prepare completes without creating an OSD + +The disks selected for the cluster still carry data from an earlier use. Prepare runs, skips them, and finishes as `Completed` without creating an OSD. This is confirmed on ACP 4.3.3 (Ceph 19.2.5), where cleaning the disks resolved it. A `ReconcileFailed` event `failed to add finalizer ... the object has been modified` next to `ReconcileSucceeded` can appear at the same time; it is not the cause. + +### Cause 2: the device name was entered with extra quotes + +The disk name was typed with extra quotes during deployment, so `CephCluster` records it literally, for example: + +```yaml +storage: + useAllDevices: false + nodes: + - name: + devices: + - name: '"vde"' +``` + +The device name is `"vde"` (with the double quotes), which matches no device on the node. Storage initialization fails because no OSD is created on `vde`. + +## Resolution + +### Cause 1: leftover data on the disks, so prepare completes without creating an OSD + +1. Confirm that no OSD was created and the tools pod is Running: + +```bash +kubectl -n rook-ceph get pods | grep -E 'rook-ceph-osd-|rook-ceph-tools' +kubectl -n rook-ceph describe cephcluster | grep -A12 -i versions +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s +kubectl -n rook-ceph logs | grep -i -E 'skipping|available' +``` + +Only `rook-ceph-osd-prepare-*` pods in `Completed`, no `rook-ceph-osd-` pod, and no OSD in `ceph -s` match this cause. If the tools pod is not Running, use the related article on the tools pod instead. + +2. On each disk node, confirm the target is an unused data disk, and find the CSI pod running on that node: + +```bash +lsblk -o NAME,SIZE,TYPE,FSTYPE,LABEL,MOUNTPOINT /dev/ +kubectl -n rook-ceph get pods -o wide | grep -i csi +``` + +3. Zap the disk from inside that CSI pod. + +> **Warning:** `ceph-bluestore-tool zap-device` erases BlueStore labels and signatures on `/dev/`. Irreversible. Precondition: only when step 1 matches, `lsblk` shows this is the disk you intend to give Ceph, it is not mounted, it is not the OS disk, and it does not back an `up` OSD. Rollback: none. + +```bash +kubectl -n rook-ceph exec -- ceph-bluestore-tool zap-device --dev /dev/ +``` + +If `exec` reports that a container must be chosen, rerun with `-c `. The signature wipe on the node in Cause 1 of [A raw disk added to a Ceph device class does not become an OSD on ACP](./A_raw_disk_added_to_a_Ceph_device_class_does_not_become_an_OSD_on_ACP.md) is the alternative when the tool is not available in the pod. + +4. If no OSD pod appears after the disks are clean, restart the operator so prepare runs again. + +> **Warning:** Restarting `rook-ceph-operator` restarts only the operator pod; prepare and reconcile pause until it is Ready. Precondition: the disks are clean. Rollback: none needed. + +```bash +kubectl -n rook-ceph rollout restart deploy/rook-ceph-operator +kubectl -n rook-ceph get pods | grep -E 'rook-ceph-osd-' +``` + +5. When OSD pods are Running, the device class appears in the pool wizard; create the pool again. + +### Cause 2: the device name was entered with extra quotes + +1. Look at the device names exactly as stored: + +```bash +kubectl -n rook-ceph get cephcluster -o jsonpath='{range .spec.storage.nodes[*]}{.name}{": "}{.devices}{"\n"}{end}' +``` + +A name such as `"\"vde\""` or one that contains `'` or `"` is this cause. + +> **Warning:** Editing `spec.storage` makes the operator prepare the devices named there. A wrong name can point prepare at a different disk. Precondition: you saved the object, and the corrected name is the free data disk that `lsblk` shows on that node. Rollback: apply the saved device entries with `kubectl -n rook-ceph edit cephcluster `. + +2. Correct the names: + +```bash +kubectl -n rook-ceph get cephcluster -o yaml > cephcluster-before.yaml +kubectl -n rook-ceph edit cephcluster +``` + +Under `spec.storage.nodes[].devices`, set each name to the bare device name, for example `- name: vde`, on every node that has the wrong value. + +3. Watch prepare create the OSDs and the pool finish: + +```bash +kubectl -n rook-ceph get pods | grep -E 'rook-ceph-osd-' +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd tree +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s +``` + +When entering disk names in the wizard, type the bare name as `lsblk` shows it (`vde`), without quotes. The `?` help icon next to each field on the create page describes what the field expects. + +## Diagnostic Steps + +```bash +kubectl -n rook-ceph get cephcluster +kubectl -n rook-ceph get pods -o wide | grep -E 'osd|tools' +kubectl -n rook-ceph get cephcluster -o yaml | grep -n -A6 'nodes:' +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s +``` + +- Prepare pods `Completed`, no OSD pod, device names correct, and `lsblk -f` or the prepare log shows the disks still hold data: Cause 1. +- Device names in `spec.storage.nodes[].devices` wrapped in extra quotes: Cause 2. +- `rook-ceph-tools` not Running, or `tty: true` on its container: see the related article on the tools pod. +- An existing cluster where a newly added disk does not become an OSD: see the related article on raw disks added to a device class. diff --git a/docs/en/solutions/Certificate_alerts_from_a_custom_VMRule_ait.alauda.io_webhook_denials_or_by-design_short-lived_certificates_on_ACP.md b/docs/en/solutions/Certificate_alerts_from_a_custom_VMRule_ait.alauda.io_webhook_denials_or_by-design_short-lived_certificates_on_ACP.md new file mode 100644 index 000000000..c568cd038 --- /dev/null +++ b/docs/en/solutions/Certificate_alerts_from_a_custom_VMRule_ait.alauda.io_webhook_denials_or_by-design_short-lived_certificates_on_ACP.md @@ -0,0 +1,177 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.0" + - "4.2" + - "4.3" +--- + +# Certificate alerts from a custom VMRule, ait.alauda.io webhook denials, or by-design short-lived certificates on ACP + +## Issue + +A certificate-related alert keeps firing although no platform certificate is actually expired. One of the following is seen: + +- The alert `平台组件证书已到期 <= 0d 且持续 1 分钟` ("platform component certificates have expired <= 0d and persisted for 1 minute") stays active after a minor upgrade. `openssl` shows the platform certificates are valid, AlertRule `cpaas-certificates-rule` has already been deleted, and restarting the monitoring components does not stop the alert. +- kube-apiserver logs cert-manager webhook denials that reference `{cpaas-ca ClusterIssuer ait.alauda.io}`, or the custom alert "apiserver read request success rate < 0.9" fires on a workload cluster and most failed reads are `403`. +- A cluster with KubeVirt virtualization always has one certificate with one day of validity left, so the "about to expire" alert never clears. +- An external monitor such as Zabbix reports Pod `node-local-certs-exporter-` in `cpaas-system` as not healthy (or failed after a scale-out), and `kubectl get pod -A` cannot find the Pod. + +Certificate expiry metrics (`certificate_expires_time`) are produced by `cluster-enhancer-apiserver`; the certificate page and alert status are served through `courier` on the `global` cluster. + +Related articles: *Certificate page empty or stale because of clock drift or a stale cluster-enhancer-apiserver metric on ACP* (the page is empty, or still shows a certificate that was renewed or removed). For a certificate that really is about to expire, see [Operator service certificates such as packageserver-service-cert about to expire or not renewed on ACP](Operator_service_certificates_such_as_packageserver-service-cert_about_to_expire_or_not_renewed_on_ACP.md). On ACP 4.1, an alert detail that shows no data because `cpaas-certificates-rule` queries time out is Cause 3 of [Certificates tab missing, platform domain certificate not listed, or certificate alert shows no data on ACP](Certificates_tab_missing_platform_domain_certificate_not_listed_or_certificate_alert_shows_no_data_on_ACP.md). + +## Root Cause + +### Cause 1: a custom VMRule still emits cpaas-certificates-rule after the AlertRule is gone + +On a VictoriaMetrics cluster, `vmalert` evaluates the rules from `VMRule` objects, which are rendered into the rule-file ConfigMaps mounted into the `vmalert` Pod. When a custom `VMRule` (one that is not generated from the platform `AlertRule`) carries the `cpaas-certificates-rule` rules, deleting AlertRule `cpaas-certificates-rule` does not remove them: the rule-file ConfigMap still contains the rules, `vmalert` keeps producing the alert, and `courier-api` keeps logging it as coming from `vmalert`. Restarting `enhancer-api`, the monitoring controller, `vmalert`, `alertmanager`, or `courier-api` does not change this. Deleting the custom `VMRule` clears the alert. + +The same symptom also appears when the AlertRule delete itself never took effect, for example because the cluster was broken and its monitoring plugin could not reconcile the change. In that case no `rpch` object exists for the change (`kubectl get rpch -A`). Either way, the fix is to remove the `VMRule` that still carries the rule. + +### Cause 2: cert-manager webhook denials for {cpaas-ca ClusterIssuer ait.alauda.io} count as 403 read failures + +Some platform certificates are deliberately not issued by cert-manager, but the platform still uses cert-manager's certificate expiry tracking for them. To do that, it registers a non-existent API group, `ait.alauda.io`, as the issuer group. Requests for that group are rejected by the cert-manager webhook, so kube-apiserver logs denials that reference `{cpaas-ca ClusterIssuer ait.alauda.io}`. + +The rejections are `403` responses and count as failed reads. A custom success-rate alert such as: + +```text +sum(irate(apiserver_request_total{code=~"20.*",verb=~"GET|LIST"}[2m]))/sum(irate(apiserver_request_total{verb=~"GET|LIST"}[2m]))<0.9 +``` + +then oscillates around 0.9 on a workload cluster. On the affected versions this behavior is expected and cannot be avoided; when cert-manager runs normally and no certificate is actually expired, the log lines can be ignored. + +### Cause 3: cdi-uploadserver-client-cert on a KubeVirt cluster is valid for one day by design + +On a cluster with KubeVirt virtualization, `cdi-uploadserver-client-cert` has a one-day lifetime and is renewed daily. It is the client TLS certificate for VM image upload, used only for mutual TLS between `cdi-uploadproxy` (or the upload client) and `cdi-uploadserver` when an ISO or qemu image is uploaded into a DataVolume. That traffic is short-lived, so the short lifetime is intended. The certificate is active and not deprecated. The platform cannot silence the alert for one individual certificate, so the "about to expire" alert keeps firing for it. + +### Cause 4: the node-local-certs-exporter Pod is short-lived and no longer exists + +`node-local-certs-exporter-` belongs to Cluster Enhancer and inspects node certificates. It runs one check, and once it has collected the certificate information it is cleaned up automatically. An external monitor that discovers Pods through kube-apiserver can catch it while it runs, or in a final failed state, and then keep alerting on a Pod that no longer exists. Its absence is normal. Why a single run ended in a failed state cannot be determined after the Pod has been cleaned up. + +## Resolution + +### Cause 1: a custom VMRule still emits cpaas-certificates-rule after the AlertRule is gone + +1. Confirm that the platform certificates are not expired (see Diagnostic Steps) and that AlertRule `cpaas-certificates-rule` is gone: + + ```bash + kubectl get alertrule -A | grep cpaas-certificates-rule + ``` + +2. Find the object that still carries the rule. Name matches are not enough, because a custom `VMRule` can have any name; search the `vmalert` rule-file ConfigMaps and the `VMRule` contents: + + ```bash + kubectl get vmrule -A + for i in 0 1 2; do kubectl get cm -n cpaas-system vm-alert-rulefiles-$i -o yaml | grep -B5 -A10 "certificate_expires_time"; done + kubectl get vmrule -A -o yaml | grep -B5 -A10 "certificate_expires_time" + kubectl get rpch -A + ``` + +3. Back up and delete that `VMRule`. + + > **Warning:** Deleting the custom `VMRule` stops every alert it generates, including alerts you may still want from it. Do this only when `openssl` shows the platform certificates are valid and this `VMRule` is still the source of `cpaas-certificates-rule` after the AlertRule is gone. Roll back with `kubectl apply -f` of the backup, which restores the same alerts. + + ```bash + kubectl get vmrule -n -o yaml > "$HOME/.yaml" + kubectl delete vmrule -n + ``` + +4. Repeat step 2. The rule-file ConfigMaps should no longer contain the rule, and the alert should resolve. + +### Cause 2: cert-manager webhook denials for {cpaas-ca ClusterIssuer ait.alauda.io} count as 403 read failures + +1. Check that cert-manager is running and that no certificate is actually failing: + + ```bash + kubectl get pods -A | grep cert-manager + kubectl get certificate -A | grep -v True + ``` + + If cert-manager is healthy and the second command lists only the header, the log lines need no fix. + +2. If a custom read-success-rate alert fires because of these 403s, edit that existing alert policy (do not create a second one) so that 4xx responses are excluded from the denominator: + + ```text + sum(irate(apiserver_request_total{code=~"20.*",verb=~"GET|LIST"}[2m]))/sum(irate(apiserver_request_total{code!~"4..",verb=~"GET|LIST"}[2m]))<0.9 + ``` + + This is a workaround: it also hides 4xx responses from other clients. For other sources of apiserver error-rate alerts, see [kube-apiserver latency alerts, noisy apiserver logs, and controller-manager or scheduler health alerts on ACP](kube-apiserver_latency_alerts_noisy_apiserver_logs_and_controller-manager_or_scheduler_health_alerts_on_ACP.md). + +### Cause 3: cdi-uploadserver-client-cert on a KubeVirt cluster is valid for one day by design + +1. Confirm the alerting certificate is `cdi-uploadserver-client-cert`: + + ```bash + kubectl get secret -A | grep cdi-uploadserver-client-cert + ``` + +2. Leave the certificate as it is and handle the alert on the policy side. Under **Operation and Maintenance Center** > **Alerts** > **Alert Policies**, find the certificate expiry policy and adjust it, for example by turning off the 30-day and 10-day pre-expiry rules and keeping the already-expired rule, or silence the policy. Changing the alert level of the built-in policy from the console may not be possible. + +3. Alternatively, lengthen the certificate lifetime through the HyperConverged (HCO) object with a CDI JSON patch annotation that sets `/spec/certConfig/client`. + + > **Warning:** HCO passes the patch to CDI, which reissues the upload client certificate with the new lifetime (ten years below). Invalid JSON in the annotation can stop HCO from reconciling CDI. Do this only when the alerting certificate is `cdi-uploadserver-client-cert`. Save the HCO object first. Roll back by removing the annotation: `kubectl annotate hco -n kubevirt kubevirt-hyperconverged containerizeddataimporter.kubevirt.io/jsonpatch-`. + + ```bash + kubectl get hco -n kubevirt kubevirt-hyperconverged -o yaml > "$HOME/kubevirt-hyperconverged.yaml" + kubectl annotate hco -n kubevirt kubevirt-hyperconverged --overwrite \ + containerizeddataimporter.kubevirt.io/jsonpatch='[{"op": "add", "path": "/spec/certConfig/client", "value": {"duration": "87600h0m0s", "renewBefore": "720h0m0s"}}]' + kubectl get secret -n kubevirt cdi-uploadserver-client-cert -o yaml | grep -E 'certConfig|certificate-not-after' + ``` + + The Secret should then carry the annotation `operator.cdi.kubevirt.io/certConfig` with `"lifetime":"87600h0m0s"`. + +### Cause 4: the node-local-certs-exporter Pod is short-lived and no longer exists + +1. Confirm the Pod is gone: + + ```bash + kubectl get pod -A -o wide | grep certs + ``` + + If needed, use the kube-apiserver audit log to confirm the Pod was created and deleted as part of a normal run. + +2. No repair is required. Tune the external monitor so that it does not alert on this short-lived Pod. + +If the Pod exists and stays in `Error` with `cert-tool export` in its log, that is a different fault: see Cause 3 of [Node kubectl fails with You must be logged in to the server or an alert names root kube config on ACP](Node_kubectl_fails_with_You_must_be_logged_in_to_the_server_or_an_alert_names_root_kube_config_on_ACP.md). + +## Diagnostic Steps + +1. Print the real certificate the alert names, to tell whether the certificate or the alert is wrong: + + ```bash + kubectl get secret -n -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject + ``` + + A `notAfter` in the future means the certificate is valid and the alert source has to be found. + +2. Check the components and rule objects that produce the alert: + + ```bash + kubectl get po -A | grep -E 'cluster-enhancer|courier|vmalert' + kubectl get vmrule -A | grep -i cert + kubectl get alertrule -A | grep cpaas-certificates-rule + ``` + +3. Match the signature: + + | What you see | Cause | + | --- | --- | + | AlertRule `cpaas-certificates-rule` absent, rule still in `vm-alert-rulefiles-*` or a `VMRule` | 1 | + | kube-apiserver log mentions `ait.alauda.io`; failed reads are mostly `403` | 2 | + | Alerting certificate is `cdi-uploadserver-client-cert` with one day left | 3 | + | External monitor alerts on a `node-local-certs-exporter-` Pod that does not exist | 4 | + | Certificate page empty, or still shows a renewed or removed certificate | *Certificate page empty or stale because of clock drift or a stale cluster-enhancer-apiserver metric on ACP* | + + To see which response codes drive a read-failure alert, query in the monitoring UI: + + ```text + sum by (code) (rate(apiserver_request_total{verb=~"GET|LIST",code!~"2.."}[5m])) + ``` + + A result dominated by `code="403"` together with `ait.alauda.io` in the kube-apiserver log matches Cause 2. diff --git a/docs/en/solutions/Certificate_and_restart_alerts_that_persist_after_an_upgrade_match_business_certificates_or_fail_to_load_on_ACP.md b/docs/en/solutions/Certificate_and_restart_alerts_that_persist_after_an_upgrade_match_business_certificates_or_fail_to_load_on_ACP.md new file mode 100644 index 000000000..006d5a49f --- /dev/null +++ b/docs/en/solutions/Certificate_and_restart_alerts_that_persist_after_an_upgrade_match_business_certificates_or_fail_to_load_on_ACP.md @@ -0,0 +1,176 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Certificate and restart alerts that persist after an upgrade, match business certificates, or fail to load on ACP + +## Issue + +After a platform upgrade, or around certificate monitoring in general, alerts do not behave as expected: + +- Kubernetes certificate alerts keep firing after an upgrade for certificate files with hundreds of negative days of validity, or for certificate Secrets that were already deleted. +- The global cluster keeps reporting `sentry-serving-cert` below `10.00d` for a workload cluster whose machines are powered off. +- Certificate expiry warnings fire 30 days and 10 days ahead (`平台组件证书的到期时间即将到期(小于10天)`), more often than the site wants. +- During a cluster upgrade, restart alerts name business containers that did not restart. +- Platform certificate alerts fire for business (workload) certificates, and the question is whether replacing VictoriaMetrics with Prometheus avoids that. +- The rule `平台组件证书已到期 <= 0d 且持续 1 分钟` (platform component certificate expired) is firing, and its chart shows only `加载失败,请 刷新` (failed to load, please refresh), so nobody can tell which certificate expired. + +## Root Cause + +### Cause 1: cluster-enhancer keeps data for certificates that no longer exist after an upgrade + +The certificate expiry data shows negative validity for files that are no longer in use, for example `/var/lib/kubelet/pki/kubelet.crt` at about `-952.1 d` on several nodes and `/etc/kubernetes/super-admin.conf` at `-16.86 d`, while current certificates such as `kubelet-client-current.pem` and `apiserver-loopback-client` are positive. Deleted certificate Secrets still appear in the real-time alerts. After the upgrade, `cluster-enhancer-controller` and `cluster-enhancer-apiserver` were out of sync and kept data for certificates and Secrets that no longer exist. + +When the VictoriaMetrics UI returns no `certificate_expires_time` series for the deleted object and the alert still shows it, the stale state is in `courier` instead; that variant is Cause 3 of [Monitoring pages stay empty because courier-api or apollo holds a stale cluster cache on ACP](Monitoring_pages_stay_empty_because_courier-api_or_apollo_holds_a_stale_cluster_cache_on_ACP.md). + +### Cause 2: sentry-serving-cert below 10.00d is reported for a powered-off workload cluster + +The global cluster reports `sentry-serving-cert` below `10.00d` for a workload cluster whose nodes are all shut down, and the cluster must be kept. On v4.1.2, with full VictoriaMetrics on both sides, restarting `courier` and `courier-api` on the global cluster cleared it. Why a powered-off cluster was still reported is not known. + +### Cause 3: the expiry warning thresholds do not match the site's requirement + +The certificate policy warns 30 days and 10 days before expiry. That frequency is a configuration choice, not a fault. Sites that want fewer warnings silence the 30-day rule and move the 10-day threshold (for example to 3 days, a site requirement and not a product default). + +### Cause 4: kube-state-metrics restarts split the restart-count series and fake an increase + +The rule `计算组件下容器组最近5分钟的重启次数 > 0 且持续 1 分钟` (Pod restarts in the last 5 minutes > 0, level `提示` info) fires several times within minutes during a cluster upgrade. It queries `cpaas_advanced_kube_pod_container_status_restarts_total` with an expression of this shape: + +```promql +sum by (kind, name, namespace, pod) (round((increase(cpaas_advanced_kube_pod_container_status_restarts_total{kind=~"Deployment",name=~"",namespace=~""}[5m]) + abs(increase(cpaas_advanced_kube_pod_container_status_restarts_total{...}[5m]))) / 2)) +``` + +When kube-state-metrics restarts, the same container gets two series that differ only in `instance` (two kube-state-metrics endpoints on port `8080`), with the same `uid` and value `1`; the first ends where the second begins. `increase(...[5m])` counts the new series' `1` as an increase from 0 and fires, although `restartCount` did not change. The expression has not been changed for this. + +### Cause 5: the certificate alert expression includes business certificates, whatever the backend + +The platform certificate alert expression matches workload certificates as well, which produces platform alerts for business certificates. The expression, not the VictoriaMetrics backend, selects them, so switching to Prometheus monitors the same certificates. + +### Cause 6: the certificate-expired rule chart fails to load because the vmselect query times out + +In a cluster-level policy `证书告警策略` (certificate alert policy), the rule `平台组件证书已到期 <= 0d 且持续 1 分钟` (level `灾难` disaster) is firing, while the 30-day and 10-day rules in the same policy are normal. The rule's expression covers every certificate in the cluster, and its query exceeds the `vmselect` query time limits, so the chart shows `加载失败,请 刷新`. A similar 4.1 symptom is Cause 3 of [Certificates tab missing, platform domain certificate not listed, or certificate alert shows no data on ACP](Certificates_tab_missing_platform_domain_certificate_not_listed_or_certificate_alert_shows_no_data_on_ACP.md). + +## Resolution + +### Cause 1: cluster-enhancer keeps data for certificates that no longer exist after an upgrade + +1. Confirm that the files with negative validity are not the certificates the node uses; a certificate that really expired needs renewal, not a restart: + + ```bash + openssl x509 -noout -enddate -in /var/lib/kubelet/pki/kubelet-client-current.pem + ls -l /var/lib/kubelet/pki/ + kubectl get secret -n + ``` + +2. Run the alert expression in the VictoriaMetrics UI. A `certificate_expires_time` series still returned for the deleted object is this cause. + +3. In the affected cluster, restart both cluster-enhancer components. + + > **Warning:** Restarting `cluster-enhancer-controller` and `cluster-enhancer-apiserver` drops their endpoints for one start; certificate monitoring for that cluster pauses and a metrics-availability alert can fire once. Precondition: the alerts name certificates or Secrets that no longer exist, and the series is still returned. Rollback is the replacement Pods becoming Ready. + + ```bash + kubectl get pods -A | grep cluster-enhancer + kubectl delete pod -n + ``` + +The Kubernetes certificate alerts, including the rows for deleted Secrets, return to normal. + +### Cause 2: sentry-serving-cert below 10.00d is reported for a powered-off workload cluster + +Keep the workload cluster, and on the global cluster restart `courier` and `courier-api`: + +> **Warning:** Restarting both pauses notification sends and the active-alert scrape until the new Pods are Ready. Precondition: the row is `sentry-serving-cert` below `10.00d` for a workload cluster whose nodes are powered off, and you are on the global cluster. Rollback is the new Pods becoming Ready. + +```bash +kubectl get pods -n cpaas-system | grep courier +kubectl delete pod -n cpaas-system +``` + +The row clears. If it returns, the cause is still open; do not delete the workload cluster for it. + +### Cause 3: the expiry warning thresholds do not match the site's requirement + +Find the certificate policy by its rule text, then, on that policy under **Operations Center > Alerts > Alert Policies**, silence the 30-day rule and change the nearer threshold to the value the site requires: + +```bash +kubectl get alertrule -A | grep -i certificate +``` + +These changes reduce frequency only; they do not reduce query load. + +### Cause 4: kube-state-metrics restarts split the restart-count series and fake an increase + +1. Compare `restartCount` with the alert: + + ```bash + kubectl get pod -n -o jsonpath='{.status.containerStatuses[*].restartCount}{"\n"}' + ``` + +2. Query the raw series in the VictoriaMetrics UI over the alert window: + + ```promql + cpaas_advanced_kube_pod_container_status_restarts_total{namespace="",pod=""} + ``` + + Two series with the same `uid` and value, one ending where the other starts, with different `instance` values, is this cause. + +3. Treat a restart alert in that kube-state-metrics window as a false positive when `restartCount` did not increase. No change is needed on the workload. + +### Cause 5: the certificate alert expression includes business certificates, whatever the backend + +Do not replace VictoriaMetrics with Prometheus to fix this. Find which certificate the alert names: + +```bash +kubectl get secret -A --field-selector type=kubernetes.io/tls | grep +``` + +A business certificate that is really near expiry is for its owner to renew. To stop the cluster-level policy from alerting on business certificates, disable it and rely on the general certificate policy from the `global` cluster, as in Cause 6. Built-in rules cannot be edited in place. + +### Cause 6: the certificate-expired rule chart fails to load because the vmselect query times out + +1. Read the current `vmselect` arguments and log on the cluster whose chart fails: + + ```bash + kubectl get vmclusters -A + kubectl get vmclusters -n cpaas-system -o jsonpath='{.spec.vmselect.extraArgs}{"\n"}' + kubectl logs -n cpaas-system vmselect-cluster-0 --tail=200 | grep -i -E 'timeout|deadline|maxQueryDuration|maxQueueDuration' + ``` + +2. Raise `search.maxQueryDuration` and `search.maxQueueDuration` (VictoriaMetrics `vmselect` flags; upstream defaults 30 seconds and 10 seconds) under `spec.vmselect.extraArgs`. + + > **Warning:** The operator rolls the `vmselect-cluster` Pods; queries and the monitoring and alert pages stall until they are Ready. Longer limits also let heavy queries hold `vmselect` CPU and memory longer. Precondition: the chart shows `加载失败,请 刷新` and the query times out, and the object is exported to `/tmp/vmcluster.yaml`. Rollback: `kubectl apply -f /tmp/vmcluster.yaml`. A plugin reconcile or upgrade can write the plugin value back. + + ```bash + kubectl get vmclusters -n cpaas-system -o yaml > /tmp/vmcluster.yaml + kubectl patch vmclusters -n cpaas-system --type=merge \ + -p '{"spec":{"vmselect":{"extraArgs":{"search.maxQueryDuration":"","search.maxQueueDuration":""}}}}' + ``` + +3. Reload the rule chart and read which certificate is at or below 0 days. Renew a certificate that has really expired. + +4. Disable the cluster-wide certificate policy from its action menu under **Operations Center > Alerts > Alert Policies**, and make sure the general certificate policy of the `global` cluster covers the platform certificates. + +## Diagnostic Steps + +1. After an upgrade, certificate alerts that name files with large negative validity or Secrets that do not exist: check the object and the series (Cause 1): + + ```bash + kubectl get secret -n + kubectl get pods -A | grep cluster-enhancer + ``` + + Expected for Cause 1: the Secret is `NotFound` and the VictoriaMetrics UI still returns its series. + +2. `sentry-serving-cert` below `10.00d` for a cluster whose nodes are all powered off: Cause 2. + +3. 30-day and 10-day warnings that are too frequent for the site: Cause 3. + +4. A restart alert during an upgrade with unchanged `restartCount` and two series that differ only in `instance`: Cause 4. + +5. A certificate alert that names a business TLS Secret: Cause 5. A certificate rule chart with `加载失败,请 刷新`: Cause 6. diff --git a/docs/en/solutions/Certificate_page_empty_or_stale_because_of_clock_drift_or_a_stale_cluster-enhancer-apiserver_metric_on_ACP.md b/docs/en/solutions/Certificate_page_empty_or_stale_because_of_clock_drift_or_a_stale_cluster-enhancer-apiserver_metric_on_ACP.md new file mode 100644 index 000000000..c3667aa6e --- /dev/null +++ b/docs/en/solutions/Certificate_page_empty_or_stale_because_of_clock_drift_or_a_stale_cluster-enhancer-apiserver_metric_on_ACP.md @@ -0,0 +1,182 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.0" + - "4.2" + - "4.3" +--- + +# Certificate page empty or stale because of clock drift or a stale cluster-enhancer-apiserver metric on ACP + +## Issue + +The platform certificate page or a certificate expiry alert does not match the certificates that actually exist: + +- The certificate management page is empty although certificate metrics exist, and the Prometheus UI of the `global` cluster warns `Detected seconds time difference between your browser and the server`. +- "Platform component certificates have expired" stays active after the certificate was renewed or removed, while VictoriaMetrics counts 0 expired certificates, or the alert data loads very slowly. +- After an upgrade, every cluster reports `alive-admin` certificates (`certmanager.-etc-kubernetes-alive-admin.conf`) as expired, with a value of about `-739,800` days. +- A Secret was rebuilt with a new certificate, and the page still shows the old date or "expired". +- `平台Kubernetes证书已到期 <= 0d 且持续 1 分钟` ("platform Kubernetes certificate has expired <= 0d for 1 minute", rule group `cpaas-cluster-rules`) names `/etc/kubernetes/kubelet.conf` or `/etc/kubernetes/admin.conf` on nodes where no matching file seems to exist. + +Certificate expiry metrics (`certificate_expires_time`) are exposed by `cluster-enhancer-apiserver` (Cluster Enhancer); node certificate files are read on the nodes; the page and alert status are served through `courier` on the `global` cluster, which refreshes its stored state about every 4 hours. + +Related articles: *Certificate alerts from a custom VMRule, ait.alauda.io webhook denials, or by-design short-lived certificates on ACP* (a custom `VMRule`, `ait.alauda.io` webhook denials, a one-day `cdi-uploadserver-client-cert`, or a missing `node-local-certs-exporter` Pod). [Monitoring pages stay empty because courier-api or apollo holds a stale cluster cache on ACP](Monitoring_pages_stay_empty_because_courier-api_or_apollo_holds_a_stale_cluster_cache_on_ACP.md) (Cause 3) covers an alert that remains after the expired certificate object was deleted. + +## Root Cause + +### Cause 1: browser time and server time differ, so the certificate page shows no data + +The certificate data exists, but the page renders nothing. Querying the metrics directly (for example `curl :19890/metrics | grep cert` from the `cluster-console` terminal of the `global` cluster) returns `certmanager_certificate_expiration_timestamp_seconds` for files such as `/etc/kubernetes/admin.conf`, `/etc/kubernetes/pki/apiserver.crt`, and `/etc/kubernetes/pki/etcd/server.crt`, together with `platform_scrape_collector_success{collector="cert"} 1`. The Prometheus UI of the `global` cluster shows: + +```text +Warning: Error fetching server time: Detected 305.3659999370575 seconds time difference between your browser and the server. Prometheus relies on accurate time and time drift might cause unexpected query results. +``` + +The clock of the machine running the browser and the node clock differ by several minutes (after time-zone conversion). Queries evaluated against the wrong time range return nothing, so the page is empty. Synchronizing node time makes the page fill again. + +### Cause 2: cluster-enhancer-apiserver is hung or cannot parse a certificate, so the expiry metric is stale + +The certificate is valid or already gone, and the "expired" alert does not clear because `cluster-enhancer-apiserver` keeps exporting a stale or invalid `certificate_expires_time`. Three forms are known; restarting `cluster-enhancer-apiserver` (together with `cluster-enhancer-controller` in the hung form) clears the alert in each: + +- **Empty alive certificate after an upgrade.** The expression `(max by(secret_namespace,name) (certificate_expires_time{secret_namespace!=""}) - time()) / (60*60*24)<=0` returns series `certmanager.-etc-kubernetes-alive-admin.conf` in `secret_namespace="cpaas-system"` with `check_time`, `not_after`, and `not_before` all `0001-01-01T00:00:00Z` and the value `-62135596800` (the zero time). The `cluster-enhancer-apiserver` log has `failed to parse certificate from secret cpaas-system/certmanager.-etc-kubernetes-alive-admin.conf, error: PEM is empty` for each control-plane node, sometimes with watch errors ending in `net/http: TLS handshake timeout`, while its Pods are `1/1 Running`. +- **Zero expiry on a region cluster.** The alert detail of `cpaas-certificates-rule` lists `certmanager.-etc-kubernetes-alive-admin.conf` entries at about `-739781 d` (the distance back to year 1, the same zero time), while other certificates such as `cluster-enhancer-apiserver-cert` are positive. `/etc/kubernetes/alive/admin.conf` on the alerting nodes has no `client-certificate-data`. Why that field was missing has not been determined. +- **Hung after certificates were removed.** Expired certificates were removed (for example `argocd-gitops-ca` and `argocd-gitops-tls` in `argocd`, by uninstalling the Argo CD GitOps plugin); VictoriaMetrics then counts 0 expired, but the alert stays and the alert data loads very slowly. `kubectl logs -f` of `cluster-enhancer-apiserver` stops at startup lines such as `Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file` and prints nothing new. + +If alert data loads slowly because `vmselect` is overloaded or `OOMKilled`, not because `cluster-enhancer-apiserver` is hung, the cause is query load, not this one. + +### Cause 3: the certificate was reissued, or its file was deleted, and the page still shows the previous state + +The status the page and alert show is refreshed about every 4 hours. After a Secret is deleted and recreated with a new certificate, the old date or "expired" remains until that refresh, or until `courier` on the `global` cluster is restarted. A variant after a renewal: the browser console reports `found duplicate series for the match group` with one `phase="Expired"` and one `phase="Normal"` series for the same certificate and `many-to-many matching not allowed`; a `courier` restart clears it. + +Node-file alerts follow the same rule. When `/etc/kubernetes/kubelet.conf` shows a negative number of days (for example about `-158 d`), the node certificate really has expired. When `/etc/kubernetes/admin.conf` shows a large negative value (for example about `-745 d`) but the file has been deleted, the collector cannot read a new file and keeps reporting the last state. In both situations, reissue the certificates and then wait four hours or restart `courier`. Stale node-file rows after an upgrade (for example `/var/lib/kubelet/pki/kubelet.crt` or `/etc/kubernetes/super-admin.conf`) have also been cleared by restarting `cluster-enhancer-controller` and `cluster-enhancer-apiserver` instead, so if a `courier` restart does not help, apply the Cause 2 restart. + +## Resolution + +### Cause 1: browser time and server time differ, so the certificate page shows no data + +1. From the `cluster-console` terminal of the `global` cluster, confirm the data exists and read the server time; compare it with the clock of the machine running the browser (convert the time zone first): + + ```bash + curl -s :19890/metrics | grep cert | head + date + ``` + +2. If the metrics are present and the clocks differ by minutes, compare node time with a reference clock and with the browser host, and fix time synchronization on whichever clock is wrong. On nodes that use chrony: + + ```bash + date -u + timedatectl status + chronyc tracking 2>/dev/null || ntpq -p + ``` + + See [Configuring chrony NTP on Cluster Nodes](Configuring_chrony_NTP_on_Cluster_Nodes.md) for the node time source. + +3. Correct the node clock. + + > **Warning:** Stepping the clock of a running node can disturb certificate validity checks, leases, and etcd on that node. Do this only when the offset is confirmed against a reference clock and the Prometheus UI shows the time-difference warning. Prefer letting chrony slew the clock; a large step belongs in a maintenance window, one control-plane node at a time. There is no rollback to the previous wrong time; if the new time source is wrong, point chrony back to the previous source. + + ```bash + systemctl restart chronyd + chronyc makestep + ``` + +4. Reload the certificate page; it should list the certificates again. + +### Cause 2: cluster-enhancer-apiserver is hung or cannot parse a certificate, so the expiry metric is stale + +1. Confirm the certificate is valid or removed (see Diagnostic Steps), then check `cluster-enhancer-apiserver` on the cluster that raised the alert: + + ```bash + kubectl get po -A | grep cluster-enhancer-apiserver + kubectl logs -n --since=1h | tail -50 + ``` + + No log lines in the last hour (the log stops at startup lines such as `Caches are synced for client-ca::...`), or `failed to parse certificate from secret ... error: PEM is empty`, matches this cause. For a removed certificate, check that its Secret is really gone: + + ```bash + kubectl get secret -n | grep + ``` + +2. Restart `cluster-enhancer-apiserver`. + + > **Warning:** Restarting `cluster-enhancer-apiserver` pauses certificate metric collection for that cluster until the new Pod is Running. Do this only when `openssl` or the certificate page already shows the alerting certificate as valid or removed, and the component is silent or logs `PEM is empty`. The Deployment recreates the Pod; there is nothing else to roll back. + + ```bash + kubectl delete pod -n + kubectl get po -n | grep cluster-enhancer-apiserver + ``` + +3. When `cluster-enhancer-apiserver` was hung (no new log lines), restart both Cluster Enhancer Pods in `cpaas-system` on that cluster. The same Warning applies; the controller is also paused until its Pod is Running. + + ```bash + kubectl get pod -n cpaas-system | grep enhancer | awk '{print $1}' | xargs kubectl delete pod -n cpaas-system + kubectl get pod -n cpaas-system | grep enhancer + ``` + +4. After one scrape interval, re-run the alert expression in the VictoriaMetrics UI. It should return no series for the fixed certificate. + +### Cause 3: the certificate was reissued, or its file was deleted, and the page still shows the previous state + +1. Print the real certificate: + + ```bash + kubectl get secret -n -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates + ``` + + For a node-file alert, print the file the alert names on that node: + + ```bash + ls -l /etc/kubernetes/kubelet.conf /etc/kubernetes/admin.conf + grep client-certificate-data /etc/kubernetes/kubelet.conf | awk '{print $2}' | base64 -d | openssl x509 -noout -dates + ``` + + If the kubeconfig has `client-certificate: ` instead of `client-certificate-data`, run `openssl x509 -noout -dates -in ` on that file. + + If the certificate in the file is expired, it is a real expiry; reissue it with the certificate renewal procedure for your ACP version before continuing. If the file no longer exists, the alert is the last stored state. + +2. When the real certificate is new (or back in place), wait about 4 hours, or restart `courier` on the `global` cluster. + + > **Warning:** Deleting the `courier` Pod fails in-flight monitoring and inspection calls and pauses notification sending until the replacement is Ready. Do this only after the certificate is confirmed valid and only the displayed state is old. The controller recreates the Pod; there is nothing to roll back. + + ```bash + kubectl -n cpaas-system get pods | grep courier + kubectl -n cpaas-system delete pod + ``` + +3. If the stale row remains and the VictoriaMetrics UI still returns the old `certificate_expires_time` series, apply the Cause 2 restart. + +## Diagnostic Steps + +1. Print the real certificate the alert or page names, to tell whether the certificate or the display is wrong: + + ```bash + kubectl get secret -n -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject + ``` + +2. Check the components that produce the status: + + ```bash + kubectl get po -A | grep -E 'cluster-enhancer|courier|vm' + ``` + +3. Query the series in the VictoriaMetrics UI: + + ```text + (max by(secret_namespace,name) (certificate_expires_time{secret_namespace!=""}) - time()) / (60*60*24)<=0 + ``` + +4. Match the signature: + + | What you see | Cause | + | --- | --- | + | Page empty, metrics present, Prometheus warns `Detected seconds time difference between your browser and the server` | 1 | + | Series with `not_after="0001-01-01T00:00:00Z"` (about `-739,800 d`), or `PEM is empty` in the `cluster-enhancer-apiserver` log | 2 | + | Certificate removed, VictoriaMetrics counts 0 expired, `cluster-enhancer-apiserver` log silent | 2 | + | Secret rebuilt or node file deleted, no stale series in VictoriaMetrics, page shows the previous state | 3 | + | Node file such as `kubelet.conf` contains a certificate that is really expired | 3 (real expiry: reissue first) | diff --git a/docs/en/solutions/Certificates_tab_missing_platform_domain_certificate_not_listed_or_certificate_alert_shows_no_data_on_ACP.md b/docs/en/solutions/Certificates_tab_missing_platform_domain_certificate_not_listed_or_certificate_alert_shows_no_data_on_ACP.md new file mode 100644 index 000000000..7d3f0ae4a --- /dev/null +++ b/docs/en/solutions/Certificates_tab_missing_platform_domain_certificate_not_listed_or_certificate_alert_shows_no_data_on_ACP.md @@ -0,0 +1,119 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.1.x + - 4.2.x +--- + +# Certificates tab missing, platform domain certificate not listed, or certificate alert shows no data on ACP + +## Issue + +One of the following happens while looking for certificate status in the ACP console: + +- After installing or upgrading the platform, the Administrator view has no standalone **Certificates** tab, although an older installation had one. +- The platform domain (the platform access address) and its certificate cannot be found on the Certificate Management page. +- The detail of a certificate expiry alert does not refresh and shows no data, for example for the rule `平台组件证书的到期时间即将到期(小于 30 天) <= 30d 且持续 1 分钟` (platform component certificate expires in less than 30 days, lasting 1 minute). +- The alert notification arrives with unrendered template text, for example `位于 global / 的证书 ( {{ $labels.name }} ) 即将过期,过期时间{{ .externalLabels.comparison }}{{ .externalLabels.threshold }}天` (the certificate in global / ( ... ) is about to expire). + +These are questions about where certificate status is displayed and how certificate alerts render, not about whether the certificate itself renews. Whether a cert-manager serving certificate renews by itself is covered in "Platform certificate expiry alert for a cert-manager serving certificate that renews by itself on ACP". + +## Root Cause + +### Cause 1: ACP 4.2 removed the legacy certificate management page + +In ACP 4.2 the legacy certificate management feature, including the standalone Certificates tab, was removed. Certificate monitoring is now shown per cluster: **Clusters > Clusters**, select the cluster, open the **Monitoring** tab, then the **Certificate Status** sub-tab. The page shows a Kubernetes certificate statistic and a platform certificate statistic, each with counts for Normal, Expires soon, Expired, and Failed, and lists the Kubernetes certificates (for example `/etc/kubernetes/admin.conf`) with node, expiry time, and status. + +Certificate monitoring is provided by the **Cluster Enhancer** cluster plugin, installed from **Marketplace > Cluster Plugins** in the Administrator view. It monitors: + +- Kubernetes component certificates, including control plane, kubelet, and kubeconfig client certificates. +- Secrets of type `kubernetes.io/tls`. +- The serving certificates kube-apiserver actually uses, checked through the `kubernetes` Endpoints. + +Status is exposed through the metrics `certificate_expires_status` and `certificate_expires_time`. If Cluster Enhancer is not installed on a cluster, its Certificate Status sub-tab has no data. + +### Cause 2: the platform domain certificate is managed separately and is not in Certificate Management + +The platform domain certificate is not maintained in Certificate Management, so it is not listed there. It is configured in `spec.http.tls` of the `prdb` resource `base` on the global cluster. Where that field only holds `secretRef.name: dex.tls`, the certificate is in Secret `cpaas-system/dex.tls`. + +### Cause 3: on ACP 4.1 the certificate alert detail shows no data because cpaas-certificates-rule queries time out + +On ACP 4.1 the alert policy `cpaas-certificates-rule` also collects the expiry times of business certificates, which it should not do. The resulting query load makes the queries time out, so the alert detail does not refresh and shows no data. This is a known issue of 4.1. The alert itself can still be valid, so check the certificate directly. + +### Cause 4: the notification shows raw {{ $labels.name }} placeholders because a custom notification template lacks the variables + +The alert from `cpaas-certificates-rule` is real, but it is delivered through a custom-developed webhook notification template that does not include the certificate alert variables. The template engine therefore leaves `{{ $labels.name }}`, `{{ .externalLabels.comparison }}`, and `{{ .externalLabels.threshold }}` unrendered. The fix belongs to whoever maintains that template. + +## Resolution + +### Cause 1: ACP 4.2 removed the legacy certificate management page + +1. Open **Clusters > Clusters**, select the cluster, and open **Monitoring > Certificate Status**. +2. If the sub-tab has no data, check that the Cluster Enhancer plugin is installed on that cluster (**Marketplace > Cluster Plugins**) and that its Pods run: + + ```bash + kubectl get pods -A | grep cluster-enhancer + ``` + +3. The same data can be queried on the monitoring query page with the metrics: + + ```text + certificate_expires_time + certificate_expires_status + ``` + +### Cause 2: the platform domain certificate is managed separately and is not in Certificate Management + +Read the certificate from where it is configured, on the global cluster: + +```bash +kubectl get prdb base -o jsonpath='{.spec.http.tls}{"\n"}' +kubectl get prdb base -o jsonpath='{.spec.http.tls.cert}' 2>/dev/null | base64 -d | openssl x509 -noout -subject -dates +kubectl get secret -n cpaas-system dex.tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject -dates +``` + +To renew it, follow "Replace the platform access certificate in dex.tls on ACP". On ACP 3.12, where the certificate is held inline in `prdb`, follow "Replace the platform access HTTPS certificate in prdb with a PEM certificate and key on ACP 3.12". + +### Cause 3: on ACP 4.1 the certificate alert detail shows no data because cpaas-certificates-rule queries time out + +1. Read the certificate name from the alert (or from the legend at the bottom of the alert monitoring panel, which lists `name` and `secret_namespace` for each series). +2. Check the certificate itself instead of the alert detail: + + ```bash + kubectl get certificate -A | grep + kubectl get certificate -n -o yaml | grep -E 'renewBefore|renewalTime|notAfter' + ``` + +3. If `renewalTime` is in the future and the Certificate is Ready, the certificate renews automatically and the alert can be ignored. Otherwise handle the certificate with the article that matches its name (see "Platform certificate expiry alert for a cert-manager serving certificate that renews by itself on ACP"). + +### Cause 4: the notification shows raw {{ $labels.name }} placeholders because a custom notification template lacks the variables + +1. Confirm that the rule returns data: paste its expression into the monitoring query page. + + ```text + (0< max by(secret_namespace,name) (certificate_expires_time{secret_namespace!=""}) - time()) / (60*60*24)<=30 + ``` + +2. If the query returns the certificate, the alert is real and the problem is the notification. Open the notification policy that the alert uses and check its template. +3. A custom-developed template has to be extended by its owner with the certificate alert variables (`$labels.name`, `externalLabels.comparison`, `externalLabels.threshold`). Until then, read the certificate name from the query result. + +## Diagnostic Steps + +1. Identify the platform version and whether the question is about the page or about the alert: + + ```bash + kubectl get prdb + kubectl get pods -A | grep -E 'cluster-enhancer|courier' + ``` + + The `prdb` output shows the platform version. Running `cluster-enhancer` Pods mean certificate monitoring is installed on that cluster. + +2. Branch: + + - No Certificates tab on 4.2 or later: Cause 1. + - Platform domain certificate not on Certificate Management: Cause 2. + - Alert detail empty or not refreshing on 4.1: Cause 3. + - Notification text shows `{{ $labels.name }}`: Cause 4. diff --git a/docs/en/solutions/ClickHouse_log_queries_fail_after_the_log_storage_had_been_working_missing_tables_or_broken_parts_on_ACP.md b/docs/en/solutions/ClickHouse_log_queries_fail_after_the_log_storage_had_been_working_missing_tables_or_broken_parts_on_ACP.md new file mode 100644 index 000000000..caf0ad569 --- /dev/null +++ b/docs/en/solutions/ClickHouse_log_queries_fail_after_the_log_storage_had_been_working_missing_tables_or_broken_parts_on_ACP.md @@ -0,0 +1,263 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.2.x + - 4.3.x +--- + +# ClickHouse log queries fail after the log storage had been working: missing tables or broken parts on ACP + +## Issue + +The ClickHouse log storage was working, and now log queries fail or return nothing. Typical reports: + +- "The last 12 hours of logs cannot be queried, but older logs still can." +- "After the ClickHouse cluster came back from exit code 137, nothing can be found." +- "`razor-0` is abnormal on the global cluster." +- "Audit and Operations Center logs are not displayed." + +The log store runs in `cpaas-system` as pods `chi-cpaas-clickhouse-replicated---0`, created from the ClickHouseInstallation `cpaas-clickhouse`, with database `observability`. `razor` is the gateway in front of it. + +The 3-replica layout uses `ReplicatedMergeTree` tables, coordinated by ClickHouse Keeper inside the ClickHouse container. There is no separate ZooKeeper Deployment, although the configuration refers to ZooKeeper. Replication is asynchronous and multi-master, so every replica accepts writes and syncs to the others. PVC usage that differs between replicas by tens of gigabytes is normal, because merges, replication lag, temporary parts and TTL cleanup differ per replica. If the same query returns the same rows on each replica, the data is consistent. A size difference is a fault only together with other symptoms. + +The shared tables include `audit`, `event`, `log_kubernetes`, `log_platform`, `log_system` and `log_workload`. Inserts go through Distributed tables such as `audit_all` and `event_all`. `SHOW TABLES FROM observability` on a healthy replica lists them all. + +Installs that never worked are covered in [ClickHouse log storage plugin install fails or ClickHouse does not start on ACP](./ClickHouse_log_storage_plugin_install_fails_or_ClickHouse_does_not_start_on_ACP.md). Out-of-memory kills and capacity problems are covered in [Logging ClickHouse runs out of disk, CPU or memory, or is OOMKilled on ACP](./Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md). + +## Root Cause + +### Cause 1: Part of the queries fail with UNKNOWN_TABLE because a replica lost its schema but is still registered + +Logs from the last hours fail to load while older ones work, and the error is `UNKNOWN_TABLE`. One replica (for example `-1-0-0`) restarted and lost its local `observability` schema. Its registration in Keeper was never removed. The replica remains an empty shell that still receives traffic, so every query routed to it fails (about one query in six in a 3-replica layout). + +### Cause 2: After recovery from exit code 137, logs cannot be found and table log_kubernetes is missing on some replicas + +The ClickHouse cluster failed with exit code 137. After its resources were raised (to 16 CPU / 16Gi) it came back, but audit, product logs and Kubernetes logs return nothing. The ClickHouse log says table `log_kubernetes` cannot be found. Tables are missing on some replicas (for example `0-0-0` and `0-2-0`) and complete on another (`0-1-0`). + +On an affected replica, `SHOW TABLES FROM observability` lists the Distributed table `log_kubernetes_all` but not the local table `log_kubernetes`. The client may also print `Effective user of the process (clickhouse) does not match the owner of the data (root).` Why the tables went missing has not been determined. Reinstalling ClickHouse with an emptied data directory restores the tables and the queries. The data directory must be cleared before the reinstall. + +### Cause 3: razor fails to initialize because ClickHouse refuses to load observability.log_workload with too many suspicious broken parts + +`razor-0` is abnormal in the console. Its events show initialization failing because ClickHouse does not load table `observability.log_workload`. The ClickHouse log reports suspicious broken parts (for example about 20 parts, 8.11 GiB in total). That is more than ClickHouse handles automatically: `merge_tree/max_suspicious_broken_parts_bytes` defaults to 1 GiB. Disk space is sufficient. + +Parts are typically broken by an abnormal node or ClickHouse restart, an interrupted merge, or a transient storage I/O error. Raising the threshold lets the table load and set the broken parts aside. Afterwards the setting can go back to the default without data loss in the loaded table. Rows in the parts that were set aside can be lost. + +### Cause 4: The ClickHouse disk filled, vector inserts failed, and after space returned ClickHouse is missing metadata files + +The log nodes ran out of disk. The `vector` container in `razor` reported, for the `event`, `audit` and `log_system` sinks: + +```text +Code: 243. DB::Exception: Cannot reserve 1.00 MiB, not enough space. (NOT_ENOUGH_SPACE) +``` + +`nevermore` then repeated `write error: failed to write to file: write /cpaas/log/logs/filebeat--1.ndjson: no space left on device`, and logs stopped arriving. Disk usage later dropped, but the components did not recover by themselves, and a `razor` restart failed: + +- The `razor` container logged `http: proxy error: dial tcp [::1]:8081: connect: connection refused` and `Remove table ttl error ... Code: 159. DB::Exception: Timeout exceeded ... (TIMEOUT_EXCEEDED)`. +- The previous `vector` log showed `vector_buffers::variants::disk_v2::writer: Last written record was unable to be deserialized. Corruption likely.` and repeated `Events dropped intentionally ... reason="Source send cancelled."`. It ended with `run "/usr/local/bin/vector" error: signal: killed`. +- ClickHouse was unhealthy, with metadata files missing that could not be recovered. + +At that point both the ClickHouse data and the `vector` disk buffer are corrupted. The recovery is a reinstall with both data directories cleared on every log node. + +### Cause 5: The clickhouse container fails its liveness probe on 8123 for days, and clickhouse-operator is in CrashLoopBackOff + +Audit logs and Operations Center logs are empty, and the monitoring dashboard shows a log collection rate of 0. `chi-cpaas-clickhouse-replicated-0-0-0` is `Running` with thousands of restarts. Its events repeat: + +```text +Container clickhouse failed liveness probe, will be restarted +Liveness probe failed: Get "http://:8123/ping": dial tcp :8123: connect: connection refused +``` + +On the global cluster, `clickhouse-operator` in `ck-system` is in `CrashLoopBackOff`, and the data disk has free space. The ClickHouse database has been broken for a long time. + +A liveness-probe kill can also show as exit code `137` with `Reason: Error`. That signature is this cause when the events show the `8123` probe failing and the operator is crash-looping. When the reason is `OOMKilled`, or there are no probe failures, see the capacity article instead. The recommended fix is to uninstall the ClickHouse log plugin, empty the logging ClickHouse data directory on the ClickHouse nodes, and redeploy. + +## Resolution + +Open a client on a ClickHouse pod for the SQL below: + +```bash +kubectl -n cpaas-system get pods | grep chi-cpaas-clickhouse +kubectl -n cpaas-system exec -it -c clickhouse -- clickhouse-client +``` + +### Cause 1: Part of the queries fail with UNKNOWN_TABLE because a replica lost its schema but is still registered + +1. Compare the schema on each replica: + + ```sql + SELECT name FROM system.tables WHERE database = 'observability' ORDER BY name; + SELECT database, table, replica_name, is_readonly, active_replicas, total_replicas + FROM system.replicas WHERE database = 'observability'; + ``` + + The ghost is the replica that lists no `observability` tables while the other replicas still count it in `total_replicas`. + +2. Remove the ghost entry from Keeper. + + > **Warning:** `SYSTEM DROP REPLICA` removes that replica's metadata from Keeper for the named table. If you name a replica that is actually healthy, it stops replicating and must be re-created. Precondition: only when `UNKNOWN_TABLE` appears on a share of the queries and exactly one replica has no `observability` tables. Run the statement on a healthy replica, naming the empty one. Rollback: none for the Keeper entry; the replica registers again when its tables are created. + + ```sql + SYSTEM DROP REPLICA '' FROM TABLE observability.log_workload; + ``` + + Repeat it for each replicated table listed in `system.replicas`. + +3. Rebuild the tables on the empty replica. On a healthy replica, list the tables and print each definition: + + ```sql + SHOW TABLES FROM observability; + SHOW CREATE TABLE observability.; + ``` + + Run each `CREATE TABLE` statement on the empty replica. If the database is gone, run `CREATE DATABASE IF NOT EXISTS observability` first. Create the replicated tables first and the Distributed tables after. The replicated tables then fetch their parts from the other replicas. + +4. Query the same time range several times. The `UNKNOWN_TABLE` errors should stop completely. + +### Cause 2: After recovery from exit code 137, logs cannot be found and table log_kubernetes is missing on some replicas + +1. Compare the table count per replica: + + ```sql + SELECT hostName(), count() FROM clusterAllReplicas('replicated', system.tables) + WHERE database = 'observability' GROUP BY 1; + ``` + + Fewer tables on some replicas, together with `log_kubernetes` not found in the log, matches this cause. + +2. Reinstall with an empty data directory. + + > **Warning:** Emptying the ClickHouse data directory and reinstalling deletes all stored logs, audit records and events. Precondition: only when this cause is confirmed and losing the stored history is accepted. The ClickHouse pods must be stopped (the plugin uninstalled) before the delete. Rollback: none. + + Uninstall the ClickHouse log storage plugin. Then, on every ClickHouse node, empty the data directory (`/cpaas/data/clickhouse` by default): + + ```bash + cd /cpaas/data/clickhouse && rm -rf ./* + ``` + +3. Reinstall the plugin with resources large enough to avoid another exit code 137. Size them from the observed usage and the free memory on the nodes. + +### Cause 3: razor fails to initialize because ClickHouse refuses to load observability.log_workload with too many suspicious broken parts + +1. Confirm the signature: + + ```bash + kubectl -n cpaas-system describe pod razor-0 | sed -n '/Events:/,$p' + kubectl -n cpaas-system logs -c clickhouse --tail=200 | grep -i 'suspicious' + ``` + +2. Raise the threshold. + + > **Warning:** A higher `max_suspicious_broken_parts_bytes` lets ClickHouse load the table and set the broken parts aside, so rows in those parts can be lost. The edit restarts all ClickHouse pods. Precondition: only when the log shows `log_workload` refused because the suspicious broken parts exceed the limit, disk space is sufficient, and the owner accepts possible log loss (and a reinstall if this does not work). Rollback: remove the setting again (step 4). + + ```bash + kubectl -n cpaas-system get clickhouseinstallations.clickhouse.altinity.com cpaas-clickhouse -o yaml > /tmp/chi.before.yaml + kubectl -n cpaas-system edit clickhouseinstallations.clickhouse.altinity.com cpaas-clickhouse + ``` + + Under `spec.configuration.settings`, add a value above the broken size (10 GiB here): + + ```yaml + merge_tree/max_suspicious_broken_parts_bytes: "10737418240" + ``` + +3. After the pods restart, check the setting and the parts that were set aside: + + ```sql + SELECT name, value, changed FROM system.merge_tree_settings WHERE name LIKE '%suspicious%'; + SELECT database, table, name, reason FROM system.detached_parts WHERE database = 'observability'; + ``` + +4. When `razor` is Ready and the tables answer, restore the default. The edit created a generated resource patch (`rpch`). Deleting that patch returns the setting to 1 GiB, and the pods rebuild once more: + + ```bash + kubectl get rpch -A | grep -i clickhouse + kubectl get rpch -o yaml > /tmp/clickhouse-rpch.yaml + kubectl delete rpch + ``` + + Confirm that `log_workload` still loads. + +### Cause 4: The ClickHouse disk filled, vector inserts failed, and after space returned ClickHouse is missing metadata files + +1. Confirm the signature: + + ```bash + df -hT /cpaas + kubectl -n cpaas-system logs razor-0 -c vector --tail=100 | grep -iE 'NOT_ENOUGH_SPACE|insert|disk|space' + kubectl -n cpaas-system logs razor-0 -c vector -p --tail=100 | grep -E 'Corruption likely|signal: killed' + kubectl -n cpaas-system logs -c clickhouse --tail=100 | grep -iE 'metadata|No such file' + ``` + +2. Free the disk permanently first. Otherwise the same sequence repeats. If `du` on the log node shows the Kafka directory (`/cpaas/data/kafka0`) as the large one, see [Logs, audit and events stop while the log Kafka is degraded on ACP](./Logs_audit_and_events_stop_while_the_log_Kafka_is_degraded_on_ACP.md). + +3. Reinstall with both data directories emptied. + + > **Warning:** Reinstalling ClickHouse and emptying its data directories removes all stored logs. Precondition: only when the disk is no longer full, `razor` still cannot start, and ClickHouse reports missing metadata files (or the `vector` disk buffer reports `Corruption likely`). Confirm the paths against the pod volume mounts on each log node. Rollback: none. + + Uninstall the ClickHouse plugin. Then, on every log node: + + ```bash + cd /cpaas/data/clickhouse && rm -rf ./* + cd /cpaas/data/vector && rm -rf ./* + ``` + +4. Reinstall the plugin, and give the `vector` container in `razor` enough memory. `razor` and log query should come back. + +This every-node cleanup applies only when ClickHouse is missing metadata files and `razor` cannot start. A single filled node, with read-only audit tables while the other replicas are healthy, is handled differently: see the capacity article. + +### Cause 5: The clickhouse container fails its liveness probe on 8123 for days, and clickhouse-operator is in CrashLoopBackOff + +1. Confirm the signature: + + ```bash + kubectl get pod -A | grep -E 'nevermore|razor|clickhouse' + kubectl -n cpaas-system describe pod chi-cpaas-clickhouse-replicated-0-0-0 | sed -n '/Events:/,$p' + kubectl -n ck-system get pods + ``` + +2. Find the host path that backs the logging ClickHouse data volume. The default is `/cpaas/data/clickhouse`. Some installations use `/cpaas/ck`, which on other clusters belongs to the MySQL slow-query ClickHouse (`chi-pxc-ck-slowsql-cluster`). Check which pod owns the path before you empty it: + + ```bash + kubectl -n cpaas-system get pod chi-cpaas-clickhouse-replicated-0-0-0 -o jsonpath='{range .spec.volumes[*]}{.name}{" "}{.hostPath.path}{.persistentVolumeClaim.claimName}{"\n"}{end}' + findmnt -T + ``` + +3. Uninstall, empty the data directory, and redeploy. + + > **Warning:** Uninstalling the ClickHouse log plugin and emptying its data directory deletes the stored logs on those nodes. Precondition: only when the liveness probe on 8123 has failed for a long time, the restarts number in the thousands, free space is not the problem, and step 2 shows that the path belongs to the logging ClickHouse pods and not to `chi-pxc-ck-slowsql-cluster`. Rollback: none. + + Uninstall the ClickHouse log plugin, empty the data directory found in step 2 on each ClickHouse node, and redeploy the plugin. + +4. Check that `clickhouse-operator` is `Running` and that the log collection rate on the dashboard is above 0. + +## Diagnostic Steps + +1. Read the error the console or `razor` reports: + + ```bash + kubectl -n cpaas-system get pods | grep -Ei 'chi-|razor' + kubectl -n cpaas-system logs razor-0 -c razor --tail=50 + kubectl -n cpaas-system describe pod razor-0 | sed -n '/Events:/,$p' + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | `UNKNOWN_TABLE` on some of the requests | Cause 1 | + | A table not found on some replicas after a recovery from exit code 137 | Cause 2 | + | `log_workload` refused because of suspicious broken parts | Cause 3 | + | `NOT_ENOUGH_SPACE` from `vector`, then missing metadata files or a corrupted `vector` buffer after the disk was freed | Cause 4 | + | Thousands of liveness-probe restarts on 8123, with `clickhouse-operator` crash-looping | Cause 5 | + +3. Exit code 137 without liveness-probe failures on `8123`, or `OOMKilled`, is a memory problem. See [Logging ClickHouse runs out of disk, CPU or memory, or is OOMKilled on ACP](./Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md). + +## Related Information + +- [ClickHouse log storage plugin install fails or ClickHouse does not start on ACP](./ClickHouse_log_storage_plugin_install_fails_or_ClickHouse_does_not_start_on_ACP.md) +- [razor does not become ready after a ClickHouse log storage install or upgrade on ACP](./razor_does_not_become_ready_after_a_ClickHouse_log_storage_install_or_upgrade_on_ACP.md) +- [Logging ClickHouse runs out of disk, CPU or memory, or is OOMKilled on ACP](./Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md) diff --git a/docs/en/solutions/ClickHouse_log_storage_fails_on_HTTPS_S3_with_a_self-signed_CA_without_the_clickhouse-ca-certificates_Secret_on_ACP.md b/docs/en/solutions/ClickHouse_log_storage_fails_on_HTTPS_S3_with_a_self-signed_CA_without_the_clickhouse-ca-certificates_Secret_on_ACP.md new file mode 100644 index 000000000..0dff06764 --- /dev/null +++ b/docs/en/solutions/ClickHouse_log_storage_fails_on_HTTPS_S3_with_a_self-signed_CA_without_the_clickhouse-ca-certificates_Secret_on_ACP.md @@ -0,0 +1,99 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.3.x +--- + +# ClickHouse log storage fails on HTTPS S3 with a self-signed CA without the clickhouse-ca-certificates Secret on ACP + +## Issue + +The ClickHouse log storage plugin is installed (from the console or from YAML) with S3 object storage, and the S3 endpoint is served over HTTPS with a certificate signed by a private or self-signed CA. The installation does not complete because ClickHouse in the ClickHouseInstallation (CHI) `cpaas-clickhouse` in `cpaas-system` cannot start. The container stdout may not show the reason. The certificate verification error for the S3 endpoint is written to `clickhouse-server.err.log` inside the ClickHouse container. + +## Root Cause + +ClickHouse verifies the TLS certificate of the S3 endpoint against the CA bundle at `/etc/ssl/certs/ca-certificates.crt` in its container. The default bundle contains only public CAs, so a certificate from a private or self-signed CA fails verification and ClickHouse cannot use the S3 disk. + +The plugin chart supports a custom CA bundle. During rendering it looks for the Secret `clickhouse-ca-certificates` (key `ca-certificates.crt`) in `cpaas-system`: + +- If the Secret exists, the chart mounts it into the ClickHouse container at `/etc/ssl/certs/ca-certificates.crt`. +- If the Secret does not exist, nothing is mounted, and the S3 certificate verification fails. + +This lookup is part of the plugin starting with ACP 4.3.2. The Secret is not created automatically and must be created by hand. The mounted file **replaces** the whole system CA bundle of the container, so it must hold the public CAs as well as the S3 CA. + +## Resolution + +1. Build a complete CA bundle: the system CA bundle followed by the CA certificate(s) of the S3 endpoint. For example, on a Linux host that has a current system bundle at the same path: + + ```bash + cat /etc/ssl/certs/ca-certificates.crt s3-ca.crt > ca-certificates.crt + ``` + + Append each additional CA certificate in PEM format, one after another. + +2. Create the Secret in `cpaas-system`: + + ```bash + kubectl -n cpaas-system create secret generic clickhouse-ca-certificates \ + --from-file=ca-certificates.crt=./ca-certificates.crt + ``` + + The same Secret as YAML: + + ```yaml + apiVersion: v1 + kind: Secret + metadata: + name: clickhouse-ca-certificates + namespace: cpaas-system + type: Opaque + stringData: + ca-certificates.crt: | + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- + ``` + +3. Redeploy the ClickHouse log storage plugin so that the chart renders again and picks up the Secret. Creating the Secret alone does not change a CHI that has already been rendered. + + > **Warning:** Redeploying the log storage plugin restarts ClickHouse, and log, event and audit writes and queries are interrupted until it is running again. Do this only after the diagnostic steps show the S3 certificate error and a CHI without the CA mount. Rollback: delete the Secret and redeploy the plugin again, which brings back the previous rendering. + +4. Check that the ClickHouse Pods become `Running` and that the plugin finishes installing: + + ```bash + kubectl -n cpaas-system get pods | grep -i clickhouse + ``` + +## Diagnostic Steps + +1. Find the ClickHouse Pods and read the server error log inside the container, not only stdout: + + ```bash + kubectl -n cpaas-system get pods | grep -i clickhouse + kubectl -n cpaas-system exec -c clickhouse -- \ + sh -c 'tail -n 200 /var/log/clickhouse-server/clickhouse-server.err.log' + ``` + + Expected for this issue: TLS or certificate verification errors for the S3 endpoint. If the log path differs, look for `clickhouse-server.err.log` under the log directory configured for the container. + +2. Check whether the CA bundle is mounted: + + ```bash + kubectl -n cpaas-system get secret clickhouse-ca-certificates + kubectl -n cpaas-system get chi cpaas-clickhouse -o yaml | grep -n -B2 -A4 clickhouse-ca-certificates + ``` + + Expected for this issue: the Secret is `NotFound` and the CHI has no volume or mount for `clickhouse-ca-certificates`. After the fix, both exist and the container has the bundle at `/etc/ssl/certs/ca-certificates.crt`. + +3. Check that the S3 endpoint certificate chains to the CA you put in the bundle: + + ```bash + openssl s_client -connect : -servername -CAfile ca-certificates.crt /dev/null | grep -E 'Verify return code' + ``` + + Expected after the bundle is correct: `Verify return code: 0 (ok)`. + +4. Rule out look-alikes. When ClickHouse logs that it cannot resolve the Keeper host, or the S3 endpoint is plain HTTP, or the endpoint uses a certificate from a public CA, the CA bundle is not the cause. diff --git a/docs/en/solutions/ClickHouse_log_storage_logs_RaftInstance_errors_with_pod_restarts_merge_CPU_spikes_or_a_hand-built_ClickHouse_on_ACP.md b/docs/en/solutions/ClickHouse_log_storage_logs_RaftInstance_errors_with_pod_restarts_merge_CPU_spikes_or_a_hand-built_ClickHouse_on_ACP.md new file mode 100644 index 000000000..779e976e5 --- /dev/null +++ b/docs/en/solutions/ClickHouse_log_storage_logs_RaftInstance_errors_with_pod_restarts_merge_CPU_spikes_or_a_hand-built_ClickHouse_on_ACP.md @@ -0,0 +1,165 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# ClickHouse log storage logs RaftInstance errors with pod restarts, merge CPU spikes, or a hand-built ClickHouse on ACP + +## Issue + +The ClickHouse log storage (Alauda Container Platform Log Storage for ClickHouse) runs in `cpaas-system` as pods `chi-cpaas-clickhouse-replicated---0` with container `clickhouse`. ClickHouse Keeper runs inside the same container, and its Raft traffic between replicas uses port `9444`. The `clickhouse` container log contains `RaftInstance` errors such as: + +```text + RaftInstance: session 13 failed to read rpc header from socket ... due to error 2, End of file +``` + +Typical questions are: + +- "ClickHouse keeps printing error logs but nothing seems broken. Is this normal?" +- "ClickHouse keeps restarting after the platform upgrade." +- "A `chi-cpaas-clickhouse-replicated` pod has high CPU and its log says `failed to read rpc header from socket`." +- "We deployed our own ClickHouse, switched the log collector to it, and log query is unavailable." + +Sometimes this line is the only symptom: no restarts, and log query and ingestion work. In that case it is benign reconnect noise (Cause 4). The line does not identify a fault by itself, because it also appears next to unrelated faults. Use the other signatures below to find the real cause. + +## Root Cause + +### Cause 1: After an upgrade, the pods restart repeatedly with Election timeout and pre-vote failures while CPU sits at its limit + +After the platform was upgraded (for example from 4.1.3 to 4.2.1), the logging ClickHouse on the `global` cluster kept restarting on all three replicas, with limits of 2 cores / 4Gi per pod. The previous `clickhouse` container log repeats: + +```text + RaftInstance: Election timeout, initiate leader election +RaftInstance: total 1 nodes (including this node) responded for pre-vote (term 16750, live 0, dead 1), at least 2 nodes should respond +failed to send prevote request: peer 3 (chi-cpaas-clickhouse-replicated-0-2.cpaas-system.svc.cluster.local:9444) is busy +peer (1) response error: failed to connect to peer 1, chi-cpaas-clickhouse-replicated-0-0.cpaas-system.svc.cluster.local:9444 +``` + +The `log/Cpaas Log Center for Clickhouse` dashboard shows CPU at 100% of the 2-core limit again and again, and memory at about 20% or less. ClickHouse needs more CPU during startup after the upgrade than the limit allows. As a result, Raft leader election between the replicas fails and the pods restart. The exact internal mechanism has not been determined. Temporarily raising the limits on the cluster plugin (to 16 cores / 16Gi) lets the pods start, and CPU then settles at about 2 cores. Once started, the pods keep running when the limits are set back to 2 cores / 4Gi. + +### Cause 2: CPU of a ClickHouse pod spikes while ClickHouse merges data parts + +A `chi-cpaas-clickhouse-replicated` pod shows high CPU, and its log shows `failed to read rpc header from socket`. The ClickHouse log shows automatic merges of data parts. The scheduled merge tasks run with high concurrency, so CPU spikes for a short time. This is known behavior. It does not affect the logging component, and no remediation is needed. A later release is expected to optimize merge CPU usage. + +### Cause 3: Log query is unavailable because ClickHouse was deployed by hand instead of through the cluster plugin + +A ClickHouse instance was deployed on a cluster by hand, and the log collector was switched to it. Afterwards log query is unavailable. The ClickHouse operator pod and `chi-cpaas-clickhouse-replicated-0-0-0` in `cpaas-system` are `Running`, and the ClickHouse log repeats `RaftInstance: session NNNN failed to read rpc header from socket ... due to error 2, End of file, ref count 1`. The platform log query works only with ClickHouse installed as the standard cluster plugin (Alauda Container Platform Log Storage for ClickHouse). After the plugin was redeployed, log query verification succeeded. + +### Cause 4: RaftInstance failed to read rpc header ... End of file repeats while ClickHouse works normally + +```text + RaftInstance: session NNNN failed to read rpc header from socket : due to error 2, End of file, and reset + RaftInstance: session NNNN failed to read header ... End of file +``` + +The ClickHouse log keeps printing these lines, often at a steady interval (for example every 10 seconds on one replica). The pods do not restart, and log query and ingestion work. The error count can look alarming in a log viewer or a log-keyword alert. + +ClickHouse Keeper keeps long-lived TCP connections between the replicas for Raft (port `9444`). When the far side closes one of these connections, the reading end gets end-of-file. Short-lived connections, such as those opened by readiness probes, end the same way. So do idle connections that a firewall or NAT device between the nodes reclaims. Keeper logs this line each time, then reconnects automatically and rebuilds the Raft session. The message is benign communication noise. It is not a Keeper failure and not a sign that the log store is short of resources, and it needs no action. + +## Resolution + +### Cause 1: After an upgrade, the pods restart repeatedly with Election timeout and pre-vote failures while CPU sits at its limit + +1. Confirm the signature: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep chi-cpaas-clickhouse-replicated + kubectl -n cpaas-system logs -c clickhouse --previous | grep -E 'Election timeout|pre-vote|prevote|failed to connect to peer' | tail -10 + kubectl -n cpaas-system top pod | grep chi-cpaas-clickhouse-replicated + ``` + + Election timeouts and pre-vote failures, with CPU pinned at the limit on every replica, match this cause. Memory far below its limit rules out an out-of-memory kill. + +2. Raise the limits temporarily. + + > **Warning:** Changing the ClickHouse resources on the cluster plugin rolls the ClickHouse pods. Log queries and inserts pause on each pod while it restarts, and `vector` buffers the data in the meantime. Precondition: only when this cause is confirmed, and the nodes have the CPU and memory for the temporary limit. Rollback: set the previous limits on the plugin again. + + Raise the ClickHouse CPU and memory limits on the Log Storage for ClickHouse cluster plugin (Marketplace, Cluster Plugins, select the cluster, update the plugin). Do not change them on the StatefulSets, because the plugin writes the old values back. + +3. Wait until all replicas are `Running` and CPU has settled. If the nodes cannot keep the larger reservation, set the limits back to the previous values. After startup completes, the pods can keep running at 2 cores / 4Gi. + +### Cause 2: CPU of a ClickHouse pod spikes while ClickHouse merges data parts + +Confirm that the spike is merge work and that logging still works: + +```bash +kubectl -n cpaas-system exec -c clickhouse -- \ + clickhouse-client --query "SELECT database, table, elapsed, progress, num_parts FROM system.merges" +kubectl -n cpaas-system top pod | grep chi-cpaas-clickhouse-replicated +``` + +Rows in `system.merges` during the spike, with log query and ingestion working, match this cause. No change is needed. If the pod restarts or queries fail at the same time, use Cause 1, or see [Logging ClickHouse runs out of disk, CPU or memory, or is OOMKilled on ACP](./Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md). + +### Cause 3: Log query is unavailable because ClickHouse was deployed by hand instead of through the cluster plugin + +1. Check whether the ClickHouse log storage is installed as the cluster plugin. On the global cluster: + + ```bash + kubectl get minfo | grep -i clickhouse | grep + ``` + + In the console, also open Marketplace, Cluster Plugins, select the cluster, and look for Alauda Container Platform Log Storage for ClickHouse. A ClickHouse that runs in the cluster with no matching plugin entry matches this cause. + +2. Replace it with the plugin. + + > **Warning:** Removing the hand-built ClickHouse deletes everything it stored. Logs written to it are lost unless they are exported first, and log query stays unavailable until the plugin is installed. Precondition: only when this cause is confirmed and the owner accepts the loss of the hand-built instance's data. Rollback: none for the removed data. + + Remove the hand-built ClickHouse with the same method that created it. Then install Alauda Container Platform Log Storage for ClickHouse on the cluster from Marketplace, Cluster Plugins. Choose the plugin version that supports the current platform version. + +3. Point the log collector at the plugin storage, and query the logs of a pod that is logging now. + +### Cause 4: RaftInstance failed to read rpc header ... End of file repeats while ClickHouse works normally + +1. Confirm that nothing else is wrong: + + ```bash + kubectl -n cpaas-system get pods | grep chi-cpaas-clickhouse-replicated + kubectl -n cpaas-system get pod -o jsonpath='{range .status.containerStatuses[*]}{.name}{" restarts="}{.restartCount}{" last="}{.lastState.terminated.reason}{"\n"}{end}' + kubectl -n cpaas-system logs -c clickhouse --since=1h \ + | grep -v 'RaftInstance' | grep -E '||MEMORY_LIMIT_EXCEEDED|Election timeout|pre-vote|Address already in use' | tail -20 + kubectl -n cpaas-system exec -c clickhouse -- \ + clickhouse-client --query "SELECT name FROM system.zookeeper WHERE path = '/'" + ``` + +2. This cause is confirmed when all of the following hold: + - The restart counts do not grow. + - The second listing is empty, meaning there is no other `` or `` line. + - The Keeper query answers with a list of znodes. + - A console log search returns current entries. + + Ignore the line. No configuration change, restart, or resource change is needed. + +3. If the restart count grows or log query fails, the line is only a bystander. Use Causes 1 to 3, or the articles under Related Information. `MEMORY_LIMIT_EXCEEDED` or CPU and memory alerts are covered in [nevermore and ClickHouse log components alert on CPU or memory, restart, or ignore new resource values on ACP](./nevermore_and_ClickHouse_log_components_alert_on_CPU_or_memory_restart_or_ignore_new_resource_values_on_ACP.md). + +## Diagnostic Steps + +1. Read the pod state and restart counts: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep -E 'clickhouse|razor' + kubectl -n cpaas-system get pod -o jsonpath='{range .status.containerStatuses[*]}{.name}{" restarts="}{.restartCount}{" exit="}{.lastState.terminated.exitCode}{"\n"}{end}' + ``` + +2. Read the previous `clickhouse` log and match it to a cause: + + | What you see | Cause | + |---|---| + | `Election timeout` and `at least 2 nodes should respond`, with CPU at the limit after an upgrade | Cause 1 | + | `failed to read rpc header from socket` with a short CPU spike and active rows in `system.merges` | Cause 2 | + | A ClickHouse that is not installed as the Log Storage for ClickHouse cluster plugin | Cause 3 | + | Only `failed to read rpc header ... End of file`, no restarts, and working log query | Cause 4, no action needed | + +3. Exit code `137` points to a memory limit or a failed liveness probe, not to Raft. See [Logging ClickHouse runs out of disk, CPU or memory, or is OOMKilled on ACP](./Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md) and [ClickHouse log queries fail after the log storage had been working: missing tables or broken parts on ACP](./ClickHouse_log_queries_fail_after_the_log_storage_had_been_working_missing_tables_or_broken_parts_on_ACP.md). + +## Related Information + +- [ClickHouse log storage plugin install fails or ClickHouse does not start on ACP](./ClickHouse_log_storage_plugin_install_fails_or_ClickHouse_does_not_start_on_ACP.md), for first installs that never become ready, including Keeper port errors. +- [Logging ClickHouse runs out of disk, CPU or memory, or is OOMKilled on ACP](./Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md) +- [ClickHouse log queries fail after the log storage had been working: missing tables or broken parts on ACP](./ClickHouse_log_queries_fail_after_the_log_storage_had_been_working_missing_tables_or_broken_parts_on_ACP.md) diff --git a/docs/en/solutions/ClickHouse_log_storage_plugin_install_fails_or_ClickHouse_does_not_start_on_ACP.md b/docs/en/solutions/ClickHouse_log_storage_plugin_install_fails_or_ClickHouse_does_not_start_on_ACP.md new file mode 100644 index 000000000..67a5e7427 --- /dev/null +++ b/docs/en/solutions/ClickHouse_log_storage_plugin_install_fails_or_ClickHouse_does_not_start_on_ACP.md @@ -0,0 +1,167 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# ClickHouse log storage plugin install fails or ClickHouse does not start on ACP + +## Issue + +Alauda Container Platform Log Storage for ClickHouse (module `logclickhouse`) is installed or reinstalled on a cluster and never reaches a working state. Typical reports: + +- "After a redeploy the ClickHouse cluster fails with a port `already in use`, but nothing on the node uses that port." +- "The pre-upgrade inspection says the logging ClickHouse is abnormal." +- "The ClickHouse storage plugin failed while waiting for resources to be ready." +- "`logclickhouse` cannot be installed, and the plugin shows `安装失败` (installation failed)." +- "The logging ClickHouse is not running, and the log only says it cannot find `storage_configuration.xml`." + +The pods in `cpaas-system` are `chi-cpaas-clickhouse-replicated---0` (ClickHouseInstallation `cpaas-clickhouse`) and `razor-` (StatefulSet `razor`). The operator is `clickhouse-operator`. Some background on the install: + +- `vector` is a container inside the `razor` pod, not a separate workload. Deploying ClickHouse log storage deploys `razor` and `vector` automatically. +- ClickHouse Keeper runs inside the ClickHouse container. In the pod, `clickhouse-server` listens on 8123, 9000, 9004, 9005, 9009, 9181 and 9444. +- The default data paths are `/cpaas/data/clickhouse` for ClickHouse and `/cpaas/data/vector` for the `vector` buffer. On nodes with an immutable OS they are under `/var/cpaas` instead. + +This article covers the cases where ClickHouse itself does not come up. When ClickHouse runs but `razor` does not become ready, see [razor does not become ready after a ClickHouse log storage install or upgrade on ACP](./razor_does_not_become_ready_after_a_ClickHouse_log_storage_install_or_upgrade_on_ACP.md). An install with HTTPS S3 storage signed by a private CA is covered in [ClickHouse log storage fails on HTTPS S3 with a self-signed CA without the clickhouse-ca-certificates Secret on ACP](./ClickHouse_log_storage_fails_on_HTTPS_S3_with_a_self-signed_CA_without_the_clickhouse-ca-certificates_Secret_on_ACP.md). + +## Root Cause + +### Cause 1: ClickHouse reports a port already in use although no host process holds it + +The logging ClickHouse fails after a redeploy and reports a port as `already in use`. The failure can first show up as an abnormal component in a pre-upgrade inspection. The ClickHouse log shows: + +```text + Context: Effective user of the process (clickhouse) does not match the owner of the data (root). +Application: Listen [0.0.0.0]:9181 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:9181 +Application: Listen [0.0.0.0]:9009 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:9009 +RaftInstance: deny, req term 6904, my term 6904, req log idx 77357199, my log idx 77357150 +``` + +Nothing listens on that port on the node. The cluster had been deployed more than once, and the PVCs of an earlier deployment were never deleted. State files left in those old volumes make the new cluster believe the port is taken. Inside the pod, `netstat -tulpn` shows `clickhouse-server` itself on 9181 and 9009. Uninstalling the logging component, deleting the old PVCs and deploying again fixes it. Deleting the PVCs is the step that cannot be skipped. + +A variant logs `Unable to resolve placement (cpaas-clickhouse-keeper)` beside the same `Address already in use` lines for 9181 and 9009, and `RaftInstance: peer (2) response error: failed to connect to peer 2`. The Keeper quorum never forms, ClickHouse is not ready, and Vector and razor fail with connection errors. That placement failure has not been fully diagnosed. The install succeeded after the leftover contents of `/cpaas/data` on the ClickHouse nodes were deleted before the install. Using Elasticsearch log storage instead is the alternative when this does not help. + +Lines such as `RaftInstance: session ... failed to read rpc header from socket ... End of file` about every 10 seconds, or `CgroupsReader: Cannot find 'kernel' in '/sys/fs/cgroup/memory.stat'`, also appear on clusters that work. They are not the failure. + +### Cause 2: The install fails and the operator ArtifactVersions stay empty because CPU passthrough is off + +Deploying the cluster plugin on a workload cluster ends in `安装失败` (installation failed) with: + +```text +failed to install chart for release "logclickhouse-operators": execution error at (install-operators/templates/subscriptions.yml:68:26): PackageManifest "cpaas-logging/clickhouse-operator" not exists. +``` + +The operator packages had been uploaded. The control-plane logs repeat `"wait csv" err="packagemanifests.packages.operators.coreos.com \"\" not found"`. `kubectl get artifactversion -A` lists `clickhouse-operator.` and the other operators in `cpaas-system` with empty `VERSION`, `PHASE` and `REASON` columns. + +The node VMs did not have CPU passthrough enabled. ClickHouse binaries require the SSE 4.2 instruction set, and a virtual CPU model that hides host CPU flags does not expose it. Without it the operator artifacts never became ready, and no PackageManifest was produced. After passthrough was enabled, the install succeeded. + +### Cause 3: ClickHouse does not start and its log stops at Can not find storage_configuration.xml in configmap, skip + +```text +[INFO] Try to parse HOST and DOMAIN ... [INFO] Try to generate keeper settings ... [INFO] Can not find storage_configuration.xml in configmap, skip +``` + +These lines are normal initialization messages, not the error. The resources configured for the ClickHouse component were too small for it to start. Raising them on the plugin let ClickHouse start normally. + +## Resolution + +### Cause 1: ClickHouse reports a port already in use although no host process holds it + +1. Confirm the signature: + + ```bash + kubectl -n cpaas-system get pod | grep -E 'click|raz' + kubectl -n cpaas-system logs -c clickhouse --tail=50 | grep -Ei 'already in use|placement' + ss -ltnp | grep -E ':(8123|9000|9009|9181|9444)\b' + kubectl -n cpaas-system get pvc | grep -Ei 'clickhouse|razor' + ``` + + Run `ss` on the node. This cause matches when the pod log says `already in use`, `ss` shows no listener on the node, and some PVCs are older than the current install. To see that ClickHouse itself holds the port inside the pod: + + ```bash + kubectl -n cpaas-system exec -c clickhouse -- netstat -tulpn + ``` + +2. Uninstall the logging component, then delete the leftover PVCs. + + > **Warning:** Deleting the old ClickHouse PVCs destroys the log data in them, and log search stays empty until the new cluster ingests again. Precondition: only when this cause is confirmed, the logging component is already uninstalled, and every PVC you delete belongs to an earlier ClickHouse deployment. Rollback: none; local volumes are gone together with their data. + + ```bash + kubectl -n cpaas-system delete pvc + ``` + + Repeat this for each leftover PVC, and confirm that the PVs are released or deleted. If the PVs use node-local storage and keep their files, also empty the host path printed in the PV's `spec.local.path`. + +3. Deploy the logging component again. + +4. For the `Unable to resolve placement (cpaas-clickhouse-keeper)` variant without leftover PVCs, list what is left under `/cpaas/data` on each ClickHouse node (`du -sh /cpaas/data/*`). + + > **Warning:** Deleting the contents of `/cpaas/data` removes every log storage data directory on that node, including Elasticsearch, Kafka or ZooKeeper data if those also run there. Precondition: only when the logging component is uninstalled, the placement error persists, and the directories hold nothing that must be kept. Rollback: none. + + Delete the leftover contents, then install again. If the placement error persists, use Elasticsearch log storage on that cluster instead. + +### Cause 2: The install fails and the operator ArtifactVersions stay empty because CPU passthrough is off + +1. Confirm the signature: + + ```bash + kubectl get artifactversion -A + kubectl get packagemanifest -A | grep -i clickhouse + grep -o -w -E 'sse4_2|avx|avx2' /proc/cpuinfo | sort | uniq -c + ``` + + Run the `grep` on a node VM. Empty `VERSION`, `PHASE` and `REASON` columns on the operator ArtifactVersions, no `clickhouse-operator` PackageManifest, and missing CPU flags all match this cause. + +2. Enable CPU passthrough on the hypervisor. + + > **Warning:** Changing the VM CPU mode needs a VM restart, which stops every pod on that node. Precondition: the node is cordoned and drained (`kubectl cordon ` and `kubectl drain --ignore-daemonsets --delete-emptydir-data`), and the hypervisor allows host passthrough for that VM. Rollback: switch the CPU mode back and restart again. + + Set the CPU mode of the node VMs to host passthrough, restart them one at a time, and run `kubectl uncordon ` when each node is back. + +3. Retry the plugin install. + +### Cause 3: ClickHouse does not start and its log stops at Can not find storage_configuration.xml in configmap, skip + +1. Check the current resources and events: + + ```bash + kubectl -n cpaas-system describe pod | grep -E 'Limits|Requests|OOMKilled|Events' -A3 + kubectl get moduleinfo -o yaml | grep -n -A6 resources + ``` + +2. Raise the resources. + + > **Warning:** Changing the resources restarts the ClickHouse pods. Precondition: the nodes have free CPU and memory for the new values. Rollback: set the previous values on the plugin. + + Raise the ClickHouse CPU and memory on the log storage plugin (Marketplace, Cluster Plugins, update the plugin). Size them from the node capacity. No single value is prescribed. + +## Diagnostic Steps + +1. Read the plugin and pod state: + + ```bash + kubectl get ars -A | grep -Ei 'click|log' + kubectl get artifactversion -A | grep -i clickhouse + kubectl -n cpaas-system get pods | grep -Ei 'click|razor' + kubectl -n ck-system get pods 2>/dev/null + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | `already in use` with no host listener, or `Unable to resolve placement (cpaas-clickhouse-keeper)` | Cause 1 | + | `PackageManifest "cpaas-logging/clickhouse-operator" not exists`, or empty operator ArtifactVersions on VMs without CPU passthrough | Cause 2 | + | A log that stops at the `storage_configuration.xml` line, with small resource limits | Cause 3 | + +3. If ClickHouse runs and `razor` stays not ready, the failure is in the `razor` initialization. See [razor does not become ready after a ClickHouse log storage install or upgrade on ACP](./razor_does_not_become_ready_after_a_ClickHouse_log_storage_install_or_upgrade_on_ACP.md). + +## Related Information + +- Related articles: [razor does not become ready after a ClickHouse log storage install or upgrade on ACP](./razor_does_not_become_ready_after_a_ClickHouse_log_storage_install_or_upgrade_on_ACP.md) covers the other first-install and upgrade failures of the same plugin. +- [ClickHouse log storage logs RaftInstance errors with pod restarts, merge CPU spikes, or a hand-built ClickHouse on ACP](./ClickHouse_log_storage_logs_RaftInstance_errors_with_pod_restarts_merge_CPU_spikes_or_a_hand-built_ClickHouse_on_ACP.md) +- [ClickHouse log queries fail after the log storage had been working: missing tables or broken parts on ACP](./ClickHouse_log_queries_fail_after_the_log_storage_had_been_working_missing_tables_or_broken_parts_on_ACP.md) diff --git a/docs/en/solutions/Clients_or_a_backup_repository_cannot_connect_to_Ceph_object_storage_on_ACP.md b/docs/en/solutions/Clients_or_a_backup_repository_cannot_connect_to_Ceph_object_storage_on_ACP.md new file mode 100644 index 000000000..93b251674 --- /dev/null +++ b/docs/en/solutions/Clients_or_a_backup_repository_cannot_connect_to_Ceph_object_storage_on_ACP.md @@ -0,0 +1,141 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# Clients or a backup repository cannot connect to Ceph object storage on ACP + +## Issue + +Applications or a backup repository cannot connect to Ceph object storage (RGW) on Alauda Container Platform: + +- Clients report a connection error and production calls fail ("Ceph cannot be connected"). Under **Platform Management > Storage Management > Distributed Storage**, Ceph itself is up. The object storage access addresses show an internal address `rook-ceph-rgw-.rook-ceph:7480` and an external address `:`, and the node in that external address is down. A follow-up question is how to avoid this single point of failure. +- Associating Ceph object storage with a backup repository fails. The Velero `BackupStorageLocation` (`bsl`) named `ceph` in `cpaas-system` is `phase: Unavailable` with: + + ```text + BackupStorageLocation "ceph" is unavailable: ... operation error S3: ListObjectsV2, ... request send failed, Get "http://rook-ceph-rgw-object-store.rook-ceph:7480/?...": dial tcp: lookup rook-ceph-rgw-object-store.rook-ceph on :53: no such host + ``` + + later `StatusCode: 404, ... api error NoSuchBucket`, and on a business cluster `kubectl get bsl` returns `error: the server doesn't have a resource type "bsl"`. + +Services in `rook-ceph`: `rook-ceph-rgw-` (ClusterIP, port `7480` to container port `8080`) and `rook-ceph-rgw--external` (NodePort, port `7480` to container port `8080`). + +## Root Cause + +### Cause 1: clients use the single node address the UI shows, and that node is down + +The external access is a NodePort Service, which is reachable on the NodePort of every node in the cluster. The external address shown in the UI is one node picked for display; it is not a virtual IP that follows a healthy node. Clients configured with only that address lose the connection when that server is down. Ceph itself does not need any repair. + +Two ways to remove the single point of failure: + +1. Put an external load balancer in front of the same NodePort on several nodes. +2. Expose RGW with a Service of type LoadBalancer (or adjust internal routing in the RGW namespace). This is a backend change; the console does not offer it for object storage. + +### Cause 2: the BackupStorageLocation is Unavailable because of a missing plugin and an in-cluster endpoint + +Two problems keep the backup repository down, one after the other: + +1. The backup plugin is not installed on the cluster that runs the backups, so that cluster has no `bsl` resource type at all. +2. The backup repository endpoint is the in-cluster Service name `rook-ceph-rgw-object-store.rook-ceph:7480`, which does not resolve from where the backup client runs (`no such host`). Once the name resolves, a wrong folder setting produces `NoSuchBucket`. + +## Resolution + +### Cause 1: clients use the single node address the UI shows, and that node is down + +1. Confirm the node state and Ceph health: + +```bash +kubectl get nodes +kubectl -n rook-ceph get svc | grep -i rgw +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s +``` + +2. If `ceph -s` is healthy and the node in the published address is `NotReady` or powered off, recover that server; access returns when it is back. Meanwhile, clients can use any other Ready node's address on the same NodePort. The UI keeps showing the single node it picked. +3. For a setup that survives one node, point an external load balancer (for example an F5) at the NodePort on several Ready nodes: + +```bash +kubectl -n rook-ceph get svc rook-ceph-rgw--external -o jsonpath='{.spec.ports[0].nodePort}{"\n"}' +kubectl get nodes -o wide +``` + +4. Alternatively, create a LoadBalancer Service for RGW. The manifest below is the generic Kubernetes form; reuse the selector of the existing in-cluster RGW Service: + +```bash +kubectl -n rook-ceph get svc rook-ceph-rgw- -o jsonpath='{.spec.selector}{"\n"}' +``` + +> **Warning:** A LoadBalancer Service publishes RGW on an address outside the cluster; anyone who can reach it can send S3 requests. Precondition: the cluster has a working LoadBalancer implementation, and the security owner approves the exposure. Rollback: `kubectl -n rook-ceph delete svc rook-ceph-rgw--lb`; the existing Services keep serving. + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: rook-ceph-rgw--lb + namespace: rook-ceph +spec: + type: LoadBalancer + selector: + # use the selector printed by the command above + app: rook-ceph-rgw + rook_object_store: + ports: + - name: http + port: 7480 + targetPort: 8080 + protocol: TCP +``` + +`kubectl -n rook-ceph get svc rook-ceph-rgw--lb` shows the external IP once it is assigned. Clients then use `http://:7480`. + +### Cause 2: the BackupStorageLocation is Unavailable because of a missing plugin and an in-cluster endpoint + +1. Check that the backup plugin is installed on the cluster that runs the backups: + +```bash +kubectl -n cpaas-system get bsl +kubectl -n cpaas-system get bsl ceph -o yaml +``` + +`error: the server doesn't have a resource type "bsl"` means the plugin is missing there. Install the backup plugin on that cluster from the platform plugin management, as on the other clusters, then read the BSL again. + +2. On the cluster that runs Ceph, get the external RGW address: + +```bash +kubectl -n rook-ceph get svc | grep rgw +kubectl get nodes -o wide +``` + +3. In the backup repository settings, set the endpoint to `http://:` (the `-external` NodePort) instead of the in-cluster name, and set the folder to `/`. Keep the provider (`aws`), `s3ForcePathStyle: "true"`, and the credential Secret as they are. +4. Confirm with an S3 client that the bucket in the repository settings exists under the same keys; `NoSuchBucket` means RGW was reached and the bucket name or path did not match: + +```bash +s3cmd --host=: --host-bucket=: --no-ssl --access_key= --secret_key= ls +``` + +5. Wait for the BSL to become available: + +```bash +kubectl -n cpaas-system get bsl ceph -o jsonpath='{.status.phase}{"\n"}' +``` + +A node-address endpoint has the single-node exposure described in Cause 1. If that node goes down, the backup repository loses RGW the same way; a load balancer address avoids that. + +## Diagnostic Steps + +```bash +kubectl get nodes -o wide +kubectl -n rook-ceph get svc -o wide | grep -i rgw +kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s +``` + +- The node in the published address is down and `ceph -s` is healthy: Cause 1. +- `ceph -s` is not healthy: fix the Ceph cluster first; an address change does not replace a down OSD. +- A BSL `Unavailable` with `no such host` for `rook-ceph-rgw-.rook-ceph`, `NoSuchBucket`, or no `bsl` resource type on the cluster: Cause 2. +- The client connects and then gets `InvalidAccessKeyId`: a credentials problem. A `Read timed out` after the bucket was found: see [Ceph OSDs nearfull, backfill blocked, or OSD usage uneven on ACP](./Ceph_OSDs_nearfull_backfill_blocked_or_OSD_usage_uneven_on_ACP.md), Cause 7. +- A `CephObjectStore` stuck in `Deleting` with `no route to host` to its own RGW: see [CephObjectStore stays in Deleting with failed to list buckets no route to host on ACP](./CephObjectStore_stays_in_Deleting_with_failed_to_list_buckets_no_route_to_host_on_ACP.md). diff --git a/docs/en/solutions/Cluster_Overview_page_shows_no_monitoring_charts_after_Cluster_Enhancer_is_installed_on_ACP.md b/docs/en/solutions/Cluster_Overview_page_shows_no_monitoring_charts_after_Cluster_Enhancer_is_installed_on_ACP.md new file mode 100644 index 000000000..a80315278 --- /dev/null +++ b/docs/en/solutions/Cluster_Overview_page_shows_no_monitoring_charts_after_Cluster_Enhancer_is_installed_on_ACP.md @@ -0,0 +1,62 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Cluster Overview page shows no monitoring charts after Cluster Enhancer is installed on ACP + +## Issue + +Under **Cluster Management** > **Clusters**, clicking a cluster opens its **Overview** (`概览`) page, and there are no monitoring charts below the cluster information. The chart area is missing entirely; this is not a set of charts that keep loading or report an error. The question that usually follows is whether the charts only appear once the Cluster Enhancer plugin is installed. + +This article applies when the **Alauda Container Platform Cluster Enhancer** plugin is installed on that cluster. In **App Marketplace Management** > **Cluster Plugins** the plugin is listed as installed, with a description saying that it enhances the cluster by backing up etcd data, monitoring certificates, and cleaning up events. + +If Cluster Enhancer is **not** installed and the cluster or host dashboards are missing from **Operations Center** > **Monitoring**, see Cause 3 of [Monitoring page freezes the browser or built-in dashboards are missing on ACP](Monitoring_page_freezes_the_browser_or_built_in_dashboards_are_missing_on_ACP.md). Charts that are present but spin or stay empty are covered in [Monitoring pages return 502 or context deadline exceeded because port 11780 on the cluster VIP is blocked on ACP](Monitoring_pages_return_502_or_context_deadline_exceeded_because_port_11780_on_the_cluster_VIP_is_blocked_on_ACP.md). + +## Root Cause + +### Cause 1: the Overview monitoring panels are removed when Cluster Enhancer is installed + +This is expected behavior, not a collection or query fault. Once Cluster Enhancer is installed on a cluster, the monitoring panels on that cluster's **Overview** page are removed. Cluster, node, container, and Pod monitoring for that cluster is shown instead in **Operations Center** > **Monitoring** > **Monitoring Dashboards**, in the built-in dashboards of the `kubernetes` group. + +## Resolution + +### Cause 1: the Overview monitoring panels are removed when Cluster Enhancer is installed + +There is nothing to repair. Use the built-in dashboards instead: + +1. Open **Operations Center** > **Monitoring** > **Monitoring Dashboards**. +2. Select the cluster and open the `kubernetes` group. +3. Use its built-in dashboards: + + | Dashboard | Name in the Chinese UI | + | --- | --- | + | Cluster Status | `集群状态` | + | Container Monitoring | `容器监控` | + | Node Monitoring | `节点监控` | + | Pod Monitoring | `容器组监控` | + +Because installing Cluster Enhancer is what removes the Overview panels, installing or reinstalling it does not bring them back. There is no setting that restores monitoring panels on the Overview page. + +If those `kubernetes` dashboards are themselves empty, treat it as a separate fault: see [Monitoring dashboard panels are blank or drop to zero while the backend has data on ACP](Monitoring_dashboard_panels_are_blank_or_drop_to_zero_while_the_backend_has_data_on_ACP.md) when the API returns data, or the port 11780 article above when the browser developer tools show HTTP 500, 502, or `context deadline exceeded`. + +## Diagnostic Steps + +1. Confirm which page is empty. The symptom here is the **Overview** page under **Cluster Management** > **Clusters**, with no chart area at all. + +2. On that cluster, list the dashboards that Cluster Enhancer synchronizes: + + ```bash + kubectl get monitordashboards.ait.alauda.io -n cpaas-system -l helm.sh/chart-name=cluster-enhancer + ``` + + Expected for this article: objects are listed, so the plugin is installed and its dashboards are present. If nothing is listed, the plugin is not installed or has not synchronized; use the "built-in dashboards are missing" article linked in the Issue section. + +3. Open the `kubernetes` group under **Operations Center** > **Monitoring** > **Monitoring Dashboards** for the same cluster. Expected: **Cluster Status** and **Node Monitoring** show data, which confirms that monitoring works and only the Overview panels were removed. To see the dashboard objects behind the group: + + ```bash + kubectl get monitordashboards.ait.alauda.io -A | grep -i kubernetes + ``` diff --git a/docs/en/solutions/Cluster_import_stuck_with_alb_module_Failed_grpcroutes_CRD_storedVersions_v1_must_appear_in_spec_versions_on_ACP.md b/docs/en/solutions/Cluster_import_stuck_with_alb_module_Failed_grpcroutes_CRD_storedVersions_v1_must_appear_in_spec_versions_on_ACP.md new file mode 100644 index 000000000..108267d10 --- /dev/null +++ b/docs/en/solutions/Cluster_import_stuck_with_alb_module_Failed_grpcroutes_CRD_storedVersions_v1_must_appear_in_spec_versions_on_ACP.md @@ -0,0 +1,89 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x +--- + +# Cluster import stuck with the alb module Failed: grpcroutes CRD status.storedVersions v1 must appear in spec.versions on ACP + +## Issue + +Importing an existing Kubernetes cluster (for example a managed cloud cluster such as Alibaba Cloud ACK) into ACP 4.0.11 never finishes and stays at the import step. On the `global` cluster, the cluster's `alb` module is `Failed`, while other modules such as `cert-manager`, `acp-business`, `marketplace`, `auth-manager`, `base` and `sentry` are `Running`. + +On the imported cluster, the `alauda-alb2` chart release keeps failing with phase `DeployFailed` and this message: + +```text +failed to install chart acp/chart-alauda-alb2-, err: failed to install chart for release "alauda-alb2": +failed to install CRD: failed to apply CustomResourceDefinition "/grpcroutes.gateway.networking.k8s.io": +CustomResourceDefinition.apiextensions.k8s.io "grpcroutes.gateway.networking.k8s.io" is invalid: +status.storedVersions[0]: Invalid value: "v1": must appear in spec.versions +``` + +The release shows `Validate` = `True` (`ChartDownloaded`), `Sync` = `False` (`Failed`) and `Health` = `Unknown`, and a Warning `ChartSynced` event repeats thousands of times. + +## Root Cause + +The target cluster already had a `grpcroutes.gateway.networking.k8s.io` CRD before it was imported, installed by something other than the platform, for example an earlier Gateway API installation. Objects of that CRD have been stored as version `v1`, so `v1` is recorded in the CRD's `status.storedVersions`. + +The ALB chart installs its own definition of the same CRD, and that definition does not list `v1` in `spec.versions`. The Kubernetes API server rejects any CRD update that would drop a version still listed in `status.storedVersions`, so applying the chart's CRD fails. The `alauda-alb2` release cannot be installed, the `alb` module stays `Failed`, and the import never reaches the ready state. + +## Resolution + +1. On the imported cluster, confirm the conflict and list any GRPCRoute objects that would be lost: + + ```bash + kubectl get crd grpcroutes.gateway.networking.k8s.io \ + -o jsonpath='spec.versions={.spec.versions[*].name} storedVersions={.status.storedVersions}{"\n"}' + kubectl get grpcroutes.gateway.networking.k8s.io -A + ``` + + Continue only if `storedVersions` contains `v1` and the release message is the one shown in the Issue section. + +2. Back up the CRD and its objects: + + ```bash + kubectl get crd grpcroutes.gateway.networking.k8s.io -o yaml > grpcroutes-crd.yaml + kubectl get grpcroutes.gateway.networking.k8s.io -A -o yaml > grpcroutes.yaml + ``` + +3. Delete the conflicting CRD. + + > **Warning:** Deleting a CRD deletes every object of that kind in the cluster. Run this only when the `storedVersions` conflict above is confirmed, and only after step 1 shows no GRPCRoute objects you still need, or they are saved by step 2. Whatever installed the existing CRD may also depend on it. To roll back, run `kubectl create -f grpcroutes-crd.yaml` and then `kubectl create -f grpcroutes.yaml`. The objects can only be restored while their version is still served by the CRD that is installed at that time. + + ```bash + kubectl delete crd grpcroutes.gateway.networking.k8s.io + ``` + +4. The chart release is retried automatically. Watch it on the imported cluster until it is synced, and confirm the `alb` module becomes `Running` on `global`: + + ```bash + kubectl get ars -A | grep -i alb + kubectl get moduleinfo -l cpaas.io/cluster-name= # on the global cluster + ``` + + Once all modules are `Running`, the import completes. + +## Diagnostic Steps + +1. On `global`, find the module that blocks the import. A stuck import with only `alb` `Failed` points at the ALB chart: + + ```bash + kubectl get moduleinfo -l cpaas.io/cluster-name= + ``` + +2. On the imported cluster, read the ALB chart release status. The `storedVersions[0]: Invalid value: "v1": must appear in spec.versions` message on `grpcroutes.gateway.networking.k8s.io` identifies this issue. A different message, for example an API version the cluster does not serve, is a different problem: + + ```bash + kubectl get ars -A | grep -i alb + kubectl get ars -n -o yaml + ``` + +3. Confirm that the CRD existed before the import and was not created by the platform. Its creation time is earlier than the import and its versions include `v1`: + + ```bash + kubectl get crd grpcroutes.gateway.networking.k8s.io \ + -o jsonpath='{.metadata.creationTimestamp} {.spec.versions[*].name}{"\n"}' + ``` diff --git a/docs/en/solutions/Cluster_plugin_install_or_upgrade_stuck_in_chart_sync_sentry_StatefulSet_Forbidden_hook_Job_or_missing_ModuleConfig.md b/docs/en/solutions/Cluster_plugin_install_or_upgrade_stuck_in_chart_sync_sentry_StatefulSet_Forbidden_hook_Job_or_missing_ModuleConfig.md new file mode 100644 index 000000000..7f12d6c7f --- /dev/null +++ b/docs/en/solutions/Cluster_plugin_install_or_upgrade_stuck_in_chart_sync_sentry_StatefulSet_Forbidden_hook_Job_or_missing_ModuleConfig.md @@ -0,0 +1,175 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "4.1" + - "4.2" + - "4.3" +--- + +# Cluster plugin install or upgrade stuck in chart sync: sentry, StatefulSet Forbidden, hook Job, or missing ModuleConfig + +## Issue + +A cluster plugin page shows `Installing` (Chinese UI: `安装中`) or `Deployment Failed` (`安装失败`), and the AppRelease (`ars`) or ModuleInfo carries a chart error: + +- Every plugin, including Alauda AI Essentials, is stuck in Installing, and no Pod in the cluster is abnormal. +- `metallb ChartDownloadFailed [1/1] failed to download chart acp/chart-alauda-metallb-plugin-v4.0.12, err: calculating blob path for ReaderAt: cannot calculate blob path from invalid digest: invalid checksum digest length: invalid argument`. +- The log center plugin (for example Log Storage for Elasticsearch `v4.1.3` on `global`) stays Installing, the Kafka consumer-lag alert from rule `cpaas-log-rules` keeps firing (`Kafka 消费队列滞后增长速率 > 0 且持续 1 分钟`, "Kafka consumer lag growth rate > 0 for 1 minute"), and the event is `failed to apply StatefulSet "cpaas-system/cpaas-kafka": StatefulSet.apps "cpaas-kafka" is invalid: spec: Forbidden: updates to statefulset spec for fields other than ...`. +- `Installation failed:[1/1] chart ait/chart-base-central-v4.2.44-cn Upgrading` for Alauda Container Platform Central Base during platform deployment. +- A business cluster ARS (for example `base`, before a cluster upgrade) stays `syncing` for a long time. +- A plugin upgrade on a business cluster (for example Prometheus) fails, and the error looks like a missing certificate. + +`sentry` installs and upgrades plugin charts on each cluster and records the result on the AppRelease. `cluster-transformer` on `global` turns ModulePlugins into ModuleConfigs and reconciles ModuleInfo. + +Related articles: "Argo Rollouts, Kubeflow Base, or a wrong-architecture cluster plugin fails to install or upgrade on ACP" covers plugin-specific install failures. + +## Root Cause + +### Cause 1: sentry stopped processing charts + +All plugins stay Installing with no abnormal Pod, or an AppRelease reports `ChartDownloadFailed ... invalid checksum digest length: invalid argument`. No underlying cause has been identified. Restarting `sentry` returns the plugins from Installing to normal and restores the AppRelease sync. + +### Cause 2: StatefulSet spec Forbidden after storageSize was changed in ModuleInfo + +`sentry failed to upgrade chart ait/chart-alauda-log-center-v4.1.3, err: failed to upgrade chart for release "logcenter": failed to apply StatefulSet "cpaas-system/cpaas-kafka" ... spec: Forbidden` repeats for days, and the log center plugin stays Installing. + +The Kafka `storageSize` in the log center ModuleInfo was changed. That value feeds the `volumeClaimTemplates` of the existing `cpaas-kafka` StatefulSet, which Kubernetes does not allow to change in place, so every chart upgrade fails. Kafka consumer-lag alerts on `alauda_log/ALAUDA_LOG_TOPIC/` can fire at the same time. The fix is to redeploy the log plugins with the planned PVC capacity, which deletes the existing log data. + +### Cause 3: Job remove-keep-annotations times out and the chart never leaves Upgrading + +`Installation failed:[1/1] chart ait/chart-base-central-v4.2.44-cn Upgrading`, and the `sentry` log for release `base-central` in `cpaas-system` shows: + +```text +warning: Failed to update release base-central version 4: release: not found +ServiceAccount base-central-upgrade-hook has skipped +recreate Job "remove-keep-annotations" success +Watching for changes to Job remove-keep-annotations with timeout of 2m0s +Jobs active: 0, jobs failed: 0, jobs succeeded: 0 +``` + +The `release: not found` warning alone does not identify this cause; the deciding lines are the wait on the Job and `Jobs active: 0`. The chart upgrade waits on hook Job `remove-keep-annotations`, which never becomes active, so the wait times out on every retry. Deleting the Job and running its operations by hand lets the installation complete. + +### Cause 4: Business cluster ARS stays syncing because a BroadcastJob is incomplete + +A business cluster ARS (or its `base` component) stays in `syncing` while a BroadcastJob of that cluster is incomplete: `update-containerd-config` with stuck jobs, or `upgrade-sysctl-config` with `FAILED 1` and an `Evicted` Pod. Adding a dummy value to the ARS values only triggers the same BroadcastJob again. Both forms are covered in "base AppRelease stays failed because the upgrade-sysctl-config BroadcastJob cannot finish on a large cluster on ACP". + +### Cause 5: ModuleInfo Reconcile error: ModuleConfig.cluster.alauda.io "" not found + +A plugin upgrade on a business cluster fails, and the `cluster-transformer` log shows: + +```text +ERROR controllers/moduleinfo_controller.go:61 ModuleInfo Reconcile '' error: ModuleConfig.cluster.alauda.io "prometheus-v3.16.11-def68b41a866" not found +``` + +The ModuleConfig of the old version (3.16) was deleted, and the chart version in the ModulePlugin `spec.mainChart` no longer matches any existing ModuleConfig, so ModuleInfo reconciliation fails. Setting that chart `version` to the version of the ModuleConfig that exists fixes it. + +## Resolution + +### Cause 1: sentry stopped processing charts + +```bash +kubectl get ars -A +kubectl get pods -A | grep sentry +kubectl -n cpaas-system logs --tail=300 > "$HOME/sentry-before-restart.log" +``` + +> **Warning:** Restarting `sentry` pauses chart install and sync on that cluster until the new Pod is Ready. Precondition: only when every plugin is stuck Installing with no abnormal Pod, or an AppRelease shows `ChartDownloadFailed` with `invalid checksum digest length`. Rollback: none needed; the Deployment recreates the Pod. + +```bash +kubectl -n cpaas-system delete pod +kubectl -n cpaas-system get pods -w | grep sentry +kubectl get ars -A +``` + +Plugins should leave Installing and the AppRelease should sync. Keep the saved `sentry` log for Alauda support if the problem returns. + +### Cause 2: StatefulSet spec Forbidden after storageSize was changed in ModuleInfo + +```bash +kubectl get ars -A | grep -i logcenter +kubectl -n cpaas-system describe ars logcenter | grep -A3 ChartSynced +kubectl get minfo | grep -i log +kubectl get minfo -o yaml | grep -n -i storageSize +kubectl -n cpaas-system get sts cpaas-kafka -o jsonpath='{.spec.volumeClaimTemplates[*].spec.resources.requests.storage}{"\n"}' +``` + +A `storageSize` in ModuleInfo that differs from the StatefulSet `volumeClaimTemplates` storage is this cause. The chart upgrade can only apply while the two values match or after the StatefulSet is recreated, which here means redeploying the plugin. + +> **Warning:** Redeploying the log plugin with new PVC sizes deletes the existing log storage data (the `cpaas-kafka` volumes and the stored logs). This cannot be undone. Precondition: only when the event shows the `cpaas-kafka` StatefulSet `spec: Forbidden` error, the `storageSize` change in ModuleInfo is confirmed, and the owner of the logs has confirmed the existing log data can be deleted. Rollback: none; export anything you must keep before you start. + +Uninstall the log collection plugin, confirm its PVCs are gone, and install it again with the planned PVC capacity: + +```bash +kubectl -n cpaas-system get pvc | grep -i kafka +``` + +### Cause 3: Job remove-keep-annotations times out and the chart never leaves Upgrading + +Find the Job in the `sentry` log and export it before touching it: + +```bash +kubectl -n cpaas-system logs --tail=500 | grep -E 'base-central|remove-keep-annotations|Watching for changes to Job|Jobs active' +kubectl get jobs -A | grep remove-keep-annotations +kubectl -n describe job remove-keep-annotations +kubectl -n get job remove-keep-annotations -o yaml > "$HOME/remove-keep-annotations.yaml" +``` + +> **Warning:** Deleting the Job removes the step the chart waits on; the operations that Job performs are not done until you run them yourself. Precondition: only when the chart is stuck at `chart ait/chart-base-central- Upgrading`, the `sentry` log shows the wait on Job `remove-keep-annotations` timing out with `Jobs active: 0`, and the Job is exported. Rollback: re-create the Job from the export if the chart must run it again. + +```bash +kubectl -n delete job remove-keep-annotations +``` + +Run the operations of the exported Job by hand: its container `command` and `args` are under `.spec.template.spec.containers[]` in the export, and the ServiceAccount it runs as is `.spec.template.spec.serviceAccountName`. Then check that the chart leaves `Upgrading`. Use the same procedure for another hook Job only when the `sentry` log shows that Job's wait timing out. + +### Cause 4: Business cluster ARS stays syncing because a BroadcastJob is incomplete + +```bash +kubectl get job,bcj -n cpaas-system +kubectl -n cpaas-system logs --tail=500 | grep -E 'update-containerd-config|upgrade-sysctl-config' +``` + +Follow "base AppRelease stays failed because the upgrade-sysctl-config BroadcastJob cannot finish on a large cluster on ACP": delete the `Evicted` Pod for a `FAILED` `upgrade-sysctl-config`, or raise `parallelism` for a stuck `update-containerd-config`, each with its Warning. + +### Cause 5: ModuleInfo Reconcile error: ModuleConfig.cluster.alauda.io "" not found + +```bash +kubectl -n cpaas-system logs deploy/cluster-transformer --tail=300 | grep -i 'ModuleConfig.cluster.alauda.io' +kubectl get mconfig | grep -i +kubectl get mplugin -o yaml > "$HOME/mplugin-.before.yaml" +grep -n -A8 'mainChart' "$HOME/mplugin-.before.yaml" +``` + +> **Warning:** The chart version in `spec.mainChart` decides which chart the platform installs for this plugin. Precondition: only when the `cluster-transformer` log reports `ModuleConfig.cluster.alauda.io "" not found` for the referenced version, and you set the version to one that `kubectl get mconfig` lists. Rollback: `kubectl apply -f "$HOME/mplugin-.before.yaml"`. + +```bash +kubectl edit mplugin +``` + +Set the `version` of the chart under `spec.mainChart` to the version of the existing ModuleConfig, save, and retry the upgrade. + +## Diagnostic Steps + +```bash +kubectl get ars -A +kubectl get minfo | grep +kubectl get pods -A | grep -E 'sentry|cluster-transformer' +kubectl -n cpaas-system logs deploy/cluster-transformer --tail=300 | grep -i error +kubectl get broadcastjob -A +``` + +- All plugins stuck with healthy Pods, or `invalid checksum digest length`: Cause 1. If `sentry` itself shows `OOMKilled` restarts, raise its memory instead of only restarting it. +- `StatefulSet ... spec: Forbidden` after a `storageSize` change: Cause 2. +- `chart ... Upgrading` with `Watching for changes to Job remove-keep-annotations` and `Jobs active: 0` in the `sentry` log: Cause 3. +- An ARS in `syncing` with an incomplete BroadcastJob (`update-containerd-config`, or `upgrade-sysctl-config` with an `Evicted` Pod): Cause 4. +- `ModuleInfo Reconcile ... ModuleConfig.cluster.alauda.io "" not found` from `cluster-transformer` during a plugin upgrade: Cause 5. + +For a release that reports `release: not found`, check whether Helm still has the release record before acting; a missing revision Secret matches that warning, and the hook Job lines decide Cause 3: + +```bash +kubectl get secret -n cpaas-system | grep 'sh.helm.release.v1.' +``` diff --git a/docs/en/solutions/Cluster_plugin_stuck_at_wait_resources_ready_Pod_Pending_cpaas_IO_error_or_missing_log_secret_on_ACP.md b/docs/en/solutions/Cluster_plugin_stuck_at_wait_resources_ready_Pod_Pending_cpaas_IO_error_or_missing_log_secret_on_ACP.md new file mode 100644 index 000000000..3328b0b2c --- /dev/null +++ b/docs/en/solutions/Cluster_plugin_stuck_at_wait_resources_ready_Pod_Pending_cpaas_IO_error_or_missing_log_secret_on_ACP.md @@ -0,0 +1,176 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.2" + - "4.3" +--- + +# Cluster plugin stuck at wait resources ready: Pod Pending, /cpaas I/O error, or missing log secret on ACP + +## Issue + +The plugin chart is applied, and the install or upgrade then waits for the plugin's workloads to become ready and never gets there: + +- The multus plugin installation hangs in Deploying. `kubectl describe ars -n cpaas-system multus` shows Reason `NotReady` with `failed to wait chart alauda-multus-plugin-v3.18.3 state, err: timed out waiting for the condition`. +- The Elasticsearch log storage plugin upgrade stays at the `CheckStatus-logcenter` step, and the progress log repeats `WaitReady: wait resources ready`. +- After the 4.3.2 logging packages (`log-agent`, `log-api`, `log-clickhouse`) were pushed, the upgrade button of the logging cluster plugins stays greyed out. Once it appears, the "Alauda Container Platform Log Collector" v4.3.2 upgrade on a business cluster fails (`安装失败`, installation failed) with `wait resources ready`. + +In each case one workload Pod of the plugin cannot become Ready, and the AppRelease (`ars`) waits for it until it times out. A plugin Pod that cannot pull its image is a different problem; see the related article below. + +Related articles: "Operator or plugin ImagePullBackOff or exec format error: registry address, 401, architecture, or search domain on ACP" covers image pull failures; "Cluster plugin install or upgrade stuck in chart sync: sentry, StatefulSet Forbidden, hook Job, or missing ModuleConfig" covers failures before the workloads are created. + +## Root Cause + +### Cause 1: a DaemonSet Pod of the plugin is Pending with Insufficient cpu on one node + +The chart waits for every Pod of its DaemonSets. When one node lacks the CPU that a DaemonSet Pod requests, that Pod stays `Pending` with `Insufficient cpu`, the DaemonSet has one Pod fewer ready than desired (for example `delete-multus-cfg-ds` in `kube-system` at 10/11), and the AppRelease stays in `WaitReady` until it times out. Once CPU is freed on that node, the Pod is scheduled and the installation completes. + +### Cause 2: the log storage Pods on one node fail because /cpaas returns Input/output error + +The log storage components (`cpaas-elasticsearch-`, `cpaas-kafka-`, `cpaas-zookeeper-` in `cpaas-system`) keep their data under `/cpaas` on the node. When the filesystem behind `/cpaas` on one node fails, every log storage Pod scheduled there is `0/1`, and the upgrade never passes `CheckStatus-logcenter` (`prepare-logcenter-upgrade` has already completed). On that node, `ls` in `/cpaas` or `/cpaas/data` returns `ls: cannot open directory .: Input/output error` (or `cannot access 'data': Input/output error`). + +When the I/O error is a transient fault of an LVM-based mount, rebooting the host restores `/cpaas`, and the log storage Pods recover on their own. When the underlying block device or its paths (for example a SAN or multipath device) are failing, a reboot does not help until the storage is repaired. + +### Cause 3: nevermore cannot start, secret log-collector-bearer-auth not found, Log Essentials on global was not upgraded first + +The Log Collector upgrade on the business cluster fails with `wait resources ready`, and the `nevermore` Pods in `cpaas-system` stay `0/4` `Init:0/1` with: + +```text +Warning FailedMount MountVolume.SetUp failed for volume "callback-secret" : secret "log-collector-bearer-auth" not found +``` + +`kubectl get secret -A | grep log-collector` returns nothing on the business cluster. + +The Secret `log-collector-bearer-auth` is distributed to business clusters by the "Alauda Container Platform Log Essentials" plugin on `global`. When Log Essentials on `global` is still on the previous release line (for example v4.2.9) while the business cluster plugins are upgraded to v4.3.2, the Secret does not exist yet. After Log Essentials is upgraded, `log-collector-bearer-auth` and `log-database-bearer-auth` are created in `cpaas-system` on the business cluster, and Log Collector and Log Storage for ClickHouse there finish their upgrade. + +A greyed-out upgrade button after the packages were pushed is cleared by restarting `cluster-transformer` on `global`. + +## Resolution + +### Cause 1: a DaemonSet Pod of the plugin is Pending with Insufficient cpu on one node + +1. Find the Pending Pod and the node it cannot fit on: + + ```bash + kubectl get ars -A + kubectl describe ars -n cpaas-system multus + kubectl get ds -A | grep multus + kubectl get pods -A --field-selector status.phase=Pending -o wide | grep multus + kubectl -n kube-system describe pod | grep -A5 Events + ``` + +2. Check the allocation of the node named by the scheduler event, or of the node missing from the `-o wide` list: + + ```bash + kubectl describe node | grep -A10 'Allocated resources' + ``` + +3. Free CPU requests on that node: move or scale down workloads that do not need to run there, or lower requests that are larger than needed. The Pending Pod schedules by itself. +4. Confirm that the DaemonSet is complete and the AppRelease leaves `WaitReady`: + + ```bash + kubectl get ds -n kube-system delete-multus-cfg-ds + kubectl get ars -n cpaas-system multus + ``` + +### Cause 2: the log storage Pods on one node fail because /cpaas returns Input/output error + +1. Find the node that runs the failing Pods: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep -E 'cpaas-elasticsearch|cpaas-kafka|cpaas-zookeeper' + ``` + +2. On that node, confirm the I/O error and look at the storage stack under the mount: + + ```bash + ls /cpaas /cpaas/data + findmnt /cpaas /cpaas/data + lsblk + lvs + dmesg -T | grep -iE 'i/o error|blk_update_request|xfs|ext4|dm-' | tail -30 + ``` + + If the device is a multipath device, also check its paths with `multipath -ll`. Failed paths, or kernel messages that another process (for example a partitioning tool) has the device open exclusively for writing, point at the storage itself; involve the storage owner before rebooting. + +3. Do not delete or clean `/cpaas/data` while it returns I/O errors; the data cannot be read in that state and may be needed after the storage is repaired. + +4. Reboot the host. + + > **Warning:** Rebooting the host stops every Pod on it, including other log storage replicas and platform components scheduled there, until the node is back. Precondition: only when `/cpaas` on that node returns `Input/output error`, the failing log storage Pods are on that node, and the other nodes can carry the load meanwhile. Drain the node first where possible. Rollback: none for the reboot itself. If `/cpaas` still fails after the reboot, the disk, its paths, or the LVM volume is damaged; keep the `dmesg` output and involve the storage owner before touching the volume. + + ```bash + kubectl cordon + kubectl drain --ignore-daemonsets --delete-emptydir-data + systemctl reboot + ``` + +5. After the node is back, confirm the mount and return the node to scheduling: + + ```bash + ls /cpaas + kubectl uncordon + kubectl -n cpaas-system get pods -o wide | grep -E 'cpaas-elasticsearch|cpaas-kafka|cpaas-zookeeper' + ``` + + The log storage Pods recover by themselves, and the upgrade passes `CheckStatus-logcenter`. + +### Cause 3: nevermore cannot start, secret log-collector-bearer-auth not found, Log Essentials on global was not upgraded first + +1. On the business cluster, confirm the mount failure and the missing Secret: + + ```bash + kubectl get pod -A | grep nevermore + kubectl -n cpaas-system describe pod | grep -A3 FailedMount + kubectl get secret -A | grep log-collector + ``` + +2. On `global`, compare the version of Log Essentials with the logging plugins of the business cluster: + + ```bash + kubectl get minfo | grep -i log + ``` + +3. Upgrade "Alauda Container Platform Log Essentials" on `global` to the same release line (for example v4.3.2) from Cluster Plugins first. +4. Check that the Secrets arrived on the business cluster, then retry the business cluster plugin upgrade: + + ```bash + kubectl get secret -n cpaas-system | grep -E 'log-collector-bearer-auth|log-database-bearer-auth' + kubectl get pod -A | grep nevermore + ``` + +5. If the upgrade button of a pushed version stays greyed out, restart `cluster-transformer` on `global`. + + > **Warning:** Restarting `cluster-transformer` pauses plugin synchronization until the new Pod is Ready. Precondition: only when the new plugin versions were pushed and the upgrade button stays greyed out. Rollback: none needed; the Deployment recreates the Pod. + + ```bash + kubectl -n cpaas-system rollout restart deployment/cluster-transformer + kubectl -n cpaas-system rollout status deployment/cluster-transformer + ``` + +## Diagnostic Steps + +1. List AppReleases that are not settled and read their reason: + + ```bash + kubectl get ars -A | grep -v -E 'Synced|Ready' + kubectl describe ars -n cpaas-system | grep -E 'Reason|Message' + ``` + + Expected for this article: Reason `NotReady` with `timed out waiting for the condition`, or a progress log that repeats `wait resources ready`. + +2. Find the plugin Pod that is not Ready and read its events: + + ```bash + kubectl get pods -A -o wide | grep -v -E 'Running|Completed' + kubectl -n describe pod | grep -A10 Events + ``` + + - `Pending` with `Insufficient cpu`: Cause 1. + - Several log storage Pods failing on one node whose `/cpaas` returns `Input/output error`: Cause 2. + - `nevermore` in `Init` with `secret "log-collector-bearer-auth" not found`: Cause 3. + - `ImagePullBackOff` or `ErrImagePull`: not this article; see the image pull article named above. diff --git a/docs/en/solutions/Cluster_plugins_stay_Deploying_or_AppReleases_stall_because_sentry_is_OOM_hung_or_CPU_bound_on_ACP.md b/docs/en/solutions/Cluster_plugins_stay_Deploying_or_AppReleases_stall_because_sentry_is_OOM_hung_or_CPU_bound_on_ACP.md new file mode 100644 index 000000000..5b152e97e --- /dev/null +++ b/docs/en/solutions/Cluster_plugins_stay_Deploying_or_AppReleases_stall_because_sentry_is_OOM_hung_or_CPU_bound_on_ACP.md @@ -0,0 +1,187 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.16" + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Cluster plugins stay Deploying or AppReleases stall because sentry is OOM, hung, or CPU-bound on ACP + +## Issue + +`sentry` is the component in `cpaas-system` of each cluster that installs, upgrades, and uninstalls plugin charts; it records the results in AppReleases (`ars`) and the chart sync records (`rels`), and applies ResourcePatches (`rpch`). When it is unhealthy, the console is up but: + +- CoreDNS, log collection, Prometheus, and other cluster plugins stay **Deploying**. +- A `sentry` Pod sits near 100% of its CPU limit for days, and a critical alert fires on `sentry` CPU usage. +- Pods fail to deploy across the cluster, and several AppReleases are `Failed`, `WaitReady`, or empty, while `sentry` Pods are `Running`. +- A ResourcePatch stays `pending` while `sentry` reports OOM. +- The `base` or `region` AppRelease of a workload cluster stays syncing, and the BroadcastJob `update-containerd-config` succeeded count rises and falls back (for example 50, 51, 52, then 50 again). + +Related article: [Platform health page shows components abnormal while their Pods are Running on ACP](Platform_health_page_shows_components_abnormal_while_their_Pods_are_Running_on_ACP.md). + +## Root Cause + +### Cause 1: sentry is OOM-killed, so AppRelease status does not sync + +`sentry` is not running its sync because it is OOM-killed, so plugin rows stay **Deploying** and AppRelease status does not sync. A higher memory limit lets it complete. + +### Cause 2: a large number of failed rels keeps sentry CPU high + +One `sentry` replica stays at about its CPU limit while the other is near 0, and the log keeps processing AppReleases with repeated `warn="set cocurrentProcessors failed, process later"` and `info="beginning wait for 45 resources with timeout of 30s"`, although `kubectl get ars -A -w` shows every AppRelease `Synced`/`Ready` and rarely updating. A CPU profile shows most of the time in Go garbage collection, with `(*AppReleaseReconciler).Process` behind it. The cluster holds a large number of failed `rels` (chart sync records), for example left from a long abnormal period while the cluster was being imported. Raising the CPU or memory limit, or moving the Pod, does not help; deleting the failed `rels` does. + +### Cause 3: sentry is hung while its Pods report Running + +Pods fail across the cluster, for example the kubelet journal repeats `Error syncing pod, skipping ... CrashLoopBackOff` for platform Pods, and `kubectl get ars -A` shows some entries `Failed` (for example `cni-kube-ovn`), `WaitReady` (`logcenter`), or with no SYNC, HEALTH, or MESSAGE value (`monitor`). `sentry` itself shows `Synced`/`Ready`, and all `sentry-*` and `sentry-service-*` Pods are `1/1 Running` with 0 restarts, but `sentry` is hung and does not synchronize. + +### Cause 4: a ResourcePatch stays pending because sentry is OOM + +The `sentry` Pods watch and enforce ResourcePatch (`rpch`) objects. While they are OOM-killed, nothing processes a new `rpch`, so it stays `pending` when a resource is changed (seen on 3.12.3). + +### Cause 5: a BroadcastJob hook cannot finish within the AppRelease timeout + +After all nodes were restarted, the `region` AppRelease on a workload cluster is upgraded again. Its chart `alauda-cluster-base` has the BroadcastJob `update-containerd-config` as a `helm.sh/hook: post-upgrade` hook with `parallelism: 1`, so it processes one node at a time. On a cluster with many nodes (for example 68), it cannot finish within the AppRelease `timeout` (240 seconds), so the upgrade is retried, the BroadcastJob is recreated, and its succeeded count resets. Deleting the BroadcastJob makes it reappear, and editing the AppRelease `timeout` is reverted automatically. On the `global` cluster the `minfo` of that cluster shows the condition `CheckStatus-region` with `status: Processing`, `reason: Syncing`, and `message: '[1/2] chart ait/chart-alauda-cluster-base- Upgrading'`. This was seen on 3.16. The same mechanism with the `upgrade-sysctl-config` BroadcastJob of the `base` AppRelease is described in [base AppRelease stays failed because the upgrade-sysctl-config BroadcastJob cannot finish on a large cluster on ACP](base_AppRelease_stays_failed_because_the_upgrade-sysctl-config_BroadcastJob_cannot_finish_on_a_large_cluster_on_ACP.md). + +## Resolution + +### Cause 1: sentry is OOM-killed, so AppRelease status does not sync + +1. Confirm the OOM: + + ```bash + kubectl get ars -A + kubectl get pods -n cpaas-system | grep sentry + kubectl describe pod -n cpaas-system | grep -E 'OOMKilled|Last State' + ``` + +2. Export the `sentry` Deployment and raise its memory limit above the usage that was OOM-killed. + + > **Warning:** A higher limit lets `sentry` use more node memory. Do this only when the Pod's last state is `OOMKilled`. To roll back, apply the export. + + ```bash + kubectl get deploy -n cpaas-system sentry -o yaml > "$HOME/sentry-deploy.yaml" + kubectl set resources deploy -n cpaas-system sentry --limits=memory= + ``` + +3. Plugin rows leave **Deploying** once `sentry` stays `Running` and AppRelease status syncs. + +### Cause 2: a large number of failed rels keeps sentry CPU high + +1. Confirm the signature: + + ```bash + kubectl top pod -n cpaas-system | grep sentry + kubectl logs -n cpaas-system --tail=300 | grep -E 'cocurrentProcessors|beginning wait for' + kubectl get rels -A | wc -l + kubectl get rels -A | grep -i fail | head + ``` + + If `rels` is not a known resource type on the cluster, find the kind of the chart sync records with `kubectl api-resources` first. + +2. Back up and delete the failed `rels`. + + > **Warning:** Deleted records cannot be restored; `sentry` creates new ones as it reconciles. Delete only objects that are already failed, and leave successful ones in place. + + ```bash + kubectl get rels -A -o yaml > "$HOME/rels-backup.yaml" + kubectl delete rels -n + ``` + +3. `sentry` CPU returns to normal once the failed records are gone. + +### Cause 3: sentry is hung while its Pods report Running + +1. Confirm the signature: + + ```bash + journalctl -u kubelet --since "30 min ago" | grep -E 'Error syncing pod|CrashLoopBackOff' | tail + kubectl get ars -A + kubectl get pods -n cpaas-system | grep sentry + ``` + +2. Recreate all `sentry` and `sentry-service` Pods. + + > **Warning:** AppRelease reconciliation pauses until the new Pods are `Ready`. Do this only when several AppReleases are `Failed`, `WaitReady`, or empty, Pods fail across the cluster, and the `sentry` Pods are `Running` with no restarts. The Pods are recreated from the same specs. + + ```bash + kubectl delete pod -n cpaas-system + kubectl get ars -A + ``` + +3. Every AppRelease returns to `Synced`/`Ready`/`chart synced` with a recent UPDATE time. If one stays `Failed`, read its message instead of restarting `sentry` again. + +### Cause 4: a ResourcePatch stays pending because sentry is OOM + +1. Confirm the OOM, find the pending object, and save the current state: + + ```bash + kubectl get deploy -A | grep -E 'sentry|cluster-transformer' + kubectl describe pod -n cpaas-system | grep -E 'OOMKilled|Last State' + kubectl get rpch -A | grep -i pending + kubectl get rpch -o yaml > "$HOME/-rpch.yaml" + kubectl get deploy -n cpaas-system sentry sentry-service cluster-transformer -o yaml > "$HOME/sentry-deploys.yaml" + ``` + +2. Delete the pending `rpch`, stop the components, raise the `sentry` limit, and start them again with their previous replica counts. + + > **Warning:** Deleting the `rpch` drops the patch it would have applied. With the three Deployments at 0 replicas, plugin and AppRelease reconciliation stops. Do this only when `sentry` is OOM-killed and an `rpch` is stuck `pending`, in a maintenance window. To roll back, apply the Deployment export and, if the patch is still wanted, the `rpch` export once `sentry` is healthy. + + ```bash + kubectl delete rpch + kubectl scale deploy -n cpaas-system sentry sentry-service cluster-transformer --replicas=0 + kubectl set resources deploy -n cpaas-system sentry --limits=memory= + kubectl scale deploy -n cpaas-system sentry --replicas= + kubectl scale deploy -n cpaas-system sentry-service --replicas= + kubectl scale deploy -n cpaas-system cluster-transformer --replicas= + ``` + +3. `sentry` stays `Running`, and a new resource change does not leave its `rpch` pending. + +### Cause 5: a BroadcastJob hook cannot finish within the AppRelease timeout + +1. Confirm the loop: + + ```bash + kubectl get broadcastjobs.operator.alauda.io -A + kubectl describe broadcastjobs.operator.alauda.io -n cpaas-system update-containerd-config | sed -n '/Events/,$p' + kubectl get broadcastjobs.operator.alauda.io -n cpaas-system update-containerd-config -o yaml | grep -n -E 'parallelism|helm.sh/hook|ttlSecondsAfterFinished|restartLimit' + kubectl get ars -n cpaas-system region -o yaml | grep -n timeout + ``` + + Expected: SUCCEEDED cycles below DESIRED, and the event shows thousands of `SuccessfulCreate` for `update-containerd-config-` Pods. + +2. Raise the parallelism so that the hook finishes within the timeout. + + > **Warning:** This BroadcastJob updates the containerd configuration on every node it runs on; with `parallelism: 10` up to ten nodes are processed at the same time. Do this only on the signature above. To roll back, set `parallelism` back to `1`; work already done on nodes is not undone. + + ```bash + kubectl patch broadcastjobs.operator.alauda.io -n cpaas-system update-containerd-config --type=merge -p '{"spec":{"parallelism":10}}' + kubectl get broadcastjobs.operator.alauda.io -n cpaas-system update-containerd-config -w + ``` + +3. SUCCEEDED reaches DESIRED, and the `region` AppRelease and the `CheckStatus-region` condition return to normal. + +## Diagnostic Steps + +```bash +kubectl get pods -n cpaas-system | grep sentry +kubectl get ars -A | grep -viE 'Ready|Succeeded|Running' || true +kubectl get broadcastjobs.operator.alauda.io -A +kubectl get rpch -A +``` + +```text +plugins stuck Deploying, sentry OOMKilled -> Cause 1 +sentry CPU high for days, cocurrentProcessors warnings, many failed rels -> Cause 2 +Pods fail cluster-wide, ars Failed or WaitReady, sentry Running with 0 restarts -> Cause 3 +rpch pending after a resource change, sentry OOMKilled -> Cause 4 +update-containerd-config SUCCEEDED cycles, CheckStatus-region Processing -> Cause 5 +``` diff --git a/docs/en/solutions/ConfigMaps_missing_after_a_namespace_restore_that_completed_without_errors_on_ACP.md b/docs/en/solutions/ConfigMaps_missing_after_a_namespace_restore_that_completed_without_errors_on_ACP.md new file mode 100644 index 000000000..3da2737d7 --- /dev/null +++ b/docs/en/solutions/ConfigMaps_missing_after_a_namespace_restore_that_completed_without_errors_on_ACP.md @@ -0,0 +1,86 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# ConfigMaps missing after a namespace restore that completed without errors on ACP + +## Issue + +A namespace was restored with the platform backup and restore tool on Alauda Container Platform. The restore task shows `Completed`, neither the backup task nor the restore task reported an error, and yet some objects (for example two ConfigMaps) are missing from the restored namespace. The objects can be recreated by importing their YAML in the console, but the question is why they are missing, and whether the restore task logs or the backup files show it. + +Typical `kubectl -n cpaas-system describe restores.velero.io ` output: + +- `Phase: Completed`, `Items Restored` equal to `Total Items`, and a non-zero `Warnings` count; +- `Spec.Included Namespaces` names the source namespace, and `Spec.Namespace Mapping` maps it to a new target namespace. + +## Root Cause + +### Cause 1: the objects were restored and deleted afterwards + +The Velero log on the restore cluster contains, for each missing ConfigMap, an `Attempting to restore ConfigMap: ` entry with `resource=configmaps` and the target namespace, followed by a `level=info` progress entry such as `Restored items out of an estimated total of `. That shows the backup carried the objects and the restore created them without error. The platform audit logs of the restore cluster then show that the ConfigMaps existed in the target namespace and were deleted manually after the restore. The restore was not at fault. + +The `describe` output and its warning count do not identify individual objects; only the per-object Velero log entries and the audit log do. + +## Resolution + +### Cause 1: the objects were restored and deleted afterwards + +Use this only when the Velero log shows the missing object being restored (Diagnostic Steps, step 2). + +1. Search the platform audit logs of the restore cluster for operations on ConfigMaps with that name in the target namespace, to find who deleted them and when. +2. Recreate the missing objects, through the console YAML import or with `kubectl` when the source namespace still holds them: + +```bash +kubectl -n get configmap -o yaml > /tmp/.yaml +``` + +Edit the file: set `metadata.namespace` to the target namespace and remove `metadata.uid`, `metadata.resourceVersion`, `metadata.creationTimestamp`, and `metadata.managedFields`. Then create the object and check the workloads that use it: + +```bash +kubectl apply -f /tmp/.yaml +kubectl -n get configmap +kubectl -n get pods +``` + +If the log has no `Attempting to restore` entry for the object, it was not restored by this task. Check whether the backup contained it, as described in [What platform backup, application snapshots, and volume snapshots cover on ACP](./What_platform_backup_application_snapshots_and_volume_snapshots_cover_on_ACP.md). + +## Diagnostic Steps + +1. Work on the cluster that runs the restore, not on the backup source cluster; it is the cluster where the `Restore` object exists: + +```bash +kubectl -n cpaas-system get restores.velero.io +kubectl -n cpaas-system get restores.velero.io -o jsonpath='{.status.phase}{" "}{.status.progress}{" warnings="}{.status.warnings}{"\n"}' +``` + +2. Search the Velero log for the missing object. Velero tags restore log entries with `restore=cpaas-system/`. A restore of a few thousand items writes many lines, so do not cut the log with a small `--tail`: + +```bash +kubectl -n cpaas-system logs deploy/velero | grep 'restore=cpaas-system/' | grep -A2 'Attempting to restore ConfigMap: ' +kubectl -n cpaas-system logs deploy/velero | grep 'restore=cpaas-system/' | grep -iE 'level=(warning|error)' +``` + +If the Velero pod restarted after the restore, add `--previous`, or read the stored restore log through the Velero CLI in the pod: + +```bash +kubectl exec -n cpaas-system deploy/velero -- ./velero restore describe --details +kubectl exec -n cpaas-system deploy/velero -- ./velero restore logs | grep '' +``` + +3. Do not treat an empty label query as proof that the restore skipped the object. Velero labels restored objects with `velero.io/restore-name`, but an object deleted after the restore is not listed either: + +```bash +kubectl -n get configmap -l velero.io/restore-name= +``` + +4. Match the result: + - `Attempting to restore ConfigMap: ` present, followed by a `Restored items` entry, and the object is gone now: Cause 1; check the audit log for the deletion. + - No entry for the object: it was not restored by this task; check the backup content. + +## Related Information + +- [Application restore fails, hangs, or restored workloads do not run on another cluster on ACP](./Application_restore_fails_hangs_or_restored_workloads_do_not_run_on_another_cluster_on_ACP.md) diff --git a/docs/en/solutions/Console_is_inaccessible_or_a_business_cluster_shows_0_nodes_because_global_cannot_reach_an_API_server_on_ACP.md b/docs/en/solutions/Console_is_inaccessible_or_a_business_cluster_shows_0_nodes_because_global_cannot_reach_an_API_server_on_ACP.md new file mode 100644 index 000000000..6f306c3d9 --- /dev/null +++ b/docs/en/solutions/Console_is_inaccessible_or_a_business_cluster_shows_0_nodes_because_global_cannot_reach_an_API_server_on_ACP.md @@ -0,0 +1,146 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Console is inaccessible or a business cluster shows 0 nodes because global cannot reach an API server on ACP + +## Issue + +The console worked before, and now one of the following happens: + +- A business cluster row shows 0 nodes, 0 containers, and 0 storage (for example after a `global` migration), while `kubectl get nodes` inside the business cluster shows `Ready`. +- The page says the system cannot be accessed (`系统无法访问`), and platform component logs time out against the in-cluster API Service. +- The console is slow, one `apollo` replica is unhealthy, and renewing credentials or restarting `apollo` does not help. +- Login fails while a burst of alerts fires and the control-plane nodes run at 100% CPU. + +Related article: [Console returns 502 or does not open after installation or a VIP change on ACP](Console_returns_502_or_does_not_open_after_installation_or_a_VIP_change_on_ACP.md). If only monitoring pages time out on port 11780, see [Monitoring pages return 502 or context deadline exceeded because port 11780 on the cluster VIP is blocked on ACP](Monitoring_pages_return_502_or_context_deadline_exceeded_because_port_11780_on_the_cluster_VIP_is_blocked_on_ACP.md). + +## Root Cause + +### Cause 1: the global cluster cannot open the business API server on 6443 + +The cluster's `CheckStatus` is `Failed` with a message such as `Get "https://:6443/api/v1/nodes?labelSelector=kubernetes.io%2Fos%3Dlinux&timeout=2m0s": context deadline exceeded`. From the `global` side, TCP to the business VIP on 6443 fails or times out, for example because a switch no longer forwards from `global` to that VIP. The business nodes and containers are not at fault. If the path was restored but the cluster sync controllers were not restarted, the 0 counts can remain. + +A failed `global` control-plane node looks similar from the browser: the whole platform is down and `apollo` cannot list API groups for the cluster `global`, because the `global` API endpoint does not answer. + +### Cause 2: one control-plane host is abnormal and components lose their API watches + +Abnormal platform components repeat client-go errors such as: + +```text +watch of *v1.Pod ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding +"Objects listed" error:Get "https://:443/api/v1/pods?resourceVersion=...": dial tcp :443: i/o timeout +``` + +Restarting the API server does not help. One master host is abnormal, and restarting that host restores the platform. The exact host fault was not identified. + +### Cause 3: a global node is NotReady + +The UI loads slowly and one `apollo` replica is unhealthy. Regenerating credentials and restarting that Pod do not help, because the node it runs on is `NotReady` or broken (for example after another container runtime was installed on it). Repairing the node restores platform access. + +### Cause 4: control-plane nodes run at 100% CPU + +The console cannot be logged into, a large set of alerts fires, and two or more control-plane nodes are at 100% CPU, shared between host agent processes and containers. A batch restart of the Pods that burn CPU brings it down, and restarting `apollo` then restores login. + +## Resolution + +### Cause 1: the global cluster cannot open the business API server on 6443 + +1. From a `global` node: + + ```bash + curl -k --max-time 5 https://:6443/readyz + kubectl get cls -A + kubectl describe cls + ``` + + `readyz` must succeed before any controller restart helps. Fix the network path, the VIP, or the failed control-plane node first. + +2. Once 6443 answers and the console still shows 0 nodes, restart the `global` cluster components whose logs show the `context deadline exceeded` timeouts to that business cluster (the cluster synchronization components). Restarting `cluster-api-provider-alauda-manager` alone may not be enough. + + > **Warning:** Restarting these controllers pauses cluster synchronization; it does not delete clusters. Do not restart them while 6443 is still closed; the new Pods only repeat the timeout. + + ```bash + for p in $(kubectl get pods -n cpaas-system -o name); do kubectl logs -n cpaas-system "$p" --all-containers --tail=200 2>/dev/null | grep -q ":6443.*context deadline exceeded" && echo "$p"; done + kubectl delete pod -n cpaas-system + ``` + +### Cause 2: one control-plane host is abnormal and components lose their API watches + +1. Find the master that is not healthy: + + ```bash + kubectl get po -A | grep -vi run + kubectl get nodes -o wide + kubectl get pods -n kube-system -l component=kube-apiserver -o wide + kubectl get pods -n kube-system -l component=etcd -o wide + ``` + + On each master: + + ```bash + uptime + dmesg -T | tail -50 + crictl ps | grep -E 'kube-apiserver|etcd' + ``` + +2. Restart the abnormal master host. + + > **Warning:** The API server and etcd member on that host stop until it boots. Do this only when the component logs show `http2: client connection lost` or `dial tcp :443: i/o timeout` and this master is the abnormal one. Confirm the other masters are healthy so that etcd keeps quorum, and keep console access to the host. A reboot cannot be rolled back. + +3. Platform pages load once the node is `Ready` and the reflector errors stop. + +### Cause 3: a global node is NotReady + +```bash +kubectl get nodes +kubectl describe node | sed -n '/Conditions/,$p' +``` + +Repair or remove the bad node. Do not keep rotating credentials once the certificate is known to be valid and only the `apollo` replica on that node is stuck. Console latency drops when every `global` node used by platform Pods is `Ready`. + +### Cause 4: control-plane nodes run at 100% CPU + +1. Confirm the saturation: + + ```bash + kubectl top nodes + ps aux --sort=-%cpu | head + ``` + +2. Restart the Pods that burn CPU on the saturated nodes, then `apollo` if login still fails. + + > **Warning:** Restarting Pods interrupts the workloads they serve, including platform components. Do this only when the masters are at 100% CPU and the login failure matches that window, preferably in a maintenance window. The Pods return from the same specs. + + ```bash + kubectl get pods -A --field-selector spec.nodeName= --no-headers + kubectl delete pod -n + kubectl delete pod -n cpaas-system + ``` + +3. Login succeeds once CPU is no longer pinned and `apollo` is `Running`. + +## Diagnostic Steps + +```bash +curl -k --max-time 5 https://:6443/readyz +kubectl get nodes +kubectl top nodes +kubectl get po -A | grep -vi run +``` + +```text +business kubectl works, global curl to :6443 fails, CheckStatus Failed -> Cause 1 +http2: client connection lost, dial tcp :443 i/o timeout, one master abnormal -> Cause 2 +slow UI, one global node NotReady, credentials already valid -> Cause 3 +login fails, master CPU at 100% -> Cause 4 +``` diff --git a/docs/en/solutions/Console_is_slow_or_shows_data_loading_errors_because_calls_through_erebus_or_the_marketplace_backends_are_slow_on_ACP.md b/docs/en/solutions/Console_is_slow_or_shows_data_loading_errors_because_calls_through_erebus_or_the_marketplace_backends_are_slow_on_ACP.md new file mode 100644 index 000000000..26e4fe8bb --- /dev/null +++ b/docs/en/solutions/Console_is_slow_or_shows_data_loading_errors_because_calls_through_erebus_or_the_marketplace_backends_are_slow_on_ACP.md @@ -0,0 +1,189 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Console is slow or shows data loading errors because calls through erebus or the marketplace backends are slow on ACP + +## Issue + +The platform is slow and data on the pages takes more than a minute to appear, or a data loading error keeps appearing and clears when the user clicks **Reload**. Related reports: + +- OperatorHub, the cluster plugin page (`集群插件`), and Tekton pages all fail to open at the same time. +- Every page is slow, with occasional no-permission errors and 502. +- The left navigation menu takes a long time to load. +- In the browser developer tools the `envs` API or other calls that go through `erebus` time out. + +`erebus` is the console backend in `cpaas-system` on the `global` cluster that proxies calls toward a target cluster, so a slow path through it makes many pages slow at once. + +Start by timing the slow request from a host inside the platform network, so that a slow client path, such as a VPN, is ruled out before any platform change. + +## Root Cause + +### Cause 1: the user's network path is slow while the platform is healthy + +Every page takes more than a minute for the user, but the same requests answer quickly from inside the platform network. The delay is on the client path, for example the VPN the user goes through to reach the platform, and no platform component is at fault. + +### Cause 2: the envs API to erebus times out and the erebus log has no error + +Browser tools show the `envs` API to `erebus` timing out, and the `erebus` Pod log on the `global` cluster has no clear exception. Restarting the `erebus` Pod restores the connection; no more specific fault was identified. + +### Cause 3: a data loading error clears on reload and erebus logs context canceled + +`erebus` logs `context canceled`, and the path from the node that runs `erebus` to the platform VIP answers normally. The failed call is the client connection dropping; after the client connects again, loading succeeds. No change is needed. + +### Cause 4: a Layer-4 load balancer with an X-Forwarded-For profile on port 6443 delays calls through erebus + +Any page can be slow, intermittently but often; some show a no-permission error and some requests return 502 from `erebus`. No container is abnormal, and `kubectl` on a workload-cluster master answers quickly while the UI is slow. A normal request takes seconds instead of milliseconds during problem periods. + +The path client > global VIP > `erebus` > workload-cluster VIP > API server reproduces the delay; the path client > workload-cluster VIP > API server does not, nor does a direct `curl` from the global nodes to the workload API server. Captures on the global nodes show many TCP retransmissions toward the workload-cluster VIP on 6443, and the load balancer capture shows retransmissions, port-reuse warnings, and duplicate ACKs ending in an RST from the client side. The F5 virtual server for API server port 6443 is a Performance (Layer 4) server with the `fastL4` profile, and its HTTP Client profile is set to `X-Forwarded-For`. After XFF was disabled on that virtual server, the problem no longer reproduced; the exact mechanism is not confirmed. Host and VM NIC queue tuning improves stability somewhat but does not remove the delay. + +### Cause 5: a decommissioned cluster is still registered + +The left navigation menu loads very slowly. A cluster that was decommissioned is still registered on the platform, and platform components keep trying to reach its VIP. + +### Cause 6: marketplace-apiserver is killed before its cache finishes loading + +OperatorHub, the cluster plugin page, and Tekton and other product pages fail at the same time, and `marketplace-apiserver` keeps restarting. On a cluster with a large number of CRDs or APIServices, `marketplace-apiserver` does not finish loading its cache before the liveness probe fails, so the Pod is restarted before it ever becomes healthy. Raising `livenessProbe.initialDelaySeconds` to `180` restores the service. + +## Resolution + +### Cause 1: the user's network path is slow while the platform is healthy + +1. In the browser developer tools (**Network**), right-click the slow request and choose **Copy as cURL**. +2. Run it from the user's machine and from a host inside the platform network (for example a `global` control-plane node or a jump host), with timing output appended: + + ```bash + curl -k -s -o /dev/null -w 'connect=%{time_connect}s first-byte=%{time_starttransfer}s total=%{time_total}s\n' https:/// + ``` + +3. A fast answer inside the network and tens of seconds from the user's machine places the delay on the client path. A long **Initial connection** or **Content download** phase in the browser **Timing** tab points the same way. Hand the path to the team that runs it; no platform change is needed. + +### Cause 2: the envs API to erebus times out and the erebus log has no error + +1. Read the `erebus` log before deleting the Pod: + + ```bash + kubectl get pods -n cpaas-system --no-headers | awk '/^erebus-/ {print}' + kubectl logs -n cpaas-system --tail=200 + ``` + +2. Restart `erebus`. + + > **Warning:** Console calls proxied by `erebus` fail until the new Pod is `Ready`. Do this only when the `envs` API to `erebus` times out and the log shows no separate fault such as clock skew, an expired platform credential, or a down business API server. The Pod is recreated from the same spec. + + ```bash + kubectl delete pod -n cpaas-system + ``` + +3. If the `envs` API still times out after the new Pod is `Running`, the fault is elsewhere. + +### Cause 3: a data loading error clears on reload and erebus logs context canceled + +Confirm the log and reload the page so that the client opens a new connection: + +```bash +kubectl logs -n cpaas-system --tail=200 | grep -F 'context canceled' +``` + +If the platform VIP does not answer from the node that runs `erebus`, the problem is reachability, not this cause. + +### Cause 4: a Layer-4 load balancer with an X-Forwarded-For profile on port 6443 delays calls through erebus + +1. From each `global` node, time the workload API server through its VIP, then reproduce the slow page and compare: + + ```bash + time curl -k -s -o /dev/null -w '%{http_code}\n' https://:6443/readyz + kubectl get pods -n cpaas-system -o wide | grep erebus + ``` + +2. On the `global` node that runs `erebus`, capture traffic toward the workload-cluster VIP while the problem reproduces: + + ```bash + tcpdump -i any -nn host and port 6443 -w /tmp/erebus-6443.pcap + ``` + + Retransmissions, port-reuse warnings, and duplicate ACKs that end in a client-side RST, only on the path through the load balancer, match this cause. + +3. On the load balancer, open the virtual server that publishes 6443 and disable the `X-Forwarded-For` HTTP Client profile. + + > **Warning:** Changing the virtual server profile affects every connection through it, not only ACP. Do this only when the virtual server is Layer 4 (`fastL4`) for API server port 6443, its HTTP Client profile is `X-Forwarded-For`, and the captures show the retransmissions above. Use a change window with the network team. To roll back, set the profile back. + +4. Watch whether the slow requests, no-permission errors, and 502 return. + +### Cause 5: a decommissioned cluster is still registered + +1. List the clusters the platform knows about and test each VIP from a `global` node: + + ```bash + kubectl get cls -A + curl -k --max-time 5 https://:6443/readyz + ``` + +2. Delete the decommissioned cluster from the platform cluster list. + + > **Warning:** Removing a cluster from the platform ends its management from the console, and it cannot be undone from the cluster list; to manage it again it must be added back. Do this only for a cluster that is decommissioned and whose VIP no longer answers. + +3. The left navigation loads normally once components stop retrying that VIP. + +### Cause 6: marketplace-apiserver is killed before its cache finishes loading + +1. Check whether the Pod is restarted by its liveness probe: + + ```bash + kubectl get deploy -A | grep marketplace-apiserver + kubectl describe pod -n | grep -E 'Restart Count|Last State|Liveness|Unhealthy|Killing' + kubectl logs -n --previous --tail=200 + kubectl get crd --no-headers | wc -l + kubectl get apiservice --no-headers | wc -l + kubectl get deploy -n marketplace-apiserver -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" "}{.livenessProbe}{"\n"}{end}' + ``` + + The signature is a growing restart count with `Liveness probe failed` events while the previous log shows the server still starting, on a cluster with many CRDs or APIServices. A Pod that crashes on its own error before the probe fires is not this cause. + +2. Export the Deployment and raise the initial delay of the liveness probe. + + > **Warning:** The patch rolls the Pod, so OperatorHub, cluster plugin, and Tekton pages stay unavailable until it is `Ready`. A longer initial delay also means a process that really hangs at startup is restarted later. To roll back, patch `initialDelaySeconds` back to the value in the export. This change can be reverted by a platform or plugin upgrade; check it again afterwards. + + ```bash + kubectl get deploy -n marketplace-apiserver -o yaml > /tmp/marketplace-apiserver.bak.yaml + kubectl patch deploy -n marketplace-apiserver --type=json -p '[{"op":"add","path":"/spec/template/spec/containers/0/livenessProbe/initialDelaySeconds","value":180}]' + kubectl rollout status deploy -n marketplace-apiserver + ``` + + The path uses container index `0`; if the `jsonpath` output lists the probe on another container, use that index. + +3. Once the Pod stays `Running` without liveness restarts, open OperatorHub, the cluster plugin page, and a Tekton page again. + +## Diagnostic Steps + +1. In the browser **Network** panel, note whether the failing call is the `envs` API or another call proxied by `erebus`, and whether it times out or returns 500. Time the same call from inside the platform network (Cause 1); if only the user's side is slow, stop there. + +2. Read the `erebus` and `apollo` logs on the `global` cluster: + + ```bash + kubectl get pods -n cpaas-system -o wide | grep -E 'erebus|apollo' + kubectl logs -n cpaas-system --since=30m --tail=200 + kubectl logs -n cpaas-system --since=30m --tail=200 | grep -i erebus + ``` + +3. Match the result: + + ```text + every page over a minute from the user's side, same call fast inside the network -> Cause 1 + envs API timeout, erebus log has no error -> Cause 2 + data loading error, reload works, erebus logs context canceled -> Cause 3 + all pages slow, 502 from erebus, retransmissions through the load balancer on 6443 -> Cause 4 + left menu slow, a decommissioned cluster still listed -> Cause 5 + OperatorHub, cluster plugins and Tekton fail together, marketplace-apiserver liveness restarts -> Cause 6 + ``` + +4. If the problem started right after a change on the hosts (DNS, NetworkManager, CNI, kernel parameters), see [Console returns 500 or data loading errors after a host network change on ACP](Console_returns_500_or_data_loading_errors_after_a_host_network_change_on_ACP.md). diff --git a/docs/en/solutions/Console_menu_bar_or_Data_Services_navigation_does_not_load_because_of_a_workload_cluster_ALB_or_a_credential_on_ACP.md b/docs/en/solutions/Console_menu_bar_or_Data_Services_navigation_does_not_load_because_of_a_workload_cluster_ALB_or_a_credential_on_ACP.md new file mode 100644 index 000000000..b80a1a3d2 --- /dev/null +++ b/docs/en/solutions/Console_menu_bar_or_Data_Services_navigation_does_not_load_because_of_a_workload_cluster_ALB_or_a_credential_on_ACP.md @@ -0,0 +1,143 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Console menu bar or Data Services navigation does not load because of a workload cluster ALB or a credential on ACP + +## Issue + +The console opens, but part of it does not load: + +- The left menu bar of platform management stays on a loading placeholder. In the browser developer tools (**Network**), only the `global-info` request stays pending. +- After middleware operators were installed, the **Data Services** left navigation shows no middleware, and the `console-plugins` response contains only `global` data. +- Platform pages freeze and only part of the interface loads. + +Related article: [Console menu or feature is missing because a feature gate, plugin, or product entry is off on ACP](Console_menu_or_feature_is_missing_because_a_feature_gate_plugin_or_product_entry_is_off_on_ACP.md). + +## Root Cause + +### Cause 1: the menu bar waits on a workload cluster whose ALB failed because a node disk is full + +`marketplace-api` on the `global` cluster collects console plugins from every cluster through that cluster's ALB, and repeats for the affected workload clusters: + +```text +consoleplugins.go:64] collect console-plugins in cluster failed. err: invalid status code: 502. response status: 502 Bad Gateway, response body: X-Error: 502 +``` + +The global `apollo` log shows cluster cache errors such as `List log-agent feature error(... Client.Timeout exceeded while awaiting headers)` and `Get monitoring feature error(... context deadline exceeded)`. On the affected workload cluster, the `cpaas-system-*` ALB Pods are `0/2` in `Error` or `Pending`, because node storage ran out. The frontend Pods there are `Running`, and restarting `apollo` and `marketplace-api` does not help. + +### Cause 2: the workload cluster ALB runs in container mode and its LoadBalancer Service stays pending + +The same `collect console-plugins ... 502` line appears, but the ALB Pods on the workload cluster are `Running`. `kubectl get alb2 -A` shows `cpaas-system` with STATE `Progressing`, REASON `wait lb svc ready`, MODE `container`, and the `LoadBalancer` Service `cpaas-system-lb-` (ports `11782` and `11780`) has EXTERNAL-IP ``. This was seen on 4.2.1 on a workload cluster deployed with the cluster-api-provider-hcs procedure, which has no ALB option: the ALB chart set container network mode, which gets no external address there. + +### Cause 3: an expired GitLab model repository token freezes the pages + +Platform pages freeze and only part of the UI loads when the token for the GitLab model repository has expired. The token is stored in the Secret `aml-image-builder-secret` in the namespace `kubeflow-admin-cpaas-io`. + +## Resolution + +### Cause 1: the menu bar waits on a workload cluster whose ALB failed because a node disk is full + +1. Find the workload cluster and its ALB state: + + ```bash + # global cluster + kubectl logs -n cpaas-system deploy/marketplace-api --tail=200 | grep 'collect console-plugins' + # affected workload cluster + kubectl get po -n cpaas-system -o wide | grep cpaas-system- + kubectl describe pod -n cpaas-system | sed -n '/Events/,$p' + ``` + +2. On the node where the ALB Pods are `Error` or `Pending`, check the disk and free space: + + ```bash + df -h + ``` + + > **Warning:** Deleting files is irreversible; copy anything you still need first, and leave live etcd data and container runtime storage in place. Do this only when `global-info` stays pending, `marketplace-api` logs 502 toward that cluster, and the node filesystem is full. + +3. Once the ALB Pods are `2/2 Running`, reload the console. The menu bar loads when `global-info` completes and `marketplace-api` stops logging 502 for that cluster. + +### Cause 2: the workload cluster ALB runs in container mode and its LoadBalancer Service stays pending + +1. On the workload cluster: + + ```bash + kubectl get alb2 -A + kubectl get svc -n cpaas-system | grep LoadBalancer + kubectl get nodes -l cpaas-system-alb + ``` + +2. Switch the ALB to host network mode. + + > **Warning:** In host network mode the ALB Pods run on the host network of the nodes labelled `cpaas-system-alb`, and their ports must be free there. Traffic through this ALB is interrupted while the Pods roll. Do this only when the ALB2 is `Progressing` with `wait lb svc ready`, mode `container`, and the LoadBalancer Service has no external IP. Export the object first; applying the export is the rollback. + + ```bash + kubectl get alb2 -n cpaas-system cpaas-system -o yaml > "$HOME/alb2-cpaas-system.yaml" + kubectl explain alb2.spec --recursive | grep -iE 'networkMode|nodeSelector' + kubectl edit alb2 -n cpaas-system cpaas-system + ``` + + Set `networkMode: host` and the node selector `cpaas-system-alb: ""` at the paths that `kubectl explain` shows. + +3. Make it permanent: on the `global` cluster, add an override to the workload cluster's ALB `minfo`: + + ```bash + kubectl get minfo | grep -i alb | grep + kubectl edit minfo + ``` + + ```yaml + spec: + valuesOverride: + _global: + global: + cluster: + type: DCS + ``` + +4. The ALB2 reaches a ready state, `marketplace-api` stops logging 502 for that cluster, and **Data Services** lists the middleware. + +### Cause 3: an expired GitLab model repository token freezes the pages + +1. Back up the Secret: + + ```bash + kubectl get secret -n kubeflow-admin-cpaas-io aml-image-builder-secret -o yaml > "$HOME/aml-image-builder-secret.yaml" + chmod 600 "$HOME/aml-image-builder-secret.yaml" + ``` + +2. Replace the token value with a current GitLab model repository token. Change only the key that stores the token. + + > **Warning:** The Secret holds a repository credential; keep the backup private. To roll back, apply the backup. + + ```bash + kubectl edit secret -n kubeflow-admin-cpaas-io aml-image-builder-secret + ``` + +3. Reload the platform pages. + +## Diagnostic Steps + +1. In the browser **Network** panel, check whether `global-info` stays pending (Cause 1) or the `console-plugins` response holds only `global` (Cause 2). + +2. On the `global` cluster, read `marketplace-api`. Expected for Causes 1 and 2: `collect console-plugins in cluster failed ... 502`. + +3. On that workload cluster: + + ```text + cpaas-system-* ALB Pods 0/2 Error or Pending, node disk full -> Cause 1 + alb2 Progressing, wait lb svc ready, MODE container, LB Service EXTERNAL-IP -> Cause 2 + ``` + +4. Pages freeze with an incomplete UI and the model repository token has expired: Cause 3. diff --git a/docs/en/solutions/Console_menu_or_feature_is_missing_because_a_feature_gate_plugin_or_product_entry_is_off_on_ACP.md b/docs/en/solutions/Console_menu_or_feature_is_missing_because_a_feature_gate_plugin_or_product_entry_is_off_on_ACP.md new file mode 100644 index 000000000..9126e2804 --- /dev/null +++ b/docs/en/solutions/Console_menu_or_feature_is_missing_because_a_feature_gate_plugin_or_product_entry_is_off_on_ACP.md @@ -0,0 +1,134 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Console menu or feature is missing because a feature gate, plugin, or product entry is off on ACP + +## Issue + +The console opens, but a menu, a feature, or a product entry is missing or shows an error, typically after an upgrade: + +- Namespace-scoped CRD management, available on 3.18, is not visible on 4.x. +- The namespace audit switch is greyed out. +- The UI cannot be switched to Chinese after an upgrade to 4.2.4. +- Load balancers are no longer listed in platform management after an upgrade. +- The data services entry is missing from the portal after an upgrade to 4.0.10. +- Clicking a cluster name in **Alerts** > **Real-time Alerts** shows `缺少翻译文件:插件console-core-service缺少zh语言翻译文件:plugin_console-core-service.json` (missing translation file: plugin console-core-service lacks the zh translation file). +- The **EXEC** entry is missing from Pod and application pages; see [EXEC and Debug entries missing from Pod and application pages until the cpc-exec feature gate is enabled on ACP](EXEC_and_Debug_entries_missing_from_Pod_and_application_pages_until_the_cpc-exec_feature_gate_is_enabled_on_ACP.md). + +Feature gates are managed at `/console-platform/feature-gate`, and are `AlaudaFeatureGate` objects (`alaudafeaturegates.alauda.io`) in `cpaas-system`. + +Related article: [Console menu bar or Data Services navigation does not load because of a workload cluster ALB or a credential on ACP](Console_menu_bar_or_Data_Services_navigation_does_not_load_because_of_a_workload_cluster_ALB_or_a_credential_on_ACP.md). + +## Root Cause + +### Cause 1: namespace-scoped CRD management is behind the namespace-resource-manage gate + +The feature was not removed in 4.x. It is off by default behind the feature gate `namespace-resource-manage`, which a search for an obvious name on the feature gate page does not find. + +### Cause 2: the namespace audit switch is greyed out because namespaceaudits is disabled + +The availability of the namespace audit switch is controlled by the `namespaceaudits` entry of a feature gate resource, not by the switch itself. While that entry is disabled, the switch stays grey. + +### Cause 3: the Chinese language pack plugin is not deployed + +After the upgrade to 4.2.4, Chinese cannot be selected until the Chinese language pack plugin is deployed. + +### Cause 4: the alb2 feature gate is off + +After an upgrade, load balancers are missing from platform management because the `alb2` `AlaudaFeatureGate` in `cpaas-system` is not enabled. + +### Cause 5: the data services ProductEntry has packType AdminPage + +`kubectl get ProductEntry console-dataservices -o yaml` shows an `alauda.io/v1alpha1` `ProductEntry` (owned by the Helm release `application-services-core` in `cpaas-system`) with `spec.entrypoint: /console-dataservices` and `spec.packType: AdminPage`. With `packType: AdminPage`, the entry is not shown on the portal. + +### Cause 6: the new web console plugin has no Chinese translation file + +In the browser developer tools on the real-time alert page, `GET /console-i18n/zh/plugin__console-core-service.json` returns `404 Not Found` with header `X-Alb-Err-Reason: BackendError ...`, and other plugin translation requests such as `plugin__web-cli-service` can fail too. The new web console plugins (`console-core`) insert jump links into the existing console pages but do not ship the matching Chinese translation file. The Chinese language pack is upgraded separately, and its latest version contains the file. + +## Resolution + +### Cause 1: namespace-scoped CRD management is behind the namespace-resource-manage gate + +Open `/console-platform/feature-gate`, search for `namespace-resource-manage`, and enable it. Namespace-scoped CRD management is shown again. + +### Cause 2: the namespace audit switch is greyed out because namespaceaudits is disabled + +1. Find the feature gate resource kind: + + ```bash + kubectl api-resources | grep -i featuregate + ``` + +2. In **Resource Management**, open the feature gate object that contains `namespaceaudits` and set its `enabled` field to `true`. + + > **Warning:** This turns on namespace audit for the platform. Do it only when the switch is greyed out and this entry is disabled. To roll back, set `enabled` back to `false`. + +### Cause 3: the Chinese language pack plugin is not deployed + +Deploy the Chinese language pack plugin from the cluster plugins page, then switch the UI language again. No feature gate change is needed. + +### Cause 4: the alb2 feature gate is off + +1. Read the gate: + + ```bash + kubectl get alaudafeaturegates -n cpaas-system alb2 -o yaml + ``` + +2. Enable it and set the annotation `cpaas.io/ignoreme` to `true`. + + > **Warning:** This enables load balancer management in the console. Apply it only when load balancers are missing after an upgrade and `spec.enabled` is not already `true`. To roll back, set `spec.enabled` to `false` and remove the annotation. + + ```bash + kubectl patch alaudafeaturegates alb2 -n cpaas-system --type=merge -p '{"spec":{"enabled":true},"metadata":{"annotations":{"cpaas.io/ignoreme":"true"}}}' + ``` + +3. Reload platform management. Load balancers are listed again. + +### Cause 5: the data services ProductEntry has packType AdminPage + +> **Warning:** Changing `packType` changes how the portal classifies the entry. Do it only when the entry is missing and `spec.packType` is `AdminPage`. Keep a copy first; to roll back, set `spec.packType` back to `AdminPage`. + +```bash +kubectl get ProductEntry console-dataservices -o jsonpath='{.spec.packType}{"\n"}' +kubectl get ProductEntry console-dataservices -o yaml > "$HOME/console-dataservices.yaml" +kubectl patch ProductEntry console-dataservices --type=merge -p '{"spec":{"packType":"Buildin"}}' +``` + +The portal shows the data services entry once `spec.packType` is `Buildin`. + +### Cause 6: the new web console plugin has no Chinese translation file + +1. Confirm the signature in the browser developer tools: the request for `/console-i18n/zh/plugin__console-core-service.json` returns 404. +2. Upgrade the Chinese language pack plugin to its latest version, which contains the translation file. +3. If the new web console is not used and the language pack cannot be upgraded yet, uninstalling the new web console plugins also removes the error. + + > **Warning:** Uninstalling those plugins removes the new web console jump links and any page that depends on them. Remove only the plugins added for the new web console, not other console plugins. To roll back, install them again. + +## Diagnostic Steps + +```bash +kubectl get alaudafeaturegates -n cpaas-system | grep -E 'alb2|exec|namespace' +kubectl get ProductEntry console-dataservices -o yaml +``` + +```text +CRD management missing on 4.x, namespace-resource-manage off -> Cause 1 +namespace audit switch greyed out -> Cause 2 +cannot switch to Chinese after 4.2.4 -> Cause 3 +load balancers missing after an upgrade, alb2 disabled -> Cause 4 +data services entry missing, packType AdminPage -> Cause 5 +plugin_console-core-service.json missing, /console-i18n/zh/... 404 -> Cause 6 +EXEC missing, cpc-exec disabled -> the cpc-exec article linked above +``` diff --git a/docs/en/solutions/Console_returns_404_shows_old_pages_or_misses_menus_after_an_upgrade_on_ACP.md b/docs/en/solutions/Console_returns_404_shows_old_pages_or_misses_menus_after_an_upgrade_on_ACP.md new file mode 100644 index 000000000..f12ea865b --- /dev/null +++ b/docs/en/solutions/Console_returns_404_shows_old_pages_or_misses_menus_after_an_upgrade_on_ACP.md @@ -0,0 +1,163 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Console returns 404, shows old pages, or misses menus after an upgrade on ACP + +## Issue + +After a platform upgrade, or on a particular version or package, the console shell does not match what is expected, in every browser: + +- requests to the base platform services return HTTP 404 while the API Pods are Running (seen on 4.0.9); +- pages mix old and new views after an upgrade from 4.0 to 4.3.2, and the `frontend` image is back on the 4.0 tag; +- the login page is blank right after images were updated as a vulnerability fix; +- the DevOps entry is missing from Platform Management and Container Management; +- Backup Management has no etcd backup page on 4.1, the etcd monitor panel is empty on 4.2 and later, or License Management is missing; +- on bare metal, the network page cannot create MetalLB, and the Kube-OVN subnet, external address pool and VLAN sections are missing; +- a ConfigMap value entered with line breaks becomes one line in YAML, and the mounted file has Windows-style line breaks; +- new jobs are not visible in the job list after an upgrade on 3.18. + +Related article: [Console white page, clipped menu, dead Create button, or empty Pod logs in one browser on ACP](Console_white_page_clipped_menu_dead_Create_button_or_empty_Pod_logs_in_one_browser_on_ACP.md). + +## Root Cause + +### Cause 1: sentry restored the frontend image to the pre-upgrade tag + +After an upgrade from 4.0 to 4.3.2 the `frontend` image was back on the 4.0 tag; the audit log showed that `sentry` had restored it. Until the image matches the platform, pages mix old static files with the new API. + +### Cause 2: the frontend image configuration does not match the upgraded platform + +After an upgrade to 4.0.9, base platform services returned HTTP 404 because the `frontend` image configuration was wrong. Pods are Running while the image they run is not the one for the upgraded platform. + +### Cause 3: images updated as a vulnerability fix break the login page + +After images were replaced as a vulnerability fix, the login page is blank and nobody can log in. Reverting those image updates restores login. + +### Cause 4: DevOps is missing on the global cluster or has the wrong version there + +The DevOps entry is read from the global cluster. DevOps installed only on a business cluster never shows the entry; it appears once DevOps is deployed on global and a Katanomi instance is created there. After global moves from 3.18 to 4.0 while `katanomi` stays on 3.18, the package for the new global version was never pushed, and both Platform Management and Container Management hide DevOps. + +### Cause 5: the menu was removed or not shipped in this version or package + +- 4.1 removed the etcd backup page from Backup Management. Backups are still visible through the API, and schedules are changed on the `etcdbackupconfiguration` object. +- From 4.2 the 4.0 etcd monitor panel is discontinued, so its charts stay empty even when etcd is healthy. +- The overseas 4.3.2 package omits License Management, including the menu entry and the path `/console-platform/manage-platform/platform-setting/license`; the China distribution has it. + +### Cause 6: the ProviderBaremetal adaptation is incomplete before 4.3.3 + +On bare metal, `Artifact` and `ArtifactVersion` for `metallb-operator` exist and marketplace logs are clean, but `console-plugins` returns no subnet payload and the cluster overview has no network-mode field. The `ProviderBaremetal` adaptation is incomplete in that build and is fixed in 4.3.3. + +### Cause 7: the ConfigMap form converts line breaks + +The ConfigMap form's character conversion turns a multi-line value into one line, and the mounted file then has Windows-style line breaks. The YAML editor keeps the breaks. The form behaves correctly from 4.1.4. + +### Cause 8: the 3.18 job list shows only the first 20 jobs + +When more than 20 jobs exist, the 3.18 list page stops at 20. The other jobs exist in the API and search finds them. 4.0 changed the list. + +## Resolution + +### Cause 1: sentry restored the frontend image to the pre-upgrade tag + +```bash +kubectl get deploy -A | grep frontend +kubectl get deploy -n \ + -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" "}{.image}{"\n"}{end}' +``` + +> **Warning:** Changing the `frontend` image rolls the console shell; users lose the page until the new Pod is Ready. Precondition: the running tag is older than the upgraded platform. Rollback: set the previous image again, which brings back the mismatched UI. + +Set the container image to the tag that belongs to the upgraded platform, then watch whether `sentry` reverts it. If it keeps writing the old tag back, the release that `sentry` reconciles still carries the old tag, and a one-off edit will not stick; check the audit log and that release before editing again. + +### Cause 2: the frontend image configuration does not match the upgraded platform + +```bash +kubectl get deploy -n cpaas-system -o custom-columns='NAME:.metadata.name,IMAGE:.spec.template.spec.containers[0].image' | grep frontend +``` + +> **Warning:** Setting a new `frontend` image rolls the console shell; users lose the page until the new Pod is Ready. Precondition: base services return 404 after an upgrade, and this image does not match that upgrade. Rollback: set the previous image. + +```bash +kubectl set image deployment/ -n cpaas-system = +kubectl rollout status deployment/ -n cpaas-system +``` + +Requests stop returning 404 once the running image matches. If `sentry` writes the old image back, see Cause 1. + +### Cause 3: images updated as a vulnerability fix break the login page + +> **Warning:** Reverting the vulnerability-fix images puts the previous tags back and re-opens what those updates closed. Precondition: the blank login page started with that image update. Keep the list of tags you revert. Re-applying the fix tags is the rollback, and it can blank the login page again; obtain a supported image set before re-applying the fix. + +Revert each image that was changed as part of the vulnerability fix to its previous tag, using the same method that applied it. Login returns. + +### Cause 4: DevOps is missing on the global cluster or has the wrong version there + +Install DevOps on the global cluster and create the Katanomi instance there, not only on the business cluster. After a global upgrade, push the DevOps package that belongs to the new global version and upgrade the component so the `katanomi` version matches. The Katanomi instance fields are described in Cause 2 of [DevOps v3 toolchain entry missing after an upgrade: Katanomi MonitorDashboard ownership conflict or missing instance](DevOps_v3_toolchain_entry_missing_after_an_upgrade_Katanomi_MonitorDashboard_ownership_conflict_or_missing_instance.md). + +### Cause 5: the menu was removed or not shipped in this version or package + +- etcd backups on 4.1: use the API, and edit the object when a schedule must change: + + ```bash + kubectl get etcdbackupconfiguration -A + kubectl edit etcdbackupconfiguration -n + ``` + +- etcd monitor panel on 4.2 and later: use the monitoring dashboard shipped with the running version instead of the 4.0 panel; do not repair the monitoring stack for it. +- License Management: check which distribution is installed. Getting the feature means installing a distribution that contains it; reinstalling the console does not add it. + +### Cause 6: the ProviderBaremetal adaptation is incomplete before 4.3.3 + +Confirm that the `metallb-operator` artifacts are present, so it is not a missing upload, then upgrade to 4.3.3 or later. The subnet, external address pool and VLAN sections and MetalLB creation return on the fixed build. + +```bash +kubectl get artifact,artifactversion -A | grep metallb +``` + +### Cause 7: the ConfigMap form converts line breaks + +Open the ConfigMap, switch to **YAML**, enter the value with real line breaks (for example as a `|` block), and apply. Mount the key again and confirm that the file has no Windows-style breaks: + +```bash +kubectl exec -n -- cat -A | head +``` + +`^M` before `$` at line ends shows Windows-style breaks. From 4.1.4 the form keeps the breaks. + +### Cause 8: the 3.18 job list shows only the first 20 jobs + +Search for the job by name before treating it as a failed create; search opens jobs beyond the first 20. Upgrading to 4.0 or later removes that list limit. + +## Diagnostic Steps + +1. Read the images and their tags, and compare them with the platform version: + + ```bash + kubectl get pods -n cpaas-system | grep -E 'frontend|archon' + kubectl get deploy -A -o wide | grep -E 'frontend|archon' + ``` + +2. Map the result: + + ```text + frontend tag older than the platform, sentry wrote it back -> Cause 1 + base services return 404 after an upgrade -> Cause 2 + blank login page right after a vulnerability image update -> Cause 3 + DevOps entry absent, katanomi old or only on a business cluster -> Cause 4 + no etcd backup page on 4.1, no License Management overseas -> Cause 5 + MetalLB and subnet pages missing on bare metal before 4.3.3 -> Cause 6 + ConfigMap value collapsed to one line after a form edit -> Cause 7 + new jobs missing from the 3.18 list, search finds them -> Cause 8 + only one browser affected -> related article + ``` + +3. A missing **EXEC** entry or another menu behind a feature gate is [Console menu or feature is missing because a feature gate, plugin, or product entry is off on ACP](Console_menu_or_feature_is_missing_because_a_feature_gate_plugin_or_product_entry_is_off_on_ACP.md). diff --git a/docs/en/solutions/Console_returns_500_or_data_loading_errors_after_a_host_network_change_on_ACP.md b/docs/en/solutions/Console_returns_500_or_data_loading_errors_after_a_host_network_change_on_ACP.md new file mode 100644 index 000000000..bac589b37 --- /dev/null +++ b/docs/en/solutions/Console_returns_500_or_data_loading_errors_after_a_host_network_change_on_ACP.md @@ -0,0 +1,141 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Console returns 500 or data loading errors after a host network change on ACP + +## Issue + +One of the following starts right after a change on the cluster hosts, not on ACP objects: + +- The management console returns `x-error 500` after node DNS was changed and NetworkManager was restarted. +- Switching between platform sub-pages sometimes returns a 500 internal server error, more often in an incognito window. +- The native application page keeps showing `数据加载错误` (data loading error), the external address pool page shows `数据加载错误 重试` (data loading error, retry), the cluster shows no subnet information, and later the whole platform is unusable. + +Platform Pods can all be `Running` while their network path is broken. The common check is to compare what works from the node with what works from inside a Pod on that node, and then to find the host change. + +If nothing changed on the hosts, and pages are slow or fail together, see [Console is slow or shows data loading errors because calls through erebus or the marketplace backends are slow on ACP](Console_is_slow_or_shows_data_loading_errors_because_calls_through_erebus_or_the_marketplace_backends_are_slow_on_ACP.md). + +## Root Cause + +### Cause 1: NetworkManager was restarted after a DNS change and Pods time out to the API Service + +Node DNS was changed with `nmcli conn mod "" ipv4.dns ""` and NetworkManager was restarted. Afterwards a platform component logs: + +```text +Error: Get "https://:443/version": dial tcp :443: i/o timeout +``` + +The same request succeeds from the node and fails from inside the Pod. From inside the Pod public name servers cannot be reached either, which first looks like a CoreDNS problem. The fault is NetworkManager running and being restarted on the node; stopping the NetworkManager service returns the environment to normal. How the restart breaks Pod traffic is not established. + +### Cause 2: a Multus CNI configuration breaks the frontend Pod's network to dex + +Inside the `frontend` container, a request to the dex Service on port 5556 times out: + +```text +curl: (28) Failed to connect to port 5556: Connection timed out +``` + +On the host, `/etc/cni/net.d` contains a Multus configuration that broke the container networking of the `frontend` Pod, so it cannot reach dex and pages return 500. Inside the container the network looks normally configured: `ip a` shows `eth0@if` with a Pod-network address and `route -n` shows the default route through `eth0`. Removing Multus and restarting the affected Pods restores the connection. + +### Cause 3: the host IPv6 kernel parameters were changed + +When the IPv6 kernel parameters on the hosts are modified, many pages show `数据加载错误` (data loading error) and the platform degrades until it is unusable. Developer tools on the native application page can look clean, with only a `heartbeat?action=false` request returning 200. Restoring the previous IPv6 parameters returns the pages to normal. + +## Resolution + +### Cause 1: NetworkManager was restarted after a DNS change and Pods time out to the API Service + +1. List what is not `Running`, and compare the node with the Pod: + + ```bash + kubectl get pod -A -o wide | grep -vi running + kubectl logs -n --tail=100 | grep -E 'i/o timeout|/version' + curl -k --max-time 5 https://:443/version + systemctl is-active NetworkManager + systemctl is-enabled NetworkManager + ``` + + Run the `curl` on the node, then the same request from inside the failing Pod with `kubectl exec` if its image has `curl`. Success on the node and a timeout in the Pod, right after a NetworkManager restart, matches this cause. + +2. Stop NetworkManager on the affected nodes. + + > **Warning:** With NetworkManager stopped, nothing reapplies the host's connection profiles until it is started again; addresses already configured stay in place. Do this only on nodes where the 500 followed a NetworkManager restart and Pods time out to the API Service while the node reaches it. Keep out-of-band console access to the node. To roll back, run `systemctl start NetworkManager`, which brings the fault back. + + ```bash + systemctl stop NetworkManager + kubectl get pod -A -o wide | grep -vi running + ``` + +3. Decide whether NetworkManager should also stay disabled at boot according to how the hosts' networking is managed. Keep the intended DNS server in the node configuration. + +### Cause 2: a Multus CNI configuration breaks the frontend Pod's network to dex + +1. From the `frontend` Pod, test dex and read the paired host interface: + + ```bash + kubectl get pod -A -o wide | grep -E 'frontend|dex' + kubectl get svc -A | grep dex + kubectl exec -n cpaas-system -- curl -k -I --max-time 5 https://:5556/ + kubectl exec -n cpaas-system -- cat /sys/class/net/eth0/iflink + ``` + +2. On the node that runs the Pod, list the CNI configuration and look up the interface index printed by `iflink`: + + ```bash + ls -l /etc/cni/net.d/ + ip -o link | awk -F': ' '{print $1": "$2}' | grep '^:' + ``` + + A Multus file in `/etc/cni/net.d` together with a dex timeout from `frontend` matches this cause. + +3. Uninstall Multus the way it was installed, make sure its configuration file is gone from `/etc/cni/net.d` on every node, and restart the affected Pods. + + > **Warning:** Uninstalling Multus removes the secondary networks it provides; every Pod attached to a `NetworkAttachmentDefinition` loses those interfaces when it is recreated. Do this only when no workload on the cluster needs Multus, in a maintenance window. To roll back, install Multus again the same way. + + ```bash + kubectl delete pod -n cpaas-system + kubectl exec -n cpaas-system -- curl -k -I --max-time 5 https://:5556/ + ``` + +### Cause 3: the host IPv6 kernel parameters were changed + +1. Compare the IPv6 parameters on an affected node with a node that was not changed, and look for recent edits: + + ```bash + sysctl -a 2>/dev/null | grep -E '^net\.ipv6\.conf\.(all|default)\.' + ls -lt /etc/sysctl.conf /etc/sysctl.d/ + grep -rn ipv6 /etc/sysctl.conf /etc/sysctl.d/ + ``` + +2. Set the changed parameters back to their previous values, and restore them in the file that was edited so that they survive a reboot. + + > **Warning:** IPv6 kernel parameters affect host and Pod networking on that node immediately. Do this only for values that differ from the healthy nodes or from the values before the change, and record the current values first. To roll back, set the recorded values again. + + ```bash + sysctl -w = + ``` + +## Diagnostic Steps + +1. Find out what changed on the hosts just before the errors started: DNS, NetworkManager, CNI files, or kernel parameters. + + ```bash + journalctl -u NetworkManager --since "-24h" --no-pager | tail -50 + ls -lt /etc/cni/net.d/ + ls -lt /etc/sysctl.conf /etc/sysctl.d/ + ``` + +2. Test the same target from the node and from inside a platform Pod on that node. Expected for this article: the node works and the Pod fails. + +3. Match the signature: + + ```text + x-error 500 after an nmcli DNS change and a NetworkManager restart, Pod i/o timeout to :443 -> Cause 1 + intermittent 500 on sub-page switch, frontend cannot reach dex :5556, Multus in /etc/cni/net.d -> Cause 2 + 数据加载错误 on many pages, no subnet information, host IPv6 parameters changed -> Cause 3 + ``` diff --git a/docs/en/solutions/Console_returns_502_or_does_not_open_after_installation_or_a_VIP_change_on_ACP.md b/docs/en/solutions/Console_returns_502_or_does_not_open_after_installation_or_a_VIP_change_on_ACP.md new file mode 100644 index 000000000..ead230101 --- /dev/null +++ b/docs/en/solutions/Console_returns_502_or_does_not_open_after_installation_or_a_VIP_change_on_ACP.md @@ -0,0 +1,197 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Console returns 502 or does not open after installation or a VIP change on ACP + +## Issue + +The console host does not answer, the browser receives HTTP 502 from ALB, or the installation wizard never leaves its first step. Typical situations: + +- After a new ACP 4.4 installation, the platform opens with an nginx `502`, although every Pod is `Running`. +- A new `global` installation stays on the repository initialization step (`初始化仓库配置`). +- An EIP is bound to the cluster, but the management page does not load through it. +- A business cluster reports an `alive` error, and the console cannot operate it. +- Web CLI drops a few seconds after it opens, and real-time log follow does the same. + +In the browser developer tools there is no HTTP response, a 502, or a long pending request: a reachability problem. An HTTP 500 with a JSON body from `apollo` is a different problem; see [Console returns HTTP 500 or a blank page because apollo or courier on the global cluster fails on ACP](Console_returns_HTTP_500_or_a_blank_page_because_apollo_or_courier_on_the_global_cluster_fails_on_ACP.md). + +Related article: [Console is inaccessible or a business cluster shows 0 nodes because global cannot reach an API server on ACP](Console_is_inaccessible_or_a_business_cluster_shows_0_nodes_because_global_cannot_reach_an_API_server_on_ACP.md). + +## Root Cause + +### Cause 1: ACP 4.4 was installed without all Required Aligned Extensions + +From ACP 4.4 the web console is not part of the Core Package. The Required Aligned Extensions must be copied into the `plugins/` directory of the extracted Core Package before the installer runs: + +- Alauda Container Platform Web Console +- Alauda Container Platform Web Console Central +- Alauda Container Platform Marketplace Web Console +- Alauda Container Platform Helm Application Catalog +- Alauda Container Platform Observability Essentials +- Alauda Container Platform Essentials +- Alauda Container Platform Cluster Essentials +- Alauda Container Platform GatewayAPI Plugin + +If they are missing, the console Pods, Services, and Endpoints are never created and ALB returns 502. Leaving out a single one gives the same 502: every Pod is `Running` and the `alb2-controller` log can even show 200 responses from the login chain. Copying additional marketplace packages beyond this set into `plugins/` makes `setup.sh` fail. + +### Cause 2: kubelet does not start because required kernel parameters are missing + +A new `global` installation stays on the repository initialization step. The `cpaas-install` logs are quiet, the `cls` object reports errors, `kubelet` is not active, and its journal says required kernel parameters are missing. No container can start until `kubelet` runs. + +### Cause 3: the alive VIP mode is not arp + +`alive` holds the cluster VIP. When the VIP mode file `/etc/kubernetes/alive/vipmode` on the nodes is not `arp`, an EIP bound to the cluster does not reach the management page. + +### Cause 4: alive crash-loops because alive.yaml names a NIC that no longer exists + +The business cluster's nodes are `Ready`, but `kubectl describe cls` shows that port 6443 on the business cluster fails, `cluster-api-provider-alauda-manager` logs an error loading the cluster, and the `alive` Pod is `CrashLoopBackOff` with a log of mostly `logrotate.state` and VIP or IPVS state. The network interface name changed (for example from `ens192` to `eth0`), but `alive.yaml` still names the old one, so `alive` cannot attach the VIP and the `global` cluster cannot reach the business API. + +### Cause 5: an external stream proxy closes long-lived connections after 5 seconds + +Web CLI opens (status 101) and dies a few seconds later. The ALB rule `cpaas-system/web-cli` logs `/auth/v1/pod/shell` with a `request_time` of a few seconds, and `web-terminal` logs `EOF`. Bypassing the external nginx makes the session stable. On that nginx, the `stream` listener that forwards 6443 has `proxy_timeout 5s`, while 443 uses `3m`. + +## Resolution + +### Cause 1: ACP 4.4 was installed without all Required Aligned Extensions + +1. Check whether the console has Pods, Services, and Endpoints. Empty results on a new 4.4 installation match this cause: + + ```bash + kubectl get pods,svc,endpoints -A | grep -Ei 'frontend|console|alb' + ``` + +2. Compare the packages that were copied into the Core Package `plugins/` directory with the eight Required Aligned Extensions: + + ```bash + ls /plugins/ + ``` + +3. To rule out a self-built VIP, test it from each control-plane node and compare: TLS handshake, status code, and agreement between nodes. A TLS handshake EOF points at the VIP (loopback access, ARP, routing, or the load balancer health checks), not at the ALB certificates: + + ```bash + curl -vk --connect-timeout 5 --resolve :443: https:/// + ``` + +4. Install again with the complete set. + + > **Warning:** The recovery runs `cleanup.sh` on every node and installs again, which tears down the failed installation. Never run it against a cluster that already serves workloads, and copy off any data you need first. A wiped installation can only be restored from backup. + + Copy exactly the eight Required Aligned Extensions (and no other marketplace packages) into `plugins/`, run `cleanup.sh` on each node, then run `setup.sh` as documented. ALB stops returning 502 once the console Service has Endpoints. + +If DNS problems stopped Kubernetes from coming up during the same installation, fix DNS and finish the cluster installation first; that is a separate, earlier fault. + +### Cause 2: kubelet does not start because required kernel parameters are missing + +1. On the node, check `kubelet` and the kernel parameters: + + ```bash + systemctl status kubelet --no-pager + journalctl -u kubelet -n 100 --no-pager + sysctl vm.overcommit_memory vm.panic_on_oom kernel.panic kernel.panic_on_oops kernel.keys.root_maxkeys kernel.keys.root_maxbytes + ``` + +2. Apply the parameters and restart `kubelet`. + + > **Warning:** `kernel.panic=10` and `kernel.panic_on_oops=1` make the node reboot itself after a kernel panic. Apply them only on nodes that will run ACP, and keep console access in case a key is rejected. To roll back, remove `/etc/sysctl.d/acp.conf`, restore the previous values, and run `sysctl` again. + + ```bash + cat >/etc/sysctl.d/acp.conf <<'EOF' + vm.overcommit_memory = 1 + vm.panic_on_oom = 0 + kernel.panic = 10 + kernel.panic_on_oops = 1 + kernel.keys.root_maxkeys = 1000000 + kernel.keys.root_maxbytes = 25000000 + EOF + sysctl -p /etc/sysctl.d/acp.conf + systemctl restart kubelet + systemctl is-active kubelet + ``` + +3. Once `kubelet` stays active, the wizard can continue. If `kubelet` is active and the wizard is still stuck, go back to the `cls` error. + +### Cause 3: the alive VIP mode is not arp + +On each node where `alive` runs: + +> **Warning:** Changing the VIP mode and restarting keepalived moves or drops the management VIP for a few seconds, and open console sessions break. Do this in a maintenance window. To roll back, write the previous mode back to the file and restart keepalived again. + +```bash +cat /etc/kubernetes/alive/vipmode +echo arp > /etc/kubernetes/alive/vipmode +kubectl get pods -n kube-system --no-headers | awk '/alive/ {print $1}' +kubectl exec -n kube-system -- pkill keepalived +``` + +The management page answers on the EIP once `alive` is `Running` again. + +### Cause 4: alive crash-loops because alive.yaml names a NIC that no longer exists + +1. Compare the live interfaces with the one in the `alive` configuration: + + ```bash + ip -br link + kubectl logs -n kube-system --tail=100 + ls /etc/kubernetes/alive + ``` + +2. Back up `alive.yaml` and change the interface to the one that `ip -br link` shows as `UP`. + + > **Warning:** A wrong interface name leaves the VIP unbound and the business cluster unreachable from `global`. Restoring the backup is the rollback. + + ```bash + cp -a /etc/kubernetes/alive/alive.yaml /root/alive.yaml.bak + vi /etc/kubernetes/alive/alive.yaml + ``` + +3. Restart `cluster-api-provider-alauda-manager` on the `global` cluster so that it reloads cluster access. + + > **Warning:** The restart briefly pauses cluster reconciliation on the `global` cluster. + + ```bash + kubectl get deploy -A | grep cluster-api-provider-alauda + kubectl rollout restart deployment -n cluster-api-provider-alauda-manager + kubectl get pods -n kube-system | grep alive + ``` + + `alive` leaves `CrashLoopBackOff`, and `kubectl describe cls` stops reporting 6443 failures. + +### Cause 5: an external stream proxy closes long-lived connections after 5 seconds + +On the external nginx that publishes the VIP, set the `stream` server that listens on 6443 to the same order of timeout as 443, then reload nginx: + +> **Warning:** A longer `proxy_timeout` keeps idle TCP connections open longer on that proxy; it does not change ACP. To roll back, restore the previous value, which brings the disconnects back. + +```nginx +# stream server that listens on 6443 +proxy_timeout 3m; +``` + +Confirm that `/auth/v1/pod/shell` stays on 101 with a growing `request_time`, and that `web-terminal` no longer logs an immediate `EOF`. + +## Diagnostic Steps + +```bash +curl -k --max-time 5 https://:6443/readyz +kubectl get pods -n kube-system | grep -E 'alive|kube-apiserver' +kubectl get pods -n cpaas-system | grep -E 'alb|frontend|apollo' +kubectl get pods,svc,endpoints -A | grep -Ei 'frontend|console' +``` + +```text +ALB 502, no console Endpoints on a new 4.4 installation, or one extension missing -> Cause 1 +kubelet inactive, installation stuck on repository initialization -> Cause 2 +EIP silent, vipmode is not arp -> Cause 3 +alive CrashLoopBackOff, alive.yaml names a NIC that does not exist -> Cause 4 +Web CLI status 101 then EOF, only through the external nginx -> Cause 5 +``` diff --git a/docs/en/solutions/Console_returns_HTTP_500_or_a_blank_page_because_apollo_or_courier_on_the_global_cluster_fails_on_ACP.md b/docs/en/solutions/Console_returns_HTTP_500_or_a_blank_page_because_apollo_or_courier_on_the_global_cluster_fails_on_ACP.md new file mode 100644 index 000000000..9b17e06b3 --- /dev/null +++ b/docs/en/solutions/Console_returns_HTTP_500_or_a_blank_page_because_apollo_or_courier_on_the_global_cluster_fails_on_ACP.md @@ -0,0 +1,195 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.14" + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Console returns HTTP 500 or a blank page because apollo or courier on the global cluster fails on ACP + +## Issue + +The browser reaches the platform host, but a page stays blank, spins, or returns HTTP 500. Typical reports are "apollo reports errors and the platform returns 500", a monitoring home page that spins for one cluster, or Operations Center saying that monitoring is not installed while the monitoring Pods are `Running`. + +Start from the failing request, not the screenshot. In the browser developer tools (**Network**), sort by status and note the first failing path and its status code: + +| Failing request | First component to check | +| --- | --- | +| Broad `/platform/...` 500, or the document itself errors after login | `apollo` and other `cpaas-system` Pods on the `global` cluster | +| `/platform/monitoring.alauda.io/v1beta1/clusters//metrics` returns 500, Grafana says `cluster not found` | the `apollo` cluster cache | +| Operations Center says monitoring is not installed, or vmui returns 403 | `courier` and `courier-api` | +| The document request never completes, or HTTP 502 from the load balancer | the entry path, not this article | + +`apollo`, `courier`, `courier-api`, `frontend`, `archon`, and `erebus` are console backends in `cpaas-system` on the `global` cluster. `apollo` aggregates the platform API, `erebus` proxies calls toward a target cluster, `courier` and `courier-api` feed the monitoring and inspection views, `frontend` serves the shell, and `archon` serves file download from containers. + +## Root Cause + +### Cause 1: apollo crash-loops with apiserver 401 because node clocks differ + +`apollo` is `CrashLoopBackOff` and its log shows cache calls returning 401 and an API server authentication failure. The node clocks are skewed, for example after nodes were powered off without stopping `kubelet` and the container runtime, or after a single master was rebooted. Token authentication between platform components fails when the clocks differ. On a 3.14 `global` cluster the same skew after one master reboot showed as `katanomi` not starting, `archon` and other components returning 401, and the platform saying the license could not be loaded. Recreating `apollo` without fixing the clocks clears the symptom only until the next authentication check. + +### Cause 2: apollo does not have the cluster in its cache + +The monitoring home page of a cluster spins, the metrics API returns 500, and the global `apollo` log contains `get cluster from cache error`, `cluster not found`, or `New manager error`. The business cluster's Prometheus answers. This happens for a newly created cluster, or when `apollo` Pods have run for a long time without picking up a new cluster. Restarting the global `apollo` Pods reloads the cluster list. Restarting `erebus` or other controllers does not. + +### Cause 3: apollo receives HTML instead of JSON from the metrics backend + +Monitoring charts stay empty after Prometheus was installed or reinstalled, while Prometheus itself returns `up`. `apollo` repeats `bad_response: invalid character '<' looking for beginning of value`: it received an HTML or XML body instead of JSON. A user-created Prometheus Deployment in `cpaas-system` is not necessarily the cause; restarting `apollo` restores the charts. + +### Cause 4: courier holds a stale view and the UI says monitoring is not installed + +Marketplace, the monitoring Pods, and the AppReleases are healthy, but Operations Center says the monitoring component is not installed and vmui returns 403. Restarting `courier` and `courier-api` reloads the view. + +### Cause 5: courier panics in the leftover resource-statistics job + +`courier` keeps restarting with a panic in the resource usage statistics path. Older versions ran that scheduled job; the current version no longer uses the feature, but the code that schedules it is still present. Scheduling the job for a date that never occurs stops the restarts. + +### Cause 6: the global control-plane disk is full of etcd backups + +The browser reports that the site cannot be reached, and `kubectl get pods -n cpaas-system` shows Pods that are not `Running`. The global control-plane filesystem is at 100% because etcd backups under `/cpaas-backup` filled it. This is a node disk fault that shows as a dead console. + +## Resolution + +### Cause 1: apollo crash-loops with apiserver 401 because node clocks differ + +1. Compare the clocks of all nodes. After a single master reboot, that node is the likely outlier: + + ```bash + date -u + timedatectl status + ``` + +2. Synchronize every control-plane and worker node to the same NTP source. + + > **Warning:** Stepping a node clock affects TLS, leases, and log order. Prefer slewing through the NTP service; if the clock must be stepped, do it in a maintenance window and expect brief API server 401 errors until `apollo` is restarted. + +3. Restart `apollo` only, and confirm the new Pod stays `Running` and the 500 is gone: + + ```bash + kubectl get pods -n cpaas-system --no-headers | awk '$1 ~ /^apollo-/ {print $1}' + kubectl delete pod -n cpaas-system + ``` + +### Cause 2: apollo does not have the cluster in its cache + +1. Confirm the log signature: + + ```bash + kubectl logs -n cpaas-system --tail=200 | grep -E 'from cache error|not found|New manager error' + ``` + +2. Restart every global `apollo` Pod. + + > **Warning:** Platform API calls, including monitoring and other aggregated pages, return errors until the new Pods are `Ready`. Do this when users can retry. The Pods are recreated from the same spec. If the new Pods log the same `cluster not found` after several minutes, the cluster object is missing or the API watch is failing; do not keep deleting Pods. + + ```bash + kubectl delete pod -n cpaas-system + ``` + +The monitoring-specific variants of this cache problem are covered in more detail in [Monitoring pages stay empty because courier-api or apollo holds a stale cluster cache on ACP](Monitoring_pages_stay_empty_because_courier-api_or_apollo_holds_a_stale_cluster_cache_on_ACP.md). + +### Cause 3: apollo receives HTML instead of JSON from the metrics backend + +1. Confirm the log line: + + ```bash + kubectl logs -n cpaas-system --tail=200 | grep bad_response + ``` + +2. Restart `apollo`. + + > **Warning:** Platform APIs return errors until the Pod is `Ready`. Do not delete a user's Prometheus just because it is in `cpaas-system`. + + ```bash + kubectl delete pod -n cpaas-system + ``` + +3. If `bad_response` continues, the upstream still returns a non-JSON body; check it before restarting `apollo` again. + +### Cause 4: courier holds a stale view and the UI says monitoring is not installed + +> **Warning:** Restarting these Deployments interrupts the monitoring and inspection views until the new Pods are `Ready`. The monitoring store keeps running. + +```bash +kubectl get deploy -n cpaas-system | grep courier +kubectl rollout restart deployment -n cpaas-system +kubectl rollout restart deployment -n cpaas-system +kubectl rollout status deployment -n cpaas-system +``` + +### Cause 5: courier panics in the leftover resource-statistics job + +1. Export the Deployment: + + ```bash + kubectl get deploy -n cpaas-system -o name | grep courier + kubectl get deploy -n cpaas-system -o yaml > "$HOME/courier-deploy.yaml" + ``` + +2. Add the following argument to the `command` of the `courier` container. It schedules the job for 31 February, which never occurs. + + > **Warning:** Editing the command rolls the Deployment and interrupts health and monitoring views until the new Pod is `Ready`. Do this only when the panic is in the resource-statistics job. To roll back, apply the export, which re-enables the job and can bring the panic back. + + ```yaml + - --record-job-spec=CRON_TZ=Asia/Shanghai 0 0 31 2 * + ``` + + ```bash + kubectl edit deploy -n cpaas-system + ``` + +3. `courier` stays `Running` after the rollout. If the argument disappears and the panic returns, the owning release is still rendering the old command. + +### Cause 6: the global control-plane disk is full of etcd backups + +1. Find what fills the disk: + + ```bash + df -h + du -sh /cpaas-backup/* 2>/dev/null | sort -h + ``` + +2. Remove old backups. + + > **Warning:** Files under `/cpaas-backup` are etcd backups and deleting them is irreversible. Copy the newest successful backup to another disk first, and never touch the live etcd data directory. If the disk is full for another reason, removing backups is not the fix. + + ```bash + rm -f /cpaas-backup/ + df -h + ``` + +3. Watch the `cpaas-system` Pods return to `Running`, then load the console again. + +## Diagnostic Steps + +1. In the browser **Network** panel, reproduce the problem once and note the first failing path and status. + +2. On the `global` cluster, list the platform Pods that are not `Running` and read the matching log: + + ```bash + kubectl get pods -n cpaas-system -o wide + kubectl get pods -n cpaas-system | grep -viE 'Running|Completed' + kubectl logs -n cpaas-system --since=30m --tail=200 + ``` + +3. Match the log text: + + ```text + 401 and CrashLoopBackOff on apollo -> Cause 1 (node time) + archon 401, license cannot be loaded after one master reboot -> Cause 1 + get cluster from cache error -> Cause 2 + bad_response: invalid character '<' -> Cause 3 + vmui 403, UI says monitoring is not installed -> Cause 4 + courier panic in the resource-statistics job -> Cause 5 + filesystem 100% and /cpaas-backup is large -> Cause 6 + ``` + +4. If only one feature path fails (logs, a registry, DevOps, a data-service instance) and `apollo` is healthy, the problem is in that product, not in the platform backends. diff --git a/docs/en/solutions/Console_returns_HTTP_500_or_clusters_show_abnormal_because_the_global-kubeconfig_or_cluster_kubeconfig_Secret_expired_on_ACP.md b/docs/en/solutions/Console_returns_HTTP_500_or_clusters_show_abnormal_because_the_global-kubeconfig_or_cluster_kubeconfig_Secret_expired_on_ACP.md new file mode 100644 index 000000000..38817eb37 --- /dev/null +++ b/docs/en/solutions/Console_returns_HTTP_500_or_clusters_show_abnormal_because_the_global-kubeconfig_or_cluster_kubeconfig_Secret_expired_on_ACP.md @@ -0,0 +1,157 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Console returns HTTP 500 or clusters show abnormal because the global-kubeconfig or cluster kubeconfig Secret expired on ACP + +## Issue + +The platform suddenly cannot be opened after running for about one year. Any of the following can be seen: + +- The page shows "Internal server error. System exception, please contact the platform administrator", `/console-portal/` returns HTTP 500, or login fails (`平台无法登录`, the platform cannot be logged in). Many platform API calls return 500 at the same time. +- The two `apollo` Pods in `cpaas-system` are `0/1 CrashLoopBackOff`. Their log shows `Updated current TLS certificate` and `Starting certificate watcher`, then `ERROR cache New cache error:could not create RESTMapper from config` and `Error: failed to initialize dexClientMgr: could not create RESTMapper from config`. Other `cpaas-system` Pods can be `Pending` or `CrashLoopBackOff`. +- `kubectl get cls` on the global cluster shows the global or a business cluster as `failed` or abnormal. Global component logs contain `Unauthorized` or `the server has asked for the client to provide credentials`. +- The cluster management page shows every node of a cluster as abnormal and leaves CPU, memory, storage, pod count, and system load empty, while `kubectl get nodes` on that cluster prints Ready. +- After the global platform was recovered, a business cluster stays abnormal and its nodes cannot be operated, although `kubectl` on that cluster works and port 6443 is reachable from global. +- A disaster-recovery standby global cannot be opened. + +On the control-plane nodes, `kubectl` often fails at the same time with `You must be logged in to the server` unless `--kubeconfig` is passed. That node file is a separate credential, covered in "Node kubectl fails with You must be logged in to the server or an alert names root kube config on ACP". + +## Root Cause + +### Cause 1: the client certificate in cpaas-system/global-kubeconfig expired + +Secret `global-kubeconfig` in namespace `cpaas-system` holds the platform-side kubeconfig that the console and platform controllers use to reach the global cluster. Under Cluster API, kubeadm issues its client certificate with a default validity of one year. On the affected releases this certificate is not renewed automatically, which is a known defect. About one year after installation it expires; platform authentication fails, `apollo` cannot create its RESTMapper and stops starting, the console returns HTTP 500, and cluster status on global goes abnormal. Restarting `apollo` alone does not help. + +Backing up and deleting the Secret makes the platform regenerate it with a renewed certificate, after which the clusters and `apollo` recover. Judge expiry by `notAfter` only: kubeadm-style signing can copy the CA's `notBefore` into the client certificate, so the gap between the two dates is not always the issued lifetime. + +Automatic renewal of this Secret is fixed in these releases: + +| Version line | Status | +|---|---| +| 4.3 | Fixed since 4.3.0 | +| 4.2 | Fixed in 4.2.5; earlier 4.2 releases are affected | +| 4.1 | Fix targeted for 4.1.6; check the release notes of your version | +| 4.0 | Fix targeted for 4.0.10; check the release notes of your version | + +After an upgrade to a release later than 4.2.6, the certificate issued in `global-kubeconfig` is valid for 10 years. Automatic renewal only prevents expiry: a Secret that is already past `notAfter` must be recreated with the procedure below on any version, including a fixed one. A disaster-recovery standby global has its own `global-kubeconfig` and needs the same renewal. + +### Cause 2: the client certificate in cpaas-system/-kubeconfig expired + +Each business cluster has its own Secret in `cpaas-system` on global, named `-kubeconfig`, which the platform uses to reach that cluster. It has the same one-year client certificate, the same defect, and the same fixed versions as Cause 1. When it expires, the console shows that cluster as abnormal, blanks its node utilization, and cannot operate its nodes, while the cluster itself is healthy. After `global-kubeconfig` was recreated, a business cluster whose own Secret is hundreds of days old in the `AGE` column stays abnormal until its Secret is recreated too. + +## Resolution + +Use a working `kubectl` on a global control-plane node. If `/root/.kube/config` on that node is also expired, pass `--kubeconfig /etc/kubernetes/admin.conf` on each command, or fix the node file first as described in "Node kubectl fails with You must be logged in to the server or an alert names root kube config on ACP". Check first that the `admin.conf` client certificate has a future `notAfter`: + +```bash +awk '/client-certificate-data/ {print $2}' /etc/kubernetes/admin.conf | base64 -d | openssl x509 -noout -dates +``` + +### Cause 1: the client certificate in cpaas-system/global-kubeconfig expired + +1. List the data keys of the Secret and print the end date of the client certificate. The kubeconfig is either under key `value`, or `client-certificate-data` is at the top level of the Secret: + + ```bash + kubectl get secret -n cpaas-system global-kubeconfig -o go-template='{{range $k, $v := .data}}{{printf "%s\n" $k}}{{end}}' + # kubeconfig under key value + kubectl get secret -n cpaas-system global-kubeconfig -o jsonpath='{.data.value}' \ + | base64 -d | awk '/client-certificate-data/ {print $2}' | base64 -d \ + | openssl x509 -noout -subject -dates + # client-certificate-data at the top level + kubectl get secret -n cpaas-system global-kubeconfig -o jsonpath='{.data.client-certificate-data}' \ + | base64 -d | openssl x509 -noout -subject -dates + ``` + + Continue only when `notAfter` is in the past. If it is in the future, this is not the cause: check `apollo`, node clocks, and node readiness instead. When nothing is broken and the question is only whether the certificate expires, answer from the dates and delete nothing. A planned renewal of a Secret that is within days of `notAfter` can be done the same way in a maintenance window. + +2. Back up and delete the Secret, and wait for the platform to regenerate it. + + > **Warning:** Deleting `cpaas-system/global-kubeconfig` removes the platform's own credential until the platform regenerates it; the console stays unavailable during that gap. Do this only when step 1 shows the certificate is expired (or within days of expiry, for a planned renewal), and only from a node where `admin.conf` works. The backup contains a client key: keep it at mode `0600` and remove it once the new end date is confirmed. Rollback is `kubectl apply -f` of the backup, which restores the old (expired) credential. On a disaster-recovery pair, repeat on the standby global. + + ```bash + umask 077 + kubectl get secret -n cpaas-system global-kubeconfig -o yaml > "$HOME/global-kubeconfig.yaml" + kubectl delete secret -n cpaas-system global-kubeconfig + kubectl get secret -n cpaas-system global-kubeconfig + ``` + +3. Print the end date again with the command from step 1; it must be in the future. Then check that `apollo` starts and the console opens: + + ```bash + kubectl get pods -A | grep apollo + ``` + +4. If `apollo` or other platform Pods are still not Ready after the new `notAfter` is in the future, or the UI and business cluster list still return 401 or 500, restart the Pods that log authentication errors, starting with `apollo`. If the global cluster status stays `failed` and `apollo` still logs a certificate error, restart the platform component `api`. + + > **Warning:** Deleting the `apollo`, `erebus`, `courier`, or `api` Pod interrupts console and cluster-proxy calls served by it until the new Pod is Running. Do this only when the recreated `global-kubeconfig` already shows a future `notAfter` and the symptoms remain. The workload recreates the Pod; there is nothing to roll back. + + ```bash + kubectl get pods -n cpaas-system --no-headers | awk '$1 ~ /apollo|erebus|courier/ {print}' + kubectl get deploy,sts -A | grep -w api + kubectl delete pod -n + ``` + +5. For a lasting fix, upgrade to a release where the Secret is renewed automatically (table in Cause 1). + +### Cause 2: the client certificate in cpaas-system/-kubeconfig expired + +1. List the cluster kubeconfig Secrets on global and print each embedded client certificate. Skip Secrets whose `notAfter` is still in the future. + + ```bash + kubectl get secret -n cpaas-system | grep kubeconfig + kubectl get secret -n cpaas-system -kubeconfig -o jsonpath='{.data.value}' \ + | base64 -d | awk '/client-certificate-data/ {print $2}' | base64 -d \ + | openssl x509 -noout -dates -subject -issuer + ``` + + If `value` is empty, list the data keys as in step 1 of Cause 1 and decode the entry that holds the client certificate. + +2. Back up and delete each expired Secret. + + > **Warning:** Deleting `cpaas-system/-kubeconfig` drops the credential platform controllers use for that cluster until the Secret is recreated; the console keeps showing that cluster as abnormal during the gap. Do this only when step 1 shows the embedded client certificate is expired, one Secret at a time, and never against a Secret that was already recreated. The backup contains a client key. Rollback is `kubectl apply -f` of the backup, which brings the expired certificate and the fault back. + + ```bash + umask 077 + kubectl get secret -n cpaas-system -kubeconfig -o yaml > "$HOME/-kubeconfig.yaml" + kubectl delete secret -n cpaas-system -kubeconfig + kubectl get secret -n cpaas-system -kubeconfig -o jsonpath='{.data.value}' \ + | base64 -d | awk '/client-certificate-data/ {print $2}' | base64 -d \ + | openssl x509 -noout -enddate + ``` + +3. Wait until the new `notAfter` is in the future, then check the cluster on global: + + ```bash + kubectl get cls -o yaml + ``` + + The conditions `HealthCheck`, `Ready`, `ControlPlaneInitialized`, `ControlPlaneReady`, and `InfrastructureReady` should be `"True"` and `phase` should be `Running`. If the cluster is still abnormal, restart the Pods as in step 4 of Cause 1. + +## Diagnostic Steps + +1. Check node readiness and the Pods that are not Running. Recreating `global-kubeconfig` does not help when a global node is NotReady; fix the node first. + + ```bash + kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes + kubectl get pods -n cpaas-system --no-headers | awk '$3 !~ /^Running$/ {print}' + kubectl logs -n cpaas-system --tail=50 | grep -i -E 'dexClientMgr|RESTMapper|certificate' + ``` + +2. A quick hint that the credentials are due is the node `AGE` column: nodes at about `365d` mean the kubeconfig Secrets of that cluster were issued about a year ago. + +3. Print `notAfter` of `global-kubeconfig` and of each `-kubeconfig` with the commands in the Resolution, including on the standby global of a disaster-recovery pair. + +4. Branch: + + - `global-kubeconfig` past `notAfter`, console 500 or `apollo` logging `could not create RESTMapper from config`: Cause 1. + - The console works, but one business cluster stays abnormal and its `-kubeconfig` is past `notAfter`: Cause 2. + - Both Secrets are in date and the log module and monitoring dashboard are what fail after more than a year of kube-apiserver uptime: see "Log module and monitoring dashboard unavailable after a kube-apiserver internal certificate expires on ACP". diff --git a/docs/en/solutions/Console_white_page_clipped_menu_dead_Create_button_or_empty_Pod_logs_in_one_browser_on_ACP.md b/docs/en/solutions/Console_white_page_clipped_menu_dead_Create_button_or_empty_Pod_logs_in_one_browser_on_ACP.md new file mode 100644 index 000000000..b0f0862eb --- /dev/null +++ b/docs/en/solutions/Console_white_page_clipped_menu_dead_Create_button_or_empty_Pod_logs_in_one_browser_on_ACP.md @@ -0,0 +1,113 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.16.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x + - 4.4.x +--- + +# Console white page, clipped menu, dead Create button, or empty Pod logs in one browser on ACP + +## Issue + +The platform API is up and the `cpaas-system` Pods are Running, but in one browser or on one PC the console misbehaves: + +- a white page, a partly drawn administrator menu, or the container platform UI does not open while another management UI on the same address does; +- **Create** for an Ingress or gateway does nothing, with no error in the browser console; +- the login page at `/console-portal` stays blank with `证书错误` (certificate error) in the address bar; +- EXEC and DEBUG open only a `文件传输` (File Transfer) button, or a `platform.cfc` download fails; +- every Pod log view is empty, and entering EXEC reports `连接失败` (connection failed); +- the console is very slow and needs manual refreshes in one browser; +- monitoring charts are blank on one PC only. + +Another browser, a private window, or another PC works. If every browser fails and the browser developer tools (F12) show HTTP 500 or 502, the problem is on the server side, for example [Console returns HTTP 500 or a blank page because apollo or courier on the global cluster fails on ACP](Console_returns_HTTP_500_or_a_blank_page_because_apollo_or_courier_on_the_global_cluster_fails_on_ACP.md). + +Related articles: [Console returns 404, shows old pages, or misses menus after an upgrade on ACP](Console_returns_404_shows_old_pages_or_misses_menus_after_an_upgrade_on_ACP.md), [Debug container blocked or left running, and console file upload or download fails on ACP](Debug_container_blocked_or_left_running_and_console_file_upload_or_download_fails_on_ACP.md). + +## Root Cause + +### Cause 1: stale browser cache or an expired idle session + +The browser holds cached static files or a session that expired while the tab was idle. Logging out and in restores a clipped menu; a private window loads the container platform at once; a different browser opens the page. There is no server-side fault. + +### Cause 2: the browser is older than the platform version supports + +The supported browser versions depend on the platform version; check the browser compatibility list in the documentation of the installed version. An older build causes client-side failures without a server error: + +- **Create** does nothing, also in a private window of the same old browser; +- a `platform.cfc` download fails on 4.3.1 in a browser below Chrome 145, while Edge succeeds; +- EXEC and DEBUG show only a **File Transfer** button in a UnionTech (UOS) browser 5.4.38 based on Chromium `83.0.4103.116`; +- the login page stays blank with `证书错误` for a `kube-ca`-issued certificate in an old browser, and displays in a newer version; +- the console is very slow in a Firefox build that is not in the validated set, while Google Chrome on the same client loads quickly; API timings in the browser trace are fast (the slowest XHR about 152 ms) and server-side checks are clean. + +### Cause 3: the PC clock is wrong + +Monitoring charts are blank on one PC while the same OS and browser draw them elsewhere, and installing another browser does not help. The query time range is likely built from the client clock. This is described in Cause 4 of [Monitoring dashboard panels are blank or drop to zero while the backend has data on ACP](Monitoring_dashboard_panels_are_blank_or_drop_to_zero_while_the_backend_has_data_on_ACP.md). + +### Cause 4: Pod logs are empty and EXEC fails because the browser is below the 4.1 requirement + +Every Pod log view is empty, including start-up and error logs, and EXEC reports a connection failure, although the application writes to standard output. On 4.1 the browser requirement includes Firefox 140 or later; a lower Firefox version produces exactly this, and both views work after the browser is upgraded. + +## Resolution + +### Cause 1: stale browser cache or an expired idle session + +1. Sign out and sign in again, then try a private window. +2. If the private window works, clear the site data for the platform address only. +3. If a second browser works and the first does not, reset or replace the broken browser profile. + +No platform Pod restart is needed when F12 shows no 5xx and `cpaas-system` is Running. + +### Cause 2: the browser is older than the platform version supports + +1. Read the browser version and its engine version on its **About** page and compare it with the compatibility list of the installed platform version. +2. Move the user to a current Google Chrome (or another browser from the list) and retry. +3. For a slow console, capture a Network trace in the slow browser and in Chrome on the same client; fast API timings and a fast Chrome load point at the browser, not the platform. + +If **Create** works but a new Layer 4 Ingress gets no address, check whether its host port (for example `32320`) is already used by another Layer 4 rule; a Layer 4 port cannot be shared, so no second address is allocated. That is not a browser fault. + +### Cause 3: the PC clock is wrong + +Compare the PC clock and time zone with a PC that renders the charts, correct them, and reload the monitoring view. Cluster NTP and the monitoring stack need no change. + +### Cause 4: Pod logs are empty and EXEC fails because the browser is below the 4.1 requirement + +1. Confirm that the application has standard output: + + ```bash + kubectl logs -n -c --tail=20 + ``` + +2. Check the browser version on its **About** page. On 4.1, Firefox below 140 matches this cause. +3. Upgrade the browser and reopen the page; logs and EXEC display normally. + +## Diagnostic Steps + +1. Confirm the server side is healthy: + + ```bash + kubectl get pods -n cpaas-system | grep -E 'frontend|archon|apollo' + ``` + + Expected: all Running. Then in the browser open F12 > **Network** and reload once. + +2. Map the result: + + ```text + private window or another browser works, no 5xx -> Cause 1 + Create is a dead click, platform.cfc fails, only File Transfer shown -> Cause 2 + blank login page with 证书错误 in an old browser only -> Cause 2 + console slow in one browser, fast in Chrome on the same client -> Cause 2 + charts blank on one PC only -> Cause 3 + all Pod logs empty, EXEC connection fails, kubectl logs has output -> Cause 4 + every browser fails, F12 shows 500 or 502 -> not this article + ``` + +3. Sign-in redirects and 401 from the identity provider are an authentication problem. A 403 with a clear `forbidden` message on one API is authorization, not a browser problem. diff --git a/docs/en/solutions/Container_runtime_conflicts_on_ACP_nodes_Docker_podman_docker-runc_and_overwritten_runtime_binaries.md b/docs/en/solutions/Container_runtime_conflicts_on_ACP_nodes_Docker_podman_docker-runc_and_overwritten_runtime_binaries.md new file mode 100644 index 000000000..57a54d4af --- /dev/null +++ b/docs/en/solutions/Container_runtime_conflicts_on_ACP_nodes_Docker_podman_docker-runc_and_overwritten_runtime_binaries.md @@ -0,0 +1,249 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" +--- + +# Container runtime conflicts on ACP nodes: Docker, podman, docker-runc, and overwritten runtime binaries + +## Issue + +A node that is being added, or that is already in the cluster, has a container runtime that no longer matches what the platform installed. Typical reports: + +- Docker was installed on a node (by hand or by mistake) and the node became abnormal; containerd disappeared from the node. +- After a reboot the node never becomes Ready, or Pods fail with `FailedCreatePodSandBox`. +- calico does not start after a kernel and NIC driver upgrade. +- The CRI version changed by itself after a reboot. +- A deployment or node add fails because of conflicting packages or reports a `runc` error. +- On a cluster created by TKE, a new node stays at `EnsureNodeReady` with cgroup errors in the network plugin events. + +The platform installs its own containerd (or Docker with `cri-docker` on Docker clusters), `runc`, `crictl`, `kubelet`, and `kubectl` in `/usr/bin` and expects exactly those versions. Installing Docker or podman from the OS repositories, an OS upgrade, or packages preinstalled in the machine image replace or remove them. The files involved on a containerd node are `containerd`, `containerd-shim`, `containerd-shim-runc-v1`, `containerd-shim-runc-v2`, `ctr`, `ctd-decoder`, `crictl`, `critest`, `containerd-stress`, `runc`, `kubelet`, and `kubectl` in `/usr/bin`, plus `/etc/containerd/config.toml` and the containerd unit file. Same-named copies in `/usr/local/bin` are found first on `PATH`. + +## Root Cause + +### Cause 1: Docker was installed on a containerd node + +A `docker-ce` installation on a node whose cluster uses containerd has different effects; compare the node with a healthy one before choosing a step: + +- Docker is installed and the node is abnormal, but containerd still runs and its binaries are identical to a healthy node. +- `yum install docker-ce` uninstalled containerd (for example on an ACP 4.0 node). +- After a reboot the node does not return to Ready: kubelet and the runtime services fail to start because `runc` and other binaries in `/usr/bin` are now the wrong versions. +- The OVN components on the node do not run (`ovs-ovn` produces no log), and Pods fail with `FailedCreatePodSandbox`. +- On a node whose overlay and underlay traffic share one NIC, deleting containers during the cleanup invalidated the `pr-bridge` bridge: the node lost its network and SSH and went `NotReady` until it was rebooted. Such a node can be recovered by removing it from the cluster, cleaning it up, and adding it again. + +### Cause 2: An OS, kernel, or driver upgrade overwrote runc and containerd, and podman was installed + +On an ACP 4.0 ARM node, calico does not start after the kernel and NIC driver are upgraded. The `runc` and `containerd` binaries in `/usr/bin` differ from those on a healthy node, and `podman` is installed. The OS packages involved on a Kylin node include `containers-common`, `containernetworking-plugins`, `docker-runc`, `podman`, and `container-selinux`. + +### Cause 3: dockerd changed version after a reboot on a Docker-runtime cluster + +The business cluster was not upgraded, but after a node reboot the CRI is a different version and `cri-docker` is broken: `dockerd` on the node no longer matches the version healthy nodes run (for example `20.10.27-2`). Other signs: + +- Workers are `NotReady`, with node events alternating between `NodeReady` and `NodeNotReady`; no Calico Pod is failing. +- `kubectl get pod -A | grep upgra` lists `--upgrade-cri` Pods in `kube-system` at `0/1`, `ContainerCreating` or `Pending`. +- The node's machine resource in `cpaas-system` on `global` has `cri.type: docker` and the Docker version in its spec, but `cri.version: Unknown` and `containerRuntimeVersion: docker://Unknown` in its status. + +### Cause 4: Docker, docker-runc, or other conflicting packages were preinstalled before ACP installed Kubernetes + +- A 4.2.2 deployment on Kylin Linux Advanced Server V10 fails until conflicting packages are removed from the node, for example `docker-runc`, `podman`, and `containernetworking-plugins`; a separate `/usr/local/bin/runc` reports `runc version 1.0.0-rc3`. +- A node add or cluster creation (for example on 4.3.1) reports a `runc` error because a `docker-runc` package from the machine image is installed. The check line is `/usr/bin服务: 服务项检查失败: /usr/local/bin/runc 不在/usr/bin下` ("service check failed: /usr/local/bin/runc is not under /usr/bin"); `rpm -qa | grep docker` returns `docker-runc-` while `systemctl -a | grep docker` shows only `var-lib-docker.mount`. +- ACP cannot reuse a Docker already installed on the host. ACP deploys the runtime itself, and a Docker installed beforehand is not compatible with the platform deployment. + +### Cause 5: A TKE-created cluster provisions new nodes with containerd while the cluster runs Docker + +Adding a node to a TKE workload cluster stays at `EnsureNodeReady`. The node is `NotReady`, calico and kube-proxy do not start, and their events report a cgroup incompatibility. `global` and the workload cluster use Docker, but the new node gets containerd, because the cluster object on `global` carries the annotation `cpaas.io/container-runtime-version: containerd://1.6.28-4,docker://20.10.24-2`. Removing only `containerd://1.6.28-4` from that annotation does not help. + +The new node shows `Container Runtime Version: containerd://1.6.28-4` and the event `InvalidDiskCapacity` (`invalid capacity 0 on image filesystem`), while every other node runs `docker://...`. `calico-node` stays `Init:0/5`, `kube-proxy` `ContainerCreating`, with repeated `FailedCreatePodSandbox`: + +```text +Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: expected cgroupsPath to be of format "slice:prefix:name" for systemd cgroups, got "/kubepods/burstable/pod.../..." instead: unknown +``` + +## Resolution + +### Cause 1: Docker was installed on a containerd node + +Compare the node with a healthy node of the same OS, architecture, and platform version: + +```bash +kubectl get node -o wide # CONTAINER-RUNTIME must be containerd:// +rpm -qa | grep -i -E 'docker|containerd|runc|podman' +systemctl list-unit-files | grep -i -E 'docker|containerd' +systemctl is-active containerd docker +md5sum /usr/bin/runc /usr/bin/containerd /usr/bin/kubelet +rpm -qf /usr/bin/containerd /usr/bin/runc +``` + +**containerd still running and the checksums match a healthy node.** Stop and disable Docker, remove the Docker package with the OS package manager, and check that the removal did not delete the containerd unit file or `/etc/containerd/config.toml`. `runc -v` must then match a healthy node. + +> **Warning:** Removing Docker stops every container that runs under `dockerd`. Review the dependency list the package manager prints before confirming, and stop if it includes packages the platform installed. Reinstalling the package is the only rollback. + +```bash +sudo systemctl stop docker docker.socket +sudo systemctl disable docker docker.socket +sudo yum remove docker-ce docker-ce-cli +runc -v +ls -l /etc/containerd/config.toml; systemctl cat containerd | head +``` + +**Runtime binaries differ from a healthy node, or containerd is gone.** Restore them as follows. + +> **Warning:** Overwriting the runtime binaries replaces the container runtime of the node. Stop kubelet and containerd first, because a busy file cannot be overwritten. With the default `KillMode=process`, running containers keep running under `containerd-shim`, but no container can be created, started, or stopped and no image pulled until both are started again. Apply only when the checksums differ from a healthy node of the same version and architecture; an amd64 binary does not run on an ARM node. Keep the replaced files in `/root/runtime-bin-backup` for rollback. + +```bash +# on the affected node +sudo yum remove -y docker-ce docker-ce-cli +sudo systemctl stop kubelet +sudo systemctl stop containerd +sudo mkdir -p /root/runtime-bin-backup +for i in ctd-decoder containerd-shim-runc-v2 ctr containerd-shim crictl containerd-shim-runc-v1 critest containerd-stress runc containerd kubelet kubectl; do + [ -e /usr/bin/$i ] && sudo cp -a /usr/bin/$i /root/runtime-bin-backup/ +done +``` + +```bash +# on the healthy node +for i in ctd-decoder containerd-shim-runc-v2 ctr containerd-shim crictl containerd-shim-runc-v1 critest containerd-stress runc containerd kubelet kubectl +do + scp -P /usr/bin/$i root@:/usr/bin +done +``` + +If `scp` fails because containerd still holds the files, disable containerd on the affected node, reboot it, and copy again. + +> **Warning:** With containerd disabled the node runs no Pods after the reboot, and the reboot drops every session on it. Do this only on the node being repaired, only when the copy fails on busy files, and from a console that does not depend on the node's network. Rollback: `sudo systemctl enable --now containerd`. + +```bash +sudo systemctl disable containerd +sudo reboot +# after the copy from the healthy node succeeded +sudo systemctl enable --now containerd +``` + +```bash +sudo systemctl start containerd kubelet +systemctl is-active containerd kubelet +kubectl get node +``` + +Optionally, the restore sequence can also remove the node's containers before the copy, between stopping kubelet and stopping containerd (`crictl` needs containerd running): + +> **Warning:** Deleting every container on the node stops all its Pods, including the CNI agents. On a node whose overlay and underlay share one NIC, it invalidates `pr-bridge`, the node loses its network and SSH, and only a reboot brings it back. Run it only from a console that does not depend on the node's network, and only when overlay and underlay do not share a NIC on that node; otherwise skip it. It cannot be undone; the Pods are recreated after kubelet starts again. + +```bash +sudo systemctl stop kubelet +sudo crictl ps -a -q | xargs -r sudo crictl rm -f +sudo systemctl stop containerd +``` + +If the node has lost its network (the shared-NIC `pr-bridge` case), reboot it from a console first. If the node cannot be repaired in place, remove it from the cluster, run the node cleanup script, and add it again. + +### Cause 2: An OS, kernel, or driver upgrade overwrote runc and containerd, and podman was installed + +Uninstall `podman`, then restore the binaries exactly as in Cause 1: + +```bash +rpm -qa | grep -i podman +sudo yum remove -y podman +md5sum /usr/bin/runc /usr/bin/containerd +``` + +After the binaries are back, check that calico starts: `kubectl get pods -A -o wide | grep `. + +### Cause 3: dockerd changed version after a reboot on a Docker-runtime cluster + +```bash +dockerd --version; docker version --format '{{.Server.Version}}' +# on the workload cluster +kubectl get pod -A | grep upgrade-cri +# on global: the machine resource of the node +kubectl get machine -n cpaas-system | grep +kubectl get machine -n cpaas-system -o yaml | grep -n -E 'cri|containerRuntimeVersion' +``` + +`docker://Unknown` in the machine status while the spec names the Docker version matches this cause. Copy `docker` and `dockerd` from a healthy node and restart Docker. + +> **Warning:** Restarting Docker restarts every container on the node. Apply only when the version differs from the healthy nodes. Keep the replaced binaries; rollback is copying them back and restarting Docker. + +```bash +sudo systemctl stop kubelet docker +sudo mkdir -p /root/runtime-bin-backup +sudo cp -a /usr/bin/docker /usr/bin/dockerd /root/runtime-bin-backup/ +# on the healthy node +scp /usr/bin/docker /usr/bin/dockerd root@:/usr/bin/ +# on the affected node +sudo systemctl start docker kubelet +systemctl status cri-docker --no-pager +``` + +`cri-docker` recovers after Docker restarts. + +### Cause 4: Docker, docker-runc, or other conflicting packages were preinstalled before ACP installed Kubernetes + +Check the host before the add or the deployment, and remove what the platform did not install: + +```bash +rpm -qa | grep -i docker +systemctl -a | grep -i docker +for x in docker docker-client docker-common docker-engine docker-proxy docker-runc podman-docker podman containernetworking-plugins apptainer containerd kubernetes kubernetes-master kubernetes-node kubernetes-client kubernetes-kubeadm; do + rpm -qa | grep -F "$x" +done | sort -u +ls -l /usr/local/bin/runc 2>/dev/null && /usr/local/bin/runc -v +``` + +> **Warning:** Removing packages from the host cannot be undone by the platform. Do it only on a machine that is not yet in a cluster, and only for runtime packages (`docker-runc`, a preinstalled `docker-ce`, `podman`, `containernetworking-plugins`). Make sure no other service on the host depends on them. + +```bash +sudo yum remove -y docker-runc podman containernetworking-plugins +``` + +For a preinstalled Docker, uninstall it, remove its leftovers, and let the platform deploy the runtime during the Kubernetes installation. After `docker-runc` is removed the `runc` error disappears; run the node add or the deployment again. + +### Cause 5: A TKE-created cluster provisions new nodes with containerd while the cluster runs Docker + +On `global`, read the runtime annotations of the cluster and compare with the runtime of its existing nodes: + +```bash +kubectl get node -o wide # on the workload cluster: CONTAINER-RUNTIME of the existing nodes +kubectl get cls -o yaml | grep -n -E 'runtime|container-runtime' +``` + +> **Warning:** The runtime annotations decide which runtime every later node of that cluster gets. Change them only when the existing nodes run Docker and new nodes come up with containerd. Save the object first; rollback is restoring the saved annotations. + +Set both runtime annotations to the Docker runtime that the existing nodes run (the `CONTAINER-RUNTIME` value above), then remove the new node, run `cleanup.sh` on it, and add it again: + +```bash +kubectl get cls -o yaml > /tmp/cls-backup.yaml +kubectl annotate cls --overwrite \ + cpaas.io/container-runtime-version='docker://' \ + cpaas.io/container-runtime='docker://' +``` + +## Diagnostic Steps + +On the affected node and on a healthy node of the same OS and version: + +```bash +rpm -qa | grep -i -E 'docker|containerd|runc|podman' || dpkg -l | grep -i -E 'docker|containerd|runc|podman' +for b in runc containerd ctr crictl kubelet kubectl; do printf '%s ' "$b"; md5sum /usr/bin/$b | cut -d' ' -f1; done +which -a runc containerd +systemctl is-enabled containerd kubelet docker 2>/dev/null +journalctl -u containerd -u kubelet -n 50 --no-pager +``` + +- A Docker package on a containerd node: Cause 1. Whether containerd is still active and the checksums match decides which branch. +- `podman` installed and binaries changed after a kernel or driver upgrade: Cause 2. +- A Docker-runtime cluster whose node reports `docker://Unknown` after a reboot: Cause 3. +- A host that is not yet in a cluster with `docker-runc`, `podman`, or `containernetworking-plugins` installed, or a `runc` error on add: Cause 4. +- A new node on a Docker cluster that comes up with `containerd://` and the `cgroupsPath` error: Cause 5. + +A node that is `NotReady` after a reboot only because the runtime and kubelet were not enabled at boot is simpler: start and enable them. + +```bash +sudo systemctl enable --now docker kubelet +``` diff --git a/docs/en/solutions/Control-plane_Pods_do_not_start_with_mkdir_var_log_pods_file_exists_after_containerd_directories_are_lost_on_ACP.md b/docs/en/solutions/Control-plane_Pods_do_not_start_with_mkdir_var_log_pods_file_exists_after_containerd_directories_are_lost_on_ACP.md new file mode 100644 index 000000000..e958299ae --- /dev/null +++ b/docs/en/solutions/Control-plane_Pods_do_not_start_with_mkdir_var_log_pods_file_exists_after_containerd_directories_are_lost_on_ACP.md @@ -0,0 +1,82 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.1" + - "4.2" + - "4.3" +--- + +# Control-plane Pods do not start with mkdir /var/log/pods: file exists after containerd directories are lost on ACP + +## Issue + +Platform access is abnormal and a cluster shows an abnormal status. On a master, kubelet and containerd are both `active (running)`, but `crictl ps -a` prints nothing at all: no container exists on the node. The kubelet log repeats: + +- `"Error getting node" err="node \"\" not found"` and `failed to list *v1.CSIDriver` +- `"Failed to create log directory for pod" err="mkdir /var/log/pods: file exists"` for `kube-system/etcd` and `kube-system/kube-controller-manager` +- `"CreatePodSandbox for pod failed" err="mkdir /var/log/pods: file exists"` and `"Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" ..."` + +Related articles: "etcd and kube-apiserver do not start on a master after runc or containerd is replaced on ACP" covers masters whose runtime binaries were replaced. + +## Root Cause + +### Cause 1: /var/lib/containerd lost its pods and io.containerd.grpc.v1.cri directories + +Under `/var/lib/containerd` on the master, the `pods` and `io.containerd.grpc.v1.cri` directories are missing. Without them the static control-plane Pods cannot create their sandboxes, and nothing runs on the node. How the directories are lost is not known. + +`mkdir` returns `file exists` when the path already exists as something other than a directory, for example a symbolic link whose target has disappeared. On a node where `/var/log/pods` is a link into `/var/lib/containerd`, losing that target produces exactly this error. + +Stopping kubelet and containerd, creating the two directories by hand, and starting both services again lets the containers start; the cluster status returns to Normal. + +## Resolution + +### Cause 1: /var/lib/containerd lost its pods and io.containerd.grpc.v1.cri directories + +Apply only when `crictl ps -a` is empty, kubelet logs `mkdir /var/log/pods: file exists` for the static Pods, and `ls` confirms that the two directories are missing. Compare with a healthy master of the same cluster first: + +```bash +systemctl is-active kubelet containerd +sudo crictl ps -a +sudo ls -la /var/lib/containerd +ls -ld /var/log/pods +readlink -f /var/log/pods || true +``` + +On the healthy master, note the owner and mode of the same entries: + +```bash +sudo stat -c '%U:%G %a %n' /var/lib/containerd/pods /var/lib/containerd/io.containerd.grpc.v1.cri +ls -ld /var/log/pods +``` + +> **Warning:** Stopping kubelet and containerd stops container management on this master until both are started again. Do it only when this cause is confirmed: no containers in `crictl ps -a`, the two directories missing, and the `mkdir /var/log/pods: file exists` error. Creating empty directories does not restore any container data that was in them; kubelet recreates the static Pods from their manifests. Rollback: start containerd and kubelet, and `sudo rmdir` a directory you created by mistake while it is still empty. + +```bash +sudo systemctl stop kubelet containerd +sudo mkdir -p /var/lib/containerd/pods /var/lib/containerd/io.containerd.grpc.v1.cri +# set owner and mode to the values stat printed on the healthy master, for example: +sudo chown : /var/lib/containerd/pods /var/lib/containerd/io.containerd.grpc.v1.cri +sudo chmod /var/lib/containerd/pods /var/lib/containerd/io.containerd.grpc.v1.cri +sudo systemctl start containerd +sudo systemctl start kubelet +sudo crictl ps -a +``` + +The etcd, kube-apiserver, and kube-controller-manager containers should appear in `crictl ps -a` within a few minutes, and the cluster status should return to Normal. + +## Diagnostic Steps + +```bash +systemctl is-active kubelet containerd +sudo crictl ps -a +sudo journalctl -u kubelet --since "30 min ago" | grep -E 'mkdir /var/log/pods|CreatePodSandbox' | tail -20 +sudo ls -la /var/lib/containerd +ls -ld /var/log/pods; readlink -f /var/log/pods || true +``` + +- kubelet and containerd running, `crictl ps -a` empty, `mkdir /var/log/pods: file exists`, and `/var/lib/containerd/pods` missing: this cause. +- Containers exist but etcd exits with `StartError`, or kubelet logs `Failed to delete cgroup paths`: the runtime binaries were replaced; see the related article. +- `/var/log/pods` is a link whose target exists and the directories are present: this cause does not apply. diff --git a/docs/en/solutions/CoreDNS_Host_Alias_records_do_not_take_effect_because_a_ResourcePatch_removes_data_hosts_from_cpaas-coredns_on_ACP.md b/docs/en/solutions/CoreDNS_Host_Alias_records_do_not_take_effect_because_a_ResourcePatch_removes_data_hosts_from_cpaas-coredns_on_ACP.md new file mode 100644 index 000000000..bcde85d99 --- /dev/null +++ b/docs/en/solutions/CoreDNS_Host_Alias_records_do_not_take_effect_because_a_ResourcePatch_removes_data_hosts_from_cpaas-coredns_on_ACP.md @@ -0,0 +1,176 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# CoreDNS Host Alias records do not take effect because a ResourcePatch removes data hosts from cpaas-coredns on ACP + +## Issue + +Static DNS records added in the cluster plugin **Alauda Container Platform Cluster DNS** (CoreDNS) under **Host Alias** do not take effect. Pods report `Could not resolve host` for the new name. After CoreDNS is restarted, names that used to work resolve to the wrong IP address. + +In `kube-system`, the ConfigMap `cpaas-coredns` has a `Corefile` that still contains the `hosts /etc/coredns/hosts` plugin, but the ConfigMap has no `hosts` key, and the hosts file inside the CoreDNS pods is empty or missing. + +## Root Cause + +The records entered under **Host Alias** are written to the `hosts` key of the ConfigMap `cpaas-coredns`. CoreDNS mounts that key as `/etc/coredns/hosts` and serves it with the `hosts` plugin (`hosts /etc/coredns/hosts { reload ... fallthrough }`). + +A `ResourcePatch` (`operator.alauda.io/v1alpha1`, short name `rpch`) applies JSON patches to the resources of a platform-managed release and keeps them applied. When a ResourcePatch for the `coredns` release targets `cpaas-coredns` with an operation such as: + +```yaml +spec: + jsonPatch: + - op: replace + path: /data/Corefile + value: | + ... + - op: remove + path: /data/hosts +``` + +the `hosts` key is stripped from the ConfigMap every time. This often happens when a ResourcePatch was created to customize the Corefile (for example to forward a private domain to a dedicated DNS server) and the `remove /data/hosts` operation was included with it. + +As a result: + +- new Host Alias entries never reach CoreDNS; +- after a restart, CoreDNS starts with no static records, and because of `fallthrough` the names are resolved by the upstream DNS instead, which can return a different, stale address. + +### Other reasons a static record does not reach CoreDNS + +The same symptom (a name added for CoreDNS never resolves in Pods) has two more causes, even when no ResourcePatch removes `/data/hosts`. + +**A direct edit of `cpaas-coredns` is converted into a ResourcePatch.** `cpaas-coredns` is a chart-rendered platform resource. The `sentry` component watches UPDATE and DELETE of objects that carry the `helm.sh/chart-name` label, through the `ValidatingWebhookConfiguration` named `sentry` (its webhook `vsentry.kb.io` calls the sentry Service `/validate`). A `kubectl edit` or `kubectl patch` of the ConfigMap returns: + +```text +admission webhook "vsentry.kb.io" denied the request: The operation is denied, but ResourcePatch is created. Please wait sentry to apply the ResourcePatch. +``` + +The edit is not rejected for good: sentry applies the ResourcePatch it created. Whichever copy sentry applied last is the one CoreDNS serves, so a ResourcePatch can keep an older hosts list after the plugin page was saved, or the opposite, and the **Host Alias** list and the ConfigMap disagree. + +**A hosts entry added by hand is in the wrong format or is overridden.** The Corefile loads the `hosts` plugin from a file (`hosts /etc/coredns/hosts`), and that file is the `hosts` data key of the same ConfigMap. A mapping written as a line inside the Corefile `hosts` block, or as a single-line data field `hosts: `, is not what the `hosts` key needs; it must be a multiline block (`hosts: |` followed by one ` ` line per entry). Even with the format corrected and CoreDNS restarted, a hand edit can still be overridden by the plugin-managed copy (the same mechanism as above); the entry then takes effect only when it is entered under **Host Alias**. + +**Host Alias** is also the way to publish additional names to Pods when the node `/etc/resolv.conf` cannot take more nameservers. + +## Resolution + +1. Back up the ConfigMap and the ResourcePatch: + + ```bash + kubectl -n kube-system get cm cpaas-coredns -o yaml > cpaas-coredns.before.yaml + kubectl get rpch -A + kubectl get rpch -o yaml > coredns-resourcepatch.before.yaml + ``` + + Add `-n ` to the last command if `kubectl get rpch -A` lists the patch in a namespace. + +2. Decide whether the other changes in the patch are still needed. Read its `spec.jsonPatch`: besides `remove /data/hosts` it may replace the Corefile, for example to add a forwarding block for a private domain. + +3. Remove the patch that strips the hosts data. + + > **Warning:** Deleting the ResourcePatch also drops every other customization it carries, such as an extra forwarding block in the Corefile; names that depend on it stop resolving. Do this only when `kubectl get rpch -o yaml` shows a `remove` operation on `/data/hosts` of `cpaas-coredns`, and only after the backups in step 1. To roll back, run `kubectl apply -f coredns-resourcepatch.before.yaml`. + + - If none of its other operations are needed any more (for example, the private DNS server it forwarded to is no longer used), delete it: + + ```bash + kubectl delete rpch + ``` + + - If the Corefile customization is still needed, edit the ResourcePatch instead and remove only the `op: remove` / `path: /data/hosts` entry from `spec.jsonPatch`: + + ```bash + kubectl edit rpch + ``` + +4. Check that the hosts data is back in the ConfigMap. If it is still empty, open **Administrator** > **Marketplace** > **Cluster Plugins**, update **Alauda Container Platform Cluster DNS**, re-enter the IP addresses and domains under **Host Alias** and click **Update**. + + ```bash + kubectl -n kube-system get cm cpaas-coredns -o jsonpath='{.data.hosts}{"\n"}' + ``` + +5. Verify from a workload Pod. The `hosts` plugin reloads the file automatically once the updated ConfigMap reaches the CoreDNS pods (kubelet syncs mounted ConfigMaps periodically, so allow a minute or two): + + ```bash + kubectl -n exec -- nslookup + ``` + +Do not add a `remove /data/hosts` operation to a ResourcePatch for CoreDNS again. Keep static records in **Host Alias** and limit the ResourcePatch to the Corefile changes that cannot be made in the plugin. + +### If a direct edit was denied by vsentry.kb.io + +The edit was converted into a ResourcePatch for sentry to apply; modification is not prohibited. Do not send a second patch. Wait, then check that the ResourcePatch was applied and that the ConfigMap contains the change: + +```bash +kubectl get validatingwebhookconfiguration sentry -o yaml +kubectl get rpch -A +kubectl get rpch -o yaml +kubectl -n kube-system get cm cpaas-coredns -o yaml +``` + +When the **Host Alias** list and the ConfigMap disagree, pick one writer and keep to it: + +- **ConfigMap edits** (when other custom lines must stay in the ConfigMap): keep editing `cpaas-coredns` and let sentry apply the ResourcePatch. +- **Host Alias only**: back up the ConfigMap and the ResourcePatch as in step 1, delete the ResourcePatch that overrides the plugin, and enter the IP and the domains (separated by spaces) under **Host Alias**. + + > **Warning:** Deleting the ResourcePatch removes every ConfigMap customization that is not on the plugin page, including a hand-written `forward` block. Confirm with the application owners which lines must be re-entered first. To roll back, apply the saved ResourcePatch and ConfigMap. + +### If a hand-added hosts entry is ignored after a restart + +1. Enter the record under **Host Alias** in **Alauda Container Platform Cluster DNS** (fields **IP** and **Domains**). On ACP 4.x this setting is reloaded without a CoreDNS restart. Then query again from the workload Pod: + + ```bash + kubectl -n exec -- nslookup + ``` + +2. If the record must stay in the ConfigMap instead, keep it in one place only (not both the Corefile and the `hosts` key), and write the `hosts` key as a multiline block: + + ```yaml + data: + hosts: | + 192.0.2.10 app.example.com + ``` + + > **Warning:** Editing `cpaas-coredns` changes DNS for every Pod in the cluster, and a malformed Corefile or hosts file can stop CoreDNS from loading. Back up the ConfigMap first (step 1). Sentry turns the write into a ResourcePatch, as described above. To roll back, run `kubectl apply -f cpaas-coredns.before.yaml`. + + If the ConfigMap edit still does not show up in answers, move the entry to **Host Alias**. + +## Diagnostic Steps + +1. Check whether the ConfigMap carries the static records. A `Corefile` that references `/etc/coredns/hosts` but no `hosts` key matches this issue: + + ```bash + kubectl -n kube-system get cm cpaas-coredns -o yaml + ``` + +2. List the ResourcePatches and look for one that targets `cpaas-coredns` with `remove` on `/data/hosts`: + + ```bash + kubectl get rpch -A + kubectl get rpch -o yaml | grep -n -B2 -A2 '/data/hosts' + ``` + +3. Check the file CoreDNS actually reads: + + ```bash + kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide + kubectl -n kube-system exec -- cat /etc/coredns/hosts + ``` + + An empty result or `No such file or directory` confirms that the records are not loaded. If the CoreDNS image has no `cat`, compare the ConfigMap instead (step 1). + +4. Rule out NodeLocal DNSCache. If the Pod's `/etc/resolv.conf` points at the NodeLocal address (for example `169.254.20.10`) and a direct query to the CoreDNS Service IP succeeds while the default one fails, the records are in CoreDNS and the cache is not forwarding that name, which is a different problem: + + ```bash + kubectl -n exec -- cat /etc/resolv.conf + kubectl -n exec -- nslookup + ``` + +5. Tell the causes apart: + + - `cpaas-coredns` has no `hosts` key and a ResourcePatch on it has `op: remove` on `/data/hosts`: the ResourcePatch removes the hosts data (this article's main cause). + - A direct edit returned `admission webhook "vsentry.kb.io" denied the request`, or the **Host Alias** list and the ConfigMap `hosts` entries differ: the edit was converted into a ResourcePatch. + - A hosts line was added to `cpaas-coredns` by hand and a CoreDNS restart did not help: check the format of the `hosts` key and move the entry to **Host Alias**. diff --git a/docs/en/solutions/CoreDNS_fails_after_a_node_restart_or_lookups_time_out_when_a_CoreDNS_node_is_shut_down_on_ACP.md b/docs/en/solutions/CoreDNS_fails_after_a_node_restart_or_lookups_time_out_when_a_CoreDNS_node_is_shut_down_on_ACP.md new file mode 100644 index 000000000..4287e60c8 --- /dev/null +++ b/docs/en/solutions/CoreDNS_fails_after_a_node_restart_or_lookups_time_out_when_a_CoreDNS_node_is_shut_down_on_ACP.md @@ -0,0 +1,118 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# CoreDNS fails after a node restart or lookups time out when a CoreDNS node is shut down on ACP + +Related articles: [Intermittent DNS failures in Pods from a new node, NIC flapping, the KubeDNS alert or CoreDNS upstream timeouts on ACP](Intermittent_DNS_failures_in_Pods_from_a_new_node_NIC_flapping_the_KubeDNS_alert_or_CoreDNS_upstream_timeouts_on_ACP.md). + +## Issue + +Cluster DNS is down or degraded after a change to the nodes, and the cause is outside the Corefile. Typical reports: + +- After the nodes restarted, CoreDNS and other basic components in the global cluster are all abnormal. +- A node that ran CoreDNS was shut down. The CoreDNS Pod restarted and application lookups timed out for a while. The question is why, and which platform components a node restart can affect. +- CoreDNS was scaled from 2 to 4 replicas with `kubectl scale`, and some time later it is back to 2. + +CoreDNS on ACP 4.x is the `coredns` Deployment in `kube-system` (Pod label `k8s-app=kube-dns`), configured by the ConfigMap `cpaas-coredns`. The CoreDNS Pods use the `/etc/resolv.conf` of the node they run on as their upstream list, and `9153` is their metrics port. + +## Root Cause + +### Cause 1: the node /etc/resolv.conf is empty after a restart, so CoreDNS has no upstream + +The default Corefile forwards with `forward . /etc/resolv.conf`. When `/etc/resolv.conf` on the node exists but is empty (no `nameserver` line), CoreDNS has no upstream and fails to start; CoreDNS normally logs `no nameservers found` in this case. Components that depend on DNS fail with it. + +The file can be reset to empty when the node restarts by tooling outside the platform, for example on the IaaS side (cloud-init, DHCP, or image tooling). Restoring the file contents returns cluster DNS to normal. + +### Cause 2: a node running CoreDNS is shut down and lookups time out until its endpoint is removed + +When a node goes down abruptly, kubelet cannot report that it is leaving. The endpoint controller removes that node's Pods from the `kube-dns` endpoints only after the node is marked `NotReady`, so there is a delay. During it, DNS caching and connection reuse in the clients keep sending queries to the offline replica until the cache expires or the connection closes, and lookups time out. + +Platform components are deliberately not pinned to specific nodes or to the masters: a pinned Pod cannot fail over when its node fails, and master resources differ between clusters, so heavy components there could destabilize the masters. The components whose node going down affects the whole cluster are: + +| Release | Component (Deployment) | Effect when its node goes down | +| --- | --- | --- | +| 3.8 | `cpaas-system/lcn-ccm-alb` | business traffic entering and leaving the cluster | +| 3.16 | the `*-alb`, `*-clb`, `*-lb` Deployments in `cpaas-system` | business traffic entering and leaving the cluster | +| 3.8, 3.16 | `cpaas-system/coredns` | in-cluster name resolution | + +On 4.x CoreDNS is `kube-system/coredns`. `raven` and `calico-kube-controllers` affect only changes such as Pod creation and deletion; running workloads are not affected. When every business team uses its own ALB on hosts it manages, CoreDNS is effectively the only shared component to plan around. + +### Cause 3: CoreDNS replica changes made with kubectl are reverted + +The CoreDNS Deployment is managed by an upper-layer platform controller, which overwrites direct edits to the Deployment. A replica count set with `kubectl scale deploy` therefore returns to the earlier value after a while. A replica count changed through the platform UI persists. + +## Resolution + +### Cause 1: the node /etc/resolv.conf is empty after a restart, so CoreDNS has no upstream + +Only when `/etc/resolv.conf` on the node running the CoreDNS Pod is empty (no `nameserver` line). + +1. Restore the file contents, from your own backup or from a healthy node of the same cluster with the same resolvers. +2. Recreate the CoreDNS Pods. A Pod's resolver file is written when the Pod is created, so a container restart alone does not pick up the restored file. + +> **Warning:** Writing the wrong resolvers into `/etc/resolv.conf` breaks external name resolution for the node and for CoreDNS, and recreating the CoreDNS Pods briefly delays cluster lookups. Make sure the nameservers you restore are the ones the cluster used before (compare with a healthy node). To roll back, restore the previous file and recreate the Pods again. + +```bash +cat /etc/resolv.conf +cp -a /etc/resolv.conf /root/resolv.conf.empty +scp :/etc/resolv.conf /etc/resolv.conf +kubectl -n kube-system delete pod -l k8s-app=kube-dns +kubectl -n kube-system get pod -l k8s-app=kube-dns -o wide -w +``` + +3. Ask the owner of the node image or IaaS why the file is emptied on restart; otherwise the next reboot repeats it. + +### Cause 2: a node running CoreDNS is shut down and lookups time out until its endpoint is removed + +No configuration change fixes a past event; the timeouts end once the endpoint is removed or the node returns. For planned work: + +1. Use a change process: report node restarts or decommissioning to the cluster administrator and announce them to the other business teams on the cluster beforehand. Keep the platform components unpinned so they can fail over. +2. Drain the node before a planned shutdown. Eviction removes the CoreDNS Pod from the endpoints before the node goes away, instead of waiting for `NotReady`. + +> **Warning:** Draining evicts every non-DaemonSet Pod on the node, and Pods with no other replica are unavailable until they start elsewhere. Make sure at least one other CoreDNS replica is Ready on another node. To roll back, run `kubectl uncordon `; evicted Pods are not moved back automatically. + +```bash +kubectl -n kube-system get pod -l k8s-app=kube-dns -o wide +kubectl drain --ignore-daemonsets --delete-emptydir-data +``` + +### Cause 3: CoreDNS replica changes made with kubectl are reverted + +Set the replica count in the platform UI rather than with `kubectl scale`, then confirm that the count holds: + +```bash +kubectl -n kube-system get deploy coredns -o jsonpath='{.spec.replicas}{"\n"}' +``` + +## Diagnostic Steps + +1. Check CoreDNS state and the previous container log: + + ```bash + kubectl -n kube-system get pod -l k8s-app=kube-dns -o wide + kubectl -n kube-system describe pod -l k8s-app=kube-dns | grep -iE 'Reason|Restart|Ready' + kubectl -n kube-system logs -l k8s-app=kube-dns --previous --tail=50 + ``` + +2. On the node of a failing CoreDNS Pod, read the resolver file. No `nameserver` line is Cause 1: + + ```bash + cat /etc/resolv.conf + grep -c '^nameserver' /etc/resolv.conf + ``` + +3. Compare the time of the lookup failures with node events. A CoreDNS restart and lookup timeouts right after its node was shut down is Cause 2: + + ```bash + kubectl get events -A --field-selector reason=NodeNotReady --sort-by=.lastTimestamp + kubectl -n kube-system get endpointslice -l kubernetes.io/service-name=kube-dns -o wide + ``` + +4. A replica count that returns to the old value after `kubectl scale` is Cause 3. diff --git a/docs/en/solutions/Create_and_manage_alert_policies_and_notification_routing_on_ACP.md b/docs/en/solutions/Create_and_manage_alert_policies_and_notification_routing_on_ACP.md new file mode 100644 index 000000000..1ac1abd7e --- /dev/null +++ b/docs/en/solutions/Create_and_manage_alert_policies_and_notification_routing_on_ACP.md @@ -0,0 +1,208 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.16" + - "4.0" + - "4.3" +--- + +# Create and manage alert policies and notification routing on ACP + +## Issue + +Common questions when setting up alerting on Alauda Container Platform: + +- How do I get an alert to the operations staff when a resource is abnormal or reaches a threshold? +- Can one alert policy apply to all clusters? +- How do I create alert policies for many namespaces in batch? +- Every alert rule uses `cpaas-admin-notification`; can I change the notification policy of all of them at once? +- How do I raise a threshold, for example make the node zombie-process alert fire only above 7? +- How do I alert on my own metric, such as JVM heap usage above 92% for 3 minutes? +- How often are alert rules evaluated? +- Is there a disk usage alert for master nodes? +- How do I list the existing alert policies and their notification channels, or get a table of the built-in alerts? +- Can a failed backup raise an alert? + +A rule that does not fire or is rejected is covered in [Alert rules do not fire, are rejected, or cannot be edited on ACP](Alert_rules_do_not_fire_are_rejected_or_cannot_be_edited_on_ACP.md). A notification that is configured and does not arrive is covered in [Alert notifications are not delivered because courier-api or apollo cannot reach the receiver on ACP](Alert_notifications_are_not_delivered_because_courier-api_or_apollo_cannot_reach_the_receiver_on_ACP.md). + +## Root Cause + +### Cause 1: notifying people needs a template, a notification policy, and an alert policy that references it + +Alerts reach people only when three pieces exist and are connected: a notification template (what the message says), a notification policy (who receives it and by which method, through notification contact groups), and an alert policy whose **Notification Policy** field references that notification policy. Built-in policies already reference one (for example `cpaas-admin-notification`); a new policy needs one selected. + +### Cause 2: alert policies are per cluster + +An alert policy belongs to one cluster, so a single policy cannot apply to all clusters. To avoid repeating the setup, create an alert template once and create the policy from it in each cluster. + +### Cause 3: there is no batch creation of alert policies + +The platform has no built-in function to create alert policies in batch, for example one per namespace. The options are creating each policy from an alert template, or creating the underlying `PrometheusRule` resources directly (creating an alert policy creates a `PrometheusRule`), with the target fields replaced in each copy. + +### Cause 4: the notification policy on alert policies cannot be changed in bulk + +There is no bulk change of the notification policy bound to alert policies. The default rules come with `cpaas-admin-notification` by design, and each policy has to be changed individually. + +### Cause 5: a threshold is part of the rule's trigger condition + +A threshold, for example of the node zombie-process rule, is changed by editing the rule's **Trigger Condition** (operator, value, and duration) inside the alert policy. The built-in node rule "Number of zombie processes on the node" in `cpaas-node-rules` can be changed to `> 7` for 1 minute this way, and the rule stays enabled. + +### Cause 6: a custom metric alert is a custom alert rule in a resource alert policy + +A custom metric such as JVM heap usage is alerted on with a rule of type **Custom Alert** (`自定义告警`) inside a resource alert policy (`资源类告警`) whose resource type is compute component (`计算组件`). On ACP 3.12 the rule takes a metric name (up to 64 characters), a PromQL expression (up to 512 characters), an optional unit and legend, a trigger condition (operator, threshold with up to 17 integer digits and 4 decimal places, and duration), and a level. After several compute components are selected, some built-in metric alerts are not supported; configuring per component supports all alert metrics. + +### Cause 7: metrics are scraped every 60 seconds and rules evaluated every 30 seconds by default + +By default metrics are scraped every 60 seconds and alert rules are evaluated every 30 seconds. The rule's duration (for example 1 minute) is counted on top of those. + +### Cause 8: node disk usage is covered by the built-in node policy + +The default node policy includes the rule `节点磁盘空间使用率 > 80% 且持续 1 分钟` (node disk space usage above 80% for 1 minute), type metric alert, level critical, enabled by default, together with the node CPU, memory, health, NodeExporter, and OOM rules. The platform inspection also has a node disk usage item, but it reflects only the moment the inspection runs. + +### Cause 9: alert policies are PrometheusRules, notification policies are notifications.ait.alauda.io + +Alert policies are stored as `prometheusrules` in each cluster, and each references its notification policies in the annotation `alert.cpaas.io/notifications`. Notification policies are `notifications.ait.alauda.io` resources on the `global` cluster; their YAML lists the bound contact groups and methods. The product documentation has no ready-made table of built-in alerts, but one can be generated from the rule labels. + +### Cause 10: backup task failures cannot be alerted on + +Application backup results are not exposed as Prometheus metrics (there is no `ServiceMonitor` for them), so no alert rule can watch them. etcd backup monitoring is different and can be configured through Prometheus. + +## Resolution + +### Cause 1: notifying people needs a template, a notification policy, and an alert policy that references it + +1. **Platform Management** > **Operations Center** > **Notification**. On **Notification Templates**, create a template of type **Alert Message**, or reuse one. +2. On **Notification Contact Groups**, create the group of receivers. +3. On **Notification Policies**, click **Create Notification Policy** and select the contact group, the methods, and the template. +4. **Operations Center** > **Alerts** > **Alert Policies**, select the cluster, and click **Create Alert Policy**, or open an existing policy with **Actions** > **Update**. +5. Under **Policy Configuration**, select the notification policy in **Notification Policy**, set **Alert Sending Interval**, and save. + +### Cause 2: alert policies are per cluster + +1. **Operations Center** > **Alerts** > **Alert Templates** > **Create Alert Template**. Choose the resource type (for example cluster) and add the rules once. +2. In each target cluster under **Alert Policies**, open the dropdown next to **Create Alert Policy** and choose **Create Alert Policy from Template**. +3. Select the template and fill in the name and display name. For a cluster template the resource is set to the current cluster. Click **Create**. + +### Cause 3: there is no batch creation of alert policies + +For a few targets, use templates (Cause 2). For many namespaces, create one policy in the console, export its `PrometheusRule`, and generate copies: + +```bash +kubectl get prometheusrules.monitoring.coreos.com -n -o yaml > /tmp/policy-template.yaml +grep -n -E 'name:|alert.cpaas.io/namespace|namespace=~|alert_involved_object_namespace|resourceVersion|uid:|creationTimestamp' /tmp/policy-template.yaml +``` + +The namespace appears in the annotation `alert.cpaas.io/namespace`, in the `namespace=~"..."` matchers of every `expr`, and in the `alert_involved_object_namespace` label of every rule. In each copy, remove `uid`, `resourceVersion`, `creationTimestamp`, and `managedFields`, give it a unique `metadata.name`, and replace the namespace in all three places. Validate, then apply: + +```bash +kubectl apply --dry-run=server -f /tmp/policy-.yaml +kubectl apply -f /tmp/policy-.yaml +``` + +Open one generated policy in the console and check that its rules and notification policy display correctly before creating the rest. + +### Cause 4: the notification policy on alert policies cannot be changed in bulk + +Change it on each alert policy: **Alert Policies** > **Actions** > **Update** > **Notification Policy**. To see which policies reference a notification policy first: + +```bash +kubectl get prometheusrules.monitoring.coreos.com -A -o yaml | grep -n 'alert.cpaas.io/notifications' +``` + +### Cause 5: a threshold is part of the rule's trigger condition + +1. **Platform Management**, select the cluster, **Operations Center** > **Alerts** > **Alert Policies**. +2. Open the policy (for example `cpaas-node-rules`) and click edit on the rule. +3. In **Edit Alert Rule**, change the value (and, if needed, the operator and duration) in **Trigger Condition**, then save the rule and the policy. + +Built-in rules can be restored from the platform's own copy, for example by an upgrade, so re-check custom thresholds on built-in policies after an upgrade. + +### Cause 6: a custom metric alert is a custom alert rule in a resource alert policy + +1. **Alert Policies** > **Create Alert Policy**. Alert type resource alert, resource type compute component, component type Deployment. Optionally choose namespaces and resource names; with none selected, any matches, up to 30 can be selected, and new namespaces are included automatically. +2. Under the alert rules, click **Add Alert Rule** and set the alert type to **Custom Alert**. +3. Enter the metric name and the expression. Scope the expression to the workload and test it in the metrics UI first. In a project or namespace view the expression must carry the namespace label, or the preview stays empty: + + ```promql + max by (pod) (jvm_memory_used_bytes{area="heap",namespace="",pod=~"-.*"} / jvm_memory_max_bytes{area="heap",namespace="",pod=~"-.*"}) + ``` + +4. Set the trigger condition to `>=` `0.92` (use `>` if `>=` is not offered) with a duration of 3 minutes, and choose the level. +5. Bind the notification policy and save. + +### Cause 7: metrics are scraped every 60 seconds and rules evaluated every 30 seconds by default + +A condition becomes visible after the next scrape (up to 60 seconds), is noticed at the next evaluation (up to 30 seconds), and fires after the rule's duration has also passed. Read the live values before relying on them: + +```bash +kubectl get vmagent -A -o jsonpath='{range .items[*]}{.metadata.name}{" scrapeInterval="}{.spec.scrapeInterval}{"\n"}{end}' +kubectl get vmalert -A -o jsonpath='{range .items[*]}{.metadata.name}{" evaluationInterval="}{.spec.evaluationInterval}{"\n"}{end}' +``` + +On a Prometheus cluster, read `spec.scrapeInterval` and `spec.evaluationInterval` from `kubectl get prometheus -A -o yaml`. An empty value means the component default. + +### Cause 8: node disk usage is covered by the built-in node policy + +Confirm that the disk rule is present and enabled: + +```bash +kubectl get prometheusrules.monitoring.coreos.com -A | grep node-rules +kubectl get prometheusrules.monitoring.coreos.com -n cpaas-system cpaas-node-rules -o yaml | grep -n -A3 node.disk.space.utilization +``` + +For another threshold or scope, create a custom policy: **Create Alert Policy**, alert type resource alert, resource type node, resource name all nodes, then add a rule with the disk metric and the threshold. If a root disk above 90% is neither alerted nor reported by inspection, contact technical support. + +### Cause 9: alert policies are PrometheusRules, notification policies are notifications.ait.alauda.io + +On each cluster, list the alert policies and their notification policies: + +```bash +kubectl get prometheusrules -A +kubectl get prometheusrules -n -o yaml | grep alert.cpaas.io/notifications +``` + +On the `global` cluster, list the notification policies and read their contact groups and methods: + +```bash +kubectl get notifications.ait.alauda.io -A +kubectl get notifications.ait.alauda.io -n -o yaml | grep -i group -C 5 +kubectl get notifications.ait.alauda.io -n -o yaml | grep -i methods -C 5 +``` + +For a table of built-in alerts (policy, alert, indicator, severity, resource kind, expression; a missing label prints an empty column): + +```bash +kubectl get prometheusrules -n cpaas-system -o json | jq -r ' + .items[] | .metadata.name as $p | .spec.groups[]?.rules[]? + | select(.alert != null) + | [$p, .alert, (.labels.alert_indicator // ""), (.labels.severity // ""), (.labels.alert_involved_object_kind // ""), (.expr | gsub("\n"; " "))] + | @tsv' +``` + +Run it on each cluster type you need (`global` and a business cluster); the built-in set differs. + +### Cause 10: backup task failures cannot be alerted on + +No alert can be configured for a failed application backup. Check results directly from the `status` of the backup objects, or look for the backup files in the target directory: + +```bash +kubectl get backups -A +``` + +If `backups` is ambiguous on the cluster, list the kinds with `kubectl api-resources | grep -i backup` and use the full name. + +## Diagnostic Steps + +1. The alert fires in alert history and nobody is told: the alert policy has no notification policy, or the notification policy has no contact group or template (Cause 1). + +2. One policy for all clusters, batch creation, or a bulk change of the notification policy: not available as a single action (Causes 2, 3, 4). + +3. A different threshold: Cause 5. A metric the platform does not ship: Cause 6. + +4. "The alert came late": add the scrape interval, the evaluation interval, and the rule's duration (Cause 7) before suspecting the evaluator. + +5. "Is there an alert for X?": search the existing policies first (Cause 9). Node disk usage is built in (Cause 8). Backup failures are not alertable (Cause 10). diff --git a/docs/en/solutions/Creating_a_custom_application_from_YAML_fails_with_applications.app.k8s.io_already_exists_on_ACP.md b/docs/en/solutions/Creating_a_custom_application_from_YAML_fails_with_applications.app.k8s.io_already_exists_on_ACP.md new file mode 100644 index 000000000..1adb9f5e3 --- /dev/null +++ b/docs/en/solutions/Creating_a_custom_application_from_YAML_fails_with_applications.app.k8s.io_already_exists_on_ACP.md @@ -0,0 +1,97 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Creating a custom application from YAML fails with applications.app.k8s.io already exists on ACP + +## Issue + +Creating a custom application from the YAML page of the console fails with a **Resource already exists** error. Renaming the application and creating it again fails the same way, while applying the same resources with `kubectl` works. + +The error details show: + +```json +{ + "StatusCode": 409, + "reason": "AlreadyExists", + "message": "applications.app.k8s.io \"\" already exists", + "details": { "group": "app.k8s.io", "kind": "applications" } +} +``` + +An `Application` with that name does exist afterwards. It was created by the same click, and its YAML shows it never assembled: + +```yaml +spec: + assemblyPhase: Failed + componentKinds: + - group: app.k8s.io + kind: Application + - group: apps + kind: StatefulSet +status: + conditions: + - type: Error + reason: Create components failed + message: applications.app.k8s.io "" already exists + totalComponents: 0 +``` + +## Root Cause + +### Cause 1: a full `Application` object was pasted into the custom application YAML + +The YAML field of the custom application page expects the workload resources only, for example a StatefulSet. The console wraps that input in its own `app.k8s.io/v1beta1` `Application` object with the name entered on the page. + +When the pasted YAML is itself an `Application` (`apiVersion: app.k8s.io/v1beta1`, `kind: Application`), the request contains an extra Application layer. The console first creates its own `Application`, then tries to create the pasted `Application` as one of its components. That component has the same name as the Application just created, so the API server returns `409 AlreadyExists`. The outer Application ends in `assemblyPhase: Failed` with `Create components failed`, and no workload is created. + +Renaming does not help because the console Application and the pasted Application get the same name again. + +## Resolution + +### Cause 1 + +1. Remove the failed Application left behind by the earlier attempts, so the name can be used again. + + > **Warning:** Deleting an `Application` can also delete resources that belong to it. Do this only when the Application shows `assemblyPhase: Failed`, the condition `Create components failed`, and `totalComponents: 0`, which means no component was created. If the Application has components, do not delete it; check what it owns first. There is nothing to roll back for a failed Application with no components. + + ```bash + kubectl -n get application -o yaml + kubectl -n delete application + ``` + +2. Create the custom application again and paste only the workload YAML, for example the StatefulSet. Do not include an `apiVersion: app.k8s.io/v1beta1` / `kind: Application` object or its `componentKinds`, `assemblyPhase` or `status` fields. + + ```yaml + apiVersion: apps/v1 + kind: StatefulSet + metadata: + name: + spec: + ... + ``` + +3. Check that the new Application assembles and lists its workload: + + ```bash + kubectl -n get application -o jsonpath='{.spec.assemblyPhase}{"\n"}' + ``` + +## Diagnostic Steps + +1. Look at the YAML that was pasted into the page. If it starts with `apiVersion: app.k8s.io/v1beta1` and `kind: Application`, this is the cause. + +2. Find the Application that the failed attempt created. Use the full resource name: the short name `app` is not registered and returns `error: the server doesn't have a resource type "app"`. + + ```bash + kubectl get application --all-namespaces | grep + kubectl -n get application -o yaml + ``` + + Expected for this cause: `componentKinds` contains `group: app.k8s.io, kind: Application`, the condition reason is `Create components failed` with the message `applications.app.k8s.io "" already exists`, and `totalComponents: 0`. + +3. If the pasted YAML contains no `Application` object and the error still appears, rule out a duplicate submission: open the browser developer tools, select **Network**, enable **Preserve log**, and click create. Two POST requests with the same URL and body, the first returning `201` and the second `409 Conflict`, are a duplicate submission, which is a different problem from this cause. diff --git a/docs/en/solutions/Cross_node_Pod_throughput_far_below_node_bandwidth_because_TX_offload_is_off_on_Kube-OVN_genev_sys_6081_on_ACP.md b/docs/en/solutions/Cross_node_Pod_throughput_far_below_node_bandwidth_because_TX_offload_is_off_on_Kube-OVN_genev_sys_6081_on_ACP.md new file mode 100644 index 000000000..3b21ff9b6 --- /dev/null +++ b/docs/en/solutions/Cross_node_Pod_throughput_far_below_node_bandwidth_because_TX_offload_is_off_on_Kube-OVN_genev_sys_6081_on_ACP.md @@ -0,0 +1,100 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Cross-node Pod throughput far below node bandwidth because TX offload is off on Kube-OVN genev_sys_6081 on ACP + +## Issue + +On a cluster that uses the Kube-OVN Geneve overlay, an `iperf` test between two nodes reaches close to the line rate of the NIC, but the same test between two Pods on those nodes reaches much less. For example, on nodes with 25 Gbit/s NICs (ARM Kunpeng servers): + +- node to node, across nodes: about 23.5 Gbit/s +- Pod to Pod, across nodes: about 9.1 Gbit/s, a loss of more than 60% + +Connectivity itself is fine: cross-node Pod traffic works, it is just slow. On the nodes, `ethtool -k genev_sys_6081` shows transmit offload switched off on the Geneve tunnel interface, with TCP segmentation offload requested but not active: + +```text +tx-checksumming: off + tx-checksum-ip-generic: off +tcp-segmentation-offload: off + tx-tcp-segmentation: off [requested on] + tx-tcp-ecn-segmentation: off [requested on] + tx-tcp-mangleid-segmentation: off [requested on] + tx-tcp6-segmentation: off [requested on] +generic-segmentation-offload: on +generic-receive-offload: on +``` + +If cross-node Pod traffic fails rather than being slow, this article does not apply; see [Cross-node Pod traffic fails on the Kube-OVN Geneve overlay because of UDP 6081, TX offload or MTU on ACP](Cross_node_Pod_traffic_fails_on_the_Kube-OVN_Geneve_overlay_because_of_UDP_6081_TX_offload_or_MTU_on_ACP.md), where turning offload **off** is the fix. + +## Root Cause + +### Cause 1: a Kube-OVN defect leaves TX offload disabled on genev_sys_6081 + +Pod traffic between nodes is encapsulated in Geneve and leaves the node through the OVS tunnel interface `genev_sys_6081`. A Kube-OVN defect leaves TX checksum offload disabled on that interface. Because TCP segmentation offload depends on TX checksum offload, the kernel also turns TSO off, which is why the TSO features read `off [requested on]`. Every encapsulated packet then has to be segmented and checksummed in software, and cross-node Pod throughput falls far below what the node itself reaches over the same NIC. + +The setting does not persist. TX offload is disabled on the interface again after the node reboots or after the `kube-ovn-cni` Pod on that node restarts. The defect is fixed in a later OVN version; until the cluster runs a fixed version, the workaround below has to be reapplied after every such event. + +## Resolution + +### Cause 1: a Kube-OVN defect leaves TX offload disabled on genev_sys_6081 + +1. On each node, confirm the symptom before changing anything. Continue only if `tx-checksumming` is `off` and the TSO features show `off [requested on]`: + + ```bash + ethtool -k genev_sys_6081 | grep -E '^tx-checksumming|tx-checksum-ip-generic|^tcp-segmentation-offload|tx-tcp' + ``` + +2. Turn TX offload on for the Geneve tunnel interface, on every node of the cluster (the physical nodes that carry Pod traffic): + + > **Warning:** This changes the offload settings of the tunnel interface that carries all cross-node Pod traffic of the node. Apply it only on nodes where `ethtool -k genev_sys_6081` shows the signature above, and test one node first. Do not apply it on a node where transmit offload was switched off on purpose to work around a NIC checksum problem (see the related article in the Issue section). If cross-node Pod traffic from the node starts failing or checksum errors rise after the change, roll back with `ethtool -K genev_sys_6081 tx off`. + + ```bash + ethtool -K genev_sys_6081 tx on + ``` + +3. Verify the result on the node. Expected: `tx-checksumming: on` and `tcp-segmentation-offload: on`: + + ```bash + ethtool -k genev_sys_6081 | grep -E '^tx-checksumming|^tcp-segmentation-offload' + ``` + +4. Repeat the cross-node Pod `iperf` test (see Diagnostic Steps). Pod-to-Pod throughput should now be close to node-to-node throughput. + +5. Reapply step 2 after every reboot of a node and after every restart of the `kube-ovn-cni` Pod on a node, until the cluster runs an OVN version that contains the fix. After such an event, repeat step 1 to check whether the interface has fallen back to `tx-checksumming: off`. + +## Diagnostic Steps + +1. Measure node-to-node throughput across two nodes, then Pod-to-Pod throughput between Pods scheduled on the same two nodes. Run `iperf3 -s` on the server side and the client against it: + + ```bash + # on node B, or in a Pod on node B + iperf3 -s + # on node A, or in a Pod on node A + iperf3 -c -t 30 -P 4 + ``` + + Expected for this article: the node-to-node result is close to the NIC line rate, and the Pod-to-Pod result is much lower, with no connection errors. + +2. Read the offload settings of the Geneve tunnel interface on both nodes: + + ```bash + ethtool -k genev_sys_6081 + ``` + + Expected for this article: `tx-checksumming: off`, `tcp-segmentation-offload: off`, and `tx-tcp-segmentation: off [requested on]`. If `tx-checksumming` is already `on`, this cause does not apply and the throughput gap has another reason. + +3. Check whether the offload state lines up with recent node reboots or `kube-ovn-cni` restarts. A node where TX offload was turned on earlier and is `off` again after a restart matches this cause: + + ```bash + kubectl -n kube-system get pod -o wide | grep kube-ovn-cni + uptime + ``` + + The `AGE` and restart count of the `kube-ovn-cni` Pod on the node, and the node uptime, show when the setting was last reset. + +4. Rule out a connectivity fault. Cross-node Pod traffic that times out or loses packets, rather than being slow, belongs to the related article on Geneve failures, not to this one. diff --git a/docs/en/solutions/Cross_node_Pod_traffic_fails_on_the_Kube-OVN_Geneve_overlay_because_of_UDP_6081_TX_offload_or_MTU_on_ACP.md b/docs/en/solutions/Cross_node_Pod_traffic_fails_on_the_Kube-OVN_Geneve_overlay_because_of_UDP_6081_TX_offload_or_MTU_on_ACP.md new file mode 100644 index 000000000..fe772f761 --- /dev/null +++ b/docs/en/solutions/Cross_node_Pod_traffic_fails_on_the_Kube-OVN_Geneve_overlay_because_of_UDP_6081_TX_offload_or_MTU_on_ACP.md @@ -0,0 +1,212 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.10.x + - 3.16.x + - 4.0.x + - 4.3.x +--- + +# Cross-node Pod traffic fails on the Kube-OVN Geneve overlay because of UDP 6081, TX offload or MTU on ACP + +## Issue + +Pods on one node cannot reach Pods on other nodes, while the hosts can reach each other and Services sometimes still answer. Typical reports: + +- Pod IP and Service access fail on and off from a few nodes, with no clear pattern. +- A new node joins, and its Pods (or CoreDNS on it) cannot reach the API server Service. `kube-ovn-pinger` on the new node logs `failed to parse config` and times out against the API. +- NodePort access through a second node is unstable, and a capture of Geneve traffic shows `IP/TCP Checksum Bad`. +- Pods fail only on physical machines while VM nodes are fine. +- After the node operating system was upgraded (for example to Kylin), some Pods fail to start because they cannot resolve or connect to Service names. +- Engineers ask how to see and change the Kube-OVN MTU and why the standard value is `1400`. +- After an MTU change, operations become slow; after the change is rolled back, containers still cannot reach external Redis or MySQL, or `ovn0` stays at `1350` instead of returning to `1400`. + +All of these involve the overlay path between nodes: Geneve on UDP `6081`, the offload settings of the encapsulation NIC, and the MTU. + +## Root Cause + +### Cause 1: the new node receives no Geneve traffic on UDP 6081 + +Kube-OVN carries Pod traffic between nodes in Geneve tunnels on UDP `6081`. When a firewall, security group or switch ACL blocks that port, `tcpdump` on the new node shows no UDP `6081` from the existing nodes, the tunnel never comes up, the new node's OVN components stay abnormal, and `kube-ovn-pinger` logs `failed to parse config` because it cannot reach the API through the overlay. + +### Cause 2: tx-checksum-ip-generic is on and Geneve packets are damaged + +With checksum offload on, the kernel leaves the checksum of encapsulated packets for the NIC to fill in. When `ethtool -k ` shows `tx-checksum-ip-generic: on` (not `off [fixed]`) and the NIC fills it in wrongly, cross-node encapsulated traffic fails or the encapsulation NIC counts checksum errors, while tiny pings still pass. In extreme cases the offload also makes connection tracking drop connections, which shows up as loss and retransmissions. + +Situations fixed by turning this offload off: + +- Pod IPs and Services intermittently unreachable from only some nodes. +- NodePort access unstable when a node forwards to another node. +- Pods in a new cluster unable to reach the `kubernetes` Service or any node while node-to-node traffic works. +- On a global cluster, `kubectl` on two of three masters unable to reach the API server, Harbor (sharing the global ALB VIP) unreachable, `console-portal` returning `500`, and `apollo` logging `Get router manager error {"error": "cluster not found"}` and `net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)`. All OVN Pods were `Running`, and restarting OVN and platform components did not help. + +Two cautions: + +- A flag that is already `off [fixed]` rules this cause out (look at Cause 1), and a flag that is `on` while the `genev_sys_6081` counters do not move is an idle tunnel interface, not a checksum fault. +- `IP/TCP Checksum Bad` on inner packets in a capture on the sending node is not proof of corruption by itself: with the offload on, `tcpdump` sees the packet before the NIC fills in the checksum. Only a capture on the receiving side is evidence. + +### Cause 3: transmit offload on a physical or Kylin node rewrites encapsulated packets + +On some physical NICs and on Kylin and other domestic operating systems, a NIC validation defect is incompatible with OVN encapsulation and modifies transmitted packets. UDP `6081` can already be open, and the node still cannot use the Pod network. Turning all transmit offload off (`ethtool -K tx off`) recovers it. This is broader than Cause 2, which turns off only `tx-checksum-ip-generic`. + +Situations fixed this way: + +- On ACP 4.3.2 ARM, Pods on a new physical machine cannot complete cross-node requests, while the masters are virtual machines. In a cluster mixing VMs and physical machines, Pods fail only on the physical machines. +- On a new Kylin node, Pods cannot reach Pods on other nodes while Services still work. A capture shows Geneve-encapsulated ICMP requests and replies between nodes with no kernel drops. `ethtool -K tx off` on the cluster bond fixes it. +- On a new node, `cert-manager-cainjector`, `kube-ovn-pinger`, `sentry` and other Pods are `CrashLoopBackOff`, and `coredns` is `Running` but `0/1`, logging `plugin/kubernetes: waiting for Kubernetes API before starting server`, `plugin/ready: Plugins not ready: "kubernetes"` and `starting server with unsynced Kubernetes API`. From the node, the API server Service on `443` answers; from inside the `coredns` network namespace it hangs. `ethtool -k ` shows `tx-checksumming: on`. +- On a new workload cluster of arm64 physical machines with bonded NICs, every `kube-ovn-pinger` Pod is `CrashLoopBackOff` with `failed to get self pod ... context deadline exceeded`. The physical NICs under the bond compute wrong checksums for Geneve packets; `tx off` is needed on every slave NIC of the bond, and the permanent fix is a NIC firmware update. The step-by-step procedure is in [kube-ovn-pinger CrashLoopBackOff context deadline exceeded on bonded NICs due to Geneve TX checksum offload on ACP](kube-ovn-pinger_CrashLoopBackOff_context_deadline_exceeded_on_bonded_NICs_due_to_Geneve_TX_checksum_offload_on_ACP.md). +- After the node OS was upgraded to Kylin, some Pods fail to start because they cannot resolve or connect to Service names. With older kernels and some domestic NICs, checksum offload makes the NIC compute wrong checksums, and the receiving NIC drops the packets as invalid. There are two fixes: upgrade the kernel to `4.19.90-25.16.v2101` or later, or turn checksum offload off with `ethtool -K tx off`. + +### Cause 4: which MTU Kube-OVN uses, and why the standard value is 1400 + +By default Kube-OVN reads the MTU of the host NIC and sets the Pod and OVS interface MTU from it, reduced in overlay networks by the encapsulation overhead: + +| Encapsulation | Pod MTU | +|---|---| +| Geneve over IPv4 | physical MTU - 100 | +| Geneve over IPv6 | physical MTU - 120 | +| VXLAN over IPv4 | physical MTU - 50 | +| VXLAN over IPv6 | physical MTU - 70 | + +The standard value of `1400` is the usual `1500` physical MTU minus `100` bytes for Geneve, which avoids fragmentation. Underlay Pods are not encapsulated, so their MTU follows the physical NIC (`1500` on a `1500` NIC). On ACP 4.x the Kube-OVN MTU can be customized, globally for overlay networks or per subnet, with the product procedure **Networking > How To > Kube-OVN > Configure MTU**. ACP 3.16 has no supported way to change the default OVN MTU. + +### Cause 5: after an MTU change is rolled back, ovn0 and some Pod interfaces keep the old MTU + +When the OVN MTU is changed (for example from `1400` to `1350`) and then rolled back, Pods created while the MTU was `1350` keep `1350` on their interfaces. In OVS an internal port such as `ovn0` cannot have a larger MTU than the other interfaces attached to the same bridge, so those Pods also hold `ovn0` at `1350`. Workloads on those Pods can fail to connect to external databases such as Redis or MySQL. This is default behavior: once every Pod with the old MTU is restarted, all interfaces become `1400` and `ovn0` and the other internal ports follow. + +## Resolution + +### Cause 1: the new node receives no Geneve traffic on UDP 6081 + +1. Open UDP `6081` from every existing node's encapsulation address to the new node, and the reverse path. Check the host firewall and any security group or switch ACL between the nodes. Restrict the source to the node CIDR. + + > **Warning:** The rule opens the Geneve port on the node. Limit the source to the cluster node prefix, apply it in a maintenance window, and keep console access. To roll back, delete the same rule (`iptables -D INPUT ...`). A network ACL outside the host has to be rolled back on that device. + + ```bash + iptables -I INPUT -p udp -s --dport 6081 -j ACCEPT + iptables -I OUTPUT -p udp -d --dport 6081 -j ACCEPT + ip6tables -I INPUT -p udp -s --dport 6081 -j ACCEPT + ``` + +2. Persist the rule with the mechanism the node already uses (`iptables-save`, `firewall-cmd --permanent --add-rich-rule`, or the cloud security group). +3. After packets show up in `tcpdump`, delete the unready `kube-ovn-cni` and `kube-ovn-pinger` Pods on that node only, so they rebuild the tunnel. + +### Cause 2: tx-checksum-ip-generic is on and Geneve packets are damaged + +1. Disable the generic IP checksum offload on the encapsulation NIC. Skip this when the driver already reports `off [fixed]` or when error counters are not increasing. + + > **Warning:** Disabling offload raises CPU use on that NIC and can lower throughput. Apply it to the encapsulation interface only, in a maintenance window. To roll back, run `ethtool -K tx-checksum-ip-generic on` if the driver allows it. + + ```bash + ethtool -K tx-checksum-ip-generic off + ethtool -k | grep tx-checksum-ip-generic + ``` + +2. The setting does not survive a reboot. To keep it, add a udev rule (replace `` in both places), reboot in a window, and confirm the flag afterwards. + + > **Warning:** A udev rule runs on every add event for that interface. A wrong interface name turns offload off on another NIC or does nothing. To roll back, delete the rule file and reload udev. + + ```bash + cat > /etc/udev/rules.d/99-ovn-fix-checksum.rules <<'RULE' + ACTION=="add", SUBSYSTEM=="net", KERNEL=="", RUN+="/sbin/ethtool -K tx-checksum-ip-generic off" + RULE + udevadm control --reload-rules && udevadm trigger + ethtool -k | grep tx-checksum-ip-generic + ``` + +### Cause 3: transmit offload on a physical or Kylin node rewrites encapsulated packets + +1. Only when cross-node Pod traffic fails on a physical node, a newly added Kylin node, or a node whose OS has the NIC validation defect, and UDP `6081` is open (otherwise it is Cause 1), turn transmit offload off on the encapsulation NIC. + + > **Warning:** `tx off` disables transmit offload on that NIC, raises CPU use and can lower throughput. Apply it to the encapsulation NIC only, in a maintenance window. To roll back, run `ethtool -K tx on` if the driver accepts it. A healthy VM node does not need this. + + ```bash + ethtool -k | grep -E '^tx|tx-checksum|tx-udp' + ethtool -K tx off + ethtool -k | grep -E 'tx-checksumming|tcp-segmentation-offload' + ``` + + Expected after the change: `tx-checksumming: off`, `tx-checksum-ip-generic: off`, `tcp-segmentation-offload: off`. + +2. On a bond, apply `tx off` to the bond and to every slave NIC (`cat /proc/net/bonding/` lists them). To keep the setting on nodes that use network-scripts files, add it to each NIC's file (keep a copy first; to roll back, restore it): + + ```bash + cp -a /etc/sysconfig/network-scripts/ifcfg- /root/ifcfg-.before + echo 'ETHTOOL_OPTS="-K tx off"' >> /etc/sysconfig/network-scripts/ifcfg- + ``` + +3. Crash-looping Pods retry on their own once the path works; `kubectl -n kube-system delete pod -l app=kube-ovn-pinger` only skips the back-off wait. Ask the NIC vendor for a firmware update that fixes the offloaded checksum. + +4. For the Kylin kernel case, the alternative to `tx off` is a kernel at `4.19.90-25.16.v2101` or later. + + > **Warning:** A kernel upgrade needs a drain and a reboot of each node. Upgrade one node at a time and keep the previous kernel in the boot menu as the rollback. + + ```bash + uname -r + kubectl drain --ignore-daemonsets --delete-emptydir-data + # install the vendor kernel package and reboot the node + kubectl uncordon + ``` + +### Cause 4: which MTU Kube-OVN uses, and why the standard value is 1400 + +Read the MTU in effect before changing anything. On ACP 4.x, follow the product procedure **Configure MTU**; on ACP 3.16 keep the default. + +```bash +kubectl -n kube-system exec -- ovs-vsctl list interface | grep -E "name|mtu" +kubectl -n exec -- ip link show eth0 +ip link show +``` + +`ip link show eth0` inside the Pod is the value the Pod really uses. + +### Cause 5: after an MTU change is rolled back, ovn0 and some Pod interfaces keep the old MTU + +1. After the MTU setting itself is back at `1400`, find the interfaces that still carry the old value. The list also prints `lo` and the physical NICs, whose MTU is legitimately different; look at the Pod veths and `ovn0`. + + ```bash + ip -o link show | awk '{for(i=1;i<=NF;i++) if($i=="mtu") print $2, $(i+1)}' | grep -v ' 1400$' + ip link show ovn0 + kubectl get pod -A -o wide --field-selector spec.nodeName= + ``` + +2. Restart the Pods whose `eth0` still reports the old MTU. When no Pod interface on the bridge is below `1400` any more, `ovn0` follows. + + > **Warning:** Restarting the Pods interrupts those workloads; roll them in a window, spare replicas first. Only after the MTU setting itself has been restored; otherwise the new Pods get the wrong value again. + + ```bash + kubectl -n exec -- ip link show eth0 + kubectl -n rollout restart deploy/ + ip link show ovn0 + ``` + +## Diagnostic Steps + +Compare a failing node with a healthy one. Check that Geneve arrives, then read the offload flags on the encapsulation NIC (often a bond): + +```bash +tcpdump -ni any -c 20 'udp and port 6081' +iptables -S | grep 6081 || true +ethtool -k | grep -E '^tx-checksumming|tx-checksum-ip-generic|tcp-segmentation-offload' +ip -s link show genev_sys_6081 +netstat -us | grep -i InCsumErrors +kubectl -n kube-system exec -- ovs-vsctl show | grep -C5 remote_ip +``` + +Test from a Pod network namespace as well as from the host. The host path working while the Pod path fails is typical for Causes 2 and 3: + +```bash +crictl ps -a | grep coredns +crictl inspect | grep -i '"pid"' +nsenter -n -t -- timeout 5 bash -c '/443' && echo open || echo blocked +timeout 5 bash -c '/443' && echo open || echo blocked +kubectl -n exec -- curl -k -m 5 https://:443/healthz +``` + +- No UDP `6081` from the other nodes is Cause 1. +- `tx-checksum-ip-generic: on` with failing cross-node traffic or climbing errors is Cause 2. +- A physical or newly added Kylin node that works only after `tx off`, Pods that cannot resolve Service names after an OS upgrade to Kylin, or bonded physical NICs where API clients crash-loop with `context deadline exceeded` to the `kubernetes` Service, is Cause 3. `InCsumErrors` increasing while the Pods fail supports it. Rule out a node firewall first: if the node itself cannot reach the API server on `6443`, the problem is in the host network, not in the overlay. +- Interfaces still at an old MTU after a rollback, with `ovn0` below `1400`, are Cause 5. diff --git a/docs/en/solutions/Cross_node_traffic_fails_in_the_host_network_path_default_route_firewall_lost_tunnel_packets_or_iptables_on_ACP.md b/docs/en/solutions/Cross_node_traffic_fails_in_the_host_network_path_default_route_firewall_lost_tunnel_packets_or_iptables_on_ACP.md new file mode 100644 index 000000000..44b9ea13a --- /dev/null +++ b/docs/en/solutions/Cross_node_traffic_fails_in_the_host_network_path_default_route_firewall_lost_tunnel_packets_or_iptables_on_ACP.md @@ -0,0 +1,201 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# Cross-node traffic fails in the host network path: default route, firewall, lost tunnel packets or iptables on ACP + +## Issue + +The CNI Pods are `Running`, yet traffic between nodes breaks. Typical reports: + +- Nodes added to a cluster through their out-of-band address become abnormal. +- An interface that the server can reach cannot be reached from inside a container on some nodes. +- Hundreds of Pods are down, and `kube-ovn-pinger` can reach its own node but no other node. +- Monitoring is abnormal on a new cluster because one `vmselect` replica has no network. +- After someone ran `iptables-restore` on a node, the workload cluster shows as abnormal in the platform, its VIP answers only some of the time, and restarting containerd does not help. On the global cluster, `erebus` logs `cluster "" is temporarily not accessible`. +- Cross-node communication fails and `ping` returns `Destination Host Prohibited`. + +Each cause below is found by testing the path between two specific nodes: their routes, the firewalls between them, and whether a packet sent by one ever arrives at the other. + +## Root Cause + +### Cause 1: nodes joined through the out-of-band IP have their only default route on the out-of-band NIC + +Hosts with an out-of-band and an in-band address (for example `bond1` and `bond0`) were added to the cluster using the out-of-band IP. On normal nodes the default route prefers `bond0` (in-band, metric `300`) with `bond1` (out-of-band, metric `301`) as backup. On the abnormal nodes the only default route is via `bond1`, and the `bond0` default route is missing. Kubernetes control-plane traffic depends on the in-band interface, so the nodes lose communication with the control plane. Restoring the default route on the joined nodes brings the node status back, but Pods still fail to start; the routing has to be correct before the node is added. + +### Cause 2: the firewall between two groups of nodes is not fully open + +The `kube-ovn-pinger` results on an affected node show the pattern: `ping pod` to most other pinger Pods reports `count: 3, loss count 3, average rtt 0.00ms`, with only a few at `loss count 0`; `ping node` reports `loss count 3` to most nodes and `loss count 0` only to a small group, which is the other nodes of the affected group plus the masters; DNS lookups of `kubernetes.default` and an external name work; `ping external address` reports `loss count 3`. The affected nodes and the other nodes belong to two different network zones or systems, and the network policies between them are not fully open for node and Pod traffic. + +### Cause 3: replies never reach the requesting node because a conflicting subnet takes the return path + +From the `kube-ovn-pinger` Pod, a request to the local node works and requests to other nodes fail. A capture on `genev_sys_6081` while a `curl` runs in the pinger's network namespace shows, on port `6443`, the requester sending repeated `[SYN]` packets and `[TCP Retransmission]` of them, and the target nodes answering with `[SYN, ACK]` (some marked `[TCP Retransmission]` and `[TCP Out-Of-Order]`) that the requester never receives. A subnet that conflicts with a cluster range sends the replies elsewhere. Removing the conflicting subnet fixes it. When the nodes are VMs, also check whether the virtualization routing mode or the physical hosts under the VMs changed. + +### Cause 4: IPIP packets from one node never arrive at one peer's NIC + +On a Calico cluster in IPIP mode, every node has a `tunl0` interface (` mtu 1480`) with a `/32` address. From one master, a ping to a worker's `tunl0` address gets `100% packet loss`, while another worker reaches it. `tcpdump` on the master shows echo requests leaving on `tunl0` with no replies; a capture of IPIP (`ip proto 4`) on `eth0` shows the worker sending IPIP packets to the master (including TCP `[S.]` replies from port `6443`), and the master's `eth0` receives none of them, with no RX or TX errors or drops. The packets are lost in the underlying network between those two hosts, outside the cluster. Pods on the affected node (for example a `vmselect` replica) lose connectivity. + +### Cause 5: iptables-restore overwrote the node rules and the workload cluster VIP flaps + +`iptables-restore` run on a node of a workload cluster overwrites the node's iptables rules. The cluster's self-built VIP (served by keepalived, the `alive` container) then answers only some of the time, and the platform marks the cluster abnormal. Restarting containerd does not help. On the global cluster both `erebus` Pods in `cpaas-system` are `Running` with no restarts, and their log repeats: + +```text +proxy.go:158] cluster "" is temporarily not accessible +Generate new reverse proxy for: 600:https://
:6443 +``` + +Restarting the keepalived container restores VIP connectivity, and the workload cluster returns to normal. + +The signature of this cause is the VIP itself failing part of the time. The same `temporarily not accessible` text, returned to UI requests and by `apollo` as `410` with reason `Gone`, can also appear while the VIP answers every time; then the problem is on the platform side, not in the node network, and the steps below do not apply. + +### Cause 6: Destination Host Prohibited, a host FORWARD rule rejects cross-node traffic + +`calico-node`, `kube-proxy` and the other node Pods are `Running`, but a cross-node `ping` gets `Destination Host Prohibited` for every packet. `iptables -L FORWARD -n --line-numbers` on the affected node shows, after the `DOCKER-USER`, `DOCKER-ISOLATION-STAGE-1`, `ACCEPT` and `DOCKER` rules, a rule `REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited`, and the command may also print `# Warning: iptables-legacy tables present, use iptables-legacy to see them`. That host firewall rule rejects the forwarded cross-node traffic. + +## Resolution + +### Cause 1: nodes joined through the out-of-band IP have their only default route on the out-of-band NIC + +1. Compare the routing table with a normal node: + + ```bash + ip route show default + ip -br addr show bond0 bond1 + ``` + +2. Remove the node, correct the default route, and join it again. Restoring the route alone brings the node status back but does not make the Pods start. + + > **Warning:** Changing the default route can cut the SSH session that uses it; work from a console. Deleting a node evicts and removes all its Pods, and the node must be joined again; drain it first. Only when the node's default route is on the out-of-band NIC and normal nodes prefer the in-band NIC. Save `ip route` before the change; to roll back, add the saved route again. + + ```bash + kubectl drain --ignore-daemonsets --delete-emptydir-data + kubectl delete node + ip route del default via dev bond1 + ip route add default via dev bond0 metric 300 + ``` + +3. Make the route persistent in the node's network configuration, with a lower metric for the in-band NIC than for the out-of-band one, confirm `ip route show default` matches a normal node, and add the node again with its in-band address. + +### Cause 2: the firewall between two groups of nodes is not fully open + +1. Use the pinger results to map which nodes cannot reach which: + + ```bash + kubectl -n kube-system get pod -l app=kube-ovn-pinger -o wide | grep + kubectl -n kube-system logs --tail=500 | grep -E 'ping (pod|node|external)' | grep -v 'loss count 0' | head -50 + ``` + +2. Group the failing targets by node. A clean split along a system or zone boundary, with only the masters reachable across it, is this cause. Have the network team open the policies between the two groups for node and Pod traffic. + +### Cause 3: replies never reach the requesting node because a conflicting subnet takes the return path + +1. Capture on the Geneve interface in one window, and run the `curl` from the pinger's network namespace in another, on the same node: + + ```bash + tcpdump -i genev_sys_6081 -n -v -w genev_sys_6081_traffic.pcap + ``` + +2. Check the route back to the source on the target node and on the requester, and compare it with the cluster subnets: + + ```bash + ip route get + ip route + kubectl get subnet -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidrBlock' + ``` + + A reply route that points somewhere other than the path back to the requester, through a range that overlaps a cluster subnet, identifies the conflicting subnet. + +3. Remove the conflicting subnet or route. + + > **Warning:** Deleting a subnet or a route that overlaps a cluster range changes where that whole range is routed. Confirm which object is the conflicting one and that nothing still needs it. Save it first (`kubectl get subnet -o yaml`, or `ip route` on the node) so it can be recreated. + +### Cause 4: IPIP packets from one node never arrive at one peer's NIC + +1. Prove the loss on the wire. Run the `ip proto 4` capture on both nodes at the same time: + + ```bash + ip a | grep tunl0 + ping -c 5 + tcpdump -i any -nnve icmp and host + tcpdump -i eth0 -nnve ip proto 4 and host and host + ip -s link show eth0 + ``` + + Packets leaving the sender's NIC that never appear on the receiver's NIC, with clean NIC counters, are this cause. + +2. Hand the captures to the IaaS or network team. Until the underlying network is repaired, take the affected node out of service (replace it if needed). + + > **Warning:** Draining the node moves or stops every Pod on it. To roll back, uncordon it once the underlying network is repaired. + + ```bash + kubectl drain --ignore-daemonsets --delete-emptydir-data + kubectl uncordon + ``` + +### Cause 5: iptables-restore overwrote the node rules and the workload cluster VIP flaps + +1. On the global cluster, confirm that `erebus` sees the cluster as unreachable, then test the workload cluster's VIP repeatedly: + + ```bash + kubectl -n cpaas-system get pod -o wide | grep erebus + kubectl -n cpaas-system logs --tail=500 | grep -E 'temporarily not accessible|Generate new reverse proxy' + for i in $(seq 1 30); do curl -k -s -o /dev/null -m 2 -w '%{http_code}\n' https://:6443/healthz; sleep 1; done | sort | uniq -c + ``` + + Restarting `erebus` can confirm that the status persists; the fault is on the VIP side. + +2. Restart the keepalived container (`alive`) on the workload cluster masters, one master at a time. + + > **Warning:** Only when the `healthz` loop above shows failures or timeouts. Restarting keepalived moves or briefly drops the VIP, so API access through the VIP is interrupted for a moment. Test the VIP between masters. There is nothing to roll back; the container comes back with its own configuration. + + ```bash + kubectl get pod -A -o wide | grep -i alive + crictl ps | grep -i alive || docker ps | grep -i alive + ``` + + If `alive` is a Pod, delete that Pod on the master so it is recreated. Otherwise restart the container with the node's runtime (`docker restart `, or `crictl stop ` for a kubelet-managed container, which kubelet then starts again). + +3. Save the node's rules (`iptables-save`) before anyone runs `iptables-restore` on a cluster node again. + +### Cause 6: Destination Host Prohibited, a host FORWARD rule rejects cross-node traffic + +Only when a cross-node ping returns `Destination Host Prohibited` and the `FORWARD` chain holds a `REJECT ... reject-with icmp-host-prohibited` rule that healthy nodes do not have, delete that rule by its line number. + +> **Warning:** Changing the host firewall opens forwarded traffic that the rule used to reject; agree it with whoever owns the node security policy. Save the rules first. To roll back, run `iptables-restore < /root/iptables.before-forward`. The line number changes when rules are added or removed, so read it again right before the delete. + +```bash +iptables-save > /root/iptables.before-forward +iptables -L FORWARD -n --line-numbers +iptables -D FORWARD +ping -c 3 +``` + +Find where the host loads the rule from (for example a saved iptables service ruleset) so that a reboot does not bring it back. + +## Diagnostic Steps + +Pick one failing pair of nodes and test the path between them in both directions: + +```bash +kubectl get nodes -o wide +ip route show default +ip route get +kubectl -n kube-system logs --tail=200 | grep 'loss count' | grep -v 'loss count 0' | head +tcpdump -ni host -c 50 +iptables -L FORWARD -n --line-numbers | grep -E 'REJECT|DROP' +``` + +- A default route on a different NIC than on healthy nodes, on nodes joined through another address, is Cause 1. +- Pinger losses that split the cluster into two groups, with masters reachable from both, are Cause 2. +- Requests that leave, answers that are sent, and answers that never arrive at the requester, together with an overlapping subnet, are Cause 3. +- Encapsulated packets that leave one NIC and never reach the other, with clean counters, are Cause 4. +- `temporarily not accessible` in `erebus` with a VIP that answers only some of the time, after `iptables-restore` on a node, is Cause 5. A VIP loop that returns only `200` while the platform still reports `temporarily not accessible` does not match Cause 5. +- `Destination Host Prohibited` with a `REJECT` rule in the `FORWARD` chain is Cause 6. +- If the cluster VIP is a domain name that a node or a Pod cannot resolve, see [Platform console does not open because the cluster VIP domain name cannot be resolved inside the cluster on ACP](Platform_console_does_not_open_because_the_cluster_VIP_domain_name_cannot_be_resolved_inside_the_cluster_on_ACP.md). +- If Geneve packets arrive but are damaged, see [Cross-node Pod traffic fails on the Kube-OVN Geneve overlay because of UDP 6081, TX offload or MTU on ACP](Cross_node_Pod_traffic_fails_on_the_Kube-OVN_Geneve_overlay_because_of_UDP_6081_TX_offload_or_MTU_on_ACP.md). diff --git a/docs/en/solutions/Custom_or_imported_monitoring_dashboards_show_parse_errors_unsupported_chart_types_or_no_data_on_ACP.md b/docs/en/solutions/Custom_or_imported_monitoring_dashboards_show_parse_errors_unsupported_chart_types_or_no_data_on_ACP.md new file mode 100644 index 000000000..d2568549c --- /dev/null +++ b/docs/en/solutions/Custom_or_imported_monitoring_dashboards_show_parse_errors_unsupported_chart_types_or_no_data_on_ACP.md @@ -0,0 +1,170 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.16 + - 4.0 + - 4.1 + - 4.2 +--- + +# Custom or imported monitoring dashboards show parse errors, unsupported chart types, or no data on ACP + +## Issue + +A dashboard that was built in, or imported into, the platform monitoring dashboards (**Operations Center** > **Monitoring** > **Monitoring Dashboards**) does not behave like it did in Grafana. Typical symptoms: + +- A panel shows `加载失败,请 刷新` (load failed, please refresh) with the error `update matchers error: 1:59: parse error: missing unit character in duration`. The query uses `[$__range]`. +- After a Grafana dashboard JSON was imported, some panels show `不支持的图表类型` (unsupported chart type) instead of a chart. +- A standard Grafana dashboard JSON (for example one supplied by a hardware or software vendor) was imported, and some panels show no data although the same expressions return data on the Prometheus page. This is sometimes described as the import "escaping" or changing characters of the original JSON. +- A user wants to create a new folder for dashboards, or wants application monitoring to show a custom dashboard selected by application labels. +- An external dashboard JSON was imported into the custom monitoring panel of an application (the **Monitoring** tab of a Deployment), and the panels show `加载失败,请 刷新`. In the browser developer tools a `monitordashboards` request fails with `mysql_up should has selector namespace=...`. +- There is no Envoy Gateway dashboard under **Monitoring Dashboards** comparable to the ALB dashboards, and 4xx/5xx response-code data is needed for alerting. +- In the monitoring dashboards the **HTTPRoute status** and **Rule Status** panels show no data while every other panel is normal. + +Built-in dashboards that freeze the browser, stay blank, or are missing are a different problem; see [Monitoring page freezes the browser or built-in dashboards are missing on ACP](Monitoring_page_freezes_the_browser_or_built_in_dashboards_are_missing_on_ACP.md) and [Monitoring dashboard panels are blank or drop to zero while the backend has data on ACP](Monitoring_dashboard_panels_are_blank_or_drop_to_zero_while_the_backend_has_data_on_ACP.md). + +## Root Cause + +### Cause 1: Grafana built-in variables such as $__range are not supported + +A query such as `sum(last_over_time(api_requests_total{appId="$appId"}[$__range]))` fails with `parse error: missing unit character in duration`. The platform monitoring dashboard does not support Grafana built-in variables such as `$__range`, and it does not support making the range selector inside a query follow a variable. The placeholder is sent unchanged, so the range selector is not a valid PromQL duration. + +### Cause 2: the imported panel type is not supported + +Some Grafana panel types are not supported by the platform dashboard. An imported panel of such a type shows `不支持的图表类型` (unsupported chart type). + +### Cause 3: imported queries keep Grafana template-variable selectors + +An imported query can keep label selectors that reference Grafana template variables, for example `DCGM_FI_DEV_POWER_USAGE{instance=~"$instance", gpu=~"$gpu"}` in the NVIDIA DCGM Exporter dashboard. If those variables are not defined on the platform dashboard, they do not resolve and the panel returns no data. + +### Cause 4: dashboard folders and label-based application dashboards + +Whether a new dashboard folder can be created depends on the version. On ACP 4.0.3 new monitoring folders cannot be created. On versions whose **Create Dashboard** form has a required **Folder** (`文件夹`) field, limited to 32 characters, typing a new name into that field creates the folder. + +Showing a custom dashboard in application monitoring based on application labels is not supported. + +### Cause 5: importing external JSON into an application's custom monitoring panel is not supported + +On the **Monitoring** tab of a Deployment, the custom panel accepts charts built in the console. Importing an external dashboard JSON into it, or changing its panels through YAML, is not a supported feature. Imported panels fail with `加载失败,请 刷新`; in the browser developer tools the `variables?expr=label_values...` and `metrics?start=...` requests fail, and a `monitordashboards` request returns `code: 500`, `reason: "Unknown"` with a message such as `mysql_up should has selector namespace=...`. The message shows that queries on this panel must carry a `namespace=` selector, which the imported query does not have. + +Importing a Grafana JSON as a dashboard under **Operations Center** > **Monitoring** > **Monitoring Dashboards** is a different feature and is not affected. + +### Cause 6: the Envoy Gateway dashboards have not been added + +The Envoy Gateway dashboards are not listed under **Monitoring Dashboards** until they are added to the cluster. They are not installed with Envoy Gateway itself. + +### Cause 7: HTTPRoute status and Rule Status panels are empty because the cluster has no HTTPRoute + +Both panels are built from HTTPRoute resources. When the cluster has no HTTPRoute, they have nothing to show. This is expected and is not a fault. + +### Cause 8: a dashboard variable expression is malformed + +An imported dashboard can carry a variable (`变量`, Variables) whose own expression is invalid, for example with an extra closing parenthesis. The panels that use the variable then show no data, while the panel expressions return data on the Prometheus page when real values are filled in. This differs from Cause 3: here the variable is defined on the dashboard, but its query cannot be parsed. + +## Resolution + +### Cause 1: Grafana built-in variables such as $__range are not supported + +Remove the variable from the range selector. Put a fixed duration in the query, or use a query that does not need the range, and select the window with the time range picker at the top of the dashboard (for example `近 7 天`, last 7 days): + +```promql +sum(last_over_time(api_requests_total{appId="$appId"}[7d])) +``` + +A simple check that the time picker drives the panel is a query such as `up{service=~".*"}`, which renders data across the selected range. + +### Cause 2: the imported panel type is not supported + +Edit each affected panel and change its type to a line chart (`趋势图`). No setting renders the unsupported Grafana panel type. + +### Cause 3: imported queries keep Grafana template-variable selectors + +1. Edit the panel query and remove the selectors that reference Grafana template variables (`$instance`, `$gpu`, and similar), or replace them with fixed values or with variables defined on the platform dashboard. +2. Run the edited expression in vmui or the Prometheus UI before saving it, for example: + + ```promql + DCGM_FI_DEV_POWER_USAGE + ``` + + With the legend `GPU {{.gpu}}`, this draws the power usage of each GPU. + +### Cause 4: dashboard folders and label-based application dashboards + +If the **Create Dashboard** form has the **Folder** field, type the new folder name (up to 32 characters) there when creating the dashboard. On a version without that field, new folders cannot be created. Label-based custom dashboards in application monitoring are not available; there is no configuration for them. + +### Cause 5: importing external JSON into an application's custom monitoring panel is not supported + +1. Build each chart with **Add Chart** (`添加图表`) on the application's custom panel instead of importing a JSON file. +2. Test each expression in vmui or the Prometheus UI first, restricted to the application's namespace: + + ```promql + mysql_up{namespace=""} + ``` + +There is no bulk-import path for application custom panels. + +### Cause 6: the Envoy Gateway dashboards have not been added + +1. Follow [How to Add Dashboards for EnvoyGateway](How_to_add_Dashboard_for_EnvoyGateway.md) (ACP 4.2.x and later) on the cluster where Envoy Gateway is installed. It applies a manifest, `envoy-gateway-dashboard.yaml`, that creates the `MonitorDashboard` resources `envoygateway-adminspace-control-panel-dashboard` and `envoygateway-adminspace-data-panel-dashboard` and the `PodMonitor` resources `envoy-gateway-controlplane-monitoring` and `envoy-gateway-proxy-monitoring` in `cpaas-system`. +2. Verify the objects: + + ```bash + kubectl -n cpaas-system get monitordashboard envoygateway-adminspace-control-panel-dashboard envoygateway-adminspace-data-panel-dashboard + kubectl -n cpaas-system get podmonitor envoy-gateway-controlplane-monitoring envoy-gateway-proxy-monitoring + ``` + +3. Open the dashboards in the `container-platform` folder. The data-plane dashboard has a **Response Codes (Route)** panel, which reads `envoy_cluster_upstream_rq_xx` grouped by `envoy_response_code_class`. For an alert on the 5xx responses of one gateway, use an expression such as: + + ```promql + sum by (envoy_cluster_name) (rate(envoy_cluster_upstream_rq_xx{gateway_namespace="", gateway="", envoy_response_code_class="5"}[2m])) + ``` + + Use `envoy_response_code_class="4"` for 4xx. + +To remove the dashboards and PodMonitors again, run `kubectl delete -f envoy-gateway-dashboard.yaml` with the same manifest. + +### Cause 7: HTTPRoute status and Rule Status panels are empty because the cluster has no HTTPRoute + +Confirm that the cluster has no HTTPRoute: + +```bash +kubectl get httproute -A +``` + +`No resources found` means the empty panels are expected and nothing needs to be fixed. The panels show data once HTTPRoute resources exist. + +### Cause 8: a dashboard variable expression is malformed + +1. Export the dashboard object and keep the export as a copy to compare against: + + ```bash + kubectl get monitordashboards.ait.alauda.io -A | grep -i + kubectl get monitordashboards.ait.alauda.io -n -o yaml > /tmp/.yaml + grep -n -E 'label_values|expr|query' /tmp/.yaml + ``` + +2. For every variable used by the empty panels, run its PromQL part in the Prometheus page or vmui: for a `label_values(,
` is the table that holds it (`filter` by default, `raw` for `PREROUTING` rules), and the security owner approves. Rollback: `iptables-restore < /root/iptables-before.rules`. + + ```bash + iptables -t
-L -n --line-numbers | grep '' + iptables -t
-D + nc -vz -w 3 + crictl pull ://: + ``` + + If firewalld or a host security agent manages the rule, remove it there as well, or it returns after a reload. + +3. For a firewall in front of Harbor (including rules in the `raw` table `PREROUTING` chain of the Harbor host), open the path to the address and port of Harbor's external URL. Image pulls by the kubelet and containerd leave from the node IPs, so allow the node IPs of every cluster that pulls. Service IPs never arrive at Harbor as a source. Pods that call Harbor themselves, such as build Pods, arrive with the node IP when their subnet translates outgoing traffic (Kube-OVN `natOutgoing: true`) and with the Pod IP otherwise: + + ```bash + kubectl get subnet -o custom-columns=NAME:.metadata.name,CIDR:.spec.cidrBlock,NAT:.spec.natOutgoing + ``` + +### Cause 3 + +Open egress to the public registry, or bring the image into Harbor. To bring it in, save the image on a machine that can download it, then load, tag, and push it on a host that reaches Harbor: + +```bash +# On the connected machine: +docker pull docker.io/library/: +docker save docker.io/library/: -o -.tar +# On a host that reaches Harbor: +docker load -i -.tar +docker tag docker.io/library/: ://: +docker push ://: +``` + +Change the workload, or the Helm chart values, to the Harbor reference, and give the workload a pull Secret for that project (see [Harbor image pull returns 401 because of the pull Secret, the project account, or node credentials](Harbor_image_pull_returns_401_because_of_the_pull_Secret_the_project_account_or_node_credentials.md)). + +### Cause 4 + +1. Measure reordering on a live transfer: + + ```bash + ss -ti dst | grep -oE 'rcv_ooopack:[0-9]+|retrans:[0-9/]+|reordering:[0-9]+' + nstat -az | grep -iE 'TCPOFOQueue|TCPSACKReorder|TCPTSReorder' + ``` + + A `rcv_ooopack` count that keeps growing during one pull points at the physical path. + +2. Have the virtualization team remove the redundant link that splits the flow, or change the uplink teaming so one flow stays on one physical path. + + > **Warning:** Impact: changing ESXi uplinks affects every VM on that host and can drop their network briefly. Precondition: a maintenance window, the current vSwitch and teaming configuration recorded, and the remaining uplink confirmed healthy. Rollback: re-add the link with the recorded configuration. + +3. Repeat the same pull and compare elapsed time and `rcv_ooopack`. + +### Cause 5 + +1. Run the same pull on a node in the external data center and on a node of the global cluster: + + ```bash + crictl pull ://: + ``` + + Layers that never leave 0.0 B before `read: connection reset by peer`, on the remote node only, confirm the cause. + +2. Have the network team adjust the WAF that intercepts the transfer, then retry the pull from the remote node. + +3. As an interim workaround while the path is broken, copy the image onto the hosts that run the workload and set the image pull policy to `IfNotPresent`. + + > **Warning:** Impact: with `IfNotPresent` those hosts do not refresh the tag until the policy changes or the local image is removed, so a later push of the same tag is not picked up and a registry outage is hidden. Precondition: the image is already on the hosts that will run the workload; use it only until the WAF change is verified. Rollback: set the policy back to `Always`. + + ```bash + kubectl patch deploy -n --type=json \ + -p '[{"op":"replace","path":"/spec/template/spec/containers/0/imagePullPolicy","value":"IfNotPresent"}]' + ``` + +### Cause 6 + +1. Confirm the node was just switched to containerd, the failing request goes to `http://...:80`, and the registry answers on its HTTPS port: + + ```bash + kubectl get node -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}{"\n"}' + crictl pull //: + nc -vz -w 3 11443 + ``` + +2. Log the node in to the registry again. + + > **Warning:** Only when the runtime switch preceded the failure and the pull goes to port 80. Impact: `nerdctl login` stores the registry credential, base64-encoded, in `/root/.docker/config.json` on this node, and the kubelet then sends it for pulls from that registry; `--insecure-registry` skips certificate verification for this login. Precondition: the account is one approved for node pulls, and the existing config file is copied aside. Rollback: `nerdctl -n k8s.io logout :11443`, or restore the copied file. + + ```bash + cp -a /root/.docker/config.json /root/docker-config.json.bak 2>/dev/null + nerdctl -n k8s.io login :11443 --insecure-registry + crictl pull //: + ``` + +3. Repeat on every switched node. If the pull still goes to port 80, compare `/etc/containerd/certs.d/` with a node that pulls (see [Harbor push or pull fails because the client uses the wrong scheme: HTTP response to HTTPS client, or port 443 refused](Harbor_push_or_pull_fails_because_the_client_uses_the_wrong_scheme_HTTP_response_to_HTTPS_client_or_port_443_refused.md)). + +## Diagnostic Steps + +1. Record the image reference, the node, and the first error line. `DeadlineExceeded` or `i/o timeout` means the TCP connection never completed; a `401`, `x509`, or `HTTP response to HTTPS client` line is not a network-path problem. + +2. Check whether every node fails or only one: + + ```bash + kubectl get pods -A -o wide | grep -E 'ImagePull|ErrImage' + ``` + + All nodes of a new or rebuilt cluster point to Cause 1 or Cause 2. + +3. From a failing node, test each port the image reference and the platform use: + + ```bash + nc -vz -w 3 + curl -sv --connect-timeout 10 http://:/v2/ + curl -sv --connect-timeout 10 https:///v2/ + ``` + + A timeout on `443` toward the global VIP selects Cause 1. A timeout toward Harbor with matching drop or reject rules on the node selects Cause 2. A timeout toward a public registry selects Cause 3. + +4. If transfers complete but slowly, read `ss -ti` during a pull. A high `rcv_ooopack` selects Cause 4; low reordering with slow transfers needs a bandwidth test on the path. + +5. A connection reset on a large pull from an external data center, with the same pull succeeding next to Harbor, selects Cause 5. + +6. Before removing firewall rules for a port 80 timeout, ask whether the node runtime was just switched from Docker to containerd. A switch, an `http://...:80` request for an HTTPS registry, and no matching drop rule select Cause 6. diff --git a/docs/en/solutions/Image_tag_cannot_be_selected_in_Update_Image_Version_because_the_integration_repositories_request_returns_500.md b/docs/en/solutions/Image_tag_cannot_be_selected_in_Update_Image_Version_because_the_integration_repositories_request_returns_500.md new file mode 100644 index 000000000..38caab447 --- /dev/null +++ b/docs/en/solutions/Image_tag_cannot_be_selected_in_Update_Image_Version_because_the_integration_repositories_request_returns_500.md @@ -0,0 +1,94 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "3.10" +--- + +# Image tag cannot be selected in Update Image Version because the integration repositories request returns 500 + +## Issue + +In the container platform, the image tag cannot be selected in the **Update Image Version** dialog of a Deployment container. The tag dropdown stays empty or fails to load, although the repository in Harbor has tags. The failure is intermittent: the same dialog sometimes works and sometimes does not. The browser developer tools show the failing request returning `500` after about 10 seconds with this body: + +```json +{ + "metadata": {}, + "status": "Failure", + "message": "Internal error occurred: unexpected error when reading response body. Please retry. Original error: net/http: request canceled (Client.Timeout or context cancellation while reading body)", + "reason": "InternalError", + "code": 500 +} +``` + +The `harbor-core` log repeats `[panic.go:969] Handler crashed with error net/http: abort Handler`, while every Harbor component is Running and none restarts often. + +## Root Cause + +### Cause 1: katanomi has run for a very long time without a restart, and its repository list requests to Harbor time out + +The dialog reads registry data through the DevOps v3 toolchain Harbor integration, which is served by katanomi (`katanomi-api`, `katanomi-plugin`, and `katanomi-controller` in `cpaas-system`, and `katanomi-operator` in `operators`). + +The dialog first lists registry projects (`GET .../projects?includeType=ImageRegistry&itemsPerPage=100&page=1`), and that call returns `200`. It then lists the repositories of the selected Harbor project: + +```text +GET /integrations/v1alpha1/namespaces//integrations//projects//repositories?itemsPerPage=100&page=1 +``` + +That call fails with the `500` above after about 10 seconds. When the katanomi Pods have been running for a very long time (for example about two years without a restart), they accumulate cached state, and with that state the repository list requests to Harbor time out. The integration API returns the timeout to the browser as `InternalError`. The `abort Handler` panic in `harbor-core` accompanies these timeouts and is consistent with Harbor's handler being aborted because the caller gave up the request, which is why Harbor itself looks healthy. + +Restarting the katanomi components, including the `katanomi-operator` Pod, reloads that state, and tag selection works again. + +## Resolution + +### Cause 1: katanomi has run for a very long time without a restart, and its repository list requests to Harbor time out + +1. Capture the requests in the browser developer tools (Network tab) while opening the tag dropdown. This cause needs both: `projects?includeType=ImageRegistry...` returns `200`, and `repositories?itemsPerPage=100&page=1` returns `500` with `Client.Timeout or context cancellation while reading body`. An empty `200` answer is not this cause. + +2. Confirm that Harbor has the repositories and answers directly, so that the fault is on the integration side: + + ```bash + curl -sk -u ':' --max-time 30 \ + -w '\ncode=%{http_code} time=%{time_total}\n' \ + 'https:///api/v2.0/projects//repositories?page=1&page_size=100' + kubectl -n get pods + kubectl -n logs --tail=300 | grep -c 'abort Handler' + ``` + +3. Read the katanomi Pod ages and restart counts: + + ```bash + kubectl get pods -A -o wide | grep katanomi + ``` + + Running katanomi Pods that are months or years old, with Harbor answering on its own, match this cause. A katanomi Pod in `CrashLoopBackOff` is a different fault. + +4. Save the logs, then restart the katanomi Pods. + + > **Warning:** Deleting the katanomi Pods makes DevOps pages, toolchain integrations, image and tag selectors, and pipeline triggers unavailable on that cluster until the new Pods are Ready. Running TaskRun Pods are not deleted. Precondition: only when the `repositories` call returns `500` with `Client.Timeout or context cancellation while reading body`, Harbor answers the same listing directly, and the katanomi Pods are Running. Do it in a quiet period. Rollback: none needed; the owners recreate the Pods from an unchanged spec. + + ```bash + mkdir -p /tmp/katanomi-logs + for p in $(kubectl -n cpaas-system get pods -o name | grep katanomi); do + kubectl -n cpaas-system logs "$p" --all-containers --tail=500 > "/tmp/katanomi-logs/$(basename "$p").log" + done + kubectl -n cpaas-system delete pod + kubectl -n operators delete pod + kubectl get pods -A | grep katanomi + ``` + +5. Open the **Update Image Version** dialog again, several times, because the fault was intermittent. The `repositories` request returns `200` well within 10 seconds and the tags are listed. If the `500` returns soon after the restart, keep the saved katanomi logs and the `harbor-core` log for the support team. + +## Diagnostic Steps + +```bash +kubectl get pods -A -o wide | grep katanomi +kubectl -n get pods +kubectl -n logs --tail=300 | grep 'abort Handler' +``` + +- Browser: `repositories?itemsPerPage=100&page=1` returns `500` with `Client.Timeout or context cancellation while reading body`, Harbor is healthy, and the katanomi Pods are very old: Cause 1. +- The image dropdown is empty in one environment only, with no `500`, and the platform project is not assigned to the Harbor project in the toolchain integration: see "Harbor toolchain integration fails: images not selectable, katanomi-plugin timeout, or push asks for credentials". +- A tag that cannot be selected in an artifact promotion form, not in the Update Image Version dialog, is a different problem. diff --git a/docs/en/solutions/Ingress_NGINX_controller_serves_the_fake_certificate_ignores_an_Ingress_or_rejects_Ingress_saves_on_ACP.md b/docs/en/solutions/Ingress_NGINX_controller_serves_the_fake_certificate_ignores_an_Ingress_or_rejects_Ingress_saves_on_ACP.md new file mode 100644 index 000000000..d6d246883 --- /dev/null +++ b/docs/en/solutions/Ingress_NGINX_controller_serves_the_fake_certificate_ignores_an_Ingress_or_rejects_Ingress_saves_on_ACP.md @@ -0,0 +1,229 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# Ingress NGINX controller serves the fake certificate, ignores an Ingress, or rejects Ingress saves on ACP + +## Issue + +These problems concern the Alauda build of Ingress NGINX Controller, deployed by `ingress-nginx-operator` from an `IngressNginx` resource (`ingressnginxes.ingress-nginx.alauda.io`), with IngressClass `ingress-nginx-operator-`. Typical reports: + +- A certificate was attached to a domain on the platform domain page, and services using that domain show a browser warning with the certificate `Kubernetes Ingress Controller Fake Certificate`, while another service whose Secret is in its own namespace works. +- An Ingress with its own TLS certificate still gets the controller's default certificate. +- Applications in one project cannot use an Ingress NGINX gateway from another project. +- An IngressClass limited with **Ingress Scope** is still selectable in other projects. +- After an Ingress was created, requests return `404` and the controller has no access log for them. +- Requests hang for about two minutes before they succeed, and the error log shows `connect() failed (110: Operation timed out) while connecting to upstream`. +- Saving an Ingress fails on ACP 4.0 when the Higress gateway is used. +- A GitLab Ingress with a `configuration-snippet` is rejected with `failed calling webhook "inject-label.cpaas.io": webhook returned response.patchType but not response.patch`. +- A gray-release (traffic-split) Ingress that repeats the host and path of the primary Ingress is rejected by the admission webhook of the platform component `metis`. + +## Root Cause + +### Cause 1: the domain default certificate option is used with Ingress NGINX + +A wildcard domain created under **Platform Management > Network Management > Domain** keeps its certificate Secret in `cpaas-system`. When an Ingress of class `ingress-nginx-operator-` selects the "domain default certificate" option, the generated YAML has `spec.tls: []` and only the annotation `alb.networking.cpaas.io/tls: =cpaas-system/`. That option, and the platform-assigned certificate in general, is designed for the ALB load balancer. Ingress NGINX lets an Ingress reference only a TLS Secret in the same namespace, so the binding fails and the controller serves its fake certificate (CN `Kubernetes Ingress Controller Fake Certificate`, O `Acme Co`). Plain `http://` access shows no certificate, which can hide the problem; the certificates themselves can be identical and valid. + +### Cause 2: several Ingress objects share one host and only one has TLS + +The Ingress with the dedicated certificate is correct (for example two host rules and a `tls` section listing both hosts with one `secretName`), but other Ingress objects use the same host names for other paths and are HTTP-only, without TLS. That conflicting configuration for the same host makes the controller's default certificate win for the domain. + +### Cause 3: the IngressNginx instance is scoped to other namespaces + +By default an Ingress NGINX instance handles Ingress resources in all namespaces that either name its IngressClass or name no IngressClass. `.spec.controller.scope.namespaceSelector` restricts it, for example `namespaceSelector: 'cpaas.io/project=demo'` limits it to namespaces labeled `cpaas.io/project=demo`. Ingress objects from other projects are then ignored. + +### Cause 4: the console lists a scoped IngressClass in every project + +The console does not filter the IngressClass list by project, so a class limited with **Ingress Scope** appears everywhere. The scope is enforced at runtime: Ingress objects from projects outside the scope are not served. + +### Cause 5: an invalid Ingress makes requests return 404 with no access log + +An invalid Ingress was added, for example one with `nginx.ingress.kubernetes.io/configuration-snippet: | gzip off;` next to `nginx.ingress.kubernetes.io/backend-protocol: HTTP` and `nginx.ingress.kubernetes.io/proxy-body-size: "0"`. Snippet annotations are disabled by default in newer Ingress NGINX versions (Cause 8). Removing the invalid Ingress is the fix. + +### Cause 6: requests wait about 131 seconds for a stale upstream, then succeed + +The controller error log shows: + +```text +[error] ... connect() failed (110: Operation timed out) while connecting to upstream ... request: "GET HTTP/1.1", upstream: "http://:80" +``` + +The access log line for the same request lists two upstreams, the old one on port 80 and the expected one on port 8080, with statuses `502, 200` and upstream times of about `131.015, 0.003`. + +An Endpoints object was applied by hand for a Service that has a selector (it carries `kubectl.kubernetes.io/last-applied-configuration`), and a Pod of another service had been associated with the Service by mistake, which produced an EndpointSlice owned by the Service with a `ready: true` endpoint whose `targetRef` is a Pod of a different service. `kube-controller-manager` logs: + +```text +"Error mirroring EndpointSlices for Endpoints, retrying" logger="endpointslice-mirroring-controller" key="/" err="error(s) deleting 1/1 EndpointSlices for / Endpoints, including: endpointslices.discovery.k8s.io \"-\" not found" +``` + +The hand-made Endpoints disrupt the native controller's takeover; its cleanup gets stuck on a slice that no longer exists, the stale slice stays, and the controller keeps trying the old upstream first. + +### Cause 7: saving an Ingress fails on 4.0 when the Higress gateway is used + +On ACP 4.0, the Ingress check webhook `check-ingress` of the `metis` Deployment in `cpaas-system` interferes with saving Ingress resources used by Higress. The same operation works on 3.16. + +### Cause 8: an Ingress with a configuration-snippet is rejected with an inject-label.cpaas.io webhook error + +Newer Ingress NGINX versions disable snippet annotations by default. A Helm-deployed GitLab whose `gitlab-webservice-default` Ingress gains a multi-line `nginx.ingress.kubernetes.io/configuration-snippet` (for WebSocket) is not served, although the controller log shows `"Found valid IngressClass"` for it, and admission fails with `failed calling webhook "inject-label.cpaas.io": webhook returned response.patchType but not response.patch`. + +### Cause 9: a gray-release Ingress with the same host and path as the primary Ingress is rejected by metis + +The `check-ingress` webhook of `metis` enforces a duplicate host and path check on Ingress resources. A gray-release Ingress (one that carries the ingress-nginx gray-release annotations, for example routing requests with the header `runenv: gray-ack` to the new version) shares host and path with the primary Ingress by design, so the check rejects it. + +From ACP 4.3, the `metis` Deployment carries `--disable-webhook=check-ingress` automatically. + +## Resolution + +### Cause 1: the domain default certificate option is used with Ingress NGINX + +Create the certificate Secret in the business namespace and reference it from the Ingress `spec.tls`; do not use the domain default certificate option with Ingress NGINX. + +```bash +kubectl -n create secret tls --cert= --key= +``` + +```yaml +spec: + ingressClassName: ingress-nginx-operator- + tls: + - hosts: + - app.example.com + secretName: +``` + +### Cause 2: several Ingress objects share one host and only one has TLS + +Change every Ingress that uses the domain to HTTPS and add the same TLS certificate to each of them. Find them with: + +```bash +kubectl get ingress -A -o json | jq -r '.items[] | select([.spec.rules[]?.host] | index("app.example.com")) | .metadata.namespace + "/" + .metadata.name + " tls=" + ((.spec.tls // []) | length | tostring)' +``` + +### Cause 3: the IngressNginx instance is scoped to other namespaces + +Check the scope of the instance. To serve Ingress objects from all projects, use an instance without `controller.scope.namespaceSelector`; otherwise adjust the selector to include the other project's namespaces. + +```bash +kubectl -n get ingressnginxes.ingress-nginx.alauda.io -o jsonpath='{.spec.controller.scope}{"\n"}' +``` + +```yaml +apiVersion: ingress-nginx.alauda.io/v1 +kind: IngressNginx +metadata: + name: +spec: + controller: + nodeSelector: + kubernetes.io/os: linux + replicaCount: 1 +``` + +### Cause 4: the console lists a scoped IngressClass in every project + +No change is needed for isolation; it works at runtime. Selecting the class from a project outside the scope does not get that Ingress served. + +### Cause 5: an invalid Ingress makes requests return 404 with no access log + +Find the invalid Ingress in the controller log or with `kubectl describe`, and remove it (or fix it, for example as in Cause 8 for snippets). + +> **Warning:** Deleting the Ingress removes its hosts from the controller. Do it only when the controller log or events name this Ingress as invalid, and save its YAML first. To roll back, apply the saved YAML after fixing it. + +```bash +kubectl -n logs deploy/-ingress-nginx-controller --since=30m | grep -iE 'error|invalid' | tail +kubectl -n get ingress -o yaml > /tmp/.yaml +kubectl -n delete ingress +``` + +### Cause 6: requests wait about 131 seconds for a stale upstream, then succeed + +Only when the access log shows the two-upstream `502, 200` pattern and an EndpointSlice of the Service lists an address that should not be there, delete that residual EndpointSlice. The requests then stop trying the old upstream. Avoid hand-made Endpoints for Services that have a selector. + +> **Warning:** Only delete a slice that contains the stale address. Deleting the wrong slice removes healthy endpoints until the controller recreates them. Save the slice YAML first; to roll back, apply it. + +```bash +kubectl -n get endpointslices -o wide +kubectl -n get endpoints -o yaml +kubectl -n kube-system logs --since=1h | grep -i 'mirroring EndpointSlices' +kubectl -n get endpointslices -o yaml > /tmp/.yaml +kubectl -n delete endpointslices +``` + +### Cause 7: saving an Ingress fails on 4.0 when the Higress gateway is used + +Add `check-ingress` to the `--disable-webhook` startup argument of the `metis` Deployment in `cpaas-system`, as in Cause 9. + +### Cause 8: an Ingress with a configuration-snippet is rejected with an inject-label.cpaas.io webhook error + +Enable snippet annotations and raise the allowed annotation risk level in the controller's global configuration (`spec.controller.config` of the IngressNginx resource): + +```yaml +spec: + controller: + config: + allow-snippet-annotations: "true" + annotations-risk-level: "Critical" +``` + +> **Warning:** Snippet annotations let anyone who can edit an Ingress inject raw nginx configuration into the shared controller, which affects every host it serves. Do this only when Ingress write access is limited to trusted users. To roll back, remove both keys from `spec.controller.config`. + +### Cause 9: a gray-release Ingress with the same host and path as the primary Ingress is rejected by metis + +Long-term fix: upgrade to ACP 4.3, where the argument is set automatically. On an earlier release, add `check-ingress` to the `--disable-webhook` argument of the `metis` Deployment. If the container already has a `--disable-webhook` argument, add `check-ingress` to it rather than adding a second flag. Only the Ingress duplicate host and path check is disabled. + +```bash +kubectl -n cpaas-system get deploy metis -o jsonpath='{.spec.template.spec.containers[*].args}{"\n"}' +``` + +> **Warning:** Only when the error comes from the `metis` webhook and, for a gray release, the rejected Ingress intentionally shares host and path with another Ingress. Impact: the platform's Ingress check is turned off for every Ingress in the cluster, so accidental duplicate hosts and paths are no longer caught at admission, and the edit rolls the `metis` Pods. Save the Deployment YAML first. To roll back, remove `check-ingress` from `--disable-webhook` again and wait for the rollout. + +```bash +kubectl -n cpaas-system get deploy metis -o yaml > /tmp/metis-deploy.before.yaml +kubectl -n cpaas-system edit deployments.apps metis +kubectl -n cpaas-system rollout status deploy/metis +``` + +```yaml +args: + - --disable-webhook=check-ingress +``` + +Then create or save the Ingress again. After an upgrade, read the arguments again to confirm the setting. + +## Diagnostic Steps + +1. Which certificate is served, and where the Ingress takes it from. `CN=Kubernetes Ingress Controller Fake Certificate` with an empty `spec.tls` is Cause 1; a dedicated `spec.tls` on one Ingress and HTTP-only Ingress objects for the same host is Cause 2: + + ```bash + openssl s_client -connect 192.0.2.10:443 -servername app.example.com /dev/null | openssl x509 -noout -subject -issuer + kubectl -n get ingress -o jsonpath='{.spec.tls}{" "}{.metadata.annotations}{"\n"}' + ``` + +2. Instances, their scope and classes (Causes 3 and 4): + + ```bash + kubectl get ingressnginxes.ingress-nginx.alauda.io -A + kubectl get ingressclass + ``` + +3. Upstream attempts per request (Cause 6). Two upstream addresses and two statuses in one access-log line mean a retry after a dead endpoint: + + ```bash + kubectl -n logs deploy/-ingress-nginx-controller --since=30m | grep 'Operation timed out' | tail -n 3 + kubectl -n get endpointslices -l kubernetes.io/service-name= -o yaml | grep -n -E 'name:|addresses' + ``` + +4. Admission failures on save show the webhook name in the API error (Causes 7, 8 and 9). List the webhooks and the `metis` arguments, and for a gray release confirm the duplicate host and path: + + ```bash + kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations | grep -iE 'ingress|inject-label|metis' + kubectl -n cpaas-system get deploy metis -o jsonpath='{.spec.template.spec.containers[*].args}{"\n"}' | grep -o 'disable-webhook=[^"]*' + kubectl -n get ingress -o custom-columns='NAME:.metadata.name,HOSTS:.spec.rules[*].host,PATHS:.spec.rules[*].http.paths[*].path' + ``` diff --git a/docs/en/solutions/Inspection_faults_x86-64-v2_CPU_error_discover-device_crash_loop_small_ClickHouse_limits_log_backlog_or_WebCLI_role_on_ACP.md b/docs/en/solutions/Inspection_faults_x86-64-v2_CPU_error_discover-device_crash_loop_small_ClickHouse_limits_log_backlog_or_WebCLI_role_on_ACP.md new file mode 100644 index 000000000..3d000fe50 --- /dev/null +++ b/docs/en/solutions/Inspection_faults_x86-64-v2_CPU_error_discover-device_crash_loop_small_ClickHouse_limits_log_backlog_or_WebCLI_role_on_ACP.md @@ -0,0 +1,175 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.16.x + - 4.2.x + - 4.3.x +--- + +# Inspection faults: x86-64-v2 CPU error, discover-device crash loop, small ClickHouse limits, log backlog, or WebCLI role on ACP + +## Issue + +A cluster inspection (for example the inspection taken before an upgrade) reports a component as abnormal, and the finding is a real fault rather than a false positive. One of these is on the report or in the pod log: + +- `olm-registry-platform` crashes with `Fatal glibc error: CPU does not support x86-64-v2`. +- Pods named `discover-device` in `nativestor-system` are in `CrashLoopBackOff` with very high restart counts. +- ClickHouse CPU on the global or a region cluster is above the 80% inspection threshold, and the log storage plugin is abnormal. +- Log search in the console returns nothing, while many business clusters write their logs into the global log storage. +- A custom platform-inspector role can do anything in WebCLI, including node login, after an upgrade from 3.16. + +For monitoring rows (a second Prometheus, a missing `vmagent-agent`, an OOMKilled `vmalert`, or a Prometheus stuck in `Init`), see [Monitoring faults: a second Prometheus, missing vmagent-agent, vmalert OOMKilled, or Prometheus stuck in Init on ACP](./Monitoring_faults_a_second_Prometheus_missing_vmagent-agent_vmalert_OOMKilled_or_Prometheus_stuck_in_Init_on_ACP.md). + +## Root Cause + +### Cause 1: Fatal glibc error: CPU does not support x86-64-v2 + +`olm-registry-platform` exits as soon as it starts, with `Fatal glibc error: CPU does not support x86-64-v2`. `/proc/cpuinfo` on that virtual machine has `sse4_2` but not `popcnt`. The image requires the x86-64-v2 instruction set level, so the guest CPU model must expose `popcnt` inside the VM. This appeared in a 4.2 inspection taken before a move to 4.3. + +### Cause 2: discover-device in nativestor-system is in CrashLoopBackOff and TopoLVM is not in use + +`discover-device` pods in `nativestor-system` are `CrashLoopBackOff` with restart counts in the tens of thousands. The workload is the device discovery used by TopoLVM local storage. On a cluster that does not use local storage, the DaemonSet has nothing to do and can be deleted. When TopoLVM volumes are in use, keep it; the crash then needs a storage investigation, not a delete. + +### Cause 3: ClickHouse is still at 2 CPU and 4Gi while the log storage plugin is abnormal + +The inspection says ClickHouse CPU on the global cluster and on a region cluster is above 80%, and the log storage plugin Alauda Container Platform Log Storage for ClickHouse is abnormal. The live limits of that ClickHouse are 2 CPU and 4Gi, which is too small for its load. The fix that matches this saturation is 4 CPU and 8Gi for both requests and limits. + +When `chi-cpaas-clickhouse` is close to 100% CPU and memory and several pods on that cluster are failing, adding resources to the cluster hosts resolves the pod errors. etcd leader changes and controller restarts flagged in the same report can be ignored once a current check shows that etcd no longer changes leader. + +### Cause 4: The platform cannot query logs because the global log Kafka is behind the business clusters + +Log search in the console returns nothing. Many business clusters write their logs into the global log storage at the same time. The global log storage consumes more slowly than the writers produce, so records pile up in the global log Kafka. After the backlog drains, log search works again. Kafka pod memory above 80% also appears on inspections by itself. The user-visible fault here is the lag, not the memory. + +### Cause 5: A custom platform-inspector role can operate WebCLI because its YAML contains acp-podsexec and acp-rootexec + +A role cloned from the platform auditor before an upgrade from 3.16 can suddenly use WebCLI for full operations, including node login. On 4.2, the built-in platform auditor has query-only WebCLI: no create, no delete, and no direct node login. The custom role YAML contains `acp-podsexec` and `acp-rootexec`. Removing those two entries, and leaving `get`, `list` and `watch`, restores query-only access. + +## Resolution + +### Cause 1: Fatal glibc error: CPU does not support x86-64-v2 + +1. Confirm the error: + + ```bash + kubectl get pods -A -o wide | grep olm-registry-platform + kubectl logs -n --tail=30 | grep x86-64-v2 + ``` + +2. On the virtual machine that runs the pod (a host shell, or `kubectl debug` on that node), check the CPU flags: + + ```bash + grep -m1 -o -w -E 'popcnt|sse4_2' /proc/cpuinfo || true + grep -c popcnt /proc/cpuinfo + ``` + + `sse4_2` without `popcnt` matches this cause. + +3. Change the guest CPU model. + + > **Warning:** Changing the guest CPU model usually needs a VM power cycle, which stops every pod on that node. Precondition: the node is cordoned and drained (`kubectl cordon ` and `kubectl drain --ignore-daemonsets --delete-emptydir-data`). Rollback: set the previous CPU model and power-cycle again. + + In the hypervisor, set a guest CPU model that exposes x86-64-v2, including `popcnt`. Power the VM off and on if the hypervisor applies the model only at boot. No image tag or Kubernetes flag bypasses this check. + +4. After boot, run `kubectl uncordon `. `grep popcnt /proc/cpuinfo` should print a match, and `olm-registry-platform` should stay up. + +### Cause 2: discover-device in nativestor-system is in CrashLoopBackOff and TopoLVM is not in use + +1. Confirm that the cluster has no TopoLVM volumes: + + ```bash + kubectl -n nativestor-system get ds,pods | grep discover-device + kubectl get sc + kubectl get pvc -A | grep -i topolvm || true + ``` + +2. Delete the DaemonSet. + + > **Warning:** Deleting the `discover-device` DaemonSet stops TopoLVM device discovery on every node. Precondition: the checks above show no TopoLVM volumes, nobody expects `nativestor` devices, and the DaemonSet YAML is saved. Skip this step when any PVC is TopoLVM-backed. Rollback: `kubectl apply -f /tmp/discover-device.before.yaml`, or reinstall the TopoLVM plugin if that YAML is incomplete. + + ```bash + kubectl -n nativestor-system get ds discover-device -o yaml > /tmp/discover-device.before.yaml + kubectl -n nativestor-system delete ds discover-device + ``` + +### Cause 3: ClickHouse is still at 2 CPU and 4Gi while the log storage plugin is abnormal + +1. Read the live requests and limits: + + ```bash + kubectl -n cpaas-system get pods | grep -i clickhouse + kubectl -n cpaas-system get pod -o jsonpath='{range .spec.containers[*]}{.name}{" req="}{.resources.requests.cpu}{"/"}{.resources.requests.memory}{" lim="}{.resources.limits.cpu}{"/"}{.resources.limits.memory}{"\n"}{end}' + ``` + +2. Raise the resources on the plugin. + + > **Warning:** Raising the limits restarts the ClickHouse pods, and log search and ingestion pause until they are Ready. Precondition: the live limits are 2 CPU and 4Gi, the log storage plugin is the abnormal one, and the nodes have the extra CPU and memory. Rollback: set the requests and limits back to 2 CPU and 4Gi on the plugin. + + In the console, open Alauda Container Platform Log Storage for ClickHouse, set CPU to 4 and memory to 8Gi for both requests and limits, and save. + +3. After the pods are Ready, watch the utilization and the plugin status. If the pods keep the old values, see [nevermore and ClickHouse log components alert on CPU or memory, restart, or ignore new resource values on ACP](./nevermore_and_ClickHouse_log_components_alert_on_CPU_or_memory_restart_or_ignore_new_resource_values_on_ACP.md). + +### Cause 4: The platform cannot query logs because the global log Kafka is behind the business clusters + +1. Confirm that the global log Kafka is lagging. Read the consumer group lag on the Grafana Kafka dashboards, or describe the groups inside `cpaas-kafka-0` with the listener and client properties that this cluster's brokers expose: + + ```bash + kubectl -n cpaas-system exec cpaas-kafka-0 -- \ + kafka-consumer-groups.sh --bootstrap-server \ + --command-config --describe --all-groups + ``` + +2. Let the backlog drain, and check that log search returns current entries once the lag is low. If the lag keeps growing, the consumer side is too slow for the writers. See [Log Kafka lag from Elasticsearch write hot spots, breaker trips, IK analyzer errors, or too few partitions on ACP](./Log_Kafka_lag_from_Elasticsearch_write_hot_spots_breaker_trips_IK_analyzer_errors_or_too_few_partitions_on_ACP.md). To reduce the load on the global cluster for good, store the business clusters' logs in log storage on those clusters. + +### Cause 5: A custom platform-inspector role can operate WebCLI because its YAML contains acp-podsexec and acp-rootexec + +1. Export the role and look for the two entries: + + ```bash + kubectl get -o yaml > .yaml + cp .yaml .before.yaml + grep -n -e acp-podsexec -e acp-rootexec .yaml + ``` + + Use the object kind that `kubectl get` shows for that role name. + +2. Remove the rules or resource names `acp-podsexec` and `acp-rootexec` from `.yaml`, and keep `get`, `list` and `watch`. The 4.2 platform auditor, which has query verbs only, is the reference. + +3. Apply the trimmed role. + + > **Warning:** Applying the trimmed role removes pod exec and node login from everyone bound to it, including any emergency use of that role. Precondition: you saved the previous YAML, and the extra verbs are the WebCLI access you want to remove. Rollback: `kubectl apply -f .before.yaml`. + + ```bash + kubectl apply -f .yaml + ``` + +## Diagnostic Steps + +1. Check the CPU flags, only on the node that runs `olm-registry-platform`: + + ```bash + kubectl logs -n --tail=50 | grep x86-64-v2 + grep -c popcnt /proc/cpuinfo + ``` + + The glibc line and a count of 0 are Cause 1. + +2. Check `nativestor-system` and the ClickHouse limits: + + ```bash + kubectl -n nativestor-system get pods + kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{range .spec.containers[*]}{.name}{"="}{.resources.limits.cpu}{"/"}{.resources.limits.memory}{" "}{end}{"\n"}{end}' | grep -i clickhouse + ``` + + A crash-looping `discover-device` with no TopoLVM volumes is Cause 2. Limits of `2` and `4Gi` with an abnormal log storage plugin are Cause 3. + +3. For empty log search, check the global Kafka lag before you treat it as a collector problem (Cause 4). Also check whether `nevermore` is `OOMKilled` (`kubectl get pods -A | grep nevermore`, then `kubectl describe pod` for `Reason: OOMKilled`). That is a collection problem: see [nevermore OOMKilled, cache written to /var/cpaas, or callback-secret mount failure on ACP](./nevermore_OOMKilled_cache_written_to_var_cpaas_or_callback-secret_mount_failure_on_ACP.md). + +4. For WebCLI access on a custom inspector, open the role YAML and search for `acp-podsexec` (Cause 5). + +## Related Information + +- Related articles: [Monitoring faults: a second Prometheus, missing vmagent-agent, vmalert OOMKilled, or Prometheus stuck in Init on ACP](./Monitoring_faults_a_second_Prometheus_missing_vmagent-agent_vmalert_OOMKilled_or_Prometheus_stuck_in_Init_on_ACP.md) covers the monitoring rows of the same inspection. +- [Logging ClickHouse runs out of disk, CPU or memory, or is OOMKilled on ACP](./Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md) diff --git a/docs/en/solutions/Inspection_reports_remaining_inodes_as_0_or_inode_and_file-handle_alerts_fire_on_ACP_nodes.md b/docs/en/solutions/Inspection_reports_remaining_inodes_as_0_or_inode_and_file-handle_alerts_fire_on_ACP_nodes.md new file mode 100644 index 000000000..bf07222c6 --- /dev/null +++ b/docs/en/solutions/Inspection_reports_remaining_inodes_as_0_or_inode_and_file-handle_alerts_fire_on_ACP_nodes.md @@ -0,0 +1,148 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Inspection reports remaining inodes as 0, or inode and file-handle alerts fire on ACP nodes + +## Issue + +On Alauda Container Platform one of the following is reported for some or all nodes of a cluster: + +- The platform inspection reports that the remaining inode count on the node is 0 (`节点内剩余inode数为0`, "the remaining inode count on the node is 0"), although the root filesystem still has inodes. +- An inode usage alert fires on some nodes. +- A file-handle usage alert fires, but the node looks fine. + +Only one of the three shapes below is a node that really cannot create files. `df -h` free bytes say nothing about inodes; compare `df -i` per mount with the number the alert or the inspection shows before cleaning anything. + +## Root Cause + +### Cause 1: df -i IUse% near 100% on the root or /var filesystem + +`df -h` still has free bytes, `df -i` on `/` or `/var` shows almost no free inodes, and the node condition mentions `nodefs.inodesFree` or `imagefs.inodesFree`. This is real inode exhaustion, caused by many tiny files, often under `/var/log`, a container log directory, or an overlay upper directory. When the kubelet `inodesFree` eviction threshold is crossed, the node also reports `DiskPressure=True`. + +### Cause 2: inspection remaining inode is 0, while df -i on / still has free inodes + +`df -i /boot/efi` is the exhausted number; `df -i /` and `df -i /var` are not. The inspection indicator `node.disk.inode.free`, defined in ConfigMap `indicators-standard-node` in `cpaas-system`, reads `node_filesystem_files_free` with `device!~"rootfs|tmpfs"` and no mount-point filter. The `/boot/efi` partition is typically vfat and reports no inode pool, so its 0 becomes the node's value. The alert is a false positive: `/boot/efi` is normally unused, the node can still create files, and scheduling is not affected. + +### Cause 3: inode or file-handle alert fires, but the node still has plenty of both + +`df -i` on the alarmed mount and `/proc/sys/fs/file-nr` show ample headroom, and kubelet reports no pressure condition. The inode and file-handle thresholds of the monitoring rules are low for that environment. No remediation is required on the nodes. + +## Resolution + +### Cause 1: df -i IUse% near 100% on the root or /var filesystem + +1. Find the directory with the most files. `du -x` stays on one filesystem; a `find` loop over `/*` also walks `/proc` and floods the output with errors such as `Too many levels of symbolic links`. + +```bash +df -i +du -x --inodes -d 1 | sort -n | tail +``` + +Repeat `du` on the largest directory until you reach the one where small files accumulate. + +2. Remove files you are allowed to lose (rotated logs, expired cache). Do not delete `/boot/efi` to fix an inode alarm. + +> **Warning:** Deleting a large set of small files is irreversible. Precondition: the directory is not a live etcd, Elasticsearch, or containerd data directory, and you have listed a sample of the names. Rollback: none. + +```bash +find -xdev -type f -name '' -print | head +find -xdev -type f -name '' -delete +df -i +``` + +Truncating a file frees bytes but not its inode, and a deleted file that a process still holds open keeps its inode until it is closed. When `df -i` is the number at 100%, the count drops only after closed files are removed. For container logs under `/var/log/pods`, let the runtime rotate them instead of removing directories it still has open. + +### Cause 2: inspection remaining inode is 0, while df -i on / still has free inodes + +1. Confirm the false positive. If `/` or `/var` is actually out of inodes, use Cause 1 instead. + +```bash +df -i / /var /boot/efi +``` + +2. On a control-plane node of the affected cluster, look at the indicator: + +```bash +kubectl -n cpaas-system get configmap indicators-standard-node -o yaml | grep -n -A3 'node.disk.inode.free' +``` + +If the indicator is not there, find the ConfigMap that holds the inode query (a search for `boot/efi` alone can miss it, because the query has no mount-point filter): + +```bash +kubectl get configmap -A -o json \ + | jq -r '.items[] | select((.data // {}) | tostring | test("node.disk.inode.free|node_filesystem_files_free")) | "\(.metadata.namespace)/\(.metadata.name)"' +``` + +3. Exclude `/boot/efi` from the query. + +> **Warning:** A bad edit can hide real filesystem alerts, not only `/boot/efi`. Precondition: `df -i /` shows free inodes, and the ConfigMap is saved. Rollback: `kubectl apply -f /tmp/indicators-standard-node.before.yaml`. Do not disable the whole inode item; that also hides a real `nodefs.inodesFree` problem. + +```bash +kubectl -n cpaas-system get configmap indicators-standard-node -o yaml > /tmp/indicators-standard-node.before.yaml +kubectl -n cpaas-system edit configmap indicators-standard-node +``` + +Search for `node.disk.inode.free` and change its `query` value from: + +```text +node_filesystem_files_free{node_name=~\"{{.name}}\",device!~\"rootfs|tmpfs\"} +``` + +to: + +```text +node_filesystem_files_free{node_name=~\"{{.name}}\",device!~\"rootfs|tmpfs\",mountpoint!~\"/boot/efi\"} +``` + +Keep the escaped quotes as they are inside the ConfigMap. If the query on your cluster has a positive mount-point list (`mountpoint=~...`) that contains `/boot/efi`, remove `/boot/efi` from that list instead. Make the change on every cluster that reports the row. + +4. Collection refreshes about every 15 minutes. To apply it sooner, restart `courier` in `cpaas-system`, then re-run the inspection and confirm the row is gone. + +> **Warning:** Restarting `courier` drops inspection and cluster-list calls until the new pod is Ready. Precondition: the ConfigMap edit is saved. Rollback: none needed; the controller recreates the pod. + +```bash +kubectl -n cpaas-system get deploy,pod | grep courier +kubectl -n cpaas-system rollout restart deploy/courier +``` + +If `courier` is not a Deployment, delete the pod that `kubectl -n cpaas-system get pod | grep courier` prints, so that its controller recreates it. + +`indicators-standard-node` is a platform object. Re-check the query after a platform upgrade, and include the change in your pre-upgrade review of platform customizations. + +### Cause 3: inode or file-handle alert fires, but the node still has plenty of both + +Confirm the headroom before you treat the alert as noise: + +```bash +df -i +cat /proc/sys/fs/file-nr +cat /proc/sys/fs/file-max +``` + +`file-nr` prints allocated handles, unused handles, and the maximum. If the allocated number is far below the maximum and `IUse%` is far below 100%, the alert threshold, not the node, is the issue, and no handling is required. If the alert is too noisy for the environment, review the threshold of that alert rule. When `IUse%` really is high, use Cause 1. + +## Diagnostic Steps + +1. Print inode usage for every mount the alert could mean, including `/boot/efi`: + +```bash +df -i / /var /var/lib /cpaas /boot/efi +kubectl describe node | grep -E 'DiskPressure|inodesFree' +``` + +2. Classify: + +- `/` or `/var` with no free inodes, possibly `DiskPressure`: Cause 1. Rank directories with `du -x --inodes -d 1`. +- Only `/boot/efi` shows 0 free inodes, and the inspection reports remaining inodes as 0: Cause 2. +- An inode or file-handle alert while `df -i` and `/proc/sys/fs/file-nr` show plenty of headroom and the node has no pressure condition: Cause 3. + +A node that is `DiskPressure` because bytes, not inodes, ran out is a different problem; use `df -h` and `du -x -d 1` to find the directory that filled the mount. diff --git a/docs/en/solutions/Inspection_reports_the_monitoring_component_abnormal_or_a_slow_collection_time_while_monitoring_works_on_ACP.md b/docs/en/solutions/Inspection_reports_the_monitoring_component_abnormal_or_a_slow_collection_time_while_monitoring_works_on_ACP.md new file mode 100644 index 000000000..9b1372087 --- /dev/null +++ b/docs/en/solutions/Inspection_reports_the_monitoring_component_abnormal_or_a_slow_collection_time_while_monitoring_works_on_ACP.md @@ -0,0 +1,115 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# Inspection reports the monitoring component abnormal or a slow collection time while monitoring works on ACP + +## Issue + +The platform inspection report marks a monitoring row as failed, but the cluster still serves metrics and the monitoring page works: + +- The basic inspection says the monitoring component or monitoring service is abnormal, while the Prometheus or VictoriaMetrics pods, the AppReleases, and the monitoring page are healthy. Prometheus returns `context deadline exceeded`, or `courier` logs a timeout while querying a metric. +- Pulling node-exporter metrics of a node times out. +- "asm-otel alert collection takes too long", or the average time to scrape `cpaas-system/cluster-enhancer-apiserver` (for example 20.28s against 10.00s) or to run monitoring queries (5.48s against 5.00s) is over the warning threshold. + +Before treating a red row as a false positive, rule out a real monitoring fault. If the API server log shows `apiserver was unable to write a JSON response: http: Handler timeout` on `GET /apis/monitoring.coreos.com/v1/prometheusrules` and an application team runs its own Prometheus in the cluster, the inspection data is genuinely wrong; see [Monitoring faults: a second Prometheus, missing vmagent-agent, vmalert OOMKilled, or Prometheus stuck in Init on ACP](Monitoring_faults_a_second_Prometheus_missing_vmagent-agent_vmalert_OOMKilled_or_Prometheus_stuck_in_Init_on_ACP.md). + +An inspection that fails with `bad_response: invalid character '<' looking for beginning of value`, with `courier` logging `Unauthorized` when it lists `clusters`, is Cause 2 of [Monitoring pages stay empty because courier-api or apollo holds a stale cluster cache on ACP](Monitoring_pages_stay_empty_because_courier-api_or_apollo_holds_a_stale_cluster_cache_on_ACP.md). + +## Root Cause + +### Cause 1: courier times out querying a monitoring target + +The inspection runs its monitoring checks through `courier`. When a query to the monitoring backend takes longer than the query timeout configured in the `monitoring` Feature, `courier` gives up with a timeout (`context deadline exceeded`), and the row says the monitoring component is abnormal although every monitoring pod and the monitoring page are fine. `ContainerStatusUnknown` pods in the same report are historical leftovers, not the failed hop. + +The item can be defined as `cpaas_cluster_prometheus_up` while the query `courier` actually runs differs: one run queried `kafka_consumergroup_lag_sum` on a business cluster that had no log storage plugin, so that series was absent, and the remaining failure was still the timeout. Raising the `monitoring` Feature timeout to 20 cleared the row. Restarting `apollo` and `courier` did not. When the row only matters as an upgrade gate, and `apollo` and the collectors log nothing, it can be recorded as a false positive; it did not block an upgrade. + +### Cause 2: node-exporter metrics take longer than the scrape timeout + +A manual request to the node-exporter metrics endpoint of the named node takes longer than the scrape limit (about 16s in one case), while the ServiceMonitor for node-exporter still has the default `scrapeTimeout` of `10s`. The node is up; the scrape is cut off. Apply this only after measuring the URL on the node the row names: another row, "average collection time 41s over 10s" for a business sample, was not fixed this way. + +### Cause 3: a collection or query time slightly above its threshold while the component is healthy + +The row names a collection or query time just above the threshold (10s for collection). For asm-otel, the pods were running without restarts, the logs had no related errors, and the time was only slightly above 10s; the warning had no impact. Rows for `cluster-enhancer-apiserver` at 20.28s and query time at 5.48s were assessed as metric request timeouts without a further cause. `cluster-enhancer-apiserver` is also the ServiceMonitor that collects `certificate_expires_time`; if that series count keeps growing, investigate it before treating a slow scrape there as harmless. + +## Resolution + +### Cause 1: courier times out querying a monitoring target + +Only when `courier` logs a timeout on a metric query, or Prometheus returns `context deadline exceeded`, and the monitoring pods and page are otherwise healthy. + +1. Save the Feature and find the timeout field: + + ```bash + kubectl get feature monitoring -o yaml > /tmp/feature-monitoring.before.yaml + kubectl get feature monitoring -o yaml | grep -n -i timeout + ``` + +2. Raise the timeout to 20 (`20s` if the field is a duration). Leave every other field as it is. + + > **Warning:** Editing `feature monitoring` changes how the console queries metrics; a wrong field can blank monitoring pages for every cluster that uses it. Precondition: the saved YAML is on disk, the timeout is confirmed in the `courier` log, and you change only the timeout that is below 20. Rollback: `kubectl apply -f /tmp/feature-monitoring.before.yaml`. + + ```bash + kubectl edit feature monitoring + ``` + +3. Re-run the inspection. If the row stays and `courier` now logs `Unauthorized` with an HTML body, follow the `bad_response` article linked in the Issue. + +### Cause 2: node-exporter metrics take longer than the scrape timeout + +1. Time the node-exporter metrics URL of the named node from a host that can reach it, and read the current timeout: + + ```bash + kubectl -n cpaas-system get servicemonitor | grep -i node + kubectl -n cpaas-system get servicemonitor -o yaml | grep -n -i -e scrapeTimeout -e interval + time curl -s -o /dev/null http://:/metrics + ``` + +2. If the measured time is above `scrapeTimeout`, raise it. + + > **Warning:** A longer scrape holds a scraper worker for up to 20s per node. Precondition: the measured time on that node is above the current timeout, and you saved the ServiceMonitor. Rollback: `kubectl apply -f /tmp/node-exporter-sm.before.yaml`. If a plugin sync restores `10s`, set the value through the monitoring plugin that owns the ServiceMonitor. + + ```bash + kubectl -n cpaas-system get servicemonitor -o yaml > /tmp/node-exporter-sm.before.yaml + kubectl -n cpaas-system edit servicemonitor + ``` + + Set `spec.endpoints[].scrapeTimeout` to `20s`; the generated Prometheus configuration then shows `scrape_timeout: 20s`. Where node scraping is defined only by a `VMNodeScrape` or `VMServiceScrape`, set that object's timeout to `20s` instead. + +### Cause 3: a collection or query time slightly above its threshold while the component is healthy + +Check the named component: + +```bash +kubectl get pods -A | grep asm-otel +kubectl -n logs --tail=200 | grep -iE 'error|timeout' +``` + +No restarts, no errors, and a time just above the threshold: close the row as no impact. A time well above the threshold on node-exporter is Cause 2. + +## Diagnostic Steps + +1. Separate a query timeout from an authorization failure: + + ```bash + kubectl -n cpaas-system logs deploy/courier --tail=200 | grep -Ei 'timeout|unauthorized|deadline|bad_response' + kubectl -n cpaas-system get pods | grep -Ei 'prometheus|vmagent|vmselect|vmstorage' + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | `context deadline exceeded` or a `courier` query timeout, monitoring pods healthy | 1 | + | `Unauthorized` on `clusters` plus `bad_response: invalid character '<'` | the `bad_response` article linked in the Issue | + | The named node's metrics URL answers slower than `scrapeTimeout` | 2 | + | A collection time just over the threshold on a healthy component | 3 | + +3. Both the monitoring pipeline row and the log pipeline row abnormal right after a primary/standby switchover, with healthy pods, is a different false positive: see [Inspection reports zero free inodes, missing event data, expired dex.tls, or abnormal pipelines after switchover on ACP](Inspection_reports_zero_free_inodes_missing_event_data_expired_dex.tls_or_abnormal_pipelines_after_switchover_on_ACP.md). diff --git a/docs/en/solutions/Inspection_reports_zero_free_inodes_missing_event_data_expired_dex.tls_or_abnormal_pipelines_after_switchover_on_ACP.md b/docs/en/solutions/Inspection_reports_zero_free_inodes_missing_event_data_expired_dex.tls_or_abnormal_pipelines_after_switchover_on_ACP.md new file mode 100644 index 000000000..6e5d2ad1f --- /dev/null +++ b/docs/en/solutions/Inspection_reports_zero_free_inodes_missing_event_data_expired_dex.tls_or_abnormal_pipelines_after_switchover_on_ACP.md @@ -0,0 +1,136 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# Inspection reports zero free inodes, missing event data, expired dex.tls, or abnormal pipelines after switchover on ACP + +## Issue + +The platform inspection report marks a node, the log pipeline, a certificate, or both pipeline rows as failed, but the cluster still serves logs, metrics, and logins. Typical report text: + +- `节点内剩余inode数为0个,低于预警阈值(5000个)` (remaining inodes on the node are 0, below the warning threshold of 5000), or the installation self-test warns about inodes while the nodes show 99% free inodes. +- A node has not sent `event` data (often `节点未发送 event 类型数据`, node has not sent event-type data), or `audit` data, for 15 minutes. Or the `日志链路组件状态` (log pipeline component status) row names `razor` with `日志未发送` (logs not sent), and the nodes it names are missing from the node list of the application log query page. +- `dex.tls` is reported expired, or another certificate has fewer than 30 days left or is named as `packageserver-service-cert` or `redis-operator-service-cert`, and a direct read of the Secret does not match. +- Inserting `audits` data into Elasticsearch is reported slow on a cluster that does not run `lanaya`. +- After a primary/standby switchover of the global cluster, both the monitoring pipeline row and the log pipeline row are abnormal (`监控组件(global/alert)健康状态不正常` and `监控组件(global/monitor)健康状态不正常`, monitoring component health abnormal), and the platform health page shows `平台监控功能故障,可能会影响集群 (global) 的功能健康检查结果的准确性,请尽快修复。` (platform monitoring is faulty, which may affect the accuracy of the health check results of cluster global), while the AppReleases and pods are all normal. + +Monitoring rows caused by query timeouts or slow scrapes are in [Inspection reports the monitoring component abnormal or a slow collection time while monitoring works on ACP](Inspection_reports_the_monitoring_component_abnormal_or_a_slow_collection_time_while_monitoring_works_on_ACP.md). + +## Root Cause + +### Cause 1: the inode count is 0 because /boot/efi (or another file system without inodes) is counted + +`df -i /boot/efi` is the empty pool, while `df -i /` and `df -i /var` have free inodes. The row is computed by the indicator `node.disk.inode.free` in the ConfigMap `indicators-standard-node` in `cpaas-system`, which reads `node_filesystem_files_free` with `device!~"rootfs|tmpfs"` and no mount-point filter, so the 0 of `/boot/efi` becomes the node's value. This is a known defect, and it does not mean the node cannot create files. The installation self-test has the same shape: it scans every device without excluding file system types, so file systems that have no inode counts report 0 remaining and fall below 5000. A later version is expected to exclude them; until then the warning can be ignored or the indicator corrected. A root or `/var` file system whose `df -i` is really exhausted is a real problem and not this cause. + +### Cause 2: the node produced no event, audit, or log data for 15 minutes + +The rule fires when no data of that type was sent from the node for 15 consecutive minutes. It means nothing of that type was produced on the node during the window, not that collection failed, and the same line can appear on every run. A "logs not sent" row naming `razor` came from nodes where system logs were visible and application logs were not, because nothing on those nodes wrote application logs; a node appears in the application log node list only after it has produced application logs. The log pipeline item checks the health of `nevermore`, `kafka`, and `lanaya`; a delay in those components is the other thing to rule out. + +### Cause 3: dex.tls is reported expired after a virtual machine restart while its dates are valid + +`openssl` on `cpaas-system/dex.tls` shows a validity period that still contains the current time. The report started after the virtual machine hosting the platform was restarted, earlier inspections had not flagged it, and later inspections returned to normal without a certificate replacement. If the dates really are in the past, this cause does not apply. + +### Cause 4: an audits-to-Elasticsearch latency row on a cluster without lanaya + +The cluster has no `lanaya` workload, so the row has nothing real to measure; it is a false alarm. Where the monitoring page or log search was really broken at the same time, the cause was elsewhere (for example a VictoriaMetrics OOM or Kafka lag). + +### Cause 5: a certificate row does not match the Secret it names + +The row says a certificate expires within 30 days, or names `packageserver-service-cert`, `redis-operator-service-cert`, or a short-lived `ca.crt`, while the Secret shows a much longer lifetime. Some certificates in these rows are renewed automatically by the platform 15 days before expiry, so the row needs no action. Replacing `dex.tls` or deleting a Certificate is wrong when the file is not the object the row measured. + +### Cause 6: warlock was not restarted after a primary/standby switchover + +The platform health page lists `vmselect-cluster` as `故障` (faulty) with 3/3 replicas and a reason such as `Service health metric is false (Error: health metric with zero value: cpaas_module_health{container="warlock", ... module="monitor", namespace="cpaas-system", ... service="warlock", ...} => 0 ...)`, while `courier` and `courier-api` are healthy. The `cpaas_module_health` value exported by `warlock` is stuck at 0 because `warlock` was not restarted after the switchover. In the switchover procedure this restart follows the restart of VictoriaMetrics and `cluster-transformer` on both global clusters, and it applies to every business cluster that sends monitoring data to the global VictoriaMetrics. + +## Resolution + +### Cause 1: the inode count is 0 because /boot/efi (or another file system without inodes) is counted + +Only when `df -i /` has free inodes and `/boot/efi` (or another file system with no inode count) is the 0. Edit the indicator on every cluster that shows the row. + +```bash +df -i / /var /boot/efi +kubectl -n cpaas-system get configmap indicators-standard-node -o yaml | grep -n -A3 'node.disk.inode.free' +``` + +> **Warning:** Excluding the wrong mount hides a real inode alert. Precondition: `df -i /` shows free inodes, and you saved the ConfigMap. Rollback: `kubectl apply -f /tmp/inode-configmap.before.yaml`. Exclude only `/boot/efi`; do not disable the whole inode item. + +```bash +kubectl -n cpaas-system get configmap indicators-standard-node -o yaml > /tmp/inode-configmap.before.yaml +kubectl -n cpaas-system edit configmap indicators-standard-node +``` + +In `node.disk.inode.free`, append `mountpoint!~\"/boot/efi\"` to the label set of `node_filesystem_files_free` when the expression has no `mountpoint` matcher. If it already has a positive `mountpoint=~` list containing `/boot/efi`, remove `/boot/efi` from that list instead. After the edit the query reads (quotes escaped as inside the ConfigMap; keep the other labels, such as the `vmcluster` value, as they are): + +```text +node_filesystem_files_free{vmcluster=\"region\",node_name=~\"{{.name}}\",device!~\"rootfs|tmpfs\",mountpoint!~\"/boot/efi\"} +``` + +The collector picks the change up in about 15 minutes. To apply it sooner, restart `courier`: + +> **Warning:** Restarting `courier` drops in-flight inspection queries until the new pod is Ready. Precondition: the ConfigMap edit is saved. Rollback: none needed; the Deployment recreates the pod. + +```bash +kubectl -n cpaas-system rollout restart deploy/courier +``` + +If the only goal is to stop the warning, the threshold can be changed in the inspection configuration; a lower threshold also hides a real exhaustion. + +### Cause 2: the node produced no event, audit, or log data for 15 minutes + +No object has to change. Treat the line as an ignorable prompt when the node is `Ready` and the 15-minute gap is the only finding. To stop it, disable the `日志链路组件状态` (log pipeline component status) item in the inspection configuration (`运维中心 → 巡检 → 巡检配置`, Operations Center → Inspection → Inspection Configuration) and watch `nevermore`, `kafka`, and `lanaya` through real-time alerts instead. Keep the item enabled if you want a real log pipeline failure to show up in the inspection; it covers more than the event sentence. + +### Cause 3: dex.tls is reported expired after a virtual machine restart while its dates are valid + +No replacement and no delete. After confirming the dates (Diagnostic Steps), wait for the next inspection. If the dates are past, follow [Replace the platform access certificate in dex.tls on ACP](Replace_the_platform_access_certificate_in_dex.tls_on_ACP.md) instead of waiting. + +### Cause 4: an audits-to-Elasticsearch latency row on a cluster without lanaya + +No change. Confirm that `lanaya` is absent and close the row as a false alarm. + +### Cause 5: a certificate row does not match the Secret it names + +Read the named Secret first (Diagnostic Steps). When it is still well inside its dates, the row does not match the file. For a certificate the platform renews automatically 15 days before expiry, check after that point that the dates moved forward. Renew only when `openssl` shows that named certificate inside the warning window, and only for that object: see [Platform certificate expiry alert for a cert-manager serving certificate that renews by itself on ACP](Platform_certificate_expiry_alert_for_a_cert_manager_serving_certificate_that_renews_by_itself_on_ACP.md), [Operator service certificates such as packageserver-service-cert about to expire or not renewed on ACP](Operator_service_certificates_such_as_packageserver-service-cert_about_to_expire_or_not_renewed_on_ACP.md), or [Expired platform component certificate Secret such as hco-webhook-service-cert not reissued on ACP](Expired_platform_component_certificate_Secret_such_as_hco-webhook-service-cert_not_reissued_on_ACP.md). + +### Cause 6: warlock was not restarted after a primary/standby switchover + +Only when a switchover happened, both pipeline rows went abnormal after it, and the pods and AppReleases are healthy. + +> **Warning:** Restarting `warlock` interrupts what it serves until the new pod is Ready. Precondition: the switchover is complete and VictoriaMetrics (`alertmanager`, `vmselect`, `vminsert`) and `cluster-transformer` were already restarted on the primary and standby global clusters. Rollback: none needed; the Deployment recreates the pod. + +Run on each affected cluster, including every business cluster that sends monitoring data to the global VictoriaMetrics: + +```bash +kubectl -n cpaas-system get pods -l service_name=warlock +kubectl delete po -n cpaas-system -l service_name=warlock +kubectl -n cpaas-system get pods -l service_name=warlock +``` + +If the earlier switchover steps were skipped, first run `kubectl delete po -n cpaas-system -l 'service_name in (alertmanager,vmselect,vminsert)'` and `kubectl delete po -n cpaas-system -l service_name=cluster-transformer` on both global clusters. Re-run the inspection, and make the `warlock` restart part of the switchover procedure. + +## Diagnostic Steps + +1. Inode row: compare mounts before editing anything. `/boot/efi` at 0 with `/` healthy is Cause 1. To list every mount that reports no inodes at all: `df -i -T | awk 'NR==1 || $3==0'`. `/` or `/var` at 0 is a real exhaustion, not this article. +2. Event or logs-not-sent row: "15 minutes" and the data type, repeating on every run, is Cause 2. On the log query page, switch to system logs for the named node; system logs present and no application pods on the node confirms it. If system logs are also missing, check the pipeline: + + ```bash + kubectl get pods -A -o wide | grep -E 'nevermore|lanaya|razor|kafka' | grep -v Running + ``` + +3. Certificate rows: + + ```bash + kubectl -n cpaas-system get secret dex.tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates -subject + kubectl -n get secret -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates + ``` + + Valid `dex.tls` dates after a VM restart is Cause 3. For a key named `ca.crt`, decode `.data['ca.crt']` the same way. A long lifetime against a "less than 30 days" row is Cause 5. A date that has passed is a real expiry. +4. Audits latency: `kubectl get pods -A | grep lanaya` returning nothing is Cause 4. +5. Both pipeline rows abnormal right after a switchover, `kubectl get ars -A` and the pods healthy, and the health page reason `health metric with zero value: cpaas_module_health{container="warlock", ...} => 0`, is Cause 6. diff --git a/docs/en/solutions/Integrated_product_link_fails_with_Redirect_to_untrusted_host_is_not_allowed_after_upgrading_to_ACP_4.3.3.md b/docs/en/solutions/Integrated_product_link_fails_with_Redirect_to_untrusted_host_is_not_allowed_after_upgrading_to_ACP_4.3.3.md new file mode 100644 index 000000000..8eae229e4 --- /dev/null +++ b/docs/en/solutions/Integrated_product_link_fails_with_Redirect_to_untrusted_host_is_not_allowed_after_upgrading_to_ACP_4.3.3.md @@ -0,0 +1,71 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.3.x +--- + +# Integrated product link fails with Redirect to untrusted host is not allowed after upgrading to ACP 4.3.3 + +## Issue + +After the platform is upgraded to ACP 4.3.3, an external product that was integrated into the console no longer opens from the platform. The product is listed under **Platform Settings** > **Product Configuration** as an integrated product, with its entry shown, single sign-on enabled and an external product link. + +Clicking the entry opens `/console/api/v2/token/redirect?redirect_url=`, and the page shows: + +```text +Redirect to untrusted host is not allowed +``` + +The same entry worked before the upgrade. + +## Root Cause + +In the new version, the console only performs token redirects to hosts that are explicitly allowed. The allowed hosts are listed under the `allowed-redirect-hosts` key of the ConfigMap `console-configmap` in the `cpaas-system` namespace of the `global` cluster. + +When the host of an integrated product link is not on that list, the console does not trust it and rejects the redirect. After the upgrade, products that were integrated earlier can be in this state. + +## Resolution + +1. On a control plane node of the `global` cluster, back up the ConfigMap and look at the current list: + + ```bash + kubectl -n cpaas-system get configmap console-configmap -o yaml > console-configmap.before.yaml + kubectl -n cpaas-system get configmap console-configmap -o jsonpath='{.data.allowed-redirect-hosts}{"\n"}' + ``` + +2. Edit the ConfigMap and add the host of every integrated product link (the host part of ``) to `allowed-redirect-hosts`. Follow the format of the existing entries, and leave the other keys, such as `enable-api-auth`, unchanged: + + ```bash + kubectl -n cpaas-system edit configmap console-configmap + ``` + +3. Restart the console frontend pods so that they load the new list. + + > **Warning:** While the frontend pods restart, the console can be briefly unavailable or show errors to logged-in users; workloads are not affected. Do this only after the `Redirect to untrusted host is not allowed` message is confirmed and the ConfigMap is updated. If the console misbehaves afterwards, restore the backup with `kubectl apply -f console-configmap.before.yaml` and restart the frontend pods again. + + ```bash + kubectl -n cpaas-system get deploy | grep -i frontend + kubectl -n cpaas-system rollout restart deploy + kubectl -n cpaas-system rollout status deploy + ``` + +4. Open the integrated product entry from the console again. It redirects to the product. + +## Diagnostic Steps + +1. Reproduce the failure in the browser and note the exact URL. The `/console/api/v2/token/redirect?redirect_url=...` path together with `Redirect to untrusted host is not allowed` identifies this issue. + +2. Compare the host of `redirect_url` with the allowed list. If the host is missing from `allowed-redirect-hosts`, this is the cause: + + ```bash + kubectl -n cpaas-system get configmap console-configmap -o jsonpath='{.data.allowed-redirect-hosts}{"\n"}' + ``` + +3. If the host is already listed but the error persists, check that the frontend pods were restarted after the change (their age is newer than the edit): + + ```bash + kubectl -n cpaas-system get pods | grep -i frontend + ``` diff --git a/docs/en/solutions/Intermittent_DNS_failures_in_Pods_from_a_new_node_NIC_flapping_the_KubeDNS_alert_or_CoreDNS_upstream_timeouts_on_ACP.md b/docs/en/solutions/Intermittent_DNS_failures_in_Pods_from_a_new_node_NIC_flapping_the_KubeDNS_alert_or_CoreDNS_upstream_timeouts_on_ACP.md new file mode 100644 index 000000000..6ad66e2a1 --- /dev/null +++ b/docs/en/solutions/Intermittent_DNS_failures_in_Pods_from_a_new_node_NIC_flapping_the_KubeDNS_alert_or_CoreDNS_upstream_timeouts_on_ACP.md @@ -0,0 +1,153 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# Intermittent DNS failures in Pods from a new node, NIC flapping, the KubeDNS alert or CoreDNS upstream timeouts on ACP + +Related articles: [CoreDNS fails after a node restart or lookups time out when a CoreDNS node is shut down on ACP](CoreDNS_fails_after_a_node_restart_or_lookups_time_out_when_a_CoreDNS_node_is_shut_down_on_ACP.md). + +## Issue + +Cluster DNS comes and goes, and the cause is outside the Corefile. Typical reports: + +- From a node that was just added, access to a Service by name works one moment and fails the next. Older nodes are fine. +- Many applications log `java.net.UnknownHostException` at the same time, together with lost heartbeats (`DiscoveryClient was unable to send heartbeat`), ZooKeeper `SessionExpiredException` and database `Communications link failure`. Node monitoring has gaps in the same window. +- An alert says `KubeDNS(:9153)` is unavailable, and it clears by itself almost immediately. +- On a global cluster whose CNI is flannel with galaxy, every console page returns `504` or an internal server error page (`服务器内部错误`), `GET /console-platform/api/v1/token/login` returns `500`, and one `tke-platform-api` replica in `cpaas-system` is in `CrashLoopBackOff`. + +CoreDNS on ACP 4.x is the `coredns` Deployment in `kube-system` (Pod label `k8s-app=kube-dns`). The CoreDNS Pods listen on `1053` inside the Pod network, use the node `/etc/resolv.conf` as their upstream list, and expose metrics on `9153`. + +## Root Cause + +### Cause 1: Service names fail intermittently only from a newly added node + +After a node is added, lookups of Service names from that node fail intermittently while older nodes are fine. CoreDNS has not yet recognized the newly added node; restarting CoreDNS restores normal access. If the new node cannot open `1053` to every CoreDNS Pod, the cause is a firewall between the node subnets instead (see Diagnostic Steps). + +### Cause 2: physical NICs flapping cause UnknownHostException bursts with connection drops + +When the physical NICs of the cluster machines go up and down, each flap interrupts DNS queries and every other connection that crosses the NIC. Several systems lose name resolution and connections in the same minutes, monitoring for the affected hosts shows gaps in CPU, memory and Pod count, and nodes drop out of the cluster count. The CNI and the operating system show nothing abnormal. + +A long node `search` list makes each interruption worse: with six suffixes on the node, several of them cluster-style, every short name is tried with each suffix in turn. Removing the `search` entries from the node `/etc/resolv.conf` stops the DNS failures, but connection drops continue until the NICs or links are repaired. + +### Cause 3: the KubeDNS :9153 unavailable alert clears by itself + +The alert names the CoreDNS metrics target (`:9153`). By the time anyone looks, it has cleared and CoreDNS is serving. A short network fluctuation can make a single query or scrape time out (the CoreDNS log for the window can show a request to port `53` timing out). A custom alert rule whose trigger timing does not match the platform's own alert behavior fires on a single missed scrape and reports every such blip. + +### Cause 4: CoreDNS cannot reach its upstream on a flannel and galaxy cluster + +One `tke-platform-api` replica in `cpaas-system` is `CrashLoopBackOff` (`0/1`, `Exit Code: 2`, more than 1,800 restarts) with events `Readiness probe failed: Get "https://:9443/healthz/ping": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)`. The CoreDNS log repeats: + +```text +[ERROR] plugin/errors: 2 HINFO: read udp :->:53: i/o timeout +``` + +The `HINFO` query is the probe CoreDNS sends at startup to detect forwarding loops, so its timeout means the CoreDNS Pod cannot reach the upstream through the Pod network. Node load is normal. The cluster network is flannel with galaxy (no `ovn` or `calico` Pods), and the `flannel` and `galaxy-daemonset` Pods in `kube-system` all show `1/1 Running`. Restarting the flannel and galaxy Pods restores the Pod network and platform access; what was wrong in the Pod network before the restart was not determined. + +## Resolution + +### Cause 1: Service names fail intermittently only from a newly added node + +Only when the failures started with the new node, other nodes resolve normally, and the `1053` check in Diagnostic Steps passes. Restart CoreDNS. + +> **Warning:** A rollout restart replaces the CoreDNS Pods one by one; lookups can stall briefly while a Pod is not Ready. Make sure more than one CoreDNS replica is Ready. No rollback is needed; the Deployment spec does not change. + +```bash +kubectl -n kube-system rollout restart deploy/coredns +kubectl -n kube-system rollout status deploy/coredns +kubectl exec -n -- nslookup ..svc.cluster.local +``` + +Run the last command several times; every attempt should answer. + +### Cause 2: physical NICs flapping cause UnknownHostException bursts with connection drops + +Only when kernel logs or the switch show the node NICs going up and down in the failure window. + +1. Repair the flapping NICs or links on the underlay. Redundant (dual-link) uplinks work as a stopgap until the repair. +2. As a DNS-side mitigation, comment out the `search` line in the node `/etc/resolv.conf` on all nodes, then recreate the affected Pods so their resolver file no longer carries those suffixes. If NetworkManager or DHCP writes the file, remove the search domains at that source too. + + > **Warning:** Removing the node `search` line changes how short names resolve for host processes and for every Pod created afterwards; names that relied on a site suffix stop expanding. Back up the file on each node and confirm the applications use full names for site domains. To roll back, restore the backup and recreate the Pods. + + ```bash + cp -a /etc/resolv.conf /root/resolv.conf.before + sed -i 's/^search /# search /' /etc/resolv.conf + grep -n search /etc/resolv.conf + ``` + +### Cause 3: the KubeDNS :9153 unavailable alert clears by itself + +If the CoreDNS Pods did not restart and the CoreDNS logs show no sustained errors in the window, treat the alert as a transient and give the custom rule a one-minute threshold, so that it fires only when the target stays down for one minute. In a Prometheus rule that is the `for` field. + +> **Warning:** A longer threshold delays real alerts by the same amount. Save the rule first and agree on the threshold with the alert owner. To roll back, apply the saved rule. + +```bash +kubectl get prometheusrule -A | grep -iE 'dns|coredns' +kubectl -n get prometheusrule -o yaml > /tmp/.before.yaml +kubectl -n edit prometheusrule +``` + +```yaml +- alert: + expr: + for: 1m +``` + +If the alert keeps firing with a one-minute threshold, it is no longer a timing artifact: collect CoreDNS logs and node network counters for the window and check Cause 2. + +### Cause 4: CoreDNS cannot reach its upstream on a flannel and galaxy cluster + +Only when the CoreDNS log shows the upstream `i/o timeout`, node resources are normal, and the cluster CNI is flannel with galaxy. Restart the flannel and galaxy Pods, then check CoreDNS and `tke-platform-api` again. + +> **Warning:** Restarting the CNI Pods interrupts Pod networking on each node while its Pods are recreated, and new Pods cannot get networking meanwhile. Keep console or SSH access to the masters that does not depend on the platform UI, and use a window in which a short Pod network interruption is acceptable. No rollback is needed; the DaemonSets recreate the Pods. + +```bash +kubectl get po -A -o wide | grep -E 'ovn|calico' +kubectl -n kube-system get ds | grep -E 'flannel|galaxy' +kubectl -n kube-system rollout restart ds/ ds/ +kubectl -n kube-system rollout status ds/ +kubectl -n kube-system rollout status ds/ +kubectl -n kube-system logs --tail=20 +kubectl -n cpaas-system get pod | grep tke-platform-api +``` + +After the restart the platform is accessible again. + +## Diagnostic Steps + +1. For a new node, query a Service FQDN repeatedly from a Pod on that node, and check that the node reaches every CoreDNS Pod on `1053`. Failures only from that node with `1053` open are Cause 1. `1053` closed to some replicas points at a firewall between node subnets; see [New node cannot reach CoreDNS or the API server because a firewall drops tunnel traffic between nodes on ACP](New_node_cannot_reach_CoreDNS_or_the_API_server_because_a_firewall_drops_tunnel_traffic_between_nodes_on_ACP.md). + + ```bash + for i in $(seq 1 20); do kubectl exec -n -- nslookup ..svc.cluster.local >/dev/null 2>&1 && echo ok || echo fail; done + kubectl -n kube-system get pod -l k8s-app=kube-dns -o wide + nc -vz -w 3 1053 + ``` + +2. For bursts of `UnknownHostException` with connection drops, look for link flaps and test Pod-to-Pod across nodes. A capture on the business node narrows which hop drops (Cause 2): + + ```bash + journalctl -k --since "" --until "" | grep -iE 'link (is )?(up|down)' + ip -s link show + kubectl exec -n -- ping -c 5 + tcpdump -i host and port -C 100 -W 5 -w /tmp/capture.pcap + ``` + +3. For the `:9153` alert, compare the alert time with CoreDNS restarts and logs, and read the rule that fired (Cause 3): + + ```bash + kubectl -n kube-system get pod -l k8s-app=kube-dns -o wide + kubectl -n kube-system logs -l k8s-app=kube-dns --since=1h | grep -iE 'timeout|error' + ``` + +4. For console errors with a crash-looping `tke-platform-api`, read the CoreDNS log and identify the CNI. `HINFO ... i/o timeout` on a flannel and galaxy cluster is Cause 4: + + ```bash + kubectl -n kube-system logs -l k8s-app=kube-dns --tail=50 | grep -E 'HINFO|i/o timeout' + kubectl -n kube-system get pod -o wide | grep -E 'flannel|galaxy|ovn|calico' + kubectl top nodes + ``` diff --git a/docs/en/solutions/Java_auto-instrumentation_init_container_is_missing_or_the_Java_agent_lands_next_to_istio-proxy_on_Service_Mesh_v2.md b/docs/en/solutions/Java_auto-instrumentation_init_container_is_missing_or_the_Java_agent_lands_next_to_istio-proxy_on_Service_Mesh_v2.md new file mode 100644 index 000000000..54f4dedde --- /dev/null +++ b/docs/en/solutions/Java_auto-instrumentation_init_container_is_missing_or_the_Java_agent_lands_next_to_istio-proxy_on_Service_Mesh_v2.md @@ -0,0 +1,163 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Service Mesh +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Java auto-instrumentation init container is missing or the Java agent lands next to istio-proxy on Service Mesh v2 + +## Issue + +Non-intrusive (automatic) tracing is configured for a Java application, but Jaeger shows no traces from that Pod. One of these is seen: + +- The Pod has no `opentelemetry-auto-instrumentation-java` init container. +- Both the Istio sidecar and the OpenTelemetry Java agent are injected, but `JAVA_TOOL_OPTIONS` is set on `istio-proxy` instead of the business container. +- Business logs carry no TraceID, so a log line cannot be matched to a trace in the Jaeger UI. +- The question is whether the platform's OpenTelemetry Operator can auto-instrument Nginx. + +The Alauda Build of OpenTelemetry v2 operator injects the agent through a pod-mutating admission webhook when the pod template carries an `instrumentation.opentelemetry.io/inject-java` annotation that points at an `Instrumentation` resource. + +Related articles: "asm-otel collectors keep restarting, Service Mesh JVM monitoring has no data, or an otel instance cannot be deleted" covers an agent that is injected but cannot reach the Collector, and missing JVM metrics. + +## Root Cause + +### Cause 1: the OpenTelemetry v2 integration was not completed in order + +The integration has an order: install Jaeger and the OpenTelemetry Collector, connect them to Elasticsearch or OpenSearch, create an `Instrumentation`, and only then annotate the Java workload. When any earlier step is missing, the webhook does not generate the `opentelemetry-auto-instrumentation-java` init container. Completing the whole sequence resolves it. + +### Cause 2: on Mesh v2 with Kubernetes earlier than 1.33, the agent targets the sidecar + +On Mesh v2 and Kubernetes versions earlier than 1.33, the business container that should receive the Java agent has to be named with an annotation; otherwise injection can apply to the Istio sidecar. From Kubernetes 1.33, sidecar init containers are supported and this does not happen. On Mesh v1, simultaneous injection of the Java agent and the Istio sidecar is supported. + +### Cause 3: the TraceID is created by the Java agent and has to be written into the log pattern + +Jaeger does not generate TraceIDs. The OpenTelemetry Java agent generates and propagates them. The Istio sidecar can generate a TraceID, but propagation across calls must then be handled by the application. The TraceID appears in business logs only when the log pattern prints it. + +### Cause 4: Nginx auto-instrumentation is enabled in `opentelemetry-operator2.v0.157.0-r1` + +The operator build `opentelemetry-operator2.v0.157.0-r1` sets `ENABLE_NGINX_AUTO_INSTRUMENTATION` to `"true"` (next to `ENABLE_GO_AUTO_INSTRUMENTATION: "true"`). The implementation is the upstream OpenTelemetry Operator Nginx auto-instrumentation of v0.157.0. + +## Resolution + +### Cause 1: the OpenTelemetry v2 integration was not completed in order + +1. Check each step and complete the first one that is missing: + 1. The OpenTelemetry v2 operator is installed and its webhook is registered. + 2. Jaeger and the Collector are deployed and connected to Elasticsearch or OpenSearch. Jaeger v2 is deployed by the OpenTelemetry v2 operator; a separate Alauda Build of Jaeger v2 install is not needed. + 3. An `Instrumentation` exists in the application namespace, or the annotation references one in another namespace as `/`. + 4. The pod template carries the injection annotation. + + ```bash + kubectl get csv -A | grep -i opentelemetry + kubectl get mutatingwebhookconfiguration | grep -i opentelemetry + kubectl get opentelemetrycollector -A + kubectl get pods -A | grep -i -E 'jaeger|otel|collector' + kubectl get instrumentation -n + ``` + +2. Create the `Instrumentation`, pointing at the Collector Service. Use the `.` form; a full name with a wrong cluster domain does not resolve. + + ```yaml + apiVersion: opentelemetry.io/v1alpha1 + kind: Instrumentation + metadata: + name: java-instrumentation + namespace: + spec: + exporter: + endpoint: http://.:4318 + propagators: + - tracecontext + - baggage + ``` + + ```bash + kubectl apply -f java-instrumentation.yaml + ``` + +3. Annotate the workload. + + > **Warning:** Patching the pod template restarts the workload's Pods. Only after steps 1.1 to 1.3 are in place. Rollback: set the annotation to `null` with the same patch, which restarts the Pods without the agent. + + ```bash + kubectl -n patch deployment --type=merge \ + -p '{"spec":{"template":{"metadata":{"annotations":{"instrumentation.opentelemetry.io/inject-java":"java-instrumentation"}}}}}' + kubectl -n get pod -o jsonpath='{.spec.initContainers[*].name}{"\n"}' + ``` + + The new Pod should list `opentelemetry-auto-instrumentation-java` among its init containers. + +### Cause 2: on Mesh v2 with Kubernetes earlier than 1.33, the agent targets the sidecar + +1. Check the Kubernetes version and the container names: + + ```bash + kubectl version + kubectl -n get deploy -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\n"}{end}' + ``` + +2. Below 1.33, name the business container with `instrumentation.opentelemetry.io/container-names`. + + > **Warning:** The Pods restart with the agent attached only to the named container. Only on Mesh v2 with Kubernetes earlier than 1.33 and both the sidecar and the agent injected. Rollback: set `instrumentation.opentelemetry.io/container-names` to `null`. + + ```bash + kubectl -n patch deployment --type=merge \ + -p '{"spec":{"template":{"metadata":{"annotations":{"instrumentation.opentelemetry.io/inject-java":"java-instrumentation","instrumentation.opentelemetry.io/container-names":""}}}}}' + ``` + + On Kubernetes 1.33 and later, and on Mesh v1, this annotation is not needed. + +### Cause 3: the TraceID is created by the Java agent and has to be written into the log pattern + +After the agent is injected, add the TraceID to the business log pattern. For Spring Boot logging: + +```properties +logging.pattern.level = trace_id=%mdc{trace_id} +``` + +Search the TraceID from a log line in the Jaeger UI to open the full call chain. For the agent's overhead, use the OpenTelemetry Java agent performance test report as the reference; no single figure applies to every application. + +### Cause 4: Nginx auto-instrumentation is enabled in `opentelemetry-operator2.v0.157.0-r1` + +1. Confirm the switch on the installed operator: + + ```bash + kubectl get csv -A | grep opentelemetry-operator2 + kubectl get csv -n -o yaml | grep -A1 -E 'ENABLE_NGINX_AUTO_INSTRUMENTATION|ENABLE_GO_AUTO_INSTRUMENTATION' + ``` + +2. Follow the upstream Nginx auto-instrumentation document for v0.157.0: the `Instrumentation` carries an `nginx` section (optionally `configFile` and `attrs`), and the workload opts in with `instrumentation.opentelemetry.io/inject-nginx: "true"`. Check the supported Nginx versions and the expected configuration layout (`/etc/nginx/nginx.conf` with a `conf.d` include in the `http` block, unless `configFile` says otherwise) against the image first. + + > **Warning:** The Pods restart with the Nginx module injected. Only when an `Instrumentation` with an `nginx` section exists in the namespace and the image meets the upstream requirements. Rollback: set the annotation to `null`. + + ```bash + kubectl -n patch deployment --type=merge \ + -p '{"spec":{"template":{"metadata":{"annotations":{"instrumentation.opentelemetry.io/inject-nginx":"true"}}}}}' + ``` + +## Diagnostic Steps + +1. Look at what was injected into a running Pod: + + ```bash + kubectl -n get pod -o jsonpath='{.metadata.annotations}{"\n"}{.spec.initContainers[*].name}{"\n"}' + kubectl -n get pod -o jsonpath='{range .spec.containers[*]}{.name}{" "}{.env[?(@.name=="JAVA_TOOL_OPTIONS")].value}{"\n"}{end}' + ``` + +2. Read the operator log for the Pod's admission: + + ```bash + kubectl -n logs --tail=200 | grep -i -E 'inject|instrumentation' + ``` + +| What you see | Cause | +| --- | --- | +| No `opentelemetry-auto-instrumentation-java` init container; Collector, Jaeger, storage, or `Instrumentation` missing | 1 | +| `JAVA_TOOL_OPTIONS` on `istio-proxy`, Kubernetes earlier than 1.33 | 2 | +| Agent injected, logs have no TraceID | 3 | +| Nginx auto-instrumentation wanted on `opentelemetry-operator2.v0.157.0-r1` | 4 | +| Agent injected into the business container, init container `terminated`, application log shows `Failed to export ... timeout` | The asm-otel article, Cause 4 | diff --git a/docs/en/solutions/Jenkins_Maven_build_fails_with_java.lang.OutOfMemoryError_Java_heap_space_although_build_node_resources_were_raised.md b/docs/en/solutions/Jenkins_Maven_build_fails_with_java.lang.OutOfMemoryError_Java_heap_space_although_build_node_resources_were_raised.md new file mode 100644 index 000000000..772b54ecc --- /dev/null +++ b/docs/en/solutions/Jenkins_Maven_build_fails_with_java.lang.OutOfMemoryError_Java_heap_space_although_build_node_resources_were_raised.md @@ -0,0 +1,89 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.0" + - "4.3" +--- + +# Jenkins Maven build fails with java.lang.OutOfMemoryError Java heap space although build node resources were raised + +## Issue + +A Java back-end project cannot be deployed by a Jenkins pipeline because the build reports that the heap is full, while front-end pipelines on the same platform deploy normally. Raising the build node resources did not help. Typically: + +- The `mvn-build` stage runs for about 15 minutes and then fails; the later `image-build` and `deploy` stages do not run. +- The stage log contains `The system is out of resources.` and `java.lang.OutOfMemoryError: Java heap space`, with a stack trace through `lombok` and `com.sun.tools.javac`, so the failure happens while the code is compiled. +- The build node resources were already raised, for example to request CPU `2` / memory `2Gi` and limit CPU `3` / memory `3Gi` on the `java` build node, and the error did not change. + +The platform default `java` build node uses the image `devops/builder-java:8`, the labels `java`, `java-openjdk8`, `maven`, `maven3.5`, and mounts the Maven cache PVC `maven-cache` at `/root/.m2/repository`. Build node resources are edited under **Toolchain Management** > **Instance Management** > the Jenkins instance > **Build Nodes** > edit the node. A project can have its own auto-assigned Jenkins instance, so the instance that runs the pipeline is not necessarily the one that was edited. + +## Root Cause + +### Cause 1: the Maven JVM heap is too small + +The Maven JVM does not have enough heap to compile and package the project. Neither Jenkins nor the size of the build container causes the failure, and a larger container limit does not, by itself, give Maven a larger heap. The JVM picks its default maximum heap when it starts, and an explicit `-Xmx` in `MAVEN_OPTS` overrides that default. Starting Maven with an explicit heap lets the pipeline deploy; in the reported case the build time also dropped from about 15 minutes to about 3 minutes. + +Raising the build node resources first does not help for two reasons: the heap does not change, and the edit may be made on a Jenkins instance that the pipeline does not use. + +## Resolution + +### Cause 1: the Maven JVM heap is too small + +Apply this only when the Maven stage log contains `java.lang.OutOfMemoryError: Java heap space`. A Pod that ends `OOMKilled` without that line has a container limit problem (see Diagnostic Steps). + +1. Set the Maven JVM heap on the packaging command in the pipeline's Maven stage: + + ```bash + MAVEN_OPTS="-Xmx2048m -Xms1024m -XX:MaxPermSize=512m" mvn clean package + ``` + + `-XX:MaxPermSize` applies only to Java 7 and earlier. A Java 8 JVM, such as the one in `builder-java:8`, prints a warning that it ignores the option and then runs. Java 17 and later JVMs can refuse to start with `Unrecognized VM option 'MaxPermSize=512m'`; on those build nodes (for example `java21`), drop that option and keep `-Xmx2048m -Xms1024m`. + +2. Keep `-Xmx` below the memory limit of the build container, with room for non-heap memory (metaspace, threads, and the compiler's native memory). A 2 GiB heap fits a `3Gi` limit. Against a node that still has a lower default limit, for example the `java21` build node default of request CPU `500m` / memory `750Mi` and limit CPU `1` / memory `2500Mi`, a 2 GiB heap leaves little room, so raise the limit as well. + + > **Warning:** Higher requests and limits on a build node apply to every pipeline that runs on that node, and the requests count against the namespace quota. The agent Pod can stay Pending if no node or quota can satisfy them. Precondition: you confirmed in Diagnostic Steps which Jenkins instance and build node the pipeline actually uses. Rollback: set the previous request and limit values back on the build node. + + Edit the build node on the Jenkins instance that the pipeline uses: **Toolchain Management** > **Instance Management** > that Jenkins instance > **Build Nodes** > the node (for example `java`) > change **Resource Requests** and **Resource Limits**. Two to three times the original values is a reasonable guideline. + +3. On a custom build node, `MAVEN_OPTS` can also be set in the node's **Environment Variables** field, so that every Maven run on it gets the heap. A per-project alternative is the standard Maven file `.mvn/jvm.config` in the repository. + +4. Run the pipeline again. The `mvn` stage finishes without `OutOfMemoryError`, followed by the image build and deploy stages. + +## Diagnostic Steps + +1. Confirm the error text in the exported pipeline log: + + ```bash + grep -nE 'OutOfMemoryError|The system is out of resources|GC overhead limit exceeded' + ``` + +2. While the pipeline runs again, find the agent Pod on the pipeline's cluster and check that it uses the instance and build node you edited: + + ```bash + kubectl get pod -A | grep java + kubectl describe pod -n + ``` + + The output shows the image (for example `devops/builder-java:8`) and the `Limits` and `Requests` of each container. If those values are not the ones you set, the pipeline runs on a different Jenkins instance or build node; edit the build node of the instance that serves the project. + +3. Check whether the JVM ran out of heap or the container was killed: + + ```bash + kubectl get pod -n -o jsonpath='{range .status.containerStatuses[*]}{.name}{" "}{.state.terminated.reason}{" "}{.lastState.terminated.reason}{"\n"}{end}' + ``` + + `OOMKilled` means the container limit is too small for the heap plus non-heap memory. `java.lang.OutOfMemoryError: Java heap space` in the log, without `OOMKilled`, is Cause 1. + +4. To see the default heap of the JVM in that container and what Maven receives, add these lines before `mvn` in the stage once: + + ```bash + echo "MAVEN_OPTS=${MAVEN_OPTS-unset}" + java -XX:+PrintFlagsFinal -version 2>/dev/null | grep -E ' MaxHeapSize ' + ``` + + A `MaxHeapSize` far below the size the project needs to compile, with `MAVEN_OPTS` unset, matches Cause 1. + +A Maven stage that fails on dependency resolution, or a green build whose image contains an old copy of a republished dependency, is a different problem; the latter is covered in "Jenkins Maven build packages an old dependency jar after the same version is republished to the private repository". diff --git a/docs/en/solutions/Jenkins_Maven_build_packages_an_old_dependency_jar_after_the_same_version_is_republished_to_the_private_repository.md b/docs/en/solutions/Jenkins_Maven_build_packages_an_old_dependency_jar_after_the_same_version_is_republished_to_the_private_repository.md new file mode 100644 index 000000000..011e0ec9b --- /dev/null +++ b/docs/en/solutions/Jenkins_Maven_build_packages_an_old_dependency_jar_after_the_same_version_is_republished_to_the_private_repository.md @@ -0,0 +1,107 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: [] +--- + +# Jenkins Maven build packages an old dependency jar after the same version is republished to the private repository + +## Issue + +A custom library is published to the private Maven repository again without raising its version number (a release version, not `-SNAPSHOT`). The Jenkins pipeline runs green and builds a new service image, but the library jar in the application's `lib` directory inside the image is still the earlier upload. The build did not fetch the republished jar from the repository, and the question is whether Jenkins caches the jar and how to replace the cached copy. + +Jenkins Java build nodes can keep the Maven local repository on persistent storage. The platform default `java` build node mounts the Maven cache PVC `maven-cache` at `/root/.m2/repository`, so artifacts downloaded by one build stay available to later builds. + +## Root Cause + +### Cause 1: the old copy of the republished release version is still in the persistent Maven cache + +The earlier jar is still in the Maven cache that the build node mounts. Because the republished jar kept the same version, the build reuses the cached copy and does not download the new one. Removing the old version directory from the cache makes later builds package the new upload. + +This matches standard Maven resolution. Maven looks for a release version in the local repository first, and once that version is present it is resolved locally. The `mvn --help` text for `-U, --update-snapshots` reads "Forces a check for missing releases and updated snapshots on remote repositories", so `-U` refreshes SNAPSHOT versions and fetches releases that are absent locally; a release that is already in the cache is used as it is. With a cache that persists across builds, the first copy of a release version stays in use until it is removed from the cache. + +Signature: the Maven stage log has no `Downloading from` / `Downloaded from` line for the republished artifact, and the jar in the cache directory is older than, and has a different checksum from, the one the repository serves now. + +## Resolution + +### Cause 1: the old copy of the republished release version is still in the persistent Maven cache + +Apply this only when Diagnostic Steps confirm all three: the repository now serves the new jar, the cached jar differs from it, and the build log shows no download of that artifact. + +1. Find the directory that holds the Maven cache. On the default `java` build node it is the volume behind PVC `maven-cache`, seen in the agent Pod at `/root/.m2/repository`. Locate its backing path: + + ```bash + kubectl get pvc -A | grep maven-cache + kubectl get pvc -n maven-cache -o jsonpath='{.spec.volumeName}{"\n"}' + kubectl get pv -o yaml + ``` + + In the PV, read `spec.hostPath.path`, `spec.local.path` together with `spec.nodeAffinity`, `spec.nfs`, or the CSI driver attributes to find the node or server and the directory. That directory is the repository root. A custom build node can mount a different claim; read its name from the agent Pod volumes (Diagnostic Steps, step 1). + +2. On that node or storage server, go to the version directory of the republished artifact. The group path is the `groupId` with `.` replaced by `/`: + + ```bash + REPO= + GAV_DIR="$REPO///" + ls -l --time-style=long-iso "$GAV_DIR" + ``` + + The jar date should match the earlier upload. + +3. Move the version directory out of the repository. + + > **Warning:** Removing the version directory makes the next build download that artifact again. If the private repository is unreachable from the build, or the version was removed from it, that build fails on dependency resolution. A build that runs on the same cache while you move the directory can fail or package a partial file. Precondition: only when Cause 1 is confirmed by Diagnostic Steps, the repository already serves the new jar, and no pipeline that uses this cache is running. Rollback: move the directory back (command below). Deleting the backup later is irreversible. + + ```bash + mkdir -p /root/maven-cache-backup + mv "$GAV_DIR" /root/maven-cache-backup/- + # rollback: + # mv /root/maven-cache-backup/- "$GAV_DIR" + ``` + +4. Run the pipeline again. The Maven stage log shows `Downloading from :` and `Downloaded from :` for `-.jar`, and the jar in the new image has the checksum of the new upload (Diagnostic Steps, step 4). Then remove the backup directory. + +Each further republish under the same version leaves the cache in the same state, so the cleanup has to be repeated. In general Maven practice, a library that changes without a version bump is published as a `-SNAPSHOT` version and the consuming build runs with `-U`, or each change gets a new release version. Running `-U` on every build adds update checks and time to the Maven stage. + +## Diagnostic Steps + +1. Confirm which cache the build uses. While the pipeline runs, describe the Java agent Pod and read the volume mounted at `/root/.m2/repository` and its claim name: + + ```bash + kubectl get pod -A | grep java + kubectl describe pod -n + ``` + +2. Check whether the build downloaded the artifact or resolved it locally. In the exported Maven stage log: + + ```bash + grep -nE 'Download(ing|ed) from .*-' + ``` + + No match means the jar came from the local repository, which is the Cause 1 path. + +3. Compare the cached jar with what the repository serves now. In the cache, or inside a running agent Pod at `/root/.m2/repository`: + + ```bash + ls -l --time-style=long-iso "$GAV_DIR" + sha1sum "$GAV_DIR/-.jar" + cat "$GAV_DIR/_remote.repositories" + ``` + + `_remote.repositories` names the repository id the jar was downloaded from. Ask that repository for the checksum of the current upload, using the URL from the build's `settings.xml`: + + ```bash + curl -sS -u "////-.jar.sha1"; echo + ``` + + Different checksums, with the cached file dated at the earlier upload, confirm Cause 1. If the repository still returns the checksum of the old jar, the republish has not reached the URL that the build resolves from (for example a group or proxy repository in front of the hosted one). Clearing the build cache would then download the same old file again, so fix the repository first. + +4. Check the jar that ended up in the image: + + ```bash + docker run --rm --entrypoint sh -c 'sha1sum /-.jar' + ``` + + If that checksum already equals the repository's new checksum, the stale jar is not coming from the Maven cache, and Cause 1 does not apply. diff --git a/docs/en/solutions/Jenkins_upgrade_stuck_admission_webhook_vjenkins.kb.io_rejects_a_Jenkins_CR_with_an_empty_spec.agent.namespace.md b/docs/en/solutions/Jenkins_upgrade_stuck_admission_webhook_vjenkins.kb.io_rejects_a_Jenkins_CR_with_an_empty_spec.agent.namespace.md new file mode 100644 index 000000000..cc591a464 --- /dev/null +++ b/docs/en/solutions/Jenkins_upgrade_stuck_admission_webhook_vjenkins.kb.io_rejects_a_Jenkins_CR_with_an_empty_spec.agent.namespace.md @@ -0,0 +1,90 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - 4.0.x +--- + +# Jenkins upgrade stuck: admission webhook vjenkins.kb.io rejects a Jenkins CR with an empty spec.agent.namespace + +## Issue + +During a toolchain upgrade (for example while the platform moves from 3.16 to 4.0), the Jenkins instance upgrade does not finish. The Jenkins operator controller log repeats a reconcile error of this form: + +```text +"msg"="Reconciler error" "error"="admission webhook \"vjenkins.kb.io\" denied the request: Jenkins.operator.devops.alauda.io \"\" is invalid: spec.agent.namespace: Invalid value: \"\": a lowercase RFC 1123 ... +``` + +The Jenkins Deployment itself still shows `1/1` ready, so Jenkins keeps serving; only the operator cannot apply the upgrade to it. + +## Root Cause + +### Cause 1: the Jenkins CR created by the older version has no spec.agent.namespace + +The new Jenkins operator installs a validating admission webhook, `vjenkins.kb.io`, that requires `spec.agent.namespace` of every `Jenkins` resource (`jenkins.operator.devops.alauda.io`) to be a valid lowercase RFC 1123 name. A Jenkins CR created by the older version never had this field, so it is empty. + +The webhook validates every update of the CR, not only updates made by users. When the operator reconciles the instance it writes its own management information back to the CR; that update is rejected because the field is empty, the reconcile fails, and the upgrade of the instance stops. The running Deployment is not touched, which is why Jenkins stays `1/1`. + +Setting `spec.agent.namespace` on the CR to the namespace where the build agents run makes the CR valid, and the operator can reconcile again. + +## Resolution + +### Cause 1 + +1. Find the Jenkins CR and confirm the field is empty (Diagnostic Steps, step 2). + +2. Decide the agent namespace. It is the namespace in which this Jenkins instance creates its build agent Pods. When the build agents already run in the same namespace as the Jenkins instance (for example `operators`), use that namespace. + +3. Set the field: + + ```bash + kubectl patch jenkins.operator.devops.alauda.io -n --type merge \ + -p '{"spec":{"agent":{"namespace":""}}}' + ``` + + For an instance named `devops-jenkins` in `operators` whose build agents run in `operators`: + + ```bash + kubectl patch jenkins.operator.devops.alauda.io devops-jenkins -n operators --type merge \ + -p '{"spec":{"agent":{"namespace":"operators"}}}' + ``` + + Once the CR is valid, the operator resumes the upgrade that was blocked, so the Jenkins instance can be updated and restarted by the operator right after the patch. Run it at a time when a Jenkins restart is acceptable. If the namespace is wrong, patch the field again with the correct namespace; an empty value is rejected by the webhook and is not a way back. + +4. Watch the operator log again (Diagnostic Steps, step 1). The `vjenkins.kb.io` denial no longer appears, and the toolchain upgrade continues. + +## Diagnostic Steps + +1. Find the Jenkins operator Pod and look for the webhook denial: + + ```bash + kubectl get pods -A | grep -i jenkins + kubectl logs -n --tail=200 | grep -i 'vjenkins.kb.io' + ``` + + A line containing `denied the request` and `spec.agent.namespace: Invalid value: ""` confirms this cause. + +2. Show the agent namespace of every Jenkins CR: + + ```bash + kubectl get jenkins.operator.devops.alauda.io -A \ + -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,AGENT_NS:.spec.agent.namespace' + ``` + + `` in `AGENT_NS` marks a CR that the webhook rejects. + +3. Confirm the webhook is installed: + + ```bash + kubectl get validatingwebhookconfiguration -o yaml | grep -n 'vjenkins.kb.io' + ``` + +4. Check that Jenkins itself is still running, which separates this cause from a Jenkins instance that is down: + + ```bash + kubectl get deploy -n + ``` + + The Jenkins Deployment shows `1/1`. If it is not ready, look at the Jenkins Pod events and logs first; this cause leaves the running instance untouched. diff --git a/docs/en/solutions/Kafka_2.x_instance_has_a_broker_offline_because_the_Alauda_Build_of_Kafka_2_operator_is_not_installed_on_ACP.md b/docs/en/solutions/Kafka_2.x_instance_has_a_broker_offline_because_the_Alauda_Build_of_Kafka_2_operator_is_not_installed_on_ACP.md new file mode 100644 index 000000000..50c607d5e --- /dev/null +++ b/docs/en/solutions/Kafka_2.x_instance_has_a_broker_offline_because_the_Alauda_Build_of_Kafka_2_operator_is_not_installed_on_ACP.md @@ -0,0 +1,59 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Application Services +ProductsVersion: [] +--- + +# Kafka 2.x instance has a broker offline because the Alauda Build of Kafka 2 operator is not installed on ACP + +## Issue + +One broker of a Data Services Kafka instance is reported offline ("a Kafka broker node is not online"). The instance runs Kafka 2.x (for example 2.7). The other brokers may still be running. + +This is an application-facing Kafka instance, not the platform log Kafka (`cpaas-kafka-*` in `cpaas-system`). For the platform log Kafka, see [Platform log Kafka brokers fail: no leader election, exit 137, Pending on DiskPressure, Ceph write errors, or OOM on ACP](Platform_log_Kafka_brokers_fail_no_leader_election_exit_137_Pending_on_DiskPressure_Ceph_write_errors_or_OOM_on_ACP.md). + +## Root Cause + +A Kafka 2.x instance is managed by the **Alauda Build of Kafka 2** operator. When that operator is not installed on the cluster, nothing reconciles the instance, so a broker that goes down is not brought back. + +Operator version `v4.0.3` of Alauda Build of Kafka 2 supports platform versions `v4.0` and `v4.1`, and Kafka `2.5`, `2.7`, and `2.8`. + +## Resolution + +1. Confirm the Kafka version of the instance and that the operator is missing (see Diagnostic Steps). +2. In the application marketplace (OperatorHub), check the Alauda Build of Kafka 2 operator versions and pick the one whose supported platform and Kafka versions match this cluster and this instance. +3. If the operator package is not yet on the platform, push it with the `violet` tool version that matches the operator package, not the `violet` version shipped with the platform. +4. Install the Alauda Build of Kafka 2 operator and wait until it reports success. +5. Restart the Kafka instance so the operator takes it over and brings the offline broker back. + + > **Warning:** Restarting the Kafka instance interrupts producers and consumers of that instance while its brokers restart. Precondition: the operator is installed and healthy, and the instance's applications can tolerate the interruption (do it in a maintenance window). Rollback: none needed for the restart itself; the operator recreates the broker pods. + + ```bash + kubectl -n get pods -o wide | grep + kubectl -n delete pod + ``` + + Restart the brokers one at a time and wait for each to be `Running` and ready before the next. + +6. Check that every broker of the instance is online. + +## Diagnostic Steps + +1. Find the instance, its namespace, and the broker that is offline: + + ```bash + kubectl get pods -A -o wide | grep -iE 'kafka|zookeeper' + kubectl -n logs -p --tail=50 + ``` + +2. Check whether the Kafka 2 operator is installed (no matching ClusterServiceVersion, or no operator pod, means it is missing): + + ```bash + kubectl get csv -A | grep -i kafka + kubectl get pods -A | grep -i kafka | grep -i operator + ``` + +3. Read the Kafka version of the instance from its custom resource or broker image, and compare it with the Kafka versions the available operator version supports. +4. If the operator is installed and healthy and a broker is still offline, the cause is elsewhere: check the broker pod events, its node, and its volume (for example a node that is `NotReady` or a volume that is full). diff --git a/docs/en/solutions/Kafka_KRaft_broker_and_controller_fail_with_No_space_left_on_device_expand_storage_through_the_RdsKafka_CR.md b/docs/en/solutions/Kafka_KRaft_broker_and_controller_fail_with_No_space_left_on_device_expand_storage_through_the_RdsKafka_CR.md new file mode 100644 index 000000000..1a4a02206 --- /dev/null +++ b/docs/en/solutions/Kafka_KRaft_broker_and_controller_fail_with_No_space_left_on_device_expand_storage_through_the_RdsKafka_CR.md @@ -0,0 +1,146 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda Application Services +ProductsVersion: + - "4.4" +--- + +# Kafka KRaft broker and controller fail with No space left on device: expand storage through the RdsKafka CR + +## Issue + +Kafka shows no data after a certain day and the platform logs stop updating. The Kafka broker Pod restarts because it exits during startup, and the KRaft controller Pods of the same instance fail as well. The previous broker log ends with: + +```text +ERROR [main] Kafka$:28 - Exiting Kafka due to fatal exception during startup. +java.nio.file.FileSystemException: /var/lib/kafka/data/kafka-log1/__cluster_metadata-0/00000000000000000000.log: No space left on device +[BrokerServer id=1] Transition from SHUTTING_DOWN to SHUTDOWN +App info kafka.server for 1 unregistered +``` + +The `kafka-log` directory and the `BrokerServer id` change with the broker. `__cluster_metadata` is the KRaft metadata log. A controller whose volume is full can report the same condition while it flushes that log, for example `KafkaStorageException: Error while flushing log for __cluster_metadata-0 in dir /var/lib/kafka/data/kafka-log ...` followed by `Caused by: java.io.IOException: No space left on device`. + +This applies to a Kafka instance managed by an `RdsKafka` custom resource in KRaft mode (`spec.mode: KRaft`, broker and controller Pods, no ZooKeeper). On ACP 4.4 the platform log Kafka can be deployed from the command line this way, for example as `log-kafka` in `cpaas-system`. In the ACP 4.4 `RdsKafka` API: + +- broker storage is `spec.storage.size`, and the KRaft controller storage is `spec.controller.storage.size`; each has its own PVCs; +- the broker data mount is `/var/lib/kafka/data` (log directory `kafka-log`); +- the Kafka operator creates the broker and controller Pods and their PVCs from that resource, so storage changes are made on the resource. + +## Root Cause + +### Cause 1: The broker and controller PVCs are full + +The broker cannot append to its KRaft metadata log (`__cluster_metadata-0/00000000000000000000.log`) because its PVC has no free space. It throws `FileSystemException ... No space left on device`, shuts down during startup, and unregisters. The KRaft controller volumes are full at the same time, so the controller containers fail too. With no broker running, nothing is written to or read from Kafka, which is why Kafka shows no data and the logs stop. + +Small controller volumes (for example `1Gi`) fill first. Expanding both sizes on the `RdsKafka` object (for example broker `50Gi` to `100Gi` and controller `1Gi` to `5Gi`) returns Kafka to normal. No data directory has to be deleted. + +## Resolution + +### Cause 1: The broker and controller PVCs are full + +Use this only when the broker's previous log shows `No space left on device` under `/var/lib/kafka/data` and the PVC or `df` confirms the volume is full. A full node-local `/cpaas/data/kafka0`, a hung storage path, or a volume that does not attach is a different fault; see Diagnostic Steps. + +1. Find the instance, its mode, and the two configured sizes: + +```bash +kubectl get rdskafka -A -o custom-columns="NAME:.metadata.name,NAMESPACE:.metadata.namespace,MODE:.spec.mode,PHASE:.status.phase,BROKER:.spec.storage.size,CONTROLLER:.spec.controller.storage.size" +``` + +If `rdskafka` is not a known resource name on the cluster, `kubectl api-resources | grep -i kafka` lists the kind. The resource type is one word, `rdskafka`; `kubectl edit rds kafka ...` does not work. + +2. List the broker and controller PVCs of the instance and confirm the StorageClass allows expansion: + +```bash +kubectl -n get pvc | grep +kubectl get sc -o jsonpath='{.allowVolumeExpansion}{"\n"}' +``` + +`true` is required. If the data sits on a node-local volume that cannot grow through the PVC, the logical volume on the node has to be extended instead; a PVC whose size changed while the directory still reports `No space left on device` is that case. + +3. Save the object, then raise both sizes on it. + +> **Warning:** Expanding storage on the `RdsKafka` object grows every broker PVC (`spec.storage.size`) or every controller PVC (`spec.controller.storage.size`) of the instance, and the operator may restart broker or controller Pods while it applies the change; log transport pauses until they are Ready. Run it only when the `No space left on device` signature is confirmed and the StorageClass reports `allowVolumeExpansion: true`, and the storage backend has the extra capacity for every replica. Expansion is one-way: Kubernetes does not accept a smaller request on an existing PVC, so there is no rollback to the old size; the saved YAML restores the other fields only. Choose the new size with retention and ingest rate in mind, and keep all brokers at the same size. + +```bash +kubectl -n get rdskafka -o yaml > /tmp/.before.yaml +kubectl -n edit rdskafka +``` + +```yaml +spec: + storage: + size: 100Gi # broker PVCs + controller: + storage: + size: 5Gi # KRaft controller PVCs +``` + +Or with one patch: + +```bash +kubectl -n patch rdskafka --type merge \ + -p '{"spec":{"storage":{"size":"100Gi"},"controller":{"storage":{"size":"5Gi"}}}}' +``` + +For the platform log Kafka the command is `kubectl -n cpaas-system edit rdskafka log-kafka`. The controller volume holds only metadata and needs far less than the brokers. Change the sizes on the `RdsKafka` object, not on the StatefulSet: Kubernetes rejects edits to `volumeClaimTemplates`. + +4. Wait until every PVC reports the new capacity and the Pods are Running: + +```bash +kubectl -n get pvc | grep +kubectl -n get pvc -o jsonpath='{.status.capacity.storage}{" "}{range .status.conditions[*]}{.type}{" "}{end}{"\n"}' +kubectl -n get pod | grep +``` + +A PVC that shows `FileSystemResizePending` has a larger volume but not yet a larger filesystem. With a CSI driver that supports only offline expansion, restart that one Pod so the volume is mounted again. + +5. Confirm the broker starts and has free space: + +```bash +kubectl -n exec -- df -h /var/lib/kafka/data +kubectl -n logs --tail=200 | grep -E 'No space left on device|Exiting Kafka|Transition from' +kubectl -n get rdskafka -o jsonpath='{.status.phase}{"\n"}' +``` + +Then check that new log data appears in Kafka and on the log query page. Once the brokers and controllers are Running, the consumers work through the backlog; query current logs after the lag has drained. If the broker volume fills again within days, review topic retention against the new size (see *Kafka log disk grows to hundreds of gigabytes or topic retention differs from the configured value on ACP*). + +## Diagnostic Steps + +1. Read the broker log from the failed start: + +```bash +kubectl get pod -A | grep -Ei 'kafka|controller' +kubectl -n logs --previous | grep -E 'No space left on device|Exiting Kafka due to fatal exception|KafkaStorageException' | tail -5 +``` + +`No space left on device` on a path under `/var/lib/kafka/data/kafka-log`, often on `__cluster_metadata-0`, with full PVCs owned by an `RdsKafka` resource, is Cause 1. + +2. Find each Pod's data mount and measure it. Print the mounts first, because controller Pods do not necessarily use the broker path: + +```bash +kubectl -n get pod -o jsonpath='{range .spec.containers[0].volumeMounts[*]}{.name}{" "}{.mountPath}{"\n"}{end}' +kubectl -n exec -- df -h +kubectl -n exec -- du -sh /var/lib/kafka/data/* +``` + +If the container restarts too quickly for `exec`, compare the PVC capacity with the sizes on the `RdsKafka` object and read the PVC usage in monitoring. + +3. Compare the `RdsKafka` sizes with what the PVCs actually have: + +```bash +kubectl -n get rdskafka -o jsonpath='{.spec.storage.size}{" "}{.spec.controller.storage.size}{"\n"}' +kubectl -n get pvc | grep +``` + +4. Rule out the neighbouring faults before expanding: + - A full node-local `/cpaas/data/kafka0`, or a host `/cpaas` at 100 percent: a node disk problem (see *Kafka and ZooKeeper down from storage stalls, a full node disk, or a probe kill during log replay on ACP*). + - The node kernel log shows iSCSI path loss, multipath `Failing path`, or EXT4 `Aborting journal`: the volume itself failed. + - The Pod reports `FailedAttachVolume`: the volume never attached. + +```bash +kubectl -n describe pod | grep -E 'FailedAttachVolume|FailedMount' +journalctl -k | grep -E 'Failing path|Aborting journal' +``` diff --git a/docs/en/solutions/Kafka_and_ZooKeeper_down_from_storage_stalls_a_full_node_disk_or_a_probe_kill_during_log_replay_on_ACP.md b/docs/en/solutions/Kafka_and_ZooKeeper_down_from_storage_stalls_a_full_node_disk_or_a_probe_kill_during_log_replay_on_ACP.md new file mode 100644 index 000000000..a4801bf98 --- /dev/null +++ b/docs/en/solutions/Kafka_and_ZooKeeper_down_from_storage_stalls_a_full_node_disk_or_a_probe_kill_during_log_replay_on_ACP.md @@ -0,0 +1,215 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda Application Services +ProductsVersion: + - "3.12" + - "3.14" + - "3.16" + - "3.18" + - "4.0" + - "4.1" + - "4.2" +--- + +# Kafka and ZooKeeper down from storage stalls, a full node disk, or a probe kill during log replay on ACP + +## Issue + +Applications lose Kafka, the Data Services page shows the Kafka instance as failed, or a create or reconcile error says: + +```text +Exceeded timeout of 300000ms while waiting for Pods resource -zookeeper- to be ready +``` + +Two instances can report that timeout at the same time. Broker Pods and ZooKeeper Pods restart together. On a KRaft-mode instance, the broker and the KRaft controller Pods both fail to come up instead, and the Kafka Pods keep restarting with no application change in that window. Check the KRaft controller Pods with the same disk and storage checks as ZooKeeper. + +Typical shapes of this failure: + +- A complete outage: ZooKeeper is gone, restarted ZooKeeper Pods cannot mount their storage, node load is in the hundreds, and Ceph raises alerts at the same time (Cause 1). +- On the log nodes, ZooKeeper and Kafka containers never start while `/cpaas` or `/cpaas/data` is at 100 percent, sometimes with the Elasticsearch API returning 503 (Cause 2). +- Only `kafka-0` fails: it is killed by its health check during startup after a JVM memory failure, with a large backlog of log segments to load (Cause 3). +- The log Kafka and Elasticsearch Pods were deleted or restarted while several nodes of the global cluster were `NotReady` (Cause 4). +- The broker PVC reports `FailedAttachVolume`, and Huawei CSI reports that `Add iscsi initiator to host` failed (Cause 5). +- The 300000 ms timeout names one ZooKeeper Pod that is not Ready, with no `OOMKilled` and no storage fault (Cause 6). + +When the report is only "Kafka is down", check Ceph and node iowait first, before reading Kafka logs or deleting any data. + +## Root Cause + +### Cause 1: Ceph OSD processes freeze and ZooKeeper misses syncLimit + +ZooKeeper is down and Kafka clients fail. Restarting ZooKeeper does not mount storage. Ceph alerts show the internal op processing of every OSD frozen in the same second, the RBD volumes behind the Kafka and ZooKeeper Pods hang, and the node load climbs very high (around 300). ZooKeeper followers must stay in sync with the leader within `syncLimit` ticks; with `syncLimit=2` and the default `tickTime` of 2000 ms that is a 4-second window, and a follower whose disk hangs exceeds it. Broker CPU is not the limiting factor. + +### Cause 2: The node directory that holds container data is full + +`/cpaas` or `/cpaas/data` on the node is at 100 percent. ZooKeeper does not start, Kafka does not start, and the Elasticsearch log API can return 503. Cleaning log files in the `/cpaas` directory of the node that runs ZooKeeper lets the containers start. The ZooKeeper data directory is not one of the files to clean. + +When the bytes are almost all `/cpaas/data/kafka0`, Kafka itself is holding them. Kafka deletes segments only when the retention time or size limit is reached, whether or not they were consumed, and the platform log topic keeps 48 hours by default. + +### Cause 3: kafka-0 is killed by the health check while replaying a large log + +The previous broker start failed because the JVM did not have enough memory. After the restart, the broker spends a long time loading accumulated segments, the health check kills the container before it becomes ready, and the cycle repeats. Stopping Kafka and ZooKeeper and deleting their data directories breaks the loop, at the cost of the topic data on those volumes. + +### Cause 4: Log Kafka and Elasticsearch were restarted because nodes were NotReady + +Several nodes of the global cluster are `NotReady`, and the log components on them are deleted or restarted. They come back by themselves once those nodes are `Ready` again. + +### Cause 5: Huawei CSI cannot attach the broker volume + +Broker Pods stay pending, or the broker Pod reports `FailedAttachVolume`. Huawei CSI returns a failure for `Add iscsi initiator to host` against a storage host object that already holds a large number of historical initiators in `offline` state. The count has reached the initiator limit of that storage host. The node IQNs themselves are unique; this is not a duplicate IQN registration. + +Other Kafka Pods can crash-loop in the same window because an application client certificate has expired. That is a separate fault and has to be fixed separately. + +### Cause 6: ZooKeeper Pod not Ready, no OOM, and a restart clears the timeout + +The 300000 ms error names one ZooKeeper Pod, that Pod is not Ready, and its events do not show `OOMKilled`. A manual restart of that Pod returns the instances to normal. The underlying reason has not been identified. + +## Resolution + +### Cause 1: Ceph OSD processes freeze and ZooKeeper misses syncLimit + +Confirm the signature before touching any broker disk: + +```bash +kubectl get pod -A | grep -Ei 'osd|ceph' +kubectl -n logs -zookeeper-0 --tail=300 | grep -Ei 'sync|fsync|timeout' +iostat -x 1 3 +``` + +> **Warning:** Restarting an OSD pauses I/O for every volume on that OSD, and rebooting a node restarts every Pod on it. Do this only when Ceph shows the OSD op freeze in the same second, the Kafka and ZooKeeper RBD volumes are hung, and ZooKeeper has missed `syncLimit`. Restart the frozen OSD Pods first, and reboot a node only if it stays in iowait after its OSDs are restarted. No Kafka data is touched by this step, so there is nothing to roll back on the Kafka side. + +```bash +kubectl -n delete pod +``` + +If the node load stays extreme and the RBD is still hung, reboot that node in a maintenance window. After the OSDs are healthy, ZooKeeper and the brokers mount their volumes again and clients reconnect. Leave the Kafka data directories in place. + +### Cause 2: The node directory that holds container data is full + +```bash +df -h /cpaas /cpaas/data +du -sh /cpaas/* /cpaas/data/* | sort -h +``` + +> **Warning:** Deleting files under `/cpaas` or `/cpaas/data` is permanent. Do it only when `df` shows that filesystem at 100 percent and ZooKeeper or Kafka fails to start because of it. Remove log files that are safe to drop, not the ZooKeeper data directory and not live Kafka volume contents. There is no undo for a deleted file. + +1. Clear the log files that `du` shows filling the filesystem until `df` reports free space. +2. The log collector queues under `/cpaas/audit`, `/cpaas/event`, and `/cpaas/log` hold logs that have not been shipped yet. Delete only inside their `diskqueue` directories, and only when the gap in collected logs is accepted. +3. Confirm the ZooKeeper and Kafka Pods reach Ready. If Kafka still logs `ZooKeeperClientTimeoutException` after space is freed, restart ZooKeeper first and then Kafka. +4. If Elasticsearch was also returning 503, recover Elasticsearch after the disk has free space; that is a log-storage step, not a Kafka topic operation. + +If the bytes are almost all `/cpaas/data/kafka0`: + +- Letting a lagging consumer catch up keeps records from being lost, but does not release space by itself. +- To get space back, expand the volume or lower retention (see *Kafka log disk grows to hundreds of gigabytes or topic retention differs from the configured value on ACP*). +- Do not delete segment files by hand on a running broker. +- Clearing `/cpaas/data/kafka0` entirely is only acceptable with Kafka, ZooKeeper, and Elasticsearch scaled to 0 and the owner accepting the loss of the whole log store, including the ZooKeeper data. + +If the full Kafka disk is a different disk from the Elasticsearch disk, deleting Elasticsearch indices does not help. + +### Cause 3: kafka-0 is killed by the health check while replaying a large log + +> **Warning:** This deletes the Kafka and ZooKeeper data directories and cannot be undone. Run it only when the previous container log shows the JVM ran out of memory, the current container is killed by the liveness or readiness probe while still loading segments, and the data owner has accepted losing the records stored in this Kafka cluster. A full disk (Cause 2) or an OSD stall (Cause 1) does not use this step. + +1. Tell a probe kill from an OOM kill with `kubectl describe pod`: a `Liveness probe failed` event versus an `OOMKilled` last state. Then print the previous log, the mounts, and the PVC names: + +```bash +kubectl -n logs -kafka-0 --previous | tail -100 +kubectl -n describe pod -kafka-0 | grep -E 'Liveness|Readiness|OOMKilled|Last State' +kubectl -n get pod -kafka-0 -o jsonpath='{range .spec.containers[0].volumeMounts[*]}{.name}{" "}{.mountPath}{"\n"}{end}' +kubectl -n get pvc +``` + +The broker data mount on a Data Services instance is `/var/lib/kafka/data`; the platform log Kafka (`cpaas-kafka-*` in `cpaas-system`) uses `/cpaas/data/kafka0`. + +2. Stop Kafka and ZooKeeper so nothing writes. The stop action in the console can leave Pods `Running`, so check `kubectl get pods` before removing files. The data lives on the PVC, so deleting a Pod does not remove it; the delete has to reach the mount. + +3. When the broker container stays up long enough: + +```bash +kubectl -n exec -kafka-0 -- ls -ld /var/lib/kafka/data +kubectl -n exec -kafka-0 -- sh -c 'rm -rf /var/lib/kafka/data/*' +``` + +Repeat for each ZooKeeper Pod, using the data mount printed for that Pod rather than the broker path. + +4. If the container restarts too quickly for `exec`, mount the claim (`data--kafka-`) on a one-off Pod that uses the broker image, remove the files under the data mount, and delete the one-off Pod. Do the same for the ZooKeeper volume only as part of the same accepted data loss. + +5. Start Kafka and ZooKeeper again. `kafka-0` starts normally once both data directories are empty. The JVM heap can be reviewed afterwards; no specific heap size is part of this recovery. + +### Cause 4: Log Kafka and Elasticsearch were restarted because nodes were NotReady + +```bash +kubectl get nodes +kubectl get pod -A | grep -E 'kafka|zookeeper|elasticsearch' +``` + +When the Kafka and Elasticsearch Pods were removed only because their nodes were `NotReady`, wait until those nodes are `Ready`. The log components return to normal with no further Pod deletion. A Kafka Pod that stays `Pending` on `DiskPressure` after its node is `Ready` is different: free the node disk, then delete that Pod so it is scheduled again. + +### Cause 5: Huawei CSI cannot attach the broker volume + +> **Warning:** Removing initiators from the storage host disconnects the matching node from that storage. Delete only initiators that are `offline` and that the storage owner has confirmed are unused. If a live initiator is removed by mistake, register it again and retry the attach. + +1. On the Huawei storage host mapped to the failing nodes, remove the historical `offline` initiators that pushed the host to its initiator limit. +2. Retry the attach: + +```bash +kubectl -n describe pod -kafka-0 | grep -A5 FailedAttachVolume +kubectl -n delete pod -kafka-0 +``` + +The new Pod should mount, `FailedAttachVolume` and the initiator error should stop, and the initiator of that node should show `online`. Also replace any expired application client certificate found in the same window; otherwise the Pod can be Running while the client still fails. + +### Cause 6: ZooKeeper Pod not Ready, no OOM, and a restart clears the timeout + +Use this only when the 300000 ms error names a ZooKeeper Pod, that Pod is not Ready, there is no `OOMKilled`, the node filesystem is not full, Ceph is not frozen, and the volume is attached. + +```bash +kubectl -n describe pod -zookeeper- | grep -E 'OOMKilled|State:|Ready' +kubectl -n get pods | grep zookeeper +``` + +> **Warning:** Deleting a ZooKeeper Pod restarts that ensemble member and briefly removes it from the quorum. Do it only when the other ZooKeeper Pods are Running and the signature above is confirmed. The StatefulSet recreates the Pod, and its data directory is kept. If the new Pod fails the same way, stop and check Causes 1, 2, and 5 instead of deleting data. + +```bash +kubectl -n delete pod -zookeeper- +kubectl -n rollout status statefulset/-zookeeper +``` + +## Diagnostic Steps + +1. Separate "Pod not Ready" from "client not authorized". A running broker that returns `TopicAuthorizationException` is a client authorization problem, not this article. + +2. Read the object named in the 300000 ms error. For a ZooKeeper Pod: + +```bash +kubectl -n describe pod -zookeeper- +kubectl -n get events --sort-by=.lastTimestamp | tail -40 +``` + +`FailedAttachVolume` is Cause 5. `FailedScheduling` or a disk-pressure taint points to Cause 2 or node disk pressure. Broker Pods that are rejected or never created while the namespace has a `LimitRange` or `ResourceQuota` are a quota problem. A container that starts and is then killed is Cause 3 only if the previous log shows a JVM memory failure during segment loading. + +3. Before any data-directory delete, check storage latency and free space. Causes 1 and 2 recover without removing Kafka segments. + +```bash +df -h +kubectl get pod -A | grep -Ei 'osd|ceph' +kubectl -n exec -kafka-0 -- df -h /var/lib/kafka/data +``` + +4. For a KRaft broker and controller crash loop, describe the controller Pod and its volume the same way and read the kernel log of its node. iSCSI `detected conn error (1022)`, `rejecting I/O to offline device`, multipath `Failing path`, and EXT4 `Aborting journal on device ...` or `Detected aborted journal` lines mean the volume behind the Pods failed; the service recovers once that volume problem is corrected, and no Kafka data has to be removed. + +```bash +journalctl -k | grep -E 'detected conn error|rejecting I/O to offline device|Failing path|Aborting journal|Detected aborted journal' +``` + +5. One broker restarting while its peers stay up, with that node's `kafka0` hundreds of gigabytes larger than the peers and an ISR that lists only that broker, is the leader-skew case in *Kafka log disk grows to hundreds of gigabytes or topic retention differs from the configured value on ACP*. Cause 3 here requires a JVM memory failure in the previous log and also deletes ZooKeeper data. + +6. For the platform log Kafka in `cpaas-system`, read the Kafka log before deleting anything. `read-only file system` on a `chmod`, `Corrupt time index found` with nothing listening on `9092`, or a snapshot restore timeout at startup are different faults and are not covered here. + +7. If the broker PVC is mounted and simply 100 percent full, expand it (see *Kafka broker PVC full or still too small after expansion on ACP*) instead of deleting the data directory. + +8. If the 300000 ms error names a ZooKeeper Pod, describe shows no `OOMKilled`, and Causes 1, 2, and 5 do not match, it is Cause 6: restart that Pod only. diff --git a/docs/en/solutions/Kafka_broker_PVC_full_or_still_too_small_after_expansion_on_ACP.md b/docs/en/solutions/Kafka_broker_PVC_full_or_still_too_small_after_expansion_on_ACP.md new file mode 100644 index 000000000..25fe2e798 --- /dev/null +++ b/docs/en/solutions/Kafka_broker_PVC_full_or_still_too_small_after_expansion_on_ACP.md @@ -0,0 +1,125 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda Application Services +ProductsVersion: + - "3.10" + - "3.12" + - "3.14" + - "3.16" + - "3.18" + - "4.0" + - "4.1" + - "4.2" +--- + +# Kafka broker PVC full or still too small after expansion on ACP + +## Issue + +A Kafka broker Pod is evicted or crash-loops, and its events or logs say the volume is full. Variants of the same problem: + +- A log Kafka volume fills its PVC (for example 30Gi) and the broker stops. The log-plugin Kafka that is created together with Elasticsearch is often requested at 20Gi; on a busy cluster that volume can reach about 18Gi in under two days. +- After someone expands the PVC, the broker still cannot create files under `/kafka`, or the Data Services page shows the instance abnormal while the broker Pod is `Running`. +- A self-managed Kafka (not created by the platform log plugin) fills `data-kafka-kafka-0`, with topic directories such as `ALAUDA_LOG_TOPIC-*` of several gigabytes each under `/var/lib/kafka/data/kafka-log0`. + +Related articles: *Kafka log disk grows to hundreds of gigabytes or topic retention differs from the configured value on ACP* (large disk use with little lag, retention, one broker holding all leaders), and *Kafka KRaft broker and controller fail with No space left on device: expand storage through the RdsKafka CR* (KRaft instances managed by `RdsKafka`). + +## Root Cause + +### Cause 1: The broker PVC has no free bytes + +The PVC used by one broker reaches 100 percent, and the broker cannot write. When the StorageClass allows volume expansion, expanding the PVC restores the broker. A single broker filling faster than its peers still needs a follow-up on which topics landed there; when that broker also holds almost every partition leader and the ISR lists only its id, see the leader-skew cause in the related retention article. + +### Cause 2: PVC capacity and the Kafka CR size disagree + +The PVC was expanded (for example from 30Gi to 60Gi), but the instance stays abnormal, or the broker still fails to create files under `/kafka`. `spec.kafka.storage.size` on the `Kafka` custom resource still holds the old value. The operator reconciles the broker storage from that field. Kubernetes refuses changes to `volumeClaimTemplates` on an existing StatefulSet, so a patch aimed at the broker StatefulSet does not update the size the operator uses. + +### Cause 3: A self-managed Kafka keeps historic log segments + +`data-kafka-kafka-0` is large (for example 250Gi requested and about 80 percent used), and `/var/lib/kafka/data/kafka-log0` holds `ALAUDA_LOG_TOPIC` partitions of several gigabytes each. The Kafka and the Elasticsearch next to it were deployed by the site, not by the platform log plugin, and historic Elasticsearch indices were still retained. Expanding the PVC brings the broker back; deleting the indices that the data owner has approved stops the growth. + +## Resolution + +### Cause 1: The broker PVC has no free bytes + +Use this only when `df` inside the broker, or the PVC metrics, show the Kafka volume at 100 percent, and the StorageClass has volume expansion enabled. + +```bash +kubectl -n get pvc +kubectl -n exec -kafka-0 -- df -h /var/lib/kafka/data +kubectl get sc -o jsonpath='{.allowVolumeExpansion}{"\n"}' +``` + +> **Warning:** A PVC can be expanded but never shrunk. Precondition: `allowVolumeExpansion` is `true`, the storage backend has the extra capacity, and you are expanding the PVC of the broker that is full. There is no rollback to the smaller size. + +```bash +kubectl -n patch pvc data--kafka-0 --type merge \ + -p '{"spec":{"resources":{"requests":{"storage":""}}}}' +kubectl -n get pvc data--kafka-0 -o jsonpath='{range .status.conditions[*]}{.type}{" "}{.status}{" "}{.message}{"\n"}{end}' +``` + +Wait until `status.capacity.storage` equals the request and `df` inside the Pod shows the new size. A condition `FileSystemResizePending` means the volume has grown but the filesystem has not yet: + +- With a CSI driver that supports online expansion, the kubelet finishes the resize while the Pod runs. +- With a driver that supports only offline expansion, the condition stays until the volume is mounted again; restart that one broker Pod. + +The log-plugin volume that starts at 20Gi uses the same patch. After the broker is up, check whether retention will fill the new size again (see the related retention article). If the Kafka CR size is still the old number once the PVC is larger, continue with Cause 2. + +### Cause 2: PVC capacity and the Kafka CR size disagree + +Compare the CR size with the PVC capacity: + +```bash +kubectl get kafka -A +kubectl -n get kafka -o jsonpath='{.spec.kafka.storage.size}{"\n"}' +kubectl -n get pvc data--kafka-0 -o jsonpath='{.status.capacity.storage}{"\n"}' +``` + +> **Warning:** Updating `spec.kafka.storage.size` restarts the Kafka instance. Run it in a maintenance window, and only when the PVC capacity is already the new size and the CR still shows the old size (instance abnormal while the Pod runs, or the broker cannot create files under `/kafka` after expansion). Set the field to the PVC's current capacity. Putting the old, smaller size back recreates the mismatch, so the only rollback is to keep the CR equal to the PVC capacity. + +```bash +kubectl -n patch kafka --type merge \ + -p '{"spec":{"kafka":{"storage":{"size":""}}}}' +kubectl -n rollout status statefulset/-kafka +kubectl -n get kafka +``` + +Do not patch `volumeClaimTemplates` on the StatefulSet; the API server rejects it, and the operator reads the size from the `Kafka` object. + +### Cause 3: A self-managed Kafka keeps historic log segments + +1. Expand the PVC with the Cause 1 procedure so the broker can start. +2. Remove only the Elasticsearch indices that the data owner has listed as disposable. Deleting Kafka topics is not part of this cleanup; the space is released on the Elasticsearch side after the volume is already larger. +3. Repeat `du` on `/var/lib/kafka/data/kafka-log0` after those indices are gone and the consumers have caught up: + +```bash +kubectl -n exec -- sh -c 'du -sh /var/lib/kafka/data/kafka-log0/* | sort -h | tail -20' +``` + +## Diagnostic Steps + +1. Decide which disk is full. The log-plugin data on the node is `/cpaas/data/kafka0`; a broker PVC is mounted at `/var/lib/kafka/data`. A full host `/cpaas` or `/cpaas/data` that keeps ZooKeeper down is a node disk problem (see *Kafka and ZooKeeper down from storage stalls, a full node disk, or a probe kill during log replay on ACP*), even when the files are Kafka logs. + +2. Compare four numbers: the PVC requested size, the PVC capacity, the filesystem size inside the Pod, and `spec.kafka.storage.size`. For a KRaft `RdsKafka`, the sizes are `spec.storage.size` and `spec.controller.storage.size` (see the related KRaft article). + +```bash +kubectl -n get pvc data--kafka-0 -o yaml +kubectl -n get kafka -o jsonpath='{.spec.kafka.storage.size}{"\n"}' +kubectl -n exec -kafka-0 -- df -h /var/lib/kafka/data +``` + +Expected for Cause 2: PVC capacity is the new size and `spec.kafka.storage.size` is still the old one. + +3. If the Pod never stays up long enough for `df`, read the previous log and the PVC events: + +```bash +kubectl -n logs -kafka-0 --previous +kubectl -n describe pvc data--kafka-0 +``` + +4. Check consumer lag before deleting anything. Lag near zero with a large directory is retention, not a fault (see the related retention article). + +5. `FailedAttachVolume` on the broker PVC is not a full disk; the volume never attached. See the Huawei CSI initiator cause in *Kafka and ZooKeeper down from storage stalls, a full node disk, or a probe kill during log replay on ACP*. diff --git a/docs/en/solutions/Kafka_broker_memory_looks_high_or_a_memory_change_or_new_broker_Pod_is_blocked_by_quota_on_ACP.md b/docs/en/solutions/Kafka_broker_memory_looks_high_or_a_memory_change_or_new_broker_Pod_is_blocked_by_quota_on_ACP.md new file mode 100644 index 000000000..4f799919c --- /dev/null +++ b/docs/en/solutions/Kafka_broker_memory_looks_high_or_a_memory_change_or_new_broker_Pod_is_blocked_by_quota_on_ACP.md @@ -0,0 +1,159 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda Application Services +ProductsVersion: + - "3.12" + - "3.14" + - "3.16" + - "3.18" + - "4.0" + - "4.1" + - "4.2" +--- + +# Kafka broker memory looks high, or a memory change or new broker Pod is blocked by quota on ACP + +## Issue + +On a running Data Services Kafka instance: + +- An inspection reports that a Kafka broker pod uses more than 80 percent of its memory, and someone asks whether that is a fault. +- Kafka memory usage climbs to about 90 percent no matter how much memory it is given. +- Updating the node spec (memory) of the instance fails, and rolling the change back fails too. The `RdsKafka` events show `ErrorKafkaStatus: Error while waiting for restarted pod -kafka-0 to become ready`, and the StatefulSet reports: + + ```text + pods "-kafka-0" is forbidden: exceeded quota: default, requested: limits.memory=8Gi,requests.memory=8Gi, used: limits.memory=28800Mi,requests.memory=28800Mi, limited: limits.memory=32Gi,requests.memory=32Gi + ``` + +- The instance shows abnormal, its `Kafka` resource is `NotReady` with `Exceeded timeout of 300000ms while waiting for Pods resource -kafka-0 in namespace to be ready`, and clients log `Timed out waiting for a node assignment. Call: fetchMetadata`. + +Related articles: [Kafka config change does not roll brokers, a broker restart hangs, or the instance stays processing on ACP](Kafka_config_change_does_not_roll_brokers_a_broker_restart_hangs_or_the_instance_stays_processing_on_ACP.md) covers rollouts and restarts that stall for reasons other than quota. Create-time failures are covered in [Kafka instance creation times out on the entity-operator, shows failed, or the cluster operator restarts on ACP](Kafka_instance_creation_times_out_on_the_entity-operator_shows_failed_or_the_cluster_operator_restarts_on_ACP.md). + +## Root Cause + +### Cause 1: Inspection flags broker memory above 80 percent + +A routine inspection reports broker memory above 80 percent of the limit (for example 87 percent). The pod is not restarting and is not `OOMKilled`. That level is normal for Kafka, not a fault, and no leak is involved. If the inspection threshold must be met, the broker memory limit is raised. + +### Cause 2: Memory usage reaches about 90 percent whatever the limit is (page cache) + +After the memory is expanded, usage again climbs to about 90 percent, with no message backlog behind it. This is expected: Kafka relies on the operating system page cache for throughput, and that cache grows into whatever memory the container is given, so the percentage stays high after every increase. No action is needed as long as the pods are not restarting because of OOM. + +### Cause 3: Spec update and rollback both fail with exceeded quota + +A memory change makes the operator stop a broker (`Killing: Stopping container kafka`) and recreate it with the new request. The StatefulSet `-kafka` then reports `FailedCreate` with the quota error, because the namespace `ResourceQuota` `default` is nearly full: in the example, 28800Mi of 32Gi is used by the other pods, so neither the new 8Gi request nor the old 4Gi request (more than the 3968Mi left) can be admitted for the pod being recreated. That is why the rollback fails the same way. The project quota on that cluster can also be fully allocated, which prevents raising the namespace quota. + +### Cause 4: The namespace LimitRange and ResourceQuota leave the broker pods without resources + +The cluster operator log repeats `Exceeded timeout of 300000ms while waiting for Pods resource -kafka- ... to be ready`, `createOrUpdate failed io.strimzi.operator.common.TimeoutException`, and `Failed to reconcile`. Clients log `Error connecting to node -kafka-2.-kafka-brokers..svc:9091` and `Timed out waiting for a node assignment. Call: fetchMetadata`. The broker pods of the Kafka `StrimziPodSet` did not come up because the namespace container limits (`LimitRange`) and `ResourceQuota` did not leave them enough resources to start (for example Kafka 3.8.0 in ZooKeeper mode with 3 brokers at CPU `500m` and memory `10Gi`, limits equal to requests). + +## Resolution + +### Cause 1: Inspection flags broker memory above 80 percent + +```bash +kubectl -n top pod -kafka-0 --containers +kubectl -n describe pod -kafka-0 | grep -E 'Last State|Reason|Restart Count' +kubectl -n get kafka -o jsonpath='{.spec.kafka.resources}{"\n"}{.metadata.ownerReferences}{"\n"}' +``` + +If the pod is not `OOMKilled` and not restarting, the usage is normal. To bring it under the inspection threshold, raise the broker memory. When `ownerReferences` names an `RdsKafka`, change the memory on the Data Services instance (its update page or its spec) so a direct edit of `Kafka` is not reverted. + +> **Warning:** Raising the broker memory rolls the broker pods one by one. Do it in a maintenance window, only when the flagged pod is a broker and it is not crash-looping. First confirm that the namespace quota has room for the larger request (Cause 3). Roll back by restoring the previous memory values. + +```bash +kubectl -n get kafka -o yaml > kafka.bak.yaml +kubectl -n edit kafka # only when the Kafka resource has no RdsKafka owner +``` + +Under `spec.kafka.resources`, raise `requests.memory` and `limits.memory` above the current limit and leave CPU unchanged. If usage climbs back to the same percentage afterwards, that is Cause 2. Growth in `-entity-operator` is an entity-operator issue, covered in the creation article linked in the Issue section. + +### Cause 2: Memory usage reaches about 90 percent whatever the limit is (page cache) + +```bash +kubectl -n get pods | grep -kafka +kubectl -n describe pod -kafka-0 | grep -E 'Last State|Reason|Restart Count' +kubectl -n top pod -kafka-0 --containers +``` + +If the restart count is not growing and the last state is not `OOMKilled`, the usage is page cache and needs no action; more memory does not lower the percentage. If the broker restarts with `OOMKilled`, the limit really is too small: raise it as in Cause 1. + +### Cause 3: Spec update and rollback both fail with exceeded quota + +```bash +kubectl -n get events --sort-by=.lastTimestamp | grep -E 'FailedCreate|ErrorKafkaStatus' | tail -20 +kubectl -n describe sts -kafka | grep -A3 FailedCreate +kubectl -n describe resourcequota default +``` + +The `FailedCreate` text names the quota, the request, the usage, and the limit. A rollback does not help when the hard limit minus `used` is below the old request as well. Check the project quota of that cluster on the project page. + +> **Warning:** Raising the project quota, and then the namespace quota `default`, lets the project and namespace schedule more CPU, memory, or storage than before. Do it only when `FailedCreate` names `exceeded quota` for a broker pod of this instance. Note the current values first; roll back by setting them again once the instance is stable. + +1. Raise the project quota for that cluster if it is fully allocated. +2. Make sure the hard limit minus `used` of the namespace quota `default` covers the broker request; raise it within the project quota if not. +3. Watch the brokers roll: + + ```bash + kubectl -n describe resourcequota default + kubectl -n rollout status sts/-kafka + kubectl -n get pods | grep -E '-(kafka|zookeeper)' + ``` + +The StatefulSet then completes its rolling update and all brokers run at the new spec. + +### Cause 4: The namespace LimitRange and ResourceQuota leave the broker pods without resources + +```bash +kubectl -n get kafka -o jsonpath='{.status.conditions}{"\n"}' +kubectl -n get strimzipodset +kubectl -n get pods | grep -kafka +kubectl -n get events --sort-by=.lastTimestamp | tail -40 +kubectl -n describe limitrange +kubectl -n describe resourcequota +``` + +Compare the broker requests and limits with the `LimitRange` maximums and with the quota hard limits minus `used`. A missing broker pod, an `exceeded quota` or LimitRange `forbidden` message, or a broker killed for lack of memory points at this cause. With `StrimziPodSet` the cluster operator creates the pods itself, so a rejected create shows up in the operator log and the namespace events, not as a StatefulSet `FailedCreate`. + +> **Warning:** Raising the namespace `LimitRange` and `ResourceQuota` lets the namespace consume more CPU and memory than before. Do it only when the events or the operator log name the quota or limit for the broker pods of this instance. Save both objects first and roll back by re-applying them once the instance is stable. + +```bash +kubectl -n get limitrange -o yaml > limitrange.bak.yaml +kubectl -n get resourcequota -o yaml > resourcequota.bak.yaml +kubectl -n edit limitrange +kubectl -n edit resourcequota +``` + +Raise the container maximums so they cover the broker request and limit, and the quota hard values so they cover current usage plus every broker pod of the instance. The same limits can be changed from the namespace settings in the console. Then watch the brokers start and the condition become `Ready`: + +```bash +kubectl -n get pods -w | grep -kafka +kubectl -n get kafka +``` + +## Diagnostic Steps + +1. Identify which pod the memory figure belongs to: + + ```bash + kubectl -n top pod --containers | grep + ``` + + A broker container: Cause 1 or Cause 2. `-entity-operator`: not this article. + +2. Memory around 90 percent that stays there after an increase, no `OOMKilled`, no backlog: Cause 2. + +3. A memory change that fails, a rollback that fails too, and `FailedCreate` with `exceeded quota: default`: Cause 3. + +4. `Kafka` condition `NotReady` with `Exceeded timeout of 300000ms while waiting for Pods resource -kafka-`, broker pods not running, and a `LimitRange` or `ResourceQuota` in the namespace: Cause 4. The same `Timed out waiting for a node assignment` text from a CLI tool while the `Kafka` resource is `Ready` and every broker pod is Running is a different problem (often a SASL listener used without client credentials). + +5. Quota pressure at a glance: + + ```bash + kubectl -n get resourcequota -o custom-columns='NAME:.metadata.name,HARD:.status.hard,USED:.status.used' + ``` + + Usage close to hard on memory or CPU confirms there is no room for a recreated broker. diff --git a/docs/en/solutions/Kafka_client_connection_failures_timeouts_TLS_certificates_authorization_NodePort_and_reverse_DNS_on_ACP.md b/docs/en/solutions/Kafka_client_connection_failures_timeouts_TLS_certificates_authorization_NodePort_and_reverse_DNS_on_ACP.md new file mode 100644 index 000000000..95e751e26 --- /dev/null +++ b/docs/en/solutions/Kafka_client_connection_failures_timeouts_TLS_certificates_authorization_NodePort_and_reverse_DNS_on_ACP.md @@ -0,0 +1,283 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda Application Services +ProductsVersion: + - "3.10" + - "3.12" + - "3.14" + - "3.16" + - "3.18" + - "4.0" + - "4.1" + - "4.2" +--- + +# Kafka client connection failures: timeouts, TLS certificates, authorization, NodePort, and reverse DNS on ACP + +## Issue + +Applications cannot reach a Data Services Kafka instance, or need to know how to connect: + +- New connections time out from outside the cluster on the external listener, while the broker pods are Running. +- The broker returns `TopicAuthorizationException: Not authorized to access topics: []` and a Spring application logs `Send failed`. Clients that used to connect without a user are rejected after authentication was turned on. +- A Kafka CLI tool run inside a broker pod fails with `TimeoutException: Timed out waiting for a node assignment. Call: listTopics`, and the broker logs `Failed authentication with /
(Unexpected Kafka request of type METADATA during SASL handshake.)`. +- A TLS client fails with `ERR_TLS_CERT_ALTNAME_INVALID`: `Failed to connect to seed broker, trying another broker from the list: Connection error: Hostname/IP does not match certificate's altnames: IP: is not in the cert's list:`. +- `openssl` on the client certificate shows it is past `notAfter`. +- Exporting the broker certificate through the console file download saves a `.crt` of 0 bytes. +- A new Kafka instance also creates a NodePort Service that is not wanted. +- Questions: can the connection address shown in Data Services change or be pinned, and can the Kafka Pod IPs be fixed? + +A user whose secret does not exist, a user stuck in `Creating`, or `Simple authorization ACL rules are configured but not supported in the Kafka cluster configuration.` is covered in [Kafka user secret missing, user stuck in Creating, ACL not supported, or uppercase topic names on ACP](Kafka_user_secret_missing_user_stuck_in_Creating_ACL_not_supported_or_uppercase_topic_names_on_ACP.md). An application outside the cluster that reports `No route to host` while curl from inside the application pod to Kafka succeeds is a network path problem, not a certificate fault. + +Listener ports of a Data Services Kafka: `9092` username/password inside the cluster, `9093` TLS inside the cluster, `9094` the external listener published through NodePort. + +## Root Cause + +### Cause 1: Kafka 2.7.0 blocks SASL_SSL threads on reverse DNS + +New connections to a Kafka 2.7.0 broker on the external SASL_SSL listener (port 9094 behind a NodePort) time out, both by Pod IP and by NodePort. One broker accumulates many `CLOSE_WAIT` sockets (around 200) and its listen socket receive queue backs up. Inside the pod, reverse lookups of client addresses hang: CoreDNS forwards those PTR queries to an upstream resolver that the pod cannot reach, and the SASL_SSL accept thread stalls in the lookup. These are the upstream Kafka defects KAFKA-13576 and KAFKA-8562. + +### Cause 2: The application client certificate is expired + +Kafka-related application pods fail because the client certificate they present is past its end date. Replacing that client certificate restores them. The operator's clients CA is a different object: on ACP 3.10.2 it is issued for about one year, and the operator renews it near expiry, which restarts brokers (see [Kafka Restarts After Client CA Certificate Expiry on ACP 3.10.2](ecosystem/kafka/Kafka_Certificate_Expiry_Restart_on_ACP_3102.md)). + +### Cause 3: Authentication and authorization are on and the client has no matching user, ACL, or SASL settings + +With authentication enabled on a listener, a client needs a user, and with `authorization.type: simple` that user needs ACL entries for the topic and for the consumer group. A user without the group ACL still fails as a consumer. + +A client that connects to a SASL listener without SASL settings (for example `kafka-topics.sh --bootstrap-server localhost:9092` with no client properties) sends ordinary Kafka requests where the broker expects a SASL handshake. The broker logs `Unexpected Kafka request of type METADATA during SASL handshake` and closes the connection, and the client keeps retrying until `Timed out waiting for a node assignment`. The source address in that broker line can be a node address (for example a Kube-OVN join subnet address when traffic arrives through a NodePort); it identifies the client, not a failing broker. + +### Cause 4: The console file download writes an empty crt + +The certificate path inside the broker pod is correct, but the console exec download saves a 0-byte `.crt`. + +### Cause 5: External access is enabled, so a NodePort Service is created + +A new Kafka gets a Service of type NodePort because the instance was created with external access on, and the external listener (named `external`, Service `-kafka-external-bootstrap`) is present in the Kafka resource. No safe way is known to drop that Service from an existing instance without recreating it. + +### Cause 6: The dialed broker IP is not in the certificate SAN + +The client performs TLS hostname verification and dials an IP that is not in the server certificate's subject alternative names. An empty list after `is not in the cert's list:` means the certificate carries no SAN at all. The failure is in the client that verifies the name, not in a broker refusing TCP. + +### Cause 7: Does the Kafka connection address in Data Services change, and can it be fixed? + +The question comes up before applications are pointed at a new instance. + +### Cause 8: Can the IP addresses of the Kafka Pods be fixed? + +The request is to give broker Pods fixed IPs, for example from a Kube-OVN underlay IP pool, so that clients or firewalls can use them. + +## Resolution + +### Cause 1: Kafka 2.7.0 blocks SASL_SSL threads on reverse DNS + +Apply this only when the broker is 2.7.0, the timeout is on 9094 or its NodePort, `CLOSE_WAIT` climbs on a broker, and PTR lookups from inside the pod are slow. + +Add host records for the client source addresses to the CoreDNS `hosts` plugin so the reverse lookup answers immediately: + +```bash +kubectl -n kube-system get cm coredns -o yaml > coredns.bak.yaml +kubectl -n kube-system edit cm coredns +``` + +```text +hosts { + 192.0.2.10 client-a.example.internal + fallthrough +} +``` + +Use the real client source addresses. CoreDNS reloads the Corefile on its own when the `reload` plugin is enabled; otherwise restart it: + +> **Warning:** Restarting CoreDNS briefly affects name resolution in the cluster. Do it one replica at a time with a rollout, and only after the ConfigMap change is saved. Roll back by re-applying `coredns.bak.yaml`. + +```bash +kubectl -n kube-system rollout restart deploy/coredns +kubectl -n exec -kafka-0 -- getent hosts 192.0.2.10 +``` + +The lookup should return immediately; then retry the external client. Moving the cluster off 2.7.0 also avoids the defect: 2.7.2, 2.8.1, and 3.0.0 do not block this way. A version change rolls the brokers; do it in a window. + +### Cause 2: The application client certificate is expired + +```bash +openssl x509 -in client.crt -noout -dates +``` + +Replace the expired certificate in the application key store or trust store and roll the application pods. + +To inspect the clients CA of the instance: + +```bash +kubectl -n get secret -clients-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > clients-ca.crt +openssl x509 -in clients-ca.crt -noout -dates +``` + +To lengthen later issuances, set both CA lifetimes on the `Kafka` resource (`3650` is a supported multi-year value) and renew: + +```yaml +spec: + clusterCa: + validityDays: 3650 + clientsCa: + validityDays: 3650 +``` + +> **Warning:** Renewing the clients CA with `strimzi.io/force-renew` restarts brokers and invalidates client certificates signed by the previous CA. Use it only when `openssl -dates` shows the CA itself is expired or close to expiry, in a maintenance window, and give clients the new CA material afterwards. An expired application certificate alone is replaced without this step. + +```bash +kubectl -n annotate secret -clients-ca-cert strimzi.io/force-renew=true --overwrite +``` + +### Cause 3: Authentication and authorization are on and the client has no matching user, ACL, or SASL settings + +Create a user bound to the instance and grant topic and group operations. For a Data Services instance the object is `RdsKafkaUser` (`middleware.alauda.io/v1`, label `middleware.alauda.io/cluster: `); for a Strimzi `Kafka` created in the management view it is `KafkaUser` (label `strimzi.io/cluster: `). The listener must have authentication (`scram-sha-512` or `tls`) and `spec.kafka.authorization.type` must be `simple`. + +```yaml +apiVersion: kafka.strimzi.io/v1beta1 +kind: KafkaUser +metadata: + name: + namespace: + labels: + strimzi.io/cluster: +spec: + authentication: + type: scram-sha-512 + authorization: + type: simple + acls: + - {host: "*", operation: Write, resource: {type: topic, name: , patternType: literal}} + - {host: "*", operation: Read, resource: {type: topic, name: , patternType: literal}} + - {host: "*", operation: Describe, resource: {type: topic, name: , patternType: literal}} + - {host: "*", operation: Read, resource: {type: group, name: , patternType: literal}} +``` + +```bash +kubectl -n apply -f kafkauser.yaml +kubectl -n get kafkauser +kubectl -n get secret -o jsonpath='{.data.password}' | base64 -d; echo +kubectl -n get svc -kafka-bootstrap -kafka-external-bootstrap +``` + +Password clients use `SASL_PLAINTEXT` on 9092 (or `SASL_SSL` on a TLS listener) with `sasl.mechanism=SCRAM-SHA-512`. TLS clients also need the cluster CA in their trust store. + +For CLI tools inside a broker pod, write a client file and pass it with `--command-config`: + +```bash +kubectl -n exec -i -kafka-0 -- sh -c 'cat > /tmp/client.properties' <<'EOF' +security.protocol=SASL_PLAINTEXT +sasl.mechanism=SCRAM-SHA-512 +sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="" password=""; +EOF +kubectl -n exec -kafka-0 -- /opt/kafka/bin/kafka-topics.sh \ + --bootstrap-server localhost:9092 --command-config /tmp/client.properties --describe --under-replicated-partitions +``` + +An empty result means no under-replicated partitions. + +### Cause 4: The console file download writes an empty crt + +Read the Secret, or copy the file with `kubectl cp`, instead of the console download: + +```bash +kubectl -n get secret -cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > cluster-ca.crt +kubectl -n cp -kafka-0: ./exported.crt +ls -l cluster-ca.crt exported.crt +``` + +Both files should be non-empty. + +### Cause 5: External access is enabled, so a NodePort Service is created + +For an instance that is not created yet, turn external access off in the create form; the `external` listener is then absent and no NodePort Service is created. For an existing instance, deleting the Service by hand is not a verified fix: the operator can recreate it while the external listener remains. + +### Cause 6: The dialed broker IP is not in the certificate SAN + +Confirm the address is missing from the SAN, using the port the application uses: + +```bash +openssl s_client -connect 192.0.2.10:9094 -servername 192.0.2.10 /dev/null | openssl x509 -noout -ext subjectAltName +``` + +No `X509v3 Subject Alternative Name` block means the certificate has no SAN. The fix is to turn off hostname verification on the failing client. + +> **Warning:** Turning off hostname verification makes the client accept any certificate signed by the trusted CA, whatever name it was issued for. Apply it only when the error is `Hostname/IP does not match certificate's altnames` and the dialed address is not in the SAN. Keep a copy of the previous client configuration; roll back by removing the setting. + +Java client (`ssl.endpoint.identification.algorithm` empty): + +```properties +ssl.endpoint.identification.algorithm= +``` + +Spring: `spring.kafka.properties.ssl.endpoint.identification.algorithm=`. librdkafka-based clients: `ssl.endpoint.identification.algorithm=none`. + +The wording `ERR_TLS_CERT_ALTNAME_INVALID` and `Failed to connect to seed broker` comes from a Node.js client such as KafkaJS, which does not read Java properties. There, skip the name check with `checkServerIdentity` while keeping the CA check (general Node.js behavior): + +```javascript +const fs = require('fs'); +const { Kafka } = require('kafkajs'); +const kafka = new Kafka({ + clientId: '', + brokers: ['192.0.2.10:9094'], + ssl: { + ca: [fs.readFileSync('/path/to/ca.crt', 'utf-8')], + checkServerIdentity: () => undefined, + }, +}); +``` + +### Cause 7: Does the Kafka connection address in Data Services change, and can it be fixed? + +The address is already fixed: it comes from the Kubernetes Services of the instance and stays the same as long as those Services are not modified. Changing it is not recommended. For external access, NodePort gives a node IP plus node port that does not change after the instance is created. + +```bash +kubectl -n get svc -l strimzi.io/cluster= +kubectl -n get svc -kafka-external-bootstrap -o jsonpath='{.spec.ports[*].nodePort}{"\n"}' +``` + +Inside the cluster, use the bootstrap Service `-kafka-bootstrap..svc` or the per-broker names `-kafka-.-kafka-brokers..svc`. + +### Cause 8: Can the IP addresses of the Kafka Pods be fixed? + +Not through a Data Services instance at present: `RdsKafka` does not support setting Pod `annotations`, which is how a Kube-OVN IP pool is normally attached, so a fixed IP pool cannot be assigned to the broker Pods. Pod IPs usually change when a Pod is recreated or rescheduled, so clients should not depend on them. Use the Service addresses from Cause 7 inside the cluster and NodePort from outside. + +## Diagnostic Steps + +1. Classify the client error: + + - Timeout only from outside, Kafka 2.7.0, `CLOSE_WAIT` on a broker: Cause 1. + - `certificate has expired`, or `openssl -dates` in the past: Cause 2. + - `TopicAuthorizationException`, or `Unexpected Kafka request of type METADATA during SASL handshake` on the broker with a client timeout: Cause 3. + - Empty file from the console download: Cause 4. + - `ERR_TLS_CERT_ALTNAME_INVALID`, `Hostname/IP does not match certificate's altnames`: Cause 6. Note the client language from the error format. + +2. Listener authentication and ports: + + ```bash + kubectl -n get kafka -o jsonpath='{.spec.kafka.listeners}{"\n"}{.spec.kafka.version}{"\n"}' + kubectl -n get svc -l strimzi.io/cluster= + ``` + + `"authentication":{"type":"scram-sha-512"}` on a listener means every client of that port, CLI tools included, needs SASL settings. + +3. Accept queue and reverse lookup (Cause 1). The broker image may lack `ss`; use `/proc/net/tcp` then (state `08` is `CLOSE_WAIT`): + + ```bash + kubectl -n exec -kafka-0 -- sh -c 'ss -ant 2>/dev/null | grep -c CLOSE-WAIT || awk "\$4==\"08\"" /proc/net/tcp /proc/net/tcp6 | wc -l' + kubectl -n exec -kafka-0 -- sh -c 'time getent hosts ' + ``` + + A multi-second PTR lookup is Cause 1. An immediate return rules it out. + +4. Find where a source address in a broker log comes from before blaming a broker: + + ```bash + kubectl get pod -A -o wide | grep
; kubectl get node -o wide | grep
+ kubectl get ip -o custom-columns='IP:.spec.ipAddress,SUBNET:.spec.subnet,POD:.spec.podName' | grep
+ ``` + +5. A timeout combined with ZooKeeper down or `FailedAttachVolume` is not a client problem; see [Kafka and ZooKeeper down from storage stalls, a full node disk, or a probe kill during log replay on ACP](Kafka_and_ZooKeeper_down_from_storage_stalls_a_full_node_disk_or_a_probe_kill_during_log_replay_on_ACP.md). diff --git a/docs/en/solutions/Kafka_config_change_does_not_roll_brokers_a_broker_restart_hangs_or_the_instance_stays_processing_on_ACP.md b/docs/en/solutions/Kafka_config_change_does_not_roll_brokers_a_broker_restart_hangs_or_the_instance_stays_processing_on_ACP.md new file mode 100644 index 000000000..e103ba024 --- /dev/null +++ b/docs/en/solutions/Kafka_config_change_does_not_roll_brokers_a_broker_restart_hangs_or_the_instance_stays_processing_on_ACP.md @@ -0,0 +1,154 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda Application Services +ProductsVersion: + - "3.12" + - "3.14" + - "3.16" + - "3.18" + - "4.0" + - "4.1" + - "4.2" +--- + +# Kafka config change does not roll brokers, a broker restart hangs, or the instance stays processing on ACP + +## Issue + +On a running Data Services Kafka instance: + +- A parameter change saved on the Data Services page does not restart the broker pods, so the new configuration never takes effect. Restarting the cluster operator lets stop, start, and restart work again for a while. The operator log shows: + + ```text + Failed to reconcile ... KafkaRoller$UnforceableProblem: Pod -kafka-2 is currently not rollable + Could not roll pod 2 due to ...ForceableProblem: Error getting broker config, retrying after at least 250ms + / will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker 2 is restarted. + ``` + +- A broker pod does not become ready after a restart, the instance reports `Error while trying to restart pod -kafka- to become ready`, and the operator shows `.status.phase (Failed) != Active` for the `RdsKafka`. +- After a node change, the instance stays in processing (`处理中`) although the Kafka pods have already moved to the new node. + +Related articles: [Kafka broker memory looks high, or a memory change or new broker Pod is blocked by quota on ACP](Kafka_broker_memory_looks_high_or_a_memory_change_or_new_broker_Pod_is_blocked_by_quota_on_ACP.md) covers rollouts that fail with `exceeded quota`. A cluster operator stuck on `Reconciliation is in progress` with no ISR line is covered in [Kafka instance creation times out on the entity-operator, shows failed, or the cluster operator restarts on ACP](Kafka_instance_creation_times_out_on_the_entity-operator_shows_failed_or_the_cluster_operator_restarts_on_ACP.md). A broker that is down because its disk is full is covered in [Kafka broker PVC full or still too small after expansion on ACP](Kafka_broker_PVC_full_or_still_too_small_after_expansion_on_ACP.md). + +## Root Cause + +### Cause 1: Rolling restart blocked because ISR is at min.insync.replicas + +The operator will not restart a broker if that would drop any partition below `min.insync.replicas`. When some partitions already have only one replica in ISR, and that replica is on the broker to be rolled, the `KafkaRoller` marks the pod `not rollable` and keeps retrying; `KafkaAvailability` lists the protected partitions with `will be underreplicated (|ISR|=1 and min.insync.replicas=1)`. `kafka-topics.sh --describe --topic __consumer_offsets` then shows partitions whose ISR is shorter than the replica list, for example `Replicas: 2,0,1 Isr: 2` and `Replicas: 1,0,2 Isr: 2,1`, while most partitions have all three in ISR. This was seen with Strimzi cluster operator 0.25.0 (the Kafka 2.7 and 2.8 era); the published procedure for this block is written for Kafka 2.7.0 and 2.8.2 on ACP 3.x (see [Resolve Kafka ISR Synchronization Issues That Block Broker Restart](ecosystem/kafka/Kafka_ISR_Synchronization_Issues.md)). Restarting the operator does not remove the ISR gap; restoring ISR does. + +### Cause 2: A broker does not become ready after a restart and the topic-store changelog has no leader + +On ACP 3.16.2 the instance logs repeat: + +```text +Cannot get current offset of topic __strimzi-topic-operator-kstreams-topic-store-changelog partition 0: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes +``` + +The stuck restart is attributed to that operator-internal changelog partition having no leader. Deleting the stuck broker pod so it starts again restores service. + +### Cause 3: The datapool keeps the old node after Kafka has moved + +After a node change, Kafka is rescheduled onto the new node, but the datapool pod is not: its node selector still names the old node, so it cannot schedule on the remaining nodes and the instance stays in processing. The datapool belongs to the MinIO component, and Kafka's scheduling change does not update MinIO. + +## Resolution + +### Cause 1: Rolling restart blocked because ISR is at min.insync.replicas + +1. Confirm the log lines and the ISR gap: + + ```bash + kubectl get pods -A -l strimzi.io/kind=cluster-operator + kubectl -n logs deploy/ --tail=500 | grep -E 'not rollable|will be underreplicated|Error getting broker config|Failed to reconcile' + kubectl -n exec -kafka-0 -- /opt/kafka/bin/kafka-topics.sh \ + --bootstrap-server localhost:9092 --describe --topic __consumer_offsets + ``` + + Use the topic named in the operator log if it is not `__consumer_offsets`. If the listener on `localhost:9092` requires SASL, add `--command-config ` to every Kafka command here. Take the broker ids from the `Replicas` line of a partition that lists every replica (ids such as `0`, `1`, `2`, or larger ids such as `1100`). + +2. Generate a reassignment plan. Create `/tmp/topic-generate.json`: + + ```json + {"topics": [{"topic": "__consumer_offsets"}], "version": 1} + ``` + + ```bash + kubectl -n cp /tmp/topic-generate.json -kafka-0:/tmp/topic-generate.json + kubectl -n exec -kafka-0 -- /opt/kafka/bin/kafka-reassign-partitions.sh \ + --bootstrap-server localhost:9092 \ + --topics-to-move-json-file /tmp/topic-generate.json \ + --broker-list ",," --generate + ``` + + Save the "Current partition replica assignment" from the output for rollback, and save the "Proposed partition reassignment configuration" as `/tmp/partition-replica-reassignment.json`. + +3. Execute it. + +> **Warning:** Partition reassignment moves replicas and adds disk and network load until it finishes. Run it only when the operator log says the broker `is currently not rollable` with `will be underreplicated`, and describe shows ISR shorter than Replicas on the topic you pass in. Use a maintenance window. Roll back by executing the saved current assignment. The alternative in the same procedure, force-deleting the stuck broker pod (`kubectl delete pod --force --grace-period=0`), can lose the latest records when ISR has a single replica; use it only when the business accepts that loss, which cannot be rolled back. + + ```bash + kubectl -n cp /tmp/partition-replica-reassignment.json -kafka-0:/tmp/partition-replica-reassignment.json + kubectl -n exec -kafka-0 -- /opt/kafka/bin/kafka-reassign-partitions.sh \ + --bootstrap-server localhost:9092 \ + --reassignment-json-file /tmp/partition-replica-reassignment.json --execute + kubectl -n exec -kafka-0 -- /opt/kafka/bin/kafka-reassign-partitions.sh \ + --bootstrap-server localhost:9092 \ + --reassignment-json-file /tmp/partition-replica-reassignment.json --verify + ``` + +4. When describe shows ISR equal to Replicas, apply the parameter change again; the brokers can roll. + +### Cause 2: A broker does not become ready after a restart and the topic-store changelog has no leader + +```bash +kubectl get rdskafka -A +kubectl -n get pods | grep +kubectl -n logs deploy/-entity-operator --all-containers --tail=300 | grep -i 'topic-store-changelog' +kubectl -n logs -kafka- --tail=300 +kubectl -n logs deploy/ --tail=500 | grep -E 'not rollable|will be underreplicated' +``` + +The changelog error can appear in the entity-operator, broker, or exporter logs of the instance. If the operator log has `not rollable` with `will be underreplicated`, use Cause 1 instead: deleting the pod there can lose records. + +> **Warning:** Deleting the stuck broker pod restarts that broker; partitions it leads move to other brokers and clients reconnect. Do it only when the instance reports `Error while trying to restart pod -kafka- to become ready`, the logs show no leader for `__strimzi-topic-operator-kstreams-topic-store-changelog`, the other brokers are Running, and the operator log has no `not rollable` line. The pod is recreated with the same PVC, so there is nothing to roll back. Add `--force --grace-period=0` only if the pod stays `Terminating`. + +```bash +kubectl -n delete pod -kafka- +kubectl -n get pods -w | grep -kafka- +kubectl get rdskafka -A +``` + +The pod should become `1/1 Running` and the `RdsKafka` phase should return to `Active`. + +### Cause 3: The datapool keeps the old node after Kafka has moved + +```bash +kubectl get pods -A -o wide | grep -i datapool +kubectl -n describe pod | grep -E 'Node-Selectors|Status:|Controlled By|Warning' +kubectl -n get pod -o jsonpath='{.metadata.ownerReferences}{"\n"}' +``` + +A Pending datapool pod whose node selector names a node Kafka has already left confirms this cause. Change the scheduling node of the MinIO component separately, to the node Kafka now runs on, with the same scheduling setting used for Kafka. + +> **Warning:** Changing the MinIO scheduling node reschedules the datapool pod. Do it only when that pod is Pending, its node selector names a node Kafka has left, and the target node is Ready. Roll back by pointing MinIO back at the previous node if that node is still schedulable. + +After the datapool pod is Running on the new node, the instance leaves processing. + +## Diagnostic Steps + +1. A console parameter change that does not roll brokers: read the operator log. `is currently not rollable` with `will be underreplicated (|ISR|=1 and min.insync.replicas=1)` and an `Isr` list shorter than `Replicas` in describe is Cause 1. Only `Reconciliation is in progress`, with no ISR line, is the stuck-operator case in the creation article. + +2. List only the partitions with a gap: + + ```bash + kubectl -n exec -kafka-0 -- /opt/kafka/bin/kafka-topics.sh \ + --bootstrap-server localhost:9092 --describe --under-replicated-partitions + ``` + + Empty output means no partition is under-replicated, so an ISR block is unlikely. A `Timed out waiting for a node assignment` from this command while every broker pod is Running usually means the listener needs SASL settings (`--command-config`). + +3. `Error while trying to restart pod -kafka- to become ready` with no leader for `__strimzi-topic-operator-kstreams-topic-store-changelog`, and no `not rollable` line: Cause 2. + +4. Instance in processing after a node change: compare the nodes of the Kafka pods and the datapool pod. Kafka moved, datapool Pending with the old node selector: Cause 3. diff --git a/docs/en/solutions/Kafka_instance_creation_times_out_on_the_entity-operator_shows_failed_or_the_cluster_operator_restarts_on_ACP.md b/docs/en/solutions/Kafka_instance_creation_times_out_on_the_entity-operator_shows_failed_or_the_cluster_operator_restarts_on_ACP.md new file mode 100644 index 000000000..09c0d3b74 --- /dev/null +++ b/docs/en/solutions/Kafka_instance_creation_times_out_on_the_entity-operator_shows_failed_or_the_cluster_operator_restarts_on_ACP.md @@ -0,0 +1,174 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda Application Services +ProductsVersion: + - "3.6" + - "3.12" + - "3.14" + - "3.16" + - "3.18" + - "4.0" + - "4.1" + - "4.2" +--- + +# Kafka instance creation times out on the entity-operator, shows failed, or the cluster operator restarts on ACP + +## Issue + +A Data Services Kafka instance does not come up, or the operator behind it is unhealthy: + +- The create fails, or the new instance sits failed, with: + + ```text + Exceeded timeout of 300000ms while waiting for Deployment resource -entity-operator in namespace to be ready + ``` + +- The Kafka service looks deployed, yet the instance still shows a creation failure in the console. +- The Kafka cluster operator pod restarts continuously (for example after an upgrade to ACP 3.14), so the console cannot create an instance or pick a namespace. +- A parameter change saved in the console never rolls the broker pods until the cluster operator pod is deleted. +- On a running instance, the memory of the entity-operator pod keeps rising and a resource increase has to be justified. + +Related articles: [No Kafka in Data Services, or Alauda Build of Kafka 2 unpublished or failing to deploy on ACP](No_Kafka_in_Data_Services_or_Alauda_Build_of_Kafka_2_unpublished_or_failing_to_deploy_on_ACP.md) covers the operator install and marketplace side. A ZooKeeper or broker pod named in the 300000 ms timeout is covered in [Kafka and ZooKeeper down from storage stalls, a full node disk, or a probe kill during log replay on ACP](Kafka_and_ZooKeeper_down_from_storage_stalls_a_full_node_disk_or_a_probe_kill_during_log_replay_on_ACP.md) and in [Kafka broker memory looks high, or a memory change or new broker Pod is blocked by quota on ACP](Kafka_broker_memory_looks_high_or_a_memory_change_or_new_broker_Pod_is_blocked_by_quota_on_ACP.md). A rollout blocked by `is currently not rollable` is covered in [Kafka config change does not roll brokers, a broker restart hangs, or the instance stays processing on ACP](Kafka_config_change_does_not_roll_brokers_a_broker_restart_hangs_or_the_instance_stays_processing_on_ACP.md). + +## Root Cause + +### Cause 1: The entity-operator Deployment never becomes Ready + +The create error names Deployment `-entity-operator` and the 300000 ms wait. On ACP 3.6 and 3.6.1 the containers of that Deployment inherit a very small namespace default limit and never become Ready (see [Kafka Entity Operator Fails to Start Because Container Limits Are Too Small](ecosystem/kafka/Kafka_Entity_Operator_Startup_Failure_Due_to_Container_Limits.md)). Later versions set explicit resources on the entity operator; the same timeout then means the pod is Pending, failing to pull its image, or crash-looping for another reason. The 300000 ms text only says that the Deployment did not become Ready in time. + +### Cause 2: The namespace quota blocks the Kafka exporter + +The Kafka brokers are running, but the instance shows creation failed. The namespace `ResourceQuota` is at its hard limit, so the Kafka exporter pod cannot be created. Raising the quota lets the exporter start, and the instance returns to normal. + +### Cause 3: The cluster operator is crash-looping or not reconciling + +Two different states: + +- On ACP 3.14 the operator pod restarts continuously. The startup failure for that upgrade is a null pointer when an `RdsTopic` exists and the matching `KafkaTopic` does not (see [Kafka Operator Startup Exception After ACP 3.14 Upgrade](ecosystem/kafka/Kafka_Operator_Startup_Exception_After_314_Upgrade.md)). +- The operator logs `Reconciliation is in progress` after an API server outage and then never moves instances; a parameter edit in the console then waits until the operator pod is deleted. This was fixed in the ACP 3.15 operator line; deleting the operator pod is the recovery on earlier builds (see [Kafka Operator Is Blocked by an Abnormal Instance](ecosystem/kafka/Kafka_Operator_Blocked_By_Abnormal_Instance.md)). + +`Reconciliation is in progress` alone does not prove a stuck operator. When the same log also has `KafkaRoller$UnforceableProblem: Pod -kafka- is currently not rollable` and `will be underreplicated (|ISR|=1 and min.insync.replicas=1) if broker is restarted`, the operator is deliberately holding the roll because of ISR, which is a different problem. + +### Cause 4: Entity operator memory keeps rising on a running instance + +The entity-operator pod of a running instance shows memory usage climbing steadily, for example from 67 percent to 92 percent of its limit, with `strimzi-kafka-operator.v4.2.0`, an entity operator sized at 1 CPU and 1000 MiB, two users, and one topic. No memory leak is known in that Kafka version and no root cause was established. Raising the entity operator to 2 CPU and 2 GiB and continuing to observe was sufficient: the instance kept operating normally. + +## Resolution + +### Cause 1: The entity-operator Deployment never becomes Ready + +```bash +kubectl -n get deploy -entity-operator +kubectl -n describe pod -l strimzi.io/name=-entity-operator +kubectl -n get limitrange -o yaml +``` + +When the described containers show a tiny CPU or memory limit inherited from the namespace `LimitRange` on ACP 3.6 or 3.6.1 (or the pod is clearly throttled or `OOMKilled` on that limit), raise the namespace default limit to a size the entity operator can start with, then recreate the pod. Deleting the pod only restarts topic and user reconciliation; brokers are not affected. + +```bash +kubectl -n delete pod -l strimzi.io/name=-entity-operator +kubectl -n rollout status deploy/-entity-operator +``` + +On later platforms, follow the `describe` output instead: image pull failures, unschedulable pods, and crashes in the `topic-operator` or `user-operator` container are ordinary workload faults. + +### Cause 2: The namespace quota blocks the Kafka exporter + +```bash +kubectl -n get pods | grep +kubectl -n get rs | grep -kafka-exporter +kubectl -n describe rs -kafka-exporter- +kubectl -n describe resourcequota +``` + +The exporter ReplicaSet event names the exceeded quota (`exceeded quota: , requested: ..., used: ..., limited: ...`). The exporter is normally named `-kafka-exporter`. + +> **Warning:** Raising a namespace `ResourceQuota` lets that namespace consume more CPU, memory, or pods than before. Do it only when the exporter event says the quota is exceeded and the instance is failed because that pod is not running. Save the quota first; roll back by re-applying the saved file. + +```bash +kubectl -n get resourcequota -o yaml > resourcequota.bak.yaml +kubectl -n edit resourcequota +``` + +Raise the exhausted hard value so it covers current usage plus the exporter request. On a project with a project quota, the namespace quota cannot exceed what the project has left; raise the project quota first if needed. After the exporter is Running, the instance status returns to normal. + +### Cause 3: The cluster operator is crash-looping or not reconciling + +```bash +kubectl get pods -A -l strimzi.io/kind=cluster-operator +kubectl -n logs deploy/ --tail=200 +kubectl -n logs deploy/ --tail=500 | grep -E 'not rollable|will be underreplicated|Error getting broker config' +``` + +If the ISR lines are present, stop here and use the config-rollout article linked in the Issue section. + +For the ACP 3.14 null pointer on an `RdsTopic` without a `KafkaTopic`: + +> **Warning:** Clearing finalizers and deleting an `RdsTopic` drops that custom resource. Do it only when the operator log names that `RdsTopic` and the failure is the startup null pointer with no matching `KafkaTopic`. Back it up first and roll back by re-creating it from the backup. On versions affected by the topic deletion defect, change `spec.topicName` to an unused name before the delete if it still points at a live topic (see [Avoid Kafka Topic Data Loss When Deleting an Abnormal Topic CR](ecosystem/kafka/Kafka_Topic_Delete_Data_Loss_Issue.md)). + +```bash +kubectl -n get rdstopic -o yaml > rdstopic.bak.yaml +kubectl -n patch rdstopic --type merge -p '{"metadata":{"finalizers":[]}}' +kubectl -n delete rdstopic +``` + +For reconciliation stuck after an API server outage, or a console parameter change that never rolls the brokers with no ISR line in the log, restart only the cluster operator. This does not restart brokers; reconciliation resumes when the new pod is up. + +```bash +kubectl delete pods -A -l strimzi.io/kind=cluster-operator +kubectl get kafka -A +``` + +Wait until each `Kafka` returns to its previous ready state. + +### Cause 4: Entity operator memory keeps rising on a running instance + +```bash +kubectl get csv -A | grep -i strimzi +kubectl -n top pod --containers | grep entity-operator +kubectl -n get kafka -o jsonpath='{.spec.entityOperator}{"\n"}' +kubectl -n get kafka -o jsonpath='{.metadata.ownerReferences}{"\n"}' +``` + +Raise the entity operator to 2 CPU and 2 GiB. On a Strimzi `Kafka` the resources are set per container, under `spec.entityOperator.topicOperator.resources` and `spec.entityOperator.userOperator.resources`; give the larger limit to the container that `kubectl top` shows growing. If `ownerReferences` shows the `Kafka` is owned by an `RdsKafka`, make the change on that instance (its update page or its spec) so the operator does not revert a direct edit. + +> **Warning:** Changing the entity operator resources recreates the entity-operator pod. Topic and user reconciliation pauses until the new pod is Ready; brokers are not restarted. Do it only when the growing pod is the entity operator. Roll back by restoring the previous `resources` values. + +```bash +kubectl -n patch kafka --type merge -p '{"spec":{"entityOperator":{"":{"resources":{"requests":{"cpu":"1","memory":"2Gi"},"limits":{"cpu":"2","memory":"2Gi"}}}}}}' +kubectl -n rollout status deploy/-entity-operator +``` + +If usage keeps climbing toward the new limit with an unchanged number of topics and users, collect the operator logs and the CSV version for escalation. Broker memory above an inspection threshold is covered in the broker memory article linked in the Issue section. + +## Diagnostic Steps + +1. Read the resource named in the 300000 ms sentence. A Deployment ending in `-entity-operator` is Cause 1. A ZooKeeper or broker pod, or a `Service`, is a different problem (see the related articles). + +2. Operator and instance health: + + ```bash + kubectl get pods -A -l strimzi.io/kind=cluster-operator + kubectl get kafka,rdskafka -A + kubectl -n get deploy,sts,po | grep + ``` + + A cluster operator pod with a growing restart count is Cause 3. + +3. Instance failed, brokers Running, exporter pod absent, and an `exceeded quota` event on its ReplicaSet: Cause 2. + +4. A console parameter change that does not roll brokers: check the operator log for `not rollable`. Present means an ISR block; absent with `Reconciliation is in progress` after an API server outage means Cause 3. + +5. Memory growth on a running instance: `kubectl -n top pod --containers`. Growth in `-entity-operator` is Cause 4; growth in a broker is not. + +6. Version skew between operator and instance (an upgraded operator leaving the instance on an old version, a Kafka 2.x instance that ignores the console): + + ```bash + kubectl -n get kafka -o yaml | grep -n -i version + ``` + + See [Kafka operator mismatch: wrong create page version, Kafka 2.x instance skipped, or ZooKeeper mode blocks upgrade on ACP](Kafka_operator_mismatch_wrong_create_page_version_Kafka_2x_instance_skipped_or_ZooKeeper_mode_blocks_upgrade_on_ACP.md). diff --git a/docs/en/solutions/Kafka_log_disk_grows_to_hundreds_of_gigabytes_or_topic_retention_differs_from_the_configured_value_on_ACP.md b/docs/en/solutions/Kafka_log_disk_grows_to_hundreds_of_gigabytes_or_topic_retention_differs_from_the_configured_value_on_ACP.md new file mode 100644 index 000000000..3b22b2764 --- /dev/null +++ b/docs/en/solutions/Kafka_log_disk_grows_to_hundreds_of_gigabytes_or_topic_retention_differs_from_the_configured_value_on_ACP.md @@ -0,0 +1,175 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda Application Services +ProductsVersion: + - "3.10" + - "3.12" + - "3.14" + - "3.16" + - "3.18" + - "4.0" + - "4.1" + - "4.2" +--- + +# Kafka log disk grows to hundreds of gigabytes or topic retention differs from the configured value on ACP + +## Issue + +One of these is reported for Kafka: + +- On the log nodes, `/cpaas/data/kafka0` or a broker PVC climbs to hundreds of gigabytes, although consumer lag is near zero and log delay is under a minute. +- One broker restarts in a loop while its peers stay up, and only that node's `kafka0` directory is hundreds of gigabytes. The peers' `/opt/kafka/logs/server.log` repeats, from `ReplicaFetcherThread`: + +```text +java.io.IOException: Connection to :9092 (id: rack: null) failed. +``` + +- Data Services topic retention was set to 2 hours, but a topic still shows 168 hours after it is created or synced in the console. + +Related articles: *Kafka broker PVC full or still too small after expansion on ACP* (expanding a full broker PVC and aligning the `Kafka` CR size), and *Avoid Kafka Topic Data Loss When Deleting an Abnormal Topic CR* (deleting an abnormal `RdsTopic` on affected versions). + +## Root Cause + +### Cause 1: Hundreds of gigabytes on disk with almost no lag + +`kafka0` holds more than 600 GB on most brokers, consumer lag is not growing, and end-to-end log delay is under one minute. Consumed records stay on disk until the retention time or the log size limit deletes them, so the volume matches roughly `daily ingest x replica count x retention` (for example 48 hours of retention with two replicas). If the log shipper does not compress, the volume is larger still. Segments of other topics (for example `devops-pipeline`) that are still inside the retention window are kept as well. This state is normal. + +It is not this cause when one broker is several times larger than its peers, that broker is restarting, and the ISR contains only that broker; that is Cause 3. + +### Cause 2: Console sync writes a 168-hour template over the retention you set + +In Data Services the global topic retention is set to 2 hours, but a topic created afterwards still shows 168 hours. Changing `log.retention.hours` and restarting Kafka does not change it, and neither does writing the same value through the configuration file. Retention configured on the Kafka instance in the Data Services console does apply to topics created from the command line. When such a topic is then synced in the console and converted to an `RdsTopic`, rds-api 3.18.1 writes a fixed template whose retention is 168 hours; the field is in the template because the page failed without it. The global setting is not wrong; the sync overwrites it on that topic. + +### Cause 3: One broker's kafka0 fills the disk because it holds the leaders + +One broker restarts repeatedly while the others stay up. That broker's `kafka0` directory has grown to roughly 400 to 500 GB and filled the disk, while the same directory on the other brokers holds only tens of gigabytes. `kafka-topics.sh --describe` shows almost all partition leaders on the restarting broker, and the ISR contains only that broker's id (for example `Leader: 1100` and `Isr: 1100` while `Replicas` lists `1100,1101,1102`). The peers log the replica fetch failure shown in Issue for `ALAUDA_LOG_TOPIC` and `ALAUDA_AUDIT_TOPIC` partitions. The full disk makes the leader broker intermittently unhealthy, which keeps interrupting replication, so the peers never catch up and the single copy keeps growing. + +Before this state becomes visible, the whole Kafka can fail to start; it starts after about 20 minutes once the health-check timing is lengthened and the JVM memory is raised, after which only the full broker keeps restarting. When the disk cannot be expanded, deleting that broker's `kafka0` directory and reducing the default retention from 48 hours to 24 hours returns Kafka to normal. + +## Resolution + +### Cause 1: Hundreds of gigabytes on disk with almost no lag + +Confirm that lag is not growing and delay is under about a minute, then treat the bytes as retained segments. List sizes and segment ages before changing retention: + +```bash +kubectl -n exec -kafka-0 -- sh -c 'du -sh /var/lib/kafka/data/* | sort -h' +kubectl -n exec -kafka-0 -- sh -c 'ls -lh /var/lib/kafka/data/- | head' +``` + +On the log nodes, the same check is `du -sh /cpaas/data/kafka0/*`. + +Capacity that matches `daily ingest x replica count x retention` can be left as it is. Shorten retention only when the owner wants less history: + +```bash +kubectl -n get kafkatopic,rdstopic +kubectl -n edit rdstopic +``` + +Set `spec.config.retention.ms` to the agreed window (48 hours is `172800000`). Segments of other topics inside the window age out with retention; do not delete them by hand. If lag on `ALAUDA_LOG_TOPIC` is high at the same time, the growth is unread data; fix the consumers before shortening retention. + +### Cause 2: Console sync writes a 168-hour template over the retention you set + +Use this only when a topic shows 168 hours after it was synced or converted to an `RdsTopic` in the console, and the retention the owner configured (for example 2 hours) is the value that should be stored. + +```bash +kubectl -n get rdstopic -o jsonpath='{.spec.config.retention\.ms}{"\n"}' +kubectl -n get rdstopic -o yaml > .rdstopic.bak.yaml +``` + +> **Warning:** Writing a shorter `retention.ms` deletes segments older than the new window, and they do not come back. Do it only for a topic whose YAML still shows the 168-hour template after a console sync, and only after the topic owner has accepted the shorter window. Roll back by setting `retention.ms` back to `604800000` (168 hours); segments already deleted stay deleted. + +```bash +kubectl -n edit rdstopic +``` + +```yaml +spec: + config: + retention.ms: "7200000" # 2 hours +``` + +The other workaround is to type the retention time in the console when the topic is created, instead of relying on the global default for a topic that the console will sync. Restarting the brokers after changing `log.retention.hours` does not replace the template value already written onto a synced topic. + +### Cause 3: One broker's kafka0 fills the disk because it holds the leaders + +1. Confirm the skew before deleting anything. On the platform log Kafka the bootstrap address is `cpaas-kafka:9092`; pass the client configuration file the brokers use for authenticated access: + +```bash +kubectl -n cpaas-system get pods -o wide | grep cpaas-kafka +kubectl -n cpaas-system exec -- \ + /opt/kafka/bin/kafka-topics.sh --bootstrap-server cpaas-kafka:9092 \ + --command-config --describe +kubectl -n cpaas-system exec -- \ + sh -c 'grep -E "Connection to .* failed|ReplicaFetcherThread" /opt/kafka/logs/server.log | tail -20' +``` + +Expected: almost every leader is the restarting broker, the ISR lists only its id, and the peers log `Connection to :9092 (id: rack: null) failed.` for that id. Run `du -sh /cpaas/data/kafka0` on each log node: the full node holds hundreds of gigabytes and the peers tens of gigabytes. + +2. If the volume can grow, expand it (see *Kafka broker PVC full or still too small after expansion on ACP*), then shorten retention so the same broker does not fill the new size. The remaining steps apply only when the disk cannot be expanded. + +3. If the restarting broker never stays up long enough to describe topics, lengthen its health-check timing and raise its JVM heap so that a start of about 20 minutes is not killed. Save the workload first; if a controller reverts the edit, make the same change on the object that owns the workload. This only makes the describe and cleanup possible; it does not stop the full broker from restarting. + +```bash +kubectl -n cpaas-system get sts cpaas-kafka -o yaml > cpaas-kafka.sts.bak.yaml +``` + +4. Pause the log consumer `razor` so it stops reading while the cleanup runs, and let consumers drain. + +> **Warning:** Stopping `razor` pauses that log component until it is scaled back. Do it only when Cause 3 is confirmed and the disk cannot be expanded. Roll back by scaling it to the replica count printed by the second command. + +```bash +kubectl get deploy,sts -A | grep -i razor +kubectl -n get / -o jsonpath='{.spec.replicas}{"\n"}' +kubectl -n scale / --replicas=0 +``` + +A Pod named `razor-0` belongs to a StatefulSet, so `` is `sts` (in `cpaas-system`). + +5. Delete the oversized copy on the full node only. + +> **Warning:** `rm -rf /cpaas/data/kafka0/*` on the full node destroys the log segments on that node, including records the lagging peers never replicated, so recent logs can be lost. It is irreversible. Run it only when Cause 3 is confirmed (one broker restarting, peers up, ISR is only that broker, this node's `kafka0` is the copy at hundreds of gigabytes), the disk cannot be expanded, no process is using the directory, and a maintenance window accepts the loss. There is no rollback. + +```bash +ls /cpaas/data/kafka0 +rm -rf /cpaas/data/kafka0/* +``` + +6. Reduce the broker default retention from 48 to 24 hours. + +> **Warning:** Setting the broker default `log.retention.hours` to `24` deletes segments older than 24 hours on every topic that inherits the default; those segments do not come back. Roll back by setting it to `48`. A topic that sets its own `retention.ms` keeps that value. + +```bash +kubectl -n cpaas-system exec -- \ + /opt/kafka/bin/kafka-configs.sh --bootstrap-server cpaas-kafka:9092 \ + --command-config \ + --alter --entity-type brokers --entity-default \ + --add-config log.retention.hours=24 +``` + +If the command fails because the broker is still down after the delete, run it once the broker is Ready. If a controller reconciles broker configuration from a Kafka object, set `log.retention.hours: 24` in that object's Kafka config (the object that currently shows 48) so the controller keeps the new default. + +7. Scale `razor` back to the saved replica count and confirm that all brokers stay Running and the ISR lists every replica again. + +## Diagnostic Steps + +1. Measure lag before deleting anything. Lag near zero plus a large directory is Cause 1. Lag climbing on `ALAUDA_LOG_TOPIC` is a consumer problem, not this article. + +2. Compare the size of `kafka0` (or `/var/lib/kafka/data`) across brokers. Similar sizes point to Cause 1; one broker many times larger, restarting, with the ISR reduced to that broker, is Cause 3. + +```bash +kubectl -n exec -- du -sh /var/lib/kafka/data +du -sh /cpaas/data/kafka0 +``` + +3. If Data Services shows 2 hours and the topic shows 168 hours after a console sync, it is Cause 2, even when a Kafka restart keeps 168 hours: + +```bash +kubectl -n get rdstopic -o jsonpath='{.spec.config}{"\n"}' +``` + +4. A JVM out-of-memory line in the previous broker log, with every broker replaying a large log at startup, is a startup failure, not Cause 3 (see *Kafka and ZooKeeper down from storage stalls, a full node disk, or a probe kill during log replay on ACP*). diff --git a/docs/en/solutions/Kafka_operator_mismatch_wrong_create_page_version_Kafka_2x_instance_skipped_or_ZooKeeper_mode_blocks_upgrade_on_ACP.md b/docs/en/solutions/Kafka_operator_mismatch_wrong_create_page_version_Kafka_2x_instance_skipped_or_ZooKeeper_mode_blocks_upgrade_on_ACP.md new file mode 100644 index 000000000..05c0c1506 --- /dev/null +++ b/docs/en/solutions/Kafka_operator_mismatch_wrong_create_page_version_Kafka_2x_instance_skipped_or_ZooKeeper_mode_blocks_upgrade_on_ACP.md @@ -0,0 +1,191 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda Application Services +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Kafka operator mismatch: wrong create page version, Kafka 2.x instance skipped, or ZooKeeper mode blocks upgrade on ACP + +## Issue + +One of these appears after a platform upgrade or after the Kafka operator version was changed: + +- The Kafka component is reported abnormal, with no error text, on a platform 4.2 cluster that still runs Kafka operator 4.1.0. +- The Data Services **Create Kafka Instance** page offers only `Kafka 4.1` or `Kafka 4.2`, although the operator was switched to a version that should offer `Kafka 3.8`. Reinstalling the operator does not change the list, and in some environments no Kafka instance can be created at all. +- A Kafka 2.x instance does not react to the console: a stop leaves it in "Stopping" with every pod still `Running`, configuration changes never reach the pods, or a broker is reported offline and nothing repairs it. `kubectl get event` in the namespace shows nothing. The cluster operator log repeats one of: + + ```text + skip reconcile (Paused), kafka version is 2.8.2 + skip reconcile , kafka version is 2.7.0 + io.vertx.core.impl.NoStackTraceThrowable: skip reconcile, not match + ``` + +- During a business-cluster upgrade, the upgrade log of the kafka functional component says the Kafka instance operator version does not match. The marketplace already shows the operator at 4.2 and the instance has not been updated. +- Questions: can the 4.3 Kafka Operator be used on a 4.1 platform, and on 4.4.0 and later are Data Services Essentials and RDS Framework still needed for Kafka? + +Operator install and publish failures (no Kafka menu, Alauda Build of Kafka 2 unpublished or failing to deploy on 4.1) are covered in [No Kafka in Data Services, or Alauda Build of Kafka 2 unpublished or failing to deploy on ACP](No_Kafka_in_Data_Services_or_Alauda_Build_of_Kafka_2_unpublished_or_failing_to_deploy_on_ACP.md). + +## Environment + +- Kafka operator package: Alauda Streaming Service for Kafka, Subscription `strimzi-kafka-operator`, CSV `strimzi-kafka-operator.v`, channel `stable`, Deployment `strimzi-cluster-operator`. It is usually installed in `kafka-system`; on clusters upgraded from 3.x it can be in another namespace such as `operators`. Find it with `kubectl get csv -A | grep -i kafka`. +- Kafka 2.x instances are served by a separate operator, Alauda Build of Kafka 2. +- The Kafka versions offered on the create page come from `ImageVersion` objects (`imageversion.middleware.alauda.io`), one per Kafka operator artifact, for example `kafka-artifact-v4.1.2` or `kafka-artifact-v4.2.0`. + +## Root Cause + +### Cause 1: Kafka operator 4.1.0 on platform 4.2 leaves the Kafka component abnormal + +`strimzi-kafka-operator.v4.1.0` is not compatible with platform 4.2. Replacing it with operator 4.1.2, the latest release of that line at the time, fixes the component: OperatorHub then shows `strimzi-kafka-operator.v4.1.2` with status `Succeeded`, and a Kafka 3.8 instance (ZooKeeper mode) runs with its broker pods `Running (1/1)` and its PVCs `Bound`. + +### Cause 2: A leftover ImageVersion from another operator version controls the create page + +`kubectl get imageversion` lists two Kafka artifacts, for example `kafka-artifact-v4.1.2` and `kafka-artifact-v4.2.0`, or `kafka-artifact-v4.1.0` and `kafka-artifact-v4.3.2`. + +The Kafka operator was moved to a different version (for example 4.2.0 to 4.1.2 on platform 4.2.4 to get Kafka 3.8, or a 4.3 operator was installed, uninstalled, and replaced by 4.1.0). The new operator installed cleanly (`Succeeded`, `InstallSucceeded`, "install strategy completed with no errors"), but the `ImageVersion` of the previous operator stayed in the cluster. The higher version takes priority, so the create page keeps offering the Kafka versions of the removed operator. When a lower Kafka Operator version is installed after a higher one, the leftover resources of the higher version must be cleaned up by hand. + +### Cause 3: No Kafka 2 operator reconciles the Kafka 2.x instance + +After an upgrade to platform 4.0 (for example from 3.16 to 4.0.9), only Alauda Streaming Service for Kafka is installed. Alauda Build of Kafka 2, the operator for Kafka 2.x instances, was not published or installed. The installed operator skips every Kafka 2.x instance on each timer reconciliation (`skip reconcile ... kafka version is 2.`, `skip reconcile, not match`) and then logs it as `reconciled`. Nothing manages the instance any more: a stop from the console never reaches the pods, configuration changes are not applied, and a broker that drops out is not repaired by an operator. Alauda Build of Kafka 2 operator `v4.0.3` supports platform `v4.0` and `v4.1` and Kafka `2.5`, `2.7`, and `2.8`. + +A `tlsSidecar ... has been deprecated` warning in the same log is unrelated and can be ignored. + +### Cause 4: Kafka operator upgraded to 4.2 while instances still run in ZooKeeper mode + +The product document "Migrate from ZooKeeper to KRaft mode" states that the v4.2.x Kafka operator supports only KRaft mode and has removed ZooKeeper mode. Before upgrading to Alauda Streaming Service for Kafka v4.2.x, every instance must be migrated to KRaft mode with the v4.1.x operator; the v4.2.x operator cannot perform that migration. Upgrading the operator to 4.2 while a ZooKeeper-mode instance exists leaves that instance behind the operator, and the cluster upgrade reports the version mismatch. + +### Cause 5: Can the 4.3 Kafka Operator be used on a 4.1 platform? + +The 4.3 operator is documented as compatible with 4.0, 4.1, and 4.2, but it gives a usable create page only when the conditions in the Resolution are met. + +### Cause 6: On 4.4.0 and later, are Data Services Essentials and RDS Framework still needed? + +The question is whether only Kafka 2 and ClickHouse Operator need to be deployed on v4.4.0 and later, without the web console plugins Alauda Container Platform Data Services Essentials and Alauda Container Platform Data Services RDS Framework. + +## Resolution + +### Cause 1: Kafka operator 4.1.0 on platform 4.2 leaves the Kafka component abnormal + +```bash +kubectl get csv -A | grep strimzi-kafka-operator +kubectl -n get subscription strimzi-kafka-operator -o jsonpath='{.status.installedCSV}{"\n"}' +``` + +When the CSV is `strimzi-kafka-operator.v4.1.0` and the platform is 4.2, replace it with operator 4.1.2 in **Administrator > App Marketplace Management > OperatorHub** (Alauda Streaming Service for Kafka, channel `stable`). If 4.1.2 is not offered, upload that operator package to the platform first with `violet push`. + +> **Warning:** Replacing the operator restarts `strimzi-cluster-operator`. Reconciliation of every Kafka instance on the cluster pauses until the new CSV is `Succeeded`, and the new operator can then roll broker pods if it changes their images or configuration. Do it only when the installed CSV is 4.1.0 on a platform 4.2 cluster. Roll back by installing 4.1.0 again from OperatorHub. + +```bash +kubectl get csv -n -w +kubectl get kafka -A +kubectl -n get pods,pvc | grep +``` + +If an instance still does not recover and its data is not needed, it can be recreated after the 4.1.2 operator is `Succeeded`. + +> **Warning:** Recreating an instance deletes its topics and messages; this cannot be undone. Do it only for an instance whose data is not needed. + +### Cause 2: A leftover ImageVersion from another operator version controls the create page + +```bash +kubectl get imageversion +kubectl get csv -A | grep strimzi-kafka-operator +``` + +The `ImageVersion` to remove is the Kafka artifact whose version does not match the installed CSV, for example `kafka-artifact-v4.2.0` while the CSV is `strimzi-kafka-operator.v4.1.2`. + +> **Warning:** Deleting an `ImageVersion` removes the Kafka versions it provides from the create page. Do it only when two `kafka-artifact-*` objects exist, the one you delete belongs to an operator version that is no longer installed, and no instance is meant to use the versions it offers. Export it first; roll back by re-creating it from that file after removing `metadata.resourceVersion`, `metadata.uid`, and `metadata.creationTimestamp`. + +```bash +kubectl get imageversion kafka-artifact-v4.2.0 -o yaml > kafka-artifact-v4.2.0.yaml +kubectl delete imageversion kafka-artifact-v4.2.0 +kubectl get imageversion +``` + +Refresh the create page. It then offers `Kafka 3.8`, and with the 4.1.x operator `ZooKeeper 模式` (ZooKeeper mode) is available. When a 4.3 operator had been installed first, clean up all leftover resources of that higher-version package, including `kafka-artifact-v4.3.2`. + +### Cause 3: No Kafka 2 operator reconciles the Kafka 2.x instance + +1. Confirm the instance version, the skip message, and that no Kafka 2 operator is installed: + + ```bash + kubectl get kafka -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,VERSION:.spec.kafka.version + kubectl get csv -A | grep -i kafka + kubectl -n logs deploy/strimzi-cluster-operator -c strimzi-cluster-operator --tail=500 | grep 'skip reconcile' + ``` + + A Kafka `2.` instance, `skip reconcile ... kafka version is 2.` for it, and only `strimzi-kafka-operator.v` (Alauda Streaming Service for Kafka) in the CSV list confirm this cause. + +2. Upload the Alauda Build of Kafka 2 operator package to the platform with `violet push`. Use the `violet` version that matches the operator package, not simply the `violet` binary that came with the platform. Use a package version that supports both the platform and the Kafka version, for example `v4.0.3` for platform 4.0 or 4.1 with Kafka 2.5, 2.7, or 2.8. + +3. Publish it and install it from **OperatorHub** (Alauda Build of Kafka 2, channel `stable`). If the package is uploaded but OperatorHub still shows it as not published, or the deployment fails on 4.1, see the article on Alauda Build of Kafka 2 publishing linked in the Issue section. Do not uninstall Alauda Streaming Service for Kafka for this. + +4. Restart the Kafka 2.x instance from the console (or repeat the stop that did nothing) so the Kafka 2 operator takes it over. + +> **Warning:** Restarting the instance rolls its broker and ZooKeeper pods; clients reconnect and partitions led by a restarting broker move. Do it only after the Alauda Build of Kafka 2 CSV is `Succeeded`, in a maintenance window. There is nothing to roll back; if the operator does not take the instance over, collect the operator logs before further changes. + +5. Check that the Kafka 2 CSV is listed and that `skip reconcile` no longer appears for the instance in the Alauda Streaming Service for Kafka log. + +Alauda Build of Kafka 2 is supported only on platform v4.0 and v4.1 (see Cause 6). + +### Cause 4: Kafka operator upgraded to 4.2 while instances still run in ZooKeeper mode + +```bash +kubectl get kafka -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,VERSION:.spec.kafka.version,METADATA:.status.kafkaMetadataState,OPERATOR:.status.operatorLastSuccessfulVersion +kubectl get csv -A | grep strimzi-kafka-operator +``` + +Any instance whose `METADATA` is `ZooKeeper` blocks the v4.2.x operator. The supported order is: keep the v4.1.x operator, migrate every instance to KRaft mode following "Migrate from ZooKeeper to KRaft mode", and only then upgrade to v4.2.x. + +When the operator has already been upgraded and the cluster upgrade is waiting on this mismatch, a possible workaround is to uninstall the Kafka operator temporarily so the cluster upgrade can continue. This workaround is not a verified procedure; prefer the supported order above whenever the upgrade can wait. + +> **Warning:** While the Kafka operator is uninstalled, nothing reconciles any Kafka instance on the cluster: configuration changes, stop and start, users, and topics are not applied, and a deleted or evicted broker pod is not recreated by the operator. Do it only when the cluster upgrade is blocked by this mismatch and at least one instance is still in ZooKeeper mode. Reinstall the operator version that matches the instances (v4.1.x for ZooKeeper-mode instances) from OperatorHub as soon as the cluster upgrade finishes; that reinstall is the rollback. + +Uninstall it from **OperatorHub** (Alauda Streaming Service for Kafka), or remove its Subscription and CSV: + +```bash +kubectl -n get subscription strimzi-kafka-operator -o yaml > strimzi-kafka-operator-sub.yaml +kubectl -n delete subscription strimzi-kafka-operator +kubectl -n delete csv +``` + +### Cause 5: Can the 4.3 Kafka Operator be used on a 4.1 platform? + +Only when both conditions of the official upgrade documentation are met. First, v4.2.x and later support only KRaft mode, so every instance must already run in KRaft mode; ZooKeeper-mode instances must be migrated with the v4.1.x operator first (Cause 4). Second, Kafka 4.2.0 requires Alauda Container Platform Data Services RDS Framework 4.2.2 or later. If the 4.3 operator is later removed in favor of a lower one, clean up its leftover resources as in Cause 2. + +### Cause 6: On 4.4.0 and later, are Data Services Essentials and RDS Framework still needed? + +Yes for the Kafka Operator: on v4.4.0 and later it depends on Alauda Container Platform Data Services Essentials and Alauda Container Platform Data Services RDS Framework, and both must be installed to support upgrades and troubleshooting. Kafka 2 Operator does not apply there: it is supported only on v4.0 and v4.1. ClickHouse Operator does not depend on these two plugins and can be deployed without them. + +```bash +kubectl get csv -A | grep -Ei 'kafka|rds-operator|clickhouse' +``` + +## Diagnostic Steps + +1. Record the platform version, the installed Kafka operator CSVs, and the Kafka artifacts: + + ```bash + kubectl get csv -A | grep -i kafka + kubectl get imageversion + kubectl -n get operatorcondition + ``` + + An `operatorcondition` with `CatalogSourcesUnhealthy` `Status: False` and `Reason: AllCatalogSourcesHealthy` is healthy; it does not explain a wrong version list. + +2. The create page offers a Kafka version of a different operator version, and two `kafka-artifact-*` objects exist: Cause 2. + +3. CSV `strimzi-kafka-operator.v4.1.0` on a platform 4.2 cluster with the component reported abnormal: Cause 1. + +4. A Kafka 2.x instance ignores console actions or keeps a broker down, the operator log says `skip reconcile ... kafka version is 2.`, and no Alauda Build of Kafka 2 CSV exists: Cause 3. Before treating a broker as offline, check its pod and the cluster state; all broker pods `1/1 Running` with the `Kafka` resource `Ready` does not by itself prove a broker failure. + +5. Operator at 4.2 or later, the instance not updated, and `status.kafkaMetadataState` is `ZooKeeper`: Cause 4. A direct platform hop from 4.0 to 4.2 with the operator stuck has the same KRaft requirement. + +6. Several middleware operators abnormal together, the operator crash-looping, or no Kafka menu at all: see the operator install article linked in the Issue section, or [Kafka instance creation times out on the entity-operator, shows failed, or the cluster operator restarts on ACP](Kafka_instance_creation_times_out_on_the_entity-operator_shows_failed_or_the_cluster_operator_restarts_on_ACP.md). diff --git a/docs/en/solutions/Kafka_user_secret_missing_user_stuck_in_Creating_ACL_not_supported_or_uppercase_topic_names_on_ACP.md b/docs/en/solutions/Kafka_user_secret_missing_user_stuck_in_Creating_ACL_not_supported_or_uppercase_topic_names_on_ACP.md new file mode 100644 index 000000000..d788e5988 --- /dev/null +++ b/docs/en/solutions/Kafka_user_secret_missing_user_stuck_in_Creating_ACL_not_supported_or_uppercase_topic_names_on_ACP.md @@ -0,0 +1,187 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda Application Services +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" +--- + +# Kafka user secret missing, user stuck in Creating, ACL not supported, or uppercase topic names on ACP + +## Issue + +A Kafka user was created from Data Services, but opening its secret (`保密字典`, Secret) says the resource does not exist. The user shows one of these: + +- `Simple authorization ACL rules are configured but not supported in the Kafka cluster configuration.` The same is often reported as "works on Kafka 2.8, fails on Kafka 2.7". +- `status.phase` stays `Creating` with `Events: `, on an instance that uses `SCRAM-SHA-512` and where user creation used to work. Other Kafka instances in the same cluster create users normally, and other recently created users of the same instance also have an empty `READY` column in `kubectl get kafkauser`. + +Also covered: client applications fail with a permission error after Kafka users and topics were migrated, and the topic creation form rejects uppercase letters in the topic name. + +A client whose user and secret exist but which gets `TopicAuthorizationException: Not authorized to access topics` needs the ACL and listener setup in [Kafka client connection failures: timeouts, TLS certificates, authorization, NodePort, and reverse DNS on ACP](Kafka_client_connection_failures_timeouts_TLS_certificates_authorization_NodePort_and_reverse_DNS_on_ACP.md). + +## Environment + +- A Data Services Kafka user is an `RdsKafkaUser` (`middleware.alauda.io/v1`, label `middleware.alauda.io/cluster: `). The operator creates a Strimzi `KafkaUser` (`kafka.strimzi.io/v1beta2`, label `strimzi.io/cluster: `) with the same name and spec. +- The `KafkaUser` is reconciled by the `user-operator` container of the Deployment `-entity-operator`, which also runs `topic-operator` and `tls-sidecar`. +- The password of a SCRAM user is kept in a Secret named `-password`, which the `RdsKafkaUser` references. + +## Root Cause + +### Cause 1: Simple authorization ACL rules are configured but not supported in the Kafka cluster configuration + +The user carries `authorization.type: simple` with ACL rules, but the instance was created without an authentication method, so authorization is not enabled in the cluster configuration. The user is not provisioned and its secret is never created. The Kafka version is not the cause: a Kafka 2.8 instance created the same way shows the same error. To use ACLs, `authorization.type: simple` must be enabled on the instance before topic or group ACLs are configured. + +### Cause 2: A user with the same name already exists + +The name entered in the Data Services form is already taken, for example by a `KafkaUser` created earlier directly through the Kafka Operator. Creating a user with the same name from Data Services fails. + +### Cause 3: The user-operator is hung while its Deployment still reports Ready + +The `user-operator` container of `-entity-operator` stops reconciling users, while the Deployment still shows `READY 1/1` and the pod is Running. Its log repeats: + +```text +WARNING: A connection to https:/// was leaked. Did you forget to close a response body? +WARN WatcherWebSocketListener:76 - Exec Failure java.io.FileNotFoundException /home/strimzi/.kube/config (No such file or directory) +``` + +Every new user of that instance stays in `Creating` and gets no secret. Authorization being disabled and a duplicate name are ruled out. Restarting the entity operator restores user creation. + +### Cause 4: Client permission error after migrating users and topics: the `-password` secret is missing + +The `-password` secret that the `RdsKafkaUser` references was not migrated, so client authentication fails. + +### Cause 5: Can a Kafka topic name contain uppercase letters? + +Yes. Kafka topic names allow uppercase letters, and so does the topic resource; only the creation form rejects them. + +## Resolution + +### Cause 1: Simple authorization ACL rules are configured but not supported in the Kafka cluster configuration + +```bash +kubectl -n get kafkauser -o jsonpath='{.spec.authorization}{"\n"}{.status.conditions}{"\n"}' +kubectl -n get kafka -o jsonpath='{.spec.kafka.authorization}{"\n"}{.spec.kafka.listeners}{"\n"}' +``` + +An empty `spec.kafka.authorization` together with the error confirms this cause. Choose one fix: + +- If the user does not need ACLs, remove `spec.authorization` from the `RdsKafkaUser`. The user is then provisioned and its secret appears. +- If ACLs are required, enable authorization on the instance through the platform, so that the cluster runs with `authorization.type: simple` and an authentication method on its listeners, and then configure the topic and group ACLs. + +Edit the `RdsKafkaUser`, not the `KafkaUser`: the operator writes the `RdsKafkaUser` spec over the `KafkaUser`. + +```bash +kubectl -n edit rdskafkauser +kubectl -n get rdskafkauser,kafkauser +``` + +### Cause 2: A user with the same name already exists + +```bash +kubectl -n get kafkauser,rdskafkauser +kubectl -n get kafkauser -o jsonpath='{.metadata.creationTimestamp}{"\n"}' +``` + +A `KafkaUser` older than the Data Services request was created earlier, outside this request. Create the Data Services user with a name that is not used in that namespace. + +### Cause 3: The user-operator is hung while its Deployment still reports Ready + +1. Rule out Cause 1 and Cause 2 with the commands above, then confirm the signature: + + ```bash + kubectl -n describe rdskafkauser + kubectl -n logs deploy/-entity-operator -c user-operator --tail=300 | grep -E 'Exec Failure|was leaked' + kubectl -n get deploy -entity-operator + ``` + + Repeated `Exec Failure java.io.FileNotFoundException /home/strimzi/.kube/config` lines, with the Deployment at `1/1`, confirm this cause. A `READY 1/1` Deployment does not prove the user-operator works. + +2. Restart the entity operator. + +> **Warning:** Restarting the entity operator restarts the `topic-operator`, `user-operator`, and `tls-sidecar` containers; topic and user reconciliation pauses until the new pod is Ready. Brokers are not restarted and clients are not affected. Do it only when users stay in `Creating` and the `user-operator` log shows the `Exec Failure ... /home/strimzi/.kube/config` lines. No rollback is needed; the Deployment recreates the pod with the same spec. + + ```bash + kubectl -n rollout restart deploy/-entity-operator + kubectl -n rollout status deploy/-entity-operator + ``` + +3. Check every user of the instance, not only the one reported. Each user that was stuck in `Creating` should become `Active`, and its secret should open: + + ```bash + kubectl -n get rdskafkauser + kubectl -n get kafkauser + kubectl -n get secret | grep + ``` + +If the `Exec Failure` lines come back later, collect the user-operator log and the operator version before restarting again. + +### Cause 4: Client permission error after migrating users and topics: the `-password` secret is missing + +```bash +kubectl -n get rdskafkauser -o jsonpath='{.spec.authentication}{"\n"}' +kubectl -n get secret -password +``` + +`Error from server (NotFound)` for `-password` confirms this cause. Reset the password of the user on its Data Services user page; the reset recreates the `-password` secret and restores access. + +> **Warning:** A reset gives the user a new password. Every client that authenticates as this user must be updated, and clients that keep the old password fail. Do it only when the `-password` secret of that user is missing. The old password cannot be restored, so plan the client updates first. + +```bash +kubectl -n get secret -password +kubectl -n get secret -o jsonpath='{.data.sasl\.jaas\.config}' | base64 -d +``` + +When users are migrated again, migrate each `-password` secret together with its `RdsKafkaUser`. + +### Cause 5: Can a Kafka topic name contain uppercase letters? + +Create the topic from YAML instead of the form. A Kubernetes object name cannot contain uppercase letters, so keep `metadata.name` lowercase and put the real topic name in `spec.topicName`. Start from an existing topic of the same instance so labels and fields match your operator version: + +```bash +kubectl -n get rdstopic -o yaml > new-topic.yaml +``` + +In `new-topic.yaml`, delete `metadata.resourceVersion`, `metadata.uid`, `metadata.creationTimestamp`, `metadata.generation`, `metadata.managedFields`, and the `status` block, then change only the two names: + +```yaml +metadata: + name: +spec: + topicName: +``` + +```bash +kubectl -n create -f new-topic.yaml +kubectl -n get rdstopic -o jsonpath='{.spec.topicName}{"\n"}' +``` + +Do not reuse a `spec.topicName` that another topic resource already uses; on some versions deleting the duplicate resource can clear the existing topic (see [Avoid Kafka Topic Data Loss When Deleting an Abnormal Topic CR](ecosystem/kafka/Kafka_Topic_Delete_Data_Loss_Issue.md)). + +## Diagnostic Steps + +1. Read the user objects: + + ```bash + kubectl -n get rdskafkauser -o yaml + kubectl -n get kafkauser -o yaml + kubectl -n get secret | grep + ``` + +2. Match the signature: + + - `Simple authorization ACL rules are configured but not supported in the Kafka cluster configuration.`: Cause 1. + - A `KafkaUser` with that name existed before the Data Services request: Cause 2. + - `phase: Creating` with no events, `spec.kafka.authorization` is `{"type":"simple"}`, no older user of that name, and `Exec Failure java.io.FileNotFoundException /home/strimzi/.kube/config` in the `user-operator` log: Cause 3. + - The user is `Active` but `-password` is missing after a migration: Cause 4. + - The user and its secrets exist and the client gets `TopicAuthorizationException`: the client connection article linked above. + +3. Check that the entity operator is running at all. A Deployment that is not Ready, or whose memory keeps rising, is covered in [Kafka instance creation times out on the entity-operator, shows failed, or the cluster operator restarts on ACP](Kafka_instance_creation_times_out_on_the_entity-operator_shows_failed_or_the_cluster_operator_restarts_on_ACP.md). + + ```bash + kubectl -n get deploy -entity-operator + kubectl -n top pod --containers | grep entity-operator + ``` diff --git a/docs/en/solutions/Keycloak_shows_HTTPS_required_the_CR_warns_about_hostname-strict_or_login_shows_Invalid_parameter_redirect_uri.md b/docs/en/solutions/Keycloak_shows_HTTPS_required_the_CR_warns_about_hostname-strict_or_login_shows_Invalid_parameter_redirect_uri.md new file mode 100644 index 000000000..ba9a58106 --- /dev/null +++ b/docs/en/solutions/Keycloak_shows_HTTPS_required_the_CR_warns_about_hostname-strict_or_login_shows_Invalid_parameter_redirect_uri.md @@ -0,0 +1,137 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Application Services +ProductsVersion: + - "4.3" +--- + +# Keycloak shows HTTPS required, the CR warns about hostname-strict, or login shows Invalid parameter redirect_uri + +## Issue + +A Keycloak 26.4 instance, managed by the Keycloak operator through a `Keycloak` custom resource, is running, but: + +- The admin console is opened through a NodePort address over plain HTTP, and the login page (`realms/master`, client `security-admin-console`) shows `We are sorry...` `HTTPS required`. +- The `Keycloak` CR is `Ready` (`status: "True"`), but its `HasErrors` condition says `warning: You need to specify these fields as the first-class citizen of the CR: hostname-strict`. +- An application or connector that signs in through Keycloak gets the realm login page with `We are sorry...` `Invalid parameter: redirect_uri`. + +## Root Cause + +### Cause 1: `HTTPS required`: the realm's `sslRequired` setting rejects the plain HTTP request + +Every Keycloak realm has an `sslRequired` setting with the values `none`, `external` (recommended for production), and `all`. With `external`, Keycloak requires HTTPS for requests from external addresses; with `all`, it requires HTTPS for every request. A request that the setting covers and that arrives over plain HTTP gets the `HTTPS required` page. Access through a NodePort over plain HTTP falls under that rule. Keycloak should be accessed over HTTPS. + +### Cause 2: `hostname-strict` is passed as a generic option instead of a first-class CR field + +The `hostname-strict` option was passed as a generic option of the instance (for example under `additionalOptions`) instead of through the dedicated fields of the CR. The operator expects hostname settings in `spec.hostname`; `spec.hostname.strict` is the entry point for `hostname-strict`. The CR stays `Ready`; the condition is a warning. + +### Cause 3: `Invalid parameter: redirect_uri`: the client's redirect URI is not registered + +Keycloak returns this page when it rejects the `redirect_uri` that the client sends in the authorization request. In standard Keycloak behavior that happens when the value is not covered by the **Valid redirect URIs** of the client named in `client_id`. Keycloak compares the URI exactly, except that a registered entry can end with a `*` wildcard. + +## Resolution + +### Cause 1: `HTTPS required`: the realm's `sslRequired` setting rejects the plain HTTP request + +1. Access Keycloak over HTTPS, for example through an Ingress with TLS or the instance's HTTPS endpoint, instead of an HTTP NodePort. + +2. For realms that you create, set `sslRequired` when the realm is imported through a `KeycloakRealmImport`: + + ```yaml + apiVersion: k8s.keycloak.org/v2alpha1 + kind: KeycloakRealmImport + metadata: + name: -import + namespace: + spec: + keycloakCRName: + realm: + realm: + enabled: true + sslRequired: external + ``` + + ```bash + kubectl api-resources | grep -i keycloak + kubectl apply -f -import.yaml + kubectl -n get keycloakrealmimports + ``` + + Use the API version that `kubectl api-resources` lists for `KeycloakRealmImport` in your cluster. `external` is the value recommended for production. + +### Cause 2: `hostname-strict` is passed as a generic option instead of a first-class CR field + +Move the hostname options into `spec.hostname` and remove `hostname-strict` from the generic options. + +> **Warning:** Changing the hostname settings restarts Keycloak and changes the URLs that Keycloak issues in redirects and tokens; clients that use a different address can stop working. Precondition: the external hostname that clients use is known and resolves to the Keycloak entry point. Save the CR first. Rollback: `kubectl apply -f keycloak-before.yaml`. + +```bash +kubectl -n get keycloak -o yaml > keycloak-before.yaml +kubectl -n edit keycloak +``` + +```yaml +spec: + hostname: + hostname: + strict: true + strictBackchannel: false # depending on the scenario +``` + +After saving, check that the `HasErrors` warning is gone (Diagnostic Steps, step 1). + +### Cause 3: `Invalid parameter: redirect_uri`: the client's redirect URI is not registered + +1. Take the `client_id` and the decoded `redirect_uri` from the failing request and compare them with the client registered in the realm (Diagnostic Steps, step 4). + +2. Make them match: either add the exact callback address that the application sends to the client's **Valid redirect URIs** in the realm, or change the application's callback address to one that is already registered. + + > **Warning:** Adding broad wildcard redirect URIs lets tokens be sent to any matching address. Precondition: register only the callback addresses that the application actually uses. Rollback: remove the added entry from the client. + +3. Retry the sign-in. + +## Diagnostic Steps + +1. Read the CR conditions: + + ```bash + kubectl -n get keycloak -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" "}{.message}{"\n"}{end}' + ``` + + `HasErrors` with `first-class citizen of the CR: hostname-strict` is Cause 2. + +2. Check how Keycloak is exposed and with which scheme the console is opened: + + ```bash + kubectl -n get svc,ingress + curl -sk -o /dev/null -w '%{http_code}\n' "https:///realms/master/protocol/openid-connect/auth?client_id=security-admin-console" + ``` + + A console opened through an `http://` NodePort address that shows `HTTPS required` is Cause 1. + +3. Read the `sslRequired` value of a realm from inside the Keycloak Pod, after logging in with an admin account: + + ```bash + kubectl -n exec -it -- sh -c \ + '/opt/keycloak/bin/kcadm.sh config credentials --config /tmp/kcadm.config --server http://localhost:8080 --realm master --user && + /opt/keycloak/bin/kcadm.sh get realms/ --config /tmp/kcadm.config --fields sslRequired' + ``` + + If the instance serves only TLS, use `--server https://localhost:8443` with `--truststore` or the instance certificate instead. + +4. For `Invalid parameter: redirect_uri`, copy the full URL of the failing login page from the browser address bar, decode it, and read the parameters: + + ```bash + python3 -c 'import sys, urllib.parse as u; q = u.parse_qs(u.urlsplit(sys.argv[1]).query); print(q.get("client_id"), q.get("redirect_uri"))' '' + ``` + + Then list the redirect URIs registered for that client, with the same `kcadm.sh` session as in step 3: + + ```bash + kubectl -n exec -it -- \ + /opt/keycloak/bin/kcadm.sh get clients -r -q clientId= --fields clientId,redirectUris --config /tmp/kcadm.config + ``` + + A `redirect_uri` that does not appear in `redirectUris` (and is not covered by a trailing `*` entry) is Cause 3. diff --git a/docs/en/solutions/Kiali_OpenID_login_fails_with_illegal_base64_data_at_input_byte_because_api_proxy_ca_data_is_malformed.md b/docs/en/solutions/Kiali_OpenID_login_fails_with_illegal_base64_data_at_input_byte_because_api_proxy_ca_data_is_malformed.md new file mode 100644 index 000000000..5fbabc991 --- /dev/null +++ b/docs/en/solutions/Kiali_OpenID_login_fails_with_illegal_base64_data_at_input_byte_because_api_proxy_ca_data_is_malformed.md @@ -0,0 +1,91 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Service Mesh +ProductsVersion: [] +--- + +# Kiali OpenID login fails with illegal base64 data at input byte because api_proxy_ca_data is malformed + +## Issue + +Opening Kiali for a workload cluster through the platform (`https:///clusters//kiali`) redirects to the ACP login page. After a successful platform login, Kiali reports that the OpenID login failed. The browser console shows: + +```text +Authentication openid_error: Unable to create a Kubernetes client from the auth token: illegal base64 data at input byte +``` + +The Kiali CR uses `spec.auth.strategy: openid`, and the Kiali pod itself is running. + +## Root Cause + +With the `openid` strategy on ACP, Kiali sends the user's token to the platform's Kubernetes API proxy (`spec.auth.openid.api_proxy`) and trusts that proxy with the CA given in `spec.auth.openid.api_proxy_ca_data`. That field must hold the platform CA certificate **base64-encoded**, exactly as it is stored in the `dex.tls` Secret on the `global` cluster. + +During installation the value is normally filled in with `export PLATFORM_CA=` and `envsubst < kiali.yaml | kubectl apply -f -`, which writes `api_proxy_ca_data: ${PLATFORM_CA}` into the Kiali CR. If the CA is not copied correctly at that point (truncated, extra or missing characters, line breaks, or the decoded PEM text instead of the base64 string), the field is not valid base64. The login through Dex succeeds, but Kiali fails when it decodes the CA to build the Kubernetes client, and reports `illegal base64 data at input byte `, where `` is the position of the first invalid character. + +## Resolution + +1. On the `global` cluster, save the platform CA as a single base64 line: + + ```bash + kubectl -n cpaas-system get secret dex.tls -o jsonpath='{.data.ca\.crt}' > platform-ca.b64 + ``` + + If `platform-ca.b64` is empty, use the `tls.crt` key instead: + + ```bash + kubectl -n cpaas-system get secret dex.tls -o jsonpath='{.data.tls\.crt}' > platform-ca.b64 + ``` + + Check that the file decodes to a certificate: + + ```bash + base64 -d platform-ca.b64 | openssl x509 -noout -subject -enddate + ``` + +2. Copy `platform-ca.b64` to a host with `kubectl` access to the workload cluster where Kiali runs, without opening and re-pasting it in an editor. + +3. Find the Kiali CR and write the correct value into `api_proxy_ca_data`: + + ```bash + kubectl get kiali -A + kubectl -n patch kiali --type merge \ + -p "{\"spec\":{\"auth\":{\"openid\":{\"api_proxy_ca_data\":\"$(tr -d '\n' < platform-ca.b64)\"}}}}" + ``` + + Editing the field directly with `kubectl edit kiali` works too, as long as the value is pasted as one unbroken line. + +4. Wait for the Kiali operator to roll out the new configuration, then open Kiali through the platform again and log in. Kiali is briefly unavailable while its pod is replaced. + + ```bash + kubectl -n get pods -w | grep kiali + ``` + +## Diagnostic Steps + +1. Check that the value in the CR decodes to a certificate. A `base64: invalid input` error, or an `openssl` error, confirms this issue: + + ```bash + kubectl -n get kiali \ + -o jsonpath='{.spec.auth.openid.api_proxy_ca_data}' | base64 -d | openssl x509 -noout -subject + ``` + +2. Compare the CR value with the `global` cluster's `dex.tls` value character by character, for example by comparing their checksums: + + ```bash + kubectl -n get kiali -o jsonpath='{.spec.auth.openid.api_proxy_ca_data}' | md5sum + kubectl -n cpaas-system get secret dex.tls -o jsonpath='{.data.ca\.crt}' | md5sum # on global; use tls.crt if ca.crt is empty + ``` + +3. Check the other OpenID settings while you are there. `api_proxy` should be `/kubernetes/`, and `issuer_uri` should point to the platform Dex (`/dex`): + + ```bash + kubectl -n get kiali -o jsonpath='{.spec.auth.openid}{"\n"}' + ``` + +4. Look for the same error in the Kiali log to confirm that the failure happens when the Kubernetes client is created, not earlier at the Dex redirect: + + ```bash + kubectl -n logs deploy/kiali --tail=300 | grep -iE 'openid|base64|token' + ``` diff --git a/docs/en/solutions/Kiali_will_not_open_or_log_in_shows_Core_istiod_Not_found_cannot_reach_VictoriaMetrics_or_shows_too_many_namespaces.md b/docs/en/solutions/Kiali_will_not_open_or_log_in_shows_Core_istiod_Not_found_cannot_reach_VictoriaMetrics_or_shows_too_many_namespaces.md new file mode 100644 index 000000000..7c6f8ba53 --- /dev/null +++ b/docs/en/solutions/Kiali_will_not_open_or_log_in_shows_Core_istiod_Not_found_cannot_reach_VictoriaMetrics_or_shows_too_many_namespaces.md @@ -0,0 +1,255 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Service Mesh +ProductsVersion: + - "4.1" + - "4.2" + - "4.3" +--- + +# Kiali will not open or log in, shows Core istiod Not found, cannot reach VictoriaMetrics, or shows too many namespaces + +## Issue + +Kiali (Alauda Build of Kiali) on Service Mesh v2 shows one of these problems: + +- Kiali access fails and its log shows nothing useful. OpenID login goes to a wrong address: behind the business-cluster ALB, `GET .../kiali/api/auth/info` returns an authorization address built from the business cluster domain instead of the ACP platform domain. +- The page shows `Cannot start authentication because it is not possible to use OpenId's authorization code flow. Check Kiali logs for more details.` at `/kiali/api/auth/openid_redirect`. +- Kiali loads, but its metrics are empty because the Kiali server cannot reach the cluster's VictoriaMetrics. +- The Overview page shows the cluster badge `Kubernetes` in error, Cluster Status `Core istiod Not found`, `Control planes (0)`, and `Data planes (0)`. +- A user who has access to only one namespace should see only that namespace in Kiali. + +A failure after the ACP login with `Authentication openid_error: Unable to create a Kubernetes client from the auth token: illegal base64 data at input byte ` is covered in "Kiali OpenID login fails with illegal base64 data at input byte because api_proxy_ca_data is malformed". + +## Root Cause + +### Cause 1: OpenID login uses the business cluster domain because `web_fqdn` is not set and the ALB sets `X-Forwarded-Host` + +The authorization endpoint that Kiali returns from `/api/auth/info` is built from `conf.Server.WebFQDN` (`GuessKialiURL` in the kiali-server source). When that value is empty, Kiali falls back in order to the `X-Forwarded-Host` request header, then the request URL host name, then the request `Host`. With `web_fqdn` not configured, the business-cluster ALB sets `X-Forwarded-Host` to the business cluster domain, so the authorization address is wrong. The ALB log on the business cluster shows Kiali requests (rule `istio-system/kiali`, upstream port `20001`) with the business cluster domain in `host`, while the same requests through the global cluster ALB under `/clusters//kiali/` carry the ACP platform domain. + +Switching the Kiali CR to `spec.auth.strategy: anonymous` loads the console through the ACP platform domain at `/clusters//kiali/console/overview`, which isolates the fault to the OpenID address. OpenID access is restored by setting `web_fqdn` and the related server settings in the Kiali CR, turning off automatic ingress generation, and creating the Ingress `kiali` in `istio-system` by hand with an annotation that clears `X-Forwarded-Host`. + +### Cause 2: OpenID discovery times out because of a node DNS search domain + +The Kiali log shows: + +```text +WRN Error when fetching OpenID provider's metadata: Get "https:///dex/.well-known/openid-configuration": context deadline exceeded (Client.Timeout exceeded while awaiting headers) +WRN Could not read the session: session [not found]: cookie [kiali-token-Kubernetes] does not exist in request +``` + +The Kiali Pod runs on a node that has an extra DNS search domain configured. Pods with the default DNS policy inherit the node's search domains, and that search domain affects name resolution, so Kiali's request to the Dex discovery endpoint times out. Removing the search domain from the node makes the Kiali page accessible. + +### Cause 3: Kiali cannot reach VictoriaMetrics through the business-cluster ALB + +Kiali opens, but a network problem prevents the Kiali server from reaching the cluster's VictoriaMetrics through the business-cluster ALB. Pointing the Kiali monitoring address at the ACP platform domain address is a workaround; the network path itself still needs a fix. + +### Cause 4: `Core istiod Not found` because Istio is not deployed + +The Kiali log shows: + +```text +WRN Cannot resolve local cluster name. Err: no istiod deployment found, no clusterName determined. Falling back to [Kubernetes] +``` + +Istio is not deployed in the cluster, so Kiali has no control plane to show. + +### Cause 5: namespace visibility follows ACP RBAC by default + +Kiali with `auth.strategy: openid` and `auth.openid.disable_rbac: false` enforces ACP RBAC, so no extra synchronization is needed. Namespaces that the user has no permission for may still be listed, but no data is displayed for them. A user given a RoleBinding in only one namespace sees only that namespace in Kiali. + +## Resolution + +### Cause 1: OpenID login uses the business cluster domain because `web_fqdn` is not set and the ALB sets `X-Forwarded-Host` + +1. Confirm the host that the ALB passes to Kiali on the business cluster, and the current Kiali settings: + + ```bash + kubectl logs -l service_name=alb2-cpaas-system --tail 200 -n cpaas-system | grep '//kiali/' + kubectl get kiali -n istio-system + kubectl get kiali -n istio-system -o jsonpath='{.spec.auth}{"\n"}{.spec.server}{"\n"}{.spec.deployment.ingress}{"\n"}' + ``` + +2. Optionally isolate the fault by switching to `anonymous` for a short test. + + > **Warning:** With `strategy: anonymous`, anyone who can reach the URL uses Kiali without logging in. Precondition: only as a short isolation test for Cause 1, with the Kiali CR backed up. Rollback: patch `strategy` back to `openid` (first command of the next step). + + ```bash + kubectl get kiali -n istio-system -o yaml > /tmp/kiali-cr.bak.yaml + kubectl patch kiali -n istio-system --type=merge -p '{"spec":{"auth":{"strategy":"anonymous"}}}' + ``` + +3. Set `spec.server.web_fqdn` to the ACP platform domain that the browser uses, and turn off automatic ingress generation with `spec.deployment.ingress.enabled: false`. Align the related server settings (`spec.server.web_root`, `spec.server.web_port`, `spec.server.web_schema`) with the platform URL in the browser. + + > **Warning:** The Kiali operator reconciles, Kiali restarts, and the generated ingress is removed, so Kiali is unreachable until the Ingress in the next step exists. Precondition: only when Cause 1 is confirmed (wrong host in `/api/auth/info` or in the ALB log, and `web_fqdn` empty) and `/tmp/kiali-cr.bak.yaml` exists. Rollback: restore the previous `spec.server` and `spec.deployment.ingress` values from the backup, and delete the manual Ingress. + + ```bash + kubectl patch kiali -n istio-system --type=merge -p '{"spec":{"auth":{"strategy":"openid"}}}' + kubectl patch kiali -n istio-system --type=merge \ + -p '{"spec":{"server":{"web_fqdn":""},"deployment":{"ingress":{"enabled":false}}}}' + ``` + +4. Create the Ingress, clearing `X-Forwarded-Host`: + + ```yaml + apiVersion: networking.k8s.io/v1 + kind: Ingress + metadata: + name: kiali + namespace: istio-system + annotations: + alb.ingress.cpaas.io/rewrite-request: '{"headers_var": {"X-Forwarded-Host": ""}}' + nginx.ingress.kubernetes.io/backend-protocol: HTTP + nginx.ingress.kubernetes.io/secure-backends: "false" + spec: + ingressClassName: cpaas-system + rules: + - http: + paths: + - path: /clusters//kiali + pathType: Prefix + backend: + service: + name: kiali + port: + number: 20001 + ``` + + ```bash + kubectl apply -f kiali-ingress.yaml + kubectl get ingress -n istio-system kiali + ``` + +5. Log in through the ACP platform domain. OpenID login succeeds. + +### Cause 2: OpenID discovery times out because of a node DNS search domain + +1. Confirm the Dex discovery timeout and find the node that runs Kiali; compare its resolver configuration with a node where Kiali works: + + ```bash + kubectl -n istio-system logs deploy/kiali --tail=300 | grep -i -E 'openid|context deadline' + kubectl -n istio-system get pod -o wide | grep kiali + kubectl -n istio-system exec deploy/kiali -- cat /etc/resolv.conf + # on the Kiali node and on a healthy node + cat /etc/resolv.conf + ``` + + An extra `search` domain on the Kiali node that the healthy nodes do not have matches this cause. + +2. Remove the extra search domain from the node. + + > **Warning:** Changing the node resolver changes name resolution for the node and for Pods created on it afterwards. Precondition: only when Cause 2 is confirmed. Back up the file first. If the file is generated by a network manager, remove the search domain in that manager's configuration instead, or it comes back. Rollback: `cp /etc/resolv.conf.bak /etc/resolv.conf` (or restore the network manager setting). + + ```bash + cp /etc/resolv.conf /etc/resolv.conf.bak + vi /etc/resolv.conf + ``` + +3. A Pod's `/etc/resolv.conf` is written when the Pod is created, so recreate the Kiali Pod if it still times out: + + ```bash + kubectl -n istio-system delete pod + ``` + + The Kiali page becomes accessible. + +### Cause 3: Kiali cannot reach VictoriaMetrics through the business-cluster ALB + +1. Read where Kiali points for metrics and confirm the failure in its log: + + ```bash + kubectl get kiali -n istio-system -o jsonpath='{.spec.external_services.prometheus}{"\n"}' + kubectl -n istio-system logs deploy/kiali --tail=200 | grep -i -E 'prometheus|metrics|timeout' + ``` + +2. As a workaround, point Kiali at the VictoriaMetrics address on the ACP platform domain. + + > **Warning:** Kiali restarts and queries metrics through the platform domain, so the platform entry carries Kiali's metric queries. Precondition: only when Cause 3 is confirmed. Back up the CR first. Rollback: `kubectl apply -f /tmp/kiali-cr-monitoring.bak.yaml`. + + ```bash + kubectl get kiali -n istio-system -o yaml > /tmp/kiali-cr-monitoring.bak.yaml + kubectl patch kiali -n istio-system --type=merge \ + -p '{"spec":{"external_services":{"prometheus":{"url":""}}}}' + ``` + +3. Fix the network path from Kiali to the business-cluster ALB separately. + +### Cause 4: `Core istiod Not found` because Istio is not deployed + +```bash +kubectl get istio,istiorevision -A +kubectl get deploy -A | grep istiod +``` + +No `Istio` object and no `istiod` Deployment confirm this cause. Deploy Istio by following the Service Mesh v2 installation documentation. Kiali shows the control plane after `istiod` is running. + +### Cause 5: namespace visibility follows ACP RBAC by default + +1. Confirm the Kiali auth settings. A working layout on the `global` cluster: + + ```yaml + auth: + strategy: openid + openid: + api_proxy: /kubernetes/global + api_proxy_ca_data: + client_id: alauda-auth + disable_rbac: false + insecure_skip_verify_tls: true + issuer_uri: /dex + username_claim: email + ``` + + ```bash + kubectl get kiali -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" "}{.spec.auth.strategy}{" disable_rbac="}{.spec.auth.openid.disable_rbac}{"\n"}{end}' + ``` + +2. With `disable_rbac: false`, grant the user access only to the target namespace, for example with a RoleBinding. With `username_claim: email`, the subject name is the user's email. Bind the namespace role that the user should have: + + ```yaml + apiVersion: rbac.authorization.k8s.io/v1 + kind: RoleBinding + metadata: + name: -kiali-view + namespace: + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: User + name: + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: + ``` + + ```bash + kubectl apply -f -kiali-rolebinding.yaml + ``` + +3. The user sees only that namespace in Kiali. + +## Diagnostic Steps + +1. Read the Kiali log first; most of these causes have a distinct line: + + ```bash + kubectl -n istio-system logs deploy/kiali --tail=300 | grep -E 'WRN|ERR' + ``` + +2. Check the auth and server settings and how Kiali is exposed: + + ```bash + kubectl get kiali -A -o yaml | grep -n -E 'strategy|web_fqdn|web_root|ingress|disable_rbac|prometheus' + kubectl get ingress -n istio-system + ``` + +| What you see | Cause | +| --- | --- | +| `/api/auth/info` or the ALB log carries the business cluster domain, and `web_fqdn` is empty | 1 | +| `Cannot start authentication ... authorization code flow`, log `context deadline exceeded` on `/dex/.well-known/openid-configuration` | 2 | +| Kiali up, metrics missing, Kiali cannot reach VictoriaMetrics through the business-cluster ALB | 3 | +| `Core istiod Not found`, `Control planes (0)`, log `no istiod deployment found ... Falling back to [Kubernetes]` | 4 | +| A user should see only their own namespaces | 5 | +| After the ACP login, `openid_error: ... illegal base64 data at input byte ` | see "Kiali OpenID login fails with illegal base64 data at input byte because api_proxy_ca_data is malformed" | diff --git a/docs/en/solutions/Kube-OVN_Pods_fail_with_AddressOutOfRange_AddressConflict_or_no_address_allocated_on_ACP.md b/docs/en/solutions/Kube-OVN_Pods_fail_with_AddressOutOfRange_AddressConflict_or_no_address_allocated_on_ACP.md new file mode 100644 index 000000000..485d61a9d --- /dev/null +++ b/docs/en/solutions/Kube-OVN_Pods_fail_with_AddressOutOfRange_AddressConflict_or_no_address_allocated_on_ACP.md @@ -0,0 +1,171 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "3.18" + - "4.0" + - "4.2" + - "4.3" +--- + +# Kube-OVN Pods fail with AddressOutOfRange, AddressConflict or no address allocated on ACP + +## Issue + +On a Kube-OVN cluster, a Pod stays in `ContainerCreating` or comes up with the wrong address. Typical reports: + +- The Pod event is `AcquireAddressFailed` with `AddressOutOfRange`, or a pinned address flaps `AddressConflict` after a fast delete. +- Pods in some namespaces fail while other namespaces work, with `request ip return 500 no address allocated to pod / provider ovn, please see kube-ovn-controller logs to find errors`. +- `arping` from a Pod gets a reply from an external host, or an external host on the underlay VLAN stops answering while a Pod holds the same address. +- In the console's fixed-IP dropdown, the subnet's `.2` address is offered although it is the u2o interconnection address. +- On ACP 3.18.2 the console does not allow adding a second IPv4 CIDR to an existing Subnet. +- The question is whether the fixed IP field takes a range such as `..`. + +Restarting `ovs-ovn` fixes none of these. + +Related articles: "Calico Pods get no address: disabled default IPPool, Raven fixed IP mismatch, reserved IPs or deleted raven on ACP"; "Multus secondary networks: unexpected end of JSON input, duplicated addresses or several addresses on one NIC on ACP"; "Kube-OVN fixed IP rejected or no address allocated because a stale IP object or logical switch port holds it on ACP". + +## Root Cause + +### Cause 1: on ACP 3.18.2, Subnet spec.cidrBlock holds one IPv4 CIDR + +A single-stack Subnet stores one CIDR in `spec.cidrBlock`; a dual-stack Subnet stores one IPv4 and one IPv6 CIDR. The console can edit that value but cannot append another IPv4 block. Editing the original CIDR in place strands the Pods that already use it. Likewise, `spec.protocol` cannot be changed after creation, so a Subnet created as `IPv4` cannot serve a workload that asks for an IPv6 address. + +### Cause 2: AddressOutOfRange in an underlay fixed-IP Deployment + +Two problems appear together: + +- The Subnet's NIC binding is wrong, so allocation is refused until the VLAN or ProviderNetwork points at the right interface. +- The number of pinned addresses equals `spec.replicas` while `strategy.type` is `RollingUpdate`. The new Pod asks for an address the old Pod has not released, and `kube-ovn-controller` still holds the old IP object. + +The same race makes a pinned address on a Deployment flap `AddressConflict` after a fast delete, because the ReplicaSet starts the replacement immediately. A StatefulSet waits until the previous ordinal is gone. + +### Cause 3: an external device or a previous platform still uses addresses in the underlay CIDR + +Dynamic allocation hands out any address of the underlay CIDR that is not listed in `spec.excludeIps`. When an external host, the gateway, or an old platform sharing the VLAN uses such an address, a Pod can be given it: `arping` from the Pod gets a reply from the external host, ICMP to that host stops returning, and OVN logs `MAC_Binding` conflicts for one `logical_port` and IP, sometimes with `OVNSB commit failed`. An address that is "usually free" is not free. + +### Cause 4: the u2o interconnection address is offered as a free static IP + +With `u2oInterconnection` enabled, the subnet's second address (`.2`) is by default the u2o interconnection address. Kube-OVN creates a `u2o-interconnection..` IP object only when the u2o address cannot use `.2` (for example because `.2` is in `excludeIps`); when `.2` is used, there is no IP object for it, which is expected. The defect is that the console then offers `.2` as a free fixed IP, and a workload pinned to it collides with the interconnection hop. It is fixed in versions after 4.2.3. + +### Cause 5: failed to patch resource ... spec.initContainers[n].lifecycle: Forbidden + +Pods in a few namespaces fail with `FailedCreatePodSandBox ... plugin type="kube-ovn" failed (add): RPC failed; request ip return 500 no address allocated to pod / provider ovn`, and `kube-ovn-controller` may also re-elect its leader. The controller log shows its annotation patch (`ovn.kubernetes.io/allocated`, `ovn.kubernetes.io/ip_address`, `ovn.kubernetes.io/logical_switch` and related keys) rejected: + +```text +failed to patch resource with json merge patch ... Pod "" is invalid: [spec.initContainers[1].lifecycle: Forbidden: may not be set for init containers without restartPolicy=Always, spec.initContainers[1].readinessProbe: Forbidden: ..., spec: Forbidden: pod updates may not change fields other than ...] +``` + +The diff in the message shows `RestartPolicy: &"Always"` changed to `nil` on the Istio native sidecar init container. The affected namespaces ran workloads into which a monitoring operator (`datakit-operator`) injected annotations alongside the Istio sidecar. The spec submitted with the annotation patch had `restartPolicy` cleared from the sidecar init container, so the API server rejected the update and no IP annotation was written. How the injector altered the spec was not determined. Restarting the injecting operator restored allocation. + +### Cause 6: the fixed IP field does not accept a range + +The fixed IP field accepts individual addresses only. Several addresses can be configured, one entry each; range notation is not accepted. This is a product limitation. + +## Resolution + +### Cause 1: on ACP 3.18.2, Subnet spec.cidrBlock holds one IPv4 CIDR + +Leave the existing `cidrBlock` unchanged. Create a second Subnet for the new prefix, bind the same namespaces with `spec.namespaces` (or set `ovn.kubernetes.io/logical_switch` on the workloads that should move), and migrate by recreating those Pods. Use the same create-and-migrate path for a protocol mismatch. + +> **Warning:** Recreating a Pod moves it to the new Subnet and drops its connections. Keep the old Subnet until every migrated workload is Ready. + +```bash +kubectl get subnet -o yaml > /root/subnet.before.yaml +kubectl apply -f - <<'EOF' +apiVersion: kubeovn.io/v1 +kind: Subnet +metadata: + name: +spec: + protocol: IPv4 + cidrBlock: 203.0.113.0/24 + gateway: 203.0.113.1 + excludeIps: + - 203.0.113.1 + namespaces: + - +EOF +``` + +Replace the documentation prefix with the real one. On releases newer than 3.18.2, check whether multiple IPv4 CIDRs are accepted before assuming this limit. + +### Cause 2: AddressOutOfRange in an underlay fixed-IP Deployment + +1. Fix the NIC binding of the underlay Subnet first if it is wrong. +2. Switch the Deployment to `strategy.type: Recreate`, or pin more addresses than `replicas` so a rolling surge has a free one. For a workload that must keep stable addresses across deletes, use a StatefulSet. +3. If an IP object is still owned by a deleted Pod, delete it only after the old Pod is gone and no running Pod uses the address. + +> **Warning:** `Recreate` stops every replica before starting new ones. Deleting an IP object frees the address for anyone; if a live Pod still uses it, you create a duplicate. Save the IP object first. Setting `RollingUpdate` back is the rollback, and brings the race back if the pin list is still as long as `replicas`. + +```bash +kubectl get pod -n -o jsonpath='{.metadata.annotations.ovn\.kubernetes\.io/ip_address}{"\n"}' +kubectl get ip -o wide | grep
+kubectl get ip -o yaml > /root/ip-object.before.yaml +kubectl delete ip +kubectl patch deploy -n --type=merge -p '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}' +``` + +### Cause 3: an external device or a previous platform still uses addresses in the underlay CIDR + +Add the external hosts, the gateway, and every address still used by a previous platform or planned for another purpose to `spec.excludeIps`, then delete the Pods that hold a conflicting address so they are allocated again outside the list. + +> **Warning:** Deleting the Pods restarts them and, if they are not pinned, gives them new addresses. `excludeIps` does not take back an address that is already allocated; the Pod delete does. Save the Subnet and restore `excludeIps` if you excluded too much. + +```bash +kubectl get subnet -o yaml > /root/subnet.before.yaml +kubectl patch subnet --type=json -p='[{"op":"add","path":"/spec/excludeIps/-","value":"198.51.100.12"}]' +kubectl delete pod -n +``` + +Repeat the patch for each address. If `excludeIps` does not exist yet, create the array with a merge patch instead of `add` on `/-`. + +### Cause 4: the u2o interconnection address is offered as a free static IP + +Add the u2o address and the gateway to `excludeIps` so the console cannot offer them. No Pod restart is needed; the exclusion applies to the next allocation. Upgrading past 4.2.3 removes the need for this workaround. + +```bash +kubectl get subnets.kubeovn.io -o jsonpath='{.status.u2oInterconnectionIP}{"\n"}{.spec.excludeIps}{"\n"}' +kubectl patch subnet --type=json -p='[{"op":"add","path":"/spec/excludeIps/-","value":""}]' +``` + +### Cause 5: failed to patch resource ... spec.initContainers[n].lifecycle: Forbidden + +Find the operator that injects into the affected namespaces, restart it, and recreate the failed Pods. Compare a failing Pod with one from a working namespace: look for the injector's annotations and for `restartPolicy: Always` on the sidecar init container. + +> **Warning:** While the injecting operator restarts, Pods created in that window miss its injection. Recreating the failed Pods restarts those workloads. Only when the controller log shows the init-container `Forbidden` rejection. + +```bash +kubectl -n kube-system logs deploy/kube-ovn-controller --tail=2000 | grep -A3 'failed to patch resource' +kubectl -n get pod -o yaml | grep -n -A3 -E 'initContainers|restartPolicy' +kubectl get deploy -A | grep -i datakit +kubectl -n rollout restart deploy/ +kubectl -n delete pod +``` + +### Cause 6: the fixed IP field does not accept a range + +Enter each address of the range as a separate entry, then check the addresses the workload actually carries: + +```bash +kubectl -n get deploy -o jsonpath='{.spec.template.metadata.annotations}{"\n"}' +kubectl -n get pod -o wide +``` + +## Diagnostic Steps + +```bash +kubectl describe pod -n +kubectl get subnet -o yaml +kubectl get ip -o wide +kubectl -n kube-system logs deploy/kube-ovn-controller --tail=500 | grep -E 'AddressOutOfRange|AddressConflict|has been assigned to|failed to patch' +``` + +- `AddressOutOfRange`: the requested address is outside `cidrBlock`, or the NIC binding made the Subnet unusable, or the pin list is as long as `replicas` (Cause 2). A missing second CIDR on 3.18.2 is Cause 1. +- `AddressConflict`: an IP object or a live owner already has the address (Cause 2). If no running Pod holds it, see "Kube-OVN fixed IP rejected or no address allocated because a stale IP object or logical switch port holds it on ACP". +- An external host answering for a Pod address, or `MAC_Binding` conflicts: Cause 3. +- `.2` offered as a fixed IP while `status.u2oInterconnectionIP` is `.2`: Cause 4. +- `failed to patch resource` with init-container `Forbidden` text: Cause 5. diff --git a/docs/en/solutions/Kube-OVN_fixed_IP_NoAvailableAddress_spaces_in_the_list_ip_pool_not_reserved_several_subnets_or_leftover_port_on_ACP.md b/docs/en/solutions/Kube-OVN_fixed_IP_NoAvailableAddress_spaces_in_the_list_ip_pool_not_reserved_several_subnets_or_leftover_port_on_ACP.md new file mode 100644 index 000000000..0579678a6 --- /dev/null +++ b/docs/en/solutions/Kube-OVN_fixed_IP_NoAvailableAddress_spaces_in_the_list_ip_pool_not_reserved_several_subnets_or_leftover_port_on_ACP.md @@ -0,0 +1,127 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.16" + - "4.0" + - "4.2" + - "4.3" +--- + +# Kube-OVN fixed IP NoAvailableAddress: spaces in the list, ip_pool not reserved, several subnets or leftover port on ACP + +## Issue + +A Kube-OVN workload with fixed IPs, or in a namespace bound to several subnets, cannot get an address although free addresses exist. Typical reports: + +- A new fixed-IP Deployment on ACP 3.16.2 fails with `AcquireAddressFailed: NoAvailableAddress` even though the address is free. +- Addresses named in `ovn.kubernetes.io/ip_pool` are taken by other Pods before the fixed-IP workload starts. +- A restarted StatefulSet Pod gets no IP in a namespace bound to two subnets, while the other subnet has plenty of free addresses. +- After a restart, a StatefulSet Pod gets no IP and the controller log says `port already exists but in switch `. +- A Pod was given an address from the subnet's reserved range, and another workload's fixed-IP Pod then failed. + +Related articles: "Kube-OVN fixed IP occupied or not released: held by another Pod, node down, duplicate entry or RollingUpdate on ACP". + +## Root Cause + +### Cause 1: spaces in the fixed IP list + +On ACP 3.16.2 with Kube-OVN 1.12.35, fixed-IP parsing is not consistent across the create and update paths. A list with spaces after the commas (`192.0.2.10, 192.0.2.11`) fails on some paths with `AcquireAddressFailed: NoAvailableAddress` and works on others. The address is inside the subnet, not in `excludeIps`, and no Pod uses it. Restarting `kube-ovn-controller`, waiting, or restarting the workload several times can clear the error temporarily. + +### Cause 2: ovn.kubernetes.io/ip_pool does not reserve addresses that no Pod is using + +An address listed in `ovn.kubernetes.io/ip_pool` and not currently held by a Pod is still free for dynamic allocation, so another Pod can take it first. Listing those addresses in the Subnet's `spec.excludeIps` stops dynamic allocation from using them, while the fixed-IP mechanism can still assign them. + +### Cause 3: a namespace bound to several overlay subnets + +A Kube-OVN overlay namespace can be bound to more than one Subnet. A Pod that does not select a Subnet may get an address from any of them; `ovn.kubernetes.io/logical_switch` selects exactly one. The CIDRs must not overlap. + +On ACP 3.16.2 (Kube-OVN v1.12.35) a confirmed defect makes a restarted StatefulSet Pod fail to get an IP when all three conditions hold: the namespace is bound to two or more subnets, the Pod template does not name a subnet, and the Pod's original subnet has no free address at that moment. The Pod does not fall back to the other subnet. The event is `AcquireAddressFailed` `NoAvailableAddress`, and the controller log repeats `allocate v4 , v6 , mac for /-0 from subnet ` followed by `error syncing '/-0': NoAvailableAddress, requeuing`. + +### Cause 4: port already exists but in switch + +A StatefulSet Pod keeps its name across restarts. In a namespace bound to several subnets, the first chosen subnet had no free address, and OVN had not removed the old Pod's logical switch port in the other subnet, so the new port with the same name was refused. The controller log shows an allocation from one subnet followed by `port already exists but in switch `. This is an OVN cleanup issue and is rarely seen. + +### Cause 5: a Pod received an address from the subnet's reserved range + +After an image update a workload restarted and was given an address inside the reserved range of its subnet, which was another workload's fixed IP, so the fixed-IP Pod could not start. This is an IPAM defect in the Kube-OVN component of that release, fixed in ACP 3.18 and later. + +## Resolution + +### Cause 1: spaces in the fixed IP list + +Write the list comma-separated with no blanks (`192.0.2.10,192.0.2.11`). Restart `kube-ovn-controller` only as a temporary recovery when the addresses are free and the list still contains spaces. + +> **Warning:** Restarting `kube-ovn-controller` pauses address allocation in the cluster until the new Pod is Ready. It does not rewrite the annotation, so the next create can fail again while spaces remain. Save the Deployment first. + +```bash +kubectl get deploy -n -o yaml > /root/deploy-ip.before.yaml +kubectl edit deploy -n +kubectl -n kube-system rollout restart deploy/kube-ovn-controller +``` + +### Cause 2: ovn.kubernetes.io/ip_pool does not reserve addresses that no Pod is using + +Add each pool address to the Subnet's `spec.excludeIps` before dynamic Pods can take it. + +> **Warning:** `excludeIps` changes which addresses dynamic Pods may receive; an address already allocated is not taken back. Save the Subnet; restoring `excludeIps` is the rollback. + +```bash +kubectl get subnet -o yaml > /root/subnet-reserve.before.yaml +kubectl patch subnet --type=json -p='[{"op":"add","path":"/spec/excludeIps/-","value":"192.0.2.10"}]' +``` + +If `excludeIps` does not exist yet, create the array with a merge patch instead of `add` on `/-`. + +### Cause 3: a namespace bound to several overlay subnets + +Pin workloads that must stay on one prefix, and every StatefulSet, to one Subnet with `ovn.kubernetes.io/logical_switch` in the Pod template (or give them fixed IPs). If a Pod is already stuck, free addresses in its original subnet, or restart `kube-ovn-controller` as a temporary fix. + +> **Warning:** Changing the StatefulSet Pod template restarts its Pods one ordinal at a time. Restarting `kube-ovn-controller` pauses allocation cluster-wide. Removing the annotation is the rollback and brings the defect back. + +```bash +kubectl get subnet -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidrBlock,NS:.spec.namespaces,AVAIL:.status.v4availableIPs' +kubectl patch sts -n --type=merge -p '{"spec":{"template":{"metadata":{"annotations":{"ovn.kubernetes.io/logical_switch":""}}}}}' +``` + +### Cause 4: port already exists but in switch + +Restart the StatefulSet Pod. If the error returns, delete only the leftover logical switch port named after the Pod, then restart the Pod. Pinning the StatefulSet to one subnet (Cause 3) prevents it. + +> **Warning:** `lsp-del` removes the port from the OVN northbound database; deleting a port of a running Pod cuts its network until it is recreated. Only when the Pod named in the port is not running. Record the `list` output first. + +```bash +kubectl ko nbctl list logical_switch_port | grep -B5 -A5 +kubectl ko nbctl lsp-del +kubectl -n delete pod +``` + +### Cause 5: a Pod received an address from the subnet's reserved range + +Upgrade to ACP 3.18 or later, which contains the fix. To unblock the fixed-IP Pod before that, recreate the Pod that took the reserved address so it is allocated again; on an affected version the wrong allocation can recur. Collect the `ovs-ovn` logs from all nodes and `kubectl get pod -A -o wide` first if the allocation needs to be traced. + +> **Warning:** Deleting the Pod that holds the address interrupts that workload and gives it a new address. Confirm it is not the workload that is supposed to own the address. + +```bash +kubectl get subnet -o jsonpath='{.spec.excludeIps}{"\n"}' +kubectl get pod -A -o wide | grep +kubectl -n delete pod +``` + +## Diagnostic Steps + +```bash +kubectl describe pod -n +kubectl get subnet -o yaml +kubectl get ns -o yaml | grep -i ovn +kubectl -n kube-system logs deploy/kube-ovn-controller --tail=2000 | grep -E 'NoAvailableAddress|port already exists|allocate v4' +``` + +- `NoAvailableAddress` with spaces in the fixed-IP list: Cause 1. +- An `ip_pool` address held by a Pod of another workload that never had it pinned: Cause 2. +- A StatefulSet in a multi-subnet namespace failing with `NoAvailableAddress` only when its original subnet is full: Cause 3. +- `port already exists but in switch`: Cause 4. +- An address inside `excludeIps` assigned to a dynamic Pod: Cause 5. diff --git a/docs/en/solutions/Kube-OVN_fixed_IP_occupied_or_not_released_held_by_another_Pod_node_down_duplicate_entry_or_RollingUpdate_on_ACP.md b/docs/en/solutions/Kube-OVN_fixed_IP_occupied_or_not_released_held_by_another_Pod_node_down_duplicate_entry_or_RollingUpdate_on_ACP.md new file mode 100644 index 000000000..f83cdd885 --- /dev/null +++ b/docs/en/solutions/Kube-OVN_fixed_IP_occupied_or_not_released_held_by_another_Pod_node_down_duplicate_entry_or_RollingUpdate_on_ACP.md @@ -0,0 +1,150 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.16" + - "4.0" + - "4.2" + - "4.3" +--- + +# Kube-OVN fixed IP occupied or not released: held by another Pod, node down, duplicate entry or RollingUpdate on ACP + +## Issue + +A workload with a fixed IP cannot start because its address is taken. Typical reports: + +- The console says the IP is in use and cannot be released, or the event is `FailedCreatePodSandBox ... plugin type="kube-ovn" failed (add): RPC failed; request ip return 500 no address allocated to pod / provider ovn`. +- A fixed IP looks stuck and is reassigned only after about 40 seconds. +- After a node dropped off the network, the replacement Pod cannot take the same fixed address. +- Two applications in one namespace list the same IP. +- A new Pod fails with `AddressOutOfRange` and its annotations show the same address twice. +- A pipeline release fails because the new Pod wants an address the old Pod still holds, or a third Pod appears that never starts after a namespace quota was raised. +- On a Calico cluster, a second replica reports `failed to allocate ... from IPAM: resource already exists` while the IP object's `containerID`, `nodeName` and `podName` are empty. + +Related articles: "Kube-OVN fixed IP NoAvailableAddress: spaces in the list, ip_pool not reserved, several subnets or leftover port on ACP"; "Kube-OVN fixed IP rejected or no address allocated because a stale IP object or logical switch port holds it on ACP" (when no running Pod holds the address). + +## Root Cause + +### Cause 1: another running application holds the address + +`kubectl get ip` and `kubectl get pods -A -o wide | grep ` show a live owner: a Pod of another application, possibly in another namespace and running for days, already uses the address pinned with `ovn.kubernetes.io/ip_pool` or `ovn.kubernetes.io/ip_address`. The address is not leaked; it is in use. + +A fixed IP can also only look stuck. The `kube-ovn-controller` log shows the new Pod failing for a while with `static address for / has been assigned to /` and `acquire address from ippool ... failed`, then `release v4 from subnet for /, add ip to released list`, and later `allocate v4 ... for /`. The new Pod gets the same IP and MAC about 40 seconds after the first failure. + +### Cause 2: the node is down and the terminating Pod still holds the fixed IP + +When the node running a fixed-IP Pod loses contact, the Pod stays `Terminating` and is not deleted, so its fixed-IP lease is not released. The replacement on another node fails network setup with an address conflict. + +### Cause 3: two applications in the same namespace are configured with the same IP + +Both workloads pin the same address, so whichever starts second fails. + +### Cause 4: Calico IP object with empty containerID, nodeName and podName + +On a Calico cluster, a second replica fails at the CNI with `failed to allocate ... from IPAM: resource already exists`. `kubectl get ip
` shows empty `containerID`, `nodeName` and `podName` while `workload` still names the application, and a live Pod already uses the address. The `raven` logs in `kube-system` show nothing. The platform IP status and Calico IPAM are out of step. + +### Cause 5: AddressOutOfRange after a duplicate IP assignment + +A new Pod fails with `AddressOutOfRange`, and its annotations show the same address assigned twice. Removing the duplicate assignment lets the Pod be created. + +### Cause 6: RollingUpdate creates the new Pod while the fixed IP is still taken + +With `strategy.type: RollingUpdate`, the new Pod is created before the old one is deleted, so the fixed address is still in use; a single-instance service with one fixed IP fails the same way on every update. A stalled rollout causes the same shortage later: a Deployment with `replicas: 2`, the default `maxSurge: 25%` and two addresses in `ovn.kubernetes.io/ip_pool` was restarted while the namespace quota was full, so the surge Pod could not be created and the rollout never finished. Deleting Pods by hand does not cancel the rollout. When the quota was raised, the rollout resumed and created a third Pod (`deployment.kubernetes.io/max-replicas: "3"` on the ReplicaSet) that could never start because both pinned addresses were in use. + +## Resolution + +### Cause 1: another running application holds the address + +Stop the application that holds the address, or change this workload's fixed IP to a free address. When the log shows the release-and-reallocate sequence above, wait for the reallocation before treating the address as leaked. + +> **Warning:** Changing `ovn.kubernetes.io/ip_pool` in the Pod template rolls the Deployment onto a different address; anything that reaches it by the old address breaks. Pick an address inside the namespace's subnet, outside `excludeIps`, with no IP object. Save the Deployment; restoring it is the rollback. + +```bash +kubectl get pods -A -o wide | grep +kubectl get ip -o wide | grep +kubectl get deploy -n -o yaml > /root/deploy-.before.yaml +kubectl -n patch deploy --type=merge -p '{"spec":{"template":{"metadata":{"annotations":{"ovn.kubernetes.io/ip_pool":""}}}}}' +kubectl -n kube-system logs deploy/kube-ovn-controller --tail=2000 | grep | grep -E 'has been assigned to|acquire address|release v4|allocate v4' +``` + +### Cause 2: the node is down and the terminating Pod still holds the fixed IP + +Force-delete the stuck Pod so the lease is released. For the longer term, pin more addresses than replicas and spread the replicas with Pod anti-affinity, so one node loss cannot use up the pins. + +> **Warning:** A force delete skips the grace period and `preStop`. Only when the node is confirmed down and unreachable; otherwise the old process may still use the address and you create a duplicate. The delete cannot be undone. + +```bash +kubectl get node +kubectl get pod -n -o wide +kubectl delete pod -n --force --grace-period=0 +``` + +### Cause 3: two applications in the same namespace are configured with the same IP + +Change one of the two settings so they no longer overlap, then restart both applications. + +```bash +kubectl get deploy,sts -n -o yaml | grep -n -E 'ip_address|ip_pool' +``` + +### Cause 4: Calico IP object with empty containerID, nodeName and podName + +Restart the `raven` Pods in `kube-system`, then restart the application. + +> **Warning:** Restarting `raven` pauses IP status updates until the new Pod is Ready, and restarting the application drops its connections. Only when the IP object has the empty fields, a live Pod uses the address, and the CNI says `resource already exists`. + +```bash +kubectl get ip
-o yaml +kubectl -n kube-system get pod -o wide | grep raven +kubectl -n kube-system delete pod +kubectl -n rollout restart deploy/ +``` + +### Cause 5: AddressOutOfRange after a duplicate IP assignment + +Inspect the annotations, keep one assignment of the duplicated address, and recreate the Pod. + +```bash +kubectl get pod -n -o jsonpath='{.metadata.annotations}{"\n"}' +``` + +### Cause 6: RollingUpdate creates the new Pod while the fixed IP is still taken + +Switch the workload to `Recreate`. If the service does not need a fixed IP, removing the fixed-IP setting also works. + +> **Warning:** `Recreate` stops the current replicas before starting new ones, so requests fail during the gap. Setting `RollingUpdate` back is the rollback and brings the failure back while the pin list is only as long as `replicas`. + +```bash +kubectl patch deploy -n --type=merge -p '{"spec":{"strategy":{"type":"Recreate","rollingUpdate":null}}}' +``` + +For a stalled rollout that left an extra Pod, scale the Deployment to `0` and back to clear the rollout state. To prevent it, pin at least `replicas + maxSurge` addresses (three for two replicas at `25%`), or set `maxSurge: 0` and accept one fewer replica during updates. + +> **Warning:** Scaling to `0` stops every replica until the new Pods are Ready. Run it in a window. + +```bash +kubectl -n get rs -o yaml | grep -E 'desired-replicas|max-replicas' +kubectl -n scale deploy --replicas=0 +kubectl -n scale deploy --replicas= +kubectl -n patch deploy --type=merge -p '{"spec":{"strategy":{"rollingUpdate":{"maxSurge":0,"maxUnavailable":1}}}}' +``` + +## Diagnostic Steps + +```bash +kubectl get ip -o wide +kubectl get pod -A -o wide | grep
+kubectl describe pod -n +kubectl get events -n --field-selector involvedObject.name= +``` + +- A running Pod of another workload on the address, or `no address allocated to pod` with the pinned address owned elsewhere: Cause 1. A release-then-reallocate sequence in the controller log: Cause 1, wait. +- The previous owner is `Terminating` on a `NotReady` node: Cause 2. +- Two workloads in the namespace with the same address in their annotations: Cause 3. +- A Calico IP object with empty `containerID`, `nodeName` and `podName` plus `resource already exists`: Cause 4. +- `AddressOutOfRange` with a duplicated address in the annotations: Cause 5. +- A failure during an update with `RollingUpdate`, or an extra Pod after a quota change: Cause 6. diff --git a/docs/en/solutions/Kube-OVN_fixed_IP_rejected_or_no_address_allocated_because_a_stale_IP_object_or_logical_switch_port_holds_it_on_ACP.md b/docs/en/solutions/Kube-OVN_fixed_IP_rejected_or_no_address_allocated_because_a_stale_IP_object_or_logical_switch_port_holds_it_on_ACP.md new file mode 100644 index 000000000..69f75a70b --- /dev/null +++ b/docs/en/solutions/Kube-OVN_fixed_IP_rejected_or_no_address_allocated_because_a_stale_IP_object_or_logical_switch_port_holds_it_on_ACP.md @@ -0,0 +1,147 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.16.x + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# Kube-OVN fixed IP rejected or no address allocated because a stale IP object or logical switch port holds it on ACP + +## Issue + +One of the following happens on a Kube-OVN cluster: + +- A business Pod with a fixed IP stays in `ContainerCreating`, and its sandbox event says `request ip return 500 no address allocated to pod`. The used-IP page lists the address as used by two containers, one of which no longer exists. +- When a container is created with a fixed IP, the console says the fixed IP cannot be used, and the address turns out to belong to a CronJob Pod that finished long ago. +- In a namespace bound to two underlay subnets, Pods take addresses from the older subnet until it is nearly full and then stay stuck with `NoAvailableAddress`, while the newer subnet has more than a thousand free addresses: + + ```text + FailedCreatePodSandBox: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "": plugin type="kube-ovn" failed (add): RPC failed; request ip return 500 no address allocated to pod / provider ovn, please see kube-ovn-controller logs to find errors + ``` + +In each case the address is held by Kube-OVN bookkeeping, not by a running Pod: an `ips.kubeovn.io` object (IP object) or an OVN logical switch port left behind by a deleted Pod, or a u2o interconnection IP object that was never created for an old subnet. If `kubectl get pod -A -o wide | grep -w
` returns a running Pod, the address is really in use and this article does not apply. + +## Root Cause + +### Cause 1: a deleted Pod's IP object and logical switch port remain on the fixed address + +A defect in the automatic garbage collection (GC) of some newer Kube-OVN versions leaves the IP object and the logical switch port of a deleted Pod on its fixed address. Environments upgraded from older versions are more likely to hit it; later versions improve the GC logic. + +The new Pod then fails with `request ip return 500 no address allocated to pod`, and the controller logs report the address pool as exhausted. The used-IP view first lists only one container on the address; after `kube-ovn-controller` restarts, it lists two, one of which no longer exists. The OVN southbound database is healthy. Deleting the stale IP object lets the new Pod start; the used-IP page shows two users until the stale logical switch port is also deleted and `kube-ovn-controller` is restarted. + +### Cause 2: the console rejects the fixed IP because IP objects of finished CronJob Pods remain + +`kubectl get pod -A -o wide | grep -w ` returns nothing, but `kubectl get ip | grep ` returns an IP object named after a historical CronJob Pod (`.`). One address can carry many such objects (dozens from the same CronJob) next to the valid object of the Pod that really uses it. + +On Kube-OVN `v1.12.35`, these objects are residue from before the current controller started; cleanup of recent CronJob Pods is normal. That version does not clean up existing residue: its cleanup deletes the OVN port first and then the IP object, and if the IP object delete fails it only logs the error and still releases the IPAM address without requeuing the object. The periodic GC iterates only existing ports and does not check IP object owners. That is why the objects persist and why several of them can record one address. The residue affects only the console, which reads IP objects rather than subnet status. + +### Cause 3: NoAvailableAddress in a namespace bound to two underlay subnets because the older subnet has no u2o IP object + +A namespace is bound to an older underlay subnet created before an upgrade (for example a `/23` with 508 used and 1 available) and a newer one (for example a `/22` with 1 used and 1020 available). Pods take addresses from the older subnet first, and once it runs out the remaining Pods stay stuck instead of using the newer one; their events repeat `AcquireAddressFailed: NoAvailableAddress`. + +`kubectl get ip | grep u2o` shows a u2o interconnection IP object for the newer subnet and none for the older one. The older subnet was carried over from an earlier version, and its u2o IP object was not created automatically during the upgrade. Its u2o address usage records are incomplete, and when the subnet is close to exhaustion the allocation state becomes inconsistent. + +This is different from a subnet whose u2o address is the default `.2`, which has no IP object by design. Here a sibling subnet of the same cluster has one, the subnet without it predates an upgrade, and allocation fails as it fills. + +## Resolution + +### Cause 1: a deleted Pod's IP object and logical switch port remain on the fixed address + +1. Prove that the Pod named by the IP object and the port is gone: + + ```bash + kubectl get ip -o wide | grep + kubectl get pod -A -o wide | grep -w + kubectl ko nbctl list logical_switch_port | grep -B3 -A8 + kubectl ko sb status + ``` + +2. Delete the stale IP object and logical switch port, then restart `kube-ovn-controller`. The logical switch port of a Pod is named `.`. + + > **Warning:** Deleting an IP object frees the address for the next allocation, and `lsp-del` removes the port from the OVN northbound database at once. If the Pod named in them is in fact running, it loses its network and the address can be handed out twice. Only when the Pod named in the IP object and the port does not exist. Save both objects first. There is no undo other than recreating the Pod. Restarting `kube-ovn-controller` pauses allocation cluster-wide until the new leader is ready. + + ```bash + kubectl get ip -o yaml > /root/ip-.yaml + kubectl delete ip + kubectl ko nbctl list logical_switch_port . > /root/lsp-.txt + kubectl ko nbctl lsp-del . + kubectl -n kube-system delete pod -l app=kube-ovn-controller + ``` + + After the delete the new Pod starts, and after the controller restart the used-IP page shows one user again. A Kube-OVN release with the improved GC prevents a recurrence. + +### Cause 2: the console rejects the fixed IP because IP objects of finished CronJob Pods remain + +1. List every IP object whose Pod does not exist. Node IP objects, u2o interconnection objects and VIP objects have no Pod by design; they are printed for review and must be left alone. + + ```bash + kubectl get ip -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.namespace}{"\t"}{.spec.podName}{"\t"}{.spec.ipAddress}{"\n"}{end}' > /root/ip-objects.txt + while IFS=$'\t' read -r name ns pod addr; do + [ -z "$ns" ] && { echo "review (no namespace): $name $addr"; continue; } + kubectl -n "$ns" get pod "$pod" >/dev/null 2>&1 || echo "orphan: $name $ns/$pod $addr" + done < /root/ip-objects.txt + kubectl get pod -A -o wide | grep -w + ``` + +2. Delete the orphan objects after review. + + > **Warning:** Deleting an IP object whose Pod is still running, or is being created at that moment, lets the address be allocated twice. Only delete objects printed as `orphan` whose Pod is confirmed absent, and keep `/root/ip-objects.txt` plus a YAML copy of each object. The delete is irreversible; recreating the Pod is the only way to get a valid object back. + + ```bash + kubectl get ip -o yaml > /root/ip-.yaml + kubectl delete ip + ``` + +3. Retry the fixed IP in the console. A CronJob whose Pods are cleaned up normally on the current version needs no change. + +### Cause 3: NoAvailableAddress in a namespace bound to two underlay subnets because the older subnet has no u2o IP object + +1. Confirm the pattern: both subnets list the namespace, the older one is nearly full, and only the newer one has a u2o IP object. + + ```bash + kubectl get subnet -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidrBlock,VLAN:.spec.vlan,NS:.spec.namespaces,U2O:.spec.u2oInterconnection,U2OIP:.status.u2oInterconnectionIP,USED:.status.v4usingIPs,AVAIL:.status.v4availableIPs' + kubectl get ip | grep u2o-interconnection + kubectl -n kube-system logs deploy/kube-ovn-controller --tail=3000 > /root/kube-ovn-controller.log + ``` + +2. Export the existing object as a template, and read the values for the older subnet: its u2o address from the Subnet status, its UID for the owner reference, and the MAC of its u2o router port from the northbound database. + + ```bash + kubectl get ip u2o-interconnection.ovn-cluster. -o yaml > /root/u2o-ip-template.yaml + kubectl get subnet -o jsonpath='{.status.u2oInterconnectionIP} {.metadata.uid}{"\n"}' + kubectl ko nbctl show ovn-cluster | grep -B2 -A2 '' + ``` + +3. In a copy of the template, change every field that names the newer subnet, its address, its MAC or its UID to the older subnet's values, and remove `metadata.uid`, `metadata.resourceVersion`, `metadata.creationTimestamp` and `status`. Use the template's name pattern (`u2o-interconnection.ovn-cluster.`). Then create the object and check its ownership and the subnet counters. + + > **Warning:** An IP object with a wrong address or MAC records a false allocation in the subnet, and other Pods can then collide with the real u2o address or be refused a free one. Only when the sibling subnet's object exists and this subnet's is missing. Keep the template and the new file. To roll back, delete the object you created. + + ```bash + cp /root/u2o-ip-template.yaml /root/u2o-ip-.yaml + vi /root/u2o-ip-.yaml + kubectl apply -f /root/u2o-ip-.yaml + kubectl get ip u2o-interconnection.ovn-cluster. -o yaml | grep -A6 ownerReferences + kubectl get subnet -o jsonpath='{.status.u2oInterconnectionIP} {.status.v4usingIPs} {.status.v4availableIPs}{"\n"}' + ``` + +4. Recreate one stuck Pod and watch its events. If Pods still do not fall back to the newer subnet when the older one is full, pin the workload to the newer subnet with the Pod annotation `ovn.kubernetes.io/logical_switch: `. + +## Diagnostic Steps + +```bash +kubectl describe pod -n | grep -A3 -E 'FailedCreatePodSandBox|AcquireAddressFailed' +kubectl get ip -o wide | grep
+kubectl get pod -A -o wide | grep -w
+kubectl ko nbctl list logical_switch_port | grep -B3 -A8
+kubectl get ip | grep u2o-interconnection +``` + +- An IP object or logical switch port for the address whose Pod does not exist, with `no address allocated to pod` on the new Pod, is Cause 1. +- A console rejection of a fixed IP whose IP object is named after a finished CronJob Pod, often with several objects for one address, is Cause 2. +- `NoAvailableAddress` in a namespace bound to two underlay subnets, with a u2o IP object for only one of them, is Cause 3. +- If `kubectl get pod -A -o wide | grep -w
` returns a running Pod, the address is really in use by that Pod. diff --git a/docs/en/solutions/Kube-OVN_subnet_planning_CIDR_changes_overlaps_subnet_type_per-namespace_IPPool_and_an_underlay_global_cluster_on_ACP.md b/docs/en/solutions/Kube-OVN_subnet_planning_CIDR_changes_overlaps_subnet_type_per-namespace_IPPool_and_an_underlay_global_cluster_on_ACP.md new file mode 100644 index 000000000..d2b333a06 --- /dev/null +++ b/docs/en/solutions/Kube-OVN_subnet_planning_CIDR_changes_overlaps_subnet_type_per-namespace_IPPool_and_an_underlay_global_cluster_on_ACP.md @@ -0,0 +1,169 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.2" + - "4.3" +--- + +# Kube-OVN subnet planning: CIDR changes, overlaps, subnet type, per-namespace IPPool and an underlay global cluster on ACP + +## Issue + +Planning questions and errors around Kube-OVN subnets: + +- The Service address pool is used up and no Service can be created, or someone wants to change the Pod, Service or join CIDR of a running cluster, or move a cluster from Calico to Kube-OVN, and asks whether a redeploy is needed. The join CIDR given at global cluster deployment was wrong. +- Creating a new subnet fails with a CIDR conflict. +- Overlay subnets were carved from the same block as an underlay subnet, and the question is whether that is safe. +- How to tell underlay subnets from overlay subnets in `kubectl get subnet`, and why the used-IP list of an underlay subnet is not sorted. +- A checklist inspection says the join address shown for the cluster in the console does not match the join address used in the backend. +- Someone wants to give one IP range of a subnet to project A and another range to project B. +- Someone asks whether the global cluster can be deployed with Kube-OVN underlay instead of overlay. +- Someone wants a `kubectl` command that prints the cluster's subnet CIDRs. + +Related articles: "Kube-OVN subnet stuck Terminating or Pods use the wrong subnet: finalizers, namespace binding or uppercase IPv6 on ACP". + +## Root Cause + +### Cause 1: there is no supported procedure to change the Pod, Service or join CIDR after deployment + +There is no ready-made, supported procedure to change the Pod subnet, the Service subnet or the join subnet of a deployed cluster (confirmed for ACP 4.3.1 global and business clusters). A change on a running cluster needs a case-by-case evaluation by the vendor's R&D, or the cluster is redeployed with the new ranges. A wrong global join CIDR on ACP 4.2.0 was handled by redeploying. There is likewise no verified method to enlarge or add a Service CIDR. For a Calico-to-Kube-OVN move, "How to Migrate CNI from Calico to Kube-OVN on ACP" covers ACP 3.18 business clusters only, not the global cluster. An overlay range causes no problem as long as it does not conflict with the production network or with the cluster CIDRs. + +### Cause 2: a new subnet's CIDR overlaps the join subnet + +A new subnet must avoid the CIDRs of all existing subnets, including `join`, which is easy to forget because no workload uses it directly. A CIDR inside the `join` range is rejected with a conflict error. + +### Cause 3: overlay subnets taken from the same block as an underlay subnet + +Splitting one block into disjoint pieces (for example two `/24` blocks for the default and `join` overlay subnets and the remaining `/23` as an underlay subnet) does not by itself break the cluster, because the ranges do not overlap. The general recommendation still stands: keep overlay ranges out of physical-network address space, because an overlapping or misconfigured subnet can confuse routing, for example by sending traffic meant for Pods to physical nodes. + +### Cause 4: an underlay subnet is the one with a VLAN + +A subnet that references a `vlan` is an underlay subnet; a subnet without one is an overlay subnet. The console lists the used addresses of an underlay subnet in no particular order; that is the current design. + +### Cause 5: the checklist join CIDR mismatch comes from the cluster's kube-ovn.cpaas.io/join-cidr annotation + +The console reads the join CIDR from the annotation `kube-ovn.cpaas.io/join-cidr` on the `Cluster` resource (`kubectl get cls `, `platform.tkestack.io/v1`, next to `cpaas.io/network-type: kube-ovn` and `kube-ovn.cpaas.io/transmit-type: overlay`). When that annotation does not hold the join CIDR the cluster really uses, the checklist reports a mismatch. + +### Cause 6: a subnet cannot be divided between projects; an IPPool binds ranges to namespaces + +The platform does not support dividing one subnet between projects. The Kube-OVN `IPPool` (`kubeovn.io/v1`) binds ranges to namespaces instead: + +- `spec.subnet` names the owning subnet; `spec.ips` accepts ``, `` and `..`, IPv4 and IPv6. +- `spec.namespaces` optionally binds namespaces; Pods in a bound namespace get addresses only from the bound pool. Several pools can be bound to one namespace. +- Pools in one subnet must not overlap. Random allocation from the subnet excludes all pool ranges. `spec.ips` can be updated at any time and takes effect immediately. + +### Cause 7: the global cluster cannot be deployed with Kube-OVN underlay + +The global cluster deployment page of the installer (port `8080`) offers only the Kube-OVN overlay configuration, and no parameter switches the global cluster to underlay at deployment time. + +### Cause 8: subnet CIDRs can be read with kubectl + +On Calico the CIDRs are in the Calico `IPPool` objects (`crd.projectcalico.org/v1`, `spec.cidr`). On Kube-OVN, `kubectl get subnet` prints a CIDR column for every subnet together with provider, VPC, protocol, NAT, default, gateway type, used and available counts, and excluded IPs. + +## Resolution + +### Cause 1: there is no supported procedure to change the Pod, Service or join CIDR after deployment + +Plan the Pod, Service and join ranges before deployment. For a running cluster, request an evaluation for that cluster or redeploy with the new ranges. For an exhausted Service CIDR, free addresses by deleting Services that are no longer used. + +> **Warning:** Deleting a Service removes its ClusterIP and DNS name immediately. Confirm with the owner that it is unused (no endpoints is a hint, not proof) and export it first; re-creating it from the export restores the object, possibly with a different ClusterIP. + +```bash +kubectl get svc -A --no-headers | wc -l +kubectl get endpoints -A | awk '$3==""' +kubectl -n get svc -o yaml > /root/svc-.yaml +kubectl -n delete svc +``` + +### Cause 2: a new subnet's CIDR overlaps the join subnet + +List all subnet CIDRs, including `join`, and choose a range that overlaps none of them: + +```bash +kubectl get subnet -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidrBlock,VLAN:.spec.vlan' +``` + +### Cause 3: overlay subnets taken from the same block as an underlay subnet + +Check that no two subnets overlap and that no overlay range is used on the physical network; if so, no change is needed. For new plans, use ranges that are not routed on the physical network for overlay subnets. + +### Cause 4: an underlay subnet is the one with a VLAN + +```bash +kubectl get subnet -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidrBlock,VLAN:.spec.vlan,PROVIDER:.spec.provider' +kubectl get vlan +``` + +An empty VLAN column means overlay. To read the used addresses of a subnet in order: + +```bash +kubectl get ip -o custom-columns='IP:.spec.ipAddress,SUBNET:.spec.subnet,POD:.spec.podName' | grep | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n +``` + +### Cause 5: the checklist join CIDR mismatch comes from the cluster's kube-ovn.cpaas.io/join-cidr annotation + +Read the real join CIDR on the business cluster, compare it with the annotation on the global cluster, and set the annotation to the real value. Then rerun the checklist. + +> **Warning:** The annotation is platform metadata; a wrong value misleads the console and the checklist. Record the current value; annotating it back is the rollback. + +```bash +# on the business cluster +kubectl get subnet join -o jsonpath='{.spec.cidrBlock}{"\n"}' +# where the cluster resource lives +kubectl get cls -o jsonpath='{.metadata.annotations.kube-ovn\.cpaas\.io/join-cidr}{"\n"}' +kubectl annotate cls kube-ovn.cpaas.io/join-cidr= --overwrite +``` + +### Cause 6: a subnet cannot be divided between projects; an IPPool binds ranges to namespaces + +Use one `IPPool` per namespace when a range must be reserved: + +```yaml +apiVersion: kubeovn.io/v1 +kind: IPPool +metadata: + name: +spec: + subnet: + ips: + - 198.51.100.50..198.51.100.150 + namespaces: + - +``` + +Use your own addresses; pools in one subnet must not overlap. + +### Cause 7: the global cluster cannot be deployed with Kube-OVN underlay + +Deploy the global cluster with the overlay configuration the page offers. + +### Cause 8: subnet CIDRs can be read with kubectl + +```bash +# Calico cluster +kubectl get ippools.crd.projectcalico.org -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidr,BLOCK:.spec.blockSize,IPIP:.spec.ipipMode,VXLAN:.spec.vxlanMode' +# Kube-OVN cluster +kubectl get subnet +``` + +The fully qualified name avoids picking up Kube-OVN's own `ippools.kubeovn.io` on a cluster that has both CRDs. + +## Diagnostic Steps + +Start from the subnet list; it answers the type question, shows overlaps and shows bound namespaces: + +```bash +kubectl get subnet -o wide +kubectl get subnet -o yaml +kubectl get ippool +``` + +- A question about changing the Pod, Service or join range, or a full Service CIDR: Cause 1. +- A creation error naming a CIDR conflict: Cause 2. Disjoint ranges cut from one block: Cause 3. +- The VLAN column decides underlay or overlay: Cause 4. +- A checklist join-CIDR mismatch: Cause 5. +- A request to divide a subnet between projects: Cause 6. An underlay global cluster: Cause 7. A command for the CIDRs: Cause 8. diff --git a/docs/en/solutions/Kube-OVN_subnet_stuck_Terminating_or_Pods_use_the_wrong_subnet_finalizers_namespace_binding_or_uppercase_IPv6_on_ACP.md b/docs/en/solutions/Kube-OVN_subnet_stuck_Terminating_or_Pods_use_the_wrong_subnet_finalizers_namespace_binding_or_uppercase_IPv6_on_ACP.md new file mode 100644 index 000000000..1bc82e5e7 --- /dev/null +++ b/docs/en/solutions/Kube-OVN_subnet_stuck_Terminating_or_Pods_use_the_wrong_subnet_finalizers_namespace_binding_or_uppercase_IPv6_on_ACP.md @@ -0,0 +1,124 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.2" + - "4.3" +--- + +# Kube-OVN subnet stuck Terminating or Pods use the wrong subnet: finalizers, namespace binding or uppercase IPv6 on ACP + +## Issue + +- A subnet stays in deleting (`Terminating`) forever, and `kubectl describe subnet` still lists `Finalizers: kubeovn.io/kube-ovn-controller`. +- After namespaces were bound to `ovn-default`, Pods in other namespaces stop getting addresses; or Pod restarts in a namespace fail with `AddressOutOfRange` because the namespace was never bound to the subnet it should use. +- The MetalLB address pool and BGP peer pages say MetalLB is not installed although it was never uninstalled, containers cannot resolve the registry domain, and the default subnet `ovn-default` is stuck in deleting. +- A namespace was moved from an overlay to an underlay subnet, and its Pods (and the fixed-IP picker) still use the overlay range. + +Related articles: "Kube-OVN subnet planning: CIDR changes, overlaps, subnet type, per-namespace IPPool and an underlay global cluster on ACP". + +## Root Cause + +### Cause 1: the subnet is stuck in Terminating because its finalizer was never removed + +Kubernetes removes a deleted object only after every entry in `metadata.finalizers` is gone. `kube-ovn-controller` did not complete its cleanup, so `kubeovn.io/kube-ovn-controller` stayed and the subnet never disappeared. `kubectl describe subnet` shows `Deletion Timestamp` set, `Finalizers: kubeovn.io/kube-ovn-controller`, and conditions `Validated` and `Ready` still `True`. + +### Cause 2: namespace binding decides which subnet a Pod uses + +Once namespaces are listed on `ovn-default`, its addresses (other than reserved IPs) are allocated only to Pods of those namespaces, so Pods elsewhere cannot get an address from it. Keep `ovn-default` unbound and give business workloads their own subnets; assigning a business subnet to the project when it is created is the way to keep business workloads off `ovn-default`. + +In the other direction, a namespace that was never bound to its intended subnet keeps getting addresses from the default subnet: `kubectl describe ns` shows `ovn.kubernetes.io/logical_switch: ovn-default` and the `ovn-default` CIDR in `ovn.kubernetes.io/cidr`, and restarts of Pods with addresses from the intended subnet fail with `AddressOutOfRange`. + +### Cause 3: ovn-default was deleted and is stuck in deleting + +Someone tried to delete `ovn-default` (and another subnet), and the deletion never completed. A half-deleted default subnet breaks the cluster: every `ars` is abnormal, containers cannot resolve the private registry domain, and the MetalLB pages report that MetalLB is not installed. Without an etcd backup, rolling back is not possible. Stopping `kube-ovn-controller`, removing the finalizer so the deletion completes, and starting the controller again brings `ovn-default` back as the default subnet with its original CIDR; other deleted subnets do not come back. Leftover interfaces from a previously tested CNI (for example `cilium_host@cilium_net`) on a node are a separate cleanup. + +### Cause 4: v6 ip address can not contain upper case, so the subnet fails validation + +A namespace was assigned to an underlay dual-stack subnet, but restarted Pods still received overlay addresses and the Pod events were clean. The `kube-ovn-controller` log shows: + +```text +subnet U2O interconnection ip , v6 ip address can not contain upper case +failed to validate subnet +``` + +and the Subnet has a `Warning` event `ValidateLogicalSwitchFailed`. The subnet failed validation and was not usable, so the namespace kept using the overlay subnet. + +## Resolution + +### Cause 1: the subnet is stuck in Terminating because its finalizer was never removed + +Remove the finalizers of a subnet that is really meant to be deleted, after its Pods and IP objects are gone. + +> **Warning:** Removing finalizers skips the controller's cleanup and is irreversible: the object is deleted as soon as the list is empty. Keep the YAML so the subnet can be recreated if needed. + +```bash +kubectl get subnet -o yaml > /root/subnet-.before.yaml +kubectl get subnet -o jsonpath='{.metadata.deletionTimestamp}{"\n"}{.metadata.finalizers}{"\n"}' +kubectl get ip | grep || true +kubectl patch subnet --type=merge -p '{"metadata":{"finalizers":null}}' +``` + +### Cause 2: namespace binding decides which subnet a Pod uses + +Leave `ovn-default` unbound. Bind each business namespace to its own subnet; add a namespace that is missing from its intended subnet. + +> **Warning:** Changing `spec.namespaces` changes where new Pods of those namespaces get addresses; running Pods keep theirs until recreated. Save the subnet first; restoring `spec.namespaces` is the rollback. + +```bash +kubectl get subnet -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidrBlock,NAMESPACES:.spec.namespaces' +kubectl describe ns | grep ovn.kubernetes.io +kubectl get subnet -o yaml > /root/subnet-.before.yaml +kubectl patch subnet --type=json -p='[{"op":"add","path":"/spec/namespaces/-","value":""}]' +``` + +If `spec.namespaces` does not exist yet, use a merge patch that sets the whole list. + +### Cause 3: ovn-default was deleted and is stuck in deleting + +Only when `ovn-default` itself has a `deletionTimestamp` and there is no etcd backup to roll back to: let the deletion complete while `kube-ovn-controller` is stopped, then start the controller so it recreates the default subnet. + +> **Warning:** While `kube-ovn-controller` is at `0` replicas, no Pod in the cluster gets an address. Removing the finalizers is irreversible. Work in a maintenance window from a session that does not depend on Pod networking, and save the Subnet YAML and the replica count. If the subnet does not come back, recreate it from the saved YAML after removing `status`, `metadata.uid`, `metadata.resourceVersion`, `metadata.deletionTimestamp` and `metadata.finalizers`. + +```bash +kubectl get subnet ovn-default -o yaml > /root/subnet-ovn-default.before.yaml +kubectl -n kube-system get deploy kube-ovn-controller -o jsonpath='{.spec.replicas}{"\n"}' +kubectl -n kube-system scale deploy kube-ovn-controller --replicas=0 +kubectl patch subnet ovn-default --type=merge -p '{"metadata":{"finalizers":null}}' +kubectl get subnet ovn-default -w +kubectl -n kube-system scale deploy kube-ovn-controller --replicas= +kubectl get subnet ovn-default -o jsonpath='{.spec.cidrBlock} {.spec.default}{"\n"}' +``` + +Stop the watch when the subnet is gone. Treat other subnets stuck in deleting the same way only if they are meant to be deleted. Check nodes for leftovers of other CNIs: `ip -br link | grep -i cilium`. + +### Cause 4: v6 ip address can not contain upper case, so the subnet fails validation + +Rewrite the IPv6 address the controller rejects in lowercase on the Subnet, then recreate the Pods that received overlay addresses. + +> **Warning:** Editing the Subnet affects allocation for every bound namespace. Change only the letter case of the address named in the log. Save the Subnet first; restoring it is the rollback. + +```bash +kubectl -n kube-system logs deploy/kube-ovn-controller --tail=2000 | grep -E 'can not contain upper case|failed to validate subnet' +kubectl get subnet -o yaml > /root/subnet-.before-lowercase.yaml +kubectl edit subnet +kubectl -n delete pod +``` + +In a mixed overlay and underlay deployment where an overlay subnet's IPv6 address also contains uppercase letters, leave that one unchanged as long as Kube-OVN reports no errors and it works, because changing it involves Service CIDR changes. + +## Diagnostic Steps + +```bash +kubectl get subnet +kubectl get subnet -o jsonpath='{.metadata.deletionTimestamp}{"\n"}{.metadata.finalizers}{"\n"}' +kubectl get ns -o yaml | grep -i ovn +kubectl get events -A --field-selector reason=ValidateLogicalSwitchFailed +``` + +- A subnet with a `deletionTimestamp` and `kubeovn.io/kube-ovn-controller` in `finalizers`: Cause 1 (Cause 3 if it is `ovn-default`, together with broken MetalLB pages and registry resolution). +- `AddressOutOfRange` on restart with the namespace annotation still `ovn.kubernetes.io/logical_switch: ovn-default` and the namespace absent from the intended subnet, or Pods elsewhere failing after namespaces were bound to `ovn-default`: Cause 2. +- `can not contain upper case` or `ValidateLogicalSwitchFailed`: Cause 4. diff --git a/docs/en/solutions/Kube-OVN_underlay_Pods_fail_with_resolve_MAC_address_timeout_or_bridge_network_InitOVSBridgeFailed_on_ACP.md b/docs/en/solutions/Kube-OVN_underlay_Pods_fail_with_resolve_MAC_address_timeout_or_bridge_network_InitOVSBridgeFailed_on_ACP.md new file mode 100644 index 000000000..1380b6adb --- /dev/null +++ b/docs/en/solutions/Kube-OVN_underlay_Pods_fail_with_resolve_MAC_address_timeout_or_bridge_network_InitOVSBridgeFailed_on_ACP.md @@ -0,0 +1,192 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# Kube-OVN underlay Pods fail with resolve MAC address timeout or bridge network InitOVSBridgeFailed on ACP + +Related articles: [Node, API and VIP access lost after creating a Kube-OVN ProviderNetwork or underlay subnet on ACP](Node_API_and_VIP_access_lost_after_creating_a_Kube-OVN_ProviderNetwork_or_underlay_subnet_on_ACP.md); [All charts stay ChartDownloading on a new Kube-OVN cluster because ovn-default was created as overlay on ACP](All_charts_stay_ChartDownloading_on_a_new_Kube-OVN_cluster_because_ovn-default_was_created_as_overlay_on_ACP.md); [Adding a node to a Kube-OVN underlay cluster in the web UI fails with network interface not found on ACP](Adding_a_node_to_a_Kube-OVN_underlay_cluster_in_the_web_UI_fails_with_network_interface_not_found_on_ACP.md). + +## Issue + +Pods on a Kube-OVN underlay subnet cannot be created, on all nodes or only on some, or the bridge network itself reports a failure. Typical signs: + +- Sandbox creation fails with an event like this, for CoreDNS right after a console deployment, or for platform Pods such as `kube-ovn-pinger`, `zeus` and `nevermore` on some nodes, or for a fixed-IP business Pod: + + ```text + Warning FailedCreatePodSandBox ... plugin type="kube-ovn" failed (add): RPC failed; request ip return 500 configure nic eth0 for pod timeout: read packet ...: is not ready for interface eth0 after 200 checks: resolve MAC address of timeout + ``` + + On a freshly deployed cluster this can look as if the CoreDNS address were already in use. + +- The bridge network (ProviderNetwork) shows status Error with `Ready: false`, and the per-node condition reads: + + ```text + Ready False InitOVSBridgeFailed failed to create and configure external bridge br-: nic does not exist + ``` + +- The underlay network breaks right after a ProviderNetwork was renamed (for example while configuring an underlay sub-interface for MetalLB), and does not recover. +- After a power cycle a VLAN sub-interface such as `bond1.131` reappears on the node, and containers on the underlay subnet stop working. + +## Root Cause + +### Cause 1: the ProviderNetwork is bound to the wrong NIC after deployment + +A cluster deployed from the console may bind the ProviderNetwork to a NIC other than the intended underlay NIC, for example when the deployment form did not allow choosing it. `br-provider` is then created on the host-network NIC. `ip r` on the node shows the default route and the host subnet on `br-provider`, while the underlay NIC holds only a kernel link route for its own subnet. Pods on the underlay subnet then try to resolve the gateway through the wrong NIC. In the `resolve MAC address of timeout` error, `` is in the host network subnet, not the underlay gateway, while the node itself can ping the underlay gateway on the intended NIC. + +### Cause 2: the bridge NIC does not exist on some nodes + +A ProviderNetwork names one `defaultInterface` for all nodes, plus optional per-node `customInterfaces` and `excludeNodes`. When a node does not have the configured NIC: + +- the bridge cannot be created there and the node condition shows `InitOVSBridgeFailed ... nic does not exist`, while nodes that have the NIC show `InitOVSBridgeSucceeded`; +- if the ProviderNetwork does not list the NIC that some nodes actually use for the underlay (for example `p40p2`), ARP for the gateway cannot complete on those nodes and sandbox creation fails with `resolve MAC address of timeout`; +- if a NIC disappears from the host (for example after a hardware or IaaS change), the ProviderNetwork shows every node under `Not Ready Nodes` and `ip a` on the nodes lists the other NICs but not the configured one. + +### Cause 3: renaming a ProviderNetwork leaves a stale OVN localnet port + +Kube-OVN does not handle a ProviderNetwork rename. The OVN logical switch port of type `localnet` for the underlay subnet (named `localnet.`) is left behind still pointing at the old provider, and the underlay network stays disconnected. Changing only the ProviderNetwork's NIC while keeping the ProviderNetwork and VLAN names does not trigger this. + +When one underlay NIC carries several VLAN subnets through sub-interfaces, give each VLAN its own ProviderNetwork. + +### Cause 4: the node OS recreates a VLAN sub-interface that conflicts with the underlay + +The underlay NIC (for example `bond1`) is attached to `ovs-system` by the ProviderNetwork. If the node OS also creates a VLAN sub-interface on that NIC (for example `bond1.131`), the sub-interface conflicts with the underlay subnet of the same VLAN, and underlay Pods time out resolving the gateway MAC, although a test VLAN interface on the node reaches the gateway. `ip a | grep bond1` shows the physical NICs enslaved to `bond1`, `bond1` with `master ovs-system`, and a leftover `bond1.131@bond1` in state `DOWN`. + +The sub-interface can come back without any script creating it. On CentOS 7, the network service (`/etc/init.d/network`) recreates it every time `network` restarts or the node boots, even with kubelet and containerd stopped. A likely explanation is that the service implicitly creates the VLAN sub-interface when the `8021q` module is loaded, `bond1` exists and VLAN 131 context is present on the system (for example an address of that VLAN's segment on `bond0`). + +## Resolution + +### Cause 1: the ProviderNetwork is bound to the wrong NIC after deployment + +Point the ProviderNetwork at the NIC that reaches the underlay gateway. + +> **Warning:** Changing `defaultInterface` re-creates the bridge on each node and interrupts underlay traffic. Save the object first and restore it to roll back. Do not select the NIC that carries the node IP; that takes node access with it. + +```bash +kubectl get provider-networks -o yaml > /root/pn-.before.yaml +kubectl edit provider-networks +``` + +After the correct NIC is set, IP allocation works again. If traffic between the host subnet and the underlay subnet is still blocked, a firewall between them has to be opened on the network side. + +If the underlay subnet has only a few usable addresses (for example about 20), it is too small for the platform components. Deploy a standard overlay cluster first and add the underlay subnet afterwards. + +### Cause 2: the bridge NIC does not exist on some nodes + +1. On every failing node, read the real NIC name: + + ```bash + ip -br link + kubectl get provider-networks -o yaml + ``` + +2. Make the ProviderNetwork match the hardware: list nodes whose underlay NIC has another name under `customInterfaces`, and put nodes that have no underlay NIC under `excludeNodes`. + + > **Warning:** Editing the ProviderNetwork reconfigures the bridge on the listed nodes and briefly interrupts underlay traffic there. Save the object first; to roll back, apply the saved copy. + + ```yaml + apiVersion: kubeovn.io/v1 + kind: ProviderNetwork + metadata: + name: + spec: + defaultInterface: + customInterfaces: + - interface: + nodes: + - + excludeNodes: + - + ``` + + For example, when only one node has the configured NIC and the others have no underlay NIC, list the others under `excludeNodes`; the bridge network then becomes available (`可用`, Available). + +3. If a NIC has vanished from a node, restore it on the host first; the ProviderNetwork cannot use an interface the kernel does not show. +4. Recreate the failed Pods, keep underlay Pods off excluded nodes with a node selector, and check gateway ARP on the underlay NIC: + + ```bash + arping -c 3 -I + ``` + +### Cause 3: renaming a ProviderNetwork leaves a stale OVN localnet port + +Only when the underlay broke right after a ProviderNetwork rename: + +1. List the `localnet` ports and find the one that still points at the old ProviderNetwork. The `network_name` option of a `localnet` port is the provider it maps to; a value that names the old ProviderNetwork marks the stale port. + + ```bash + kubectl get provider-networks,vlan,subnet + kubectl ko nbctl show | grep -B2 -A3 localnet | tee /root/localnet.before + kubectl ko nbctl lsp-get-options localnet. + kubectl ko vsctl get open . external-ids:ovn-bridge-mappings + ``` + +2. Delete the stale port and restart `kube-ovn-controller` so that it creates the port again for the current provider. + + > **Warning:** `lsp-del` removes the port from the OVN northbound database at once, and the underlay subnet has no path to the physical network until `kube-ovn-controller` creates it again. Delete only the `localnet` port confirmed in step 1, and keep the saved listing. The delete cannot be undone except by the controller recreating the port. Restarting `kube-ovn-controller` pauses IP allocation until the new leader is ready. + + ```bash + kubectl ko nbctl lsp-del localnet. + kubectl -n kube-system rollout restart deploy/kube-ovn-controller + kubectl -n kube-system rollout status deploy/kube-ovn-controller + kubectl ko nbctl lsp-get-options localnet. + ``` + +3. Keep ProviderNetwork and VLAN names stable afterwards. To use a different interface, change only the NIC. + +### Cause 4: the node OS recreates a VLAN sub-interface that conflicts with the underlay + +1. When a Pod is already stuck, delete the leftover sub-interface. This needs root; without `sudo` the command returns `RTNETLINK answers: Operation not permitted`. + + > **Warning:** Only when the interface is a leftover VLAN sub-interface on the NIC that the ProviderNetwork has attached to `ovs-system`, and nothing on the host uses it. Deleting a sub-interface that carries a host address cuts that address. To roll back, recreate it with `ip link add link name . type vlan id `. + + ```bash + ip a | grep + sudo ip link del . + ``` + + The stuck Pod then starts. + +2. Remove the source of the interface from the OS network configuration if you can find it. If you cannot, add a udev rule that deletes the interface whenever it appears (replace `bond1.131` with your conflicting interface): + + > **Warning:** The rule deletes the named interface every time it appears, including when someone creates it on purpose. Name the interface exactly. To roll back, delete the rule file and reload udev. + + ```bash + cat > /etc/udev/rules.d/99-disable-bond1.131.rules <<'RULE' + SUBSYSTEM=="net", KERNEL=="bond1.131", ENV{SYSTEMD_ALIAS}="*", RUN+="/bin/sh -c '/sbin/ip link set bond1.131 down; /sbin/ip link delete bond1.131'" + RULE + udevadm control --reload-rules + udevadm trigger + ``` + +3. Confirm the rule: restart the network service, or reboot once in a maintenance window, and check that the interface does not come back. + + > **Warning:** Restarting the network service briefly drops every host interface and can cut the session you are using. Run it from a console, in a maintenance window, one node at a time. + + ```bash + systemctl restart network + ip link show . + ``` + +## Diagnostic Steps + +```bash +kubectl get provider-networks -o yaml +kubectl get vlan,subnet +ip -br addr +ip route +ip -d link show | grep -A1 -E 'vlan|br-' +kubectl get provider-networks -o jsonpath='{range .status.conditions[*]}{.node}{" "}{.type}{" "}{.status}{" "}{.reason}{" "}{.message}{"\n"}{end}' +kubectl ko nbctl show | grep -B2 -A3 localnet +``` + +- `br-provider` holding the node address and default route right after deployment, with `resolve MAC address of timeout` for a gateway in the host network subnet, is Cause 1. +- A bridge network that fails only on nodes missing the interface, a node condition `InitOVSBridgeFailed` with `nic does not exist`, or gateway ARP failing only on nodes whose underlay NIC is not listed, is Cause 2. +- An underlay failure that started right after renaming a ProviderNetwork, with a `localnet` port whose `network_name` is the old name, is Cause 3. +- An unexpected VLAN sub-interface after a reboot, or a `DOWN` sub-interface such as `bond1.131@bond1` on a NIC whose master is `ovs-system` while underlay Pods time out resolving the gateway MAC, is Cause 4. +- If the underlay gateway does not answer ARP from any NIC of the node, the fault is in the switch or gateway configuration, not in the ProviderNetwork. diff --git a/docs/en/solutions/Kube-OVN_underlay_multicast_not_delivered_DDS_IGMP_membership_limit_or_a_sender_that_never_joins_the_group_on_ACP.md b/docs/en/solutions/Kube-OVN_underlay_multicast_not_delivered_DDS_IGMP_membership_limit_or_a_sender_that_never_joins_the_group_on_ACP.md new file mode 100644 index 000000000..d8f4553cf --- /dev/null +++ b/docs/en/solutions/Kube-OVN_underlay_multicast_not_delivered_DDS_IGMP_membership_limit_or_a_sender_that_never_joins_the_group_on_ACP.md @@ -0,0 +1,113 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Kube-OVN underlay multicast not delivered: DDS, IGMP membership limit or a sender that never joins the group on ACP + +## Issue + +Pods on a Kube-OVN underlay subnet send or receive multicast, and the data does not arrive. Unicast between the same endpoints works. Typical reports: + +- DDS traffic from one underlay Pod to another is missing or arrives only after several minutes, on the same node or across nodes. The same receiver accepts DDS immediately when the sender is a physical host using that sender's address. +- An application cannot join one more multicast group, and its multicast send or receive breaks; joining works again after another group is dropped. +- A Pod sends to a multicast group. The packets can be captured on the host's physical NIC but not on the switch, and receivers that joined the group get nothing. `ping -c 3 ` from the Pod shows `100% packet loss`. + +## Root Cause + +### Cause 1: underlay DDS depends on platform multicast support that is enabled separately + +On ACP 4.2 with business Pods on a Kube-OVN underlay network, DDS can fail or arrive late between Pods while unicast is normal, across several applications, while a physical sender on the same address works. This is the platform multicast path, not IPAM and not a single application's socket. Enabling multicast for this scenario is a separate request to the platform vendor; there is no in-cluster switch for it. + +### Cause 2: net.ipv4.igmp_max_memberships is 20, so a new group cannot be joined + +`net.ipv4.igmp_max_memberships` defaults to `20` and applies per network namespace. Once the Pod has joined 20 groups, the next `setsockopt` join fails and the application's multicast traffic for that group breaks. Leaving one group allows the new join; raising the sysctl keeps the cap from being hit again. + +### Cause 3: the sending application never joins the target group + +The application sends UDP to the target group but never joins it with `setsockopt(IP_ADD_MEMBERSHIP)`. The kernel therefore sends no IGMP report for that group, and OVN records no `IGMP_Group` entry for it. With IGMP snooping, OVN forwards ordinary multicast only to ports that have joined the group, so the packets match no forwarding flow and are dropped before they reach the bridge port and the switch. A group the Pod did join (for example the group it receives from) works, because it has an `IGMP_Group` entry. + +The evidence on the Pod and the node: + +- `ip maddr` in the container lists the joined groups on `eth0` and has no entry for the target group. +- `tcpdump -i eth0 -nnve igmp` on the node shows the gateway's `igmp query v3`, and the Pod's port answers with `igmp v3 report, 1 group record(s) [gaddr is_ex, 0 source(s)]` only for the joined group, never for the target group. +- The OVN `IGMP_Group` table has an entry for the joined group and none for the target group. + +This is an application-side problem; the network configuration is working as designed. + +## Resolution + +### Cause 1: underlay DDS depends on platform multicast support that is enabled separately + +Rule out Cause 3 first. If unicast works, the application joins its groups, and DDS still fails between Pods while a physical sender on the same address succeeds, request multicast enablement for the cluster. Locate where the packets stop and attach the result: + +```bash +kubectl exec -n -- ip maddr +kubectl exec -n -- ip route +tcpdump -ni -c 20 host +``` + +On the receiver node, a packet on the Pod veth means the CNI delivered it. A packet on the physical NIC and none on the veth means the underlay path did not forward the group. A packet on the veth and no group membership in the receiver means the application did not join. + +### Cause 2: net.ipv4.igmp_max_memberships is 20, so a new group cannot be joined + +Only when a new join fails and the sysctl in the Pod prints `20`: leave a group the application owner confirms is no longer needed, or raise the limit. + +> **Warning:** A higher limit uses more kernel memory for membership state in that network namespace. A value written with `sysctl -w` in a running Pod is lost when the Pod is recreated. Writing back the previous value (normally `20`) is the rollback. Dropping a group stops reception for it immediately; rejoining reverses it. + +```bash +kubectl exec -n -- sysctl net.ipv4.igmp_max_memberships +kubectl exec -n -- sysctl -w net.ipv4.igmp_max_memberships=64 +kubectl ko sbctl list Multicast_Group +``` + +If `kubectl ko` is not available on the operator node, take the `kubectl-ko` helper from the `kube-ovn-pinger` image the cluster already runs. To keep a higher limit across restarts, set the sysctl in the Pod spec only if the kubelet already allows that key; otherwise reapply it after each recreate. + +### Cause 3: the sending application never joins the target group + +Have the application join the target group with `setsockopt(IP_ADD_MEMBERSHIP)` before sending, so the kernel sends an IGMP report and OVN creates the `IGMP_Group` entry. Then check again on the node: + +```bash +kubectl exec -n -- ip maddr +tcpdump -i -nnve igmp +kubectl ko sbctl list IGMP_Group +``` + +The target group should now appear in `ip maddr`, in an IGMP report from the Pod's port, and in `IGMP_Group`. + +## Diagnostic Steps + +1. Confirm unicast and read the group memberships of sender and receiver: + + ```bash + kubectl get pod -n -o wide + kubectl exec -n -- ping -c 3 + kubectl exec -n -- ip maddr + kubectl exec -n -- ip maddr + ``` + +2. Watch IGMP on the node of the sending Pod and read OVN's group table: + + ```bash + tcpdump -i -nnve igmp + kubectl ko sbctl list IGMP_Group + ``` + + No membership, no IGMP report and no `IGMP_Group` entry for the group the Pod sends to: Cause 3. + +3. Check the membership cap: + + ```bash + kubectl exec -n -- sysctl net.ipv4.igmp_max_memberships + ``` + + A join error at 20 memberships: Cause 2. + +4. Unicast works, groups are joined, and DDS from a Pod sender fails while a physical sender on the same address succeeds: Cause 1. + +If unicast to the underlay gateway already fails, the problem is not multicast; see "Kube-OVN underlay Pods fail with resolve MAC address timeout or bridge network InitOVSBridgeFailed on ACP". diff --git a/docs/en/solutions/LDAP_user_who_has_never_logged_in_gets_an_invalid_password_error_on_the_ACP_login_page.md b/docs/en/solutions/LDAP_user_who_has_never_logged_in_gets_an_invalid_password_error_on_the_ACP_login_page.md new file mode 100644 index 000000000..dd093adb7 --- /dev/null +++ b/docs/en/solutions/LDAP_user_who_has_never_logged_in_gets_an_invalid_password_error_on_the_ACP_login_page.md @@ -0,0 +1,56 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# LDAP user who has never logged in gets an invalid password error on the ACP login page + +## Issue + +A user tries to log in to the ACP console with a directory (LDAP / Active Directory) account for the first time. The account has no platform user yet, because directory users are only created in ACP when they log in or are synchronized. The login page reports that the password is invalid, although the password has been verified and works elsewhere. + +Other directory users log in normally, and the user is testing from a network segment that can reach the platform. Often the organization has more than one directory domain, and the failing user belongs to a domain other than the one most users are in. + +## Root Cause + +The login page does not reveal whether an account exists. When the account cannot be found and when the password is wrong, the platform returns the same invalid-password error, by design: a distinct "user not found" answer would let an attacker probe for valid account names. An invalid-password error for a user whose password is known to be correct therefore usually means that the platform **cannot find the account**. + +For an LDAP identity provider (IDP), ACP looks up a login account by combining the IDP's user **Filter** with an equality match on the configured **Login Field** attributes, below the user **Base DN**, in the directory server the IDP is connected to. A user who is not in that directory, is outside the Base DN, or is excluded by the Filter is never found, and the login fails with the invalid-password error. This happens, for example, when the user's accounts live in a separate directory domain whose accounts are not present in the directory the IDP queries. + +## Resolution + +1. Confirm which directory, Base DN and Filter the platform uses: open **Users** > **IDP** and check the LDAP IDP's server address, **Search Setting** (**Filter**, **Base DN**) and **Field Mapping** (**Login Field**). + +2. Make the user reachable through that configuration. Choose what fits the directory layout: + + - Have the directory administrator make the user's account available in the directory the IDP queries, for example by synchronizing the other domain's accounts into it, so that the user is found below the configured Base DN and matches the Filter. + - Or adjust the IDP's **Base DN** and **Filter** so that they include the user's branch, if the account is already in that directory but outside the configured search. + +3. If the IDP has **Auto Sync** enabled or you synchronize manually, run the synchronization and check that the user now appears under **Users**. Otherwise, the platform user is created at the first successful login. + +4. Ask the user to log in again with the same login attribute (for example `sAMAccountName` or `mail`) configured as **Login Field**. + +## Diagnostic Steps + +1. Read the authentication log on the `global` cluster around the failing login. Find the `apollo` pod and search its log for the username: + + ```bash + kubectl get pod -A | grep apollo + kubectl logs -n cpaas-system | grep -C 5 + ``` + +2. Query the directory with the IDP's own settings from a host whose network is equivalent to the cluster. No result means the platform cannot find the user, which explains the invalid-password error: + + ```bash + ldapsearch -H ldap://:389 -D '' -W \ + -b '' '(&(=))' dn + ``` + + One entry returned means the account is found; then the password itself, an account lock, or the account being excluded by policy is the next thing to check. + +3. Compare with a user who can log in: if that user is in a different domain or branch than the failing one, the IDP's directory or Base DN covers only the working one. + +4. Rule out network reachability by testing from a segment that is known to reach the platform. A user who cannot reach the platform at all sees a connection error, not an invalid-password error. diff --git a/docs/en/solutions/Leftover_Service_Mesh_v1_recreates_istio-system_against_Mesh_v2_or_Istio_Secret_certificate_alerts_remain_on_ACP.md b/docs/en/solutions/Leftover_Service_Mesh_v1_recreates_istio-system_against_Mesh_v2_or_Istio_Secret_certificate_alerts_remain_on_ACP.md new file mode 100644 index 000000000..deb786cb8 --- /dev/null +++ b/docs/en/solutions/Leftover_Service_Mesh_v1_recreates_istio-system_against_Mesh_v2_or_Istio_Secret_certificate_alerts_remain_on_ACP.md @@ -0,0 +1,118 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Service Mesh +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Leftover Service Mesh v1 recreates istio-system against Mesh v2 or Istio Secret certificate alerts remain on ACP + +## Issue + +After Service Mesh v1 (ASM v1) was removed, or while Mesh v2 is being installed on a cluster that used to run v1, one of these is seen: + +- Creating Istio-type resources for Mesh v2 fails. Namespace `istio-system` and ServiceAccount `istio-reader-service-account` reappear on the business cluster on their own after they are deleted. +- Certificate monitoring keeps reporting that Istio Secrets are expired or about to expire, although ASM v1 has been uninstalled. + +Mesh v1 and Mesh v2 must not be deployed on one cluster at the same time; two Istio installs cannot coexist. Mesh v2 support starts with platform 4.1.1, ASM v1 is no longer maintained, and Service Mesh v1 is removed in platform 4.4. + +Related articles: "Service Mesh delete is stuck on a failed to get cluster webhook denial or on pods that still have sidecar injection" covers a delete that does not finish. "Platform certificate expiry alert for a cert-manager serving certificate that renews by itself on ACP" covers Istio CA alerts of a mesh that is still installed. + +## Root Cause + +### Cause 1: Mesh v1 `servicemesh` and `servicemeshgroup` are still on `global` + +The `global` cluster still holds Mesh v1 `servicemesh` and `servicemeshgroup` objects. `global-asm-controller` sees them and keeps creating `istio-system` and related objects on the business cluster. Those objects collide with the Mesh v2 install. + +### Cause 2: Istio Secrets left behind by the ASM v1 uninstall + +The Secrets named in the alert are leftovers of the ASM v1 uninstall, not the certificate of a live mesh that failed to renew. Nothing renews them any more, so the alert keeps firing until they are removed. + +## Resolution + +### Cause 1: Mesh v1 `servicemesh` and `servicemeshgroup` are still on `global` + +1. On `global`, confirm the v1 objects and find the controller; on the business cluster, back up the Mesh v2 objects: + + ```bash + # global cluster + kubectl get servicemesh,servicemeshgroup -A + kubectl get servicemesh,servicemeshgroup -A -o yaml > /tmp/mesh-v1-leftovers.bak.yaml + kubectl get deploy -A | grep global-asm-controller + # business cluster + kubectl get istio,istiocni -A -o yaml > /tmp/mesh-v2.bak.yaml + ``` + +2. Delete the v1 objects on `global`. + + > **Warning:** Mesh v1 reconciliation stops for every business cluster these objects point at. Only when the v1 objects exist and `istio-system` is being recreated against a v2 install, and the backup exists. Rollback: `kubectl apply -f /tmp/mesh-v1-leftovers.bak.yaml` if v1 is still needed and v2 is not installed. If the delete is denied with `failed to get cluster`, follow the stuck-delete article first. + + ```bash + kubectl delete servicemesh -n + kubectl delete servicemeshgroup -n + ``` + +3. On the business cluster, delete the conflicting Mesh v2 resources and `istio-system`. + + > **Warning:** This removes the mesh control plane, gateways, and sidecar injection on that cluster; meshed calls fail until Mesh v2 is installed again. Only after step 2 is done and `/tmp/mesh-v2.bak.yaml` exists; use a maintenance window. Rollback: install Mesh v2 again; use the backup only as a reference for its settings. + + ```bash + kubectl delete istio -n + kubectl delete istiocni + kubectl delete namespace istio-system + ``` + +4. Install Mesh v2 again only after `kubectl get servicemesh,servicemeshgroup -A` on `global` shows nothing for this cluster and `istio-system` is no longer recreated. + +### Cause 2: Istio Secrets left behind by the ASM v1 uninstall + +1. List the Secrets the alert names and confirm that no mesh still uses them: + + ```bash + kubectl get secret -A | grep -Ei 'istio|cacert' + kubectl get servicemesh,istio,istiocni -A + kubectl get pods -A | grep -Ei 'istiod|istio-operator|asm-controller' + ``` + + A Secret referenced by a running Mesh v2 control plane, or by a mesh that is still installed, is not a leftover; leave it. + +2. Back up and delete each leftover Secret. + + > **Warning:** Deleting a Secret that a live control plane uses breaks that mesh's certificates. Only when ASM v1 is uninstalled, no v1 `servicemesh` remains, and no running `istiod` mounts the Secret. The backup contains a private key; store it securely. Rollback: `kubectl apply -f /tmp/istio-secret-.bak.yaml`. + + ```bash + kubectl get secret -n -o yaml > /tmp/istio-secret-.bak.yaml + kubectl delete secret -n + ``` + +3. Repeat for each Secret the alert names. The certificate alerts stop after the leftovers are gone. + +## Diagnostic Steps + +1. On `global`: + + ```bash + kubectl get servicemesh,servicemeshgroup -A + kubectl get deploy -A | grep -E 'global-asm-controller|asm-controller' + ``` + +2. On the business cluster: + + ```bash + kubectl get istio,istiocni -A + kubectl get ns istio-system + kubectl get sa -n istio-system istio-reader-service-account + kubectl get pods -n istio-system + ``` + +| What you see | Cause | +| --- | --- | +| `istio-system` or `istio-reader-service-account` returns after deletion, `global` still has `servicemesh` or `servicemeshgroup`, Mesh v2 resources fail to create | 1 | +| Certificate alert on Istio Secrets after ASM v1 is uninstalled, no running `istiod` uses them | 2 | +| Alert on `asm-istio-cacerts-` for a mesh that is still installed | Not a leftover; see the certificate expiry article | diff --git a/docs/en/solutions/LoadBalancer_VIP_unreachable_after_project_isolation_or_with_a_MetalLB_address_outside_the_node_segment_on_ACP.md b/docs/en/solutions/LoadBalancer_VIP_unreachable_after_project_isolation_or_with_a_MetalLB_address_outside_the_node_segment_on_ACP.md new file mode 100644 index 000000000..15b0c7d40 --- /dev/null +++ b/docs/en/solutions/LoadBalancer_VIP_unreachable_after_project_isolation_or_with_a_MetalLB_address_outside_the_node_segment_on_ACP.md @@ -0,0 +1,127 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" + - "4.3" +--- + +# LoadBalancer VIP unreachable after project isolation or with a MetalLB address outside the node segment on ACP + +## Issue + +The external IP that MetalLB assigned to a `LoadBalancer` Service does not answer on its port: + +- After the cluster network policy "Isolation between Projects" was turned on, every VIP exposed through MetalLB `LoadBalancer` Services became unreachable. Turning the isolation off brings them back. +- A tool exposed by MetalLB (for example Harbor) cannot be reached from the global cluster. The integration fails with `katanomi-plugin to harbor domain address request timeout`. The domain resolves, but from a global master and from inside the `katanomi-plugin` container network namespace: + - `ping ` returns `Destination Port Unreachable`, `4 packets transmitted, 0 received, +4 errors, 100% packet loss`; + - `curl -v https://:443 --insecure` and `telnet 443` hang at `Trying ...`. + +## Root Cause + +### Cause 1: with isolation between projects, LoadBalancer traffic arrives from the join subnet + +On a Kube-OVN cluster, traffic that arrives for a `LoadBalancer` Service is SNATed to the node's `ovn0` address, not to the node IP. `ovn0` sits in the Kube-OVN join subnet (the `Subnet` named `join`), so the backend Pod sees a source address from the join subnet. The isolation policy does not allow that subnet, so the traffic is dropped. + +The cluster network policy introduced in ACP 4.2 is based on `AdminNetworkPolicy` / `ClusterNetworkPolicy`, supports CIDR matching, and can be configured on the console page; the older cluster network policy lacks this. Kube-OVN does not support skipping ACL checks for specific subnets, so the fix is to allow the join subnet explicitly while keeping isolation on. + +### Cause 2: the MetalLB address is in a segment the nodes are not in + +In MetalLB layer 2 mode, a node answers ARP for the VIP on its own network, so the external address pool must be Layer 2 reachable from the nodes that announce it. When the nodes have their addresses in one `/24` and the MetalLB address is in a different `/24` that the nodes are not attached to, no node can answer for the VIP and it is unreachable from other networks. This is not a firewall problem; compare the VIP with the node segment before requesting firewall changes. + +## Resolution + +### Cause 1: with isolation between projects, LoadBalancer traffic arrives from the join subnet + +Only when the VIPs fail with isolation on and work with it off, on Kube-OVN. Read the join subnet CIDR: + +```bash +kubectl get subnet join -o jsonpath='{.spec.cidrBlock}{"\n"}' +ip -4 addr show ovn0 +``` + +On ACP 4.2 and later, keep isolation enabled and allow the join subnet on the cluster network policy page: + +1. Go to **Networking > Cluster Network Policies > Config Network Policy**. +2. With **Isolation between Projects** enabled, add an **Access Using IP Addresses** entry. +3. Set **Source IP Segment** to the join subnet CIDR and **Target** to the project whose LoadBalancer VIPs must be reachable. + +For a namespace that keeps its own `NetworkPolicy`, the equivalent ingress rule is: + +```yaml +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: allow-join-subnet + namespace: +spec: + podSelector: {} + policyTypes: + - Ingress + ingress: + - from: + - ipBlock: + cidr: +``` + +> **Warning:** The join subnet is the source of all node-to-Pod traffic on Kube-OVN, so allowing it lets every node, and anything SNATed through a node, reach the selected Pods. Narrow `podSelector` and `ports` to the LoadBalancer backends where required. Save the current policies first. Roll back with `kubectl -n delete networkpolicy allow-join-subnet`, or by removing the entry on the policy page. + +```bash +kubectl get networkpolicy -A -o yaml > /tmp/all-netpol.before.yaml +kubectl -n apply -f allow-join-subnet.yaml +curl -sv --max-time 5 http://:/ -o /dev/null +``` + +### Cause 2: the MetalLB address is in a segment the nodes are not in + +Change the address pool to a free address in the nodes' segment, then update everything that refers to the old address (DNS or CoreDNS host records, integrations, client configuration). + +> **Warning:** The Service gets a new external IP; clients and records that use the old address stop working until updated. Only when the VIP is outside every subnet configured on the announcing nodes, and the new address is not used by any other host. Save the pool and Service YAML; restoring them is the rollback. + +```bash +kubectl get ipaddresspools.metallb.io -A -o yaml > /tmp/metallb-pools.before.yaml +kubectl -n edit ipaddresspools.metallb.io +kubectl -n get svc -o wide +curl -kv --max-time 5 https://:443/ -o /dev/null +``` + +Test the new address from a global node before re-running the integration. If the integration uses a domain name held in a CoreDNS host record, point that record at the new address; see "CoreDNS Host Alias records do not take effect because a ResourcePatch removes data hosts from cpaas-coredns on ACP". + +## Diagnostic Steps + +1. Confirm the failing endpoint is the MetalLB external IP and Service port, and read the MetalLB logs: + + ```bash + kubectl get svc -A -o wide | grep LoadBalancer + kubectl get pod -A | grep -i metallb + kubectl -n logs --tail=50 + ``` + +2. Check whether isolation between projects is enabled and whether the VIPs failed when it was turned on: + + ```bash + kubectl get adminnetworkpolicy + kubectl get networkpolicy -A + ``` + + Yes to both: Cause 1. + +3. Compare the VIP with the node addresses: + + ```bash + kubectl get nodes -o wide + ip -4 addr show + ip -4 route + ``` + + A VIP in no node subnet: Cause 2. To see the failure from the calling component, repeat the test inside its network namespace on a global master: + + ```bash + crictl ps | grep katanomi-plugin + crictl inspect | grep -i '"pid"' + nsenter -t -n curl -v --max-time 10 https://:443 --insecure + ``` + +If an application Pod (not the VIP) broke after a policy change, see "Application breaks after a NetworkPolicy change: Ingress, Pod-to-Pod ping, Redis Sentinel or Harbor on ACP". diff --git a/docs/en/solutions/Local_overrides_reverted_or_blocking_an_ACP_upgrade_ResourcePatch_ALB_ConfigMap_chart_and_feature_gate_changes.md b/docs/en/solutions/Local_overrides_reverted_or_blocking_an_ACP_upgrade_ResourcePatch_ALB_ConfigMap_chart_and_feature_gate_changes.md new file mode 100644 index 000000000..86216a731 --- /dev/null +++ b/docs/en/solutions/Local_overrides_reverted_or_blocking_an_ACP_upgrade_ResourcePatch_ALB_ConfigMap_chart_and_feature_gate_changes.md @@ -0,0 +1,246 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.6" + - "3.8" + - "3.12" + - "3.16" + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Local overrides reverted or blocking an ACP upgrade: ResourcePatch, ALB ConfigMap, chart and feature gate changes + +## Issue + +An ACP upgrade is blocked by a pre-upgrade check, or it completes and a hand-edited setting returns to the catalog value. Typical signatures: + +- The pre-upgrade checklist prints `ERROR: check alb configmap patch`. +- `kubectl get rpch -A` returns a long list of `ResourcePatch` objects. +- New Pods in a Service Mesh namespace log `FailedToRetrieveImagePullSecret` naming `global-registry-auth`. +- Charts are reported missing from the local template repository after a Pod moved to another node, or hand edits disappear after a HelmRequest sync. +- ALB on a business cluster restarts while only the `global` cluster is being upgraded. +- ALB is missing after the upgrade to 4.3. +- A custom inspector role can suddenly use WebCLI to exec into Pods and log in to nodes. +- The documented fix for checklist item `check_alb_configmap_ge_3_12` fails with `unknown field "sepc.confg"`. + +ACP stores some local changes as `ResourcePatch` objects (`rpch`) and re-applies them on every reconcile. A direct edit of a generated ConfigMap or of a Helm-rendered object is not such a patch, so the next sync puts the catalog value back. + +## Root Cause + +### Cause 1: ERROR: check alb configmap patch + +The checklist names an ALB ConfigMap that was edited in place. An in-place ConfigMap edit is not an override the upgrade preserves: the upgrade rewrites the ConfigMap from the chart, and the edit is gone afterwards even though the checklist only warned about it. + +### Cause 2: ResourcePatch (rpch) entries block or silently override the upgrade + +`rpch` objects record personalized edits of platform components. Before a business-cluster upgrade (for example 4.2.2 to 4.2.5) or a `global` patch upgrade (for example 4.1.2 to 4.1.5), every entry has to be classified by what it changes. Image overrides, including old hotfix images, are not safe to carry across an upgrade. CPU and memory overrides usually are. Empty patches that only store an annotation and an update timestamp are noise. A patch whose effect nobody can explain is a stop sign. Which patches are kept is a decision for each environment; there is no global allow-list. + +### Cause 3: Subnet isolation or a temporary otel injection image is still in place + +For the upgrade from 3.18.1 to 4.2.5, subnet resource isolation and cross-subnet access that were changed away from the platform default have to be restored to the default before the upgrade and configured again from the 4.2.5 documentation afterwards. A changed Harbor storage address, the etcd-mirror synchronization of Prometheus resources, and nevermore `compression=none` can stay in place across that upgrade. + +A temporary image used so that the OpenTelemetry injection jar can start despite wrong file permissions must be checked against the target 4.x build. If the target build contains the fix, revert the temporary image before the upgrade. If it does not, the image is an image-type `rpch` and Cause 2 applies as soon as a fixed image exists. + +### Cause 4: Chart template not found, and manual edits reappear as the chart + +`/cpaas/captain-chartmuseum`, the local template repository, is a hostPath. When its Pod is rescheduled onto another node, that node does not have the files and the chart is reported missing. This happens on every move of the Pod (for example during a 3.6 to 3.8 upgrade). A related failure is a HelmRequest (`helmrequests.app.alauda.io`) resync that redeploys the chart and overwrites objects that were edited by hand. + +### Cause 5: FailedToRetrieveImagePullSecret global-registry-auth + +After a business cluster that uses Service Mesh is upgraded, new microservice Pods raise: + +`FailedToRetrieveImagePullSecret: Unable to retrieve some image pull secrets (global-registry-auth); attempting to pull the image may not succeed` + +`IstioOperator` and the injector ConfigMap `istio-sidecar-injector-1-20` list `imagePullSecrets: global-registry-auth`. Since the 3.16 injection change, that name is added to the `istio-proxy` injection. The Secret does not exist in the business namespace, so every injection logs the warning. + +### Cause 6: ALB is not enabled after the upgrade to 4.3 + +On 4.3, ALB is off unless its feature gate `alaudafeaturegates/alb2` in `cpaas-system` is enabled. Existing ALB installations can keep running. A new 4.3 deployment should not enable ALB as the long-term design: ALB is deprecated from 4.3.1, and the replacement is `ingress-nginx-operator` or Envoy Gateway. + +### Cause 7: A custom inspector role can use WebCLI on nodes + +A custom platform-inspector role that was limited before the upgrade can exec everywhere afterwards, including node login. Its manifest contains `acp-podsexec` and `acp-rootexec`. The built-in platform auditor role on 4.2 is query-only on WebCLI and has no create, delete, or node login permission; the custom role picked up the extra entries and was not trimmed. + +### Cause 8: Business-cluster ALB restarts during the global upgrade + +Only `global` is upgraded, but ALB on a business cluster restarts. The ALB Deployment on the business cluster no longer matches the HelmRequest on `global`, so the sync during the global upgrade writes the HelmRequest's desired state and the business ALB rolls. Once the pre-upgrade logs and audit data are gone, the field that differed cannot be reconstructed; only the current mismatch can be checked. + +### Cause 9: check_alb_configmap_ge_3_12 fix fails with unknown field "sepc.confg" + +The checklist reports `check_alb_configmap_ge_3_12` as failed, and applying the documented fix fails with `unknown field "sepc.confg"` (the field the fix writes is the ALB's `spec.config`). `spec.config` belongs to the `v2beta1` version of the ALB2 resource. The update was sent without selecting resource type `alb2.v2beta1.crd.alauda.io`. Repeating it with that resource type succeeds. + +## Resolution + +### Cause 1: ERROR: check alb configmap patch + +Export the ConfigMap before the upgrade and compare it afterwards. If the upgrade restored the chart copy, put the required keys back through the ALB overwrite mechanism, not through another `kubectl edit` of the ConfigMap. + +```bash +kubectl get configmap -n -o yaml > -configmap.yaml +``` + +> **Warning:** Re-applying a full ConfigMap YAML with `kubectl apply` fights the controller and can drop keys the chart added during the upgrade. Use the ALB overwrite path for that setting and keep `-configmap.yaml` as the backup. If the overwrite itself is wrong, remove the overwrite entry so the chart value returns; do not delete the ALB instance to recover one ConfigMap key. + +### Cause 2: ResourcePatch (rpch) entries block or silently override the upgrade + +Export every patch, then classify each entry: + +```bash +kubectl get rpch -A -o yaml > rpch-backup.yaml +``` + +| What the patch changes | Before the upgrade | +| --- | --- | +| Image, including an old hotfix image | Delete it. | +| CPU or memory only | May stay. | +| CoreDNS host resolution, or a dex access address | May stay through the upgrade. Delete it afterwards and move the data into plugin management or the platform access-address configuration. | +| Empty patch (annotation and timestamp only) | Delete. | +| A file under the component configuration | If nothing depends on it strongly, back up the live resource, delete the `rpch`, and restore the intended content after the upgrade. | +| Startup arguments or an environment variable of a component | May stay, but the flag's location can move. If the component then crashes or conflicts, delete that `rpch`. | +| Anything you cannot explain | Stop. Neither delete it nor keep it blindly. | + +Entries of type `Artifact` can be left out of this classification; confirm the type before you skip one. + +> **Warning:** Deleting an image `rpch` makes the component run the catalog image at the next reconcile, which can restart it and drop the hotfix. Take `rpch-backup.yaml` first. If the catalog image is worse than the hotfix and the upgrade has not started, restore that one object with `kubectl apply`. Re-applying an old image patch after the upgrade puts the old image back; do that only to recover a failed upgrade, and remove it again before the next attempt. Deleting a CPU or memory patch resets the component to the chart requests, which can cause OOM or throttling. + +```bash +kubectl delete rpch -n +``` + +### Cause 3: Subnet isolation or a temporary otel injection image is still in place + +Restore subnet resource isolation and cross-subnet access to the platform default before the 4.2.5 upgrade, then configure them again following the 4.2.5 documentation. Do not carry the pre-4.2 isolation syntax across and edit it in place. + +For the OpenTelemetry injection image, confirm whether the target 4.x build fixes the jar permission failure. If it does, point the injection back to the catalog image before the upgrade. If it does not, keep the workaround until a fixed image is uploaded, then delete it as an image `rpch` (Cause 2). + +> **Warning:** Restoring subnet isolation to the default opens or closes cross-subnet paths that the custom policy enforced. Do it in the maintenance window with the previous manifest saved. Roll back by re-applying that manifest only if the upgrade has not started; once the cluster is on 4.2.5, configure it from the new documentation instead. Reverting the otel image before the target build has the fix brings back the Pod start failure. + +### Cause 4: Chart template not found, and manual edits reappear as the chart + +Find the Pod that mounts the hostPath and the node that has the files: + +```bash +kubectl get pods -A -o json | python3 -c 'import json,sys +d=json.load(sys.stdin) +for p in d.get("items",[]): + for v in p["spec"].get("volumes",[]): + path=(v.get("hostPath") or {}).get("path") or "" + if "captain-chartmuseum" in path: + print(p["metadata"]["namespace"], p["metadata"]["name"], p["spec"].get("nodeName"), path)' +``` + +On that node, `ls /cpaas/captain-chartmuseum` must show the charts. For the duration of the upgrade, keep the Pod on that node by setting a nodeSelector `kubernetes.io/hostname=` on its owner. + +> **Warning:** A nodeSelector pins the template repository to one node; if that node is drained, the charts disappear again. Removing the nodeSelector returns the Pod to normal scheduling and loses the charts unless the files were copied to shared storage first. Do not delete the Pod to "refresh" a missing chart. + +Put every intentional difference into Helm values and change it through those values. A hand edit of a rendered Deployment is overwritten the next time the HelmRequest syncs. + +### Cause 5: FailedToRetrieveImagePullSecret global-registry-auth + +Confirm the Secret is absent, then remove `global-registry-auth` from the ServiceMesh configuration (`IstioOperator` at `spec.values.global.imagePullSecrets`) and from the injector ConfigMap that still lists it. Creating the missing Secret is not the fix. + +```bash +kubectl get secret global-registry-auth -n +kubectl get servicemesh -A +kubectl get istiooperator -A +kubectl get configmap -A | grep istio-sidecar-injector +``` + +> **Warning:** Restarting microservice Pods re-injects the sidecar and drops their current connections. Do it after neither the ServiceMesh nor the injector names `global-registry-auth`, in a window those services can restart. Roll back by restoring the previous ServiceMesh and ConfigMap YAML and restarting again. + +```bash +kubectl -n rollout restart deploy +``` + +### Cause 6: ALB is not enabled after the upgrade to 4.3 + +If the cluster already serves traffic with ALB and the gate is off, enable it. On a new 4.3 installation, do not enable ALB; plan `ingress-nginx-operator` or Envoy Gateway instead. From 4.3.1, treat enabling ALB as a temporary bridge. + +```bash +kubectl get alaudafeaturegates -n cpaas-system alb2 -o yaml +``` + +> **Warning:** Enabling the gate reconciles ALB on a cluster where it is deprecated from 4.3.1. Enable it only to keep an ALB that is already in use. To turn it off again, patch `spec.enabled` to `false` with the same annotation. Do not delete the `alb2` feature-gate object. + +```bash +kubectl patch alaudafeaturegates alb2 -n cpaas-system --type=merge -p '{"spec":{"enabled":true},"metadata":{"annotations":{"cpaas.io/ignoreme":"true"}}}' +``` + +Keep the `cpaas.io/ignoreme` annotation in the patch so the setting persists. Read the object again and require `spec.enabled: true`. + +### Cause 7: A custom inspector role can use WebCLI on nodes + +Find the custom role YAML that contains `acp-podsexec` or `acp-rootexec`, remove those entries, and keep `get`, `list`, and `watch`. The 4.2 platform auditor is the reference for query-only access. + +```bash +grep -n -e acp-podsexec -e acp-rootexec .yaml +``` + +> **Warning:** Applying the trimmed role removes node login and Pod exec from everyone bound to it. Save the previous YAML and restore it if the trim removes a permission the role still needs. Do not delete the whole role to remove two entries; that locks out the inspectors. + +```bash +kubectl apply -f .yaml +``` + +If the role is not a plain ClusterRole, apply it through the same API the platform uses for custom roles. + +### Cause 8: Business-cluster ALB restarts during the global upgrade + +Before upgrading `global`, compare the business ALB Deployment with the HelmRequest on `global` that owns it. Put the intended values on the HelmRequest so the sync is a no-op, or schedule the business ALB restart inside the global window and inform the application owners. + +```bash +kubectl get helmrequests.app.alauda.io -A +kubectl -n get deploy -o yaml > business-alb-deploy.yaml +``` + +> **Warning:** Editing the HelmRequest to match a hand-tuned Deployment rolls ALB when the request syncs. Change it at the start of the window, not in the middle of the global upgrade. To undo a bad edit, re-apply the previous HelmRequest YAML. Deleting the HelmRequest removes the release, not just the drift. + +Aligning the HelmRequest only removes this drift-induced restart. The ALB restart that comes with upgrading the cluster that runs the ALB cannot be avoided or deferred; see "ALB load balancer Pods restart during an ACP upgrade and the restart cannot be deferred". + +### Cause 9: check_alb_configmap_ge_3_12 fix fails with unknown field "sepc.confg" + +Repeat the documented update with resource type `alb2.v2beta1.crd.alauda.io` selected. In the console, pick that resource type before editing. From the command line, name the fully qualified resource: + +```bash +kubectl get alb2.v2beta1.crd.alauda.io -A +kubectl -n get alb2.v2beta1.crd.alauda.io -o yaml > -alb2.yaml +``` + +> **Warning:** Changing `spec.config` reconciles the ALB and can restart its Pods, which interrupts traffic through it briefly. Precondition: `-alb2.yaml` saved and a maintenance window. Rollback: `kubectl apply -f -alb2.yaml`. + +```bash +kubectl -n edit alb2.v2beta1.crd.alauda.io +``` + +Make the change the checklist item's documentation describes, save, and run the checklist again. + +## Diagnostic Steps + +List the overrides before judging upgrade readiness. A healthy node list does not mean the upgrade keeps local edits. + +```bash +kubectl get rpch -A +kubectl get helmrequests.app.alauda.io -A +kubectl get alaudafeaturegates -n cpaas-system alb2 -o jsonpath='{.spec.enabled}{"\n"}' +``` + +Use the first matching signature: + +1. Checklist text `check alb` with `configmap patch`: Cause 1. +2. Any `rpch` row: Cause 2. Image rows are deleted; a row is not kept just because the component is currently healthy. +3. Subnet isolation or cross-subnet access is non-default and the target is 4.2.5, or an otel injection image is a known temporary build: Cause 3. +4. A chart template cannot be found, or a hand edit vanished after a sync, and a volume uses `/cpaas/captain-chartmuseum`: Cause 4. +5. Event text naming `global-registry-auth`: Cause 5. Check the business namespace for the Secret before editing the injector. +6. ALB missing on 4.3 and `alaudafeaturegates/alb2` not enabled: Cause 6. +7. A custom inspector role can exec or log in to nodes, and its YAML contains `acp-podsexec` or `acp-rootexec`: Cause 7. +8. Business ALB restarted while only `global` changed, and its Deployment does not match the HelmRequest on `global`: Cause 8. +9. The `check_alb_configmap_ge_3_12` fix fails with `unknown field "sepc.confg"` or `unknown field "spec.config"`: Cause 9. + +A `ResourcePatch` whose owner cannot say what it changes stays a stop. Do not delete it to make `kubectl get rpch` shorter. diff --git a/docs/en/solutions/Log_Kafka_lag_from_Elasticsearch_write_hot_spots_breaker_trips_IK_analyzer_errors_or_too_few_partitions_on_ACP.md b/docs/en/solutions/Log_Kafka_lag_from_Elasticsearch_write_hot_spots_breaker_trips_IK_analyzer_errors_or_too_few_partitions_on_ACP.md new file mode 100644 index 000000000..40e6d2284 --- /dev/null +++ b/docs/en/solutions/Log_Kafka_lag_from_Elasticsearch_write_hot_spots_breaker_trips_IK_analyzer_errors_or_too_few_partitions_on_ACP.md @@ -0,0 +1,207 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.8.x + - 3.14.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Log Kafka lag from Elasticsearch write hot spots, breaker trips, IK analyzer errors, or too few partitions on ACP + +## Issue + +Logs are late or missing, and the platform log Kafka is backed up. Unlike a plain capacity shortage, the lag has a recognizable shape: + +- Two Elasticsearch nodes sit at nearly 100% CPU while a third one idles. +- `lanaya` reports a `Data too large` circuit-breaker message. +- `lanaya` loops on `bulk response error`, and the Elasticsearch log shows IK analyzer offset errors. +- The lag is spread evenly over every partition of a topic that has too few partitions for the clusters it serves, while Elasticsearch has headroom. + +The application log topic is `ALAUDA_LOG_TOPIC`, and its platform consumer group is `alauda_log`. `lanaya` (Deployment in `cpaas-system`) consumes it and bulk-writes to Elasticsearch, taking the next batch of a partition only after the current one is persisted. + +When the lag sits on a single partition, see [Log Kafka lag on a single partition from oversized log entries or a stuck offset on ACP](./Log_Kafka_lag_on_a_single_partition_from_oversized_log_entries_or_a_stuck_offset_on_ACP.md). + +## Root Cause + +### Cause 1: Application log indices have one primary shard, so two nodes carry all writes + +The daily application index (`log-workload-`) is created with one primary shard. It therefore lands on two Elasticsearch nodes, one for the primary and one for the replica. Those two nodes sit at about 96% CPU while the third idles, and Kafka backs up. Scaling Elasticsearch out does not spread the load. The index can stay `yellow` (1 primary, 1 replica, hundreds of millions of documents), with one shard stuck initializing while the index keeps growing. + +The shard count comes from the `log.json` index template (`"index_patterns": "log-*"`, `"index.lifecycle.name": "alauda-log"`). The platform manages that template through key `log.json` of ConfigMap `aiops-logging-config` in `cpaas-system`. Changing `"number_of_shards"` from 1 to `3`, while keeping `"number_of_replicas": 1` and `"refresh_interval": "10s"`, spreads the writes, and log queries return to normal. + +When `lanaya` already has as many replicas as the topic has partitions and the daily index still has a single shard on one hot node, the backlog can also clear on its own once the Elasticsearch load drops. Check the `log.json` template in that case as well. + +### Cause 2: lanaya writes fail with Data too large from the parent circuit breaker + +`lanaya` logs a write error such as: + +```text +Data too large, data for [] would be [8958381455/8.3gb], which is larger than the limit of [8958371430/8.3gb], usages [request=0/0b, fielddata=7652620260/7.1gb, ... +``` + +The Elasticsearch pods are healthy. Fielddata holds most of the breaker budget (7.1 GB of 8.3 GB here), so each bulk write trips the parent breaker, and the displayed logs fall behind. Raising `indices.breaker.total.limit` to `92%` and `indices.breaker.fielddata.limit` to `85%` restores writes. Treat this as a temporary fix. Adding heap is the lasting one. + +### Cause 3: lanaya repeats bulk response error because the Elasticsearch IK analyzer rejects documents + +`lanaya` cannot consume and loops on: + +```text +elasticsearch/elasticsearch.go:166] Database bulk error {"bulkSize": 10000, ... "error": "bulk response error"} +consumer/consumer.go:107] Failed to insert message ... "error": "bulk response error" +consumer/consumer.go:48] Failed to consume message, try to reconnect {"category": "log", ... "error": "bulk response error"} +``` + +The platform Elasticsearch log (for example Elasticsearch 6.8.23 on ACP 3.14.1) repeats `failed to execute bulk item` for `log-platform-` with: + +```text +java.lang.IllegalArgumentException: startOffset must be non-negative, and endOffset must be >= startOffset, and offsets must not go backwards startOffset=846,endOffset=847,lastStartOffset=847 for field 'log_data' +``` + +A line such as `(index) index {[log-platform-][_doc][...], source[n/a, actual length: [2.7kb], max length: 2kb]}` may be the first error found. It only shows a truncated document in the log message and is not the cause. Without the Elasticsearch hotfix, the platform is affected by a known IK analyzer issue that produces these offset errors on `log_data`. Replacing the Elasticsearch image with the fixed one lets `lanaya` consume normally again. + +### Cause 4: The topic has too few partitions for the clusters it serves + +An Elasticsearch with 3 master and 8 data nodes stores logs for 15 clusters. Elasticsearch runs at about 60% memory and 40% CPU, the indices are normal, and the cluster is still behind. `_cluster/health` is `green` with 0 unassigned shards, and every log component is `Running` without restarts. The `lanaya` logs show delays (`dly`) of tens of hours on the log partitions. + +Consumption parallelism is too low for the volume. Expanding the log topic from 30 to 60 partitions makes the platform backlog, and the backlog of external consumer groups, fall steadily. `lanaya` replicas beyond the partition count add nothing, and adding replicas does not help while Elasticsearch is the limit. A `razor` line such as `Validate token error ... the server has asked for the client to provide credentials` in the same period is not related. + +## Resolution + +Load the Elasticsearch credentials once for the API calls below: + +```bash +ES_USER=$(kubectl -n cpaas-system get secret elasticsearch-basic-auth -o jsonpath='{.data.username}' | base64 -d) +ES_PASS=$(kubectl -n cpaas-system get secret elasticsearch-basic-auth -o jsonpath='{.data.password}' | base64 -d) +es() { kubectl -n cpaas-system exec cpaas-elasticsearch-0 -c cpaas-elasticsearch -- \ + curl -sS -u "${ES_USER}:${ES_PASS}" "$@"; } +``` + +### Cause 1: Application log indices have one primary shard, so two nodes carry all writes + +1. Confirm the signature: + + ```bash + es 'http://cpaas-elasticsearch:9200/_cat/shards/log-workload-*?v&s=index' | tail -20 + es 'http://cpaas-elasticsearch:9200/_cat/nodes?v&h=name,cpu,heap.percent' + es 'http://cpaas-elasticsearch:9200/_template?pretty' | grep -n -E '"log-\*"|number_of_shards|number_of_replicas|alauda-log' + ``` + + One primary (`p`) and one replica (`r`) per daily application index, with two nodes near 96% CPU, matches this cause. + +2. Change the shard count in the ConfigMap. The platform reverts direct template edits, so edit it here. + + > **Warning:** Changing `number_of_shards` in the `log.json` template affects every index created from it afterwards. More shards per daily index means more shards in the cluster in total. Precondition: only when the daily application index has one primary and two nodes carry the write load. Rollback: `kubectl apply -f /tmp/aiops-logging-config.before.yaml`; indices that already exist keep the shard count they were created with. + + ```bash + kubectl -n cpaas-system get cm aiops-logging-config -o yaml > /tmp/aiops-logging-config.before.yaml + kubectl -n cpaas-system edit cm aiops-logging-config + ``` + + In key `log.json`, set `"number_of_shards": 3`, and keep `"number_of_replicas": 1` and `"refresh_interval": "10s"`. + +3. Existing indices keep their shard count, so the load spreads from the next daily index onward. + +### Cause 2: lanaya writes fail with Data too large from the parent circuit breaker + +1. Confirm the signature: + + ```bash + kubectl -n cpaas-system logs deploy/lanaya --tail=200 | grep 'Data too large' + es 'http://cpaas-elasticsearch:9200/_nodes/stats/breaker?pretty' | grep -E '"(parent|fielddata)"|limit_size|estimated_size' + ``` + +2. Raise the breaker limits. + + > **Warning:** Raising the breaker limits lets Elasticsearch use more heap before it rejects requests. If the heap is really exhausted, the node can hit a Java `OutOfMemoryError` instead of rejecting the write. Precondition: only when `lanaya` logs `Data too large` and fielddata dominates the usage. Save the current settings first. Rollback: set both keys to `null` to return to the defaults. + + ```bash + es 'http://cpaas-elasticsearch:9200/_cluster/settings?pretty' > /tmp/es-cluster-settings.before.json + es -X PUT -H 'Content-Type: application/json' \ + 'http://cpaas-elasticsearch:9200/_cluster/settings' \ + -d '{"persistent":{"indices.breaker.total.limit":"92%","indices.breaker.fielddata.limit":"85%"}}' + ``` + +3. Writes should resume immediately. If the breaker keeps tripping, add heap by raising the Elasticsearch memory limit on the log storage plugin. + +### Cause 3: lanaya repeats bulk response error because the Elasticsearch IK analyzer rejects documents + +1. Confirm the signature: + + ```bash + kubectl -n cpaas-system logs deploy/lanaya --tail=500 | grep -E 'bulk response error' | tail -5 + for p in $(kubectl -n cpaas-system get pod -l service_name=cpaas-elasticsearch -o name); do + kubectl -n cpaas-system logs "$p" -c cpaas-elasticsearch --tail=2000 | grep -m3 'startOffset must be non-negative' + done + kubectl -n cpaas-system get sts cpaas-elasticsearch -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' + ``` + + `startOffset must be non-negative ... for field 'log_data'` in the Elasticsearch log, together with `bulk response error` in `lanaya`, matches this cause. + +2. Ask Alauda support for the Elasticsearch hotfix image that matches the platform version. Compare its tag with the image printed above to see whether the hotfix is already applied. + +3. Apply the hotfix image. + + > **Warning:** Replacing the Elasticsearch image rolls every Elasticsearch pod, and search and ingestion pause on each pod while it restarts. Precondition: only when the offset error is confirmed, the hotfix image is the one provided for this platform version, and you saved the StatefulSet (and the `rpch` that pins the image, if one exists). Rollback: apply the saved StatefulSet or `rpch` to return to the previous image. + + ```bash + kubectl -n cpaas-system get sts cpaas-elasticsearch -o yaml > /tmp/cpaas-elasticsearch.sts.before.yaml + kubectl get rpch -A | grep -i elasticsearch + ``` + + Apply the image the way support specifies for that version. Then confirm that `lanaya` stops logging `bulk response error` and that the consume rate rises. + +### Cause 4: The topic has too few partitions for the clusters it serves + +1. Read the current partition count. Use the listener and client configuration that this cluster's brokers expose: + + ```bash + kubectl -n cpaas-system exec cpaas-kafka-0 -- \ + kafka-topics.sh --bootstrap-server \ + --command-config --describe --topic ALAUDA_LOG_TOPIC | head -3 + ``` + +2. Add partitions. + + > **Warning:** A partition count can only be increased. New partitions change which partition a key maps to, and they use more broker disk and file handles. Precondition: only when Elasticsearch has headroom (Causes 1 to 3 are ruled out) and the lag persists across many partitions. Rollback: none; a topic cannot shrink its partition count. + + ```bash + kubectl -n cpaas-system exec cpaas-kafka-0 -- \ + kafka-topics.sh --bootstrap-server \ + --command-config --alter --topic ALAUDA_LOG_TOPIC --partitions 60 + ``` + +3. Watch the platform and external consumer groups in the describe output. Both backlogs should fall steadily. Scale `lanaya` only up to the partition count. + +## Diagnostic Steps + +1. Read per-partition lag from the `log/Cpaas Log Stack for Elasticsearch` dashboard or with `kafka-consumer-groups.sh --describe --group alauda_log`. + +2. Check the consumer and the nodes: + + ```bash + kubectl -n cpaas-system logs deploy/lanaya --tail=300 | grep -Ei 'Data too large|bulk response error|Failed to insert' + es 'http://cpaas-elasticsearch:9200/_cat/nodes?v&h=name,cpu,heap.percent' + es 'http://cpaas-elasticsearch:9200/_cat/indices?v' | grep -v green + ``` + +3. Match the signature: + + | What you see | Cause | + |---|---| + | Two nodes near 100% CPU and a third idle, with today's `log-workload-*` index at one primary | Cause 1 | + | `Data too large` with fielddata holding most of the breaker | Cause 2 | + | `bulk response error` in `lanaya` and `startOffset must be non-negative` in the Elasticsearch log | Cause 3 | + | Lag spread evenly over every partition, Elasticsearch well below its limits, and all indices `green` | Cause 4 | + +## Related Information + +- Related articles: [Log Kafka lag on a single partition from oversized log entries or a stuck offset on ACP](./Log_Kafka_lag_on_a_single_partition_from_oversized_log_entries_or_a_stuck_offset_on_ACP.md) covers lag that sits on one partition. +- [Elasticsearch log index replica count reverts after editing the index template on ACP](./Elasticsearch_log_index_replica_count_reverts_after_editing_the_index_template_on_ACP.md) +- [Log search misses entries: IK analyzer tokens, uppercase keywords, clock skew, or page scope on ACP](./Log_search_misses_entries_IK_analyzer_tokens_uppercase_keywords_clock_skew_or_page_scope_on_ACP.md) diff --git a/docs/en/solutions/Log_Kafka_lag_on_a_single_partition_from_oversized_log_entries_or_a_stuck_offset_on_ACP.md b/docs/en/solutions/Log_Kafka_lag_on_a_single_partition_from_oversized_log_entries_or_a_stuck_offset_on_ACP.md new file mode 100644 index 000000000..9d80cd201 --- /dev/null +++ b/docs/en/solutions/Log_Kafka_lag_on_a_single_partition_from_oversized_log_entries_or_a_stuck_offset_on_ACP.md @@ -0,0 +1,149 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.8.x + - 3.14.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Log Kafka lag on a single partition from oversized log entries or a stuck offset on ACP + +## Issue + +Logs are late or missing, and the lag of the platform log Kafka sits on **one** partition of `ALAUDA_LOG_TOPIC`, while the other partitions are at 1 or 2. Typical reports: + +- "Kafka is backed up and log query fails." +- "One partition has all the lag." +- "One partition does not consume logs, and `lanaya` shows many errors." +- The alert `Kafka 消费队列滞后增长速率 > 0 且持续 1 分钟` ("Kafka consumer lag growth rate above 0 for 1 minute") fires again and again. + +The application log topic is `ALAUDA_LOG_TOPIC`, and its platform consumer group is `alauda_log`. `lanaya` (Deployment in `cpaas-system`) consumes the topic and bulk-writes to Elasticsearch. It takes the next batch of a partition only after the current batch is persisted, so one bulk that keeps failing stops that partition. The `lanaya` bulk size is set with the `-es-bulk-size` flag (default 10000). The Elasticsearch settings are in ConfigMap `es-config` in `cpaas-system`. + +Per-partition lag is shown on the Grafana dashboards `log/Cpaas Log Stack for Elasticsearch` (Lag detail by `alauda_log/`) and `Platform Middleware / Kafka Cluster Status`. You can also read it from `kafka-consumer-groups.sh --describe`. + +When the lag is spread over many partitions, or two Elasticsearch nodes carry all the load, see [Log Kafka lag from Elasticsearch write hot spots, breaker trips, IK analyzer errors, or too few partitions on ACP](./Log_Kafka_lag_from_Elasticsearch_write_hot_spots_breaker_trips_IK_analyzer_errors_or_too_few_partitions_on_ACP.md). + +## Root Cause + +### Cause 1: Lag sits on one partition because a few log entries are too large + +All the lag is on the log topic, on one partition. On the `log/Cpaas Log Stack for Elasticsearch` dashboard the production and consumption rates match, while the lag of one partition rises in a straight line past 40,000. Every other partition stays at 1 or 2. The broker logs show only routine segment deletion. Scaling `lanaya` out to more replicas has no effect. + +The `lanaya` log for that partition shows repeated `Try to record message` entries and Kafka reader entries with a very large `"Content-Length"` (about 10 MB). Filtered by the consumer id, the log shows a loop: + +```text +Try to do bulk {"from": 701, "to": 1402, "batch": 701} +Do bulk error ... "error": "[413 Request Entity Too Large] " +Failed to insert message, abort +Try to do consume again +``` + +Elasticsearch returns `413` for the bulk, because some application log entries are individually huge. Raising `http.max_content_length` to `2000mb` in ConfigMap `es-config`, and setting `-es-bulk-size=100` on `lanaya`, makes the partition move again. A small bulk size lowers throughput, so the lasting fix is to stop the application from writing oversized entries, and then return `-es-bulk-size` to its default. + +### Cause 2: One partition stops consuming at a fixed offset, and the stuck record is skipped with kafka-delete-records.sh + +One partition of the log topic does not consume at all. Its `CURRENT-OFFSET` stays at one fixed offset, while `lanaya` logs many errors. If the `lanaya` lines for that partition show the `413` loop, use Cause 1 first. No more specific root cause is known. The remediation is to delete the records of that partition up to its latest offset with `kafka-delete-records.sh`, so that the consumer moves past the record it cannot process. + +## Resolution + +The Kafka commands below run inside `cpaas-kafka-0`. Use the listener and client configuration that this cluster's brokers expose. On some clusters the plaintext `localhost:9092` listener times out, while the TLS listener works with the client properties file mounted in the pod. Do not mix the two. If a script is not on the `PATH`, it is in `/opt/kafka/bin/`. + +### Cause 1: Lag sits on one partition because a few log entries are too large + +1. Find the `lanaya` lines for the lagging partition, then the bulk loop for its consumer: + + ```bash + for p in $(kubectl -n cpaas-system get pods -o name | grep lanaya); do + echo "== $p" + kubectl -n cpaas-system logs "$p" --tail 10000 | grep '"ptn": ""' | grep log | tail -3 + done + kubectl -n cpaas-system logs --tail 10000 | grep '' | grep -E 'Try to do bulk|413|abort' | tail -10 + ``` + + Replace `` with the lagging partition, and `` with the id shown on those lines (for example `log-01-04i`). A `413 Request Entity Too Large` loop for that consumer matches this cause. + +2. Save the current configuration: + + ```bash + kubectl -n cpaas-system get cm es-config -o yaml > /tmp/es-config.before.yaml + kubectl -n cpaas-system get deploy lanaya -o yaml > /tmp/lanaya.before.yaml + ``` + +3. Raise the request size limit and lower the bulk size. + + > **Warning:** A larger `http.max_content_length` lets one request use more heap, and a smaller `-es-bulk-size` lowers `lanaya` throughput on every partition. The `lanaya` edit restarts its pods. `http.max_content_length` is a static Elasticsearch node setting, so it takes effect only after the Elasticsearch pods restart. Restart them one at a time. Elasticsearch does not accept a value above about 2 GB, so do not go past `2000mb`. Precondition: only when this cause is confirmed (lag on one partition and `413` in `lanaya` for it), and the Elasticsearch heap is about half of its memory limit. Rollback: apply `/tmp/es-config.before.yaml` and `/tmp/lanaya.before.yaml`, then restart Elasticsearch one pod at a time. + + ```bash + kubectl -n cpaas-system edit cm es-config + kubectl -n cpaas-system edit deploy lanaya + ``` + + Set `http.max_content_length: 2000mb` in `es-config`, and add `-es-bulk-size=100` to the `lanaya` arguments. Restart the Elasticsearch pods one at a time: + + ```bash + for p in $(kubectl -n cpaas-system get pod -l service_name=cpaas-elasticsearch -o name); do + kubectl -n cpaas-system delete "$p"; sleep 15 + kubectl -n cpaas-system wait --for=condition=Ready "$p" --timeout=15m + done + ``` + +4. After the partition catches up and the application no longer writes oversized entries, remove `-es-bulk-size` again. + +### Cause 2: One partition stops consuming at a fixed offset, and the stuck record is skipped with kafka-delete-records.sh + +1. Read the stuck offset and the log-end offset of each partition: + + ```bash + kubectl -n cpaas-system exec cpaas-kafka-0 -- \ + kafka-consumer-groups.sh --bootstrap-server \ + --command-config --describe --group alauda_log + ``` + + A partition whose `CURRENT-OFFSET` does not move between runs, while the other partitions advance, matches this cause. + +2. Delete the records of that partition up to the log-end offset. + + > **Warning:** `kafka-delete-records.sh` permanently deletes every record in that partition below the offset you give. With the log-end offset, the whole unconsumed range of that partition never reaches the log store. Precondition: only when one partition is stuck at a fixed offset while the others consume, Cause 1 is ruled out, and losing those records is accepted. Rollback: none. + + ```bash + kubectl -n cpaas-system exec -i cpaas-kafka-0 -- sh -c 'cat > /tmp/delete-records.json' <<'EOF' + {"partitions": [{"topic": "ALAUDA_LOG_TOPIC", "partition": , "offset": }], "version": 1} + EOF + kubectl -n cpaas-system exec cpaas-kafka-0 -- \ + kafka-delete-records.sh --bootstrap-server \ + --command-config --offset-json-file /tmp/delete-records.json + ``` + + Use the topic and partition that the describe output named. + +3. Run the describe again and confirm that `CURRENT-OFFSET` moves. + +## Diagnostic Steps + +1. Read per-partition lag from the dashboard, or with the group describe in Cause 2. + +2. Check the consumer: + + ```bash + kubectl -n cpaas-system logs deploy/lanaya --tail=300 | grep -Ei '413|Failed to insert|abort' + ``` + +3. Match the signature: + + | What you see | Cause | + |---|---| + | Lag on one partition, and `lanaya` repeats `413 Request Entity Too Large` for it | Cause 1 | + | Lag on one partition whose `CURRENT-OFFSET` stays at one fixed offset, without the `413` loop | Cause 2 | + | Lag on many partitions, or two Elasticsearch nodes near 100% CPU | See the related article | + +## Related Information + +- Related articles: [Log Kafka lag from Elasticsearch write hot spots, breaker trips, IK analyzer errors, or too few partitions on ACP](./Log_Kafka_lag_from_Elasticsearch_write_hot_spots_breaker_trips_IK_analyzer_errors_or_too_few_partitions_on_ACP.md) covers lag spread over many partitions and lag caused by the Elasticsearch side. +- [Logs, audit and events stop while the log Kafka is degraded on ACP](./Logs_audit_and_events_stop_while_the_log_Kafka_is_degraded_on_ACP.md) diff --git a/docs/en/solutions/Log_archive_output_platform_logs_with_workload_logs_flat_S3_objects_file_retention_or_undeletable_.nfs_files_on_ACP.md b/docs/en/solutions/Log_archive_output_platform_logs_with_workload_logs_flat_S3_objects_file_retention_or_undeletable_.nfs_files_on_ACP.md new file mode 100644 index 000000000..78d20c683 --- /dev/null +++ b/docs/en/solutions/Log_archive_output_platform_logs_with_workload_logs_flat_S3_objects_file_retention_or_undeletable_.nfs_files_on_ACP.md @@ -0,0 +1,157 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Log archive output: platform logs with workload logs, flat S3 objects, file retention, or undeletable .nfs files on ACP + +## Issue + +The log archive (log sync to external NFS or S3) is running, and its output is not what the owner expected: + +- Only application (workload) logs are enabled, but system component logs from `cpaas-system` and `kube-system` are archived as well. +- After switching the archive from NFS to S3, the bucket has no directory hierarchy; only the log files appear. +- Business logs must be kept for 3 years, but the archive holds only about half a year; the oldest daily directory is about 180 days old. +- `log-exporter-config` says `path: /cpaas/data/logarchive`, but `ls /cpaas/data/logarchive` on the host says `No such file or directory`. +- Does the hostPath archive directory grow without limit? +- Are audit logs archived? +- Deleting old archived logs fails, and files whose names start with `.nfs` appear in the archive directory; their content looks like the consumed Kafka records. + +Deploying the archive (the `/cpaas/data/logarchive` mount, the Kafka address, real-time copy versus move after N days) is in [Archive Elasticsearch logs to NFS: logarchive mount, log center Kafka address, and archive limits on ACP](Archive_Elasticsearch_logs_to_NFS_logarchive_mount_log_center_Kafka_address_and_archive_limits_on_ACP.md). + +The documented archive ("How to archive logs to third-party storage") consists of three objects in `cpaas-system`: + +- Secret `log-export-s3-secret` with `access_key_id` and `secret_access_key` (S3 output only). +- ConfigMap `log-exporter-config` with the file `export.yml`: `scopes` (`system`, `workload`, `kubernetes`, `platform`, each `true` or `false`) and `output` (`type` `local` or `s3`, `path: /cpaas/data/logarchive`, `layout: TimePrefixed`, `max_size`, `compression` such as `gzip`, `file_type: txt`, and for S3 the endpoint, bucket, `s3forcepathstyle`, and `insecure`). +- Deployment `log-exporter` (also seen as `log-export`): an alpine init container that runs `chown`, and a `razor` container that runs `/razor consumer ... --database-type=file --export-config=/etc/log-export/export.yml` against the log center Kafka. + +## Root Cause + +### Cause 1: on Elasticsearch, the Workload scope also exports platform logs, and the Platform scope has no effect + +The archiver selects each log item by its index prefix: `log-system` follows `scopes.system`, `log-kubernetes` follows `scopes.kubernetes`, `log-platform` follows `scopes.platform`, and every other value falls to the default branch, which follows `scopes.workload`. With Elasticsearch log storage, the index prefix is empty for both application (workload) and product (platform) logs, so both fall to the default branch: with Workload on, platform logs are archived too; with Workload off, neither is; the Platform switch changes nothing. System and Kubernetes logs are not affected. No setting separates them. + +### Cause 2: S3 objects have no directory hierarchy + +This is expected. The date-time / project / cluster directories exist only in the temporary local path; the uploaded objects are the log files themselves. + +### Cause 3: archive files older than 180 days are deleted by a cron job on the host, not by the platform + +The platform does not limit how long archive files are kept. A scheduled task on the host that mounts the archive volume ran a cleanup script that took a target directory (`TARGET_DIR="${1:-.}"`) and a day threshold (`DAYS_THRESHOLD="${2:-180}"`), listed subdirectories with `find "$TARGET_DIR" -maxdepth 1 -type d`, compared their modification time (`stat -c %Y`), and ran `rm -rf` on those older than the threshold. With the default of 180, about half a year of daily directories remained. + +### Cause 4: the output path in log-exporter-config is a path inside the pod + +`output.path: /cpaas/data/logarchive` is where the archive container writes. What backs it is the volume mounted at that path in the Deployment. When that volume is a PVC on an NFS StorageClass, the files are on the NFS share of that PV, not in a host directory of that name. Only in the hostPath layout is the same path also the host directory. + +### Cause 5: the hostPath archive directory is not cleaned by the platform + +Nothing in the archive caps or cleans the hostPath directory; it grows until something removes old files. A scheduled cleanup on the node is the way to bound it. + +### Cause 6: audit logs are not an archive scope + +The archive covers log data only: system, application (workload), Kubernetes, and product (platform) logs. Audit data is not archived. For a 6-month log backup requirement, the archive is the recommended path rather than a 180-day retention in the log Elasticsearch, which uses a large amount of storage and affects Elasticsearch performance. + +### Cause 7: archived files cannot be deleted, and .nfs files appear in the archive directory + +The log archive does not create `.nfs` files itself. They are the NFS client's protection for open files: when a file that a process still holds open is deleted on an NFS mount, the client renames it to `.nfs` instead of removing it, and removes it when the last process closes it. Archived files that cannot be deleted, with `.nfs` files left in their place, come from a known defect in the log archiving solution that keeps files open; it is fixed in versions after 3.18.3. + +## Resolution + +### Cause 1: on Elasticsearch, the Workload scope also exports platform logs, and the Platform scope has no effect + +Plan the archive storage for workload plus platform logs when Workload is on, and do not rely on the Platform switch. Check the live scopes: + +```bash +kubectl -n cpaas-system get configmap log-exporter-config -o jsonpath='{.data.export\.yml}' +``` + +### Cause 2: S3 objects have no directory hierarchy + +No change. Find files by object name and time in the bucket. + +### Cause 3: archive files older than 180 days are deleted by a cron job on the host, not by the platform + +Find the scheduled task on the node that mounts the archive volume and read its threshold: + +```bash +crontab -l +grep -r -l -iE 'rm -rf|find ' /etc/cron.d/ 2>/dev/null +cat +``` + +> **Warning:** The script runs `rm -rf` on archive directories; a wrong directory argument or a threshold in the wrong unit deletes archive data permanently. Precondition: you saved the current crontab, the first argument is the archive directory, and the new threshold is in days (about 1095 for 3 years). Rollback: restore the saved crontab line; deleted directories do not return. + +```bash +crontab -l > /tmp/crontab.before +crontab -e +``` + +Change the day threshold argument of the cleanup line to the number of days that must be kept, and size the share for that period (a 1.7T share was 54% used with about 180 days on it). + +### Cause 4: the output path in log-exporter-config is a path inside the pod + +Read the volume mounted at that path, then where that volume lives: + +```bash +kubectl -n cpaas-system get deploy | grep -E 'log-export' +kubectl -n cpaas-system get deploy -o jsonpath='{range .spec.template.spec.volumes[*]}{.name}{" pvc="}{.persistentVolumeClaim.claimName}{" hostPath="}{.hostPath.path}{"\n"}{end}' +kubectl -n cpaas-system get pvc -o jsonpath='{.spec.volumeName}{"\n"}' +df -h | grep +``` + +For a PVC the files are on the PV's share; for hostPath they are on the node that runs the pod. + +### Cause 5: the hostPath archive directory is not cleaned by the platform + +Add a scheduled cleanup on each node that runs the archive pod. + +> **Warning:** The cleanup deletes archive files permanently. Precondition: `` is the hostPath archive directory (check it with the Cause 4 commands), `` is at least the period that must be kept, and you ran the `find` without `-delete` first. Rollback: remove the crontab line; deleted files do not return. + +```bash +find -type f -mtime + | head +( crontab -l; echo "0 2 * * * find -type f -mtime + -delete" ) | crontab - +``` + +### Cause 6: audit logs are not an archive scope + +Audit records are not archived. Keep the needed audit days in the log storage retention (read the maximum on the plugin form) and size the disk for it. For logs, deploy the archive with the scopes that are required. + +### Cause 7: archived files cannot be deleted, and .nfs files appear in the archive directory + +1. Confirm which process holds the files (on the node that mounts the share): + + ```bash + ls -la /.nfs* | head + lsof +D 2>/dev/null | head + kubectl -n cpaas-system get deploy,pods | grep -E 'log-export' + ``` + +2. Do not delete the `.nfs` files by hand; the NFS client removes each one when the process that holds it closes it. +3. Upgrade the log archiving solution to a version after 3.18.3, which fixes the defect. + +## Diagnostic Steps + +1. Read the archive configuration and workload: + + ```bash + kubectl -n cpaas-system get configmap log-exporter-config -o yaml + kubectl -n cpaas-system get deploy,pods | grep -E 'log-export' + kubectl -n cpaas-system logs deploy/ --tail=100 + ``` + +2. Store type: `kubectl get pods -A | grep -iE 'cpaas-elasticsearch|clickhouse'`. Platform logs in the archive with only Workload on, on Elasticsearch, is Cause 1. +3. Archive directories disappearing at a fixed age that matches a host cron script threshold is Cause 3: + + ```bash + ls | head + ``` + +4. `No such file or directory` on the host for the ConfigMap path is Cause 4. +5. `.nfs` files next to archived files that cannot be deleted, on an NFS-backed archive, is Cause 7. diff --git a/docs/en/solutions/Log_module_and_monitoring_dashboard_unavailable_after_a_kube-apiserver_internal_certificate_expires_on_ACP.md b/docs/en/solutions/Log_module_and_monitoring_dashboard_unavailable_after_a_kube-apiserver_internal_certificate_expires_on_ACP.md new file mode 100644 index 000000000..48d19563f --- /dev/null +++ b/docs/en/solutions/Log_module_and_monitoring_dashboard_unavailable_after_a_kube-apiserver_internal_certificate_expires_on_ACP.md @@ -0,0 +1,178 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.12.x + - 4.0.x + - 4.2.x + - 4.3.x + - 4.4.x +--- + +# Log module and monitoring dashboard unavailable after a kube-apiserver internal certificate expires on ACP + +## Issue + +One of the following happens on a cluster that has been running for a long time, while the nodes can still be Ready: + +- The log module and the monitoring dashboard can no longer be opened. The kube-apiserver process on a control-plane node has been running for more than one year. +- An inspection says the `apiserver-loopback-client` certificate expires in a few days. +- The console shows `服务器内部错误` (server internal error), and platform Pods such as `dex` and `tke-platform` went down and cannot start again. +- After the kube-apiserver, kube-controller-manager, and kube-scheduler certificates were renewed, Pods are no longer scheduled and started automatically. +- A Deployment stays in Progressing while its Pods are Running, and restarting or deleting the Pods does not help. The Kubernetes Certificates Rotator plugin was uninstalled earlier. + +## Root Cause + +Kubernetes control-plane processes load some certificates only at startup. A process that keeps running holds what it loaded, so a certificate can be expired in memory while the files on disk are valid, and a renewed file on disk has no effect until the process restarts. + +### Cause 1: kube-apiserver has been running for more than one year and an internal certificate it holds has expired + +`kube-apiserver` stayed up for more than one year, and an internal certificate held by that process reached `notAfter` while the process kept running. The log module and the monitoring dashboard both fail until the kube-apiserver container is restarted. A restart does not rewrite any file on disk: it regenerates the in-memory loopback certificate (Cause 2) and reloads the files under `/etc/kubernetes/pki/`. If a file on disk is itself past `notAfter`, the new process loads the same expired file, so that file has to be renewed first (Cause 4). + +On ACP, kube-apiserver runs as a static Pod. etcd is a different static Pod (`etcd-`, label `component=etcd`, in `kube-system`), and restarting kube-apiserver does not touch it. + +### Cause 2: the apiserver-loopback-client certificate is about to expire or has expired + +`apiserver-loopback-client` is a self-signed certificate that kube-apiserver generates in memory at startup, valid for one year, and uses only to call itself. It has no file on disk and cannot be rotated externally, so it expires one year after the process started. Restarting kube-apiserver is the only fix, because the new process generates a new pair. + +When the loopback certificate has already expired, the effect can be a full platform outage: the console shows an internal error, and platform Pods such as `dex` and `tke-platform` cannot start again. Restarting kube-apiserver, and then the affected Pods, restores the platform. + +### Cause 3: control-plane certificates were renewed on disk but the components were not restarted + +The certificates of kube-apiserver, kube-controller-manager, and kube-scheduler were renewed on disk, but the running components still hold the old certificates because they were not restarted, and Pods stop being scheduled. Restarting the static Pods makes them load the new files. + +### Cause 4: the Certificates Rotator plugin was uninstalled and the Kubernetes component certificates expired + +The Kubernetes Certificates Rotator plugin issues short-lived certificates (365 days for control-plane certificates). When the plugin is uninstalled, the certificates it already wrote stay short-lived and are no longer rotated, so they expire. Workloads then show symptoms such as a Deployment stuck in Progressing. Regenerating the Kubernetes component certificates returns the cluster to normal. + +## Resolution + +### Cause 1: kube-apiserver has been running for more than one year and an internal certificate it holds has expired + +1. On the control-plane node, print when kube-apiserver started, the certificate the running process serves on the loopback name, and the certificate files on disk: + + ```bash + ps -eo lstart,cmd | grep kube-apiserver | grep -v grep + openssl s_client -connect 127.0.0.1:6443 -servername apiserver-loopback-client /dev/null | openssl x509 -noout -subject -dates + openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates -subject -issuer + openssl x509 -in /etc/kubernetes/pki/apiserver-kubelet-client.crt -noout -dates -subject -issuer + openssl x509 -in /etc/kubernetes/pki/apiserver-etcd-client.crt -noout -dates -subject -issuer + openssl x509 -in /etc/kubernetes/pki/front-proxy-client.crt -noout -dates -subject -issuer + ``` + + Continue when the process started more than one year ago, the certificate it holds is past `notAfter`, and the files on disk are in date. If a file on disk is past `notAfter`, renew it first with Cause 4. + +2. Restart the kube-apiserver container by stopping it; kubelet starts a new one from the static Pod manifest. + + > **Warning:** Restarting kube-apiserver on this node drops the API requests this instance is handling; on a single control-plane cluster the API is unavailable until the new container is Ready. Use a maintenance window and restart one control-plane node at a time. Do this only after step 1 confirmed the signature. Certificate files are not modified, and there is no rollback to the previous process. If the new container does not become Ready, read the kubelet logs on that node and confirm that `/etc/kubernetes/manifests/kube-apiserver.yaml` is in place. + + ```bash + crictl ps --name kube-apiserver + crictl stop + crictl ps --name kube-apiserver + ``` + + `kubectl delete pod -n kube-system kube-apiserver-` does not restart it: that object is only the mirror of the static Pod, and kubelet recreates the mirror while the container keeps running. The same applies to the other static Pods. + +3. When the CREATED time of the new container is current, open the log module and the monitoring dashboard again. + +### Cause 2: the apiserver-loopback-client certificate is about to expire or has expired + +1. Restart kube-apiserver on each control-plane node in turn, during an off-peak window, by moving its static Pod manifest out of the manifests directory and back. Use a directory outside `/etc/kubernetes/manifests` for the temporary copy, because kubelet reads every manifest file in that directory. Wait until each node's kube-apiserver is Running before moving to the next node. + + > **Warning:** While the manifest is out, kube-apiserver on this node is stopped and requests to it fail; on a single control-plane cluster the whole API is down until the Pod is back. Do this one node at a time. Rollback is to move the manifest back, which is also the second half of the procedure; never leave it in the backup directory. + + ```bash + mkdir -p /etc/kubernetes/manifests-backup + mv /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/manifests-backup/ + # wait until the container is gone + crictl ps | grep kube-apiserver + mv /etc/kubernetes/manifests-backup/kube-apiserver.yaml /etc/kubernetes/manifests/ + crictl ps | grep kube-apiserver + kubectl get pod kube-apiserver- -n kube-system + ``` + + Moving the file back before kubelet has stopped the old container can leave the old process running. The `crictl stop` method of Cause 1 is equivalent. + +2. If platform Pods such as `dex` or `tke-platform` stayed down while the loopback certificate was expired, restart them after every kube-apiserver has been restarted. + + > **Warning:** Deleting these Pods interrupts login (`dex`) and the platform API (`tke-platform`) until the replacements are Running. Do this only after the kube-apiserver restarts, and only for Pods that are not Running. Their workloads recreate them. + + ```bash + kubectl get pods -A | grep -v -E 'Running|Completed' + kubectl delete pod -n + ``` + +### Cause 3: control-plane certificates were renewed on disk but the components were not restarted + +1. Confirm that the files on disk are newer than the running processes: + + ```bash + ls -l --time-style=full-iso /etc/kubernetes/pki/*.crt + ps -eo lstart,cmd | grep -E 'kube-apiserver|kube-controller-manager|kube-scheduler' | grep -v grep + ``` + +2. Restart the three components one node at a time with the manifest move of Cause 2 (its Warning applies). kube-controller-manager and kube-scheduler are leader-elected, so another control-plane node takes over while one is stopped. + + ```bash + mkdir -p /etc/kubernetes/manifests-backup + for c in kube-apiserver kube-controller-manager kube-scheduler; do + mv /etc/kubernetes/manifests/$c.yaml /etc/kubernetes/manifests-backup/ + sleep 20 + mv /etc/kubernetes/manifests-backup/$c.yaml /etc/kubernetes/manifests/ + sleep 30 + crictl ps | grep $c + done + ``` + +3. Check that new Pods are scheduled. + +### Cause 4: the Certificates Rotator plugin was uninstalled and the Kubernetes component certificates expired + +1. Print the component certificates on each control-plane node: + + ```bash + kubeadm certs check-expiration 2>/dev/null + for f in /etc/kubernetes/pki/*.crt; do echo "$f"; openssl x509 -in "$f" -noout -enddate; done + ``` + +2. Regenerate the expired certificates with the `cert-renew` tooling, following "Recovering ACP Kubernetes Certificates with Kubernetes Certificates Rotator". That procedure backs up `/etc/kubernetes`, re-signs the files with the cluster CA on every node, and restarts the components. + + On a control plane that has the `kubeadm` binary, `kubeadm certs renew all` is the standard kubeadm alternative. + + > **Warning:** Renewing control-plane certificates rewrites the files under `/etc/kubernetes/pki` and the kubeconfig files kubeadm manages on this node, and the components must be restarted afterwards (Cause 3). Do this only when step 1 shows files past `notAfter`. Back up `/etc/kubernetes` first; rollback is to restore that copy and restart the static Pods. Repeat on each control-plane node. + + ```bash + cp -a /etc/kubernetes /etc/kubernetes.bak-$(date +%Y%m%d) + kubeadm certs renew all + ``` + + `kubeadm certs renew all` issues the kubeadm default one-year lifetime and also rewrites the client certificate in `/etc/kubernetes/admin.conf`, which is then no longer a long-lived credential. Print its `notAfter` before relying on it. + +3. Restart the control-plane components with Cause 3 and confirm that the Deployment leaves Progressing. + +## Diagnostic Steps + +1. Separate a long-running kube-apiserver from expired files and from an expired platform credential: + + ```bash + ps -eo lstart,cmd | grep kube-apiserver | grep -v grep + openssl x509 -in /etc/kubernetes/pki/apiserver-kubelet-client.crt -noout -enddate + openssl s_client -connect 127.0.0.1:6443 -servername apiserver-loopback-client /dev/null | openssl x509 -noout -enddate + ``` + +2. Branch: + + - Log module and monitoring dashboard unavailable, kube-apiserver up for more than one year, the held certificate past `notAfter`, files on disk in date: Cause 1. + - Inspection names `apiserver-loopback-client`, or the console shows an internal error and `dex` or `tke-platform` cannot start while the loopback certificate is expired: Cause 2. + - Certificate files on disk are newer than the process start, and Pods stopped scheduling: Cause 3. + - A client outside the node, such as a pipeline `kubectl`, fails with `x509: certificate has expired or is not yet valid` against `https://
:6443`, and the log module and monitoring dashboard are not the symptom: first identify the certificate that address serves and compare it with `apiserver.crt` on each control-plane node. A load balancer or VIP in front of port 6443 can present its own certificate. + + ```bash + echo | openssl s_client -connect
:6443 2>/dev/null | openssl x509 -noout -subject -issuer -dates + openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -subject -issuer -dates + ``` + - Deployment stuck in Progressing, Certificates Rotator uninstalled, component certificate files expired: Cause 4. + - The console returns HTTP 500 and the platform `global-kubeconfig` Secret is expired: not this article. See "Console returns HTTP 500 or clusters show abnormal because the global-kubeconfig or cluster kubeconfig Secret expired on ACP". diff --git a/docs/en/solutions/Log_query_shows_no_logs_HTTP_500_slow_long_ranges_or_an_empty_real-time_log_on_ACP.md b/docs/en/solutions/Log_query_shows_no_logs_HTTP_500_slow_long_ranges_or_an_empty_real-time_log_on_ACP.md new file mode 100644 index 000000000..c1bbb943f --- /dev/null +++ b/docs/en/solutions/Log_query_shows_no_logs_HTTP_500_slow_long_ranges_or_an_empty_real-time_log_on_ACP.md @@ -0,0 +1,192 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Log query shows no logs, HTTP 500, slow long ranges, or an empty real-time log on ACP + +## Issue + +The log store holds the data, but the console does not show it. Typical reports: + +- "No logs in Operations Center, although `razor`, ClickHouse and `nevermore` are normal and their logs show no errors." +- "ClickHouse runs normally, and the ClickHouse database contains the data." +- "The real-time log is empty, and the raw logs are garbled." +- "A keyword is found over the last 6 hours but shows no data over the last day." +- "A filter that matches 59 lines still takes two minutes." +- "Cluster logs are empty after we moved to ClickHouse, and the browser shows HTTP 500." + +The failure is on the query or display path between the console and the storage, not in collection. Open the browser developer tools on the failing page first. The status and body of the log query request decide which cause applies. + +Keyword, time-window and page-scope mistakes are covered in [Log search misses entries: IK analyzer tokens, uppercase keywords, clock skew, or page scope on ACP](./Log_search_misses_entries_IK_analyzer_tokens_uppercase_keywords_clock_skew_or_page_scope_on_ACP.md). + +## Root Cause + +### Cause 1: Operations Center shows no logs while the whole pipeline is healthy, because the clock of the computer running the browser is wrong + +After log center, ClickHouse and the log collector were installed (ACP 4.3.3), Operations Center shows no logs, even with product logs and Kubernetes logs both selected in the collector. `nevermore` uploads normally, and the `vector`, ClickHouse, `courier-api` and `alb` logs show no errors. A manual query in ClickHouse finds current rows. Synchronizing the node clocks and reinstalling the components with their data cleared does not help. + +The system time of the computer that opens the console is wrong, and that alone is enough to leave Operations Center empty. The log pipeline itself works. + +### Cause 2: The real-time log view stays empty because a reverse proxy in front of the platform does not forward WebSocket + +The pod's real-time log in the console is empty or never loads, and raw logs may look garbled. The backend pod returns the data without error. Two breaks can be on this path: + +- A node clock is out of sync, which makes log API authentication fail, and the platform gateway blocks the WebSocket requests. Calibrating the time fixes this part. +- An Nginx reverse proxy was added in front of the platform VIP without WebSocket forwarding settings, so the `wss` handshake never passes. Going to the platform VIP directly restores the live log. + +### Cause 3: A query over a long time range returns no data or takes minutes, because Elasticsearch or the storage is overloaded + +A keyword finds matches over the last 6 hours but shows no data over the last day. The global log storage had 3 log nodes with a 1 core / 2Gi limit, and one project kept logs for 30 days. A 3-day keyword query on one namespace matched about 14 million entries. The page also states that only the first 10,000 matched entries can be viewed and asks for a narrower filter. When the range holds a large log volume, filtering for one entry overwhelms Elasticsearch and the query fails. + +On ClickHouse log storage, a filtered query that matched only about 59 lines took almost 2 minutes, and the query memory chart spiked to 6 to 7 GiB again and again. The storage was NAS. The log component capacity planning documentation gives the benchmark disk as 6000 IOPS and 250 MB/s read/write on a dedicated SSD mount. Storage below that needs a larger-scale configuration with more CPU and memory. An `fio` test against those numbers showed that the disk was too slow. + +When every Elasticsearch query is slow and the Elasticsearch logs report errors, the component lacks the capacity for the query load. Raising the Elasticsearch memory limit triggers a rolling update, after which index queries return to normal and the errors stop. + +### Cause 4: Queries return HTTP 500 and global apollo logs cache error after the store was switched to ClickHouse + +Elasticsearch was replaced by ClickHouse on a workload cluster. ClickHouse holds the log rows, but the console shows `共 0 条日志` ("0 logs in total"). In the browser developer tools, requests to `logs?action=aggregation&type=workload...` fail with: + +```text +code: 500, message: "unknown action: aggregation", reason: "Unknown", status: "Failure" +``` + +`apollo` on the global cluster logs `cache error`. Global `apollo` reaches the workload cluster's ClickHouse through that cluster's load balancer `/clickhouse` endpoint. A known issue keeps `apollo` on a stale cache of the cluster's data-access information after the switch. Restarting `apollo` on the global cluster makes the logs queryable. + +The ClickHouse replicas may also log `RaftInstance: session ... failed to read rpc header from socket ... End of file` repeatedly, and Vector may log ClickHouse health-check failures. Neither causes the 500. + +### Cause 5: The global cluster cannot reach the business cluster apiserver on 6443 + +The business cluster's `nevermore` is Ready, `razor` logs authentication failures, and the global `cluster-transformer` cannot reach the business cluster's API VIP on port 6443. Queries for that cluster return nothing. Opening that path from all global nodes, not only the first control-plane node, restores queries. + +Port 11781 is another port that centralized logging uses between the global and business clusters. A closed 11781 can appear beside the 6443 break without being its cause. Do not disable 11781 on a centralized-logging design that still needs it. Port 11780 on the log cluster's load balancer is the log sink port from platform 4.0 onward, so log shipping connections land on it. A connection count on 11780 that rises from almost 0 to several thousand after an upgrade to 4.0 is expected. + +## Resolution + +### Cause 1: Operations Center shows no logs while the whole pipeline is healthy, because the clock of the computer running the browser is wrong + +1. Compare the clock of the computer that runs the browser with a cluster node. On a node: + + ```bash + date -u + ``` + + On a Windows client, `w32tm /stripchart /computer: /samples:3` shows the offset from a time server. A client clock that is off while the nodes agree matches this cause. + +2. Correct the client's date, time and time zone, or enable automatic time synchronization on it (on Windows, `w32tm /resync` after a time source is configured). + +3. Reload the console and query again. Opening the console from another computer with a correct clock is a quick confirmation. Check this before you reinstall any log component. + +### Cause 2: The real-time log view stays empty because a reverse proxy in front of the platform does not forward WebSocket + +1. Check the node clocks first with `date -u` on each node, and fix any skew. + +2. Open the browser developer tools on the pod log page and look at the `wss` request. A handshake that fails only through the proxy, and works against the platform VIP directly, matches this cause. + +3. Add WebSocket forwarding to the Nginx `location` that proxies the platform, for example: + + ```nginx + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + proxy_read_timeout 3600s; + ``` + + Reload Nginx and retry. Until the proxy is fixed, use the platform VIP directly. + +### Cause 3: A query over a long time range returns no data or takes minutes, because Elasticsearch or the storage is overloaded + +1. Narrow the time range, and add exact conditions (cluster, namespace, pod) before the keyword. + +2. If even small result sets take minutes, measure the log storage disk on the log storage node, against the filesystem of the data directory: + + ```bash + fio --name=randwrite --directory= --rw=randwrite --bs=4k --size=1G \ + --ioengine=libaio --direct=1 --iodepth=32 --runtime=60 --time_based --group_reporting + ``` + + Remove the test file afterwards. Compare the result with the documented benchmark of 6000 IOPS and 250 MB/s on a dedicated SSD. A disk below the requirement has to be replaced, or the log storage sized up with more CPU and memory. Storing each cluster's logs in that cluster also spreads the query load. + +3. When every Elasticsearch query is slow and the Elasticsearch log shows errors, compare its memory with the limit: + + ```bash + kubectl -n cpaas-system top pod | grep cpaas-elasticsearch + kubectl -n cpaas-system logs cpaas-elasticsearch-0 -c cpaas-elasticsearch --tail=100 | grep -E 'WARN|ERROR' + ``` + + > **Warning:** Raising the Elasticsearch memory limit rolls every Elasticsearch pod, and search and ingestion pause on each pod while it restarts. Precondition: only when queries are slow across the board, Elasticsearch memory is near its limit, and the nodes can give the new limit. Rollback: set the previous limit back on the log storage plugin. + + Change the memory limit on the log storage plugin, and let the rolling update finish before you test again. + +### Cause 4: Queries return HTTP 500 and global apollo logs cache error after the store was switched to ClickHouse + +1. On the global cluster, check `apollo`: + + ```bash + kubectl -n cpaas-system get pods | grep apollo + kubectl -n cpaas-system logs deploy/apollo --tail=100 | grep -i 'cache error' + ``` + +2. Restart `apollo`. If `get pods` shows a different owner kind, use that kind in the restart. + + > **Warning:** Restarting `apollo` on the global cluster briefly interrupts the console API calls it serves. Precondition: only when ClickHouse on the workload cluster holds current rows, the browser shows HTTP 500 with `unknown action: aggregation` on log queries, and global `apollo` logs `cache error`. Rollback: none needed; the rollout does not change the spec. + + ```bash + kubectl -n cpaas-system rollout restart deploy/apollo + kubectl -n cpaas-system rollout status deploy/apollo + ``` + +### Cause 5: The global cluster cannot reach the business cluster apiserver on 6443 + +1. From each global node, test the business cluster API VIP (the address in that cluster's kubeconfig): + + ```bash + nc -vz 6443 + nc -vz 11781 + ``` + +2. Restore the 6443 path (security group, route or VIP listener). Queries should return without a pod restart. + +3. Leave 11781 enabled when centralized logging is in use. Disable the 11781 load-balancer health check only after you have confirmed that nothing listens on it and the design does not need the port. + +## Diagnostic Steps + +1. Open the browser developer tools on the failing page and read the log query request: + + | What you see | Cause | + |---|---| + | No HTTP error, current rows in ClickHouse or Elasticsearch, every component healthy, and still no logs | Cause 1, compare the browser computer's clock with a node | + | A `wss` request that fails while `kubectl logs` works | Cause 2 | + | A narrow range works and a wide one fails, or small results take minutes | Cause 3 | + | HTTP 500 with `unknown action: aggregation` | Cause 4, read global `apollo` for `cache error` | + | Queries empty for one business cluster only, with `razor` authentication failures there | Cause 5, test 6443 from the global nodes | + +2. To confirm that the store holds the rows, query ClickHouse directly from a replica pod: + + ```bash + kubectl -n cpaas-system exec -it chi-cpaas-clickhouse-replicated-0-0-0 -c clickhouse -- clickhouse-client + ``` + + ```sql + SELECT timestamp, log_data FROM observability.log_workload_all + WHERE match(namespace, '') AND match(workload, '') + AND timestamp > now() - INTERVAL 1 HOUR + ORDER BY timestamp DESC LIMIT 20; + ``` + + Rows here with nothing on the page point to the display path (Causes 1, 2 and 4). + +3. A 401 or 403 on the query request is an authentication problem and not covered here. + +## Related Information + +- [Log search misses entries: IK analyzer tokens, uppercase keywords, clock skew, or page scope on ACP](./Log_search_misses_entries_IK_analyzer_tokens_uppercase_keywords_clock_skew_or_page_scope_on_ACP.md) +- [Logging ClickHouse runs out of disk, CPU or memory, or is OOMKilled on ACP](./Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md) diff --git a/docs/en/solutions/Log_search_misses_entries_IK_analyzer_tokens_uppercase_keywords_clock_skew_or_page_scope_on_ACP.md b/docs/en/solutions/Log_search_misses_entries_IK_analyzer_tokens_uppercase_keywords_clock_skew_or_page_scope_on_ACP.md new file mode 100644 index 000000000..d80927db1 --- /dev/null +++ b/docs/en/solutions/Log_search_misses_entries_IK_analyzer_tokens_uppercase_keywords_clock_skew_or_page_scope_on_ACP.md @@ -0,0 +1,177 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.16.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x + - 4.4.x +--- + +# Log search misses entries: IK analyzer tokens, uppercase keywords, clock skew, or page scope on ACP + +## Issue + +Elasticsearch or ClickHouse log storage is up, the indices or tables exist, and the console still shows nothing, the wrong time window, or lines that do not match the keyword. Typical reports: + +- "A Chinese sentence I copied from the log cannot be found." +- "Searching `ERROR` in uppercase shows unrelated logs, but lowercase works." +- "The project filter also shows another project." +- "Logs from `cpaas-system` cannot be seen in the console." +- "Only the last 30 minutes are shown." + +Use this article when a keyword that you can see in `kubectl logs` does not match, when two nodes disagree on the date, or when the page shows a different scope from the line you are looking for. If logs cannot be viewed at all, first check the per-node agent: an `OOMKilled` or restarting `nevermore` is a collection problem, not a query problem. If the Kafka consumer lag is hours, fix the lag first; a deployment name that appears only after you widen the time range is a sign of lag, not of a query bug. + +## Root Cause + +### Cause 1: On Elasticsearch storage, the IK analyzer lowercases text and splits long Chinese strings + +With Elasticsearch and the IK analyzer, the console search is a token query, not a `grep` of the raw line. + +- **Uppercase does not match.** The tokens are lowercase, so searching the lowercase form returns the line. Wrapping the original text in quotes does not turn it into an exact-string query. For example, logs that contain `CN=cluster-operator,O=io.strimzi` are found by `cn` and not by `CN` (`共 0 条日志`, "0 logs in total"). +- **A Chinese compound fails when the dictionary did not emit that compound.** `资产` ("asset") and `管理` ("management") match, but `资产管理` ("asset management") does not. Two-character pieces match more often than a long phrase, so split the phrase (for example search `没有` ("not") instead of `没有执行` ("not executed")). +- **Chinese quotation marks, URLs and `/` are split as well**, so the original string misses. An unquoted search that returns nothing and a quoted search that returns inaccurate rows are the same keyword-matching limitation. +- **A long alphanumeric ID can lose its digits at query time, depending on the IK build.** For example, `RF20260928140102047100190` matched only `RF` on one 4.2 environment and matched exactly on another. With one IK build (8.19.10), the query-time `ik_smart` analyzer drops the numeric part, so the query becomes `rf`. Another build (8.19.14) keeps the whole string. The index-time `ik_max_word` analyzer keeps the full string and its fragments on both. On the affected build, quoted searches returned hundreds of thousands of entries that merely share the `RF` prefix. The unquoted lowercase ID returned only exact matches, and the unquoted uppercase ID returned nothing. +- **A Chinese sentence copied from a displayed line returns nothing, while showing all logs displays it.** Log queries follow the Elasticsearch tokenization, so some Chinese phrases can be found and others cannot. This behavior cannot be changed while the log storage is Elasticsearch. ClickHouse log storage supports full-text search and does not use IK. The log content field accepts `AND` or `OR` combinations, for example `"the connection is unavailable" AND "rpc error"`, but not both operators in one query. + +No index setting is known that disables IK in place. + +### Cause 2: Node clocks disagree, so the selected range is empty + +`nevermore` stamps records with one node's clock, and the store is queried with another. When those clocks are far apart (for example months), searching either window returns nothing, because the pipeline's "now" and the application's "now" are not the same range. + +On an Elasticsearch cluster, the same skew creates indices whose names and `@timestamp` values do not line up. The cluster stays `yellow`. Deleting the data does not help, because the next index is wrong again until the clocks match. An existing yellow index cannot be repaired into a correct time range. + +The clock that matters can also be the client's. When node clocks agree and the pipeline is healthy, a wrong system time on the computer that runs the browser leaves Operations Center empty. See [Log query shows no logs, HTTP 500, slow long ranges, or an empty real-time log on ACP](./Log_query_shows_no_logs_HTTP_500_slow_long_ranges_or_an_empty_real-time_log_on_ACP.md). + +### Cause 3: The page has a different scope from the line you are searching + +Several "no data" reports are about the console scope, not a broken indexer: + +- **Product logs.** Namespace `cpaas-system` holds platform product logs. Observability, Logs lists application logs only. With product-log collection enabled, those lines are under Platform Management, Operations Center, Logs. +- **Default time window.** Observability, Logs opens on the last 30 minutes, which hides every earlier line. Change the time range in the upper-right corner. The lines were not removed by retention. +- **Pod filter.** A pod chosen in the query filter applies only after the confirm button in that filter is clicked. Without it, results come from other pods. +- **Wrong project.** A project named `default` reached through a non-standard entry path may not exist, so a namespace's logs look empty there. Switch to the project that owns the namespace and select the namespace there. +- **One cluster at a time.** An administrator query covers one cluster at a time. There is no cross-cluster aggregate. +- **Export limit.** Export stops at one million rows. Narrow the time range and the workload, export in batches, and merge the files outside the platform. To export one project's logs from the built-in Elasticsearch, filter Operations Center, Logs by that project and export the result. +- **Missing component on 4.0.** A data-service component such as RabbitMQ is missing from the component dropdown on 4.0 when its pod has no Project label, because the query joins pod, namespace and project by that label. This is fixed in 4.1. On 4.0, add the Project label that the other visible components carry. +- **ALB history.** An ALB pod in `cpaas-system` shows only its real-time log under the cluster's pods. Its history is under Platform Management, Operations Center, Logs, with Log Type `Product Logs`, a wider time range, and a filter for the ALB pod. The page shows only the first 10,000 matched logs. +- **ALB access logs.** Inbound ALB access logs are product logs. Set the log type to Product Logs, filter by the load balancer's component name as the component dropdown lists it (for example `alb2-cpaas-system`), and enter `client_addr` in Log Content. Each match is a JSON access record with fields such as `client_addr`, `host`, `request`, `status` and `upstream_addr`. Without product-log collection, read the recent records from the `nginx` container of the ALB pod (`kubectl logs -c nginx`). +- **GBK bytes.** The raw-log view shows GBK bytes, while the live tail, which is decoded as UTF-8, looks fine. Forcing the raw response to UTF-8 fixes the display. The file on disk is not corrupt. +- **NUL bytes.** A line full of `0x00` bytes renders as empty or as `NUL`, depending on the viewer. The zeros come from the application's write or its log rotation, not from the store. +- **Garbled live view before 4.0.** Chinese characters that are intermittently garbled only in the pod's real-time log view come from one UTF-8 character being split across two transmissions. This is fixed in platform 4.0. + +### Cause 4: On ClickHouse storage, an uppercase keyword returns unrelated lines, or a project filter also returns a project with a longer name + +Two behaviors of a ClickHouse-backed query page: + +- Searching the log content for an uppercase keyword such as `ERROR` returns lines that do not contain it (`[WARN ]` lines, retry messages, stack-trace lines), while the same keyword in lowercase works. A direct ClickHouse query with `log_data ILIKE '%ERROR%'` for the same pod and time range returns only correct lines, so the stored data is right. This is a product bug in the log query parsing. It affects v4.3.1 to v4.3.5 and v4.4.0 to v4.4.1, and it is fixed in v4.4.2 and later and in v4.5. +- Filtering by project `` also returns logs of a project named `-`. The project keyword is matched fuzzily by design. + +## Resolution + +### Cause 1: On Elasticsearch storage, the IK analyzer lowercases text and splits long Chinese strings + +1. Search with the lowercase English token, or with the short Chinese token that IK emits. Quotes around the original uppercase or compound string do not change the analyzer. For a long alphanumeric ID on an IK build that drops the digits, use the unquoted lowercase ID. + +2. To see which tokens a build produces, run `_analyze` with `ik_smart` against a log index: + + ```bash + ES_USER=$(kubectl -n cpaas-system get secret elasticsearch-basic-auth -o jsonpath='{.data.username}' | base64 -d) + ES_PASS=$(kubectl -n cpaas-system get secret elasticsearch-basic-auth -o jsonpath='{.data.password}' | base64 -d) + kubectl -n cpaas-system exec cpaas-elasticsearch-0 -c cpaas-elasticsearch -- \ + curl -sS -u "${ES_USER}:${ES_PASS}" -H 'Content-Type: application/json' \ + 'http://cpaas-elasticsearch:9200//_analyze?pretty' \ + -d '{"analyzer": "ik_smart", "text": ""}' + ``` + +3. If the requirement is an exact raw substring, move that cluster's log storage to ClickHouse. Confirm which store the cluster has before you treat this as a bug: + + ```bash + kubectl -n cpaas-system get pods | grep -Ei 'elastic|clickhouse' + ``` + +### Cause 2: Node clocks disagree, so the selected range is empty + +1. On each node that runs `nevermore` or the log store, compare the clocks: + + ```bash + date -u + ``` + + A gap of minutes is enough to push a short window off the record. A gap of months means the log center cannot be the source of truth until NTP is fixed. + +2. Fix time synchronization. + + > **Warning:** Stepping a node clock affects certificates, leases and audit order, not only log search. Precondition: the skew is confirmed with `date -u` on the `nevermore` node and on the ClickHouse or Elasticsearch node, and you have a maintenance window for chrony or the hypervisor clock. Rollback: restore the previous NTP configuration. Do not set the clock of a single control-plane node by hand. + +3. If the business requires clocks that differ, use the workload's live log (`kubectl logs`), and do not expect the log center to reconcile the ranges. Yellow Elasticsearch indices created by the skew stay yellow, and deleting them without fixing NTP recreates them. + +### Cause 3: The page has a different scope from the line you are searching + +Use the entry and filter that match the line, as listed in Root Cause: + +- For `cpaas-system` product logs, use Platform Management, Operations Center, Logs. +- Widen the upper-right time range before you investigate Kafka or the store. +- Click the confirm button after you choose a pod. +- Export in windows that stay under one million rows. +- For GBK, set the raw-log request charset to UTF-8. +- For NUL bytes, count them in the file inside the container: + + ```bash + kubectl exec -n -c -- \ + python3 -c 'import pathlib; b=pathlib.Path("").read_bytes(); print(b.count(b"\x00"))' + ``` + + A large count means the application or its log rotation wrote the zeros. Fix that writer, because reindexing stores the same bytes. + +### Cause 4: On ClickHouse storage, an uppercase keyword returns unrelated lines, or a project filter also returns a project with a longer name + +1. Confirm that the stored rows are correct by querying ClickHouse directly from a replica pod: + + ```bash + kubectl -n cpaas-system exec -it chi-cpaas-clickhouse-replicated-0-0-0 -c clickhouse -- clickhouse-client + ``` + + ```sql + SELECT timestamp, log_data + FROM observability.log_workload_all + WHERE match(namespace, '') AND match(workload, '') + AND timestamp > now() - INTERVAL 1 HOUR + AND log_data ILIKE '%ERROR%' + ORDER BY timestamp DESC + LIMIT 20; + ``` + + If the database name differs on this version, list it with `SHOW DATABASES`. + +2. Correct rows here, and unrelated rows on the page for the same uppercase keyword, match the parsing bug. Until the platform is upgraded to v4.4.2 or later (or v4.5), search with the lowercase form. + +3. To trace one page request, open the browser console (F12) and run the search. Copy the `request-id` from the response headers of the search request, and look it up in the `razor-api` logs on the global cluster: + + ```bash + kubectl -n cpaas-system get pods | grep razor-api + kubectl -n cpaas-system logs | grep '' + ``` + +4. For the project filter, nothing needs fixing. When two project names share a prefix, narrow the result with the namespace or workload filter. + +## Diagnostic Steps + +1. Decide whether the record exists. A green Elasticsearch `_cat/indices`, or a ClickHouse table with today's rows, means the indexer ran, and this article applies. An empty index together with Kafka lag means a collection or consumption problem. + +2. Repeat the query with the lowercase or short token (Cause 1) and with a wide time range. A page left on the default last 30 minutes is Cause 3, not a missing index. If the wide range works and the node clocks differ, it is Cause 2. + +3. On ClickHouse storage, an uppercase keyword that returns unrelated lines while a direct `ILIKE` query returns the right rows is Cause 4. So is a project filter that also returns a project whose name starts with the same text. + +4. A single namespace or component missing while everything else is searchable is Cause 3. For one business cluster that returns nothing, test the 6443 path from the global cluster as described in [Log query shows no logs, HTTP 500, slow long ranges, or an empty real-time log on ACP](./Log_query_shows_no_logs_HTTP_500_slow_long_ranges_or_an_empty_real-time_log_on_ACP.md). + +## Related Information + +- [Log query shows no logs, HTTP 500, slow long ranges, or an empty real-time log on ACP](./Log_query_shows_no_logs_HTTP_500_slow_long_ranges_or_an_empty_real-time_log_on_ACP.md) +- [Log Kafka lag from Elasticsearch write hot spots, breaker trips, IK analyzer errors, or too few partitions on ACP](./Log_Kafka_lag_from_Elasticsearch_write_hot_spots_breaker_trips_IK_analyzer_errors_or_too_few_partitions_on_ACP.md) diff --git a/docs/en/solutions/Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md b/docs/en/solutions/Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md new file mode 100644 index 000000000..b73ca87d3 --- /dev/null +++ b/docs/en/solutions/Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md @@ -0,0 +1,244 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.10.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Logging ClickHouse runs out of disk, CPU or memory, or is OOMKilled on ACP + +## Issue + +The log store is ClickHouse, not Elasticsearch, and one of these happens: + +- A log query returns a memory error, or the console shows no logs. +- A node hits 100% disk, and Kube-OVN, backup and the log pods fail together. +- ClickHouse pods restart with exit code `137`, or with `OOMKilled` in their events. + +Two different ClickHouse installations appear in these problems, and cleaning one does not fix the other: + +- The **logging ClickHouse** holds application, audit and platform logs (pods `chi-cpaas-clickhouse-replicated---0` in `cpaas-system`). +- The **MySQL slow-query ClickHouse** is `chi-pxc-ck-slowsql-cluster` (StatefulSet `chi-pxc-ck-slowsql-cluster-0-0`, pod `chi-pxc-ck-slowsql-cluster-0-0-0`). Its data is often on `/cpaas/ck`. + +A third kind is a ClickHouse that a team runs with its own ClickHouseInstallation in its own namespace (container `clickhouse-pod`). Cause 6 covers it. + +## Root Cause + +### Cause 1: ClickHouse slow-query logs filled the node disk + +`du` on the affected node stops in the slow-query volume, not in Elasticsearch. These files grow even when no MySQL pod is present, because the slow-query ClickHouse keeps its own log tables, including `system.query_log`. Other platform pods then fail because the node cannot write. + +A `pxc` `chi-ck` pod in `cpaas-system` that is abnormal and does not start is this default slow-query deployment as well. When nobody uses it, stop it by setting its replicas to 0. To remove the feature completely, uninstall its operator (its CSV and Subscription go away with it), set the ClickHouse replica count to 0, and delete the PVC. The operator does not recreate the instance after that. + +### Cause 2: The logging ClickHouse is out of CPU or memory + +Both CPU and memory of the logging ClickHouse are at 100%, and the console cannot query. Scaling out the cluster restores queries. A smaller example is one ClickHouse node at 2 CPU / 4Gi receiving logs from two clusters: query memory runs out, and raising that node's resources stops the overflow. Neither situation involves a corrupt table, so add capacity before you consider a reinstall. + +Raising memory does not always help. In one cluster, a log query kept exceeding the memory limit after the plugin memory was raised from 4G to 10G. The error stopped only after the environment was scaled out. A `MEMORY_LIMIT_EXCEEDED` with a `maximum` of about 3.6 GiB at a 4Gi limit is covered in [nevermore and ClickHouse log components alert on CPU or memory, restart, or ignore new resource values on ACP](./nevermore_and_ClickHouse_log_components_alert_on_CPU_or_memory_restart_or_ignore_new_resource_values_on_ACP.md). + +### Cause 3: One logging node filled, and after expansion the audit tables are read-only + +The pod restart loop says the disk is full. After expansion the pod is `Running`, and the console works for a while if you widen the time range. Then queries fail again. The database log on that node shows errors, some audit tables are read-only (`is_readonly = 1`), and the three nodes do not hold the same amount of data. + +A likely way to get there is an earlier uninstall and reinstall in which only some of the nodes had their data cleaned. Expansion alone does not make the replicas match. The repair is to remove the bad node from the ClickHouse cluster, join it again, and reinstall ClickHouse. + +A size difference between replicas is not this cause by itself. Replicas of the 3-replica logging ClickHouse normally differ by tens of gigabytes. + +### Cause 4: System component logs dominate the ClickHouse volume + +Most of the stored volume comes from `cpaas-system` workloads. Their log level can be lowered directly on the component. The collector has no switch that drops one namespace's debug lines, so the volume has to be cut at the source or kept for fewer days. Do not change the log level of the MetalLB components without first confirming the change with product support. + +### Cause 5: Logging ClickHouse pods exit with code 137 and the plugin memory limit is too low + +All three logging ClickHouse pods restart repeatedly, with container exit code `137`. Raising the ClickHouse plugin memory limit stops the restarts. CPU and memory utilization can stay high afterwards. + +Exit code `137` means only that the container received SIGKILL. A kernel out-of-memory kill is normally reported as `Reason: OOMKilled`. With `Reason: Error`, the kubelet may have killed the container after failed liveness probes. If the events repeat `Liveness probe failed` on `:8123/ping` with `connection refused` for days, and `clickhouse-operator` in `ck-system` is in `CrashLoopBackOff`, the database is broken rather than short of memory. See [ClickHouse log queries fail after the log storage had been working: missing tables or broken parts on ACP](./ClickHouse_log_queries_fail_after_the_log_storage_had_been_working_missing_tables_or_broken_parts_on_ACP.md) (Cause 5). The same article covers tables that are still missing after the restarts stop (Cause 2). + +### Cause 6: A ClickHouse managed by its own ClickHouseInstallation restarts with OOMKilled + +The container of a ClickHouse that is not the logging plugin keeps restarting, and its events show OOM. The instance is a `ClickHouseInstallation` in its own namespace, for example with pod template `clickhouse:21.8` and container `clickhouse-pod`. The fix is made on the ClickHouseInstallation, because the operator rebuilds the StatefulSet from it. Raise the memory requests and limits of `clickhouse-pod` (to 8Gi in a confirmed fix). Set `max_server_memory_usage` below the limit (6Gi, `6442450944`), so that ClickHouse refuses work before the kernel kills it. + +## Resolution + +### Cause 1: ClickHouse slow-query logs filled the node disk + +1. Confirm which volume is full before you delete anything: + + ```bash + df -hT / /cpaas /cpaas/ck + kubectl -n cpaas-system get sts,pod,pvc | grep -i slowsql + ``` + +2. If the slow-query instance is in use, find the largest tables and truncate only the log table at the top of that list (usually `system.query_log`): + + ```bash + kubectl -n cpaas-system exec chi-pxc-ck-slowsql-cluster-0-0-0 -c clickhouse-pod -- clickhouse-client --query \ + "SELECT database, table, formatReadableSize(sum(bytes_on_disk)) AS size FROM system.parts WHERE active GROUP BY database, table ORDER BY sum(bytes_on_disk) DESC LIMIT 10" + ``` + + > **Warning:** `TRUNCATE` permanently removes all rows of that table. Precondition: only for a ClickHouse system log table (such as `system.query_log`) that the list above shows as the largest. Never truncate tables of the `observability` database. Rollback: none. + + ```bash + kubectl -n cpaas-system exec chi-pxc-ck-slowsql-cluster-0-0-0 -c clickhouse-pod -- clickhouse-client --query "TRUNCATE TABLE system.query_log" + ``` + + Do not delete files under `/cpaas/ck` by hand while the slow-query pod runs. + +3. If the slow-query instance is not used, stop it instead. + + > **Warning:** Setting the replica count to 0 and deleting the PVC removes the slow-query history and stops slow-query collection. MySQL pods are not affected. Uninstalling the operator also removes its CSV and Subscription, so reinstall the operator if another ClickHouse still needs it. Precondition: the pod you stop is `chi-pxc-ck-slowsql-cluster-0-0-0`, not the logging cluster, and nobody uses the slow-query console. Rollback: scale the replicas back to the previous count. Data on the deleted PVC does not return. + + ```bash + kubectl -n cpaas-system scale sts/chi-pxc-ck-slowsql-cluster-0-0 --replicas=0 + kubectl -n cpaas-system delete pvc + ``` + + If an operator scales the StatefulSet back up, set the replicas to 0 on the ClickHouseInstallation that owns it, uninstall that operator, and delete the PVC again. + +### Cause 2: The logging ClickHouse is out of CPU or memory + +1. Check usage on the logging pods. Ignore pods that match `slowsql`. + + ```bash + kubectl -n cpaas-system top pod | grep -i clickhouse + kubectl -n cpaas-system get pods -o wide | grep -i clickhouse + ``` + +2. Add capacity. + + > **Warning:** Raising the resources of the logging ClickHouse, or scaling it, restarts replicas, and queries fail over while they roll. Precondition: `top` shows CPU or memory near 100% on the logging pods (not on `chi-pxc-ck-slowsql-cluster`), and the nodes have the extra CPU and memory. Rollback: set the previous values on the plugin again. + + Set the new CPU and memory on the Log Storage for ClickHouse cluster plugin, so the plugin does not write the old values back. No single size is prescribed. Size from `top` and the free memory on the nodes. A node that ingests logs from two clusters needs a larger size even when average CPU looks moderate. When more memory does not stop query memory errors, scale the environment out instead. + +### Cause 3: One logging node filled, and after expansion the audit tables are read-only + +1. Expand the disk or the PVC so the pod stays `Running`. A wider time range in the console only proves that some parts are readable. + +2. On each replica, compare the read-only flags. Newer images use `clickhouse client`, older images `clickhouse-client`. + + ```bash + kubectl -n cpaas-system exec -c clickhouse -- clickhouse-client --query \ + "SELECT database, table, is_readonly, absolute_delay FROM system.replicas ORDER BY is_readonly DESC" + ``` + +3. Rejoin the bad node. + + > **Warning:** Removing the bad logging node and reinstalling ClickHouse drops parts that existed only on that node, and log search is down during the reinstall. Precondition: the disk is already large enough for the pod to run, `system.replicas` shows read-only audit tables on that node, console queries still fail after expansion, and you have a maintenance window. A replica whose size differs from the other two is not enough by itself. Rollback: none for parts that were only on the removed node. + + Take the abnormal node out of the ClickHouse cluster, join it again, and reinstall the log storage plugin through the original install path. Do not empty the data on the healthy replicas to force them to match, and do not uninstall again without removing the bad node. + +### Cause 4: System component logs dominate the ClickHouse volume + +1. Rank namespaces and components by log volume in the console. + +2. Lower the log level on the `cpaas-system` Deployments that dominate. Use the setting that each component already documents: a flag such as `--v`, an environment variable such as `LOG_LEVEL`, or a configuration file. Leave the MetalLB components at their shipped level. + +3. If the volume cannot be cut at the source, shorten the log retention in the log storage settings. + +### Cause 5: Logging ClickHouse pods exit with code 137 and the plugin memory limit is too low + +1. Read the last termination and the probe events: + + ```bash + kubectl -n cpaas-system get pod -o jsonpath='{range .status.containerStatuses[*]}{.name}{" exit="}{.lastState.terminated.exitCode}{" reason="}{.lastState.terminated.reason}{"\n"}{end}' + kubectl -n cpaas-system describe pod | grep -i 'liveness probe failed' + kubectl -n ck-system get pods + ``` + + If the liveness probe on `8123` has failed for days and `clickhouse-operator` is in `CrashLoopBackOff`, stop here and use the article in Cause 5 of Root Cause. + +2. Raise the memory limit. + + > **Warning:** Raising memory on the logging ClickHouse rolls its pods, and queries and inserts pause on each pod while it restarts. Precondition: the pods are the logging cluster, not `chi-pxc-ck-slowsql-cluster`, the last exit code is 137, and the nodes have free memory for the new limit. Rollback: set the previous memory limit again. + + Set the new memory limit on the ClickHouse log storage plugin, sized from the usage that led to exit code 137 and from the free memory on the node. If the plugin form is not available, the StatefulSet can be changed directly, but the plugin can write it back: + + ```bash + kubectl -n cpaas-system set resources sts/ \ + -c --limits=memory= + ``` + + Read the container name with `kubectl -n cpaas-system get pod -o jsonpath='{.spec.containers[*].name}'`. + +3. Watch that the restarts stop. + +### Cause 6: A ClickHouse managed by its own ClickHouseInstallation restarts with OOMKilled + +1. Confirm the kill reason and the current memory, and save the ClickHouseInstallation: + + ```bash + kubectl -n get chi + kubectl -n get pod -o jsonpath='{range .status.containerStatuses[*]}{.name}{" exit="}{.lastState.terminated.exitCode}{" reason="}{.lastState.terminated.reason}{"\n"}{end}' + kubectl -n get pod -o jsonpath='{range .spec.containers[*]}{.name}{" req="}{.resources.requests.memory}{" lim="}{.resources.limits.memory}{"\n"}{end}' + kubectl -n get chi -o yaml > /tmp/.before.yaml + ``` + + Only `reason=OOMKilled` on the ClickHouse container (or OOM in the pod events) matches this cause. `Error` with a failing probe on `8123` does not. + +2. Edit the ClickHouseInstallation. + + > **Warning:** Editing the ClickHouseInstallation makes the operator roll the ClickHouse pods of that instance, which stops serving queries and inserts while each pod restarts. Precondition: only when `OOMKilled` is confirmed, the backup above is saved, and the node has the new memory free. Rollback: `kubectl apply -f /tmp/.before.yaml` returns the previous resources and settings, and the pods roll again. + + ```bash + kubectl -n edit chi + ``` + + Change only these three things: `max_server_memory_usage` under `spec.configuration.settings`, and the memory requests and limits of the ClickHouse container in the pod template. Use this instance's pod template name and container name: + + ```yaml + spec: + configuration: + settings: + max_server_memory_usage: 6442450944 # 6Gi, below the 8Gi limit + templates: + podTemplates: + - name: clickhouse:21.8 + spec: + containers: + - name: clickhouse-pod + resources: + requests: + memory: 8Gi + limits: + memory: 8Gi + ``` + + 8Gi and 6Gi are example values. Size both from the instance's usage, and keep `max_server_memory_usage` below the limit. + +3. Watch that the restarts stop. + +## Diagnostic Steps + +1. Separate the ClickHouse installations: + + ```bash + kubectl get chi -A + kubectl -n cpaas-system get pods,sts,pvc | grep -Ei 'clickhouse|slowsql|chi-' + du -sh /cpaas/ck 2>/dev/null + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | `/cpaas/ck` full with `slowsql` pods, or an abnormal `pxc` `chi-ck` pod that nobody uses | Cause 1 | + | Logging pods at 100% CPU or memory | Cause 2 | + | One logging node with read-only audit tables after expansion | Cause 3 | + | `cpaas-system` components dominate the log volume in the console | Cause 4 | + | Logging ClickHouse pods whose last exit code is 137, without days of failed liveness probes on `8123` | Cause 5 | + | A ClickHouse outside the logging plugin with container `clickhouse-pod`, last state `OOMKilled` | Cause 6, change the ClickHouseInstallation and not the StatefulSet | + +3. The monitoring graphs of the node can stay low while a container is killed. `dmesg` reports `Memory cgroup out of memory` when a container hits its own limit, regardless of node memory. + +## Related Information + +- [ClickHouse log queries fail after the log storage had been working: missing tables or broken parts on ACP](./ClickHouse_log_queries_fail_after_the_log_storage_had_been_working_missing_tables_or_broken_parts_on_ACP.md) +- [nevermore and ClickHouse log components alert on CPU or memory, restart, or ignore new resource values on ACP](./nevermore_and_ClickHouse_log_components_alert_on_CPU_or_memory_restart_or_ignore_new_resource_values_on_ACP.md) +- [ClickHouse log storage logs RaftInstance errors with pod restarts, merge CPU spikes, or a hand-built ClickHouse on ACP](./ClickHouse_log_storage_logs_RaftInstance_errors_with_pod_restarts_merge_CPU_spikes_or_a_hand-built_ClickHouse_on_ACP.md) diff --git a/docs/en/solutions/Logs_audit_and_events_stop_while_the_log_Kafka_is_degraded_on_ACP.md b/docs/en/solutions/Logs_audit_and_events_stop_while_the_log_Kafka_is_degraded_on_ACP.md new file mode 100644 index 000000000..26ad2a65b --- /dev/null +++ b/docs/en/solutions/Logs_audit_and_events_stop_while_the_log_Kafka_is_degraded_on_ACP.md @@ -0,0 +1,214 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Logs, audit and events stop while the log Kafka is degraded on ACP + +## Issue + +Platform logs, audit records and events stop arriving, and the log storage Kafka is part of the reason. Typical reports: + +- "The Events tab in the application view is empty, and pods cannot be debugged." +- "Logs, audit and events stop six days ago." +- "One of three Kafka nodes failed, log collection stopped, and `razor` memory spiked." +- "The log disk was full, we deleted the data, and the observability page still shows no logs." + +The broker is the log storage Kafka in `cpaas-system` (StatefulSet `cpaas-kafka`, pods `cpaas-kafka-0..2`, with `cpaas-zookeeper`). Its data is on node-local storage, typically `/cpaas/data/kafka0`, plus `/cpaas/data/kafka` and `/cpaas/data/zookeeper`. `lanaya` consumes the log topics into Elasticsearch. `razor` writes audit, events and logs into Kafka on clusters where it is deployed. + +When a broker does not start at all, see [cpaas-kafka broker fails to start: corrupt segment, no space, broker id conflict, or deleted StatefulSet on ACP](./cpaas-kafka_broker_fails_to_start_corrupt_segment_no_space_broker_id_conflict_or_deleted_StatefulSet_on_ACP.md). + +## Root Cause + +### Cause 1: cpaas-kafka-0 is abnormal with a large kafka0 directory, and logs, audit and events stop + +The application Events tab is empty, and a pod that cannot start cannot be debugged from the console. `kubectl describe pod` and `kubectl logs` from the backend still show the events and logs. Logs, audit records and events in platform management reach back only to a date several days earlier. The log storage plugin is stuck in `部署中` ("deploying"). + +`cpaas-kafka-0` is `0/1 Running`, while the other brokers, ZooKeeper, Elasticsearch and `razor` are `1/1`. The Grafana dashboard Platform Middleware / Log Center Plugin Status shows only 2 brokers up, and consumer group `alauda_audit` far behind on every `ALAUDA_AUDIT_TOPIC` partition. `du -sh *` under `/cpaas/data/` on the log storage node shows `kafka0` at tens of gigabytes. + +After the owner confirmed that the accumulated data could be discarded, the consumers and Kafka were stopped and the `kafka0` directory was emptied. The log storage plugin then recovered, and the Events page showed events again. + +A large `kafka0` alone is not this cause. When every broker is `Running` and the lag is still moving, it is consumption lag, which drains without deleting segments once `lanaya` and Elasticsearch catch up. + +### Cause 2: One broker fails and every write fails, and razor is OOMKilled, because each partition needs two in-sync replicas + +On a three-broker cluster, one broker failed. Log collection stopped, `razor` memory climbed, and all `razor` pods restarted with last state `Terminated`, `OOMKilled`, exit code `137`. Before the restart, `razor` repeated: + +```text +server/server.go:78 | server | Failed to write kafka {"category": "logs", "error": "context deadline exceeded"} +``` + +It did not move to the healthy brokers. Its `--kafka-endpoint` listed all three broker addresses on port `9092`, so the client configuration is not the fault. + +The broker configuration has `default.replication.factor=2` and `min.insync.replicas=2`. `kafka-topics.sh --describe` on `ALAUDA_LOG_TOPIC` shows replication factor 2 and `min.insync.replicas: 2`, and the producers use `acks=all`. Every partition has one of its two replicas on the failed broker, so every ISR drops to 1, below the minimum, and every write is refused. `razor` buffers what it cannot send until it is OOMKilled. When the broker comes back, memory and collection recover. + +For `acks=all` producers, a topic tolerates one broker down only when the replication factor minus `min.insync.replicas` is at least 1. For example, factor 3 with minimum 2, or factor 2 with minimum 1. + +### Cause 3: After a full log disk was cleaned, the observability page still shows no logs until Kafka and ZooKeeper start again in order + +The log disk filled, and data was deleted to free it. The observability page still shows no logs. Deleting the Kafka and ZooKeeper data does not help, and neither does deleting the Elasticsearch, Kafka and ZooKeeper data and then the Elasticsearch log indices. Logs become visible again after `nevermore` is restarted and the Kafka and ZooKeeper data is deleted once more. + +The failure is tied to the startup order of ZooKeeper, Kafka and Elasticsearch. Start ZooKeeper first, then Kafka once ZooKeeper is fully ready, then Elasticsearch once Kafka is fully ready. It is not known which of the last steps made the difference, so the Resolution repeats all of them in that order. + +## Resolution + +The Kafka commands below run inside `cpaas-kafka-0`. Use the listener and client properties file that this cluster's brokers expose. If a script is not on the `PATH`, it is in `/opt/kafka/bin/`. + +### Cause 1: cpaas-kafka-0 is abnormal with a large kafka0 directory, and logs, audit and events stop + +1. Confirm the signature. On the log storage node: + + ```bash + cd /cpaas/data && du -sh * + ``` + + On the cluster: + + ```bash + kubectl get pod -A -owide | grep -E 'kafka|razor|zookeeper|elas' + ``` + + In Grafana, Platform Middleware / Log Center Plugin Status, check how many brokers are up and the lag per consumer group. + +2. Record the replica counts, then stop the consumers and Kafka: + + ```bash + kubectl -n cpaas-system get deploy lanaya + kubectl -n cpaas-system get sts cpaas-kafka + kubectl -n cpaas-system scale deploy lanaya --replicas=0 + kubectl -n cpaas-system scale sts cpaas-kafka --replicas=0 + ``` + + If the cluster also runs `razor` against this Kafka, stop it as well. + +3. Empty `kafka0`. + + > **Warning:** Emptying `kafka0` discards every log, audit and event record still in Kafka, and those records never reach the store. Precondition: only when `cpaas-kafka-0` is abnormal, the directory is large, the owner has confirmed that the accumulated data can be discarded, and the consumers and Kafka are stopped. Rollback: none. + + On each log node: + + ```bash + cd /cpaas/data/kafka0 && rm -rf ./* + ``` + +4. Start Kafka, then the consumers, with their previous replica counts: + + ```bash + kubectl -n cpaas-system scale sts cpaas-kafka --replicas= + kubectl -n cpaas-system scale deploy lanaya --replicas= + ``` + + When the plugin is healthy, open an application's Events tab. + +### Cause 2: One broker fails and every write fails, and razor is OOMKilled, because each partition needs two in-sync replicas + +1. Confirm the signature: + + ```bash + kubectl -n cpaas-system exec cpaas-kafka-0 -- \ + kafka-topics.sh --bootstrap-server \ + --command-config --describe --topic ALAUDA_LOG_TOPIC + kubectl -n cpaas-system logs -p | grep 'Failed to write kafka' + kubectl -n cpaas-system describe pod | grep -E 'OOMKilled|Restart Count' + ``` + + `ReplicationFactor: 2` with `min.insync.replicas=2`, and an `Isr` of one broker for every partition while one broker is down, match this cause. + +2. Bring the failed broker back first. That alone restores collection. + +3. To tolerate one broker down in the future, change the topic configuration together with product support, in a maintenance window. Read the current topic configuration: + + ```bash + kubectl -n cpaas-system exec cpaas-kafka-0 -- \ + kafka-configs.sh --bootstrap-server \ + --command-config --describe --entity-type topics --entity-name ALAUDA_LOG_TOPIC + ``` + + > **Warning:** Lowering `min.insync.replicas` to 1 lets a write succeed with a single copy, so losing that broker before replication catches up loses those records. Raising the replication factor instead needs a partition reassignment and more disk on every broker. Precondition: only when this cause is confirmed and the change is agreed. Rollback: `kafka-configs.sh ... --alter --entity-type topics --entity-name ALAUDA_LOG_TOPIC --delete-config min.insync.replicas` returns the topic to the broker default. + + Check `ALAUDA_AUDIT_TOPIC` and `ALAUDA_EVENT_TOPIC` the same way, because `razor` also writes audit records and events. + +### Cause 3: After a full log disk was cleaned, the observability page still shows no logs until Kafka and ZooKeeper start again in order + +1. Record the replica counts, confirm that the disk has room, and check whether the log storage workloads came up together after the cleanup: + + ```bash + df -hT /cpaas /cpaas/data + kubectl -n cpaas-system get sts cpaas-zookeeper cpaas-kafka cpaas-elasticsearch + kubectl -n cpaas-system get pods -o wide | grep -E 'cpaas-zookeeper|cpaas-kafka|cpaas-elasticsearch|nevermore' + ``` + +2. Restart the collector. + + > **Warning:** Restarting DaemonSet `nevermore` pauses log, audit and event collection on each node while its pod restarts. Precondition: only when logs are still missing after the disk cleanup and the log storage pods are `Running`. Rollback: none needed; the DaemonSet recreates the pods. + + ```bash + kubectl -n cpaas-system rollout restart ds/nevermore + kubectl -n cpaas-system rollout status ds/nevermore + ``` + +3. If logs are still missing, stop Elasticsearch, Kafka and ZooKeeper: + + ```bash + kubectl -n cpaas-system scale sts cpaas-elasticsearch cpaas-kafka cpaas-zookeeper --replicas=0 + kubectl -n cpaas-system get pods | grep -E 'cpaas-zookeeper|cpaas-kafka|cpaas-elasticsearch' + ``` + +4. Empty the Kafka and ZooKeeper directories. + + > **Warning:** Emptying `/cpaas/data/zookeeper`, `/cpaas/data/kafka` and `/cpaas/data/kafka0` deletes every log, audit and event record still in Kafka and the ZooKeeper state of the log cluster, and those records never reach storage. Elasticsearch indices are not touched. Precondition: only when the log disk was already cleaned, logs are still not visible after the `nevermore` restart, and all three StatefulSets are at 0 replicas. Rollback: none. + + On each Kafka node, empty the directories and keep them: + + ```bash + cd /cpaas/data/zookeeper && rm -rf ./* + cd /cpaas/data/kafka && rm -rf ./* + cd /cpaas/data/kafka0 && rm -rf ./* + ``` + +5. Start ZooKeeper, then Kafka, then Elasticsearch, and wait for each to be fully ready before the next. Use the replica counts you recorded: + + ```bash + kubectl -n cpaas-system scale sts cpaas-zookeeper --replicas= + kubectl -n cpaas-system rollout status sts/cpaas-zookeeper + kubectl -n cpaas-system scale sts cpaas-kafka --replicas= + kubectl -n cpaas-system rollout status sts/cpaas-kafka + kubectl -n cpaas-system scale sts cpaas-elasticsearch --replicas= + kubectl -n cpaas-system rollout status sts/cpaas-elasticsearch + ``` + + If `kubectl get deploy,sts` shows Elasticsearch as a Deployment on this cluster, scale `deploy/cpaas-elasticsearch` instead. + +6. When all pods are `Running`, query current logs on the observability page. + +## Diagnostic Steps + +1. Read the broker and consumer state: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep -E 'kafka|zookeeper|razor|lanaya' + kubectl -n cpaas-system logs -p cpaas-kafka-0 | tail -30 + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | `cpaas-kafka-0` at `0/1` with a very large `kafka0` and old data, and the plugin stuck deploying | Cause 1 | + | Every broker `Running` and lag still moving | Consumption lag, do not empty `kafka0` | + | All brokers but one up, writes refused, and `razor` `OOMKilled` | Cause 2, read the topic describe for the factor and `min.insync.replicas` | + | Every log pod `Running` after a full-disk cleanup, and still no logs on the observability page | Cause 3 | + +## Related Information + +- Related articles: [cpaas-kafka broker fails to start: corrupt segment, no space, broker id conflict, or deleted StatefulSet on ACP](./cpaas-kafka_broker_fails_to_start_corrupt_segment_no_space_broker_id_conflict_or_deleted_StatefulSet_on_ACP.md) covers brokers that do not start. +- [Log Kafka lag on a single partition from oversized log entries or a stuck offset on ACP](./Log_Kafka_lag_on_a_single_partition_from_oversized_log_entries_or_a_stuck_offset_on_ACP.md) +- [Elasticsearch log storage Pending or every cluster loses logs after the log disk fills on ACP](./Elasticsearch_log_storage_Pending_or_every_cluster_loses_logs_after_the_log_disk_fills_on_ACP.md) diff --git a/docs/en/solutions/Maven_stage_misses_Nexus_mirrorOf_stays_central_after_mavenMirrorURL_or_501_HTTPS_Required_after_generate-settings.md b/docs/en/solutions/Maven_stage_misses_Nexus_mirrorOf_stays_central_after_mavenMirrorURL_or_501_HTTPS_Required_after_generate-settings.md new file mode 100644 index 000000000..c47edf162 --- /dev/null +++ b/docs/en/solutions/Maven_stage_misses_Nexus_mirrorOf_stays_central_after_mavenMirrorURL_or_501_HTTPS_Required_after_generate-settings.md @@ -0,0 +1,95 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.0" + - "4.3" +--- + +# Maven stage misses Nexus: mirrorOf stays central after mavenMirrorURL, or 501 HTTPS Required after generate-settings + +## Issue + +The Maven stage of a DevOps (Tekton) pipeline does not use the built-in Nexus as intended. One of these symptoms is shown: + +- The standard Maven task is given `mavenMirrorURL` for the built-in Nexus, but the generated settings keep `central`. Changing that parameter so that `mirrorOf` becomes `*` is not possible. +- Connectors created from the `maven-settings` workspace do not put dependency downloads on Nexus. +- The `generate-settings` step succeeds, Nexus was not selected for the run, and the Maven stage then fails with `Non-resolvable import POM` and `501 HTTPS Required`. The transfer names `https://maven.aliyun.com/repository/spring` and `https://repo.maven.apache.org/maven2/`. + +Maven settings travel in the `maven-settings` workspace. The built-in Nexus is the Service `nexus-nxrm-ha` in the namespace `devops-system`. Related articles: "DevOps Node.js or apt downloads fail: gnutls_handshake failed, npm install hangs, or the Nexus connection is reset" covers npm and network failures. + +## Root Cause + +### Cause 1: `mavenMirrorURL` always generates `mirrorOf` `central` + +The `mavenMirrorURL` parameter of the Maven task always generates a mirror with `mirrorOf` `central`. No value of that parameter yields `*`. Artifacts requested from any other repository id skip the mirror. The workspace Maven connector mounts `settings.xml` and is limited to proxy and authentication unless `useAsMirror=true` is set and the connector URL is a repository that can serve as a mirror. + +### Cause 2: `501 HTTPS Required` after `generate-settings` when Nexus is not set + +When Nexus is not specified, the pipeline generates default Maven settings, and `generate-settings` still succeeds. With those defaults the Maven stage fails to resolve an import POM, for example: + +```text +Non-resolvable import POM: org.springframework.boot:spring-boot-dependencies:pom:2.1.11.RELEASE +... could not be transferred from https://maven.aliyun.com/repository/spring ... +501 HTTPS Required ... https://repo.maven.apache.org/maven2/ +``` + +Specifying Maven settings, so that the generated default configuration is not used, clears the failure. This is not Cause 1: `mavenMirrorURL` is not involved, and the failure comes from the generated defaults, not from a mirror left at `central`. + +## Resolution + +### Cause 1: `mavenMirrorURL` always generates `mirrorOf` `central` + +Apply this only when the settings rendered by the task contain `central`, or the connector is mounted and traffic still misses Nexus. + +1. Stop using `mavenMirrorURL` for a full mirror. Put a `settings.xml` into the `maven-settings` workspace, or change the Task template so that it uses that file. To make the connector itself the mirror, set `useAsMirror=true` and aim the connector at the mirror repository URL. + + > **Warning:** A `mirrorOf` of `*` sends every repository id to one Nexus URL. Precondition: only when Nexus already proxies those artifacts, you have a copy of the old settings, and the rendered settings show `central` (or a connector that is not a mirror). Rollback: put the old settings back and run the pipeline again. + + ```xml + + + + internal-nexus + http://nexus-nxrm-ha.devops-system.svc.cluster.local/repository/maven-public/ + * + + + + ``` + + `maven-public` stands for the repository path on your Nexus; replace it with the real one. + +2. Run the pipeline again. The downloads appear in Nexus. + +### Cause 2: `501 HTTPS Required` after `generate-settings` when Nexus is not set + +Apply this only when `generate-settings` succeeded, Nexus was not selected on that run, and the Maven log contains `Non-resolvable import POM`, `https://maven.aliyun.com/repository/spring`, and `501 HTTPS Required`. + +1. Specify Maven settings so that the pipeline does not apply the generated defaults. Place the file in the `maven-settings` workspace, and point it at a repository that already contains the artifacts this build resolves. The mirror block shown for Cause 1 is one way to do it. + + > **Warning:** `*` sends every repository id, including plugin repositories, to that one URL. Precondition: only when the log signature above is in this run, Nexus was not selected, and that URL already proxies or hosts the artifacts the build needs. Keep the previous settings file. Rollback: restore that file and run the pipeline again. If the rendered `` is already `central` because `mavenMirrorURL` was set, use Cause 1 instead of adding a second mirror. + +2. Run the pipeline again. The Maven stage no longer prints `501 HTTPS Required` or the Aliyun spring repository URL. If `` later shows `central` and downloads still miss Nexus, that is Cause 1. + +## Diagnostic Steps + +From the Maven step, find the settings file and read the effective settings: + +```bash +find /workspace /root /home -name settings.xml -print +mvn help:effective-settings -N +``` + +- `central` after `mavenMirrorURL` was set, and downloads that never appear in Nexus: Cause 1. `*` already present, with failures after that, is a Nexus content or network problem, not this parameter. +- `generate-settings` is green, the Maven stage is red, Nexus was left unset, and the log shows the lines below: Cause 2. + + ```text + Non-resolvable import POM + https://maven.aliyun.com/repository/spring + 501 HTTPS Required + ``` + +A Maven stage that ends in `taskruntimeout`, or a JAR that is produced but omits files with Chinese names, is not a mirror problem. diff --git a/docs/en/solutions/Metrics_Service_availability_alert_is_not_1_while_the_target_Pod_looks_healthy_on_ACP.md b/docs/en/solutions/Metrics_Service_availability_alert_is_not_1_while_the_target_Pod_looks_healthy_on_ACP.md new file mode 100644 index 000000000..5f484f840 --- /dev/null +++ b/docs/en/solutions/Metrics_Service_availability_alert_is_not_1_while_the_target_Pod_looks_healthy_on_ACP.md @@ -0,0 +1,208 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Metrics Service availability alert is not 1 while the target Pod looks healthy on ACP + +## Issue + +The real-time alert page reports that the availability of a Metrics Service obtained by Prometheus is not 1 (`the availability of Metrics Service != 1`), or "Prometheus failed to fetch metrics" (`Prometheus获取metrics异常`), or a node kubelet health alert. The rule stays firing for one or three minutes and sometimes clears by itself, often over and over. The Service named in the alert and its Pods are `Running`. `curl` from inside the Pod returns metrics, or an anonymous request to the Service returns HTTP 403, which only shows that the process answers. In vmagent targets the endpoint is `down`, or `up` for that Service is `0`. + +The built-in rule on a cluster has this shape (the `vmcluster` label is the cluster name): + +```promql +max by (namespace, service) (up{vmcluster="",service!="kube-prometheus-exporter-dockerd",job!="apiserver",job!="kube-controller-manager",job!="kube-scheduler",job!="kubelet",job!="kube-dns",job!="kube-etcd"}) != 1 +``` + +The alert reports that a scrape failed. Each cause below is a different reason for the failed scrape. + +## Root Cause + +### Cause 1: the node firewall blocks the metrics port + +`curl` of `/metrics` inside the Pod succeeds. From another node the same endpoint IP and port fails with `No route to host` or `Failed to connect`, and vmagent lists the target as `down`. The host firewall zone on the node that hosts the endpoint does not allow the metrics port, for example `10254` of an ingress-nginx controller metrics Service. This is covered in detail in Cause 3 of [Node firewall or iptables rules block kubelet 10250, metrics 10254 or control-plane traffic on ACP](Node_firewall_or_iptables_rules_block_kubelet_10250_metrics_10254_or_control_plane_traffic_on_ACP.md). + +### Cause 2: calico-felix-metrics-svc is scraped while Felix metrics are disabled + +`calico-felix-metrics-svc` availability is `0`. The `calico-node` template sets `FELIX_PROMETHEUSMETRICSENABLED=false`, but the Service is still exposed on port `9091` with selector `k8s-app=calico-node`, so monitoring scrapes an endpoint that is turned off. After an upgrade to 4.3.2 (cpaas-calico chart `v4.3.30`) the same state was found with Endpoints on `:9091`; there the Service was confirmed as no longer used, left over from before the upgrade. + +### Cause 3: the metrics backend is OOMKilled, so the alert clears by itself + +Availability drops below 1 and later recovers with no scrape change. The backend named by the alert is restarting with `OOMKilled` (for example `clickhouse-operator`). Each restart takes the metrics endpoint away until the Pod is back. + +### Cause 4: cluster-enhancer-apiserver scrape duration reaches the scrape timeout + +The alert names `cpaas-system` / `cluster-enhancer-apiserver`, and `up` is `0` for minutes or hours, often on a cluster where the alert is sent again and again. The Pods are `Running` with normal CPU and memory. An anonymous `curl` to port `8443` returns HTTP 403 `forbidden`, which is the expected answer of this endpoint. `scrape_duration_seconds` for job `cluster-enhancer-apiserver` sits at or above the configured scrape timeout (for example `45.001` against a timeout of `45` with a sampling interval of `60`, the VictoriaMetrics plugin defaults). The scrape does not finish inside the timeout, so `up` is `0`. This was seen with image `cluster-enhancer-apiserver:v4.2.6`. + +### Cause 5: the alb image serves metrics over HTTP + +"Prometheus failed to fetch metrics" fires. On the targets page `serviceMonitor/cpaas-system/kube-prometheus-cpaas-components-1/1` is `0/5 up`, and the ALB targets (ports `1936` and `11782`, endpoint `https-metrics`) report: + +```text +Get "https://:/metrics": http: server gave HTTP response to HTTPS client +``` + +The `alb` image had been changed away from the version the platform ships, and that image serves its metrics over plain HTTP. + +### Cause 6: kubelet health flips because one container produces duplicate metric data + +The node kubelet health alert fires (rule `sum(up{job="kubelet"}) / count(up{job="kubelet"})!=1` for 1 minute), and `up{job="kubelet"}` for one node (`endpoint="https-metrics"`, port `10250`) flips between 0 and 1. Restarting kubelet and containerd does not help. Node events can repeat `Starting kubelet.`, `InvalidDiskCapacity` / `invalid capacity 0 on image filesystem`, and `NodeNotReady` followed by `NodeReady`. A request to the kubelet metrics endpoint fails as a whole because one container on that node produces a duplicate series: + +```text +An error has occurred while serving metrics: collected metric "kubelet_container_log_filesystem_used_bytes" { label:" > label:" > label:" > ... } was collected before with the same name and label values +``` + +When storage on the node had been cleaned by hand earlier, leftovers most likely made that container's metric conflict. + +## Resolution + +### Cause 1: the node firewall blocks the metrics port + +Run the checks in the linked article: the metrics path answers inside the Pod and on its own node, fails from another node, and `firewall-cmd --list-all` on the owning node does not list the port. + +> **Warning:** Opening the port on the node firewall exposes that metrics port to every source the zone allows. Precondition: the three checks match. Use the active zone if it is not `public`. Rollback: `firewall-cmd --zone=public --remove-port=10254/tcp --permanent` and `firewall-cmd --reload`. + +```bash +firewall-cmd --zone=public --add-port=10254/tcp --permanent +firewall-cmd --reload +``` + +After the next scrape the target leaves `down`. + +### Cause 2: calico-felix-metrics-svc is scraped while Felix metrics are disabled + +1. Confirm the mismatch: + + ```bash + kubectl get svc -A | grep calico-felix-metrics + kubectl -n kube-system get ds calico-node -o yaml | grep -n -A1 FELIX_PROMETHEUSMETRICSENABLED + ``` + +2. Where the Service is confirmed unused (as on 4.3.2 after the upgrade), back it up and delete it. + + > **Warning:** Deleting `calico-felix-metrics-svc` removes that scrape target. Precondition: Felix metrics are disabled, the alert names this Service, and nothing else uses it. On other versions confirm that it is unused first. Rollback: `kubectl apply -f /tmp/calico-felix-metrics-svc.yaml`. + + ```bash + kubectl -n kube-system get svc calico-felix-metrics-svc -o yaml > /tmp/calico-felix-metrics-svc.yaml + kubectl -n kube-system delete svc calico-felix-metrics-svc + ``` + +Felix metrics can be disabled on purpose, as in [Disable Calico Node Metrics Port](How_to_Disable_Calico_Node_Metrics_Port.md); in that case the Service is the leftover, not the setting. + +### Cause 3: the metrics backend is OOMKilled, so the alert clears by itself + +1. Confirm that the last termination is `OOMKilled` and that availability returns to 1 after each restart: + + ```bash + kubectl get pod -A | grep + kubectl -n describe pod | grep -E 'OOMKilled|Exit Code|Restart Count' + ``` + +2. Raise memory on the object that owns that workload, above the working set that was killed. + + > **Warning:** Raising memory rolls the backend; its control loop and metrics endpoint pause until the new Pod is Ready. Precondition: the export below is saved. Rollback: set the previous memory limit on the same object. + + ```bash + kubectl -n get deploy -o yaml > /tmp/backend-deploy.yaml + kubectl -n edit deploy + ``` + + If the edit is written back within a minute, an operator or plugin owns the size; make the same change on that owner instead. + +### Cause 4: cluster-enhancer-apiserver scrape duration reaches the scrape timeout + +1. In vmui on the affected cluster, compare scrape duration with the timeout: + + ```promql + scrape_duration_seconds{vmcluster="",job="cluster-enhancer-apiserver"} + ``` + +2. Read the current sampling interval and scrape timeout in the VictoriaMetrics plugin configuration (Marketplace > Cluster Plugins, update the VictoriaMetrics monitoring plugin of that cluster). They are stored as `spec.config.components.vmagent.scrapeInterval` and `scrapeTimeout` on its ModuleInfo, default `60` and `45`. + +3. When duration is at or above the timeout, set the sampling interval to `90` and the scrape timeout to `60` in the plugin configuration and save. The interval must be greater than the timeout. + + > **Warning:** A longer timeout and interval reduce scrape frequency for every target of that vmagent and let slow scrapes hold resources longer. Precondition: `up` is 0, the Pod is Running, and `scrape_duration_seconds` is at or above the current timeout. Rollback: set the interval back to `60` and the timeout back to `45` in the same form. + +4. After the next scrapes, `scrape_duration_seconds` stays below the new timeout, `up` returns to `1`, and the alert stops. + +If the Pod has stayed Running with `up` at 0 for hours and the duration is not pinned at the timeout, restarting the Pod also restored scraping: + +> **Warning:** Restarting `cluster-enhancer-apiserver` drops its endpoint for one start. Precondition: the duration query does not show the timeout case above. Rollback is the new Pod reaching Ready. + +```bash +kubectl -n cpaas-system get pod | grep cluster-enhancer-apiserver +kubectl -n cpaas-system delete pod +``` + +HTTP 403 from an anonymous `curl` is not the reason `up` is 0; do not restart for it. + +### Cause 5: the alb image serves metrics over HTTP + +1. Compare the `alb` image tag with the one the platform version ships (a healthy cluster of the same version is the easiest reference): + + ```bash + kubectl get deploy,ds -A -o wide | grep -i alb + ``` + +2. Set the image back. + + > **Warning:** Changing the `alb` image rolls the ALB Pods and can briefly interrupt traffic through them. Precondition: the running image differs from the shipped one and the alert started after that change. Rollback: `kubectl apply -f /tmp/alb-deploy.yaml`. + + ```bash + kubectl -n get deploy -o yaml > /tmp/alb-deploy.yaml + kubectl -n set image deploy/ = + ``` + +The scrape recovers and the alert clears. + +### Cause 6: kubelet health flips because one container produces duplicate metric data + +1. Fetch the kubelet metrics and read the error: + + ```bash + kubectl get --raw "/api/v1/nodes//proxy/metrics" > /tmp/kubelet-metrics.txt; echo "exit=$?" + ``` + + Or from the node, with the kubelet client certificate: + + ```bash + curl -sk https://:10250/metrics --cacert /etc/kubernetes/pki/ca.crt \ + --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key | head -3 + ``` + +2. Confirm that the Pod named in the duplicate-metric error runs on this node, then recreate it. + + > **Warning:** Recreating the Pod interrupts that workload until it is Running again. Precondition: the kubelet metrics error names that container's metric and the Pod is on the flapping node. Rollback is the controller recreate. + + ```bash + kubectl -n delete pod + ``` + +After the restart the duplicate data is gone and the kubelet health of the node recovers. + +## Diagnostic Steps + +1. Read the alert labels and query `up` for that namespace and Service. `up=0` with a Running Pod is this article. + + ```bash + kubectl get pod -A | grep -E 'vmagent|cluster-enhancer-apiserver|calico-node' + ``` + +2. In-Pod `curl` works and another node gets `No route to host`: Cause 1. + +3. `calico-felix-metrics-svc` at 0 with `FELIX_PROMETHEUSMETRICSENABLED=false`: Cause 2. + +4. The alert clears by itself and the backend's last state is `OOMKilled`: Cause 3. + +5. `cluster-enhancer-apiserver` Running, anonymous `curl` returns 403, and `scrape_duration_seconds` equals the scrape timeout: Cause 4. Expected after the fix: duration below the new timeout and `up` at `1`. + +6. Targets page shows `server gave HTTP response to HTTPS client` on ALB endpoints after an image change: Cause 5. + +7. Kubelet health flapping on one node, and the kubelet metrics endpoint returns `was collected before with the same name and label values`: Cause 6. diff --git a/docs/en/solutions/Monitoring_components_OOMKilled_or_near_their_limits_vmagent_vmstorage_kube-state_Prometheus_vm-operator_on_ACP.md b/docs/en/solutions/Monitoring_components_OOMKilled_or_near_their_limits_vmagent_vmstorage_kube-state_Prometheus_vm-operator_on_ACP.md new file mode 100644 index 000000000..d2019efa3 --- /dev/null +++ b/docs/en/solutions/Monitoring_components_OOMKilled_or_near_their_limits_vmagent_vmstorage_kube-state_Prometheus_vm-operator_on_ACP.md @@ -0,0 +1,184 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.12.x + - 3.14.x + - 3.16.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Monitoring components OOMKilled or near their limits: vmagent, vmstorage, kube-state, Prometheus, vm-operator on ACP + +## Issue + +A platform monitoring component in `cpaas-system` runs out of memory or stays near its limit: + +- `vmagent-agent` restarts with `OOMKilled` / exit code `137`, ends in `CrashLoopBackOff` (`1/2` Ready), or fails its readiness probe so charts have gaps. Critical `cpaas-deployment-rules` alerts fire for `vmagent-agent`, and some nodes or the whole cluster have no data. +- `vmstorage-cluster` sits above 90% CPU or memory on one or every replica, or restarts with `OOMKilled`, and the page says some monitoring cannot be displayed (`部分监控无法显示`). +- `kube-prometheus-exporter-kube-state` is `OOMKilled`, and the Deployment CPU and memory trend panels are empty, or an inspection flags it as abnormal. +- The `prometheus` container is `OOMKilled` and monitoring data stops. +- `vm-operator` restarts on OOM and cluster monitoring is abnormal. + +Sizes set directly on the live Deployment, StatefulSet or VictoriaMetrics object are often written back. The persistent place is the monitoring plugin configuration: Marketplace > Cluster Plugins > update the monitoring plugin > choose a custom scale and set the component quota. It is stored on the plugin's ModuleInfo as `spec.config.components..resources` (for example `vmagent`, `vmstorage`, `vmselect`, `kubeStateExporter`, `prometheus`). + +## Root Cause + +### Cause 1: vmagent is undersized for the scrape and remote-write load + +`kubectl describe pod` shows `Last State: Terminated`, `Reason: OOMKilled`, `Exit Code: 137`, often with thousands of restarts. The log shows a normal start (`kubernetes_sd_configs: added targets: `) and the kill a minute or two later, and after a restart: + +```text +queue/... size (... bytes) is bigger than writer offset (... bytes); this may be the case on unclean shutdown (OOM, kill -9, hardware reset) +``` + +That line is a consequence of the OOM, not a second fault. The agent cache, scraping, and temporary remote-write buffers together exceed the limit (for example about 1.2Gi of cache plus about 200MB of scraping against a 2Gi limit). Triggers seen: a global `vmagent` at 1 CPU / 1Gi; a monitoring configuration change in the plugin; bulk, high-frequency calls to the platform API for container metrics, which also restart `vmselect`; and failed or timed-out scrapes cached in memory. Restarts that look causeless are still OOM: 15s samples are aggregated over 60s, so charts and the 90% alert can miss the peak. When the agent is near its limit without being killed, readiness probe failures show up as chart gaps instead of restarts. + +A `vmagent` that restarts with `remote_write` `502 Bad Gateway` and is not `OOMKilled` is query overload from a very large `certificate_expires_time` series, and more memory does not fix it. Read the limit of the `vmagent` container, not of the `config-reloader` sidecar in the same Pod. + +### Cause 2: vmstorage memory or CPU limit is too small for the stored series + +Restart count is 0 and usage stays above 90%: the limit itself caps the component (for example memory request `1000Mi` / limit `2000Mi`, or a 4-core CPU limit on all three replicas). Tuning the VMCluster cache-hit setting and restarting does not lower CPU. + +When the limit is far below the working set, one replica OOM-restarts (`OOMKilled`, exit `137`, memory at 155.80% of its limit, hundreds of restarts). That case is misleading: `vmagent` logs that it cannot write, restarting `vmagent` brings data back for a while, and the gap returns. The real failure is in `vminsert` and `vmstorage`: + +```text +# vmagent +sending a block with size 258764 bytes to "2:secret-url" was rejected (skipping the block): status code 400; response body: duplicate sample for timestamp +# vminsert +cannot make a copy #2 out of 2 copies according to -replicationFactor=2 ... since a part of storage nodes is temporarily unavailable +cannot dial storageNode "vmstorage-cluster-0.vmstorage-cluster.cpaas-system:8400": ... connect: connection refused +``` + +`successfully dialed -storageNode` follows each time the replica returns, so the gap looks intermittent. + +### Cause 3: kube-state-metrics is sized by /pod_nanny + +The container command is `/pod_nanny` with `--container=exporter-kube-state`, `--cpu`, `--extra-cpu`, `--memory`, `--extra-memory`, `--threshold` and `--deployment=kube-prometheus-exporter-kube-state`. The addon resizer sets CPU from `--cpu` plus `--extra-cpu` per node and memory from `--memory` plus `--extra-memory` per node, so a patch to `resources` loses to the next nanny loop. Starting flags `--cpu=100m`, `--extra-cpu=5m`, `--memory=130Mi`, `--extra-memory=20Mi`, `--threshold=5` had already resized the Pod to `540m` / `1890Mi` and it still OOMed. On 3.12.2 the visible symptom was empty Deployment trend panels while kube-state restarted with `OOMKilled`. + +### Cause 4: the Prometheus container limit is too small + +The `prometheus` container is `OOMKilled` / exit `137` (for example `prometheus-kube-prometheus-0-0` at `Running (3/4)` with limits `cpu: 2` / `memory: 8000Mi`), and the business cluster's data stops, including in the global views. Doubling the limit let it stay up; 4 CPU / 16Gi stopped the restarts of an `8000Mi` instance. + +### Cause 5: vm-operator is OOMKilled + +After the cluster recovered from an outage, global monitoring dashboards were abnormal and `vm-operator` on the business cluster restarted with `OOMKilled`. Hosts near 100% memory at the same time are a separate problem to handle on the nodes. + +## Resolution + +### Cause 1: vmagent is undersized for the scrape and remote-write load + +1. Confirm `137` / `OOMKilled` and read the real `vmagent` container limit: + + ```bash + kubectl -n cpaas-system describe pod | grep -E 'Exit Code|Reason|OOMKilled' + kubectl -n cpaas-system get pod -o jsonpath='{range .spec.containers[*]}{.name}{" req="}{.resources.requests.memory}{" lim="}{.resources.limits.memory}{"\n"}{end}' + ``` + +2. Raise `vmagent` memory in the plugin configuration (custom scale). Sizes that stopped the restarts: `1000Mi` or 1Gi to 2Gi; 2Gi to 4Gi; `6000Mi`; 2 CPU / `2000Mi` where readiness failed above 80% memory. Under bulk API pulls, double CPU and memory of both `vmagent` and `vmselect`, then watch for another OOM. + +3. If the plugin cannot be updated, edit the VMAgent object as a stop-gap: + + > **Warning:** Editing VMAgent `agent` restarts `vmagent`; scraping and remote write pause until the new Pod is Ready, and a size the nodes cannot hold leaves it Pending. Precondition: the last termination is `OOMKilled`, or readiness failures coincide with memory near the limit. Rollback: `kubectl apply -f /tmp/vmagent-agent.yaml`. Carry the value into the plugin configuration afterwards, or a reconcile writes it back. + + ```bash + kubectl -n cpaas-system get vmagent agent -o yaml > /tmp/vmagent-agent.yaml + kubectl -n cpaas-system edit vmagent agent + ``` + +4. Confirm the running Pod shows the new limit and stops restarting. Do not delete the remote-write queue directory; `vmagent` repairs the queue on its next clean start. + +### Cause 2: vmstorage memory or CPU limit is too small for the stored series + +1. When `vmagent` restarts only help for a while, read `vmstorage` and `vminsert` first: + + ```bash + kubectl -n cpaas-system describe pod | grep -E 'Reason|Exit Code|Restart Count' + kubectl -n cpaas-system logs -c vmagent --tail=300 | grep -E 'duplicate sample for timestamp|was rejected' + kubectl -n cpaas-system logs --tail=300 | grep -E 'cannot make a copy|cannot dial storageNode|broken pipe' + ``` + +2. Raise the `vmstorage` limit in the plugin configuration, not with `kubectl edit sts vmstorage-cluster` (the plugin page stays read-only until the scale is switched to custom). Changes that cleared the alert: memory limit `2000Mi` to `4000Mi`; CPU limit 4 to 5 cores; CPU `1000m` to `2000m` where only gaps were seen. Without a better measurement, double the current limit and watch. + + > **Warning:** Changing `vmstorage` resources rolls the replicas one by one; writes and queries to a restarting replica fail over to the others, and with replication factor 2 a second failure during the roll can leave gaps. Rollback: set the previous values in the same form. + +3. Confirm the new limit on the Pods: + + ```bash + kubectl -n cpaas-system get pod | grep vmstorage + kubectl -n cpaas-system get pod -o jsonpath='{.spec.containers[0].resources}{"\n"}' + ``` + +An `OOMKilled` `vmstorage` is fixed by more `vmstorage` memory, not by another `vmagent` restart. + +### Cause 3: kube-state-metrics is sized by /pod_nanny + +1. Print the nanny flags: + + ```bash + kubectl -n cpaas-system get deploy kube-prometheus-exporter-kube-state -o yaml | grep -n -E 'pod_nanny|--cpu|--memory|extra-cpu|extra-memory|threshold' + ``` + +2. Raise both the base and the per-node extra so the result lands above the OOM working set (for example from `130Mi` / `20Mi` upwards). + + > **Warning:** Changing `--memory` or `--cpu` makes `/pod_nanny` resize and restart `exporter-kube-state`; kube-state series pause during the restart. Precondition: the four old flag values are noted. Rollback: set them back. If the old args return within a minute, the flags are chart-owned: set the Kube State Exporter quota in the plugin configuration instead. + + ```bash + kubectl -n cpaas-system edit deploy kube-prometheus-exporter-kube-state + ``` + +3. The limits on the new Pod follow the nanny formula, not numbers typed under `resources`. Empty Deployment trend panels return as soon as the exporter stays up. + +### Cause 4: the Prometheus container limit is too small + +1. Confirm the termination reason and rule out a corrupt WAL: + + ```bash + kubectl -n cpaas-system describe pod | grep -E 'Reason|Exit Code|OOMKilled' + kubectl -n cpaas-system logs -c prometheus --previous --tail=100 | grep -E 'invalid magic number|checksum mismatch|WAL' + ``` + + `invalid magic number` or `checksum mismatch` is a damaged segment that more memory does not repair. A long WAL replay with `Startup probe failed: HTTP probe failed with statuscode: 503` is covered in [Prometheus WAL replay slow or pod restarts before replay finishes on ACP](Prometheus_WAL_replay_slow_or_pod_restarts_before_replay_finishes_on_ACP.md). + +2. Double the Prometheus limit in the plugin configuration (for an `8000Mi` instance, 4 CPU / 16Gi stopped the restarts). + + > **Warning:** The new limit rolls the Prometheus Pod; queries stop until it is Ready, and a size that does not fit the node leaves it Pending. Precondition: the last termination is `OOMKilled` and the log shows no corrupt segment. Rollback: set the previous limit in the same form. + +### Cause 5: vm-operator is OOMKilled + +```bash +kubectl -n cpaas-system get pod | grep vm-operator +kubectl -n cpaas-system describe pod | grep -E 'Reason|Exit Code|OOMKilled' +``` + +> **Warning:** Raising `vm-operator` resources restarts the operator; VictoriaMetrics objects are not reconciled while it is down, while running `vmagent`, `vmselect` and `vmstorage` keep serving. Recreating the other monitoring Pods afterwards interrupts queries and scraping until each is Ready. Precondition: `vm-operator` is restarting with `OOMKilled`. Rollback: set the previous resources in the same form. + +Raise the `vm-operator` resources in the plugin configuration, then recreate the other monitoring Pods in `cpaas-system` one component at a time. Cluster monitoring displays normally afterwards. + +## Diagnostic Steps + +1. Describe the restarting Pod. `OOMKilled` on `vmagent`, with or without the queue-offset line, is Cause 1: + + ```bash + kubectl -n cpaas-system get pod | grep -E 'vmagent|vmstorage|vminsert|vmselect|kube-state|prometheus|vm-operator' + kubectl -n cpaas-system describe pod | grep -E 'Reason|Exit Code|Restart Count' + ``` + +2. High but stable usage on `vmstorage` with 0 restarts, or an `OOMKilled` `vmstorage` behind a `vmagent` that recovers only after restarts, is Cause 2. + +3. For kube-state, print the `/pod_nanny` flags before touching `resources` (Cause 3). + +4. When `vmagent` restarts with `502` on remote write and is not `OOMKilled`, count the certificate series in vmui before adding memory: + + ```promql + count({__name__="certificate_expires_time"}) + ``` + + A very large count is query overload, not this article. + +5. `prometheus` `OOMKilled` with no WAL checksum line is Cause 4. `vm-operator` `OOMKilled` is Cause 5. diff --git a/docs/en/solutions/Monitoring_dashboard_panels_are_blank_or_drop_to_zero_while_the_backend_has_data_on_ACP.md b/docs/en/solutions/Monitoring_dashboard_panels_are_blank_or_drop_to_zero_while_the_backend_has_data_on_ACP.md new file mode 100644 index 000000000..ef96eb255 --- /dev/null +++ b/docs/en/solutions/Monitoring_dashboard_panels_are_blank_or_drop_to_zero_while_the_backend_has_data_on_ACP.md @@ -0,0 +1,151 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.12 + - 3.16 + - 3.18 + - 4.0 + - 4.1 + - 4.2 + - 4.3 +--- + +# Monitoring dashboard panels are blank or drop to zero while the backend has data on ACP + +## Issue + +The monitoring backend has data, but monitoring panels do not show it as expected. Typical symptoms: + +- The application monitoring overview, a namespace dashboard, or the CPU and memory **usage rate** panels show no data. A related request is a command that lists which workloads in the cluster have no CPU or memory limit. +- The network packet send or receive rate drops to 0 periodically, or its latest value reads 0, while traffic is flowing. +- Right after monitoring was deployed, the page is empty or abnormal for several minutes. +- Every monitoring page and vmui show no data (Node and Pod counts `0`, totals `无数据`, no data) although all VictoriaMetrics Pods are `Running`. + +Before changing any dashboard, run the panel's expression in vmui or the Prometheus UI (`https:///clusters//prometheus-0`), or read the response in the browser developer tools (F12, **Network**). If the query returns data and the chart does not, use this article. If the query is empty too, the problem is in the query path; see [Monitoring pages return 502 or context deadline exceeded because port 11780 on the cluster VIP is blocked on ACP](Monitoring_pages_return_502_or_context_deadline_exceeded_because_port_11780_on_the_cluster_VIP_is_blocked_on_ACP.md). + +Related article: [Monitoring page freezes the browser or built-in dashboards are missing on ACP](Monitoring_page_freezes_the_browser_or_built_in_dashboards_are_missing_on_ACP.md) covers frozen pages and missing dashboards. + +## Root Cause + +### Cause 1: there is nothing to graph + +- A namespace dashboard is blank while a single compute-component panel has data, and the namespace has no native application deployed. The namespace-level dashboard and the application monitoring overview monitor native applications only, so no data is the expected result. +- CPU usage rate and memory usage rate show no data when the container has no CPU or memory limit, because the ratio to the limit cannot be computed. This applies to the Pod and application monitoring overview dashboards as well (also with VictoriaMetrics on 4.2.2). The same charts still offer a usage-amount view. + +Neither case is a broken scraper. + +### Cause 2: packet-rate panels drop to zero because the expression uses irate + +The packets received and sent per second keep oscillating: the curve shows data and then resets to zero, on business Pods and in vmui. The namespace dashboard expression uses `irate()`, which uses only the last two samples in the window; a sustained download still produces periodic zeros, so the drops come from the calculation, not from traffic stopping. + +Two related faces: + +- The **latest value** of a packet panel reads 0 on Pods that serve traffic. Packets do not arrive every second, so the latest point can be 0 while the enlarged chart (three-dot menu > **View**) shows the rate fluctuating. Bytes can be non-zero while packets read 0 at the same instant. +- The result depends on the query `step`. The front end uses `step=30` for ranges up to 30 minutes (and in the enlarged view up to 1 day), `step=60` in the small panel from 1 hour, and `step=300` for 3 and 7 days. A packet receive-rate panel (`网络报文量接收速率`) can return data at `step=30` and 0 at 60 or 300, in the Prometheus UI as well. + +### Cause 3: monitoring was just deployed + +On a newly built environment the monitoring page can be empty or abnormal right after deployment, while the VictoriaMetrics Pods are `Running` and data is already being written. Metrics can take about 10 minutes after the monitoring plugin is installed to appear on the **Monitoring** page. + +### Cause 4: the clock of the computer running the browser is wrong + +When the clock of the computer running the browser does not match the cluster time, the cluster monitoring page, the monitoring dashboards, and vmui all show no data, and business-cluster CPU and memory panels are empty, although every `vm` Pod is `Running`. The query time range is likely built from the client clock, so it does not cover the stored samples. + +## Resolution + +### Cause 1: there is nothing to graph + +1. Check whether the namespace has any native application. If it has none, a blank namespace-level dashboard is expected and no change is needed. +2. If the namespace runs workloads and the blank panel is a usage-rate panel, check whether the workloads declare CPU and memory limits. Without a limit, the usage-amount view is what the chart can show. Setting limits caps the container, so it is a decision for the workload owner, not a monitoring fix. + + ```bash + kubectl -n get deploy,sts,ds + kubectl -n get deploy,sts,ds -o jsonpath='{range .items[*]}{.metadata.name}{" limits="}{.spec.template.spec.containers[0].resources.limits}{"\n"}{end}' + ``` + + This reads only the first container of each workload in one namespace. + +3. To list, across the whole cluster, every Pod that has at least one container without a CPU limit or without a memory limit (read-only, requires `jq`): + + ```bash + kubectl get pods -A -o json | + jq -r ' + ["Namespace", "Pod"], + ( + .items[] + | select( + any( + .spec.containers[]; + (.resources.limits.cpu // "") == "" + or + (.resources.limits.memory // "") == "" + ) + ) + | [.metadata.namespace, .metadata.name] + ) + | @tsv + ' | column -t + ``` + + Pods of one Deployment share its name as a prefix. Init containers are not checked by this filter. + +### Cause 2: packet-rate panels drop to zero because the expression uses irate + +1. Find the namespace dashboards (names starting with `ns` in `cpaas-system`) and back them up: + + ```bash + kubectl get monitordashboards.ait.alauda.io -n cpaas-system -o name | grep '/ns' + kubectl get monitordashboards.ait.alauda.io -n cpaas-system -o yaml > /tmp/ns-monitordashboards.yaml + ``` + +2. Replace `irate` with `rate` in the packet-rate expressions only. + + > **Warning:** The change affects the packet-rate panels of every namespace that uses the dashboard, and a bulk replace outside those expressions changes unrelated panels. Do this only when vmui shows the same reset to zero and the packet expression in the backup uses `irate`. To roll back, run `kubectl apply -f /tmp/ns-monitordashboards.yaml`. Stored samples are not affected. + + ```bash + kubectl edit monitordashboards.ait.alauda.io -n cpaas-system + ``` + + No monitoring Pod restart is required. + +3. For the step-dependent zero, select the last 30 minutes, or use the enlarged view for ranges up to 1 day (`step=30`). +4. When the only complaint is a latest value of 0, open the enlarged chart and read the curve. The latest value of this metric has little meaning on its own; nothing needs to change. + +### Cause 3: monitoring was just deployed + +Check that the monitoring Pods are `Running`, wait about 10 minutes, and reload the page: + +```bash +kubectl -n cpaas-system get pod | grep -Ei 'vm|prometheus' +``` + +If the page is still empty after that, follow the port 11780 article linked in the Issue section. + +### Cause 4: the clock of the computer running the browser is wrong + +1. Print the time on a cluster node and compare it with the clock of the computer running the browser, or with the `start` and `end` parameters of a query in the browser developer tools: + + ```bash + date -u '+%Y-%m-%d %H:%M:%S UTC' + ``` + +2. Correct the time of the client computer and reload. The data is displayed once the two clocks match. + +## Diagnostic Steps + +1. Run the panel expression in vmui or the Prometheus UI and read the F12 response for the chart. Expected for this article: the API returns data and the chart does not show it as expected. + +2. Blank usage-rate panels: check whether the namespace has native applications and whether the containers declare limits (Cause 1). Expected: no application, or an empty `limits=` field. + +3. Packet panels that drop to 0: open the dashboard object and look for `irate` in the packet expressions (Cause 2): + + ```bash + kubectl get monitordashboards.ait.alauda.io -n cpaas-system -o yaml | grep -n 'irate' + ``` + +4. A page that is empty only in the first minutes after deployment, with healthy Pods: Cause 3. + +5. Every page and vmui empty with healthy Pods: compare the client clock with a node (Cause 4). Expected: a difference large enough to move the query window away from the stored samples. diff --git a/docs/en/solutions/Monitoring_faults_a_second_Prometheus_missing_vmagent-agent_vmalert_OOMKilled_or_Prometheus_stuck_in_Init_on_ACP.md b/docs/en/solutions/Monitoring_faults_a_second_Prometheus_missing_vmagent-agent_vmalert_OOMKilled_or_Prometheus_stuck_in_Init_on_ACP.md new file mode 100644 index 000000000..7d9e2a6c7 --- /dev/null +++ b/docs/en/solutions/Monitoring_faults_a_second_Prometheus_missing_vmagent-agent_vmalert_OOMKilled_or_Prometheus_stuck_in_Init_on_ACP.md @@ -0,0 +1,161 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.16.x + - 4.2.x + - 4.3.x +--- + +# Monitoring faults: a second Prometheus, missing vmagent-agent, vmalert OOMKilled, or Prometheus stuck in Init on ACP + +## Issue + +A cluster inspection reports monitoring items as abnormal (the report may only say `集群巡检项异常`, "cluster inspection item abnormal", without naming a row), or platform monitoring data looks wrong, and the cause is a real fault rather than a false positive. One of these is in the report, the events, or a pod log: + +- Events or the API server log contain `apiserver was unable to write a JSON response: http: Handler timeout` for `GET /apis/monitoring.coreos.com/v1/prometheusrules`. An application team has deployed its own Prometheus in the cluster. +- The VictoriaMetrics row on a global cluster (including a standby global cluster) says that the `vmagent-agent` Deployment does not exist. +- `vmalert` or another `vm*` container in `cpaas-system` restarts with `OOMKilled`, and monitoring for many clusters drops in and out. +- The cluster monitoring component is abnormal and cannot start (`集群监控组件异常 无法启动`). One Prometheus replica stays in `Init`, and its events say that the data volume cannot be mounted. + +When the report names no specific row, run the first check in Diagnostic Steps before you ask for more detail. It is read-only and separates these causes from each other. + +## Root Cause + +### Cause 1: A second Prometheus makes the API server time out on prometheusrules + +Events and API server logs show `apiserver was unable to write a JSON response: http: Handler timeout` while serving `GET /apis/monitoring.coreos.com/v1/prometheusrules`. A Prometheus installed by an application team runs in the cluster next to the platform monitoring stack. That second Prometheus is what makes platform monitoring and the inspection data wrong. Removing it restores platform monitoring. The platform monitoring release in `cpaas-system` is not the one to remove. + +### Cause 2: deployments.apps "vmagent-agent" not found + +The VictoriaMetrics AppRelease in `cpaas-system` stays `NotReady` with the event `deployments.apps "vmagent-agent" not found`. `kubectl get deploy -n cpaas-system vmagent-agent` confirms that the Deployment is gone. The VMAgent custom resource named `agent` can still exist, with status `expanding`. The `vm-operator` logs show no error or warning, the audit logs show no delete of that Deployment, and no resource patch (`rpch`) explains it. Updating the `victoriametrics` AppRelease, so that the controller renders the chart again, brings the Deployment back. Why the Deployment disappeared is not known. + +### Cause 3: A VictoriaMetrics monitoring container is OOMKilled because its memory limit is too low + +A global VictoriaMetrics monitoring container restarts with `OOMKilled` (for example `vmalert-alert` in `cpaas-system`, with both pods abnormal). Several business clusters store their metrics on the global cluster, and the memory limit of that component is too low for the load. Raising the memory limit on the monitoring component resolves it. This is not Cause 2: here the Deployment exists and is being killed. + +### Cause 4: One Prometheus replica is stuck in Init because its data volume cannot be mounted on a failing storage backend + +The monitoring component is reported abnormal and cannot start. The stack is Prometheus (not VictoriaMetrics). One Prometheus replica stays in `Init`, and the pod events show that the Prometheus database volume cannot be mounted. The PVC is on Ceph storage, and the storage side has a disk problem. After the storage problem is repaired, the volume mounts and monitoring returns to normal, with no change to the monitoring plugin. This is not an OOM restart: the container never starts, so there is no `OOMKilled` last state. + +## Resolution + +### Cause 1: A second Prometheus makes the API server time out on prometheusrules + +1. Find the Prometheus instances that are not the platform release: + + ```bash + kubectl get prometheus -A + kubectl get deploy,sts -A | grep -i prometheus + ``` + +2. Remove the application team's Prometheus. + + > **Warning:** Uninstalling that Prometheus stops the application team's metrics, rules and any dashboard that reads it. Precondition: the `Handler timeout` line is present, and you have identified the object by namespace and name as not belonging to the platform monitoring in `cpaas-system`. Rollback: reinstall it from the team's manifest. Leave the platform monitoring plugin in place. + + Uninstall it with the same method that installed it: the team's Helm release, or `kubectl delete` of their `Prometheus` and its operator. + +3. Check that `GET /apis/monitoring.coreos.com/v1/prometheusrules` answers again and that the inspection items that depend on platform monitoring clear. + +### Cause 2: deployments.apps "vmagent-agent" not found + +1. Confirm that the Deployment is really absent: + + ```bash + kubectl -n cpaas-system get deploy vmagent-agent + kubectl -n cpaas-system describe ars victoriametrics + kubectl -n cpaas-system get vmagent,ars | grep -i agent + ``` + +2. Save the AppRelease, then update it. + + > **Warning:** Updating the `victoriametrics` AppRelease renders the chart again and can roll the VictoriaMetrics pods, and scrapes pause during that rollout. Precondition: the Deployment is missing, and you saved the AppRelease YAML on the global cluster where the event appears. Rollback: `kubectl apply -f /tmp/victoriametrics-ars.before.yaml`. Update the AppRelease; do not delete it. + + ```bash + kubectl -n cpaas-system get ars victoriametrics -o yaml > /tmp/victoriametrics-ars.before.yaml + ``` + + Run the VictoriaMetrics plugin update in the console, so that this object is rewritten and the chart renders again. + +3. Check that `vmagent-agent` exists and that the AppRelease becomes Ready: + + ```bash + kubectl -n cpaas-system get deploy vmagent-agent + kubectl -n cpaas-system get ars victoriametrics + ``` + +### Cause 3: A VictoriaMetrics monitoring container is OOMKilled because its memory limit is too low + +1. Confirm that the container's last state is `OOMKilled`: + + ```bash + kubectl -n cpaas-system get pods | grep -E 'vmagent|vmalert|vmstorage|vminsert' + kubectl -n cpaas-system get pod -o jsonpath='{range .status.containerStatuses[*]}{.name}{" last="}{.lastState.terminated.reason}{"\n"}{end}' + kubectl -n cpaas-system get pod -o jsonpath='{range .spec.containers[*]}{.name}{" lim="}{.resources.limits.memory}{"\n"}{end}' + ``` + +2. Raise the memory limit. + + > **Warning:** Raising the limit rolls the pods of that component, and alerting or scraping pauses briefly while they restart. Precondition: `OOMKilled` is confirmed, and the node has free memory for the new limit. Rollback: set the previous limit on the plugin again. + + Raise the memory limit of the OOMKilled component on the monitoring plugin that owns the pod, so that the plugin does not write the old value back. Size it from the usage that led to the kill and from the free memory on the node. + +3. Watch that the restarts stop. + +### Cause 4: One Prometheus replica is stuck in Init because its data volume cannot be mounted on a failing storage backend + +1. Find out which monitoring stack the cluster uses and which pod does not start: + + ```bash + kubectl -n cpaas-system get pods | grep -E 'prometheus|vmstorage|vmselect|vmagent|vmalert' + kubectl -n cpaas-system describe pod | sed -n '/Events/,$p' + ``` + +2. From the events, take the PVC behind the volume that fails to mount, and find its StorageClass and provisioner: + + ```bash + kubectl -n cpaas-system get pod -o jsonpath='{range .spec.volumes[*]}{.name}{" "}{.persistentVolumeClaim.claimName}{"\n"}{end}' + kubectl -n cpaas-system get pvc -o wide + kubectl get sc -o jsonpath='{.provisioner}{"\n"}' + ``` + +3. Repair the storage backend, not the monitoring component. For Rook Ceph, check the cluster health and the OSDs first: + + ```bash + kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph -s + kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd tree + ``` + + A down OSD or a failed disk is a storage repair. Follow the procedures for that backend. + +4. After the storage side is healthy, watch the Prometheus pod leave `Init`. No change to the monitoring plugin or its resources is needed. + +## Diagnostic Steps + +1. Read the lines that separate the causes, before you uninstall or update anything: + + ```bash + kubectl get events -A --sort-by=.lastTimestamp | grep -i 'Handler timeout' || true + kubectl -n kube-system logs kube-apiserver- --tail=100 | grep -F 'prometheusrules' || true + kubectl get prometheus -A + kubectl -n cpaas-system get deploy vmagent-agent + kubectl -n cpaas-system get pods | grep -E 'prometheus|vmalert|vmagent|vmstorage' + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | `http: Handler timeout` on the `prometheusrules` path, and a Prometheus that is not the platform release | Cause 1 | + | `deployments.apps "vmagent-agent" not found` | Cause 2 | + | `OOMKilled` on a `vm*` container | Cause 3 | + | A Prometheus pod in `Init` whose events report a volume mount failure | Cause 4, check the storage backend of its PVC | + +3. A red monitoring row with healthy pods and only a slow scrape or query is usually a false positive, not one of these causes. + +## Related Information + +- Related articles: [Inspection faults: x86-64-v2 CPU error, discover-device crash loop, small ClickHouse limits, log backlog, or WebCLI role on ACP](./Inspection_faults_x86-64-v2_CPU_error_discover-device_crash_loop_small_ClickHouse_limits_log_backlog_or_WebCLI_role_on_ACP.md) covers the other real faults that a cluster inspection reports. +- [vmagent fails to scrape node-exporter 9100 metrics because its basicAuth does not match node-exporter on ACP](./vmagent_fails_to_scrape_node-exporter_9100_metrics_because_its_basicAuth_does_not_match_node-exporter_on_ACP.md) diff --git a/docs/en/solutions/Monitoring_page_freezes_the_browser_or_built_in_dashboards_are_missing_on_ACP.md b/docs/en/solutions/Monitoring_page_freezes_the_browser_or_built_in_dashboards_are_missing_on_ACP.md new file mode 100644 index 000000000..bb4a1795b --- /dev/null +++ b/docs/en/solutions/Monitoring_page_freezes_the_browser_or_built_in_dashboards_are_missing_on_ACP.md @@ -0,0 +1,151 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.16 + - 3.18 + - 4.0 + - 4.1 + - 4.2 + - 4.3 +--- + +# Monitoring page freezes the browser or built-in dashboards are missing on ACP + +## Issue + +The monitoring backend is healthy, but the monitoring pages themselves do not work. Typical symptoms: + +- Opening the cluster monitoring page spins and then freezes the browser tab, or it loads for minutes. The Administrator overview may fail to load CPU and memory. **Operations Center** > **Monitoring** > **Monitoring Dashboards** may stay on `加载中,请稍候` (loading, please wait) and freeze as soon as the `global` cluster is selected. +- After an upgrade from 3.18, opening **Clusters** > `global` > **Monitoring** freezes the browser tab. +- Cluster and host dashboards are missing from **Operations Center** > **Monitoring**, or a newly created cluster shows an older-style monitoring page than its neighbours. +- Business clusters that are still on 3.16 under a 4.0 `global` cluster show only the overview dashboard. + +All monitoring Pods and `apollo` are `Running`. + +Related articles: [Monitoring dashboard panels are blank or drop to zero while the backend has data on ACP](Monitoring_dashboard_panels_are_blank_or_drop_to_zero_while_the_backend_has_data_on_ACP.md) covers individual panels that are empty; [Cluster Overview page shows no monitoring charts after Cluster Enhancer is installed on ACP](Cluster_Overview_page_shows_no_monitoring_charts_after_Cluster_Enhancer_is_installed_on_ACP.md) covers the cluster **Overview** page. + +## Root Cause + +### Cause 1: a MonitorDashboard has gridPos.true instead of gridPos.y + +A `monitordashboards.ait.alauda.io` object has the key `true` in a `gridPos` block where the key must be `y`. This is the known Kubernetes YAML behavior that turns the key `y` into `true` (kubernetes/kubernetes issue 102308). The front end cannot lay out the dashboard and the browser freezes. + +- A compatibility fix in the MonitorDashboard webhook shipped in 3.18, but a dashboard object that was never regenerated still carries the bad key. On 4.1.3 the affected object was the `advanced-search` MonitorDashboard in `cpaas-system` on the `global` cluster. +- The same bad key was written into `cluster-overview-monitor-dashboard` when **Alauda Container Platform Cluster Enhancer** v4.1.2 was installed while monitoring was down. Cluster Enhancer 4.1.7 contains the fix for that install-time defect. + +### Cause 2: dashboard objects left over from a 3.18 deployment + +After an upgrade from 3.18 to 4.1, `monitordashboards.ait.alauda.io` and `grafanadashboard` objects created by the 3.18 deployment remain although their plugins are gone, and they are incompatible with the new monitoring deployment. Examples are `grafanadashboard` objects named `courier` and `hermes`, old `monitordashboards.ait.alauda.io` objects in `cpaas-system` (such as `cluster-overview-monitor-dashboard`, `cluster-etcd-monitor-dashboard`, `kubernetes-nodes-monitor-dashboard`, `ns-pods-overview`) and in `kube-public` (such as `ns-application-`, `user-alb-status-`), all with the age of the previous deployment. Accessing the platform through another address does not change the freeze, and the `apollo` log shows only routine INFO lines. + +### Cause 3: Cluster Enhancer is not installed + +From 4.2.4, the cluster and host dashboards are provided by the **Alauda Container Platform Cluster Enhancer** plugin. Without it: + +- **Operations Center** > **Monitoring** has no cluster or host dashboards. +- A newly created cluster shows the older-style monitoring page (tabs `ApiServer`, `Controller Manager`, `Scheduler`, `ETCD`, and `集群网络` (cluster network)), while an upgraded cluster with the same versions shows the new one, which adds `集群概览` (cluster overview), `证书状态` (certificate status), and `etcd 备份` (etcd backup). + +On a healthy cluster `cluster-overview-monitor-dashboard` in `cpaas-system` carries the labels `cpaas.io/dashboard.folder: cluster`, `cpaas.io/dashboard.tag.built-in: "true"`, and `helm.sh/chart-name: cluster-enhancer`. On an affected cluster `kubectl get monitordashboards.ait.alauda.io -A` lists only component and namespace dashboards (`albstatus`, `coredns`, `kube-ovn-*`, `ns-application`, `ns-pod`, and similar) and lacks `cluster-overview-monitor-dashboard`, `cluster-certificates-monitor-dashboard`, `cluster-etcd-monitor-dashboard`, `cluster-etcd-backup-monitor-dashboard`, `cluster-kube-apiserver-monitor-dashboard`, `cluster-kube-controller-manager-monitor-dashboard`, `cluster-kube-scheduler-monitor-dashboard`, `cluster-network-monitor-dashboard`, and the `kubernetes-*-monitor-dashboard` set. Copying those objects from another cluster does not help: the page does not request them. + +### Cause 4: business clusters on 3.16 under a 4.0 global cluster + +When the `global` cluster has been upgraded to 4.0 (VictoriaMetrics) and business clusters are still on 3.16 (Prometheus), the monitoring dashboard page for those business clusters has only the overview, and even cluster status is missing. The cause is the version mismatch between `global` and the business clusters. On such a business cluster `kubectl get monitordashboards.ait.alauda.io -A` returns a single object, `kubernetes-cluster` in `cpaas-system`. + +## Resolution + +### Cause 1: a MonitorDashboard has gridPos.true instead of gridPos.y + +1. Back up the dashboard objects and look for the bad key: + + ```bash + kubectl get monitordashboards.ait.alauda.io -A + kubectl get monitordashboards.ait.alauda.io -n cpaas-system -o yaml > /tmp/monitordashboards-backup.yaml + grep -n -A3 gridPos /tmp/monitordashboards-backup.yaml + ``` + + Continue only when a `gridPos` block contains the key `true` instead of `y`. + +2. If the object belongs to the `advanced-search` plugin, remove the plugin and deploy it again, which regenerates its dashboard object. + + > **Warning:** Removing the `advanced-search` plugin takes its function away until it is deployed again. Do this only when the frozen page is traced to the `advanced-search` MonitorDashboard with `true` under `gridPos`. To roll back, deploy the plugin again with its previous settings. + +3. For any other object, restore the key `y` in each affected `gridPos` block. If the affected panel is not needed, deleting that panel from the object also works. + + > **Warning:** A mistaken edit leaves the page frozen or changes the dashboard layout. Do this only when the backup shows `true` under `gridPos` where `y` belongs. To roll back, run `kubectl apply -f /tmp/monitordashboards-backup.yaml`. No collected data is deleted. + + ```bash + kubectl edit monitordashboards.ait.alauda.io -n cpaas-system + ``` + +4. When the object was written by Cluster Enhancer 4.1.2, upgrade Cluster Enhancer to 4.1.7 or later so that the defect does not recur. + +### Cause 2: dashboard objects left over from a 3.18 deployment + +1. List the dashboard objects, pick out the ones created by the 3.18 deployment (compare `metadata.creationTimestamp` and the release labels with objects of the plugin that is installed now), and back everything up: + + ```bash + kubectl get monitordashboards.ait.alauda.io -A --sort-by=.metadata.creationTimestamp + kubectl get grafanadashboard -A --sort-by=.metadata.creationTimestamp + kubectl get monitordashboards.ait.alauda.io -A -o yaml > /tmp/monitordashboards-all.yaml + kubectl get grafanadashboard -A -o yaml > /tmp/grafanadashboard-all.yaml + ``` + +2. Delete the old objects. + + > **Warning:** Deleting dashboard objects removes those dashboards from the monitoring pages. Deleting an object that belongs to an installed plugin removes a current dashboard until the plugin restores it. Do this only after an upgrade from 3.18, for objects that predate the upgrade and whose plugin is no longer installed, and only after the two backup files exist. To roll back, run `kubectl apply -f` with the backup files. + + ```bash + kubectl -n delete monitordashboards.ait.alauda.io + kubectl -n delete grafanadashboard + ``` + +3. The deletes can create `ResourcePatch` (`rpch`) objects. List them; deleting the `rpch` objects created by these deletes makes the installed plugin update and recreate its own dashboards: + + ```bash + kubectl get rpch -A + kubectl delete rpch + ``` + +4. Reload the monitoring page. It no longer freezes. + +### Cause 3: Cluster Enhancer is not installed + +1. Confirm that the plugin is not installed on the cluster: the second command returns nothing. + + ```bash + kubectl get monitordashboards.ait.alauda.io -A | grep -i -E 'cluster|node|host' + kubectl get monitordashboards.ait.alauda.io -n cpaas-system -l helm.sh/chart-name=cluster-enhancer + ``` + +2. Install **Alauda Container Platform Cluster Enhancer** on that cluster from **Cluster Plugins**. The dashboards are synchronized after the installation. Copying `monitordashboard` objects by hand is not a substitute. + +3. Reload the **Monitoring** page. The new layout shows all tabs and the overview panels have data. + +### Cause 4: business clusters on 3.16 under a 4.0 global cluster + +The workaround is to take the dashboard objects from the `global` cluster, adjust them, and apply them to the business cluster. Remove the server-generated metadata before applying: + +```bash +# on the global cluster +kubectl get monitordashboards.ait.alauda.io -n cpaas-system -o yaml > /tmp/.yaml +# remove uid, resourceVersion, creationTimestamp and managedFields, then on the business cluster: +kubectl apply -f /tmp/.yaml +``` + +> **Warning:** A dashboard object from the newer version can reference metrics or recording rules that the 3.16 cluster does not have, and those panels then stay empty. Do this only on a business cluster that is still on 3.16 under a 4.0 `global` cluster, after saving a copy of any object of the same name that already exists there. To roll back, run `kubectl delete -f /tmp/.yaml` on the business cluster. + +## Diagnostic Steps + +1. For a frozen tab, look for the bad key. Output that shows `true:` inside a `gridPos` block matches Cause 1: + + ```bash + kubectl get monitordashboards.ait.alauda.io -A -o yaml | grep -n -A3 gridPos | grep -n 'true:' + ``` + +2. If there is no bad key and the cluster was upgraded from 3.18, list dashboard objects by age. Objects with the age of the 3.18 deployment whose plugin is gone match Cause 2. + +3. For missing cluster or host dashboards, run the `helm.sh/chart-name=cluster-enhancer` query from Cause 3. An empty result matches Cause 3. In the browser developer tools (**Network**), a healthy page requests `monitordashboards?labelSelector=cpaas.io/dashboard.folder=cluster` in `cpaas-system`. + +4. For a business cluster that shows only the overview, compare its version with `global`, and list its dashboards. A single `kubernetes-cluster` object on a 3.16 business cluster under a 4.0 `global` matches Cause 4. diff --git a/docs/en/solutions/Monitoring_pages_return_502_or_context_deadline_exceeded_because_port_11780_on_the_cluster_VIP_is_blocked_on_ACP.md b/docs/en/solutions/Monitoring_pages_return_502_or_context_deadline_exceeded_because_port_11780_on_the_cluster_VIP_is_blocked_on_ACP.md new file mode 100644 index 000000000..193473d45 --- /dev/null +++ b/docs/en/solutions/Monitoring_pages_return_502_or_context_deadline_exceeded_because_port_11780_on_the_cluster_VIP_is_blocked_on_ACP.md @@ -0,0 +1,100 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.16 + - 3.18 + - 4.0 + - 4.1 + - 4.2 + - 4.3 +--- + +# Monitoring pages return 502 or context deadline exceeded because port 11780 on the cluster VIP is blocked on ACP + +## Issue + +The cluster monitoring page shows "loading" for every chart, or the cluster overview shows the banner `监控组件异常,只能获取部分监控数据,请修复。` (the monitoring component is abnormal; only part of the monitoring data can be retrieved, please fix it). The monitoring plugin deployed successfully and `vmagent`, `vminsert`, `vmselect`, `vmstorage`, `vm-operator` and the Prometheus Pods in `cpaas-system` are `Running`. In the browser developer tools (F12, **Network**) the monitoring call fails with HTTP 502, 500, or 503, or `vm-operator` and the global query path log `context deadline exceeded`, while a `curl` from inside the workload cluster returns JSON. + +ACP publishes workload-cluster monitoring through the cluster VIP on **port 11780**. Queries from the `global` cluster look like: + +```text +https://:11780/clusters//vmselect/api/v1/query +``` + +The `global` components that turn the response into a page are `courier`, `courier-api`, and `apollo`, all in `cpaas-system`. + +Related article: [Monitoring pages stay empty because courier-api or apollo holds a stale cluster cache on ACP](Monitoring_pages_stay_empty_because_courier-api_or_apollo_holds_a_stale_cluster_cache_on_ACP.md) covers the case where port 11780 answers but the global components still return errors. + +## Root Cause + +### Cause 1: TCP to port 11780 on the VIP does not complete + +From a `global` node, TCP to the workload-cluster VIP on port 11780 does not complete, so the query times out with `context deadline exceeded`, vmui in a browser returns 502, and `courier-api` logs that the business VIP on 11780 is unreachable. Any device on that path can be responsible: + +- The VIP does not forward 11780 to one or more of the master addresses behind it, or an external load balancer's backend pool for 11780 does not list the nodes that actually own the port (for example nodes that have been drained). +- A firewall, WAF, or other security device in front of the VIP drops 11780. In the browser `GET /platform/monitoring.alauda.io/v1beta1/clusters//metrics?indicators=...` returns `503` with `"reason": "ServiceUnavailable"` and a message naming `Get "http://:11780/clusters//prometheus/api/v1/query?query=..."`. The cluster status can still be `正常` (normal). +- The business cluster and the `global` cluster share one VIP served by HAProxy on the nodes (for example the business API on `:16443` and the `global` API on `:6443`), and the `backend` for 11780 lists the `global` masters instead of the business masters. `courier-api` logs `EOF` for both Alertmanager and Prometheus requests, for example `List alerts error ... "https://:11780/clusters//alertmanager/api/v2/alerts?active=true...": EOF`, and `/platform/monitoring.alauda.io/v1beta1/clusters//metrics` returns `500`. +- A newly added cluster has 11780 closed from the `global` cluster to its VIP and shows only part of the monitoring data. +- The reverse direction: the workload cluster reaches the `global` VIP on 11780 and that VIP does not forward it, while the same request to a `global` master IP works. Then global monitoring is normal and only the workload cluster is abnormal. + +### Cause 2: the load balancer on 11780 presents no TLS certificate + +Port 11780 is reachable, but `openssl s_client` through an external load balancer completes without a certificate, while the same handshake to a master node IP returns the platform certificate. The platform queries `https://:11780`, so the HTTPS request fails even though TCP works. The `vmselect` and `vmalertmanager-alertmanager` logs stay quiet and the page returns 500. A TCP check such as `telnet` only proves the first step and does not detect this. + +## Resolution + +### Cause 1: TCP to port 11780 on the VIP does not complete + +1. From a `global` node, compare the VIP with a master IP of the workload cluster: + + ```bash + curl -vk --connect-timeout 5 "https://:11780/clusters//vmselect/api/v1/query?query=up" + curl -vk --connect-timeout 5 "https://:11780/clusters//vmselect/api/v1/query?query=up" + ``` + +2. If only the VIP fails, open TCP 11780 on every master address behind the VIP, and make sure an external load balancer's pool for 11780 lists the nodes that listen on it. If a firewall or WAF is in front of the VIP, have the network team allow 11780 to the VIP. The browser 502 clears as soon as 11780 is open; no VictoriaMetrics Pod needs a restart. + +3. If the VIP is HAProxy on the nodes, find the configuration file that the running process loaded and read the 11780 backend: + + ```bash + ps -ef | grep [h]aproxy + grep -n -B2 -A8 11780 + ``` + + > **Warning:** Changing the 11780 backend and reloading HAProxy moves monitoring traffic for every cluster that shares the VIP, and a wrong backend list cuts monitoring for the cluster that was working. Do this only when the 11780 backend lists the `global` masters while the request is meant for the business cluster, and save a copy of the file first. To roll back, restore the saved file and reload HAProxy again. + + Point the 11780 backend at the business cluster's master nodes and reload HAProxy with the method already used on those nodes. + +4. For the reverse direction, test from a workload node to the `global` VIP and to a `global` master IP on 11780. If only the VIP fails, the VIP forwarding for 11780 must be fixed on the network side. + +### Cause 2: the load balancer on 11780 presents no TLS certificate + +1. Compare the TLS handshake through the VIP and directly to a master node: + + ```bash + openssl s_client -connect :11780 -servername :11780 -servername from cache error"`. +- An inspection or monitoring API call fails with `bad_response: invalid character '<' looking for beginning of value`. +- After a disaster-recovery switchover of the `global` cluster, every business cluster's Prometheus is shown as not deployed although it is running. +- A severe alert for an expired certificate stays up after the certificate object was deleted. + +The `global` components that serve these pages are `courier`, `courier-api`, and `apollo` in `cpaas-system`. + +Related article: [Monitoring pages return 502 or context deadline exceeded because port 11780 on the cluster VIP is blocked on ACP](Monitoring_pages_return_502_or_context_deadline_exceeded_because_port_11780_on_the_cluster_VIP_is_blocked_on_ACP.md) covers the network path; check it first if a `curl` to port 11780 on the VIP times out. + +## Root Cause + +### Cause 1: courier-api has a stale cluster list + +`courier-api` logs a cache error, vmui already has series, the VictoriaMetrics Pods are `Running` and not CPU-bound, and a newly built or newly attached cluster does not load in the UI. `courier-api` has not picked up the new cluster. The same applies after the Prometheus of a workload cluster is reinstalled: the cluster's monitoring stays empty until `courier-api` is restarted, while `global` monitoring is unaffected. Why `courier-api` does not pick up the reinstalled backend on its own is not known. A `config-reloader` line such as `Post "http://localhost:8429/-/reload": dial tcp [::1]:8429: connect: connection refused` can appear at the same time; it is a missed reload, not the reason the cluster list is stale. + +### Cause 2: the response body is HTML, not JSON + +`bad_response: invalid character '<' looking for beginning of value` means the component expected JSON and received an HTML page. Two sources are known: + +- `courier` is `Unauthorized` when it lists the `clusters` resource on the `global` cluster (for example after `global` worker nodes were moved), so the inspection never loads a real metrics URL. +- `apollo` logs the same parse error while querying Prometheus, for example for `container_memory_working_set_bytes` and CPU utilization. A self-managed Prometheus Deployment in `cpaas-system` whose labels do not match the platform release can sit beside the real backend; the `up` query in the Prometheus UI still succeeds. + +### Cause 3: a certificate alert remains after the expired certificate object was deleted + +The managed cluster works, but the platform still shows a severe alert for an expired certificate whose object has already been deleted. Residual collection data held by `courier` keeps the alert up; `courier` also refreshes this state on its own in about 4 hours. When the `certificate_expires_time` series for the deleted object is still returned by VictoriaMetrics, the stale data is held by Cluster Enhancer instead, and restarting `courier` does not remove it. + +### Cause 4: apollo cannot find the cluster in its cache + +The `global` `apollo` keeps a cache of clusters. When the cache lacks a cluster, every monitoring request for it fails. The `apollo` log shows lines such as: + +```text +ERROR Get cluster from cache error {"error": "cluster not found"} +ERROR New manager error {"error": "cluster not found"} +Parse cluster error {"error": "get cluster from cache error"} +cache.cache/cluster List clusters error, sleep 15s, unauthorized +``` + +for requests such as `/v2/metrics//query`, `/v2/metrics//indicators`, or `/platform/monitoring.alauda.io/v1beta1/clusters//metrics`. A line such as `json: cannot unmarshal array into Go struct field jwtToken.aud of type string` can appear alongside them. + +The stale cache shows up for a newly created cluster, right after monitoring was deployed (the page does not load even after a refresh), and after a `global` primary-to-standby switchover. It is easily mistaken for a missing firewall rule when the clusters are in separate network segments opened per port. The exact reason the cache entry goes stale is not known; restarting `apollo` rebuilds it. + +On a standby `global` cluster that is not serving traffic, an abnormal monitoring status is expected; check the primary. + +## Resolution + +### Cause 1: courier-api has a stale cluster list + +> **Warning:** Recreating `courier-api` fails in-flight monitoring and inspection calls until the new Pod is `Ready`. Do this only when the `courier-api` log shows the cache error, or when a workload cluster's monitoring stays empty after its Prometheus was reinstalled and the Prometheus Pods there are `Running`, preferably in a quiet period if this `global` cluster serves many workload clusters. The Deployment recreates the Pod; nothing needs rolling back if the new Pod is healthy. + +```bash +kubectl -n cpaas-system get pod -o wide | grep courier-api +kubectl -n cpaas-system delete pod +``` + +Wait until the replacement is `Ready`, then reload the cluster in the UI. If the `courier` log shows it still serves the old cluster list, recreate `courier` as well, after `courier-api` is back, not at the same time: + +> **Warning:** Recreating `courier` fails cluster-list calls until the new Pod is `Ready`. The controller recreates the Pod; do not change the Pod template. + +```bash +kubectl -n cpaas-system delete pod +``` + +### Cause 2: the response body is HTML, not JSON + +1. Read which component receives the HTML before restarting anything: + + ```bash + kubectl -n cpaas-system logs deploy/courier --tail=200 | grep -Ei 'unauthorized|clusters' + kubectl -n cpaas-system logs deploy/apollo --tail=200 | grep -F "bad_response" + ``` + +2. If `courier` is denied listing `clusters`, recreate its Pod so it rebuilds its cache. + + > **Warning:** In-flight inspection calls fail until the new `courier` Pod is `Ready`. Do this only when the log shows `Unauthorized` on `clusters`. The controller recreates the Pod. + + ```bash + kubectl -n cpaas-system delete pod + ``` + +3. If `apollo` logs `bad_response` against Prometheus, recreate the `apollo` Pod. + + > **Warning:** Recreating `apollo` interrupts monitoring queries for every cluster until the Pod is `Ready`. Do this only after the log shows `bad_response` against Prometheus. The controller recreates the Pod; no data is deleted. + + ```bash + kubectl -n cpaas-system delete pod + ``` + +### Cause 3: a certificate alert remains after the expired certificate object was deleted + +1. Run the alert expression in the VictoriaMetrics UI. If no `certificate_expires_time` series is returned for the deleted object while the alert still shows it, the stale state is in `courier`. +2. Restart `courier`, or wait about 4 hours for it to refresh on its own. + + > **Warning:** Deleting the `courier` Pod fails in-flight monitoring and inspection calls and pauses notification sends and the active-alert scrape until the replacement is `Ready`. Do this only when the expired certificate object is already gone and the alert still fires. The controller recreates the Pod. + + ```bash + kubectl -n cpaas-system get pod | grep courier + kubectl -n cpaas-system delete pod + ``` + +3. If the series for the deleted object is still returned, restarting `courier` alone does not help; restart the Cluster Enhancer components `cluster-enhancer-controller` and `cluster-enhancer-apiserver` instead: + + > **Warning:** Certificate monitoring and etcd-backup status for that cluster are unavailable until both Pods are `Ready`. Do this only when the VictoriaMetrics UI still returns the series for a deleted certificate. The Deployments recreate the Pods. + + ```bash + kubectl get deploy -A | grep cluster-enhancer + kubectl -n rollout restart deploy/ deploy/ + ``` + +### Cause 4: apollo cannot find the cluster in its cache + +1. Read the `apollo` log on the `global` cluster: + + ```bash + kubectl -n cpaas-system get pod | grep apollo + kubectl -n cpaas-system logs --tail=300 | grep -Ei 'cache|not found|cluster' + ``` + +2. After a disaster-recovery switchover, first make sure the platform access domain resolves to the standby cluster VIP on the hosts, then check it from inside the cluster: + + ```bash + kubectl exec -it -n cpaas-system deployments/sentry -- nslookup $(kubectl get services -n kube-system kube-dns -o jsonpath='{.spec.clusterIP}') + ``` + + If the in-cluster answer is still the old VIP, recreate the `coredns` Pods in `kube-system` and repeat until it resolves to the standby VIP. Restarting `apollo` before DNS is correct only rebuilds the same stale view. + +3. Recreate the `apollo` Pod. + + > **Warning:** Recreating `apollo` interrupts monitoring queries for every cluster until the new Pod is `Ready`. Do this only when the `apollo` log or the F12 response shows `get cluster from cache error` or a cluster that is not found, and the monitoring Pods on that workload cluster are `Running`. The controller recreates the Pod; no stored data is touched. + + ```bash + kubectl -n cpaas-system delete pod + ``` + +4. If the page is still empty after `apollo` is `Ready`, test port 11780 as described in the related article. + +## Diagnostic Steps + +1. Confirm that collection works: the monitoring Pods are `Running` and vmui inside the workload cluster returns data. + + ```bash + kubectl -n cpaas-system get pod -o wide | grep -Ei 'vmagent|vmselect|vminsert|vmstorage|prometheus|courier|apollo' + ``` + +2. From a `global` node, `curl` port 11780 on the workload VIP. Expected for this article: JSON is returned. A timeout belongs to the related article. + +3. Match the log or F12 response to a cause: + - `cache error` in `courier-api`: Cause 1. + - The cluster's Prometheus was reinstalled shortly before its monitoring went empty, and `global` monitoring is normal: Cause 1. + - `Unauthorized` on `clusters` in `courier`, or `bad_response: invalid character '<'`: Cause 2. + - `get cluster from cache error` or `cluster not found` in `apollo` or in the F12 response: Cause 4, even when port 11780 is open. + +4. A certificate alert that survives the deletion of the expired certificate object: Cause 3. Use the VictoriaMetrics UI result to decide between `courier` and Cluster Enhancer. diff --git a/docs/en/solutions/Monitoring_plugin_install_or_redeploy_fails_ResourcePatch_conflict_IPv6_Service_mkdirOnNode_or_alertmanager-pv-0.md b/docs/en/solutions/Monitoring_plugin_install_or_redeploy_fails_ResourcePatch_conflict_IPv6_Service_mkdirOnNode_or_alertmanager-pv-0.md new file mode 100644 index 000000000..817784d46 --- /dev/null +++ b/docs/en/solutions/Monitoring_plugin_install_or_redeploy_fails_ResourcePatch_conflict_IPv6_Service_mkdirOnNode_or_alertmanager-pv-0.md @@ -0,0 +1,157 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Monitoring plugin install or redeploy fails: ResourcePatch conflict, IPv6 Service, mkdirOnNode, or alertmanager-pv-0 + +## Issue + +Installing, redeploying or running the platform monitoring plugin (Prometheus or VictoriaMetrics) fails in one of these ways: + +- A VictoriaMetrics redeploy fails because the new vmagent objects are not created; a VictoriaMetrics uninstall stays in Uninstalling; or the Prometheus plugin shows `部署中` (deploying) forever. +- The monitoring AppRelease fails with an error that requires an IPv6 address on an IPv4 single-stack cluster. +- VictoriaMetrics install fails in `mkdirOnNode` with an invalid Pod name that contains a space, such as `- -var-cpaas-monitoring-alertmanager`, and `metadata.name: Invalid value` against the RFC 1123 subdomain rule. +- A VictoriaMetrics install waits on an unbound PVC because PV `alertmanager-pv-0` was never created. +- On a standby global cluster with high-availability Prometheus, `prometheus-1` and `prometheus-2` are rebuilt again and again, and an alert reports Prometheus status 0. + +Related article: [Monitoring plugin uninstall or Prometheus to VictoriaMetrics switch leaves ModuleInfo, AppRelease, or Pods on ACP](Monitoring_plugin_uninstall_or_Prometheus_to_VictoriaMetrics_switch_leaves_ModuleInfo_AppRelease_or_Pods_on_ACP.md). + +## Root Cause + +### Cause 1: leftover ResourcePatch (rpch) records conflict with the monitoring objects + +Earlier changes to monitoring components left `rpch` records (sentry resource patches). They conflict with newly deployed objects, so a VictoriaMetrics redeploy does not create the new vmagent resources. The same records block other transitions: a VictoriaMetrics uninstall stays in Uninstalling while `Active` records of release `cpaas-system/victoriametrics` target VMAgent `agent`, VMCluster `cluster` and PersistentVolume `alertmanager-pv-0`. A Prometheus plugin stays in deploying when an `rpch` of release `cpaas-system/kube-prometheus` targets PersistentVolume `prometheus-pv-0` with a `jsonPatch` that adds `/metadata/deletionTimestamp` and `/metadata/deletionGracePeriodSeconds: 0`, so it keeps re-applying a delete to the PV. + +### Cause 2: a Kyverno policy writes IPv6 into Services on an IPv4 cluster + +The cluster has no IPv6. A Kyverno policy mutates `Service` objects and writes an IPv6 `ipFamilies` (or a similar IPv6 field), and the monitoring chart then fails to create its Service. + +### Cause 3: the LocalVolume storage path has a leading space + +The installer builds the temporary directory-init Pod name from the node name and the LocalVolume `storage.path`, and does not strip spaces. A value of `" /var/cpaas/monitoring"` instead of `/var/cpaas/monitoring` produces an invalid name. + +### Cause 4: an old Prometheus PVC still holds the alertmanager-pv name + +An old Prometheus PVC is still bound to a PV that uses the same name pattern, `alertmanager-pv-`, that the VictoriaMetrics deployment wants to create, so `alertmanager-pv-0` is not created and the new PVC stays unbound. + +### Cause 5: the standby global etcd sync overwrites prometheus-1 and prometheus-2 + +In a primary/standby global setup, `etcd-mirror` Pods in `kube-system` sync etcd from the primary to the standby. ConfigMap `etcd-sync-ignore-text` in `kube-system` has ignore entries only for `prometheus-0` (`/registry/secrets/cpaas-system/prometheus-kube-prometheus-0`, `...-0-tls-assets`, `...-0-web-config`, `/registry/monitoring.coreos.com/prometheuses/cpaas-system/kube-prometheus-0`), so the sync keeps overwriting the objects of the other two replicas. This is a known issue. + +## Resolution + +### Cause 1: leftover ResourcePatch (rpch) records conflict with the monitoring objects + +1. Check for Pods stuck in `Terminating`, then list and back up the monitoring `rpch` records on the business cluster: + + ```bash + kubectl get pods -A | grep -E 'vm|prometheus' | grep Terminating + kubectl get rpch -A | grep -Ei 'vm|prometheus' + kubectl get rpch -A -o yaml > /tmp/rpch-backup.yaml + kubectl get rpch -n -o yaml | grep -n -E 'release|kind|name:|deletionTimestamp' + ``` + +2. Delete only the records that belong to the monitoring release and target the objects that are not being created or deleted. + + > **Warning:** Deleting an `rpch` drops that sentry patch; an intentional customization stored only in it is gone on the next reconcile. Precondition: the deploy, redeploy or uninstall is not progressing, and the records belong to VictoriaMetrics or Prometheus monitoring objects. Rollback: `kubectl apply -f /tmp/rpch-backup.yaml`. + + ```bash + kubectl delete rpch -n + ``` + +3. The components are created and reach Running, the uninstall completes, or the plugin changes to `部署成功` (deployed). Retry the redeploy if it does not continue by itself. + +### Cause 2: a Kyverno policy writes IPv6 into Services on an IPv4 cluster + +1. Find the policy whose mutation writes IPv6 into Services: + + ```bash + kubectl get policy,clusterpolicy -A + kubectl -n cpaas-system get svc -o yaml | grep -n -i ipfamil + ``` + +2. Remove that policy, or rewrite its mutation so it no longer injects IPv6. + + > **Warning:** Deleting a Kyverno policy stops that control for every object it matched, not only the monitoring Service. Precondition: the policy YAML is saved and its mutation is the IPv6 field the AppRelease rejected. Rollback: `kubectl apply -f /tmp/kyverno-policy-backup.yaml`. + + ```bash + kubectl get clusterpolicy -o yaml > /tmp/kyverno-policy-backup.yaml + kubectl delete clusterpolicy + ``` + + Use `policy -n ` for a namespaced policy. The next reconcile creates the Service with IPv4 only and the AppRelease becomes Ready. + +### Cause 3: the LocalVolume storage path has a leading space + +The init Pod was rejected by the API, so there is nothing to delete. Correct the LocalVolume storage path in the VictoriaMetrics install parameters (`spec.config.storage.path` on the ModuleInfo) so it has no leading or trailing space, then install again: + +```text +storage.path: /var/cpaas/monitoring +``` + +The default path is `/cpaas/monitoring`; on Alauda OS it is `/var/cpaas/monitoring`. + +### Cause 4: an old Prometheus PVC still holds the alertmanager-pv name + +```bash +kubectl get pvc -A | grep -Ei 'alertmanager|prome' +kubectl get pv | grep alertmanager-pv +``` + +> **Warning:** Deleting the old Prometheus PVC also deletes its PV, and the old Alertmanager data on it is gone. Precondition: the PV `alertmanager-pv-` is bound to a PVC of the earlier Prometheus deployment and the VictoriaMetrics PVC waits for that name. Copy the data off first if it is needed. There is no rollback after the PV is deleted. + +```bash +kubectl -n delete pvc +``` + +The VictoriaMetrics deployment then creates `alertmanager-pv-` itself and the PVC binds. + +### Cause 5: the standby global etcd sync overwrites prometheus-1 and prometheus-2 + +1. Check the ignore list: + + ```bash + kubectl get cm -n kube-system etcd-sync-ignore-text -o yaml | grep prome + ``` + +2. Recommended: reinstall Prometheus on that global cluster as single-node instead of high-availability. + + > **Warning:** Reinstalling Prometheus stops monitoring on that cluster until the new install is Ready, and history is not carried over unless the volumes are reused. Precondition: the rebuilt Pods are `prometheus-1` and `prometheus-2` on the standby global and the ignore list lacks them. Rollback: reinstall in the previous mode. + +3. Alternative: add ignore entries for `prometheus-1` and `prometheus-2` under `data.ignore-equal.txt`, copying the format of the existing `prometheus-0` entries. + + > **Warning:** Editing `etcd-sync-ignore-text` creates an `rpch`, which may block later upgrades. Precondition: same signature as above, and a saved copy. Rollback: `kubectl apply -f /tmp/etcd-sync-ignore-text.yaml`, then remove the generated `rpch`. + + ```bash + kubectl get cm -n kube-system etcd-sync-ignore-text -o yaml > /tmp/etcd-sync-ignore-text.yaml + kubectl edit cm -n kube-system etcd-sync-ignore-text + ``` + +## Diagnostic Steps + +1. Read the module and release state: + + ```bash + kubectl get minfo -A | grep -Ei 'prometheus|victoriametrics' + kubectl -n cpaas-system get ars + kubectl get rpch -A | grep -Ei 'vm|prometheus' + ``` + +2. A redeploy, uninstall or "deploying" state that does not progress, with monitoring `rpch` records present: Cause 1. A record whose `jsonPatch` adds `deletionTimestamp` to a PV is the deploying variant. + +3. An AppRelease error about IPv6 on a single-stack cluster: Cause 2. + +4. `mkdirOnNode` and an RFC 1123 name with a space in it: Cause 3. Expected after the fix: the install proceeds past `mkdirOnNode`. + +5. A PVC waiting for `alertmanager-pv-0` while an old Prometheus PVC holds an `alertmanager-pv-` volume: Cause 4. + +6. `prometheus-1` and `prometheus-2` rebuilt in a loop only on the standby global, with only `prometheus-0` in `etcd-sync-ignore-text`: Cause 5. diff --git a/docs/en/solutions/Monitoring_plugin_uninstall_or_Prometheus_to_VictoriaMetrics_switch_leaves_ModuleInfo_AppRelease_or_Pods_on_ACP.md b/docs/en/solutions/Monitoring_plugin_uninstall_or_Prometheus_to_VictoriaMetrics_switch_leaves_ModuleInfo_AppRelease_or_Pods_on_ACP.md new file mode 100644 index 000000000..7853fc0f2 --- /dev/null +++ b/docs/en/solutions/Monitoring_plugin_uninstall_or_Prometheus_to_VictoriaMetrics_switch_leaves_ModuleInfo_AppRelease_or_Pods_on_ACP.md @@ -0,0 +1,182 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Monitoring plugin uninstall or Prometheus to VictoriaMetrics switch leaves ModuleInfo, AppRelease, or Pods on ACP + +## Issue + +A monitoring plugin is being uninstalled, or a cluster is being moved from Prometheus to VictoriaMetrics, and the platform does not reach a clean state: + +- The Prometheus uninstall never finishes: the global `ModuleInfo` (`minfo`) stays `deleting`, while `prometheus-operator` on the workload cluster is already gone. +- The functional-component page still says Prometheus Operator is abnormal after the cluster moved to VictoriaMetrics, and the namespace shown on the `minfo` contains no operator objects. +- VictoriaMetrics is installed and Running, but a Prometheus AppRelease (`ars`) is still `Syncing`, and Prometheus Pods such as `kube-prometheus-grafana`, `kube-prometheus-thanos-query`, `prometheus-operator` or `cpaas-monitor-prometheus-adapter` remain. +- All VictoriaMetrics Pods are Running, but the VictoriaMetrics `minfo` on global stays `Processing`. +- Installing VictoriaMetrics fails on a taint that says Prometheus is already running, although Marketplace lists Prometheus as not installed; or two VictoriaMetrics instances run on one cluster. + +Plugin state is the global `ModuleInfo` plus the workload-cluster `AppRelease` in `cpaas-system`; `sentry` applies the releases, and `cluster-transformer` recreates a `minfo` that is deleted while it is running. + +Related articles: [Monitoring plugin install or redeploy fails: ResourcePatch conflict, IPv6 Service, mkdirOnNode, or alertmanager-pv-0](Monitoring_plugin_install_or_redeploy_fails_ResourcePatch_conflict_IPv6_Service_mkdirOnNode_or_alertmanager-pv-0.md), [AppRelease abnormal because a ModuleInfo of a deleted cluster remains on ACP](AppRelease_abnormal_because_a_ModuleInfo_of_a_deleted_cluster_remains_on_ACP.md). + +## Root Cause + +### Cause 1: the post-delete job times out on a very large set of PrometheusRules + +The uninstall job and the chart cleanup script walk every `PrometheusRule`. With a very large set the job hits its deadline and retries forever, so the `minfo` stays `deleting`. Pods of the old release can keep running after the AppRelease is gone, and if the `Alertmanager` and `Prometheus` custom resources were not deleted, a `prometheus-operator` that comes back adopts them again and replica counts look wrong. + +### Cause 2: the Prometheus Operator plugin is still installed after the move to VictoriaMetrics + +The `minfo` still points at the old operator namespace, but that namespace has no Prometheus Operator Pod, CSV or PackageManifest (seen on 3.18.2 with VictoriaMetrics live). Restarting `sentry`, editing the `minfo` version, or patching an unrelated AppRelease does not clear it. Deleting the `minfo` while `cluster-transformer` runs makes it come back; deleting it with `cluster-transformer` stopped fails the `vmoduleinfo.kb.io` webhook. Removing the Prometheus Operator plugin itself clears the row. + +### Cause 3: a previous Prometheus uninstall left its AppRelease behind its finalizers + +The VictoriaMetrics `minfo` is `Running`, global has no Prometheus `minfo`, and on the workload cluster the Prometheus `ars` is still `Syncing` or `Ready` with a status saying the kube-prometheus chart is Installing. The earlier uninstall did not finish, and the `ars` stays until its `finalizers` are removed. + +### Cause 4: a leftover Prometheus monitor AppRelease keeps the VictoriaMetrics minfo in Processing + +A Prometheus uninstall got stuck during an earlier upgrade; its `minfo` was deleted by hand but its AppReleases were not. VictoriaMetrics was then deployed on top of that state, so its `minfo` never leaves `Processing`, and a normal VictoriaMetrics uninstall cannot complete either. + +### Cause 5: Prometheus and VictoriaMetrics cannot run side by side + +The platform Prometheus and VictoriaMetrics share the same node-exporter, so they are not allowed to coexist and the install conflict is expected. After an upgrade from 3.18 to 4.1, the Prometheus Pods were still running while Marketplace listed Prometheus as not installed. Two platform VictoriaMetrics instances on one 4.2 global cluster made the data services page show wrong data. A second, self-installed open-source VictoriaMetrics next to the platform one uses the same upstream CRDs and may cause data conflicts; it is not recommended. + +## Resolution + +### Cause 1: the post-delete job times out on a very large set of PrometheusRules + +1. Count the rules and read the module state: + + ```bash + kubectl get prometheusrule -A --no-headers | wc -l + kubectl get moduleinfo -A | grep -i prometheus + kubectl -n cpaas-system get pod | grep -Ei 'prometheus|alertmanager|sentry' + ``` + +2. Back up the rules off the cluster, then delete them per namespace that holds a large share. + + > **Warning:** Deleting `PrometheusRule` objects removes those alerts until they are restored, and the backup file is the only rollback. Precondition: `/tmp/prometheusrule-backup.yaml` is copied off the cluster, and missing alerts are acceptable for the window. Do not delete `cpaas-cluster-rules` unless it is part of the blocking set and is in the backup. Rollback: `kubectl apply -f /tmp/prometheusrule-backup.yaml`. + + ```bash + kubectl get prometheusrule -A -o yaml > /tmp/prometheusrule-backup.yaml + kubectl delete prometheusrule -n --all + ``` + +3. Let `sentry` retry the post-delete job. If it does not retry, recreate the `sentry` Pod. + + > **Warning:** Recreating `sentry` pauses module installs and uninstalls platform-wide until it is Ready. Precondition: the rule backlog is already deleted, or the job times out again. Rollback is the controller starting a new Pod. + + ```bash + kubectl get pod -A | grep sentry + kubectl -n delete pod + ``` + +4. Remove leftover operator objects so a later install does not adopt them. + + > **Warning:** Deleting the `Prometheus` and `Alertmanager` resources tears down their StatefulSets and stops scraping and notification. Precondition: this is the uninstall you intended, and the data on the PVC is expendable or copied. Rollback: reinstall the plugin, which creates new resources. + + ```bash + kubectl -n cpaas-system get prometheus,alertmanager + kubectl -n cpaas-system delete prometheus + kubectl -n cpaas-system delete alertmanager + ``` + +### Cause 2: the Prometheus Operator plugin is still installed after the move to VictoriaMetrics + +```bash +kubectl get minfo -A | grep -i prometheus +kubectl get pod,csv,packagemanifest -A | grep -i prometheus +kubectl -n cpaas-system get pod | grep -Ei 'cluster-transformer|sentry' +``` + +> **Warning:** Removing the Prometheus Operator plugin deletes any operands it still has. Precondition: VictoriaMetrics already serves the cluster, and any `PrometheusRule` you still need is backed up (Cause 1). Rollback: install the plugin again from the catalog. + +Uninstall the Prometheus Operator plugin through the platform with `cluster-transformer` left running; do not delete the `minfo` by hand. If a VictoriaMetrics install in the same window fails on a `webhook.cert-manager.io` `context deadline exceeded` timeout, repair the cert-manager webhook before retrying. + +### Cause 3: a previous Prometheus uninstall left its AppRelease behind its finalizers + +1. Confirm VictoriaMetrics is the live stack and the Prometheus `ars` is the leftover: + + ```bash + kubectl get minfo -A | grep -Ei 'prometheus|victoriametrics' + kubectl -n cpaas-system get ars + kubectl get pods -A -o wide | grep prometheus | grep -iv exporter + ``` + +2. Save the `ars`, clear its finalizers and delete it. + + > **Warning:** This drops the leftover Prometheus release, including its Grafana, Thanos query, operator and adapter, and any scraping that still depended on it. Precondition: the VictoriaMetrics `minfo` is `Running`, no Prometheus `minfo` exists, and the `ars` shows the old kube-prometheus chart Installing. Rollback after a successful delete: install the Prometheus plugin again; old Prometheus data is reattached only if its PVC still exists. + + ```bash + kubectl -n cpaas-system get ars -o yaml > /tmp/prometheus-ars.yaml + kubectl -n cpaas-system patch ars --type=json -p '[{"op":"remove","path":"/metadata/finalizers"}]' + kubectl -n cpaas-system delete ars --wait=false + ``` + +3. Delete the leftover Prometheus Pods and Jobs from the list (for example `cpaas-monitor-prometheus-adapter`, `kube-prometheus-grafana`, `kube-prometheus-thanos-query`, `prometheus-operator`). Skip every name that belongs to the VictoriaMetrics release. + + > **Warning:** Deleting them stops the old Prometheus query path and adapter. Precondition: each name is in the leftover list and vmui already returns samples. Rollback: reinstall Prometheus. + + ```bash + kubectl -n cpaas-system delete pod + ``` + +### Cause 4: a leftover Prometheus monitor AppRelease keeps the VictoriaMetrics minfo in Processing + +1. Identify the leftover monitor `ars` created by the earlier Prometheus install: + + ```bash + kubectl get minfo -A | grep -Ei 'victoriametrics|prometheus' + kubectl -n cpaas-system get ars + kubectl -n cpaas-system get ars -o yaml | grep -n -i -E 'chart|prometheus' + ``` + +2. Save and delete the VictoriaMetrics `minfo` (on global), the VictoriaMetrics `ars` and the leftover monitor `ars` (on the workload cluster), then deploy VictoriaMetrics again. + + > **Warning:** This removes VictoriaMetrics monitoring of that cluster until it is redeployed; data on a retained volume is reattached only if the same PVC is reused. Precondition: the VictoriaMetrics `minfo` is stuck in `Processing`, a Prometheus monitor `ars` predates it, and a normal uninstall cannot complete. Rollback: redeploy the plugin; the saved YAML is for reference, not for reapplying the stuck state. + + ```bash + kubectl get minfo -o yaml > /tmp/vm-minfo.yaml + kubectl -n cpaas-system get ars -o yaml > /tmp/vm-ars.yaml + kubectl -n cpaas-system get ars -o yaml > /tmp/monitor-ars.yaml + kubectl delete minfo + kubectl -n cpaas-system delete ars + ``` + + If an object hangs on finalizers, handle it as in Cause 3. + +### Cause 5: Prometheus and VictoriaMetrics cannot run side by side + +```bash +kubectl get minfo -A | grep -Ei 'prometheus|victoriametrics' +kubectl get ds -A | grep -i exporter-node +``` + +Run one platform monitoring solution per cluster. Switch by uninstalling the current plugin through the platform before installing the other; clean up leftovers of an incomplete switch with Causes 3 and 4 rather than deleting the Prometheus DaemonSet and Pods by hand, which left the overview without cluster metrics. For two platform VictoriaMetrics instances on global, uninstall the duplicate. For a business-owned second VictoriaMetrics, do not install it on a cluster that already runs the platform one. + +## Diagnostic Steps + +1. Read the module phase on global and the AppReleases on the workload cluster before restarting anything: + + ```bash + kubectl get minfo -A + kubectl -n cpaas-system get ars + ``` + +2. `deleting` plus a large `PrometheusRule` count: Cause 1. + +3. An abnormal Prometheus Operator row with an empty namespace while VictoriaMetrics is live: Cause 2. + +4. VictoriaMetrics `minfo` `Running`, no Prometheus `minfo`, and a Prometheus `ars` still Installing: Cause 3. + +5. VictoriaMetrics `minfo` stuck `Processing` with a leftover Prometheus monitor `ars`: Cause 4. + +6. A taint that says Prometheus is running, or two VictoriaMetrics installs: Cause 5. Expected after cleanup: exactly one monitoring `minfo` per cluster, `Running`. diff --git a/docs/en/solutions/Moving_Service_Mesh_v2_tracing_storage_from_Elasticsearch_to_OpenSearch_with_Jaeger_v2.md b/docs/en/solutions/Moving_Service_Mesh_v2_tracing_storage_from_Elasticsearch_to_OpenSearch_with_Jaeger_v2.md new file mode 100644 index 000000000..70d0065d1 --- /dev/null +++ b/docs/en/solutions/Moving_Service_Mesh_v2_tracing_storage_from_Elasticsearch_to_OpenSearch_with_Jaeger_v2.md @@ -0,0 +1,205 @@ +--- +kind: + - How To +products: + - Alauda Service Mesh +ProductsVersion: + - "4.2" + - "4.3" + - "4.4" +--- + +# Moving Service Mesh v2 tracing storage from Elasticsearch to OpenSearch with Jaeger v2 + +## Issue + +A Service Mesh v2 tracing pipeline stores traces in Elasticsearch and has to move to OpenSearch. The questions are whether the move affects business traffic, which component versions fit together, and what the steps are. Related questions: + +- Is Alauda Build of Jaeger v2 now deployed independently of ASM, and how does the mesh send traces to it? +- Which Elasticsearch version does the tracing feature support, for example whether an existing ES7 can be used? +- Must the host that runs the Jaeger v2 installation have `jq`? + +The layout described for Cause 1 is that of ACP 4.2.2: an OpenTelemetry Collector sends traces to a Jaeger v2 service in `jaeger-system` that writes to Elasticsearch. The Alauda Build of OpenTelemetry v2 operator (adapted to ACP 4.2) deploys Jaeger v2; OpenSearch Operator v2.8 from the App Marketplace deploys OpenSearch 3.x. + +## Root Cause + +### Cause 1: the tracing pipeline uses the legacy mesh integration with Elasticsearch storage + +Moving to OpenSearch means deploying a Jaeger v2 instance integrated with OpenSearch and pointing the OpenTelemetry Collector at it. Switching the backend does not affect business traffic; a small amount of trace data can be lost during the switch. + +### Cause 2: Jaeger v2 is the standalone product Alauda Distributed Tracing + +Alauda Build of Jaeger v2 is installed and run as the standalone product Alauda Distributed Tracing; it is no longer part of ASM. ASM only exports Istio spans to the deployed Jaeger v2 and OpenTelemetry Collector. One tracing stack can serve both the mesh and other workloads in the cluster; its default namespace is `jaeger-system`. The old stack, Alauda Build of Jaeger (Jaeger 1.60.0) plus Alauda Build of OpenTelemetry, is deprecated in ACP 4.4 and will be removed; the old Distributed Tracing pages are removed in ACP 4.4, with ACP 4.3 as the last version that contains them. + +### Cause 3: Elasticsearch 8.x is the supported version + +The Alauda Distributed Tracing documentation (Configuration > Storage Backend > Elasticsearch) lists Elasticsearch 8.x as supported. Jaeger detects the version from the Elasticsearch root endpoint and then uses the matching index mappings and REST APIs; the version can also be set explicitly with the `version` property of the Elasticsearch storage configuration. ES7 is not listed as supported. + +### Cause 4: the Jaeger v2 installation depends on `jq` + +The Jaeger v2 installation depends on `jq`, so the host that runs the installation must have it. + +## Resolution + +### Cause 1: the tracing pipeline uses the legacy mesh integration with Elasticsearch storage + +Install OpenSearch Operator v2.8 from the App Marketplace and create an OpenSearch 3.x cluster. Back up the Collector configuration: + +```bash +kubectl get opentelemetrycollector -A +kubectl -n get opentelemetrycollector -o yaml > /tmp/otel-collector.bak.yaml +kubectl -n jaeger-system get all +``` + +Then choose one of the three paths. + +#### Path A: historical traces are not needed + +1. Deploy a new Jaeger v2 service integrated with OpenSearch, following the Jaeger v2 deployment procedure of the OpenTelemetry v2 operator. Set `JAEGER_INSTANCE_NAME` to a value other than `jaeger`, for example `jaeger-opensearch`, so it does not collide with the old instance. +2. Point the Collector exporter at the new service; copy TLS and other settings from the exporter that points at the old Jaeger. + + ```yaml + spec: + config: + exporters: + otlp/jaeger-opensearch: + endpoint: ..svc:4317 + service: + pipelines: + traces: + exporters: + - otlp/jaeger-opensearch + ``` + + > **Warning:** The Collector restarts and spans in flight can be lost. Only when the new Jaeger v2 is Running and connected to OpenSearch, and the backup exists. Rollback: `kubectl apply -f /tmp/otel-collector.bak.yaml`. + + ```bash + kubectl -n edit opentelemetrycollector + kubectl -n get pods -w + ``` + +3. Confirm new traces in the new Jaeger UI, then uninstall the old Jaeger v2 service in `jaeger-system` with the uninstall step of the procedure that deployed it. + + > **Warning:** Traces stored through the old Jaeger can no longer be queried once it is removed. Only after new traces are confirmed and only when historical traces are not needed. Rollback: reinstall the old Jaeger against Elasticsearch to read old traces. + +#### Path B: historical traces must be kept for a period + +Deploy the new Jaeger v2 as in Path A. Add a second traces pipeline so data goes to both services; keep the receivers and processors of the existing pipeline: + +```yaml +spec: + config: + exporters: + otlp/jaeger: + endpoint: .jaeger-system.svc:4317 + otlp/jaeger-opensearch: + endpoint: ..svc:4317 + service: + pipelines: + traces: + receivers: [otlp] + processors: [batch] + exporters: [otlp/jaeger] + traces/opensearch: + receivers: [otlp] + processors: [batch] + exporters: [otlp/jaeger-opensearch] +``` + +Apply it with `kubectl edit` under the same Warning as Path A step 2. After the retention period, remove the `traces` pipeline and the `otlp/jaeger` exporter, and uninstall the old service under the Warning of Path A step 3. + +#### Path C: test environment where a short trace gap is acceptable + +Uninstall the old Jaeger v2 first, then deploy the new one with `JAEGER_INSTANCE_NAME=jaeger`, integrated with OpenSearch. Because the name stays `jaeger`, the Collector exporter can stay unchanged; confirm the Service it points to exists after the new instance is Ready. + +> **Warning:** No traces are stored between the uninstall and the new instance becoming Ready, and old traces are no longer reachable. Test environments only. Rollback: reinstall Jaeger v2 against Elasticsearch to read old traces. + +```bash +kubectl -n jaeger-system get opentelemetrycollector,deploy,svc +kubectl -n jaeger-system get svc -o custom-columns='NAME:.metadata.name,PORTS:.spec.ports[*].port' +``` + +### Cause 2: Jaeger v2 is the standalone product Alauda Distributed Tracing + +1. For a new deployment, follow the current Alauda Distributed Tracing installation guide for the backend you use (Elasticsearch or OpenSearch) instead of the deprecated tracing documents; existing deployments follow its migration guide. The order is: + 1. Cluster plugin Alauda Build of Jaeger v2. + 2. OpenTelemetry v2 Operator. + 3. Deploy Jaeger v2. + 4. Deploy the Collector. + + ```bash + kubectl get pods -n jaeger-system + kubectl get svc -n jaeger-system | grep -E 'otel-collector|jaeger' + ``` + +2. To send mesh traces, add an `extensionProviders` entry on the `Istio` resource that points at the Collector, and reference it from a `Telemetry` resource. The provider name is an example and the sampling percentage is a site choice. If the cluster DNS domain is not `cluster.local`, use the cluster's domain in the service name. + + > **Warning:** istiod pushes new configuration to every proxy, and a `Telemetry` in `istio-system` changes tracing for the whole mesh. Copy existing `extensionProviders` entries into the patch, because a merge patch replaces the whole list. Rollback: `kubectl apply -f /tmp/istio-tracing.bak.yaml` (remove `metadata.resourceVersion` on conflict) and `kubectl delete telemetry -n istio-system mesh-default`. + + ```bash + kubectl get istio -o yaml > /tmp/istio-tracing.bak.yaml + kubectl get istio -o jsonpath='{.spec.values.meshConfig.extensionProviders}{"\n"}' + kubectl patch istio --type=merge -p '{"spec":{"values":{"meshConfig":{"extensionProviders":[{"name":"otel-tracing","opentelemetry":{"service":"otel-collector.jaeger-system.svc.cluster.local","port":4317}}]}}}}' + ``` + + ```yaml + apiVersion: telemetry.istio.io/v1 + kind: Telemetry + metadata: + name: mesh-default + namespace: istio-system + spec: + tracing: + - providers: + - name: otel-tracing + randomSamplingPercentage: 10.0 + ``` + + ```bash + kubectl apply -f mesh-default-telemetry.yaml + ``` + +3. To remove the mesh integration later, remove the `Istio` and `Telemetry` configuration first; leave Jaeger and the Collector in place while other workloads still use tracing. + +### Cause 3: Elasticsearch 8.x is the supported version + +Use Elasticsearch 8.x. If only ES7 is available, it can be tested, but it is not a supported combination. Read the version Jaeger will detect: + +```bash +curl -s -k -u ':' https://:9200/ | grep '"number"' +``` + +### Cause 4: the Jaeger v2 installation depends on `jq` + +Install `jq` on the host that runs the installation, then rerun the step: + +```bash +command -v jq || echo 'jq is missing' +yum install -y jq # or: apt-get install -y jq; use an internal mirror on an offline host +jq --version +``` + +## Diagnostic Steps + +1. Confirm the layout before starting: + + ```bash + kubectl get csv -A | grep -i -E 'opentelemetry|opensearch' + kubectl get pods -n jaeger-system + kubectl -n get opentelemetrycollector -o jsonpath='{.spec.config}{"\n"}' + ``` + +2. After the switch, check that the Collector exports without errors and the new Jaeger receives data: + + ```bash + kubectl -n logs --tail=200 | grep -i -E 'error|export' + kubectl -n logs --tail=200 | grep -i -E 'opensearch|error' + ``` + +| What you see | Next step | +| --- | --- | +| Collector logs export errors to the new endpoint | Check the Service name and port in the exporter | +| New Jaeger logs OpenSearch connection errors | Check the OpenSearch address and credentials given to Jaeger | +| No mesh spans in the standalone Jaeger v2 | Cause 2: check `extensionProviders` on `Istio` and the `Telemetry` provider name | +| Elasticsearch root endpoint reports a 7.x `number` | Cause 3 | +| Installation stops because `jq` is not found | Cause 4 | diff --git a/docs/en/solutions/Multus_secondary_networks_unexpected_end_of_JSON_input_duplicated_addresses_or_several_addresses_on_one_NIC_on_ACP.md b/docs/en/solutions/Multus_secondary_networks_unexpected_end_of_JSON_input_duplicated_addresses_or_several_addresses_on_one_NIC_on_ACP.md new file mode 100644 index 000000000..20dda4b62 --- /dev/null +++ b/docs/en/solutions/Multus_secondary_networks_unexpected_end_of_JSON_input_duplicated_addresses_or_several_addresses_on_one_NIC_on_ACP.md @@ -0,0 +1,84 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "3.18" + - "4.0" + - "4.2" + - "4.3" +--- + +# Multus secondary networks: unexpected end of JSON input, duplicated addresses or several addresses on one NIC on ACP + +## Issue + +A Pod with secondary networks attached through Multus (`k8s.v1.cni.cncf.io/networks` and `NetworkAttachmentDefinition`) does not get the NICs or addresses that were intended: + +- The Pod is scheduled, then sandbox creation fails, and Multus reports `failed to unmarshal Spec.Config: unexpected end of JSON input` for `multus-cni-network`. +- The Pod should get three VLAN addresses and instead shows two identical ones. +- A multi-homed Pod should receive several addresses from the same subnet on one NIC, and no annotation makes Kube-OVN do it. + +Related articles: "Kube-OVN Pods fail with AddressOutOfRange, AddressConflict or no address allocated on ACP"; "Calico Pods get no address: disabled default IPPool, Raven fixed IP mismatch, reserved IPs or deleted raven on ACP". + +## Root Cause + +### Cause 1: failed to unmarshal Spec.Config: unexpected end of JSON input + +`GetCNIConfig` cannot parse the NetworkAttachmentDefinition: its `spec.config` (or the annotation that carries it) is truncated or is not valid JSON. Every new Pod that references the attachment fails the same way; Pods that already started keep their NIC. + +### Cause 2: k8s.v1.cni.cncf.io/networks lists the same attachment twice + +The annotation repeats one attachment and omits another, for example `default/vlan161,default/vlan161` instead of `default/vlan161,default/vlan162`. IPAM does what the annotation asks, so the Pod gets two addresses from the same network. + +### Cause 3: one NIC cannot be given several addresses in the same subnet automatically + +Kube-OVN and OVN cannot allocate several addresses from one subnet onto a single NIC of a Pod, and there is no annotation that turns it on. This is a current product gap, not a misconfigured Subnet. + +## Resolution + +### Cause 1: failed to unmarshal Spec.Config: unexpected end of JSON input + +Replace `spec.config` with a complete JSON document, validate it before updating the object, then delete the failed Pod so Multus reads the new config. + +> **Warning:** The replacement must be the full JSON string; there is no partial update. Save the object first and re-apply the saved copy to roll back. + +```bash +kubectl get network-attachment-definitions.k8s.cni.cncf.io -A +kubectl get network-attachment-definitions.k8s.cni.cncf.io -n -o yaml > /root/nad.before.yaml +python3 -m json.tool < /tmp/nad-config.json > /dev/null && echo valid +kubectl edit network-attachment-definitions.k8s.cni.cncf.io -n +kubectl -n delete pod +``` + +For a safe way to rewrite the field, see "Updating a NetworkAttachmentDefinition's spec.config JSON Safely". + +### Cause 2: k8s.v1.cni.cncf.io/networks lists the same attachment twice + +Set the annotation to the distinct attachments, in the order of the addresses you intend, and let the Pod be recreated. + +> **Warning:** Patching the Pod template rolls the workload. + +```bash +kubectl patch deploy -n --type=merge -p '{"spec":{"template":{"metadata":{"annotations":{"k8s.v1.cni.cncf.io/networks":"default/vlan160,default/vlan161,default/vlan162"}}}}}' +``` + +Use your own NetworkAttachmentDefinition names; the example shows each VLAN listed once. + +### Cause 3: one NIC cannot be given several addresses in the same subnet automatically + +Give each address its own NIC and its own NetworkAttachmentDefinition, or use one address. If a single NIC must carry several addresses from one subnet, raise a product requirement; there is no cluster-side switch. + +## Diagnostic Steps + +```bash +kubectl -n describe pod | grep -A5 -E 'FailedCreatePodSandBox|multus' +kubectl -n get pod -o jsonpath='{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/networks}{"\n"}' +kubectl -n exec -- ip -br addr +``` + +- `failed to unmarshal Spec.Config: unexpected end of JSON input`: Cause 1. +- The same attachment name twice in the annotation and two identical addresses in the Pod: Cause 2. +- A request for several same-subnet addresses on one NIC: Cause 3. diff --git a/docs/en/solutions/NFS_PVC_stays_Pending_csi-nfs-controller_mount_fails_with_rpc.statd_exit_status_32_or_access_denied_on_ACP.md b/docs/en/solutions/NFS_PVC_stays_Pending_csi-nfs-controller_mount_fails_with_rpc.statd_exit_status_32_or_access_denied_on_ACP.md new file mode 100644 index 000000000..25a4c23f3 --- /dev/null +++ b/docs/en/solutions/NFS_PVC_stays_Pending_csi-nfs-controller_mount_fails_with_rpc.statd_exit_status_32_or_access_denied_on_ACP.md @@ -0,0 +1,245 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# NFS PVC stays Pending: csi-nfs-controller mount fails with rpc.statd, exit status 32, or access denied on ACP + +## Issue + +A PVC on an NFS StorageClass (provisioner `nfs.csi.k8s.io`) stays `Pending`, or the console keeps showing it as `处理中` (Processing), and no PV is created. `kubectl describe pvc` shows: + +```text +ExternalProvisioning Waiting for a volume to be created either by the external provisioner 'nfs.csi.k8s.io' or manually by the system administrator +Provisioning External provisioner is provisioning volume for claim "/" +ProvisioningFailed failed to provision volume with StorageClass "": rpc error: code = Internal desc = failed to mount nfs server: ... +``` + +The NFS plugin Pods are Running (`csi-nfs-controller` 4/4, `csi-nfs-node` 3/3), and often `showmount -e` lists the export and a manual NFS mount on a node works. The `nfs` container log of `csi-nfs-controller` shows the mount failing with one of: + +- `rpc.statd is not running but is required for remote locking`, `cannot create /run/rpc.statd.lock`, `Mount failed: exit status 32` +- `mount.nfs: access denied by server while mounting :` + +Sometimes the PVC is created only after the NFS version in the StorageClass is changed, or after `nolock` is added to its `mountOptions`. + +At `CreateVolume`, `csi-nfs-controller` mounts the export itself, on the node where it runs, with the StorageClass `server`, `share`, and `mountOptions`, and then creates a per-volume directory. A mount that works by hand from another node, or with other options, does not prove that this mount works. + +Related articles: *NFS PVC stays Pending: csi-nfs-controller mount times out or mkdir of the volume directory is denied on ACP* (the mount times out, or succeeds and `mkdir` fails). For the StorageClass fields see [Dynamic NFS storage provisioning for Tekton CICD on ACP](Dynamic_NFS_storage_provisioning_for_Tekton_CICD_on_ACP.md). + +## Root Cause + +`access denied by server while mounting` has four causes here (Causes 1, 2, 3, and 4); the table in Diagnostic Steps separates them. + +### Cause 1: the NFSv3 mount needs rpc.statd, and it is not running on the node + +An NFSv3 mount without `nolock` needs `rpc.statd` for remote locking. The CSI container cannot start it itself, because its `/run` is read-only: + +```text +/usr/sbin/start-statd: 10: cannot create /run/rpc.statd.lock: Read-only file system +mount.nfs: rpc.statd is not running but is required for remote locking. +mount.nfs: Either use '-o nolock' to keep locks local, or start statd. +Mount failed: exit status 32 +GRPC error: rpc error: code = Internal desc = failed to mount nfs server: rpc error: code = Internal desc = mount failed: exit status 32 +``` + +The node that hosts `csi-nfs-controller` has `rpc-statd` stopped, so the mount fails. The same root problem has three other forms: + +- An NFSv3 StorageClass whose `mountOptions` omit `nolock`: the manual mount with `nolock` works, the PVC never gets a PV, and adding `nolock` to the class restores provisioning. +- A NAS that must stay on NFSv3 (for example a Huawei NAS through the default NFS CSI plugin): NFS 4.0 and 4.1 work through the plugin and NFS 3.0 fails, while a manual v3 mount on the node succeeds. Installing the NFS client utilities and running `rpcbind` and `rpc-statd` on the nodes fixes it. +- The provisioner reports `failed to get nfs info from nfs host` and `access denied by server while mounting`, the server is reachable, a manual `mount -t nfs -o hard,nfsvers=3,nolock` succeeds, and `rpc-statd.service` is inactive. Restarting and enabling `rpc-statd.service`, then connecting the StorageClass again, restores PVC creation. + +### Cause 2: the StorageClass NFS version is one the server does not serve + +The `nfsvers` in the StorageClass `mountOptions` is used by `csi-nfs-controller` for its own mount. If the server does not serve that version, every `CreateVolume` fails, and changing the version on the StorageClass makes the next PVC succeed. Examples: + +- The server does not support NFSv4 at all and the class uses v4. +- The server runs v4.1 and the storage was integrated as v3 (`nfsvers=3`, `nolock`). The controller reports `access denied by server while mounting`, and a manual v3 mount shows: + + ```text + mount -t nfs -o hard,nfsvers=3,nolock : /test -v + mount.nfs: portmap query retrying: RPC: Program not registered + mount.nfs: mount(2): Permission denied + mount.nfs: access denied by server while mounting : + ``` + + The same mount without the v3 option succeeds, and `showmount -e` still lists the export. Integrating the NFS storage again with version v4.1 selected makes provisioning and mounting work. + +The platform supports NFS protocol versions `v3`, `v4.0`, and `v4.1`. + +### Cause 3: the StorageClass share path has no leading / + +The `share` parameter of the StorageClass is written as a relative name (for example `sfs-example-0001`) instead of the export path (`/sfs-example-0001`). `csi-nfs-controller` mounts exactly that string, so the server refuses an export it does not have: + +```text +failed to provision volume with StorageClass "": rpc error: code = Internal desc = failed to mount nfs server: rpc error: code = Internal desc = mount failed: exit status 32 +Mounting arguments: -t nfs -o hard,nfsvers=3,nolock :sfs-example-0001 /tmp/pvc- +Output: mount.nfs: access denied by server while mounting :sfs-example-0001 +``` + +`rpc-statd` can be active on every node, and a manual mount of `:/sfs-example-0001` (with the slash) succeeds on the controller node. On some NAS devices no error text is shown at all; the PVC simply stays Pending until `share` is corrected. + +### Cause 4: the NAS authorizes only some nodes, not the node running csi-nfs-controller + +Mount permission on the NAS was granted to the worker nodes, not to the control-plane nodes (or to some other subset). Dynamic provisioning is done by `csi-nfs-controller`, which mounts the export from whichever node it runs on. By default it is not pinned to a fixed node (its `nodeSelector` is `kubernetes.io/os: linux`), so it can land on a node that has no mount permission on the NAS, and PVC creation fails with a permission error right after the StorageClass is created. + +## Resolution + +### Cause 1: the NFSv3 mount needs rpc.statd, and it is not running on the node + +Use one of two repairs: run `rpc-statd` on the nodes, or add `nolock` to the StorageClass. Use the service when the NAS has to stay on NFSv3 with locking; use `nolock` when the error suggests it or the service cannot be enabled. + +**Run rpc-statd on the nodes.** Run this on every node that can run `csi-nfs-controller` or a Pod that uses the volume. If `rpc.statd` is not installed, install the node OS package that provides `rpcbind` and `rpc-statd` (the NFS client utilities) first. + +> **Warning:** `rpcbind` and `rpc-statd` listen on the node and take part in NFSv3 locking; restarting `rpc-statd` briefly interrupts NFSv3 lock recovery on that node. Do this only when a Cause 1 signature is confirmed and the unit is inactive. Roll back with `sudo systemctl disable --now rpc-statd rpcbind`. + +```bash +systemctl status rpc-statd rpcbind +sudo systemctl enable --now rpcbind +sudo systemctl enable --now rpc-statd +rpcinfo -p 127.0.0.1 +``` + +`rpcinfo` should list program `100000` (portmapper) and program `100024` (status). In the `failed to get nfs info from nfs host` form, also connect the StorageClass to the NFS storage again after the service is running. + +**Or add nolock to the StorageClass.** Kubernetes stores mount options without the `-o` flag. + +```bash +kubectl edit storageclass +``` + +```yaml +mountOptions: + - hard + - nfsvers=3 + - nolock +``` + +`mountOptions` on an existing class can be changed; `parameters` cannot. Existing PVs keep the options they were provisioned with. + +**Then recreate the Pending claim.** + +> **Warning:** Deleting a PVC whose PV uses reclaim policy `Delete` makes the provisioner remove the backend directory. Do this only when the claim is still Pending, `spec.volumeName` is empty, and no application data was written. A directory the provisioner has already removed cannot be restored; if `spec.volumeName` is set, stop and check the reclaim policy first. + +```bash +kubectl -n get pvc -o jsonpath='{.spec.volumeName}{"\n"}' +kubectl -n get pvc -o yaml > "$HOME/.yaml" +kubectl -n delete pvc +``` + +Create it again from your manifest (remove `metadata.uid`, `metadata.resourceVersion`, and `status` if you reuse the saved copy). + +### Cause 2: the StorageClass NFS version is one the server does not serve + +1. From the node that runs `csi-nfs-controller`, try the class's version and another version: + + ```bash + kubectl get storageclass -o jsonpath='{.mountOptions}{"\n"}' + sudo mount -t nfs -o nfsvers=4.1 : /mnt && sudo umount /mnt + sudo mount -t nfs -o nfsvers=3,nolock : /mnt && sudo umount /mnt + rpcinfo -p + ``` + + `portmap query retrying: RPC: Program not registered` on the v3 mount means the server does not offer NFSv3 to this client. A server that serves only NFSv4 may not answer `rpcinfo` at all. On the NFS server, `nfsstat -s` shows which versions it serves. + +2. Set the version that mounted in the StorageClass `mountOptions` (for a server without NFSv4, `nfsvers=3` plus `nolock` as in Cause 1), or integrate the NFS storage again with that version selected in the **NFS Protocol Version** field of the StorageClass form. Then create the PVC again. + + ```bash + kubectl edit storageclass + ``` + +### Cause 3: the StorageClass share path has no leading / + +1. Read the class and the controller's mount arguments: + + ```bash + kubectl get storageclass -o jsonpath='{.parameters.server}{" "}{.parameters.share}{"\n"}' + kubectl -n cpaas-system logs deploy/csi-nfs-controller -c nfs --tail=500 | grep -E 'Mounting arguments|access denied' + showmount -e + ``` + + A `share` without a leading `/`, or mount arguments with no `/` after the colon, matches this cause. `showmount -e` lists the export paths exactly as the server publishes them (NFSv3 servers; an NFSv4-only server may not answer). + +2. StorageClass `parameters` cannot be updated in place, so recreate the class under the same name with the corrected `share`. + + > **Warning:** While the class is deleted, new PVCs that name it cannot be provisioned. Deleting a StorageClass does not remove existing PVs or PVCs. Do this only when the Cause 3 signature is confirmed. Back up the class first; roll back with `kubectl apply -f /tmp/.yaml`. + + ```bash + kubectl get storageclass -o yaml > /tmp/.yaml + cp /tmp/.yaml /tmp/-fixed.yaml + vi /tmp/-fixed.yaml + kubectl delete storageclass + kubectl apply -f /tmp/-fixed.yaml + ``` + + In the copy, set `share: /` and remove `metadata.uid`, `metadata.resourceVersion`, `metadata.creationTimestamp`, and `metadata.managedFields`. + +3. If a claim still has no PV after a few minutes, recreate it as in Cause 1 (only while it holds no data). + +### Cause 4: the NAS authorizes only some nodes, not the node running csi-nfs-controller + +1. Find the node that runs the controller and compare it with the NAS client allowlist: + + ```bash + kubectl get pods -A -o wide | grep csi-nfs-controller + ``` + +2. Grant NAS mount permission (the client allowlist or export rule) to every node in the cluster. This is the recommended fix, because `csi-nfs-controller` can be rescheduled to any node, and Pods mount NFS through `csi-nfs-node` on whichever node they run. + +3. If new nodes should not need NAS permission, pin `csi-nfs-controller` to a labeled set of nodes that the NAS authorizes. Restarting or recreating the `csi-nfs-controller` Pod does not affect existing PVCs or PVs. + + > **Warning:** Saving the Deployment restarts `csi-nfs-controller`; provisioning pauses until it is Ready, and stays stopped if no labeled node is schedulable. A plugin reinstall or upgrade can drop the edit. Do this only when the Cause 4 signature is confirmed and every labeled node can mount the export. Back up the Deployment first; roll back with `kubectl -n cpaas-system apply -f /tmp/csi-nfs-controller.yaml`, or set the `nodeSelector` back to `kubernetes.io/os: linux`. + + ```bash + kubectl label node storage.example/nas-access=allowed + kubectl -n cpaas-system get deploy csi-nfs-controller -o yaml > /tmp/csi-nfs-controller.yaml + kubectl -n cpaas-system edit deploy csi-nfs-controller + ``` + + ```yaml + spec: + template: + spec: + nodeSelector: + storage.example/nas-access: allowed + ``` + + `storage.example/nas-access` is a placeholder label key; every node that runs NFS workloads still needs NAS permission. + +## Diagnostic Steps + +1. Read the claim's events, the class, and the controller log: + + ```bash + kubectl -n describe pvc + kubectl get storageclass -o yaml + kubectl -n cpaas-system get pods -o wide | grep csi-nfs + kubectl -n cpaas-system logs deploy/csi-nfs-controller -c nfs --tail=200 + ``` + +2. On the node that runs `csi-nfs-controller`, repeat the controller's mount by hand with the class's `server`, `share`, and options: + + ```bash + systemctl status rpc-statd rpcbind + sudo mount -t nfs -v -o hard,nfsvers=3,nolock : /mnt && sudo umount /mnt + ``` + +3. Match the signature: + + | Controller log or manual test | Cause | + | --- | --- | + | `rpc.statd is not running`, `cannot create /run/rpc.statd.lock`, `exit status 32` | 1 | + | `access denied by server`; a manual mount with `nolock` works; `rpc-statd` inactive | 1 | + | `portmap query retrying: RPC: Program not registered` with `nfsvers=3`; a mount without `nfsvers=3` works | 2 | + | Mounting arguments `:` with no `/` after the colon | 3 | + | Manual mount denied from the controller node, allowed from a node the NAS authorizes | 4 | + | `timeout after 110s`, or `failed to make subdirectory ... permission denied` | *NFS PVC stays Pending: csi-nfs-controller mount times out or mkdir of the volume directory is denied on ACP* | + +A host mount that works does not by itself clear a Pending claim: the controller uses host networking and its own node, StorageClass options, and `share` string, so a failure that appears only in the controller log still blocks provisioning. diff --git a/docs/en/solutions/NFS_PVC_stays_Pending_csi-nfs-controller_mount_times_out_or_mkdir_of_the_volume_directory_is_denied_on_ACP.md b/docs/en/solutions/NFS_PVC_stays_Pending_csi-nfs-controller_mount_times_out_or_mkdir_of_the_volume_directory_is_denied_on_ACP.md new file mode 100644 index 000000000..1487aab98 --- /dev/null +++ b/docs/en/solutions/NFS_PVC_stays_Pending_csi-nfs-controller_mount_times_out_or_mkdir_of_the_volume_directory_is_denied_on_ACP.md @@ -0,0 +1,124 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# NFS PVC stays Pending: csi-nfs-controller mount times out or mkdir of the volume directory is denied on ACP + +## Issue + +A PVC on an NFS StorageClass (provisioner `nfs.csi.k8s.io`) stays `Pending`, or the console keeps showing it as `处理中` (Processing) or creating, and no PV is created. The NFS plugin Pods are Running. The PVC events or the `nfs` container log of `csi-nfs-controller` show one of: + +```text +failed to mount nfs server: rpc error: code = Internal desc = mount volume : to /tmp/pvc- timeout after 110s +``` + +```text +GRPC error: rpc error: code = Internal desc = failed to make subdirectory: mkdir /tmp/pvc-/pvc-: permission denied +``` + +A typical case is an application that creates its own claims, for example a Harbor deployment whose PostgreSQL PVCs (`pgdata-acid-postgresql-0`, `pgdata-acid-postgresql-1`) are never provisioned, or a new RWX PVC created right after an NFS StorageClass was added. + +At `CreateVolume`, `csi-nfs-controller` mounts the export on the node where it runs, under `/tmp/pvc-`, creates a per-volume directory in it, and unmounts it again. These two failures sit on either side of that mount. + +Related articles: *NFS PVC stays Pending: csi-nfs-controller mount fails with rpc.statd, exit status 32, or access denied on ACP* (the mount itself is refused). + +## Root Cause + +### Cause 1: the provisioner mounts the export but cannot create the volume directory in it + +The mount succeeds, then the provisioner fails to create the per-volume directory because the NFS server has not granted the required permissions on the export directory. The driver log shows the export mounted on `/tmp/pvc-`, then: + +```text +GRPC error: rpc error: code = Internal desc = failed to make subdirectory: mkdir /tmp/pvc-/pvc-: permission denied +``` + +and then the temporary mount being unmounted again (`unmount volume ... successfully`). The claim stays Pending. Once the server-side permissions are opened, the PVC is created. A PVC that stays in "creating" right after a new StorageClass was created can also come from an NFS server configuration that is not finished (export not defined, clients not allowed, or service not started). + +### Cause 2: the cluster nodes cannot reach the NFS server + +The provisioner cannot complete the mount at all. The PVC events cycle between `Provisioning` (`External provisioner is provisioning volume for claim ...`) and `ProvisioningFailed` with `mount volume : to /tmp/pvc- timeout after 110s`, and the provisioner log counts repeated failures (for example `failures=9`). Network connectivity from the cluster nodes to the NFS storage is missing, for example because of routing, a firewall, or a storage-side client allowlist. For example, on ACP 4.3.0 the PostgreSQL PVCs that a Harbor deployment creates in `cpaas-solution` stay Pending this way when the nodes cannot reach the NFS NAS. When the cluster nodes cannot reach the NFS server address at all, opening network access from the nodes to the server is the whole fix; the PVC is then created and mounted normally. + +## Resolution + +### Cause 1: the provisioner mounts the export but cannot create the volume directory in it + +1. Search the controller log for the exact text: + + ```bash + kubectl -n cpaas-system logs deploy/csi-nfs-controller -c nfs --tail=500 | grep -E 'failed to make subdirectory|permission denied' + ``` + +2. From the node that hosts `csi-nfs-controller`, mount with the StorageClass options and try to create a directory: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep csi-nfs-controller + sudo mount -t nfs -o hard,nfsvers=3,nolock : /mnt + sudo mkdir /mnt/provision-test && sudo rmdir /mnt/provision-test + sudo umount /mnt + ``` + + Use the `nfsvers` and options from the StorageClass. If the mount succeeds and `mkdir` fails, this cause is confirmed. + +3. Grant the permissions on the NFS server side. The provisioner creates directories as root, so on a Linux `nfsd` an export with `root_squash` over a root-owned directory blocks it; compare the export with one that provisions correctly (`exportfs -v`). On a storage appliance, set the equivalent permission for the export on the array. If the server configuration is incomplete, finish it first (export defined, clients allowed, service started). + + > **Warning:** Widening export permissions lets the allowed clients create and change files in the export. Do this only when the mount succeeds and the `mkdir` test fails. Back up the export configuration first; roll back by restoring `/etc/exports` from the backup and running `sudo exportfs -ra` (on an appliance, revert the export setting). + + ```bash + sudo cp -a /etc/exports /etc/exports.bak + sudo vi /etc/exports + sudo exportfs -ra + sudo exportfs -v + ``` + +4. Repeat the `mkdir` test, then create the PVC again. The claim should become `Bound`. + +### Cause 2: the cluster nodes cannot reach the NFS server + +1. From the node that hosts `csi-nfs-controller` (and from the other nodes), test the path to the server: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep csi-nfs-controller + rpcinfo -p + timeout 5 bash -c '/2049' && echo reachable + ``` + + A timeout or no route confirms this cause. + +2. Fix the network path between the cluster nodes and the NFS server: routing, firewalls in between, and the storage-side client allowlist. Allow NFS (TCP 2049) and, for NFSv3, the RPC services the server registers (portmapper on 111 and the ports `rpcinfo -p` lists). + +3. When the connectivity test succeeds from every node, create the claim again. If only one node cannot reach the server while others can, investigate that node's routes and source address. + +## Diagnostic Steps + +1. Read the claim's events and the controller log: + + ```bash + kubectl -n describe pvc + kubectl -n cpaas-system get pods -o wide | grep csi-nfs + kubectl -n cpaas-system logs deploy/csi-nfs-controller -c nfs --tail=200 + ``` + +2. On the node that runs `csi-nfs-controller`, repeat the controller's mount by hand: + + ```bash + sudo mount -t nfs -v -o hard,nfsvers=3,nolock : /mnt + ``` + +3. Match the signature: + + | Controller log or manual test | Cause | + | --- | --- | + | Mount succeeds, then `failed to make subdirectory ... permission denied`; manual `mkdir` in the export fails | 1 | + | `timeout after 110s`; manual mount or `/dev/tcp//2049` hangs from the controller node | 2 | + | `rpc.statd`, `exit status 32`, or `access denied by server while mounting` | *NFS PVC stays Pending: csi-nfs-controller mount fails with rpc.statd, exit status 32, or access denied on ACP* | + +The controller Pod uses host networking and the node's route table, so a test from a different node, or from inside an application Pod, does not prove the controller's path works. diff --git a/docs/en/solutions/Namespace_Stuck_in_Terminating_Due_to_Lingering_Finalizers.md b/docs/en/solutions/Namespace_Stuck_in_Terminating_Due_to_Lingering_Finalizers.md index 768ffb054..594785a35 100644 --- a/docs/en/solutions/Namespace_Stuck_in_Terminating_Due_to_Lingering_Finalizers.md +++ b/docs/en/solutions/Namespace_Stuck_in_Terminating_Due_to_Lingering_Finalizers.md @@ -36,6 +36,8 @@ A namespace wedged in `Terminating` therefore means at least one finalizer is no - The controller is running but cannot complete its cleanup (a downstream API unreachable, a missing credential, a webhook timing out). - The object is a custom resource whose CRD itself has been deleted; without the CRD the controller is gone, leaving orphaned instances. +A related case on the ACP `global` cluster does not involve a finalizer on any object. When a namespace on `global` has the same name as a workload cluster that was created and later deleted, the namespace holds `OperatorView` objects. They are served from memory by `marketplace-apiserver` in `cpaas-system`, which does not accept DELETE for them. The namespace controller cannot empty the namespace, and the namespace condition `NamespaceDeletionContentFailure` reads `Failed to delete all resource types, 1 remaining: OperatorView resource does not support DELETE requests`. Editing or patching the namespace finalizer does not help. Once `marketplace-apiserver` is stopped, its APIService `v1alpha2.app.alauda.io` has no endpoints (`False (MissingEndpoints)`), and the deletion is then blocked by `NamespaceDeletionDiscoveryFailure` with `stale GroupVersion discovery: app.alauda.io/v1alpha2` until that APIService is removed. See Step 3 below. + ## Resolution The safe fix is to find the specific resources holding finalizers, identify the owning controller, and let that controller finish its job. Forcing the finalizer off should be reserved for the case where the controller is known to be gone and its residual state is confirmed to be safe to drop. @@ -68,6 +70,28 @@ kubectl patch / -n "$NS" \ Repeat until every lingering object is gone. The namespace then transitions out of `Terminating` within a few seconds. +Step 3 — on the `global` cluster, when the namespace condition names `OperatorView resource does not support DELETE requests` and the namespace belongs to a workload cluster of the same name that no longer exists: + +> **Warning:** While `marketplace-apiserver` is scaled to zero and its APIService is removed, the APIs it serves, including `OperatorView` and `app.alauda.io/v1alpha2`, are unavailable on the `global` cluster, and console pages that read it can fail. Do this only when the namespace is confirmed to be the leftover of a deleted cluster and the condition above is present. Save both objects first. Rollback: apply the saved APIService and scale `marketplace-apiserver` back to the recorded replica count. + +```bash +NS= +kubectl get ns "$NS" -o jsonpath='{range .status.conditions[*]}{.type}: {.message}{"\n"}{end}' +kubectl -n cpaas-system get deploy marketplace-apiserver -o jsonpath='{.spec.replicas}{"\n"}' +kubectl -n cpaas-system get deploy marketplace-apiserver -o yaml > /tmp/marketplace-apiserver.yaml +kubectl get apiservice v1alpha2.app.alauda.io -o yaml > /tmp/apiservice-v1alpha2.app.alauda.io.yaml + +# 1. stop the aggregated API server that serves OperatorView +kubectl -n cpaas-system scale deploy marketplace-apiserver --replicas=0 + +# 2. the namespace now reports stale discovery for app.alauda.io/v1alpha2; remove the APIService +kubectl get apiservice | grep -v True +kubectl delete apiservice v1alpha2.app.alauda.io +kubectl get ns "$NS" +``` + +When the namespace is gone, apply `/tmp/apiservice-v1alpha2.app.alauda.io.yaml` again if the APIService has not been recreated, and scale `marketplace-apiserver` back to the recorded replica count. Then check that `kubectl get apiservice v1alpha2.app.alauda.io` shows `True`. + ## Diagnostic Steps Before forcing anything, gather evidence so you can justify the decision to remove a finalizer: @@ -78,6 +102,14 @@ Inspect the namespace object itself for its own finalizer — the `kubernetes` f kubectl get ns scheduling -o yaml | grep -A3 finalizers ``` +Read the namespace conditions. `NamespaceDeletionContentFailure` with `OperatorView resource does not support DELETE requests`, or `NamespaceDeletionDiscoveryFailure` with `stale GroupVersion discovery: app.alauda.io/v1alpha2`, points to the `global` cluster case handled in Step 3, not to a finalizer: + +```bash +kubectl get ns scheduling -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}' +kubectl get operatorview -n scheduling +kubectl get apiservice | grep -v True +``` + Check whether any CRD has been deleted while instances still exist; orphaned CRs are a very common cause: ```bash diff --git a/docs/en/solutions/Namespace_quota_exceeds_the_project_quota_a_project_shows_the_whole_clusters_resources_or_system_namespaces_on_ACP.md b/docs/en/solutions/Namespace_quota_exceeds_the_project_quota_a_project_shows_the_whole_clusters_resources_or_system_namespaces_on_ACP.md new file mode 100644 index 000000000..be4fa27d5 --- /dev/null +++ b/docs/en/solutions/Namespace_quota_exceeds_the_project_quota_a_project_shows_the_whole_clusters_resources_or_system_namespaces_on_ACP.md @@ -0,0 +1,141 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.1" + - "4.1.4" + - "4.2" + - "4.3" +--- + +# Namespace quota exceeds the project quota, a project shows the whole cluster's resources, or system namespaces on ACP + +## Issue + +The project view shows quota, resources, or namespaces that do not match what the project should own. Typical questions: + +- "The project has a resource quota, but a namespace created in it has no limit." The namespace quota can then exceed the project quota, and the console does not force a quota when the namespace is created. +- A namespace quota change is rejected. +- "The `cpaas-system` quota is allocated unreasonably and cannot be managed." `cpaas-system` appears with a project and quota figures. +- "How do I give a user `kube-system` access?" The project namespace list shows `命名空间 (0)` (Namespaces (0)) with `无可用命名空间,请联系管理员创建命名空间或添加已有命名空间权限` (no namespace available, ask an administrator), and a chart deployed from Platform Management > App Store Management > Template Repository offers only `default`. +- "How do I dedicate a few nodes to one project?" After nodes are dedicated, a namespace with no project quota still shows the whole cluster's resources. +- "Which namespaces on this cluster are built in by the platform?" A security scan flags `acp-storage` and `argo-rollouts` and asks whether they can be removed. + +## Root Cause + +### Cause 1: namespace quota fields are optional, and a namespace can be created without a quota limit + +Project quota (`projectquota.auth.alauda.io`) is enforced only when a namespace `ResourceQuota` is created or updated: the sum of the same keys across the project's namespaces must stay within the project's hard limits. On the namespace creation page the resource quota is optional; a resource type with no quota is unbounded, so the namespace can consume all of that resource allocated to the project. A key that is not written into the namespace `ResourceQuota` is not limited by the project quota. Separately, creating a namespace in a project that has a quota can intermittently produce a namespace with no quota limit at all; this is a known product defect with a fix planned for 4.3, and no workaround is available for it. + +### Cause 2: a namespace quota change that would exceed the project quota is rejected + +Namespace quotas are carved out of the project quota. A change that would push the project total above its hard limit is rejected; the project quota has to be raised first. + +### Cause 3: the `cpaas-system` project shown from the cluster view is display only + +There is no real `cpaas-system` project. When `cpaas-system` is opened from a cluster's container groups, the project shown is for display only, and its overview and resource limits can be ignored. + +### Cause 4: `kube-system` and `cpaas-system` are not offered as project namespaces by design + +System namespaces such as `kube-system` and `cpaas-system` are not selectable in the project namespace list or the template deployment form. Making them selectable is a feature request, not a setting. + +### Cause 5: dedicated nodes are a Node Isolation Policy, and resource figures follow quota, not nodes + +Nodes are dedicated to a project with a Node Isolation Policy (Platform Management > Security Management) with **Project Exclusive** (`项目独占`) on: only the projects in the policy may use the nodes. Resource pages on the container platform show resources by project and namespace, not by node. A project without a quota can use the whole cluster, so its namespace shows the whole cluster's totals even when its Pods can land only on the dedicated nodes. Project exclusivity and project quota can be used together. This was confirmed on 4.1.4. + +### Cause 6: platform namespaces depend on which components are installed + +There is no single official list of platform namespaces. Common ones: + +- Always present: `cpaas-system` (platform components) and `kube-system` (Kubernetes system components), plus the Kubernetes defaults `default`, `kube-public`, and `kube-node-lease`. +- Present when the related component is deployed: `argocd`, `cert-manager` (certificate management), `katanomi-system` (pipeline katanomi components), `kubevirt` (virtualization), `rook-ceph` (Ceph storage), `nativestor-system` (TopoLVM local storage), `operators` (pipeline-related operators such as `devops-tool-operator-controller-manager`, `harbor-operator-controller-manager`, `katanomi-api`), `rds-system` and `redis` (data services), `argo-rollouts` (default namespace of Alauda Build of Argo Rollouts), and `acp-storage` (default namespace of Alauda Build of Local Storage and of Alauda Container Platform Storage Essentials). + +These are system-component namespaces, not business namespaces to delete. + +## Resolution + +### Cause 1: namespace quota fields are optional, and a namespace can be created without a quota limit + +1. Find the project's namespaces that have no `ResourceQuota`, and compare the keys of an existing quota with the project's: + + ```bash + PROJECT= + for ns in $(kubectl get namespace -l "cpaas.io/project=${PROJECT}" -o name | cut -d/ -f2); do + printf '%s ' "$ns"; kubectl get resourcequota -n "$ns" --no-headers 2>/dev/null | wc -l + done + kubectl get projectquota -l "cpaas.io/project=${PROJECT}" -o yaml + kubectl get resourcequota -n -o yaml + ``` + + `0` means no `ResourceQuota`; a quota whose `hard` lacks `requests.cpu`, `limits.cpu`, `requests.memory`, or `limits.memory` leaves those resources unbounded. + +2. Set the missing values in the console as a normal namespace quota update (Project Management > Namespaces > the namespace > resource quota). The platform rejects a value above what the project has left. When creating namespaces, fill in requests and limits instead of leaving them unlimited. +3. For the intermittent defect where a namespace in a quota-limited project is created with no quota limit, upgrade to 4.3, where the fix is planned. + +### Cause 2: a namespace quota change that would exceed the project quota is rejected + +Raise the project quota on the project details page first, or reduce other namespaces' quotas, then change the namespace quota: + +```bash +kubectl get projectquota -l "cpaas.io/project=" -o yaml +kubectl get resourcequota -n -o yaml +``` + +### Cause 3: the `cpaas-system` project shown from the cluster view is display only + +No action is needed. + +### Cause 4: `kube-system` and `cpaas-system` are not offered as project namespaces by design + +No setting changes this. Submit a feature request if it is needed. + +### Cause 5: dedicated nodes are a Node Isolation Policy, and resource figures follow quota, not nodes + +1. Go to Platform Management > Security Management > Node Isolation Policy > **Create**. Enter a name, turn on **Project Exclusive** (`项目独占`), select the project and the nodes, read the confirmation dialog, and create the policy. + + > **Warning:** Existing Pods of the project are moved onto the policy's nodes only when they are rebuilt; with Project Exclusive on, Pods of other projects already on those nodes are not evicted automatically. Only when these nodes should serve this project alone; plan the rescheduling in a maintenance window. Rollback: delete the policy, or turn Project Exclusive off. + +2. Find workloads of other projects still running on a dedicated node and restart them so they are scheduled elsewhere: + + ```bash + kubectl get pods -A -o wide --field-selector spec.nodeName= + ``` + + > **Warning:** A restart recreates all Pods of the workload; a single-replica workload is briefly unavailable. Only for other projects' workloads still on the dedicated nodes. Rollback: none needed. + + ```bash + kubectl rollout restart deployment/ -n + ``` + +3. To see the project's own capacity, set a project quota sized to the total resources of the dedicated nodes, then view it under Project Management > Namespaces. Node status is checked in the cluster view. + +### Cause 6: platform namespaces depend on which components are installed + +For a namespace not in the list, look at what runs in it and how it is labeled: + +```bash +kubectl get pods -n +kubectl get namespace --show-labels +kubectl get deploy,sts,ds -n +``` + +Pods named after platform components (for example `topolvm-*` or `katanomi-*`) identify a platform namespace. A `cpaas.io/project` label that names a business project identifies a project namespace. Answer a scan finding on `acp-storage` or `argo-rollouts` by showing the owning component running there. + +## Diagnostic Steps + +```bash +kubectl get namespace -l "cpaas.io/project=" +kubectl get resourcequota,limitrange -n +kubectl get projectquota -l "cpaas.io/project=" +``` + +| What you see | Cause | +| --- | --- | +| Namespace in a quota-limited project has no `ResourceQuota`, or its quota lacks CPU or memory keys | 1 | +| Namespace quota update rejected as exceeding the project quota | 2 | +| `cpaas-system` shows a project and quota that cannot be managed | 3 | +| `kube-system` or `cpaas-system` missing from the project namespace list or the chart form | 4 | +| Nodes needed for one project only, or a namespace shows the whole cluster's resources | 5 | +| Which namespaces belong to the platform, or a scan flags `acp-storage` or `argo-rollouts` | 6 | diff --git a/docs/en/solutions/Native_application_update_fails_with_selector_does_not_match_template_labels_on_ACP.md b/docs/en/solutions/Native_application_update_fails_with_selector_does_not_match_template_labels_on_ACP.md new file mode 100644 index 000000000..d63488388 --- /dev/null +++ b/docs/en/solutions/Native_application_update_fails_with_selector_does_not_match_template_labels_on_ACP.md @@ -0,0 +1,87 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Native application update fails with selector does not match template labels on ACP + +## Issue + +A native application shows **Failed** in the console after an update, although its Deployments show **Running** with all replicas ready. The update never reaches the Deployment. + +The `archon` component logs on the global cluster show the update being rejected by the Kubernetes API server: + +```text +handler.go:66] Handle request error: Deployment.apps "" is invalid: spec.template.metadata.labels: Invalid value: {"app.cpaas.io/name":".", ...}: `selector` does not match template `labels` +``` + +The same log can also contain lines such as `Warning: unknown field "spec.componentTemplates[N].apiVersion"`. The line that names the rejected Deployment is the `selector` error above. + +## Root Cause + +### Cause 1: the Deployment selector pins `app.cpaas.io/name` to a value the platform overwrites + +When the platform updates a native application, it writes the standard label `app.cpaas.io/name=.` into the Pod template of each Deployment that belongs to the application. + +A Deployment that was created earlier with a selector containing a different value for the same key, for example `app.cpaas.io/name=`, conflicts with that label. After the platform overwrites the Pod template label, `spec.selector.matchLabels` no longer matches `spec.template.metadata.labels`. Kubernetes validation rejects any Deployment whose selector does not match its own template labels, so the update fails and the native application is marked as failed. + +The existing Pods keep running because the old ReplicaSet is untouched, which is why the Deployments still report Running. + +`spec.selector` of an `apps/v1` Deployment is immutable. The selector cannot be corrected in place: the Deployment has to be recreated with a selector that agrees with the platform label. + +A configuration error in another resource of the same application can also leave the native application in an abnormal state. The `archon` log names the resource and the reason in each case, so read it before changing anything. + +## Resolution + +### Cause 1 + +1. Fix the source that generates the Deployment selector, for example the `selectorLabels` helper of the Helm chart or the template used to render the Deployment. If the selector uses the `app.cpaas.io/name` label, set its value in the `.` format, which is the value the platform injects: + + ```yaml + selector: + matchLabels: + app.cpaas.io/name: . + ``` + +2. Save the current Deployment before removing it: + + ```bash + kubectl -n get deployment -o yaml > -backup.yaml + ``` + +3. Delete the existing Deployment and redeploy the application, so the Deployment is created again with the corrected selector. + + > **Warning:** Deleting the Deployment deletes its Pods. The workload is unavailable until the new Deployment's Pods are Ready. Do this only when the `archon` log shows `` `selector` does not match template `labels` `` for this Deployment and the business allows the workload to be rebuilt. Roll back by applying the saved `-backup.yaml` (remove `metadata.resourceVersion`, `metadata.uid` and `status` first). + + ```bash + kubectl -n delete deployment + ``` + + Then redeploy the native application from the console, or re-run the release that renders the corrected template. + +4. Confirm that the native application status changes from **Failed** to **Running**, and that `archon` no longer logs the error for this Deployment. + +## Diagnostic Steps + +1. Find the `archon` Pods on the global cluster and search their logs for the rejected update: + + ```bash + kubectl get pods -A | grep archon + kubectl -n logs --tail=500 | grep -E 'Handle request error|does not match' + ``` + + Expected for this cause: `Deployment.apps "" is invalid: spec.template.metadata.labels ... `selector` does not match template `labels``. If the error names another resource of the application, fix that resource instead. + +2. Compare the Deployment selector with the value the platform injects: + + ```bash + kubectl -n get deployment \ + -o jsonpath='{.spec.selector.matchLabels}{"\n"}{.spec.template.metadata.labels}{"\n"}' + ``` + + Expected for this cause: the selector contains `app.cpaas.io/name` with a value other than `.`, for example the bare Deployment name. + +3. Check the other Deployments of the same application. A Deployment whose selector does not use `app.cpaas.io/name`, or already uses `.`, updates normally, which separates this cause from a general application controller problem. diff --git a/docs/en/solutions/New_node_cannot_reach_CoreDNS_or_the_API_server_because_a_firewall_drops_tunnel_traffic_between_nodes_on_ACP.md b/docs/en/solutions/New_node_cannot_reach_CoreDNS_or_the_API_server_because_a_firewall_drops_tunnel_traffic_between_nodes_on_ACP.md new file mode 100644 index 000000000..26be10de6 --- /dev/null +++ b/docs/en/solutions/New_node_cannot_reach_CoreDNS_or_the_API_server_because_a_firewall_drops_tunnel_traffic_between_nodes_on_ACP.md @@ -0,0 +1,95 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# New node cannot reach CoreDNS or the API server because a firewall drops tunnel traffic between nodes on ACP + +Related articles: [Workload cluster abnormal or monitoring empty in the console because global cannot reach its ports on ACP](Workload_cluster_abnormal_or_monitoring_empty_in_the_console_because_global_cannot_reach_its_ports_on_ACP.md). + +## Issue + +Pods on one node, often a newly added one, cannot reach cluster services, while the other nodes are fine: + +- A worker that was just added cannot open CoreDNS port `53`, while an older worker can. +- The log component status is abnormal because the log collection Pod (`nevermore`) fails to start on one node only, while it runs on the other nodes. This can show up right after the log storage is switched (for example from Elasticsearch to ClickHouse) and the log collector is reinstalled, but the switch is not the cause. + +## Root Cause + +### Cause 1: a firewall between node subnets drops Calico IPIP traffic + +Both CoreDNS replicas run in one node subnet, and the new node is in another. From the new node, Pod port `1053` on one replica answers and the other replica does not. On a Calico cluster in IPIP mode, Pod-to-Pod traffic between nodes is carried as IP protocol `4` (IPIP). The firewall between the subnets does not allow IPIP, so the tunnel is dropped even where the TCP port looks open. Allowing IPIP restores communication. + +A Kube-OVN cluster uses a different encapsulation (Geneve on UDP `6081`), so the rule it needs is different (Cause 2). + +### Cause 2: a firewall blocks the Kube-OVN network and the kubernetes Service path from one node + +The firewall on the path from the node restricts two things: the Kube-OVN network between that node and the other nodes, and access to the API server behind the `kubernetes` Service on `443`. A Pod on that node cannot reach the API server through the Service, and Pod traffic to other nodes is dropped, so DaemonSet Pods such as the log collector fail on that node only. Opening that access returns the log component to normal. + +## Resolution + +### Cause 1: a firewall between node subnets drops Calico IPIP traffic + +Only when the CNI is Calico in IPIP mode, the new node is in a different subnet from the CoreDNS Pods, those Pods are Ready, and `1053` to the replicas is only partly open. + +1. On the firewall between the node subnets, permit IP protocol `4` (IPIP) between the cluster node prefixes. Opening TCP `53` alone does not carry the tunnel. + + > **Warning:** An IPIP permission lets every encapsulated Pod flow cross, not only DNS. Make sure both CIDRs are cluster node subnets. To roll back, delete the same firewall entry. A short loss of cross-subnet Pod traffic is possible while the device commits the policy. + +2. If the filtering device is host `iptables` rather than a middlebox, the equivalent on each node is below. Persist it with the mechanism the node already uses (`iptables-save` or `firewall-cmd`); to roll back, run the same commands with `-D` instead of `-I`. + + ```bash + iptables -I INPUT -p 4 -s -j ACCEPT + iptables -I OUTPUT -p 4 -d -j ACCEPT + ``` + +3. The new worker should now open `1053` on every CoreDNS Pod IP and `53` on the `kube-dns` ClusterIP. + +### Cause 2: a firewall blocks the Kube-OVN network and the kubernetes Service path from one node + +Only when the failing Pod is on a single node, Pods on that node cannot reach the `kubernetes` Service on `443`, and the Kube-OVN tunnel between that node and the others fails while the other nodes are fine. + +1. On the firewall of that node, and on any device between it and the other nodes, open the Kube-OVN tunnel traffic between all cluster nodes (Geneve UDP `6081` by default) and the API server path behind the `kubernetes` Service (the endpoint addresses and port listed in its EndpointSlice, `6443` on the control-plane nodes). + + > **Warning:** Opening ports on a node firewall widens what can reach that node. Limit the source to the cluster node prefix and save the current rule set first. To roll back, remove the rules you added or restore the saved set. A firewall reload can briefly drop existing flows on that node. + + If the node uses `firewalld`, one way that keeps the source scoped is to trust the node prefix: + + ```bash + firewall-cmd --list-all > /root/firewalld.before.txt + firewall-cmd --permanent --zone=trusted --add-source= + firewall-cmd --reload + ``` + +2. Recreate the collector Pod on that node and confirm it stays `Running`: + + ```bash + kubectl get pod -A -o wide | grep nevermore | grep + kubectl -n delete pod + ``` + +## Diagnostic Steps + +1. For a new node and CoreDNS, compare the node subnet with the subnet of the nodes running CoreDNS, and test `1053` to every replica. Partial `1053` reachability across that boundary on Calico IPIP is Cause 1: + + ```bash + kubectl -n kube-system get pod -l k8s-app=kube-dns -o wide + nc -vz -w 3 1053 + ``` + +2. For a DaemonSet Pod failing on one node only, test that node's path to the API server and to the other nodes. A timeout here while the same test passes from a healthy node is Cause 2. `kube-ovn-pinger` on the failing node logs the node-to-node and Pod-to-Pod checks that fail: + + ```bash + kubectl get endpointslice -n default -l kubernetes.io/service-name=kubernetes + nc -vz -w 3 6443 + kubectl -n kube-system get pod -l app=kube-ovn-pinger -o wide | grep + kubectl -n kube-system logs --tail=50 + firewall-cmd --list-all + ``` + +3. If `1053` is reachable from the new node and lookups still fail only there, see [Intermittent DNS failures in Pods from a new node, NIC flapping, the KubeDNS alert or CoreDNS upstream timeouts on ACP](Intermittent_DNS_failures_in_Pods_from_a_new_node_NIC_flapping_the_KubeDNS_alert_or_CoreDNS_upstream_timeouts_on_ACP.md). If the firewall is on the node itself and blocks kubelet or metrics ports, see [Node firewall or iptables rules block kubelet 10250, metrics 10254 or control-plane traffic on ACP](Node_firewall_or_iptables_rules_block_kubelet_10250_metrics_10254_or_control_plane_traffic_on_ACP.md). diff --git a/docs/en/solutions/Nexus_database_export_task_unreachable_during_migration_to_Alauda_DevOps_v4_after_deleting_the_Nexus_Services_first.md b/docs/en/solutions/Nexus_database_export_task_unreachable_during_migration_to_Alauda_DevOps_v4_after_deleting_the_Nexus_Services_first.md new file mode 100644 index 000000000..6c52ad88d --- /dev/null +++ b/docs/en/solutions/Nexus_database_export_task_unreachable_during_migration_to_Alauda_DevOps_v4_after_deleting_the_Nexus_Services_first.md @@ -0,0 +1,107 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: [] +--- + +# Nexus database export task unreachable during migration to Alauda DevOps v4 after deleting the Nexus Services first + +## Issue + +While migrating a platform-deployed Nexus to Alauda DevOps v4 (the Nexus version itself does not change), the migration steps are run in their numbered order: + +- 1.1 Service shutdown: uninstall the old-version operator, then delete the Nexus Services with `kubectl -n ${INSTANCE_NAMESPACE} delete service -l release=${INSTANCE_NAME}`. +- 1.2 Back up Blob storage with `kubectl cp` from the Nexus Pod. +- 1.3 Back up the database by creating an **Admin - Export databases for backup** task under **Admin Dashboard > System > Tasks** in the Nexus web UI. + +After step 1.1, step 1.3 cannot be performed: the Nexus web UI, where the export task is created, is no longer reachable. + +## Root Cause + +### Cause 1: the Services were deleted before the backups were taken + +Step 1.3 is done inside the Nexus web UI, and the UI is reached through the Nexus Services. Step 1.1 deletes those Services. Running 1.1 first removes access to the UI, so the database export task cannot be created or run. + +The correct execution order is: complete steps 1.2 and 1.3 first, while the Services still exist, and only then perform step 1.1. + +## Resolution + +### Cause 1 + +1. Prepare a local backup directory with enough free disk space for the blob store and the database export, and set the variables used by the steps: + + ```bash + mkdir -p nexus-backup + export INSTANCE_NAME= + export INSTANCE_NAMESPACE= + kubectl -n ${INSTANCE_NAMESPACE} get pod -l release=${INSTANCE_NAME} + export POD_NAME= + ``` + +2. Step 1.2, back up Blob storage: + + ```bash + mkdir -p nexus-backup/blobs + kubectl -n ${INSTANCE_NAMESPACE} cp ${POD_NAME}:/nexus-data/blobs nexus-backup/blobs + ``` + +3. Step 1.3, back up the database in the Nexus web UI: + 1. Go to **Admin Dashboard > System > Tasks** and create a task. + 2. Type: **Admin - Export databases for backup**. + 3. Task name: any name. + 4. Backup location: a directory path inside the Nexus data volume directory. + 5. Frequency: manual. + 6. Run the task and confirm that it completes. + + The export files are written inside the Nexus data volume, and step 1.2 copies only `/nexus-data/blobs`. Copy the export out of the Pod as well, so that the database backup does not exist only on the volume being migrated: + + ```bash + mkdir -p nexus-backup/db + kubectl -n ${INSTANCE_NAMESPACE} cp ${POD_NAME}: nexus-backup/db + ls -la nexus-backup/db + ``` + +4. Step 1.1, shut down the service, only after the blob backup and the database export are both in `nexus-backup/`. Uninstall the old-version operator first; otherwise the deleted Services are recreated automatically. Then save and delete the Services: + + > **Warning:** After the Services are deleted, Nexus is no longer reachable by users, build tools or the web UI. Do this only after steps 1.2 and 1.3 have completed and the backups are confirmed. To restore access, re-apply the saved Service manifests (remove `metadata.resourceVersion`, `metadata.uid`, `spec.clusterIP` and `spec.clusterIPs` first). + + ```bash + kubectl -n ${INSTANCE_NAMESPACE} get service -l release=${INSTANCE_NAME} -o yaml > nexus-backup/services.yaml + kubectl -n ${INSTANCE_NAMESPACE} delete service -l release=${INSTANCE_NAME} + ``` + +5. Continue with the remaining migration steps. + +If the Services were already deleted before the database export, the web UI can still be reached directly on the Pod, without a Service, by port-forwarding to the Nexus HTTP port (8081 by default in Nexus): + +```bash +kubectl -n ${INSTANCE_NAMESPACE} port-forward pod/${POD_NAME} 8081:8081 +``` + +Open `http://localhost:8081` on the machine running the command, then perform step 1.3 and copy the export out of the Pod as described above. + +## Diagnostic Steps + +1. Check whether the Nexus Services still exist: + + ```bash + kubectl -n ${INSTANCE_NAMESPACE} get service -l release=${INSTANCE_NAME} + ``` + + `No resources found` means step 1.1 has already been run; take the database export through the port-forward described above before going on. + +2. Check that the Nexus Pod is still running, which steps 1.2 and 1.3 need: + + ```bash + kubectl -n ${INSTANCE_NAMESPACE} get pod -l release=${INSTANCE_NAME} -o wide + ``` + +3. After step 1.3, check that the export files exist in the backup location set in the task: + + ```bash + kubectl -n ${INSTANCE_NAMESPACE} exec ${POD_NAME} -- ls -la + ``` + + An empty directory means the export task did not run or did not complete. diff --git a/docs/en/solutions/No_Kafka_in_Data_Services_or_Alauda_Build_of_Kafka_2_unpublished_or_failing_to_deploy_on_ACP.md b/docs/en/solutions/No_Kafka_in_Data_Services_or_Alauda_Build_of_Kafka_2_unpublished_or_failing_to_deploy_on_ACP.md new file mode 100644 index 000000000..f4b3bb745 --- /dev/null +++ b/docs/en/solutions/No_Kafka_in_Data_Services_or_Alauda_Build_of_Kafka_2_unpublished_or_failing_to_deploy_on_ACP.md @@ -0,0 +1,175 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda Application Services +ProductsVersion: + - "3.6" + - "3.12" + - "3.14" + - "3.16" + - "3.18" + - "4.0" + - "4.1" + - "4.2" +--- + +# No Kafka in Data Services, or Alauda Build of Kafka 2 unpublished or failing to deploy on ACP + +## Issue + +Kafka cannot be offered to business namespaces because its operator is missing or unhealthy: + +- The Data Services menu lists other services (for example MySQL, Redis, PostgreSQL) but no Kafka, so there is no **Create Kafka Instance** button. `kubectl get po -A | grep kafka` lists only `cpaas-kafka-0/1/2` in `cpaas-system`. Often the request started as "connect a Kafka web UI such as kafka-ui to the platform Kafka". +- OperatorHub shows Alauda Build of Kafka 2 as not published (`未上架`), or its install fails on ACP 4.1. +- After the global cluster was upgraded (for example from 3.18 to 4.0.12), a business cluster no longer lists MySQL or Kafka, while they still appear on the global cluster. +- Kafka, MySQL, and RabbitMQ operators all show abnormal at the same time. + +Related articles: [Kafka instance creation times out on the entity-operator, shows failed, or the cluster operator restarts on ACP](Kafka_instance_creation_times_out_on_the_entity-operator_shows_failed_or_the_cluster_operator_restarts_on_ACP.md) covers create failures once the operator is installed. A Kafka 2.x instance that the installed operator skips is covered in [Kafka operator mismatch: wrong create page version, Kafka 2.x instance skipped, or ZooKeeper mode blocks upgrade on ACP](Kafka_operator_mismatch_wrong_create_page_version_Kafka_2x_instance_skipped_or_ZooKeeper_mode_blocks_upgrade_on_ACP.md). + +## Root Cause + +### Cause 1: Data Services has no Kafka menu because the Kafka operator is not deployed + +The `cpaas-kafka-*` pods in `cpaas-system` are the platform's own Kafka, used by platform components, not a business instance. The Data Services Kafka feature switch is off and the Kafka operator (Alauda Streaming Service for Kafka) is not installed, so Data Services offers no Kafka. A web UI or business traffic should use a Data Services Kafka instance; the platform Kafka is not reconfigured for that. + +### Cause 2: Alauda Build of Kafka 2 fails to deploy on ACP 4.1 because Kafka Operator 3.x earlier than 4.2 is not deployed first + +On ACP 4.1 (for example 4.1.3), installing Alauda Build of Kafka 2 fails at the operator deployment. On platform 4.1, Kafka Operator 3.x must be deployed before Kafka Operator 2.x, and that 3.x operator must be a version earlier than 4.2: Kafka Operator 3.x 4.2 and later no longer supports Kafka 2.x. + +### Cause 3: The Alauda Build of Kafka 2 package has not been pushed with the matching violet + +OperatorHub shows Alauda Build of Kafka 2 (for example `4.0.3` on a 4.0.9 cluster) as not published, and no `kafka-operator-2x` `artifactversion` exists yet. The operator package has to be uploaded with `violet push`, using the `violet` version that matches the package rather than simply the `violet` that came with the platform. + +### Cause 4: The Kafka 2 bundle artifact name does not match OperatorHub + +The package is uploaded, `artifactversion` for `kafka-operator-2x` is `Present` and `Success`, and `packagemanifest` `strimzi-kafka-operator-2x` exists, but the marketplace still shows Alauda Build of Kafka 2 as not published. OperatorHub does not accept an artifact named `kafka-operator-2x`; the name needs the `operatorhub-` prefix. + +### Cause 5: Global and business clusters are on incompatible versions + +The global cluster is 4.0.12 after an upgrade from 3.18 (Data Services Essentials `v4.0.23` on the global side) while the business cluster is still on the previous version. MySQL and Kafka disappear from the business cluster view. Errors in other products (for example service mesh) in the same window come from the same version skew. + +### Cause 6: MySQL (MGR) operator 4.4.4 with MySQL 8.0 makes several middleware operators abnormal + +Kafka, MySQL, and RabbitMQ operators show abnormal together. MGR Operator 4.4.4 does not support MySQL 8.0. The Kafka operator status clears after the MySQL operator and Data Services Essentials are moved back to the 4.3 line, not after a Kafka restart. + +## Resolution + +### Cause 1: Data Services has no Kafka menu because the Kafka operator is not deployed + +```bash +kubectl get po -A | grep -i kafka +kubectl get csv -A | grep -i kafka +kubectl get packagemanifest -A | grep -i kafka +``` + +If the only Kafka pods are `cpaas-kafka-*` in `cpaas-system` and no Kafka operator CSV exists: + +1. Enable the Kafka feature switch for Data Services. +2. Install Alauda Streaming Service for Kafka from **OperatorHub**: target cluster, channel `stable`, namespace `kafka-system`, and the operator version that matches your platform (for example `4.1.0`). On clusters that still need Kafka 2.x, respect the version limit in Cause 2. +3. Confirm the operator is running, then reload Data Services. The Kafka menu and **Create Kafka Instance** appear. + + ```bash + kubectl -n kafka-system get pods + kubectl get csv -A | grep -i kafka + ``` + +4. Create the business instance. For an SSL client such as kafka-ui, the create form offers in-cluster access with link encryption (Plain/SCRAM-SHA-512, or TLS with SCRAM-SHA-512 or TLS authentication) and out-of-cluster `NodePort` access with TLS. Deploy the community web UI separately and point it at the instance. Client certificates and users are covered in [Kafka client connection failures: timeouts, TLS certificates, authorization, NodePort, and reverse DNS on ACP](Kafka_client_connection_failures_timeouts_TLS_certificates_authorization_NodePort_and_reverse_DNS_on_ACP.md). + +### Cause 2: Alauda Build of Kafka 2 fails to deploy on ACP 4.1 because Kafka Operator 3.x earlier than 4.2 is not deployed first + +```bash +kubectl get packagemanifest,artifact,artifactversion -A | grep kafka +kubectl get csv -A | grep -i kafka +``` + +If `artifactversion` `kafka-operator-2x.*` is `Present` and `Success` and the only problem is that the marketplace shows the package as not published, use Cause 4. If the Kafka Operator 2.x deployment fails and no Kafka Operator 3.x is installed, or the installed 3.x is 4.2 or later: + +1. Deploy Kafka Operator 3.x at a version earlier than 4.2 from the marketplace. +2. Deploy Kafka Operator 2.x (Alauda Build of Kafka 2) after the 3.x operator is running. +3. Confirm both CSVs are `Succeeded` with `kubectl get csv -A | grep -i kafka`. + +Do not upgrade the 3.x operator to 4.2 or later on a cluster that still needs Kafka 2.x. + +### Cause 3: The Alauda Build of Kafka 2 package has not been pushed with the matching violet + +1. Confirm the package objects are missing: + + ```bash + kubectl get packagemanifest,artifact,artifactversion -A | grep kafka + ``` + +2. Push the Alauda Build of Kafka 2 package with the `violet` version that matches it (check with `violet version`), choosing a package version that supports the platform and the Kafka version (`v4.0.3` supports platform 4.0 and 4.1 with Kafka 2.5, 2.7, and 2.8): + + ```bash + violet push .tgz \ + --platform-address "https://" \ + --platform-username "" \ + --platform-password "" \ + --clusters "" + ``` + +3. Publish it and install it from **OperatorHub** (channel `stable`). If `artifactversion` is now `Present`/`Success` and the package is still not published, continue with Cause 4. + +### Cause 4: The Kafka 2 bundle artifact name does not match OperatorHub + +> **Warning:** Deleting the `artifactversion` and the old `artifact` removes the objects OperatorHub is currently ignoring. Do it only when `kafka-operator-2x` is `Present` and `Success` but Alauda Build of Kafka 2 still shows as not published, and only after both objects are saved to YAML. Roll back by re-applying the backup files. + +```bash +kubectl get packagemanifest,artifact,artifactversion -A | grep kafka +kubectl -n cpaas-system get artifactversion -o yaml > artifactversion.bak.yaml +kubectl -n cpaas-system get artifact kafka-operator-2x -o yaml > artifact.bak.yaml +kubectl -n cpaas-system delete artifactversion +``` + +Use the `artifactversion` name in your cluster (for the 4.0.3 bundle it is `kafka-operator-2x.v4.0.3`). Copy `artifact.bak.yaml` to `operatorhub-kafka-operator-2x.yaml` and edit it: + +- Remove `resourceVersion`, `uid`, `creationTimestamp`, `generation`, `managedFields`, and the whole `status` block. +- Set `metadata.name` to `operatorhub-kafka-operator-2x`. +- Set the `spec.artifactVersionSelector.matchLabels` key `cpaas.io/artifact-version` to `operatorhub-kafka-operator-2x`. +- Keep the existing `registry`, `repository`, and `type: bundle` values. + +```bash +kubectl -n cpaas-system apply -f operatorhub-kafka-operator-2x.yaml +kubectl -n cpaas-system delete artifact kafka-operator-2x +``` + +Publish the bundle again the same way it was first uploaded. Confirm that `kubectl -n cpaas-system get artifact operatorhub-kafka-operator-2x -o yaml` reports a synced `status` and that OperatorHub lists the Kafka 2 operator as available. + +### Cause 5: Global and business clusters are on incompatible versions + +Upgrade the business cluster, and the middleware components on it, to versions compatible with the global cluster. No configuration change brings the menus back while the business cluster stays on the old version. Data Services Essentials `v4.0.23` is the global version in this situation, not a downgrade target. + +### Cause 6: MySQL (MGR) operator 4.4.4 with MySQL 8.0 makes several middleware operators abnormal + +> **Warning:** Moving the MySQL operator to 4.3.2 and Data Services Essentials to 4.3.3 downgrades those two components and restarts their workloads. Do it only when Kafka, MySQL, and RabbitMQ operators went abnormal together and the installed MGR or MySQL operator is 4.4.4 with MySQL 8.0. Returning to 4.4.4 brings the incompatibility back. + +Install MySQL operator 4.3.2 and Data Services Essentials 4.3.3 through the same marketplace path used for 4.4.4. The Kafka operator status recovers with them. + +```bash +kubectl get csv -A | grep -Ei 'mysql|kafka|rabbit' +``` + +## Diagnostic Steps + +1. Is any business Kafka operator installed? + + ```bash + kubectl get csv -A | grep -i kafka + kubectl get po -A | grep -i kafka + ``` + + No Kafka operator CSV and only `cpaas-kafka-*` pods in `cpaas-system`: Cause 1. + +2. Marketplace state of the Kafka 2 package: + + ```bash + kubectl get packagemanifest,artifact,artifactversion -A | grep kafka + ``` + + No `kafka-operator-2x` objects at all: Cause 3. `Present` plus `Success` with a name that lacks `operatorhub-`: Cause 4. The package is published but its deployment fails on 4.1: Cause 2. + +3. Menus missing only on business clusters right after a global upgrade: compare the cluster versions on the cluster list. Different minor versions: Cause 5. + +4. Several middleware operators abnormal on the same day as a MySQL operator change: compare the CSV versions with Cause 6 before restarting Kafka. diff --git a/docs/en/solutions/No_default_etcd_backup_on_ACP_Cluster_Enhancer_not_installed_or_etcd-backup-default_not_created.md b/docs/en/solutions/No_default_etcd_backup_on_ACP_Cluster_Enhancer_not_installed_or_etcd-backup-default_not_created.md new file mode 100644 index 000000000..f4f5c4eb6 --- /dev/null +++ b/docs/en/solutions/No_default_etcd_backup_on_ACP_Cluster_Enhancer_not_installed_or_etcd-backup-default_not_created.md @@ -0,0 +1,94 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" + - "4.3" +--- + +# No default etcd backup on ACP: Cluster Enhancer not installed or etcd-backup-default not created + +## Issue + +- An inspection finds no etcd backup on a cluster. After the Cluster Enhancer plugin is installed, `kubectl get ebc -A` is empty and `kubectl get acj,bcj -A | grep -i etcd` shows no backup job. +- Creating the default backup by hand fails: `kubectl create -f ebc.yaml` for `etcd-backup-default` returns `Error from server (MethodNotAllowed): error when creating "ebc.yaml": the server does not allow this method on the requested resource` (HTTP 405). +- On a 4.2.1 `global` cluster the etcd backup feature is missing, and the monitoring dashboard list shows only `container-platform`, `log`, and `monitor`, without `cluster` and `kubernetes`. +- After the platform moved from 3.16 to 4.2, it is unclear whether the etcd backup policy reached the workload clusters, including workload clusters still on 3.16. + +Related articles: "etcd backup dates missing in the console, backup tab gone, duplicate copies, or retention to change on ACP" covers an existing backup configuration; "Scheduled etcd backups stop after an upgrade because the etcd-backup-default AdvancedCronJob stops triggering on ACP" covers backups that stop. + +## Root Cause + +### Cause 1: Cluster Enhancer is installed but does not create the default backup until create-etcd-backup-default is true + +The default etcd backup (`EtcdBackupConfiguration`, short name `ebc`, named `etcd-backup-default`) is created by the Cluster Enhancer API Deployment (its name contains `cluster-enhancer-api`) only when its argument `--create-etcd-backup-default` is `true`. Without it, no `ebc` and no backup jobs exist, and the object cannot be created by hand because the API does not allow `create` (`MethodNotAllowed`). Setting the argument to `true` creates the object. For example, on a 4.3 cluster with the `Alauda Container Platform Cluster Enhancer` plugin `v4.3.0`, `kubectl get ebc -A` then lists `etcd-backup-default` with schedule `0 0 * * *`, PAUSED `false`, local path `/cpaas/backup`, TTL `604800`, and 0 records. When the default backup path is also missing, the argument `--default-etcd-backup-path=` (for example `/var/cpaas/backup`) is added next to it. + +### Cause 2: the Cluster Enhancer plugin is not installed on the cluster + +etcd backup rules come from the `Alauda Container Platform Cluster Enhancer` cluster plugin, which backs up etcd data, monitors certificates, cleans up events, and provides the `cluster` and `kubernetes` monitoring dashboards. A cluster without the plugin has neither the etcd backup feature nor those two dashboards. After the plugin is installed (for example version `v4.2.6` on 4.2.1), the dashboard list shows `container-platform`, `cluster`, `log`, `monitor`, and `kubernetes`, and etcd backup is available through `EtcdBackupConfiguration`. + +The policy is per cluster: deploying the etcd backup plugin on the global cluster after an upgrade from 3.16 to 4.2 does not push the policy onto every workload cluster. Backup rules exist on the clusters where Cluster Enhancer is deployed after the 4.2 upgrade, and workload clusters left on 3.16 keep their 3.16 backup rules. + +## Resolution + +### Cause 1: Cluster Enhancer is installed but does not create the default backup until create-etcd-backup-default is true + +1. Confirm that the plugin is installed, no `ebc` exists, and find the Deployment: + + ```bash + kubectl get mplugin cluster-enhancer -o yaml | grep -iE 'version|status' | head + kubectl get ebc -A + kubectl get acj,bcj -A | grep -i etcd || true + kubectl get deploy -A | grep cluster-enhancer-api + kubectl -n get deploy -o jsonpath='{.spec.template.spec.containers[*].args}{"\n"}' + ``` + +2. Save the Deployment: + + ```bash + kubectl -n get deploy -o yaml > /tmp/cluster-enhancer-api.yaml + ``` + +3. Set `--create-etcd-backup-default=true` in the container arguments, and add `--default-etcd-backup-path=` if no default backup path is configured. + + > **Warning:** Editing the Deployment restarts the Cluster Enhancer API Pods; backup and enhancement API calls pause until the new Pods are Ready. Precondition: only when the default etcd backup objects are absent and the argument is not already `true`. Rollback: `kubectl apply -f /tmp/cluster-enhancer-api.yaml`. + + ```bash + kubectl -n edit deploy + kubectl -n rollout status deploy + ``` + +4. Check that the default backup now exists: + + ```bash + kubectl get ebc -A + ``` + + Expected: `etcd-backup-default` with a schedule, PAUSED `false`, a local path, and a TTL. + +### Cause 2: the Cluster Enhancer plugin is not installed on the cluster + +1. Check whether the plugin and a backup configuration exist on the cluster in question: + + ```bash + kubectl get deploy -A | grep cluster-enhancer + kubectl get etcdbackupconfiguration -A + ``` + +2. Install the `Alauda Container Platform Cluster Enhancer` cluster plugin from **Administrator → Marketplace → Cluster Plugins** for that cluster. +3. Check the dashboard list (it should now include `cluster` and `kubernetes`) and run `kubectl get etcdbackupconfiguration -A` again. If no default object appears, continue with Cause 1. +4. To change a backup rule, edit `EtcdBackupConfiguration` on the cluster whose rule you mean to change. A workload cluster still on 3.16 keeps its existing 3.16 backup configuration. + +## Diagnostic Steps + +```bash +kubectl get deploy -A | grep cluster-enhancer +kubectl get ebc -A +kubectl get acj,bcj -A | grep -i etcd +``` + +- Cluster Enhancer Deployments present, `ebc` empty, or `MethodNotAllowed` (405) when creating an `ebc` by hand: Cause 1. Expected: the `cluster-enhancer-api` arguments do not contain `--create-etcd-backup-default=true`. +- No Cluster Enhancer Deployment on the cluster, no etcd backup, and no `cluster` or `kubernetes` dashboards: Cause 2. +- An `ebc` exists and backups ran before but stopped: see "Scheduled etcd backups stop after an upgrade because the etcd-backup-default AdvancedCronJob stops triggering on ACP". diff --git a/docs/en/solutions/Node_API_and_VIP_access_lost_after_creating_a_Kube-OVN_ProviderNetwork_or_underlay_subnet_on_ACP.md b/docs/en/solutions/Node_API_and_VIP_access_lost_after_creating_a_Kube-OVN_ProviderNetwork_or_underlay_subnet_on_ACP.md new file mode 100644 index 000000000..e8ab76347 --- /dev/null +++ b/docs/en/solutions/Node_API_and_VIP_access_lost_after_creating_a_Kube-OVN_ProviderNetwork_or_underlay_subnet_on_ACP.md @@ -0,0 +1,104 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# Node, API and VIP access lost after creating a Kube-OVN ProviderNetwork or underlay subnet on ACP + +Related articles: [Kube-OVN underlay Pods fail with resolve MAC address timeout or bridge network InitOVSBridgeFailed on ACP](Kube-OVN_underlay_Pods_fail_with_resolve_MAC_address_timeout_or_bridge_network_InitOVSBridgeFailed_on_ACP.md); [All charts stay ChartDownloading on a new Kube-OVN cluster because ovn-default was created as overlay on ACP](All_charts_stay_ChartDownloading_on_a_new_Kube-OVN_cluster_because_ovn-default_was_created_as_overlay_on_ACP.md). + +## Issue + +One of the following happens right after an underlay network is set up with Kube-OVN: + +- A ProviderNetwork (bridge network) was just created and the cluster becomes unavailable: the login page opens but login fails, the cluster VIP does not answer `ping`, and a new interface `br-` appears on every node with the same addresses as the node NIC. +- A new cluster is being created with Kube-OVN, transport mode `Underlay`, a custom default subnet and a VLAN ID. As soon as creation starts, the masters can no longer be logged into, and the switch reports MAC or IP conflicts such as `ARP_USER_DUPLICATE_IPADDR_DETECT` (`user IP address conflict`). +- Before setting up an underlay network on bonded NICs, an engineer asks how the bond is configured and what the platform needs. + +All three come from the choice of the node NIC that the ProviderNetwork uses, or from addresses that collide with the node network. + +## Root Cause + +### Cause 1: the ProviderNetwork defaultInterface is the NIC that carries the node IP + +When a ProviderNetwork is created, Kube-OVN moves the NIC named in `defaultInterface` (or `customInterfaces`) into the OVS bridge `br-`. The node addresses and all routes of that NIC move to the bridge. If that NIC is the one that carries the node IP (for example `defaultInterface: bond0` when `bond0` holds the node address), node, API server and VIP access break on every node. The underlay NIC must be a separate NIC, not the one that carries the node IP. + +The resource is named `provider-networks` (`provider-networks.kubeovn.io`, cluster-scoped). `kubectl delete ProviderNetwork ` fails with `the server doesn't have a resource type "ProviderNetwork"`. + +### Cause 2: the underlay subnet chosen at cluster creation overlaps the node network + +When a cluster is created with an underlay default subnet that lies in the same segment as the node IPs, platform component Pods receive addresses from that subnet as soon as creation starts. Those addresses collide with node addresses in the same VLAN: the switch sees one IP on several MACs (`ARP_USER_DUPLICATE_IPADDR_DETECT`), and the masters become unreachable. Stopping the creation or scale-out task restores node access. + +### Cause 3: NIC bonding for an underlay network is done on the node, not by the platform + +An underlay network needs the gateway address, the subnet, the VLAN ID and the NIC. The platform does not configure bonds. Node-level NIC configuration, including bond, trunk and VLAN, is done in the node operating system. The platform then creates a bridge on top of that interface. If the bond is missing on some nodes, or it is the NIC that holds the node IP, the bridge either fails on those nodes or takes node access with it (Cause 1). + +## Resolution + +### Cause 1: the ProviderNetwork defaultInterface is the NIC that carries the node IP + +1. From a session that still reaches the API (a node console, or any master that still answers), save and delete the ProviderNetwork. + + > **Warning:** Deleting a ProviderNetwork removes the bridge from every node, and underlay Pods that use it lose their network. Here that is the intended effect, because node access is already broken. There is no undo; to roll back, recreate the object from the saved YAML (with a corrected NIC). + + ```bash + kubectl get provider-networks + kubectl get provider-networks -o yaml > /root/pn-.yaml + kubectl delete provider-networks + ip -br addr show br- || true + ``` + + After the delete, the node addresses return to the NIC and node, API and VIP access come back. + +2. Choose a `defaultInterface` from `ip -br addr` that does not hold the node IP, and create the ProviderNetwork again with it. + +### Cause 2: the underlay subnet chosen at cluster creation overlaps the node network + +1. Stop the cluster creation or scale-out task that started handing out addresses. Node access comes back once it stops. +2. Plan an underlay subnet that does not include any node address, and recreate the cluster with it. Exclude every node, gateway and VIP address that must stay in the same VLAN. + + ```bash + kubectl get nodes -o wide + kubectl get subnet -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidrBlock,VLAN:.spec.vlan,EXCLUDE:.spec.excludeIps' + ``` + +### Cause 3: NIC bonding for an underlay network is done on the node, not by the platform + +1. Configure the bond (and any trunk or VLAN) in the operating system of every underlay node. NetworkManager is the preferred tool for node network management, because the platform's functional tests run with it. +2. Check that the bond exists with the same name on all those nodes, and that it is not the NIC that carries the node IP: + + ```bash + ip -d link show + cat /proc/net/bonding/ + ip -br addr show + ``` + +3. Give the platform the bond name, whether the switch port is a trunk or an access (VLAN) port, the VLAN ID, the subnet and the gateway. Nodes whose underlay NIC has another name are listed under `customInterfaces` of the ProviderNetwork; nodes without an underlay NIC go under `excludeNodes` (see the related article on `InitOVSBridgeFailed`). + +## Diagnostic Steps + +1. Check where the node address sits. If it is on `br-` and no longer on the physical NIC or bond, Cause 1 applies: + + ```bash + ip -br addr + ip route + kubectl get provider-networks -o yaml | grep -E 'name:|defaultInterface|interface:' + ``` + +2. Compare the underlay subnet with the node addresses. A subnet CIDR that contains node IPs, together with duplicate-IP alarms on the switch, is Cause 2: + + ```bash + kubectl get nodes -o wide + kubectl get subnet -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidrBlock,VLAN:.spec.vlan' + ``` + +3. For a bond, confirm that it exists and has the expected members on every underlay node before the bridge network is created (Cause 3): + + ```bash + ip -d link show | grep -A1 -E 'bond|vlan' + ``` diff --git a/docs/en/solutions/Node_NotReady_caused_by_network_storage_or_host_level_faults_on_ACP.md b/docs/en/solutions/Node_NotReady_caused_by_network_storage_or_host_level_faults_on_ACP.md new file mode 100644 index 000000000..bb10442e1 --- /dev/null +++ b/docs/en/solutions/Node_NotReady_caused_by_network_storage_or_host_level_faults_on_ACP.md @@ -0,0 +1,225 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.x + - 4.x +--- + +# Node NotReady caused by network, storage or host-level faults on ACP + +## Issue + +`kubectl get nodes` shows one or more nodes `NotReady`, and the reports look like this: + +- "The business cluster went down again", "a large number of nodes are not ready", "the cluster status is Abnormal". +- Applications or middleware (for example Redis) restart and fail over on their own, with nobody deploying anything. +- A node "drops out" for about 30 seconds and recovers by itself, more and more often. +- A node shows abnormal on the platform and cannot be reached over SSH. +- Pods on one node stay `Terminating`, new Pods cannot be scheduled there, and the event says the node has a taint. +- After a host reboot, one Pod stays `0/1` with repeated `SandboxChanged` and `FailedKillPod ... failed to find plugin "galaxy-veth" in path [/opt/cni/bin]` events. +- A workload cluster shows `Abnormal`, and its KubeadmControlPlane has `MachinesReady` `Unknown` with `Kubelet stopped posting node status.` +- "An application Pod keeps restarting", but the Pod is really being recreated on other nodes. + +A node is `NotReady` when its kubelet stops posting status to kube-apiserver within the node monitor grace period. The node then carries the `node.kubernetes.io/unreachable` or `node.kubernetes.io/not-ready` taint, so its Pods are evicted and recreated elsewhere, which looks like an application restart. This article covers cases where the kubelet itself is fine but the network, the storage under the host, or the host as a whole stops it from reporting. A kubelet or container runtime that is broken on the node is a different problem (see *Node NotReady on ACP with kubelet "PLEG is not healthy"* and *Diagnosing a node that reports NotReady on Alauda Container Platform*). Many Pods `Unknown` after an IaaS upgrade renamed the node NICs while the IPs stayed the same, with `kube-ovn-cni` logging `iface has no valid IP address`, is a Kube-OVN interface-binding problem and is not covered here; restarting the OVN/OVS containers does not pick up host NIC changes (see *Restarting OVN/OVS containers does not reload host NIC configuration changes on ACP*). + +## Root Cause + +### Cause 1: Network instability between the nodes and the API server VIP + +The kubelet log around the NotReady window contains node-status update failures and connection timeouts to `https://:6443`. **Several nodes go NotReady at the same moment**, while kube-apiserver has not restarted and the masters have low load. Typical kubelet lines: + +```text +"Error updating node status, will retry" err="error getting node \"\": ... net/http: request canceled (Client.Timeout exceeded while awaiting headers)" +"Unable to update node status" err="update node status exceeds retry count" +"Failed to update lease" ... kube-node-lease ... net/http: request canceled +"failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease" +http2: client connection lost +read tcp ...: use of closed network connection +"Failed to make webhook authenticator request" / "Unable to authenticate the request due to an error" +"Error processing volume" ... failed to fetch PVC from API server +dial tcp :6443: connect: no route to host +``` + +Ten nodes can go `NotReady` together, with dozens of health-check, restart, and volume-mount failures in the same minutes; Redis instances can switch master and replica each time their node goes `NotReady`. Even short packet loss between the nodes and `VIP:6443` is enough to make status reports fail; the nodes need continuous, stable reachability to the VIP. Evidence is packet loss between the masters, or a site-wide network outage; the nodes recover by themselves once the network recovers. + +A **security device** on the path causes the same signature. Only some nodes log large numbers of failed connections to the API server and go `NotReady`; their Pods show `TaintManagerEviction: Marking for deletion Pod /` and are recreated elsewhere. A firewall exception does not help, a port check on the VIP side shows nothing, and the apiserver logs have no errors. A packet capture on an affected node shows repeated `[TCP Retransmission]` `[PSH, ACK]` packets from the node to `:6443` with no response: a security device is intercepting that traffic. + +### Cause 2: The node NIC intermittently goes down + +The node goes NotReady for about 30 seconds and recovers on its own. Node monitoring shows **gaps in the metrics** over the same windows, while the monitoring components have not restarted. `dmesg -T` shows the interface (for example `eth0`) not ready, or the link down, at the matching times. Losing link cuts both the kubelet heartbeat and metric collection. Why the NIC loses readiness is outside Kubernetes: the host network configuration, the virtual switch, or the physical link. + +### Cause 3: Underlying storage stalls block the host + +During the incident window the IaaS or storage team reports a storage anomaly, host load rises sharply, `iowait` is high, and the kubelet **stops writing logs** before the node goes NotReady. The cluster status can show `Abnormal`; more than 30 nodes can go NotReady together. When the disks behind the hosts block on I/O, the kubelet cannot keep its heartbeat. A typical cascade: + +1. Cluster memory overcommit above 200% and actual node memory use above 90%, with high Pod density. +2. Heavy page-cache reclaim, which turns into large amounts of disk I/O. +3. The I/O reaches the per-node storage QoS limit (for example 30 MB/s) and exhausts the shared storage bandwidth. +4. I/O blocks, kubelet heartbeats time out, and the nodes become `NotReady`. +5. Eviction and rescheduling load the neighbouring nodes, and the failure spreads. + +On a workload cluster managed from the global cluster, the same chain shows in the KubeadmControlPlane status: `RollingOut`, `Remediating`, `ScalingDown`, and `ScalingUp` are `False`, but: + +```text +type: MachinesReady +status: Unknown +reason: ReadyUnknown +message: ... NodeHealthy: Node.AllConditions: Kubelet stopped posting node status. ... Control plane components: Node is unreachable +``` + +If `dmesg` shows `INFO: task ... blocked for more than 120 seconds` or `nfs: server ... not responding` with many `D`-state processes, see *Node NotReady with high load average and D-state processes on ACP*. If `/var/lib/containerd` returns `Input/output error` and kube-apiserver refuses connections, the host disk itself has failed. + +### Cause 4: The host itself is hung or unreachable + +The node is `NotReady` and **SSH to it fails or hangs**. The platform web CLI cannot reach it either; only the IaaS console shows the VM or server. With no shell on the node, the cause cannot be determined from Kubernetes. The hosts return to normal after a reboot at the infrastructure level. + +### Cause 5: Pods stuck in Terminating on a NotReady or just-restarted node + +Pods that ran on the NotReady node stay `Terminating` and cannot be deleted normally, and replacements for a StatefulSet or a node-pinned workload do not start. Pod deletion is completed by the kubelet on the Pod's node; if that kubelet is down or cannot reach the API server, it can neither stop the containers nor report back, so the Pod object is never finalised. + +A variant appears after a host restart: the old Pod hangs while terminating because the CNI plugin binary is not found during teardown. The Pod shows `0/1` with 0 restarts while other Pods on the node run, and its events repeat: + +```text +SandboxChanged: Pod sandbox changed, it will be killed and re-created. +FailedKillPod: error killing pod: failed to "KillPodSandbox" for "" with KillPodSandboxError: "rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"_\" network: failed to find plugin \"galaxy-veth\" in path [/opt/cni/bin]" +``` + +## Resolution + +### Cause 1: Network instability between the nodes and the API server VIP + +1. Confirm the kubelet errors are connectivity errors and that kube-apiserver and the masters are healthy (Diagnostic Steps 2 to 4). +2. Hand the path between the nodes and `VIP:6443` (switches, load balancer or keepalived VIP, firewalls) to the network team, with the timestamps of the NotReady windows and the packet-loss evidence. +3. If a packet capture shows the node's packets to `:6443` retransmitted with no answer while the VIP side is healthy, have the network and security teams check security devices (IPS, firewalls, anti-intrusion appliances) on the path. Adding the affected nodes to the security device's allow list stops the recurrences. +4. No change on the nodes is needed; they return to `Ready` once the network is stable. Then check that evicted workloads have been recreated: + +```bash +kubectl get nodes +kubectl get pods -A -o wide | grep -viE 'Running|Completed' +``` + +### Cause 2: The node NIC intermittently goes down + +Have the host or IaaS owner check the interface configuration and the virtual or physical link for the interface named in `dmesg`. Kubernetes needs no action; the node recovers each time the link returns, and the recurrences stop once the link is stable. + +### Cause 3: Underlying storage stalls block the host + +1. Reduce cluster load: scale down or stop non-critical workloads, and lower ingress traffic into the cluster. +2. Have the storage team restore the storage queue and free storage bandwidth. +3. Reboot the nodes that do not return to `Ready` by themselves once I/O recovers. + +> **Warning:** Rebooting a node kills every container on it, and single-replica services on that node are unavailable until it returns. Precondition: storage I/O has recovered, and the node has stayed `NotReady` after that or its shell is unresponsive. Reboot one node at a time, starting with nodes that hold no unique stateful replicas. A reboot cannot be rolled back; an unclean reboot may need a filesystem check on boot. + +```bash +# from the IaaS console, or if the shell still responds: +systemctl reboot +``` + +4. Afterwards, remove the memory pressure that started the reclaim storm: lower the overcommit, or add memory or nodes. The root cause of the storage anomaly itself belongs to the infrastructure team. + +### Cause 4: The host itself is hung or unreachable + +Before rebooting, take a screenshot of the IaaS VM console; kernel panic, hung-task, or OOM messages are often visible there. Then reboot from the infrastructure side. + +> **Warning:** A hard reset or power cycle is an unclean shutdown: Pods on the host restart, and in-flight application writes can be lost. Precondition: the node is confirmed unreachable over SSH and from the console, and the infrastructure owner has checked the host. Irreversible. + +After it boots, collect the logs of the previous boot so the hang can be analysed (`journalctl -b -1` needs a persistent journal; otherwise read `/var/log/messages`): + +```bash +journalctl -b -1 -p warning --no-pager | tail -n 300 +journalctl -b -1 -u kubelet --no-pager | tail -n 200 +``` + +### Cause 5: Pods stuck in Terminating on a NotReady or just-restarted node + +- If the node will come back soon (network or IaaS recovery is in progress), wait. When the kubelet reconnects, it completes the pending deletions itself, and the taint clears once the node is `Ready`. +- If the node stays down, or the Pod is stuck after a host restart and nothing else is wrong, force-delete the stuck Pod so its controller can create a new one. + +> **Warning:** A force delete only removes the Pod object from the API server; it does not confirm that the containers are stopped. Precondition: the node is confirmed powered off or isolated, or (after a restart) the old containers are confirmed gone with `crictl ps -a` or `docker ps -a` on the node. Otherwise, for a StatefulSet or any workload writing to shared storage, two instances can run at once and corrupt data. A force delete cannot be undone, but the controller recreates the Pod. + +```bash +kubectl get pods -A -o wide --field-selector spec.nodeName= | grep Terminating +kubectl delete pod -n --grace-period=0 --force +``` + +For the `failed to find plugin "galaxy-veth" in path [/opt/cni/bin]` case after a reboot, force-deleting the Pod with the command above lets a new instance start. Use it only when that event repeats for the old Pod and the node is otherwise `Ready`. Compare `/opt/cni/bin` with a healthy node, in case other Pods on the node hit the same teardown error: + +```bash +kubectl get events -n --field-selector involvedObject.name= | grep -E 'SandboxChanged|FailedKillPod' +ls /opt/cni/bin/ +``` + +## Diagnostic Steps + +1. If the cluster status is abnormal and the CNI is suspected, first test the API VIP on `6443` from a global node and from a node of the affected cluster, and compare with a master IP. A VIP that does not answer while a master IP does is a network-side VIP forwarding fault; nothing inside the cluster needs to change. + +```bash +nc -vz -w 3 6443 +curl -k --max-time 5 https://:6443/healthz || true +curl -k --max-time 5 https://:6443/healthz || true +``` + +2. Find out how many nodes went NotReady, and when. Many nodes at the same moment point to Cause 1 or 3; a single node to Cause 2 or 4. + +```bash +kubectl get nodes -o wide +kubectl get events -A --field-selector reason=NodeNotReady --sort-by=.lastTimestamp +kubectl describe node | sed -n '/Conditions:/,/Addresses:/p' +kubectl describe node | grep -A3 Taints +``` + +3. On an affected node, read the kubelet log for the window. Timeouts to `:6443` and failed status or lease updates are Cause 1. A kubelet that **stops logging altogether** points to I/O blocking (Cause 3) or a hung host (Cause 4). + +```bash +journalctl -u kubelet --since "" --until "" --no-pager \ + | grep -iE 'node status|lease|6443|timeout|deadline' | tail -n 50 +``` + +For a workload cluster shown `Abnormal`, read the KubeadmControlPlane conditions from the global cluster; `MachinesReady` `Unknown` with `Kubelet stopped posting node status` names the control-plane node to look at: + +```bash +kubectl get kubeadmcontrolplane -A +kubectl get kubeadmcontrolplane -n -o yaml | grep -B2 -A8 'type: MachinesReady' +``` + +4. Rule out the API server: the kube-apiserver Pods have not restarted, and the masters are not overloaded. + +```bash +kubectl -n kube-system get pods -l component=kube-apiserver -o wide +kubectl top nodes +``` + +5. Measure the path to the VIP from the affected node and keep it running until the next occurrence. A `000` code or multi-second times that line up with the NotReady windows are Cause 1. + +```bash +ping -i 0.2 -c 300 +while true; do + echo "$(date '+%F %T') $(curl -sk -o /dev/null -w '%{http_code} %{time_total}s' --max-time 5 https://:6443/healthz)" + sleep 1 +done +``` + +If the VIP answers from other hosts but the affected node still times out, capture its traffic during a failure; retransmissions with no reply point to a device on the path: + +```bash +tcpdump -i any -nn host and port 6443 -c 200 -w /tmp/vip-6443.pcap +tcpdump -nn -r /tmp/vip-6443.pcap | head -n 50 +kubectl get events -A --sort-by=.lastTimestamp | grep -E 'TaintManagerEviction|NodeNotReady' | tail -n 20 +``` + +6. Check the kernel log for NIC events at the NotReady times (Cause 2): + +```bash +dmesg -T | grep -iE 'link is not ready|link is down|link is up|NIC Link' +``` + +7. Check for I/O blocking (Cause 3): high `wa` in `top`, high `await` and `%util` in `iostat`, and hung-task messages: + +```bash +top -b -n 1 | head -n 5 +iostat -x 1 5 +dmesg -T | grep -iE 'blocked for more than|hung_task|I/O error' +``` diff --git a/docs/en/solutions/Node_NotReady_with_containerd_not_running_or_cni_plugin_not_initialized_on_ACP.md b/docs/en/solutions/Node_NotReady_with_containerd_not_running_or_cni_plugin_not_initialized_on_ACP.md new file mode 100644 index 000000000..a6f1d2bc3 --- /dev/null +++ b/docs/en/solutions/Node_NotReady_with_containerd_not_running_or_cni_plugin_not_initialized_on_ACP.md @@ -0,0 +1,143 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.x + - 4.x +--- + +# Node NotReady with containerd not running or "cni plugin not initialized" on ACP + +## Issue + +Typical reports: + +- After the cluster was restarted, some nodes stay `NotReady` and cannot be connected. +- In a newly deployed environment, some masters of a workload cluster are `NotReady`; `kubectl describe node` says the CNI is broken, but the OVN containers are Running: + + ```text + Ready False KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized + ``` + +- During an upgrade a node becomes `NotReady` and platform Pods on it (for example `zeus` in `cpaas-system`) fail to start, while `calico-node` is running and logs no errors. The kubelet log repeats: + + ```text + "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" + ``` + +- After a GPU node was rebooted, many Pods on it cannot start, and the containerd log shows the `loopback` CNI plugin crashing. + +The kubelet is running and reports to the API server, but it reports the node as not ready because the container runtime is missing or has not loaded the CNI configuration. Memory, disk, and PID pressure conditions are normal, and the other nodes of the cluster are `Ready`. + +If the runtime runs but hangs (`PLEG is not healthy`, `container runtime is down`), see [Node NotReady on ACP with kubelet "PLEG is not healthy"](Node_NotReady_with_PLEG_is_not_healthy.md). If the kubelet cannot reach the API server at all, see [Node NotReady caused by network, storage or host-level faults on ACP](Node_NotReady_caused_by_network_storage_or_host_level_faults_on_ACP.md). CoreDNS problems right after a node restart are covered in [CoreDNS fails after a node restart or lookups time out when a CoreDNS node is shut down on ACP](CoreDNS_fails_after_a_node_restart_or_lookups_time_out_when_a_CoreDNS_node_is_shut_down_on_ACP.md). + +## Root Cause + +### Cause 1: containerd did not start after a node or cluster restart + +After a cluster or host restart, `systemctl status containerd` on the `NotReady` nodes shows the service `inactive (dead)` or `failed`, and `crictl ps` cannot connect to the runtime socket. Without containerd the kubelet cannot run any Pod, so the node cannot become `Ready`. Why containerd did not start is not known; starting it by hand recovers the node. + +### Cause 2: `cni plugin not initialized` while the kube-ovn CNI Pod is Running + +The node condition shows `NetworkPluginNotReady ... cni plugin not initialized`. At the same time `NetworkUnavailable` is `False` with reason `JoinSubnetGatewayReachable`; the `kube-ovn-cni-` Pod in `kube-system` on that node has its `cni-server` container `Running` with 0 restarts and its init containers `install-cni` and `hostpath-init` completed; and containerd is running. This happened on two of three control-plane nodes of a newly deployed 4.2 workload cluster (Kubernetes `v1.33.6`). containerd had not picked up the CNI configuration that kube-ovn installs, although both looked healthy. Restarting containerd makes the nodes `Ready`; the underlying cause is not confirmed. + +### Cause 3: A corrupted plugin binary in `/opt/cni/bin` (`portmap`, `loopback`) + +A Pod network is set up by running every plugin in the CNI chain, so one damaged binary breaks it even when the main network plugin is healthy. Two forms: + +- `network plugin is not ready: cni config uninitialized` on a Docker-runtime node with Calico during an upgrade. All Calico Pods are Running, the `calico-node` log on the node has no errors, and the Calico configuration file and binaries are present. Restarting `calico-node` and the kubelet does not help. The kubelet also logs `"Error syncing pod, skipping"` for `cpaas-system/zeus-*` Pods and `KillPodSandbox` failures with `networkPlugin cni failed to teardown pod ... network: cni config uninitialized`. `ls -l /opt/cni/bin/` shows `portmap` at 1572864 bytes on the affected node and about 3.6 MB on a healthy node, while the other binaries match. +- Pods cannot start after a reboot of a (GPU) node, and the kubelet and containerd logs show the `loopback` plugin binary crashing during Pod network setup. The `md5sum` of `/opt/cni/bin/loopback` differs from every healthy node. + +How the binaries were damaged is not known: around an upgrade for `portmap`, around a reboot for `loopback`. + +## Resolution + +### Cause 1: containerd did not start after a node or cluster restart + +```bash +systemctl start containerd +systemctl status containerd --no-pager | head -n 5 +systemctl is-enabled containerd +kubectl get node +``` + +The node should return to `Ready`. If `systemctl is-enabled containerd` does not print `enabled`, compare it with a healthy node of the same cluster and enable it the same way (`systemctl enable containerd`); otherwise the node stays `NotReady` again after the next reboot. If containerd fails to start, read `journalctl -u containerd -b --no-pager | tail -n 50` for the reason. + +### Cause 2: `cni plugin not initialized` while the kube-ovn CNI Pod is Running + +Restart containerd on each affected node, one node at a time. + +> **Warning:** Restarting containerd briefly interrupts container operations on the node: for a few seconds no Pods are created, stopped, or probed. Running containers keep running in their shims. Do it only when the node condition shows `cni plugin not initialized` and the `kube-ovn-cni` Pod on that node is Running (step 2 of Diagnostic Steps). No rollback is needed. + +```bash +systemctl restart containerd +systemctl status containerd --no-pager | head -n 5 +kubectl get node +kubectl describe node | grep -E 'Ready|NetworkPluginNotReady' +``` + +If the condition stays, check that the CNI configuration and binaries exist on the node (step 3 of Diagnostic Steps) and compare them with a `Ready` node. + +### Cause 3: A corrupted plugin binary in `/opt/cni/bin` (`portmap`, `loopback`) + +Replace the damaged binary with the copy from a healthy node of the same cluster. The node returns to `Ready` (`portmap`), or the Pods start normally (`loopback`). + +> **Warning:** A binary from a node with a different architecture or CNI version breaks Pod networking on this node. Do it only when the binary's size or checksum differs from healthy nodes (step 4 of Diagnostic Steps), and take the copy from a healthy node of the same cluster, OS, and architecture. Keep the old file to roll back. + +```bash +cp -a /opt/cni/bin/portmap /root/portmap.corrupt.$(date +%s) +scp :/opt/cni/bin/portmap /opt/cni/bin/portmap +chmod 755 /opt/cni/bin/portmap +md5sum /opt/cni/bin/portmap # must match the healthy node +kubectl get node +``` + +Use the same steps for `loopback` or any other binary that differs. Compare all binaries at once, because more than one can be damaged. + +## Diagnostic Steps + +1. Read the node condition and the runtime state: + + ```bash + kubectl get nodes -o wide + kubectl describe node | sed -n '/Conditions:/,/Addresses:/p' + systemctl status containerd kubelet --no-pager | grep -E '●|Active' + journalctl -u kubelet --since "-30min" --no-pager | grep -iE 'NetworkPluginNotReady|cni|runtime' | tail -n 20 + ``` + + `containerd` not active: Cause 1. `cni plugin not initialized` with containerd active: Cause 2. `cni config uninitialized` with a healthy Calico, or a plugin binary crashing: Cause 3. + +2. Check the kube-ovn CNI Pod on the node: + + ```bash + kubectl -n kube-system get pods -o wide --field-selector spec.nodeName= | grep kube-ovn-cni + crictl ps -a | grep -E 'cni-server|install-cni|hostpath-init' + ``` + + Expected for Cause 2: `cni-server` Running, `install-cni` and `hostpath-init` Exited (completed). + +3. Check the CNI configuration and plugin binaries containerd loads, and what it reports about loading them: + + ```bash + ls -l /etc/cni/net.d/ /opt/cni/bin/ + crictl info | grep -iE -A3 'lastCNILoadStatus|NetworkReady' + journalctl -u containerd --since "-30min" --no-pager | grep -i cni | tail -n 20 + ``` + + Compare the file lists with a `Ready` node of the same cluster. + +4. Compare every CNI plugin binary with a healthy node (Cause 3). A size or checksum that differs is the damaged one: + + ```bash + md5sum /opt/cni/bin/* > /tmp/cni-$(hostname).md5 + ssh 'md5sum /opt/cni/bin/*' > /tmp/cni-healthy.md5 + diff /tmp/cni-healthy.md5 /tmp/cni-$(hostname).md5 + grep -o '"type": *"[a-z0-9-]*"' /etc/cni/net.d/*.conflist # plugins in the chain + journalctl -u kubelet --since "-30min" --no-pager | grep -E 'cni config uninitialized|loopback|portmap' | tail -n 10 + ``` + + No `diff` output means the binaries match the healthy node. + +5. Runtime binaries that were replaced or conflict (Docker on a containerd node, overwritten `runc`) are covered in [Container runtime conflicts on ACP nodes: Docker, podman, docker-runc, and overwritten runtime binaries](Container_runtime_conflicts_on_ACP_nodes_Docker_podman_docker-runc_and_overwritten_runtime_binaries.md). diff --git a/docs/en/solutions/Node_add_check_times_out_immediately_because_the_cluster_exec_API_returns_HTTP_500_on_ACP.md b/docs/en/solutions/Node_add_check_times_out_immediately_because_the_cluster_exec_API_returns_HTTP_500_on_ACP.md new file mode 100644 index 000000000..84b771705 --- /dev/null +++ b/docs/en/solutions/Node_add_check_times_out_immediately_because_the_cluster_exec_API_returns_HTTP_500_on_ACP.md @@ -0,0 +1,84 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Node add check times out immediately because the cluster exec API returns HTTP 500 on ACP + +## Issue + +Adding a node to a cluster fails at the availability check with no delay. The add-node dialog returns `检查超时,请重试` ("check timed out, please retry") at once, while it still shows `节点可用性检查中...` ("node availability check in progress"). The node is reachable, and the error does not match the network state. + +In the browser developer tools, the request of the add-node dialog fails: + +```text +POST /apis/platform.tkestack.io/v1/clusters//exec?ignore_warning=true +500 Internal Server Error +``` + +The `tke-platform-api` log on the `global` cluster has no x509 error for `https://localhost:9443/...`. + +`tke-platform-api` and `tke-platform-controller` run on the `global` cluster in `cpaas-system`. On some `global` clusters (for example on ACP 4.2) there is no `tke-platform-api` Pod, and `cluster-api-provider-alauda` manages node addition instead; check which one yours runs with `kubectl get deploy -A | grep -E 'tke-platform|cluster-api-provider-alauda'`. + +Related articles: [tke-platform-api serves an expired certificate after running for a year on ACP](tke-platform-api_serves_an_expired_certificate_after_running_for_a_year_on_ACP.md), for the same immediate check timeout when the `tke-platform-api` log shows x509 errors against `localhost:9443`. + +## Root Cause + +### Cause 1: the cluster exec API of tke-platform-api returns 500 with no log cause + +The add-node dialog runs its availability check through the cluster `exec` API (`/apis/platform.tkestack.io/v1/clusters//exec`), which `tke-platform-api` serves. When that API returns HTTP 500, the dialog reports it as an immediate check timeout instead of a real one. No underlying cause has been identified for a 500 that comes without a `localhost:9443` x509 line in the log. Restarting `tke-platform-api` restores the `exec` API, and the node availability check recovers. + +## Resolution + +### Cause 1: the cluster exec API of tke-platform-api returns 500 with no log cause + +1. Confirm the 500 in the browser developer tools: the `exec?ignore_warning=true` request of the add-node dialog. + +2. Read the `tke-platform-api` log on `global`. If it shows x509 errors for `localhost:9443` (`CertificateInvalidError` or `certificate has expired`), follow the related article on the expired certificate instead. + + ```bash + kubectl -n cpaas-system get pods -o wide | grep tke-platform-api + kubectl -n cpaas-system logs deploy/tke-platform-api --tail=300 | grep -i -E 'exec|error|panic' + kubectl -n cpaas-system logs deploy/tke-platform-api --tail=500 | grep -E 'localhost:9443|CertificateInvalidError|certificate has expired' + ``` + +3. Check that the clock is correct on `global` and on the node, and that the node is reachable over SSH from `global`: + + ```bash + date -u + ``` + +4. Restart `tke-platform-api`. + + > **Warning:** Restarting `tke-platform-api` briefly interrupts cluster and node management on `global` for every cluster; running workloads are not affected. Apply only when the `exec` request returns 500 and the clock and the network to the node have been checked. Rollback: none needed; the Deployment recreates the Pods. + + ```bash + kubectl -n cpaas-system rollout restart deployment/tke-platform-api + kubectl -n cpaas-system rollout status deployment/tke-platform-api + ``` + +5. Retry the node addition. If the `exec` request still returns 500, collect the `tke-platform-api` log from the time of the request and contact support. + +## Diagnostic Steps + +```bash +# global cluster +kubectl get deploy -A | grep -E 'tke-platform|cluster-api-provider-alauda' +kubectl -n cpaas-system get pods -o wide | grep tke-platform-api +kubectl -n cpaas-system logs deploy/tke-platform-api --tail=500 | grep -E 'localhost:9443|CertificateInvalidError|certificate has expired|panic' +date -u +``` + +- The `exec` request returns 500 at once and the log has no `localhost:9443` x509 line: this article. +- The `exec` request returns 500 and the log shows `localhost:9443` with an x509 error: see [tke-platform-api serves an expired certificate after running for a year on ACP](tke-platform-api_serves_an_expired_certificate_after_running_for_a_year_on_ACP.md). +- `date -u` on `global` or on the node is wrong: fix the clock; this article does not apply. +- The `exec` request returns 200, or 504 after a long wait: a real check timeout; check SSH and network reachability to the node. +- There is no `tke-platform-api` Deployment and `cluster-api-provider-alauda` runs instead: this article does not apply; node addition is managed by `cluster-api-provider-alauda`. diff --git a/docs/en/solutions/Node_capacity_pod_limit_or_data_disk_after_resizing_or_replacing_node_hardware_on_ACP.md b/docs/en/solutions/Node_capacity_pod_limit_or_data_disk_after_resizing_or_replacing_node_hardware_on_ACP.md new file mode 100644 index 000000000..5125dbf25 --- /dev/null +++ b/docs/en/solutions/Node_capacity_pod_limit_or_data_disk_after_resizing_or_replacing_node_hardware_on_ACP.md @@ -0,0 +1,170 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.x + - 4.x +--- + +# Node capacity, pod limit or data disk after resizing or replacing node hardware on ACP + +## Issue + +After hardware work on a node, the platform does not reflect the change, or the node needs its data disk rebuilt: + +- The node's CPU or memory was expanded, but the cluster still shows the old values. For example, a node expanded from 16C32G to 48C96G shows 48 cores on the host and 16 in the console; or `free -g` already shows the new memory, while the node details page shows `31.27 Gi` and the node list tooltip `63.27Gi` (Cause 1). +- Pods cannot be created because the node reached its Pod limit, or a large node (for example 1T of memory with 256 Pods at 50% memory use) should run more Pods. Is the limit a cluster or a node setting, and does an upgrade reset it? (Cause 2) +- A passthrough data disk that held `/var/lib/docker` and `/cpaas` was replaced online with a blank disk, and the node has to be recovered (Cause 3). + +Related articles: *Shutting down or rebooting ACP nodes for maintenance* (cordon, drain, and node-by-node reboots), *Safely resizing CPU and memory on Alauda Container Platform nodes*, and *Changing the kubelet maxPods setting on immutable infrastructure nodes*. + +## Root Cause + +### Cause 1: The kubelet still reports the old capacity after a CPU or memory resize + +The kubelet reports node capacity from what the OS exposes, and keeps reporting the old capacity until it is restarted. The OS itself may not expose hot-added resources: added CPUs can stay offline, and the kubelet only counts online CPUs. Memory hot-added to a VM (for example on a hyper-converged platform) can show in `free` while the node object keeps the old `Capacity` (for example `memory: 32792048Ki`, `Allocatable` `memory: 29842928Ki`) even after a kubelet restart; only a node reboot makes Kubernetes report the correct memory. + +### Cause 2: maxPods in the node's kubelet configuration limits the Pod count + +The Pod limit shown on the node details page (allocated and allocatable, for example `allocated: 224; allocatable: 256`) comes from `maxPods` in that node's `/var/lib/kubelet/config.yaml`, which is why it can differ between nodes and between similar clusters (for example 256 on some nodes and 210 on others): + +- It is a **per-node** setting; the platform has no function to change it on all nodes of a cluster in bulk. +- It can be raised above 256, for example to 300 or 512. The node still fills up by resource requests: if Pods request large amounts of CPU or memory, the node is full before it reaches the new maximum. +- Upgrades normally do not touch this setting. If an upgrade does revert it, change it again by hand. + +### Cause 3: The data disk behind /var/lib/docker and /cpaas was replaced with a blank disk + +The node's runtime data (`/var/lib/docker`) and platform data (`/cpaas`) lived on two partitions of a passthrough disk. After an online replacement with a blank disk, the original data is gone and the mounts point at nothing. The new disk has to be partitioned, formatted, and mounted to the original plan with the runtime and kubelet stopped, after which images and Pod data are pulled and recreated. + +The order matters for any disk change on a node. When a raw disk is added to a containerd node while kubelet and containerd were not stopped correctly and the containers on the node were not deleted, the platform registry's MinIO instances can fail to form a cluster, and restarted component Pods then stay pulling images. Stop kubelet, delete the containers on the node, and stop containerd before the disk change. Recovering the registry MinIO cluster after it has broken is not covered here. + +## Resolution + +### Cause 1: The kubelet still reports the old capacity after a CPU or memory resize + +Restarting the kubelet is a required step after expanding a host's resources. + +> **Warning:** Restarting the kubelet does not stop running containers, but the node can briefly report `NotReady`, and probes and status updates pause for a few seconds. No rollback is needed. + +```bash +systemctl daemon-reload +systemctl restart kubelet.service +kubectl describe node | grep -E -A7 '^(Capacity|Allocatable):' +``` + +If the value is still old, check what the OS sees: + +```bash +lscpu | grep -iE '^CPU\(s\)|On-line|Off-line' +free -h +``` + +- **`Off-line CPU(s) list` is not empty** (for example 16 CPUs with 8-15 offline): bring the added CPUs online and restart the kubelet again, or reboot the node. + +```bash +for c in /sys/devices/system/cpu/cpu[0-9]*/online; do [ "$(cat $c)" = 0 ] && echo 1 > $c; done +lscpu | grep -iE 'On-line|Off-line' +systemctl restart kubelet.service +``` + +- **Memory was hot-added and the node object still shows the old value after the kubelet restart**: reboot the node with the cordon, drain, and reboot procedure in *Shutting down or rebooting ACP nodes for maintenance*. + +If the platform's monitoring view of the node fails with `dial tcp :11780: connect: connection refused` (HTTP 503 `ServiceUnavailable`), that is a separate problem: the load balancer's port forwarding for port 11780 to the cluster's Prometheus is misconfigured. Fix the load balancer rule; node capacity is not involved. + +### Cause 2: maxPods in the node's kubelet configuration limits the Pod count + +Raise `maxPods` in the kubelet configuration on each node that needs it. Add the key at the top level of the `KubeletConfiguration` if it is absent. + +> **Warning:** A syntax error in `/var/lib/kubelet/config.yaml` stops the kubelet, and the node goes `NotReady`. Precondition: keep a backup copy, and make sure the Pod network has enough IP addresses for the higher Pod count. Roll back by restoring the copy and restarting the kubelet. + +```bash +cp -a /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.bak +grep -n -i 'max' /var/lib/kubelet/config.yaml +vi /var/lib/kubelet/config.yaml # set or add: maxPods: +systemctl restart kubelet +kubectl describe node | grep -iE -A6 'Capacity|Allocatable' +``` + +Expected, for `maxPods: 210`: `pods: 210` under both `Capacity` and `Allocatable`. When the other nodes cannot take the Pods because their TopoLVM space is nearly exhausted, raising the Pod limit on the nodes that still have space lets the Pods be placed. + +On immutable-infrastructure nodes managed by Machine Configuration (for example HCS clusters), node files are reconciled by the platform and a direct edit is configuration drift; change `maxPods` there with *Changing the kubelet maxPods setting on immutable infrastructure nodes*. + +### Cause 3: The data disk behind /var/lib/docker and /cpaas was replaced with a blank disk + +Drain the node first (see *Shutting down or rebooting ACP nodes for maintenance*). + +> **Warning:** Partitioning and formatting destroy everything on the target disk. Precondition: `lsblk -f` confirms the target is the new, empty disk (no filesystem, not mounted, expected size), not a disk in use, and the node is drained. Partitioning and formatting are irreversible. + +1. Stop the runtime and the kubelet, and keep them from starting at boot: + +```bash +systemctl stop kubelet docker +systemctl disable kubelet docker +``` + +2. If anything readable is still under `/var/lib/docker` or `/cpaas`, copy it elsewhere first. If the data is already lost, skip this and the copy-back later. + +3. Unmount the old disk and clean up the old directories: + +```bash +umount /var/lib/docker /cpaas 2>/dev/null || umount -l /var/lib/docker /cpaas +rm -rf /var/lib/docker/* # only if anything is left on the root filesystem under the mount point +``` + +4. Partition and format the new disk to the original plan, with the same partition sizes and filesystem types as a healthy node (`lsblk -f` there). For Docker `overlay2` on XFS, the filesystem must be created with `ftype=1`: + +```bash +lsblk -f +parted -s /dev/ mklabel gpt +parted -s /dev/ mkpart docker 0% +parted -s /dev/ mkpart cpaas 100% +mkfs.xfs -n ftype=1 /dev/ +mkfs.xfs /dev/ +``` + +5. Point `/etc/fstab` at the new partitions by UUID, replacing the old entries, and mount them: + +```bash +blkid /dev/ /dev/ +vi /etc/fstab # UUID= /var/lib/docker xfs defaults 0 0 ; UUID= /cpaas xfs defaults 0 0 +mount -a +findmnt /var/lib/docker; findmnt /cpaas +``` + +6. Copy back any backed-up data, then start and re-enable the services and uncordon the node: + +```bash +systemctl enable --now docker +systemctl enable --now kubelet +kubectl uncordon +``` + +The node then runs from the new data disk, and images and Pod data are pulled and recreated. + +## Diagnostic Steps + +1. Compare what the node object reports with what the host sees. A mismatch after a resize is Cause 1. + +```bash +kubectl describe node | grep -E -A7 '^(Capacity|Allocatable):' +lscpu | grep -iE '^CPU\(s\)|On-line|Off-line' +free -g +``` + +2. For Pods that cannot be created on a node, compare the running Pod count with `Allocatable: pods`, and read `maxPods`. A node at its Pod count is Cause 2. + +```bash +kubectl get pods -A -o wide --field-selector spec.nodeName= --no-headers | wc -l +kubectl describe node | grep -iE -A6 'Allocatable' +grep -n -i 'maxPods' /var/lib/kubelet/config.yaml +``` + +3. After a disk replacement, check what is mounted where and compare with a healthy node. Missing or empty `/var/lib/docker` and `/cpaas` mounts are Cause 3. + +```bash +lsblk -f +findmnt /var/lib/docker; findmnt /cpaas +grep -E 'docker|cpaas' /etc/fstab +``` diff --git a/docs/en/solutions/Node_disk_alert_notification_shows_an_empty_mountpoint_or_no_value_on_ACP.md b/docs/en/solutions/Node_disk_alert_notification_shows_an_empty_mountpoint_or_no_value_on_ACP.md new file mode 100644 index 000000000..8643c6b1c --- /dev/null +++ b/docs/en/solutions/Node_disk_alert_notification_shows_an_empty_mountpoint_or_no_value_on_ACP.md @@ -0,0 +1,95 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" +--- + +# Node disk alert notification shows an empty mountpoint or no value on ACP + +## Issue + +The built-in node disk alert (policy `cpaas-node-rules`, indicator `node.disk.space.utilization`) fires for disk usage above the threshold, but the mountpoint inside the summary is blank: + +```text +磁盘分区(挂载点 )的空间使用率超过80.00% +``` + +(disk partition (mountpoint ) space usage exceeds 80.00%), with a trigger value such as `80.20%`. A webhook forward of the same alert shows `no value` where the disk path should be. The `courier-api` log line `Try to record` for the alert lists labels such as `device: /dev/mapper/-`, `alert_resource: cpaas-node-rules`, and `alert_involved_object_kind: Node`, with no `mountpoint`, and status `Ready`, phase `Done`: the message was delivered, only the field is empty. + +Related article: [Notification template cannot show metric labels such as job, instance, or the workload name on ACP](Notification_template_cannot_show_metric_labels_such_as_job_instance_or_the_workload_name_on_ACP.md). + +## Root Cause + +### Cause 1: the disk summary references mountpoint, and the alert series no longer has that label + +The `node.disk.space.utilization` expression in the PrometheusRule `cpaas-system/cpaas-node-rules` has this shape: + +```promql +max by (node_name,host_ip,node,ip,instance,device) (avg by(node_name,host_ip,node,ip,instance,device,mountpoint) (node_filesystem_used{node=~".*",device!~"rootfs|tmpfs|shm",mountpoint!~"/run/containerd/.*"}) / avg by(node_name,host_ip,node,ip,instance,device,mountpoint) (node_filesystem_size_bytes{...}))>0.5 +``` + +The outer `max by (...)` aggregates by `device` and drops `mountpoint`, so the result series carry only `device`, `host_ip`, `instance`, `ip`, `node`, and `node_name`. The alert summary comes from the ConfigMap `indicators-standard-node-alerts` in `cpaas-system` on the `global` cluster. Under the indicator `node.disk.space.utilization`, its `summaryZH` is `磁盘分区(挂载点{{.labels.mountpoint}})的空间使用率{{.externalLabels.comparison}}{{.externalLabels.threshold}}`, and `summaryEN` references `{{.labels.mountpoint}}` as well. With the label absent, `{{ .externalLabels.summary }}` renders an empty mountpoint. + +Other ConfigMaps with indicator definitions are not read for this summary, so editing them has no effect. + +## Resolution + +### Cause 1: the disk summary references mountpoint, and the alert series no longer has that label + +1. Read the rule expression, the summary, and the live labels: + + ```bash + kubectl get prometheusrule -n cpaas-system cpaas-node-rules -o yaml | grep node.disk.space.utilization -A 10 + kubectl get cm -n cpaas-system indicators-standard-node-alerts -o yaml | grep node.disk.space.utilization -C10 + ``` + + Run the disk expression from the PrometheusRule in the metrics UI. Continue only if `mountpoint` is absent and `device` is present in the result. + +2. On the `global` cluster, export the ConfigMap: + + ```bash + kubectl get cm -n cpaas-system indicators-standard-node-alerts -o yaml > /tmp/indicators-standard-node-alerts.yaml + ``` + +3. Under `node.disk.space.utilization`, replace `{{.labels.mountpoint}}` with `{{.labels.device}}` in `summaryZH`, and in `summaryEN` if it also references `mountpoint`. + + > **Warning:** This changes the summary text of all node disk alerts, and a wrong label name keeps printing a blank or `no value`. Do this only when the query result has `device` and not `mountpoint`. To roll back, run `kubectl apply -f /tmp/indicators-standard-node-alerts.yaml`. A platform upgrade can restore the shipped copy of this ConfigMap. + + ```bash + kubectl edit cm -n cpaas-system indicators-standard-node-alerts + ``` + + The resulting `summaryZH`: + + ```text + 磁盘分区(挂载点{{.labels.device}})的空间使用率{{.externalLabels.comparison}}{{.externalLabels.threshold}} + ``` + +4. Confirm on a business cluster that the indicator definition shows the new text: + + ```bash + kubectl get cm -n cpaas-system indicators-standard-node-alerts -o yaml | grep -A3 node.disk.space.utilization + ``` + +5. Wait for the next node disk alert, or test as below. The alert now shows the device, for example `磁盘分区(挂载点/dev/vda15)的空间使用率超过1.00%`, in firing and resolved notifications. + + To test immediately, lower the threshold of the disk rule in `cpaas-node-rules` temporarily. + + > **Warning:** A lower threshold fires a node disk alert for every node and filesystem above the new value, and each is sent to every notification policy bound to `cpaas-node-rules`. Do this only to verify the summary change, after writing down the current threshold and duration and warning the recipients. To roll back, set the threshold back (80% for 1 minute by default); the test alerts then resolve. + +This shows the device instead of the mountpoint; it does not restore a mountpoint value. Only filesystems that the collection rules keep appear: a partition without a filesystem, or the parent disk itself, has no alert series. + +## Diagnostic Steps + +1. Run the disk expression from `cpaas-node-rules` in the metrics UI. Expected for this issue: the series have `device` and no `mountpoint`. + +2. Read the summary in `indicators-standard-node-alerts`. Expected for this issue: it references `{{.labels.mountpoint}}`. + +3. Find the `Try to record` line for the alert in the `courier-api` log. Phase `Done` confirms the notification was delivered and only the field is empty: + + ```bash + kubectl logs -n cpaas-system deploy/courier-api --tail=500 | grep 'node.disk.space.utilization' + ``` diff --git a/docs/en/solutions/Node_events_show_CIDRNotAvailable_and_some_nodes_have_an_empty_podCIDR_on_a_Kube-OVN_cluster_on_ACP.md b/docs/en/solutions/Node_events_show_CIDRNotAvailable_and_some_nodes_have_an_empty_podCIDR_on_a_Kube-OVN_cluster_on_ACP.md new file mode 100644 index 000000000..e4e1e1a14 --- /dev/null +++ b/docs/en/solutions/Node_events_show_CIDRNotAvailable_and_some_nodes_have_an_empty_podCIDR_on_a_Kube-OVN_cluster_on_ACP.md @@ -0,0 +1,79 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Node events show CIDRNotAvailable and some nodes have an empty podCIDR on a Kube-OVN cluster on ACP + +## Issue + +On a cluster whose CNI is Kube-OVN, node events with reason `CIDRNotAvailable` appear, and the following command returns a value on some nodes and nothing on others: + +```bash +kubectl get node -o jsonpath='{.spec.podCIDR}{"\n"}{.spec.podCIDRs}{"\n"}' +``` + +The question is whether these events and the empty fields affect the cluster network. + +Related articles: "Kube-OVN Pods fail with AddressOutOfRange, AddressConflict or no address allocated on ACP". + +## Root Cause + +### Cause 1: CIDRNotAvailable comes from Kubernetes node CIDR allocation, which Kube-OVN does not use + +In Kubernetes, `spec.podCIDR` and `spec.podCIDRs` on a Node are filled by the node IPAM controller of kube-controller-manager when node CIDR allocation is enabled (`--allocate-node-cidrs` with a `--cluster-cidr`). When that controller cannot assign a range to a node, it records a `CIDRNotAvailable` event for the node and leaves the field empty. + +Kube-OVN does not take Pod addresses from the Node `podCIDR`. It allocates them from its own `Subnet` resources (the default subnet, or the subnet bound to the project or namespace). For that reason the `CIDRNotAvailable` events have no impact on the Kube-OVN cluster network itself. Why the controller could not allocate a range in a particular cluster is a separate question and does not change this conclusion. + +## Resolution + +### Cause 1: CIDRNotAvailable comes from Kubernetes node CIDR allocation, which Kube-OVN does not use + +No change is required for the Kube-OVN network. Confirm that Pod networking is healthy on the nodes with an empty `podCIDR`: + +1. Pods on such a node have addresses and are `Running`: + + ```bash + kubectl get pods -A -o wide --field-selector spec.nodeName= + ``` + +2. The Kube-OVN subnets are ready and have free addresses: + + ```bash + kubectl get subnet + ``` + +3. `kube-ovn-controller` reports no allocation failures: + + ```bash + kubectl -n kube-system logs deploy/kube-ovn-controller --tail=500 | grep -iE 'no address allocated|NoAvailableAddress' + ``` + +If all three are clean, the events can be ignored for the network. If Pods do fail to get addresses, the cause is in Kube-OVN allocation, not in the Node `podCIDR`; see the related article. + +## Diagnostic Steps + +1. Compare the nodes that have a `podCIDR` with those that do not: + + ```bash + kubectl get nodes -o custom-columns='NAME:.metadata.name,PODCIDR:.spec.podCIDR,PODCIDRS:.spec.podCIDRs' + kubectl get events -A --field-selector reason=CIDRNotAvailable + ``` + +2. Check whether kube-controller-manager allocates node CIDRs, and from which range: + + ```bash + kubectl -n kube-system get pods -l component=kube-controller-manager -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.containers[*].command}{"\n"}{end}' | tr ',' '\n' | grep -E 'allocate-node-cidrs|cluster-cidr|node-cidr-mask-size' + ``` + +3. Confirm the cluster CNI is Kube-OVN and Pod addresses come from its subnets: + + ```bash + kubectl get subnet -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidrBlock,NS:.spec.namespaces' + kubectl get pods -A -o wide | head + ``` + +Pod addresses inside the Kube-OVN subnet CIDRs on nodes with and without a `podCIDR` confirm that the events do not affect the cluster network. diff --git a/docs/en/solutions/Node_firewall_or_iptables_rules_block_kubelet_10250_metrics_10254_or_control_plane_traffic_on_ACP.md b/docs/en/solutions/Node_firewall_or_iptables_rules_block_kubelet_10250_metrics_10254_or_control_plane_traffic_on_ACP.md new file mode 100644 index 000000000..d9568a023 --- /dev/null +++ b/docs/en/solutions/Node_firewall_or_iptables_rules_block_kubelet_10250_metrics_10254_or_control_plane_traffic_on_ACP.md @@ -0,0 +1,203 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# Node firewall or iptables rules block kubelet 10250, metrics 10254 or control-plane traffic on ACP + +## Issue + +A firewall or extra iptables rules on the cluster nodes themselves (not a firewall between sites) break part of the cluster. Typical reports: + +- Web CLI to a node fails, and logs of Pods on that node cannot be viewed in the platform. The node is still `Ready` and scheduling works. The Web CLI session shows the banner `远程终端连接已断开` (remote terminal disconnected) and: + + ```text + warning: couldn't attach to pod/nsenter-, falling back to streaming logs: Internal error occurred: error sending request: Post "https://:10250/attach/cpaas-system/nsenter-/nsenter?input=1&output=1&tty=1": dial tcp :10250: connect: no route to host + Error from server: Get "https://:10250/containerLogs/cpaas-system/nsenter-/nsenter": dial tcp :10250: connect: no route to host + ``` + +- Container services on a two-node deployment look hung with no error, one of the nodes is not displayed, and container logs cannot be viewed. +- `kube-ovn-pinger` does not start on some nodes of a new cluster, and `kubectl` on the master cannot fetch logs from those nodes, while the nodes show `Ready`. +- Platform inspection, check "Status of monitor related component", returns Warning with `Availability of service(cpaas-system/alb2-cpaas-system) for monitoring component to get Metrics is not working properly.`, and the same for the ingress-nginx controller `-metrics` Services. No functional impact is visible. +- The production API server is down, and traffic from one master to the other masters does not pass. +- Alerts fire after iptables rules were added to the `raw` table to stop outside access to the nodes, and they do not clear at once after the rules are removed. + +## Root Cause + +### Cause 1: dial tcp :10250: connect: no route to host + +Exec (Web CLI) and log retrieval are API server to kubelet connections on `10250`. From the global cluster and from other nodes, TCP to `10250` on the affected node fails (`telnet: Unable to connect to remote host: No route to host`), while `telnet 10250` on the node itself connects. In the other direction the node reaches the API server on `6443`, so kubelet keeps reporting status and the node stays `Ready`. + +The block can be: + +- iptables rules on the node that reject all external access to `10250`. `no route to host` while the host is up is what a `REJECT` with `icmp-host-prohibited` returns, which is the reject type firewalld uses by default. +- a network policy outside the node that has not been fully opened from the masters to the nodes. Then `telnet 10250` from the master stays at `Trying ...` instead of failing fast, and a `kube-ovn-pinger` on a working node reports `loss count 3` for pings to the pinger Pods and the node addresses of the affected nodes. + +### Cause 2: kube-ovn-pinger still cannot reach the API server Service after 10250 is open + +After `10250` is opened, the logs become readable and show `kube-ovn-pinger` failing to connect to the API server Service, while TCP to the API VIP on `6443` works from the same nodes. Restarting the Kube-OVN `ovs-ovn`, `kube-ovn-cni` and `kube-ovn-pinger` Pods on those nodes restores them; why the Service path failed before the restart was not determined. + +A `curl` from a container on such a node that returns `curl: (60) SSL certificate problem: unable to get local issuer certificate` for `https://:443` or `https://:6443` shows that the path is reachable: the error comes after the TCP and TLS exchange started, and only means `curl` has no CA bundle for the cluster certificate. + +### Cause 3: monitoring cannot scrape metrics on 10254 because the node firewall does not allow it + +In the workload cluster VMUI, the job `-ingress-nginx-controller-metrics` shows `up` at `0`. The controller Pods are `1/1 Running`, and the `*-controller-metrics` Services are `ClusterIP` on `10254/TCP`. From another node, port `80` of the Pod IP returns nginx `404 Not Found`, but `curl :10254/metrics` returns `curl: (7) Failed to connect to port 10254 after 0 ms: Could not connect to server`. `curl 127.0.0.1:10254/metrics` inside the Pod returns data, and so does the same request from the Pod's own node. + +On the node, `systemctl status firewalld` is `active (running)` and `firewall-cmd --list-all` shows zone `public (active)` with no `10254` in its port list. The node firewall does not allow `10254/tcp`, so the monitoring component cannot reach the metrics endpoints from outside the node. (The `alb2-cpaas-system` warning in the same inspection can appear while its VMUI job still returns data.) + +### Cause 4: a firewall enabled on the masters blocks master-to-master traffic and the API server goes down + +A firewall enabled on the masters blocks the traffic between them, the API server becomes unavailable, and traffic from one master to the others does not pass. + +### Cause 5: iptables raw table rules raise alerts and the cluster recovers slowly after their removal + +Rules in the `raw` table `PREROUTING` chain see every packet entering the node before connection tracking and before the kube-proxy and CNI chains, including Service, tunnel and Pod traffic. Rules added there to block outside access to in-cluster services also disturb the cluster, and high-load alerts fire. After the rules are removed, the cluster returns to normal gradually rather than at once. + +Restricting all ports on all nodes with iptables without affecting the cluster has no product solution. To restrict access to workload services, use **Network > Network Policies** in the Container Platform view; network policies cover the ports that workloads expose, not every port on the node. + +## Resolution + +### Cause 1: dial tcp :10250: connect: no route to host + +Only when `10250` fails from other nodes, answers locally, and the node is `Ready`. + +- **Block outside the node** (network policy): open `10250` from the control-plane nodes to every node. +- **Block on the node**: find the rule that rejects `10250`, or rejects everything after an allow list, and remove only that rule by line number. + + > **Warning:** Flushing a whole table (`iptables -F`) also removes the kube-proxy and CNI chains until those components re-sync, which interrupts Service and Pod traffic on the node. Deleting the wrong rule can open the node wider than intended. Save the rules first and identify the rule by line number. To roll back, run `iptables-restore < /root/iptables.before`. + + ```bash + iptables-save > /root/iptables.before + iptables -L INPUT -n --line-numbers | grep -E 'REJECT|DROP|10250' + iptables -D INPUT + ``` + +- **firewalld on the node**: add the port, scoped to the cluster node range. + + > **Warning:** A rich rule with a source range that is too wide exposes the kubelet API on that node. Save the current configuration first. To roll back, run the same command with `--remove-rich-rule` and reload. + + ```bash + firewall-cmd --state + firewall-cmd --list-all > /root/firewalld.before.txt + firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="" port port="10250" protocol="tcp" accept' + firewall-cmd --reload + ``` + +Persist an iptables change with the mechanism the node already uses, then test from a master and from the global cluster: + +```bash +nc -vz -w 3 10250 +kubectl logs -n --tail=5 +``` + +### Cause 2: kube-ovn-pinger still cannot reach the API server Service after 10250 is open + +Only after Cause 1 is fixed and `kube-ovn-pinger` on the same nodes still logs API server Service connection failures while the VIP on `6443` answers. Restart the three Kube-OVN Pods on each affected node. + +> **Warning:** Restarting `ovs-ovn` and `kube-ovn-cni` on a node interrupts Pod networking on that node until they are Ready again, and new Pods cannot get networking meanwhile. Do one node at a time. No rollback is needed; the DaemonSets recreate the Pods. + +```bash +kubectl -n kube-system get pod -o wide | grep -E 'ovs-ovn|kube-ovn-cni|kube-ovn-pinger' | grep +kubectl -n kube-system delete pod +kubectl -n kube-system get pod -o wide | grep +``` + +### Cause 3: monitoring cannot scrape metrics on 10254 because the node firewall does not allow it + +Only when `10254/metrics` answers inside the Pod and on its node, and fails from another node. Allow `10254` on the node firewall from the addresses the monitoring component scrapes from. + +> **Warning:** Opening a port on the node firewall widens what can reach the node. Limit the source to the cluster range and save the current configuration. To roll back, remove the rule (`--remove-rich-rule` with the same text) and reload. + +```bash +firewall-cmd --list-all > /root/firewalld.before.txt +firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="" port port="10254" protocol="tcp" accept' +firewall-cmd --reload +curl -s http://:10254/metrics | head -5 +``` + +If the site's rule is simply to open the port in the active zone, the plain form is `firewall-cmd --permanent --zone=public --add-port=10254/tcp` followed by `firewall-cmd --reload` (roll back with `--remove-port=10254/tcp`). Afterwards, `up` for the `-controller-metrics` job in VMUI should return to `1`; re-run the inspection. + +### Cause 4: a firewall enabled on the masters blocks master-to-master traffic and the API server goes down + +Only when the firewall on the masters is active and master-to-master traffic fails. Restore the traffic between the masters: either return the firewall to its previous state, or allow the control-plane ports between the master addresses. In general Kubernetes terms these are `6443` (API server), `2379` and `2380` (etcd) and `10250` (kubelet), plus the CNI traffic between nodes; the full list for ACP nodes is the communication matrix in the installation documentation. + +> **Warning:** Rules on a broken control plane are added node by node; keep console or out-of-band access to each master and agree with the security owner on the option used. To roll back, remove the added rules. + +```bash +systemctl is-active firewalld +firewall-cmd --list-all +for p in 6443 2379 2380 10250; do nc -vz -w 3 $p; done +firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="" accept' +firewall-cmd --reload +``` + +If the site decides to turn the firewall off instead: + +> **Warning:** Stopping and disabling firewalld removes every rule it enforced on that master, now and after reboot. Only when the security owner accepts an unfiltered node and `firewall-cmd --list-all` was saved. To roll back, run `systemctl enable --now firewalld`, which re-applies the saved permanent configuration. + +```bash +systemctl stop firewalld +systemctl disable firewalld +``` + +Then check etcd and the API server: + +```bash +kubectl -n kube-system get pod -l component=etcd -o wide +kubectl get --raw='/readyz?verbose' | tail -5 +``` + +### Cause 5: iptables raw table rules raise alerts and the cluster recovers slowly after their removal + +Only when the alerts started with rules added to the `raw` table. Remove those rules; expect recovery to be gradual. + +> **Warning:** Removing the rules restores outside access that the security team wanted to block. Save the table and agree with the security owner. To roll back, run `iptables-restore -T raw < /root/iptables-raw.before` (or re-add the rules), which brings the alerts back. + +```bash +iptables-save -t raw > /root/iptables-raw.before +iptables -t raw -S +iptables -t raw -D PREROUTING +``` + +Use network policies to restrict access to workload services instead. + +## Diagnostic Steps + +1. Test the port from several directions. Failing from other nodes while answering locally means a filter on the path or on the node: + + ```bash + nc -vz -w 3 10250 + ssh 'curl -sk https://127.0.0.1:10250/healthz; echo' + kubectl get node + ``` + +2. List what filters the node. Look for rules that are not from kube-proxy (`KUBE-`), Calico (`cali-`) or Kube-OVN (`ovn`): + + ```bash + systemctl is-active firewalld + iptables-save | grep -vE 'KUBE-|cali-|ovn' | grep -E 'REJECT|DROP' + iptables -t raw -S + ``` + +3. For a metrics alert, compare local and remote scrapes. Run the second `curl` from a different node and again on the Pod's own node. `Could not connect to server` from the other node, data on the own node, and no `10254` in `firewall-cmd --list-all` is Cause 3: + + ```bash + kubectl -n exec -- curl -s 127.0.0.1:10254/metrics | head -3 + curl -s --max-time 3 http://:10254/metrics | head -3 + firewall-cmd --list-all | grep 10254 + ``` + +4. Map the findings: + + - Logs and Web CLI fail with `no route to host` on `10250`, node `Ready`: Cause 1. + - `10250` open, `kube-ovn-pinger` still cannot reach the API server Service: Cause 2. + - `kube-ovn-pinger` `CrashLoopBackOff` on every node of a new cluster with `failed to get self pod ... context deadline exceeded`, no rejecting rule, bonded physical NICs: not a firewall; see [Cross-node Pod traffic fails on the Kube-OVN Geneve overlay because of UDP 6081, TX offload or MTU on ACP](Cross_node_Pod_traffic_fails_on_the_Kube-OVN_Geneve_overlay_because_of_UDP_6081_TX_offload_or_MTU_on_ACP.md). + - Metrics target down, `10254` answers locally only: Cause 3. + - API server down, master-to-master traffic blocked, firewall active: Cause 4. + - Alerts began with `raw` table rules: Cause 5. + - The global cluster cannot reach a workload cluster's VIP ports: see [Workload cluster abnormal or monitoring empty in the console because global cannot reach its ports on ACP](Workload_cluster_abnormal_or_monitoring_empty_in_the_console_because_global_cannot_reach_its_ports_on_ACP.md). diff --git a/docs/en/solutions/Node_kubectl_fails_with_You_must_be_logged_in_to_the_server_or_an_alert_names_root_kube_config_on_ACP.md b/docs/en/solutions/Node_kubectl_fails_with_You_must_be_logged_in_to_the_server_or_an_alert_names_root_kube_config_on_ACP.md new file mode 100644 index 000000000..f2bc9060b --- /dev/null +++ b/docs/en/solutions/Node_kubectl_fails_with_You_must_be_logged_in_to_the_server_or_an_alert_names_root_kube_config_on_ACP.md @@ -0,0 +1,125 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.12.x + - 4.0.x + - 4.2.x + - 4.3.x + - 4.4.x +--- + +# Node kubectl fails with You must be logged in to the server or an alert names root kube config on ACP + +## Issue + +One of the following happens on a control-plane node: + +- `kubectl` fails, or only works when `--kubeconfig` points at `/etc/kubernetes/admin.conf`: + + ```text + "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials" + error: You must be logged in to the server (the server has asked for the client to provide credentials) + ``` + +- The platform raises `平台组件证书的到期时间即将到期(小于 30 天) <= 30d 且持续 1 分钟` (platform component certificate expires in less than 30 days), and the alert object is a file on a node rather than a Secret: `name: /root/.kube/config,node: `. `kubectl` on that node still works. The questions are whether the file renews automatically, how to read its expiry date on the node, and how to back it up before changing it. +- Pod `node-local-certs-exporter` on the global cluster is in `Error`, and its log mentions `cert-tool export` and an invalid ConfigMap key. + +Whether Kubernetes client certificates that are valid for one year renew automatically: the node file `/root/.kube/config` does not. The platform Secrets `global-kubeconfig` and `-kubeconfig` are a different credential, covered in "Console returns HTTP 500 or clusters show abnormal because the global-kubeconfig or cluster kubeconfig Secret expired on ACP". + +## Root Cause + +### Cause 1: the client certificate in /root/.kube/config is past notAfter + +Decoding `client-certificate-data` in `/root/.kube/config` shows `Issuer: CN = kubernetes`, `Subject: O = system:masters, CN = kubernetes-admin`, and a `notAfter` about one year after issue. That one-year client certificate is not renewed automatically, so `kubectl` with the default kubeconfig is rejected once it expires. + +`/etc/kubernetes/admin.conf` on the same node holds a long-lived client certificate for the same identity. From ACP 4.4 its default validity is 10 years; older installations can have a longer one. Copying `admin.conf` over `/root/.kube/config` restores `kubectl`. The long lifetime holds only while `admin.conf` has not been re-issued by `kubeadm certs renew all`, which rewrites it with the kubeadm default one-year certificate, so always print its `notAfter` before relying on it. + +### Cause 2: the one-year certificate in /root/.kube/config is inside the 30-day alert window + +Certificate monitoring checks `/root/.kube/config` on each node, which is why the alert carries a `node` label, and it fires once fewer than 30 days remain. Because nothing renews the file, the alert stays until the file is replaced. Copying the still-valid `admin.conf` over it before the old certificate expires renews it, and the alert clears. + +Other objects listed in the same alert follow their own renewal. For example, `asm-istio-cacerts-` in `cpaas-system` on global is renewed automatically by cert-manager about 15 days before expiry and needs no action while that cluster exists. + +### Cause 3: node-local-certs-exporter rejects /root/.kube/config and cert-tool export fails + +`node-local-certs-exporter` checks Kubernetes certificate expiry on the node and writes the result to ConfigMap `node-local-certs-`. It builds a data key from the kubeconfig path and the context name, such as `root_.kube_config:`. A ConfigMap key must match `[-._a-zA-Z0-9]+`, so a context name with a colon makes `cert-tool export` fail, and the Pod goes to `Error`. This is a known defect fixed in ACP 4.4. Replacing `/root/.kube/config` with `/etc/kubernetes/admin.conf` restores the Pod. + +## Resolution + +### Cause 1: the client certificate in /root/.kube/config is past notAfter + +1. Print both client certificates on the node: + + ```bash + awk '/client-certificate-data/ {print $2}' /root/.kube/config | base64 -d | openssl x509 -noout -dates -subject -issuer + awk '/client-certificate-data/ {print $2}' /etc/kubernetes/admin.conf | base64 -d | openssl x509 -noout -dates -subject -issuer + ``` + + Continue when `/root/.kube/config` is expired or unreadable and `admin.conf` still has a future `notAfter`. + +2. Back up the file and copy `admin.conf` over it. + + > **Warning:** The copy overwrites `/root/.kube/config` for root on this node; contexts and users that existed only in the old file are gone from the active file. Do this only when the `admin.conf` certificate printed in step 1 is in date, on the node whose `kubectl` or exporter is failing. Rollback is `cp -a /root/.kube/config.bak /root/.kube/config`. This does not rotate the API server or etcd serving certificates. + + ```bash + cp -a /root/.kube/config /root/.kube/config.bak + cp /etc/kubernetes/admin.conf /root/.kube/config + kubectl get nodes + ``` + + `kubectl get nodes` should list the cluster. Repeat on every control-plane node, not only the one in the open shell. + +3. If the console also returns an internal error, check the platform kubeconfig Secrets as described in "Console returns HTTP 500 or clusters show abnormal because the global-kubeconfig or cluster kubeconfig Secret expired on ACP". + +### Cause 2: the one-year certificate in /root/.kube/config is inside the 30-day alert window + +1. Log in to the node the alert names and print both certificates with step 1 of Cause 1. Continue when `/root/.kube/config` shows about one year of validity with fewer than 30 days left, and `admin.conf` shows a `notAfter` well beyond it. If `admin.conf` also shows only about one year, it may have been re-issued by `kubeadm certs renew all`; do not copy it. + +2. Back up the file and copy `admin.conf` over it, with the Warning and commands of step 2 of Cause 1. Then confirm the new end date: + + ```bash + awk '/client-certificate-data/ {print $2}' /root/.kube/config | base64 -d | openssl x509 -noout -enddate + ``` + + It now prints the `admin.conf` end date. Repeat on every node the alert names; each node has its own copy of the file. The alert clears after the next check. + +3. For Secret-type objects in the same alert, check the Certificate on global instead of changing anything: + + ```bash + kubectl get certificate -n cpaas-system asm-istio-cacerts- -o custom-columns='NAME:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status,RENEW:.status.renewalTime,SECRET:.spec.secretName' + ``` + +### Cause 3: node-local-certs-exporter rejects /root/.kube/config and cert-tool export fails + +1. Confirm the signature in the Pod log: `cert-tool export` and an invalid ConfigMap key with a colon (`root_.kube_config:`), or a format error while reading `/root/.kube/config`. + + ```bash + kubectl get pods -A | grep node-local-certs-exporter + kubectl logs -n --tail=40 + ``` + +2. On the node the Pod runs on, replace the file with step 2 of Cause 1, including the backup and its Warning. The exporter Pod then leaves `Error`. Clusters that run the exporter of a release before 4.4 need this workaround; upgrading to 4.4 fixes the key generation. + +## Diagnostic Steps + +1. On a control-plane node, separate an expired client file from an expired serving certificate: + + ```bash + awk '/client-certificate-data/ {print $2}' /root/.kube/config | base64 -d | openssl x509 -noout -dates -subject -issuer + awk '/client-certificate-data/ {print $2}' /etc/kubernetes/admin.conf | base64 -d | openssl x509 -noout -dates -subject -issuer + openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates -subject -issuer + openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates -subject -issuer + ``` + + Copying `admin.conf` fixes only the client file. If `apiserver.crt` or another serving file is the expired object, it needs its own renewal. + +2. Read every object in a certificate alert. A `name:` that is a path plus a `node:` label is a node file; a `name:` with `secret_namespace:` is a Secret. + +3. Branch: + + - `kubectl` fails with `You must be logged in to the server` and `/root/.kube/config` is past `notAfter`: Cause 1. + - The alert names `/root/.kube/config` and the file still has fewer than 30 days left: Cause 2. + - `node-local-certs-exporter` is `Error` with a `cert-tool export` key error: Cause 3. diff --git a/docs/en/solutions/Node_maintenance_on_ACP_OS_reinstall_or_replacement_hostname_IP_NIC_and_cluster_name_changes.md b/docs/en/solutions/Node_maintenance_on_ACP_OS_reinstall_or_replacement_hostname_IP_NIC_and_cluster_name_changes.md new file mode 100644 index 000000000..2de53ce6d --- /dev/null +++ b/docs/en/solutions/Node_maintenance_on_ACP_OS_reinstall_or_replacement_hostname_IP_NIC_and_cluster_name_changes.md @@ -0,0 +1,152 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - "3.8" + - "4.1" + - "4.2" + - "4.3" +--- + +# Node maintenance on ACP: OS reinstall or replacement, hostname, IP, NIC, and cluster name changes + +## Issue + +Planned changes to the identity of nodes that are already in a cluster raise the same questions repeatedly: + +- A machine's OS was reinstalled. How does it rejoin the cluster? +- Compute nodes must move from CentOS to Kylin, keeping their IPs, and TopoLVM is in use. +- Node hostnames are inconsistent. Can the hostnames be changed? Can the node names be changed? +- The node IPs must change. Is a redeploy needed, and how do removed nodes come back? Can the VIP and node IPs of a whole `global` and workload cluster be changed? +- VMs move to another virtualization platform and the NIC name changes (for example from `ens192` to `ens3`). +- Can the cluster be renamed? + +Related articles: "Node maintenance on ACP: resizing nodes and masters, host shutdown, and disk expansion" covers resource changes. + +## Root Cause + +### Cause 1: A reinstalled machine is a new host to the cluster + +After an OS reinstall the host has none of the files the cluster gave it, while the cluster may still hold its node object and network or storage state. Remove the node from the cluster, run the node cleanup script on the host before it rejoins, then add the node again. + +### Cause 2: Replacing the OS of compute nodes that keep their IPs + +Changing compute nodes from CentOS to Kylin Linux Advanced Server V10 (for example on ACP 4.2.1) means taking each node out of the cluster, reinstalling it, and rejoining it with the same IP. The target OS and kernel must be in the verified list of the platform version. Nodes that host log or monitoring storage, middleware, DevOps components, or TopoLVM volumes need their data moved first. + +### Cause 3: The node name is fixed at join; hostnames of IP-named nodes can change + +A joined node does not support changing its Node name; the name is fixed when the node joins. + +- Nodes named by hostname: changing the hostname afterwards can make node identity, certificates, kubelet registration, and the platform display disagree. Keep the hostnames unchanged. +- Nodes named by IP: the hostnames can be changed. + +### Cause 4: Node IPs cannot be changed in place + +- A worker's IP is changed by deleting the node in the UI and adding it again with the new IP. Changing the IPs of all machines at once is not supported (for example on TKE 3.8.1). +- Nodes already removed because their IPs changed cannot be re-added through a backend method. Run `cleanup.sh` and `init.sh` on the node and add it again under Platform Management > Cluster. The cleanup script deletes the cluster data on that node. +- A workload cluster whose IPs must all change and that runs no business workloads yet is deleted and recreated with the new IPs, rather than changed node by node. +- There is no procedure for changing both the cluster VIP and the node IPs of a running `global` cluster and its workload clusters with workloads and Harbor on them. That is an architecture and network re-planning project that needs a separate evaluation with Alauda. + +### Cause 5: VMs moved to another virtualization platform get a new NIC name + +Moving nodes to another virtualization platform requires shutting the VMs down. When IPs, disks, and paths stay the same, the only platform-side impact is the NIC name. On Calico, `IP_AUTODETECTION_METHOD` of the `calico-node` DaemonSet matches interface names (for example `interface=^eth0$`) and must also match the new name. On Kube-OVN, the tunnel interface setting of the node is the equivalent; when a migrated master's NIC name changes, `ovn-cni` can fail to start until the node annotation is updated to the new interface name. + +### Cause 6: The cluster name cannot be modified + +Renaming a cluster after it is deployed is not supported. The only option is to redeploy the cluster with the new name. Only the display name can be edited. + +## Resolution + +### Cause 1: A reinstalled machine is a new host to the cluster + +Remove the node from the cluster through the platform, run the cleanup script for your platform version on the host, prepare the host as for a new node, and add it again. The same sequence applies to nodes that must be repartitioned (for example to give `/var/lib/containerd` and `/cpaas` their own partitions) while they run no workloads: delete the node, run the cleanup script, partition, run the initialization script, add it again. + +```bash +kubectl get node +kubectl get machine -A | grep +``` + +### Cause 2: Replacing the OS of compute nodes that keep their IPs + +1. Confirm the target OS and kernel against the verified list of your version (`cat /etc/os-release` and `uname -r` on a host already running it). +2. List what runs on the nodes: log and monitoring storage, middleware, DevOps components, TopoLVM volumes. +3. If spare capacity allows, add the new nodes first. Then cordon and drain the nodes being replaced and wait until their Pods run on other nodes. +4. For TopoLVM, back up or migrate TopoLVM-backed services, then take the node out of TopoLVM following the product documentation for removing a TopoLVM storage node (see also "Preserving TopoLVM Data of KubeVirt VMs When Reinstalling Node Operating Systems on ACP"). +5. Remove the node from the cluster, reinstall the OS, prepare the host, and add it again with the same IP. + +> **Warning:** `kubectl drain` evicts every Pod on the node that is not a DaemonSet Pod; single-replica workloads are unavailable until they start elsewhere. Data on local TopoLVM volumes of that node is lost with the reinstall unless it was backed up or migrated first. Confirm the data is backed up or not needed before step 5. Rollback of a drain is `kubectl uncordon `; a reinstall cannot be rolled back. + +```bash +kubectl cordon +kubectl drain --ignore-daemonsets --delete-emptydir-data +kubectl get pods -A -o wide --field-selector spec.nodeName= +kubectl -n nativestor-system get topolvmcluster topolvm -o yaml +``` + +### Cause 3: The node name is fixed at join; hostnames of IP-named nodes can change + +Check how the nodes are named (Platform Management > Cluster Management > Nodes, or compare `NAME` with `INTERNAL-IP` below) and compare with the current OS hostname: + +```bash +kubectl get nodes -o wide +hostname; hostnamectl +``` + +- Named by hostname: keep the hostname each node had when it joined; there is no rename procedure. +- Named by IP: change the hostnames as required, then confirm every node is still `Ready`. + +### Cause 4: Node IPs cannot be changed in place + +For one worker or compute node, delete it in the UI, then on the host: + +> **Warning:** `cleanup.sh` removes the cluster data on that node (container data, kubelet and Kubernetes files). Confirm that nothing that must be kept lives in those paths before running it; it cannot be undone. + +```bash +sudo bash cleanup.sh +sudo bash init.sh +ip -br addr +``` + +Then add the node with its new IP under Platform Management > Cluster. If the add reports that a NIC cannot be found (for example `检查网卡:找不到网卡: `, "check NIC: NIC not found"), enter the interface that now carries the node IP. An expired license must be renewed before nodes can be added. For a whole cluster with no workloads, delete it and create it again with the new addresses. + +### Cause 5: VMs moved to another virtualization platform get a new NIC name + +Move one node at a time: cordon and drain it, shut it down, move it, start it, uncordon it, and only then take the next. Before the first node with the new name starts, let Calico match both names. + +> **Warning:** A wrong `IP_AUTODETECTION_METHOD` makes `calico-node` pick no address or the wrong one when the DaemonSet rolls, which breaks Pod traffic on every node. Save the current value; rollback is setting it back. + +```bash +kubectl get ds -A | grep calico-node +kubectl -n get ds calico-node -o yaml | grep -A2 IP_AUTODETECTION_METHOD +kubectl -n set env ds/calico-node IP_AUTODETECTION_METHOD='interface=^(ens192|ens3)$' +``` + +The regular expression must keep every interface name that nodes use today and add the new one. Use the namespace that `kubectl get ds -A` prints (commonly `kube-system`). + +As an alternative to moving the `global` workers, they can be replaced: add new workers; cordon the old ones; uninstall the log collection components (`nevermore`) on `global` and the business clusters and the `global` Elasticsearch log storage plugin; evict the Pods to the new workers and check `kubectl get pod -A | grep -v Running`; reinstall the Elasticsearch log storage plugin pinned to the new nodes; reinstall log collection with `global` as the storage cluster. + +### Cause 6: The cluster name cannot be modified + +Plan the name before creation. To use another name, create the cluster again with that name. For an imported cluster, re-importing it is the alternative to assess. + +```bash +kubectl get cls +``` + +## Diagnostic Steps + +```bash +kubectl get nodes -o wide +kubectl get machine -A +cat /etc/os-release; uname -r +ip -br addr +``` + +- A host whose OS was reinstalled while its node object still exists: Cause 1. +- An OS change on nodes that keep their IPs, especially with TopoLVM volumes: Cause 2. +- `NAME` equal to the hostname versus equal to `INTERNAL-IP` decides whether hostnames may change: Cause 3. +- Any node IP change: Cause 4. +- A NIC name that changed after a VM migration, with `calico-node` or `ovn-cni` failing on that node: Cause 5. +- A request to rename a cluster: Cause 6. diff --git a/docs/en/solutions/Node_maintenance_on_ACP_resizing_nodes_and_masters_host_shutdown_and_disk_expansion.md b/docs/en/solutions/Node_maintenance_on_ACP_resizing_nodes_and_masters_host_shutdown_and_disk_expansion.md new file mode 100644 index 000000000..d5dfe58d0 --- /dev/null +++ b/docs/en/solutions/Node_maintenance_on_ACP_resizing_nodes_and_masters_host_shutdown_and_disk_expansion.md @@ -0,0 +1,145 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - "3.8" + - "4.1" + - "4.2" + - "4.3" +--- + +# Node maintenance on ACP: resizing nodes and masters, host shutdown, and disk expansion + +## Issue + +Planned resource work on nodes that are already in a cluster raises the same questions repeatedly: + +- CPU or memory was hot-added to a node VM, but the node still shows the old capacity. +- The three masters of a business cluster are oversized and the CPU license is exhausted. Can they be reduced to free cores for new workers? +- Control-plane, monitoring, and log nodes must become smaller. How do I tell whether a node can be shrunk, and can the Pod requests on it be lowered? +- The hypervisor hosts need a fix. Can the platform VMs be shut down one host at a time? +- After a node disk expansion, restarted component Pods stay in the image-pulling state. + +Related articles: "Node maintenance on ACP: OS reinstall or replacement, hostname, IP, NIC, and cluster name changes" covers changes to node identity. + +## Root Cause + +### Cause 1: Node capacity does not change after a CPU or memory hot-add + +kubelet reports the CPU and memory capacity it read when it started. After a hot expansion the node keeps showing the old capacity until kubelet is restarted on that node. + +### Cause 2: Shrinking a master or other node is a per-node resize gated by Pod requests + +Reducing oversized masters (for example three 64-core, 256 GB business-cluster masters at low utilization with no business workloads or logs, reduced to 16 cores and 32 GB to free licensed cores) is feasible as a per-node resize, one master at a time. + +A node can be reduced only when both its utilization and its request rate are low, and at least 3 control-plane nodes must remain. Utilization alone is misleading: a control-plane node with 8 cores and 31.34 Gi can show 11.79% CPU utilization but an 84.00% CPU request rate (limit rate 329.10%), with requests coming largely from Pods such as `region-alb` (2050m), `vmagent-agent` (600m), `ovn-central` (300m), `kube-apiserver` (250m), and `ovs-ovn` and `kube-ovn-monitor` (200m each), well above their actual use. Pod request values like these can be lowered appropriately before the node is resized. + +### Cause 3: Host shutdowns must not take several platform VMs down at once + +A hypervisor host often carries several platform VMs. Shutting the host down stops all of them together, which can take more than one master down and lose the etcd quorum. Back up etcd first, then handle one VM at a time. + +### Cause 4: Pods stay pulling images after a disk expansion because the registry's MinIO cluster broke + +After a node disk expansion, restarted component Pods stay in the image-pulling state. The platform image registry's MinIO is abnormal: its three instances cannot form a cluster, so the registry is unavailable. A new raw disk was added directly while the node's containers were still in place and kubelet and containerd had not been stopped correctly; the existing containers then read their original data abnormally, which broke the MinIO cluster. Restarting MinIO and the registry restores it. + +## Resolution + +### Cause 1: Node capacity does not change after a CPU or memory hot-add + +> **Warning:** Restarting kubelet briefly stops status updates for that node; running Pods keep running. Do it in a maintenance window, one node at a time. There is no rollback; if kubelet does not come back, read `journalctl -u kubelet`. + +```bash +kubectl get node -o jsonpath='{.status.capacity}{"\n"}' +sudo systemctl restart kubelet +systemctl status kubelet --no-pager +kubectl get node -o jsonpath='{.status.capacity}{"\n"}' +``` + +The general cordon, drain, resize, uncordon loop is in "Safely resizing CPU and memory on Alauda Container Platform nodes". + +### Cause 2: Shrinking a master or other node is a per-node resize gated by Pod requests + +Check usage and placement first. Node names on ACP are often the node IP; use the exact name from `kubectl get nodes`. + +```bash +kubectl get nodes -o wide +kubectl top node +kubectl get pods -A -o wide --field-selector spec.nodeName= +``` + +The request rate decides as much as the usage. The node's monitoring tab in the console shows CPU and memory request rates; the same figures are in `kubectl describe node`: + +```bash +kubectl describe node | sed -n '/Allocated resources/,/Events/p' +kubectl get pods -A --field-selector spec.nodeName= \ + -o custom-columns='NAMESPACE:.metadata.namespace,POD:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory' +``` + +A node whose usage is low but whose request rate is high cannot simply be made smaller: the Pods placed there would no longer fit in its allocatable resources, and Pods that do not fit are rejected (for example `OutOfcpu`) or stay `Pending`. Lower the requests of the Pods that over-request first, then resize. Keep at least 3 control-plane nodes throughout. + +> **Warning:** A master that is powered off for the resize removes one etcd member and one API server from service. Never take a second master down until the first is back. Back up etcd before starting, and make sure the new size still covers the control-plane Pods listed above. Rollback is resizing the VM back to its old size. + +For each master in turn: shut it down, resize it, start it, restart and check kubelet, and wait until the node and its etcd Pod are back before moving on: + +```bash +sudo systemctl restart kubelet +systemctl status kubelet --no-pager +kubectl get node +kubectl -n kube-system get pods -l component=etcd -o wide +``` + +### Cause 3: Host shutdowns must not take several platform VMs down at once + +Back up etcd first. Then choose one of two plans: + +- **Business can stop:** stop the business containers, restart the compute nodes one by one and confirm each is normal before the next, then restart the control-plane nodes one by one, then start the business again. +- **Business keeps running:** for one VM at a time, mark it unschedulable and evict the business containers, confirm the business recovered elsewhere, shut down the VM for the fix, start it, make it schedulable again, then move to the next VM. + +> **Warning:** Evicting Pods interrupts single-replica workloads until they start on another node. Do not shut down several platform VMs at the same time; two masters down at once loses the etcd quorum of a three-master cluster. Data services such as MySQL or Redis in the cluster need their own plan based on how they are deployed. Rollback for a VM is `kubectl uncordon` after it is back. + +```bash +kubectl cordon +kubectl drain --ignore-daemonsets --delete-emptydir-data +# shut down, fix, and start the VM, then: +kubectl get node +kubectl uncordon +``` + +### Cause 4: Pods stay pulling images after a disk expansion because the registry's MinIO cluster broke + +Find the MinIO and registry Pods and read their logs; MinIO instances that cannot form a cluster are this cause: + +```bash +kubectl get pods -A -o wide | grep -i -E 'minio|registry' +kubectl logs -n --tail=200 +kubectl describe pod -n | sed -n '/Events/,$p' +``` + +> **Warning:** Restarting MinIO and the registry makes image pulls and pushes against the platform registry fail until the new Pods are Ready. Precondition: the stuck Pods pull from the platform registry and the MinIO logs or status show the instances are not forming a cluster. The controllers recreate the Pods, so there is nothing to roll back; the restart does not repair data that is actually damaged on disk. + +```bash +kubectl delete pod -n +kubectl delete pod -n +kubectl get pods -A -o wide | grep -i -E 'minio|registry' +``` + +The registry returns to normal and the component Pods finish pulling. For the next disk change on a node, stop kubelet, delete the containers on the node, and stop containerd before the disk is added. + +## Diagnostic Steps + +Before and after each maintenance step: + +```bash +kubectl get nodes -o wide +kubectl get node -o jsonpath='{.status.capacity}{"\n"}{.status.allocatable}{"\n"}' +kubectl get pods -A -o wide --field-selector spec.nodeName= | grep -v Running +kubectl -n kube-system get pods -l component=etcd -o wide +``` + +- Capacity still shows the old CPU or memory after a resize: kubelet has not been restarted on that node (Cause 1). +- A node with low usage but a high request rate: lower requests before shrinking it (Cause 2). +- More than one master on the same hypervisor host that is about to be shut down: Cause 3. +- Pods stuck pulling images after a disk expansion, with the platform registry's MinIO unable to form a cluster: Cause 4. +- A node that is `NotReady` after it is back is not a maintenance-plan question; see "Diagnosing a node that reports NotReady on Alauda Container Platform". diff --git a/docs/en/solutions/Node_runtime_or_kubelet_directories_return_Input_output_error_or_df_hangs_on_ACP.md b/docs/en/solutions/Node_runtime_or_kubelet_directories_return_Input_output_error_or_df_hangs_on_ACP.md new file mode 100644 index 000000000..34d955860 --- /dev/null +++ b/docs/en/solutions/Node_runtime_or_kubelet_directories_return_Input_output_error_or_df_hangs_on_ACP.md @@ -0,0 +1,173 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.x + - 4.x +--- + +# Node runtime or kubelet directories return Input/output error, or df hangs, on ACP + +## Issue + +Typical reports: + +- All services scheduled to one node are abnormal, cannot be deleted, and cannot be exec'd into, and it comes back after the node is rebooted. +- `ls` under `/var/lib` prints `ls: cannot access 'containerd': Input/output error` (Chinese locale: `ls: 无法访问 'containerd': 输入/输出错误`), and the entry is listed as `d?????????`. The same can happen for `/cpaas`, which holds platform component data. +- The kubelet on one worker will not start: `failed to run Kubelet: unable to determine image API version: rpc error ... /var/lib/docker/image: input/output error`. +- After a storage failure the platform cannot select clusters or namespaces, and kube-apiserver on a master refuses connections on `6443`. +- The platform Grafana page returns `login grafana with unexpected response` (code 500) while some nodes are `NotReady` with disk problems. +- A container keeps failing with `Back-off restarting failed container`, although the same image works elsewhere. +- The platform suddenly raises many alerts, all about DaemonSets, and `df -h` on one node hangs. + +These are faults in the host disks, below Kubernetes. Related article: [Node unstable from a GPU NVRM log flood, a faulty memory module, or a read-only cpuset cgroup on ACP](Node_unstable_from_a_GPU_NVRM_log_flood_a_faulty_memory_module_or_a_read-only_cpuset_cgroup_on_ACP.md) covers GPU, memory, and cgroup faults. Storage that stalls rather than fails is covered in [Node NotReady caused by network, storage or host-level faults on ACP](Node_NotReady_caused_by_network_storage_or_host_level_faults_on_ACP.md) and [Node NotReady with high load average and D-state processes on ACP](NotReady_Node_with_High_Load_and_Many_D_State_Processes.md). + +## Root Cause + +### Cause 1: I/O errors on the disk that holds the runtime, kubelet, or platform data + +`ls /var/lib/containerd` (or `/var/lib/docker`, `/var/lib/kubelet`, `/cpaas`) returns `Input/output error`. The data disk under that directory is damaged and can no longer be read or written reliably; restarts and Pod evictions only move the symptom. Stopping containerd and cleaning up its processes helps for a while, but the fault returns. + +Errors on the node: + +```text +ls: cannot access 'containerd': Input/output error +d????????? ? ? ? ? ? containerd +FATA[0000] execing command in container: ... failed to load task: no running task found: task not found: not found +Fail to write "stderr" log to log file "/var/log/pods///2.log" error="write ...: input/output error" +"command failed" err="failed to run Kubelet: unable to determine image API version: rpc error ... /var/lib/docker/image: input/output error" +FATA[0000] failed to try resolving symlinks in path "/var/log/pods/kube-system_kube-apiserver-_/kube-apiserver/1.log": lstat /var/lib/containerd: input/output error +``` + +Kernel messages with them: + +```text +VFS: Open an exclusive opened block device for write vdb [ sgdisk]. +blk_update_request: I/O error, dev sdb, sector +sd 0:0:2:0: timing out command, waited 180s +XFS (dm-1): metadata I/O error in "xlog_iodone" ... error 5 +XFS (dm-1): Log I/O Error Detected. Shutting down filesystem +xfs_imap_to_bp: xfs_trans_read_buf() returned error -5 +``` + +Variants: + +- **containerd data disk on a worker.** Every Pod on the node is abnormal, cannot be deleted, and `crictl exec` fails with `no running task found`. When `/var/log/pods` is a symlink into `/var/lib/containerd`, container log writes fail too. The kernel can repeat `VFS: Open an exclusive opened block device for write [ sgdisk]` about every 10 seconds, meaning an `sgdisk` process keeps opening the in-use disk for writing; what starts it is not known. CNI teardown errors such as `dial unix /run/openvswitch/kube-ovn-daemon.sock: connect: connection refused` at the same time are side effects. +- **kubelet will not start on a Docker worker.** `systemctl status kubelet` shows `activating (auto-restart) (Result: exit-code)` with `status=1/FAILURE`. `mount | grep sdb` returns nothing because the disk is used through LVM; `lsblk` shows the `sdb` → `dm-*` → mount point chain. +- **`/var/lib/containerd` on a master.** kube-apiserver cannot run, `curl -k https://:6443` returns `Connection refused`, and the platform cannot select the cluster or its namespaces. +- **Nodes `NotReady` with faulty disks.** Platform Pods on them crash-loop, for example `kube-prometheus-grafana-*` in `cpaas-system`, which makes the Grafana page return `login grafana with unexpected response`. +- **Several directories on several nodes.** `df -h /var/lib/containerd` returns `Input/output error`, and `dmesg` shows command timeouts, `I/O error, dev sdc`, then `xfs_do_force_shutdown(0x2)`. An XFS filesystem in that state stays shut down until it is unmounted and mounted again, even after the disk is readable. +- **Formatted or broken data disks.** A custom container that mounts a file-server path keeps failing with `BackOff` because some nodes had a data disk formatted and another node had a broken disk. + +### Cause 2: The root filesystem hangs + +On the node, `df -h` hangs, and `strace df -h` shows it stuck on one mount point; `cd /; ls` hangs too. DaemonSet Pods on that node fail to start, so the platform raises a burst of DaemonSet alerts at once. The root disk mount is faulty and every process that touches it blocks; `df` can be stuck on a directory where no disk is actually mounted. A reboot restores the node. + +## Resolution + +### Cause 1: I/O errors on the disk that holds the runtime, kubelet, or platform data + +1. Move workloads off the node with `kubectl cordon ` and `kubectl drain --ignore-daemonsets --delete-emptydir-data`. +2. Give the disk errors (and any repeating `sgdisk` line with its process) to the IaaS or hardware owner to repair or replace the disk. Containers on the node cannot run normally until the disk is fixed. Stopping the runtime and cleaning up its processes is only temporary relief. +3. If the disk was replaced with an empty one, or `lsblk -f` shows a disk was formatted (its data is gone), rebuild the node data directories as in [Node capacity, pod limit or data disk after resizing or replacing node hardware on ACP](Node_capacity_pod_limit_or_data_disk_after_resizing_or_replacing_node_hardware_on_ACP.md). +4. If the same disks were repaired, cleaned, and re-added, remount the affected directories so a shut-down XFS filesystem comes back. The order below is general Linux and Kubernetes practice. containerd normally uses `KillMode=process`, so stopping it leaves `containerd-shim` processes and their containers running and still holding the mounts; stop the Pods while containerd still runs. + +> **Warning:** `crictl stopp` stops every Pod sandbox on the node, including DaemonSet and static Pods; on a control-plane node that takes its kube-apiserver and etcd member offline until the kubelet starts again. Do it only when the infrastructure side confirms the disk is readable again (no new `I/O error` lines in `dmesg`), the node is drained, and, on a master, the other masters are healthy and etcd is backed up. If the mount fails, keep the services stopped and return the disk to its owner. + +```bash +systemctl show containerd -p KillMode +systemctl stop kubelet +crictl stopp $(crictl pods -q) +systemctl stop containerd +ps -eo pid,cmd | grep -E 'containerd-shim' | grep -v grep # must print nothing +umount /var/lib/containerd /cpaas +``` + +If `umount` reports the target is busy, find and stop what still uses it: + +```bash +fuser -vm /var/lib/containerd /cpaas +lsof +f -- /var/lib/containerd +``` + +Then mount and start again: + +```bash +mount -a +findmnt /var/lib/containerd; findmnt /cpaas +ls /var/lib/containerd > /dev/null && echo readable +systemctl start containerd kubelet +kubectl uncordon +``` + +5. For a master whose `/var/lib/containerd` returns I/O errors, reboot it after the storage owner has repaired the underlying storage. + +> **Warning:** Rebooting a control-plane node takes its kube-apiserver and etcd member offline until it is back. Do it only when `ls /var/lib/containerd` on that master returns `Input/output error`, the storage owner has confirmed the repair, and the other masters are healthy. A reboot cannot be undone. + +```bash +systemctl reboot +# afterwards, from another master: +curl -k --max-time 5 https://:6443/healthz +kubectl -n kube-system get pods -l component=kube-apiserver -o wide +``` + +After the host-side repair, platform Pods such as Grafana return to normal on their own; check them with `kubectl -n cpaas-system get pods -o wide | grep `. + +### Cause 2: The root filesystem hangs + +> **Warning:** A forced reboot kills every Pod on the node, and blocked filesystem writes may be lost. Do it only when `df` or `ls` on `/` hang and `strace` confirms the process is blocked on the filesystem. A reboot cannot be undone. + +```bash +systemctl reboot -f # use the IaaS console reset if the shell cannot run it +``` + +After the reboot, check for a leftover taint that keeps Pods `Pending` (for example one `alb` replica). Remove only a taint that is not one of the automatic `node.kubernetes.io/*` taints and is not part of your node-role design: + +```bash +kubectl describe node | grep -A3 Taints +kubectl taint nodes :- +``` + +Have the IaaS owner check the root disk and its storage path so the hang does not recur. + +## Diagnostic Steps + +1. Disk errors and the device behind the data directories (Cause 1): + + ```bash + dmesg -T | grep -iE 'I/O error|blk_update_request|Buffer I/O|EXT4-fs error|XFS .*error|sgdisk|exclusive opened' | tail -n 30 + findmnt -T /var/lib/containerd; findmnt -T /var/lib/docker; findmnt -T /var/lib/kubelet; findmnt -T /cpaas + lsblk -f + ls /var/lib/containerd > /dev/null && echo readable + ls -ld /var/log/pods + ``` + + Expected on a healthy node: no I/O error lines and `readable`. `findmnt` and `lsblk` show the device even behind LVM or multipath. A symlinked `/var/log/pods` shares the same disk. + +2. A repeating `sgdisk` line: + + ```bash + dmesg -T | grep -c 'exclusive opened block device' + ps -ef | grep [s]gdisk + ``` + + Record the process and give it to the IaaS owner with the disk errors. + +3. On a master, an apiserver that refuses `6443` while `ls /var/lib/containerd` fails is the master variant of Cause 1: + + ```bash + curl -k --max-time 5 https://:6443 + crictl ps -a | grep kube-apiserver + ``` + +4. Root filesystem hang (Cause 2). Run each command with a timeout so the shell itself does not hang: + + ```bash + timeout 10 df -h || echo "df hung" + timeout 10 strace -f df -h 2>&1 | tail -n 5 + timeout 10 ls / > /dev/null || echo "ls / hung" + ``` + +5. Many Pods on one node failing while the kernel log is clean points elsewhere: a hung runtime is covered in [Node NotReady on ACP with kubelet "PLEG is not healthy"](Node_NotReady_with_PLEG_is_not_healthy.md), and missing containerd directories in [Control-plane Pods do not start with mkdir /var/log/pods: file exists after containerd directories are lost on ACP](Control-plane_Pods_do_not_start_with_mkdir_var_log_pods_file_exists_after_containerd_directories_are_lost_on_ACP.md). diff --git a/docs/en/solutions/Node_unstable_from_a_GPU_NVRM_log_flood_a_faulty_memory_module_or_a_read-only_cpuset_cgroup_on_ACP.md b/docs/en/solutions/Node_unstable_from_a_GPU_NVRM_log_flood_a_faulty_memory_module_or_a_read-only_cpuset_cgroup_on_ACP.md new file mode 100644 index 000000000..703423ce3 --- /dev/null +++ b/docs/en/solutions/Node_unstable_from_a_GPU_NVRM_log_flood_a_faulty_memory_module_or_a_read-only_cpuset_cgroup_on_ACP.md @@ -0,0 +1,136 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.x + - 4.x +--- + +# Node unstable from a GPU NVRM log flood, a faulty memory module, or a read-only cpuset cgroup on ACP + +## Issue + +Typical reports: + +- A node is abnormal, services cannot be deployed to it, and `journalctl -u kubelet` prints nothing. The node is `NotReady` with `PLEG is not healthy`, and a reboot does not fix it. +- A memory fault on a node caused a service disruption. +- The business network on one worker (for example an ARM node) is unreachable although the switch configuration is fine, and the kube-ovn Pods fail with `/sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice: no such file or directory`. + +These are faults in the GPU, memory, or OS configuration of the host. Kubernetes only shows their effect; each cause gives the host-side signature that identifies it. Disk I/O errors and a hanging root filesystem are covered in [Node runtime or kubelet directories return Input/output error, or df hangs, on ACP](Node_runtime_or_kubelet_directories_return_Input_output_error_or_df_hangs_on_ACP.md). A PLEG problem without a GPU log flood is covered in [Node NotReady on ACP with kubelet "PLEG is not healthy"](Node_NotReady_with_PLEG_is_not_healthy.md). + +## Root Cause + +### Cause 1: A GPU that failed to initialise floods the kernel log + +On a GPU node, `journalctl -fu kubelet` and `journalctl -fu docker` print nothing. `systemctl status systemd-journald` shows a continuous stream of `systemd-journald[]: Missed kernel messages`, with journald using several GiB of memory. `dmesg -T` is full of `NVRM` errors. The node looks like a PLEG or runtime problem: + +```text +Ready False KubeletNotReady PLEG is not healthy: pleg was last seen active 38m36.485032165s ago; threshold is 3m0s +"PLEG: Write status" err="rpc error: code = DeadlineExceeded desc = operation timeout: context deadline exceeded" +``` + +while `MemoryPressure`, `DiskPressure`, `PIDPressure`, `KernelDeadlock`, and `NetworkUnavailable` are all `False`. The kernel messages identify the card: + +```text +NVRM: gpuHandleSanityCheckRegReadError_GM107: Possible bad register read: addr: 0x110100, regvalue: 0xbadf5620, error code: Unknown SYS_PRI_ERROR_CODE +NVRM: kflcnWaitForHalt_TU102: Timeout waiting for Falcon to halt +NVRM: GSP failed to halt with GFW_BOOT: (progress 0xff) +NVRM: failed to wait for GFW boot complete: 0x65 VBIOS version +NVRM: (the GPU may be in a bad state and may need to be reset) +NVRM: RmInitAdapter: Cannot initialize GSP firmware RM +NVRM: GPU <0000:BB:DD.F>: RmInitAdapter failed! (0x62:0x65:2015) +NVRM: rm_init_adapter failed, device minor number 0 +``` + +One GPU card failed to initialise. Its very frequent `NVRM` messages overflow the kernel log buffer, journald drops the journal, runtime, and kubelet logs, and the node becomes unstable. + +### Cause 2: Faulty memory module + +Services on one node are disrupted at a specific moment and may recover on their own. The hardware log (BMC, iLO, iDRAC, or IaaS events) or the kernel log reports memory errors for that host. A failing memory module (DIMM) crashes or stalls whatever was using the bad memory. Taking the faulty memory offline or replacing it during a planned shutdown makes the node healthy again. + +### Cause 3: `/sys/fs/cgroup/cpuset` is read-only, so kube-ovn Pods cannot start + +On one node `/sys/fs/cgroup/cpuset` has become read-only, so the cpuset path for Pods cannot be used. The OVN Pods fail with: + +```text +/sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice: no such file or directory +``` + +and the node's Pod network fails, although the physical network is fine. Why the cgroup filesystem became read-only is not known; a reboot restores it. + +## Resolution + +### Cause 1: A GPU that failed to initialise floods the kernel log + +1. Raise the GPU errors with the hardware vendor before continuing Kubernetes-level troubleshooting. +2. Identify the faulty card from the PCI address in the `RmInitAdapter failed!` line, or with `nvidia-smi`. +3. Reset that card with a PCIe Function Level Reset, which recovers the card and the node. + +> **Warning:** Resetting a GPU kills every process using it. Do it only when no workload uses the card (cordon the node and drain or stop the GPU Pods) and the PCI address is confirmed as the faulty card. The reset itself cannot be undone; if it does not recover the card, the card needs vendor service. + +```bash +kubectl cordon +nvidia-smi -q | grep -E 'Bus Id|Product Name' +# by PCI address (domain:bus:device.function, from dmesg or nvidia-smi): +echo 1 > /sys/bus/pci/devices/<0000:BB:DD.F>/reset +# or with the NVIDIA tool: +nvidia-smi --gpu-reset -i +``` + +4. Confirm that `dmesg` stops showing new `NVRM` errors and that the journal shows kubelet and runtime logs again, then `kubectl uncordon `. + +### Cause 2: Faulty memory module + +Take the faulty memory offline or replace it in a planned shutdown. Follow [Shutting down or rebooting ACP nodes for maintenance](Shutting_down_or_rebooting_ACP_nodes_for_maintenance.md): evict Pods, back up etcd if it is a control-plane node, shut down, repair, boot, and check that the platform and applications recover. + +### Cause 3: `/sys/fs/cgroup/cpuset` is read-only, so kube-ovn Pods cannot start + +Reboot the host; this restores write access to `/sys/fs/cgroup/cpuset/`, and the OVN components and business network recover. Ask the OS owner to check the cgroup filesystem afterwards. + +> **Warning:** A reboot restarts every Pod on the node. Do it only when `/sys/fs/cgroup/cpuset` is confirmed read-only on that node and the kube-ovn Pods fail with the `kubepods-burstable.slice: no such file or directory` error. Drain the node first if it still runs workloads. A reboot cannot be undone. + +```bash +kubectl drain --ignore-daemonsets --delete-emptydir-data --timeout=10m +systemctl reboot +# after boot: +mount | grep 'cgroup/cpuset' +kubectl -n kube-system get pods -o wide --field-selector spec.nodeName= | grep -E 'kube-ovn|ovs' +kubectl uncordon +``` + +The cpuset mount should show `rw`, and the kube-ovn Pods on the node should be Running. + +## Diagnostic Steps + +1. GPU log flood (Cause 1): + + ```bash + kubectl describe node | grep -E 'Ready|PLEG' + systemctl status systemd-journald --no-pager | grep -iE 'Missed .* kernel messages|Memory' + journalctl -u systemd-journald --since "-1h" --no-pager | grep -iE 'lost|missed|overrun' + dmesg -T | grep -c NVRM + dmesg -T | grep -E 'RmInitAdapter failed|rm_init_adapter failed|may need to be reset' | tail -n 10 + ``` + + A large `NVRM` count together with `Missed ... kernel messages` is Cause 1. The PCI address in `RmInitAdapter failed!` is the card to reset. `PLEG is not healthy` with a clean kernel log is not this cause. + +2. Memory errors (Cause 2): + + ```bash + dmesg -T | grep -iE 'EDAC|Hardware Error|mce|memory error' | tail -n 20 + ras-mc-ctl --errors 2>/dev/null | tail -n 20 + ``` + + Also check the server BMC event log or the IaaS host events for the same time. Corrected or uncorrected memory errors for that host point at Cause 2. + +3. cpuset cgroup state (Cause 3): + + ```bash + mount | grep 'cgroup/cpuset' # "ro" in the options means read-only + ls /sys/fs/cgroup/cpuset/kubepods.slice/ 2>&1 | head + kubectl -n kube-system get pods -o wide --field-selector spec.nodeName= | grep kube-ovn + ``` + + `ro` on the cpuset mount with failing kube-ovn Pods is Cause 3. A node whose kube-ovn CNI fails after a NIC rename or a clock change is covered in [kube-ovn-cni fails on one node after a NIC rename, clock skew or hostname change on ACP](kube-ovn-cni_fails_on_one_node_after_a_NIC_rename_clock_skew_or_hostname_change_on_ACP.md). diff --git a/docs/en/solutions/Node_zombie_process_alert_keeps_switching_between_firing_and_resolved_on_ACP.md b/docs/en/solutions/Node_zombie_process_alert_keeps_switching_between_firing_and_resolved_on_ACP.md new file mode 100644 index 000000000..c7f5cd9fb --- /dev/null +++ b/docs/en/solutions/Node_zombie_process_alert_keeps_switching_between_firing_and_resolved_on_ACP.md @@ -0,0 +1,68 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.0.x +--- + +# Node zombie process alert keeps switching between firing and resolved on ACP + +## Issue + +Notifications for the node zombie process alert keep arriving as firing (`告警中`) and then resolved (`已恢复`), again and again, although the zombie process count seems unchanged. The rule is in the `uni-node-rules` alert policy: `节点的僵尸进程数 > 10 且持续 1 分钟` (number of zombie processes on the node > 10 for 1 minute). The chart of the rule in the policy may show `无数据,请确认表达式涉及到的监控指标是否存在` (no data, check whether the metrics in the expression exist), which suggests a monitoring problem. Changing the alert sending interval does not stop the alternation. + +Related article: [Alert repeat interval cannot be set to 4 hours, or the global sending interval does not take effect on ACP](Alert_repeat_interval_cannot_be_set_to_4_hours_or_the_global_sending_interval_does_not_take_effect_on_ACP.md). + +## Root Cause + +### Cause 1: the zombie process count on the node really crosses the threshold back and forth + +The rule expression is `node_processes_state{state="Z"}>10`. Zombie processes on the node appear and disappear intermittently, so the count goes above 10 for more than a minute, falls below it, and rises again. Monitoring collects the metric normally, and each crossing correctly produces a firing and then a resolved notification. The "no data" message on the policy chart does not mean that the metric is missing: the same expression returns data in vmui. + +## Resolution + +### Cause 1: the zombie process count on the node really crosses the threshold back and forth + +1. Confirm that the metric is collected and see how it moves. In vmui on that cluster, run the rule expression and the raw series over the last hours: + + ```promql + node_processes_state{state="Z"} > 10 + node_processes_state{state="Z"} + ``` + + Values above 10 on the node named in the alert (for example 18 or 128), alternating with lower values, confirm this cause. + +2. Confirm that the notifications match those crossings. The `courier-api` log in `cpaas-system` records each firing and resolved event of the alert: + + ```bash + kubectl get pods -n cpaas-system | grep courier-api + kubectl logs -n cpaas-system --tail 10000 | grep '' + ``` + + Expected: entries for alert `节点的僵尸进程数` with `alert_resource: uni-node-rules`, `state: "Z"`, a current value such as `"11"`, and `status: "resolved"` alternating with firing entries at the times the notifications arrived. + +3. Find out why zombie processes keep appearing on the node. A zombie is a child process that has exited but has not been reaped by its parent; on the node, list them with their parents: + + ```bash + ps -eo pid,ppid,stat,comm | awk '$3 ~ /^Z/' + ``` + + The parent process (`ppid`) is the one that does not reap its children. Exec probes are a common source; see [Defunct Processes From Exec Probes Driving Worker Node Pressure](Defunct_Processes_From_Exec_Probes_Driving_Worker_Node_Pressure.md). Fix or restart the owning workload. + +4. If the alternation is expected for that node and the alert should be quieter, change the rule's trigger condition (operator, value and duration) inside the alert policy, as described in [Create and manage alert policies and notification routing on ACP](Create_and_manage_alert_policies_and_notification_routing_on_ACP.md). + +## Diagnostic Steps + +1. Run `node_processes_state{state="Z"}` in vmui for the node in the alert. Data present and crossing 10 repeatedly: Cause 1. No data at all in vmui: a collection problem, not this article. + +2. Compare the notification times with the `courier-api` log entries for the alert (Resolution step 2). Matching firing/resolved pairs show that the alerting chain works as designed. + +3. On the node, count zombies a few times over several minutes: + + ```bash + for i in 1 2 3 4 5; do ps -eo stat | grep -c '^Z'; sleep 60; done + ``` + + Expected for this cause: counts that move above and below 10. diff --git a/docs/en/solutions/Notification_template_cannot_show_metric_labels_such_as_job_instance_or_the_workload_name_on_ACP.md b/docs/en/solutions/Notification_template_cannot_show_metric_labels_such_as_job_instance_or_the_workload_name_on_ACP.md new file mode 100644 index 000000000..b9fefe2e8 --- /dev/null +++ b/docs/en/solutions/Notification_template_cannot_show_metric_labels_such_as_job_instance_or_the_workload_name_on_ACP.md @@ -0,0 +1,115 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.14" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Notification template cannot show metric labels such as job, instance, or the workload name on ACP + +## Issue + +When an alert notification template (email, robot, or webhook) is customized, one of the following comes up: + +- A template that uses `{{ .labels.job }}`, `{{ .externalLabels.instance }}`, or another label seen on the metric in the Prometheus UI renders nothing. +- The alert message should contain the name of the workload, for example the StatefulSet or Deployment that triggered it, and it is not clear which variable holds it. +- An application alert policy that targets `.*` does not show the specific compute component name. +- Which fields does a standard alert notification carry, and which variables can a webhook template use? + +Related article: [Node disk alert notification shows an empty mountpoint or no value on ACP](Node_disk_alert_notification_shows_an_empty_mountpoint_or_no_value_on_ACP.md). + +## Root Cause + +### Cause 1: templates can read only the labels of the fired alert and the platform variables + +The template can use only: + +- `.labels`: the labels on the fired alert. These are the alert-rule labels, for example `alert_cluster`, `alert_for`, `alert_indicator`, `alert_involved_object_kind`, `alert_involved_object_name`, `alert_involved_object_namespace`, `alert_kind`, `alert_name`, `alert_namespace`, `alert_project`, `alert_resource`, `alertgroup`, `alertname`, `severity`, and `vmcluster`, plus any series labels that the rule expression keeps after aggregation. +- `.externalLabels`: values provided by the platform: `comparison`, `currentValue`, `object`, `severity`, `status`, `summary`, and `threshold`. + +Metric labels that are not on the fired alert cannot be added. For example, a test template that printed all labels of a `workload.pod.status.phase.not.running` alert on a Deployment showed only rule labels: `job`, `instance`, `endpoint`, and `service` from the Prometheus UI were not there. Custom variables cannot be defined: the display names and formats of the supported variables can be changed, but not the variables themselves. The metric name is available as `{{ .labels.alert_indicator }}` and the trigger value as `{{ .externalLabels.currentValue }}`. The supported list is in the "Reference Variables" section of the notification management documentation (**Observability** > **Monitoring** > **Operations Guide** > **Notification Management**). + +The "More webhook advanced variables (Alpha)" option is the `webhook-template` feature gate. Enabling it does not add `job` or `instance`. + +### Cause 2: the resource name is in alert_involved_object_name and the object in externalLabels.object + +For resource-type alerts, the workload is carried in the rule labels. For example, an alert from `cpaas-statefulset-rules` on `workload.pod.container.cpu.utilization` shows the object as `cpaas-system/工作负载vmselect-cluster` (`工作负载` means workload), and its labels include `alert_involved_object_namespace`, `alert_involved_object_name`, `alert_involved_object_kind: StatefulSet`, `alert_project`, `namespace`, `pod`, `container`, and `alertname`. When the policy's resource object sets a namespace and a resource name, the notification includes them. + +The notified user is not a template field: recipients, method, and template are chosen in the notification policy. + +### Cause 3: a target of .* has no single component name + +An application alert policy generated from a template that covers every Deployment in the namespace uses `.*` as the target. One policy then matches every Deployment, so the alert has no single component name to print in `alert_involved_object_name`. + +## Resolution + +### Cause 1: templates can read only the labels of the fired alert and the platform variables + +1. To see exactly which keys an alert carries, add these two lines to a test template (for example a group robot template), trigger one alert, and read the message: + + ```text + labels: {{ .labels }} + externalLabels: {{ .externalLabels }} + ``` + +2. Reference any key printed on the `labels:` line as `{{ .labels. }}`, and any key on the `externalLabels:` line as `{{ .externalLabels. }}`. A key that is not printed, such as `job`, cannot be used. + +3. To show the advanced variable list, an administrator opens `/console-platform/feature-gate` and enables `webhook-template`. + + > **Warning:** Enabling `webhook-template` changes how webhook alert bodies are built and may change the format of webhook alerts that receivers already parse. Do this only when the basic variables are not enough and every webhook receiver has been confirmed to accept the new format. To roll back, disable the gate on the same page. + +### Cause 2: the resource name is in alert_involved_object_name and the object in externalLabels.object + +In the notification template (**Operations Center** > **Notification** > **Templates**), use: + +- `{{ .labels.alert_involved_object_name }}` for the resource name, for example the StatefulSet name. +- `{{ .labels.alert_involved_object_namespace }}` for its namespace. +- `{{ .externalLabels.object }}` for the alert object as the platform displays it. + +A webhook template that uses the default variables, created at **Operations Center** > **Notification** > **Create Notification Template**, message type **Alert Message**: + +```text +{ + "status": "{{ .externalLabels.status }}", + "severity": "{{ .externalLabels.severity }}", + "cluster": "{{ .labels.alert_cluster }}", + "object": "{{ .externalLabels.object }}", + "resource_name": "{{ .labels.alert_involved_object_name }}", + "policy": "{{ .labels.alert_resource }}", + "summary": "{{ .externalLabels.summary }}", + "current_value": "{{ .externalLabels.currentValue }}", + {{- if eq .externalLabels.status "告警中" }} + "alert_time": "{{ dateFormatWithZone .startsAt "2006-01-02 15:04:05" "Asia/Shanghai" }}" + {{- else }} + "alert_time": "{{ dateFormatWithZone .startsAt "2006-01-02 15:04:05" "Asia/Shanghai" }}", + "recovery_time": "{{ dateFormatWithZone .endsAt "2006-01-02 15:04:05" "Asia/Shanghai" }}" + {{- end}} +} +``` + +The keys on the left are what the receiver sees; rename them to the receiver's schema. `告警中` is the firing status value. The `if` branch keeps the body valid JSON in both states: the firing branch has no trailing comma, and the recovery branch adds the recovery time. Bind the template to the notification policy. A webhook body is rendered from the template only when the `webhook-template` gate is on; see [Webhook alert notifications rejected as InvalidJson or sent without the notification template on ACP](Webhook_alert_notifications_rejected_as_InvalidJson_or_sent_without_the_notification_template_on_ACP.md), which also covers quoting values that can contain double quotes. + +A custom request header that a receiver requires is not a template variable; it is stored in a `NotificationSender` Secret. See [Webhook alert notifications fail with invalid header field value or miss a required custom header on ACP](Webhook_alert_notifications_fail_with_invalid_header_field_value_or_miss_a_required_custom_header_on_ACP.md). + +### Cause 3: a target of .* has no single component name + +To show one component's name, scope the policy to that component instead of `.*`. A policy that keeps the wildcard cannot print a single component name. Check what the policy targets: + +```bash +kubectl get prometheusrule -n -o yaml | grep alert_involved_object_name +``` + +## Diagnostic Steps + +1. Print `{{ .labels }}` and `{{ .externalLabels }}` in a test template and trigger one alert. Expected: the key you want is either on one of the two lines (use it), or absent (Cause 1; it cannot be added). + +2. For a workload name, check that `alert_involved_object_name` is on the `labels:` line (Cause 2). + +3. If `alert_involved_object_name` is `.*` in the PrometheusRule, the policy is a wildcard (Cause 3). diff --git a/docs/en/solutions/OIDC_clients_and_pages_are_sent_to_the_wrong_Dex_issuer_address_on_ACP.md b/docs/en/solutions/OIDC_clients_and_pages_are_sent_to_the_wrong_Dex_issuer_address_on_ACP.md new file mode 100644 index 000000000..4118517e5 --- /dev/null +++ b/docs/en/solutions/OIDC_clients_and_pages_are_sent_to_the_wrong_Dex_issuer_address_on_ACP.md @@ -0,0 +1,96 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.10" + - "3.10.2" + - "3.16" + - "4.0" + - "4.2" + - "4.2.1" + - "4.3" + - "4.3.2" + - "4.3.3" +--- + +# OIDC clients and pages are sent to the wrong Dex issuer address on ACP + +## Issue + +The platform Dex advertises an issuer address that does not match how users or OIDC clients reach the platform. One of these is reported: + +- Users open the platform through one address (for example a floating IP in front of the HA VIP), and Service Mesh v2 pages such as Kiali or Jaeger, each with its own access address, redirect to the VIP for Dex. +- SonarQube reports `Issuer URL in provider metadata doesn't match the issuer URI specified in plugin configuration`, and other OIDC clients (GitLab, Harbor, Nexus) fail the same way when they are configured with `https:///dex`. The Dex discovery document on the DNS name still returns `"issuer": "https:///dex"`. + +Related articles: "Dex login options, CORS, SSO user identity, and the Dex deployment layout on ACP" and "Dex update fails on ACP 3.10 or account login returns 401 after the Dex permanent token changes". + +## Root Cause + +### Cause 1: Dex does not switch the issuer when more than one access address is in use + +Dex publishes one issuer and does not change it dynamically according to the address the browser used. Registering extra callback URLs does not make the issuer follow the browser host. When the console and the Mesh pages are opened on different addresses, the pages that use Dex are sent to the issuer address. The supported layout is one domain name for access. + +### Cause 2: Dex still advertises the IP issuer after the platform moved to a DNS name + +The platform was installed with an IP address as its platform access address. A DNS name and a new certificate were added later by rotating the TLS certificates of the platform access addresses. That rotation lets browsers reach the DNS name; it does not change the Dex issuer. The current product does not support converting an IP-based platform access address to a DNS name in place, so there is no in-place fix on the existing platform. The way to get a DNS issuer is to redeploy the platform with the domain name as the access address. + +On ACP 4.3.2 the default platform access address cannot be changed after installation, and neither can the internal registry address; additional access addresses can still be added. + +## Resolution + +### Cause 1: Dex does not switch the issuer when more than one access address is in use + +Publish one domain name and open the platform, Service Mesh, Kiali, and Jaeger on that domain. + +Only when a product page must stay up without Dex, and its owners accept an unauthenticated UI, these per-product changes remove the dependency on the issuer: + +> **Warning:** Kiali with `spec.auth.strategy: anonymous` no longer asks who the user is; anyone who can open the Kiali address can use it. Precondition: only when Kiali's OpenID redirect to the VIP is the confirmed problem and a single domain is not available. Rollback: `kubectl apply -f /tmp/kiali.bak.yaml`. + +```bash +kubectl get kiali -A +kubectl get kiali -n -o yaml > /tmp/kiali.bak.yaml +kubectl patch kiali -n --type=merge -p '{"spec":{"auth":{"strategy":"anonymous"}}}' +``` + +> **Warning:** Removing `spec.additionalContainers` removes the `oauth2-proxy` container from the OpenTelemetryCollector, and pointing the Ingress at `jaeger-collector-extension:16686` serves Jaeger without that proxy, unauthenticated. Precondition: only when Jaeger is the page that bounces to the VIP for Dex. Rollback: `kubectl apply -f /tmp/otelcol.bak.yaml` and restore the previous Ingress backend. + +```bash +kubectl get opentelemetrycollector -A +kubectl get opentelemetrycollector -n -o yaml > /tmp/otelcol.bak.yaml +kubectl patch opentelemetrycollector -n --type=json \ + -p='[{"op":"remove","path":"/spec/additionalContainers"}]' +``` + +Then change the backend of the Jaeger Ingress to the Service `jaeger-collector-extension`, port `16686`. + +### Cause 2: Dex still advertises the IP issuer after the platform moved to a DNS name + +1. Confirm the issuer that OIDC clients receive on the DNS name, and the certificate that name presents: + + ```bash + curl -k -sS https:///dex/.well-known/openid-configuration | jq -r .issuer + openssl s_client -connect :443 -servername /dev/null \ + | openssl x509 -noout -subject -ext subjectAltName + ``` + + `issuer` is `https:///dex`: the platform was installed on the IP, and the issuer cannot be changed in place. + +2. Plan a redeployment of the platform with the domain name as the platform access address, and configure the OIDC clients with `https:///dex` after that. Redeploying is not realistic once user clusters already run workloads. Until then, OIDC clients that require an exact issuer match can only use the IP issuer that the platform already advertises. + +3. If the certificate subject or SAN does not contain the DNS name, also replace the platform access certificate with one that contains it. A browser warning such as `Not secure` on the DNS name with a certificate CN for a different name is a separate certificate problem. + +Adding a domain only for toolchain callbacks, without changing the issuer, is covered in "Toolchain SSO on a domain name after an IP-based install, and kube-apiserver OIDC for an external UI on ACP". + +## Diagnostic Steps + +```bash +curl -k -sS https:///dex/.well-known/openid-configuration | jq -r .issuer +kubectl -n cpaas-system get cm dex-configmap -o yaml | grep -n issuer +kubectl get oauth2clients.dex.coreos.com -n cpaas-system +``` + +- Several browser hosts are in use, and the console or Mesh pages jump to the VIP for Dex: Cause 1. +- An OIDC client reports `Issuer URL in provider metadata doesn't match`, and the discovery `issuer` on the DNS name is still the install-time IP: Cause 2. +- `Unregistered redirect_uri` for `/console-portal/` on a toolchain host, with `client_id=alauda-auth`: not an issuer problem; see "Harbor SSO through the platform Dex fails: wrong issuer, OAuth2Client name, or Unregistered redirect_uri console-portal". diff --git a/docs/en/solutions/Old_business_TLS_certificate_still_served_after_an_Ingress_ALB_rule_or_upstream_load_balancer_update_on_ACP.md b/docs/en/solutions/Old_business_TLS_certificate_still_served_after_an_Ingress_ALB_rule_or_upstream_load_balancer_update_on_ACP.md new file mode 100644 index 000000000..58e96b340 --- /dev/null +++ b/docs/en/solutions/Old_business_TLS_certificate_still_served_after_an_Ingress_ALB_rule_or_upstream_load_balancer_update_on_ACP.md @@ -0,0 +1,143 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "3.18" + - "4.0" + - "4.2" + - "4.3" +--- + +# Old business TLS certificate still served after an Ingress, ALB rule, or upstream load balancer update on ACP + +## Issue + +A business certificate, a new wildcard certificate, or an Ingress TLS Secret was updated for a published service, and clients still receive the previous certificate. An incognito browser window, or `openssl s_client`, still shows the old `notBefore` / `notAfter` dates and issuer. A typical report is "the domain certificate replacement does not take effect". + +This article covers the certificate that a published service presents through ALB (and, where present, an Ingress or an upstream load balancer). The platform access certificate (`dex.tls`) and platform component Secrets are covered elsewhere: see [Replace the platform access certificate in dex.tls on ACP](Replace_the_platform_access_certificate_in_dex.tls_on_ACP.md) and [Update a platform domain certificate stored in a cpaas-system Secret on ACP](Update_a_platform_domain_certificate_stored_in_a_cpaas-system_Secret_on_ACP.md). An ingress-nginx controller that serves a renewed certificate only after a restart is Cause 2 of [Business TLS certificate not renewed, not reloaded by ingress-nginx, or rejected with a format error on import on ACP](Business_TLS_certificate_not_renewed_not_reloaded_by_ingress-nginx_or_rejected_with_a_format_error_on_import_on_ACP.md). + +Related articles: *HTTPS to a published service fails with plain HTTP sent to HTTPS port, ERR_SSL_PROTOCOL_ERROR, or TLS alert 80 on ACP* (handshake and protocol errors instead of an old certificate). + +## Root Cause + +Three places can each present a certificate for the same hostname: the ALB rule for that host and port, an Ingress for that host, and an upstream load balancer in front of the cluster. Clients see whichever certificate is presented by the hop that terminates TLS for them. Updating the certificate in one place does not update the others. + +### Cause 1: the Ingress certificate changed and the ALB rule still selects the old one + +The business certificate lives in a Secret (sometimes with a copy in a ConfigMap), and only the Ingress was updated. The certificate selected on the ALB rule for that host was not updated with it, so the ALB keeps serving the old leaf. An Ingress annotation of the form `alb.networking.cpaas.io/tls: =/` and the ALB rule select a Secret; replacing a ConfigMap that holds a copy, or creating a new Secret that the rule does not reference, does not change what the ALB serves. + +A hostname that matches no rule certificate falls through to the default certificate of the HTTPS port. Such names keep the default certificate even after the rule for other hosts is updated; for them, the port default certificate is what has to change. + +### Cause 2: the ALB serves the new certificate and the upstream load balancer still serves the old one + +A new wildcard certificate was selected for the service, there is no second Ingress for that host, and `openssl s_client` against the ALB node shows the new leaf. Users who come through an upstream load balancer (for example an F5 appliance) still get the previous certificate, because that device terminates TLS and presents its own certificate configuration. + +Two designs are valid, and they have different fixes: + +- The upstream device should keep terminating TLS: its certificate has to be updated. +- Only the ALB certificate should be presented: the upstream device should stop terminating TLS and pass TCP through to ALB port 443. + +Forwarding decrypted HTTP from the upstream device to the ALB HTTPS port meets neither goal (see the related article on `The plain HTTP request was sent to HTTPS port`). + +### Cause 3: the certificate was replaced on the ALB and the Ingress certificate was left unchanged + +The service is published with both an Ingress and an ALB rule. The certificate selected on the ALB was replaced, but the Ingress TLS Secret was not, so clients that reach the service through the Ingress keep receiving the previous leaf. Replacing the Ingress certificate as well makes the new domain certificate take effect. + +If the direct ALB check still shows the old leaf, this is Cause 1, not Cause 3. + +## Resolution + +### Cause 1: the Ingress certificate changed and the ALB rule still selects the old one + +1. Read the annotation, the leaf the ALB serves, and the certificate in the Secret: + + ```bash + kubectl get ingress -n -o yaml + echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -dates -subject -issuer + kubectl get secret -n -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject -issuer + ``` + +2. When the Secret holds the new certificate and the ALB still serves the old leaf, open the ALB rule for that host and port and select the same Secret that the Ingress annotation names. This is a normal rule update; it does not need an ALB restart. + +3. If the hostname does not match a rule certificate and the HTTPS port default is the old leaf, update the default certificate of that port instead. + +4. Repeat the `openssl s_client` check against the ALB node and in an incognito browser window. Both should show the new leaf. + +### Cause 2: the ALB serves the new certificate and the upstream load balancer still serves the old one + +1. Compare the ALB node with the public name users type: + + ```bash + echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -dates -subject -issuer + echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -dates -subject -issuer + kubectl get ingress -A -o yaml | grep -F '' + ``` + +2. If the ALB node shows the new certificate, the public name shows the old one, and no other Ingress serves the host: + - When the upstream device should remain the TLS terminator, replace the certificate on the upstream device. + - When only the ALB certificate should be presented, remove the certificate from the upstream device and configure it to pass TCP through to ALB port 443, instead of updating its certificate in place. + +3. If the ALB node itself is still old, go back to Cause 1. If clients reach an Ingress whose Secret was not replaced, use Cause 3. + +### Cause 3: the certificate was replaced on the ALB and the Ingress certificate was left unchanged + +1. Compare the Ingress TLS Secret with the leaf the ALB already serves: + + ```bash + kubectl get ingress -A -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.spec.tls}{"\n"}{end}' + kubectl get secret -n -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject -issuer + ``` + +2. Replace the Ingress TLS Secret with the new certificate and key. + + > **Warning:** Applying a new TLS Secret changes the certificate that the Ingress presents for this host. A mismatched key or a not-yet-valid certificate breaks HTTPS for the host until the previous Secret is restored. Do this only when the service has both an Ingress and an ALB rule, the ALB certificate was already replaced, and this Ingress Secret still holds the old leaf. Back up the Secret first; roll back with `kubectl apply -f` of the backup. + + ```bash + umask 077 + kubectl get secret -n -o yaml > "$HOME/.yaml" + openssl x509 -noout -modulus -in tls.crt | openssl md5 + openssl rsa -noout -modulus -in tls.key | openssl md5 + kubectl create secret tls --cert=tls.crt --key=tls.key -n \ + --dry-run=client -o yaml | kubectl apply -f - + ``` + + The two `md5` values must match before you apply (for an RSA key; for an EC key compare `openssl x509 -noout -pubkey` with `openssl pkey -pubout`). + +3. Check the public name again: + + ```bash + echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -dates -subject -issuer + ``` + + If the Ingress Secret is already new and the ALB leaf is old, use Cause 1 instead. + +## Diagnostic Steps + +Collect three leaves before changing a rule or an upstream device: the Secret the Ingress names, the ALB node, and the public name. + +```bash +kubectl get ingress -n -o yaml +kubectl get secret -n -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject -issuer +echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -dates -subject -issuer +echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -dates -subject -issuer +``` + +| Secret | ALB node | Public name | Cause | +| --- | --- | --- | --- | +| new | old | old | 1 (rule certificate, or the HTTPS port default for an unmatched name) | +| new | new | old, no other Ingress for the host | 2 | +| Ingress Secret old, ALB rule certificate already replaced | new | old | 3 | + +If the client gets no certificate at all, a handshake alert, or an HTTP error instead of an old leaf, see the related article named under Issue. diff --git a/docs/en/solutions/OpenTelemetry_Collector_logs_cannot_list_resources_pods_because_its_ServiceAccount_has_no_ClusterRoleBinding_on_ACP.md b/docs/en/solutions/OpenTelemetry_Collector_logs_cannot_list_resources_pods_because_its_ServiceAccount_has_no_ClusterRoleBinding_on_ACP.md new file mode 100644 index 000000000..fdadf662a --- /dev/null +++ b/docs/en/solutions/OpenTelemetry_Collector_logs_cannot_list_resources_pods_because_its_ServiceAccount_has_no_ClusterRoleBinding_on_ACP.md @@ -0,0 +1,73 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# OpenTelemetry Collector logs cannot list resources pods because its ServiceAccount has no ClusterRoleBinding on ACP + +## Issue + +The OpenTelemetry Collector log repeats an RBAC error that engineers usually quote as `cannot list resources "pod" in API group`. The full Kubernetes form is: + +```text +pods is forbidden: User "system:serviceaccount::otel-collector" cannot list resource "pods" in API group "" at the cluster scope +``` + +The Collector Pod is Running, but processors that look up Kubernetes metadata (for example Pod attributes on spans) fail. The Collector was deployed by following a tracing deployment document, often into an application namespace instead of the namespace that document uses. + +Related articles: "asm-otel collectors keep restarting, Service Mesh JVM monitoring has no data, or an otel instance cannot be deleted" covers the Service Mesh collectors and a Java agent that cannot reach the Collector. + +## Root Cause + +The Collector runs as a ServiceAccount that the operator creates in the namespace of the `OpenTelemetryCollector` resource, for example `system:serviceaccount::otel-collector`. A Kubernetes ClusterRoleBinding names one ServiceAccount in one namespace. The deprecated deployment document grants cluster-wide read access only to `system:serviceaccount:cpaas-system:otel-collector` (ClusterRoleBinding `otel-collector:cpaas-system:cluster-admin`). When the Collector is deployed in any other namespace, the ServiceAccount actually in use has no binding and cannot list Pods across the cluster. + +The old tracing deployment documents are deprecated: the old Distributed Tracing pages are removed in ACP 4.4, and ACP 4.3 is the last version that contains them. Current deployments follow the Alauda Distributed Tracing documentation. + +## Resolution + +1. Redeploy the tracing stack by following the current Alauda Distributed Tracing documentation, for example "Installing Alauda Distributed Tracing (OpenSearch)" for distributed-tracing 2.1, instead of the deprecated document. This is the confirmed fix: the current procedure creates the Collector together with the permissions it needs. + + > **Warning:** Removing the old Collector stops trace and metric collection until the new Collector is Running; spans sent in that window are lost. Back up the old resource first with `kubectl -n get opentelemetrycollector -o yaml > /tmp/otel-collector.bak.yaml`. Rollback: apply the backup. + +2. If the existing Collector has to stay where it is until the redeployment, the binding can be pointed at the ServiceAccount that is really used. Grant only what the Collector needs; the example ClusterRole below allows reading Pods, Namespaces, and ReplicaSets, which is what Kubernetes attribute lookups usually need: + + ```bash + kubectl create clusterrole otel-collector-read --verb=get,list,watch --resource=pods,namespaces,replicasets.apps + kubectl create clusterrolebinding otel-collector-read: \ + --clusterrole=otel-collector-read --serviceaccount=:otel-collector + ``` + + Rollback: `kubectl delete clusterrolebinding otel-collector-read:` and `kubectl delete clusterrole otel-collector-read`. + +3. After the change the Collector log should no longer show the `cannot list resource "pods"` error. + +## Diagnostic Steps + +1. Read the full error and the ServiceAccount the Collector Pod runs as: + + ```bash + kubectl logs -n --tail=80 + kubectl get pod -n -o jsonpath='{.spec.serviceAccountName}{"\n"}' + kubectl get sa -n otel-collector -o jsonpath='{.metadata.ownerReferences}{"\n"}' + ``` + + The ServiceAccount is typically `otel-collector`, owned by the `OpenTelemetryCollector` resource. + +2. Test the permission of exactly that ServiceAccount: + + ```bash + kubectl auth can-i list pods -A --as=system:serviceaccount::otel-collector + ``` + + `no` confirms the missing permission. + +3. Find which subject the existing binding names: + + ```bash + kubectl get clusterrolebinding -o yaml | grep -B 8 -A 12 'name: otel-collector' + ``` + + A binding whose subject is ServiceAccount `otel-collector` in namespace `cpaas-system`, while the Collector runs in another namespace, matches this article. diff --git a/docs/en/solutions/OperatorHub_instances_or_pages_fail_from_a_broken_11780_VIP_mapping_or_marketplace-apiserver_restarts_on_ACP.md b/docs/en/solutions/OperatorHub_instances_or_pages_fail_from_a_broken_11780_VIP_mapping_or_marketplace-apiserver_restarts_on_ACP.md new file mode 100644 index 000000000..2280d48fa --- /dev/null +++ b/docs/en/solutions/OperatorHub_instances_or_pages_fail_from_a_broken_11780_VIP_mapping_or_marketplace-apiserver_restarts_on_ACP.md @@ -0,0 +1,119 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# OperatorHub instances or pages fail from a broken 11780 VIP mapping or marketplace-apiserver restarts on ACP + +## Issue + +An instance of an Operator-based tool cannot be created on a workload cluster although the Operator looks healthy, or the OperatorHub page itself cannot be opened. Typical reports: + +- GitLab (or Nexus, or another Operator on the same cluster) cannot create an instance. In OperatorHub the creation page shows `Resource is being initialized, please try again later` and stays there. +- The Operator Pods on the workload cluster (for example `gitlab-ce-operator`, `nexus-ce-operator-controller-manager`) are `Running`, the Operator log shows nothing wrong, and its CRDs and resources look normal. +- On the global cluster the OLM-related Pods are all `Running`, but their logs keep reporting that the `console-plugins` of that workload cluster cannot be retrieved, with `502 Bad Gateway`. +- OperatorHub, the cluster plugin pages and the Tekton pages cannot be opened at all, and the marketplace apiserver in the backend has an abnormal status. + +## Root Cause + +### Cause 1: marketplace-api on global gets 502 from one workload cluster because the 11780 VIP port mapping is broken + +The instance form depends on the global cluster reaching the workload cluster. `marketplace-api` on the global cluster calls the workload cluster's VIP on port `11780`, which the workload cluster ALB serves. When the port mapping for `11780` on the workload cluster VIP (on a firewall NAT, load balancer or VIP gateway) is not connected, global cannot reach the workload cluster on that port, and `marketplace-api` logs: + +```text +collect console-plugins in cluster failed. err: invalid status code: 502. response status: 502 Bad Gateway, response body: X-Error: 502 +``` + +OperatorHub resource initialization then fails, and no instance can be created for any Operator on that cluster. Restoring the `11780` VIP port mapping makes instance creation work again. + +### Cause 2: marketplace-apiserver is killed by its liveness probe before its cache loads + +Several console pages that depend on the marketplace fail together: OperatorHub, the cluster plugin list, Tekton and others. `marketplace-apiserver` loads a cache of the cluster's CRD and APIService resources at startup. In an environment with a large number of those resources, the load has not finished when the liveness probe starts to count failures, so the container is killed and started again before it can serve. Raising `livenessProbe.initialDelaySeconds` of `marketplace-apiserver` to `180` lets the cache finish loading, and the pages recover. + +## Resolution + +### Cause 1: marketplace-api on global gets 502 from one workload cluster because the 11780 VIP port mapping is broken + +Only when `marketplace-api` (or the OLM Pods) on global log `502` for that one workload cluster, and a connection from the global nodes to the workload VIP on `11780` fails or does not reach the workload ALB. + +1. Restore the port mapping of the workload cluster VIP for `11780` on the device that does the mapping, so that it forwards to the workload cluster ALB that serves `11780`. This is usually done by the network team. + + > **Warning:** Changing NAT or load-balancer mappings on a shared device can affect other services that use it. Confirm the backends are the workload cluster ALB nodes for that cluster, save the device configuration, and use a change window agreed with the network owner. To roll back, restore the saved mapping. + +2. Verify from more than one global node. The first two commands should connect and get an HTTP response from the ALB instead of a timeout, and the `marketplace-api` log should stop reporting `502` for that cluster: + + ```bash + nc -vz -w 3 11780 + curl -sv --max-time 5 http://:11780/ -o /dev/null + kubectl logs -n cpaas-system deploy/marketplace-api --since=10m | grep -E '502|console-plugins' + ``` + +3. Create the instance again in OperatorHub. + +### Cause 2: marketplace-apiserver is killed by its liveness probe before its cache loads + +Only when the `marketplace-apiserver` Pod keeps being restarted by its liveness probe (or never becomes ready) and the cluster holds a large number of CRDs or APIServices. A `502` toward one workload cluster with a healthy `marketplace-apiserver` is Cause 1 instead. + +1. Find the workload, save it, and read the current probe delay per container: + + ```bash + kubectl get deploy -A | grep -i marketplace + kubectl -n get deploy -o yaml > /tmp/marketplace-apiserver.before.yaml + kubectl -n get deploy -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" "}{.livenessProbe.initialDelaySeconds}{"\n"}{end}' + ``` + +2. Raise the initial delay to `180` seconds. + + > **Warning:** Changing the probe rolls the `marketplace-apiserver` Pod, so OperatorHub and cluster plugin pages stay unavailable until the new Pod is ready. A longer initial delay also means a hung container is detected later after each start. To roll back, patch `initialDelaySeconds` back to the saved value, or run `kubectl apply -f /tmp/marketplace-apiserver.before.yaml`. + + ```bash + kubectl -n patch deploy -p '{"spec":{"template":{"spec":{"containers":[{"name":"","livenessProbe":{"initialDelaySeconds":180}}]}}}}' + kubectl -n rollout status deploy/ + ``` + + If the patch is denied by `vsentry.kb.io` with a message that a `ResourcePatch` was created, the change is applied through the platform's `sentry` component; wait and check the Deployment again instead of patching a second time. + +3. Reload OperatorHub, the cluster plugin page and the Tekton pages. This is a temporary change: note it and check the value again after a platform upgrade. + +## Diagnostic Steps + +1. On the workload cluster, rule out the Operator. Running Pods with a clean log mean the failure is earlier in the path: + + ```bash + kubectl get pod -A | grep -E 'gitlab-ce-operator|nexus-ce-operator' + kubectl -n logs --tail=50 + ``` + +2. On the global cluster, check the marketplace and OLM Pods and look for `502` toward one cluster: + + ```bash + kubectl get pod -A | grep -E 'market|olm' + kubectl logs -n cpaas-system deploy/marketplace-api --tail=200 | grep -E '502|console-plugins' + ``` + +3. From several global nodes, test the workload cluster VIP on `6443` and `11780`. `6443` answering while `11780` fails points at the `11780` mapping (Cause 1): + + ```bash + for p in 6443 11780; do nc -vz -w 3 $p; done + ``` + +4. When whole pages fail rather than one instance form, check the `marketplace-apiserver` Pod for liveness-probe restarts and count the API resources it has to load. Restarts with `Liveness probe failed` events in a cluster with many CRDs or APIServices is Cause 2: + + ```bash + kubectl get pod -A | grep -i marketplace + kubectl -n describe pod | grep -iE 'Liveness|Restart Count|Killing' + kubectl -n logs --previous --tail=50 + kubectl get crd --no-headers | wc -l + kubectl get apiservice --no-headers | wc -l + ``` + +5. Look-alikes: + + - `11780` accepts connections but the load balancer forwards it to the wrong backends: fix the load-balancer backends; the symptom is the same `502`. + - `marketplace-api` gets `502` for a cluster whose ALB Pods are not Ready: fix the workload cluster ALB first. + - Global cannot reach the workload VIP on `6443` either: see [Workload cluster abnormal or monitoring empty in the console because global cannot reach its ports on ACP](Workload_cluster_abnormal_or_monitoring_empty_in_the_console_because_global_cannot_reach_its_ports_on_ACP.md). diff --git a/docs/en/solutions/Operator_or_plugin_ImagePullBackOff_or_exec_format_error_registry_address_401_architecture_or_search_domain_on_ACP.md b/docs/en/solutions/Operator_or_plugin_ImagePullBackOff_or_exec_format_error_registry_address_401_architecture_or_search_domain_on_ACP.md new file mode 100644 index 000000000..c4f54b4c5 --- /dev/null +++ b/docs/en/solutions/Operator_or_plugin_ImagePullBackOff_or_exec_format_error_registry_address_401_architecture_or_search_domain_on_ACP.md @@ -0,0 +1,240 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.1" + - "4.2" +--- + +# Operator or plugin ImagePullBackOff or exec format error: registry address, 401, architecture, or search domain on ACP + +## Issue + +- An operator deployed from OperatorHub (for example `knative-operator` on `global`, installed for all namespaces) stays in Deploying (`部署中`), and its Pod events say the image cannot be pulled. +- The Knative Operator from the marketplace cannot be installed into a namespace because its image cannot be downloaded. Editing the image on the Deployment works for a while and is then changed back. +- Installing the ClickHouse plugin on `global` times out and the console shows the installation as failed; `cpaas-elasticsearch-0` is in `ImagePullBackOff`. +- Cluster functional components stay in upgrading, and the `connectors-operator` Pod on an ARM business cluster crashes with `exec /manager: exec format error`. +- After the global cluster recovers from a failure, platform component Pods in a workload cluster stay in `ImagePullBackOff`; a manual pull from the platform registry fails with `received unexpected HTTP status: 500 Internal Server Error`. + +A plugin that waits for a Pod that is Pending or failing on its node for reasons other than the image is covered in "Cluster plugin stuck at wait resources ready: Pod Pending, /cpaas I/O error, or missing log secret on ACP". + +## Root Cause + +### Cause 1: the registry address was changed and the running Pod still pulls from the old address + +The Pod event is `BackOff: Back-off pulling image "/:"`, repeated a very large number of times, and the registry in the event is not the one the platform is configured with now. After the image registry address of the platform is changed, an operator Pod that was created before the change (for example `knative-operator` in namespace `operators`) still references the old address, so every pull fails and the operator stays in Deploying. Recreating the Pod makes it pull from the current address. + +### Cause 2: the image points to a registry domain the cluster cannot reach, and Deployment edits are reverted + +The events of the operator Pods (for example `knative-operator-` and `operator-webhook-`) show: + +```text +Failed to pull image "/:": rpc error: code = DeadlineExceeded desc = failed to pull and unpack image ... failed to resolve reference ... dial tcp ...:443: i/o timeout +``` + +followed by `Back-off pulling image` and `Error: ImagePullBackOff`. + +The operator's image references use a registry domain that the cluster cannot resolve or reach. Normally Metis, the platform webhook controller that passes values from upper-level resources down to workloads, replaces such image addresses with the platform registry address; when it does not, the Pods keep the unreachable domain. The same repository and tag pull fine from the platform registry address of the cluster (Clusters → `` → Overview → Basic Info → Image Repository Address, for example an `IP:11443` address). Editing the image on the Deployment does not last, because the operator Deployment is owned by its ClusterServiceVersion (CSV) and OLM writes the CSV's spec back. Changing the image address in the CSV propagates to the Deployments, and the operator starts. + +For Knative, the components from the platform marketplace are supported; open-source Knative Operators that are not from the product are not. + +### Cause 3: 401 Unauthorized from the registry token service, a platform image cannot be pulled + +A platform Pod (for example `cpaas-elasticsearch-0` in `cpaas-system`, image `ait/elasticsearch-basic-http:`) cannot pull its image: + +```text +failed to authorize: failed to fetch oauth token: unexpected status from GET request to https:///auth/token?scope=repository%3Aait%2Felasticsearch-basic-http%3Apull&service=token-service: 401 Unauthorized +``` + +then `ErrImagePull`, `ImagePullBackOff`, and `Back-off pulling image`; a plugin install that needs the Pod times out. The registry token service refused the pull with `401`. Pulling the image manually on the node and then recreating the Pod recovers it. + +### Cause 4: exec format error, the operator package that was pushed has no image for the cluster's architecture + +`kubectl logs` of the operator Pod returns `exec /manager: exec format error`. The image itself was pulled successfully, the `manager` container restarts, and the event is `Back-off restarting failed container manager`. The cluster runs on another architecture than the image (for example `global` is x86 and the business cluster is ARM). + +The operator package that was pushed contained only the x86 image, so the ARM nodes run an x86 binary. Pushing the multi-architecture (hybrid) package of the same operator version with `violet push --force` overwrites the package, the business cluster pulls the ARM image, and the Pod reaches `Running`. Triggering the upgrade of the stuck components from OperatorHub then clears the upgrading state. + +### Cause 5: pulls return 500 because a node search domain makes registry name lookups time out + +Platform component Pods on a workload cluster stay in `ImagePullBackOff`. A manual pull on a node fails with: + +```text +Error response from daemon: received unexpected HTTP status: 500 Internal Server Error +``` + +and `curl -kv` against the platform registry, and against its backend address, also returns `500`. + +The node `/etc/resolv.conf` (written by NetworkManager) carries an extra search domain such as `linklocal`. Pods with `dnsPolicy: ClusterFirst` inherit the node search domains after the cluster suffixes, with `options ndots:5`; the `alb2` container of the global ALB, for example, shows `search cpaas-system.svc.cluster.local svc.cluster.local cluster.local linklocal`. Every lookup of the registry domain and of in-cluster Service names is then also tried with `.linklocal` appended, CoreDNS forwards those names upstream, and the lookups time out. The CoreDNS log shows: + +```text +[ERROR] plugin/errors: 2 .linklocal. A: read udp :->:53: i/o timeout +``` + +for names such as the platform registry domain, `registry.kube-system.svc.cluster.local.linklocal.`, and `erebus.cpaas-system.svc.cluster.local.linklocal.`. The component that serves the registry cannot resolve what it needs in time and answers `500`. Removing the search domain from the node `/etc/resolv.conf` and recreating the ALB and CoreDNS Pods restores the pulls. + +## Resolution + +### Cause 1: the registry address was changed and the running Pod still pulls from the old address + +1. Compare the registry in the event with the current Image Repository Address of the cluster: + + ```bash + kubectl get pods -A | grep -i + kubectl -n describe pod | grep -E 'Image:|Back-off pulling' + ``` + +2. Recreate the Pod. + + > **Warning:** Deleting the operator Pod restarts the operator; its reconciliation pauses until the new Pod is Running. Precondition: only when the registry address was changed and the Pod event still pulls from the old address. Rollback: none needed; the Deployment recreates the Pod. + + ```bash + kubectl -n delete pod + kubectl -n get pod -w | grep + ``` + +3. If the new Pod still names the old registry, the address is stored in the workload spec; continue with Cause 2. + +### Cause 2: the image points to a registry domain the cluster cannot reach, and Deployment edits are reverted + +1. Read the image of the Deployment and find the CSV: + + ```bash + kubectl get po -A | grep + kubectl -n get deploy -o jsonpath='{.spec.template.spec.containers[*].image}{"\n"}' + kubectl get csv -A | grep + kubectl get po -A | grep metis + ``` + +2. On a node, test both addresses. The domain failing and the platform registry address working is this cause: + + ```bash + crictl pull /: + crictl pull /: + ``` + +3. Save the CSV: + + ```bash + kubectl -n get csv -o yaml > "$HOME/.before.yaml" + ``` + +4. Change the image in the CSV, not on the Deployment. + + > **Warning:** Saving the CSV makes OLM roll the operator Deployments with the new image; the operator restarts. Precondition: only when the Pods are in `ImagePullBackOff` for an image whose registry domain the cluster cannot reach, and the same repository and tag pull from the platform registry address. Rollback: `kubectl apply -f "$HOME/.before.yaml"`. + + ```bash + kubectl -n edit csv + ``` + + Replace `` with the platform registry address in every image under `.spec.install.spec.deployments[].spec.template.spec.containers[].image` (and `relatedImages` if present), and save. + +5. Check the Pods: + + ```bash + kubectl get po -A | grep + ``` + +If other operators from the marketplace show the same unreplaced domain, collect the Metis Pod log (`kubectl -n logs --tail=300`) for Alauda technical support before editing more CSVs. + +### Cause 3: 401 Unauthorized from the registry token service, a platform image cannot be pulled + +1. On the node that runs the Pod, pull the image the event names: + + ```bash + kubectl -n cpaas-system get pod cpaas-elasticsearch-0 -o wide + kubectl -n cpaas-system describe pod cpaas-elasticsearch-0 | grep -E 'Image:|Failed' + crictl pull /ait/elasticsearch-basic-http: + ``` + + If the plain pull also returns `401`, pull with the platform registry credentials: + + ```bash + crictl pull --creds ':' /ait/elasticsearch-basic-http: + ``` + +2. Recreate the Pod. + + > **Warning:** Deleting the Pod restarts that Elasticsearch replica; it rejoins after it starts. Precondition: only after the image is present on that node (`crictl images | grep elasticsearch-basic-http`) and the Pod is in `ImagePullBackOff` with the `401 Unauthorized` token error. Rollback: none needed; the StatefulSet recreates the Pod. + + ```bash + kubectl -n cpaas-system delete pod cpaas-elasticsearch-0 + kubectl -n cpaas-system get pod cpaas-elasticsearch-0 -w + ``` + +3. Retry the plugin install. + +### Cause 4: exec format error, the operator package that was pushed has no image for the cluster's architecture + +1. Compare the node architecture with the crash: + + ```bash + kubectl get nodes -o custom-columns=NAME:.metadata.name,ARCH:.status.nodeInfo.architecture + kubectl -n get pods + kubectl -n logs --previous --tail=20 + ``` + +2. Push the multi-architecture package of the same version. + + > **Warning:** `--force` overwrites the package, Artifact, and ArtifactVersion already on the platform for this operator version. Precondition: only when the operator Pod logs `exec format error` on nodes of one architecture and the package you push is the multi-architecture (hybrid) build of the same version. Rollback: none for the overwritten content; push the previous file again with `--force` if needed. + + ```bash + violet push .tgz --platform-address --platform-username --platform-password '' --clusters --force + ``` + +3. Trigger the upgrade of the stuck components from OperatorHub on that cluster, and check that the Pod pulls the ARM image and runs: + + ```bash + kubectl -n get pods -w + ``` + +### Cause 5: pulls return 500 because a node search domain makes registry name lookups time out + +1. Confirm the search domain in a platform Pod and the timeouts in CoreDNS (on the global cluster for the global ALB): + + ```bash + kubectl -n cpaas-system get pods | grep alb + kubectl -n cpaas-system exec -c alb2 -- cat /etc/resolv.conf + kubectl -n kube-system logs -l k8s-app=kube-dns --tail=200 | grep -i 'i/o timeout' + ``` + +2. On each node whose `/etc/resolv.conf` carries the extra search domain, back the file up and comment out the `search` entry. + + > **Warning:** Removing the node search domain changes how host processes resolve short names; anything on the host that relied on that suffix must use full names afterwards. Precondition: only when Pod `resolv.conf` files carry the site suffix and CoreDNS logs `i/o timeout` for names with that suffix appended. Rollback: restore the backup file and recreate the Pods again. + + ```bash + cp -a /etc/resolv.conf /root/resolv.conf.before + grep -nE '^(search|domain)' /etc/resolv.conf + sed -i -E 's/^(search|domain)([[:space:]])/# \1\2/' /etc/resolv.conf + cat /etc/resolv.conf + ``` + + When NetworkManager writes the file, also remove the search domain from the NetworkManager connection, or the next NetworkManager restart puts it back. + +3. Recreate the ALB and CoreDNS Pods so that their `/etc/resolv.conf` is written again. + + > **Warning:** Restarting the ALB interrupts the traffic it carries, including console access, until the new Pods are Ready, and a CoreDNS restart briefly delays lookups. Precondition: the node files are fixed first. Rollback: none needed; the controllers recreate the Pods. + + ```bash + kubectl -n cpaas-system delete pod + kubectl -n kube-system rollout restart deploy/coredns + ``` + +4. Pull the image again on a workload cluster node and watch the Pods leave `ImagePullBackOff`. + +For the same search-domain mechanism behind console and DevOps timeouts, see "Platform UI stuck loading or DevOps errors from Service name timeouts caused by node search domains on ACP". + +## Diagnostic Steps + +```bash +kubectl get pods -A | grep -E 'ImagePullBackOff|ErrImagePull|CrashLoopBackOff' +kubectl -n describe pod | grep -E 'Image:|Failed|Back-off' +kubectl -n logs --previous --tail=20 +``` + +- A back-off on an image from a registry address that was changed: Cause 1. +- `i/o timeout` or a resolution failure for a build registry domain, with Deployment edits reverted: Cause 2. +- `failed to fetch oauth token ... 401 Unauthorized`: Cause 3. +- A successful pull followed by `exec format error` on nodes of one architecture: Cause 4. +- `received unexpected HTTP status: 500` from the platform registry, and CoreDNS `i/o timeout` errors for names ending in an extra suffix such as `.linklocal.`: Cause 5. diff --git a/docs/en/solutions/Operator_service_certificates_such_as_packageserver-service-cert_about_to_expire_or_not_renewed_on_ACP.md b/docs/en/solutions/Operator_service_certificates_such_as_packageserver-service-cert_about_to_expire_or_not_renewed_on_ACP.md new file mode 100644 index 000000000..bc1621198 --- /dev/null +++ b/docs/en/solutions/Operator_service_certificates_such_as_packageserver-service-cert_about_to_expire_or_not_renewed_on_ACP.md @@ -0,0 +1,149 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.12.x + - 3.14.x + - 3.18.x + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# Operator service certificates such as packageserver-service-cert about to expire or not renewed on ACP + +## Issue + +An alert or a platform inspection says an operator service certificate is about to expire, or has expired, and the question is whether it renews by itself. Typical alert text from policy `cpaas-certificates-rule`: + +- `packageserver-service-cert即将过期,过期时间低于30.00d天` (packageserver-service-cert is about to expire, less than 30.00 days) +- `位于 / harbor-operator 的证书 ( harbor-operator-controller-manager-service-cert ) 即将过期,过期时间低于30.00d天` (the certificate in / harbor-operator is about to expire) +- "platform component certificate expiring soon (less than 10 days) <= 10d" + +Typical names are `packageserver-service-cert` (namespace `cpaas-system`), `katanomi-operator-service-cert`, `redis-operator-service-cert`, `rds-operator-controller-manager-service-cert` (namespace `operators`), `harbor-operator-controller-manager-service-cert` (namespace `operators` or `harbor-operator`), `jaeger-operator-service-cert` (namespace `istio-system`), `metallb-operator-webhook-server-service-cert` (namespace `metallb-system`), and `opentelemetry-operator-controller-manager-service-cert`. + +Related reports: the cert-manager `Certificate` for `packageserver-service-cert` passed its `renewalTime` and nothing renewed; the platform certificate list shows Certificate `packageserver-service-cert` with Ready `False`; or the Secret was deleted, came back with a new two-year date, and the inspection or certificate page still flags it. + +## Root Cause + +### Cause 1: an operator service certificate renews about one day before expiry, so the 30-day and 10-day alerts fire first + +Operator Lifecycle Manager (OLM) generates and rotates the certificates of the webhooks and API services that an operator's ClusterServiceVersion (CSV) defines. These service certificates are owned by `olm-operator` (or, for the katanomi certificates, by `katanomi-operator`), not by cert-manager. cert-manager only monitors their expiry. A 30-day `renewBefore` shown on a `Certificate` object applies only to certificates that cert-manager issues. + +The owning operator normally rotates each certificate about one day before it expires, so the 30-day and 10-day alerts fire long before anything rotates. When the alert is inside that window, no action is required. To renew earlier, back up and delete the Secret: the operator writes a new certificate, valid for about two years. Silencing the alert is the other option. + +The certificate status page and the inspection keep showing the old date until `courier` on the global cluster refreshes, about every 4 hours, or until `courier` is restarted. + +### Cause 2: packageserver-service-cert is rotated by OLM at CSV status.certsRotateAt, not at the Certificate renewalTime + +`packageserver-service-cert` is an OLM self-signed certificate. OLM issues it and writes the Secret directly, so the cert-manager `Certificate` of the same name can show a `renewalTime` 30 days before expiry while OLM has not rotated the certificate the alert counts. The Certificate can also show condition `Issuing` `True` with reason `SecretMismatch` and message `Existing private key is not up to date for spec: [spec.privateKey.algorithm]`, while `Ready` is `True`. Nothing rotates at that `renewalTime`. + +The time OLM rotates it is `status.certsRotateAt` on the CSV `packageserver` in `cpaas-system`, about one day before the Secret's `notAfter`. Moving `certsRotateAt` to an earlier time makes OLM rotate immediately, and the certificate then shows Normal with a new two-year expiry. If this certificate does expire, the operator components fail and the OperatorHub page cannot be displayed. + +### Cause 3: Certificate packageserver-service-cert shows Ready False while tls.crt in the Secret is valid + +The certificate packageserver actually uses is `tls.crt` in Secret `cpaas-system/packageserver-service-cert`, which OLM writes and renews when it is due. A `Ready=False` status on the `Certificate` object has no functional impact on packageserver while the certificate in the Secret is valid. No action is needed. + +## Resolution + +### Cause 1: an operator service certificate renews about one day before expiry, so the 30-day and 10-day alerts fire first + +1. On the cluster the alert names (often a business cluster, not global), find the Secret and print its dates, and read the CSV rotation times: + + ```bash + kubectl get secrets -A | grep -E 'packageserver-service-cert|katanomi-operator-service-cert|redis-operator-service-cert|rds-operator-controller-manager-service-cert|harbor-operator-controller-manager-service-cert|jaeger-operator-service-cert|metallb-operator-webhook-server-service-cert' + kubectl get secret -n -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject -issuer + kubectl get csv -A -o yaml | grep -n certsRotateAt + ``` + + If `notAfter` is more than a day ahead and an early renewal is not required, wait for the operator to rotate it. + +2. To renew now, or when the certificate has already expired, back up and delete the Secret. + + > **Warning:** Deleting the Secret removes the certificate the operator webhook or API service is serving until `olm-operator` writes a new one; API calls that hit that webhook fail in the gap. Do this only for the operator service certificate names above, after `openssl` confirmed the dates, and only with a YAML backup. Rollback is `kubectl apply -f` of the backup, which restores the old certificate; apply it also if the Secret is not recreated. + + ```bash + umask 077 + kubectl get secret -n -o yaml > "$HOME/.yaml" + kubectl delete secret -n + kubectl get secret -n + kubectl get secret -n -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject -issuer + ``` + + The new `notAfter` should be about two years ahead. If the Secret does not come back, restore the backup and see "Platform component certificate not renewed: stale private key, issuerRef group, or alert name that differs from the Secret on ACP" (for example `rds-operator-controller-manager-service-cert` with `issuerRef.group: ait.alauda.io`). + +3. If the certificate page or a re-run inspection still shows the previous date, wait about 4 hours, or restart the `courier` and `courier-api` Pods in `cpaas-system` on the global cluster and run the inspection again. + + > **Warning:** Restarting the global `courier` Pods interrupts alert delivery and console certificate status until they are Running again. Do this only after the new Secret's `notAfter` is in the future. The workload recreates the Pods. + + ```bash + kubectl get pods -n cpaas-system | grep courier + kubectl delete pod -n cpaas-system + ``` + +4. The alert can also be silenced: **Platform Management > Operations Center > Alerts > Alert Policies**, policy `cpaas-certificates-rule` on the cluster, action **Set Silence**. + +### Cause 2: packageserver-service-cert is rotated by OLM at CSV status.certsRotateAt, not at the Certificate renewalTime + +1. Read both schedules and the Secret dates: + + ```bash + kubectl -n cpaas-system get csv packageserver -o jsonpath='{.status.certsRotateAt}{"\n"}' + kubectl get certificate -n cpaas-system packageserver-service-cert -o yaml | grep -E 'renewBefore|renewalTime|notAfter|reason|message' + kubectl get secret -n cpaas-system packageserver-service-cert -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates + ``` + + `certsRotateAt` about one day before `notAfter` is the normal OLM schedule. + +2. To rotate earlier, either back up and delete the Secret (Cause 1, step 2), or move `certsRotateAt` so that OLM rotates now. + + > **Warning:** Patching `status.certsRotateAt` makes OLM rotate `packageserver-service-cert` at the new time; the packageserver API service serves the new certificate from then on, and requests in flight during the rotation can fail. Do this only when the CSV shows `certsRotateAt` about one day before `notAfter` and the Secret has not been rotated. Save the CSV first. Once OLM has rotated the certificate there is nothing to roll back. `--subresource=status` needs kubectl 1.24 or later. Use a full RFC 3339 time, already past or a few minutes ahead. + + ```bash + kubectl -n cpaas-system get csv packageserver -o yaml > "$HOME/csv-packageserver.yaml" + kubectl patch csv -n cpaas-system packageserver \ + --type merge \ + --subresource=status \ + --patch '{"status":{"certsRotateAt":""}}' + kubectl get secret -n cpaas-system packageserver-service-cert -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates + ``` + + The new `notAfter` should be about two years ahead. Restart `courier` as in Cause 1 if the page keeps the old date. + +### Cause 3: Certificate packageserver-service-cert shows Ready False while tls.crt in the Secret is valid + +1. Read the certificate from the Secret, not from the Certificate status: + + ```bash + kubectl get secret -n cpaas-system packageserver-service-cert -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates + kubectl get certificate -n cpaas-system packageserver-service-cert + ``` + +2. When `notAfter` is in the future, packageserver is unaffected and no action is needed; OLM renews the Secret when it is due. Only when `notAfter` is close or past and the Secret has not been renewed, use step 2 of Cause 1. + +## Diagnostic Steps + +1. Read the alert's cluster, namespace, and certificate name, switch to that cluster, and print the dates: + + ```bash + kubectl get secret -A | grep + kubectl get secret -n -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates + kubectl get certificate -A | grep + ``` + +2. Branch: + + - An operator service certificate from the list above near expiry: Cause 1. + - `packageserver-service-cert` whose Certificate shows `SecretMismatch` with `[spec.privateKey.algorithm]` and passed `renewalTime` without rotating: read CSV `certsRotateAt`, then Cause 2. + - Certificate `packageserver-service-cert` Ready `False` while the Secret's `notAfter` is in the future: Cause 3, no action. + - Secret recreated with a new date, and the page or inspection still flags it: step 3 of Cause 1. + - Secret not recreated after the delete, or not found on the cluster you searched: "Platform component certificate not renewed: stale private key, issuerRef group, or alert name that differs from the Secret on ACP". + - A `*-serving-cert` with about 30 days left: "Platform certificate expiry alert for a cert-manager serving certificate that renews by itself on ACP". diff --git a/docs/en/solutions/Operator_stays_Processing_base-operator_logs_certwatch_errors_or_a_CSV_keeps_raising_DisabledCopiedCSVs_events_on_ACP.md b/docs/en/solutions/Operator_stays_Processing_base-operator_logs_certwatch_errors_or_a_CSV_keeps_raising_DisabledCopiedCSVs_events_on_ACP.md new file mode 100644 index 000000000..2717d6841 --- /dev/null +++ b/docs/en/solutions/Operator_stays_Processing_base-operator_logs_certwatch_errors_or_a_CSV_keeps_raising_DisabledCopiedCSVs_events_on_ACP.md @@ -0,0 +1,108 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Operator stays Processing, base-operator logs certwatch errors, or a CSV keeps raising DisabledCopiedCSVs events on ACP + +## Issue + +One of the following is seen on a cluster whose operators otherwise run normally: + +- The cluster functional-component page shows `redis-operator` as Processing, and it stays there. The operator Pod is not crash-looping. A managed instance under that operator is paused, and the ModuleInfo (`minfo`) of the component reports Processing. +- `base-operator` logs keep printing certificate reread errors such as `error re-reading certificate` with `tls: private key does not match public key`, and reconciliation reports `ProductBase Reconcile error: ProductBase need retry`. `base-operator` is otherwise healthy. +- A ClusterServiceVersion (CSV) keeps raising the same warning event, for example `DisabledCopiedCSVs: CSV copying disabled for operators/katanomi-operator.v3.20.36` hundreds of times over a few weeks, and the question is whether it is a fault. + +`redis-operator` that stays Processing during a platform upgrade because its Subscription waits for manual approval is a different cause; see "ACP upgrade stuck on an operator, OLM catalog, or operator-managed component". + +## Root Cause + +### Cause 1: redis-operator stays Processing because a managed instance is paused + +When an instance managed by an operator is paused, the platform's operator component logic reports the cluster functional component as Processing on its `minfo`. This is the expected display for a paused instance, not an operator failure. While the instance stays paused, the component stays Processing; when the instance is resumed, `minfo` and the functional-component page follow. + +### Cause 2: error re-reading certificate, tls: private key does not match public key + +`base-operator` watches its certificate files with certwatch, which relies on file system notifications. During certificate rotation `tls.crt` is updated first and `tls.key` after it. A reread that runs in this gap sees a certificate and key that do not belong together, or a file that is not fully written yet, and logs one of: + +```text +error re-reading certificate ... tls: private key does not match public key +error re-reading certificate ... tls: failed to find any PEM data in key input +error re-reading certificate ... tls: failed to find any PEM data in certificate input +``` + +When `tls.key` has finished updating, the pair matches again. `base-operator` and `prdb` stay healthy through these lines, the errors do not affect use of the product, and a `ProductBase need retry` from the same gap clears with the file update. + +### Cause 3: repeated DisabledCopiedCSVs warning events on katanomi or Knative operator CSVs + +The event list of the `operators` namespace shows, for example: + +```text +DisabledCopiedCSVs: CSV copying disabled for operators/knative-operator.v3.20.10 +DisabledCopiedCSVs: CSV copying disabled for operators/katanomi-operator.v3.20.36 +``` + +with counts in the hundreds over about two weeks. OLM has CSV copying turned off (`spec.features.disableCopiedCSVs: true` on the `OLMConfig` named `cluster`), and the event reports that state for each cluster-wide operator CSV. It is informational and not an error; the operators work normally and the event count simply grows. + +## Resolution + +### Cause 1: redis-operator stays Processing because a managed instance is paused + +1. List ModuleInfo and the instances the operator manages, and match a Processing `minfo` to a paused instance: + + ```bash + kubectl get moduleinfo -A + kubectl get pods -A | grep redis-operator + kubectl api-resources | grep -i redis + kubectl get -A + ``` + +2. If the instance is meant to stay paused, no action is needed; Processing stays with it. +3. If the instance should run again, resume it through the same control that paused it (for example in the instance's page in the console). After the instance is no longer paused, `minfo` leaves Processing and the functional-component page follows. + +If every managed instance is running and `minfo` is still Processing, this cause does not apply; check the CSV phase (`kubectl get csv -A | grep -i redis`) and the operator Pod status. + +### Cause 2: error re-reading certificate, tls: private key does not match public key + +1. Confirm the log lines and that the components are otherwise Ready: + + ```bash + kubectl get deploy,pods -A | grep -E 'base-operator|prdb' + kubectl logs -n deploy/base-operator --tail=200 + ``` + +2. If the lines match this cause and `base-operator` and `prdb` are healthy, leave them alone. They stop after `tls.key` is written; no recovery operation is required. + +A certificate error that continues after rotation has finished, or a `base-operator` Pod that is not Ready, is not this gap. Collect the steady-state log before changing any certificate files. + +### Cause 3: repeated DisabledCopiedCSVs warning events on katanomi or Knative operator CSVs + +1. Read the events and the OLM setting that produces them: + + ```bash + kubectl -n operators get events --field-selector reason=DisabledCopiedCSVs + kubectl get events -A --field-selector involvedObject.kind=ClusterServiceVersion | grep -iE 'katanomi|knative' + kubectl get olmconfig cluster -o jsonpath='{.spec.features.disableCopiedCSVs}{"\n"}' + ``` + +2. An event that says CSV copying is disabled for that CSV, with `disableCopiedCSVs` set to `true`, is expected. No action is required. + +For what the setting controls and how to change it, see "Controlling Copied CSVs on ACP with the OLMConfig disableCopiedCSVs feature". + +## Diagnostic Steps + +```bash +kubectl get moduleinfo -A +kubectl get csv -A | grep -iE 'redis|katanomi|knative' +kubectl get pods -A | grep -E 'redis-operator|base-operator|prdb' +kubectl get olmconfig cluster -o jsonpath='{.spec.features.disableCopiedCSVs}{"\n"}' +``` + +- Processing `minfo`, a paused managed instance, and the operator Pod `Running`: Cause 1. +- `error re-reading certificate` naming a key/certificate mismatch or missing PEM data, while `base-operator` and `prdb` are healthy and the lines stop after rotation: Cause 2. +- A recurring `DisabledCopiedCSVs` warning on katanomi or Knative operator CSVs with `disableCopiedCSVs` printed as `true`: Cause 3. diff --git a/docs/en/solutions/PVC_expansion_stuck_in_Resizing_because_the_Trident_backend_limitVolumeSize_is_below_the_requested_size_on_ACP.md b/docs/en/solutions/PVC_expansion_stuck_in_Resizing_because_the_Trident_backend_limitVolumeSize_is_below_the_requested_size_on_ACP.md new file mode 100644 index 000000000..ad3850bb9 --- /dev/null +++ b/docs/en/solutions/PVC_expansion_stuck_in_Resizing_because_the_Trident_backend_limitVolumeSize_is_below_the_requested_size_on_ACP.md @@ -0,0 +1,153 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# PVC expansion stuck in Resizing because the Trident backend limitVolumeSize is below the requested size on ACP + +## Issue + +A PersistentVolumeClaim (PVC) on a NAS StorageClass served by NetApp Trident (driver `ontap-nas`) is expanded, for example from `5Gi` to `500Gi`, and the expansion never completes: + +- `spec.resources.requests.storage` of the PVC shows the new size (`500Gi`), but `status.capacity.storage` and the bound PV still show the old size (`5Gi`). +- The PVC carries the condition `type: Resizing` and does not leave it. +- Trying to correct the size in the console is rejected because the new capacity must be larger than the value already in `spec` (for a `500Gi` request, the console asks for at least `501Gi`). The console cannot shrink the request back. +- The Trident controller log shows the resize being refused, for example: + +```text +unsupported capacity range ... requested size: 536870912000 > the size limit: 214748364800 +``` + +The StorageClass already has `allowVolumeExpansion: true`, and there is no LimitRange or ResourceQuota in the namespace that explains the rejection. + +## Root Cause + +Trident backends can cap the size of a single volume with the backend option `limitVolumeSize`. Trident checks every create and resize request against that cap and refuses any request above it. When a backend (for example a `TridentBackendConfig` deployed through the CSI Helm chart) is configured with `limitVolumeSize: 200g`, every PVC on that backend is limited to about 200 GiB. An expansion to `500Gi` (536870912000 bytes) is above the cap (214748364800 bytes), so the storage side never grows the volume. + +Kubernetes has already accepted the larger `spec.resources.requests.storage` at this point. Kubernetes does not allow a PVC request to be lowered, so the PVC stays in `Resizing` with a request it can never reach, and the console only offers sizes above the current request. + +## Resolution + +Decide first which size the volume really needs: + +- **The needed size fits under the backend limit** (for example `100Gi` under a `200g` cap): cancel the failed expansion and expand again within the limit, using option A or option B below. +- **The needed size is above the backend limit**: the cap is a setting of the Trident backend. Changing `limitVolumeSize` is a storage administration decision that the owner of the backend and the NAS must make. After the backend limit has been raised, the pending resize can be retried. Do not edit the Trident controller Deployment. The limit is not a controller argument. + +### Option A: cancel the failed expansion and rebind the PVC to its PV + +This follows the product procedure **Recover From PVC Expansion Failure** and keeps the data on the original volume. + +> **Warning:** This procedure deletes the PVC object. If the PV reclaim policy is still `Delete` when the PVC is deleted, the volume and its data are deleted with it. Do it only after step 1 shows `Retain` on the PV. Stop the workload that mounts the PVC first. Rollback: as long as the PV is `Retain`, the data stays on the PV and can be bound again by a PVC that names it in `volumeName`. + +1. Record the PV, its reclaim policy, and the current sizes, and save both objects: + + ```bash + PV=$(kubectl -n get pvc -o jsonpath='{.spec.volumeName}') + kubectl get pv "$PV" -o jsonpath='{.spec.persistentVolumeReclaimPolicy} {.spec.capacity.storage}{"\n"}' + kubectl -n get pvc -o yaml > /tmp/.yaml + kubectl get pv "$PV" -o yaml > /tmp/"$PV".yaml + kubectl patch pv "$PV" -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}' + ``` + +2. Scale the workload that uses the PVC to zero, then delete the PVC: + + ```bash + kubectl -n scale / --replicas=0 + kubectl -n delete pvc + ``` + +3. Remove the old claim reference so the PV becomes `Available`: + + ```bash + kubectl patch pv "$PV" --type json -p '[{"op":"remove","path":"/spec/claimRef"}]' + kubectl get pv "$PV" + ``` + +4. Recreate the PVC with the same name, StorageClass and access modes, `volumeName` set to the original PV, and a request that the PV can satisfy. The PV still has its old capacity (`5Gi` in the example), so the new request must not be larger than that capacity, or the PVC cannot bind: + + ```yaml + apiVersion: v1 + kind: PersistentVolumeClaim + metadata: + name: + namespace: + spec: + accessModes: + - + storageClassName: + volumeName: + resources: + requests: + storage: + ``` + +5. Restore the original reclaim policy of the PV (the value recorded in step 1): + + ```bash + kubectl patch pv "$PV" -p '{"spec":{"persistentVolumeReclaimPolicy":""}}' + ``` + +6. Expand the PVC again, this time to a size at or below the backend `limitVolumeSize`, and start the workload: + + ```bash + kubectl -n patch pvc -p '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}' + kubectl -n scale / --replicas= + ``` + +### Option B: copy the data to a new PVC + +Use this when you prefer not to touch the PV binding. + +> **Warning:** The workload must be stopped while the data is copied, and the old PVC must not be deleted until the copy is verified. Rollback: keep the old PVC and point the workload back at it. + +1. Stop the workload (scale its Deployment, StatefulSet or ReplicaSet to `0`). +2. Create a new PVC of the needed size (at or below the backend limit) on the same StorageClass. +3. Run a temporary Pod that mounts both PVCs and copy the data, for example with `cp -a` or `rsync -a`. +4. Verify the copy, point the workload at the new PVC (or recreate the old PVC name on the new volume), and start the workload. +5. Delete the old PVC only after the workload runs correctly on the new one. + +## Diagnostic Steps + +1. Compare the requested and the actual size of the PVC. A larger `spec` than `status` together with a `Resizing` condition means that the resize was accepted by Kubernetes but not done by the storage: + + ```bash + kubectl -n get pvc \ + -o jsonpath='spec={.spec.resources.requests.storage} status={.status.capacity.storage}{"\n"}{.status.conditions}{"\n"}' + kubectl get pv -o jsonpath='{.spec.capacity.storage}{"\n"}' + ``` + + Expected for this issue: `spec=500Gi status=5Gi`, a `Resizing` condition, and a PV that still shows `5Gi`. + +2. Read the PVC events. An expansion that is only slow keeps showing progress. This issue shows a failed resize: + + ```bash + kubectl -n describe pvc + ``` + +3. Find the rejection in the Trident controller log: + + ```bash + kubectl -n trident get pods + kubectl -n trident logs --all-containers --tail=500 | grep -iE 'size limit|unsupported capacity range' + ``` + + Expected: `requested size: > the size limit: `. Divide the byte values by 1073741824 to get GiB. + +4. Read the limit on the backend: + + ```bash + kubectl -n trident get tridentbackendconfig + kubectl -n trident get tridentbackendconfig -o jsonpath='{.spec.limitVolumeSize}{"\n"}' + ``` + + Expected: a value such as `200g` that is smaller than the requested size. When the backend was deployed with Helm, the same value is in the release values (`helm -n trident get values `), and that is where it has to be changed so that the next upgrade does not reset it. + +5. Rule out look-alikes. When the StorageClass has no `allowVolumeExpansion: true`, the expansion is refused right away instead of hanging in `Resizing`. A namespace LimitRange or ResourceQuota rejects the PVC edit itself, with an admission error: + + ```bash + kubectl get storageclass -o jsonpath='{.allowVolumeExpansion}{"\n"}' + kubectl -n get limitrange,resourcequota + ``` diff --git a/docs/en/solutions/PXC_slow_query_ClickHouse_restarts_fills_its_disk_or_its_LoadBalancer_blocks_an_upgrade_on_ACP.md b/docs/en/solutions/PXC_slow_query_ClickHouse_restarts_fills_its_disk_or_its_LoadBalancer_blocks_an_upgrade_on_ACP.md new file mode 100644 index 000000000..9c026416a --- /dev/null +++ b/docs/en/solutions/PXC_slow_query_ClickHouse_restarts_fills_its_disk_or_its_LoadBalancer_blocks_an_upgrade_on_ACP.md @@ -0,0 +1,219 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Application Services + - Alauda Container Platform +ProductsVersion: + - "3.10" + - "3.14" + - "4.0" + - "4.1" +--- + +# PXC slow-query ClickHouse restarts, fills its disk, or its LoadBalancer blocks an upgrade on ACP + +## Issue + +The ClickHouse that stores MySQL PXC slow-query logs (Pods `chi-pxc-ck-slowsql-*` or `chi-pxc-ckslowsql`, Service `clickhouse-pxc-ck`, usually in `cpaas-system`) shows one of these: + +- The slow-query component restarts again and again; events on `chi-pxc-ck-slowsql-cluster` or `chi-pxc-ckslowsql` show the container exiting and coming back (Cause 1). +- Slow query has to be turned off and `chi-pxc-ckslowsql` removed on ACP 3.10 (Cause 2). +- Before a platform upgrade, the pre-check stops on a MetalLB Service: `clickhouse-pxc-ck` in `cpaas-system` is of type `LoadBalancer` (Cause 3). +- The ClickHouse container of `chi-pxc-ck-slowsql-cluster-0-0-0` sits at about 80 to 90 percent memory with no Pod events (Cause 4). +- The `pxc-ck-slowsql-cluster` Pod takes a lot of storage (for example 60G), most of it in `value.bin` files (Cause 5). + +`chi-pxc-ck-slowsql-cluster` collects slow-query logs for MySQL PXC instances. Raising memory, uninstalling the feature, and changing the Service type solve different problems; deleting the PVC belongs only to the uninstall. + +A ClickHouse instance that a user created in an application namespace (Pods named `chi-----0`) is a different workload. The 2Gi and 4Gi values below come from the slow-query component and do not size such an instance; the general way to change ClickHouse operator memory is in the Cause 1 resolution, steps 3 and 4. + +## Root Cause + +### Cause 1: chi-pxc-ck-slowsql-cluster is OOMKilled with a 2Gi memory limit + +The container's last termination reason is `OOMKilled`. The memory limit of the slow-query ClickHouse container was 2Gi (also shown as 2G). The process is killed whenever it crosses that limit, so the Pod keeps restarting. Raising the limit to 4Gi stops the restarts. If slow query is not used, the component can be removed instead (Cause 2). + +### Cause 2: Slow query has to be uninstalled on ACP 3.10 + +This is a removal procedure, not a crash. On 3.10, uninstalling the ClickHouse operator also deletes its `csv` and `subs`. The instance is then stopped by setting the ClickHouse replica count to 0 and deleting the matching PVC. + +### Cause 3: Service clickhouse-pxc-ck is of type LoadBalancer and the upgrade pre-check fails + +The pre-upgrade checklist reports a MetalLB LoadBalancer for Service `clickhouse-pxc-ck` in `cpaas-system`. The slow-query ClickHouse Service does not need an external address. Changing the Service type to `ClusterIP` lets the pre-check continue. + +### Cause 4: Memory stays near 80 to 90 percent and no data service is in use + +`chi-pxc-ck-slowsql-cluster-0-0-0` shows ClickHouse memory around 80 to 90 percent, with no Pod events, so this is not the OOM restart of Cause 1. On ACP 4.1.5, a platform with no Data Services page has no PXC or MGR instances to manage, so the Pod is not serving any database. No further reason for the memory level is known. Scaling the workload to 0 removes the memory use from the node. This is not the 3.10 uninstall: no operator CSV is removed and the PVC is kept. + +### Cause 5: Storage keeps growing, mostly value.bin files + +`value.bin` is a ClickHouse table data file (a column file of a table part); here it holds the historical slow-query data. Before ACP 3.14, the ClickHouse used by PXC could not be configured to clean up slow-query and ClickHouse logs periodically, so storage use only grows. On 3.14, the fix is to adjust the data retention time of the ClickHouse used by PXC, as described in the platform documentation of that name for your version. + +## Resolution + +### Cause 1: chi-pxc-ck-slowsql-cluster is OOMKilled with a 2Gi memory limit + +1. Confirm the last termination reason is `OOMKilled` (exit code `137`) and that the limit is 2Gi. Note the container name that owns the limit. + +```bash +kubectl get pod -n \ + -o jsonpath='{range .status.containerStatuses[*]}{.name}{" exit="}{.lastState.terminated.exitCode}{" reason="}{.lastState.terminated.reason}{"\n"}{end}' +kubectl get pod -n \ + -o jsonpath='{range .spec.containers[*]}{.name}{" limit="}{.resources.limits.memory}{"\n"}{end}' +``` + +2. Set that container's memory limit to 4Gi on the object that controls the Pod. When the ClickHouse operator owns the StatefulSet, a change made only on the StatefulSet is reverted, so change the operator's resource (steps 3 and 4). Edit the StatefulSet directly only when nothing owns it (step 5). + +3. Find out which object owns the StatefulSet. Pods and StatefulSets named `chi-...` are normally generated by the ClickHouse operator from a `ClickHouseInstallation` (short name `chi`). + +```bash +kubectl get sts -n \ + -o jsonpath='{range .metadata.ownerReferences[*]}{.kind}{" "}{.name}{"\n"}{end}' +kubectl get chi -A +``` + +4. If the owner is a `ClickHouseInstallation`, edit the memory of the container in its pod template. Use the template name and container name the resource already has, and change nothing else. + +> **Warning:** Saving the `ClickHouseInstallation` makes the operator roll the ClickHouse Pod, and slow-query reads fail until it is Ready again; MySQL itself keeps running. Do this only when `OOMKilled` is confirmed on the ClickHouse container, and only when the node has room for the new size, otherwise the Pod stays Pending. Save the resource first (`kubectl get chi -n -o yaml > chi-backup.yaml`). Roll back by setting the same fields to their previous values. + +```bash +kubectl -n edit chi +``` + +```yaml +spec: + templates: + podTemplates: + - name: + spec: + containers: + - name: + resources: + limits: + memory: 4Gi +``` + +Optionally, set `spec.configuration.settings.max_server_memory_usage` (in bytes) below the container memory limit, with the memory request and limit at the same value. That caps ClickHouse's own memory use under the limit; it has fixed OOM on user ClickHouse instances, and the slow-query 2Gi-to-4Gi fix did not need it. + +5. If the StatefulSet has no `ClickHouseInstallation` owner, set the limit on the StatefulSet. + +> **Warning:** Changing the limit rolls the slow-query Pod and drops in-flight queries to it; MySQL itself keeps running. Only when `OOMKilled` with a 2Gi limit is confirmed and the node has 4Gi free. Do not scale the StatefulSet to 0 or delete its PVC for an OOM. Roll back by setting the limit to 2Gi again. + +```bash +kubectl set resources -n sts/ -c --limits=memory=4Gi +``` + +### Cause 2: Slow query has to be uninstalled on ACP 3.10 + +1. Find the ClickHouse operator objects that back `chi-pxc-ckslowsql` or `pxc-ck`, then uninstall the operator from the operator catalog. On 3.10 the uninstall deletes the operator `csv` and `subs`. + +```bash +kubectl get csv,sub -A | grep -Ei 'clickhouse|pxc-ck|ckslow' +``` + +2. Read the slow-query PVC name from the Pod while it still exists. MySQL member PVCs use the instance name and `pxc-`; the slow-query PVC name contains `ck` or `slowsql`. + +```bash +kubectl get pod -n \ + -o jsonpath='{range .spec.volumes[*]}{.persistentVolumeClaim.claimName}{"\n"}{end}' +``` + +3. Set the ClickHouse replica count to 0 and delete that PVC. + +> **Warning:** Replica count 0 stops slow-query collection, and deleting the PVC permanently deletes the stored slow-query logs. Do this only when the goal is to remove the feature, not as an OOM fix. There is no rollback for the deleted PVC; a later install starts empty. + +```bash +kubectl scale -n sts/ --replicas=0 +kubectl delete pvc -n +``` + +### Cause 3: Service clickhouse-pxc-ck is of type LoadBalancer and the upgrade pre-check fails + +> **Warning:** Changing `clickhouse-pxc-ck` from `LoadBalancer` to `ClusterIP` removes the external address MetalLB published for it, and clients outside the cluster that used it will fail. Do it only when the upgrade pre-check names this Service and nothing outside the cluster needs the slow-query address. Roll back by re-applying the saved YAML. + +```bash +kubectl get svc clickhouse-pxc-ck -n cpaas-system -o yaml > clickhouse-svc-backup.yaml +kubectl patch svc clickhouse-pxc-ck -n cpaas-system --type merge -p '{"spec":{"type":"ClusterIP"}}' +kubectl get svc clickhouse-pxc-ck -n cpaas-system -o jsonpath='{.spec.type}{"\n"}' +kubectl get endpoints clickhouse-pxc-ck -n cpaas-system -o wide +``` + +`spec.type` should be `ClusterIP`, and the endpoints should still list the slow-query Pod addresses if the Pod is running. Then rerun the pre-check. + +### Cause 4: Memory stays near 80 to 90 percent and no data service is in use + +1. Confirm the Pod is the slow-query store, memory is high without `OOMKilled`, and no PXC or MGR instance needs those logs: + +```bash +kubectl get pods -A | grep -E 'chi-pxc-ck-slowsql|pxc-|mysql-' +kubectl get mysql -A +kubectl top pod -n chi-pxc-ck-slowsql-cluster-0-0-0 +kubectl get pod -n chi-pxc-ck-slowsql-cluster-0-0-0 \ + -o jsonpath='{.metadata.ownerReferences[*].kind}{" "}{.metadata.ownerReferences[*].name}{"\n"}' +``` + +2. Scale the owning workload to 0. + +> **Warning:** Replica count 0 stops PXC slow-query collection. Do it only when memory stays high on `chi-pxc-ck-slowsql-cluster` and no data service, PXC, or MGR instance is in use. The PVC is kept. Scale back to the previous replica count to undo it. If a controller creates the Pod again, set the replica count to 0 on that owner object instead. + +```bash +kubectl scale -n sts/ --replicas=0 +``` + +### Cause 5: Storage keeps growing, mostly value.bin files + +1. Confirm which tables hold the space. Add `--user` and `--password` to `clickhouse-client` if the server requires them. + +```bash +kubectl exec -n chi-pxc-ck-slowsql-cluster-0-0-0 -- du -sh /var/lib/clickhouse +kubectl exec -n chi-pxc-ck-slowsql-cluster-0-0-0 -- clickhouse-client --query \ + "SELECT database, table, formatReadableSize(sum(bytes_on_disk)) AS size, sum(rows) AS rows FROM system.parts WHERE active GROUP BY database, table ORDER BY sum(bytes_on_disk) DESC LIMIT 20" +``` + +2. Adjust the retention time of the slow-query data as the platform documentation on the data retention time of the ClickHouse used by PXC describes for your version. In ClickHouse, a table retention period is a TTL; read the current definition first to see the time column and any existing TTL. + +```bash +kubectl exec -n chi-pxc-ck-slowsql-cluster-0-0-0 -- clickhouse-client --query "SHOW CREATE TABLE .
" +``` + +> **Warning:** A shorter TTL makes ClickHouse delete slow-query rows older than the new period during background merges; deleted history cannot be recovered. Only when the space is confirmed to be in the slow-query tables listed in step 1 and older history is no longer needed. Rolling back to the previous TTL only keeps future data longer. MySQL PXC is not affected. + +```bash +kubectl exec -n chi-pxc-ck-slowsql-cluster-0-0-0 -- clickhouse-client --query \ + "ALTER TABLE .
MODIFY TTL + INTERVAL DAY" +``` + +3. Repeat step 1 over the following days. Space is released as old parts are merged away, not at the moment of the change. + +## Diagnostic Steps + +1. Separate an OOM from a pre-check finding. OOM is a restarting Pod; the pre-check finding is a Service type, even when the Pod is healthy. + +```bash +kubectl get pods -n cpaas-system | grep -E 'pxc-ck|ckslow' +kubectl get svc clickhouse-pxc-ck -n cpaas-system +kubectl describe pod -n | grep -E 'OOMKilled|Limits|Restart' +``` + +2. Read the limit actually enforced. Cause 1 applies when the OOM-killed container shows `2Gi` or `2G`. With any other limit, 4Gi is not the fix; size from actual usage and free node memory. + +3. Before an uninstall, list the operator subscription and the PVCs so you delete the slow-query volume and not a MySQL PXC volume: + +```bash +kubectl get csv,sub,pvc -n cpaas-system | grep -Ei 'clickhouse|pxc-ck|slowsql' +``` + +High memory without `OOMKilled` while `kubectl get mysql -A` is empty is Cause 4: scale to 0, do not delete the PVC. + +4. If the OOM-killed ClickHouse Pod runs in an application namespace and is not the slow-query store, the slow-query causes do not apply. Confirm `OOMKilled` or exit code `137` (a container whose last state is `Error` without `OOMKilled` was not killed for memory, and more memory is not its fix), then raise the memory above the current limit on its `ClickHouseInstallation` as in the Cause 1 resolution, steps 3 and 4. Do not scale it to 0 or delete its PVC for an OOM. + +```bash +kubectl get pod -n \ + -o jsonpath='{range .status.containerStatuses[*]}{.name}{" reason="}{.lastState.terminated.reason}{" exit="}{.lastState.terminated.exitCode}{"\n"}{end}' +``` + +5. Disk growth without OOM or high memory, with `value.bin` files taking the space, is Cause 5: + +```bash +kubectl exec -n chi-pxc-ck-slowsql-cluster-0-0-0 -- sh -c 'find /var/lib/clickhouse -name value.bin -size +100M | head' +``` diff --git a/docs/en/solutions/Pipeline_component_status_Unknown_because_tekton-operator-webhook-certs_holds_a_future-dated_certificate_on_ACP.md b/docs/en/solutions/Pipeline_component_status_Unknown_because_tekton-operator-webhook-certs_holds_a_future-dated_certificate_on_ACP.md new file mode 100644 index 000000000..13f4a4df9 --- /dev/null +++ b/docs/en/solutions/Pipeline_component_status_Unknown_because_tekton-operator-webhook-certs_holds_a_future-dated_certificate_on_ACP.md @@ -0,0 +1,95 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: [] +--- + +# Pipeline component status Unknown because tekton-operator-webhook-certs holds a future-dated certificate on ACP + +## Issue + +On a business cluster, the cluster component **Pipeline** (the Tekton operator) shows status Unknown, for example during a routine inspection of cluster components. + +Decoding the certificates stored in the Secret `tekton-operator-webhook-certs` in the `tekton-operator` namespace shows a validity period that starts in the future: + +```text +notBefore=Oct 29 19:00:14 2027 GMT +notAfter=Nov 5 19:00:14 2027 GMT +``` + +## Root Cause + +### Cause 1: the webhook certificate was issued while the cluster clock was wrong + +The Tekton operator keeps the serving certificate of its webhook in the Secret `tekton-operator-webhook-certs`. If the cluster clock was set ahead at some point when the certificate was generated, the certificate carries a `notBefore` date in the future. + +A TLS client rejects a certificate whose `notBefore` date has not been reached yet. Even after the clock is correct again, the stored certificate stays not yet valid until that date, and the Pipeline component status is Unknown. Replacing the Secret with an empty one and restarting the operator makes the operator generate a new certificate with the current time. + +## Resolution + +### Cause 1 + +1. Check that the clock on every node of the cluster is correct before regenerating the certificate. A certificate regenerated on a wrong clock has the same problem. + + ```bash + date -u + chronyc tracking + ``` + + Run these on each node. If the time is wrong, fix time synchronization first (see `Configuring_chrony_NTP_on_Cluster_Nodes.md`). + +2. Save the current Secret: + + ```bash + kubectl -n tekton-operator get secret tekton-operator-webhook-certs -o yaml > tekton-operator-webhook-certs-backup.yaml + ``` + +3. Replace the Secret with an empty Secret of the same name. + + > **Warning:** Until the operator writes a new certificate, the Tekton operator webhook cannot serve TLS, and changes to Tekton operator resources (such as `TektonConfig`) can fail admission. Do this only when the certificate in the Secret has a `notBefore` date in the future and the node clocks are correct. Roll back by applying `tekton-operator-webhook-certs-backup.yaml` (remove `metadata.resourceVersion` and `metadata.uid` first) and restarting the operator again. + + ```bash + kubectl -n tekton-operator delete secret tekton-operator-webhook-certs + kubectl -n tekton-operator create secret generic tekton-operator-webhook-certs + ``` + +4. Restart the Tekton operator so that it generates a new certificate into the empty Secret: + + > **Warning:** The operator Pods restart. Pipelines that are already running are not stopped, but operator reconciliation pauses until the new Pods are Ready. Roll back as in step 3. + + ```bash + kubectl -n tekton-operator get deployment + kubectl -n tekton-operator rollout restart deployment + kubectl -n tekton-operator rollout status deployment + ``` + +5. Check the certificate dates again (see Diagnostic Steps) and confirm that the Pipeline component status returns to normal. + +## Diagnostic Steps + +1. List the keys of the Secret and print the validity of each certificate: + + ```bash + kubectl -n tekton-operator get secret tekton-operator-webhook-certs -o jsonpath='{.data}' | jq -r 'keys[]' + kubectl -n tekton-operator get secret tekton-operator-webhook-certs \ + -o jsonpath='{.data.}' | base64 -d | openssl x509 -noout -dates + ``` + + Expected for this cause: `notBefore` is later than the current time (`date -u`). A key that holds a private key instead of a certificate fails with `unable to load certificate ... Expecting: TRUSTED CERTIFICATE`; that is expected for that key and is not the fault. + +2. Compare with the current time on the nodes: + + ```bash + date -u + ``` + + If `notBefore` is in the past and `notAfter` is in the future, the certificate is valid and the Unknown status has another cause. If `notAfter` is in the past, the certificate has expired, which is a different problem from a future-dated certificate. + +3. Check the operator Pods and their logs for TLS errors: + + ```bash + kubectl -n tekton-operator get pods + kubectl -n tekton-operator logs deployment/ --tail=200 | grep -iE 'x509|certificate|tls' + ``` diff --git a/docs/en/solutions/Pipeline_deploy_task_fails_with_x509_certificate_signed_by_unknown_authority_from_the_Connectors_CSI_kubeconfig.md b/docs/en/solutions/Pipeline_deploy_task_fails_with_x509_certificate_signed_by_unknown_authority_from_the_Connectors_CSI_kubeconfig.md new file mode 100644 index 000000000..f024a3012 --- /dev/null +++ b/docs/en/solutions/Pipeline_deploy_task_fails_with_x509_certificate_signed_by_unknown_authority_from_the_Connectors_CSI_kubeconfig.md @@ -0,0 +1,124 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: [] +--- + +# Pipeline deploy task fails with x509 certificate signed by unknown authority from the Connectors CSI kubeconfig + +## Issue + +A Tekton pipeline task that updates a workload with `kubectl apply` (for example a `workload-apply` task that renders a Deployment and applies it) fails in its `apply` step. The task uses the kubeconfig mounted into the workspace: + +```text +==> Using kubeconfig from workspace: /workspace/kubeconfig/kubeconfig +error: error validating "/shared/rendered.yaml": error validating data: failed to download openapi: Get "https:///kubernetes//openapi/v2?timeout=32s": tls: failed to verify certificate: x509: certificate signed by unknown authority; if you choose to ignore these errors, turn validation off with --validate=false +==> Apply failed, continue to finalize flow +``` + +The following steps then fail and the PipelineRun ends as failed. The same pipeline worked before, and the platform address itself is reachable with a trusted certificate from other clients. + +The same step can instead fail with an expired certificate on the same platform path, while the browser still shows a valid certificate for the platform address: + +```text +error: error validating "/shared/rendered.yaml": error validating data: failed to download openapi: Get "https:///kubernetes//openapi/v2?timeout=32s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time is after ; if you choose to ignore these errors, turn validation off with --validate=false +``` + +## Root Cause + +### Cause 1: the Connectors CSI component keeps a certificate it loaded at startup + +When a pipeline runs, its Pod mounts a CSI volume provided by Connectors (driver `connectors.alauda.io`). At mount time the Connectors CSI component uses a CA to sign a certificate and injects it into the Pod, together with the kubeconfig the task uses. + +In Connectors versions before 1.14.0, the CSI component loads this certificate material only once, when it starts. The certificate is valid for 60 days and is rotated upstream. A CSI component that has been running longer than that keeps injecting the stale certificate, so the kubeconfig in the pipeline Pod no longer matches and the TLS check fails with `x509: certificate signed by unknown authority`. + +Connectors 1.14.0 fixes this: the certificate is rotated automatically upstream, and the CSI component detects the change and reloads it. + +### Cause 2: the certificate used by Connectors expired on connectors-operator v1.12.0 + +With `connectors-operator.v1.12.0` installed in namespace `connectors-operator`, the certificate that the pipeline's request runs into belongs to Connectors. On Connectors Pods that have been running for a long time this certificate can expire, and then `kubectl` validation through the platform `/kubernetes//` path fails with `x509: certificate has expired or is not yet valid`. Restarting the Connectors Pods generates a new certificate, valid for 60 days, so the restart is a temporary fix. The permanent fix is to upgrade `connectors-operator` from v1.12.0 to v1.14. + +## Resolution + +### Cause 1 + +**Workaround:** restart all Connectors components so that the CSI component loads the current certificate. + +1. List the Connectors workloads: + + ```bash + kubectl get deployments,daemonsets -A | grep -i connectors + ``` + +2. Restart each of them: + + > **Warning:** While the Connectors Pods restart, pipeline Pods that start at that moment cannot mount their Connectors volumes and stay in the init phase or fail; Pods that are already running keep their mounts. Do this only when the error is `x509: certificate signed by unknown authority` from the workspace kubeconfig, and preferably when no pipelines are starting. A restart has nothing to roll back; re-run any PipelineRun that failed during the restart. + + ```bash + kubectl -n rollout restart deployment/ + kubectl -n rollout restart daemonset/ + kubectl -n rollout status deployment/ + kubectl -n rollout status daemonset/ + ``` + +3. Re-run the PipelineRun and confirm that the `apply` step succeeds. + +**Permanent fix:** upgrade Connectors to 1.14.0 or later, where the CSI component refreshes the certificate when it changes. Without the upgrade, the error comes back each time the certificate is rotated while the CSI component keeps running. + +### Cause 2 + +1. Confirm the installed version and the Pods: + + ```bash + kubectl get csv -n connectors-operator | grep connectors-operator + kubectl get pod -n connectors-operator + ``` + +2. Restart the Connectors Pods: + + > **Warning:** Deleting the Connectors Pods interrupts the Connectors API, CSI, proxy, and the git, harbor, and oci plugins until the replacements are Running; pipeline runs that start in the gap can fail. Do this only when the pipeline error is `x509: certificate has expired or is not yet valid` on the platform `/kubernetes//` path, the platform certificate in the browser is valid, and `connectors-operator.v1.12.0` is installed. The workloads recreate the Pods; there is nothing to roll back. + + ```bash + for p in api controller-manager csi git-plugin harbor-plugin oci-plugin proxy; do + kubectl get pod -n connectors-operator -o name | grep "connectors-${p}" | xargs -r kubectl delete -n connectors-operator + done + kubectl get pod -n connectors-operator + ``` + + The `operator-connectors-operator` and `operator-connectors-ui-plugin` Pods do not need to be restarted. + +3. Re-run the PipelineRun. + +**Permanent fix:** upgrade `connectors-operator` to v1.14 through OperatorHub, following the upgrade procedure for that operator. Until then, repeat the restart before the 60-day certificate expires. + +Do not use `--validate=false` as a fix. It only skips the OpenAPI schema download; the apply itself talks to the same API endpoint with the same kubeconfig and fails TLS verification in the same way. + +## Diagnostic Steps + +1. Confirm that the failing step uses the mounted kubeconfig: the log shows `Using kubeconfig from workspace: /workspace/kubeconfig/kubeconfig` before the x509 error. + +2. Check the Connectors version: + + ```bash + kubectl get csv -A | grep -i connectors + ``` + + A version earlier than 1.14.0 is affected. + +3. Check how long the Connectors Pods have been running: + + ```bash + kubectl get pods -A -o wide | grep -i connectors + ``` + + Pods older than about 60 days (the certificate validity) make this cause likely. After the restart, the next pipeline run succeeding confirms it. + +4. To rule out a platform certificate problem, check the platform address from outside the pipeline, for example in a browser or with `curl -v https:///`. A trusted certificate there, together with the failure only inside pipeline Pods, points to the injected kubeconfig and not to the platform endpoint. + +5. Match the error text to the cause. `x509: certificate signed by unknown authority` is Cause 1. `x509: certificate has expired or is not yet valid` on the platform `/kubernetes//` path with `connectors-operator.v1.12.0` installed is Cause 2. If the URL in the error is the API server directly (`https://
:6443/openapi/v2`) instead of the platform path, the request does not go through Connectors and neither cause applies; print the certificate that address serves: + + ```bash + echo | openssl s_client -connect
:6443 2>/dev/null | openssl x509 -noout -subject -issuer -dates + ``` diff --git a/docs/en/solutions/Pipeline_takes_more_than_40_minutes_because_the_Maven_stage_runs_mvn_-U.md b/docs/en/solutions/Pipeline_takes_more_than_40_minutes_because_the_Maven_stage_runs_mvn_-U.md new file mode 100644 index 000000000..9633b35a6 --- /dev/null +++ b/docs/en/solutions/Pipeline_takes_more_than_40_minutes_because_the_Maven_stage_runs_mvn_-U.md @@ -0,0 +1,48 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.3" +--- + +# Pipeline takes more than 40 minutes because the Maven stage runs mvn -U + +## Issue + +A pipeline needs more than 40 minutes to finish, and releases slip because of the wait. The run does complete: it is not stuck in Pending, and it does not fail with `index-pack` or `OOMKilled`. Per-stage timing shows that the Maven stage is the long part, and the Maven command in that stage includes `-U` (for example `mvn -U clean package`). + +## Root Cause + +### Cause 1: the Maven stage forces dependency updates with `-U` + +`-U` (`--update-snapshots`) makes Maven check the remote repositories for updated snapshots and for missing releases on every run. When the Maven stage is the stage that dominates the run and its command contains `-U`, that forced update check is the extra time in the stage, and removing it is the change to make. Removing `-U` is the optimization direction that matches the stage timing; measure the result on your own pipeline. + +## Resolution + +### Cause 1: the Maven stage forces dependency updates with `-U` + +1. Time the stages before changing the command. Continue only when Maven is the long stage and the command contains `-U`: + + ```bash + kubectl get pipelinerun -n + kubectl logs -n | grep -E 'mvn |BUILD ' + ``` + +2. Remove `-U` from the Maven invocation in the stage script, so that the stage uses the dependencies already resolved instead of forcing an update on every run. + + > **Warning:** Without `-U`, a newly published snapshot is not picked up until the local repository is refreshed or `-U` is added back for one run. Precondition: only when stage timing shows the Maven stage takes tens of minutes and the command includes `-U`. Rollback: restore `-U` on that Maven step. Leave the Maven cache volume in place. + +3. Compare the duration of the next run's Maven stage with the previous run. If Maven is still the long stage after `-U` is gone, the force update was not the only cost. Measure the remaining Maven log (download and compile phases) instead of changing clone memory or Pod scheduling. + +## Diagnostic Steps + +```bash +kubectl get pipelinerun -n +kubectl logs -n | grep -E 'mvn |BUILD |Downloading|Downloaded' +``` + +- A completed run whose Maven stage dominates the total time, with `mvn -U` in the command and many `Downloading` lines for metadata on every run: Cause 1. +- A run that never starts is a scheduling problem (Pod Pending), not Maven. +- A Node.js task killed for memory, or a clone failing in `index-pack`, is not related to the Maven `-U` flag. diff --git a/docs/en/solutions/Pipeline_task_Pod_does_not_start_connector-csi_OOMKilled_controller_image_pull_Connector_placeholder_or_PodSecurity.md b/docs/en/solutions/Pipeline_task_Pod_does_not_start_connector-csi_OOMKilled_controller_image_pull_Connector_placeholder_or_PodSecurity.md new file mode 100644 index 000000000..f4ca938f2 --- /dev/null +++ b/docs/en/solutions/Pipeline_task_Pod_does_not_start_connector-csi_OOMKilled_controller_image_pull_Connector_placeholder_or_PodSecurity.md @@ -0,0 +1,167 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.2.0" + - "4.3" + - "4.3.2" +--- + +# Pipeline task Pod does not start: connector-csi OOMKilled, controller image pull, Connector placeholder, or PodSecurity + +## Issue + +A DevOps v4 (Tekton) pipeline is created or run, and a task Pod does not start. One of these symptoms is shown: + +- The gitclone Pod stays unstarted while its volume is being mounted. +- The continuous-build API returns HTTP 500, and `katanomi-controller` cannot pull its own image. +- A buildah Pod stays in `Init:0/3`, and kubelet repeats: + + ```text + Warning FailedMount ... kubelet MountVolume.SetUp failed for volume "ws-" : rpc error: code = Unknown desc = failed to get Connector /{selectedConnectors()}: Connector.connectors.alauda.io "{selectedConnectors()}" not found + ``` + +- The TaskRun Pod is rejected at admission: + + ```text + pods "" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (containers "prepare", "place-scripts", "step-clone" ...) + ``` + + followed by the `capabilities.drop`, `runAsNonRoot`, and `seccompProfile` requirements. Namespaces are created with `pod-security.kubernetes.io/enforce: restricted` by default, and relaxing an application namespace to `privileged` is not acceptable because it places no constraint on user deployments. + +A clone that starts and then fails with `fatal: could not read Username` is a missing Git credential, not a mount failure. A step that fails inside Maven, the JDK, or Node.js after the Pod is Running is a build-environment problem. + +## Root Cause + +### Cause 1: gitclone Pod does not start because `connector-csi` is `OOMKilled` + +The gitclone Pod waits for a volume provided by the `connector-csi` component. The `connector-csi` Pod log shows a timeout fetching a ConfigMap, which looks like an API problem, but the Pod's container status shows that it was `OOMKilled` and restarted. The mount handler does not stay alive long enough to finish the volume setup, so the gitclone Pod never leaves the mount phase. Raising the `connector-csi` memory limit lets the mount complete. A ConfigMap RBAC error without `OOMKilled` is not this cause. + +### Cause 2: continuous build returns HTTP 500 because `katanomi-controller` cannot pull its image + +The controller Pod is not Ready because its image pull fails, so the API it serves returns HTTP 500. Once the pull can succeed, recreating the Pod clears the 500. A 500 while the controller Pod is Running and its image is pulled is a different fault. + +### Cause 3: the buildah Pod volume carries the `{selectedConnectors()}` placeholder + +A pipeline created in the UI uses the buildah task to build and push to Harbor. The Pod's CSI volume (driver `connectors.alauda.io`, attributes `configuration.names: registry-config`, `connectors: '{selectedConnectors()}'`) still carries the placeholder instead of the Connector selected in the UI, so the CSI driver looks up a Connector that does not exist. This is a known defect in Alauda DevOps Pipelines `tektoncd-operator.v4.8.0` (seen with ACP 4.2.0 and `connectors-operator.v1.9.0`), fixed in Alauda DevOps Pipelines 4.10. + +### Cause 4: the namespace enforces `restricted` and the Tekton containers do not meet it + +The namespace where the pipeline runs enforces the `restricted` Pod Security Standard, and the Tekton init containers (`prepare`, `place-scripts`) and step containers (`step-*`) do not set the required securityContext fields. The Pod is rejected at admission and never exists. + +The Tekton `set-security-context` feature flag is not sufficient on its own. In upstream Tekton the flag only gives a restricted-compatible securityContext to the containers that Tekton injects (init containers such as `prepare` and `place-scripts`, and the affinity assistant). `step-*` containers keep the securityContext that the Task defines, and a BuildKit step that requests `privileged: true` is still rejected by `restricted`. BuildKit build tasks require privileged containers; with privilege turned off the build fails with `operation not permitted`. + +## Resolution + +### Cause 1: gitclone Pod does not start because `connector-csi` is `OOMKilled` + +1. Confirm `OOMKilled` on `connector-csi` and find its owner: + + ```bash + kubectl get pods -A | grep connector-csi + kubectl -n get pod -o jsonpath='{.metadata.ownerReferences[0].kind}{" "}{.metadata.ownerReferences[0].name}{"\n"}{.status.containerStatuses[*].name}{" "}{.status.containerStatuses[*].lastState.terminated.reason}{"\n"}' + ``` + + `OOMKilled` in the output is required before the change. Note the owner kind, its name, and the container name, and record the current limits. + +2. Raise that container's memory limit on the owning workload. Doubling the current limit is a starting point. Keep the request less than or equal to the new limit. + + > **Warning:** `kubectl set resources` rolls the `connector-csi` Pods. New gitclone mounts fail until the new Pod is Ready; task Pods that already mounted are not affected. Precondition: only when the last terminated reason is `OOMKilled`. Rollback: set the memory limit back to the recorded value with the same command. If a controller reverts the change, the limit is owned by an operator resource or plugin configuration; set it there. + + ```bash + kubectl -n set resources / -c --limits=memory= --requests=memory= + kubectl -n rollout status / + ``` + + For example, when the old limit was `256Mi`: `--limits=memory=512Mi --requests=memory=256Mi`. + +3. Watch the new Pod. Another `OOMKilled` means the limit is still too low; raise it again. The gitclone Pod passes the mount and starts its container. + +### Cause 2: continuous build returns HTTP 500 because `katanomi-controller` cannot pull its image + +1. Find the controller Pod and read why the pull failed: + + ```bash + kubectl get pods -A | grep katanomi-controller + kubectl -n describe pod | sed -n '/Events/,$p' + ``` + + `ImagePullBackOff` or `ErrImagePull` on `katanomi-controller`, together with HTTP 500 from continuous build, is this cause. + +2. Fix the reason for the pull failure first (registry reachability, scheme, or pull rate), or the replacement Pod hits the same error. + +3. Recreate the Pod. + + > **Warning:** Deleting the `katanomi-controller` Pod stops continuous-build reconciliation until the replacement is Ready. Precondition: only when the Pod event is an image-pull failure and the registry path is restored. Rollback: none needed; the owner recreates the Pod from an unchanged spec. + + ```bash + kubectl -n delete pod + kubectl -n rollout status deploy/katanomi-controller + ``` + + If the owner is not a Deployment of that name, wait on the owner from `ownerReferences` instead. The new Pod is Running and continuous-build calls no longer return 500. + +### Cause 3: the buildah Pod volume carries the `{selectedConnectors()}` placeholder + +1. Confirm the placeholder in the Pod spec and record the operator versions: + + ```bash + kubectl -n describe pod | sed -n '/Events/,$p' + kubectl -n get pod -o yaml | grep -n 'selectedConnectors' + kubectl get csv -A | grep -Ei 'tektoncd-operator|connectors-operator' + ``` + + The literal `{selectedConnectors()}` in the volume attributes, with `tektoncd-operator` older than 4.10, is this cause. + +2. Download the Alauda DevOps Pipelines 4.10 package from the Alauda application marketplace (for example `tektoncd-operator.latest.amd64.v4.10.0.tgz`, listed as compatible with ACP 4.0 to 4.3), publish it to the cluster, and upgrade. + + > **Warning:** Upgrading Alauda DevOps Pipelines rolls the Tekton controllers and webhooks; PipelineRuns that start during the rollout can fail admission or wait. Precondition: only when the FailedMount names `{selectedConnectors()}` and the installed `tektoncd-operator` is older than 4.10. Upgrade in a quiet window and note the current CSV. Treat the upgrade as one-way: going back requires reinstalling the previous version. + +3. Run the pipeline again. The buildah Pod leaves `Init`, and the CSI volume names the real Connector. + +### Cause 4: the namespace enforces `restricted` and the Tekton containers do not meet it + +1. Read the namespace labels and the containers named in the admission error: + + ```bash + kubectl get ns --show-labels | tr ',' '\n' | grep pod-security + kubectl get taskrun -n -o jsonpath='{.status.conditions[0].message}{"\n"}' + ``` + +2. Run pipelines in a namespace that is used only for pipeline runs, and change only that namespace's container security policy (Pod Security Admission level) to privileged, from the namespace settings in the console or with the label. Leave the application namespaces at `restricted`, so user deployments stay constrained. + + > **Warning:** The `privileged` level lets any Pod in that namespace run privileged containers, mount host paths, and run as root. Precondition: only when the error is `violates PodSecurity "restricted:latest"` for Tekton containers such as `prepare`, `place-scripts`, or `step-*`, and the namespace is used for pipeline runs. Record the current labels. Rollback: set `pod-security.kubernetes.io/enforce` back to the recorded value. + + ```bash + kubectl label ns pod-security.kubernetes.io/enforce=privileged --overwrite + kubectl get ns -L pod-security.kubernetes.io/enforce + ``` + + The expected result is `privileged` on the pipeline namespace only and `restricted` on the application namespaces. + +3. Run the pipeline again in the pipeline namespace. The TaskRun Pod is created. If a deploy step then fails with `Forbidden` on an application namespace, grant the pipeline's ServiceAccount the permissions it needs in that namespace. + +To check whether the Tekton flag is set and which containers the error lists: + +```bash +kubectl get tektonpipelines.operator.tekton.dev -o yaml | grep -n 'set-security-context' +kubectl get cm -A | grep feature-flags +``` + +`step-*` containers in the message depend on the securityContext the Task defines, not on the flag. `privileged` listed for BuildKit steps means the namespace that runs them needs the privileged level. + +## Diagnostic Steps + +```bash +kubectl get pods -A | grep -Ei 'connector-csi|gitclone|katanomi-controller' +kubectl get events -n --sort-by=.lastTimestamp | grep -Ei 'OOMKilled|FailedMount|Forbidden|Failed' | tail -40 +``` + +- gitclone stuck mounting, and `connector-csi` last state `OOMKilled`: Cause 1. +- HTTP 500 from continuous build, and `katanomi-controller` in `ImagePullBackOff`: Cause 2. +- `FailedMount` naming `{selectedConnectors()}`: Cause 3. +- `violates PodSecurity "restricted:latest"` for `prepare`, `place-scripts`, or `step-*`: Cause 4. + +A Pending gitclone Pod whose event is `0/N nodes are available` is a scheduling problem, not CSI. A rejection that names the `katanomi-setting-init` init container is covered in "Tekton pipeline Pod rejected by Pod Security admission because of the katanomi-setting-init init container". BuildKit steps rejected for `privileged` are covered in "Image build fails with PodSecurity privileged, a RUN step failing on one node, or a buildah certificate error". diff --git a/docs/en/solutions/Pipeline_task_list_is_empty_tasks_are_missing_or_TektonHub_cannot_be_viewed_on_ACP.md b/docs/en/solutions/Pipeline_task_list_is_empty_tasks_are_missing_or_TektonHub_cannot_be_viewed_on_ACP.md new file mode 100644 index 000000000..a1d64d4ab --- /dev/null +++ b/docs/en/solutions/Pipeline_task_list_is_empty_tasks_are_missing_or_TektonHub_cannot_be_viewed_on_ACP.md @@ -0,0 +1,227 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.3" + - "4.3.2" +--- + +# Pipeline task list is empty, tasks are missing, or TektonHub cannot be viewed on ACP + +## Issue + +One of the following happens in the pipeline (Tekton) console: + +- On the graphical pipeline creation page, **Add Task** opens an empty task list. A task the user expects, such as `npm install`, appears to be missing, but in fact no Hub task is listed at all. The Pipeline Tasks page also lists nothing, so a new pipeline cannot be composed, even though Tekton Pipelines itself reports Ready. +- TektonHub cannot be opened, and reinstalling the Tekton operator fails. +- After the pipeline component is upgraded from 4.12.0 to 4.15.1, every Task disappears from the page, although the upgrade did not delete Task resources. +- Every CICD task is gone, a release cannot be started, and `devops-api` is unhealthy. +- A custom or private catalog added to Hub never shows its tasks, or edits to the `tekton-hub-api` ConfigMap are reverted. + +The task list reads its data from TektonHub (or, from pipeline version 4.14, from ArtifactHub Shim). The versions 4.12, 4.14 and 4.15.1 in this article are versions of the pipeline (Tekton operator) component, not of the platform; confirm them with `TektonConfig` before changing anything. + +## Root Cause + +### Cause 1: TektonHub is not installed + +The Tasks page and the **Add Task** dialog have no Hub to read, so the list is empty while Tekton Pipelines is Ready. Restarting the pipeline controllers does not fill the list. On platform 4.3.2, TektonHub was simply not installed; installing a TektonHub instance from the application marketplace restored the task list. + +On the 4.12 operator line, the operator creates TektonHub automatically only when both conditions hold: `TektonConfig.spec.profile` is `all`, and `AUTOINSTALL_TEKTONHUB` in ConfigMap `tekton-operator/tekton-config-defaults` is `"true"`. A stock install leaves the flag `"false"`, so Hub never appears. From pipeline version 4.14 the built-in Tekton Hub is no longer bundled, and 4.15.1 cannot re-enable it through this preset; that is Cause 3. + +### Cause 2: the Triggers webhook Role and RoleBinding are missing after an operator uninstall + +After the Tekton operator was uninstalled, the reinstall failed because the Triggers webhook `Role` and `RoleBinding` were gone. Without them the webhook cannot start, Hub stays unreachable, and the operator install does not finish. + +### Cause 3: from pipeline 4.14, Tekton Hub is no longer bundled + +The upgrade to 4.15.1 does not delete Task resources. From 4.14, the built-in Tekton Hub is no longer shipped, and catalog access moves to the standalone ArtifactHub Shim. Tasks stay off the page until the `artifacthub-shim` plugin is installed, and existing custom catalogs have to be migrated to it. If the Git host that serves a catalog repository no longer answers, that catalog stays empty even with `artifacthub-shim` running. + +### Cause 4: `devops-api` is wedged + +The task list used for release is empty and a release cannot be started because `devops-api` is unhealthy. A slow etcd can leave the component wedged. Restarting `devops-api` brings the CICD tasks back; it does not repair the underlying etcd performance problem. + +### Cause 5: the operator owns `tekton-hub-api` and overwrites direct edits + +When Hub is deployed through `TektonConfig`, the operator owns the `tekton-hub-api` ConfigMap and reverts direct edits, including its `CATALOGS` value. On `tektoncd-operator` release 4.12 the supported path is to override `CATALOGS` through `TektonConfig.spec.hub.options.configMaps`. A private catalog repository needs both `url` and `sshurl`, plus the fixed-name SSH Secret `tekton-hub-api-ssh-crds` (keys `id_rsa`, `id_rsa.pub`, `known_hosts`) in the namespace where Hub runs. Hub rereads catalogs after `CATALOG_REFRESH_INTERVAL` or when the `tekton-hub-api` Pod restarts. Hub itself must be enabled first (Cause 1). + +## Resolution + +### Cause 1: TektonHub is not installed + +1. Confirm that no TektonHub exists and check the pipeline version: + + ```bash + kubectl get tektonhub.operator.tekton.dev -A + kubectl get tektonconfig config -o jsonpath='{.status.version}{"\n"}{.spec.profile}{"\n"}' + kubectl -n tekton-operator get cm tekton-config-defaults -o jsonpath='{.data.AUTOINSTALL_TEKTONHUB}{"\n"}' + ``` + + No TektonHub row means this cause. If the pipeline version is 4.14 or later, go to Cause 3 instead. + +2. Install a TektonHub instance from the application marketplace on the cluster that runs the pipelines. This is the fix that restored the list on platform 4.3.2. Then reopen **Add Task**. + +3. On the 4.12 operator line, the auto-install switch is an alternative to step 2. + + > **Warning:** Setting the flag and restarting `tekton-operator` pulls Hub images and creates Hub API and database Pods; setting `spec.profile` to `all` installs every component of that profile, not only Hub. Only when the list is empty, no TektonHub exists, and the flag is `"false"`. Rollback: restore the saved ConfigMap and the previous `spec.profile`, then restart the operator. Turning the flag off does not delete a TektonHub that was already created. + + ```bash + kubectl -n tekton-operator get cm tekton-config-defaults -o yaml > /tmp/tekton-config-defaults.yaml + kubectl -n tekton-operator patch cm tekton-config-defaults --type=merge -p '{"data":{"AUTOINSTALL_TEKTONHUB":"true"}}' + # only when spec.profile is not already "all" + kubectl patch tektonconfig config --type=merge -p '{"spec":{"profile":"all"}}' + kubectl -n tekton-operator rollout restart deploy/tekton-operator + kubectl -n tekton-operator rollout status deploy/tekton-operator + kubectl get tektonhub.operator.tekton.dev -A + ``` + + Confirm the operator Deployment name with `kubectl -n tekton-operator get deploy` first. + +4. If a TektonHub object exists and the list is still empty, check that the Hub API and database Pods are Ready and that the catalog has synchronized: + + ```bash + kubectl get tektonhub.operator.tekton.dev -A -o jsonpath='{range .items[*]}{.metadata.name}{" "}{range .status.conditions[*]}{.type}{"="}{.status}{" "}{end}{"\n"}{end}' + kubectl get pods -A | grep -Ei 'tekton-hub|hub-api|hub-db' + kubectl logs -n --tail=100 | grep -Ei 'catalog|sync|error' + ``` + + Repair a Pod that is not Ready, or the catalog error in the API log, before installing a second Hub. + +### Cause 2: the Triggers webhook Role and RoleBinding are missing after an operator uninstall + +1. Confirm that the Role and RoleBinding are gone and that the webhook log shows a forbidden error: + + ```bash + kubectl get deploy,pods -A | grep -i 'triggers-webhook\|tekton-triggers' + kubectl get role,rolebinding -A | grep -i trigger + kubectl logs -n deploy/ --tail=80 + ``` + +2. Restore both objects from a healthy cluster on the same operator version, or from that version's Triggers static manifests (installer sets named like `trigger-main-static`). Do not bind the webhook ServiceAccount to `cluster-admin` instead. If `TektonConfig` is Not Ready because an installer set is stuck terminating, follow "TektonConfig stuck Not Ready after Tekton operator upgrade on ACP" instead. + + ```bash + kubectl apply -f /tmp/triggers-webhook-role.yaml + kubectl apply -f /tmp/triggers-webhook-rolebinding.yaml + ``` + +3. Restart the webhook. + + > **Warning:** Admission for trigger resources is unavailable for a few seconds; EventListener and trigger creates fail until the Pod is Ready. Only after the restored Role and RoleBinding come from the same operator version. Rollback: delete the applied objects if they were wrong; a restart alone does not remove them. + + ```bash + kubectl -n rollout restart deploy/ + kubectl -n rollout status deploy/ + ``` + +4. Retry the operator install. If the task list is then empty, continue with Cause 1 or Cause 3. + +### Cause 3: from pipeline 4.14, Tekton Hub is no longer bundled + +1. Confirm the version and whether ArtifactHub Shim runs: + + ```bash + kubectl get tektonconfig config -o jsonpath='{.status.version}{"\n"}' + kubectl get pods -A | grep -i artifacthub + ``` + +2. On 4.14 or later with no ArtifactHub Shim workload, install the `artifacthub-shim` plugin through the same plugin installation path used for other DevOps plugins. The Cause 1 flag does not bring the built-in Hub back on these versions. +3. Migrate existing custom catalogs to ArtifactHub Shim, following its documentation for the cross-namespace global `repository` ConfigMap. Copy the shape from a cluster where the task list already loads instead of inventing keys. +4. If the list is still empty, check that the Git host named in the repository configuration answers, and point the configuration at a reachable host: + + ```bash + kubectl get configmap -A | grep -i repository + kubectl exec -n -- getent hosts + ``` + +### Cause 4: `devops-api` is wedged + +1. Confirm `devops-api` is not Ready or its log shows it is stuck. A healthy `devops-api` with an empty list is Cause 1 or Cause 3. + + ```bash + kubectl get deploy,pods -A | grep devops-api + kubectl -n logs deploy/ --tail=80 + ``` + +2. Restart it. + + > **Warning:** The CICD task list and release API are unavailable until the new Pod is Ready. Only when every CICD task is gone and `devops-api` is unhealthy. If API calls are still slow from an etcd problem, wait until they answer, or the component wedges again. Rollback: none; a restart changes no configuration. + + ```bash + kubectl -n rollout restart deploy/ + kubectl -n rollout status deploy/ + ``` + + If the list empties again soon after, the etcd performance problem is still open; repeating the restart is not a fix. + +### Cause 5: the operator owns `tekton-hub-api` and overwrites direct edits + +1. Read the current list; the override replaces the whole `CATALOGS` value: + + ```bash + kubectl -n get cm tekton-hub-api -o jsonpath='{.data.CATALOGS}{"\n"}' + kubectl get tektonconfig config -o yaml > /tmp/tektonconfig.yaml + ``` + +2. Put the full list, with your entries appended, under the override in `TektonConfig` `config` (example for `tektoncd-operator` release 4.12; keep your own existing entries): + + ```yaml + spec: + hub: + options: + disabled: false + configMaps: + tekton-hub-api: + data: + CATALOGS: | + - name: catalog + org: tektoncd + type: community + provider: github + url: file:////mnt/git/catalog.git + revision: master + - name: private-catalog + org: example + type: private + provider: gitlab + url: https:///devops/private-tekton-catalog.git + sshurl: git@:devops/private-tekton-catalog.git + revision: main + ``` + + > **Warning:** The operator rolls `tekton-hub-api`, so the Hub UI and task list are briefly unavailable; a malformed `CATALOGS` value can leave Hub with no catalogs. Rollback: `kubectl apply -f /tmp/tektonconfig.yaml`, or remove `spec.hub.options.configMaps`. + + ```bash + kubectl edit tektonconfig config + ``` + +3. For a private repository, create the SSH Secret in the Hub namespace: + + ```bash + kubectl -n create secret generic tekton-hub-api-ssh-crds \ + --from-file=id_rsa=/path/to/id_rsa \ + --from-file=id_rsa.pub=/path/to/id_rsa.pub \ + --from-file=known_hosts=/path/to/known_hosts + ``` + +4. Wait for `CATALOG_REFRESH_INTERVAL`, or restart the `tekton-hub-api` Pod. The catalog's tasks should appear in the Hub list. + +## Diagnostic Steps + +```bash +kubectl get tektonhub.operator.tekton.dev -A +kubectl get tektonconfig config -o jsonpath='{.status.version}{"\n"}{.spec.profile}{"\n"}' +kubectl -n tekton-operator get cm tekton-config-defaults -o jsonpath='{.data.AUTOINSTALL_TEKTONHUB}{"\n"}' +kubectl get role,rolebinding -A | grep -i 'triggers-webhook\|tekton-triggers' +kubectl get pods -A | grep -Ei 'artifacthub|devops-api' +``` + +| What you see | Cause | +| --- | --- | +| Empty list, no TektonHub object, pipeline version below 4.14 | 1 | +| Operator reinstall fails, Triggers webhook Role or RoleBinding missing, forbidden in the webhook log | 2 | +| Empty list after an upgrade to 4.15.1, or on 4.14, no ArtifactHub Shim workload | 3 | +| Every CICD task gone, `devops-api` not Ready | 4 | +| Custom catalog never appears, or a `tekton-hub-api` edit is reverted | 5 | + +A stage list that is empty in a DevOps v4 pipeline while TektonHub is installed and Ready, with no API error, is not explained by this article. diff --git a/docs/en/solutions/Planning_an_ACP_upgrade_to_4.2.6_the_4.0.12_starting_version_PostgreSQL_operator_order_and_Kafka_2.x_instances.md b/docs/en/solutions/Planning_an_ACP_upgrade_to_4.2.6_the_4.0.12_starting_version_PostgreSQL_operator_order_and_Kafka_2.x_instances.md new file mode 100644 index 000000000..3a3e05b0d --- /dev/null +++ b/docs/en/solutions/Planning_an_ACP_upgrade_to_4.2.6_the_4.0.12_starting_version_PostgreSQL_operator_order_and_Kafka_2.x_instances.md @@ -0,0 +1,86 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda Application Services +ProductsVersion: + - 4.0.x + - 4.2.x +--- + +# Planning an ACP upgrade to 4.2.6: the 4.0.12 starting version, PostgreSQL operator order and Kafka 2.x instances + +## Issue + +A platform is upgraded in several hops, for example from ACP 3.18.1 to 4.0 and then to 4.2.6, and the plan is blocked by one of these questions: + +- The 4.2 documentation says the supported path is `4.0` → `4.2`, but the published upgrade path for 4.2.6 shows 4.0.12 as its starting version. Can a cluster on an earlier 4.0 patch, such as 4.0.10, go straight to 4.2.6? +- After the platform reaches 4.0.12 the PostgreSQL operator is on its 4.0 line (for example 4.0.25). The compatibility lists seem to block both orders: the PostgreSQL operator 4.0 line is not listed for ACP 4.2, the 4.1 line is not listed for ACP 4.0, and the operator itself is upgraded one minor line at a time (4.0 → 4.1 → 4.2 → 4.3). Which goes first: the platform upgrade to 4.2.6, or the operator upgrade to 4.1? +- Kafka instances still run Kafka 2.x (for example 2.5), and the Kafka operator that comes with the 4.2.6 platform is on the 4.3 line (for example 4.3.3 or 4.3.5). Does that operator support Kafka 2.x? + +## Root Cause + +### Cause 1: the published upgrade path for 4.2.6 starts from 4.0.12 + +The documentation describes supported upgrades between minor versions (`4.0` → `4.2`). The recommended upgrade path published for a target release is more specific: it names the patch version that the upgrade to that release starts from. For 4.2.6 that starting version is 4.0.12. A cluster on an earlier 4.0 patch is first upgraded to 4.0.12 and then to 4.2.6; from 3.18.1 the path is 3.18.1 → 4.0.12 → 4.2.6. A plan made for an earlier 4.2 patch (for example 3.18.1 → 4.0.10 → 4.2.5) is adjusted when the target changes to 4.2.6. + +### Cause 2: upgrade the PostgreSQL operator to 4.1 before the platform upgrade + +With the constraints above, neither order keeps every component inside its compatibility list for the whole upgrade. The recommended order is to upgrade the PostgreSQL operator from 4.0 to 4.1 first, while the platform is still on 4.0.12, and to upgrade the platform to 4.2.6 afterwards. Upgrading the operator first has the smaller impact. + +### Cause 3: Kafka 2.x is not managed by the 4.2 and later Kafka operator lines + +The Kafka operator lines from 4.2 on no longer manage Kafka 2.x instances, so the 4.3 operator that comes with the 4.2.6 platform does not cover an existing Kafka 2.5 instance. On ACP 4.2, Kafka 2.x instances run when the Kafka operator 4.1 line (`kafka-operator` 4.1) is deployed together with the Kafka 2 operator (`kafka-operator-2x`). This combination has been verified only to the point that the Kafka 2.x Pods run; other functions were not tested on ACP 4.2. + +Both of these operators are past the end of their support: they receive no further support and no vulnerability fixes. Running Kafka 2.x on ACP 4.2 this way is a stopgap until the instances are moved to a Kafka version that the current operator line supports. + +## Resolution + +### Cause 1 + +1. Check the current platform version and compare it with the starting version in the recommended upgrade path published for the target release. +2. If the cluster is on a 4.0 patch earlier than 4.0.12, upgrade it to 4.0.12 first, following the ACP 4.0 upgrade documentation, and validate it. +3. Then upgrade from 4.0.12 to 4.2.6, following the ACP 4.2 upgrade documentation. + +> **Warning:** Each platform upgrade restarts platform components and upgrades every cluster, `global` first. Precondition: the pre-upgrade checks of the target version pass. Rollback: follow the recovery guidance in the upgrade documentation of the target version, and take the backups it requires before you start. + +### Cause 2 + +1. On ACP 4.0.12, back up every PostgreSQL instance and confirm that all instances are `Running` (Diagnostic Steps, step 2). +2. Upgrade the PostgreSQL operator from the 4.0 line to the 4.1 line in OperatorHub. + + > **Warning:** While the operator is replaced, no PostgreSQL instance is reconciled, and the new operator reconciles every instance when it starts. OLM does not downgrade an operator in place, so the way back is restoring the instance backups taken in step 1. Do this in a maintenance window, and only when every instance is healthy beforehand. + +3. Wait until the 4.1 operator is `Succeeded` and every instance is `Running` again. +4. Upgrade the platform from 4.0.12 to 4.2.6 (Cause 1). +5. Continue the operator upgrade one minor line at a time (4.1 → 4.2 → 4.3) as the target platform version requires, checking the instances after each step. + +### Cause 3 + +1. Before the platform upgrade, list the Kafka instances and their versions (Diagnostic Steps, step 3). If any instance runs Kafka 2.x, do not let the Kafka operator move to the 4.2 or 4.3 line on that cluster while the instance exists. +2. To keep Kafka 2.x instances running on ACP 4.2, deploy the Kafka operator 4.1 line together with the Kafka 2 operator (`kafka-operator-2x`) instead of the 4.3 operator. Expect only the Pods to be covered by testing on ACP 4.2, and no support or vulnerability fixes for either operator. +3. Plan the move of every Kafka 2.x instance to a Kafka version supported by a current operator line, following the product upgrade documentation of the Kafka operator (moving to the 4.2 line also requires KRaft mode; see the product document "Migrate from ZooKeeper to KRaft mode"). + +## Diagnostic Steps + +1. Platform version: check the current platform version in the platform console and compare it with the starting version of the published upgrade path for the target release. + +2. PostgreSQL operator and instances: + + ```bash + kubectl get csv -A | grep -i postgres + kubectl get postgresql -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,VER:.spec.postgresql.version,STATUS:.status.PostgresClusterStatus + ``` + + The CSV version shows the operator line (`4.0.x`, `4.1.x`, ...). Every instance should show `Running` before and after each operator step. + +3. Kafka operators and instance versions: + + ```bash + kubectl get csv -A | grep -i kafka + kubectl get rdskafka -A -o yaml | grep -nE '^\s+(name|namespace|version):' + ``` + + A `version` of `2.x` on any instance means the cluster needs the Kafka 4.1 operator plus the Kafka 2 operator on ACP 4.2 (Cause 3). A Kafka CSV on the `4.2.x` or `4.3.x` line with such an instance present is the combination to avoid. diff --git a/docs/en/solutions/Platform_UI_stuck_loading_or_DevOps_errors_from_Service_name_timeouts_caused_by_node_search_domains_on_ACP.md b/docs/en/solutions/Platform_UI_stuck_loading_or_DevOps_errors_from_Service_name_timeouts_caused_by_node_search_domains_on_ACP.md new file mode 100644 index 000000000..fab7e82d2 --- /dev/null +++ b/docs/en/solutions/Platform_UI_stuck_loading_or_DevOps_errors_from_Service_name_timeouts_caused_by_node_search_domains_on_ACP.md @@ -0,0 +1,193 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda DevOps +ProductsVersion: + - 4.3.x +--- + +# Platform UI stuck loading or DevOps errors from Service name timeouts caused by node search domains on ACP + +## Issue + +The platform opens slowly or not at all, while the platform Pods all look `Running`. Typical reports: + +- After login, the page keeps loading and never enters the platform. In browser developer tools, `GET /console/api/v1/envs` returns `500 Internal Server Error` with: + + ```text + {"status": "Failure", "message": "failed to reach validation endpoint: Get \"https://erebus.cpaas-system.svc.cluster.local/kubernetes/global/api\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "reason": "InternalError", "code": 500} + ``` + + The `erebus` Pod is `Running` and its log shows only `Starting https proxy server on: :9443` and repeated `Generate new reverse proxy for: ...` lines. +- The platform cannot be opened right after a 4.3 deployment. Requests to `erebus` time out, and the `apollo` log shows connection timeouts to `erebus` or failures resolving a Service. +- The global `apollo` log repeats `List clusters error, sleep 15s`, `Get cluster resource error`, `List log-agent feature error` or `Use global as log-center cluster is empty`, with requests to `https://kubernetes.default.svc.cluster.local/...` or `https://erebus.cpaas-system.svc.cluster.local/...` failing with `net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)` or `context deadline exceeded`. +- Platform access is very slow, and the image selection list loads for a long time or shows nothing when creating a Deployment. +- The DevOps UI shows an error, or GitLab or Harbor integration fails, with `dial tcp: lookup katanomi-plugin.cpaas-system.svc.cluster.local: i/o timeout` (or the same for `katanomi-plugin.katanomi-operator.svc.cluster.local`) in a `500` `InternalError` response. When integrating an external GitLab, the name that fails is the in-cluster `katanomi-plugin` Service, not the GitLab host. +- The image repository dropdown keeps loading for one project; the request `repositories?itemsPerPage=100&page=1` returns `500`. +- On a business cluster, `alertmanager` fails to start and nothing listens on `9093`. +- Every page takes more than a minute to display data, with no specific error. + +Restarting `erebus`, `apollo` or every Pod in `cpaas-system` alone changes nothing. + +## Root Cause + +### Cause 1: extra search domains in the node /etc/resolv.conf + +For Pods with `dnsPolicy: ClusterFirst`, the kubelet appends the node `search` domains to the Pod search list after the cluster suffixes, with `options ndots:5`. A name such as `erebus.cpaas-system.svc.cluster.local` has four dots, so the resolver tries it with every search suffix before the absolute name. Site suffixes (for example `localdomain` or `novalocal`, often written by NetworkManager) are not cluster zones, so CoreDNS forwards those tries to the upstream resolver. When that resolver is slow or drops them, each try waits for its timeout, the calling component's own deadline expires first, and the error looks like an `erebus` or Service failure. The CoreDNS log shows it directly: + +```text +[ERROR] plugin/errors: 2 erebus.cpaas-system.svc.cluster.local.localdomain. A: read udp :->:53: i/o timeout +``` + +A container's `/etc/resolv.conf` is written when the Pod is created. Fixing or rebooting the node does not change running Pods; they keep the old search list until they are recreated. That is why restarting containers or the host alone does not help. + +This cause covers the login page that keeps loading, the 4.3 platform that does not open, the slow image selection page, the DevOps UI error, the `katanomi-plugin` timeouts during GitLab integration, and the `alertmanager` that does not start. For comparison, a healthy node has no `search` line, for example a single nameserver with `options timeout:1` and `options single-request-reopen`. + +### Cause 2: an in-cluster name is still forwarded upstream after the node fix + +After the node `search` conflict is fixed and the Pods are recreated, component logs can still report resolution timeouts for `katanomi-plugin.cpaas-system`. The Pods use `ClusterFirst` and no host network, and manual lookups from the container succeed, but the CoreDNS log shows the name being forwarded to the external upstream and timing out, so the `kubernetes` plugin is not answering it. Why it is forwarded was not determined. A static CoreDNS hosts record that maps `katanomi-plugin.cpaas-system` to its Service ClusterIP stops the timeouts, the external forwarding and the UI errors. + +### Cause 3: katanomi-controller CPU saturation makes the image dropdown time out + +The image dropdown fails for one project only, with `dial tcp: lookup katanomi-plugin.cpaas-system.svc.cluster.local: i/o timeout`, while `katanomi-api`, `katanomi-controller`, `katanomi-plugin` and `katanomi-operator` are all `1/1 Running` and the Harbor instance is associated with the project. Along the path: + +- the `nginx` container log of the `global-alb2` Pod shows `repositories?itemsPerPage=100&page=N` requests to `katanomi-api` returning `500` after about 10 to 16 seconds, and one `499`, while other pages return `200` in 1 to 2 seconds; +- the `katanomi-api` log shows `"list integrationclass error"` when calling `erebus`. + +A large number of concurrent repository requests drives `katanomi-controller` to its CPU limit, and the UI requests behind the dropdown time out under that load. Raising the controller CPU to 2 cores makes the service stable. + +### Cause 4: the image list is empty when creating a Deployment and a CoreDNS restart restores it + +The image list is not displayed when creating a Deployment, and nothing else is found. Restarting CoreDNS on the cluster brings the list back; the reason was not identified. + +### Cause 5: the whole platform is slow because of the users' VPN path + +Every page loads slowly and data takes more than a minute to show, with no failing request. When the delay follows the users' VPN path, the platform is not at fault, and the fix belongs to whoever runs the VPN. + +## Resolution + +### Cause 1: extra search domains in the node /etc/resolv.conf + +Only when the node `/etc/resolv.conf` has a `search` (or `domain`) line with site suffixes and the platform errors are Service-name timeouts. + +1. Comment out the extra `search` entries on every node of the affected cluster (global and workload clusters, if both have them). Keep the `nameserver` lines. If NetworkManager or DHCP writes the file, remove the search domains at that source as well, or the next restart puts them back. + + > **Warning:** Removing the node search domains changes how host processes resolve short names; anything on the host that relied on a site suffix must use full names afterwards. Back up the file on each node. To roll back, restore the backup and recreate the Pods again. + + ```bash + cp -a /etc/resolv.conf /root/resolv.conf.before + grep -nE '^(search|domain)' /etc/resolv.conf + sed -i -E 's/^(search|domain)([[:space:]])/# \1\2/' /etc/resolv.conf + cat /etc/resolv.conf + ``` + +2. Recreate the Pods so that the kubelet writes a new Pod `/etc/resolv.conf`. The smallest set that recovers the platform is CoreDNS plus `apollo`, `erebus`, the frontend and the katanomi Pods; recreating every Pod in `cpaas-system` also works. + + > **Warning:** Deleting these Pods makes the console and DevOps functions unavailable until the replacements are Ready, and a CoreDNS restart briefly delays lookups. Fix the node files first. No rollback is needed; the controllers recreate the Pods. + + ```bash + kubectl -n kube-system rollout restart deploy/coredns + kubectl -n cpaas-system get pod -o name | grep -E 'apollo|erebus|frontend|katanomi' | xargs kubectl -n cpaas-system delete + ``` + +3. Recreate business Pods that show the same timeouts (for example a business-cluster `alertmanager`) the same way. Then confirm a platform Pod has no site suffix: + + ```bash + kubectl -n cpaas-system exec deploy/apollo -- cat /etc/resolv.conf + ``` + +### Cause 2: an in-cluster name is still forwarded upstream after the node fix + +Only when the CoreDNS log shows `katanomi-plugin.cpaas-system` (or another platform Service name) being forwarded to the upstream after Cause 1 is fixed. + +1. Read the Service ClusterIP: + + ```bash + kubectl -n cpaas-system get svc katanomi-plugin -o jsonpath='{.spec.clusterIP}{"\n"}' + ``` + +2. Add a static record ` katanomi-plugin.cpaas-system` under **Host Alias** in the cluster plugin **Alauda Container Platform Cluster DNS** (**Administrator > Marketplace > Cluster Plugins**), then restart CoreDNS and watch its log. See [CoreDNS Host Alias records do not take effect because a ResourcePatch removes data hosts from cpaas-coredns on ACP](CoreDNS_Host_Alias_records_do_not_take_effect_because_a_ResourcePatch_removes_data_hosts_from_cpaas-coredns_on_ACP.md) if the record does not take effect. + + > **Warning:** A hosts record pins the ClusterIP. If the Service is deleted and recreated with a new ClusterIP, the record keeps answering the old address and the component breaks again. Note that the entry exists so it can be removed later. To roll back, delete the record and restart CoreDNS. + + ```bash + kubectl -n kube-system rollout restart deploy/coredns + kubectl -n kube-system logs -l k8s-app=kube-dns --tail=100 | grep katanomi-plugin + ``` + +### Cause 3: katanomi-controller CPU saturation makes the image dropdown time out + +Only when `katanomi-controller` CPU sits at its limit while the dropdown fails. Raise the controller CPU to 2 cores in the katanomi instance resource (`katanomi-global`, under **Operators > katanomi-operator** in the console), under `helmValues` `controller.resources`. + +> **Warning:** Changing the resources rolls `katanomi-controller`, and DevOps operations pause until it is Ready. Make sure the node has 2 free cores, and save the instance YAML. To roll back, apply the saved YAML. + +```bash +kubectl -n cpaas-system top pod | grep katanomi +kubectl api-resources | grep -i katanomi +kubectl get -A | grep katanomi-global +kubectl get -n -o yaml > /tmp/katanomi.before.yaml +kubectl edit -n +``` + +```yaml +helmValues: + controller: + resources: + limits: + cpu: "2" +``` + +### Cause 4: the image list is empty when creating a Deployment and a CoreDNS restart restores it + +Only after the checks for Causes 1 to 3 found nothing. Restart CoreDNS and reload the Deployment creation page. + +> **Warning:** A rollout restart replaces the CoreDNS Pods one at a time, so cluster lookups can stall for a moment. Make sure at least two CoreDNS replicas are Ready. No rollback is needed. + +```bash +kubectl -n kube-system rollout restart deploy/coredns +kubectl -n kube-system rollout status deploy/coredns +``` + +### Cause 5: the whole platform is slow because of the users' VPN path + +Do not change nodes or Pods for this signature. Time the platform once from a client that goes through the VPN and once from a host that reaches the platform without it (for example a global cluster node), and hand the comparison to the VPN owner: + +```bash +curl -sk -o /dev/null -w 'connect=%{time_connect}s tls=%{time_appconnect}s first_byte=%{time_starttransfer}s total=%{time_total}s\n' https:/// +``` + +A large gap in `connect` or `first_byte` between the two, with fast times from the host, places the delay on the VPN path. + +## Diagnostic Steps + +1. In the browser developer tools, find the request that hangs. A `500` with `failed to reach validation endpoint` on `erebus`, or a timeout on an `erebus` URL, points to name resolution inside the platform Pods. If no request fails and every request is uniformly slow, time the platform from a host that does not use the users' VPN first (Cause 5). + +2. Read the node and Pod resolver files. A site suffix in either is Cause 1: + + ```bash + cat /etc/resolv.conf + kubectl -n cpaas-system exec deploy/apollo -- cat /etc/resolv.conf + kubectl -n cpaas-system logs deploy/apollo --tail=100 | grep -iE 'erebus|timeout|resolve' + ``` + +3. Time a lookup of the platform Service from inside a platform Pod whose image has `sh` and `nslookup`. Seconds for the name without a trailing dot and milliseconds for the absolute name confirm the search expansion cost: + + ```bash + kubectl -n cpaas-system exec deploy/apollo -- sh -c 'time nslookup erebus.cpaas-system.svc.cluster.local' + kubectl -n cpaas-system exec deploy/apollo -- sh -c 'time nslookup erebus.cpaas-system.svc.cluster.local.' + ``` + +4. For DevOps errors, check katanomi and its endpoints, then CoreDNS for forwarded platform names (Cause 2) and controller CPU (Cause 3): + + ```bash + kubectl get pod -A | grep katanomi + kubectl get ep -A | grep katanomi + kubectl -n kube-system logs -l k8s-app=kube-dns --tail=200 | grep -iE 'cpaas-system|i/o timeout' + kubectl -n cpaas-system top pod | grep katanomi + kubectl -n cpaas-system logs -c nginx | grep 'repositories?itemsPerPage' + kubectl -n cpaas-system logs deploy/katanomi-api --tail=200 | grep 'list integrationclass error' + ``` + + A CoreDNS log line whose name ends in a site suffix (for example `.localdomain.`) with `i/o timeout` is Cause 1. A platform Service name forwarded upstream with no suffix after the node fix is Cause 2. Slow `500` or `499` answers from `katanomi-api` in the ALB log, with `katanomi-controller` at its CPU limit, is Cause 3. diff --git a/docs/en/solutions/Platform_access_fails_or_navigation_entries_disappear_after_a_load_balancer_DNS_or_access_address_change_on_ACP.md b/docs/en/solutions/Platform_access_fails_or_navigation_entries_disappear_after_a_load_balancer_DNS_or_access_address_change_on_ACP.md new file mode 100644 index 000000000..3b2ce3303 --- /dev/null +++ b/docs/en/solutions/Platform_access_fails_or_navigation_entries_disappear_after_a_load_balancer_DNS_or_access_address_change_on_ACP.md @@ -0,0 +1,141 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Platform access fails or navigation entries disappear after a load balancer, DNS, or access address change on ACP + +## Issue + +Something outside the ACP objects changed how the platform address, the API endpoint, or the platform domain is reached. Typical symptoms: + +- The platform cannot be reached at its VIP after NGINX or an external load balancer was adjusted, although nothing changed in the cluster. +- The platform answers by IP, but login through the domain shows `授权码不合法` (invalid authorization code). +- After a new `global` installation, the administrator no longer sees **Operations Center** or **Marketplace Management**, in every browser and in an incognito window. +- After a `global` upgrade, the overview and App Market pages do not render. +- How do I change the platform domain, and does the old address still work after a new one is added? + +Check, in order: is a `global` control-plane node abnormal, is the domain registered as a platform access address (`prdb` `base`), and does the platform domain resolve inside the cluster. + +## Root Cause + +### Cause 1: the platform VIP does not answer because one global control-plane node is abnormal + +After a load balancer outside the cluster was adjusted, the `global` VIP does not open the console. One control-plane node is abnormal, and rebooting it restores VIP access. Why the node failed was not established. + +### Cause 2: the domain is not registered as a platform access address + +Login by IP works and login by domain shows `授权码不合法`. The domain is not a platform access address: **Platform Access Address** lists only the default IP, and `kubectl get prdb base -o yaml` on the `global` cluster has no entry for the domain under `spec.alternativeURLs`. + +### Cause 3: the platform domain does not resolve inside the cluster + +**Operations Center** and **Marketplace Management** disappear for `admin`; under **Cluster Management** only **Clusters**, **Cloud Credentials**, **Custom Resource Definitions**, and **Resource Management** remain, and the user menu shows only **Appearance**, **Language**, and **Log Out**. Opening the Marketplace Management URL (`/console-platform/manage-platform/cluster/global/catalog-management/operator`) redirects elsewhere. The platform looks healthy: AppReleases are `Synced`/`Ready`, the role bindings exist, and the navigation requests (`selfsubjectaccessreviews`, `userbindings`, `navconfig-platform.yaml`, `custom-navconfig-platform`) return 200 or 201. + +From a Pod such as `pinger`, the platform domain is `unknown host`. The domain exists only in hosts files, not in DNS. A hosts entry that had been added by hand to the CoreDNS ConfigMap was lost, so in-cluster components cannot resolve the platform domain and the navigation modules are not shown. + +### Cause 4: CoreDNS runs on global nodes that have no resolver + +After a `global` upgrade, the overview and App Market pages do not render. Some `global` nodes have no DNS configuration in `/etc/resolv.conf`; when CoreDNS runs on those nodes, it cannot resolve the platform name. + +### Cause 5: an address added under Other Access Address does not replace the deployment address + +In **Administrator** > **System Settings** > **Platform Parameters**, the platform access URLs show the deployment address marked `Default`. The **Update Platform Access URLs** dialog has an **Other Access Address** list. An address added there is an additional way to reach the platform; it does not replace the deployment address, so the `Default` address keeps working. There is no one-step automation for changing the platform domain. + +## Resolution + +### Cause 1: the platform VIP does not answer because one global control-plane node is abnormal + +```bash +kubectl get nodes +``` + +> **Warning:** Rebooting a control-plane node restarts its static Pods and drops its API server until the node is back. Do this only when the `global` VIP does not open the console and this node is the abnormal one, in a maintenance window, with console access to the other masters. + +Reboot that node. The VIP answers once the node is `Ready`. If domain login then shows `授权码不合法`, continue with Cause 2. + +### Cause 2: the domain is not registered as a platform access address + +1. Check the registered addresses: + + ```bash + kubectl get prdb base -o yaml | grep -i url -A5 + ``` + +2. Log in by IP. Under **Platform Management** > **Platform Settings** > **Platform Parameters** > **Platform Access Address**, choose **Update Platform Access Address** and add the domain. Removing the entry again is the rollback. + +3. `kubectl get prdb base -o yaml` now lists the domain under `spec.alternativeURLs`, and login by domain succeeds. + +### Cause 3: the platform domain does not resolve inside the cluster + +1. Test the domain from inside the cluster and find the CoreDNS module object: + + ```bash + kubectl get pods -A | grep pinger + kubectl exec -n -- ping -c 1 + kubectl get minfo -A | grep -i coredns + ``` + + `unknown host`, with healthy AppReleases, matches this cause. If `pinger` is `Pending` because every node is tainted, use any Pod that has `ping` or `nslookup`. + +2. Make the domain resolvable in a way that persists. Preferably add it to the DNS servers that the nodes use. Otherwise add a **Host Alias** to the **Alauda Container Platform Cluster DNS** plugin as described in [Platform console does not open because the cluster VIP domain name cannot be resolved inside the cluster on ACP](Platform_console_does_not_open_because_the_cluster_VIP_domain_name_cannot_be_resolved_inside_the_cluster_on_ACP.md), or add the entry under `spec.config.hosts` of the CoreDNS `minfo`. Entries added only by hand to the CoreDNS ConfigMap can be lost. + + > **Warning:** The CoreDNS configuration drives name resolution for the whole `global` cluster. Do this only when the platform domain is not served by any DNS server. Export the object first; applying the export is the rollback. + + ```bash + kubectl get minfo -o yaml > "$HOME/coredns-minfo.yaml" + kubectl edit minfo + ``` + + If `spec.config.hosts` already has an entry, copy its exact shape. Otherwise: + + ```yaml + spec: + config: + hosts: + - domains: + - + ip: 192.0.2.10 + ``` + +3. The left navigation, including **Operations Center** and **Marketplace Management**, returns once the domain resolves from inside the cluster. + +### Cause 4: CoreDNS runs on global nodes that have no resolver + +```bash +kubectl get pods -n kube-system -o wide | grep coredns +``` + +On each node that hosts CoreDNS, `/etc/resolv.conf` must list a resolver that can answer the platform DNS name. Add the resolver that the other `global` nodes use, then delete the CoreDNS Pod on that node so that it is recreated and reads the new configuration. The overview and App Market render once the platform name resolves from every CoreDNS Pod. + +### Cause 5: an address added under Other Access Address does not replace the deployment address + +To let users reach the platform by a new domain, add it under **Other Access Address** in **Update Platform Access URLs**. The `Default` deployment address keeps working alongside it. Replacing the `Default` address is not a console operation; contact technical support for that. Check the current addresses with: + +```bash +kubectl get prdb base -o yaml | grep -i url -A5 +``` + +## Diagnostic Steps + +```bash +kubectl get nodes +kubectl get prdb base -o yaml | grep -i url -A5 +kubectl get minfo -A | grep -i coredns +kubectl exec -n -- ping -c 1 +``` + +```text +VIP dead after a load balancer change, one control-plane node abnormal -> Cause 1 +domain login 授权码不合法, IP login works, domain missing from alternativeURLs -> Cause 2 +Operations Center and Marketplace Management missing, unknown host in a Pod -> Cause 3 +overview and App Market blank, a CoreDNS node has no resolver -> Cause 4 +new domain added, old Default address still works -> Cause 5, by design +``` diff --git a/docs/en/solutions/Platform_certificate_expiry_alert_for_a_cert_manager_serving_certificate_that_renews_by_itself_on_ACP.md b/docs/en/solutions/Platform_certificate_expiry_alert_for_a_cert_manager_serving_certificate_that_renews_by_itself_on_ACP.md new file mode 100644 index 000000000..9186583f9 --- /dev/null +++ b/docs/en/solutions/Platform_certificate_expiry_alert_for_a_cert_manager_serving_certificate_that_renews_by_itself_on_ACP.md @@ -0,0 +1,120 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.12.x + - 3.18.x + - 4.0.x + - 4.2.x + - 4.3.x + - 4.4.x +--- + +# Platform certificate expiry alert for a cert-manager serving certificate that renews by itself on ACP + +## Issue + +The platform raises a certificate alert such as "platform component certificate expires in less than 30 days", "less than 10 days", or "platform component certificate has expired". In Chinese consoles the rule reads `平台组件证书的到期时间即将到期(小于30天) <= 30d 且持续 1 分钟` (platform component certificate expires in less than 30 days, lasting 1 minute). The summary names `//`, for example `global/cpaas-system/base-operator.tls`. Or an inspection report or the certificate status page lists a few platform certificates with about 30, 9, or 1 days left, and the question is whether they renew by themselves. + +Typical names are `sentry-serving-cert`, `capi-serving-cert`, `compliance-manager-cert`, `katanomi-artifacts-serving-cert`, `global-asm-serving-cert`, `base-operator.tls`, `argocd-secret`, `asm-istio-cacerts-`, `asm-istio-cacert-global`, and `cacerts-ca-cert.pem`. + +## Root Cause + +### Cause 1: a cert-manager serving certificate still has about 30 days left and will renew by itself + +Platform serving certificates are issued and renewed by the platform cert-manager. The lifetime and the renewal point are set on each `Certificate` object (`spec.duration`, `spec.renewBefore`), and cert-manager renews the certificate at `status.renewalTime`, which lies ahead of `status.notAfter`. A healthy Certificate shows Ready `True` with the message `Certificate is up to date and has not expired`. A remaining validity of about 30 days on these certificates is therefore normal and needs no manual step: the 30-day alert fires at, or before, the point where cert-manager renews. When the renewal point passes, the certificate is re-signed and the alert recovers by itself. + +The renewal point belongs to each object, so read it instead of assuming one number: + +| Certificate | Renewal behavior | +|---|---| +| `*-serving-cert` in `cpaas-system` (`sentry-serving-cert`, `capi-serving-cert`, `katanomi-artifacts-serving-cert`, `global-asm-serving-cert`), `compliance-manager-cert`, `base-operator.tls` | Renewed by cert-manager at `renewalTime`; about 30 days left is the normal window | +| `asm-istio-cacerts-` in `global/cpaas-system` (ACP 4.x Istio intermediate CA for a business cluster) | `duration: 1440h0m0s`, `renewBefore: 360h0m0s`: valid about 60 days and renewed about 15 days before expiry, so the 30-day alert fires about 15 days before renewal | +| `cacerts-ca-cert.pem` in `istio-system` (ACP 3.x Istio intermediate CA) | `duration: 1440h0m0s`, `renewBefore: 720h0m0s`, renewed by cert-manager on the business cluster | +| `metallb-operator` certificate | Renewed 24 hours before expiry | +| `argocd-secret` on global | Renewed by argocd-operator when it is about to expire; it can still be unrenewed with less than 29 days left | +| Operator service certificates (`packageserver-service-cert`, `*-operator-service-cert`) | Rotated by the owning operator about one day before expiry; see "Operator service certificates such as packageserver-service-cert about to expire or not renewed on ACP" | + +Because the Istio intermediate CA is valid for only about two months, a recurring under-30-day report on `asm-istio-cacerts-` is expected. The global cluster raises it under names that refer to a business cluster, because the intermediate CA for each business cluster is signed and rotated by cert-manager on global and distributed to that cluster by `global-asm-controller`. A `ca.crt` with a very long (for example 100-year) lifetime in the same Secret is the root CA, not the certificate the alert counts. Read `tls.crt`. + +The following are not renewed automatically and need action when they approach expiry: + +- `asm-controller-webhook-serving-cert` (the documentation lists it as not renewed automatically). +- `dex.tls` when the platform was deployed with a certificate supplied by its owner. See "Replace the platform access certificate in dex.tls on ACP". +- The one-year Harbor notary TLS certificate. See "Harbor notary-server fails with an expired x509 certificate on ACP". +- The node file `/root/.kube/config`, which the same alert can name. See "Node kubectl fails with You must be logged in to the server or an alert names root kube config on ACP". + +A certificate that is not a platform object is renewed by the application owner. For example, a Secret such as `sentry-nginx-tls` in a `sentry-` namespace belongs to a separately deployed Sentry instance, not to the platform's built-in Sentry component. + +## Resolution + +### Cause 1: a cert-manager serving certificate still has about 30 days left and will renew by itself + +1. On the cluster the alert names, read the Certificate's readiness, renewal time, and Secret, then print the leaf certificate: + + ```bash + kubectl get certificate -n \ + -o custom-columns='NAME:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status,RENEW:.status.renewalTime,SECRET:.spec.secretName' + kubectl get secret -n -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject -issuer + ``` + +2. If the Certificate is Ready `True`, `renewalTime` is in the future, and `notAfter` is still ahead, leave the Secret in place. After `renewalTime`, run the commands again and confirm that `notAfter` moved forward. The alert clears by itself. A recurring Istio CA alert that is explained by the table above can be silenced. + +3. To renew now instead of at the automatic point, back up and delete the Secret so that cert-manager reissues it. For the Istio intermediate CA on a business cluster that uses the certificate management plugin, restart `istiod` after the delete. + + > **Warning:** Deleting a cert-manager-managed Secret leaves the component without its certificate until cert-manager writes the new one, and restarting `istiod` briefly interrupts certificate issuance to mesh sidecars. Do this only for the cert-manager-managed names in the table above, and only when an immediate renewal is needed or the certificate is already past `notAfter`. Do not delete `asm-istio-cacerts-` for a cluster that still exists unless the renewal is really needed: it holds that cluster's Istio intermediate CA. Take the backup first. Rollback is `kubectl apply -f` of the backup, which puts the previous certificate back. + + ```bash + umask 077 + kubectl get secret -n -o yaml > "$HOME/.yaml" + kubectl delete secret -n + kubectl get secret -n -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates + # Istio certificate on a business cluster only: restart istiod after the delete + kubectl get deploy -n istio-system | grep istiod + kubectl rollout restart deployment -n istio-system + ``` + +4. If the Secret is not regenerated after the delete, restart cert-manager on the cluster that holds the Secret. + + > **Warning:** Deleting the cert-manager Pods pauses issuance and renewal of every cert-manager certificate on that cluster until the new Pods are Running; certificates already issued keep working. Do this only after the Secret backup exists and the Secret was deleted in step 3 but was not regenerated. The Deployment recreates the Pods. If the Secret still does not come back, restore it with `kubectl apply -f` of the backup. + + ```bash + kubectl get pods -A | grep cert-manager + kubectl delete pod -n + kubectl get secret -n + ``` + +## Diagnostic Steps + +1. Read the alert's cluster, namespace, and certificate name. When the alert does not name the certificate, scroll to the bottom of the alert monitoring detail panel; the legend lists each alerting series with `name` and `secret_namespace`. A series whose latest value is exactly `30.0 d` has just entered the alert window. + +2. On that cluster, print the leaf and, for comparison, the CA in the same Secret: + + ```bash + kubectl get certificate -A | grep + kubectl get secret -n -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject -issuer + kubectl get secret -n -o jsonpath='{.data.ca\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject -issuer + ``` + + The `tls.crt` dates are what the alert counts. + +3. For `asm-istio-cacerts-`, confirm that the cluster still exists: + + ```bash + kubectl get cls | grep + ``` + + A cluster that no longer exists leaves a leftover Secret. That case is covered in "Expired leftover certificates that no component uses on ACP". + +4. Branch: + + - Ready `True`, future `renewalTime`, a name from the table: Cause 1, wait. + - The Certificate condition reads `Existing private key is not up to date for spec: [spec.keyAlgorithm]`, or the alert name matches no Secret: "Platform component certificate not renewed: stale private key, issuerRef group, or alert name that differs from the Secret on ACP". + - The Secret is already past `notAfter`: "Expired platform component certificate Secret such as hco-webhook-service-cert not reissued on ACP". + - An operator service certificate (`packageserver-service-cert`, `-operator-service-cert`): "Operator service certificates such as packageserver-service-cert about to expire or not renewed on ACP". diff --git a/docs/en/solutions/Platform_component_certificate_not_renewed_stale_private_key_issuerRef_group_or_alert_name_that_differs_from_the_Secret_on_ACP.md b/docs/en/solutions/Platform_component_certificate_not_renewed_stale_private_key_issuerRef_group_or_alert_name_that_differs_from_the_Secret_on_ACP.md new file mode 100644 index 000000000..170dc6586 --- /dev/null +++ b/docs/en/solutions/Platform_component_certificate_not_renewed_stale_private_key_issuerRef_group_or_alert_name_that_differs_from_the_Secret_on_ACP.md @@ -0,0 +1,192 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.12.x + - 3.18.x + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# Platform component certificate not renewed: stale private key, issuerRef group, or alert name that differs from the Secret on ACP + +## Issue + +A platform component certificate has expired, or is close to expiry, and did not renew by itself. Typical reports: + +- On a cluster with DevOps components, `devops-tool-operator-controller-manager-service-cert` is expired. An inspection can report it as `devops-tool-operator-controller-manager-service-cert 证书有效期还剩-61天` (remaining validity -61 days). After its Secret was rebuilt, the certificate page still shows it as expired, and the browser console shows `found duplicate series for the match group` and `many-to-many matching not allowed` on `/v2/metrics/global/query`. +- The alert says `katanomi-artifacts-serving-cert` in `operators` has less than `30.00d` left, but `kubectl get secrets -A | grep katanomi-artifacts-serving-cert` returns nothing, and rebuilding another katanomi Secret does not change the alert. +- `rds-operator-controller-manager-service-cert` is expired on global (the certificate page can truncate it to `rds-operator-controller-manager-service-cer`), or its Secret was deleted by hand and never reissued. +- An inspection names a certificate, and a search for it on the cluster you are logged in to finds no Secret. + +## Root Cause + +### Cause 1: the Certificate reports Existing private key is not up to date for spec: [spec.keyAlgorithm] + +The cert-manager `Certificate` (for example `devops-tool-operator-controller-manager-service-cert` in `operators`, with `duration: 17520h0m0s`, `renewBefore: 720h0m0s`, issuer `ClusterIssuer` `cpaas-ca`) shows the condition `Existing private key is not up to date for spec: [spec.keyAlgorithm]` and, once the date passes, `Certificate expired`. The private key stored in the Secret does not match the key algorithm in the Certificate spec, and the certificate is not re-issued. This is a known issue with this credential: deleting the Secret once lets cert-manager write a new key and certificate, after which cert-manager manages it normally. The operator that serves the certificate must then be restarted to load it. + +The issue is identified by the `spec.keyAlgorithm` condition, not by the certificate name. Read the conditions on your cluster before treating an alert for this name as this cause. + +After the renewal, the certificate page can keep showing the old state. The monitoring data then holds two series for the same certificate, one with `phase="Expired"` and one with `phase="Normal"`, and the page query fails with `found duplicate series for the match group ... many-to-many matching not allowed: matching labels must be unique on one side`. Restarting `courier` on the global cluster clears the old series. + +### Cause 2: the alert names a certificate, and no Secret with that name exists + +The alert carries the certificate name, and the Secret that stores it can have a different name. The certificate name appears in the Secret's annotations, or in `spec.secretName` of its `Certificate`. For `katanomi-artifacts-serving-cert`, the Secret is `katanomi-artifacts-webhook-service-cert` in `operators`, alongside other katanomi Secrets such as `katanomi-operator-service-cert` and `katanomi-webhook-server-cert`. Deleting one of those other Secrets reissues a different certificate and leaves the alert unchanged. + +Like most platform component certificates, this one normally renews automatically before it expires, so an alert that is only inside the 30-day window can be left alone. Delete-to-reissue does not work for every platform component certificate: it depends on the component and the platform version, because the management layer above the Secret differs. + +### Cause 3: issuerRef has group ait.alauda.io, so cert-manager does not reissue rds-operator-controller-manager-service-cert + +The `Certificate` `rds-operator-controller-manager-service-cert` (`duration: 17520h0m0s`, `renewBefore: 720h0m0s`) contains: + +```yaml +issuerRef: + group: ait.alauda.io + kind: ClusterIssuer + name: cpaas-ca +``` + +cert-manager's built-in issuers handle only `issuerRef` entries of group `cert-manager.io` (or no group). A Certificate that names another group is left to an external issuer, so cert-manager does not issue it from the `cpaas-ca` ClusterIssuer, and it is not reissued when it expires or when its Secret is deleted. Removing the `group` field from `issuerRef` lets cert-manager issue the certificate; the Secret is updated with a new certificate and the alert clears. + +### Cause 4: the alerting certificate is on another cluster + +The alert or inspection row names a certificate that belongs to another cluster of the same platform, for example a business cluster while the search was run on global. Searching on the wrong cluster finds nothing. On the right cluster the Secret exists and can be renewed with the method that matches its name. + +## Resolution + +### Cause 1: the Certificate reports Existing private key is not up to date for spec: [spec.keyAlgorithm] + +1. Read the Certificate and its conditions, and print the certificate dates: + + ```bash + kubectl get cert -A | grep devops-tool + kubectl get cert -n operators devops-tool-operator-controller-manager-service-cert -o yaml + kubectl get secret -n operators devops-tool-operator-controller-manager-service-cert -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject -issuer + ``` + + Continue only when the condition text is `Existing private key is not up to date for spec: [spec.keyAlgorithm]`, or the certificate is expired or near expiry and has not renewed. If `kubectl get cert -A` printed another namespace, use it. + +2. Back up and delete the Secret, then check that it was reissued. + + > **Warning:** Deleting the Secret removes the certificate the devops-tool operator serves until cert-manager writes a new one; clients of that service fail TLS in the gap. Do this only when step 1 confirmed the condition or the expiry. Rollback is `kubectl apply -f` of the backup, which restores the old certificate. + + ```bash + umask 077 + kubectl get secret -n operators devops-tool-operator-controller-manager-service-cert -o yaml > "$HOME/devops-tool-operator-controller-manager-service-cert.yaml" + kubectl delete secret -n operators devops-tool-operator-controller-manager-service-cert + kubectl get secret -n operators devops-tool-operator-controller-manager-service-cert -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject -issuer + kubectl get cert -n operators devops-tool-operator-controller-manager-service-cert + ``` + + The new `notAfter` must be later than before, and the Certificate must be Ready. + +3. Restart the operator Pod so that it loads the new certificate. + + > **Warning:** Deleting the `devops-tools-operator` Pod stops that operator's reconciliation until the Deployment starts a new Pod. Do this only after the new Secret shows a future `notAfter`. The Deployment recreates the Pod. + + ```bash + kubectl get pods -A | grep devops-tools-operator + kubectl delete pod -n + ``` + +4. If the certificate page still lists the certificate as expired, and the browser console shows `found duplicate series` with `phase="Expired"` and `phase="Normal"` for the same name, restart `courier` on the global cluster. + + > **Warning:** Restarting the global `courier` Pods interrupts alert delivery and the console certificate status until they are Running again. Do this only after the new Secret's `notAfter` is in the future and the page still shows the old state. The Deployment recreates the Pods. + + ```bash + kubectl get pods -A | grep courier + kubectl delete pod -n + ``` + +### Cause 2: the alert names a certificate, and no Secret with that name exists + +1. Map the certificate name to its Secret before deleting anything: + + ```bash + kubectl get certificate -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,SECRET:.spec.secretName' \ + | grep + kubectl get secret -A \ + -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.metadata.annotations}{"\n"}{end}' \ + | grep + ``` + +2. Print the `tls.crt` dates of the Secret found. If they are only inside the 30-day window, leaving it for automatic renewal is the default answer. + +3. To renew now, back up and delete that Secret (for `katanomi-artifacts-serving-cert`: `katanomi-artifacts-webhook-service-cert` in `operators`). + + > **Warning:** Deleting `katanomi-artifacts-webhook-service-cert` removes the webhook certificate of the katanomi artifacts component until it is reissued; calls to that webhook fail in the gap. Do this only when the lookup in step 1 has tied this Secret to the alerting certificate name. Rollback is `kubectl apply -f` of the backup. This method is not confirmed for every platform component, so do not apply it to another Secret by analogy. + + ```bash + umask 077 + kubectl get secret -n operators katanomi-artifacts-webhook-service-cert -o yaml > "$HOME/katanomi-artifacts-webhook-service-cert.yaml" + kubectl delete secret -n operators katanomi-artifacts-webhook-service-cert + kubectl get secret -n operators katanomi-artifacts-webhook-service-cert -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates + ``` + +4. Restart `courier` on the global cluster (step 4 of Cause 1, with its Warning) so that the page shows the new date. Without the restart the page can lag for a while. + +### Cause 3: issuerRef has group ait.alauda.io, so cert-manager does not reissue rds-operator-controller-manager-service-cert + +1. Confirm the signature: the certificate is expired or its Secret is missing after a delete, and `issuerRef` contains `group: ait.alauda.io`. + + ```bash + kubectl get certificate -A | grep rds-operator-controller-manager-service-cert + kubectl get certificate -n operators rds-operator-controller-manager-service-cert -o jsonpath='{.spec.issuerRef}{"\n"}{.spec.secretName}{"\n"}' + kubectl get secret -n operators rds-operator-controller-manager-service-cert -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates + ``` + + If the Certificate is in another namespace, use that namespace. + +2. Remove `issuerRef.group`. + + > **Warning:** Removing `issuerRef.group` hands issuance of this certificate to cert-manager and the `cpaas-ca` ClusterIssuer; cert-manager rewrites the Secret with a new key and certificate, and the rds-operator serves it from then on. Do this only for `rds-operator-controller-manager-service-cert` when step 1 confirmed the signature; do not apply it to other certificates. Back up the Certificate and the Secret first. Rollback is `kubectl apply -f` of the Certificate backup, which puts the group back; the reissued certificate stays until the Secret backup is also applied. + + ```bash + umask 077 + kubectl get certificate -n operators rds-operator-controller-manager-service-cert -o yaml > "$HOME/rds-operator-certificate.yaml" + kubectl get secret -n operators rds-operator-controller-manager-service-cert -o yaml > "$HOME/rds-operator-secret.yaml" 2>/dev/null + kubectl patch certificate -n operators rds-operator-controller-manager-service-cert --type=json \ + -p '[{"op": "remove", "path": "/spec/issuerRef/group"}]' + kubectl get certificate -n operators rds-operator-controller-manager-service-cert + kubectl get secret -n operators rds-operator-controller-manager-service-cert -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates + ``` + + `kubectl edit` of the Certificate, deleting the `group` line, has the same effect. The new `notAfter` should be about two years ahead. Check the alert again the next day. + +### Cause 4: the alerting certificate is on another cluster + +1. Read the cluster name from the alert or inspection row, switch to that cluster's kubeconfig (or the console terminal of that cluster), and search there: + + ```bash + kubectl get secret -A | grep + kubectl get secret -n -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates + ``` + +2. Renew it with the method that matches the name on that cluster: Cause 1 for `devops-tool-operator-controller-manager-service-cert`, or "Operator service certificates such as packageserver-service-cert about to expire or not renewed on ACP" for operator service certificates. The same Warning and backup apply. Do not delete a different Secret on the current cluster in its place. + +## Diagnostic Steps + +1. Print the Certificate status and the Secret it names: + + ```bash + kubectl get certificate -A | grep -v True + kubectl get certificate -n \ + -o jsonpath='{.spec.secretName}{"\n"}{.spec.issuerRef}{"\n"}{range .status.conditions[*]}{.type}{"="}{.status}{" "}{.message}{"\n"}{end}' + ``` + +2. Branch: + + - A condition reads `Existing private key is not up to date for spec: [spec.keyAlgorithm]`: Cause 1. + - The page still shows a renewed certificate as expired and the browser console reports `duplicate series` with `phase="Expired"` and `phase="Normal"`: step 4 of Cause 1. + - The alert name matches no Secret on the alert's cluster: Cause 2. No Secret on the cluster you searched, and the alert names a different cluster: Cause 4. + - `issuerRef.group: ait.alauda.io` on `rds-operator-controller-manager-service-cert`, and the certificate expired or was not reissued after a delete: Cause 3. + - The Certificate is Ready, `notAfter` is in the future, and only the 30-day alert fired: "Platform certificate expiry alert for a cert-manager serving certificate that renews by itself on ACP". diff --git a/docs/en/solutions/Platform_console_does_not_open_because_the_cluster_VIP_domain_name_cannot_be_resolved_inside_the_cluster_on_ACP.md b/docs/en/solutions/Platform_console_does_not_open_because_the_cluster_VIP_domain_name_cannot_be_resolved_inside_the_cluster_on_ACP.md new file mode 100644 index 000000000..4858612f4 --- /dev/null +++ b/docs/en/solutions/Platform_console_does_not_open_because_the_cluster_VIP_domain_name_cannot_be_resolved_inside_the_cluster_on_ACP.md @@ -0,0 +1,110 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Platform console does not open because the cluster VIP domain name cannot be resolved inside the cluster on ACP + +## Issue + +The ACP console cannot be opened, although every platform pod on the `global` cluster is `Running` and the API server certificates are not expired. Restarting platform components does not help. Platform components that proxy requests to the cluster may log that the cluster is temporarily not accessible. + +The cluster's VIP (the cluster access address entered at installation) is a **domain name**, not an IP address. + +## Root Cause + +When the cluster VIP is configured as a domain name, platform components inside the cluster must resolve that name to reach the cluster. Name resolution inside the cluster works as follows: + +- Pods that use cluster DNS ask CoreDNS. CoreDNS answers from its own records (such as **Host Alias** entries) and forwards every other name to the DNS servers in the node's `/etc/resolv.conf`. +- Host-network components use the node's `/etc/resolv.conf` directly. + +If the DNS servers listed in the nodes' default `/etc/resolv.conf` cannot resolve the VIP domain name, and CoreDNS has no record for it, nothing inside the cluster can resolve the cluster address. The components keep running, but their requests to the cluster fail, so the console does not open. + +A browser on a workstation may still resolve the name (for example through a local hosts file or a different DNS server), which hides the problem from the outside. + +## Resolution + +Make the VIP domain name resolvable inside the cluster, in one of two ways. + +1. **Fix the node DNS (preferred).** Make sure the DNS servers in `/etc/resolv.conf` on every cluster node can resolve the VIP domain name: add the record on those DNS servers, or point the nodes to DNS servers that have it. This covers both host-network components and Pods, because CoreDNS forwards to the same servers. + + After the change, check on each node: + + ```bash + cat /etc/resolv.conf + nslookup + ``` + + If NetworkManager or another tool generates `/etc/resolv.conf`, make the change there (for example `nmcli con mod ipv4.dns ""`), so it survives a reboot. + + CoreDNS Pods normally run with `dnsPolicy: Default` and copy the node's resolver configuration only when they are created, so they keep forwarding to the old servers until they are recreated. Check the policy, and restart CoreDNS after every node has been changed: + + > **Warning:** Restarting CoreDNS briefly interrupts DNS for Pods in the cluster while the new Pods start. Do it only after `nslookup ` works on every node. No rollback is needed for the restart itself. + + ```bash + kubectl -n kube-system get deploy | grep -i coredns + kubectl -n kube-system get deploy -o jsonpath='{.spec.template.spec.dnsPolicy}{"\n"}' + kubectl -n kube-system rollout restart deployment/ + kubectl -n kube-system rollout status deployment/ + ``` + +2. **Add a static record to CoreDNS.** If the node DNS servers cannot be changed, add the record to the cluster DNS plugin: + + 1. Open **Administrator** > **Marketplace** > **Cluster Plugins**, find **Alauda Container Platform Cluster DNS** and click **Update**. + 2. Under **Host Alias**, enter the VIP address as **IP** and the VIP domain name under **Domains**, then click **Update**. + + If the console cannot be opened at all, add the record from the command line on a `global` control plane node as an emergency measure, in hosts-file format, to the `hosts` key of the ConfigMap `cpaas-coredns` in `kube-system`. + + > **Warning:** `cpaas-coredns` drives DNS for every Pod in the cluster, and a syntax error in it can break all in-cluster name resolution. Do this only when step 2 of the Diagnostic Steps shows that the node DNS cannot resolve the VIP name and option 1 is not possible. Keep every existing line of `hosts` and only append the new one. To roll back, run `kubectl apply -f cpaas-coredns.before.yaml`. + + ```bash + kubectl -n kube-system get cm cpaas-coredns -o yaml > cpaas-coredns.before.yaml + kubectl -n kube-system edit cm cpaas-coredns + ``` + + ```yaml + data: + hosts: | + + + ``` + + A manual edit of this ConfigMap can be overwritten when the plugin is reconciled. Once the console opens, enter the same record under **Host Alias** so that it persists. A CoreDNS record only helps Pods that use cluster DNS. Host-network components still depend on the node DNS (option 1). + +3. Reload the console in the browser and confirm it opens. + +## Diagnostic Steps + +1. Confirm that the cluster VIP is a domain name. Find the address the cluster was installed with, for example in the `server:` field of the kubeconfig used on the control plane nodes: + + ```bash + grep 'server:' ~/.kube/config /etc/kubernetes/*.conf 2>/dev/null + ``` + +2. On a cluster node, resolve the name through the node's DNS servers. `NXDOMAIN`, `server can't find` or a timeout matches this issue: + + ```bash + nslookup + for ns in $(awk '/^nameserver/{print $2}' /etc/resolv.conf); do nslookup $ns; done + ``` + +3. Resolve the name from inside a Pod that has `nslookup` (or `getent`). `unknown host` or `can't resolve` confirms that cluster DNS cannot resolve it either: + + ```bash + kubectl -n exec -- nslookup + ``` + +4. Rule out an unreachable VIP: if the name resolves but the address does not answer, the problem is the VIP itself, not DNS: + + ```bash + curl -k -m 5 https://:6443/healthz + ``` + +5. Rule out a look-alike: if the VIP domain resolves inside Pods and the `tke-platform-api` log on `global` contains `x509` errors for `localhost:9443`, the console failure comes from an expired certificate held by a long-running `tke-platform-api`, not from DNS: + + ```bash + kubectl -n cpaas-system logs deploy/tke-platform-api --tail=500 | grep -E 'localhost:9443|certificate has expired' + ``` diff --git a/docs/en/solutions/Platform_health_page_shows_components_abnormal_while_their_Pods_are_Running_on_ACP.md b/docs/en/solutions/Platform_health_page_shows_components_abnormal_while_their_Pods_are_Running_on_ACP.md new file mode 100644 index 000000000..1273324af --- /dev/null +++ b/docs/en/solutions/Platform_health_page_shows_components_abnormal_while_their_Pods_are_Running_on_ACP.md @@ -0,0 +1,96 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.16" + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Platform health page shows components abnormal while their Pods are Running on ACP + +## Issue + +The platform health page or the cluster status marks components as abnormal, but their Pods are `Running`: + +- **Platform Health Status** marks the `global` cluster unhealthy with the banner `平台监控功能故障,可能会影响集群(global)的功能健康检查结果的准确性,请尽快修复` (platform monitoring failure may affect the accuracy of the health check results of the global cluster; please fix it). Most monitoring components (`courier`, `courier-api`, `kube-prometheus-exporter-*`, `vmagent-agent`, `vmalertmanager-alertmanager`, `vminsert-cluster`, `vmselect-cluster`, `vmstorage-cluster`) are shown faulty with connectivity `否` (no) although their replicas are complete. The page can also raise false Pod-restart alerts. +- The cluster status shows the `coredns` component unhealthy. + +Related article: [Cluster plugins stay Deploying or AppReleases stall because sentry is OOM, hung, or CPU-bound on ACP](Cluster_plugins_stay_Deploying_or_AppReleases_stall_because_sentry_is_OOM_hung_or_CPU_bound_on_ACP.md). + +## Root Cause + +### Cause 1: monitoring data is backed up, so the health records are not refreshed + +The health page reads `modulehealthrecords`, and the module health query uses the metric `cpaas_module_health`. When `vmstorage` is CPU-throttled, monitoring data backs up and has gaps, and the health check keeps reading historical fault data, so the `modulehealthrecord` status is not refreshed. In VMUI `cpaas_module_health` stays flat for hours. The marked Pods are `Running`, their Endpoints are populated, and direct requests to their Service IPs are answered. Recreating `courier` and `courier-api` does not refresh the page. + +When the backlog is released all at once, a burst of alerts can fire; for example `courier-api` sends `workload.pod.restarted.count` notifications that a DingTalk robot rejects with `errcode 660026`, `"errmsg":"sending too many messages per minute"`. The alerts recover after some time, and the health page returns to normal hours later. + +`vmstorage` CPU also rises when it merges parts, under heavy `vmselect` query or `vminsert` write load, and with high disk I/O wait (for example a volume on a slow SAN). + +### Cause 2: a ResourcePatch with spec.delete true removed the Service used by the coredns health check + +The cluster `modulehealth` object names a Service for the `coredns` health check. On the affected cluster, `kubectl get rpch` lists a ResourcePatch in phase `Active` for release `cpaas-system/kube-prometheus`, kind `Service`, namespace `kube-system`, target `kube-prometheus-exporter-coredns`, and its YAML (`operator.alauda.io/v1alpha1`, `ResourcePatch`) has `spec.delete: true`. That patch removed the Service, so the health check cannot collect the status. Restoring the configuration brings the monitoring target back and clears the unhealthy mark. + +## Resolution + +### Cause 1: monitoring data is backed up, so the health records are not refreshed + +1. Confirm that the components are reachable and the data is stale: + + ```bash + kubectl get modulehealthrecords -A + kubectl get ep -A | grep -E 'vm|courier' + kubectl top pod -A | grep -E 'vmstorage|vmagent' + kubectl logs -n cpaas-system deploy/courier-api --since=2h | grep -E 'errcode|sending too many messages' | tail + ``` + + In VMUI, query `cpaas_module_health` over the last 24 hours. A flat series while the page is red, populated Endpoints, and answers from the marked Services match this cause. + +2. Raise the `vmstorage` CPU limit and the `vmagent` memory limit. + + > **Warning:** Higher limits let these Pods consume more of the node. Do this only when `vmstorage` CPU is throttled and the marked components answer directly. Export the object that holds the limits first; applying the export is the rollback. + +3. Watch the health page over the following days. If the `vmstorage` volume has poor I/O, plan to move it to faster storage. + +### Cause 2: a ResourcePatch with spec.delete true removed the Service used by the coredns health check + +1. Find the Service and the ResourcePatch: + + ```bash + kubectl get modulehealth -A -o yaml + kubectl get svc -n kube-system kube-prometheus-exporter-coredns + kubectl get rpch -A + kubectl get rpch -A -o yaml | grep -B10 -A10 'kube-prometheus-exporter-coredns' + ``` + +2. Back up the ResourcePatch, then delete it or edit it so that it no longer deletes the Service. + + > **Warning:** This changes what the `kube-prometheus` release keeps in `kube-system`. Do it only when the `coredns` health mark is abnormal, the Service is missing, and this ResourcePatch has `spec.delete: true` against it. To roll back, apply the export, which deletes the Service again. + + ```bash + kubectl get rpch -o yaml > "$HOME/coredns-exporter-rpch.yaml" + kubectl delete rpch + ``` + +3. Confirm that the Service exists again. If it is still missing, compare with a cluster whose `coredns` check is healthy and restore the same Service. The monitoring target comes back and the `coredns` mark clears. + +## Diagnostic Steps + +1. For a red health page, compare the page with the Pods and Endpoints, and query `cpaas_module_health` in VMUI. Expected for Cause 1: Pods `Running`, Endpoints populated, and a flat series. + +2. For an unhealthy `coredns` mark: + + ```bash + kubectl get svc -n kube-system kube-prometheus-exporter-coredns + kubectl get rpch -A | grep -i coredns + ``` + + Expected for Cause 2: `NotFound` for the Service and an `Active` ResourcePatch with `spec.delete: true` targeting it. diff --git a/docs/en/solutions/Platform_log_Kafka_brokers_fail_no_leader_election_exit_137_Pending_on_DiskPressure_Ceph_write_errors_or_OOM_on_ACP.md b/docs/en/solutions/Platform_log_Kafka_brokers_fail_no_leader_election_exit_137_Pending_on_DiskPressure_Ceph_write_errors_or_OOM_on_ACP.md new file mode 100644 index 000000000..acb548515 --- /dev/null +++ b/docs/en/solutions/Platform_log_Kafka_brokers_fail_no_leader_election_exit_137_Pending_on_DiskPressure_Ceph_write_errors_or_OOM_on_ACP.md @@ -0,0 +1,231 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.14.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Platform log Kafka brokers fail: no leader election, exit 137, Pending on DiskPressure, Ceph write errors, or OOM on ACP + +## Issue + +The platform log Kafka (`cpaas-kafka-*` with `cpaas-zookeeper-*` in `cpaas-system`) fails as a broker, not as a slow consumer. Typical reports: + +- `lanaya` cannot connect to Kafka, and the broker log says a leader election failed. +- `cpaas-kafka-1` and `cpaas-kafka-2` are in `Error` with tens of thousands of restarts and exit code `137`; adding replicas does not help. +- A `cpaas-kafka` pod is `Pending` and log delivery has stopped. +- Observability shows no logs; the logging pods start healthy and later go abnormal, and the Kafka log shows data write failures on a Ceph-backed volume. +- The broker log shows `java.lang.OutOfMemoryError: Java heap space` or `Direct buffer memory`, logs pile up, and `razor` logs `Send kafka msg error` with `context deadline exceeded` or `kafka write errors`. + +This article is only for the platform log pipeline. A Data Services Kafka instance used by applications is a different workload; do not apply the wipe, `Pending`, or metadata branches here to it. + +Related articles: [cpaas-kafka broker fails to start: corrupt segment, no space, broker id conflict, or deleted StatefulSet on ACP](cpaas-kafka_broker_fails_to_start_corrupt_segment_no_space_broker_id_conflict_or_deleted_StatefulSet_on_ACP.md), [Kafka and ZooKeeper down from storage stalls, a full node disk, or a probe kill during log replay on ACP](Kafka_and_ZooKeeper_down_from_storage_stalls_a_full_node_disk_or_a_probe_kill_during_log_replay_on_ACP.md), [Logs, audit, and events stop while the log Kafka is degraded on ACP](Logs_audit_and_events_stop_while_the_log_Kafka_is_degraded_on_ACP.md), and [cpaas-kafka retention hours, memory limit above 80 percent, or KAFKA_HEAP_OPTS not applied on ACP](cpaas-kafka_retention_hours_memory_limit_above_80_percent_or_KAFKA_HEAP_OPTS_not_applied_on_ACP.md). + +## Root Cause + +### Cause 1: the brokers cannot elect a leader because the Kafka metadata is corrupt + +The brokers are up enough to log, and the log says a leader election failed because metadata is corrupt. A normal restart does not elect a leader. Clearing the Kafka data directory restores elections. Lag and a large on-disk queue of the `nevermore` log agent under `/cpaas` (the `audit`, `event`, and `log` trees and their `diskqueue` directories) can remain afterwards; clearing those trees on every node and restarting `nevermore` drains it, at the cost of every record that had not been shipped yet. + +### Cause 2: cpaas-kafka pods restart with exit code 137 and extra replicas still crash + +Exit code `137` means the container received `SIGKILL`: either the kernel OOM killer (`OOMKilled` in the container's last state) or the kubelet after a failed liveness probe (a `Liveness probe failed` event). A failing or full disk does not produce 137 directly, but it can slow startup until the probe kills the container. When the loop persists after disk and memory are checked and scaling out has failed, the recovery that works is a full rebuild of the log cluster: stop the writers and the stores, clear the Elasticsearch, Kafka, and ZooKeeper data directories, and start the stores in order. Every stored log and the ZooKeeper state of the log cluster are lost. + +A broker that is killed by its probe while it is still loading segments is the narrower replay case in the Kafka and ZooKeeper article linked above; that recovery does not delete Elasticsearch data. + +### Cause 3: the Kafka pod is Pending because its node has DiskPressure + +Disk usage crossed the kubelet eviction threshold, the Kafka pod was evicted, and it stays `Pending`, so log delivery stops. Expanding or freeing the disk and then deleting the pod lets it schedule again. The same `Pending` state clears after `NotReady` nodes recover; restarting pods does not fix a `NotReady` node. + +### Cause 4: Kafka data writes fail on a Ceph-backed volume + +Kafka logs from before a restart show data write failures, and the Kafka PVC is on Ceph. Expanding that Kafka volume brings Kafka back, and log display returns (one case: ACP 4.2.1 on x86 with MicroOS nodes). + +### Cause 5: cpaas-kafka runs out of JVM heap or direct buffer memory + +```text +java.lang.OutOfMemoryError: Java heap space +ERROR kafka.network.Processor - Processor got uncaught exception. +java.lang.OutOfMemoryError: Direct buffer memory +``` + +`ReplicaFetcherThread` also logs `Error sending fetch request ... java.lang.OutOfMemoryError: Java heap space`. Log volume grew beyond what the Kafka configuration can carry, the broker JVM ran out of memory, and Kafka stopped processing log data. `nevermore` builds up a local backlog meanwhile; when Kafka recovers it sends the backlog in a burst, and `apollo` on the global cluster can hit OOM and restart, which partly affects the console. + +Raising only the container resources does not stop the OOM; `lanaya` can then also go OOM. Raising the broker heap (`KAFKA_HEAP_OPTS`) stops it. Check the running JVM afterwards: on some images the start script sets its own heap, and the environment value never reaches the broker (see the retention and heap article linked above). + +## Resolution + +Some of the workloads below are Deployments on one version and StatefulSets on another. If `kubectl scale` rejects the kind, use the kind printed by `kubectl -n cpaas-system get deploy,sts`. + +### Cause 1: the brokers cannot elect a leader because the Kafka metadata is corrupt + +1. Stop the writer and the brokers: + + ```bash + kubectl -n cpaas-system scale deploy/lanaya --replicas=0 + kubectl -n cpaas-system scale sts/cpaas-kafka --replicas=0 + ``` + +2. Clear the Kafka data directory on each broker node. + + > **Warning:** Clearing the Kafka data directory destroys every record on that broker; consumers lose the backlog. Precondition: only when the broker log shows a failed leader election or corrupt metadata, a normal restart did not elect a leader, both workloads above are at 0, `lsof` shows no Kafka process in the directory, and the path is the Kafka volume (often `/cpaas/data/kafka0`), not ZooKeeper. Rollback: none. + + Delete the contents, not the mount directory itself: + + ```bash + rm -rf /* + ``` + +3. Scale `cpaas-kafka` back to its previous replica count, wait until a leader is elected, then scale `lanaya` back. +4. Only if lag remains and the `nevermore` queue trees under `/cpaas` are large, clear them and restart `nevermore`. + + > **Warning:** Clearing `/cpaas/audit`, `/cpaas/event`, and `/cpaas/log` drops every record that never reached Kafka, and also removes the agent's own logs under `/cpaas/log`. Precondition: Kafka elects leaders again, and `du` shows these agent trees, not the Elasticsearch data directory. Rollback: none; `nevermore` recreates empty queues. + + ```bash + du -sh /cpaas/audit /cpaas/event /cpaas/log + rm -rf /cpaas/audit/* /cpaas/event/* /cpaas/log/* + kubectl -n cpaas-system rollout restart ds/nevermore + ``` + +### Cause 2: cpaas-kafka pods restart with exit code 137 and extra replicas still crash + +1. Find who sends the kill before you delete anything: + + ```bash + kubectl -n cpaas-system describe pod cpaas-kafka-1 | grep -E -A3 'Last State|Liveness probe failed' + df -h /cpaas /cpaas/data + dmesg -T | grep -iE 'oom|i/o error' | tail + ``` + +2. Record the replica counts: + + ```bash + kubectl -n cpaas-system get deploy,sts | grep -E 'lanaya|devops-controller|courier|razor|cpaas-kibana|cpaas-elasticsearch|cpaas-kafka|cpaas-zookeeper' + ``` + +3. Stop the writers and the stores (skip names that do not exist, such as `cpaas-elasticsearch-master` on a small-scale layout). + + > **Warning:** This stops log collection, log search, Kibana, and the inspection queries served by `courier` for the whole cluster, and the next step deletes every stored log, every unconsumed Kafka record, and the ZooKeeper state of the log cluster. Precondition: exit 137 persists after the disk and memory checks, scaling out already failed, and the owner accepts a full rebuild of the log cluster. Rollback: none for the data. + + ```bash + kubectl -n cpaas-system scale deploy/lanaya deploy/devops-controller deploy/courier deploy/razor deploy/cpaas-kibana --replicas=0 + kubectl -n cpaas-system scale sts/cpaas-elasticsearch sts/cpaas-elasticsearch-master sts/cpaas-kafka sts/cpaas-zookeeper --replicas=0 + ``` + +4. When every pod is gone, on each data node record owner and mode, then clear the contents of each data directory (take the paths from the pods' volume mounts; never a directory that also holds etcd or application data): + + ```bash + stat -c '%U:%G %a %n' + rm -rf /* /* /* + ``` + + Recreate any directory you removed with the same owner and mode. + +5. Start back to the recorded counts, waiting for each to be Ready: ZooKeeper, Kafka, Elasticsearch master, Elasticsearch data, then `cpaas-kibana`, `razor`, `courier`, `lanaya`, and `devops-controller`. The order of the stateless components after the stores has not been shown to matter. +6. Query a pod that is logging now; older records were deleted on purpose. + +### Cause 3: the Kafka pod is Pending because its node has DiskPressure + +```bash +kubectl describe node | grep -E 'DiskPressure|MemoryPressure|Ready' +kubectl -n cpaas-system describe pod cpaas-kafka-0 | sed -n '/Events:/,$p' +``` + +Free or extend the filesystem that `df` shows full. When `DiskPressure` is `False` and the node is `Ready`: + +> **Warning:** Deleting the Kafka pod interrupts produce and consume until the StatefulSet recreates it. Precondition: the node condition that kept it `Pending` is gone. Rollback: none needed; the StatefulSet recreates the pod. + +```bash +kubectl -n cpaas-system delete pod cpaas-kafka-0 +``` + +Delete Elasticsearch pods the same way only if they are also stuck on that condition. Do not clear data directories for a `Pending` pod. If the node is `NotReady`, repair the node; the pods become Ready once it is. + +### Cause 4: Kafka data writes fail on a Ceph-backed volume + +```bash +kubectl -n cpaas-system get pvc | grep -i kafka +kubectl -n cpaas-system logs cpaas-kafka-0 --tail=80 +kubectl -n cpaas-system exec cpaas-kafka-0 -- df -h +``` + +A hostPath or node partition such as `/cpaas/data/kafka0` is not this procedure; grow that filesystem on the node instead. + +> **Warning:** A PVC expansion generally cannot be shrunk afterwards; topic data is not deleted by it. Precondition: only when the broker log shows write failures and the Kafka PVC is Ceph-backed. Pick `` larger than the current capacity. Rollback: none (no shrink). If the StorageClass does not allow expansion, expand the volume from the storage side, and restart the broker pod only if it does not see the new space. + +```bash +kubectl -n cpaas-system patch pvc --type merge \ + -p '{"spec":{"resources":{"requests":{"storage":""}}}}' +``` + +When the PVC and `cpaas-kafka` pods are healthy, query a pod that is logging now. If the `logcenter` ModuleInfo declares a Kafka storage size (`spec.config.components.kafka.storageSize`), set it to the same value. + +### Cause 5: cpaas-kafka runs out of JVM heap or direct buffer memory + +1. Confirm the error and read the current settings: + + ```bash + kubectl -n cpaas-system logs cpaas-kafka-0 --tail=200 | grep -E 'OutOfMemoryError|Direct buffer memory' + kubectl -n cpaas-system get sts cpaas-kafka -o yaml | grep -E -A2 'KAFKA_HEAP_OPTS|resources:' + kubectl -n cpaas-system exec cpaas-kafka-0 -- ps -ef | grep '[j]ava' | grep -o -- '-Xm[sx][^ ]*' + ``` + +2. Raise the Kafka resources through the platform chart override on the global cluster, so a reconcile does not revert them. The platform applies the change in about 5 to 10 minutes. + + > **Warning:** Editing `prdb base` changes platform-level chart overrides on the global cluster, and the new Kafka resources roll the `cpaas-kafka` pods; produce and consume pause on each broker while it restarts. Precondition: only when the broker log shows the `OutOfMemoryError` lines above, you saved the object, and the nodes have the memory for the new limit. Rollback: `kubectl apply -f /tmp/prdb-base.before.yaml`. + + ```bash + kubectl get prdb base -o yaml > /tmp/prdb-base.before.yaml + kubectl edit prdb base + ``` + + One working override (keep any keys already under `valuesOverride`): + + ```yaml + spec: + valuesOverride: + ait/chart-kafka-zookeeper: + cpaasKafka: + resources: + requests: + cpu: 200m + memory: 256Mi + limits: + cpu: "2" + memory: 4Gi + kafka: + jvm_size: -Xmx1g -Xms512m + ``` + +3. If the OOM continues, raise the broker heap through the same override path. Keep the heap well below the container memory limit, because direct buffers and the page cache live outside the heap. +4. Confirm that the running broker JVM uses the new heap: the `-Xmx` of the Java process that loads `kafka_server_jaas.conf` must match. If it stays at `-Xmx1G` while the environment shows the new value, the start script overrides it; follow the heap cause in the retention and heap article linked above. +5. Watch `sum(kafka_consumergroup_lag{})` until it trends down, and watch `lanaya` and `apollo` for OOM restarts while the `nevermore` backlog drains. + +## Diagnostic Steps + +1. Broker state and the last fatal log: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep -E 'kafka|zookeeper|lanaya|razor' + kubectl -n cpaas-system logs -p cpaas-kafka-0 --tail=50 + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | Leader election fails, metadata corrupt | 1 | + | Exit code `137` in a tight loop, scale-out does not help | 2 | + | Pod `Pending`, node `DiskPressure` or `NotReady` | 3 | + | Write failures in the broker log, Ceph-backed Kafka PVC | 4 | + | `OutOfMemoryError: Java heap space` or `Direct buffer memory` | 5 | + +3. Distinguish the two kinds of exit 137: `Reason: OOMKilled` in `Last State` is memory; a `Liveness probe failed` event with `Reason: Error` is the probe. +4. After any wipe, check a pod that is logging right now, not a historical time range. diff --git a/docs/en/solutions/Platform_registry_pull_fails_with_authorization_failed_or_insufficient_scope_auth.yaml_targets_or_node_login.md b/docs/en/solutions/Platform_registry_pull_fails_with_authorization_failed_or_insufficient_scope_auth.yaml_targets_or_node_login.md new file mode 100644 index 000000000..c6dc8ab17 --- /dev/null +++ b/docs/en/solutions/Platform_registry_pull_fails_with_authorization_failed_or_insufficient_scope_auth.yaml_targets_or_node_login.md @@ -0,0 +1,133 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Platform registry pull fails with authorization failed or insufficient_scope: auth.yaml targets or node login + +## Issue + +Nodes reach the platform image registry (the built-in registry of the global cluster, often addressed as `:11443`), but pulls are denied: + +- After the core environment is upgraded, custom-built images stored under the `/solution` path fail to pull. +- Plugin images newly added to the built-in registry on port `11443` fail to pull in Pods and with `crictl pull` on the node, with `authorization failed`. +- Images uploaded to the platform registry cannot be pulled in a business cluster, although they are present in the registry storage: + + ```text + PullImage from image service failed ... pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed + ``` + +Related articles: [Registry push fails with insufficient_scope or authentication required on ACP](Registry_push_fails_with_insufficient_scope_or_authentication_required_on_ACP.md). + +## Root Cause + +The platform registry authorizes requests with `/etc/kubernetes/registry/auth.yaml` on the global control-plane nodes. Each user entry grants `actions` (`pull`, `push`, `delete`) on one or more `target` patterns (regular expressions when `useRegexp: true`). A node that presents no credential is handled by the `_anonymous` entry, which by default grants `pull` only on a few repositories (for example `tkestack/.*` and `windows/.*`). + +### Cause 1: the user's pull target does not include the solution repository + +The user's `pull` entry lists the platform prefixes only, for example `target: (acp|ait|devops|asm|alauda|3rdparty|ops|middleware|automation|frontend|developer\-platform)/.*` with `useRegexp: true`, and the `_anonymous` entry covers only `tkestack/.*` and `windows/.*`. Neither matches `solution/...`, so pulls of custom-built images under `/solution` are denied. Adding `solution` to the user's `target` and restarting the registry fixes it. + +### Cause 2: the repository of newly added images has no anonymous pull entry + +Node pulls without an image pull secret go through the `_anonymous` entry. When new images are added to a repository that no anonymous `target` covers (for example `3rdparty`), the pulls fail with `authorization failed`. Adding an anonymous pull target for that repository in `auth.yaml` is the fix. + +### Cause 3: the business-cluster node has no valid credential for the repository + +On a business-cluster node, `crictl pull :11443//:` returns `insufficient_scope: authorization failed`, while DNS resolution and TCP to port `11443` work and the image is present in the registry storage (MinIO under `/cpaas/minio` on each global control-plane node). Adding an anonymous pull entry to `auth.yaml` does not help. Logging the node in to the registry again with `nerdctl -n k8s.io login :11443 --insecure-registry` makes the pulls work. The underlying mechanism has not been identified. + +## Resolution + +Before any edit of `auth.yaml`: the file exists on every global control-plane node, and all copies must stay identical. Compare them first: + +```bash +md5sum /etc/kubernetes/registry/auth.yaml +``` + +### Cause 1 + +1. Find the entry of the affected user: + + ```bash + grep -n -B3 -A8 'useRegexp' /etc/kubernetes/registry/auth.yaml + ``` + +2. Add `solution` to the user's `target` alternation, in the same form as the other prefixes, for example `(acp|ait|...|developer\-platform|solution)/.*`, then restart the registry so it reads the file. + + > **Warning:** Only when the failing images are under `/solution` and no `target` of the user or of `_anonymous` matches `solution/`. Impact: a wrong edit can deny pulls of platform images to every cluster that uses this registry, and the restart interrupts pushes and pulls until the registry is Ready. Precondition: a copy of the file on each node, only the `target` of the affected user changes, and every global control-plane node gets the same change. Rollback: restore the copies and restart the registry again. + + ```bash + cp -a /etc/kubernetes/registry/auth.yaml /root/auth.yaml.bak + vi /etc/kubernetes/registry/auth.yaml + kubectl get pods -A | grep -i registry + kubectl rollout restart deploy/ -n + kubectl rollout status deploy/ -n + ``` + +3. Retry the pull of a custom-built image under `/solution`. A pull from another repository is not the check. + +### Cause 2 + +1. Read the `_anonymous` entry on each global control-plane node: + + ```bash + grep -n -A12 '_anonymous' /etc/kubernetes/registry/auth.yaml + ``` + +2. Add an anonymous pull target for the failing repository in the same format as the existing ones (for example a target `3rdparty/.*` with action `pull` next to `tkestack/.*`), change nothing else, and restart the registry as in Cause 1. + + > **Warning:** Only when the failing repository is not covered by any anonymous target and the pulls are made without an image pull secret. Impact: anyone who can reach the registry can then pull every image in that repository without credentials; a YAML error in the file can stop the registry from authenticating anyone. Precondition: the images in that repository may be readable by anyone who reaches the registry, a copy of the file is kept on each node, and every node gets the same edit. Rollback: restore the copies and restart the registry. + +3. Retry `crictl pull` of one of the new images on a node. If it still fails with `insufficient_scope`, continue with Cause 3. + +### Cause 3 + +1. Confirm the image is in the registry storage, on a global control-plane node: + + ```bash + du -sh /cpaas/minio + ls /cpaas/minio/*/registry/docker/registry/v2/repositories// + ``` + +2. From the failing node, confirm name resolution and the port, and reproduce the error: + + ```bash + nslookup + nc -vz -w 3 11443 + crictl pull :11443//: + ``` + +3. Log the node in to the registry again. + + > **Warning:** Only when the image is present, the port answers, and the pull returns `insufficient_scope: authorization failed`. Impact: `nerdctl login` stores the registry credential, base64-encoded, in the Docker config file of the user who runs it (`/root/.docker/config.json` for root), and the kubelet then sends it for pulls from that registry; `--insecure-registry` skips certificate verification for this login. Precondition: the account is one the registry owner approves for node pulls, and the existing config file is copied aside. Rollback: `nerdctl -n k8s.io logout :11443`, or restore the copied file. + + ```bash + cp -a /root/.docker/config.json /root/docker-config.json.bak 2>/dev/null + nerdctl -n k8s.io login :11443 --insecure-registry + crictl pull :11443//: + ``` + +4. Repeat on every node that must pull the image. If you added an anonymous entry to `auth.yaml` while testing and it is not part of the intended configuration, remove it again on every global control-plane node. + +## Diagnostic Steps + +1. Separate a permission denial from a missing tag: list the tags with an account that can read the repository. An empty list means the image is not there. + +2. Read the entries that could match the failing repository: + + ```bash + grep -n -B2 -A12 -E '_anonymous|useRegexp' /etc/kubernetes/registry/auth.yaml | grep -v -i password + ``` + + - Custom-built images under `/solution`, with no `target` matching `solution/`: Cause 1. + - A repository with newly added images, pulled without an image pull secret, with no anonymous `target` covering it: Cause 2. + +3. `insufficient_scope: authorization failed` on a business-cluster node, with the image present under `/cpaas/minio` and port `11443` reachable: Cause 3. Try the node login before editing `auth.yaml`. + +4. A `401` from Harbor rather than from the platform registry is covered in [Harbor image pull returns 401 because of the pull Secret, the project account, or node credentials](Harbor_image_pull_returns_401_because_of_the_pull_Secret_the_project_account_or_node_credentials.md). diff --git a/docs/en/solutions/Plugin_install_or_uninstall_stuck_or_Marketplace_and_OperatorHub_empty_sentry_OOMKilled_or_cluster-transformer_errors.md b/docs/en/solutions/Plugin_install_or_uninstall_stuck_or_Marketplace_and_OperatorHub_empty_sentry_OOMKilled_or_cluster-transformer_errors.md new file mode 100644 index 000000000..b1f09f97c --- /dev/null +++ b/docs/en/solutions/Plugin_install_or_uninstall_stuck_or_Marketplace_and_OperatorHub_empty_sentry_OOMKilled_or_cluster-transformer_errors.md @@ -0,0 +1,164 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Plugin install or uninstall stuck, or Marketplace and OperatorHub empty: sentry OOMKilled or cluster-transformer errors + +## Issue + +- Uninstalling the monitoring or logging plugin of a cluster hangs: the plugin stays in deleting and its Pods are not removed. Removing the `finalizers` from the ModuleInfo makes the entry vanish from the console, but the resources stay on the cluster. +- All functional components of a business cluster are abnormal with `ChartDownloadFailed [1/1] failed to download chart acp/chart-acp-business-v4.0.32, err: EOF`, although the business cluster reaches `global` on ports 443 and 11443. +- `kubectl get ars -n cpaas-system` shows `prometheus` with `ChartDownloadFailed` / `failed to download chart ... err: EOF`, or `logcenter` stuck in `Syncing` with `chart ... Installing`. +- The NeuVector plugin uninstall from the console stays in uninstalling and its Pods are not deleted. +- The `global` cluster UI shows the Alauda Container Platform Marketplace component as abnormal, while `kubectl get ars` looks normal. +- On `global`, the `alauda-operators` AppRelease reports `failed to resync chart for release "install-operators": execution error at (install-operators/templates/notes.yaml:14:8)`, and OperatorHub shows no operators. + +`sentry` (Deployment `sentry` in `cpaas-system` on each cluster) installs, upgrades, and removes plugin charts and records the result on the `ars` (AppRelease) objects. `cluster-transformer` on `global` turns ModulePlugins into ModuleConfigs and reconciles ModuleInfo (`minfo`) and ClusterModule. + +Related articles: "sentry at full CPU or a redeployed component fails because of Failed or leftover ResourcePatches on ACP". + +## Root Cause + +### Cause 1: sentry is OOMKilled or short of resources, so chart sync, install, and uninstall stop + +`kubectl describe pod` on the `sentry` Pod shows `Last State: Terminated`, `Reason: OOMKilled`, `Exit Code: 137`, with a growing restart count, and the AppReleases of the cluster stay in `Syncing` / `Installing` or report `ChartDownloadFailed ... err: EOF`. The `sentry` Pods can still show `1/1 Running`. + +- System plugin uninstall fails (for example on an ACP 3.12 workload cluster): `logcenter` is `Syncing` with `[1/1] chart ait/chart-alauda-log-center-v3.12.38 Installing`, `prometheus` is `ChartDownloadFailed` with `[2/2] failed to download chart ait/chart-kube-prometheus-v3.12.9, err: EOF`, and the `sentry` container restarts with `OOMKilled` under a `4Gi` memory limit. Raising the memory limit and restarting `sentry` lets the AppReleases sync and the uninstall complete. +- A monitoring uninstall is stuck with the Prometheus ModuleInfo in deleting. Removing its `finalizers` removes the component from the console only; the backend resources stay. With `sentry` given more memory, deploying the component again from the console and uninstalling it again completes the uninstall. Removing ModuleInfo finalizers does not perform an uninstall. +- Every functional component of a business cluster is abnormal (`ChartDownloadFailed ... err: EOF`) while the network to `global` is fine: `sentry` does not have enough resources to reconcile every component at the same time. Increasing its resources returns the AppReleases to normal. + +### Cause 2: A plugin uninstall stays in uninstalling with no identified cause (NeuVector) + +The NeuVector plugin uninstall from the console stays in uninstalling, and the Pods in `neuvector` are not deleted. No underlying cause has been identified. The uninstall is finished by deleting the plugin's ModuleInfo, then the resources in the `neuvector` namespace and the namespace itself. Because deleting the ModuleInfo alone only clears the console entry (Cause 1), the namespace contents are removed by hand. + +### Cause 3: Marketplace component abnormal, get registry client is nil + +The Marketplace component shows abnormal on `global`, AppReleases look normal, and the `cluster-transformer` log repeats: + +```text +ModuleInfo Reconcile ... error: ... error calling imageExist: get registry client is nil +get moduleconfig structure error: template: :34:16: executing "" at : error calling imageExist: get registry client is nil +ClusterModule Process '' error: handle plugin marketplace failed: ClusterModule need retry +``` + +with the same `need retry` for `auth-manager`, `acp-business`, and `sentry`, and `ProcessPlugin ... get render data metallb error: ... get registry client is nil`. `cluster-transformer` cannot reach the image registry, so the `imageExist` check in the plugin templates fails and ModuleInfo / ClusterModule processing stops. Restoring network connectivity to the registry returns the component to normal. + +### Cause 4: failed to resync chart for release "install-operators" and OperatorHub on global is empty + +```text +failed to resync chart for release "install-operators": execution error at (install-operators/templates/notes.yaml:14:8) +``` + +appears on the `alauda-operators` AppRelease of `global`, and OperatorHub in the App Marketplace lists no operators. No underlying cause has been identified. Restarting `cluster-transformer` and `sentry` restores service. + +## Resolution + +### Cause 1: sentry is OOMKilled or short of resources, so chart sync, install, and uninstall stop + +On the affected cluster: + +```bash +kubectl get ars -n cpaas-system +kubectl -n cpaas-system get pods | grep sentry +kubectl -n cpaas-system describe pod | grep -E -A6 'Last State|Limits|Requests' +kubectl -n cpaas-system get deploy sentry -o yaml > "$HOME/sentry.before.yaml" +``` + +> **Warning:** Changing the `sentry` resources restarts it; chart installs, upgrades, and uninstalls on the cluster pause until the new Pod is Ready, and a higher request reserves more node memory. Precondition: only when the `sentry` container's last state is `OOMKilled` (or it cannot keep up with every AppRelease) and AppReleases are stuck in `Syncing`, `Installing`, or `ChartDownloadFailed ... EOF`. Rollback: `kubectl apply -f "$HOME/sentry.before.yaml"`. + +```bash +kubectl -n cpaas-system set resources deploy sentry --limits=memory= +kubectl -n cpaas-system rollout status deploy sentry +``` + +Choose a limit above the one the container was killed at. After a few minutes, check that `kubectl -n cpaas-system get deploy sentry -o jsonpath='{.spec.template.spec.containers[*].resources}'` still shows the new limit; if it reverted, ask Alauda support where the `sentry` resources are configured on your version. Then watch the AppReleases: + +```bash +kubectl get ars -n cpaas-system -w +``` + +For a stuck uninstall whose ModuleInfo finalizers were already removed while the resources are still on the cluster: deploy the plugin again from the console, wait until it is installed, and uninstall it again. Do not remove ModuleInfo finalizers as a way to uninstall while the plugin's resources are still on the cluster. + +One finalizer removal is valid: the Log Agent plugin stays in Uninstalling, its workloads (`nevermore` and `kubernetes-event-exporter`) are already deleted from that cluster, and only the logagent `minfo` remains, in Deleting, with `finalizers: - moduleinfo`. There, removing the finalizer on `global` finishes the uninstall. + +> **Warning:** Removing `finalizers` lets the ModuleInfo be deleted without the controller's cleanup, and anything that cleanup would have removed stays on the cluster. Precondition: only with the exact Log Agent signature above. Save the `minfo` YAML first. Rollback: none for the deleted object; deploy the plugin again from the console if it was removed by mistake. + +```bash +kubectl get minfo -o yaml > "$HOME/.yaml" +kubectl patch minfo --type=merge -p '{"metadata":{"finalizers":null}}' +``` + +### Cause 2: A plugin uninstall stays in uninstalling with no identified cause (NeuVector) + +Rule out Cause 1 first: `sentry` must be Running without `OOMKilled` restarts; otherwise fix it and retry the uninstall from the console. Then look at what is left: + +```bash +kubectl get minfo | grep -i neuvector +kubectl -n neuvector get all +kubectl get all,cm,secret,pvc -n neuvector -o yaml > "$HOME/neuvector-namespace.yaml" +``` + +> **Warning:** This removes NeuVector from the cluster without the plugin's own cleanup; container security policies and NeuVector data in that namespace are deleted. Irreversible apart from reinstalling the plugin. Precondition: only when the console uninstall of NeuVector is stuck in uninstalling, `sentry` is healthy, and the export above is written. Rollback: install the NeuVector plugin again from Cluster Plugins. + +```bash +kubectl delete minfo +kubectl -n neuvector delete all --all +kubectl delete namespace neuvector +kubectl get namespace neuvector +``` + +If the namespace stays in `Terminating`, list what is left with `kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -n neuvector` and delete those objects. + +### Cause 3: Marketplace component abnormal, get registry client is nil + +```bash +kubectl -n cpaas-system logs deploy/cluster-transformer --tail=500 | grep -E 'registry client is nil|need retry' +``` + +Take the registry address from Clusters > `global` > Overview > Basic Info > Image Repository Address, and test it from the `global` control-plane nodes: + +```bash +curl -k -s -o /dev/null -w '%{http_code}\n' https:///v2/ +``` + +`000` or a timeout means the path is broken. Restore network connectivity from `global` to the registry (routing, firewall, DNS, or the registry service itself). The `need retry` errors stop and the Marketplace component returns to normal once the registry is reachable. + +### Cause 4: failed to resync chart for release "install-operators" and OperatorHub on global is empty + +```bash +kubectl get ars -A | grep -i alauda-operators +kubectl -n describe ars alauda-operators | grep -i -A3 'resync' +``` + +> **Warning:** Restarting `cluster-transformer` and `sentry` pauses plugin processing and chart sync on `global` until the new Pods are Ready; installs and upgrades in flight wait. Precondition: only when `alauda-operators` shows the `install-operators` `notes.yaml` resync error and OperatorHub on `global` is empty. Rollback: none needed; the Deployments recreate the Pods. + +```bash +kubectl -n cpaas-system rollout restart deploy/cluster-transformer deploy/sentry +kubectl -n cpaas-system rollout status deploy/cluster-transformer +kubectl -n cpaas-system rollout status deploy/sentry +kubectl get ars -A | grep -i alauda-operators +``` + +## Diagnostic Steps + +```bash +kubectl get ars -A | grep -v -E 'Synced|Ready' +kubectl -n cpaas-system get pods | grep -E 'sentry|cluster-transformer' +kubectl -n cpaas-system describe pod | grep -A5 'Last State' +kubectl -n cpaas-system logs deploy/cluster-transformer --tail=300 | grep -E 'registry client is nil|need retry' +``` + +- `sentry` `OOMKilled` (exit code `137`) with AppReleases stuck or `ChartDownloadFailed ... EOF`, or a stuck uninstall: Cause 1. +- A NeuVector uninstall stuck while `sentry` is healthy: Cause 2. +- `get registry client is nil` from `cluster-transformer`: Cause 3. +- The `install-operators` `notes.yaml` resync error with an empty OperatorHub: Cause 4. +- One `sentry` Pod near its CPU limit, or `Failed` ResourcePatches: see the related ResourcePatch article. diff --git a/docs/en/solutions/Pod_or_Service_traffic_fails_on_some_nodes_because_of_the_node_kernel_OS_hypervisor_or_network_path_on_ACP.md b/docs/en/solutions/Pod_or_Service_traffic_fails_on_some_nodes_because_of_the_node_kernel_OS_hypervisor_or_network_path_on_ACP.md new file mode 100644 index 000000000..d21a208ae --- /dev/null +++ b/docs/en/solutions/Pod_or_Service_traffic_fails_on_some_nodes_because_of_the_node_kernel_OS_hypervisor_or_network_path_on_ACP.md @@ -0,0 +1,145 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.18" + - "4.0" + - "4.2" + - "4.3" +--- + +# Pod or Service traffic fails on some nodes because of the node kernel, OS, hypervisor or network path on ACP + +## Issue + +The report names the network plugin (Kube-OVN or Calico), but the CNI Pods are healthy or recover by themselves. Typical descriptions: + +- An Istio sidecar fails its startup probe because DNS lookups of `istiod` time out: `lookup istiod.istio-system.svc on :53: read udp :->:53: i/o timeout`. +- A load test fails a few times a day with `Internal Server Error` while no Pod restarts, and log collection drops to zero for a few seconds. +- One Pod of a Deployment cannot reach a ClusterIP (`curl -v` stays at `Trying :...`) while its sibling can. +- Kube-OVN components alert and restart together while the cluster looks fine, or the whole business cluster misbehaves for hours. +- Pods fail with `FailedCreatePodSandBox ... failed to reserve sandbox name`, and the kernel log repeats `unregister_netdevice: waiting for lo to become free`. +- A platform component cannot connect to a Service even though a test from inside the container works. +- Pods on one node cannot reach the apiserver Service while the host can, or newly added nodes behind an ALB return `504`. + +Each cause below lies in the node operating system, the kernel, the virtualization layer or the physical network, not in the CNI configuration. + +Related articles: "calico-node fails to start or Pods hang in ContainerCreating: dummy module, corrupted image, clock skew or x509 on ACP". + +## Root Cause + +### Cause 1: Kylin V10 kernel nf_conntrack defect drops NAT replies + +Kylin V10 kernels older than `4.19.90-52.33.v2207.ky10` have a connection-tracking defect, confirmed by the OS vendor, that breaks reverse NAT translation. Replies that need reverse NAT, such as DNS replies and Service traffic, are dropped by the kernel. It has appeared on `4.19.90-52.22.v2207.ky10.x86_64` and `4.19.90-24.4.v2101.ky10.aarch64`, in three forms: + +- Intermittent Istio sidecar startup failures for some microservices, with `istiod` lookups timing out and `citadelclient failed to sign CSR` and `sds failed to warm certificate` following. `nslookup` of the same name from another container on the node works. Restarting OVS on the node clears it only temporarily. +- Under heavy load, readiness probes fail for a moment and Pods are removed from the Service Endpoints, so the external load balancer has no backend. No Pod restarts. +- One Pod cannot reach a ClusterIP while a sibling can, with DNS, routes, NetworkPolicy, Endpoints and the OVN load balancer all correct. Restarting the Pod helps temporarily; after the kernel upgrade it stayed normal. + +### Cause 2: the node intermittently loses the apiserver VIP + +The node log shows lease renewals failing with `dial tcp :6443: connect: no route to host`. The VIP and its backends are healthy; the node itself cannot reach the VIP, goes `NotReady`, and the OVN components on it restart as a consequence. Continuous ping probes from nodes to the VIP show rounds of heavy or total packet loss and RTT spikes of seconds, aligned with `NodeNotReady` events. Uneven traffic graphs and gaps in node monitoring are side effects. The fault is in the underlying network between the nodes and the VIP. + +### Cause 3: unregister_netdevice: waiting for lo to become free + +A kernel network-namespace leak keeps the loopback device of old Pod sandboxes busy, so those sandboxes are never torn down. On the affected nodes: + +- Platform Pods show `FailedCreatePodSandBox: ... context deadline exceeded`, then repeat `failed to reserve sandbox name "___0": name ... is reserved for ""`. +- Fixed-IP workloads fail with `AcquireAddressFailed` / `NoAvailableAddress`, and `kube-ovn-controller` logs `static address has been assigned to /`, because the old Pods holding the fixed IPs cannot be removed. +- `dmesg` keeps printing `unregister_netdevice: waiting for lo to become free. Usage count = `. + +The OS vendor (Kylin) confirmed it as a known kernel issue. Restarting `ovn-controller` does not help; nodes without the message are unaffected. + +### Cause 4: the VM source/destination check is enabled on the virtualization platform + +The hypervisor drops traffic whose source or destination is a Pod address when the source/destination check (anti-spoofing) is enabled on the node VMs. A platform component that needs a Service fails to start while a test from inside its container succeeds, and cleaning the node or restarting kubelet and the container runtime does not help. + +### Cause 5: one node uses a different iptables backend or has leftover legacy rules + +- A node whose iptables runs in nf_tables mode (for example TencentOS `3.1`, kernel `5.4`, iptables `1.8.5`) joined a cluster whose other nodes use legacy mode (TencentOS `2.4`, kernel `4.19`, iptables `1.4.21`). nf_tables and legacy rule tables are separate, so the rules written by Calico and kube-proxy do not apply to that node's traffic: its host reaches the apiserver Service, every Pod on it times out, and the Calico Pods on it fail to start. Flushing iptables and restarting `calico-node` and `kube-proxy` does not help. Node admission does not check OS or iptables consistency. +- On TencentOS `3.1` with the ACP `3.12` Calico, after an upgrade one Pod (for example `calico-kube-controllers`) cannot be reached from its own node: `ip r get ` returns `local ... dev lo`. Leftover `iptables-legacy` rules that this `calico-node` version does not clean up are the cause; later versions clean them up. +- Newly added TencentOS Server 3 nodes could not carry Calico container traffic at all (ALB `504`, NodePort, Pod IP and Service access to their Pods all failing) while `tunl0` and `cali*` routes looked normal. The mechanism was not determined; replacing them with nodes on the OS version the rest of the cluster runs restored service. + +## Resolution + +### Cause 1: Kylin V10 kernel nf_conntrack defect drops NAT replies + +Upgrade the kernel on all nodes to `4.19.90-52.33.v2207.ky10` or later; the recommended stable version is `kernel-4.19.90-52.44.v2207.ky10`. Until then, restarting OVS on an affected node (deleting its `ovs-ovn` Pod) is a temporary mitigation. Collect `/var/log/kube-ovn` and `/var/log/openvswitch` from the node before restarting if the cause is not yet confirmed. + +> **Warning:** A kernel upgrade needs a reboot. Drain each node first, upgrade one node at a time, and keep the previous kernel in the boot menu as the rollback. Deleting the `ovs-ovn` Pod interrupts Pod traffic on that node until the new Pod is Ready. + +```bash +uname -r +kubectl drain --ignore-daemonsets --delete-emptydir-data +# install the vendor kernel package, reboot the node, then: +kubectl uncordon +kubectl -n kube-system delete pod +``` + +### Cause 2: the node intermittently loses the apiserver VIP + +Hand the path between the nodes and the VIP to the network or IaaS team with evidence. Run a continuous probe from several nodes and keep it until the next alert: + +```bash +journalctl -u kubelet --since '-2h' | grep -E 'no route to host|lease' | tail +mkdir -p /var/log/network_check +while true; do ping -c 50 -i 0.2 -q | tail -2 | sed "s/^/$(date '+%F %T') /"; done >> /var/log/network_check/vip_ping.log +``` + +Stop the loop with `Ctrl-C`. Rounds with non-zero loss, matched against `kubectl get events -A --field-selector reason=NodeNotReady`, are the evidence. See also "Node NotReady caused by network, storage or host-level faults on ACP". + +### Cause 3: unregister_netdevice: waiting for lo to become free + +Upgrade the kernel to the vendor version that fixes the leak, only on nodes where the `waiting for lo` message repeats. + +> **Warning:** The kernel upgrade needs a drain and a reboot of each node. Keep the old kernel as the boot fallback. + +```bash +dmesg -T | grep 'waiting for lo to become free' | tail -3 +uname -r +``` + +### Cause 4: the VM source/destination check is enabled on the virtualization platform + +Disable the source/destination check for every VM that is a cluster node, on the virtualization platform, then retry the component. No cluster change is needed. + +### Cause 5: one node uses a different iptables backend or has leftover legacy rules + +Keep the same OS version, kernel and iptables backend on all nodes of a cluster; replace or reinstall the odd node to match the others. + +For the TencentOS `3.1` variant on ACP `3.12`, flush the leftover legacy rules from inside the `calico-node` Pod on that node. Upgrading `calico-node` to a version that cleans them up is the permanent fix. + +> **Warning:** The flush removes every rule in the `iptables-legacy` filter, raw, mangle and nat tables of the host. Felix rewrites its own rules, but other software on the node that relies on legacy rules loses them. Only when the node matches this variant and `iptables-legacy-save` shows rules. Save them first; restoring the file brings the fault back. + +```bash +kubectl -n kube-system exec -c calico-node -- iptables-legacy-save > /root/iptables-legacy.before +kubectl -n kube-system exec -c calico-node -- sh -c 'iptables-legacy -F; iptables-legacy -F -t raw; iptables-legacy -F -t mangle; iptables-legacy -F -t nat' +curl -m 5 : +``` + +When new nodes on a different OS version cannot carry Pod traffic and no cause is found, replace them with nodes on the OS version the rest of the cluster runs. + +## Diagnostic Steps + +Before changing the CNI, check whether the fault follows a node property: + +```bash +kubectl get nodes -o wide +uname -r; grep -E '^(NAME|VERSION)=' /etc/os-release +dmesg -T | grep -E 'nf_conntrack|waiting for lo' | tail +journalctl -u kubelet --since '-1h' | grep -E 'no route to host|timeout' | tail +iptables --version +iptables-legacy-save 2>/dev/null | wc -l +iptables-nft-save 2>/dev/null | wc -l +``` + +- A Kylin V10 kernel older than `4.19.90-52.33` with DNS timeouts to `istiod`, Endpoint flaps under load, or one Pod that cannot reach a correct ClusterIP: Cause 1. +- `no route to host` toward the VIP in the kubelet log, with `NodeNotReady` events: Cause 2. +- `waiting for lo to become free` together with `failed to reserve sandbox name`: Cause 3. +- Pod traffic failing only on VMs whose source/destination check is on: Cause 4. +- `iptables v1.8.x (nf_tables)` on the failing node and `(legacy)` on the others, or a TencentOS `3.1` node whose `calico-node` still sees `iptables-legacy` rules: Cause 5. + +For Pods that cannot reach the apiserver Service because a host `FORWARD` rule or firewall drops cross-node traffic, see "Cross-node traffic fails in the host network path: default route, firewall, lost tunnel packets or iptables on ACP". diff --git a/docs/en/solutions/Pod_traffic_fails_on_Calico_because_of_an_IPPool_destination_MTU_firewall_BGP_sessions_or_IPv6_routes_on_ACP.md b/docs/en/solutions/Pod_traffic_fails_on_Calico_because_of_an_IPPool_destination_MTU_firewall_BGP_sessions_or_IPv6_routes_on_ACP.md new file mode 100644 index 000000000..fcac6cafe --- /dev/null +++ b/docs/en/solutions/Pod_traffic_fails_on_Calico_because_of_an_IPPool_destination_MTU_firewall_BGP_sessions_or_IPv6_routes_on_ACP.md @@ -0,0 +1,211 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Pod traffic fails on Calico because of an IPPool destination, MTU, firewall, BGP sessions or IPv6 routes on ACP + +## Issue + +On a cluster that uses Calico as the CNI, Pod traffic fails or times out while the node itself is fine. Typical reports: + +- `telnet ` from the node prints `Connected to ...`, but the same command from inside a Pod (or after `nsenter -t -n`) stays at `Trying ...`. +- Pods on some nodes cannot reach a VM outside the cluster while Pods on other nodes can, and the nodes differ in `net.ipv4.conf.all.rp_filter`. +- One business container cannot open an external port, while an agent container in the same Pod, other Pods on the node, and the host all can. +- Business nginx times out toward a master NodePort after the MTU of the physical NIC was changed. +- A Pod-to-Pod call times out, and a capture shows the request already inside the destination Pod. +- ClusterIP `telnet` fails right after a node gained an IPv6 address and IPv6 routes. +- After a node was removed and a new node with the same IP was added, Pods on the new node cannot resolve names, and `ping` from the node to a CoreDNS Pod IP returns `Destination Net Unreachable`. +- In a BGP cluster with IPIP and VXLAN off, traffic from containers to outside services arrives from the node address, and the question is whether Calico encapsulates it. + +Related articles: "Calico ipipMode reverts, Felix metrics off, bandwidth annotation breaks Pods, systemd-udevd loops on calico_tmp on ACP". + +## Root Cause + +### Cause 1: the external destination lies inside a Calico IPPool CIDR + +Calico treats every address inside the CIDR of any `IPPool` as a cluster address. Its `cali-nat-outgoing` chain masquerades Pod traffic only when the destination is outside all pools: + +```text +MASQUERADE all -- 0.0.0.0/0 0.0.0.0/0 /* cali:... */ match-set cali40masq-ipam-pools src ! match-set cali40all-ipam-pools dst +``` + +When the external host's address falls inside a pool CIDR, Pod traffic to it is not source-NATed and is routed as if it were in-cluster traffic (over IPIP when `ipipMode: Always`). The request leaves with the Pod address, and the remote host cannot answer it. The node uses its own routing table, so the same connection works from the node. + +The pool can be the active pool (for example `default-ipv4-ippool` with a `/16` that also covers a range used outside the cluster) or a pool that is disabled and unused: a disabled pool still counts. A difference in `rp_filter` between nodes looks like the cause and is not. + +### Cause 2: a non-Calico firewall rule on the node blocks one container + +The business container hangs on an external address. The host, an agent container in the same Pod, and other Pods on the node connect. `net.ipv4.ip_forward` is `1`, and the container's default route is the normal Calico route `default via 169.254.1.1 dev eth0`. The node has firewall rules that are not written by Calico or kube-proxy, and they conflict with the container's traffic. Correcting those rules restores the connection. + +### Cause 3: the physical NIC MTU was lowered and running Pods kept the old veth MTU + +With the IPIP overlay, the tunnel adds 20 bytes to every packet. When the physical NIC MTU was reduced (for example to 1496) and running Pods were not recreated, their veths stayed at 1480, so encapsulated packets exceeded the NIC MTU and were fragmented or dropped. The symptom is intermittent timeouts, for example business nginx failing toward a master NodePort, while small API checks on the master, host load and kube-proxy logs look normal. + +### Cause 4: the callee Pod ACKs at once and the application does not read + +A caller times out toward one replica. A capture shows the request in the callee Pod's kernel within about a millisecond and an immediate TCP ACK, but the application thread does not read the request for tens of seconds. When it resumes it sends a burst, and the caller has already reset. Other replicas of the same workload are clean. This is an application stall, not a Calico drop; changing IP pools, BGP or MTU does not shorten it. + +### Cause 5: ClusterIP fails after an IPv6 address and IPv6 routes were added to a node + +On a dual-stack Calico cluster, a node was given an IPv6 address and IPv6 routes, and ClusterIP access from or through that node then failed. The mechanism was not determined. Recreating that node's `calico-node` and `kube-proxy` Pods restored ClusterIP access. + +### Cause 6: BGP: Unexpected connect from unknown address, after a node was re-added with the same IP + +After a node is removed and a new node with the same IP joins, an existing node's BIRD can fail to create a BGP session for the new node and reject its connections. There are then no routes to the Pods on the new node: from the new node, `ping` to the CoreDNS Service address works while `ping` to a CoreDNS Pod IP fails with `Destination Net Unreachable`, and its Pods cannot resolve names. TCP `179` from the new node is refused on the affected peers, whose `calico-node` log repeats: + +```text +bird: BGP: Unexpected connect from unknown address (port ) +``` + +`calicoctl node status` can still show every peer `up`, but `birdcl show protocols` on the affected peer lists `Mesh_*` sessions to the other nodes and no entry for the new node. This is a low-probability timing defect in Calico BGP peering. Restarting `calico-node` on the new node does not help; restarting it on the affected peer does. + +### Cause 7: container egress leaves with the node address because the IPPool has natOutgoing: true + +In BGP mode with `ipipMode` and `vxlanMode` off, nothing is encapsulated. When the pool has `natOutgoing: true`, Calico source-NATs container traffic that leaves the cluster pools to the node address, so external services see the node, and return traffic comes back to the node and is translated back there. This is the expected behavior of that setting. + +## Resolution + +### Cause 1: the external destination lies inside a Calico IPPool CIDR + +1. List the pools and check which CIDR contains the destination: + + ```bash + kubectl get ippools.crd.projectcalico.org -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidr,DISABLED:.spec.disabled,NAT:.spec.natOutgoing' + ``` + +2. If the destination is inside the pool the cluster actually uses, reach the external host through an address outside every pool CIDR (for example, have the network team map the service to such an address), then test again from the Pod. Do not shrink or delete a pool that holds Pod addresses. + +3. If the destination is inside a disabled pool that no Pod uses, back the pool up and delete it, so traffic to that range is NATed like other external traffic. Leave `rp_filter` as it is. + + > **Warning:** Deleting an `IPPool` that still has allocations strands those Pods. Only when no Pod address lies in its CIDR. Roll back by applying the saved YAML. If a platform `Subnet` with the same name manages the pool, handle that object too, or the pool comes back. + + ```bash + kubectl get pod -A -o wide | grep '' || true + kubectl get ippools.crd.projectcalico.org -o yaml > /root/ippool-.yaml + kubectl delete ippools.crd.projectcalico.org + ``` + +### Cause 2: a non-Calico firewall rule on the node blocks one container + +Compare the node's firewall with a node where every container connects, and remove or correct the rules that are neither Calico's (`cali-*`) nor kube-proxy's (`KUBE-*`) and that affect the container's egress. + +> **Warning:** Editing the filter table can cut SSH and API access. Work from a console, save the rules first, and change only rules that a healthy node does not have. Roll back with `iptables-restore < /root/iptables.before`. + +```bash +iptables-save > /root/iptables.before +iptables -S +sysctl net.ipv4.ip_forward +``` + +Retry from the business container after the rules match a healthy node. + +### Cause 3: the physical NIC MTU was lowered and running Pods kept the old veth MTU + +Put the physical NIC MTU back to the value it had before the change, so that the Pod MTU plus the 20-byte IPIP header fits again, and persist it in the node network configuration so a reboot does not reapply the lower value. + +> **Warning:** Changing the NIC MTU disturbs large in-flight packets on that port. Only when the NIC MTU was reduced and running Pods still show the old veth MTU. Record the current value; setting it again is the rollback (and brings the fragmentation back). + +```bash +ip link show +ip -d link show | grep -E 'mtu|veth' | head +ip link set dev mtu +``` + +### Cause 4: the callee Pod ACKs at once and the application does not read + +No CNI change. Capture on the callee node long enough to see whether the application reads the payload. An immediate kernel ACK followed by tens of seconds of silence from the process, with other replicas healthy, goes to the application owner. + +```bash +tcpdump -ni any -c 50 host and port +``` + +### Cause 5: ClusterIP fails after an IPv6 address and IPv6 routes were added to a node + +Delete the `calico-node` and `kube-proxy` Pods on that node so the DaemonSets recreate them. + +> **Warning:** Until the new Pods are Ready the node's routes and Service rules are reprogrammed and in-flight connections on it reset. Only when ClusterIP access broke after the node received IPv6 addresses and routes. Restart one node at a time and confirm ClusterIP from a Pod on that node before the next. Nothing needs to be rolled back. + +```bash +kubectl -n kube-system get pod -o wide | grep -E 'calico-node|kube-proxy' | grep +kubectl -n kube-system delete pod +kubectl -n kube-system rollout status ds/calico-node +``` + +### Cause 6: BGP: Unexpected connect from unknown address, after a node was re-added with the same IP + +1. Find the peers that have no session to the re-added node: + + ```bash + kubectl -n kube-system logs -c calico-node --tail=500 | grep 'Unexpected connect from unknown address' + kubectl -n kube-system exec -c calico-node -- birdcl -s /var/run/calico/bird.ctl show protocols + ``` + +2. Restart `calico-node` on each affected peer. + + > **Warning:** Restarting `calico-node` on a peer withdraws and re-learns its BGP routes, so Pods on it can briefly lose cross-node traffic. Only on peers whose log shows the message above or whose protocol list lacks the new node. Restart one peer at a time; the DaemonSet recreates the Pod, so there is nothing to roll back. + + ```bash + kubectl -n kube-system delete pod + kubectl -n kube-system exec -c calico-node -- birdcl -s /var/run/calico/bird.ctl show protocols | grep -i mesh + ``` + +3. If it recurs, check session establishment with `calicoctl get bgpsessions -o wide`. If it is empty, do a rolling restart of every `calico-node` Pod except the one on the re-added node, one at a time. + +### Cause 7: container egress leaves with the node address because the IPPool has natOutgoing: true + +No fault to repair. Confirm the setting: + +```bash +kubectl get ippools.crd.projectcalico.org -o custom-columns='NAME:.metadata.name,CIDR:.spec.cidr,NAT:.spec.natOutgoing,IPIP:.spec.ipipMode,VXLAN:.spec.vxlanMode' +``` + +Showing Pod addresses to the outside instead is a design change, not a fix: with NAT off, the external network has to route the Pod CIDR back to the nodes. + +## Diagnostic Steps + +1. Compare the node and the Pod toward the same destination, and check whether the destination is inside a pool: + + ```bash + telnet + nsenter -t -n telnet + iptables -t nat -L cali-nat-outgoing -n + kubectl get ippools.crd.projectcalico.org -o yaml | grep -E 'name:|cidr:|disabled:|natOutgoing:' + ``` + + The node connects, the Pod stays at `Trying`, and the destination is inside a pool CIDR: Cause 1. + +2. Check whether only one container fails. Run the same test from another Pod on the node; if only one container fails, compare `iptables -S` with a healthy node: Cause 2. + +3. Check MTU and fragmentation: + + ```bash + kubectl exec -n -- ip route + ip link show + ping -M do -s 1400 -c 2 || true + ``` + + Timeouts that start when the NIC MTU was lowered while Pod veths kept the old MTU: Cause 3. + +4. An immediate ACK in the callee Pod followed by a silent application: Cause 4. A ClusterIP failure that starts with an IPv6 route change on dual-stack: Cause 5. + +5. For a re-added node, check BGP on the peers: + + ```bash + kubectl -n kube-system logs -c calico-node --tail=500 | grep -E 'Unexpected connect|BGP' + ss -anp | grep ':179' + ``` + + `Unexpected connect from unknown address `: Cause 6. + +6. External services see the node address and the pool has `natOutgoing: true`: Cause 7, expected behavior. + +If cross-node `ping` returns `Destination Host Prohibited`, see "Cross-node traffic fails in the host network path: default route, firewall, lost tunnel packets or iptables on ACP". diff --git a/docs/en/solutions/Pods_cannot_resolve_intranet_or_external_names_nameserver_order_internal_DNS_server_or_debug_container_ping_on_ACP.md b/docs/en/solutions/Pods_cannot_resolve_intranet_or_external_names_nameserver_order_internal_DNS_server_or_debug_container_ping_on_ACP.md new file mode 100644 index 000000000..ce3996900 --- /dev/null +++ b/docs/en/solutions/Pods_cannot_resolve_intranet_or_external_names_nameserver_order_internal_DNS_server_or_debug_container_ping_on_ACP.md @@ -0,0 +1,135 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Pods cannot resolve intranet or external names: nameserver order, internal DNS server or debug container ping on ACP + +## Issue + +Pods fail to resolve names outside the cluster. Typical reports: + +- External names from Pods succeed and then fail. The node `/etc/resolv.conf` lists several `nameserver` lines, mixing an intranet DNS that does not recurse to the public Internet with one that does. +- The node scale-out precheck stops and says `/etc/resolv.conf` may contain only two `nameserver` entries, while the site needs intranet and public resolution. +- Applications must reach external systems by domain name, and those names are served only by an internal DNS server: `nslookup ` from a Pod returns `NXDOMAIN` or times out. +- `ping ` in a container debug session reports that the name cannot be resolved, while `ping
` of the same host works. + +Related articles: "Pods fail to resolve Service names or resolve them to external addresses: search domains and ndots on ACP"; "Host DNS changes do not reach CoreDNS: systemd-resolved, kubelet resolvConf, missing nameserver or a new upstream on ACP". + +## Root Cause + +### Cause 1: a list of nameservers is not split-horizon + +A list of nameservers in `/etc/resolv.conf` is not "ask the next server when this one says no such name". A resolver stops at the first definitive `NXDOMAIN`. Putting an intranet server that does not recurse in front of a public server therefore fails public names, and the reverse order fails intranet names. CoreDNS inherits the same list through `forward . /etc/resolv.conf`. ACP nodes also do not accept more than two `nameserver` lines: the scale-out precheck refuses a longer list, and that limit is not bypassed by a sysctl or a kubelet flag. + +### Cause 2: external system names exist only on an internal DNS server + +The lookup path is: business Pod, then `kube-dns` (CoreDNS), then `forward`, then the nameservers in `/etc/resolv.conf` of the node that runs the CoreDNS Pod. If neither the Corefile `forward` target nor those node nameservers is the internal DNS server, the internal zone cannot be answered. Either pointing the CoreDNS `forward` at the internal DNS server, or adding the server to the node resolver file on every node and restarting CoreDNS, makes the names resolvable. + +### Cause 3: ping by name fails in a musl-based debug container + +The container debug image is built on the musl C library. Its `ping` requests IPv6 by default and fails at once when the name has no `AAAA` record; a glibc-based `ping` falls back to IPv4. Cluster DNS is not failing. This is a property of the base image, not a CoreDNS setting. + +## Resolution + +### Cause 1: a list of nameservers is not split-horizon + +Give the nodes one resolver address that forwards by zone (intranet zones to the intranet servers, everything else to the public servers); publish it twice if a second address is needed for redundancy, and point every node's `/etc/resolv.conf` at those addresses only. + +When only Pods need the split and the node file must keep two lines, put per-zone `forward` stanzas in the Corefile instead. Example shape with documentation addresses: + +```text +intranet.example.com:1053 { + forward . 192.0.2.53 192.0.2.54 +} +.:1053 { + errors + kubernetes cluster.local in-addr.arpa ip6.arpa { + fallthrough in-addr.arpa ip6.arpa + } + forward . /etc/resolv.conf +} +``` + +> **Warning:** A broken zone stanza black-holes that zone for every Pod. Confirm each upstream answers a known name with `dig` from a node before adding it, and save the ConfigMap. The platform turns the ConfigMap edit into a ResourcePatch; wait until it is applied before testing. Roll back with `kubectl apply -f /tmp/cpaas-coredns.before.yaml`. + +```bash +kubectl -n kube-system get cm cpaas-coredns -o yaml > /tmp/cpaas-coredns.before.yaml +kubectl -n kube-system edit cm cpaas-coredns +``` + +Static names that are not a whole zone belong in a CoreDNS Host Alias (an IP, then domains separated by spaces), not in a third `nameserver` line. + +### Cause 2: external system names exist only on an internal DNS server + +Only when the internal DNS server answers the name (`dig @ ` from a node) and the Pod does not. Use one of the two approaches. + +**Corefile forward.** When the internal server should answer every non-cluster name, replace the target of the default `forward`; when it should answer only some zones, add a per-zone stanza as in Cause 1 and keep `forward . /etc/resolv.conf`: + +```text +.:1053 { + # other plugins unchanged + forward . 192.0.2.53 +} +``` + +> **Warning:** Changing the default `forward` sends every external lookup in the cluster to the new server; if it does not recurse for public names, public lookups fail for all Pods. Save the ConfigMap and confirm with `dig` that the server answers both internal names and the public names the cluster needs. Wait for the ResourcePatch to be applied. Roll back with `kubectl apply -f /tmp/cpaas-coredns.before.yaml`. + +```bash +kubectl -n kube-system get cm cpaas-coredns -o yaml > /tmp/cpaas-coredns.before.yaml +kubectl -n kube-system edit cm cpaas-coredns +kubectl exec -n -- nslookup +``` + +**Node resolver.** Add the server as a `nameserver` line in `/etc/resolv.conf` on every node, then restart CoreDNS so its Pods receive the new list. The two-line limit and the first-`NXDOMAIN` behavior of Cause 1 apply to this file. The full procedure is Cause 4 of "Host DNS changes do not reach CoreDNS: systemd-resolved, kubelet resolvConf, missing nameserver or a new upstream on ACP". + +> **Warning:** Restarting CoreDNS briefly delays lookups cluster-wide. Back up `/etc/resolv.conf` on each node; restoring it and restarting CoreDNS again is the rollback. + +```bash +cp -a /etc/resolv.conf /root/resolv.conf.before +kubectl -n kube-system rollout restart deploy/coredns +kubectl -n kube-system rollout status deploy/coredns +``` + +Adding the server to `spec.dnsConfig.nameservers` of individual workloads is a third possibility that affects only those workloads; a hand edit of the file inside a running container is lost on the next recreate. + +### Cause 3: ping by name fails in a musl-based debug container + +No cluster change fixes this. Test name resolution directly, and reachability by address: + +```bash +nslookup +ping -c 3 +``` + +A glibc-based image resolves the IPv4 address and pings it. + +## Diagnostic Steps + +1. Ask each node nameserver for the failing name; a definitive `NXDOMAIN` from the first one explains a failure even when the second one knows the name: + + ```bash + cat /etc/resolv.conf + dig @ +short + dig @ +short + kubectl -n kube-system get pod -l k8s-app=kube-dns -o wide + ``` + +2. Read the Corefile forward targets: + + ```bash + kubectl -n kube-system get cm cpaas-coredns -o jsonpath='{.data.Corefile}{"\n"}' | grep -B2 -A5 forward + ``` + +3. Map the result: + + - Public names flap, the node has several nameservers, or the precheck says only two are allowed: Cause 1. + - The name is answered by the internal DNS server and returns `NXDOMAIN` or a timeout from the Pod: Cause 2. + - `ping` by name fails only in the debug container while `nslookup` of the same name works: Cause 3. + +If `nslookup` to `kube-dns` times out instead of returning `NXDOMAIN`, check NetworkPolicies and the Pod's route to the CoreDNS ClusterIP before changing resolvers. diff --git a/docs/en/solutions/Pods_fail_on_one_Kube-OVN_node_after_Docker_or_a_second_runtime_is_installed_or_old_cluster_state_remains_on_ACP.md b/docs/en/solutions/Pods_fail_on_one_Kube-OVN_node_after_Docker_or_a_second_runtime_is_installed_or_old_cluster_state_remains_on_ACP.md new file mode 100644 index 000000000..50e7c4934 --- /dev/null +++ b/docs/en/solutions/Pods_fail_on_one_Kube-OVN_node_after_Docker_or_a_second_runtime_is_installed_or_old_cluster_state_remains_on_ACP.md @@ -0,0 +1,119 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.12.x + - 3.18.x + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# Pods fail on one Kube-OVN node after Docker or a second runtime is installed or old cluster state remains on ACP + +Related articles: [kube-ovn-cni fails on one node after a NIC rename, clock skew or hostname change on ACP](kube-ovn-cni_fails_on_one_node_after_a_NIC_rename_clock_skew_or_hostname_change_on_ACP.md). + +## Issue + +Pods on one node of a Kube-OVN cluster cannot be created or cannot reach the API server, although the CNI looks healthy elsewhere. Typical reports: + +- Pods on one node fail with `request ip return 500 configure nic failed no packets received from gateway` after someone installed Docker or a second containerd on the host. +- A platform component such as `zeus` panics because it times out connecting to the API server. From that node's Pods, `curl` to the `kubernetes` Service on `443` times out, while the API server node on `6443` answers: + + ```text + curl: (28) Failed to connect to port 443 after 130856 ms: Couldn't connect to server + curl: (28) Failed to connect to port 443 after 5002 ms: Timeout was reached + ``` + +- On a newly added node, `kube-ovn-pinger` is in `CrashLoopBackOff` (`Exit Code: 1`, hundreds of restarts), other platform Pods crash-loop, and the kubelet log repeats `"Unable to authenticate the request due to an error" err="[invalid bearer token, invalid signature, no keys found]"`. It looks as if the Kube-OVN components cannot connect to the API server. +- The node reaches a destination under the host policy, but Pods on it cannot. + +## Root Cause + +### Cause 1: Docker or a second containerd replaced or disturbed the platform runtime + +The event text looks like a Kube-OVN gateway failure (`no packets received from gateway`), but the runtime is what broke: + +- Docker installed by hand overwrites containerd-related binaries. Pod sandboxes cannot be created until those binaries are restored. On a single-node ACP 4.0.2 cluster, an extra containerd installed on the node broke the runtime the same way; recreating `ovn0` and clearing the OVN database did not help, and replacing the runtime files did. +- When Docker is started next to the platform's containerd and left running, it rewrites `docker0`, iptables NAT and `FORWARD` rules, and Pod forwarding on that node breaks. Pods then reach the API server node on `6443` but not the `kubernetes` Service on `443`, and components such as `zeus` panic because they cannot create their metrics handler. + +A platform node cannot keep a second runtime that rewrites its iptables. + +### Cause 2: the node kept state from a deleted cluster, or already had a runtime or another vendor's CNI + +- The machine was previously a node of a cluster that was removed, and the node cleanup was never run, so container services and iptables rules are still present. The node reaches destinations under the host policy, but Pods cannot. +- A newly added node already had containerd installed and another vendor's network plugin deployed. On it, `kube-ovn-pinger` (`pinger` container) is `CrashLoopBackOff` with `Exit Code: 1`, `zeus` and other platform Pods crash-loop, and the kubelet log repeats `invalid bearer token, invalid signature, no keys found`. The containerd log has `an error occurred when try to find sandbox: not found` and image pulls fail with `dial tcp :80: i/o timeout`. Such a node does not meet the platform's node software and directory requirements, which ask for conflicting packages to be removed and leftover runtime and kubelet directories to be deleted before the node is added. The exact failure mechanism on such a node was not determined. + +## Resolution + +### Cause 1: Docker or a second containerd replaced or disturbed the platform runtime + +1. Confirm the signature from the `kube-ovn-pinger` Pod on that node, and check whether Docker is running: + + ```bash + kubectl -n kube-system exec -- curl -vk --connect-timeout 5 --max-time 10 https://:443/healthz + kubectl -n kube-system exec -- curl -k -m 5 https://:6443/healthz + systemctl is-active docker containerd + ip link show docker0 + iptables -t nat -S | grep -i docker | head + rpm -qf /usr/bin/containerd /usr/bin/runc + ``` + +2. Restore the platform runtime. Keep this order: stop kubelet, containerd and Docker; uninstall the Docker packages; copy the platform binaries; start containerd and kubelet. Removing a Docker package deletes the files it owns, which can include `/usr/bin/containerd` and `/usr/bin/runc`, so uninstalling after the copy deletes the restored binaries again. A running `containerd` cannot be overwritten in place (`Text file busy`). + + The binaries to copy back from a healthy node of the same ACP version and architecture are `containerd`, `containerd-shim`, `containerd-shim-runc-v1`, `containerd-shim-runc-v2`, `containerd-stress`, `ctr`, `crictl`, `critest`, `runc`, `ctd-decoder`, `kubelet` and `kubectl`. + + > **Warning:** Stopping Docker and replacing `containerd` and `kubelet` stops every container that runs under `dockerd`, stops all container creation and probes on the node, and can leave it `NotReady`. Drain the node first if the API answers. Containers under the platform containerd keep running under `containerd-shim` only when the containerd unit has `KillMode=process` (check `systemctl show containerd -p KillMode`); with `control-group` or `mixed`, stopping containerd stops every Pod on the node. Work from a console, copy the current binaries aside first, and only copy from a node on the same ACP build and architecture. To roll back, restore the saved binaries and start the previous runtime. + + ```bash + mkdir -p /root/runtime-before && cp -a /usr/bin/containerd* /usr/bin/ctr /usr/bin/runc /usr/bin/kubelet /root/runtime-before/ 2>/dev/null + systemctl stop kubelet containerd + systemctl disable --now docker docker.socket || true + # uninstall the Docker packages reported by rpm -qf / rpm -qa | grep docker + # copy the binaries listed above from the healthy node to the same paths + which -a runc containerd + systemctl start containerd + systemctl start kubelet + ``` + +3. If `which -a runc containerd` still lists a copy under `/usr/local/bin` after the restore, Pods keep failing until that leftover is moved aside. +4. When Docker must stay on the machine for other applications, take the node out of the cluster instead: cut its access to the API server and delete the Node object. + +### Cause 2: the node kept state from a deleted cluster, or already had a runtime or another vendor's CNI + +- **Node from a deleted cluster.** Run the platform's node cleanup script for a removed node: when a node is deleted from the cluster in the web UI, the dialog offers **Download Cleanup Script**; run it on the node after the deletion. Then add the node again if it should be part of the cluster, and retest from a Pod. + + > **Warning:** The cleanup removes CNI state and rules from the node and is not fully reversible. Do not run it on a node that is still a healthy member of the cluster. A broad `iptables -F` or `ip link delete` on OVS bridges is not a substitute: it blackholes the node. Save the rules first. + + ```bash + iptables-save > /root/iptables.before-node-cleanup + crictl ps + systemctl list-units --type=service --all | grep -Ei 'docker|kubelet|containerd' + ``` + +- **New node with a pre-installed runtime or another vendor's CNI.** Remove the node from the cluster, uninstall packages that conflict with the platform components, delete the directories listed in the node software and directory requirements, and add the node again. + + > **Warning:** The commands below destroy every container image, container, Pod log and kubelet client certificate on the host, and they are irreversible. Only on a node that has already been removed from the cluster (`kubectl get node ` returns `NotFound`), with its runtime stopped, and only when the `invalid bearer token` or pre-installed runtime signature is confirmed. Do not run them on a member node. + + ```bash + kubectl get node + systemctl stop kubelet containerd docker 2>/dev/null || true + rm -rf /var/lib/docker /var/lib/nerdctl /opt/nerdctl/ /var/lib/containerd /var/log/pods /var/lib/kubelet/pki + ``` + +## Diagnostic Steps + +```bash +kubectl -n kube-system get pod -o wide | grep -E 'kube-ovn-cni|ovs-ovn|kube-ovn-pinger' | grep +kubectl -n kube-system describe pod | grep -A3 FailedCreatePodSandBox +crictl version +systemctl is-active containerd kubelet docker || true +journalctl -u kubelet --since '-30m' | grep -E 'invalid bearer token|Unable to authenticate' | tail -5 +journalctl -u containerd --since '-30m' | grep -E 'find sandbox|i/o timeout' | tail -5 +``` + +- `docker` active on a containerd node, or runtime binaries that changed after a Docker install, is Cause 1. On that node, a Pod reaching the API server node on `6443` but not the `kubernetes` Service on `443` points to the same cause. +- `invalid bearer token` in the kubelet log of a new node that already had containerd or another vendor's CNI, or a node that came from a deleted cluster, is Cause 2. +- If `kube-ovn-cni` itself fails with `failed to parse config` or `failed to initialize config` for an old NIC name, see the related article. diff --git a/docs/en/solutions/Pods_fail_to_resolve_Service_names_or_resolve_them_to_external_addresses_search_domains_and_ndots_on_ACP.md b/docs/en/solutions/Pods_fail_to_resolve_Service_names_or_resolve_them_to_external_addresses_search_domains_and_ndots_on_ACP.md new file mode 100644 index 000000000..49c05e08a --- /dev/null +++ b/docs/en/solutions/Pods_fail_to_resolve_Service_names_or_resolve_them_to_external_addresses_search_domains_and_ndots_on_ACP.md @@ -0,0 +1,131 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Pods fail to resolve Service names or resolve them to external addresses: search domains and ndots on ACP + +## Issue + +A Pod fails names that cluster DNS answers when asked directly, resolves a Service to the wrong address, or spends a long time resolving. Typical reports: + +- Application logs show `Name or service not known`, `UnknownHostException` or `SearchDomainUnknownHostException` for a name like `.` (one dot) or a short Service name, while `..svc.cluster.local` returns `NOERROR`. +- An in-cluster Service such as `mysql` starts returning an external address after someone pointed containers at an external resolver; the node resolves public names, the Pod does not unless a public resolver is written into the container. +- A task hangs for a long time at one stage with the default `dnsPolicy: ClusterFirst`, and does not hang with `dnsPolicy: Default`. + +A Pod's `/etc/resolv.conf` is generated by the kubelet. With `dnsPolicy: ClusterFirst`, `nameserver` is the `kube-dns` ClusterIP, `options ndots:5` is set, and `search` lists `.svc.cluster.local`, `svc.cluster.local`, `cluster.local`, plus the search or domain suffix of the node's own `/etc/resolv.conf`. + +Related articles: "Pods cannot resolve intranet or external names: nameserver order, internal DNS server or debug container ping on ACP". + +## Root Cause + +### Cause 1: an expanded name returns NXDOMAIN while the FQDN returns NOERROR + +With `ndots:5`, a name with fewer than five dots is tried with each search suffix before it is tried as an absolute name. When the node's `/etc/resolv.conf` has a company `domain` or `search` line, the kubelet copies it into the Pod search list. The CoreDNS log then shows `NOERROR` for `..svc.cluster.local` and `NXDOMAIN` for expansions such as the name with the namespace suffix appended twice or with the company domain appended. Resolvers such as Netty fail the whole lookup when one expanded query returns `NXDOMAIN`, even though another expansion is the real Service. CoreDNS itself answers the FQDN quickly. + +A readiness failure of one CoreDNS replica on `http://:8181/ready` (`i/o timeout`) in the same window is a separate fault and does not explain these `NXDOMAIN` answers. + +### Cause 2: a Service name resolves to an external address after external DNS was added to the container + +Writing nameservers into a container's `/etc/resolv.conf` bypasses `kube-dns`. The remaining search list, or a search domain inherited from the node, then lets a short name (for example `mysql`) match an external zone. CoreDNS learns the node's resolvers only through `forward . /etc/resolv.conf` on the node where the CoreDNS Pod runs; a nameserver inside one container is not cluster DNS, and it is lost when the Pod is recreated. + +If only some nodes have a route to the public resolver, CoreDNS Pods on the other nodes time out on external names while in-cluster names still work. + +### Cause 3: the task hangs with dnsPolicy ClusterFirst and not with dnsPolicy Default + +With `ClusterFirst`, every external name with fewer than five dots is first tried with each search suffix, often for both A and AAAA records, and each attempt is a round trip. An application that resolves many external names, or retries after slow answers, accumulates that delay. `dnsPolicy: Default` copies the node file, which has no cluster search list and the default `ndots:1`, so the expansions disappear. Lowering `ndots` to `1` on the workload reduces resolution time while keeping cluster DNS. + +## Resolution + +### Cause 1: an expanded name returns NXDOMAIN while the FQDN returns NOERROR + +Only when CoreDNS shows `NOERROR` for the FQDN and `NXDOMAIN` for an expansion that repeats the namespace or appends the company domain. Apply both changes: + +1. Point the application at the short Service name or at the full FQDN `..svc.cluster.local` (a trailing dot forces an absolute query). +2. Remove the company `domain` / `search` suffix from `/etc/resolv.conf` on every node, keep the `nameserver` lines, and recreate the application Pods. + +> **Warning:** Editing the node file changes resolution for host processes and for every new Pod. Copy the file on each node first; restoring it and recreating the Pods is the rollback. In-cluster Service names keep working through `kube-dns`. + +```bash +cp -a /etc/resolv.conf /root/resolv.conf.before +vi /etc/resolv.conf # delete only the domain or search token with the company suffix +kubectl delete pod -n +kubectl exec -n -- cat /etc/resolv.conf +``` + +New Pods should list `search .svc.cluster.local svc.cluster.local cluster.local` without the company domain. See also "Platform UI stuck loading or DevOps errors from Service name timeouts caused by node search domains on ACP". + +### Cause 2: a Service name resolves to an external address after external DNS was added to the container + +Fix the nodes, not the container. Remove the extra search domain from `/etc/resolv.conf` on all nodes; on the nodes that run CoreDNS, put the resolver that answers public names first. Restart CoreDNS so it re-reads the file, then recreate the test Pod. + +> **Warning:** Cluster lookups stall briefly while CoreDNS restarts, and recreated application Pods drop their connections. Confirm the first nameserver answers every zone you need and save `/etc/resolv.conf` first; restoring it and restarting CoreDNS again is the rollback. + +```bash +cp -a /etc/resolv.conf /root/resolv.conf.before +kubectl -n kube-system rollout restart deploy/coredns +kubectl -n kube-system rollout status deploy/coredns +kubectl delete pod -n +kubectl exec -n -- nslookup kubernetes.default +kubectl exec -n -- nslookup +``` + +If public resolution works only on some nodes, keep `/etc/resolv.conf` identical on all nodes and schedule the Pods that need the public network onto the nodes that have the route (taints on the other nodes, plus a matching toleration and node affinity on those workloads). For one external name with a fixed address, add a CoreDNS Host Alias instead of a container entry; see "CoreDNS Host Alias records do not take effect because a ResourcePatch removes data hosts from cpaas-coredns on ACP". + +### Cause 3: the task hangs with dnsPolicy ClusterFirst and not with dnsPolicy Default + +Keep `dnsPolicy: ClusterFirst` and set `ndots` to `1` in the workload's `dnsConfig`: + +```yaml +spec: + template: + spec: + dnsPolicy: ClusterFirst + dnsConfig: + options: + - name: ndots + value: "1" +``` + +> **Warning:** Patching the Pod template rolls the workload. With `ndots:1`, a name such as `.` is tried as an absolute name first, so the application should use short names or full FQDNs. Save the Deployment; applying the saved copy is the rollback. + +```bash +kubectl -n get deployment -o yaml > /tmp/.before.yaml +kubectl -n patch deployment --type=strategic \ + -p '{"spec":{"template":{"spec":{"dnsConfig":{"options":[{"name":"ndots","value":"1"}]}}}}}' +kubectl -n rollout status deployment +kubectl exec -n -- cat /etc/resolv.conf +``` + +The new Pod shows `options ndots:1` with the same `nameserver` and `search` lines. The same change on a platform component is described in "ArgoCD repo-server DNS lookups leak upstream because of pod ndots:5 search expansion". + +## Diagnostic Steps + +1. Compare the node file, the Pod file, and one absolute query: + + ```bash + cat /etc/resolv.conf + kubectl exec -n -- cat /etc/resolv.conf + kubectl exec -n -- nslookup + kubectl exec -n -- nslookup ..svc.cluster.local + ``` + +2. Turn on CoreDNS query logging if needed (add `log` to the server block of `cpaas-coredns`; it reloads without a restart) and read the answers: + + ```bash + kubectl -n kube-system edit cm cpaas-coredns + kubectl -n kube-system logs -l k8s-app=kube-dns --tail=50 + ``` + +3. Map the result: + + - FQDN `NOERROR`, expanded name `NXDOMAIN`, company domain in the Pod search list: Cause 1. + - Short Service name returns an off-cluster address and the container's `/etc/resolv.conf` was edited: Cause 2. + - Hang with `ClusterFirst`, none with `dnsPolicy: Default`, `options ndots:5` in the Pod: Cause 3. + +Read the cause from the CoreDNS log rather than from a monitoring chart before changing resolvers. diff --git a/docs/en/solutions/PostgreSQL_instance_abnormal_disk_full_WAL_buildup_replica_cannot_rejoin_or_writes_fail_on_ACP.md b/docs/en/solutions/PostgreSQL_instance_abnormal_disk_full_WAL_buildup_replica_cannot_rejoin_or_writes_fail_on_ACP.md new file mode 100644 index 000000000..81f6608e6 --- /dev/null +++ b/docs/en/solutions/PostgreSQL_instance_abnormal_disk_full_WAL_buildup_replica_cannot_rejoin_or_writes_fail_on_ACP.md @@ -0,0 +1,230 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Application Services +ProductsVersion: + - "3.18" + - "4.1" + - "4.3" +--- + +# PostgreSQL instance abnormal: disk full, WAL buildup, replica cannot rejoin, or writes fail on ACP + +## Issue + +A Data Services PostgreSQL instance is reported abnormal. The custom resource is `postgresql` (`acid.zalan.do/v1`), and the members are Patroni-managed Pods `-0` and `-1`. One of these is visible: + +- The PostgreSQL log says the disk is full: `OSError: [Errno 28] No space left on device: '/home/postgres/pgdata/pgroot/data/recovery.conf'`, followed by `WARNING: Postgresql is not running.` (Cause 1) +- The primary cannot serve requests and its data volume is full of WAL; storage dashboards can show the nodes at 99% disk usage although the business data is only megabytes (Cause 2). +- One member Pod is `0/1 Running` while the other is `1/1`, and `patronictl list` shows that member as `Replica` in state `starting`, with an empty `TL` and `Lag in MB` of `unknown`, while the other member is the running `Leader`. It looks like split-brain (Cause 3). +- Containers of the instance do not start, with `SMT Alignment Error: requested 1 cpus not multiple cpus per core = 2` (Cause 4). +- SQL statements fail and the application cannot write data (Cause 5). +- After its host node hung and was rebooted, one replica does not start while the primary runs normally; its log shows that `pg_controldata` failed and `/home/postgres/pgdata/pgroot/data/global/pg_control` could not be read (Cause 6). +- A replica is `stopped` in `patronictl list` and its log repeats `data dir for the cluster is not empty, but system ID is invalid; consider doing reinitialize`. Data Services can show the instance as `Error` while its Pods are `Running`, or the upgrade checklist reports PostgreSQL abnormal (Cause 7). + +## Root Cause + +### Cause 1: The data volume is full + +The PV that backs the PostgreSQL data directory has no free space. The `postgres` container of the member logs: + +```text +INFO: Lock owner: None; I am -1 +INFO: starting as a secondary +ERROR: Exception during execution of long running task restarting after failure +... + in write_recovery_conf +OSError: [Errno 28] No space left on device: '/home/postgres/pgdata/pgroot/data/recovery.conf' +WARNING: Postgresql is not running. +``` + +followed by `Database cluster state: shutting down`. Inside the Pod, `df -h` shows the data volume on `/home/postgres/pgdata` at 100% (for example Size `25G`, Used `25G`, Avail `104K`), while `sv status` still lists `cron`, `patroni`, and `pgqd` as `run`. Patroni cannot write `recovery.conf`, so PostgreSQL on that member does not start. Expand the PV, or delete data that is no longer needed. If the space is taken by WAL rather than by tables, see Cause 2. + +### Cause 2: WAL fills the primary because replication has been broken for a long time + +When primary-replica replication fails for a long time, the primary keeps every WAL segment the replica has not received. The WAL fills the primary's disk and the primary can no longer serve. Slow disks make it worse, because the replica falls behind faster than it can replay. Recovery: expand the primary PVC, rebuild the replica, and let the primary drop the old WAL once the rebuilt replica is streaming. + +On TopoLVM storage, the reported disk usage is the capacity already allocated to volumes, not what the data inside them uses. Cleaning other data (for example Elasticsearch indices) does not lower that figure. + +### Cause 3: Replica stays starting with Lag unknown after the WAL it needs is lost + +`patronictl list` shows the replica `starting` with no timeline and `Lag in MB` `unknown`, and the leader `running` on its timeline. The WAL the replica needs to catch up is no longer available (for example the WAL archive was lost), so it cannot heal on its own. `patronictl reinit` of that member from the leader recovers it. + +### Cause 4: SMT Alignment Error: requested 1 cpus not multiple cpus per core = 2 + +The nodes have NUMA enabled, and the kubelet CPU Manager and Topology Manager policies require SMT alignment. Under that policy an exclusive CPU request must be a multiple of the CPUs per physical core (2 on these nodes). The PostgreSQL container requests 1 CPU, and the kubelet rejects it at start. Setting the CPU resources to an even value fixes it. The rule applies to any Data Services Pod scheduled on those nodes; see the ACP 4.3 documentation on CPU Manager and Topology Manager policies. + +### Cause 5: SQL writes fail because the primary/standby switchover is stuck + +SQL statements against the instance fail and no data can be written because a primary/standby switchover is stuck. Restarting the PostgreSQL Pod of the instance restores writes. What leaves the switchover stuck has not been identified. + +### Cause 6: Replica cannot start because global/pg_control cannot be read after a node crash + +The host node became unresponsive (for example because of an operating system kernel bug). After the node was fixed and rebooted, the primary ran normally and the replica did not. The replica's log shows `pg_controldata` failing because `global/pg_control` cannot be read: the control file of that member was damaged by the crash. Resynchronizing the replica from the primary, by deleting the replica's PVC and Pod, returns it to normal. + +### Cause 7: Replica stopped with "data dir for the cluster is not empty, but system ID is invalid" + +The replica's `postgres` container repeats `INFO: data dir for the cluster is not empty, but system ID is invalid; consider doing reinitialize`. `patronictl list` shows it as `Replica`, state `stopped`, `Lag in MB` `unknown`, while the leader is `running` and logs `no action. I am (-0), the leader with the lock`. Patroni does not start PostgreSQL on a non-empty data directory whose system identifier it cannot read, so the replica never rejoins. The instance page can show `could not sync statefulsets: could not recreate pods: skipping switchover: no switchover candidate found`. Bootstrap lines such as `chmod: changing permissions of '/var/log/pg_log': Operation not permitted` and `Disabling 'bypass_api_service'` are not related. + +`patronictl reinit` of the replica can return `Success` while the member stays in `creating replica` with Lag `unknown`. Scaling the instance from 2 members to 1 and back to 2, which recreates the replica from the leader, restores the cluster. + +## Resolution + +### Cause 1: The data volume is full + +```bash +kubectl get pvc -n +kubectl exec -n -0 -c postgres -- df -h /home/postgres/pgdata +kubectl get sc -o jsonpath='{.allowVolumeExpansion}{"\n"}' +``` + +> **Warning:** A PVC can be expanded but never shrunk. Precondition: `allowVolumeExpansion` is `true`, the storage backend has the capacity, and you expand the PVC of the member whose disk is full. There is no rollback for the size; a smaller volume needs a new instance and a data migration. + +Raise the volume size of the instance and of the full claim to the same value, so the operator does not work against the new size: + +```bash +kubectl patch postgresql -n --type=merge -p '{"spec":{"volume":{"size":""}}}' +kubectl patch pvc -n -p '{"spec":{"resources":{"requests":{"storage":""}}}}' +kubectl get pvc -n -w +``` + +If the data owner prefers to free space instead, delete data that is no longer needed through SQL. + +### Cause 2: WAL fills the primary because replication has been broken for a long time + +Find the leader, how far the replica is behind, and how much of the disk is WAL: + +```bash +kubectl exec -n -0 -c postgres -- patronictl list +kubectl exec -n -c postgres -- du -sh /home/postgres/pgdata/pgroot/data/pg_wal +kubectl exec -n -c postgres -- psql -U postgres -c 'select slot_name, active, restart_lsn from pg_replication_slots' +``` + +A large `pg_wal`, a replica far behind or not streaming, and an inactive replication slot for that replica fit this cause. Expand the leader PVC first (Cause 1) so the leader can serve while the replica is rebuilt. Then rebuild the replica by deleting its PVC and then its Pod; the StatefulSet creates both again and the member clones itself from the leader. + +> **Warning:** Deleting the replica PVC destroys that member's copy of the database. The new member copies the full data set from the leader, which loads the leader's disk and network until it finishes. Precondition: `patronictl list` shows the Pod you delete as `Replica`, not `Leader`; the leader is `running`; a backup of the leader exists. The PVC delete is irreversible. Run it only against the replica. + +```bash +kubectl get pod -n -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}{"\n"}' +kubectl delete pvc -n --wait=false +kubectl delete pod -n +kubectl get pod,pvc -n -w +``` + +When `patronictl list` shows the rebuilt member `streaming` (or `running` with a small lag), the leader can remove the old WAL. Check `du -sh` on `pg_wal` again; the space is freed after the next checkpoints. + +### Cause 3: Replica stays starting with Lag unknown after the WAL it needs is lost + +```bash +kubectl exec -n -0 -c postgres -- patronictl list +kubectl logs -n -c postgres --tail=100 +``` + +Reinitialize only the member that shows `Replica` / `starting` / `unknown`, while the other member is `Leader` / `running`. + +> **Warning:** `patronictl reinit` erases the data directory of the named member and copies it again from the leader. Anything that existed only on that member is lost. Precondition: the named member is the replica, the leader is running and serving, and a backup of the leader exists. Irreversible for the member's local data. Name only the replica member. + +```bash +kubectl exec -it -n -c postgres -- patronictl reinit +``` + +The command can run from any member Pod, including the `0/1` replica itself; Patroni asks for confirmation. Watch `patronictl list` until the member shows the leader's timeline and a numeric lag. If it stays in `creating replica` with Lag `unknown`, or its log says `system ID is invalid`, continue with Cause 7. + +### Cause 4: SMT Alignment Error: requested 1 cpus not multiple cpus per core = 2 + +```bash +kubectl get pod -n -o jsonpath='{.status.reason}{" "}{.status.message}{"\n"}' +kubectl get pod -n -o jsonpath='{range .spec.containers[*]}{.name}{" "}{.resources}{"\n"}{end}' +``` + +On the node named in the Pod spec: + +```bash +grep -i -E 'cpuManagerPolicy|cpuManagerPolicyOptions|full-pcpus-only|topologyManagerPolicy' -A2 /var/lib/kubelet/config.yaml +``` + +> **Warning:** Changing the instance CPU recreates its Pods; each member restarts once and the leader can fail over. Precondition: the nodes have enough free CPU for the even value, and the application owner accepts a short disconnect. Rollback is setting the previous value again, which brings the error back on these nodes. + +Set the CPU request and limit of the instance to an even number (for example `2`) with the instance Update action in Data Services. + +### Cause 5: SQL writes fail because the primary/standby switchover is stuck + +```bash +kubectl exec -n -0 -c postgres -- patronictl list +kubectl logs -n -0 -c postgres --tail=100 +kubectl exec -n -c postgres -- psql -U postgres -c 'select pg_is_in_recovery()' +``` + +No member `running` as `Leader`, a pending switchover, or a leader that still answers `t` to `pg_is_in_recovery()` while writes fail fits this cause. + +> **Warning:** Deleting a PostgreSQL Pod restarts that member and clients see a disconnect; if it holds the leader lock, Patroni elects a leader again. Precondition: this cause is confirmed with `patronictl list`. Delete Pods only, never PVCs and never the `postgresql` resource. No rollback is needed; the StatefulSet recreates the Pod on its existing PVC. + +```bash +kubectl delete pod -n +kubectl exec -n -0 -c postgres -- patronictl list +``` + +Retry a write from the application. If writes still fail, collect `patronictl list` and the `postgres` logs of both members before any further restart. + +### Cause 6: Replica cannot start because global/pg_control cannot be read after a node crash + +```bash +kubectl exec -n -c postgres -- patronictl list +kubectl logs -n -c postgres --tail=200 | grep -i -E 'pg_control|pg_controldata' +kubectl get pod -n -o jsonpath='{.spec.nodeName}{" "}{.spec.volumes[*].persistentVolumeClaim.claimName}{"\n"}' +``` + +The node that hung must be healthy again (`Ready`, the kernel issue fixed), and the primary must be `Leader` and `running`. + +> **Warning:** Deleting the replica's PVC permanently removes its data directory, including the damaged `pg_control`. The rebuilt member copies the whole database from the primary, which loads the primary until the copy finishes. Precondition: the `pg_control` read failure is confirmed on a member that `patronictl list` shows as the replica, never on the leader; the leader is `running` and a backup of it exists. Irreversible for the replica's local data. + +```bash +kubectl delete pvc -n --wait=false +kubectl delete pod -n +kubectl get pod,pvc -n -w +``` + +When the new member shows `streaming` with the leader's timeline, the replica is resynchronized. If the leader itself reports the `pg_control` error, this procedure does not apply; collect both logs and contact support. + +### Cause 7: Replica stopped with "data dir for the cluster is not empty, but system ID is invalid" + +```bash +kubectl logs -n -c postgres --tail=200 | grep 'system ID is invalid' +kubectl exec -n -c postgres -- patronictl list +kubectl get postgresql -n -o yaml > -before.yaml +``` + +Continue only when the member that logs the message is `Replica` / `stopped` and the other is `Leader` / `running`. First run the Cause 3 `patronictl reinit` against that replica. If it stays in `creating replica` with Lag `unknown`, scale the instance to 1 member and back to 2, with the instance Update action in Data Services or on the resource. + +> **Warning:** Scaling to 1 member removes the highest-ordinal Pod (`-1`) and leaves the instance without a replica until it is scaled back; if that Pod held the leader lock, Patroni fails over. Precondition: `patronictl list` shows `-1` as the `stopped` replica and `-0` as `Leader` / `running`, and a backup of the leader exists. Never scale to 0. Rollback is setting `numberOfInstances` back to its previous value. If the broken replica is not the highest-ordinal Pod, scaling down removes the wrong member; use the Cause 6 replica rebuild instead. + +```bash +kubectl patch postgresql -n --type=merge -p '{"spec":{"numberOfInstances":1}}' +kubectl get pod,pvc -n -w +kubectl patch postgresql -n --type=merge -p '{"spec":{"numberOfInstances":2}}' +kubectl exec -n -0 -c postgres -- patronictl list +``` + +Patch back to 2 only after `-1` is gone. If the new `-1` mounts its old PVC again and returns with the same log, use the Cause 6 replica PVC procedure (replica only). The member is healthy when it shows `streaming` or `running` with the leader's timeline and a numeric lag; then rerun the upgrade checklist. + +## Diagnostic Steps + +```bash +kubectl get postgresql,pods,pvc -n -o wide +kubectl exec -n -0 -c postgres -- patronictl list +kubectl exec -n -0 -c postgres -- df -h /home/postgres/pgdata +kubectl get events -n --sort-by=.lastTimestamp | tail -n 30 +``` + +| Signature | Cause | +| --- | --- | +| `No space left on device` on `recovery.conf`, `df` at 100%, no WAL backlog | Cause 1 | +| `pg_wal` large on the leader, replica far behind or not streaming | Cause 2 | +| Replica `starting`, empty TL, `Lag in MB` `unknown`, leader `running` | Cause 3 | +| `SMT Alignment Error: requested 1 cpus not multiple cpus per core = 2` | Cause 4 | +| Writes fail, switchover stuck in `patronictl list` | Cause 5 | +| Replica log: `pg_controldata` fails, `global/pg_control` unreadable, after its node crashed | Cause 6 | +| Replica `stopped`, Lag `unknown`, log `system ID is invalid; consider doing reinitialize` | Cause 7 | + +If a member's container is not running, `patronictl list` still works from the other member. If both members are down because both disks are full, expand both PVCs before any rebuild. diff --git a/docs/en/solutions/Pre-upgrade_checklist_reports_a_CIDR_mismatch_a_missing_is-global_label_the_NFS_plugin_or_node_isolation_on_ACP.md b/docs/en/solutions/Pre-upgrade_checklist_reports_a_CIDR_mismatch_a_missing_is-global_label_the_NFS_plugin_or_node_isolation_on_ACP.md new file mode 100644 index 000000000..1f22a379a --- /dev/null +++ b/docs/en/solutions/Pre-upgrade_checklist_reports_a_CIDR_mismatch_a_missing_is-global_label_the_NFS_plugin_or_node_isolation_on_ACP.md @@ -0,0 +1,156 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.1" + - "4.2" +--- + +# Pre-upgrade checklist reports a CIDR mismatch, a missing is-global label, the NFS plugin, or node isolation on ACP + +## Issue + +The pre-upgrade checklist prints one of the following lines before an ACP upgrade, and it is not clear whether the line is a real blocker or what to change: + +- `集群的 cluster cidr不一致,cluster资源上的为:,kubeadm config上的为: cluster-cidr` (the cluster CIDR is inconsistent: the Cluster resource has ``, the kubeadm config has `cluster-cidr`). On the cluster, `kubectl get cm -n kube-system kubeadm-config -o yaml | grep -iE "cluster-cidr"` prints only the parameter name, not a CIDR. +- `ERROR: global集群不存在is-global的label` (the global cluster has no `is-global` label). +- `ERROR: 集群 已部署 NFS 插件,请确认 NFS 的协议版本,如果为 NFSv3 则需要手动添加 nolock 参数或者升级后将 NFS 插件的 readOnlyRootFilesystem 修改为 false` (the NFS plugin is deployed on cluster ``; confirm the NFS protocol version; for NFSv3 add the `nolock` option, or after the upgrade set `readOnlyRootFilesystem` of the NFS plugin to `false`). +- `升级前,请确认 global 集群中的节点是否已 启用 global Cluster Platform Node Isolation 。如需解决方案,请联系技术支持。` (before upgrading, confirm whether the global cluster nodes have global Cluster Platform Node Isolation enabled; contact technical support for a solution). + +## Root Cause + +### Cause 1: CIDR mismatch line that shows the parameter name cluster-cidr instead of a value + +The checklist compares the CIDRs on the platform's Cluster resource with the values in the `kube-system/kubeadm-config` ConfigMap. Newer kubeadm configurations (the `v1beta4` configuration API) store the controller-manager flags under `controllerManager.extraArgs` as a list of `name`/`value` pairs: + +```yaml +controllerManager: + extraArgs: + - name: allocate-node-cidrs + value: "false" + - name: cluster-cidr + value: + - name: node-cidr-mask-size + value: "24" +``` + +The CIDR is on the `value:` line after `- name: cluster-cidr`. Older configurations write `cluster-cidr: ` on one line. With the list layout, a one-line match for `cluster-cidr` returns only the name line, so the checklist prints `cluster-cidr` where a CIDR is expected. When the YAML is read in full and the value matches the Cluster resource, the line is a parsing artifact and nothing has to change. + +The same comparison answers other CIDR lines of the checklist: compare the cluster overview page with `kubeadm-config` and, on Kube-OVN clusters, with the `join` subnet. When the CIDR shown on the cluster's CIDR page is the one in use, the CIDR item can be ignored. + +### Cause 2: the global cluster has no is-global label + +The cluster label check applies to upgrades to 3.8 or later. It requires the label `is-global=true` on both global cluster resources: `cluster global`, and the `cluster.v1alpha1.clusterregistry.k8s.io` object named `global` in `cpaas-system`. When either object lacks the label, the checklist stops with this error. + +### Cause 3: the NFS plugin is deployed on the cluster + +The checklist flags every cluster that has the NFS plugin deployed, because an NFSv3 export needs either the `nolock` mount option or `readOnlyRootFilesystem: false` on the NFS plugin after the upgrade. The line is printed once per cluster, for example for both `global` and a business cluster. The check does not test whether the plugin is actually in use on that cluster, so it also appears for a cluster where the plugin is installed but unused. + +### Cause 4: Confirm whether global Cluster Platform Node Isolation is enabled + +The item refers to the node isolation policy of the global cluster. It is a confirmation prompt: when the policy is not enabled, the item does not apply. + +The same pre-upgrade step can also ask to upload the TopoLVM plugin before upgrading. That is the Alauda Build of TopoLVM entry for the matching node architecture in the App Marketplace (for a 4.2.1 target: channel `alpha`, version `v4.2.4`), which is offered for x86, ARM, and heterogeneous architectures. + +## Resolution + +### Cause 1: CIDR mismatch line that shows the parameter name cluster-cidr instead of a value + +Compare the two sources for each cluster the checklist names, one cluster at a time. + +1. In the console, open Cluster Management, select the cluster, and record the pod CIDR, the service CIDR, and on Kube-OVN clusters the join CIDR from the cluster overview page. +2. On a control-plane node of that cluster, read the values in use. `-A1` prints the `value:` line that follows each `name:` line; `podSubnet` and `serviceSubnet` are printed on their own lines: + + ```bash + kubectl get cm -n kube-system kubeadm-config -o yaml | grep -iE -A1 "cluster-cidr|subnet" + ``` + + On a Kube-OVN cluster also read the join subnet: + + ```bash + kubectl get subnet join -o jsonpath='{.spec.cidrBlock}{"\n"}' + ``` + +3. If every value matches the overview page, the checklist line is a parsing artifact. Ignore it and keep the comparison with the upgrade plan. + +If a value really differs from what the cluster is running, this cause does not apply. Send the comparison and the checklist output to Alauda technical support before the upgrade window. + +### Cause 2: the global cluster has no is-global label + +1. On a control-plane node of the global cluster, check the label on both objects: + + ```bash + kubectl get cluster global --show-labels | grep is-global + kubectl get cluster.v1alpha1.clusterregistry.k8s.io -n cpaas-system global --show-labels | grep is-global + ``` + +2. If either command prints nothing, add the label to that object: + + ```bash + kubectl label cluster global is-global=true + kubectl label cluster.v1alpha1.clusterregistry.k8s.io -n cpaas-system global is-global=true + ``` + +3. Run the checklist again. + +Add the label only to the global cluster's objects. A label added by mistake to another cluster is removed with `kubectl label cluster is-global-`. + +### Cause 3: the NFS plugin is deployed on the cluster + +1. On a node of each named cluster that mounts the export, read the NFS protocol version. `vers=` in the mount options is the version in use. Also list the StorageClasses, PVs, and NFS Pods: + + ```bash + grep -E ' nfs4? ' /proc/mounts + kubectl get sc -o custom-columns=NAME:.metadata.name,PROVISIONER:.provisioner,MOUNTOPTIONS:.mountOptions + kubectl get pv -o custom-columns=NAME:.metadata.name,SC:.spec.storageClassName,CLAIM:.spec.claimRef.name + kubectl get pods -A | grep -i nfs + ``` + + If `vers=4` or later is in use, the NFSv3 condition does not apply. + +2. **The cluster does not use the NFS plugin.** Uninstall the NFS plugin from that cluster through the platform's cluster plugin management before the upgrade. + + > **Warning:** Only when no PV on that cluster uses the NFS StorageClass and no workload mounts the NFS export, as shown by the `kubectl get pv` and `kubectl get sc` output above. Impact: new PVCs on the NFS StorageClass can no longer be provisioned on that cluster. Save the PV list first. Rollback: install the NFS plugin again from cluster plugin management with the same export settings. + +3. **The cluster uses the NFS plugin over NFSv3.** Add `nolock` to the mount options of the NFS StorageClass. Back the StorageClass up first: + + ```bash + kubectl get sc -o yaml > .yaml + ``` + + > **Warning:** With `nolock`, NFS file locks are local to each client node, so two nodes writing to the same file are not protected from each other by NFS locking. Precondition: the export is confirmed to be NFSv3 and the backup file exists. Rollback: remove `nolock` again with `kubectl apply -f .yaml`. A StorageClass change applies to PVs provisioned after it; each existing PV keeps the options in its own `spec.mountOptions`. + + If the StorageClass already has a `mountOptions` list, append to it: + + ```bash + kubectl patch sc --type=json -p '[{"op":"add","path":"/mountOptions/-","value":"nolock"}]' + ``` + + If it has no `mountOptions` field yet, create the list: + + ```bash + kubectl patch sc --type=merge -p '{"mountOptions":["nolock"]}' + ``` + +The checklist also offers setting `readOnlyRootFilesystem` to `false` on the NFS plugin after the upgrade as the alternative. This article covers the `nolock` path only. + +### Cause 4: Confirm whether global Cluster Platform Node Isolation is enabled + +1. Open the node isolation policy setting of the global cluster in the console. +2. If the policy is not enabled, ignore this checklist item. +3. If it is enabled, contact Alauda technical support for the solution before the upgrade, as the item instructs. + +For the TopoLVM upload, read the node architecture and pick the TopoLVM package for it (for example `topolvm-operator.alpha.arm64.v4.2.4.tgz` when the nodes are ARM): + +```bash +kubectl get nodes -o custom-columns=NAME:.metadata.name,ARCH:.status.nodeInfo.architecture +``` + +## Diagnostic Steps + +1. A CIDR line that shows a parameter name (`cluster-cidr`) or an empty value where a CIDR is expected: read `kubeadm-config` with `grep -A1` and compare with the overview page and, on Kube-OVN, the `join` subnet. Expected for Cause 1: the `value:` line carries the same CIDR as the overview page. +2. `global集群不存在is-global的label`: the two `--show-labels` commands of Cause 2. Expected for Cause 2: at least one of them prints nothing. +3. `已部署 NFS 插件`: read `vers=` from `/proc/mounts` and list the PVs on the NFS StorageClass. No PVs: uninstall the unused plugin. `vers=3` with PVs in use: add `nolock`. `vers=4` or later: the NFSv3 condition does not apply. +4. `global Cluster Platform Node Isolation`: check the node isolation policy of the global cluster. Not enabled: ignore. Enabled: contact technical support. diff --git a/docs/en/solutions/Preparing_a_host_before_adding_it_to_an_ACP_cluster_node_check_items_disks_init_script_DefaultTasksMax_root_access.md b/docs/en/solutions/Preparing_a_host_before_adding_it_to_an_ACP_cluster_node_check_items_disks_init_script_DefaultTasksMax_root_access.md new file mode 100644 index 000000000..319555c41 --- /dev/null +++ b/docs/en/solutions/Preparing_a_host_before_adding_it_to_an_ACP_cluster_node_check_items_disks_init_script_DefaultTasksMax_root_access.md @@ -0,0 +1,182 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - "4.1" + - "4.2" + - "4.3" +--- + +# Preparing a host before adding it to an ACP cluster: node check items, disks, init script, DefaultTasksMax, root access + +## Issue + +Before deploying the platform or adding a node, the following questions come up: + +- Which operating-system and kernel parameters must be changed on the host, and why? +- How much disk does a new worker node need, and which mount points? +- What has to be run on a new server before it is added to an existing cluster? +- Why did the initialization script comment out `UsePAM` in `/etc/ssh/sshd_config`? +- What should `DefaultTasksMax` be, and which ports does a business cluster need? +- Can the deployment use a non-root user with limited rights? + +Skipping these steps shows up later as node availability check failures, kernels the CNI cannot work with, or installs that stop on a missing package. + +Related articles: "Adding a node to an ACP cluster fails on UseDNS or UsePAM, the node does not boot after init.sh, or a cloned node fails at yum" covers failures during the add; "Which operating systems, kernels, and CPU architectures can run in an ACP cluster" covers OS, kernel, and architecture support. + +## Root Cause + +### Cause 1: the node check has mandatory items that depend on the platform version + +The requirements are version-specific and are published in the product documentation under **Installation → Node Preprocessing → Node Check** (on a running platform, open **Help Documentation** in the top-right corner). For ACP 4.1.6 the check splits configuration into items that must pass and scenario-specific checks. The mandatory items include: + +- Kernel command line: `transparent_hugepage=never`; on CentOS 7.x also `cgroup.memory=nokmem`. +- Kernel modules: the `ip_vs` series and `nf_conntrack`; with Kube-OVN as CNI also `geneve` and `openvswitch`. +- AppArmor / SELinux, the host firewall, and swap. +- SSH: the user's permissions, and `UseDNS` and `UsePAM` in `sshd_config`. + +### Cause 2: the initialization script comments out UsePAM + +The platform drives nodes through automated SSH logins and remote commands. The initialization script comments out `UsePAM` (which was `yes`) so that PAM modules do not interfere with those automated logins and commands. The node check inspects `UsePAM` and `UseDNS`, and warns with `/etc/ssh/sshd_config文件中 UsePAM 参数被注释或值不为 no,请调整` (the `UsePAM` parameter in `/etc/ssh/sshd_config` is commented out or not `no`; adjust it). That warning is non-blocking: node preprocessing sets it to prevent SSH timeouts during the add, and it is not a problem once the node has joined. + +### Cause 3: worker disk layout follows usage, not the existing nodes + +For ACP 4.3.1 worker nodes, the recommended starting point is: + +| Mount point | Minimum | +|---|---| +| `/cpaas` | 100 GB | +| `/var/lib/containerd` | 200 GB | + +The real need depends on the number of Pods, image sizes and how many versions are kept, temporary container data, log volume, and local storage. Disks can start at the initial size and be expanded as usage grows. A new node does not have to copy the disks of the existing nodes. A node that will later host log or monitoring storage needs `/cpaas` on its own mount. + +### Cause 4: a new server must be initialized before it is added + +To add a server to a business cluster (for example on ACP 4.2.1): check the node against the deployment documentation of that version, run `init.sh` from the installer package of the same version on the node, then add the node through the platform. The `init.sh` kept on the first control-plane node of `global` (under `/root`) can also be copied to the new node and run there. + +### Cause 5: DefaultTasksMax must be infinity, and the business cluster ports + +For a production plan on Kylin (kernel `4.19.90-89.26.v2401`), two check items apply: `DefaultTasksMax` in `/etc/systemd/system.conf` must be `infinity`, and a 4.x business cluster needs ports 6443 and 11780 open for this purpose; 11781 is not needed. A change to `/etc/systemd/system.conf` takes effect only after `systemctl daemon-reexec`; `daemon-reload` rereads unit files but not the manager configuration. After `daemon-reexec`, only units started afterwards get the new default. + +### Cause 6: the deployment needs root or sudo on the nodes + +Deployment creates directories such as `/var/lib/containerd/`, `/var/lib/kubelet/`, `/var/lib/etcd/`, and `/cpaas/`, and runs partitioning commands. A non-root user with a restricted set of rights is not supported; the deployment needs `root` or a user with sudo rights. + +## Resolution + +### Cause 1: the node check has mandatory items that depend on the platform version + +Apply the items from the Node Check page of your version. The commands below cover the items listed above. + +> **Warning:** Changing the kernel command line requires a reboot, and a typo in the boot entry can leave the host unbootable. Do it before the host joins, or drain it first. Rollback: remove the arguments with the same tool (`grubby --update-kernel=ALL --remove-args=...`) and reboot. Disabling swap and the firewall changes host behavior for every process; follow your site's security approval for the firewall item. + +```bash +# kernel command line (distributions that use grubby, such as CentOS and Kylin) +cat /proc/cmdline +sudo grubby --update-kernel=ALL --args="transparent_hugepage=never" +# CentOS 7.x only +sudo grubby --update-kernel=ALL --args="cgroup.memory=nokmem" + +# kernel modules, loaded now and at boot +sudo modprobe -a ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack +printf '%s\n' ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack | sudo tee /etc/modules-load.d/acp.conf +# Kube-OVN clusters only +sudo modprobe -a geneve openvswitch +printf '%s\n' geneve openvswitch | sudo tee -a /etc/modules-load.d/acp.conf + +# swap +sudo swapoff -a +``` + +Comment out the swap line in `/etc/fstab` so swap stays off after a reboot. Reboot once after the command-line change and confirm with `cat /proc/cmdline`. + +### Cause 2: the initialization script comments out UsePAM + +The change is intentional; keep it. If the availability check warns about `UsePAM` or `UseDNS`, set both explicitly to `no`, then check the values sshd actually uses: + +```bash +sudo grep -n -E '^\s*#?\s*(UsePAM|UseDNS)' /etc/ssh/sshd_config +# after setting "UsePAM no" and "UseDNS no": +sudo sshd -t && echo "sshd_config syntax OK" +sudo systemctl restart sshd +sudo sshd -T | grep -i -E '^(usepam|usedns) ' +``` + +If a site baseline requires PAM for SSH logins, raise it with Alauda technical support before re-enabling `UsePAM`. + +### Cause 3: worker disk layout follows usage, not the existing nodes + +Create the mounts before the node is added, sized from the table and the expected workload: + +```bash +lsblk -o NAME,SIZE,TYPE,MOUNTPOINT,FSTYPE +df -h /cpaas /var/lib/containerd +findmnt /cpaas /var/lib/containerd +``` + +Plan a separate `/cpaas` volume on any node that may later become a log or monitoring storage node. + +### Cause 4: a new server must be initialized before it is added + +1. Run the checks of your version on the new server. +2. Run the installer's `init.sh` of the same version: + + ```bash + # on the new server, from the extracted installer package of the same version + sudo bash ./init.sh + ``` + + The script path inside the package can differ between versions; use the one named in that version's deployment documentation. Start the scripts with `bash`, not `sh`; an add-node script run with `sh` fails on its syntax. + +3. Add the node from the cluster's node page, and check afterwards that its monitoring metrics are collected. + +### Cause 5: DefaultTasksMax must be infinity, and the business cluster ports + +> **Warning:** `systemctl daemon-reexec` restarts the systemd manager in place; running services keep running, but a broken `/etc/systemd/system.conf` affects every unit started afterwards. Apply it before the node joins, after backing up the file. Rollback: restore the backup and run `systemctl daemon-reexec` again. + +```bash +sudo cp /etc/systemd/system.conf /root/system.conf.bak +if grep -qE '^\s*#?\s*DefaultTasksMax=' /etc/systemd/system.conf; then + sudo sed -ri 's/^\s*#?\s*DefaultTasksMax=.*/DefaultTasksMax=infinity/' /etc/systemd/system.conf +else + echo 'DefaultTasksMax=infinity' | sudo tee -a /etc/systemd/system.conf +fi +sudo systemctl daemon-reexec +systemctl show --property=DefaultTasksMax +``` + +The expected output is `DefaultTasksMax=infinity`. Units that are already running keep the old value until they restart or the node reboots; because node preprocessing happens before installation, containerd and kubelet start with the new value. + +Open ports 6443 and 11780 for the business cluster as listed above. + +### Cause 6: the deployment needs root or sudo on the nodes + +Give the deployment user root, or sudo without a password prompt (the availability check runs sudo non-interactively): + +```bash +id +sudo -l -U +``` + +## Diagnostic Steps + +Run on the host before adding it, and compare with a node already in the cluster: + +```bash +cat /etc/os-release; uname -r +cat /proc/cmdline +cat /sys/kernel/mm/transparent_hugepage/enabled +lsmod | grep -E 'ip_vs|nf_conntrack|geneve|openvswitch' +swapon --show +getenforce 2>/dev/null || true +systemctl is-active firewalld 2>/dev/null || true +sudo sshd -T | grep -i -E '^(usepam|usedns) ' +df -h /cpaas /var/lib/containerd +hostname +systemctl show --property=DefaultTasksMax +rpm -qa | grep -i -E 'docker|podman' || true +``` + +Expected results: `[never]` in the transparent hugepage file, the listed modules loaded, an empty `swapon --show`, `usepam no` and `usedns no`, `DefaultTasksMax=infinity`, a lowercase hostname, and no preinstalled Docker or podman packages. A missing module or an active `UsePAM` appears in the availability check as a failure or warning. diff --git a/docs/en/solutions/Preparing_the_extension_package_list_for_an_ACP_upgrade_fails_or_is_incomplete_violet_list_and_apps_yaml.md b/docs/en/solutions/Preparing_the_extension_package_list_for_an_ACP_upgrade_fails_or_is_incomplete_violet_list_and_apps_yaml.md new file mode 100644 index 000000000..1332aba2e --- /dev/null +++ b/docs/en/solutions/Preparing_the_extension_package_list_for_an_ACP_upgrade_fails_or_is_incomplete_violet_list_and_apps_yaml.md @@ -0,0 +1,184 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "4.0" + - "4.2" + - "4.3" +--- + +# Preparing the extension package list for an ACP upgrade fails or is incomplete: violet list and apps.yaml + +## Issue + +While collecting the extension packages for a platform upgrade: + +- The extension list exported with `violet` is pasted into the Alauda Customer Portal download page and the upload returns an error. After the YAML is fixed, Marketplace > Batch Download > Upgrade > Post-ACP v4.0 Upgrades (for example target `v4.3.0`, architecture `amd64`) shows `No Application` in the Packages list. +- After `apps.yaml` is imported, many packages are listed with several versions each. +- `apps.yaml` entries do not match on the Alauda Customer Portal (`logapi`, `logagent` reported missing), a scenario category misses `log-center`, and some entries show "no package for this version" (`暂无此版本包`). +- Before an upgrade (for example from 4.2.4 to 4.3) some extension packages are not yet released, and self-uploaded plugins have no matching version. +- `violet list` fails on 3.16.2, or after the upgrade from 3.16.x to 4.0.12 the package lookup still goes through `app.yaml`. +- While upgrading DevOps to v3.20 following the DevOps v3 upgrade documentation, the `Alauda DevOps Nexus v3` and `Alauda DevOps Harbor v3` packages cannot be found. +- On a new 4.0.7 `global` deployed with the 4.0.7 core package, the 4.0.7 extension package was pushed with `violet push`, and the VM monitoring, log collection, and ES storage plugins still show as not listed (`未上架`). +- Packages that are published (`已上架`) but not installed should be taken down (`下架`) so that production matches a test environment. +- `violet list --platform-address ... --platform-token ... --output-file ./apps.yaml` returns `null` for every plugin before an upgrade to 4.3.3, although Cluster Plugins on `global` shows installed plugins. + +## Root Cause + +### Cause 1: The extension YAML lacks the AppScenarioManifest header, or lists plugins that are not installed + +The exported YAML starts directly with the `applications:` key and lacks a top-level `AppScenarioManifest:` line above it, which causes the upload error. Separately, plain `violet list` lists every package uploaded to the platform, not only the installed ones, which produces the `No Application` ("no updates needed") result. `violet list --installed` lists only installed plugins. Adding the `AppScenarioManifest:` line and regenerating the list with `violet list --installed` fixes both. If some target-version plugins are still being released, wait for their packages before upgrading. + +### Cause 2: The upgrade prompt lists every compatible version of each package + +By default the upgrade prompt lists every extension package version that is compatible with the target version. Downloading all listed versions is the expected default. + +### Cause 3: An artifact ID changed or a scenario category missed a package + +When new artifact versions were published, some artifact IDs changed: the names gained a `-` (`log-api`, `log-agent`), so the Alauda Customer Portal treats the new version as a different application and the old `apps.yaml` entry no longer matches. Both exist under scenario category `common ACP Extension`. The 4.0 names `logagent` and `logcenter` became `log-agent` and `log-center` in 4.3. `log-center` was missing from that category and has since been added. + +For an entry with "no package for this version", the portal cannot tell whether the component was removed in the target version or merged into the core package; confirm each entry by hand. For `logagent` and `logcenter` it is the rename above. For `harbor-operator` and `nexus-operator` on 4.3 it means DevOps v3 is not supported there; migrate to DevOps v4. + +### Cause 4: Some packages for the target version are not released yet, or have no platform version + +The upgrade is held because some required extension packages are not yet released, and self-uploaded plugins show no match. Comparing every package in the environment's `apps.yaml` with the available target-version packages gives two lists: packages to download for the upgrade, and self-uploaded plugins with no matching platform version. An installed platform-provided plugin with no released target package holds the upgrade until the package is released. Self-uploaded plugins with no platform version do not affect the upgrade. For 4.3: OAM has no 4.3 packages, so nothing is downloaded for it; Ingress-nginx is no longer available as a plugin, and `ingress-nginx-operator`, included in the core package, replaces it. + +The same gap applies to the DevOps v3.20 upgrade: the `Alauda DevOps Nexus v3` and `Alauda DevOps Harbor v3` packages were published only after a request to Alauda. + +### Cause 5: After 3.16 to 4.0.12 the package list must not be taken from app.yaml + +For the upgrade from ACP 3.16.x to 4.0.12, the package list is not retrieved through `app.yaml`. The packages for that upgrade come from the dedicated `ACP Upgrade from v3.x to v4.x` batch-download scenario. `violet list` reads the application list already present in the environment and is the inventory to compare against that download set; plain `violet list` also includes packages that are uploaded but not installed. Later upgrades that start on 4.0 or later do import `apps.yaml` on the Post-ACP v4.0 Upgrades page (Cause 1). No cause is known for the `violet list` failure on 3.16.2; `violet list` works on a 3.16 `global` control-plane node with the violet client that matches the target version. + +### Cause 6: The published 4.0.7 extension package was missing plugins + +The 4.0.7 extension package published at one time did not contain several plugins. The package was rebuilt and published again; download the corrected package. + +### Cause 7: Published but not installed packages do not have to be taken down + +Packages that are published but not installed do not affect the upgrade process, and there is no takedown procedure for them. Leave them in place. + +### Cause 8: violet list returns null for every plugin before the upgrade to 4.3.3 + +With the `violet` version used before the upgrade to 4.3.3, `violet list` outputs only the application catalog and does not show installation status, so the installed state comes back as `null`. This is expected and does not mean the plugins are uninstalled. + +## Resolution + +### Cause 1: The extension YAML lacks the AppScenarioManifest header, or lists plugins that are not installed + +Regenerate the list from installed plugins only: + +```bash +violet list --installed --platform-address --platform-username --platform-password '' --output-file apps.yaml +head -n 5 apps.yaml +``` + +If the file does not begin with `AppScenarioManifest:`, add that line above `applications:` (entries carry the installed cluster and version and a `type` of `ModulePlugin`, `OperatorBundle`, or `Chart`): + +```bash +sed -i '1i AppScenarioManifest:' apps.yaml +head -n 3 apps.yaml +``` + +Import the file again on the upgrade download page and confirm that the Packages list shows updates for the plugins that are actually installed. + +### Cause 2: The upgrade prompt lists every compatible version of each package + +Download the versions the upgrade prompt lists; they are all compatible with the target version. + +### Cause 3: An artifact ID changed or a scenario category missed a package + +For each entry the portal reports as missing: + +1. Search for the name with a `-` added (`log-api`, `log-agent`) and in scenario category `common ACP Extension`. +2. Check that `log-center` is in your download set. +3. For each "no package for this version" entry, confirm in the target version's release notes or with Alauda support whether the component was removed or merged into the core package. Drop it from the list only after that confirmation. + +Compare the final download set with the installed list: + +```bash +violet list --installed --platform-address --platform-token '' +``` + +### Cause 4: Some packages for the target version are not released yet, or have no platform version + +When some target packages are not released or plugins show no match, send the environment's `apps.yaml`, the architecture, and the current platform version to Alauda support for a comparison, and download the packages on the resulting "must download" list. When the portal import shows every required package as available, no review is needed. Leave self-uploaded plugins with no platform version out of the upgrade. For OAM on 4.3 there is nothing to download; for Ingress-nginx, plan to install `ingress-nginx-operator` from the core package. If a package that a product upgrade document requires cannot be found on the download site, report the package names to Alauda support. + +```bash +kubectl get nodes -o custom-columns=NAME:.metadata.name,ARCH:.status.nodeInfo.architecture +``` + +### Cause 5: After 3.16 to 4.0.12 the package list must not be taken from app.yaml + +Download the packages for the 3.16.x to 4.0.12 upgrade from the batch-download scenario `ACP Upgrade from v3.x to v4.x` (App Marketplace > Batch Download > Upgrade). On the environment being upgraded, list what is installed, on the first control-plane node of the `global` cluster and with the violet client that matches the target version: + +```bash +violet list +``` + +Use that list as the bill of extension packages to download, leaving out rows for packages that are uploaded but not installed. If `violet list` still fails with the target-version client, contact Alauda support. + +### Cause 6: The published 4.0.7 extension package was missing plugins + +List what the downloaded package contains and compare it with the plugins that show as not listed: + +```bash +tar -tf | grep -i -E 'monitor|log|elasticsearch|es' +kubectl get moduleplugins | grep -i -E 'monitor|log|elasticsearch' +``` + +If the plugins are not in the archive, download the extension package again from the official site, push it, and check Cluster Plugins: + +```bash +violet push --platform-address --platform-username --platform-password '' +``` + +If the republished package still lacks them, report the missing plugin names to Alauda support. + +### Cause 7: Published but not installed packages do not have to be taken down + +Confirm that the packages in question are really not installed: + +```bash +violet list --platform-address --platform-username --platform-password '' +violet list --installed --platform-address --platform-username --platform-password '' +``` + +A package that appears only in the first list is published but not installed; leave it in place. An installed package is not covered by this conclusion. + +### Cause 8: violet list returns null for every plugin before the upgrade to 4.3.3 + +First rule out a command that did not reach the platform. Use the real address and token, not the template placeholders: + +```bash +violet version +violet list --platform-address "https://" --platform-token "" --output-file ./apps.yaml +ls -l ./apps.yaml +cat ./apps.yaml +``` + +If the file is written and the plugins still show `null`: + +1. Record the installed plugins and their versions from Cluster Plugins on the `global` cluster. That list is the inventory for the upgrade. +2. After the platform is upgraded to 4.3.3, run the inventory again with the `violet` client that matches 4.3.3. + +## Diagnostic Steps + +```bash +violet version +violet list --installed --platform-address --platform-token '' +head -n 3 apps.yaml +``` + +- A portal upload error on a YAML whose first line is `applications:` rather than `AppScenarioManifest:`, or `No Application` from a list that includes uninstalled plugins: Cause 1. +- Many versions per package after importing `apps.yaml`: Cause 2. +- `logapi` / `logagent` missing, `log-center` missing from a scenario, or "no package for this version": Cause 3. +- Target packages not yet released, self-uploaded plugins with no match, or packages required by a product upgrade document missing from the download site: Cause 4. +- A 3.16 to 4.0.12 lookup that still depends on `app.yaml`: Cause 5. +- Plugins still not listed after the whole extension package was pushed, because the archive does not contain them: Cause 6. +- A request to take down published but not installed packages: Cause 7. +- `null` for every plugin from `violet list` before the upgrade to 4.3.3, while Cluster Plugins shows plugins installed: Cause 8. + +When `violet list` returns nothing useful, also check what the command covers and who runs it. `violet list` lists extensions with the Aligned or Agnostic life cycle (cluster plugins of those types, installed operators, and Helm chart applications); Core plugins follow the cluster upgrade and are never listed. The platform account must have role property `System` and role name `platform-admin-system`; a `Custom` role cannot use the tool. Token-based inventory needs a recent `violet` client; older clients can fail with `failed to get user info from platform server`. diff --git a/docs/en/solutions/Project_administrators_lose_permission_to_manage_template_repositories_in_Catalog_after_upgrading_to_ACP_4.3.md b/docs/en/solutions/Project_administrators_lose_permission_to_manage_template_repositories_in_Catalog_after_upgrading_to_ACP_4.3.md new file mode 100644 index 000000000..a15de99e6 --- /dev/null +++ b/docs/en/solutions/Project_administrators_lose_permission_to_manage_template_repositories_in_Catalog_after_upgrading_to_ACP_4.3.md @@ -0,0 +1,91 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.3.x +--- + +# Project administrators lose permission to manage template repositories in Catalog after upgrading to ACP 4.3 + +## Issue + +After the platform is upgraded to ACP 4.3, users who hold the project administrator role can no longer manage template repositories in **Catalog** > **Helm Charts**. Before the upgrade, the same users could upload new chart versions to the template repositories assigned to their project. After the upgrade, the users have no permission to manage the template repositories. + +## Root Cause + +### Cause 1: the acp-catalog rule of the project administrator RoleTemplate is read-only after the upgrade + +Platform roles are defined by `RoleTemplate` objects on the `global` cluster. Each entry in `spec.rules` points at a `FunctionResource` through `functionResourceRef` and lists the verbs the role gets on that function. The built-in project administrator role is backed by the RoleTemplate `legacy-project-admin-system`. + +ACP 4.3 changed the permissions for template repositories. After the upgrade, the `legacy-project-admin-system` rule with `functionResourceRef: acp-catalog` grants only `get`, `list`, and `watch`, which are read-only verbs, while the neighbouring management rules of the same template (for example `acp-batchaction`, `acp-imagewhitelist`, and `acp-autoscalers`) grant `"*"`. Every user bound to the project administrator role therefore loses write access to template repositories. Changing the verbs of the `acp-catalog` rule in that RoleTemplate restores the permission. + +## Resolution + +### Cause 1 + +Run these steps against the `global` cluster with a platform administrator account. + +1. Save the current RoleTemplate so that you can roll back: + + ```bash + kubectl get roletemplate legacy-project-admin-system -o yaml > legacy-project-admin-system.bak.yaml + ``` + +2. Confirm the signature of this cause (see Diagnostic Steps, step 1): the `acp-catalog` rule lists only `get`, `list`, and `watch`. + +3. Grant management verbs on the `acp-catalog` rule. + + > **Warning:** This changes a built-in role. Every user bound to the project administrator role, in every project, gets the new permission on template repositories as soon as the platform regenerates the role. Do it only when step 2 confirmed that the `acp-catalog` rule is read-only and project administrators are expected to manage template repositories. To roll back, edit the RoleTemplate again and set the `acp-catalog` verbs back to the values saved in `legacy-project-admin-system.bak.yaml`. + + ```bash + kubectl edit roletemplate legacy-project-admin-system + ``` + + Find the rule for `acp-catalog` and change its verbs, for example to the same value that the neighbouring management rules use: + + ```yaml + spec: + rules: + - functionResourceRef: acp-catalog + verbs: + - "*" + ``` + + Leave all other rules unchanged. + +4. Ask an affected user to sign in again and repeat the template repository action, for example **Catalog** > **Helm Charts** > **Add Template**, select a repository assigned to the project, then **Upload Template**. The template status changes from `Uploading` to `Upload Successful`. + +5. The change is made on a built-in role. After every later platform upgrade, run Diagnostic Steps, step 1 again to check that the `acp-catalog` rule still has the verbs you set. + +## Diagnostic Steps + +1. On the `global` cluster, list the rules of the project administrator RoleTemplate: + + ```bash + kubectl get roletemplate legacy-project-admin-system \ + -o jsonpath='{range .spec.rules[*]}{.functionResourceRef}{" "}{.verbs}{"\n"}{end}' + ``` + + Output that matches this cause: + + ```text + acp-catalog ["get","list","watch"] + acp-batchaction ["*"] + acp-imagewhitelist ["*"] + acp-autoscalers ["*"] + ... + ``` + + If `acp-catalog` already has `"*"` or other write verbs, the role is not the problem; continue with step 3. + +2. To see which resources the `acp-catalog` function covers, read the FunctionResource: + + ```bash + kubectl get functionresource acp-catalog -o yaml + ``` + +3. Rule out the look-alike: template repositories are added by platform administrators, and a project can upload templates only to a Chart or OCI Chart repository that is assigned to it with **Management** permission. In the console, check that the repository the user is trying to manage is assigned to the user's project with that permission. A repository that is not assigned to the project, or is assigned without **Management**, is refused for every project user, whatever the role grants. + +4. Check that the affected user actually holds the project administrator role in that project (platform **Role Management** or the user's role list). A user with a custom role is governed by that role's RoleTemplate, not by `legacy-project-admin-system`; inspect its `acp-catalog` rule with the command in step 1, replacing the template name. diff --git a/docs/en/solutions/Project_quota_and_namespace_ResourceQuota_aggregate_limits_overcommit_Pod_count_and_node_capacity_on_ACP.md b/docs/en/solutions/Project_quota_and_namespace_ResourceQuota_aggregate_limits_overcommit_Pod_count_and_node_capacity_on_ACP.md new file mode 100644 index 000000000..6717a711f --- /dev/null +++ b/docs/en/solutions/Project_quota_and_namespace_ResourceQuota_aggregate_limits_overcommit_Pod_count_and_node_capacity_on_ACP.md @@ -0,0 +1,202 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.16" + - "4.1" +--- + +# Project quota and namespace ResourceQuota: aggregate limits, overcommit, Pod count, and node capacity on ACP + +## Issue + +Quota behavior looks wrong, or the quota model needs explaining: + +- A project has CPU and memory quota on a cluster. Two namespaces are created in it without quota numbers, and their combined usage is expected to stay inside the project quota. +- A project has a storage quota, the namespace was created without one, and the Container Platform view of the namespace shows no storage quota. +- A namespace cannot hold more than 1000 Pods, and the console offers no field to raise that number. +- The namespace Pod limit "does not take effect": the limit is 20, yet more than 30 Pods exist in the namespace, many of them `Completed`, and the quota shows something like `pods: 4/20`. +- Cluster resources are almost fully allocated to projects while real utilization is below 10 percent. Can projects be overcommitted? +- A node has 96 physical CPU cores but `Allocatable` shows 92. How do project quotas map to physical node resources, and why do the CPU and memory totals on the project page not match real usage? +- How are business Pods kept off the nodes that host logging and monitoring components? +- A namespace has more than one quota object, for example the platform quota `default` and a hand-made `-quota`, with different values for the same resource. Which one applies? + +## Root Cause + +### Cause 1: Namespaces without their own quota do not share the project quota as a combined limit + +The project quota is a configuration boundary for allocation. Resource limits are enforced per namespace, by the namespace `ResourceQuota`. Namespaces cannot be left without quota numbers and still have their combined usage bounded by the project quota. + +### Cause 2: A namespace without its own storage quota shows none + +When only the project has a storage quota and none is allocated to the namespace, the Container Platform view shows no storage quota for the namespace. This is expected. The namespace storage limit is then determined by the maximum available storage. + +### Cause 3: The 1000 Pods per namespace cap is `spec.hard.pods` of the namespace ResourceQuota + +On ACP 3.16 the cap of 1000 Pods per namespace is the `spec.hard.pods` value of the namespace's `ResourceQuota`. The console does not expose that value, but it can be changed on the `ResourceQuota` directly. + +### Cause 4: The Pod quota does not count Completed Pods + +The `pods` entry of a `ResourceQuota` counts only Pods in a non-terminal state. A Pod whose `.status.phase` is `Succeeded` or `Failed` is terminal, and `Completed` Pods are in that state. On ACP 4.1.2, for example, the quota `default` shows `pods: 4/20` while 31 Pods exist. Native `ResourceQuota` cannot be made to count completed Pods; this is Kubernetes design (see the Kubernetes `ResourceQuota` object count documentation). + +### Cause 5: Project quotas may add up to more than the cluster has + +The platform supports overcommit between projects. With 200 CPU cores in the cluster, projects A, B, and C can each be allocated 80 cores, 240 in total. Inside one project the namespace quotas are bounded by the project quota: with namespaces `ns1` and `ns2` in project A, `ns1` can have 30 cores and `ns2` 50 cores, as long as `ns1 + ns2 <= 80`. + +### Cause 6: Project quotas are administrator limits, not node capacity, and Allocatable excludes reserved resources + +- Project quotas are based on Kubernetes `ResourceQuota`, which limits `requests.cpu`, `requests.memory`, `limits.cpu`, `limits.memory`, Pod and PVC counts, and extended resources. The effective values are those in `status.hard`. +- A `ResourceQuota` is an upper limit set by an administrator. It is not calculated from physical resources and is not bound to nodes. The platform does not keep the sum of project quotas within the schedulable capacity (that is the overcommit in Cause 5). +- Pods are scheduled by their `requests` against node `Allocatable`. Business capacity is the total `Allocatable` of the worker nodes that run business workloads, minus system components, DaemonSets, a reserve for node failures, and a reserve for rolling upgrades. +- `Allocatable` lower than `Capacity` (92 of 96 cores) is the resource reserved for the system and Kubernetes components through the kubelet `kubeReserved`, `systemReserved`, or `reservedSystemCPUs` settings. +- The CPU and memory totals on the project page are a resource overview. They do not combine actual usage and, with many projects, cannot be calculated accurately. + +### Cause 7: Keeping business Pods off logging and monitoring nodes needs taints and tolerations + +Without taints, any Pod can be scheduled on the logging and monitoring nodes. Node labels, `NoSchedule` taints, tolerations, and node affinity make those nodes a separate infrastructure pool. Components that already tolerate the taint keep running there (for example, ClickHouse carries an `infra` toleration, as do most platform components); ordinary business Pods are no longer scheduled there. Components deployed later, such as MetalLB, must be checked for the toleration separately. + +### Cause 8: Several ResourceQuota objects in one namespace all apply + +Kubernetes evaluates every `ResourceQuota` in the namespace. Creating or scaling a Pod must fit all of them, so for each resource the effective limit is the smallest hard value among the quotas that list it. Resources listed in only one quota are limited by that quota alone. The platform-managed quota is the one named `default`. For example, with `default` at `pods: 1k`, `requests.cpu: 128` and a second quota `-quota` at `pods: 10k`, `cpu: 120`, the namespace can run at most 1000 Pods and CPU requests up to 120 cores. In a `ResourceQuota`, `cpu` and `requests.cpu` (and `memory` and `requests.memory`) both limit the sum of requests, so they count as the same resource when comparing quotas. + +## Resolution + +### Cause 1: Namespaces without their own quota do not share the project quota as a combined limit + +Give each namespace its own quota, with the namespace values adding up to no more than the project quota (Cause 5). + +```bash +kubectl get resourcequota -n -o yaml +kubectl get resourcequota -n -o yaml +``` + +### Cause 2: A namespace without its own storage quota shows none + +No action is needed. To confirm, look for a storage entry (`requests.storage` or a per-StorageClass entry) in the namespace quota: + +```bash +kubectl get resourcequota -n -o yaml | grep -i storage +``` + +No output means the namespace has no storage quota of its own. + +### Cause 3: The 1000 Pods per namespace cap is `spec.hard.pods` of the namespace ResourceQuota + +Find the quota object (on ACP 4.1 the namespace quota is normally named `default`) and save it: + +```bash +kubectl get resourcequota -n +kubectl get resourcequota -n -o yaml > "$HOME/resourcequota-.yaml" +``` + +> **Warning:** This changes how many non-terminal Pods the namespace may run. A value below the current count blocks new Pods until existing ones end; a higher value lets the namespace create more Pods on the cluster. Do it only when the cap comes from `spec.hard.pods` of this `ResourceQuota`. Roll back with `kubectl apply -f` of the saved file or by patching the previous value back. + +```bash +kubectl patch resourcequota -n --type=merge -p '{"spec":{"hard":{"pods":""}}}' +kubectl get resourcequota -n -o jsonpath='{.spec.hard.pods}{"\t"}{.status.hard.pods}{"\n"}' +``` + +Both values should show the new limit. It is not known whether a later quota save in the console keeps this value; read `spec.hard.pods` again after any console edit of that quota. + +### Cause 4: The Pod quota does not count Completed Pods + +No fix exists in native `ResourceQuota`. Confirm the numbers agree: + +```bash +kubectl get resourcequota -n +kubectl get pods -n --field-selector=status.phase!=Succeeded,status.phase!=Failed --no-headers | wc -l +kubectl get pods -n --field-selector=status.phase=Succeeded --no-headers | wc -l +``` + +The first count matches the used `pods` value of the quota; the second is the `Completed` Pods it does not count. + +### Cause 5: Project quotas may add up to more than the cluster has + +Overcommit between projects is allowed. Within one project, keep the sum of namespace quotas at or below the project quota. + +```bash +kubectl get resourcequota -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,LIMITS_CPU:.status.hard.limits\.cpu,USED_LIMITS_CPU:.status.used.limits\.cpu' +``` + +Pods are still placed only where node `Allocatable` has room for their `requests` (Cause 6). + +### Cause 6: Project quotas are administrator limits, not node capacity, and Allocatable excludes reserved resources + +Plan project quotas from business capacity as defined above, and read the node reserve from the kubelet configuration: + +```bash +kubectl get nodes -o custom-columns='NAME:.metadata.name,CPU_CAPACITY:.status.capacity.cpu,CPU_ALLOCATABLE:.status.allocatable.cpu,MEM_CAPACITY:.status.capacity.memory,MEM_ALLOCATABLE:.status.allocatable.memory' +kubectl get --raw "/api/v1/nodes//proxy/configz" | jq '.kubeletconfig | {kubeReserved, systemReserved, reservedSystemCPUs}' +``` + +The difference between capacity and allocatable should match the reserved values (plus any eviction threshold for memory). Use monitoring, not the project page totals, for real CPU and memory usage. + +### Cause 7: Keeping business Pods off logging and monitoring nodes needs taints and tolerations + +Before tainting, read the tolerations of every Pod that must stay on those nodes, and use the key those tolerations already match (`infra` for ClickHouse): + +```bash +kubectl get pods -A -o wide --field-selector spec.nodeName= +kubectl get pod -n -o jsonpath='{.spec.tolerations}{"\n"}' +``` + +> **Warning:** A `NoSchedule` taint stops new Pods without a matching toleration from being scheduled on the node, including platform add-ons deployed later that lack it. Running Pods are not evicted. Taint only after confirming that every component that must run on these nodes tolerates it. Roll back by removing the taint with the trailing `-` command. + +```bash +kubectl label nodes = +kubectl taint nodes =:NoSchedule +# rollback +kubectl taint nodes =:NoSchedule- +``` + +Afterwards, check that the logging and monitoring Pods still schedule on those nodes, and check later components such as MetalLB for the toleration. + +### Cause 8: Several ResourceQuota objects in one namespace all apply + +List what each quota limits and compare the hard values per resource: + +```bash +kubectl get resourcequota -n +kubectl get resourcequota -n -o custom-columns='NAME:.metadata.name,HARD:.status.hard,USED:.status.used' +``` + +For each resource, the smallest hard value is the one that blocks a new Pod. To make one value authoritative, change the quota that is stricter, or remove the extra quota object and keep `default`. + +> **Warning:** Deleting a `ResourceQuota` removes the limits it alone enforces. Delete only a quota that was added by hand, never `default`, and save it first (`kubectl get resourcequota -n -o yaml > .yaml`). Roll back with `kubectl apply -f .yaml`. + +After an edit, compare `spec.hard` with `status.hard`: the quota controller updates `status.hard` shortly after `spec.hard` changes, and admission uses the new value once it is there. + +## Diagnostic Steps + +1. Decide whether the question is about enforcement or display. List the quota objects and the project quota: + + ```bash + kubectl get resourcequota -A + kubectl get projectquotas.auth.alauda.io -A + ``` + + Namespaces with no `ResourceQuota` that are expected to share the project limit: Cause 1. A namespace quota without a storage entry: Cause 2. More than one `ResourceQuota` in a namespace: Cause 8. + +2. For a Pod count question, read `spec.hard.pods` and `status.used.pods`: + + ```bash + kubectl get resourcequota -n -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.hard.pods}{"\t"}{.status.used.pods}{"\n"}{end}' + kubectl get pods -n --no-headers | awk '{print $3}' | sort | uniq -c + ``` + + A cap at 1000 that the console cannot change: Cause 3. More Pods listed than `status.used.pods`, the extra ones `Completed`: Cause 4. + +3. For allocation questions, compare the sum of project quotas with cluster capacity (Cause 5), and node `Capacity` with `Allocatable` and the kubelet reserve (Cause 6). + +4. For placement of business Pods, list node taints: + + ```bash + kubectl get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints' + ``` + + No taint on the logging and monitoring nodes: Cause 7. + +5. A rollout that stalls with `FailedCreate ... exceeded quota` is covered in [Deployment rollout stalls with FailedCreate exceeded quota and no built-in alert on ACP](Deployment_rollout_stalls_with_FailedCreate_exceeded_quota_and_no_built_in_alert_on_ACP.md). diff --git a/docs/en/solutions/Project_webhook_alert_notifications_miss_custom_headers_because_webhook-header-secret_is_only_in_cpaas-system_on_ACP.md b/docs/en/solutions/Project_webhook_alert_notifications_miss_custom_headers_because_webhook-header-secret_is_only_in_cpaas-system_on_ACP.md new file mode 100644 index 000000000..2049829ac --- /dev/null +++ b/docs/en/solutions/Project_webhook_alert_notifications_miss_custom_headers_because_webhook-header-secret_is_only_in_cpaas-system_on_ACP.md @@ -0,0 +1,91 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Project webhook alert notifications miss custom headers because webhook-header-secret is only in cpaas-system on ACP + +## Issue + +Alert notifications sent to a third-party Webhook endpoint work for notification rules created by the platform administrator, but fail for notification rules created under **Project Management** (often the ones that cover a workload cluster). The receiving endpoint rejects the request because a required custom header is missing, for example: + +```text +Required request header ... is not present +``` + +The platform-level setup follows the product documentation: a Secret `webhook-header-secret` of type `NotificationSender` exists in `cpaas-system` on the `global` cluster, and the platform receiver group carries the annotation `cpaas.io/notification.webhook.config: webhook-header-secret`. The question is then whether workload clusters and projects use the same Secret, and where to configure it. + +## Root Cause + +The custom headers of a Webhook notification come from a `NotificationSender` Secret that the receiver group (`NotificationGroup`, `ait.alauda.io`) references by name through the `cpaas.io/notification.webhook.config` annotation. Each key under the Secret's `data` becomes an HTTP header name. + +The product documentation only describes creating that Secret in `cpaas-system`, which is where platform-level receiver groups live. Notification rules and receiver groups created under **Project Management** are tenant-isolated and are stored in the namespace whose name is the project name. A receiver group in that namespace cannot reference a Secret in `cpaas-system`, so the headers are never added and the endpoint rejects the request. + +There is no separate per-cluster configuration: what matters is the namespace of the receiver group, not the cluster that raised the alert. + +## Resolution + +Create the header Secret in the namespace named after the project, on the `global` cluster, and reference it from the project's receiver group. + +1. On a control plane node of the `global` cluster, find the receiver group used by the failing notification rule and note its namespace (the project name): + + ```bash + kubectl get notificationgroups.ait.alauda.io -A + ``` + +2. Create the Secret in that namespace. Use the same header names as the platform Secret; every value must be base64-encoded: + + ```yaml + apiVersion: v1 + kind: Secret + metadata: + name: webhook-header-secret + namespace: + type: NotificationSender + data: + : + ``` + + ```bash + kubectl apply -f webhook-header-project.yaml + ``` + + To reuse the values of the platform Secret, print them with `kubectl get secret webhook-header-secret -n cpaas-system -o yaml` and copy the `data` section. + +3. Make sure the project receiver group references the Secret. Edit it and add the annotation if it is missing: + + ```bash + kubectl edit notificationgroups.ait.alauda.io -n + ``` + + ```yaml + metadata: + annotations: + cpaas.io/notification.webhook.config: webhook-header-secret + ``` + +4. Trigger or wait for an alert covered by the project notification rule and confirm that the endpoint no longer reports the missing header. + +Repeat the steps for every project whose notification rules send to a Webhook endpoint that requires custom headers. + +## Diagnostic Steps + +1. List the receiver groups with their namespace and header annotation. A failing group in a project namespace whose annotation points to a Secret that does not exist in that namespace matches this issue: + + ```bash + kubectl get notificationgroups.ait.alauda.io -A \ + -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,HEADER_SECRET:.metadata.annotations.cpaas\.io/notification\.webhook\.config' + ``` + +2. Check that the referenced Secret exists in the same namespace as the receiver group and has the expected type and keys: + + ```bash + kubectl get secret webhook-header-secret -n -o jsonpath='{.type}{"\n"}{.data}{"\n"}' + ``` + + Expected: `NotificationSender` and the header names the endpoint requires. `NotFound` means the Secret exists only in `cpaas-system`. + +3. If the platform receiver group in `cpaas-system` fails in the same way, the Secret is not namespace-mismatched. Check instead that the annotation is present on that group and that the Secret contains the header the endpoint asks for. diff --git a/docs/en/solutions/Pushed_Helm_chart_version_not_shown_because_violet_reused_an_Artifact_of_another_chart_repository_on_ACP.md b/docs/en/solutions/Pushed_Helm_chart_version_not_shown_because_violet_reused_an_Artifact_of_another_chart_repository_on_ACP.md new file mode 100644 index 000000000..a0edd7d6d --- /dev/null +++ b/docs/en/solutions/Pushed_Helm_chart_version_not_shown_because_violet_reused_an_Artifact_of_another_chart_repository_on_ACP.md @@ -0,0 +1,128 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Pushed Helm chart version not shown because violet reused an Artifact of another chart repository on ACP + +## Issue + +A middleware chart package (for example `nacos.ALL.v2.5.1-260420.tgz`) was pushed with `violet push` and the push finished, but the new version does not appear in the platform: Catalog > Helm Charts, the `public-charts` repository, and the version list show nothing new or only an old version. Pushing the same package again does not help; the `violet` output reports that everything already exists: + +```text +... already exists, skip push +artifact already exist, will create artifactversion cluster=global type=chart +artifactversion . already exist, skip it. cluster=global type=chart +``` + +The platform has more than one chart repository (for example the built-in chart repository and a Harbor chart repository), and the other repository already holds an older template of the same chart (for example `nacos` 2.2.3 in the Harbor chart repository, while the new package is `nacos` `v2.5.1-260420` for `public-charts`). + +The `artifact` / `artifactversion` objects exist, so this is not a push that never created them. + +## Root Cause + +### Cause 1: violet reused an Artifact owned by another chart repository + +The push succeeds: the images and the chart are stored, and an `ArtifactVersion` for the new tag is created in `cpaas-system` on the `global` cluster. The problem is the `Artifact` it is attached to. + +`Artifact` objects (`artifacts.app.alauda.io`) are named after the chart, and only one object with that name can exist in `cpaas-system`. When the chart name is already taken by an Artifact that belongs to a different chart repository, `violet` does not create a new Artifact. It logs `artifact already exist, will create artifactversion` and attaches the new `ArtifactVersion` to the existing Artifact through an ownerReference. For example: + +| Object | Labels and fields | +| --- | --- | +| Artifact `nacos` (existing) | `cpaas.io/chart-purpose: library-charts`, `cpaas.io/library: stable`, `cpaas.io/managed-by: violet`; `spec.artifactVersionSelector.matchLabels` `cpaas.io/artifact-version: nacos`; `imagePullSecrets` `ctr-library-charts`; `repository: middleware/chart-nacos` | +| ArtifactVersion `nacos.v2.5.1-260420` (new) | `cpaas.io/chart-purpose: public-charts`, `tag: v2.5.1-260420`, ownerReference to Artifact `nacos` | + +The repository label of the parent Artifact (`library-charts`) does not match the repository of the new version (`public-charts`), so the chart list for `public-charts` is not refreshed and the new version is not displayed. + +Because the `ArtifactVersion` already exists after the first push, every later push of the same package only logs `artifactversion . already exist, skip it`. Restarting platform components, re-syncing the chart repository, or pushing again does not change the parent Artifact. + +## Resolution + +### Cause 1: violet reused an Artifact owned by another chart repository + +Apply this only when all of the following are confirmed: + +- the `violet` output contains `artifact already exist, will create artifactversion`; +- `kubectl -n cpaas-system get artifact ` returns one Artifact whose `cpaas.io/chart-purpose` label names a different repository than the label on the new `ArtifactVersion`; +- the old chart in that other repository has the same name and is the version you intend to retire. + +1. Decide whether the old chart is still needed (for example, an old template that is incompatible with the current platform version and not used). Check whether any application still deploys from it: + + ```bash + kubectl get helmrequest -A | grep + kubectl get artifactversion -n cpaas-system --show-labels | grep + ``` + +2. Keep a copy of the objects before you change anything: + + ```bash + kubectl -n cpaas-system get artifact -o yaml > artifact--backup.yaml + kubectl -n cpaas-system get artifactversion . -o yaml > artifactversion--backup.yaml + ``` + +> **Warning:** Deleting the old template from the other chart repository removes that chart version from Catalog. The old version can no longer be selected for new deployments or upgrades, and applications deployed from it can no longer be upgraded or redeployed from that template. When the old `Artifact` is removed, Kubernetes garbage collection also removes the `ArtifactVersion` objects that list it as owner, including the version you just pushed; that is why the package is pushed again in step 4. Precondition: the `artifact already exist` signature and the repository-label mismatch are confirmed, the owner of the old chart agrees that it is retired, and the backups from step 2 and the old chart package (for example `helm pull oci://// --version `, or a download from Harbor) are saved. Rollback: upload the saved old package to its chart repository again; that brings the old template back, and with it the same name conflict. + +3. Delete the old `` template from the other chart repository (in the Harbor project that holds it). Then check whether the conflicting Artifact is gone: + + ```bash + kubectl -n cpaas-system get artifact --show-labels + ``` + + If it is still present with the other repository's label, collect `kubectl -n cpaas-system get artifact -o yaml` and contact Alauda support before pushing again. + +4. Push the new package again with the same command as before: + + ```bash + violet push .ALL..tgz \ + --platform-address "https://" \ + --platform-username "" \ + --platform-password "" + ``` + + This time `violet` creates the Artifact for the new chart. Images that are already present are still reported as `already exists, skip push`; that is expected. + +5. Verify the result. The `public-charts` repository lists `` under the published packages (Helm Chart), and Catalog > Helm Charts shows the application template with version ``: + + ```bash + kubectl -n cpaas-system get artifact --show-labels + kubectl -n cpaas-system get artifactversion . --show-labels + kubectl get artifactversion -A | grep + ``` + + Both objects now carry `cpaas.io/chart-purpose: public-charts`. + +## Diagnostic Steps + +1. Read the `violet push` output. `already exists, skip push` on the images only means the images were pushed before. The line that identifies this cause is `artifact already exist, will create artifactversion cluster=global type=chart`, followed on a re-push by `artifactversion . already exist, skip it`. + +2. List the chart repositories and look for a second repository that already holds the same chart name: + + ```bash + kubectl get chartrepos.app.alauda.io -A + kubectl get chartrepo -n cpaas-system -o custom-columns=NAME:.metadata.name,TYPE:.spec.type,URL:.spec.url,PHASE:.status.phase + ``` + +3. Find the Artifact and the ArtifactVersions for the chart on the `global` cluster. One Artifact that is much older than the push, plus both the old and the new version, matches this cause: + + ```bash + kubectl get artifact -A | grep + kubectl get artifactversion -A | grep + ``` + +4. Compare the repository labels of the parent Artifact and the new ArtifactVersion, and confirm the ownerReference: + + ```bash + kubectl -n cpaas-system get artifact -o yaml | grep -n -E 'chart-purpose|repository|imagePullSecrets|artifact-version' + kubectl -n cpaas-system get artifactversion . \ + -o jsonpath='purpose={.metadata.labels.cpaas\.io/chart-purpose} owner={.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name} tag={.spec.tag}{"\n"}' + ``` + +| Observation | Conclusion | +| --- | --- | +| Artifact `cpaas.io/chart-purpose` differs from the new ArtifactVersion's (for example `library-charts` vs `public-charts`), owner is that Artifact | This cause | +| Artifact and new ArtifactVersion carry the same repository label, upload succeeded, version still missing | The repository index did not refresh; check the chart repository sync state | +| ArtifactVersion of `type: chart` with empty PHASE, Artifact labels match the repository | Normal state of a chart package; chart artifacts are not reconciled and carry no phase | +| No `artifact` / `artifactversion` for the chart at all after a successful push | The push did not register the chart; this cause does not apply | diff --git a/docs/en/solutions/Pushed_package_not_listed_where_expected_global-only_plugins_cluster_affinity_catalog_source_or_plugin_type_on_ACP.md b/docs/en/solutions/Pushed_package_not_listed_where_expected_global-only_plugins_cluster_affinity_catalog_source_or_plugin_type_on_ACP.md new file mode 100644 index 000000000..093b3e47a --- /dev/null +++ b/docs/en/solutions/Pushed_package_not_listed_where_expected_global-only_plugins_cluster_affinity_catalog_source_or_plugin_type_on_ACP.md @@ -0,0 +1,203 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.2" + - "4.3" +--- + +# Pushed package not listed where expected: global-only plugins, cluster affinity, catalog source, or plugin type on ACP + +## Issue + +`violet push` succeeded and the objects exist, but the package does not show where it is expected: + +- The push targeted a DevOps or other business cluster and the Cluster Plugins page there stays empty (GitOps, `asm-global`). +- `violet push` reported success, the plugin is not under the business cluster's Cluster Plugins, and a business-side Harbor reports an image as present although it is not there. +- A cluster plugin can only be uploaded to `global`, and it is unclear which setting has to change, or how to name another cluster as the upload target. +- MetalLB was pushed with `--clusters `, the output says `moduleplugin metallb has been updated successfully cluster=global type=plugin`, and Cluster Plugins for that workload cluster (an HCS cluster) shows `No data` for MetalLB. +- The ClickHouse log plugin (`logclickhouse`) was uploaded for a dual-stack cluster and never appears in that cluster's plugin list, even after restarts and re-uploads. +- Flagger packages were pushed and are not in the expected package list. +- Kyverno was pushed, Cluster Plugins shows it as listed, and OperatorHub shows it as not listed. +- After a directory push, some `.tgz` packages show neither `present` nor `success`, and pushing again changes nothing. +- On ACP 4.0.3 no standalone ASM plugin package can be found. +- On a new ACP 4.3.0 installation, `envoy-gateway-operator` and `ingress-nginx-operator` seem absent from the marketplace and no Pods exist for them. + +Each of these is expected behavior for that package. A push whose `artifact` / `artifactversion` were never created is a different problem (see "violet push succeeds but the plugin is not published: Artifact, ModuleConfig, or chart missing on ACP"). + +## Root Cause + +### Cause 1: The plugin can only run on global + +The GitOps plugin is restricted to the `global` cluster: a successful push for a DevOps cluster does not create a plugin entry on that cluster, and raising the GitOps package to 4.3.0 does not remove the restriction. On ACP 4.0.3 the `asm-global` cluster plugin behaves the same way and is displayed only in the `global` cluster. + +### Cause 2: violet push without --clusters targets global and its built-in registry + +`violet push` without `--clusters ` uploads to the `global` cluster, and the default push target is the `global` cluster's built-in registry, not a business-side Harbor; an image check in that Harbor does not tell whether the push worked. + +For cluster plugins this is not a default that can be changed. The `--clusters` parameter is not needed when uploading a cluster plugin, because the platform distributes the plugin to clusters automatically according to its affinity configuration; if `--clusters` is given, it is ignored. A cluster plugin is always uploaded to `global`, and no plugin configuration needs to change. Older built-in help that says plugins go "only to the `global` cluster" describes the same behavior. `--clusters` matters for operator packages (`type=bundle`), not for cluster plugins (`type=plugin`). + +### Cause 3: ModulePlugin clusterAffinity does not match the target cluster + +A ModulePlugin can carry an `affinity` block, and the plugin is offered only to clusters that match it. Re-uploading and restarting `cluster-transformer` or OLM does not change that. + +- `clusterAffinity.matchLabels`: the ClickHouse log plugin `logclickhouse` (version 4.3.1) carries `cpaas.io/protocol-stack: IPv4/IPv6`, so it is not offered to a cluster without that label (for example a newly installed ACP 4.3.2 x86 dual-stack cluster). Removing that entry from `moduleplugin logclickhouse` makes the plugin appear and publishable. +- `clusterAffinity.matchExpressions`: the `metallb` ModulePlugin (`metallb.amd64.v4.2.5.tgz`) carries key `cluster-type`, operator `In`, values `Baremetal` and `DCS`. An HCS workload cluster does not match, so MetalLB is registered on `global` but never offered for that cluster, and pushing again only prints `already exists, skip push` and `moduleplugin metallb has been updated successfully cluster=global type=plugin`. Adding `HCS` to the values makes MetalLB available for that cluster. + +### Cause 4: The package is registered in the system catalog, not the platform catalog + +The Flagger packages have catalog source `system`, not `platform`, so they are not listed in the platform catalog; they appear under Installed Plugins. Both are installed automatically by the `global` cluster controller when a service mesh is created, so no manual installation is needed. + +### Cause 5: The package is a cluster plugin, not an Operator + +The `violet push` output says `moduleplugin has been created successfully cluster=global type=plugin`. Kyverno `v4.2.3` (`kyverno.ALL.v4.2.3.tgz`) is delivered as a cluster plugin (`moduleplugin`, `type=plugin`), not as an Operator, so it never appears in OperatorHub. The "Kyverno Operator" card that a search for `kyverno` finds in OperatorHub on `global` (Community, provided by Kyverno, status `未上架`, not listed) is a separate community operator; its status is unrelated, and Kyverno is not deployed from Operators. + +Cluster plugins are managed through the cluster-level CRDs `ModulePlugin`, `ModuleConfig`, and `ModuleInfo`. A plugin is published with `violet` to the `global` cluster only, and the platform then creates its `ModulePlugin` and `ModuleConfig` there; it can be installed on `global` or on any workload cluster, and installing it creates a `ModuleInfo`. Even `--clusters "global,,"` reports `cluster=global type=plugin`. + +### Cause 6: Chart-type packages show no present or success status + +After a directory push (for example with `violet` 4.2.9), `kubectl get artifactversion -A | grep -vi succe` lists ArtifactVersions in `cpaas-system` with empty PHASE and REASON columns (for example `chart-milvus-operator.v1.3.5`, `dify.v1.11.4`, `kubeflow-model-registry.v1.10.10`, `label-studio.v1.21.0-2`), and their artifact is `type: chart`. Artifacts of type `chart` are not reconciled, so their ArtifactVersion has no phase or status. `present: true` with the version in `spec` confirms the push succeeded. The empty status needs no fix. It does not prove the version is displayed; see "Pushed Helm chart version not shown because violet reused an Artifact of another chart repository on ACP" when the new chart version is also missing from the platform. + +### Cause 7: ACP 4.0.3 ships plugins inside the extension package + +In 4.0.3 the extension package is not split into individual plugin packages. It bundles every plugin, including ASM. Take the ASM plugin from the 4.0.3 extension package. + +### Cause 8: Operators shipped with the release show as ready and have no Pods until they are deployed + +`envoy-gateway-operator` and `ingress-nginx-operator` are shipped with 4.3.0. In the platform's operator list they are in the `ready` state, next to many operators in the `absent` state, which makes them easy to overlook. No Pods exist because neither operator has been deployed yet. + +## Resolution + +### Cause 1: The plugin can only run on global + +Push and install the plugin on the `global` cluster, then open Cluster Plugins on `global`. Confirm the same restriction before treating any other package as global-only; it is established for GitOps and `asm-global`. + +```bash +violet push --platform-address= --platform-username= --platform-password='' --clusters global .tgz +``` + +### Cause 2: violet push without --clusters targets global and its built-in registry + +Look for the plugin on `global` first: + +```bash +kubectl get moduleplugins | grep -i +kubectl get artifact -A | grep -i +``` + +To make an operator package (`type=bundle`) available to a business cluster, name that cluster explicitly: + +```bash +violet push --platform-address= --platform-username= --platform-password='' --clusters .tgz +``` + +For a cluster plugin (`type=plugin` in the push output), push without `--clusters`. If it is not offered to the cluster you expect, read its affinity (Cause 3) instead of pushing again with another `--clusters` value. + +### Cause 3: ModulePlugin clusterAffinity does not match the target cluster + +Read the affinity on the ModulePlugin and compare it with the labels and type of the target cluster: + +```bash +kubectl get moduleplugin -o yaml > "$HOME/moduleplugin-.before.yaml" +grep -n -A8 'affinity' "$HOME/moduleplugin-.before.yaml" +``` + +> **Warning:** Changing `clusterAffinity` makes the plugin eligible for clusters the package author excluded. Installing it on a cluster the plugin does not support can fail or misbehave. Precondition: only when the plugin is missing from the cluster plugin list, its affinity does not match the target cluster, you have confirmed the plugin is meant to run there, and the export above is kept. Rollback: `kubectl apply -f "$HOME/moduleplugin-.before.yaml"`. A later push of the same package may also restore the original ModulePlugin. + +```bash +kubectl edit moduleplugin +``` + +For a `matchLabels` entry such as `cpaas.io/protocol-stack: IPv4/IPv6` on `logclickhouse`, delete that line (and the now-empty `matchLabels` / `clusterAffinity` keys if nothing else remains). For a `matchExpressions` list such as the `metallb` `cluster-type` values, add the missing cluster type instead of removing the expression: + +```yaml +spec: + affinity: + clusterAffinity: + matchExpressions: + - key: cluster-type + operator: In + values: + - Baremetal + - DCS + - HCS +``` + +Save, then open Cluster Plugins for the target cluster. + +### Cause 4: The package is registered in the system catalog, not the platform catalog + +```bash +kubectl get packagemanifest -A -o custom-columns=NAME:.metadata.name,CATALOG:.status.catalogSource | grep -i flagger +``` + +Catalog `system` means the package is expected to be absent from the platform catalog; look for it under Installed Plugins. For Flagger, no manual installation is needed. + +### Cause 5: The package is a cluster plugin, not an Operator + +```bash +kubectl get moduleplugins | grep -i kyverno +kubectl get artifact -A | grep -i kyverno +``` + +If the ModulePlugin is missing, push the package: + +```bash +violet push kyverno.ALL.v4.2.3.tgz --platform-address --platform-username --platform-password '' +``` + +Then open Cluster Plugins, select the cluster that should run Kyverno in the cluster selector, and install it there. The Upload Packages page (App Market Management, `上架软件包`) with the same cluster selector shows whether the package is offered to that cluster. Check the result on `global`: + +```bash +kubectl get minfo | grep -i kyverno +``` + +### Cause 6: Chart-type packages show no present or success status + +```bash +kubectl get artifactversion -A | grep -vi succe +kubectl -n cpaas-system get artifactversion -o yaml | grep -n -E 'present|tag|type' +kubectl -n cpaas-system get artifact -o yaml | grep -n 'type:' +``` + +`type: chart` with `spec.present: true` and the expected version is the expected state. Pushing again does not add a status. Look for the chart under Catalog > Helm Charts, not under Cluster Plugins or OperatorHub. + +### Cause 7: ACP 4.0.3 ships plugins inside the extension package + +Unpack the 4.0.3 extension package and push the ASM plugin from it with `violet`: + +```bash +tar -tf | grep -i asm +``` + +The `asm-global` plugin from that package is shown on `global` only (Cause 1). + +### Cause 8: Operators shipped with the release show as ready and have no Pods until they are deployed + +```bash +kubectl get artifactversion -A | grep -iE 'envoy-gateway-operator|ingress-nginx-operator' +kubectl get packagemanifest -A | grep -iE 'envoy-gateway|ingress-nginx' +``` + +When both are present, deploy the operator you need from OperatorHub. Pods appear only after the operator is installed. + +## Diagnostic Steps + +```bash +kubectl get moduleplugins +kubectl get moduleplugin -o yaml | grep -n -A8 affinity +kubectl get packagemanifest -A -o custom-columns=NAME:.metadata.name,CATALOG:.status.catalogSource +kubectl get artifact,artifactversion -A | grep +``` + +- An empty plugin page on a non-global cluster for GitOps or `asm-global`: Cause 1. +- A push without `--clusters`, or a question of how to push a cluster plugin to a cluster other than `global`: Cause 2. +- A `clusterAffinity` (`matchLabels`, or a `matchExpressions` list such as `cluster-type`) that the target cluster does not satisfy: Cause 3. +- Catalog source `system`: Cause 4. +- A `type=plugin` ModulePlugin missing from OperatorHub: Cause 5. +- A chart-type package with `spec.present: true` and no status: Cause 6. +- A 4.0.3 plugin with no standalone package: Cause 7. +- Operators listed as `ready` with no Pods: Cause 8. diff --git a/docs/en/solutions/Redis_for_Harbor_unavailable_no_Data_Services_license_or_Redis_in_cluster_mode_reported_as_empty_secrets.md b/docs/en/solutions/Redis_for_Harbor_unavailable_no_Data_Services_license_or_Redis_in_cluster_mode_reported_as_empty_secrets.md new file mode 100644 index 000000000..7f7b8b8a5 --- /dev/null +++ b/docs/en/solutions/Redis_for_Harbor_unavailable_no_Data_Services_license_or_Redis_in_cluster_mode_reported_as_empty_secrets.md @@ -0,0 +1,116 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps + - Alauda Application Services +ProductsVersion: [] +--- + +# Redis for Harbor unavailable: no Data Services license, or Redis in cluster mode reported as empty secrets + +## Issue + +Harbor cannot be deployed because its Redis is not available or not accepted: + +- While preparing Redis for a Harbor v4 deployment, the Select Instance Type page blocks creation of a `Redis` instance with `请在 Data Services 视图使用此功能` ("use this function in the Data Services view"), and the create action is disabled. The list also shows `ActiveRedis`, `ActiveRedisConnection`, `DistributedRedisCluster`, and `RedisClusterBackup`. Opening Data Services then shows `无法访问此页面` ("the page cannot be accessed"), states that the Data Services product function cannot be used, and points to `许可证管理 / 平台许可证` (License Management / Platform License) with an `立即导入` (Import now) action. +- Deploying Harbor reports that the secrets are empty, although the Secrets named by the deployment do contain data. + +Related articles: [harbor-core cannot connect to Redis: NOAUTH from Sentinel, or ERR invalid password after a Redis password change](harbor-core_cannot_connect_to_Redis_NOAUTH_from_Sentinel_or_ERR_invalid_password_after_a_Redis_password_change.md). + +## Root Cause + +### Cause 1: Data Services is not licensed + +Redis provisioning for Harbor v4 from the platform goes through Data Services ("v4" is the ACP packaging name, not the Helm chart or Harbor version). When Data Services has not been purchased, its license is absent, the Data Services view cannot be opened, and a Redis instance cannot be created there. This is a licensing condition, not a fault. Harbor can instead be deployed against Redis and PostgreSQL that you provide, passed to Harbor through credential Secrets. + +### Cause 2: the Redis instance runs in cluster mode + +When the Redis instance configured for Harbor runs in cluster mode, deploying Harbor reports that the secrets are empty, even though the Secrets contain data. The error comes from the unsupported Redis mode, not from missing Secret content. **Harbor does not support Redis cluster mode**; it needs a Redis that is not in cluster mode (Sentinel or standalone). + +## Resolution + +### Cause 1 + +1. Check `许可证管理 / 平台许可证` (Platform License) for a Data Services license. If Data Services is purchased later, import the license from that page; Redis can then be created from the Data Services view. + +2. Without the license, provide Redis and PostgreSQL yourself and give Harbor their connection details in Opaque Secrets in the Harbor namespace. Follow the "Harbor credentials" page of the Alauda Build of Harbor installation documentation for the Harbor version you deploy: it lists the Secret keys, the supported PostgreSQL versions, and how the Secrets are referenced from the `Harbor` resource. The Harbor version decides which PostgreSQL version is required (for example, Harbor 2.12.z requires PostgreSQL 14.x; Harbor 2.14.z supports PostgreSQL 15.x and 16.x). + + Redis in Sentinel mode uses the keys `password`, `address`, and `masterName`: + + ```bash + NS= + kubectl -n "$NS" create secret generic harbor-redis \ + --from-literal=password='' \ + --from-literal=address='192.0.2.11:26379,192.0.2.12:26379,192.0.2.13:26379' \ + --from-literal=masterName='mymaster' + ``` + + Redis in standalone mode uses `host`, `port`, and `password` instead. + + PostgreSQL uses `host`, `port`, `username`, `password`, `database`, and `sslmode`. The database must already exist and be empty; `harbor_db` below is only an example name and must be the database you created: + + ```bash + kubectl -n "$NS" create secret generic harbor-database \ + --from-literal=host='192.0.2.20' \ + --from-literal=port='5432' \ + --from-literal=username='postgres' \ + --from-literal=password='' \ + --from-literal=database='harbor_db' \ + --from-literal=sslmode='require' + ``` + +3. Use a Redis without an independent Sentinel password: Harbor does not send the Sentinel password and fails with `NOAUTH` (see the related article). + +### Cause 2 + +1. Confirm that the Secrets are not empty and that Redis runs in cluster mode: + + ```bash + NS= + kubectl -n "$NS" get secret -o go-template='{{range $k, $v := .data}}{{$k}} {{len $v}}{{"\n"}}{{end}}' + kubectl get redis -A + kubectl -n get redis -o yaml | grep -n -i -E 'arch|mode|cluster' + ``` + + The cause is confirmed when every key has a non-zero length and the Redis instance is in cluster mode. + +2. Create a Redis instance that is not in cluster mode (Sentinel or standalone) in Data Services, and copy its credential Secret into the Harbor namespace if it lives elsewhere: + + ```bash + kubectl get secret -n -l middleware.instance/type=Redis + ``` + +3. Point the `Harbor` resource at the new instance. + + > **Warning:** Only when the deployment reports empty secrets, the Secrets are not empty, and Redis is in cluster mode. Impact: the operator restarts Harbor components, and cache, sessions, and queued jobs on the old Redis are not carried over. Precondition: the new Redis instance is Ready and its Secret is in the Harbor namespace; save the current resource with the first command below. Rollback: apply the saved file, which returns the unsupported mode and the same error. Do not delete the cluster-mode instance until Harbor works on the new one; its deletion cannot be undone. + + ```bash + NAME= + kubectl -n "$NS" get harbor "$NAME" -o yaml > "$HOME/harbor-$NAME-before.yaml" + kubectl -n "$NS" patch harbor "$NAME" --type=merge -p '{ + "spec": {"helmValues": {"redis": {"type": "external", "external": { + "addr": ":26379,:26379,:26379", + "existingSecret": "", + "existingSecretKey": "password", + "sentinelMasterSet": "mymaster" + }}}} + }' + ``` + + For a standalone replacement, use its `host:port` as `addr` and leave `sentinelMasterSet` unset. + +## Diagnostic Steps + +1. For Cause 1 the evidence is in the UI: the text `请在 Data Services 视图使用此功能` when creating Redis, and `无法访问此页面` pointing to the platform license page when Data Services is opened. Check the Platform License page before investigating a fault. + +2. For Cause 2, list the key sizes of every Secret the deployment references (command in Cause 2, step 1). Expected: non-zero sizes. Then check the Redis architecture. Expected: cluster mode. + +3. Once Harbor is deployed, read the core log for Redis errors: + + ```bash + CORE=$(kubectl -n get pods -o name | grep harbor-core | head -1) + kubectl -n logs "$CORE" --tail=200 | grep -iE 'redis|sentinel|NOAUTH|invalid password' + ``` + + Expected: no Redis errors. `NOAUTH` or `ERR invalid password` is covered by the related article. diff --git a/docs/en/solutions/Registry_push_fails_with_insufficient_scope_or_authentication_required_on_ACP.md b/docs/en/solutions/Registry_push_fails_with_insufficient_scope_or_authentication_required_on_ACP.md new file mode 100644 index 000000000..b8fcdd3fa --- /dev/null +++ b/docs/en/solutions/Registry_push_fails_with_insufficient_scope_or_authentication_required_on_ACP.md @@ -0,0 +1,111 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform + - Alauda DevOps +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Registry push fails with insufficient_scope or authentication required on ACP + +## Issue + +Login reaches the registry, but the account is not allowed to push, or the push carries no credential: + +- `nerdctl` or `ctr` push fails, after the registry address itself is correct, with: + + ```text + push access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed + ``` + + The account is the registry user entered when the platform was deployed (for example `cpaas`). A push as `admin` is not rejected. +- A pipeline Buildah task pulls the base image, then the push fails with: + + ```text + Error: pushing image "//:" to "docker:////:": trying to reuse blob sha256: at destination: checking whether a blob sha256: exists in //: authentication required + ``` +- A new global cluster cannot reuse a Harbor that was deployed with the previous platform version. + +A `docker login` or push that fails with `http: server gave HTTP response to HTTPS client` never reaches the account check; see [Harbor push or pull fails because the client uses the wrong scheme: HTTP response to HTTPS client, or port 443 refused](Harbor_push_or_pull_fails_because_the_client_uses_the_wrong_scheme_HTTP_response_to_HTTPS_client_or_port_443_refused.md). For pull denials from the platform registry, see [Platform registry pull fails with authorization failed or insufficient_scope: auth.yaml targets or node login](Platform_registry_pull_fails_with_authorization_failed_or_insufficient_scope_auth.yaml_targets_or_node_login.md). + +## Root Cause + +### Cause 1: the platform-deployment registry user has only pull permission + +In `/etc/kubernetes/registry/auth.yaml` on the global control-plane nodes, the registry user entered during platform deployment has only `pull` on target `.*`, while `admin` has `pull`, `push`, and `delete` on `.*`. A push with that user fails with `insufficient_scope: authorization failed`. Pushing with an account that has push permission (`admin`) is the fix. + +### Cause 2: the Buildah task has no registry credentials bound for the push + +In an ACP UI pipeline that uses the built-in Buildah task, the base-image pull succeeds and the push fails with `authentication required` on the blob check, because the task has no registry credentials for the push. The task has an optional `registry-config` workspace that provides a `config.json` at the workspace root, which Buildah uses to authenticate to the registry. Binding that workspace to the docker-credentials Secret of the target Harbor integration supplies the credentials. + +### Cause 3: the Harbor integration on the new global cluster has wrong credentials + +When a new global cluster is set up to reuse the Harbor deployed with the previous platform version, the integration does not work if the credentials configured for the existing Harbor are wrong. Removing the integration and adding it again with the correct credentials restores access. + +## Resolution + +### Cause 1 + +1. With the registry address in `/etc/containerd/certs.d` already correct, read the actions granted to the account you pushed with (password lines filtered out): + + ```bash + grep -n -B2 -A8 '' /etc/kubernetes/registry/auth.yaml | grep -v -i password + ``` + + Only `pull` for that user, and `pull`, `push`, `delete` for `admin`, confirms the cause. + +2. Push as `admin`: + + ```bash + printf '%s' "$REGISTRY_ADMIN_PASSWORD" | nerdctl login : --username admin --password-stdin + nerdctl push ://: + ``` + + Do not put the `admin` password into a workload image pull Secret. For pipelines, use an account that has push permission on the target project. + +### Cause 2 + +1. Open the pipeline stage that runs the Buildah task and check its settings: `skipPush` must be `false`, and the `source` workspace is bound to the build workspace. + +2. Find the docker-credentials Secret of the Harbor integration for the target registry. In the UI it appears as ` (harbor) - docker-credentials`; the Secret is named `docker-credentials-`. Print its type and key names only: + + ```bash + kubectl get secret -n | grep docker-credentials + kubectl get secret -n \ + -o go-template='{{.type}}{{"\n"}}{{range $k, $v := .data}}{{$k}}{{"\n"}}{{end}}' + ``` + +3. Bind the task's `registry-config` workspace to that Secret and run the pipeline again. + +4. If the push still fails with `authentication required`, confirm that the account in the credential can push to the Harbor project (log in and push with it from the command line). + +### Cause 3 + +1. Confirm a login with the credentials you intend to use: + + ```bash + printf '%s' "$HARBOR_PASSWORD" | docker login : --username --password-stdin + ``` + + A failed login means the credentials are still wrong; do not re-add the integration until it succeeds. + +2. On the new global cluster, remove the Harbor integration and add it again with those credentials, from the same platform integration screen used to add it. + + > **Warning:** Impact: while the integration is removed, pipelines and projects that use it cannot reach Harbor. Precondition: the login in step 1 succeeded, and the integration settings (name, address, assigned projects) are recorded so they can be entered again. Rollback: add the integration again with the recorded settings. + +## Diagnostic Steps + +1. Separate a permission denial from a missing tag or a scheme error. An `http: server gave HTTP response to HTTPS client` or x509 error happens before the account check. + +2. `insufficient_scope: authorization failed` on push with the platform-deployment user selects Cause 1. Check the user's actions in `auth.yaml` for `push`. + +3. A Buildah task whose pull works and whose push fails with `authentication required` on the blob check selects Cause 2. Check the task's `registry-config` workspace binding first. + +4. A new global cluster that cannot open an older Harbor selects Cause 3. Test the configured username and password with `docker login` before changing anything else. + +5. A UI upload through a toolchain integration that fails with `unauthorized to access repository: ..., action: push` while the same account pushes from the command line is covered in [Harbor returns 401 because of time skew, the node auth container, Harbor Core, or a shared integration credential](Harbor_returns_401_because_of_time_skew_the_node_auth_container_Harbor_Core_or_a_shared_integration_credential.md). diff --git a/docs/en/solutions/Removing_Rook-Ceph_fails_in_the_UI_or_the_CephCluster_stays_in_Deleting_on_ACP.md b/docs/en/solutions/Removing_Rook-Ceph_fails_in_the_UI_or_the_CephCluster_stays_in_Deleting_on_ACP.md new file mode 100644 index 000000000..ee1d2aa0a --- /dev/null +++ b/docs/en/solutions/Removing_Rook-Ceph_fails_in_the_UI_or_the_CephCluster_stays_in_Deleting_on_ACP.md @@ -0,0 +1,95 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Removing Rook-Ceph fails in the UI, or the CephCluster stays in Deleting on ACP + +## Issue + +Ceph distributed storage has to be removed from a cluster of Alauda Container Platform, and the removal does not finish: + +- Deleting the distributed storage from the UI fails. +- The product cleanup procedure was followed and every PVC on Ceph was deleted, but `kubectl get cephclusters -A` keeps showing `rook-ceph/ceph-cluster` with PHASE `Deleting`, MESSAGE `Deleting the CephCluster`, and HEALTH `HEALTH_WARN`. Removing the finalizers once did not make it go away. +- Some PVCs on Ceph belong to cluster services, and it is unclear what removing Ceph does to them. + +This article covers the whole `CephCluster` (`cephclusters.ceph.rook.io` in namespace `rook-ceph`, for example `ceph-cluster` with `DATADIRHOSTPATH` `/var/lib/rook`; Rook puts the finalizer `cephcluster.ceph.rook.io` on it). A single `CephObjectStore` stuck in `Deleting` while the cluster is healthy is covered in [CephObjectStore stays in Deleting with failed to list buckets no route to host on ACP](./CephObjectStore_stays_in_Deleting_with_failed_to_list_buckets_no_route_to_host_on_ACP.md). + +## Root Cause + +### Cause 1: distributed storage deletion fails in the UI + +When the UI deletion fails, Ceph is removed manually with the product documentation topic "Cleanup Distributed Storage". That procedure removes the distributed storage resources in order, starting with the VolumeSnapshotClasses and then the StorageClasses, and requires every PVC and PV that uses Ceph to be deleted first. + +Removing Ceph deletes the data in every PVC Ceph backs, including PVCs of platform or cluster services, not only business PVCs. The impact on those services has to be settled before deletion. + +### Cause 2: PHASE Deleting, MESSAGE Deleting the CephCluster, because finalizers remain + +After the PVCs were removed and the cleanup procedure was followed, the `CephCluster` object still carries finalizers, so Kubernetes keeps it in `Deleting`. A partial finalizer edit leaves some of them behind and the object stays. + +## Resolution + +### Cause 1: distributed storage deletion fails in the UI + +1. List what still uses Ceph: + +```bash +kubectl get pvc -A -o wide | grep -E 'ceph|rbd|cephfs' +kubectl get pv | grep -E 'ceph|rbd|cephfs' +kubectl get sc +kubectl get volumesnapshotclass +``` + +2. For each PVC, find its owner. If any belongs to a platform or cluster service, decide with the service owner how its data is migrated or backed up first. + +> **Warning:** Removing Ceph deletes the data in every Ceph-backed PVC, including PVCs of cluster services, and those services stop working. Precondition: every Ceph PVC is listed, each owner has agreed, and data that must survive is migrated or backed up outside this Ceph cluster. Rollback: none; a new Ceph cluster does not bring the data back. + +3. Open "Cleanup Distributed Storage" in the product documentation for the installed version (also reachable from the help entry in the top-right corner of the platform) and run its steps in order. If the `CephCluster` then stays in `Deleting`, continue with Cause 2. + +### Cause 2: PHASE Deleting, MESSAGE Deleting the CephCluster, because finalizers remain + +Use this only when the cleanup procedure has been followed up to the cluster deletion, no PVC or PV that uses Ceph remains, and the `CephCluster` stays in `Deleting`. + +1. Check what is still there: + +```bash +kubectl get cephclusters -A +kubectl -n rook-ceph get cephcluster ceph-cluster -o jsonpath='{.metadata.finalizers}{"\n"}' +kubectl get pv | grep -E 'ceph|rbd|cephfs' +kubectl -n rook-ceph logs deploy/rook-ceph-operator --tail=100 +``` + +If the operator log names objects that still depend on the cluster, clean those up first, following the procedure, before touching finalizers. + +> **Warning:** Removing the finalizers lets Kubernetes delete the `CephCluster` object without waiting for Rook to finish its cleanup. Rook then no longer manages anything of this cluster; data on the OSD disks and under `/var/lib/rook` on the nodes stays until the procedure's later steps clean it. Precondition: you have decided to remove this Ceph cluster, every Ceph PVC and PV is gone, and the earlier steps of the procedure are done. Rollback: none. + +2. Remove all finalizers and check the result: + +```bash +kubectl -n rook-ceph patch cephcluster ceph-cluster --type merge -p '{"metadata":{"finalizers":null}}' +kubectl -n rook-ceph get cephcluster ceph-cluster -o jsonpath='{.metadata.finalizers}{"\n"}' +kubectl get cephclusters -A +``` + +Editing the object with `kubectl edit cephclusters -n rook-ceph ceph-cluster` and deleting the whole `finalizers` list has the same effect; check afterwards that the list is really empty. `No resources found` from `kubectl get cephclusters -A` means the object is gone. + +3. Continue with the remaining steps of the cleanup procedure (operator removal and disk cleanup on the nodes). + +## Diagnostic Steps + +```bash +kubectl get cephclusters -A +kubectl -n rook-ceph get cephcluster ceph-cluster -o yaml | grep -A5 finalizers +kubectl get pvc -A | grep -E 'ceph|rbd|cephfs' +kubectl -n rook-ceph logs deploy/rook-ceph-operator --tail=100 +``` + +- UI deletion fails and Ceph PVCs still exist: Cause 1. Settle the data impact, then run the cleanup procedure. +- `Deleting` / `Deleting the CephCluster` after the PVCs and PVs are gone, with a non-empty `finalizers` list: Cause 2. +- The operator log names dependent objects: clean those first, following the procedure. diff --git a/docs/en/solutions/Removing_a_TopoLVM_storage_node_or_releasing_its_disks_on_ACP.md b/docs/en/solutions/Removing_a_TopoLVM_storage_node_or_releasing_its_disks_on_ACP.md new file mode 100644 index 000000000..7053b5bc1 --- /dev/null +++ b/docs/en/solutions/Removing_a_TopoLVM_storage_node_or_releasing_its_disks_on_ACP.md @@ -0,0 +1,140 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Removing a TopoLVM storage node or releasing its disks on ACP + +## Issue + +One or more nodes have to leave TopoLVM on Alauda Container Platform, for example before the node is reclaimed or its operating system is reinstalled, or its raw disks must be freed for another consumer such as Ceph. Unused PVCs or mounted workloads may still be on that node. + +Or the release was already attempted: a prepare job still errors, `lsblk` still shows content on the local disks, `TopolvmCluster` status still lists nodes other than the one that should remain, and `dd` was then run on the node that was supposed to keep TopoLVM. + +A related question is what deletes the data inside a TopoLVM PVC. TopoLVM volumes are node-local LVM logical volumes. Data is lost when the PVC, PV, or `LogicalVolume` is deleted with the volume behind it, when a node is removed from `TopolvmCluster` or its disks are released with the sequence below, or when the data disk of a node that should keep its data is wiped (`dd`, a reinstall that touches data disks, or an uninstall and reinstall of TopoLVM). Removing a node never copies its logical volumes anywhere. Decide first whether the data on the node is gone, backed up, or copied by the application. + +## Root Cause + +### Cause 1: a TopoLVM storage node has to be removed, or its disks released, while PVCs are still on it + +The supported procedure is "Scenario 3: Remove a TopoLVM Storage Node" in the product documentation topic "Remove Disks, Device-Class Volume Groups, or Nodes from LVM Storage". It applies when the node cannot be recovered or will be permanently removed from `TopolvmCluster`, and no device-class volume group on the node is to be kept. Residual PVCs pinned to the node, including unused ones, block handing its disks to another consumer until they are cleaned up. + +### Cause 2: the prepare job still errors after a node release, and dd was run on the node that should keep its data + +On the node that was meant to keep TopoLVM: + +- its `status.nodeStorageState` entry is `phase: Unknown`, while a released node can still show `phase: Ready`; +- its `topolvm-node-` pod is `3/4 CrashLoopBackOff` with many restarts, while `topolvm-controller` and `topolvm-operator` are Running; +- its `topolvm-prepare-vg` log, after the usual device skips (`unsupported diskType lvm`, `contains a filesystem "LVM2_member"`, `has child, considering the child instead`), ends with an error raised from `checkVgIfShrink`: + +```text +prepare-vg: list pv for vg - failed err:exit status 5 +parse out failed cmd:vgs pv_name - +``` + +`pvs` on that node lists only the operating-system volume group; no physical volume exists for the `-` group that the ConfigMap and the status name. `dd` against the disk that held that volume group destroyed it. Restarting the operator does not change this. + +## Resolution + +### Cause 1: a TopoLVM storage node has to be removed, or its disks released, while PVCs are still on it + +Prerequisites: + +- Workloads that use PVCs on the target node are stopped or migrated, and the remaining nodes can carry them. +- The data on the node is backed up or confirmed not needed. + +1. Save the cluster object and stop the operator so it does not write the old node back: + +```bash +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm -o yaml > /tmp/topolvmcluster.before.yaml +kubectl -n nativestor-system scale deployment topolvm-operator --replicas=0 +``` + +2. List every TopoLVM PV pinned to the node. Do not skip a line because the PVC looks unused: + +```bash +kubectl get pv -o json | jq -r --arg NODE "" ' + .items[] + | select(.spec.csi.driver=="topolvm.cybozu.com") + | select(.spec.nodeAffinity.required.nodeSelectorTerms[]?.matchExpressions[]? + | select(.key=="topology.topolvm.cybozu.com/node") | .values[]? == $NODE) + | [.metadata.name, .status.phase, .spec.claimRef.namespace, .spec.claimRef.name] + | @tsv' +``` + +An empty list means no TopoLVM volume is pinned to that node. + +> **Warning:** The deletes below remove every TopoLVM PVC, PV, and LogicalVolume pinned to the node, and the data on those volumes is destroyed. Precondition: the workloads are stopped, a backup exists if the data matters, and the node name in the query is the node you mean to retire. Rollback: none after the LogicalVolume is deleted. + +3. For each line, delete the objects. The LogicalVolume name equals the PV name: + +```bash +kubectl delete pvc -n +kubectl delete pv +kubectl delete logicalvolumes.topolvm.cybozu.com +``` + +4. Run `kubectl -n nativestor-system edit topolvmclusters.topolvm.cybozu.com topolvm` and delete the whole `nodeName: ` element under `spec.storage.deviceClasses`. +5. Run `kubectl -n nativestor-system edit configmaps lvmdconfig-` and delete the entire `lvmd.yaml` and `status.json` sections. +6. Start the operator again and confirm the node is gone from `status.nodeStorageState`: + +```bash +kubectl -n nativestor-system scale deployment topolvm-operator --replicas=1 +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm \ + -o jsonpath='{.status.nodeStorageState}' | jq . +``` + +If only the spec was edited and a `topolvm-prepare-vg` Job for the node stays `Pending`, the node is still in status; clean the status as well instead of repeatedly deleting Jobs. + +When nodes are being replaced and the StorageClass reclaim policy is `Retain`, release each PV before removing the node: delete the application that no longer uses the volumes, delete the matching `logicalvolume` and then the PV, and confirm with `lvs` that the LV is gone (see [TopoLVM capacity stays full after PVCs are deleted on ACP](./TopoLVM_capacity_stays_full_after_PVCs_are_deleted_on_ACP.md), Cause 2). A full replacement of workers in the global cluster, for example for an unused registry instance, ran in this order: delete the registry's template repositories and the registry instance; release the Retain PVs; check on each node that the local storage space is released; clean up the TopoLVM resources; uninstall the log collection and log storage plugins; cordon the nodes, force-evict their pods, and check that nothing is scheduled back; delete the nodes; redeploy TopoLVM and the registry on the new nodes. + +> **Warning:** Draining evicts every pod on the node, including pods with local data (`--delete-emptydir-data`). Precondition: the TopoLVM volumes on the node are released and the plugins that keep data there are uninstalled. Rollback: `kubectl uncordon ` allows scheduling again; evicted pods are recreated elsewhere by their owners. + +```bash +kubectl cordon +kubectl drain --ignore-daemonsets --delete-emptydir-data --force +kubectl get pods -A -o wide --field-selector spec.nodeName= +``` + +### Cause 2: the prepare job still errors after a node release, and dd was run on the node that should keep its data + +1. Confirm that the volume group is really gone: + +```bash +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm \ + -o jsonpath='{.status.nodeStorageState}' | jq . +pvs +vgs - +``` + +`vgs` failing for the name in status, no physical volume for it in `pvs`, and the `list pv for vg ... exit status 5` log confirm this cause. If `pvs` shows the device in a TopoLVM volume group with a different name, that is a name mismatch, not damage; see [TopoLVM lvmd exits with should have at least one device-class or Error: not found on ACP](./TopoLVM_lvmd_exits_with_should_have_at_least_one_device-class_or_Error_not_found_on_ACP.md). + +2. There is no in-place undo for `dd` on the disk that held the volume group you meant to keep. Treat the bytes on that disk as lost. The way forward is to uninstall and reinstall TopoLVM on the cluster. + +> **Warning:** Uninstalling TopoLVM after `dd` on the disk you meant to keep discards that node's local volumes. Precondition: this cause is confirmed, a backup exists or the data is expendable, and you are in a maintenance window. Rollback: none. + +To avoid this: when status still lists nodes that were removed, clean the stale status first. Never wipe the disk of a node that is meant to keep its data. + +## Diagnostic Steps + +```bash +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm -o yaml +kubectl get cm,deploy -n nativestor-system +kubectl get job,pod -n nativestor-system -o wide | grep prepare-vg +pvs +vgs +``` + +- A request to remove a live node, or to free its disks for another consumer: Cause 1. +- `prepare-vg: list pv for vg - failed err:exit status 5`, no physical volume for that group in `pvs`, and `dd` already run on the node you meant to keep: Cause 2. + +## Related Information + +- [TopoLVM node pod does not start after a reboot because its data disk was formatted on ACP](./TopoLVM_node_pod_does_not_start_after_a_reboot_because_its_data_disk_was_formatted_on_ACP.md): uses the removal sequence above and then adds the node back. +- [Preserving TopoLVM Data of KubeVirt VMs When Reinstalling Node Operating Systems on ACP](./Preserving_TopoLVM_Data_of_KubeVirt_VMs_When_Reinstalling_Node_Operating_Systems_on_ACP.md) diff --git a/docs/en/solutions/Replace_the_platform_access_HTTPS_certificate_in_prdb_with_a_PEM_certificate_and_key_on_ACP_3.12.md b/docs/en/solutions/Replace_the_platform_access_HTTPS_certificate_in_prdb_with_a_PEM_certificate_and_key_on_ACP_3.12.md new file mode 100644 index 000000000..50c9204a3 --- /dev/null +++ b/docs/en/solutions/Replace_the_platform_access_HTTPS_certificate_in_prdb_with_a_PEM_certificate_and_key_on_ACP_3.12.md @@ -0,0 +1,110 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.12.x +--- + +# Replace the platform access HTTPS certificate in prdb with a PEM certificate and key on ACP 3.12 + +## Issue + +On ACP 3.12, the HTTPS certificate of the platform access address (the domain users open the console with) has to be replaced, for example because it is about to expire. The product documentation pages about certificates cover Ingress and load balancer certificates, not the certificate of the platform itself. + +The new certificate is often delivered as two files, such as `.key` and `.pem`, and there is no file named `.crt` or `.cert`, which makes it unclear which file goes where. + +## Root Cause + +On ACP 3.12 the platform access certificate is held in the `prdb` resource on the `global` cluster, in `spec.http.tls.cert` (certificate) and `spec.http.tls.key` (private key), both base64-encoded. Changing these fields is how the certificate is replaced. + +This procedure is for ACP 3.12. On ACP 4.x the platform access certificate is the `dex.tls` Secret in `cpaas-system`; follow "Rotate TLS Certs of Platform Access Addresses" in the product documentation instead. + +A `.pem` file is just a PEM-encoded file. When it starts with `-----BEGIN CERTIFICATE-----`, it is the certificate and is used in place of a `.cert`/`.crt` file. The `.key` file is the private key. + +## Resolution + +Run the steps on the first control plane node of the `global` cluster. + +1. Prepare the files. Copy the certificate (`.pem`) to `custom.cert` and the key to `custom.key`, then check that they belong together and cover the platform domain: + + ```bash + cp .pem custom.cert + cp .key custom.key + openssl x509 -in custom.cert -noout -subject -enddate -ext subjectAltName + openssl x509 -in custom.cert -noout -pubkey | openssl md5 + openssl pkey -in custom.key -pubout | openssl md5 + ``` + + Continue only if the two hashes are identical, `notAfter` is in the future, and the subject or subject alternative names include the platform access domain. If the issuer supplied intermediate certificates, append them to `custom.cert` after the server certificate. + +2. Back up the current resource: + + ```bash + kubectl get prdb -o yaml > /backup-prdb.yaml + ``` + +3. Generate the base64 values: + + ```bash + cat custom.cert | base64 -w 0; echo + cat custom.key | base64 -w 0; echo + ``` + +4. Write them into `prdb`. + + > **Warning:** Saving this change redeploys the platform access components with the new certificate. HTTPS access and logins can be interrupted until the rollout completes, and a wrong certificate or key makes the console unreachable over HTTPS. Do this in a maintenance window and only after step 1 succeeded. To roll back, edit `prdb` again and put back the `spec.http.tls.cert` and `spec.http.tls.key` values saved in `/backup-prdb.yaml`. + + ```bash + kubectl edit prdb + ``` + + Set `spec.http.tls.cert` to the first value and `spec.http.tls.key` to the second. Add either key if it is missing. + +5. Wait until the platform releases in `cpaas-system` are ready again (note the namespace spelling; a typo such as `cpass-system` returns `No resources found`): + + ```bash + kubectl get appreleases -n cpaas-system + ``` + +6. Check that the `dex` pod was recreated after the change. If its age shows it was not, delete it so that it restarts with the new certificate. + + > **Warning:** Users cannot log in while the `dex` pod restarts, usually for less than a minute. Delete it only if it was not recreated automatically. The Deployment recreates it, so no rollback is needed. + + ```bash + kubectl get po -A | grep dex + kubectl -n cpaas-system delete pod + ``` + +7. Log in to the platform and check the certificate the browser receives. + +## Diagnostic Steps + +1. Check the certificate currently served on the platform address: + + ```bash + echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -subject -issuer -dates + ``` + +2. Check what `prdb` holds and compare it with the served certificate: + + ```bash + kubectl get prdb -o jsonpath='{.items[0].spec.http.tls.cert}' | base64 -d | openssl x509 -noout -subject -dates + ``` + + After the replacement both commands show the new subject and dates. + +3. Identify an unlabeled file by its first line: `-----BEGIN CERTIFICATE-----` is a certificate, `-----BEGIN PRIVATE KEY-----` or `-----BEGIN RSA PRIVATE KEY-----` is a key: + + ```bash + head -1 .pem .key + ``` + +4. If the platform releases do not become ready after the edit, inspect the one that is not ready: + + ```bash + kubectl get appreleases -n cpaas-system + kubectl describe apprelease -n cpaas-system + ``` diff --git a/docs/en/solutions/Replace_the_platform_access_certificate_in_dex.tls_on_ACP.md b/docs/en/solutions/Replace_the_platform_access_certificate_in_dex.tls_on_ACP.md new file mode 100644 index 000000000..d201c0f7a --- /dev/null +++ b/docs/en/solutions/Replace_the_platform_access_certificate_in_dex.tls_on_ACP.md @@ -0,0 +1,170 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.12.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x + - 4.4.x +--- + +# Replace the platform access certificate in dex.tls on ACP + +## Issue + +One of the following concerns the certificate that the platform access address presents: + +- The browser warns that the platform address is not private, the HTTPS certificate is about to expire, or the site still shows an openssl self-signed certificate after the platform certificate was deleted. +- A certificate added on an ACP page never appears when users open the platform. +- Nodes fail image pulls from the platform registry host with `x509: certificate has expired or is not yet valid`. +- An inspection reports `dex.tls` as expired, but the console still opens over HTTP, or the report appeared right after a virtual machine restart. +- In a disaster-recovery deployment, business clusters show the `base` component abnormal in minfo before an upgrade, and `cluster-transform` logs that the primary and standby global `tls.crt` are inconsistent. Or the standby global fails. +- During installation or later: "where do I add the ACP certificate?", "how do I replace the platform certificate with a trusted one?", "can the platform move from an IP address to a domain name?", or "which format of a wildcard certificate should be downloaded?". + +This certificate is not the client certificate inside `admin.conf`, not Secret `global-kubeconfig`, and not an Ingress or ALB certificate of a business service. On ACP 3.12, where the certificate is held inline in the `prdb` resource, see "Replace the platform access HTTPS certificate in prdb with a PEM certificate and key on ACP 3.12". + +## Root Cause + +### Cause 1: cpaas-system/dex.tls is expired, self-signed, or not the certificate uploaded on a page + +Secret `dex.tls` in `cpaas-system` on the global cluster holds the leaf certificate that the platform access address, and the platform registry, present. Certificate `dex-serving-cert` in the same namespace is the cert-manager object that owns a platform-issued copy. Uploading a certificate on an ACP page does not replace this Secret. + +Whether `dex.tls` renews by itself depends on how the platform was deployed. When the platform generated its own certificate, `dex-serving-cert` manages it and it renews automatically. When the platform was deployed with an existing certificate supplied by its owner, `dex.tls` has to be replaced by hand when that certificate approaches its end date. The same replacement applies when the current material is the temporary openssl self-signed certificate left after a deletion. On ACP 4.0 disaster-recovery deployments, the standby global needs the same replacement. + +After `dex.tls` is replaced, business clusters can keep listing it as about to expire on the previous date. Those business-cluster copies renew automatically. + +### Cause 2: dex.tls is reported expired but its dates are valid, or users only use HTTP + +An inspection run right after a virtual machine restart can report `dex.tls` as expired while its `notAfter` is still in the future; later inspection runs are clean. Separately, an expired `dex.tls` does not block users who reach the platform over HTTP only; it matters for HTTPS clients and for nodes that pull from the platform registry. + +### Cause 3: the platform address cannot be changed, so a trusted certificate must match the existing name + +Certificates are configured for the platform access address; they are not added to a business cluster's API server. The access address of an already deployed platform cannot currently be changed, so a trusted certificate can only be used if it is issued for the name the platform already uses. There is no established procedure to turn an IP platform address into a domain name. A DNS name added later for browser access to a platform deployed on a VIP shows "not secure", and no certificate configuration removes that warning. + +### Cause 4: the primary and standby global clusters use different platform certificates + +In a disaster-recovery deployment both global clusters must present the same platform access certificate. When their `tls.crt` differ, `cluster-transform` reports the inconsistency, the business clusters show `base` abnormal in minfo, the minfo version does not match the cluster version, and refreshing the minfo label does not help. Putting the same certificate on both globals is the fix. A failed standby global can likewise be recovered by updating `dex.tls` on the standby. + +### Cause 5: deployment-time certificate choices + +When no certificate issued by a public or enterprise CA is available for the platform domain at installation, a self-signed certificate can be used; browsers warn until a trusted certificate for the same domain replaces it. When a wildcard certificate is downloaded from a provider that offers nginx, apache, tomcat, IIS, and other server types, choose nginx: it delivers a PEM certificate and key. + +## Resolution + +### Cause 1: cpaas-system/dex.tls is expired, self-signed, or not the certificate uploaded on a page + +1. Confirm that the leaf clients receive is `dex.tls`: + + ```bash + echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -dates -subject -issuer + kubectl get secret dex.tls -n cpaas-system -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject -issuer + ``` + + Stop if the two differ: an upstream load balancer may present its own certificate, or the certificate may be held in `prdb`. If a node reports `x509: certificate has expired or is not yet valid` and its clock is wrong, fix the clock instead. + +2. Put the replacement PEM files on the global control-plane node you run `kubectl` from: `tls.crt` (the certificate, followed by any intermediates from the issuer) and `tls.key` (the private key). Check them: + + ```bash + openssl x509 -in tls.crt -noout -pubkey | openssl md5 + openssl pkey -in tls.key -pubout | openssl md5 + openssl x509 -in tls.crt -noout -dates -subject -issuer -ext subjectAltName + ``` + + Continue only when the two hashes match, the new `notAfter` is in the future, and the certificate names the current platform host. + +3. Back up, delete, and recreate the certificate. + + > **Warning:** These commands delete Certificate `dex-serving-cert` and Secret `dex.tls` in `cpaas-system`; HTTPS to the platform address fails until `kubectl create secret tls` completes. Use a maintenance window and run the commands only when step 1 showed that the served leaf is `dex.tls` and step 2 verified the new pair. Rollback is `kubectl apply -f` of the two backup files, which restores the previous certificate. On an ACP 4.0 disaster-recovery pair, repeat the same steps on the standby global. + + ```bash + umask 077 + kubectl get certificate -n cpaas-system dex-serving-cert --ignore-not-found=true -o yaml > "$HOME/dex-serving-cert.yaml" + kubectl get secret -n cpaas-system dex.tls -o yaml > "$HOME/dex.tls.yaml" + kubectl delete certificate -n cpaas-system dex-serving-cert --ignore-not-found=true + kubectl delete secret -n cpaas-system dex.tls + kubectl create secret tls dex.tls --cert=tls.crt --key=tls.key -n cpaas-system + ``` + +4. Verify the Secret and the served leaf, then retry any failed image pull: + + ```bash + kubectl get secret dex.tls -n cpaas-system -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject -issuer + echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -dates -subject -issuer + ``` + + Both show the new subject and dates. The backup `dex.tls.yaml` contains a private key; remove it once the new certificate is confirmed. If business clusters still report the previous `dex.tls` expiry, wait for their automatic renewal and check that their `notAfter` moves forward. + +### Cause 2: dex.tls is reported expired but its dates are valid, or users only use HTTP + +1. Print the dates before any change (step 1 of Cause 1). If `notAfter` is still in the future, leave `dex.tls` in place; the report clears on a later inspection run. +2. If the dates are really past, replace the certificate with Cause 1 when HTTPS clients or the platform registry must trust it. An environment that is reached only over HTTP keeps working without the replacement. + +### Cause 3: the platform address cannot be changed, so a trusted certificate must match the existing name + +1. Read the name the platform uses today from the certificate it serves: + + ```bash + echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -subject -ext subjectAltName -dates + ``` + +2. Obtain a trusted certificate issued for that name, and replace it with Cause 1 in a maintenance window. +3. A move from an IP address to a domain name has no established procedure; raise it as a separate request with support. + +### Cause 4: the primary and standby global clusters use different platform certificates + +1. On the primary global and on the standby global, compare the certificate and read the `cluster-transform` log: + + ```bash + kubectl get secret dex.tls -n cpaas-system -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -fingerprint -sha256 -subject -dates + kubectl get pods -A | grep cluster-transform + kubectl logs -n --tail=200 | grep -i tls + ``` + +2. If the fingerprints differ, replace the certificate with the same `tls.crt` and `tls.key` on both globals using Cause 1, including its Warning. +3. Check the business clusters' `base` status in minfo again. + +### Cause 5: deployment-time certificate choices + +1. Without a CA-issued certificate, install with a self-signed certificate. Replacing it later with a trusted certificate for the same domain is a separate maintenance-window change (Cause 1). +2. For a wildcard certificate download, choose the nginx type and check the pair before installation: + + ```bash + openssl x509 -in tls.crt -noout -subject -ext subjectAltName -dates + openssl x509 -in tls.crt -noout -pubkey | openssl md5 + openssl pkey -in tls.key -pubout | openssl md5 + ``` + +## Diagnostic Steps + +1. Print what port 443 on the platform host actually sends, `dex.tls`, and `prdb` `base` if it holds an inline certificate: + + ```bash + echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -dates -subject -issuer + kubectl get secret dex.tls -n cpaas-system -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject -issuer + kubectl get prdb base -o jsonpath='{.spec.http.tls.cert}' 2>/dev/null \ + | base64 -d | openssl x509 -noout -dates -subject -issuer + ``` + + The object whose subject and dates match the client error is the one to replace. If `dex.tls` already holds the new certificate but the browser does not, check a load balancer in front of the platform. + +2. Branch: + + - Browser or registry leaf equals `dex.tls` and is expired, self-signed, or untrusted: Cause 1. + - Inspection says expired but `notAfter` is in the future, or only HTTP is used: Cause 2. + - A question about replacing the certificate with a trusted one, or moving from IP to a domain name: Cause 3. + - Disaster-recovery pair with business-cluster `base` abnormal and `cluster-transform` reporting inconsistent `tls.crt`: Cause 4. + - Questions during installation: Cause 5. + - A platform component certificate in `cpaas-system` with about 30 days left: a different object, see "Platform certificate expiry alert for a cert-manager serving certificate that renews by itself on ACP". diff --git a/docs/en/solutions/Replacing_or_rejoining_an_ACP_master_node_same-IP_replacement_etcd-join_extra_masters_and_control-plane_requests.md b/docs/en/solutions/Replacing_or_rejoining_an_ACP_master_node_same-IP_replacement_etcd-join_extra_masters_and_control-plane_requests.md new file mode 100644 index 000000000..555468d17 --- /dev/null +++ b/docs/en/solutions/Replacing_or_rejoining_an_ACP_master_node_same-IP_replacement_etcd-join_extra_masters_and_control-plane_requests.md @@ -0,0 +1,151 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Replacing or rejoining an ACP master node: same-IP replacement, etcd-join, extra masters, and control-plane requests + +## Issue + +- A master has to be replaced (virtual machine to physical machine, or the reverse), and the question is whether to delete it and add it again. +- After a master was removed and added back, kube-apiserver restarts and nothing listens on `127.0.0.1:2379`. +- An extra master is requested so that external clients can call `http://:6443` and master CPU is spread out, or apiserver is to run on two workers. +- During a drill, a newly added machine is expected to become a master after a master fails. +- Master component CPU and memory requests look too small. + +Related articles: "ACP control-plane limits: apiserver address change, snapshot.db scope, total master loss, x86 to ARM, and planned shutdown" covers changes that have no supported procedure. + +## Root Cause + +### Cause 1: Replace one master and keep the same node IP + +When the new machine keeps the old master IP, the path for a business cluster is to delete that master in the platform UI and add it again, one node at a time. After each node, etcd membership and the Kube-OVN `OVN_Northbound` and `OVN_Southbound` databases must list the new master and must not list the removed one. + +The Global Cluster Master Node Reset and Replacement operation guide is validated for v3.18.x, v4.0.x, v4.1.x, and v4.2.x, not for v4.3.x; on 4.3 use it as a reference only after a test cluster has been through it. Keeping the IP is a precondition in both paths. Adding a spare node now so that it can take over a master role later is not possible, because the node role is chosen when the node joins and cannot be changed afterwards. + +### Cause 2: After rejoin, nothing listens on 127.0.0.1:2379 because etcd-join did not finish + +The master was removed and added back. kube-apiserver is already generated and keeps exiting, and nothing listens on `127.0.0.1:2379`, so kube-apiserver cannot reach etcd. While the node was out, CoreDNS P99 latency on another master could stay high. The kubeadm etcd join phase did not run; `kubeadm join phase etcd-join --config=/etc/kubernetes/kubeadm-config.yaml --v=5` on the cleaned node completes it. This is not a WAL corruption and not a reason to delete `/var/lib/etcd`. + +### Cause 3: Another master only to expose its :6443 or spread load + +Adding control-plane nodes is supported, and the count should stay odd (3 or 5). Publishing `http://:6443` to external clients, or running apiserver alone on former workers to spread load, is not an accepted layout and does not meet the need to spread pressure. + +### Cause 4: A machine added in an expansion plan stays a compute node + +In a drill that adds a device and then fails a master, the new device is expanded as a compute node, not imaged as a management node. The failed master is recovered first with the replacement procedure (Cause 1) until the cluster is back to three masters; after that a further master or worker failure still leaves the platform running. No step turns the new device into a master. + +### Cause 5: apiserver and etcd requests are adjusted in their static Pod YAML + +Master system-component requests are changed by editing the static Pod YAML of the components, including kube-apiserver and etcd. There is no fixed target value; use the size planned for the cluster. etcd Pods that are OOM-killed (with etcd sync then failing) are addressed by increasing the etcd component's resources. + +## Resolution + +### Cause 1: Replace one master and keep the same node IP + +Continue only when the replacement machine uses the same IP as the master being removed, you are on 3.18, 4.0, 4.1, or 4.2 (or on 4.3 after a test cluster completed the procedure), and you replace one master at a time. Masters that also run workloads or pin log and monitoring Pods need extra planning; confirm the node is a clean master before the delete. + +> **Warning:** Deleting a master in the platform UI removes a control-plane node and its etcd member. Do it only when the replacement keeps the same IP, `endpoint health` shows the other members healthy, and only one master at a time. The delete cannot be undone; the rollback is to add a machine back at that IP and wait until `member list` and both OVN cluster statuses agree. + +Wait until the delete finishes before the add. After adding the new machine in the UI with the same IP, confirm membership before touching the next master: + +```bash +kubectl -n kube-system exec etcd- -- etcdctl \ + --endpoints=https://127.0.0.1:2379 \ + --cacert=/etc/kubernetes/pki/etcd/ca.crt \ + --cert=/etc/kubernetes/pki/etcd/server.crt \ + --key=/etc/kubernetes/pki/etcd/server.key \ + member list -w table +``` + +The removed master's peer URL must be absent and the new master's present. On Kube-OVN clusters, check the OVN databases in Deployment `ovn-central` in `kube-system`: + +```bash +kubectl -n kube-system exec deploy/ovn-central -c ovn-central -- ovn-nbctl cluster/status OVN_Northbound +kubectl -n kube-system exec deploy/ovn-central -c ovn-central -- ovn-sbctl cluster/status OVN_Southbound +``` + +Both must show the new master and not the removed one. If `-c ovn-central` fails, read the container name from `kubectl -n kube-system get deploy ovn-central -o yaml`. Then check etcd health: + +```bash +kubectl -n kube-system exec etcd- -- etcdctl \ + --endpoints=https://127.0.0.1:2379 \ + --cacert=/etc/kubernetes/pki/etcd/ca.crt \ + --cert=/etc/kubernetes/pki/etcd/server.crt \ + --key=/etc/kubernetes/pki/etcd/server.key \ + endpoint health --cluster +``` + +Proceed to the next master only when that command reports every member healthy and the two OVN statuses match the new set. If the replacement uses a different IP, this procedure does not apply; update the VIP forwarding policy to the new IP before clients use it, and follow the platform procedure for replacing masters with new IPs. + +### Cause 2: After rejoin, nothing listens on 127.0.0.1:2379 because etcd-join did not finish + +Apply only when this master was removed and is being added back, kube-apiserver is restarting, and nothing listens on `127.0.0.1:2379`. Confirm the other members are healthy first: + +```bash +ss -lnt | grep 2379 || true +ls -l /etc/kubernetes/kubeadm-config.yaml +``` + +> **Warning:** `kubeadm join phase etcd-join` adds this node's etcd member and starts it. Run it only on the cleaned node that is missing from `member list` or has no listener on `127.0.0.1:2379` after a rejoin, in a maintenance window, with the other members healthy. If the phase errors, stop and collect `member list` from a healthy master; do not wipe `/var/lib/etcd` unless the WAL is corrupt (`walpb: crc mismatch`). + +```bash +sudo kubeadm join phase etcd-join --config=/etc/kubernetes/kubeadm-config.yaml --v=5 +``` + +Afterwards `ss -lnt | grep 2379` shows a listener and kube-apiserver stops restarting. Check membership with the `member list` command from Cause 1 before the next master change. + +### Cause 3: Another master only to expose its :6443 or spread load + +If a real extra master is needed for quorum, keep the control-plane count odd (3 or 5), add it through the normal master-add path with its own IP, and check membership as in Cause 1. Do not build an external-endpoint or worker-hosted apiserver layout by hand. + +### Cause 4: A machine added in an expansion plan stays a compute node + +Treat the new device as a worker. If a master has already failed, restore that master with Cause 1 until `member list` shows three healthy members, then continue the drill. + +```bash +kubectl get nodes -l node-role.kubernetes.io/control-plane= -o wide +kubectl get nodes -o wide +``` + +### Cause 5: apiserver and etcd requests are adjusted in their static Pod YAML + +```bash +sudo ls -l /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/manifests/etcd.yaml +sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /var/tmp/kube-apiserver.yaml.before +sudo cp /etc/kubernetes/manifests/etcd.yaml /var/tmp/etcd.yaml.before +``` + +> **Warning:** Saving `/etc/kubernetes/manifests/kube-apiserver.yaml` or `etcd.yaml` restarts that static Pod on this node. One master at a time, in a maintenance window, and change only `resources.requests`. A request larger than the node's allocatable CPU or memory leaves the static Pod Pending. Keep the backups outside `/etc/kubernetes/manifests/`. Rollback: copy the backups back. + +Edit `resources.requests`, save, and wait until the Pod is Ready before the next master: + +```bash +kubectl -n kube-system get pods -l 'component in (kube-apiserver,etcd)' -o wide +``` + +## Diagnostic Steps + +```bash +kubectl -n kube-system get pods -l component=etcd -o wide +kubectl get nodes -l node-role.kubernetes.io/control-plane= -o wide +kubectl get machine -A +``` + +- Same IP, one master, version 3.18 to 4.2 (or 4.3 after a test run): Cause 1. +- Spare node added ahead of time so it can later become a master: not possible; the role is fixed at join (Cause 1). +- Rejoin, kube-apiserver restarting, nothing on `127.0.0.1:2379`: Cause 2. +- Request for an extra endpoint or worker-hosted apiservers: Cause 3. +- New device during a drill, expected to become a master: Cause 4. +- apiserver or etcd `requests` too small, or etcd OOM-killed: Cause 5. + +A `member list` that still contains a node you already deleted is an unfinished Cause 1 check, not a reason to wipe `/var/lib/etcd` on the surviving members. diff --git a/docs/en/solutions/Scheduled_etcd_backups_stop_after_an_upgrade_because_the_etcd-backup-default_AdvancedCronJob_stops_triggering_on_ACP.md b/docs/en/solutions/Scheduled_etcd_backups_stop_after_an_upgrade_because_the_etcd-backup-default_AdvancedCronJob_stops_triggering_on_ACP.md new file mode 100644 index 000000000..0fd297aa7 --- /dev/null +++ b/docs/en/solutions/Scheduled_etcd_backups_stop_after_an_upgrade_because_the_etcd-backup-default_AdvancedCronJob_stops_triggering_on_ACP.md @@ -0,0 +1,99 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.3.x +--- + +# Scheduled etcd backups stop after an upgrade because the etcd-backup-default AdvancedCronJob stops triggering on ACP + +## Issue + +A routine check of a cluster that was upgraded shows that the last etcd backup is several days old: + +- The newest snapshot file in the backup directory on the control plane nodes is from a date shortly after the upgrade, and nothing newer has been written. +- `kubectl get broadcastjobs -n cpaas-system` lists `etcd-backup-default-` BroadcastJobs only up to that date. No newer ones are created, even though the older ones completed (`DESIRED 3`, `SUCCEEDED 3`). The last one can also show one failed node. +- The `etcd-backup` feature is enabled, but the cluster has no etcd backup page in the console. + +## Root Cause + +Scheduled etcd backups are driven by the AdvancedCronJob (`acj`) `etcd-backup-default` in `cpaas-system`. At each scheduled time it creates a BroadcastJob (`bcj`) named `etcd-backup-default-` that runs the backup on every control plane node. The AdvancedCronJob works out its next trigger time from the BroadcastJobs it owns. + +During an upgrade, some BroadcastJobs can be created from the AdvancedCronJob without the schedule annotation. The AdvancedCronJob then fails to compute its next trigger time, and from then on it never triggers again. The object still exists and looks healthy, so the problem only shows up as backups that stop. This happens only occasionally. It is a known defect, and recreating the AdvancedCronJob clears it. + +The missing etcd backup page is a separate issue. The page is provided by the **Alauda Container Platform Cluster Enhancer** plugin, so a cluster where this plugin is not installed shows no backup page even though backups run in the background. + +## Resolution + +1. If the cluster has no etcd backup page, download the **Alauda Container Platform Cluster Enhancer** package that matches the platform version, upload it with `violet push`, and install it on the cluster from **Administrator** > **Marketplace** > **Cluster Plugins**. This restores the page. It does not by itself restart the stopped schedule. + +2. Save the AdvancedCronJob and record its schedule: + + ```bash + kubectl -n cpaas-system get acj etcd-backup-default -o yaml > /tmp/acj-etcd-backup-default.yaml + grep -n -E 'schedule|paused' /tmp/acj-etcd-backup-default.yaml + ``` + +3. Prepare a copy that can be applied again. In `/tmp/acj-etcd-backup-default.recreate.yaml`, remove `metadata.resourceVersion`, `metadata.uid`, `metadata.creationTimestamp`, `metadata.managedFields` and the whole `status` block: + + ```bash + cp /tmp/acj-etcd-backup-default.yaml /tmp/acj-etcd-backup-default.recreate.yaml + vi /tmp/acj-etcd-backup-default.recreate.yaml + ``` + +4. Delete the AdvancedCronJob and apply the saved copy. + + > **Warning:** While the AdvancedCronJob is deleted, the cluster has no scheduled etcd backup. Do this only when the diagnostic steps confirm that `etcd-backup-default` exists but has created no BroadcastJob since the last good backup, and only after `/tmp/acj-etcd-backup-default.yaml` has been saved. Rollback: `kubectl apply -f /tmp/acj-etcd-backup-default.recreate.yaml`. + + ```bash + kubectl -n cpaas-system delete acj etcd-backup-default + kubectl apply -f /tmp/acj-etcd-backup-default.recreate.yaml + kubectl -n cpaas-system get acj etcd-backup-default + ``` + +5. Verify that the schedule fires. Either wait for the next scheduled time, or set the schedule temporarily a few minutes ahead, confirm that a new BroadcastJob runs, and then set the schedule back to the value recorded in step 2: + + ```bash + kubectl -n cpaas-system edit acj etcd-backup-default # temporary schedule, then restore it + kubectl -n cpaas-system get broadcastjobs | grep etcd-backup-default + ``` + + A new `etcd-backup-default-` BroadcastJob should appear and reach `SUCCEEDED` equal to `DESIRED`, and a new snapshot file should appear on each control plane node. + +6. Repeat the procedure on every cluster that shows the same symptom. + +## Diagnostic Steps + +1. Check the dates of the snapshot files on each control plane node. Use the backup path configured for the cluster: + + ```bash + ls -lt | head + ``` + + Expected for this issue: the newest file is several days old. + +2. List the backup BroadcastJobs and compare their ages with the schedule: + + ```bash + kubectl get broadcastjobs -A | grep etcd-backup + ``` + + Expected for this issue: the newest `etcd-backup-default-` is as old as the newest snapshot, and no newer one exists. If new BroadcastJobs keep appearing but end in `FAILED`, the schedule is working and the backup run itself fails. That is a different problem: read the logs of the failed `etcd-backup` Pods instead. + +3. Confirm that the AdvancedCronJob exists and is not paused: + + ```bash + kubectl -n cpaas-system get acj etcd-backup-default -o jsonpath='{.spec.schedule} paused={.spec.paused}{"\n"}' + ``` + + Expected for this issue: the expected schedule and `paused=` empty or `false`. The object exists, but nothing has been created since the last good backup. + +4. When the backup page is missing, check in **Administrator** > **Marketplace** > **Cluster Plugins** whether **Alauda Container Platform Cluster Enhancer** is installed on that cluster. On the global cluster, the installed plugins of a cluster can also be listed with: + + ```bash + kubectl get moduleinfo -l cpaas.io/cluster-name= | grep -i enhancer + ``` + + No output means that the plugin is not installed on that cluster. diff --git a/docs/en/solutions/Security_scan_reports_a_weak_Grafana_admin_password_equal_to_the_username_on_ACP.md b/docs/en/solutions/Security_scan_reports_a_weak_Grafana_admin_password_equal_to_the_username_on_ACP.md new file mode 100644 index 000000000..927fdadf1 --- /dev/null +++ b/docs/en/solutions/Security_scan_reports_a_weak_Grafana_admin_password_equal_to_the_username_on_ACP.md @@ -0,0 +1,116 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# Security scan reports a weak Grafana admin password equal to the username on ACP + +## Issue + +A security scan of a cluster's monitoring endpoint reports a weak password on the Grafana login page, for example on a URL of the form: + +```text +http://:11780/clusters//grafana/login +``` + +The finding says that the Grafana `admin` user accepts the password `admin`, the same as the username. Operators then ask whether Grafana has a password of its own, separate from the platform login, and where to change it. + +## Root Cause + +### Cause 1: the platform Grafana admin password is still `admin` + +The platform Grafana runs in the `cpaas-system` namespace of each cluster (Pods named `kube-prometheus-grafana-`). It has its own `admin` account, separate from platform users, and that account's password is stored in Grafana itself. When the password has not been changed, it is `admin`, the same as the username, and the scanner reports it as a weak credential. + +The platform also keeps the Grafana credentials in the Secret `kube-prometheus-grafana-basic-auth` in `cpaas-system`. The Secret has the keys `username`, `password`, `guestUsername` and `guestPassword`. In this state, `username` and `password` both decode to `admin`. The password in Grafana and the `password` value in this Secret must match. So a password change has to be made in both places, and the `prometheus` and `warlock` components of that cluster are restarted afterwards. + +## Resolution + +### Cause 1 + +Run the steps on the cluster named in the scanned URL (`/clusters//grafana`), with `kubectl` access to that cluster. Choose the new password first; `` below stands for it. + +1. Find the Grafana Pod: + + ```bash + kubectl -n cpaas-system get pod | grep kube-prometheus-grafana + ``` + +2. Back up the credentials Secret and decode its current values: + + ```bash + kubectl -n cpaas-system get secret kube-prometheus-grafana-basic-auth -o yaml > kube-prometheus-grafana-basic-auth.bak.yaml + kubectl -n cpaas-system get secret kube-prometheus-grafana-basic-auth -o jsonpath='{.data.username}' | base64 -d; echo + kubectl -n cpaas-system get secret kube-prometheus-grafana-basic-auth -o jsonpath='{.data.password}' | base64 -d; echo + ``` + + `admin` and `admin` confirm the finding. + +3. Reset the Grafana admin password inside the Grafana Pod. + + > **Warning:** Impact: from this step until the Secret is updated in step 4, the password in Grafana and the copy in the Secret differ. Platform features that log in to Grafana with the Secret can fail during that time. Precondition: the backup from step 2 exists, and you run step 4 right after this one. Rollback: run the same command again with the old password (the value decoded in step 2), then restore the Secret with `kubectl apply -f kube-prometheus-grafana-basic-auth.bak.yaml`. Rolling back brings the weak password back. + + ```bash + kubectl -n cpaas-system exec -it -- \ + grafana-cli --homepath=/usr/share/grafana admin reset-admin-password '' + ``` + + If the Pod has more than one container, add `-c ` (list the containers with `kubectl -n cpaas-system get pod -o jsonpath='{.spec.containers[*].name}'`). + +4. Write the same password into the Secret. Change only the `password` key, and leave `username`, `guestUsername` and `guestPassword` unchanged: + + ```bash + NEW_B64=$(echo -n '' | base64 -w0) + kubectl -n cpaas-system patch secret kube-prometheus-grafana-basic-auth \ + --type merge -p "{\"data\":{\"password\":\"${NEW_B64}\"}}" + kubectl -n cpaas-system get secret kube-prometheus-grafana-basic-auth -o jsonpath='{.data.password}' | base64 -d; echo + ``` + + The last command must print the new password exactly. + +5. Restart `prometheus` and `warlock` on the same cluster. + + > **Warning:** Impact: while the Pods restart, metric queries, monitoring charts and alert evaluation on this cluster are briefly unavailable, and a short gap can appear in the collected data. Precondition: steps 3 and 4 are done and the Secret prints the new password. Rollback: none needed for a restart, because the controllers recreate the Pods. Choose a low-traffic time. + + ```bash + kubectl get pods -A | grep -E 'warlock|prometheus' + kubectl -n delete pod + kubectl -n delete pod + kubectl get pods -A | grep -E 'warlock|prometheus' + ``` + + Delete the Pods of the `warlock` and Prometheus workloads that the first command prints. Do not delete the Grafana Pod in this step. Wait until the recreated Pods are `Running` and ready. + +6. Verify and rescan. In the commands below, `` is the scanned URL without the trailing `/login`, for example `http://:11780/clusters//grafana`: + + ```bash + curl -sk -u 'admin:admin' -o /dev/null -w '%{http_code}\n' '/api/user' + curl -sk -u 'admin:' -o /dev/null -w '%{http_code}\n' '/api/user' + ``` + + Expect `401` for `admin:admin` and `200` for the new password. Then open the cluster monitoring pages and Grafana from the platform console, check that they still load, and run the security scan again. + +## Diagnostic Steps + +1. Confirm that the platform Grafana runs on the scanned cluster: + + ```bash + kubectl -n cpaas-system get pod | grep kube-prometheus-grafana + kubectl -n cpaas-system get secret | grep grafana + ``` + + Expected: a `Running` `kube-prometheus-grafana-` Pod and the Secret `kube-prometheus-grafana-basic-auth`. If there is no Grafana Pod, the scanned address belongs to a different Grafana, not to the platform one in `cpaas-system`. If the Secret has another name, use the name printed here in the Resolution steps. + +2. Confirm the weak credential from a host that can reach the scanned URL: + + ```bash + curl -sk -u 'admin:admin' -o /dev/null -w '%{http_code}\n' '/api/user' + ``` + + Expected: `200` while the weak password is still active, and `401` after the fix. + +3. After the change, check that Grafana and the Secret hold the same password. Decode the Secret as in step 2 of the Resolution, then test that value against Grafana with the second `curl` in step 6. A `401` for the Secret's value means the two differ. Repeat step 4 with the password that was set in Grafana, then repeat step 5. + +4. If the console reports `dial tcp
:11780: i/o timeout` when it opens Grafana, the problem is network reachability of port `11780`, not the password. See [Workload cluster abnormal or monitoring empty in the console because global cannot reach its ports on ACP](Workload_cluster_abnormal_or_monitoring_empty_in_the_console_because_global_cannot_reach_its_ports_on_ACP.md). diff --git a/docs/en/solutions/Service_Mesh_delete_is_stuck_on_a_failed_to_get_cluster_webhook_denial_or_on_pods_that_still_have_sidecar_injection.md b/docs/en/solutions/Service_Mesh_delete_is_stuck_on_a_failed_to_get_cluster_webhook_denial_or_on_pods_that_still_have_sidecar_injection.md new file mode 100644 index 000000000..6ed86c627 --- /dev/null +++ b/docs/en/solutions/Service_Mesh_delete_is_stuck_on_a_failed_to_get_cluster_webhook_denial_or_on_pods_that_still_have_sidecar_injection.md @@ -0,0 +1,168 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Service Mesh +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Service Mesh delete is stuck on a failed to get cluster webhook denial or on pods that still have sidecar injection + +## Issue + +A service mesh, or its gateway, will not uninstall and the console stays on deleting. One of these is seen: + +- The business cluster that the mesh belonged to no longer exists. Deleting the `servicemesh` object with `kubectl` is denied: + + ```text + admission webhook "mservicemesh.alauda.io" denied the request: failed to get cluster "": clusters.clusterregistry.k8s.io "" not found + ``` + + The `servicemesh` can still show `Deployed`, and removing its `finalizers` is denied with the same message. +- The ASM delete does not complete, and the `asm-operator` log reports that pods outside the mesh workloads being removed still have sidecar injection enabled. After that is fixed, `istio-operator` still does not go away. + +`servicemesh` objects live on the `global` cluster as `asm.alauda.io/v1alpha1` in `cpaas-system`, and the object name is the business cluster name. An unused mesh whose ASM process consumes a lot of CPU is removed through this same uninstall path. + +Related articles: "Leftover Service Mesh v1 recreates istio-system against Mesh v2 or Istio Secret certificate alerts remain on ACP" covers leftovers after an uninstall. + +## Root Cause + +### Cause 1: the `mservicemesh.alauda.io` webhook cannot find the deleted cluster + +The mutating webhook configuration `servicemesh-v1alpha1-mutation` (webhook `mservicemesh.alauda.io`, served by `asm-controller`) looks up the business cluster record for every change to a `servicemesh`. When that record is gone, the lookup fails and the webhook denies the delete and any finalizer patch, so the object can never be removed while the webhook is in place. + +### Cause 2: other workloads still have sidecar injection enabled + +ASM is deleted only after sidecar injection is disabled on every pod that still has it. A workload outside the mesh, for example `ingress-nginx`, can carry the sidecar injection annotation on its pod template; `asm-operator` then waits indefinitely. After injection is turned off, a leftover `istio-operator` object can still hold the delete and has to be force-removed. + +## Resolution + +### Cause 1: the `mservicemesh.alauda.io` webhook cannot find the deleted cluster + +Run these steps on the `global` cluster. + +1. If the cluster record still exists, this cause does not apply; remove the meshed workloads and delete normally: + + ```bash + kubectl get clusters.clusterregistry.k8s.io + kubectl get servicemesh -n cpaas-system -o yaml > /tmp/servicemesh-.bak.yaml + kubectl -n cpaas-system delete servicemesh --wait + ``` + +2. When the record is `not found`, back up the webhook configuration and the mesh objects, and inspect the webhook: + + ```bash + kubectl get mutatingwebhookconfiguration servicemesh-v1alpha1-mutation -o yaml > /tmp/servicemesh-v1alpha1-mutation.bak.yaml + kubectl get servicemesh -A -o yaml > /tmp/servicemesh-all.bak.yaml + kubectl get mutatingwebhookconfiguration servicemesh-v1alpha1-mutation \ + -o jsonpath='{range .webhooks[*]}{.name}{" "}{.clientConfig.service.name}{" "}{.failurePolicy}{"\n"}{end}' + ``` + +3. Take the webhook out of the path, using one of the two methods below. + + > **Warning:** While the webhook is bypassed, ServiceMesh changes on this `global` cluster are not validated by it. Only when the delete fails because the cluster record is missing and the backup files exist; use a maintenance window. Rollback: `kubectl apply -f /tmp/servicemesh-v1alpha1-mutation.bak.yaml` (remove `metadata.resourceVersion`, `uid`, `managedFields`, and `status` from the file if apply complains). + + - Method A: set `failurePolicy` of the `asm-controller` webhook to `Ignore`. This resolved a delete for a mesh whose cluster no longer existed. Kubernetes applies `failurePolicy` only when the webhook call itself fails; if the delete still returns the `failed to get cluster` denial after this change, use Method B. + + ```bash + kubectl patch mutatingwebhookconfiguration servicemesh-v1alpha1-mutation --type=json \ + -p '[{"op":"replace","path":"/webhooks/0/failurePolicy","value":"Ignore"}]' + ``` + + Use the index of the `mservicemesh.alauda.io` entry from the listing in step 2 if it is not `0`. + + - Method B: delete the webhook configuration. + + ```bash + kubectl delete mutatingwebhookconfiguration servicemesh-v1alpha1-mutation + ``` + +4. Delete the mesh object. + + > **Warning:** The ServiceMesh object and the gateways and injection it owns are removed. Because the cluster record is gone, the object cannot be recreated until that record exists again; treat this delete as irreversible. + + ```bash + kubectl get servicemesh -A + kubectl delete servicemesh -n cpaas-system + ``` + +5. Restore the webhook immediately: + + ```bash + kubectl apply -f /tmp/servicemesh-v1alpha1-mutation.bak.yaml + kubectl get mutatingwebhookconfiguration servicemesh-v1alpha1-mutation -o jsonpath='{.webhooks[*].failurePolicy}{"\n"}' + ``` + + A gateway that was stuck in deleting leaves that state once the webhook no longer denies the request. + +### Cause 2: other workloads still have sidecar injection enabled + +1. Read the `asm-operator` log and find the workloads it names: + + ```bash + kubectl get pods -A | grep asm-operator + kubectl -n istio-system logs --tail=200 | grep -i -E 'sidecar|inject' + kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" "}{.spec.containers[*].name}{"\n"}{end}' | grep istio-proxy + kubectl -n get deploy -o jsonpath='{.spec.template.metadata.annotations}{"\n"}' + ``` + +2. Set the Istio injection annotation `sidecar.istio.io/inject` to `"false"` on each named workload. Use `daemonset` instead of `deployment` where the workload is a DaemonSet. + + > **Warning:** The workload rolls and its new pods start without `istio-proxy`; ingress traffic through `ingress-nginx` can drop briefly. Only for workloads the `asm-operator` log names, and only when the mesh is really being removed. Rollback: set the annotation back to `"true"` (re-injection works only while the mesh is still installed). + + ```bash + kubectl -n patch deployment --type=merge \ + -p '{"spec":{"template":{"metadata":{"annotations":{"sidecar.istio.io/inject":"false"}}}}}' + kubectl -n rollout status deployment + ``` + +3. Retry the ASM delete. If it stops on `istio-operator`, find which object is left: + + ```bash + kubectl get ns | grep istio-operator + kubectl get pods -A | grep istio-operator + kubectl get istiooperator -A + kubectl get istiooperator -n -o jsonpath='{.metadata.deletionTimestamp}{" "}{.metadata.finalizers}{"\n"}' + ``` + +4. Force-remove the remaining object. + + > **Warning:** A force delete skips the controller's cleanup, so resources it would have removed can remain and need manual cleanup. Only after injection is disabled on the named workloads and the remaining `istio-operator` object has a `deletionTimestamp` or is stuck in `Terminating`. Rollback: none; keep the backup as a reference. + + ```bash + kubectl get istiooperator -n -o yaml > /tmp/istiooperator.bak.yaml + # stuck IstioOperator resource + kubectl patch istiooperator -n --type=merge -p '{"metadata":{"finalizers":null}}' + # Pod stuck in Terminating + kubectl delete pod -n --grace-period=0 --force + ``` + + Run only the line that matches the object left. The ASM delete should then complete. If the `servicemesh` on `global` is then denied with `failed to get cluster`, continue with Cause 1. + +## Diagnostic Steps + +1. On `global`, check the mesh object, the cluster record, and the webhook: + + ```bash + kubectl -n cpaas-system get servicemesh + kubectl get clusters.clusterregistry.k8s.io + kubectl get mutatingwebhookconfiguration servicemesh-v1alpha1-mutation + ``` + +2. On the business cluster, check what blocks `asm-operator`: + + ```bash + kubectl -n istio-system logs --tail=200 | grep -i -E 'sidecar|inject' + kubectl get ns,istiooperator -A | grep -i istio-operator + ``` + +| What you see | Cause | +| --- | --- | +| Delete denied by `mservicemesh.alauda.io` with `failed to get cluster` and the cluster record `not found` | 1 | +| ASM delete hangs, `asm-operator` log names pods that are still injected (for example `ingress-nginx`), or `istio-operator` stays | 2 | +| `istio-system` keeps returning, or Istio Secret certificate alerts after Mesh v1 is gone | The leftover Mesh v1 article | diff --git a/docs/en/solutions/Service_Mesh_tracing_Kiali_trace_links_return_404_a_workload_has_no_data_or_jaeger-prod-query_crashes_on_ACP.md b/docs/en/solutions/Service_Mesh_tracing_Kiali_trace_links_return_404_a_workload_has_no_data_or_jaeger-prod-query_crashes_on_ACP.md new file mode 100644 index 000000000..a68814a39 --- /dev/null +++ b/docs/en/solutions/Service_Mesh_tracing_Kiali_trace_links_return_404_a_workload_has_no_data_or_jaeger-prod-query_crashes_on_ACP.md @@ -0,0 +1,177 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Service Mesh +ProductsVersion: + - "4.2" + - "4.3" +--- + +# Service Mesh tracing: Kiali trace links return 404, a workload has no data, or jaeger-prod-query crashes on ACP + +## Issue + +One of these tracing symptoms appears on a cluster where the mesh is installed and running: + +- In ASM v2, every link from Kiali to the details of a trace opens a 404 page. +- Kiali pages for one workload show no data: Outbound Metrics show `No data available` for Request duration, Request size, Response size, gRPC received, and gRPC sent; only the TCP charts have data, and the Traces tab is empty. +- The container platform Observability tracing page shows no data and the error `There was an error quering for traces:Unknown Error` (the page text has that spelling). +- The call chain keeps showing an `unknown` request, and nobody can tell who sends it. +- After ASM v1 is deployed, Jaeger does not start and `jaeger-prod-query` in `istio-system` keeps crashing. +- After the platform log storage was switched from Elasticsearch to ClickHouse, `jaeger-prod-es-index-cleaner` in `istio-system` keeps reporting `failed to query indices and unmarshall response body`. + +## Root Cause + +### Cause 1: Kiali `external_url` for tracing is not set + +Kiali builds trace detail links from `spec.external_services.tracing.external_url` of the Kiali custom resource. The documented tracing block (`provider: jaeger`, `use_grpc: true`, `internal_url: "http://jaeger-prod-query.istio-system:16685/jaeger"`) shows `external_url` commented out as optional. Left unset, every trace detail link returns 404. The value has the form `/clusters//istio/jaeger`. + +### Cause 2: `service.istio.io/canonical-name` does not match the Deployment name + +The sidecar is injected and the Service port name (for example `http-8040-8040`) lets Istio detect HTTP, yet Kiali shows no request metrics. The Pod label `service.istio.io/canonical-name` is set to the Service name instead of the Deployment name, so telemetry is reported under a name Kiali does not look up for this workload. In the Jaeger UI, selecting the service `.` returns the traces that Kiali does not show. + +### Cause 3: the container platform tracing page is a separate product from Service Mesh v2 tracing + +This is a product boundary, not a fault. Mesh v1 bundles Jaeger v1 with a custom tracing page; on 4.x a separate tracing feature for containers was derived from that page and released as a standalone UI plugin with its own backend and indices. Mesh v2 integrates Jaeger in the open-source way, and its traces are read in the Jaeger UI or in Kiali. When the container tracing backend is not deployed, the page's API calls `GET /clusters//acp/jaeger/api/services` and `.../api/spans` return `404` with response headers `via: alb` and `x-alb-err-reason: BackendError`, although the mesh Jaeger UI (instance `jaeger-prod` in `istio-system`) lists services and the `istio-tracing--jaeger-span-` indices contain documents. This was confirmed on ACP 4.2.2. + +### Cause 4: a Prometheus scrape appears as an `unknown` caller + +The ztunnel or `istio-proxy` log of the destination workload shows an inbound `access connection complete` entry about once a minute from the same source address. That address belongs to a Prometheus instance scraping the mesh workloads. Those scrapes appear in the call chain as `unknown`. This is expected behavior. + +### Cause 5: the ACP 4.3.0 `asm-operator` generates a 44-byte `cookie_secret` + +The `jaeger-prod-query` Pod has two containers, `jaeger-query` and `oauth2-proxy`. The `oauth2-proxy` container exits with: + +```text +invalid configuration: cookie_secret must be 16, 24, or 32 bytes to create an AES cipher, but is 44 bytes +``` + +Secret `oauth2proxy-secret` in `istio-system` is generated by ASM (Helm release `asm`, owned by `Asm/asm`). The ACP 4.3.0 `asm-operator` writes a 44-byte value, which `oauth2-proxy` rejects. This is a known issue of that package and is fixed in the ACP v4.3.1 `asm-operator` package. A plain `kubectl logs | grep error` shows nothing because the error is in the `oauth2-proxy` container. + +### Cause 6: ASM v1 tracing supports only Elasticsearch + +`jaeger-es-index-cleaner` is a Job that removes old Elasticsearch tracing indices (traces are kept for 7 days by default). ASM v1 tracing supports only Elasticsearch as storage, not ClickHouse. After Elasticsearch is removed, the cleaner still calls the old address, receives a portal HTML page, and cannot parse it. Log queries on ClickHouse are unaffected. ASM v2 has no `jaeger-es-index-cleaner`. + +## Resolution + +### Cause 1: Kiali `external_url` for tracing is not set + +1. Confirm the field is empty: + + ```bash + kubectl get kiali -A + kubectl get kiali -n -o jsonpath='{.spec.external_services.tracing}{"\n"}' + ``` + +2. Set it. + + > **Warning:** The Kiali operator restarts Kiali, so the Kiali UI is briefly unavailable. Only when trace links return 404 and `external_url` is empty. Rollback: `kubectl apply -f /tmp/kiali.bak.yaml`. + + ```bash + kubectl get kiali -n -o yaml > /tmp/kiali.bak.yaml + kubectl patch kiali -n --type=merge \ + -p '{"spec":{"external_services":{"tracing":{"external_url":"/clusters//istio/jaeger"}}}}' + ``` + + `` is the platform address used in the browser and `` is the cluster that runs Jaeger. After Kiali is Ready, trace links should open the Jaeger trace. + +### Cause 2: `service.istio.io/canonical-name` does not match the Deployment name + +1. Compare the label with the Deployment name and check the sidecar and port name: + + ```bash + kubectl -n get pod -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{end}' + kubectl -n get pod -o jsonpath='{.metadata.labels.service\.istio\.io/canonical-name}{" "}{.metadata.labels.service\.istio\.io/canonical-revision}{"\n"}' + kubectl -n get svc -o jsonpath='{range .spec.ports[*]}{.name}{" "}{.port}{"\n"}{end}' + ``` + +2. Set the label on the Pod template; a label on a running Pod is lost when it is recreated. + + > **Warning:** The workload rolls, and its telemetry is reported under the new name from then on. Only when the label differs from the Deployment name and Jaeger shows traces under `.`. Rollback: patch the previous value back. + + ```bash + kubectl -n patch deployment --type=merge \ + -p '{"spec":{"template":{"metadata":{"labels":{"service.istio.io/canonical-name":""}}}}}' + kubectl -n rollout status deployment + ``` + + If Kiali still reports `Pod has no Istio sidecar`, injection is the problem, not the label. + +### Cause 3: the container platform tracing page is a separate product from Service Mesh v2 tracing + +1. Confirm that Mesh v2 traces are stored and readable: + + ```bash + kubectl get jaeger -n istio-system + kubectl get pods -n istio-system | grep -E 'jaeger|kiali' + ``` + + Open the mesh Jaeger UI or the Kiali Traces tab and check that traces are listed. + +2. In the browser developer tools on the tracing page, a `404` on `/acp/jaeger/api/services` or `/acp/jaeger/api/spans` with `x-alb-err-reason: BackendError` matches this cause. +3. Read Mesh v2 traces in the Jaeger UI or Kiali. No mesh change makes the container platform tracing page show Mesh v2 traces. If the Jaeger UI is also empty, this cause does not apply. + +### Cause 4: a Prometheus scrape appears as an `unknown` caller + +1. Map the recurring source address to a Pod: + + ```bash + kubectl logs -n --tail=1000 | grep 'access connection complete' | grep '' + kubectl logs -n -c istio-proxy --tail=1000 + kubectl get pod -A -o wide | grep '' + ``` + + Use the ztunnel log in ambient mode and the `istio-proxy` log in sidecar mode. + +2. An address that resolves to a Prometheus Pod (or another scraper) is expected traffic. Stopping that scrape removes the `unknown` entry; no mesh object needs to change. + +### Cause 5: the ACP 4.3.0 `asm-operator` generates a 44-byte `cookie_secret` + +1. Confirm the signature: + + ```bash + kubectl logs -n istio-system -c oauth2-proxy --tail=50 + kubectl get secret -n istio-system oauth2proxy-secret -o jsonpath='{.data.OAUTH2_PROXY_COOKIE_SECRET}' | base64 -d | wc -c + ``` + + A length of `44` with the log line above matches this cause. + +2. Replace the `asm-operator` package and redeploy the mesh. Do not hand-edit the generated Secret. + + > **Warning:** Uninstalling the mesh removes all mesh components from the managed clusters; injection, gateways, and tracing stop until ASM v1 is deployed again. Only when the platform is 4.3.0, the log shows the 44-byte error, and a maintenance window is open. Rollback: not useful; the 4.3.0 package brings the crash back. + + 1. Uninstall the mesh. If the delete gets stuck, see "Service Mesh delete is stuck on a failed to get cluster webhook denial or on pods that still have sidecar injection". + 2. On the `global` cluster, go to **Marketplace** > **Upload Packages** > **Operator**, delete the 4.3.0 `asm-operator` package, and upload the ACP v4.3.1 `asm-operator` package. + 3. Deploy ASM v1 again. + +3. All Pods in `istio-system` should be Running, with `jaeger-prod-query` at `2/2` and no restarts. + +### Cause 6: ASM v1 tracing supports only Elasticsearch + +1. Confirm the error and the address the cleaner calls: + + ```bash + kubectl get cronjob,job -n istio-system | grep es-index-cleaner + kubectl logs -n istio-system --tail=50 + kubectl get cronjob -n istio-system -o jsonpath='{.spec.jobTemplate.spec.template.spec.containers[*].args}{"\n"}' + ``` + +2. ClickHouse cannot replace Elasticsearch for ASM v1 tracing. Either keep an Elasticsearch backend for ASM v1 tracing, or move the mesh to ASM v2, where the cleaner does not exist and tracing uses the separate stack described in "Moving Service Mesh v2 tracing storage from Elasticsearch to OpenSearch with Jaeger v2". Moving editions means uninstalling ASM v1 and installing ASM v2; evaluate the differences in sidecar injection, gateway deployment, and the effect on Alauda AI if it uses the mesh before choosing it. + +## Diagnostic Steps + +```bash +kubectl get kiali -A +kubectl get pods -n istio-system | grep -E 'jaeger|kiali' +kubectl logs -n istio-system -c oauth2-proxy --tail=20 +``` + +| What you see | Cause | +| --- | --- | +| Trace link from Kiali opens 404, `external_url` empty | 1 | +| One workload: no request metrics, Traces tab empty, Jaeger has traces under `.`, canonical-name is the Service name | 2 | +| Platform tracing page `Unknown Error`, `/acp/jaeger/api/*` returns 404 with `x-alb-err-reason: BackendError`, Jaeger UI has traces | 3 | +| `unknown` request about once a minute, source address is a Prometheus Pod | 4 | +| `jaeger-prod-query` crash loop, `oauth2-proxy` logs `cookie_secret ... but is 44 bytes`, platform 4.3.0 | 5 | +| `jaeger-prod-es-index-cleaner` logs `failed to query indices and unmarshall response body`, log storage moved to ClickHouse | 6 | diff --git a/docs/en/solutions/Service_mesh_history_is_not_visible_after_switching_the_Elasticsearch_integration_to_another_cluster_on_ACP.md b/docs/en/solutions/Service_mesh_history_is_not_visible_after_switching_the_Elasticsearch_integration_to_another_cluster_on_ACP.md new file mode 100644 index 000000000..9af17cea5 --- /dev/null +++ b/docs/en/solutions/Service_mesh_history_is_not_visible_after_switching_the_Elasticsearch_integration_to_another_cluster_on_ACP.md @@ -0,0 +1,41 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Service Mesh +ProductsVersion: [] +--- + +# Service mesh history is not visible after switching the Elasticsearch integration to another cluster on ACP + +## Issue + +A service mesh stores its data in the platform Elasticsearch of the `global` cluster, and it has to move to the Elasticsearch of another cluster. Administrators ask whether the setting can be changed in place, and after the change they find that the data recorded before the switch no longer appears in the mesh views. + +The mesh's **Update Elasticsearch Integration Configuration** dialog shows **Source** set to **Platform** (the alternative is **External**), **Cluster** set to `global`, and an automatically filled **Access Address** that ends in `/es_proxy`. + +## Root Cause + +The Elasticsearch integration of a service mesh can be changed directly: selecting another cluster with **Source** = **Platform** points the mesh at that cluster's platform Elasticsearch, and the access address is generated automatically. + +The switch does not migrate data. Historical data stays in the Elasticsearch of the original cluster. After the switch, the mesh creates new indices in the new Elasticsearch, writes new data there, and queries only those new indices. Data recorded before the switch is therefore not visible from the mesh, although it still exists in the original Elasticsearch. + +## Resolution + +1. Open the service mesh and its **Update Elasticsearch Integration Configuration** dialog. + +2. Keep **Source** as **Platform**, select the target cluster in **Cluster**, and check that **Access Address** is regenerated for that cluster (it still ends in `/es_proxy`). + +3. Click **Update**. + + > **Warning:** After the update, the mesh writes to and reads from the new Elasticsearch only. Data recorded before the switch is not copied and is no longer shown in the mesh, although it remains in the original Elasticsearch. Plan the switch at a time when the loss of history in the mesh views is acceptable, and keep the original Elasticsearch until its data is no longer needed. To roll back, open the same dialog, select the original cluster again and click **Update**. Data written to the new cluster in the meantime stays there and is not copied back. + +4. Generate some traffic in the mesh and confirm that new data appears in the mesh views. + +## Diagnostic Steps + +1. Reopen **Update Elasticsearch Integration Configuration** and check that **Cluster** shows the new cluster and **Access Address** points to it. + +2. If the new data appears but the old data does not, this is the expected result of the switch, not a failure: the old data is still in the original cluster's Elasticsearch. + +3. If no new data appears either, check that the platform Elasticsearch of the target cluster is installed and healthy before suspecting the mesh configuration. diff --git a/docs/en/solutions/Shutting_down_or_rebooting_ACP_nodes_for_maintenance.md b/docs/en/solutions/Shutting_down_or_rebooting_ACP_nodes_for_maintenance.md new file mode 100644 index 000000000..d486044f3 --- /dev/null +++ b/docs/en/solutions/Shutting_down_or_rebooting_ACP_nodes_for_maintenance.md @@ -0,0 +1,228 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - 3.x + - 4.x +--- + +# Shutting down or rebooting ACP nodes for maintenance + +## Issue + +Typical questions before planned work on nodes: + +- "We need to reboot some worker nodes. Can we just reboot the servers?" +- "A server's RAID card or memory failed and must be shut down for replacement. What must the platform side do?" +- "One global (management) server needs a memory replacement. What is the shutdown plan, how do we back up etcd, and what do we check after the restart?" +- "A node of the logging cluster hosts Kafka, ZooKeeper and Elasticsearch and will be shut down for repair. Do the logging components need any preparation?" +- "We will upgrade the kernel, or the OS (for example Kylin SP2 to SP3). What must be done before the reboot?" +- "The IaaS will migrate all VMs for a hardware expansion, with short interruptions on every node. Will the cluster be affected?" +- "After patching we rebooted the nodes; the Pods on each node were evicted and recreated elsewhere, so replicas are now unevenly spread. Does the platform taint the node during a reboot, and can the eviction time be controlled?" +- "All three masters of a test environment restarted at the same time. Nodes look normal; is anything else needed?" + +These procedures take nodes down one at a time while the rest of the cluster keeps running. Shutting down every node of the platform at once (a planned power outage of the whole site) is not covered here. + +Related articles: *Node capacity, pod limit or data disk after resizing or replacing node hardware on ACP* (capacity that does not update after a resize, `maxPods`, and replacing a data disk), and *Safely resizing CPU and memory on Alauda Container Platform nodes*. + +## Root Cause + +These are planned operations, not faults. The procedures exist for three reasons: + +- **Pods move only if they are evicted first.** If a node is rebooted without preparation, its Pods stay bound to it and are unavailable until it returns. Pods pinned to the node (`nodeSelector`, local volumes) never move and are unavailable while it is down. +- **Control-plane and global nodes hold etcd and platform components.** Losing one member of a 3-node etcd keeps quorum, but the data has to be protected against a second failure during the work. +- **Node failures are handled with taints.** About 40 seconds after a node stops reporting, kube-controller-manager adds the `node.kubernetes.io/not-ready` or `node.kubernetes.io/unreachable` taint with effect `NoExecute`. The platform does not add any taint of its own during a reboot. + +### Cause 1: A worker node is rebooted or shut down + +The node's business Pods have to be evicted first so they run elsewhere during the work. Without preparation, Pods are deleted when their toleration for the `NoExecute` taint expires (300 seconds by default in upstream Kubernetes, unless the Pod sets its own) and replaced on other nodes; they are not moved back afterwards, which leaves replicas unevenly spread. Replacement Pods that cannot be placed show `FailedScheduling ... node(s) had untolerated taint {node.kubernetes.io/unreachable: }`. + +### Cause 2: A control-plane node or a global-cluster node is shut down + +etcd runs on the control-plane nodes. With 3 control-plane nodes, one can be down while etcd keeps quorum; an etcd backup protects against a second failure. While a global node is down, platform pages, log and monitoring collection, and components fixed to that node can be affected; business services already running in business clusters are generally not affected. + +### Cause 3: A node of the logging cluster (Kafka, ZooKeeper, Elasticsearch) is shut down + +The logging components on that node have to be stopped before the node goes down, whether the Elasticsearch instance on it is a data node or a master node. + +### Cause 4: Kernel or OS upgrade with reboots + +Each node is rebooted, so the worker procedure applies node by node, after an etcd backup. There is no platform-specific OS upgrade procedure; the OS upgrade is carried out by the user, and the target OS version must be one the platform supports. + +### Cause 5: IaaS hardware expansion or VM migration + +Whether preparation is needed depends on whether the cluster runs business workloads at all. Decide this from what is deployed, not from the cluster's name or type: a new business cluster can still hold only platform base components. + +## Resolution + +### Cause 1: A worker node is rebooted or shut down + +Work on **one node at a time**; start the next node only when the previous one is `Ready` and its Pods are running. + +1. Check that the remaining nodes can absorb the node's Pods: + +```bash +kubectl describe nodes | grep -A8 'Allocated resources' +kubectl get pdb -A +``` + +2. Mark the node unschedulable and evict its Pods. In the console this is the node's **Evict Pods** action; from the CLI: + +> **Warning:** Eviction restarts every non-DaemonSet Pod of the node on other nodes, and `--delete-emptydir-data` deletes their `emptyDir` data. Single-replica services are briefly unavailable, and Pods pinned to this node stay down until it returns. Precondition: the other nodes have room, and PodDisruptionBudgets allow the eviction. Roll back with `kubectl uncordon `; evicted Pods do not move back by themselves. + +```bash +kubectl cordon +kubectl drain --ignore-daemonsets --delete-emptydir-data --timeout=10m +``` + +3. Reboot, or shut down and do the hardware work (RAID controller, memory, VM specification): + +```bash +systemctl reboot # or: shutdown -h now +``` + +4. When the node is back and `Ready`, re-enable scheduling and check for Pods that are not running: + +```bash +kubectl get node +kubectl uncordon +kubectl get pods -A -o wide | grep -viE 'Running|Completed' +``` + +Platform components on a worker need no extra handling; they recover by themselves after the restart. Rebooting directly without cordon and eviction is not recommended on clusters that carry business workloads. + +**Keeping Pods on a node through a short reboot.** To keep Pods bound to the node for up to 15 minutes while it reboots, so that only their restart count increases, set both tolerations in each workload's Pod template: + +```yaml +spec: + tolerations: + - key: "node.kubernetes.io/not-ready" + operator: "Exists" + effect: "NoExecute" + tolerationSeconds: 900 + - key: "node.kubernetes.io/unreachable" + operator: "Exists" + effect: "NoExecute" + tolerationSeconds: 900 +``` + +> **Warning:** Changing the Pod template rolls out new Pods for the workload. A longer toleration also delays replacement when a node really fails: its Pods stay unavailable for up to `tolerationSeconds` before they are recreated elsewhere. Agree the value with the application owner and apply it in a change window. Roll back by removing the tolerations or restoring the previous value. + +```bash +kubectl get pod -n -o jsonpath='{range .spec.tolerations[*]}{.key}{" "}{.effect}{" "}{.tolerationSeconds}{"\n"}{end}' +``` + +On clusters whose node configuration is managed by Machine Configuration, a node left cordoned also keeps its machine configuration pool from reporting all machines ready: a node counts as ready only when it is `NodeReady=True` and not unschedulable. For example, a cordoned master with its configuration applied shows `MachineCount = 3`, `UpdatedMachineCount = 3`, `ReadyMachineCount = 2`, with no errors in the controller logs. That is expected; uncordon the node when its maintenance is over: + +```bash +kubectl get nodes | grep SchedulingDisabled +kubectl uncordon +``` + +### Cause 2: A control-plane node or a global-cluster node is shut down + +1. Confirm that the cluster has 3 control-plane nodes, and shut down only one at a time. +2. Back up etcd first. Use the platform's etcd backup feature if it is configured (see the etcd backup documentation for your ACP version), and take a manual snapshot from the etcd static Pod. Read the certificate paths and data directory from the Pod spec instead of guessing them: + +```bash +kubectl -n kube-system get pods -l component=etcd -o wide +kubectl -n kube-system get pod etcd- -o yaml | grep -E -- '--(cert-file|key-file|trusted-ca-file|data-dir)=' +kubectl -n kube-system exec etcd- -- etcdctl \ + --endpoints=https://127.0.0.1:2379 \ + --cacert= --cert= --key= \ + snapshot save /snapshot-before-maintenance.db +``` + +Copy the snapshot file off the node (it is under the data directory on the host) and confirm its size is not zero. + +3. Shut the node down normally: + +```bash +shutdown -h now +``` + +4. After the repair and power-on, check that the node is `Ready` and the platform Pods on it are healthy, then handle any component that does not recover. On a node that runs Ceph, also check that Ceph has returned to normal. + +```bash +kubectl get nodes +kubectl -n kube-system get pods -o wide | grep +kubectl get pods -A -o wide | grep | grep -viE 'Running|Completed' +``` + +### Cause 3: A node of the logging cluster (Kafka, ZooKeeper, Elasticsearch) is shut down + +Stop the logging components first, then shut the node down for repair. Cordon and drain are the worker procedure for business workloads, not part of this step. To confirm the Elasticsearch role or cluster health before the shutdown, use read-only checks only. After the repair and power-on, confirm the node is `Ready` and the logging components on it are running again: + +```bash +kubectl get node +kubectl get pods -A -o wide | grep | grep -viE 'Running|Completed' +``` + +### Cause 4: Kernel or OS upgrade with reboots + +Back up etcd first (Cause 2). Then: + +**Business can be stopped:** + +1. Stop the business containers. +2. Upgrade and reboot the compute nodes one by one, confirming each node is normal before the next. +3. Upgrade the control-plane nodes one by one. +4. Start the business. + +**Business cannot be stopped:** + +1. For each compute node: cordon it, evict the business Pods (Cause 1), and confirm the business has recovered on other nodes. +2. Upgrade the kernel and reboot. Confirm the node has rejoined normally, then uncordon it. +3. Repeat for the other compute nodes, then upgrade the control-plane nodes one by one. + +Schedule an OS version upgrade in a coordinated maintenance window, with the same node-by-node approach. + +### Cause 5: IaaS hardware expansion or VM migration + +Ask the cluster owner, and list what runs outside the platform's own namespaces (read-only): + +```bash +kubectl get ns +kubectl get deploy,sts -A +``` + +- **No business workloads** (only platform base components): there is no significant risk. The expansion and VM migration can go ahead directly, and the hosts can simply be rebooted. Wait a while after the reboot, then list the Pods that are not running: + +```bash +kubectl get po -A | grep -vi run +``` + +- **Business workloads**: migrate or restart nodes one at a time with the Cause 1 procedure, for example shutting each node down, giving it a new VM specification or moving it to another resource pool, and restarting it, with each node `Ready` before the next. + +After any VM migration, check the container runtime on each node; a migration can corrupt the runtime data directory and runtime binaries. + +## Diagnostic Steps + +Before maintenance: + +```bash +kubectl get nodes -o wide +kubectl get pods -A -o wide --field-selector spec.nodeName= +kubectl get pdb -A +kubectl -n kube-system get pods -l component=etcd -o wide +``` + +Pods with a local PV, or a `nodeSelector` or `nodeName` for this node, will not move; plan their downtime. Fewer than 3 etcd members means no control-plane node can be shut down safely. + +After maintenance: + +```bash +kubectl get node +kubectl get pods -A | grep -vi run +``` + +After an unplanned restart of all masters at once, the same check is the first step. Periodic job Pods in `cpaas-system` (names such as `--`) in state `0/1 Completed` with 0 restarts are normal; with no remaining alerts, no further action is needed. + +If a node gets stuck during `shutdown`, power it on again, check that the system time is right with `date`, and collect the shutdown logs for analysis: + +```bash +journalctl --since "" > ~/shutdown-log.log +ls -lh ~/shutdown-log.log +``` diff --git a/docs/en/solutions/Storage_navigation_entries_added_by_plugins_or_Operators_cannot_be_hidden_with_custom-navconfig-platform_on_ACP.md b/docs/en/solutions/Storage_navigation_entries_added_by_plugins_or_Operators_cannot_be_hidden_with_custom-navconfig-platform_on_ACP.md new file mode 100644 index 000000000..f4d3a2768 --- /dev/null +++ b/docs/en/solutions/Storage_navigation_entries_added_by_plugins_or_Operators_cannot_be_hidden_with_custom-navconfig-platform_on_ACP.md @@ -0,0 +1,64 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.3.x +--- + +# Storage navigation entries added by plugins or Operators cannot be hidden with custom-navconfig-platform on ACP + +## Issue + +On ACP 4.3, an administrator wants to hide some entries under **Storage** in the left navigation of the **Administrator** view, such as **Distributed Storage**, **Local Storage** and **Object Storage**. Following "Customizing the Left Navigation" in the product documentation, `"hidden": true` is added for those entries in the `buildIn` data of the ConfigMap `custom-navconfig-platform` (namespace `cpaas-system`, `global` cluster), and the browser is refreshed. The entries are still shown. + +Hiding the whole `storage_management` group, or the documented built-in children such as `storageclasses`, works as expected. + +## Root Cause + +`custom-navconfig-platform` only customizes the **built-in** navigation. The built-in children of `storage_management` are `storageclasses` (StorageClasses), `nav_pv` (PersistentVolumes), `object_storage_class` (Object StorageClass) and `bucket` (Buckets). Keys that do not match a built-in item have no effect. + +The **Distributed Storage**, **Local Storage** and **Object Storage** entries are not built-in items: they are added to the navigation when the corresponding storage plugins or Operators are installed. There is no separate switch to hide them, so no key in `custom-navconfig-platform` removes them. + +## Resolution + +1. Decide whether the storage capability behind the entry is used. An entry contributed by a plugin or Operator disappears only when that plugin or Operator is uninstalled. + +2. If the capability is not needed, uninstall the corresponding plugin or Operator from **Administrator** > **Marketplace** (**Cluster Plugins** or **OperatorHub**, depending on how it was installed). Its navigation entry is removed with it. + + > **Warning:** Uninstalling a storage plugin or Operator removes the storage service it provides. Workloads that use volumes or buckets from it lose access to that storage, and data held only there can become unavailable. Uninstall it only after confirming that no PersistentVolumeClaim, StorageClass or bucket in any cluster still depends on it. Reinstalling the plugin or Operator restores the entry and the service, but not data that was removed with it. + +3. If the capability is in use, keep the plugin or Operator installed: its entry cannot be hidden. + +4. For the built-in items, hiding keeps working as documented. For example, to hide the PersistentVolumes entry: + + ```yaml + apiVersion: v1 + kind: ConfigMap + metadata: + name: custom-navconfig-platform + namespace: cpaas-system + data: + buildIn: | + { + "nav_pv": { + "hidden": true + } + } + ``` + + Refresh the browser after editing. + +## Diagnostic Steps + +1. Check what is currently configured. The ConfigMap must be in `cpaas-system` on the `global` cluster: + + ```bash + kubectl -n cpaas-system get configmap custom-navconfig-platform -o yaml + kubectl -n cpaas-system get configmap | grep navconfig + ``` + +2. Compare the keys under `buildIn` with the built-in storage items `storage_management`, `storageclasses`, `nav_pv`, `object_storage_class` and `bucket`. Keys such as `distributed_storage` or `local_storage` are not built-in names and are ignored. + +3. Check whether the entry belongs to an installed storage plugin or Operator: it appears after that component is installed and is absent on clusters where it is not installed. Such an entry cannot be hidden through `custom-navconfig-platform`. diff --git a/docs/en/solutions/TektonHub_cannot_be_viewed_or_the_pipeline_task_page_is_blank_after_a_Pipelines_upgrade_on_ACP.md b/docs/en/solutions/TektonHub_cannot_be_viewed_or_the_pipeline_task_page_is_blank_after_a_Pipelines_upgrade_on_ACP.md new file mode 100644 index 000000000..cd89db40c --- /dev/null +++ b/docs/en/solutions/TektonHub_cannot_be_viewed_or_the_pipeline_task_page_is_blank_after_a_Pipelines_upgrade_on_ACP.md @@ -0,0 +1,107 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "3.14" + - "3.16" + - "4.2" + - "4.3" +--- + +# TektonHub cannot be viewed, or the pipeline task page is blank after a Pipelines upgrade on ACP + +## Issue + +- TektonHub cannot be opened, and uninstalling the Tekton operator and installing it again fails. +- After a Pipelines upgrade the pipeline task page is blank, and `kubectl get tektonhub.operator.tekton.dev -A` returns `No resources found`, although `TektonConfig.spec.profile` is already `all`. + +## Root Cause + +### Cause 1: TektonHub cannot be viewed and the operator reinstall fails because the Triggers webhook Role is missing + +The Tekton Triggers webhook runs with its own ServiceAccount, which is granted its permissions by a `Role` and `RoleBinding`. When those two objects are missing (for example after an uninstall that removed them), the webhook cannot work, the operator install does not finish, and TektonHub stays unreachable. Restoring the `Role` and `RoleBinding` and restarting the webhook brings the operator install and TektonHub back. + +### Cause 2: pipeline task page blank after a Pipelines upgrade because AUTOINSTALL_TEKTONHUB is false + +The task page reads tasks from TektonHub. ConfigMap `tekton-config-defaults` in namespace `tekton-operator` has `AUTOINSTALL_TEKTONHUB` set to `"false"`. With that default the operator does not create the TektonHub custom resource, even with profile `all`, so no Hub exists and the page is blank. The value is injected into the operator's environment and is not reloaded while the Pod runs, so the operator Pod has to be recreated after the change; deleting the Pod is preferred over `rollout restart`. + +This applies to Tekton operator versions that still ship the built-in TektonHub. Check that the installed Pipelines version provides TektonHub before turning the flag on. + +## Resolution + +### Cause 1: TektonHub cannot be viewed and the operator reinstall fails because the Triggers webhook Role is missing + +1. Find the Triggers webhook and confirm that its `Role` and `RoleBinding` are absent: + + ```bash + kubectl get deploy,pods -A | grep -iE 'triggers-webhook|tekton-triggers' + kubectl get role,rolebinding -A | grep -i trigger + kubectl -n logs deploy/ --tail=80 + ``` + + A missing Role for the webhook ServiceAccount, or a `forbidden` error in that log, matches this cause. + +2. Export the same `Role` and `RoleBinding` from a healthy cluster that runs the same Tekton operator version, or take them from the Triggers release manifests of that version, and apply them here. + + > **Warning:** Applying a Role and RoleBinding grants the verbs in that file to the webhook ServiceAccount. A copy from a different operator version can be too wide or too narrow. Precondition: only when the Tekton operator reinstall fails after an uninstall and these Triggers webhook objects are missing. Keep the YAML you applied. Rollback: delete the Role and RoleBinding you added. + + ```bash + kubectl apply -f + kubectl apply -f + ``` + +3. Restart the webhook. + + > **Warning:** Restarting the Triggers webhook drops admission for trigger resources until the new Pod is Ready; creating or changing EventListeners and other trigger objects fails in that window. Precondition: only after the Role and RoleBinding from step 2 are in place. Rollback: none for the restart; delete the objects from step 2 if they were the wrong ones. + + ```bash + kubectl -n rollout restart deploy/ + kubectl -n rollout status deploy/ + ``` + +4. Retry the operator install. TektonHub should open, and the webhook Pod should stay Ready. + +### Cause 2: pipeline task page blank after a Pipelines upgrade because AUTOINSTALL_TEKTONHUB is false + +1. Confirm that no TektonHub exists, the profile is `all`, and the flag is `false`: + + ```bash + kubectl get tektonhub.operator.tekton.dev -A + kubectl get tektonconfig -o jsonpath='{range .items[*]}{.metadata.name}{" profile="}{.spec.profile}{"\n"}{end}' + kubectl -n tekton-operator get configmap tekton-config-defaults -o jsonpath='{.data.AUTOINSTALL_TEKTONHUB}{"\n"}' + ``` + +2. Save the ConfigMap, set the flag, and recreate the operator Pod. + + > **Warning:** Setting `AUTOINSTALL_TEKTONHUB` to `true` makes the operator create TektonHub (`tektonhub/hub`, with its Deployments in `tekton-pipelines`) and pull its images; deleting the operator Pod pauses Tekton component reconciliation until the new Pod is Ready. Precondition: only when no TektonHub exists, `TektonConfig.spec.profile` is `all`, and `AUTOINSTALL_TEKTONHUB` is `false`. Rollback: apply the saved ConfigMap and delete the operator Pod again; if `tektonhub/hub` remains, delete it with `kubectl delete tektonhub hub`, which removes the Hub UI. + + ```bash + kubectl -n tekton-operator get configmap tekton-config-defaults -o yaml > /tmp/tekton-config-defaults.yaml + kubectl -n tekton-operator patch configmap tekton-config-defaults --type merge -p '{"data":{"AUTOINSTALL_TEKTONHUB":"true"}}' + kubectl -n tekton-operator delete pod -l name=tekton-operator + ``` + +3. Wait for TektonHub and check its resources: + + ```bash + kubectl wait --for=condition=Ready tektonhub/hub --timeout=600s + kubectl get tektonhub.operator.tekton.dev -A + kubectl -n tekton-pipelines get svc,deploy,pod | grep tekton-hub + ``` + +4. Reload the pipeline task page. + +## Diagnostic Steps + +```bash +kubectl get tektonhub.operator.tekton.dev -A +kubectl get csv -A | grep -i tekton +kubectl get role,rolebinding -A | grep -i trigger +kubectl -n tekton-operator get configmap tekton-config-defaults -o jsonpath='{.data.AUTOINSTALL_TEKTONHUB}{"\n"}' +``` + +- The Tekton operator reinstall fails, and no Role or RoleBinding exists for the Triggers webhook: Cause 1. +- `No resources found` for TektonHub, profile `all`, and `AUTOINSTALL_TEKTONHUB` printed as `false`: Cause 2. +- A TektonHub object exists and its Pods are Running while the page is still blank: neither cause; check the Hub API Pod log before installing anything. diff --git a/docs/en/solutions/Tekton_pipeline_Maven_task_fails_with_java.lang.OutOfMemoryError_Java_heap_space_on_ACP.md b/docs/en/solutions/Tekton_pipeline_Maven_task_fails_with_java.lang.OutOfMemoryError_Java_heap_space_on_ACP.md new file mode 100644 index 000000000..445270353 --- /dev/null +++ b/docs/en/solutions/Tekton_pipeline_Maven_task_fails_with_java.lang.OutOfMemoryError_Java_heap_space_on_ACP.md @@ -0,0 +1,115 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - 4.4.x +--- + +# Tekton pipeline Maven task fails with java.lang.OutOfMemoryError Java heap space on ACP + +## Issue + +A Tekton pipeline fails in its Maven build task. The task log ends with: + +```text +[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin::compile (default-compile) on project : Fatal error compiling: java.lang.OutOfMemoryError: Java heap space -> [Help 1] +``` + +The question that follows is how to give the pipeline's Maven task more resources. The Pod of the task is not necessarily `OOMKilled`; the Java process inside it runs out of heap. + +The same error in a Jenkins pipeline is covered in "Jenkins Maven build fails with java.lang.OutOfMemoryError Java heap space although build node resources were raised". + +## Root Cause + +### Cause 1: the Maven JVM heap is too small for the build + +`java.lang.OutOfMemoryError: Java heap space` is raised by the JVM when the heap reaches its maximum size (`-Xmx`). The compiler runs inside the Maven JVM, so a large project can exhaust the heap during `compile` even though the container still has free memory. Raising only the container memory does not raise the heap if the JVM keeps the same `-Xmx`; the heap limit has to be raised, and the container limit has to leave room for it. + +Two places control this in a Tekton pipeline: + +- The `mavenOpts` parameter of the Maven task. Its value is appended after the task's default JVM options, so an `-Xmx` set there takes effect. The Maven step has a memory limit of `2Gi`, so the heap plus metaspace set through `mavenOpts` must fit inside `2Gi`. +- The `taskRunSpecs` of the PipelineRun. An entry for the Maven task can set larger compute resources for that task and add JVM options through an environment variable, for builds that need more than the default step limit allows. + +## Resolution + +### Cause 1 + +Use option A when a heap that fits in `2Gi` is enough. Use option B when the build needs more memory than that. + +**Option A: set the heap through the `mavenOpts` task parameter** + +1. Edit the pipeline and, on the Maven task, set the `mavenOpts` parameter, for example to a 1 GiB heap: + + ```yaml + params: + - name: mavenOpts + value: "-Xmx1g" + ``` + + Put JVM options only in `mavenOpts`. Do not add them to the `goals` list; entries in `goals` are passed to Maven as goals and arguments, not to the JVM. + +2. Keep the heap plus metaspace below the `2Gi` step limit, with room for the JVM's other native memory. TaskRuns that do not set `mavenOpts` keep the previous default behaviour, so only runs started after the change use the new heap. + +3. Run the pipeline again. + +**Option B: give the Maven task more resources with `taskRunSpecs`** + +1. Open the failed run and choose **Re-run Pipeline**, then add a `taskRunSpecs` entry to the PipelineRun YAML. Set `pipelineTaskName` to the name of the Maven task as it is written in your pipeline (`maven` in this example): + + ```yaml + spec: + taskRunSpecs: + - pipelineTaskName: maven + computeResources: + requests: + cpu: "4" + memory: 8Gi + limits: + cpu: "4" + memory: 16Gi + podTemplate: + env: + - name: JAVA_TOOL_OPTIONS + value: "-Xms4g -Xmx12g -XX:MaxMetaspaceSize=2g" + ``` + + The values are an example: keep `-Xmx` plus `MaxMetaspaceSize` below the memory limit, with headroom for non-heap memory. + + > **Warning:** The larger requests count against the namespace ResourceQuota and must fit on a node. If they cannot be satisfied, the task Pod stays `Pending` and the run does not start the Maven task. Apply this only after Diagnostic Steps confirmed `java.lang.OutOfMemoryError: Java heap space`. To roll back, re-run without the `taskRunSpecs` entry. + +2. To make the change permanent, add the same `taskRunSpecs` entry wherever the PipelineRun is created for this pipeline (for example its trigger template), instead of adding it on every re-run. + +3. Run the pipeline. The Maven task completes without `OutOfMemoryError`. + +## Diagnostic Steps + +1. Confirm the error type in the failed task's log. `java.lang.OutOfMemoryError: Java heap space` is this cause. `GC overhead limit exceeded` is also heap exhaustion and has the same fix. + +2. Check whether the task container was killed instead: + + ```bash + kubectl get pod -n -l tekton.dev/pipelineRun= \ + -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.containerStatuses[*].state.terminated.reason}{"\n"}{end}' + ``` + + `OOMKilled` on the Maven task Pod means the container limit is smaller than the heap plus non-heap memory: lower `-Xmx` or raise the limit (option B). `Error` together with the `Java heap space` message in the log is Cause 1 with a heap that is too small. + +3. Check the memory limits the Maven task Pod actually received, after a change: + + ```bash + kubectl get pod -n \ + -o jsonpath='{range .spec.containers[*]}{.name}{" limits="}{.resources.limits.memory}{" requests="}{.resources.requests.memory}{"\n"}{end}' + ``` + + With option B, the Maven step container shows the limit set in `taskRunSpecs`. If it still shows the old value, the `pipelineTaskName` in the entry does not match the task name in the pipeline. + +4. Check which JVM options reach Maven: + + ```bash + kubectl get pod -n \ + -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{range .env[*]}{" "}{.name}={.value}{"\n"}{end}{end}' + ``` + + With option B, `JAVA_TOOL_OPTIONS` is listed on the Maven step container, and the task log contains `Picked up JAVA_TOOL_OPTIONS: ...` from the JVM. When the same option appears more than once, the JVM uses the last one; options given on the command line, such as those from `mavenOpts`, come after `JAVA_TOOL_OPTIONS`. If the heap does not change with option B, set the same `-Xmx` through `mavenOpts` as well, keeping it below the container limit shown in step 3. diff --git a/docs/en/solutions/Tekton_pipeline_Pod_rejected_by_Pod_Security_admission_because_of_the_katanomi-setting-init_init_container.md b/docs/en/solutions/Tekton_pipeline_Pod_rejected_by_Pod_Security_admission_because_of_the_katanomi-setting-init_init_container.md new file mode 100644 index 000000000..6f2b6e259 --- /dev/null +++ b/docs/en/solutions/Tekton_pipeline_Pod_rejected_by_Pod_Security_admission_because_of_the_katanomi-setting-init_init_container.md @@ -0,0 +1,88 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - 4.3.x +--- + +# Tekton pipeline Pod rejected by Pod Security admission because of the katanomi-setting-init init container + +## Issue + +A Tekton pipeline (for example one using the official `git-clone` and `buildah` tasks) cannot run in a namespace that enforces the `restricted` Pod Security Standard. The admission event names the init container `katanomi-setting-init` as not meeting the namespace security policy, in the usual Pod Security admission form: + +```text +pods "" is forbidden: violates PodSecurity "restricted:latest": ... (container "katanomi-setting-init" ...) +``` + +Setting `set-security-context: "true"` under `spec.pipeline` of the `TektonConfig` does not change the result; the same event is reported again. + +## Root Cause + +### Cause 1: the DevOps v3 components inject `katanomi-setting-init` into pipeline Pods + +`katanomi-setting-init` is not part of the Tekton task. It is an init container injected by the DevOps v3 (katanomi) components while they are installed on the cluster: + +- **v3 pipelines** do not support running in a `restricted` namespace. +- **v4 (Tekton) pipelines** also get the `katanomi-setting-init` init container injected automatically as long as the v3 components are still installed. The container does not satisfy the `restricted` policy, so the whole Pod is rejected. + +`set-security-context` in the `TektonConfig` applies to the containers Tekton creates. It does not remove or change the injected `katanomi-setting-init` container, which is why changing it has no effect. + +## Resolution + +### Cause 1 + +Choose the option that matches how the cluster still uses DevOps v3. + +**Option A: v3 is no longer needed.** Uninstall the DevOps v3 components. After that, v4 pipeline Pods no longer carry `katanomi-setting-init` and can run in a `restricted` namespace. + +> **Warning:** Uninstalling the DevOps v3 components removes the v3 pipeline functions, including Jenkins-based pipelines that still run on v3. Do this only after every pipeline that still depends on v3 has been migrated to v4 and verified. Reinstalling the v3 components is the only way back. + +**Option B: v3 and v4 must run in parallel for a while** (for example while Jenkins pipelines are still being migrated). Keep the v3 components and run the v4 pipelines in a namespace whose Pod Security level the pipeline Pods satisfy, and plan the move to a `restricted` namespace after the migration to v4 is finished. + +1. Check what a lower level would allow before changing it. The server-side dry run prints a warning for every existing Pod that would still be rejected: + + ```bash + kubectl label --dry-run=server --overwrite ns pod-security.kubernetes.io/enforce=baseline + ``` + +2. Apply the level that the pipeline Pods pass: + + > **Warning:** Lowering the enforced Pod Security level allows more privileged Pods in the namespace for every workload there, not only the pipelines. Prefer a dedicated namespace for pipelines. Record the current label first and roll back by setting it again. + + ```bash + kubectl get ns --show-labels + kubectl label --overwrite ns pod-security.kubernetes.io/enforce= + ``` + +3. Run the pipeline again and confirm that the Pod is admitted. + +## Diagnostic Steps + +1. Read the event of the rejected Pod or TaskRun and check which container is named: + + ```bash + kubectl -n get events --sort-by=.lastTimestamp | grep -i podsecurity + kubectl -n describe taskrun | sed -n '/Events/,$p' + ``` + + Expected for this cause: the violation lists `katanomi-setting-init`. If it lists only Tekton containers such as `prepare`, `place-scripts` or `step-*`, the problem is the securityContext of the Tekton containers, not the v3 injection. + +2. Check the namespace Pod Security labels: + + ```bash + kubectl get ns -o jsonpath='{.metadata.labels}{"\n"}' + ``` + + Expected: `pod-security.kubernetes.io/enforce: restricted`. + +3. Check whether the DevOps v3 (katanomi) components are still installed: + + ```bash + kubectl get katanomis.operators.katanomi.dev -A + kubectl get pods -A | grep -i katanomi + ``` + + Resources and Pods present mean the v3 components are installed and will keep injecting `katanomi-setting-init`. diff --git a/docs/en/solutions/Toolchain_SSO_on_a_domain_name_after_an_IP-based_install_and_kube-apiserver_OIDC_for_an_external_UI_on_ACP.md b/docs/en/solutions/Toolchain_SSO_on_a_domain_name_after_an_IP-based_install_and_kube-apiserver_OIDC_for_an_external_UI_on_ACP.md new file mode 100644 index 000000000..1b4ae3f1e --- /dev/null +++ b/docs/en/solutions/Toolchain_SSO_on_a_domain_name_after_an_IP-based_install_and_kube-apiserver_OIDC_for_an_external_UI_on_ACP.md @@ -0,0 +1,89 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.1" + - "4.2" + - "4.3" + - "4.3.2" +--- + +# Toolchain SSO on a domain name after an IP-based install, and kube-apiserver OIDC for an external UI on ACP + +## Issue + +One of these is reported: + +- The platform was installed and is opened by IP address, and toolchain SSO (Harbor, GitLab, and similar) should use a domain name instead. Adding the DNS name alone does not switch SSO. +- Headlamp, or a similar cluster UI, needs kube-apiserver OIDC flags, and the concern is that the API would then accept only OIDC, which would break the ACP console, in-cluster components, and kubectl, and that an upgrade would remove the flags. + +Related articles: "OIDC clients and pages are sent to the wrong Dex issuer address on ACP", "Harbor SSO through the platform Dex fails: wrong issuer, OAuth2Client name, or Unregistered redirect_uri console-portal", and "GitLab or Argo CD login through the platform Dex fails: host required, certificate verify, invalid_client, LDAP groups". + +## Root Cause + +### Cause 1: a domain for toolchain SSO needs an access address, a certificate, and updated client callbacks + +Publishing a DNS name does not change anything on the platform by itself. For toolchain SSO to work on the domain, the domain has to be added as a platform access address, a certificate for that name has to be installed, and each toolchain OAuth2 client callback has to use the domain. A reinstall is normally not needed for this. These steps do not change the default access URL, and they do not change the issuer that Dex publishes. On ACP 4.3.2 the default platform access address and the internal registry address cannot be edited after installation; only additional access addresses can be added. Redeploying with the domain as the default is the only way to change the default itself, and it is not realistic once user clusters already run workloads. + +### Cause 2: kube-apiserver OIDC flags add OIDC authentication without turning off ServiceAccount or certificate authentication + +Adding the OIDC flags to the kube-apiserver static Pod manifest on every control-plane node enables OIDC login for the external UI. ServiceAccount and client-certificate authentication keep working, so the ACP console, in-cluster components, and kubectl are not affected. A platform upgrade does not revert this manifest edit, and the standard product supports it. Passing the same parameters through installer `extraArgs` has not been shown to work; edit the manifest instead. + +This is not how a third-party application obtains a token for the ACP API proxy: the platform API proxy accepts tokens from the `alauda-auth` client only. + +## Resolution + +### Cause 1: a domain for toolchain SSO needs an access address, a certificate, and updated client callbacks + +1. Publish DNS for the platform name. +2. Add the name as an additional platform access address. +3. Install a platform access certificate that contains the DNS name. +4. Edit each toolchain OAuth2 client and set its callback (redirect) URLs to `https:///...`: + + ```bash + kubectl get oauth2clients.dex.coreos.com -n cpaas-system -o yaml > /tmp/oauth2clients.bak.yaml + kubectl edit oauth2clients.dex.coreos.com -n cpaas-system + ``` + + > **Warning:** Changing redirect URLs breaks callbacks that still use the old string. Precondition: only after the domain resolves and serves a certificate that contains the name. Rollback: `kubectl apply -f /tmp/oauth2clients.bak.yaml`. + +If an OIDC client then reports that the issuer in the provider metadata does not match the issuer it was configured with, the discovery document still advertises the install-time IP; see "OIDC clients and pages are sent to the wrong Dex issuer address on ACP". + +### Cause 2: kube-apiserver OIDC flags add OIDC authentication without turning off ServiceAccount or certificate authentication + +Apply this only when an external UI must talk to kube-apiserver with OIDC, and a maintenance window covers a rolling kube-apiserver restart. + +1. On one control-plane node, back up the static Pod manifest and add the OIDC arguments, keeping every existing argument: + + ```yaml + - --authorization-mode=Node,RBAC + - --oidc-issuer-url=https:// + - --oidc-username-claim=email + - --oidc-client-id= + ``` + + > **Warning:** kubelet restarts kube-apiserver when the manifest is saved; API calls on that node fail until the Pod is Ready. Precondition: only when the issuer and client id are the identity provider the UI should trust, and the other control-plane nodes are Ready. Rollback: copy `/tmp/kube-apiserver.yaml.bak` back to `/etc/kubernetes/manifests/kube-apiserver.yaml`. An upgrade does not revert this file, so removing the flags is a manual restore from the backup. + + ```bash + sudo cp -a /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml.bak + sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml + kubectl get nodes + ``` + +2. Wait until the kube-apiserver Pod on that node is Ready, then repeat on the next control-plane node, one node at a time. + +3. After every control plane is Ready, confirm that an in-cluster ServiceAccount call and a kubectl call still succeed, then test the OIDC UI. + +## Diagnostic Steps + +```bash +curl -k -sS https:///dex/.well-known/openid-configuration | jq -r .issuer +openssl s_client -connect :443 -servername /dev/null | openssl x509 -noout -subject -ext subjectAltName +kubectl get oauth2clients.dex.coreos.com -n cpaas-system -o yaml | grep -n -i redirect +kubectl -n kube-system get pods -o wide | grep kube-apiserver +``` + +- The toolchains should use a new domain, the platform was installed on an IP, and the client callbacks or the certificate do not contain the domain: Cause 1. +- An external UI needs kube-apiserver OIDC, and the console must keep working: Cause 2. After the change, the kube-apiserver Pods are Running on every control-plane node and both kubectl and the OIDC UI work. diff --git a/docs/en/solutions/TopoLVM_PVC_expansion_stays_Resizing_spans_two_disks_or_pods_stay_on_the_old_node_on_ACP.md b/docs/en/solutions/TopoLVM_PVC_expansion_stays_Resizing_spans_two_disks_or_pods_stay_on_the_old_node_on_ACP.md new file mode 100644 index 000000000..64fcee397 --- /dev/null +++ b/docs/en/solutions/TopoLVM_PVC_expansion_stays_Resizing_spans_two_disks_or_pods_stay_on_the_old_node_on_ACP.md @@ -0,0 +1,133 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# TopoLVM PVC expansion stays Resizing, spans two disks, or pods stay on the old node on ACP + +## Issue + +One of the following is seen with TopoLVM (LVM storage, CSI driver `topolvm.cybozu.com`, operands in `nativestor-system`) on Alauda Container Platform: + +- After a PVC was expanded (for example from 10Gi to 40Gi), `lvs -o devices` shows its logical volume on two physical disks, and it looks as if the volume were split. +- A TopoLVM volume has to be expanded, or a PVC expansion stays in `Resizing` / expanding and never finishes. This often happens when several PVCs (for example the brokers of a three-replica Kafka) are given all remaining capacity of the device class at once. +- New TopoLVM nodes were added and have free capacity, but existing pods stay on the old nodes. After the old storage node was deleted, the pod stays `Pending` with an event such as `0/15 nodes are available: 3 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 4 node(s) didn't match Pod's node affinity/selector, 8 node(s) had volume node affinity conflict`. +- The expansion entry of a TopoLVM PVC looks grayed out in the console while editing the YAML works, or a second disk was added to a node and the existing PVC still cannot grow into it. + +For rejected access modes, mount errors, and new claims that cannot be scheduled, see [TopoLVM PVC rejects an access mode, fails to mount, or cannot be scheduled on ACP](./TopoLVM_PVC_rejects_an_access_mode_fails_to_mount_or_cannot_be_scheduled_on_ACP.md). + +## Root Cause + +### Cause 1: an expanded LV shows extents on two physical disks + +This is ordinary LVM. When the volume group was built from two physical volumes, `lvextend` takes free extents from either disk. The filesystem stays one filesystem. The pod does not have to move, and the extents do not have to be migrated onto one disk. + +### Cause 2: a PVC has to be expanded, or the expansion stays in progress + +Two different operations are called "expansion": + +- Adding capacity to the device class (the pool): **Add Storage Device** on the Local Storage page, with a whole bare disk. +- Growing one claim: editing `spec.resources.requests.storage` of the PVC. The application manifest does not have to change. In a multi-replica application, every replica PVC is a separate volume and is expanded on its own. + +An expansion stalls in two known situations: + +- The new disk added to the pool still carries old signatures, so the pool did not actually grow. +- Several PVCs are enlarged in one step to consume all remaining published capacity. The volume group cannot satisfy every request plus LVM metadata, so one or more PVCs stay in `Resizing`. TopoLVM on ACP does not support a `spare-gb` setting to keep free extents in reserve. + +### Cause 3: pods remain on the old node after new TopoLVM nodes are added + +A TopoLVM PV has node affinity `topology.topolvm.cybozu.com/node` set to the node that holds its logical volume. The scheduler will not place the pod on another node, and TopoLVM does not copy logical volumes between nodes. Replacing a server and reusing its IP address does not move the old volume group onto the new machine. When the old node object has been deleted, the pod event shows `volume node affinity conflict`, because the volume still points at the deleted node. + +### Cause 4: the expansion entry looks grayed out, or a second disk does not add capacity to the existing PVC + +TopoLVM PVCs support expansion, from **Actions > Expand** on the PVC detail page or by editing the PVC YAML. A PVC is one logical volume in one volume group, so it can only grow by the free space of that volume group. + +The platform supports several disks on one node. A disk added with **Storage Management > Local Storage > topolvm > Storage Devices > Add Storage Device** (`添加存储设备`, "Add Storage Device"; it asks for a node, a device class, and a storage device) can become a separate volume group instead of joining the volume group of the existing disk. In that case it serves new PVCs only and does not add capacity to existing ones. + +## Resolution + +### Cause 1: an expanded LV shows extents on two physical disks + +No repair is needed. Confirm that the volume group contains both disks and leave the logical volume where LVM placed it: + +```bash +vgs --units g +pvs --units g +lvs -o lv_name,vg_name,lv_size,devices --units g +``` + +Do not run `pvmove` to make the layout look tidier. It is a long exclusive operation on the volume group and is not required. + +### Cause 2: a PVC has to be expanded, or the expansion stays in progress + +1. To grow the pool, add a whole bare disk to the device class from **Storage Management > Local Storage**. This does not resize existing PVCs. +2. To grow a claim, edit its request. TopoLVM expands online; the pod does not need a restart. + +> **Warning:** A larger PVC request allocates extents immediately and cannot be shrunk through Kubernetes. Precondition: `vgs` free space on the node that owns the volume is larger than the increase, and you are not giving the last free extents to several PVCs at once. Rollback: none for the allocated extents; setting the old request back does not shrink the logical volume. + +```bash +kubectl patch pvc -n --type=merge \ + -p '{"spec":{"resources":{"requests":{"storage":""}}}}' +kubectl get pvc -n -o jsonpath='{.status.capacity.storage}{"\n"}{.status.conditions}{"\n"}' +``` + +3. Repeat for each replica PVC. +4. If the condition stays `Resizing`: + - If the sum of the new sizes exceeds `vg_free` on the node, lower the requests, or add capacity to the device class first and then patch again. Expanding the device class first and re-running the resize is what clears the stall when several PVCs were given all remaining capacity. + - If a disk was just added and the pool did not grow, clear the leftover signatures on that unused disk before you patch again. + +Do not delete a PVC that still holds data to "unstick" an expansion. + +### Cause 3: pods remain on the old node after new TopoLVM nodes are added + +A pod cannot be rescheduled to a new node while it uses the old PV. To move it: + +1. Copy the data at application level (dump and restore, or a replica rebuild) onto a new PVC provisioned on the new node. +2. Switch the workload to the new PVC. + +The StorageClass option **PVC Reconstruction** rebuilds a claim on another node after a wait, and the rebuilt claim does not contain the original data. Do not use it as a migration tool. + +A Velero backup of the claim is only useful if the copy is application-consistent. A single-node TopoLVM volume is not a portable disk, and re-adding a replacement server under the old IP does not attach the old volume group. + +### Cause 4: the expansion entry looks grayed out, or a second disk does not add capacity to the existing PVC + +1. Expand the PVC from **Actions > Expand** on its detail page, or edit `spec.resources.requests.storage` as in Cause 2. The same Warning applies. +2. Check which volume group holds the PVC and whether the new disk joined it: + +```bash +kubectl get pv -o jsonpath='{.spec.csi.volumeHandle}{"\n"}' +lvs -o lv_name,vg_name,lv_size,devices | grep +pvs -o pv_name,vg_name,pv_size,pv_free +vgs +``` + +If `pvs` shows the new disk in a different volume group, it only serves new PVCs of the device class that volume group belongs to. + +To give existing PVCs more room on a node whose TopoLVM disk is a virtual disk or a LUN, grow that disk in place and run `pvresize` on it. This enlarges the volume group that already holds the PVCs; see [TopoLVM device class capacity does not change after the underlying disk is expanded on ACP](./TopoLVM_device_class_capacity_does_not_change_after_the_underlying_disk_is_expanded_on_ACP.md). Adding another disk helps only new PVCs. In both cases, each PVC is then expanded on its own as in Cause 2. + +## Diagnostic Steps + +```bash +kubectl get pvc -n -o yaml +kubectl describe pvc -n +kubectl get pv -o jsonpath='{.spec.csi.driver}{"\n"}{.spec.nodeAffinity}{"\n"}' +kubectl get pod -n -o jsonpath='{.spec.nodeName}{"\n"}{.status.phase}{"\n"}' +``` + +- `lvs -o devices` lists two physical disks of one volume group for the expanded LV: Cause 1. +- `FileSystemResizePending`, a long `Resizing` condition, or several PVCs stuck expanding after they were given all remaining capacity: Cause 2. Compare the requested increase with `vgs` free space on the node. +- The pod is `Pending` with `volume node affinity conflict`, or the PV node affinity names a node that no longer exists or is not the new node: Cause 3. +- A grayed-out expansion entry, or a second disk that `pvs` shows as a separate volume group: Cause 4. +- `did not have enough free storage` on a new claim after a disk was moved to a new device class is a scheduling problem; see the access-mode and scheduling article linked in the Issue. + +## Related Information + +- [TopoLVM capacity stays full after PVCs are deleted on ACP](./TopoLVM_capacity_stays_full_after_PVCs_are_deleted_on_ACP.md) +- [TopoLVM thin provisioning, over-provisioned capacity, and pod IOPS limits on ACP](./TopoLVM_thin_provisioning_over-provisioned_capacity_and_pod_IOPS_limits_on_ACP.md) diff --git a/docs/en/solutions/TopoLVM_PVC_rejects_an_access_mode_fails_to_mount_or_cannot_be_scheduled_on_ACP.md b/docs/en/solutions/TopoLVM_PVC_rejects_an_access_mode_fails_to_mount_or_cannot_be_scheduled_on_ACP.md new file mode 100644 index 000000000..189856de1 --- /dev/null +++ b/docs/en/solutions/TopoLVM_PVC_rejects_an_access_mode_fails_to_mount_or_cannot_be_scheduled_on_ACP.md @@ -0,0 +1,172 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# TopoLVM PVC rejects an access mode, fails to mount, or cannot be scheduled on ACP + +## Issue + +A PVC on a TopoLVM StorageClass, or the pod that uses it, does not come up on Alauda Container Platform: + +- The PVC stays `Pending` with `ProvisioningFailed` and `rpc error: code = InvalidArgument desc = unsupported access mode: MULTI_NODE_MULTI_WRITER`. +- The pod reports `FailedMount` / `MountVolume.SetUp failed` with `target device is already formatted with different filesystem: volume=, current=jmicron_raid_member, new=xfs`. +- Creating a pod fails with a scheduling error, and some nodes are already at their pod limit. +- Pods stay `Pending` with `0/N nodes are available: ... did not have enough free storage`, for example after a disk was removed from TopoLVM and added to a new device class, or the node's available TopoLVM space shows 0 although its device class is Ready. + +Background: TopoLVM is node-local LVM, and the provisioner is `topolvm.cybozu.com`. The stock StorageClass on current clusters is `topolvm-hdd` (`csi.storage.k8s.io/fstype=xfs`, `topolvm.cybozu.com/device-class=hdd`); other clusters use another name such as `sc-topolvm`. `CSIDriver/topolvm.cybozu.com` uses `fsGroupPolicy: ReadWriteOnceWithFSType` and `attachRequired: false`. The stock class uses `volumeBindingMode: WaitForFirstConsumer`, so a PVC with no consuming pod stays `Pending` without a `ProvisioningFailed` event; that wait is not a storage failure. + +## Root Cause + +### Cause 1: unsupported access mode: MULTI_NODE_MULTI_WRITER + +`spec.accessModes` is `ReadWriteMany` or `ReadOnlyMany`. A TopoLVM logical volume exists on one node only, and there is no shared backend behind the StorageClass, so the driver rejects every multi-node mode. Adding the annotation `cpaas.io/access-mode` with `ReadWriteOnce,ReadWriteMany` to the StorageClass does not change this: the annotation is saved and the driver still rejects the request. See also [TopoLVM rejects ReadWriteMany PVCs on Alauda Container Platform](./TopoLVM_rejects_ReadWriteMany_PVCs_on_Alauda_Container_Platform.md). + +### Cause 2: target device is already formatted with different filesystem + +The logical volume handed to kubelet still carries a foreign signature (for example leftover RAID metadata, reported as `current=jmicron_raid_member`) next to the real filesystem, so the node plugin refuses to mount or format it. The decisive sign is that two tools disagree about the same LV: `blkid -c /dev/null -o export ` prints `TYPE=jmicron_raid_member`, while `lsblk -f` shows the same LV as `xfs` with a filesystem UUID. The xfs filesystem is intact; the `current=` value is the stale signature. This can show up when a pod is rebuilt (for example after an image change) and remounts an existing volume whose data is still needed. + +### Cause 3: some nodes are at the pod limit and the other nodes have almost no TopoLVM space + +The scheduler cannot place the new pod anywhere: the nodes with free TopoLVM capacity have already reached their maximum schedulable pod count (the standard scheduler message for this is `Too many pods`), and the nodes that still have pod slots have almost no free space in the TopoLVM device class. + +### Cause 4: did not have enough free storage because no device class on the node has default: true + +A StorageClass without the parameter `topolvm.cybozu.com/device-class` is served by the device class marked `default: true` on each node. If the node's class entry in `TopolvmCluster` (namespace `nativestor-system`) does not have `default: true`, TopoLVM finds no default device class on that node and treats its capacity as unavailable. The node then shows 0 available TopoLVM space and the scheduler reports `did not have enough free storage`, even though the cluster status looks healthy (`phase: Ready`, `failClasses: []`, the class in `successClasses` with `state: Ready` and `message: create successful`). + +This happens, for example, after a disk is removed from TopoLVM and added to a new device class on a node, and the pods use the default StorageClass. The event often also lists `volume node affinity conflict`, a node affinity or selector miss, and the master taint; those lines alone are not this cause. + +## Resolution + +### Cause 1: unsupported access mode: MULTI_NODE_MULTI_WRITER + +Recreate the PVC with `ReadWriteOnce` or `ReadWriteOncePod`. The access mode of an existing claim cannot be changed into one the driver accepts; create a new claim. Use the StorageClass name this cluster actually has: + +```yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: + namespace: +spec: + accessModes: + - ReadWriteOnce + storageClassName: topolvm-hdd + resources: + requests: + storage: 10Gi +``` + +`ReadWriteOnce` means one node, not one pod: two pods on the same node can mount the volume. A volume that must be mounted read-write from several nodes needs a shared storage class such as Ceph distributed storage. Editing `cpaas.io/access-mode` does not provide ReadWriteMany on TopoLVM. + +### Cause 2: target device is already formatted with different filesystem + +Use this only when the event contains `target device is already formatted with different filesystem`, `current=` is a foreign type (for example `jmicron_raid_member`), and `new=` is the filesystem the StorageClass asks for (for example `xfs`). + +1. Find the logical volume. In the PV, `spec.csi.volumeHandle` is the LV name and `nodeAffinity` names the node: + +```bash +kubectl get pv -o yaml +``` + +2. On that node, compare the views and list the signature offsets. The LV path is `/dev//`; in `lsblk` the LV appears as `-` with each hyphen inside the names doubled. + +```bash +lvs -o lv_name,lv_path,vg_name,lv_size | grep +blkid -c /dev/null -o export +lsblk -f | grep +wipefs -n +``` + +`blkid -c /dev/null` bypasses the blkid cache. `wipefs -n` lists every signature with its byte offset. Note the offset of the line whose type matches `current=` only. The `xfs` line is the filesystem the pod must mount. + +> **Warning:** `wipefs -o ` erases the signature at that byte offset on ``. A wrong offset can destroy the filesystem the application needs. This is not `wipefs -a` and not a format. Precondition: the FailedMount text matches this cause, the offset belongs to the foreign type and not to `xfs`, the path is the LV of this PV, and a backup exists if the data cannot be rebuilt. Rollback: none for the erased bytes. + +```bash +wipefs -o +``` + +After the foreign signature is cleared, the volume mounts and the pod recovers. If `wipefs -n` shows only the filesystem named by `new=`, there is nothing to clear; do not run `wipefs -o`. + +### Cause 3: some nodes are at the pod limit and the other nodes have almost no TopoLVM space + +1. Confirm both conditions before changing anything: + +```bash +kubectl get node -o jsonpath='capacity={.status.capacity.pods} allocatable={.status.allocatable.pods}{"\n"}' +kubectl get pods -A --field-selector spec.nodeName= --no-headers | wc -l +kubectl describe pod -n | grep -A 20 Events +``` + +On the nodes that still have pod slots, check TopoLVM free space with `vgs`. + +2. Raise the maximum schedulable pod count (kubelet `maxPods`) on the nodes that reached the limit, using the node maintenance procedure your platform version documents. Raising it on the full nodes is what lets the pods schedule. +3. Confirm that allocatable pods increased and the `Pending` pods schedule: + +```bash +kubectl get node -o jsonpath='capacity={.status.capacity.pods} allocatable={.status.allocatable.pods}{"\n"}' +``` + +If the only nodes with free pod slots have a full TopoLVM device class, a new TopoLVM volume still cannot be placed there; that needs more device-class capacity. + +### Cause 4: did not have enough free storage because no device class on the node has default: true + +Use this only when the event includes `did not have enough free storage` (or the node shows 0 available TopoLVM space while its class is Ready), the node's class entry has no `default: true`, and the StorageClass in use names no device class (or is the default StorageClass after a disk move). + +1. Check the StorageClass. An empty second line means it names no device class and relies on the default one: + +```bash +kubectl get sc -o jsonpath='{.metadata.annotations.storageclass\.kubernetes\.io/is-default-class}{"\n"}{.parameters.topolvm\.cybozu\.com/device-class}{"\n"}' +``` + +2. Save the cluster object: + +```bash +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm -o yaml > /tmp/topolvmcluster.before.yaml +``` + +> **Warning:** Editing `TopolvmCluster` changes which nodes the default device class schedules onto; a wrong `nodeName` marks a different node. Precondition: this cause is confirmed, the YAML is saved, and you change `default` only on the affected node's entry for the intended class. Rollback: set `default` back to `false` (or remove the field if it was absent) and save. + +3. Edit the cluster object and, under `spec.storage.deviceClasses`, set `default: true` on the affected node's class. Keep the existing `devices` list: + +```bash +kubectl -n nativestor-system edit topolvmclusters.topolvm.cybozu.com topolvm +``` + +```yaml +- nodeName: + classes: + - className: + default: true +``` + +4. Check that the node's available space is no longer 0 and that the `Pending` PVCs bind and the pods start. + +If that class must not become the default, create a new StorageClass whose `parameters` set `topolvm.cybozu.com/device-class: `, and create new PVCs on it. Kubernetes does not allow the `parameters` of an existing StorageClass to be changed. + +## Diagnostic Steps + +When the only report is that a pod cannot be created, start with the scheduler event and the pod count per node: + +```bash +kubectl describe pod -n | grep -A 20 Events +kubectl get pods -A --field-selector=status.phase!=Succeeded,status.phase!=Failed -o custom-columns=NODE:.spec.nodeName --no-headers | sort | uniq -c +kubectl get nodes -o custom-columns=NAME:.metadata.name,PODS:.status.allocatable.pods +kubectl get pvc -n -o yaml +kubectl get sc -o yaml +kubectl describe csidriver topolvm.cybozu.com +``` + +- `ProvisioningFailed` with `MULTI_NODE_MULTI_WRITER`: Cause 1. +- A `Pending` PVC with `WaitForFirstConsumer` and no pod: waiting for a consumer, not a failure. +- `FailedMount` with `already formatted with different filesystem`: Cause 2. +- Some nodes have as many pods as allocatable pods while the others have little TopoLVM free space: Cause 3. +- `did not have enough free storage`, a StorageClass with no device-class parameter, and no `default: true` on the node's class: Cause 4. +- A pod already bound to a PV on an old or deleted node (`volume node affinity conflict`): see [TopoLVM PVC expansion stays Resizing, spans two disks, or pods stay on the old node on ACP](./TopoLVM_PVC_expansion_stays_Resizing_spans_two_disks_or_pods_stay_on_the_old_node_on_ACP.md), Cause 3. diff --git a/docs/en/solutions/TopoLVM_capacity_stays_full_after_PVCs_are_deleted_on_ACP.md b/docs/en/solutions/TopoLVM_capacity_stays_full_after_PVCs_are_deleted_on_ACP.md new file mode 100644 index 000000000..f6eb51023 --- /dev/null +++ b/docs/en/solutions/TopoLVM_capacity_stays_full_after_PVCs_are_deleted_on_ACP.md @@ -0,0 +1,206 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.16.x + - 3.18.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# TopoLVM capacity stays full after PVCs are deleted on ACP + +## Issue + +The Local Storage page, or a node capacity alert, still shows a TopoLVM device class as full on Alauda Container Platform: + +- PVCs and PVs were deleted, the PVs show as pending reclaim, and `vgs` free space did not come back. Sometimes the logical volume was deleted on the host with `lvremove` and the page still did not change. +- The PVCs and PVs are gone, a search by their UUID finds no PVC, and the space still shows as used. +- One node sits at the alert threshold while another TopoLVM node on the same cluster is mostly free. +- The TopoLVM capacity alert stays firing after a NAS was expanded and `df` on the NAS mount already dropped. +- The Local Storage page shows `0/0` capacity while a test pod mounts a TopoLVM volume normally. +- A workload deployment (for example GitLab) stays on Deploying and never finishes. + +How TopoLVM counts capacity: published capacity is the volume group size minus the logical volumes the driver still accounts for. It is not `df` inside a pod. A 500Gi PVC consumes 500Gi of schedulable capacity even when its filesystem is almost empty, and deleting files inside the container does not shrink the logical volume. The storage-node usage percentage is the sum of the PVC requested sizes. The StorageClass reclaim policy decides whether deleting a PVC also deletes its logical volume. + +This describes a thick device class. A thin device class (`thinPoolConfig` in `TopolvmCluster`) publishes over-provisioned capacity, and its real limit is the thin pool's data usage on the node; see [TopoLVM thin provisioning, over-provisioned capacity, and pod IOPS limits on ACP](./TopoLVM_thin_provisioning_over-provisioned_capacity_and_pod_IOPS_limits_on_ACP.md). + +## Root Cause + +### Cause 1: Bound TopoLVM PVs still reserve the capacity + +`kubectl get pv` still shows `Bound` volumes of driver `topolvm.cybozu.com` on the node that looks full, and their pods or StatefulSets are running. The page is correct: those volumes are the workload. + +A node at the alert threshold while another TopoLVM node is much lower is the same accounting, split unevenly across nodes. If the sum of PVC requests on the alerting node matches the reported usage, the usage is expected. TopoLVM is local storage, so restarting a pod does not move it to an emptier node. + +### Cause 2: PV Released with reclaim policy Retain, LogicalVolume still present + +The PVC is gone, the PV phase is `Released`, and `persistentVolumeReclaimPolicy` is `Retain`. With Retain, Kubernetes does not call CSI `DeleteVolume` when the PVC is removed, so a `LogicalVolume` object with the same name as the PV still exists and `lvs` still shows the LV. The console lists such PVs as pending reclaim. A StorageClass created with reclaim policy Retain leaves one of these behind for every deleted PVC. Running `lvremove` on the host without deleting the API objects does not update the page. + +Example from `kubectl get pv`: + +```text +NAME CAPACITY RECLAIM POLICY STATUS CLAIM STORAGECLASS +pvc- 30Gi Retain Released / sc-topolvm +pvc- 5Gi Retain Released / sc-topolvm +``` + +### Cause 3: spare-gb is not a supported parameter + +TopoLVM on ACP (confirmed on ACP 4.3 with TopoLVM 4.3.6) does not accept a `spare-gb` parameter to make the scheduler keep free extents. There is no values key that turns it on. + +### Cause 4: the capacity alert stays firing after NAS usage drops + +The TopoLVM capacity alert reports space allocated by the TopoLVM device class. It does not measure the bytes used inside a PVC or the utilization of a NAS export. Expanding a NAS changes the NAS `df` number and not the device-class allocation the alert uses. + +### Cause 5: Local Storage capacity shows 0/0 because Prometheus is not installed + +The TopoLVM pods are Running and a test claim mounts, but the Local Storage page shows `0/0`. Prometheus is not installed on the platform. Once Prometheus is installed, the capacity figures are displayed; storage itself was never affected. + +### Cause 6: a deployment stays on Deploying because one component's storage request is larger than TopoLVM can allocate + +One component asks for more storage than the free capacity of the device class (for example the Gitaly component of GitLab), so that volume is never created and the deployment never finishes. + +### Cause 7: LogicalVolume objects remain with no matching PV + +The PVC and PV were deleted, but `kubectl get logicalvolumes -o wide` still lists `LogicalVolume` objects whose names match no PV. Read with `-o yaml`, such an object shows `apiVersion: topolvm.cybozu.com/v1`, finalizer `topolvm.cybozu.com/logicalvolume`, `size` and `currentSize`, a `volumeID`, and the `nodeName`. On that node `lvs | grep ` still shows the LV (attributes `-wi-a-----`), while `lsblk | grep ` shows no mount point. The orphaned objects keep the LVs allocated. Why they were not removed with the PVs is not known. + +## Resolution + +### Cause 1: Bound TopoLVM PVs still reserve the capacity + +Do not delete Bound PVs to change the graph. Either add another bare disk to the device class, or delete the workload that owns the claim; the claim then follows the reclaim policy. With `Delete`, removing the PVC removes the PV and the LogicalVolume, and the space returns. With `Retain`, removing the PVC turns the volume into Cause 2. + +List the TopoLVM volumes and their owners before deciding: + +```bash +kubectl get pv -o json | jq -r ' + .items[] + | select(.spec.csi.driver=="topolvm.cybozu.com") + | [.metadata.name, .status.phase, .spec.persistentVolumeReclaimPolicy, .spec.capacity.storage, + ((.spec.claimRef.namespace // "") + "/" + (.spec.claimRef.name // "")), + (.spec.nodeAffinity.required.nodeSelectorTerms[]?.matchExpressions[]? + | select(.key=="topology.topolvm.cybozu.com/node") | .values[0])] + | @tsv' +kubectl get pod -A -o json | jq -r --arg NS "" --arg PVC "" ' + .items[] + | select(.metadata.namespace==$NS) + | select(.spec.volumes[]? | .persistentVolumeClaim.claimName==$PVC) + | [.metadata.namespace, .metadata.name, .spec.nodeName, .status.phase] + | @tsv' +``` + +On the node, compare with LVM. A large LV with no Bound PV is Cause 2 or Cause 7: + +```bash +vgs --units g -o vg_name,pv_count,lv_count,vg_size,vg_free +lvs -a --units g -o lv_name,vg_name,lv_size,lv_attr,pool_lv,origin,data_percent,metadata_percent --sort=-lv_size +``` + +To move a pod to a less-used node, its PVC has to be deleted and a new one provisioned on the other node, with the data copied by the application. + +### Cause 2: PV Released with reclaim policy Retain, LogicalVolume still present + +A cleanup candidate meets all of these: phase `Released`, policy `Retain`, the PVC in `claimRef` is gone, and no Pod or StatefulSet references the claim. Anything still `Bound` is Cause 1. + +> **Warning:** Deleting the PV and the `logicalvolumes.topolvm.cybozu.com` object of the same name deletes the volume and its data. Precondition: all candidate conditions above are met, and a backup exists if anyone might need the files. Rollback: none. `lvremove` on the host is not a substitute and does not update the page. + +1. Save both objects, then delete them. The LogicalVolume name equals the PV name: + +```bash +kubectl get pv -o yaml > /tmp/.before.yaml +kubectl get logicalvolumes.topolvm.cybozu.com -o yaml > /tmp/.lv.before.yaml +kubectl delete logicalvolumes.topolvm.cybozu.com +kubectl delete pv +``` + +The expected output includes `logicalvolume.topolvm.cybozu.com "" deleted`. Deleting the LogicalVolume first and then the PV, or the other way around, ends with both gone. + +2. On the node named by `topology.topolvm.cybozu.com/node`, `lvs | grep ` lists the LV before the delete and prints nothing after it. `spec.csi.volumeHandle` in the PV is the LV name. +3. Repeat per candidate, then check `vgs` and the Local Storage page. The page can show the old number for several minutes after the objects and the LV are gone. + +If the LV is already gone from the host but the API objects remain, still delete the PV and the LogicalVolume. If the PVC and PV were already deleted, find the LogicalVolume by the PV name and delete it the same way: + +```bash +kubectl get logicalvolume | grep +kubectl delete logicalvolume +``` + +Never run `kubectl delete pv --all`, and never use this sequence on a Bound PV. + +### Cause 3: spare-gb is not a supported parameter + +Do not add `spare-gb` to the StorageClass, the `TopolvmCluster`, or a ModulePlugin `valuesOverride`; it is not honored. Keep free extents by not assigning the whole volume group to PVCs. Several expansions that together request all remaining capacity will stall; that is a sizing limit. Add a disk when the device class is really out of extents. + +### Cause 4: the capacity alert stays firing after NAS usage drops + +Judge capacity from the device-class allocation and the PVC requests, not from the NAS percentage: + +```bash +df -hT +vgs --units g +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm \ + -o jsonpath='{.status.nodeStorageState}' | jq . +``` + +If the device class needs more space, expand the node disk or add a disk to the device class. Expanding the NAS does not clear this alert. If the disk was already grown in the hypervisor and the pool did not follow, see [TopoLVM device class capacity does not change after the underlying disk is expanded on ACP](./TopoLVM_device_class_capacity_does_not_change_after_the_underlying_disk_is_expanded_on_ACP.md). + +### Cause 5: Local Storage capacity shows 0/0 because Prometheus is not installed + +Use this only when the TopoLVM pods are Running, a test claim mounts, and the page shows `0/0`. Confirm that Prometheus is not installed, then install it on the platform. The capacity figures return; no TopoLVM object has to change. + +### Cause 6: a deployment stays on Deploying because one component's request is larger than TopoLVM can allocate + +Compare the component's storage request with the free capacity, lower that request to a size the pool can provision, and deploy again. Do not delete unrelated PVCs to force the original size through. + +```bash +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm \ + -o jsonpath='{.status.nodeStorageState}' | jq . +vgs --units g +``` + +### Cause 7: LogicalVolume objects remain with no matching PV + +1. List LogicalVolume names with no PV of the same name, and read their node and volume ID: + +```bash +comm -23 \ + <(kubectl get logicalvolumes.topolvm.cybozu.com -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | sort) \ + <(kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | sort) +kubectl get logicalvolume -o jsonpath='{.spec.nodeName}{" "}{.status.volumeID}{" "}{.spec.size}{"\n"}' +kubectl get volumeattachment | grep +``` + +2. On that node, `lvs | grep ` shows the LV that still holds the space, and `lsblk | grep ` shows whether anything mounts it. + +> **Warning:** Deleting the LogicalVolume object lets TopoLVM delete the LV on the node; its data is gone. Precondition: no PV with the same name exists, no VolumeAttachment or mount references the `volumeID`, and the object is saved. Rollback: none for the data. + +```bash +kubectl get logicalvolume -o yaml > /tmp/.yaml +kubectl delete logicalvolume +``` + +3. Repeat per orphan. After the orphans are deleted, the TopoLVM capacity displays normally. + +## Diagnostic Steps + +```bash +kubectl get sc -o custom-columns='NAME:.metadata.name,PROV:.provisioner,RECLAIM:.reclaimPolicy,BINDING:.volumeBindingMode' +kubectl get pv | grep -E 'topolvm|Released|Bound' +kubectl get logicalvolumes.topolvm.cybozu.com +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm \ + -o jsonpath='{.status.nodeStorageState}' | jq . +``` + +- The provisioner must be `topolvm.cybozu.com`; a full Ceph pool is a different product. +- If the class has `thinPoolConfig`, check `data_percent` and `metadata_percent` of the thin pool in `lvs` on the node before reading the page figure. +- `WaitForFirstConsumer` with a `Pending` PVC and no pod is not full capacity; the claim has not been provisioned yet. +- Bound PVs whose requests add up to the usage: Cause 1. `Released` + `Retain`: Cause 2. A `spare-gb` request: Cause 3. Alert still firing after a NAS `df` drop: Cause 4. `0/0` with healthy pods: Cause 5. A deployment stuck on Deploying with one oversized request: Cause 6. LogicalVolume objects with no PV of the same name: Cause 7. + +## Related Information + +- [TopoLVM PVC expansion stays Resizing, spans two disks, or pods stay on the old node on ACP](./TopoLVM_PVC_expansion_stays_Resizing_spans_two_disks_or_pods_stay_on_the_old_node_on_ACP.md) +- [TopoLVM PVC rejects an access mode, fails to mount, or cannot be scheduled on ACP](./TopoLVM_PVC_rejects_an_access_mode_fails_to_mount_or_cannot_be_scheduled_on_ACP.md) diff --git a/docs/en/solutions/TopoLVM_device_class_capacity_does_not_change_after_the_underlying_disk_is_expanded_on_ACP.md b/docs/en/solutions/TopoLVM_device_class_capacity_does_not_change_after_the_underlying_disk_is_expanded_on_ACP.md new file mode 100644 index 000000000..fc7214651 --- /dev/null +++ b/docs/en/solutions/TopoLVM_device_class_capacity_does_not_change_after_the_underlying_disk_is_expanded_on_ACP.md @@ -0,0 +1,105 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.16.x + - 3.18.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# TopoLVM device class capacity does not change after the underlying disk is expanded on ACP + +## Issue + +A disk that TopoLVM already uses was expanded in place (a virtual disk grown in the hypervisor, or a LUN grown on the array), and the TopoLVM device class on Alauda Container Platform still shows the old capacity. Typical signs: + +- The **Storage Device** tab shows the expanded block device (for example 750 Gi), while the logical-pool view keeps the previous capacity of the device class (for example 500 Gi on `hdd`). +- `lsblk` on the node shows the new size, while `pvs` and `vgs` still show the old one. For example, `lsblk` shows `vdb` at `310G`, `pvs` shows `/dev/vdb` in the TopoLVM volume group with PSize `<200.00g` and PFree `1020.00m`, and `vgs` shows the volume group at VSize `<200.00g`. +- The only report is "the disk is full and the PVC cannot be expanded". For example a ZooKeeper volume filled up, its PVC expansion did not succeed, the device class showed `400/500 Gi` on each node, and `lsblk -l` showed `sdb` at `1000G`. + +The same question also comes up before expanding: on a virtual-machine node with one raw disk for TopoLVM, should the disk be grown in the IaaS layer, or should another disk be added? Growing the existing disk followed by `pvresize` adds the capacity to the volume group that already holds the PVCs. A disk added with **Add Storage Device** can become a separate volume group and then serves only new PVCs. + +This article covers only a disk that is already a physical volume of a TopoLVM device class. Growing a single PVC is covered in [TopoLVM PVC expansion stays Resizing, spans two disks, or pods stay on the old node on ACP](./TopoLVM_PVC_expansion_stays_Resizing_spans_two_disks_or_pods_stay_on_the_old_node_on_ACP.md). + +## Root Cause + +### Cause 1: lsblk shows the new disk size, pvs and vgs still show the old size + +TopoLVM is built on LVM and publishes capacity from the volume group. When the block device under an existing physical volume grows, the LVM physical-volume metadata keeps the old size until `pvresize` is run on that device. The volume group, and therefore the TopoLVM device class, keep the old size too. TopoLVM does not run `pvresize` for an in-place disk grow; the PV has to be resized by hand. Adding a new disk to the device class is the path TopoLVM handles itself. + +## Resolution + +### Cause 1: lsblk shows the new disk size, pvs and vgs still show the old size + +Use this only when `lsblk` is larger than `pvs` for the same path, and that path is a physical volume of the device class that shows the old capacity. + +1. Compare the sizes on the node that owns the disk: + +```bash +lsblk -o NAME,PATH,SIZE,TYPE /dev/ +pvs --units g -o pv_name,pv_size,pv_free,vg_name /dev/ +vgs --units g +``` + +> **Warning:** `pvresize` updates the LVM physical-volume metadata so the volume group includes the extents the hypervisor added. Existing logical volumes stay in place. Precondition: the `lsblk` size is larger than the `pvs` size for this path, the path belongs to the device class that shows the old capacity, and the hypervisor or array grow has finished. Rollback: none; do not try to shrink the physical volume afterwards. + +2. Resize the physical volume and confirm that the volume group grew: + +```bash +pvresize /dev/ +pvs --units g -o pv_name,pv_size,pv_free,vg_name /dev/ +vgs --units g +``` + +Run `pvresize` once per grown physical volume, on the node that owns it. After that, the logical-pool capacities on the Local Storage page update. + +3. If the TopoLVM capacity still shows the old size after `pvresize`, restart that node's `topolvm-node` pod. + +> **Warning:** Deleting the `topolvm-node` pod pauses new mounts and capacity reporting on that node until the replacement is Ready. Mounts that already succeeded stay mounted. Precondition: `pvresize` has succeeded on this node and the capacity has not updated. Rollback: none needed; the pod is recreated. + +```bash +kubectl -n nativestor-system get pod -o wide | grep topolvm-node | grep +kubectl -n nativestor-system delete pod +``` + +4. Expand the PVCs again. `pvresize` does not grow existing PVCs by itself; each PVC is expanded by editing `spec.resources.requests.storage`. In the ZooKeeper example, `pvresize /dev/sdb`, a `topolvm-node` restart, and a new PVC expansion let the expansion succeed, and the pods returned to `Running`. + +## Diagnostic Steps + +1. On the storage node, compare the block device with the LVM view of the same device: + +```bash +lsblk /dev/ +pvs /dev/ +vgs +``` + +Example (shortened): + +```text +NAME SIZE TYPE +vdb 310G disk + +PV VG PSize PFree +/dev/vdb <200.00g 1020.00m + +VG #PV #LV VSize VFree + 1 13 <200.00g 1020.00m +``` + +2. `lsblk` larger than `PSize` is Cause 1. A PVC expansion that fails while the device class shows less than the disk size `lsblk` reports is the same cause. Equal sizes mean the grow has not reached the node, or a different disk was expanded. + +3. After `pvresize`, check the device class capacity: + +```bash +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm \ + -o jsonpath='{.status.nodeStorageState}' | jq . +``` + +## Related Information + +- [TopoLVM capacity stays full after PVCs are deleted on ACP](./TopoLVM_capacity_stays_full_after_PVCs_are_deleted_on_ACP.md): device-class usage is the sum of PVC requests. diff --git a/docs/en/solutions/TopoLVM_device_class_or_storage_devices_show_Unavailable_or_Unknown_on_ACP.md b/docs/en/solutions/TopoLVM_device_class_or_storage_devices_show_Unavailable_or_Unknown_on_ACP.md new file mode 100644 index 000000000..e1cc380d5 --- /dev/null +++ b/docs/en/solutions/TopoLVM_device_class_or_storage_devices_show_Unavailable_or_Unknown_on_ACP.md @@ -0,0 +1,158 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# TopoLVM device class or storage devices show Unavailable or Unknown on ACP + +## Issue + +On the Local Storage page of Alauda Container Platform, a TopoLVM device class or several storage devices are shown as unavailable (`不可用`, "Unavailable"), or the node's local-storage status is `Unknown`. Typical reports: + +- After a node reboot the local-storage status was `Unknown`, and the device class was then deleted from `TopolvmCluster` by hand while the logical volumes were still on the disk. +- The device class is unavailable, sometimes with no error text, and the question is whether it affects use and how to clear it. +- NVMe devices in one or more device classes are unavailable, and the `topolvm-operator` log repeats `cluster is not health should requene`. +- Under **Storage Management > Local Storage > topolvm > Storage Devices** the list can be empty (`无设备`, "No devices"). + +Operands are in `nativestor-system`; the cluster object is `topolvm`; per-node lvmd settings are a ConfigMap such as `lvmdconfig-` (use the name `kubectl get cm -n nativestor-system` prints). + +Related articles: [TopoLVM lvmd exits with should have at least one device-class or Error: not found on ACP](./TopoLVM_lvmd_exits_with_should_have_at_least_one_device-class_or_Error_not_found_on_ACP.md) (the volume group name on the node differs from the configuration, or the volume groups are missing). + +## Root Cause + +### Cause 1: the device class was deleted after a reboot while the logical volumes were still on disk + +After a reboot showed `Unknown`, the device class was removed from `TopolvmCluster` by hand. `kubectl get topolvmclusters.topolvm.cybozu.com -n nativestor-system topolvm -o yaml` then shows `useAllDevices`, `useAllNodes`, and `useLoop` all `false`, and `status.nodeStorageState` holds only a failed class, for example `className: sdc`, device `/dev/sdc1`, `state: Offline`, message `disk may has filesystem or is not raw disk please check`, class message `create failed`. The class that still holds the data (for example `sdb`, volume group `sdb-`) is no longer listed, while its logical volumes are still on the disk. This is outside the supported node and device removal procedures, and full recovery is not guaranteed. + +### Cause 2: devices Offline because the stored disk serial lacks the suffix udev reports + +The `topolvm-operator` log repeats: + +```text +topolvm-controller: failed to reconcile failed to reconcile cluster "topolvm": cluster is not health should requene +``` + +and the `topolvm-prepare-vg-` Job is removed and recreated in a loop. In `topolvmcluster/topolvm` status the device has a `serial` and `state: Offline`, and its class is `state: UnReady`. On the node, `udevadm info /dev/` reports `ID_SERIAL=_1`, while `status.json` in the node's lvmd ConfigMap holds the same serial without the `_1` suffix. Because the stored serial does not match, TopoLVM treats the disk as gone. + +### Cause 3: the device class shows unavailable, and restarting the operator and the controller clears it + +The device class is unavailable, none of the shapes above or in the related article matches (names agree, logical volumes intact, serials match, the class is still in the spec), and restarting `topolvm-operator` and `topolvm-controller` brings the class back. The underlying reason for this shape is not known. + +## Resolution + +### Cause 1: the device class was deleted after a reboot while the logical volumes were still on disk + +The recommended path is to migrate the data off the node, then uninstall and redeploy TopoLVM. The temporary procedure below lets the existing volumes be created and mounted again, but it is not a clean steady state: the node stays `Unknown`, and upgrades and later TopoLVM updates are blocked while the operator is stopped. Risks include an overwritten configuration, a volume group that does not match, existing logical volumes that are not recognized, and PVC mounts that then fail permanently. Validate it on a replica environment of the same version and architecture first. + +1. Stop the operator from its CSV so it does not rewrite the ConfigMap. Set `spec.install.spec.deployments` for the deployment named `topolvm-operator` to `replicas: 0`, and confirm the operator pod is gone. + +> **Warning:** Operator replicas 0 stops reconciliation for every TopoLVM node, not only this one. Precondition: `lvs` still shows the volumes you need, the CSV is saved, and you accept the `Unknown` status and blocked upgrades. Rollback: set the replicas back to `1`. + +```bash +kubectl get csv -A | grep topolvm +kubectl -n nativestor-system get csv -o yaml > /tmp/topolvm-csv.before.yaml +kubectl -n nativestor-system edit csv +``` + +2. On ConfigMap `lvmdconfig-`, keep the existing `devices` and `status.json` keys, do not set `immutable`, and add `lvmd.yaml` if it is missing. The volume group must be the one `vgs` shows, which still holds the logical volumes. Leave other disks out. + +```yaml +lvmd.yaml: | + socket-name: /run/topolvm/lvmd.sock + device-classes: + - name: + volume-group: + default: true +``` + +```bash +kubectl -n nativestor-system get cm lvmdconfig- -o yaml > /tmp/lvmdconfig.before.yaml +kubectl -n nativestor-system edit cm lvmdconfig- +kubectl -n nativestor-system get cm lvmdconfig- -o yaml | grep lvmd.yaml +``` + +3. Restore the node under `spec.storage.deviceClasses` of `topolvmcluster/topolvm`: the same `className`, `default: true`, the device path with `type: disk`, and `volumeGroup` set to ``. Do not add disks that are not in that volume group. + +> **Warning:** A wrong `volumeGroup` or device path makes the node plugin look at the wrong disk. Precondition: `vgs` and `lvs` match the names you write, and the cluster YAML is saved. Rollback: restore the saved cluster object; fix the ConfigMap first rather than scaling the operator up to undo a bad name. + +4. Finish in this order: + 1. In the CSV, set the `topolvm-operator` replicas to `1`, so the operator recreates `topolvm-node` for the node. + 2. The recreated `topolvm-node` fails to start. Set the replicas in the CSV back to `0` and confirm the operator pod is gone. + 3. Re-apply the `lvmd.yaml` edit from step 2 and check it. + 4. Confirm that `topolvm-node` on that node starts, with the operator still at 0 replicas. + +When PVCs mount and capacity is displayed while the node is still `Unknown`, stop here; do not scale the operator up. Plan the data migration and the TopoLVM redeploy. + +### Cause 2: devices Offline because the stored disk serial lacks the suffix udev reports + +1. On each affected node, compare the udev serial with the stored one: + +```bash +udevadm info --query=property --name=/dev/ | grep '^ID_SERIAL=' +kubectl -n nativestor-system get cm -o name | grep -i lvm +kubectl -n nativestor-system get cm -o jsonpath='{.data.status\.json}{"\n"}' +``` + +> **Warning:** Scaling `topolvm-operator` to 0 pauses reconciliation for every TopoLVM node, and a wrong serial keeps the device Offline. Existing mounts keep working. Precondition: the udev `ID_SERIAL` and the stored serial differ for this device (for example only the `_1` suffix is missing), and the ConfigMap is saved. Rollback: apply the saved ConfigMap while the operator is still at 0, then scale it back to 1. + +2. Stop the operator, set the device's serial in `status.json` to the exact `ID_SERIAL` value udev printed, then start the operator: + +```bash +kubectl -n nativestor-system get cm -o yaml > /tmp/lvmd-configmap.before.yaml +kubectl -n nativestor-system scale deployment topolvm-operator --replicas=0 +kubectl -n nativestor-system edit cm +kubectl -n nativestor-system scale deployment topolvm-operator --replicas=1 +``` + +If OLM sets the operator back to 1 replica during the edit, stop it from the CSV as in Cause 1. + +3. The ConfigMap then shows the device `Online` and the class `Ready`, and `topolvmcluster` shows the disk online. If the console still shows the devices unavailable after the serial is corrected, contact support with the operator log and the ConfigMap; an additional fix was needed for the console to show the device classes as healthy (`健康`) and available (`可用`). + +### Cause 3: the device class shows unavailable, and restarting the operator and the controller clears it + +1. Before restarting, collect what is needed to find the cause if it returns: whether any disk operation was done on the node, the cluster object, and the prepare pod's events and logs. Run the commands where a kubeconfig for the cluster is available (`The connection to the server localhost:8080 was refused` means the host has no kubeconfig; it is not a TopoLVM symptom). + +```bash +kubectl get po -A -o wide | grep topolvm +kubectl get topolvmclusters -n nativestor-system topolvm -o yaml +kubectl -n nativestor-system describe pod +kubectl -n nativestor-system logs --all-containers +``` + +> **Warning:** Restarting the operator and the controller recreates those pods; new mounts wait until they are Ready, and mounts that already succeeded stay mounted. Precondition: no volume group mismatch, no deleted class, and no serial mismatch. Rollback: none needed. + +2. Restart both: + +```bash +kubectl -n nativestor-system rollout restart deployment topolvm-operator +kubectl -n nativestor-system rollout restart deployment topolvm-controller +kubectl -n nativestor-system get pods -o wide +``` + +Every node then shows a normal status (`正常`), and the device class shows available (`可用`) with its usage. If it becomes unavailable again, collect the prepare pod logs before restarting a second time. A `topolvm-prepare-vg` pod that stays `Pending` for a node that has already left the cluster is a different problem (stale node status). + +## Diagnostic Steps + +```bash +kubectl -n nativestor-system get pods -o wide +kubectl -n nativestor-system get topolvmcluster topolvm -o yaml +kubectl -n nativestor-system get cm lvmdconfig- -o yaml +kubectl -n nativestor-system logs deploy/topolvm-operator --tail=100 +``` + +On the node: + +```bash +pvs +vgs +lvs -a +udevadm info --query=property --name=/dev/ | grep '^ID_SERIAL=' +``` + +- `Unknown` after a reboot, a device class someone deleted from the spec, and `lvs` still showing data: Cause 1. +- Devices `Offline`, the operator repeating `cluster is not health should requene`, and a serial in `status.json` that lacks the suffix `udevadm info` shows: Cause 2. +- Unavailable with none of the above, and with volume group names that match: Cause 3. +- `lvmd` exiting with `should have at least one device-class` or `Error: not found`: see Related articles. diff --git a/docs/en/solutions/TopoLVM_device_unavailable_or_PVC_unusable_because_the_backing_disk_failed_disappeared_or_was_decommissioned_on_ACP.md b/docs/en/solutions/TopoLVM_device_unavailable_or_PVC_unusable_because_the_backing_disk_failed_disappeared_or_was_decommissioned_on_ACP.md new file mode 100644 index 000000000..944ff474d --- /dev/null +++ b/docs/en/solutions/TopoLVM_device_unavailable_or_PVC_unusable_because_the_backing_disk_failed_disappeared_or_was_decommissioned_on_ACP.md @@ -0,0 +1,196 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# TopoLVM device unavailable or PVC unusable because the backing disk failed, disappeared, or was decommissioned on ACP + +## Issue + +On **Storage Management > Local Storage > topolvm > Storage Devices** a TopoLVM device (for example `/dev/sdc`) is listed as unavailable (`不可用`, "Unavailable"), and one of the following is reported: + +- After the servers were powered off and on again, TopoLVM on a node does not start, and the workloads whose PVCs live on that node (for example a registry or a Git service) do not start either. +- A disk on a TopoLVM node is damaged, and a PVC on it can no longer be recognized or mounted. The question is whether rebuilding the PVC helps. +- `discover-device` fails with `failed to list all devices: exit status 127` on some nodes, although `lsblk`, `blkid`, and the `lvm2` packages are installed. +- A disk that TopoLVM used has been taken offline for good, and its entries still show in TopoLVM. + +TopoLVM volumes are LVM logical volumes on the node's own disk. When that disk is gone, the logical volumes on it are gone too; TopoLVM does not keep a second copy. Recovering the data means restoring it from the application's own backup or replica. + +Operands run in `nativestor-system`; the cluster object is `topolvmclusters.topolvm.cybozu.com/topolvm`; the per-node lvmd settings are ConfigMap `lvmdconfig-`. + +Related articles: [TopoLVM does not offer a disk or topolvm-prepare-vg fails to create the volume group on ACP](./TopoLVM_does_not_offer_a_disk_or_topolvm-prepare-vg_fails_to_create_the_volume_group_on_ACP.md) (the disk is present but still carries old signatures, or a new disk has to be added), [TopoLVM lvmd exits with should have at least one device-class or Error: not found on ACP](./TopoLVM_lvmd_exits_with_should_have_at_least_one_device-class_or_Error_not_found_on_ACP.md) (the disk is present but the volume group named in `lvmd.yaml` is missing or named differently), [TopoLVM device class or storage devices show Unavailable or Unknown on ACP](./TopoLVM_device_class_or_storage_devices_show_Unavailable_or_Unknown_on_ACP.md). + +## Root Cause + +### Cause 1: the device is unavailable after a power cycle and lsblk no longer lists the disk + +`lsblk` on the node lists the other disks (the operating-system disk and its partitions) but not the disk that the TopoLVM device names. The disk dropped off the node after the power cycle, so the device class on that node has no volume group, TopoLVM on that node does not start, and the pods whose PVCs live there stay down. This is a hardware or virtualization-layer fault of the disk, and the disk may have to be replaced. + +### Cause 2: a damaged disk leaves a PVC on it unusable + +A disk on a TopoLVM node is damaged, and the PV and PVC on it can no longer be recognized or mounted. Rebuilding the PVC creates a new, empty volume; it does not recover what was on the damaged disk. + +### Cause 3: discover fails with exit status 127 and the GPT main header of the disk is corrupted + +`discover-device` logs `failed to list all devices: exit status 127` on some nodes while the device tools are installed, and `topolvm-controller` repeats `probe failed` with `csidrivers.storage.k8s.io "topolvm.cybozu.com" not found`. A GPT check of a disk on those nodes reports a damaged primary partition table: + +```text +Invalid CRC on main header data; loaded backup partition table. +Main and backup partition tables differ! +One or more CRCs don't match. You should repair the disk! +Main header: ERROR +Backup header: OK +Main partition table: ERROR +Backup partition table: OK +Invalid partition data! +``` + +The failure was resolved by repairing that disk and rebuilding TopoLVM. The same `exit status 127` is also the signature of a CPU without `x86-64-v2` support, see [TopoLVM operator or discover-device crashes with CPU does not support x86-64-v2 on ACP](./TopoLVM_operator_or_discover-device_crashes_with_CPU_does_not_support_x86-64-v2_on_ACP.md); check the CPU flags as well. + +### Cause 4: a disk taken offline for good still appears in TopolvmCluster and in the lvmd ConfigMap + +A disk was physically taken offline, but TopoLVM still lists it. The device is still in the node's `devices` list under `spec.storage.deviceClasses` of `TopolvmCluster`, and it is still an entry of `deviceStates` in `status.json` of ConfigMap `lvmdconfig-`, for example: + +```json +"deviceStates": [{"name":"/dev/vdb","state":"Online"},{"name":"/dev/vdc","state":"Online"}] +``` + +Those leftover entries keep the node and the device class abnormal on the Local Storage page. + +## Resolution + +### Cause 1: the device is unavailable after a power cycle and lsblk no longer lists the disk + +1. Confirm, from the node and from the kernel log, that the disk is gone rather than renamed: + +```bash +lsblk +dmesg -T | grep -i -E 'sd[a-z]|nvme|I/O error' | tail -50 +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm \ + -o jsonpath='{.status.nodeStorageState}' | jq . +``` + +2. Have the hardware or virtualization team check the disk and, if needed, replace it. +3. A new disk does not carry the old volume group or its logical volumes. Restore the PVCs that lived there from the application's backup or replicas. Remove the old device entries as in Cause 4, and add the new raw disk with **Add Storage Device** (see the related article on adding a disk). + +### Cause 2: a damaged disk leaves a PVC on it unusable + +Rebuild the PVC only when the data on it is not needed or can be restored. First record which PV and node the claim uses: + +```bash +kubectl get pvc -n -o jsonpath='{.spec.volumeName}{"\n"}' +kubectl get pv -o jsonpath='{.spec.csi.driver}{" "}{.spec.nodeAffinity}{"\n"}' +kubectl get pod -n -o wide | grep +kubectl get pvc -n -o yaml > /tmp/.before.yaml +``` + +> **Warning:** Impact: deleting the PVC and PV drops the claim and whatever data was still on that volume; the restarted pod gets a new, empty volume. Precondition: only when the disk under this PV is confirmed damaged and the data is not needed or is backed up elsewhere. Rollback: none; restore from the application's backup. + +```bash +kubectl delete pvc -n +kubectl delete pv +kubectl delete pod -n +kubectl get pvc -n +``` + +The new claim is provisioned on a node whose device class is healthy. If the pod's owner does not create its PVC from a template (a StatefulSet `volumeClaimTemplates` does), recreate the PVC from the saved manifest, without its `spec.volumeName`, before you restart the pod. + +### Cause 3: discover fails with exit status 127 and the GPT main header of the disk is corrupted + +1. Confirm with a read-only check. `sgdisk -v` (verify) and `gdisk -l` only read the disk: + +```bash +lsblk -f +sgdisk -v /dev/ +gdisk -l /dev/ +``` + +Look for `Invalid CRC on main header data`, `Main and backup partition tables differ!`, `Main header: ERROR`, or `Main partition table: ERROR`, with `Backup header: OK`. Do not use `sgdisk -e` as a check: it writes to the disk (it moves the backup GPT structures to the end of the disk and writes the partition table back). + +2. Repair the disk, choosing by what it holds: + - The disk is a TopoLVM data disk and holds nothing you need: return it to a raw disk by removing its signatures and partition table, as in Cause 1 of [TopoLVM does not offer a disk or topolvm-prepare-vg fails to create the volume group on ACP](./TopoLVM_does_not_offer_a_disk_or_topolvm-prepare-vg_fails_to_create_the_volume_group_on_ACP.md). + - The partitions are still needed: rebuild the main header from the intact backup header. Agree this with whoever owns the disk first. + +> **Warning:** Impact: `sgdisk -e` (and the recovery menu of `gdisk`) rewrites the GPT headers and partition table on `/dev/`; a wrong disk, or a backup table that is itself stale, writes a bad partition table and can make every partition on the disk unreachable. Precondition: only when this cause is confirmed with the read-only check above, the disk is the right one, the GPT has been saved with `sgdisk --backup`, and the disk owner has agreed. Rollback: `sgdisk --load-backup=/root/-gpt.bak /dev/` restores the saved headers and table. + +```bash +sgdisk --backup=/root/-gpt.bak /dev/ +sgdisk -e /dev/ +sgdisk -v /dev/ +``` + +3. Rebuild the TopoLVM cluster on the repaired disks afterwards, and confirm `discover-device` no longer logs `exit status 127`. + +### Cause 4: a disk taken offline for good still appears in TopolvmCluster and in the lvmd ConfigMap + +Volumes that lived on that disk are already lost. The supported procedure is "Scenario 1: Remove a volume group device from a device-class volume group" in the product documentation topic "Remove Disks, Device-Class Volume Groups, or Nodes from LVM Storage". It applies when the node is still reachable, the device class provision type is `Thick`, and the volume group still has at least one healthy disk. If the failed disk was the only disk of that volume group, so `vgs` no longer lists the group, use Scenario 2 of the same topic (remove the device-class volume group); if the node itself is retired, use Scenario 3 (see [Removing a TopoLVM storage node or releasing its disks on ACP](./Removing_a_TopoLVM_storage_node_or_releasing_its_disks_on_ACP.md)). + +1. Save both objects and identify the logical volumes that used the failed disk: + +```bash +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm -o yaml > /tmp/topolvmcluster.before.yaml +kubectl -n nativestor-system get cm lvmdconfig- -o yaml > /tmp/lvmdconfig.before.yaml +kubectl -n nativestor-system scale deployment topolvm-operator --replicas=0 +vgs +lvs -a -o +devices 2>/dev/null | egrep "/dev/|unknown" | awk '{print $1}' +``` + +2. Each name the `lvs` line prints is a logical volume on the failed disk; its PV has that name as `spec.csi.volumeHandle`: + +```bash +kubectl get pv -o json | jq -r --arg HANDLE ' + .items[] | select(.spec.csi.volumeHandle == $HANDLE) + | [.metadata.name, .spec.claimRef.namespace, .spec.claimRef.name] | @tsv' +``` + +3. Stop the workloads that use those PVCs. + +> **Warning:** Impact: these commands delete the PVC, the PV, the `logicalvolumes.topolvm.cybozu.com` object, and the logical volume, and drop the missing disk from the volume group. Data on those volumes is not recoverable. Precondition: only when the disk is confirmed gone for good, the workloads are stopped, and the names come from the `lvs` and `jq` output above. Rollback: none. + +```bash +kubectl delete pvc -n +kubectl delete pv +kubectl delete logicalvolumes.topolvm.cybozu.com +lvremove / +vgreduce --removemissing +``` + +4. Remove the device entries, then start the operator. + +> **Warning:** Impact: while `topolvm-operator` is at 0 replicas no device class is reconciled on any node, and removing the entries drops TopoLVM's record of that disk. Existing mounts on other disks keep working. Precondition: only when the disk is physically offline for good, both objects are saved, and you remove only the failed device on the target node. Rollback: while the operator is still at 0, apply the saved `TopolvmCluster` and ConfigMap. + +```bash +kubectl -n nativestor-system edit topolvmclusters.topolvm.cybozu.com topolvm +kubectl -n nativestor-system edit cm lvmdconfig- +kubectl -n nativestor-system scale deployment topolvm-operator --replicas=1 +``` + +In `TopolvmCluster`, delete the failed device from the target node's `devices` list. In `status.json` of `lvmdconfig-`, delete that device's element from `deviceStates` and keep the others, for example leave only `{"name":"/dev/vdc","state":"Online"}` when `/dev/vdb` is the device that is gone. + +5. Confirm the device is gone from status. Every node then shows Normal (`正常`) on the Local Storage view and the device class shows Available (`可用`): + +```bash +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm -o jsonpath='{.status.nodeStorageState}' | jq +``` + +## Diagnostic Steps + +```bash +lsblk +kubectl -n nativestor-system get pods -o wide +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm -o yaml +kubectl -n nativestor-system get cm -o name | grep -i lvm +kubectl logs -n nativestor-system --tail=50 +``` + +- A device on the Storage Devices tab that `lsblk` on its node no longer shows after a power cycle: Cause 1. +- A PVC that cannot be mounted because its disk is damaged (`dmesg` shows I/O errors for that disk): Cause 2. +- `failed to list all devices: exit status 127`, CPU flags complete, and `Main header: ERROR` from `sgdisk -v` or `gdisk -l`: Cause 3. +- A disk that is gone for good while `TopolvmCluster` `devices` and `deviceStates` in `lvmdconfig-` still list it: Cause 4. +- Data inside a container is missing while the disk is healthy: check that the PVC is still bound to the original PV (`kubectl get pvc -n -o jsonpath='{.spec.volumeName}'`) and that `lvs -a` on the PV's node still lists the logical volume named by the PV's `spec.csi.volumeHandle`. A recreated PVC or PV is a new, empty volume. For what deletes TopoLVM data, see [Removing a TopoLVM storage node or releasing its disks on ACP](./Removing_a_TopoLVM_storage_node_or_releasing_its_disks_on_ACP.md). diff --git a/docs/en/solutions/TopoLVM_does_not_offer_a_disk_or_topolvm-prepare-vg_fails_to_create_the_volume_group_on_ACP.md b/docs/en/solutions/TopoLVM_does_not_offer_a_disk_or_topolvm-prepare-vg_fails_to_create_the_volume_group_on_ACP.md new file mode 100644 index 000000000..ba8e27289 --- /dev/null +++ b/docs/en/solutions/TopoLVM_does_not_offer_a_disk_or_topolvm-prepare-vg_fails_to_create_the_volume_group_on_ACP.md @@ -0,0 +1,196 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.16.x + - 3.18.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# TopoLVM does not offer a disk or topolvm-prepare-vg fails to create the volume group on ACP + +## Issue + +A data disk cannot be used by TopoLVM on Alauda Container Platform: + +- TopoLVM deployment reports an error on a data disk, TopoLVM cannot be created because the disk shows as unavailable, or a bare disk is missing from the addable device list. +- `pvcreate` reports `Device /dev/ excluded by a filter.`, or a device class reports `disk may has filesystem or is not raw disk please check`. +- The `topolvm-prepare-vg` log shows `Cannot use /dev/: device is partitioned` followed by `create vg - retry failed err:exit status 5`. +- A newly added disk stays "device unavailable" while the TopoLVM pod for that node keeps being recreated. +- A device class stays Offline or shows `0 GiB`, and `topolvm-prepare-vg` logs `device:/dev/mapper/mpatha is not available`. +- Adding a disk for expansion finishes with no extra capacity. + +The operands live in `nativestor-system`. The cluster object is `topolvmclusters.topolvm.cybozu.com/topolvm` (`TopolvmCluster`). Preparation is the Job `topolvm-prepare-vg`; discovery is the `discover-device` pod. + +TopoLVM needs a whole disk of type `disk`. A partition such as `/dev/sdb1` is rejected. TopoLVM does not require a particular RAID level: a hardware RAID volume (for example RAID 5) qualifies when the operating system sees it as a whole disk with no partitions, no filesystem, and no RAID member signature. `/dev/sda` is often the operating-system disk; never wipe a disk because its name matches an example. + +Related articles: [Former TopoLVM data disk stays unavailable because old LVM and thin-pool mappings remain on ACP](./Former_TopoLVM_data_disk_stays_unavailable_because_old_LVM_and_thin-pool_mappings_remain_on_ACP.md), [TopoLVM device class capacity does not change after the underlying disk is expanded on ACP](./TopoLVM_device_class_capacity_does_not_change_after_the_underlying_disk_is_expanded_on_ACP.md) (a disk TopoLVM already uses was grown in place), [TopoLVM lvmd exits with should have at least one device-class or Error: not found on ACP](./TopoLVM_lvmd_exits_with_should_have_at_least_one_device-class_or_Error_not_found_on_ACP.md). + +## Root Cause + +### Cause 1: Device excluded by a filter, or the disk still has a filesystem, partition, or LVM signature + +The disk still carries an old signature: `wipefs -n` or `lsblk -f` shows `LVM2_member`, a partition table, a filesystem, or a foreign member such as `jmicron_raid_member`. TopoLVM does not offer such a disk, `pvcreate` refuses it, and pool expansion does nothing until those signatures are removed. A partition table on a whole disk shows up as the `pvcreate` that prepare runs on the host failing: + +```text +prepare-vg: start provision with node status +nsenter -u -i -n -m -p -t 1 lvm pvcreate /dev/nvme0n1 +Cannot use /dev/nvme0n1: device is partitioned +prepare-vg: create vg - retry failed err:exit status 5 +``` + +The message `disk may has filesystem or is not raw disk please check` is also written when the device is already a physical volume of a TopoLVM volume group whose name differs from the one TopoLVM recorded. Then `pvs` shows the device in a VG named `-` and `lvs` shows live logical volumes. That is not a dirty disk, and wiping it destroys the PVC data (see the related article on lvmd). Run `pvs` and `lvs` before any wipe. + +If `wipefs -n` is empty and `pvcreate` still prints `excluded by a filter`, the LVM devices file is the next check, not a format. + +### Cause 2: the disk is not in TopolvmCluster.spec.storage.deviceClasses + +`TopolvmCluster` lists other devices on that `nodeName` and not the target path, and the `discover-device` logs have no line for it. A disk that was wiped but never added is not used. The UI **Add Storage Device** action is the normal path once an initial device class exists. On a cluster whose Local Storage page shows a healthy TopoLVM install but refuses the first node and disk, writing the node and device into `TopolvmCluster` by hand makes later UI adds work. + +### Cause 3: prepare-vg: device:/dev/mapper/mpatha is not available + +The prepare log shows `sys: device:dm-12 is available` and, on the next line, `prepare-vg: device:/dev/mapper/mpatha is not available`. Inventory and prepare disagree about one multipath device (`/dev/mapper/mpath*`, type `mpath`), for example an iSCSI LUN. The device class becomes Offline with `0 GiB`, and the prepare Job can show Succeeded while a new Job is created about every 30 seconds. `wipefs -n` is empty; this is not a dirty disk. The TopoLVM operator of ACP 3.16 (v3.16.6) does not support multipath devices. `/dev/dm-N` is not a stable name to configure instead. + +### Cause 4: newly added node disk is blank, the device stays unavailable and the pod is recreated repeatedly + +A disk was added to a node for TopoLVM. The device stays unavailable and the TopoLVM pod for that node keeps being recreated. `lsblk -f`, `blkid`, and `wipefs -n` show no filesystem, partition table, or LVM signature: the disk is blank, not dirty. Formatting it on the host made the device available and stopped the pod recreation. The exact filesystem type that was written is not known. + +A different shape with the same symptom: `topolvm-prepare-vg` and `topolvm-node` pods rebuild forever while `pvs`, `vgs`, and `lvs` already show the complete PV, VG, and LV and the logs are quiet. That disk is not blank, so this cause does not apply. + +## Resolution + +Identify the device with the commands in Diagnostic Steps before any wipe. Stop if the path is mounted, is the parent of `/`, `/boot`, or `/var`, or is already a PV in `pvs` that an application still uses. + +### Cause 1: Device excluded by a filter, or the disk still has a filesystem, partition, or LVM signature + +> **Warning:** Impact: `wipefs -af` deletes every filesystem, RAID, LVM, and partition signature on that path, and `sgdisk --zap-all` destroys the GPT and the protective MBR; the data is not recoverable from the node. Precondition: only when `wipefs -n` lists at least one leftover signature; `lsblk`, `findmnt`, `blkid`, and `pvs` show the disk is unused; it is not the disk that holds `/`; and no bound TopoLVM PV points at it. If `wipefs -n` prints nothing, there is nothing to wipe: go to the devices-file check below, or to Cause 4. Rollback: none; restore only from a backup taken before the wipe. + +```bash +lsblk /dev/ +wipefs -af /dev/ +sgdisk --zap-all /dev/ +wipefs -n /dev/ +lsblk /dev/ +``` + +The last `lsblk` must show the disk as `disk` with no child partitions and no mountpoint, and `wipefs -n` must print nothing. If `sgdisk` is not installed and `wipefs -n` is already empty, the `wipefs -af` step alone is enough. Then add the device (Cause 2). If `topolvm-prepare-vg` already failed on the old signature, delete that Job so the operator starts a new one; deleting the Job does not delete a volume group: + +```bash +kubectl delete job -n nativestor-system +``` + +If `pvcreate` still prints `excluded by a filter` after the wipe, the LVM devices file may be omitting the disk: + +```bash +lvmconfig devices/use_devicesfile +lvmdevices +``` + +> **Warning:** `lvmdevices --adddev` allows LVM to claim `/dev/`. Precondition: `use_devicesfile` is 1 and this is the TopoLVM data disk, not a disk you meant to keep out of LVM. Rollback: `lvmdevices --deldev /dev/`. + +```bash +lvmdevices --adddev /dev/ +``` + +Do not change the global `filter` in `/etc/lvm/lvm.conf` for this. If the disk still shows old TopoLVM thin-pool mappings in `dmsetup ls`, `wipefs` cannot remove them; see the related article on leftover mappings. + +### Cause 2: the disk is not in TopolvmCluster.spec.storage.deviceClasses + +Prefer **Storage Management > Local Storage > Storage Devices > Add Storage Device** and select the clean whole disk. If the page will not offer the node, edit the object. + +> **Warning:** A wrong `devices[].name` makes prepare-vg operate on that path, and `useAllDevices: true` can pull in the OS disk. Precondition: Cause 1 is clear, you saved the YAML, and the path is the intended whole disk. Rollback: remove the device entry you added and save again. + +```bash +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm -o yaml > /tmp/topolvmcluster.before.yaml +kubectl -n nativestor-system edit topolvmclusters.topolvm.cybozu.com topolvm +``` + +Add the node and device under `spec.storage.deviceClasses` (names are examples; keep `useAllDevices`, `useAllNodes`, and `useLoop` false unless the cluster already sets them): + +```yaml +spec: + storage: + useAllDevices: false + useAllNodes: false + useLoop: false + deviceClasses: + - nodeName: + classes: + - className: hdd + default: true + devices: + - name: /dev/ + type: disk +``` + +`type` must be `disk`. Wait until `status.nodeStorageState` shows the class Ready and the UI capacity includes the new size; a Job phase of Succeeded is not that check. After a device is added the capacity can take about 10 minutes to appear, for each extra raw disk. + +### Cause 3: prepare-vg: device:/dev/mapper/mpatha is not available + +1. Do not wipe the LUN and do not switch the class to `/dev/dm-N`. +2. To stop the 30-second Job recreation, remove the `mpath` device from `spec.storage.deviceClasses` with the same edit as Cause 2. That leaves the class without this LUN; it does not delete data on the LUN. +3. Give TopoLVM a non-multipath whole disk, or upgrade to an ACP version whose TopoLVM release notes list multipath support and add the LUN with the path that version's device list offers. + +### Cause 4: newly added node disk is blank, the device stays unavailable and the pod is recreated repeatedly + +Decide by what the disk holds, not by the error alone: + +- `wipefs -n` or `lsblk -f` lists old signatures: Cause 1. +- `wipefs -n` prints nothing and `pvcreate` prints `excluded by a filter`: neither wipe nor format; check the LVM devices file as in Cause 1. +- `wipefs -n` and `blkid` print nothing, there is no filter error, and the disk is the new, empty data disk: this cause. +- `pvs`, `vgs`, and `lvs` already show the complete TopoLVM PV, VG, and LV: neither wipe nor format here. Removing the node from `TopolvmCluster`, wiping the disk, and enrolling the node again is the recovery for that shape, and only when `lvs` shows no logical volume that backs a bound PV (see [Removing a TopoLVM storage node or releasing its disks on ACP](./Removing_a_TopoLVM_storage_node_or_releasing_its_disks_on_ACP.md)). + +Take the format from a disk that already works in the same device class on another node: + +```bash +# on a node where the same device class is Ready +lsblk -f /dev/ +blkid /dev/ || true +# on the new node +lsblk -o NAME,PATH,TYPE,SIZE,FSTYPE,MOUNTPOINT,PKNAME /dev/ +wipefs -n /dev/ || true +findmnt -n -o SOURCE / +``` + +> **Warning:** Impact: formatting writes a new signature to the whole disk and destroys anything on it. Precondition: the device is the newly added data disk, not the disk that holds `/`, `/boot`, or `/var`, not mounted, not in `pvs`, and `wipefs -n` shows no signatures. Rollback: none. If you cannot tell which format the working disk uses, do not guess; collect the output above and contact support. + +After the disk matches the working disk, delete a failed `topolvm-prepare-vg` Job so the operator creates a new one, and confirm the class is Ready in `status.nodeStorageState` and the pod on that node stops being recreated. + +## Diagnostic Steps + +1. Confirm the operands are up, then read device-class status. A missing `TopolvmCluster` is a controller problem, not a reason to wipe. + +```bash +kubectl get deploy,ds,job,pod -n nativestor-system -o wide +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm \ + -o jsonpath='{.status.nodeStorageState}' | jq . +``` + +2. On the node, prove identity and ownership: + +```bash +lsblk -o NAME,KNAME,PATH,TYPE,SIZE,FSTYPE,MOUNTPOINT,PKNAME /dev/ +blkid /dev/ || true +wipefs -n /dev/ || true +findmnt -n -o SOURCE / +pvs; vgs; lvs +multipath -ll +``` + +3. Read the prepare log: + +```bash +kubectl logs -n nativestor-system job/ --all-containers --tail=200 +``` + +- `excluded by a filter`, `device is partitioned`, or `disk may has filesystem` with a signature in `wipefs -n` and no TopoLVM VG in `pvs`: Cause 1. +- `disk may has filesystem or is not raw disk please check` while `pvs` shows the device in a TopoLVM `-` VG with live LVs: a VG-name mismatch, see the related lvmd article. +- Clean disk not listed in `TopolvmCluster`: Cause 2. +- `device:/dev/mapper/mpath* is not available` and a new prepare Job every 30 seconds: Cause 3. +- Blank disk, no filter error, pod recreated repeatedly: Cause 4. +- A pod mount error `target device is already formatted with different filesystem` is a signature on the logical volume, not on the disk; see [TopoLVM PVC rejects an access mode, fails to mount, or cannot be scheduled on ACP](./TopoLVM_PVC_rejects_an_access_mode_fails_to_mount_or_cannot_be_scheduled_on_ACP.md), Cause 2. +- In-use disk grown in place, so the Storage Device size is larger than `pvs`: see the related capacity article. diff --git a/docs/en/solutions/TopoLVM_lvmd_exits_with_should_have_at_least_one_device-class_or_Error_not_found_on_ACP.md b/docs/en/solutions/TopoLVM_lvmd_exits_with_should_have_at_least_one_device-class_or_Error_not_found_on_ACP.md new file mode 100644 index 000000000..82edf4956 --- /dev/null +++ b/docs/en/solutions/TopoLVM_lvmd_exits_with_should_have_at_least_one_device-class_or_Error_not_found_on_ACP.md @@ -0,0 +1,172 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# TopoLVM lvmd exits with should have at least one device-class or Error: not found on ACP + +## Issue + +Storage on one node of Alauda Container Platform is abnormal, and the logs point at the TopoLVM node plugin. The `lvmd` container of `topolvm-node` in `nativestor-system` exits with one of these, or pods that should use that node stay `Pending` with no node they can schedule on: + +```text +Error: should have at least one device-class +``` + +```text +lvmd info: "configuration file loaded: " device_classes="[0xc0005ff270 0xc0005ff2c0]" file_name="/etc/topolvm/lvmd.yaml" socket_name="/run/topolvm/lvmd.sock" +not found +Error: not found +``` + +The `lvmd-config` log of the operator can also report `node all volume groups are not available` for several nodes. Restarting `topolvm-node` does not help. + +Per-node lvmd settings are a ConfigMap in `nativestor-system`, named `lvmdconfig-` or `lvmd-config-node-` depending on the version; use the name `kubectl get cm -n nativestor-system` prints. The cluster object is `topolvmcluster/topolvm`. + +Related articles: [TopoLVM device class or storage devices show Unavailable or Unknown on ACP](./TopoLVM_device_class_or_storage_devices_show_Unavailable_or_Unknown_on_ACP.md) (the device class was deleted after a reboot, or devices are shown unavailable); [TopoLVM node pod does not start after a reboot because its data disk was formatted on ACP](./TopoLVM_node_pod_does_not_start_after_a_reboot_because_its_data_disk_was_formatted_on_ACP.md) (`Error: not found` with no physical volume left on the disk). + +## Root Cause + +### Cause 1: the volume group name on the node is not the name in lvmd or in TopolvmCluster + +On the node, `vgs` prints a volume group name that is not the name configured for that node, so `lvmd` has nothing to serve. A typical chain: + +1. `kubectl logs -n nativestor-system -c lvmd` loads `/etc/topolvm/lvmd.yaml` with `device_classes="[]"` and exits with `Error: should have at least one device-class`. +2. The `prepare-vg` container of the same pod skips other disks (for example `skipping device "nvme6n1" because it contains a filesystem "ext4"`), then logs `prepare-vg: start provision vg`, `prepare-vg: cm is existing check if need update`, `prepare-vg: device /dev/nvme1n1 is not available`, and `op-k8sutil: patching previous cm lvmdconfig-`. +3. ConfigMap `lvmdconfig-` then holds `device-classes: []` in `lvmd.yaml`. Its `status.json` lists the class as failed, with `vgName` `ssd-`, the device in state `Offline`, and the message `disk may has filesystem or is not raw disk please check`. +4. On the node, `pvs` shows `/dev/nvme1n1` in a different volume group, `ssd-`, and `lvs` shows that this group holds the active logical volumes of the existing PVCs. + +The disk is not dirty. TopoLVM looks for `ssd-`, finds the device used by `ssd-`, marks it unavailable, and writes an empty device-class list. + +### Cause 2: lvmd exits with Error: not found because the volume groups in lvmd.yaml do not exist on the node + +The node's `lvmd.yaml` declares device classes whose `volume-group` names (`-`) are missing on the node. `vgs` lists only the operating-system groups, and `pvs` shows the data disks as physical volumes that belong to no volume group. `lvmd` cannot find the groups it was told to serve, so it exits. On a healthy TopoLVM node the same disks hold those volume groups. + +### Cause 3: the volume group and thin pool on one node were wiped, so pods stay Pending + +`topolvm-node` runs on several nodes. On one node the volume group and the logical volumes are gone and the data disk is bare. Pods stay `Pending` because no node can take them: the wiped node cannot provide TopoLVM capacity, and the healthy node already runs a replica that anti-affinity will not double up. After the volume group is recreated, the pods still fail, because their PVC objects exist while the logical volumes behind them were wiped. + +## Resolution + +### Cause 1: the volume group name on the node is not the name in lvmd or in TopolvmCluster + +1. On the node, print the live volume group and its logical volumes. In the cluster, save the configured objects: + +```bash +vgs +pvs -o pv_name,vg_name,pv_size +lvs -o lv_name,vg_name,lv_size,lv_attr +kubectl -n nativestor-system get cm -o name | grep -i lvmd +kubectl -n nativestor-system get cm lvmdconfig- -o yaml > /tmp/lvmdconfig.before.yaml +kubectl -n nativestor-system get topolvmcluster topolvm -o yaml > /tmp/topolvmcluster.before.yaml +``` + +> **Warning:** A wrong volume group name makes `topolvm-node` look for a group that is not on the disk. Deleting Deployment `topolvm-node-` recreates the node plugin and pauses new mounts on that node until it is Ready; mounts that already succeeded stay mounted. Precondition: `vgs` on this node shows the name you write, and both objects are saved. Rollback: restore the saved ConfigMap and cluster object, then delete the Deployment again so it reloads them. + +2. Correct the ConfigMap first: `lvmd.yaml` and `status.json` must both describe the live volume group (`ssd-` in the example). In `lvmd.yaml` the field is `volume-group`; the layout is: + +```yaml +lvmd.yaml: | + socket-name: /run/topolvm/lvmd.sock + device-classes: + - name: + volume-group: + default: true +``` + +```bash +kubectl -n nativestor-system edit cm lvmdconfig- +``` + +3. Correct `TopolvmCluster`: in the node's device class entry, set `volumeGroup` to the same live name. Leave devices that are not in this volume group alone. + +```bash +kubectl -n nativestor-system edit topolvmcluster topolvm +``` + +4. Recreate the node plugin so it reloads the configuration: + +```bash +kubectl -n nativestor-system delete deploy topolvm-node- +``` + +5. Confirm: + +```bash +kubectl -n nativestor-system get cm lvmdconfig- -o jsonpath='{.data.lvmd\.yaml}{"\n"}{.data.status\.json}{"\n"}' +kubectl -n nativestor-system get pods -o wide | grep +``` + +After the names match, the components return to Running and storage on the node recovers. In this cause, `disk may has filesystem or is not raw disk please check` refers to the TopoLVM volume group that holds the PVC data: do not wipe the disk. + +### Cause 2: lvmd exits with Error: not found because the volume groups in lvmd.yaml do not exist on the node + +Use this only when the `lvmd` log ends with `Error: not found`, every `volume-group` name in the node's `lvmd.yaml` is absent from `vgs`, and `pvs` shows the intended data disk as a physical volume with no volume group. If `pvs` prints no physical volume at all and `lsblk -f` shows a filesystem on the data disk, the disk was formatted; see the related article on a formatted data disk instead. + +1. Read the names and the disks: + +```bash +kubectl -n nativestor-system get cm -o jsonpath='{.data.lvmd\.yaml}{"\n"}' +vgs +pvs +lsblk /dev/ +``` + +2. Take the class-to-disk mapping from the platform storage view or from a healthy node. The healthy node's volume group names carry a different UUID; type the names from this node's `lvmd.yaml`. + +> **Warning:** `vgcreate` writes volume group metadata on `/dev/`. Precondition: this cause is confirmed, `pvs` shows `/dev/` with an empty VG column, `lvs` shows nothing on it, and the `volume-group` name you type is copied exactly from this node's `lvmd.yaml` for the class mapped to this disk. Rollback: `vgremove ` while the new group still holds no logical volume. + +3. Create one volume group per missing class on its disk, then restart the node plugin pod: + +```bash +vgcreate /dev/ +vgs +kubectl -n nativestor-system delete pod +``` + +TopoLVM then starts on the node. If the class uses a thin pool, the thin pool must exist as well (see Cause 3). + +### Cause 3: the volume group and thin pool on one node were wiped, so pods stay Pending + +Use this only when the node's data disk is already bare and the lvmd ConfigMap still describes the volume group and thin pool the node should have. + +> **Warning:** Creating a volume group on the disk destroys whatever is still on it. Precondition: `lsblk` and `vgs` show the disk bare, the ConfigMap is the only source of the names, and you accept that the old data is not coming back. Rollback: none. + +1. Recreate the volume group and the thin-pool logical volume so they match the names and the disk path already in the ConfigMap. If the ConfigMap does not contain those names, stop; do not invent new names. TopoLVM scheduling on the node returns once the thin pool matches the ConfigMap. +2. Delete the PVCs whose logical volumes were wiped and the stuck pods, so the workload creates new claims and rebuilds those replicas. + +> **Warning:** Deleting a PVC drops the claim; with reclaim policy `Delete` the PV object goes too. Precondition: `lvs` on every node shows no LV for that claim, and the application can rebuild a replica. Rollback: none. + +```bash +kubectl get pvc -n +kubectl delete pvc -n +kubectl delete pod -n +``` + +Anti-affinity against the replica still running on the healthy node is why the new pods cannot start until this node is usable again. Restore the node rather than placing both replicas on the surviving node. + +## Diagnostic Steps + +```bash +kubectl -n nativestor-system get pods -o wide +kubectl -n nativestor-system logs deploy/topolvm-node- -c lvmd --tail=100 +kubectl -n nativestor-system logs deploy/topolvm-node- -c prepare-vg --tail=100 +kubectl -n nativestor-system get cm lvmdconfig- -o yaml +kubectl -n nativestor-system get topolvmcluster topolvm -o yaml +``` + +On the node: + +```bash +pvs +vgs +lvs -a +``` + +- `Error: should have at least one device-class`, `lvmd.yaml` with `device-classes: []`, and `pvs` showing the device in a `-` volume group other than the `vgName` in `status.json`: Cause 1. Do not restart repeatedly; correct the names. +- `Error: not found` while `vgs` lacks the groups named in `lvmd.yaml` and `pvs` shows the data disk with an empty VG column: Cause 2. Compare names before running `vgcreate`. +- A bare disk where the ConfigMap still describes a volume group and thin pool, and pods `Pending`: Cause 3. +- `Error: not found` with empty `pvs` and a filesystem on the data disk: the formatted-disk article in Related articles. diff --git a/docs/en/solutions/TopoLVM_node_pod_does_not_start_after_a_reboot_because_its_data_disk_was_formatted_on_ACP.md b/docs/en/solutions/TopoLVM_node_pod_does_not_start_after_a_reboot_because_its_data_disk_was_formatted_on_ACP.md new file mode 100644 index 000000000..cd919d58e --- /dev/null +++ b/docs/en/solutions/TopoLVM_node_pod_does_not_start_after_a_reboot_because_its_data_disk_was_formatted_on_ACP.md @@ -0,0 +1,112 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# TopoLVM node pod does not start after a reboot because its data disk was formatted on ACP + +## Issue + +After a kernel upgrade or a node reboot on Alauda Container Platform, the TopoLVM pod on that node does not start. Workloads whose PVCs live on the node cannot mount, and new TopoLVM volumes are not created there. + +The `driver-registrar` container of the node's `topolvm-node` pod in `nativestor-system` waits for the CSI socket and gives up: + +```text +"Still connecting" address="unix:///run/topolvm/csi-topolvm.sock" +"Error connecting to CSI driver" err="context deadline exceeded" +``` + +The previous log of the `lvmd` container (`/lvmd --config=/etc/topolvm/lvmd.yaml --container=true`) loads the configuration with `socket_name="/run/topolvm/lvmd.sock"` and exits with `not found` / `Error: not found`. + +On the node, `pvs`, `vgs`, and `lvs` print nothing for the data disks, `pvscan` prints `No matching physical volumes found`, and `lsblk -f` shows a filesystem (for example `xfs`) on the disk TopoLVM used, with no `LVM2_member` anywhere. + +The same `Error: not found` while `pvs` still lists the data disk is a different problem; see [TopoLVM lvmd exits with should have at least one device-class or Error: not found on ACP](./TopoLVM_lvmd_exits_with_should_have_at_least_one_device-class_or_Error_not_found_on_ACP.md). + +## Root Cause + +### Cause 1: lvmd Error: not found, empty pvs and vgs, and the TopoLVM data disk now carries a filesystem + +The LVM physical volume, the volume group, and the logical volumes TopoLVM used on the node no longer exist, because the data disk was formatted. `lvmd` cannot find any volume group named in `lvmd.yaml` and exits, so the CSI socket `/run/topolvm/csi-topolvm.sock` is never created and `driver-registrar` times out. + +A typical way this happens: the node does not come up cleanly during the reboot after a kernel upgrade, and a disk is formatted to bring it back (for example as `/var/lib/containerd`). Kernel device names such as `sdb` and `sdc` can swap between boots, so the disk formatted as `sdb` may be the disk TopoLVM used before the reboot. The LVM data on the formatted disk is lost. + +A loaded device-mapper (`lsmod | grep dm` lists `dm_mod`) and the new kernel are not the fault; the missing LVM metadata is. + +## Resolution + +### Cause 1: lvmd Error: not found, empty pvs and vgs, and the TopoLVM data disk now carries a filesystem + +Use this only when all of these hold on the node: `lvmd` ends with `Error: not found`; `pvs`, `vgs`, and `lvs` show no TopoLVM physical volume or volume group; `pvscan` prints `No matching physical volumes found`; and `lsblk -f` shows a filesystem instead of `LVM2_member` on the disk TopoLVM used. Ask whether anyone formatted or wiped a disk on that node after the reboot. + +The TopoLVM volumes that lived on that disk are gone and cannot be recovered in place. Plan to restore those applications from their own backups or replicas. + +1. Work out which physical disk TopoLVM expects and what is on each disk now. Kernel names can change between boots; the device paths in `TopolvmCluster` may be stable `/dev/disk/by-id/...` paths. + +```bash +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm -o yaml > /tmp/topolvmcluster.before.yaml +grep -A 8 "nodeName: " /tmp/topolvmcluster.before.yaml +``` + +On the node: + +```bash +ls -l /dev/disk/by-id/ | grep -v part +lsblk -f +findmnt -o TARGET,SOURCE,FSTYPE | grep -E '/dev/sd|/dev/vd|/dev/nvme' +``` + +A disk that now holds a mounted filesystem the node needs (for example `/var/lib/containerd`) is not a candidate for TopoLVM. + +2. List the claims that the removal will delete: + +```bash +kubectl get pv -o json | jq -r --arg NODE "" ' + .items[] + | select(.spec.csi.driver=="topolvm.cybozu.com") + | select(.spec.nodeAffinity.required.nodeSelectorTerms[]?.matchExpressions[]? + | select(.key=="topology.topolvm.cybozu.com/node") | .values[]? == $NODE) + | [.metadata.name, .status.phase, .spec.claimRef.namespace, .spec.claimRef.name] + | @tsv' +``` + +> **Warning:** The removal sequence scales `topolvm-operator` to 0, deletes every TopoLVM PVC, PV, and LogicalVolume pinned to this node, and removes the node from `TopolvmCluster`. Workloads using those claims stop until new claims are created. Precondition: this cause is confirmed, so the logical volumes behind those objects are already gone, and the application owners agree to recreate their volumes. Rollback: none for the deleted claims; restore the node element from `/tmp/topolvmcluster.before.yaml` to undo the spec edit. + +3. Remove the node from TopoLVM with the sequence in [Removing a TopoLVM storage node or releasing its disks on ACP](./Removing_a_TopoLVM_storage_node_or_releasing_its_disks_on_ACP.md), Cause 1. +4. Add the node back, from **Storage Management > Local Storage > Storage Devices > Add Storage Device**, or by restoring the node's element with `kubectl -n nativestor-system edit topolvmclusters.topolvm.cybozu.com topolvm`. Select the data disk by its stable `/dev/disk/by-id/` path and check it against step 1: it must not be the disk that now holds a mounted filesystem. If the page does not offer the disk because it still carries the filesystem from the format, the leftover signature has to be cleared first; confirm with `lsblk -f` that the disk is the intended, unmounted one before wiping anything. +5. Verify: + +```bash +kubectl -n nativestor-system get pod -o wide | grep +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm \ + -o jsonpath='{.status.nodeStorageState}' | jq . +``` + +The prepare pod for the node finishes `Completed`, `topolvm-node` is Running with all containers ready, and the node's class is Ready. Then let the affected workloads create new PVCs and restore their data. + +To prevent this: before formatting or wiping a disk on a TopoLVM node, especially after a reboot, check it with `lsblk -f` and `ls -l /dev/disk/by-id/`. An `LVM2_member` signature means it is a TopoLVM physical volume. Using `/dev/disk/by-id/` paths in `TopolvmCluster` also protects against kernel name changes; see [TopoLVM storage breaks when VM disk names change: use /dev/disk/by-id paths in TopolvmCluster on ACP](./TopoLVM_storage_breaks_when_VM_disk_names_change_use_devdiskby-id_paths_in_TopolvmCluster_on_ACP.md). + +## Diagnostic Steps + +```bash +kubectl -n nativestor-system get pod -o wide | grep topolvm-node +kubectl -n nativestor-system logs -c driver-registrar --tail=50 +kubectl -n nativestor-system logs -c lvmd --previous --tail=50 +``` + +On the node: + +```bash +uname -r +pvs +vgs +lvs -a +pvscan +lsblk -f +``` + +- `driver-registrar` timing out on `/run/topolvm/csi-topolvm.sock` only says the node plugin never became ready; read the `lvmd` log next. +- `Error: not found` with empty `pvs`, `vgs`, and `lvs`, `No matching physical volumes found` from `pvscan`, and a filesystem on the disk TopoLVM used: Cause 1. +- `Error: not found` while `pvs` still lists the data disk: see the article linked in the Issue. diff --git a/docs/en/solutions/TopoLVM_operator_or_discover-device_crashes_with_CPU_does_not_support_x86-64-v2_on_ACP.md b/docs/en/solutions/TopoLVM_operator_or_discover-device_crashes_with_CPU_does_not_support_x86-64-v2_on_ACP.md new file mode 100644 index 000000000..20545d2da --- /dev/null +++ b/docs/en/solutions/TopoLVM_operator_or_discover-device_crashes_with_CPU_does_not_support_x86-64-v2_on_ACP.md @@ -0,0 +1,138 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# TopoLVM operator or discover-device crashes with CPU does not support x86-64-v2 on ACP + +## Issue + +TopoLVM cannot be deployed or does not discover disks on some nodes of Alauda Container Platform. One of the following is seen in namespace `nativestor-system`: + +- The TopoLVM deployment in the console stays at `installing`. The `topolvm-operator` pod is `0/1` `CrashLoopBackOff` with a large restart count, and its log is a single line: + + ```text + Fatal glibc error: CPU does not support x86-64-v2 + ``` + +- Deploying TopoLVM fails with an error that the CPU version is not supported, and the nodes are virtual machines. +- The navigation configuration looks empty, and the `discover-device` pods crash-loop on the control-plane nodes while the same pods on the workers are Running. Their log repeats: + + ```text + discover: can not list disk err:failed to list all devices: exit status 127 + discover: failed to update device configmap: failed to list all devices: exit status 127 + ``` + +- `discover-device` pods run on control-plane nodes that must stay free of workloads, and the goal is to keep them off for good. + +## Root Cause + +### Cause 1: the virtual machine has no CPU passthrough, and deploy reports that the CPU version is not supported + +The nodes are virtual machines with CPU passthrough turned off, so the guest sees a generic CPU model instead of the host CPU. TopoLVM requires a CPU that the generic model does not provide, and deploy fails with the unsupported CPU version error. + +### Cause 2: discover-device crash-loops on control-plane nodes because their CPUs lack x86-64-v2 + +`discover-device` is the disk inventory pod. Its image needs the x86-64-v2 instruction set. On the control-plane nodes, `/proc/cpuinfo` lacks flags such as `sse4_1`, `sse4_2`, `popcnt`, and `ssse3`, while the workers have them. A test discover pod scheduled onto such a node logs `Fatal glibc error: CPU does not support x86-64-v2`. Because the pod cannot start there, it never creates the lvmd ConfigMap for that node: `kubectl get cm -n nativestor-system` lists `lvmdconfig-` for each worker and `nativestor-setting`, and nothing for the control-plane nodes. + +The required host tools can all be present (`lvm`, `pvs`, `vgs`, `lvs`, `nsenter`, `lsblk`, from the `lvm2` and `util-linux` packages); `exit status 127` here is not a missing package. + +### Cause 3: topolvm-operator exits with Fatal glibc error: CPU does not support x86-64-v2 + +The CPU of the node that runs `topolvm-operator` does not expose x86-64-v2, which the glibc in the operator image requires, so the process exits at startup. The console stays at `installing`, the pod crash-loops, and Service `topolvm-cluster-metric` (selector `name=topolvm-operator`) has no endpoints because the operator never becomes ready. This is the same CPU baseline as Cause 2, hit by the operator instead of `discover-device`. + +### Cause 4: discover-device runs on control-plane nodes because of DISCOVER_DEVICE_TOLERATIONS + +The tolerations of `discover-device` come from the key `DISCOVER_DEVICE_TOLERATIONS` in ConfigMap `nativestor-setting` in `nativestor-system`: + +```yaml +DISCOVER_DEVICE_TOLERATIONS: | + - operator: Exists +``` + +A toleration with only `operator: Exists` tolerates every taint, including the control-plane taint, so `discover-device` is also placed on those nodes. + +## Resolution + +### Cause 1: the virtual machine has no CPU passthrough, and deploy reports that the CPU version is not supported + +1. Compare the CPU flags with a node where TopoLVM already runs, if there is one. A short, generic flag list together with the unsupported CPU version error is this cause. + +```bash +grep -m1 flags /proc/cpuinfo +``` + +2. Enable CPU passthrough for the virtual machine in the hypervisor that owns it (the setting name depends on the hypervisor), then deploy TopoLVM again. No Kubernetes setting replaces this. + +```bash +kubectl -n nativestor-system get pods +``` + +### Cause 2: discover-device crash-loops on control-plane nodes because their CPUs lack x86-64-v2 + +1. Confirm the split between control-plane nodes and workers: + +```bash +kubectl get pods -A -o wide | grep discover-device +kubectl get cm -n nativestor-system | grep -i lvmd +``` + +2. Check the CPU flags on a crashing node. Any `MISSING` line means the CPU does not meet x86-64-v2. The loop checks the flags x86-64-v2 needs (`pni` is how `/proc/cpuinfo` names SSE3): + +```bash +for f in ssse3 sse4_1 sse4_2 popcnt cx16 lahf_lm pni; do grep -qw "$f" /proc/cpuinfo && echo "$f present" || echo "$f MISSING"; done +``` + +3. Provide x86-64-v2 on those CPUs. For virtual machines this is the hypervisor CPU model or passthrough change from Cause 1. Wiping disks, restarting the pod, or reinstalling host packages does not add CPU instructions. + +Until the CPU meets the baseline, a node that cannot run `discover-device` does not appear as a selectable disk node. Place TopoLVM data disks on the nodes where discovery works. To keep the pods off those nodes entirely, see Cause 4. + +### Cause 3: topolvm-operator exits with Fatal glibc error: CPU does not support x86-64-v2 + +1. Find the node the operator runs on and confirm the log line: + +```bash +kubectl get pod -n nativestor-system -l name=topolvm-operator -o wide +kubectl get ep topolvm-cluster-metric -n nativestor-system +kubectl -n nativestor-system logs +``` + +2. On that node, run the CPU flag loop from Cause 2. +3. Fix it on the IaaS side: enable CPU passthrough for the virtual machine, or change the guest CPU model so that it exposes x86-64-v2. No switch inside the cluster works around it. +4. Run the flag loop again; every line should say `present`. Then check that the operator pod becomes Running and the console leaves `installing`: + +```bash +kubectl -n nativestor-system get pods +``` + +### Cause 4: discover-device runs on control-plane nodes because of DISCOVER_DEVICE_TOLERATIONS + +The `- operator: Exists` entry under `DISCOVER_DEVICE_TOLERATIONS` can be deleted so that `discover-device` is no longer scheduled onto control-plane nodes. Do this only when there is a specific reason, for example nodes that must not run workloads or whose pod network cannot reach the management network. + +> **Warning:** After the toleration is removed, `discover-device` no longer runs on tainted nodes, so their disks are not discovered or offered to TopoLVM. Precondition: no TopoLVM data disk is meant to be used on those nodes, and the ConfigMap is saved. Rollback: put the saved `DISCOVER_DEVICE_TOLERATIONS` value back. + +```bash +kubectl -n nativestor-system get cm nativestor-setting -o yaml > /tmp/nativestor-setting.before.yaml +kubectl -n nativestor-system edit cm nativestor-setting +kubectl get pods -A -o wide | grep discover-device +``` + +If `discover-device` pods are still on those nodes after the edit, collect the output and the DaemonSet YAML before deleting anything. + +## Diagnostic Steps + +```bash +kubectl -n nativestor-system get pods -o wide +kubectl -n nativestor-system logs --tail=50 +kubectl -n nativestor-system logs +grep -m1 flags /proc/cpuinfo +``` + +- An unsupported CPU version error on a virtual machine during deploy: Cause 1. +- `failed to list all devices: exit status 127` only on control-plane nodes, with the glibc `x86-64-v2` line or missing CPU flags: Cause 2, even when every host tool is installed. +- The same glibc line in the `topolvm-operator` log, usually reported as a console stuck at `installing` with the operator in `CrashLoopBackOff` and no endpoints behind `topolvm-cluster-metric`: Cause 3. Read the operator log first. +- `discover-device` pods on control-plane nodes that must stay free of workloads: Cause 4. +- `failed to list all devices: exit status 127` on nodes whose CPU flags are all present can also come from a damaged disk partition table (for example `Invalid CRC on main header data` in a GPT check). Check the CPU flags first, then the disk. diff --git a/docs/en/solutions/TopoLVM_storage_breaks_when_VM_disk_names_change_use_devdiskby-id_paths_in_TopolvmCluster_on_ACP.md b/docs/en/solutions/TopoLVM_storage_breaks_when_VM_disk_names_change_use_devdiskby-id_paths_in_TopolvmCluster_on_ACP.md new file mode 100644 index 000000000..4ad3d38fa --- /dev/null +++ b/docs/en/solutions/TopoLVM_storage_breaks_when_VM_disk_names_change_use_devdiskby-id_paths_in_TopolvmCluster_on_ACP.md @@ -0,0 +1,115 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# TopoLVM storage breaks when VM disk names change: use /dev/disk/by-id paths in TopolvmCluster on ACP + +## Issue + +The cluster nodes are virtual machines, and TopoLVM (LVM storage) uses their virtual disks. When a VM is rebuilt, its disks can come back under different kernel names (for example `/dev/sdb` becomes `/dev/sdc`, or `/dev/vdb` becomes `/dev/vdc`), and TopoLVM storage on that node becomes abnormal. The question is whether disks can be added to TopoLVM by a stable path such as `/dev/disk/by-id/...` instead. + +## Root Cause + +The disk picker in the console fills the TopolvmCluster with kernel device names such as `/dev/sdb` or `/dev/vdb`. Kernel device names are assigned at boot in discovery order and are not guaranteed to stay the same when a VM is rebuilt or its disks are reattached. After a rename, `spec.storage.deviceClasses[].classes[].devices[].name` in the TopolvmCluster no longer points to the disk that holds the volume group. + +Using kernel names is a console behavior, not a backend limitation. The TopolvmCluster accepts a `/dev/disk/by-id/...` path in `devices[].name`, both when a TopolvmCluster is created and when an existing one is edited in YAML. + +Pick the right kind of stable path: + +- Use the `by-id` link that is derived from the disk serial number or WWN (prefixes `virtio-`, `wwn-`, `scsi-`, `nvme-`). +- Do not use `/dev/disk/by-path/...`. It follows the PCI slot and changes when the slot changes, so it does not solve this problem. +- A `by-id` link is only as stable as the disk serial. If a rebuild also changes the serial, the `by-id` link changes too. Keep the disk serial unchanged on the virtualization side. + +## Resolution + +### New TopolvmCluster + +Write the TopolvmCluster in YAML and set each `devices[].name` to the serial-based `/dev/disk/by-id/...` path of the disk instead of the kernel name. + +### Existing TopolvmCluster: switch one node to by-id paths + +The switch is safe only when the path points to the same disk that is already in the volume group, and only when the change replaces paths without adding or removing disks. + +> **Warning:** Do not add or remove disks in the same edit as the path change. When the number of devices changes, the volume-group shrink logic compares device paths as strings and can treat the existing physical volume (still known by its old kernel name) as a disk to remove. Do one node at a time. Save the TopolvmCluster first. Rollback: apply the saved YAML, which puts the kernel names back. + +1. Save the current TopolvmCluster: + + ```bash + kubectl -n nativestor-system get topolvmcluster + kubectl -n nativestor-system get topolvmcluster -o yaml > /tmp/topolvmcluster-.yaml + ``` + +2. On the node, confirm that the `by-id` link resolves to the disk that `pvs` reports for the volume group: + + ```bash + ls -l /dev/disk/by-id/ + readlink -f /dev/disk/by-id/ + pvs -o pv_name,vg_name,pv_uuid + ``` + + The output of `readlink -f` must be the same device as the `pv_name` of the volume group, for example `/dev/vdb`. + +3. Edit the TopolvmCluster and change only `devices[].name` of that node under `spec.storage.deviceClasses[].classes[]` from the kernel name to the `by-id` path: + + ```bash + kubectl -n nativestor-system edit topolvmcluster + ``` + +4. Wait for the node's `topolvm-prepare-vg` run to finish and check its log. With an unchanged number of devices, the shrink step is skipped with the message `no need shrink vg for disk name change`, and the existing volume group and logical volumes are not touched: + + ```bash + kubectl -n nativestor-system get pods -o wide | grep topolvm-prepare-vg + kubectl -n nativestor-system logs | grep -i shrink + ``` + + The `lvmd.yaml` in the node's lvmd configuration needs no change. It records only the volume group name, not device paths. + +5. Align the status with the spec. The TopolvmCluster status does not switch to the `by-id` path on its own. After `topolvm-prepare-vg` has finished, edit the node's ConfigMap in `nativestor-system` (named `lvmdconfig-`). In its `status.json`, set `successClasses[].deviceStates[].name` to the same `by-id` path as in the spec, and leave `serial` unchanged. The controller syncs this status into `TopolvmCluster.status`. + + ConfigMap names are truncated for long node names, so match the ConfigMap by its `node-name` annotation: + + ```bash + kubectl -n nativestor-system get cm -l topolvm/lvmdconfig=lvmdconfig \ + -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.node-name}{"\n"}{end}' + kubectl -n nativestor-system get cm -o yaml > /tmp/.yaml + kubectl -n nativestor-system edit cm + ``` + +6. Repeat for the next node. + +### Known side effect + +On ACP 4.3.3, the storage performance monitoring page queries metrics by kernel device name. After a disk is switched to a `by-id` path, storage keeps working, but the performance page may show no data for that disk. + +## Diagnostic Steps + +1. See which device names the TopolvmCluster uses: + + ```bash + kubectl -n nativestor-system get topolvmcluster -o yaml | grep -n -A3 'devices:' + ``` + + Kernel names such as `/dev/sdb` or `/dev/vdb` mean that the node is exposed to renaming. + +2. On an affected node, compare the configured name with the disk that LVM actually uses: + + ```bash + pvs -o pv_name,vg_name,pv_uuid + ls -l /dev/disk/by-id/ | grep -v part + ``` + + If the `pv_name` of the volume group differs from the name in the TopolvmCluster, the disk was renamed. + +3. Check that the `by-id` link you plan to use is based on the serial and stays the same across a rebuild. Compare the link before and after a test rebuild, or check the serial that the virtualization platform assigns to the disk: + + ```bash + lsblk -o NAME,SERIAL,WWN + ``` + + An empty `SERIAL` and `WWN` means that the disk has no serial-based `by-id` link. Set a serial on the virtualization side first. diff --git a/docs/en/solutions/TopoLVM_thin_provisioning_over-provisioned_capacity_and_pod_IOPS_limits_on_ACP.md b/docs/en/solutions/TopoLVM_thin_provisioning_over-provisioned_capacity_and_pod_IOPS_limits_on_ACP.md new file mode 100644 index 000000000..0f7e25829 --- /dev/null +++ b/docs/en/solutions/TopoLVM_thin_provisioning_over-provisioned_capacity_and_pod_IOPS_limits_on_ACP.md @@ -0,0 +1,121 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x +--- + +# TopoLVM thin provisioning, over-provisioned capacity, and pod IOPS limits on ACP + +## Issue + +Questions and symptoms around TopoLVM thin provisioning on Alauda Container Platform 4.2: + +- Can TopoLVM create thin-provisioned logical volumes, and how is that turned on? +- Can a pod's IOPS be capped with cgroups v2 and an `IOLimit` custom resource? +- After a thin device class was created, the StorageClass shows about twice the physical capacity (for example 6.4T while only a 3.5T disk is used). +- A PVC on a thin device class cannot be created (`ProvisioningFailed`), or an application writing to a thin volume fails with `No space left on device`, although the console shows plenty of capacity and a single disk looks big enough. + +Thin provisioning here means LVM thin pools that TopoLVM is configured to use. It is not VDO deduplication under the volume group. + +## Root Cause + +### Cause 1: thin logical volumes are chosen with Provision Config when the TopolvmCluster is deployed + +Thin-provisioned LVM is supported on ACP 4.2. It is selected with **Provision Config** when the `TopolvmCluster` is deployed: the **Provision Type** field offers `Thin` and `Thick`. + +### Cause 2: pod IOPS limiting is not a supported TopoLVM feature + +Limiting a pod's IOPS with cgroups v2 and an `IOLimit` custom resource is not supported by TopoLVM on the platform. + +### Cause 3: capacity shows about double the disk size because the thin device class has overprovisionRatio 2 + +The device class has a `thinPoolConfig`, for example: + +```yaml +thinPoolConfig: + name: hdd-thin-pool + overprovisionRatio: 2 + sizePercent: 90 +``` + +On the node, `lvs` shows the thin pool LV (`hdd-thin-pool`) in the volume group `hdd-`. With `overprovisionRatio: 2`, the capacity TopoLVM publishes is twice the physical pool size, so the StorageClass total is not wrong: it is thin over-provisioning. + +### Cause 4: the thin pool on the selected node is full, so new volumes and writes fail + +The published (over-provisioned) capacity is not physical space. What can actually be written depends on the thin pool's `Data%` and `Meta%` on the node the PVC is scheduled to. When the thin pool reaches its data limit, LVM cannot create new thin volumes there (`ProvisioningFailed` on the PVC), and writes to existing thin volumes in that pool fail (for example `No space left on device` in the application). The free space `vgs` shows outside the pool (the part not covered by `sizePercent`) cannot be used for thin volumes. For example: a pool of about 190g, three thin volumes with a combined virtual size of 240g, and `Data%` at `100.00`. + +## Resolution + +### Cause 1: thin logical volumes are chosen with Provision Config when the TopolvmCluster is deployed + +In the console, open **Storage > Local Storage > Config local storage**, step **Create Cluster**, Storage Plan `TopoLVM`, namespace `nativestor-system`. In the storage entry, open **Provision Config** and set **Provision Type** to `Thin`. These parameters then appear: + +| Parameter | Default | +|---|---| +| Over Provision Ratio | `2` | +| Size Percent | `90` | +| Metadata Size Calculation Policy | `Host` (other value: `Static`) | +| Chunk Size | none given | + +Adjust them or keep the defaults. Thin is chosen at deploy time. No YAML field, ModulePlugin key, or procedure is documented for switching an existing thick device class to thin, or a thin class to thick; treat that as an open question for support rather than assuming either answer. + +After a thin deploy, the volume group TopoLVM created contains a thin pool, and new volumes are allocated from it: + +```bash +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm -o yaml +lvs -a -o lv_name,vg_name,lv_attr,pool_lv +``` + +### Cause 2: pod IOPS limiting is not a supported TopoLVM feature + +There is no supported substitute: no `IOLimit` object, cgroup setting, or StorageClass parameter provides per-pod IOPS limits for TopoLVM volumes. + +### Cause 3: capacity shows about double the disk size because the thin device class has overprovisionRatio 2 + +No repair is needed when the over-provisioning is intended. Physical writes are still limited by the thin pool, so monitor it on each node: + +```bash +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm -o yaml | grep -B 2 -A 4 thinPoolConfig +lvs -o lv_name,vg_name,lv_size,data_percent,metadata_percent +``` + +### Cause 4: the thin pool on the selected node is full, so new volumes and writes fail + +1. Find the node the PVC was scheduled to, and check the thin pool there: + +```bash +kubectl get pvc -n -o jsonpath='{.metadata.annotations.volume\.kubernetes\.io/selected-node}{"\n"}' +lvs -o lv_name,vg_name,lv_size,lv_attr,data_percent,metadata_percent,pool_lv +vgs -o vg_name,vg_size,vg_free +``` + +A thin pool with `Data%` or `Meta%` at or near 100 confirms this cause. + +2. Map the thin volumes to PVs and PVCs, and decide which data can go: + +```bash +kubectl get logicalvolumes.topolvm.cybozu.com +kubectl get pv | grep topolvm +``` + +3. Free or add physical space: + - Delete PVCs on that node that are confirmed unused. Space returns only if the reclaim policy is `Delete`; with `Retain`, see [TopoLVM capacity stays full after PVCs are deleted on ACP](./TopoLVM_capacity_stays_full_after_PVCs_are_deleted_on_ACP.md). + - Or add a bare disk to that node's device class (**Storage Management > Local Storage > Add Storage Device**) so new PVCs can be created. A disk added this way can become a separate volume group and then does not help existing thin volumes. + +> **Warning:** Do not run `lvremove` on a thin volume that backs a Bound PV. Its data cannot be recovered, and the API objects stay behind. + +## Diagnostic Steps + +```bash +kubectl -n nativestor-system get topolvmclusters.topolvm.cybozu.com topolvm -o yaml +kubectl -n nativestor-system get pods +lvs -o lv_name,vg_name,lv_size,lv_attr,data_percent,metadata_percent,pool_lv +``` + +- No `thinPoolConfig` on the device class: the cluster was not deployed with thin provisioning (Cause 1). +- An `IOLimit` object in the namespace does not change Cause 2. +- Capacity about twice the disk size on a class with `overprovisionRatio: 2`: Cause 3. +- `ProvisioningFailed` or `No space left on device` with the thin pool `Data%` near 100 on the selected node: Cause 4. diff --git a/docs/en/solutions/Traefik_gateway_X-Forwarded-For_missing_ports_80_and_443_underscore_headers_weighted_routes_and_route_export_on_ACP.md b/docs/en/solutions/Traefik_gateway_X-Forwarded-For_missing_ports_80_and_443_underscore_headers_weighted_routes_and_route_export_on_ACP.md new file mode 100644 index 000000000..f26e8813b --- /dev/null +++ b/docs/en/solutions/Traefik_gateway_X-Forwarded-For_missing_ports_80_and_443_underscore_headers_weighted_routes_and_route_export_on_ACP.md @@ -0,0 +1,196 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" +--- + +# Traefik gateway: X-Forwarded-For missing, ports 80 and 443, underscore headers, weighted routes and route export on ACP + +## Issue + +The customized Traefik gateway (`traefik.traefik.alauda.io`), delivered for specific projects, is in use, and one of these comes up: + +- Traefik cannot get the client IP from the request headers. The access log for one entrypoint shows `ClientHost` `127.0.0.6` and `request_X-Real-Ip` `127.0.0.6` and no `X-Forwarded-For` value, although the client sent one. The response carries `server: istio-envoy`. +- Does Traefik drop request headers whose names contain an underscore? +- Can Traefik's default ports 80 and 443 be closed, or opened again? +- Where is a `TraefikService` maintained? The console has no page for it, and the goal is a gray release (weighted split between a stable and a new version) without the service mesh module. +- Can routes be exported and imported between environments by copying the IngressRoute YAML from the console? + +## Root Cause + +### Cause 1: the entrypoint has no forwardedHeaders trust + +Traefik trusts `X-Forwarded-*` headers only from addresses listed in `forwardedHeaders.trustedIPs`, or from every source when `forwardedHeaders.insecure` is true, and the setting is per entrypoint. Requests reached Traefik through an Istio sidecar, so the source Traefik saw was `127.0.0.6` (the address the sidecar uses for inbound traffic to the application container). The custom entrypoint that received them (for example `mytcp` on ACP 4.2.2) had no `forwardedHeaders` setting, so the client's `X-Forwarded-For` did not appear in the log, while other ports could keep it. + +### Cause 2: Traefik accepts header names with underscores by default + +Traefik supports request headers whose names contain underscores by default; no extra configuration is needed. This differs from Envoy Gateway, which rejects such headers by default. + +### Cause 3: ports 80 and 443 are the web and websecure entries of the instance + +Ports 80 and 443 are the `web` and `websecure` entries under `spec.ports` of the Traefik instance, and their `expose.default` field controls whether they are published. + +### Cause 4: TraefikService has no UI, and a weighted IngressRoute does the gray release + +Traefik is a project-specific customization, and the console does not support `TraefikService`. A gray release without the service mesh module is done with an `IngressRoute` whose route lists two backend Services with weights. + +### Cause 5: copying the IngressRoute YAML is a valid way to move routes + +Opening a route under **Resource Management > IngressRoute**, copying its YAML, cleaning it up and creating it in another environment is a viable way to export and import Traefik routes, provided the target environment is compatible and already holds everything the route depends on. + +## Resolution + +### Cause 1: the entrypoint has no forwardedHeaders trust + +Only when the access log of a specific entrypoint lacks the `X-Forwarded-For` value the request carried. Add `forwardedHeaders.insecure: true` to that port of the Traefik instance, at the same level as `expose`, `exposedPort` and `port`: + +```yaml +spec: + ports: + mytcp: + expose: + default: true + exposedPort: 9000 + forwardedHeaders: + insecure: true + port: 19090 +``` + +Keep your own entry name and ports; only add the `forwardedHeaders` block. Traefik also accepts `forwardedHeaders.trustedIPs` with a list of trusted source CIDRs instead of `insecure`. + +> **Warning:** `insecure: true` makes the entrypoint trust `X-Forwarded-For` from any source, so a client that reaches the port directly can set any address, and backends or allowlists that read the header will believe it. The Traefik Pods can roll, dropping connections. Use a maintenance window and save the instance first; removing `forwardedHeaders` or applying the saved file is the rollback. + +```bash +kubectl get traefik.traefik.alauda.io -n -o yaml > /tmp/traefik-before.yaml +kubectl edit traefik.traefik.alauda.io -n +kubectl rollout status deploy/ -n +kubectl logs -n deploy/ --since=5m | grep '' +``` + +The new log entry should show the `X-Forwarded-For` value. Removing the Istio sidecar from the Traefik Pod, so that Traefik is the only proxy on the hop, is a separate option; the `forwardedHeaders` change alone is enough. + +### Cause 2: Traefik accepts header names with underscores by default + +No change is needed. If such a header really goes missing behind Traefik, test the hop (Diagnostic Steps) before changing any setting. + +### Cause 3: ports 80 and 443 are the web and websecure entries of the instance + +Set `spec.ports.web.expose.default` and `spec.ports.websecure.expose.default` to `false` to close them, or `true` to open them again. Example that keeps a custom port and closes both: + +```yaml +spec: + ports: + mytcp: + expose: + default: true + exposedPort: 9000 + port: 9000 + web: + expose: + default: false + websecure: + expose: + default: false +``` + +Leave any `forwardedHeaders` settings that the entries already have. + +> **Warning:** Clients that use ports 80 or 443 on this instance lose access as soon as the change applies. Confirm no route still needs `web` or `websecure`, and save the instance; setting `expose.default` back to `true` is the rollback. + +```bash +kubectl edit traefik.traefik.alauda.io -n +kubectl get svc -n -o custom-columns='NAME:.metadata.name,TYPE:.spec.type,PORTS:.spec.ports[*].port' +``` + +### Cause 4: TraefikService has no UI, and a weighted IngressRoute does the gray release + +Give the stable (v1) and gray (v2) versions their own Services, then create a weighted `IngressRoute`. This example sends 90 percent of `app.example.com` to `app-v1` and 10 percent to `app-v2` on entrypoint `web` (use `websecure` for HTTPS, or another exposed entrypoint if `web` is closed): + +```yaml +apiVersion: traefik.io/v1alpha1 +kind: IngressRoute +metadata: + name: app-ingress-gray + namespace: +spec: + entryPoints: + - web + routes: + - match: Host(`app.example.com`) + kind: Rule + services: + - name: app-v1 + kind: Service + port: 80 + weight: 90 + - name: app-v2 + kind: Service + port: 80 + weight: 10 +``` + +```bash +kubectl get svc app-v1 app-v2 -n +kubectl apply -f /tmp/app-ingress-gray.yaml +kubectl get ingressroute app-ingress-gray -n -o yaml +``` + +Change the two `weight` values and apply again to shift traffic to v2. + +### Cause 5: copying the IngressRoute YAML is a valid way to move routes + +1. Check that the target environment is ready: + - Traefik is installed with the matching IngressRoute CRD, and the `apiVersion` is compatible (for example `traefik.io/v1alpha1`). + - The backend Services, Middlewares, TLS Secrets and TLSOption/TLSStore resources that the route references exist there. + - Environment-specific settings are prepared: entrypoints, domain DNS, the VIP or LoadBalancer Service, and the namespaces the Traefik instance watches. + + ```bash + kubectl get crd | grep -i traefik + kubectl get traefik.traefik.alauda.io -A + kubectl get svc,secret -n + ``` + +2. Export the route, remove the server-generated fields (`metadata.uid`, `metadata.resourceVersion`, `metadata.creationTimestamp`, `metadata.generation`, `metadata.managedFields` and `status`), and adjust the domain, certificate, Services and entrypoints for the target. + + ```bash + kubectl get ingressroute -n -o yaml > /tmp/.yaml + vi /tmp/.yaml + ``` + +3. Create it in the target environment and test the domain: + + ```bash + kubectl apply -f /tmp/.yaml + kubectl get ingressroute -n + ``` + +## Diagnostic Steps + +1. Find the instance, its Pods and the entrypoint that handles the request: + + ```bash + kubectl get traefik.traefik.alauda.io -A + kubectl get traefik.traefik.alauda.io -n -o yaml | sed -n '/^ ports:/,/^ [a-z]/p' + kubectl get pods -n -o wide + ``` + +2. For a missing client address, send a request with a known header and read its log line: + + ```bash + curl -v -H 'X-Forwarded-For: 203.0.113.10' http://:/ + kubectl logs -n | grep '' + kubectl get pod -n -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{end}' + ``` + + `ClientHost` and `request_X-Real-Ip` of `127.0.0.6`, no `X-Forwarded-For` value, and no `forwardedHeaders` on that port: Cause 1. An `istio-proxy` container means a sidecar sits in front of Traefik. + +3. For an underscore header, send it through Traefik to a backend that logs headers and compare with a request straight to the Service: + + ```bash + curl -v -H 'x_custom_header: test' http://:/ + ``` + +4. For ports 80 and 443, read `spec.ports.web` and `spec.ports.websecure` and the ports on the Traefik Service. For gray release or route export, confirm the CRD and the Services with `kubectl get ingressroute -A` and `kubectl get svc -n `. diff --git a/docs/en/solutions/Update_a_platform_domain_certificate_stored_in_a_cpaas-system_Secret_on_ACP.md b/docs/en/solutions/Update_a_platform_domain_certificate_stored_in_a_cpaas-system_Secret_on_ACP.md new file mode 100644 index 000000000..e49da63f3 --- /dev/null +++ b/docs/en/solutions/Update_a_platform_domain_certificate_stored_in_a_cpaas-system_Secret_on_ACP.md @@ -0,0 +1,162 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Update a platform domain certificate stored in a cpaas-system Secret on ACP + +## Issue + +One of the following applies to a certificate bound to a domain under **Platform Management > Network Management**: + +- A certificate under **Network Management > Certificates** shows a warning icon with the tooltip `证书不在有效期内` (certificate not within its validity period), and a new certificate for the same domain is at hand. +- A certificate management system has to update domain certificates automatically. The questions are where the certificates bound to domains under **Network Management > Domains** are stored, whether they are ConfigMaps, and whether the public key and private key shown in the **Update Domain** dialog can be changed through an API. +- On ACP 3.18.1 the GitLab, Harbor, Nexus, or Jenkins certificate was replaced in the Secret that the toolchain instance names, and after some time the Secret is back to the old certificate. +- The platform domain certificate is held in Secret `cpaas-system/legal-domain.tls` and has to be renewed. + +This article covers domain certificates. The certificate of the platform access address itself (`dex.tls`) is covered in "Replace the platform access certificate in dex.tls on ACP". Business certificates in project Secrets are covered in "Business TLS certificate not renewed, not reloaded by ingress-nginx, or rejected with a format error on import on ACP". + +## Root Cause + +### Cause 1: domain certificates are Secrets in cpaas-system, updated through the Kubernetes Secret API + +All platform domain certificates are stored as Secrets in namespace `cpaas-system` with label `global-secret=true`, not in ConfigMaps. A certificate imported when a domain is created carries the annotation `cpaas.io/domain-name-ref` with the domain name. A certificate assigned to all projects also carries the label `project.cpaas.io/ALL_ALL: "true"`. The certificate data is in `data.tls.crt` and `data.tls.key`. + +The platform has no separate API for modifying a domain certificate. The console itself updates it with a `PATCH` to `/api/v1/namespaces/cpaas-system/secrets/`. An external certificate management system does the same through the native Kubernetes Secret API, patching `tls.crt` and `tls.key` (base64) of that Secret. Editing those two fields with `kubectl edit` or `kubectl apply` also updates the certificate: the Certificates page then shows the new validity on the same Secret, still associated with the same domain, and the warning icon disappears. Certificates are changed one Secret at a time; there is no batch replacement. + +A platform API token for automation is created in the console: open the avatar menu in the top-right corner, select **Personal Info**, open the **API Token** tab, and use **Add API Token**. + +### Cause 2: on ACP 3.18.1 a toolchain TLS Secret edit is reverted because the domain is managed on the Domains page + +On ACP 3.18.1 the access domains of the DevOps v3 toolchains (GitLab, Harbor, Nexus, Jenkins) are exposed through the platform ALB, and each toolchain instance names its TLS Secret in a `secretName` field. That Secret, of type `kubernetes.io/tls` in `cpaas-system`, carries the labels `global-secret: "true"` and `project.cpaas.io/ALL_ALL: "true"`. These labels mark it as a certificate managed by the platform's domain management (**Network Management > Domains**). The platform therefore writes the domain's certificate back over a direct edit of the Secret, and after a while every toolchain Secret shows the old certificate again. Changing the certificate on the **Domains** page makes the change stick. + +### Cause 3: the domain certificate in cpaas-system/legal-domain.tls is renewed by recreating the Secret and restarting global-alb2 + +On some installations, for example trial or demo environments, the platform domain certificate is stored in Secret `legal-domain.tls` in `cpaas-system`. It is renewed by backing up that Secret, deleting it, recreating it from the new certificate and key with `kubectl create secret tls`, and restarting the `global-alb2` component so that it serves the new certificate. + +## Resolution + +Before any change, verify the new pair: the two commands below must print the same hash, `notAfter` must be in the future, and the certificate must name the domain. + +```bash +openssl x509 -in tls.crt -noout -pubkey | openssl md5 +openssl pkey -in tls.key -pubout | openssl md5 +openssl x509 -in tls.crt -noout -dates -subject -ext subjectAltName +``` + +### Cause 1: domain certificates are Secrets in cpaas-system, updated through the Kubernetes Secret API + +1. On the global cluster, list the domain certificate Secrets and find the one for a domain: + + ```bash + kubectl get secrets -n cpaas-system -l global-secret=true + kubectl get secrets -n cpaas-system -l global-secret=true \ + -o go-template='{{range .items}}{{if eq (index .metadata.annotations "cpaas.io/domain-name-ref") ""}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' + ``` + +2. Back up the Secret and patch `tls.crt` and `tls.key`. + + > **Warning:** Patching the Secret changes the certificate served for that domain at once; a key that does not match the certificate, or a certificate for another name, breaks HTTPS for every service that uses it. Do this only after the pair check above succeeded. Rollback is `kubectl apply -f` of the backup. + + ```bash + umask 077 + kubectl get secret -n cpaas-system -o yaml > "$HOME/.yaml" + kubectl patch secret -n cpaas-system --type merge \ + -p "{\"data\":{\"tls.crt\":\"$(base64 -w 0 tls.crt)\",\"tls.key\":\"$(base64 -w 0 tls.key)\"}}" + kubectl get secret -n cpaas-system -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject + ``` + + From an automation system, send the same change as an HTTP `PATCH` to the Secret, with the API token from Cause 1. `${APISERVER}` is the Kubernetes API endpoint of the global cluster. The values must be base64-encoded on a single line, which `base64 -w 0` produces. + + ```bash + curl -X PATCH -H "Content-Type: application/strategic-merge-patch+json" \ + -H "Authorization: Bearer ${TOKEN}" --cacert /etc/kubernetes/pki/ca.crt \ + "${APISERVER}/api/v1/namespaces/cpaas-system/secrets/" \ + -d "{\"data\": {\"tls.crt\": \"$(base64 -w 0 tls.crt)\", \"tls.key\": \"$(base64 -w 0 tls.key)\"}}" + ``` + +3. Check the **Certificates** page: the validity is the new one and the `证书不在有效期内` icon is gone. If the Secret returns to the old certificate after a while, use Cause 2. + +### Cause 2: on ACP 3.18.1 a toolchain TLS Secret edit is reverted because the domain is managed on the Domains page + +1. Read the `secretName` field from the toolchain instance, then check the labels of that Secret and the certificate it holds: + + ```bash + kubectl get secret -n cpaas-system -o jsonpath='{.type}{"\n"}{.metadata.labels}{"\n"}' + kubectl get secret -n cpaas-system -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject -ext subjectAltName + ``` + + The labels `global-secret: "true"` and `project.cpaas.io/ALL_ALL: "true"` mean that the certificate is managed through **Network Management > Domains**. + +2. Update the certificate on the Domains page. + + > **Warning:** Updating the domain certificate changes the certificate every toolchain and service on that domain presents, at once. A certificate that does not cover the domain (for example the wildcard `*.`) or a key that does not match breaks HTTPS for all of them. Do this only after the pair check above. Keep the old certificate and key files; rollback is to update the domain again with them. + + 1. Open **Platform Management > Network Management > Domains** and choose the update action for the domain (for example `*.`). + 2. Keep the certificate toggle enabled. + 3. Paste or import the new public key (certificate) and private key. + 4. Click **Update**. + +3. Check the toolchain Secret again with step 1, and check the certificate served on each toolchain hostname: + + ```bash + echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -dates -subject + ``` + +### Cause 3: the domain certificate in cpaas-system/legal-domain.tls is renewed by recreating the Secret and restarting global-alb2 + +1. Print the current certificate and the new one: + + ```bash + kubectl get secret -n cpaas-system legal-domain.tls -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject -ext subjectAltName + kubectl get secret -n cpaas-system legal-domain.tls -o jsonpath='{.metadata.labels}{"\n"}{.metadata.annotations}{"\n"}' + openssl x509 -in .pem -noout -dates -subject -ext subjectAltName + ``` + +2. Recreate the Secret and restart `global-alb2`. + + > **Warning:** Between the delete and the create, the platform domain has no certificate in this Secret, and restarting `global-alb2` briefly interrupts traffic through the global ALB. Do this in a maintenance window and only when the domain certificate is in `cpaas-system/legal-domain.tls` and a verified new pair for the same domain is on this node. Rollback is `kubectl apply -f` of the backup, followed by the same restart. + + ```bash + umask 077 + kubectl get secret -n cpaas-system legal-domain.tls -o yaml > "$HOME/legal-domain.tls.yaml" + kubectl delete secret -n cpaas-system legal-domain.tls + kubectl create secret tls legal-domain.tls --cert=.pem --key=.key -n cpaas-system + kubectl get deploy -n cpaas-system | grep global-alb2 + kubectl rollout restart deployment -n cpaas-system global-alb2 + kubectl rollout status deployment -n cpaas-system global-alb2 + echo | openssl s_client -connect :443 -servername 2>/dev/null \ + | openssl x509 -noout -dates -subject + ``` + + `kubectl create secret tls` does not copy labels or annotations from the old Secret. If the domain does not pick up the new Secret, compare its labels and annotations with the backup and add the missing ones with `kubectl label` and `kubectl annotate`. If `kubectl get deploy` does not list `global-alb2`, restart the workload kind that `kubectl get pods -n cpaas-system | grep global-alb2` shows. + +## Diagnostic Steps + +1. Find the Secret behind the certificate and read its labels, annotations, and dates: + + ```bash + kubectl get secrets -n cpaas-system -l global-secret=true --show-labels + kubectl get secret -n -o jsonpath='{.metadata.labels}{"\n"}{.metadata.annotations}{"\n"}' + kubectl get secret -n -o jsonpath='{.data.tls\.crt}' \ + | base64 -d | openssl x509 -noout -dates -subject -issuer + ``` + +2. Branch: + + - Where a domain certificate is stored, how to update it from automation or `kubectl`, or the Certificates page shows `证书不在有效期内`: Cause 1. + - A toolchain TLS Secret with `global-secret: "true"` returns to the old certificate after a direct edit: Cause 2. + - The domain certificate is Secret `cpaas-system/legal-domain.tls`: Cause 3. + - The Secret already holds the new certificate but clients still receive the old one: check the ALB rule or an upstream load balancer in front of the platform, which can serve its own copy. diff --git a/docs/en/solutions/Upgrading_a_Data_Services_PostgreSQL_14_instance_to_PostgreSQL_16_on_ACP_4.2.md b/docs/en/solutions/Upgrading_a_Data_Services_PostgreSQL_14_instance_to_PostgreSQL_16_on_ACP_4.2.md new file mode 100644 index 000000000..c5baecb93 --- /dev/null +++ b/docs/en/solutions/Upgrading_a_Data_Services_PostgreSQL_14_instance_to_PostgreSQL_16_on_ACP_4.2.md @@ -0,0 +1,55 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Application Services +ProductsVersion: + - 4.2.x +--- + +# Upgrading a Data Services PostgreSQL 14 instance to PostgreSQL 16 on ACP 4.2 + +## Issue + +On ACP 4.2 (for example 4.2.3), Data Services offers PostgreSQL 14 and PostgreSQL 16 for new instances. An existing instance runs PostgreSQL 14 and has to move to PostgreSQL 16. The upgrade documentation only says that the PostgreSQL server version has to be upgraded manually, and the backup and restore documentation describes restoring into a new instance; neither gives the steps for moving an existing instance from 14 to 16. The question is whether updating the instance and selecting the new version is enough. + +## Root Cause + +### Cause 1: the major version is changed on the instance update page + +No separate migration procedure is needed for this path. On ACP 4.2, the update page of a PostgreSQL instance has a database version field that offers `PostgreSQL 14` and `PostgreSQL 16`. Updating the instance and switching that field from `PostgreSQL 14` to `PostgreSQL 16` upgrades the existing instance to 16. + +A PostgreSQL major version change rewrites the instance's system catalogs for the new version and requires the database server to restart, so it is a planned maintenance operation even though it is started from one form field. + +## Resolution + +### Cause 1 + +1. Check that the instance is healthy before the upgrade (Diagnostic Steps, step 1). Do not start a major version upgrade on an instance that is not `Running`. + +2. Take a backup of the instance and confirm that it completed. + + > **Warning:** A major version upgrade cannot be undone by selecting `PostgreSQL 14` again: a data directory upgraded to 16 cannot be opened by PostgreSQL 14. The instance is unavailable to clients while the server restarts on the new version. Precondition: a completed backup of the instance, a maintenance window, and applications that are tested against PostgreSQL 16. Rollback: restore the backup into a PostgreSQL 14 instance and point the applications at it. + +3. In the console, open the PostgreSQL instance, choose **Update**, and change the database version from `PostgreSQL 14` to `PostgreSQL 16`. Leave the other fields (replicas, CPU, memory, storage class) as they are, then confirm. + +4. Wait until the instance returns to `Running`, then verify the server version (Diagnostic Steps, step 2) and test the applications that use it. + +## Diagnostic Steps + +1. Check the current major version and status of the instances: + + ```bash + kubectl get postgresql -A \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,VER:.spec.postgresql.version,STATUS:.status.PostgresClusterStatus + ``` + + Before the upgrade the instance shows `VER` `14` and `STATUS` `Running`. After the update is submitted, `VER` shows `16`; wait until `STATUS` is `Running` again. A status such as `SyncFailed` or `UpdateFailed` means the change was not applied; inspect `kubectl describe postgresql -n ` before retrying. + +2. Confirm the version that the server actually runs: + + ```bash + kubectl exec -n -0 -- psql -U postgres -c 'SHOW server_version;' + ``` + + The output starts with `16`. A value starting with `14` means the server has not restarted on the new version yet. diff --git a/docs/en/solutions/User_group_member_gets_a_data_error_or_no_permissions_while_a_direct_role_binding_works_on_ACP.md b/docs/en/solutions/User_group_member_gets_a_data_error_or_no_permissions_while_a_direct_role_binding_works_on_ACP.md new file mode 100644 index 000000000..c242b0384 --- /dev/null +++ b/docs/en/solutions/User_group_member_gets_a_data_error_or_no_permissions_while_a_direct_role_binding_works_on_ACP.md @@ -0,0 +1,135 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.2.6" +--- + +# User group member gets a data error or no permissions while a direct role binding works on ACP + +## Issue + +A user (local, synchronized from LDAP, or from a third-party identity source) is a member of a user group, and the group is bound to a platform or namespace role, including platform administrator. The user can log in but cannot do what a user with the same role bound directly can do: + +- Batch delete of Pods on the container platform shows a data error and deletes nothing; the same batch delete succeeds for a local platform administrator. +- Creating a namespace, or opening one, reports insufficient permissions. Assigning the role directly to the account makes it work. +- "The user group is authorized, but the user has no permissions after logging in": any role granted through that group, including a custom namespace role, is missing. +- Users added to a group with a role association cannot open the related namespaces, while a newly created group with a role association works. +- Pipeline (Tekton) pages do not load or query for a user whose permissions come only from a group, while every other feature works for that user. + +The failure is authorization through the group, not a bad password. + +## Root Cause + +### Cause 1: metis does not apply group authorization to batch delete + +`auth-controller2` logs show the batch-delete request intercepted by a webhook, and metis validation rejects it. Metis lacks group authorization: a permission that exists on the group is not applied when metis evaluates the user. A directly bound role does not take this path. This is a platform defect; the fix is planned for 4.2.6. + +### Cause 2: group role permissions do not take effect for namespace creation on 3.18 and 4.1.0 + +On 3.18, and on 4.1.0, permissions granted through a user group did not take effect for namespace creation. On 4.1.0 this affects LDAP users and local users in a local group alike. It is fixed starting in 4.2. + +### Cause 3: the group's name does not start with `g-` + +Authorization discards every permission of a group whose Kubernetes object name lacks the `g-` prefix. A group created in the web console gets a generated `g-` name; a group without the prefix was created another way, for example by applying an object with a hand-chosen name. The role content is irrelevant. + +### Cause 4: the group was never actually associated with the role + +A component problem while the group and its role association were first created left the role unbound, although the console showed the association. In the browser, `selfsubjectaccessreviews` returns `201` while the namespace page's requests return `403` (the namespace itself, `resourcequota`, `pod`, `workload`, `application`, `configmaps?labelSelector=features.cpaas.io%2Ftype%3DCustomResourceLimitation`, the `features` queries for `prometheus` and `victoriametrics`, and `log-agent`), and `custom-navconfig-acp` returns `404`. A new group with the role associated works, which shows the original association was never stored. The exact mechanism was not confirmed. + +### Cause 5: the pipeline Hub API of Tekton operator 4.10 does not honor group bindings + +The group's UserBindings are `Success` and synced to all clusters, and the user has every other feature, but the pipeline pages fail. The pipeline Hub API in Tekton operator (Alauda DevOps Pipelines) 4.10 does not support authorization through user-group bindings, so permissions granted only through a group are ignored by the pipeline pages. Directly bound roles are honored. Group authorization for pipelines needs Alauda DevOps Pipelines 4.14 or later together with Alauda Artifact Hub Shim v1.0.0, which are compatible only with platform v4.4 and v4.3.3. + +## Resolution + +### Cause 1: metis does not apply group authorization to batch delete + +1. Upgrade to the release that contains the metis group-authorization fix (planned for 4.2.6); check the release notes of the target version. +2. Until then, bind the role directly to the affected user in Platform Management > User Management, the same way the working local account is bound, and retry the batch delete. Rollback: remove that direct binding; the group binding is unchanged. + +### Cause 2: group role permissions do not take effect for namespace creation on 3.18 and 4.1.0 + +1. Upgrade to 4.2 or later, then create the namespace again as the group user without a direct binding. +2. While the upgrade is scheduled, a direct binding of the same role to the user also makes namespace creation succeed. Rollback: delete that direct binding. + +### Cause 3: the group's name does not start with `g-` + +An object name cannot be renamed, so the fix is a new group. + +1. Export the old group and its bindings: + + ```bash + kubectl get groups.auth.alauda.io + kubectl get groups.auth.alauda.io -o yaml > old-group.before.yaml + kubectl get userbinding -o yaml > userbindings.before.yaml + ``` + +2. In Platform Management > User Management, create a new user group in the web console and confirm its name starts with `g-`. +3. Import the affected users, and bind the role the old group had, with the same project or namespace scope. +4. Log in as an affected user and confirm the permissions. +5. Remove the old group. + + > **Warning:** Deleting the old group removes it and its bindings for every member; members not moved to the new group keep nothing from it (they had nothing in effect while this cause applied). Only after the new `g-` group is verified. Rollback: `kubectl apply -f old-group.before.yaml` and recreate its bindings from `userbindings.before.yaml`; a restored group still lacks the prefix. + +### Cause 4: the group was never actually associated with the role + +1. Save the original group's bindings so a recurrence can be compared: + + ```bash + kubectl get userbinding -o yaml | grep -n + kubectl get userbinding -o yaml > userbinding.before.yaml + ``` + +2. Create a new user group, associate the role the original group should have had, add the users, and confirm access. +3. Retire the original group after the new one is verified. If the problem reappears, collect the full `auth-controller2` log and the UserBinding YAML: + + ```bash + kubectl get pods -A | grep auth-controller2 + kubectl logs -n > auth-controller2.log + ``` + +### Cause 5: the pipeline Hub API of Tekton operator 4.10 does not honor group bindings + +1. Workaround: grant the role directly to each user who needs the pipeline pages, instead of only through the group. Rollback: remove the direct binding. +2. Permanent fix: upgrade Alauda DevOps Pipelines to 4.14 or later and deploy Alauda Artifact Hub Shim v1.0.0. Because these require platform v4.4 or v4.3.3, upgrade the platform first if it is on another version. + +## Diagnostic Steps + +1. Compare the group user with a directly bound user: + + ```bash + kubectl auth can-i create namespaces --as= + kubectl auth can-i create namespaces --as= + kubectl auth can-i get pods -n --as= + ``` + + `auth can-i` answers Kubernetes RBAC only. A `yes` together with a console data error on batch delete still fits Cause 1, because metis rejects the call after RBAC. + +2. Check the group name and its bindings: + + ```bash + kubectl get groups.auth.alauda.io + kubectl get userbinding | grep + kubectl get userbinding -o jsonpath='{.status}{"\n"}' + ``` + +3. For batch delete, read the intercepting component: + + ```bash + kubectl logs -n --tail=400 + ``` + +| What you see | Cause | +| --- | --- | +| Batch delete data error, local user works, `auth-controller2` log shows the metis webhook, 4.x before 4.2.6 | 1 | +| Namespace creation denied for group users, direct role works, platform 3.18 or 4.1 | 2 | +| No permission at all through one group, group name lacks the `g-` prefix | 3 | +| Group name has `g-`, namespace page requests return `403` while `selfsubjectaccessreviews` returns `201`, a new group works | 4 | +| Only the pipeline pages fail for a group-only user, group UserBindings `Success`, Tekton operator 4.10 | 5 | diff --git a/docs/en/solutions/Vector_log_forwarding_to_external_Kafka_or_OpenTelemetry_fails_on_ACP.md b/docs/en/solutions/Vector_log_forwarding_to_external_Kafka_or_OpenTelemetry_fails_on_ACP.md new file mode 100644 index 000000000..92def0847 --- /dev/null +++ b/docs/en/solutions/Vector_log_forwarding_to_external_Kafka_or_OpenTelemetry_fails_on_ACP.md @@ -0,0 +1,240 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# Vector log forwarding to external Kafka or OpenTelemetry fails on ACP + +## Issue + +Vector forwards platform logs to a target outside the platform, such as an external Kafka or an OpenTelemetry (OTLP) endpoint, and one of the following happens: + +- The Vector Kafka sink fails SASL authentication (for example the broker answers with a password or credentials error), although the user, the password and the permissions are correct. +- Vector exits immediately with status 78 and `No provider for SASL mechanism GSSAPI` when the destination Kafka uses Kerberos. +- The external Kafka topic is auto-created but stays empty. +- The platform log page goes blank right after forwarding to Kafka was configured, while the external Kafka receives logs. +- Platform log search fails after compute nodes were added, on a cluster that uses Vector forwarding. +- Ingress logs need to be forwarded to an OTLP endpoint. + +The in-cluster log pipeline can be healthy while the forwarding path is not. Vector runs as the `vector` container of the `razor` pod in `cpaas-system`. Its configuration is in Secret `cpaas-vector` (key `vector.yaml`), and changes to it may be carried by a resource patch (`rpch`). + +## Root Cause + +### Cause 1: The Vector Kafka sink fails SASL authentication because the sink enables TLS and the broker does not + +Vector forwards logs to a Kafka with authentication enabled, and authentication fails. The Kafka user has full permissions, the username and password are correct, and TLS is not enabled on the Kafka listener. The sink has `sasl.enabled: true` with `mechanism: PLAIN`. It also has `tls.enabled: true` with `verify_certificate: false`, `verify_hostname: false`, and `librdkafka_options` `"enable.ssl.certificate.verification": "false"`. + +The sink speaks TLS to a plaintext listener, so the SASL exchange never completes, and the failure looks like an authentication error. Setting `tls.enabled: false` in the sink fixes it, and forwarding returns to normal. + +### Cause 2: The Vector librdkafka build has no GSSAPI, so a Kerberos Kafka sink exits with status 78 + +Vector loads its configuration and exits with status 78. The error is: + +```text +Sink "output_logs_kafka": creating kafka producer failed: Client creation error: No provider for SASL mechanism GSSAPI: recompile librdkafka with libsasl2 or openssl support. Current build options: PLAIN SASL_SCRAM OAUTHBEARER. +``` + +A warning that transform `dispatcher_unmatched` has no consumers may appear first, and it is not the failure. The librdkafka in the Vector image is built with `PLAIN`, `SASL_SCRAM` and `OAUTHBEARER` only. The SASL mechanism `GSSAPI` (Kerberos) is not in this build, so the Kafka producer is not created. No setting enables GSSAPI in this image. + +### Cause 3: The external Kafka topic is created and stays empty because the Vector condition matches only one cluster + +The log storage is centralized ClickHouse on the global cluster. Platform log queries work, `razor` is `Running`, and Vector forwards to an external Kafka. Kafka auto-created the topic, which proves connectivity, but the topic holds no logs. The forwarding configuration's `condition` filters events by `.cluster`, and as written it does not cover the clusters whose logs are wanted. Rewriting it as an OR over all those cluster names, `(.cluster == "" || .cluster == "")`, delivers logs from all of them. + +### Cause 4: The platform log page is empty after ClickHouse-to-Kafka forwarding was configured, while the external Kafka receives logs + +Logs arrive continuously in the external Kafka, so collection and forwarding work, but the platform log page shows nothing from ClickHouse. `razor` logs no error, ClickHouse runs well below its limits, and restarting `razor` changes nothing. + +The applied Vector configuration kept the `clickhouse` sink (`endpoint: "http://cpaas-clickhouse:8123"`, `database: observability`, `table: log_kubernetes_all`, `auth.strategy: "basic"`, `skip_unknown_fields: true`, a disk buffer). It added an `output_logs_kafka` sink of `type: kafka`. A special character somewhere in that configuration is suspected but has not been confirmed. Deleting the `rpch` that carried the configuration restores the default, and logs display again. Re-applying the configuration carefully keeps them displaying. + +### Cause 5: Log search fails after new nodes until Vector forwarding is disabled and enabled again + +Log search from the platform fails after compute nodes are added (for example 10 nodes) on a cluster that uses Vector forwarding. The forwarding path is what broke. Disabling forwarding and then enabling it again makes queries return. No deeper defect has been identified. + +### Cause 6: Forwarding ingress logs to an OpenTelemetry endpoint + +Vector can forward ingress logs to an OTLP endpoint. The working setup reshapes each event into OTLP JSON (`resourceLogs`) with a `remap` transform. It sends the result through an `opentelemetry` sink over HTTP POST to the endpoint's `/v1/logs` path, with JSON encoding, `content-type: application/json`, and the TLS CA file. A wrong path on the endpoint side makes the delivery fail. + +## Resolution + +### Cause 1: The Vector Kafka sink fails SASL authentication because the sink enables TLS and the broker does not + +1. Find out which security protocol the broker exposes on the port the sink uses. Ask the Kafka owner, or read the broker's `listeners` and `listener.security.protocol.map`. Also note which SASL mechanism the listener enables (`PLAIN`, `SCRAM-SHA-256` or `SCRAM-SHA-512`). + +2. Read the sink: + + ```bash + kubectl -n cpaas-system get secret cpaas-vector -o jsonpath='{.data.vector\.yaml}' | base64 -d | grep -n -A20 'type: kafka' + ``` + + `tls.enabled: true` against a `SASL_PLAINTEXT` listener matches this cause. The sink's `sasl.mechanism` must also be the mechanism that the listener enables, in the same spelling. + +3. Correct the sink in the Vector configuration that holds it. Save the current configuration first. + + > **Warning:** Changing the Vector configuration restarts the forwarding pipeline, and a wrong sink can stop forwarding or the platform's own ClickHouse sink. Precondition: the listener protocol and mechanism are confirmed. Rollback: re-apply the saved configuration. + + ```yaml + sinks: + : + type: kafka + inputs: + - + bootstrap_servers: : + topic: + encoding: + codec: json + sasl: + enabled: true + mechanism: PLAIN + username: + password: + tls: + enabled: false + ``` + + Remove the `librdkafka_options` SSL entries that were only there for the TLS attempt. Validate the file with `vector validate` in the `vector` container before you apply it. + +4. Check that the topic receives records. + +### Cause 2: The Vector librdkafka build has no GSSAPI, so a Kerberos Kafka sink exits with status 78 + +1. Confirm the signature: + + ```bash + kubectl -n cpaas-system logs -c vector --tail=30 + ``` + + The `GSSAPI` line and exit status 78 match this cause. + +2. This Vector image cannot use Kerberos to authenticate to Kafka. Use one of the mechanisms the build supports (`PLAIN`, `SCRAM`, or `OAUTHBEARER`) on a listener of the destination Kafka, and configure the sink with it. + +### Cause 3: The external Kafka topic is created and stays empty because the Vector condition matches only one cluster + +1. Find the `condition` in the Kafka forwarding configuration: + + ```bash + kubectl get rpch -A | grep -Ei 'vector|razor|clickhouse' + kubectl -n cpaas-system get secret cpaas-vector -o jsonpath='{.data.vector\.yaml}' | base64 -d | grep -n -B2 -A4 'condition' + ``` + +2. Widen the condition. + + > **Warning:** Widening the condition sends more clusters' logs to the external Kafka, which adds traffic and storage on the external side. Nothing changes in the platform storage. Precondition: only when the topic exists, stays empty, and the condition names fewer clusters than you intend to forward. Rollback: restore the saved configuration and restart `razor`. + + Write the condition as one expression that names every cluster to forward: + + ```yaml + condition: '(.cluster == "" || .cluster == "")' + ``` + + Apply the configuration the same way it was created, then restart `razor`. + +3. Read a few records from the topic. + +### Cause 4: The platform log page is empty after ClickHouse-to-Kafka forwarding was configured, while the external Kafka receives logs + +1. Save the forwarding patch: + + ```bash + kubectl get rpch -A | grep -Ei 'vector|razor|clickhouse' + kubectl get rpch -o yaml > /tmp/forwarding-rpch.yaml + ``` + +2. Delete it. + + > **Warning:** Deleting the forwarding `rpch` stops forwarding to the external Kafka until you apply it again, and returns the Vector pipeline to the default. Precondition: only when the external Kafka receives logs, the platform page is empty, and `razor` and ClickHouse are healthy. Rollback: `kubectl apply -f /tmp/forwarding-rpch.yaml`. + + ```bash + kubectl delete rpch + ``` + +3. Confirm that the platform page shows current logs. + +4. Apply the forwarding configuration again. Keep the `clickhouse` sink next to the new `kafka` sink, and check the file for stray quotes, non-ASCII punctuation or other special characters. Confirm that both the page and the external topic receive logs. + +### Cause 5: Log search fails after new nodes until Vector forwarding is disabled and enabled again + +> **Warning:** Turning Vector forwarding off stops the copy to the external target until it is turned on again, so external consumers see a gap for that interval. Precondition: only when search failed after compute nodes were added and forwarding is in use. Rollback: enable forwarding again, which is also the second step of the fix. + +Disable Vector forwarding with the same switch that enabled it, then enable it again. Query a workload that is logging now. + +### Cause 6: Forwarding ingress logs to an OpenTelemetry endpoint + +Add a transform and a sink to the Vector configuration. `filter_workload_ingress` is the input name of the ingress filter in this example. Use the name that your configuration defines. + +```yaml +transforms: + remap_otlp_ingress: + type: remap + inputs: + - filter_workload_ingress + source: | + ts = to_unix_timestamp(timestamp(.timestamp) ?? now(), unit: "nanoseconds") + body = string(.message) ?? encode_json(.) + . = { + "resourceLogs": [{ + "resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "acp-ingress"}}]}, + "scopeLogs": [{ + "scope": {"name": "vector"}, + "logRecords": [{"timeUnixNano": ts, "body": {"stringValue": body}}] + }] + }] + } +sinks: + output_logs_ingress: + type: opentelemetry + inputs: + - remap_otlp_ingress + protocol: + type: http + uri: https:///v1/logs + method: post + encoding: + codec: json + framing: + method: newline_delimited + batch: + max_events: 1 + request: + headers: + content-type: application/json + tls: + ca_file: /var/ca-bundle.crt + buffer: + type: disk + max_size: +``` + +Set `max_size` to the disk buffer size in bytes (1 GiB is 1073741824). The `protocol` block takes the options of Vector's `http` sink. Check the key placement against the Vector version in the `vector` container (`vector --version`), and run `vector validate` on the file before you apply it. Make sure the URI ends in `/v1/logs`. + +## Diagnostic Steps + +1. Find the forwarding configuration and the Vector state: + + ```bash + kubectl get secret -A | grep -E 'cpaas-vector' + kubectl get rpch -A | grep -Ei 'vector|razor|clickhouse' + kubectl -n cpaas-system get pods | grep -Ei 'razor' + kubectl -n cpaas-system logs -c vector --tail=50 + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | SASL authentication fails with correct credentials, and the sink sets `tls.enabled: true` against a broker without TLS | Cause 1 | + | Exit status 78 and `No provider for SASL mechanism GSSAPI` | Cause 2 | + | An external topic that exists and stays empty | Cause 3 | + | An empty platform page right after Kafka forwarding was added, while the external topic fills | Cause 4 | + | Search that broke only after new nodes were added, on a cluster with forwarding | Cause 5 | + | OTLP deliveries fail | Cause 6, check the URI path and the CA file | + +3. If `ALAUDA_LOG_TOPIC` lag is growing and forwarding is not configured, the forwarding path is not involved. See [Log Kafka lag from Elasticsearch write hot spots, breaker trips, IK analyzer errors, or too few partitions on ACP](./Log_Kafka_lag_from_Elasticsearch_write_hot_spots_breaker_trips_IK_analyzer_errors_or_too_few_partitions_on_ACP.md). + +## Related Information + +- [Log query shows no logs, HTTP 500, slow long ranges, or an empty real-time log on ACP](./Log_query_shows_no_logs_HTTP_500_slow_long_ranges_or_an_empty_real-time_log_on_ACP.md) diff --git a/docs/en/solutions/Web_terminal_exec_window_debug_container_or_console_file_transfer_fails_on_ACP.md b/docs/en/solutions/Web_terminal_exec_window_debug_container_or_console_file_transfer_fails_on_ACP.md new file mode 100644 index 000000000..6c93f3e74 --- /dev/null +++ b/docs/en/solutions/Web_terminal_exec_window_debug_container_or_console_file_transfer_fails_on_ACP.md @@ -0,0 +1,204 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.3" + - "4.4" +--- + +# Web terminal, exec window, debug container, or console file transfer fails on ACP + +## Issue + +The console opens, but entering a Pod or a node from it does not work. Typical reports: + +- After an upgrade, users who could enter a Pod before cannot. If the **EXEC** entry is missing altogether, enable the `cpc-exec` feature gate as described in [EXEC and Debug entries missing from Pod and application pages until the cpc-exec feature gate is enabled on ACP](EXEC_and_Debug_entries_missing_from_Pod_and_application_pages_until_the_cpc-exec_feature_gate_is_enabled_on_ACP.md). +- The exec pop-up window opens as a blank white page. +- The terminal does not open, or **Reconnect** does nothing. +- After an upgrade the per-cluster CLI entry is gone. +- A debug container never finishes initializing. +- **File Transfer** is greyed out with a message that `tar` is missing, although `tar` is in the container. +- `nsenter` Pods in `Error` are found on every cluster during an inspection. +- Exec sessions drop often. +- `kubectl get ns` works on a master node but fails on worker nodes and in the Web CLI with `connection refused` on `localhost:8080`. + +The `web-terminal` Pods in `cpaas-system` serve the terminal sessions. + +## Root Cause + +### Cause 1: the platform is opened over HTTP and the exec window stays blank + +Running exec from the container page opens a pop-up window that stays blank white while the platform is accessed over `http://`. The blank window is attributed to the concurrency limit of HTTP access; opening the platform over HTTPS fixes it. (Browsers limit the number of HTTP/1.1 connections per host, while HTTP/2 over HTTPS multiplexes requests over one connection.) + +### Cause 2: the terminal access token expired + +The `web-terminal` log repeats `invalid access token: authentication head is invalid` and `failed open terminal for user` with an empty user, for the `global` cluster and for business clusters, while the Pod is running and its initial request to the API server succeeded. The terminal session was left open so long that its access token expired. + +### Cause 3: the disk that holds the web-terminal audit log is full + +**Reconnect** has no effect. The `web-terminal` Pods are `Running` with no restarts, and the log shows: + +```text +login termail audit failed. ... error: write /var/audit/cpaas-web-terminal-audit.log: no space left on device +failed open terminal for user: ... +``` + +`web-terminal` cannot write its audit log and refuses to open terminals. Earlier `etcdserver: request timed out` errors can appear in the same log. + +### Cause 4: the CLI entry moved to a single Web CLI icon + +After an upgrade the per-cluster CLI entry is no longer in the cluster list. By design, the CLI entry moved to a single **Web CLI** icon at the top right of the platform management view, served at `/console-webcli/`. In a working setup the **Alauda Container Platform Web Terminal** plugin is installed on the `global` cluster and the `web-cli` feature gate is enabled. The Web CLI lists the `global` nodes and the business clusters, and a node shell opens with `This is a shell for node, the host root directory is mounted to /host`. + +### Cause 5: the debug container image registry address was not replaced + +New Pods hang while the debug container initializes, and Pod events show the debug container image pull failing because the registry address in the debug image was not replaced. A debug container in an older Pod on the same cluster still works. Restarting `metis` on the affected cluster fixes new Pods. + +### Cause 6: file transfer detection fails although tar exists + +The **File Transfer** button is grey and the tooltip says `该容器内不存在tar工具,不支持文件传输功能` (the container has no tar tool; file transfer is not supported), while `tar` is present in the container and `kubectl cp` works. Setting `LANG=en_US.UTF-8` and `LC_ALL=en_US.UTF-8` on the container restores the detection; upgrading the platform to 3.16 or later is the lasting fix. + +### Cause 7: nsenter Pods left in Error by the platform CLI + +Using the platform CLI tool (the **CLI Tool** entry in the cluster's **Actions** menu, or the terminal icon in the top bar) creates an `nsenter` Pod. A Pod that ended in an error state got stuck during the later cleanup. Such Pods can be deleted and have no impact on the platform or on workloads. + +### Cause 8: exec sessions drop because of the network path + +The exec window has no fixed timeout: the front end sends a ping every 5 seconds, which keeps the connection open, and exec connects through to the API server. Frequent disconnects therefore point at the network path, for example a proxy with a short idle timeout. + +### Cause 9: kubectl on a worker node or in a Web CLI node shell has no kubeconfig + +A worker node has no kubeconfig by default, so `kubectl` falls back to `http://localhost:8080` and gets `connect: connection refused`. A node shell in the Web CLI starts inside the CLI container with the host root mounted at `/host`; until `chroot /host` is run, it cannot read the host's configuration. + +## Resolution + +### Cause 1: the platform is opened over HTTP and the exec window stays blank + +Check the scheme in the address bar. If it is `http://`, open the platform through its `https://` access address and run exec again. No platform object changes. + +### Cause 2: the terminal access token expired + +```bash +kubectl get pods -n cpaas-system | grep web-terminal +kubectl logs -n cpaas-system --tail=100 | grep -F 'invalid access token' +``` + +Close the old terminal session and open a new one, so that the page sends a current token. No Pod restart is required. If a brand-new session still logs `authentication head is invalid`, it is not this cause. + +### Cause 3: the disk that holds the web-terminal audit log is full + +1. Confirm the error and find the full filesystem: + + ```bash + kubectl get pods -n cpaas-system -o wide | grep -E 'web-terminal|node-terminal' + kubectl logs -n cpaas-system --tail=500 | grep -E 'no space left on device|failed open terminal|etcdserver' + kubectl exec -n cpaas-system -- df -h /var/audit + ``` + + If `/var/audit` is a host path, run `df -h` on the node that hosts the Pod. + +2. Free space on that path, or expand it, until usage is well below the kubelet eviction threshold (about 85%). + + > **Warning:** Deleting files is irreversible; copy anything you still need off the node first, and leave etcd data and container runtime storage in place. Do this only when the `web-terminal` log reports `no space left on device`. + +3. Recreate the Pod. Open CLI sessions drop until it is `Ready`. + + ```bash + kubectl delete pod -n cpaas-system + ``` + +### Cause 4: the CLI entry moved to a single Web CLI icon + +Use the **Web CLI** icon at the top right, or open `/console-webcli/`. If it does not load, check the **Alauda Container Platform Web Terminal** plugin on the `global` cluster and the `web-cli` feature gate. + +### Cause 5: the debug container image registry address was not replaced + +1. Confirm that the event is an image pull failure of the debug container: + + ```bash + kubectl describe pod -n | sed -n '/Events/,$p' + kubectl get pods -n cpaas-system | grep metis + ``` + +2. Restart `metis` on the affected cluster. + + > **Warning:** Debug image handling and the other functions served by `metis` are interrupted until the new Pod is `Ready`. Do this only when the debug container pull fails because the registry address was not replaced. The Pod is recreated from the same spec. + + ```bash + kubectl delete pod -n cpaas-system + ``` + +3. Create a new test Deployment and add a debug container; the pull succeeds. If the image is missing from the registry itself, restarting `metis` does not help. + +### Cause 6: file transfer detection fails although tar exists + +1. Confirm `tar` in the container: + + ```bash + kubectl exec -n -c -- tar --version + ``` + +2. Set `LANG=en_US.UTF-8` and `LC_ALL=en_US.UTF-8` on the container and let the workload roll. + + > **Warning:** Changing the workload's environment rolls its Pods. To roll back, remove the two variables. On platform versions earlier than 3.16 this is a workaround; upgrading to 3.16 or later is the lasting fix. + +3. Open **File Transfer** again. + +### Cause 7: nsenter Pods left in Error by the platform CLI + +```bash +kubectl get pods -A -o wide | grep nsenter | grep -v Running +``` + +> **Warning:** Deleting an `nsenter` Pod ends any CLI session still using it. Delete only Pods in `Error` or another terminated state. A new CLI session creates a new Pod. + +```bash +kubectl delete pod -n +``` + +### Cause 8: exec sessions drop because of the network path + +Exec into containers of different projects, and record whether all sessions drop or only some, and after how long. Drops after a fixed number of seconds point at a proxy idle timeout on the path, for example an external nginx `stream` listener with a short `proxy_timeout`; see Cause 5 of [Console returns 502 or does not open after installation or a VIP change on ACP](Console_returns_502_or_does_not_open_after_installation_or_a_VIP_change_on_ACP.md). + +### Cause 9: kubectl on a worker node or in a Web CLI node shell has no kubeconfig + +In a node shell of the Web CLI, enter the host first: + +```bash +chroot /host +kubectl get ns +``` + +On a worker node, point `kubectl` at a kubeconfig with `--kubeconfig ` or at `$HOME/.kube/config`. + +> **Warning:** A kubeconfig copied from a master node usually carries cluster-admin credentials, and anyone with access to the worker node can then administer the cluster. Use the narrowest role that works; delete the file to roll back. + +```bash +kubectl --kubeconfig get ns +``` + +## Diagnostic Steps + +```bash +kubectl get pods -n cpaas-system | grep -E 'web-terminal|metis' +kubectl logs -n cpaas-system --tail=80 +``` + +```text +exec pop-up blank white, platform opened over http:// -> Cause 1 +invalid access token: authentication head is invalid -> Cause 2 +write /var/audit/cpaas-web-terminal-audit.log: no space left on device -> Cause 3 +per-cluster CLI entry gone after an upgrade, /console-webcli/ works -> Cause 4 +debug container image pull fails, registry address not replaced -> Cause 5 +File Transfer grey, tar exists, kubectl cp works -> Cause 6 +nsenter Pods in Error on every cluster -> Cause 7 +exec sessions drop often -> Cause 8 +kubectl on a worker or in a Web CLI node shell: localhost:8080 refused -> Cause 9 +no EXEC entry at all -> the cpc-exec article linked in the Issue section +``` diff --git a/docs/en/solutions/Webhook_alert_notifications_fail_with_invalid_header_field_value_or_miss_a_required_custom_header_on_ACP.md b/docs/en/solutions/Webhook_alert_notifications_fail_with_invalid_header_field_value_or_miss_a_required_custom_header_on_ACP.md new file mode 100644 index 000000000..df8f59f4f --- /dev/null +++ b/docs/en/solutions/Webhook_alert_notifications_fail_with_invalid_header_field_value_or_miss_a_required_custom_header_on_ACP.md @@ -0,0 +1,161 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.3" + - "4.4" +--- + +# Webhook alert notifications fail with invalid header field value or miss a required custom header on ACP + +## Issue + +A webhook alert notification that needs custom HTTP request headers (for example an app ID and an access key for an SMS gateway or a third-party alert platform) does not reach the receiver: + +- Alert history shows the webhook notification with `"type":"Ready"`, `"status":"False"`, `"phase":"Failed"`, and `courier-api` logs `Failed to send webhook ... net/http: invalid header field value for "X-Open-Api-App-Access-Key"` (the header name differs per receiver). The receiver logs nothing. +- Or the receiver rejects the request because a required header is missing, for example: + + ```text + {"data":null,"status":"fail","msg":"Required request header 'appkey' for method parameter type String is not present","code":"400"} + ``` + + The platform side can still show the webhook send as successful. + +Custom headers are not template variables. The header names and Base64-encoded values are stored under `data` in a Secret of `type: NotificationSender` (for example `webhook-header-secret` in `cpaas-system` on the `global` cluster), and the receiver group (`notificationgroups.ait.alauda.io`) that holds the webhook URL references that Secret with the annotation `cpaas.io/notification.webhook.config`. The platform has no built-in SMS method; an SMS gateway is called through a webhook notification with custom headers. + +For receiver groups created under **Project Management**, the Secret must be in the project's namespace; see [Project webhook alert notifications miss custom headers because webhook-header-secret is only in cpaas-system on ACP](Project_webhook_alert_notifications_miss_custom_headers_because_webhook-header-secret_is_only_in_cpaas-system_on_ACP.md). + +## Root Cause + +### Cause 1: invalid header field value because the Base64 value ends with a newline + +The header values in the Secret were Base64-encoded from the output of a plain `echo`, which appends a newline, so each decoded value ends with `\n`. HTTP header values cannot contain a newline, and the Go HTTP client in `courier-api` refuses to build the request. Nothing is sent, and the notification phase is `Failed`. The header name in the error is printed in canonical form: `X-OPEN-API-APP-ACCESS-KEY` in the Secret appears as `X-Open-Api-App-Access-Key` in the log. + +### Cause 2: the receiver reports the header as not present because the value was built without echo -n + +The receiver answers `400` with `Required request header '' ... is not present`, while the platform side shows the send as successful. The value of the header (for example an encrypted `appKey`) was produced from a plain `echo` without `-n`, so the stray characters from the trailing newline broke it and the request went out without a usable header. This is the same mistake as Cause 1, with a different signature: here the request is sent and rejected by the receiver. + +### Cause 3: the receiver group is not associated with the header Secret + +The receiver group has a webhook URL but no `cpaas.io/notification.webhook.config` annotation, so the platform sends the alert without the custom headers and the receiver rejects it. This also explains a header that suddenly goes missing after the receiver group was recreated or edited. + +## Resolution + +### Cause 1: invalid header field value because the Base64 value ends with a newline + +1. Confirm the signature: + + ```bash + kubectl get pods -A | grep courier-api + kubectl logs -n cpaas-system deploy/courier-api --tail=500 | grep -F 'invalid header field value' + ``` + +2. Decode every value in the header Secret and check for a line break. `true` in the second column means the value carries a newline: + + ```bash + kubectl get secret -n cpaas-system webhook-header-secret -o json \ + | jq -r '.data | to_entries[] | "\(.key)\t\(.value | @base64d | test("[\\r\\n]"))"' + ``` + + Without `jq`, decode one key and look for `\n` at the end of the output: + + ```bash + kubectl get secret -n cpaas-system webhook-header-secret -o jsonpath='{.data.X-OPEN-API-APP-ACCESS-KEY}' | base64 -d | od -c | tail -n 3 + ``` + +3. Export the Secret: + + ```bash + kubectl get secret -n cpaas-system webhook-header-secret -o yaml > /tmp/webhook-header-secret.yaml + ``` + +4. Re-encode each affected value with `echo -n` and write it back. Repeat for every key that showed `true`: + + > **Warning:** The new values are sent to the receiver on the next notification, for every receiver group that references this Secret. Do this only when `courier-api` logs `invalid header field value` and the decoded value ends with a newline, and only with the exact plaintext the receiver issued. To roll back, run `kubectl apply -f /tmp/webhook-header-secret.yaml`. + + ```bash + kubectl patch secret -n cpaas-system webhook-header-secret --type merge \ + -p "{\"data\":{\"X-OPEN-API-APP-ACCESS-KEY\":\"$(echo -n '' | base64 -w0)\"}}" + kubectl patch secret -n cpaas-system webhook-header-secret --type merge \ + -p "{\"data\":{\"X-OPEN-API-APPID\":\"$(echo -n '' | base64 -w0)\"}}" + ``` + +5. Run the decode check again; every key prints `false`. Trigger one alert and confirm that the log no longer shows `invalid header field value`. + +If the receiver then returns `InvalidJson`, the header is fixed and the body is the next problem; see [Webhook alert notifications rejected as InvalidJson or sent without the notification template on ACP](Webhook_alert_notifications_rejected_as_InvalidJson_or_sent_without_the_notification_template_on_ACP.md). A template for an SMS gateway maps the gateway's JSON fields to platform variables; the keys and the phone number belong to the receiver: + +```text +{ + "requestId": "{{ .fingerprint }}", + "requestTime": "{{ dateFormatWithZone .startsAt "2006-01-02 15:04:05" "Asia/Shanghai" }}", + "messageTarget": "", + "messageContent": "{{ .externalLabels.summary }}" +} +``` + +### Cause 2: the receiver reports the header as not present because the value was built without echo -n + +1. Confirm the `400` text and find the header Secret: + + ```bash + kubectl logs -n deploy/courier-api --tail=300 | grep -F "appkey" + kubectl get secret -n cpaas-system --field-selector type=NotificationSender + kubectl get secret -n cpaas-system -o yaml > /tmp/.yaml + ``` + +2. Produce the plaintext without a trailing newline, and pass that output to the encryption or encoding the receiver requires: + + ```bash + echo -n '' + ``` + +3. Write the new value into the Secret. + + > **Warning:** The new value is sent to the third party on the next notification. Do this only when the receiver reports the header as missing with code `400` and the previous value was produced with plain `echo`. To roll back, run `kubectl apply -f /tmp/.yaml`. If the header is still missing after `echo -n` was used, check the receiver-group association (Cause 3). + +4. Send one test and confirm that the `400` line is gone and the receiver gets the notification. + +### Cause 3: the receiver group is not associated with the header Secret + +1. On the `global` cluster, find the receiver group that holds the webhook URL and check for the annotation: + + ```bash + kubectl get notificationgroups.ait.alauda.io -n cpaas-system + kubectl get notificationgroups.ait.alauda.io -n cpaas-system -o yaml | grep -n -E 'notification.webhook.config|webhook' + kubectl get secret -n cpaas-system webhook-header-secret + ``` + +2. Add the annotation, with the name of the header Secret as its value. + + > **Warning:** Every webhook notification sent to this receiver group then carries the headers stored in the Secret. Do this only when the receiver reports the header as missing and the group has no `cpaas.io/notification.webhook.config` annotation. Export the group first with `kubectl get notificationgroups.ait.alauda.io -n cpaas-system -o yaml > /tmp/notificationgroup.yaml`. To roll back, run `kubectl annotate notificationgroups.ait.alauda.io -n cpaas-system cpaas.io/notification.webhook.config-`. + + ```bash + kubectl annotate notificationgroups.ait.alauda.io -n cpaas-system \ + cpaas.io/notification.webhook.config=webhook-header-secret + ``` + +3. Trigger one alert. Notifications return to normal as soon as the association exists. + +## Diagnostic Steps + +1. Read the sender log for the header: + + ```bash + kubectl logs -n deploy/courier-api --tail=300 | grep -i -E 'appkey|header|400|Failed to send' + ``` + + - `Failed to send webhook` with `net/http: invalid header field value for ""` and phase `Failed`: Cause 1. Nothing reached the receiver, so do not change the template or the network for this error. + - `Required request header ''` with code `400`: Cause 2 or Cause 3. + +2. List the receiver groups with their header annotation. A group without the annotation is Cause 3; a group in a project namespace is the project article linked in the Issue section: + + ```bash + kubectl get notificationgroups.ait.alauda.io -A \ + -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,HEADER_SECRET:.metadata.annotations.cpaas\.io/notification\.webhook\.config' + ``` + +3. Run the decode check from Cause 1. Expected after the fix: every key prints `false`. + +4. The header arrives and the receiver still rejects the body: the payload is the problem; see the InvalidJson article. diff --git a/docs/en/solutions/Webhook_alert_notifications_rejected_as_InvalidJson_or_sent_without_the_notification_template_on_ACP.md b/docs/en/solutions/Webhook_alert_notifications_rejected_as_InvalidJson_or_sent_without_the_notification_template_on_ACP.md new file mode 100644 index 000000000..2db7a36bb --- /dev/null +++ b/docs/en/solutions/Webhook_alert_notifications_rejected_as_InvalidJson_or_sent_without_the_notification_template_on_ACP.md @@ -0,0 +1,163 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.14" + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# Webhook alert notifications rejected as InvalidJson or sent without the notification template on ACP + +## Issue + +Webhook alert notifications reach the receiver, but the receiver rejects the body or the body does not follow the configured notification template. Typical symptoms: + +- Feishu or another webhook receiver returns `InvalidJson`, often `invalid character 'å' looking for beginning of value`. The message that does arrive has empty severity, cluster, target, and description, and a zero timestamp. +- A third-party receiver gets some alerts and not others; `webhook-business` logs a render failure for the missing ones. +- The receiver gets the raw Alertmanager alert (top-level `labels`, `annotations`, `startsAt`, `endsAt` = `0001-01-01T00:00:00Z` while firing, `fingerprint`, `generatorURL`, `status`) instead of the fields in the template, for example after an upgrade. +- On v3.18.2 a customized webhook template has no effect, the courier logs never mention the target address, and alert history shows `notifications: null` with Ready reason `NoNotifications`. +- Webhook alerts are not sent according to the template while the `webhook-template` feature gate is on. + +The webhook body is rendered from the alert's labels, annotations, and status fields. For workload alerts that includes the resources that triggered it, such as `.labels.alert_involved_object_namespace` and the involved object's name, so a receiver can identify the target from those fields. No other identifier is added. + +Related article: [DingTalk robot rejects alert notifications with errcode 300001 or 310000 on ACP](DingTalk_robot_rejects_alert_notifications_with_errcode_300001_or_310000_on_ACP.md). + +## Root Cause + +### Cause 1: InvalidJson because the template is still Markdown or HTML + +Webhook rendering requires the rendered string to be one JSON value. When the notification template is still the default Markdown or HTML sample, the receiver's parser fails at the first character, so the fields it would have read stay empty and the timestamp is zero. `apollo` logs `InvalidJson`. + +### Cause 2: invalid JSON because printf "%q" is wrapped in extra quotes + +`webhook-business` renders some alerts and drops others with invalid JSON. The field that breaks is typically: + +```text +"meta_info": "{{ printf "%q" .labels.alert_involved_object_namespace }}" +``` + +`printf "%q"` already emits a double-quoted string, so wrapping it in another pair of quotes produces `""value""`, which is not valid JSON. Alerts whose labels do not exercise that branch still send, so the failure looks intermittent. Field names that the receiver requires (for example `severity`, `alarm_name`, `alarm_content`) belong to the receiver's schema, not to the platform. + +### Cause 3: the webhook-template feature gate is off on 4.x, so the template is ignored + +When the `AlaudaFeatureGate` `webhook-template` in `cpaas-system` has `spec.enabled` other than `true` (or its console switch is off), rendering never runs: `notificationmessage.spec.body`, the raw Alertmanager alert, is posted and the template has no effect. This can appear after an upgrade, for example from 3.14.2 to 4.2.4. The `Webhook begin` entry in the `apollo` log then shows the untransformed alert. The gate is Alpha and was added in 3.14; its description warns that enabling it may change the format of existing webhook alerts. + +### Cause 4: webhookTemplate is off on v3.18.2 + +On platform v3.18.2, custom webhook alert templates are supported, but the feature gate `webhookTemplate` is disabled by default. With it off, alert history shows `notifications: null` and Ready reason `NoNotifications`, and the courier logs do not contain the target webhook address. This is the 3.18 values key `global.featureGate.webhookTemplate`, not the 4.x object of Cause 3. + +### Cause 5: the receiver does not need rendering, but webhook-template is on + +Webhook processing includes a JSON rendering step. Some receivers need it and others do not. A receiver that does not expect a rendered body can fail while `webhook-template` is on, and work again once the gate is off. This is the reverse of Cause 3: the same gate, in the other state, fixes a different receiver. + +## Resolution + +### Cause 1: InvalidJson because the template is still Markdown or HTML + +Replace the Markdown or HTML sample with one JSON object that the receiver accepts. For a Feishu custom bot, a text message has this shape (match the receiver's own documentation if it differs): + +```json +{ + "msg_type": "text", + "content": { + "text": "{{ .labels.alertname }} {{ .annotations.description }}" + } +} +``` + +If a value can contain quotes or newlines, use the quoting form in Cause 2. Send again and confirm that `apollo` no longer logs `InvalidJson`. + +### Cause 2: invalid JSON because printf "%q" is wrapped in extra quotes + +Remove the extra quotes, because `%q` already adds them: + +```text +"meta_info": {{ printf "%q" .labels.alert_involved_object_namespace }} +``` + +For a single token with no quote and no newline, plain interpolation is also valid; do not mix the two forms: + +```text +"meta_info": "{{ .labels.alert_involved_object_namespace }}" +``` + +Use the receiver's key names. Re-send an alert that previously failed. `webhook-business` logs a successful render and the receiver accepts the body. A body that is valid JSON but still lacks a key the receiver requires is a schema mismatch; compare it with a payload the receiver accepts. + +### Cause 3: the webhook-template feature gate is off on 4.x, so the template is ignored + +1. Read the gate: + + ```bash + kubectl get alaudafeaturegates.alauda.io webhook-template -n cpaas-system -o jsonpath='{.spec.enabled}{"\n"}' + ``` + +2. Enable it on the console page `/console-platform/feature-gate`, or patch the object. + + > **Warning:** Enabling `webhook-template` changes the body of every webhook notification that has a template. Receivers that parse today's raw body receive the rendered template instead, and a template that is still Markdown starts failing with `InvalidJson` (Cause 1). Do this only when the webhook section of the notification template is valid JSON for the receiver. To roll back, run the same patch with `false`. + + ```bash + kubectl patch AlaudaFeatureGate webhook-template -n cpaas-system \ + --type='json' -p='[{"op": "replace", "path": "/spec/enabled", "value": true}]' + ``` + + If the object was changed through the API and the console switch still shows off, turn the console switch on as well. + +3. Trigger an alert. The body follows the template. When no template is bound at all, the native Alertmanager body is posted; bind a notification template if the receiver needs a specific schema. + +### Cause 4: webhookTemplate is off on v3.18.2 + +Confirm the version and the empty notification row first. Then set the platform values key: + +```yaml +global: + featureGate: + webhookTemplate: true +``` + +> **Warning:** This turns on custom webhook template rendering. Receivers that parsed the previous body see the template instead, and a template that is not JSON fails as in Cause 1. Do this only on v3.18.2 with `NoNotifications` and the gate still off. To roll back, set the key back to `false`. + +Apply it in the platform values that already carry `global.featureGate`. Check the template format, trigger one alert, and confirm that the courier logs now contain the target webhook address. + +### Cause 5: the receiver does not need rendering, but webhook-template is on + +1. Confirm that the gate is on: + + ```bash + kubectl get alaudafeaturegates.alauda.io webhook-template -n cpaas-system -o jsonpath='{.spec.enabled}{"\n"}' + ``` + +2. Turn the gate off on `/console-platform/feature-gate`, or patch the object. + + > **Warning:** `webhook-template` is platform-wide. Turning it off stops template rendering for every webhook notification, so receivers that depend on a rendered body (for example a DingTalk robot that needs `msgtype`, or a receiver whose schema the template builds) start receiving the native alert body. Do this only when no other webhook receiver relies on a rendered template; list the notification policies that use a webhook method and confirm with their owners first. To roll back, set the gate back to `true`. + + ```bash + kubectl patch AlaudaFeatureGate webhook-template -n cpaas-system \ + --type='json' -p='[{"op": "replace", "path": "/spec/enabled", "value": false}]' + ``` + +3. Trigger one alert and confirm that the receiver accepts it. No per-receiver setting for this gate is available. + +## Diagnostic Steps + +1. Copy the response line of one failed send: + + ```bash + kubectl get pods -n cpaas-system | grep -E 'apollo|courier-api' + kubectl logs -n cpaas-system deploy/courier-api --tail=300 | grep -i -E 'errcode|InvalidJson|Webhook begin|invalid header' + kubectl logs -n cpaas-system --tail=200 | grep -i -E 'InvalidJson|Webhook begin' + ``` + +2. Map it to a cause: + - `InvalidJson`, or a rendered body that starts with `#`, `<`, or a letter: Cause 1. + - A render error in `webhook-business` on some alerts only, on a field wrapped with `printf "%q"`: Cause 2. + - The receiver gets the raw Alertmanager alert, or `Webhook begin` shows it: Cause 3. Expected gate value: not `true`. + - `notifications: null` and `NoNotifications` on v3.18.2: Cause 4. + - Messages not sent as expected while the gate is already `true`, for a receiver that does not need rendering: Cause 5. + +3. A `courier-api` `Failed to send webhook` error with no HTTP response, such as `invalid header field value`, means nothing was posted; the gate is not the cause. See [Webhook alert notifications fail with invalid header field value or miss a required custom header on ACP](Webhook_alert_notifications_fail_with_invalid_header_field_value_or_miss_a_required_custom_header_on_ACP.md). diff --git a/docs/en/solutions/What_platform_backup_application_snapshots_and_volume_snapshots_cover_on_ACP.md b/docs/en/solutions/What_platform_backup_application_snapshots_and_volume_snapshots_cover_on_ACP.md new file mode 100644 index 000000000..c1155d1be --- /dev/null +++ b/docs/en/solutions/What_platform_backup_application_snapshots_and_volume_snapshots_cover_on_ACP.md @@ -0,0 +1,187 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - 3.14.x + - 4.2.x + - 4.3.x +--- + +# What platform backup, application snapshots, and volume snapshots cover on ACP + +## Issue + +Before choosing a backup feature on Alauda Container Platform, these questions come up: + +- Is the data of a native application snapshot stored in etcd? Does it cover only YAML, and does it miss ConfigMaps, Secrets, and ALB resources? Should the platform backup and restore component use S3 storage, and does it back up everything in a namespace? +- An application backup has both "Kubernetes resources" and "PVC data volumes" selected. Is the data inside the volumes backed up? +- A namespace runs MySQL and Redis data services. Is a full namespace backup, PVCs included, enough? +- How are applications, ConfigMaps, and Secrets moved from namespace B in cluster A to namespace D in cluster C, or a whole namespace from a shared cluster to a dedicated one? +- Can an existing namespace be renamed without affecting its pods? +- Where is the "restore volume snapshot" action? +- Releases go out through the platform API. Can the resources be backed up before each release and rolled back automatically if the release goes wrong? +- Some ConfigMaps are missing after a namespace restore. Can the backup be inspected to see whether it contained them? + +## Root Cause + +### Cause 1: native application snapshot versus the platform backup and restore component + +On ACP 3.14.1, the native application snapshot backs up only PVC data; it does not back up Kubernetes resource objects, ConfigMaps, Secrets, or ALB resources, and it does not store anything in etcd. The platform backup and restore component (Velero) performs a namespace-level full backup that includes ConfigMaps and Secrets; S3 external storage is recommended for it. Whether ALB resources are covered depends on the backup policy. + +When an application backup has both "Kubernetes resources" and "PVC data volumes" selected, the data inside the volumes is backed up too: Velero saves the namespace's Kubernetes resources and the data on the volumes bound to its PVCs to the configured backup repository, as a file-level backup with the restic uploader. On restore, Velero recreates the resources and writes the volume data back into the new PVCs. + +### Cause 2: a namespace backup is not enough for MySQL and Redis + +A namespace-level backup, even with PVCs, cannot guarantee data consistency for databases and risks data corruption. Data services support only their own dedicated backup and restore. Their resources are excluded from the namespace backup with a label selector, and MySQL and Redis are backed up with the data services' own backup. + +### Cause 3: migrating an application or namespace to another cluster + +Cross-cluster migration uses the platform backup and restore flow: back up the source namespace with an application backup policy, then run an application restore task on the target cluster. Source and target clusters should have similar CPU and memory capacity, so migrated pods do not stay `Pending`. + +### Cause 4: a namespace cannot be renamed in place + +A namespace name cannot be changed, and a rename affects the resources in it. Renaming means creating a new namespace and moving the resources into it. + +### Cause 5: there is no standalone "restore from volume snapshot" action + +Data is restored by creating a new PVC from the volume snapshot and mounting that PVC in the workload. + +### Cause 6: no built-in pre-release backup and automatic rollback through the API + +The platform does not support backing up the current resources and rolling back automatically through the API. Cluster backup and restore (**Cluster > Backup and Restore**) is for disaster recovery, not release rollback. What exists on ACP 4.2 is the native application version snapshot, which is recorded automatically only for changes made in the web console, and Deployment-level rollback. + +### Cause 7: checking whether a backup contained a specific ConfigMap or Secret + +A namespace-level backup by the platform component includes the namespace's ConfigMaps and Secrets. After a restore, the Velero log of the restore cluster names each restored object in an `Attempting to restore ConfigMap: ` entry, which shows that the backup carried it. Without a restore, the backup's resource list shows it. + +## Resolution + +### Cause 1: native application snapshot versus the platform backup and restore component + +Use the platform backup and restore component with S3 external storage when ConfigMaps, Secrets, and the rest of the namespace must be protected. To include volume data, select both "Kubernetes resources" and "PVC data volumes" in the application backup. On ACP 3.14.1, confirm that volume snapshots or a storage-level backup are available for the StorageClass in use before relying on the component for PVC data. Before relying on it for ALB rules, check what the backup policy includes. + +```bash +kubectl get storageclass +kubectl get volumesnapshotclass +``` + +To check that a finished backup carried volume data, list its pod volume backups and the repository: + +```bash +kubectl -n cpaas-system get podvolumebackups.velero.io -l velero.io/backup-name= +kubectl -n cpaas-system get backuprepositories.velero.io +``` + +### Cause 2: a namespace backup is not enough for MySQL and Redis + +1. Back up MySQL and Redis with the data services' dedicated backup and restore. +2. Exclude their resources from the namespace backup with `NotIn` expressions in the backup's `labelSelector`: + +```yaml +labelSelector: + matchExpressions: + # resources with label a=b are not backed up + - key: a + operator: NotIn + values: ["b"] + # resources with label c=d are not backed up + - key: c + operator: NotIn + values: ["d"] +``` + +Replace `a`/`b` and `c`/`d` with labels that the data-service objects (instances, StatefulSets, pods, PVCs) carry and the business workloads do not. List the labels first: + +```bash +kubectl -n get statefulset,pod,pvc --show-labels | grep -iE 'mysql|redis' +``` + +`NotIn` also keeps objects that do not have the key, so business objects without the label stay in the backup. Test a restore of the filtered backup in a non-production namespace. + +### Cause 3: migrating an application or namespace to another cluster + +1. Install the backup and restore component and give the cluster a default S3 storage. +2. Create an application backup policy for the source namespace and run it. +3. On the target cluster, create an application restore task and select the backup from the restore task list. A different target namespace is set with namespace mapping (`spec.namespaceMapping` on the Velero `Restore` object). +4. Check capacity first so restored pods can schedule: + +```bash +kubectl describe nodes | grep -A6 'Allocated resources' +``` + +Restored objects that refer to their source namespace, nodes, registry, or StorageClass are covered in [Application restore fails, hangs, or restored workloads do not run on another cluster on ACP](./Application_restore_fails_hangs_or_restored_workloads_do_not_run_on_another_cluster_on_ACP.md). + +### Cause 4: a namespace cannot be renamed in place + +Create the new namespace and move the resources into it. The platform backup and restore feature (Cause 3, with namespace mapping) covers Kubernetes resources and PVC data and is recommended when many resources have to move. For a few services, recreating them by hand in the new namespace is fine. Delete the old namespace only after the workloads in the new one are verified. + +> **Warning:** Deleting the old namespace deletes every object in it, and PVCs whose PV reclaim policy is `Delete` lose their data. Precondition: the new namespace runs correctly and its PVCs hold the data. Rollback: none except restoring from a backup. + +```bash +kubectl delete namespace +``` + +### Cause 5: there is no standalone "restore from volume snapshot" action + +1. In the left navigation, click **Storage > PersistentVolumeClaims (PVC)**. +2. Click the name of the backed-up PVC and select the **Volume Snapshots** tab. +3. Locate the snapshot, click ⋮, and choose **Create PersistentVolumeClaim**. +4. Fill in the PVC parameters and click **Create**. +5. Mount it, for example in a Deployment: **Compute Components > Deployments** > the Deployment > **Actions > Update**; in the **Container Group** section, change the storage volume to the new PVC and click **Update**. + +The update rolls the Deployment's pods; afterwards the restored data is at the volume mount path. The product documentation describes the same procedure in "Restore Data" and in the best practice on using volume snapshots for data backup. + +### Cause 6: no built-in pre-release backup and automatic rollback through the API + +Options: + +- Native applications: create a version snapshot by hand before each API-driven release, because API deployments are not recorded. Console path: **Container Platform > Application Management > Native Applications** > application details > **Version Snapshot** > **Create**. To roll back, open **History Versions**, click the version number, and select **Roll Back to This Version**. At most 6 snapshots are kept; the oldest is deleted when the limit is exceeded. No API is documented for creating or rolling back snapshots. +- Individual Deployments: roll back with `ac rollout undo` (or the upstream `kubectl rollout undo`). +- Automation: build it on the native Kubernetes API, which the platform forwards under its own URL prefix. + +```bash +kubectl rollout history deployment/ -n +``` + +> **Warning:** `rollout undo` replaces the Deployment's pod template with the previous revision and rolls all pods. Precondition: the previous revision is the one you want (check the history). Rollback: `kubectl rollout undo deployment/ -n --to-revision=` back to the revision you left. + +```bash +ac rollout undo deployment/ -n +``` + +### Cause 7: checking whether a backup contained a specific ConfigMap or Secret + +After a restore, search the Velero log of the restore cluster: + +```bash +kubectl -n cpaas-system logs deploy/velero | grep 'restore=cpaas-system/' | grep '' +``` + +An `Attempting to restore ConfigMap: ` entry means the object was in the backup; see [ConfigMaps missing after a namespace restore that completed without errors on ACP](./ConfigMaps_missing_after_a_namespace_restore_that_completed_without_errors_on_ACP.md) for what to check next. + +Without a restore, print the backup's resource list with the Velero CLI in the Velero pod, on a cluster whose backup storage holds the backup: + +```bash +kubectl -n cpaas-system get backups.velero.io -o jsonpath='{.status.phase}{" "}{.status.progress}{"\n"}' +kubectl exec -n cpaas-system deploy/velero -- ./velero backup describe --details | grep -E 'v1/ConfigMap|/' +``` + +The `Resource List` section lists objects as `/` under each kind. An object that is not listed was not in the backup; then check the policy's included namespaces and its `labelSelector` (Cause 2). + +## Diagnostic Steps + +```bash +kubectl -n cpaas-system get backups.velero.io,schedules.velero.io +kubectl get volumesnapshot -A +kubectl -n get pvc +``` + +- Snapshot versus platform backup scope, or whether PVC data is included: Cause 1. +- A namespace with MySQL or Redis: Cause 2. +- Moving a namespace or application to another cluster or namespace: Cause 3. +- Renaming a namespace: Cause 4. +- Restoring a PVC from a volume snapshot: Cause 5. +- Pre-release backup and rollback for API deployments: Cause 6. +- Whether a backup contained a specific object: Cause 7. diff --git a/docs/en/solutions/Which_address_goes_into_Active_Global_Cluster_ETCD_Endpoints_when_enabling_etcd_sync_for_global_DR_on_ACP.md b/docs/en/solutions/Which_address_goes_into_Active_Global_Cluster_ETCD_Endpoints_when_enabling_etcd_sync_for_global_DR_on_ACP.md new file mode 100644 index 000000000..8417ec7d0 --- /dev/null +++ b/docs/en/solutions/Which_address_goes_into_Active_Global_Cluster_ETCD_Endpoints_when_enabling_etcd_sync_for_global_DR_on_ACP.md @@ -0,0 +1,104 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - 4.3.x +--- + +# Which address goes into Active Global Cluster ETCD Endpoints when enabling etcd sync for global DR on ACP + +## Issue + +While deploying a standby `global` cluster for disaster recovery (DR), the last installation step is to enable etcd synchronization. On the standby cluster's Web Console, under **Marketplace** > **Cluster Plugins** with the `global` cluster selected, the **etcd Synchronizer** plugin is installed with these parameters: + +- **Active Global Cluster VIP** +- **Active Global Cluster ETCD Endpoints** +- **Standby Cluster ETCD Endpoints** +- **Active Global Cluster Token Secret** +- **Data Check Interval** +- **Print detail logs** + +The installation guide only says to "configure **Active Global Cluster ETCD Endpoints** correctly" when port `2379` is not forwarded through a load balancer, and it is not obvious which address that field expects: the active cluster or the standby cluster, a VIP or the individual control plane nodes. + +## Root Cause + +### Cause 1: the field names the etcd of the active cluster, reached through its VIP on port 2379 + +The etcd Synchronizer runs on the **standby** `global` cluster. It reads etcd data from the **active** (primary) `global` cluster on port `2379` and writes it into the standby cluster's local etcd. **Active Global Cluster ETCD Endpoints** tells it where to read from, so it must point at the active cluster, not at the standby. The value is the VIP of the active `global` cluster followed by port `2379`, for example `https://:2379`. + +**Standby Cluster ETCD Endpoints** is the local side and normally keeps its default value. + +## Resolution + +### Cause 1: the field names the etcd of the active cluster, reached through its VIP on port 2379 + +1. On the **active** `global` cluster, read the bearer token that the standby cluster uses to access the active cluster's API server: + + ```bash + kubectl -n cpaas-system get secret k8sadmin -o jsonpath='{.data.token}' | base64 -d + ``` + +2. On the **standby** `global` cluster, store that token in the `etcd-sync-active-cluster-token` Secret, under the data key `token`: + + ```bash + ACTIVE_CLUSTER_TOKEN='' + kubectl -n cpaas-system create secret generic etcd-sync-active-cluster-token \ + --from-literal=token="${ACTIVE_CLUSTER_TOKEN}" \ + --dry-run=client -o yaml | kubectl apply -f - + ``` + +3. Make sure the standby cluster's control plane nodes can reach port `2379` on the active cluster's VIP over TCP (see Diagnostic Steps). If a load balancer is used for this port, it must forward in TCP mode; L7 forwarding is not supported. + +4. Open the **standby** cluster's Web Console through its own VIP, switch to the **Administrator** view, go to **Marketplace** > **Cluster Plugins**, select the `global` cluster, and install the **etcd Synchronizer** plugin with: + + | Parameter | Value | + | --- | --- | + | Active Global Cluster VIP | VIP of the active `global` cluster | + | Active Global Cluster ETCD Endpoints | VIP of the active `global` cluster with port `2379`, for example `https://:2379` | + | Standby Cluster ETCD Endpoints | Default value, unless the local etcd is exposed through a different endpoint | + | Active Global Cluster Token Secret | `etcd-sync-active-cluster-token` | + | Data Check Interval | Default value | + | Print detail logs | Disabled, unless troubleshooting | + +5. During installation the `etcd-sync-bootstrap` Job runs before the `etcd-sync` Deployment starts. Verify the Job and the resources it prepares, on the standby cluster: + + ```bash + kubectl get job -n cpaas-system etcd-sync-bootstrap + kubectl logs -n cpaas-system job/etcd-sync-bootstrap + kubectl get secret -n cpaas-system remote-etcd-ca + kubectl get issuer -n cpaas-system remote-etcd-issuer + kubectl get certificate -n cpaas-system remote-etcd-client + kubectl get secret -n cpaas-system remote-etcd-client + ``` + +6. Verify that the sync Pods run and that the leader has started syncing: + + ```bash + kubectl get po -n cpaas-system -l app=etcd-sync + leader_pod=$(kubectl get lease -n cpaas-system etcd-sync-mirror -o jsonpath='{.spec.holderIdentity}') + kubectl logs -n cpaas-system "$leader_pod" | grep -E "Acquired leader lease|Start Sync update" + ``` + +## Diagnostic Steps + +1. From a control plane node of the **standby** cluster, check that port `2379` on the active cluster's VIP accepts TCP connections: + + ```bash + nc -vz -w 3 2379 + ``` + + Expected: the connection succeeds. A timeout or `Connection refused` means the standby cluster cannot reach the active etcd through that address, and the network path or load balancer forwarding has to be fixed before the plugin can sync. + +2. Check the values the plugin was installed with. **Active Global Cluster VIP** and **Active Global Cluster ETCD Endpoints** must both refer to the **active** cluster; **Standby Cluster ETCD Endpoints** refers to the standby cluster. + +3. Check that the token Secret exists on the standby cluster and holds the key `token`: + + ```bash + kubectl -n cpaas-system get secret etcd-sync-active-cluster-token -o jsonpath='{.data}' | grep -o '"token"' + ``` + + Expected output: `"token"`. + +4. Check the leader's log on the standby cluster. Expected: `Acquired leader lease` followed by `Start Sync update`. If the sync Pods do not start at all, see [etcd-sync Pod stuck in ContainerCreating with failed to sync secret cache on the standby global cluster on ACP](etcd-sync_Pod_stuck_in_ContainerCreating_with_failed_to_sync_secret_cache_on_the_standby_global_cluster_on_ACP.md). diff --git a/docs/en/solutions/Which_operating_systems_kernels_and_CPU_architectures_can_run_in_an_ACP_cluster.md b/docs/en/solutions/Which_operating_systems_kernels_and_CPU_architectures_can_run_in_an_ACP_cluster.md new file mode 100644 index 000000000..59eff3ae1 --- /dev/null +++ b/docs/en/solutions/Which_operating_systems_kernels_and_CPU_architectures_can_run_in_an_ACP_cluster.md @@ -0,0 +1,115 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - "3.10" + - "3.14" + - "3.16" + - "4.1" + - "4.2" + - "4.3" +--- + +# Which operating systems, kernels, and CPU architectures can run in an ACP cluster + +## Issue + +Before adding a node, upgrading a node kernel, or mixing node types, the following questions come up: + +- Is a newer Kylin kernel (for example for a security patch) supported? A newer openEuler release? Rocky Linux 10.1? An Ubuntu HWE kernel? +- Can CentOS and Kylin nodes, or Kylin and Ubuntu nodes, share one cluster, with some nodes running only ALB? +- Can x86 and ARM nodes share one 4.x cluster? Does a Hygon C86 cluster accept x86 nodes? Can an ARM global cluster manage Intel x86 servers? Are Kunpeng 920 (aarch64) hosts supported? +- The kernel of a new node changed after a reboot and its NIC stopped working. Did the platform initialization upgrade the kernel? + +Related articles: "Preparing a host before adding it to an ACP cluster: node check items, disks, init script, DefaultTasksMax, root access"; "Shutting down or rebooting ACP nodes for maintenance" covers rebooting nodes one by one for a kernel or OS upgrade. + +## Root Cause + +### Cause 1: only the OS releases and kernels in the version's baseline are supported + +Each platform release documents its own verified OS and kernel list in the node preprocessing section of its installation documentation. A node outside that list can fail the availability check or run with components that do not work on it. + +Known baselines and answers: + +- **3.14.2**, on both ARM (Kunpeng 920) and x86: Kylin Linux Advanced Server V10, V10-SP1, and V10-SP2, with the official built-in kernels of those releases only. The kernel `A.B.C` must be `4.19.90`; the suffix after `-` may differ, so the full build string does not have to match a tested kernel. Tested kernels include `4.19.90-11.ky10`, `4.19.90-23.8.v2101.ky10`, and `4.19.90-24.4.v2101.ky10` on both architectures, and `4.19.90-17.ky10.aarch64` on ARM. SP1 `4.19.90-23.72` and SP2 `4.19.90-25.63` fit the rule when each is the official built-in kernel of its release. V10-SP3 is not in the 3.14 baseline, so an SP3 kernel such as `4.19.90-52.64` is not supported on 3.14.2 even though its `A.B.C` is `4.19.90`. A vendor-fixed or already-installed kernel that is not an official built-in kernel of V10, V10-SP1, or V10-SP2 (for example `4.19.90-52.22.v2207.ky10` on aarch64) is outside the tested list, and stable operation is not guaranteed. +- **3.14.1**: Kylin V10 / SP1 / SP2, Ubuntu 20.04 / 22.04, and CentOS 7.6–7.9 with a kernel from `3.10.0-1127` up to but not including 3.11. Oracle Linux 9.7 is not adapted (`kube-ovn-cni` crashes on its ipset), and Ubuntu 24.04 is not verified. +- **4.2.3** supports a broader OS and kernel list, which adds Kylin V10-SP3 and MicroOS 5.5, among others. +- **Ubuntu**: the stock kernels `5.4.0-135-generic` (Ubuntu 20.04) and `5.15.0-56-generic` (Ubuntu 22.04) are the baseline; HWE kernels such as `6.8.0-106-generic` are not supported (on 4.1.2, such a server was reinstalled with a 5.15 kernel). +- **Rocky Linux 10.1** is not on the supported OS list of ACP 4.3. The availability check shows `检查操作系统:未支持的操作系统: Rocky Linux 10.1` (OS check: unsupported operating system), `EnsurePrepareNode` passes, and the add stops at `EnsureContainerRuntime` with `FailedInit`. +- **openEuler 24.03 (LTS-SP4)** is reported as an unsupported version by the 4.x availability check. +- On ACP 4.2.1, a failure prompt shown while adding a node whose OS version is listed as supported is non-blocking; the deployment can be continued past it. + +### Cause 2: mixed operating systems or CPU architectures in one cluster + +- Nodes of different operating systems can share a cluster as long as every node has the same CPU architecture and an OS within the version's baseline; the platform does not require one OS for all nodes. On 3.16 and 3.10, CentOS and Kylin nodes can share a cluster. One cluster may contain both Kylin and Ubuntu nodes, and dedicating some Kylin nodes to ALB only is feasible. +- On ACP 4.x, x86 and ARM nodes in one cluster need the multi-architecture (heterogeneous) images, uploaded once from the multi-architecture product package of the matching version. The 3.x-era step of setting the label `cpaas.io/node-arch-constraint` to `""` on the cluster's `clusters.platform.tkestack.io` resource on global is no longer needed on 4.x; the label is not present by default. An x86-only installation rejects ARM nodes with `硬件架构不匹配` (hardware architecture mismatch). +- A Hygon C86 cluster supports adding x86 nodes. +- A platform whose global cluster runs on ARM, with a business cluster of ARM and C86 servers, can manage Intel x86 servers by putting them in a separately created cluster instead of adding them to the existing mixed business cluster; the x86 package is compatible for managing those servers. +- Hosts with the aarch64 instruction set on Kunpeng 920 are supported in principle; no validated configuration or version is stated for that answer, and it says nothing about mixing x86 and aarch64 in one cluster. + +### Cause 3: the kernel of a new node changed after a reboot because of automatic OS updates, not the platform + +The platform initialization script does not upgrade the node kernel. When a node comes up on a newer kernel after a reboot (for example `5.15.0-187` instead of `5.15.0-119` on Ubuntu) and its NIC then fails on a kernel compatibility problem, the operating system's automatic updates are the likely source: the Unattended Upgrades service installs new kernel packages, and the next reboot boots them. + +## Resolution + +### Cause 1: only the OS releases and kernels in the version's baseline are supported + +1. Before adding a node or upgrading its kernel, read the OS and kernel on every node and compare them with the supported list of your platform version: + + ```bash + kubectl get nodes -o custom-columns=NAME:.metadata.name,OS:.status.nodeInfo.osImage,KERNEL:.status.nodeInfo.kernelVersion + cat /etc/os-release + uname -r + ``` + +2. On 3.14.2, an official built-in kernel of Kylin V10 / SP1 / SP2 is within the baseline when the `A.B.C` in `uname -r` is `4.19.90`. A kernel from another `A.B.C`, any SP3 kernel, or a vendor-fixed kernel outside the tested list is not. +3. For an OS that is not in the list of your version (for example Rocky Linux 10.1 on 4.3), reinstall the host with a supported OS before adding it; the add cannot be completed on it. +4. Reboot nodes one at a time for a kernel upgrade, as described in "Shutting down or rebooting ACP nodes for maintenance". + +### Cause 2: mixed operating systems or CPU architectures in one cluster + +1. Read the architecture of each host; `uname -m` prints `x86_64` on Hygon C86 and Intel x86 hosts and `aarch64` on Kunpeng hosts, and that value decides which product package a node needs: + + ```bash + uname -m + kubectl get nodes -o custom-columns=NAME:.metadata.name,ARCH:.status.nodeInfo.architecture + ``` + +2. For mixed CPU architectures on 4.x, install or upgrade from the multi-architecture product package of your version and upload its images; there is no label to change. To confirm the label is absent on an existing 4.x cluster, run on the global cluster: + + ```bash + kubectl get clusters.platform.tkestack.io -o jsonpath='{.metadata.labels}' | grep -o 'cpaas.io/node-arch-constraint[^,]*' || echo "no node-arch-constraint label" + ``` + +3. For mixed operating systems, check that every node's OS and kernel are in the version's baseline (Cause 1). + +### Cause 3: the kernel of a new node changed after a reboot because of automatic OS updates, not the platform + +1. Confirm the source of the kernel change on the node: + + ```bash + uname -r + ls /boot | grep -E 'vmlinuz|initrd' + systemctl status unattended-upgrades --no-pager + cat /etc/apt/apt.conf.d/20auto-upgrades + grep -h -E 'Start-Date|linux-image' /var/log/apt/history.log 2>/dev/null | tail -n 20 + sudo zgrep -h -i 'linux-image' /var/log/unattended-upgrades/unattended-upgrades.log* 2>/dev/null | tail -n 20 + ``` + + `APT::Periodic::Unattended-Upgrade "1";` in `20auto-upgrades` means automatic upgrades are enabled, and a `linux-image` package in the logs with a date matching the change confirms the source. + +2. Hand the NIC recovery and the kernel analysis to the owners of the OS and the infrastructure. Whatever way a kernel got onto a node, it still has to be in the version's baseline (Cause 1). + +## Diagnostic Steps + +```bash +kubectl get nodes -o custom-columns=NAME:.metadata.name,OS:.status.nodeInfo.osImage,KERNEL:.status.nodeInfo.kernelVersion,ARCH:.status.nodeInfo.architecture +cat /etc/os-release; uname -r; uname -m +``` + +- The availability check reports `未支持的操作系统` (unsupported operating system), or the planned OS or kernel is not in the supported list of the version: Cause 1. +- A question about mixing operating systems or architectures, or an add rejected with `硬件架构不匹配`: Cause 2. +- A kernel in `uname -r` that differs from the one the node had at initialization, with no platform action in between, and Unattended Upgrades enabled: Cause 3. diff --git a/docs/en/solutions/Workload_changes_reverted_not_applied_or_restarting_Pods_OAM_Helm_template_sync_StatefulSet_partition_reloader_on_ACP.md b/docs/en/solutions/Workload_changes_reverted_not_applied_or_restarting_Pods_OAM_Helm_template_sync_StatefulSet_partition_reloader_on_ACP.md new file mode 100644 index 000000000..727a05f45 --- /dev/null +++ b/docs/en/solutions/Workload_changes_reverted_not_applied_or_restarting_Pods_OAM_Helm_template_sync_StatefulSet_partition_reloader_on_ACP.md @@ -0,0 +1,165 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.12.x + - 3.14.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# Workload changes reverted, not applied, or restarting Pods: OAM, Helm template sync, StatefulSet partition, reloader on ACP + +## Issue + +A workload is changed and the cluster does something else: + +- A manual edit of a Deployment that an OAM application created is reverted. +- A manual edit of the `resources` of a Deployment in an OAM application without `apply-once` is not reverted, while the controller log keeps printing `patching object`. +- A native application or a Helm-deployed application (for example Redis) is updated without anyone touching it: its history shows a new revision, its Pods restart several times, and manual configuration changes are lost. +- An application deployed with Helm is now maintained by editing its Deployments in the UI, and the question is whether it must be detached from its template. +- After a parameter change in the console, a StatefulSet Pod does not restart. +- Updating a ConfigMap makes a Deployment restart by itself. + +Each case is a controller doing what it is configured to do, or a known product defect. + +## Root Cause + +### Cause 1: the OAM application restores the Deployment it rendered + +An OAM application with a custom template manages the service lifecycle and restores the Deployment it rendered to the template's desired state (seen on 4.0.2). An in-place edit of a ComponentDefinition does not reach existing applications, which keep the version they reference, and there is no dry-run interface to preview the render of a new template. + +### Cause 2: OAM synchronizes only the fields the template defines + +The application controller re-applies the template periodically; its log repeats `"patching object" name="" resource="apps/v1, Kind=Deployment"` for the Service, PersistentVolumeClaim, ConfigMap, Deployment and `StartStopScalerTrait` of the application, for example every five minutes. Fields that the template does not define, such as `resources` in this case, are outside the synchronized state, so a manual value stays. + +### Cause 3: a captain restart re-synchronizes Helm-based applications and redeploys them + +When `captain` (captain-controller) restarts, a defect in its check for whether a HelmRequest changed triggers a resynchronization of the downstream Helm templates, which re-applies the application's resources. The result is a new application revision without user action, Pods that restart, and manual changes that are reverted. The Kubernetes audit log then shows `patch` requests on the application's resources, for example `/api/v1/namespaces//services/?fieldManager=manager`, by user `system:serviceaccount:kube-system:admin` with user agent `Go-http-client/2.0`, setting `app.kubernetes.io/managed-by: Helm` on objects annotated with `meta.helm.sh/release-name`. A Redis StatefulSet hit this way can also leave a deleted Pod outside the StatefulSet's management, so the new Pod cannot start. The defect is in ACP 3.12 and 3.14 and is fixed in 3.15 and later. + +### Cause 4: a Helm-deployed application maintained in the UI must be detached from its template + +As long as the application is still attached to its Helm template, the Helm-managed state (the values) can be applied again, for example when `captain` on the global cluster restarts (Cause 3), and UI edits are overwritten. + +### Cause 5: rollingUpdate.partition is higher than the Pod ordinal + +With `updateStrategy.type: RollingUpdate` and `rollingUpdate.partition: `, only Pods whose ordinal is greater than or equal to the partition are updated when the template changes. A StatefulSet with one replica (ordinal 0) and `partition: 4` replaces no Pod after a change, for example to a ConfigMap script mounted as `/scripts/start.sh` with `subPath: start.sh`. + +### Cause 6: the workload carries reloader.stakater.com/auto: "true" + +That annotation makes the workload restart whenever its ConfigMap changes, so it reads the new configuration. Some platform component charts set it in their packages; on business workloads it was added by the workload owner. There is no global switch. + +## Resolution + +### Cause 1: the OAM application restores the Deployment it rendered + +1. Export the application, then add the `apply-once` policy so the rendered Deployments can be changed while the template keeps publishing: + + ```bash + kubectl get applications.core.oam.dev -n -o yaml > "$HOME/-oam.yaml" + kubectl edit applications.core.oam.dev -n + ``` + + ```yaml + spec: + policies: + - name: apply-once + type: apply-once + properties: + enable: true + ``` + + > **Warning:** With `apply-once` the application no longer restores drift; manual edits stay until the next publish changes those fields. Precondition: direct edits to the rendered workloads are intended. Rollback: apply the exported YAML, which removes the policy and brings back the revert. + +2. For a template change, keep the existing ComponentDefinition and publish a new versioned one instead of editing it in place: keep `web-service-v1`, publish `web-service-v2` with the new parameters (for example CPU and memory defaults `*"500m"` and `*"512Mi"`), then change the component `type` in the Application from `web-service-v1` to `web-service-v2` and apply. The Deployment is re-rendered; the Application does not have to be deleted. + +Detaching the application from its template also stops the revert, but gives up the template. + +### Cause 2: OAM synchronizes only the fields the template defines + +Expected behavior. Check which fields the template defines: + +```bash +kubectl get pods -A | grep -i vela +kubectl logs -n --tail=200 | grep -F 'patching object' +kubectl get applications.core.oam.dev -n -o yaml | grep -n -A5 'resources' +``` + +To have the template enforce `resources`, define them in the OAM template. + +### Cause 3: a captain restart re-synchronizes Helm-based applications and redeploys them + +1. Match the unexpected revision with a `captain` restart and with the audit log entries described above: + + ```bash + kubectl get pods -n cpaas-system | grep captain + kubectl get helmrequest -A | grep + ``` + +2. Stop template sync for the affected application by annotating its HelmRequest (in the console: Platform Management > Cluster > Resource Management, search for the `helmrequest`, edit its YAML; or with `kubectl` on the cluster that holds it): + + ```bash + kubectl get helmrequest -n -o yaml > "$HOME/-hr.yaml" + kubectl annotate helmrequest -n captain-no-sync="true" + ``` + + > **Warning:** `captain-no-sync: "true"` stops captain from synchronizing this HelmRequest, so later template or values changes are not deployed until it is removed. Precondition: the syncs of this HelmRequest restart the application. Rollback: `kubectl annotate helmrequest -n captain-no-sync-`; the next sync can redeploy the application and restart its Pods. + +3. Long term: upgrade to 3.15 or later, where the defect is fixed. For Redis, prefer Operator mode over Helm mode, and keep the main configuration of business charts in values so a resync does not lose it. + +### Cause 4: a Helm-deployed application maintained in the UI must be detached from its template + +Use the application's detach-from-template (`脱离模板`) action before maintaining its Deployments and StatefulSets directly in the UI. Afterwards Helm values are no longer applied to it. + +### Cause 5: rollingUpdate.partition is higher than the Pod ordinal + +```bash +kubectl get sts -n -o jsonpath='{.spec.replicas}{" "}{.spec.updateStrategy}{"\n"}' +kubectl get sts -n -o jsonpath='{.status.currentRevision}{" "}{.status.updateRevision}{"\n"}' +``` + +A partition greater than or equal to the replica count, with `currentRevision` different from `updateRevision`, means no Pod moves to the new template. + +> **Warning:** Setting the partition to 0 rolls every Pod of the StatefulSet to the new template, one at a time from the highest ordinal. Precondition: the partition was not left in place on purpose for a staged rollout, and the restarts are acceptable. Rollback: set the partition back; Pods already replaced stay on the new revision. + +```bash +kubectl patch sts -n --type merge -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":0}}}}' +kubectl rollout status sts -n +``` + +### Cause 6: the workload carries reloader.stakater.com/auto: "true" + +```bash +kubectl get deployment,sts,ds -A -o yaml | grep -n reloader.stakater.com +``` + +Remove the annotation from each business workload that should not restart on ConfigMap changes; leave the platform components as their charts set them. + +> **Warning:** Without the annotation, a ConfigMap change no longer restarts the workload, and an application that reads its configuration only at start keeps the old values until it is restarted another way. Rollback: add the annotation again. + +```bash +kubectl annotate deployment -n reloader.stakater.com/auto- +``` + +## Diagnostic Steps + +```bash +kubectl get deploy,sts -n -o yaml | grep -n -E 'ownerReferences|reloader|partition|captain|app.oam.dev|meta.helm.sh' +kubectl get helmrequest -A | grep +kubectl get applications.core.oam.dev -A | grep +``` + +```text +edit of an OAM-rendered Deployment reverted -> Cause 1 +edit of resources not reverted, controller logs patching object -> Cause 2 +new revision or Pod restarts with no user action, right after captain restart -> Cause 3 +Helm-deployed application to be maintained in the UI -> Cause 4 +StatefulSet change did not roll the Pod, partition >= replicas -> Cause 5 +ConfigMap update restarts the workload, reloader annotation present -> Cause 6 +``` + +For Cause 3, the audit log entries by `system:serviceaccount:kube-system:admin` with `fieldManager=manager` at the time of the new revision, and no user request, confirm the resync. diff --git a/docs/en/solutions/Workload_cluster_abnormal_or_monitoring_empty_in_the_console_because_global_cannot_reach_its_ports_on_ACP.md b/docs/en/solutions/Workload_cluster_abnormal_or_monitoring_empty_in_the_console_because_global_cannot_reach_its_ports_on_ACP.md new file mode 100644 index 000000000..668f57779 --- /dev/null +++ b/docs/en/solutions/Workload_cluster_abnormal_or_monitoring_empty_in_the_console_because_global_cannot_reach_its_ports_on_ACP.md @@ -0,0 +1,174 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# Workload cluster abnormal or monitoring empty in the console because global cannot reach its ports on ACP + +Related articles: [New node cannot reach CoreDNS or the API server because a firewall drops tunnel traffic between nodes on ACP](New_node_cannot_reach_CoreDNS_or_the_API_server_because_a_firewall_drops_tunnel_traffic_between_nodes_on_ACP.md). + +## Issue + +The global console shows a workload cluster as abnormal, the container overview does not open, Grafana opened from the platform UI returns an error, or monitoring and logs stay empty after a cluster is imported. After the global worker nodes are migrated, GitOps sync or business-cluster monitoring fails. Platform Inspection cannot connect to Prometheus. Typical error text: + +- Console request for a workload cluster: + + ```text + "status": "Failure", "code": 500, Get "https://erebus.cpaas-system.svc.cluster.local/kubernetes//version?timeout=10s": net/http: request canceled (Client.Timeout exceeded while awaiting headers) + ``` + +- Grafana opened from **Operations Center > Monitoring > Grafana**: `"reason": "Unknown"`, `"code": 500`, `Post "http://
:11780/clusters//grafana/login": dial tcp
:11780: i/o timeout`. +- A GitOps `ApplicationSet` shows Sync Failed while health is Healthy: `ComparisonError: Failed to load live state: failed to get cluster info for "https://:6443": error synchronizing cache state : Get "https://:6443/version?timeout=32s": dial tcp :6443: i/o timeout`. +- Every business cluster overview page shows the banner `监控组件异常,只能获取部分监控数据,请修复。` ("Monitoring component abnormal, only partial monitoring data can be obtained, please fix.") while the cluster status is normal. + +What the global cluster has to reach on a workload cluster: + +| Port | What uses it | +| --- | --- | +| `6443` | Kubernetes API. `erebus` in `cpaas-system` proxies the console to this port; Argo CD uses it for GitOps. | +| `11780` | Monitoring. `apollo` queries `http://:11780/clusters//prometheus/api/v1/query`; Grafana uses the same path. | +| `11781` | Logs, once log data is sunk to Elasticsearch on the workload cluster. | +| `16443` | A workload VIP port used as the API server address on some sites (Cause 4). | +| `2379` | etcd client. Required between active and standby in a disaster-recovery failover, together with `80` and `443`. | + +The workload cluster also has to reach the global cluster's image registry. For an imported external cluster to show logs and monitoring, open `6443`, `11780` and `11781` (once logs are sunk to that cluster) from every global node to the imported cluster's control plane (its VIP or the control-plane node addresses), and open access from the imported cluster to the global registry. + +## Root Cause + +### Cause 1: erebus times out on 6443 or apollo on 11780 because the path from some global nodes is closed + +`erebus` logs `timeout or abort` while calling `https://:6443`, and `apollo` logs `context deadline exceeded` for `http://:11780/clusters//prometheus/api/v1/query`. Every node in the workload cluster can be `Ready`, and `cpaas-system` and `kube-system` Pods on both clusters can look healthy. + +The path from the global nodes to that VIP is only partly open. A check from one global node, or only from the first master, is not enough: each global node needs `6443` and `11780` to the workload VIP. `11781` is the log path on the same VIP; closing it alone does not blank the container overview. Grafana from the platform UI uses `11780` as well, so an unopened `11780` to the address in the Grafana error makes monitoring look abnormal even when the monitoring components are healthy. Restarting `apollo` does not open a port that the firewall drops. + +### Cause 2: an imported cluster's monitoring endpoint fell back to the master internal IP + +A cluster was imported with a public VIP, and monitoring stays empty. From the `apollo` container, TCP to the workload node on `11780` fails. `kubectl get nodes -o wide` on the workload cluster shows no public `ExternalIP` on the masters, and nodes labeled `cpaas.io/system-alb` lack it too. The Endpoints object `cluster-proxy-` then contains the master internal IP instead of the VIP that global can route to. + +Two gaps stack: the firewall from global to the workload control plane does not allow `6443`, `11780` and `11781`, and the monitoring component still targets the fallen-back internal address. + +### Cause 3: Pods on newly migrated global workers cannot reach the business VIP + +After the global cluster's worker nodes are migrated, GitOps applications fail with `ComparisonError: Failed to load live state ... dial tcp :6443: i/o timeout` (or `Failed to load target state: failed to get cluster version for cluster "https://:6443" ... (retried 3 times).`), or monitoring disappears for every business cluster with the banner above, and the `apollo` log shows `Prometheus query error`. + +The Argo CD `argocd-gitops-application-controller-0` / `-1` Pods, or the global `apollo` Pods, run on the new global workers (role ``). From a new worker, `curl -v -k https://:6443` and `telnet 6443` hang at `Trying...`; from a global master the same `curl` returns `403 Forbidden` (`User "system:anonymous" cannot get path "/"`), which proves the path from the master is open. The firewall toward the business clusters allows only the older global node addresses. Platform Pods are not pinned to masters, so after the migration they run on workers that have no path: `6443` for Argo CD, `11780` and `11781` for `apollo`. + +### Cause 4: global cannot reach the workload cluster VIP on 16443 + +The workload cluster is reported abnormal with the `erebus ... version?timeout=10s ... Client.Timeout exceeded` error. On the global cluster, `kubectl get minfo -A | grep | grep -vi run` lists that cluster's ModuleInfo objects (for example `logcenter`, `velero`, `kubernetes`, `prometheus`, `cosi`) as `Failed`. Restarting `erebus`, `apollo` and `archon` changes nothing, and the workload nodes and `kube-system` are healthy. + +The workload nodes use `:16443` as the API server address (the `server` in `/etc/kubernetes/kubelet.conf`). From a workload node, `curl -k https://:16443` returns `403 Forbidden` for `system:anonymous`; from the global master, `curl -kv https://:16443` stops at `Trying ...` with no response. A network firewall blocks global's access to that VIP port. + +### Cause 5: Platform Inspection cannot reach Prometheus because the prdb domain resolves to an external load balancer + +Platform Inspection reports that Prometheus cannot be connected for a cluster, although Prometheus is healthy: the `prometheus` ModuleInfo is `Running`, `kubectl get ars -A | grep prome` shows `Synced` and `Ready`, the Prometheus, Alertmanager, Thanos query and Grafana Pods in `cpaas-system` are `Running`, and `sentry` logs no errors. `courier` connects using the address stored in `prdb`. That address is a domain name, and the domain resolves to the address of an external load balancer (for example an F5) that `courier` cannot reach. This was seen on ACP 3.14. + +## Resolution + +### Cause 1: erebus times out on 6443 or apollo on 11780 because the path from some global nodes is closed + +1. Confirm the gap from more than one global node: + + ```bash + nc -vz -w 3 6443 + nc -vz -w 3 11780 + nc -vz -w 3 11781 + ``` + +2. Open `6443` and `11780` from every global node to the workload VIP, and `11781` as well when log views must work, using the firewall's own syntax. + + > **Warning:** Opening these ports to a source wider than the global node prefix exposes the workload API and monitoring listener. Limit the source to the global node addresses and keep the previous rule set. To roll back, delete the rule you added. + +3. After `nc` succeeds from every global node, open the container overview and a monitoring page as the acceptance test. + +### Cause 2: an imported cluster's monitoring endpoint fell back to the master internal IP + +Only when the `cluster-proxy-` addresses are internal master IPs and the import VIP is a different, reachable address. + +1. Open `6443`, `11780` and `11781` from global to the workload control-plane addresses, including the public VIP, as in Cause 1. +2. On the global cluster, edit the monitoring ModuleInfo (`minfo`) of that workload cluster and set its access address under `spec` to the VIP that global can reach. Copy the address field from the monitoring `minfo` of a healthy imported cluster rather than guessing the key. + + ```bash + kubectl get endpoints -A | grep cluster-proxy- + kubectl get minfo -A | grep -i + kubectl get minfo -n -o yaml > /tmp/minfo.before.yaml + ``` + + > **Warning:** Saving a ModuleInfo rolls the monitoring stack for that cluster, and scrapes pause until the new Pods are Ready. Change only the address that points at the VIP. To roll back, run `kubectl apply -f /tmp/minfo.before.yaml`. + + ```bash + kubectl edit minfo -n + ``` + +3. After about five minutes, TCP `11780` from `apollo` to the VIP should succeed and the monitoring page should fill in. Also confirm that the workload cluster can reach the global registry on the registry port; a closed registry path fails image pulls independently of the monitoring page. + +### Cause 3: Pods on newly migrated global workers cannot reach the business VIP + +Only when the failing Pod runs on a new global worker and the same `nc` succeeds from a global master. + +1. Request the permanent fix: a firewall rule from every global worker node to each business cluster VIP on `6443` (GitOps) and on `11780` and `11781` (monitoring). +2. As a temporary measure, delete the affected Pods so they are scheduled again, and check where the replacements land; the scheduler can pick another new worker. + + > **Warning:** Deleting `application-controller` pauses GitOps reconciliation, and deleting `apollo` blanks monitoring pages, until the new Pod is Ready. Make sure no sync you depend on is in progress. No rollback is needed; the controller recreates the Pod. + + ```bash + kubectl get nodes + kubectl get pod -A -o wide | grep -E 'argocd-gitops-application-controller|apollo' + kubectl -n delete pod + kubectl get pod -A -o wide | grep -E 'argocd-gitops-application-controller|apollo' + ``` + +### Cause 4: global cannot reach the workload cluster VIP on 16443 + +Only when the port fails from global and answers from inside the workload cluster. Have the network team allow the global network segment to the workload cluster segment on TCP `6443`, `11780`, `15012` and `16443`. Restarting global components does not substitute for it. + +```bash +grep server /etc/kubernetes/kubelet.conf +curl -k https://:16443 +for p in 6443 11780 15012 16443; do nc -vz -w 3 $p; done +kubectl get minfo -A | grep | grep -vi run +``` + +Run the first two on a workload node (`403 Forbidden` for `system:anonymous` is the healthy answer) and the rest on a global master. After the firewall change every port reports connected, the ModuleInfo objects leave `Failed`, and the clusters show Normal. + +### Cause 5: Platform Inspection cannot reach Prometheus because the prdb domain resolves to an external load balancer + +Only when the domain in `prdb` resolves to an address `courier` cannot reach. Add a CoreDNS static record that maps that domain to the global cluster's actual API server VIP (**Host Alias** in the cluster plugin **Alauda Container Platform Cluster DNS**), then rerun the inspection. + +```bash +kubectl get prdb -A -o yaml | grep -iE 'https?://' +kubectl get pod -A | grep -E 'courier|warlock' +kubectl -n exec -- nslookup +kubectl get minfo -A | grep prome +``` + +After the record points the domain at the global API server VIP, the inspection completes without the Prometheus fault. + +## Diagnostic Steps + +1. From a global node that is not the only master you usually test, probe the workload VIP. A timeout on `6443` or `11780` is Cause 1. A timeout only on `11781` blocks sunk logs and does not explain an `erebus` API timeout: + + ```bash + nc -vz -w 3 6443 + nc -vz -w 3 11780 + nc -vz -w 3 11781 + kubectl -n cpaas-system logs deploy/erebus --tail=50 | grep -E 'timeout or abort|6443' + kubectl -n cpaas-system logs deploy/apollo --tail=50 | grep -E '11780|context deadline exceeded' + ``` + +2. On the workload cluster, see whether the proxy target drifted to an internal IP. Internal addresses in `cluster-proxy-` while the cluster was imported with a public VIP is Cause 2: + + ```bash + kubectl get nodes -o wide + kubectl get nodes -l cpaas.io/system-alb -o yaml | grep -A3 addresses + kubectl get endpoints -A | grep cluster-proxy + ``` + +3. Compare `nc` to the business VIP from a new global worker and from a master (Cause 3), and from global versus inside the workload cluster (Cause 4). Resolve the `prdb` domain from `courier` for inspection failures (Cause 5). + +4. If TCP to `11780` and `11781` succeeds and the monitoring page is still empty, the port is not the problem; check the HTTPS answer on `https://:11780` and the `apollo` and `courier` logs for cache or `cluster not found` errors. diff --git a/docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md b/docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md index 177349739..febc31a95 100644 --- a/docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md +++ b/docs/en/solutions/acp/NodeLocal_DNSCache_S2_Workarounds.md @@ -10,11 +10,13 @@ id: KB261000048 # Temporary Workarounds for Common NodeLocal DNSCache Issues in Field Environments -This article provides temporary workarounds for three common NodeLocal DNSCache issues in field environments: +This article provides temporary workarounds for five common NodeLocal DNSCache issues in field environments: - DNS resolution is affected when the `node-cache` Pod on a node is unavailable. - Health check port `8080` conflicts. - External monitoring systems or dashboards cannot collect NodeLocal DNSCache metrics. +- A custom domain added as a hosts record in CoreDNS is not resolved through NodeLocal DNSCache. +- `node-local-dns` Pods remain after the plugin is uninstalled. These workarounds are temporary. Manual changes may be overwritten after plugin upgrade, plugin reinstall, platform reconciliation, chart re-rendering, or node rebuild. Perform the change in a maintenance window. @@ -154,6 +156,39 @@ ss -ltnp | grep ':8080' The expected result is that `18080` is listened on by NodeLocal DNSCache, and `8080` is no longer listened on by NodeLocal DNSCache. +### Reverse case: a host process already holds port 8080 + +**Symptom:** While NodeLocal DNSCache is deployed, the `node-local-dns` Pod on one node does not start, and its log shows: + +```text +listen tcp 169.254.20.10:8080: bind: address already in use +``` + +**Cause:** A process on that node already listens on port `8080` on all addresses (`0.0.0.0:8080`, shown by `lsof` as `TCP *:webcache (LISTEN)`). A wildcard listener holds the port for every local address, so the cache cannot bind its health address on `8080` and exits on that node only. The listener can be an ordinary host service, for example a Java service started from a user's home directory under a non-root account; `/proc//cgroup` shows whether it is a container workload. + +**Resolution:** Identify the process and its owner first: + +```bash +kubectl -n kube-system logs --previous | grep 'address already in use' +ss -ltnp | grep ':8080' +lsof -i :8080 +ls -l /proc//exe /proc//cwd +cat /proc//cgroup +systemctl status +``` + +Then either move the NodeLocal DNSCache health port as described above, or, if the owner of the host service agrees, stop that service. If `systemctl status ` names a unit, stop the unit (`systemctl stop `), because a supervised service starts again after a plain `kill`; otherwise use `kill `. + +> **Warning:** Stopping the process stops the host service that owns port `8080` on this node, and its in-flight work is lost. Do it only after the service owner confirms it can stop on this node or move to another port or host. To roll back, start the service again with its own start method; `node-local-dns` then fails to bind again on this node. + +Confirm that the port is free (the `ss` output is empty), then recreate the cache Pod on that node: + +```bash +ss -ltnp | grep ':8080' +kubectl -n kube-system delete pod +kubectl -n kube-system get pod -o wide | grep node-local-dns +``` + ## Issue 3: NodeLocal DNSCache metrics cannot be collected externally **Symptom:** External monitoring systems or dashboards cannot access NodeLocal DNSCache metrics. @@ -202,3 +237,60 @@ Confirm that the Corefile is updated, and verify that metrics can be accessed fr ```bash kubectl -n "$NS" get cm "$CM" -o yaml | grep 'prometheus' ``` + +## Issue 4: A CoreDNS hosts record is not resolved through NodeLocal DNSCache + +**Symptom:** A domain was added as a hosts record in CoreDNS (for example under **Host Alias**). In a test Pod, `nslookup ` returns the address, but plain `nslookup ` fails, and the Pod's `/etc/resolv.conf` points at the NodeLocal DNSCache address (for example `169.254.20.10`). When the same record is added to the upstream DNS instead, both forms resolve. + +**Cause:** NodeLocal DNSCache reads its own Corefile from its ConfigMap, not from `cpaas-coredns`. Its Corefile has server blocks only for the cluster zones, such as `cluster.local:53` and `in-addr.arpa:53`, each with `forward . { force_tcp }`, plus a catch-all `.:53` block. Only the zones that have their own block are forwarded to CoreDNS. Any other name, including a custom domain, falls into `.:53` and goes to the node's upstream DNS without passing through CoreDNS, so a record that exists only in CoreDNS is never seen on the default path. A record in the upstream DNS is visible on both paths because both end at that upstream. + +**Resolution:** Add a server block for the custom domain to the NodeLocal DNSCache ConfigMap, so that queries for that zone are forwarded to CoreDNS. Copy the existing `in-addr.arpa:53` block exactly and change only the zone name. Keep the `bind` line and the `forward` target as they appear in the live ConfigMap, including any `__PILLAR__` token. For example: + +```text +example.internal:53 { + errors + cache 30 + reload + bind 169.254.20.10 __PILLAR__DNS__SERVER__ + forward . { + force_tcp + } + prometheus 169.254.20.10:9253 +} +``` + +Save the ConfigMap before editing: + +```bash +kubectl -n "$NS" get cm "$CM" -o jsonpath='{.data.Corefile}{"\n"}' +kubectl -n "$NS" get cm "$CM" -o yaml > /tmp/nodelocaldns.before.yaml +``` + +> **Warning:** A syntax error in this Corefile stops the cache from loading the new configuration, and every Pod that uses NodeLocal DNSCache on the affected nodes can lose DNS. Do this only when the record already answers `nslookup `. To roll back, run `kubectl apply -f /tmp/nodelocaldns.before.yaml`. + +```bash +kubectl -n "$NS" edit cm "$CM" +``` + +The `reload` plugin picks up the new Corefile after the kubelet refreshes the mounted ConfigMap, which can take about a minute. If the new zone is still not served after that, restart the DaemonSet; DNS from Pods on a node can pause while its cache Pod restarts. + +```bash +kubectl -n "$NS" rollout restart ds "$DS" +kubectl -n "$NS" rollout status ds "$DS" +kubectl exec -n -- nslookup +``` + +The last command, without an explicit server, is the acceptance test. + +## Issue 5: `node-local-dns` Pods remain after the plugin is uninstalled + +**Symptom:** The NodeLocal DNSCache plugin was uninstalled and the page reports the uninstall as successful, but `kubectl -n kube-system get ds,pod | grep node-local-dns` still lists the DaemonSet and its Pods. + +**Cause:** This is expected product behavior. The `node-local-dns` DaemonSet and its Pods remaining after the plugin is uninstalled does not mean that the uninstall failed. + +**Resolution:** No recovery action is required. To confirm that what remains is the retained DaemonSet: + +```bash +kubectl -n kube-system get ds -o wide | grep -iE 'node-?local' +kubectl -n kube-system get pod -o wide | grep node-local-dns +``` diff --git a/docs/en/solutions/asm-otel_collectors_keep_restarting_Service_Mesh_JVM_monitoring_has_no_data_or_an_otel_instance_cannot_be_deleted.md b/docs/en/solutions/asm-otel_collectors_keep_restarting_Service_Mesh_JVM_monitoring_has_no_data_or_an_otel_instance_cannot_be_deleted.md new file mode 100644 index 000000000..e41340791 --- /dev/null +++ b/docs/en/solutions/asm-otel_collectors_keep_restarting_Service_Mesh_JVM_monitoring_has_no_data_or_an_otel_instance_cannot_be_deleted.md @@ -0,0 +1,181 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Service Mesh +ProductsVersion: + - "3.18" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# asm-otel collectors keep restarting, Service Mesh JVM monitoring has no data, or an otel instance cannot be deleted + +## Issue + +One of the following is seen: + +- The platform components `asm-otel-backend-collector` and `asm-otel-collector` keep restarting and cannot be restarted cleanly. The image field shown for them in the console keeps changing. +- Service Mesh JVM monitoring, or the `OpenTelemetry Java JVM` dashboard, shows no data for a Java application. +- While OpenTelemetry is being uninstalled so that OpenTelemetry v2 can be installed, the otel instance is deleted but stays in place, and the OpenTelemetry v2 installation is blocked. + +Related articles: "OpenTelemetry Collector logs cannot list resources pods because its ServiceAccount has no ClusterRoleBinding on ACP" covers a Collector that cannot list Pods. "Java auto-instrumentation init container is missing or the Java agent lands next to istio-proxy on Service Mesh v2" covers a pod where the Java agent was never injected. + +## Root Cause + +### Cause 1: a leftover 3.x `opentelemetry-operator` in `cpaas-system` fights the asm-otel collectors + +A Deployment named `opentelemetry-operator` runs in `cpaas-system`. That Deployment belongs only to 3.x environments. It conflicts with the current `asm-otel-*` components and keeps changing them, so both collectors restart. The Mesh v2 OpenTelemetry operator that serves tracing (for example `opentelemetry-operator-controller-manager` in `istio-system`) is not this leftover. Removing the old Deployment returns the collectors to normal; other objects left by that old operator should be cleaned up so they do not cause a later fault. + +### Cause 2: `asm-otel` was OOMKilled + +JVM metrics are not collected because the `asm-otel` collection component was OOMKilled. Raising its CPU and memory allocation lets it run and collect again. + +### Cause 3: the otel instance still carries a finalizer + +The OpenTelemetry Operator was uninstalled before the otel instance was removed, so nothing clears the instance's `finalizers` and the delete never completes. Removing the finalizer lets the delete finish, and the OpenTelemetry v2 installation is no longer blocked. + +### Cause 4: the Java agent cannot resolve the exporter endpoint in the `Instrumentation` + +The Java agent is injected (the auto-instrumentation init container ran and is `terminated`), but the exporter endpoint in the `Instrumentation` resource uses a full service name that does not resolve from the application Pod, for example a mistyped or wrong cluster domain in `http://otel-collector..svc.cluster.local:4318`. The agent's metric export times out, so no JVM series reach the Collector. The application container log shows: + +```text +ERROR io.opentelemetry.exporter.internal.http.HttpExporter - Failed to export metrics. The request could not be executed. Full error message: timeout +java.io.InterruptedIOException: timeout +``` + +The documented endpoint form `http://.:4318` resolves through the Pod's DNS search path. + +## Resolution + +### Cause 1: a leftover 3.x `opentelemetry-operator` in `cpaas-system` fights the asm-otel collectors + +1. Confirm that both collectors restart and that an `opentelemetry-operator` Deployment runs in `cpaas-system`: + + ```bash + kubectl get pod -A | grep -i opentelemetry-operator + kubectl get deploy,pods -n cpaas-system | grep -E 'opentelemetry-operator|asm-otel' + ``` + + Two operators, an old `opentelemetry-operator-` in `cpaas-system` (typically more than a year old) and `opentelemetry-operator-controller-manager-` in `istio-system`, match this cause. + +2. Back up and delete the old Deployment. + + > **Warning:** This stops the 3.x operator in `cpaas-system`. Only when both `asm-otel-backend-collector` and `asm-otel-collector` restart, the image field keeps changing, and this Deployment is in `cpaas-system`. Rollback: `kubectl apply -f /tmp/opentelemetry-operator-deploy.bak.yaml`. + + ```bash + kubectl get deploy -n cpaas-system opentelemetry-operator -o yaml > /tmp/opentelemetry-operator-deploy.bak.yaml + kubectl delete deploy -n cpaas-system opentelemetry-operator + kubectl get pods -A | grep -E 'asm-otel|opentelemetry-operator' + ``` + + The collectors should stay Running and the image field should stop changing. + +3. Clean up the other objects of the old operator that still exist. List them first: + + ```bash + kubectl get mutatingwebhookconfiguration,validatingwebhookconfiguration | grep opentelemetry-operator + kubectl get certificate,issuer,svc,role,rolebinding,sa,servicemonitor -n cpaas-system | grep opentelemetry-operator + kubectl get clusterrole,clusterrolebinding | grep opentelemetry-operator + ``` + + > **Warning:** These webhook, RBAC, certificate, and Service objects are the old operator's admission and identity; cluster-scoped deletes affect every namespace. Only after the Deployment in step 2 is gone, and only for objects with these exact names. Back up each object with `kubectl get [-n cpaas-system] -o yaml > /tmp/.bak.yaml` first. Rollback: `kubectl apply` of the backup (remove `resourceVersion`, `uid`, and `managedFields` from a webhook backup if apply complains). + + ```bash + kubectl delete mutatingwebhookconfiguration opentelemetry-operator-mutation + kubectl delete validatingwebhookconfiguration opentelemetry-operator-validation + kubectl delete certificate opentelemetry-operator-serving-cert -n cpaas-system + kubectl delete issuer opentelemetry-operator-selfsigned-issuer -n cpaas-system + kubectl delete svc opentelemetry-operator-webhook opentelemetry-operator -n cpaas-system + kubectl delete role,rolebinding opentelemetry-operator-leader-election -n cpaas-system + kubectl delete clusterrole,clusterrolebinding opentelemetry-operator-manager + kubectl delete sa opentelemetry-operator -n cpaas-system + kubectl delete servicemonitor opentelemetry-operator -n cpaas-system + ``` + + Run a line only for an object the listing showed. + +### Cause 2: `asm-otel` was OOMKilled + +1. Confirm the last state is OOMKilled: + + ```bash + kubectl get pods -A | grep asm-otel + kubectl describe pod -n | grep -A8 'Last State' + ``` + +2. Raise CPU and memory. On a ServiceMesh install, component sizes are set under `spec.componentConfig` of the ServiceMesh resource (entries `asmOtelCollector` and `asmOtelCollectorLB` for `asm-otel-collector` and `asm-otel-backend-collector`). Change them there so the next reconcile does not restore the old limits. + + > **Warning:** A higher request can leave the Pod Pending until a node has the capacity; a limit that is still too low leaves it OOMKilled. Only when JVM monitoring is empty and the Pod's last state is OOMKilled. Rollback: apply the backup. + + ```bash + kubectl get servicemesh -A -o yaml | grep -n -i -E 'asmOtel' + kubectl get servicemesh -n -o yaml > /tmp/servicemesh-otel.bak.yaml + kubectl edit servicemesh -n + ``` + +3. The new Pod should not be OOMKilled and its log should progress; JVM series are then collected again. + +### Cause 3: the otel instance still carries a finalizer + +1. Confirm the instance has a `deletionTimestamp` and a finalizer, and that the old operator is already gone: + + ```bash + kubectl get opentelemetrycollector,instrumentation -A + kubectl get -n -o jsonpath='{.metadata.deletionTimestamp}{" "}{.metadata.finalizers}{"\n"}' + kubectl get csv -A | grep -i opentelemetry + ``` + +2. Remove the finalizer. + + > **Warning:** The instance is deleted without the operator's cleanup; objects it created that are not garbage-collected through owner references stay and must be removed by hand. Only when the instance has a `deletionTimestamp`, a finalizer is set, and the old OpenTelemetry Operator is uninstalled. Rollback: the delete is irreversible; recreate the instance from the backup after removing `metadata.finalizers`, `deletionTimestamp`, `resourceVersion`, and `uid`. + + ```bash + kubectl get -n -o yaml > /tmp/otel-instance.bak.yaml + kubectl patch -n --type=merge -p '{"metadata":{"finalizers":null}}' + ``` + + `` is the kind of the stuck instance, for example `opentelemetrycollector`. Then continue with the OpenTelemetry v2 installation. + +### Cause 4: the Java agent cannot resolve the exporter endpoint in the `Instrumentation` + +1. Read the endpoint and test it from the application Pod (or any Pod in the same namespace that has `curl`): + + ```bash + kubectl get instrumentation -n -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.exporter.endpoint}{"\n"}{end}' + kubectl -n logs -c --tail=200 | grep -i 'Failed to export' + kubectl -n exec -- curl -s -o /dev/null -w '%{http_code}\n' http://.:4318 + ``` + + `could not resolve` for the configured name and an HTTP code (for example `404`) for the short name means the Collector port is reachable and only the name is wrong. + +2. Change the endpoint to `http://.:4318`. + + > **Warning:** The new endpoint applies to Pods injected after the change; restart the instrumented workloads so they pick it up, which recreates their Pods. Rollback: restore the previous endpoint value and restart again. + + ```bash + kubectl -n patch instrumentation --type=merge \ + -p '{"spec":{"exporter":{"endpoint":"http://.:4318"}}}' + kubectl -n rollout restart deployment + ``` + +3. Also check that the workload uses an `Instrumentation` it can find: declare one in each application namespace, or reference one in another namespace with the annotation value `/`, for example `instrumentation.opentelemetry.io/inject-java: "/"`. If the Collector has no logs pipeline, setting the environment variable `OTEL_LOGS_EXPORTER=none` in the `Instrumentation` (`spec.env`) stops the agent from exporting logs to it. + +4. After the restart, the `Failed to export metrics` errors should stop and the JVM dashboard should start receiving series. + +## Diagnostic Steps + +```bash +kubectl get pods -A | grep -E 'asm-otel|opentelemetry-operator' +kubectl get deploy -n cpaas-system | grep opentelemetry-operator +kubectl get instrumentation -A +``` + +| What you see | Cause | +| --- | --- | +| Both asm-otel collectors restart, image field keeps changing, `opentelemetry-operator` Deployment in `cpaas-system` next to the one in `istio-system` | 1 | +| JVM monitoring empty, `asm-otel` last state OOMKilled | 2 | +| otel instance stays after delete, `finalizers` set, operator already uninstalled | 3 | +| JVM dashboard empty, agent injected, application log shows `Failed to export metrics ... timeout`, configured endpoint does not resolve | 4 | diff --git a/docs/en/solutions/auth-controller2_webhook_certificate_missing_or_log_burst_monitoring_query_credentials_and_Connector_tokens_on_ACP.md b/docs/en/solutions/auth-controller2_webhook_certificate_missing_or_log_burst_monitoring_query_credentials_and_Connector_tokens_on_ACP.md new file mode 100644 index 000000000..11a41eb47 --- /dev/null +++ b/docs/en/solutions/auth-controller2_webhook_certificate_missing_or_log_burst_monitoring_query_credentials_and_Connector_tokens_on_ACP.md @@ -0,0 +1,155 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.18" + - "4.2" + - "4.3.3" +--- + +# auth-controller2 webhook certificate missing or log burst, monitoring query credentials, and Connector tokens on ACP + +## Issue + +One of these is reported on the platform: + +- `auth-controller2` never starts. The Pod event is `MountVolume.SetUp failed for volume "cert": secret "auth-controller2-webhook-cert" not found`, and an upgrade of `3rdparty/chart-cert-manager` fails while creating a ResourceQuota at `metis-webhook.cpaas-system.svc:443/check-resourcequota` with `x509: certificate signed by unknown authority`. +- One daily `log-platform` index is suddenly many times larger than its neighbours (about a terabyte or more), and the question is which component produced it, or whether `auth-controller2` logs surge and the controller will crash while administrators are adding permissions for users. +- An external caller needs a credential for a Prometheus or VictoriaMetrics query, or the API token page shows `Permissions required. Please contact your administrator.` +- `kube-prometheus-exporter-node` uses a different username on some clusters. +- A Kubernetes-type Connector shows authentication failed. + +## Root Cause + +### Cause 1: `auth-controller2-webhook-cert` is not issued because a second cert-manager produces conflicting CertificateRequests + +The certificate volume of `auth-controller2` is not optional (`optional: false`), so the Pod cannot start without the Secret `auth-controller2-webhook-cert` in `cpaas-system`. When a non-platform cert-manager and the platform cert-manager are both present, more than one `CertificateRequest` exists for the certificate, and admission to the `metis-validation` webhook fails with `x509: certificate signed by unknown authority` (also logged as `tls: failed to verify certificate: x509: certificate signed by unknown authority`). The webhook Secret comes back after `metis-validation` is set to ignore failures, the extra `CertificateRequest` objects are removed (one is kept), and cert-manager is restarted. This is not the case where the metis Pod is down and does not answer. + +### Cause 2: a one-time `auth-controller2` log burst while user permissions are being added + +A daily `log-platform` index can grow far beyond its neighbours when `auth-controller2` writes a burst of log lines while operations staff add permissions for users. In one such case the large day held about 62 million documents and about 2.1 TB, against about 18 to 39 million documents and 8 to 23 GB on neighbouring days. A product-log query on the two `auth-controller2` Pods returned about 13 million entries, about 10 million of them in a single interval, with the surrounding intervals near zero. During the burst, container `auth-controller2-manager` rose from about 750 Mi to about 1.1 Gi and returned to baseline, far below its 6144 Mi limit, so the Pod was not near an OOM kill; a "utilization from requests" figure above 1000 % only reflects its small memory request. No crash occurred, the burst did not repeat, and later daily indices returned to normal size. Why the permission changes produced that many lines was not determined. + +### Cause 3: external Prometheus or VictoriaMetrics queries need the matching credential + +Platform API queries use a platform API token. Native Prometheus queries use the basic-auth Secret `kube-prometheus-prometheus-basic-auth` in `cpaas-system`. Native VictoriaMetrics queries use `victoriametrics-basic-auth`. The API token page hides token creation from users who lack that page permission, which produces the `Permissions required` message; the monitoring endpoints below are an alternative for a monitoring caller that reads every cluster. + +### Cause 4: the `kube-prometheus-exporter-node` credential changed in 4.2 + +`kube-prometheus-exporter-node` does not use the same username and password on every cluster. Before 4.2 it uses a fixed credential. From 4.2 the username is `monitor@cpaas.io` and the password is a random string. The difference is a security policy change in 4.2, not a cluster that lost its configuration. + +### Cause 5: a Kubernetes Connector uses an endpoint that does not match its token + +The Connector object is created and then shows authentication failed. The usual mismatches are: the endpoint is a cluster API server (`https://192.0.2.10:6443`) but the token is a platform API token, or the endpoint is the ACP platform address but the token is a cluster ServiceAccount token; the Secret that was inspected is not the Secret named on the Connector; or `secretRef` is not under `spec.auth` with `bearerTokenAuth`. A token created with `kubectl create token` is valid for one hour unless the API server lifetime flags say otherwise. + +## Resolution + +### Cause 1: `auth-controller2-webhook-cert` is not issued because a second cert-manager produces conflicting CertificateRequests + +1. Confirm that the Secret is absent and the Pod event names it: + + ```bash + kubectl -n cpaas-system get secret auth-controller2-webhook-cert + kubectl -n cpaas-system describe pod + kubectl get validatingwebhookconfiguration,mutatingwebhookconfiguration -o name | grep metis-validation + kubectl get certificaterequest.cert-manager.io -A + ``` + +2. Set `failurePolicy: Ignore` on the `metis-validation` webhook entries (at least the one whose service is `metis-webhook` and path `/check-resourcequota`). + + > **Warning:** `failurePolicy: Ignore` lets requests through while the webhook certificate is untrusted, including requests the webhook would have rejected. Precondition: only when `auth-controller2-webhook-cert` is missing and admission to `metis-webhook.cpaas-system.svc:443/check-resourcequota` fails with `x509: certificate signed by unknown authority`. Rollback: `kubectl apply -f /tmp/metis-validation.bak.yaml`. + + ```bash + kubectl get validatingwebhookconfiguration metis-validation -o yaml > /tmp/metis-validation.bak.yaml + kubectl edit validatingwebhookconfiguration metis-validation + ``` + +3. Keep one `CertificateRequest`, delete the extra ones created by the non-platform cert-manager, and restart cert-manager. + + > **Warning:** Deleting a `CertificateRequest` drops that issuance. Precondition: only when both cert-managers have produced multiple `CertificateRequest` objects for this certificate and the Secret is still missing. Save the YAML first. Re-applying a deleted `CertificateRequest` retries issuance. + + ```bash + kubectl get certificaterequest.cert-manager.io -A -o yaml > /tmp/cr.bak.yaml + kubectl delete certificaterequest.cert-manager.io -n + kubectl get deploy -A | grep cert-manager + kubectl -n rollout restart deploy/ + kubectl -n cpaas-system get secret auth-controller2-webhook-cert + ``` + +4. `auth-controller2` starts after the Secret exists. Once the webhook certificate verifies (the `x509` error is gone), restore the previous `failurePolicy` of `metis-validation` from `/tmp/metis-validation.bak.yaml`. + +Do not install a second cert-manager next to the platform cert-manager. + +### Cause 2: a one-time `auth-controller2` log burst while user permissions are being added + +1. Find the source of the volume before treating it as business logs. In the Operations Center, open **Logs > Query**, select product logs, narrow the time range to the large day, filter by the `auth-controller2` Pod names, and look at the histogram for a single spike. + +2. Check the Pod's memory in monitoring for the same interval, and its current state: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep auth-controller2 + kubectl -n cpaas-system get pod -o jsonpath='{range .status.containerStatuses[*]}{.name}{" restarts="}{.restartCount}{" lastState="}{.lastState}{"\n"}{end}' + kubectl -n cpaas-system top pod --containers + ``` + + A single spike at the time permissions were being granted, memory back at baseline, and no restart or `OOMKilled` last state need no action. + +3. If the burst repeats, collect a sample of the repeated lines while it happens, together with the permission operations being performed, for the support team: + + ```bash + kubectl -n cpaas-system logs -c auth-controller2-manager --since=10m > /tmp/auth-controller2-burst.log + wc -l /tmp/auth-controller2-burst.log + ``` + +### Cause 3: external Prometheus or VictoriaMetrics queries need the matching credential + +Platform API, with a platform API token: + +```text +https:///v2/metrics//prometheus/query +``` + +Native Prometheus, with the username and password from the Secret: + +```bash +kubectl get secret -n cpaas-system kube-prometheus-prometheus-basic-auth -o jsonpath='{.data.username}' | base64 -d; echo +kubectl get secret -n cpaas-system kube-prometheus-prometheus-basic-auth -o jsonpath='{.data.password}' | base64 -d; echo +``` + +```text +https:///clusters//prometheus-0/api/v1/query +``` + +VictoriaMetrics uses the Secret `victoriametrics-basic-auth` and paths of the form `/clusters/global/vmselect/api/v1/query` and `/api/v1/query_range` under the same prefix. Replacing `vmselect` in that path with `prometheus-0` gives the native Prometheus form; the query APIs are the same. + +### Cause 4: the `kube-prometheus-exporter-node` credential changed in 4.2 + +Read the credential that the cluster actually uses and compare it with the cluster version. On 4.2 and later, expect username `monitor@cpaas.io` and a random password. No reset is needed to make clusters of different versions "match". + +### Cause 5: a Kubernetes Connector uses an endpoint that does not match its token + +1. Match the endpoint and the token. A cluster API server address takes a bearer token from that cluster. An ACP platform address takes a platform API token. The Secret name and namespace on the Connector must be the Secret you filled in. + +2. To use a ServiceAccount token for a cluster API server endpoint, create one for the ServiceAccount the Connector should act as, store it in the Secret key `token`, and reference the Secret under `spec.auth.bearerTokenAuth.secretRef` (name and namespace). + + > **Warning:** Anyone who can read the Connector Secret can call the API as that ServiceAccount until the token expires (one hour by default). Precondition: only when the Connector endpoint and this ServiceAccount are the pair that should authenticate. Rollback: replace the `token` value or delete the key; a token already issued stays valid until it expires. The API server flags `--service-account-extend-token-expiration` and `--service-account-max-token-expiration` affect the lifetime of tokens issued later. + + ```bash + kubectl -n create token + ``` + +3. After the endpoint, the token type, and `secretRef` agree, the Connector authenticates and can be used from a pipeline Task. + +## Diagnostic Steps + +```bash +kubectl -n cpaas-system get pods | grep auth-controller +kubectl -n cpaas-system get secret auth-controller2-webhook-cert +kubectl -n cpaas-system get secret kube-prometheus-prometheus-basic-auth +``` + +- Event `secret "auth-controller2-webhook-cert" not found`, a `metis-validation` `x509` error, a second cert-manager, and more than one `CertificateRequest`: Cause 1. If the metis Pod itself is not Running, the webhook does not answer at all, which is a different problem. +- One daily `log-platform` index many times larger than its neighbours, and a product-log query showing a single `auth-controller2` spike at the time user permissions were added, with no restart: Cause 2. +- An external metrics query needs a credential: Cause 3. Exporter username differs by cluster version: Cause 4. +- Connector authentication failed: Cause 5. diff --git a/docs/en/solutions/base_AppRelease_stays_failed_because_the_upgrade-sysctl-config_BroadcastJob_cannot_finish_on_a_large_cluster_on_ACP.md b/docs/en/solutions/base_AppRelease_stays_failed_because_the_upgrade-sysctl-config_BroadcastJob_cannot_finish_on_a_large_cluster_on_ACP.md new file mode 100644 index 000000000..4327fe42a --- /dev/null +++ b/docs/en/solutions/base_AppRelease_stays_failed_because_the_upgrade-sysctl-config_BroadcastJob_cannot_finish_on_a_large_cluster_on_ACP.md @@ -0,0 +1,155 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# base AppRelease stays failed because the upgrade-sysctl-config BroadcastJob cannot finish on a large cluster on ACP + +## Issue + +On a cluster with many nodes (more than a hundred), the `base` AppRelease (`ar`, also shown as `ars base`) in `cpaas-system` stays in a failed state. In `cpaas-system`, the BroadcastJob `upgrade-sysctl-config` is created again and again and never completes: + +```bash +kubectl -n cpaas-system get ar base +kubectl -n cpaas-system get broadcastjob upgrade-sysctl-config +``` + +The same AppRelease can also stay in `syncing` on a cluster of any size, for example when the pre-upgrade check of a business cluster shows its base component (Cluster Base, AppRelease `base`) in `syncing` and it never finishes, or a business cluster ARS stays `syncing` for a long time. `kubectl get bcj -n cpaas-system` then shows a BroadcastJob (`broadcastjobs.operator.alauda.io`, short name `bcj`) whose `SUCCEEDED` count stays below `DESIRED`, or with `FAILED` greater than `0`. Adding a dummy key to the ARS values to force a new sync does not help. + +## Root Cause + +The `base` AppRelease includes the BroadcastJob `upgrade-sysctl-config`, which runs a Pod on every node. The AppRelease has a timeout (`spec.timeout`) within which its resources must become ready. On a large cluster the BroadcastJob cannot finish on all nodes within that timeout. The sentry component that watches the release then judges the operation as timed out and runs it again. The new run starts over and times out the same way, so the `base` AppRelease never becomes healthy. + +### A Pod of upgrade-sysctl-config was Evicted + +`kubectl get job,bcj -n cpaas-system` shows `upgrade-sysctl-config` with, for example, `DESIRED 5`, `ACTIVE 0`, `SUCCEEDED 4`, `FAILED 1`, and one `upgrade-sysctl-config-` Pod in `cpaas-system` is `Evicted` while the others are `Completed`. Because of the failed Pod the BroadcastJob never counts all nodes as succeeded, and the base component stays in `syncing`. Ordinary Jobs in the same list (for example `keep-marketplace-resources`, `Complete`) are not involved. Deleting the Evicted Pod lets the base component recover. + +### update-containerd-config does not finish + +The `sentry` log ties the long `syncing` state of a business cluster ARS to BroadcastJob `update-containerd-config`, and some of its jobs are stuck. `update-containerd-config` synchronizes and updates the containerd configuration on the nodes and is triggered whenever the business cluster ARS is updated. Raising the BroadcastJob `parallelism` to `10` lets every job finish with `success`, and the ARS returns to normal. Why individual jobs get stuck is not known. + +## Resolution + +Use one of the two options. + +### Option 1: delete the BroadcastJob + +> **Warning:** Deleting `upgrade-sysctl-config` stops any of its Pods that are still running, so the sysctl update may not have run on every node. Do this only when the diagnostic steps show that the `base` AppRelease is failing on this BroadcastJob and that it keeps restarting. Save the object first. Rollback: `kubectl apply -f /tmp/upgrade-sysctl-config.yaml`. + +```bash +kubectl -n cpaas-system get broadcastjob upgrade-sysctl-config -o yaml > /tmp/upgrade-sysctl-config.yaml +kubectl -n cpaas-system delete broadcastjob upgrade-sysctl-config +kubectl -n cpaas-system get ar base -w +``` + +### Option 2: give the AppRelease more time + +> **Warning:** While `cluster-transformer` is scaled to zero, that platform controller is not running on this cluster. Keep the window short and scale it back as soon as the `base` AppRelease is healthy. Rollback: scale `cluster-transformer` back to the recorded replica count and patch `spec.timeout` back to the recorded value. + +1. Record the current replica count of `cluster-transformer` and the current timeout of the `base` AppRelease: + + ```bash + kubectl -n cpaas-system get deploy cluster-transformer -o jsonpath='{.spec.replicas}{"\n"}' + kubectl -n cpaas-system get ar base -o jsonpath='{.spec.timeout}{"\n"}' + ``` + +2. Stop `cluster-transformer` and raise the timeout (in seconds): + + ```bash + kubectl -n cpaas-system scale deploy/cluster-transformer --replicas=0 + kubectl -n cpaas-system patch ar base --type merge -p '{"spec":{"timeout":1800}}' + ``` + +3. Wait until the `base` AppRelease is healthy, then scale `cluster-transformer` back to the recorded replica count: + + ```bash + kubectl -n cpaas-system get ar base -w + kubectl -n cpaas-system scale deploy/cluster-transformer --replicas= + ``` + +The two options above apply when the BroadcastJob is only slow. The following sections apply to the other two causes in Root Cause. + +### Evicted Pod in upgrade-sysctl-config: delete the Evicted Pod + +1. Confirm the signature and read the eviction message, which names the node resource that ran short: + + ```bash + kubectl get job,bcj -n cpaas-system + kubectl -n cpaas-system get pods -o wide | grep upgrade-sysctl-config + kubectl -n cpaas-system get pod -o jsonpath='{.status.phase} {.status.reason}: {.status.message}{"\n"}' + ``` + +2. Check that the node no longer reports that pressure; a node still under memory or disk pressure has to be cleared first, otherwise a new Pod there can be evicted again: + + ```bash + kubectl describe node | grep -E 'MemoryPressure|DiskPressure|PIDPressure' + ``` + +3. Save the Evicted Pod, because its status is the only record of the eviction: + + ```bash + kubectl -n cpaas-system get pod -o yaml > "$HOME/.yaml" + ``` + +> **Warning:** Deleting the Pod object removes its eviction status and message from the cluster; the Pod is not running, so no workload stops. Precondition: only when `upgrade-sysctl-config` (or the BroadcastJob the base component waits on) shows `FAILED` greater than `0`, the failed Pod is `Evicted`, the other Pods of that BroadcastJob are `Completed`, and the YAML above is saved. Rollback: none for the Pod object; the saved YAML keeps the eviction details. + +4. Delete the Evicted Pod and watch the base component: + + ```bash + kubectl -n cpaas-system delete pod + kubectl get job,bcj -n cpaas-system + kubectl get ars -A | grep -i base + ``` + +Confirm that the component shows synced on the cluster page before you start a business cluster upgrade. If it stays in `syncing`, collect the BroadcastJob YAML and the `sentry` log before trying anything else. + +### update-containerd-config does not finish: raise its parallelism + +```bash +kubectl get broadcastjob -A | grep update-containerd-config +kubectl -n get broadcastjob update-containerd-config -o yaml > "$HOME/update-containerd-config.yaml" +``` + +> **Warning:** A higher `parallelism` updates the containerd configuration on more nodes at the same time. Precondition: only when the ARS stays `syncing` and the `sentry` log points to an incomplete `update-containerd-config` BroadcastJob. Rollback: set `parallelism` back to the value in the export. + +```bash +kubectl -n patch broadcastjob update-containerd-config --type=merge -p '{"spec":{"parallelism":10}}' +kubectl -n get broadcastjob update-containerd-config -w +``` + +Wait until all jobs are `success`, then check that the ARS leaves `syncing`. + +## Diagnostic Steps + +1. Check the state of the `base` AppRelease and its conditions: + + ```bash + kubectl -n cpaas-system get ar base + kubectl -n cpaas-system get ar base -o yaml | grep -n -i -A5 'conditions\|timeout' + ``` + +2. Check the BroadcastJob and its progress. Compare `DESIRED` with the node count. A job that keeps a new, short age and never reaches `SUCCEEDED` equal to `DESIRED` is being restarted: + + ```bash + kubectl get nodes --no-headers | wc -l + kubectl -n cpaas-system get broadcastjob upgrade-sysctl-config + kubectl -n cpaas-system get pods -o wide | grep upgrade-sysctl-config + ``` + +3. Rule out a BroadcastJob that fails rather than one that is only slow. If its Pods end in `Error` or stay `Pending` on some nodes, a longer timeout does not help. Read those Pods first: + + ```bash + kubectl -n cpaas-system get pods | grep upgrade-sysctl-config | grep -v -E 'Running|Completed' + kubectl -n cpaas-system describe pod + ``` + +4. List every BroadcastJob and read the `sentry` log, because the BroadcastJob that holds the AppRelease differs between cases. `FAILED` greater than `0` with an `Evicted` Pod is the Evicted-Pod cause. An `update-containerd-config` that stays below `DESIRED` with stuck jobs, named in the `sentry` log, is the `update-containerd-config` cause. If every BroadcastJob and Job is complete and the `sentry` log shows `Watching for changes to Job ` followed by `Jobs active: 0`, the chart is waiting on a hook Job instead: + + ```bash + kubectl get bcj -A + kubectl get pods -A | grep -E 'upgrade-sysctl-config|update-containerd-config' + kubectl -n cpaas-system logs --tail=500 | grep -E 'update-containerd-config|upgrade-sysctl-config|Watching for changes to Job|Jobs active' + ``` diff --git a/docs/en/solutions/calico-node_fails_to_start_or_Pods_hang_in_ContainerCreating_dummy_module_corrupted_image_clock_skew_or_x509_on_ACP.md b/docs/en/solutions/calico-node_fails_to_start_or_Pods_hang_in_ContainerCreating_dummy_module_corrupted_image_clock_skew_or_x509_on_ACP.md new file mode 100644 index 000000000..2b7715b65 --- /dev/null +++ b/docs/en/solutions/calico-node_fails_to_start_or_Pods_hang_in_ContainerCreating_dummy_module_corrupted_image_clock_skew_or_x509_on_ACP.md @@ -0,0 +1,144 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "3.18" + - "4.0" + - "4.2" + - "4.3" +--- + +# calico-node fails to start or Pods hang in ContainerCreating: dummy module, corrupted image, clock skew or x509 on ACP + +## Issue + +On a Calico cluster, `calico-node` does not start on a node, or Pods on a node stay in `ContainerCreating` with Calico errors. Typical reports: + +- `calico-node` fails with `failed to connect kubernetes svc:443`. +- `calico-node` does not start on a node right after a VM live migration, and container DNS and logins fail. +- Pods stay in `ContainerCreating`, the kubelet log shows Calico errors while setting up the Pod network, and after a restart `calico-node` fails with `x509: certificate has expired or is not yet valid: current time ... is before ...`. +- After every node of the cluster was restarted, the Pods on one node stay in `ContainerCreating` with `plugin type="calico" failed (add): error getting ClusterInformation: ... x509: certificate signed by unknown authority`. + +Each cause is a property of the node (kernel modules, local image store, clock), not of the Calico configuration. + +Related articles: "Pod or Service traffic fails on some nodes because of the node kernel, OS, hypervisor or network path on ACP". + +## Root Cause + +### Cause 1: calico-node: failed to connect kubernetes svc:443, and the dummy kernel module is missing + +`calico-node` fails to start with `failed to connect kubernetes svc:443`. The node has no dummy interface, and `lsmod` shows no `dummy` module because the module file is missing from `/lib/modules/$(uname -r)`. + +### Cause 2: calico-node image layers are corrupted after a VM live migration + +After the node VM was live-migrated, the local image layers of the `calico-node` image were corrupted, so the container could not start and Pod networking on the node failed. + +### Cause 3: node clock out of sync, install-cni fails x509 verification + +Pods stay in `ContainerCreating`, and restarting Calico makes it worse: `calico-node` no longer starts. Its `install-cni` container keeps exiting `1`, and its log shows the CNI binaries written to `/host/opt/cni/bin` followed by: + +```text +[ERROR] cni-installer/token_watch.go 108: Unable to create token for CNI kubeconfig error=Post ".../api/v1/namespaces/kube-system/serviceaccounts/calico-cni-plugin/token": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time is before +[FATAL] cni-installer/install.go 499: ... +``` + +The node clock was behind the start of the certificate's validity, so `install-cni` failed TLS verification when requesting the `calico-cni-plugin` token, and the Pod network could not be set up. + +### Cause 4: x509: certificate signed by unknown authority from the Calico plugin after a full cluster restart + +After every node was restarted, the Pods on one node stayed in `ContainerCreating` with: + +```text +Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "": plugin type="calico" failed (add): error getting ClusterInformation: Get "https://:443/apis/crd.projectcalico.org/v1/clusterinformations/default": tls: failed to verify certificate: x509: certificate signed by unknown authority +``` + +followed by `Pod sandbox changed, it will be killed and re-created.` Every `calico-node` Pod was `1/1 Running` with routine logs, restarting `calico-node` did not help, and the node clock was correct. The Calico CNI plugin on that node could not verify the API server certificate; the exact mechanism was not determined. Restarting the node restored its Pods. This differs from Cause 3, where the message says the certificate `is not yet valid`. + +## Resolution + +### Cause 1: calico-node: failed to connect kubernetes svc:443, and the dummy kernel module is missing + +Copy the `dummy` module from a node that runs exactly the same kernel version, rebuild the module index, load it, and recreate `calico-node`. + +> **Warning:** A module built for another kernel can fail to load or destabilize the host. Compare `uname -r` on both nodes first. Roll back by removing the copied file and running `depmod -a` again. + +```bash +uname -r +lsmod | grep dummy || modinfo dummy +# on a healthy node with the same kernel: +find /lib/modules/$(uname -r) -name 'dummy.ko*' +# copy that file to the same path on the affected node, then on the affected node: +depmod -a +modprobe dummy +kubectl -n kube-system delete pod +``` + +### Cause 2: calico-node image layers are corrupted after a VM live migration + +Stop kubelet, delete the local `calico-node` image, and start kubelet so a clean copy is pulled. + +> **Warning:** With kubelet stopped, the node does not manage Pods. The pull after the delete fails if the node cannot reach the registry, so check registry access first. The delete cannot be undone; the pull replaces the image. + +```bash +systemctl stop kubelet +crictl images | grep calico/node +crictl rmi +systemctl start kubelet +``` + +On a Docker node use `docker images` and `docker rmi -f` instead of `crictl`. + +### Cause 3: node clock out of sync, install-cni fails x509 verification + +Compare the clock with a healthy node and synchronize it from the site time source; restarting Calico before that does not help. Then retry the Pods. + +```bash +date -u; timedatectl +chronyc tracking || true +chronyc makestep +``` + +### Cause 4: x509: certificate signed by unknown authority from the Calico plugin after a full cluster restart + +Rule out the clock first (Cause 3), then restart the affected node. + +> **Warning:** Restarting the node stops every Pod on it. Drain it first and restart one node at a time; uncordon it when it is back. Nothing needs to be rolled back. + +```bash +kubectl drain --ignore-daemonsets --delete-emptydir-data +# on the node: +systemctl reboot +# from a master, once the node is Ready: +kubectl uncordon +``` + +## Diagnostic Steps + +1. Read the Pod events and the kubelet log on the affected node: + + ```bash + kubectl describe pod -n | grep -A3 FailedCreatePodSandBox + journalctl -u kubelet --since '-30m' | grep -i calico | tail -n 20 + kubectl get pods -A -o wide | grep -E 'calico|ContainerCreating' + ``` + +2. Read the `install-cni` log when `calico-node` is not starting: + + ```bash + kubectl -n kube-system logs -c install-cni --previous | grep -E 'x509|is before|token' + ``` + +3. Check the node properties: + + ```bash + lsmod | grep dummy + date -u; timedatectl + ``` + +- `failed to connect kubernetes svc:443` with no `dummy` module: Cause 1. +- A failure that starts right after a VM live migration: Cause 2. +- `x509: certificate has expired or is not yet valid: current time ... is before ...` from `install-cni`, with a node clock behind the other nodes: Cause 3. +- `x509: certificate signed by unknown authority` from the Calico plugin after a full cluster restart, with a correct clock: Cause 4. diff --git a/docs/en/solutions/cpaas-kafka_broker_fails_to_start_corrupt_segment_no_space_broker_id_conflict_or_deleted_StatefulSet_on_ACP.md b/docs/en/solutions/cpaas-kafka_broker_fails_to_start_corrupt_segment_no_space_broker_id_conflict_or_deleted_StatefulSet_on_ACP.md new file mode 100644 index 000000000..4ee8ce8cc --- /dev/null +++ b/docs/en/solutions/cpaas-kafka_broker_fails_to_start_corrupt_segment_no_space_broker_id_conflict_or_deleted_StatefulSet_on_ACP.md @@ -0,0 +1,246 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# cpaas-kafka broker fails to start: corrupt segment, no space, broker id conflict, or deleted StatefulSet on ACP + +## Issue + +A broker of the log storage Kafka does not start, or the Kafka is missing altogether. Typical reports: + +- "The ES log component keeps showing Installing (`正在安装`)." +- "Kafka for logging filled its PVC and still fails after the PVC was expanded." +- "Kafka cannot write to its data disk: `No space left on device`." +- "One Kafka pod crashes while Elasticsearch log storage is being deployed." +- "Kafka cannot start again after a reinstall: broker id conflict." +- "The `cpaas-kafka` StatefulSet was deleted." + +The broker is the log storage Kafka in `cpaas-system` (StatefulSet `cpaas-kafka`, pods `cpaas-kafka-0..2`, with `cpaas-zookeeper`), not a Data Services Kafka. Its data is on node-local storage, typically `/cpaas/data/kafka0` (seen inside the pod as `/kafka0`), plus `/cpaas/data/kafka` and `/cpaas/data/zookeeper`. + +When the brokers run but logs, audit and events stop, see [Logs, audit and events stop while the log Kafka is degraded on ACP](./Logs_audit_and_events_stop_while_the_log_Kafka_is_degraded_on_ACP.md). + +## Root Cause + +### Cause 1: No space left on device under /kafka0, and expanding the PVC does not help + +```text +ERROR Error while loading log dir /kafka0 (org.apache.kafka.storage.internals.log.LogDirFailureChannel) +java.nio.file.FileSystemException: /kafka0/ALAUDA_EVENT_TOPIC-2/00000000000000000007.snapshot: No space left on device +``` + +Kafka filled its volume and crashed. What fixes it depends on what backs the PV: + +- **Node-local PV** (`spec.local` or `spec.hostPath`). By default, `cpaas-kafka` uses PVs backed by the node's local storage, so the real capacity is the node's local filesystem. Expanding the PVC (for example from 30Gi to 60Gi) changes the number but not the space, and the pod still cannot create files in `/kafka0`. Patching `volumeClaimTemplates` is rejected, because that StatefulSet field is immutable. Extending the logical volume behind the path on the node fixes Kafka. +- **PV from a StorageClass that supports expansion.** Expanding the PVC adds real space, and Kafka recovers. + +After space is added, ZooKeeper may log `ERROR Unsuccessful handshake with session 0x0 (org.apache.zookeeper.server.NettyServerCnxnFactory)`. This message is known, and it can be ignored while ZooKeeper is `Running`. + +### Cause 2: Kafka exits at startup with a fault in an unsafe memory access operation in LogSegment.recover + +The previous log of the broker ends with: + +```text +ERROR Exiting Kafka due to fatal exception during startup. (kafka.Kafka$) +java.lang.InternalError: a fault occurred in an unsafe memory access operation + at org.apache.kafka.storage.internals.log.LogSegment.recover(LogSegment.java:498) + at kafka.log.LogLoader.recoverSegment(LogLoader.scala:371) +``` + +Kafka cannot read its on-disk segments while it recovers them at startup, so the broker never starts. The pod is `CrashLoopBackOff` with thousands of restarts, while the other brokers, Elasticsearch and `lanaya` are `Running`. `kubectl get ars -A` shows every chart `Synced`/`Ready` except `logcenter`, which is `Synced` with health `WaitReady`. That keeps the Elasticsearch log component in Installing. The fix is to stop Kafka and ZooKeeper, clear the contents of their data directories on all three log nodes, and scale both back to 3 replicas. + +### Cause 3: One Kafka pod crashes at startup with a broker id conflict + +One `cpaas-kafka` pod crash-loops while its peers run, and its broker id is the same as another broker's, or conflicts with data left from an earlier installation. Upstream Kafka reports these states with messages such as `InconsistentBrokerIdException: Configured broker.id ... doesn't match stored broker.id ... in meta.properties`, or with a broker that cannot register because its id is already registered in ZooKeeper. Two variants exist: + +- While Elasticsearch was being deployed as the log storage, one Kafka node failed to start because its broker id duplicated the id of another Kafka node. Giving each Kafka node a different broker id and restarting Kafka fixed it. +- After the log storage was reinstalled, Kafka could not start because of a broker id conflict. The data of the previous installation had not been cleaned before the reinstall. Clearing the `zookeeper`, `kafka` and `kafka0` directories under `/cpaas/data` on the three Kafka nodes fixes it. + +### Cause 4: The cpaas-kafka StatefulSet was deleted + +The log storage Kafka StatefulSet is gone. A resource patch (`rpch`) for Kafka may exist that only adds configuration, and it can be left alone. Kafka can be recreated from the StatefulSet YAML of a healthy environment with the same platform and plugin version. After that, stop `lanaya`, Kafka and ZooKeeper, clean their directories, and start them in the order ZooKeeper, Kafka, `lanaya`. Log queries then work again. + +## Resolution + +### Cause 1: No space left on device under /kafka0, and expanding the PVC does not help + +1. Find the PV behind the broker and what backs it: + + ```bash + kubectl get pod -A | grep -E 'zook|kafka' + kubectl -n cpaas-system get pvc | grep kafka + kubectl get pv -o jsonpath='{.spec.local.path}{.spec.hostPath.path}{"\n"}{.spec.nodeAffinity}{"\n"}{.spec.storageClassName}{"\n"}' + ``` + +2. If the PV has a `local` or `hostPath` path, find the filesystem behind that path on the node the PV is pinned to: + + ```bash + df -hT + lvs + vgs + ``` + +3. Add space. + - For a node-local PV, extend the logical volume. + + > **Warning:** Extending a logical volume and its filesystem changes the node's disk layout. An online extension is normally safe, but a wrong logical volume or volume group name can consume space that another workload needs. Precondition: only when the Kafka log shows `No space left on device` under `/kafka0`, `df` on the node shows the logical volume behind the PV path full, and the volume group has free extents (`vgs`). Rollback: shrinking is not supported online for XFS and is risky for ext4, so treat the extension as permanent. + + ```bash + lvextend -r -L + /dev// + df -hT + ``` + + `-r` grows the filesystem together with the logical volume. + - For a PV from an expandable StorageClass, expand the PVC instead: + + ```bash + kubectl -n cpaas-system patch pvc -p '{"spec":{"resources":{"requests":{"storage":""}}}}' + ``` + +4. Restart the broker and check a current log query: + + ```bash + kubectl -n cpaas-system delete pod + ``` + + A ZooKeeper `Unsuccessful handshake with session 0x0` line after recovery needs no action. + +### Cause 2: Kafka exits at startup with a fault in an unsafe memory access operation in LogSegment.recover + +1. Confirm the signature: + + ```bash + kubectl get po -A | grep -E 'kafka|lanaya|elastic' + kubectl get ars -A + kubectl -n cpaas-system logs -p cpaas-kafka-0 | grep -B2 -A6 'unsafe memory access' + ``` + +2. Stop Kafka and ZooKeeper: + + ```bash + kubectl -n cpaas-system scale sts cpaas-kafka cpaas-zookeeper --replicas=0 + ``` + +3. Empty their data directories. + + > **Warning:** Clearing the Kafka and ZooKeeper data directories deletes every unconsumed log, audit and event record, and the ZooKeeper state of the log cluster. Precondition: only when the previous Kafka log shows the `LogSegment.recover` fault above and both StatefulSets are at 0 replicas. Rollback: none. + + On each of the three log nodes, empty the directories but keep them. The `cd ... &&` form stops before `rm` if a directory is missing: + + ```bash + cd /cpaas/data/kafka && rm -rf ./* + cd /cpaas/data/kafka0 && rm -rf ./* + cd /cpaas/data/zookeeper && rm -rf ./* + ``` + +4. Start both again and watch: + + ```bash + kubectl -n cpaas-system scale sts cpaas-kafka cpaas-zookeeper --replicas=3 + kubectl -n cpaas-system get po -w | grep -E 'kafka|zookeeper' + ``` + +### Cause 3: One Kafka pod crashes at startup with a broker id conflict + +1. Find the Kafka nodes. In the console, open Marketplace, Cluster Plugins, select the cluster, and open the log storage plugin. Its details list the three Kafka nodes. + +2. Compare the broker ids. The stored id is the `broker.id` line of `meta.properties` in the data directory (`/kafka0` inside the pod, `/cpaas/data/kafka0` on the node): + + ```bash + kubectl -n cpaas-system get pods -o wide | grep -E 'cpaas-kafka|cpaas-zookeeper' + kubectl -n cpaas-system logs -p | grep -i -E 'broker.?id' | tail -20 + for p in cpaas-kafka-0 cpaas-kafka-1 cpaas-kafka-2; do + echo "== $p"; kubectl -n cpaas-system exec "$p" -- cat /kafka0/meta.properties 2>/dev/null + done + ``` + + A crashing pod cannot be opened with `exec`. Read its file on the node with `cat /cpaas/data/kafka0/meta.properties`. + +3. When two live brokers have the same id and the data on each node belongs to the current installation, make the ids distinct and restart Kafka. Compare with a healthy installation of the same version before you edit anything. + +4. When the log storage was reinstalled on nodes that still hold the previous installation's data, clear that data. First stop both StatefulSets: + + ```bash + kubectl -n cpaas-system scale sts cpaas-kafka cpaas-zookeeper --replicas=0 + kubectl -n cpaas-system get pods | grep -E 'cpaas-kafka|cpaas-zookeeper' + ``` + + > **Warning:** Emptying `/cpaas/data/zookeeper`, `/cpaas/data/kafka` and `/cpaas/data/kafka0` deletes every record still in Kafka and the ZooKeeper state of the log cluster. Unconsumed logs, audit records and events are lost. Elasticsearch indices are not touched. Precondition: only when the broker id conflict appeared after a reinstall, the data under these directories is left from the previous installation, and both StatefulSets are at 0 replicas. Rollback: none. + + On each of the three Kafka nodes, delete the contents and keep the directories: + + ```bash + cd /cpaas/data/zookeeper && rm -rf ./* + cd /cpaas/data/kafka && rm -rf ./* + cd /cpaas/data/kafka0 && rm -rf ./* + ``` + + Then start both again and watch until every pod is `Running`: + + ```bash + kubectl -n cpaas-system scale sts cpaas-kafka cpaas-zookeeper --replicas=3 + kubectl -n cpaas-system get pods -w | grep -E 'cpaas-kafka|cpaas-zookeeper' + ``` + +### Cause 4: The cpaas-kafka StatefulSet was deleted + +1. Confirm that the StatefulSet is missing, and list the resource patches: + + ```bash + kubectl -n cpaas-system get sts | grep -E 'kafka|zookeeper' + kubectl get rpch -A + ``` + +2. On a healthy environment with the same platform and plugin version, export the StatefulSet: + + ```bash + kubectl -n cpaas-system get sts cpaas-kafka -o yaml > /tmp/cpaas-kafka.sts.yaml + ``` + + Remove `metadata.uid`, `metadata.resourceVersion`, `metadata.creationTimestamp`, `metadata.managedFields` and `status`. Replace node names, addresses and storage values that belong to the other environment. + +3. Apply it on the affected cluster: + + ```bash + kubectl -n cpaas-system apply -f /tmp/cpaas-kafka.sts.yaml + ``` + +4. Clean and restart the pipeline in order. + + > **Warning:** Cleaning the Kafka and ZooKeeper directories drops every record that was in Kafka. Precondition: Kafka was recreated from the exported YAML, and `lanaya`, Kafka and ZooKeeper are stopped. Rollback: none. + + Stop `lanaya`, Kafka and ZooKeeper, and empty the Kafka and ZooKeeper data directories as in Cause 2. Start ZooKeeper, then Kafka, then `lanaya`, and wait for each to be Ready before the next. Keep watching the Kafka pods afterwards, and run a log query. + +## Diagnostic Steps + +1. Read the broker state and the last fatal line: + + ```bash + kubectl -n cpaas-system get sts | grep -E 'kafka|zookeeper' + kubectl -n cpaas-system get pods -o wide | grep -E 'kafka|zookeeper' + kubectl -n cpaas-system logs -p cpaas-kafka-0 | tail -30 + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | `No space left on device` under `/kafka0` | Cause 1, check what backs the PV and not only the PVC size | + | `unsafe memory access` in `LogSegment.recover` | Cause 2 | + | One broker crash-looping with "broker id" in its previous log, two brokers with the same `broker.id` in `meta.properties`, or a conflict right after a reinstall | Cause 3 | + | No `cpaas-kafka` StatefulSet | Cause 4 | + +3. A Kafka in KRaft mode (broker and controller pods, data under `/var/lib/kafka/data`, `No space left on device` on `__cluster_metadata`) is not this log storage Kafka. Check its disk the same way: find the PV behind the data directory, and add space to the volume that actually backs it. + +## Related Information + +- Related articles: [Logs, audit and events stop while the log Kafka is degraded on ACP](./Logs_audit_and_events_stop_while_the_log_Kafka_is_degraded_on_ACP.md) covers a large `kafka0` backlog, one broker down with `min.insync.replicas=2`, and logs that stay missing after a disk cleanup. +- [Kafka Scale Switching](./Kafka_Scale_Switching.md) diff --git a/docs/en/solutions/cpaas-kafka_retention_hours_memory_limit_above_80_percent_or_KAFKA_HEAP_OPTS_not_applied_on_ACP.md b/docs/en/solutions/cpaas-kafka_retention_hours_memory_limit_above_80_percent_or_KAFKA_HEAP_OPTS_not_applied_on_ACP.md new file mode 100644 index 000000000..fc9a0bd0e --- /dev/null +++ b/docs/en/solutions/cpaas-kafka_retention_hours_memory_limit_above_80_percent_or_KAFKA_HEAP_OPTS_not_applied_on_ACP.md @@ -0,0 +1,160 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# cpaas-kafka retention hours, memory limit above 80 percent, or KAFKA_HEAP_OPTS not applied on ACP + +## Issue + +Questions and problems with the Kafka of the platform log storage plugin (StatefulSet `cpaas-kafka` in `cpaas-system`): + +- How long does the log storage Kafka keep messages, and how is the retention (`log.retention.hours`) changed? +- The log nodes have a single disk, and during business peaks Kafka data could fill `/`. Can Kafka keep less? +- A rule requires Kafka memory usage at or below 80%, and the `cpaas-kafka` pods run at 80.70% to 82.00% of their container memory limit. +- `KAFKA_HEAP_OPTS` on the StatefulSet says `-Xms6g -Xmx6g` (or `2g`), but the broker Java process still runs with `-Xmx1G -Xms1G`. Logs pile up, and `razor` logs `Send kafka msg error {"topic": "ALAUDA_LOG_TOPIC", ..., "error": "context deadline exceeded"}` and `"error": "kafka write errors (1/1)"`, also for `ALAUDA_AUDIT_TOPIC`. + +This Kafka is not a Data Services instance and is not listed on the Data Services page. Its broker settings are environment variables on the container, for example: + +| Variable | Value seen | +| --- | --- | +| `KAFKA_LOG_RETENTION_HOURS` | `"48"` | +| `KAFKA_LOG_ROLL_HOURS` | `"12"` | +| `KAFKA_LOG_SEGMENT_DELETE_DELAY_MS` | `"0"` | +| `KAFKA_LOG_DIRS` | `/kafka0` | +| `KAFKA_HEAP_OPTS` | `-Xmx1g -Xms512m` | + +Container resources were limits `cpu: "2"` / `memory: 4Gi` and requests `cpu: 200m` / `memory: 256Mi` in one environment. The plugin keeps Kafka settings in the `logcenter` ModuleInfo (`spec.config.components.kafka`). The default Kafka backlog alerts are in the alert policy `cpaas-log-rules` on the global cluster. + +## Root Cause + +### Cause 1: Kafka keeps log data for 48 hours, set by KAFKA_LOG_RETENTION_HOURS + +By default the log storage Kafka keeps data for 2 days: `KAFKA_LOG_RETENTION_HOURS: "48"` in the `cpaas-kafka` environment. Where the variable is absent from the StatefulSet, the default is expected to be the same. Retention is changed by adding or editing that variable, which restarts the Kafka pods. + +### Cause 2: Kafka data fills a single disk during peaks, and retention is the trade-off + +When the Kafka log directory sits on the same file system as `/`, peak log volume can fill `/` and make the node abnormal. A lower retention reduces that risk but shortens the time available to clear a backlog: with 24 hours, any log data not written to the log store within 24 hours during a backlog is lost. Kafka removes segments only once they pass the retention age; a consumer that catches up does not free disk by itself. + +### Cause 3: cpaas-kafka container memory is above an 80% requirement + +With a `4Gi` limit and about 3.2Gi used, container memory usage is 80.70% to 82.00%. With no backlog, this is normal: Kafka relies on the page cache, which is counted in container memory. To meet a rule of 80% or less, raise the container memory limit. + +### Cause 4: KAFKA_HEAP_OPTS does not reach the broker JVM because the start script sets its own heap + +The container starts the broker through `/opt/kafka/bin/start-kafka.sh`, which calls `/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties`. In the affected image, `kafka-server-start.sh` contains an unconditional `export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"`, which overwrites the value from the environment. The StatefulSet and `printenv` show the configured heap, while the broker JVM (`java -Xmx1G -Xms1G -Djava.security.auth.login.config=/opt/kafka/conf/kafka_server_jaas.conf -server -XX:+UseG1GC ...`) runs with 1G. Threads such as `kafka-scheduler`, `data-plane-kafka`, and `ExpirationReaper` belong to that same broker JVM; `kafka_exporter` in the same pod is a separate, non-Java process. + +With a 1G heap under a high log volume, the brokers cannot keep up: replica fetchers log `Error sending fetch request ... java.io.IOException: Connection to was disconnected before the response was read`, the controller repeats `Processing automatic preferred replica leader election`, and `razor` writes time out. Mounting a corrected `kafka-server-start.sh` from a ConfigMap and restarting Kafka makes the heap setting take effect. Large on-disk queues on some nodes also made Kafka fail while loading them at startup and needed cleanup. If the backlog persists after the heap is right, the remaining limit is capacity: Kafka and Elasticsearch sharing the same nodes could not sustain the log volume, and log collection for the noisiest namespace was disabled temporarily. + +## Resolution + +Save the StatefulSet before any change: + +```bash +kubectl -n cpaas-system get sts cpaas-kafka -o yaml > /tmp/cpaas-kafka-sts.before.yaml +``` + +### Cause 1: Kafka keeps log data for 48 hours, set by KAFKA_LOG_RETENTION_HOURS + +```bash +kubectl -n cpaas-system get sts cpaas-kafka -o jsonpath='{range .spec.template.spec.containers[*].env[*]}{.name}{"="}{.value}{"\n"}{end}' | grep -E '^KAFKA_LOG_(RETENTION|ROLL|DIRS|SEGMENT)' +``` + +> **Warning:** Changing the StatefulSet environment restarts the `cpaas-kafka` pods one by one; ingestion slows and lag rises until they are Ready. A shorter retention deletes older segments, including data not consumed yet. Precondition: consumer lag is near zero, and the change was tried on a test cluster of the same version. Rollback: `kubectl apply -f /tmp/cpaas-kafka-sts.before.yaml`; deleted segments do not return. + +```bash +kubectl -n cpaas-system set env sts/cpaas-kafka KAFKA_LOG_RETENTION_HOURS= +kubectl -n cpaas-system rollout status sts/cpaas-kafka +``` + +Editing the YAML (`kubectl -n cpaas-system edit sts cpaas-kafka`, or the StatefulSet's YAML in the console) is the same change: add or modify `name: KAFKA_LOG_RETENTION_HOURS` with a quoted value such as `"48"`. If a plugin update restores the old value, read the environment again and repeat. + +### Cause 2: Kafka data fills a single disk during peaks, and retention is the trade-off + +1. Size retention as the hours of peak log volume that fit on the Kafka disk with headroom, but no shorter than the longest backlog you must survive. Set it as in Cause 1. +2. Watch the Kafka backlog alerts in `cpaas-log-rules` and set thresholds that warn before data reaches the retention age. +3. Plan a separate disk for log data if `/` is the only disk. + +For a log node whose disk is already full of Elasticsearch and Kafka data, free Elasticsearch space by deleting old indices as in [/cpaas/data full with ZooKeeper down or Elasticsearch on the root disk on ACP](cpaas_data_full_with_ZooKeeper_down_or_Elasticsearch_on_the_root_disk_on_ACP.md), and let the shorter Kafka retention remove old segments. + +### Cause 3: cpaas-kafka container memory is above an 80% requirement + +```bash +kubectl -n cpaas-system top pod | grep cpaas-kafka +kubectl -n cpaas-system get sts cpaas-kafka -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" "}{.resources}{"\n"}{end}' +``` + +Pick a limit that puts usage below 80% (usage divided by 0.8, rounded up; about 3.2Gi needs more than 4Gi). Page cache can grow toward the new limit, so check again after a few days. + +> **Warning:** Changing the resources restarts the `cpaas-kafka` pods one by one. Precondition: the Kafka nodes have the extra memory free and consumer lag is near zero. Rollback: `kubectl apply -f /tmp/cpaas-kafka-sts.before.yaml`. + +```bash +kubectl -n cpaas-system edit sts cpaas-kafka +kubectl -n cpaas-system rollout status sts/cpaas-kafka +``` + +Set `resources.limits.memory` of the Kafka container. If a plugin update restores the old limit, set the same value in the log storage plugin configuration. + +### Cause 4: KAFKA_HEAP_OPTS does not reach the broker JVM because the start script sets its own heap + +1. Compare the three values and read the script: + + ```bash + kubectl -n cpaas-system get sts cpaas-kafka -o jsonpath='{range .spec.template.spec.containers[*].env[*]}{.name}={.value}{"\n"}{end}' | grep HEAP + kubectl -n cpaas-system exec cpaas-kafka-0 -- printenv KAFKA_HEAP_OPTS + kubectl -n cpaas-system exec cpaas-kafka-0 -- sh -c "ps -ef | grep '[k]afka_server_jaas' | grep -o -- '-Xm[sx][^ ]*'" + kubectl -n cpaas-system exec cpaas-kafka-0 -- grep -n 'KAFKA_HEAP_OPTS' /opt/kafka/bin/kafka-server-start.sh + ``` + + The environment showing the configured heap, the broker process showing `-Xmx1G`, and an unconditional `export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"` in the script confirm this cause. + +2. Copy the script out and correct it so the environment value wins, for example by wrapping the line in the upstream Kafka form `if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"; fi`: + + ```bash + kubectl -n cpaas-system exec cpaas-kafka-0 -- cat /opt/kafka/bin/kafka-server-start.sh > /tmp/kafka-server-start.sh + vi /tmp/kafka-server-start.sh + kubectl -n cpaas-system create configmap cpaas-kafka-server-start --from-file=kafka-server-start.sh=/tmp/kafka-server-start.sh + ``` + +3. Mount it over the original and restart Kafka. + + > **Warning:** Changing the StatefulSet restarts the `cpaas-kafka` pods; log produce and consume pause on each broker while it restarts. Set `KAFKA_HEAP_OPTS` well below the container memory limit, or the pod is `OOMKilled`. Precondition: Cause 4 is confirmed and the StatefulSet YAML is saved. Rollback: `kubectl apply -f /tmp/cpaas-kafka-sts.before.yaml`, then delete the ConfigMap. A plugin update can replace the StatefulSet; check the mount after every update. + + ```bash + kubectl -n cpaas-system edit sts cpaas-kafka + ``` + + Add a volume from ConfigMap `cpaas-kafka-server-start` with `defaultMode: 0755`, and in the Kafka container a `volumeMount` with `mountPath: /opt/kafka/bin/kafka-server-start.sh` and `subPath: kafka-server-start.sh`. Then: + + ```bash + kubectl -n cpaas-system rollout status sts/cpaas-kafka + kubectl -n cpaas-system exec cpaas-kafka-0 -- sh -c "ps -ef | grep '[k]afka_server_jaas' | grep -o -- '-Xm[sx][^ ]*'" + ``` + + The broker process must now show the configured heap. + +4. If a broker fails while loading a large on-disk queue at startup, clear that queue before restarting it; see Cause 1 of [Platform log Kafka brokers fail: no leader election, exit 137, Pending on DiskPressure, Ceph write errors, or OOM on ACP](Platform_log_Kafka_brokers_fail_no_leader_election_exit_137_Pending_on_DiskPressure_Ceph_write_errors_or_OOM_on_ACP.md). +5. If the backlog persists with the right heap, add capacity: move Kafka or Elasticsearch to separate nodes, or reduce the log volume (for example by disabling log collection for a noisy namespace) until capacity is added. + +## Diagnostic Steps + +1. Confirm it is the platform log Kafka: `kubectl -n cpaas-system get sts,pods | grep cpaas-kafka`. +2. Retention and disk: + + ```bash + lsblk + kubectl -n cpaas-system exec cpaas-kafka-0 -- df -h /kafka0 + du -sh /cpaas/data/kafka* + ``` + + Kafka data on the same disk as `/`, or near the disk's size, is Cause 2. +3. `kubectl top` usage divided by the limit above 80%, with no lag, is Cause 3. If consumer lag is also growing, fix the backlog first. +4. Heap: when the broker JVM's `-Xmx` differs from `KAFKA_HEAP_OPTS` in the environment, read the start script (Cause 4). The Java process that loads `kafka_server_jaas.conf` is the broker; `kafka_exporter` is not a Java process. +5. Backlog alerts: in the global cluster's alert policies, check that `cpaas-log-rules` exists and its Kafka rules are normal. A firing lag rule means data older than `KAFKA_LOG_RETENTION_HOURS` in the backlog is at risk. diff --git a/docs/en/solutions/cpaas_data_full_with_ZooKeeper_down_or_Elasticsearch_on_the_root_disk_on_ACP.md b/docs/en/solutions/cpaas_data_full_with_ZooKeeper_down_or_Elasticsearch_on_the_root_disk_on_ACP.md new file mode 100644 index 000000000..7332a3213 --- /dev/null +++ b/docs/en/solutions/cpaas_data_full_with_ZooKeeper_down_or_Elasticsearch_on_the_root_disk_on_ACP.md @@ -0,0 +1,264 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.12.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# /cpaas/data full with ZooKeeper down or Elasticsearch on the root disk on ACP + +## Issue + +The log storage on Alauda Container Platform is abnormal because `/cpaas` is full, or Elasticsearch fills a disk it shares with the system. Typical reports: + +- Logs cannot be viewed in the ACP console, although `kubectl logs` still shows them; the log storage plugin of the global cluster is abnormal, and `df` shows `/cpaas` at 100% on its storage node. +- The log components are abnormal, and Kafka and ZooKeeper do not start. +- A production pod (for example Redis) cannot be scheduled and the event mentions `disk-pressure`. +- Many pods, including `nevermore`, keep restarting, and `/` is above 85%. +- Local disk on the global masters is high because Elasticsearch and MinIO both store data there. +- After space was freed, Elasticsearch shard recovery is very slow (`"decision" : "THROTTLE"`, `reached the limit of incoming shard recoveries [2]`), or Kafka still exits with `kafka.zookeeper.ZooKeeperClientTimeoutException`. +- The Elasticsearch log storage plugin keeps restarting with `/cpaas` at 100%, and the stored logs do not need to be kept. +- `/cpaas` is full of data that is not the running log store, such as Prometheus monitoring data or ClickHouse log data from an earlier installation. +- The platform cannot be opened after the log disk filled: `kubectl get ars -A` shows `auth-manager` as `Failed`, and `apollo` stays abnormal after the disk was freed. + +Logging components on ACP 4.x are in `cpaas-system` (StatefulSet `cpaas-elasticsearch`, DaemonSet `nevermore`). Audit indices or leftover Elasticsearch data filling `/cpaas`, and index deletes that free nothing, are covered in [/cpaas filled by Elasticsearch audit indices or leftover Elasticsearch data on ACP](./cpaas_filled_by_Elasticsearch_audit_indices_or_leftover_Elasticsearch_data_on_ACP.md), together with how to read the Elasticsearch credentials. + +## Root Cause + +### Cause 1: the log storage plugin is abnormal and /cpaas is at 100%, so the console shows no logs + +The console log query reads from the log storage, while `kubectl logs` reads from the kubelet on the node. When `/cpaas` on the log storage node is full, the log storage plugin of the global cluster becomes abnormal and log query returns nothing, while `kubectl logs` keeps working. No ZooKeeper error is needed for this shape. Once the disk is expanded, the logging components return to normal without further action, and the backlog queued while the disk was full is consumed before log query catches up. + +### Cause 2: /cpaas/data is 100%, ZooKeeper does not start, and the Elasticsearch API returns 503 + +The directory used by Elasticsearch is full, the API returns 503, and ZooKeeper, which shares the same filesystem (for example `/cpaas/data`), fails to start. The ZooKeeper side looks like this, for example on a log storage node whose `/cpaas` (`/dev/mapper/cpaas_vg-cpaas_lv`, 500G) has a few hundred KB left: + +```text +chmod: changing permissions of '/opt/kafka/libs/...': Read-only file system +bash: /opt/kafka/bin/init.sh: No such file or directory +ERROR Invalid config, exiting abnormally +org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error processing /opt/kafka/config/zoo.cfg +Caused by: java.lang.IllegalArgumentException: serverid null is not a number +ERROR Exiting JVM with code 2 +``` + +Kafka then connects to `cpaas-zookeeper:2181`, loses the channel (`EndOfStreamException: channel for sessionid 0x0 is lost`), and exits with `kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING`, so Kafka crash-loops behind ZooKeeper. The same full volume can sit behind the local PVs `elasticsearch-log-node-pv-*`, `elasticsearch-log-master-pv-*`, and `zookeeper-pv-*` on several log nodes at once. + +A full filesystem by itself returns `No space left on device`. `Read-only file system` can also mean the mount has gone read-only; check `findmnt -o TARGET,OPTIONS -T /cpaas/data`. If it is `ro`, freeing space does not help until the node filesystem is repaired. + +ZooKeeper snapshots and transaction logs live inside its data directory (`version-2`); its runtime logs are a separate tree. The data directory, and `myid` in it, are not disposable. + +### Cause 3: FailedScheduling lists node.kubernetes.io/disk-pressure because Elasticsearch shares the root disk + +`/cpaas` has no disk of its own, so Elasticsearch data fills the root filesystem, and the kubelet taints the node with `node.kubernetes.io/disk-pressure`. The scheduling event lists that taint, often next to a control-plane taint or a volume node-affinity conflict: + +```text +FailedScheduling: 0/13 nodes are available: 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 3 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }, 5 node(s) had volume node affinity conflict. +``` + +The same root cause shows up as `nevermore` and other pods restarting while `/` is above 85%. + +### Cause 4: the global masters hold both Elasticsearch and MinIO + +Both components store data on the local disk of the global masters. There is no procedure to move Elasticsearch onto other global nodes; that relocation is a feature request. What can be done now is index cleanup or a larger disk. + +### Cause 5: after space is freed, shard recovery is throttled and Kafka still cannot reach ZooKeeper + +The components do not all come back by themselves after the disk is cleaned. Elasticsearch recovers shards at most 2 incoming and 2 outgoing at a time per node by default, so the allocation explain shows `THROTTLE` and recovery of a large store is slow; raising concurrency to 10 and recovery bandwidth to 400MB for the duration, then reverting, speeds it up. Kafka lost its ZooKeeper connection while the disk was full and keeps failing with `Fatal error during KafkaServer startup` and `ZooKeeperClientTimeoutException`; restarting the ZooKeeper pods first and then the Kafka pods restores it. + +### Cause 6: /cpaas is 100% full and the stored logs can be discarded + +The log storage plugin keeps restarting because Elasticsearch, Kafka, and ZooKeeper under `/cpaas/data` cannot write, and the owner accepts losing all stored logs. Stopping the log components, clearing the Elasticsearch, Kafka, and ZooKeeper data on the data nodes, and starting the stores in order brings the plugin back. This is the only full-disk case in which ZooKeeper data is cleared. In a narrower variant only the ZooKeeper and Kafka data were cleared; the Elasticsearch indices came back yellow and recovered once their shards were allocated again. Check with `du` which directory actually holds the bytes first: in one case `/cpaas/data/elasticsearch/data` was only 157G of a full 1.8T. + +### Cause 7: /cpaas is filled by data that is not the current log store + +`du` names a directory that does not belong to the running Elasticsearch: Prometheus monitoring data (a `monitoring` directory) on the same disk as the log storage, or ClickHouse log data under `/cpaas/data` from an earlier ClickHouse log storage installation that the console no longer lists. Removing that data frees the space. `/cpaas/audit`, `/cpaas/event`, and `/cpaas/log` are not this cause; they are the live queue of the `nevermore` log agent. + +### Cause 8: the console is unreachable after the log disk filled, and apollo stays abnormal + +`auth-manager` is `Failed`, and `apollo`, Kafka, and ZooKeeper pods are abnormal. Kafka and ZooKeeper failed because `/cpaas` on the log storage node was full. After the disk was freed, `apollo` stayed abnormal and recovered only after the Secret `global-kubeconfig` was backed up and deleted and `apollo` was restarted. Why that Secret blocked `apollo` is not known. + +## Resolution + +Start with `df` and `du`, so that you do not delete Elasticsearch data to fix a log collector cache, or the reverse: + +```bash +df -hT /cpaas /var /var/cpaas +du -x -d 1 -h /cpaas /var/cpaas 2>/dev/null +``` + +### Cause 1: the log storage plugin is abnormal and /cpaas is at 100%, so the console shows no logs + +Expanding the disk is the fix. Deleting historical indices (Cause 1 of the audit-indices article linked in the Issue) is the alternative when expansion is not possible. + +1. Find what backs `/cpaas` on the log storage node: + +```bash +findmnt -T /cpaas -o TARGET,SOURCE,FSTYPE +lsblk +vgs; lvs +``` + +> **Warning:** Growing the volume under a mounted `/cpaas` changes the block layer of the live log store; a wrong device or a failed resize can leave the filesystem unmountable. Precondition: `/cpaas` is the full filesystem, it is an LVM logical volume, the virtual disk was already enlarged (or a new disk was added), and the `pvs`/`vgs`/`lvs` output is saved. Rollback: a filesystem grow cannot be undone; restore from backup if the mount fails. + +2. Grow the physical volume and the logical volume together with its filesystem: + +```bash +pvresize /dev/ +lvextend -r -l +100%FREE /dev// +df -h /cpaas +``` + +For a new disk, run `vgextend /dev/` instead of `pvresize`. + +3. Watch the logging pods return to Running: + +```bash +kubectl -n cpaas-system get pod | grep -Ei 'elastic|kafka|zookeeper|lanaya|nevermore' +``` + +4. Logs written while the disk was full reach the console only after the queued backlog is consumed. Wait for that before you test log query. + +### Cause 2: /cpaas/data is 100%, ZooKeeper does not start, and the Elasticsearch API returns 503 + +1. Map the full volume to the local PVs it backs before deciding what to clean: + +```bash +df -h /cpaas +kubectl get pv | grep -E 'elasticsearch-log|zookeeper' +kubectl -n cpaas-system get pod -o wide | grep -Ei 'zookeeper|zk|kafka' +kubectl -n cpaas-system logs --tail=50 | grep -E 'serverid|Read-only|Exiting' +``` + +2. Free space by expanding the disk (as in Cause 1) or by removing Elasticsearch data as described in the audit-indices article, whichever `du` supports. Do not delete the ZooKeeper data directory to make room. Trim old ZooKeeper snapshots only with ZooKeeper's own purge (`zkCleanup.sh` or `autopurge`, which keeps at least the three newest snapshots). +3. After `/cpaas` has free space, recreate the Elasticsearch pods if they are crash-looping on the full disk. + +> **Warning:** Deleting the Elasticsearch pod interrupts log search and ingest until it is Ready. Precondition: the disk is no longer 100%. Rollback: none needed; the StatefulSet recreates the pod. + +```bash +kubectl -n cpaas-system get pod | grep -Ei 'elastic|zookeeper|kafka' +kubectl -n cpaas-system delete pod cpaas-elasticsearch-0 +``` + +If the API still returns 503 after the disk is free and the pods are Running, contact support for the Elasticsearch cluster recovery procedure. Do not start a full log-cluster rebuild that deletes ZooKeeper data just because ZooKeeper failed on a full disk. + +### Cause 3: FailedScheduling lists node.kubernetes.io/disk-pressure because Elasticsearch shares the root disk + +1. Confirm that `/cpaas` has no mount of its own: + +```bash +findmnt -T /cpaas -o TARGET,SOURCE,FSTYPE +ls /cpaas +du -sh /cpaas/* | grep G +``` + +The same `SOURCE` as `/` confirms it. `ls /cpaas` on such a node lists `audit`, `data`, `event`, and `log`. + +2. If these nodes are not meant to be log storage nodes (they are not monitoring-data storage nodes), uninstall Elasticsearch from them through the plugin path used to install logging, then clear the leftover files once no Elasticsearch process has them open. + +> **Warning:** Uninstalling Elasticsearch on these nodes drops the log indices stored there. Precondition: the nodes are not monitoring-data storage nodes, and the log owner accepts the loss. Rollback: reinstall; deleted indices do not return. + +```bash +kubectl -n cpaas-system get sts,pod | grep -i elastic +df -hT / /cpaas +kubectl get pod -A -o wide --field-selector spec.nodeName= | grep -v Running +``` + +3. When Elasticsearch has to stay on these nodes, clean up the log data and mount a separate disk for `/cpaas` (log storage). Before cleaning, find which components produce the log volume (Diagnostic Steps) so the new disk does not fill the same way. + +### Cause 4: the global masters hold both Elasticsearch and MinIO + +```bash +kubectl get pod -A -o wide | grep -Ei 'elastic|minio' +du -x -d 1 -h /cpaas +``` + +Clean up indices as described in the audit-indices article, or expand the disk. Moving Elasticsearch to other global nodes has no supported procedure. + +### Cause 5: after space is freed, shard recovery is throttled and Kafka still cannot reach ZooKeeper + +1. Check why shards wait (set `ES_USER` and `ES_PASS` as in the audit-indices article, and `ES_HOST` and `ES_PORT` as in Diagnostic Steps): + +```bash +curl -s -u "$ES_USER:$ES_PASS" "http://$ES_HOST:$ES_PORT/_cluster/allocation/explain?pretty" | grep -E '"decision"|reached the limit' +``` + +2. Raise the recovery limits for the duration, and reset them when health is green: + +```bash +curl -s -u "$ES_USER:$ES_PASS" -X PUT "http://$ES_HOST:$ES_PORT/_cluster/settings" -H 'Content-Type: application/json' \ + -d '{"transient":{"cluster.routing.allocation.node_concurrent_recoveries":10,"indices.recovery.max_bytes_per_sec":"400mb"}}' +# when health is green: +curl -s -u "$ES_USER:$ES_PASS" -X PUT "http://$ES_HOST:$ES_PORT/_cluster/settings" -H 'Content-Type: application/json' \ + -d '{"transient":{"cluster.routing.allocation.node_concurrent_recoveries":null,"indices.recovery.max_bytes_per_sec":null}}' +``` + +3. If Kafka still fails on ZooKeeper, restart ZooKeeper first, then Kafka. + +> **Warning:** Restarting ZooKeeper and Kafka pauses log ingestion until both are Ready. Precondition: `/cpaas` has free space and the previous Kafka log shows `ZooKeeperClientTimeoutException`. Rollback: none needed; the StatefulSets recreate the pods. + +```bash +kubectl -n cpaas-system logs cpaas-kafka-0 -p --tail 50 | grep -E 'Fatal error during KafkaServer startup|ZooKeeperClientTimeoutException' +kubectl -n cpaas-system delete pods -l service_name=cpaas-zookeeper +kubectl -n cpaas-system rollout status sts/cpaas-zookeeper +kubectl -n cpaas-system delete pods -l service_name=cpaas-kafka +kubectl -n cpaas-system rollout status sts/cpaas-kafka +``` + +### Cause 6: /cpaas is 100% full and the stored logs can be discarded + +Only when `du` shows the log store holds the bytes and the owner accepts losing every stored log and unconsumed Kafka record. Follow the stop, clear, and start sequence in Cause 2 of [Platform log Kafka brokers fail: no leader election, exit 137, Pending on DiskPressure, Ceph write errors, or OOM on ACP](Platform_log_Kafka_brokers_fail_no_leader_election_exit_137_Pending_on_DiskPressure_Ceph_write_errors_or_OOM_on_ACP.md), with the data directories under `/cpaas/data` (for example `elasticsearch`, `kafka0`, and `zookeeper`). Record owner and mode with `stat` before clearing, and give recreated directories the same. If the log nodes are cordoned to keep them for logging, uncordon them before scaling up and cordon them again once the pods are Running. For the narrower variant, clear only the Kafka and ZooKeeper directories, then retry allocation (`_cluster/reroute?retry_failed=true`) until the indices are green. + +### Cause 7: /cpaas is filled by data that is not the current log store + +```bash +du -sh /cpaas/* /cpaas/data/* 2>/dev/null | sort -h | tail -20 +kubectl get pods -A -o wide | grep | grep -Ei 'prometheus|clickhouse|elasticsearch' +``` + +> **Warning:** Uninstalling Prometheus monitoring and clearing its data deletes the stored metrics and stops collection until monitoring is installed again. Precondition: the owner accepted removing Prometheus monitoring from this cluster, and the directory is its data directory on each node. Rollback: reinstall the monitoring plugin; old metrics do not return. + +Uninstall the Prometheus monitoring plugin, then on each node run `rm -rf /cpaas//*`. + +> **Warning:** Deleting old ClickHouse data removes those logs permanently. Precondition: no ClickHouse pod runs on the node (`kubectl get pods -A -o wide | grep -i clickhouse`), and the owner accepts the loss. Rollback: none. + +Remove the leftover ClickHouse data with `rm -rf /cpaas/data/`. Clean it before reinstalling ClickHouse log storage on that node. + +### Cause 8: the console is unreachable after the log disk filled, and apollo stays abnormal + +Recover the disk first. Then, only when `apollo` is still abnormal after Kafka and ZooKeeper are Running: + +> **Warning:** Deleting `global-kubeconfig` removes a Secret that `apollo` reads; what recreates it is not documented. Precondition: the backup file is written and readable, the disk is no longer full, and Kafka and ZooKeeper are Running. Rollback: `kubectl apply -f /tmp/global-kubeconfig.bak.yaml`. + +```bash +kubectl get secret -A | grep global-kubeconfig +kubectl -n get secret global-kubeconfig -o yaml > /tmp/global-kubeconfig.bak.yaml +kubectl -n delete secret global-kubeconfig +kubectl -n rollout restart deploy/ +``` + +If `apollo` stays abnormal, restore the backup and collect its log. To keep the log disk from filling again, set a project-level log retention and watch index sizes. + +## Diagnostic Steps + +1. Split `/cpaas` from `/var/cpaas` with `df` and `du`. A full `/var` with a small `/cpaas` is the log collector cache, not an Elasticsearch problem. +2. Console shows no logs, `kubectl logs` works, the global log storage plugin is abnormal, and `/cpaas` is 100% without ZooKeeper errors: Cause 1. Expand first, then wait for the backlog. +3. ZooKeeper `serverid null is not a number` together with `/cpaas` at 100%: Cause 2. +4. `FailedScheduling` with `node.kubernetes.io/disk-pressure` while Elasticsearch shares `/`: Cause 3. +5. Elasticsearch and MinIO together on the global masters: Cause 4. +6. To see which components hold the log volume, list indices by size (credentials as in the audit-indices article): + +```bash +ES_HOST=$(kubectl get svc -n cpaas-system cpaas-elasticsearch -o jsonpath='{.spec.clusterIP}') +ES_PORT=$(kubectl get svc -n cpaas-system cpaas-elasticsearch -o jsonpath='{.spec.ports[0].port}') +curl -u "$ES_USER:$ES_PASS" "http://$ES_HOST:$ES_PORT/_cat/indices?v&s=store.size:desc" | head -n 20 +``` + +7. After cleanup, `THROTTLE` in the allocation explain or `ZooKeeperClientTimeoutException` in the previous Kafka log: Cause 5. A full log store whose data may be discarded: Cause 6. The largest directory is not Elasticsearch, Kafka, ZooKeeper, or the `nevermore` queue: Cause 7. `auth-manager` `Failed` and `apollo` abnormal after the disk is free: Cause 8. +8. If `ls /cpaas` or `ls /cpaas/data` returns `Input/output error` instead of listing files, the disk is not full but the file system or the block device under it has failed. Check `dmesg` and the device (`lsblk`, and `multipath -ll` on multipath storage) before freeing space or deleting data. diff --git a/docs/en/solutions/cpaas_filled_by_Elasticsearch_audit_indices_or_leftover_Elasticsearch_data_on_ACP.md b/docs/en/solutions/cpaas_filled_by_Elasticsearch_audit_indices_or_leftover_Elasticsearch_data_on_ACP.md new file mode 100644 index 000000000..c0b63c8ac --- /dev/null +++ b/docs/en/solutions/cpaas_filled_by_Elasticsearch_audit_indices_or_leftover_Elasticsearch_data_on_ACP.md @@ -0,0 +1,232 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.12.x + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# /cpaas filled by Elasticsearch audit indices or leftover Elasticsearch data on ACP + +## Issue + +A node alarm on Alauda Container Platform names `/cpaas`, or `df` shows `/cpaas` near or at 100%, and the bytes are Elasticsearch data. Typical reports: + +- Node disk usage is above 85%, and the question is which data it is. +- Application-log retention in the log center was shortened, and the disk did not shrink. +- Elasticsearch log search fails, `curl` against the Elasticsearch API returns 503, or a delete of old indices returns success and the disk does not move. +- `/cpaas/data/elasticsearch` is very large although the cluster now uses an external Elasticsearch. +- Logs were moved to a new disk mounted at the old path, and the root disk stays full although `du` finds nothing large. +- Elasticsearch keeps restarting with a CPU alarm after an application upgrade, and the system log indices are the large ones. + +Do not treat every directory under `/cpaas` as disposable log cache: ZooKeeper data can live on the same filesystem. When `/cpaas/data` is full and ZooKeeper does not start, when the console shows no logs while `kubectl logs` works, or when Elasticsearch shares the root disk, see [/cpaas/data full with ZooKeeper down or Elasticsearch on the root disk on ACP](./cpaas_data_full_with_ZooKeeper_down_or_Elasticsearch_on_the_root_disk_on_ACP.md). + +Logging components on ACP 4.x are in `cpaas-system`: StatefulSet `cpaas-elasticsearch` and DaemonSet `nevermore`. On older versions the log workloads can be Deployments; check with `kubectl -n cpaas-system get deploy,sts | grep elastic` and use `deploy/` in the scale commands below if that is what the output shows. From inside the Elasticsearch pod the API is `http://cpaas-elasticsearch:9200`; a port-forward to `127.0.0.1:9200` is refused even when Elasticsearch is healthy. Basic-auth credentials are in Secret `elasticsearch-basic-auth` in `cpaas-system`; if that Secret does not exist, read the credentials the pod actually mounts (for example keys `ES_USER` and `ES_PASSWORD` of Secret `acp-config-secret`). + +## Root Cause + +### Cause 1: audit indices fill /cpaas after only application-log retention was shortened + +`_cat/indices` shows that the large indices are audit logs. Audit retention is a separate setting from application-log retention, and indices already written are not removed the moment a setting changes. Deleting those indices frees the space. + +If the audit growth started after an upgrade and the audit documents are dominated by Kube-OVN node `setCondition` updates, the source is an audit flood that is fixed in ACP 4.2.5; the retention and delete steps here are the relief until that upgrade. Workload indices are named like `log-workload-YYYYMMDD` on the 4.x stack, so use the size list rather than assuming a prefix. + +### Cause 2: deleting historical indices has no effect because the Elasticsearch data node already failed + +`/cpaas` on an Elasticsearch data node is full and the cluster is already in a failed state, so the index API is not a usable cleanup tool. Elasticsearch also sets `index.blocks.read_only_allow_delete` on indices when a node crosses the flood-stage disk watermark (95% by default). That block stops writes but still allows deletes; a delete that frees nothing here is the failed cluster, not the block. The block has to be cleared after the disk has free space so that ingest resumes. + +There are two recoveries: expand the disk, wait until health is not red, clear the block, and delete indices; or stop Elasticsearch and delete its data directories on all Elasticsearch data nodes, which drops every indexed log on those nodes. The second path also restores the platform when `/cpaas` doubles as the registry storage path on a global master and a package push fails because the disk is full of Elasticsearch data (with the API returning `master_not_discovered_exception` and status 503). The ZooKeeper data directory is never part of this cleanup. + +### Cause 3: leftover Elasticsearch data that the running process does not use + +`du` under `/cpaas` finds a large Elasticsearch tree that no running Elasticsearch process uses, for example data left after a move or an upgrade. Removing the leftover tree frees the space while the running Elasticsearch stays healthy. This is different from deleting the live data directory in Cause 2. + +After a switch to an external Elasticsearch, `/cpaas/data/elasticsearch` on a global node can still hold the old indices. For example, `du -sh *` shows `nodes` at 114G; under `nodes/0`, `indices` is 114G, `node.lock` 0, and `_state` 64K, and all index directories are dated long before the switch. + +### Cause 4: the old log files are hidden under the mount of the new log disk + +The logs were migrated to another disk that is mounted on the old path (for example `/cpaas/data`), and the root file system stays full. The files that were written to that path before the new disk was mounted are still on the root file system, but the mount hides them: `du` on the mounted path only sees the new disk, so nothing large shows up. They can be removed only by unmounting the new disk or by reaching the underlying file system through a second mount point. In the recovery that cleared this, the log files on the new disk were removed as well, and the log history was not kept. + +### Cause 5: system log collection left on after an application upgrade floods Elasticsearch + +After an application upgrade, system log collection stayed enabled, the system log indices grew quickly, and Elasticsearch kept restarting with a CPU alarm. Disabling system log collection and deleting those indices stopped the restarts and the alarm. There is no per-project collection quota and no switch to collect only some log levels; what exists is a retention policy per project and the retention of `log-workload` in the log policy settings. + +## Resolution + +Start with `df` and `du`, so that you do not delete Elasticsearch data to fix a log collector cache, or the reverse: + +```bash +df -hT /cpaas /var /var/cpaas +du -x -d 1 -h /cpaas /var/cpaas 2>/dev/null +``` + +A full `/var` with a small `/cpaas` is the log collector cache (`/var/cpaas`, `diskqueue` `seg` files), not an Elasticsearch index problem. + +### Cause 1: audit indices fill /cpaas after only application-log retention was shortened + +1. Shorten the audit retention in the log center. Changing only the application-log retention leaves these indices. +2. List indices by size. The commands put the password on the command line of the admin host; do not share the shell transcript. + +```bash +ES_USER=$(kubectl -n cpaas-system get secret elasticsearch-basic-auth \ + -o jsonpath='{.data.username}' | base64 -d) +ES_PASS=$(kubectl -n cpaas-system get secret elasticsearch-basic-auth \ + -o jsonpath='{.data.password}' | base64 -d) +kubectl -n cpaas-system exec cpaas-elasticsearch-0 -c cpaas-elasticsearch -- \ + curl -sS -u "${ES_USER}:${ES_PASS}" \ + 'http://cpaas-elasticsearch:9200/_cat/indices?v&h=index,docs.count,store.size&s=store.size:desc' +``` + +> **Warning:** `DELETE /` removes that index and its documents permanently. Precondition: `_cat/indices` shows the name is an expired audit or log index, the cluster health API answers, and audit retention has been shortened. Rollback: none without a snapshot stored outside this disk. Do not use a wildcard delete. + +3. Delete the expired indices one by one: + +```bash +kubectl -n cpaas-system exec cpaas-elasticsearch-0 -c cpaas-elasticsearch -- \ + curl -sS -u "${ES_USER}:${ES_PASS}" -X DELETE \ + "http://cpaas-elasticsearch:9200/" +``` + +4. Watch `df -h /cpaas` after each batch. Each node removes the shard files after it applies the delete, so usage can lag the HTTP 200 briefly. + +### Cause 2: deleting historical indices has no effect because the Elasticsearch data node already failed + +Prefer expansion: + +1. Grow the disk or filesystem that holds `/cpaas`. +2. Wait until the Elasticsearch pods are Ready and `_cluster/health` is not red. +3. Clear the read-only block, then delete indices as in Cause 1. + +> **Warning:** Clearing `index.blocks.read_only_allow_delete` lets Elasticsearch write again; if the disk is still above the flood-stage watermark, the block returns, and writes can fill the last free blocks. Precondition: `df` shows free space on the data node. Rollback: Elasticsearch re-applies the block if the watermark is still crossed; you can also set it back to `true`. + +```bash +kubectl -n cpaas-system exec cpaas-elasticsearch-0 -c cpaas-elasticsearch -- \ + curl -sS -u "${ES_USER}:${ES_PASS}" -X PUT \ + -H 'Content-Type: application/json' \ + 'http://cpaas-elasticsearch:9200/_all/_settings' \ + -d '{"index.blocks.read_only_allow_delete": null}' +``` + +If the disk cannot be expanded and the API is dead, stop Elasticsearch and remove its data directories on every Elasticsearch data node. That is a full log-history wipe. + +> **Warning:** Stopping `cpaas-elasticsearch` and deleting its data directories destroys every log index on those nodes. Precondition: expansion is impossible, index deletes already failed, the Elasticsearch data directory was identified from the pod volume mounts, it is not the ZooKeeper data directory, and the loss of log history is accepted. Rollback: none without an external snapshot. + +```bash +kubectl -n cpaas-system get sts,pod | grep -i elastic +kubectl -n cpaas-system scale sts/cpaas-elasticsearch --replicas=0 +``` + +If a `cpaas-elasticsearch-master` StatefulSet exists and its pods mount the directory you are about to delete, scale it to 0 as well, or it can recreate files while you delete them. If it does not exist, the data nodes are all the Elasticsearch nodes. Then, on each data node, after the pods are gone and `lsof` shows no Elasticsearch process in the directory: + +```bash +lsof +D | head +rm -rf +``` + +Scale the StatefulSets back to their previous replica counts. If the pods do not form a cluster afterwards, contact support for the Elasticsearch cluster recovery procedure rather than deleting more directories. + +### Cause 3: leftover Elasticsearch data that the running process does not use + +1. Compare the `du` output with the live pod mounts. Only a directory that no Elasticsearch process has open is leftover: + +```bash +kubectl -n cpaas-system get pod -o name | grep -i elastic +kubectl -n cpaas-system get pod cpaas-elasticsearch-0 \ + -o jsonpath='{range .spec.containers[*].volumeMounts[*]}{.name}{" "}{.mountPath}{"\n"}{end}' +lsof +D | head +``` + +> **Warning:** Removing a live Elasticsearch path corrupts the cluster. Precondition: `lsof` shows no Elasticsearch process in the directory, and the live mount path is a different directory. Rollback: none after deletion; move the tree aside first and delete the copy only after Elasticsearch has stayed healthy across new writes. + +```bash +mv .aside +``` + +2. For the leftover after a switch to external Elasticsearch, inspect the tree under `/cpaas/data/elasticsearch`: + +```bash +cd /cpaas/data/elasticsearch && du -sh * +du -sh nodes/0/* +ls -l --time-style=+%Y-%m-%d nodes/0/indices | head +kubectl get pod -A -o wide | grep -i elastic +lsof +D /cpaas/data/elasticsearch/nodes/0 | head +``` + +3. If the history must be kept, copy `nodes/0/indices` to another host or disk first. + +> **Warning:** Deleting the contents of `nodes/0/indices` removes the old local indices permanently. Precondition: platform logs go to the external Elasticsearch, no Elasticsearch pod runs on this node and `lsof` prints nothing for the path, the index directories predate the switch, and a backup exists if the history may be needed. Rollback: restore from that backup. + +```bash +rm -rf /cpaas/data/elasticsearch/nodes/0/indices/* +df -h /cpaas +``` + +### Cause 4: the old log files are hidden under the mount of the new log disk + +1. Confirm that the root file system is full while `du` cannot account for it, and find the mount that sits on the old path: + +```bash +df -hT / /cpaas /cpaas/data +du -x -d 1 -h / 2>/dev/null | sort -h | tail +findmnt -T /cpaas/data -o TARGET,SOURCE,FSTYPE +``` + +2. Stop the log pods that hold the mount (scale the log storage workloads to 0 as in Cause 2 and record their replica counts), then reach the root file system through a second mount point and list what lies under the old path: + +```bash +mkdir -p /mnt/rootfs +mount --bind / /mnt/rootfs +du -sh /mnt/rootfs/cpaas/data/* 2>/dev/null +``` + +> **Warning:** Deleting the hidden tree removes the old log files permanently. Precondition: the path is under `/mnt/rootfs` (the root file system seen without the new disk), not the live mount of the new log disk, and `lsof` shows no process using it. Rollback: none. + +```bash +rm -rf /mnt/rootfs/cpaas/data/ +umount /mnt/rootfs +df -h / +``` + +3. Scale the log storage workloads back to their recorded replica counts. + +### Cause 5: system log collection left on after an application upgrade floods Elasticsearch + +1. Confirm that system log indices dominate (credentials as in Cause 1): + +```bash +kubectl -n cpaas-system exec cpaas-elasticsearch-0 -c cpaas-elasticsearch -- \ + curl -sS -u "${ES_USER}:${ES_PASS}" \ + 'http://cpaas-elasticsearch:9200/_cat/indices/log-system-*?v&h=index,docs.count,store.size&s=store.size:desc' +``` + +2. Disable system log collection in the log collector settings. +3. Delete the large system log indices by name, with the same warning and command as Cause 1. +4. To keep the remaining volume down, shorten the retention of `log-workload` in the log policy settings or set a retention for the noisy project. + +## Diagnostic Steps + +1. Split `/cpaas` from `/var/cpaas` with `df` and `du` as at the start of the Resolution. +2. If the large tree is Elasticsearch data, check health first (set `ES_USER` and `ES_PASS` as in Cause 1): + +```bash +kubectl -n cpaas-system get pod,sts | grep -Ei 'elastic|nevermore|zookeeper' +kubectl -n cpaas-system exec cpaas-elasticsearch-0 -c cpaas-elasticsearch -- \ + curl -sS -u "${ES_USER}:${ES_PASS}" 'http://cpaas-elasticsearch:9200/_cluster/health?pretty' +``` + +3. Health green and the large indices are audit: Cause 1. Health red, 503, or a delete that frees nothing: Cause 2. A large Elasticsearch directory that `lsof` shows no process using: Cause 3. A full root file system that `du` cannot account for, with a log disk mounted on the old path: Cause 4. Large `log-system-*` indices after an application upgrade: Cause 5. +4. To see which components produce the log volume, list indices by size from a master node through the Service address: + +```bash +ES_HOST=$(kubectl get svc -n cpaas-system cpaas-elasticsearch -o jsonpath='{.spec.clusterIP}') +ES_PORT=$(kubectl get svc -n cpaas-system cpaas-elasticsearch -o jsonpath='{.spec.ports[0].port}') +curl -u "$ES_USER:$ES_PASS" "http://$ES_HOST:$ES_PORT/_cat/indices?v&s=store.size:desc" | head -n 20 +``` + +Indices are daily, for example `log-platform-` or `audit-`. Within a `log-platform` index, the largest producers are typically platform pods such as `lanaya`, `cpaas-monitor-prometheus-adapter`, `vmselect-cluster-0`, and `speaker`. diff --git a/docs/en/solutions/crictl_pull_fails_with_unexpected_commit_digest_while_ctr_pull_works_because_of_hosts.toml_entry_order_on_ACP.md b/docs/en/solutions/crictl_pull_fails_with_unexpected_commit_digest_while_ctr_pull_works_because_of_hosts.toml_entry_order_on_ACP.md new file mode 100644 index 000000000..7d3be25bb --- /dev/null +++ b/docs/en/solutions/crictl_pull_fails_with_unexpected_commit_digest_while_ctr_pull_works_because_of_hosts.toml_entry_order_on_ACP.md @@ -0,0 +1,92 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# crictl pull fails with unexpected commit digest while ctr pull works because of hosts.toml entry order on ACP + +## Issue + +A registry (for example Harbor exposed through a NodePort, with HTTPS added in front of it by custom forwarding rules) is reachable by domain name. Pulling an image by that domain works with `docker`, `nerdctl` and `ctr`, but `crictl pull` on the same node fails, and kubelet image pulls that go through the same CRI path fail too: + +```text +"PullImage from image service failed" err="rpc error: code = FailedPrecondition desc = failed to pull and unpack image \"//:\": failed commit on ref \"unknown-sha256:\": unexpected commit digest sha256:, expected sha256:: failed precondition" +FATA[0000] pulling image: rpc error: code = FailedPrecondition desc = failed to pull and unpack image ...: failed precondition +``` + +Clearing the containerd image store (`/var/lib/containerd`) and restarting containerd and kubelet does not help, and the same error comes back. + +## Root Cause + +`crictl` talks to the containerd CRI plugin, and the CRI plugin resolves registries through the host configuration files under `/etc/containerd/certs.d/`. On the node, the wildcard configuration `/etc/containerd/certs.d/0.0.0.0_0/hosts.toml` holds two entries, one for `https://*` and one for `http://*`. containerd tries the `[host."..."]` entries of a `hosts.toml` in the order in which they appear in the file. + +When the `http://*` entry comes before the `https://*` entry, the CRI path contacts the registry domain over plain HTTP first. Behind forwarding rules that provide HTTPS for the registry, what comes back on that path is not the expected image content. Its digest does not match the digest from the manifest, and containerd aborts the commit with `unexpected commit digest ... failed precondition`. + +Other clients are not affected because they do not resolve the registry through this file in the same way. For example, `ctr` ignores `/etc/containerd/certs.d` unless `--hosts-dir` is passed, and `docker` uses its own daemon configuration. + +## Resolution + +> **Warning:** A wrong `hosts.toml` breaks every CRI image pull on the node, which includes the kubelet pulls for new Pods. Edit one node first and verify before changing the others. Rollback: copy the saved file back. + +1. Save the current file: + + ```bash + sudo cp /etc/containerd/certs.d/0.0.0.0_0/hosts.toml /tmp/hosts.toml.bak + ``` + +2. Edit `/etc/containerd/certs.d/0.0.0.0_0/hosts.toml` so that the `https://*` entry comes **before** the `http://*` entry. Keep the existing settings of each entry: + + ```toml + [host."https://*"] + capabilities = ["pull", "resolve", "push"] + skip_verify = true + + [host."http://*"] + capabilities = ["pull", "resolve", "push"] + skip_verify = true + ``` + +3. Pull the image again with `crictl`: + + ```bash + crictl pull //: + ``` + +4. Apply the same order on the other nodes that pull from this registry. + +## Diagnostic Steps + +1. Reproduce with both clients on the same node: + + ```bash + crictl pull //: + ctr -n k8s.io images pull //: + ``` + + Expected for this issue: `crictl` fails with `unexpected commit digest ... failed precondition`, while `ctr` completes. + +2. Check the order of the entries in the wildcard host configuration: + + ```bash + sudo grep -n 'host\.' /etc/containerd/certs.d/0.0.0.0_0/hosts.toml + ls /etc/containerd/certs.d/ + ``` + + Expected for this issue: `[host."http://*"]` is listed before `[host."https://*"]`. Also check whether a directory named after the registry domain exists under `/etc/containerd/certs.d/`. A host-specific `hosts.toml` there applies to that registry instead of the wildcard one, and its entry order matters in the same way. + +3. Confirm which directory the CRI plugin reads: + + ```bash + sudo grep -n -A2 'config_path' /etc/containerd/config.toml + ``` + + Expected: `config_path` points at `/etc/containerd/certs.d`. + +4. Rule out a corrupted local store. If the error remains after the entry order is corrected, check the containerd log for the pull. An identical digest error across different registries points to something else, such as a proxy that rewrites responses: + + ```bash + journalctl -u containerd --since "10 min ago" | grep -E 'commit digest|FailedPrecondition' + ``` diff --git a/docs/en/solutions/devops-controller_or_devops-apiserver_does_not_start_or_fails_requests_on_ACP.md b/docs/en/solutions/devops-controller_or_devops-apiserver_does_not_start_or_fails_requests_on_ACP.md new file mode 100644 index 000000000..faae8e5ad --- /dev/null +++ b/docs/en/solutions/devops-controller_or_devops-apiserver_does_not_start_or_fails_requests_on_ACP.md @@ -0,0 +1,144 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "3.12" + - "4.3" +--- + +# devops-controller or devops-apiserver does not start or fails requests on ACP + +## Issue + +The DevOps API components in `cpaas-system` on the `global` cluster are down or fail requests, and pipeline pages that depend on them stay empty or stale. One of these signatures is shown: + +- The `devops-controller` Pods are not Ready. The controller log repeats `Failed to watch v1alpha1.CodeRepository`, and the integrated GitLab answers `Internal server error` on `GET /api/v4/projects?membership=true&per_page=100`. +- On a standby (backup) data center, `devops-apiserver` and `devops-controller` are both down, and the controller log says it cannot get a ConfigMap because it has no permission. +- `devops-apiserver` crash-loops with `panic: unable to create REST storage for a resource due to context deadline exceeded, will die`, after normal INFO lines such as `cacher (*devops.JenkinsBinding): initialized`. +- Updating a pipeline fails with HTTP 500 and a message like `Get "https://[::1]:443/apis/devops.alauda.io/v1alpha1/clusterpipelinetasktemplates/?resourceVersion=0": x509: certificate has expired or is not yet valid`, while the platform certificate pages show nothing relevant as expired. + +A GitLab integration whose projects API returns 500 while these components are healthy is covered in "GitLab toolchain integration fails: token-gitlab not found, Internal error, projects API 500, or an extra bot project". An empty task list with these components Ready is covered in "Pipeline task list is empty, tasks are missing, or TektonHub cannot be viewed on ACP". + +## Root Cause + +### Cause 1: `Failed to watch v1alpha1.CodeRepository` because the integrated GitLab returns Internal server error + +`devops-controller` watches `CodeRepository` objects and calls each integrated GitLab to list projects. When one GitLab answers `Internal server error` on `/api/v4/projects`, the watch fails and the controller Pods do not become Ready. The broken dependency is that GitLab integration; deleting the abnormal integration lets `devops-controller` start again. Restarting the controller does not clear it. + +### Cause 2: the DevOps API ServiceAccount lacks permission to read ConfigMaps + +On a standby data center the ServiceAccount used by the DevOps API components had fewer permissions than on the primary. The controller cannot get the ConfigMap it needs, and both `devops-apiserver` and `devops-controller` stay down. Granting the ServiceAccount the missing permission lets both recover. + +### Cause 3: etcd does not answer in time while `devops-apiserver` builds its storage + +At startup `devops-apiserver` connects to etcd and creates REST storage for its resources. If storage creation does not finish in time, the process panics from `registry.RESTInPeace` and the Pod restarts. On ACP 3.12 this was caused by an abnormal control-plane node; after the workload moved off that node the panics stopped. The first lines of the log are ordinary INFO messages, so the panic is visible only in the full log of a failed start. + +### Cause 4: the DevOps API uses a serving certificate that has expired + +The 500 response comes from a call to the aggregated `devops.alauda.io/v1alpha1` API. When no certificate in the platform views matches the expiry time in the message, the most likely explanation is that the devops component on the `global` cluster is still presenting a serving certificate that has expired. The exact mechanism is not confirmed; the recommended action is to restart `devops-apiserver` and `devops-controller`. + +## Resolution + +### Cause 1: `Failed to watch v1alpha1.CodeRepository` because the integrated GitLab returns Internal server error + +1. Confirm the log signature. A controller that is down for an image pull is not this cause. + + ```bash + kubectl -n cpaas-system get pods | grep devops-controller + kubectl -n cpaas-system logs deploy/devops-controller --tail=80 | grep -E 'CodeRepository|Internal server error' + ``` + +2. Identify the GitLab integration whose host appears in that log, and check that its projects API really fails: + + ```bash + curl -sk -H "PRIVATE-TOKEN: " "https:///api/v4/projects?membership=true&per_page=100" | head -c 300 + ``` + +3. Delete that integration from the console page where it was created (the toolchain integration list), after recording its settings. + + > **Warning:** Deleting the GitLab integration removes that code-host binding; repositories, triggers, and pipelines that use it stop resolving until it is created again. Only when `devops-controller` is not Ready, the log shows `Failed to watch v1alpha1.CodeRepository`, and this GitLab returns `Internal server error` on `/api/v4/projects`. Rollback: recreate the integration with the recorded settings once the GitLab projects API returns a project list again. Do not delete a healthy integration. + +4. Watch the controller recover: + + ```bash + kubectl -n cpaas-system rollout status deploy/devops-controller + ``` + + If the Pods stay down and the log no longer mentions GitLab, follow the new log. + +### Cause 2: the DevOps API ServiceAccount lacks permission to read ConfigMaps + +1. Find the ServiceAccount and test the permission named in the log: + + ```bash + kubectl -n cpaas-system get deploy devops-controller -o jsonpath='{.spec.template.spec.serviceAccountName}{"\n"}' + kubectl auth can-i get configmaps -n cpaas-system --as=system:serviceaccount:cpaas-system: + kubectl get clusterrolebinding -o wide | grep + ``` + + `no` for that ServiceAccount is this cause. Compare its bindings with the same ServiceAccount on the primary data center. + +2. Restore the binding the primary cluster has for this ServiceAccount. If that binding is not available, binding the ServiceAccount to `cluster-admin` also lets the components recover. + + > **Warning:** A `cluster-admin` binding gives every Pod that runs as this ServiceAccount full control of the cluster. Only when the log shows the ConfigMap permission error and `kubectl auth can-i` returns `no`, and only when the primary's exact binding cannot be copied. Rollback: `kubectl delete clusterrolebinding `. + + ```bash + kubectl create clusterrolebinding --clusterrole=cluster-admin --serviceaccount=cpaas-system: + kubectl -n cpaas-system rollout restart deploy/devops-controller + ``` + +3. `devops-apiserver` and `devops-controller` should become Ready and the permission error should leave the log. + +### Cause 3: etcd does not answer in time while `devops-apiserver` builds its storage + +1. Read the log of the failed start and check the control plane: + + ```bash + kubectl -n cpaas-system logs deploy/devops-apiserver --previous | grep -E 'panic|deadline' + kubectl get nodes -l node-role.kubernetes.io/control-plane= + kubectl -n kube-system get pods -l component=etcd -o wide + ``` + + The panic together with a NotReady control-plane node or an etcd member that is not Running is this cause. + +2. Restarting `devops-apiserver` does not help while etcd is slow. Repair the control-plane node, or move off it, following the cluster's control-plane maintenance procedure. `devops-apiserver` should then start without the panic; if the panic returns, the control-plane problem is still open. + +### Cause 4: the DevOps API uses a serving certificate that has expired + +1. Only when the 500 body contains `x509: certificate has expired or is not yet valid` for a `devops.alauda.io` path. Check the APIService and how long the Pods have run: + + ```bash + kubectl get apiservice | grep devops.alauda.io + kubectl -n cpaas-system get pods -o wide | grep -E 'devops-apiserver|devops-controller' + kubectl -n cpaas-system logs deploy/devops-apiserver --tail=100 | grep -i x509 + ``` + +2. Restart both components on the `global` cluster. + + > **Warning:** The DevOps API is unavailable until the new Pods are Ready; pipeline pages and API calls fail in that window. Rollback: none; a restart changes no configuration. + + ```bash + kubectl -n cpaas-system rollout restart deploy/devops-apiserver deploy/devops-controller + kubectl -n cpaas-system rollout status deploy/devops-apiserver + kubectl -n cpaas-system rollout status deploy/devops-controller + ``` + +3. Save the pipeline again. If the x509 error returns after the restart, the certificate in its Secret is itself expired; collect the `devops-apiserver` log and `kubectl get apiservice -o yaml` for the product team. + +## Diagnostic Steps + +```bash +kubectl -n cpaas-system get deploy,pods | grep -E 'devops-controller|devops-apiserver' +kubectl -n cpaas-system logs deploy/devops-controller --tail=80 +kubectl -n cpaas-system logs deploy/devops-apiserver --previous --tail=200 +``` + +| What you see | Cause | +| --- | --- | +| `Failed to watch v1alpha1.CodeRepository` plus GitLab `Internal server error` on `/api/v4/projects` | 1 | +| Controller log: cannot get a ConfigMap, no permission; `kubectl auth can-i` returns `no` | 2 | +| `devops-apiserver` panics with `unable to create REST storage ... context deadline exceeded` | 3 | +| Pipeline update returns 500 with `x509: certificate has expired or is not yet valid` on a `devops.alauda.io` path | 4 | +| GitLab webhook delivery fails while `devops-controller` is Ready | Not this article | diff --git a/docs/en/solutions/docker_push_to_Harbor_from_Docker_Desktop_times_out_with_proxyconnect_i_o_timeout.md b/docs/en/solutions/docker_push_to_Harbor_from_Docker_Desktop_times_out_with_proxyconnect_i_o_timeout.md new file mode 100644 index 000000000..8e8a4f82c --- /dev/null +++ b/docs/en/solutions/docker_push_to_Harbor_from_Docker_Desktop_times_out_with_proxyconnect_i_o_timeout.md @@ -0,0 +1,65 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.3" +--- + +# docker push to Harbor from Docker Desktop times out with proxyconnect i/o timeout + +## Issue + +`docker push` from a workstation running Docker Desktop (for example on macOS) to the platform Harbor times out. The error names a proxy rather than Harbor: + +```text +proxyconnect tcp: dial tcp :3128: i/o timeout +``` + +The proxy on port `3128` is Docker Desktop's own HTTP proxy (`http.docker.internal:3128`). `docker info` shows an `HTTP Proxy` / `HTTPS Proxy`, and its `No Proxy` list contains only entries such as `hubproxy.docker.internal`, not the Harbor address. + +Related articles: [Harbor push of large image layers fails with unexpected EOF, timeout awaiting response headers, or 408 on ACP](Harbor_push_of_large_image_layers_fails_with_unexpected_EOF_timeout_awaiting_response_headers_or_408_on_ACP.md) and [Harbor image push fails while harbor-nginx restarts, or is rejected with too large on a NodePort Harbor](Harbor_image_push_fails_while_harbor-nginx_restarts_or_is_rejected_with_too_large_on_a_NodePort_Harbor.md). + +## Root Cause + +### Cause 1: Docker Desktop sends the registry traffic to its proxy + +Docker Desktop is configured with an HTTP/HTTPS proxy, and the Harbor address is not in its `No Proxy` (bypass) list. The engine therefore sends the push to the proxy instead of connecting to Harbor directly. The proxy cannot reach the internal Harbor address, so the connection times out before any layer is uploaded. Adding the Harbor address to the `No Proxy` list in Docker Desktop stops the timeout. + +## Resolution + +### Cause 1: Docker Desktop sends the registry traffic to its proxy + +1. In Docker Desktop, open the proxy settings (**Settings > Resources > Proxies** in current releases) and add the Harbor host or IP address, with and without the port as your clients use it, to the bypass / `No Proxy` list. Keep the existing entries. +2. Apply the change. Docker Desktop restarts its engine, which stops running containers on that workstation until it is back. +3. Check that the address is now bypassed and retry: + + ```bash + docker info | grep -iE 'HTTP Proxy|HTTPS Proxy|No Proxy' + docker login : + docker push ://: + ``` + +On a Linux host with `dockerd`, the same rule applies: the daemon's proxy configuration must list the Harbor address in `NO_PROXY` / `no-proxy`. + +If the timeout is gone but large layers now stop in `Pushing` and the push ends with `unexpected EOF`, the proxy is fixed and the remaining failure is on the network path; continue with Cause 2 of [Harbor push of large image layers fails with unexpected EOF, timeout awaiting response headers, or 408 on ACP](Harbor_push_of_large_image_layers_fails_with_unexpected_EOF_timeout_awaiting_response_headers_or_408_on_ACP.md). Increasing the Docker Desktop memory does not fix that. + +## Diagnostic Steps + +1. Read the engine's proxy settings and the shell's proxy environment: + + ```bash + docker info | grep -iE 'HTTP Proxy|HTTPS Proxy|No Proxy' + env | grep -iE 'http_proxy|https_proxy|no_proxy' + ``` + + An `HTTP Proxy` / `HTTPS Proxy` entry, with a `No Proxy` list that does not contain the Harbor address, matches this cause. + +2. Test Harbor directly, bypassing any proxy: + + ```bash + curl -vk --noproxy '*' --connect-timeout 5 https://:/v2/ + ``` + + A response (`401` is normal without credentials) proves that Harbor is reachable directly and only the proxy path fails. A timeout here as well means the workstation cannot reach Harbor at all; check the network path and firewall instead, see [Image registry or Harbor unreachable or slow because of firewall rules, blocked ports, or the network path](Image_registry_or_Harbor_unreachable_or_slow_because_of_firewall_rules_blocked_ports_or_the_network_path.md). diff --git a/docs/en/solutions/ecosystem/nacos/How_to_Deploy_Nacos_2.2.md b/docs/en/solutions/ecosystem/nacos/How_to_Deploy_Nacos_2.2.md index b85ff4a18..08040e4a8 100644 --- a/docs/en/solutions/ecosystem/nacos/How_to_Deploy_Nacos_2.2.md +++ b/docs/en/solutions/ecosystem/nacos/How_to_Deploy_Nacos_2.2.md @@ -257,3 +257,45 @@ Nacos resolves configs by composing names; the log line is expected during start ### Q8. After a Nacos 2.2.3 ephemeral instance is taken offline, the pod remains registered under the service Known community issue affecting Nacos 2.2.3. Fixed in 2.3.x. Track upstream: . To eliminate the symptom permanently, redeploy Nacos using the [Nacos 2.5 plan](./How_to_Deploy_Nacos_2.5.md). + +### Q9. How do I move Nacos members on TopoLVM to new nodes, for example before reinstalling the node operating system? + +TopoLVM volumes are local to one node, so a Nacos Pod cannot move to another node together with its PVCs. Move the members one at a time and let each one come up on a new node with new, empty volumes. Nacos configuration is kept in MySQL (see the MySQL prerequisite above), which this procedure does not touch. + +1. Prepare one new node per Nacos member (for example four new nodes for a four-member cluster), each with TopoLVM storage and enough free capacity for the member's log and data volumes. +2. Taint every node that currently runs a Nacos Pod, so that the Pods cannot be scheduled back onto them: + + ```bash + kubectl get pod -n -o wide | grep -i nacos + kubectl taint nodes nacos-migration=true:NoSchedule + ``` + +3. For one member, delete its log PVC and its data PVC first, then delete the Pod. Taking member 0 as an example: + + > **Warning:** Deleting the PVCs permanently deletes the member's local volumes on the old node. Move only one member at a time, and only while all other members are Ready, so that the cluster keeps serving. Do not reinstall an old node before every member has moved. Rollback for a member that has not been deleted yet: remove the taint from its node (`kubectl taint nodes nacos-migration=true:NoSchedule-`). + + ```bash + kubectl get pvc -n + kubectl delete pvc -n --wait=false + kubectl delete pod -n + ``` + +4. Wait until the recreated Pod is running on a new node and is Ready, with new PVCs bound: + + ```bash + kubectl get pod -n -o wide -w + kubectl get pvc -n + ``` + +5. Repeat steps 3 and 4 for the next member until every member runs on a new node. Then remove the taints from the old nodes or take the nodes out of service. + +### Q10. Can workloads in another cluster reach this Nacos through its `svc.cluster.local` names? + +No. The in-cluster names in `NACOS_SERVERS` (`nacos-server-.nacos-server..svc.cluster.local:8848`) resolve only inside the cluster where Nacos runs, and the multi-cluster Headless Service domain names (`.amcs-..svc.cluster.local`) are not a supported way to reach Nacos. Cross-cluster access to Nacos is supported only by exposing the Nacos service outside its cluster through `NodePort` or `LoadBalancer`, as described under Architecture Overview, and pointing the clients in the other cluster at that address. Expose the gRPC port as well, and keep it at exactly the HTTP port plus 1000 on the outside (for example `8848` and `9848`). + +```bash +kubectl -n get svc -o wide +kubectl -n get sts -o jsonpath='{range .spec.template.spec.containers[0].env[*]}{.name}{"="}{.value}{"\n"}{end}' | grep -E 'NACOS_SERVERS|MODE|PREFER_HOST_MODE' +``` + +Only `ClusterIP` or headless Services in the output means Nacos is not yet reachable from another cluster. diff --git a/docs/en/solutions/etcd-sync_Pod_stuck_in_ContainerCreating_with_failed_to_sync_secret_cache_on_the_standby_global_cluster_on_ACP.md b/docs/en/solutions/etcd-sync_Pod_stuck_in_ContainerCreating_with_failed_to_sync_secret_cache_on_the_standby_global_cluster_on_ACP.md new file mode 100644 index 000000000..51b6fafce --- /dev/null +++ b/docs/en/solutions/etcd-sync_Pod_stuck_in_ContainerCreating_with_failed_to_sync_secret_cache_on_the_standby_global_cluster_on_ACP.md @@ -0,0 +1,91 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# etcd-sync Pod stuck in ContainerCreating with failed to sync secret cache on the standby global cluster on ACP + +## Issue + +On the standby `global` cluster of a primary/standby (disaster recovery) setup, a new `etcd-sync` Pod in `cpaas-system` stays in `ContainerCreating`. `kubectl describe pod` shows: + +```text +Warning FailedMount ... MountVolume.SetUp failed for volume "etcd-sync-token" : failed to sync secret cache: timed out waiting for the condition +``` + +The Secrets the Pod uses (`etcd-sync-token`, `local-etcd-client`, `remote-etcd-client`) are listed by `kubectl get secret -n cpaas-system`, and other `etcd-sync` or `etcd-sync-monitor` Pods may still be `Running`. Listing Secrets across all namespaces fails with an error like: + +```text +Error from server (InternalError): Internal error occurred: unable to transform key "/registry/secrets//": invalid padding on input +``` + +## Root Cause + +The Kubernetes API server encrypts Secrets at rest with the keys in its encryption provider configuration, `/etc/kubernetes/encryption-provider.conf` on each control plane node. etcd synchronization copies the primary cluster's etcd data, including Secrets encrypted with the primary cluster's keys, into the standby cluster's etcd. The standby cluster can only decrypt that data when its control plane nodes use the same encryption provider configuration as the primary cluster. + +For that reason the standby installation procedure requires `/etc/kubernetes/encryption-provider.conf` to be copied from the first control plane node of the primary cluster to every standby control plane node **before** the standby cluster is installed. When this step is skipped, the standby cluster is installed with its own keys. The API server cannot decrypt the synchronized Secrets and returns `unable to transform key ... invalid padding on input`. The kubelet then cannot read the Secret behind the `etcd-sync-token` volume, and the Pod never leaves `ContainerCreating`. + +## Resolution + +The encryption configuration of a cluster is fixed when the cluster is installed. A standby cluster that was installed without the primary's encryption configuration has to be rebuilt. + +> **Warning:** Rebuilding the standby cluster removes it and its data. The primary cluster and its workloads are not affected, but there is no disaster recovery standby until the rebuild and the etcd synchronization have finished. Do this only when the diagnostic steps confirm the `invalid padding on input` error and a differing `encryption-provider.conf` on the standby control plane nodes. Plan a maintenance window for it. There is no rollback once the standby has been removed. Keep the primary cluster untouched throughout. + +1. Remove the standby cluster according to the product documentation. + +2. Log in to the first control plane node of the **primary** cluster and copy the encryption configuration to every control plane node of the standby cluster: + + ```bash + for i in + do + ssh "@$i" "sudo mkdir -p /etc/kubernetes/" + scp /etc/kubernetes/encryption-provider.conf "@$i:/tmp/encryption-provider.conf" + ssh "@$i" "sudo install -o root -g root -m 600 /tmp/encryption-provider.conf /etc/kubernetes/encryption-provider.conf && rm -f /tmp/encryption-provider.conf" + done + ``` + +3. Install the standby cluster again by following the standby installation steps of the global cluster disaster recovery documentation. The Platform Access Address, the certificate fields and the image repository fields must be the same as on the primary cluster. + +4. Enable etcd synchronization again and confirm that the `etcd-sync` Pods become `Running`: + + ```bash + kubectl -n cpaas-system get pods | grep etcd-sync + ``` + +## Diagnostic Steps + +1. Confirm the mount failure on the Pod: + + ```bash + kubectl -n cpaas-system get pods | grep etcd-sync + kubectl -n cpaas-system describe pod | grep -A3 FailedMount + ``` + + Expected for this issue: `failed to sync secret cache: timed out waiting for the condition` for volume `etcd-sync-token`. + +2. Confirm that the Secrets exist, so that the problem is not a missing Secret: + + ```bash + kubectl -n cpaas-system get secret etcd-sync-token local-etcd-client remote-etcd-client + ``` + + If a Secret is missing (`NotFound`), create it as described in the etcd synchronization steps instead. That is a different cause. + +3. Read Secrets through the API server: + + ```bash + kubectl get secret -A > /dev/null + ``` + + Expected for this issue: `unable to transform key "/registry/secrets/...": invalid padding on input`. This means that etcd holds Secrets the API server cannot decrypt. + +4. Compare the encryption configuration on the control plane nodes of both clusters: + + ```bash + sudo sha256sum /etc/kubernetes/encryption-provider.conf + ``` + + Run this on the first control plane node of the primary cluster and on each standby control plane node. Different checksums confirm that the file was not copied before the standby cluster was installed. diff --git a/docs/en/solutions/etcd-sync_check_reports_localMissedKeys_or_localSurplusKeys_or_what_a_primary_standby_global_pair_replicates_on_ACP.md b/docs/en/solutions/etcd-sync_check_reports_localMissedKeys_or_localSurplusKeys_or_what_a_primary_standby_global_pair_replicates_on_ACP.md new file mode 100644 index 000000000..54c5b8b5d --- /dev/null +++ b/docs/en/solutions/etcd-sync_check_reports_localMissedKeys_or_localSurplusKeys_or_what_a_primary_standby_global_pair_replicates_on_ACP.md @@ -0,0 +1,152 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" + - "4.3" +--- + +# etcd-sync check reports localMissedKeys or localSurplusKeys, or what a primary/standby global pair replicates on ACP + +## Issue + +On the standby global cluster of a primary/standby (disaster recovery) setup, the sync check of the etcd Synchronizer (`curl "http:///check"`) or a daily inspection lists keys under `LOCAL ETCD missed keys` / `localMissedKeys` or `LOCAL ETCD surplus keys` / `localSurplusKeys`, and it is not clear whether they need action. Examples: + +- Right after disaster recovery is set up: `localMissedKeys [0]` and `localSurplusKeys [2]`, the two keys being `/registry/auth.alauda.io/disabletokens/` and `/registry/monitoring.coreos.com/servicemonitors/cpaas-system/etcd-sync-monitor`. +- A daily inspection with `localMissedKeys [4]` and `localSurplusKeys [88]`. +- `/registry/secrets/cpaas-logging/clickhouse-operator` stays in the missed list after the `etcd-sync` Pod is restarted several times. +- Dozens of missed keys under `/registry/ait.alauda.io/notifications/...`, or a question about which objects the pair replicates, or whether the nodes of the two clusters need full network connectivity. + +Missed keys exist on the primary and not on the standby. Surplus keys exist only on the standby. + +Related articles: "etcd Synchronizer plugin install stuck or etcd-sync does not start on the standby global cluster on ACP"; "etcd-sync Pod stuck in ContainerCreating with failed to sync secret cache on the standby global cluster on ACP". + +## Root Cause + +### Cause 1: platform surplus keys and rotated platform objects are expected differences + +Some keys differ between the clusters by design and need no action: + +- `/registry/monitoring.coreos.com/servicemonitors/cpaas-system/etcd-sync-monitor`: the ServiceMonitor of the etcd Synchronizer, which is installed only on the standby, so the primary has no such key. +- `/registry/auth.alauda.io/disabletokens/`: created when an account logs out, to ban that user's unexpired tokens. +- dex `authrequests` and `refreshtokens`, and `auth.alauda.io` sessions and groups. +- Rotated certificate and token Secrets in platform namespaces, for example `cpaas-system/dex.tls-*`, `etcd-ca-*`, `base-operator.tls-*`, `argocd-manager-token-*`, and operator service certificates such as `/-service-cert`. +- Platform alert and monitoring rules (`ait.alauda.io/alertrules/cpaas-system/cpaas-*-rules`, `monitoring.coreos.com/prometheusrules/cpaas-system/cpaas-*-rules`, `operator.victoriametrics.com/vmrules/cpaas-system/cpaas-*-rules`), and the etcd-sync monitor dashboard. +- Inspection results (`ait.alauda.io/inspections/inspection--` and `inspectiondetails/...`), a `cluster.alauda.io/moduleinfoes/` entry, and an empty `productentries` key. + +With `localMissedKeys [0]`, the synchronization is normal; surplus keys from the list above do not affect business workloads or platform use. A daily inspection whose 4 missed and 88 surplus keys all fall into these classes needs no handling either. + +### Cause 2: a few missed keys clear after the etcd-sync leader Pod is restarted + +Missed keys are often caused by garbage collection during synchronization, when objects arrive in an order that leaves their owner references unresolved. Restarting the current etcd-sync leader Pod runs a full sync again and clears them. + +### Cause 3: the clickhouse-operator Secret stays missed and a restart does not create it + +`localMissedKeys [1]`: `/registry/secrets/cpaas-logging/clickhouse-operator`, a Secret with an `ownerReferences` entry of kind `ClusterServiceVersion` (for example `clickhouse-operator.v4.2.1`). Restarting the `etcd-sync` Pod more than once leaves the key missing. The standby copy has to be created by hand, with its owner reference pointing to the CSV on the standby. + +### Cause 4: plugins, feature gates, component YAML edits, OperatorHub installs, and customized keys are not in the sync set + +From 4.2 on, the sync program runs on the standby cluster and pulls from the primary: a full sync when it starts, then incremental sync. Only the keys listed in the ConfigMap that the sync workload mounts are replicated. Installed plugins, feature-gate settings, YAML edits of platform components, and OperatorHub installs are not replicated; each of those changes has to be applied on both clusters. Customized keys are also outside the set, for example custom notification objects (`/registry/ait.alauda.io/notificationgroups//...`, `notifications/...`, `notificationtemplates/...`). Their absence on the standby is expected, not a failed sync. + +Losing one primary etcd member does not stop synchronization. Losing two or more interrupts it until the primary recovers; the sync Pod restart then runs a full sync again. + +### Cause 5: which network the primary and standby global clusters need + +The two clusters do not need full node-to-node connectivity. The reference design: + +- Both global clusters are installed with the same platform access domain. Normally it resolves to the primary cluster's VIP; on failover it is re-resolved to the standby cluster's VIP. +- Each cluster has its own VIP, and each VIP forwards TCP `443` (and `80` when the platform is accessed over HTTP), `6443`, `2379` (unless the etcd endpoints are reached directly), and `11443` to that cluster's own control-plane nodes. +- Those ports are allowed in both directions between the primary and standby networks, because the roles reverse after failover. +- etcd data is copied from the primary to the standby by the etcd Synchronizer. + +## Resolution + +### Cause 1: platform surplus keys and rotated platform objects are expected differences + +Compare each listed key with the classes above. If `localMissedKeys` is `0` and every surplus key belongs to those classes, no action is needed. Do not delete these keys from the standby. Only a difference in a key that the business itself writes needs to be reviewed with the operations team before anything is deleted. + +### Cause 2: a few missed keys clear after the etcd-sync leader Pod is restarted + +1. Find the current leader Pod: + + ```bash + kubectl get po -n cpaas-system -l app=etcd-sync + kubectl get lease -n cpaas-system etcd-sync-mirror -o jsonpath='{.spec.holderIdentity}{"\n"}' + ``` + + On versions without this Lease, restart one of the `etcd-sync` Pods. + +2. Restart it. + + > **Warning:** Deleting the leader Pod restarts the sync process; synchronization pauses until a Pod takes over and runs a full sync. Precondition: only when the missed list is small and does not name a key covered by Cause 3. Rollback: none needed; the Deployment recreates the Pod. + + ```bash + kubectl delete pod -n cpaas-system + ``` + +3. Run the check again. A business key that is still missing after the restart is Cause 3 or Cause 4, not a reason for further restarts. + +### Cause 3: the clickhouse-operator Secret stays missed and a restart does not create it + +1. On the primary, export the Secret. On the standby, read the uid of the ClickHouse operator CSV: + + ```bash + # primary + kubectl -n cpaas-logging get secret clickhouse-operator -o yaml > /tmp/clickhouse-operator.secret.yaml + # standby + kubectl -n cpaas-logging get csv | grep -i clickhouse + kubectl -n cpaas-logging get csv -o jsonpath='{.metadata.uid}{"\n"}' + ``` + +2. In the saved file, remove `metadata.creationTimestamp`, `metadata.resourceVersion`, `metadata.uid`, and `metadata.managedFields`, and set the `uid` under `ownerReferences` (kind `ClusterServiceVersion`) to the standby CSV uid just printed. Use the CSV name that `get csv` shows on the standby. +3. Apply it on the standby. + + > **Warning:** This creates or overwrites `clickhouse-operator` in `cpaas-logging` on the standby. Precondition: only when that name is the missed key and an `etcd-sync` restart did not create it. A wrong owner uid leaves the operator unable to adopt the Secret. Rollback: `kubectl -n cpaas-logging delete secret clickhouse-operator` if the apply created it; if a Secret existed before, save it first and re-apply that copy. + + ```bash + kubectl apply -f /tmp/clickhouse-operator.secret.yaml + ``` + + Expected: `secret/clickhouse-operator created`. Run the check again; the key is no longer listed. + +### Cause 4: plugins, feature gates, component YAML edits, OperatorHub installs, and customized keys are not in the sync set + +1. Read the sync scope from the ConfigMap that the sync workload mounts, on the cluster that runs the sync program: + + ```bash + kubectl get deploy -n cpaas-system | grep -E 'etcd-sync|etcd-mirror' + kubectl -n cpaas-system get deploy -o jsonpath='{.spec.template.spec.volumes}{"\n"}' + kubectl -n cpaas-system get configmap -o yaml + ``` + +2. Apply plugin installs, feature-gate changes, platform component YAML edits, and OperatorHub installs on the other global cluster as well. +3. Handle missing customized keys with the team that owns the customization; they are not synchronized automatically. +4. Before a planned switchover, take an etcd backup on both the active and the standby cluster. + +### Cause 5: which network the primary and standby global clusters need + +From a host outside each cluster, check that the VIP forwards the ports to its own control-plane nodes, once for the primary VIP and once for the standby VIP: + +```bash +for p in 443 6443 2379 11443; do nc -vz -w 3 $p; done +``` + +Also check the same ports from the standby network to the primary and back. + +## Diagnostic Steps + +Run the sync check on a control-plane node of the standby global cluster. The ClusterIP of the `etcd-sync-monitor` Service is wrapped in brackets when it is IPv6: + +```bash +mirror_svc=$(kubectl get svc -n cpaas-system etcd-sync-monitor -o jsonpath='{.spec.clusterIP}') +if [[ $mirror_svc =~ ^[0-9a-fA-F:]+$ ]]; then mirror_host="[$mirror_svc]"; else mirror_host="$mirror_svc"; fi +curl -g "http://${mirror_host}/check" +``` + +- `localMissedKeys [0]` and only platform surplus keys such as `etcd-sync-monitor` and `disabletokens`: Cause 1, normal. +- A few missed keys of platform objects: Cause 2. +- `clickhouse-operator` still missed after a restart: Cause 3. +- Missing plugins, feature gates, or customized keys such as `ait.alauda.io/notifications`: Cause 4. +- A question about network connectivity between the two clusters: Cause 5. diff --git a/docs/en/solutions/etcd_Synchronizer_plugin_install_stuck_or_etcd-sync_does_not_start_on_the_standby_global_cluster_on_ACP.md b/docs/en/solutions/etcd_Synchronizer_plugin_install_stuck_or_etcd-sync_does_not_start_on_the_standby_global_cluster_on_ACP.md new file mode 100644 index 000000000..6970559ac --- /dev/null +++ b/docs/en/solutions/etcd_Synchronizer_plugin_install_stuck_or_etcd-sync_does_not_start_on_the_standby_global_cluster_on_ACP.md @@ -0,0 +1,168 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.2" +--- + +# etcd Synchronizer plugin install stuck or etcd-sync does not start on the standby global cluster on ACP + +## Issue + +The etcd Synchronizer cluster plugin of a primary/standby (disaster recovery) global setup does not come up on the standby global cluster: + +- Installing or reinstalling the plugin (Administrator → Marketplace → Cluster Plugins → `global`) stays in the installing state, and `kubectl get ars -A` shows: + + ```text + cpaas-system etcd-sync ChartValuesRenderFailed failed to render values for chart ait/chart-etcd-sync-v4.2.1, err: run script error: exit status 2, stderr: base64: invalid input... + ``` + +- During a disaster-recovery upgrade (for example 4.0.9 to 4.2.4) the sync plugin is deployed on the standby global cluster and never becomes Ready. +- `etcd-sync` fails to deploy while the historical primary cluster still runs an `etcd-mirror` Deployment. +- The `ars` of the plugin on the standby times out calling the etcd sync API. +- The plugin has to be removed or installed again with `kubectl`. + +Related articles: "etcd-sync check reports localMissedKeys or localSurplusKeys, or what a primary/standby global pair replicates on ACP" covers a running synchronizer; "etcd-sync Pod stuck in ContainerCreating with failed to sync secret cache on the standby global cluster on ACP" covers `invalid padding on input`. + +## Root Cause + +### Cause 1: ChartValuesRenderFailed with base64: invalid input because the token was entered in plaintext + +The installation takes a token for the active global cluster as a parameter. The chart's values-rendering script expects that token base64-encoded and decodes it. A plaintext token makes the script fail with `base64: invalid input`, the AppRelease stays in `ChartValuesRenderFailed`, and the installation never reaches the point of creating Pods. Replacing the token with its base64-encoded form lets the installation complete. + +On plugin versions that read the token from the `etcd-sync-active-cluster-token` Secret instead of a parameter, the Secret is created from the plaintext token as the installation documentation describes. + +### Cause 2: Secret etcd-ca is missing in cpaas-system, the sync plugin does not start + +The sync plugin needs the Secret `etcd-ca` in `cpaas-system`. When it is missing on the primary and the standby (for example after a 4.0.9 to 4.2.4 disaster-recovery upgrade), the plugin deployed on the standby does not become Ready. Creating the Secret on both clusters from the etcd CA certificate and key, then redeploying the plugin, brings it up. + +### Cause 3: a leftover etcd-mirror Deployment on the historical primary blocks etcd-sync + +An `etcd-mirror` Deployment from an earlier synchronization setup is still running on the cluster that used to be the primary. It is not the current sync plugin, and while it exists, `etcd-sync` fails to deploy. Deleting the leftover Deployment clears the install. + +### Cause 4: the plugin was installed while the platform domain still resolved to the standby + +The standby cluster is installed while the platform domain temporarily points at the standby VIP, and the domain has to point back at the primary VIP before the etcd Synchronizer is installed. When the plugin is installed while the domain still resolves to the standby, the standby's `ars` times out calling the etcd sync API. After the domain resolves to the primary again and the plugin is redeployed, the timeout clears. + +### Cause 5: the plugin is installed and uninstalled through its ModuleInfo + +The etcd Synchronizer, like other cluster plugins, is represented by a ModuleInfo (`minfo`). Deleting that object uninstalls the plugin, and creating it again installs it. + +## Resolution + +### Cause 1: ChartValuesRenderFailed with base64: invalid input because the token was entered in plaintext + +1. Confirm the render failure: + + ```bash + kubectl get ars -n cpaas-system etcd-sync + kubectl describe ars -n cpaas-system etcd-sync | grep -iE 'reason|message|base64' + ``` + +2. Encode the token without a trailing newline: + + ```bash + echo -n '' | base64 -w0; echo + ``` + +3. Enter the encoded value as the token in the plugin's installation parameters and install again. +4. Check that the AppRelease leaves the failed state and the Pods run: + + ```bash + kubectl get ars -n cpaas-system etcd-sync + kubectl get po -n cpaas-system -l app=etcd-sync + ``` + +### Cause 2: Secret etcd-ca is missing in cpaas-system, the sync plugin does not start + +1. On the primary and on the standby, check the Secret and the CA files on a control-plane node: + + ```bash + kubectl get secret -n cpaas-system etcd-ca + sudo ls -l /etc/kubernetes/pki/etcd/ca.crt /etc/kubernetes/pki/etcd/ca.key + ``` + +2. On each cluster where it is missing, create it from that cluster's etcd CA. + + > **Warning:** The Secret stores the etcd CA private key in `cpaas-system`. Precondition: only when `etcd-ca` is missing and the plugin does not start; if the Secret exists, do not recreate it. Rollback: `kubectl delete secret -n cpaas-system etcd-ca` for a Secret this command just created; do not delete one that other components already use. + + ```bash + kubectl create secret tls etcd-ca \ + --cert=/etc/kubernetes/pki/etcd/ca.crt \ + --key=/etc/kubernetes/pki/etcd/ca.key \ + -n cpaas-system + ``` + +3. Redeploy the sync plugin after both clusters have the Secret. + +### Cause 3: a leftover etcd-mirror Deployment on the historical primary blocks etcd-sync + +1. On the historical primary cluster, not on the standby, find the leftover: + + ```bash + kubectl get deploy -A | grep etcd-mirror + ``` + +2. Save and delete it. + + > **Warning:** Deleting `etcd-mirror` stops that old sync process. Precondition: only the leftover Deployment on the historical primary, only while the `etcd-sync` install is failing. Do not delete an `etcd-sync` Deployment in this step. Rollback: `kubectl apply -f /tmp/etcd-mirror.yaml`. + + ```bash + kubectl -n get deploy etcd-mirror -o yaml > /tmp/etcd-mirror.yaml + kubectl -n delete deploy etcd-mirror + ``` + +3. Retry the etcd-sync install. + +### Cause 4: the plugin was installed while the platform domain still resolved to the standby + +1. Check where the platform domain resolves, from a standby node and from inside the cluster: + + ```bash + getent hosts + kubectl exec -n cpaas-system deployments/sentry -- nslookup + kubectl get ars -A | grep -i etcd + ``` + +2. Point the domain at the primary global cluster's VIP. +3. Redeploy the etcd Synchronizer on the standby global cluster, as the installation documentation describes, and check that the `ars` timeout is gone. + +### Cause 5: the plugin is installed and uninstalled through its ModuleInfo + +1. Find the object: + + ```bash + kubectl get minfo -A | grep -i etcd-sync + ``` + +2. Save it and delete it to uninstall. + + > **Warning:** Uninstalling the etcd Synchronizer stops disaster-recovery synchronization. Uninstalling it while the standby is missing data that the primary holds can make owner references resolve incorrectly and lead to workload-cluster node `Machine` objects being deleted. Precondition: only when the sync check reports no missed or surplus keys that are unexplained, and the name is the etcd-sync plugin. Rollback: `kubectl apply -f /tmp/etcd-sync-minfo.yaml`. + + ```bash + kubectl get minfo -o yaml > /tmp/etcd-sync-minfo.yaml + kubectl delete minfo + ``` + + Add `-n ` to both commands if the list printed a namespace. + +3. To install it again, apply a ModuleInfo for `etcd-sync` taken from the saved object (remove `metadata.resourceVersion`, `metadata.uid`, and `status` first), not a hand-written manifest. + +## Diagnostic Steps + +```bash +kubectl get ars -A | grep -i etcd +kubectl get pods -A | grep -E 'etcd-sync|etcd-mirror' +kubectl get secret -n cpaas-system etcd-ca +kubectl get minfo -A | grep -i etcd +kubectl logs -n cpaas-system --tail=200 +``` + +- `ChartValuesRenderFailed` with `base64: invalid input`: Cause 1; no Pods exist yet. +- Plugin Pods not Ready and `etcd-ca` reported `NotFound`: Cause 2. +- Install fails and the historical primary still lists an `etcd-mirror` Deployment: Cause 3. +- `ars` times out on the etcd sync API, and the platform domain resolved to the standby when the plugin was installed: Cause 4. +- Removing or reinstalling the plugin from `kubectl`: Cause 5. diff --git a/docs/en/solutions/etcd_and_kube-apiserver_do_not_start_on_a_master_after_runc_or_containerd_is_replaced_on_ACP.md b/docs/en/solutions/etcd_and_kube-apiserver_do_not_start_on_a_master_after_runc_or_containerd_is_replaced_on_ACP.md new file mode 100644 index 000000000..9c4a94ccf --- /dev/null +++ b/docs/en/solutions/etcd_and_kube-apiserver_do_not_start_on_a_master_after_runc_or_containerd_is_replaced_on_ACP.md @@ -0,0 +1,190 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.1" + - "4.2" + - "4.3" +--- + +# etcd and kube-apiserver do not start on a master after runc or containerd is replaced on ACP + +## Issue + +After a kernel upgrade, or after Docker (`docker-ce`) was installed on a master, the static Pods `etcd-` and `kube-apiserver-` in `kube-system` stay in `CrashLoopBackOff` or never start, and every Pod on that node fails to start. Typical signs: + +- kubelet repeats `Failed to delete cgroup paths`. +- `crictl ps -a | grep etcd` shows etcd `Exited` with `"reason": "StartError"` and `"exitCode": 128`, and `crictl logs` prints nothing because the container never started. +- containerd logs a `shim-run-v2` error, and etcd and kube-apiserver stay down even after `/usr/bin` was recopied from a healthy node. +- Copying `containerd` and `runc` into `/usr/bin` and restarting kubelet does not help while an older `runc` under `/usr/local/bin` is still the one found first on `PATH`. +- The command line stops working after Docker was installed, and after Docker is uninstalled the scheduler still does not start. +- calico-node restarts with a port conflict, and two calico containers cannot be deleted. + +Related articles: "Control-plane Pods do not start with mkdir /var/log/pods: file exists after containerd directories are lost on ACP" covers a master where kubelet and containerd run but no container exists. "Container runtime conflicts on ACP nodes: Docker, podman, docker-runc, and overwritten runtime binaries" covers the same overwrite on worker nodes. + +## Root Cause + +### Cause 1: /usr/local/bin/runc is an old build after a kernel upgrade + +After a kernel upgrade (for example on Kylin), etcd and kube-apiserver on the upgraded master enter `CrashLoopBackOff` and kubelet cannot create the sandbox. `which -a runc` lists `/usr/local/bin/runc` before `/usr/bin/runc`. The first binary is `runc 1.0.0-rc3`, a build shipped with the OS image, which does not match the new kernel's IPC namespace handling, so sandbox creation fails. The working binary is the platform's `runc` under `/usr/bin` (for example `1.1.12`). Recopying `/usr/bin` and restarting kubelet and containerd does not help until `/usr/local/bin/runc` is moved aside and the runtime is restarted. + +### Cause 2: Installing docker-ce replaced runc and containerd + +`yum install docker-ce` replaced the `runc` and `containerd` binaries. After a reboot no Pod starts, and kubelet repeats `Failed to delete cgroup paths` from `pod_container_manager_linux.go:210`. On a containerd cluster the overwrite can show up only after a reboot: `crictl inspect` of the etcd container shows `"state": "CONTAINER_EXITED"`, `"exitCode": 128`, `"reason": "StartError"`, and: + +```text +failed to create containerd task: failed to create shim task: OCI runtime create failed: container_linux.go:311: creating new parent process caused "container_linux.go:1572: running lstat on namespace path \"/proc//ns/ipc\" caused \"lstat /proc//ns/ipc: no such file or directory\"": unknown +``` + +`systemctl status docker` shows `docker.service` `active (running)` on the master. If several masters are affected (for example one failed and a second was rebooted), etcd loses quorum. Restoring an etcd backup does not help; restoring the overwritten binaries from a healthy node brings etcd back and the nodes to `Ready`. + +Restoring the binaries without removing Docker is enough for etcd and kube-apiserver, which use the host network, but a running `dockerd` on a platform node is not supported: it rewrites `docker0`, iptables, NAT, and `FORWARD` rules, which breaks Pod-network traffic on the node (for example Pods that cannot reach the `kubernetes` Service on `443`) while host-network static Pods keep working. + +### Cause 3: /usr/local/bin still has Docker runtime binaries + +Docker installed on the containerd node left `runc`, `containerd`, or `containerd-shim*` copies in `/usr/local/bin`, which are found first on `PATH`. containerd logs a `shim-run-v2` error, and etcd and kube-apiserver stay down after `/usr/bin` was recopied. After `systemctl stop kubelet` and `systemctl stop containerd`, many `/usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id ...` processes can remain. Moving the leftovers aside, copying the containerd binaries again, and restarting containerd and kubelet is the recovery; a reboot may also be needed. + +### Cause 4: After Docker is uninstalled the scheduler stays down until a reboot + +The cluster is abnormal, `kubectl` cannot run, and Docker is installed on a node. After the Docker package is uninstalled the scheduler still cannot start; rebooting that node brings the scheduler and the cluster back. A reboot alone is enough only when the Docker packages did not own the platform runtime files. + +## Resolution + +Before any step: if the API server still answers, cordon and drain the node (`kubectl drain --ignore-daemonsets --delete-emptydir-data`). When etcd or kube-apiserver on this master is down and `kubectl` does not answer, skip the drain and work from a console session. Static Pods are not evicted by a drain either way. + +### Cause 1: /usr/local/bin/runc is an old build after a kernel upgrade + +Apply only when etcd and kube-apiserver are in `CrashLoopBackOff` after a kernel upgrade, `/usr/local/bin/runc --version` prints `1.0.0-rc3` (or another build older than the healthy node's), and `/usr/bin/runc` matches the healthy node: + +```bash +which -a runc +/usr/local/bin/runc --version +/usr/bin/runc --version +``` + +> **Warning:** Moving `/usr/local/bin/runc` changes which runc creates the next sandbox. New containers wait until containerd is restarted. Rollback: `sudo mv /usr/local/bin/runc.old /usr/local/bin/runc` and restart containerd again. Keep the old file. + +```bash +sudo mv /usr/local/bin/runc /usr/local/bin/runc.old +hash -r +command -v runc +runc --version +sudo systemctl restart containerd +sudo systemctl restart kubelet +``` + +`runc --version` must now match `/usr/bin/runc`. etcd and kube-apiserver should leave `CrashLoopBackOff` without a manifest edit and without deleting `/var/lib/etcd`. + +### Cause 2: Installing docker-ce replaced runc and containerd + +Compare the node with a healthy master of the same cluster: + +```bash +which -a runc containerd +runc --version +containerd --version +ls -l "$(command -v runc)" "$(command -v containerd)" +rpm -q docker-ce || true +rpm -qf "$(command -v runc)" "$(command -v containerd)" +``` + +If `rpm -qf` shows that a Docker package (for example `docker-ce` or `containerd.io`) owns `runc` or `containerd` and you also intend to remove Docker, remove it before the copy: removing the package deletes the files it owns, including freshly restored ones. + +Copy `runc` and `containerd` from the healthy node to `/var/tmp/runc.from-healthy` and `/var/tmp/containerd.from-healthy` on this node. If the healthy node's `ls -l` shows other differences, also copy `ctd-decoder`, `containerd-shim-runc-v2`, `ctr`, `containerd-shim`, `crictl`, `containerd-shim-runc-v1`, `critest`, `containerd-stress`, `kubelet`, and `kubectl` into `/usr/bin`. + +> **Warning:** Replacing `runc` and `containerd` and restarting containerd stops new container creation on this node until the units are back. Running containers survive the stop only when the containerd unit has `KillMode=process` (check `systemctl show containerd -p KillMode`); with `control-group` or `mixed`, stopping containerd stops every container on the node, including static Pods that are still up. Apply only when the cgroup-path log or the `StartError` is confirmed and the healthy-node binaries differ. Stop kubelet and containerd before the copy; overwriting a running `containerd` fails with `Text file busy`. Match owner and mode to the healthy node. Rollback: stop kubelet and containerd, copy `/var/tmp/runc.before-restore` and `/var/tmp/containerd.before-restore` back, and start containerd and kubelet. This does not touch etcd data. + +```bash +RUNC="$(command -v runc)"; CONTAINERD="$(command -v containerd)" +sudo cp -a "$RUNC" /var/tmp/runc.before-restore +sudo cp -a "$CONTAINERD" /var/tmp/containerd.before-restore +sudo systemctl stop kubelet containerd +sudo cp -a /var/tmp/runc.from-healthy "$RUNC" +sudo cp -a /var/tmp/containerd.from-healthy "$CONTAINERD" +sudo systemctl start containerd +sudo systemctl start kubelet +``` + +If `which` on the healthy node and on this node disagree, restore onto the healthy node's path and move any extra copy under `/usr/local/bin` aside (Cause 3). Pods, including the static Pods, should start once kubelet can create cgroups again. If Pods on this node then cannot reach the `kubernetes` Service, stop and remove Docker, or take the node out of the cluster if Docker must stay. + +### Cause 3: /usr/local/bin still has Docker runtime binaries + +Apply only when the runtime is containerd, Docker was installed on the node, containerd logs `shim-run-v2`, and `/usr/local/bin` holds `runc`, `containerd`, or `containerd-shim` files that a healthy node does not have: + +```bash +rpm -qa | grep docker || true +ls -l /usr/local/bin | grep -E 'runc|containerd|shim' || true +which -a runc containerd +ps -ef | grep '[c]ontainerd-shim-runc-v2' | wc -l +``` + +> **Warning:** Moving the `/usr/local/bin` binaries changes which shim the next container uses. Move them only when they are confirmed Docker leftovers and `/usr/bin` already matches a healthy node. New containers wait until containerd and kubelet are restarted. Rollback: `sudo mv /var/tmp/usr-local-bin-docker/ /usr/local/bin/` for each moved file. + +```bash +sudo mkdir -p /var/tmp/usr-local-bin-docker +for f in runc containerd containerd-shim containerd-shim-runc-v1 containerd-shim-runc-v2; do + if [ -e "/usr/local/bin/$f" ]; then sudo mv "/usr/local/bin/$f" /var/tmp/usr-local-bin-docker/; fi +done +hash -r +sudo systemctl restart containerd +sudo systemctl restart kubelet +``` + +Copy the healthy `/usr/bin` set again if `which` still disagrees with the healthy node. Leave `/usr/local/bin` without a `runc` unless a unit, script, or containerd setting calls that path by full name; in that case copy `/usr/bin/runc` there instead. etcd and kube-apiserver should start without deleting the WAL. + +If calico-node then restarts with a port conflict and `crictl ps -a | grep calico` shows two containers that cannot be deleted, stop kubelet, reboot the host, delete the stale calico container, and start kubelet. + +> **Warning:** Stopping kubelet and rebooting stops Pods on this node until the host is back and kubelet is started. Only when the duplicate calico containers remain after the binary restore and calico-node is in a port conflict. Rollback of the kubelet stop before the reboot: `sudo systemctl start kubelet`. + +```bash +sudo systemctl stop kubelet +sudo shutdown -r now +# after the host is back +sudo crictl ps -a | grep calico +sudo crictl rm +sudo systemctl start kubelet +``` + +### Cause 4: After Docker is uninstalled the scheduler stays down until a reboot + +List the Docker packages and check whether they own the platform runtime files: + +```bash +rpm -qa | grep docker || true +rpm -qf /usr/bin/containerd /usr/bin/runc +``` + +If a Docker package (for example `containerd.io` or `docker-ce`) owns them, removing it deletes those files; restore them from a healthy node (Cause 2) before the reboot. + +> **Warning:** Removing Docker and rebooting stops containers on this node until the host is back. Only when Docker was installed on a containerd node and the cluster command line is already dead. Reinstalling Docker is not a rollback, because Docker caused the fault. If etcd is still down after the reboot and `/usr/local/bin` holds runtime binaries, use Cause 3. + +Remove the listed Docker packages with the package manager, then reboot: + +```bash +sudo shutdown -r now +``` + +After the reboot, `kubectl get pods -n kube-system -l component=kube-scheduler` should show the scheduler Running. + +## Diagnostic Steps + +```bash +which -a runc containerd +runc --version +containerd --version +sudo journalctl -u kubelet --since "1 hour ago" | grep -F "Failed to delete cgroup paths" || true +systemctl is-active docker 2>/dev/null || true +sudo crictl ps -a | grep etcd +sudo crictl inspect | grep -E '"reason"|"exitCode"|"message"' +kubectl -n kube-system get pods -l 'component in (etcd,kube-apiserver)' -o wide +``` + +- `/usr/local/bin/runc` is `1.0.0-rc3` (or older than the healthy node's) after a kernel upgrade, static Pods in `CrashLoopBackOff`: Cause 1. +- `Failed to delete cgroup paths`, or etcd `StartError` / `lstat /proc//ns/ipc: no such file or directory` with `docker.service` running on a containerd node: Cause 2. `rpm -q docker-ce` returning a package supports it. +- `shim-run-v2` and Docker leftovers in `/usr/local/bin`: Cause 3. +- Docker installed, command line dead, scheduler down after the uninstall: Cause 4. +- `walpb: crc mismatch` with binaries that match a healthy node: a corrupt etcd member, not a binary restore. + +A node that never had Docker installed and whose `which -a runc` lists only `/usr/bin/runc` at the healthy version is not covered here. diff --git a/docs/en/solutions/etcd_backup_dates_missing_in_the_console_backup_tab_gone_duplicate_copies_or_retention_to_change_on_ACP.md b/docs/en/solutions/etcd_backup_dates_missing_in_the_console_backup_tab_gone_duplicate_copies_or_retention_to_change_on_ACP.md new file mode 100644 index 000000000..ca2dd9371 --- /dev/null +++ b/docs/en/solutions/etcd_backup_dates_missing_in_the_console_backup_tab_gone_duplicate_copies_or_retention_to_change_on_ACP.md @@ -0,0 +1,172 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "4.0" + - "4.1" + - "4.2" +--- + +# etcd backup dates missing in the console, backup tab gone, duplicate copies, or retention to change on ACP + +## Issue + +- The etcd backup task looks like it skipped dates: every control-plane node has a schedule such as `0 18 * * *`, and some dates are absent from the console. +- The Administrator cluster-management page has no backup-management tab, so the backup policy and a restore cannot be started from that screen. +- Every day a new backup is written to `/cpaas-backup`, and it is not clear how long snapshots are kept or how to shorten the retention. A directory `backup_` sits next to `cpaas-backup` on `/`. +- The backup retention on a standby global cluster has to change without switching the platform domain to it. +- Each etcd backup run writes two copies. + +Related articles: "No default etcd backup on ACP: Cluster Enhancer not installed or etcd-backup-default not created"; "Scheduled etcd backups stop after an upgrade because the etcd-backup-default AdvancedCronJob stops triggering on ACP". + +## Root Cause + +### Cause 1: repeated label updates on one EtcdBackupRecord collapse the console list, the files are on disk + +The scheduled backups run (the `etcd-backup-default` job logs show the execution), and the `snapshot-etcd-*.db` files for the missing dates exist in the backup directory (for example `/mnt/cpaas-backup/` or `/cpaas/backup`). The `EtcdBackupRecord` objects and the audit log show the same resource being label-updated repeatedly, so the console lists one record of that name. The snapshots are not missing, and no backup has to be regenerated. + +### Cause 2: on 4.1 the backup-management tab was removed, the backup configuration still exists + +On 4.1 the backup-management page of cluster management has been removed. The backup itself keeps running: `cluster-enhancer-apiserver` Pods are `1/1 Running`, and the `EtcdBackupConfiguration` `etcd-backup-default` exists, is not paused, and keeps creating records. Records are viewed from the etcd backup monitoring dashboard (Operations Center → Monitoring → Dashboards → Switch → Cluster → etcd backup), and policy changes are made on the `EtcdBackupConfiguration` object. A restore screen has never been provided in the console; restoring etcd is not started from the UI. + +### Cause 3: retention is spec.localStorage.ttl, and backup_ is an upgrade copy + +Daily backups to `/cpaas-backup` are the normal behavior. Snapshot retention is `spec.localStorage.ttl` on the `EtcdBackupConfiguration` named `etcd-backup-default` (`apiVersion: enhancement.cluster.alauda.io/v1`), in seconds, for example `ttl: "7776000"` (90 days) with `localStorage.path: /cpaas-backup` and `schedule: 0 0 * * *`. The same object also carries the annotations `life` and `path`; retention is changed through `spec.localStorage.ttl`, not through the `life` annotation. The change applies without a component restart, but snapshots already past the new retention are removed on the next scheduled backup, not at the moment of the edit. In the console, the object is found under Resource Management; searching Custom Resource Definitions for `EtcdBackupConfiguration` returns nothing. + +`cpaas-backup` on `/` is the etcd backup data directory and has to be kept. `backup_` (for example `backup_` plus a date stamp) is written by a platform upgrade and exists only on clusters that were upgraded. It holds an `etcd` and a `kubernetes` subdirectory (with `admin.conf`, `kubeadm-config.yaml`, `encryption-provider.conf`, `kubelet.conf`, and the `manifests`, `pki`, `audit`, and `patches` directories). That upgrade copy can be deleted. + +### Cause 4: on a standby global cluster, retention is the LIFE variable of the etcd backup CronJob + +When the platform domain is not switched to the standby global cluster, its console backup page is not the path. The retention period there is the `LIFE` environment variable, in seconds, of the etcd backup CronJob (for example container `etcd-backup` running `/backup-local.sh` with `NAME=etcd-backup-default` and `LIFE="1296000"`, 15 days). + +### Cause 5: on 4.2.4 the etcd-backup-upgrade-k8s bcj writes a second copy + +Each run produces two copies because a `bcj` named `etcd-backup-upgrade-k8s` sits beside the regular backup configuration and fires too. Removing that bcj's annotations, labels, and owner references stops the second copy. Why both objects fire is not established, so treat this as a workaround. + +## Resolution + +### Cause 1: repeated label updates on one EtcdBackupRecord collapse the console list, the files are on disk + +Look on the control-plane nodes before scheduling another backup: + +```bash +kubectl get cronjob -A | grep etcd-backup +kubectl get etcdbackuprecord -A +kubectl get etcdbackupconfiguration -A -o yaml | grep -A3 localStorage +sudo ls -l /mnt/cpaas-backup/snapshot-etcd-*.db /cpaas/backup/snapshot-etcd-*.db /cpaas-backup/snapshot-etcd-*.db 2>/dev/null +``` + +Check the path printed on the configuration object as well. If the dated `snapshot-etcd-*.db` files exist, the single console row is a display effect of repeated label updates on one `EtcdBackupRecord`. Leave the files. If no file exists on any path, this cause does not apply. + +### Cause 2: on 4.1 the backup-management tab was removed, the backup configuration still exists + +1. Confirm that the backup is still in place: + + ```bash + kubectl get pods -A | grep cluster-enhancer-apiserver + kubectl get etcdbackupconfiguration -A -o yaml + ``` + +2. Read records from the etcd backup monitoring dashboard. +3. To change the policy, save the object, edit the saved copy, and apply it. + + > **Warning:** Applying a changed `EtcdBackupConfiguration` changes the schedule, path, or retention of the platform etcd backup; a bad path or a much smaller `ttl` can remove snapshots you still need on the next run. Precondition: only when you intend that policy change, and only after the backup copy exists. Rollback: apply the saved copy again before the next scheduled run. + + ```bash + kubectl get etcdbackupconfiguration -A -o yaml > /tmp/etcd-backup-default.yaml + cp /tmp/etcd-backup-default.yaml /tmp/etcd-backup-default.new.yaml + vi /tmp/etcd-backup-default.new.yaml + kubectl apply -f /tmp/etcd-backup-default.new.yaml + ``` + +### Cause 3: retention is spec.localStorage.ttl, and backup_ is an upgrade copy + +1. Read the current retention, path, and schedule. An empty NAMESPACE column means the object is cluster-scoped; drop `-n` then: + + ```bash + kubectl get etcdbackupconfiguration -A + kubectl get etcdbackupconfiguration etcd-backup-default -n -o yaml > /tmp/etcd-backup-default.yaml + grep -n -A3 'localStorage' /tmp/etcd-backup-default.yaml + ``` + +2. Set the retention you want, in seconds, as a quoted string like the existing value (`604800` is 7 days, `2592000` is 30 days). + + > **Warning:** Lowering `ttl` makes the next scheduled backup delete every snapshot older than the new value. Precondition: only when that retention is intended, after saving the object. Rollback: `kubectl apply -f /tmp/etcd-backup-default.yaml` before the next scheduled run; snapshots already removed by a run do not come back. Leave the `life` annotation alone; it is not the setting. + + ```bash + kubectl patch etcdbackupconfiguration etcd-backup-default -n --type=merge \ + -p '{"spec":{"localStorage":{"ttl":"604800"}}}' + ``` + + Old snapshots are removed by the next scheduled backup. + +3. To free the upgrade copy on `/`, check what it is first: + + ```bash + sudo ls -ld /cpaas-backup /backup_* 2>/dev/null || true + sudo ls /backup_ + ``` + + > **Warning:** Deleting `backup_` removes the configuration and etcd copy that the upgrade took, and cannot be undone. Precondition: only that directory, after `ls` shows it is the upgrade backup (`etcd` and `kubernetes` subdirectories) and not `cpaas-backup`, and only when the upgrade has completed. Rollback: none, unless you archived the directory elsewhere first. + + ```bash + sudo rm -rf /backup_ + ``` + +### Cause 4: on a standby global cluster, retention is the LIFE variable of the etcd backup CronJob + +1. Find the CronJob and save it: + + ```bash + kubectl get cronjob -A | grep -i etcd-backup + kubectl -n get cronjob -o yaml > /tmp/etcd-backup-cronjob.yaml + grep -n -A1 'name: LIFE' /tmp/etcd-backup-cronjob.yaml + ``` + +2. Set `LIFE` to the retention you want, in seconds. + + > **Warning:** A smaller `LIFE` removes older standby snapshots on the next run. Precondition: only on the CronJob whose container is `etcd-backup`, and only with the value you intend. Rollback: `kubectl apply -f /tmp/etcd-backup-cronjob.yaml`. + + ```bash + kubectl -n set env cronjob/ -c etcd-backup LIFE=1296000 + ``` + +### Cause 5: on 4.2.4 the etcd-backup-upgrade-k8s bcj writes a second copy + +1. Confirm the extra object: + + ```bash + kubectl get bcj -A | grep etcd-backup + ``` + +2. Save it, then clear its annotations, labels, and owner references. + + > **Warning:** This detaches `etcd-backup-upgrade-k8s` from its owner; the duplicate stops only as long as no controller writes those fields back. It does not delete snapshot files already written. Precondition: only when `etcd-backup-upgrade-k8s` exists and each run writes two copies. Rollback: `kubectl apply -f /tmp/etcd-backup-upgrade-k8s.yaml`. + + ```bash + kubectl -n get bcj etcd-backup-upgrade-k8s -o yaml > /tmp/etcd-backup-upgrade-k8s.yaml + kubectl -n annotate bcj etcd-backup-upgrade-k8s --all- + kubectl -n label bcj etcd-backup-upgrade-k8s --all- + kubectl -n patch bcj etcd-backup-upgrade-k8s --type=json -p='[{"op":"remove","path":"/metadata/ownerReferences"}]' + ``` + + If `ownerReferences` is already absent, skip the patch. The next backup run should write one copy. + +## Diagnostic Steps + +```bash +kubectl get etcdbackupconfiguration -A +kubectl get etcdbackuprecord -A +kubectl get cronjob -A | grep etcd-backup +kubectl get bcj -A | grep etcd-backup +``` + +- Dated snapshot files exist on disk while the console shows one `EtcdBackupRecord`: Cause 1. +- 4.1, no backup-management tab, `etcd-backup-default` present: Cause 2. +- A retention change, or `backup_` next to `cpaas-backup`: Cause 3. Expected: `spec.localStorage.ttl` in the YAML, and `etcd` and `kubernetes` subdirectories in `backup_`. +- Retention on a standby global cluster without a domain switch: Cause 4. +- Two copies per run and `etcd-backup-upgrade-k8s` in the `bcj` list: Cause 5. +- No `EtcdBackupConfiguration` at all: see the related article on the missing default backup. diff --git a/docs/en/solutions/etcd_quota_stays_full_from_a_large_live_key_set_Pod_bursts_or_a_database_over_2_GB_on_ACP.md b/docs/en/solutions/etcd_quota_stays_full_from_a_large_live_key_set_Pod_bursts_or_a_database_over_2_GB_on_ACP.md new file mode 100644 index 000000000..ed5b2697f --- /dev/null +++ b/docs/en/solutions/etcd_quota_stays_full_from_a_large_live_key_set_Pod_bursts_or_a_database_over_2_GB_on_ACP.md @@ -0,0 +1,156 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "2.6" + - "3.16" + - "4.1" + - "4.2" + - "4.3" +--- + +# etcd quota stays full from a large live key set, Pod bursts, or a database over 2 GB on ACP + +## Issue + +- etcd DB SIZE stays above about 1 GB after compaction, close to the default 2 GiB quota, and the key count is large (for example tens of thousands of ReplicaSets). +- A workload creates and deletes many Pods in bursts; during a load test etcd returns `Error from server : etcdserver: mvcc:database space exceeded` after about 1,000 Pods, every Pod create and delete fails, and deleted Pods stay in `Terminated`. +- kube-apiserver (for example on a standby `global` cluster) restarts again and again, its log shows `etcdserver: leader changed`, and every member's database is over 2 GB. + +On ACP, etcd runs as static Pods `etcd-` in `kube-system` (label `component=etcd`). The manifest is `/etc/kubernetes/manifests/etcd.yaml` and the data directory is `/var/lib/etcd`. + +Related articles: "etcd reports database space exceeded, or compaction frees only tens of megabytes on ACP" covers the snapshot, compaction, defrag, and `alarm disarm` steps used below. + +## Root Cause + +### Cause 1: DB SIZE stays above 1 GB after compaction because the live key set is large + +`endpoint status` shows DB SIZE around 1.9 GB on every member, and a prefix count shows a large live set (for example 52126 replicasets, 14579 services, 12352 secrets, 9761 pods, 7522 configmaps, 7262 deployments). Compaction drops the file to about 1.2 GB, which is still a capacity risk next to the default 2 GiB quota. A ten-minute watch shows `/registry/minions/` and `/registry/leases/` as the most frequently changing keys, so no single client loop is the source. + +ReplicaSets often dominate: about 5000 Deployments with 5 to 10 kept revisions each produce tens of thousands of ReplicaSets, because old ReplicaSets are kept up to the Deployment's `spec.revisionHistoryLimit` (Kubernetes default 10). Other large prefixes can be `rolebindings`, `aquasecurity.github.io` objects, `cert-manager.io`, and `events`. + +kube-apiserver compacts etcd about every 5 minutes. If kube-apiserver has been restarting continuously, that compaction does not run and revisions accumulate. + +Once compaction has done what it can, raise `quota-backend-bytes` to 8 GiB (`8589934592`) and keep that value permanently. + +### Cause 2: Pod create and delete bursts fill the quota + +A workload starts many Pods per task and deletes them when the task ends. On a v3.16.2 cluster (etcd 3.5.15), `endpoint status` can show DB SIZE 4.6 GB and `du -sh /var/lib/etcd/` 4.8G while a key count under `/registry` finds only about 2700 keys in total. A file that large over so few live keys is revision history from the Pod churn, not live data. The temporary relief is `--auto-compaction-retention=2h` together with `--quota-backend-bytes=8589934592`, followed by compaction and defrag; restore the previous parameters afterwards. + +### Cause 3: kube-apiserver restarts on etcdserver: leader changed while every member's database is over 2 GB + +kube-apiserver restarts frequently and its log before each restart shows etcd leader changes. The etcd log repeats `"msg":"apply request took too long"` with `"expected-duration":"100ms"`; read-only range requests on `/registry/events/` (`limit:500`) take 100 to 236 ms, some requests take 1 to 3 seconds (`"took":"3.427755923s"`) and end with `"error":"etcdserver: leader changed"`, and `/readyz` fails with `[-]linearizable_read failed: etcdserver: leader changed` and `"status-code":503`. The three members' databases are 2.3G, 2.4G, and 2.5G. After an etcd compaction, kube-apiserver stops restarting. + +## Resolution + +Run the diagnostic commands first and apply only the cause whose signature you confirmed. Take a snapshot from a healthy member before any change, as described in the related article. The helper below runs `etcdctl` inside an etcd Pod with the ACP certificate paths: + +```bash +etcdctl_on() { kubectl -n kube-system exec "etcd-$1" -- etcdctl \ + --endpoints=https://127.0.0.1:2379 \ + --cacert=/etc/kubernetes/pki/etcd/ca.crt \ + --cert=/etc/kubernetes/pki/etcd/server.crt \ + --key=/etc/kubernetes/pki/etcd/server.key "${@:2}"; } +etcdctl_on snapshot save /var/lib/etcd/etcd-before-change.db +``` + +Copy the snapshot from `/var/lib/etcd` on that node to `/var/tmp` before you continue. + +### Cause 1: DB SIZE stays above 1 GB after compaction because the live key set is large + +Compact first (compact to the current revision, then defrag each member, leader last; no `alarm disarm` unless `NOSPACE` is set). Confirm kube-apiserver is not crash-looping, otherwise the periodic compaction stalls again: + +```bash +kubectl -n kube-system get pods -l component=kube-apiserver -o wide +``` + +Then raise the quota only when DB SIZE is still above about 1 GB after a successful compaction, the prefix count shows a large live set, and the filesystem that holds `/var/lib/etcd` has room for an 8 GiB file. Set it where etcd's arguments come from on this cluster. + +When the control plane is generated from a `kubeadmConfigSpec`, set the value there, as a quoted string: + +```yaml +kubeadmConfigSpec: + clusterConfiguration: + etcd: + local: + extraArgs: + quota-backend-bytes: "8589934592" +``` + +When kubelet runs etcd from `/etc/kubernetes/manifests/etcd.yaml`, add the flag next to the other etcd arguments on each master, one master at a time. If both the manifest and `kubeadmConfigSpec` exist, set the same value in both so they do not disagree. + +```yaml +# in the etcd container command list of /etc/kubernetes/manifests/etcd.yaml +- --quota-backend-bytes=8589934592 +``` + +> **Warning:** Raising the quota does not shrink etcd; it lets the member grow up to 8 GiB. Saving `etcd.yaml`, or applying the kubeadm extra argument, restarts that etcd member. One master at a time, after compaction, in a maintenance window, and only with free space for an 8 GiB `snap/db`. Rollback: set the previous value again only after DB SIZE is under that value. Without the flag, etcd uses 2147483648 (2 GiB); putting that back while the file is larger turns `NOSPACE` on again. + +After the Pod is recreated, its arguments must contain `--quota-backend-bytes=8589934592`: + +```bash +kubectl -n kube-system get pod etcd- -o yaml | grep quota-backend-bytes +``` + +To reduce the number of ReplicaSets, lower `spec.revisionHistoryLimit` on Deployments that do not need many rollback revisions. + +### Cause 2: Pod create and delete bursts fill the quota + +Raise the quota as in Cause 1 so that writes can resume, and add `--auto-compaction-retention=2h` temporarily. + +> **Warning:** Saving `etcd.yaml` restarts this member. One master at a time, after a snapshot, and only while `mvcc: database space exceeded` is confirmed and Pod churn is the source. A `2h` retention discards revisions faster than before, so clients with old resourceVersions see compaction errors sooner. Rollback: remove `--auto-compaction-retention=2h` once DB SIZE is back under the quota you will keep, and save again. + +```yaml +# temporary, next to the other etcd arguments in /etc/kubernetes/manifests/etcd.yaml +- --auto-compaction-retention=2h +- --quota-backend-bytes=8589934592 +``` + +When the file is under the new quota, run `alarm disarm`, then compact and defrag each member. Restore the previous parameters when DB SIZE allows the previous quota. + +### Cause 3: kube-apiserver restarts on etcdserver: leader changed while every member's database is over 2 GB + +Apply only when `endpoint status` shows every member above about 2 GB, kube-apiserver's pre-restart log shows `etcdserver: leader changed`, and the etcd log has `apply request took too long` on large range reads: + +```bash +kubectl -n kube-system logs etcd- --tail=500 | grep -E 'changed leader|leader changed|linearizable_read' || true +``` + +Take the snapshot, compact to the current revision (no `alarm disarm` unless `NOSPACE` is set), and defrag each member, leader last. kube-apiserver stops restarting after the compaction. Leader changes while the database is small are not fixed by compaction; check etcd disk latency and packet loss between the masters instead. + +## Diagnostic Steps + +```bash +kubectl -n kube-system get pods -l component=etcd -o wide +etcdctl_on endpoint status -w table +etcdctl_on alarm list +``` + +- A large live prefix count, usually `replicasets`: Cause 1. +- A load test that creates and deletes Pods until `database space exceeded`, with a multi-gigabyte file over a few thousand keys: Cause 2. +- Every member above 2 GB while kube-apiserver restarts on `leader changed`: Cause 3. +- `NOSPACE` without one of these patterns, or a DB SIZE that barely moved after compaction: see the related article. + +Count keys by prefix: + +```bash +etcdctl_on get / --prefix --keys-only \ + | awk -F '/' '{ h[$3]++ } END {for (k in h) print h[k], k}' | sort -nr | head -30 +``` + +Watch which keys change for ten minutes, to rule out a single client writing in a loop: + +```bash +kubectl -n kube-system exec etcd- -- sh -c \ + 'export ETCDCTL_API=3; timeout 600 etcdctl \ + --endpoints=https://127.0.0.1:2379 \ + --cacert=/etc/kubernetes/pki/etcd/ca.crt \ + --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \ + --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \ + watch --prefix / -w fields' > /tmp/etcd-watch.txt +grep -w Key /tmp/etcd-watch.txt | awk '{print $NF}' | tr -d '"' | sort | uniq -c | sort -nr | head -20 +``` + +Lease and node-status keys at the top (for example about 300 changes per lease and 120 per `/registry/minions/` in ten minutes) are normal heartbeat traffic. diff --git a/docs/en/solutions/etcd_reports_database_space_exceeded_or_compaction_frees_only_tens_of_megabytes_on_ACP.md b/docs/en/solutions/etcd_reports_database_space_exceeded_or_compaction_frees_only_tens_of_megabytes_on_ACP.md new file mode 100644 index 000000000..090134142 --- /dev/null +++ b/docs/en/solutions/etcd_reports_database_space_exceeded_or_compaction_frees_only_tens_of_megabytes_on_ACP.md @@ -0,0 +1,146 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "2.6" + - "3.16" + - "4.1" + - "4.2" + - "4.3" +--- + +# etcd reports database space exceeded, or compaction frees only tens of megabytes on ACP + +## Issue + +The cluster reports `etcdserver: no space`, `Error from server : etcdserver: mvcc:database space exceeded`, or `etcdserver: mvcc: database space exceeded`. Pods cannot be created or deleted and stay in `Terminated`; `kubectl` writes fail. The platform raises an etcd db size alarm, or an inspection reports the etcd database over its threshold (for example about 800 MB, 1 GB, 1.4 GB, or 1.5 GB). Sometimes compaction has already been run and the file on disk barely shrinks. + +On ACP, etcd runs as static Pods `etcd-` in `kube-system` (label `component=etcd`). The manifest is `/etc/kubernetes/manifests/etcd.yaml` and the data directory is `/var/lib/etcd`. + +Related articles: "etcd quota stays full from a large live key set, Pod bursts, or a database over 2 GB on ACP" covers a database that stays large after compaction. + +## Root Cause + +### Cause 1: etcdserver: mvcc: database space exceeded + +The database file reached the backend quota and etcd raised the `NOSPACE` alarm. `etcdctl alarm list` shows `NOSPACE`, and in `endpoint status -w table` the ERRORS column reads `memberID: alarm:NOSPACE` on every member, even when only one member's file is large (for example 2.2 GB on a follower and 547 MB on the others). While the alarm is set, etcd rejects writes, so object deletes fail as well as creates. + +The repair is `etcdctl compact`, then `defrag` on each member, then `alarm disarm`. Compaction drops historical revisions, defrag returns the freed pages to the filesystem, and disarm clears the alarm once the file is under the quota. After this sequence the members can drop to around 400 MB with an empty ERRORS column. + +### Cause 2: Compaction frees only tens of megabytes and snap/db stays about 1.4G + +An inspection reports the etcd database usage (`ETCD数据库使用量`) above the warning threshold, a key count does not show one runaway prefix, and compaction frees only about 50 MB. On disk, `member` is about 1.7G, `snap` about 1.4G, `wal` about 367M, and `snap/db` about 1.4G. Endpoint status is healthy, DB SIZE matches that file, and no leader change is in progress. + +The inspection series is `cluster.kube.etcd.db.size.mb.per.instance`, query `etcd_mvcc_db_total_size_in_bytes{namespace="kube-system"} / 1024 / 1024`, from inspection item `cluster-etcd-database-usage` (`inspectionitems.ait.alauda.io`). It measures the on-disk MVCC file, including pages that compaction already marked free. Compaction alone does not shrink the file; defrag rewrites `snap/db` without the free pages. + +## Resolution + +Run the diagnostic commands first and apply only the cause whose signature you confirmed. Define a helper for `etcdctl` inside an etcd Pod (certificate paths as on ACP; check them first with `ls /etc/kubernetes/pki/etcd/ca.crt /etc/kubernetes/pki/etcd/server.crt /etc/kubernetes/pki/etcd/server.key` on a master, and do not use the commands as written if the paths differ): + +```bash +etcdctl_on() { kubectl -n kube-system exec "etcd-$1" -- etcdctl \ + --endpoints=https://127.0.0.1:2379 \ + --cacert=/etc/kubernetes/pki/etcd/ca.crt \ + --cert=/etc/kubernetes/pki/etcd/server.crt \ + --key=/etc/kubernetes/pki/etcd/server.key "${@:2}"; } +``` + +### Take a snapshot first + +Take a snapshot from a healthy member before any compaction or defrag. This is also the command-line form of a manual etcd backup. Find the `hostPath` whose `volumeMounts.mountPath` is `/var/lib/etcd` in `kubectl -n kube-system get pod etcd- -o yaml`; `snapshot save` writes inside the container, and the file lands on that hostPath on the node. The etcd image usually has no `tar`, so `kubectl cp` cannot copy the file out. Leave enough free space for another full copy of the database. + +```bash +etcdctl_on snapshot save /var/lib/etcd/etcd-before-compact.db +``` + +On that node, move the file out of the data directory: + +> **Warning:** The `rm -f` deletes only `/var/lib/etcd/etcd-before-compact.db` (or the same file name on the hostPath), and only after `ls` shows a non-empty copy under `/var/tmp`. Never delete anything inside `member/`. If the copy is missing, skip the `rm` and copy again. + +```bash +sudo cp /var/lib/etcd/etcd-before-compact.db /var/tmp/etcd-before-compact.db +sudo ls -l /var/tmp/etcd-before-compact.db +sudo rm -f /var/lib/etcd/etcd-before-compact.db +``` + +When the etcd Pod cannot be exec'd, run `etcdctl` on the master itself with `ETCDCTL_API=3`, the member endpoints on port 2379, and the certificates under `/etc/kubernetes/pki/etcd/` (`peer.crt`/`peer.key` or `healthcheck-client.crt`/`healthcheck-client.key`). + +### Cause 1: etcdserver: mvcc: database space exceeded + +Apply only when `alarm list` prints `NOSPACE` or the member log contains `database space exceeded`, and the snapshot exists. + +Read the current revision. The table form does not show it on every `etcdctl` build; the JSON form prints it in the response header: + +```bash +etcdctl_on endpoint status -w table +etcdctl_on endpoint status --write-out=json | grep -o '"revision":[0-9]*' +``` + +> **Warning:** `etcdctl compact ` drops every revision older than that number; clients that still hold those resourceVersions get a compaction error. Compaction is cluster-wide and cannot be undone. Run it once, with the current revision, after the snapshot exists. A wrong revision is not a reason to wipe `/var/lib/etcd`. + +```bash +etcdctl_on compact +``` + +Defragment one member at a time, the current leader (`IS LEADER` true in `endpoint status`) last, and wait until `endpoint health` of that member is true before the next. On a file of several gigabytes add `--command-timeout=300s`; on a file of about 1 GB, `--command-timeout=120s` is enough. Each member returns `Finished defragmenting etcd member[...]`. + +> **Warning:** `defrag` blocks the member while it rewrites `snap/db`; expect an API latency spike and a possible leader change. Maintenance window, snapshot saved, compact already run, one member at a time. If a member does not come back, stop. Defrag has no data undo. A snapshot restore is a rollback only for the whole cluster (every member restored from the same file after quorum is lost); one member cannot be restored from it into a running cluster. + +```bash +etcdctl_on defrag --command-timeout=300s +``` + +> **Warning:** `alarm disarm` lifts the read-only `NOSPACE` condition for the whole cluster. Run it only when `endpoint status` DB SIZE is below the quota after compact and defrag; disarming while the file is still over quota lets the alarm return. There is no separate re-arm step. + +```bash +etcdctl_on alarm disarm +etcdctl_on alarm list +``` + +`alarm list` should be empty and the console message should clear; writes, including Pod deletes, work again. Run `endpoint status -w table` against all members afterwards. If one member answers `context deadline exceeded`, check that member's health and etcd log before closing. + +When compaction cannot bring the file under the default 2 GiB quota because the live key set is large, raise `--quota-backend-bytes` to `8589934592` first (see the related article on the etcd quota), wait for the member to come back, `alarm disarm`, compact to the current revision, then defrag each member with `--command-timeout=300s`. + +### Cause 2: Compaction frees only tens of megabytes and snap/db stays about 1.4G + +An inspection that flags the size while DB SIZE is only slightly above 500 MB needs no immediate action: keep monitoring and compact once the database approaches 1 GB. An inspection that reports the threshold reached is answered with the snapshot and compaction steps of Cause 1, without `alarm disarm` unless `NOSPACE` is set. + +When compaction has already run, DB SIZE dropped by only tens of megabytes, `snap/db` is still about 1.4G, and `endpoint status` shows no alarm and a stable leader, defrag each member, leader last. The default 5-second `etcdctl` client timeout can expire before defrag returns on a 1 GB file, so add `--command-timeout=120s` (a client wait, not an etcd setting). Changing `quota-backend-bytes` is not needed for this signature. + +> **Warning:** `defrag` rewrites this member's `snap/db` and can move the leader. Maintenance window, snapshot saved, one member at a time, and only for the "compaction freed tens of MB, file still about 1.4G" signature. If the member does not return, stop; the snapshot is for a whole-cluster restore only. + +```bash +etcdctl_on defrag --command-timeout=120s +``` + +## Diagnostic Steps + +```bash +kubectl -n kube-system get pods -l component=etcd -o wide +etcdctl_on endpoint status -w table +etcdctl_on alarm list +``` + +- `NOSPACE` in `alarm list`, or `alarm:NOSPACE` in the ERRORS column: Cause 1. +- A healthy endpoint whose DB SIZE barely moved after compact: Cause 2. +- A large live key count, usually `replicasets`, Pod create/delete bursts, or every member above 2 GB with kube-apiserver restarting on `leader changed`: see the related etcd quota article. + +Count keys by prefix to see whether one resource type dominates: + +```bash +etcdctl_on get / --prefix --keys-only \ + | awk -F '/' '{ h[$3]++ } END {for (k in h) print h[k], k}' | sort -nr | head -30 +``` + +On the node, `du -sh /var/lib/etcd/member /var/lib/etcd/member/snap /var/lib/etcd/member/wal /var/lib/etcd/member/snap/db` separates a large WAL from a large `snap/db`. + +The inspection item behind the MB threshold: + +```bash +kubectl get inspectionitems.ait.alauda.io -A | grep etcd +kubectl get inspectionitems.ait.alauda.io cluster-etcd-database-usage -o yaml | grep indicator +``` + +etcd backup files that fill `/` do not move `etcd_mvcc_db_total_size_in_bytes`; that is a disk-space problem, not this metric. diff --git a/docs/en/solutions/harbor-core_cannot_connect_to_Redis_NOAUTH_from_Sentinel_or_ERR_invalid_password_after_a_Redis_password_change.md b/docs/en/solutions/harbor-core_cannot_connect_to_Redis_NOAUTH_from_Sentinel_or_ERR_invalid_password_after_a_Redis_password_change.md new file mode 100644 index 000000000..fc4e8f79a --- /dev/null +++ b/docs/en/solutions/harbor-core_cannot_connect_to_Redis_NOAUTH_from_Sentinel_or_ERR_invalid_password_after_a_Redis_password_change.md @@ -0,0 +1,162 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "4.4" +--- + +# harbor-core cannot connect to Redis: NOAUTH from Sentinel, or ERR invalid password after a Redis password change + +## Issue + +`harbor-core` keeps restarting, or the Harbor login page reports that the core service is unavailable, and the core log shows that it cannot connect to Redis. Two signatures: + +- A Harbor that uses Redis in Sentinel mode never comes up, or creating a standalone (single-instance) Harbor fails, and the core log has: + + ```text + GetMasterAddrByName master="mymaster" failed: NOAUTH Authentication required + redis: all sentinels specified in configuration are unreachable + ``` + +- After the password of an external Redis was changed by editing the Redis Secret, `harbor-core` no longer starts, although `redis-cli` with the new password answers `PONG` and restarting `harbor-core` does not help: + + ```text + [/lib/cache/cache.go:126]: failed to ping redis://:xxxxx@:6379/0?idle_timeout_seconds=30 : ERR invalid password + [FATAL] [/core/main.go:161]: failed to initialize cache: retry timeout: ERR invalid password + ``` + +Related articles: [Redis for Harbor unavailable: no Data Services license, or Redis in cluster mode reported as empty secrets](Redis_for_Harbor_unavailable_no_Data_Services_license_or_Redis_in_cluster_mode_reported_as_empty_secrets.md). + +## Root Cause + +### Cause 1: the Redis instance has an independent Sentinel password + +Harbor connects to Redis in Sentinel mode through `spec.helmValues.redis.external` on the `Harbor` resource: `addr` lists the Sentinel endpoints on port 26379 (for example `rfs--.rfs--hl.:26379` for ordinals 0, 1, and 2), `existingSecret` names the Redis credential Secret, `existingSecretKey` is `password`, and `sentinelMasterSet` is `mymaster`. + +When the Redis instance is created in Data Services with both `Set Redis Password` and `Set Sentinel Password` enabled, its generated credential Secret (for example `redis--default-credential`) carries both a `password` and a `sentinelPassword` key. Harbor reads only the key named in `existingSecretKey` and never sends the Sentinel password. The Sentinel nodes reject Harbor's master lookup with `NOAUTH`, every Sentinel counts as unreachable, and `harbor-core` cannot find the Redis master. + +**Harbor does not support a Redis Sentinel independent password.** The Redis instance used by Harbor must have `Set Sentinel Password` disabled. The same limitation makes creation of a standalone Harbor fail. A manual `redis-cli` that succeeds with the Sentinel password does not rule this out, because Harbor does not send that password. + +### Cause 2: the operator did not carry a changed external Redis password into harbor-core + +With an external Redis (`redis.type: external`, `existingSecret: harbor-redis`, `existingSecretKey: password`), changing the password by editing the `harbor-redis` Secret does not update `REDIS_URL` in the Harbor core ConfigMap. `harbor-core` keeps authenticating with the old password, and restarting it re-reads the same stale ConfigMap. Triggering a Harbor instance update through the operator regenerates the configuration and `harbor-core` recovers. + +## Resolution + +### Cause 1 + +1. Confirm the cause (see Diagnostic Steps): `NOAUTH Authentication required` from `GetMasterAddrByName` in core, and `Set Sentinel Password` enabled on the Redis instance (or a `sentinelPassword` key in its credential Secret). + +2. In Data Services, open the Redis instance, go to its connection (access) settings, keep `Set Redis Password` enabled, and turn off `Set Sentinel Password`. + + > **Warning:** Only when Cause 1 is confirmed. Impact: the Sentinel endpoints of this Redis instance stop requiring the Sentinel password; the Redis data password stays in force. Other clients of this instance that send the Sentinel password must be checked. Rollback: enable `Set Sentinel Password` again, which returns Harbor to the `NOAUTH` failure. + +3. Watch `harbor-core` until the `NOAUTH` lines stop and the Pod becomes Ready: + + ```bash + NS= + kubectl -n "$NS" get pods -w | grep harbor-core + CORE=$(kubectl -n "$NS" get pods -o name | grep harbor-core | head -1) + kubectl -n "$NS" logs "$CORE" --tail=50 | grep -E 'NOAUTH|sentinels' || echo "no Sentinel auth errors" + ``` + +4. If the option cannot be turned off on the existing instance, recreate the Redis instance with `Set Sentinel Password` disabled. + + > **Warning:** Only when Cause 1 is confirmed and the Sentinel password cannot be disabled on the existing instance. Impact: deleting the Redis instance deletes its data; Harbor keeps cache, session, and job-queue data in Redis, so logged-in sessions end and queued jobs are lost. Precondition: the instance is used only by this Harbor (or Harbor is not yet in service), and a maintenance window. Deletion cannot be undone. + + Find the credential Secret Data Services generated for the new instance, and copy it into the Harbor namespace if it lives elsewhere: + + ```bash + kubectl get secret -n -l middleware.instance/type=Redis + ``` + +5. If the new instance's Sentinel addresses or Secret name differ from what the `Harbor` resource references, update the `Harbor` resource. + + > **Warning:** Only when the replacement Redis has a different address or Secret name. Impact: the operator restarts Harbor components; login, push, and pull fail until the Pods are Ready. Precondition: save the current resource with the first command below. Rollback: apply the saved file. + + ```bash + NS= + NAME= + kubectl -n "$NS" get harbor "$NAME" -o yaml > "$HOME/harbor-$NAME-before.yaml" + kubectl -n "$NS" patch harbor "$NAME" --type=merge -p '{ + "spec": {"helmValues": {"redis": {"type": "external", "external": { + "addr": "rfs--0.rfs--hl.:26379,rfs--1.rfs--hl.:26379,rfs--2.rfs--hl.:26379", + "existingSecret": "", + "existingSecretKey": "password", + "sentinelMasterSet": "mymaster" + }}}} + }' + ``` + +### Cause 2 + +1. Test the new password from a host that can reach Redis, then compare the password in the Secret with the one in the core ConfigMap without printing either: + + ```bash + redis-cli -u "redis://:@:6379/0" PING + NS= + kubectl -n "$NS" get secret harbor-redis -o jsonpath='{.data.password}' | base64 -d | sha256sum + CM=$(kubectl -n "$NS" get cm -o name | grep -- '-core$' | head -1) + kubectl -n "$NS" get "$CM" -o yaml | grep REDIS_URL \ + | sed -E 's#.*://[^:]*:([^@]*)@.*#\1#' | sort -u \ + | while read -r p; do printf '%s' "$p" | sha256sum; done + ``` + + The cause is confirmed when `PING` returns `PONG`, core logs `ERR invalid password`, and the ConfigMap digest differs from the Secret digest. A password with special characters can appear URL-encoded in `REDIS_URL`; decode it before comparing if the digests differ only for that reason. + +2. Trigger a Harbor instance update through the operator: open the instance in the Harbor operator's update view and submit it, or make a change to the `Harbor` resource that the operator reconciles. + + > **Warning:** Only when the cause is confirmed in step 1. Impact: the operator regenerates the Harbor configuration and restarts components; login, push, and pull fail until the Pods are Ready. Precondition: a short maintenance window and the `Harbor` resource saved. Rollback: apply the saved file (the stale password in the ConfigMap is not a state to return to). + + ```bash + NAME= + kubectl -n "$NS" get harbor "$NAME" -o yaml > "$HOME/harbor-$NAME-before-redis.yaml" + kubectl -n "$NS" patch harbor "$NAME" --type=merge \ + -p '{"spec":{"helmValues":{"core":{"resources":{"requests":{"cpu":""}}}}}}' + kubectl -n "$NS" get pods -w | grep harbor-core + ``` + +3. Run the digest comparison again: the two values match and the `ERR invalid password` lines stop. + +## Diagnostic Steps + +1. Read the core log: + + ```bash + NS= + kubectl get harbor -A + CORE=$(kubectl -n "$NS" get pods -o name | grep harbor-core | head -1) + kubectl -n "$NS" logs "$CORE" --tail=200 | grep -iE 'redis|sentinel|NOAUTH|invalid password' + ``` + + `NOAUTH Authentication required` from `GetMasterAddrByName` points to Cause 1; `ERR invalid password` with `redis.type: external` after a password change points to Cause 2. + +2. For Cause 1, check what Harbor is configured with and which keys the credential Secret carries (key names only): + + ```bash + kubectl -n "$NS" get harbor -o jsonpath='{.spec.helmValues.redis}{"\n"}' + kubectl -n "$NS" get secret \ + -o go-template='{{range $k, $v := .data}}{{$k}}{{"\n"}}{{end}}' + ``` + + Expected for Cause 1: a `sentinelPassword` key next to `password`, and `existingSecretKey: password` on the Harbor resource. + +3. Query Sentinel directly on port 26379 (add `-c ` if the Pod has more than one container): + + ```bash + kubectl -n exec rfs--0 -- \ + redis-cli -p 26379 SENTINEL get-master-addr-by-name mymaster + ``` + + `NOAUTH Authentication required` confirms that Sentinel requires its own password; after the option is disabled the command returns the master address and port. `redis-cli` defaults to port 6379, the data port, where a `SENTINEL` command returns `unknown command`. + +4. `redis: all sentinels specified in configuration are unreachable` alone is not specific to Cause 1. If `GetMasterAddrByName` fails with `dial tcp: lookup rfs--.rfs--hl. ...: no such host` instead of `NOAUTH`, the Sentinel Pods do not exist yet. Check them before changing any password setting: + + ```bash + kubectl -n get pods -o wide | grep -E 'rfs-|rfr-' + kubectl -n describe pod rfs--0 | sed -n '/Events:/,$p' + ``` + + `Init:ImagePullBackOff` or `Init:0/1` on the `rfs-` Pods with `Failed to pull image ... i/o timeout` is an image-pull problem on the node, not a password problem. diff --git a/docs/en/solutions/harbor-nginx_or_portal_does_not_start_or_keeps_restarting_IP_address_family_unsupported_or_OOMKilled.md b/docs/en/solutions/harbor-nginx_or_portal_does_not_start_or_keeps_restarting_IP_address_family_unsupported_or_OOMKilled.md new file mode 100644 index 000000000..3b482fdd1 --- /dev/null +++ b/docs/en/solutions/harbor-nginx_or_portal_does_not_start_or_keeps_restarting_IP_address_family_unsupported_or_OOMKilled.md @@ -0,0 +1,102 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "3.12" + - "4.2" + - "4.3" +--- + +# harbor-nginx or portal does not start or keeps restarting: IP address family unsupported or OOMKilled + +## Issue + +The Harbor web entry point is down or unstable because the `harbor-nginx` Pod (and sometimes the portal Pod) does not become Ready or keeps restarting: + +- `harbor-nginx` and the portal Pod never become Ready, and their logs contain `IP address family was unsupported`. +- `harbor-nginx` keeps restarting, and `kubectl describe pod` shows `Last State: Terminated, Reason: OOMKilled`. The Harbor service looks unstable even though `harbor-core` is up. + +Related articles: [Harbor core service unavailable or internal server error, stuck instance status, or Pods cycling after an edit](Harbor_core_service_unavailable_or_internal_server_error_stuck_instance_status_or_Pods_cycling_after_an_edit.md). + +## Root Cause + +### Cause 1: Harbor requires IPv6 on a cluster where IPv6 is disabled + +The Harbor resource is configured with IPv6 enabled, so nginx and the portal try to listen on IPv6. On a cluster where IPv6 is disabled, the listen fails with `IP address family was unsupported` and the Pods never become Ready. Setting the Harbor resource to IPv4 only, under `spec.helmValues.ipFamily`, fixes it. + +### Cause 2: the harbor-nginx memory limit is too low + +`harbor-nginx` exceeds its memory limit and is OOMKilled. A change made directly on the `harbor-nginx` Deployment is overwritten by the operator's sync unless the Deployment carries the annotation `skip-sync: "true"`. Adding that annotation and raising the memory limit to twice the original value stops the restarts. + +## Resolution + +### Cause 1 + +1. Confirm the error in the nginx and portal logs, and that the nodes are IPv4-only (see Diagnostic Steps). Save the current `ipFamily` block if one is set: + + ```bash + NS= + NAME= + kubectl -n "$NS" get harbor "$NAME" -o jsonpath='{.spec.helmValues.ipFamily}{"\n"}' + ``` + +2. Set IPv4 only. + + > **Warning:** Only when the nginx and portal logs say the IP address family is unsupported and the cluster is IPv4-only. Impact: the operator reconciles and restarts Harbor components. Precondition: a short maintenance window and the previous `ipFamily` values saved. Rollback: patch the previous values back; on an IPv4-only cluster they are the failing configuration, so roll back only when moving to a cluster that provides IPv6. + + ```bash + kubectl -n "$NS" patch harbor "$NAME" --type=merge -p \ + '{"spec":{"helmValues":{"ipFamily":{"ipv6":{"enabled":false},"ipv4":{"enabled":true}}}}}' + kubectl -n "$NS" get pods | grep -E 'nginx|portal' + ``` + +3. Wait until the nginx and portal Pods are Ready and check that their logs no longer contain the address-family error. + +### Cause 2 + +1. Confirm OOMKilled and read the current memory limit: + + ```bash + NS= + kubectl -n "$NS" describe pod | grep -A3 'Last State' + DEP=$(kubectl -n "$NS" get deploy -o name | grep harbor-nginx | head -1) + kubectl -n "$NS" get "$DEP" -o jsonpath='{.spec.template.spec.containers[0].resources.limits.memory}{"\n"}' + ``` + +2. Mark the Deployment so the operator does not overwrite it, and double the memory limit (`512Mi` becomes `1Gi`, `1Gi` becomes `2Gi`). + + > **Warning:** Only when `harbor-nginx` is OOMKilled. Impact: `skip-sync: "true"` stops the operator from syncing this Deployment, so later changes from the Harbor resource no longer reach it until the annotation is removed; the Pod restarts once with the new limit and may use twice the previous memory on its node. Precondition: the node has that much allocatable memory. Rollback: set the limit back to the printed value and remove the annotation (`kubectl -n "$NS" annotate "$DEP" skip-sync-`); removing only the annotation lets the operator restore the old limit and the OOMKilled loop. + + ```bash + kubectl -n "$NS" annotate "$DEP" skip-sync="true" --overwrite + kubectl -n "$NS" set resources "$DEP" --limits=memory= + kubectl -n "$NS" rollout status "$DEP" + ``` + +3. Watch the Pod. Another OOMKilled means doubling was not enough for that workload. + +For other Harbor components, such as `harbor-registry`, the limit is set on the Harbor resource instead; see [harbor-registry is OOMKilled with exit code 137, or a resource change on its Deployment is reverted](harbor-registry_is_OOMKilled_with_exit_code_137_or_a_resource_change_on_its_Deployment_is_reverted.md). + +## Diagnostic Steps + +```bash +NS= +kubectl -n "$NS" get pods -o wide | grep -E 'nginx|portal' +NGINX=$(kubectl -n "$NS" get pods -o name | grep harbor-nginx | head -1) +kubectl -n "$NS" logs "$NGINX" --tail=80 +kubectl -n "$NS" describe "$NGINX" | grep -A3 'Last State' +``` + +| What you see | Cause | +| --- | --- | +| `IP address family was unsupported` in nginx or portal logs, Pods never Ready | Cause 1 | +| `harbor-nginx` restarts, `Last State` `OOMKilled` | Cause 2 | +| `harbor-notary-server` restarts with `certificate has expired`, or the portal says the core service is unavailable | Related article above | + +To confirm the cluster is IPv4-only for Cause 1, check the node addresses; only IPv4 entries are expected: + +```bash +kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.addresses[*].address}{"\n"}{end}' +``` diff --git a/docs/en/solutions/harbor-registry_is_OOMKilled_with_exit_code_137_or_a_resource_change_on_its_Deployment_is_reverted.md b/docs/en/solutions/harbor-registry_is_OOMKilled_with_exit_code_137_or_a_resource_change_on_its_Deployment_is_reverted.md new file mode 100644 index 000000000..824f14cc7 --- /dev/null +++ b/docs/en/solutions/harbor-registry_is_OOMKilled_with_exit_code_137_or_a_resource_change_on_its_Deployment_is_reverted.md @@ -0,0 +1,146 @@ +--- +kind: + - Troubleshooting +products: + - Alauda DevOps +ProductsVersion: + - "3.14" + - "3.18" +--- + +# harbor-registry is OOMKilled with exit code 137, or a resource change on its Deployment is reverted + +## Issue + +The Harbor `registry` component keeps restarting and its log shows no obvious error, or it has hit its CPU or memory limit and the image registry is unavailable. Typical signs: + +- The `harbor-registry` Pod's `RESTARTS` count grows, it may alternate between `Running` and `CrashLoopBackOff`, and its last state shows exit code `137`. This can start after the node container runtime was switched from Docker to containerd. +- An engineer raises the limits on the `harbor-registry` Deployment, and the change is reverted shortly afterwards. +- Clicking Update on the Harbor instance reports that some operators are not ready because they were never deployed. After the Redis and PostgreSQL operators are installed, Update asks for a storage class that a local-PV instance does not have. + +## Root Cause + +### Cause 1: the registry container is OOMKilled at its memory limit + +The kernel kills the `registry` container when it reaches its memory limit. The container's last state shows reason `OOMKilled` and exit code `137` (`128 + 9`, the process ended by `SIGKILL`), and the registry log has no error because the process was killed from outside. Raising the resources of the Harbor instance stops the restarts. Because the operator owns the Deployment, the change must be made on the Harbor instance, not on the Deployment (see Cause 2). + +### Cause 2: the Harbor operator owns the component resources and reverts Deployment edits + +Harbor component resources are controlled by the `Harbor` custom resource. The Harbor operator reconciles the Deployments from it, so a manual edit of the Deployment limits is written back. With `spec.resourceAssign.type: GlobalAssign`, the operator derives per-component values from one global setting (for example cpu `"16"` and memory `16Gi`) and writes them to `status.helmValues`; those status values are what the Deployments get. The API group can be `operator.devops.alauda.io/v1alpha1` or `operator.alaudadevops.io/v1alpha1` depending on the Harbor line; `kubectl get harbor` works for both. + +Two ways give a component its own size: + +- **The Update form** on the instance details page, which switches from global to per-component assignment. It requires the Redis and PostgreSQL operators to be installed. On platform 3.18 it also moves Harbor's PostgreSQL and Redis to Data Services-managed instances on TopoLVM storage, which is why it asks for a storage class that an instance with `persistence.type: LocalPath` does not have. That is an upgrade path, not only a limit change. +- **Component resources in the Harbor resource** under `spec.helmValues`. For the registry container the path is `spec.helmValues.registry.registry.resources`: the outer `registry` is the Harbor registry component, the inner `registry` is its registry container, next to `controller` (the `registryctl` container). Other components, such as `core`, sit at the same level as the outer `registry`. This path works on a `LocalPath` instance without a storage migration. A common mistake is `spec.helmValues.registry.resources`, which the operator does not use for the registry container. + +## Resolution + +### Cause 1 + +1. Confirm the OOM kill and read the current values (Diagnostic Steps, steps 1 and 2). + +2. Raise the memory of the container whose last state is `OOMKilled` on the Harbor resource: `registry` at `spec.helmValues.registry.registry.resources`, or `registryctl` at `spec.helmValues.registry.controller.resources`. Choose a limit above the current one in `status.helmValues` and above the usage seen just before the kills, and keep the request no higher than a node can schedule. You can also update the instance resources from the UI. + + > **Warning:** Only when the `OOMKilled` / `137` signature is confirmed. Impact: the operator reconciles the Harbor Deployments; `harbor-registry` Pods roll, and pushes and pulls are interrupted until they are Ready. Precondition: the backup below is saved and the new limit fits on the nodes. Rollback: `kubectl apply -f /tmp/harbor-before-oom.yaml`. + + ```bash + kubectl -n get harbor -o yaml > /tmp/harbor-before-oom.yaml + kubectl -n edit harbor + ``` + + Merge the block into the existing `spec.helmValues`; do not replace keys such as `externalURL`, `expose`, or `persistence`: + + ```yaml + spec: + helmValues: + registry: + registry: + resources: + limits: + memory: + requests: + memory: + ``` + +3. Confirm the value reached the operator and the Deployment, and that the restart count stops growing under normal load: + + ```bash + kubectl -n get harbor -o jsonpath='{.status.helmValues.registry.registry.resources}{"\n"}' + kubectl -n get deploy -o jsonpath='{.spec.template.spec.containers[*].resources}{"\n"}' + kubectl -n rollout status deployment + kubectl get pods -A | grep harbor-registry + ``` + + If the new Pod is `Pending`, read its events for `Insufficient memory` and lower the request or free a node. If the Deployment does not change, the edit missed the field the operator reads. + +If all Harbor Pods start cycling after the edit, check whether the registry PVC is larger than the size declared in the Harbor resource; see [Harbor core service unavailable or internal server error, stuck instance status, or Pods cycling after an edit](Harbor_core_service_unavailable_or_internal_server_error_stuck_instance_status_or_Pods_cycling_after_an_edit.md) (Cause 4). + +### Cause 2 + +1. Save the resource and read how resources are assigned: + + ```bash + kubectl get harbor -A + kubectl -n get harbor -o yaml > /tmp/harbor-resources.yaml + grep -n -E 'resourceAssign|GlobalAssign|currentVersion|LocalPath|persistence' /tmp/harbor-resources.yaml + kubectl -n get harbor -o jsonpath='{.status.helmValues.registry}{"\n"}' + ``` + +2. Choose the path: + + - Use the Update form only when the move of PostgreSQL and Redis to Data Services instances on TopoLVM is the change you intend. If Update says operators were never deployed, install the Redis and PostgreSQL operators first. + - Otherwise, especially on a `LocalPath` instance, add component resources to the Harbor resource. Size them for your workload, comparing with the current value in `status.helmValues`. Example values for the registry container: + + > **Warning:** Impact: the operator reconciles the Harbor Deployments; `harbor-registry` Pods roll, and a limit that is still too small leaves the registry unavailable. Precondition: `/tmp/harbor-resources.yaml` is a complete copy of the object. Rollback: `kubectl apply -f /tmp/harbor-resources.yaml`. + + ```bash + kubectl -n edit harbor + ``` + + ```yaml + spec: + helmValues: + registry: + registry: + resources: + limits: + cpu: 300m + memory: 800Mi + requests: + cpu: 150m + memory: 400Mi + ``` + + Leave other components out unless you mean to resize them, or add them at the same level as `registry` (for example `spec.helmValues.core.resources`). + +3. Confirm with the commands in Cause 1, step 3. Do not edit the Deployment again: that is the change the operator reverts. When `harbor-registry` is Ready, pull one known tag. + +## Diagnostic Steps + +1. Read the last termination state of each container in the registry Pod, its events, and the previous log: + + ```bash + kubectl -n get pod -o jsonpath='{range .status.containerStatuses[*]}{.name}{" restarts="}{.restartCount}{" reason="}{.lastState.terminated.reason}{" exit="}{.lastState.terminated.exitCode}{"\n"}{end}' + kubectl -n get events --field-selector involvedObject.name= + kubectl -n logs -c registry --previous --tail=50 + ``` + + `OOMKilled` with exit code `137` is Cause 1. Exit code `137` without `OOMKilled` is a SIGKILL from elsewhere: check the events for a failing liveness probe before changing memory. A storage, configuration, or permission error at the end of the previous log points at the registry storage instead. + +2. Compare memory use with the limit, and look for the kernel OOM line on the node that ran the Pod. Usage that climbs to the limit before each restart confirms Cause 1 and gives the floor for the new limit: + + ```bash + kubectl top pod -n --containers + journalctl -k --since "1 hour ago" | grep -iE 'out of memory|oom-kill|killed process' + ``` + +3. Check whether the operator owns the values: + + ```bash + kubectl -n describe deploy | grep -A8 -E 'Limits|Requests' + kubectl -n get harbor -o yaml | grep -n -E 'resourceAssign|GlobalAssign|LocalPath' + ``` + + `GlobalAssign` and a reverted Deployment edit are Cause 2. + +4. If the restarting component is `harbor-nginx` rather than `harbor-registry`, see [harbor-nginx or portal does not start or keeps restarting: IP address family unsupported or OOMKilled](harbor-nginx_or_portal_does_not_start_or_keeps_restarting_IP_address_family_unsupported_or_OOMKilled.md). If nodes cannot pull images after a Docker-to-containerd switch (`dial tcp :80: i/o timeout`), see [Image registry or Harbor unreachable or slow because of firewall rules, blocked ports, or the network path](Image_registry_or_Harbor_unreachable_or_slow_because_of_firewall_rules_blocked_ports_or_the_network_path.md) (Cause 6). diff --git a/docs/en/solutions/kube-apiserver_latency_alerts_noisy_apiserver_logs_and_controller-manager_or_scheduler_health_alerts_on_ACP.md b/docs/en/solutions/kube-apiserver_latency_alerts_noisy_apiserver_logs_and_controller-manager_or_scheduler_health_alerts_on_ACP.md new file mode 100644 index 000000000..baf647e05 --- /dev/null +++ b/docs/en/solutions/kube-apiserver_latency_alerts_noisy_apiserver_logs_and_controller-manager_or_scheduler_health_alerts_on_ACP.md @@ -0,0 +1,231 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.1" + - "4.2" + - "4.3" +--- + +# kube-apiserver latency alerts, noisy apiserver logs, and controller-manager or scheduler health alerts on ACP + +## Issue + +- Alerts say kube-apiserver write latency is over 0.50 seconds, or the read success rate is under 0.80 or 0.90, while the static Pods are Running. `kubectl` is slow only for some calls, often Pod creates. +- An apiserver error-rate alert arrives, or the apiserver log is full of `invalid bearer token` and OpenAPI download errors such as `v1.enhancement.cluster.alauda.io ... resource not found`, and a control-plane restart is being considered. +- `cluster-enhancer-apiserver` uses a lot of memory and its resources need to change. +- The monitoring page shows kube-controller-manager as down, but `telnet 127.0.0.1 10257` on the node succeeds and the Pod is healthy. On a managed cluster the platform raises `集群内KubeScheduler的健康状态 != 1 且持续 1 分钟` and `集群内ControllerManager的健康状态 != 1 且持续 1 分钟` ("KubeScheduler / ControllerManager health status in the cluster != 1 for 1 minute", policy `cpaas-cluster-rules`), while both Pods are `1/1 Running` with no restarts. +- `kubectl get componentstatuses` shows `controller-manager` and `scheduler` as `Unhealthy`. +- An inspection reports apiserver errors tied to an already-compacted etcd revision and a connection reset. +- The apiserver error-rate alert rises and stays high after worker nodes were removed from the cluster, and the failed requests are mostly `404` on `nodes`. + +Check admission webhooks before scaling kube-apiserver or compacting etcd. + +## Root Cause + +### Cause 1: An admission webhook has no backend (dial tcp :443: connect: connection refused) + +kube-apiserver logs `dial tcp :443: connect: connection refused` while calling a mutating admission webhook (for example `chaosblade-operator-arm64.chaosblade.svc`), and `POST pods` p99 is about 1.25 seconds. The webhook Service still exists, but its Endpoints and EndpointSlice have no addresses: the operator Deployment is `READY 0/0`. The `MutatingWebhookConfiguration` still matches Pod creates with `failurePolicy: Ignore` and `timeoutSeconds: 10`, so each create waits on the dead backend and then continues. Deleting that MutatingWebhookConfiguration stops the alert. The same applies to any webhook configuration whose Service has no endpoints. + +### Cause 2: invalid bearer token and v1.enhancement.cluster.alauda.io resource not found + +The apiserver log contains, in volume: + +- `Unable to authenticate the request` with `err="invalid bearer token"` +- `Unhandled Error` with `loading OpenAPI spec for "v1.enhancement.cluster.alauda.io" failed` and `failed to download v1.enhancement.cluster.alauda.io: resource not found` + +The first line comes from the nevermore log agent; platform components call kube-apiserver with tokens and other methods, so these lines can appear while the platform works normally. The second comes from `cluster-enhancer-apiserver`. While kube-apiserver serves normally, these lines do not affect nevermore, `cluster-enhancer-apiserver`, or kube-apiserver and can be ignored. Expanding the audit policy to identify the client is not needed when node time is consistent across clusters and the platform works. + +The ignore conclusion covers the OpenAPI line only while the aggregated API behind it is up: APIService `v1.enhancement.cluster.alauda.io` reports `Available=True` and its Service has Ready endpoints. When the APIService reports `Available=False` (for example `MissingEndpoints` or `FailedDiscoveryCheck`), requests to that API group fail, and the line is a symptom of the missing backend, for example because the backend Pods have no node left to run on. + +### Cause 3: cluster-enhancer-apiserver memory is high and a resource change needs a restart + +`cluster-enhancer-apiserver` serves the `EtcdBackupConfiguration` API, certificate monitoring, etcd backup storage, and Prometheus metrics for that API. `cluster-enhancer-controller` cleans expired cert-manager certificates, monitoring dashboard configuration, and event records. Restarting `cluster-enhancer-apiserver` for a resource change does not affect workloads that are already running. + +### Cause 4: kube-controller-manager or kube-scheduler listens only on 127.0.0.1 + +Monitoring probes kube-controller-manager on `:10257` (and kube-scheduler on `:10259`), but the process listens only on `127.0.0.1`, or a firewall change blocks the node address: + +- Firewall: `telnet 127.0.0.1 10257` connects and `telnet 10257` is refused because the planned iptables forwarding rules were applied incorrectly. The fix restores the rules as planned and aligns the Prometheus Service, ServiceMonitor, and target port with the port that answers. +- Bind address: `netstat -lntp` shows kube-controller-manager on `127.0.0.1:10257` and kube-scheduler on `127.0.0.1:10259`. Forwarding cannot make `:10257` connect until the bind address changes. A burst of `pods is forbidden: User "system:kube-scheduler" cannot list resource "pods"` lines before `successfully acquired lease kube-system/kube-scheduler` in the scheduler log is not the cause. Scheduling and control are not affected. + +### Cause 5: kubectl get componentstatuses shows controller-manager and scheduler Unhealthy + +`kubectl get componentstatuses` prints `controller-manager` and `scheduler` as `Unhealthy` with `dial tcp 127.0.0.1:10257: connect: connection refused` (and the same for `127.0.0.1:10259`), warns that `v1 ComponentStatus` is deprecated in v1.19+, and shows `etcd-0` `Healthy`. This deprecated probe output has no impact on the service and needs no remediation. + +### Cause 6: apiserver errors from an already-compacted revision, with no leader change + +The requested historical revision was already removed by etcd auto-compaction, and the connection was reset. etcd has not restarted and is not changing leader. These errors have no impact and can be ignored. A cluster that also logs `leader changed` is not covered by this conclusion. + +### Cause 7: removed worker machines are still running and keep requesting their deleted Node objects + +Worker nodes were removed from the cluster, so their Node objects no longer exist, but the machines themselves were left running. Each of them keeps sending `GET` requests to kube-apiserver for its own Node object, and every such request returns `404`. The constant 404 stream raises the apiserver error rate even though the cluster itself is healthy. The 404 rate jumps when the nodes are removed and then stays flat; nearly all of it is `verb=GET` on resource `nodes`, while `PATCH` on `nodes/status` stays at zero and other resources show almost no errors. Shutting down the removed machines brings the error rate back to normal. + +## Resolution + +### Cause 1: An admission webhook has no backend + +Apply only when the apiserver log names this webhook, its Service has empty endpoints, and the operator has no ready Pods: + +```bash +kubectl get mutatingwebhookconfiguration -o yaml > /tmp/.yaml +kubectl get endpoints,endpointslice -n +kubectl get deploy,pods -n +``` + +> **Warning:** Deleting a MutatingWebhookConfiguration stops that mutation for every matched resource. Do it only when the log shows `connection refused` to that webhook and its Service has no endpoints; never for a webhook that still has ready Pods. With `failurePolicy: Ignore`, creates were already allowed after the timeout, so the delete removes the wait rather than changing allow versus deny. Rollback: `kubectl apply -f /tmp/.yaml`. + +```bash +kubectl delete mutatingwebhookconfiguration +``` + +Watch `POST pods` latency afterwards, and repeat the endpoint check for every other webhook before changing etcd. + +### Cause 2: invalid bearer token and v1.enhancement.cluster.alauda.io resource not found + +Check the APIService first; `readyz` reports kube-apiserver itself and does not show whether an aggregated API has a backend: + +```bash +kubectl get apiservice v1.enhancement.cluster.alauda.io +kubectl get apiservice v1.enhancement.cluster.alauda.io -o jsonpath='{.spec.service.namespace}/{.spec.service.name}{"\n"}{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}' +kubectl get endpoints -n +kubectl get --raw='/readyz?verbose' +``` + +- `AVAILABLE True`, endpoints present, and `readyz` succeeds: no change. Do not restart the control plane, rotate tokens, or delete `cluster-enhancer-apiserver` because of these lines. +- `AVAILABLE False`, or a Service without endpoint addresses: find out why the backend Pods are missing or not Ready (for example after worker nodes were removed), and fix that: + + ```bash + kubectl get nodes -o wide + kubectl get pods -A -o wide | grep cluster-enhancer-apiserver + kubectl describe pod -n + ``` + +For `invalid bearer token`, compare the clock on every node of every cluster before closing: + +```bash +date -u '+%F %T' +timedatectl | grep -E 'synchronized|NTP' +``` + +### Cause 3: cluster-enhancer-apiserver memory is high and a resource change needs a restart + +Change the resources where the platform owns that Deployment; a hand edit of the Pod template is reverted when a controller reconciles it. The restart does not disturb running workloads. + +```bash +kubectl get pods -A -o wide | grep cluster-enhancer-apiserver +``` + +### Cause 4: kube-controller-manager or kube-scheduler listens only on 127.0.0.1 + +Confirm the split between localhost and the node IP, and save iptables before editing: + +```bash +sudo ss -lntp | grep -E ':10257|:10259' +sudo iptables-save > /tmp/iptables-before-kcm.rules +kubectl get svc,servicemonitor -A | grep -i -E 'controller-manager|scheduler' +``` + +**Firewall:** restore the forwarding rules from the approved firewall plan (the exact rules are site-specific), then point the Service and ServiceMonitor at the port that answers. + +> **Warning:** Changing filter or nat rules can black-hole node traffic unrelated to port 10257. Edit only the rules the firewall plan uses for these ports, and only after localhost works while the node IP does not. Rollback: `sudo iptables-restore < /tmp/iptables-before-kcm.rules`. + +**Bind address on a cluster reconciled from `kubeadmConfigSpec`:** set the listen address there: + +```yaml +kubeadmConfigSpec: + clusterConfiguration: + controllerManager: + extraArgs: + bind-address: "::" + scheduler: + extraArgs: + bind-address: "::" +``` + +**Bind address on a cluster whose control plane runs from static manifests:** change `--bind-address` from `127.0.0.1` to the node IP that monitoring scrapes. + +> **Warning:** Changing `bind-address` restarts kube-controller-manager and kube-scheduler on each control-plane node; edit one master at a time, in a maintenance window, and only when `ss` shows the ports bound to `127.0.0.1`. The metrics and health ports then answer on the node address, so keep the firewall plan in front of them. Keep backups outside `/etc/kubernetes/manifests/`; a copy left in that directory is started as a second static Pod. Rollback: restore the backups (or set `bind-address` back to `127.0.0.1`) and the previous ServiceMonitor. + +```bash +sudo cp /etc/kubernetes/manifests/kube-controller-manager.yaml /var/tmp/kube-controller-manager.yaml.before +sudo cp /etc/kubernetes/manifests/kube-scheduler.yaml /var/tmp/kube-scheduler.yaml.before +sudo sed -i 's/--bind-address=127.0.0.1/--bind-address=/' /etc/kubernetes/manifests/kube-controller-manager.yaml +sudo sed -i 's/--bind-address=127.0.0.1/--bind-address=/' /etc/kubernetes/manifests/kube-scheduler.yaml +kubectl get pods -n kube-system -l 'component in (kube-controller-manager,kube-scheduler)' -o wide +``` + +After the Pods are Ready, `telnet 10257` from the Prometheus node should connect and the health alerts should clear. + +### Cause 5: kubectl get componentstatuses shows controller-manager and scheduler Unhealthy + +When that output is the only complaint and the Pods are Running, no change is needed: no firewall edit, no bind-address change, and no restart. If the monitoring page is also red, use Cause 4. + +```bash +kubectl get componentstatuses +kubectl -n kube-system get pods -l 'component in (kube-controller-manager,kube-scheduler,etcd)' -o wide +``` + +### Cause 6: apiserver errors from an already-compacted revision, with no leader change + +```bash +kubectl -n kube-system get pods -l component=etcd +kubectl -n kube-system logs etcd- --tail=200 | grep -F 'leader changed' || true +``` + +With a compacted-revision error plus a connection reset, no etcd restart, and no leader changes, ignore the lines. A `NOSPACE` alarm or leader changes need the etcd quota or disk investigation instead. + +### Cause 7: removed worker machines are still running and keep requesting their deleted Node objects + +1. Break the error rate down by code, then by resource, in the monitoring UI: + + ```text + sum by (code) (apiserver_request_count_rate_5m_sum_by_instance_and_code) + topk(20, sum by (resource, code) (apiserver_request_count_rate_5m_sum_by_resource_and_code{code=~"(4|5)..",code!="429"})) + sum by (verb, subresource) (rate(apiserver_request_total{job="apiserver",resource="nodes",code="404"}[5m])) + ``` + + This cause matches when `code="404", resource="nodes"` dominates, almost all of it is `verb="GET"`, and the rise started when the nodes were removed. + +2. Confirm the machines that send these requests are no longer cluster members: their names are absent from `kubectl get nodes`, and the machines are still powered on. + +3. Shut down the removed machines. + + > **Warning:** Shutting a machine down stops everything still running on it. Do this only for machines whose Node objects were deliberately removed from the cluster and that carry nothing else in use. Roll back by powering the machine on again; it will resume the 404 requests until it is shut down or rejoined. + + If a machine has to stay powered on for another purpose, stop and disable the kubelet on it instead, so that it no longer contacts the cluster: + + ```bash + systemctl disable --now kubelet + ``` + +4. Re-run the queries in step 1 after a few minutes. The `404` rate on `nodes` should drop to near zero and the error-rate alert should clear. + +## Diagnostic Steps + +Webhooks first, then the log signatures, then the controller-manager port: + +```bash +kubectl get mutatingwebhookconfiguration -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.webhooks[*].clientConfig.service.namespace}{"/"}{.webhooks[*].clientConfig.service.name}{"\tfail="}{.webhooks[*].failurePolicy}{"\ttimeout="}{.webhooks[*].timeoutSeconds}{"\n"}{end}' +kubectl get validatingwebhookconfiguration -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' +kubectl -n kube-system logs kube-apiserver- --tail=200 | grep -E 'connection refused|invalid bearer token|enhancement.cluster.alauda.io' || true +kubectl get apiservice | grep -v 'True' || true +``` + +For each webhook Service namespace printed above: + +```bash +kubectl get endpoints,pods,deploy -n +``` + +- Empty endpoints plus `connection refused`: Cause 1. +- The two log lines with a healthy `readyz` and APIService `v1.enhancement.cluster.alauda.io` `AVAILABLE True`: Cause 2. The same OpenAPI line with that APIService `False`: start from the missing backend Pods. +- High memory on `cluster-enhancer-apiserver`: Cause 3. +- `10257` open on `127.0.0.1` and refused on the node IP, or the KubeScheduler / ControllerManager health alerts with ports bound to `127.0.0.1`: Cause 4. +- `kubectl get componentstatuses` Unhealthy with the Pods Running and no monitoring failure: Cause 5. +- Compacted-revision errors with a quiet etcd: Cause 6. +- The error rate is mostly `404` from `GET` on `nodes`, starting when worker nodes were removed: Cause 7. + +Read-success alerts that are mostly HTTP 403 are authorization failures, not a webhook timeout. Capture one request's user and verb before deleting any webhook. When the 403s come with cert-manager webhook denials for `{cpaas-ca ClusterIssuer ait.alauda.io}` in the kube-apiserver log, they are expected; see *Certificate alerts from a custom VMRule, ait.alauda.io webhook denials, or by-design short-lived certificates on ACP* (Cause 2). diff --git a/docs/en/solutions/kube-ovn-cni_fails_on_one_node_after_a_NIC_rename_clock_skew_or_hostname_change_on_ACP.md b/docs/en/solutions/kube-ovn-cni_fails_on_one_node_after_a_NIC_rename_clock_skew_or_hostname_change_on_ACP.md new file mode 100644 index 000000000..ebd9063eb --- /dev/null +++ b/docs/en/solutions/kube-ovn-cni_fails_on_one_node_after_a_NIC_rename_clock_skew_or_hostname_change_on_ACP.md @@ -0,0 +1,168 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.12.x + - 3.18.x + - 4.0.x + - 4.2.x + - 4.3.x +--- + +# kube-ovn-cni fails on one node after a NIC rename, clock skew or hostname change on ACP + +Related articles: [Pods fail on one Kube-OVN node after Docker or a second runtime is installed or old cluster state remains on ACP](Pods_fail_on_one_Kube-OVN_node_after_Docker_or_a_second_runtime_is_installed_or_old_cluster_state_remains_on_ACP.md). + +## Issue + +`kube-ovn-cni` keeps restarting on one node or a group of nodes, or Pods there cannot be created, and the network plugin itself is not what broke. Typical reports: + +- After a master VM was migrated, `kube-ovn-cni` on it does not start. It is in `BackOff`, and the `cni-server` log ends with `"failed to parse config" err="network interface not exist"`. +- After an IaaS upgrade renamed the NICs of every worker (IPs unchanged), many Pods are `Unknown`, `kube-ovn-cni` logs `iface has no valid IP address`, and restarting `ovs-ovn` does not help. +- Adding a node fails with `Check network interface`, or a node whose NIC was renamed never brings OVN up. +- A node that was powered off came back with a wrong clock, and `kube-ovn-cni` and `ovs-ovn` there cannot create Pods. +- A node (for example a GPU node) was removed from the cluster, its hostname was changed, and after it rejoined, the component Pods on it are abnormal. + +Each cause below is a property of the host: its interface names, its clock, or its identity in the OVN database. + +## Root Cause + +### Cause 1: Check network interface, the configured interface name is not on the node + +The cluster was created with one interface name, but some machines use another name, so those nodes never finish joining. Separately, renaming a NIC on a node that is already in the cluster leaves OVN unable to start, because restarting `ovs-ovn` does not re-read host NIC configuration. At cluster creation the platform can take a different interface name per node. For a node already in the cluster, check Cause 2 first; re-adding the node after the name is corrected is the fallback. + +### Cause 2: failed to parse config or failed to initialize config for the old NIC after a VM migration or IaaS upgrade + +`kube-ovn-cni` reads the tunnel interface of a node from the Node annotation `ovn.kubernetes.io/tunnel_interface`. The per-node annotation overrides the DaemonSet `--iface` argument. When the NIC is renamed on the host side (VM migration, IaaS upgrade) and the annotation still holds the old name, `kube-ovn-cni` cannot build its configuration and `cni-server` keeps restarting. + +After a VM migration (Kube-OVN `v1.9.25`), the events show `NodeNotReady`, `SandboxChanged`, `Error: services have not yet been read at least once, cannot construct envvars`, repeated restarts of the `cni-server` container and `Back-off restarting failed container`. The `cni-server` log reads: + +```text +Find node tunnel interface name: +network interface not exist +failed to find iface , network interface not exist +"failed to parse config" err="network interface not exist" +``` + +After an IaaS upgrade that renamed the NICs but kept the IPs (Kube-OVN `v1.15.17`), the old name may still exist without the node IP: + +```text +iface has no valid IP address +"failed to initialize config" err="iface has no valid IP address" +``` + +Restarting `ovs-ovn` does not help in either case. Setting the annotation to the current interface name does. + +### Cause 3: the node clock is far from the cluster after a power cycle + +After a node is shut down and powered on again, its clock can be far from the rest of the cluster. `kube-ovn-cni` and `ovs-ovn` on that node then cannot create Pods, a large set of Pods stays abnormal, and new Pods cannot be created there. A clock hours ahead of the API server also makes the API server reject that node's clients. The platform requires node clocks to stay within 10 seconds of each other. + +### Cause 4: the node rejoined with a new hostname and OVN still holds its old chassis + +When a node is removed, renamed and joined again (seen on ACP 3.12), the OVN southbound database still holds a chassis record for the old hostname. The component Pods on the node are abnormal until that stale chassis is destroyed. A node that moves between clusters or gets a new management IP can leave the same stale row. + +## Resolution + +### Cause 1: Check network interface, the configured interface name is not on the node + +- For a cluster that has not finished creating, recreate it and set each node's interface to the name shown by `ip -br link` on that machine. Do not keep a single global name that only some nodes have. +- For a node already in the cluster whose interface was renamed, try Cause 2 first. If the `kube-ovn-cni` log shows no Cause 2 signature, or the annotation did not recover the node, remove the node, fix the host interface name (or the per-node interface setting) so it matches what Kube-OVN was given, and add the node again through the platform node workflow. + + > **Warning:** `kubectl drain` evicts workloads and `kubectl delete node` removes the node from scheduling. Pods with local storage stay stranded. Add the node back before leaving the maintenance window. Precondition: no Cause 2 signature, or the Cause 2 annotation did not recover the node. With a Cause 2 signature, do not drain or delete the node; annotate it instead. + + ```bash + kubectl drain --ignore-daemonsets --delete-emptydir-data + kubectl delete node + ``` + +### Cause 2: failed to parse config or failed to initialize config for the old NIC after a VM migration or IaaS upgrade + +1. Confirm the signature, and compare the annotation with the interfaces the node really has. `ip link` prints a VLAN interface as `@` (for example `bond0.1341@bond0`); compare the part before `@` with the annotation. + + ```bash + kubectl -n kube-system logs -c cni-server --tail=100 | grep -E 'tunnel interface|not exist|no valid IP|failed to (parse|initialize) config' + kubectl get node -o jsonpath='{.metadata.annotations.ovn\.kubernetes\.io/tunnel_interface}{"\n"}' + kubectl -n kube-system get ds kube-ovn-cni -o yaml | grep -- '--iface' + ip -br addr + ip route show default + kubectl get node -o yaml > /root/node-.before.yaml + ``` + +2. Set the annotation to the NIC that now holds the node IP. + + > **Warning:** The annotation decides which NIC Kube-OVN uses for the tunnel on this node; a wrong name leaves the node without a working tunnel. Only when the log names the old NIC (`not exist` or `has no valid IP address`), `ip -br addr` shows the node IP on another NIC, and the NIC changed on the host side. To roll back, restore the annotation from the saved node YAML. + + ```bash + kubectl annotate node --overwrite ovn.kubernetes.io/tunnel_interface= + kubectl -n kube-system get pod -o wide | grep -E 'kube-ovn-cni|ovs-ovn' | grep + ``` + + After a VM migration, `kube-ovn-cni` starts after the annotation update; no Pod deletion, drain or node re-add is needed. Annotate every renamed node. + +3. If `kube-ovn-cni` stays in back-off (as after the IaaS upgrade), restart it on that node. + + > **Warning:** Deleting the `kube-ovn-cni` Pod stops sandbox creation on that node until the new Pod is Ready. Only after the annotation names the NIC that holds the node IP. The DaemonSet recreates the Pod; there is nothing to roll back. + + ```bash + kubectl -n kube-system delete pod -l app=kube-ovn-cni --field-selector spec.nodeName= + ``` + +### Cause 3: the node clock is far from the cluster after a power cycle + +1. Compare the clock with a healthy node, and set it from the same time source the healthy nodes use: + + ```bash + timedatectl + chronyc tracking || true + ``` + +2. Delete the `kube-ovn-cni` and `ovs-ovn` Pods on that node so the DaemonSets recreate them. + + > **Warning:** Deleting `ovs-ovn` drops forwarding on that node until the new Pod is Ready. Confirm the clock is correct first. Stepping the clock by hours can expire TLS sessions on the node; that is part of the repair, and local clients reconnect. There is no rollback of a deleted Pod; the DaemonSet creates another. + + ```bash + kubectl -n kube-system delete pod + ``` + +3. If a TopoLVM volume-prepare Pod in `nativestor-system` is still not `Completed` after the network Pods are Ready, delete that prepare Pod too; it is recreated and reaches `Completed`. Skip this when that namespace has no stuck prepare Pod. + + > **Warning:** Deleting the TopoLVM prepare Pod retries volume-group setup and can disrupt a volume that was mid-prepare. + + ```bash + kubectl -n nativestor-system delete pod + ``` + +### Cause 4: the node rejoined with a new hostname and OVN still holds its old chassis + +Find the chassis whose hostname is the node's old name, destroy only that row, and restart the OVN Pods on the node. + +> **Warning:** Destroying a chassis takes effect immediately. Destroying the chassis of a live node cuts that node's Pod traffic until its `ovn-controller` registers again. Match the old hostname exactly and save the list first. There is no undo other than the node registering again. + +```bash +kubectl ko sbctl --columns=_uuid,name,hostname list chassis | tee /root/chassis.before +kubectl ko sbctl destroy chassis +kubectl -n kube-system delete pod +kubectl ko sbctl --columns=name,hostname list chassis | grep -A1 +``` + +## Diagnostic Steps + +Look at the host before touching Kube-OVN objects: + +```bash +kubectl get nodes -o wide +kubectl -n kube-system get pod -o wide | grep -E 'kube-ovn-cni|ovs-ovn|kube-ovn-pinger' | grep +kubectl -n kube-system describe pod | tail -20 +kubectl -n kube-system logs -c cni-server --tail=100 +ip -br link +timedatectl +kubectl ko sbctl --columns=name,hostname list chassis +``` + +- A node-add check that names an interface the machine does not have is Cause 1. +- `network interface not exist` (`failed to parse config`) or `iface has no valid IP address` (`failed to initialize config`) after a VM migration or IaaS upgrade is Cause 2. +- A clock that differs from healthy nodes by more than a few seconds (often hours) after power-on is Cause 3. +- A rejoined node whose old hostname is still listed in the chassis table is Cause 4. +- If Pods fail on the node with `no packets received from gateway` after Docker or another runtime was installed, or the node used to belong to another cluster, see the related article. diff --git a/docs/en/solutions/kube-ovn-pinger_CrashLoopBackOff_context_deadline_exceeded_on_bonded_NICs_due_to_Geneve_TX_checksum_offload_on_ACP.md b/docs/en/solutions/kube-ovn-pinger_CrashLoopBackOff_context_deadline_exceeded_on_bonded_NICs_due_to_Geneve_TX_checksum_offload_on_ACP.md new file mode 100644 index 000000000..cd52fe327 --- /dev/null +++ b/docs/en/solutions/kube-ovn-pinger_CrashLoopBackOff_context_deadline_exceeded_on_bonded_NICs_due_to_Geneve_TX_checksum_offload_on_ACP.md @@ -0,0 +1,96 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# kube-ovn-pinger CrashLoopBackOff context deadline exceeded on bonded NICs due to Geneve TX checksum offload on ACP + +## Issue + +Right after a workload cluster is created on physical machines whose network uses a bond, every `kube-ovn-pinger` pod in `kube-system` is `0/1` `CrashLoopBackOff`. Other pods that talk to the API server, such as `kube-prometheus-exporter-kube-state` and `vm-operator` in `cpaas-system`, can crash-loop as well. + +The `kube-ovn-pinger` log shows that the pod cannot reach the API server through the `kubernetes` Service ClusterIP: + +```text +config.go:162] failed to get self pod kube-system/kube-ovn-pinger-: Get "https://:443/api/v1/namespaces/kube-system/pods/kube-ovn-pinger-?timeout=15s": context deadline exceeded (Client.Timeout exceeded while awaiting headers) +"failed to parse config" err=... context deadline exceeded +``` + +`` is the ClusterIP of the `default/kubernetes` Service (`443/TCP`). The nodes themselves can reach each other. + +## Root Cause + +Kube-OVN carries pod traffic between nodes in Geneve tunnels (UDP `6081`). With TX checksum offload enabled, the kernel leaves the checksum of the encapsulated packets for the NIC to fill in. On some physical NICs, the checksum computed after offload for Geneve-encapsulated packets is wrong, so the receiving side drops the packets. + +Pod-to-API-server traffic through the `kubernetes` Service crosses that path, so it times out, and `kube-ovn-pinger` exits because it cannot load its own pod. Host-to-host traffic is not encapsulated and keeps working, which is why the nodes look healthy. + +A bond is a virtual interface: the offload is actually performed by its slave (member) NICs, so the setting has to be changed on every slave NIC, not only on the bond. + +The permanent fix is a NIC firmware update from the NIC vendor. Disabling TX offload is the workaround. + +## Resolution + +1. On each affected node, list the slave NICs of the bond: + + ```bash + cat /proc/net/bonding/ | grep 'Slave Interface' + ``` + +2. Disable TX offload on every slave NIC. + + > **Warning:** Changing NIC offload settings can briefly disturb traffic on that NIC and moves checksum work to the CPU, which increases CPU usage under heavy traffic. Apply it only when the checksum-offload signature in the Diagnostic Steps is confirmed, and on production clusters apply it one node at a time. To roll back, run `ethtool -K tx on` and remove the persistent setting from step 3. + + ```bash + ethtool -K tx off + ethtool -k | grep -E 'tx-checksumming|tx-checksum-ip-generic' + ``` + + Expected: `tx-checksumming: off`. + +3. Make the setting persistent across reboots. On systems that use `network-scripts`, add the option to the configuration file of each slave NIC: + + ```bash + echo 'ETHTOOL_OPTS="-K tx off"' >> /etc/sysconfig/network-scripts/ifcfg- + ``` + + On systems managed by NetworkManager without `network-scripts`, use the equivalent persistent setting of that tool. + +4. Delete the crash-looping pods so they restart immediately instead of waiting for the back-off: + + ```bash + kubectl -n kube-system delete pod -l app=kube-ovn-pinger + ``` + + Confirm they become `1/1 Running`. + +5. Contact the NIC vendor for a firmware update that fixes Geneve checksum offload. After the firmware is updated, TX offload can be re-enabled and verified. + +## Diagnostic Steps + +1. Check whether the failure is limited to pod-to-Service traffic. From the node, the API server answers; from inside the pod network, the ClusterIP times out: + + ```bash + kubectl get svc -n default kubernetes + curl -k -m 5 https://:6443/healthz # on the node: ok + ``` + + If the node cannot reach `:6443` either, the problem is in the host network (firewall, routing), not in the overlay. + +2. Check the offload state of the bond's slave NICs. `tx-checksumming: on` on a slave NIC of the encapsulation path matches this issue; `off [fixed]` rules it out: + + ```bash + ethtool -k | grep -E 'tx-checksumming|tx-checksum-ip-generic' + ``` + +3. Capture Geneve traffic on the receiving node. Packets arriving with bad checksums, or not arriving at all while the sender transmits, point at the offload: + + ```bash + tcpdump -i -nn -vv udp port 6081 + ``` + + A capture taken on the sending node shows bad checksums whenever offload is on, because `tcpdump` sees the packet before the NIC fills in the checksum. Only a capture on the receiving side is evidence. + +4. If disabling TX offload on all slave NICs makes `kube-ovn-pinger` recover, the cause is confirmed. diff --git a/docs/en/solutions/kubectl_returns_the_server_has_asked_for_the_client_to_provide_credentials_after_ac_login_or_ac_login_fails_on_ACP.md b/docs/en/solutions/kubectl_returns_the_server_has_asked_for_the_client_to_provide_credentials_after_ac_login_or_ac_login_fails_on_ACP.md new file mode 100644 index 000000000..92aa30665 --- /dev/null +++ b/docs/en/solutions/kubectl_returns_the_server_has_asked_for_the_client_to_provide_credentials_after_ac_login_or_ac_login_fails_on_ACP.md @@ -0,0 +1,113 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.2" +--- + +# kubectl returns the server has asked for the client to provide credentials after ac login, or ac login fails, on ACP + +## Issue + +One of the following happens with the `ac` CLI or with platform API tokens: + +- The day after `ac login`, `kubectl` fails while the console still works: + + ```text + error: You must be logged in to the server (the server has asked for the client to provide credentials) + ``` + +- `ac login` returns HTTP 400, while `curl` against the platform address returns 302. +- The question is whether `ac login --idp` works with an Azure AD account. +- On ACP 4.2, a user token is needed from the command line or an API instead of the console. + +Related articles: "Forgotten local admin password on ACP: reset the password hash, set a chosen password, and password policy limits" covers local passwords. "ACP console session survives cookie deletion or a browser close, or the login page shows the previous account" covers browser sessions. + +## Root Cause + +### Cause 1: the kubeconfig written by `ac login` carries a one-day token + +`ac login`, as described in the Configure ACP CLI documentation, writes a kubeconfig whose token is valid for one day. After it expires, every `kubectl` call is rejected with the credentials error, while the console session is unaffected. + +### Cause 2: the installed `ac` binary is too old + +An old `ac` build sends a login request the platform rejects with 400, although the platform itself answers (`curl` returns 302). Reinstalling `ac` makes login succeed. + +### Cause 3: `ac login --idp` supports Windows AD, not Azure AD + +`ac login --idp` works with a Windows AD identity provider. Azure AD is not supported by that command. + +### Cause 4: on ACP 4.2 a user token is created with an API call and shown only once + +On ACP 4.2 a user token can be created with `POST /api/v1/users/{userId}/tokens`, authenticated, with a body that includes the token name and the expiration time. The response contains the token string, which cannot be read again after creation. + +## Resolution + +### Cause 1: the kubeconfig written by `ac login` carries a one-day token + +1. Create a long-lived token on the user's personal information page in the console. +2. In the kubeconfig written by `ac login`, replace the token of that user: + + ```yaml + users: + - name: + user: + token: + ``` + +3. Confirm that a call succeeds: + + ```bash + kubectl get --raw=/readyz + ``` + + It should print `ok` instead of the credentials error. + +### Cause 2: the installed `ac` binary is too old + +1. Compare the CLI and the platform response: + + ```bash + ac version + curl -k -sS -o /dev/null -w '%{http_code}\n' https:/// + ``` + + A 302 from `curl` with a 400 from `ac login` matches this cause. + +2. Download and install `ac` again as described in the Configure ACP CLI documentation for this platform version, then log in again. + +### Cause 3: `ac login --idp` supports Windows AD, not Azure AD + +Use `ac login --idp` for Windows AD identity providers. Azure AD users log in through the console. + +### Cause 4: on ACP 4.2 a user token is created with an API call and shown only once + +Send an authenticated `POST` to `/api/v1/users//tokens` with a JSON body that contains the token name and the expiration time. Take the exact field names from the API reference of your platform version. + +```bash +curl -k -sS -X POST \ + -H "Authorization: Bearer ${TOKEN}" \ + -H 'Content-Type: application/json' \ + --data '' \ + "https:///api/v1/users//tokens" +``` + +Copy the token string out of the response and store it securely; it cannot be retrieved later. Put it in a kubeconfig as in Cause 1 if `kubectl` should use it. + +## Diagnostic Steps + +```bash +kubectl config view --minify -o jsonpath='{.users[0].name}{"\n"}' +kubectl get --raw=/readyz +ac version +``` + +| What you see | Cause | +| --- | --- | +| `kubectl` credentials error about a day after `ac login`, console still works | 1 | +| `ac login` returns 400, `curl` to the platform returns 302 | 2 | +| Azure AD account cannot log in with `ac login --idp` | 3 | +| A user token is needed without the console on ACP 4.2 | 4 | +| The credentials error comes from a node's `/root/.kube/config` or `admin.conf`, not from `ac login` | Not this article | diff --git a/docs/en/solutions/nevermore_OOMKilled_cache_written_to_var_cpaas_or_callback-secret_mount_failure_on_ACP.md b/docs/en/solutions/nevermore_OOMKilled_cache_written_to_var_cpaas_or_callback-secret_mount_failure_on_ACP.md new file mode 100644 index 000000000..19af11b4e --- /dev/null +++ b/docs/en/solutions/nevermore_OOMKilled_cache_written_to_var_cpaas_or_callback-secret_mount_failure_on_ACP.md @@ -0,0 +1,207 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.18.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# nevermore OOMKilled, cache written to /var/cpaas, or callback-secret mount failure on ACP + +## Issue + +Application pods are logging and `kubectl logs` shows the lines, but the log console is empty or hours behind. On the nodes, one of the following is seen: + +- The `nevermore` DaemonSet in `cpaas-system` is restarting, and its main container or its `audit` container shows `OOMKilled`. On the node, `dmesg -T | grep -i -E 'out of memory|killed process'` shows `Memory cgroup out of memory` every few minutes, while the cluster monitoring shows node CPU and memory at only 20% to 30%. +- `/var` is at 100% because files appeared under `/var/cpaas`. +- On a fresh install, the `nevermore` pods sit in `Init:0/1` or `ContainerCreating` with `secret "log-collector-bearer-auth" not found`. + +`nevermore` is the per-node log agent. It reads container logs, Kubernetes audit records and events, and produces them into the platform Kafka, from which they are written into Elasticsearch or ClickHouse. Its local queue is under `/cpaas` (`/cpaas/log`, `/cpaas/audit`, `/cpaas/event`), and its own log is `/cpaas/log/logs/nevermore`. + +A `Memory cgroup out of memory` kill means that a container reached its **own** memory limit. That happens regardless of how much memory the node has free, which is why node-level monitoring can look low at the same time. + +When `nevermore` is `Pending`, fails on `filebeat.lock`, or is `OOMKilled` with a large disk queue after the log store was down, see [nevermore Pending, Error with filebeat.lock, or OOMKilled after a disk queue backlog on ACP](./nevermore_Pending_Error_with_filebeat_lock_or_OOMKilled_after_a_disk_queue_backlog_on_ACP.md). + +## Root Cause + +### Cause 1: The nevermore main container or its audit container is OOMKilled by its memory limit + +`kubectl describe` on a `nevermore-*` pod shows `Reason: OOMKilled` and exit code 137, either on the main container or on the container named `audit`. The container names differ between versions (for example `nevermore-audit`, `nevermore-event` and `nevermore-log` on older releases), so read them from the DaemonSet. While the container is down, the agent stops shipping, and the records wait in its on-disk queue under `/cpaas`. + +Consumer lag is the gap between what was produced and what was consumed, so a stopped producer does not raise it. The lag on `ALAUDA_LOG_TOPIC` climbs when the restarted agent flushes its queue faster than the consumer reads it. It falls once the container stays up and the consumer catches up. + +The node's log volume exceeds what the current limit allows. The main container stops restarting after its resources are raised (to about 2 CPU and 2Gi). An `audit` container OOM is cleared the same way, by raising that container's resources. + +One workload can be the whole cause. On one cluster, the `log` container of `nevermore` ran at its 2000m CPU limit, and its memory spiked toward the 4096Mi limit. The other containers stayed low, and the cache under `/cpaas/log/data` was small. The real-time application log showed about 600 thousand entries per minute, nearly all from one test workload running an `echo` loop with no resource limits. Stopping that application ended the OOM, and the next day the `log` container ran at about 19m CPU and 552Mi. + +A full node disk can also push the agent over its limit. Free the disk before you only add memory, or the same pod is OOMKilled again. + +### Cause 2: spec.valuesOverride on the logagent ModuleInfo points the cache at /var/cpaas + +`df` shows `/var` at 100%, and `du -sh /var/cpaas` shows several gigabytes, while `/cpaas` itself may still have room. The installed log agent module object (kind ModuleInfo, short name `minfo`, on the global cluster) has a `spec.valuesOverride` that changed the `nevermore` local cache directory. The product default is `/cpaas/`. After the override, `event`, `log` and `audit` are written onto the smaller `/var` partition. + +Deleting that `spec.valuesOverride` puts the cache back on `/cpaas/`. Files that are already under `/var/cpaas` are not moved by the revert. + +### Cause 3: FailedMount for volume callback-secret, secret log-collector-bearer-auth not found + +The pod event is: + +```text +MountVolume.SetUp failed for volume "callback-secret": secret "log-collector-bearer-auth" not found +``` + +The agent cannot start because the Secret that the callback volume mounts was never created. For example, on a 4.3 business cluster every `nevermore` pod sits at `0/4` `Init:0/1`. `kubectl get secret -A | grep log` lists `log-callback-secret` and `log-center-secret` in `cpaas-system`, but no `log-collector-bearer-auth`. + +Reinstalling the Log Collector does not create the Secret. The Alauda Container Platform Log Essentials plugin (`log-essentials`) creates it. Install Log Essentials on the cluster before the Log Collector, then restart Deployment `cluster-transformer` in `cpaas-system` so that it regenerates the Secret. + +## Resolution + +### Cause 1: The nevermore main container or its audit container is OOMKilled by its memory limit + +1. List the containers and the last termination: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep nevermore + kubectl -n cpaas-system get ds nevermore -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" "}{.resources.limits.memory}{"\n"}{end}' + kubectl -n cpaas-system describe pod | grep -E 'OOMKilled|Exit Code|Limits|Requests' + ``` + +2. Before raising limits, check whether one workload floods its output or the disk is full: + + ```bash + kubectl -n cpaas-system top pod --containers | grep nevermore + du -sh /cpaas/log/data/* 2>/dev/null + df -hT /cpaas + ``` + + Run `du` and `df` on the node. If only the `log` container is hot and the cache is small, find the workload with the largest real-time log volume in the console, and stop or fix it. Free a full disk first. + +3. Raise the resources of the container that was OOMKilled. + + > **Warning:** Raising CPU and memory on DaemonSet `nevermore` rolls the agent on every node, and collection pauses on a node while its pod restarts. Precondition: the node has the extra memory free, and you noted the previous requests and limits. A module controller can revert a direct DaemonSet edit. If the pod spec snaps back, set the same numbers in the log agent ModuleInfo `spec.valuesOverride`, under the resources key that object already uses. Rollback: `kubectl set resources` with the previous values, or re-apply the saved ModuleInfo. + + ```bash + kubectl -n cpaas-system set resources ds/nevermore \ + -c --requests=cpu=2,memory=2Gi --limits=cpu=2,memory=2Gi + ``` + + Repeat the command with `-c` for the `audit` container if that is the one with `OOMKilled`. Size the `audit` container above the usage that killed it, instead of copying 2Gi onto every container. + +4. Watch that `RESTARTS` stops increasing, then confirm that the Kafka lag is falling: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep nevermore + ``` + +### Cause 2: spec.valuesOverride on the logagent ModuleInfo points the cache at /var/cpaas + +1. On the global cluster, save the ModuleInfo: + + ```bash + kubectl get minfo | grep logagent | grep + kubectl get minfo -o yaml > /tmp/logagent-minfo.before.yaml + kubectl get minfo -o jsonpath='{.spec.valuesOverride}' + ``` + +2. Remove the cache-path override. + + > **Warning:** Clearing the whole `spec.valuesOverride` drops every override on that module, including resource overrides from Cause 1, and `nevermore` rolls back to the default cache directory `/cpaas/`. Precondition: `/cpaas` has free space, you saved the YAML, and you read the override so that other keys can be put back. Rollback: `kubectl apply -f /tmp/logagent-minfo.before.yaml`. If other keys must stay, delete only the key that sets the cache path. + + ```bash + kubectl edit minfo + ``` + + Wait until DaemonSet `nevermore` is Ready, then check on the node that the default directories exist: + + ```bash + ls -ld /cpaas/event /cpaas/log /cpaas/audit + ``` + +3. If `/cpaas/event`, `/cpaas/log` or `/cpaas/audit` does not exist yet and the only copy is under `/var/cpaas`, move that directory once. Skip the move when `/cpaas/` already exists, because that tree is the live queue. + + > **Warning:** The directories under `/var/cpaas` hold unshipped cache, and a later delete of the source is irreversible. Precondition: `nevermore` has been reverted to `/cpaas/`, and `du` showed these directories under `/var/cpaas`, not `/var/log` or `/var/lib`. Rollback: move the directory back, only if `nevermore` has not started a new queue at the destination. + + ```bash + mkdir -p /cpaas + for d in event log audit; do + if [ -d "/var/cpaas/$d" ] && [ ! -e "/cpaas/$d" ]; then + mv "/var/cpaas/$d" "/cpaas/$d" + fi + done + ``` + +4. Free what remains under `/var/cpaas`. + + > **Warning:** `rm -rf` under `/var/cpaas` deletes log cache that has not been shipped, so search has a gap for those records. Precondition: the path that `du` printed is under `/var/cpaas`, and `nevermore` is already writing the restored `/cpaas/` tree. When `/var` is at 100% and pods cannot start, deleting the `logs` directory first gives temporary relief before the revert takes effect. Rollback: none. + + ```bash + du -sh /var/cpaas /var/cpaas/* + rm -rf /var/cpaas//logs + ``` + +### Cause 3: FailedMount for volume callback-secret, secret log-collector-bearer-auth not found + +1. Confirm the signature: + + ```bash + kubectl get secret -A | grep log + kubectl -n cpaas-system get pods | grep nevermore + kubectl -n cpaas-system describe pod | grep -A2 FailedMount + ``` + +2. Install the `log-essentials` plugin from the cluster plugin page, where the log collector was installed. Reinstalling only the Log Collector does not create the Secret. + +3. Restart the transformer so that it regenerates the Secret. + + > **Warning:** Restarting `cluster-transformer` pauses plugin status updates during the rollout. It does not delete log data. Precondition: `log-essentials` is installed or installing. Rollback: none needed; a restart has no spec change to undo. + + ```bash + kubectl -n cpaas-system rollout restart deployment/cluster-transformer + kubectl -n cpaas-system rollout status deployment/cluster-transformer + kubectl -n cpaas-system get secret log-collector-bearer-auth + kubectl -n cpaas-system get pods | grep nevermore + ``` + + The Secret must exist, and the `nevermore` pods must leave `Init` and `ContainerCreating`. + +4. If Log Essentials was already installed and `log-collector-bearer-auth` is still missing after the transformer rollout, this procedure does not apply. That situation needs a different investigation, for example on a standby global cluster. + +## Diagnostic Steps + +1. Compare the source with the agent before you look at Kafka: + + ```bash + kubectl logs -n -c --tail=20 + kubectl -n cpaas-system get pods -o wide | grep nevermore + ``` + + If the line is absent from the source, this article does not apply. + +2. On the node that reports OOM kills, read the kernel log and the container that was killed: + + ```bash + dmesg -T | grep -i -E 'out of memory|killed process' | tail -5 + kubectl -n cpaas-system describe pod | grep -E 'OOMKilled|Exit Code|Limits|Requests' + ``` + + `Memory cgroup out of memory` naming a `nevermore` process, together with `OOMKilled` on a `nevermore` container, is Cause 1. + +3. For the cache path, run `df -hT /var /cpaas` and `du -sh /var/cpaas` on the node, then read the `logagent` ModuleInfo `spec.valuesOverride` on the global cluster. A cache path under `/var/cpaas` is Cause 2. + +4. The `callback-secret` event with pods at `Init:0/1`, on a cluster without Log Essentials, is Cause 3. + +5. When the pod is up, read the agent log for enqueue or connection errors: + + ```bash + kubectl -n cpaas-system logs -c --tail=100 + ``` + +## Related Information + +- [nevermore Pending, Error with filebeat.lock, or OOMKilled after a disk queue backlog on ACP](./nevermore_Pending_Error_with_filebeat_lock_or_OOMKilled_after_a_disk_queue_backlog_on_ACP.md) +- [nevermore and ClickHouse log components alert on CPU or memory, restart, or ignore new resource values on ACP](./nevermore_and_ClickHouse_log_components_alert_on_CPU_or_memory_restart_or_ignore_new_resource_values_on_ACP.md) diff --git a/docs/en/solutions/nevermore_Pending_Error_with_filebeat_lock_or_OOMKilled_after_a_disk_queue_backlog_on_ACP.md b/docs/en/solutions/nevermore_Pending_Error_with_filebeat_lock_or_OOMKilled_after_a_disk_queue_backlog_on_ACP.md new file mode 100644 index 000000000..4a3c8a434 --- /dev/null +++ b/docs/en/solutions/nevermore_Pending_Error_with_filebeat_lock_or_OOMKilled_after_a_disk_queue_backlog_on_ACP.md @@ -0,0 +1,160 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.14.x + - 4.0.x + - 4.1.x + - 4.2.x + - 4.3.x +--- + +# nevermore Pending, Error with filebeat.lock, or OOMKilled after a disk queue backlog on ACP + +## Issue + +The `nevermore` log agent does not run on some nodes. Typical reports: + +- "The log collection component is abnormal", and `nevermore` pods are `Pending` for days. +- "`nevermore` is stuck": one pod is `3/4 Error` while the others are `4/4 Running`. +- A `nevermore` pod cannot start and its status is `OOMKilled` after the log store was down. + +`nevermore` is the per-node DaemonSet in `cpaas-system` that collects container logs, Kubernetes audit records and events. Its local queue is under `/cpaas` (`/cpaas/log`, `/cpaas/audit`, `/cpaas/event`), with segment files `*.seg` under, for example, `/cpaas/log/data/diskqueue`. Its own log is `/cpaas/log/logs/nevermore`. The container names differ between releases (for example `audit` and `log`, or `nevermore-audit`, `nevermore-event` and `nevermore-log`). Read them from the DaemonSet before you pass `-c`: + +```bash +kubectl -n cpaas-system get ds nevermore -o jsonpath='{.spec.template.spec.containers[*].name}' +``` + +When `nevermore` is `OOMKilled` without a queue backlog, writes its cache under `/var/cpaas`, or waits for the `log-collector-bearer-auth` Secret, see [nevermore OOMKilled, cache written to /var/cpaas, or callback-secret mount failure on ACP](./nevermore_OOMKilled_cache_written_to_var_cpaas_or_callback-secret_mount_failure_on_ACP.md). + +## Root Cause + +### Cause 1: nevermore cannot start because the node has too little memory + +`kubectl get ars -A` shows `logagent` as `Synced`, with health `WaitReady` and message `chart synced`. One or more `nevermore` pods are `0/4` `Pending`, and their events read: + +```text +Warning FailedScheduling default-scheduler 0/59 nodes are available: 1 Insufficient memory. preemption: 0/59 nodes are available: 59 No preemption victims found for incoming pod.. +``` + +A DaemonSet pod is bound to one node, so `1 Insufficient memory` refers to that node. `kubectl logs` on the pending pod prints only `Defaulted container ...`, because a pod that never started has no container log. The node is a small machine (for example 2 CPU / 4 GB) that cannot fit the `nevermore` requests next to its other pods. Giving the machine more memory lets the pod schedule. + +### Cause 2: nevermore is OOMKilled because its disk queue filled while Elasticsearch was down + +The `nevermore` pod cannot start, and its status is `OOMKilled`. Elasticsearch had been abnormal, so `nevermore` could not ship, and the collected data piled up in the local disk queue. `/cpaas/log/data/diskqueue` is large. Repairing Elasticsearch and then deleting the `*.seg` files under `/cpaas/log/data/diskqueue` returns `nevermore` to normal. Raising the limits alone does not empty this queue. + +The same cache family also has `/cpaas/audit/data/diskqueue` and `/cpaas/event/data/diskqueue`. Clean those only when `du` shows that they are large, and only the files inside `diskqueue`. Do not delete `/cpaas/log`, `/cpaas/audit` or `/cpaas/event` themselves. Cleaning `diskqueue` while Elasticsearch is still down frees disk space only until the queue refills. + +### Cause 3: An old nevermore log container stays on the node and blocks the new one + +One `nevermore` pod is `3/4 Error` with a few restarts, while the others are `4/4 Running`. `kubectl logs -p ` defaults to the audit container and answers `previous terminated container "audit" in pod "" not found`. The failing container is the log container, and its previous log is: + +```text +Exiting: /cpaas/log/data/filebeat.lock: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data) +``` + +On its node, `crictl ps -a | grep never` shows the new pod's log container `Exited` with several restarts. It also shows the log container of an old `nevermore` pod, created months earlier, still `Running`, while the old pod's other containers are `Exited`. The old container holds `/cpaas/log/data/filebeat.lock` and prevents the new one from starting. Deleting the old container fails, and killing its process by PID does not remove it either. Rebooting the node clears it, and `nevermore` recovers. Why the container could not be removed is not known. + +## Resolution + +### Cause 1: nevermore cannot start because the node has too little memory + +1. Confirm the signature: + + ```bash + kubectl get ars -A | grep logagent + kubectl get pod -A | grep nevermore | grep -v Run + kubectl -n cpaas-system describe pod | sed -n '/Events:/,$p' + kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.capacity.cpu,MEM:.status.capacity.memory + kubectl describe node | sed -n '/Allocated resources/,/Events/p' + ``` + + `0/N nodes are available: 1 Insufficient memory` on a `Pending` `nevermore` pod, on a node too small for its requests, matches this cause. + +2. Add memory (and CPU) to the node. For a VM, this usually needs the VM to be stopped. + + > **Warning:** Resizing the node VM restarts it, which stops every pod on that node. Precondition: the node is cordoned and drained (`kubectl cordon ` and `kubectl drain --ignore-daemonsets --delete-emptydir-data`). Rollback: resize the VM back; run `kubectl uncordon ` when it is up. + +3. After the node is back, run `kubectl uncordon `. The DaemonSet pod then schedules. + +### Cause 2: nevermore is OOMKilled because its disk queue filled while Elasticsearch was down + +1. Repair Elasticsearch first, or the queue refills. See [Elasticsearch log indices red or yellow with unassigned shards on ACP](./Elasticsearch_log_indices_red_or_yellow_with_unassigned_shards_on_ACP.md). + +2. On the node, check the queue size: + + ```bash + du -sh /cpaas/log/data/diskqueue /cpaas/audit/data/diskqueue /cpaas/event/data/diskqueue + ls /cpaas/log/data/diskqueue | head + ``` + +3. Delete the queued segment files and restart the pod. + + > **Warning:** Deleting `*.seg` files removes queued log records that were never shipped, so search has a permanent gap for that window. Precondition: only when `nevermore` is `OOMKilled`, the disk queue is large, and Elasticsearch has already been repaired. Do not delete the parent `/cpaas/log` tree. Rollback: none. + + ```bash + find /cpaas/log/data/diskqueue -name '*.seg' -delete + kubectl -n cpaas-system delete pod + ``` + + If `du` also showed a large audit or event `diskqueue`, the same gap applies when you delete the segment files there. + +### Cause 3: An old nevermore log container stays on the node and blocks the new one + +1. Confirm the signature: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep never + kubectl -n cpaas-system logs -c -p | grep 'filebeat.lock' + ``` + + On the node: + + ```bash + crictl ps -a | grep -i nevermore + crictl inspect | grep -E '"pid"|"state"' + ``` + + `data path already locked by another beat` in the new log container, together with a `Running` log container whose pod ID belongs to an old `nevermore` pod, matches this cause. + +2. Try to remove the old container with `crictl rm -f `. If that fails and killing its PID does not remove it, reboot the node. + + > **Warning:** Rebooting the node stops every pod on it. Precondition: only when an old `nevermore` log container remains, `crictl rm -f` and killing its PID both failed, and the node is cordoned and drained. Rollback: none needed; uncordon the node after the reboot. + + ```bash + kubectl cordon + kubectl drain --ignore-daemonsets --delete-emptydir-data + ``` + + On the node: + + ```bash + systemctl reboot + ``` + +3. After the node is back, run `kubectl uncordon `, and check that `nevermore` is `Running`. + +## Diagnostic Steps + +1. Read the pod state and the node: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep nevermore + kubectl -n cpaas-system describe pod | grep -E 'State|Reason|Exit Code|Limits' -A2 + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | `Pending` with `Insufficient memory` on a small node | Cause 1 | + | `OOMKilled` with a large `/cpaas/log/data/diskqueue` after Elasticsearch trouble | Cause 2, repair Elasticsearch before deleting `*.seg` | + | `filebeat.lock: data path already locked by another beat`, with an old container that cannot be removed | Cause 3 | + | `OOMKilled` with no queue backlog | See the related article on limit-driven OOM | + +## Related Information + +- [nevermore OOMKilled, cache written to /var/cpaas, or callback-secret mount failure on ACP](./nevermore_OOMKilled_cache_written_to_var_cpaas_or_callback-secret_mount_failure_on_ACP.md) +- [nevermore and ClickHouse log components alert on CPU or memory, restart, or ignore new resource values on ACP](./nevermore_and_ClickHouse_log_components_alert_on_CPU_or_memory_restart_or_ignore_new_resource_values_on_ACP.md) diff --git a/docs/en/solutions/nevermore_and_ClickHouse_log_components_alert_on_CPU_or_memory_restart_or_ignore_new_resource_values_on_ACP.md b/docs/en/solutions/nevermore_and_ClickHouse_log_components_alert_on_CPU_or_memory_restart_or_ignore_new_resource_values_on_ACP.md new file mode 100644 index 000000000..e829a6466 --- /dev/null +++ b/docs/en/solutions/nevermore_and_ClickHouse_log_components_alert_on_CPU_or_memory_restart_or_ignore_new_resource_values_on_ACP.md @@ -0,0 +1,249 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.3.x +--- + +# nevermore and ClickHouse log components alert on CPU or memory, restart, or ignore new resource values on ACP + +## Issue + +One of the following is reported for the log components: + +- One `nevermore` pod restarted abnormally. Only the log container on that node has high memory, and `nevermore` on every other node is fine. +- On a cluster with dozens of nodes, `nevermore` CPU on one node keeps alerting while the other nodes are normal. The node's CPU is high only at intervals. +- ClickHouse log storage keeps raising high CPU and memory alarms, and adding resources does not stop them. +- The ClickHouse log shows `Memory limit (total) exceeded` with `MEMORY_LIMIT_EXCEEDED`. Log queries may fail at the same time. +- On the global cluster, the ClickHouse StatefulSet runs with values different from the plugin configuration, and restarting `clickhouse-operator` changes nothing. +- The ClickHouse CPU alarm fires on a regular schedule (for example about once an hour, for about 10 minutes), or node load on the management cluster rises at the same time. +- After an upgrade, a new ClickHouse limit set in the console does not take effect. +- "What are the `nevermore` and `zeus` services for? They take 10 cores on a 64-core server. Can that be reduced?" + +`nevermore` is the node log collector and runs one pod per node. Its local buffer on the node is `/cpaas/log/data/diskqueue`. ClickHouse is the log store of Alauda Container Platform Log Storage for ClickHouse, managed by `clickhouse-operator` through a ClickHouseInstallation (`chi`). The ClickHouse log often also contains `RaftInstance ... failed to read rpc header from socket ... End of file` every few seconds. That line is benign and unrelated to CPU or memory. + +## Root Cause + +### Cause 1: The nevermore log container is OOMKilled on one node after a short memory surge + +Pod OOM events rise on one node (for example 6 in 30 minutes) and stay at 0 on the others. The `nevermore` log container on that node has a 4G limit. Its usual memory use is a few percent to just over 10%, but a short surge passes 4G and the container is killed. After the log container memory was raised to 5G, it recovered and usage returned below 10%. + +### Cause 2: A nevermore CPU alert on one node comes from bursts of business logs + +Business logs on that node are written in concentrated bursts. Log file rotation, compression and batch sending then push `nevermore` CPU up for a while, and the alert fires. If the node's local log directory or disk is slow, `nevermore` builds a backlog and processes it all at once, which makes the spike sharper. `/cpaas/log/data/diskqueue` grows during the alert. With a 2-core CPU limit on the log container, the recommendation is 4 cores. The queued data is compressed, so the business log that caused the burst cannot always be identified. Such an alert can clear without further change. + +### Cause 3: ClickHouse CPU and memory alarms persist after expansion (known issue) + +ClickHouse log storage keeps raising CPU and memory alarms, and more resources do not help. This is a known issue in the log component. It is fixed in `log-4.3.2` (ACP 4.3.2) and not in 4.3.1. + +### Cause 4: MEMORY_LIMIT_EXCEEDED in MergeTreeBackgroundExecutor or in log queries + +The ClickHouse log shows: + +```text +Memory limit (total) exceeded: would use 3.64 GiB (attempt to allocate chunk of 4361312 bytes), maximum: 3.60 GiB +``` + +with `MEMORY_LIMIT_EXCEEDED` on `MergeTreeBackgroundExecutor`. The `maximum` is close to the 4Gi container limit: background merges, and queries, need more memory than the ClickHouse limit allows. The console can then fail to show logs, and the `vector` sinks retry their inserts. Raising ClickHouse from 2 CPU / 4Gi to 4 CPU / 8Gi removes the error. An `Address already in use` line for `0.0.0.0:9004` in the same log was not related. + +### Cause 5: The ClickHouse StatefulSet does not follow the configured resources + +The plugin and the `chi` both say 2 CPU / 4G, and no resource patch (`rpch`) exists. However, the StatefulSet runs two replicas at 3 CPU / 6G and one at 4 CPU / 6G. Restarting `clickhouse-operator` does not update the StatefulSet or its pods. Saving the resources on the plugin again triggers the update, and the StatefulSet returns to the configured values. + +When the global cluster nodes are too small for ClickHouse (8 CPU per node, with ClickHouse alone driving node CPU high), the lasting change is to move ClickHouse off the global cluster. Store the global cluster's audit data in a business cluster's log storage instead. + +### Cause 6: Periodic CPU spikes from MergeTree background merges + +The CPU alarm repeats at regular intervals, or CPU spikes about every hour and raises node load on the management cluster. ClickHouse does not restart, and its log during a spike looks the same as at other times. The pod graph shows CPU at 90% to 100% at regular intervals, often as two adjacent peaks at the limit, with 10% to 20% in between. On the node, `top` puts `clickhouse-serv` far above every other process during a spike. + +The MergeTree background merge periodically combines small data parts of a partition into larger ones and applies TTL deletes. With high merge-thread concurrency, it takes a lot of CPU for a short time. Raising the limit does not remove the spike: after a 4-core limit was raised to 6 cores, the merges used all 6. This is a known issue, and an optimization has been requested. It does not affect normal ClickHouse operation. A slow `/cpaas` disk makes merges last longer. + +### Cause 7: A resource change in the console does not apply because a resource patch already exists + +An earlier backend change to the ClickHouse resources created a resource patch (`rpch`). While that `rpch` exists, resource changes made in the console do not reach the ClickHouse StatefulSet. The patch targets the ClickHouseInstallation, not the StatefulSet, and appears like this: + +```text +rp- Active cpaas-system/logclickhouse clickhouse.altinity.com/v1 ClickHouseInstallation cpaas-system cpaas-clickhouse +``` + +This is the opposite situation to Cause 5, where no `rpch` exists. + +### Cause 8: nevermore and zeus appear to take 10 CPU cores + +`nevermore` is the log collection component. `zeus` reads the certificate information of the Kubernetes components on each node, generates Secrets from it, and creates the matching Certificate resources so that those certificates can be monitored. + +The "10 cores" is the sum of the containers' CPU **limits**, not their requests and not their actual usage. Pod scheduling counts only requests against the node's allocatable CPU, and the sum of limits on a node may exceed the node's physical CPU. The figure therefore takes nothing away from scheduling capacity. + +## Resolution + +### Cause 1: The nevermore log container is OOMKilled on one node after a short memory surge + +1. Read the containers and their limits: + + ```bash + kubectl -n cpaas-system get ds nevermore -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" "}{.resources.limits}{"\n"}{end}' + ``` + +2. Raise the log container memory limit. + + > **Warning:** Changing the DaemonSet restarts `nevermore` on every node, one at a time, and log collection on a node pauses while its pod restarts. Precondition: the OOM events are on the `nevermore` log container, and each node has the extra memory free. Rollback: set the previous limit with the same command. + + ```bash + kubectl -n cpaas-system set resources ds/nevermore -c --limits=memory=5Gi + ``` + +3. If the log collection plugin renders `nevermore` with its own resource values, set the same limit there. After the next plugin update, check that the DaemonSet still has the new limit. + +### Cause 2: A nevermore CPU alert on one node comes from bursts of business logs + +1. Confirm that the CPU on that node's `nevermore` is intermittent and lines up with business log bursts or `diskqueue` growth (Diagnostic Steps 1 and 2). + +2. Raise the log container CPU limit so the peaks are absorbed. A CPU limit does not reserve resources up front. + + > **Warning:** This restarts `nevermore` on every node, as in Cause 1. Precondition: the nodes have CPU headroom at peak. Rollback: set the CPU limit back to 2. + + ```bash + kubectl -n cpaas-system set resources ds/nevermore -c --limits=cpu=4 + ``` + +3. Compare the node's business log volume with other nodes, and check the write speed of the local log disk. A slow disk keeps the backlog pattern even with a higher limit. + +### Cause 3: ClickHouse CPU and memory alarms persist after expansion (known issue) + +Upgrade the log components to `log-4.3.2` (ACP 4.3.2), using the package provided by Alauda support. More CPU and memory do not fix this cause. A ClickHouse that is still at 2 CPU / 4Gi and logs `MEMORY_LIMIT_EXCEEDED` needs Cause 4 first. + +### Cause 4: MEMORY_LIMIT_EXCEEDED in MergeTreeBackgroundExecutor or in log queries + +1. Read the live requests and limits of every replica: + + ```bash + kubectl -n cpaas-system get pods | grep chi-cpaas-clickhouse + kubectl -n cpaas-system get pod -o jsonpath='{range .spec.containers[*]}{.name}{" req="}{.resources.requests.cpu}{"/"}{.resources.requests.memory}{" lim="}{.resources.limits.cpu}{"/"}{.resources.limits.memory}{"\n"}{end}' + ``` + + Continue only when the `maximum` in the error is close to the current memory limit (for example 3.60 GiB at 4Gi). + +2. Raise the resources. + + > **Warning:** Saving new resources on the plugin rolls the ClickHouse pods, and log writes and queries are degraded until every replica is Ready again. Precondition: the nodes that run ClickHouse have 4 CPU and 8Gi free for each replica. Rollback: set the plugin back to the previous values and save. + + In the console, open the Alauda Container Platform Log Storage for ClickHouse plugin on the affected cluster. Set CPU to 4 and memory to 8Gi, and save. + +3. When the pods are Ready, read the limits again. If they still show the old values, check for a resource patch or a StatefulSet that does not follow the plugin (Causes 5 and 7). + +### Cause 5: The ClickHouse StatefulSet does not follow the configured resources + +1. Confirm the drift (Diagnostic Step 4): the StatefulSet differs from both the plugin and the `chi`, and no `rpch` exists for it. + +2. Re-apply the resources from the plugin. + + > **Warning:** Re-applying the resources rolls the ClickHouse pods, and log writes and queries are degraded until all replicas are Ready. Precondition: the drift is confirmed. Rollback: none needed; this applies the values the plugin already holds. + + Open Alauda Container Platform Log Storage for ClickHouse in the console, change the CPU and memory values and save, so that an update is triggered. Check that the StatefulSet now matches. Restarting `clickhouse-operator` alone does not do this. + +3. If the global nodes are too small to host ClickHouse, move it off the global cluster. + + > **Warning:** Uninstalling ClickHouse log storage from the global cluster deletes the logs stored there, which is irreversible. Precondition: a business cluster's log storage is set up to receive the global cluster's audit data, and the owner accepts losing the stored global logs. Rollback: reinstall the plugin on the global cluster; the old data does not return. + + Uninstall the plugin from the global cluster in the console. Then confirm that new global audit records arrive in the business cluster's log storage. + +### Cause 6: Periodic CPU spikes from MergeTree background merges + +1. Confirm that the spikes line up with merges (Diagnostic Step 5). + +2. Make sure the CPU limit leaves the node headroom during a spike. Raising the limit does not remove the spike. + +3. To stop frequent alerts in the meantime, lengthen the trigger duration of the ClickHouse CPU alert rule beyond the length of one merge spike. Edit the rule in the alert policy that holds it, and note the old duration so that you can set it back. + +4. If the log components are older than `log-4.3.2` and the alarms persist after expansion, also apply Cause 3. + +### Cause 7: A resource change in the console does not apply because a resource patch already exists + +1. Confirm that a patch exists for ClickHouse: + + ```bash + kubectl get rpch -A | grep -i clickhouse + ``` + +2. Set the resources on the StatefulSet. + + > **Warning:** Editing the StatefulSet rolls the ClickHouse pods, and log writes and queries are degraded until all replicas are Ready. Precondition: the `rpch` exists, you saved the StatefulSet YAML, and the nodes have the capacity. Rollback: `kubectl apply -f /tmp/clickhouse-sts.before.yaml`. + + ```bash + kubectl -n cpaas-system get sts -o yaml > /tmp/clickhouse-sts.before.yaml + kubectl -n cpaas-system edit sts + ``` + + Set the container resources there. Console changes stay blocked while the `rpch` exists. + +### Cause 8: nevermore and zeus appear to take 10 CPU cores + +No reduction is needed. The 10 cores are the combined CPU limits of `nevermore` and `zeus`. Limits do not reserve CPU and do not reduce what the scheduler can place on the node. The only condition is that the actual CPU usage of all containers on the node stays below the node's total CPU. Show the three numbers side by side: requests, limits and live usage (Diagnostic Step 6). + +## Diagnostic Steps + +1. For `nevermore`, find the node, the container and the reason: + + ```bash + kubectl -n cpaas-system get pods -o wide | grep nevermore + kubectl -n cpaas-system get pod -o jsonpath='{range .status.containerStatuses[*]}{.name}{" restarts="}{.restartCount}{" last="}{.lastState.terminated.reason}{"\n"}{end}' + kubectl -n cpaas-system top pod --containers | grep nevermore + ``` + + `OOMKilled` on the log container of one node is Cause 1. CPU near the 2-core limit on one node, without restarts, is Cause 2. + +2. On the alerting node, watch the disk queue during an alert: + + ```bash + du -sh /cpaas/log/data/diskqueue + top -b -n 1 | head -20 + ``` + + Growth during the alert and a quiet queue outside it support Cause 2. + +3. Check the ClickHouse version and log lines: + + ```bash + kubectl -n cpaas-system logs -c clickhouse --tail=500 | grep -E 'MEMORY_LIMIT_EXCEEDED|Memory limit' + ``` + + Alarms that stay after expansion on log components older than 4.3.2 are Cause 3. `MEMORY_LIMIT_EXCEEDED` with a `maximum` near the limit is Cause 4. + +4. Compare the plugin, the `chi` and the StatefulSet: + + ```bash + kubectl -n cpaas-system get chi cpaas-clickhouse -o yaml | grep -n -A6 resources + kubectl -n cpaas-system get sts | grep -i clickhouse + kubectl -n cpaas-system get sts -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" "}{.resources}{"\n"}{end}' + kubectl get rpch -A 2>/dev/null | grep -i clickhouse || true + ``` + + `chi` values that differ from the StatefulSet with no `rpch` are Cause 5. A console change that does not apply while an `rpch` for ClickHouse exists is Cause 7. + +5. During a periodic CPU spike, list the running merges: + + ```bash + kubectl -n cpaas-system exec -c clickhouse -- clickhouse-client \ + --query "SELECT database, table, elapsed, progress, num_parts FROM system.merges" + ``` + + Active merges during each spike, with an unchanged log and no restart, are Cause 6. + +6. For CPU attributed to `nevermore` and `zeus`, compare requests, limits and usage on one node: + + ```bash + kubectl get pods -A -o wide --field-selector spec.nodeName= | grep -E 'nevermore|zeus' + kubectl -n get pod -o jsonpath='{range .spec.containers[*]}{.name}{" req="}{.resources.requests.cpu}{" lim="}{.resources.limits.cpu}{"\n"}{end}' + kubectl -n top pod --containers + kubectl describe node | sed -n '/Allocated resources/,/Events/p' + ``` + + A large total under `lim=` with small requests and low `top` values is Cause 8. In `Allocated resources`, the `Limits` column may be above 100%. The `Requests` column is what scheduling uses. + +## Related Information + +- [ClickHouse log storage logs RaftInstance errors with pod restarts, merge CPU spikes, or a hand-built ClickHouse on ACP](./ClickHouse_log_storage_logs_RaftInstance_errors_with_pod_restarts_merge_CPU_spikes_or_a_hand-built_ClickHouse_on_ACP.md), for the benign `RaftInstance ... End of file` line. +- [Logging ClickHouse runs out of disk, CPU or memory, or is OOMKilled on ACP](./Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md) +- [nevermore OOMKilled, cache written to /var/cpaas, or callback-secret mount failure on ACP](./nevermore_OOMKilled_cache_written_to_var_cpaas_or_callback-secret_mount_failure_on_ACP.md) diff --git a/docs/en/solutions/razor_crash-loops_or_audit_logs_stop_TOO_MANY_PARTS_BufferTooSmall_vector_OOMKilled_or_stale_ClickHouse_token_on_ACP.md b/docs/en/solutions/razor_crash-loops_or_audit_logs_stop_TOO_MANY_PARTS_BufferTooSmall_vector_OOMKilled_or_stale_ClickHouse_token_on_ACP.md new file mode 100644 index 000000000..5fd4ffaa9 --- /dev/null +++ b/docs/en/solutions/razor_crash-loops_or_audit_logs_stop_TOO_MANY_PARTS_BufferTooSmall_vector_OOMKilled_or_stale_ClickHouse_token_on_ACP.md @@ -0,0 +1,242 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# razor crash-loops or audit logs stop: TOO_MANY_PARTS, BufferTooSmall, vector OOMKilled, or stale ClickHouse token on ACP + +## Issue + +The log storage is ClickHouse, and one of these is seen on the cluster that runs it: + +- Audit search is empty, or has nothing for the last day, while application logs or events still arrive. The `vector` container of `razor` in `cpaas-system` logs `Too many partitions for single INSERT block (more than 100)` or `Code: 252 ... (TOO_MANY_PARTS)`. +- `razor-0` is `1/2` or `CrashLoopBackOff`, and `vector` logs `failed to encode record: BufferTooSmall`. +- `razor` is `CrashLoopBackOff`, its probes fail with `dial tcp :8081: connect: connection refused`, and the log query page fails with a ClickHouse `500` error while the ClickHouse pods log no errors. +- `razor` logs `Failed to create project table` and ClickHouse reports `SYNTAX_ERROR` at `'-'`. +- After an upgrade from 4.3.1 to 4.3.2, ClickHouse looks empty although no table was dropped. +- Logs, events, and audit all stop on the same day while every pod is `Running`, and `nevermore` logs timeouts to `razor`. +- ClickHouse log storage was just deployed, the log module shows nothing, and ClickHouse has no new rows. + +`razor` runs two containers: `razor` and `vector`. Vector writes to ClickHouse and keeps a persistent disk buffer. Readiness, liveness, and startup probes use TCP port `8081`, which is served only while `vector` is running, so any `vector` crash makes the pod fail its probes. `razor` is a StatefulSet on most installs; check with `kubectl -n cpaas-system get deploy,sts | grep razor`. + +## Root Cause + +### Cause 1: inserts into observability.audit_all exceed max_partitions_per_insert_block + +The ClickHouse tables are partitioned by day, and ClickHouse rejects an INSERT that touches more than `max_partitions_per_insert_block` partitions (default 100). A replay of historical audit data, or a very large Vector batch, puts more than 100 days into one INSERT. Vector treats the rejection as transient and retries the same batch, the disk buffer fills (about 2.5 GB in one case), and real-time audit stops. A restart alone does not clear it, because the buffer is on disk. `event_all` can stay current, which shows that ClickHouse, the network, and `razor` are reachable and only the audit sink is stuck. + +The same limit can stop logs, events, and audit together: no data after a given date in ClickHouse, all pods `Running`, `vector` logging on the `clickhouse` sink with `component_id=audit`: + +```text +Code: 252. DB::Exception: Too many partitions for single INSERT block (more than 100). The limit is controlled by 'max_partitions_per_insert_block' setting. ... (TOO_MANY_PARTS) +``` + +and the `http` source (`path=/v4/callback/logs`) reporting `component_events_dropped` with `reason="Source send cancelled."`. Reducing the Vector batch size (or raising the partition limit for the duration of the replay) lets inserts succeed again. `system.errors` counts the `TOO_MANY_PARTS` failures (tens of thousands in one case). + +### Cause 2: vector exits on BufferTooSmall, and the razor container then sits at its CPU limit + +The `razor` container is running and `vector` is not. Vector logs `Disk buffer writer has encountered an unrecoverable error` and `failed to encode record: BufferTooSmall`, then stops, and the probes on `8081` are refused. PVC `data-razor-0` is not full and the node filesystem has space. `razor` sent Vector a batch that was too large to encode. When the crash loop slows, the `razor` container CPU can sit at 100% of a 1-core limit; restarting does not change it, and raising the limit to 2 cores leaves it running at about 1.5 cores with no errors. + +### Cause 3: a project name containing a hyphen makes ClickHouse reject the project table + +`razor` creates a per-project table for a project-level retention policy. When the project name contains a hyphen, the statement (` ON CLUSTER replicated`) fails with `SYNTAX_ERROR` at `'-'`, and `razor` logs `Failed to create project table` repeatedly. The shared tables in database `observability` (`audit`, `event`, `log_kubernetes`, `log_platform`, `log_system`, `log_workload`, `meter_detail`, `meter_summary`, `schema_migrations`) are not affected. + +### Cause 4: after the upgrade from 4.3.1 to 4.3.2, razor and nevermore keep a stale ClickHouse token + +ClickHouse looks empty after the upgrade, but no rows were deleted. `razor` and `nevermore` keep ClickHouse authentication that the upgrade did not refresh. This is a known defect on that upgrade path. Forcing the token in Secret `log-database-bearer-auth` to expire and restarting `razor-api` and `nevermore` makes the data visible again. + +Log queries belong on the primary global cluster; a standby global cluster whose log feature points at a domain name is not a log query endpoint. + +### Cause 5: nevermore upload requests to razor time out although razor is reachable + +`nevermore` logs: + +```text +Failed to publish events ... Post "http://razor:8080/v4/callback/logs": context deadline exceeded (Client.Timeout exceeded while awaiting headers) +``` + +A manual request to the same URL from a `nevermore` pod returns `Unauthorized with error:authorization header is empty`, so the endpoint answers. Each upload batch is so large that the request times out. Lowering the upload `batch_size` from 1000 to 100 restores the connection. This can come together with Cause 1. + +### Cause 6: nevermore requests to the Kubernetes API time out, and then vector is OOMKilled + +On a newly deployed ClickHouse log storage, the log module shows nothing, a direct ClickHouse query finds no new rows, and the query page returns no error, so the break is before ClickHouse. `nevermore` requests to the Kubernetes API are timing out. Restarting `nevermore` clears that, and the next hop then fails: the `vector` container in `razor` is `OOMKilled`. Both fixes are needed: restart `nevermore`, then raise the `vector` memory limit. + +### Cause 7: vector is OOMKilled at its memory limit, razor crash-loops, and log queries return a ClickHouse 500 error + +The `vector` container reaches its memory limit (`memory: 1Gi`, with a request of `256Mi`, in one case) and is killed. `kubectl describe pod` shows `State: Waiting`, `Reason: CrashLoopBackOff`, `Last State: Terminated`, `Reason: OOMKilled`, and a growing restart count. Nothing listens on `8081`, so `razor` logs `dial tcp :8081: connect: connection refused`, the pod never becomes ready, and the log query page fails with a ClickHouse `500` error. The ClickHouse pods themselves are healthy and log no errors. Raising the `vector` memory limit restores log queries. + +## Resolution + +### Cause 1: inserts into observability.audit_all exceed max_partitions_per_insert_block + +1. Confirm the error and read the Vector configuration: + + ```bash + kubectl -n cpaas-system logs razor-0 -c vector --tail=1500 | grep -iE 'audit|Code:|TOO_MANY|partition|dropped' -C3 + kubectl -n cpaas-system get secret cpaas-vector -o yaml > /tmp/cpaas-vector.before.yaml + kubectl -n cpaas-system get secret cpaas-vector -o jsonpath='{.data.vector\.yaml}' | base64 -d > /tmp/vector.yaml + grep -n max_events /tmp/vector.yaml + ``` + + Repeat the log command for `razor-1` if it exists. + +2. Edit `/tmp/vector.yaml` so the audit sink has `sinks.audit.batch.max_events: 50`. If the file has only one `batch.max_events` (for example `10000000`), set it to `99` until the backlog is written, then restore the saved value. + + > **Warning:** Patching Secret `cpaas-vector` and restarting `razor` reloads Vector; ingestion through `razor` pauses until the pod is Ready. Precondition: only when `Too many partitions for single INSERT block (more than 100)` or `TOO_MANY_PARTS` is confirmed. Rollback: `kubectl apply -f /tmp/cpaas-vector.before.yaml` and restart `razor` again. + + ```bash + kubectl -n cpaas-system patch secret cpaas-vector --type merge \ + -p "{\"data\":{\"vector.yaml\":\"$(base64 -w0 /tmp/vector.yaml)\"}}" + kubectl -n cpaas-system rollout restart sts/razor + kubectl -n cpaas-system rollout status sts/razor + ``` + +3. If the stuck batch still fails, raise the ClickHouse `max_partitions_per_insert_block` setting above 100 while the backlog drains, and set it back once `audit_all` receives current timestamps. Read the current value first: + + ```bash + kubectl -n cpaas-system exec -c -- \ + clickhouse-client --query "SELECT value FROM system.settings WHERE name = 'max_partitions_per_insert_block'" + ``` + +4. Query audit for the current hour. A gap matching the time the sink was stuck can remain. + +### Cause 2: vector exits on BufferTooSmall, and the razor container then sits at its CPU limit + +1. Confirm which container stops and why: + + ```bash + kubectl -n cpaas-system get pod razor-0 -o jsonpath='{range .status.containerStatuses[*]}{.name}{" restarts="}{.restartCount}{" last="}{.lastState.terminated.reason}{"\n"}{end}' + kubectl -n cpaas-system logs razor-0 -c vector --previous --tail=80 + ``` + +2. If `razor` also logs `Failed to create project table`, fix Cause 3 first. +3. If the `razor` container stays at its CPU limit, raise it. + + > **Warning:** Changing the `razor` container resources rolls the workload; ingestion and queries pause during the roll. Precondition: only when `BufferTooSmall` is confirmed and the `razor` container is at its CPU limit, and the node has a free core. Rollback: set the previous limit again. If a plugin update puts the old value back, set it again the same way. + + ```bash + kubectl -n cpaas-system set resources sts/razor -c razor --limits=cpu=2 + ``` + +### Cause 3: a project name containing a hyphen makes ClickHouse reject the project table + +```bash +kubectl -n cpaas-system logs -c razor --tail=80 | grep -E 'Failed to create project table|SYNTAX_ERROR' +``` + +> **Warning:** Deleting the retention policy removes that project's retention rule; its logs follow the default retention afterwards. Precondition: only when the failing statement names a project that contains a hyphen. Rollback: none for the deleted policy. + +Delete that project's log retention policy in the console. The `Failed to create project table` line stops. If CPU stays high afterwards, apply Cause 2. + +### Cause 4: after the upgrade from 4.3.1 to 4.3.2, razor and nevermore keep a stale ClickHouse token + +> **Warning:** Expiring the token and restarting `razor-api` and `nevermore` pauses log queries and collection during the restarts. Precondition: only when ClickHouse looks empty after the upgrade from 4.3.1 to 4.3.2. Rollback: restore the saved Secret and restart the same two workloads. + +```bash +kubectl -n cpaas-system get secret log-database-bearer-auth -o yaml > /tmp/log-database-bearer-auth.before.yaml +kubectl -n cpaas-system annotate secret log-database-bearer-auth \ + razor-api/token-expires-at="2000-01-01T00:00:00Z" --overwrite +kubectl -n cpaas-system rollout restart deploy/razor-api +kubectl -n cpaas-system rollout restart ds/nevermore +kubectl -n cpaas-system rollout status deploy/razor-api +kubectl -n cpaas-system rollout status ds/nevermore +``` + +Refresh the log page after both are Ready. + +### Cause 5: nevermore upload requests to razor time out although razor is reachable + +```bash +kubectl -n cpaas-system logs --all-containers --tail=200 | grep -iE 'razor|timeout|Failed to publish' +kubectl -n cpaas-system exec -c log -- curl -sS http://razor:8080/v4/callback/logs +kubectl -n cpaas-system get cm nevermore-config -o yaml > /tmp/nevermore-config.before.yaml +grep -n batch_size /tmp/nevermore-config.before.yaml +``` + +An `Unauthorized` answer means `razor` is reachable. If the container has no `curl`, run the request from another pod in `cpaas-system`. + +> **Warning:** Editing `nevermore-config` and restarting `nevermore` pauses collection on each node while its pod restarts; a batch that is too small slows shipping on busy nodes. Precondition: only when `nevermore` logs upload timeouts and a manual request to `razor` is answered. Rollback: `kubectl apply -f /tmp/nevermore-config.before.yaml` and restart `nevermore` again. + +```bash +kubectl -n cpaas-system edit cm nevermore-config +kubectl -n cpaas-system rollout restart ds/nevermore +``` + +Change the upload `batch_size` from `1000` to `100`, then watch the timeouts stop and current rows arrive in ClickHouse. + +### Cause 6: nevermore requests to the Kubernetes API time out, and then vector is OOMKilled + +1. Confirm that nothing new reaches ClickHouse (use `DESCRIBE TABLE observability.log_workload` if the time column has another name): + + ```bash + kubectl -n cpaas-system exec -c -- \ + clickhouse-client --query "SELECT max(timestamp) FROM observability.log_workload" + kubectl -n cpaas-system logs --all-containers --tail=300 | grep -iE 'timeout|deadline|apiserver|443' + ``` + +2. Restart `nevermore`. + + > **Warning:** Restarting `nevermore` pauses log, audit, and event collection on each node while its pod restarts. Precondition: only when ClickHouse has no new rows and `nevermore` logs Kubernetes API timeouts. Rollback: none needed; the DaemonSet recreates the pods. + + ```bash + kubectl -n cpaas-system rollout restart ds/nevermore + kubectl -n cpaas-system rollout status ds/nevermore + ``` + +3. If `vector` is then `OOMKilled`, raise its memory as in Cause 7. + +### Cause 7: vector is OOMKilled at its memory limit, razor crash-loops, and log queries return a ClickHouse 500 error + +1. Read the last termination reason and the current limit of each container: + + ```bash + for p in $(kubectl -n cpaas-system get pod -o name | grep razor); do + echo "== $p" + kubectl -n cpaas-system get "$p" -o jsonpath='{range .status.containerStatuses[*]}{.name}{" restarts="}{.restartCount}{" last="}{.lastState.terminated.reason}{"\n"}{end}' + done + kubectl -n cpaas-system get sts razor -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{" "}{.resources.limits.memory}{"\n"}{end}' + ``` + +2. Raise the memory limit of the `vector` container. + + > **Warning:** Editing the `razor` workload restarts its pods; log ingestion and queries pause until they are Ready. Precondition: only when the `vector` container's last state is `OOMKilled`, and the node has the memory for the new limit. Rollback: set the previous memory limit again. + + ```bash + kubectl -n cpaas-system edit sts razor + ``` + + In the `vector` container, raise `resources.limits.memory` (one recovery used `4Gi`; size it from observed usage and the node's free memory). In that recovery the edit generated a resource patch (`rpch`) that kept the new value through later reconciles; check that the value is still in place after the next plugin update. + +3. When `vector` stops restarting and the pod is Ready, run the log query again. + +## Diagnostic Steps + +1. Identify which container fails and what it logs: + + ```bash + kubectl -n cpaas-system get pods | grep -E 'razor|nevermore' + kubectl -n cpaas-system describe pod razor-0 | sed -n '/Containers:/,/Conditions:/p' + kubectl -n cpaas-system logs razor-0 -c vector --previous --tail=60 + kubectl -n cpaas-system logs razor-0 -c razor --tail=40 + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | `TOO_MANY_PARTS` or `more than 100` on the ClickHouse sink | 1 | + | `BufferTooSmall` in the previous `vector` log | 2 | + | `Failed to create project table` with `SYNTAX_ERROR` at `'-'` | 3 | + | ClickHouse looks empty right after the 4.3.1 to 4.3.2 upgrade | 4 | + | `nevermore` times out on `/v4/callback/logs`, manual request answered | 5 | + | No new ClickHouse rows and `nevermore` times out against the Kubernetes API | 6 | + | `vector` last state `OOMKilled`, `8081` refused, query page shows ClickHouse `500` | 7 | + +3. A `connection refused` on `8081` only says that `vector` is down; always read the previous `vector` log and the last termination reason before changing anything. A ClickHouse pod that is itself `OOMKilled` or exits with `137` is a ClickHouse capacity problem, see [Logging ClickHouse runs out of disk, CPU or memory, or is OOMKilled on ACP](Logging_ClickHouse_runs_out_of_disk_CPU_or_memory_or_is_OOMKilled_on_ACP.md). + +Related articles: [razor does not become ready after a ClickHouse log storage install or upgrade on ACP](razor_does_not_become_ready_after_a_ClickHouse_log_storage_install_or_upgrade_on_ACP.md), [nevermore OOMKilled, cache written to /var/cpaas, or callback-secret mount failure on ACP](nevermore_OOMKilled_cache_written_to_var_cpaas_or_callback-secret_mount_failure_on_ACP.md). diff --git a/docs/en/solutions/razor_does_not_become_ready_after_a_ClickHouse_log_storage_install_or_upgrade_on_ACP.md b/docs/en/solutions/razor_does_not_become_ready_after_a_ClickHouse_log_storage_install_or_upgrade_on_ACP.md new file mode 100644 index 000000000..bb0337245 --- /dev/null +++ b/docs/en/solutions/razor_does_not_become_ready_after_a_ClickHouse_log_storage_install_or_upgrade_on_ACP.md @@ -0,0 +1,225 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 4.2.x + - 4.3.x +--- + +# razor does not become ready after a ClickHouse log storage install or upgrade on ACP + +## Issue + +Alauda Container Platform Log Storage for ClickHouse is installed, reinstalled or upgraded. ClickHouse itself runs, but `razor` never becomes ready. `razor` is the gateway in front of the logging ClickHouse, StatefulSet `razor` in `cpaas-system`. `vector` is a container inside the `razor` pod. Typical reports: + +- "`razor` will not start after the install." +- "The `vector` container fails with a read-only file system." +- "`razor` stays in `Error` after the platform upgrade." +- "`razor-0` is `1/2 Running` and restarting on a new deploy." + +The `razor` log may end with: + +```text +Try to do recover from dirty +Failed to prepare {"error": "migration aborted after recoverd"} +``` + +The default data paths are `/cpaas/data/clickhouse` for ClickHouse and `/cpaas/data/vector` for the `vector` buffer. On an immutable OS the writable root for platform data is `/var/cpaas` instead of `/cpaas`. + +When ClickHouse itself never starts, see [ClickHouse log storage plugin install fails or ClickHouse does not start on ACP](./ClickHouse_log_storage_plugin_install_fails_or_ClickHouse_does_not_start_on_ACP.md). + +## Root Cause + +### Cause 1: On an immutable OS, razor fails with a read-only file system or never starts because the vector path lacks /var + +Two variants share this cause: + +- The install from the console wrote the ClickHouse path as `/var/cpaas/data/clickhouse`, but passed `/cpaas/data/vector` to `vector`, without the `/var` prefix. `razor` cannot start. Editing the ClickHouse ModuleInfo in place to `/var/cpaas/data/vector` does not refresh the path of the existing `razor` PVC and PV. Recreating the ModuleInfo with the corrected path does. +- After a disk replacement moved the writable path from `/cpaas` to `/var/cpaas`, the `vector` container fails with `Configuration error` on every sink (`audit`, `event`, `log kubernetes`, `log platform`, `log system`, `log workload`): + + ```text + error occurred when building buffer: failed to build individual stage 0: failed to load/create ledger: ledger I/O error: Read-only file system (os error 30) + run "/usr/local/bin/vector" error: exit status 78 + ``` + + `mount | grep cpaas` on the host shows the disks mounted `rw` on `/var/cpaas`, so the path `vector` was given is not the writable one. The `Transform "dispatcher.unmatched" has no consumers` warning in the same log is not the failure. + +### Cause 2: razor stays in Error after an upgrade because migration_versions_new has a dirty record + +After a platform upgrade, the ClickHouse component is installed and `razor` never starts. Its log shows the migration check, an automatic recovery attempt, and the abort: + +```text +clickhouse/store_setup.go:94 | server | Try to do recover from dirty +server/options.go:169 | server | Failed to prepare {"error": "migration aborted after recoverd"} +``` + +The latest record in the ClickHouse table `migration_versions_new` is marked dirty. `razor` cannot recover from it. Emptying the ClickHouse data directory with `razor` and every ClickHouse replica stopped, then starting them again, lets `razor` start. That discards all stored ClickHouse data. + +### Cause 3: razor fails with migration aborted after recoverd on a new deploy because one replica is missing tables + +While the ClickHouse log storage is being deployed, `razor-0` is `1/2 Running` and restarting. Its log shows `curVersion:10 dirty:false` and `run migration error:no change` against one replica (`chi-cpaas-clickhouse-replicated-0-0:8123`). Against another replica (`chi-cpaas-clickhouse-replicated-0-1:8123`) it shows: + +```text +Try to do recover from dirty +Failed to prepare {"error": "migration aborted after recoverd"} +Failed to apply to server {"error": "migration aborted after recoverd"} +``` + +A later log shows the migration statement and the real error: + +```text +CREATE TABLE IF NOT EXISTS audit_all ON CLUSTER replicated AS audit ENGINE = Distributed(replicated, observability, audit, rand()); +Code: 60. DB::Exception: Table observability.audit does not exist. (UNKNOWN_TABLE) +``` + +ClickHouse had been uninstalled and reinstalled with new PVCs and PVs. The database and table initialization had not completed on one replica, so tables of the `observability` database (such as `audit`) were missing there. The migration against that replica failed and then aborted. This is a known ClickHouse log storage issue. The fix is to create the missing tables on that replica from the definitions on a healthy replica. Unlike Cause 2, the migration record here is not dirty. + +## Resolution + +### Cause 1: On an immutable OS, razor fails with a read-only file system or never starts because the vector path lacks /var + +1. Confirm the signature. On the global cluster: + + ```bash + kubectl get moduleinfo -A | grep -i clickhouse + kubectl get moduleinfo -o yaml | grep -n -E '/cpaas/data|/var/cpaas' + ``` + + On the affected cluster: + + ```bash + kubectl -n cpaas-system get pv | grep razor + kubectl -n cpaas-system logs razor-1 -c vector --tail=50 | grep -E 'os error 30|exit status' + ``` + + On the node, run `mount | grep cpaas`. This cause matches when the ClickHouse path is under `/var/cpaas` and the `vector` path under `/cpaas` in the same ModuleInfo. It also matches when `vector` reports `Read-only file system (os error 30)` while the host mount is `rw`. + +2. Save the ModuleInfo and prepare a corrected copy: + + ```bash + kubectl get moduleinfo -o yaml > /tmp/clickhouse-minfo.backup.yaml + cp /tmp/clickhouse-minfo.backup.yaml /tmp/clickhouse-minfo.new.yaml + ``` + + In `/tmp/clickhouse-minfo.new.yaml`, change the `vector` path to `/var/cpaas/data/vector`. Remove `metadata.uid`, `metadata.resourceVersion`, `metadata.creationTimestamp`, `metadata.managedFields` and `status`. + +3. Recreate the plugin from the corrected copy. + + > **Warning:** Uninstalling ClickHouse from the console removes the log storage and its PVCs, and log search is empty until the recreated plugin ingests again. Precondition: only when this cause is confirmed and the ModuleInfo backup from step 2 is saved on the global cluster. Rollback: apply the saved backup unchanged to return to the previous state. + + Uninstall ClickHouse from the console and wait until the ModuleInfo is gone. Then: + + ```bash + kubectl apply -f /tmp/clickhouse-minfo.new.yaml + kubectl -n cpaas-system get pv | grep razor + ``` + +4. Check that the new `razor` PV points under `/var/cpaas`. An in-place edit of the old ModuleInfo is not enough, because it leaves the `razor` PVC and PV on the old path. + +### Cause 2: razor stays in Error after an upgrade because migration_versions_new has a dirty record + +1. Confirm the signature: + + ```bash + kubectl -n cpaas-system logs -c razor --tail=40 | grep -E 'recover from dirty|migration aborted' + kubectl -n cpaas-system exec -c clickhouse -- clickhouse-client --query \ + "SELECT version, dirty, sequence FROM migration_versions_new ORDER BY sequence DESC LIMIT 1" + ``` + + `dirty` = 1 on the latest row matches this cause. + +2. Stop `razor` and every ClickHouse replica. Record the current `razor` replica count first. + + > **Warning:** Emptying the ClickHouse data directory permanently removes every log, audit and event row stored in this ClickHouse. Precondition: only when this cause is confirmed (`Try to do recover from dirty` followed by `migration aborted after recoverd`, and the latest `migration_versions_new` row is dirty), `razor` and all ClickHouse StatefulSets are at 0 replicas, and the data loss is accepted. Rollback: none. + + ```bash + kubectl -n cpaas-system get sts razor + kubectl -n cpaas-system scale sts razor --replicas=0 + kubectl -n cpaas-system scale sts chi-cpaas-clickhouse-replicated-0-0 --replicas=0 + kubectl -n cpaas-system scale sts chi-cpaas-clickhouse-replicated-0-1 --replicas=0 + kubectl -n cpaas-system scale sts chi-cpaas-clickhouse-replicated-0-2 --replicas=0 + ``` + +3. On every node that runs a logging ClickHouse pod, empty the logging data directory. This is not the MySQL slow-query store, which some clusters keep at `/cpaas/ck`. + + ```bash + cd /cpaas/data/clickhouse && rm -rf ./* + ``` + +4. Start ClickHouse again, then `razor` with its previous replica count: + + ```bash + kubectl -n cpaas-system scale sts chi-cpaas-clickhouse-replicated-0-0 --replicas=1 + kubectl -n cpaas-system scale sts chi-cpaas-clickhouse-replicated-0-1 --replicas=1 + kubectl -n cpaas-system scale sts chi-cpaas-clickhouse-replicated-0-2 --replicas=1 + kubectl -n cpaas-system scale sts razor --replicas= + ``` + +### Cause 3: razor fails with migration aborted after recoverd on a new deploy because one replica is missing tables + +1. Compare the `observability` tables on each replica: + + ```bash + for p in chi-cpaas-clickhouse-replicated-0-0-0 chi-cpaas-clickhouse-replicated-0-1-0 chi-cpaas-clickhouse-replicated-0-2-0; do + echo "== $p" + kubectl -n cpaas-system exec "$p" -c clickhouse -- clickhouse-client --query \ + "SELECT name FROM system.tables WHERE database = 'observability' ORDER BY name" + done + kubectl -n cpaas-system logs razor-0 -c razor --tail=200 | grep -E 'UNKNOWN_TABLE|does not exist' + ``` + + A replica that lacks tables the others have (for example `audit`), with `UNKNOWN_TABLE` in the `razor` log for that replica, matches this cause. + +2. Export the table definitions from a healthy replica: + + ```bash + kubectl -n cpaas-system exec chi-cpaas-clickhouse-replicated-0-0-0 -c clickhouse -- clickhouse-client --format TSVRaw --query \ + "SELECT name AS table_name, create_table_query FROM system.tables WHERE database = 'observability'" > /tmp/observability-ddl.tsv + ``` + +3. Create the missing tables on the affected replica. + + > **Warning:** Creating tables by hand on a replica changes its schema outside the `razor` migration. A statement for a table that already exists on that replica fails, and a wrong statement can register a table that does not match its peers. Precondition: only when this cause is confirmed, the statements come from a healthy replica of the same installation, and you run only the statements for tables missing on the target replica. Rollback: `DROP TABLE observability.
` on that replica removes a table created by mistake. + + Open a client on the replica that is missing tables: + + ```bash + kubectl -n cpaas-system exec -it -c clickhouse -- clickhouse-client + ``` + + Run the `create_table_query` of each missing table. Run the base tables first and the `*_all` Distributed tables that reference them afterwards. + +4. Repeat the comparison from step 1. Then restart `razor-0` so that it runs the migration again, and check that it becomes Ready: + + ```bash + kubectl -n cpaas-system delete pod razor-0 + kubectl -n cpaas-system get pods | grep razor + ``` + +## Diagnostic Steps + +1. Read the plugin and pod state: + + ```bash + kubectl get ars -A | grep -Ei 'click|log' + kubectl -n cpaas-system get pods | grep -Ei 'click|razor' + kubectl -n cpaas-system logs razor-0 -c razor --tail=100 + kubectl -n cpaas-system logs razor-0 -c vector --tail=100 + ``` + +2. Match the signature: + + | What you see | Cause | + |---|---| + | A mix of `/var/cpaas` and `/cpaas` paths in the ModuleInfo on an immutable OS, or `Read-only file system (os error 30)` in the `vector` container | Cause 1 | + | `razor` in `Error` after an upgrade, `Try to do recover from dirty`, and the latest `migration_versions_new` row has `dirty` = 1 | Cause 2 | + | `migration aborted after recoverd` on a new deploy, `UNKNOWN_TABLE` for an `observability` table, and one replica missing tables the others have | Cause 3 | + +3. The `RaftInstance ... End of file` lines that ClickHouse prints repeatedly also appear on correctly deployed clusters. They do not identify any of these causes. + +## Related Information + +- Related articles: [ClickHouse log storage plugin install fails or ClickHouse does not start on ACP](./ClickHouse_log_storage_plugin_install_fails_or_ClickHouse_does_not_start_on_ACP.md) covers the first-install failures where ClickHouse itself does not come up. +- [ClickHouse log queries fail after the log storage had been working: missing tables or broken parts on ACP](./ClickHouse_log_queries_fail_after_the_log_storage_had_been_working_missing_tables_or_broken_parts_on_ACP.md) diff --git a/docs/en/solutions/sentry_at_full_CPU_or_a_redeployed_component_fails_because_of_Failed_or_leftover_ResourcePatches_on_ACP.md b/docs/en/solutions/sentry_at_full_CPU_or_a_redeployed_component_fails_because_of_Failed_or_leftover_ResourcePatches_on_ACP.md new file mode 100644 index 000000000..4a1fbe8ae --- /dev/null +++ b/docs/en/solutions/sentry_at_full_CPU_or_a_redeployed_component_fails_because_of_Failed_or_leftover_ResourcePatches_on_ACP.md @@ -0,0 +1,140 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.12" + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# sentry at full CPU or a redeployed component fails because of Failed or leftover ResourcePatches on ACP + +## Issue + +- One of the two `sentry` Pods is almost idle and the other runs at about 100% of its CPU limit (around 2000m). The spikes can come and go within minutes. +- The platform cert-manager component keeps upgrading and never finishes. +- After ClickHouse was installed no ClickHouse instance appears, and redeploying the ClickHouse plugin fails. +- `olm-registry-custom`, `olm-registry-platform`, and `olm-registry-system` are all `0/1` `CrashLoopBackOff` a few minutes after they were created. + +`sentry` (Deployment `sentry` in `cpaas-system` on each cluster) installs, upgrades, and removes plugin charts and records the result on the `ars` (AppRelease) objects. A ResourcePatch (`rpch`) records a change made to a platform-managed resource; `sentry` applies it on top of the chart every time the release is reconciled. A ResourcePatch is not removed when its plugin is uninstalled, and it is applied again when the plugin is reinstalled. + +Related articles: "Plugin install or uninstall stuck, or Marketplace and OperatorHub empty: sentry OOMKilled or cluster-transformer errors". + +## Root Cause + +### Cause 1: One sentry Pod at 100% CPU because Failed ResourcePatches are retried in a tight loop + +The busy `sentry` Pod logs `ResourcePatch Reconcile` every few seconds for the same objects, with errors such as: + +```text +Apply ResourcePatch rp-4mcvk for apps/v1.Deployment cert-manager/cert-manager-cainjector in Release cpaas-system/cert-manager error: error in add for path: '/spec/template/spec/containers/1': Unable to access invalid index: 1: invalid index referenced +``` + +The idle Pod writes no log lines. Three ResourcePatches are in `Failed`, targeting release `cpaas-system/cert-manager` and the `cert-manager-cainjector`, `cert-manager`, and `cert-manager-webhook` Deployments. Each tries to patch `/spec/template/spec/containers/1`, a second container that does not exist in those Deployments, so `sentry` retries them continuously. Deleting those ResourcePatches brings the busy Pod down from about 2000m to about 456m CPU. + +### Cause 2: cert-manager keeps upgrading because the platform cert-manager was replaced with community images + +The cert-manager component stays in upgrading. `kubectl get rpch -A | grep cert-manager` shows the ResourcePatches for the `cert-manager-cainjector`, `cert-manager-webhook`, and `cert-manager` Deployments and the `cert-manager-webhook` and `cert-manager` Services in `Failed` (the RBAC and ServiceAccount ones `Active`), and the `cert-manager-controller` container runs `jetstack/cert-manager-controller:v1.16.1` with `--leader-election-namespace=kube-system` and `--max-concurrent-challenges=60`. + +The cert-manager deployment script of the Kubeflow 1.10 community release, run when Kubeflow was deployed, replaced the platform cert-manager images with community images. That conflicts with the platform component, so its upgrade stays stuck. Restore every cert-manager component to the platform's own images and do not modify platform components. Restoring the images does not affect platform certificates. The Kubeflow certificates issued by the community controller (in `ai-platform`, `kserve`, and `kubeflow`, some managed by Argo CD) stay, but the platform cert-manager manages only platform certificates and will not renew them; the team that deployed Kubeflow has to plan their renewal. + +### Cause 3: A leftover ResourcePatch breaks a redeployed component + +A component that was removed and deployed again fails, and a ResourcePatch for it from the earlier deployment is still in the cluster. + +- ClickHouse: the ClickHouse plugin deployment fails because of a leftover ClickHouse ResourcePatch; after it is deleted, the plugin deploys normally. Separately, the ClickHouse operator (for example `clickhouse-operator.v4.2.1` in `cpaas-logging`, `Succeeded`) does not create an instance by itself; the instance has to be created manually. +- OLM: all three `olm-registry-*` Pods are `0/1` `CrashLoopBackOff` with the same restart count, and ResourcePatches related to them exist. After the patches that are no longer needed are deleted, the OLM components run normally. + +## Resolution + +### Cause 1: One sentry Pod at 100% CPU because Failed ResourcePatches are retried in a tight loop + +```bash +kubectl -n cpaas-system top pod | grep sentry +kubectl -n cpaas-system logs --tail=300 | grep -E 'ResourcePatch|invalid index' +kubectl get rpch -A | grep -i failed +``` + +Back up every ResourcePatch the log names: + +```bash +mkdir -p "$HOME/rpch-backup" +for rp in ; do kubectl get rpch "$rp" -o yaml > "$HOME/rpch-backup/$rp.yaml"; done +``` + +> **Warning:** Deleting a ResourcePatch drops the customization it carries from that component. A `Failed` patch is not being applied, so nothing that currently runs changes. Precondition: only for ResourcePatches in `Failed` whose `invalid index referenced` errors repeat in the `sentry` log, with the backups written and the owner of the customization informed. Rollback: `kubectl apply -f "$HOME/rpch-backup/.yaml"` (remove `metadata.resourceVersion` and `metadata.uid` first). + +```bash +kubectl delete rpch +kubectl -n cpaas-system top pod | grep sentry +``` + +If `kubectl get rpch -A` shows a namespace for these objects, add `-n ` to the `get` and `delete` commands. + +When the spikes are short and gone before you can look, check for the same retry pattern after the fact: `Failed` ResourcePatches, and the same object names repeating in the `sentry` log. Raising the `sentry` CPU limit does not stop a retry loop. + +### Cause 2: cert-manager keeps upgrading because the platform cert-manager was replaced with community images + +Confirm the community images and record which certificates depend on them: + +```bash +kubectl get ars -A | grep -i cert +kubectl get rpch -A | grep cert-manager +kubectl get deploy -A | grep cert-manager +kubectl -n get deploy -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.template.spec.containers[*].image}{"\n"}{end}' +kubectl get certs -A +kubectl -n get deploy -o yaml > "$HOME/cert-manager-deploy.before.yaml" +``` + +Images such as `jetstack/cert-manager-controller:` instead of images from the platform registry are this cause. Get the platform images and container settings for your platform version from a cluster on the same version where cert-manager is healthy (same command), or from Alauda support. + +> **Warning:** Restoring the platform cert-manager images changes the controller that issues certificates; the Kubeflow certificates created by the community controller stay but are no longer renewed by the platform cert-manager. Precondition: only when the cert-manager Deployments run community images after a Kubeflow community deployment script and the cert-manager upgrade is stuck; agree with the team that owns Kubeflow how its certificates will be renewed first. Rollback: `kubectl apply -f "$HOME/cert-manager-deploy.before.yaml"`, which returns the conflicting state. + +```bash +kubectl -n set image deploy/ =/: +kubectl get ars -A | grep -i cert +``` + +Repeat for each cert-manager Deployment, and bring the container arguments back to the platform values if they differ (the community controller runs with `--leader-election-namespace=kube-system`). Do not run third-party scripts that modify platform components. + +### Cause 3: A leftover ResourcePatch breaks a redeployed component + +Find the ResourcePatches for the component, back them up, and confirm with their owner that they are no longer needed: + +```bash +kubectl get rpch -A +kubectl get rpch -A -o yaml | grep -n -i -E '|release:' +mkdir -p "$HOME/rpch-backup" +kubectl get rpch -o yaml > "$HOME/rpch-backup/.yaml" +``` + +> **Warning:** Deleting a ResourcePatch removes the customization it would apply to the component when it is deployed again. Precondition: only when the redeployed component (the ClickHouse plugin, or all `olm-registry-*` Pods in `CrashLoopBackOff`) fails, the ResourcePatch belongs to that component's earlier deployment, its owner confirmed it is not needed, and the backup is written. Rollback: `kubectl apply -f "$HOME/rpch-backup/.yaml"`. + +```bash +kubectl delete rpch +``` + +Redeploy the ClickHouse plugin, or watch the OLM registry Pods recover (`kubectl -n cpaas-system get pods -w | grep olm-registry`). For ClickHouse, create the ClickHouse instance manually after the plugin is deployed: + +```bash +kubectl get csv -n cpaas-logging | grep -i clickhouse +``` + +## Diagnostic Steps + +```bash +kubectl -n cpaas-system get pods | grep sentry +kubectl -n cpaas-system top pod | grep sentry +kubectl get rpch -A +kubectl get rpch -A | grep -i failed +kubectl -n cpaas-system logs --tail=300 | grep -E 'ResourcePatch|invalid index' +``` + +- One `sentry` Pod near its CPU limit logging `ResourcePatch Reconcile` with `invalid index referenced`, and `Failed` ResourcePatches: Cause 1. +- A stuck cert-manager upgrade with `jetstack/...` images: Cause 2. +- A redeployed ClickHouse plugin or OLM registry failing while an old ResourcePatch for it exists: Cause 3. +- `sentry` restarting with `OOMKilled`: see the related article on stuck installs and uninstalls. diff --git a/docs/en/solutions/tke-platform-api_serves_an_expired_certificate_after_running_for_a_year_on_ACP.md b/docs/en/solutions/tke-platform-api_serves_an_expired_certificate_after_running_for_a_year_on_ACP.md new file mode 100644 index 000000000..97a4e2322 --- /dev/null +++ b/docs/en/solutions/tke-platform-api_serves_an_expired_certificate_after_running_for_a_year_on_ACP.md @@ -0,0 +1,175 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - 3.12.x + - 3.18.x + - 4.1.x + - 4.2.x + - 4.3.x + - 4.4.x +--- + +# tke-platform-api serves an expired certificate after running for a year on ACP + +## Issue + +On the global cluster, operations that go through the platform API stop working while the cluster itself looks healthy, and neither the platform access certificate nor the Kubernetes certificates have expired. Each case shows an x509, TLS handshake, or port 9443 error in the `tke-platform-api` log, in a caller's log, in the API response, or on the login page: + +- A cluster plugin (for example NFS) deployed from the cluster plugin page stays in "Deploying" forever, although the plugin works on the nodes. On the business cluster the plugin AppRelease is `Synced` and `Ready`; on global, `kubectl get minfo -A | grep ` shows that cluster's minfo stuck at `Processing`. +- Adding a node is stuck at "check timeout" although the node's network and ports are reachable, and `tke-platform-api` logs `http: TLS handshake error from ...: remote error: tls: bad certificate`. +- Adding a node returns `检查超时,请重试` ("check timed out, please retry") at once, while the dialog still shows `节点可用性检查中...` ("node availability check in progress"). In the browser developer tools, `POST /apis/platform.tkestack.io/v1/clusters//exec?ignore_warning=true` returns `500 Internal Server Error`, and the `tke-platform-api` log shows x509 errors for calls to `https://localhost:9443/apis/platform.tkestack.io/v1/...`. +- Creating a business cluster fails with `error trying to reach service: stream error: stream ID ; INTERNAL_ERROR; received from peer`. +- Adding a node or evicting containers fails with `x509: certificate has expired or is not yet valid: current time is after `. +- Changing a cluster's display name fails. The `PATCH` to `/apis/platform.tkestack.io/v1/clusters/` returns `422` (`reason: Invalid`) with `Internal error: get cluster's credential error: Get "https://localhost:9443/apis/platform.tkestack.io/v1/clustercredentials/": x509: certificate has expired or is not yet valid`. +- The login page reports an expired certificate and nobody can log in. It shows `ErrCreateAuthReq` with `Get ".../api/v1/namespaces/cpaas-system/resourcequotas": x509: certificate has expired or is not yet valid: current time is after `. + +`tke-platform-api` and `tke-platform-controller` run on the `global` cluster in `cpaas-system`. On some `global` clusters (for example on ACP 4.2) there is no `tke-platform-api` Pod, and `cluster-api-provider-alauda` manages node addition instead; check which one yours runs with `kubectl get deploy -A | grep -E 'tke-platform|cluster-api-provider-alauda'`. + +## Root Cause + +### Cause 1: the tke-platform-api Pod has been running for more than a year and its in-Pod certificate expired + +The certificate used inside the `tke-platform-api` Pod is valid for one year. A Pod that has run longer than that serves an expired certificate, and callers of `tke-platform-api` on port 9443 fail. For a stuck plugin, `cluster-transformer` on global logs errors against port 9443 while the plugin's minfo stays `Processing`; forcing a reconcile of the ARS does not help, because the call to `tke-platform-api` is the part that fails. For a stuck node addition, `tke-platform-api` logs `tls: bad certificate` handshake errors from a local client, along with `http2: stream closed`. + +The process uses that certificate for its own calls to `localhost:9443`, and neither of the following log lines names the certificate. The Pod age is a guide, not a threshold: a Pod that has run for about 300 days can already fail this way. When the add-node `exec` request returns 500, the log repeats for the cluster being added to: + +```text +apiserver received an error that is not an metav1.Status: &url.Error{Op:"Get", URL:"https://localhost:9443/apis/platform.tkestack.io/v1/clusters/", Err:x509.CertificateInvalidError{... Reason:1, Detail:"current time is after "}} +``` + +When a business cluster is created, the create path panics in `(*Strategy).PrepareForCreate` (`pkg/platform/registry/cluster/strategy.go`): + +```text +apiserver panic'd on POST /apis/platform.tkestack.io/v1/clusters +get cluster's credential error: Get "https://localhost:9443/apis/platform.tkestack.io/v1/clustercredentials?fieldSelector=clusterName%3D...": x509: certificate has expired or is not yet valid: current time is after +``` + +Restarting `tke-platform-api` loads a valid certificate, after which the plugin deploys, the node is added, and the cluster is created. + +### Cause 2: on 3.12, tke-platform-api logs x509: certificate has expired on node add or eviction + +On ACP 3.12 (TKE-based), `tke-platform-api` reports `x509: certificate has expired or is not yet valid: current time ... is after ...` when nodes are added or containers are evicted. This is a known component bug of 3.12, fixed in later versions. Restarting `tke-platform-api` and `tke-platform-controller` restores the functions; the error line can still appear in the `tke-platform-controller` log afterwards. Log lines such as `attempting to acquire leader lease /tke-platform-controller` and `lock is held by tke-platform-controller-... and has not yet expired` are normal with two replicas, because the other Pod holds the leader lease. + +### Cause 3: the login page shows an expired certificate managed by tke-platform-api and dex + +The certificate behind the login page's `ErrCreateAuthReq` error is managed jointly by `tke-platform-api` and `dex`. When it has expired, users cannot log in. Restarting both components restores the login page. + +### Cause 4: updating a cluster fails with get cluster's credential error and x509 against localhost:9443 + +The platform fetches the cluster credential (`clustercredentials`) from `https://localhost:9443`. `tke-platform-api` caches the certificate it uses there, so after that certificate expires it keeps using the expired one, and every cluster update that needs the credential fails with `422`. Restarting `tke-platform-api` restores the call. + +## Resolution + +### Cause 1: the tke-platform-api Pod has been running for more than a year and its in-Pod certificate expired + +1. Check the Pod age and the logs, and confirm that the platform access certificate is valid and the clock on `global` is correct: + + ```bash + kubectl get pods -A -o wide | grep tke-platform-api + kubectl logs -n --tail=200 | grep -i -E 'tls|x509|handshake' + kubectl -n cpaas-system logs deploy/tke-platform-api --tail=500 | grep -E 'localhost:9443|CertificateInvalidError|certificate has expired|panic' + echo | openssl s_client -connect :443 2>/dev/null | openssl x509 -noout -dates + date -u + kubectl get pods -A | grep cluster-transformer + kubectl logs -n --tail=200 | grep 9443 + kubectl get minfo -A | grep + ``` + + Continue when the `tke-platform-api` Pod has run for a long time (about a year, sometimes less) and the logs show port 9443 or `localhost:9443` x509 errors or `tls: bad certificate` handshake errors, while the platform access certificate's `notAfter` is still in the future and `date -u` is correct. + +2. Restart `tke-platform-api`. + + > **Warning:** Deleting the `tke-platform-api` Pod interrupts the platform API calls that Pod handles until its replacement is Running; plugin deployment, cluster and node management for every cluster, and console actions that use it fail in the gap. Running workloads are not affected. Do this only when step 1 confirmed the signature and the clock on `global` is correct (an x509 time error can also come from a wrong clock). With several replicas, delete one Pod at a time and wait for it to be `Running` before the next. The workload recreates the Pod; there is nothing else to roll back. + + ```bash + kubectl delete pod -n + kubectl get pods -n | grep tke-platform-api + ``` + + A rolling restart of the Deployment is equivalent: + + ```bash + kubectl -n cpaas-system rollout restart deployment/tke-platform-api + kubectl -n cpaas-system rollout status deployment/tke-platform-api + ``` + +3. Retry the operation (plugin deployment, node addition, or cluster creation). For a plugin, watch its ARS and the Pods: + + ```bash + kubectl get ars -A | grep + kubectl get pods -A | grep -vi running + ``` + +### Cause 2: on 3.12, tke-platform-api logs x509: certificate has expired on node add or eviction + +1. Restart `tke-platform-api` with step 2 of Cause 1 and its Warning. +2. Restart `tke-platform-controller` the same way, one Pod at a time. + + > **Warning:** Deleting a `tke-platform-controller` Pod pauses the reconciliation that Pod leads until the other replica takes the lease or the new Pod starts. The workload recreates the Pod. + + ```bash + kubectl get pods -A | grep tke-platform-controller + kubectl delete pod -n + kubectl logs -n --tail=50 + ``` + +3. Check that adding a node or evicting a container works. Upgrade to a later version to remove the bug. + +### Cause 3: the login page shows an expired certificate managed by tke-platform-api and dex + +1. Restart `tke-platform-api` with step 2 of Cause 1. +2. Restart `dex`. + + > **Warning:** Deleting the `dex` Pod interrupts login to the platform until the new Pod is Running. Do this only when the login page reports the expired certificate. The workload recreates the Pod. + + ```bash + kubectl get pods -A | grep dex + kubectl delete pod -n + ``` + +3. If the login page still shows an expired certificate, compare the served certificate with the platform access certificate, as described in "Replace the platform access certificate in dex.tls on ACP". + +### Cause 4: updating a cluster fails with get cluster's credential error and x509 against localhost:9443 + +1. Confirm that the error names `localhost:9443` and `clustercredentials`: + + ```bash + kubectl get pods -A | grep tke-platform-api + kubectl logs -n --tail=200 | grep -i -E 'x509|clustercredentials' + ``` + +2. Restart `tke-platform-api` with step 2 of Cause 1, one Pod at a time. For this cause the precondition is the error above, not the Pod age. +3. Retry the cluster edit. + +## Diagnostic Steps + +1. Note the Pod age of the platform API components, search the `tke-platform-api` log for the `localhost:9443` signature, and check the clock: + + ```bash + kubectl get pods -A | grep -E 'tke-platform-api|tke-platform-controller|dex' + kubectl -n cpaas-system logs deploy/tke-platform-api --tail=500 | grep -E 'localhost:9443|CertificateInvalidError|certificate has expired|panic' + date -u + ``` + +2. Find who is failing and against which port: a stuck plugin's ARS and the `cluster-transformer` log, or the `tke-platform-api` log when node addition times out. + + ```bash + kubectl get ars -A + kubectl describe ars -n + ``` + +3. Branch: + + - Pod older than a year, errors to port 9443, `tls: bad certificate` handshake failures, or a plugin minfo stuck at `Processing` while its AppRelease is `Ready`: Cause 1. + - The add-node `exec` request returns 500 at once in the browser developer tools and the log shows `localhost:9443` with an x509 error, or a cluster create fails with `INTERNAL_ERROR` and the `PrepareForCreate` panic: Cause 1. + - `date -u` on `global` or on the node is wrong: fix the clock; this article does not apply. + - The add-node `exec` request returns 500 and the log has no `localhost:9443` x509 line: see "Node add check times out immediately because the cluster exec API returns HTTP 500 on ACP". + - The `exec` request returns 200, or 504 after a long wait: a real check timeout; check SSH and network reachability to the node. + - ACP 3.12 and `current time ... is after ...` on node add or eviction: Cause 2. + - Login page shows `ErrCreateAuthReq` with `x509: certificate has expired or is not yet valid`: Cause 3. + - A cluster update returns `422` with `get cluster's credential error` and x509 against `https://localhost:9443/.../clustercredentials/...`: Cause 4. + - The same `get cluster's credential error ... localhost:9443 ... x509: certificate has expired` text after a finalizer was removed from a cluster stuck in `Terminating` whose `ClusterCredential` is missing: not this article; recreate the credential. + - The platform cannot be opened at all, every component is Running, and no log or page shows an x509, handshake, or port 9443 error: not this article. When the cluster VIP is a domain name, see "Platform console does not open because the cluster VIP domain name cannot be resolved inside the cluster on ACP". diff --git a/docs/en/solutions/violet_push_fails_to_parse_a_package_or_a_new_operator_version_is_skipped_or_missing_on_ACP.md b/docs/en/solutions/violet_push_fails_to_parse_a_package_or_a_new_operator_version_is_skipped_or_missing_on_ACP.md new file mode 100644 index 000000000..32b63907f --- /dev/null +++ b/docs/en/solutions/violet_push_fails_to_parse_a_package_or_a_new_operator_version_is_skipped_or_missing_on_ACP.md @@ -0,0 +1,106 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "4.0" + - "4.2" + - "4.3" +--- + +# violet push fails to parse a package, or a new operator version is skipped or missing on ACP + +## Issue + +- A push fails immediately with `failed to parse package and manifest for package .tgz: unexpected EOF`. +- GPU Operator and SR-IOV Network uploads fail while every other required plugin from the same set (for example ACP 4.3.2) uploads. +- A new operator version was pushed (for example Tekton 4.14, or `tektoncd-operator` 4.15.1), but the page and `PackageManifests` still show only the old version, and `violet` printed `WARN artifactversion operatorhub-.v already exist, skip it`. +- An operator package synchronized from `global` is visible as a software package but not in OperatorHub on a business cluster. + +Related articles: "violet push succeeds but the plugin is not published: Artifact, ModuleConfig, or chart missing on ACP" covers successful pushes whose objects never appear. A pushed Helm chart that reports `artifact already exist, will create artifactversion ... type=chart` is covered in "Pushed Helm chart version not shown because violet reused an Artifact of another chart repository on ACP". + +## Root Cause + +### Cause 1: failed to parse package and manifest, unexpected EOF + +`violet` cannot read the archive (for example `workbench-operator.stable.amd64.v0.2.1.tgz`): the archive was corrupted during upload. Matching MD5 sums across several downloads do not rule this out. Uploading the package again clears the error. + +### Cause 2: GPU Operator or SR-IOV package is missing required fields + +On ACP 4.3.2, the GPU Operator and SR-IOV Network marketplace packages were missing required fields that belong in `moduleplugin.yaml`; fetching the marketplace package again leaves the same gap. `gpu-operator` was re-released, and version `v26.3.2-default.1.gcdaaba55` is the build to select. The `sriov-network-plugin` package was still in testing and not released for production use; it is not the production plugin, and its official release has to be awaited. + +### Cause 3: New version pushed, ArtifactVersion is Failed or skipped as already existing + +`violet` prints `WARN artifactversion operatorhub-.v already exist, skip it`, and the new version is missing from the page and from `kubectl get packagemanifest`. The ArtifactVersion shows `phase=Failed`, `reason=Error`, with a bundle pull error such as `failed to do request: Head "http:///v2//-bundle/manifests/": dial tcp :80: connect: no route to host` or `... not found`. + +An earlier push left the ArtifactVersion in the cluster, but the OLM registry could not pull the bundle image it points to; the `olm-registry-platform` log repeats `failed to add bundle` for it. A plain re-push sees the existing ArtifactVersion and skips it, and restarting `olm`, `packageserver`, and `catalog-operator` does not help. Pushing again with `violet push --force` overwrites the existing objects, and the new version appears. A version that does not show up after release, with no error text, is recovered the same way. An operator package synchronized from `global` that is missing from OperatorHub on a business cluster is restored by uploading the package again; no underlying cause is known for that form. + +## Resolution + +### Cause 1: failed to parse package and manifest, unexpected EOF + +Test the file on disk before another push; `tar` reporting an unexpected end of archive means the file is not a complete package: + +```bash +tar -tzf .tgz >/dev/null +``` + +Obtain the package again from the distribution source and push that copy: + +```bash +violet push --platform-address= --platform-username= --platform-password='' .tgz +``` + +`violet` should report a successful push and no `unexpected EOF`. + +### Cause 2: GPU Operator or SR-IOV package is missing required fields + +Select the re-released GPU Operator build `v26.3.2-default.1.gcdaaba55`. For SR-IOV Network, wait for the officially released plugin; the testing package is not for production use. + +```bash +kubectl get moduleplugins | grep -iE 'gpu|sriov' +``` + +### Cause 3: New version pushed, ArtifactVersion is Failed or skipped as already existing + +On the cluster that should offer the version, read the ArtifactVersion phase and message: + +```bash +kubectl get packagemanifest,artifact,artifactversion -A | grep +kubectl -n cpaas-system get artifactversion operatorhub-.v -o yaml +kubectl -n cpaas-system logs deploy/olm-registry-platform --tail=300 | grep -i 'failed to add bundle' +``` + +> **Warning:** `--force` overwrites the package, Artifact, and ArtifactVersion already on the platform for this package version and architecture; any earlier content stored under the same name is replaced. Precondition: only when the version is missing from `PackageManifests`, and either `violet` printed `already exist, skip it` for that ArtifactVersion or the ArtifactVersion is `Failed` with a bundle pull error. Rollback: there is no undo of the overwrite; push the previous tarball again with `--force` if you need the old content back. + +```bash +violet push --platform-address= --platform-username= --platform-password='' --clusters --force .tgz +``` + +Check that the version is listed: + +```bash +kubectl get packagemanifest -A | grep +kubectl -n cpaas-system get artifactversion | grep +``` + +If the ArtifactVersion still fails with `no route to host`, the cluster cannot reach the registry host in that message. Fix that network path or push to a registry the cluster can reach; another `--force` does not change the address. + +For an operator package that is visible as a software package but missing from OperatorHub on a business cluster, upload the package again for that cluster. + +## Diagnostic Steps + +```bash +violet version +tar -tzf .tgz >/dev/null +kubectl get moduleplugins | grep +kubectl -n cpaas-system get artifactversion | grep +``` + +- `unexpected EOF` from `violet` or from `tar`: Cause 1. +- A GPU or SR-IOV upload error that names missing fields, while other plugins succeed: Cause 2. +- An ArtifactVersion in `Failed` with a bundle pull error, or `artifactversion operatorhub-... already exist, skip it`: Cause 3. +- The same `already exist` message with `type=chart`: see the related Helm chart article. +- No `artifact` / `artifactversion` at all after a successful push: see the related article on plugins that are not published. diff --git a/docs/en/solutions/violet_push_fails_with_no_package_found_code_000_HTTP_500_or_502_or_invalid_reference_format_on_ACP.md b/docs/en/solutions/violet_push_fails_with_no_package_found_code_000_HTTP_500_or_502_or_invalid_reference_format_on_ACP.md new file mode 100644 index 000000000..3dadf7a1b --- /dev/null +++ b/docs/en/solutions/violet_push_fails_with_no_package_found_code_000_HTTP_500_or_502_or_invalid_reference_format_on_ACP.md @@ -0,0 +1,220 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# violet push fails with no package found, code 000, HTTP 500 or 502, or invalid reference format on ACP + +## Issue + +Uploading an extension package or operator with `violet push` stops with an error: + +- `Error: no package found in ./plugins/.gz`, although the archive opens fine in an archive tool (for example `edp-keycloak-operator-v1.23.0.gz` downloaded from Alauda Marketplace), or `Error: no package found in installer-core-v4.3.2-hybrid.tar`. +- The push to the platform repository returns `code 000`, and afterwards ModuleInfo reports `moduleconfig not found`, although the previous version pushed to the same repository worked. +- A batch `violet push` of every extension package before an upgrade succeeds for most packages and fails for some with `500 Internal Server Error`. +- `violet push` skips the images that already exist and then repeats `copy error: writing blob: uploading layer chunked: received unexpected HTTP status: 502 Bad Gateway`. +- `violet push ... --dest-repo "https://:"` fails with `invalid destination name docker://https://:/...: invalid reference format`, and `docker login` to the platform registry on the same host fails with an X509 error. +- Pushing a chart package to the platform's built-in public chart repository with the platform address, username, and password fails with an invalid username or password. + +Related articles: "violet push goes to the wrong global cluster, a wrong package cannot be deleted, or violet or kubectl is missing on ACP" covers pushes that complete in the wrong place or cannot be undone; "violet push fails to parse a package, or a new operator version is skipped or missing on ACP" covers `unexpected EOF`; "violet push succeeds but the plugin is not published: Artifact, ModuleConfig, or chart missing on ACP" covers pushes that report success. + +## Root Cause + +### Cause 1: Error: no package found, the file is not in the violet package layout + +`violet push` accepts only packages in the Violet package layout. A package produced by the standard Violet process (`violet create`, then `violet package `) is named `PLUGIN_NAME.BUNDLE_VERSION.tgz`, for example `edp-keycloak-operator.v1.23.0.tgz`, and pushes successfully. A `.gz` downloaded from Alauda Marketplace that differs in structure gives `no package found`; renaming or repacking it as `.tgz` or `.tar.gz` gives the same error. The error also appears when the file is not a plugin package at all: `installer-core-v4.3.2-hybrid.tar` is an installer archive, so `violet` finds no package metadata in it. + +### Cause 2: code 000, the platform repository could not be reached, and ModuleInfo reports moduleconfig not found + +`code 000` means that no HTTP response was received: the host running `violet` could not reach the repository. The plugin artifact was never stored, so the ModuleConfig for that version was never created, and ModuleInfo cannot find it. Once the repository is reachable again, the push succeeds and the ModuleConfig appears. + +A host that cannot resolve the registry name fails the same way before any HTTP exchange, with an error such as `dial tcp: lookup : no such host`. + +### Cause 3: some packages of a batch push fail with 500 Internal Server Error + +Most images print `found image ... already exists, skip push`; the failing ones print: + +```text +copy error: copying image 2/2 from manifest list: writing blob: uploading layer chunked: received unexpected HTTP status: 500 Internal Server Error +push artifact .v to cluster global failed. error: copying image 2/2 from manifest list: ... +failed to push images via skopeo mode for package /.tgz +``` + +The same error can name `copying image 1/2`. After a failure, `violet push` continues with the remaining packages. The registry rejected individual layer uploads; pushing only the failed packages again after the run has finished completes them. + +### Cause 4: 502 Bad Gateway while uploading layers, the platform registry is OOMKilled and restarting + +The push skips the images that already exist and fails on the first layer it has to upload, again and again. The registry Pod that receives the push keeps restarting: its `registry` container shows `Last State: Terminated`, `Reason: OOMKilled`, `Exit Code: 137` (for example with limits `cpu: 1`, `memory: 2Gi`). Each restart cuts off the layer upload in progress, and the gateway in front of the registry answers `502`. After the registry's CPU and memory limits are raised, it stops restarting and the push completes. + +### Cause 5: invalid reference format because --dest-repo has https://, and docker login fails with X509 + +```text +ERRO invalid destination name docker://https://:/:: invalid reference format +WARN failed to push images via skopeo mode for package .tgz, error: invalid reference format +Error: failed to push one or more packages. +``` + +Two separate problems on the host that runs the push: + +- `violet` builds the destination image name from `--dest-repo`. An image reference cannot contain a URL scheme, so `--dest-repo https://:` makes every image fail with `invalid reference format`. Passwords that contain shell special characters also break when they are not single-quoted. +- `docker login` to the platform registry fails with an X509 error when the registry address is not configured in Docker's `daemon.json` on that host. + +### Cause 6: a chart push to the built-in public chart repository is rejected as invalid username or password + +Pushing the raw chart to the built-in `public chart` repository with the platform credentials is rejected as an invalid username or password. Packaging the chart with `violet` and pushing it with `violet push` succeeds. + +## Resolution + +### Cause 1: Error: no package found, the file is not in the violet package layout + +1. If you already have a `.tgz` named `PLUGIN_NAME.BUNDLE_VERSION.tgz` (for example from Alauda technical support), push that file: + + ```bash + violet push --platform-address= --platform-username= --platform-password='' edp-keycloak-operator.v1.23.0.tgz + ``` + +2. Otherwise build the package with the standard process and push the generated `.tgz`: + + ```bash + violet create + violet package + ls *.tgz + ``` + +Repacking a Marketplace `.gz` by hand does not help. Do not push installer archives such as `installer-core--.tar` with `violet push`; they are not plugin packages. + +### Cause 2: code 000, the platform repository could not be reached, and ModuleInfo reports moduleconfig not found + +1. From the host that runs `violet`, check name resolution and reachability of the repository. Any HTTP status means the host is reachable; `000` or a connection error means it is not: + + ```bash + getent hosts + curl -k -s -o /dev/null -w '%{http_code}\n' https://:/v2/ + ``` + +2. Restore access to the repository: the network path, DNS (or an `/etc/hosts` entry on the push host), the proxy, or the repository service itself. +3. Push the same package again, then check that the ModuleConfig exists on `global`: + + ```bash + kubectl get moduleconfigs | grep + kubectl get moduleinfo | grep + ``` + + `moduleconfig not found` clears once the version has actually been pushed. + +### Cause 3: some packages of a batch push fail with 500 Internal Server Error + +1. Let the running push finish completely. +2. Collect the failed packages from its output (the `push artifact to cluster failed` and `failed to push images via skopeo mode for package ` lines). +3. Push only those: + + ```bash + mkdir -p ./retry + cp .tgz .tgz ./retry/ + violet push ./retry --platform-address= --platform-username= --platform-password='' + ``` + +If the same packages fail again with `500`, check the registry Pod as in Cause 4. + +### Cause 4: 502 Bad Gateway while uploading layers, the platform registry is OOMKilled and restarting + +1. Find the registry Pod on the cluster that the push targets and read why it restarted: + + ```bash + kubectl get pods -A | grep -i registry + kubectl -n describe pod | grep -E -A6 'Last State|Limits|Requests|Controlled By' + ``` + + `Reason: OOMKilled` with `Exit Code: 137` on the `registry` container, while the push repeats `502 Bad Gateway`, is this cause. + +2. Save the workload before changing it: + + ```bash + kubectl -n get deploy -o yaml > "$HOME/.before.yaml" + ``` + +3. Raise the registry CPU and memory limits. + + > **Warning:** Changing the registry resources restarts the registry Pod; image pulls and pushes against it fail until the new Pod is Ready, and Pods scheduled in that window can be in `ImagePullBackOff` for a while. Precondition: only when the `registry` container's last state is `OOMKilled` (exit code `137`) and the push fails with `502 Bad Gateway`; do it when no push or large rollout is running. Rollback: `kubectl apply -f "$HOME/.before.yaml"`. + + ```bash + kubectl -n set resources deploy -c registry --limits=cpu=,memory= + kubectl -n rollout status deploy + ``` + + Size the memory limit above the peak the container reached before the OOM. + +4. After a few minutes, check that the Deployment still carries the new limits. If they went back, the values are managed by the component that deploys the registry; collect the Deployment's labels and annotations and ask Alauda technical support where to set them. +5. When the registry stays Running, push the package again; images that already exist are skipped. + +### Cause 5: invalid reference format because --dest-repo has https://, and docker login fails with X509 + +1. Give `--dest-repo` as `host:port` with no scheme, and single-quote the passwords: + + ```bash + violet push .tgz \ + --platform-address \ + --platform-username \ + --platform-password '' \ + --dest-repo ":" \ + --username "" \ + --password '' \ + --debug + ``` + + The output should end with `successfully create artifact resources for package `. If an earlier failed attempt left an artifact behind, the push may need `--force`; read the Warning about `--force` in the related article on wrong packages before using it. + +2. For the X509 error on `docker login`, check the Docker daemon configuration on the push host: + + ```bash + cat /etc/docker/daemon.json + docker login : + ``` + + For a registry whose certificate the host does not trust, the Docker setting that takes a registry address is `insecure-registries`: + + ```json + { + "insecure-registries": [":"] + } + ``` + + > **Warning:** Docker reads `daemon.json` only when it starts. Restarting Docker stops the containers on this host unless `live-restore` is enabled, and an invalid `daemon.json` keeps Docker from starting. `insecure-registries` also turns off TLS verification for that registry from this host. Precondition: only on the host that runs `violet`, when `docker login` to the platform registry fails with an X509 error. Back the file up and merge the key into the existing content instead of replacing it. Rollback: restore the backup and restart Docker again. + + ```bash + cp /etc/docker/daemon.json /etc/docker/daemon.json.bak + vi /etc/docker/daemon.json + systemctl restart docker + docker login : + ``` + +### Cause 6: a chart push to the built-in public chart repository is rejected as invalid username or password + +Package the chart with the packaging flow of the `violet` version that matches the platform, then push the resulting package with `violet push` instead of pushing the raw chart with the platform credentials: + +```bash +violet push --platform-address= --platform-username= --platform-password='' .tgz +``` + +## Diagnostic Steps + +```bash +violet version +getent hosts +curl -k -s -o /dev/null -w '%{http_code}\n' https://:/v2/ +kubectl get pods -A | grep -i registry +``` + +- `no package found` on a Marketplace `.gz` or an installer `.tar`: Cause 1. +- `code 000`, or `lookup ... no such host`, then `moduleconfig not found`: Cause 2. Expected: the `curl` above prints `000` or a connection error. +- `500 Internal Server Error` for some packages of a batch push while the registry Pod is stable: Cause 3. +- `502 Bad Gateway` while uploading layers, with the registry Pod `OOMKilled`: Cause 4. +- `invalid reference format` with `docker://https://` in the destination name, or an X509 error on `docker login`: Cause 5. +- A raw chart push to `public chart` rejected as invalid credentials: Cause 6. diff --git a/docs/en/solutions/violet_push_goes_to_the_wrong_global_cluster_a_wrong_package_cannot_be_deleted_or_violet_or_kubectl_is_missing_on_ACP.md b/docs/en/solutions/violet_push_goes_to_the_wrong_global_cluster_a_wrong_package_cannot_be_deleted_or_violet_or_kubectl_is_missing_on_ACP.md new file mode 100644 index 000000000..64dd09d4c --- /dev/null +++ b/docs/en/solutions/violet_push_goes_to_the_wrong_global_cluster_a_wrong_package_cannot_be_deleted_or_violet_or_kubectl_is_missing_on_ACP.md @@ -0,0 +1,124 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "4.0" + - "4.1" + - "4.2" + - "4.3" +--- + +# violet push goes to the wrong global cluster, a wrong package cannot be deleted, or violet or kubectl is missing on ACP + +## Issue + +- Cluster plugins pushed for the standby global cluster still show as not listed (`未上架`) there (for example Log Essentials and Log Storage for Elasticsearch), although `violet push` reported no error. +- A package was uploaded by mistake (wrong architecture or wrong build), and the console says it cannot be deleted. +- Deploying an immutable upload package fails because `packtool` cannot run `kubectl`. +- `violet` is not available on the operator workstation, and the question is whether a container of the platform Core already has it, or where a specific version can be downloaded. On ACP 4.2.1 the console has no upload (`上架`) button like the one in 3.18. + +Related articles: "violet push fails with no package found, code 000, HTTP 500 or 502, or invalid reference format on ACP" covers pushes that stop with an error; "Pushed package not listed where expected: global-only plugins, cluster affinity, catalog source, or plugin type on ACP" covers packages that were registered correctly but are not offered on a cluster. + +## Root Cause + +### Cause 1: the platform address given to violet resolves to the other global cluster + +In a primary/standby global setup both clusters share one platform domain, which normally resolves to the primary. When `violet push` is given that domain as `--platform-address` on a host where it resolves to the other global cluster, the packages are registered there, the push reports success, and the intended standby cluster shows nothing new. Running the push with `--platform-address` set to the standby cluster's VIP registers the packages on the standby. + +### Cause 2: a wrongly uploaded package cannot be deleted from the console + +The console refuses to delete the package. `violet push --force` overwrites and updates an existing package with the same image and architecture on the platform, so pushing the correct package for the same image and architecture (for example ARM) with `--force` replaces the wrong upload. + +### Cause 3: packtool fails because kubectl is not in PATH + +Deploying an immutable upload package runs `packtool`, which calls `kubectl` during execution. The failure is not a push to the image registry: `kubectl` is not available through `$PATH` on the node where `packtool` runs. After `kubectl` is installed on that node, the deployment completes. + +### Cause 4: violet is not part of the platform Core and is downloaded separately + +The `violet` tool is not built into the platform Core package, so it cannot be copied out of a running Core container. On ACP 4.2.1 the console has no 3.18-style upload button; packages are uploaded with `violet`. `violet` is downloaded from the Alauda customer portal, under the command-line tools of the matching ACP release. + +## Resolution + +### Cause 1: the platform address given to violet resolves to the other global cluster + +1. On the host that runs `violet`, check where the platform domain resolves, and compare it with the standby cluster's VIP: + + ```bash + getent hosts + ``` + +2. If the domain resolves to the other global cluster, run the push with the standby cluster's VIP as the platform address: + + ```bash + violet push .tgz --platform-address "https://" --platform-username --platform-password '' + ``` + +3. On the standby global cluster, check that the packages are registered: + + ```bash + kubectl get artifact,artifactversion -A | grep -i + kubectl get moduleplugins | grep -i + ``` + +If the push still does not register on the standby cluster, follow the Upload Packages procedure in the documentation for your platform version step by step for the standby cluster. + +### Cause 2: a wrongly uploaded package cannot be deleted from the console + +1. Get the correct package for the same image and architecture as the wrong upload. +2. Push it with `--force`. + + > **Warning:** `--force` overwrites the existing package with the same image and architecture on the platform. The previous upload is replaced, not kept. Precondition: only when the package already on the platform is the wrong one, the console refuses to delete it, and the file you push is the correct package for the same image and architecture. Rollback: none for the overwritten content; push the earlier file again with `--force` if you need it back. + + ```bash + violet push --platform-address= --platform-username= --platform-password='' --force .tgz + ``` + +3. Check that the console now shows the correct version and architecture for the package. + +### Cause 3: packtool fails because kubectl is not in PATH + +1. On the node where `packtool` runs, check for `kubectl`: + + ```bash + command -v kubectl || echo "kubectl not in PATH: $PATH" + ``` + +2. Install a `kubectl` that matches the cluster version (for example copied from a control-plane node) into a directory on `$PATH`, and check that it reaches the cluster: + + ```bash + install -m 0755 /usr/local/bin/kubectl + kubectl version + ``` + +3. Run the immutable upload package deployment again. + +### Cause 4: violet is not part of the platform Core and is downloaded separately + +1. Some platform builds serve `violet` themselves. Check whether your platform answers on this path before looking elsewhere: + + ```bash + curl -k -I https:///platform/tools/download/violet/violet_linux_amd64 + ``` + +2. Otherwise download it from the Alauda customer portal: Products (`产品`) → Product Download (`产品下载`) → the release line in the left sidebar (for example v4.0 → v4.0.9, or v4.2 → v4.2.1) → on the Alauda Container Platform page, under Type (`类型`), select Command-line tools (`命令行工具`) and choose the architecture: x86, ARM, or Hybrid (`异构`). The other types on that page are Core Package (`核心包`) and Extensions (`扩展`). +3. Use the `violet` of the same version as the platform, and check it: + + ```bash + violet version + ``` + +## Diagnostic Steps + +```bash +getent hosts +command -v kubectl +violet version +kubectl get artifact,artifactversion -A | grep -i +``` + +- The push reports success, nothing appears on the standby global cluster, and the platform domain resolves to the other global cluster from the push host: Cause 1. +- A wrong upload that the console will not delete: Cause 2. +- `packtool` stops on a missing `kubectl`: Cause 3. Expected: `command -v kubectl` prints nothing on that node. +- No `violet` binary on the workstation, or no upload button in the 4.2 console: Cause 4. diff --git a/docs/en/solutions/violet_push_succeeds_but_the_plugin_is_not_published_Artifact_ModuleConfig_or_chart_missing_on_ACP.md b/docs/en/solutions/violet_push_succeeds_but_the_plugin_is_not_published_Artifact_ModuleConfig_or_chart_missing_on_ACP.md new file mode 100644 index 000000000..d32a2aee6 --- /dev/null +++ b/docs/en/solutions/violet_push_succeeds_but_the_plugin_is_not_published_Artifact_ModuleConfig_or_chart_missing_on_ACP.md @@ -0,0 +1,169 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: + - "3.16" + - "4.0" + - "4.2" + - "4.3" +--- + +# violet push succeeds but the plugin is not published: Artifact, ModuleConfig, or chart missing on ACP + +## Issue + +`violet push` finishes without an error, yet the plugin cannot be used: + +- Cluster Plugins shows the package as not published (`未上架`), or the Publish Software Package page lists it with no version. OperatorHub can show the same package as not onboarded. +- The uploaded packages menu lists the package with the right upload time while Cluster Plugins shows nothing (for example Alauda Build of NPU Operator and Alauda Build of Node Feature Discovery). +- The marketplace keeps showing the old version after a new one was pushed (for example `hyperflux` 1.4.2 on ACP 4.2.3). +- `kubectl get moduleplugins` lists the plugin, but `kubectl get moduleconfigs` returns nothing for it and the page never offers it (for example Machine Configuration). +- An ARM plugin (for example the DCS provider on ACP 4.2.4) shows `State: Absent` and no version in Cluster Plugins, with the banner "This plugin has not been uploaded to the current cluster", and cannot be deployed. +- The upload command for the HAMi plugin completed, the platform registry is Harbor, and the target cluster has no HAMi plugin; Harbor instances in other regions also lack its charts. + +Related articles: "violet push fails to parse a package, or a new operator version is skipped or missing on ACP" covers push errors and missing versions. "Pushed package not listed where expected: global-only plugins, cluster affinity, catalog source, or plugin type on ACP" covers packages that exist but are shown only on `global`, only in another catalog, or without a status by design. + +## Root Cause + +### Cause 1: Artifact and ArtifactVersion were never created because cluster-transformer stopped processing the ModulePlugin + +`violet push` reports success, the module-plugin object (`mplugin`) exists, often with no `status` field at all, while `artifact` and `artifactversion` for the package do not. `cluster-transformer` has not written those objects, so the plugin page has nothing to render. The upload itself succeeded; restarting Deployment `cluster-transformer` in `cpaas-system` makes the objects appear. Pushing the same tarball again, restarting the OLM catalog Pods, `catalog-operator`, or `marketplace-apiserver`, or re-pushing with a newer `violet` does not help. Why `cluster-transformer` stops processing the ModulePlugin is not known. + +Variants of the same signature: + +- `violet push` prints `already exists, skip push` for the images and `moduleplugin has been updated successfully cluster=global type=plugin`; `kubectl get artifact -A | grep ` returns nothing; the `marketplace-apiserver` log only shows `"OnModulePluginChanged events" logger="storage-provider" eventType="ADDED" name=""`. +- A ModuleConfig exists without a `status`, and none of the `cluster-transformer`, catalog, OLM, or ModuleInfo logs mention the plugin (for example `asm-global` 4.3.4, ARM build, on ACP 4.3.1). +- Only the `moduleplugin` exists for a new version (for example `hyperflux` 1.4.2 pushed with `violet` 4.2.11). Deleting the old version from the uploaded packages page leaves no artifacts at all; restarting `cluster-transformer` and then pushing the plugin again makes the new version display. +- After a package was first pushed to a control-plane node address by mistake and then to the platform access address, the UI still shows it as not uploaded although `apollo`, `cluster-transformer`, and `cluster-transformer-apiservice` are `1/1 Running`. Restarting those Pods brings the plugins to listed (`已上架`). + +### Cause 2: ModulePlugin exists but no ModuleConfig was generated + +`kubectl get moduleplugins` returns the plugin with the pushed chart version (for example `machine-config` `v4.0.18`, chart `ait/chart-machine-config`), `kubectl get moduleconfigs` returns nothing for it, and a repeated `violet push ... --clusters=global --force` changes nothing. This is the same `cluster-transformer` stall as Cause 1, one object later. Restarting `cluster-transformer` produces the ModuleConfig and the plugin can be installed. + +### Cause 3: sentry cannot pull the plugin chart because the registry's anonymous access was removed + +`violet` reports success (for example `found image .../ait/chart-cluster-api-provider-kubeadm:v1.0.8 already exists, skip push` and `moduleplugin cluster-api-provider-kubeadm has been created successfully cluster=global type=plugin`), but Marketplace > Cluster Plugins on `global` shows the plugin with `State: Absent`, an empty Version, and the banner "This plugin has not been uploaded to the current cluster. To install it, please complete the upload process first." No `Artifact`, `ArtifactVersion`, or PackageManifest exists, and the ModulePlugin YAML carries an error that `sentry` failed to pull the plugin chart. + +At the same time the `catalog-operator` log can show `failed to list bundles: rpc error: code = Unavailable ... dial tcp :50051: connect: connection refused`, `failed to populate resolver cache from source platform/cpaas-system`, and `ResolutionFailed` / `CatalogSourcesUnhealthy`; restarting the OLM catalog Pod does not change the plugin state. The ModulePlugin error is the one that decides this cause. + +The anonymous user was deleted from the platform image registry (and the registry credentials were changed), so `sentry` can no longer pull the chart and the plugin never moves from not listed to listed. Re-downloading and re-pushing the package does not help. Adding the anonymous user back to the registry restores the chart pull. + +### Cause 4: The Harbor registry has no mlops project, so the HAMi images and charts were never stored + +The HAMi and DCGM exporter images and charts are pushed under the `mlops` repository path. AI components are not part of the standard product, so Harbor does not contain the `mlops` project by default, and without it the push of those artifacts fails even though the command reports success. After the `mlops` project is created and the package is pushed again, the Harbor of the `global` cluster's region holds `mlops/dcgm-exporter`, `mlops/chart-dcgm-exporter`, `mlops/hami-webui-fe-oss`, `mlops/hami-webui-be-oss`, `mlops/chart-hami-webui`, `mlops/hami`, and `mlops/chart-hami`. When several regions each have their own Harbor, the other Harbor instances still lack them: push once against each Harbor (creating `mlops` wherever it is missing), or use Harbor replication. + +## Resolution + +### Cause 1: Artifact and ArtifactVersion were never created because cluster-transformer stopped processing the ModulePlugin + +Confirm the push reported success, `mplugin` exists, and `artifact` / `artifactversion` for the package do not. Skip this restart when the plugin is global-only and the push targeted another cluster. + +```bash +kubectl get mplugin -A +kubectl get artifact,artifactversion -A | grep -i +kubectl get mplugin -o jsonpath='{.status}{"\n"}' +kubectl -n cpaas-system get deploy cluster-transformer +``` + +> **Warning:** Restarting `cluster-transformer` pauses plugin synchronization until the new Pod is Ready; in-flight plugin installs wait. Precondition: only when `violet push` succeeded, `mplugin` for the package exists, and `artifact` / `artifactversion` for it are missing (or the publish page shows the package with no version). Rollback: none needed. If the page is still empty after the new Pod is Ready, collect the controller log and the `mplugin` YAML before repeating the restart. + +```bash +kubectl -n cpaas-system rollout restart deployment/cluster-transformer +kubectl -n cpaas-system rollout status deployment/cluster-transformer +kubectl get artifact,artifactversion -A | grep -i +``` + +The plugin page should show the package as published, with a version. If a new version still does not show after the restart, push the same package once more: + +```bash +violet push --platform-address= --platform-username= --platform-password='' .tgz +``` + +If the UI still shows the plugin as not uploaded while `artifact` / `artifactversion` now exist, restart `apollo` and `cluster-transformer-apiservice` as well. + +> **Warning:** Restarting `apollo` briefly interrupts the console views it serves, and restarting `cluster-transformer-apiservice` briefly interrupts the plugin API. Precondition: only after the `cluster-transformer` restart, when `artifact` / `artifactversion` exist and the UI still shows the plugin as not uploaded. Rollback: none needed; the Deployments recreate the Pods. + +```bash +kubectl -n cpaas-system get pods | grep -E 'apollo|transformer' +kubectl -n cpaas-system delete pod +``` + +If `artifact` is still missing after the restart, check the package (see the related article on parse errors) or the chart pull (Cause 3) instead of restarting again. + +### Cause 2: ModulePlugin exists but no ModuleConfig was generated + +Run on `global`: + +```bash +kubectl get moduleplugins -o yaml +kubectl get moduleconfigs | grep +``` + +When the ModulePlugin shows the chart version you pushed and no ModuleConfig exists for it, restart `cluster-transformer` with the Warning and commands in Cause 1 (the precondition extends to this signature). Then check again: + +```bash +kubectl get moduleconfigs | grep +``` + +The ModuleConfig appears and the plugin becomes installable from Cluster Plugins. + +### Cause 3: sentry cannot pull the plugin chart because the registry's anonymous access was removed + +Read the ModulePlugin and the `sentry` log for the chart pull error: + +```bash +kubectl get moduleplugin -o yaml | grep -n -i -B2 -A4 'pull\|error' +kubectl -n cpaas-system logs --tail=300 | grep -i '' +kubectl -n cpaas-system logs deploy/catalog-operator --tail=200 | grep -E 'failed to list bundles|resolver cache|CatalogSourcesUnhealthy' +``` + +Check whether the platform image registry still accepts anonymous access. A request without credentials that used to succeed and now returns `401` means anonymous access is gone: + +```bash +curl -k -s -o /dev/null -w '%{http_code}\n' https:///v2//tags/list +``` + +Restore the anonymous user on the platform image registry, configured as before the change (compare with a healthy environment of the same version). If the registry credentials were also changed, make sure the credentials the platform uses for the registry match the new ones. Then check that the ModulePlugin error clears and that `artifact` / `artifactversion` appear: + +```bash +kubectl get artifact,artifactversion -A | grep -i '' +``` + +### Cause 4: The Harbor registry has no mlops project, so the HAMi images and charts were never stored + +Check whether the project exists in each Harbor the clusters pull from: + +```bash +curl -k -s -u ':' "https:///api/v2.0/projects?name=mlops" +curl -k -s -u ':' "https:///api/v2.0/projects/mlops/repositories?page_size=50" | grep -o '"name":"[^"]*"' +``` + +An empty project list is this cause. Create the `mlops` project in that Harbor (Harbor UI > Projects > New Project, or the API below), then push the HAMi package again: + +```bash +curl -k -s -u ':' -H 'Content-Type: application/json' -X POST "https:///api/v2.0/projects" -d '{"project_name":"mlops"}' +violet push --platform-address= --platform-username= --platform-password='' .tgz +``` + +Check that the `mlops` repositories listed in the Root Cause now exist. For Harbor instances in other regions, repeat the project creation and the push against each Harbor, or replicate the `mlops` project with a Harbor replication rule. + +## Diagnostic Steps + +Separate a controller stall from a bad file before restarting anything: + +```bash +violet version +kubectl get mplugin -A +kubectl get artifact,artifactversion -A +kubectl get moduleconfigs | grep +kubectl get moduleplugin -o yaml | grep -n -i 'failed to pull' +tar -tzf .tgz >/dev/null +``` + +- Push success, `mplugin` present (often with no `status`), `artifact` / `artifactversion` absent: Cause 1. +- ModulePlugin present, no ModuleConfig: Cause 2. +- A ModulePlugin error that `sentry` failed to pull the chart: Cause 3. +- HAMi (or another AI component) missing after a successful push to a Harbor registry without an `mlops` project: Cause 4. +- `unexpected EOF` from `violet` or `tar`, or an ArtifactVersion that is `Failed` or skipped as already existing: see the related article on parse errors and missing versions. diff --git a/docs/en/solutions/vmagent_fails_to_scrape_node-exporter_9100_metrics_because_its_basicAuth_does_not_match_node-exporter_on_ACP.md b/docs/en/solutions/vmagent_fails_to_scrape_node-exporter_9100_metrics_because_its_basicAuth_does_not_match_node-exporter_on_ACP.md new file mode 100644 index 000000000..119d96cc6 --- /dev/null +++ b/docs/en/solutions/vmagent_fails_to_scrape_node-exporter_9100_metrics_because_its_basicAuth_does_not_match_node-exporter_on_ACP.md @@ -0,0 +1,84 @@ +--- +kind: + - Troubleshooting +products: + - Alauda Container Platform +ProductsVersion: [] +--- + +# vmagent fails to scrape node-exporter 9100 metrics because its basicAuth does not match node-exporter on ACP + +## Issue + +After the VictoriaMetrics monitoring plugin is deployed, vmagent reports errors when it scrapes the node-exporter endpoint `https://:9100/metrics`, and the node-exporter targets are down. A manual request to the endpoint without credentials is refused with HTTP `401 Unauthorized`. The node-exporter DaemonSet `kube-prometheus-exporter-node` in `cpaas-system` itself is healthy, with all Pods scheduled, ready and available. + +## Root Cause + +The node-exporter of the VictoriaMetrics plugin protects `:9100/metrics` with TLS and HTTP basic authentication. The DaemonSet starts node-exporter with `--web.config.file=/etc/node-exporter/config.yaml`, mounted from the ConfigMap `victoriametrics-exporter-node-config` (key `config.yaml`). The TLS certificate and key come from the Secret `victoriametrics-exporter-node-ca`. The web config lists the users that are accepted (`basic_auth_users`, with a bcrypt hash of each password). + +vmagent sends the username and password configured in the `basicAuth` section of its node-exporter scrape object. When that username and password do not match a user in the node-exporter web config, every scrape is answered with `401` and all node-exporter targets go down. Nothing on the node is broken. The two sides of the credential simply disagree. + +## Resolution + +> **Warning:** Restarting vmagent and the node-exporter DaemonSet causes a short gap in node metrics. Save the objects you change first. Rollback: apply the saved YAML and restart the same components again. Do this only after the diagnostic steps show that a request with the vmagent credentials returns `401`. + +1. Save the node-exporter web config and the Secret that vmagent uses (see Diagnostic Steps for how to find it): + + ```bash + kubectl -n cpaas-system get cm victoriametrics-exporter-node-config -o yaml > /tmp/victoriametrics-exporter-node-config.yaml + kubectl -n cpaas-system get secret -o yaml > /tmp/.yaml + ``` + +2. Make the two sides consistent. Use one of these: + + - Set the username and password in the Secret that the vmagent scrape object references to a user that the node-exporter web config accepts. + - Or put the vmagent username into `basic_auth_users` of `config.yaml` in `victoriametrics-exporter-node-config`, with a bcrypt hash of the vmagent password. A bcrypt hash can be generated, for example, with: + + ```bash + htpasswd -nbBC 10 "" '' | tr -d ':\n'; echo + ``` + +3. Restart node-exporter and vmagent so that both load the new configuration: + + ```bash + kubectl -n cpaas-system rollout restart ds/kube-prometheus-exporter-node + kubectl -n cpaas-system get deploy,sts | grep -i vmagent + kubectl -n cpaas-system rollout restart / + ``` + +4. Check that the node-exporter targets are up again, and that a request with the vmagent credentials now returns metrics (see Diagnostic Steps, step 3). + +If the change is reverted after a plugin reconcile or upgrade, make the same change through the plugin configuration instead of editing the generated objects only. + +## Diagnostic Steps + +1. Read how the node-exporter is started and where its web config comes from: + + ```bash + kubectl -n cpaas-system get ds kube-prometheus-exporter-node -o yaml | grep -n -E 'web.config.file|configMap|secretName|name: ' + kubectl -n cpaas-system get cm victoriametrics-exporter-node-config -o jsonpath='{.data.config\.yaml}'; echo + ``` + + Expected: `--web.config.file=/etc/node-exporter/config.yaml`, and a `basic_auth_users` map in `config.yaml`. Note the username(s). + +2. Find the basicAuth that vmagent uses for node-exporter, and decode its username and password: + + ```bash + kubectl -n cpaas-system get servicemonitor,vmnodescrape,vmservicescrape 2>/dev/null | grep -i node + kubectl -n cpaas-system get / -o yaml | grep -n -A6 basicAuth + kubectl -n cpaas-system get secret -o jsonpath='{.data.username}' | base64 -d; echo + kubectl -n cpaas-system get secret -o jsonpath='{.data.password}' | base64 -d; echo + ``` + + A username that is not in `basic_auth_users` already confirms the mismatch. + +3. Test the endpoint from a node with and without the vmagent credentials: + + ```bash + curl -k -s -o /dev/null -w '%{http_code}\n' https://:9100/metrics + curl -k -s -o /dev/null -w '%{http_code}\n' -u ':' https://:9100/metrics + ``` + + Expected for this issue: `401` for both requests. After the fix, the request with credentials returns `200`. + +4. Rule out look-alikes. A connection error or timeout instead of `401` points to the network or to the node-exporter Pod, not to credentials. A `200` with credentials but down targets points to the scrape configuration (port, scheme, timeout), not to basicAuth.