🚨 ArgoCD Deployment Failed: agentic-platform-engineering-demo

[github-actions]

ArgoCD Deployment Failure

Application: agentic-platform-engineering-demo
Timestamp: 2026-03-29T00:04:27.256Z

Cluster Information

Field	Value
Cluster Name	msftgbb
Resource Group	agentic-platform-engineering
Namespace	default

Application Status

Field	Value
Health Status	Degraded
Sync Status	Synced
Revision	3cbff7db97e79fffbefa46861326d0e5ed7c010e
Repository	https://github.com/microsoftgbb/agentic-platform-engineering.git

Error Message

successfully synced (all tasks run)

Troubleshooting Commands

# Check application status in ArgoCD
argocd app get agentic-platform-engineering-demo

# Check pods in namespace
kubectl get pods -n default

# Describe failed pods
kubectl describe pods -n default

# Get pod logs
kubectl logs -n default <pod-name>

# Check events
kubectl get events -n default --sort-by='.lastTimestamp'

Quick Links

• ArgoCD UI
• Source Repository

---

This issue was automatically created by ArgoCD Notifications

[github-actions]

🤖 Cluster Doctor is on the case!

Work has begun on this issue. A GitHub Action has been spun up to analyze and diagnose the cluster.

🔗 View the running GitHub Action

[github-actions]

🤖 Cluster Doctor Diagnostic Report

Cluster Identity Verified ✅

• AKS cluster msftgbb in resource group agentic-platform-engineering confirmed via Azure management API + kubectl API server response (canadacentral, AKS 1.33.7).

---

Root Cause: Image Name Typo → ImagePullBackOff → Stuck Rolling Update

Finding

A one-character typo in Act-3/argocd/apps/broken-aks-store-all-in-one.yaml caused the order-service Deployment to reference a non-existent container image:

- image: ghcr.io/azure-samples/aks-store-demo/order-servic:2.1.0
+ image: ghcr.io/azure-samples/aks-store-demo/order-service:2.1.0

The trailing e was missing from order-service, introduced in commit 3cbff7db (2026-03-26T19:36:27Z).

---

Evidence

Pod Status

NAME                                READY   STATUS             RESTARTS   AGE
order-service-74887bf86-lxmxs       0/1     ImagePullBackOff   0          2d4h   ← BROKEN (new RS, bad image)
order-service-575df9db99-x6kmq      1/1     Running            0          ~2m    ← OK (old RS, correct image)

Pod Events (13,881 failures over 2d4h)

Warning  Failed   3m34s (x13881 over 2d4h)  kubelet  Error: ImagePullBackOff
Normal   BackOff  3m45s (x13880 over 2d4h)  kubelet  Back-off pulling image "ghcr.io/azure-samples/aks-store-demo/order-servic:2.1.0"

Deployment Condition

Conditions:
  Progressing  False  ProgressDeadlineExceeded   ← rolling update permanently blocked
  Available    True   MinimumReplicasAvailable    ← old pod kept alive
OldReplicaSets:  order-service-575df9db99 (1/1) ← correct image, kept running
NewReplicaSet:   order-service-74887bf86  (1/1) ← broken image, stuck

ArgoCD Health Explanation

ArgoCD Sync Status = Synced (repo matches cluster — both have the typo). Health = Degraded because the Deployment's ProgressDeadlineExceeded condition causes ArgoCD to mark the app unhealthy.

---

Impact Assessment

Component	Impact
order-service	Degraded — rolling update stuck; old pod still serving traffic
End-to-end order flow	Functional — old replica set pod (575df9db99) kept running by Kubernetes RollingUpdate safety
ArgoCD app health	Degraded — will resolve after fix is synced
Other services	✅ All 1/1 Running

---

Remediation

A PR has been created to fix the typo:
➡️ PR #41: #41

After Merge — Validation Steps

# 1. Confirm new pod is running with correct image
kubectl get pods -n default -l app=order-service

# 2. Confirm deployment is healthy
kubectl get deployment order-service -n default
# Expected: READY 1/1, UP-TO-DATE 1, AVAILABLE 1

# 3. Confirm no more ImagePullBackOff events
kubectl get events -n default | grep order-service

Rollback (if needed)

kubectl rollout undo deployment/order-service -n default

---

Diagnosed and remediated autonomously by Cluster Doctor at 2026-03-29T00:07Z

[github-actions]

🔄 Deployment Failed Again

Timestamp: 2026-03-29T00:13:58.448Z
Revision: 3cbff7db97e79fffbefa46861326d0e5ed7c010e

Error:

successfully synced (all tasks run)

[github-actions]

🔄 Deployment Failed Again

Timestamp: 2026-03-29T00:14:49.914Z
Revision: 3cbff7db97e79fffbefa46861326d0e5ed7c010e

Error:

successfully synced (all tasks run)