feat(helm): add PodMonitors for the workloads that serve metrics

.agents/skills/update-against-main/SKILL.md

@@ -0,0 +1,35 @@
+---
+name: update-against-main
+description: Merge agent-substrate/substrate main into the kagent-dev/substrate fork's main branch, resolve conflicts, validate the result, and safely update the fork. Use only when explicitly synchronizing the fork's main branch with upstream main. Do not use for updating, rebasing, or resolving conflicts in feature branches or pull requests.
+---
+
+# Update Against Main
+
+This skill applies only to synchronizing the fork's `main` branch. Do not invoke it for a feature branch or PR merely because that branch is behind or conflicts with `main`.
+
+1. Confirm the worktree, current branch, tracking branch, and remotes. Do not disturb unrelated changes.
+2. Fetch `origin/main` and `upstream/main`, inspect their divergence, and create a dated backup branch from `origin/main`.
+3. Rebuild `main` from `upstream/main` by replaying only intentional fork feature commits in dependency order. Drop merge commits and fork commits superseded by upstream.
+4. Resolve conflicts in favor of current upstream APIs while preserving the remaining fork features. Inspect the resulting diff and linear history.
+5. Keep Helm charts synchronized with their corresponding manifests. When either changes, inspect and update the other while preserving intentional Helm templating and conditionals, then run `make verify-helm-template` and `make verify-crd-chart` and compare any relevant resources not covered by those checks.
+6. Run `make test` and `make verify`.
+7. Run the real Kind E2E matrix from `.github/workflows/pr-workflow.yaml`, but use agentgateway for all fork testing:
+   - Use a dedicated cluster name and kubeconfig; record the temporary assets created by this run. Before recreating with `hack/create-kind-cluster.sh`, delete any old cluster owned by this sync using `hack/kind.sh delete cluster --name "$cluster_name"` with its dedicated `KUBECONFIG`.
+   - Install the control plane with `hack/install-ate-kind.sh --deploy-ate-system --atenet-dataplane=agentgateway`.
+   - Deploy the micro-VM demo with `hack/run-microvm-demo-kind.sh --skip-control-plane` so it does not reinstall the control plane.
+   - Deploy the gVisor counter demo and both standard egress demos.
+   - The full gVisor suite: `hack/run-e2e-kind.sh -v -args --no-color`
+   - The full micro-VM suite with the CI environment: `E2E_SANDBOX_CLASS=microvm hack/run-e2e-kind.sh -v -args --no-color`
+   - Switch egress to agentgateway sdsmint, then run the MITM trust and targeted networking lanes for both runtimes exactly as the workflow specifies.
+   - Verify the live router and egress workloads use agentgateway. Never use Envoy for fork validation.
+8. Treat `go test ./internal/e2e/...` without `-args --e2e` as compilation/package testing, not E2E coverage.
+9. Do not push when unit, verification, or E2E checks fail or cannot run. Report the exact blocker instead.
+10. After all checks pass, verify the worktree and rewritten commits, then update the fork with `git push --force-with-lease origin main`. Never use an unguarded force push.
+11. Clean up temporary assets before finishing, including on failure or cancellation:
+   - Stop this run's test/install processes and port-forwards. Save any diagnostics needed to explain failures before tearing down workloads.
+   - Delete the task-owned Kind cluster with `hack/kind.sh delete cluster --name "$cluster_name"` using its dedicated `KUBECONFIG`. Verify both the cluster and its node containers are gone before removing the kubeconfig.
+   - Remove this run's disposable assets: generated micro-VM disks and images, downloaded bundles, build outputs, scratch scripts, and temporary kubeconfigs. Remove task-only Docker images, containers, and volumes once no longer in use. Preserve shared assets, caches, registries, and unrelated clusters; do not use global Docker prune commands.
+   - After a successful push, remove clean temporary worktrees with `git worktree remove` from another checkout. Preserve backup branches, unpushed commits, uncommitted changes, and diagnostics needed for unresolved failures.
+   - If teardown stalls (for example, Docker reports no exit event), inspect only the task's node containers, retry scoped deletion once, and report any remaining resources and exact blocker. Do not restart the global Docker daemon or kill unrelated processes. Report cleanup separately from validation so leftover assets are not hidden by passing tests.
+
+Use the current CI workflow as the source of truth for cluster setup, images, demos, runtime coverage, and environment variables, with the agentgateway-only override above. Never claim E2E passed unless workloads ran against the cluster.

.github/workflows/helm-e2e.yaml

@@ -0,0 +1,125 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: helm-e2e
+on:
+  pull_request:
+  push:
+    branches: [main]
+permissions:
+  contents: read
+jobs:
+  e2e-test:
+    runs-on: ubuntu-latest
+    env:
+      VERSION: helm-e2e
+      E2E_ATENET_DATAPLANE: agentgateway
+      E2E_CREDENTIAL_PROVIDER: "1"
+      E2E_EGRESS_MITM: "1"
+    steps:
+    - name: Checkout
+      uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0
+    - name: Setup Go
+      uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5.6.0
+      with:
+        go-version-file: go.mod
+    - name: Setup Helm
+      uses: azure/setup-helm@v4
+    - name: Cache micro-VM assets
+      uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0
+      with:
+        path: bin/microvm-assets/amd64
+        key: microvm-assets-amd64-${{ hashFiles('hack/microvm-assets/assemble.sh') }}
+    - name: Enable KVM
+      run: |
+        echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' \
+          | sudo tee /etc/udev/rules.d/99-kvm4all.rules
+        sudo udevadm control --reload-rules
+        sudo udevadm trigger --name-match=kvm
+    - name: Create cluster
+      run: hack/create-kind-cluster.sh
+    - name: Label nodes with the installed version
+      run: kubectl label nodes --all ate.dev/substrate-version=${VERSION}
+    - name: Create install namespace
+      run: kubectl create namespace ate-system
+    - name: Install observability fixtures
+      run: |
+        kubectl apply -f manifests/ate-install/kind/otel-collector.yaml
+        kubectl apply -f manifests/ate-install/kind/prometheus.yaml
+    - name: Build chart images
+      run: |
+        # Pushed under the image's real path (kagent-dev/substrate/<component>):
+        # the chart composes {registry}/{repository}/{component}, so the local
+        # registry serves each image where the default repository expects it --
+        # the same path-preserving rule a production mirror follows.
+        for component in ateapi atecontroller atelet podcertcontroller atenet credential-provider/kubernetes-secrets; do
+          KO_DOCKER_REPO="localhost:5001/kagent-dev/substrate/${component##*/}" \
+            ./hack/run-tool.sh ko build --bare --tags helm-e2e \
+              --platform linux/amd64 "./cmd/${component}"
+        done
+    - name: Install Agent Substrate with Helm
+      run: |
+        helm upgrade --install substrate-crds charts/substrate-crds
+        helm upgrade --install substrate charts/substrate \
+          --namespace ate-system \
+          --create-namespace \
+          -f internal/e2e/suites/credentials/values.yaml \
+          --set image.registry=localhost:5001 \
+          --set image.tag=helm-e2e \
+          --set 'atelet.extraArgs[0]=--localhost-registry-replacement=kind-registry:5000' \
+          --set otel.endpoint=http://opentelemetry-collector.otel-system.svc:4317 \
+          --set postgres.resources.requests.cpu=500m
+    - name: Bootstrap mTLS authorities
+      run: |
+        hack/install-ate-kind.sh --create-podcertificate-controller-cas
+        hack/install-ate-kind.sh --create-jwt-authority-pool-secret
+        hack/install-ate-kind.sh --create-actor-id-ca-pool-secret
+        hack/install-ate-kind.sh --create-actor-id-ca-certs-secret
+        hack/install-ate-kind.sh --create-api-authentication-config
+        hack/install-ate-kind.sh --create-egress-mitm-ca-pool-secret
+    - name: Wait for Helm install
+      run: |
+        helm upgrade substrate charts/substrate \
+          --namespace ate-system \
+          --reuse-values \
+          --wait --timeout=10m
+    - name: Enable NFS
+      run: |
+        sudo modprobe nfs || true
+        sudo modprobe nfsd || true
+    - name: Install CSI NFS driver
+      run: hack/install-ate-kind.sh --setup-csi=nfs
+    - name: Deploy micro-VM counter demo
+      # The deploy creates the substrate ActorTemplate and waits for its golden
+      # snapshot internally; the ActorTemplate CRD (and its Ready condition)
+      # no longer exists to wait on.
+      run: hack/run-microvm-demo-kind.sh --skip-control-plane
+    - name: Deploy gVisor counter demo
+      run: hack/install-ate-kind.sh --deploy-demo-counter
+    - name: Deploy egress demo
+      run: hack/install-ate-kind.sh --deploy-demo-egress-mitm
+    - name: Run E2E tests (gVisor)
+      run: hack/run-e2e-kind.sh -v -args --no-color
+    - name: Run E2E tests (micro-VM)
+      env:
+        E2E_SANDBOX_CLASS: microvm
+      run: hack/run-e2e-kind.sh ./internal/e2e/suites/demo -v -args --no-color
+    - name: Dump diagnostics on failure
+      if: failure()
+      run: |
+        kubectl --context kind-kind get workerpool,pods -A -o wide || true
+        for p in $(kubectl --context kind-kind get pods -n ate-system -o name 2>/dev/null); do
+          echo "=== logs: ate-system/${p} ==="
+          kubectl --context kind-kind logs -n ate-system "$p" --all-containers --tail=300 || true
+        done

.github/workflows/helm-verify.yaml

@@ -0,0 +1,39 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Enforces the chart checks that were previously developer-run only:
+# verify-helm-template's own comment says "Used in CI", but no workflow ran
+# it -- so a chart change that broke hack/render-manifests.sh shipped a green
+# PR and was caught in review by hand.
+name: helm-verify
+on:
+  pull_request:
+  push:
+    branches: [main]
+permissions:
+  contents: read
+jobs:
+  helm-verify:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: azure/setup-helm@v4
+      - name: Lint the chart
+        run: helm lint charts/substrate charts/substrate-crds
+      - name: Verify committed manifests match the chart output
+        run: make verify-helm-template
+      - name: Install helm-unittest
+        run: helm plugin install https://github.com/helm-unittest/helm-unittest --version v1.0.2
+      - name: Run the chart unit tests
+        run: helm unittest charts/substrate

.github/workflows/pr-workflow.yaml

@@ -60,15 +60,11 @@ jobs:
   e2e-test-matrix:
     name: E2E (${{ matrix.dataplane }})
     runs-on: ubuntu-latest
-    continue-on-error: ${{ matrix.experimental }} # TODO: Make AgentGateway required once tests show stability
     strategy:
       fail-fast: false
       matrix:
         include:
-        - dataplane: envoy
-          experimental: false
         - dataplane: agentgateway
-          experimental: true
     env:
       E2E_ATENET_DATAPLANE: ${{ matrix.dataplane }}
     steps:
@@ -102,7 +98,7 @@ jobs:
     - name: Install Agent Substrate (${{ matrix.dataplane }})
       # The dataplane selection applies to both the ingress router and egress
       # gateway.
-      run: hack/install-ate-kind.sh --deploy-ate-system --atenet-dataplane=${{ matrix.dataplane }}
+      run: hack/install-ate-kind.sh --deploy-ate-system --atenet-dataplane=${{ matrix.dataplane }} --rollout-timeout=300s
     - name: Enable NFS
       # Load NFS kernel modules so in-cluster NFS server and CSI driver can run.
       run: |
@@ -112,11 +108,8 @@ jobs:
       run: hack/install-ate-kind.sh --setup-csi=nfs
     - name: Deploy micro-VM counter demo
       # Stages the (cached) assets into the cluster's rustfs and deploys the
-      # counter-microvm demo onto the control plane installed above. The demo
-      # redeploys the control plane, so retain the selected dataplane.
-      env:
-        ATE_ATENET_DATAPLANE: ${{ matrix.dataplane }}
-      run: hack/run-microvm-demo-kind.sh
+      # counter-microvm demo onto the control plane installed above.
+      run: hack/run-microvm-demo-kind.sh --skip-control-plane
     - name: Deploy gVisor counter demo
       run: hack/install-ate-kind.sh --deploy-demo-counter
     - name: Deploy egress demos
@@ -196,8 +189,7 @@ jobs:
         kubectl --context kind-kind get pods -A -l ate.dev/worker-pool \
           -o 'custom-columns=:.metadata.namespace,:.metadata.name' --no-headers 2>/dev/null \
           | while read -r ns name; do dump "$ns" "$name"; done
-  # Preserve the required-check name while the concrete Envoy and AgentGateway
-  # executions run as entries in the shared matrix above.
+  # Preserve the required-check name for the dataplane matrix above.
   e2e-test:
     name: e2e-test
     needs: e2e-test-matrix