kuberesolver watch fails with 403 despite RBAC allowing it — login 504 on self-hosted Cloud (K8s 1.33)

[saranshflip]

Describe the bug

On a self-hosted Pixie Cloud deployment, the login flow fails at the final step with a 504 Gateway Timeout returned by the cloud-proxy (openresty/nginx) proxy on POST /api/auth/login.

Root cause traced to api-server's gRPC client-side load balancer (github.com/sercand/kuberesolver/v3, registered via kuberesolver.RegisterInCluster() in src/shared/services/service_flags.go). It resolves backend addresses (e.g. auth-service) via kubernetes:///<service>.<namespace>:<port> URIs, which requires watching pods/services/endpoints through the Kubernetes API. api-server's logs are flooded with:

kuberesolver: watching ended with error='invalid response code 403', will reconnect again

Because the watch never succeeds, api-server can never resolve auth-service's pod IP, the gRPC call hangs, and cloud-proxy eventually times out with 504.

Note: I was not able to conclusively confirm this is a reproducible bug in Pixie's code versus an authorization policy specific to our cluster's platform — see the "What we ruled out" section below. Posting here in case others hit the same symptom, and because the missing discovery.k8s.io/endpointslices RBAC grant seems like a potential gap regardless.

What we ruled out on our end

• kubectl auth can-i {get,list,watch} {pods,services,endpoints,endpointslices} --as=system:serviceaccount:plc:default -n plc → yes for everything, including after manually patching pl-kuberesolver-role (k8s/cloud/base/kuberesolver_role.yaml) to add discovery.k8s.io/endpointslices.
• No NetworkPolicy present in the namespace.
• No Calico/Cilium/Istio policies installed on the cluster.
• Gatekeeper (OPA) webhooks only intercept CREATE/UPDATE on specific object kinds (deployments, services, pods, etc.) — never GET/WATCH — so irrelevant here.
• Restarting the api-server pod (fresh service-account token) didn't change anything.
• Plain DNS resolution of auth-service.plc.svc.cluster.local works fine and returns the correct pod IP — it's specifically the K8s-API-watch-based resolver that fails, not DNS.

To Reproduce

1. Deploy Pixie Cloud (self-hosted) to a K8s v1.33.x cluster.
2. Port-forward cloud-proxy and browse to /auth/login.
3. Log in with valid credentials (Hydra/Kratos flow completes).
4. On the final POST /api/auth/login call, observe 504 Gateway Timeout in the browser and kuberesolver: watching ended with error='invalid response code 403' spamming in api-server logs.

Expected behavior
Login completes successfully; api-server resolves auth-service (and other backends) via kuberesolver without 403s.

Questions for maintainers/community

1. Does kuberesolver v3.0.0 (as pinned in go.mod) ever watch discovery.k8s.io/v1 EndpointSlices instead of/in addition to core v1 Endpoints? kuberesolver_role.yaml only grants pods/services/endpoints, not endpointslices — this looks like a potential RBAC gap for clusters/K8s versions where resolution behavior differs.
2. This cluster is on K8s v1.33.7, where core v1 Endpoints is now deprecated (the API server emits a deprecation warning on every request) in favor of discovery.k8s.io/v1 EndpointSlice. Should Pixie's client-side gRPC resolver move to EndpointSlices for forward compatibility with 1.33+?
3. Has anyone else seen kuberesolver return 403 on a watch that kubectl auth can-i says should be allowed? We suspect this may be an authorization layer specific to our managed cluster platform rather than a Pixie bug, but wanted to check for prior reports of this interaction with kuberesolver/client-go.

Logs
api-server logs show the repeating kuberesolver: watching ended with error='invalid response code 403' message. (./px collect-logs not run, since this is a self-hosted deployment and root cause was isolated to a specific pod/service, not the full vizier/cloud log bundle.)

App information

• Pixie Cloud version: 0.1.9 (image tag across all cloud-* services)
• Pixie CLI version: 0.8.8+Distribution.042e356.20250117230946.1.Homebrew
• K8s cluster version: v1.33.7
• Node OS/Kernel: Debian GNU/Linux 11 (bullseye), 5.10.0-19-cloud-amd64, containerd 1.6.36
• Browser: N/A (symptom only visible in-browser; root cause is server-side)

Additional context
Relevant RBAC definition: k8s/cloud/base/kuberesolver_role.yaml. kuberesolver registration: src/shared/services/service_flags.go.