Count all observed network failures by bounded error code

[hiroTamada]

Summary

Count every accepted, deduplicated Network.loadingFailed outcome in the existing always-on in-memory monitor, independently of customer telemetry. Add kernel_chromium_network_failures_total{error_code,canceled} to the existing /metrics collector without another CDP connection or changes to ChromeCollector, capture domains, body/script capture, or lifecycle machinery.

• Carry explicit finished/failed terminal kind: empty error text is still a failure; a finished event cannot increment failures even if it contains error fields.
• Strict finite allowlist of 83 Chromium codes, including both ERR_CONNECTION_TIMED_OUT and ERR_TIMED_OUT, connection/DNS/SSL/certificate/proxy/tunnel/HTTP2/QUIC/blocked/abort errors. Missing/empty text becomes unknown; everything else becomes other. Never label with raw error text.
• Retain the CDP canceled boolean separately so cancellations with missing/unrecognized text remain distinguishable. Blocked outcomes still count without adding a blockedReason dimension.
• Emit all 170 possible series from the first scrape, including zeros before capture readiness. Later first failures can then be diffed against zero rather than becoming the first sample of a sparse series.
• Preserve reset/completed/readiness metrics, the 8,192-identity first-terminal-wins retention bound, generation handling, and totals across reconnects. New API-owned monitor state starts at zero. Summed failures never exceed completed; the RESET bucket summed across cancellation states equals the legacy reset counter.

Related collector change: https://github.com/kernel/infra/pull/1336. Builds on the merged always-on monitor foundation in #387, using a fresh branch from current main.

Rollout and compatibility

Apply collector label preservation first, image second. The generic kernel_ filter already passes the name, but the old collector's keep_keys would erase its dimensions. The paired collector retains only this family's bounded labels alongside metro/platform and normalizes unsupported codes. Old images have no failure family; existing metrics remain compatible during mixed-version rollout. Future taxonomy additions need coordinated image/collector updates. Per-target first-scrape baselines and existing CDP coverage/dedup limitations still apply; these are observed terminal outcomes, not lossless socket counts.

No control-plane/metro-api change, HTTP-status metrics, request-type/top-level dimension, dashboard, alert, deployment, or Chromium patch is included.

Validation

With Chromium 152.0.7977.82 installed locally:

• GOCACHE=/tmp/go-build KERNEL_CDPMONITOR_CHROME_E2E=1 go test -count=1 -race -v ./lib/cdpmonitor ./lib/metrics ./cmd/api/api passed. No CDP monitor opt-in tests skipped.
• Full non-Docker package suite passed on the final run with the same environment: go test -count=1 -race -v $(go list ./... | grep -v /e2e$). 574 top-level tests passed; 62 unrelated REPL/recording tests skipped because esbuild/ffmpeg are absent. An earlier broader run reproduced the existing TestUpstreamManagerDetectsChromiumAndRestart TempDir cleanup race; unchanged, passed on rerun.
• GOCACHE=/tmp/go-build go vet ./... passed.
• Table/race tests cover every supported code, strict normalization, empty/arbitrary text, cancellations and blocked outcomes, finished-vs-failed dispatch, exact bucket totals, duplicate collisions, bounded retention, detached snapshots, concurrent events/scrapes, and reconnect generations.
• Real local HTTP/TCP fixtures, confirmed through an independent CDP observer: 10 resets produce exactly +10 RESET/+10 failures/+10 completed; refusal and abort each add exactly one correctly labeled failure; HTTP 200/500 add completed only. A TCP listener that never answers TLS produces a real net::ERR_TIMED_OUT after Chromium's 30s SSL handshake deadline, with exactly one timeout failure. These are browser-produced failures, not injected CDP outcomes.
• Existing frame/worker, telemetry off/on/off, same-URL reconnect, actual Chrome restart, cleanup-failure and race regressions pass. New assertions retain known-code totals across reconnect/restart.
• Terminal dispatch microbenchmark: 2,149 ns/op, 303 B/op, 6 allocs/op (local ingestion only, not browser workload overhead).

No full Docker image, production, suspend/resume, or snapshot-fork validation was performed. Known unrelated same-process-iframe telemetry re-enable behavior is unchanged.

---

Note

Medium Risk
Expands the public Prometheus surface with 170 new labeled series and requires coordinated collector rollout; core request handling is unchanged but mis-timed deploy could drop label dimensions in downstream metrics.

Overview
Adds labeled failure breakdown to the always-on CDP network monitor and /metrics, without changing telemetry capture or the existing reset/completed/up counters.

Network.loadingFailed is now classified with an explicit terminal kind so loadingFinished never increments failure buckets. Failures are bucketed by a bounded error_code (83 allowlisted Chromium net::ERR_* values plus unknown/other) and CDP’s canceled flag. kernel_chromium_network_failures_total{error_code,canceled} is exported with all 170 series pre-emitted at zero on first scrape; legacy ERR_CONNECTION_RESET totals still align with kernel_chromium_connection_resets_total.

NetworkMetrics / NetworkSnapshot now include a detached Failures map; ApiService.NetworkMetrics passes it through to metrics.NetworkCollector. Docs call for rolling out the paired infra collector label preservation before this image so dimensions are not stripped.

^{Reviewed by Cursor Bugbot for commit 859a089. Bugbot is set up for automated code reviews on this repo. Configure here.}