You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
proxy: label the request-outcome metric by destination node to isolate a drained/stale node #3759
The request-outcome metric added in #3758 (client_proxy.request.outcome, labeled outcome) is not labeled by node. That is enough to see that a spike of upstream_unreachable is happening, but not which orchestrator node the unreachable dials are going to — which is exactly what you need to tell "one node drained and its routing records are stale" apart from "the whole fleet is sick".
Motivation
During a node scale-down, client-proxy keeps resolving sandboxes whose Redis routing record still points at a removed node (the record's TTL is sandbox_max_length_in_hours, deleted only on a clean MarkStopping; an abruptly-killed node leaves it live for hours). Resolution succeeds — it returns the stale OrchestratorIP — and the failure only surfaces at dial time, as the upstream_unreachable outcome. Without a node label on that outcome, the dashboard shows a 502 spike but not the offending node, so on-call still has to scrape logs for dial tcp <nodeIP>:5007 to find it.
This is the missing half of the node-churn observability story: #3692 split the message, #3758 added the outcome dimension, this adds the node dimension so upstream_unreachable{node="..."} points straight at the drained node.
Proposal
Attribute each proxied-request outcome to the destination node:
Add Node string to pool.Destination — client-proxy sets it to the resolved SandboxInfo.OrchestratorID (the worker instance; bounded cardinality, same label the hugepage-ceiling metric already uses as node.id).
client-proxy records outcome + node; the orchestrator sandbox proxy leaves Node empty (its "node" is itself — not meaningful), so its metric is unchanged.
Result: client_proxy.request.outcome{outcome="upstream_unreachable", node="<orchestrator-id>"} — a per-node unreachable rate that isolates a drained/stale node directly.
Depends on #3758 (same metric + callback). Refs #3757, #3691.
Activity