Skip to content

proxy: label the request-outcome metric by destination node to isolate a drained/stale node #3759

Open

Description

@AdaAibaby

Summary

The request-outcome metric added in #3758 (client_proxy.request.outcome, labeled outcome) is not labeled by node. That is enough to see that a spike of upstream_unreachable is happening, but not which orchestrator node the unreachable dials are going to — which is exactly what you need to tell "one node drained and its routing records are stale" apart from "the whole fleet is sick".

Motivation

During a node scale-down, client-proxy keeps resolving sandboxes whose Redis routing record still points at a removed node (the record's TTL is sandbox_max_length_in_hours, deleted only on a clean MarkStopping; an abruptly-killed node leaves it live for hours). Resolution succeeds — it returns the stale OrchestratorIP — and the failure only surfaces at dial time, as the upstream_unreachable outcome. Without a node label on that outcome, the dashboard shows a 502 spike but not the offending node, so on-call still has to scrape logs for dial tcp <nodeIP>:5007 to find it.

This is the missing half of the node-churn observability story: #3692 split the message, #3758 added the outcome dimension, this adds the node dimension so upstream_unreachable{node="..."} points straight at the drained node.

Proposal

Attribute each proxied-request outcome to the destination node:

  1. Add Node string to pool.Destination — client-proxy sets it to the resolved SandboxInfo.OrchestratorID (the worker instance; bounded cardinality, same label the hugepage-ceiling metric already uses as node.id).
  2. Extend the OnRequestOutcome callback (from feat(proxy): request-outcome metric distinguishing routing (500) / upstream (502) / client-closed (499) #3758) to carry the node: func(ctx, outcome, node string). The pool's ErrorHandler/ModifyResponse already hold the Destination, so they pass t.Node; the pre-routing branches in handler() (resolution never produced a destination) pass "" (recorded as node="unknown").
  3. client-proxy records outcome + node; the orchestrator sandbox proxy leaves Node empty (its "node" is itself — not meaningful), so its metric is unchanged.

Result: client_proxy.request.outcome{outcome="upstream_unreachable", node="<orchestrator-id>"} — a per-node unreachable rate that isolates a drained/stale node directly.

Depends on #3758 (same metric + callback). Refs #3757, #3691.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions