Skip to content

feat(upstream): follow the Pod behind a Kubernetes endpoint for slow start - #13958

Open
AlinsRan wants to merge 1 commit into
apache:masterfrom
AlinsRan:feat/slow-start-kubernetes
Open

AlinsRan wants to merge 1 commit into
apache:masterfrom
AlinsRan:feat/slow-start-kubernetes

Conversation

@AlinsRan

Copy link
Copy Markdown
Contributor

Follows #13941, which covered the general data plane capability. This one teaches it about Kubernetes, and settles the naming.

What was missing

An upstream backed by Kubernetes discovery already ramps: endpoints reach the balancer as ordinary nodes, a new one is new to the node set, and it warms up. What slow start could not see is which workload answers on an address.

A cluster recycles addresses quickly, so both halves of that were wrong:

  • a Pod that took over an address another Pod had just released inherited its maturity and went straight to full weight, which is exactly the burst slow start exists to prevent;
  • a Pod that moved to another address was ramped again from the minimum although it had been serving all along.

Pod identity

apisix/discovery/kubernetes/core.lua now carries targetRef.uid in the node metadata, for EndpointSlice and for the older Endpoints API, and node_id in apisix/slow_start.lua prefers it:

endpoint lifecycle identity
backed by a Pod targetRef.uid and the port
no targetRef (an external address, say) host and port, as before
a configured node host and port, a domain node its hostname

Everything else follows from that: a Pod changing address keeps its ramp, an address changing Pod starts a new one. The uid also makes the node set change when a Pod is replaced in place, so the picker is rebuilt rather than kept from the previous version.

slow_start.default_weight

Kubernetes endpoints carry no weight of their own, so the registry gives every one of them the same default_weight. An upstream that wants its ramp to end somewhere else can now say so:

{
  "type": "roundrobin",
  "service_name": "default/orders:http",
  "discovery_type": "kubernetes",
  "slow_start": {
    "slow_start_time_seconds": 300,
    "min_weight_percent": 1,
    "default_weight": 100
  }
}

It is rejected on anything but a discovery_type: kubernetes upstream. Elsewhere the weights a registry hands out mean something - a Nacos service with a 10/20/70 split, for one - and silently flattening them would be worse than not offering the field.

Deliberately out of scope

Two things the RFC lists for this stage are not here, and I would rather say so than leave them looking forgotten:

  • Sharing endpoint observation times between upstreams. The RFC records first_observed_at at the discovery layer so two upstreams referencing the same Service agree on when an endpoint appeared. Each upstream already keeps its own lifecycle, and the only case the shared record changes is an upstream created while the Service was already running, whose nodes are a mature baseline either way.
  • Making the LIST transactional. A relist republishes slices one at a time, so a node can briefly leave the set. The tombstone window already covers that: it comes back inside one slow start window and resumes its ramp rather than restarting it. Buffering a whole LIST before publishing it is a change to the discovery module's own contract, not to slow start, and belongs with a change that has a reason of its own.

Naming

@shreemaan-abhishek's review of #13941 pointed out that the feature answered to two names: the configuration was warm_up_conf, while the module, the shared dict, the log lines and the test files all said slow start - and the key held a field called slow_start_time_seconds, so both names met inside one object.

Slow start is what the rest of the world calls this. Envoy has slow_start_config, NGINX Plus a slow_start parameter on an upstream server, HAProxy slowstart. The weight curve here is Envoy's and two of the field names already match it, so the configuration key becomes slow_start and everything reads the same way. It also drops a _conf suffix that no other upstream field carries.

The field names inside it do not change, and nothing has been released with the old key, so there is nothing to migrate.

Tests

  • t/node/upstream-slow-start.t — a Pod keeps its ramp across an address change, an address taken over by a different Pod ramps again, and default_weight is the target a Kubernetes endpoint ramps to while a Nacos upstream keeps its own weights.
  • t/admin/upstream-slow-start.tdefault_weight accepted on a Kubernetes upstream and rejected anywhere else.
  • t/kubernetes/discovery/kubernetes2.t — an endpoint with a targetRef carries the Pod uid, one without it carries none. Runs in the ci-kubernetes job, which needs a cluster.

Everything except the t/kubernetes case was run locally; that one needs the kind cluster the CI job brings up.

… the feature slow start

Two things, both on top of apache#13941.

Slow start already ramps the endpoints a Kubernetes upstream discovers, since
they reach the balancer as ordinary nodes. What it could not see is which
workload answers on an address. A cluster recycles addresses quickly, so a
Pod that took one over inherited the maturity of the Pod that had it before
and went straight to full weight, while a Pod that moved to another address
was ramped again from the minimum although it had been serving all along.
The discovery module now carries `targetRef.uid` in the node metadata, for
EndpointSlice and for the older Endpoints API, and the lifecycle identity
prefers it over host and port. An endpoint with no Pod behind it keeps the
address identity it had.

`slow_start.default_weight` comes with it. Kubernetes endpoints carry no
weight of their own, so the registry gives them all the same one; an upstream
that wants its ramp to end somewhere else says so. It is rejected on anything
but a `discovery_type: kubernetes` upstream, where the weights a registry
hands out mean something and are left alone.

The second change is the name. The configuration was `warm_up_conf` while the
module, the shared dict, the log lines and the test files all said slow start,
and the key held a field called `slow_start_time_seconds`, so the two names
met inside one object. Slow start is what the rest of the world calls this -
Envoy has `slow_start_config`, NGINX Plus a `slow_start` parameter, HAProxy
`slowstart` - and the weight curve here is Envoy's, so the key becomes
`slow_start`. The fields inside it do not change, and nothing has been
released with the old key.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant