Conversation
… the feature slow start Two things, both on top of apache#13941. Slow start already ramps the endpoints a Kubernetes upstream discovers, since they reach the balancer as ordinary nodes. What it could not see is which workload answers on an address. A cluster recycles addresses quickly, so a Pod that took one over inherited the maturity of the Pod that had it before and went straight to full weight, while a Pod that moved to another address was ramped again from the minimum although it had been serving all along. The discovery module now carries `targetRef.uid` in the node metadata, for EndpointSlice and for the older Endpoints API, and the lifecycle identity prefers it over host and port. An endpoint with no Pod behind it keeps the address identity it had. `slow_start.default_weight` comes with it. Kubernetes endpoints carry no weight of their own, so the registry gives them all the same one; an upstream that wants its ramp to end somewhere else says so. It is rejected on anything but a `discovery_type: kubernetes` upstream, where the weights a registry hands out mean something and are left alone. The second change is the name. The configuration was `warm_up_conf` while the module, the shared dict, the log lines and the test files all said slow start, and the key held a field called `slow_start_time_seconds`, so the two names met inside one object. Slow start is what the rest of the world calls this - Envoy has `slow_start_config`, NGINX Plus a `slow_start` parameter, HAProxy `slowstart` - and the weight curve here is Envoy's, so the key becomes `slow_start`. The fields inside it do not change, and nothing has been released with the old key.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows #13941, which covered the general data plane capability. This one teaches it about Kubernetes, and settles the naming.
What was missing
An upstream backed by Kubernetes discovery already ramps: endpoints reach the balancer as ordinary nodes, a new one is new to the node set, and it warms up. What slow start could not see is which workload answers on an address.
A cluster recycles addresses quickly, so both halves of that were wrong:
Pod identity
apisix/discovery/kubernetes/core.luanow carriestargetRef.uidin the node metadata, for EndpointSlice and for the older Endpoints API, andnode_idinapisix/slow_start.luaprefers it:targetRef.uidand the porttargetRef(an external address, say)Everything else follows from that: a Pod changing address keeps its ramp, an address changing Pod starts a new one. The uid also makes the node set change when a Pod is replaced in place, so the picker is rebuilt rather than kept from the previous version.
slow_start.default_weightKubernetes endpoints carry no weight of their own, so the registry gives every one of them the same
default_weight. An upstream that wants its ramp to end somewhere else can now say so:{ "type": "roundrobin", "service_name": "default/orders:http", "discovery_type": "kubernetes", "slow_start": { "slow_start_time_seconds": 300, "min_weight_percent": 1, "default_weight": 100 } }It is rejected on anything but a
discovery_type: kubernetesupstream. Elsewhere the weights a registry hands out mean something - a Nacos service with a 10/20/70 split, for one - and silently flattening them would be worse than not offering the field.Deliberately out of scope
Two things the RFC lists for this stage are not here, and I would rather say so than leave them looking forgotten:
first_observed_atat the discovery layer so two upstreams referencing the same Service agree on when an endpoint appeared. Each upstream already keeps its own lifecycle, and the only case the shared record changes is an upstream created while the Service was already running, whose nodes are a mature baseline either way.Naming
@shreemaan-abhishek's review of #13941 pointed out that the feature answered to two names: the configuration was
warm_up_conf, while the module, the shared dict, the log lines and the test files all said slow start - and the key held a field calledslow_start_time_seconds, so both names met inside one object.Slow start is what the rest of the world calls this. Envoy has
slow_start_config, NGINX Plus aslow_startparameter on an upstream server, HAProxyslowstart. The weight curve here is Envoy's and two of the field names already match it, so the configuration key becomesslow_startand everything reads the same way. It also drops a_confsuffix that no other upstream field carries.The field names inside it do not change, and nothing has been released with the old key, so there is nothing to migrate.
Tests
t/node/upstream-slow-start.t— a Pod keeps its ramp across an address change, an address taken over by a different Pod ramps again, anddefault_weightis the target a Kubernetes endpoint ramps to while a Nacos upstream keeps its own weights.t/admin/upstream-slow-start.t—default_weightaccepted on a Kubernetes upstream and rejected anywhere else.t/kubernetes/discovery/kubernetes2.t— an endpoint with atargetRefcarries the Pod uid, one without it carries none. Runs in theci-kubernetesjob, which needs a cluster.Everything except the
t/kubernetescase was run locally; that one needs the kind cluster the CI job brings up.