diff --git a/README.md b/README.md index 6e23109..3e697c5 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ per-cloud glue. The API follows a Karpenter-style split: ## Features -- **Kubernetes-native experience.** Existing Deployments and Jobs opt in via labels; no new workload type and no per-cloud glue. +- **Kubernetes-native experience.** Existing Deployments or Jobs opt in via labels; no new workload type and no per-cloud glue. - **Multi-provider support.** Modal and AWS today, with more to come. - **Policy-driven placement.** Flexible NodePool spec to pick providers by cost, availability, and region, with failover and network egress policy. - **Cost tracking.** Out-of-the-box per-instance cost tracking. diff --git a/docs/metrics.md b/docs/metrics.md index 8b74572..5cb961f 100644 --- a/docs/metrics.md +++ b/docs/metrics.md @@ -148,14 +148,16 @@ lives on the claim instead (below). Sharing series across claims narrows that problem rather than removing it, because `increase()` still needs a sample from *before* the charge and a series' first sample has none. Counter series are also **process-local**: a redeploy or a leader handoff starts the new process with none of them. So -when the accrual loop starts — which under leader election is the moment this process starts charging -— it publishes every already-anchored, priced claim's label set at `0` under all three billing -phases, a baseline for the first window any of them will book. The seeding pass is one interval ahead -of the first tick, so a scrape lands in between. +`markPhase` publishes every priced, anchored claim's label set at `0` under all three billing phases +— a baseline for the first window any of them will book — on **every** reconcile pass rather than +once at startup. -That covers the fleet across a restart, which is where the money is. It does **not** cover a label -set appearing *after* the pass — a shape or a tenant seen for the first time mid-process — whose -first window is still booked onto a newborn series. See [Known gaps](#known-gaps). +Being level-triggered is what makes one seeding point cover two problems that look separate. A claim +born mid-process, carrying a shape or a tenant seen for the first time, is baselined the moment it +becomes chargeable; and a restart re-baselines the whole fleet off the informer's initial sync, well +before the accrual loop's first tick an interval later. Either way the baseline precedes the first +charge by about one accrual interval, so the whole mechanism assumes a scrape interval shorter than +that; see [Known gaps](#known-gaps). **Instance-level** cost — infrastructure spend, blind to the workload on top, charging each claim's whole rate. @@ -442,18 +444,19 @@ knowing before trusting a dashboard. - **A series' first charge is invisible to `increase()`.** A counter's first sample carries no information — `increase()` recovers a *rise* between two samples — so dollars that arrive on a series' first sample are in `nebula_cost_usd_total` but in no `increase()` or `rate()` query over - it. The baselines seeded when the accrual loop starts (see [Cost](#cost)) cover every claim already - billing, so a restart does not cost the fleet a window. Two cases remain, both of them worse with - `--cost-labels` on, where a series belongs to one tenant rather than a whole shape: - - a **label set first seen mid-process** — a new tenant, or a shape nothing was running on at - startup — gets no baseline at all, so its first window is lost to `increase()` for the life of - that process. Nothing re-seeds; the accrual loop seeds once. - - an **instance born and gone inside one scrape interval**, which no baseline can help: the - baseline and the charge land in the same scrape either way. - - Both bias toward *undercounting*, and a tenant whose only job is short can read as zero spend. - Cross-check against `status.estimatedCostUSD` and the `claim finalized` log, which are the durable - records. Closing this properly means a durable per-window event stream, not a counter. + it. Baselines are what keep a claim from relying on sharing a series with another: `markPhase` + re-publishes them on every pass, so a mid-process label set — a new tenant, or a shape nothing was + running on at startup — is covered as well as a claim that predates the process. But the mechanism + is worth nothing unless **the scrape interval is shorter than `accrualInterval`**: the gap between a + baseline and the first window charged on it is one tick, and a scrape has to land inside it. At a + 60s scrape it will not. + + One case no baseline can help: an **instance born and gone inside one scrape interval**, where the + baseline and the charge land in the same scrape regardless. It biases toward *undercounting*, and + with `--cost-labels` on — where a series belongs to one tenant rather than a whole shape — a tenant + whose only job is that short can read as zero spend. Cross-check against + `status.estimatedCostUSD` and the `claim finalized` log, which are the durable records. Closing it + properly means a durable per-window event stream, not a counter. - **`EST_COST` understates a reclaimed instance; the metric does not.** The field is not written on the deletion path (the object is going away), so it misses the window still open at teardown — which the counter *does* book (see [Cost](#cost)). The two therefore disagree by up to one diff --git a/internal/controller/cost_accrual.go b/internal/controller/cost_accrual.go index 87f6c1f..ac881e6 100644 --- a/internal/controller/cost_accrual.go +++ b/internal/controller/cost_accrual.go @@ -83,8 +83,6 @@ func (a *CostAccrual) Start(ctx context.Context) error { log := logf.Log.WithName("cost-accrual") log.Info("starting cost accrual", "interval", a.interval) - a.seedBaselines(ctx) - ticker := time.NewTicker(a.interval) defer ticker.Stop() @@ -98,46 +96,6 @@ func (a *CostAccrual) Start(ctx context.Context) error { } } -// seedBaselines publishes a zero-valued series for every claim that was already chargeable when -// this process took over, before the first tick can charge one. -// -// Counter series are process-local, so without this the first post-restart window of the WHOLE FLEET -// lands on a newborn series, which increase() cannot difference (see metrics.TouchSeries). Prometheus -// recovers some of that by reset detection, but only for an unchanged scrape target: a redeploy's new -// Pod carries a new instance label, and a handoff to another replica is a different target entirely. -// -// Its position in Start is both the once-per-process guard and the ordering that matters — leader -// election makes it the moment this process starts charging, and the first tick is an interval later, -// so a scrape lands between a baseline and the window it explains. A claim that becomes chargeable -// after this pass gets no baseline; see docs/metrics.md, Known gaps. -// -// Best-effort: a failed List costs one window's visibility, never the window itself, which the anchor -// still carries. -func (a *CostAccrual) seedBaselines(ctx context.Context) { - log := logf.Log.WithName("cost-accrual") - - ctx, cancel := context.WithTimeout(ctx, accrualTimeout) - defer cancel() - - var claims nebulav1alpha1.NodeClaimList - if err := a.List(ctx, &claims); err != nil { - log.Error(err, "listing nodeclaims to seed cost baselines") - return - } - seeded := 0 - for i := range claims.Items { - nc := &claims.Items[i] - // finalRate rather than billingRate: a Terminated claim still books a window when its Pod - // is finally deleted (see settleFinalCost), so it needs a baseline as much as a live one. - if _, ok := finalRate(nc); !ok || nc.Status.LastAccruedAt == nil { - continue - } - metrics.TouchSeries(claimLabels(nc), nc.Status.CostLabels, billingPhases...) - seeded++ - } - log.Info("seeded cost baselines", "claims", seeded) -} - // accrueAll checkpoints every billing claim once. Errors are logged per claim and never // propagated: one unwritable claim must not cost the rest of the fleet its window. func (a *CostAccrual) accrueAll(ctx context.Context) { @@ -227,25 +185,28 @@ func billingRate(nc *nebulav1alpha1.NodeClaim) (float64, bool) { default: return 0, false } - return finalRate(nc) + return parsePrice(nc) } // billingPhases is every phase a window can be booked under: the two billingRate admits, plus -// Terminated for the last window settleFinalCost closes. seedBaselines publishes all three up front -// rather than guessing: one always stays empty, since a reclaimed instance passes through Terminating -// and a self-terminated one does not, and which of the two it will be is not knowable in advance. +// Terminated for the last window settleFinalCost closes. seedClaimBaseline publishes all three up +// front rather than guessing: one always stays empty, since a reclaimed instance passes through +// Terminating and a self-terminated one does not, and which of the two it will be is not knowable +// in advance. var billingPhases = []string{ string(nebulav1alpha1.NodeClaimBound), string(nebulav1alpha1.NodeClaimTerminating), string(nebulav1alpha1.NodeClaimTerminated), } -// finalRate is billingRate without the phase gate. Terminated is excluded there because a claim -// sits in that phase until its Pod is deleted and charging those idle hours would be wrong — but the -// window between the last checkpoint and the instance's death is real, and settleFinalCost is the -// only place left that can book it. Nothing may use this to decide that a claim ACCRUES; seedBaselines -// is the one other caller, and it only asks whether a window could ever be booked. -func finalRate(nc *nebulav1alpha1.NodeClaim) (float64, bool) { +// parsePrice is billingRate without the phase gate: whether an instance of this shape costs money, +// not whether this claim is accruing right now. Nothing may use it to decide that a claim ACCRUES. +// +// Both callers want the ungated form because of Terminated — the phase billingRate refuses, since a +// claim sits there until its Pod is deleted and charging those idle hours would be wrong. +// settleFinalCost books the one window that is real, between the last checkpoint and the instance's +// death; seedClaimBaseline only asks whether a window could ever be booked at all. +func parsePrice(nc *nebulav1alpha1.NodeClaim) (float64, bool) { rate, err := strconv.ParseFloat(nc.Status.PriceUSDPerHour, 64) // ParseFloat accepts "NaN" and "Inf" without an error, and both slip past rate <= 0. Left // unchecked, one such value reaches the ledger, and from there it is unrecoverable: the total @@ -262,7 +223,7 @@ func finalRate(nc *nebulav1alpha1.NodeClaim) (float64, bool) { func costSoFar(nc *nebulav1alpha1.NodeClaim) float64 { total, err := strconv.ParseFloat(nc.Status.EstimatedCostUSD, 64) // Non-finite is what makes the zero here load-bearing rather than tidy: a ledger already - // holding "NaN" (see finalRate) would otherwise stay poisoned for the claim's whole life, + // holding "NaN" (see parsePrice) would otherwise stay poisoned for the claim's whole life, // since every later total is derived from this read. if err != nil || math.IsNaN(total) || math.IsInf(total, 0) || total < 0 { return 0 @@ -301,6 +262,31 @@ func stampAccrualStart(nc *nebulav1alpha1.NodeClaim) bool { return true } +// seedClaimBaseline publishes a zero-valued series for each phase this claim could ever book a +// window under, so the first charge has an earlier sample to be differenced against — without one, +// those dollars are in the counter but in no increase() query, which is what a billing consumer +// runs (see metrics.TouchSeries). +// +// LEVEL-TRIGGERED on purpose, which is why one call site covers two problems that look separate. +// A claim born mid-process is seeded when it first becomes chargeable; and because counter series +// are process-local, a restart or leader handoff re-seeds the whole fleet off the initial sync, +// well before the accrual loop's first tick an interval later. A once-at-startup pass covers only +// the second, and only for claims that existed by then. +// +// parsePrice rather than billingRate: a Terminated claim still books a window when its Pod is +// finally deleted (see settleFinalCost), so it needs a baseline as much as a live one. Both gates +// are omissions — no anchor or no price means no window can ever be booked here, and a baseline +// for one would be an empty promise that only costs cardinality. +// +// Worth nothing unless the scrape interval is shorter than accrualInterval: the gap this opens is +// one tick, and a scrape has to land inside it. See docs/metrics.md, Known gaps. +func seedClaimBaseline(nc *nebulav1alpha1.NodeClaim) { + if _, ok := parsePrice(nc); !ok || nc.Status.LastAccruedAt == nil { + return + } + metrics.TouchSeries(claimLabels(nc), nc.Status.CostLabels, billingPhases...) +} + // settleFinalCost books the window still open at teardown and logs what the instance cost over its // whole life. // @@ -313,7 +299,7 @@ func stampAccrualStart(nc *nebulav1alpha1.NodeClaim) bool { // The lifetime figure itself only goes to the log; status is deleted with the object. func settleFinalCost(ctx context.Context, nc *nebulav1alpha1.NodeClaim) { var final float64 - if rate, ok := finalRate(nc); ok && nc.Status.LastAccruedAt != nil { + if rate, ok := parsePrice(nc); ok && nc.Status.LastAccruedAt != nil { open := time.Since(nc.Status.LastAccruedAt.Time) // Clamped in Terminated alone: its anchor froze when the instance died and the claim may // have sat for days since. A Terminating claim is still running, so its whole open window is diff --git a/internal/controller/cost_accrual_test.go b/internal/controller/cost_accrual_test.go index 6a73f30..f5f7732 100644 --- a/internal/controller/cost_accrual_test.go +++ b/internal/controller/cost_accrual_test.go @@ -31,10 +31,12 @@ import ( apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/client/fake" "sigs.k8s.io/controller-runtime/pkg/client/interceptor" "sigs.k8s.io/controller-runtime/pkg/event" + "sigs.k8s.io/controller-runtime/pkg/reconcile" nebulav1alpha1 "github.com/InftyAI/Nebula/api/v1alpha1" "github.com/InftyAI/Nebula/pkg/metrics" @@ -600,32 +602,121 @@ func TestMarkPhase_StampsAttributionThatIsThenBooked(t *testing.T) { } NewCostAccrual(c).accrueAll(context.Background()) - // No baseline anywhere: accrueAll is driven directly, so nothing here has seeded one. That is the - // gap this claim would fall into if it had been born mid-process — see seedBaselines. + // The two zeroes are the point, not noise. They are what stands between this claim's first + // window and a series whose first sample already holds money: with them, the 10 below is a RISE + // that increase() can difference; without them it is invisible to every billing query. // - // The exposition format is line-oriented, so this cannot be wrapped. + // The exposition format is line-oriented, so these cannot be wrapped. //nolint:lll wantSeries := ` # HELP nebula_cost_usd_total Cumulative USD billed by external instances, added window by window as it accrues. # TYPE nebula_cost_usd_total counter nebula_cost_usd_total{accelerator="H100",accelerator_count="1",capacity_type="OnDemand",example_com_org_id="acme",phase="Bound",provider="fake",region="us-east-1"} 10 +nebula_cost_usd_total{accelerator="H100",accelerator_count="1",capacity_type="OnDemand",example_com_org_id="acme",phase="Terminated",provider="fake",region="us-east-1"} 0 +nebula_cost_usd_total{accelerator="H100",accelerator_count="1",capacity_type="OnDemand",example_com_org_id="acme",phase="Terminating",provider="fake",region="us-east-1"} 0 ` if err := testutil.CollectAndCompare(metrics.CostTotal, strings.NewReader(wantSeries)); err != nil { t.Fatal(err) } } -// A restart throws away every series markPhase baselined when these claims were born, and it will -// not run again for a claim that is already anchored. Seeding is what stops the whole fleet's first -// post-restart window from landing on newborn series that increase() cannot difference. -func TestSeedBaselines(t *testing.T) { +// Seeding follows the patch, never the stamp. An anchor that failed to persist opens no window, so a +// series minted alongside it advertises spend that cannot arrive — and for a claim whose writes keep +// failing it is pure cardinality on a counter that never releases a series. +// +// The last pass is the restart: series are process-local, so a claim that is already anchored — one +// this process never stamped — has to be re-seeded off the initial sync, without a status write. +// That is why the seed is level-triggered rather than riding the stamp's once-per-claim gate. +func TestMarkPhase_SeedsBaseline(t *testing.T) { + metrics.CostTotal.Reset() + + pod := gpuPod("H100", 1, "1", "1Gi") + withInstanceID(pod, "sb-1") + + nc := newClaim("c1", "p1", "default", "uid-1", "fake") + nc.Spec.Accelerator = "H100:1" + nc.Spec.Region = "us-east-1" + nc.Spec.CapacityType = nebulav1alpha1.CapacityOnDemand + + refuse := true + writes := 0 + s := testScheme(t) + c := fake.NewClientBuilder(). + WithScheme(s). + WithObjects(pod, nc). + WithStatusSubresource(&nebulav1alpha1.NodeClaim{}). + WithInterceptorFuncs(interceptor.Funcs{ + SubResourceUpdate: func(ctx context.Context, cl client.Client, sub string, + obj client.Object, opts ...client.SubResourceUpdateOption) error { + if refuse { + return apierrors.NewInternalError(errors.New("status write refused")) + } + writes++ + return cl.SubResource(sub).Update(ctx, obj, opts...) + }, + }). + Build() + pp := &pricedProvider{fakeProvider: &fakeProvider{name: "fake"}, rate: 10} + r := &NodeClaimReconciler{Client: c, Scheme: s, Providers: func(name string) (provider.Provider, bool) { + return pp, name == pp.name + }} + + if _, err := r.Reconcile(context.Background(), reconcile.Request{ + NamespacedName: types.NamespacedName{Name: "c1"}, + }); err == nil { + t.Fatal("reconcile succeeded with the status write refused; the error must reach the caller") + } + if n := testutil.CollectAndCount(metrics.CostTotal); n != 0 { + t.Fatalf("%d series after a failed patch, want 0 — the seed must not outrun the anchor", n) + } + + refuse = false + reconcileClaim(t, r, "c1") + + if got := getClaim(t, c, "c1"); got.Status.LastAccruedAt == nil { + t.Fatal("no accrual anchor persisted; the seed below would have nothing to baseline") + } + // Zero, not absent: the dollars come later, and a baseline's whole job is to be the sample + // before them. See metrics.TouchSeries for why RecordWindow still refuses a zero WINDOW. + // + // The exposition format is line-oriented, so these cannot be wrapped. + //nolint:lll + want := ` +# HELP nebula_cost_usd_total Cumulative USD billed by external instances, added window by window as it accrues. +# TYPE nebula_cost_usd_total counter +nebula_cost_usd_total{accelerator="H100",accelerator_count="1",capacity_type="OnDemand",phase="Bound",provider="fake",region="us-east-1"} 0 +nebula_cost_usd_total{accelerator="H100",accelerator_count="1",capacity_type="OnDemand",phase="Terminated",provider="fake",region="us-east-1"} 0 +nebula_cost_usd_total{accelerator="H100",accelerator_count="1",capacity_type="OnDemand",phase="Terminating",provider="fake",region="us-east-1"} 0 +` + if err := testutil.CollectAndCompare(metrics.CostTotal, strings.NewReader(want)); err != nil { + t.Fatal(err) + } + + // Reset stands in for a restart or a leader handoff: the claim is untouched and still anchored, + // but every series this process published is gone. Nothing here needs a write — the anchor is + // already durable — so a seed gated on the stamp would leave the whole fleet's first + // post-restart window landing on newborn series, an interval before the accrual loop's first tick. + metrics.CostTotal.Reset() + reconcileClaim(t, r, "c1") + + if err := testutil.CollectAndCompare(metrics.CostTotal, strings.NewReader(want)); err != nil { + t.Fatal(err) + } + if writes != 1 { + t.Fatalf("%d status writes, want 1 — re-seeding a settled claim must not patch it", writes) + } +} + +// Which claims get a baseline at all, over the shapes one reconcile pass cannot cover. +func TestSeedClaimBaseline(t *testing.T) { + metrics.CostTotal.Reset() hour := time.Hour // Every claim gets a shape of its own, or a claim wrongly skipped would hide behind the series // another one seeded — only "same-shape" shares deliberately. shape := billingClaim("same-shape", "10.0000", &hour) // Anchored and priced, so its last window is still booked at teardown: it needs a baseline as - // much as a running claim does, which is why seeding asks finalRate rather than billingRate. + // much as a running claim does, which is why seeding asks parsePrice rather than billingRate. terminated := billingClaim("terminated", "10.0000", &hour) terminated.Status.Phase = nebulav1alpha1.NodeClaimTerminated terminated.Spec.Accelerator = "A100:2" @@ -636,8 +727,11 @@ func TestSeedBaselines(t *testing.T) { cpu := billingClaim("cpu", "0.1000", &hour) cpu.Spec.Accelerator = "" - a, _ := newAccrual(t, billingClaim("bound", "10.0000", &hour), shape, terminated, unanchored, unpriced, cpu) - a.seedBaselines(context.Background()) + for _, nc := range []*nebulav1alpha1.NodeClaim{ + billingClaim("bound", "10.0000", &hour), shape, terminated, unanchored, unpriced, cpu, + } { + seedClaimBaseline(nc) + } // Three shapes, three phases each. The two claims sharing H100:1 collapse onto one set of series // — a baseline is per SERIES, not per claim — while L4 and T4 are absent entirely: no window can @@ -663,32 +757,6 @@ nebula_cost_usd_total{accelerator="none",accelerator_count="none",capacity_type= } } -// Seeding must land BEFORE the first tick, or it baselines a series the same tick already charged -// and buys nothing. Only Start guarantees that ordering, so it is asserted through Start — with a -// claim whose whole hour is chargeable, so a tick that beat the seed would show up as a value. -func TestStart_SeedsBeforeCharging(t *testing.T) { - hour := time.Hour - a, _ := newAccrual(t, billingClaim("bound", "10.0000", &hour)) - - ctx, cancel := context.WithCancel(context.Background()) - defer cancel() - go func() { _ = a.Start(ctx) }() - - // a.interval is a minute, so anything observed within seconds is the seed, not a tick. - deadline := time.Now().Add(5 * time.Second) - for testutil.CollectAndCount(metrics.CostTotal) < len(billingPhases) { - if time.Now().After(deadline) { - t.Fatalf("collected %d series, want %d baselines seeded on the way to the first tick", - testutil.CollectAndCount(metrics.CostTotal), len(billingPhases)) - } - time.Sleep(10 * time.Millisecond) - } - if got := testutil.ToFloat64(metrics.CostTotal.WithLabelValues( - "modal", "none", "none", "H100", "1", "Bound")); got != 0 { - t.Fatalf("the Bound baseline holds %v, want 0 — seeding may not charge anything", got) - } -} - // The stamp is what makes a crash before the first checkpoint lossless, so it must land exactly // when the claim becomes chargeable — and never move once set. func TestStampAccrualStart(t *testing.T) { diff --git a/internal/controller/nodeclaim_controller.go b/internal/controller/nodeclaim_controller.go index 05dd59b..e6dadf4 100644 --- a/internal/controller/nodeclaim_controller.go +++ b/internal/controller/nodeclaim_controller.go @@ -339,10 +339,16 @@ func (r *NodeClaimReconciler) markPhase(ctx context.Context, nc *nebulav1alpha1. if stampAccrualStart(nc) { changed = true } - if !changed { - return nil + if changed { + if err := r.patchStatus(ctx, nc); err != nil { + return err + } } - return r.patchStatus(ctx, nc) + // After the patch, and on every pass rather than only the one that stamps: the anchor has to be + // durable before a baseline claims a window exists, and re-publishing is what makes this cover a + // restart too (see seedClaimBaseline). Idempotent — a series that exists is left where it is. + seedClaimBaseline(nc) + return nil } // recordInstanceID copies the instance id off the served Pod into status.InstanceID when it diff --git a/pkg/metrics/cost.go b/pkg/metrics/cost.go index c1b68d8..f2355cf 100644 --- a/pkg/metrics/cost.go +++ b/pkg/metrics/cost.go @@ -27,8 +27,7 @@ import ( // CostTotal accumulates spend one CLOSED WINDOW at a time, which is what makes it billable: // increase(...[w]) over any window is a pure function of that window, so a consumer replaying // an old window re-derives the same dollars and can upsert them idempotently. That holds only for a -// series scraped before its first charge, which is what TouchSeries is for — and not at all for one -// born after that pass has run. +// series scraped before its first charge, which is what TouchSeries is for. // // Deliberately carries no claim identity. A per-claim series would churn — one per instance ever // created, retained until the process exits — and worse, a claim that lived and died between two @@ -100,7 +99,7 @@ func RecordWindow(l Labels, phase string, attribution map[string]string, usd flo // Non-finite is checked separately because it passes every comparison above: a counter cannot // be decremented, so one NaN added here is permanent, and it spreads — any sum() spanning that // series is NaN too, taking the whole fleet's cost query with it. Cheaper to refuse here than - // to trust every caller's own parsing (see controller.finalRate). + // to trust every caller's own parsing (see controller.parsePrice). if usd <= 0 || math.IsNaN(usd) || math.IsInf(usd, 0) { return } @@ -123,9 +122,9 @@ func RecordWindow(l Labels, phase string, attribution map[string]string, usd flo // it is the ordinary way to make rate() work over label values not known until runtime. // // Series are PROCESS-local, so the baseline has to be republished by whatever process is doing the -// charging — see controller.CostAccrual.seedBaselines, which is the one caller. A scrape still has to -// land between the baseline and the charge, so a claim that becomes chargeable after the seeding pass -// is not covered at all; see docs/metrics.md. +// charging, on every pass rather than once at startup — see controller.seedClaimBaseline, the one +// caller. A scrape still has to land between the baseline and the first charge, which is what keeps +// this a mitigation rather than a fix; see docs/metrics.md. func TouchSeries(l Labels, attribution map[string]string, phases ...string) { for _, phase := range phases { values := l.values(append([]string{phase}, attributionValues(attribution)...)...)