Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions docs/metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ claim's whole rate.
over a hand-maintained list price — cost *incurred*, which is what "accrued" means, not cost any
provider has confirmed. Nothing below has been invoiced.

**One arithmetic, two views.** A leader-elected loop closes each claim's window every minute:
**One arithmetic, two views.** A leader-elected loop closes each claim's window every 30 seconds:
it charges `priceUSDPerHour × (now − status.lastAccruedAt)`, adds it to
`NodeClaim.status.estimatedCostUSD` (the `EST_COST` column) and re-anchors, then books the *same*
window on the counter. So the counter is the fleet's stream of charges and the field is the
Expand All @@ -180,8 +180,8 @@ nc-train-0 modal H100:8 23.7000 148.12500000 Bound

`EST_COST` carries eight decimals to the rate's four because each checkpoint measures its window
from the field it wrote last time, so digits dropped there are money dropped, not display noise: at
four decimals a claim under $0.003/hour rounded back to its previous value every minute and never
accrued at all. Round it when you show it.
four decimals a claim under $0.006/hour rounds back to its previous value every tick and never
accrues at all. Round it when you show it.

The counter is advanced only *after* the field's patch lands. A counter has no idempotency key, so
a window booked before its write was durable would be charged twice: the anchor would not have
Expand Down Expand Up @@ -448,8 +448,9 @@ knowing before trusting a dashboard.
re-publishes them on every pass, so a mid-process label set — a new tenant, or a shape nothing was
running on at startup — is covered as well as a claim that predates the process. But the mechanism
is worth nothing unless **the scrape interval is shorter than `accrualInterval`**: the gap between a
baseline and the first window charged on it is one tick, and a scrape has to land inside it. At a
60s scrape it will not.
baseline and the first window charged on it is one tick, and a scrape has to land inside it. The
tick is 30s, so a 15s scrape clears it and a 60s one does not — verify yours before trusting any
`increase()` figure here.

One case no baseline can help: an **instance born and gone inside one scrape interval**, where the
baseline and the charge land in the same scrape regardless. It biases toward *undercounting*, and
Expand All @@ -465,7 +466,7 @@ knowing before trusting a dashboard.
running — a freshness limit, not an error, since a window not charged now is charged in full next
tick.
- **The last window of a self-terminated instance is capped, not measured.** A `Terminated` claim's
anchor froze when the instance died, and nothing since distinguishes "died a minute ago" from
anchor froze when the instance died, and nothing since distinguishes "died 30 seconds ago" from
"died while the manager was down three hours ago", so teardown charges one interval either way
(see [Cost](#cost)). The cap is the safe direction — it cannot bill idle hours — but a preemption
during an outage is undercharged by the whole outage. Nothing is booked at all if the Pod object
Expand Down
26 changes: 16 additions & 10 deletions internal/controller/cost_accrual.go
Original file line number Diff line number Diff line change
Expand Up @@ -36,19 +36,23 @@ import (

// accrualInterval is how often spend is checkpointed. It is a WRITE cadence, not an accounting
// resolution: every window is measured from the persisted anchor, so a longer interval trades
// freshness of the EST_COST column for fewer writes, never accuracy. One minute is ~8.3 writes/s
// across a 500-claim fleet, about a sixth of the client's rate budget (see the QPS in cmd/main.go)
// — the point where this stops being free and starts competing with the reconcilers for it.
const accrualInterval = time.Minute
// freshness of the EST_COST column for fewer writes, never accuracy.
//
// What it does bound is the gap a baseline has to survive — a scrape must land between a series'
// zero sample and its first charge, one tick later, or those dollars reach no increase() query at
// all (see seedClaimBaseline). Thirty seconds keeps that reachable for the usual 15s scrape, at
// ~16.7 writes/s across a 500-claim fleet: a third of the client's rate budget (see the QPS in
// cmd/main.go), which is where this stops being free and starts competing with the reconcilers.
const accrualInterval = 30 * time.Second

// accrualTimeout bounds one whole tick, List plus every write. A tick that cannot finish loses
// nothing: the anchors it did not reach are still where they were, so the next tick charges the
// same windows.
//
// Now equal to accrualInterval, which is safe rather than tidy: ticks run sequentially and a
// Equal to accrualInterval, which is safe rather than tidy: ticks run sequentially and a
// time.Ticker drops the ticks a slow receiver missed instead of queueing them, so the worst case is
// back-to-back ticks, and re-deriving a window from its anchor cannot double-charge it.
const accrualTimeout = time.Minute
const accrualTimeout = accrualInterval
Comment thread
kerthcet marked this conversation as resolved.

// CostAccrual advances each claim's durable spend ledger on a ticker.
//
Expand Down Expand Up @@ -233,11 +237,13 @@ func costSoFar(nc *nebulav1alpha1.NodeClaim) float64 {

// costDecimals is how many fractional digits the LEDGER keeps — more than priceDecimals, because a
// rate is an input that is written once while a total is an accumulator that is re-read and
// re-written every minute. Rounding it at each checkpoint would quantize the effective rate onto the
// re-written every window. Rounding it at each checkpoint would quantize the effective rate onto the
// grid: the residue is discarded rather than carried, so a claim cheaper than half a grid step per
// window freezes forever (at four digits, anything under $0.003/hr — a Modal CPU-only sandbox), and
// one just above it is charged the rounded-UP step every tick. Eight digits puts that error below
// 0.01% at any rate a catalog carries, at the cost of a wider EST_COST column.
// window freezes forever (at four digits and a 30s tick, anything under $0.006/hr — a Modal CPU-only
// sandbox), and one just above it is charged the rounded-UP step every tick. A shorter interval only
// sharpens that, since it shrinks the charge and not the step. Eight digits holds the drift under
// 0.03% at the cheapest rate a catalog carries and far under it everywhere else, at the cost of a
// wider EST_COST column.
const costDecimals = 8

func formatCost(usd float64) string {
Expand Down
12 changes: 8 additions & 4 deletions internal/controller/cost_accrual_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -155,8 +155,8 @@ func TestCostAccrual_AccumulatesAcrossTicks(t *testing.T) {

// Every checkpoint re-reads the ledger to measure the next window, so the digits it keeps ARE the
// accounting resolution — see costDecimals. Driven at the real cadence, a cheap claim is where that
// shows: at four decimals $0.0028/hr froze at $0.0001 and never moved again, while $0.01/hr was
// charged a rounded-up step every minute and ran 20% over.
// shows: at four decimals and the one-minute cadence this ran at then, $0.0028/hr froze at $0.0001
// and never moved again, while $0.01/hr was charged a rounded-up step every tick and ran 20% over.
func TestCostAccrual_ChargesCheapClaimsAtTheirRealRate(t *testing.T) {
// Modal's CPU-only components: a 20-millicore reservation, a 64MiB one, and one whole core.
for _, rate := range []string{"0.0028", "0.0100", "0.1419"} {
Expand All @@ -166,7 +166,7 @@ func TestCostAccrual_ChargesCheapClaimsAtTheirRealRate(t *testing.T) {
base := nc.Status.LastAccruedAt.Time
a, c := newAccrual(t, nc)

const ticks = 240 // four hours at the one-minute write cadence
const ticks = 480 // four hours at the 30s write cadence
Comment thread
kerthcet marked this conversation as resolved.
for i := 1; i <= ticks; i++ {
at := base.Add(time.Duration(i) * accrualInterval)
a.now = func() time.Time { return at }
Expand All @@ -179,7 +179,11 @@ func TestCostAccrual_ChargesCheapClaimsAtTheirRealRate(t *testing.T) {
}
want := hourly * float64(ticks) * accrualInterval.Hours()
total, _ := ledger(t, c, "cheap")
if math.Abs(total-want)/want > 1e-4 {
// Half a rounding step is discarded per tick and never carried, so the bar is that
// residue against the SMALLEST per-window charge a catalog produces — $0.0028/hr over
// 30s, i.e. ~2e-4 relative. Anything worse means costDecimals is too coarse for the
// interval, which is the failure this test exists to catch.
if math.Abs(total-want)/want > 3e-4 {
t.Fatalf("status.estimatedCostUSD %v after %d ticks at $%s/hr, want %v", total, ticks, rate, want)
}
if got := booked(t); math.Abs(got-total) > 1e-9 {
Expand Down
Loading