feat: support cost tracking

.gitignore

@@ -42,4 +42,5 @@ __pycache__
 *.tgz
 
 .devcontainer
-bin
+bin
+.claude

CLAUDE.md

@@ -0,0 +1,18 @@
+# Nebula
+
+## Comments
+
+This codebase comments the *why*, not the *what*. Match that — and match its density too,
+don't exceed it.
+
+- Never restate the code. `// increment the counter` above `n++` is noise.
+- Don't repeat a rationale a nearby doc comment already gives. Name the symbol that holds
+  it (`see Offering.GPUCount`) and move on.
+- Doc comments: the contract, plus the one constraint a caller would otherwise get wrong.
+  2-5 lines. Go longer only for a real invariant — a subtle failure mode, a decision kept
+  on purpose, a gotcha in an external API.
+- Inline comments: only where a reader would otherwise draw the wrong conclusion. Not one
+  per branch, not one per field assignment.
+- One comment on the tricky invariant beats five on the obvious steps.
+
+If a comment is longer than the code it explains, it is usually the wrong comment.

api/v1alpha1/nodeclaim_types.go

@@ -138,6 +138,81 @@ type NodeClaimStatus struct {
 	// Endpoint is the reachable address (e.g. SSH host:port) once ready.
 	// +optional
 	Endpoint string `json:"endpoint,omitempty"`
+
+	// PriceUSDPerHour is what this instance costs per hour in USD, as a decimal string
+	// ("7.9000"), resolved from the provider's catalog against the served Pod's shape.
+	// Status, not spec: it is a derived result nobody can declare up front. A string
+	// because it is written to be READ — a print column can only echo a field, never
+	// scale a fixed-point integer back into currency.
+	//
+	// Empty means UNPRICED, not free: the provider implements no provider.Pricer, or its
+	// catalog has no row for this candidate. Consumers must skip such a claim rather than
+	// count it as $0.
+	//
+	// Written once and never refreshed, so a catalog edit cannot retroactively reprice a
+	// running instance and rewrite the cost history it has already reported.
+	// +optional
+	PriceUSDPerHour string `json:"priceUSDPerHour,omitempty"`
+
+	// EstimatedCostUSD is what this instance has cost SO FAR, as a decimal string ("412.94000000")
+	// — PriceUSDPerHour integrated over the time it has held an instance. A string for the same
+	// reason the rate is one: it exists to be read off a print column.
+	//
+	// More decimals than the rate, and they are not decoration: each checkpoint re-reads this field
+	// to measure the next window, so digits dropped here are money dropped, and a cheap enough claim
+	// would round back to where it started every minute and never accrue at all. Round it for
+	// display; never round it back into this field.
+	//
+	// ESTIMATED, and the name says so on purpose: it is our own arithmetic over a list price
+	// (see PriceUSDPerHour), not a figure any provider has confirmed. Nothing here has been
+	// invoiced. Reconcile against the provider's billing export before anyone is charged.
+	//
+	// Within those limits it is the AUTHORITATIVE total, not the Prometheus counter: it is
+	// written exactly once per window and survives a restart. It is also only LIVE cost — it
+	// dies with the claim, so it is not a history. The metric is what outlives an instance.
+	// +optional
+	EstimatedCostUSD string `json:"estimatedCostUSD,omitempty"`
+
+	// LastAccruedAt is how far cost accrual has counted: an ANCHOR for the next measurement,
+	// not a note about the last one. "Accrued" in the accounting sense — cost incurred but not
+	// yet invoiced, which is all EstimatedCostUSD ever holds.
+	//
+	// It is the reason a restart loses nothing: the next window is rate x (now - LastAccruedAt),
+	// so time that passed while Nebula was down is still counted on recovery instead of vanishing
+	// with the in-memory total.
+	//
+	// The invariant that makes it safe: this NEVER moves past cost that has been durably
+	// recorded, because it advances only in the same patch that writes EstimatedCostUSD. A
+	// failed write therefore loses nothing — the same window is counted next time.
+	//
+	// Unset means "not counting yet". Never treat it as the epoch, which would charge decades
+	// on the first tick.
+	// +optional
+	LastAccruedAt *metav1.Time `json:"lastAccruedAt,omitempty"`
+
+	// CostLabels attributes this claim's spend to whoever asked for it: the values the served
+	// Pod carried for the label keys the operator configured (--cost-labels). Keyed by the POD
+	// label key, verbatim — so this map reads like the Pod it came from. The metric emits under a
+	// derived name Prometheus accepts ("example.com/org-id" here is example_com_org_id there), so
+	// do not expect the two to match by string.
+	//
+	// Status rather than spec because it is observed from the Pod, and it sits with the rest of
+	// the billing record for a reason: it is stamped in the SAME patch that opens the accrual
+	// anchor, so no window can ever be charged before its attribution is known.
+	//
+	// Written once, on first observation, and never refreshed — relabeling a Pod must not
+	// retroactively re-attribute spend already reported under the old values. A name the
+	// current --cost-labels no longer lists is ignored rather than cleaned up; one it lists but
+	// this map lacks reports as "none".
+	//
+	// Nil and EMPTY differ, which is why this field has no omitempty: nil means nothing has been
+	// observed yet (or --cost-labels is unset), while an empty map is the settled fact that the
+	// Pod carried none of the configured keys. With omitempty an empty map would not survive the
+	// write, so every reconcile would re-read the Pod and a label added later could still move
+	// the claim's attribution.
+	// +optional
+	// +kubebuilder:validation:Nullable
+	CostLabels map[string]string `json:"costLabels"`
 }
 
 // +kubebuilder:object:root=true
@@ -147,6 +222,8 @@ type NodeClaimStatus struct {
 // +kubebuilder:printcolumn:name="Region",type=string,JSONPath=`.spec.region`
 // +kubebuilder:printcolumn:name="ACCELERATOR",type=string,JSONPath=`.spec.accelerator`
 // +kubebuilder:printcolumn:name="CAPACITY_TYPE",type=string,JSONPath=`.spec.capacityType`
+// +kubebuilder:printcolumn:name="PRICE/HR",type=string,JSONPath=`.status.priceUSDPerHour`
+// +kubebuilder:printcolumn:name="EST_COST",type=string,JSONPath=`.status.estimatedCostUSD`
 // +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
 // +kubebuilder:printcolumn:name="Instance",type=string,JSONPath=`.status.instanceID`
 // +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`

cmd/main.go

@@ -49,6 +49,7 @@ import (
 	webhookv1 "github.com/InftyAI/Nebula/internal/webhook/v1"
 	nebulacert "github.com/InftyAI/Nebula/pkg/cert"
 	"github.com/InftyAI/Nebula/pkg/failover"
+	nebulametrics "github.com/InftyAI/Nebula/pkg/metrics"
 	"github.com/InftyAI/Nebula/pkg/provider"
 	awsprovider "github.com/InftyAI/Nebula/pkg/provider/aws"
 	"github.com/InftyAI/Nebula/pkg/provider/fake"
@@ -93,6 +94,7 @@ func main() {
 	var secureMetrics bool
 	var enableHTTP2 bool
 	var kubeletAddr, kubeletClientCA string
+	var costLabels string
 	var tlsOpts []func(*tls.Config)
 	flag.StringVar(&metricsAddr, "metrics-bind-address", "0", "The address the metrics endpoint binds to. "+
 		"Use :8443 for HTTPS or :8080 for HTTP, or leave as 0 to disable the metrics service.")
@@ -119,6 +121,14 @@ func main() {
 			"serves TLS without client verification, because which CA signs the API server's kubelet "+
 			"client certificate is not portable across distributions; restrict the port with a "+
 			"NetworkPolicy, or set this to your API server's kubelet client CA.")
+	flag.StringVar(&costLabels, "cost-labels", "",
+		"Comma-separated Pod label keys whose values attribute cost, e.g. "+
+			"\"example.com/org-id,team_id\". Qualified keys are fine: the metric label is the whole key "+
+			"with '/', '-' and '.' folded to '_' (example.com/org-id is queried as example_com_org_id), so "+
+			"two keys that would collide there are rejected at startup. Empty (the default) reports cost "+
+			"by candidate shape only. Changing this changes the identity of every cost series. Values "+
+			"come from Pod labels and are NOT capped: the manager warns once if they push the cost "+
+			"metric past 5000 series, but pick keys an admission policy constrains.")
 	opts := zap.Options{
 		Development: true,
 	}
@@ -127,6 +137,21 @@ func main() {
 
 	ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts)))
 
+	// Before anything can record: the counter's label names are fixed when it is constructed, and
+	// a bad --cost-labels must stop the process rather than silently report every tenant as "none".
+	attribution, err := nebulametrics.ParseCostLabels(costLabels)
+	if err != nil {
+		setupLog.Error(err, "invalid --cost-labels")
+		os.Exit(1)
+	}
+	if err := nebulametrics.InitCost(attribution); err != nil {
+		setupLog.Error(err, "configuring the cost metric")
+		os.Exit(1)
+	}
+	if len(attribution) > 0 {
+		setupLog.Info("cost attribution enabled", "labels", attribution)
+	}
+
 	// if the enable-http2 flag is false (the default), http/2 should be disabled
 	// due to its vulnerabilities. More specifically, disabling http/2 will
 	// prevent from being vulnerable to the HTTP/2 Stream Cancellation and
@@ -418,6 +443,16 @@ func setupControllers(mgr ctrl.Manager, blocklist *failover.Blocklist, kubeletSr
 			return fmt.Errorf("unable to create Pod webhook: %w", err)
 		}
 	}
+
+	// Cost accrual is a clock-driven loop, not a reconciler, so it is added directly. It opts into
+	// leader election by NOT implementing LeaderElectionRunnable, which buys two things: N replicas
+	// would mean N times the writes, and each would book windows into its OWN cost counter, leaving
+	// increase() to read a series that holds only the races that replica won. The ledger itself is
+	// safe either way (see CostAccrual.accrue).
+	if err := mgr.Add(controller.NewCostAccrual(mgr.GetClient())); err != nil {
+		return fmt.Errorf("unable to add the cost accrual loop: %w", err)
+	}
+
 	// +kubebuilder:scaffold:builder
 	return nil
 }

config/crd/bases/nebula.inftyai.com_nodeclaims.yaml

@@ -29,6 +29,12 @@ spec:
     - jsonPath: .spec.capacityType
       name: CAPACITY_TYPE
       type: string
+    - jsonPath: .status.priceUSDPerHour
+      name: PRICE/HR
+      type: string
+    - jsonPath: .status.estimatedCostUSD
+      name: EST_COST
+      type: string
     - jsonPath: .status.phase
       name: Phase
       type: string
@@ -141,23 +147,100 @@ spec:
               NodeClaimStatus is the durable record reconciled against the provider by the
               poll loop.
             properties:
+              costLabels:
+                additionalProperties:
+                  type: string
+                description: |-
+                  CostLabels attributes this claim's spend to whoever asked for it: the values the served
+                  Pod carried for the label keys the operator configured (--cost-labels). Keyed by the POD
+                  label key, verbatim — so this map reads like the Pod it came from. The metric emits under a
+                  derived name Prometheus accepts ("example.com/org-id" here is example_com_org_id there), so
+                  do not expect the two to match by string.
+
+                  Status rather than spec because it is observed from the Pod, and it sits with the rest of
+                  the billing record for a reason: it is stamped in the SAME patch that opens the accrual
+                  anchor, so no window can ever be charged before its attribution is known.
+
+                  Written once, on first observation, and never refreshed — relabeling a Pod must not
+                  retroactively re-attribute spend already reported under the old values. A name the
+                  current --cost-labels no longer lists is ignored rather than cleaned up; one it lists but
+                  this map lacks reports as "none".
+
+                  Nil and EMPTY differ, which is why this field has no omitempty: nil means nothing has been
+                  observed yet (or --cost-labels is unset), while an empty map is the settled fact that the
+                  Pod carried none of the configured keys. With omitempty an empty map would not survive the
+                  write, so every reconcile would re-read the Pod and a label added later could still move
+                  the claim's attribution.
+                type: object
               endpoint:
                 description: Endpoint is the reachable address (e.g. SSH host:port)
                   once ready.
                 type: string
+              estimatedCostUSD:
+                description: |-
+                  EstimatedCostUSD is what this instance has cost SO FAR, as a decimal string ("412.94000000")
+                  — PriceUSDPerHour integrated over the time it has held an instance. A string for the same
+                  reason the rate is one: it exists to be read off a print column.
+
+                  More decimals than the rate, and they are not decoration: each checkpoint re-reads this field
+                  to measure the next window, so digits dropped here are money dropped, and a cheap enough claim
+                  would round back to where it started every minute and never accrue at all. Round it for
+                  display; never round it back into this field.
+
+                  ESTIMATED, and the name says so on purpose: it is our own arithmetic over a list price
+                  (see PriceUSDPerHour), not a figure any provider has confirmed. Nothing here has been
+                  invoiced. Reconcile against the provider's billing export before anyone is charged.
+
+                  Within those limits it is the AUTHORITATIVE total, not the Prometheus counter: it is
+                  written exactly once per window and survives a restart. It is also only LIVE cost — it
+                  dies with the claim, so it is not a history. The metric is what outlives an instance.
+                type: string
               instanceID:
                 description: |-
                   InstanceID is the provider's identifier for the external instance (e.g. a
                   RunPod pod id). This is the field that must not be lost: the terminate
                   finalizer uses it to reclaim the instance.
                 type: string
+              lastAccruedAt:
+                description: |-
+                  LastAccruedAt is how far cost accrual has counted: an ANCHOR for the next measurement,
+                  not a note about the last one. "Accrued" in the accounting sense — cost incurred but not
+                  yet invoiced, which is all EstimatedCostUSD ever holds.
+
+                  It is the reason a restart loses nothing: the next window is rate x (now - LastAccruedAt),
+                  so time that passed while Nebula was down is still counted on recovery instead of vanishing
+                  with the in-memory total.
+
+                  The invariant that makes it safe: this NEVER moves past cost that has been durably
+                  recorded, because it advances only in the same patch that writes EstimatedCostUSD. A
+                  failed write therefore loses nothing — the same window is counted next time.
+
+                  Unset means "not counting yet". Never treat it as the epoch, which would charge decades
+                  on the first tick.
+                format: date-time
+                type: string
               nodeName:
                 description: NodeName is the virtual Node created for this instance,
                   once it exists.
                 type: string
               phase:
                 description: Phase is the coarse lifecycle state.
                 type: string
+              priceUSDPerHour:
+                description: |-
+                  PriceUSDPerHour is what this instance costs per hour in USD, as a decimal string
+                  ("7.9000"), resolved from the provider's catalog against the served Pod's shape.
+                  Status, not spec: it is a derived result nobody can declare up front. A string
+                  because it is written to be READ — a print column can only echo a field, never
+                  scale a fixed-point integer back into currency.
+
+                  Empty means UNPRICED, not free: the provider implements no provider.Pricer, or its
+                  catalog has no row for this candidate. Consumers must skip such a claim rather than
+                  count it as $0.
+
+                  Written once and never refreshed, so a catalog edit cannot retroactively reprice a
+                  running instance and rewrite the cost history it has already reported.
+                type: string
             type: object
         type: object
     served: true

docs/architecture.md

@@ -568,7 +568,12 @@ Two design rules are worth knowing here, because they constrain the code above:
   `Handler.metricLabels` and `placementLabels` are field-for-field mirrors.
 - Every label is bounded by **configuration** (NodePools, provider catalogs), never by
   workload. Nothing derived from a Pod name, UID, namespace or unresolved user-supplied
-  pool label is ever a label value.
+  pool label is ever a label value. Cost is no exception, which is why per-instance spend
+  is a NodeClaim status field rather than a label on the counter. Cost *attribution* is the
+  single opt-in breach of that rule — `--cost-labels` promotes chosen Pod labels onto the
+  counter — so it ships off by default, and warns rather than capping when the series count
+  says an operator chose badly: a billing metric that rewrote its own labels to defend
+  itself would be wrong where nobody could see it.
 
 See [metrics.md](metrics.md) for the full series list, label semantics, example queries
 and the two known gaps.