fix: prometheus increase error - #92
Conversation
Signed-off-by: kerthcet <kerthcet@gmail.com>
Signed-off-by: kerthcet <kerthcet@gmail.com>
There was a problem hiding this comment.
🟡 Changes recommended
Baseline seeding is now only triggered via markPhase (Pod-present path), which can miss already-anchored claims entering teardown with no served Pod after restart/hand-off, allowing settleFinalCost to book a first-sample charge that increase() may not observe.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR aims to prevent Prometheus increase()/rate() from missing cost when a counter series’ first observed sample already contains money, by reworking how cost “baseline” (zero-valued) series are published for nebula_cost_usd_total.
Changes:
- Move baseline seeding away from a once-at-startup
CostAccrual.Start()pass and intoNodeClaimReconciler.markPhase()(level-triggered reseeding behavior). - Refactor and rename the ungated price parsing helper (
finalRate→parsePrice) and update call sites/comments. - Update tests and documentation to reflect the new seeding approach and its limitations.
File summaries
| File | Description |
|---|---|
| README.md | Small wording tweak in the features list. |
| pkg/metrics/cost.go | Updates metric comments to point at the new seeding approach (seedClaimBaseline / parsePrice). |
| internal/controller/nodeclaim_controller.go | Seeds cost baselines after a successful status patch and on subsequent passes. |
| internal/controller/cost_accrual.go | Removes startup seeding, introduces seedClaimBaseline, and renames finalRate → parsePrice. |
| internal/controller/cost_accrual_test.go | Reworks tests to validate per-reconcile baseline seeding and restart reseeding behavior. |
| docs/metrics.md | Updates documentation to describe the new level-triggered baseline republishing model and caveats. |
Review details
Suppressed comments (1)
docs/metrics.md:452
- The statement “At a 60s scrape it will not.” is too categorical: with
accrualInterval = 1mand scrape interval also 60s, whether a scrape lands between baseline publish and the first booked window is timing-dependent. This section should describe the reliability/guarantee rather than implying it never works.
is worth nothing unless **the scrape interval is shorter than `accrualInterval`**: the gap between a
baseline and the first window charged on it is one tick, and a scrape has to land inside it. At a
60s scrape it will not.
- Files reviewed: 6/6 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
/lgtm |
InftyAI-Agent
left a comment
There was a problem hiding this comment.
Approved: PR has both lgtm and approved labels
InftyAI-Agent
left a comment
There was a problem hiding this comment.
Approved: PR has both lgtm and approved labels
What this PR does / why we need it
Which issue(s) this PR fixes
Fixes #
Special notes for your reviewer
Does this PR introduce a user-facing change?