Don't zero out Gauges. - #813
Conversation
krajorama
left a comment
There was a problem hiding this comment.
I think it makes sense to keep the current behavior to be in line with what client_golang and client_java does. It could be the basis of introducing constant labels later. See comment inside.
a40779b to
2a2b386
Compare
Also does some lifecycle work for how and when stores are initialized, in preparation for more extensive changes for prometheus#812 Fixes prometheus#622 Signed-off-by: Jason Marshall <jdmarshall@users.noreply.github.com>
I'm trying to change that behavior at the moment (#812, #814), as a side effect of fixing three other issues. Some of the code in this PR is in service of that work, but could be split out. |
|
Thinking about this more: in general it seems the issue is that there's no separation between constructing a constant label metric versus dynamic label metric like in Go (e.g. NewCounter vs NewCounterVec). Interestingly Java is in the same state as client_js right now: https://prometheus.github.io/client_java/getting-started/labels/#initializing-label-values . @zeitlinger is it causing issues for Java that metrics without labels are automatically set to 0 and are exposed? Is there plans to change it? BTW, isn't simply removing the current zeroing out in this PR a breaking change? Someone might rightfully depend on it. IMHO the absolute cleanest solution would be to introduce NewCounter/NewCounterVec factory functions that cleanly separate the two use cases. |
|
Oh it's definitely a breaking change, which is why #622 is marked as semver-major. But it's been complained about by a few different people. Gauges getting sampled as zeros is no bueno when you're using min/max or average aggregation. As for the factory function, that's one of the goals of #812, but it's a fair point that I should write that down since it's all in my head at the moment. In fact I might need to make an epic ('project') for this work. |
|
Also, I still don't see any evidence in the Go code of Gauges being initialized to zeros. You gave Counter examples, which absolutely do need to be initialized to 0. |
|
@krajorama This change has no affect on Counter, which is intended to be initialized to zero. This is gauge, where no initial value is correct or can be. We don't know what's being measured. CPU%, req/s, heap usage. |
It's relatively easy to reproduce the Gauge behavior with client_golang, I have a dumb little testbed that I use and tweak all the time. The constant labels version does init to 0. Whereas the dynamic labels one doesn't: https://github.com/krajorama/promtestsource/blob/dynamic-gauge/promtestsource.go#L208 . From the code it seems like client_java does the same, but I have not tested it directly.
Ok, that's fair.
That's also fair, interestingly the specification still calls for starting at 0 (and may offer a way to init to non-zero): https://prometheus.io/docs/instrumenting/writing_clientlibs/#gauge. But the spec doesn't say if this has to exposed or not. Which would be very impossible for dynamic labels anyway. Summary:
I think a way forward could be to do this breaking change, but also add a new API that allows for explicitly setting constant labels (even zero) so that you can switch to using that if you're depending on the current behavior. The new API would allow 0..n constant labels (label name+value). What do you think? |
|
Actually you may not need a new API at all , just document that you need to observe a value to make it appear right away, right? |
|
@krajorama I have captured your research as a requirement of #812 which sounds reasonable to me. I still think it's wrong but probably consistency matters here. What I'd like to do is converse with the other implementers and ask why we are collectively doing it this way and whether my reasons for suggesting we all stop are more compelling for their reasons for doing it this way. But that would have required me to register for the convention and I hadn't been aware it existed until after the cutoff date. |
krajorama
left a comment
There was a problem hiding this comment.
Approved as agreed, acknowledging it's a breaking change. Will have to be explained in the release notes.
Added some Claude findings as comments, could be follow up as well.
| }); | ||
| await expectValue(0); | ||
|
|
||
| await expect(instance.get()).resolves.toBeEmpty; |
There was a problem hiding this comment.
claude claims this is no-op, should be something like
expect((await instance.get()).values).toEqual([]);
|
|
||
| instance.reset(); | ||
| expect((await instance.get()).values[0].value).toEqual(0); | ||
| expect(await instance.get()).toBeEmpty; |
There was a problem hiding this comment.
claude claims this is no-op, should be something like
expect((await instance.get()).values).toEqual([]);
This PR started as exploratory work for a different issue - there's been a TODO that I added to metric.js that fights the work needed for #812. I had already decided that doing #812 all in a single PR would probably get a little bit on the long side, so I was looking for some stopping point in the middle to split it up.
I noticed that if #622 were fixed, then a default
reset()implementation would be used by 2 of the 4 builtin metric types.This PR initializes the store in metrics.js, and then retains the LabelMap object across
reset()calls, and removes the zeroing logic fromGauge.reset(), thereby fixing #622.