Skip to content

fix: bound Prometheus histogram memory between scrapes - #363

Open
korkin25 wants to merge 1 commit into
icoretech:mainfrom
korkin25:fix/upstream-bounded-histograms
Open

fix: bound Prometheus histogram memory between scrapes#363
korkin25 wants to merge 1 commit into
icoretech:mainfrom
korkin25:fix/upstream-bounded-histograms

Conversation

@korkin25

@korkin25 korkin25 commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Prometheus Core 1.2.1 stores each distribution observation until the next scrape.
With telemetry enabled and no scraper, routine database queries keep growing the
raw-sample ETS table: a synthetic run increased its memory from 181,384 bytes at
1,000 events to 16,015,816 bytes at 100,000 events.

Aggregate histograms when observations arrive, retaining one bucket/count/sum row
per metric and label combination. Keep Core's scalar handlers and exporter, the
existing inclusive bucket boundaries and fractional sums, and the metrics
endpoint's authorization and response format. Memory depends on distinct series
and configured buckets rather than observations between scrapes.

Validation of the identical collector/reporter/test implementation: nine regression
tests passed, including one million events without scraping, concurrent writers
and scrapes, exact boundaries and sums, malformed measurements, and collector
restart/detachment. The no-scrape test retained one aggregate and zero raw samples,
using 1,449 ETS words at both 100,000 and 1,000,000 events. Compilation, strict Credo
and compile-connected xref checks passed for that implementation. No dependencies
are added or upgraded.

@masterkain masterkain self-assigned this Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants