Skip to content

[CASSANDRA-20333][trunk] Reduce DecayingEstimatedHistogramReservoir update cost with per-thread update buffers - #5185

Open
netudima wants to merge 1 commit into
apache:trunkfrom
netudima:CASSANDRA-20333-trunk
Open

netudima wants to merge 1 commit into
apache:trunkfrom
netudima:CASSANDRA-20333-trunk

Conversation

@netudima

Copy link
Copy Markdown
Contributor


Reservoir updates accumulate in a thread-local buffer instead of going to the shared striped arrays one at a time. An update encodes into a single int -- reservoir id, bucket index, and a delta from the buffer's baseline second -- so the buffer is a plain int[], shared by every reservoir the thread updates, with a single writer and multiple readers.

A flush counts identical entries through an open addressing hashtable, which collapses repeats into one atomic update per bucket. Many histograms record a near-constant value (written row counts, for instance) and latencies are typically multi-modal, so that aggregation removes most of the atomic traffic.

A flush also groups one reservoir's entries together, so each reservoir is applied in a single call that reuses the exponential decay factors it computed and touches the reservoir's arrays with better locality. Sorting proved too slow for this, so grouping uses a second open addressing hashtable keyed by reservoir id: it counts the entries per reservoir, turns those counts into ranges, and compacts each entry into its reservoir's range. Every structure a flush uses is reused, so a flush allocates nothing.

Metric reads flush the buffers first. A scrape may read thousands of histograms in a row, so a time check for the last flush is introduced to reduce flush rate

Additionally:

  • findIndex reads a short[] lookup table instead of computing log2 in floating point
  • the bucket arrays carry a leading cache line of padding, so no live bucket shares a line with the array length field that every bounds check reads
  • MonotonicClock.nowInSec() is sampled by the approximate clocks, so the decay path no longer divides nanoseconds on every update

NOTE: snapshot merging rescaled by a wrong landmark difference divided

patch by Dmitry Konstantinov; reviewed by TBD for CASSANDRA-20333

…d update buffers

Reservoir updates accumulate in a thread-local buffer instead of going to the
shared striped arrays one at a time. An update encodes into a single int --
reservoir id, bucket index, and a delta from the buffer's baseline second -- so
the buffer is a plain int[], shared by every reservoir the thread updates, with
a single writer and multiple readers.

A flush counts identical entries through an open addressing hashtable, which
collapses repeats into one atomic update per bucket. Many histograms record a
near-constant value (written row counts, for instance) and latencies are
typically multi-modal, so that aggregation removes most of the atomic traffic.

A flush also groups one reservoir's entries together, so each reservoir is
applied in a single call that reuses the exponential decay factors it computed
and touches the reservoir's arrays with better locality. Sorting proved too slow
for this, so grouping uses a second open addressing hashtable keyed by reservoir
id: it counts the entries per reservoir, turns those counts into ranges, and
compacts each entry into its reservoir's range. Every structure a flush uses is
reused, so a flush allocates nothing.

Metric reads flush the buffers first. A scrape may read thousands of histograms in
a row, so a time check for the last flush is introduced to reduce flush rate

Additionally:
 - findIndex reads a short[] lookup table instead of computing log2 in floating
   point
 - the bucket arrays carry a leading cache line of padding, so no live bucket
   shares a line with the array length field that every bounds check reads
 - MonotonicClock.nowInSec() is sampled by the approximate clocks, so the decay
   path no longer divides nanoseconds on every update

NOTE: snapshot merging rescaled by a wrong landmark difference divided

patch by Dmitry Konstantinov; reviewed by TBD for CASSANDRA-20333
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant