diff --git a/.gitignore b/.gitignore index 61e98e96a3..543983a2d6 100644 --- a/.gitignore +++ b/.gitignore @@ -56,6 +56,7 @@ CLAUDE.md # AI review/agent tooling scratch state .sphinx/ +/.scratch/ .skill-builder-temp/ .claude/scheduled_tasks.lock @@ -64,3 +65,4 @@ CLAUDE.md # AI investigation scratch state (never committed) .investigations/ +/notes/ diff --git a/.knowledgebase/.cairn b/.knowledgebase/.cairn new file mode 100644 index 0000000000..3e6a3b04e9 --- /dev/null +++ b/.knowledgebase/.cairn @@ -0,0 +1 @@ +{"generator": "cairn", "retired_threshold": 10} diff --git a/.knowledgebase/memories/ancestor-walk-early-stop-needs-flag.md b/.knowledgebase/memories/ancestor-walk-early-stop-needs-flag.md new file mode 100644 index 0000000000..fff6251012 --- /dev/null +++ b/.knowledgebase/memories/ancestor-walk-early-stop-needs-flag.md @@ -0,0 +1,22 @@ +--- +id: ancestor-walk-early-stop-needs-flag +title: trackLeakAccumulation's ancestor walk may stop early only at entries flagged ancestors_recorded +kind: finding +tags: [refchains, leak-fanout, stw] +applies_to: [ddprof-lib/src/main/cpp/referenceChainWalk.cpp] +source: feedback-1/find-ancestor-walk-early-stop-soundness +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + +Stopping the ancestor walk at the first parent already present in `_leak_parent_fanout` is +unsound. `seedLeakAccumulationForNewlyWatchedKlass()` inserts direct parents without walking their +ancestors, so a present entry does not imply its chain to the root was recorded. + +`LeakParentFanoutEntry::ancestors_recorded` is set by the walk and left false by the seed path; the +walk stops only when it is set. Regression tests: `TrackLeakAccumulationWalksPastSeededAncestor`, +`TrackLeakAccumulationSiblingsShareRecordedAncestors`. + +Refuted alternative: "break on any existing ancestor" drops chains above seeded entries. diff --git a/.knowledgebase/memories/assert-state-gate-ndebug.md b/.knowledgebase/memories/assert-state-gate-ndebug.md new file mode 100644 index 0000000000..c582460dd2 --- /dev/null +++ b/.knowledgebase/memories/assert-state-gate-ndebug.md @@ -0,0 +1,21 @@ +--- +id: assert-state-gate-ndebug +title: State read only by assert() must be gated on NDEBUG (ASSERT_ONLY), not DEBUG (DEBUG_ONLY) +kind: finding +tags: [build, gtest, assertions, macros] +applies_to: [ddprof-lib/src/main/cpp/common.h, build-logic/conventions/src/main/kotlin/com/datadoghq/native/gtest/GtestTaskBuilder.kt] +source: feedback-1/find-c6-debugonly-wrong-gate +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + +`assert()` is compiled in whenever `NDEBUG` is undefined. `DEBUG_ONLY()` keys on `DEBUG`. +The two diverge in the gtest release build: `GtestTaskBuilder.kt` removes `-DNDEBUG` to keep +assertions on but does not add `-DDEBUG`. A variable declared under `DEBUG_ONLY()` and read in +an `assert()` therefore fails to compile there (or, if the assert is also hidden, that build +silently loses the check). + +Use `ASSERT_ONLY()` from `common.h` (`#ifndef NDEBUG`) for state whose only readers are asserts. +Keep `DEBUG_ONLY()` for code that genuinely belongs to `DEBUG` builds. diff --git a/.knowledgebase/memories/class-tag-mint-serialized.md b/.knowledgebase/memories/class-tag-mint-serialized.md new file mode 100644 index 0000000000..253b1f3274 --- /dev/null +++ b/.knowledgebase/memories/class-tag-mint-serialized.md @@ -0,0 +1,23 @@ +--- +id: class-tag-mint-serialized +title: Class-tag minting must go through ClassTagAllocator::getOrMint() (GetTag/SetTag under one mutex) +kind: finding +tags: [jvmti, tags, concurrency, refchains, liveness] +applies_to: [ddprof-lib/src/main/cpp/classTagAllocator.h, ddprof-lib/src/main/cpp/livenessTracker.cpp, ddprof-lib/src/main/cpp/referenceChainWalk.cpp, ddprof-lib/src/main/cpp/referenceChainAnchors.cpp] +source: feedback-1/find-c9-classtag-race +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + +JVMTI has no tag compare-and-set. The reference-chain tracker (`resolveLoadedClasses()`, +`resolveContainerInterfaceTags()`) and `LivenessTracker::mintStableClassTagIfNeeded()` can tag the +same class concurrently. A bare GetTag -> mint -> SetTag sequence loses updates: one side caches +a tag the class object no longer carries, which silently breaks attribution for that class for +the rest of the session (no crash). + +`ClassTagAllocator::getOrMint()` runs the sequence under a function-local `Mutex`. It is a +blocking mutex, not a spinlock, because it is held across JVMTI calls and application threads can +be callers. New class-tag minting sites must use it. `ClassTagGetOrMintRaceTest` (8 threads, slow +injected GetTag) fails if the lock is removed. diff --git a/.knowledgebase/memories/container-runtime-macos-colima.md b/.knowledgebase/memories/container-runtime-macos-colima.md new file mode 100644 index 0000000000..958a610e6e --- /dev/null +++ b/.knowledgebase/memories/container-runtime-macos-colima.md @@ -0,0 +1,31 @@ +--- +id: container-runtime-macos-colima +title: Container tests on macOS: podman is broken locally, use colima+docker +kind: finding +tags: [] +applies_to: [] +source: implement-github-issue-sphinx/container-runtime-macos-colima +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + +# Container tests on macOS: podman is broken locally, use colima+docker + +**Observed symptom (verified on this machine):** every `podman run` command aborts with +`error getting credentials` before the container starts. Cause: podman resolves the Datadog +registry credentials via the `ddtool` credHelper declared in ~/.docker/config.json +(credHelpers for registry-staging.ddbuild.io / registry.ddbuild.io / registry.us1-build.fed.dog); +ddtool fetches an identity JWT from vault.us1-build.fed.dog, which /etc/hosts blackholes to +127.0.0.1 ("Explicitely fail probe to fed vault"). Fatal because podman treats helper failure +as an error. + +**Working path:** colima (docker runtime) + docker CLI client (`brew install docker`, +`brew link --overwrite docker` if a Docker.app bash-completion symlink blocks linking). +Colima context already set in ~/.docker/config.json (currentContext: colima). +`./utils/run-containers-tests.sh --container=docker ...` works; Docker Hub pulls +(debian/eclipse-temurin base images) do not touch the ddtool helpers. + +Docs updated: AGENTS.md, README.md, utils/README.md (commit b412cd098). Script default +(podman, for Linux CI) intentionally unchanged. diff --git a/.knowledgebase/memories/dead-end-cross-thread-free.md b/.knowledgebase/memories/dead-end-cross-thread-free.md new file mode 100644 index 0000000000..a1860347a8 --- /dev/null +++ b/.knowledgebase/memories/dead-end-cross-thread-free.md @@ -0,0 +1,17 @@ +--- +id: dead-end-cross-thread-free +title: Cross-thread free (producer/consumer) is not a driver of glibc arena blowup; no free hook is needed to detect it +kind: dead-end +tags: [glibc, free, tcache, arena] +applies_to: [] +source: malloc-arena-detection/hyp-cross-thread-free +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + +A chunk freed by a thread other than its allocator goes into the freeing thread's tcache (7 per bin), otherwise +back to the OWNING arena (fastbin via CAS or under that arena's lock; 2.39 `malloc.c:4547-4549`, `4621`, +`4640-4646`). The memory returns to the arena it came from and is reused there. Extra retention is bounded +(tcache, fastbins blocking coalescing). The blowup driver is arena count x per-arena peak plus non-trimming. diff --git a/.knowledgebase/memories/dead-end-free-ratio-arena-signal.md b/.knowledgebase/memories/dead-end-free-ratio-arena-signal.md new file mode 100644 index 0000000000..2a14b016e5 --- /dev/null +++ b/.knowledgebase/memories/dead-end-free-ratio-arena-signal.md @@ -0,0 +1,18 @@ +--- +id: dead-end-free-ratio-arena-signal +title: The free-held ratio (fordblks / arena) does not identify glibc arena blowup; use absolute free-held bytes +kind: dead-end +tags: [glibc, mallinfo, detection, arena] +applies_to: [] +source: malloc-arena-detection/find-detection-signature +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + +After the frees, the ratio of free to committed arena bytes is about 100% both with the default arena limit and +with a quota-aware `MALLOC_ARENA_MAX`, so a ratio threshold cannot separate the pathological case from a healthy +one. What differs is the absolute amount: the default limit held 3.7-9x more bytes. Committed bytes also mislead, +because they include in-use memory that fewer arenas do not reclaim; free-held bytes (`mallinfo2().fordblks`) are +the reclaimable part. diff --git a/.knowledgebase/memories/durations-use-nanotime-not-tsc.md b/.knowledgebase/memories/durations-use-nanotime-not-tsc.md new file mode 100644 index 0000000000..214145c36f --- /dev/null +++ b/.knowledgebase/memories/durations-use-nanotime-not-tsc.md @@ -0,0 +1,20 @@ +--- +id: durations-use-nanotime-not-tsc +title: Measure elapsed durations with OS::nanotime(), not TSC::ticks() deltas +kind: finding +tags: [timing, tsc, refchains] +applies_to: [ddprof-lib/src/main/cpp/referenceChainAnchors.cpp] +source: feedback-1/find-c10-tsc-underflow +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + +`TSC::ticks()` reads rdtsc on x86, which is not guaranteed monotonic across cores/sockets. An +end-minus-start `u64` delta that steps back underflows to a huge value. In the reference-chain +tracker such a value fed the safepoint cost accounting, which then disabled the search for the +rest of the process. + +For durations that feed budgets or accumulators, use `OS::nanotime()` (monotonic clock). Its cost +is negligible at per-pass/per-follow sites. diff --git a/.knowledgebase/memories/find-c10-tsc-underflow.md b/.knowledgebase/memories/find-c10-tsc-underflow.md new file mode 100644 index 0000000000..d8796be58e --- /dev/null +++ b/.knowledgebase/memories/find-c10-tsc-underflow.md @@ -0,0 +1,28 @@ +--- +id: find-c10-tsc-underflow +title: C10 partially valid: x86 rdtsc may step back; unguarded u64 deltas -> ~6e9 ms pain debt disables RC search for process life; 5 such sites +kind: finding +tags: [pr-797, tsc, pacing, pain-budget] +applies_to: [] +source: feedback-1/find-c10-tsc-underflow +status: stale +recorded: 2026-11-02 +updated: 2026-11-02 +valid_at: 2026-11-02 +invalid_at: 2026-11-02 +retired_reason: "marked stale" +--- + + + + +# C10 partially valid: x86 rdtsc may step back; unguarded u64 deltas -> ~6e9 ms pain debt disables RC search for process life; 5 such sites + +## Reasoning chain +tsc.h:88 rdtsc-_offset. ticks_to_millis wraps to ~6e9 ms -> _search_pain_ms (traversal:1070) -> canAffordNewSearch false. Sites anchors:109, traversal:204,629,886,1041. Comment traversal:1042 falsely says monotonic. Recommend OS::nanotime() for durations, keep TSC for JFR _start_time. + +## Evidence +- Full write-up: reports/PR797Review5357763282.md; raw notes evidence/D.md. Lines at 9521394e8. + +## Follow-up (2026-10-01) +- nanotime cost is fine: every TSC pair brackets one FollowReferences (an STW heap op of ms), O(batches+anchors) per pass. Linux vDSO clock_gettime costs tens of ns (a syscall of about 1µs without vDSO); macOS uses mach_absolute_time. The same function already uses nanotime around GetObjectsWithTags (traversal:544-548). diff --git a/.knowledgebase/memories/find-c2-settag-779-conflict.md b/.knowledgebase/memories/find-c2-settag-779-conflict.md new file mode 100644 index 0000000000..85f5ac71ef --- /dev/null +++ b/.knowledgebase/memories/find-c2-settag-779-conflict.md @@ -0,0 +1,26 @@ +--- +id: find-c2-settag-779-conflict +title: C2 valid+severe: #779 and #797 share VM::jvmti() tag space; #779 positive count vs #797 negative identity collide on JDK8 +kind: finding +tags: [pr-797, classTagAllocator, jvmti-tags, pr-779] +applies_to: [] +source: feedback-1/find-c2-settag-779-conflict +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + + +# C2 valid+severe: #779 and #797 share VM::jvmti() tag space; #779 positive count vs #797 negative identity collide on JDK8 + +## Reasoning chain +#779 SetTag via VM::jvmti() gated hotspot_version()==8; RCT has no JDK gate, hasLeakSignal() true when liveness off (<11). #779-first: adopted positive tag hits *tag_ptr>0 arm (referenceChainWalk.cpp:401) -> improveChain on unrelated frontier entry; releaseSearchTags clears #779 dedup tag. #797-first: #779 over-patches and overwrites negative tag. Fix: separate jvmtiEnv (A for RC+LT or B for #779) + defensive asserts. + +## Evidence +- Full write-up: reports/PR797Review5357763282.md; raw notes evidence/A.md. Lines at 9521394e8. + +## Follow-up (2026-10-01) +- Masking is not feasible. GetObjectsWithTags uses exact match (anchors:531,615; labels:212; traversal:545,936), so the identity would change on every #779 re-patch. The combination is also a read-modify-write and JVMTI has no tag CAS. +- A second env is just `JavaVM::GetEnv(JVMTI)` again: no re-attach, callable in the live phase. `can_tag_objects` is in HotSpot's always set (jdk8u jvmtiManageCapabilities.cpp init_always_capabilities). Use DisposeEnvironment at shutdown. +- Recommend B: move only #779's GetTag/SetTag pair, which is JDK8 HotSpot only, so no J9 concern. A would touch ~90 tag lines in 6 TUs, and FollowReferences/GetObjectsWithTags would have to move with them. diff --git a/.knowledgebase/memories/find-ddtrace-smaps-not-baseline.md b/.knowledgebase/memories/find-ddtrace-smaps-not-baseline.md new file mode 100644 index 0000000000..6c70b90d95 --- /dev/null +++ b/.knowledgebase/memories/find-ddtrace-smaps-not-baseline.md @@ -0,0 +1,32 @@ +--- +id: find-ddtrace-smaps-not-baseline +title: dd-trace-java smaps events are opt-in, JDK 23+ only, and do not identify malloc arenas +kind: finding +tags: [dd-trace-java, smaps, jfr, always-on] +applies_to: [] +source: malloc-arena-detection/find-ddtrace-smaps-not-baseline +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + + +# dd-trace-java smaps events are opt-in, JDK 23+ only, and do not identify malloc arenas + +## Reasoning chain +dd-trace-java (HEAD c3ea2067) emits `datadog.SmapEntry` (one event per mapping, @Period beginChunk) and +`datadog.AggregatedSmapEntry` (rss summed by nmtCategory). Both default off +(`profiling.smap.collection.enabled`, `profiling.smap.aggregation.enabled`). Registration requires Linux, non-J9, +not Oracle JDK8, and the `System.map` DiagnosticCommand (JDK 23+). Classification is by exact start address +against System.map descriptions; arena heaps land in UNKNOWN/UNDEFINED. No 64MB-alignment/PROT_NONE heuristic. +Pure Java, no ddprof involvement. + +## Evidence +- profiling-controller-openjdk/.../events/SmapEntryEvent.java:14-18, AggregatedSmapEntryEvent.java:14,36-39 +- SmapEntryCache.java:41,154-271,274-309; ProfilingConfig.java:252-260; SmapEntryFactory.java:27-58; OpenJdkController.java:250-271 + +## What this rules out +- Using smaps as the always-on detection baseline (off by default, JDK>=23). +- Still useful as offline confirmation: when enabled, per-mapping data lets a backend count 64MB-aligned anon + regions (arena heaps) post hoc — a complementary, not primary, signal. diff --git a/.knowledgebase/memories/find-deoptimize-frames-escape-barrier.md b/.knowledgebase/memories/find-deoptimize-frames-escape-barrier.md new file mode 100644 index 0000000000..de646023ba --- /dev/null +++ b/.knowledgebase/memories/find-deoptimize-frames-escape-barrier.md @@ -0,0 +1,61 @@ +--- +id: find-deoptimize-frames-escape-barrier +title: IterateOverReachableObjects forces EA-object materialization: 626 DeoptimizeFrames proven, not GetObjectsWithTags +kind: finding +tags: [hotspot, jvmti, escape-analysis, deoptimization, cost-model, parallel-session] +applies_to: [ddprof-lib/src/main/cpp/referenceChains.cpp] +source: writing-any-code-thoroughly/find-deoptimize-frames-escape-barrier +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + + +# IterateOverReachableObjects forces EA-object materialization: 626 DeoptimizeFrames proven, not GetObjectsWithTags + +## Reasoning chain +Parallel session PROVED (earlier hand-waving corrected) the source of the 626 +DeoptimizeFrames in the recording: + +- Path: IterateOverReachableObjects (BFS root enumeration) -> HotSpot + JvmtiTagMap -> EscapeBarrier -> materializes EA-optimized (scalar-replaced) + objects across ALL threads -> Deoptimization::deoptimize_frame per eligible + frame -> posts VM_DeoptimizeFrame, then HeapWalkOperation. Verified in + jdk-27+35 source (escapeBarrier.cpp -> deoptimize_frame, + deoptimization.cpp:1553). +- Reproduced standalone (Temurin 26): each enumeration = 1 global handshake + (HandshakeAllThreads) + 8 deopts + 1 walk; with -XX:-DoEscapeAnalysis: + 0 deopts. GetObjectsWithTags is NOT the source. +- Only frames holding objects that DO NOT EXIST ON THE HEAP are affected: + scalar-replaced allocations (object lives only in registers/stack slots of + the compiled frame) or locked-in-compiled-frame monitors. Normal frame-held + oops are walked directly - those threads are untouched. +- Deopt semantics: per-frame, per-activation. The killed activation continues + interpreted until it returns (hot loop: OSR rescues after ~10k+ backedges + + compile latency). The nmethod stays valid - subsequent invocations enter + compiled code directly, NO re-warmup, no recompilation storm. This is NOT + jdk.Deoptimization (uncommon trap): no speculation failure, no method + invalidation. +- Recurrence shape: each enumeration pass re-materializes whatever + EA-optimized frames are live at that moment, so ~5 passes/s during warmup + re-deopts a fresh batch (~48 eligible frames per handshake observed) on an + EA-heavy C2 workload like prof-analyzer (446 threads). +- Measured: 626 ops / 0.24 ms total VM-op time - the deopts themselves are + cheap; the real cost is the 13 global handshakes plus the interpreter dip + for each killed frame's remaining lifetime. 253 HeapWalks during the ~50s + episode each paid handshake + deopt batch. + +## Evidence +- Parallel-session reproducers: /tmp/rc-investigation.JRtDyL (Temurin 26 + standalone: handshake/deopt/walk counts, +/-EscapeAnalysis). + +## What this rules out +- GetObjectsWithTags as the DeoptimizeFrame source. +- Recompilation-storm theories: forced per-frame materialization deopts do + not invalidate nmethods. + +## Cost implication for the fix sizing +Every candidate-less or not, each BFS pass pays handshake + deopt batch - +bounding pass counts (the TTL/no-progress exits) bounds this cost; the +candidate-gate fixes (A/B) do exactly that by preventing pointless searches. diff --git a/.knowledgebase/memories/find-detection-signature.md b/.knowledgebase/memories/find-detection-signature.md new file mode 100644 index 0000000000..96c30c003d --- /dev/null +++ b/.knowledgebase/memories/find-detection-signature.md @@ -0,0 +1,38 @@ +--- +id: find-detection-signature +title: Measured signature of the pattern: arena count above 8 x quota CPUs with ~1 attached thread per arena; free-held ratio alone does not discriminate +kind: finding +tags: [detection, measurement, arena, reproducer] +applies_to: [] +source: malloc-arena-detection/find-detection-signature +status: stale +recorded: 2026-11-02 +updated: 2026-11-02 +valid_at: 2026-11-02 +invalid_at: 2026-11-02 +retired_reason: "marked stale" +--- + + + + +# Measured signature of the pattern: arena count above 8 x quota CPUs with ~1 attached thread per arena; free-held ratio alone does not discriminate + +## Reasoning chain +From 144 reproducer results (find-repro-arm64-results): +- Thread-private arenas: in the retained phase with T <= 64, walk_count = T + 2 and max_attached = 1 -> every + allocating thread owns an arena. With a binding limit (ARENA_MAX 2/16) attached rises to 23-155 (all targets). +- Free-held ratio (sum_sys - in_use) / sum_sys is ~100% after frees in BOTH default and capped runs (2.39 mi2), + so a ratio threshold cannot tell the pathological case from a healthy one. The discriminator is magnitude + relative to what a quota-aware limit would retain: T=256 default / quota-aware(16) = 3.7x-9x across all targets. +- Hence the rule shape: (1) arena_count > 8 x ceil(quota CPUs) [config not quota-aware AND exercised], plus + (2) sum of non-main system_mem large relative to the container memory limit. The percentage in (2) is a policy + choice; the reproducer cannot calibrate it (workload-dependent per-thread peak). +- Remedy hint the event can carry: MALLOC_ARENA_MAX ~ 8 x quota CPUs (measured 4-9x lower retention). + +## Evidence +- build/arena-repro/results/*.txt; summarize.py table. + +## What this rules out +- A pure fragmentation-ratio detector (free/arena) as the arena-blowup signal. +- Requires attached_threads, present in all three calibrated layouts seen (absent only in upstream 2.17-2.22). diff --git a/.knowledgebase/memories/find-detector-rule.md b/.knowledgebase/memories/find-detector-rule.md new file mode 100644 index 0000000000..eb20af082e --- /dev/null +++ b/.knowledgebase/memories/find-detector-rule.md @@ -0,0 +1,118 @@ +--- +id: find-detector-rule +title: Detector rule: warn when arena_count > 8 x ceil(quota CPUs) AND non-main arenas hold more than P% (default 25%, override arenawarn=) of native headroom (memory limit - max heap) +kind: finding +tags: [detection, rule, arena, design, thresholds] +applies_to: [] +source: malloc-arena-detection/find-detector-rule +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + + +# Detector rule: warn when arena_count > 8 x ceil(quota CPUs) AND non-main arenas hold more than P% (default 25%, override arenawarn=) of native headroom (memory limit - max heap) + +## Purpose (agreed with user 2026-10-26) +The warning reports ONE specific, fixable problem: glibc malloc holds a lot of native memory across many per-thread +arenas because its arena limit ignores the container CPU quota; remedy MALLOC_ARENA_MAX=N. +- Condition 1 = cause (limit not quota-aware AND exercised); condition 2 = relevance (headroom is only the yardstick + for "material", NOT the subject of the warning); recommendedArenaMax = remedy. +- NOT a general low-native-memory warning (metaspace/stacks/direct buffers do not satisfy condition 1); NOT an OOM + prediction. +- "Held" = arena system_mem (reserved), not in use. Free share only via optional freeHeldBytes when mallinfo2 is + available (PROF-16350). + +## Rule +Evaluated once per JFR chunk on the flush path (not signal context). + +Inputs: +- `arena_count`, per-arena `system_mem` and `attached_threads`, `main` flag: lock-free arena walk with runtime + layout calibration (find-arena-walk-safe, find-el7-layout-backport). +- `quota_cpus`: cgroup v2 `cpu.max` quota/period, else v1 `cpu.cfs_quota_us`/`cpu.cfs_period_us`. +- `mem_limit`: cgroup v2 `memory.max`, else v1 `memory.limit_in_bytes`. +- `arena_limit` and its source: `GLIBC_TUNABLES` glibc.malloc.arena_max, else `MALLOC_ARENA_MAX`, else + 8 x get_nprocs() (online CPUs, the value glibc uses). + +Conditions (both required): +1. `arena_count > 8 x ceil(quota_cpus)` — the limit is not quota-aware AND the workload used the extra arenas. +2. `sum(system_mem of non-main arenas) > P% of native_headroom`, native_headroom = mem_limit - max_heap — the + extra arenas hold a material share of the memory not reserved for the Java heap. + Uses committed arena bytes, NOT the free-held ratio (find-detection-signature: that ratio is ~100% after frees + in both bad and healthy runs). + +Characterization reported with the warning (not a gate): +- `attached_threads` distribution: ~1 per arena = thread-private arenas, the many-threads x small-allocs pattern + (reproducer: T <= 64 -> arena_count = T + 2, attached 1 on all targets; binding limit -> 23-155). +- Largest per-arena `system_mem` / `max_system_mem` (per-thread peak). + +Remedy carried in the event: `MALLOC_ARENA_MAX = 8 x ceil(quota_cpus)` (reproducer, T=256: 4-9x less retained). + +## Where the rule runs +Decision (user, 2026-10-26): evaluate the rule IN THE PROFILER and emit a dedicated structured warning JFR event +with the relevant details; do not rely on backend computation. Rationale: only the profiler knows input validity +(calibrated layout, walk status, attached_threads availability); evaluation cost is negligible (comparisons over +values already collected per chunk); one versioned, tested implementation. +Conditions agreed in the proposal: +- P is runtime-configurable (profiler argument, conservative default) — a hardcoded P needs an agent release per change. +- Raw inputs travel with the warning or in a lean stats event (arena_count, sums, largest arena, quota, mem limit, + limit source, layout, walk status) — needed to debug false positives/negatives and to calibrate P. +- Edge-triggered emission: on condition entry, on worsening, on clearing — not one event per chunk. +- Not `profiler.Log` (jfrMetadata.cpp:425): free text only, and the Log::log -> JFR path is commented out + (log.cpp:97); a structured event avoids backend string parsing. + +## Degraded paths +- No CPU quota: condition 1 cannot exceed glibc's own limit, so the rule does not fire. Correct for the + quota-mismatch case; plain many-thread blowup on an unconstrained host is NOT covered by this rule. +- No cgroup memory limit: physical RAM as mem_limit (memoryLimitSource=physical); headroom computed from it. +- Zing: walk turned off (no fault recovery there; q-zing-walk-safety) -> no warning, walkStatus=unsupported-jvm. +- Walk disabled (no calibrated layout passes, seed lands only in main_arena, unsupported allocator): no warning; + the stats event reports walkStatus with the reason. +- `attached_threads` absent (upstream 2.17-2.22 layout): emit without the characterization. +- mallinfo2 unavailable (glibc < 2.33, or PROF-16350 unfixed): condition 2 falls back to committed non-main + system_mem (may overstate reclaimable memory) -> PROF-16350 is a prerequisite for an accurate rule. + +## Condition 2 measure (revised after JVM baseline, find-jvm-baseline) +- Use FREE-HELD bytes (mallinfo2 fordblks: reserved but not in use) as condition 2's numerator, not committed + non-main system_mem: fewer arenas reclaim free space, not in-use bytes. x86_64 JDK8 http: committed 46.7 MiB + (fires) vs free-held 26.5 MiB (silent at 32 MiB threshold). On aarch64/2.43 both are equal (2 MiB THP floor is + free top space). +- Fallback without mallinfo2 (glibc < 2.33 or PROF-16350 unfixed): committed non-main system_mem, flagged in the event. +- mallinfo2 includes main_arena free space (not caused by arena count); accepted, condition 1 gates the cause. +- Condition 1 does not filter JVMs (arenas ~= threads > 8 x quota almost always); it remains the cause gate that + guarantees MALLOC_ARENA_MAX would bind. + +## Native headroom and threshold P (user, 2026-10-26: good default + override; heap defaults are not a basis, apps set -Xmx) +- max_heap = effective `Runtime.maxMemory()` (reflects -Xmx / MaxRAMPercentage / ergonomics), resolved ONCE at + profiler start (JNIEnv available) and cached — fixed for process lifetime; no JNI on the flush path. + Precedent: HeapUsage::getMaxHeap() (hotspot/vmStructs.cpp:1087), cached by the liveness tracker + (livenessTracker.h:417). That helper is under hotspot/ -> shared detector code must use the JVM abstraction layer + or call Runtime.maxMemory() via JNI itself (standard API on HotSpot/J9/Zing). +- mem_limit = cgroup limit, else physical RAM (sysinfo); memoryLimitSource reports which. +- Fallbacks: max_heap unknown or Long.MAX_VALUE -> headroom = mem_limit (headroomSource=limit-only); + max_heap >= mem_limit -> headroom = mem_limit and heapExceedsLimit=true. +- Max (not committed) heap is correct: the heap may grow to max; the question is whether heap + native fits. +- Default P = 25% of native headroom; override `arenawarn=` (arguments.cpp style), `arenawarn=0` disables + the warning, stats event stays. Policy default: headroom also holds metaspace, code cache, thread stacks, GC + structures, direct buffers; stranded arenas from the quota mismatch should not take a quarter of it. +- Not measurable by the reproducer (no Java heap; per-thread peak is workload-specific). The rule is insensitive to + the exact P because the two regimes differ 3.7-9x at T=256. Refine from datadog.MallocArenaStats, which must carry + maxHeap and memoryLimit so headroom and P can be recomputed offline. +- Superseded earlier draft: 10% of the memory limit justified by JDK MaxRAMPercentage defaults — rejected (user): + apps commonly set -Xmx explicitly, so heap defaults are not a valid basis. + +## Events +- `datadog.MallocArenaWarning` (state raised/worsened/cleared; rule inputs; thresholdPercent; threadPrivateArenas, + largest arena; recommendedArenaMax; glibcVersion, layout, walkStatus). +- `datadog.MallocArenaStats` lean per-chunk raw values for every process where the walk runs. +- Hysteresis: `worsened` when nonMainArenaBytes grows >= 25% over the last emitted warning; `cleared` when either + condition turns false. + +## Evidence +- Implementation ticket: PROF-16351 (2026-10-26), full design + open questions. +- build/arena-repro/results/*.txt (144 results, 6 targets), summarize.py; doc/temp/MallocArenaDetectionOptions.md + "Detection signature" (worktree start-cairn-investigation). + +## What this rules out +- A fragmentation-ratio detector; a version-table-only walk; gating on mallinfo2. diff --git a/.knowledgebase/memories/find-el7-missing-gtest-fixture.md b/.knowledgebase/memories/find-el7-missing-gtest-fixture.md new file mode 100644 index 0000000000..681a2141d8 --- /dev/null +++ b/.knowledgebase/memories/find-el7-missing-gtest-fixture.md @@ -0,0 +1,35 @@ +--- +id: find-el7-missing-gtest-fixture +title: PR #800 CI failure root cause and fix (closed) +kind: finding +tags: [ci, el7, gtest, rc6] +applies_to: [ddprof-test/src/test/java/com/datadoghq/profiler/referencechains/ReferenceChainJfrParserTest.java] +source: failing-caching-enabled-caching/find-el7-missing-gtest-fixture +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + + +# PR #800 CI failure root cause and fix (closed) + +## Symptom +PR #800 (jb/rc-6-java-tests) failing on el7 CI runners. Investigation opened this +morning for the failure report (title mentioned Gradle "Caching not enabled / +Caching disabled for ExternalDependencyInstrumentingArtifactTransform"); the concrete +failing test was ReferenceChainJfrParserTest. + +## Root cause (confirmed) +el7 CI runners have no Google Test installed, so the gtest-driven JFR roundtrip tool +never produces the fixture file the Java parser test expects; the test failed on the +missing file, not on parser logic. + +## Fix +`Assumptions.assumeTrue` skip when the fixture is absent (JUnit 5), commit 218c2b7cc +on jb/rc-6-java-tests (pushed). jb/rc-7-tooling forked before that commit, so the same +fix was re-applied there during the humanize pass (currently in the uncommitted tree). + +## Verification +ev-pr800-checks-green — `gh pr checks 800` after the fix: every check passes +(only `fuzz` 'skipping', an expected skip). diff --git a/.knowledgebase/memories/find-existing-mallinfo-snapshot.md b/.knowledgebase/memories/find-existing-mallinfo-snapshot.md new file mode 100644 index 0000000000..8eae6733ef --- /dev/null +++ b/.knowledgebase/memories/find-existing-mallinfo-snapshot.md @@ -0,0 +1,32 @@ +--- +id: find-existing-mallinfo-snapshot +title: A process-wide mallinfo2() snapshot already runs on every JFR chunk finish, independent of nativemem +kind: dead-end +tags: [mallinfo, counters, always-on, existing-infra] +applies_to: [] +source: malloc-arena-detection/find-existing-mallinfo-snapshot +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + + +# A process-wide mallinfo2() snapshot already runs on every JFR chunk finish, independent of nativemem + +## Reasoning chain +`Recording::finishChunk` path calls `updateNativeMemStats()` (ddprof-lib/src/main/cpp/flightRecorder.cpp:1082) which calls +`updateMallocArenaStats()` (ddprof-lib/src/main/cpp/flightRecorder.cpp:2214). That calls `mallinfo2()` (2170) when glibc >= 2.33 headers +(DD_HAVE_MALLINFO2, ddprof-lib/src/main/cpp/flightRecorder.cpp:13-17) and `glibcMallocActive()` (no `mallctl`/`tc_malloc_size`, not ASan/TSan). +Published as internal counters (Counters::set, compiled in for all Linux/macOS configs incl. release via -DCOUNTERS, build-logic/conventions/src/main/kotlin/com/datadoghq/native/config/ConfigurationPresets.kt:93): MALLOC_ARENA_BYTES (mi.arena), MALLOC_IN_USE_BYTES (uordblks), MALLOC_FREE_HELD_BYTES (fordblks), +MALLOC_TRIMMABLE_BYTES (keepcost), MALLOC_MMAP_BYTES (hblkhd) -> writeCounters() (ddprof-lib/src/main/cpp/flightRecorder.cpp:1083,2296) emits one `datadog.ProfilerCounter` event (startTime,name,count; ddprof-lib/src/main/cpp/jfrMetadata.cpp:344) per counter per chunk; also readable via JavaProfiler.getDebugCounters() (ddprof-lib/src/main/java/com/datadoghq/profiler/JavaProfiler.java:444). This is the profiler's internal diagnostics table, not a user-facing signal. Added in #759 (2b4ace6c7). +So the "always-on cheap poll" baseline exists; the cost was already accepted at once-per-chunk cadence. + +## Evidence +- ddprof-lib/src/main/cpp/flightRecorder.cpp:2159-2161 comment: mallinfo2 walks every arena taking each lock; flush path only. +- ddprof-lib/src/main/cpp/flightRecorder.cpp:2178-2182 Counters::set(...) +- Gap: no arena COUNT, no per-arena distribution, no thread-count correlation, no attribution; built only against glibc>=2.33 headers. +- Possible doc inaccuracy to verify: comment at 2171 says `arena` = "bytes obtained from the OS via brk"; glibc int_mallinfo sums system_mem over ALL arenas (pending glibc-source check). + +## What this rules out +- Building a new mallinfo poller from scratch: the detector should extend this snapshot, not duplicate it. diff --git a/.knowledgebase/memories/find-glibc243-fastbins-removed.md b/.knowledgebase/memories/find-glibc243-fastbins-removed.md new file mode 100644 index 0000000000..51ea7bcfd9 --- /dev/null +++ b/.knowledgebase/memories/find-glibc243-fastbins-removed.md @@ -0,0 +1,35 @@ +--- +id: find-glibc243-fastbins-removed +title: glibc 2.43 removed fastbins from malloc_state: a fourth layout (next 2072, attached 2088, system_mem 2096, max 2104) +kind: finding +tags: [glibc, malloc_state, layout, calibration, glibc-2.43] +applies_to: [] +source: malloc-arena-detection/find-glibc243-fastbins-removed +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + + +# glibc 2.43 removed fastbins from malloc_state: a fourth layout (next 2072, attached 2088, system_mem 2096, max 2104) + +## Reasoning chain +The JVM baseline check used eclipse-temurin:21-jdk, which now ships Ubuntu GLIBC 2.43-2ubuntu2.4. The walk failed +with `implausible` under all three known layouts: system_mem read 0 on the seed arena itself. The detector disabled +itself safely. glibc-2.43 malloc/malloc.c:1722-1759: struct malloc_state = mutex, flags, top, last_remainder, +bins[254], binmap[4], next, next_free, attached_threads, system_mem, max_system_mem — no fastbinsY, no +have_fastchunks. glibc-2.41 and 2.42 still contain fastbinsY (raw source grep), so the change landed in 2.43. +heap_info unchanged (pagesize field, 48 bytes; arena.c:74-78) and `a = h->ar_ptr = (mstate)(h + 1)` still holds +(arena.c:639), so the structural check is unaffected. +Added candidate {2072, 2088, 2096, 2104}, preferred for >= 2.43. Reproducer on glibc 2.43 arm64 (T=64): layout=3, +walk_count 66 == malloc_info 66, all system_mem identical (max_diff 0). + +## Evidence +- build/arena-repro/arena_repro.c g_candidates[3]; JVM debug output `DEBUG cand=0..2 fail=implausible at=0 sys=0`. + +## What this rules out +- Treating the candidate list as closed: any glibc release can change malloc_state. The detector must fail + closed (no warning, walkStatus reason in the stats event), and new layouts need a candidate plus a + malloc_info-backed test per supported distro image. Strengthens the case for a CI check that runs the walk + against malloc_info on current distro images. diff --git a/.knowledgebase/memories/find-jvm-baseline.md b/.knowledgebase/memories/find-jvm-baseline.md new file mode 100644 index 0000000000..2a00030ce7 --- /dev/null +++ b/.knowledgebase/memories/find-jvm-baseline.md @@ -0,0 +1,40 @@ +--- +id: find-jvm-baseline +title: JVM baseline: arena count ~= thread count, so condition 1 never filters; on aarch64 glibc 2.43 the warnings are true positives (MALLOC_ARENA_MAX=8 cuts JVM RSS 44-60%) +kind: finding +tags: [jvm, validation, baseline, arena, rule] +applies_to: [] +source: malloc-arena-detection/find-jvm-baseline +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + + +# JVM baseline: arena count ~= thread count, so condition 1 never filters; on aarch64 glibc 2.43 the warnings are true positives (MALLOC_ARENA_MAX=8 cuts JVM RSS 44-60%) + +## Reasoning chain +Probe: build/arena-repro/jvm/arena_preload.c (LD_PRELOAD sampler reusing the walk; maxHeap passed via env), Workloads.java +(idle / http: HttpServer pool 16 + 16 clients / deflate: 64 threads using java.util.zip.Deflater), run_jvm.sh. +Matrix: Temurin 8/17/21 (Ubuntu GLIBC 2.43), arm64 native, container 512 MiB, -Xmx384m (headroom 128 MiB, P=25% -> 32 MiB), +quotas 1/2/4. 27 runs, walk ok in every sample (layout 3, 2.43). +- Arenas ~= threads: idle 14 (JDK21), 15 (JDK17), 34 (JDK8); http 49-69; deflate 78-98. Condition 1 + (arenas > 8 x quota) was true in every run except idle JDK17/21 at quota 2/4 -> condition 1 does not filter JVMs; + condition 2 decides. +- aarch64 committed (= free-held, = resident) arena bytes: idle 26-68 MiB, http 116-180, deflate 156-196. Rule fired on + all JDK8 runs and all http/deflate runs; silent on idle JDK17/21 (25-30 MiB < 32). +- Remedy test (quota 1): JDK8 idle default 34 arenas, arena RSS 66 MiB, VmRSS 87 MiB -> MALLOC_ARENA_MAX=8: 8 arenas, + arena RSS 14, VmRSS 35 (-60%). JDK21 http default 49 arenas, arena RSS 96, VmRSS 171 -> ARENA_MAX=8: arena RSS 14, + VmRSS 96 (-44%). Warnings flipped to 0 with the remedy -> the default-config warnings were true positives. +- x86_64 (qemu-emulated, system_mem only; smaps RSS not emulated): idle 2.4-5.5 MiB committed; http JDK21 22.8 MiB + (free 6.6), JDK8 46.7 MiB (free 26.5) -> committed criterion fires on JDK8 http, free-held criterion does not. + +## Evidence +- build/arena-repro/jvm/results/jdk*-q*-*.txt, summarize_jvm.py (worktree start-cairn-investigation, gitignored). + +## What this rules out +- Fear that the rule mostly flags healthy JVMs: on aarch64/2.43 the flagged memory is real RSS and reclaimable. +- Committed bytes as condition 2: it includes in-use bytes that fewer arenas do not reclaim -> use free-held + (mallinfo2 fordblks) when available; fall back to committed only without mallinfo2. +- Residual: x86_64 RSS not measured natively (emulator); x86_64 native check would close it. diff --git a/.knowledgebase/memories/find-liveheap-generations-equivalence.md b/.knowledgebase/memories/find-liveheap-generations-equivalence.md new file mode 100644 index 0000000000..7c7561a13f --- /dev/null +++ b/.knowledgebase/memories/find-liveheap-generations-equivalence.md @@ -0,0 +1,43 @@ +--- +id: find-liveheap-generations-equivalence +title: generations and liveheap drive the same liveness mechanisms; liveheap additionally preserves stack traces +kind: finding +tags: [liveness, generations, liveheap, semantics] +applies_to: [ddprof-lib/src/main/cpp/arguments.cpp, ddprof-lib/src/main/cpp/livenessTracker.cpp] +source: writing-any-code-thoroughly/find-liveheap-generations-equivalence +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + + +# generations and liveheap drive the same liveness mechanisms; liveheap additionally preserves stack traces + +## Reasoning chain +Operator-stated semantics, verified against source. `memory=b:l|L` +sets `_record_liveness` (+`_record_heap_usage` for 'L'); `generations=` sets +`_gc_generations`. Both flags feed `ObjectSampler::onSample -> +LivenessTracker::track()` (objectSampler.cpp:151). The leak-signal machinery — +per-klass population table (cleanup_table population accumulation, +livenessTracker.cpp:278/362), heap-floor ring (onGC, :2538), secondsToOOM() +(:1574), maybeForceCleanup() (:2482) — was gated on `_gc_generations` alone, +which is why liveheap-only deployments produced HeapLiveObject samples but +never filled the population table. + +Per the operator: EITHER option allows constructing the population-trend +signal; liveheap = generations + stacktrace preservation for live samples. +The documented intent "liveheap profiles imply generations" did not exist in +the native parser before this fix. + +## Evidence +- arguments.cpp:200-260 (memory option), 282-291 (generations option) +- livenessTracker.cpp:2096-2104 (initialize: both flags read) +- Verified the agent side (dd-trace-java@jb/reference_chains e188d0ff7f + + 158078255f) forces `generations=true` when reference chains are enabled — + but the deployed runtime recorded gcGenerations=false anyway. + +## What this rules out +- "Missing generations flag is only an agent wiring problem" — the native + parser had no coupling either; the profiler must treat either source as + sufficient. diff --git a/.knowledgebase/memories/find-refchain-cacheleak-inorder-flake.md b/.knowledgebase/memories/find-refchain-cacheleak-inorder-flake.md new file mode 100644 index 0000000000..f65a2d2b08 --- /dev/null +++ b/.knowledgebase/memories/find-refchain-cacheleak-inorder-flake.md @@ -0,0 +1,30 @@ +--- +id: find-refchain-cacheleak-inorder-flake +title: shouldReconstructReferrerChainThroughUnboundedCacheLeak flakes in-order, pre-existing +kind: finding +tags: [flaky-test, e2e, pr-921, pre-existing] +applies_to: [ddprof-test/src/test/java/com/datadoghq/profiler/referencechains/ReferenceChainTrackingTest.java] +source: writing-any-code-thoroughly/find-refchain-cacheleak-inorder-flake +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + + +# shouldReconstructReferrerChainThroughUnboundedCacheLeak flakes in-order, pre-existing + +Verified by stash-bisection (2026-10-29): the in-order failure ("Never observed a +datadog.ReferenceChain event whose chain[0] is CachedPayload after N cached entries +across up to 16 rounds") reproduces on the COMMITTED tree (aa527d27d) as well as on +the delta-fix tree - committed: 1 pass / 1 fail; delta: 3 fail / 1 isolated pass. +Passes reliably with -Ptests=...shouldReconstructReferrerChainThroughUnboundedCacheLeak +(single method, fresh JVM). The sphinx verify-v1 failure and the fix child's +repeated-run loop were chasing this flake, NOT a delta regression. + +Tracked as PROF-16383 (filed 2026-10-29, assigned to the operator); linked from PR #921 via a CI-triage comment. Fix candidate noted in the ticket: force System.gc() per growth round + failure-message diagnostics. + +Shape: heavy E2E leak scenario, shared no-forkEvery JVM, self-adjusting 16-round +budget, per-test stop()/start() (@AfterEach), population/hook state from the @Order(1) +GC-root test. Sensitive to machine load and in-class ordering. NOT addressed in this +PR (out of scope; separate follow-up if wanted). diff --git a/.knowledgebase/memories/find-repro-arm64-results.md b/.knowledgebase/memories/find-repro-arm64-results.md new file mode 100644 index 0000000000..9c1061aeb1 --- /dev/null +++ b/.knowledgebase/memories/find-repro-arm64-results.md @@ -0,0 +1,43 @@ +--- +id: find-repro-arm64-results +title: Reproducer (arm64, glibc 2.17-el7/2.28/2.39): walk == malloc_info exactly; arenas = min(threads+2, 8*online CPUs) even under a 2-CPU quota; freed memory stays held +kind: finding +tags: [reproducer, arena, measurement, arm64] +applies_to: [] +source: malloc-arena-detection/find-repro-arm64-results +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + + +# Reproducer (arm64, glibc 2.17-el7/2.28/2.39): walk == malloc_info exactly; arenas = min(threads+2, 8*online CPUs) even under a 2-CPU quota; freed memory stays held + +## Reasoning chain +Workload: T threads, 8 MiB each of 16..1024 B mallocs, free all but every 8th, barrier, probe ("retained"), +free rest, join, probe ("threads-exited"). Probe thread owns the seed arena (adds 1 arena; main adds 1). +- Walk: ok on all 72 results; count and per-arena system_mem identical to malloc_info (N/N, max_diff 0); + sum_sys == mallinfo2.arena on 2.39. +- Default limit 128 (16 online CPUs). quota2_default (podman --cpus=2, cpu.max 200000/100000) still 128 arenas + -> glibc ignores the CPU quota (empirical confirmation of find-arena-limit-ignores-cgroup). +- Retention after all memory freed and all threads exited (sum_sys MiB / RSS MiB), T=256: + default ~1152-1186 / ~1169-1204 on all three; MALLOC_ARENA_MAX=2: 95-299 / 113-316; + quota2 + MALLOC_ARENA_MAX=16 (8 x quota): 132-313 / 149-330. In-use after exit ~0 (2.39 mi2). +- Default footprint scales linearly: T=4 33 MiB, 16 132, 64 528, 256 ~1170 (capped by 128 arenas). +- Per-arena max_sys ~8-16 MiB default vs 100-600 MiB with 2 arenas (shared); max_attached 1 default + (thread-private) vs 33-155 when shared (during retained phase). +- In the "retained" phase (1/8 live, pinned) shared arenas do NOT help much (e.g. 2.39 T=256: 1224 vs 1083 MiB): + the win is reuse after frees, not fragmentation. + +## Evidence +- build/arena-repro/results/*-arm64.txt (worktree start-cairn-investigation, gitignored); summarize.py. +- Podman gotcha: remote client queries all credHelpers (ddtool) even for docker.io; run with + DOCKER_CONFIG/REGISTRY_AUTH_FILE pointing at an empty {} config; use fully qualified image names. + +## What this rules out +- Doubt that the walk reads the same values glibc reports (exact match). +- amd64 (qemu-emulated) matrix: walk ok in 72/72; malloc_info match N/N in 71/72. One OL7 quota2 threads-exited + run had 127/128 (one arena differed 8.1 MiB); 3 reruns matched 128/128 -> transient race between the two + non-atomic reads (malloc_info first, walk second), not reproduced. Same retention pattern: T=256 default + 872-1047 MiB sum_sys vs quota2+ARENA_MAX=16 107-193 MiB. RSS under qemu is inflated by the emulator and not used. diff --git a/.knowledgebase/memories/find-tls-pool-exhaustion.md b/.knowledgebase/memories/find-tls-pool-exhaustion.md new file mode 100644 index 0000000000..c4df5ac871 --- /dev/null +++ b/.knowledgebase/memories/find-tls-pool-exhaustion.md @@ -0,0 +1,68 @@ +--- +id: find-tls-pool-exhaustion +title: 64-slot ThreadLocalDataPool exhausted by unregistered threads: 6.85M dropped samples in staging prof-analyzer +kind: finding +tags: [malloc-tracer, threadlocaldatapool, nativemem, dropped-samples, staging] +applies_to: [ddprof-lib/src/main/cpp/mallocTracer.cpp, ddprof-lib/src/main/cpp/threadLocalDataPool.h] +source: writing-any-code-thoroughly/find-tls-pool-exhaustion +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + + +# 64-slot ThreadLocalDataPool exhausted by unregistered threads: 6.85M dropped samples in staging prof-analyzer + +## Reasoning chain +Both `thread_local_pool_exhausted` and `samples_dropped_thread_local` equal +6,853,922 in rec1 because they increment IN PAIRS: every failed +`ProfiledThread::acquireCurrent()` bumps the pool counter (in claim()) and the +caller's drop counter. + +Mechanism: +- ThreadLocalDataPool has a hard 64 slots (threadLocalDataPool.h:9-13), sized + for "JVM internal threads that start before the pthread_create interceptor + is initialized - not many, 64 slots should be plenty". +- MallocTracer::maybeRecord() calls acquireCurrent() on EVERY intercepted + malloc/free BEFORE the sampling gate (mallocTracer.cpp:311-322) - placement + is deliberate (first-touch priming comment), but with nativemem on + (modeMask 27) the pool becomes the acquisition fallback for EVERY + unregistered thread. +- prof-analyzer is a Rust+JVM hybrid: 471 peak / 2,561 accumulated JVM threads + (jdk.JavaThreadStatistics) plus tokio/grpc-C-core/zstd natives. Once >64 + concurrently-live unregistered threads exist, every malloc on every other + thread fails forever: SignalBlocker + failed claim + 2 counter increments + (~6,700 drop-pairs/s over the ~17 min recording). +- Impact: nativemem (and CPU/wall) coverage silently vanishes for those + threads; only the counters show it. + +Operator decision (2026-10-28): do NOT patch now; file a Jira ticket and +investigate whether a gate reorder interferes with the planned +sampler-per-thread design. Filed PROF-16366 +(https://datadoghq.atlassian.net/browse/PROF-16366). + +## Refined reorder design (would-be patch shape, NOT implemented) +Naively moving acquireCurrent() after shouldSample() SKEWS SAMPLING: the +reentrant hook call (profiler-internal malloc inside recordMalloc) would +decrement the global poisson counter before discovering the CS is held. The +safe shape is two paths: +- TLS present: ProfiledThread::current() (never claims) -> CriticalSection + try-enter -> shouldSample -> recordMalloc. Reentrant call: CS try fails -> + skip BEFORE touching the counter. Registered threads keep the cheap TLS hit. +- No TLS (pre-profiler/foreign thread): shouldSample FIRST; only on an actual + sample, acquireCurrent() (pool claim at sample rate ~7/s) -> CS -> record. +Effect: the pool stops being a per-malloc client; claims happen at sample rate +and in signal-handler priming only - i.e. pre-profiler threads, per the +operator's rule. + +## Evidence +- Feasibility analysis (recorded in PROF-16366): + shouldSample() is thread-independent (global poisson counter, + mallocTracer.cpp:286-300). +- Pool capacity math: ProfiledThread is a few hundred bytes; 64 -> 512 slots + is ~0.25 MB. + +## What this rules out +- "The drops are reference-chain related" - independent subsystem; the counts + are identical across both recordings and unrelated to BFS activity. diff --git a/.knowledgebase/memories/find-urgent-path-candidate-less-search.md b/.knowledgebase/memories/find-urgent-path-candidate-less-search.md new file mode 100644 index 0000000000..bf21644d16 --- /dev/null +++ b/.knowledgebase/memories/find-urgent-path-candidate-less-search.md @@ -0,0 +1,60 @@ +--- +id: find-urgent-path-candidate-less-search +title: Second root cause: urgent-OOM latch authorized a candidate-less search (exposed by generations=true experiment) +kind: finding +tags: [reference-chains, urgent-oom, warmup, second-root-cause] +applies_to: [ddprof-lib/src/main/cpp/referenceChains.cpp, ddprof-lib/src/main/cpp/livenessTracker.cpp] +source: writing-any-code-thoroughly/find-urgent-path-candidate-less-search +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + + +# Second root cause: urgent-OOM latch authorized a candidate-less search (exposed by generations=true experiment) + +## Reasoning chain +Operator added `generations=true` to staging args; rec3 +(~/Downloads/20261029-085919_prof-analyzer_AaGG44LTAABrEm9tdOyoPAAA/main.jfr, +agent 1.69.0-SNAPSHOT~2a7b8e49bb, gcGenerations=true, fresh watermark 262144/16.8MB +=> process restarted) still showed the collector churning. Discriminator: +`liveness_urgent_boost_admits = 2495` (0 in rec1/rec2) -> the urgent-OOM path, +not the short-circuit, authorized the search. + +Mechanism (verified at v_1.52.1 tag - deployed binary contains the gate): +1. Fresh process -> heap floor rises during warmup -> secondsToOOM()'s linear + extrapolation reads < 5 min -> isUrgent() latches -> one search entitled + with NO candidate requirement (old hasLeakSignal urgent branch). +2. The search cannot exit early: completion requires a pass with no truncated + work (referenceChainTraversal.cpp:1113 - every pass truncates on this heap), + and the 30-pass no-progress abandonment is suppressed while urgent (:1140 + `!isUrgent()`). +3. Result: candidate-less whole-heap BFS to the auto-tuned TTL (up to 30 min); + zero chains (nothing tagged), zero abandonments (RECORDING_END emits none). + +## Fix (A+B, implemented 2026-10-29, uncommitted, same worktree change set) +- A: hasLeakSignal() returns selectLeakCandidates(probe,1) > 0 only - the + urgent latch no longer entitles searches. _urgent_search_spent removed + entirely (behaviorally dead). Urgency still drives the ramp (cadence, pause + target, budget boost, 100% liveness admission) and suppresses the + no-progress abandonment for legitimate candidate-driven searches. +- B: warmup guard in isUrgent() - a projection in the urgent band whose + observation window (ring time span, new secondsToOOM(out_window) out-param) + is shorter than OOM_URGENT_THRESHOLD_S (300s) neither latches nor touches + the release counter. The 30-min ramp still sees the raw projection. +- Tests: WarmupShortWindowDoesNotLatchUrgency (new, B); + UrgentLatchWithoutCandidateDoesNotStartSearch (rewrites + UrgentOOMProjectionBypassesCandidateGate, which asserted the removed + bypass); OomUrgencyTest project() reseated to 15s sample spacing (window + 435s >= 300s) and BFS CanaryStuck seeding to 60s spacing - short-window + seedings would silently never latch under B. +- Docs: ReferenceChainsSignalsExplained.md (search gate = candidate only; + warmup guard documented). + +Verification: full :ddprof-lib:gtestDebug 1017 OK / 0 failed (2026-10-29). + +## What this rules out +- "The fix from find-fix-leak-signal-gate was wrong" - it was correct for the + gcGenerations=false signature; rec3 simply exercised the second, independent + authorization path. diff --git a/.knowledgebase/memories/glibc-243-aarch64-arena-thp-floor.md b/.knowledgebase/memories/glibc-243-aarch64-arena-thp-floor.md new file mode 100644 index 0000000000..4db9b210de --- /dev/null +++ b/.knowledgebase/memories/glibc-243-aarch64-arena-thp-floor.md @@ -0,0 +1,21 @@ +--- +id: glibc-243-aarch64-arena-thp-floor +title: glibc 2.43 on aarch64 sizes every arena heap in 2 MiB THP units; with THP madvise each arena costs at least 2 MiB RSS +kind: finding +tags: [glibc-2.43, aarch64, thp, arena, rss, jvm] +applies_to: [] +source: malloc-arena-detection/find-aarch64-243-thp-floor +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + +glibc 2.43 defines `DEFAULT_THP_PAGESIZE` = 2 MiB for aarch64 (`sysdeps/unix/sysv/linux/aarch64/malloc-hugepages.h:19`) +and sets `thp_pagesize` "even if thp_mode is never" (`malloc.c:1912-1915`); `new_heap` sizes heaps in that unit +(`arena.c:446`). With kernel THP mode `madvise`, glibc advises huge pages, so each arena is at least 2 MiB resident +even when a thread uses a few KiB. x86_64 and generic builds default to 0; glibc 2.42 lacks this code. + +Measured in JVMs (Temurin 8/17/21 on Ubuntu glibc 2.43, aarch64): every non-main arena = 2048 KiB committed and +resident; idle JVMs held 26-68 MiB in arenas with < 4 MiB in use. `MALLOC_ARENA_MAX=8` cut JVM RSS from 87 to +35 MiB (idle JDK 8) and from 171 to 96 MiB (JDK 21 under HTTP load). diff --git a/.knowledgebase/memories/glibc-arena-limit-ignores-cpu-quota.md b/.knowledgebase/memories/glibc-arena-limit-ignores-cpu-quota.md new file mode 100644 index 0000000000..e4e84c8479 --- /dev/null +++ b/.knowledgebase/memories/glibc-arena-limit-ignores-cpu-quota.md @@ -0,0 +1,23 @@ +--- +id: glibc-arena-limit-ignores-cpu-quota +title: glibc's malloc arena limit is 8 x online CPUs and ignores the container CPU quota +kind: finding +tags: [glibc, malloc, arena, container, cgroup] +applies_to: [] +source: malloc-arena-detection/find-arena-limit-ignores-cgroup +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + +glibc gives each thread its own malloc arena until the arena limit is reached. On 64-bit the limit is +8 x `__get_nprocs()`, fixed once after the first 8 arenas exist (glibc 2.17 `malloc/arena.c:848`, 2.28 `:888`, +2.39 `:834`). `__get_nprocs()` reads the online CPUs (`/sys/devices/system/cpu/online`), so it ignores the cgroup +CPU quota. glibc 2.35-2.38 used the affinity mask instead (cpuset, still not the quota); 2.39 reverted that. + +Consequence: a container with a 2-CPU quota on a 16-CPU host keeps a limit of 128 arenas; measured, 256 threads +created 128 arenas under `--cpus=2`. glibc never frees arenas; an exited thread's arena goes to a reuse list. + +Override: `MALLOC_ARENA_MAX=N` or `GLIBC_TUNABLES=glibc.malloc.arena_max=N`. With N = 8 x quota CPUs, a +256-thread small-allocation workload retained 4-9x less memory than with the default limit. diff --git a/.knowledgebase/memories/glibc-arena-list-walk.md b/.knowledgebase/memories/glibc-arena-list-walk.md new file mode 100644 index 0000000000..242d9173e0 --- /dev/null +++ b/.knowledgebase/memories/glibc-arena-list-walk.md @@ -0,0 +1,24 @@ +--- +id: glibc-arena-list-walk +title: The glibc arena list can be walked lock-free from any arena and self-validated with the (mstate)(heap_info + 1) placement invariant +kind: finding +tags: [glibc, malloc_state, arena, lock-free] +applies_to: [] +source: malloc-arena-detection/find-arena-walk-safe +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + +- `malloc_state->next` forms a circular list through `main_arena`. New arenas are inserted under `list_lock` + and published after `atomic_write_barrier` (2.39 `arena.c:669-676`); arena structs are never freed. A walk + with acquire loads is memory-safe; field values are racy snapshots. +- Every non-main arena is placed right after its first `heap_info`: `a = h->ar_ptr = (mstate)(h + 1)` + (2.17 `arena.c:730`, 2.28 `:703`, 2.39 `:649`, 2.43 `:639`). So `(a & (HEAP_MAX_SIZE-1)) == sizeof(heap_info)` + (32 before 2.35, 48 from 2.35) and `heap_info->ar_ptr == a`. Exactly one node fails this: `main_arena`. +- Per-arena `system_mem` (committed span, updated on every grow/shrink) and `attached_threads` (2.23+) read + this way matched `malloc_info()` exactly on glibc 2.17 (EL7), 2.28, 2.39 and 2.43, aarch64 and x86_64. +- Seed: any arena pointer works (list is circular); a thread's own malloc usually yields a non-main arena, but + past the arena limit `reused_arena` can hand out `main_arena`. +- Read through `SafeAccess`; recovery exists only on HotSpot and OpenJ9, so do not run such a walk on Zing. diff --git a/.knowledgebase/memories/glibc-arena-retention.md b/.knowledgebase/memories/glibc-arena-retention.md new file mode 100644 index 0000000000..7a56cff785 --- /dev/null +++ b/.knowledgebase/memories/glibc-arena-retention.md @@ -0,0 +1,24 @@ +--- +id: glibc-arena-retention +title: glibc arenas keep freed memory: interior free chunks are never released and the heap top shrinks only above trim_threshold +kind: finding +tags: [glibc, malloc, arena, fragmentation, mmap-threshold] +applies_to: [] +source: malloc-arena-detection/find-mmap-threshold-drift +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + +- Each non-main arena reserves 64 MiB heaps (`HEAP_MAX_SIZE` = 2 x `DEFAULT_MMAP_THRESHOLD_MAX` on 64-bit). The + committed part grows to the arena's peak. +- `free()` shrinks a non-main heap only from the top and only when the top chunk exceeds `trim_threshold` + (2.39 `arena.c:545-606`); interior free chunks stay committed. `malloc_trim(0)` madvises interior free pages + of every arena, but shrinks the top only for `main_arena` (2.39 `malloc.c:5198`). +- Dynamic mmap threshold: freeing an mmapped chunk larger than `mmap_threshold` (up to 32 MiB) raises + `mmap_threshold` to that size and `trim_threshold` to 2x (2.39 `__libc_free`, `malloc.c:3379-3384`). Later + allocations up to 32 MiB then land in arenas, and with `trim_threshold` at 64 MiB a heap top effectively + never shrinks. Any explicit `mallopt`/tunable for these thresholds disables the dynamic behaviour. +- Result: retained arena memory ~= number of arenas x per-arena peak, even after all memory is freed and the + threads have exited (measured: ~1.2 GiB held, ~0 in use, 256 threads). diff --git a/.knowledgebase/memories/glibc-mallinfo-cost.md b/.knowledgebase/memories/glibc-mallinfo-cost.md new file mode 100644 index 0000000000..0b28355236 --- /dev/null +++ b/.knowledgebase/memories/glibc-mallinfo-cost.md @@ -0,0 +1,21 @@ +--- +id: glibc-mallinfo-cost +title: mallinfo2() and malloc_info() lock each arena and walk all its free chunks; use them at JFR-chunk cadence only +kind: finding +tags: [glibc, mallinfo, cost, locking] +applies_to: [ddprof-lib/src/main/cpp/flightRecorder.cpp] +source: malloc-arena-detection/find-mallinfo-cost +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + +`int_mallinfo` walks every fastbin and regular-bin chunk of an arena (2.39 `malloc.c:5261-5300`), so the cost +is O(free chunks). `__libc_mallinfo2` visits arenas one after another and holds each arena's mutex for that +arena's whole walk; allocating threads of that arena stall meanwhile. `malloc_info()` does the same walk plus the +heap chain and writes XML through stdio. `mallinfo2` exists since glibc 2.33. Neither gives an arena count. + +Field semantics (easy to get wrong): `arena` = sum of `system_mem` over ALL arenas (not brk only); `keepcost` = +top chunk of `main_arena` only; `fordblks` = free bytes held in all arenas (including the unused top space). +Acceptable once per JFR chunk on the flush path; never from a signal handler or at higher frequency. diff --git a/.knowledgebase/memories/glibc-malloc-state-layout-calibration.md b/.knowledgebase/memories/glibc-malloc-state-layout-calibration.md new file mode 100644 index 0000000000..43f7633d8e --- /dev/null +++ b/.knowledgebase/memories/glibc-malloc-state-layout-calibration.md @@ -0,0 +1,30 @@ +--- +id: glibc-malloc-state-layout-calibration +title: malloc_state field offsets differ by glibc version, distro build and architecture; calibrate them at runtime +kind: finding +tags: [glibc, malloc_state, layout, backport, calibration] +applies_to: [] +source: malloc-arena-detection/find-el7-layout-backport +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + +Known layouts (x86_64 == aarch64 for upstream; offsets in bytes): + +| Layout | next | attached_threads | system_mem | max_system_mem | +|---|---|---|---|---| +| upstream 2.17-2.22 | 2152 | - | 2168 | 2176 | +| upstream 2.23-2.26 | 2152 | 2168 | 2176 | 2184 | +| upstream 2.27-2.42 | 2160 | 2176 | 2184 | 2192 | +| upstream 2.43+ (fastbins removed) | 2072 | 2088 | 2096 | 2104 | + +Version alone is not enough: Oracle Linux 7 glibc 2.17 uses the 2.27-2.42 layout on aarch64 +(`2.17-326.0.6.el7_9.3`) and the 2.23-2.26 layout on x86_64 (`2.17-326.0.9.el7_9.3`) because of vendor backports. +glibc 2.43 removed `fastbinsY` and `have_fastchunks`, shifting everything by 88 bytes. + +Calibrate: try each candidate (version-preferred first) and accept the first whose full walk passes the +structural check (see glibc-arena-list-walk), returns to the seed, and yields plausible values (`system_mem` +non-zero and page-aligned, `max_system_mem >= system_mem`). Fail closed when none passes. New glibc releases can +add layouts, so test against `malloc_info()` on current distro images. diff --git a/.knowledgebase/memories/glibc-pointer-to-arena.md b/.knowledgebase/memories/glibc-pointer-to-arena.md new file mode 100644 index 0000000000..7e5e8ee5ca --- /dev/null +++ b/.knowledgebase/memories/glibc-pointer-to-arena.md @@ -0,0 +1,18 @@ +--- +id: glibc-pointer-to-arena +title: A pointer returned by glibc malloc maps to its arena without locks: check IS_MMAPPED, then NON_MAIN_ARENA, then read heap_info->ar_ptr +kind: finding +tags: [glibc, malloc, arena, attribution] +applies_to: [] +source: malloc-arena-detection/find-ptr-to-arena +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + +The size word at `p - 8` carries `IS_MMAPPED` (0x2) and `NON_MAIN_ARENA` (0x4). Check `IS_MMAPPED` first: +mmapped chunks carry no arena bit. For a non-main chunk, `heap_info` sits at `p & ~(HEAP_MAX_SIZE - 1)` and its +first field `ar_ptr` is the arena (true in 2.17 through 2.43). `HEAP_MAX_SIZE` is 64 MiB on 64-bit, but +`glibc.malloc.hugetlb >= 2` (2.35+) changes it to 4 x the huge page size. Strip the MTE tag on aarch64. +Main-arena chunks give no address (`main_arena` and `narenas` are static, not exported). diff --git a/.knowledgebase/memories/jvm-threads-get-private-arenas.md b/.knowledgebase/memories/jvm-threads-get-private-arenas.md new file mode 100644 index 0000000000..6403bdf6fa --- /dev/null +++ b/.knowledgebase/memories/jvm-threads-get-private-arenas.md @@ -0,0 +1,22 @@ +--- +id: jvm-threads-get-private-arenas +title: In a JVM the glibc arena count is about the thread count, so 'arenas > 8 x quota CPUs' is true for almost every small-container JVM +kind: finding +tags: [jvm, glibc, arena, container] +applies_to: [] +source: malloc-arena-detection/find-jvm-baseline +status: stale +recorded: 2026-11-02 +updated: 2026-11-02 +valid_at: 2026-11-02 +invalid_at: 2026-11-02 +retired_reason: "marked stale" +--- + + + +JVM-internal threads (GC workers, JIT compilers, runtime threads) and application pools each get their own arena +until glibc's limit. Measured (Temurin 8/17/21): idle 14-34 arenas, HTTP server with clients 49-69, 64 +`Deflater` threads 78-98; nearly all with `attached_threads == 1`. Any check of the form +`arena_count > 8 x ceil(quota CPUs)` therefore holds for almost every JVM at quotas of 1-4 CPUs; it indicates that +`MALLOC_ARENA_MAX` would take effect, not that memory is wasted. Judge the waste by bytes (free-held), not count. diff --git a/.knowledgebase/memories/jvmti-class-tags-negative-shared-env.md b/.knowledgebase/memories/jvmti-class-tags-negative-shared-env.md new file mode 100644 index 0000000000..e6e7222065 --- /dev/null +++ b/.knowledgebase/memories/jvmti-class-tags-negative-shared-env.md @@ -0,0 +1,30 @@ +--- +id: jvmti-class-tags-negative-shared-env +title: JVMTI object tags are shared per env; class-identity tags are negative and a positive tag on a class is never a class tag +kind: finding +tags: [jvmti, tags, refchains, liveness] +applies_to: [ddprof-lib/src/main/cpp/classTagAllocator.h, ddprof-lib/src/main/cpp/referenceChainWalk.cpp] +source: feedback-1/find-c2-settag-779-conflict +status: stale +recorded: 2026-11-02 +updated: 2026-11-02 +valid_at: 2026-11-02 +invalid_at: 2026-11-02 +retired_reason: "marked stale" +--- + + + +JVMTI tags are per `jvmtiEnv`, and everything that uses `VM::jvmti()` shares one tag per object. +The reference-chain tracker and LivenessTracker use the tag on a class object as its identity and +mint it from `ClassTagAllocator`, which hands out strictly negative values. Positive tags on class +objects occur legitimately: `heapReferenceCallback()` admits a not-yet-class-tagged class object +reached over a non-CLASS edge as an ordinary frontier object (frontier tags are positive). Any +other feature that stores data in a class tag through the same env collides with both. + +Rules: +- Never adopt a positive tag as a class tag. `ClassTagAllocator::getOrMint()` returns 0 for it, and + `resolveLoadedClasses()` neither adopts nor re-resolves positive tags. +- A feature that needs its own per-class tag data should obtain a separate `jvmtiEnv` (another + `GetEnv` call) rather than share `VM::jvmti()`. Masking bits in a shared tag does not work: + `GetObjectsWithTags` matches exact values and JVMTI has no tag compare-and-set. diff --git a/.knowledgebase/memories/mallinfo2-compiled-out-in-release.md b/.knowledgebase/memories/mallinfo2-compiled-out-in-release.md new file mode 100644 index 0000000000..3d298151f5 --- /dev/null +++ b/.knowledgebase/memories/mallinfo2-compiled-out-in-release.md @@ -0,0 +1,23 @@ +--- +id: mallinfo2-compiled-out-in-release +title: The mallinfo2() arena counters are compiled out of shipped glibc binaries, because the release build uses glibc 2.28 headers +kind: finding +tags: [mallinfo, build, release, glibc, counters] +applies_to: [ddprof-lib/src/main/cpp/flightRecorder.cpp, .gitlab/base/el8/Dockerfile, .gitlab/scripts/rebuild-images.sh] +source: malloc-arena-detection/find-mallinfo-snapshot-dead-in-release +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + +`DD_HAVE_MALLINFO2` is defined only when the glibc headers at COMPILE time are 2.33+ +(`flightRecorder.cpp`, top of file), and `Recording::updateMallocArenaStats()` is empty without it. The shipped +glibc libraries are built in the AlmaLinux 8 image (glibc 2.28; `rebuild-images.sh` maps +`BUILD_IMAGE_{X64,ARM64}_GLIBC` to `.gitlab/base/el8/Dockerfile`), so the `malloc_*` counters are always 0 in +production. Verified on ddprof 1.52.1: the counter name is present, no `mallinfo2` import. +`mallocArenaStats_ut.cpp` is behind the same macro and runs on newer-glibc test images, so CI never exercises the +shipped binary. + +Rule: never gate a runtime libc feature on `__GLIBC_MINOR__`; resolve it at runtime (`dlsym`). Tracked in +PROF-16350. diff --git a/.knowledgebase/memories/n-002-root-cause-terminal-release.md b/.knowledgebase/memories/n-002-root-cause-terminal-release.md new file mode 100644 index 0000000000..64e5b07677 --- /dev/null +++ b/.knowledgebase/memories/n-002-root-cause-terminal-release.md @@ -0,0 +1,78 @@ +--- +id: n-002-root-cause-terminal-release +title: n-002: ROOT CAUSE — terminal COMPLETED releases tags; restart storm makes EXPANDED unobservable +kind: finding +tags: [reference-chains, root-cause, jvmti] +applies_to: [] +source: frontier-wait-untagged/n-002-root-cause-terminal-release +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + +# n-002: ROOT CAUSE — terminal COMPLETED releases tags; restart storm makes EXPANDED unobservable + +- status: confirmed +- evidence: build/logs/20261014-174441 (instrumented child section /tmp/child-section2.log), OpenJDK jvmtiTagMap.cpp + +## Chain (each step verified) + +1. The scenario wires no representative → `_watched_leak_klass_count == 0`. +2. Each pass: static sweep admits LEAK_BUFFER's SynchronizedList (verified: + `sweep_static_edge class_tag=-8 referee_tag=0 referee_klass=34` once per + walk — 36 = 4 statics x 9 laps), expandFrontier expands it, nothing pending + → `!has_pending_frontier && _watched_leak_klass_count == 0` → + **SearchState::COMPLETED** (referenceChainTraversal.cpp:1110). +3. The terminal pass ITSELF releases every frontier tag: + `if (_search_state != RUNNING) { _tags_released = releaseSearchTags(...); }` + (referenceChainTraversal.cpp:1192, and the terminal no-op branch at 1008). +4. `shouldRunPass()` sees terminal, `canAffordNewSearch()` cools the safepoint + pain budget (~1.5-83ms) then `restartSearch()` — identical walk repeats + (edges_admitted=19058 twice); `_watched_leak_klass_count==0` again → + COMPLETED → release. Storm of 9+ identical searches in the 45s window. +5. The scenario's poll (`getReferenceChainFrontierStateForTest0`) takes + `_engine_lock` — it can only read BETWEEN passes (blocked mid-pass), and + between passes the tags are released → `GetTag(holder)=0` → state -1. + The 45s wait can NEVER observe EXPANDED in this regime. + +## Why intermittent + +Pass/fail depends on whether any pass stays RUNNING long enough (e.g. a +truncated pass → `has_pending_frontier` true → no COMPLETED → tags persist → +poll observes EXPANDED). The one observed passing run (--no-daemon, +build/logs/20261014-164358, `[chain-found] byte[]`) had a different pass +history. Machine load/scheduling decides. + +## Instrumentation notes (reverted) + +- Temporary TEST_LOG lines were added to referenceChainWalk.cpp / + referenceChainTraversal.cpp and reverted via git checkout (worktree clean). +- JVMTI facts verified from OpenJDK master jvmtiTagMap.cpp: + `iterate_over_class` reports static fields unconditionally once the class + mirror is visited; the callback's return gates the visit + (`res & JVMTI_VISIT_OBJECTS (0x100) → check_for_visit(referree)`; + returning 0 prunes). jvmtiHeapReferenceKind: CLASS_LOADER=4, + PROTECTION_DOMAIN=6, INTERFACE=7, STATIC_FIELD=8, CONSTANT_POOL=9, + SUPERCLASS=10. +- Statics of the scenario class ARE reported by FollowReferences every lap + (LEAK_BUFFER edge included) — the native sweep design works; the failure is + the observe-window race, not a missing admission. + +## Pre-bug state + +The EXPANDED-wait + seam machinery arrived with the branch's review-fix +commits (a28220dc1/fa035dfa4/13bbd3d65/7414cd169/a849954f8). Bisection: 2/2 +failures at a849954f8 (pre-address commit) → NOT introduced by the address +commit 6855e4b5e. + +## Fix directions (decision open) + +A. Test-side: wire a representative (setKlassPopulationRepresentativeForTest0) + like LeakingCacheScenario → `_watched_leak_klass_count > 0` → search stays + RUNNING → tags persist → wait observes EXPANDED. Changes the tracker mode + the scenario runs in (canary/rotation behavior). +B. Test-side: replace the EXPANDED wait with a restart-storm-tolerant signal + (pass-counter deltas + rotation seam), accepting tags are ephemeral. +C. Product-side: keep tags on terminal-COMPLETED (release only on + ABANDONED/restart) — changes product semantics, needs justification. diff --git a/.knowledgebase/memories/n-005-rep-drift-and-record-loss.md b/.knowledgebase/memories/n-005-rep-drift-and-record-loss.md new file mode 100644 index 0000000000..ad31a21bb7 --- /dev/null +++ b/.knowledgebase/memories/n-005-rep-drift-and-record-loss.md @@ -0,0 +1,94 @@ +--- +id: n-005-rep-drift-and-record-loss +title: n-005: two residual flake mechanisms after the seam fallback; fresh-chunk-per-attempt fix +kind: finding +tags: [reference-chains, flake, chunk] +applies_to: [] +source: frontier-wait-untagged/n-005-rep-drift-and-record-loss +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + +# n-005: two residual flake mechanisms after the seam fallback; fresh-chunk-per-attempt fix + +## Status: FIXED - batch F 8/8 PASS; batch G confirmation + commit pending + +## Mechanism 1 (B5): representative drift +klassPopulationSetRepresentativeForTest REPLACES reps with [lateChunk] per round call, but after +the rounds end, real allocation sampling re-populates the byte[] population entry's reps +(rep_count 1 -> 3 observed post-rounds, non-lateChunk at head). resolveCandidateRepresentative +then returns ANOTHER byte[]: the poll leak-tags/intercepts/builds THAT object's chain (entry +22365, leak 1073742079) while lateChunk - the scenario's actual seam target - holds an unbuilt +plain-admission entry (22363, cached=0, no leak tag). Seam: 0 forever. +Fix applied: tail loop re-pins the representative (later superseded by mechanism-2 fix). + +## Mechanism 2 (C8): allocation-tracking record loss [THE DEEPER ONE] +With the rep re-pinned correctly (rep_count=1, rep == lateChunk), lateChunk STILL reads tag=0 +at 2134 consecutive polls: it never acquired a leak tag. tagLeakInstances tagged=19 (klass, +main-tid) with need_set=0 - the 19 tagged objects are others. The leak-tag pipeline keys on +the chunk's allocation-tracking record (records minted ONLY at allocation); table churn evicts +an old chunk's record, and once gone, that object can never be leak-tagged/intercepted/ +chain-built again. Whether lateChunk's record survives the whole run is the remaining lottery. + +## Fix (batch D, running): fresh-chunk-per-tail-attempt +Each tail attempt appends a NEW chunk to LEAK_BUFFER (still appended long after the collection +was EXPANDED - the scenario's invariant holds) and watches IT: fresh allocation => guaranteed- +live record => explicit poll tags it immediately => next anchor descend intercepts => +later attempt's poll builds its chain (rep path, not the 8-slot discovered path) => seam returns +its tag => dump => JFR match. LEAK_BUFFER_CAPACITY raised to cover tail chunks (elementData +must never reallocate). Also keeps the no-local discipline (chunk reachable only via the static +field). + +## Seam fallback (kept, from n-004): leak-tag branch in resolvedChainTargetTagForTest scans +_resolved_chains by event target tag - covers the leak-tag-on-object state across generations. +INVESTIGATE trace lines still in referenceChainEvents.cpp - MUST BE REMOVED before commit. + +## Verified remote state: ~/go/src/github.com/DataDog/java-profiler at 6855e4b5e + uncommitted: +referenceChainEvents.cpp (fallback+traces), referenceChains.h (contract comment), +StaticFieldGrowingCollectionScenario.java (v2 rep wiring + stack-local removal + fresh-chunk tail). +Debug level via /tmp/ddprof_root/refchains_debug_level = 2. Compiler: -Pnative.forceCompiler=g++. +Batch results: A3 1/1 PASS, B 2/5, C 4/8 (mechanisms 1+2). Batch D (fresh-chunk) pending. + +## Mechanism 3 (E-mode, batch E: 5/8): safepoint pain budget starvation +E8 dissection: chain WAS built for the watched chunk (target_tag=22343, root_kind=8 STATIC, +leak_tag=0 - PLAIN admission path needs NO tracking record, just reachability via elementData), +yet the seam read tag=0 at ALL ~40 explicit attempts. Tail of log: engine frozen in terminal +state - `canAffordNewSearch blocked by safepoint_pain_budget` x1902 consecutive 10Hz iterations +(vs 340 in passing E2), `shouldRunPass terminal_blocked tags_released=1`. A cold external JVM +search costs seconds of measured safepoint pain; default cap 5% of wall-clock clears the debt +in ~a minute; the tail's 60s window gets 1-2 search restarts at best - pure box-load lottery. +Note D8 already proved fresh chunks get plain-admitted+built when searches DO run (43 builds +leak_tag=0). The fresh-chunk rotation's D-era 0/8 was self-inflicted: rotating EVERY attempt read +each chunk before its maturation (interception+build spans >1.5s); fixed in E by +TAIL_CHUNK_ATTEMPTS=8 rotation window. + +## Fix 3 (batch F, running): `:painbudget=100` in the flaky test's start command +(ExternalProcessReferenceChainTest:~166, alongside the already-raised budget/pausetarget). +100% = debt clears as fast as it accrues; cadence still bounded by each search's own ~seconds +of work. Test-only config, no product change. + +## Full layer map (each layer evidence-backed, all distinct): +1. seam leak-tag fallback (n-004 corrected fix) - leak-tag observation states across generations +2. byte[] rep + stack-local removal (n-002) - candidate klass must be byte[] +3. tail rep re-pin (mechanism 1) - post-round rep drift +4. fresh-chunk rotation w/ 8-attempt maturation (mechanism 2) - tracking-record loss is moot via + plain admission, but the chunk needs several attempts to mature +5. painbudget=100 (mechanism 3) - engine must actually run searches during the tail + +## VERIFICATION RESULTS (final) +- Batch F (painbudget=100 + all prior layers): **8/8 PASS** on workspace-jb-tester. +- Batch G (2 confirmation runs, INVESTIGATE traces stripped, only the leak-tag fallback kept in + referenceChainEvents.cpp): running, pid 93447, results /tmp/pr800-batchG.txt, logs /tmp/pr800-runG*.log. +- Final local diff (4 files, worktree address-pr-800 at 6855e4b5e, branch jb/rc-6-java-tests): + referenceChainEvents.cpp (seam leak-tag fallback only, 21 diff lines), + referenceChains.h (contract comment for resolvedChainTargetTagForTest), + ExternalProcessReferenceChainTest.java (:painbudget=100 + rationale comment), + StaticFieldGrowingCollectionScenario.java (TAIL_ATTEMPTS/TAIL_CHUNK_ATTEMPTS constants, + LEAK_BUFFER_CAPACITY covers rotations, tail: fresh-chunk rotation with 8-attempt maturation + window + per-attempt rep re-pin + explicit poll before each seam read). +- Batch history on this flake: pre-fix ~40-60% fail; fix-A-era A3 1/1, B 2/5, C 4/8 (mech 1+2), + D 0/8 (self-inflicted every-attempt rotation), E 5/8 (mech 3), F 8/8. +- Remaining steps: batch G green -> spotlessApply (./.claude/commands/build-and-summarize) -> + commit 4 files on jb/rc-6-java-tests -> push -> re-check PR 800 CI. diff --git a/.knowledgebase/memories/nativemem-global-sampling-cas.md b/.knowledgebase/memories/nativemem-global-sampling-cas.md new file mode 100644 index 0000000000..62e0b7e1e3 --- /dev/null +++ b/.knowledgebase/memories/nativemem-global-sampling-cas.md @@ -0,0 +1,19 @@ +--- +id: nativemem-global-sampling-cas +title: nativemem's unsampled malloc path does a CAS on one process-global counter, which contends across threads +kind: finding +tags: [nativemem, overhead, contention, malloc] +applies_to: [ddprof-lib/src/main/cpp/mallocTracer.cpp] +source: malloc-arena-detection/find-hook-global-cas +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + +On every intercepted allocation while running, `MallocTracer::maybeRecord` acquires the `ProfiledThread`, enters a +`CriticalSection`, then `shouldSample()` runs a CAS loop on the global `_bytes_until_sample`. Under many threads +doing small mallocs, all threads write one cache line, so the unsampled cost grows with thread count. Running +nativemem always-on with a sparse interval is therefore not cheap as implemented. Fix direction (PROF-16349): +per-thread countdown in `ProfiledThread`, as HotSpot's `ThreadHeapSampler` does; statistically equivalent because +per-thread Poisson sampling is independent and memoryless. diff --git a/.knowledgebase/memories/ongc-testable-without-jvm.md b/.knowledgebase/memories/ongc-testable-without-jvm.md new file mode 100644 index 0000000000..b9b02ae58d --- /dev/null +++ b/.knowledgebase/memories/ongc-testable-without-jvm.md @@ -0,0 +1,21 @@ +--- +id: ongc-testable-without-jvm +title: LivenessTracker::onGC() records heap-floor samples in gtest without a JVM; container-usage tests must resolve the limit first +kind: finding +tags: [gtest, liveness, cgroup, testing] +applies_to: [ddprof-lib/src/main/cpp/livenessTracker.cpp, ddprof-lib/src/main/cpp/os_linux.cpp, ddprof-lib/src/test/cpp/livenessTracker_ut.cpp] +source: feedback-1/find-gtest-ongc-without-jvm +status: active +recorded: 2026-11-02 +valid_at: 2026-11-02 +--- + + +With no JVM, `HeapUsage::get(false)` leaves `_used` at `(size_t)-1`, so `resolvePostGcHeapUsage()` +returns a positive value and `onGC()` reaches `recordHeapFloorSample`. `onGC()`'s container +branch is therefore drivable from a plain gtest once `_initialized` and `_gc_generations` are set +and `_heap_floor_recording_disabled_for_test` is false. No mock JVM or HeapUsage seam is needed. + +`OS::getContainerMemoryUsage()` returns -1 until the first `OS::getContainerMemoryLimit()` call has +picked the usage file (os_linux.cpp, resolved once). A test asserting on recorded container usage +must call `getContainerMemoryLimit()` first; otherwise it passes vacuously on Linux too. diff --git a/AGENTS.md b/AGENTS.md index fec4357392..4e2d1f87bd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,3 +1,14 @@ + +## Cairn knowledgebase (optional) + +If `.knowledgebase/memories/` exists, this repo carries distilled +institutional memory from past cairn investigations (confirmed findings +and refuted dead ends); if your harness also provides the cairn-memory +skill, consult it before design, implementation, or review work here. +If cairn is not available in your environment, skip this section — +nothing else in this file depends on it. + +