diff --git a/AGENTS.md b/AGENTS.md index a574e785fa..5a5f072b4a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -146,7 +146,7 @@ before working on a subsystem. | String dictionary (arena chunks, rotation/clear protocols) | [doc/architecture/StringDictionary.md](doc/architecture/StringDictionary.md) | | RefCountGuard (lock-free RAII reader-drain before reclamation) | [doc/architecture/RefCountGuard.md](doc/architecture/RefCountGuard.md) | | TLS context (thread-local tracing context, distributed tracing) | [doc/architecture/TLSContext.md](doc/architecture/TLSContext.md) | -| Profiling engines (CPU, wall-clock, allocation, native memory, liveness) | [doc/architecture/ProfilingEngines.md](doc/architecture/ProfilingEngines.md) | +| Profiling engines (CPU, wall-clock, allocation, native memory, liveness; thread-registration gates) | [doc/architecture/ProfilingEngines.md](doc/architecture/ProfilingEngines.md) | | JFR integration (FlightRecorder, metadata, constant pools, buffers) | [doc/architecture/JfrIntegration.md](doc/architecture/JfrIntegration.md) | | Live-heap reference chains (implementation, signal scheduling, design) | [doc/architecture/LiveHeapReferenceChainsImplementation.md](doc/architecture/LiveHeapReferenceChainsImplementation.md), [doc/architecture/ReferenceChainsSignalsExplained.md](doc/architecture/ReferenceChainsSignalsExplained.md), [doc/ReferenceChainsDesign.md](doc/ReferenceChainsDesign.md), [doc/ReferenceChainsCollectionSummary.md](doc/ReferenceChainsCollectionSummary.md) | | Native memory profiling (malloc/free interception, sampling) | [doc/architecture/NativeMemoryProfiling.md](doc/architecture/NativeMemoryProfiling.md) | @@ -272,6 +272,42 @@ Otherwise a signal can sample a half-torn-down thread, or re-acquire a thread that keeps running silently stops profiling that thread — use `SignalBlocker` for a temporary window. +**Thread registration.** The `pthread_create` hook and the JVMTI `ThreadStart`/ +`ThreadEnd` callbacks run without `_state_lock` and can overlap `Profiler::start()`/ +`stop()`, while engines size, replace or tear down their per-thread tables (`CTimer`'s +`_timers`, `PerfEvents`' `_events`). So: +- reach the engines only through `Profiler::registerThread()`/`unregisterThread()`, + never `_cpu_engine->registerThread()` directly. These calls check a gate + (`_engines_ready` / `_thread_tables_stable`), count themselves in flight + (`HookCallScope`), and run with thread cancellation disabled; +- an engine with a per-thread table sizes it first, then calls + `Profiler::openThreadRegistration()`, then lists the existing threads. Free or + replace a table only while its gate is closed: closing a gate + (`closeThreadRegistration()`, `ThreadTablesFreeze`) waits until no call is inside an + engine. Each gate counts its calls separately: `closeThreadRegistration()` waits only + for registration calls, because unregistration stays open during `stop()`; + `ThreadTablesFreeze` waits for both. The wait is bounded; when it reports failure, + free or replace no table. A `ThreadTablesFreeze` timeout fails the operation; a + `closeThreadRegistration()` timeout puts the profiler in `ERROR` (`stop()` still + completes, since it frees no table), so no later `start()` replaces a table under + the stuck call. Engine + resources kept across `stop()` are released (`Engine::releaseThreads()`) when another + engine replaces the CPU engine. Known exception: `PerfEvents::start()` still + replaces `_events` without a gate (PROF-16359). `Profiler::startCpuEngine()` opens the + registration gate once any CPU engine started, so the gate is open iff it did, and it + admits only the CPU engine. A wall engine that needs `registerThread()` needs its own + gate: opening this one would also admit a CPU engine that failed; +- an engine's `stop()` must tolerate a concurrent `unregisterThread()`: `stop()` closes + only the registration gate, and unregistration stays open while the engines stop; +- close unregistration (`ThreadTablesFreeze`) only around an actual table replacement + or a CPU engine change, never for a whole `start()`: an exit dropped while it is closed leaves a stale slot + that a thread reusing the tid cannot claim, and `start()`/`stop()` cycles repeat; +- restore the cancellation state outside every destructor-bearing scope, never from a + destructor: current glibc acts on a pending asynchronous cancellation as soon as it + is re-enabled, and unwinding out of an implicitly `noexcept` destructor terminates. + +Details: [ProfilingEngines.md](doc/architecture/ProfilingEngines.md#thread-registration). + **JVMTI callbacks.** Callbacks run on whichever JVM thread triggers the event — including compiler, VM and GC threads that never went through our `pthread_create` hook — so call `ProfiledThread::initCurrentThreadSignalSafe()` first. diff --git a/ddprof-lib/src/main/cpp/ctimer.h b/ddprof-lib/src/main/cpp/ctimer.h index cfc79414ac..2b7fcce2da 100644 --- a/ddprof-lib/src/main/cpp/ctimer.h +++ b/ddprof-lib/src/main/cpp/ctimer.h @@ -1,5 +1,6 @@ /* * Copyright 2017 Andrei Pangin + * Copyright 2026, Datadog, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -25,6 +26,8 @@ #include class CTimer : public Engine { + friend class CTimerTestAccessor; + private: // cppcheck-suppress unusedPrivateFunction static void signalHandler(int signo, siginfo_t *siginfo, void *ucontext); @@ -41,6 +44,13 @@ class CTimer : public Engine { static int _max_timers; static int *_timers; + static bool resizeTimers(int max_timers); + // Sizes the timer table for start(), replacing it under a + // Profiler::ThreadTablesFreeze only when the size changes. + static Error sizeTimerTable(int max_timers); + // start() with the timer table size passed in rather than read from + // pid_max, so a test can drive the allocation-failure path. + Error startWithTableSize(Arguments &args, int max_timers); int registerThread(int tid); void unregisterThread(int tid); diff --git a/ddprof-lib/src/main/cpp/ctimer_linux.cpp b/ddprof-lib/src/main/cpp/ctimer_linux.cpp index f1e1a8a418..ea7e1e3784 100644 --- a/ddprof-lib/src/main/cpp/ctimer_linux.cpp +++ b/ddprof-lib/src/main/cpp/ctimer_linux.cpp @@ -24,6 +24,7 @@ #include "debugSupport.h" #include "libraries.h" #include "log.h" +#include "nativeMem.h" #include "profiler.h" #include "samplerPerf.h" #include "signalCookie.h" @@ -54,7 +55,7 @@ bool CTimer::_enabled = false; int CTimer::_signal; int CTimer::registerThread(int tid) { - if (tid >= _max_timers) { + if (tid < 0 || tid >= _max_timers) { Log::warn("tid[%d] > pid_max[%d]. Restart profiler after changing pid_max", tid, _max_timers); return -1; @@ -119,7 +120,7 @@ int CTimer::registerThread(int tid) { } void CTimer::unregisterThread(int tid) { - if (tid >= _max_timers) { + if (tid < 0 || tid >= _max_timers) { return; } // Atomic acquire to avoid possible leak when unregistering @@ -131,6 +132,51 @@ void CTimer::unregisterThread(int tid) { } } +// Sizes the tid-indexed timer table and accounts it under NM_CTIMER. Called +// from sizeTimerTable() under a Profiler::ThreadTablesFreeze, so no +// thread-hook call (pthread hook or JVMTI ThreadStart/ThreadEnd) can be inside +// the table while it is replaced. +// +// On allocation failure the table is left empty (_max_timers == 0), so the tid +// bound checks reject every tid instead of indexing a NULL table, and the next +// start() retries. The table currently held always has _max_timers slots, so +// that is the size released from NM_CTIMER. +bool CTimer::resizeTimers(int max_timers) { + if (max_timers == _max_timers) { + return true; + } + free(_timers); + NativeMem::record(NM_CTIMER, -(long long)((size_t)_max_timers * sizeof(int))); + _timers = (int *)calloc(max_timers, sizeof(int)); + if (_timers == NULL) { + _max_timers = 0; + return false; + } + _max_timers = max_timers; + NativeMem::record(NM_CTIMER, (long long)((size_t)max_timers * sizeof(int))); + return true; +} + +// A restart with an unchanged pid_max keeps the table and never closes thread +// unregistration, so an exit during start() still releases its timer. Only an +// actual replacement freezes the hook calls. If one is still inside the table +// after the wait budget, start() fails and keeps the old table; whether a +// later start() can succeed depends on the stuck call (see +// Profiler::startCpuEngine()). +Error CTimer::sizeTimerTable(int max_timers) { + if (max_timers == _max_timers) { + return Error::OK; + } + Profiler::ThreadTablesFreeze freeze(Profiler::instance()); + if (!freeze.drained()) { + return Error("thread hook calls did not drain"); + } + if (!resizeTimers(max_timers)) { + return Error("Failed to allocate CPU timer table"); + } + return Error::OK; +} + Error CTimer::check(Arguments &args) { timer_t timer; if (timer_create(CLOCK_THREAD_CPUTIME_ID, NULL, &timer) < 0) { @@ -142,6 +188,10 @@ Error CTimer::check(Arguments &args) { } Error CTimer::start(Arguments &args) { + return startWithTableSize(args, OS::getMaxThreadId()); +} + +Error CTimer::startWithTableSize(Arguments &args, int max_timers) { if (args._interval < 0) { return Error("interval must be positive"); } @@ -150,11 +200,9 @@ Error CTimer::start(Arguments &args) { _cstack = args._cstack; _signal = SIGPROF; - int max_timers = OS::getMaxThreadId(); - if (max_timers != _max_timers) { - free(_timers); - _timers = (int *)calloc(max_timers, sizeof(int)); - _max_timers = max_timers; + Error error = sizeTimerTable(max_timers); + if (error) { + return error; } // Prime the origin-check cache from this non-signal context before any @@ -165,6 +213,10 @@ Error CTimer::start(Arguments &args) { OS::installSignalHandler(_signal, signalHandler); + // The table is sized: open the hook gate before listing threads (see + // Profiler::openThreadRegistration()). + Profiler::instance()->openThreadRegistration(); + // Register all existing threads. Individual failures are benign — a thread // may exit between listThreads() and registerThread(), and new threads // will register themselves on creation. check() already validated that the diff --git a/ddprof-lib/src/main/cpp/engine.h b/ddprof-lib/src/main/cpp/engine.h index c9e75e2986..2deffca625 100644 --- a/ddprof-lib/src/main/cpp/engine.h +++ b/ddprof-lib/src/main/cpp/engine.h @@ -1,5 +1,6 @@ /* * Copyright 2017 Andrei Pangin + * Copyright 2026, Datadog, Inc. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -53,6 +54,10 @@ class Engine { virtual int registerThread(int tid) { return -1; } virtual void unregisterThread(int tid) {} + // Releases the per-thread resources the engine keeps across stop(). Called + // when another engine replaces it in its Profiler slot: thread exits no + // longer reach it after that, so nothing else would release them. + virtual void releaseThreads() {} virtual void enableEvents(bool enabled) { // do nothing diff --git a/ddprof-lib/src/main/cpp/nativeMem.h b/ddprof-lib/src/main/cpp/nativeMem.h index cbcb70844d..c0253a7d5e 100644 --- a/ddprof-lib/src/main/cpp/nativeMem.h +++ b/ddprof-lib/src/main/cpp/nativeMem.h @@ -14,10 +14,19 @@ // per-category live byte gauges partition the total: sum(category) == total, // with no double counting. // -// These gauges track memory that is actually touched (and therefore resident), -// not address space that has merely been reserved. NM_CALLTRACE in particular -// counts the bytes LinearAllocator::alloc() has bump-allocated, NOT the -// capacity of the 8 MiB chunks backing them -- see linearAllocator.cpp. +// With the exceptions below, these gauges track memory that is actually touched +// (and therefore resident), not address space that has merely been reserved. +// NM_CALLTRACE in particular counts the bytes LinearAllocator::alloc() has +// bump-allocated, NOT the capacity of the 8 MiB chunks backing them -- see +// linearAllocator.cpp. +// +// Exceptions: the tid-indexed tables of CTimer (NM_CTIMER, +// pid_max * sizeof(int)) and PerfEvents (part of NM_PERF, +// pid_max * sizeof(PerfEvent), two cache lines per slot) are counted at their +// full calloc'd size. That memory is mostly untouched address space -- only the +// pages holding live tids become resident -- so for these two categories the +// gauge overstates RSS and should be read as an upper bound. NM_PERF's +// per-thread ring pages (2 pages each) are touched and resident. // // Consequently NM_CALLTRACE is no longer an "arena reserved" figure with // CALLTRACE_STORAGE_BYTES as its used slice: both now count touched bytes, and @@ -25,6 +34,12 @@ // not chunk slack. Do not compute arena waste as the difference between them. // DICTIONARY_ARENA_WASTE_BYTES remains a genuine nested waste figure for the // DICTIONARY arena. None of these nested counters are summed in here. +// +// The exported per-category counter names (native_mem_{live,avg,max, +// chunk_overhead,post_flush_live,post_flush_max}_bytes., written by +// Recording::writeNativeMem()) derive from this table, and the set grows when +// a category is added, so consumers should match native_mem_*_bytes. counters +// by prefix rather than hard-code the category list. #define DD_NATIVE_MEM_CATEGORY_TABLE(X) \ X(CALLTRACE, "calltrace") \ X(DICTIONARY, "dictionary") \ @@ -38,6 +53,7 @@ X(METHOD_MAP, "method_map") \ X(LIVENESS, "liveness") \ X(WALLCLOCK, "wallclock") \ + X(CTIMER, "ctimer") \ X(MISC, "misc") #define X_NM_ENUM(a, b) NM_##a, diff --git a/ddprof-lib/src/main/cpp/perfEvents.h b/ddprof-lib/src/main/cpp/perfEvents.h index 560d214352..6f9eb32a72 100644 --- a/ddprof-lib/src/main/cpp/perfEvents.h +++ b/ddprof-lib/src/main/cpp/perfEvents.h @@ -52,6 +52,7 @@ class PerfEvents : public Engine { virtual int registerThread(int tid); virtual void unregisterThread(int tid); + void releaseThreads(); long interval() const { return _interval; } const char *name() { return "PerfEvents"; } diff --git a/ddprof-lib/src/main/cpp/perfEvents_linux.cpp b/ddprof-lib/src/main/cpp/perfEvents_linux.cpp index 05c1e82454..842e57db00 100644 --- a/ddprof-lib/src/main/cpp/perfEvents_linux.cpp +++ b/ddprof-lib/src/main/cpp/perfEvents_linux.cpp @@ -962,16 +962,29 @@ Error PerfEvents::start(Arguments &args) { // sites); capture the old size first since _max_events is overwritten below. long long old_bytes = (_events != NULL) ? (long long)((size_t)_max_events * sizeof(PerfEvent)) : 0; + // The table is the only record of which fd and ring belong to which tid + // (stop() keeps them). Release them while it still exists; otherwise + // they leak, and stay armed, once it is freed. + releaseThreads(); free(_events); if (old_bytes > 0) { NativeMem::record(NM_PERF, -old_bytes); } _events = (PerfEvent *)calloc(max_events, sizeof(PerfEvent)); - _max_events = max_events; - if (_events != NULL) { - NativeMem::record(NM_PERF, - (long long)((size_t)max_events * sizeof(PerfEvent))); + if (_events == NULL) { + // Back to the not-started value, so registerThread() and + // unregisterThread() reject every tid. Return before the signal + // handler, the hook and the registration gate are set up: start() + // fails, so the CPU engine is neither registered nor enabled for this + // recording. The next start() retries the allocation. + _max_events = -1; + Log::error("perfEvents: could not allocate the table for %d threads; " + "PerfEvents is disabled for this recording", max_events); + return Error("Failed to allocate perf event table"); } + _max_events = max_events; + NativeMem::record(NM_PERF, + (long long)((size_t)max_events * sizeof(PerfEvent))); } OS::installSignalHandler(SIGPROF, signalHandler); @@ -980,6 +993,10 @@ Error PerfEvents::start(Arguments &args) { __atomic_store_n(_pthread_entry, (void *)pthread_setspecific_hook, __ATOMIC_RELEASE); + // _events is sized: open the hook gate before listing threads (see + // Profiler::openThreadRegistration()). + Profiler::instance()->openThreadRegistration(); + // Create perf_events for all existing threads int threads_sz = 0, threads_cap; int *threads = (int *)malloc((threads_cap = 1024) * sizeof(int)); @@ -1052,12 +1069,28 @@ void PerfEvents::stop() { // check whether the thread has been registered already on start. } +// stop() keeps every fd and ring, and only unregisterThread() releases them +// when a thread exits. Releases them all when nothing else could: +// Profiler::replaceCpuEngine() calls this under a ThreadTablesFreeze once +// another CPU engine replaces this one, and start() calls it before it frees +// the table on a pid_max change. The start() call runs without a freeze, so a +// concurrent unregisterThread() of the same tid is still possible there +// (PROF-16359). +void PerfEvents::releaseThreads() { + for (int tid = 0; tid < _max_events; tid++) { + unregisterThread(tid); + } +} + int PerfEvents::walkKernel(int tid, const void **callchain, int max_depth, StackContext *java_ctx) { if (!(_ring & RING_KERNEL)) { // we are not capturing kernel stacktraces return 0; } + if (_events == NULL || tid < 0 || tid >= _max_events) { + return 0; // no table: the allocation failed (see start()) + } PerfEvent *event = &_events[tid]; if (!event->tryLock()) { @@ -1148,6 +1181,12 @@ int PerfEvents::walkKernel(int tid, const void **callchain, int max_depth, } void PerfEvents::resetBuffer(int tid) { + // Fds left armed by an earlier recording keep delivering SIGPROF, and the + // handler calls this on every exit, also after a failed table allocation + // left _events NULL (_max_events -1). + if (_events == NULL || tid < 0 || tid >= _max_events) { + return; + } PerfEvent *event = &_events[tid]; if (!event->tryLock()) { return; // the event is being destroyed diff --git a/ddprof-lib/src/main/cpp/profiler.cpp b/ddprof-lib/src/main/cpp/profiler.cpp index 94f8df8d0d..b61a127f47 100644 --- a/ddprof-lib/src/main/cpp/profiler.cpp +++ b/ddprof-lib/src/main/cpp/profiler.cpp @@ -48,22 +48,31 @@ #include #include +#include #include #include #include #include #include +#include +#include #include #include #include #include #include #include +#include #include +// Budget for waitForHookCalls(). Hook calls do a few syscalls per engine, so +// a full second means a call is stuck (or the in-flight count leaked). +static const long HOOK_CALLS_DRAIN_TIMEOUT_NS = 1000000000L; + // The instance is not deleted on purpose, since profiler structures // can be still accessed concurrently during VM termination Profiler *const Profiler::_instance = new Profiler(); +long long Profiler::_hook_drain_timeout_ns = HOOK_CALLS_DRAIN_TIMEOUT_NS; volatile bool Profiler::_signals_initialized = false; volatile bool Profiler::_need_JDK_8313796_workaround = true; @@ -108,8 +117,9 @@ void Profiler::onThreadStart(jvmtiEnv *jvmti, JNIEnv *jni, jthread thread) { } } - _cpu_engine->registerThread(tid); - _wall_engine->registerThread(tid); + // Through the same gate and in-flight count as the pthread hook: disabling + // the JVMTI event in stop() does not wait for callbacks already running. + registerThread(tid); } void Profiler::onThreadEnd(jvmtiEnv *jvmti, JNIEnv *jni, jthread thread) { @@ -140,8 +150,7 @@ void Profiler::onThreadEnd(jvmtiEnv *jvmti, JNIEnv *jni, jthread thread) { // partially-torn-down thread (PROF-14674). { blockProfilingForExit(); - _cpu_engine->unregisterThread(tid); - _wall_engine->unregisterThread(tid); + unregisterThread(tid); LivenessTracker::instance()->releaseThreadLocalState(); ProfiledThread::release(); } @@ -159,14 +168,131 @@ void Profiler::onThreadEnd(jvmtiEnv *jvmti, JNIEnv *jni, jthread thread) { if (_reference_chains_active) { ReferenceChainTracker::instance()->unregisterThreadObject(jni, tid); } - _cpu_engine->unregisterThread(tid); - _wall_engine->unregisterThread(tid); + unregisterThread(tid); LivenessTracker::instance()->releaseThreadLocalState(); } +// Marks a native-thread hook call as in flight for the duration of a +// Profiler::registerThread()/unregisterThread() call, so a gate can be closed +// and the calls that already passed it waited for. Each gate has its own +// counter (_register_calls_inflight, _unregister_calls_inflight). The +// increment and the gate +// load in the caller are seq_cst, pairing with the seq_cst gate store and +// counter load on the closing side: either the hook sees the gate closed, or +// the closer sees the call in flight and waits for it. Registration and +// unregistration both raise the counter on every call. +class HookCallScope { +public: + explicit HookCallScope(std::atomic &inflight) : _inflight(inflight) { + _inflight.fetch_add(1, std::memory_order_seq_cst); + } + ~HookCallScope() { _inflight.fetch_sub(1, std::memory_order_release); } + HookCallScope(const HookCallScope &) = delete; + HookCallScope &operator=(const HookCallScope &) = delete; + +private: + std::atomic &_inflight; +}; + +void Profiler::openThreadRegistration() { + _thread_tables_stable.store(true, std::memory_order_seq_cst); + _engines_ready.store(true, std::memory_order_seq_cst); +} + +bool Profiler::closeThreadRegistration() { + _engines_ready.store(false, std::memory_order_seq_cst); + return waitForHookCalls(/*include_unregistration=*/false); +} + +bool Profiler::waitForHookCalls(bool include_unregistration) { + // Hook calls only do a few syscalls per engine, so the wait is short. It is + // bounded so that a call that never leaves (or a leaked in-flight count) + // fails the caller instead of wedging it under _state_lock. + // clock_gettime() rather than OS::nanotime(): see SamplerPerf::primeClock() + // in start(). + auto inflight = [this, include_unregistration]() { + int calls = _register_calls_inflight.load(std::memory_order_seq_cst); + if (include_unregistration) { + calls += _unregister_calls_inflight.load(std::memory_order_seq_cst); + } + return calls; + }; + if (inflight() == 0) { + return true; + } + long long budget = _hook_drain_timeout_ns; + struct timespec deadline; + if (clock_gettime(CLOCK_MONOTONIC, &deadline) != 0) { + Log::warn("Profiler: clock_gettime(CLOCK_MONOTONIC) failed (errno=%d) " + "waiting for thread hook calls", errno); + return false; + } + // Division/modulo rather than a single carry: the budget may exceed 2 s. + long long nsec = (long long)deadline.tv_nsec + budget; + deadline.tv_sec += (time_t)(nsec / 1000000000LL); + deadline.tv_nsec = (long)(nsec % 1000000000LL); + + while (inflight() != 0) { + struct timespec now; + if (clock_gettime(CLOCK_MONOTONIC, &now) != 0) { + Log::warn("Profiler: clock_gettime(CLOCK_MONOTONIC) failed (errno=%d) " + "waiting for thread hook calls", errno); + return false; + } + if (now.tv_sec > deadline.tv_sec || + (now.tv_sec == deadline.tv_sec && now.tv_nsec >= deadline.tv_nsec)) { + int remaining = inflight(); + Log::warn("Profiler: timed out after %ldms waiting for %d in-flight " + "thread hook call(s)", (long)(budget / 1000000LL), remaining); + return false; + } + sched_yield(); + } + return true; +} + +// Thread cancellation is disabled for the whole hook call. The engine calls +// reach cancellation points (close() in PerfEvents::unregisterThread(), stdio +// in Log::warn()); acting on a cancellation there would unwind through +// ~HookCallScope(), which aborts with the statically linked libgcc (see +// run_with_cleanup() in libraryPatcher_linux.cpp), or skip it on musl and +// leave the in-flight count raised. Every later closeThreadRegistration() and +// ThreadTablesFreeze would then time out, so stop() and every start() that +// replaces a table would fail. +// +// This is a manual disable/restore pair rather than a guard on purpose: with +// asynchronous cancellation, current glibc's pthread_setcancelstate() acts on a +// pending request as soon as cancellation is re-enabled, so the restore must +// not run inside a destructor, which is implicitly noexcept and would +// terminate. It runs here, after every destructor-bearing scope of the call +// has ended. int Profiler::registerThread(int tid) { - return _instance->_cpu_engine->registerThread(tid) | - _instance->_wall_engine->registerThread(tid); + int cancel_state; + pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &cancel_state); + int result = registerThreadInEngines(tid); + pthread_setcancelstate(cancel_state, nullptr); + return result; +} + +void Profiler::unregisterThread(int tid) { + int cancel_state; + pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &cancel_state); + unregisterThreadInEngines(tid); + pthread_setcancelstate(cancel_state, nullptr); +} + +int Profiler::registerThreadInEngines(int tid) { + // The counter is raised before the gate is read, both seq_cst, so the + // closing side either sees this call in flight or this call sees the gate + // closed. A thread created before the gate opens is covered by the CPU + // engine's start(), which lists threads after opening it. + HookCallScope scope(_instance->_register_calls_inflight); + if (!_instance->_engines_ready.load(std::memory_order_seq_cst)) { + return -1; // not registered, as Engine::registerThread() reports it + } + // The gate admits only the CPU engine (see _engines_ready). + Engine *cpu = __atomic_load_n(&_instance->_cpu_engine, __ATOMIC_ACQUIRE); + return cpu->registerThread(tid); } #ifdef UNIT_TEST static std::atomic g_test_last_unregistered_tid{-1}; @@ -179,19 +305,28 @@ void Profiler::resetUnregisterObservableForTest() { } #endif -void Profiler::unregisterThread(int tid) { +void Profiler::unregisterThreadInEngines(int tid) { + HookCallScope scope(_instance->_unregister_calls_inflight); + // Closed only inside a ThreadTablesFreeze (see _thread_tables_stable). + if (!_instance->_thread_tables_stable.load(std::memory_order_seq_cst)) { + return; + } #ifdef UNIT_TEST // In gtest, _cpu_engine/_wall_engine are null (profiler not started). - // Record the tid so integration tests can verify the call happened without - // crashing on the null engine dereference. This bypasses the real engine - // unregister path entirely, so that path is covered only by JVM-level tests, + // Record the tid so integration tests can verify the call passed the gate + // without crashing on the null engine dereference. This bypasses the real + // engine unregister path, so that path is covered only by JVM-level tests, // not by these gtests. UNIT_TEST is defined solely for the gtest binaries // (see GtestTaskBuilder); the shipped library never compiles this branch. g_test_last_unregistered_tid.store(tid, std::memory_order_relaxed); return; #endif - _instance->_cpu_engine->unregisterThread(tid); - _instance->_wall_engine->unregisterThread(tid); + // start() may replace the engine slots concurrently. Either engine is safe + // to call: unregisterThread() releases only the exiting thread's own slot. + __atomic_load_n(&_instance->_cpu_engine, __ATOMIC_ACQUIRE) + ->unregisterThread(tid); + __atomic_load_n(&_instance->_wall_engine, __ATOMIC_ACQUIRE) + ->unregisterThread(tid); } const char *Profiler::asgctError(int code) { @@ -1006,6 +1141,77 @@ void Profiler::disableEngines() { _wall_engine->enableEvents(false); } +// Keeps the registration gate open iff the CPU engine started (see +// _engines_ready): an engine that failed must never see the hook. Opening it +// here is a no-op for engines that opened it in their start() (CTimer, +// PerfEvents), and opens it for the table-less ones (ITimer, ITimerJvmti, J9). +bool Profiler::startCpuEngine(Arguments &args) { + Error error = _cpu_engine->start(args); + if (error) { + Log::warn("%s", error.message()); + // The engine may have opened the gate before failing. If a registration + // call is still inside it after the wait budget, that call may keep using + // the engine's table, and only start() replaces tables. ERROR rejects + // every later start(); the caller undoes what its start() set up. A + // failed table freeze in the engine's start() ends here too when the + // stuck call is a registration call; otherwise the failure is retryable. + if (!closeThreadRegistration()) { + Log::error("Profiler: a thread registration call did not leave the CPU " + "engine; the profiler is disabled"); + _state.store(ERROR, std::memory_order_release); + } + return false; + } + openThreadRegistration(); + return true; +} + +// Undoes what start() set up once _jfr.start() succeeded, when start() then +// fails without a running recording: engine events, the library trap and the +// refresher, the JFR recording, and the library patches. The patches are +// restored after JFR stops, as in stop(); a later start() patches again from +// Libraries::updateSymbols(). Without this, a start() that left the profiler +// in ERROR would leave the hooks patched for the rest of the process. +void Profiler::rollBackStart() { + disableEngines(); + switchLibraryTrap(false); + _libs->stopRefresher(); + lockAll(); + _jfr.stop(); + unlockAll(); + LibraryPatcher::unpatch_libraries(); +} + +// stop() leaves an engine's per-thread resources in place (PerfEvents keeps +// its fds so that unregisterThread() closes them when threads exit). Once the +// slot points at another engine, thread exits no longer reach the previous +// one, so it releases them first. The freeze keeps registerThread() and +// unregisterThread() off both engines meanwhile. +bool Profiler::replaceCpuEngine(Engine *next) { + Engine *previous = _cpu_engine; + if (previous != NULL && previous != next) { + ThreadTablesFreeze freeze(this); + if (!freeze.drained()) { + return false; + } + previous->releaseThreads(); + __atomic_store_n(&_cpu_engine, next, __ATOMIC_RELEASE); + return true; + } + __atomic_store_n(&_cpu_engine, next, __ATOMIC_RELEASE); + return true; +} + +void Profiler::enableCpuEngine(int activated) { + // Only when the CPU engine itself started: another mode may have activated + // after it failed, and fds or timers left armed by an earlier recording + // (PerfEvents keeps its fds across stop()) would otherwise emit CPU samples + // into this recording. + if (activated & EM_CPU) { + _cpu_engine->enableEvents(true); + } +} + void Profiler::segvHandler(int signo, siginfo_t *siginfo, void *ucontext) { // J9 installs a SIGSEGV handler that uses siglongjmp() to recover from // null-pointer-check faults during normal Java execution. When we chain to @@ -1604,8 +1810,17 @@ Error Profiler::start(Arguments &args, bool reset) { _thread_filter.remove(slot_id); // Remove from filtering initially (matches onThreadStart behavior) } - _cpu_engine = selectCpuEngine(args); - _wall_engine = selectWallEngine(args); + // Registration stays closed until the CPU engine starts; unregistration + // stays open (see _engines_ready). No wait here: only replaceCpuEngine() and + // the CPU engine's table sizing touch per-thread tables, each behind its own + // ThreadTablesFreeze. + _engines_ready.store(false, std::memory_order_seq_cst); + if (!replaceCpuEngine(selectCpuEngine(args))) { + // Nothing replaced or released yet. Retryable, like the earlier error + // returns in this function. + return Error("thread hook calls did not drain; retry start()"); + } + __atomic_store_n(&_wall_engine, selectWallEngine(args), __ATOMIC_RELEASE); _cstack = args._cstack; _force_jmethodID = args._force_jmethodID; if (_cstack == CSTACK_DEFAULT) { @@ -1677,15 +1892,18 @@ Error Profiler::start(Arguments &args, bool reset) { // can fire. SamplerPerf::primeClock(); + // error is Error::OK here: every earlier failure returned. A CPU engine + // failure is recoverable and logged by startCpuEngine(). int activated = 0; - if ((_event_mask & EM_CPU) && _cpu_engine != &noop_engine) { - error = _cpu_engine->start(args); - if (error) { - Log::warn("%s", error.message()); - error = Error::OK; // recoverable - } else { - activated |= EM_CPU; - } + if ((_event_mask & EM_CPU) && _cpu_engine != &noop_engine && + startCpuEngine(args)) { + activated |= EM_CPU; + } + if (state() == ERROR) { + // startCpuEngine() could not close the registration gate after the CPU + // engine failed. Nothing else has started; undo the setup above. + rollBackStart(); + return Error("thread hook calls did not drain; the profiler is disabled"); } if ((_event_mask & EM_WALL) && _wall_engine != &noop_engine) { // Enable wall clock BEFORE starting its pthread: timerLoopCommon() checks @@ -1737,12 +1955,7 @@ Error Profiler::start(Arguments &args, bool reset) { Log::warn("%s", error.message()); if (_event_mask == EM_NATIVEMEM) { // nativemem is the only requested mode: propagate the real error - disableEngines(); - switchLibraryTrap(false); - _libs->stopRefresher(); - lockAll(); - _jfr.stop(); - unlockAll(); + rollBackStart(); return error; } error = Error::OK; // recoverable when other modes are also active @@ -1787,8 +2000,9 @@ Error Profiler::start(Arguments &args, bool reset) { // ever sees finds the engine fully wired. Unlike wall clock, CPU handlers // re-check _enabled on every signal, so enabling here (rather than before // start()) does not lose samples. - // Paired with drainInflight() on the stop side. - _cpu_engine->enableEvents(true); + // Paired with drainInflight() on the stop side. enableCpuEngine() enables + // it only if the CPU engine itself started. + enableCpuEngine(activated); _state.store(RUNNING, std::memory_order_release); _start_time = time(NULL); @@ -1796,13 +2010,7 @@ Error Profiler::start(Arguments &args, bool reset) { return Error::OK; } // no engine was activated; perform cleanup - disableEngines(); - switchLibraryTrap(false); - _libs->stopRefresher(); - - lockAll(); - _jfr.stop(); - unlockAll(); + rollBackStart(); return Error( "Neither CPU, wallclock nor allocation profiling could be started"); @@ -1830,6 +2038,22 @@ Error Profiler::stop() { return Error("signal handlers did not drain; teardown skipped, retry stop()"); } + // Close the registration gate and wait for the calls already past it, so no + // thread arms a timer or opens an fd after the engines below stop. + // Unregistration stays open (see _thread_tables_stable). The engine stops + // below tolerate a concurrent unregisterThread() because none of them frees + // a per-thread table: CTimer releases slots with a per-slot CAS, and + // PerfEvents::stop() is a no-op. For the same reason the stop still + // completes if a registration call is stuck inside an engine after the wait + // budget; the profiler then ends in ERROR, which rejects every later + // start(), the only place a table is replaced. Unregistration calls are not + // waited for here: they may keep arriving during stop() under thread churn. + bool hooks_drained = closeThreadRegistration(); + if (!hooks_drained) { + Log::error("Profiler: thread hook calls did not drain; stopping and " + "disabling the profiler"); + } + if (_event_mask & EM_ALLOC) _alloc_engine->stop(); if (_reference_chains_active) { @@ -1950,6 +2174,10 @@ Error Profiler::stop() { // owned by library metadata, so we must keep library patches active until after serialization LibraryPatcher::unpatch_libraries(); + if (!hooks_drained) { + _state.store(ERROR, std::memory_order_release); + return Error("thread hook calls did not drain; the profiler is stopped and disabled"); + } _state.store(IDLE, std::memory_order_release); return Error::OK; } @@ -1963,17 +2191,21 @@ Error Profiler::check(Arguments &args) { error = checkJvmCapabilities(); + // check() runs while the profiler is IDLE, after stop(): unregistration is + // still open then, and the previous recording's engine may hold fds that + // only its unregisterThread() closes. Selecting into the slots here would + // route those exits to the new engine and leak the fds, so check() selects + // into locals. checkState() rejects a RUNNING profiler before this point. + Engine *cpu_engine = NULL; if (!error && (args._event != NULL || args._cpu >= 0)) { - _cpu_engine = selectCpuEngine(args); - error = _cpu_engine->check(args); + cpu_engine = selectCpuEngine(args); + error = cpu_engine->check(args); } if (!error && args._wall >= 0) { - _wall_engine = selectWallEngine(args); - error = _wall_engine->check(args); + error = selectWallEngine(args)->check(args); } if (!error && args._memory >= 0) { - _alloc_engine = selectAllocEngine(args); - error = _alloc_engine->check(args); + error = selectAllocEngine(args)->check(args); } if (!error && args._nativemem >= 0) { error = malloc_tracer.check(args); @@ -1984,7 +2216,7 @@ Error Profiler::check(Arguments &args) { if (!error) { if (args._cstack == CSTACK_DWARF && !DWARF_SUPPORTED) { return Error("DWARF unwinding is not supported on this platform"); - } else if (args._cstack == CSTACK_LBR && _cpu_engine != &perf_events) { + } else if (args._cstack == CSTACK_LBR && cpu_engine != &perf_events) { return Error("Branch stack is supported only with PMU events"); } else if (_cstack >= CSTACK_VM && !(VMStructs::hasStackStructs() && OS::isLinux())) { return Error( diff --git a/ddprof-lib/src/main/cpp/profiler.h b/ddprof-lib/src/main/cpp/profiler.h index 6c85387d0a..bcc90e27d5 100644 --- a/ddprof-lib/src/main/cpp/profiler.h +++ b/ddprof-lib/src/main/cpp/profiler.h @@ -89,6 +89,56 @@ class alignas(alignof(SpinLock)) Profiler { Mutex _state_lock; std::atomic _state; + // Thread-registration gates. The pthread_create hook and the JVMTI + // ThreadStart/ThreadEnd callbacks reach the engines only through + // registerThread()/unregisterThread(), without _state_lock. The two gates + // keep them off the per-thread tables that start() sizes, replaces or + // releases. This is the reference for the contract in code; the rest is in + // doc/architecture/ProfilingEngines.md#thread-registration. + // + // Registration gate: open iff the CPU engine started, and it admits only + // that engine; a wall engine that needs registerThread() needs its own gate. + // Set by openThreadRegistration(), from the CPU engine's start() once its + // table is sized and from startCpuEngine() on success. Cleared by start(), + // closeThreadRegistration() (stop(), failed CPU engine start) and + // ThreadTablesFreeze. + std::atomic _engines_ready; + // Unregistration gate: open except inside a ThreadTablesFreeze. It stays + // open while idle, because PerfEvents keeps its per-thread fds across + // stop() and only unregisterThread() closes the fd of a thread that exits + // after stop(). An exit that arrives while it is closed is dropped, and its + // slot may stay occupied (PROF-16359). + std::atomic _thread_tables_stable; + // Hook calls past a gate that may still be inside an engine, counted per + // gate. closeThreadRegistration() waits only for registration calls: + // unregistration stays open during stop(), and under thread churn its + // count may not reach zero within the wait budget, although no call is + // stuck. ThreadTablesFreeze closes both gates and waits for both counts. + // Kept off _state's cache line: the hook writes them on every native thread + // start and exit, while isRunning() reads _state on every intercepted + // socket call. + alignas(DEFAULT_CACHE_LINE_SIZE) std::atomic _register_calls_inflight; + std::atomic _unregister_calls_inflight; + // Wait budget of waitForHookCalls(); HOOK_CALLS_DRAIN_TIMEOUT_NS (1 s) in + // profiler.cpp. Written only by the friend test accessor. long long: test + // budgets of several seconds must not overflow a 32-bit long. + static long long _hook_drain_timeout_ns; + + // Closes the registration gate and waits for the registration calls already + // past it to leave the engines. Returns false if one is still in flight + // after the wait budget (logged); the caller must not free or replace any + // table then. stop() and startCpuEngine() put the profiler in ERROR on that + // result, so no later start() replaces a table under the stuck call. + bool closeThreadRegistration(); + // Waits until no registration call (and, with include_unregistration, no + // unregistration call) is inside an engine, for at most + // _hook_drain_timeout_ns. Returns false on timeout (logged); the caller + // must not free or replace any table then. + bool waitForHookCalls(bool include_unregistration); + // Gate checks and engine calls behind registerThread()/unregisterThread(), + // run with thread cancellation disabled. + static int registerThreadInEngines(int tid); + static void unregisterThreadInEngines(int tid); // class unload hook Trap _class_unload_hook_trap; typedef void (*NotifyClassUnloadedFunc)(void *); @@ -115,6 +165,11 @@ class alignas(alignof(SpinLock)) Profiler { ThreadFilter _thread_filter; CallTraceStorage _call_trace_storage; FlightRecorder _jfr; + // Replaced only by start(), while thread unregistration is open, so the + // hook paths (registerThreadInEngines(), unregisterThreadInEngines()) load + // them with acquire and start() stores them with release. check() selects + // into locals and runs only while IDLE, where the slots still route exits + // to the previous recording's engine. Engine *_cpu_engine; Engine *_wall_engine = NULL; Engine *_alloc_engine; @@ -167,6 +222,20 @@ class alignas(alignof(SpinLock)) Profiler { static bool prewarmUnwinder(); void disableEngines(); + // Starts _cpu_engine. On success opens thread registration (idempotent for + // engines that already opened it); on failure closes it again and logs the + // error. If closing times out it also puts the profiler in ERROR. Returns + // whether the engine started. + bool startCpuEngine(Arguments &args); + // Installs next as _cpu_engine. When it replaces another engine, the + // previous one releases the per-thread resources it keeps across stop() + // (Engine::releaseThreads()) under a ThreadTablesFreeze. Returns false, + // replacing nothing, if hook calls do not drain. + bool replaceCpuEngine(Engine *next); + // Undoes start()'s setup after _jfr.start() when start() then fails. + void rollBackStart(); + // Enables the CPU engine's events iff EM_CPU is in activated. + void enableCpuEngine(int activated); void onThreadStart(jvmtiEnv *jvmti, JNIEnv *jni, jthread thread); void onThreadEnd(jvmtiEnv *jvmti, JNIEnv *jni, jthread thread); @@ -248,8 +317,41 @@ class alignas(alignof(SpinLock)) Profiler { } public: + // Closes both thread-registration gates (see _engines_ready) and waits for + // the hook calls already past them. Construct it only around an actual + // table replacement (CTimer::sizeTimerTable()) or a CPU engine change + // (replaceCpuEngine()), never for a whole start(): exits are dropped while + // it is alive. The destructor reopens unregistration; registration reopens + // only when the CPU engine starts. + class ThreadTablesFreeze { + public: + explicit ThreadTablesFreeze(Profiler *profiler) + : _profiler(profiler), _drained(false) { + // Both gates: registerThread() must not reach a table being replaced + // or released either. + _profiler->_engines_ready.store(false, std::memory_order_seq_cst); + _profiler->_thread_tables_stable.store(false, std::memory_order_seq_cst); + _drained = _profiler->waitForHookCalls(/*include_unregistration=*/true); + } + ~ThreadTablesFreeze() { + _profiler->_thread_tables_stable.store(true, std::memory_order_seq_cst); + } + // false if hook calls were still in flight after the wait budget; the + // caller must then fail without replacing any table. + bool drained() const { return _drained; } + ThreadTablesFreeze(const ThreadTablesFreeze &) = delete; + ThreadTablesFreeze &operator=(const ThreadTablesFreeze &) = delete; + + private: + Profiler *_profiler; + bool _drained; + }; + Profiler() - : _state_lock(), _state(State::NEW), _class_unload_hook_trap(2), + : _state_lock(), _state(State::NEW), _engines_ready(false), + _thread_tables_stable(true), _register_calls_inflight(0), + _unregister_calls_inflight(0), + _class_unload_hook_trap(2), _notify_class_unloaded_func(NULL), _thread_info(), _class_map(1), _string_label_map(2), _context_value_map(3), _thread_filter(), _call_trace_storage(), _jfr(), _cpu_engine(NULL), _wall_engine(NULL), @@ -374,6 +476,14 @@ class alignas(alignof(SpinLock)) Profiler { Error checkState(); void logStats(); void switchThreadEvents(jvmtiEventMode mode); + /** + * Opens both thread-registration gates (see _engines_ready). A CPU engine + * with a per-thread table calls it from its start() once the table is sized + * and before it lists the existing threads, so the hook covers a thread + * that starts or exits while start() registers the existing ones. + * startCpuEngine() calls it again once the CPU engine started. Idempotent. + */ + void openThreadRegistration(); /** * Remote symbolication packed data layout (BCI_NATIVE_FRAME_REMOTE): diff --git a/ddprof-lib/src/test/cpp/ctimer_ut.cpp b/ddprof-lib/src/test/cpp/ctimer_ut.cpp new file mode 100644 index 0000000000..a112720893 --- /dev/null +++ b/ddprof-lib/src/test/cpp/ctimer_ut.cpp @@ -0,0 +1,1107 @@ +/* + * Copyright 2026, Datadog, Inc. + * SPDX-License-Identifier: Apache-2.0 + */ + +#include + +#ifdef __linux__ + +#include "arguments.h" +#include "common.h" +#include "ctimer.h" +#include "engine.h" +#include "nativeMem.h" +#include "os.h" +#include "profiler.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +// mallinfo2() reports the bytes in use only with glibc's own allocator: it is +// glibc 2.33+ only, and the sanitizer allocators report zeros. Nested so that +// __GLIBC_PREREQ is expanded only where it is defined (not on musl). +#if defined(__GLIBC__) && !defined(ASAN_ENABLED) && !defined(TSAN_ENABLED) +#if __GLIBC_PREREQ(2, 33) +#define HAS_GLIBC_MALLINFO2 1 +#endif +#endif + +// The failure-path tests force calloc to fail with a negative count, which +// makes count * size overflow. The CI sanitizer jobs run gtest binaries +// without *SAN_OPTIONS, so allocator_may_return_null defaults to 0 and the +// sanitizer reports that overflow as a fatal error instead of returning NULL. +#if defined(ASAN_ENABLED) || defined(TSAN_ENABLED) +#define SKIP_IF_CALLOC_OVERFLOW_ABORTS() \ + GTEST_SKIP() << "sanitizer allocators abort on calloc overflow" +#else +#define SKIP_IF_CALLOC_OVERFLOW_ABORTS() +#endif + +class ProfilerTestAccessor { +public: + static bool enginesReady(Profiler *p) { + return p->_engines_ready.load(std::memory_order_acquire); + } + // Registration and unregistration calls in flight, together. + static int hookCallsInflight(Profiler *p) { + return p->_register_calls_inflight.load(std::memory_order_acquire) + + p->_unregister_calls_inflight.load(std::memory_order_acquire); + } + static bool closeThreadRegistration(Profiler *p) { + return p->closeThreadRegistration(); + } + static bool startCpuEngine(Profiler *p, Arguments &args) { + return p->startCpuEngine(args); + } + static void enableCpuEngine(Profiler *p, int activated) { + p->enableCpuEngine(activated); + } + static bool replaceCpuEngine(Profiler *p, Engine *next) { + return p->replaceCpuEngine(next); + } + static Engine *cpuEngine(Profiler *p) { return p->_cpu_engine; } + static State state(Profiler *p) { return p->state(); } + + // Restores the profiler state on destruction, so a test that drives the + // singleton into ERROR does not leak that into later tests. + class StateRestore { + public: + explicit StateRestore(Profiler *p) : _p(p), _state(p->state()) {} + ~StateRestore() { _p->_state.store(_state, std::memory_order_release); } + StateRestore(const StateRestore &) = delete; + StateRestore &operator=(const StateRestore &) = delete; + + private: + Profiler *_p; + State _state; + }; + // Runs body(drained) while a real ThreadTablesFreeze is alive. + template static void withThreadTablesFreeze(Profiler *p, F body) { + Profiler::ThreadTablesFreeze freeze(p); + body(freeze.drained()); + } + + // Sets the hook-call wait budget for its lifetime and restores the previous + // one on destruction, including when a test assertion returns early. + class DrainBudget { + public: + explicit DrainBudget(long long ns) : _saved(Profiler::_hook_drain_timeout_ns) { + Profiler::_hook_drain_timeout_ns = ns; + } + ~DrainBudget() { Profiler::_hook_drain_timeout_ns = _saved; } + DrainBudget(const DrainBudget &) = delete; + DrainBudget &operator=(const DrainBudget &) = delete; + + private: + long long _saved; + }; + // Counts a hook call in flight on one gate's counter for its lifetime, as if + // one were stuck inside an engine, without a thread or an engine to hold it. + class InflightHold { + public: + explicit InflightHold(std::atomic &counter) : _counter(counter) { + _counter.fetch_add(1, std::memory_order_seq_cst); + } + ~InflightHold() { _counter.fetch_sub(1, std::memory_order_release); } + InflightHold(const InflightHold &) = delete; + InflightHold &operator=(const InflightHold &) = delete; + + private: + std::atomic &_counter; + }; + // A registration call held in flight. + class HookCallHold : public InflightHold { + public: + explicit HookCallHold(Profiler *p) : InflightHold(p->_register_calls_inflight) {} + }; + // An unregistration call held in flight. + class UnregisterCallHold : public InflightHold { + public: + explicit UnregisterCallHold(Profiler *p) + : InflightHold(p->_unregister_calls_inflight) {} + }; + static bool threadTablesStable(Profiler *p) { + return p->_thread_tables_stable.load(std::memory_order_acquire); + } + static void setThreadTablesStable(Profiler *p, bool stable) { + p->_thread_tables_stable.store(stable, std::memory_order_seq_cst); + } + + // Installs the given engines for its lifetime and puts the previous ones + // back on destruction, including when a test assertion returns early. + class EngineSwap { + public: + EngineSwap(Profiler *p, Engine *cpu, Engine *wall) + : _p(p), _cpu(p->_cpu_engine), _wall(p->_wall_engine) { + p->_cpu_engine = cpu; + p->_wall_engine = wall; + } + ~EngineSwap() { + _p->_cpu_engine = _cpu; + _p->_wall_engine = _wall; + } + EngineSwap(const EngineSwap &) = delete; + EngineSwap &operator=(const EngineSwap &) = delete; + + private: + Profiler *_p; + Engine *_cpu; + Engine *_wall; + }; + + // Opens the registration gate for its lifetime and closes it on + // destruction, including when a test assertion returns early. Declare it + // after EngineSwap, so the gate closes before the engines are put back. + class RegistrationGate { + public: + explicit RegistrationGate(Profiler *p) : _p(p) { + p->openThreadRegistration(); + } + ~RegistrationGate() { _p->closeThreadRegistration(); } + RegistrationGate(const RegistrationGate &) = delete; + RegistrationGate &operator=(const RegistrationGate &) = delete; + + private: + Profiler *_p; + }; +}; + +// Polls cond until it holds or a 5 s deadline passes, so a gate regression +// fails the test instead of hanging the gtest binary. +template static bool waitFor(Cond cond) { + auto deadline = std::chrono::steady_clock::now() + std::chrono::seconds(5); + while (!cond()) { + if (std::chrono::steady_clock::now() > deadline) { + return false; + } + std::this_thread::yield(); + } + return true; +} + +// Exposes CTimer's tid-indexed timer table. The table accessors work without +// starting the engine; startWithTableSize() does start it (installs the +// SIGPROF handler and arms a timer for every live thread), and only +// StartOpensThreadRegistration uses it on a real table. +class CTimerTestAccessor { +public: + static bool resizeTimers(int max_timers) { return CTimer::resizeTimers(max_timers); } + static Error sizeTimerTable(int max_timers) { + return CTimer::sizeTimerTable(max_timers); + } + static int maxTimers() { return CTimer::_max_timers; } + static int *timers() { return CTimer::_timers; } + static Error startWithTableSize(CTimer &engine, Arguments &args, + int max_timers) { + return engine.startWithTableSize(args, max_timers); + } + static int registerThread(CTimer &engine, int tid) { + return engine.registerThread(tid); + } + static void unregisterThread(CTimer &engine, int tid) { + engine.unregisterThread(tid); + } + + // Hands the current table to the caller without freeing it, so a test can + // run against an empty table and later reinstate the original one. + static void detach() { + CTimer::_timers = nullptr; + CTimer::_max_timers = 0; + } + + // Frees the test-owned table and reinstates the detached one, without + // touching the gauge (the fixture restores that separately). + static void reattach(int max_timers, int *timers) { + free(CTimer::_timers); + CTimer::_timers = timers; + CTimer::_max_timers = max_timers; + } + + // Sets CTimer's timer signal for its lifetime and restores the previous one. + // It is 0 until a CTimer has started, and timer_create() rejects signal 0. + class SignalSwap { + public: + explicit SignalSwap(int signal) : _signal(CTimer::_signal) { + CTimer::_signal = signal; + } + ~SignalSwap() { CTimer::_signal = _signal; } + SignalSwap(const SignalSwap &) = delete; + SignalSwap &operator=(const SignalSwap &) = delete; + + private: + int _signal; + }; + + // Installs a caller-owned table for its lifetime and puts the previous one + // back on destruction, without freeing either. + class TimerTableSwap { + public: + TimerTableSwap(int *timers, int max_timers) + : _timers(CTimer::_timers), _max_timers(CTimer::_max_timers) { + CTimer::_timers = timers; + CTimer::_max_timers = max_timers; + } + ~TimerTableSwap() { + CTimer::_timers = _timers; + CTimer::_max_timers = _max_timers; + } + TimerTableSwap(const TimerTableSwap &) = delete; + TimerTableSwap &operator=(const TimerTableSwap &) = delete; + + private: + int *_timers; + int _max_timers; + }; +}; + +// Returns the current SIGPROF handler address (sa_handler and sa_sigaction +// share storage), to check whether a call installed a handler. +static void *sigprofHandler() { + struct sigaction sa; + sigaction(SIGPROF, nullptr, &sa); + return (void *)sa.sa_sigaction; +} + +// Size of a timer table with n slots, as recorded under NM_CTIMER. +static long long bytes(int n) { return (long long)((size_t)n * sizeof(int)); } + +class CTimerNativeMemTest : public ::testing::Test { +protected: + int _saved_max; + int *_saved_timers; + long long _saved_live; + + void SetUp() override { + _saved_max = CTimerTestAccessor::maxTimers(); + _saved_timers = CTimerTestAccessor::timers(); + _saved_live = NativeMem::live(NM_CTIMER); + // Detach the existing table so resizeTimers() neither frees it nor + // subtracts its size; the gauge starts from zero for this test. + CTimerTestAccessor::detach(); + NativeMem::setLive(NM_CTIMER, 0); + } + + void TearDown() override { + CTimerTestAccessor::reattach(_saved_max, _saved_timers); + NativeMem::setLive(NM_CTIMER, _saved_live); + } +}; + +// The timer table is reported under NM_CTIMER, and a resize replaces the old +// size rather than adding to it. +TEST_F(CTimerNativeMemTest, TimerTableIsAccounted) { + CTimerTestAccessor::resizeTimers(1024); + ASSERT_NE(nullptr, CTimerTestAccessor::timers()); + EXPECT_EQ(bytes(1024), NativeMem::live(NM_CTIMER)); + + CTimerTestAccessor::resizeTimers(4096); + ASSERT_NE(nullptr, CTimerTestAccessor::timers()); + EXPECT_EQ(bytes(4096), NativeMem::live(NM_CTIMER)); + + CTimerTestAccessor::resizeTimers(512); + EXPECT_EQ(bytes(512), NativeMem::live(NM_CTIMER)); +} + +// Restarting with an unchanged pid_max keeps the table and the gauge as-is. +// The sentinel shows the table was not reallocated: a free + calloc of the +// same size can return the same block, but calloc zeroes it. +TEST_F(CTimerNativeMemTest, SameSizeDoesNotReaccount) { + ASSERT_TRUE(CTimerTestAccessor::resizeTimers(2048)); + int *table = CTimerTestAccessor::timers(); + table[2047] = 0x5a5a5a5a; + + ASSERT_TRUE(CTimerTestAccessor::resizeTimers(2048)); + EXPECT_EQ(table, CTimerTestAccessor::timers()); + EXPECT_EQ(0x5a5a5a5a, CTimerTestAccessor::timers()[2047]); + EXPECT_EQ(bytes(2048), NativeMem::live(NM_CTIMER)); +} + +// A failed allocation leaves an empty table rather than a NULL table with a +// non-zero bound, releases the old table's bytes from the gauge, and a later +// resize to a valid size succeeds. +TEST_F(CTimerNativeMemTest, FailedAllocationLeavesEmptyTable) { + SKIP_IF_CALLOC_OVERFLOW_ABORTS(); + ASSERT_TRUE(CTimerTestAccessor::resizeTimers(1024)); + + EXPECT_FALSE(CTimerTestAccessor::resizeTimers(-1)); + EXPECT_EQ(nullptr, CTimerTestAccessor::timers()); + EXPECT_EQ(0, CTimerTestAccessor::maxTimers()); + EXPECT_EQ(0, NativeMem::live(NM_CTIMER)); + + ASSERT_TRUE(CTimerTestAccessor::resizeTimers(1024)); + EXPECT_NE(nullptr, CTimerTestAccessor::timers()); + EXPECT_EQ(1024, CTimerTestAccessor::maxTimers()); + EXPECT_EQ(bytes(1024), NativeMem::live(NM_CTIMER)); +} + +// start() opens the native-thread registration gate once the table is sized, +// so a thread that starts or exits while start() registers the existing +// threads is handled by the pthread hook instead of being missed. The engine +// is stopped, the gate closed and the SIGPROF disposition restored before the +// fixture restores the table, so later tests see the process as it was. The +// engine is installed as the Profiler's CPU engine while the gate is open, so +// a hook call in that window reaches it instead of a null engine slot. +TEST_F(CTimerNativeMemTest, StartOpensThreadRegistration) { + Profiler *profiler = Profiler::instance(); + ASSERT_FALSE(ProfilerTestAccessor::enginesReady(profiler)); + struct sigaction saved_sigprof; + sigaction(SIGPROF, nullptr, &saved_sigprof); + CTimer engine; + Engine wall; + ProfilerTestAccessor::EngineSwap swap(profiler, &engine, &wall); + Arguments args; + + Error error = CTimerTestAccessor::startWithTableSize(engine, args, + OS::getMaxThreadId()); + bool opened = ProfilerTestAccessor::enginesReady(profiler); + // Ignore SIGPROF before stopping, so a signal an armed timer already + // generated is discarded instead of hitting a restored SIG_DFL. + signal(SIGPROF, SIG_IGN); + engine.stop(); + ProfilerTestAccessor::closeThreadRegistration(profiler); + sigaction(SIGPROF, &saved_sigprof, nullptr); + + EXPECT_FALSE((bool)error); + EXPECT_TRUE(opened); +} + +// start() fails when the timer table cannot be allocated, and returns before +// it installs the SIGPROF handler or arms a timer for any thread. +TEST_F(CTimerNativeMemTest, StartFailsWhenTableAllocationFails) { + SKIP_IF_CALLOC_OVERFLOW_ABORTS(); + CTimer engine; + Arguments args; + void *handler_before = sigprofHandler(); + + Error error = CTimerTestAccessor::startWithTableSize(engine, args, -1); + EXPECT_TRUE((bool)error); + EXPECT_EQ(handler_before, sigprofHandler()); + EXPECT_FALSE(ProfilerTestAccessor::enginesReady(Profiler::instance())); + EXPECT_STREQ("Failed to allocate CPU timer table", error.message()); + EXPECT_EQ(nullptr, CTimerTestAccessor::timers()); + EXPECT_EQ(0, CTimerTestAccessor::maxTimers()); + EXPECT_EQ(0, NativeMem::live(NM_CTIMER)); +} + +// A restart with an unchanged pid_max keeps the table without freezing thread +// unregistration, so a thread that exits during start() still releases its +// timer. A hook call is held in flight: a freeze would wait for it and time +// out, so success shows that none ran. +TEST_F(CTimerNativeMemTest, SameSizeRestartDoesNotFreezeUnregistration) { + Profiler *profiler = Profiler::instance(); + ASSERT_TRUE(CTimerTestAccessor::resizeTimers(64)); + int *table = CTimerTestAccessor::timers(); + ProfilerTestAccessor::DrainBudget budget(100000000LL); + + Error error = Error::OK; + { + ProfilerTestAccessor::HookCallHold hold(profiler); + error = CTimerTestAccessor::sizeTimerTable(64); + } + + EXPECT_FALSE((bool)error); + EXPECT_EQ(table, CTimerTestAccessor::timers()); + EXPECT_EQ(64, CTimerTestAccessor::maxTimers()); + EXPECT_TRUE(ProfilerTestAccessor::threadTablesStable(profiler)); +} + +// A size change replaces the table only under a freeze. While a hook call is +// still in flight after the wait budget, start() fails retryably and keeps the +// old table, and unregistration is open again afterwards. +TEST_F(CTimerNativeMemTest, ResizeKeepsTableWhenHookCallsDoNotDrain) { + Profiler *profiler = Profiler::instance(); + ASSERT_TRUE(CTimerTestAccessor::resizeTimers(64)); + int *table = CTimerTestAccessor::timers(); + ProfilerTestAccessor::DrainBudget budget(100000000LL); + + Error error = Error::OK; + { + ProfilerTestAccessor::HookCallHold hold(profiler); + error = CTimerTestAccessor::sizeTimerTable(128); + } + + EXPECT_TRUE((bool)error); + EXPECT_STREQ("thread hook calls did not drain", + error.message()); + EXPECT_EQ(table, CTimerTestAccessor::timers()); + EXPECT_EQ(64, CTimerTestAccessor::maxTimers()); + EXPECT_EQ(bytes(64), NativeMem::live(NM_CTIMER)); + EXPECT_TRUE(ProfilerTestAccessor::threadTablesStable(profiler)); + EXPECT_FALSE(ProfilerTestAccessor::enginesReady(profiler)); +} + +// With no hook call in flight, a size change replaces the table and reopens +// unregistration. +TEST_F(CTimerNativeMemTest, ResizeReplacesTableWhenHookCallsDrain) { + Profiler *profiler = Profiler::instance(); + ASSERT_TRUE(CTimerTestAccessor::resizeTimers(64)); + + Error error = CTimerTestAccessor::sizeTimerTable(128); + + EXPECT_FALSE((bool)error); + EXPECT_NE(nullptr, CTimerTestAccessor::timers()); + EXPECT_EQ(128, CTimerTestAccessor::maxTimers()); + EXPECT_EQ(bytes(128), NativeMem::live(NM_CTIMER)); + EXPECT_TRUE(ProfilerTestAccessor::threadTablesStable(profiler)); +} + +// A resize frees the old table, which the NM_CTIMER gauge alone cannot show: +// it is decremented whether or not the old table is freed. glibc's +// mallinfo2() reports the bytes the allocator has handed out (heap plus mmap +// chunks), so replacing an N1-slot table with an N2-slot one raises that by +// about (N2 - N1) slots when the old table is freed, and by N2 slots when it +// leaks. The sanitizer allocators and musl do not report through mallinfo2(). +TEST_F(CTimerNativeMemTest, ResizeFreesOldTable) { +#ifndef HAS_GLIBC_MALLINFO2 + GTEST_SKIP() << "needs glibc mallinfo2() without a sanitizer allocator"; +#else + const int old_slots = 1 << 16; + const int new_slots = 1 << 17; + ASSERT_TRUE(CTimerTestAccessor::resizeTimers(old_slots)); + struct mallinfo2 before = mallinfo2(); + ASSERT_TRUE(CTimerTestAccessor::resizeTimers(new_slots)); + struct mallinfo2 after = mallinfo2(); + + long long in_use_before = (long long)(before.uordblks + before.hblkhd); + long long in_use_after = (long long)(after.uordblks + after.hblkhd); + long long grown = in_use_after - in_use_before; + // Halfway between the freed (new - old) and leaked (new) outcomes. + EXPECT_LT(grown, bytes(new_slots) - bytes(old_slots) / 2); +#endif +} + +// Profiler::registerThread() is reached from the pthread hook without +// _state_lock. Until start() has sized the engines' per-thread structures +// (CTimer::resizeTimers()) and set the engines-ready flag, it must return +// without touching the engines. In this binary the profiler is never +// started, so its engine pointers are null and any engine call would crash. +// The call must also leave the in-flight count at zero, or stop() would wait +// on it forever. +// +// Unregistration has its own gate, which stays open while the profiler is +// idle: PerfEvents keeps per-thread fds across stop(), and only +// unregisterThread() closes the fd of a thread that exits after stop(). +TEST(ProfilerEnginesGateTest, RegisterThreadSkipsEnginesUntilReady) { + Profiler *profiler = Profiler::instance(); + ASSERT_FALSE(ProfilerTestAccessor::enginesReady(profiler)); + EXPECT_EQ(-1, Profiler::registerThread(OS::threadId())); + EXPECT_EQ(0, ProfilerTestAccessor::hookCallsInflight(profiler)); + EXPECT_TRUE(ProfilerTestAccessor::threadTablesStable(profiler)); +} + +// A CPU engine whose registerThread() blocks until released, to hold a real +// Profiler::registerThread() call inside the engine. +class BlockingEngine : public Engine { +public: + std::atomic entered{false}; + std::atomic released{false}; + // Thread cancellation state seen inside the call (PTHREAD_CANCEL_*), or -1. + std::atomic cancel_state_inside{-1}; + + int registerThread(int tid) override { + int state; + pthread_setcancelstate(PTHREAD_CANCEL_DISABLE, &state); + pthread_setcancelstate(state, nullptr); + cancel_state_inside.store(state); + entered.store(true); + waitFor([this] { return released.load(); }); + return 0; + } +}; + +// stop() relies on closeThreadRegistration() not returning while a hook call +// that passed the gate is still inside an engine; otherwise CTimer::stop() or +// the next resizeTimers() could run under it. +TEST(ProfilerEnginesGateTest, CloseThreadRegistrationWaitsForHookCalls) { + Profiler *profiler = Profiler::instance(); + BlockingEngine cpu; + Engine wall; + ProfilerTestAccessor::EngineSwap swap(profiler, &cpu, &wall); + ProfilerTestAccessor::RegistrationGate gate(profiler); + + // Well above the 50 ms check below and below BlockingEngine's 5 s hold, so a + // slow sanitizer host does not turn the wait into a false timeout. + ProfilerTestAccessor::DrainBudget budget(4000000000LL); + + std::thread hook([] { Profiler::registerThread(OS::threadId()); }); + bool entered = waitFor([&] { return cpu.entered.load(); }); + std::atomic closed{false}; + std::atomic closed_ok{false}; + std::thread closer([&] { + closed_ok.store(ProfilerTestAccessor::closeThreadRegistration(profiler)); + closed.store(true); + }); + // Wait until the closer has cleared the gate, so it is in the wait loop. + bool gate_closed = + waitFor([&] { return !ProfilerTestAccessor::enginesReady(profiler); }); + std::this_thread::sleep_for(std::chrono::milliseconds(50)); + EXPECT_TRUE(gate_closed); + EXPECT_FALSE(closed.load()); + + cpu.released.store(true); + hook.join(); + closer.join(); + EXPECT_TRUE(closed.load()); + EXPECT_TRUE(closed_ok.load()); + EXPECT_TRUE(entered); + EXPECT_EQ(0, ProfilerTestAccessor::hookCallsInflight(profiler)); +} + +// The wait is bounded: a hook call stuck inside an engine makes +// closeThreadRegistration() report failure after the budget instead of +// hanging stop() under _state_lock. The registration gate stays closed, and +// the unregistration gate is not touched. +TEST(ProfilerEnginesGateTest, CloseThreadRegistrationTimesOutOnStuckHookCall) { + Profiler *profiler = Profiler::instance(); + BlockingEngine cpu; + Engine wall; + ProfilerTestAccessor::EngineSwap swap(profiler, &cpu, &wall); + ProfilerTestAccessor::RegistrationGate gate(profiler); + ProfilerTestAccessor::DrainBudget budget(200000000LL); + + std::thread hook([] { Profiler::registerThread(OS::threadId()); }); + bool entered = waitFor([&] { return cpu.entered.load(); }); + auto begin = std::chrono::steady_clock::now(); + bool ok = ProfilerTestAccessor::closeThreadRegistration(profiler); + long long elapsed_ms = std::chrono::duration_cast( + std::chrono::steady_clock::now() - begin) + .count(); + bool ready = ProfilerTestAccessor::enginesReady(profiler); + int inflight = ProfilerTestAccessor::hookCallsInflight(profiler); + bool stable = ProfilerTestAccessor::threadTablesStable(profiler); + + cpu.released.store(true); + hook.join(); + + EXPECT_TRUE(entered); + EXPECT_FALSE(ok); + EXPECT_FALSE(ready); + EXPECT_EQ(1, inflight); + EXPECT_TRUE(stable); + EXPECT_GE(elapsed_ms, 150); + EXPECT_LT(elapsed_ms, 3000); + EXPECT_EQ(0, ProfilerTestAccessor::hookCallsInflight(profiler)); +} + +// stop() keeps unregistration open, and under thread churn unregistration +// calls can overlap without pause. closeThreadRegistration() therefore waits +// only for registration calls: an unregistration in flight does not make +// stop() time out and put the profiler in ERROR. +TEST(ProfilerEnginesGateTest, CloseThreadRegistrationIgnoresInFlightUnregistration) { + Profiler *profiler = Profiler::instance(); + ProfilerTestAccessor::RegistrationGate gate(profiler); + ProfilerTestAccessor::DrainBudget budget(100000000LL); + + bool closed; + { + ProfilerTestAccessor::UnregisterCallHold hold(profiler); + closed = ProfilerTestAccessor::closeThreadRegistration(profiler); + } + + EXPECT_TRUE(closed); + EXPECT_FALSE(ProfilerTestAccessor::enginesReady(profiler)); + EXPECT_EQ(0, ProfilerTestAccessor::hookCallsInflight(profiler)); +} + +// A table replacement or a CPU engine change must exclude unregistration as +// well, so ThreadTablesFreeze also waits for unregistration calls. +TEST(ProfilerEnginesGateTest, ThreadTablesFreezeWaitsForInFlightUnregistration) { + Profiler *profiler = Profiler::instance(); + ProfilerTestAccessor::DrainBudget budget(100000000LL); + + bool drained = true; + { + ProfilerTestAccessor::UnregisterCallHold hold(profiler); + ProfilerTestAccessor::withThreadTablesFreeze( + profiler, [&](bool d) { drained = d; }); + } + + EXPECT_FALSE(drained); + EXPECT_TRUE(ProfilerTestAccessor::threadTablesStable(profiler)); + EXPECT_EQ(0, ProfilerTestAccessor::hookCallsInflight(profiler)); +} + +// The ThreadTablesFreeze a CPU engine's start() takes around a table +// replacement closes both gates before it waits for hook +// calls already inside an engine, does not return while one is, keeps +// unregisterThread() off the engines while alive, and on destruction reopens +// unregistration but not registration (only a CPU engine that started opens +// that). +TEST(ProfilerEnginesGateTest, ThreadTablesFreezeClosesGatesAndWaitsForHookCalls) { + Profiler *profiler = Profiler::instance(); + BlockingEngine cpu; + Engine wall; + ProfilerTestAccessor::EngineSwap swap(profiler, &cpu, &wall); + ProfilerTestAccessor::RegistrationGate gate(profiler); + ProfilerTestAccessor::DrainBudget budget(4000000000LL); + int tid = OS::threadId(); + + std::thread hook([] { Profiler::registerThread(OS::threadId()); }); + bool entered = waitFor([&] { return cpu.entered.load(); }); + std::atomic in_body{false}; + std::atomic drained_inside{false}; + std::atomic stable_inside{true}; + std::atomic ready_inside{true}; + std::atomic unreg_inside{0}; + std::thread freezer([&] { + ProfilerTestAccessor::withThreadTablesFreeze(profiler, [&](bool drained) { + drained_inside.store(drained); + stable_inside.store(ProfilerTestAccessor::threadTablesStable(profiler)); + ready_inside.store(ProfilerTestAccessor::enginesReady(profiler)); + Profiler::resetUnregisterObservableForTest(); + Profiler::unregisterThread(tid); + unreg_inside.store(Profiler::lastUnregisteredTidForTest()); + in_body.store(true); + }); + }); + bool cleared = + waitFor([&] { return !ProfilerTestAccessor::threadTablesStable(profiler); }); + std::this_thread::sleep_for(std::chrono::milliseconds(50)); + bool in_body_early = in_body.load(); + + cpu.released.store(true); + hook.join(); + freezer.join(); + + EXPECT_TRUE(entered); + EXPECT_TRUE(cleared); + EXPECT_FALSE(in_body_early); + EXPECT_TRUE(in_body.load()); + EXPECT_TRUE(drained_inside.load()); + EXPECT_FALSE(stable_inside.load()); + EXPECT_FALSE(ready_inside.load()); + EXPECT_EQ(-1, unreg_inside.load()); + EXPECT_TRUE(ProfilerTestAccessor::threadTablesStable(profiler)); + EXPECT_FALSE(ProfilerTestAccessor::enginesReady(profiler)); + EXPECT_EQ(0, ProfilerTestAccessor::hookCallsInflight(profiler)); + Profiler::resetUnregisterObservableForTest(); +} + +// A freeze whose wait times out reports it, so start() can fail without +// replacing any table, and its destructor still reopens unregistration. +TEST(ProfilerEnginesGateTest, ThreadTablesFreezeReportsUndrainedOnStuckHookCall) { + Profiler *profiler = Profiler::instance(); + BlockingEngine cpu; + Engine wall; + ProfilerTestAccessor::EngineSwap swap(profiler, &cpu, &wall); + ProfilerTestAccessor::RegistrationGate gate(profiler); + ProfilerTestAccessor::DrainBudget budget(200000000LL); + + std::thread hook([] { Profiler::registerThread(OS::threadId()); }); + bool entered = waitFor([&] { return cpu.entered.load(); }); + bool body_ran = false; + bool drained_inside = true; + ProfilerTestAccessor::withThreadTablesFreeze(profiler, [&](bool drained) { + body_ran = true; + drained_inside = drained; + }); + bool stable_after = ProfilerTestAccessor::threadTablesStable(profiler); + bool ready_after = ProfilerTestAccessor::enginesReady(profiler); + + cpu.released.store(true); + hook.join(); + + EXPECT_TRUE(entered); + EXPECT_TRUE(body_ran); + EXPECT_FALSE(drained_inside); + EXPECT_TRUE(stable_after); + EXPECT_FALSE(ready_after); + EXPECT_EQ(0, ProfilerTestAccessor::hookCallsInflight(profiler)); +} + +// A CPU engine fake: start() optionally opens the registration gate the way +// CTimer/PerfEvents do, and fails or succeeds. Counts enableEvents(true) and +// registerThread() calls. +class FakeCpuEngine : public Engine { +public: + FakeCpuEngine(bool fail, bool open_gate) : _fail(fail), _open_gate(open_gate) {} + std::atomic gate_open_in_start{false}; + std::atomic enable_true_calls{0}; + std::atomic register_calls{0}; + + Error start(Arguments &args) override { + if (_open_gate) { + Profiler::instance()->openThreadRegistration(); + gate_open_in_start.store( + ProfilerTestAccessor::enginesReady(Profiler::instance())); + } + return _fail ? Error("injected CPU engine start failure") : Error::OK; + } + void enableEvents(bool enabled) override { + if (enabled) { + enable_true_calls.fetch_add(1); + } + } + int registerThread(int tid) override { + register_calls.fetch_add(1); + return 0; + } + +private: + bool _fail; + bool _open_gate; +}; + +// A CPU engine that opened the gate and then failed leaves it closed, so the +// hook never reaches an engine that did not start. +TEST(ProfilerEnginesGateTest, FailedCpuEngineStartClosesThreadRegistration) { + Profiler *profiler = Profiler::instance(); + ASSERT_FALSE(ProfilerTestAccessor::enginesReady(profiler)); + FakeCpuEngine cpu(/*fail=*/true, /*open_gate=*/true); + Engine wall; + ProfilerTestAccessor::EngineSwap swap(profiler, &cpu, &wall); + Arguments args; + State state_before = ProfilerTestAccessor::state(profiler); + + bool started = ProfilerTestAccessor::startCpuEngine(profiler, args); + bool ready = ProfilerTestAccessor::enginesReady(profiler); + int registered = Profiler::registerThread(OS::threadId()); + // Restores the closed gate if the call under test regressed. + ProfilerTestAccessor::closeThreadRegistration(profiler); + + EXPECT_FALSE(started); + EXPECT_TRUE(cpu.gate_open_in_start.load()); + EXPECT_FALSE(ready); + EXPECT_EQ(-1, registered); + EXPECT_EQ(0, cpu.register_calls.load()); + EXPECT_EQ(0, ProfilerTestAccessor::hookCallsInflight(profiler)); + EXPECT_TRUE(ProfilerTestAccessor::threadTablesStable(profiler)); + // The gate closed within the budget, so the profiler stays usable. + EXPECT_EQ(state_before, ProfilerTestAccessor::state(profiler)); +} + +// If the gate cannot be closed after the CPU engine failed, a hook call may +// still be inside that engine's table, and only start() replaces tables. The +// profiler goes to ERROR, which rejects every later start(). +TEST(ProfilerEnginesGateTest, StuckGateAfterCpuEngineFailurePutsProfilerInError) { + Profiler *profiler = Profiler::instance(); + ASSERT_FALSE(ProfilerTestAccessor::enginesReady(profiler)); + ProfilerTestAccessor::StateRestore restore(profiler); + FakeCpuEngine cpu(/*fail=*/true, /*open_gate=*/true); + Engine wall; + ProfilerTestAccessor::EngineSwap swap(profiler, &cpu, &wall); + ProfilerTestAccessor::DrainBudget budget(100000000LL); + Arguments args; + + bool started; + { + ProfilerTestAccessor::HookCallHold hold(profiler); + started = ProfilerTestAccessor::startCpuEngine(profiler, args); + } + + EXPECT_FALSE(started); + EXPECT_EQ(ERROR, ProfilerTestAccessor::state(profiler)); + EXPECT_FALSE(ProfilerTestAccessor::enginesReady(profiler)); + EXPECT_EQ(0, ProfilerTestAccessor::hookCallsInflight(profiler)); +} + +// A CPU engine that counts releaseThreads() calls and records the gates it +// saw while releasing. +class ReleasingEngine : public Engine { +public: + std::atomic release_calls{0}; + std::atomic stable_inside{true}; + std::atomic ready_inside{true}; + + void releaseThreads() override { + release_calls.fetch_add(1); + stable_inside.store( + ProfilerTestAccessor::threadTablesStable(Profiler::instance())); + ready_inside.store( + ProfilerTestAccessor::enginesReady(Profiler::instance())); + } +}; + +// stop() leaves an engine's per-thread resources in place (PerfEvents keeps +// its fds open), and only unregisterThread() releases them as threads exit. +// When start() selects another CPU engine, exits no longer reach the previous +// one, so it releases them before the slot changes, with both gates closed. +TEST(ProfilerEnginesGateTest, CpuEngineChangeReleasesPreviousEngineThreads) { + Profiler *profiler = Profiler::instance(); + ReleasingEngine previous; + FakeCpuEngine next(/*fail=*/false, /*open_gate=*/false); + Engine wall; + ProfilerTestAccessor::EngineSwap swap(profiler, &previous, &wall); + + bool replaced = ProfilerTestAccessor::replaceCpuEngine(profiler, &next); + + EXPECT_TRUE(replaced); + EXPECT_EQ(1, previous.release_calls.load()); + EXPECT_FALSE(previous.stable_inside.load()); + EXPECT_FALSE(previous.ready_inside.load()); + EXPECT_EQ(&next, ProfilerTestAccessor::cpuEngine(profiler)); + EXPECT_TRUE(ProfilerTestAccessor::threadTablesStable(profiler)); + EXPECT_EQ(0, ProfilerTestAccessor::hookCallsInflight(profiler)); +} + +// Restarting with the same CPU engine keeps its per-thread resources, so +// PerfEvents reuses its fds across stop()/start(). +TEST(ProfilerEnginesGateTest, SameCpuEngineKeepsItsThreads) { + Profiler *profiler = Profiler::instance(); + ReleasingEngine engine; + Engine wall; + ProfilerTestAccessor::EngineSwap swap(profiler, &engine, &wall); + + bool replaced = ProfilerTestAccessor::replaceCpuEngine(profiler, &engine); + + EXPECT_TRUE(replaced); + EXPECT_EQ(0, engine.release_calls.load()); + EXPECT_EQ(&engine, ProfilerTestAccessor::cpuEngine(profiler)); +} + +// If hook calls do not drain, the engine change is refused: the previous +// engine keeps its resources and its slot, and start() fails retryably. +TEST(ProfilerEnginesGateTest, CpuEngineChangeKeepsPreviousEngineWhenHookCallsDoNotDrain) { + Profiler *profiler = Profiler::instance(); + ReleasingEngine previous; + FakeCpuEngine next(/*fail=*/false, /*open_gate=*/false); + Engine wall; + ProfilerTestAccessor::EngineSwap swap(profiler, &previous, &wall); + ProfilerTestAccessor::DrainBudget budget(100000000LL); + + bool replaced; + { + ProfilerTestAccessor::HookCallHold hold(profiler); + replaced = ProfilerTestAccessor::replaceCpuEngine(profiler, &next); + } + + EXPECT_FALSE(replaced); + EXPECT_EQ(0, previous.release_calls.load()); + EXPECT_EQ(&previous, ProfilerTestAccessor::cpuEngine(profiler)); + EXPECT_TRUE(ProfilerTestAccessor::threadTablesStable(profiler)); +} + +// A CPU engine without a per-thread table (ITimer-like) never opens the gate +// itself; startCpuEngine() opens it once the engine started, so the hook +// reaches it. +TEST(ProfilerEnginesGateTest, StartedCpuEngineOpensThreadRegistration) { + Profiler *profiler = Profiler::instance(); + ASSERT_FALSE(ProfilerTestAccessor::enginesReady(profiler)); + FakeCpuEngine cpu(/*fail=*/false, /*open_gate=*/false); + Engine wall; + ProfilerTestAccessor::EngineSwap swap(profiler, &cpu, &wall); + Arguments args; + + bool started = ProfilerTestAccessor::startCpuEngine(profiler, args); + bool ready = ProfilerTestAccessor::enginesReady(profiler); + Profiler::registerThread(OS::threadId()); + int registered = cpu.register_calls.load(); + bool closed_ok = ProfilerTestAccessor::closeThreadRegistration(profiler); + + EXPECT_TRUE(started); + EXPECT_TRUE(ready); + EXPECT_EQ(1, registered); + EXPECT_TRUE(closed_ok); + EXPECT_EQ(0, ProfilerTestAccessor::hookCallsInflight(profiler)); +} + +// The CPU engine is enabled only if its own start() succeeded, even when +// another mode activated. +TEST(ProfilerEnginesGateTest, CpuEngineEnabledOnlyIfItStarted) { + Profiler *profiler = Profiler::instance(); + FakeCpuEngine cpu(/*fail=*/true, /*open_gate=*/false); + Engine wall; + ProfilerTestAccessor::EngineSwap swap(profiler, &cpu, &wall); + Arguments args; + + bool started = ProfilerTestAccessor::startCpuEngine(profiler, args); + ProfilerTestAccessor::closeThreadRegistration(profiler); + int activated = EM_WALL | (started ? EM_CPU : 0); + ProfilerTestAccessor::enableCpuEngine(profiler, activated); + EXPECT_FALSE(started); + EXPECT_EQ(0, cpu.enable_true_calls.load()); + + ProfilerTestAccessor::enableCpuEngine(profiler, EM_CPU | EM_WALL); + EXPECT_EQ(1, cpu.enable_true_calls.load()); +} + +// With the gate closed registerThread() returns before reaching the engine; +// with it open the call reaches the engine and returns its result. The +// in-flight count is back at zero afterwards either way. +TEST(ProfilerEnginesGateTest, RegisterThreadReachesEngineOnlyWhenGateOpen) { + Profiler *profiler = Profiler::instance(); + ASSERT_FALSE(ProfilerTestAccessor::enginesReady(profiler)); + FakeCpuEngine cpu(/*fail=*/false, /*open_gate=*/false); + Engine wall; + ProfilerTestAccessor::EngineSwap swap(profiler, &cpu, &wall); + + EXPECT_EQ(-1, Profiler::registerThread(OS::threadId())); + EXPECT_EQ(0, cpu.register_calls.load()); + EXPECT_EQ(0, ProfilerTestAccessor::hookCallsInflight(profiler)); + { + ProfilerTestAccessor::RegistrationGate gate(profiler); + EXPECT_EQ(0, Profiler::registerThread(OS::threadId())); + EXPECT_EQ(1, cpu.register_calls.load()); + EXPECT_EQ(0, ProfilerTestAccessor::hookCallsInflight(profiler)); + } + EXPECT_FALSE(ProfilerTestAccessor::enginesReady(profiler)); +} + +// A wall engine that counts registerThread() calls. +class CountingWallEngine : public Engine { +public: + std::atomic register_calls{0}; + + int registerThread(int tid) override { + register_calls.fetch_add(1); + return 0; + } +}; + +// The registration gate is open iff the CPU engine started, so it admits only +// the CPU engine. A wall engine never receives the hook through it: it would +// otherwise be called whenever the CPU engine started, even if the wall engine +// did not, and a wall engine that opened the gate itself would admit a CPU +// engine that failed. +TEST(ProfilerEnginesGateTest, RegistrationGateAdmitsOnlyCpuEngine) { + Profiler *profiler = Profiler::instance(); + ASSERT_FALSE(ProfilerTestAccessor::enginesReady(profiler)); + FakeCpuEngine cpu(/*fail=*/false, /*open_gate=*/false); + CountingWallEngine wall; + ProfilerTestAccessor::EngineSwap swap(profiler, &cpu, &wall); + + int result; + { + ProfilerTestAccessor::RegistrationGate gate(profiler); + result = Profiler::registerThread(OS::threadId()); + } + + EXPECT_EQ(0, result); + EXPECT_EQ(1, cpu.register_calls.load()); + EXPECT_EQ(0, wall.register_calls.load()); + EXPECT_EQ(0, ProfilerTestAccessor::hookCallsInflight(profiler)); +} + +// A negative tid must not index before CTimer's table. The page before the +// table is inaccessible, so an access through _timers[-1] faults. +TEST(CTimerTableTest, NegativeTidIsRejected) { + size_t page_size = (size_t)sysconf(_SC_PAGESIZE); + char *base = (char *)mmap(nullptr, 2 * page_size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + ASSERT_NE(MAP_FAILED, (void *)base); + ASSERT_EQ(0, mprotect(base, page_size, PROT_NONE)); + int *table = (int *)(base + page_size); + { + CTimerTestAccessor::TimerTableSwap swap(table, 16); + CTimer engine; + // unregisterThread() first: registerThread() may fail earlier, in + // timer_create, so this is the discriminating call. + CTimerTestAccessor::unregisterThread(engine, -1); + EXPECT_EQ(-1, CTimerTestAccessor::registerThread(engine, -1)); + for (int i = 0; i < 16; i++) { + EXPECT_EQ(0, table[i]) << "slot " << i; + } + } + munmap(base, 2 * page_size); +} + +// A tid equal to the bound must not index one past CTimer's table. The table +// is sized to the calling thread's own tid, so that tid is one past its end, +// and the page after the table is inaccessible. The tid is live and the timer +// signal is valid, so a registerThread() that wrongly passed the bound would +// get past timer_create() and fault on the CAS into the guard page. +TEST(CTimerTableTest, TidAtBoundIsRejected) { + size_t page_size = (size_t)sysconf(_SC_PAGESIZE); + int tid = OS::threadId(); + ASSERT_GT(tid, 0); + size_t table_bytes = (size_t)tid * sizeof(int); + size_t table_pages = (table_bytes + page_size - 1) / page_size; + size_t map_bytes = (table_pages + 1) * page_size; + char *base = (char *)mmap(nullptr, map_bytes, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + ASSERT_NE(MAP_FAILED, (void *)base); + char *guard = base + table_pages * page_size; + ASSERT_EQ(0, mprotect(guard, page_size, PROT_NONE)); + int *table = (int *)(guard - table_bytes); + { + CTimerTestAccessor::TimerTableSwap swap(table, tid); + CTimerTestAccessor::SignalSwap signal(SIGPROF); + CTimer engine; + CTimerTestAccessor::unregisterThread(engine, tid); + EXPECT_EQ(-1, CTimerTestAccessor::registerThread(engine, tid)); + EXPECT_EQ(0, table[tid - 1]); + } + munmap(base, map_bytes); +} + +// Thread body for the cancellation test: makes a hook call with asynchronous +// cancellation enabled, then reaches a cancellation point. A plain C-style +// thread is used so that no C++ frame with cleanups sits above the call when +// the cancellation unwinds it. +static std::atomic survived_cancellation{false}; +static void *registerWithAsyncCancel(void *) { + pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, nullptr); + Profiler::registerThread(OS::threadId()); + pthread_testcancel(); + survived_cancellation.store(true); + return nullptr; +} + +// A cancellation requested during a hook call stays pending until the call has +// left every destructor-bearing scope. Current glibc acts on a pending +// asynchronous request as soon as cancellation is re-enabled, so re-enabling it +// from a destructor would unwind out of an implicitly noexcept function and +// terminate the process. Older glibc (2.35) keeps it pending until the next +// cancellation point instead. Either way the process must survive, the thread +// must not run past that cancellation point, and the in-flight count must be +// back at zero. The join value is not checked: a forced unwind that crosses +// differently versioned libgcc copies (see run_with_cleanup() in +// libraryPatcher_linux.cpp) does not always report PTHREAD_CANCELED. +TEST(ProfilerEnginesGateTest, CancellationDuringHookCallActsAfterTheCall) { +#if defined(TSAN_ENABLED) + GTEST_SKIP() << "TSan does not support asynchronous thread cancellation"; +#endif + Profiler *profiler = Profiler::instance(); + BlockingEngine cpu; + Engine wall; + ProfilerTestAccessor::EngineSwap swap(profiler, &cpu, &wall); + ProfilerTestAccessor::RegistrationGate gate(profiler); + + survived_cancellation.store(false); + pthread_t thread; + ASSERT_EQ(0, pthread_create(&thread, nullptr, registerWithAsyncCancel, + nullptr)); + bool entered = waitFor([&] { return cpu.entered.load(); }); + // Cancellation is disabled inside the hook call, so this stays pending. + EXPECT_EQ(0, pthread_cancel(thread)); + cpu.released.store(true); + EXPECT_EQ(0, pthread_join(thread, nullptr)); + + EXPECT_TRUE(entered); + // The hook call ran with cancellation disabled; without that, the request + // could act at a cancellation point inside the engine and unwind through + // ~HookCallScope(). + EXPECT_EQ(PTHREAD_CANCEL_DISABLE, cpu.cancel_state_inside.load()); + EXPECT_EQ(0, ProfilerTestAccessor::hookCallsInflight(profiler)); + EXPECT_FALSE(survived_cancellation.load()); +} + +// Unregistration is skipped, without touching the engines, while start() may +// replace their per-thread tables, and goes through once the tables are +// stable. In this binary the call records the tid instead of calling the +// (null) engines, so the recorded tid shows whether it passed the gate. +TEST(ProfilerEnginesGateTest, UnregisterThreadSkipsEnginesWhileTablesFrozen) { + Profiler *profiler = Profiler::instance(); + int tid = OS::threadId(); + + ProfilerTestAccessor::setThreadTablesStable(profiler, false); + Profiler::resetUnregisterObservableForTest(); + Profiler::unregisterThread(tid); + EXPECT_EQ(-1, Profiler::lastUnregisteredTidForTest()); + EXPECT_EQ(0, ProfilerTestAccessor::hookCallsInflight(profiler)); + + ProfilerTestAccessor::setThreadTablesStable(profiler, true); + Profiler::unregisterThread(tid); + EXPECT_EQ(tid, Profiler::lastUnregisteredTidForTest()); + Profiler::resetUnregisterObservableForTest(); +} + +#endif // __linux__ diff --git a/ddprof-lib/src/test/cpp/perfEvents_ut.cpp b/ddprof-lib/src/test/cpp/perfEvents_ut.cpp index 59bb7c6915..477584ed24 100644 --- a/ddprof-lib/src/test/cpp/perfEvents_ut.cpp +++ b/ddprof-lib/src/test/cpp/perfEvents_ut.cpp @@ -53,4 +53,15 @@ TEST(PerfEventsCompareTidsTest, SortsCreationOrderedTidsWithReuse) { } } +// A failed table allocation leaves _events NULL and _max_events -1, which is +// also the state before the first start(). Fds left armed by an earlier +// recording keep delivering SIGPROF, and the handler calls resetBuffer() on +// every exit, so resetBuffer() must not index the missing table. This checks +// only that no crash happens. Checking that releaseThreads() closes real fds +// needs slot setters in perfEvents_linux.cpp, which this binary does not have +// (PROF-16359). +TEST(PerfEventsTableTest, MissingTableIsNotIndexed) { + PerfEvents::resetBuffer(5); +} + #endif // __linux__ diff --git a/ddprof-test/src/test/java/com/datadoghq/profiler/CheckCommandTest.java b/ddprof-test/src/test/java/com/datadoghq/profiler/CheckCommandTest.java new file mode 100644 index 0000000000..451bbde46d --- /dev/null +++ b/ddprof-test/src/test/java/com/datadoghq/profiler/CheckCommandTest.java @@ -0,0 +1,40 @@ +/* + * Copyright 2026, Datadog, Inc. + * SPDX-License-Identifier: Apache-2.0 + */ + +package com.datadoghq.profiler; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import org.junit.jupiter.api.Test; + +/** + * The check command validates a profiler command without starting anything. + * While a recording runs it is rejected, and the recording must carry on. After + * stop() it validates the command and leaves the profiler idle. + */ +public class CheckCommandTest extends AbstractProfilerTest { + + @Override + protected String getProfilerCommand() { + return "cpu=1ms"; + } + + @Test + public void checkIsRejectedWhileRecording() throws Exception { + IllegalStateException e = assertThrows(IllegalStateException.class, + () -> profiler.execute("check,cpu=1ms")); + assertEquals("Profiler already started", e.getMessage()); + assertTrue(profiler.execute("status").contains("Profiling is running")); + } + + @Test + public void checkSucceedsWhenIdle() throws Exception { + stopProfiler(); + assertTrue(profiler.execute("check,cpu=1ms").startsWith("OK")); + assertTrue(profiler.execute("status").contains("Profiler is not active")); + } +} diff --git a/doc/README.md b/doc/README.md index 39d51ddf17..a6b20560f4 100644 --- a/doc/README.md +++ b/doc/README.md @@ -22,7 +22,7 @@ All documentation files use **PascalCase** naming (e.g., `BuildSystemGuide.md`). - [JfrIntegration](architecture/JfrIntegration.md) - Native JFR writer: recordings, buffers, metadata, constant pools - [LiveHeapReferenceChainsImplementation](architecture/LiveHeapReferenceChainsImplementation.md) - Current reference-chain implementation - [NativeMemoryProfiling](architecture/NativeMemoryProfiling.md) - malloc interception and allocation sampling -- [ProfilingEngines](architecture/ProfilingEngines.md) - CPU, wall-clock, allocation, native-memory and liveness engines +- [ProfilingEngines](architecture/ProfilingEngines.md) - CPU, wall-clock, allocation, native-memory and liveness engines; thread-registration gates and in-flight drain - [RefCountGuard](architecture/RefCountGuard.md) - Lock-free RAII reference-counting primitive used to drain readers before resource reclamation - [ReferenceChainsSignalsExplained](architecture/ReferenceChainsSignalsExplained.md) - Reference-chain scheduling and cost control - [JvmSupportLayer](architecture/JvmSupportLayer.md) - JVM-implementation abstraction (HotSpot, J9, Zing) diff --git a/doc/architecture/JfrIntegration.md b/doc/architecture/JfrIntegration.md index aec55b2abb..f25275b993 100644 --- a/doc/architecture/JfrIntegration.md +++ b/doc/architecture/JfrIntegration.md @@ -62,10 +62,10 @@ event silently. (`flightRecorder.h:84`, `283`). They are indexed by the profiler's lock stripe, not by thread: -1. `Profiler::getLockIndex(tid)` hashes the thread ID to a stripe - (`profiler.cpp:229-234`). +1. `Profiler::getLockIndex(tid)` hashes the thread ID to a stripe. 2. The caller `tryLock()`s that stripe, then up to two others; if all three are - taken the sample is dropped (`profiler.cpp:679-682`). + taken the sample is dropped (`Profiler::recordSample()` and the other + `record*()` methods). 3. With the stripe lock held it calls `FlightRecorder::recordEvent(lock_index, ...)`, which writes into `_buf[lock_index]` (`flightRecorder.cpp:2866-2907`). @@ -138,7 +138,8 @@ events write the context snapshot taken when the allocation was tracked - `wallClockEpoch()`, `recordTraceRoot()`, `recordQueueTime()`, `recordDatadogSetting()`, `recordHeapUsage()` and the reference-chain writers use `_buf[lock_index]`; their `Profiler` callers take the stripe lock first - (`profiler.cpp:788-894`, `1854-1866`). + (`Profiler::writeDatadogProfilerSetting()`, `writeHeapUsage()`, and the + reference-chain drains in `dump()` and `stop()`). - `recordLog()` builds the event in an `alloca` buffer and writes it to the file at once, without a stripe lock (`flightRecorder.cpp:2945-2966`). - `cpuMonitorCycle()` writes `jdk.CPULoad` into `_cpu_monitor_buf`, which @@ -195,12 +196,12 @@ properties and native libraries can be turned off with `NO_SYSTEM_INFO`, ### How `Profiler` drives it -`Profiler::dump()` (`profiler.cpp:2024-2096`) first emits pending +`Profiler::dump()` first emits pending reference-chain events under a stripe lock, refreshes libraries and thread names, then runs `_jfr.dump()` inside `rotateDictsAndRun()`. `Profiler::stop()` -runs `_jfr.stop()` the same way (`profiler.cpp:1946`). +runs `_jfr.stop()` the same way. -`rotateDictsAndRun()` (`profiler.h:216-239`): +`rotateDictsAndRun()` (`profiler.h:245-268`): 1. Blocks profiling signals on the dumping thread (`SignalBlocker`). 2. Rotates `_class_map`, `_string_label_map` and `_context_value_map`, so each @@ -257,7 +258,7 @@ static string table filled by `getId()` (`jfrMetadata.h:114-122`); `Profiler::start()` calls `JfrMetadata::reset()` then `initialize()` with `args._context_attributes`, truncated to `DD_TAGS_CAPACITY`, before -`_jfr.start()` (`profiler.cpp:1657-1665`). `reset()` deletes the tree and +`_jfr.start()`. `reset()` deletes the tree and re-registers `"root"` as string 0 so the root element's name stays valid (`jfrMetadata.cpp:43-60`). Both require `Profiler::_state_lock` and must not run while engines are active (`jfrMetadata.h:276-284`). @@ -373,7 +374,7 @@ turns this off; it is on by default (`flightRecorder.cpp:1244-1291`, `writeStackTraces()` (`flightRecorder.cpp:1925-1979`) is the only consumer of stored traces. It calls `Profiler::processCallTraces()`, which hands the lambda to `CallTraceStorage::processTraces()`, or an empty set when stack traces are -omitted (`profiler.h:346-354`). Rotation, liveness preservation and the reserved +omitted (`profiler.h:377-385`). Rotation, liveness preservation and the reserved trace IDs are covered in [CallTraceStorage](CallTraceStorage.md). The callback runs inside `processTraces()`, so the `CallTrace` pointers stay valid while it runs. diff --git a/doc/architecture/JvmSupportLayer.md b/doc/architecture/JvmSupportLayer.md index 8a0ecccac3..583786ea43 100644 --- a/doc/architecture/JvmSupportLayer.md +++ b/doc/architecture/JvmSupportLayer.md @@ -83,11 +83,11 @@ predicates. More detection-dependent setup: - `can_retransform_any_class` is disabled on J9 (`vmEntry.cpp:502`). - The JFR `RequestStackTrace` extension is probed only on HotSpot (`vmEntry.cpp:519-521`). - The profiler installs its SIGSEGV/SIGBUS handlers only on HotSpot and J9 - (`profiler.cpp:1168`). -- On Zing, `GC_TRACES | LAST_JAVA_PC` is forced into `_safe_mode` (`profiler.cpp:1587`). + (`Profiler::setupSignalHandlers()`). +- On Zing, `GC_TRACES | LAST_JAVA_PC` is forced into `_safe_mode` (`Profiler::start()`). `JVMSupport::initialize()` (`jvmSupport.cpp:32-57`) runs after detection, from -`Profiler::start()`/`Profiler::init()` (`profiler.cpp:1417`, `profiler.cpp:1437`). +`Profiler::start()`/`Profiler::init()`. It calls `JVMThread::initialize()`, then checks the `ProfiledThread` TLS key. If either step fails, profiling is disabled. @@ -116,7 +116,7 @@ gets HotSpot declarations, but it should still call only `JVMSupport::*`. Callers in shared code: -- Stack walking calls `walkJavaStack` (`profiler.cpp:717`). +- Stack walking calls `walkJavaStack` (`Profiler::recordSample()`). - `stackWalker.cpp:80-82` and `perfEvents_linux.cpp` call `isJitCode` and `canUnwind`. Both are signal-handler paths, and on non-HotSpot VMs they reduce to constant returns. @@ -183,7 +183,7 @@ Caveats: (for example, a build without JFR), `initLazyResolution()` fails, and `initExecution()` falls back to `Partial_loaded` (`jvmSupport.cpp:86-91`). - **`cstack=vm` needs `VMStructs`.** If `VMStructs::hasStackStructs()` is false, - `cstack=vm` is downgraded to DWARF or `no` (`profiler.cpp:1625-1629`). + `cstack=vm` is downgraded to DWARF or `no` (`Profiler::start()`). ### `j9/` @@ -196,7 +196,7 @@ Caveats: - **The ASGCT-or-JVMTI choice is made per engine.** `Profiler::selectCpuEngine` and `selectWallEngine` choose `J9WallClock` (the JVMTI path) when any of these - holds (`profiler.cpp:1274-1285`, `profiler.cpp:1321-1333`): + holds: - `J9Support::shouldUseAsgct()` is false. The system property `dd.profiling.ddprof.j9.sampler` set to `jvmti` makes it false (`j9/j9Support.h:108-126`). @@ -223,7 +223,7 @@ behavior is shared code that branches on `VM::isZing()`: - Java stacks are collected with ASGCT (`jvmSupport.cpp:112-117`). - Runtime entries are marked (`vmEntry.cpp:378-379`). -- `_safe_mode` is forced (`profiler.cpp:1587`). +- `_safe_mode` is forced (`Profiler::start()`). Caveat: `JVMThread::nativeThreadId()` sends Zing to `VMThread::nativeThreadId()`. That function returns -1 unless diff --git a/doc/architecture/ProfilingEngines.md b/doc/architecture/ProfilingEngines.md index e40fc6f6df..ea0c3fa5cb 100644 --- a/doc/architecture/ProfilingEngines.md +++ b/doc/architecture/ProfilingEngines.md @@ -46,7 +46,7 @@ the header and no `CpuEngine::` member is defined anywhere. `PerfEvents`, ### Engine slots in `Profiler` -`profiler.cpp:73-82` declares each engine as a file-static singleton: +The top of `profiler.cpp` declares each engine as a file-static singleton: `noop_engine` (a bare `Engine`, used when a mode is off), `malloc_tracer`, `perf_events`, `wall_asgct_engine`, `wall_jvmti_engine`, `j9_engine`, `itimer`, `itimer_jvmti`, `ctimer`, and `ctimer_jvmti`. `ObjectSampler` is a @@ -55,9 +55,9 @@ separate singleton (`ObjectSampler::instance()`, | Slot | Chosen by | Notes | |------|-----------|-------| -| `_cpu_engine` | `selectCpuEngine()` (`profiler.cpp:1269`) | Receives `registerThread`/`unregisterThread` and `enableEvents` | -| `_wall_engine` | `selectWallEngine()` (`profiler.cpp:1316`) | Same as above | -| `_alloc_engine` | `selectAllocEngine()` (`profiler.cpp:1351`) | `ObjectSampler` if `VM::canSampleObjects()`, else `noop_engine` | +| `_cpu_engine` | `selectCpuEngine()` | Receives `registerThread`/`unregisterThread` and `enableEvents` | +| `_wall_engine` | `selectWallEngine()` | Receives `unregisterThread` and `enableEvents`, not `registerThread`: the registration gate admits only the CPU engine (see Thread registration) | +| `_alloc_engine` | `selectAllocEngine()` | `ObjectSampler` if `VM::canSampleObjects()`, else `noop_engine` | `MallocTracer` has no slot. `Profiler::start()`/`stop()` call `malloc_tracer` directly. `LivenessTracker` and `NativeMem` are not `Engine`s. @@ -65,11 +65,26 @@ directly. `LivenessTracker` and `NativeMem` are not `Engine`s. ## Lifecycle ``` -Profiler::start() profiler.cpp:1453 - selectCpuEngine / selectWallEngine +Profiler::start() + _engines_ready cleared registration closed until the CPU engine + starts; unregistration stays open + selectCpuEngine / selectWallEngine engine pointers stored with release + replaceCpuEngine() on an engine change, the previous engine + releases its threads under + ThreadTablesFreeze; timeout -> retryable error _jfr.start() SamplerPerf::primeClock() before any handler can fire - _cpu_engine->start() (EM_CPU) failure -> warning, continue + Profiler::startCpuEngine() (EM_CPU) failure -> warning, continue + _cpu_engine->start() + CTimer: ThreadTablesFreeze only if the table size changes; + on a wait timeout (1 s) it fails and keeps its + table (ERROR below if a registration call is stuck) + PerfEvents: replaces its table without the freeze (PROF-16359); + allocation failure -> error, engine stays off + CTimer/PerfEvents: openThreadRegistration() after sizing, before listing threads + on failure: closeThreadRegistration(); timeout -> ERROR, start() undoes + its setup and fails + on success: openThreadRegistration() no-op for CTimer/PerfEvents _wall_engine->enableEvents(true) BEFORE start(): see below _wall_engine->start() (EM_WALL) _alloc_engine = selectAllocEngine(); start() (EM_ALLOC) @@ -77,38 +92,61 @@ Profiler::start() profiler.cpp:1453 NativeSocketSampler::start() (EM_NATIVESOCKET) if anything activated: switchThreadEvents(JVMTI_ENABLE) - _cpu_engine->enableEvents(true) LAST + Profiler::enableCpuEngine() LAST, only if EM_CPU activated ``` - The wall engine is enabled before its thread starts because `timerLoopCommon()` checks `_enabled` once on entry and returns if it is - false (`wallClock.h:50`, `profiler.cpp:1690-1696`). -- The CPU engine is enabled last (`profiler.cpp:1791`). CPU handlers check - `_enabled` on every signal, so nothing is lost by enabling late. + false (`wallClock.h:50`, the EM_WALL branch of `Profiler::start()`). +- The CPU engine is enabled last (`Profiler::enableCpuEngine()`). CPU handlers check + `_enabled` on every signal, so nothing is lost by enabling late. It is not + enabled when its own `start()` failed, even if another mode activated, so fds + or timers left armed by an earlier recording cannot emit CPU samples. - A failing CPU, wall, or alloc `start()` is logged and the remaining modes still start. Start fails only if no mode activated. -`Profiler::stop()` (`profiler.cpp:1811`) runs in reverse: +`Profiler::stop()` runs in reverse: 1. `disableEngines()` calls `enableEvents(false)` on the CPU and wall engines - only (`profiler.cpp:1004`). + only. 2. `SignalInflight::drain()` waits until every handler that already passed its `_enabled` check has left. If the drain times out, `stop()` returns an error - and tears nothing down (`profiler.cpp:1824`). -3. Then, in order: `_alloc_engine->stop()`, reference-chain teardown, + and tears nothing down. +3. `closeThreadRegistration()` clears `_engines_ready` and waits until no + registration call that passed the check is still inside an engine. + No `registerThread()` call (pthread hook or JVMTI `ThreadStart`) can then arm + a timer or open an fd after the engines stop, or still be using a table when + the next `start()` resizes it. The wait is bounded (1 s). On timeout `stop()` + logs an error, still completes every step below, and ends in `ERROR` instead + of `IDLE`. That is safe because no step frees a per-thread table, and + `ERROR` rejects every later `start()`, the only place a table is replaced, + so the stuck call can never reach freed memory. The cost: if the stuck call + is inside `CTimer::registerThread()` and finishes after `CTimer::stop()`, + its timer stays armed for good; its `SIGPROF` is dropped by the disabled + handler. + Unregistration is not closed here: JVMTI `ThreadEnd` and native thread exits + may still reach `unregisterThread()` while the engines stop. `CTimer::stop()` + and `CTimer::unregisterThread()` each release a slot with a per-slot CAS and + neither frees the table; `PerfEvents::stop()` is a no-op and keeps its fds, + which only `unregisterThread()` closes; the wall engines and `ITimer` have + no `unregisterThread()`. Only a CPU engine's `start()` replaces its table: + `CTimer` under `ThreadTablesFreeze`, `PerfEvents` without it (PROF-16359). +4. Then, in order: `_alloc_engine->stop()`, reference-chain teardown, `malloc_tracer.stop()`, the socket sampler, `_wall_engine->stop()`, `_cpu_engine->stop()`. -`Profiler::check()` (`profiler.cpp:1957`) runs the same selection and calls -each engine's `check()` without starting anything. +`Profiler::check()` runs the same selection and calls each engine's `check()` +without starting anything. It runs only while the profiler is `IDLE` +(`checkState()` rejects a `RUNNING` profiler) and selects into locals, so it +never replaces the engine slots that thread exits still route to. ### Thread registration Per-thread engines (`CTimer`, `PerfEvents`) learn about threads in three ways: 1. **JVMTI** `ThreadStart`/`ThreadEnd` reach `Profiler::onThreadStart()`/ - `onThreadEnd()`, which call `_cpu_engine->registerThread(tid)` and - `_wall_engine->registerThread(tid)` (`profiler.cpp:111-112`, `143-144`). + `onThreadEnd()`, which call `Profiler::registerThread()`/ + `unregisterThread()`, so they share the gates described in item 2. When the thread's `ProfiledThread` still exists at thread end, unregistration runs with profiling signals blocked (`blockProfilingForExit()`). @@ -116,10 +154,65 @@ Per-thread engines (`CTimer`, `PerfEvents`) learn about threads in three ways: wraps the start routine (`start_routine_wrapper()`) and calls `Profiler::registerThread()` (`libraryPatcher_linux.cpp:553`). On exit, `unregister_and_release()` calls `Profiler::unregisterThread()` - (`libraryPatcher_linux.cpp:249`). + (`libraryPatcher_linux.cpp:249`). The hook runs without `_state_lock` + and is installed before the engines start, so both calls are gated: + - `registerThread()` needs `_engines_ready`, which `CTimer::start()` and + `PerfEvents::start()` set via `Profiler::openThreadRegistration()` once + their table is sized and before they list threads, and which + `Profiler::startCpuEngine()` sets once any CPU engine (including the + table-less `ITimer`, `ITimerJvmti` and J9 engines) has started. It is + cleared by `start()`, by `stop()`, by a failed CPU engine start and by + `ThreadTablesFreeze`, so the gate is open iff the CPU engine started. + Wall-only, noop and failed-CPU recordings leave it closed. Because the + gate tracks the CPU engine, it admits only that engine: + `registerThread()` never calls the wall engine. No wall engine implements + `registerThread()`; one that ever does needs a gate of its own, because + opening this one would also admit a CPU engine that failed. While the + gate is closed, `registerThread()` returns before it touches the + in-flight counter. + - `PerfEvents::start()`'s stale-fd sweep may close the fd of a tid the hook + registered after the thread list was taken, and such a thread is then not + sampled for the recording (PROF-16359). + - `unregisterThread()` needs `_thread_tables_stable`, which is cleared only + while `CTimer::sizeTimerTable()` replaces the timer table + (`ThreadTablesFreeze`), that is, when the table size changes: the first + start, or `pid_max` changed. A restart that keeps the table never clears + it. `PerfEvents::start()` replaces its table without clearing it + (PROF-16359). An exit that arrives while the gate is closed is + dropped, and if the thread is still listed when the engine lists + threads, its slot stays occupied after it dies (PROF-16359). The gate + stays set while idle, because `PerfEvents::stop()` keeps per-thread fds + open and this call closes the fd of a thread that exits after `stop()`. + - `start()` replaces `_cpu_engine`/`_wall_engine` while unregistration is + open, so it stores the pointers with release and the hook paths load them + with acquire. An unregistration that reaches the previous engine only + releases that thread's own slot. `check()` never writes the slots. 3. **`PerfEvents` only**: a `pthread_setspecific` GOT hook on the JVM library - (`perfEvents_linux.cpp:176`), installed in `PerfEvents::start()` - (`perfEvents_linux.cpp:980`). + (`pthread_setspecific_hook()`), installed in `PerfEvents::start()`. + +Every `Profiler::registerThread()`/`unregisterThread()` call counts itself in +flight (`HookCallScope`) for as long as it may be inside an engine, on its +gate's own counter (`_register_calls_inflight`, `_unregister_calls_inflight`). +`closeThreadRegistration()` waits, at most 1 s, only for registration calls, so +no `registerThread()` call is inside an engine once `stop()` tears it down. +Unregistration calls are not waited for there: they stay allowed during `stop()`, +and under thread churn their count may not reach zero within the budget although +none is stuck. `ThreadTablesFreeze` closes both gates and waits for both counts, +so no call at all is inside an engine while a table is replaced or a replaced +CPU engine releases its threads. On a freeze timeout the operation fails and +replaces nothing; on a `closeThreadRegistration()` timeout the profiler goes to +`ERROR` (see `stop()` above and `Profiler::startCpuEngine()`). The gate flags and +the counters use `seq_cst` on both sides: either a call sees the gate closed, or +the closer sees the call and waits. + +Thread cancellation is disabled for the length of each call, because the engine +calls reach cancellation points (`close()`, `Log::warn()`). Acting on a +cancellation there would unwind through `~HookCallScope()`, which aborts with the +statically linked libgcc, or, on musl, skip it and leave the count raised. The +state is restored by a plain call made after every destructor-bearing scope of the +call, not by a guard. Current glibc acts on a pending asynchronous cancellation as +soon as it is re-enabled, and unwinding out of an implicitly `noexcept` destructor +terminates the process. Each engine's `start()` also registers every thread that already exists. @@ -143,7 +236,8 @@ Profiler::recordSample[Delegated]() `SIGNAL_HANDLER_GUARD_OR_DROP` is defined at `guards.h:115`. Inside `recordSample()`, the handler tries to take one of `CONCURRENCY_LEVEL` (16) striped spin locks with `tryLock()`, three attempts at most. If all three fail, -the sample is dropped and counted as `ticks_skipped` (`profiler.cpp:680-691`). +the sample is dropped and counted as `ticks_skipped` (`Profiler::recordSample()` and the other +`record*()` methods). No handler blocks. The `*Jvmti` variants call `recordSampleDelegated()` instead, which asks @@ -154,7 +248,7 @@ HotSpot's JFR `RequestStackTrace` JVMTI extension to walk the stack ### Selection -`selectCpuEngine()` (`profiler.cpp:1269-1314`): +`Profiler::selectCpuEngine()`: ``` cpu not requested and event is NULL/"noop" -> noop_engine @@ -190,24 +284,27 @@ resolved through `Arguments::cpuSamplerInterval()`. - **Mechanism**: `registerThread(tid)` creates one POSIX timer per thread with the raw `__NR_timer_create` syscall. The clock is `((~tid) << 3) | 6`, the thread's per-thread scheduler CPU clock - (`ctimer_linux.cpp:45-47`). The raw syscall is used because the libc wrapper - accepts only predefined clocks (`ctimer_linux.cpp:85`). The timer notifies + (`thread_cpu_clock()` in `ctimer_linux.cpp`). The raw syscall is used because the libc wrapper + accepts only predefined clocks. The timer notifies with `SIGEV_THREAD_ID` targeting `tid`, carrying - `sival_ptr = SignalCookie::cpu()` (`ctimer_linux.cpp:72-83`). It is armed + `sival_ptr = SignalCookie::cpu()` (`CTimer::registerThread()`). It is armed periodic with `timer_settime`. -- **Signal**: `SIGPROF` (`ctimer_linux.cpp:151`). +- **Signal**: `SIGPROF` (set in `CTimer::startWithTableSize()`). - **Origin check**: the handler accepts only `si_code == SI_TIMER` carrying the CPU cookie. Anything else is forwarded with `OS::forwardForeignSignal()` - (`ctimer_linux.cpp:261-265`). + (`CTimer::signalHandler()`). - **Per-thread table**: `_timers[tid]` holds `timer_id + 1` (0 means empty), sized from `OS::getMaxThreadId()`. Registration and unregistration claim slots with CAS, so concurrent register/unregister of one tid deletes the - timer exactly once (`ctimer_linux.cpp:93-131`). A tid at or above - `_max_timers` logs a warning and is not profiled. -- **`stop()`** unregisters every slot (`ctimer_linux.cpp:181-187`). + timer exactly once (`CTimer::registerThread()`, `unregisterThread()`). A tid + at or above + `_max_timers` logs a warning and is not profiled. The table is accounted + under `NM_CTIMER` at its full allocated size (see Profiler Self-Accounting). +- **`stop()`** unregisters every slot. - **`CTimerJvmti`**: runs `CTimer::start()` and then replaces the `SIGPROF` handler with one that calls `recordSampleDelegated(ucontext, ...)` - (`ctimer_linux.cpp:196-252`). It has no ASGCT fallback. + (`CTimerJvmti::start()`, `CTimerJvmti::signalHandler()`). It has no ASGCT + fallback. ### `PerfEvents` (`perfEvents_linux.cpp`, Linux only) @@ -228,13 +325,28 @@ resolved through `Arguments::cpuSamplerInterval()`. - **Thread hook**: `start()` patches the JVM library's `pthread_setspecific` import so that HotSpot setting or clearing its thread TLS key registers or unregisters the thread (`perfEvents_linux.cpp:171-204`). -- **`stop()` is a no-op** (`perfEvents_linux.cpp:1047`): fds stay open across +- **`stop()` is a no-op**: fds stay open across stop/start, and `registerThread()` skips threads that already have an fd. After stop, the cleared `_enabled` flag keeps handlers from recording. - **Memory**: the `_events` array and any 2-page mmap ring pages are accounted - under `NM_PERF`. + under `NM_PERF`. The array is counted at its full allocated size, which is + mostly untouched address space (see Profiler Self-Accounting). +- **Table replacement**: when `pid_max` changed, `start()` releases the old + table's fds and rings (`releaseThreads()`) and then frees and reallocates + `_events`. It does not exclude thread unregistration or the signal-path + readers (`walkKernel()`, `resetBuffer()`) while it does so: a handler that + already holds a slot pointer can still use the freed table (PROF-16359). +- **Allocation failure**: if the table cannot be allocated, `start()` sets + `_max_events` back to -1, logs an error and fails before it installs the + signal handler, the hook or the registration gate, so the engine stays off + for the recording. `walkKernel()` and `resetBuffer()` skip the missing table: + fds left armed by an earlier recording keep delivering `SIGPROF`. +- **Engine change**: `releaseThreads()` closes every fd and unmaps every ring. + `Profiler::replaceCpuEngine()` calls it when `start()` selects another CPU + engine, because thread exits stop reaching PerfEvents once it leaves the + slot. - `CSTACK_LBR` and kernel stacks (`walkKernel`) are available only with this - engine (`profiler.cpp:1622`, `1642`). + engine (`Profiler::start()`, `Profiler::getNativeTrace()`). ### `ITimer` / `ITimerJvmti` (`itimer.cpp`, all platforms) @@ -269,7 +381,7 @@ dedicated profiler pthread picks threads and signals each one directly. ### Selection -`selectWallEngine()` (`profiler.cpp:1316-1349`): on OpenJ9, either +`Profiler::selectWallEngine()`: on OpenJ9, either `J9WallClock` (JVMTI polling) or `WallClockASGCT`. On HotSpot with `jvmtistacks` and `VM::canRequestStackTrace()`, `WallClockJvmti`. Otherwise, `WallClockASGCT`; a requested `JVMTI` sampler that is not available falls back @@ -340,7 +452,7 @@ running. The source tree has no TLAB-specific engine class. Heap allocation sampling is JVMTI `SampledObjectAlloc`. `BCI_ALLOC_OUTSIDE_TLAB` exists in `vmEntry.h:27` -and is handled in `profiler.cpp:328` and `hotspot/hotspotSupport.cpp:123`, but no +and is handled in `Profiler::getNativeTrace()` and `hotspot/hotspotSupport.cpp:123`, but no engine produces it; `ObjectSampler` always records `BCI_ALLOC`. - **Capability**: `VM::canSampleObjects()` requires @@ -409,7 +521,7 @@ chains are described in survivors. `cleanup_table()` runs from `flush()`, `stop()`, the overflow path in `track()`, and `maybeForceCleanup()` (called from the reference-chain thread). -- **Output**: `Profiler::dump()` calls `flush()` (`profiler.cpp:2035`). The +- **Output**: `Profiler::dump()` calls `flush()`. The flush runs a cleanup and then, under the exclusive lock, emits one `BCI_LIVENESS` deferred sample per surviving entry through `recordDeferredSample()` (`livenessTracker.cpp:1962`). When heap-usage @@ -459,9 +571,9 @@ In summary: ## Profiler Self-Accounting (`NativeMem`, `nativeMem.{h,cpp}`) `nativeMem.{h,cpp}` is **not** a profiling engine and does not observe -application allocations. It tracks the profiler's **own** native memory in 13 -categories (`NM_CALLTRACE`, `NM_DICTIONARY`, ..., `NM_PERF`, `NM_LIVENESS`, -`NM_WALLCLOCK`, `NM_MISC`; `nativeMem.h`). +application allocations. It tracks the profiler's **own** native memory in 14 +categories, defined in `nativeMem.h`: `NM_CALLTRACE`, `NM_DICTIONARY`, ..., +`NM_PERF`, `NM_LIVENESS`, `NM_WALLCLOCK`, `NM_CTIMER`, `NM_MISC`. - `record(cat, delta)` is a relaxed atomic add plus a lock-free CAS on the per-category peak. It is async-signal-safe, which matters because the @@ -476,9 +588,26 @@ categories (`NM_CALLTRACE`, `NM_DICTIONARY`, ..., `NM_PERF`, `NM_LIVENESS`, exported as the `NATIVE_MEM_LIVE_BYTES`, `NATIVE_MEM_AVG_BYTES`, and `NATIVE_MEM_MAX_BYTES` counters (`flightRecorder.cpp:2201-2203`). -The engines above feed it: `PerfEvents` reports `NM_PERF`, -`timerLoopCommon()` reports `NM_WALLCLOCK`, and `LivenessTracker` reports -`NM_LIVENESS`. +The engines above feed it: `PerfEvents` reports its tid-indexed event table +(up to `pid_max × sizeof(PerfEvent)` bytes, two cache lines per slot; address +space) and the per-thread ring pages (resident) as `NM_PERF`, `CTimer` reports +its tid-indexed timer table (up to `pid_max × 4` bytes, `pid_max × sizeof(int)`; +address space) as `NM_CTIMER`, `timerLoopCommon()` reports `NM_WALLCLOCK`, and +`LivenessTracker` reports `NM_LIVENESS`. + +Unlike the other categories, the two tid-indexed tables are allocated address +space, not resident memory: each is `calloc`ed at `pid_max` slots, but only the +pages holding live tids are ever touched. With a `pid_max` of 4194304 the +`ctimer` gauge reads about 16 MiB and the table part of the `perf` gauge about +512 MiB, while the resident share is usually far lower. Read both as upper +bounds. Only `NM_PERF`'s ring pages are resident. + +Each category is exported per chunk as +`native_mem_{live,avg,max,chunk_overhead,post_flush_live,post_flush_max}_bytes.` +(`Recording::writeNativeMem()`), with names taken from +`DD_NATIVE_MEM_CATEGORY_TABLE`. The set grows when categories are added, so +consumers should match these counters by prefix rather than hard-code the +category list. ## Counters by Engine diff --git a/doc/reference/ProfilerMemoryRequirements.md b/doc/reference/ProfilerMemoryRequirements.md index 03ddb57510..9dd378d795 100644 --- a/doc/reference/ProfilerMemoryRequirements.md +++ b/doc/reference/ProfilerMemoryRequirements.md @@ -7,11 +7,14 @@ The Datadog Java Profiler has inherent memory requirements due to its architectu The sizes below are estimates. The profiler's own native allocations are measured at runtime: `NativeMem` (`ddprof-lib/src/main/cpp/nativeMem.h`) keeps live, average and peak bytes for each category (`calltrace`, `dictionary`, `line_tables`, `thread_local`, -`jfr_buffers`, `method_map`, ...), and every chunk carries them as -`datadog.ProfilerCounter` events named `native_mem_live_bytes.`, -`native_mem_avg_bytes.`, `native_mem_max_bytes.` and -`native_mem_chunk_overhead_bytes.`. jmethodIDs are allocated by the JVM and are -not part of these figures. +`jfr_buffers`, `method_map`, `perf`, `ctimer`, ...), and every chunk carries them as +`datadog.ProfilerCounter` events named +`native_mem_{live,avg,max,chunk_overhead,post_flush_live,post_flush_max}_bytes.` +(`Recording::writeNativeMem()`). jmethodIDs are allocated by the JVM and are +not part of these figures. The category set is not fixed, so match these counters by +prefix rather than by a fixed list. The `ctimer` gauge and the +table part of the `perf` gauge count reserved, mostly untouched tid-indexed tables +(`pid_max` slots each), so they overstate RSS and are upper bounds. ## Memory Requirements