Potential memory leak and race with the JVMTI wallclock sampler

[zhengyu123]

What does this PR do?:
Release jthread local reference to prevent memory leak.

Profiler uses jvmtiError GetAllThreads(jvmtiEnv* env, jint* threads_count_ptr, jthread** threads_ptr) to obtain a list of running threads. The document states

On return, the jthread* points to a newly allocated array of size *threads_count_ptr. The array should be freed with [Deallocate](https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html#Deallocate). The objects returned by threads_ptr are JNI local references and must be [managed](https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html#refs).

The returned array contains JNI local references of threads, should be managed by caller, which means the caller should manage the life cycle of returned JNI local reference. In this case, we should delete those JNI local references to avoid the leak.

Also, JVMTI GetAllTheads() snapshots alive threads, the returned JNI local references only guarantee that Thread objects are not reclaimed by GCs, it does not prevent underneath native thread from exiting, so that, we have to be extremely careful when examining captured thread's native structures, as they may no longer be valid.

Motivation:
Make JVMTI wallclock sampler useable.

Additional Notes:

How to test the change?:
Run:

java -javaagent:/Users/zhengyu.gu/ws/dd-java-agent.jar -Ddd.profiling.enabled=true -Ddd.profiling.upload.period=10 -Ddd.profiling.start-force-first=true -Ddd.profiling.ddprof.debug.lib=/Users/zhengyu.gu/go/src/github.com/DataDog/java-profiler/ddprof-lib/build/lib/main/debug/macos/arm64/libjavaProfiler.dylib -Ddd.env=workspace-jb -Ddd.service=akka-uct -XX:NativeMemoryTracking=summary -Ddd.profiling.smap.aggregation.enabled=false -Ddd.profiling.experimental.ddprof.wall.jvmti=true -Ddd.profiling.ddprof.wall.context.filter=false -jar renaissance-gpl-0.16.0.jar akka-uct -r 500000

It crashes without this fix, no crash with the fix.

For Datadog employees:

• If this PR touches code that signs or publishes builds or packages, or handles
  credentials of any kind, I've requested a review from @DataDog/security-design-and-guidance.
• This PR doesn't touch any of that.
• JIRA: PROF-10859

Unsure? Have a question? Request a review!

[github-actions]

🔧 Report generated by pr-comment-cppcheck

CppCheck Report

Errors (2)

• There is an unknown macro here somewhere. Configuration is required. If LP64_ONLY is a macro then please configure it.
• syntax error

Warnings (4)

• The class 'RecordingBuffer' defines member function with name 'flushIfNeeded' also defined in its parent class 'Buffer'.
• Member variable 'JitWriteProtection::_restore' is not initialized in the constructor.
• Member variable 'JitWriteProtection::_restore' is not initialized in the constructor.
• Member variable 'ElfParser::_vaddr_diff' is not initialized in the constructor.

Style Violations (301)

• Class 'Arguments' has a constructor with 1 argument that is not explicit.
• struct member 'Multiplier::symbol' is never used.
• struct member 'Multiplier::multiplier' is never used.
• class member 'Error::OK' is never used.
• Class 'AsyncSampleMutex' has a constructor with 1 argument that is not explicit.
• The class 'NativeFunc' does not declare a constructor although it has private member variables which likely require initialization.
• Local variable 'count' shadows outer function
• Local variable 'blob' shadows outer function
• Local variable 'i' shadows outer variable
• Parameter 'prefixes' can be declared as reference to const
• Variable 'cb1' can be declared as pointer to const
• Variable 'cb2' can be declared as pointer to const
• Variable 'fd1' can be declared as pointer to const
• Variable 'fd2' can be declared as pointer to const
• Parameter 'name' can be declared as pointer to const
• Consider using std::transform algorithm instead of a raw loop.
• struct member 'ContextPage::capacity' is never used.
• struct member 'ContextPage::storage' is never used.
• class member 'Contexts::_max_pages' is never used.
• class member 'Contexts::_pages' is never used.
• The function 'check' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'start' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'registerThread' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'unregisterThread' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'name' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'interval' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'check' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'start' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'stop' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'enableEvents' overrides a function in a base class but is not marked with a 'override' specifier.
• class member 'CTimer::_cstack' is never used.
• class member 'CTimer::_signal' is never used.
• class member 'CTimer::_max_timers' is never used.
• class member 'CTimer::_timers' is never used.
• class member 'Shims::_tid_setter_ref' is never used.
• Class 'Dictionary' has a constructor with 1 argument that is not explicit.
• struct member 'DictRow::keys' is never used.
• struct member 'DictRow::next' is never used.
• Operator '|' with one operand equal to zero is redundant.
• Operator '|' with one operand equal to zero is redundant.
• Operator '|' with one operand equal to zero is redundant.
• Local variable 'table' shadows outer function
• Local variable 'pc_off' shadows outer variable
• Variable 'table' can be declared as pointer to const
• struct member 'FrameDesc::fp_off' is never used.
• struct member 'FrameDesc::pc_off' is never used.
• class member 'DwarfParser::_name' is never used.
• class member 'DwarfParser::_image_base' is never used.
• class member 'DwarfParser::_capacity' is never used.
• class member 'DwarfParser::_prev' is never used.
• class member 'DwarfParser::_data_align' is never used.
• Class 'WallClockEpochEvent' has a constructor with 1 argument that is not explicit.
• class member 'LockEvent::_address' is never used.
• class member 'LockEvent::_timeout' is never used.
• The function 'name' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'interval' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'check' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'start' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'stop' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'enableEvents' overrides a function in a base class but is not marked with a 'override' specifier.
• class member 'ITimer::_cstack' is never used.
• The class 'J9WallClock' does not declare a constructor although it has private member variables which likely require initialization.
• The function 'name' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'interval' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'start' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'stop' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'enableEvents' overrides a function in a base class but is not marked with a 'override' specifier.
• class member 'J9WallClock::_max_stack_depth' is never used.
• class member 'J9WallClock::_running' is never used.
• Class 'Element' has a constructor with 1 argument that is not explicit.
• Class 'NoField' has a constructor with 1 argument that is not explicit.
• The function 'is_nofield' overrides a function in a base class but is not marked with a 'override' specifier.
• class member 'JfrMetadata::_initialized' is never used.
• class member 'JVM::_is_readable_pointer_resolved' is never used.
• class member 'JVM::_is_readable_pointer' is never used.
• The class 'VirtualSpaceSummary' does not declare a constructor although it has private member variables which likely require initialization.
• The class 'GCHeapSummary' does not declare a constructor although it has private member variables which likely require initialization.
• The class 'CollectedHeapWrapper' does not declare a constructor although it has private member variables which likely require initialization.
• class member 'VirtualSpaceSummary::_committed_end' is never used.
• class member 'GCHeapSummary::vptr' is never used.
• class member 'CollectedHeapWrapper::vptr' is never used.
• class member 'CollectedHeapWrapper::_gc_heap_log' is never used.
• class member 'CollectedHeapWrapper::_capacity_at_last_gc' is never used.
• class member 'CollectedHeapWrapper::_used_at_last_gc' is never used.
• struct member 'Chunk::prev' is never used.
• struct member 'Chunk::offs' is never used.
• struct member 'Chunk::_padding' is never used.
• class member 'LinearAllocator::_chunk_size' is never used.
• class member 'LinearAllocator::_tail' is never used.
• class member 'LinearAllocator::_reserve' is never used.
• Class 'SpinLock' has a constructor with 1 argument that is not explicit.
• struct member 'TrackingEntry::skipped' is never used.
• class member 'LivenessTracker::MAX_TRACKING_TABLE_SIZE' is never used.
• class member 'LivenessTracker::MIN_SAMPLING_INTERVAL' is never used.
• class member 'LivenessTracker::_table_lock' is never used.
• class member 'Log::_file' is never used.
• class member 'Log::LEVEL_NAME' is never used.
• The function 'check' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'start' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'stop' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'interval' overrides a function in a base class but is not marked with a 'override' specifier.
• class member 'ObjectSampler::CONFIG_UPDATE_CHECK_PERIOD_SECS' is never used.
• class member 'JitWriteProtection::_restore' is never used.
• class member 'OS::page_size' is never used.
• class member 'OS::page_mask' is never used.
• The function 'rewind' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'next' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'size' overrides a function in a base class but is not marked with a 'override' specifier.
• Variable 'entry' can be declared as pointer to const
• The function 'rewind' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'next' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'size' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'check' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'start' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'enableEvents' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'registerThread' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'unregisterThread' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'interval' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'enableEvents' overrides a function in a base class but is identical to the overridden function
• The function 'registerThread' overrides a function in a base class but is identical to the overridden function
• The function 'unregisterThread' overrides a function in a base class but is identical to the overridden function
• The function 'check' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'start' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'stop' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'registerThread' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'unregisterThread' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'interval' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'name' overrides a function in a base class but is not marked with a 'override' specifier.
• The function 'enableEvents' overrides a function in a base class but is not marked with a 'override' specifier.
• class member 'PerfEvents::_max_events' is never used.
• class member 'PerfEvents::_events' is never used.
• class member 'PerfEvents::_event_type' is never used.
• class member 'PerfEvents::_ring' is never used.
• class member 'PerfEvents::_cstack' is never used.
• class member 'PerfEvents::_use_mmap_page' is never used.
• Class 'ReservoirSampler' has a constructor with 1 argument that is not explicit.
• Parameter 'ptr' can be declared as pointer to const. However it seems that 'load32' is a callback function, if 'ptr' is declared with const you might also need to cast function pointer(s).
• Parameter 'ptr' can be declared as pointer to const
• Parameter 'ptr' can be declared as pointer to const. However it seems that 'load32' is a callback function, if 'ptr' is declared with const you might also need to cast function pointer(s).
• Class 'SymbolDesc' has a constructor with 1 argument that is not explicit.
• Class 'MemoryMapDesc' has a constructor with 1 argument that is not explicit.
• Local variable 'section' shadows outer function
• Local variable 'section' shadows outer function
• Local variable 'section' shadows outer function
• Variable 'ident' can be declared as pointer to const
• Parameter 'section' can be declared as pointer to const
• Variable 'checked' can be declared as pointer to const
• Variable 'pheader' can be declared as pointer to const
• Variable 'sym' can be declared as pointer to const
• Variable 'sym' can be declared as pointer to const
• Variable 'chain' can be declared as pointer to const
• Variable 'plt' can be declared as pointer to const
• Variable 'note' can be declared as pointer to const
• Variable 'sym' can be declared as pointer to const
• struct member 'RootSymbolEntry::name' is never used.
• struct member 'RootSymbolEntry::kind' is never used.
• class member 'ElfParser::_root_symbols' is never used.
• Class 'ExclusiveLock' has a constructor with 1 argument that is not explicit.
• Class 'SharedLock' has a constructor with 1 argument that is not explicit.
• Same expression on both sides of '-'.
• Local variable 'count' shadows outer function
• Parameter 'failures' can be declared as reference to const
• class member 'ProfiledThread::_buffer_size' is never used.
• class member 'ProfiledThread::_buffer' is never used.
• Local variable 'word' shadows outer function
• class member 'ThreadInfo::_thread_names' is never used.
• class member 'ThreadInfo::_thread_ids' is never used.
• Condition 'enabled()' is always false
• The function 'loadAcquire' is never used.
• The function 'message' is never used.
• The function 'hasOption' is never used.
• The function 'cpuSamplerInterval' is never used.
• The function 'acquired' is never used.
• The function 'offset' is never used.
• The function 'skip' is never used.
• The function 'put16' is never used.
• The function 'put64' is never used.
• The function 'putFloat' is never used.
• The function 'putVar64' is never used.
• The function 'findBlobByAddress' is never used.
• The function 'binarySearch' is never used.
• The function 'dump' is never used.
• The function 'findSymbolsByPrefix' is never used.
• The function 'findImport' is never used.
• The function 'patchImport' is never used.
• The function 'findFrameDesc' is never used.
• The function 'libIndex' is never used.
• The function 'isMarked' is never used.
• The function 'minAddress' is never used.
• The function 'maxAddress' is never used.
• The function 'contains' is never used.
• The function 'get' is never used.
• The function 'getPage' is never used.
• The function 'get_tag' is never used.
• The function 'getCounters' is never used.
• The function 'getCounter' is never used.
• The function 'describeCounters' is never used.
• The function 'units' is never used.
• The function 'check' is never used.
• The function 'start' is never used.
• The function 'supported' is never used.
• The function 'setSighandlerTid' is never used.
• The function 'bounded_lookup' is never used.
• The function 'stop' is never used.
• The function 'interval' is never used.
• The function 'enableEvents' is never used.
• The function 'updateCounter' is never used.
• The function 'registerThread' is never used.
• The function 'unregisterThread' is never used.
• The function 'hasChanged' is never used.
• The function 'updateNumSamplableThreads' is never used.
• The function 'updateNumSuccessfulSamples' is never used.
• The function 'updateNumFailedSamples' is never used.
• The function 'updateNumExitedThreads' is never used.
• The function 'updateNumPermissionDenied' is never used.
• The function 'endEpoch' is never used.
• The function 'clean' is never used.
• The function 'newEpoch' is never used.
• The function 'encode' is never used.
• The function 'decode' is never used.
• The function 'threadEntry' is never used.
• The function 'sampleIdleThreads' is never used.
• The function 'field' is never used.
• The function 'category' is never used.
• The function 'root' is never used.
• The function 'strings' is never used.
• The function 'jniExceptionCheck' is never used.
• The function 'is_readable_pointer' is never used.
• The function 'used' is never used.
• The function 'alloc' is never used.
• The function 'level' is never used.
• The function 'next' is never used.
• The function 'cputime' is never used.
• The function 'micros' is never used.
• The function 'processStartTime' is never used.
• The function 'sleep' is never used.
• The function 'ntoh64' is never used.
• The function 'threadId' is never used.
• The function 'schedPolicy' is never used.
• The function 'threadName' is never used.
• The function 'listThreads' is never used.
• The function 'isLinux' is never used.
• The function 'installSignalHandler' is never used.
• The function 'replaceSigsegvHandler' is never used.
• The function 'replaceSigbusHandler' is never used.
• The function 'sendSignalToThread' is never used.
• The function 'getCpuDescription' is never used.
• The function 'getProcessCpuTime' is never used.
• The function 'getTotalCpuTime' is never used.
• The function 'copyFile' is never used.
• The function 'truncateFile' is never used.
• The function 'fileSize' is never used.
• The function 'freePageCache' is never used.
• The function 'mallocArenaMax' is never used.
• The function 'walkKernel' is never used.
• The function 'resetBuffer' is never used.
• The function 'getEventName' is never used.
• The function 'compute' is never used.
• The function 'sample' is never used.
• The function 'is_probably_rust_legacy' is never used.
• The function 'demangle' is never used.
• The function 'initSafeFetch' is never used.
• The function 'safeFetch32' is never used.
• The function 'skipLoad' is never used.
• The function 'skipLoadArg' is never used.
• The function 'tryLockShared' is never used.
• The function 'withinCurrentStack' is never used.
• The function 'restore' is never used.
• The function 'stackAt' is never used.
• The function 'clearParsingCaches' is never used.
• The function 'parseLibraries' is never used.
• The function 'isRootSymbol' is never used.
• The function 'forTid' is never used.
• The function 'inBuffer' is never used.
• The function 'tid' is never used.
• The function 'noteCPUSample' is never used.
• The function 'lookupWallclockCallTraceId' is never used.
• The function 'recordCallTraceId' is never used.
• The function 'enterCrashHandler' is never used.
• The function 'exitCrashHandler' is never used.
• The function 'resetCrashHandler' is never used.
• The function 'isDeepCrashHandler' is never used.
• The function 'unwindFailures' is never used.
• The function 'bitmapAddressFor' is never used.
• The function 'accept' is never used.
• The function 'remove' is never used.
• The function 'addressOfSize' is never used.
• The function 'isValid' is never used.
• The function 'getThreadId' is never used.
• The function 'clearAll' is never used.
• The function 'updateThreadName' is never used.
• The function 'reportCounters' is never used.
• The function 'convertJvmExecutionState' is never used.
• The function 'initialized' is never used.
• The function 'ticks' is never used.
• The function 'frequency' is never used.
• The function 'ticks_to_millis' is never used.
• The function 'record' is never used.
• The function 'forEach' is never used.
• The function 'countFailures' is never used.
• The function 'collectAndReset' is never used.

[github-actions]

🔧 Report generated by pr-comment-scanbuild

[jbachorik]

Looks good

[r1viollet]

Not my domain, though I think returning on threads_count == 0 is a functional difference.
I think it is OK to stop the profiler if there are no threads.

[jbachorik]

Yes, we should always have at least one thread reported by JVMTI

[r1viollet]

LGTM

[r1viollet]

I allowed myself to pull main into your branch, I'm chasing flaky tests down and wanted to check if some of them were fixed.

[jbachorik]

I found a few things I think are not right. Please, re-check and eventually fix. Thanks!

[jbachorik]

LGTM!