[sundapeng]

Purpose

Containers reading through paimon-jindo, such as compaction TaskManagers, get OOMKilled while the Java heap is far from full. JindoSDK keeps a readahead cache of tens of MiB per open read stream outside the JVM, and glibc keeps most of it resident after the streams close. Nothing in the logs shows how many streams were open or how much native memory Jindo held when the container died.

This PR adds an opt-in reporter to paimon-jindo:

• JindoIOStats counts the streams opened through a JindoFileIO that sets jindo.io-stats.interval, including multipart (two-phase) output streams: currently open, peak since the previous line, opened, closed and bytes. The counters live in the class loader of paimon-jindo, which is the whole JVM unless jobs bundle their own copy.
• A stream closed twice, even concurrently, is counted once. A close that throws is not counted, and a retried close still reaches Jindo. Until a close succeeds, the stream stays counted as open because it may still hold native memory, so the reporter keeps logging it. A multipart stream holds no native stream and cannot retry closeForCommit(), so any attempt counts it closed.
• With jindo.io-stats.interval set (for example 1 min), a daemon thread logs one INFO line per interval while streams are open or have been opened or closed since the previous line. Paimon does not close FileIOs, so the thread is not tied to them: it stops after an interval without stream activity, and the next open starts it again. Its context class loader is the one that loaded paimon-jindo, so it does not keep a finished job's class loader alive. When FileIOs set different intervals, the shortest is used. If the thread cannot be started, for example when no native thread can be created near the memory limit, a WARN is logged, the stream is returned as usual, and a later stream retries after one interval. The default is 0, which disables both the counting and the reporter.
• The line combines the stream counters with:
  • JindoSDK's native metrics: prefetch cache size and capacity, open native streams, prefetch fallbacks on insufficient memory.
  • Process memory: the RSS split, heap, direct memory, metaspace, threads, and whether jemalloc is loaded.
  • The memory cgroup (v1 or v2): usage and limit, working set, peak, limit hits and OOM kills. Limits are hierarchical, so these come from the level between the process's cgroup and the mount root whose working set is closest to its limit.
• The thread samples the cgroup every second and logs the same line at WARN when the working set of a cgroup level first crosses 80%, 90% or 95% of its limit, resetting below 70%. Each level keeps its own alarm, so a parent's WARN does not mute a child. The last line before an OOM kill therefore shows what was open.
• Native metrics are read only through cached file systems, which are never closed. A failure is logged at DEBUG and turns that section off; it never affects reads or writes.

Example (JDK 11, 16 read streams in a 1100 MiB memory cgroup):

WARN  JindoIOStats - Container memory working set is at 85% of the limit. Jindo IO stats: input open=16 peak=16 opened=16 closed=0 read=384MiB; output open=0 peak=0 opened=0 closed=0 written=0MiB; jindo prefetchCache=346MiB/4423MiB readStreams=16 writeStreams=0 memInsufficient=0; process rss=1003MiB anon=939MiB file=63MiB shmem=0MiB threads=42 heap=69MiB/494MiB/494MiB direct=0MiB metaspace=13MiB jemalloc=false; cgroup usage=943MiB/1100MiB workingSet=943MiB peak=943MiB limitHits=0 oomKills=0

Tests

• JindoIOStatsTest (19 tests) covers:
  • the counters and peak, and stream counting through HadoopCompliantFileIO with a mocked JindoHadoopSystem (including double close and byte counts);
  • a failed close that stays counted as open until it is retried, and streams of a FileIO with the interval unset, which are not counted;
  • multipart output streams, including one whose upload fails to start;
  • concurrent opens and closes, where every sample keeps 0 <= open <= peak, and two threads closing the same stream, which counts one close;
  • the reporter stopping once streams are idle and starting again on the next open, and a stream opened and closed between its idle check and its shutdown, which keeps it running;
  • a reporter thread that fails to start, which leaves the stream usable and is retried by a later stream, and the shortest interval winning;
  • activity detection after closes;
  • parsing the native metrics JSON;
  • cgroup v1, v2 (including a limited parent above an unlimited child) and none;
  • the alarm levels, also when a child cgroup becomes the level closest to its limit after its parent warned.
• All 52 tests in paimon-jindo pass, and so do checkstyle and spotless.
• I also ran it by hand on JDK 11 against OSS with jindosdk 6.10.2, in memory-limited cgroups:
  • 1100 MiB limit: 16 read streams logged the 85% WARN above. After they closed, prefetchCache was 0 MiB while anon stayed at 934 MiB, which is glibc retention rather than Jindo.
  • 960 MiB limit: the 90% WARN, with input open=16 and prefetchCache=308MiB, was the last line before the kernel OOM-killed the JVM.
• Overhead: about 50 us per one-second sample, and about 0.2 ms per line plus about 0.4 ms for the native metrics dump.