2.7 Native Memory Tracking

The Native Memory Tracking (NMT) is a Java HotSpot VM feature that tracks internal memory usage for a Java HotSpot VM. For details about NMT scope, how to enable NMT, and other additional usage details, see Native Memory Tracking.

Since NMT doesn't track memory allocations by non-JVM code, you may have to use tools supported by the operating system to detect memory leaks in native code.

The following sections describe how to monitor VM internal memory allocations and diagnose VM memory leaks.

Use NMT to Detect a Memory Leak
How to Monitor VM Internal Memory
NMT Memory Categories

2.7.1 Use NMT to Detect a Memory Leak

Follow these steps to detect a memory leak.

Start the JVM with summary or detail tracking using the command line option: -XX:NativeMemoryTracking=summary or -XX:NativeMemoryTracking=detail.
Establish an early baseline - use NMT baseline feature to get a baseline to compare during development and maintenance by running: jcmd <pid> VM.native_memory baseline.
Monitor memory changes using: jcmd <pid> VM.native_memory detail.diff.
If the application leaks a small amount of memory, it make take a while to show up.

2.7.2 How to Monitor VM Internal Memory

Used with The jcmd Utility, Native Memory Tracking can be set up to monitor memory and ensure that an application does not start to use increasing amounts of memory during development or maintenance. See Table 2-1 for details about NMT memory categories.

The following sections describe how to get summary or detail data for NMT and describes how to interpret the sample output.

Interpret sample output: From the sample output below, you will see reserved and committed memory. Note that only committed memory is actually used. For example, if you run with -Xms100m -Xmx1000m, the JVM will reserve 1000 MB for the Java Heap. Since the initial heap size is only 100 MB, only 100MB will be committed to begin with. For a 64-bit machine where address space is almost unlimited, there is no problem if a JVM reserves a lot of memory. The problem arises if more and more memory gets committed, which may lead to swapping or native OOM situations.

Arena is a chunk of memory allocated using malloc. Memory is freed from these chunks in bulk, when exiting a scope or leaving an area of code. These chunks may be reused in other subsystems to hold temporary memory, for example, pre-thread allocations. Arena malloc policy ensures no memory leakage. So Arena is tracked as a whole and not individual objects. Some amount of initial memory can not by tracked.

Enabling NMT will result in a 5-10 percent JVM performance drop and memory usage for NMT adds 2 machine words to all malloc memory as malloc header. NMT memory usage is also tracked by NMT.

Get summary data: To get a summary view of native memory usage, start the JVM with command line option: -XX:NativeMemoryTracking=summary.

Example 2-2 is a sample output that describes NMT for track level set to summary. One way to get this sample output is to run: jcmd <pid> VM.native_memory summary.

Example 2-2 Get Summary Data for NMT

Total:  reserved=664192KB,  committed=253120KB                                           <--- total memory tracked by Native Memory Tracking
 
-                 Java Heap (reserved=516096KB, committed=204800KB)                      <--- Java Heap
                            (mmap: reserved=516096KB, committed=204800KB)
 
-                     Class (reserved=6568KB, committed=4140KB)                          <--- class metadata
                            (classes #665)                                               <--- number of loaded classes
                            (malloc=424KB, #1000)                                        <--- malloc'd memory, #number of malloc
                            (mmap: reserved=6144KB, committed=3716KB)
 
-                    Thread (reserved=6868KB, committed=6868KB)
                            (thread #15)                                                 <--- number of threads
                            (stack: reserved=6780KB, committed=6780KB)                   <--- memory used by thread stacks
                            (malloc=27KB, #66)
                            (arena=61KB, #30)                                            <--- resource and handle areas
 
-                      Code (reserved=102414KB, committed=6314KB)
                            (malloc=2574KB, #74316)
                            (mmap: reserved=99840KB, committed=3740KB)
 
-                        GC (reserved=26154KB, committed=24938KB)
                            (malloc=486KB, #110)
                            (mmap: reserved=25668KB, committed=24452KB)
 
-                  Compiler (reserved=106KB, committed=106KB)
                            (malloc=7KB, #90)
                            (arena=99KB, #3)
 
-                  Internal (reserved=586KB, committed=554KB)
                            (malloc=554KB, #1677)
                            (mmap: reserved=32KB, committed=0KB)
 
-                    Symbol (reserved=906KB, committed=906KB)
                            (malloc=514KB, #2736)
                            (arena=392KB, #1)
 
-           Memory Tracking (reserved=3184KB, committed=3184KB)
                            (malloc=3184KB, #300)
 
-        Pooled Free Chunks (reserved=1276KB, committed=1276KB)
                            (malloc=1276KB)
 
-                   Unknown (reserved=33KB, committed=33KB)
                            (arena=33KB, #1)

Get detail data: To get a more detailed view of native memory usage, start the JVM with command line option: -XX:NativeMemoryTracking=detail. This will track exactly what methods allocate the most memory. Enabling NMT will result in 5-10 percent JVM performance drop and memory usage for NMT adds 2 words to all malloc memory as malloc header. NMT memory usage is also tracked by NMT.

Example 2-3 shows a sample output for virtual memory for track level set to detail. One way to get this sample output is to run: jcmd <pid> VM.native_memory detail.

Example 2-3 Get detail Data for NMT

Virtual memory map: [0x8f1c1000 - 0x8f467000] reserved 2712KB for Thread Stack from [Thread::record_stack_base_and_size()+0xca] [0x8f1c1000 - 0x8f467000] committed 2712KB from [Thread::record_stack_base_and_size()+0xca] [0x8f585000 - 0x8f729000] reserved 1680KB for Thread Stack from [Thread::record_stack_base_and_size()+0xca] [0x8f585000 - 0x8f729000] committed 1680KB from [Thread::record_stack_base_and_size()+0xca] [0x8f930000 - 0x90100000] reserved 8000KB for GC from [ReservedSpace::initialize(unsigned int, unsigned int, bool, char*, unsigned int, bool)+0x555] [0x8f930000 - 0x90100000] committed 8000KB from [PSVirtualSpace::expand_by(unsigned int)+0x95] [0x902dd000 - 0x9127d000] reserved 16000KB for GC from [ReservedSpace::initialize(unsigned int, unsigned int, bool, char*, unsigned int, bool)+0x555] [0x902dd000 - 0x9127d000] committed 16000KB from [os::pd_commit_memory(char*, unsigned int, unsigned int, bool)+0x36] [0x9127d000 - 0x91400000] reserved 1548KB for Thread Stack from [Thread::record_stack_base_and_size()+0xca] [0x9127d000 - 0x91400000] committed 1548KB from [Thread::record_stack_base_and_size()+0xca] [0x91400000 - 0xb0c00000] reserved 516096KB for Java Heap <--- reserved memory range from [ReservedSpace::initialize(unsigned int, unsigned int, bool, char*, unsigned int, bool)+0x190] <--- callsite that reserves the memory [0x91400000 - 0x93400000] committed 32768KB from [VirtualSpace::initialize(ReservedSpace, unsigned int)+0x3e8] <--- committed memory range and its callsite [0xa6400000 - 0xb0c00000] committed 172032KB from [PSVirtualSpace::expand_by(unsigned int)+0x95] <--- committed memory range and its callsite [0xb0c61000 - 0xb0ce2000] reserved 516KB for Thread Stack from [Thread::record_stack_base_and_size()+0xca] [0xb0c61000 - 0xb0ce2000] committed 516KB from [Thread::record_stack_base_and_size()+0xca] [0xb0ce2000 - 0xb0e83000] reserved 1668KB for GC from [ReservedSpace::initialize(unsigned int, unsigned int, bool, char*, unsigned int, bool)+0x555] [0xb0ce2000 - 0xb0cf0000] committed 56KB from [PSVirtualSpace::expand_by(unsigned int)+0x95] [0xb0d88000 - 0xb0d96000] committed 56KB from [CardTableModRefBS::resize_covered_region(MemRegion)+0xebf] [0xb0e2e000 - 0xb0e83000] committed 340KB from [CardTableModRefBS::resize_covered_region(MemRegion)+0xebf] [0xb0e83000 - 0xb7003000] reserved 99840KB for Code from [ReservedSpace::initialize(unsigned int, unsigned int, bool, char*, unsigned int, bool)+0x555] [0xb0e83000 - 0xb0e92000] committed 60KB from [VirtualSpace::initialize(ReservedSpace, unsigned int)+0x3e8] [0xb1003000 - 0xb139b000] committed 3680KB from [VirtualSpace::initialize(ReservedSpace, unsigned int)+0x37a] [0xb7003000 - 0xb7603000] reserved 6144KB for Class from [ReservedSpace::initialize(unsigned int, unsigned int, bool, char*, unsigned int, bool)+0x555] [0xb7003000 - 0xb73a4000] committed 3716KB from [VirtualSpace::initialize(ReservedSpace, unsigned int)+0x37a] [0xb7603000 - 0xb760b000] reserved 32KB for Internal from [PerfMemory::create_memory_region(unsigned int)+0x8ba] [0xb770b000 - 0xb775c000] reserved 324KB for Thread Stack from [Thread::record_stack_base_and_size()+0xca] [0xb770b000 - 0xb775c000] committed 324KB from [Thread::record_stack_base_and_size()+0xca]

Get diff from NMT baseline: For both summary and detail level tracking, you can set baseline once the application is up and running. Do this by running jcmd <pid> VM.native_memory baseline after some warm up of the application. Then, you can run: jcmd <pid> VM.native_memory summary.diff or jcmd <pid> VM.native_memory detail.diff.

Example 2-4 shows a sample output for the summary difference in native memory usage since the baseline and is a great way to find memory leaks.

Example 2-4 Get diff from NMT Baseline

Total: reserved=664624KB -20610KB, committed=254344KB -20610KB <--- total memory changes vs. earlier baseline. '+'=increase '-'=decrease - Java Heap (reserved=516096KB, committed=204800KB) (mmap: reserved=516096KB, committed=204800KB) - Class (reserved=6578KB +3KB, committed=4530KB +3KB) (classes #668 +3) <--- 3 more classes loaded (malloc=434KB +3KB, #930 -7) <--- malloc'd memory increased by 3KB, but number of malloc count decreased by 7 (mmap: reserved=6144KB, committed=4096KB) - Thread (reserved=60KB -1129KB, committed=60KB -1129KB) (thread #16 +1) <--- one more thread (stack: reserved=7104KB +324KB, committed=7104KB +324KB) (malloc=29KB +2KB, #70 +4) (arena=31KB -1131KB, #32 +2) <--- 2 more arenas (one more resource area and one more handle area) - Code (reserved=102328KB +133KB, committed=6640KB +133KB) (malloc=2488KB +133KB, #72694 +4287) (mmap: reserved=99840KB, committed=4152KB) - GC (reserved=26154KB, committed=24938KB) (malloc=486KB, #110) (mmap: reserved=25668KB, committed=24452KB) - Compiler (reserved=106KB, committed=106KB) (malloc=7KB, #93) (arena=99KB, #3) - Internal (reserved=590KB +35KB, committed=558KB +35KB) (malloc=558KB +35KB, #1699 +20) (mmap: reserved=32KB, committed=0KB) - Symbol (reserved=911KB +5KB, committed=911KB +5KB) (malloc=519KB +5KB, #2921 +180) (arena=392KB, #1) - Memory Tracking (reserved=2073KB -887KB, committed=2073KB -887KB) (malloc=2073KB -887KB, #84 -210) - Pooled Free Chunks (reserved=2624KB -15876KB, committed=2624KB -15876KB) (malloc=2624KB -15876KB)

Example 2-5 is a sample output that shows the detail difference in native memory usage since the baseline and is a great way to find memory leaks.

Example 2-5 Get detail diff for NMT

Details: [0x01195652] ChunkPool::allocate(unsigned int)+0xe2 (malloc=482KB -481KB, #8 -8) [0x01195652] ChunkPool::allocate(unsigned int)+0xe2 (malloc=2786KB -19742KB, #134 -618) [0x013bd432] CodeBlob::set_oop_maps(OopMapSet*)+0xa2 (malloc=591KB +6KB, #681 +37) [0x013c12b1] CodeBuffer::block_comment(int, char const*)+0x21 <--- [callsite address] method name + offset (malloc=562KB +33KB, #35940 +2125) <--- malloc'd amount, increased by 33KB #malloc count, increased by 2125 [0x0145f172] ConstantPool::ConstantPool(Array<unsigned char>*)+0x62 (malloc=69KB +2KB, #610 +15) ... [0x01aa3ee2] Thread::allocate(unsigned int, bool, unsigned short)+0x122 (malloc=21KB +2KB, #13 +1) [0x01aa73ca] Thread::record_stack_base_and_size()+0xca (mmap: reserved=7104KB +324KB, committed=7104KB +324KB)