Why does a JVM report more committed memory than the linux process resident set size? -
when running java app (in yarn) native memory tracking enabled (-xx:nativememorytracking=detail see https://docs.oracle.com/javase/8/docs/technotes/guides/vm/nmt-8.html , https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html), can see how memory jvm using in different categories.
my app on jdk 1.8.0_45 shows:
native memory tracking: total: reserved=4023326kb, committed=2762382kb - java heap (reserved=1331200kb, committed=1331200kb) (mmap: reserved=1331200kb, committed=1331200kb) - class (reserved=1108143kb, committed=64559kb) (classes #8621) (malloc=6319kb #17371) (mmap: reserved=1101824kb, committed=58240kb) - thread (reserved=1190668kb, committed=1190668kb) (thread #1154) (stack: reserved=1185284kb, committed=1185284kb) (malloc=3809kb #5771) (arena=1575kb #2306) - code (reserved=255744kb, committed=38384kb) (malloc=6144kb #8858) (mmap: reserved=249600kb, committed=32240kb) - gc (reserved=54995kb, committed=54995kb) (malloc=5775kb #217) (mmap: reserved=49220kb, committed=49220kb) - compiler (reserved=267kb, committed=267kb) (malloc=137kb #333) (arena=131kb #3) - internal (reserved=65106kb, committed=65106kb) (malloc=65074kb #29652) (mmap: reserved=32kb, committed=32kb) - symbol (reserved=13622kb, committed=13622kb) (malloc=12016kb #128199) (arena=1606kb #1) - native memory tracking (reserved=3361kb, committed=3361kb) (malloc=287kb #3994) (tracking overhead=3075kb) - arena chunk (reserved=220kb, committed=220kb) (malloc=220kb)
this shows 2.7gb of committed memory, including 1.3gb of allocated heap , 1.2gb of allocated thread stacks (using many threads).
however, when running ps ax -o pid,rss | grep <mypid> or top shows 1.6gb of res/rss resident memory. checking swap says none in use:
free -m total used free shared buffers cached mem: 129180 99348 29831 0 2689 73024 -/+ buffers/cache: 23633 105546 swap: 15624 0 15624
why jvm indicate 2.7gb memory committed when 1.6gb resident? did rest go?
i'm beginning suspect stack memory (unlike jvm heap) seems precommitted without becoming resident , on time becomes resident high water mark of actual stack usage.
yes, malloc/mmap lazy unless told otherwise. pages backed physical memory once they're accessed.
gc heap memory gets touched copying collector or pre-zeroing (-xx:+alwayspretouch), it'll resident. thread stacks otoh aren't affected this.
for further confirmation can use pmap -x <java pid> , cross-reference rss of various address ranges output virtual memory map nmt.
reserved memory has been mmaped prot_none. means virtual address space ranges have entries in page tables , not used other mmap/malloc calls. still cause page faults being forwarded process sigsegv, i.e. accessing them error.
this important have contiguous address ranges available future use, in turn simplifies pointer arithmetic.
committed-but-not-backed-by-storage memory has been mapped - example - prot_read | prot_write accessing still causes page fault. page fault silently handled kernel backing actual memory , returning execution if nothing happened.
i.e. it's implementation detail/optimization won't noticed process itself.
to give breakdown of concepts:
used heap: amount of memory occupied live objects according last gc
committed: address ranges have been mapped other prot_none. may or may not backed physical or swap due lazy allocation , paging.
reserved: total address range has been pre-mapped via mmap particular memory pool.
reserved − committed difference consists of prot_none mappings, guaranteed not backed physical memory
resident: pages in physical ram. means code, stacks, part of committed memory pools portions of mmaped files have been accessed , allocations outside control of jvm.
virtual: sum of virtual address mappings. covers committed, reserved memory pools mapped files or shared memory. number informative since jvm can reserve large address ranges in advance or mmap large files.
Comments
Post a Comment