Java 程序内存不足问题求解

1.问题描述

1. 服务忽然宕机，查看服务器发现生成了很多 hs_err_pidxxxxxx.log 这样的 log 文件。下面我会提出来文件的一些内容。
2. 查看服务的日志文件有如下错误提示

# There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 2863661056 bytes for committing reserved memory. # An error report file with more information is saved as: # /neworiental/software/bootstrap/hs_err_pid17840.log Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000006c0000000, 2863661056, 0) failed; error='Cannot allocate memory' (errno=12) 

1. JVM 参数

-Xms4G -Xmx4G -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/xxx/software/bootstrap/ 

2. 问题

1. 简单的查了查，对 jvm 不是特别的熟悉。不太清楚这个问题是 jvm 分配的内存有点小，服务内存溢出导致的，还是服务器内存资源不够了，导致服务产生异常。
2. 我加了 HeapDumpOnOutOfMemoryError 这个参数，如果是服务运行期间内存溢出，为什么没有产生 dump 文件。
3. 希望有了解这块内容的大佬帮忙看看，提供些思路感谢。

3. hs_err_pidxxxxxx.log 文件

Memory: 4k page, physical 16260372k(425348k free), swap 4194300k(874708k free) vm_info: Java HotSpot(TM) 64-Bit Server VM (25.172-b11) for linux-amd64 JRE (1.8.0_172-b11), built on Mar 28 2018 21:44:09 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8) time: Sun Oct 9 00:28:39 2022 elapsed time: 0 seconds (0d 0h 0m 0s) ^C [dontovertime@HYY-BMPP-P-sql96d40-09 bootstrap]$ tail -10000f hs_err_pid18191.log # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 2863661056 bytes for committing reserved memory. # Possible reasons: # The system is out of physical RAM or swap space # In 32 bit mode, the process size limit was hit # Possible solutions: # Reduce memory load on the system # Increase physical memory or swap space # Check if swap backing store is full # Use 64 bit Java on a 64 bit OS # Decrease Java heap size (-Xmx/-Xms) # Decrease number of Java threads # Decrease Java thread stack sizes (-Xss) # Set larger code cache with -XX:ReservedCodeCacheSize= # This output file may be truncated or incomplete. # # Out of Memory Error (os_linux.cpp:2640), pid=18191, tid=0x00007f6c15c70700 # # JRE version: (8.0_172-b11) (build ) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.172-b11 mixed mode linux-amd64 compressed oops) # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again --------------- T H R E A D --------------- Current thread (0x00007f6c0c00a800): JavaThread "Unknown thread" [_thread_in_vm, id=18192, stack(0x00007f6c15b71000,0x00007f6c15c71000)] Stack: [0x00007f6c15b71000,0x00007f6c15c71000], sp=0x00007f6c15c6d2b0, free space=1008k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0xad221a] VMError::report_and_die()+0x2ba V [libjvm.so+0x50185b] report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x8b V [libjvm.so+0x92fee3] os::Linux::commit_memory_impl(char*, unsigned long, bool)+0x123 V [libjvm.so+0x930109] os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x29 V [libjvm.so+0x9273ca] os::commit_memory(char*, unsigned long, unsigned long, bool)+0x2a V [libjvm.so+0x99d363] PSVirtualSpace::expand_by(unsigned long)+0x53 V [libjvm.so+0x98d477] PSOldGen::initialize(ReservedSpace, unsigned long, char const*, int)+0xb7 V [libjvm.so+0x2dbb8a] AdjoiningGenerations::AdjoiningGenerations(ReservedSpace, GenerationSizer*, unsigned long)+0x39a V [libjvm.so+0x9514a6] ParallelScavengeHeap::initialize()+0x1d6 V [libjvm.so+0xa9a253] Universe::initialize_heap()+0xf3 V [libjvm.so+0xa9a7be] universe_init()+0x3e V [libjvm.so+0x643de5] init_globals()+0x65 V [libjvm.so+0xa7ef7e] Threads::create_vm(JavaVMInitArgs*, bool*)+0x23e V [libjvm.so+0x6d8b34] JNI_CreateJavaVM+0x74 C [libagentloader.so+0x1146d] JNI_CreateJavaVM+0x8ed --------------- P R O C E S S --------------- Java Threads: ( => current thread ) Other Threads: =>0x00007f6c0c00a800 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=18192, stack(0x00007f6c15b71000,0x00007f6c15c71000)] VM state:not at safepoint (not fully initialized) VM Mutex/Monitor currently owned by a thread: None GC Heap History (0 events): No events Deoptimization events (0 events): No events Classes redefined (0 events): No events Internal exceptions (0 events): No events Events (0 events): No events --------------- S Y S T E M --------------- OS:CentOS Linux release 7.7.1908 (Core) uname:Linux 3.10.0-514.6.1.el7.x86_64 #1 SMP Wed Jan 18 13:06:36 UTC 2017 x86_64 libc:glibc 2.17 NPTL 2.17 rlimit: STACK 8192k, CORE 0k, NPROC 63433, NOFILE 102536, AS infinity load average:3.00 3.58 2.35 /proc/meminfo: MemTotal: 16260372 kB MemFree: 425348 kB MemAvailable: 367844 kB Buffers: 0 kB Cached: 235068 kB SwapCached: 148856 kB Active: 13576768 kB Inactive: 1854056 kB Active(anon): 13530096 kB Inactive(anon): 1771152 kB Active(file): 46672 kB Inactive(file): 82904 kB Unevictable: 0 kB Mlocked: 0 kB SwapTotal: 4194300 kB SwapFree: 874708 kB Dirty: 488 kB Writeback: 0 kB AnonPages: 15048848 kB Mapped: 49124 kB Shmem: 105552 kB Slab: 162980 kB SReclaimable: 94124 kB SUnreclaim: 68856 kB KernelStack: 27168 kB PageTables: 65984 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 12324484 kB Committed_AS: 33627812 kB VmallocTotal: 34359738367 kB VmallocUsed: 195136 kB VmallocChunk: 34359326716 kB HardwareCorrupted: 0 kB AnonHugePages: 5726208 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 87936 kB DirectMap2M: 5154816 kB DirectMap1G: 13631488 kB CPU:total 8 (initial active 8) (1 cores per cpu, 1 threads per core) family 6 model 79 stepping 1, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, aes, clmul, erms, 3dnowpref, tsc, tscinvbit /proc/cpuinfo: processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 79 model name : Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz stepping : 1 microcode : 0xb00001b cpu MHz : 2199.038 cache size : 30720 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 20 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep bogomips : 4399.99 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 79 model name : Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz stepping : 1 microcode : 0xb00001b cpu MHz : 2199.038 cache size : 30720 KB physical id : 2 siblings : 1 core id : 0 cpu cores : 1 apicid : 2 initial apicid : 2 fpu : yes fpu_exception : yes cpuid level : 20 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep bogomips : 4399.99 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: processor : 2 vendor_id : GenuineIntel cpu family : 6 model : 79 model name : Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz stepping : 1 microcode : 0xb00001b cpu MHz : 2199.038 cache size : 30720 KB physical id : 4 siblings : 1 core id : 0 cpu cores : 1 apicid : 4 initial apicid : 4 fpu : yes fpu_exception : yes cpuid level : 20 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep bogomips : 4399.99 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: processor : 3 vendor_id : GenuineIntel cpu family : 6 model : 79 model name : Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz stepping : 1 microcode : 0xb00001b cpu MHz : 2199.038 cache size : 30720 KB physical id : 6 siblings : 1 core id : 0 cpu cores : 1 apicid : 6 initial apicid : 6 fpu : yes fpu_exception : yes cpuid level : 20 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep bogomips : 4399.99 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: processor : 4 vendor_id : GenuineIntel cpu family : 6 model : 79 model name : Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz stepping : 1 microcode : 0xb00001b cpu MHz : 2199.038 cache size : 30720 KB physical id : 8 siblings : 1 core id : 0 cpu cores : 1 apicid : 8 initial apicid : 8 fpu : yes fpu_exception : yes cpuid level : 20 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep bogomips : 4399.99 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: processor : 5 vendor_id : GenuineIntel cpu family : 6 model : 79 model name : Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz stepping : 1 microcode : 0xb00001b cpu MHz : 2199.038 cache size : 30720 KB physical id : 10 siblings : 1 core id : 0 cpu cores : 1 apicid : 10 initial apicid : 10 fpu : yes fpu_exception : yes cpuid level : 20 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep bogomips : 4399.99 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: processor : 6 vendor_id : GenuineIntel cpu family : 6 model : 79 model name : Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz stepping : 1 microcode : 0xb00001b cpu MHz : 2199.038 cache size : 30720 KB physical id : 12 siblings : 1 core id : 0 cpu cores : 1 apicid : 12 initial apicid : 12 fpu : yes fpu_exception : yes cpuid level : 20 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep bogomips : 4399.99 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: processor : 7 vendor_id : GenuineIntel cpu family : 6 model : 79 model name : Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz stepping : 1 microcode : 0xb00001b cpu MHz : 2199.038 cache size : 30720 KB physical id : 14 siblings : 1 core id : 0 cpu cores : 1 apicid : 14 initial apicid : 14 fpu : yes fpu_exception : yes cpuid level : 20 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase smep bogomips : 4399.99 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: Memory: 4k page, physical 16260372k(425348k free), swap 4194300k(874708k free) vm_info: Java HotSpot(TM) 64-Bit Server VM (25.172-b11) for linux-amd64 JRE (1.8.0_172-b11), built on Mar 28 2018 21:44:09 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8) time: Sun Oct 9 00:28:39 2022 elapsed time: 0 seconds (0d 0h 0m 0s)