一、背景
周末仓内派单程序奔溃，程序启动时间：2021-07-22 10:38:19；程序奔溃时间：2021-07-25 11:12:27。
2021-07-25 11:30左右群里有反映收不到拣货派单结果，经查询程序崩溃了。程序于2021-07-25 12:06:49重新启动。启动时添加了JVM的GC垃圾回收监控，-Xloggc:./wse-gc-%t.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=20M -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCCause。
目前该参内派单程序刚上线不久，业务量不大，为了便于及时修改项目相关算法及配置，单点部署在34这台机器上，等项目稳定后迁移至docker环境并分布式部署。所以目前属于单点没有监控的状态。
相关分析报告情况如下。
二、相关环境情况
2.1.机器物理内存和核数
64G/16C
[ops@dis-algo34 pick-task-assign-job]$ free -m
total used free shared buff/cache available
Mem: 64264 17561 37012 9 9690 46106
Swap: 0 0 0
[ops@dis-algo34 pick-task-assign-job]$ top
top - 16:16:33 up 95 days, 4:40, 1 user, load average: 0.05, 0.16, 0.26
Tasks: 220 total, 1 running, 215 sleeping, 4 stopped, 0 zombie
%Cpu0 : 0.3 us, 0.0 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 : 0.7 us, 0.3 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu5 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu6 : 0.3 us, 0.3 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu7 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu8 : 0.3 us, 0.0 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu9 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu10 : 0.3 us, 0.0 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu11 : 0.3 us, 0.3 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu12 : 0.3 us, 0.3 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu13 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu14 : 0.3 us, 0.3 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu15 : 0.3 us, 0.7 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 65806676 total, 38360376 free, 17231528 used, 10214772 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 47969452 avail Mem
2.2.仓内派单java程序程序启动参数
我们使用的是jdk1.8，程序启动参数如下:
java -Dspring.profiles.active=tce -Djava.library.path=/data/cplex -jar app.jar
由于启动时没有指定具体的堆栈大小，使用默认配置，查看默认配置：
[ops@dis-algo34 pick-task-assign-job]$ java -XX:+PrintCommandLineFlags -version
-XX:InitialHeapSize=1052906816 -XX:MaxHeapSize=16846509056 -XX:+PrintCommandLineFlags -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseParallelGC
java version "1.8.0_251"
Java(TM) SE Runtime Environment (build 1.8.0_251-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.251-b08, mixed mode)
[ops@dis-algo34 pick-task-assign-job]$
初始堆内存InitialHeapSize：1052906816字节=1G ；最大堆内存MaxHeapSize：16846509056字节=16G。JVM初始分配的内存默认是物理内存的1/64；JVM最大分配的内存默认是物理内存的1/4。
默认空余堆内存小于40%时，JVM就会增大堆直到-Xmx的最大限制；空余堆内存大于70%时，JVM会减少堆直到-Xms的最小限制。因此服务器一般设置-Xms、-Xmx相等以避免在每次GC后调整堆的大小。使用-XX:-UseAdaptiveSizePolicy 避免堆大小的自适应扩缩容。
2.3.机器使用情况
目前该机器是公用的，上面部署了很多其他程序。
派单程序崩溃时其他程序的资源使用情况如下：
pid=28179的程序内存一直是超过40%，cpu的瞬时使用率也很高。这个程序一直占用40%以上的内存可能也是派单程序奔溃的外部原因之一。
三、日志分析
程序上线运行了一周多的时间，周六突然程序挂掉了，日志的最后几行显示：
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000513200000, 3075473408, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 3075473408 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /data/Projects/pick-task-assign-job/hs_err_pid5421.log
上面报错：Cannot allocate memory，不能分配内存；“内存不足，Java运行时环境无法继续。本地内存分配3075473408字节（3G）失败。错误信息存储在/data/Projects/pick-task-assign-job/hs_err_pid5421.log文件中”。
JVM出现的致命错误，会在默认工作目录下面生成了hs_err_pid.log 日志文件。其中包含了导致JVM crash 的重要信息（可以在虚拟参数中添加 -XX:ErrorFie=./hss_err_pid.log指定生成路径。）
hs_err_pid.log 日志文件包含：日志头文件，导致crash的线程信息，所有的线程信息，安全点和锁信息，堆信息，本地代码缓存，编译事件，gc相关记录，jvm内存映射，jvm启动参数，服务器信息。下面简单看下hs_err_pid5421.log文件：
[ops@dis-algo34 pick-task-assign-job]$ cat hs_err_pid5421.log
# 1.首先是日志头信息。这里已经详细告诉我们原因了：进程号5421,由于物理内存不足，已经出现Out of Memory Error错误。同时提供了解决方案：减轻机器内存负载，设置Xmx /xms减少java堆内存空间大小，如果是64位的操作系统建议使用64bit java，减少java线程数量，减少每个线程堆栈大小-Xss。
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 3075473408 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# JVM is running with Zero Based Compressed Oops mode in which the Java heap is
# placed in the first 32GB address space. The Java Heap base address is the
# maximum limit for the native heap growth. Please use -XX:HeapBaseMinAddress
# to set the Java Heap base and to place the Java Heap above 32GB virtual address.
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2749), pid=5421, tid=0x00007ff48d6d6700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_251-b08) (build 1.8.0_251-b08)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.251-b08 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
#2.导致程序crash的线程信息。导致程序奔溃的线程id=5442，线程类型是：VMThread，这种类型的线程是jvm内部线程。还有其他类型的线程，如JavaThread表示执行的是java线程，GCTaskThread是执行gc的线程，CompilerThread是用来调用JITing实时编译装卸class，WatcherThread是jvm周期性任务调度的线程。
--------------- T H R E A D ---------------
Current thread (0x00007ff4e4272000): VMThread [stack: 0x00007ff48d5d7000,0x00007ff48d6d7000] [id=5442]
Stack: [0x00007ff48d5d7000,0x00007ff48d6d7000], sp=0x00007ff48d6d53f0, free space=1016k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xad9dc5] VMError::report_and_die()+0x2e5
V [libjvm.so+0x4e1d67] report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x67 # 内存不足导致分配失败这就是JVM发生致命的原因所在
V [libjvm.so+0x912cb0] os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x100
V [libjvm.so+0x90a2df] os::commit_memory(char*, unsigned long, unsigned long, bool)+0x1f
V [libjvm.so+0x98f0e6] PSVirtualSpace::expand_by(unsigned long)+0x56
V [libjvm.so+0x97e3bb] PSOldGen::expand_by(unsigned long) [clone .part.28]+0x1b
V [libjvm.so+0x97e578] PSOldGen::expand(unsigned long) [clone .part.29]+0xa8
V [libjvm.so+0x97f121] PSOldGen::resize(unsigned long)+0xe1
V [libjvm.so+0x98525c] PSParallelCompact::invoke_no_policy(bool) [clone .part.118]+0xa2c
V [libjvm.so+0x98d075] PSScavenge::invoke()+0x215
V [libjvm.so+0x93cbf8] ParallelScavengeHeap::failed_mem_allocate(unsigned long)+0x68 #内存分配失败
V [libjvm.so+0xadb913] VM_ParallelGCFailedAllocation::doit()+0x93#内存分配失败
V [libjvm.so+0xae0b66] VM_Operation::evaluate()+0x46
V [libjvm.so+0xadee9d] VMThread::evaluate_operation(VM_Operation*) [clone .constprop.44]+0xcd
V [libjvm.so+0xadf483] VMThread::loop()+0x3a3
V [libjvm.so+0xadf858] VMThread::run()+0x78
V [libjvm.so+0x9102e2] java_start(Thread*)+0x102
#下面这句：ParallelGCFailedAllocation，表示执行Parallel垃圾回收过程中内存分派失败，这可能导致的内存溢出。
VM_Operation (0x00007ff27a5f1ba0): ParallelGCFailedAllocation, mode: safepoint, requested by thread 0x00007fefa1290000
#3.下面是程序奔溃时正在运行的线程信息，其中双引号中是线程名称。最后一个线程就是导致程序奔溃的线程
--------------- P R O C E S S ---------------
#3.1.java线程
Java Threads: ( => current thread )
0x00007ff1ba676800 JavaThread "Unknown thread" [_thread_blocked, id=24432, stack(0x00007ff281a9d000,0x00007ff28229d000)]
0x00007ff306e4d000 JavaThread "Unknown thread" [_thread_blocked, id=24431, stack(0x00007ff27b5f6000,0x00007ff27bdf6000)]
0x00007ff2ffb8b000 JavaThread "Unknown thread" [_thread_blocked, id=24430, stack(0x00007ff27adf5000,0x00007ff27b5f5000)]
0x00007ff30204d000 JavaThread "Unknown thread" [_thread_blocked, id=24429, stack(0x00007ff27a5f4000,0x00007ff27adf4000)]
0x00007fefa1290000 JavaThread "Unknown thread" [_thread_blocked, id=24428, stack(0x00007ff279df3000,0x00007ff27a5f3000)]
0x00007fefc871a800 JavaThread "Keep-Alive-Timer" daemon [_thread_blocked, id=24450, stack(0x00007ff282cdd000,0x00007ff282dde000)]
0x00007fee88687000 JavaThread "ForkJoinPool.commonPool-worker-13" daemon [_thread_blocked, id=5498, stack(0x00007ff287eff000,0x00007ff288000000)]
0x00007fedbfaef000 JavaThread "ForkJoinPool.commonPool-worker-4" daemon [_thread_blocked, id=32156, stack(0x00007ff285b09000,0x00007ff285c0a000)]
0x00007fee9eb0f800 JavaThread "ForkJoinPool.commonPool-worker-11" daemon [_thread_in_native, id=705, stack(0x00007ff282bdc000,0x00007ff282cdd000)]
0x00007ff4cb461800 JavaThread "ForkJoinPool.commonPool-worker-2" daemon [_thread_blocked, id=30106, stack(0x00007ff2840ef000,0x00007ff2841f0000)]
0x00007ff356726000 JavaThread "ForkJoinPool.commonPool-worker-5" daemon [_thread_blocked, id=13133, stack(0x00007ff28229d000,0x00007ff28239e000)]
0x00007ff3c8077800 JavaThread "lettuce-eventExecutorLoop-1-16" daemon [_thread_blocked, id=28265, stack(0x00007ff2839ea000,0x00007ff283aeb000)]
0x00007ff3c8077000 JavaThread "lettuce-eventExecutorLoop-1-15" daemon [_thread_blocked, id=28235, stack(0x00007ff48a177000,0x00007ff48a278000)]
0x00007ff3c805b000 JavaThread "lettuce-eventExecutorLoop-1-14" daemon [_thread_blocked, id=25593, stack(0x00007ff282dde000,0x00007ff282edf000)]
0x00007ff3c805a800 JavaThread "lettuce-eventExecutorLoop-1-13" daemon [_thread_blocked, id=25569, stack(0x00007ff282edf000,0x00007ff282fe0000)]
0x00007ff3c8059800 JavaThread "lettuce-eventExecutorLoop-1-12" daemon [_thread_blocked, id=22888, stack(0x00007ff282fe0000,0x00007ff2830e1000)]
0x00007ff3c8081000 JavaThread "lettuce-eventExecutorLoop-1-11" daemon [_thread_blocked, id=22872, stack(0x00007ff2832e3000,0x00007ff2833e4000)]
0x00007ff3c8080800 JavaThread "lettuce-eventExecutorLoop-1-10" daemon [_thread_blocked, id=19918, stack(0x00007ff2830e1000,0x00007ff2831e2000)]
0x00007ff3c805d000 JavaThread "lettuce-eventExecutorLoop-1-9" daemon [_thread_blocked, id=19905, stack(0x00007ff2831e2000,0x00007ff2832e3000)]
0x00007ff3c805c800 JavaThread "lettuce-eventExecutorLoop-1-8" daemon [_thread_blocked, id=16974, stack(0x00007ff2833e4000,0x00007ff2834e5000)]
0x00007ff3c805e000 JavaThread "lettuce-eventExecutorLoop-1-7" daemon [_thread_blocked, id=16965, stack(0x00007ff2836e7000,0x00007ff2837e8000)]
0x00007ff3c8051000 JavaThread "lettuce-eventExecutorLoop-1-6" daemon [_thread_blocked, id=14162, stack(0x00007ff2834e5000,0x00007ff2835e6000)]
0x00007ff3c8050000 JavaThread "lettuce-eventExecutorLoop-1-5" daemon [_thread_blocked, id=14161, stack(0x00007ff2835e6000,0x00007ff2836e7000)]
0x00007ff3c807e800 JavaThread "lettuce-eventExecutorLoop-1-4" daemon [_thread_blocked, id=11346, stack(0x00007ff2837e8000,0x00007ff2838e9000)]
0x00007ff3c807b000 JavaThread "lettuce-eventExecutorLoop-1-3" daemon [_thread_blocked, id=11345, stack(0x00007ff2838e9000,0x00007ff2839ea000)]
0x00007ff3c8079800 JavaThread "lettuce-eventExecutorLoop-1-2" daemon [_thread_blocked, id=8628, stack(0x00007ff283aeb000,0x00007ff283bec000)]
0x00007ff3c800e800 JavaThread "lettuce-eventExecutorLoop-1-1" daemon [_thread_blocked, id=8627, stack(0x00007ff283bec000,0x00007ff283ced000)]
0x00007ff334064800 JavaThread "lettuce-timer-3-1" [_thread_blocked, id=8607, stack(0x00007ff283ced000,0x00007ff283dee000)]
0x00007ff3e4008800 JavaThread "logback-8" daemon [_thread_blocked, id=6327, stack(0x00007ff283fee000,0x00007ff2840ef000)]
0x00007ff2cc002000 JavaThread "logback-7" daemon [_thread_blocked, id=6212, stack(0x00007ff2841f0000,0x00007ff2842f1000)]
0x00007ff3b0004800 JavaThread "ConsumeMessageThread_20" [_thread_blocked, id=6200, stack(0x00007ff2847f6000,0x00007ff2848f7000)]
0x00007ff3b8003000 JavaThread "ConsumeMessageThread_19" [_thread_blocked, id=6177, stack(0x00007ff2842f1000,0x00007ff2843f2000)]
0x00007ff310001800 JavaThread "logback-6" daemon [_thread_blocked, id=6162, stack(0x00007ff2843f2000,0x00007ff2844f3000)]
0x00007ff288028000 JavaThread "wse-pool-19" [_thread_blocked, id=6139, stack(0x00007ff2844f3000,0x00007ff2845f4000)]
0x00007ff288026800 JavaThread "wse-pool-18" [_thread_blocked, id=6136, stack(0x00007ff2845f4000,0x00007ff2846f5000)]
0x00007ff3c800b800 JavaThread "ConsumeMessageThread_18" [_thread_blocked, id=6130, stack(0x00007ff2846f5000,0x00007ff2847f6000)]
0x00007ff288025000 JavaThread "wse-pool-17" [_thread_blocked, id=5971, stack(0x00007ff2848f7000,0x00007ff2849f8000)]
0x00007ff288023000 JavaThread "wse-pool-16" [_thread_blocked, id=5970, stack(0x00007ff2849f8000,0x00007ff284af9000)]
0x00007ff288021000 JavaThread "wse-pool-15" [_thread_blocked, id=5969, stack(0x00007ff284af9000,0x00007ff284bfa000)]
0x00007ff288020000 JavaThread "wse-pool-14" [_thread_blocked, id=5968, stack(0x00007ff284bfa000,0x00007ff284cfb000)]
0x00007ff414004800 JavaThread "logback-5" daemon [_thread_blocked, id=5966, stack(0x00007ff285e0a000,0x00007ff285f0b000)]
0x00007ff3d8001000 JavaThread "ConsumeMessageThread_17" [_thread_blocked, id=5958, stack(0x00007ff284cfb000,0x00007ff284dfc000)]
0x00007ff3e0002000 JavaThread "ConsumeMessageThread_20" [_thread_blocked, id=5955, stack(0x00007ff284dfc000,0x00007ff284efd000)]
0x00007ff3c0004800 JavaThread "ConsumeMessageThread_16" [_thread_blocked, id=5953, stack(0x00007ff284efd000,0x00007ff284ffe000)]
0x00007ff28801e800 JavaThread "wse-pool-13" [_thread_blocked, id=5868, stack(0x00007ff284ffe000,0x00007ff2850ff000)]
0x00007ff288010000 JavaThread "wse-pool-12" [_thread_blocked, id=5867, stack(0x00007ff2850ff000,0x00007ff285200000)]
0x00007ff3a0003000 JavaThread "ConsumeMessageThread_15" [_thread_blocked, id=5865, stack(0x00007ff285200000,0x00007ff285301000)]
0x00007ff3ac003000 JavaThread "ConsumeMessageThread_19" [_thread_blocked, id=5861, stack(0x00007ff285301000,0x00007ff285402000)]
0x00007ff3b4003000 JavaThread "ConsumeMessageThread_14" [_thread_blocked, id=5844, stack(0x00007ff285402000,0x00007ff285503000)]
0x00007ff28800f000 JavaThread "wse-pool-11" [_thread_blocked, id=5842, stack(0x00007ff285503000,0x00007ff285604000)]
0x00007ff288001800 JavaThread "wse-pool-10" [_thread_blocked, id=5841, stack(0x00007ff285604000,0x00007ff285705000)]
0x00007ff2cc001000 JavaThread "logback-4" daemon [_thread_blocked, id=5840, stack(0x00007ff285705000,0x00007ff285806000)]
0x00007ff3c8008800 JavaThread "ConsumeMessageThread_18" [_thread_blocked, id=5839, stack(0x00007ff285806000,0x00007ff285907000)]
0x00007ff3b0003000 JavaThread "ConsumeMessageThread_13" [_thread_blocked, id=5830, stack(0x00007ff285907000,0x00007ff285a08000)]
0x00007ff3e0003800 JavaThread "ConsumeMessageThread_17" [_thread_blocked, id=5826, stack(0x00007ff285a08000,0x00007ff285b09000)]
0x00007ff288007000 JavaThread "wse-pool-9" [_thread_blocked, id=5807, stack(0x00007ff2808fd000,0x00007ff2809fe000)]
0x00007ff288018000 JavaThread "wse-pool-8" [_thread_blocked, id=5806, stack(0x00007ff2809fe000,0x00007ff280aff000)]
0x00007ff3cc006800 JavaThread "ConsumeMessageThread_16" [_thread_blocked, id=5784, stack(0x00007ff27e5fb000,0x00007ff27e6fc000)]
0x00007ff314bf4000 JavaThread "kafka-producer-network-thread | producer-1" daemon [_thread_in_native, id=5764, stack(0x00007ff2f82e9000,0x00007ff2f83ea000)]
0x00007ff314bae000 JavaThread "Druid-ConnectionPool-Destroy-519492428" daemon [_thread_blocked, id=5763, stack(0x00007ff2f85ea000,0x00007ff2f86eb000)]
0x00007ff314bad000 JavaThread "Druid-ConnectionPool-Create-519492428" daemon [_thread_blocked, id=5762, stack(0x00007ff2f86eb000,0x00007ff2f87ec000)]
0x00007ff314baa800 JavaThread "Druid-ConnectionPool-Destroy-1052253947" daemon [_thread_blocked, id=5761, stack(0x00007ff2f87ec000,0x00007ff2f88ed000)]
0x00007ff314ba9800 JavaThread "Druid-ConnectionPool-Create-1052253947" daemon [_thread_blocked, id=5760, stack(0x00007ff2f88ed000,0x00007ff2f89ee000)]
0x00007ff3a0001000 JavaThread "ConsumeMessageThread_12" [_thread_blocked, id=5748, stack(0x00007ff2f8dee000,0x00007ff2f8eef000)]
0x00007ff288016000 JavaThread "wse-pool-7" [_thread_blocked, id=5744, stack(0x00007ff2f90ef000,0x00007ff2f91f0000)]
0x00007ff288014800 JavaThread "wse-pool-6" [_thread_blocked, id=5743, stack(0x00007ff2f91f0000,0x00007ff2f92f1000)]
0x00007ff288012800 JavaThread "wse-pool-5" [_thread_blocked, id=5742, stack(0x00007ff2f92f1000,0x00007ff2f93f2000)]
0x00007ff288011000 JavaThread "wse-pool-4" [_thread_blocked, id=5741, stack(0x00007ff2f93f2000,0x00007ff2f94f3000)]
0x00007ff414002800 JavaThread "logback-3" daemon [_thread_blocked, id=5739, stack(0x00007ff2f94f3000,0x00007ff2f95f4000)]
0x00007ff3ac001000 JavaThread "ConsumeMessageThread_15" [_thread_blocked, id=5738, stack(0x00007ff2f95f4000,0x00007ff2f96f5000)]
0x00007ff3a8001000 JavaThread "ConsumeMessageThread_11" [_thread_blocked, id=5727, stack(0x00007ff2f96f5000,0x00007ff2f97f6000)]
0x00007ff3b4001000 JavaThread "ConsumeMessageThread_14" [_thread_blocked, id=5720, stack(0x00007ff2f97f6000,0x00007ff2f98f7000)]
0x00007ff28800d800 JavaThread "wse-pool-3" [_thread_blocked, id=5703, stack(0x00007ff2f98f7000,0x00007ff2f99f8000)]
0x00007ff28800b800 JavaThread "wse-pool-2" [_thread_blocked, id=5702, stack(0x00007ff2f99f8000,0x00007ff2f9af9000)]
0x00007ff3b0001000 JavaThread "ConsumeMessageThread_13" [_thread_blocked, id=5697, stack(0x00007ff2f9af9000,0x00007ff2f9bfa000)]
0x00007ff2d4021000 JavaThread "Druid-ConnectionPool-Destroy-669284403" daemon [_thread_blocked, id=5689, stack(0x00007ff2f9bfa000,0x00007ff2f9cfb000)]
0x00007ff2d4020000 JavaThread "Druid-ConnectionPool-Create-669284403" daemon [_thread_blocked, id=5688, stack(0x00007ff2f9cfb000,0x00007ff2f9dfc000)]
0x00007ff2d4015000 JavaThread "Druid-ConnectionPool-Destroy-85748029" daemon [_thread_blocked, id=5687, stack(0x00007ff2f9dfc000,0x00007ff2f9efd000)]
0x00007ff2d4014000 JavaThread "Druid-ConnectionPool-Create-85748029" daemon [_thread_blocked, id=5686, stack(0x00007ff2f9efd000,0x00007ff2f9ffe000)]
0x00007ff2c8008000 JavaThread "Druid-ConnectionPool-Destroy-294111720" daemon [_thread_blocked, id=5683, stack(0x00007ff3080f0000,0x00007ff3081f1000)]
0x00007ff2c8007000 JavaThread "Druid-ConnectionPool-Create-294111720" daemon [_thread_blocked, id=5682, stack(0x00007ff3081f1000,0x00007ff3082f2000)]
0x00007ff288009800 JavaThread "wse-pool-1" [_thread_blocked, id=5661, stack(0x00007ff3082f2000,0x00007ff3083f3000)]
0x00007ff288008800 JavaThread "wse-pool-0" [_thread_blocked, id=5660, stack(0x00007ff3083f3000,0x00007ff3084f4000)]
0x00007ff414001000 JavaThread "logback-2" daemon [_thread_blocked, id=5659, stack(0x00007ff3084f4000,0x00007ff3085f5000)]
0x00007ff4e4009800 JavaThread "DestroyJavaVM" [_thread_blocked, id=5422, stack(0x00007ff4ea15a000,0x00007ff4ea25a000)]
0x00007ff4e4e8c800 JavaThread "http-nio-8080-Acceptor-0" daemon [_thread_in_native, id=5649, stack(0x00007ff3087f5000,0x00007ff3088f6000)]
0x00007ff4e4c83800 JavaThread "http-nio-8080-ClientPoller-1" daemon [_thread_blocked, id=5648, stack(0x00007ff3088f6000,0x00007ff3089f7000)]
0x00007ff4e5cb1000 JavaThread "http-nio-8080-ClientPoller-0" daemon [_thread_blocked, id=5647, stack(0x00007ff3089f7000,0x00007ff308af8000)]
0x00007ff4e47a8800 JavaThread "http-nio-8080-exec-10" daemon [_thread_blocked, id=5646, stack(0x00007ff308af8000,0x00007ff308bf9000)]
0x00007ff4e4954000 JavaThread "http-nio-8080-exec-9" daemon [_thread_blocked, id=5645, stack(0x00007ff308bf9000,0x00007ff308cfa000)]
0x00007ff4e4bd7800 JavaThread "http-nio-8080-exec-8" daemon [_thread_blocked, id=5644, stack(0x00007ff308cfa000,0x00007ff308dfb000)]
0x00007ff4e5018000 JavaThread "http-nio-8080-exec-7" daemon [_thread_blocked, id=5643, stack(0x00007ff308dfb000,0x00007ff308efc000)]
0x00007ff4e516f800 JavaThread "http-nio-8080-exec-6" daemon [_thread_blocked, id=5642, stack(0x00007ff308efc000,0x00007ff308ffd000)]
0x00007ff4e49b1800 JavaThread "http-nio-8080-exec-5" daemon [_thread_blocked, id=5641, stack(0x00007ff3180f0000,0x00007ff3181f1000)]
0x00007ff4e56cb800 JavaThread "http-nio-8080-exec-4" daemon [_thread_blocked, id=5640, stack(0x00007ff3181f1000,0x00007ff3182f2000)]
0x00007ff4e4f40000 JavaThread "http-nio-8080-exec-3" daemon [_thread_blocked, id=5639, stack(0x00007ff3182f2000,0x00007ff3183f3000)]
0x00007ff4e4d20000 JavaThread "http-nio-8080-exec-2" daemon [_thread_blocked, id=5638, stack(0x00007ff3183f3000,0x00007ff3184f4000)]
0x00007ff4e4d1f000 JavaThread "http-nio-8080-exec-1" daemon [_thread_blocked, id=5637, stack(0x00007ff3184f4000,0x00007ff3185f5000)]
0x00007ff4e4988800 JavaThread "NioBlockingSelector.BlockPoller-1" daemon [_thread_blocked, id=5636, stack(0x00007ff3185f5000,0x00007ff3186f6000)]
0x00007ff4e4fbf000 JavaThread "scheduling-1" [_thread_blocked, id=5635, stack(0x00007ff3186f6000,0x00007ff3187f7000)]
0x00007ff2b0001800 JavaThread "kafka-coordinator-heartbeat-thread | kafkaListenerFeature" daemon [_thread_blocked, id=5633, stack(0x00007ff3187f7000,0x00007ff3188f8000)]
0x00007ff2a0002000 JavaThread "kafka-coordinator-heartbeat-thread | kafkaListenerFeature" daemon [_thread_blocked, id=5632, stack(0x00007ff3188f8000,0x00007ff3189f9000)]
0x00007ff298001800 JavaThread "kafka-coordinator-heartbeat-thread | kafkaListenerFeature" daemon [_thread_blocked, id=5631, stack(0x00007ff3189f9000,0x00007ff318afa000)]
0x00007ff2a8009800 JavaThread "kafka-coordinator-heartbeat-thread | kafkaListenerFeature" daemon [_thread_blocked, id=5630, stack(0x00007ff318afa000,0x00007ff318bfb000)]
0x00007ff290004000 JavaThread "kafka-coordinator-heartbeat-thread | kafkaListenerFeature" daemon [_thread_blocked, id=5629, stack(0x00007ff318bfb000,0x00007ff318cfc000)]
0x00007ff4e5e95000 JavaThread "kafkaListenerFeature-4-C-1" [_thread_blocked, id=5628, stack(0x00007ff318efc000,0x00007ff318ffd000)]
0x00007ff4e5e93800 JavaThread "ThreadPoolTaskScheduler-1" [_thread_blocked, id=5627, stack(0x00007ff3280f2000,0x00007ff3281f3000)]
0x00007ff4e5e78800 JavaThread "kafkaListenerFeature-3-C-1" [_thread_blocked, id=5626, stack(0x00007ff3281f3000,0x00007ff3282f4000)]
0x00007ff4e5e77000 JavaThread "ThreadPoolTaskScheduler-1" [_thread_blocked, id=5625, stack(0x00007ff3282f4000,0x00007ff3283f5000)]
0x00007ff4e5e5c000 JavaThread "kafkaListenerFeature-2-C-1" [_thread_blocked, id=5624, stack(0x00007ff3283f5000,0x00007ff3284f6000)]
0x00007ff4e5e5a000 JavaThread "ThreadPoolTaskScheduler-1" [_thread_blocked, id=5623, stack(0x00007ff3284f6000,0x00007ff3285f7000)]
0x00007ff4e5e3f800 JavaThread "kafkaListenerFeature-1-C-1" [_thread_blocked, id=5622, stack(0x00007ff3285f7000,0x00007ff3286f8000)]
0x00007ff4e5e3f000 JavaThread "ThreadPoolTaskScheduler-1" [_thread_blocked, id=5621, stack(0x00007ff3286f8000,0x00007ff3287f9000)]
0x00007ff4e5e24000 JavaThread "kafkaListenerFeature-0-C-1" [_thread_blocked, id=5620, stack(0x00007ff3287f9000,0x00007ff3288fa000)]
0x00007ff4e5e19000 JavaThread "ThreadPoolTaskScheduler-1" [_thread_blocked, id=5619, stack(0x00007ff3288fa000,0x00007ff3289fb000)]
0x00007ff4e5132000 JavaThread "quartzScheduler_QuartzSchedulerThread" [_thread_blocked, id=5617, stack(0x00007ff328bfb000,0x00007ff328cfc000)]
0x00007ff4e5993800 JavaThread "quartzScheduler_Worker-10" [_thread_blocked, id=5616, stack(0x00007ff328efc000,0x00007ff328ffd000)]
0x00007ff4e5991800 JavaThread "quartzScheduler_Worker-9" [_thread_blocked, id=5615, stack(0x00007ff36c0f2000,0x00007ff36c1f3000)]
0x00007ff4e53f0800 JavaThread "quartzScheduler_Worker-8" [_thread_blocked, id=5614, stack(0x00007ff36c1f3000,0x00007ff36c2f4000)]
0x00007ff4e53ef000 JavaThread "quartzScheduler_Worker-7" [_thread_blocked, id=5613, stack(0x00007ff36c2f4000,0x00007ff36c3f5000)]
0x00007ff4e5518800 JavaThread "quartzScheduler_Worker-6" [_thread_blocked, id=5612, stack(0x00007ff36c3f5000,0x00007ff36c4f6000)]
0x00007ff4e5517800 JavaThread "quartzScheduler_Worker-5" [_thread_blocked, id=5611, stack(0x00007ff36c4f6000,0x00007ff36c5f7000)]
0x00007ff4e5972800 JavaThread "quartzScheduler_Worker-4" [_thread_blocked, id=5610, stack(0x00007ff36c5f7000,0x00007ff36c6f8000)]
0x00007ff4e5971800 JavaThread "quartzScheduler_Worker-3" [_thread_blocked, id=5609, stack(0x00007ff36c6f8000,0x00007ff36c7f9000)]
0x00007ff4e4a51800 JavaThread "quartzScheduler_Worker-2" [_thread_blocked, id=5608, stack(0x00007ff36c7f9000,0x00007ff36c8fa000)]
0x00007ff4e4a50800 JavaThread "quartzScheduler_Worker-1" [_thread_blocked, id=5607, stack(0x00007ff36c8fa000,0x00007ff36c9fb000)]
0x00007ff37c19a000 JavaThread "Druid-ConnectionPool-Destroy-1470358122" daemon [_thread_blocked, id=5606, stack(0x00007ff36cbfb000,0x00007ff36ccfc000)]
0x00007ff37c1a4800 JavaThread "Druid-ConnectionPool-Create-1470358122" daemon [_thread_blocked, id=5605, stack(0x00007ff36ccfc000,0x00007ff36cdfd000)]
0x00007ff35c04f000 JavaThread "Abandoned connection cleanup thread" daemon [_thread_blocked, id=5603, stack(0x00007ff4542b8000,0x00007ff4543b9000)]
0x00007ff38c3a9800 JavaThread "lettuce-epollEventLoop-4-16" daemon [_thread_blocked, id=5567, stack(0x00007ff4547b9000,0x00007ff4548ba000)]
0x00007ff38c3a7000 JavaThread "lettuce-epollEventLoop-4-15" daemon [_thread_blocked, id=5566, stack(0x00007ff4548ba000,0x00007ff4549bb000)]
0x00007ff38c3a5000 JavaThread "lettuce-epollEventLoop-4-14" daemon [_thread_blocked, id=5565, stack(0x00007ff4549bb000,0x00007ff454abc000)]
0x00007ff38c3a3000 JavaThread "lettuce-epollEventLoop-4-13" daemon [_thread_blocked, id=5564, stack(0x00007ff4700c8000,0x00007ff4701c9000)]
0x00007ff38c3a1800 JavaThread "lettuce-epollEventLoop-4-12" daemon [_thread_blocked, id=5563, stack(0x00007ff4701c9000,0x00007ff4702ca000)]
0x00007ff38c39f800 JavaThread "lettuce-epollEventLoop-4-11" daemon [_thread_blocked, id=5562, stack(0x00007ff4702ca000,0x00007ff4703cb000)]
0x00007ff38c39d800 JavaThread "lettuce-epollEventLoop-4-10" daemon [_thread_blocked, id=5561, stack(0x00007ff4703cb000,0x00007ff4704cc000)]
0x00007ff38c39b800 JavaThread "lettuce-epollEventLoop-4-9" daemon [_thread_blocked, id=5560, stack(0x00007ff4704cc000,0x00007ff4705cd000)]
0x00007ff38c39a000 JavaThread "lettuce-epollEventLoop-4-8" daemon [_thread_blocked, id=5559, stack(0x00007ff4705cd000,0x00007ff4706ce000)]
0x00007ff38c398000 JavaThread "lettuce-epollEventLoop-4-7" daemon [_thread_blocked, id=5558, stack(0x00007ff4706ce000,0x00007ff4707cf000)]
0x00007ff38c396000 JavaThread "lettuce-epollEventLoop-4-6" daemon [_thread_blocked, id=5557, stack(0x00007ff4707cf000,0x00007ff4708d0000)]
0x00007ff38c394800 JavaThread "lettuce-epollEventLoop-4-5" daemon [_thread_blocked, id=5556, stack(0x00007ff4708d0000,0x00007ff4709d1000)]
0x00007ff38c392800 JavaThread "lettuce-epollEventLoop-4-4" daemon [_thread_blocked, id=5555, stack(0x00007ff4709d1000,0x00007ff470ad2000)]
0x00007ff38c391000 JavaThread "lettuce-epollEventLoop-4-3" daemon [_thread_blocked, id=5554, stack(0x00007ff470ad2000,0x00007ff470bd3000)]
0x00007ff38c38f000 JavaThread "lettuce-epollEventLoop-4-2" daemon [_thread_blocked, id=5553, stack(0x00007ff470bd3000,0x00007ff470cd4000)]
0x00007ff38c336000 JavaThread "lettuce-epollEventLoop-4-1" daemon [_thread_blocked, id=5552, stack(0x00007ff471cd5000,0x00007ff471dd6000)]
0x00007ff3d400a000 JavaThread "ConsumeMessageThread_8" [_thread_blocked, id=5551, stack(0x00007ff4723e4000,0x00007ff4724e5000)]
0x00007ff3cc004800 JavaThread "ConsumeMessageThread_9" [_thread_blocked, id=5550, stack(0x00007ff4724e5000,0x00007ff4725e6000)]
0x00007ff3e0009000 JavaThread "ConsumeMessageThread_10" [_thread_blocked, id=5549, stack(0x00007ff4725e6000,0x00007ff4726e7000)]
0x00007ff3c8006800 JavaThread "ConsumeMessageThread_7" [_thread_blocked, id=5548, stack(0x00007ff4726e7000,0x00007ff4727e8000)]
0x00007ff3c8004800 JavaThread "ConsumeMessageThread_6" [_thread_blocked, id=5547, stack(0x00007ff4727e8000,0x00007ff4728e9000)]
0x00007ff3d8005000 JavaThread "ConsumeMessageThread_5" [_thread_blocked, id=5546, stack(0x00007ff4728e9000,0x00007ff4729ea000)]
0x00007ff3d0005000 JavaThread "ConsumeMessageThread_4" [_thread_blocked, id=5545, stack(0x00007ff4729ea000,0x00007ff472aeb000)]
0x00007ff3c0003000 JavaThread "ConsumeMessageThread_3" [_thread_blocked, id=5544, stack(0x00007ff472aeb000,0x00007ff472bec000)]
0x00007ff3b8001000 JavaThread "ConsumeMessageThread_2" [_thread_blocked, id=5543, stack(0x00007ff472bec000,0x00007ff472ced000)]
0x00007ff3bc001000 JavaThread "ConsumeMessageThread_1" [_thread_blocked, id=5542, stack(0x00007ff472ced000,0x00007ff472dee000)]
0x00007ff3e0006800 JavaThread "ConsumeMessageThread_12" [_thread_blocked, id=5541, stack(0x00007ff472dee000,0x00007ff472eef000)]
0x00007ff3d4008800 JavaThread "ConsumeMessageThread_10" [_thread_blocked, id=5540, stack(0x00007ff472eef000,0x00007ff472ff0000)]
0x00007ff3a4008000 JavaThread "ConsumeMessageThread_1" [_thread_blocked, id=5539, stack(0x00007ff472ff0000,0x00007ff4730f1000)]
0x00007ff3e0005000 JavaThread "ConsumeMessageThread_3" [_thread_blocked, id=5538, stack(0x00007ff4730f1000,0x00007ff4731f2000)]
0x00007ff3c8003000 JavaThread "ConsumeMessageThread_9" [_thread_blocked, id=5537, stack(0x00007ff4731f2000,0x00007ff4732f3000)]
0x00007ff3cc003000 JavaThread "ConsumeMessageThread_11" [_thread_blocked, id=5536, stack(0x00007ff4732f3000,0x00007ff4733f4000)]
0x00007ff3d8003000 JavaThread "ConsumeMessageThread_7" [_thread_blocked, id=5535, stack(0x00007ff4733f4000,0x00007ff4734f5000)]
0x00007ff3cc001000 JavaThread "ConsumeMessageThread_5" [_thread_blocked, id=5534, stack(0x00007ff4734f5000,0x00007ff4735f6000)]
0x00007ff3d0004000 JavaThread "ConsumeMessageThread_2" [_thread_blocked, id=5533, stack(0x00007ff4735f6000,0x00007ff4736f7000)]
0x00007ff3d4006800 JavaThread "ConsumeMessageThread_6" [_thread_blocked, id=5532, stack(0x00007ff4736f7000,0x00007ff4737f8000)]
0x00007ff3c8001000 JavaThread "ConsumeMessageThread_8" [_thread_blocked, id=5531, stack(0x00007ff4737f8000,0x00007ff4738f9000)]
0x00007ff3c0001000 JavaThread "ConsumeMessageThread_4" [_thread_blocked, id=5530, stack(0x00007ff4738f9000,0x00007ff4739fa000)]
0x00007ff3f00e3800 JavaThread "NettyClientPublicExecutor_16" [_thread_blocked, id=5529, stack(0x00007ff4739fa000,0x00007ff473afb000)]
0x00007ff3dc00d800 JavaThread "NettyClientPublicExecutor_15" [_thread_blocked, id=5528, stack(0x00007ff473afb000,0x00007ff473bfc000)]
0x00007ff3e4006800 JavaThread "NettyClientPublicExecutor_14" [_thread_blocked, id=5527, stack(0x00007ff473bfc000,0x00007ff473cfd000)]
0x00007ff3ec008800 JavaThread "NettyClientPublicExecutor_13" [_thread_blocked, id=5526, stack(0x00007ff473cfd000,0x00007ff473dfe000)]
0x00007ff3f00e1800 JavaThread "NettyClientPublicExecutor_12" [_thread_blocked, id=5525, stack(0x00007ff473dfe000,0x00007ff473eff000)]
0x00007ff3dc00b800 JavaThread "NettyClientPublicExecutor_11" [_thread_blocked, id=5524, stack(0x00007ff473eff000,0x00007ff474000000)]
0x00007ff3dc00a000 JavaThread "NettyClientPublicExecutor_10" [_thread_blocked, id=5523, stack(0x00007ff4780ee000,0x00007ff4781ef000)]
0x00007ff3e4004800 JavaThread "NettyClientPublicExecutor_9" [_thread_blocked, id=5522, stack(0x00007ff4781ef000,0x00007ff4782f0000)]
0x00007ff3ec006800 JavaThread "NettyClientPublicExecutor_8" [_thread_blocked, id=5521, stack(0x00007ff4782f0000,0x00007ff4783f1000)]
0x00007ff3f00e0800 JavaThread "NettyClientPublicExecutor_7" [_thread_blocked, id=5520, stack(0x00007ff4783f1000,0x00007ff4784f2000)]
0x00007ff3e4003000 JavaThread "NettyClientPublicExecutor_6" [_thread_blocked, id=5519, stack(0x00007ff4784f2000,0x00007ff4785f3000)]
0x00007ff3ec005000 JavaThread "NettyClientPublicExecutor_5" [_thread_blocked, id=5518, stack(0x00007ff4785f3000,0x00007ff4786f4000)]
0x00007ff3f0001800 JavaThread "NettyClientPublicExecutor_4" [_thread_blocked, id=5517, stack(0x00007ff4786f4000,0x00007ff4787f5000)]
0x00007ff3dc001000 JavaThread "NettyClientPublicExecutor_3" [_thread_blocked, id=5516, stack(0x00007ff4787f5000,0x00007ff4788f6000)]
0x00007ff3fc030000 JavaThread "NettyClientWorkerThread_4" [_thread_blocked, id=5515, stack(0x00007ff4798f7000,0x00007ff4799f8000)]
0x00007ff3e4001000 JavaThread "NettyClientPublicExecutor_2" [_thread_blocked, id=5514, stack(0x00007ff4799f8000,0x00007ff479af9000)]
0x00007ff3fc02e800 JavaThread "NettyClientWorkerThread_3" [_thread_blocked, id=5513, stack(0x00007ff47aafa000,0x00007ff47abfb000)]
0x00007ff3ec003000 JavaThread "NettyClientPublicExecutor_1" [_thread_blocked, id=5512, stack(0x00007ff47abfb000,0x00007ff47acfc000)]
0x00007ff3fc008000 JavaThread "NettyClientWorkerThread_2" [_thread_blocked, id=5511, stack(0x00007ff47bcfd000,0x00007ff47bdfe000)]
0x00007ff4e4415800 JavaThread "RebalanceService" [_thread_blocked, id=5510, stack(0x00007ff47bdfe000,0x00007ff47beff000)]
0x00007ff4e5c08000 JavaThread "PullMessageService" [_thread_blocked, id=5509, stack(0x00007ff47beff000,0x00007ff47c000000)]
0x00007ff3fc016800 JavaThread "NettyClientWorkerThread_1" [_thread_blocked, id=5495, stack(0x00007ff4822d8000,0x00007ff4823d9000)]
0x00007ff4e5c05000 JavaThread "NettyClientSelector_1" [_thread_blocked, id=5494, stack(0x00007ff4823d9000,0x00007ff4824da000)]
0x00007ff4e5b17000 JavaThread "MQClientFactoryScheduledThread" [_thread_blocked, id=5493, stack(0x00007ff4824da000,0x00007ff4825db000)]
0x00007ff4e52f8000 JavaThread "ClientHouseKeepingService" daemon [_thread_blocked, id=5492, stack(0x00007ff4825db000,0x00007ff4826dc000)]
0x00007ff4e57c2000 JavaThread "container-0" [_thread_blocked, id=5485, stack(0x00007ff4830dc000,0x00007ff4831dd000)]
0x00007ff408003800 JavaThread "Catalina-utility-2" [_thread_blocked, id=5484, stack(0x00007ff4831dd000,0x00007ff4832de000)]
0x00007ff4e57b4000 JavaThread "Catalina-utility-1" [_thread_blocked, id=5483, stack(0x00007ff4832de000,0x00007ff4833df000)]
0x00007ff4e4dc6000 JavaThread "logback-1" daemon [_thread_blocked, id=5465, stack(0x00007ff489354000,0x00007ff489455000)]
0x00007ff4e42de000 JavaThread "Service Thread" daemon [_thread_blocked, id=5458, stack(0x00007ff48a579000,0x00007ff48a67a000)]
0x00007ff4e42d3000 JavaThread "C1 CompilerThread11" daemon [_thread_blocked, id=5457, stack(0x00007ff48a67b000,0x00007ff48a77b000)]
0x00007ff4e42d1000 JavaThread "C1 CompilerThread10" daemon [_thread_blocked, id=5456, stack(0x00007ff48a77c000,0x00007ff48a87c000)]
0x00007ff4e42cf000 JavaThread "C1 CompilerThread9" daemon [_thread_blocked, id=5455, stack(0x00007ff48a87d000,0x00007ff48a97d000)]
0x00007ff4e42cc800 JavaThread "C1 CompilerThread8" daemon [_thread_blocked, id=5454, stack(0x00007ff48a97e000,0x00007ff48aa7e000)]
0x00007ff4e42ca800 JavaThread "C2 CompilerThread7" daemon [_thread_blocked, id=5453, stack(0x00007ff48cacc000,0x00007ff48cbcc000)]
0x00007ff4e42c8800 JavaThread "C2 CompilerThread6" daemon [_thread_blocked, id=5452, stack(0x00007ff48cbcd000,0x00007ff48cccd000)]
0x00007ff4e42c6000 JavaThread "C2 CompilerThread5" daemon [_thread_blocked, id=5451, stack(0x00007ff48ccce000,0x00007ff48cdce000)]
0x00007ff4e42bc000 JavaThread "C2 CompilerThread4" daemon [_thread_blocked, id=5450, stack(0x00007ff48cdcf000,0x00007ff48cecf000)]
0x00007ff4e42ba000 JavaThread "C2 CompilerThread3" daemon [_thread_blocked, id=5449, stack(0x00007ff48ced0000,0x00007ff48cfd0000)]
0x00007ff4e42b7800 JavaThread "C2 CompilerThread2" daemon [_thread_blocked, id=5448, stack(0x00007ff48cfd1000,0x00007ff48d0d1000)]
0x00007ff4e42b6000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=5447, stack(0x00007ff48d0d2000,0x00007ff48d1d2000)]
0x00007ff4e42b3000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=5446, stack(0x00007ff48d1d3000,0x00007ff48d2d3000)]
0x00007ff4e42b1000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=5445, stack(0x00007ff48d2d3000,0x00007ff48d3d4000)]
0x00007ff4e4280000 JavaThread "Finalizer" daemon [_thread_blocked, id=5444, stack(0x00007ff48d3d4000,0x00007ff48d4d5000)]
0x00007ff4e427b800 JavaThread "Reference Handler" daemon [_thread_blocked, id=5443, stack(0x00007ff48d4d5000,0x00007ff48d5d6000)]
#3.2.其它线程。这些事JVM内部线程
#1.虚拟机线程VMThread（这个线程5442就是导致程序奔溃的线程）；
#2.WatcherThread：jvm周期性任务调度的线程
Other Threads:
=>0x00007ff4e4272000 VMThread [stack: 0x00007ff48d5d7000,0x00007ff48d6d7000] [id=5442]
0x00007ff4e42e1000 WatcherThread [stack: 0x00007ff48a479000,0x00007ff48a579000] [id=5459]
#4.程序奔溃时虚拟机状态信息：at safepoint表示所有线程都因为虚拟机等待状态而阻塞，等待一个虚拟机操作完成。这里是因为在垃圾回收，所以会阻塞等待垃圾回收完成。
VM state:at safepoint (normal execution)
#5.程序奔溃时锁信息
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x00007ff4e4006770] ExpandHeap_lock - owner thread: 0x00007ff4e4272000
[0x00007ff4e4006df0] Threads_lock - owner thread: 0x00007ff4e4272000
[0x00007ff4e40072f0] Heap_lock - owner thread: 0x00007fefa1290000
heap address: 0x00000003d3c00000, size: 16068 MB, Compressed Oops mode: Zero based, Oop shift amount: 3
Narrow klass base: 0x0000000000000000, Narrow klass shift: 3
Compressed class space size: 1073741824 Address: 0x00000007c0000000
#6.程序奔溃时堆信息，内存的详细使用情况
Heap:
PSYoungGen total 4890112K, used 143712K [0x0000000671400000, 0x00000007a8300000, 0x00000007c0000000) #年轻代共4G，用了1.1G
eden space 4691456K, 0% used [0x0000000671400000,0x0000000671400000,0x000000078f980000)
from space 198656K, 72% used [0x000000079bd80000,0x00000007a49d8120,0x00000007a7f80000)
to space 200704K, 0% used [0x000000078f980000,0x000000078f980000,0x000000079bd80000)
ParOldGen total 5232640K, used 5232632K [0x00000003d3c00000, 0x0000000513200000, 0x0000000671400000) #老年代5G 用了5G
object space 5232640K, 99% used [0x00000003d3c00000,0x00000005131fe1b8,0x0000000513200000) #空间使用99%
Metaspace used 101388K, capacity 106705K, committed 107712K, reserved 1144832K
class space used 12164K, capacity 13051K, committed 13312K, reserved 1048576K
Card table byte_map: [0x00007ff4d2470000,0x00007ff4d43d3000] byte_map_base: 0x00007ff4d05d2000
Marking Bits: (ParMarkBitMap*) 0x00007ff4e9623fe0
Begin Bits: [0x00007ff4949e0000, 0x00007ff4a44f0000)
End Bits: [0x00007ff4a44f0000, 0x00007ff4b4000000)
Polling page: 0x00007ff4ea266000
#7.本地代码缓存。这是一块用于编译和保存本地代码的内存；注意是本地代码，它和Metaspace（元空间）是不一样的，元空间是用来存放jvm和java类的元数据的。
CodeCache: size=245760Kb used=73103Kb max_used=74041Kb free=172656Kb
bounds [0x00007ff4d4793000, 0x00007ff4d90d3000, 0x00007ff4e3793000]
total_blobs=19194 nmethods=18292 adapters=810
compilation: enabled
#8.编译事件
Compilation events (10 events):
Event: 260825.612 Thread 0x00007ff4e42d1000 nmethod 25775 0x00007ff4d4e2c810 code [0x00007ff4d4e2c9a0, 0x00007ff4d4e2cca8]
Event: 260825.613 Thread 0x00007ff4e42d3000 nmethod 25774 0x00007ff4d6bb1ed0 code [0x00007ff4d6bb20c0, 0x00007ff4d6bb2898]
Event: 260865.508 Thread 0x00007ff4e42cc800 25776 3 sun.misc.Cleaner::create (19 bytes)
Event: 260865.508 Thread 0x00007ff4e42d1000 25778 s 3 sun.misc.Cleaner::add (26 bytes)
Event: 260865.508 Thread 0x00007ff4e42d1000 nmethod 25778 0x00007ff4d4e2c250 code [0x00007ff4d4e2c3c0, 0x00007ff4d4e2c710]
Event: 260865.509 Thread 0x00007ff4e42cc800 nmethod 25776 0x00007ff4d6c90590 code [0x00007ff4d6c90740, 0x00007ff4d6c90df0]
Event: 260865.509 Thread 0x00007ff4e42cc800 25777 3 sun.misc.Cleaner::<init> (24 bytes)
Event: 260865.509 Thread 0x00007ff4e42cc800 nmethod 25777 0x00007ff4d52874d0 code [0x00007ff4d5287640, 0x00007ff4d5287950]
Event: 261186.686 Thread 0x00007ff4e42b6000 25779 4 java.net.SocketTimeoutException::<init> (6 bytes)
Event: 261186.689 Thread 0x00007ff4e42b6000 nmethod 25779 0x00007ff4d6c67ed0 code [0x00007ff4d6c68040, 0x00007ff4d6c683a8]
#9.程序奔溃时最近几次的GC垃圾回收相关记录：10次fullGC，eden取全部回收，OldGen回收后还有98%-99%是被占用的，而且full GC出现的频率很高，可能有内存泄漏的情况
GC Heap History (10 events):
Event: 260986.480 GC heap after
Heap after GC invocations=5466 (full 6):
PSYoungGen total 4808704K, used 189670K [0x0000000671400000, 0x00000007a2d80000, 0x00000007c0000000)
eden space 4613632K, 0% used [0x0000000671400000,0x0000000671400000,0x000000078ad80000)
from space 195072K, 97% used [0x0000000796d00000,0x00000007a26398f0,0x00000007a2b80000)
to space 196096K, 0% used [0x000000078ad80000,0x000000078ad80000,0x0000000796d00000)
ParOldGen total 5232640K, used 5167250K [0x00000003d3c00000, 0x0000000513200000, 0x0000000671400000)
object space 5232640K, 98% used [0x00000003d3c00000,0x000000050f2248d0,0x0000000513200000) #第5466次调用fullGC后，内存使用98%
Metaspace used 101391K, capacity 106711K, committed 107712K, reserved 1144832K
class space used 12164K, capacity 13052K, committed 13312K, reserved 1048576K
}
Event: 261046.853 GC heap before
{Heap before GC invocations=5467 (full 6):
PSYoungGen total 4808704K, used 4803302K [0x0000000671400000, 0x00000007a2d80000, 0x00000007c0000000)
eden space 4613632K, 100% used [0x0000000671400000,0x000000078ad80000,0x000000078ad80000)
from space 195072K, 97% used [0x0000000796d00000,0x00000007a26398f0,0x00000007a2b80000)
to space 196096K, 0% used [0x000000078ad80000,0x000000078ad80000,0x0000000796d00000)
ParOldGen total 5232640K, used 5167250K [0x00000003d3c00000, 0x0000000513200000, 0x0000000671400000)
object space 5232640K, 98% used [0x00000003d3c00000,0x000000050f2248d0,0x0000000513200000) #第5467次调用fullGC前，内存使用98%
Metaspace used 101391K, capacity 106711K, committed 107712K, reserved 1144832K
class space used 12164K, capacity 13052K, committed 13312K, reserved 1048576K
Event: 261046.916 GC heap after
Heap after GC invocations=5467 (full 6):
PSYoungGen total 4809728K, used 189466K [0x0000000671400000, 0x00000007a5600000, 0x00000007c0000000)
eden space 4613632K, 0% used [0x0000000671400000,0x0000000671400000,0x000000078ad80000)
from space 196096K, 96% used [0x000000078ad80000,0x00000007966868c0,0x0000000796d00000)
to space 197120K, 0% used [0x0000000799580000,0x0000000799580000,0x00000007a5600000)
ParOldGen total 5232640K, used 5181087K [0x00000003d3c00000, 0x0000000513200000, 0x0000000671400000)
object space 5232640K, 99% used [0x00000003d3c00000,0x000000050ffa7d38,0x0000000513200000) #第5467次调用fullGC后，内存使用99%
Metaspace used 101391K, capacity 106711K, committed 107712K, reserved 1144832K
class space used 12164K, capacity 13052K, committed 13312K, reserved 1048576K
}
Event: 261107.680 GC heap before
{Heap before GC invocations=5468 (full 6):
PSYoungGen total 4809728K, used 4803098K [0x0000000671400000, 0x00000007a5600000, 0x00000007c0000000)
eden space 4613632K, 100% used [0x0000000671400000,0x000000078ad80000,0x000000078ad80000)
from space 196096K, 96% used [0x000000078ad80000,0x00000007966868c0,0x0000000796d00000)
to space 197120K, 0% used [0x0000000799580000,0x0000000799580000,0x00000007a5600000)
ParOldGen total 5232640K, used 5181087K [0x00000003d3c00000, 0x0000000513200000, 0x0000000671400000)
object space 5232640K, 99% used [0x00000003d3c00000,0x000000050ffa7d38,0x0000000513200000)#第5468次调用fullGC前，内存使用99%
Metaspace used 101391K, capacity 106711K, committed 107712K, reserved 1144832K
class space used 12164K, capacity 13052K, committed 13312K, reserved 1048576K
Event: 261107.740 GC heap after
Heap after GC invocations=5468 (full 6):
PSYoungGen total 4851200K, used 187546K [0x0000000671400000, 0x00000007a5880000, 0x00000007c0000000)
eden space 4654080K, 0% used [0x0000000671400000,0x0000000671400000,0x000000078d500000)
from space 197120K, 95% used [0x0000000799580000,0x00000007a4ca6af0,0x00000007a5600000)
to space 197120K, 0% used [0x000000078d500000,0x000000078d500000,0x0000000799580000)
ParOldGen total 5232640K, used 5196567K [0x00000003d3c00000, 0x0000000513200000, 0x0000000671400000)
object space 5232640K, 99% used [0x00000003d3c00000,0x0000000510ec5cd0,0x0000000513200000) #第5468次调用fullGC后，内存使用99%
Metaspace used 101391K, capacity 106711K, committed 107712K, reserved 1144832K
class space used 12164K, capacity 13052K, committed 13312K, reserved 1048576K
}
Event: 261187.245 GC heap before
{Heap before GC invocations=5469 (full 6):
PSYoungGen total 4851200K, used 4841626K [0x0000000671400000, 0x00000007a5880000, 0x00000007c0000000)
eden space 4654080K, 100% used [0x0000000671400000,0x000000078d500000,0x000000078d500000)
from space 197120K, 95% used [0x0000000799580000,0x00000007a4ca6af0,0x00000007a5600000)
to space 197120K, 0% used [0x000000078d500000,0x000000078d500000,0x0000000799580000)
ParOldGen total 5232640K, used 5196567K [0x00000003d3c00000, 0x0000000513200000, 0x0000000671400000)
object space 5232640K, 99% used [0x00000003d3c00000,0x0000000510ec5cd0,0x0000000513200000) #第5469次调用fullGC前，内存使用99%
Metaspace used 101391K, capacity 106711K, committed 107712K, reserved 1144832K
class space used 12164K, capacity 13052K, committed 13312K, reserved 1048576K
Event: 261187.288 GC heap after
Heap after GC invocations=5469 (full 6):
PSYoungGen total 4851200K, used 191201K [0x0000000671400000, 0x00000007a7f80000, 0x00000007c0000000)
eden space 4654080K, 0% used [0x0000000671400000,0x0000000671400000,0x000000078d500000)
from space 197120K, 96% used [0x000000078d500000,0x0000000798fb87b0,0x0000000799580000)
to space 198656K, 0% used [0x000000079bd80000,0x000000079bd80000,0x00000007a7f80000)
ParOldGen total 5232640K, used 5210187K [0x00000003d3c00000, 0x0000000513200000, 0x0000000671400000)
object space 5232640K, 99% used [0x00000003d3c00000,0x0000000511c12da0,0x0000000513200000)#第5469次调用fullGC后，内存使用99%
Metaspace used 101391K, capacity 106711K, committed 107712K, reserved 1144832K
class space used 12164K, capacity 13052K, committed 13312K, reserved 1048576K
}
Event: 261248.905 GC heap before
{Heap before GC invocations=5470 (full 6):
PSYoungGen total 4851200K, used 4845281K [0x0000000671400000, 0x00000007a7f80000, 0x00000007c0000000)
eden space 4654080K, 100% used [0x0000000671400000,0x000000078d500000,0x000000078d500000)
from space 197120K, 96% used [0x000000078d500000,0x0000000798fb87b0,0x0000000799580000)
to space 198656K, 0% used [0x000000079bd80000,0x000000079bd80000,0x00000007a7f80000)
ParOldGen total 5232640K, used 5210187K [0x00000003d3c00000, 0x0000000513200000, 0x0000000671400000)
object space 5232640K, 99% used [0x00000003d3c00000,0x0000000511c12da0,0x0000000513200000) #第5470次调用fullGC前，内存使用99%
Metaspace used 101391K, capacity 106711K, committed 107712K, reserved 1144832K
class space used 12164K, capacity 13052K, committed 13312K, reserved 1048576K
Event: 261248.948 GC heap after
Heap after GC invocations=5470 (full 6):
PSYoungGen total 4890112K, used 194517K [0x0000000671400000, 0x00000007a8300000, 0x00000007c0000000)
eden space 4691456K, 0% used [0x0000000671400000,0x0000000671400000,0x000000078f980000)
from space 198656K, 97% used [0x000000079bd80000,0x00000007a7b75650,0x00000007a7f80000)
to space 200704K, 0% used [0x000000078f980000,0x000000078f980000,0x000000079bd80000)
ParOldGen total 5232640K, used 5218201K [0x00000003d3c00000, 0x0000000513200000, 0x0000000671400000)
object space 5232640K, 99% used [0x00000003d3c00000,0x00000005123e65a0,0x0000000513200000) #第5470次调用fullGC后，内存使用99%
Metaspace used 101391K, capacity 106711K, committed 107712K, reserved 1144832K
class space used 12164K, capacity 13052K, committed 13312K, reserved 1048576K
}
Event: 261248.948 GC heap before
{Heap before GC invocations=5471 (full 7):
PSYoungGen total 4890112K, used 194517K [0x0000000671400000, 0x00000007a8300000, 0x00000007c0000000)
eden space 4691456K, 0% used [0x0000000671400000,0x0000000671400000,0x000000078f980000)
from space 198656K, 97% used [0x000000079bd80000,0x00000007a7b75650,0x00000007a7f80000)
to space 200704K, 0% used [0x000000078f980000,0x000000078f980000,0x000000079bd80000)
ParOldGen total 5232640K, used 5218201K [0x00000003d3c00000, 0x0000000513200000, 0x0000000671400000)
object space 5232640K, 99% used [0x00000003d3c00000,0x00000005123e65a0,0x0000000513200000) #第5471次调用fullGC前，内存使用99%
Metaspace used 101391K, capacity 106711K, committed 107712K, reserved 1144832K
class space used 12164K, capacity 13052K, committed 13312K, reserved 1048576K
#10.程序奔溃时非优化事件
Deoptimization events (10 events):
Event: 183256.499 Thread 0x00007ff4e5b17000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007ff4d8ead114 method=ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy.isTriggeringEvent(Ljava/io/File;Ljava/lang/Object;)Z @ 31
Event: 188056.499 Thread 0x00007ff3fc016800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007ff4d8d1db04 method=java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object; @ 147
Event: 189591.336 Thread 0x00007ff38c391000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007ff4d7f684cc method=io.netty.channel.epoll.EpollEventLoop.epollWait(Z)I @ 29
Event: 212562.434 Thread 0x00007ff38c3a9800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00007ff4d887d334 method=io.lettuce.core.protocol.RedisStateMachine.readLine(Lio/netty/buffer/ByteBuf;)Ljava/nio/ByteBuffer; @ 23
Event: 213358.334 Thread 0x00007ff38c39d800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00007ff4d887d334 method=io.lettuce.core.protocol.RedisStateMachine.readLine(Lio/netty/buffer/ByteBuf;)Ljava/nio/ByteBuffer; @ 23
Event: 213475.092 Thread 0x00007ff38c39d800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00007ff4d887d334 method=io.lettuce.core.protocol.RedisStateMachine.readLine(Lio/netty/buffer/ByteBuf;)Ljava/nio/ByteBuffer; @ 23
Event: 214383.434 Thread 0x00007ff38c392800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00007ff4d887d334 method=io.lettuce.core.protocol.RedisStateMachine.readLine(Lio/netty/buffer/ByteBuf;)Ljava/nio/ByteBuffer; @ 23
Event: 215716.499 Thread 0x00007ff4e5c05000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007ff4d76096e0 method=io.netty.channel.nio.NioEventLoop.select(Z)V @ 122
Event: 238192.333 Thread 0x00007ff38c396000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007ff4d7f684cc method=io.netty.channel.epoll.EpollEventLoop.epollWait(Z)I @ 29
Event: 238605.046 Thread 0x00007ff37c1a4800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007ff4d5882778 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 6
Classes redefined (0 events):
No events
#11.程序奔溃时内部异常。可以看到主要是socket超时异常
Internal exceptions (10 events):
Event: 260986.649 Thread 0x00007ff288025000 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x000000067aab6e98) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 736]
Event: 261011.428 Thread 0x00007ff28800d800 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x0000000702733ab0) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 736]
Event: 261016.425 Thread 0x00007ff288018000 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x0000000703e66e30) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 736]
Event: 261096.428 Thread 0x00007ff288018000 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x000000073f56ef88) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 736]
Event: 261101.368 Thread 0x00007ff28801e800 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x000000073fe3e198) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 736]
Event: 261126.806 Thread 0x00007ff288009800 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x00000006a9b43330) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 736]
Event: 261131.428 Thread 0x00007ff288014800 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x00000006ce14bd38) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 736]
Event: 261186.685 Thread 0x00007ff288025000 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x000000077f1c1780) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 736]
Event: 261191.457 Thread 0x00007ff288007000 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x00000006a3c88db8) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 736]
Event: 261191.720 Thread 0x00007ff288007000 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x00000006a3cc1310) thrown at [/scratch/jenkins/workspace/8-2-build-linux-amd64/jdk8u251/737/hotspot/src/share/vm/prims/jni.cpp, line 736]
#12.程序奔溃时，最近10个事件。最后一个是JVM内存分配失败
Events (10 events):
Event: 261248.889 Thread 0x00007ff3237a2800 Thread exited: 0x00007ff3237a2800
Event: 261248.889 Thread 0x00007ff306e4d000 Thread exited: 0x00007ff306e4d000
Event: 261248.891 Thread 0x00007ff332d49800 Thread exited: 0x00007ff332d49800
Event: 261248.894 Thread 0x00007fefa1290000 Thread exited: 0x00007fefa1290000
Event: 261248.904 Thread 0x00007fefa1290000 Thread added: 0x00007fefa1290000
Event: 261248.904 Thread 0x00007ff30204d000 Thread added: 0x00007ff30204d000
Event: 261248.904 Thread 0x00007ff2ffb8b000 Thread added: 0x00007ff2ffb8b000
Event: 261248.904 Thread 0x00007ff306e4d000 Thread added: 0x00007ff306e4d000
Event: 261248.904 Thread 0x00007ff1ba676800 Thread added: 0x00007ff1ba676800
Event: 261248.904 Executing VM operation: ParallelGCFailedAllocation
#13.动态库信息。这些信息是虚拟机奔溃时的虚拟内存列表区域。它可以告诉奔溃时哪些类库正在被使用，位置在哪里，还有堆栈和守护等信息
Dynamic libraries:
3d3c00000-513200000 rw-p 00000000 00:00 0
5ca700000-671400000 ---p 00000000 00:00 0
671400000-7a8300000 rw-p 00000000 00:00 0
7a8300000-7c0000000 ---p 00000000 00:00 0
7c0000000-7c0d00000 rw-p 00000000 00:00 0
7c0d00000-800000000 ---p 00000000 00:00 0
562a3e8a8000-562a3e8a9000 r-xp 00000000 fd:01 920956 /usr/java/jdk1.8.0_251/bin/java
562a3eaa8000-562a3eaa9000 r--p 00000000 fd:01 920956 /usr/java/jdk1.8.0_251/bin/java
562a3eaa9000-562a3eaaa000 rw-p 00001000 fd:01 920956 /usr/java/jdk1.8.0_251/bin/java
562a40a50000-562a43108000 rw-p 00000000 00:00 0 [heap]
7fec54000000-7fec5778e000 rw-p 00000000 00:00 0
7ff28229d000-7ff2822a0000 ---p 00000000 00:00 0
7ff2822a0000-7ff2823de000 rw-p 00000000 00:00 0
7ff2823de000-7ff28259e000 ---p 00000000 00:00 0
7ff28259e000-7ff28279e000 rw-p 00000000 00:00 0
7ff28279e000-7ff2827d6000 r-xp 00000000 fd:01 919143 /usr/java/jdk1.8.0_251/jre/lib/amd64/libsunec.so
7ff2827d6000-7ff2829d5000 ---p 00038000 fd:01 919143 /usr/java/jdk1.8.0_251/jre/lib/amd64/libsunec.so
7ff2829d5000-7ff2829da000 r--p 00037000 fd:01 919143 /usr/java/jdk1.8.0_251/jre/lib/amd64/libsunec.so
7ff2829da000-7ff2829dc000 rw-p 0003c000 fd:01 919143 /usr/java/jdk1.8.0_251/jre/lib/amd64/libsunec.so
7ff2829dc000-7ff282bdc000 rw-p 00000000 00:00 0
7ff282bdc000-7ff282bdf000 ---p 00000000 00:00 0
7ff282bdf000-7ff282cdd000 rw-p 00000000 00:00 0
7ff282cdd000-7ff282ce0000 ---p 00000000 00:00 0
7ff282ce0000-7ff282dde000 rw-p 00000000 00:00 0
7ff282dde000-7ff282de1000 ---p 00000000 00:00 0
7ff282de1000-7ff282edf000 rw-p 00000000 00:00 0
7ff285b09000-7ff285b0c000 ---p 00000000 00:00 0
7ff285b0c000-7ff285e0a000 rw-p 00000000 00:00 0
7ff285e0a000-7ff285e0d000 ---p 00000000 00:00 0
7ff285e0d000-7ff285f0b000 rw-p 00000000 00:00 0
7ff285f0b000-7ff285f20000 r-xp 00000000 fd:01 3937 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7ff285f20000-7ff28611f000 ---p 00015000 fd:01 3937 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7ff28611f000-7ff286120000 r--p 00014000 fd:01 3937 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7ff286120000-7ff286121000 rw-p 00015000 fd:01 3937 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7ff286121000-7ff28620a000 r-xp 00000000 fd:01 8779 /usr/lib64/libstdc++.so.6.0.19
7ff28620a000-7ff28640a000 ---p 000e9000 fd:01 8779 /usr/lib64/libstdc++.so.6.0.19
7ff28640a000-7ff286412000 r--p 000e9000 fd:01 8779 /usr/lib64/libstdc++.so.6.0.19
7ff286412000-7ff286414000 rw-p 000f1000 fd:01 8779 /usr/lib64/libstdc++.so.6.0.19
7ff286414000-7ff286429000 rw-p 00000000 00:00 0
7ff286429000-7ff287b3b000 r-xp 00000000 fd:10 1075364526 /data/cplex/libcplex1260remotejni.so
7ff287b3b000-7ff287d3b000 ---p 01712000 fd:10 1075364526 /data/cplex/libcplex1260remotejni.so
7ff287d3b000-7ff287d3c000 r--p 01712000 fd:10 1075364526 /data/cplex/libcplex1260remotejni.so
7ff287d3c000-7ff287ef6000 rw-p 01713000 fd:10 1075364526 /data/cplex/libcplex1260remotejni.so
7ff287ef6000-7ff287eff000 rw-p 00000000 00:00 0
7ff287eff000-7ff287f02000 ---p 00000000 00:00 0
7ff4549be000-7ff455abd000 rw-p 00000000 00:00 0
7ff455abd000-7ff45c000000 r--p 00000000 fd:01 4236 /usr/lib/locale/locale-archive
7ff45c000000-7ff45fffe000 rw-p 00000000 00:00 0
7ff45fffe000-7ff460000000 ---p 00000000 00:00 0
7ff460000000-7ff464000000 rw-p 00000000 00:00 0
7ff471cd5000-7ff471cd8000 ---p 00000000 00:00 0
7ff471cd8000-7ff471fd6000 rw-p 00000000 00:00 0
7ff471fd6000-7ff471fe3000 r-xp 00000000 fd:01 20664 /tmp/libnetty_transport_native_epoll_x86_648817920238216405072.so (deleted)
7ff471fe3000-7ff4721e2000 ---p 0000d000 fd:01 20664 /tmp/libnetty_transport_native_epoll_x86_648817920238216405072.so (deleted)
7ff4721e2000-7ff4721e4000 rw-p 0000c000 fd:01 20664 /tmp/libnetty_transport_native_epoll_x86_648817920238216405072.so (deleted)
7ff4721e4000-7ff4723e4000 rw-p 00000000 00:00 0
7ff4723e4000-7ff4723e7000 ---p 00000000 00:00 0
7ff4837df000-7ff4839df000 rw-p 00000000 00:00 0
7ff4839df000-7ff4839f5000 r-xp 00000000 fd:01 7744 /usr/lib64/libresolv-2.17.so
7ff4839f5000-7ff483bf5000 ---p 00016000 fd:01 7744 /usr/lib64/libresolv-2.17.so
7ff483bf5000-7ff483bf6000 r--p 00016000 fd:01 7744 /usr/lib64/libresolv-2.17.so
7ff483bf6000-7ff483bf7000 rw-p 00017000 fd:01 7744 /usr/lib64/libresolv-2.17.so
7ff483bf7000-7ff483bf9000 rw-p 00000000 00:00 0
7ff483bf9000-7ff483bff000 r-xp 00000000 fd:01 3925 /usr/lib64/libnss_dns-2.17.so
7ff483bff000-7ff483dfe000 ---p 00006000 fd:01 3925 /usr/lib64/libnss_dns-2.17.so
7ff483dfe000-7ff483dff000 r--p 00005000 fd:01 3925 /usr/lib64/libnss_dns-2.17.so
7ff483dff000-7ff483e00000 rw-p 00006000 fd:01 3925 /usr/lib64/libnss_dns-2.17.so
7ff483e00000-7ff484000000 rw-p 00000000 00:00 0
7ff484000000-7ff487bee000 rw-p 00000000 00:00 0
7ff487bee000-7ff488000000 ---p 00000000 00:00 0
7ff4880e7000-7ff4882e7000 rw-p 00000000 00:00 0
7ff4882e7000-7ff4884e7000 rw-p 00000000 00:00 0
7ff4884e7000-7ff4886e7000 rw-p 00000000 00:00 0
7ff4886e7000-7ff4888e7000 rw-p 00000000 00:00 0
7ff4888e7000-7ff4888f8000 r-xp 00000000 fd:01 919104 /usr/java/jdk1.8.0_251/jre/lib/amd64/libnio.so
7ff4888f8000-7ff488af7000 ---p 00011000 fd:01 919104 /usr/java/jdk1.8.0_251/jre/lib/amd64/libnio.so
7ff488af7000-7ff488af8000 r--p 00010000 fd:01 919104 /usr/java/jdk1.8.0_251/jre/lib/amd64/libnio.so
7ff488af8000-7ff488af9000 rw-p 00011000 fd:01 919104 /usr/java/jdk1.8.0_251/jre/lib/amd64/libnio.so
7ff488af9000-7ff488cf9000 rw-p 00000000 00:00 0
7ff488cf9000-7ff488ef9000 rw-p 00000000 00:00 0
7ff488f54000-7ff489154000 rw-p 00000000 00:00 0
7ff489154000-7ff489354000 rw-p 00000000 00:00 0
7ff489354000-7ff489357000 ---p 00000000 00:00 0
7ff489357000-7ff489655000 rw-p 00000000 00:00 0
7ff489655000-7ff489956000 rw-p 00000000 00:00 0
7ff489956000-7ff48996c000 r-xp 00000000 fd:01 919167 /usr/java/jdk1.8.0_251/jre/lib/amd64/libnet.so
7ff48996c000-7ff489b6b000 ---p 00016000 fd:01 919167 /usr/java/jdk1.8.0_251/jre/lib/amd64/libnet.so
7ff489b6b000-7ff489b6c000 r--p 00015000 fd:01 919167 /usr/java/jdk1.8.0_251/jre/lib/amd64/libnet.so
7ff489b6c000-7ff489b6d000 rw-p 00016000 fd:01 919167 /usr/java/jdk1.8.0_251/jre/lib/amd64/libnet.so
7ff489b6d000-7ff489b76000 r-xp 00000000 fd:01 919116 /usr/java/jdk1.8.0_251/jre/lib/amd64/libmanagement.so
7ff489b76000-7ff489d75000 ---p 00009000 fd:01 919116 /usr/java/jdk1.8.0_251/jre/lib/amd64/libmanagement.so
7ff489d75000-7ff489d76000 r--p 00008000 fd:01 919116 /usr/java/jdk1.8.0_251/jre/lib/amd64/libmanagement.so
7ff489d76000-7ff489d77000 rw-p 00009000 fd:01 919116 /usr/java/jdk1.8.0_251/jre/lib/amd64/libmanagement.so
7ff489d77000-7ff489f77000 rw-p 00000000 00:00 0
7ff493a02000-7ff494000000 ---p 00000000 00:00 0
7ff494049000-7ff4940df000 r--s 0110d000 fd:01 918937 /usr/java/jdk1.8.0_251/jre/lib/ext/jfxrt.jar
7ff4940df000-7ff4b4000000 rw-p 00000000 00:00 0
7ff4ced5c000-7ff4d0000000 ---p 00000000 00:00 0
7ff4d0013000-7ff4d0063000 rw-p 00000000 00:00 0
7ff4d0063000-7ff4d023e000 r--s 03dbf000 fd:01 919174 /usr/java/jdk1.8.0_251/jre/lib/rt.jar
7ff4d023e000-7ff4d0378000 rw-p 00000000 00:00 0
7ff4d90d3000-7ff4e3793000 ---p 00000000 00:00 0
7ff4e3793000-7ff4e37ae000 r-xp 00000000 fd:01 919135 /usr/java/jdk1.8.0_251/jre/lib/amd64/libzip.so
7ff4e37ae000-7ff4e39ad000 ---p 0001b000 fd:01 919135 /usr/java/jdk1.8.0_251/jre/lib/amd64/libzip.so
7ff4e39ad000-7ff4e39ae000 r--p 0001a000 fd:01 919135 /usr/java/jdk1.8.0_251/jre/lib/amd64/libzip.so
7ff4e39ae000-7ff4e39af000 rw-p 0001b000 fd:01 919135 /usr/java/jdk1.8.0_251/jre/lib/amd64/libzip.so
7ff4e39af000-7ff4e39bb000 r-xp 00000000 fd:01 5760 /usr/lib64/libnss_files-2.17.so
7ff4e39bb000-7ff4e3bba000 ---p 0000c000 fd:01 5760 /usr/lib64/libnss_files-2.17.so
7ff4e3bba000-7ff4e3bbb000 r--p 0000b000 fd:01 5760 /usr/lib64/libnss_files-2.17.so
7ff4e3bbb000-7ff4e3bbc000 rw-p 0000c000 fd:01 5760 /usr/lib64/libnss_files-2.17.so
7ff4e3bbc000-7ff4e3bc2000 rw-p 00000000 00:00 0
7ff4e3bc2000-7ff4e3bee000 r-xp 00000000 fd:01 919146 /usr/java/jdk1.8.0_251/jre/lib/amd64/libjava.so
7ff4e3bee000-7ff4e3dee000 ---p 0002c000 fd:01 919146 /usr/java/jdk1.8.0_251/jre/lib/amd64/libjava.so
7ff4e3dee000-7ff4e3def000 r--p 0002c000 fd:01 919146 /usr/java/jdk1.8.0_251/jre/lib/amd64/libjava.so
7ff4e3def000-7ff4e3df1000 rw-p 0002d000 fd:01 919146 /usr/java/jdk1.8.0_251/jre/lib/amd64/libjava.so
7ff4e3df1000-7ff4e3dfe000 r-xp 00000000 fd:01 919147 /usr/java/jdk1.8.0_251/jre/lib/amd64/libverify.so
7ff4e3dfe000-7ff4e3ffd000 ---p 0000d000 fd:01 919147 /usr/java/jdk1.8.0_251/jre/lib/amd64/libverify.so
7ff4e3ffd000-7ff4e3fff000 r--p 0000c000 fd:01 919147 /usr/java/jdk1.8.0_251/jre/lib/amd64/libverify.so
7ff4e3fff000-7ff4e4000000 rw-p 0000e000 fd:01 919147 /usr/java/jdk1.8.0_251/jre/lib/amd64/libverify.so
7ff4e4000000-7ff4e8000000 rw-p 00000000 00:00 0
7ff4e8005000-7ff4e800a000 r--s 00040000 fd:01 918942 /usr/java/jdk1.8.0_251/jre/lib/ext/sunjce_provider.jar
7ff4e800a000-7ff4e800e000 r--s 00041000 fd:01 918941 /usr/java/jdk1.8.0_251/jre/lib/ext/sunpkcs11.jar
7ff4e800e000-7ff4e8011000 r--s 0001a000 fd:01 919204 /usr/java/jdk1.8.0_251/jre/lib/jce.jar
7ff4e8011000-7ff4e8024000 r--s 0034b000 fd:01 919049 /usr/java/jdk1.8.0_251/jre/lib/resources.jar
7ff4e8024000-7ff4e8029000 r--s 000a8000 fd:01 918933 /usr/java/jdk1.8.0_251/jre/lib/jsse.jar
7ff4e8029000-7ff4e8045000 r--s 00393000 fd:01 918936 /usr/java/jdk1.8.0_251/jre/lib/ext/cldrdata.jar
7ff4e8045000-7ff4e8046000 ---p 00000000 00:00 0
7ff4e8046000-7ff4e8146000 rw-p 00000000 00:00 0
7ff4e8146000-7ff4e814d000 r-xp 00000000 fd:01 7747 /usr/lib64/librt-2.17.so
7ff4e814d000-7ff4e834c000 ---p 00007000 fd:01 7747 /usr/lib64/librt-2.17.so
7ff4e834c000-7ff4e834d000 r--p 00006000 fd:01 7747 /usr/lib64/librt-2.17.so
7ff4e834d000-7ff4e834e000 rw-p 00007000 fd:01 7747 /usr/lib64/librt-2.17.so
7ff4e834e000-7ff4e844f000 r-xp 00000000 fd:01 3938 /usr/lib64/libm-2.17.so
7ff4e844f000-7ff4e864e000 ---p 00101000 fd:01 3938 /usr/lib64/libm-2.17.so
7ff4e864e000-7ff4e864f000 r--p 00100000 fd:01 3938 /usr/lib64/libm-2.17.so
7ff4e864f000-7ff4e8650000 rw-p 00101000 fd:01 3938 /usr/lib64/libm-2.17.so
7ff4e8650000-7ff4e9340000 r-xp 00000000 fd:01 919113 /usr/java/jdk1.8.0_251/jre/lib/amd64/server/libjvm.so
7ff4e9340000-7ff4e953f000 ---p 00cf0000 fd:01 919113 /usr/java/jdk1.8.0_251/jre/lib/amd64/server/libjvm.so
7ff4e953f000-7ff4e95d5000 r--p 00cef000 fd:01 919113 /usr/java/jdk1.8.0_251/jre/lib/amd64/server/libjvm.so
7ff4e95d5000-7ff4e9606000 rw-p 00d85000 fd:01 919113 /usr/java/jdk1.8.0_251/jre/lib/amd64/server/libjvm.so
7ff4e9606000-7ff4e9641000 rw-p 00000000 00:00 0
7ff4e9641000-7ff4e9805000 r-xp 00000000 fd:01 3908 /usr/lib64/libc-2.17.so
7ff4e9805000-7ff4e9a04000 ---p 001c4000 fd:01 3908 /usr/lib64/libc-2.17.so
7ff4e9a04000-7ff4e9a08000 r--p 001c3000 fd:01 3908 /usr/lib64/libc-2.17.so
7ff4e9a08000-7ff4e9a0a000 rw-p 001c7000 fd:01 3908 /usr/lib64/libc-2.17.so
7ff4e9a0a000-7ff4e9a0f000 rw-p 00000000 00:00 0
7ff4e9a0f000-7ff4e9a11000 r-xp 00000000 fd:01 3928 /usr/lib64/libdl-2.17.so
7ff4e9a11000-7ff4e9c11000 ---p 00002000 fd:01 3928 /usr/lib64/libdl-2.17.so
7ff4e9c11000-7ff4e9c12000 r--p 00002000 fd:01 3928 /usr/lib64/libdl-2.17.so
7ff4e9c12000-7ff4e9c13000 rw-p 00003000 fd:01 3928 /usr/lib64/libdl-2.17.so
7ff4e9c13000-7ff4e9c2a000 r-xp 00000000 fd:01 920384 /usr/java/jdk1.8.0_251/lib/amd64/jli/libjli.so
7ff4e9c2a000-7ff4e9e29000 ---p 00017000 fd:01 920384 /usr/java/jdk1.8.0_251/lib/amd64/jli/libjli.so
7ff4e9e29000-7ff4e9e2a000 r--p 00016000 fd:01 920384 /usr/java/jdk1.8.0_251/lib/amd64/jli/libjli.so
7ff4e9e2a000-7ff4e9e2b000 rw-p 00017000 fd:01 920384 /usr/java/jdk1.8.0_251/lib/amd64/jli/libjli.so
7ff4e9e2b000-7ff4e9e42000 r-xp 00000000 fd:01 3935 /usr/lib64/libpthread-2.17.so
7ff4e9e42000-7ff4ea041000 ---p 00017000 fd:01 3935 /usr/lib64/libpthread-2.17.so
7ff4ea041000-7ff4ea042000 r--p 00016000 fd:01 3935 /usr/lib64/libpthread-2.17.so
7ff4ea042000-7ff4ea043000 rw-p 00017000 fd:01 3935 /usr/lib64/libpthread-2.17.so
7ff4ea043000-7ff4ea047000 rw-p 00000000 00:00 0
7ff4ea047000-7ff4ea069000 r-xp 00000000 fd:01 4297 /usr/lib64/ld-2.17.so
7ff4ea069000-7ff4ea06a000 rw-p 00000000 00:00 0
7ff4ea06a000-7ff4ea074000 r--s 00116000 fd:01 918946 /usr/java/jdk1.8.0_251/jre/lib/ext/localedata.jar
7ff4ea074000-7ff4ea080000 r--s 04612000 fd:10 1132777118 /data/Projects/pick-task-assign-job/app.jar
7ff4ea080000-7ff4ea151000 rw-p 00000000 00:00 0
7ff4ea151000-7ff4ea159000 rw-s 00000000 fd:01 266446 /tmp/hsperfdata_ops/5421
7ff4ea159000-7ff4ea15d000 ---p 00000000 00:00 0
7ff4ea15d000-7ff4ea25e000 rw-p 00000000 00:00 0
7ff4ea25e000-7ff4ea260000 r--s 0000d000 fd:01 918944 /usr/java/jdk1.8.0_251/jre/lib/ext/sunec.jar
7ff4ea260000-7ff4ea265000 r--s 002f9000 fd:01 919048 /usr/java/jdk1.8.0_251/jre/lib/charsets.jar
7ff4ea265000-7ff4ea266000 rw-p 00000000 00:00 0
7ff4ea266000-7ff4ea267000 ---p 00000000 00:00 0
7ff4ea267000-7ff4ea268000 rw-p 00000000 00:00 0
7ff4ea268000-7ff4ea269000 r--p 00021000 fd:01 4297 /usr/lib64/ld-2.17.so
7ff4ea269000-7ff4ea26a000 rw-p 00022000 fd:01 4297 /usr/lib64/ld-2.17.so
7ff4ea26a000-7ff4ea26b000 rw-p 00000000 00:00 0
7ffc4c2fa000-7ffc4c31b000 rw-p 00000000 00:00 0 [stack]
7ffc4c3e0000-7ffc4c3e2000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
#14.程序启动时指定的参数信息
VM Arguments:
jvm_args: -Dspring.profiles.active=tce -Djava.library.path=/data/cplex
java_command: app.jar
java_class_path (initial): app.jar
Launcher Type: SUN_STANDARD
#15.程序运行的环境信息
Environment Variables:
JAVA_HOME=/usr/java/default
JRE_HOME=/usr/java/default/jre
CLASSPATH=.:/usr/java/default/lib:/usr/java/default/jre/lib
PATH=/data/anaconda3/bin:/data/anaconda3/condabin:/usr/java/default/bin:/usr/local/swig-4.0.2/bin:/usr/local/go/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/ucloud_bi/spark/bin:/home/ops/.local/bin:/home/ops/bin
SHELL=/bin/bash
#16.信号处理信息。
Signal Handlers:
SIGSEGV: [libjvm.so+0xada700], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xada700], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0x90e240], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0x90e240], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0x90e240], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0x90e240], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGUSR2: [libjvm.so+0x90e110], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGINT: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGTERM: [libjvm.so+0x90e730], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGQUIT: [libjvm.so+0x90e730], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
#17.下面是机器的相关信息。包括操作系统的版本、内存、cpu等信息
--------------- S Y S T E M ---------------
OS:CentOS Linux release 7.6.1810 (Core)
uname:Linux 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64
libc:glibc 2.17 NPTL 2.17
rlimit: STACK 8192k, CORE 0k, NPROC 1000000, NOFILE 65536, AS infinity
load average:4.02 3.61 3.14
#内存信息
/proc/meminfo:
MemTotal: 65806676 kB
MemFree: 443840 kB
MemAvailable: 389336 kB
Buffers: 3896 kB
Cached: 278636 kB
SwapCached: 0 kB
Active: 64351060 kB
Inactive: 153520 kB
Active(anon): 64228544 kB
Inactive(anon): 3252 kB
Active(file): 122516 kB
Inactive(file): 150268 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 5416 kB
Writeback: 0 kB
AnonPages: 64221128 kB
Mapped: 36820 kB
Shmem: 9964 kB
Slab: 279032 kB
SReclaimable: 204580 kB
SUnreclaim: 74452 kB
KernelStack: 10400 kB
PageTables: 139052 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 32903336 kB
Committed_AS: 51608244 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 118528 kB
VmallocChunk: 34359611132 kB
HardwareCorrupted: 0 kB
AnonHugePages: 10240 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 178024 kB
DirectMap2M: 9259008 kB
DirectMap1G: 59768832 kB
#cpu信息。包括每颗cpu信息，这里是16核。
container (cgroup) information:
container_type: cgroupv1
cpu_cpuset_cpus: 0-15
cpu_memory_nodes: 0
active_processor_count: 16
cpu_quota: -1
cpu_period: 100000
cpu_shares: -1
memory_limit_in_bytes: -1
memory_and_swap_limit_in_bytes: -1
memory_soft_limit_in_bytes: -1
memory_usage_in_bytes: 66050940928
memory_max_usage_in_bytes: 0
CPU:total 16 (initial active 16) (16 cores per cpu, 1 threads per core) family 6 model 85 stepping 5, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, rtm, 3dnowpref, lzcnt, tsc, bmi1, bmi2, adx
#cpu详情
/proc/cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 85
model name : Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz
stepping : 5
microcode : 0x1
cpu MHz : 2095.072
cache size : 28160 KB
physical id : 0
siblings : 16
core id : 0
cpu cores : 16
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 arat avx512_vnni
bogomips : 4190.14
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 85
model name : Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz
stepping : 5
microcode : 0x1
cpu MHz : 2095.072
cache size : 28160 KB
physical id : 0
siblings : 16
core id : 1
cpu cores : 16
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 arat avx512_vnni
bogomips : 4190.14
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
。。。。。。。。。。
processor : 15
vendor_id : GenuineIntel
cpu family : 6
model : 85
model name : Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz
stepping : 5
microcode : 0x1
cpu MHz : 2095.072
cache size : 28160 KB
physical id : 0
siblings : 16
core id : 15
cpu cores : 16
apicid : 15
initial apicid : 15
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 arat avx512_vnni
bogomips : 4190.14
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
#程序奔溃时机器内存信息。空闲物理内存只有443M了。swap没有可用的了。
Memory: 4k page, physical 65806676k(443840k free), swap 0k(0k free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (25.251-b08) for linux-amd64 JRE (1.8.0_251-b08), built on Mar 11 2020 23:20:16 by "java_re" with gcc 7.3.0
#程序奔溃的时间
time: Sun Jul 25 11:12:33 2021
timezone: CST
elapsed time: 261254 seconds (3d 0h 34m 14s)
[ops@dis-algo34 pick-task-assign-job]$
#总结一下：这个问题主要是指线上服务器部署了许多程序，有些邮件统计是每天定时启动，当这些定时脚本启动时，就占用了机器特别多内存。导致内存负载过高，操作系统就会干掉一些进程，所以解决的办法就是把一些程序移到其它机器或者升级内存空间，也可以采用日志头文件中介绍的方法。
上面日志的最有用部分有两处。一是：#9.程序奔溃时最近几次的GC垃圾回收相关记录：fullGC，eden取全部回收，OldGen回收后还有98%-99%是被占用的，而且full GC出现的频率很高，可能有内存泄漏的情况。二是：日志最下面：程序奔溃时机器内存信息，可以看出这时机器扩容已经没有内存了。
重新启动程序，两天后，查看所有的FullGC执行的情况
(base) [ops@dis-algo34 pick-task-assign-job]$ cat wse-gc-2021-07-25_12-21-39.log.0.current | grep -i full
2021-07-25T12:21:40.877+0800: 1.304: [Full GC (Metadata GC Threshold) [PSYoungGen: 6080K->0K(558592K)] [ParOldGen: 104K->5893K(286208K)] 6184K->5893K(844800K), [Metaspace: 21004K->21003K(1069056K)], 0.0237441 secs] [Times: user=0.17 sys=0.01, real=0.02 secs]
2021-07-25T12:21:42.576+0800: 3.002: [Full GC (Metadata GC Threshold) [PSYoungGen: 8676K->0K(971264K)] [ParOldGen: 7173K->15275K(355840K)] 15849K->15275K(1327104K), [Metaspace: 34880K->34880K(1081344K)], 0.0753956 secs] [Times: user=0.64 sys=0.01, real=0.08 secs]
2021-07-25T12:21:46.124+0800: 6.550: [Full GC (Metadata GC Threshold) [PSYoungGen: 10232K->0K(984064K)] [ParOldGen: 24879K->33351K(440832K)] 35111K->33351K(1424896K), [Metaspace: 58110K->58091K(1101824K)], 0.1923452 secs] [Times: user=1.85 sys=0.03, real=0.19 secs]
2021-07-25T12:23:00.712+0800: 81.138: [Full GC (Metadata GC Threshold) [PSYoungGen: 8344K->0K(2166784K)] [ParOldGen: 72684K->69273K(593920K)] 81028K->69273K(2760704K), [Metaspace: 94305K->94126K(1136640K)], 0.2455757 secs] [Times: user=1.93 sys=0.06, real=0.25 secs]
2021-07-25T18:11:21.406+0800: 20981.833: [Full GC (Ergonomics) [PSYoungGen: 2544K->0K(224256K)] [ParOldGen: 593362K->484582K(1138688K)] 595906K->484582K(1362944K), [Metaspace: 97269K->96856K(1138688K)], 0.5887687 secs] [Times: user=5.21 sys=0.13, real=0.59 secs]
2021-07-26T10:39:42.090+0800: 80282.516: [Full GC (Ergonomics) [PSYoungGen: 53947K->6613K(1707008K)] [ParOldGen: 1136516K->1138656K(2050048K)] 1190464K->1145269K(3757056K), [Metaspace: 100956K->100897K(1142784K)], 1.4794690 secs] [Times: user=19.05 sys=0.17, real=1.48 secs]
2021-07-27T08:45:41.681+0800: 159842.107: [Full GC (Ergonomics) [PSYoungGen: 50648K->879K(2814464K)] [ParOldGen: 2047436K->2049748K(3226624K)] 2098085K->2050627K(6041088K), [Metaspace: 108690K->108660K(1148928K)], 1.9388877 secs] [Times: user=21.61 sys=0.29, real=1.94 secs]
(base) [ops@dis-algo34 pick-task-assign-job]$
观察到所有的fullGC后内存的回收效率比较低。比如最后一次，回收前堆内存是2098085K，回收后堆内存是2050627K；这种情况可能有两个原因，一个是内存泄漏，一个是回收时又生产了很多对象。
四、程序内存分析
从上面的GC部分日志分析发现，系统奔溃时的几次Full GC日志中，full GC出现的频率很高，基本60s出现一次，而且GC回收后堆内存任然有98%-99%的占用，可能有内存泄漏的情况。
所以项目重新启动并运行一段时间后，使用jvm内存调优工具Arthas，dump下内存并使用MAT分析（dump内存时会阻塞程序，不能线上dump），结果如下。
dump操作：
(base) [ops@dis-algo34 pick-task-assign-job]$ java -jar arthas-boot.jar
[INFO] arthas-boot version: 3.5.3
[INFO] Process 30482 already using port 3658
[INFO] Process 30482 already using port 8563
[INFO] Found existing java process, please choose one and input the serial number of the process, eg : 1. Then hit ENTER.
* [1]: 30482 app.jar
1
[INFO] arthas home: /home/ops/.arthas/lib/3.5.3/arthas
[INFO] The target process already listen port 3658, skip attach.
[INFO] arthas-client connect 127.0.0.1 3658
,---. ,------. ,--------.,--. ,--. ,---. ,---.
/ O \ | .--. ''--. .--'| '--' | / O \ ' .-'
| .-. || '--'.' | | | .--. || .-. |`. `-.
| | | || |\ \ | | | | | || | | |.-' |
`--' `--'`--' '--' `--' `--' `--'`--' `--'`-----'
wiki https://arthas.aliyun.com/doc
tutorials https://arthas.aliyun.com/doc/arthas-tutorials.html
version 3.5.3
main_class
pid 30482
time 2021-07-26 12:44:02
[arthas@30482]$
[arthas@30482]$
[arthas@30482]$
[arthas@30482]$ heapdump
Dumping heap to /tmp/heapdump2021-07-26-16-454750796091920514123.hprof ... #文件所在目录
Heap dump file created
[arthas@30482]$
[ops@dis-algo34 tmp]$ ll -th /tmp/heapdump2021-07-26-16-454750796091920514123.hprof
-rw------- 1 ops ops 4.8G Jul 26 16:46 /tmp/heapdump2021-07-26-16-454750796091920514123.hprof #查看内存dump文件大小 超过4G，windows的文件系统不支持超过4G的文件，只能在linux下分析了。
[ops@dis-algo34 tmp]$
下载解压MAT，并修改配置
[ops@dis-algo34 tmp]$ wget http://iso.mirrors.ustc.edu.cn/eclipse/mat/1.8/rcp/MemoryAnalyzer-1.8.0.20180604-linux.gtk.x86_64.zip
[ops@dis-algo34 tmp]$ unzip MemoryAnalyzer-1.8.0.20180604-linux.gtk.x86_64.zip
[ops@dis-algo34 mat]$ cd mat
[ops@dis-algo34 mat]$ vim MemoryAnalyzer.ini # 修改MAT的内存大小， 注意这个大小要根据你dump文件大小来的，如果dump文件是5GB那么 这里最好配>5GB 否则会报MAT内存不足的异常
[ops@dis-algo34 mat]$ ./ParseHeapDump.sh ../heapdump2021-07-26-16-454750796091920514123.hprof org.eclipse.mat.api:suspects org.eclipse.mat.api:overview org.eclipse.mat.api:top_components
#如果启动报错，则修改ParseHeapDump.sh
[ops@dis-algo34 mat]$ vim ParseHeapDump.sh
#!/bin/sh
#注释掉下面这句
#"$(dirname -- "$0")"/MemoryAnalyzer -consolelog -application org.eclipse.mat.api.parse "$@"
#添加下面这句
/usr/java/jdk1.8.0_251/bin/java -Xmx5g -Xms5g -jar plugins/org.eclipse.equinox.launcher_1.5.0.v20180512-1130.jar -consoleLog -consolelog -application org.eclipse.mat.api.parse "$@"
#重新运行
[ops@dis-algo34 mat]$ ./ParseHeapDump.sh ../heapdump2021-07-26-16-454750796091920514123.hprof org.eclipse.mat.api:suspects org.eclipse.mat.api:overview org.eclipse.mat.api:top_components
Task: Parsing heapdump2021-07-26-16-454750796091920514123.hprof
[Task: Parsing /data/Projects/pick-task-assign-job/dump-analysis/mat/../heapdump2021-07-26-16-454750796091920514123.hprof
[
Subtask: Scanning /data/Projects/pick-task-assign-job/dump-analysis/mat/../heapdump2021-07-26-16-454750796091920514123.hprof
[
[INFO] Detected compressed references, because with uncompressed 64-bit references the array at 0x671443ae0 would overlap the array at 0x671443ac8
[....................
[INFO] Wrote threads call stacks to /data/Projects/pick-task-assign-job/dump-analysis/mat/../heapdump2021-07-26-16-454750796091920514123.threads
[....................
[INFO] Heap /data/Projects/pick-task-assign-job/dump-analysis/mat/../heapdump2021-07-26-16-454750796091920514123.hprof contains 81,886,109 objects
[....................
Subtask: Extracting objects from /data/Projects/pick-task-assign-job/dump-analysis/mat/../heapdump2021-07-26-16-454750796091920514123.hprof
[................................................................................Task: Removing unreachable objects
[
Subtask: Searching for unreachable objects
[
Subtask: Marking reachable objects
[.
Subtask: Re-indexing objects
[.
[INFO] Removed 51,559,853 unreachable objects using 1,892,483,968 bytes
[..
Subtask: Re-indexing classes
[...
Subtask: Writing /data/Projects/pick-task-assign-job/dump-analysis/mat/../heapdump2021-07-26-16-454750796091920514123.idx.index
[....
Subtask: Writing /data/Projects/pick-task-assign-job/dump-analysis/mat/../heapdump2021-07-26-16-454750796091920514123.o2c.index
[.....
Subtask: Writing /data/Projects/pick-task-assign-job/dump-analysis/mat/../heapdump2021-07-26-16-454750796091920514123.a2s.index
[......
Subtask: Re-indexing outbound index
[.......
Subtask: Writing /data/Projects/pick-task-assign-job/dump-analysis/mat/../heapdump2021-07-26-16-454750796091920514123.inbound.index
[........
Subtask: Writing /data/Projects/pick-task-assign-job/dump-analysis/mat/../heapdump2021-07-26-16-454750796091920514123.outbound.index
[.........
Subtask: Writing /data/Projects/pick-task-assign-job/dump-analysis/mat/../heapdump2021-07-26-16-454750796091920514123.o2hprof.index
[.........Task: Calculating Dominator Tree
[
Subtask: Dominator Tree calculation
[
Subtask: Depth-first search
[..............................
Subtask: Computing dominators
[.........................................................
Subtask: Calculate retained sizes
[......................................................................
Subtask: Create dominators index file
[..........................................................................................]
#最终生成三个报告文件
[ops@dis-algo34 dump-analysis]$ ll -th
total 4.9G
-rw-r--r-- 1 root root 280K Jul 26 17:32 heapdump2021-07-26-16-454750796091920514123_Top_Components.zip #内存中的对象数量排名
-rw-r--r-- 1 root root 46K Jul 26 17:31 heapdump2021-07-26-16-454750796091920514123_System_Overview.zip#综合分析
-rw-r--r-- 1 root root 57K Jul 26 17:31 heapdump2021-07-26-16-454750796091920514123_Leak_Suspects.zip #内存泄漏分析
下载到本地解压查看。
内存中的对象占比
从上面的分析来看，ilog.cplex.CplexI存在内存泄漏的可能，需要review代码。
Cplex是IBM开发用于运筹学最优解求解的求解器，这部分是一个同事写的，写完后集成到项目中的。经过代码分析，发现Cplex实例化后，使用完了没有调用end()方法，怀疑这就是内存泄漏的问题。
五、问题原因总结及解决
经过上面分析，复盘崩溃时的场景：1.项目启动时，未配置JVM内存分派，使用默认配置；默认配置是，初始堆大小是1G（默认物理内存的1/64），最大堆内存是16G（默认是物理内存的1/4）；堆内分年轻代和老年代，比例默认是1:2，所以当程序不断扩容后，年轻代最多占用5.33G，老年代最多占用10.66G。2.由于程序刚启动时使用的内存较小，随着程序的运行时间变成，开始执行minor gc和full gc做垃圾回收，由于存在cplex内存泄漏，每次都会有一部分内存因为泄漏回收不了，而且泄漏的内存越来越多，当已分配的堆内存不够用时，就扩容，直到16G堆内存全部分配完，程序出现奔溃。 3.但是根据日志，奔溃时，程序的内存内没有达到16G，所以JVM继续分配内存，此时正好有其他程序那一段时间内存使用很多(超过40%)，导致没有可用的物理内存分配了，最终程序奔溃。
5.1 外部原因及解决
目前该机器上运行的程序比较多。可能有些程序使用内存过大，或者瞬时的内存使用过大，导致当前这个java程序在分派内存时，内存不够用，导致程序奔溃。
解决办法：
- 程序的所有人在启动程序时，最好对程序使用的资源做限制，并且经常查看项目运行的情况，做到在程序崩溃前预知风险，提前解决问题。
- 最好能有相关报警。
- 合理的将程序部署在四台机器上，不要把自己的所有服务都部署在同一台机器上。
5.2. 内部原因
通过内存分析，发现有内存泄漏的可能；同时项目启动时最好预估内存使用情况并指定内存大小，尽量减少系统的自动扩缩容。
解决办法：
- review代码，查看cplex部分是否有内存泄漏的情况。经过代码review发现，cplex使用完后没有调用end方法。修改代码后重新发布，再dump下内存，发现已经没有cplex相关的对象了。
- 项目启动时指定堆内存的初始值和最大值，并保持相等，这样就不用自动扩容了： -Xms3072m -Xmx3072m；关闭堆内存的自动调整**-XX:-UseAdaptiveSizePolicy ；调整老年代和新生代比例，同时增加survivor区的比例，-XX:NewRatio=1 -XX:SurvivorRatio=6；将MetaspaceSize和MaxMetaspaceSize设置成一样的值，并设置得比初始值要大，64G的机器设置为1G，-XX:MetaspaceSize=1024m -XX:MaxMetaspaceSize=1024m **。
- 修改代码中关于线程池相关的配置，根据目前仓的数量，决定有多少线程。
六、调优过程中常用命令
jmap -heap $pid
jstat -gcmetacapacity -h5 $pid1000