线程堆栈是个很强大的东西,可以看到进程正在运行的所有线程信息
1.1 kill -3 PID 命令
堆栈信息会打到进程的后台启动nohup.out的文件里面
1.2 jboss配置前台请求线程数:
jboss-4.2.3.GA/server/default/deploy/jboss-web.deployer/server.xml
<Connector port="17100" address="0.0.0.0"
maxThreads="250" strategy="ms" maxHttpHeaderSize="8192"
emptySessionPath="true" protocol="
HTTP/1.1"
RIEncoding="utf-8" compression="on" compressionMinSize="2048"
noCompressionUserAgents="gozilla,traviata"
compressableMimeType="text/html,text/xml,text/
javascript,text/css,text/plain,application/xhtml+xml,application/x-javascript,application/javascript,text/xhtml"
minSpareThreads="50" maxSpareThreads="200" enableLookups="false" redirectPort="8443" acceptCount="100"
connectionTimeout="20000" disableUploadTimeout="true" />
1.3 关注前台发起的线程
红色部分的线程是前台发起的线程,这个线程的最大数可以在jboss里面配置,系统压力大的时候这里的jboss前台请求的线程来不及释放来来处理新的请求,关注请求的是哪部分业务代码
"http-0.0.0.0-17100-250" daemon prio=1 tid=0x00002aaab0039270 nid=0x5485 waiting for monitor entry [0x000000006196a000..0x000000006196bca0]
at org.apache.log4j.Category.callAppenders(Category.java:201)
- waiting to lock <0x00002b21f8c70060> (a org.apache.log4j.spi.RootLogger)
at org.apache.log4j.Category.forcedLog(Category.java:388)
at org.apache.log4j.Category.info(Category.java:663)
"http-0.0.0.0-17100-249" daemon prio=1 tid=0x00002aaab0038b80 nid=0x5484 waiting for monitor entry [0x0000000063384000..0x0000000063385c20]
at org.apache.log4j.Category.callAppenders(Category.java:201)
- waiting to lock <0x00002b21f8c70060> (a org.apache.log4j.spi.RootLogger)
at org.apache.log4j.Category.forcedLog(Category.java:388)
at org.apache.log4j.Category.log(Category.java:853)
at org.apache.commons.logging.impl.Log4JLogger.info(Log4JLogger.java:193)
1.4 关注runnable状态的线程
"http-0.0.0.0-17100-185" daemon prio=1 tid=0x00002aaaada1fce0 nid=0x61a5 runnable [0x000000005b302000..0x000000005b305ea0]
~~
at com.huawei.nsm.formconfig.cache.Impl.FormConfigObjectCacheImpl.getFormsFromDB(FormConfigObjectCacheImpl.java:127)
at com.huawei.nsm.formconfig.cache.Impl.FormConfigObjectCacheImpl.flushCache(FormConfigObjectCacheImpl.java:76)
at com.huawei.nsm.formconfig.common.Form.deepClone(Form.java:212)
at com.huawei.nsm.formconfig.service.impl.Form
ServiceImpl.copyForm(FormServiceImpl.java:367)
at com.huawei.nsm.formconfig.action.FormAction.editForm(FormAction.java:155)
系统压力大,很大程度取决于这部分的线程在干什么,是否消耗了大量的系统资源,而且进一步关注是哪个线程是跑的业务代码的线程。
2014-02-19 XX 局点XX系统缓慢就是因为有13个编辑表单的线程在并发跑,而且都是runnable状态的
1.5 关注waiting for monitor entry
"Scheduler_Worker-27" prio=1 tid=0x00002aaac6721710 nid=0x7cad waiting for monitor entry [0x000000004e134000..0x000000004e134d20]
at java.lang.Long.valueOf(Long.java:550)
at com.huawei.nsm.formconfig.common.FormFieldStoreValue.<init>(FormFieldStoreValue.java:18)
at sun.reflect.GeneratedConstructorAccessor111.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:501)
at org.
hibernate.tuple.PojoInstantiator.instantiate(PojoInstantiator.java:88)
at org.hibernate.tuple.PojoInstantiator.instantiate(PojoInstantiator.java:100)
~~
at com.huawei.nsm.workflow.service.AutoHDSapJob.executeJob(AutoHDSapJob.java:35)
at com.huawei.nsm.framework.schedule.Job.execute(Job.java:34)
at org.quartz.core.JobRunShell.run(JobRunShell.java:199)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546)
2014-03-29 XX局点XX系统无法访问可以
在线程中出现14个此类线程
分析业务,此定时器应该没5分钟运行一次,
发现驻留如此多的线程在堆栈中是不正常的
1.6 关注读取数据库线程
"http-0.0.0.0-17100-185" daemon prio=1 tid=0x00002aaaada1fce0 nid=0x61a5 runnable [0x000000005b302000..0x000000005b305ea0]
at oracle.jdbc.driver.PhysicalConnection.getCharBuffer(PhysicalConnection.java:7018)
at oracle.jdbc.driver.OracleStatement.prepareAccessors(OracleStatement.java:907)
at oracle.jdbc.driver.T4CTTIdcb.receiveCommon(T4CTTIdcb.java:261)
~~
- locked <0x00002ab8eda121f0> (a oracle.jdbc.driver.T4CConnection)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1203)
at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:342)
at org.hibernate.jdbc.AbstractBatcher.getResultSet(AbstractBatcher.java:186)
at org.hibernate.loader.Loader.getResultSet(Loader.java:1787)
at org.hibernate.loader.Loader.doQuery(Loader.java:674)
如果出现没有分页的查询需要从数据库load大量的数据,这类线程多卡在读取数据的过程中。