Hadoop下遇到OOM：unable to create new native thread_JAVA_编程开发_程序员俱乐部

在hadoop下执行任务出现了这个异常：

class="MsoNormal">Exception in thread "main" org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.OutOfMemoryError: unable to create new native thread

???????? at java.lang.Thread.start0(Native Method)

???????? at java.lang.Thread.start(Thread.java:640)

???????? at org.apache.hadoop.mapred.JvmManager$JvmManagerForType$JvmRunner.kill(JvmManager.java:553)

???????? at org.apache.hadoop.mapred.JvmManager$JvmManagerForType.killJvmRunner(JvmManager.java:317)

???????? at org.apache.hadoop.mapred.JvmManager$JvmManagerForType.killJvm(JvmManager.java:297)

???????? at org.apache.hadoop.mapred.JvmManager$JvmManagerForType.taskKilled(JvmManager.java:289)

???????? at org.apache.hadoop.mapred.JvmManager.taskKilled(JvmManager.java:158)

???????? at org.apache.hadoop.mapred.TaskRunner.kill(TaskRunner.java:801)

???????? at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.kill(TaskTracker.java:3274)

???????? at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.jobHasFinished(TaskTracker.java:3246)

???????? at org.apache.hadoop.mapred.TaskTracker.purgeTask(TaskTracker.java:2281)

???????? at org.apache.hadoop.mapred.TaskTracker.fatalError(TaskTracker.java:3666)

???????? at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)

???????? at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

???????? at java.lang.reflect.Method.invoke(Method.java:597)

???????? at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)

???????? at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)

???????? at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)

???????? at java.security.AccessController.doPrivileged(Native Method)

???????? at javax.security.auth.Subject.doAs(Subject.java:396)

???????? at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)

???????? at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)?

???????? at org.apache.hadoop.ipc.Client.call(Client.java:1107)

???????? at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)

???????? at com.sun.proxy.$Proxy1.fatalError(Unknown Source)

???????? at org.apache.hadoop.mapred.Child.main(Child.java:310)

?

然后，为了了解这个异常，我开始了百度之旅：

也就是说在压力环境下拥有大量的线程，或者本地内存耗尽时，企图创建新的线程时抛出。

而系统能创建的线程数的计算公式如下：?
(MaxProcessMemory?-?JVMMemory?-?ReservedOsMemory)?/?(ThreadStackSize)?=?Number?of?threads?
MaxProcessMemory?? 指的是一个进程的最大内存
JVMMemory?????????????? JVM内存
ReservedOsMemory??保留的操作系统内存
ThreadStackSize??????? 线程栈的大小

表象便是当JVM的heap size设置过大时，thread的创建数量便会减少。

因为当JVM申请分配heap size的内存的时候，会马上占有这些内存，因为这些内存是连续的。

所以会导致没有办法创建本地的线程。

?

JVM的堆和进程所需的栈都会消耗进程地址空间(address space)，而地址空间则取决于操作系统。

而进程栈和一些库也会需要这个寻址空间。

?

有2个方法可以提高创建线程的数量：
1.减少JVM的heap size；
2.减少单个线程栈的大小，在JVM启动中使用-Xss参数。目前JDK1.4中每个线程栈所大小是256K,1.5是1M.

在Linux下

查了一下man ulimit，有下面解释
-a???? All current limits are reported
-c???? The maximum size of core files created
-d???? The maximum size of a process data segment
-f???? The maximum size of files created by the shell
-l???? The maximum size that may be locked into memory
-m???? The maximum resident set size (has no effect on Linux)
-n???? The maximum number of open file descriptors (most systems do not allow this value to be set)
-p???? The pipe size in 512-byte blocks (this may not be set)
-s???? The maximum stack size
-t???? The maximum amount of cpu time in seconds
-u???? The maximum number of processes available to a single user
-v???? The maximum amount of virtual memory available to the shell

其中：只有“max user processes”会对所有application能创建总的线程数有限制

?

refer to：

http://www.iteye.com/topic/654172

http://www.iteye.com/topic/503349

http://bbs.csdn.net/topics/350248404

?

?

?