hadoop2.7.2 节点nodemanager启动失败

2016-07-08 06:48:32 7k 阅读 AI 写同款 · GEO 优化AI 选题数据监测一键分发 ›

解决Hadoop2.7.4中nodemanager无法启动的问题在虚拟机环境下，3个节点，每台也就分配2G内存，以前安装CDH都能勉强跑起来，现在用hadoop2.7.4居然跑不动了。报错关键字为： doesn't satisfy minimum allocations 2017-09-29 14:34:41,991 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManage 阅读详情

这是我在master上启动 hadoop 返回的状态

[root@banhai2 hadoop]# start-all.sh 

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh

Starting namenodes on [192.168.1.168]

192.168.1.168: starting namenode, logging to /home/hadoop/hadoop-2.7.2/logs/hadoop-root-namenode-banhai2.out

192.168.1.167: starting datanode, logging to /home/hadoop/hadoop-2.7.2/logs/hadoop-root-datanode-banhai3.out

Starting secondary namenodes [192.168.1.168]

192.168.1.168: starting secondarynamenode, logging to /home/hadoop/hadoop-2.7.2/logs/hadoop-root-secondarynamenode-banhai2.out

starting yarn daemons

starting resourcemanager, logging to /home/hadoop/hadoop-2.7.2/logs/yarn-root-resourcemanager-banhai2.out

192.168.1.167: starting nodemanager, logging to /home/hadoop/hadoop-2.7.2/logs/yarn-root-nodemanager-banhai3.out

[root@banhai2 hadoop]# jps

4215 NameNode

4574 ResourceManager

4846 Jps

4408 SecondaryNameNode

而在Slave上也启动DataNode：

[root@banhai3 logs]# jps

26101 DataNode

26349 Jps

但是就是启动不了nodemanager。
也按照网上的修改了但是就是不行。求解
这是 yarn-root-nodemanager-banhai3.out

Jul 08, 2016 6:28:10 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register

INFO: Registering org.apache.hadoop.yarn.server.nodemanager.webapp.NMWebServices as a root resource class

Jul 08, 2016 6:28:10 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register

INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class

Jul 08, 2016 6:28:10 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register

INFO: Registering org.apache.hadoop.yarn.server.nodemanager.webapp.JAXBContextResolver as a provider class

Jul 08, 2016 6:28:10 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate

INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 AM'

Jul 08, 2016 6:28:10 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider

INFO: Binding org.apache.hadoop.yarn.server.nodemanager.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope "Singleton"

Jul 08, 2016 6:28:10 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider

INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton"

Jul 08, 2016 6:28:11 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider

INFO: Binding org.apache.hadoop.yarn.server.nodemanager.webapp.NMWebServices to GuiceManagedComponentProvider with the scope "Singleton"

这是：yarn-root-nodemanager-banhai3.log 内容：

2016-07-07 17:55:42,932 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NodeManager

STARTUP_MSG:   host = banhai3/127.0.0.1

STARTUP_MSG:   args = []

STARTUP_MSG:   version = 2.7.2

TARTUP_MSG:   build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r b165c4fe8a74265c792ce23f546c64604acf0e41; compiled by 'jenkins' on 2016-01-26T00:08Z

STARTUP_MSG:   java = 1.7.0_79

************************************************************/

2016-07-07 17:55:42,937 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: registered UNIX signal handlers for [TERM, HUP, INT]

2016-07-07 17:55:43,340 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

2016-07-07 17:55:43,677 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher

···其中都是INFO···

2016-07-07 17:55:43,806 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.event.LocalizerEventType for class org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerTracker

2016-07-07 17:55:43,822 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class org.apache.hadoop.mapred.ShuffleHandler which has a name of 'httpshuffle'. Because these are not the same tools trying to send ServiceData and read Service Meta Data may have issues unless the refer to the name in the config.



···INFO···

2016-07-07 17:55:45,078 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Unexpected error starting NodeStatusUpdater

org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN signal from Resourcemanager ,Registration of NodeManager failed, Message from ResourceManager: NodeManager from  localhost doesn't satisfy minimum allocations, Sending SHUTDOWN signal to the NodeManager.

        at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:270)

        at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:196)

        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)

        at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)

        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:271)

        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)

        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:486)

        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:533)

...全文

hadoop-2.7.4-nodemanager无法启动问题解决方案近期新配了hadoop-274版本的集群，发现有一个小问题，DataNode无法顺利启动。问题日志如下： resourcemanager_log 2017-09-20 03:54:24,165 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: NodeManager from ub 阅读详情

依然那么楞

Hadoop生态社区 20,842 成员 4,695 内容

+ 关注 + 加入社区

切换为时间正序

英雄史诗 2019-02-14

打赏
举报

<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>8</value>
</property>


<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>8000</value>
</property>

我的2.7.4，设置成这样就可以；
查资料说：如果自己不设置，就会用默认值启动；但默认值又不符合hadoop要求，就报错了。
我的这里设的比实际的大的多（我电脑实际只有4G内存，4核CPU）。应该是上边说的。

依然那么楞 2016-07-12

打赏
举报

引用 5 楼 ling811 的回复:

[quote=引用 1 楼 Evankaka 的回复:] NodeManager from localhost doesn't satisfy minimum allocations, 好像是内存不够了。

我添加的2048了都不够吗？为什么我还是启动不起来[/quote] 嗯我把yarn-env.sh中的JAVA_HEAP_MAX=-Xmx3072m 改成3G了这次nodemanager启动起来了~ 谢谢了~~~

依然那么楞 2016-07-11

打赏
举报

<property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>2048</value> <discription>单个任务可申请最少内存，默认1024MB</discription> </property> 我添加到了2048 怎么还是启动不了，启动nodeManager 加上参数-Xmx2048M 怎么个写法

引用 2 楼 ArayChou 的回复:

https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml 应该调小这个参数,yarn.scheduler.minimum-allocation-mb 默认值为1G 或者启动nodeManager加上参数 -Xmx2048M 试试

ArayChou 2016-07-11

打赏
举报

引用 5 楼 ling811 的回复:

我添加的2048了都不够吗？为什么我还是启动不起来

帅哥,这个配置是要求NodeManager至少有2G的内存,比起以前的1G,要求更高了.... (其实,你可以将这里修改成512试验一下) 要提高NodeManger的内存,配置在这里:~/hadoop-2.7.2/etc/hadoop/yarn-env.sh

依然那么楞 2016-07-11

打赏
举报

引用 1 楼 Evankaka 的回复:

NodeManager from localhost doesn't satisfy minimum allocations,
好像是内存不够了。

我添加的2048了都不够吗？为什么我还是启动不起来

ArayChou 2016-07-11

打赏
举报

引用 3 楼 ling811 的回复:

<property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>2048</value> <discription>单个任务可申请最少内存，默认1024MB</discription> </property> 我添加到了2048 怎么还是启动不了，启动nodeManager 加上参数-Xmx2048M 怎么个写法

yarn.scheduler.minimum-allocation-mb这个配置是对NodeManager的内存的最小要求,上应该将他设置得更小才可以让你的NodeManager工作起来. 不过实际上要根据需求来定多少. 可以调大NodeManager的内存,配置文件是在hadoop-2.7.2/etc/hadoop/yarn-env.sh;

[aray@hadoop1 hadoop]$ grep "HEAP" ~/hadoop-2.7.2/etc/hadoop/yarn-env.sh
JAVA_HEAP_MAX=-Xmx1000m
# For setting YARN specific HEAP sizes please use this
# YARN_HEAPSIZE=1000
if [ "$YARN_HEAPSIZE" != "" ]; then
  JAVA_HEAP_MAX="-Xmx""$YARN_HEAPSIZE""m"

上面贴出的是默认的配置,可以看出,nodemanager的默认heap大小是1000M,小于上面个默认配置的1024M的要求. 我没有相关经验,如果你还有问题,可以搜索搜索

ArayChou 2016-07-10

打赏
举报

https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml 应该调小这个参数,yarn.scheduler.minimum-allocation-mb 默认值为1G 或者启动nodeManager加上参数 -Xmx2048M 试试

Evankaka 2016-07-08

打赏
举报

NodeManager from localhost doesn't satisfy minimum allocations, 好像是内存不够了。

hadoop2.7.2 节点nodemanager启动失败

相关推荐