请教hadoop2.2 wordcount运行卡住问题？

sdjnzqr 2014-03-21 05:01:36
各位大神，小弟最近由于工作需要开始研究hadoop，选择了hadoop2.2版本，安装过程一切正常
安装完成之后，运行wordcount卡住了，查看日志也没有见明显的错误，请教大神们帮忙看看

安装环境：
虚拟机：virtualBox
操作系统：centOS6.5 64位单机
网络：NAT模式
用户：hadoop（普通用户）

安装过程过程如下：
1.安装openJDK1.7 。
2.根据源码编译出对应的64位版本。


[INFO] ------------------------------------------------------------------------

[INFO] Reactor Summary:

[INFO]

[INFO] Apache Hadoop Main ................................ SUCCESS [4.082s]

[INFO] Apache Hadoop Project POM ......................... SUCCESS [2.888s]

[INFO] Apache Hadoop Annotations ......................... SUCCESS [5.947s]

[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.908s]

[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [3.449s]

[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [10.062s]

[INFO] Apache Hadoop Auth ................................ SUCCESS [8.267s]

[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [4.943s]

[INFO] Apache Hadoop Common .............................. SUCCESS [2:36.409s]

[INFO] Apache Hadoop NFS ................................. SUCCESS [12.805s]

[INFO] Apache Hadoop Common Project ...................... SUCCESS [0.073s]

[INFO] Apache Hadoop HDFS ................................ SUCCESS [2:18.842s]

[INFO] Apache Hadoop HttpFS .............................. SUCCESS [33.530s]

[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [59.857s]

[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [7.996s]

[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.093s]

[INFO] hadoop-yarn ....................................... SUCCESS [1:36.068s]

[INFO] hadoop-yarn-api ................................... SUCCESS [1:16.596s]

[INFO] hadoop-yarn-common ................................ SUCCESS [46.203s]

[INFO] hadoop-yarn-server ................................ SUCCESS [0.226s]

[INFO] hadoop-yarn-server-common ......................... SUCCESS [16.424s]

[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [32.148s]

[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [7.221s]

[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [24.729s]

[INFO] hadoop-yarn-server-tests .......................... SUCCESS [1.746s]

[INFO] hadoop-yarn-client ................................ SUCCESS [11.026s]

[INFO] hadoop-yarn-applications .......................... SUCCESS [0.145s]

[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [5.940s]

[INFO] hadoop-mapreduce-client ........................... SUCCESS [0.191s]

[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [50.833s]

[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [4.777s]

[INFO] hadoop-yarn-site .................................. SUCCESS [0.370s]

[INFO] hadoop-yarn-project ............................... SUCCESS [23.235s]

[INFO] hadoop-mapreduce-client-common .................... SUCCESS [35.375s]

[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [6.272s]

[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [18.120s]

[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [10.007s]

[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [13.135s]

[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [4.226s]

[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [13.207s]

[INFO] hadoop-mapreduce .................................. SUCCESS [9.270s]

[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [9.905s]

[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [39.657s]

[INFO] Apache Hadoop Archives ............................ SUCCESS [4.801s]

[INFO] Apache Hadoop Rumen ............................... SUCCESS [11.171s]

[INFO] Apache Hadoop Gridmix ............................. SUCCESS [10.053s]

[INFO] Apache Hadoop Data Join ........................... SUCCESS [7.957s]

[INFO] Apache Hadoop Extras .............................. SUCCESS [6.399s]

[INFO] Apache Hadoop Pipes ............................... SUCCESS [17.653s]

[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [3.980s]

[INFO] Apache Hadoop Tools ............................... SUCCESS [0.060s]

[INFO] Apache Hadoop Distribution ........................ SUCCESS [27.891s]

[INFO] Apache Hadoop Client .............................. SUCCESS [10.050s]

[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.414s]

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 18:31.143s

[INFO] Finished at: Fri Mar 21 14:22:53 CST 2014

[INFO] Final Memory: 117M/411M

[INFO] ------------------------------------------------------------------------

2.按照网上的安装步骤一步步安装hadoop，相应的配置文件：
1）core-site.xml


<configuration>

   <property>

      <name>fs.defaultFS</name>

      <value>hdfs://localhost:8020</value>

      <final>true</final>

   </property>

</configuration>

2）hdfs-site.xml
<configuration>

 <property>

   <name>dfs.namenode.name.dir</name>

   <value>file:/home/hadoop/dfs/name</value>

 </property>

 <property>

   <name>dfs.datanode.data.dir</name>

   <value>file:/home/hadoop/dfs/data</value>

 </property>

</configuration>

3）mapred-site.xml
<configuration>

  <property>

    <name>mapreduce.framework.name</name>

    <value>yarn</value>

  </property>

  <property>

    <name>mapreduce.jobtracker.system.dir</name>

    <value>file:/home/hadoop/mapred/system</value>

  </property>

  <property>

    <name>mapreduce.cluster.local.dir</name>

    <value>file:/home/hadoop/mapred/local</value>

  </property>

</configuration>

4）yarn-site.xml
 <property>

   <name>yarn.nodemanager.aux-services</name>

   <value>mapreduce_shuffle</value>

 </property>

 <property>

 <name>yarn.resourcemanager.address</name>

 <value>127.0.0.1:8032</value>

 </property>

 <property>

 <name>yarn.resourcemanager.scheduler.address</name>

 <value>127.0.0.1:8030</value>

 </property>

 <property>

 <name>yarn.resourcemanager.resource-tracker.address</name>

 <value>127.0.0.1:8031</value>

 </property>

...全文
2201 8 打赏 收藏 转发到动态 举报
写回复
用AI写文章
8 条回复
切换为时间正序
请发表友善的回复…
发表回复
ccdbeyond 2015-12-15
  • 打赏
  • 举报
 回复
请问楼主是如何解决的最后？急急急！
Search_Truth 2014-08-07
  • 打赏
  • 举报
 回复
楼主我也遇到了同样的问题。 请问楼主是怎么解决的呢。
HuntRed 2014-07-31
  • 打赏
  • 举报
 回复
检查 hadoop-yarn-nodemanager status 如果hadoop-yarn-nodemanager启动失败，作业卡住等待分配资源
XingYunBenChen 2014-06-25
  • 打赏
  • 举报
 回复
遇到同样的问题， 请问是怎么解决的？
gnr163 2014-04-13
  • 打赏
  • 举报
 回复
请问具体是如何解决的呢。。
sdjnzqr 2014-03-25
  • 打赏
  • 举报
 回复
问题解决： 可能的原因是没有在 /etc/hosts文件中没有配置本地地址和hsotname的对应关系。 造成在hadoop运行过程中根据hostname找IP时找到了一个错误的IP。
sdjnzqr 2014-03-24
  • 打赏
  • 举报
 回复
在配置hdfs-site.xml中增加配置代码： 
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>
不起作用
sdjnzqr 2014-03-21
  • 打赏
  • 举报
 回复
根据wordCount教程：准备数据： 
[root@centos hadoop]# hadoop fs -ls /input
Found 2 items
-rw-r--r--   3 hadoop supergroup       1118 2014-03-21 15:40 /input/test1.txt
-rw-r--r--   3 hadoop supergroup       1118 2014-03-21 15:40 /input/test2.txt
运行wordCount： 
[hadoop@centos hadoop]$ sudo bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /input/ /output/result
14/03/21 15:53:37 INFO client.RMProxy: Connecting to ResourceManager at /127.0.0.1:8032
14/03/21 15:53:39 INFO input.FileInputFormat: Total input paths to process : 2
14/03/21 15:53:40 INFO mapreduce.JobSubmitter: number of splits:2
14/03/21 15:53:40 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
14/03/21 15:53:40 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/03/21 15:53:40 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/03/21 15:53:40 INFO Configuration.deprecation: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
14/03/21 15:53:40 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/03/21 15:53:40 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/03/21 15:53:40 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/03/21 15:53:40 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/03/21 15:53:40 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/03/21 15:53:40 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/03/21 15:53:40 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/03/21 15:53:40 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/03/21 15:53:42 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1395387332626_0001
14/03/21 15:53:43 INFO impl.YarnClientImpl: Submitted application application_1395387332626_0001 to ResourceManager at /127.0.0.1:8032
14/03/21 15:53:43 INFO mapreduce.Job: The url to track the job: http://centos:8088/proxy/application_1395387332626_0001/
14/03/21 15:53:43 INFO mapreduce.Job: Running job: job_1395387332626_0001
PS:网络上有的帖子都执行到这一步就完了 后台yarn-root-resourcemanager-centos.log日志： 
[root@centos logs]# tail -f yarn-root-resourcemanager-centos.log
2014-03-21 15:35:39,283 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8030: starting
2014-03-21 15:35:39,796 INFO org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.yarn.api.ApplicationClientProtocolPB to the server
2014-03-21 15:35:39,797 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8032
2014-03-21 15:35:39,803 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8032: starting
2014-03-21 15:35:39,806 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2014-03-21 15:35:40,019 INFO org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.yarn.server.api.ResourceManagerAdministrationProtocolPB to the server
2014-03-21 15:35:40,020 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8033
2014-03-21 15:35:40,028 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2014-03-21 15:35:40,058 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8033: starting
2014-03-21 15:53:38,856 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new applicationId: 1
2014-03-21 15:53:43,475 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Storing application with id application_1395387332626_0001
2014-03-21 15:53:43,472 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Application with id 1 submitted by user root
2014-03-21 15:53:43,482 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root     IP=127.0.0.1    OPERATION=Submit Application Request    TARGET=ClientRMService RESULT=SUCCESS  APPID=application_1395387332626_0001
2014-03-21 15:53:43,483 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1395387332626_0001 State change from NEW to NEW_SAVING
2014-03-21 15:53:43,507 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Storing info for app: application_1395387332626_0001
2014-03-21 15:53:43,571 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1395387332626_0001 State change from NEW_SAVING to SUBMITTED
2014-03-21 15:53:43,580 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1395387332626_0001_000001
2014-03-21 15:53:43,619 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1395387332626_0001_000001 State change from NEW to SUBMITTED
2014-03-21 15:53:43,674 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application application_1395387332626_0001 from user: root activated in queue: default
2014-03-21 15:53:43,675 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Application added - appId: application_1395387332626_0001 user: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue$User@4ac9aa21, leaf-queue: default #user-pending-applications: 0 #user-active-applications: 1 #queue-pending-applications: 0 #queue-active-applications: 1
2014-03-21 15:53:43,675 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue: Application added - appId: application_1395387332626_0001 user: root leaf-queue of parent: root #applications: 1
2014-03-21 15:53:43,676 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Application Submission: appattempt_1395387332626_0001_000001, user: root queue: default: capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>usedCapacity=0.0, absoluteUsedCapacity=0.0, numApps=1, numContainers=0, currently active: 1
2014-03-21 15:53:43,678 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1395387332626_0001_000001 State change from SUBMITTED to SCHEDULED
2014-03-21 15:53:43,679 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1395387332626_0001 State change from SUBMITTED to ACCEPTED
实在不知道哪里的原因卡住，请教大神帮忙提示一下不胜感激
hadoop wordcount卡住
2、Ubuntu 12.10 +Hadoop 1.2.1版本集群配置：http://blog.csdn.net/xjavasunjava/article/details/12013677。1、hadoop1.2.1伪分布模式安装教程：http://www.cnblogs.com/fabulousyoung/p/4074197.html。安装完hadoop后，运行hadoop自带的程序wordcount时，出现卡住的问题，等再长的时间也无法计算完。在网上查找资料后得到，这一般是因为网络没有设置好引起的。...
Hadoop运行wordcount实例任务卡在job running的多种情况及解决方法
第一种：配置问题 这是别人的图片，据楼主排查解决是因为hosts配置问题？？？我想知道hosts没配置好，Hadoop是怎么撑到MR的… 现象：各种无法运行、启动 解决办法： 1、修改日志级别 export HADOOP_ROOT_LOGGER=DEBUG,console 查看下详细信息，定位到具体问题解决 第二种：服务器问题 **现象：**运行到job时卡住不动 **原因：**服务器配置低下，内存小或磁盘小 **解决办法：**修改yarn.site.xml配置 <!--每个磁盘的磁盘利用率百分比
hadoop运行自带wordcount卡住：INFO mapreduce.Job:job_1585916162475_0005
hadoop运行自带wordcount卡住：INFO mapreduce.Job:job_1585916162475_0005 ##本人hadoop小白，在搭建好hadoop集群后运行hadoop自带wordount程序 在hadoop文件夹中创建wordcount.txt文件，内容不限，一些简单的英文即可 1.启动hadoop 2.进入hadoop目录，运行下面指令 $hadoop fs -mk...
hadoop3.1.3wordcount实例讲解
近日在CSDN与小破站查找有关linux hadoop 的wordcount实例讲解，上面的某些视频或文章有些过时，于是笔者便自己写一篇实例讲解，希望对大家有所帮助。（笔者是初学者，若有编写错误的地方，望指正：）） 一、环境准备： 1、笔者的hadoop是3.1.3版本，java是1.8.0版本。这是我的环境变量设置，在.bashrc里，读者需根据自己的安装位置灵活更改。 这是我的cor...
Hadoop 运行wordcount任务卡在job running的一种解决办法
Hadoop 运行wordcount任务卡在runnning job的一种解决办法 在使用Mac系统构建本地伪分布式hadoop系统，遇到如下问题 网上有较多的解决办法，但是均不能解决我的问题，后来经过google查询之后，对yarn-site.xml 添加以下配置信息 &lt;property&gt; &lt;name&gt;yarn.nodemanager.disk-he...
Hadoop生态社区

20,808

社区成员

4,690

社区内容

发帖
与我相关
我的任务
社区描述
Hadoop生态大数据交流社区，致力于有Hadoop，hive，Spark，Hbase，Flink，ClickHouse，Kafka，数据仓库，大数据集群运维技术分享和交流等。致力于收集优质的博客
社区管理员
  • 分布式计算/Hadoop社区
  • 涤生大数据
加入社区
  • 近7日
  • 近30日
  • 至今

加载中

查看更多榜单
社区公告
暂无公告

试试用AI创作助手写篇文章吧

+ 用AI写文章