hadoop2.7.0 统计字符数量，报“The auxService:mapreduce_shuffle does not exist”异常

军大_j 2015-06-02 03:59:10
下面内容是我执行命令输出的内容：
-----------------------------------------------------------------------------
[root@ikaixin01 hadoop-2.7.0]# hadoop jar /cloud/hadoop-2.7.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.0.jar wordcount /words /count
Java HotSpot(TM) Client VM warning: You have loaded library /cloud/hadoop-2.7.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
15/06/02 02:19:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/06/02 02:19:14 INFO client.RMProxy: Connecting to ResourceManager at ikaixin01/192.168.137.88:8032
15/06/02 02:19:16 INFO input.FileInputFormat: Total input paths to process : 1
15/06/02 02:19:16 INFO mapreduce.JobSubmitter: number of splits:1
15/06/02 02:19:16 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1433170139587_0005
15/06/02 02:19:17 INFO impl.YarnClientImpl: Submitted application application_1433170139587_0005
15/06/02 02:19:17 INFO mapreduce.Job: The url to track the job: http://ikaixin01:8088/proxy/application_1433170139587_0005/
15/06/02 02:19:17 INFO mapreduce.Job: Running job: job_1433170139587_0005
15/06/02 02:19:28 INFO mapreduce.Job: Job job_1433170139587_0005 running in uber mode : false
15/06/02 02:19:28 INFO mapreduce.Job: map 0% reduce 0%
15/06/02 02:19:29 INFO mapreduce.Job: Task Id : attempt_1433170139587_0005_m_000000_0, Status : FAILED
Container launch failed for container_1433170139587_0005_01_000002 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:375)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

15/06/02 02:19:31 INFO mapreduce.Job: Task Id : attempt_1433170139587_0005_m_000000_1, Status : FAILED
Container launch failed for container_1433170139587_0005_01_000003 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:375)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

15/06/02 02:19:33 INFO mapreduce.Job: Task Id : attempt_1433170139587_0005_m_000000_2, Status : FAILED
Container launch failed for container_1433170139587_0005_01_000004 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:375)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

15/06/02 02:19:37 INFO mapreduce.Job: map 100% reduce 100%
15/06/02 02:19:38 INFO mapreduce.Job: Job job_1433170139587_0005 failed with state FAILED due to: Task failed task_1433170139587_0005_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

15/06/02 02:19:38 INFO mapreduce.Job: Counters: 4
Job Counters
Other local map tasks=3
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=0
Total time spent by all reduces in occupied slots (ms)=0
[root@ikaixin01 hadoop-2.7.0]#
------------------------------------------------------
网上查，都描述的是yarn-site.xml 缺少了aux-service的配置。
我的配置：
<property>
<name>yarn.nodemanager.aux-service</name>
<value>mapreduce_shuffle</value>
</property>

麻烦大拿们帮忙看看是什么问题导致。
...全文
4313 7 打赏 收藏 转发到动态 举报
写回复
用AI写文章
7 条回复
切换为时间正序
请发表友善的回复…
发表回复
qq_35382930 2016-06-21
  • 打赏
  • 举报
 回复
请问楼主你在修改yarn配置之后再怎么做呢，直接又运行一次hadoop jar /cloud/hadoop-2.7.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.0.jar wordcount /words /count 呢，还是需要再重新格式阿虎集群呢
MamyA 2015-07-14
  • 打赏
  • 举报
 回复
请问大神你是如何配置好的，我用之前旧的版本的搭建方式换到hadoop2.7上，各种不成功呢。请问主要要配置哪些文件？求解。。。。
学战到底 2015-06-25
  • 打赏
  • 举报
 回复
大神，我的配置没问题，也出现这个错误，请教。。。。
q1066186643 2015-06-04
  • 打赏
  • 举报
 回复
谢谢分享..........
qq_28741651 2015-06-04
  • 打赏
  • 举报
 回复
编码要认真啊。
军大_j 2015-06-02
  • 打赏
  • 举报
 回复
<name>yarn.nodemanager.aux-service</name>这个写错了，应该是： <name>yarn.nodemanager.aux-services</name> 编码要认真啊。
Hadoop之hadoop-mapreduce-examples测试执行及报错处理
Hadoop Map/Reduce是一个使用简易的软件框架，基于它写出来的应用程序能够运行在由上千个商用机器组成的大型集群上，并以一种可靠容错的方式并行处理上T级别的数据集。
HADOOP编译WordCount.java文件报错
问题现象使用javac编译hadoop例子wordcount.java出现报错如下。
hadoop jar执行报错java.io.FileNotFoundException: /tmp/hadoop-unjarxxx(No space left on device)终极解决方案
原因 MapReduce提交过程中，会把jar解压到/tmp目录下生成临时文件夹hadoop-unjarxxx，任务执行完成后，会主动删除这个临时目录hadoop-unjarxxx, 但是我们偶尔会遇到一些情况，导致hadoop任务异常结束，没有清理掉这个任务的临时目录，临时文件就会越来越多了。 可能导致/tmp磁盘空间不足或者inode索引不足导致报错，因此使用下面命令查找错误，查看使用率 df -h 查看磁盘 df -i 查看inode 解决 思路就是降低jar包大小并定时清除/tmp/hadoop
spark-sql写入对象存储路径不存在问题（异常路径自动消失）
由于多个 Task 同时写数据到 HDFS，如何保证要么所有 Task 写的所有文件要么同时对外可见，要么同时对外不可见，即保证数据一致性同一 Task 可能因为 Speculation 而存在两个完全相同的 Task 实例写相同的数据到 HDFS中，如何保证只有一个 commit 成功对于大 Job（如具有几万甚至几十万 Task），如何高效管理所有文件V1 committer（即的值为 1），commit 过程如下Task 线程将 TaskAttempt 数据写入。
Ambari2.7.0 + HDP3.1.4.0安装,hdfs数据备份和恢复，hive数据备份和恢复，hbase数据备份和恢复,常见错误总结，Ambari卸载，hadoop-ha,hive和ES整合
目录 1 Ambari + HDP离线安装 1.1 介绍 1.1.1 Ambari介绍 1.1.2 HDP 1.1.3 HDP-UTILS 1.2 登录ambari官网地址 1.3 Ambari和HDP下载 1.4 系统要求 1.4.1 软件需求 1.5 修改打开文件的最大数量 1.6 集群节点规划 1.7 防火墙设置 1.8 关闭selinux 1.9 安装jdk 1.10 设置hostname...
云存储

932

社区成员

652

社区内容

发帖
与我相关
我的任务
社区描述
云计算 云存储相关讨论
社区管理员
  • 云存储
加入社区
  • 近7日
  • 近30日
  • 至今

加载中

查看更多榜单
社区公告
暂无公告

试试用AI创作助手写篇文章吧

+ 用AI写文章