mapreduce 错误：failed Map Tasks exceeded allowed limit

h941578187 2014-01-02 08:56:26
程序在处理20M一下的文件时没有问题，超过20以后执行一直失败，报错信息：如提示
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:576)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.mapred.Child.main(Child.java:249)

执行过程打印信息：
packageJobJar: [/tmp/RtmpbrxmDJ/rmr-local-env5b7e66089a6f, /tmp/RtmpbrxmDJ/rmr-global-env5b7e6fc159e8, /tmp/RtmpbrxmDJ/rmr-streaming-map5b7e11d32952, /tmp/RtmpbrxmDJ/rmr-streaming-reduce5b7e40455eb9, /home/hadoop/tmp/hadoop-unjar4800417805397835354/] [] /tmp/streamjob7747076070321950699.jar tmpDir=null
14/01/02 03:36:20 INFO mapred.FileInputFormat: Total input paths to process : 1
14/01/02 03:36:20 INFO streaming.StreamJob: getLocalDirs(): [/home/hadoop/mapreduce/local]
14/01/02 03:36:20 INFO streaming.StreamJob: Running job: job_201401020330_0003
14/01/02 03:36:20 INFO streaming.StreamJob: To kill this job, run:
14/01/02 03:36:20 INFO streaming.StreamJob: /root/hadoop-1.1.2/libexec/../bin/hadoop job -Dmapred.job.tracker=192.168.2.6:9001 -kill job_201401020330_0003
14/01/02 03:36:20 INFO streaming.StreamJob: Tracking URL: http://192.168.2.6:50030/jobdetails.jsp?jobid=job_201401020330_0003
14/01/02 03:36:22 INFO streaming.StreamJob: map 0% reduce 0%
14/01/02 03:36:37 INFO streaming.StreamJob: map 2% reduce 0%
14/01/02 03:36:42 INFO streaming.StreamJob: map 3% reduce 0%
14/01/02 03:36:45 INFO streaming.StreamJob: map 6% reduce 0%
14/01/02 03:36:48 INFO streaming.StreamJob: map 8% reduce 0%
14/01/02 03:36:51 INFO streaming.StreamJob: map 12% reduce 0%
14/01/02 03:36:54 INFO streaming.StreamJob: map 14% reduce 0%
14/01/02 03:36:58 INFO streaming.StreamJob: map 18% reduce 0%
14/01/02 03:37:01 INFO streaming.StreamJob: map 21% reduce 0%
14/01/02 03:37:04 INFO streaming.StreamJob: map 22% reduce 0%
14/01/02 03:37:07 INFO streaming.StreamJob: map 25% reduce 0%
14/01/02 03:37:10 INFO streaming.StreamJob: map 28% reduce 0%
14/01/02 03:37:13 INFO streaming.StreamJob: map 31% reduce 0%
14/01/02 03:37:30 INFO streaming.StreamJob: map 35% reduce 0%
14/01/02 03:37:31 INFO streaming.StreamJob: map 44% reduce 0%
14/01/02 03:37:34 INFO streaming.StreamJob: map 46% reduce 0%
14/01/02 03:37:35 INFO streaming.StreamJob: map 47% reduce 0%
14/01/02 03:37:37 INFO streaming.StreamJob: map 49% reduce 0%
14/01/02 03:37:40 INFO streaming.StreamJob: map 50% reduce 0%
14/01/02 03:37:41 INFO streaming.StreamJob: map 52% reduce 0%
14/01/02 03:37:43 INFO streaming.StreamJob: map 55% reduce 0%
14/01/02 03:37:46 INFO streaming.StreamJob: map 58% reduce 0%
14/01/02 03:37:50 INFO streaming.StreamJob: map 60% reduce 0%
14/01/02 03:37:55 INFO streaming.StreamJob: map 62% reduce 0%
14/01/02 03:37:56 INFO streaming.StreamJob: map 64% reduce 0%
14/01/02 03:37:58 INFO streaming.StreamJob: map 65% reduce 0%
14/01/02 03:38:04 INFO streaming.StreamJob: map 67% reduce 0%
14/01/02 03:38:06 INFO streaming.StreamJob: map 71% reduce 0%
14/01/02 03:38:09 INFO streaming.StreamJob: map 74% reduce 0%
14/01/02 03:38:12 INFO streaming.StreamJob: map 76% reduce 0%
14/01/02 03:38:15 INFO streaming.StreamJob: map 80% reduce 0%
14/01/02 03:38:18 INFO streaming.StreamJob: map 83% reduce 0%
14/01/02 03:38:22 INFO streaming.StreamJob: map 85% reduce 0%
14/01/02 03:38:25 INFO streaming.StreamJob: map 87% reduce 0%
14/01/02 03:38:28 INFO streaming.StreamJob: map 91% reduce 0%
14/01/02 03:38:31 INFO streaming.StreamJob: map 93% reduce 0%
14/01/02 03:38:35 INFO streaming.StreamJob: map 95% reduce 0%
14/01/02 03:38:39 INFO streaming.StreamJob: map 98% reduce 0%
14/01/02 03:38:42 INFO streaming.StreamJob: map 100% reduce 0%
14/01/02 03:39:13 INFO streaming.StreamJob: map 50% reduce 0%
14/01/02 03:39:27 INFO streaming.StreamJob: map 51% reduce 0%
14/01/02 03:39:30 INFO streaming.StreamJob: map 52% reduce 0%
14/01/02 03:39:34 INFO streaming.StreamJob: map 53% reduce 0%
14/01/02 03:39:48 INFO streaming.StreamJob: map 53% reduce 17%
14/01/02 03:39:50 INFO streaming.StreamJob: map 56% reduce 17%
14/01/02 03:39:53 INFO streaming.StreamJob: map 59% reduce 17%
14/01/02 03:39:56 INFO streaming.StreamJob: map 63% reduce 17%
14/01/02 03:39:59 INFO streaming.StreamJob: map 65% reduce 17%
14/01/02 03:40:02 INFO streaming.StreamJob: map 67% reduce 17%
14/01/02 03:40:04 INFO streaming.StreamJob: map 69% reduce 17%
14/01/02 03:40:07 INFO streaming.StreamJob: map 73% reduce 17%
14/01/02 03:40:10 INFO streaming.StreamJob: map 75% reduce 17%
14/01/02 03:40:14 INFO streaming.StreamJob: map 78% reduce 17%
14/01/02 03:40:17 INFO streaming.StreamJob: map 80% reduce 17%
14/01/02 03:40:20 INFO streaming.StreamJob: map 83% reduce 17%
14/01/02 03:40:24 INFO streaming.StreamJob: map 86% reduce 17%
14/01/02 03:40:26 INFO streaming.StreamJob: map 88% reduce 17%
14/01/02 03:40:29 INFO streaming.StreamJob: map 90% reduce 17%
14/01/02 03:40:32 INFO streaming.StreamJob: map 94% reduce 17%
14/01/02 03:40:36 INFO streaming.StreamJob: map 96% reduce 17%
14/01/02 03:40:39 INFO streaming.StreamJob: map 98% reduce 17%
14/01/02 03:40:42 INFO streaming.StreamJob: map 100% reduce 17%
14/01/02 03:40:47 INFO streaming.StreamJob: map 50% reduce 17%
14/01/02 03:41:02 INFO streaming.StreamJob: map 52% reduce 17%
14/01/02 03:41:05 INFO streaming.StreamJob: map 55% reduce 17%
14/01/02 03:41:08 INFO streaming.StreamJob: map 57% reduce 17%
14/01/02 03:41:11 INFO streaming.StreamJob: map 60% reduce 17%
14/01/02 03:41:14 INFO streaming.StreamJob: map 63% reduce 17%
14/01/02 03:41:17 INFO streaming.StreamJob: map 66% reduce 17%
14/01/02 03:41:20 INFO streaming.StreamJob: map 68% reduce 17%
14/01/02 03:41:23 INFO streaming.StreamJob: map 70% reduce 17%
14/01/02 03:41:26 INFO streaming.StreamJob: map 74% reduce 17%
14/01/02 03:41:29 INFO streaming.StreamJob: map 75% reduce 17%
14/01/02 03:41:32 INFO streaming.StreamJob: map 78% reduce 17%
14/01/02 03:41:35 INFO streaming.StreamJob: map 81% reduce 17%
14/01/02 03:41:39 INFO streaming.StreamJob: map 83% reduce 17%
14/01/02 03:41:41 INFO streaming.StreamJob: map 86% reduce 17%
14/01/02 03:41:44 INFO streaming.StreamJob: map 88% reduce 17%
14/01/02 03:41:47 INFO streaming.StreamJob: map 91% reduce 17%
14/01/02 03:41:50 INFO streaming.StreamJob: map 94% reduce 17%
14/01/02 03:41:53 INFO streaming.StreamJob: map 96% reduce 17%
14/01/02 03:41:56 INFO streaming.StreamJob: map 99% reduce 17%
14/01/02 03:42:01 INFO streaming.StreamJob: map 50% reduce 17%
14/01/02 03:42:10 INFO streaming.StreamJob: map 52% reduce 17%
14/01/02 03:42:13 INFO streaming.StreamJob: map 56% reduce 17%
14/01/02 03:42:16 INFO streaming.StreamJob: map 59% reduce 17%
14/01/02 03:42:19 INFO streaming.StreamJob: map 61% reduce 17%
14/01/02 03:42:22 INFO streaming.StreamJob: map 64% reduce 17%
14/01/02 03:42:25 INFO streaming.StreamJob: map 66% reduce 17%
14/01/02 03:42:28 INFO streaming.StreamJob: map 69% reduce 17%
14/01/02 03:42:31 INFO streaming.StreamJob: map 72% reduce 17%
14/01/02 03:42:34 INFO streaming.StreamJob: map 75% reduce 17%
14/01/02 03:42:37 INFO streaming.StreamJob: map 77% reduce 17%
14/01/02 03:42:40 INFO streaming.StreamJob: map 80% reduce 17%
14/01/02 03:42:43 INFO streaming.StreamJob: map 83% reduce 17%
14/01/02 03:42:46 INFO streaming.StreamJob: map 85% reduce 17%
14/01/02 03:42:49 INFO streaming.StreamJob: map 88% reduce 17%
14/01/02 03:42:52 INFO streaming.StreamJob: map 90% reduce 17%
14/01/02 03:42:55 INFO streaming.StreamJob: map 94% reduce 17%
14/01/02 03:42:59 INFO streaming.StreamJob: map 96% reduce 17%
14/01/02 03:43:02 INFO streaming.StreamJob: map 98% reduce 17%
14/01/02 03:43:07 INFO streaming.StreamJob: map 50% reduce 17%
14/01/02 03:43:10 INFO streaming.StreamJob: map 100% reduce 100%
14/01/02 03:43:10 INFO streaming.StreamJob: To kill this job, run:
14/01/02 03:43:10 INFO streaming.StreamJob: /root/hadoop-1.1.2/libexec/../bin/hadoop job -Dmapred.job.tracker=192.168.2.6:9001 -kill job_201401020330_0003
14/01/02 03:43:10 INFO streaming.StreamJob: Tracking URL: http://192.168.2.6:50030/jobdetails.jsp?jobid=job_201401020330_0003
14/01/02 03:43:10 ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201401020330_0003_m_000000
14/01/02 03:43:10 INFO streaming.StreamJob: killJob...
Streaming Command Failed!
Error in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce, :
hadoop streaming failed with error code 1
Calls: QRcount -> mapreduce -> mr
Execution halted
[root@localhost shell]#

恳求诸位不吝赐教啊~~~
...全文
1112 7 打赏 收藏 转发到动态 举报
写回复
用AI写文章
7 条回复
切换为时间正序
请发表友善的回复…
发表回复
lele12138 2015-03-10
  • 打赏
  • 举报
 回复
引用 5 楼 h941578187 的回复:
谢谢大师和少主的关注，问题已经解决，是由于处理的文件编码格式造成的，doc的编码格式文件在处理的有问题，在map的时候做个操作就ok了！
那为什么20M之内木有问题呢
lele12138 2015-03-10
  • 打赏
  • 举报
 回复
引用 3 楼 tntzbzc 的回复:
性能瓶颈很可能是你的配置有问题 stream配置有问题嘛，检查一下 还有就是检查你的 tasktracker log 看看报什么错 从打印日志来看，肯定有task fail了 最后问一下，你的MR怎么写的？ 有没有做过把数据暂放在内存类似的操作
大神 ，我也报了这个错，其中就有把内容缓存在内存的操作。请问是对内存大小有限制吗？ 下面是我的部分代码 perm = new Path(FileOutputFormat.getOutputPath(job), name); // 临时本地路径，存储临时的索引结果 temp = job.getLocalPath("index/_" + Integer.toString(new Random().nextInt())); fs.delete(perm, true); writer = new IndexWriter(FSDirectory.open(new File(fs.startLocalOutput(perm, temp).toString())), new StandardAnalyzer(Version.LUCENE_30), true, IndexWriter.MaxFieldLength.LIMITED);
h941578187 2014-01-15
  • 打赏
  • 举报
 回复
谢谢大师和少主的关注，问题已经解决，是由于处理的文件编码格式造成的，doc的编码格式文件在处理的有问题，在map的时候做个操作就ok了！
少主无翼 2014-01-07
  • 打赏
  • 举报
 回复
检查日志，这些信息看不出来什么，感觉不太想内存问题，20M的输入文件能吃多少内存呢
撸大湿 2014-01-06
  • 打赏
  • 举报
 回复
性能瓶颈很可能是你的配置有问题
stream配置有问题嘛，检查一下

还有就是检查你的 tasktracker log 看看报什么错
从打印日志来看，肯定有task fail了

最后问一下，你的MR怎么写的？
有没有做过把数据暂放在内存类似的操作
h941578187 2014-01-06
  • 打赏
  • 举报
 回复
设置过了，我用的是单机版的伪分布式。这个是不是在性能上有处理上线啊！
撸大湿 2014-01-03
  • 打赏
  • 举报
 回复
敢问HEAP SIZE多大 搞大一点再试试
第四章：Hadoop编程
MapReduce Shuffle过程使用MRUnit进行单元测试过程MapReduce 数据压缩：Snappy,Gzip,LZOMapReduce Partitioner,Combiner实现及应用MapReduce编程：实现数据去重，数据二次排序，倒排索引使用MapReduce实现微博情感分析与单元测试使用MapReduce用户终端情况分析与单元测试
java堆空间此许可证无效,错误：Java堆空间
In Ubuntu, when I am running the hadoop example :$bin/hadoop jar hadoop-examples-1.0.4.jar grep input output 'dfs[a-z.]+'$echo $HADOOP_HEAPSIZE2000In log, I am getting the error as :INFO mapred.JobCli...
hadoop steaming中ERROR：Job not successful. of failed Reduce Tasks exceeded allowed limit.的解决办法
用hadoop streaaming写了个小的mapreduce实验程序，脚本如下： #! /bash/sh hadoop jar /usr/share/hadoop/contrib/streaming/hadoop-streaming-1.2.1.jar \ -input intest \ -output outtest \ -mapper cat \ -reducer cat\
解决RHadoop错误：PipeMapRed.waitOutputThreads(): subprocess failed with code 1
RHadoop实践系列文章，包含了R语言与Hadoop结合进行海量数据分析。Hadoop主要用来存储海量数据，R语言完成MapReduce 算法，用来替代Java的MapReduce实现。有了RHadoop可以让广大的R语言爱好者，有更强大的工具处理大数据1G, 10G, 100G, TB, PB。 由于大数据所带来的单机性能问题，可能会一...
hadoop distcp踩坑记
distcp（分布式拷贝）是用于大规模集群内部和集群之间拷贝的工具。 它使用Map/Reduce实现文件分发，错误处理和恢复，以及报告生成。 它把文件和目录的列表作为map任务的输入，每个任务会完成源列表中部分文件的拷贝。 1、在nn1上执行 hadoop distcp hdfs://source-nn1:9000/user/xxx.txt hdfs://dest-nn1:9000/ ...
Hadoop生态社区

20,808

社区成员

4,690

社区内容

发帖
与我相关
我的任务
社区描述
Hadoop生态大数据交流社区，致力于有Hadoop，hive，Spark，Hbase，Flink，ClickHouse，Kafka，数据仓库，大数据集群运维技术分享和交流等。致力于收集优质的博客
社区管理员
  • 分布式计算/Hadoop社区
  • 涤生大数据
加入社区
  • 近7日
  • 近30日
  • 至今

加载中

查看更多榜单
社区公告
暂无公告

试试用AI创作助手写篇文章吧

+ 用AI写文章