请问mapreduce.reduce.shuffle.memory.limit.percent这个参数时在哪个文件里修改的
如题,运行hadoop执行Wordcount程序时,经常出现map 进程一直增长,reduce归零的情况
13/12/02 20:02:06 INFO mapreduce.Job: map 100% reduce 2%
13/12/02 20:02:18 INFO mapreduce.Job: Task Id : attempt_1385983958793_0001_r_000000_1, Status : FAILED
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#1
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:379)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:58)
at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:45)
at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.<init>(InMemoryMapOutput.java:63)
at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:297)
at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:287)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:360)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:295)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:154)
和以上错误信息相似。
在网上查找资料说是跟shuffle消耗内存有关
需要将默认的mapreduce.reduce.shuffle.memory.limit.percent参数0.25调低
但我找/etc/hadoop/mapred-site.xml.temllate下没有这个参数配置啊
我应该到哪个文件去找mapreduce的参数?
我用的是hadoop-2.4.1
求大神指点!!!