hadoop wordcount 问题

[kangle000]

刚刚开始学习hadoop，搭好环境跑了一下wordcount程序。出现一个问题。我的输入数据有5.8G，采用mapred.map.tasks=2,mapred.reduce.tasks=2时，没有问题。当mapred.map.task2=2,maprd.reduce.tasks=1时，在reduce阶段出现问题，reduce一直进行到89%，就进行不下去，然后retry，最后任务失败。
tasknode的错误如下。

java.io.IOException: subprocess still running
R/W/S=340694604/0/0 in:439605=340694604/775 [rec/s] out:0=0/775 [rec/s]
minRecWrittenToEnableSkip_=9223372036854775807 LOGNAME=null
HOST=null
USER=search
HADOOP_USER=null
last Hadoop input: |null|
last tool output: |null|
Date: Thu Feb 02 16:08:02 CST 2012
Broken pipe
at org.apache.hadoop.streaming.PipeReducer.reduce(PipeReducer.java:131)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:518)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:419)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)

stderr logs

Traceback (most recent call last):
  File "/taskTracker/search/jobcache/job_201202021401_0005/attempt_201202021401_0005_r_000000_0/work/./reduce.py", line 16, in ?
    wc[word] = count
MemoryError

map的源码如下：

#! /usr/bin/python
import sys

for line in sys.stdin:
    line =  line.strip()
    words = line.split()
    for word in words:
        print '%s\t%s' % (word,1)

reduce的源码如下

#!/usr/bin/python
import sys

wc = {}

for line in sys.stdin:
    line = line.strip()
    word,count = line.split('\t',1)
    try:
        count = int(count)
    except Error:
        pass
    if wc.has_key(word):
        wc[word += count
    else:
        wc[word] = count
for key in wc.keys():
    print '%s\t%s' % (key, wc[key])