目前可以说HADOOP和TOMCAT之间已经通了，HDFS操作比如文件上传，删除都没问题。但是MR就出问题了。目前程序能跑，但是跑的时候报错：
2014-02-13 17:50:30,616 WARN [http-8080-1] conf.Configuration (Configuration.java:warnOnceIfDeprecated(981)) - mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
2014-02-13 17:50:30,617 WARN [http-8080-1] conf.Configuration (Configuration.java:warnOnceIfDeprecated(981)) - mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
2014-02-13 17:50:30,825 INFO [http-8080-1] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(454)) - Submitting tokens for job: job_1392171916215_0091
2014-02-13 17:50:31,246 INFO [http-8080-1] mapred.YARNRunner (YARNRunner.java:createApplicationSubmissionContext(367)) - Job jar is not present. Not adding any jar to the list of resources.
2014-02-13 17:50:31,290 INFO [http-8080-1] client.YarnClientImpl (YarnClientImpl.java:submitApplication(124)) - Submitted application application_1392171916215_0091 to ResourceManager at HADOOP1/172.16.0.137:9080
2014-02-13 17:50:31,354 INFO [http-8080-1] mapreduce.Job (Job.java:submit(1273)) - The url to track the job: http://HADOOP1:8088/proxy/application_1392171916215_0091/
2014-02-13 17:50:31,355 INFO [http-8080-1] mapreduce.Job (Job.java:monitorAndPrintJob(1318)) - Running job: job_1392171916215_0091
2014-02-13 17:50:37,880 INFO [http-8080-1] mapreduce.Job (Job.java:monitorAndPrintJob(1339)) - Job job_1392171916215_0091 running in uber mode : false
2014-02-13 17:50:37,882 INFO [http-8080-1] mapreduce.Job (Job.java:monitorAndPrintJob(1346)) - map 0% reduce 0%
2014-02-13 17:50:41,953 INFO [http-8080-1] mapreduce.Job (Job.java:printTaskEvents(1425)) - Task Id : attempt_1392171916215_0091_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.springforhadoop.mapReduce.MyMapper not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1774)
at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:715)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:338)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:160)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:155)
Caused by: java.lang.ClassNotFoundException: Class com.springforhadoop.mapReduce.MyMapper not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1680)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1772)
... 8 more
2014-02-13 17:50:46,001 INFO [http-8080-1] mapreduce.Job (Job.java:printTaskEvents(1425)) - Task Id : attempt_1392171916215_0091_m_000000_1, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.springforhadoop.mapReduce.MyMapper not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1774)
at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:715)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:338)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:160)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:155)
Caused by: java.lang.ClassNotFoundException: Class com.springforhadoop.mapReduce.MyMapper not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1680)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1772)
... 8 more
2014-02-13 17:50:50,038 INFO [http-8080-1] mapreduce.Job (Job.java:printTaskEvents(1425)) - Task Id : attempt_1392171916215_0091_m_000000_2, Status : FAILED
Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.springforhadoop.mapReduce.MyMapper not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1774)
at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:196)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:715)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:338)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:160)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:155)
Caused by: java.lang.ClassNotFoundException: Class com.springforhadoop.mapReduce.MyMapper not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1680)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1772)
... 8 more
2014-02-13 17:52:32,879 INFO [http-8080-1] mapreduce.Job (Job.java:monitorAndPrintJob(1346)) - map 100% reduce 0%
2014-02-13 17:52:32,885 INFO [http-8080-1] mapreduce.Job (Job.java:monitorAndPrintJob(1359)) - Job job_1392171916215_0091 failed with state FAILED due to: Task failed task_1392171916215_0091_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
2014-02-13 17:52:33,009 INFO [http-8080-1] mapreduce.Job (Job.java:monitorAndPrintJob(1364)) - Counters: 6
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=3
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=10238
Total time spent by all reduces in occupied slots (ms)=0
导致每次任务都失败，其实就是mapper类找不到，但是我甚至已经把MAP类和REDUCE类写到JOB类中了，而且安装网上说的 job.setJarByClass(HellozyxMapRed.class);也设置了，但是还是跑不通。望高手指点。
下面是MAPREDUCE类的源码：
package com.springforhadoop.mapReduce;
import java.io.IOException;
import java.util.Iterator;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;
import org.springframework.stereotype.Service;
import org.springframework.test.context.ContextConfiguration;
@ContextConfiguration("spring/spring3mvc-servlet.xml")
@Service
public class HellozyxMapRed {
private static Log tLog=LogFactory.getLog(HellozyxMapRed.class);
private static final Configuration conf = new Configuration();
@Autowired
private MyInputFormat MyInputFormat;
public MyInputFormat getMyInputFormat() {
return MyInputFormat;
}
public void setMyInputFormat(MyInputFormat myInputFormat) {
MyInputFormat = myInputFormat;
}
public void HellozyxMapRedRun() throws IOException, InterruptedException, ClassNotFoundException{
// ApplicationContext ctx = new ClassPathXmlApplicationContext("/MR/mapreduce.xml");
Path file=new Path("upload/mobile2.txt");
Path outFile=new Path("upload/countResult");
Job job=new Job(conf,"TxtCounter");
job.setJarByClass(HellozyxMapRed.class);
job.setJarByClass(MyMapper.class);
job.setJarByClass(MyReduce.class);
FileInputFormat.addInputPath(job, file);
FileOutputFormat.setOutputPath(job, outFile);
job.setMapperClass(MyMapper.class);
job.setReducerClass(MyReduce.class);
// job.setMapOutputKeyClass(LongWritable.class);
// job.setMapOutputValueClass(Text.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.waitForCompletion(true);
}
}
class MyMapper extends Mapper<LongWritable, Text, Text, IntWritable>{
private static Log tLog=LogFactory.getLog(MyMapper1.class);
protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{
tLog.info("_+_+_+_+_+_+ now is map");
// String str1="i am your father, i am the greatest man";
String[] strs=value.toString().split(",");
// String[] strs=str1.split(" ");
for(String str:strs){
context.write(new Text(str), new IntWritable(1));
}
}
}
class MyReduce extends Reducer<Text,IntWritable, Text, IntWritable>{
// private static Logger logger=Logger.getLogger(TxtReducer.class);
private static Log tLog=LogFactory.getLog(MyReduce1.class);
protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException{
tLog.info("++++++++++++++++++start reduce job");
int sum=0;
Iterator<IntWritable> it=values.iterator();
while(it.hasNext()){
IntWritable value=it.next();
sum+=value.get();
tLog.info("key is "+key+" sum is"+sum);
}
context.write(key, new IntWritable(sum));
}
}
