基于mapreduce的knn算法实现中遇到的问题

[longyixian]

请问一下我下面是什么错误，谢谢诸位了
14/09/03 08:08:01 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
14/09/03 08:08:01 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
14/09/03 08:08:01 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
14/09/03 08:08:02 INFO input.FileInputFormat: Total input paths to process : 1
14/09/03 08:08:02 INFO mapred.JobClient: Running job: job_local_0001
14/09/03 08:08:02 INFO input.FileInputFormat: Total input paths to process : 1
14/09/03 08:08:02 INFO mapred.MapTask: io.sort.mb = 100
14/09/03 08:08:03 INFO mapred.MapTask: data buffer = 79691776/99614720
14/09/03 08:08:03 INFO mapred.MapTask: record buffer = 262144/327680
14/09/03 08:08:03 WARN mapred.LocalJobRunner: job_local_0001
java.lang.ClassCastException: class Point2D
at java.lang.Class.asSubclass(Class.java:3018)
at org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:599)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:791)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:524)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
14/09/03 08:08:03 INFO mapred.JobClient: map 0% reduce 0%
14/09/03 08:08:03 INFO mapred.JobClient: Job complete: job_local_0001
14/09/03 08:08:03 INFO mapred.JobClient: Counters: 0

[longyixian]

引用 4 楼 wulinshishen 的回复:

没有Reducer的话，设置job.setNumReduceTasks(0);要不默认使用一个Reducer类的

加上这行代码，确实ok了，但是程序运行有问题。还请麻烦看看哪里写的不对 输出文件如下： Point2D@11bed71 d1 Point2D@11bed71 d1 Point2D@11bed71 d1 Point2D@11bed71 d1 Point2D@11bed71 d1 Point2D@11bed71 d1 Point2D@11bed71 d2 Point2D@11bed71 d2 Point2D@11bed71 d2 Point2D@11bed71 d2 Point2D@11bed71 d2 Point2D@11bed71 d2 Point2D@11bed71 d3 Point2D@11bed71 d3 Point2D@11bed71 d3 Point2D@11bed71 d3 Point2D@11bed71 d3 Point2D@11bed71 d3 Point2D@11bed71 d4 Point2D@11bed71 d4 Point2D@11bed71 d4 Point2D@11bed71 d4 Point2D@11bed71 d4 Point2D@11bed71 d4 Point2D@11bed71 d5 Point2D@11bed71 d5 Point2D@11bed71 d5 Point2D@11bed71 d5 Point2D@11bed71 d5 Point2D@11bed71 d5 Point2D@11bed71 d6 Point2D@11bed71 d6 Point2D@11bed71 d6 Point2D@11bed71 d6 Point2D@11bed71 d6 Point2D@11bed71 d6 我的data.txt里的内容是 d1 6.0 104.0 A d2 2.0 101.0 A d3 1.0 90.0 A d4 101.0 8.0 R d5 99.0 10.0 R d6 98.0 5.0 R 我的test.txt里的内容是 t1 5.0 100.0 t2 3.0 95.0 t3 8.0 110.0 t4 110.0 5.0 t5 99.0 11.0 t6 98.0 6.0

[人生偌只如初见]

我稍候写一下KNN代码，写好发你！

[longyixian]

引用 6 楼 wulinshishen 的回复:

我稍候写一下KNN代码，写好发你！

好的！谢谢你啦！

[longyixian]

引用 6 楼 wulinshishen 的回复:

我稍候写一下KNN代码，写好发你！

不知道您发到哪里？我留个qq邮箱吧1124250368 再次感谢！

[人生偌只如初见]

http://blog.csdn.net/fighting_one_piece/article/details/39030969 希望对你有帮助

[zixin1990]

引用 9 楼 wulinshishen 的回复:

http://blog.csdn.net/fighting_one_piece/article/details/39030969 希望对你有帮助

很详细的博客，赞！

[人生偌只如初见]

java.lang.ClassCastException: class Point2D 这有提示，你在仔细看看代码

[longyixian]

这是Point2D里的代码

import java.io.DataInput;

import java.io.DataOutput;

import java.io.IOException;


import org.apache.hadoop.io.Writable;


public class Point2D implements Writable {

		public double y;
	
	public String x;
	
	public String getX() {return x;}
	
	public double getY() {return y;}

	
	public void readFields(DataInput in) throws IOException {
	
		x = in.readUTF();
		
	y = in.readDouble();
		
}

		
public void write(DataOutput out) throws IOException {
	
		out.writeChars(x);
		
	out.writeDouble(y);
	
	}

	}

[longyixian]

这是mapreduce里的代码,求指教看看哪里写的不对

import java.io.IOException;
import java.net.URI;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;



public 
class knn {
	
     public static class TokenizerMapper extends
			Mapper<LongWritable, Text, Point2D, Text> {

		public Text one = new Text();
		public Point2D Point2DD = new Point2D();

		public void map(LongWritable key, Text value, Context context)
				throws IOException, InterruptedException {

			String uri = "hdfs://localhost:9000/user/input/test.txt";
			Configuration conf = new Configuration();
			FileSystem fs = FileSystem.get(URI.create(uri), conf);
			FSDataInputStream hdfsInStream = fs.open(new Path(uri));
			byte[] ioBuffer = new byte[13];
			int readLen = hdfsInStream.read(ioBuffer);
			double testx, testy, datax, datay, distance;
			while (-1 != readLen) {

				String res = new String(ioBuffer);
				readLen = hdfsInStream.read(ioBuffer);
				String tline = res.toString();
				String[] tlineSplit = tline.split(" ");
				testx = Double.parseDouble(tlineSplit[1]);
				testy = Double.parseDouble(tlineSplit[2]);

				String dline = value.toString();
				String[] dlineSplit = dline.split(" ");

				datax = Double.parseDouble(dlineSplit[1]);
				datay = Double.parseDouble(dlineSplit[2]);
				distance = Math.sqrt((testx - datax) * (testx - datax)
						+ (testy - datay) * (testy - datay));

				Point2DD.x = tlineSplit[0];
				//Point2DD.setX(tlineSplit[0]);
				Point2DD.y = distance;
				one.set(dlineSplit[0]);
				// Point2DD(tlineSplit[0],distance);
				context.write(Point2DD, one);
			}

		}
	}

     public static void main(String[] args) throws Exception {
        
    	 Configuration conf = new Configuration();
    
         if (args.length != 2) {
             System.err.println("Usage: wordcount  ");
             System.exit(2);
         }
      
         Job job = new Job(conf, "word count");
         job.setJarByClass(knn.class);
         job.setMapperClass(TokenizerMapper.class);
         //job.setReducerClass(IntSumReducer.class);
         job.setMapOutputKeyClass(Point2D.class);
         job.setMapOutputValueClass(Text.class);
         job.setOutputKeyClass(Point2D.class);
         job.setOutputValueClass(Text.class);
      
         
         FileInputFormat.addInputPath(job, new Path(args[0]));
         FileOutputFormat.setOutputPath(job, new Path(args[1]));

         System.exit(job.waitForCompletion(true) ? 0 : 1);
         
     }

}

[人生偌只如初见]

没有Reducer的话，设置job.setNumReduceTasks(0);要不默认使用一个Reducer类的

[bigsolomon]

赞一个赞一个

[金色华年373]

非常好灰常好

[longyixian]

引用 9 楼 wulinshishen 的回复:

http://blog.csdn.net/fighting_one_piece/article/details/39030969 希望对你有帮助

谢谢！

[changuncle]

看不太懂啊，不过没关系，帮顶一下。

[GW786228836]

[伊顺鸣]

不错的啊。。。。

[angelyly]

时间有点久远了，不知道博主还会不会看到这个回复，我在做hadoop的文本分类，希望能有一份您的工程，求看到

[angelyly]

楼主好，我在做hadoop平台的文本分类，不知楼主是否还有工程文件，求分享一份，求看到，[e-mail]