[求助]mapreduce+hbase在eclipse下报错的问题

dunerunner 2014-03-05 10:51:04
小弟在win7里装了虚拟机，虚拟机里装了两个linux并配置好了hadoop和hbase
单独的hadoop mapreduce和hbase的代码在win7里用eclipse都能跑
但是mapreduce+hbase的读写代码(代码是《HBase官方文档 0.97》的)在eclipse中跑就跑出异常:
Exception in thread "main" java.lang.IllegalArgumentException: Pathname /E:/workspace_hadoop/zTestHBase/hbasejars/protobuf-java-2.5.0.jar from hdfs://172.17.40.250:9000/E:/workspace_hadoop/zTestHBase/hbasejars/protobuf-java-2.5.0.jar is not a valid DFS filename.

	at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:184)

	at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:92)

	at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1106)

	at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)

	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)

	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)

	at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)

	at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224)

	at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:93)

	at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57)

	at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:264)

	at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:300)

	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:387)

	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)

	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)

	at java.security.AccessController.doPrivileged(Native Method)

	at javax.security.auth.Subject.doAs(Subject.java:415)

	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)

	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)

	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1286)

	at hmrDemo.HBaseWithMapWithoutReduce.main(HBaseWithMapWithoutReduce.java:46)


这个异常真是莫名其妙

没办法只能把类打成jar包，传到linux中，在里边执行 hadoop jar XXX.jar 来跑，是没有问题的
但是这样严重影响效率啊！！
求解
代码没问题是来自《HBase 官方文档 0.97》的 而且在配有hadoop和hbase的linux环境下是可以正确执行的！
...全文
987 14 打赏 收藏 转发到动态 举报
AI 作业
写回复
用AI写文章
14 条回复
切换为时间正序
请发表友善的回复…
发表回复
dunerunner 2014-03-27
  • 打赏
  • 举报
 回复
多谢版主以及大家的帮助。
dunerunner 2014-03-27
  • 打赏
  • 举报
 回复
忘了结贴了，这个问题解决了，就是在windows下测试的时候尽量不使用TableMapReduceUtil.initTableMapperJob()方法 具体配置要看TableMapReduceUtil.initTableMapperJob()中的源码进行手动设置……
「已注销」 2014-03-22
  • 打赏
  • 举报
 回复
附上，我的代码 import hbasetest.TestHFile; import java.io.IOException; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.hbase.KeyValue; import org.apache.hadoop.hbase.client.HTable; import org.apache.hadoop.hbase.io.ImmutableBytesWritable; import org.apache.hadoop.hbase.mapreduce.HFileOutputFormat; import org.apache.hadoop.hbase.mapreduce.KeyValueSortReducer; import org.apache.hadoop.hbase.util.Bytes; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Mapper.Context; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; public class HFileTest { public static class TestHFileToHbaseMapper extends Mapper{ protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String[] values=value.toString().split(",",-1); byte[] subject=Bytes.toBytes(values[0]); ImmutableBytesWritable rowKey=new ImmutableBytesWritable(subject); KeyValue k=new KeyValue(subject,"subject?object".getBytes(),null,values[1].getBytes()); if(k!=null){ context.write(rowKey, k); } } } public static void main(String[] args)throws IOException, InterruptedException, ClassNotFoundException { Configuration conf = HBaseConfiguration.create(); Job job = new Job(conf, "HFileTest"); job.setJarByClass(HFileTest.class); job.setOutputKeyClass(ImmutableBytesWritable.class); job.setOutputValueClass(KeyValue.class); job.setMapperClass(TestHFileToHbaseMapper.class); job.setReducerClass(KeyValueSortReducer.class); job.setOutputFormatClass(HFileOutputFormat.class); HFileOutputFormat.configureIncrementalLoad(job, new HTable(conf,"P_SO")); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } }
「已注销」 2014-03-22
  • 打赏
  • 举报
 回复
版主，你好！我最近在学习hadoop和hbase,在实现hbase数据生成hfile文件时，一直出现错误，不知道怎么解决？ 14/03/22 21:28:31 WARN mapred.LocalJobRunner: job_local_0001 java.lang.IllegalArgumentException: Can't read partitions file at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:116) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:677) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:756) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) Caused by: java.io.FileNotFoundException: File _partition.lst does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:397) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:796) at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1475) at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1470) at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.readPartitions(TotalOrderPartitioner.java:301) at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:88) ... 6 more 14/03/22 21:28:31 INFO mapred.JobClient: map 0% reduce 0% 14/03/22 21:28:31 INFO mapred.JobClient: Job complete: job_local_0001 14/03/22 21:28:31 INFO mapred.JobClient: Counters: 0 不知道版主有没遇到这样的问题，求助
撸大湿 2014-03-08
  • 打赏
  • 举报
 回复
8楼的问题 参照置顶帖 如何在Map和Reduce之间传递静态变量：http://bbs.csdn.net/topics/390418508 或者用Distributed file 解决
撸大湿 2014-03-08
  • 打赏
  • 举报
 回复 


import java.io.IOException;



import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.hbase.HBaseConfiguration;

import org.apache.hadoop.hbase.client.Result;

import org.apache.hadoop.hbase.client.Scan;

import org.apache.hadoop.hbase.io.ImmutableBytesWritable;

import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;

import org.apache.hadoop.hbase.mapreduce.TableMapper;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.lib.output.NullOutputFormat;



public class HBaseWithMapWithoutReduce {



	private static class MyMapper extends TableMapper<Text, LongWritable> {



		public void map(ImmutableBytesWritable key, Result value, Context context) throws IOException, InterruptedException {

			System.err.println(Text.decode(key.get()));

			System.err.println(value.toString());

			System.err.println(context.toString());

		}



	}



	public static void main(String[] args) throws Exception {

		if (args == null || args.length != 2) {

			args = new String[] { "content", "Hello" };

		}

		String tableName = "wordcount";



		Configuration conf = HBaseConfiguration.create();

		conf.set("mapred.job.tracker", "JT HOST或IP:9001");

		conf.set("hbase.zookeeper.quorum", "ZK HOST或IP");

		conf.set("hbase.zookeeper.property.clientPort", "2181");

		Job job = new Job(conf, "ExampleRead");

		job.setJarByClass(HBaseWithMapWithoutReduce.class);



		Scan scan = new Scan();

		scan.setCaching(500);

		scan.setCacheBlocks(false);



		TableMapReduceUtil.initTableMapperJob(tableName, scan, MyMapper.class, null, null, job);

		job.setOutputFormatClass(NullOutputFormat.class);





		boolean b = job.waitForCompletion(true);

		if (!b) {

			throw new IOException("error with job!");

		}

	}

}


添加好hadoop plugin
配好locations
run as hadoop

结果

不用去管那个ERROR，是我测试环境有台机器宕机造成的
撸大湿 2014-03-07
  • 打赏
  • 举报
 回复
你的eclipse项目lib中有没有添加protobuf-java-2.5.0.jar和 guava-XX.X.X.jar
dunerunner 2014-03-07
  • 打赏
  • 举报
 回复
在windows里这些属性是怎么传的
代码里打印也看不到相应的配置属性
Iterator<Entry<String, String>> iterator = config.iterator();

while (iterator.hasNext()) {

Entry<String, String> entry = iterator.next();

System.err.println(entry.getKey() + " -> " + entry.getValue());

}

最开始有个用户名问题，非得把windows的adminstrator改成我linux的用户才行……属性里也没找到配置项
dunerunner 2014-03-07
  • 打赏
  • 举报
 回复
大湿建个QQ群吧，方便学习交流
dunerunner 2014-03-07
  • 打赏
  • 举报
 回复
有的，有protobuf-java-2.5.0.jar和guava-12.0.1.jar
撸大湿 2014-03-06
  • 打赏
  • 举报
 回复
/E:/workspace_hadoop ？？？ hdfs://172.17.40.250:9000/E:/workspace_hadoop ？？？？ 什么情况，你的DistributedFile 怎么会去读这种目录？ 把你的hmrDemo.HBaseWithMapWithoutReduce.java代码贴出来
dunerunner 2014-03-06
  • 打赏
  • 举报
 回复
引用 2 楼 tntzbzc 的回复:
/E:/workspace_hadoop ？？？ hdfs://172.17.40.250:9000/E:/workspace_hadoop ？？？？ 什么情况，你的DistributedFile 怎么会去读这种目录？ 把你的hmrDemo.HBaseWithMapWithoutReduce.java代码贴出来

package hmrDemo;

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
import org.apache.hadoop.hbase.mapreduce.TableMapper;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.output.NullOutputFormat;

public class HBaseWithMapWithoutReduce {

	private static class MyMapper extends TableMapper<Text, LongWritable> {

		public void map(ImmutableBytesWritable key, Result value, Context context) throws IOException, InterruptedException {
			System.err.println(Text.decode(key.get()));
			System.err.println(value.toString());
			System.err.println(context.toString());
		}
		
	}

	public static void main(String[] args) throws Exception {
		if (args == null || args.length != 2) {
			args = new String[] {"content", "Hello"};
		}
		String tableName = "wordcount";
		
		Configuration config = HBaseConfiguration.create();
		
		Job job = Job.getInstance(config, "ExampleRead");
		job.setJarByClass(HBaseWithMapWithoutReduce.class);
		
		Scan scan = new Scan();
		scan.setCaching(500);
		scan.setCacheBlocks(false);
		
		TableMapReduceUtil.initTableMapperJob(tableName, scan, MyMapper.class, null, null, job);
		job.setOutputFormatClass(NullOutputFormat.class);
		
//		Iterator<Entry<String, String>> iterator = config.iterator();
//		while (iterator.hasNext()) {
//			Entry<String, String> entry = iterator.next();
//			System.err.println(entry.getKey() + " -> " + entry.getValue());
//		}
		
		boolean b = job.waitForCompletion(true);
		if (!b) {
			throw new IOException("error with job!");
		}
	}
}
就是照着文档中的demo敲的代码
阿狸-天道 2014-03-06
  • 打赏
  • 举报
 回复
我也是这种问题，在windows下不能自行，在linux下就可以 public class Hbase{ enum Counter{ LINESKIP, } private static final String sourceTable = "log_original"; public static class Mapper1 extends TableMapper<Text, IntWritable> { public void map(ImmutableBytesWritable row, Result hbaseResult, Context context) throws IOException, InterruptedException{ for (KeyValue rowQualifierAndValue : hbaseResult.list()) { String fieldName = new String( rowQualifierAndValue.getQualifier()); String fieldValue = new String(rowQualifierAndValue.getValue()); // if (fieldName.equalsIgnoreCase("time") // || fieldName.equalsIgnoreCase("msg")) { // System.out.println(hbaseResult.getRow()+"------------------------"); // insertSolr(new String(hbaseResult.getRow()),fieldName,fieldValue); // } } } } public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException { // UserGroupInformation ugi = UserGroupInformation.createRemoteUser("hadoop"); // try { // ugi.doAs(new PrivilegedExceptionAction<Void>() { // public Void run() throws Exception { Configuration conf = HBaseConfiguration.create(); // Configuration conf =new Configuration(); conf.set("mapred.job.tracker", "m.soc.com:9001"); // conf.set("hadoop.job.user", "hadoop"); // conf.set("hadoop.job.ugi","hadoop,hadoop"); // conf.set("hbase.zookeeper.quorum","s1.soc.com,s2.soc.com,s3.soc.com,s4.soc.com"); // conf.set("hbase.zookeeper.property.clientPort", "2181"); Job job = new Job(conf); job.setJobName("Hbase"); job.setJarByClass(Hbase.class); Scan scan = new Scan(); scan.setCaching(1024); scan.setCacheBlocks(false); TableMapReduceUtil.initTableMapperJob(sourceTable, scan, Mapper1.class, Text.class, IntWritable.class, job); job.setOutputFormatClass(NullOutputFormat.class); job.waitForCompletion(true); // return null; // } // }); // } catch (InterruptedException e) { // e.printStackTrace(); // } } }
dunerunner 2014-03-05
  • 打赏
  • 举报
 回复
大湿来救命！
hadoop-eclipse-plugin-1.0.0（5个版本）
hadoop-eclipse-plugin-1.0.0（5个版本），足够满足你的需求，你可以选择合适你的。
hadoop权威指南第三版 中文 pdf
本书从Hadoop的缘起开始，由浅入深，结合理论和实践，全方位地介绍Hadoop这一高性能处理海量数据集的理想工具。全书共14章，3个附录，涉及的主题包括：Hadoop简介；MapReduce简介；Hadoop分布式文件系统；Hadoop的I/O、MapReduce应用程序开发；MapReduce的工作机制；MapReduce的类型和格式；MapReduce的特性；如何安装Hadoop集群，如何管理Hadoop；Pig简介；Hbase简介；ZooKeeper简介，最后还提供了丰富的案例分析。
Hadoop源码分析 完整版 共55章
caibinbupt的Hadoop源码分析完整版，包括 HDFS 和 MapReduce。 HDFS: 41章 MapReduce: 14章
Hadoop核心架构HDFS+MapReduce+Hbase+Hive内部机理详解
编者按：HDFS和MapReduce是Hadoop的两大核心，除此之外Hbase、Hive这两个核心工具也随着Hadoop发展变得越来越重要。本文作者张震的博文《Thinking in BigDate（八）大数据Hadoop核心架构HDFS+MapReduce+Hbase+Hive内部机理详解》从内部机理详细的分析了HDFS、MapReduce、Hbase、Hive的运行机制，从底层到数据管理
大数据Hadoop核心架构HDFS+MapReduce+Hbase+Hive内部机理详解
通过对Hadoop分布式计算平台最核心的分布式文件系统HDFS、MapReduce处理过程，以及数据仓库工具Hive和分布式数据库Hbase的介绍。基本涵盖了Hadoop分布式平台的所有技术核心。从体系架构到数据定义到数据存储再到数据处理，从宏观到微观的系统介绍，为Hadoop平台上大规模的数据存储和任务处理打下基础。部分参考《Hadoop实战指南》...
Hadoop生态社区

20,848

社区成员

4,696

社区内容

发帖
与我相关
我的任务
社区描述
Hadoop生态大数据交流社区，致力于有Hadoop，hive，Spark，Hbase，Flink，ClickHouse，Kafka，数据仓库，大数据集群运维技术分享和交流等。致力于收集优质的博客
社区管理员
  • 分布式计算/Hadoop社区
  • 涤生大数据
加入社区
  • 近7日
  • 近30日
  • 至今

加载中

查看更多榜单
社区公告
暂无公告

试试用AI创作助手写篇文章吧

+ 用AI写文章