1,261
社区成员
发帖
与我相关
我的任务
分享spark读取hive，写入hbase
有一个文件hive_file里面写了多个hive表的名称，我要用spark 读取hive_file的内容，遍历每行，把hive表的数据写入到hbase。
自己试了spark hivecontext读取一个表可以，但是在hive_file内容foreach遍历时候，就不能读取了，有做过的求教。
自己试了spark hivecontext读取一个表可以，但是在hive_file内容foreach遍历时候，就不能读取了，有做过的求教。
...全文
请发表友善的回复…
发表回复
LinkSe7en 2017-05-03
- 打赏
- 举报
val lines = sc.textFile("hdfs://myspark1:9000/testsrc/hive_tab", 1);
lines.foreach({ line =>
println(line)
//读取hive表数据,输出
val goodStudentsDF = hiveContext.sql("select * from " + line)
.rdd.foreach(row => {
println("row.size:" + row.size + " " + row.getString(0))
})
})
你这里，foreach是在executor上执行的。executor哪有hiveContext对象？你需要把lines collect回driver，用普通scala集合的foreach去做
mmmc_a 2017-05-03
- 打赏
- 举报
求解答。。。。。。。。
代码：
def main(args: Array[String]): Unit = {
//初始化context
val conf = new SparkConf()
.setAppName("HiveDataSource").setMaster("local");
val sc = new SparkContext(conf);
val hiveContext = new HiveContext(sc);
//从文件中读取hive表名称
val lines = sc.textFile("hdfs://myspark1:9000/testsrc/hive_tab", 1);
lines.foreach({ line =>
println(line)
//读取hive表数据,输出
val goodStudentsDF = hiveContext.sql("select * from " + line)
.rdd.foreach(row => {
println("row.size:" + row.size + " " + row.getString(0))
})
})
sc.stop()
}
错误：
17/05/02 21:11:03 INFO Configuration.deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
17/05/02 21:11:03 INFO Configuration.deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
student_infos
17/05/02 21:11:06 ERROR executor.Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.NullPointerException
at org.apache.spark.sql.hive.HiveContext.parseSql(HiveContext.scala:277)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:725)
at HiveReader$$anonfun$main$1.apply(HiveReader.scala:27)
at HiveReader$$anonfun$main$1.apply(HiveReader.scala:25)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:890)
at org.apache.spark.rdd.RDD$$anonfun$foreach$1$$anonfun$apply$28.apply(RDD.scala:890)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1848)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1848)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
17/05/02 21:11:06 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.NullPointerException
