云存储
hive join 错误，求大神
墨衍会员 · AI 创作全网分发 墨衍智能分发 通过墨衍一键同步至各平台 可同步平台 CSDN微信公众号微博知乎掘金百家号博客园抖音小红书 你的文章也可以这样分发 墨衍会员 ¥399起/年，写一次发全网
我也要 关注 + 加入社区执行hive sql的时候
hive> select * from records r join records2 r2 on r.year=r2.year join records3 r3 on r3.year=r2.year;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201311101215_106238, Tracking URL = http://hadoop-master.TB.com:50030/jobdetails.jsp?jobid=job_201311101215_106238
Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=hadoop-master.TB.com:8021 -kill job_201311101215_106238
Hadoop job information for Stage-1: number of mappers: 2; number of reducers: 1
2013-12-18 11:00:21,940 Stage-1 map = 0%, reduce = 0%
2013-12-18 11:00:28,378 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 1.7 sec
2013-12-18 11:00:29,416 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 1.7 sec
2013-12-18 11:00:30,459 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 1.7 sec
2013-12-18 11:00:31,495 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 1.7 sec
2013-12-18 11:00:32,535 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 1.7 sec
2013-12-18 11:00:34,237 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 1.7 sec
2013-12-18 11:00:35,270 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 1.7 sec
2013-12-18 11:00:36,310 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:37,424 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:38,459 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:39,483 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:40,736 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:41,779 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:42,814 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:44,069 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:45,110 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:46,130 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:47,162 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:48,189 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:49,438 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:50,481 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:51,938 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:52,970 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:54,007 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:55,043 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:56,082 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:57,113 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:58,150 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:59,177 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1.7 sec
MapReduce Total cumulative CPU time: 1 seconds 700 msec
Ended Job = job_201311101215_106238 with errors
Error during job, obtaining debugging information...
Examining task ID: task_201311101215_106238_m_000003 (and more) from job job_201311101215_106238
Exception in thread "Thread-40" java.lang.RuntimeException: Error while reading from task log url
at org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:130)
at org.apache.hadoop.hive.ql.exec.JobDebugger.showJobFailDebugInfo(JobDebugger.java:211)
at org.apache.hadoop.hive.ql.exec.JobDebugger.run(JobDebugger.java:81)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Server returned HTTP response code: 400 for URL: http://hadoop-sl002.TB.com:50060/tasklog?taskid=attempt_201311101215_106238_m_000000_1&start=-8193
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
at java.net.URL.openStream(URL.java:1010)
at org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:120)
... 3 more
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 2 Reduce: 1 Accumulative CPU: 1.7 sec HDFS Read: 366 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 1 seconds 700 msec
hadoop 的JOBID的日志错误提示
2013-12-18 11:07:01,210 FATAL ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"year":"1990","standard":30}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:550)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:390)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:324)
at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
at org.apache.hadoop.mapred.Child.main(Child.java:260)
Caused by: java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.hadoop.io.IntWritable
at org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyIntObjectInspector.get(LazyIntObjectInspector.java:38)
at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getInt(PrimitiveObjectInspectorUtils.java:519)
at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$IntConverter.convert(PrimitiveObjectInspectorConverter.java:145)
at org.apache.hadoop.hive.ql.udf.generic.GenericUDFUtils$ConversionHelper.convertIfNecessary(GenericUDFUtils.java:345)
at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.evaluate(GenericUDFBridge.java:181)
at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.evaluate(ExprNodeGenericFuncEvaluator.java:163)
at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:225)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:531)
... 9 more
hive> select * from records r join records2 r2 on r.year=r2.year join records3 r3 on r3.year=r2.year;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_201311101215_106238, Tracking URL = http://hadoop-master.TB.com:50030/jobdetails.jsp?jobid=job_201311101215_106238
Kill Command = /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=hadoop-master.TB.com:8021 -kill job_201311101215_106238
Hadoop job information for Stage-1: number of mappers: 2; number of reducers: 1
2013-12-18 11:00:21,940 Stage-1 map = 0%, reduce = 0%
2013-12-18 11:00:28,378 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 1.7 sec
2013-12-18 11:00:29,416 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 1.7 sec
2013-12-18 11:00:30,459 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 1.7 sec
2013-12-18 11:00:31,495 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 1.7 sec
2013-12-18 11:00:32,535 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 1.7 sec
2013-12-18 11:00:34,237 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 1.7 sec
2013-12-18 11:00:35,270 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 1.7 sec
2013-12-18 11:00:36,310 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:37,424 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:38,459 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:39,483 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:40,736 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:41,779 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:42,814 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:44,069 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:45,110 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:46,130 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:47,162 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:48,189 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:49,438 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:50,481 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:51,938 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:52,970 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:54,007 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:55,043 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:56,082 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:57,113 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:58,150 Stage-1 map = 50%, reduce = 17%, Cumulative CPU 1.7 sec
2013-12-18 11:00:59,177 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 1.7 sec
MapReduce Total cumulative CPU time: 1 seconds 700 msec
Ended Job = job_201311101215_106238 with errors
Error during job, obtaining debugging information...
Examining task ID: task_201311101215_106238_m_000003 (and more) from job job_201311101215_106238
Exception in thread "Thread-40" java.lang.RuntimeException: Error while reading from task log url
at org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:130)
at org.apache.hadoop.hive.ql.exec.JobDebugger.showJobFailDebugInfo(JobDebugger.java:211)
at org.apache.hadoop.hive.ql.exec.JobDebugger.run(JobDebugger.java:81)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Server returned HTTP response code: 400 for URL: http://hadoop-sl002.TB.com:50060/tasklog?taskid=attempt_201311101215_106238_m_000000_1&start=-8193
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
at java.net.URL.openStream(URL.java:1010)
at org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getErrors(TaskLogProcessor.java:120)
... 3 more
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 2 Reduce: 1 Accumulative CPU: 1.7 sec HDFS Read: 366 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 1 seconds 700 msec
hadoop 的JOBID的日志错误提示
2013-12-18 11:07:01,210 FATAL ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"year":"1990","standard":30}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:550)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:390)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:324)
at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
at org.apache.hadoop.mapred.Child.main(Child.java:260)
Caused by: java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.hadoop.io.IntWritable
at org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyIntObjectInspector.get(LazyIntObjectInspector.java:38)
at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getInt(PrimitiveObjectInspectorUtils.java:519)
at org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$IntConverter.convert(PrimitiveObjectInspectorConverter.java:145)
at org.apache.hadoop.hive.ql.udf.generic.GenericUDFUtils$ConversionHelper.convertIfNecessary(GenericUDFUtils.java:345)
at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.evaluate(GenericUDFBridge.java:181)
at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.evaluate(ExprNodeGenericFuncEvaluator.java:163)
at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:225)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:531)
... 9 more
...全文
相关推荐
hive优化总结
JOIN操作优化JOIN操作是Hive性能优化的关键之一。在执行JOIN操作时，应该将小表放在前面，以避免内存溢出错误。同时，应该使用MAPJOIN操作，以提高 JOIN 的效率。4.
通俗易懂的Hive join几种形式原理详解
表的Join是数据分析处理过程中必不可少的操作，Hive同样支持Join的语法，Hive Join的底层还是通过MapReduce来实现的，Hive实现Join时，为了提高MapReduce的性能，提供了多种Join方案来实现，例如适合小表Join大表的Map Join，大表Join大表的Reduce Join，以及大表Join的优化方案Bucket Join等。 1 Map Join 应用场景 适合于小表join大表或者小表Join小表 原理 将小的那份数据给每个MapTask的内存都放一.
Hive那些事儿之八-大数据踩过的坑——Hive insert
"Hive在大数据处理中的一个常见操作是数据导入，但在实际使用中可能会遇到各种问题。本文主要讨论了在使用Hive进行数据插入时遇到的一个坑，并提供了相应的解决方案。"在大数据处理中，Hive作为一
hive参数配置说明大全
18. hive.exec.script.maxerrsize该参数决定了通过TRANSFROM/MAP/REDUCE所执行的用户脚本所允许的最大序列化错误数，默认值为100000。
Hive sql优化
**JOIN数据倾斜** - **JOIN原则**：在进行JOIN操作时，应该将较小的表放置在JOIN操作符的左边，因为Hive在执行JOIN时会尝试将左侧表的内容加载到内存中，将小表放在左边有助于减少内存溢出的风险
