Hortonworks sandbox2.0 hive中进行select group by问题

jpjiang4648 2014-01-28 08:55:30
在Hortonworks sandbox2.0中进行hive操作,select后跟一个group by 顺利执行,但是跟两个以上group by会出错,请教各位大神。

我的环境是:在windows 7 上安装了虚拟机,装了Hortonworks sandbox2.0.6,对应Hadoop版本是2.2.0,Hive版本是0.12。

情况一:Select后跟一个group by

select trim(pID),max(trim(oID)),max(md),max(newDate)
from table_name group by trim(pID);

顺利执行

情况二:Select后跟二个及以上group by
select trim(pID),max(trim(oID)),trim(cID), trim(prID),max(md),max(newDate)
from table_name group by trim(pID), trim(cID), trim(prID);

出现错误:
14/01/26 21:40:15 ERROR exec.Task: Ended Job = job_1390796761818_0004 with errors
14/01/26 21:40:16 INFO impl.YarnClientImpl: Killing application application_1390796761818_0004
14/01/26 21:40:16 INFO ql.Driver: </PERFLOG method=task.MAPRED.Stage-1 start=1390801038548 end=1390801216121 duration=177573>
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
14/01/26 21:40:16 ERROR ql.Driver: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
14/01/26 21:40:16 INFO ql.Driver: </PERFLOG method=Driver.execute start=1390801038543 end=1390801216121 duration=177578>
MapReduce Jobs Launched:
14/01/26 21:40:16 INFO ql.Driver: MapReduce Jobs Launched:
14/01/26 21:40:16 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
Job 0: Map: 3 Reduce: 1 Cumulative CPU: 18.51 sec HDFS Read: 0 HDFS Write: 0 FAIL
14/01/26 21:40:16 INFO ql.Driver: Job 0: Map: 3 Reduce: 1 Cumulative CPU: 18.51 sec HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 18 seconds 510 msec
14/01/26 21:40:16 INFO ql.Driver: Total MapReduce CPU Time Spent: 18 seconds 510 msec
14/01/26 21:40:16 ERROR beeswax.BeeswaxServiceImpl: Exception while processing query
BeeswaxException(message:Driver returned: 2. Errors: OK
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapred.reduce.tasks=<number>
Starting Job = job_1390796761818_0004, Tracking URL = http://sandbox.hortonworks.com:8088/proxy/application_1390796761818_0004/
Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1390796761818_0004
Hadoop job information for Stage-1: number of mappers: 3; number of reducers: 1
2014-01-26 21:37:28,063 Stage-1 map = 0%, reduce = 0%
2014-01-26 21:38:29,232 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 38.41 sec
2014-01-26 21:38:30,698 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 38.41 sec
2014-01-26 21:38:32,745 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 39.67 sec
2014-01-26 21:38:33,774 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 39.67 sec
2014-01-26 21:38:34,959 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 42.34 sec
2014-01-26 21:38:36,021 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 42.34 sec
2014-01-26 21:38:37,074 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 43.42 sec
2014-01-26 21:38:38,114 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 43.42 sec
2014-01-26 21:38:39,185 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 43.42 sec
2014-01-26 21:38:40,218 Stage-1 map = 0%, reduce = 0%
2014-01-26 21:39:40,252 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 24.74 sec
2014-01-26 21:39:41,372 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 24.74 sec
2014-01-26 21:39:42,422 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 16.97 sec
2014-01-26 21:39:43,464 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 18.51 sec
2014-01-26 21:39:44,495 Stage-1 map = 0%, reduce = 0%
2014-01-26 21:40:13,540 Stage-1 map = 100%, reduce = 100%
MapReduce Total cumulative CPU time: 18 seconds 510 msec
...全文
339 回复 打赏 收藏 转发到动态 举报
写回复
用AI写文章
回复
切换为时间正序
请发表友善的回复…
发表回复
内容概要:本文提出了一种结合在线鲁棒主成分分析(RPCA)模型与长短期记忆(LSTM)循环网络的商品需求预测方法,并提供了完整的Python代码实现。该方法首先利用RPCA模型对原始商品需求时间序列进行分解,分离出低秩的潜在趋势成分与稀疏的异常波动成分,有效实现数据去噪与异常值修正,提升输入数据的鲁棒性;随后将净化后的数据输入LSTM网络,充分挖掘时间序列的长期依赖关系与时序模式,从而提高对未来需求的预测精度。整个模型设计针对实际商业场景普遍存在的数据噪声大、波动剧烈、突发性事件干扰等问题,展现出较强的稳定性与预测能力。文通过实验验证了该混合模型在多个指标上优于传统统计模型及单一LSTM模型,体现了其在复杂环境下的优越性能。; 适合人群:具备一定Python编程能力和机器学习基础知识,从事数据分析、供应链管理、电商运营、零售优化及相关领域研究的研发人员或研究生;特别适合关注时间序列预测、深度学习建模以及鲁棒数据处理技术的技术人员。; 使用场景及目标:①应用于电商平台、零售企业或制造行业的销量预测,以支持库存优化、生产计划制定与物流调度决策;②为科研工作者提供一种融合鲁棒统计与深度学习的预测建模范例,推动高噪声环境下预测算法的创新与复现研究;③帮助开发者深入理解RPCA与LSTM的集成机制,掌握复杂预测模型的构建、训练与调优流程。; 阅读建议:建议读者结合所提供的Python代码逐步实现模型,重点理解RPCA在数据预处理阶段的作用机制以及LSTM网络的结构设计与超参数配置。学习过程应在真实或模拟数据集上复现实验结果,对比不同参数设置下的模型表现,以深化对模型内在工作原理的理解。同时可进一步探索其他深度学习模型(如GRU、Transformer)与鲁棒分解方法(如VMD、STL)的融合可能性,拓展应用场景。

20,842

社区成员

发帖
与我相关
我的任务
社区描述
Hadoop生态大数据交流社区,致力于有Hadoop,hive,Spark,Hbase,Flink,ClickHouse,Kafka,数据仓库,大数据集群运维技术分享和交流等。致力于收集优质的博客
社区管理员
  • 分布式计算/Hadoop社区
  • 涤生大数据
加入社区
  • 近7日
  • 近30日
  • 至今
社区公告
暂无公告

试试用AI创作助手写篇文章吧