HIVE SQL耗时过长问题
如下语句: 跑了12个小时,没跑完
select
count(1),count(case when msisdn<>'' then 1 else null end)
from
zb_dwd.dwd_d_ia_mme_part
where date_id ='20180925';
HADOOP JOB LIST: 改作业占用资源:
内存:1046016M
CPU: 171
想不通,一天数据才200多个GB,内存1个TB了, 为甚一条语句跑了他妈12小时 跑不完,请大神指点
建表语句:
CREATE TABLE `zb_dwd.dwd_d_ia_mme_part`(
`xdr_id` string,
`imsi` string,
`imei` string,
`msisdn` string,
`starttime` string,
`endtime` string,
`group_id` string,
`code` string,
`tac` string,
`cell_id` string)
PARTITIONED BY (
`date_id` string,
`hour_id` string)
命令:hadoop fs -du -h hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925
分区结果如下:
94.0 M hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=00
100.3 M hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=01
101.4 M hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=02
81.5 M hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=03
95.6 M hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=04
80.4 M hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=05
101.4 M hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=06
87.9 M hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=07
132.1 M hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=08
18.6 G hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=09
19.5 G hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=10
19.4 G hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=11
18.9 G hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=12
18.8 G hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=13
19.4 G hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=14
18.3 G hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=15
18.0 G hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=16
18.0 G hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=17
17.4 G hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=18
17.2 G hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=19
16.1 G hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=20
13.5 G hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=21
13.9 G hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=22
11.7 G hdfs://hacluster/user/hive/warehouse/zb_dwd.db/dwd_d_ia_mme_part/date_id=20180925/hour_id=23