hive启动mapreduce时卡死

[惑先生]

问题描述
1、hadoop集群部署在腾讯云上，一个master，两个slave
2、hadoop可以跑官方wordcount，能出结果
3、hadoop的版本是3.1.2，hive的版本是3.1.1
4、hive在跑select count(1) from t_test时卡死，事实上t_test表中只有两条数据
5、hive卡死的现象如下：

hive> set mapreduce.job.reduces=2;
hive> set hive.compute.query.using.stats=false;
hive> select count(1) from t_test;
Query ID = root_20191130113749_ff62498c-ee7b-41a5-a4e4-bc3653b074d1
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>

6、在/tmp/host/hive.log中发现时有mr的job创建的，如下：
2019-11-30T11:37:55,318 WARN [614f0786-de9d-4388-8fd4-28ecbf967e14 main] mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2019-11-30T11:37:55,399 INFO [614f0786-de9d-4388-8fd4-28ecbf967e14 main] mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1575042073347_0015



7、但是在http://134.175.61.114:8088/cluster/里面却找不见这个job

[惑先生]

几个小时过去来，终于看到错误信息了： Hive Session ID = d802bdae-f12c-4074-be9e-9c425bffed34 hive> set mapreduce.job.reduces=2; hive> set hive.compute.query.using.stats=false; hive> select count(1) from t_test; Query ID = root_20191130113749_ff62498c-ee7b-41a5-a4e4-bc3653b074d1 Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1575042073347_0015, Tracking URL = http://master:8088/proxy/application_1575042073347_0015/ Kill Command = /opt/hadoop/bin/mapred job -kill job_1575042073347_0015 Hadoop job information for Stage-1: number of mappers: 0; number of reducers: 0 2019-11-30 12:41:14,331 Stage-1 map = 0%, reduce = 0% Ended Job = job_1575042073347_0015 with errors Error during job, obtaining debugging information... FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask MapReduce Jobs Launched: Stage-Stage-1: HDFS Read: 0 HDFS Write: 0 FAIL Total MapReduce CPU Time Spent: 0 msec hive> Socket error Event: 32 Error: 10053. Connection closing...Socket close.

[惑先生]

补充一下： 昨天晚上挂起来跑了一夜，早上发现在http://134.175.61.114:8088/cluster/发现来job，信息如下： User: root Name: select count(1) from t_test (Stage-1) Application Type: MAPREDUCE Application Tags: Application Priority: 0 (Higher Integer value indicates higher priority) YarnApplicationState: FAILED Queue: default FinalStatus Reported by AM: FAILED Started: Sat Nov 30 02:13:22 +0800 2019 Elapsed: 24mins, 30sec Tracking URL: History Log Aggregation Status: DISABLED Application Timeout (Remaining Time): Unlimited Diagnostics: Application application_1575042073347_0009 failed 2 times due to ApplicationMaster for attempt appattempt_1575042073347_0009_000002 timed out. Failing the application. Unmanaged Application: false Application Node Label expression: <Not set> AM container Node Label expression: <DEFAULT_PARTITION>

[惑先生]

如果感兴趣，我可以提供系统账号，查看更丰富详细的日志

[RivenDong]

你不加前面两条，直接执行 select count(1) from t_test; 会怎么样？

[惑先生]

引用 6 楼 RivenDong 的回复:

[quote=引用 5 楼 ytoms的回复:][quote=引用 4 楼 RivenDong 的回复:] 你不加前面两条，直接执行 select count(1) from t_test; 会怎么样？

很快就返回了，不过是0，因为读的统计信息，不走MR[/quote] 什么叫读的是统计信息？HQL默认就会转换为mr去执行的，本身就不用设置[/quote] hive默认设置 hive.compute.query.using.stats=true //改为false 不从元数据取值，可计数，但性能不高，不推荐

[惑先生]

引用 4 楼 RivenDong 的回复:

你不加前面两条，直接执行 select count(1) from t_test; 会怎么样？

很快就返回了，不过是0，因为读的统计信息，不走MR

[RivenDong]

引用 5 楼 ytoms的回复:

[quote=引用 4 楼 RivenDong 的回复:] 你不加前面两条，直接执行 select count(1) from t_test; 会怎么样？

很快就返回了，不过是0，因为读的统计信息，不走MR[/quote] 什么叫读的是统计信息？HQL默认就会转换为mr去执行的，本身就不用设置

[Onioncat_]

这个怎么解决的，大佬