spark分布式集群安装配置问题

[HITxuefeng]

spark正常安装配置启动之后，在master节点上只有worker进程，而没有master进程，这是为什么啊！

root@master:/opt/spark-2.1.0-bin-hadoop2.7/logs# start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /opt/spark-2.1.0-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.master.Master-1-master.out
slave1: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-2.1.0-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave1.out
master: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-2.1.0-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-master.out
slave2: starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark-2.1.0-bin-hadoop2.7/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-slave2.out
root@master:/opt/spark-2.1.0-bin-hadoop2.7/logs# jps
11440 Worker
11486 Jps


我查看日志如下：
root@master:/opt/spark-2.1.0-bin-hadoop2.7/logs# vi spark-root-org.apache.spark.deploy.master.Master-1-master.out

Spark Command: /opt/jdk1.8.0_121/bin/java -cp /opt/spark-2.1.0-bin-hadoop2.7/conf/:/opt/spark-2.1.0-bin-hadoop2.7/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host master --port 7077 --webui-port 18080
========================================
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/04/12 10:16:19 INFO Master: Started daemon with process name: 11338@master
17/04/12 10:16:19 INFO SignalUtils: Registered signal handler for TERM
17/04/12 10:16:19 INFO SignalUtils: Registered signal handler for HUP
17/04/12 10:16:19 INFO SignalUtils: Registered signal handler for INT
17/04/12 10:16:19 WARN MasterArguments: SPARK_MASTER_IP is deprecated, please use SPARK_MASTER_HOST
17/04/12 10:16:19 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/04/12 10:16:20 INFO SecurityManager: Changing view acls to: root
17/04/12 10:16:20 INFO SecurityManager: Changing modify acls to: root
17/04/12 10:16:20 INFO SecurityManager: Changing view acls groups to:
17/04/12 10:16:20 INFO SecurityManager: Changing modify acls groups to:
17/04/12 10:16:20 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
17/04/12 10:16:20 WARN Utils: Service 'sparkMaster' could not bind on port 7077. Attempting port 7078.
17/04/12 10:16:20 WARN Utils: Service 'sparkMaster' could not bind on port 7078. Attempting port 7079.
17/04/12 10:16:20 WARN Utils: Service 'sparkMaster' could not bind on port 7079. Attempting port 7080.
17/04/12 10:16:20 WARN Utils: Service 'sparkMaster' could not bind on port 7080. Attempting port 7081.
17/04/12 10:16:20 WARN Utils: Service 'sparkMaster' could not bind on port 7081. Attempting port 7082.

[HITxuefeng]

怎么解决呢？我${SPARK_HOME}/conf/slaves 已经把master的host写进去，并且在slave节点上都有worker进程

[HITxuefeng]

root@master:~# vi /opt/spark-2.1.0-bin-hadoop2.7/conf/slaves # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # A Spark Worker will be started on each of the machines listed below. master slave1 slave2

[HITxuefeng]

已经OK了，我在spark配置文件中写了一行： export SPARK_MASTER_HOST=7080 感谢二楼！

[LinkSe7en]

17/04/12 10:16:20 WARN Utils: Service 'sparkMaster' could not bind on port 7077. Attempting port 7078. 17/04/12 10:16:20 WARN Utils: Service 'sparkMaster' could not bind on port 7078. Attempting port 7079. 17/04/12 10:16:20 WARN Utils: Service 'sparkMaster' could not bind on port 7079. Attempting port 7080. 17/04/12 10:16:20 WARN Utils: Service 'sparkMaster' could not bind on port 7080. Attempting port 7081. 17/04/12 10:16:20 WARN Utils: Service 'sparkMaster' could not bind on port 7081. Attempting port 7082. 端口被占用了呗。 另外你看看你${SPARK_HOME}/conf/slaves 是不是把master的host写进去了。。。