新手，IDEA 远程提交任务失败

lonz 2018-11-07 04:54:51

刚开始学习spark 。在linux上安装了 spark-master + work;
#!/usr/bin/env bash
export SPARK_WORKER_MEMORY=3000m
#本机ip
SPARK_LOCAL_IP=10.1.5.172
JAVA_HOME=/JAVA/jdk1.8.0_181
SPARK_MASTER_HOST=10.1.5.172

在windows 上搭建开发环境，用的是IDEA 。配置完开发环境。写代码。build 都正常。内容代码如下：



import org.apache.spark.{SparkConf,SparkContext}



object Main {

  def main(args: Array[String]): Unit = {

    var masterUrl = "spark://10.1.5.172:7077"

    var inputPath="D:\\\\SparkWork\\helloSpark.txt"



    val conf = new SparkConf()

      .setAppName("myProject1")

      .setMaster(masterUrl)

      .set("spark.cores.max","2")

      .set("spark.executor.memory","512m")



    val sc = new SparkContext(conf)

    val textFile = sc.textFile(inputPath)

    println(textFile.count())

  }

}

提交服务器计算：
在IDEA出现以下情况，一致未能计算完成

2018-11-07 16:40:46 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20181107164005-0001/23 is now RUNNING
2018-11-07 16:40:46 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Asked to remove non-existent executor 22
2018-11-07 16:40:48 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20181107164005-0001/23 is now EXITED (Command exited with code 1)
2018-11-07 16:40:48 INFO StandaloneSchedulerBackend:54 - Executor app-20181107164005-0001/23 removed: Command exited with code 1
2018-11-07 16:40:48 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20181107164005-0001/24 on worker-20181107155456-10.1.5.172-33496 (10.1.5.172:33496) with 2 core(s)
2018-11-07 16:40:48 INFO BlockManagerMaster:54 - Removal of executor 23 requested
2018-11-07 16:40:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Asked to remove non-existent executor 23
2018-11-07 16:40:48 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20181107164005-0001/24 on hostPort 10.1.5.172:33496 with 2 core(s), 512.0 MB RAM
2018-11-07 16:40:48 INFO BlockManagerMasterEndpoint:54 - Trying to remove executor 23 from BlockManagerMaster.
2018-11-07 16:40:48 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20181107164005-0001/24 is now RUNNING

无限制循环下去。

在spark ui上

Running Applications (1)
Application ID Name Cores Memory per Executor Submitted Time User State Duration
app-20181107164005-0001 (kill) myProject1 2 512.0 MB 2018/11/07 16:40:05 user RUNNING 1.2 min

任务如下：
Executor Summary (79)
ExecutorID Worker Cores Memory State Logs
78 worker-20181107155456-10.1.5.172-33496 2 512 RUNNING stdout stderr
14 worker-20181107155456-10.1.5.172-33496 2 512 EXITED stdout stderr
47 worker-20181107155456-10.1.5.172-33496 2 512 EXITED stdout stderr
25 worker-20181107155456-10.1.5.172-33496 2 512 EXITED stdout stderr

stderr错误如下：
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1713)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:63)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:188)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:293)
at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:201)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:64)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
... 4 more
Caused by: java.io.IOException: Failed to connect to az-tdr-1/192.168.30.59:61237
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:245)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:187)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:198)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: win-pc/192.168.3.59:61237
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
... 1 more
Caused by: java.net.ConnectException: Connection refused

请各位赐教！

...全文