新手,IDEA 远程提交任务失败
#!/usr/bin/env bash
export SPARK_WORKER_MEMORY=3000m
#本机ip
SPARK_LOCAL_IP=10.1.5.172
JAVA_HOME=/JAVA/jdk1.8.0_181
SPARK_MASTER_HOST=10.1.5.172
在windows 上 搭建开发环境,用的是IDEA 。配置完开发环境。 写代码。build 都正常。内容代码如下:
import org.apache.spark.{SparkConf,SparkContext}
object Main {
def main(args: Array[String]): Unit = {
var masterUrl = "spark://10.1.5.172:7077"
var inputPath="D:\\\\SparkWork\\helloSpark.txt"
val conf = new SparkConf()
.setAppName("myProject1")
.setMaster(masterUrl)
.set("spark.cores.max","2")
.set("spark.executor.memory","512m")
val sc = new SparkContext(conf)
val textFile = sc.textFile(inputPath)
println(textFile.count())
}
}
提交服务器计算:
在IDEA出现以下情况,一致未能计算完成
2018-11-07 16:40:46 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20181107164005-0001/23 is now RUNNING
2018-11-07 16:40:46 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Asked to remove non-existent executor 22
2018-11-07 16:40:48 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20181107164005-0001/23 is now EXITED (Command exited with code 1)
2018-11-07 16:40:48 INFO StandaloneSchedulerBackend:54 - Executor app-20181107164005-0001/23 removed: Command exited with code 1
2018-11-07 16:40:48 INFO StandaloneAppClient$ClientEndpoint:54 - Executor added: app-20181107164005-0001/24 on worker-20181107155456-10.1.5.172-33496 (10.1.5.172:33496) with 2 core(s)
2018-11-07 16:40:48 INFO BlockManagerMaster:54 - Removal of executor 23 requested
2018-11-07 16:40:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Asked to remove non-existent executor 23
2018-11-07 16:40:48 INFO StandaloneSchedulerBackend:54 - Granted executor ID app-20181107164005-0001/24 on hostPort 10.1.5.172:33496 with 2 core(s), 512.0 MB RAM
2018-11-07 16:40:48 INFO BlockManagerMasterEndpoint:54 - Trying to remove executor 23 from BlockManagerMaster.
2018-11-07 16:40:48 INFO StandaloneAppClient$ClientEndpoint:54 - Executor updated: app-20181107164005-0001/24 is now RUNNING
无限制循环下去。
在spark ui上
Running Applications (1)
Application ID Name Cores Memory per Executor Submitted Time User State Duration
app-20181107164005-0001 (kill) myProject1 2 512.0 MB 2018/11/07 16:40:05 user RUNNING 1.2 min
任务如下:
Executor Summary (79)
ExecutorID Worker Cores Memory State Logs
78 worker-20181107155456-10.1.5.172-33496 2 512 RUNNING stdout stderr
14 worker-20181107155456-10.1.5.172-33496 2 512 EXITED stdout stderr
47 worker-20181107155456-10.1.5.172-33496 2 512 EXITED stdout stderr
25 worker-20181107155456-10.1.5.172-33496 2 512 EXITED stdout stderr
stderr错误如下:
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1713)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:63)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:188)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:293)
at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:201)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:64)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
... 4 more
Caused by: java.io.IOException: Failed to connect to az-tdr-1/192.168.30.59:61237
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:245)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:187)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:198)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: win-pc/192.168.3.59:61237
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
... 1 more
Caused by: java.net.ConnectException: Connection refused
请各位赐教!
相关推荐
idea远程调试代码(图文详细)
这几天,我做的项目中需要使用第三方的 API,在第三方的 API 回调时,出现各种错误,需要远程调试。之前做远程调试的时候,我只会在代码中输出日志,记录下来做分析处理,但这样做既麻烦又费时,往往还不能准确定位问题,恶心至极。之前做 .NET 开发时用过 Remote Debug 这个功能,想想 Idea 这么牛逼,是不是也有这个功能,上网一查,还真有,共享出来,与诸君共勉。
IDEA使用Git将本地工程推送到远程仓库
本篇文章介绍一下,IDEA开发工具如何使用Git将本地工程推送到远程仓库,具体步骤如下所示。目录(1)创建gitee远程仓库 (2)创建本地工程(3)初始化本地仓库(4)提交代码到本地仓库(5)推送本地仓库到远程仓库(6)远程仓库查看是否成功,环境准备,安装Git工具 安装IDEA开发工具 搭建一个maven工程 (1)创建gitee远程仓库 这里使用gitee码云作为远程仓库,创建【idea_git_test】远程仓库,然后初始化仓库即可。找到仓库地址,这个地址.
误将.idea文件提交至git后删除的操作方法
主要介绍了误将.idea文件提交至git后删除的操作方法,本文给大家介绍的非常详细,对大家的学习或工作具有一定的参考借鉴价值,需要的朋友可以参考下
IDEA中Maven依赖下载失败解决方案
使用IDEA进行Maven项目开发时,时不时会遇到pom.xml报错的情况,其中很大概率是因为Maven依赖的jar包下载失败,找来找去也没有找到是什么问题,困扰了很多程序猿,这里给出IDEA中Maven依赖下载失败解决方案,给大家参考,实测有用。 文章目录首先检查网络有没有问题,确定网络没有问题,请看下一步多次点击重新导入Maven依赖的按钮设置自动导入Maven依赖在IDEA中找到Maven的配置文件的地址,然后检查配置的远程仓库或者镜像有没有问题如果上面几步都没有解决问题,可以使用以下脚本删除Mvaen中的lastUpdated文件Maven仓库依赖存在依旧报错 首先检查网络有没有问题,
Tomcat使用IDEA远程Debug调试的讲解
今天小编就为大家分享一篇关于Tomcat使用IDEA远程Debug调试的讲解,小编觉得内容挺不错的,现在分享给大家,具有很好的参考价值,需要的朋友一起跟随小编来看看吧


