spark 集群远程提交作业抛异常

黑色的月牙 2018-01-17 04:39:58
提交的命令如下:
[b]bash spark-submit --class com.dqcheck.rules.DataRepeatCheck --master spark://172.100.90.21:7077 jar路径

18/01/17 16:35:41 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://10.111.134.43:4040
18/01/17 16:35:41 INFO spark.SparkContext: Added JAR file:/extend/codeguan/sparkjava-pi/target/spark-pi-1.0-SNAPSHOT.jar at spark://10.111.134.43:43531/jars/spark-pi-1.0-SNAPSHOT.jar with timestamp 1516178141961
18/01/17 16:35:42 INFO client.StandaloneAppClient$ClientEndpoint: Connecting to master spark://172.16.50.21:7077...
18/01/17 16:35:42 INFO client.TransportClientFactory: Successfully created connection to /172.16.50.21:7077 after 29 ms (0 ms spent in bootstraps)
18/01/17 16:35:42 WARN client.StandaloneAppClient$ClientEndpoint: Failed to connect to master 172.16.50.21:7077
org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:100)
at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:108)
at org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1$$anon$1.run(StandaloneAppClient.scala:106)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: java.io.StreamCorruptedException: invalid stream header: 01000D31
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:808)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:301)
at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:63)
at org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:63)
at org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:122)


.....................................
....................................
8/01/17 16:36:42 ERROR cluster.StandaloneSchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.
18/01/17 16:36:42 WARN cluster.StandaloneSchedulerBackend: Application ID is not initialized yet.
18/01/17 16:36:42 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37027.
18/01/17 16:36:42 INFO netty.NettyBlockTransferService: Server created on 10.111.134.43:37027
18/01/17 16:36:42 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
18/01/17 16:36:42 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.111.134.43, 37027, None)
18/01/17 16:36:42 INFO server.AbstractConnector: Stopped Spark@21a5fd96{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
18/01/17 16:36:42 INFO storage.BlockManagerMasterEndpoint: Registering block manager 10.111.134.43:37027 with 366.3 MB RAM, BlockManagerId(driver, 10.111.134.43, 37027, None)
18/01/17 16:36:42 INFO ui.SparkUI: Stopped Spark web UI at http://10.111.134.43:4040
18/01/17 16:36:42 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.111.134.43, 37027, None)
18/01/17 16:36:42 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, 10.111.134.43, 37027, None)
18/01/17 16:36:42 INFO cluster.StandaloneSchedulerBackend: Shutting down all executors
18/01/17 16:36:42 INFO cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
18/01/17 16:36:42 WARN client.StandaloneAppClient$ClientEndpoint: Drop UnregisterApplication(null) because has not yet connected to master
18/01/17 16:36:42 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
18/01/17 16:36:42 INFO memory.MemoryStore: MemoryStore cleared
18/01/17 16:36:42 INFO storage.BlockManager: BlockManager stopped
18/01/17 16:36:42 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
18/01/17 16:36:42 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
18/01/17 16:36:42 ERROR spark.SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
at scala.Predef$.require(Predef.scala:224)
at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:524)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2509)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:909)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:901)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:901)
at com.chinacloud.metagrid.dqcheck.rules.DataRepeatCheck.main(DataRepeatCheck.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
18/01/17 16:36:42 INFO spark.SparkContext: SparkContext already stopped.
18/01/17 16:36:42 INFO spark.SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem
at scala.Predef$.require(Predef.scala:224)
at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:524)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2509)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:909)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:901)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:901)
at com.chinacloud.metagrid.dqcheck.rules.DataRepeatCheck.main(DataRepeatCheck.java:47)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

...全文
2370 1 打赏 收藏 转发到动态 举报
写回复
用AI写文章
1 条回复
切换为时间正序
请发表友善的回复…
发表回复
张某码 2018-02-01
  • 打赏
  • 举报
回复
初步判断是序列化问题

20,807

社区成员

发帖
与我相关
我的任务
社区描述
Hadoop生态大数据交流社区,致力于有Hadoop,hive,Spark,Hbase,Flink,ClickHouse,Kafka,数据仓库,大数据集群运维技术分享和交流等。致力于收集优质的博客
社区管理员
  • 分布式计算/Hadoop社区
  • 涤生大数据
加入社区
  • 近7日
  • 近30日
  • 至今
社区公告
暂无公告

试试用AI创作助手写篇文章吧