springboot 提交 spark任务报错

daiqw 2018-07-12 06:17:37
新手刚刚自学spark,遇到一个棘手的问题,在linux系统上用命令提交没什么问题,但是在springboot框架里面提交spark任务总是报错,调了好久都没解决, spark 1.6.2 scala 2.10.5,请大神指教
2018-07-12 18:09:11.031 WARN 12076 --- [nio-8080-exec-1] o.a.s.deploy.rest.RestSubmissionClient : Unable to connect to server spark://192.168.1.229:7077.
2018-07-12 18:09:11.223 WARN 12076 --- [nio-8080-exec-1] org.apache.hadoop.util.NativeCodeLoader : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-07-12 18:09:11.267 INFO 12076 --- [nio-8080-exec-1] org.apache.spark.SecurityManager : Changing view acls to: cp110
2018-07-12 18:09:11.268 INFO 12076 --- [nio-8080-exec-1] org.apache.spark.SecurityManager : Changing modify acls to: cp110
2018-07-12 18:09:11.268 INFO 12076 --- [nio-8080-exec-1] org.apache.spark.SecurityManager : SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(cp110); users with modify permissions: Set(cp110)
2018-07-12 18:09:11.931 INFO 12076 --- [nio-8080-exec-1] org.apache.spark.util.Utils : Successfully started service 'driverClient' on port 62815.
2018-07-12 18:09:12.015 ERROR 12076 --- [huffle-client-0] o.a.s.network.client.TransportClient : Failed to send RPC 7209509134568250263 to /192.168.1.229:7077: java.lang.AbstractMethodError: org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;

java.lang.AbstractMethodError: org.apache.spark.network.protocol.MessageWithHeader.touch(Ljava/lang/Object;)Lio/netty/util/ReferenceCounted;
at io.netty.util.ReferenceCountUtil.touch(ReferenceCountUtil.java:77) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:810) [netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) [netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111) ~[netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) [netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) [netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:816) [netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723) [netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:302) [netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738) [netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730) [netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:38) [netty-all-4.1.25.Final.jar:4.1.25.Final]
at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:1081) [netty-all-4.1.25.Final.jar:4.1.25.Final]


...全文
499 回复 打赏 收藏 转发到动态 举报
写回复
用AI写文章
回复
切换为时间正序
请发表友善的回复…
发表回复

1,261

社区成员

发帖
与我相关
我的任务
社区描述
Spark由Scala写成,是UC Berkeley AMP lab所开源的类Hadoop MapReduce的通用的并行计算框架,Spark基于MapReduce算法实现的分布式计算。
社区管理员
  • Spark
  • shiter
加入社区
  • 近7日
  • 近30日
  • 至今
社区公告
暂无公告

试试用AI创作助手写篇文章吧