java.lang.ClassNotFoundException: XXX

[wozaiztam]

本人刚学Spark，有个最简单的例子，但是我在spark中运行报以上错误，

package example

import com.typesafe.scalalogging.LazyLogging
import org.apache.spark.{SparkConf, SparkContext}

object SparkExample extends LazyLogging {

  private val master = "spark://master:7077"
  private val appName = "example-spark"
  private val stopWords = Set("gif", "png")

  def main(args: Array[String]): Unit = {

    val conf = new SparkConf()
      .setMaster(master)
      .setAppName(appName)

    val sc = new SparkContext(conf)

    val lines = sc.textFile("file:/home/qiujl/access.log")
    val wordsCount = WordsCount.count(sc, lines, stopWords)

    val counts = wordsCount.collect().mkString("[", ", ", "]")
    logger.info(counts)
  }
}

报以下错误：
[qwwsl@master ~]$ spark-submit --master spark://master:7077 --num-executors 48 --driver-memory 2g --executor-memory 7g --executor-cores 3 /home/qiujl/example-spark-master.jar
18/05/30 10:27:38 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
java.lang.ClassNotFoundException: example.SparkExample
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:235)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:836)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

[tianfang]

java.lang.ClassNotFoundException: example.SparkExample 你的代码没有打入jar

[wozaiztam]

引用 1 楼 tianfang 的回复:

java.lang.ClassNotFoundException: example.SparkExample 你的代码没有打入jar

能具体点吗？我刚自学，不太清楚

[D.Foil]

打包的时候要把代码加进去，命令指明哪个class，--class example.SparkExample

[wozaiztam]

引用 3 楼 wR__500 的回复:

打包的时候要把代码加进去，命令指明哪个class，--class example.SparkExample

请问能具体点吗？写在代码哪里？还是配置中？

[4qw]

看情况是,你打包时,默认指定main class,具体你可以看下如何打包默认指定方法,另外spark可以默认本地执行的,不必非得提交到服务器,可以用val sparkConf = new SparkConf().setAppName("SparkTest").setMaster("local"),就可以在window开发环境下执行;你可以看下你打的包中的内容,能否找到该类,可以用jd-gui查看jar包.

[cocoa2003]

submit的时候需要把jar包指定一下，同时把jar包上传到服务器的spark下面。
例如${SPARK_HOME}/bin/spark-submit --master spark://10.70.64.15:8080 --jars ./lib/*.jar --class org.apache.sparkexamples.SparkSQL2 ./jar/first.jar