scala - Spark Runtime Error - ClassDefNotFound: SparkConf -

after installing , building apache spark (albeit quite few warnings), compilation of our spark application (using "sbt package") completes successfully. however, when trying run our application using spark-submit script, runtime error results states sparkconf class definition not found. sparkconf.scala file present on our system, seems if not being built correctly. ideas on how solve this?

user@compname:~/documents/testapp$ /opt/spark/spark-1.4.0/bin/spark-submit --master local[4] --jars /opt/spark/spark-1.4.0/jars/elasticsearch-hadoop-2.1.0.beta2.jar target/scala-2.11/sparkesingest_2.11-1.0.0.jar ~/desktop/csv/data.csv es-index localhost warning: local jar /opt/spark/spark-1.4.0/jars/elasticsearch-hadoop-2.1.0.beta2.jar not exist, skipping. log4j:warn no appenders found logger (app). log4j:warn please initialize log4j system properly. log4j:warn see http://logging.apache.org/log4j/1.2/faq.html#noconfig more info. using spark's default log4j profile: org/apache/spark/log4j-defaults.properties 15/07/01 13:56:58 info sparkcontext: running spark version 1.4.0 15/07/01 13:56:59 warn nativecodeloader: unable load native-hadoop library platform... using builtin-java classes applicable 15/07/01 13:56:59 warn utils: hostname, compname resolves loopback address: 127.0.1.1; using [ip address] instead (on interface eth0) 15/07/01 13:56:59 warn utils: set spark_local_ip if need bind address 15/07/01 13:56:59 info securitymanager: changing view acls to: user 15/07/01 13:56:59 info securitymanager: changing modify acls to: user 15/07/01 13:56:59 info securitymanager: securitymanager: authentication disabled; ui acls disabled; users view permissions: set(user); users modify permissions: set(user) 15/07/01 13:56:59 info slf4jlogger: slf4jlogger started 15/07/01 13:56:59 info remoting: starting remoting 15/07/01 13:56:59 info remoting: remoting started; listening on addresses :[akka.tcp://sparkdriver@[ip address]] 15/07/01 13:56:59 info utils: started service 'sparkdriver' on port 34276. 15/07/01 13:56:59 info sparkenv: registering mapoutputtracker 15/07/01 13:56:59 info sparkenv: registering blockmanagermaster 15/07/01 13:56:59 info diskblockmanager: created local directory @ /tmp/spark-c206e297-c2ef-4bbf-9bd2-de642804bdcd/blockmgr-8d273f32-589e-4f55-98a2-cf0322a05d45 15/07/01 13:56:59 info memorystore: memorystore started capacity 265.4 mb 15/07/01 13:56:59 info httpfileserver: http file server directory /tmp/spark-c206e297-c2ef-4bbf-9bd2-de642804bdcd/httpd-f4c3c67a-d058-4aba-bd65-5352feb5f12e 15/07/01 13:56:59 info httpserver: starting http server 15/07/01 13:56:59 info utils: started service 'http file server' on port 33599. 15/07/01 13:56:59 info sparkenv: registering outputcommitcoordinator 15/07/01 13:56:59 info utils: started service 'sparkui' on port 4040. 15/07/01 13:56:59 info sparkui: started sparkui @ http://[ip address]:4040 15/07/01 13:57:00 error sparkcontext: jar not found @ file:/opt/spark/spark-1.4.0/jars/elasticsearch-hadoop-2.1.0.beta2.jar 15/07/01 13:57:00 info sparkcontext: added jar file:/home/user/documents/testapp/target/scala-2.11/sparkesingest_2.11-1.0.0.jar @ http://[ip address]:33599/jars/sparkesingest_2.11-1.0.0.jar timestamp 1435784220028 15/07/01 13:57:00 info executor: starting executor id driver on host localhost 15/07/01 13:57:00 info utils: started service 'org.apache.spark.network.netty.nettyblocktransferservice' on port 44746. 15/07/01 13:57:00 info nettyblocktransferservice: server created on 44746 15/07/01 13:57:00 info blockmanagermaster: trying register blockmanager 15/07/01 13:57:00 info blockmanagermasterendpoint: registering block manager localhost:44746 265.4 mb ram, blockmanagerid(driver, localhost, 44746) 15/07/01 13:57:00 info blockmanagermaster: registered blockmanager 15/07/01 13:57:00 info memorystore: ensurefreespace(143840) called curmem=0, maxmem=278302556 15/07/01 13:57:00 info memorystore: block broadcast_0 stored values in memory (estimated size 140.5 kb, free 265.3 mb) 15/07/01 13:57:00 info memorystore: ensurefreespace(12635) called curmem=143840, maxmem=278302556 15/07/01 13:57:00 info memorystore: block broadcast_0_piece0 stored bytes in memory (estimated size 12.3 kb, free 265.3 mb) 15/07/01 13:57:00 info blockmanagerinfo: added broadcast_0_piece0 in memory on localhost:44746 (size: 12.3 kb, free: 265.4 mb) 15/07/01 13:57:00 info sparkcontext: created broadcast 0 textfile @ ingest.scala:159 exception in thread "main" java.lang.noclassdeffounderror: org/apache/spark/sparkconf     @ org.elasticsearch.spark.rdd.compatutils.<clinit>(compatutils.java:20)     @ java.lang.class.forname0(native method)     @ java.lang.class.forname(class.java:274)     @ org.elasticsearch.hadoop.util.objectutils.loadclass(objectutils.java:71)     @ org.elasticsearch.spark.package$.<init>(package.scala:14)     @ org.elasticsearch.spark.package$.<clinit>(package.scala)     @ build.ingest$.main(ingest.scala:176)     @ build.ingest.main(ingest.scala)     @ sun.reflect.nativemethodaccessorimpl.invoke0(native method)     @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:57)     @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43)     @ java.lang.reflect.method.invoke(method.java:606)     @ org.apache.spark.deploy.sparksubmit$.org$apache$spark$deploy$sparksubmit$$runmain(sparksubmit.scala:664)     @ org.apache.spark.deploy.sparksubmit$.dorunmain$1(sparksubmit.scala:169)     @ org.apache.spark.deploy.sparksubmit$.submit(sparksubmit.scala:192)     @ org.apache.spark.deploy.sparksubmit$.main(sparksubmit.scala:111)     @ org.apache.spark.deploy.sparksubmit.main(sparksubmit.scala) caused by: java.lang.classnotfoundexception: org.apache.spark.sparkconf     @ java.net.urlclassloader$1.run(urlclassloader.java:366)     @ java.net.urlclassloader$1.run(urlclassloader.java:355)     @ java.security.accesscontroller.doprivileged(native method)     @ java.net.urlclassloader.findclass(urlclassloader.java:354)     @ java.lang.classloader.loadclass(classloader.java:425)     @ java.lang.classloader.loadclass(classloader.java:358)     ... 17 more 15/07/01 13:57:00 info sparkcontext: invoking stop() shutdown hook 15/07/01 13:57:00 info sparkui: stopped spark web ui @ http://[ip address]:4040 15/07/01 13:57:00 info dagscheduler: stopping dagscheduler 15/07/01 13:57:00 info mapoutputtrackermasterendpoint: mapoutputtrackermasterendpoint stopped! 15/07/01 13:57:00 info utils: path = /tmp/spark-c206e297-c2ef-4bbf-9bd2-de642804bdcd/blockmgr-8d273f32-589e-4f55-98a2-cf0322a05d45, present root deletion. 15/07/01 13:57:00 info memorystore: memorystore cleared 15/07/01 13:57:00 info blockmanager: blockmanager stopped 15/07/01 13:57:01 info blockmanagermaster: blockmanagermaster stopped 15/07/01 13:57:01 info sparkcontext: stopped sparkcontext 15/07/01 13:57:01 info outputcommitcoordinator$outputcommitcoordinatorendpoint: outputcommitcoordinator stopped! 15/07/01 13:57:01 info utils: shutdown hook called 15/07/01 13:57:01 info utils: deleting directory /tmp/spark-c206e297-c2ef-4bbf-9bd2-de642804bdcd 

here build.sbt file:

scalaversion := "2.11.6"  name := "sparkesingest"  version := "1.0.0"  librarydependencies ++= seq(     "org.apache.spark" %% "spark-core" % "1.4.0" % "provided",     "org.apache.spark" %% "spark-streaming" % "1.4.0" % "provided",     "org.apache.spark" %% "spark-sql" % "1.4.0" % "provided",     "org.elasticsearch" % "elasticsearch-hadoop" % "2.1.0.beta2" exclude("org.spark-project.akka", "akka-remote_2.10") exclude("org.spark-project.akka", "akka-slf4j_2.10") exclude("org.json4s", "json4s-ast_2.10") exclude("org.apache.spark", "spark-catalyst_2.10") exclude("com.twitter", "chill_2.10") exclude("org.apache.spark", "spark-sql_2.10") exclude("org.json4s", "json4s-jackson_2.10") exclude("org.json4s", "json4s-core_2.10") exclude("org.apache.spark", "spark-core_2.10")   )  if ( system.getenv("query_es_resource") != null) {   println("[info] using lib/es-hadoop-build-snapshot/ unmanagedbase dir")   unmanagedbase <<= basedirectory { base => base / "lib/es-hadoop-build-snapshot" } } else {   println("[info] using lib/ unmanagedbase dir")   unmanagedbase <<= basedirectory { base => base / "lib" } }  resolvers += "conjars.org" @ "http://conjars.org/repo"  resolvers += "clojars" @ "https://clojars.org/repo" 

is spark jar inside jar submitting ? seems tell sbt you're providing jar don't see "unmanagedjars in compile += file(...)" in you're sbt, if you're counting jar in machine suggest wouldn't since mat lead kind of problems.

try unrar you're jar , see if spark jar there, if not use sbt-assembly or other tool of you're choice.