CODEWITHSUNDEEP
scalaakka
WestCoastProjects
WestCoastProjects

Reputation: 63082

Mixing up of Scala library supplied akka-actors.jar and application specific version

The following SOF explains why akka related method not found exceptions may occur: TL;DR scala boot loader puts scala libraries version of akka-actors first in classpath

RemoteActorRefProvider ClassNotFound

However simply removing the scala library version in favor of the custom one causes other issues:

14/10/28 09:30:23 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(steve); users with modify permissions: Set(steve)
java.lang.NoSuchMethodError: com.typesafe.config.Config.getDuration(Ljava/lang/String;Ljava/util/concurrent/TimeUnit;)J
    at akka.util.Helpers$ConfigOps$.akka$util$Helpers$ConfigOps$$getDuration$extension(Helpers.scala:125)

So it is not clear what is the correct way to handle custom akka-actor jars.

Upvotes: 1

Views: 1717

Answers (2)

nir
nir

Reputation: 3868

Please look at this JIRA. I think spark-core 1.3.1 package is including incorrect typesafe config APIs. https://issues.apache.org/jira/browse/SPARK-9441 Try following maven or equivalent gradle ,sbt dependencies

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<exclusions>
  <exclusion>
   <artifactId>config</artifactId>
   <groupId>com.typesafe</groupId>
  </exclusion>
 </exclusions>
</dependency>

<dependency>
 <groupId>com.typesafe</groupId>
 <artifactId>config</artifactId>
 <version>1.2.1</version>
</dependency>

Upvotes: 0

WestCoastProjects
WestCoastProjects

Reputation: 63082

The workaround for this is a bit surprising. The scala interpreter is not the way to go. Instead fall back to the (drum roll..) java interpreter to run the (yes scala!) program.

The reason is that the scala interpreter adds in various things - including the akka classes into the boot classpath. Apparently no usable way around that.

To run with java we will need to include the $SCALA_home/lib/* in the classpath. After that - it works just fine.

Upvotes: 1

Related Questions