CODEWITHSUNDEEP
scalasbtofflineremote-accessapache-spark
tsjnsn
tsjnsn

Reputation: 451

Using SBT on a remote node without internet access via SSH

I am trying to write a Spark program with Scala on a remote machine, but that machine has no internet access. Since I am using the pre-built version for Hadoop, I am able to run the pre-compiled examples:

[user@host spark-0.7.2]$ ./run spark.examples.LocalPi

but I can't compile anything that references spark on the machine:

$ scalac PiEstimate.scala
PiEstimate.scala:1: error: not found: object spark
import spark.SparkContext
       ^

Normally, I would use SBT to take care of any dependencies, but the machine does not have internet access, and tunneling internet through SSH is not possible.

Is it possible to compile an SBT project on a remote machine that has no internet access? Or how could I manually link the Spark dependencies to the Scala compiler.

Upvotes: 2

Views: 909

Answers (2)

savx2
savx2

Reputation: 1071

I know this is an old post but I had to deal with this issue recently. I solved it by removing the dependencies from my .sbt file and adding the spark jar (spark-home/assembly/target/scala.2-10/spark-[...].jar) under my-project-dir/lib directory. You can also point to it using unmanagedBase = file("/path/to/jars/") Then I could use sbt package as usually

Upvotes: 0

Josh Rosen
Josh Rosen

Reputation: 13841

If you're compiling your Spark program through scalac, you'll have to add Spark's jars to scalac's classpath; I think this should work:

scalac -classpath "$SPARK_HOME/target/scala-*/*.jar" PiEstimate.scala

Upvotes: 1

Related Questions