How to import own scala package using spark-shell?

Question

I have written a class for spark-ml library that uses another classes from it. If to be clear, my class is a wrapper for RandomForestClassifier. Now I want to have an opportunity to import this class from spark-shell.

So the question is: how to make package containing my own class that it will be able to be imported from spark-shell? Many thanks!

Answer 1

If you want to import uncompiled files like Hello.scala, do below in spark shell:

scala> :load ./src/main/scala/Hello.scala

Answer 2

Read the docs:

In the Spark shell, a special interpreter-aware SparkContext is already created for you, in the variable called sc. Making your own SparkContext will not work. You can set which master the context connects to using the --master argument, and you can add JARs to the classpath by passing a comma-separated list to the --jars argument. You can also add dependencies (e.g. Spark Packages) to your shell session by supplying a comma-separated list of maven coordinates to the --packages argument. Any additional repositories where dependencies might exist (e.g. SonaType) can be passed to the --repositories argument.