CODEWITHSUNDEEP
apache-sparkversionmahout
user3309479
user3309479

Reputation: 327

Mahout and Spark version compatibility

I am currently using Mahout 0.13.0. When I try to use spark-itemsimilarity I get an error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/SparkConf

I suspect it may be an issue with Spark version, but I can't find what version of Spark do I need. I am currently using Spark 2.1.0.

So my question is, with which version of Spark is Mahout 0.13.0 compatible?

Upvotes: 2

Views: 453

Answers (2)

rawkintrevo
rawkintrevo

Reputation: 658

There are no binaries for Mahout 0.13.0 on Spark 2.1 / Scala 2.11, however it is possible to build Mahout for Spark 2.1 (unfortunately there are no profiles, have to do it all with D switches)

to build you need to do the following:

git clone http://github.com/apache/mahout

cd mahout

mvn clean package -Dspark.version=2.1.0 -Dscala.version=2.11.8 -Dscala.compat.version=2.11

Also, Pat is correct- release coming hopefully very very soon, will have binaries.

Upvotes: 1

pferrel
pferrel

Reputation: 5702

I think there is a maven profile in the develop branch for building with Spark 2.x and Scala 2.11 but no artifacts published until 0.13.1 coming in a week or so.

It is compatible with both, just too many combinations of artifacts to catch up with.

Upvotes: 0

Related Questions