CODEWITHSUNDEEP
hadoopapache-spark
horatio1701d
horatio1701d

Reputation: 9159

Building spark without any hadoop dependencies

I found some references to -Phadoop-provided flag for building spark without hadoop libraries but cannot find a good example of how to use it. how can I build spark from source and make sure it does not add any of it's own hadoop dependencies. it looks like when I built the latest spark it included a bunch of 2.8.x hadoop stuff which conflicts with my cluster hadoop version.

Upvotes: 2

Views: 656

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191711

Spark has download options for "pre-built with user-provided Hadoop", which are consequently named with spark-VERSION-bin-without-hadoop.tgz

If you would really like to build it, then run this from the project root

./build/mvn -Phadoop-provided -DskipTests clean package

Upvotes: 3

Related Questions