CODEWITHSUNDEEP
javaapache-spark
salman hussain
salman hussain

Reputation: 51

apache spark implementation

I tried alot to find the best solution to implement apache spark in my web application in java using struts2 framework.

I have few machine to make it use for driver and workers.

I have a web application that do hell lots of DB operation and I want it to be done by spark using hdfs.

I am able to run spark on a single standalone java project on local but I want to use it as cluster.

I read so many threads and information available on web but I am not able to find the way to implement it on cluster so that whatever huge processing that is required by my application will be done by spark.

I am not in a state to use the paid services like cloudera or amazon service...

Upvotes: 1

Views: 128

Answers (1)

tsar2512
tsar2512

Reputation: 2994

Here is a step-by-step guide regarding installation of Spark in masters and slave. It is pretty comprehensive:

http://data-flair.training/blogs/install-deploy-run-spark-2-x-multi-node-cluster-step-by-step-guide/

If successful, you should be able to see a spark-ui on the master which will show the entire cluster. Admittedly the setup is a lengthy process, I myself have gone through the process, so if you have any specific questions feel free to ask.

Use the spark-submit script to launch your applications once done:

./bin/spark-submit \
  --class <main-class> \
  --master <master-url> \
  --deploy-mode <deploy-mode> \
  --conf <key>=<value> \
  ... # other options
  <application-jar> \
  [application-arguments]

Make sure to give the masters url to run your application in the cluster mode instead of local[*] etc.

Upvotes: 1

Related Questions