CODEWITHSUNDEEP
apache-sparkspark-submit
learning_dev
learning_dev

Reputation: 145

Spark job not using the worker nodes on the cluster

I have set up spark on a cluster of 3 nodes, one is my namenode-master (named h1) and other two are my datanode-workers (named h2 and h3). When I give the command to run a spark job on my master, it seems like the job is not getting distributed to the workers and it is just being done on the master. The command I gave to run the spark job is 

bin/spark-submit --class org.dataalgorithms.chap07.spark.FindAssociationRules /home/ubuntu/project_spark/data-algorithms-1.0.0.jar ./in/xaa

The reason why I think its just running on the master is because when I go on the Spark Application GUI I just see the master h1 in the executor list. I would think I should see h2 and h3 my worker nodes too here? SparkUI

Correct me if I am wrong. I am a newbie so please excuse me for my ignorance.

Upvotes: 3

Views: 3311

Answers (2)

learning_dev
learning_dev

Reputation: 145

Thank you for all the help and suggestions. I tried many of them but ended up with some or the other error. What helped me is specifying the --master spark://IP:PORT with my regular command. So my new execution command looked like this

bin/spark-submit --class org.dataalgorithms.chap07.spark.FindAssociationRules --master spark://IP:PORT /home/ubuntu/project_spark/data-algorithms-1.0.0.jar ./in/xaa

This started my spark job in a truly distributed cluster mode

Upvotes: 0

wandermonk
wandermonk

Reputation: 7386

You haven't specified the mode you are deploying your job. You need to specify the --deploy-mode to deploy the job to the cluster and also you need to specify the --master which can be YARN/Mesos.

Also, when you specify YARN you need to make sure the resources you are using like executor-memory, executor-cores and num-executors are managed by cluster manager i.e YARN. YARN provides you different schedulers to allocate resources. So, you need to check the type of scheduler you have configured.

Read about schedulers here

https://blog.cloudera.com/blog/2016/01/untangling-apache-hadoop-yarn-part-3/

spark-submit --num-executors 50 --executor-memory 4G --executor-cores 4  --master yarn --deploy-mode cluster

Upvotes: 2

Related Questions