CODEWITHSUNDEEP
hadoopapache-sparkhadoop-yarn
user3648294
user3648294

Reputation:

Spark not able to run in yarn cluster mode

I am trying to execute my code on a yarn cluster

The command which I am using is 

$SPARK_HOME/bin/spark-submit \
    --class "MyApp" \
    target/scala-2.10/my-application_2.10-1.0.jar \
    --master yarn-cluster \
    --num-executors 3 \
    --driver-memory 6g \
    --executor-memory 7g \
    <outputPath>

But, I can see that this program is running only on the localhost.

Its able to read the file from hdfs.

I have tried this in standalone mode and it works fine.

Please suggest where is it going wrong.

Upvotes: 5

Views: 1238

Answers (1)

user3648294
user3648294

Reputation:

I am using Hadoop2.4 with Spark 1.1.0 . I was able to get it running in the cluster mode.

To solve it we simply removed all the configuration files from all the slave nodes. Earlier we were running in the standalone mode and that lead to duplicating the configuration on all the slaves. Once that was done it ran as expected in cluster mode. Although performance is not up to the standalone mode.

Thanks.

Upvotes: 2

Related Questions