CODEWITHSUNDEEP
scalaapache-sparkcassandra-2.1webui
thomas poidevin
thomas poidevin

Reputation: 185

Running Applications doesn t appear spark web Ui but runs

i need your help, i created 2 apps (one which using spray framework and the other one receive messages from kafka and send it to cassandra). Both run all the time and should never stop. I m in standalone on the server and my conf is : 

- In spark_env.sh :

SPARK_MASTER_IP=MYIP
SPARK_EXECUTOR_CORES=2
SPARK_MASTER_PORT=7077
SPARK_EXECUTOR_MEMORY=4g
#SPARK_WORKER_PORT=65000
MASTER=spark://${SPARK_MASTER_IP}:${SPARK_MASTER_PORT}
SPARK_LOCAL_IP=MYIP
SPARK_MASTER_WEBUI_PORT=8080

- In spark_env.sh :
spark.master                     spark://MYIPMASTER:7077
spark.eventLog.enabled           true
spark.eventLog.dir               /opt/spark-1.6.1-bin-hadoop2.6/spark-events
spark.history.fs.logDirectory    /opt/spark-1.6.1-bin-hadoop2.6/logs
spark.io.compression.codec       lzf
spark.cassandra.connection.host MYIPMASTER
spark.cassandra.auth.username   LOGIN
spark.cassandra.auth.password   PASSWORD

I can access on both pages : MYIP:8080/ and MYIP:4040/ But on http://MYIP:8080/, i see only my workers , i can t see my application which running.

When i submit i use this : 

/opt/spark-1.6.1-bin-hadoop2.6/bin/spark-submit --class MYCLASS --verbose --conf spark.eventLog.enable=true --conf spark.master.ui.port=8080 --master local[2] /opt/spark-1.6.1-bin-hadoop2.6/jars/MYJAR.jar

Why ? Could you help me?

Thanks a lot :)

Upvotes: 1

Views: 4178

Answers (3)

Deepak Motlani
Deepak Motlani

Reputation: 1

Mode: Standalone cluster

1> bin/spark-submit --class com.deepak.spark.App ../spark-0.0.2-SNAPSHOT.jar --master spark://172.29.44.63:7077, was not working because master was specified after the jar

2> bin/spark-submit --class com.deepak.spark.App --master spark://172.29.44.63:7077 ../spark-0.0.2-SNAPSHOT.jar, this worked

Upvotes: 0

Zhong Dai
Zhong Dai

Reputation: 514

In terms of the master, spark-submit will respect the setting by following orders,

  1. The master URL in your application code, which is the SparkSession.builder().master("...")
  2. The --master parameter for the spark-submit command
  3. The default configuration in your spark-defaults.conf

Upvotes: 0

Hokam
Hokam

Reputation: 924

In your spark-submit command you are using the --master as local[2] which is submitting the application in local mode. If you wants to run it on the standalone cluster that you are running then you should pass spark master URL in master option i.e. --master spark://MYIPMASTER:7077

Upvotes: 4

Related Questions