CODEWITHSUNDEEP
apache-spark
Suraj Karale
Suraj Karale

Reputation: 21

I want to run spark shell in client mode?

Spark context available as 'sc' (master = yarn, app id = application_1519491124804_0002).

I need master = yarn-client

error:

Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". 
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 
18/02/24 22:27:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
18/02/24 22:27:29 WARN Utils: Your hostname, suraj resolves to a loopback address: 127.0.1.1; using 192.168.43.193 instead (on interface wlan0) 
18/02/24 22:27:29 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 
18/02/24 22:27:32 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. 
Spark context Web UI available at http://192.168.43.193:4040 Spark context available as 'sc' (master
= yarn, app id = application_1519491124804_0002). 
Spark session available as 'spark'. 
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/    
   /___/ .__/\_,_/_/ /_/\_\   version 2.2.1
      /_/
          Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_161) Type in expressions to have them evaluated. Type :help for more information.

Upvotes: 0

Views: 1951

Answers (1)

OneCricketeer
OneCricketeer

Reputation: 191701

I need master = yarn-client

In Spark 2.x master = yarn-client is deprecated.

spark-shell --master yarn --deploy-mode client is the correct way to run the shell

The default deploy-mode is client

Upvotes: 2

Related Questions