CODEWITHSUNDEEP
apache-sparkpyspark
ERJAN
ERJAN

Reputation: 24500

how to make pyspark run smoothly without these warnings?

I installed spark and made pyspark work by checking "bin\pyspark" in anaconda prompt.

I do have winutils.exe installed.

However it gives me this warning:

(base) C:\Users\Admin\Desktop\spark\spark-2.4.6-bin-hadoop2.6>bin\pyspark
Python 3.7.6 (default, Jan  8 2020, 20:23:39) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/C:/Users/Admin/Desktop/spark/spark-2.4.6-bin-hadoop2.6/jars/spark-unsafe_2.11-2.4.6.jar) to method java.nio.Bits.unaligned()
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
20/06/16 03:48:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.4.6
      /_/

Using Python version 3.7.6 (default, Jan  8 2020 20:23:39)
SparkSession available as 'spark'.

How to get rid of this warnings?

Upvotes: 4

Views: 1170

Answers (1)

Wiki_91
Wiki_91

Reputation: 21

Goto this file "org/apache/spark/log4j-defaults.properties" and change/add below property,

log4j.rootCategory=ERROR, console

Upvotes: 1

Related Questions