CODEWITHSUNDEEP
pythonapache-sparkpyspark
user13485171
user13485171

Reputation: 63

How to set pyspark logging level to debug?

Can anyone help me with the spark configuration needed to set logging level to debug and capture more logs

Upvotes: 5

Views: 7648

Answers (1)

notNull
notNull

Reputation: 31490

Set setLogLevel property to DEBUG in sparksession.

from pyspark.sql import SparkSession
spark = SparkSession.builder.master('local').appName('test').getOrCreate()
spark.sparkContext.setLogLevel("DEBUG")

#or

log4j = sc._jvm.org.apache.log4j
log4j.LogManager.getRootLogger().setLevel(log4j.Level.DEBUG)

#for custom log
log4j.LogManager.getRootLogger().info("custom log")
#2020-05-06 13:43:42 INFO  root:? - custom log

Upvotes: 8

Related Questions