CODEWITHSUNDEEP
pythonapache-sparkloggingpysparkpytest
user4046073
user4046073

Reputation: 871

Unable to use logging in PySpark

I am trying to find out how to setup logging in pyspark for pytest. However both methods below has an error TypeError: 'JavaPackage' object is not callable. I'm using python 3.7 and pyspark <2.4.0 in pycharm. Could you please help? Many thanks.

    logger = logging.getlogger('py4j')
    logger.setlevel(logger.WARN)

    log4jLogger = spark.sparkContext._jvm.org.apache.log4j
    log = log4jLogger.logManager.getLogger(__name__)

Upvotes: 1

Views: 1562

Answers (1)

pltc
pltc

Reputation: 6082

Looks like the way you're referring to Java object is incorrect, try this instead

log4jLogger = spark.sparkContext._jvm.org.apache.log4j
logger = log4jLogger.LogManager.getLogger("MyLogger")
logger.setLevel(log4jLogger.Level.DEBUG)

Upvotes: 4

Related Questions