CODEWITHSUNDEEP
luigi
simon
simon

Reputation: 2821

How do I change the logging level in luigi on windows?

I have created a luigi.cfg in my execution directory; and set the LUIGI_CONFIG_PATH variable to point to a copy of this; and set my own logging configuration file to INFO. Yet I still get all the DEBUG messages.

The config file is:

[core]
log_level=INFO
no_configure_logging=True

Upvotes: 2

Views: 1232

Answers (4)

Deke Eusey
Deke Eusey

Reputation: 1

This worked for me with luigi 3.0.2, to exclude luigi DEBUG messages, and include luigi INFO and higher severity messages, in my log:

import os
import logging
import luigi

# Initialize logging.  This does NOT set the luigi log level:
logging.basicConfig(
    filename=os.path.join(os.path.dirname(__file__), 'My_Example.log'), 
    format='%(asctime)s %(levelname)s | %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S',
    filemode='w',
    level=logging.INFO)

# SET THE luigi LOG LEVEL: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
luigi.interface.InterfaceLogging.setup(type('opts',
                                            (),
                                            {   'background': None,
                                                'logdir': None,
                                                'logging_conf_file': None,
                                                'log_level': 'INFO' # <<<<<<<<<<
                                            }))

""" Report the luigi logging level by writing to the log file
    that is specified above via logging.basicConfig() """
luigi_interface_log_level = logging.getLogger('luigi-interface').level
logging.info(f"logging.getLogger('luigi-interface').level"
             f" = {luigi_interface_log_level} = "
             f"logging.{logging._levelToName[luigi_interface_log_level]}")

The output of the above, in file My_Example.log, is:

2023-10-13 11:22:33 INFO | logging configured by default settings
2023-10-13 11:22:33 INFO | logging.getLogger('luigi-interface').level = 20 = logging.INFO

The message 'logging configured by default settings', which is logged during execution of luigi.interface.InterfaceLogging.setup(), is OK. The luigi log level specification (by "'log_level': 'INFO'" in the above case) is effective for subsequent logging by luigi - so luigi DEBUG messages are not logged. 'logging configured by default settings' can be excluded from the log by moving the call to logging.basicConfig() to after the call to luigi.interface.InterfaceLogging.setup()

Upvotes: 0

Deke Eusey
Deke Eusey

Reputation: 1

This worked for me with luigi 3.0.2, to exclude luigi DEBUG messages, and include luigi INFO and higher severity messages, in my log:

import os
import logging
import luigi

# Initialize logging:
logging.basicConfig(
    filename=os.path.join(os.path.dirname(__file__), 'My_Example.log'), 
    format='%(asctime)s %(levelname)s | %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S',
    filemode='w',
    level=logging.INFO)

# Set the luigi log level: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
luigi.interface.InterfaceLogging.setup(type('opts',
                                            (),
                                            {   'background': None,
                                                'logdir': None,
                                                'logging_conf_file': None,
                                                'log_level': 'INFO' # <<<<<<<<<<
                                            }))

""" Report the luigi logging level
    by writing to the log file specified above via logging.basicConfig() """
luigi_interface_log_level = logging.getLogger('luigi-interface').level
logging.info(f"logging.getLogger('luigi-interface').level"
             f" = {luigi_interface_log_level} = "
             f"logging.{logging._levelToName[luigi_interface_log_level]}")

The output of the above, in file My_Example.log, is:

2023-10-13 11:22:33 INFO | logging configured by default settings
2023-10-13 11:22:33 INFO | logging.getLogger('luigi-interface').level = 20 = logging.INFO

The message 'logging configured by default settings', which is logged during execution of luigi.interface.InterfaceLogging.setup(), is OK. The log level specification (by "'log_level': 'INFO'" in the above case) is effective. Logging this message can be prevented by placing the call to logging.basicConfig() after the call to luigi.interface.InterfaceLogging.setup()

Upvotes: 0

Gabriel Magalhaes
Gabriel Magalhaes

Reputation: 56

You can add to your code:

luigi.interface.InterfaceLogging.setup(luigi.interface.core())

Luigi initializes its logging configuration (accessing luigi.cfg file) the first time you execute it. So, if you are adding a different configuration, it might get overridden when luigi executes. This command will initialize luigi's interface logging beforehand to avoid that.

Upvotes: 0

simon
simon

Reputation: 2821

Seems luigi sets its own logging anyway. This turns it off.

luigi.interface.setup_interface_logging.has_run = True

Upvotes: 1

Related Questions