CODEWITHSUNDEEP
python-2.7jenkinsluigi
sakurashinken
sakurashinken

Reputation: 4080

How to log from Python Luigi

I am trying to build a strategy to log from Luigi in such a way that there is a configurable list of outputs including stdout and a custom list of files. I would like to be able to set the logging level at runtime. Our system uses Luigi to call spark from Jenkins. Thank you in advance.

Upvotes: 10

Views: 9411

Answers (2)

Evhz
Evhz

Reputation: 9248

Inside any of the Task class methods, you can do:

class Agg(luigi.Task):
  _date = luigi.DateParameter()

  def output(self):
    return luigi.LocalTarget("file_%.txt" % self._date)

  def run(self):
    # Use the luigi-interface to log to console
    logger = logging.getLogger('luigi-interface')
    logger.info("Running --> Agg.Task")

Upvotes: 15

mfcabrera
mfcabrera

Reputation: 781

Have you checked the logging_conf_file parameter of the configuration? you can set up there all your configuration regarding logging using Python's standard logging mechanism.

For some examples see:

Upvotes: 0

Related Questions