CODEWITHSUNDEEP
pythonloggingray
Han Zheng
Han Zheng

Reputation: 319

How can I use the python logging in Ray?

I use the logging module in the main function/process, it works well, but it seems can't work in Actor process/subprocess. How to make it work? In the sample below code, logging.info work in the main process but failed in the worker process. Thanks.

import logging
import ray

@ray.remote
class Worker(object):
   ...

   def train(self):
       logging.info("fail print")


...

worker = Worker.remote()

ray.get(worker.train.remote())

logging.info("successful print")

Upvotes: 16

Views: 9500

Answers (3)

KG3
KG3

Reputation: 21

A little late to the game here, but to get around this I do something like the following:

import ray
import os
import importlib
import logging

@ray.remote()
def my_remote_func_w_custom_logger():

    logging.shutdown()
    importlib.reload(logging)

    pid = os.getpid()

    logging.basicConfig(format= f"%(asctime)s - %(levelname)s - %(funcName)s - PID {pid} - %(message)s", 
                        level= logging.INFO,
                        datefmt= '%m/%d/%Y %I:%M:%S %p',
                        filename= 'remote_func.log')
   
    logger = logging.getLogger(__name__)

    logger.info("This is the remote function.")

So essentially, reload the logging module after you import ray and then specify your logging config. I don't know if this is the best solution, but it seems to work.

I've run into this problem with other packages as well, not just ray, but I have not done enough research to understand why some packages have this behavior while others do not.

Upvotes: 2

crypdick
crypdick

Reputation: 19894

I've had trouble with Ray suppressing console output from loggers. To get around that, add a StreamHandler:

your_logger.addHandler(logging.StreamHandler())

Upvotes: 1

Robert Nishihara
Robert Nishihara

Reputation: 3362

There are a couple things to be careful about.

  • First, you should create a new logger inside of the worker because the worker runs on a different Python process. If you try to use a logger that you created outside of the worker within the worker, then Ray will try to pickle the logger and send it to the worker process, and Python loggers typically do not behave correctly when pickled and unpickled.
  • Second, you have to make sure the logging level is set correctly. I'm using logger.warning instead of logger.info because the Python logging level is set to `warning by default.

Here is a working example:

import logging
import ray

logger = logging.getLogger(__name__)

@ray.remote
class Worker(object):
    def __init__(self):
        self.logger = logging.getLogger(__name__)
    def train(self):
        self.logger.warning("print from inside worker")


ray.init()

worker = Worker.remote()

ray.get(worker.train.remote())

logger.warning("print from outside worker")

Upvotes: 19

Related Questions