CODEWITHSUNDEEP
pythonpython-3.xloggingmultiprocessingconcurrent.futures
zephyr
zephyr

Reputation: 1900

Logging in main process when using concurrent futures ProcessPoolExecutor

Any ideas on how can I get forked processes to log to main process ?

My setup is like this:

from itertools import repeat
from concurrent.futures import ProcessPoolExecutor

logger = logging.getLogger(__name__)

def worker_process(i, logger):
    logger.info(f"{i}")

with ProcessPoolExecutor(max_workers=4) as executor:
    logger.info("Starting")
    executor.map(worker_process, range(5), repeat(logger))

Upvotes: 1

Views: 5242

Answers (1)

kalatabe
kalatabe

Reputation: 2989

Your worker processes do log to the main thread. You're not seeing any log output because the default log level for your logger is higher than INFO.

Set the logging to INFO and you'll see output:

from itertools import repeat
from concurrent.futures import ProcessPoolExecutor
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)


def worker_process(i, logger):
    logger.info(f"{i}")


with ProcessPoolExecutor(max_workers=4) as executor:
    executor.map(worker_process, range(5), repeat(logger))

Result:

INFO:__main__:0
INFO:__main__:4
INFO:__main__:3
INFO:__main__:1
INFO:__main__:2

Upvotes: 2

Related Questions