CODEWITHSUNDEEP
python-3.xloggingjupyter-notebook
N8_Coder
N8_Coder

Reputation: 803

Show logging output in terminal when calling a notebook

I want to execute a notebook from a python bash script and show all logging outputs in the terminal. The problem is that only the output of the main.sh script is shown and not from the notebook foo.ipynb.

I already tried different settings for the logger from the internet, but did not get it to work. Any ideas how to set up the logger config correctly?

Here is an example:

main.sh:

#!/usr/bin/python3

import subprocess
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

logger.info("Calling Notebook...")
subprocess.check_call("jupyter nbconvert --to notebook --execute --inplace foo.ipynb",  shell=True)
logger.info("Finished")

and foo.ipynb

import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

logger.info('Start ...')
a = 3
logger.info('End...')

I cannot find a correct configuration for the logger to see the logging statements from the notebook in my terminal.

Output after running ./main.sh

INFO:__main__:Calling Notebook...
[NbConvertApp] WARNING | Config option `template_path` not recognized by `NotebookExporter`.
[NbConvertApp] Converting notebook foo.ipynb to notebook
[NbConvertApp] Executing notebook with kernel: python3
[NbConvertApp] Writing 983 bytes to foo.ipynb
INFO:__main__:Finished

Upvotes: 3

Views: 2414

Answers (2)

Hamanth Gousik
Hamanth Gousik

Reputation: 11

When subprocess is called a separate shell is used to execute the script foo.py, thus the logging is done on that shell. If you wish to have all the logging information in a same file use a logger file to get the logging data.

logging.basicConfig(filename="logging.log", level=logging.DEBUG, format='%(asctime)s - %(relativeCreated)6d - %(threadName)s - %(message)s ', filemode= 'a')

Upvotes: 1

Dillon Davis
Dillon Davis

Reputation: 7740

The issue appears to be that the jupyter process is capturing the stderr stream rather than redirecting it to the terminal. I would try adding the switch --stderr first:

subprocess.check_call("jupyter nbconvert --to notebook --stderr --execute --inplace foo.ipynb",  shell=True)

and if that fails, try adding --stdout as well or in place of --stderr. Lastly, I've read in a few places that you need to use --to html in place of other options in order to receive stdout/stderr messages. So you may need to try:

subprocess.check_call("jupyter nbconvert --to html --stdout --stderr --execute --inplace foo.ipynb",  shell=True)

Or possibly without the --stderr if it complains that it is not a valid option (can't test it first-hand, can't find any official documentation on it, I just stumbled upon a couple of snippets using it).

Hopefully one of these will force jupyter to redirect your stderr stream to the terminal.

Upvotes: 2

Related Questions