How do I convert a IPython Notebook into a Python file via commandline?

Question

I'm looking at using the *.ipynb files as the source of truth and programmatically 'compiling' them into .py files for scheduled jobs/tasks.

The only way I understand to do this is via the GUI. Is there a way to do it via command line?

Answer 1

You can convert all Notebook files to python files by doing this:

pip install jupytext
jupytext --to py *.ipynb 

Answer 2

The following answer worked for me (based on Yaach's answer from October 25) but mine needed to convert several files at the same time:

1. Launch Anaconda prompt
2. CD to your directory which has several ipnyb files, for example, C:\myfolder
3. Run: jupyter nbconvert coding-challenge-*.ipynb --to python

Note the wild card character. The result was several coding_challeng-*.py files in the same folder.

Answer 3

Common unflexible way

Most would suggest the nbconvert method, but the output is also kinda ugly containing tons of comments:

jupyter nbconvert --to script [YOUR_NOTEBOOK].ipynb

Cell level extractions

As notebooks are basically JSON, then you can write your own module extracting specific cells of your notebooks using tags like "###EXTRACT###" as the first line of the cells you want to extract.

Here is how to use the module I wrote:

# Importing Custom Module
from ipynb_exporter import NotebookModuleBuilder as nmb

# Exporting labeled cells
nmb().ipynb_to_file(ipynb_path="mynotebook.ipynb",
                    label="###EXTRACT###",
                    py_path="mynotebookcells.py")

Here is the module:

import json

class NotebookModuleBuilder():
    """ Class helps you extract code cells from ipynb-files by using tags """
    
    @staticmethod
    def _read_json(path: str) -> dict:
        """ Reads a json-file and returns a dictionary
            
        Args:
            path: Path to jupyter notebook (.ipynb)
            
        Returns:
            dictionary representation of notebook
        """
        file = open(path, mode= "r", encoding= "utf-8")
        myfile = file.read()
        myjson = json.loads(myfile)
        file.close()
        return myjson

    @staticmethod
    def _get_code_cells(dictionary: dict) -> list:
        """ Finds cells of ipynb with code
        
        Args:
            dictionary: Dictionary from importing a ipynb notebook
            
        Returns:
            List of code cells
        """
        code_cells = [cell for cell in dictionary['cells'] if cell['cell_type']=='code']
        return code_cells

    @staticmethod
    def _get_labeled_cells(code_cells: dict, label="###EXPORT###") -> dict:
        """ Gets cells with the specified label
        
        Args:
            code_cells: Dictionary with code cells
            
        Returns:
            Dictionary with labeled cells
        """
        label = label + "\n"
        sourced_cells = [cell for cell in code_cells if len(cell['source']) > 0]
        labeled_cells = [cell['source'] for cell in sourced_cells if cell['source'][0]==label]
        return labeled_cells

    @staticmethod
    def _write_to_file(labeled_cells: dict, output_file: str) -> None:
        """ Writes the labeled cells to a file
        
        Args:
            labeled_cells: Dictionary with cells that should be written to a file
        """
        flattened_lists = '\n\n'.join([''.join(labeled_cell[1:]) for labeled_cell in labeled_cells])
        file = open(output_file, 'w')
        file.write(flattened_lists)
        file.close()
    
    def ipynb_to_file(self, ipynb_path: str, py_path: str, label: str = '###EXTRACT###') -> None:
        """ Writes cells labeled with ###EXTRACT### in ipynb into a py-file
        
        Args:
            label: Lable that in first line of a cell to match
            ipynb_path: Input path to ipynb-notebook
            py_path: Output path to py-file
        """
        json_file = self._read_json(ipynb_path)
        code_cells = self._get_code_cells(json_file)
        labeled_cells = self._get_labeled_cells(code_cells,label)
        self._write_to_file(labeled_cells, py_path)

Answer 4

jupyter nbconvert main.ipynb --to python

Answer 5

I have built a function to achieve this. The user does not have to install anything to use it.

#!/usr/bin/python


# A short routine to convert a Jupyter Notebook to a Python file

import json

def ipynb_to_py(input_ipynb_file,output_py_file=None):
    """
    Generate a Python script (.py) that includes all source code from the input Jupyter notebook (.ipynb).
    
    The user can input a Jupyter Notebook file from the current working directory or from a path.
    
    If the name for output Python file is not specified, 
    the output file name will copy the file name of the input Jupyter Notebook, 
    but the file exention will be changed from ".ipynb" chanegd to ".py".
    And the output Python file will be saved at the same directory of the input Jupyter Notebook.
    For example:
    ipynb_to_py("test-jupyternotebook.ipynb")
    ipynb_to_py("./test-input-dir/test-jupyternotebook.ipynb")
    
    The user can also specify an output file name that ends with ".py".
    If the output file name is provided, but no path to output file is added, 
    the file will be saved at the current working directory.
    For example:
    ipynb_to_py("test-jupyternotebook.ipynb","test1.py")
    ipynb_to_py("./test-input-dir/test-jupyternotebook.ipynb","test2.py")
        
    The user can save out the file at a target directory by adding a path to the output file.
    For example: 
    ipynb_to_py("test-jupyternotebook.ipynb","./test-outputdir/test3.py")
    ipynb_to_py("./test-input-dir/test-jupyternotebook.ipynb","./test-output-dir/test4.py")
    
    This function does not edit or delete the original input Jupyter Notebook file.
    
    Args:
    -----
        input_ipynb_file: The file name string for the Jupyter Notebook (ends with ".ipynb")
        output_py_file (optional): The file name for Python file to be created (ends with ".py"). 
    
    Returns:
    --------
        A Python file containing all source code in the Jupyter Notebook.
        
    Example usages:
    ---------------
        ipynb_to_py("test-jupyternotebook.ipynb")
        ipynb_to_py("./test-input-dir/test-jupyternotebook.ipynb")
        ipynb_to_py("test-jupyternotebook.ipynb","test1.py")
        ipynb_to_py("test-jupyternotebook.ipynb","./test-outputdir/test2.py")
        ipynb_to_py("test-jupyternotebook.ipynb","./test-outputdir/test3.py")
        ipynb_to_py("./test-input-dir/test-jupyternotebook.ipynb","./test-output-dir/test4.py")
             
    """
    # Check if the input file is a Jupyter Notebook
    if input_ipynb_file.endswith(".ipynb"):
        
        # Open the input Jupyter Notebook file
        notebook = open(input_ipynb_file)
        
        # Read its content in the json format
        notebook_content = json.load(notebook)

        # Only extract the source code snippet from each cell in the input Jupyter Notebook
        source_code_snippets = [cell['source'] for cell in notebook_content['cells']]
        
        # If the name for output Python file is not specified,
        # The name of input Jupyter Notebook will be used after changing ".ipynb" to ".py".
        if output_py_file == None:
            output_py_file = input_ipynb_file.split('.ipynb')[0]+".py"
        else:
            pass

        # Create a Python script to save out all the extracted source code snippets
        output_file = open(output_py_file,'w')

        # Print out each line in each source code snippet to the output file
        for snippet in source_code_snippets:
            for line in snippet:
                # Use end='' to avoid creating unwanted gaps between lines
                print(line,end = '',file = output_file)
            # At end of each snippet, move to the next line before printing the next one
            print('',sep = '\n',file=output_file)

        # Close the output file
        output_file.close()
        print("The path to output file:",output_py_file)
        
    else:
        print("The input file must be a Jupyter Notebook (in .ipynb format)!")
        
def main():
    pass

if __name__ == "__main__":
    main()

Answer 6

I found that there are 2 ways to convert Jupyter Notebook to plain Python script with command line. Below is the example Jupyter notebook and output from both tools.

1. Use nbconvert

The nbconvert is tool that is used in Jupyter Notebook User Interface in Download as functionality. It can be used as command line tool:

jupyter nbconvert --to python notebook.ipynb

Example Python script:

2. Use jupytext

The jupytext is a package to keep .ipynb file synchronized with .py file. It can be also used to convert .ipynb files in command line. It supports several types of conversion:

Convert to Python script with light format

jupytext --to py notebook.ipynb             

Example python script:

Convert to Python script with percent format

jupytext --to py:percent notebook.ipynb 

Example Python script:

Answer 7

The given solution works only for converting a single .py file. Here is a solution that converts all .py files in a directory and sub-directories.

First, you need to install the tool that convert only one file at a time like ipynb-py-convert

pip install ipynb-py-convert

Then cd into the place of your folder having the .py files and directories. Then we run the tool recursively on all files in the directory and subdirectories:

In powershell:

foreach ($f in Get-ChildItem "." -Filter *.ipynb -Recurse){ ipynb-py-convert $f.FullName "$($f.FullName.Substring(0,$f.FullName.Length-6)).py"}

Now if you would like to convert the otherway around from .ipynb to .py also with batch convert, you can run:

foreach ($f in Get-ChildItem "." -Filter *.py -Recurse){ ipynb-py-convert $f.FullName "$($f.FullName.Substring(0,$f.FullName.Length-3)).ipynb"}

This helped me a lot while exploring .py files. I make a copy of the project, run this code and quickly in Jupiter test different parts of the code as cells and so on. I hope it helps more people.

Answer 8

no file/directory error

On my mint [ubuntu] system at work, even though jupyter was already installed and notebooks worked, jupyter nbconvert --to script gave the error no file/directory until I did a separate

sudo apt-get install jupyter-nbconvert

Then all was fine with the conversion. I just wanted to add this in case anyone hits the same error (for me it was confusing as I thought the no file error referred to the notebook, which was definitely there in the local directory, took me a while to realize the subcommand was not installed).

Answer 9

Here is a jq solution that could be situationally useful. Remember, notebooks are just json.

jq -r '.cells[] | select(.cell_type  == "code") | .source[] | rtrimstr("\n")' $filename

Answer 10

Using nbconvert 6.07 and jupyter client 6.1.12:

Convert jupyter notebook to python script

$ jupyter nbconvert mynotebook.ipynb --to python

Convert jupyter notebook to python script specifying output filename

$ jupyter nbconvert mynotebook.ipnb --to python --output myscript.py

Answer 11

The %notebook foo.ipynb magic command will export the current IPython to "foo.ipynb".

More info by typing %notebook?

Answer 12

The following example turns an Iron Python Notebook called a_notebook.ipynb into a python script called a_python_script.py leaving out the cells tagged with the keyword remove, which I add manually to the cells that I don't want to end up in the script, leaving out visualizations and other steps that once I am done with the notebook I don't need to be executed by the script.

import nbformat as nbf
from nbconvert.exporters import PythonExporter
from nbconvert.preprocessors import TagRemovePreprocessor

with open("a_notebook.ipynb", 'r', encoding='utf-8') as f:
    the_notebook_nodes = nbf.read(f, as_version = 4)

trp = TagRemovePreprocessor()

trp.remove_cell_tags = ("remove",)

pexp = PythonExporter()

pexp.register_preprocessor(trp, enabled= True)

the_python_script, meta = pexp.from_notebook_node(the_notebook_nodes)

with open("a_python_script.py", 'w', encoding='utf-8') as f:
    f.writelines(the_python_script)

Answer 13

I understand this is an old thread. I have faced the same issue and wanted to convert the .pynb file to .py file via command line.

My search took me to ipynb-py-convert

By following below steps I was able to get .py file

1. Install "pip install ipynb-py-convert"
2. Go to the directory where the ipynb file is saved via command prompt
3. Enter the command

> ipynb-py-convert YourFileName.ipynb YourFilename.py

Eg:. ipynb-py-convert getting-started-with-kaggle-titanic-problem.ipynb getting-started-with-kaggle-titanic-problem.py

Above command will create a python script with the name "YourFileName.py" and as per our example it will create getting-started-with-kaggle-titanic-problem.py file

Answer 14

Jupytext is nice to have in your toolchain for such conversions. It allows not only conversion from a notebook to a script, but you can go back again from the script to notebook as well. And even have that notebook produced in executed form.

jupytext --to py notebook.ipynb                 # convert notebook.ipynb to a .py file
jupytext --to notebook notebook.py              # convert notebook.py to an .ipynb file with no outputs
jupytext --to notebook --execute notebook.py    # convert notebook.py to an .ipynb file and run it 

Answer 15

I had this problem and tried to find the solution online. Though I found some solutions, they still have some problems, e.g., the annoying Untitled.txt auto-creation when you start a new notebook from the dashboard.

So eventually I wrote my own solution:

import io
import os
import re
from nbconvert.exporters.script import ScriptExporter
from notebook.utils import to_api_path


def script_post_save(model, os_path, contents_manager, **kwargs):
    """Save a copy of notebook to the corresponding language source script.

    For example, when you save a `foo.ipynb` file, a corresponding `foo.py`
    python script will also be saved in the same directory.

    However, existing config files I found online (including the one written in
    the official documentation), will also create an `Untitile.txt` file when
    you create a new notebook, even if you have not pressed the "save" button.
    This is annoying because we usually will rename the notebook with a more
    meaningful name later, and now we have to rename the generated script file,
    too!

    Therefore we make a change here to filter out the newly created notebooks
    by checking their names. For a notebook which has not been given a name,
    i.e., its name is `Untitled.*`, the corresponding source script will not be
    saved. Note that the behavior also applies even if you manually save an
    "Untitled" notebook. The rationale is that we usually do not want to save
    scripts with the useless "Untitled" names.
    """
    # only process for notebooks
    if model["type"] != "notebook":
        return

    script_exporter = ScriptExporter(parent=contents_manager)
    base, __ = os.path.splitext(os_path)

    # do nothing if the notebook name ends with `Untitled[0-9]*`
    regex = re.compile(r"Untitled[0-9]*$")
    if regex.search(base):
        return

    script, resources = script_exporter.from_filename(os_path)
    script_fname = base + resources.get('output_extension', '.txt')

    log = contents_manager.log
    log.info("Saving script at /%s",
             to_api_path(script_fname, contents_manager.root_dir))

    with io.open(script_fname, "w", encoding="utf-8") as f:
        f.write(script)

c.FileContentsManager.post_save_hook = script_post_save

To use this script, you can add it to ~/.jupyter/jupyter_notebook_config.py :)

Note that you may need to restart the jupyter notebook / lab for it to work.

Answer 16

There's a very nice package called nb_dev which is designed for authoring Python packages in Jupyter Notebooks. Like nbconvert, it can turn a notebook into a .py file, but it is more flexible and powerful because it has a lot of nice additional authoring features to help you develop tests, documentation, and register packages on PyPI. It was developed by the fast.ai folks.

It has a bit of a learning curve, but the documentation is good and it is not difficult overall.

Answer 17

If you want to convert all *.ipynb files from current directory to python script, you can run the command like this:

jupyter nbconvert --to script *.ipynb

Answer 18

For converting all *.ipynb format files in current directory to python scripts recursively:

for i in *.ipynb **/*.ipynb; do 
    echo "$i"
    jupyter nbconvert  "$i" "$i"
done

Answer 19

If you don't want to output a Python script every time you save, or you don't want to restart the IPython kernel:

On the command line, you can use nbconvert:

$ jupyter nbconvert --to script [YOUR_NOTEBOOK].ipynb

As a bit of a hack, you can even call the above command in an IPython notebook by pre-pending ! (used for any command line argument). Inside a notebook:

!jupyter nbconvert --to script config_template.ipynb

Before --to script was added, the option was --to python or --to=python, but it was renamed in the move toward a language-agnostic notebook system.

Answer 20

Following the previous example but with the new nbformat lib version :

import nbformat
from nbconvert import PythonExporter

def convertNotebook(notebookPath, modulePath):

  with open(notebookPath) as fh:
    nb = nbformat.reads(fh.read(), nbformat.NO_CONVERT)

  exporter = PythonExporter()
  source, meta = exporter.from_notebook_node(nb)

  with open(modulePath, 'w+') as fh:
    fh.writelines(source.encode('utf-8'))

Answer 21

Here is a quick and dirty way to extract the code from V3 or V4 ipynb without using ipython. It does not check cell types, etc.

import sys,json

f = open(sys.argv[1], 'r') #input.ipynb
j = json.load(f)
of = open(sys.argv[2], 'w') #output.py
if j["nbformat"] >=4:
        for i,cell in enumerate(j["cells"]):
                of.write("#cell "+str(i)+"\n")
                for line in cell["source"]:
                        of.write(line)
                of.write('\n\n')
else:
        for i,cell in enumerate(j["worksheets"][0]["cells"]):
                of.write("#cell "+str(i)+"\n")
                for line in cell["input"]:
                        of.write(line)
                of.write('\n\n')

of.close()

Answer 22

You can do this from the IPython API.

from IPython.nbformat import current as nbformat
from IPython.nbconvert import PythonExporter

filepath = 'path/to/my_notebook.ipynb'
export_path = 'path/to/my_notebook.py'

with open(filepath) as fh:
    nb = nbformat.reads_json(fh.read())

exporter = PythonExporter()

# source is a tuple of python source code
# meta contains metadata
source, meta = exporter.from_notebook_node(nb)

with open(export_path, 'w+') as fh:
    fh.writelines(source)