CODEWITHSUNDEEP
pythonjupyter-notebookjupyter
Nope
Nope

Reputation: 353

Use Jupyter notebooks as function calls

How to invoke a *.ipynb file as if it was a generic python function accepting an input and producing an output?

For example, graph_parser.ipynb is a Jupyter notebook, it needs as input some file path (e.g. .\location\big_graph.txt) and produces as output some pyhton object. Immagine a generic python script, for example running on a web-server, calling the notebook with notebook_result = run_notebook("local/path/to/graph_parser.ipynb", ".\location\big_graph.txt") and getting the pyhton object produced by the notebook assigned to notebook_result.

How would you go about achieving this? How to invoke a *.ipynb file, passing in some arguments and getting the output of the computation out? I know the tool nbconvert provides a python API to execute a *.ipynb file programmatically so I suppose one way would be to hardcode in the *.ipynb file an input and an output folder but I am looking for a more flexible apporach.

Motivation

I would like to use Jupyter Notebooks as function calls because they offer a fast way of visualising and improving processing steps in complex pipelines. In a project I was involve we ended duplicating the code of a notebook for converting an edge list into a complex python object. We needed the notenook because it was useful for experimenting but we also needed a standalone python function to parse graphs server-side.

Upvotes: 0

Views: 360

Answers (1)

Wayne
Wayne

Reputation: 9790

You seem to be seeking Papermill. It has a lot of ways to run and is flexible. You can program inside the notebook you'll call to make the Python object you want, such as a pickled set of data, etc.. See this article with a similar title as your post.

Related to this you could get most of the way using jupytext to run the notebook code. Jupytext let's you convert to python if you want and then you could make it ipython by changing the extension to .ipy if you have magic. It can also execute notebooks and depending on how you do things, you can inject things into them before you run them or make templates.

Getting more complex than papermill, there's ploomber.

Upvotes: 1

Related Questions