CODEWITHSUNDEEP
pythonjupyter-notebookjupyterpapermill
NikPapadopoulos
NikPapadopoulos

Reputation: 41

Kernel dies while executing papermill

I am trying to automatically generate several notebooks with papermill using the following code:

template = "preprocessing_template.ipynb"
parameters = {"allowed_values":[0,1], "epsilon":0.01, "to_csv":True}
kernel_name = "my_env"
grid_folders_csv_names_outputs = [("dir_a", "a.csv","output_a.ipynb"),\
                                  ("dir_b", "b.csv","output_b.ipynb"),\
                                  ("dir_c", "c.csv","output_c.ipynb"),
                                 [

def execute_notebook(template,output_file,parameters,kernel_name,grid_folder,csv_name):

    parameters["grid_folder"]=grid_folder
    parameters["csv_name"]=csv_name
    nb = pm.execute_notebook(template,
                        output_file,
                        parameters=parameters,
                        kernel_name=kernel_name)
    #nbs = []
    #nbs.append(nb)
    return nb

for grid_folder, csv_name, output_file in tqdm(grid_folders_csv_names_outputs):
    execute_notebook(template,output_file,parameters,kernel_name,grid_folder,csv_name)

Nevertheless, I get an error message

DeadKernelError: Kernel died

which seems to happen because the memory is completely flooded as I have checked by monitoring the system while running the code. Apparently the code snippet above generates several processes to run the parameterized notebooks, and these processes are not killed once an iteration is complete, thus accumulating in memory. What could I do to solve this issue? I am using Ubuntu 18.04 and the code is run in an anaconda 2019.10 environment.

Upvotes: 4

Views: 1490

Answers (0)

Related Questions