CODEWITHSUNDEEP
pythonjupyter-notebook
Pablito
Pablito

Reputation: 97

Why does Jupyter Notebook suddenly slow down so much?

I'm processing customer and purchase data on Jupyter Notebook. I was comfortably writing and executing codes on it, but all of a sudden, it has slowed down and got to take forever to execute even one simple code like print('A'). The worst thing is that it's not showing me any error, so I have absolutely no idea about what's wrong with Jupyter Notebook or my codes.

The original data is kinda big. I merged two data sets which have 424,699 rows and 22 columns, and 4,308,392 rows and 39 columns in total respectively. 

The versions:
Python → 3.7.4
Jupyter Notebook → 6.0.0
windows 10 pro

I just want to boost the speed of execution on Jupyter Notebook.

Upvotes: 4

Views: 20146

Answers (1)

PV8
PV8

Reputation: 6260

Probably your memory use gets quite high, and then the jupyter notebook slows down, as it goes on your hard disk then. The risk is also that it can crash soon.

Try to get clean all the data you do not need anymore. If you do not need a dataset after the merge, delete it. How to delete multiple pandas (python) dataframes from memory to save RAM?

a, b, c = pd.DataFrame(), pd.DataFrame(), pd.DataFrame()
lst = [a, b, c]
del a, b, c # dfs still in list
del lst     # memory release now

In this thread you can get an idea how to track your memory and cpu use in python: How to get current CPU and RAM usage in Python?

#!/usr/bin/env python
import psutil
# gives a single float value
psutil.cpu_percent()
# gives an object with many fields
psutil.virtual_memory()
# you can convert that object to a dictionary 
dict(psutil.virtual_memory()._asdict())

Here is also an overview how much the different datatypes using of your memory depending on your system: In-memory size of a Python structure

Upvotes: 6

Related Questions