CODEWITHSUNDEEP
pythonpandasnumpymemoryout-of-memory
Jacob Kim
Jacob Kim

Reputation: 31

How to solve memory error? Should I increase memory limit?

I was loading this but it says error.

import pandas as pd
import numpy as np

userMovie = np.load('userMovieMatrixAction.npy')

numberUsers, numberGenreMovies = userMovie.shape

genreFilename = 'Action.csv'
genre = pd.read_csv(genreFilename)

MemoryError: Unable to allocate 3.63 GiB for an array with shape (487495360,) and data type float64 What can I do? It's driving me crazy.

Upvotes: 1

Views: 646

Answers (2)

Paraskevas Paraskevas
Paraskevas Paraskevas

Reputation: 17

  1. First things first you don't need float64 you can save this as float32 using the casting pandas function you can find here https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.astype.html ..even if you were working on a neural network you would try to save as float32, you will reduce your byte footprint in this way.
  2. You may need to build a generator to allow the data to be loaded into memory when needed in chunks not entirely in memory as you are trying to do. links to Python guide https://docs.python.org/3/howto/functional.html#generators Kaggle notebook using generators https://www.kaggle.com/code/vbookshelf/python-generators-to-reduce-ram-usage-part-2/notebook

You are starting to build a recommender system as far I can see ....

Upvotes: 2

Cardstdani
Cardstdani

Reputation: 5223

If the program runs out of memory, it seems like an issue with overcommit handling of your operative system. If you are in Linux, you can try to run the following command to enable "always overcommit" mode, which can help you load the 3.63GiB npy file with numpy:

$ echo 1 > /proc/sys/vm/overcommit_memory

Upvotes: 2

Related Questions