CODEWITHSUNDEEP
pythonarraysnumpy
ramailo sathi
ramailo sathi

Reputation: 355

memory profiling: list comprehension vs. numpy array

I am trying to perform memory profiling of list vs numpy arrays.

%%file memory.py

import numpy as np

@profile
def allocate():
    vector_list = [float(i) for i in range(10000)]
    np.arange(0,10000,dtype='d')

allocate()

Running memory profiler in the shell:

!python -m memory_profiler memory.py

gives the following output:

Line #    Mem usage    Increment   Line Contents
================================================
     4   39.945 MiB    0.000 MiB   @profile
     5                             def allocate():
     6   39.949 MiB    **0.004 MiB**       vector_list = [float(i) for i in range(10000)]
     7   40.039 MiB    **0.090 MiB**       np.arange(0,10000,dtype='d')

Increment in memory of line 6 vs line 7 shows that numpy array was way more expensive than a list. What am I doing wrong?

Upvotes: 0

Views: 539

Answers (1)

AGN Gazer
AGN Gazer

Reputation: 8388

I do not know what is memory profiler reporting - I get very different numbers from you:

Line #    Mem usage    Increment   Line Contents
================================================
     3   41.477 MiB    0.000 MiB   @profile
     4                             def allocate():
     5   41.988 MiB    0.512 MiB       vector_list = [float(i) for i in range(10000)]
     6   41.996 MiB    0.008 MiB       np.arange(0,10000,dtype='d')

I would recommend the following two links for your reading: Python memory usage of numpy arrays and Size of list in memory

I have also modified your code as follows:

import numpy as np
import sys

@profile
def allocate():
    vector_list = [float(i) for i in range(10000)]
    npvect = np.arange(0,10000,dtype='d')
    listsz = sum(map(sys.getsizeof, vector_list)) + sys.getsizeof(vector_list)
    print("numpy array size: {}\nlist size: {}".format(npvect.nbytes, listsz)) 
    print("getsizeof(numpy array): {}\n".format(sys.getsizeof(npvect))) 

allocate()

and it outputs:

numpy array size: 80000
list size: 327632
getsizeof(numpy array): 80096

Upvotes: 3

Related Questions