CODEWITHSUNDEEP
pythonlistpython-3.xpipgenerator
SaiKiran
SaiKiran

Reputation: 6504

No module named mem_profile

I'm using this program to measure the time and memory used by two functions and compare which is better for processing a large amount of data. My understanding is that to measure the memory usage we need the mem_profile module, but during the pip install mem_profile it gave me the error No module named mem_profile.

import mem_profile
import random
import time

names = ['Kiran','King','John','Corey']
majors = ['Math','Comps','Science']

print 'Memory (Before): {}Mb'.format(mem_profile.memory_usage_resource())

def people_list(num_people):
    results = []
    for i in num_people:
        person = {
                    'id':i,
                    'name': random.choice(names),
                    'major':random.choice(majors)
                  }
        results.append(person)
    return results

def people_generator(num_people):
    for i in xrange(num_people):
        person = {
                    'id':i,
                    'name': random.choice(names),
                    'major':random.choice(majors)
                  }
        yield person

t1 = time.clock()
people = people_list(10000000)
t2 = time.clock()


# t1 = time.clock()
# people = people_generator(10000000)
# t2 = time.clock()

print 'Memory (After): {}Mb'.format(mem_profile.memory_usage_resource())
print 'Took {} Seconds'.format(t2-t1)

What has caused this error? And are there any alternative packages I could use instead?

Upvotes: 8

Views: 19068

Answers (10)

Devansh
Devansh

Reputation: 1267

Use this for calculating time:

import time

time_start = time.time()
#run your code
time_elapsed = (time.time() - time_start)

As referenced by the Python documentation:

time.time() → float Return the time in seconds since the epoch as a floating point number. The specific date of the epoch and the handling of leap seconds is platform dependent. On Windows and most Unix systems, the epoch is January 1, 1970, 00:00:00 (UTC) and leap seconds are not counted towards the time in seconds since the epoch. This is commonly referred to as Unix time. To find out what the epoch is on a given platform, look at gmtime(0).

Note that even though the time is always returned as a floating point number, not all systems provide time with a better precision than 1 second. While this function normally returns non-decreasing values, it can return a lower value than a previous call if the system clock has been set back between the two calls.

The number returned by time() may be converted into a more common time format (i.e. year, month, day, hour, etc…) in UTC by passing it to gmtime() function or in local time by passing it to the localtime() function. In both cases a struct_time object is returned, from which the components of the calendar date may be accessed as attributes.

Reference: https://docs.python.org/3/library/time.html#time.time

Use this for calculating memory:

import resource

resource.getrusage(resource.RUSAGE_SELF).ru_maxrss

Reference: http://docs.python.org/library/resource.html

Use this if you using python 3.x:

Reference: https://docs.python.org/3/library/timeit.html

Upvotes: 7

Mj Khodadadi
Mj Khodadadi

Reputation: 41

I went through the same tutorial the library name is memory_profiler in order to install, you can use the following

pip install -U memory_profiler

to importing

import memory_profiler

note: the library does not have memory_usage_resource function but you can use memory_usage with the same functionality. instead of using clock() function use time() function

import memory_profiler
import random
import time

names=['John', 'Jane', 'Adam','Steve', 'Rick','George','Paul','Bill','Bob']
majors=['Math','Engineering','ComSic','Arts','Stuck Broker']

print ('Memory (Before): {} MB'.format(memory_profiler.memory_usage()))
#using list
def people_list(num_people):
    result = []
    for i in range(num_people):
        person={
            'id':i,
            'name':random.choice(names),
            'major':random.choice(majors)
        }
        result.append(person)
    return result


#using generators
def people_generator(num_people):
    for i in range(num_people):
        person={
            'id':i,
            'name':random.choice(names),
            'major':random.choice(majors)
        }
        yield person


# t1=time.time()
# people_list(1000000)
# t2=time.time()

t1=time.time()
people_generator(1000000)
t2=time.time()

print('Memory (After): {} MB'.format(memory_profiler.memory_usage()))
print ('Took {} seconds'.format(t2-t1))

Upvotes: 1

Wolftales
Wolftales

Reputation: 25

A couple of python 3.8 updates since time.clock() was removed, and print() has evolved.

Thanks everyone for this discussion, and definitely thanks to Corey Schafer great video.

import memory_profiler as mem_profile
import random
import time

names = ['John', 'Corey', 'Adam', 'Steve', 'Rick', 'Thomas']
majors = ['Math', 'Engineering', 'CompSci', 'Arts', 'Business']

print(f'Memory (Before): {mem_profile.memory_usage()}Mb')

def people_list(num_people):
    result = []
    for i in range(num_people):
        person = {
                    'id': i,
                    'name': random.choice(names),
                    'major': random.choice(majors)
                }
        result.append(person)
    return result

def people_generator(num_people):
    for i in range(num_people):
        person = {
                    'id': i,
                    'name': random.choice(names),
                    'major': random.choice(majors)
                }
        yield person

# t1 = time.process_time()
# people = people_list(1000000)
# t2 = time.process_time()

t1 = time.process_time()
people = people_generator(1000000)
t2 = time.process_time()


print(f'Memory (After): {mem_profile.memory_usage()}Mb')

print(f'Took {t2-t1} Seconds')

Upvotes: 1

Adebayo Ibro
Adebayo Ibro

Reputation: 131

Was going through the same tutorial and encountered the same problem. But upon further research, I discovered the author of the tutorial used a package called memory_profiler, whose main file he changed to mem_profile, which he imported in the code tutorial.

Just go ahead and do pip install memory_profiler. Copy and rename the file to mem_profile.py in your working directory and you should be fine. If you are on Windows, make sure you install the dependent psutil package as well.

Hope this helps somebody.

Upvotes: 13

H S Umer farooq
H S Umer farooq

Reputation: 1081

1)First import module

pip install memory_profiler

2)include it in your code like this

import memory_profiler as mem_profile

3)change code

mem_profile.memory_usage_psutil() to memory_usage()

4)convert you print statements like this

print('Memory (Before): ' + str(mem_profile.memory_usage()) + 'MB' )
print('Memory (After) : ' + str(mem_profile.memory_usage()) + 'MB')
print ('Took ' + str(t2-t1) + ' Seconds')

5)you will have something like this code:

import memory_profiler as mem_profile
import random
import time

names = ['John', 'Corey', 'Adam', 'Steve', 'Rick', 'Thomas']
majors = ['Math', 'Engineering', 'CompSci', 'Arts', 'Business']

# print('Memory (Before): {}Mb '.format(mem_profile.memory_usage_psutil()))
print('Memory (Before): ' + str(mem_profile.memory_usage()) + 'MB' )

def people_list(num_people):
    result = []
    for i in range(num_people):
        person = {
                    'id': i,
                    'name': random.choice(names),
                    'major': random.choice(majors)
                }
        result.append(person)
    return result

def people_generator(num_people):
    for i in range(num_people):
        person = {
                    'id': i,
                    'name': random.choice(names),
                    'major': random.choice(majors)
                }
        yield person

# t1 = time.clock()
# people = people_list(1000000)
# t2 = time.clock()

t1 = time.clock()
people = people_generator(1000000)
t2 = time.clock()

# print 'Memory (After) : {}Mb'.format(mem_profile.memory_usage_psutil())
print('Memory (After) : ' + str(mem_profile.memory_usage()) + 'MB')

# print 'Took {} Seconds'.format(t2-t1)
print ('Took ' + str(t2-t1) + ' Seconds')

Now it work fine i m using python 3.6 and its working without any error.

Upvotes: 20

SKRelan
SKRelan

Reputation: 119

Adding to Adebayo Ibro's answer above. Do the following :

  • In terminal, run $ pip install memory_profiler
  • In your script, replace import mem_profile with import memory_profiler as mem_profile
  • In your script, replace all mem_profile.memory_usage_resource() with mem_profile.memory_usage().

Hope this helps!

Upvotes: 5

iDuran
iDuran

Reputation: 398

I just encountered the same problem. I solved it installing memory_profiler ($ pip install -U memory_profiler), and them modify the program as follows:

import memory_profiler
...
print('Memory (Before): {}Mb'.format(memory_profiler.memory_usage()))

Upvotes: 2

Surya Agustian
Surya Agustian

Reputation: 31

That module is hand written (not in python packages). I got this from Corey Schafer's comment in his youtube video. Just save this code as the module's name:

from pympler import summary, muppy
import psutil
import resource
import os
import sys

def memory_usage_psutil():
    # return the memory usage in MB
    process = psutil.Process(os.getpid())
    mem = process.get_memory_info()[0] / float(2 ** 20)
    return mem

def memory_usage_resource():
    rusage_denom = 1024.
    if sys.platform == 'darwin':
        # ... it seems that in OSX the output is different units ...
        rusage_denom = rusage_denom * rusage_denom
    mem = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / rusage_denom
    return mem

Upvotes: 3

user6511855
user6511855

Reputation:

Much simple with sys

    import sys
    ...
    print ('Memory (Before): {0}Mb'.format(sys.getsizeof([])))

Upvotes: 0

Corey Goldberg
Corey Goldberg

Reputation: 60604

during the pip install mem_profile it gave me error No module named mem_profile.

by default, pip will download packages from PyPI. No package exists on PyPI named "mem_profile", so of course you will get an error.

for timing blocks of code, the timeit module is what you want to use: https://docs.python.org/library/timeit.html

Upvotes: -1

Related Questions