CODEWITHSUNDEEP
pythonpython-importmemory-profiling
angelo_poerio
angelo_poerio

Reputation: 21

Importing a python module takes too much memory

I am working on a quite big existing Python application and I am now trying to reduce its memory usage.
Thanks to memory_profiler I got this:

23   11.289 MiB    0.434 MiB    from remote import settings
24   14.422 MiB    3.133 MiB    from remote.controller import ChannelManager
25   14.422 MiB    0.000 MiB    from remote import channel as channel_module

As you can see at line 24 the memory usage increases of 3.13MB.
Why importing a simple class (ChannelManager) should use so much memory?
ChannelManager is not a complex class, it just wraps some logic (spawning some processes and threads).
Thank you

Upvotes: 2

Views: 8420

Answers (2)

Dunes
Dunes

Reputation: 40713

from remote.controller import ChannelManager will import the entire remote.controller module into memory if it has not already been imported. As such ChannelManager may not be the culprit at all. It could be that remote.controller has a whole host of dependencies that aren't loaded into memory at that point in time.

I would profile the module by itself too where the memory is being used. In all likelihood it's a one off cost, that isn't worth trying to optimise.

Upvotes: 1

user559633
user559633

Reputation:

Unless those three megabytes are an issue, I would spend my time profiling the application itself as there's likely low-hanging fruit in your actual logic for cleanup.

When you import a module, Python compiles that module into bytecode (commonly seen as .pyc files) and stores their content into memory. The module that you're importing (e.g. remote.controller.ChannelManager) is likely many lines or references an object that makes its compiled form take up that amount of space.

Your options are either to accept that as a cost of using the ChannelManager object or pick a smaller module in order to bring that usage down. I would heavily suggest the former and look for portions of code that use more memory -- especially if you're on Python 2.x and there are areas that iterators could be employed.

Upvotes: 4

Related Questions