CODEWITHSUNDEEP
pythonperformancememory-management
New User
New User

Reputation: 51

How to import a class from a module without importing the whole module

I have a python module which is really big (more than 1 Gigabyte) and I'm importing a class from this module in another python script with the command from module import class. The thing is, when I firt launch my python script, the memory consumption is really high and the script takes a very long time to execute (few minutes!). When launching it after that, it takes significantly less time (a few seconds) and uses less memory but still a lot for me.

What I think my script does is that it loads all the data from the module when I first launch it into memory which is why it takes so much time and memory.

Is there a way to change that and not have my script import the whole module but only specific parts which I would like ?

Thanks for taking the time to answer :)

Upvotes: 5

Views: 6906

Answers (2)

big meanie
big meanie

Reputation: 61

ok, this is going to be a bit of black-magic but if you only wish to import or run specific parts of a module then you can use the ast module to sift out what you want.

This is also occasionally a useful troubleshooting tool, if you wish to only run parts of your script and it couples well with the use of subprocess module as well.

But be really careful about the use of eval. and you are going to have to open the module as a file.

import ast

astparse = ast.parse(code_as_string)

astbodypart = astparse.body[:2]

astCobbledModule = ast.Module(body = astbodypart, type_ignores = "")

astunparse = ast.unparse(astCobbledModule)

exec(astunparse)

Upvotes: 2

bruno desthuilliers
bruno desthuilliers

Reputation: 77912

Short answer: no, there's no way to avoid this. The first time a module is imported in a gien process, all it's top-level statements (imports, def, class, and of course assignment) are executed to build the runtime module object. That's how Python work, and there are very valid reasons for it to work that way.

Now the solution here is quite simple: 1/ split your gigantic module into proper (high cohesion / low coupling) modules and only import the parts you need, and 2/ instead of defining gigabytes of data at the top-level, encapsulate this part in functions with some caching system to avoid useless re-computations.

Upvotes: 4

Related Questions