CODEWITHSUNDEEP
pythonproject-structure
veich
veich

Reputation: 539

Python imports for machine learning

I'm switching from R to Python for new machine learning project I'm doing.

I found one code architecture which seems ok (image below) but I have problems with with implementation if I want to keep my code DRY.

enter image description here

So basically I have a piece of code (functions) that I want to share across different modules (namely data, features and models located under src module). That code in this case contains functions for loading and saving the data.

The problem is that I cannot import functions from upper module (for example if I had functions.py directly under src folder)

I could add separate functions.py inside every module (data, features, models) but that would defeat the purpose of me trying to create well structured project.

I assume I'm not the only one with this issue.

How is this usually handled?

How do you create shared module that can be used by all most deeply nested modules? (I'd like to import the same module into make_dataset.py, build_features.py, predict_model.py, train_model.py)

EDIT:

I also forgot to mention I use virtualenv and global space should not be contaminated.

SOLUTION:

So what I ended up doing was I added the following code to my top package __init__.py file: import sys, os fullPathToProjectFolder = os.path.dirname(os.path.realpath(__file__)) sys.path.insert(0, fullPathToProjectFolder) print('__init__ project packege linked: ' + fullPathToProjectFolder)

So whenever you use anything from the main package or the subpackage it is automatically linked (because when you import main package then __init__.py file is automatically called). And every module can make it's imports relative to top package.

E.g: if train_model.py wants to import build_features.py it would do it like so:

from features import build_features

Upvotes: 2

Views: 893

Answers (2)

Dwi Sulfahnur
Dwi Sulfahnur

Reputation: 46

You need to make __init__.py file in your directory would you make as a module. So functions.py or whatever is in the folder that you can call with import. I hope it works https://docs.python.org/3/tutorial/modules.html#packages

Upvotes: 0

Maaaaa
Maaaaa

Reputation: 388

You need to add your project to your system path so that python can find the module.

This can be done within python:

import sys
sys.path.insert(0, fullPathToProjectFolder)

Then you will be able to call your modules with

import general_functions

Upvotes: 1

Related Questions