Python 'ImportError: No module named' when importing modules across packages

Question

I came across a Python 'ImportError: No module named...' error attempting to import a Python module which resides in another Python package from a module which resides within another Python package. The following image shows the directory structure:

It has to be noted that I get this error only when I run the script from my terminal whereas when executing through PyCharm, this script successfully runs. The error when executing from terminal is as follows:

Traceback (most recent call last):
File "social_networks/linked_data.py", line 15, in <module>
    from text_analysis.text_refinement import camel_case_split
ImportError: No module named 'text_analysis'

I have tried different ways of importing such as the following without any success:

Method-1:

sys.path.insert(0, os.path.realpath('../text_analysis'))
from text_analysis.text_refinement import camel_case_split

Method-2:

from text_analysis.text_refinement import camel_case_split

What is the solution for this issue?

Answer 1

Short version

Change it to:

sys.path.insert(0, os.path.realpath('./'))
from text_analysis.text_refinement import camel_case_split

Or:

sys.path.insert(0, os.path.realpath('./text_analysis'))
from text_refinement import camel_case_split

Long version

I have recreated your project structure on my machine and managed to make it work. Let's go step by step, so that we can figure out what's happening.

First of all, I see you're working on your project in PyCharm. It automatically adds a project root to PYTHONPATH. You can read about this in detail in this thread. Since PyCharm takes care of path business for you, you do not really need

sys.path.insert(0, os.path.realpath('../text_analysis'))

for your code to run. The path will still be added, but it will not be used to locate the package. Try it on your machine. I think you will find it to be true. You can easily check out paths by running

for path in sys.path:
    print(path)

While this is interesting information, it does not answer your question how to run it from a terminal. To understand why it is not run from the script, lets see what Python paths you would have upon executing the (slightly modified) commands in method-1:

sys.path.insert(0, os.path.realpath('../text_analysis'))

try:
    from text_analysis.text_refinement import camel_case_split
    camel_case_split()
except:
    for path in sys.path:
        print(path)

# output:
# ~/text_analysis (where ../text_analysis path points to)
# ~/social-network-interest-engine/social_networks (where your file is)
# ... (several other irrelevant paths) ...

We can see that '../text_analysis' points one directory above what you need. What would happen if we deleted one of the full stops, and instead wrote './text_analysis'? Output seems to be what we need:

# output:
# ~/social-network-interest-engine/text_analysis
# ~/social-network-interest-engine/social_networks

But we still have not imported the function. We know this because we reach the except part, which prints the paths. Looking at import, we can see that we have text_analysis.text_refinement. Do we really need to state the directory name if we already added it to the path? No, we do not. If we write

from text_refinement import camel_case_split

instead, we find that the function has finally been imported. Following this logic, and assuming we wanted to leave text_analysis.text_refinement in import statement (for whatever reason), we could add path differently, too:

sys.path.insert(0, os.path.realpath('./'))

Note, however, that this way of inserting the path is somewhat brittle. The starting location is the path from which you call python python_file.py If you navigated to different directory, you would need to adjust os.path.realpath accordingly. What you can do instead:

sys.path.insert(0, 'full/path/to/application/app/folder')

Though this assumes that the directory/structure of your project will not change.

For a more in-depth overview of paths and imports, you can read more about importing stuff from different folders here, and if you prefer relative path imports, this is a useful thread. Of course, official documentation is also a good place to start.