Python script runs in PyCharm but not Git Bash

Question

Say I've got a arbitrarily big, modular Python 2.7 codebase:

project
↳ docs
↳ etc
↳ package
  ↳ module1
    ↳ submodule1
      ↳ subsubmodule1
        ↳ __init__.py
      ↳ subsubmodule2 (... and so on)
      ↳ __init__.py
    ↳ submodule2
      ↳ subsubmodule1
        ↳ __init__.py
      ↳ subsubmodule2 (... and so on)
      ↳ __init__.py
    ↳ submodule3 (... and so on)
      ↳ __init__.py
  ↳ module2
    ↳ submodule1
      ↳ __init__.py
    ↳ submodule2 (... and so on)
      ↳ __init__.py
    ↳ __init__.py        
  ↳ module3 (... and so on)
    ↳ __init__.py
  ↳ __init__.py
↳ test

• project is the root folder - it's a PyCharm project and it's not a module.
• project\package is the root Python module for the project. It contains many subdirectories, each of which is a Python module named moduleN.
• Each project\package\moduleN module contains many subdirectories, each of which is a Python module named submoduleN ... and so on and so forth.

---

Say I have a particular Python script called foo.py that I would like to run, and it's located within one of the infinitely many submodules under package:

# foo.py:

from package.module2.submodule3 import foo
print foo.bar()

When the script is ran from PyCharm with Ctrl+F9: no problem, foo.bar() prints.

But when the script is ran from the Git Bash terminal, from the home directory, with:

python path/to/project/package/module4/submodule6/subsubmobile5/foo.py

The following error is thrown:

ImportError: No module named package.module2.submodule3

---

I would like to know what I need to do in order to get my script to run on Git Bash, and why there is a discrepancy in the first place between PyCharm and Git Bash. Is it something to do with PYTHONPATH?

---

EDIT:

• Many StackOverflow answers suggest using some iteration of sys.path.append() hacks. None of these suggestions work in my case.
• I set up a .bashrc file with export PYTHONPATH=absolute/path/to/project/package, where package is my source root in PyCharm, but the import error is still thrown. Relative paths do not work either. I have verified that the paths are correct with echo $PYTHONPATH. export PYTHONPATH=absolute/path/to/project similarly does not work.
• The working directory doesn't seem to matter - Git Bash fails every time regardless of the working directory and PyCharm works every time regardless of the working directory.

EDIT 2:

The issue is still unresolved but it may have to do with PYTHONPATH not being set correctly on Git Bash. When Add Content Roots to PYTHONPATH is unticked, PyCharm throws the same import error as Git Bash.

Answer 1

Ultimately, Laurent's answers helped me figure this out.

I used:

    import sys
    for p in sys.path:
        print(p)

to help me determine that there was indeed a discrepancy between what's being added to PYTHONPATH on PyCharm versus Git Bash. On PyCharm, both /path/to/project and /path/to/project/package are being added to PYTHONPATH. On Git Bash, only one is being added.

So I modified my export statement in .bashrc such that it appends both paths to PYTHONPATH; i.e. from:

export PYTHONPATH="${PYTHONPATH}:absolute/path/to/project/package"

to:

export PYTHONPATH="${PYTHONPATH}:absolute/path/to/project:absolute/path/to/project/package"

The import error was then resolved.¹ Similarly, the script works as-intended on the Windows command prompt when the PATH AND PYTHONPATH user environmental variables are similarly declared.

My best guess is that setting the Sources Root in PyCharm to something other than the project directory may require multiple additions to PYTHONPATH.

_{1 On a site note: my script then started hanging indefinitely which was resolved by way of aliasing Python: alias python="winpty python".}

EDIT:

Git Bash operates under MinGW64 - where lists of Windows paths in export statements are subject to Posix path conversion prior to being processed by Python's import resolution.

So technically, export PYTHONPATH="absolute/path/to/project;absolute/path/to/project/package" should correctly resolve to two paths.

Answer 2

Notice: the solution exposed below won't work if your application/library has third-party dependencies. For that, you need a virtualenv. See my previous answer.

You have the following tree structure in path\to\project:

path\to\project
├───docs
├───etc
├───package
│   │   __init__.py
│   ├───module1 [...]
│   ├───module2
│   │   │   __init__.py
│   │   ├───submodule1 [...]
│   │   ├───submodule2 [...]
│   │   └───submodule3
│   │           foo.py
│   │           __init__.py
│   ├───module3 [...]
│   └───module4
│       │   __init__.py
│       └───submodule6
│           │   __init__.py
│           └───subsubmobile5
│                   foo.py
│                   __init__.py
└───tests

Here, package is the root Python package of your project, and path\to\project is your project source directory (the one that PyCharm use).

To run a script in this packages, for instance the script package/module4/submodule6/subsubmobile5/foo.py, you need to set the PYTHONPATH to the root of the project, i.e. path\to\project.

But, since you are on Windows and run Git Bash, you need to convert the Windows path to a valid Linux path, with forward slash.

One way to do that is as follow:

$ export PYTHONPATH=path/to/project && python path/to/project/package/module4/submodule6/subsubmobile5/foo.py

You can even execute the module package.module4.submodule6.subsubmobile5.foo:

$ export PYTHONPATH=path/to/project && python -m package.module4.submodule6.subsubmobile5.foo

Answer 3

The right way to solve your problem is not to hack the PYTHONPATH, because this won't be useful if your library/application has third-party dependencies.

To work properly, PyCharm used two things:

• the Source Root (which is added to the PYTHONPATH), and,
• the Project Interpreter (i.e. the virtualenv),

To check that, open a Terminal view in PyCharm, and try:

$ python
>>> import sys
>>> for p in sys.path:
...     print(p)
...

The right way is using a virtualenv.

To create a virtualenv on Windows, go to a directory where you want to create a virtualenv. It can be in a unique directory (like recommended by pew), or in your project directory (usually in .venv).

virtualenv -p C:\Python27\python.exe yourvenv

Then, activate your virtualenv and install your application in development/editable mode:

yourvenv\Scripts\activate

cd path\to\project\
pip install -e .

Here, you have installed your library/application with all its dependencies. The -e flag means "editable" (which match the old "develop" mode), See pip documentation for that.

Whenever you want to run a script, you can do as follow:

yourvenv\Scripts\activate
python -m package.module4.submodule6.subsubmobile5.foo

On Windows, you can also do:

yourvenv\Scripts\activate
python path\to\project\package\module4\submodule6\subsubmobile5\foo.py

On Git bash, you do:

source yourvenv/Scripts/activate
python path/to/project/package/module4/submodule6/subsubmobile5/foo.py

If you want to call your Python script from another batch, you can do:

yourvenv/Scripts/python.exe -m package.module4.submodule6.subsubmobile5.foo
yourvenv/Scripts/python.exe path/to/project/package/module4/submodule6/subsubmobile5/foo.py