CODEWITHSUNDEEP
pythonsetuptoolspackagingsetup.pyproject-structure
lexadler
lexadler

Reputation: 255

Packaging python project with multiple directories

I need some explanation on working with setuptools and find_packages function. I have a project structure like this:

├── project_dir_1
│   ├── module.py
│   ├── __init__.py
├── my_project
│   ├── cli.py
│   ├── subdir1
│   │   ├── __init__.py
│   │   ├── module.py
│   ├── conf
│   │   ├── module.py
│   │   ├── params
│   │   │   ├── config.yml
│   │   ├── __init__.py
│   ├── subdir2
│   │   ├── module.py
│   ├── __init__.py
│   └── version.py
├── project_dir_2
│   ├──  subdir1
│   │   ├── module.py
│   │   ├── __init__.py
│   ├── __init__.py
├── README.md
├── requirements.txt
├── setup.py
└── tests
    └── test_main.py

Actually all my code in the my_project dir and I also have two additonal dirs project_dir_1 and project_dir_2 that contains necessary external modules from where I should do imports both in the package code and in another projects code where this package will be installed in venv. I have setup script like this:

setup(
    name='my_project',
    version='0.0.1',
    description='Python library.',
    license='license',
    author='me',
    author_email='my_email',
    entry_points={'console_scripts': ['my_project=my_project.cli:main']},
    python_requires='>=3.7',
    packages=find_packages(
        include=['my_project', 'project_dir_1', 'project_dir_2', 'my_project.*', 'project_dir_1.*', 'project_dir_2.*']
    ),
    install_requires=list(open(join(dirname(__file__), 'requirements.txt')).read().split()),
)

When I activate venv in another project folder and trying to install package from package root folder like python ..\package_root\setup.py install everything seems to works fine during install. And pip list shows all dependencies and my_project 0.0.1. But if I'm trying to import something from my_project using the venv interpreter I got an error: ModuleNotFoundError: No module named 'my_project'. The same result if I'm trying to import something like from project_dir_1 import module which is also necessary. Also when I just run my_project from shell addressing to cli I got an error:

Traceback (most recent call last):
  File "/home/developer/another_project/env/bin/my_project", line 11, in <module>
    load_entry_point('my_project==0.0.1', 'console_scripts', 'my_project')()
  File "/home/developer/another_project/env/lib/python3.8/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/home/developer/another_project/env/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
    return ep.load()
  File "/home/developer/another_project/env/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2443, in load
    return self.resolve()
  File "/home/developer/another_project/env/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2449, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
ModuleNotFoundError: No module named 'my_project'

So what is the right way to organize this complex project structure and include all necessary code in setup.py to get the setuptools install the package correctly? I need some better understanding of python projects packaging but still don't get an answers for this case from seeking the docs.

Upvotes: 5

Views: 2600

Answers (1)

hoefling
hoefling

Reputation: 66531

find_packages will resolve paths relative to current working directory, so calling it outside of the project root dir will effectively install nothing (check whether you see any sources installed by e.g. running

$ pip show -f my_project

, I bet nothing will be listed). You have to force switching to the project root dir in the setup script, e.g. add a magic line to your setup script:

# setup.py

import os
from setuptools import setup

# old-style for python 2
os.chdir(os.path.normpath(os.path.join(os.path.abspath(__file__), os.pardir)))

# new style for python 3
from pathlib import Path
os.chdir(Path(__file__).parent.absolute())

setup(...)

Upvotes: 2

Related Questions