CODEWITHSUNDEEP
pythonsetuptoolssetup.pypython-packaging
bicyclesonthemoon
bicyclesonthemoon

Reputation: 142

how to add package_data from outside src subdirectory so that they are actually installed?

I am aware that there were already similar questions but the answers provided there do not help my situation. I believe that the situation has changed since then with .whl files introduced and made default etc.

pip 21.2.4
python 3.9.5

The situation is like this. I have a python project for which I'm generating released packages. The project will not be uploaded to the public repository because it is a library for controlling a custom device at my job place.

In the released package I want to not only include the .py files but also additional files like templates for configuration files, examples of usage, other doc. etc.

Let's focus on the templates as rest will be analogical.

My current project structure (names changed):

deviceFFS
+-dist
| \-generated release packages...
+-src
| \-deviceffs
|   +-__init__.py
|   \-other .py files...
+-template
| +-template_A.ini
| \-template_B.ini
+-tests
| \-some tests...
+-venv
| \-...
+-LICENSE
+-MANIFEST.in
+-pyproject.toml
+-README.md
+-requirements.txt
+-setup.cfg
\-setup.py

And content of config files BEFORE I tried to add additional files:
setup.cfg:

[metadata]
name = deviceffs
version = 0.1.2
author = B
author_email = B
description = device for my job

[options]
python_requires= >=3.9
install_requires=
        configparser
        pyserial
package_dir=
        =src
packages=find:

[options.packages.find]
where=src

setup.py:

from setuptools import setup

setup()

pyproject.toml:

[build-system]
requires = [
        "setuptools",
        "wheel"
]
build-backend = "setuptools.build_meta"

Ok, so now I want to add the template files:
template/template_A.ini
template/template_B.ini

FIRST APROACH:

I edited MANIFEST.in:

include template/*

and added to setup.cfg:

include_package_data=true

With this done, the generated (python -m build) .tar.gz file did indeed contain the "template" subdirectory. don't know about the .hwl file because I don't know how to preview those. However if I install this package in custom project (with venv) using python -m pip install -vvv I see that these files are not installed anywhere no matter if I install from .tar.gz or from .whl. I use -vvv to see each file copy operation done by pip to see if it was installed or not.

SECOND APROACH:

instead of MANIFEST.in use package_data. I revert changes from previous aproach and instead I edit setup.py:

setup(package_data={"deviceffs": ["template/*"]})

unfortunately this way I would be able the files from '.\src\deviceffs\template' and not from '.\template' as the paths are relative to the.

And I don't want to put the templates (and in future also usage examples, other descriptions) to .\src and also don't want to distribute them separately to the package.

THIRD APROACH:

instead of 'package_data' use 'data_files'

This was a solution which seemed to work in the past. However now pip uses .whl by default (even if install from .tar.gz) and this appears to ignore 'data_files' totally.

FINALLY:

I'm out of ideas.
I lost most of today's day at work trying to make the generated release include my files, instead of actually working on the actual content of the files.

Information from the internet are contradicting each other.
I'm looking for advice how to correctly configure this.
Thanks in advance.

edited to add python and pip version.

Upvotes: 2

Views: 2361

Answers (2)

sinoroc
sinoroc

Reputation: 22295

Looks like you need to work with package_dir. I am not too sure about the resulting package tree structure that you are aiming for, so I will guess a bit.

If I assume that you want to have the template files in a package deviceffs.template, then it becomes clear that setuptools find_packages function can not find the deviceffs.templates package as there is no src/devicesffs/template directory. So we can not rely on find_packages exclusively and need to modify the list of packages on the fly in setup.py.

[Off the top of my head, without testing]

setup.py:

#!/usr/bin/env python"
import setuptools
setuptools.setup(
    packages=setuptools.find_packages() + ['deviceffs.template'],
    package_dir={
        'deviceffs': 'src/deviceffs',
        'deviceffs.template': 'template',
    },
    # The rest is in 'setup.cfg'
)

Something like that. You might also need to add a template/__init__.py file.

Upvotes: 1

xplo4d
xplo4d

Reputation: 260

adding recursive-include template/ *.ini to MANIFEST.in worked for me

Upvotes: -1

Related Questions