CODEWITHSUNDEEP
pythonpipsetuptools
nmz787
nmz787

Reputation: 2170

Python setuptools/pip packing data files into your package

I have this git repo structure:

.gitignore
JSONs/subdirA/some.json
JSONs/subdirB/other.json
MyPackage/__init__.py
MyPackage/myModule.py

How do I properly pack the JSONs folder into MyPackage/JSONs, without moving it there permanently (mostly because customers use this git repo directly for non-python usage, and the folder at the top of the repo is easy/intuitive... But now I also want to release this same dir into my PyPi package)?

I've tried adding it to the MANIFEST.in and then playing with data_files in setup.py as well as package_data... But to no avail. Maybe some .pyc or cached build files got the best of me... But I haven't figured it out from all the other (not quite) duplicate questions as they don't specifically call out their directory structure and desired final location.

I've tried resorting to os.walk and shutil.copy before the call to setup and then deleting that directory after setup... While it seems to work locally, when pushing to our local devpi package server, something goes wrong. Is my goal totally off the radar for setuptools/pip ideals??? Or am I just missing some key understanding? Please enlighten me!

Upvotes: 1

Views: 769

Answers (1)

sinoroc
sinoroc

Reputation: 22295

Something like the following could help:

First we need to make sure that the json files are added to the source distribution.

MANIFEST.in:

recursive-include JSONs *.json

Then in the actual setup script, the list of packages has to be modified on the fly to take into account the target package structure.

setup.py:

#!/usr/bin/env python3

import setuptools

PACKAGES = (
    setuptools.find_packages(exclude=['JSONs*'])
    +
    [
        f'MyPackage.{package}'
        for package
        in setuptools.find_namespace_packages(include=['JSONs*'])
    ]
)

setuptools.setup(
    packages=PACKAGES,
    package_dir={
        'MyPackage.JSONs': 'JSONs',
    },
    include_package_data=True,
    #
    name='Something',
    version='1.2.3',
)

JSONs/subdirA/some.json :

{"Marco": "Polo"}

Such package data can be read like so:

MyPackage/myModule.py:

import pkgutil

print(pkgutil.get_data('MyPackage', 'JSONs/subdirA/some.json').decode())

And use it like in the following:

$ python -m pip install .
$ # Move to another directory to prevent that the current working directory
$ # ... overshadows the installed project
$ cd ..
$ python -m MyPackage.myModule
{"Marco": "Polo"}

Upvotes: 4

Related Questions