Cannot install two packages that use the same namespace

Question

UPDATE: When installing both packages using setup.py alone they install just fine. When extracting the tarballs generated by sdist and installing them the same error occurs. This means that the problem is somewhere inside setuptools I guess.

I developed two projects that have two namespace packages: testsuite and testsuite.prettyprint. Both of these namespace packages' __init__.py contain:

__import__('pkg_resources').declare_namespace(__name__)

Here's the setup.py for testsuite.prettyprint.outcomes:

import pkgutil
from setuptools import setup


def get_packages():
    return [name for _, name, is_package in pkgutil.walk_packages('.') if name.startswith('testsuite') and is_package]

dependencies = ['nose2>=0.4.6', 'colorama>=0.2.5']

setup(
    name='testsuite-prettyprint-outcomes',
    version='0.1.0-alpha.1',
    packages=get_packages(),
    url='',
    license='BSD3',
    author='Omer Katz',
    author_email='omer.drow@gmail.com',
    description='testsuite-prettyprint-outcomes is a nose2 plugin that prettyprints test outcomes.',
    namespace_packages=['testsuite', 'testsuite.prettyprint'],
    install_requires=dependencies
)

and here is the setup.py for testsuite.prettyprint.traceback:

import pkgutil
import sys

from setuptools import setup


def get_packages():
    return [name for _, name, is_package in pkgutil.walk_packages('.') if name.startswith('testsuite') and is_package]

dependencies = ['nose2>=0.4.6', 'pygments>=1.6']

if sys.platform == 'win32':
    dependencies.append('colorama>=0.2.5')

setup(
    name='testsuite-prettyprint-traceback',
    version='0.1.0-alpha.2',
    packages=get_packages(),
    url='',
    license='BSD3',
    author='Omer Katz',
    author_email='omer.drow@gmail.com',
    description='testsuite-prettyprint-traceback is a nose2 plugin that prettyprints traceback on failures and errors.',
    namespace_packages=['testsuite', 'testsuite.prettyprint'],
    install_requires=dependencies
)

When installing them both it refuses to install one:

pip install testsuite-prettyprint-outcomes testsuite-prettyprint-traceback --use-mirrors
Downloading/unpacking testsuite-prettyprint-outcomes
  Downloading testsuite-prettyprint-outcomes-0.1.0-alpha.1.tar.gz
  Running setup.py egg_info for package testsuite-prettyprint-outcomes

Downloading/unpacking testsuite-prettyprint-traceback
  Downloading testsuite-prettyprint-traceback-0.1.0-alpha.2.tar.gz
  Running setup.py egg_info for package testsuite-prettyprint-traceback

Downloading/unpacking nose2>=0.4.6 (from testsuite-prettyprint-outcomes)
  Running setup.py egg_info for package nose2

    warning: no previously-included files matching '__pycache__' found anywhere in distribution
    warning: no previously-included files matching '*~' found anywhere in distribution
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
Downloading/unpacking colorama>=0.2.5 (from testsuite-prettyprint-outcomes)
  Downloading colorama-0.2.5.zip
  Running setup.py egg_info for package colorama

Downloading/unpacking pygments>=1.6 (from testsuite-prettyprint-traceback)
  Downloading Pygments-1.6.tar.gz (1.4MB): 1.4MB downloaded
  Running setup.py egg_info for package pygments

Downloading/unpacking six>=1.1,<1.2 (from nose2>=0.4.6->testsuite-prettyprint-outcomes)
  Running setup.py egg_info for package six

Installing collected packages: testsuite-prettyprint-outcomes, testsuite-prettyprint-traceback, nose2, colorama, pygments, six
  Running setup.py install for testsuite-prettyprint-outcomes
    Skipping installation of /home/omer/.virtualenvs/test/lib/python3.2/site-packages/testsuite/__init__.py (namespace package)
    Skipping installation of /home/omer/.virtualenvs/test/lib/python3.2/site-packages/testsuite/prettyprint/__init__.py (namespace package)

    Installing /home/omer/.virtualenvs/test/lib/python3.2/site-packages/testsuite_prettyprint_outcomes-0.1.0_alpha.1-py3.2-nspkg.pth
  Running setup.py install for testsuite-prettyprint-traceback
    error: package directory 'testsuite/prettyprint/outcomes' does not exist
    Complete output from command /home/omer/.virtualenvs/test/bin/python3.2 -c "import setuptools;__file__='/home/omer/.virtualenvs/test/build/testsuite-prettyprint-traceback/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-12l9lq-record/install-record.txt --single-version-externally-managed --install-headers /home/omer/.virtualenvs/test/include/site/python3.2:
    running install

running build

running build_py

creating build

creating build/lib

creating build/lib/testsuite

copying testsuite/__init__.py -> build/lib/testsuite

creating build/lib/testsuite/prettyprint

copying testsuite/prettyprint/__init__.py -> build/lib/testsuite/prettyprint

error: package directory 'testsuite/prettyprint/outcomes' does not exist

----------------------------------------
Command /home/omer/.virtualenvs/test/bin/python3.2 -c "import setuptools;__file__='/home/omer/.virtualenvs/test/build/testsuite-prettyprint-traceback/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-12l9lq-record/install-record.txt --single-version-externally-managed --install-headers /home/omer/.virtualenvs/test/include/site/python3.2 failed with error code 1 in /home/omer/.virtualenvs/test/build/testsuite-prettyprint-traceback
Storing complete log in /home/omer/.pip/pip.log

I can't figure out what is wrong. Even if you change the order of installation it says it can't find the other.

Answer 1

The names of your packages look wrong. I just separated a project out into subpackages, and one thing I did differently was to make each name match the components of the namespace_packages.

So, for testsuite.prettyprint.outcomes:

setup(
       name='testsuite.prettyprint.outcomes',
       [...] ,
       namespace_packages=['testsuite', 'testsuite.prettyprint']
)

And for testsuite.prettyprint.traceback:

setup(
       name='testsuite.prettyprint.traceback',
       [...] ,
       namespace_packages=['testsuite', 'testsuite.prettyprint']
)

For this to work properly, you'll need to provide __init__.py scripts, like you've already shown, for all the parent namespace_package levels (i.e. down to testsuite.prettyprint).

Good examples of namespace_packages in production packages can be found in the zope sub-packages.

e.g. See the zope.app.cache setup.py script, at http://svn.zope.org/zope.app.cache/trunk/setup.py?view=markup

Answer 2

I see you are using virtualenv. Usually when I encounter that error your are facing is because the packages don't work well with virtualenv.

Have you tried installing the packages on your main python install folder? (not in virtualenv)

I think this happens because some setup.py files make assumptions about the host environment and don't follow the setup.py best practices.

If you are still stuck give it a try.

Answer 3

After installing one of your packages and downloading the other…

You're not including testsuite/__init__.py and testsuite/prettyprint/__init__.py in the source files.

The docs on Namespace Packages the setuptools/pkg_resources way explains:

Note, by the way, that your project's source tree must include the namespace packages' __init__.py files (and the __init__.py of any parent packages), in a normal Python package layout.

If you don't actually install these files, they don't do any good. You just end up with a testsuite with nothing in it but prettyprint, and that has nothing in it but outcomes, so testsuite and testsuite.prettyprint are not packages at all, much less namespace packages.