Coverage.py gives wrong stat when called with setuptools

Question

Setup

Pytest and Coverage Settings

In my python project myproject I use the following setup.cfg:

...
[aliases]
test = pytest

[tool:pytest]
addopts = --durations=5 --cov
testpaths = tests

[coverage:run]
branch = True
source = myproject
...

As you see I use pytest and have aliased it to test, so I can call it through setuptools (python setup.py test).

Directory Layout

My project structure looks like the following:

.
├── MANIFEST.in
├── README.rst
├── tests
│   └── test_version.py
└── myproject
    ├── __init__.py
    └── _version.py

Please note, there is no .coveragerc file, so all settings are comming from setup.py

The Problem

When I run pytest directly, coverage gives me right stats:

running pytest
running egg_info
writing requirements to myproject.egg-info/requires.txt
writing myproject.egg-info/PKG-INFO
writing top-level names to myproject.egg-info/top_level.txt
writing dependency_links to myproject.egg-info/dependency_links.txt
writing entry points to myproject.egg-info/entry_points.txt
reading manifest file 'myproject.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*.pyo' found under directory 'tests'
writing manifest file 'myproject.egg-info/SOURCES.txt'
running build_ext
================================================================================ test session starts ================================================================================
platform darwin -- Python 2.7.13, pytest-3.0.7, py-1.4.33, pluggy-0.4.0 -- /Users/someuser/.virtualenvs/myproject/bin/python
cachedir: .cache
rootdir: /Users/someuser/Desktop/myproject, inifile: setup.cfg
plugins: cov-2.4.0
collected 7 items 

tests/test_version.py::test_versionInfoIsExposed PASSED
tests/test_version.py::test_versionInfoIsASemanticVersionTuple PASSED
tests/test_version.py::test_versionIsExposed PASSED
tests/test_version.py::test_versionIsASemanticVersionString PASSED
tests/test_version.py::test_versionLooksGeneratedFromVersionInfo PASSED

---------- coverage: platform darwin, python 2.7.13-final-0 ----------
Name                            Stmts   Miss Branch BrPart  Cover
-----------------------------------------------------------------
myproject/__init__.py               1      0      0      0   100%
myproject/_version.py               3      0      0      0   100%
-----------------------------------------------------------------
TOTAL                               4      0      0      0   100%
...

When I run python setup test, coverage gives me different (and wrong) stats:

running pytest
running egg_info
writing requirements to myproject.egg-info/requires.txt
writing myproject.egg-info/PKG-INFO
writing top-level names to myproject.egg-info/top_level.txt
writing dependency_links to myproject.egg-info/dependency_links.txt
writing entry points to myproject.egg-info/entry_points.txt
reading manifest file 'myproject.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*.pyo' found under directory 'tests'
writing manifest file 'myproject.egg-info/SOURCES.txt'
running build_ext
================================================================================ test session starts ================================================================================
platform darwin -- Python 2.7.13, pytest-3.0.7, py-1.4.33, pluggy-0.4.0 -- /Users/someuser/.virtualenvs/myproject/bin/python
cachedir: .cache
rootdir: /Users/someuser/Desktop/myproject, inifile: setup.cfg
plugins: cov-2.4.0
collected 7 items 

tests/test_version.py::test_versionInfoIsExposed PASSED
tests/test_version.py::test_versionInfoIsASemanticVersionTuple PASSED
tests/test_version.py::test_versionIsExposed PASSED
tests/test_version.py::test_versionIsASemanticVersionString PASSED
tests/test_version.py::test_versionLooksGeneratedFromVersionInfo PASSED

---------- coverage: platform darwin, python 2.7.13-final-0 ----------
Name                            Stmts   Miss Branch BrPart  Cover
-----------------------------------------------------------------
myproject/__init__.py               1      1      0      0     0%
myproject/_version.py               3      3      0      0     0%
-----------------------------------------------------------------
TOTAL                               4      4      0      0     0%
...

Does anyone know why this happens?

Answer 1

If someone stumbles upon this, I finally found the solution. In my setup.cfg I had the following line inside the metadata section:

[metadata]
...
version = attr: myproject._version.__version__
...

This caused the module myproject.__init__ and all lines in myproject._version.py to be run before the tests ran and as such coverage had no chance to tell, that those line were actually covered by the tests.

To solve that issue I just read the __version__ information inside setup.py without importing the module:

# inside setup.py
_locals = {}
with open('sirup/_version.py') as fp:
    exec(fp.read(), None, _locals)
version = _locals['__version__']

setup(version=version)