CODEWITHSUNDEEP
python-3.xpytesttravis-cipytest-cov
Kevin Winata
Kevin Winata

Reputation: 444

Pytest coverage return different coverage result when testing CLI applications in Travis-CI

My project is a python CLI application and I use Travis for the CI. I test it locally using the script

script:
  - pytest pacco/cli/tests.py --cov=pacco --cov-config=.coveragerc
  - pytest --doctest-modules --cov=pacco --cov-append --cov-config=.coveragerc --cov-report xml
  - coverage report -m

and it gives me the result of 

Name                                           Stmts   Miss  Cover   Missing
----------------------------------------------------------------------------
pacco/__init__.py                                  1      0   100%
pacco/cli/__init__.py                              0      0   100%
pacco/cli/commands/__init__.py                     0      0   100%
pacco/cli/commands/binary.py                      85      3    96%   16, 30, 53
pacco/cli/commands/pacco.py                       13      0   100%
pacco/cli/commands/registry.py                    65      1    98%   27
pacco/cli/commands/remote.py                      37      6    84%   29-34
pacco/cli/commands/utils/__init__.py               0      0   100%
pacco/cli/commands/utils/command_abstract.py      57     27    53%   21-22, 27-39, 51-63, 67-74
pacco/cli/commands/utils/output_stream.py         17      2    88%   16, 24
pacco/cli/entry_point.py                           4      0   100%
pacco/cli/test_utils.py                           54      0   100%
pacco/cli/tests.py                               115      1    99%   195
pacco/manager/__init__.py                          0      0   100%
pacco/manager/file_based/__init__.py               0      0   100%
pacco/manager/file_based/package_binary.py        30      1    97%   40
pacco/manager/file_based/package_manager.py       28      2    93%   37, 58
pacco/manager/file_based/package_registry.py     130      9    93%   65, 71, 100, 108, 145, 159, 217, 224, 226
pacco/manager/file_based/remote.py                44     10    77%   17, 20, 48, 52-57, 60
pacco/manager/interfaces/__init__.py               0      0   100%
pacco/manager/interfaces/package_binary.py         8      2    75%   21, 30
pacco/manager/interfaces/package_manager.py       13      4    69%   22, 31, 43, 56
pacco/manager/interfaces/package_registry.py      22      8    64%   23, 37, 46, 60, 66, 79, 91, 105
pacco/manager/interfaces/remote.py                14      3    79%   14, 18, 21
pacco/manager/remote_factory.py                    7      3    57%   7-10
pacco/manager/remote_manager.py                   66      4    94%   93, 127, 140, 163
pacco/manager/utils/__init__.py                    0      0   100%
pacco/manager/utils/cache.py                      28      0   100%
pacco/manager/utils/clients.py                   130     70    46%   28, 38, 47, 58, 68, 78, 117, 131-146, 150-151, 154-160, 163, 166, 169-170, 173-175, 178, 181, 184-196, 199-217, 220-222
----------------------------------------------------------------------------
TOTAL                                            968    156    84%

But, when it is run in the Travis, the result is only

Name                                           Stmts   Miss  Cover   Missing
----------------------------------------------------------------------------
pacco/__init__.py                                  1      0   100%
pacco/cli/__init__.py                              0      0   100%
pacco/cli/commands/__init__.py                     0      0   100%
pacco/cli/commands/binary.py                      85     69    19%   15-17, 20-34, 37-56, 59-69, 75-85, 88-107
pacco/cli/commands/pacco.py                       13      4    69%   9, 12, 15, 19
pacco/cli/commands/registry.py                    65     55    15%   11-15, 21-29, 35-40, 46-63, 69-76, 82-91, 97-105
pacco/cli/commands/remote.py                      37     29    22%   10-13, 19-34, 45-48, 54-57, 63-66
pacco/cli/commands/utils/__init__.py               0      0   100%
pacco/cli/commands/utils/command_abstract.py      57     45    21%   10-14, 20-41, 44-48, 51-63, 67-74, 77
pacco/cli/commands/utils/output_stream.py         17     10    41%   6-7, 10, 13-18, 21, 24
pacco/cli/entry_point.py                           4      1    75%   10
pacco/cli/test_utils.py                           54      0   100%
pacco/cli/tests.py                               115      1    99%   195
pacco/manager/__init__.py                          0      0   100%
pacco/manager/file_based/__init__.py               0      0   100%
pacco/manager/file_based/package_binary.py        30      6    80%   40, 56-60
pacco/manager/file_based/package_manager.py       28      2    93%   37, 58
pacco/manager/file_based/package_registry.py     130     12    91%   65, 71, 100, 108, 145, 159, 182, 186, 196, 217, 224, 226
pacco/manager/file_based/remote.py                44     10    77%   17, 20, 48, 52-57, 60
pacco/manager/interfaces/__init__.py               0      0   100%
pacco/manager/interfaces/package_binary.py         8      2    75%   21, 30
pacco/manager/interfaces/package_manager.py       13      4    69%   22, 31, 43, 56
pacco/manager/interfaces/package_registry.py      22      8    64%   23, 37, 46, 60, 66, 79, 91, 105
pacco/manager/interfaces/remote.py                14      3    79%   14, 18, 21
pacco/manager/remote_factory.py                    7      3    57%   7-10
pacco/manager/remote_manager.py                   66      4    94%   93, 127, 140, 163
pacco/manager/utils/__init__.py                    0      0   100%
pacco/manager/utils/cache.py                      28      5    82%   26, 34-37
pacco/manager/utils/clients.py                   130     70    46%   28, 38, 47, 58, 68, 78, 117, 131-146, 150-151, 154-160, 163, 166, 169-170, 173-175, 178, 181, 184-196, 199-217, 220-222
----------------------------------------------------------------------------
TOTAL                                            968    343    65%

You can see the full log in the travis.

I believe that the source of the problem is that because it's a CLI application, that I test it using subprocess.run(CLI_COMMAND) such that the pytest-cov cannot detect the trace from the child process. But it works on my local machine.

When I check the missing lines in the travis coverage result, it looks like that the body of the functions is all ignored.

I suspect that it might be because of the different Python environment (related to the sys.settrace behavior).

Upvotes: 1

Views: 1502

Answers (1)

hauntsaninja
hauntsaninja

Reputation: 1009

If, as Kevin mentions, it's possible for you to call a function to run your CLI directly, that's preferable to using subprocesses.

I ran into the same issue, however, directly calling into my code wasn't an option for my use case. Like OP, I was using pytest-cov to measure code coverage through subprocesses. pytest-cov's documentation contains this small note, that annoyingly isn't mentioned on the page dedicated to subprocess pitfalls (https://pytest-cov.readthedocs.io/en/latest/readme.html#limitations)

For subprocess measurement environment variables must make it from the main process to the subprocess. The python used by the subprocess must have pytest-cov installed. The subprocess must do normal site initialisation so that the environment variables can be detected and coverage started.

Anyway, I was installing pytest-cov as part of tox workflow, so it presumably got installed into whatever environment tox created. Meanwhile, Travis was installing my package in .travis.yml in Travis' environment. Shelling out to my CLI app meant that it then used the Python interpreter from Travis' environment, which didn't have pytest-cov installed. The fix that worked for me was to pip install my package as part of tox.

Upvotes: 1

Related Questions