CODEWITHSUNDEEP
pythontestingtravis-citox
ababic
ababic

Reputation: 263

Travis jobs reporting success, even though tests fail (using tox)

I'm looking specifically at the following build:
https://travis-ci.org/ababic/wagtailmenus/builds/267670218

All jobs seem to be reporting as successful, even though they all have a single, deliberately failing test, and this has been happening on different builds on the same project for at least the last 2 days.

The configuration in my .travis.yml hasn't changed significantly in a while, apart from switching to 'trusty' from 'precise' - and changing that back seems not to fix the issue.

My tox.ini hasn't been changed in a while either.

I tried forcing tox to an earlier version already, which didn't seem to help.

I know it's got to be something to do with tox or Travis, but that's where my knowledge ends. Any help at all would be greatly appreciated.

Upvotes: 1

Views: 185

Answers (1)

Oliver Bestwalter
Oliver Bestwalter

Reputation: 5397

I had a look at the project and this has nothing to do with either tox or travis. The problem is that runtests.py used in tox always returns with exitcode 0 whatever happens. Tox (and in extension Travis) needs an exitcode != 0 to be able to know that something went wrong.

relevant code in runtests.py:

[...]

def runtests():
    [...]
    try:
        execute_from_command_line(argv)
    except:
        pass

if __name__ == '__main__':
    runtests()

I did not check what execute execute_from_command_line exactly does but I would reckon that it returns an error code if something went wrong (or raises an exception if something went really wrong).

Therefore I would rewrite the code above like this:

import sys
[...]

def runtests():
    [...]
    return execute_from_command_line(argv)

if __name__ == '__main__':
    sys.exit(runtests())

This way you pass through whatever the function you run has to report about the outcome of your tests and exit the script with that as error code or if an exception is raised, the traceback is printed and the script also returns with a non zero code.

Upvotes: 3

Related Questions