CODEWITHSUNDEEP
pythonlanguage-agnosticbashtestingcommand-line
intuited
intuited

Reputation: 24034

testing command line utilities

I'm looking for a way to run tests on command-line utilities written in bash, or any other language.

I'd like to find a testing framework that would have statements like

setup:
    command = 'do_awesome_thing'
    filename = 'testfile'
    args = ['--with', 'extra_win', '--file', filename]
    run_command command args

test_output_was_correct
    assert_output_was 'Creating awesome file "' + filename + '" with extra win.'

test_file_contains_extra_win
    assert_file_contains filename 'extra win'

Presumably the base test case would set up a temp directory in which to run these commands, and remove it at teardown.

I would prefer to use something in Python, since I'm much more familiar with it than with other plausible candidate languages.

I imagine that there could be something using a DSL that would make it effectively language-agnostic (or its own language, depending on how you look at it); however this might be less than ideal, since my testing techniques usually involve writing code that generates tests.

It's a bit difficult to google for this, as there is a lot of information on utilities which run tests, which is sort of the converse of what I'm looking for.

Support for doctests embedded in the output of command --help would be an extra bonus :)

Upvotes: 11

Views: 4526

Answers (4)

iLoveTux
iLoveTux

Reputation: 3625

I know that this question is old, but since I was looking for an answer, I figured I would add my own for anyone else who happens along.

Full disclaimer: The project I am mentioning is my own, but it is completely free and open source.

I ran into a very similar problem, and ended up rolling my own solution. The test code will look like this:

from CLITest import CLITest, TestSuite
from subprocess import CalledProcessError


class TestEchoPrintsToScreen(CLITest):
    '''Tests whether the string passed in is the string
    passed out'''

    def test_output_contains_input(self):
        self.assertNotIsInstance(self.output, CalledProcessError)
        self.assertIn("test", self.output)

    def test_ouput_equals_input(self):
        self.assertNotIsInstance(self.output, CalledProcessError)
        self.assertEqual("test", self.output)

suite = TestSuite()

suite.add_test(TestEchoPrintsToScreen("echo test"))

suite.run_tests()

This worked well enough to get me through my issues, but I know it could use some more work to make it as robust as possible (test discovery springs to mind). It may help, and I always love a good pull request.

Upvotes: 1

frankc
frankc

Reputation: 11473

outside of any prepackaged testing framework that may exist but I am unaware of, I would just point out that expect is an awesome and so underutilized tool for this kind of automation, especially if you want to support multistage interaction, which is to say not just send a command and check output but respond to output with more input. If you wind up building your own system, it's worth looking into.

There is also python reimplementation of expect called pexpect.There may be some direct interfaces to the expect library available as well. I'm not a python guy so I couldn't tell you much about them.

Upvotes: 0

Ian Bicking
Ian Bicking

Reputation: 9922

Check out ScriptTest :

from scripttest import TestFileEnvironment

env = TestFileEnvironment('./scratch')

def test_script():
    env.reset()
    result = env.run('do_awesome_thing testfile --with extra_win --file %s' % filename)
    # or use a list like ['do_awesome_thing', 'testfile', ...]
    assert result.stdout.startswith('Creating awesome file')
    assert filename in result.files_created

It's reasonably doctest-usable as well.

Upvotes: 13

Daniel Rodriguez MSFT
Daniel Rodriguez MSFT

Reputation: 83

Well... What we usually do (and one of the wonders of O.O. languages) is to write all the components of an application before actually make the application. Every component might have an standalone way to be executed, for testing purpose (command line, usually), that also allows you to think in them as complete programs each by each, and use them in future projects. If what you want is to test the integrity of an existing program... well, I think the best way is to learn in deep how it work, or even deeper: read the source. Or even deeper: develop a bot to force-test it :3

Sorry that's what I have .-.

Upvotes: 1

Related Questions