CODEWITHSUNDEEP
perltestingcpan
Chas. Owens
Chas. Owens

Reputation: 64929

How can I tell CPAN Testers how to setup the environment for my tests?

I am writing tests that require external software (Amazon's local DynamoDB server) to be installed and running. Is there some way to tell CPAN Testers what to do?

Or should I just download the server and start it myself in the test setup? That would require Java 6.x or newer to be installed. So I think I am back to the first question.

In case people don't know, CPAN Testers is a group of people who test all of CPAN using automated scripts called smokers.

Further background:

Right now, CPAN Testers shows 227 machines pass all tests for Amazon::DynamoDB, but that is misleading since only one of the over seven thousand tests is currently being run: use_ok( 'Amazon::DynamoDB' );. The rest are hidden behind unless statements:

unless ( $ENV{'AMAZON_DYNAMODB_EXPENSIVE_TESTS'} ) {
    plan skip_all => 'Testing this module for real costs money.';
}

And a significant number of the tests do not pass. I have fixed that, but testing requires either the setting of three environment variables in the tester's environment and money (the current way):

AMAZON_DYNAMODB_EXPENSIVE_TESTS=1
EC2_ACCESS_KEY=<user's AWS access key>
EC2_SECRET_KEY=<user's AWS secret key>

or the installation of the local version of Amazon DynamoDB. If this module is released as is, it will appear broken on all machines it runs on that don't have the prerequisite environment setup (ie it will erroneously appear broken rather than erroneously appear to be working).

Upvotes: 2

Views: 252

Answers (4)

daxim
daxim

Reputation: 39158

Is there some way to tell CPAN Testers what to do?

This is more of a social problem than a technical one.

You can ask the regulars on cpan-testers-discuss to manually set up the requirements; there's precedent for doing so. Not everyone will oblige, of course.

Another possibility is to reach out to your module's users and ask them to become ad-hoc test reporters via Task::CPAN::Reporter/cpanm-reporter or similar.

Upvotes: 2

mob
mob

Reputation: 118615

Call die from Makefile.PL or Build.PL if the prerequisites for building your module cannot be satisfied. On CPANTesters, aborting from the Makefile will give you an NA test result instead of a FAIL test result, and does not reflect poorly on your module and your build process.

# Makefile.PL
...
if ($ENV{AUTOMATED_TESTING}) {
    if (!$ENV{AMAZON_DYNAMODB_EXPENSIVE_TESTS} ||
            !$ENV{EC2_ACCESS_KEY} ||
            !$ENV{EC2_SECRET_KET}) {
        die "To test this module, you must set the environment\n",
            "variables  EC2_ACCESS_KEY, EC2_SECRET_KEY, and\n",
            "AMAZON_DYNAMODB_EXPENSIVE_TESTS.  Be advised that\n",
            "running these test will result in charges against\n",
            "your AWS account.";
    }
}
...

Upvotes: 3

amon
amon

Reputation: 57640

CPAN Testers run the same tests that your module will run upon installation. Should your tests install other software on the machine? Probably not. Instead, the tests should fail loudly when its prerequisites are not met.

You should also draw a distinction between author tests and installation tests. There is no expectation that the installation tests verify all the functionality. Expensive tests (in this case, tests that literally cost money) shouldn't be part of that. You can run them yourself before you release. However, it might be better to put them in xt/ and guard them with the EXTENDED_TESTING variable instead of a non-standard environment variable. See also the Lancaster Consensus for a discussion of various environment variables during testing of Perl projects.

You can also consider using a different provider for your more thorough tests than the donated CPAN Testers capacity, e.g. by setting up Travis CI for your project. Since they give you a container to play around, you can install extra software. You can also securely provide credentials to your tests. In contrast, the main advantage of CPAN Testers is the diverse range of operating systems, i.e. the lack of control over the testing environment.

Upvotes: 4

Tanktalus
Tanktalus

Reputation: 22274

Look at what other CPAN modules that have external dependencies do, and do something like that.

For example, look at the DBI drivers for various databases. While File and SQLite come with their own prereqs, the same is not true for others like Oracle and DB2. Or look at wxGTK which, IIRC, uses an Alien package to install Wx.

In your case, I would suggest more along the lines of the DBD drivers than embedding through Alien, but you have to make that choice.

Upvotes: 1

Related Questions