How can I perform mutation testing of my Java program?

Question

Can anyone provide suggestions of tools that can be used to make mutations within a Java program at a source-code (not byte-code) level? I need to seed my source code with faults. I would prefer an application with a GUI, if one exists.

Answer 1

I've found https://pitest.org, which I quite like. With MuJava I have exceptions caused by @Override annotations. Seems that it doesnt' support Java 5/6.

Answer 2

"This has little application in the real world". Disagree strongly, I contract at a very big media company in the UK and PIT (http://pitest.org/) is extremely useful for mutation testing. Most of the CI builds are set fail without 85% coverage PIT mutation testing. Re: "replacing some random characters..." Mutation does not have to mean that. The PIT mutations are restricted to compilable changes.

Answer 3

The Major mutation framework (Major's website) provides a compiler-integrated mutator and a mutation analyzer for JUnit tests.

Major's mutator is integrated in the Java 7 compiler. It provides several mutation operators and supports two options for mutating source code:

1. Generate and embed all mutants during compilation
2. Generate mutants and export mutated source files

Answer 4

Here it is a benchmark :

MμClipse only supports JUnit 3 and is no longer maintained. Jester as for it, is laborious and requires a complicated configuration; plus is not maintained anymore. The best tool I could find is Javalanche I had wrote a entire article about this !

Answer 5

If your requirement really is for source code mutation then the options I'm aware of are

Jester - Provides a limited number of mutation operators that are quite unstable. Jeff Offutt described it as a very expensive way to apply branch testing. I believe it is possible to define your own operators however, so this probably isn't entirely fair.

Mu Java - Doesn't support Java 5

Judy - Don't know much about this other than that it exists

Lava - Sounds pretty basic and I don't think it supports java 5

From the answer further down I see your now using PIT. I think that's a truly great choice (I wrote it so may have a certain bias), but it's a byte code mutation system.

Answer 6

This is the technique used by the open source test coverage tool Jester. A look at it's source might be helpful if you are unable to find a suitable tool and want to build one yourself.

Answer 7

When I took a software engineering class on testing at GMU, we used Mu Java, for one of the chapters. HTH