Do scientific studies exist enabling practitioners to make quantitative decisions regarding benefits of using Agile for projects?

Question

Core question

Are there any scientific studies that have compared Agile methodologies (such as Scrum) to other software development methodologies (such as Waterfall), resulting in recommendations that companies can use to quantify the advantage (or disadvantage) of using the studied methodologies as a factor affecting likelihood of project success?

I think agile is a nice way of working, but is there any scientific basis for it?

Background:

I'm looking for something like this, except this study only deals with TDD. Have there been studies about agile in general?

I'm asking this because I met a agile-zealot who claimed the agile way of working produces better software than all the other ways of working. He presented this as a fact, which I don't think it is.

Answer 1

This paper analyzes the return on investment (ROI) of Agile vs Traditional methods. It's based on actual results from published studies.

From the abstract:

Agile Methods ROI was four times more than expensive Traditional Methods, two times less than inexpensive ones, and the best Agile and Traditional Methods had equal ROI.

Answer 2

This interview with Linda Rising on InfoQ addresses your question to some extent. She talks about the effectiveness of placebos and our beliefs in medicine and how these same things might relate to Software Development. Basically, are we in the Agile community giving our selves a "Sugar Pill"?

Interview Excerpt

Science is about experiments, about holding an idea for a short period of time and testing it and then examining the results of that test to find out whether or not the hypothesis holds up. That's really what Agile is about. Agile is about small experiments. I now believe that everything we do, not just software development, but our lives should be a series of small experiments.

We bring in every possible stakeholder, we bring in customers, we bring in users, testers work with developers. We are always examining carefully those sugar pills. Does it really work? What do you think? Are you happy with the results? That's the only thing that saves us - Agile itself. This series of small experiments.

Answer 3

No, it's not scientifically or otherwise proven. To "prove" it would mean either:

• To demonstrate its validity from an analytical point of view.
• To show that it works in practice and then infer its validity

An analytical approach is unfeasible, since we are dealing with people here rather than simple systems. You cannot formalise teams and organisations.

Empirical studies, on the other hand, have been conducted, but the outcomes are unconclusive. Robert Glass shows some interesting results in his book Software Creativity 2.0, for example.

So no, agile is not proven. Not even close. :-)

Answer 4

What is proven to me is the statistical failure of waterfall i.e. scientific management applied to software development. Agile, as a movement, is just an answer to this empirical evidence (see for example the CHAOS reports).

Answer 5

I don't think it's possible to "prove" such a thing.

I'll go even further, and say that I don't think it's really possible to study the issue of "Software Development productivity" with such a study. Which is mainly why all the evidence we ever really have to go on is what experienced people tell us (and unfortunately, each one has a different view on various methodologies).

There's a simple reason for this: people are completely different. Sit down a team of 5 people for a project of a few months (which is more, I'm guessing, than most studies ever manage; let's see anyone finance a few months of developer time), and you're bound to get completely different results. The problem is, there is no way to sepearate the many different factors here:

1. Ability of the individual programmers.
2. Dedication/effort put in by the programmers.
3. Experience with the tools.
4. The ability of whoever is acting as the team leader (just following a methodology isn't enough. If someone doesn't know how to manage a team, the methodology won't really be well represented).

And there are probably many more factors.

So what I'm trying to say is, don't believe studies that have "proven" that one methodology/tool/anything works better than others. They're almost impossible to do.

Answer 6

Certain aspects of scrum have supporting empirical evidence. Quite a number of empirical studies of different part of scrum have been done. I've heard Jeff Sutherland ( http://jeffsutherland.com/scrum/ the inventor of scrum) mention lots of specific studies and observations in his talks.

Agile in general is just an umbrella term designed to keep different political groups moderately happy. Don't expect to see an experiment prove anything general about all of "agile". It's too vague to be useful.

Answer 7

Scientific? Well, I'm very impressed of Alistair Cockburn work. Listen to him here

Alistair Cockburn had been a hardware designer and researcher for 16 years when IBM asked him to write a methodology for object-oriented projects. He's spent the last decade studying and writing about software development and learned that some of the most successful projects have the simplest processes. In 2001 he and 16 other software-development heavyweights met to discuss so-called lightweight methodologies, and one result was the Agile Software Development Manifesto, which includes four value statements: individuals and interactions over processes and tools; working software over comprehensive documentation; customer collaboration over contract negotiation; and responding to change over following a plan.

Answer 8

Laurie Williams from NCSU published a lot of really interesting studies on the effectiveness of pair programming, and then started dealing with more facets of agile.