CODEWITHSUNDEEP
cucumberbddspecflowgherkin
Mark Chidlow
Mark Chidlow

Reputation: 1472

Gherkin scenario outlines or multiple scenarios?

Is there a preferred approach when defining acceptance criteria in Gherkin format for BDD?

Should I split out the scenarios as follows...

Scenario: User saves contact details
Given I am on the contact details page
When I enter the following details
| email          | phone     |
| [email protected] | 012345678 |
And I save the details
Then the details are correctly saved

Scenario: User saves contact details with a very long email address
Given I am on the contact details page
When I enter the following details
| email                                                         | phone     |
| [email protected] | 012345678 |
And I save the details
Then the details are correctly saved

Scenario: User saves contact details with a very long phone number

etc

Or should I use a single scenario with multiple outlines?

Scenario: User saves contact details
Given I am on the contact details page
When I enter the following details
| email   | phone   |
| <email> | <phone> |
And I save the details
Then the details are correctly saved

Examples: 
| email                                                         | phone                 |
| [email protected]                                                | 012345678             |
| [email protected] | 012345678             |
| [email protected]                                                | 012345678901234567890 |

The latter may be easier to adapt/add to but to me loses the basic concept of scenarios.

UPDATE I have just come across the following article, which discusses this exact point. Suggesting the latter is a better approach. https://www.hindsightsoftware.com/blog/scenario-outlines

It suggests something like the following:

Scenario: User saves contact details
Given I am on the contact details page
When I enter the following details
| email   | phone   |
| <email> | <phone> |
And I save the details
Then the details are correctly saved

Examples: 
| scenario                  | email                                                         | phone                 |
| basic details saved       | [email protected]                                                | 012345678             |
| very long email address   | [email protected] | 012345678             |
| very long phone number    | [email protected]                                                | 012345678901234567890 |

Upvotes: 1

Views: 3807

Answers (1)

Greg Burghardt
Greg Burghardt

Reputation: 18773

As a general matter of practice, if you vary the input, but expect the same output then you have a good case for a scenario outline.

You can parameterize the Then steps, but I advise against that, because changing the assertion of a test means you have fundamentally changed the test. It's a philosophical and code maintenance issue.

Consider the max length for a phone number, which by your example should be 11 characters. If 11 characters is the max, and you have a scenario outline that tests both sides of that boundary, then the scenario has multiple reasons to fail. First, when 11 is no longer the max length, and furthermore if 12 is now the max length, that test will fail too.

Tests with multiple reasons to fail are "flaky"

The scenario outline below varies the assertion as well as the input, which means this test has multiple reasons to fail.

Scenario Outline: User saves contact details
    Given I am on the contact details page
    When I enter the following details
        | email          | phone   |
        | [email protected] | <Phone> |
    And I save the details
    Then the details <Assertion> correctly saved

Examples:
    | Phone                  | Assertion |
    | 012345678901234567890  | Are       |
    | 0123456789012345678901 | Are Not   |

I would even advise separating the e-mail scenarios from the phone number scenarios.

Every test should only have one reason to fail.

The two scenarios below seem like duplicates of one another, but keeping the most of the input consistent, and only varying one of the inputs gives you more stable tests that fail for obvious reasons:

Scenario Outline: User saves contact phone number
    Given I am on the contact details page
    When I enter the following details
        | email          | phone   |
        | [email protected] | <Phone> |
    And I save the details
    Then the details are correctly saved

Examples:
    | Phone                 |
    | 012345678             |
    | 012345678901234567890 |

Scenario Outline: User saves contact e-mail address
    Given I am on the contact details page
    When I enter the following details
        | email   | phone     |
        | <Email> | 012345678 |
    And I save the details
    Then the details are correctly saved

Examples:
    | Email                                                         |
    | [email protected]                                                |
    | [email protected] |

Now when a scenario fails, the scenario name gives you a hint about which part of the application may be to blame, which aids debugging.

Upvotes: 3

Related Questions