Reputation: 18043

How can I retry a failed test?

I'll sometimes have 1 or 2 tests that fail in CI, and rerunning the build causes them to pass.

How can I automatically re-run these flaky tests so my build will pass the first time? Is there something similar to mocha's this.retries?

For example, I have a test that fails with "The element has an effective height of 0x0" about 10% of the time:

cy.visit('/')
cy.get('.my-element').click() // sometimes fails with not visible error

Upvotes: 18

Answers (4)

M. Justin

Reputation: 21094

Cypress supports test retries as of version 5.0.0, released on 8/19/2020. These are present to reduce test flakiness and continuous integration build failures. This feature is documented in the online Cypress documentation under the Test Retries guide.

By default, tests will not retry when they fail. To retry failing tests, test retries need to be enabled in the configuration.

Retries can be configured separately for run mode (cypress run) vs. open mode (cypress open), as these will typically be different.

There are two ways to configure retries: globally, and per test or test suite.

Global Configuration

The Cypress configuration file (cypress.json by default) allows configuring the retry attempt either per mode or for all modes.

To use a different value per mode:

{
  "retries": {
    "runMode": 2,
    "openMode": 0
  }
}

To use the same value for both modes:

{
  "retries": 1
}

Single test configuration

A test's configuration can specify the number of retry attempts specific to that test:

// Customize retry attempts for an individual test
describe('User sign-up and login', () => {
  // `it` test block with no custom configuration
  it('should redirect unauthenticated user to sign-in page', () => {
    // ...
  })

  // `it` test block with custom configuration
  it(
    'allows user to login',
    {
      retries: {
        runMode: 2,
        openMode: 1,
      },
    },
    () => {
      // ...
    }
  )
})

Single test suite configuration

A test suite's configuration can specify the number of retry attempts for each test within that suite:

// Customizing retry attempts for a suite of tests
describe('User bank accounts', {
  retries: {
    runMode: 2,
    openMode: 1,
  }
}, () => {
  // The per-suite configuration is applied to each test
  // If a test fails, it will be retried
  it('allows a user to view their transactions', () => {
    // ...
  }

  it('allows a user to edit their transactions', () => {
    // ...
  }
})

Upvotes: 0

kuceb

Reputation: 18043

Update (v5.0.0)

Cypress now has built-in retry support.

You can set test retries in Cypress 5.0 via configuration in cypress.json

{
  "retries": 1
}

or specify different options for runMode and openMode:

{
  "retries": {
    "runMode": 1,
    "openMode": 3
  }
}

runMode allows you to define the number of test retries when running cypress run

openMode allows you to define the number of test retries when running cypress open

You can turn on test retries for just a single test or suite via test options:

it('my test', {
  retries: 2
}, () => {
  // ...
})

// or

describe('my suite', {
  retries: 2
}, () => {
  // ...
})

If a test fails in a beforeEach, afterEach, or in the test body, it will be retried. Failures in beforeAll and afterAll hooks will not retry.

Old answer:

Official Support for test retries is on the way, but there's a plugin for that. cypress-plugin-retries

Disclosure: I'm the creator of the plugin.

Installation

Add the plugin to devDependencies

npm install -D cypress-plugin-retries

At the top of cypress/support/index.js:

require('cypress-plugin-retries')

Usage

Use the environment variable CYPRESS_RETRIES to set the retry number:

CYPRESS_RETRIES=2 npm run cypress

or use Cypress.env('RETRIES') in your spec file:

Cypress.env('RETRIES', 2)

or on a per-test or per-hook basis, set the retry number:

Note: this plugin adds Cypress.currentTest and you should only access it in the context of this plugin.

it('test', () => {
    Cypress.currentTest.retries(2)
})

Note: Please refer to this issue for updates about official cypress retry support

Upvotes: 16

Shri_Automation

Reputation: 191

For retry, you can add it in your config as below

{
  "retries": {
    // Configure retry attempts for `cypress run`
    // Default is 0
    "runMode": 2,
    // Configure retry attempts for `cypress open`
    // Default is 0
    "openMode": 0
  }
}

but instead of adding above I prefer to add (Which works for both run and open mode)

"retries": 1

Points to consider

You need cypress version > 5
As of now in cypress, you can retry test, not entire spec which will be a great ability to make tests more robust. I have already voted this as a critical feature (Also this one of the most voted for now) Ref: https://portal.productboard.com/cypress-io/1-cypress-dashboard/tabs/1-under-consideration

Upvotes: 4