CODEWITHSUNDEEP
continuous-integrationcypress
Dan
Dan

Reputation: 1411

Cypress retries making test always pass

I am attempting to use the Cypress (at 7.6.0) retries feature as per https://docs.cypress.io/guides/guides/test-retries#How-It-Works

but for some reason it seems to be not working, in that a test that is guaranteed to always fail:

describe('Deliberate fail', function() {
    it('Make an assertion that will fail', function() {
        expect(true).to.be.false;
    });
});

When run from the command line with the config retries set to 1,

npx cypress run --config retries=1 --env server_url=http://localhost:3011 -s cypress/integration/tmp/deliberate_fail.js

it seems to pass, with the only hint that something is being retried being the text "Attempt 1 of 2" and the fact that a screenshot has been made:

enter image description here

The stats on the run look also to be illogical: 1 test 0 passing 0 failing 1 skipped (but does not appear as skipped in summary)

Exactly the same behavior when putting the "retries" option in cypress.json, whether as a single number or options for runMode or openMode.

And in "open" mode, the test does not retry but just fails.

I am guessing that I'm doing something face-palmingly wrong, but what?

Upvotes: 2

Views: 1749

Answers (2)

Dan
Dan

Reputation: 1411

Found the cause .. to fix the problem that Cypress does not abort a spec file when one of the test steps (the it() steps) fails, I had the workaround for this very old issue https://github.com/cypress-io/cypress/issues/518 implemented

//
// Prevent it just running on if a step fails
// https://github.com/cypress-io/cypress/issues/518
//

afterEach(function() {
    if (this.currentTest.state === 'failed') {
        Cypress.runner.stop()
    }
});

This means that a describe() will stop on fail, but does not play well with the retry apparently.

My real wished-for use case is to retry at the describe() level, but that may ne expensive as the above issue just got resolved but the solution is only available to those on the Business plan at $300/mo. https://github.com/cypress-io/cypress/issues/518#issuecomment-809514077

Upvotes: 0

Mikkel
Mikkel

Reputation: 7777

I think your problem is that you are not testing anything. Cypress will retry operations that involve the DOM, because the DOM takes time to render. Retrying is more efficient than a straight wait, because it might happen quicker.

So I reckon because you are just comparing 2 literal values, true and false, Cypress says, "Hey, there is nothing to retry here, these two values are never going to change, I'm outta here!"

I was going to say, if you set up a similar test with a DOM element, it might behave as you are expecting, but in fact it will also stop after the first attempt, because when it finds the DOM element, it will stop retrying. The purpose of the retry is to allow the element to be instantiated rather than retrying because the value might be different.

I will admit that I could be wrong in this, but I have definitely convinced myself - what do you think?

Upvotes: 1

Related Questions