Cypress soft assertions with cy.origin - Why is it passing a test with failed assertions?

Question

I think we all know that cypress and soft assertions has been discussed to death, and various solutions to implementing soft solutions are out there. For some time now, I've been using the following code, based on the old stack question Does cypress support soft assertion?:

let isSoftAssertion = false;
let errors = [];

chai.softExpect = function ( ...args ) {
    isSoftAssertion = true;
    return chai.expect(...args);
},
chai.softAssert = function ( ...args ) {
    isSoftAssertion = true;
    return chai.assert(...args);
}

const origAssert = chai.Assertion.prototype.assert;
chai.Assertion.prototype.assert = function (...args) {
    if ( isSoftAssertion ) {
        try {
            origAssert.call(this, ...args)
        } catch ( error ) {
            errors.push(error);
        }
        isSoftAssertion = false;
    } else {

        origAssert.call(this, ...args)
    }
};

// monkey-patch `Cypress.log` so that the last `cy.then()` isn't logged to command log
const origLog = Cypress.log;
Cypress.log = function ( data ) {
    if ( data && data.error && /soft assertions/i.test(data.error.message) ) {
        data.error.message = '\n\n\t' + data.error.message + '\n\n';
        throw data.error;
    }
    return origLog.call(Cypress, ...arguments);
};

// monkey-patch `it` callback so we insert `cy.then()` as a last command 
// to each test case where we'll assert if there are any soft assertion errors
function itCallback ( func ) {
    func();
    cy.then(() => {
        if ( errors.length ) {
            const _ = Cypress._;
            let msg = '';

            if ( Cypress.browser.isHeaded ) {

                msg = 'Failed soft assertions... check log above ↑';
            } else {

                _.each( errors, error => {
                    msg += '\n' + error;
                });

                msg = msg.replace(/^/gm, '\t');
            }

            throw new Error(msg);
        }
    });
}

const origIt = window.it;
window.it = (title, func) => {
    origIt(title, func && (() => itCallback(func)));
};
window.it.only = (title, func) => {
    origIt.only(title, func && (() => itCallback(func)));
};
window.it.skip = (title, func) => {
    origIt.skip(title, func);
};

beforeEach(() => {
    errors = [];
});
afterEach(() => {
    errors = [];
    isSoftAssertion = false;
});

This has been working beautifully... until now! I'm now need to test cross-origin... For assertions against my base URL, nothing has changed.... but when I try and use softExpect within cy.origin, the assertions are correctly passed / failed, but the parent test is passed regardless. For example:

describe('dummy test', () => {

    it('tests without cy.origin - this will fail correctly', () => {
        chai.softExpect(1).to.equal(2)
    })

    it('tests without cy.origin - the assertion will fail, but test is passed', () => {
        cy.origin('some.url', () => {
            Cypress.require('../../support/modules/soft_assertions.js')
            chai.softExpect(1).to.equal(3)
        })
    })    
})

Can anyone advise why this is? Or how I can pass the fact that 1+ softassertion has failed back to the top level "it" block where cy.origin has been used?

TIA!

Answer 1

The short answer to Why is it passing a test is that cy.origin() gives you a different instance of Cypress. Any modules imported with Cypress.require() also have a separate instance.

The outer (main) instance does not know about soft assertions found in the inner origin instance.

The quickest fix would be to alter the module along the lines of the Singleton pattern, but I'm stuck with your code because of the monkey patches.

---

I do it based on this question How can I use soft assertion in Cypress using the npm package soft-assert.

You may be able to adapt the pattern to your code, so I'll keep the example as general as possible.

In general, to use any package across multiple origins

• import the package with Cypress.require() inside the origin callback (already done in your case)

• Yield data back from cy.origin() as per Yielding-a-value

• Coordinate the data from origin with the main Cypress instance

Implementation

Example soft-assert implementation

cypress/support/e2e

const jsonAssertion = require("soft-assert")

Cypress.Commands.add('softAssert', (actual, expected, message) => {
  jsonAssertion.softAssert(actual, expected, message)
  if (jsonAssertion.jsonDiffArray?.length) {
    jsonAssertion.jsonDiffArray.forEach(diff => {
      const log = Cypress.log({
        name: 'Soft assertion error',
        displayName: 'softAssert',
        message: diff.error.message.split('message:')[1].trim()
      })
      log.set({state: 'failed', end: true})
    })
  }
  cy.wrap(jsonAssertion, {log:false})  // return the internal instance (data only)
});

Cypress.Commands.add('combineSoftResults', (resultsFromOrigin) => {
  resultsFromOrigin.jsonDiffArray.forEach(jsonDiff => {
    const {actual, expected, message} = jsonDiff.error
    jsonAssertion.softAssert(actual, expected, message)  // sync with main instance
  })
})

Cypress.Commands.add('softAssertAll', () => {
  if (jsonAssertion.softAssertCount) throw new Error('Failed soft assertions')
})

Test

I've added a new cy.origin2() command to combine results, but you can just apply the pattern given in Cypress docs if preferred.

context('soft assert', () => {

  Cypress.Commands.add('origin2', (...args) => {
    cy.origin(...args).then(result => {
      cy.combineSoftResults(result)
    })
  })

  it('testing numbers with soft-assert', () => {
    cy.softAssert(1, 2)                                // main test soft fail
  
    cy.origin2('https://example.com', () => {
      Cypress.require('../support/e2e')
      cy.softAssert(1, 3)                              // origin soft fail
    })
  
    cy.softAssert(1, 4)                                // main test soft fail

    cy.softAssertAll()                                 // throw to fail test
  })
})

Results

With soft fails inside and outside cy.origin()

With soft fails only inside cy.origin()

Without a call to cy.softAssertAll(), test passes but shows soft failures.