Hari Venkadesh
Reputation: 13
Test execution(Jira Xray) created but cucumber json testcases are not mapped to the test execution
After hitting this /rest/raven/2.0/import/execution/cucumber/multipart? api with below ~info and ~AI_sample cucumber json - it creates test execution but cucumber testcases are not mapped to the test execution, I also tried to import it manually (import Execution results ) no testcases mapped to the testexecution, I am not sure how to validate cucumber and what I missed here
info:
{
"fields": {
"project": {
"key": "IXXXX"
},
"summary": "Test Execution for cucumber Execution",
"description": "For more info please check [here|https://www.example.com]",
"issuetype": {
"id": "10009"
}
}
}
AI_sample cucumber json:
[
{
"uri": "features/Login.feature",
"elements": [
{
"name": "Successful login with valid credentials",
"id": "successful-login-with-valid-credentials",
"steps": [
{
"name": "I navigate to the login page",
"keyword": "Given ",
"result": {
"status": "passed"
}
},
{
"name": "I enter valid credentials",
"keyword": "When ",
"result": {
"status": "passed"
}
},
{
"name": "I am redirected to the dashboard",
"keyword": "Then ",
"result": {
"status": "passed"
}
}
]
},
{
"name": "Unsuccessful login with invalid credentials",
"id": "unsuccessful-login-with-invalid-credentials",
"steps": [
{
"name": "I navigate to the login page",
"keyword": "Given ",
"result": {
"status": "passed"
}
},
{
"name": "I enter invalid credentials",
"keyword": "When ",
"result": {
"status": "failed",
"errormessage": "Invalid username or password"
}
},
{
"name": "I see an error message",
"keyword": "Then ",
"result": {
"status": "passed"
}
}
]
}
]
},
{
"uri": "features/Account.feature",
"elements": [
{
"name": "Creating a new account",
"id": "creating-a-new-account",
"steps": [
{
"name": "I navigate to the account creation page",
"keyword": "Given ",
"result": {
"status": "passed"
}
},
{
"name": "I enter all required details",
"keyword": "When ",
"result": {
"status": "passed"
}
},
{
"name": "I successfully create a new account",
"keyword": "Then ",
"result": {
"status": "passed"
}
}
]
}
]
}
]
Upvotes: 0
Views: 123
Answers (1)
Sérgio
Reputation: 2129
First, I assume that you're using Xray on Jira Datacenter (and not on Cloud). The flow for integrating with Cucumber, and other based Gherkin frameworks, is described in detail here. We cannot just import the results; the results (on the Cucumber JSON report) need to contain references to existing Cucumber/Gherkin Tests in Xray; so you need to have Test issues corresponding to each Scenario/Scenario Outline in your .feature file.
In sum, you have 2 flows: one where you have Xray has the master for editing the Gherkin scenarios, or another one where you use Git (or other SCVS) as master to edit the Gherkin scenarios. If you wish to follow the latest flow, you need to synchronize (i.e., import) the Cucumber Scenarios to Xray using a proper REST API endpoint.
You can have a look at this tutorial where you can see the 2 flows.
Note that you cannot push the results and have Xray autoprovision Cucumber Tests in Jira; that's not possible. You always need to have the Cucumber Tests beforehand in Xray so that you can report against them.
Looking at the Cucumber JSON you shared, I don't see there references to the keyword "Scenario" or "Scenario Outline"; additionally, there are no `@TEST_xxx" tags on those scenarios. So it seems that you are not following none of the 2 flows I mentioned above, or else you would have the references to the Test issues on the generated report implicitly, as one of the steps of the flows it to export the .feature file out of Jira, so it generates it with the annotations automatically on the Gherkin.
Here's an example of one Cucumber JSON report that contains that information.
[
{
"line": 2,
"elements": [
{
"start_timestamp": "2024-05-10T20:55:45.711Z",
"before": [
{
"result": {
"duration": 1404619000,
"status": "passed"
},
"match": {
"location": "webdemo.StepDefinitions.setUp(io.cucumber.java.Scenario)"
}
}
],
"line": 5,
"name": "Valid Logout",
"description": "",
"id": "as-a-user--i-can-logout-the-application;valid-logout",
"after": [
{
"result": {
"duration": 233798000,
"status": "passed"
},
"match": {
"location": "webdemo.StepDefinitions.tearDown(io.cucumber.java.Scenario)"
}
},
{
"output": [
"Finished scenario: Valid Logout"
],
"result": {
"duration": 423000,
"status": "passed"
},
"match": {
"location": "webdemo.StepDefinitions.afterScenario(io.cucumber.java.Scenario)"
}
}
],
"type": "scenario",
"keyword": "Scenario",
"steps": [
{
"result": {
"duration": 460327000,
"status": "passed"
},
"line": 6,
"name": "user is on the welcome page",
"match": {
"location": "webdemo.StepDefinitions.user_is_on_the_welcome_page()"
},
"keyword": "Given "
},
{
"result": {
"error_message": "org.openqa.selenium.NoSuchElementException: no such element: Unable to locate element: {\"method\":\"link text\",\"selector\":\"logout\"}\n (Session info: chrome-headless-shell\u003d124.0.6367.156)\nFor documentation on this error, please visit: https://selenium.dev/exceptions/#no_such_element\nBuild info: version: \u00274.0.0-beta-4\u0027, revision: \u002729f46d02dd\u0027\nSystem info: host: \u0027Sergios-MBP.lan\u0027, ip: \u00272001:818:ebff:be00:1484:bdf8:b397:fea6%en0\u0027, os.name: \u0027Mac OS X\u0027, os.arch: \u0027x86_64\u0027, os.version: \u002714.4.1\u0027, java.version: \u002721.0.2\u0027\nDriver info: org.openqa.selenium.chrome.ChromeDriver\nCommand: [483719b41c711baaf97e6e8038cc07e8, findElement {using\u003dlink text, value\u003dlogout}]\nCapabilities {acceptInsecureCerts: false, browserName: chrome-headless-shell, browserVersion: 124.0.6367.156, chrome: {chromedriverVersion: 123.0.6312.122 (31f8248cdd9..., userDataDir: /var/folders/2w/5zhst_816kl...}, fedcm:accounts: true, goog:chromeOptions: {debuggerAddress: localhost:64252}, javascriptEnabled: true, networkConnectionEnabled: false, pageLoadStrategy: normal, platform: MAC, platformName: MAC, proxy: Proxy(), se:cdp: ws://localhost:64252/devtoo..., se:cdpVersion: 124.0.6367.156, setWindowRect: true, strictFileInteractability: false, timeouts: {implicit: 0, pageLoad: 300000, script: 30000}, unhandledPromptBehavior: dismiss and notify, webauthn:extension:credBlob: true, webauthn:extension:largeBlob: true, webauthn:extension:minPinLength: true, webauthn:extension:prf: true, webauthn:virtualAuthenticators: true}\nSession ID: 483719b41c711baaf97e6e8038cc07e8\n\tat java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)\n\tat java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)\n\tat java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)\n\tat org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:200)\n\tat org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:133)\n\tat org.openqa.selenium.remote.codec.w3c.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:53)\n\tat org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:184)\n\tat org.openqa.selenium.remote.service.DriverCommandExecutor.invokeExecute(DriverCommandExecutor.java:162)\n\tat org.openqa.selenium.remote.service.DriverCommandExecutor.execute(DriverCommandExecutor.java:137)\n\tat org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:612)\n\tat org.openqa.selenium.remote.RemoteWebDriver$Mechanism$2.findElement(RemoteWebDriver.java:1251)\n\tat org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:379)\n\tat org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:373)\n\tat com.xray.tutorials.pages.WelcomePage.logout(WelcomePage.java:23)\n\tat webdemo.StepDefinitions.user_chooses_to_logout(StepDefinitions.java:47)\n\tat ✽.user chooses to logout(file:///Users/sergio/exps/tutorial-java-cucumber-selenium/features/1_XT-6.feature:7)\n",
"duration": 23477000,
"status": "failed"
},
"line": 7,
"name": "user chooses to logout",
"match": {
"location": "webdemo.StepDefinitions.user_chooses_to_logout()"
},
"keyword": "When "
},
{
"result": {
"duration": 33000,
"status": "skipped"
},
"line": 8,
"name": "login page should be open",
"match": {
"location": "webdemo.StepDefinitions.login_page_should_be_open()"
},
"keyword": "Then "
}
],
"tags": [
{
"name": "@REQ_XT-6"
},
{
"name": "@TEST_XT-10"
}
]
}
],
"name": "As a user, I can logout the application",
"description": "",
"id": "as-a-user--i-can-logout-the-application",
"keyword": "Feature",
"uri": "file:features/1_XT-6.feature",
"tags": [
{
"name": "@REQ_XT-6",
"type": "Tag",
"location": {
"line": 1,
"column": 1
}
}
]
},
{
"line": 2,
"elements": [
{
"start_timestamp": "2024-05-10T20:55:47.891Z",
"before": [
{
"result": {
"duration": 578606000,
"status": "passed"
},
"match": {
"location": "webdemo.StepDefinitions.setUp(io.cucumber.java.Scenario)"
}
}
],
"line": 5,
"name": "Valid Login",
"description": "",
"id": "as-a-user--i-can-login-the-application;valid-login",
"after": [
{
"result": {
"duration": 81443000,
"status": "passed"
},
"match": {
"location": "webdemo.StepDefinitions.tearDown(io.cucumber.java.Scenario)"
}
},
{
"output": [
"Finished scenario: Valid Login"
],
"result": {
"duration": 178000,
"status": "passed"
},
"match": {
"location": "webdemo.StepDefinitions.afterScenario(io.cucumber.java.Scenario)"
}
}
],
"type": "scenario",
"keyword": "Scenario",
"steps": [
{
"result": {
"duration": 491618000,
"status": "passed"
},
"line": 6,
"name": "browser is opened to login page",
"match": {
"location": "webdemo.StepDefinitions.browser_is_opened_to_login_page()"
},
"keyword": "Given "
},
{
"result": {
"duration": 363270000,
"status": "passed"
},
"line": 7,
"name": "user \"demo\" logs in with password \"mode\"",
"match": {
"arguments": [
{
"val": "\"demo\"",
"offset": 5
},
{
"val": "\"mode\"",
"offset": 34
}
],
"location": "webdemo.StepDefinitions.user_logs_in_with_password(java.lang.String,java.lang.String)"
},
"keyword": "When "
},
{
"result": {
"duration": 11501000,
"status": "passed"
},
"line": 8,
"name": "welcome page should be open",
"match": {
"location": "webdemo.StepDefinitions.welcome_page_should_be_open()"
},
"keyword": "Then "
}
],
"tags": [
{
"name": "@REQ_XT-5"
},
{
"name": "@TEST_XT-7"
}
]
},
{
"start_timestamp": "2024-05-10T20:55:49.425Z",
"before": [
{
"result": {
"duration": 597129000,
"status": "passed"
},
"match": {
"location": "webdemo.StepDefinitions.setUp(io.cucumber.java.Scenario)"
}
}
],
"line": 11,
"name": "Invalid Login",
"description": "",
"id": "as-a-user--i-can-login-the-application;invalid-login",
"after": [
{
"result": {
"duration": 81064000,
"status": "passed"
},
"match": {
"location": "webdemo.StepDefinitions.tearDown(io.cucumber.java.Scenario)"
}
},
{
"output": [
"Finished scenario: Invalid Login"
],
"result": {
"duration": 229000,
"status": "passed"
},
"match": {
"location": "webdemo.StepDefinitions.afterScenario(io.cucumber.java.Scenario)"
}
}
],
"type": "scenario",
"keyword": "Scenario",
"steps": [
{
"result": {
"duration": 453456000,
"status": "passed"
},
"line": 12,
"name": "browser is opened to login page",
"match": {
"location": "webdemo.StepDefinitions.browser_is_opened_to_login_page()"
},
"keyword": "Given "
},
{
"result": {
"duration": 427893000,
"status": "passed"
},
"line": 13,
"name": "user \"dummy\" logs in with password \"password\"",
"match": {
"arguments": [
{
"val": "\"dummy\"",
"offset": 5
},
{
"val": "\"password\"",
"offset": 35
}
],
"location": "webdemo.StepDefinitions.user_logs_in_with_password(java.lang.String,java.lang.String)"
},
"keyword": "When "
},
{
"result": {
"duration": 10580000,
"status": "passed"
},
"line": 14,
"name": "error page should be open",
"match": {
"location": "webdemo.StepDefinitions.error_page_should_be_open()"
},
"keyword": "Then "
}
],
"tags": [
{
"name": "@REQ_XT-5"
},
{
"name": "@TEST_XT-8"
}
]
},
{
"start_timestamp": "2024-05-10T20:55:50.998Z",
"before": [
{
"result": {
"duration": 580970000,
"status": "passed"
},
"match": {
"location": "webdemo.StepDefinitions.setUp(io.cucumber.java.Scenario)"
}
}
],
"line": 24,
"name": "Login With Invalid Credentials Should Fail",
"description": "",
"id": "as-a-user--i-can-login-the-application;login-with-invalid-credentials-should-fail;;2",
"after": [
{
"result": {
"duration": 72742000,
"status": "passed"
},
"match": {
"location": "webdemo.StepDefinitions.tearDown(io.cucumber.java.Scenario)"
}
},
{
"output": [
"Finished scenario: Login With Invalid Credentials Should Fail"
],
"result": {
"duration": 165000,
"status": "passed"
},
"match": {
"location": "webdemo.StepDefinitions.afterScenario(io.cucumber.java.Scenario)"
}
}
],
"type": "scenario",
"keyword": "Scenario Outline",
"steps": [
{
"result": {
"duration": 545656000,
"status": "passed"
},
"line": 18,
"name": "browser is opened to login page",
"match": {
"location": "webdemo.StepDefinitions.browser_is_opened_to_login_page()"
},
"keyword": "Given "
},
{
"result": {
"duration": 328858000,
"status": "passed"
},
"line": 19,
"name": "user \"invalid\" logs in with password \"mode\"",
"match": {
"arguments": [
{
"val": "\"invalid\"",
"offset": 5
},
{
"val": "\"mode\"",
"offset": 37
}
],
"location": "webdemo.StepDefinitions.user_logs_in_with_password(java.lang.String,java.lang.String)"
},
"keyword": "When "
},
{
"result": {
"duration": 8815000,
"status": "passed"
},
"line": 20,
"name": "error page should be open",
"match": {
"location": "webdemo.StepDefinitions.error_page_should_be_open()"
},
"keyword": "Then "
}
],
"tags": [
{
"name": "@REQ_XT-5"
},
{
"name": "@TEST_XT-9"
}
]
},
{
"start_timestamp": "2024-05-10T20:55:52.538Z",
"before": [
{
"result": {
"duration": 579073000,
"status": "passed"
},
"match": {
"location": "webdemo.StepDefinitions.setUp(io.cucumber.java.Scenario)"
}
}
],
"line": 25,
"name": "Login With Invalid Credentials Should Fail",
"description": "",
"id": "as-a-user--i-can-login-the-application;login-with-invalid-credentials-should-fail;;3",
"after": [
{
"result": {
"duration": 78805000,
"status": "passed"
},
"match": {
"location": "webdemo.StepDefinitions.tearDown(io.cucumber.java.Scenario)"
}
},
{
"output": [
"Finished scenario: Login With Invalid Credentials Should Fail"
],
"result": {
"duration": 142000,
"status": "passed"
},
"match": {
"location": "webdemo.StepDefinitions.afterScenario(io.cucumber.java.Scenario)"
}
}
],
"type": "scenario",
"keyword": "Scenario Outline",
"steps": [
{
"result": {
"duration": 526710000,
"status": "passed"
},
"line": 18,
"name": "browser is opened to login page",
"match": {
"location": "webdemo.StepDefinitions.browser_is_opened_to_login_page()"
},
"keyword": "Given "
},
{
"result": {
"duration": 362822000,
"status": "passed"
},
"line": 19,
"name": "user \"demo\" logs in with password \"invalid\"",
"match": {
"arguments": [
{
"val": "\"demo\"",
"offset": 5
},
{
"val": "\"invalid\"",
"offset": 34
}
],
"location": "webdemo.StepDefinitions.user_logs_in_with_password(java.lang.String,java.lang.String)"
},
"keyword": "When "
},
{
"result": {
"duration": 9527000,
"status": "passed"
},
"line": 20,
"name": "error page should be open",
"match": {
"location": "webdemo.StepDefinitions.error_page_should_be_open()"
},
"keyword": "Then "
}
],
"tags": [
{
"name": "@REQ_XT-5"
},
{
"name": "@TEST_XT-9"
}
]
},
{
"start_timestamp": "2024-05-10T20:55:54.098Z",
"before": [
{
"result": {
"duration": 593300000,
"status": "passed"
},
"match": {
"location": "webdemo.StepDefinitions.setUp(io.cucumber.java.Scenario)"
}
}
],
"line": 26,
"name": "Login With Invalid Credentials Should Fail",
"description": "",
"id": "as-a-user--i-can-login-the-application;login-with-invalid-credentials-should-fail;;4",
"after": [
{
"result": {
"duration": 77271000,
"status": "passed"
},
"match": {
"location": "webdemo.StepDefinitions.tearDown(io.cucumber.java.Scenario)"
}
},
{
"output": [
"Finished scenario: Login With Invalid Credentials Should Fail"
],
"result": {
"duration": 145000,
"status": "passed"
},
"match": {
"location": "webdemo.StepDefinitions.afterScenario(io.cucumber.java.Scenario)"
}
}
],
"type": "scenario",
"keyword": "Scenario Outline",
"steps": [
{
"result": {
"duration": 879007000,
"status": "passed"
},
"line": 18,
"name": "browser is opened to login page",
"match": {
"location": "webdemo.StepDefinitions.browser_is_opened_to_login_page()"
},
"keyword": "Given "
},
{
"result": {
"duration": 346027000,
"status": "passed"
},
"line": 19,
"name": "user \"invalid\" logs in with password \"invalid\"",
"match": {
"arguments": [
{
"val": "\"invalid\"",
"offset": 5
},
{
"val": "\"invalid\"",
"offset": 37
}
],
"location": "webdemo.StepDefinitions.user_logs_in_with_password(java.lang.String,java.lang.String)"
},
"keyword": "When "
},
{
"result": {
"duration": 9807000,
"status": "passed"
},
"line": 20,
"name": "error page should be open",
"match": {
"location": "webdemo.StepDefinitions.error_page_should_be_open()"
},
"keyword": "Then "
}
],
"tags": [
{
"name": "@REQ_XT-5"
},
{
"name": "@TEST_XT-9"
}
]
},
{
"start_timestamp": "2024-05-10T20:55:56.006Z",
"before": [
{
"result": {
"duration": 599761000,
"status": "passed"
},
"match": {
"location": "webdemo.StepDefinitions.setUp(io.cucumber.java.Scenario)"
}
}
],
"line": 27,
"name": "Login With Invalid Credentials Should Fail",
"description": "",
"id": "as-a-user--i-can-login-the-application;login-with-invalid-credentials-should-fail;;5",
"after": [
{
"result": {
"duration": 229111000,
"status": "passed"
},
"match": {
"location": "webdemo.StepDefinitions.tearDown(io.cucumber.java.Scenario)"
}
},
{
"output": [
"Finished scenario: Login With Invalid Credentials Should Fail"
],
"result": {
"duration": 184000,
"status": "passed"
},
"match": {
"location": "webdemo.StepDefinitions.afterScenario(io.cucumber.java.Scenario)"
}
}
],
"type": "scenario",
"keyword": "Scenario Outline",
"steps": [
{
"result": {
"duration": 425929000,
"status": "passed"
},
"line": 18,
"name": "browser is opened to login page",
"match": {
"location": "webdemo.StepDefinitions.browser_is_opened_to_login_page()"
},
"keyword": "Given "
},
{
"result": {
"duration": 377786000,
"status": "passed"
},
"line": 19,
"name": "user \"demo\" logs in with password \"mode\"",
"match": {
"arguments": [
{
"val": "\"demo\"",
"offset": 5
},
{
"val": "\"mode\"",
"offset": 34
}
],
"location": "webdemo.StepDefinitions.user_logs_in_with_password(java.lang.String,java.lang.String)"
},
"keyword": "When "
},
{
"result": {
"error_message": "org.junit.ComparisonFailure: expected:\u003c[Welcome] Page\u003e but was:\u003c[Error] Page\u003e\n\tat org.junit.Assert.assertEquals(Assert.java:117)\n\tat org.junit.Assert.assertEquals(Assert.java:146)\n\tat webdemo.StepDefinitions.error_page_should_be_open(StepDefinitions.java:71)\n\tat ✽.error page should be open(file:///Users/sergio/exps/tutorial-java-cucumber-selenium/features/2_XT-5.feature:20)\n",
"duration": 4803000,
"status": "failed"
},
"line": 20,
"name": "error page should be open",
"match": {
"location": "webdemo.StepDefinitions.error_page_should_be_open()"
},
"keyword": "Then "
}
],
"tags": [
{
"name": "@REQ_XT-5"
},
{
"name": "@TEST_XT-9"
}
]
}
],
"name": "As a user, I can login the application",
"description": "",
"id": "as-a-user--i-can-login-the-application",
"keyword": "Feature",
"uri": "file:features/2_XT-5.feature",
"tags": [
{
"name": "@REQ_XT-5",
"type": "Tag",
"location": {
"line": 1,
"column": 1
}
}
]
}
]
Of course, you could somehow shortcut the flows that I mentioned and create some dummy Cucumber Tests in Xray, and then annotate locally your .feature file, and add there the references to (optionaly) the Story and the Test in Jira. Then whenever you run it, the report will have those references.. so that if you import it, the results will be linked correct against the issues in Jira/Xray.
# CALC-7931 the issue key of the corresponding Story to link the tests to
@REQ_CALC-7931
Feature: As a user, I can add two numbers
# CALC-1234 the issue key of the corresponding Cucumber Test issue to report results against
@TEST_CALC-1234
Scenario: simple integer addition
Given I have entered 1 into the calculator
And I have entered 2 into the calculator
When I press add
Then the result should be 3 on the screen
Upvotes: 0
Related Questions
- Error in Xray Rest API Request to execution cucumber Json result test in Jira
- Jira/xRay - Embedded element in JSON-Report has the wrong name in Jira
- Cucumber report attachment - json format - XRay Jira API
- Custom field for test plan in jira xray
- Import cucumber results to Jira/Xray due the curl
- How to send test information within Test Execution in Jira Xray?
- Import Cucumber results to Jira/Xray
- How to import test execution results using REST API and XRay for JIRA and Cucumber + Xray
- Update Cucumber features and test execution results on Xray in Jira
- Node.js: Sending Cucumber JSON results to Jira's Xray - jira-client-xray gives HTTP 405 error