How to query for flakey tests using GitLab's REST or GraphQL API

Question

I'm trying to query our instance of GitLab for flakey tests.

If I navigate to a URL of this format, https://our-gitlab-instance.com/the-group/the-project/-/pipelines/365771/tests/suite.json?build_ids[]=1671439, we get some nice JSON which lists stats about the tests.

I'm specifically looking for flakey test cases which looks something like this:

{
  "status": "success",
  "name": "The name of the test",
  "classname": "the/path/to/the/test.pw.ts",
  "file": null,
  "execution_time": 77.511,
  "system_output": null,
  "stack_trace": null,
  "recent_failures": {
      "count": 18,
      "base_branch": "develop"
  },
},

Initially, I was trying to use GraphQL:

query {
  project(fullPath: "path_to/my_project") {
    pipelines {
      nodes {
        testSuite {
          testCases {
            nodes {
              recentFailures {
                baseBranch
                count
              }
            }
          }
        }
      }
    }
  }
}

GraphQL seems to provide the right fields, however, testSuite requires an argument buildIds of type [ID!]!.

I can get the data I need with this query:

query {
  project(fullPath: "path_to/my_project") {
    pipeline(iid: "261264") {
      testSuite(buildIds: "1671439") {
        testCases {
          nodes {
            recentFailures {
              count
              baseBranch
            }
          }
        }
      }
    }
  }
}

But this involves creating a separate query for each job. As I'm wanting to enumerate over all builds over a given timeframe, this seems like a lot.

I have not yet found any REST endpoint that provides this information.

Are there any other options I haven't considered?

Answer 1

I have found a solution that involves three API calls. This solution is good example of an N + 1 problem.

1. Fetch each failed pipeline:

GET :gitlab_url/api/v4/projects/:project_id/pipelines?status=failed&per_page=:per_page&page=:page&created_after=:created_after&source=parent_pipeline

This is an authenticated request that requires the PRIVATE-TOKEN header to be set to an access token with the required permissions.

This URL has the following params:

• :gitlab_url - the URL of your GitLab instance
• :project_id - the id of the project you are interested in
• status=failed - only return failed pipelines
• per_page=:per_page - the number of results to return (defaults to 20)
• page=:page - the page to fetch
• created_after=:created_after - return records after this date
• source=parent_pipeline - by default, child pipelines are not included in the results. To return child pipelines, set source to parent_pipeline. I'm not yet sure if you need to query twice; once with the source params and once without

2. Iterate over each pipeline, and it's jobs:

GET GET :gitlab_url/api/v4/projects/:project_id/pipelines?scope[]=failed&include_retried=true

This is an authenticated request that requires the PRIVATE-TOKEN header to be set to an access token with the required permissions.

This URL has the following params:

• :gitlab_url - the URL of your GitLab instance
• :project_id - the id of the project you are interested in
• scope[]=failed - only return failed jobs
• include_retried=true - include retried jobs in the response. Defaults to false.

3. Fetch test cases for each job:

POST :gitlab_url/api/graphql

This is an authenticated request that requires the Authorization header to be set to an access token with the required permissions of the format Bearer :your_access_token.

Additionally, include the header Content-Type with value application/json.

This URL has the following params:

• :gitlab_url - the URL of your GitLab instance

Set the body to the following:

query {
  project(fullPath: "productdev/insight") {
    pipeline(iid: ":pipeline_iid") {
      testSuite(buildIds: ":job_id") {
        testCases {
          nodes {
            name
            status
            classname
            recentFailures {
              count
              baseBranch
            }
            file
            executionTime
            systemOutput
            stackTrace
            attachmentUrl
          }
        }
      }
    }
  }
}

This query has the following params:

• :pipeline_iid - This is the iid for the pipeline. this is not a spelling mistake; it is NOT the id.
• job_id - The id of the job

This third query returns the JSON I am expecting:

{
  "status": "success",
  "name": "The name of the test",
  "classname": "the/path/to/the/test.pw.ts",
  "file": null,
  "execution_time": 77.511,
  "system_output": null,
  "stack_trace": null,
  "recent_failures": {
      "count": 18,
      "base_branch": "develop"
  },
},