Performance info for each commit in GitHub or GitLab CI

Question

I am using GitLab to host a project where performance is the main focus. I currently have tests which assert that certain tasks finish in given time.

I'd like to expand my CI configuration to measure how a particular commit or pull request affects performance and have the information available in a convenient format (charts, performance change by commit, etc.).

Basically I want the equivalent of coveralls.io except for performance instead of coverage.

I'm interested in all solutions which work on GitHub or GitLab.

Please note that this question is not about how to do performance testing in CI, it's about presenting the obtained data in a convenient way.

Answer 1

GitHub or GitLab are only source control hosting services, my expectation is that you need to have a piece of software which will orchestrate the build, to wit:

1. Obtain the source code via commit hook
2. Build the source code
3. Run tests
4. Publish artifacts

This is normally done by a CI server and if you don't have one - you will need to choose it and if you don't have a better candidate - I'd recommend going for Jenkins which is some form of an industry standard.

Jenkins in its turn has Performance Plugin which can build performance trends charts for the following formats:

• JMeter XML and CSV format, also Summarizer log output
• Taurus Tool Final Stats XML (through it, you can publish summaries from JMeter, Gatling, Grinder, Siege, ab, Selenium and many others)
• JUnit format (used by SoapUI for example)
• Twitter's Iago
• wrk

So if your tests produce results in one of the above formats you will be able to have charts like:

In addition you will be able to conditionally mark your build as failed depending on different criteria (% of errors, response time exceeds thresholds, current build result is worse than previous, etc.)