CODEWITHSUNDEEP
asp.netcontinuous-integrationteamcity
Massimiliano Peluso
Massimiliano Peluso

Reputation: 26737

Continuous integration and Automated testing strategy

I’m working in a company that uses Continuous Integration (TeamCity). Every time someone does a check in the CI software starts a build and runs all the Unit/Automated Test . The problem is that we got more than 7000 unit tests + 756 automated tests (used to test the JavaScript as we got a very complex UI logic for making calculation etc). As you can imaging every time someone does a check in the all process takes more than 2 hours to go through all the steps (build-unitest-automated test) so that I need to wait that much before I can get a result to understand if my check in has broken perhaps an automated test or a unit test. Worst situation is when more than one people check in something so that TeamCity start queue up the build and before I can get a valid result (udated) I can wait up to half day ! what strategy should we adopt to speed up a bit this process? Is it a best practice run all the automated tests even against a little change?

Upvotes: 4

Views: 1125

Answers (4)

KIR
KIR

Reputation: 5712

Have you considered using pre-tested commits? If you run a remote run build (without committing in to VCS), you can be sure that you didn't break anything in VCS (just because you didn't commit yet). And you can continue working without problems. If the build is successful, you can commit your change (even if you made some other changes in the same files) - if you commit via TeamCity's plugin it will commit exactly the code you sent to the server for the testing.

This way, you don't have to wait until TeamCity's build has finished to continue working with it.

Disclaimer: I'm a TeamCity developer.

Upvotes: 0

Chris Marisic
Chris Marisic

Reputation: 33128

I'm going to talk in theory, I have yet to put it into practice but CI is on my goals to have up and humming by the end of the summer.

From statements I've seen made by the people that have earned the most respect from me in developers the most common element for CI I've seen in regards to the testing strategy is to split your tests into Long Running and Short Running.

Then you would want to configure that standard check ins kick off the short running test for basic validation of the solution. Then on the nightly builds, and for deployment builds is the only time you NEED to run the full test suite to give your validation of regression tests.

Aside/Alternate answer: Seeing as I haven't setup CI for myself yet, I had never understood the TeamCity business model that they making the pricing based on build agents. Now I understand why multiple build agents really start to matter if your test suite takes that long, being able to run 5 builds at once becomes much more important. So one option could be to just spend more money and stick a band-aid on the bullet hole for now.

Upvotes: 1

thekbb
thekbb

Reputation: 7934

I would look at breaking up your test suite in two ways - with the goal of making it so you and your team can check in, go get a cup of coffee and have some meaningful feedback from team city when you get back to your desk.

  1. decide what you really want to test on every commit, move the remaining tests to a suite that runs at a scheduled interval (hourly, nightly - whatever works for you).

  2. If the set of tests agreed upon to run every commit is still large - break that set up and distribute across multiple nodes running in parallel.

You may also want to beef up your CI machine, depending on the nature of your stuff have the working directory for the tests live in tmpfs (RAM disk).

Upvotes: 3

Dennis Traub
Dennis Traub

Reputation: 51694

Continuous Integration works best with a distributed version control system like Git or Mercurial.

Every developer can check in often into their local repository without triggering the whole integration and UI testing ceremony all the time.

Once a feature is finished locally, it can be checked in to the central repository. Thus the CI server runs all the time-consuming tests only when new features and/or fixes have been added.

Upvotes: 0

Related Questions