CODEWITHSUNDEEP
gitlabgitlab-ci-runner
Bart
Bart

Reputation: 1957

gitlab runner errors occasionally

I have gitlab setup with runners on dedicated VM machine (24GB 12 vCPUs and very low runner concurrency=6).

Everything worked fine until I've added more Browser tests - 11 at the moment. These tests are in stage browser-test and start properly.

My problem is that, it sometimes succeeds and sometimes not, with totally random errors.

Sometimes it cannot resolve host, other times unable to find element on page..

If I rerun these failed tests, all goes green always.

Anyone has an idea on what is going wrong here?

BTW... I've checked, this dedicated VM is not overloaded...

Upvotes: 2

Views: 370

Answers (1)

Bart
Bart

Reputation: 1957

I have resolved all my initial issues (not tested with full machine load so far), however, I've decided to post some of my experiences.

First of all, I was experimenting with gitlab-runner concurrency (to speed things up) and it turned out, that it really quickly filled my storage space. So for anybody experiencing storage shortcomings, I suggest installing this package

Secondly, I was using runner cache and artifacts, which in the end were cluttering my tests a bit, and I believe, that was the root cause of my problems.

My observations:

  1. If you want to take advantage of cache in gitlab-runner, remember that by default it is accessible on host where runner starts only, and remember that cache is retrieved on top of your installation, meaning it overrides files from your project.
  2. Artifacts are a little bit more flexible, cause they are stored/fetched from your gitlab installation. You should develop your own naming convention (using vars) for them to control, what is fetched/cached between stages and to make sure all is working, as you would expect.
  3. Cache/Artifacts in your tests should be used with caution and understanding, cause they can introduce tons of problems, if not used properly...

Side note:

Although my VM machine was not overloaded, certain lags in storage were causing timeouts in the network and finally in Dusk, when running multiple gitlab-runners concurrently...

Update as of 2019-02:

Finally, I have tested this on a full load, and I can confirm my earlier side note, about machine overload is more than true.

After tweaking Linux parameters to handle big load (max open files, connections, sockets, timeouts, etc.) on hosts running gitlab-runners, all concurrent tests are passing green, without any strange, occasional errors.

Hope it helps anybody with configuring gitlab-runners...

Upvotes: 1

Related Questions