CODEWITHSUNDEEP
amazon-web-servicesamazon-ebsamazon-efs
sorin
sorin

Reputation: 170598

Can I improve performance of a build server by switching from EBS to EFS storage?

I am wondering if I can obtain serious improvements in performance by reconfiguring our Jenkins build server to use EFS (AWS NFS implementation) instead of EBS disks.

While EFS is about 3x more expensive per gigabyte the real cost is probably going to be only around 2x more expensive because on EFS you pay only for used space, as opposed to EBS where you pay all of it.

Also EFS has another very important advantage: it does scale without having to take anything down for upgrade. Resizing EBS disk is a time consuming operations that involves downtimes.

This question is not about the cost, is more about performance as if I can improve the build speed even by 20% the storage cost would clearly be overcomed (not to mention the advantage of needing less maintenance later).

Upvotes: 3

Views: 2721

Answers (2)

Steven Huwig
Steven Huwig

Reputation: 20794

From my direct experience, this is a very bad idea for a Jenkins server. We thought to save ourselves the administrative and automation overheads of creating, expanding, and otherwise managing EBS volumes, so we put our Jenkins home on an EFS mount.

The trouble is that Jenkins builds often involve lots of tiny files (for example, Javascript npm modules), which are the worst-case scenario for EFS, and indeed any NFS implementation. File-based storage requires server round-trips for each file access. In our specific case, cleaning out workspaces of even small projects can take several minutes on a Jenkins server with its home directory on EFS.

Save yourself the trouble, learn from our mistakes; we are going to undo this choice. Your Jenkins server will almost certainly be much slower than one based on EBS.

Upvotes: 12

sorin
sorin

Reputation: 170598

Here are my intermediary results of my attempt of using AWS EFS for storing Jenkins home directory (which includes the workspaces).

My mistake was that I missed this well hidden page about EFS performance which I would summarize that unless you want to store huge amount of data on EFS, it can burst only for 0.5% of the day.... where burst is what we would all expect as the normal performance.

It seems the EFS is not only damn slow, it is extremely slow, so slow that I failed to do an rsync of only 8GB of data from the local EBS volume to the EFS one.

root@hostname:/efs# time rsync -ah --info=progress2 /jenkins/ /efs
        816.72M   6%  609.02kB/s    0:21:49 (xfr#12490, ir-chk=1009/273305)
          2.71G  18%  871.55kB/s    0:50:40 (xfr#42955, ir-chk=1070/306870)

The average speed was around 1.5mB/s which is ridiculous.

Due to this I decided not even to test running a jenkins build job jenkins on it.

I tried to see if this was caused by AWS that was limiting my speed but the monitoring on EFS does not point that's the case. I think that's the expected performance that you may get if you have to handle lots of small files. Have a look at the screenshot: enter image description here

Upvotes: 3

Related Questions