Not enough space when running Hudson jobs

Question

My Hudson jobs are crashing on each run with this error:

    Caused by: java.io.IOException: error=12, Not enough space
     at java.lang.UNIXProcess.forkAndExec(Native Method)

I found documention on StackOverflow and on the Jenkins website regarding this error, which indicate a problem of swap space (https://wiki.jenkins-ci.org/display/JENKINS/IOException+Not+enough+space).

However, maybe my problem is different or not, but if I launch the process manually it works fine. A weird thing is I see different resuls from top of from prstat:

Specs:

• Hudson processes are running in their own Unix user
• OS: SunOS dc5c00-d12 5.10 Generic_147440-19 sun4v sparc sun4v

• Memory:

  • from top:

    32G phys mem, 6255M free mem, 16G total swap, 16G free swap
    

  • from prstat

    NPROC USERNAME  SWAP   RSS MEMORY      TIME  CPU
    50 user1      12G   12G    39%  89:02:31 0.3%
    36 user2      11G 6779M    21% 155:17:41 0.0%
    26 user3      10G 8509M    26% 4787:37:4 8.0%
     6 hudson    572M  556M   1.7%   0:08:25 0.0%
    57 root      280M  285M   0.9% 138:46:05 0.0%
    

Can anywone confirm if I have a swap issue? top shows 16GB free...

EDIT: results from swap -s (after problem being remporarly resolved)

    total: 19940168k bytes allocated + 12578048k reserved = 32518216k used, 4118208k available

.

Answer 1

It is certainly a swap issue.

top is reporting as free swap blocks that do not contain paginated data. However, even while unused, some of these blocks can be reserved (i.e untouched still allocated virtual memory). When you have no more blocks to back memory reservations, you got this "Not enough space" exception.

swap -s shows your applications are reserving more that 12 GB while your swap area is just 16 GB. I would double the size of your swap to prevent virtual memory shortage in your case.