Raku `precomp` compilation error and how to overcome?

Question

I have a medium sized Raku project that all of a sudden started to throw the following error on startup:

===SORRY!=== Error while compiling /home/patrickb/repos/RakudoCIBot/service.raku
===SORRY!=== Error while compiling /home/patrickb/repos/RakudoCIBot/lib/RakudoCIBot.rakumod (RakudoCIBot)
Now hiting the error and sleeping
===SORRY!=== Error while compiling /home/patrickb/repos/RakudoCIBot/lib/Routes.rakumod (Routes)
Earlier failure:
 Failed to open file /home/patrickb/.raku/precomp/F494CC98E40B399BDACD0E2436C176FDE8706DE8/28/287E340591A8C5DE2625947EEC5BEFDF29E8EA4F: Too many open files
  in any statement_control at /home/patrickb/rrepos/install/share/perl6/lib/Perl6/Grammar.moarvm line 1

Final error:
 Type check failed in binding; expected IO::Handle but got Failure (Failure.new(exceptio...)
at /home/patrickb/repos/RakudoCIBot/lib/Routes.rakumod (Routes):10

at /home/patrickb/repos/RakudoCIBot/lib/RakudoCIBot.rakumod (RakudoCIBot):15

at /home/patrickb/repos/RakudoCIBot/service.raku:3

Deleting the /home/patrickb/repos/RakudoCIBot/.precomp/ folder makes it start again successfully once. But that's neither a solution, nor an acceptable workaround.

What's happening and what can I do to fix this?

Answer 1

_{The people in the #raku IRC channel were super helpful in solving this.}

Stopping the process at the point it errors (I compiled my own Rakudo and put a sleep right before where the error message is printed.) and looking at /proc/$PID/fd of one of the rakudo compiler processes showed that it has exactly 1024 files open (ls -1 /proc/$PID/fd | wc -l). Most of those files look as follows:

lr-x------. 1 patrickb patrickb 64 Jun 13 17:20 965 -> /home/patrickb/.raku/precomp/F494CC98E40B399BDACD0E2436C176FDE8706DE8/EA/EA68508E655FD3F22D1AE0A8666B86469073473

So it's the compiler that has lots of precompiled files open. That's actually quite plausible, as the dependency trees can easily comprise > 1000 files in non trivial projects with one or two larger dependencies. (In my case there is Red and Cro in the deps.)

ulimit -Sn prints 1024. So I have a rather low limit set. Calling ulimit -n 4096 before running my application makes it work.

Considering all of the above, the clean solution seems to be to add the following to /etc/security/limits.conf:

# Increase nofile soft from the default of 1024.
# The Rakudo compiler sometimes hits the limit.
*               soft    nofile          4096