CODEWITHSUNDEEP
linuxulimit
Pavel Komarov
Pavel Komarov

Reputation: 1246

Raise number of files a process can open beyond 2^20

I am running a python process that opens a lot of memory-mapped files at once. Currently it fails with an OSError because it tries to open more than the system wants to allow. I need to set the ulimit -n to at least 4.6million, but doing so causes

bash: ulimit: open files: cannot modify limit: Operation not permitted.

But I am root! 😭 How could this not be allowed?

However, if I attempt to set the limit to anything less than 2^20, it works without a problem. Same thing inside python using the resource package: Below 2^20 works; over 2^20 says ValueError: not allowed to raise maximum limit.

I've edited /proc/sys/fs/file-max to be 100million. I've edited /etc/security/limits.conf to have nofile limits of 100million. I've rebooted. I've edited DefaultLimitNOFILE in /etc/systemd/user.conf and /etc/systemd/system.conf to be 100million. I've rebooted again. I've edited /etc/pam.d/common-session to have the line session required pam_limits.so. I've rebooted again. No dice.

How can I raise the limit? There is no physical barrier here that makes it impossible. This machine is extremely powerful and should be able to manage that many open file handles. According to this the hard-coded limit in the kernel is about 4billion.

Upvotes: 3

Views: 1984

Answers (1)

Pavel Komarov
Pavel Komarov

Reputation: 1246

Okay, I got it. You can't just set fs.file-max. You also have to set fs.nr_open, which has a default value of 2^20. I also removed the /etc/pam.d/common-session I created and commented out the session required pam_limits.so line in /etc/pam.d/sudo.

Upvotes: 2

Related Questions