CODEWITHSUNDEEP
pythoncentoscentos5
Z Jones
Z Jones

Reputation: 2035

How do you compile python with >1024 file descriptors?

The workaround for an issue in supervisord is to: "compile a Python that supports > 1024 file descriptors"

https://github.com/Supervisor/supervisor/issues/26

Can someone please walk me through what changes are necessary to accomplish this? I have the python 2.7.2 source extracted and ready to go.

Running centos 5.6, if that matters.

Thanks.

Update: ulimit -n is already set to 65535. This is the full error I'm getting when starting supervisord:

Traceback (most recent call last): File "/usr/local/bin/supervisord", line 8, in load_entry_point('supervisor==3.0a10', 'console_scripts', 'supervisord')() File "/usr/local/lib/python2.7/site-packages/supervisor-3.0a10-py2.7.egg/supervisor/supervisord.py", line 372, in main go(options) File "/usr/local/lib/python2.7/site-packages/supervisor-3.0a10-py2.7.egg/supervisor/supervisord.py", line 382, in go d.main()
File "/usr/local/lib/python2.7/site-packages/supervisor-3.0a10-py2.7.egg/supervisor/supervisord.py", line 95, in main self.run()
File "/usr/local/lib/python2.7/site-packages/supervisor-3.0a10-py2.7.egg/supervisor/supervisord.py", line 112, in run self.runforever()
File "/usr/local/lib/python2.7/site-packages/supervisor-3.0a10-py2.7.egg/supervisor/supervisord.py", line 230, in runforever r, w, x = self.options.select(r, w, x, timeout)
File "/usr/local/lib/python2.7/site-packages/supervisor-3.0a10-py2.7.egg/supervisor/options.py", line 1113, in select return select.select(r, w, x, timeout) ValueError: filedescriptor out of range in select()

Upvotes: 1

Views: 2124

Answers (1)

Keith
Keith

Reputation: 43034

That's actually the limit of the underlying select(2) system call.

From the man page:

An fd_set is a fixed size buffer.  Executing FD_CLR() or FD_SET() with a value of fd 
that  is  negative  or  is equal  to  or  larger  than  FD_SETSIZE will result in 
undefined behavior.

And the standard FD_SETSIZE is 1024.

/usr/include/linux/posix_types.h:#define __FD_SETSIZE   1024

So it's not a Python issue. The poll(2) and epoll(2) system calls have a much larger limit. What you really need to do use use the select.epoll object (still in the select module) instead of `select.

Upvotes: 4

Related Questions