CODEWITHSUNDEEP
c++performanceunixnetwork-programming
Anerudhan Gopal
Anerudhan Gopal

Reputation: 389

Select v/s Multithreading in an Multicore Environment

I am writing an application which works on 3 tiers Each having multiple instances and forming a mesh like relationship.

Eg. Let us consider a Matrix type situation:

                     L11 L12

                 L21 L22 L23 L24

                    L31 L32 L33

L11 has connections with L21,L22,L23,L24
L21 has connections with L21,L22,L23,L24
L21 has connection with L11,L12,L31,L32,L33... and so on

Presently I am using UNIX domain sockets for communication with select call(to determine where the data has come from). How does a timeout on select call affect CPU usage??

Will it be better to spawn a new thread per connection or persist with the select model. Typically the horizontal scalability at L1 may be around 5, L2 will be around 5 and L3 around 10.

Is there a penalty for select system call if the bit fields passed to select is high? In case of Multi threaded application whether to go with a Blocking system socket or Non-Blocking?

Is there a difference between UDP/TCP in UNIX-Domain socket??

Note: I will be using systems of 12-24 cores ... Probably this is what made me think why not use separate threads and not select call ..

Note 2: Will there be buffer overflows in Unix Domain Sockets ?? As far as i read these are lossless and inorder ...

Upvotes: 1

Views: 1279

Answers (1)

Jeremy Friesner
Jeremy Friesner

Reputation: 73091

I am writing an application which works on 3 tiers Each having multiple instances and forming a mesh like relationship.

Are all of the nodes in your Matrix running on the same host, or are they running on multiple hosts across a network?

How does a timeout on select call affect CPU usage??

Only in the obvious way, in that every time your select() call times out you're going to have to go for another (otherwise maybe avoidable) spin around your event loop. So, for example always passing a timeout of zero (or near-zero) would be bad because then your thread would be wasting CPU cycles busy-looping. But the occasional timeout shouldn't be a problem. My recommendation is to be purely event-based as much as possible, and only use timeouts when they are unavoidable (e.g. because such-and-such an event has to happen at a particular time, unrelated to any I/O traffic).

Will it be better to spawn a new thread per connection or persist with the select model. Typically the horizontal scalability at L1 may be around 5, L2 will be around 5 and L3 around 10.

For this scale of operation (dozens of connections but not hundreds) select() will work fine. It's only when you get up into hundreds or thousands of connections that select() starts to fall down, and even then you could move to something like kqueue() or epoll() rather than going multithreaded.

Is there a penalty for select system call if the bit fields passed to select is high?

I'm not 100% sure what you mean by "bit fields is high", but if you mean "there are lots of sockets specified", then the main penalty is that select() will only work with file descriptors whose integer values are less than FD_SETSIZE (typically 1024) -- which means by implication that select() also can't track more than FD_SETSIZE sockets at a time. (Exception: Under Windows, select() will handle sockets with arbitrary integer values, although it still won't handle more than FD_SETSIZE sockets at a time)

In case of Multi threaded application whether to go with a Blocking system socket or Non-Blocking?

If it were me, I would still use non-blocking I/O even in a multithreaded application, because blocking I/O makes certain things like a clean shutdown of the application problematic. For example, if you have a thread that is blocked inside recv(), how do you safely get that thread to exit so that you can clean up any shared resources? You can't send a signal because you don't know which thread would receive it, and you can't rely on recv() returning naturally within a finite amount of time either. On the other hand, if your target thread only ever blocks inside select(), you can send the thread a byte on one of the FDs it is select()'ing on to wake it up and tell it go away.

Is there a difference between UDP/TCP in UNIX-Domain socket??

Unix-Domain sockets don't use UDP or TCP (since they never go across the network). They do use SOCK_STREAM and SOCK_DGRAM, though, which behave similarly (in most respects) to using TCP and UDP sockets to localhost.

Note: I will be using systems of 12-24 cores ... Probably this is what made me think why not use separate threads and not select call ..

The question to ask yourself is, is your application likely to be compute-bound or I/O bound? If it's compute-bound (and the computations can be parallelized) you might benefit from farming out the computations to multiple threads. If it's I/O bound, OTOH, there won't be any advantage to going multithreaded, because in that case the bottleneck is likely to be your network card (and/or the network itself), and your gigabit-Ethernet jack is still going to be sending a maximum of 1Gb/sec whether it is fed by one thread or many. If all your communication is with other processes on the same host, OTOH, there might be some benefit to multiple threads, but my guess is that it would be minimal, since it looks like you will already be using all of your cores for the dozen or so processes that you have.

Note 2: Will there be buffer overflows in Unix Domain Sockets ?? As far as i read these are lossless and inorder ...

There shouldn't be. SOCK_STREAM Unix-domain sockets work very much like TCP sockets in my experience. (I've never used SOCK_DGRAM Unix-domain sockets, but I would imagine they'd behave similarly to UDP sockets, and would even drop packets sometimes e.g. when the receiving process can't keep up with the sender)

Upvotes: 2

Related Questions