CODEWITHSUNDEEP
c
Lostsoul
Lostsoul

Reputation: 25999

using pthreads as a thread pool with queue?

I'm very new to C so sorry if this question is very off. I have used Java executor service to make fixed thread pools and was having trouble understanding how to do something similar in C.

I learned how to create threads in c using pthreads(which seems easy enough) but I'm unsure how to create a queue that a fixed number of threads consume? All the tutorials I have done so far either start the threads in their main statement on their own or they do it within a for loop. If I do this approach then I'll have millions of threads(1 for each work item) when all I want is 3 or 4 and have them processing a queue.

Is this possible and if so what do I need to learn? If its not possible with pthreads, then I'm happy to use something else, I am developing on a mac and going to deploy it on linux.

Upvotes: 2

Views: 1414

Answers (1)

paxdiablo
paxdiablo

Reputation: 881403

You can do this with a uniproducer/multiconsumer model reasonably easily by using condition variables. Keep in mind this is one architecture, others are certainly possible.

In the main thread, you simply create the queue, mutex and condition variable, then start up as many threads as you want running, pseudo-code such as:

glbQueue = []
glbMutex = new mutex
glbCondVar = new condvar
for i = 1 to 10:
    start thread using thrdFn

Next step is to add whatever workitems you need to the queue (using the mutex) and kicking the condition variable to wake up threads as needed:

while workitem = getNextWorkItem():
    lock glbMutex
    glbQueue.append (workItem)
    kick glbCondVar
    unlock glbMutex

Once all work items are done, you wait for the queue to empty, then you post some sentinel items to shut down the threads then wait for them to finish before exiting.

lock glbMutex
while glbQueue is not empty:
    kick glbCondVar
    unlock glbMutex.
    sleep for a bit
    lock glbMutex
unlock glbMutex.

for i = 1 to 10:
    lock glbMutex
    glbQueue.append (endWorkItem)
    kick glbCondVar
    unlock glbMutex.
    wait for any one thread to exit
exit

The threads that do the work are also relatively simple. First, they run in an infinite loop waiting for the condition variable to be kicked. Within that loop, they process work items until no more are available, then they go back to sleep.

Once the end work item has been received by a thread, it exits, guaranteeing that each thread gets one end item.

In other words, something like:

initialise
stillGoing = true
lock glbMutex
while stillGoing:
    wait on glbCondVar using glbMutex
    while stillGoing and glbQueue is not empty:
        extract workItem from glbQueue to thread local storage
        unlock glbMutex.
        if workItem is endWorkItem:
            stillGoing = false
        else:
            do the work specified by workItem
        lock glbMutex
unlock glbMutex
clean up
exit thread

That basically allows you to have a fixed number of threads processing items on the queue and the queue itself is protected by the mutex so that there's no contention between the worker threads or the main thread.

Upvotes: 2

Related Questions