Unexpected semaphore behavior, threads executing beyond sem_wait()

Question

I have been trying to wrap my head around the concept of semaphores in C, and I have had some limited success. From what I understand, in C, if the value of a semaphore is 0, a sem_wait() should result in that thread blocking until the value of that semaphore is no longer 0.

My problem is this: I have written some very quick example code (below), and I'm not sure why, but once the threads are created, they seem to execute code beyond the sem_wait() even though the semaphore's value appears to be 0. I'm not sure why this would be.

Edit: As per Perception's advice, I checked the return value of sem_wait(), and it seems to be setting errno to "Operation Timed Out." As far as I can tell, this shouldn't be happening unless I am using sem_timedwait(). Still digging...

Edit 2: Oop. Should have read my output more closely. It's actually setting it to "Function not implemented."

#include <stdio.h>
#include <unistd.h>
#include <pthread.h>
#include <semaphore.h>
#include <errno.h>

// vars
int jobsInQueue, currentJob;
sem_t *semaphore;
pthread_t threads[10];
int runningThreads = 0;

// prototypes
void *do_work(void *arg);
void add_job();

int main()
{
    // i for the for loop used to create the threads
    int i;

    // counter for jobs in the queue
    jobsInQueue = 0;

    // indicator for the current job
    currentJob = 0;

    // indicator for whether we have reached the limit imposed in the while loop used for adding jobs
    int reachedlimit = 0;

    // create the semaphore
    semaphore = sem_open("semaphore", O_CREAT, 0600, 0);

    // get the value of the semaphore and temporarily store it in reachedlimit
    sem_getvalue(semaphore, &reachedlimit);

    // print off the value of the semaphore.  I think I'm crazy because the threads are executing code
    // before the semaphore is posted to, but this appears to be zero...
    fprintf(stderr, "semaphore: %d", reachedlimit);
    fflush(stderr);

    // set reachedlimit back to zero because we expect it to be zero below
    reachedlimit = 0;

    for(i = 0; i < 10; ++i)
    {
        // create a pthread
        pthread_create(&threads[i], NULL, &do_work, (void *)i);

        // increment the number of running threads
        runningThreads++;
    }

    // sleep for a couple of seconds just as separator space
    sleep(2);

    // while there are threads running
    while(runningThreads > 0)
    {
        // sleep for a tenth of a second
        usleep(100000);

        // after that, if there are 1000 or more jobs in the queue, we've reached the number of total jobs we want
        if(jobsInQueue >= 1000) reachedlimit = 1;

        // if we haven't reached that, then add another job
        if(reachedlimit == 0) add_job();

        // print that we're still sleeping and the number of jobs in the queue.
        fprintf(stderr, "Still sleeping. Jobs in queue: %d\n", jobsInQueue);
        fflush(stderr);
    }
}

void *do_work(void *arg)
{
    // when the thread is created, print this thread's number to the console
    fprintf(stderr, "I am thread %d.\n", (int)arg);
    fflush(stderr);

    // then loop infinitely doing the following...
    while(1)
    {
        // wait until the semaphore's value is no longer zero <-- doesn't seem to do this
        sem_wait(semaphore);

        // if we are on the 1000th job, terminate the thread
        if (currentJob >= 1000) {
            runningThreads--;
            fprintf(stderr, "Thread %d terminated", (int)arg);
            fflush(stderr);
            pthread_exit((void *)1);
        }

        // otherwise, increment the current job counter
        currentJob++;

        // tell the console that this thread took a job
        fprintf(stderr, "Thread %d: I took a job.: %d\n", (int)arg, currentJob);
        fflush(stderr);

        // subtract one from the count of jobs in the queue
        jobsInQueue--;

        // sleep for at least one second before taking another job
        sleep(1);
    }

    // this will never happen because the while loop will never be broken
    runningThreads--;
    return NULL;
}

void add_job()
{
    // increment the count of jobs in the queue
    jobsInQueue++;

    // print that a job has been added
    fprintf(stderr, "Job added\n");
    fflush(stderr);

    // post to the semaphore, which should essentially release the job for "processing" if I understand correctly.
    sem_post(semaphore);
}

Some sample output:

semaphore: 0
I am thread 0.
I am thread 1.
Thread 0: I took a job.: 1
I am thread 2.
I am thread 3.
Thread 1: I took a job.: 2
I am thread 4.
I am thread 5.
I am thread 6.
Thread 2: I took a job.: 3
I am thread 7.
I am thread 8.
Thread 3: I took a job.: 4
I am thread 9.
Thread 4: I took a job.: 5
Thread 5: I took a job.: 6
Thread 6: I took a job.: 7
Thread 7: I took a job.: 8
Thread 8: I took a job.: 9
Thread 9: I took a job.: 10
Thread 0: I took a job.: 12
Thread 4: I took a job.: 11
Thread 5: I took a job.: 13
Thread 6: I took a job.: 14
Thread 1: I took a job.: 15
Thread 8: I took a job.: 17
Thread 3: I took a job.: 16
Thread 7: I took a job.: 18
Thread 2: I took a job.: 19
Thread 9: I took a job.: 20
Thread 0: I took a job.: 21
Thread 1: I took a job.: 22
Thread 8: I took a job.: 23
Thread 3: I took a job.: 24
Thread 5: I took a job.: 25
Thread 7: I took a job.: 26
Thread 6: I took a job.: 27
Thread 2: I took a job.: 29
Thread 4: I took a job.: 28
Thread 9: I took a job.: 30
Job added
Still sleeping. Jobs in queue: -29
Job added
Still sleeping. Jobs in queue: -28
Job added
Still sleeping. Jobs in queue: -27
Job added
Still sleeping. Jobs in queue: -26
Job added
Still sleeping. Jobs in queue: -25
Job added
Still sleeping. Jobs in queue: -24
Job added
Still sleeping. Jobs in queue: -23
Job added
Still sleeping. Jobs in queue: -22
Job added
Still sleeping. Jobs in queue: -21
Thread 3: I took a job.: 31
Thread 0: I took a job.: 32
Thread 5: I took a job.: 33
Thread 2: I took a job.: 34
Thread 1: I took a job.: 35
Thread 7: I took a job.: 36
Thread 9: I took a job.: 37
Thread 8: I took a job.: 38
Thread 6: I took a job.: 39
Thread 4: I took a job.: 40
Job added
Still sleeping. Jobs in queue: -30
Job added
Still sleeping. Jobs in queue: -29
Job added
Still sleeping. Jobs in queue: -28
Job added
Still sleeping. Jobs in queue: -27
Job added
Still sleeping. Jobs in queue: -26
Job added
Still sleeping. Jobs in queue: -25

Answer 1

Semaphores persist after a process dies unless you specifically unlink them. The behavior you're seeing is due to the threads pulling old jobs sem_post'd to the semaphore by a previous process. Your sem_getvalue call would show the existence of those old jobs if the call actually worked, but it is failing and you're not noticing because you aren't checking sem_getvalue's return value. The "Function not implemented" errno value is in fact from sem_getvalue, not sem_wait.

Add

sem_unlink("semaphore");

before your call to sem_open and the weird behavior will go away.