Pthread Scheduling policy and priority

Question

I have four threads which are waiting on a condition variable and fifth thread posts condition variable when all four threads are waiting. When I set thread priority to maximum that is 99, threads switch takes a lot of time which is far from acceptable. Can anybody please take a look and tell what's happening ?

#define N_WORK_THREADS        4
pthread_mutex_t count_mutex     = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t  condition_var   = PTHREAD_COND_INITIALIZER;

void *functionCount1(void * arg);
void *functionCount2(void * arg);

int  count = 0;
int  valid = 0;
int thread_personal[N_WORK_THREADS];

static int display_thread_sched_attr(int id)
{
  int policy, s;
  struct sched_param param;

  s = pthread_getschedparam(pthread_self(), &policy, &param);
  if (s != 0) { printf("pthread_getschedparam"); return 1; }

  printf("Thread Id=%d policy=%s, priority=%d\n",id,
          (policy == SCHED_FIFO)  ? "SCHED_FIFO" : (policy == SCHED_RR)    ? "SCHED_RR" : (policy == SCHED_OTHER) ? "SCHED_OTHER" : "???",
          param.sched_priority);

  return 0;
}


int main(void)
{
  pthread_t       thread_work[N_WORK_THREADS];
  pthread_t       thread;
  int             i,s;
  pthread_attr_t  attr;
  struct          sched_param param;

  s = pthread_attr_init(&attr);
  if (s != 0) { printf("pthread_attr_init"); return 1; }

  s = pthread_attr_setinheritsched(&attr, PTHREAD_EXPLICIT_SCHED);
  if (s != 0) { printf("pthread_attr_setinheritsched"); return 1; } 

  s = pthread_attr_setschedpolicy(&attr, SCHED_RR);
  if (s != 0) { printf("pthread_attr_setschedpolicy"); return 1; }

  param.sched_priority = 99;
  s = pthread_attr_setschedparam(&attr, &param);
  if (s != 0) { printf("pthread_attr_setschedparam"); return 1; }

  for (i=0; i<N_WORK_THREADS; i++) { thread_personal[i] = 0; }

  for (i=0; i<N_WORK_THREADS; i++) { pthread_create( &thread_work[i], &attr, &functionCount1, (void *)i); }

  param.sched_priority = 99;
  s = pthread_attr_setschedparam(&attr, &param);
  if (s != 0) { printf("pthread_attr_setschedparam"); return 1; }
  pthread_create( &thread, &attr, &functionCount2, (void *)N_WORK_THREADS);

  for (i=0; i<N_WORK_THREADS; i++) { pthread_join( thread_work[i], NULL); }
  pthread_join( thread, NULL);

  for (i=0; i<N_WORK_THREADS; i++) { printf("Thread Id=%d Mutex USed=%d\n",i,thread_personal[i]); }

  exit(EXIT_SUCCESS);
}

void *functionCount1(void * arg)
{
  int i;
  int id = (int) arg;

  display_thread_sched_attr(id);

  for(i=0; i<10; i++)
  {
    pthread_mutex_lock( &count_mutex );

    thread_personal[id] += 1;

    while (((count>>id) & 0x1) == 0)
    {
      pthread_cond_wait( &condition_var, &count_mutex );
    }
    count = count^ (1<<id);

    printf("Thread Id %d: Valid = %d\n",id,valid);

    pthread_mutex_unlock( &count_mutex );
  }

  return NULL;
}

void *functionCount2(void * arg)
{
  int check;
  int id = (int) arg;

  display_thread_sched_attr(id);

  check =0;
  while (check < 10)
  {    
    pthread_mutex_lock( &count_mutex );

    if (count == 0)
    {
      pthread_cond_broadcast ( &condition_var );
      count =0xF;
      printf("Thread Id %d: Counter = %d\n",id,check);
      valid = check++;
    }

    pthread_mutex_unlock( &count_mutex );
  }
  return NULL;
}

Answer 1

I'm unable to test your program with the scheduling policy code enabled because the program simply doesn't work when that's in there (as I mention in a comment: Linux 3.16.0 x86_64 with gcc 4.8.4).

But I'm guessing that your problem might be due to the loop in functionCount2():

while (check < 10)
{    
    pthread_mutex_lock( &count_mutex );

    if (count == 0)
    {
      pthread_cond_broadcast ( &condition_var );
      count =0xF;
      printf("Thread Id %d: Counter = %d\n",id,check);
      valid = check++;
    }

    pthread_mutex_unlock( &count_mutex );
}

In general, acquisition of mutex objects in pthreads is not guaranteed to be fair or FIFO (though to be honest, I'm not sure how thread scheduling policies might affect it). What I believe is happening is that this loop releases count_mutex then immediately re-acquires it even though other threads are blocked waiting to claim the mutex. And with the scheduling policy in place, this may occur until the thread uses its quantum.