Multiple Processes and Threads synchronization and ordering in C

Question

What I'm trying to accomplish is to create 3 processes and a new thread within each new process that handles anonymously mapped shared memory. I'm still new to multithreading and IPC so maybe I haven't used them correctly but I've tried Mutexes, Semaphores, and Condition Variables and while I get close to passing memory between all of them it either hangs or does not seem to pass the information completely.

What I want to do is:

1. The first fork()'d process create a thread and read in from a file.
2. The second process to make a new thread to handle the shared memory to alter it.
3. The third process to create a thread to once again alter the shared memory and then output it to a file.

I know I can accomplish this with a pipe and only the 3 processes but I wanted to experiment with the shared memory and threading.

Is it possible to order the processes in such a way that the

• 1st process -> 1st thread -> 2nd process -> 2nd thread -> 3rd process -> 3rd thread

Will run like this? Is there any way to do this in general for N processes and N threads? I feel like I should be able to but I'm not familiar enough with this yet. Any thoughts and advice are appreciated, Thanks in advance.

Edit: I found a solution to this in calling fork() but reordering the called threads in the processes. Check below for the example.

Updated question: I found that when memcpy()'ing in a certain thread I could update a buffer in the shared memory for one buffer but not for another. In the next thread my one buffer was full as it was supposed to be but the other one was not even though a quick check showed that there was data in both buffers before leaving the thread? What sort of concurrency error have I run into? Thoughts and advice are appreciated, Thanks in advance.

Answer 1

I figured out a solution thanks to @MichaelGoren and some slight modifications. I essentially reordered the threads called in the process order. Instead of using semaphores, I had my parent threads wait for their children to finish. So,

1. I called fork() for my first process and told the parent to wait for its child. I also had a Pipe between this and the next fork(). I called my 3rd thread in this parent.
2. I called my next process and made it wait for its child and put my 2nd thread in this. This child now about to be a parent caller of its own had another pipe between this and the last child.
3. I called my first thread in the last child.

I also used some shared memory as stated before.

In order it went 2nd child -> 1st thread -> 1st child -> 2nd thread -> Parent -> 3rd thread.

Now, I'm not sure if this is fool proof so feel free to correct me if this may not be concurrent.

Answer 2

The question is rather general, wide and deep. Therefore my answer will get you a very initial entry into multithreading/multiprocessing. First of all, I would divide the question into two main problems: 1) Threads synchronization inside single process. 2) Single thread process synchronization. Later you may combine (1) and (2), not forgetting special issues and side effects caused by mixing threads and forks.

1. Threads synchronization.

Let's assume we have a process with 3 (or M, in a general case) threads t1, t2 and t3, that should run in a strong sequence: t1, t2 and t3. Even if you create these threads in exactly this sequence, no one can guarantee that they will be started in that sequence. You may block the execution of t2 and t3 by forcing them to wait on two conditional variables: c1 and c2. The above conditional variables will signal the t2 and t3 threads, when the t1 and t2 threads are ready. The t1 thread will "unblock" the c1 variable, the t2 thread will "unblock" c2. So, all that you need are two (or M-1) conditional variables associated with two (or M-1) mutexes and two (or M-1) boolean responsible for the threads status. Here is a simplified code. Although it works (you may compile it with gcc -lpthread), your "real" code should organize the variables into structures, check for errors etc.

#include <pthread.h>
#include <stdio.h>

// These are mentioned mutexes, conditional variables and booleans.
// They are global, so any thread can access them.
pthread_mutex_t m1 = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t  c1 = PTHREAD_COND_INITIALIZER;
int t1_is_ready = 0;

pthread_mutex_t m2 = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t  c2 = PTHREAD_COND_INITIALIZER;
int t2_is_ready = 0;

// This function is invoked by the t1 thread.
// It does not depend on conditional variables and always is executed first.
// When it's ready it sets its ready status to one and signal to the thread t2
// via c2, that the status has changed.
void* t1_work(void* arg)
{
  pthread_mutex_lock(&m1);
  printf("thread t1 does its work\n");
  t1_is_ready = 1;
  pthread_mutex_unlock(&m1);
  pthread_cond_signal(&c1);

  pthread_exit(NULL);
} 

// This function is invoked by the t2 thread.
// It is blocked until t1 setis is ready status to 1.
// Pay your attention how to wait on a conditional variable:
// with locked mutex and in a loop, to prevent spurious wakes up. 
void* t2_work(void* arg)
{
  pthread_mutex_lock(&m1);
  while(t1_is_ready == 0)
  {
    pthread_cond_wait(&c1, &m1);
  }
  pthread_mutex_unlock(&m1);

  pthread_mutex_lock(&m2);
  printf("thread t2 does its work\n");
  t2_is_ready = 1;
  pthread_mutex_unlock(&m2);
  pthread_cond_signal(&c2);

  pthread_exit(NULL);
}

// This function is invoked by the t3 thread.
void* t3_work(void* arg)
{
  pthread_mutex_lock(&m2);
  while(t2_is_ready == 0)
  {
    pthread_cond_wait(&c2, &m2);
  }

  printf("thread t3 does its work\n");
  pthread_mutex_unlock(&m2);

  pthread_exit(NULL);
}

int main(void)
{
  pthread_t t1;
  pthread_t t2;
  pthread_t t3;

  pthread_create(&t1, NULL, t1_work, NULL);
  pthread_create(&t2, NULL, t2_work, NULL);
  pthread_create(&t3, NULL, t3_work, NULL);

  pthread_join(t1, NULL);
  pthread_join(t2, NULL);
  pthread_join(t3, NULL);

  return 0;
}

2. Process synchronization

Here is a good article with code examples of process synchronization: http://www.linuxdevcenter.com/pub/a/linux/2007/05/24/semaphores-in-linux.html?page=5 Since you are planning to create your processes by the fork() function, they will be so called "related". If you have 3 (or N) running processes create 2 (N-1) semaphores, using them in a way you have synchronized your threads.