C MPI - spawning multiple threads in batches

Question

I have a basic question about MPI programming in C. Essentially what I want is that there is a master process that spawns a specific number of child processes, collects some information from all of them (waits until all of the children finish), calculates some metric, based on this metric it decides if it has to spawn more threads... it keeps doing this until the metric meets some specific condition. I have searched through the literature, to no avail. How can this be done? any pointers?.

Thanks for the help.

Courtesy : An introduction to the Message Passing Interface (MPI) using C. In the "complete parallel program to sum an array", lets say, "for some lame reason", I want the master process to sum the contents of the array twice. I.e in the first iteration, the master process starts the slave processes which compute the sum of the arrays, once they are done and the master process returns the value, I would like to invoke the master process to reinvoke another set of threads to do the computation again. Why would the code below not work? I added a while loop around the master process process which spawns the slave processes.

  #include <stdio.h>
  #include <mpi.h>

  #define max_rows 100000
  #define send_data_tag 2001
  #define return_data_tag 2002

  int array[max_rows];
  int array2[max_rows];

main(int argc, char **argv) 
 {
   long int sum, partial_sum,number_of_times;

  number_of_times=0;

  MPI_Status status;

  int my_id, root_process, ierr, i, num_rows, num_procs,
     an_id, num_rows_to_receive, avg_rows_per_process, 
     sender, num_rows_received, start_row, end_row, num_rows_to_send;

  /* Now replicte this process to create parallel processes.
   * From this point on, every process executes a seperate copy
   * of this program */

  ierr = MPI_Init(&argc, &argv);

  root_process = 0;

  /* find out MY process ID, and how many processes were started. */

  ierr = MPI_Comm_rank(MPI_COMM_WORLD, &my_id);
  ierr = MPI_Comm_size(MPI_COMM_WORLD, &num_procs);

  if(my_id == root_process) {

     /* I must be the root process, so I will query the user
      * to determine how many numbers to sum. */

     //printf("please enter the number of numbers to sum: ");
     //scanf("%i", &num_rows);

      num_rows=10;

      while (number_of_times<2)
      {

      number_of_times++;
      start_row=0;
      end_row=0;



     if(num_rows > max_rows) {
        printf("Too many numbers.\n");
        exit(1);
     }

     avg_rows_per_process = num_rows / num_procs;

     /* initialize an array */

     for(i = 0; i < num_rows; i++) {
        array[i] = i + 1;
     }

     /* distribute a portion of the bector to each child process */

     for(an_id = 1; an_id < num_procs; an_id++) {
        start_row = an_id*avg_rows_per_process + 1;
        end_row   = (an_id + 1)*avg_rows_per_process;

        if((num_rows - end_row) < avg_rows_per_process)
           end_row = num_rows - 1;

        num_rows_to_send = end_row - start_row + 1;

        ierr = MPI_Send( &num_rows_to_send, 1 , MPI_INT,
              an_id, send_data_tag, MPI_COMM_WORLD);

        ierr = MPI_Send( &array[start_row], num_rows_to_send, MPI_INT,
              an_id, send_data_tag, MPI_COMM_WORLD);
     }

     /* and calculate the sum of the values in the segment assigned
      * to the root process */

     sum = 0;
     for(i = 0; i < avg_rows_per_process + 1; i++) {
        sum += array[i];   
     } 

     printf("sum %i calculated by root process\n", sum);

     /* and, finally, I collet the partial sums from the slave processes, 
      * print them, and add them to the grand sum, and print it */

     for(an_id = 1; an_id < num_procs; an_id++) {

        ierr = MPI_Recv( &partial_sum, 1, MPI_LONG, MPI_ANY_SOURCE,
              return_data_tag, MPI_COMM_WORLD, &status);

        sender = status.MPI_SOURCE;

        printf("Partial sum %i returned from process %i\n", partial_sum, sender);

        sum += partial_sum;
     }

     printf("The grand total is: %i\n", sum);


      }



  }

  else {

     /* I must be a slave process, so I must receive my array segment,
      * storing it in a "local" array, array1. */

     ierr = MPI_Recv( &num_rows_to_receive, 1, MPI_INT, 
           root_process, send_data_tag, MPI_COMM_WORLD, &status);

     ierr = MPI_Recv( &array2, num_rows_to_receive, MPI_INT, 
           root_process, send_data_tag, MPI_COMM_WORLD, &status);

     num_rows_received = num_rows_to_receive;

     /* Calculate the sum of my portion of the array */

     partial_sum = 0;
     for(i = 0; i < num_rows_received; i++) {
        partial_sum += array2[i];
     }

     /* and finally, send my partial sum to hte root process */

     ierr = MPI_Send( &partial_sum, 1, MPI_LONG, root_process, 
           return_data_tag, MPI_COMM_WORLD);
  }


  ierr = MPI_Finalize();
 }

Answer 1

You should start by looking at MPI_Comm_spawn and collective operations. To collect information from old child processes,one would typically use MPI_Reduce.

This stackoverflow question might also be helpful.

...to spawn more threads...

I guess you meant the right thing since you used "process" instead of "thread" mostly, but just to clarify: MPI only deals with processes and not with threads.

I'm not sure how well you know MPI already - let me know if my answer was any help or if you need more hints.

Answer 2

The MPI-2 standard includes process management functionality. It's described in detail in Chapter 5. I have not used it myself though, so perhaps someone else may weigh in with more practical hints.