Accomplishing asynchronous behaviour with pipes

Question

I came across a bit of an interesting problem during piping. Suppose I want to emulate asynchronous behaviour between pipes, such that I have a single parent process and multiple child processes. The parent and child processes can talk to each other. However, something I have come across is the communication is atomic so far. That is that if I have a dataset

• 1. First create all the children
• 2. Send a CHUNK_SIZE of data of the dataset to the children via a loop
• 3. Children does some computation or nothing and passes its data back to the parent
• 4. Parent receives the data from its children

However, the trouble is that my communication is atomic, I have to send all my data and then read it. The sending is fine since the child reads its almost instantaneously and thus do not need to worry about the pipe filling up. However, all of the children do their work and return back their data, but the parent won't read this until it finishes sending all the data. That is, a pipe can fill up for one of the children to the parent during step 3 before the parent has even reached step 4. One way to avoid this is to just send data and then receive before we send another dataset again, but this in a way ruins the whole point. Another way is to yet fork() again and have another child deal with this. But this is just moving the problem to somewhere else. I think this is a problem with my design, but it seems impossible because my code in a single process must be in two places at once! (that is reading and writing data to the children). Is there any best practice or hindsight on designing "true" asynchronous behaviour using only pipes in this methodology?

Answer 1

You can do that using select. Select will tell you when any of your pipes are ready to be read from or written to.