`boost::process` with standard output redirection fails randomly with Ubuntu 16

Question

I try to use boost::process, it looks like very buggy part of boost; anyway, may be anyone knows workaround.

The most common task is to execute process and get its full (really full) output. Also generally output may be binary one and hence we can not use string in generic case.

Asynchronous example from boost documentation doesn't work, other articles on this forum mention that already, so I tried to use the most simple synchronous algorithm. Of course, I know about deadlocks risk, but boost doesn't come to this point, it fails before.

The code idea follows:

bool ReadPipe(boost::process::ipstream &pipe, vector<char> &output)
{
   char buffer[4096];
   pipe.read(buffer, 4096);
   auto bytesRead = pipe.gcount();
   if (bytesRead)
      output.insert(output.end(), buffer, buffer + bytesRead);
   return bytesRead != 0;
}
boost::process::ipstream output;
vector<char> processOutput;
string cmdline = "somthing";
boost::process::child c(cmdLine.c_str(),
   boost::process::std_in.close(),
   boost::process::std_out > output);
while (c.running())
   Reader::ReadPipe(output, processOutput);
Reader::ReadPipe(output, processOutput);

In this code we create process, redirect its standard output to ipstream, read it while application runs, and read possible rest of data after application existed.

On Windows it works OK. On Ubuntu 16 it sometimes works, sometimes returns partial output and sometimes returns no output.

Does anyone have idea why it is so unstable, and is there any realistic way to use boost::process to get full, possibly binary output from any application, just like Linux terminal can do it?

Answer 1

Using running() invites a race condition.

If the program exits before you've consumed all its output, you will stop consuming. That's explicitly what you coded.

Look At It Closely

Use this contrived example that exacerbates the problem so that it's reliably reproduced:

#include <boost/process.hpp>

namespace Reader {
    static constexpr size_t buf_size = 4096;

    bool ReadPipe(boost::process::ipstream &pipe, std::vector<char> &output)
    {
        std::this_thread::sleep_for(std::chrono::milliseconds(500));

        char buffer[buf_size];
        pipe.read(buffer, sizeof(buffer));
        auto bytesRead = pipe.gcount();
        if (bytesRead)
            output.insert(output.end(), buffer, buffer + bytesRead);

        return bytesRead != 0;
    }
}

#include <iostream>

int main() {
    boost::process::ipstream output;
    std::vector<char> processOutput;
    std::string cmdline = "/bin/bash";
    boost::process::child c(cmdline.c_str(), std::vector<std::string> { "-c", "(dd if=/dev/urandom bs=1024 count=10 | xxd); echo -e '\\nComplete, bye!'" },
            boost::process::std_in.close(),
            boost::process::std_out > output);

    while (c.running())
        Reader::ReadPipe(output, processOutput);

    std::cout.write(processOutput.data(), processOutput.size()) << std::endl;
}

Piping the output through |tail prints something like

As you can see, it's indeed incomplete. Reducing the dd size e.g.

boost::process::child c(cmdline.c_str(), std::vector<std::string> { "-c", "(dd if=/dev/urandom bs=32 count=10 | xxd); echo -e '\\nComplete, bye!'" },

Shows the expected

There's A Bear On The Road

There's an ugly catch to that: tutorial example has the following ominous warning:

So, in fact it's not completely clear whether it is safe to drain the input buffer after the process exited. My hunch tells me this is actually fine, but, more importantly, things can be a lot simpler:

Luckily, We Can Fly (and bears can't)

#include <boost/process.hpp>
#include <boost/asio.hpp>
#include <iostream>

int main() {
    std::future<std::vector<char>> processOutput;
    std::string cmdline = "/bin/bash";
    boost::asio::io_service io;
    boost::process::child c(cmdline.c_str(), std::vector<std::string> { "-c", "(dd if=/dev/urandom bs=1024 count=10 | xxd); echo -e '\\nComplete, bye!'" },
            boost::process::std_in.close(),
            boost::process::std_out > processOutput,
            io);

    io.run();
    c.wait();

    auto output = processOutput.get();
    std::cout.write(output.data(), output.size()) << std::endl;
}

This works as expected without the race condition. For more complicated usage (e.g. "live" dialogue with the child process where the input determines the output dynamically), consider using the async interfaces, e.g. boost process running() and exit_code() thread safety or Boost::process output blank lines