Calling external program using boost::process causes caller to hang (Linux)

Question

I am using boost::process to call an external program - the external program reads input via stdin, and writes to stdout and stderr. The external program is as follows (expects a single argument - the path to a file for debugging)

#include <fstream>
#include <iostream>
#include <stdexcept>
#include <string>
#include <vector>

int main(int argc, char** argv)
{
    try
    {
        if (argc != 2)
        {
            throw std::logic_error("Expected two arguments");
        }

        std::ofstream ofs(argv[1]);

        std::vector<std::string> someTestInput;

        ofs << "Starting program..." << std::endl;

        // Read from cin
        {
            ofs << "Reading from cin..." << std::endl;
            std::string input;
            while (std::getline(std::cin, input))
            {
                ofs << "Received from cin: " << input << std::endl;
                someTestInput.emplace_back(input);
            }

            ofs << "Finished receiving from cin..." << std::endl;
        }

        // Error if nothing has been input
        if (someTestInput.empty())
        {
            throw std::logic_error("Expected some input and received nothing...");
        }

        ofs << "Writing to cout..." << std::endl;

        // Write to cout
        for (const auto& output : someTestInput)
        {
            std::cout << output << '\n';
        }

        ofs << "Finished!\n";
    }
    catch (std::exception& e)
    {
        std::cerr << "Error caught: " << e.what() << '\n';
        return 1;
    }

    return 0;
}

The caller expects 2+ arguments, one of which is the path to the external program, and the rest are passed on as arguments to the external program.

It hangs while waiting for the process to exit, and it seems like the external program is waiting for an EOF from stdin.

#include <memory>
#include <vector>
#include <boost/iostreams/device/file_descriptor.hpp>
#include <boost/iostreams/stream.hpp>
#include <boost/process.hpp>

int main(int argc, char** argv)
{
    try
    {
        if (argc < 2)
        {
            throw std::logic_error("Expecting at least 2 arguments...");
        }

        std::vector<std::string> args;

        for (int i = 1; i < argc; ++i)
        {
            args.emplace_back(argv[i]);
        }

        std::cout << "Creating stdout, stderr pipes...\n";

        // Create pipes for stdout, stderr
        boost::process::pipe pstdout = boost::process::create_pipe();
        boost::process::pipe pstderr = boost::process::create_pipe();

        std::cout << "Mapping pipes to sources...\n";

        // Map pipe source from stdout and stderr to sources
        boost::iostreams::file_descriptor_source sourcestdout(pstdout.source, boost::iostreams::close_handle);
        boost::iostreams::file_descriptor_source sourcestderr(pstderr.source, boost::iostreams::close_handle);

        std::cout << "Setting up streams for the sources...\n";

        // And set up streams for the sources
        boost::iostreams::stream<boost::iostreams::file_descriptor_source> istdout(sourcestdout);
        boost::iostreams::stream<boost::iostreams::file_descriptor_source> istderr(sourcestderr);

        std::unique_ptr<boost::process::child> p;

        // Want to check for process result, but also need to ensure stdin handle is closed properly,
        // so place everything in separate scope
        {
            std::cout << "Mapping pipes to sinks...\n";

            // Map pipe sink from stdout and stderr to sinks
            boost::iostreams::file_descriptor_sink sinkstdout(pstdout.sink, boost::iostreams::close_handle);
            boost::iostreams::file_descriptor_sink sinkstderr(pstderr.sink, boost::iostreams::close_handle);

            std::cout << "Creating stdin pipe, mapping to source and sink...\n";

            boost::process::pipe pstdin = boost::process::create_pipe();

            // For stdin, map pipe to source and sink as before - want it to close on exiting this scope
            boost::iostreams::file_descriptor_sink sinkstdin(pstdin.sink, boost::iostreams::close_handle);
            boost::iostreams::file_descriptor_source sourcestdin(pstdin.source, boost::iostreams::close_handle);
            boost::iostreams::stream<boost::iostreams::file_descriptor_sink> ostdin(sinkstdin);

            std::cout << "Calling process... \n";

            // Call process
            p = std::unique_ptr<boost::process::child>(new boost::process::child(boost::process::execute(
                boost::process::initializers::set_args(args),
                boost::process::initializers::throw_on_error(),
                boost::process::initializers::bind_stdout(sinkstdout),
                boost::process::initializers::bind_stderr(sinkstderr),
                boost::process::initializers::bind_stdin(sourcestdin)
                )));

            std::cout << "Sending test data...\n";

            // Send some test data to cin - comment out the below to test for error case
            ostdin << "Test Input 1\n";
            ostdin << "Some\n";
            ostdin << "Useful\n";
            ostdin << "Data\n";

            std::cout << "Test data sent, exiting scope...\n";
        }

        std::cout << "Check if process has exited...\n";

        // Check if process has exited OK - if not, report errors
        if (boost::process::wait_for_exit(*p))
        {
            std::cout << "Has not exited OK, reporting problems...\n";

            // Gather output from stderr
            std::string error;
            while (std::getline(istderr, error))
            {
                std::cout << "Error: " << error << '\n';
            }

            throw std::logic_error("Problem executing TestProgram...");
        }

        std::cout << "Exited OK, here is output from the callee...\n";

        // Gather the output
        std::string output;
        while (std::getline(istdout, output))
        {
            std::cout << output << '\n';
        }
    }
    catch (std::exception& e)
    {
        std::cerr << "Error: " << e.what() << '\n';
        return 1;
    }
}

I was under the impression that placing my stdin pipe and related sources/sinks within a scope will guarantee they're closed, and therefore send the EOF.

The same code works perfectly under Windows (VS2013, boost_1_53).

I am using boost_1_53, boost-process 0.5, gcc 4.8.2.

Answer 1

That does not happen, because there's still a pipe handle open in the child process; that is only closed on posix if you set it explicitly (on windows it is done automatically). So you'd need to add something like that:

#if defined (BOOST_POSIX_API)
fcntl(pstdout.sink, F_SETFD, FD_CLOEXEC);
fcntl(pstderr.sink, F_SETFD, FD_CLOEXEC);
#endif

I would however recommend to use boost.asio and wait asynchronously for the exit of the subprocess and close the pipes there.

Just FYI: I've worked on boost-process 0.6 which has a different interface but makes the asio stuff much easier. This will hopefully be in review in October/November, so it might become an official boost library soon. It's currently in beta so you might want to check that one out.