CODEWITHSUNDEEP
perlipcpipefilehandle
theory
theory

Reputation: 9897

What is the preferred cross-platform IPC Perl module?

I want to create a simple IO object that represents a pipe opened to another program to that I can periodically write to another program's STDIN as my app runs. I want it to be bullet-proof (in that it catches all errors) and cross-platform. The best options I can find are:

open

sub io_read {
    local $SIG{__WARN__} = sub { }; # Silence warning.
    open my $pipe, '|-', @_ or die "Cannot exec $_[0]: $!\n";
    return $pipe;
}

Advantages:

Disadvantages

IO::Pipe

sub io_read {
    IO::Pipe->reader(@_);
}

Advantages:

Disadvantages

IPC::Run

There is no interface for writing to a file handle in IPC::Run, only appending to a scalar. This seems…weird.

IPC::Run3

No file handle interface here, either. I could use a code reference, which would be called repeatedly to spool to the child, but looking at the source code, it appears that it actually writes to a temporary file, and then opens it and spools its contents to the pipe'd command's STDIN. Wha?

IPC::Cmd

Still no file handle interface.

What am I missing here? It seems as if this should be a solved problem, and I'm kind of stunned that it's not. IO::Pipe comes closest to what I want, but the lack of $SIG{PIPE} error handling and the lack of support for Windows is distressing. Where is the piping module that will JDWIM?

Upvotes: 16

Views: 2385

Answers (3)

Guido Flohr
Guido Flohr

Reputation: 2350

It is possible to redirect stdandard output and standard error from an (almost) arbitrary number of child processes and read them asynchronously triggered by select(). The child processes do not have to co-operate, like in the accepted answer, although it is helpful if they use line-buffered or unbuffered I/O.

For POSIX/Un*x you just use the regular combination of pipe(), fork(), POSIX::dup(), exec(), and select() or one of the higher level modules you have mentioned, like IPC::Open3(), IPC::Run() or so.

None of that works under Windows because the Perl emulation of fork() and exec() is not sufficient for that task, and select() only works on sockets.

What you have to do for Windows is:

  1. Create a socketpair() (with the socketpair() emulation of Perl).
  2. Enable non-blocking I/O on the read end of the socket pair with $true = 1; ioctl $child_stdout, 0x8004667e, \$true.
  3. dup() STDOUT and save the duplicate descriptor.
  4. dup() the write end of the socketpair replacing STDOUT.
  5. Create the child process with Win32::Process::Create(), not with fork() and exec().
  6. Restore STDOUT.
  7. Wait for the child to write (resp. flush()) to standard output with select() and sysread() it from the read end of the socket pair.

The procedure for STDERR is identical. For STDIN you have to swap the read ends and write ends.

The constant 0x8004667e is called FIONBIO in the Windows header files but not available in Perl.

I have written a blog post with more in-depth information: http://www.guido-flohr.net/platform-independent-asynchronous-child-process-ipc/. A git repository with working code in Perl and C can be found here: https://github.com/gflohr/platform-independent-ipc.

Upvotes: 0

theory
theory

Reputation: 9897

Thanks to guidance from @ikegami, I have found that the best choice for interactively reading from and writing to another process in Perl is IPC::Run. However, it requires that the program you are reading from and writing to have a known output when it is done writing to its STDOUT, such as a prompt. Here's an example that executes bash, has it run ls -l, and then prints that output:

use v5.14;
use IPC::Run qw(start timeout new_appender new_chunker);

my @command = qw(bash);

# Connect to the other program.
my ($in, @out);
my $ipc = start \@command,
    '<' => new_appender("echo __END__\n"), \$in,
    '>' => new_chunker, sub { push @out, @_ },
    timeout(10) or die "Error: $?\n";

# Send it a command and wait until it has received it.
$in .= "ls -l\n";
$ipc->pump while length $in;

# Wait until our end-of-output string appears.
$ipc->pump until @out && @out[-1] =~ /__END__\n/m;

pop @out;
say @out;

Because it is running as an IPC (I assume), bash does not emit a prompt when it is done writing to its STDOUT. So I use the new_appender() function to have it emit something I can match to find the end of the output (by calling echo __END__). I've also used an anonymous subroutine after a call to new_chunker to collect the output into an array, rather than a scalar (just pass a reference to a scalar to '>' if you want that).

So this works, but it sucks for a whole host of reasons, in my opinion:

  • There is no generally useful way to know that an IPC-controlled program is done printing to its STDOUT. Instead, you have to use a regular expression on its output to search for a string that usually means it's done.
  • If it doesn't emit one, you have to trick it into emitting one (as I have done here—god forbid if I should have a file named __END__, though). If I was controlling a database client, I might have to send something like SELECT 'IM OUTTA HERE';. Different applications would require different new_appender hacks.
  • The writing to the magic $in and $out scalars feels weird and action-at-a-distance-y. I dislike it.
  • One cannot do line-oriented processing on the scalars as one could if they were file handles. They are therefore less efficient.
  • The ability to use new_chunker to get line-oriented output is nice, if still a bit weird. That regains a bit of the efficiency on reading output from a program, though, assuming it is buffered efficiently by IPC::Run.

I now realize that, although the interface for IPC::Run could potentially be a bit nicer, overall the weaknesses of the IPC model in particular makes it tricky to deal with at all. There is no generally-useful IPC interface, because one has to know too much about the specifics of the particular program being run to get it to work. This is okay, maybe, if you know exactly how it will react to inputs, and can reliably recognize when it is done emitting output, and don't need to worry much about cross-platform compatibility. But that was far from sufficient for my need for a generally useful way to interact with various database command-line clients in a CPAN module that could be distributed to a whole host of operating systems.

In the end, thanks to packaging suggestions in comments on a blog post, I decided to abandon the use of IPC for controlling those clients, and to use the DBI, instead. It provides an excellent API, robust, stable, and mature, and suffers none of the drawbacks of IPC.

My recommendation for those who come after me is this:

  • If you just need to execute another program and wait for it to finish, or collect its output when it is done running, use IPC::System::Simple. Otherwise, if what you need to do is to interactively interface with something else, use an API whenever possible. And if it's not possible, then use something like IPC::Run and try to make the best of it—and be prepared to give up quite a bit of your time to get it "just right."

Upvotes: 6

Levi Beers
Levi Beers

Reputation: 1

I've done something similar to this. Although it depends on the parent program and what you are trying to pipe. My solution was to spawn a child process (leaving $SIG{PIPE} to function) and writing that to the log, or handling the error in what way you see fit. I use POSIX to handle my child process and am able to utilize all the functionality of the parent. However if you're trying to have the child communicate back to the parent - then things get difficult. Do you have an example of the main program and what you're trying to PIPE?

Upvotes: 0

Related Questions