CODEWITHSUNDEEP
mysqlperlgzip
terdon
terdon

Reputation: 3380

Perl gives "gzip: stdout: Broken pipe" error when opening gzipped files, but only if connecting to a DB

Consider the following program, running on a Linux machine, which opens a gzipped input file:

#!/usr/bin/env perl

open (my $fileHandle, "-|", "/bin/zcat $ARGV[0]");
my $ff = <$fileHandle>;
close($fileHandle);

That works as expected (it does nothing, but prints no error):

$ bar.pl file.gz
$

Now, if I use the same code but previously connect to a MySQL database, gzip will complain (you can run the code directly, this is an open DB and the credentials will work):

#!/usr/bin/env perl
use DBI;
use strict;
use warnings;

my $dsn = "DBI:mysql:database=hg19;host=genome-mysql.cse.ucsc.edu";
my $db =  DBI->connect($dsn, 'genomep', 'password');
my $dbResults = $db->prepare("show tables");
my $ret = $dbResults->execute();
$dbResults->finish();
$db->disconnect();

open (my $fileHandle, "-|", "/bin/zcat $ARGV[0]");
my $ff = <$fileHandle>;
close($ff);

Running the above gives:

$ foo.pl file.gz 

gzip: stdout: Broken pipe

This is obviously part of a much more complicated program, but I've managed to trim it down to this silly snippet that reproduces the issue.

What's going on? Why does connecting to a DB affect how gzip behaves? Note that everything seems to work (in the actual program, I do something useful with the gzipped data) but why am I getting that error message?

It turns out this behavior is specific to (slightly) older versions of Perl and/or DBI. On the machines where it failed, I have:

However, on another two machines it did work. These had:

And

Upvotes: 3

Views: 7844

Answers (2)

derobert
derobert

Reputation: 51167

At least here, it appears that the MySQL libraries (probably) are masking (ignoring) SIGPIPE, and that's what you're seeing. Comparing strace outputs, I see a line like this in the MySQL run:

rt_sigaction(SIGPIPE, {sa_handler=SIG_IGN, sa_mask=[PIPE], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f78bdf16840}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0

And it turns out you can duplicate the behavior easily w/o MySQL:

$SIG{PIPE} = 'IGNORE';

open (my $fileHandle, "-|", "/bin/zcat $ARGV[0]");
my $ff = <$fileHandle>;
close($ff);

Or, alternatively, you can reset the signal to the default handler to make the message go away, even after connecting to MySQL by setting it to DEFAULT instead of IGNORE.

This is, by the way, documented behavior of the MySQL library:

To avoid aborting the program when a connection terminates, MySQL blocks SIGPIPE on the first call to mysql_library_init(), mysql_init(), or mysql_connect().

(It may also depend on the gzip version; maybe some versions of gzip set up signal handlers on init.)

Ultimately, what you're seeing is that if gzip gets a SIGPIPE, it just exits. If it gets an error back from write (because SIGPIPE is ignored), it prints an error message.

Upvotes: 4

g_bor
g_bor

Reputation: 1092

Most probably the following is happening: gzip tries to write to the pipe, the program on your side is not reading up to eof, the closes the pipe. Gzip then receives a SIGPIPE, and dies with this error message. Can you confirm that this is taking place?

Upvotes: 1

Related Questions