CODEWITHSUNDEEP
phpdebianpipegzipio-redirection
Kyle Spier-Swenson
Kyle Spier-Swenson

Reputation: 1

pipe stdout to gzip of long running program in a way that lets you read it mid run

I have a php irc bot that logs to stdout ran using nohup via a while true: php bot.php; sleep 60 style bash script to act like a watchdog.

It outputs alot of data, mostly noise, but can be helpful debugging state during errors, so i started piping stdout thru gzip, the only issue is that i can't read the file until it closes, so debugging issues found in production can become a pain since i have to shut everything down to access the logs.

I was hoping there might be a simple solution this, I found one around setting the buffer sizes using stdbuf but it didn't seem to help, i think maybe gzip might be doing its own buffering.

Is there a better way to send stdout of a long running program to a compressed file that allows for peaking in the file mid run?

Upvotes: 0

Views: 442

Answers (2)

Mark Adler
Mark Adler

Reputation: 112284

You can use gzlog.c/gzlog.h. It writes short log messages to a gzip file efficiently, leaving a complete and correct gzip file after each write. It does this by appending the log messages uncompressed until a threshold is reached, at which point the uncompressed log messages are compressed.

Upvotes: 0

Sammitch
Sammitch

Reputation: 32232

The way that most daemons tend to handle this is to listen for a signal like SIGHUP or one of the SIGUSRs, flush their log buffers to disk, then close and reopen the logfile handle. This is so that a system utility like logrotate can have already renamed the file and simply signal to the running process to flush and recreate.

Once you've got that sorted out you can make use of existing log rotation services to rotate and compress your log files for you based on time and/or size constraints, rather than having to worry about implementing these vagaries in your application.

Upvotes: 1

Related Questions