CODEWITHSUNDEEP
pythoniogzip
user1908678
user1908678

Reputation:

Appending to gzip file in Python doesnt work

I have following function : 

def save(msg):
    with gzip.open("ircbot.log.gz", "ab") as f:
        f.write(msg+'\n')
        f.close()
        return "Succesfully logged: "+msg

I want to add each msg to .log file but it doesnt work, only first msg is saved.

For example after calling those functions :

save('first')
save('second')
save('third')

.log file contains only 'first'.

For simple .txt file it works fine. Gzip doesn't support appending to file?

Upvotes: 1

Views: 2841

Answers (3)

Mark Adler
Mark Adler

Reputation: 112374

Concatenating gzip streams to make an extractable gzip file (i.e. f.close() after each message) does work, as you discovered. This is because the gzip standard requires a compliant decompressor to look for another gzip stream after it decodes the current one. However, assuming that your messages are relatively short, e.g. a line or two, then the resulting gzip file will be larger, not smaller than a simple text file with the messages. Each message will have at least the overhead of a gzip header and trailer of 18 bytes, and the data will likely expand by five bytes, adding 23 bytes per message.

The alternative of not using f.close() after each message will result in real compression of the data, by writing a single gzip stream where the compression of later messages can take advantage of redundancy with earlier messages. However this has the downside of never having a complete and correct gzip file until you do finally call f.close(). In addition, messages will not be written at all (again if they are short), until enough have been accumulated to compress a block. Then a bunch will be written in a burst, and the file will wait again for more to be accumulated.

There is a solution to this, but I don't think python has a complete enough interface to zlib to permit it. You can look at an example in C, gzlog.h and gzlog.c, which writes log entries to a gzip file immediately, and always leaves the log file in a complete and correct state.

Upvotes: 3

Dima Tisnek
Dima Tisnek

Reputation: 11781

Works for me, without the extra f.close(), Linux, python-2.7, both with gzip file created by this script and a gzip file created by regular gzip command.

Upvotes: 0

user1908678
user1908678

Reputation:

Ok, I figured it out.

I used Altap Salmander to extract .gz and view log file (F3 function).

When I extracted gz file in classic explorer with 7zip all msgs were there.

Upvotes: 0

Related Questions