CODEWITHSUNDEEP
pythoncompression
MY_1129
MY_1129

Reputation: 317

Decompress bz2 files

I would like to decompress the files in different directories which are in different routes. And codes as below and the error is invalid data stream. Please help me out. Thank you so much.

import sys
import os
import bz2
from bz2 import decompress

path = "Dir"
for(dirpath,dirnames,files)in os.walk(path):
   for file in files:
       filepath = os.path.join(dirpath,filename)
       newfile = bz2.decompress(file)
       newfilepath = os.path.join(dirpath,newfile)

Upvotes: 29

Views: 47232

Answers (4)

fantabolous
fantabolous

Reputation: 22706

This is much faster for large files because it incrementally writes the output, without storing the entire decompressed file in memory:

import bz2,shutil
filepath = 'test.txt.bz2'
with bz2.BZ2File(filepath) as fr, open(filepath[:-4],"wb") as fw:
    shutil.copyfileobj(fr,fw)

Credit to https://stackoverflow.com/a/49073452/3427777

Upvotes: 0

Jochen Ritzel
Jochen Ritzel

Reputation: 107598

bz2.decompress takes compressed data and inflates it. You pass a filename, not the data in the file!

Do this instead:

zipfile = bz2.BZ2File(filepath) # open the file
data = zipfile.read() # get the decompressed data
newfilepath = filepath[:-4] # assuming the filepath ends with .bz2
open(newfilepath, 'wb').write(data) # write a uncompressed file

Upvotes: 27

Juraj Ivančić
Juraj Ivančić

Reputation: 725

bz2.compress/decompress work with binary data:

>>> import bz2
>>> compressed = bz2.compress(b'test_string')
>>> compressed
b'BZh91AY&SYJ|i\x05\x00\x00\x04\x83\x80\x00\x00\x82\xa1\x1c\x00 \x00"\x03h\x840"
P\xdf\x04\x99\xe2\xeeH\xa7\n\x12\tO\x8d \xa0'
>>> bz2.decompress(compressed)
b'test_string'

In short - you need to process file contents manually. In case you have very large files you should prefer using bz2.BZ2Decompressor to bz2.decompress, because the latter requires that you store the entire file in a byte array.

for filename in files:
    filepath = os.path.join(dirpath, filename)
    newfilepath = os.path.join(dirpath,filename + '.decompressed')
    with open(newfilepath, 'wb') as new_file, open(filepath, 'rb') as file:
        decompressor = BZ2Decompressor()
        for data in iter(lambda : file.read(100 * 1024), b''):
            new_file.write(decompressor.decompress(data))

You can also use bz2.BZ2File to make this even simpler:

for filename in files:
    filepath = os.path.join(dirpath, filename)
    newfilepath = os.path.join(dirpath, filename + '.decompressed')
    with open(newfilepath, 'wb') as new_file, bz2.BZ2File(filepath, 'rb') as file:
        for data in iter(lambda : file.read(100 * 1024), b''):
            new_file.write(data)

Upvotes: 27

michaelmeyer
michaelmeyer

Reputation: 8205

This should work

for file in files:
    archive_path = os.path.join(dirpath,filename)
    outfile_path = os.path.join(dirpath, filename[:-4])
    with open(archive_path, 'rb') as source, open(outfile_path, 'wb') as dest:
        dest.write(bz2.decompress(source.read()))

Upvotes: 3

Related Questions