CODEWITHSUNDEEP
pythonftpftplibgzip
Luca
Luca

Reputation: 1810

Retrieve data from gz file on FTP server without writing it locally

I would like to retrieve the data inside a compressed gz file stored on an FTP server, without writing the file to the local archive.

At the moment I have done

from ftplib import FTP
import gzip

ftp = FTP('ftp.server.com')
ftp.login()  
ftp.cwd('/a/folder/')

fileName = 'aFile.gz'

localfile = open(fileName,'wb')
ftp.retrbinary('RETR '+fileName, localfile.write, 1024)

f = gzip.open(localfile,'rb')
data = f.read()

This, however, writes the file "localfile" on the current storage.

I tried to change this in

from ftplib import FTP
import zlib

ftp = FTP('ftp.server.com')
ftp.login()  
ftp.cwd('/a/folder/')

fileName = 'aFile.gz'

data = ftp.retrbinary('RETR '+fileName, zlib.decompress, 1024)

but, ftp.retrbinary does not output the output of its callback. Is there a way to do this?

Upvotes: 2

Views: 2713

Answers (1)

Martin Prikryl
Martin Prikryl

Reputation: 202272

A simple implementation is to:

import gzip
from io import BytesIO
import shutil
from ftplib import FTP

ftp = FTP('ftp.example.com')
ftp.login('username', 'password')

flo = BytesIO()

ftp.retrbinary('RETR /remote/path/archive.tar.gz', flo.write)

flo.seek(0)

with open('archive.tar', 'wb') as fout, gzip.GzipFile(fileobj = flo) as gzip:
    shutil.copyfileobj(gzip, fout)

The above loads whole .gz file to a memory. What can be inefficient for large files. A smarter implementation would stream the data instead. But that would probably require implementing a smart custom file-like object.

See also Get files names inside a zip file on FTP server without downloading whole archive.

Upvotes: 6

Related Questions