CODEWITHSUNDEEP
pythonfilegoogle-cloud-storage
Alon Barad
Alon Barad

Reputation: 2013

Download file from GCS by chunks (from byte)

I have many .rar files in my Google Cloud Storage bucket and I'm trying to download all of them one by one, parse them and delete them locally.

Sometimes the connection is lost and then I want to continue the download from where it was stopped but for some reason when the download is done, the .rar file is corrupted.

This is my Python code:

def download_gcs_file(blob):
    start = None
    to_download_path = os.path.join('/downloads', blob.name)
    if os.path.exists(to_download_path):
        start = os.path.getsize(to_download_path)
    blob.download_to_filename(
        filename=to_download_path,
        start=start,
    )

    return to_download_path

if __name__ == '__main__':
    retry.api.retry_call(
        f=download_gcs_file,
        tries=-1,
        delay=5,
        exceptions=(
            requests.exceptions.ChunkedEncodingError,
            requests.exceptions.ReadTimeout,
            requests.exceptions.ConnectionError,
        ),
        fkwargs={
            'blob': blob,
        },

Upvotes: 1

Views: 1774

Answers (1)

Rafael Lemos
Rafael Lemos

Reputation: 5829

I don't believe that this is possible without using the official SDK, I would recommend you try using it, since it is fault tolerant and will retry and resume downloads that are in progress automatically.

You can find an example implementation using the Official Python SDK in this documentation.

Upvotes: 0

Related Questions