CODEWITHSUNDEEP
python-3.xdownloadurllib
Rankinstudio
Rankinstudio

Reputation: 612

Python 3 urllib urlretrieve retry on download fail

I have a program that downloads files from a server. They are anywhere from 2mb to 8mb. It runs through a loop and grabs the number of files I request. The problem is that my internet sucks out here in the middle of the freekin' desert. While everything works beautifully most of the time, sometimes the internet drops during a urllib.request.urlretrieve request and freezes up the program. I need a way to get urllib to detect when the net has dropped, and retry the file until it comes back up again. Any help appreciated!!!

Example of what I am doing:

try:
    numimgs = len(imgsToGet)

    path1 = "LEVEL II" #HIGHTEST FOLDER
    self.fn = imgs.split('/')[-1] #SPLIT OUT NAME FROM LEFT
    path2 = self.fn[:4] #SPLIT OUT KICX
    path3 = self.fn.split('_')[1] #SPLIT OUT DATE
    savepath = os.path.join(path1, path2, path3) #LEVEL II / RADAR / DATE PATH

    if not os.path.isdir(savepath): #See if it exists
        os.makedirs(savepath) #If not, make it

    fileSavePath = os.path.join(path1, path2, path3, self.fn)

    if os.path.isfile(fileSavePath): #chcek to see if image path already exists
        self.time['text'] = self.fn + ' exists \n'
        continue

    #DOWNLOAD PROGRESS
    def reporthook(blocknum, blocksize, totalsize):
        percent = 0
        readsofar = blocknum * blocksize
        if totalsize > 0:
            percent = readsofar * 1e2 / totalsize
            if percent >= 100:
                percent = 100

            s = "\r%5.1f%% %*d / %d" % (
                percent, len(str(totalsize)), readsofar, totalsize)

            self.time['text'] = 'Downloading File: '+str(curimg)+ ' of '+str(numimgs)+' '+self.fn+'' + s

            if readsofar >= totalsize: # near the end
                self.time['text'] = "Saving File..."
        else: # total size is unknown
            self.time['text'] = "read %d\n" % (readsofar)

        #UPDATE PROGRESSBAR
        self.pb.config(mode="determinate")
        if percent > 0:
            self.dl_p = round(percent,0)
            self.pb['value'] = self.dl_p
            self.pb.update()
        if percent > 100:
            self.pb['value'] = 0
            self.pb.update()

    urllib.request.urlretrieve(imgs, fileSavePath, reporthook)

except urllib.error.HTTPError as err: #catch 404 not found and continue
    if err.code == 404:
        self.time['text'] = ' Not Found'
        continue

Cheers,

David

Upvotes: 4

Views: 4781

Answers (2)

jarzyn
jarzyn

Reputation: 327

You can wrap it up with retrying library

from retrying import retry

# will retry forever, check documentation for more examples
@retry
def download_file(imgs, fileSavePath, reporthook):
    urllib.request.urlretrieve(imgs, fileSavePath, reporthook)

Upvotes: 0

Chinmay Kamat
Chinmay Kamat

Reputation: 301

You could place the code in a try except block, with a counter. Here is what I had done:

remaining_download_tries = 15

while remaining_download_tries > 0 :
    try:
        urlretrieve(CaseLawURL,File_Path_and_Name)
        print("successfully downloaded: " + CaseLawURL)
        time.sleep(0.1)
    except:
        print("error downloading " + CaseLawURL +" on trial no: " + str(16 - remaining_download_tries))
        remaining_download_tries = remaining_download_tries - 1
        continue
    else:
        break

I hope the code is self explanatory. Regards

Upvotes: 4

Related Questions