CODEWITHSUNDEEP
web-scrapingpython-requestshttpresponse
workin 4weekend
workin 4weekend

Reputation: 371

list of URLs, try URL request 3 times, if condition, move to next URL

I have a list of URLs to access for scraping. If the status code is not 200, i want to try 2 more times. If the status code is not 200 in 3 times, I want it to move to the next URL in the list.

This is what I am thinking code is for example

url_list = [url1, url2, url3, url4, url5]

for url in url_list:
    requests.get(url)
    if requests.status_code == 200:
       do scraping

    if requests.status_code != 200:
       try url again 2 more times then move on to next url in list

I have tried defining a function, ranges, adding i = 0 , then i =+ 1 as a loop, and raising exceptions

Upvotes: 0

Views: 305

Answers (1)

chitown88
chitown88

Reputation: 28640

There might be a better way, but essentially just piggy backing off your idea, do something like:

for url in url_list:
    try_again = 3
    while try_again != 0:
        response = requests.get(url)
        if response.status_code == 200:
            print('do scraping')
            try_again = 0

        if response.status_code != 200:
            try_again -= 1
            print ('Request failed. Attempts remaining: %s' %try_again)
            if try_again == 0:
                print ('Tried 3 times. Moving to next URL.')

Upvotes: 1

Related Questions