CODEWITHSUNDEEP
python
RoyMWell
RoyMWell

Reputation: 209

checking if download link works in python

Im currently using the following code to download gz file. The url of the gz file will be constructed from pieces of information provided by the user:

generalUrl = theWebsiteURL + "/" + packageName

So generalURl can contain something like: http://www.example.com/blah-0.1.0.tar.gz

        res = requests.get(generalUrl)
        res.raise_for_status()

The problem I have here is; I have a list of websites for the variable called theWebsiteURL. I need to check all of these websites to see which ones have the package in packageName available for download. I would prefer not to download the package during the confirmation.

Once the code goes through the list of websites to discover which ones have the package, I then want to pick the first website from the list of websites that were found to have the package and automatically download the package from it.

something like this:

#!/usr/bin/env python2.7
listOfWebsites = [ website1, website2, website3, website4, and so on ]
goodWebsites = []
for eachWebsite in listOfWebsites:
        genURL = eachWebsite + "/" + packageName
        res = requests.get(genUrl)
        res.raise_for_status()
        if raise_for_status == "200"
             goodWebsites.append(genURL)

This is where my imagination stops. I need assistance completing this. Not even sure I'm going about it the right way.

Upvotes: 0

Views: 1085

Answers (1)

Gevorg Davoian
Gevorg Davoian

Reputation: 524

You can try to send a HEAD request first in order to check that the URL is valid, and only then download the package via a GET request.

#!/usr/bin/env python2.7
listOfWebsites = [ website1, website2, website3, website4, and so on ]
goodWebsites = []
for eachWebsite in listOfWebsites:
    genURL = eachWebsite + "/" + packageName
    res = requests.head(genUrl)
    if res.ok:
         goodWebsites.append(genURL)

Upvotes: 5

Related Questions