CODEWITHSUNDEEP
pythonurllib
Shamik
Shamik

Reputation: 1609

Does urllib.request.urlopen(url) use cache?

I have this long list of URL that I need to check response code of, where the links are repeated 2-3 times. I have written this script to check the response code of each URL. 

connection =urllib.request.urlopen(url)
return connection.getcode()

The URL comes in XML in this format 

 < entry key="something" > url</entry>
 < entry key="somethingelse" > url</entry>

and I have to associate the response code with the attribute Key so I don't want to use a SET.

Now I definitely don't want to make more than 1 request for the same URL so I was searching whether urlopen uses cache or not but didn't find a conclusive answer. If not what other technique can be used for this purpose.

Upvotes: 1

Views: 2136

Answers (3)

Arsh Singh
Arsh Singh

Reputation: 2116

You can store the urls in a dictionary (urls = {}) as you make a request and check if you have already made a req to that url later:

if key not in urls:
  connection = urllib.request.urlopen(url)
  urls[key] = url
  return connection.getcode()

BTW if you make requests to the same urls repeatedly (multiple runs of the script), and need a persistent cache, i recommend using requests with requests-cache

Upvotes: 2

djhoese
djhoese

Reputation: 3667

How are you associating the URL with the key? A dictionary?

You can use a dictionary to map the URL to it's response and any other information you need to keep track of. If the URL is already in the dictionary then you know the response. So you have one dictionary:

url_cache = {
    "url1" : ("response", [key1,key2])
    }

If you need to organize things differently it shouldn't be too hard with another dictionary.

Upvotes: 1

Uku Loskit
Uku Loskit

Reputation: 42040

Why don't you create a python set() of the URLs? That way each url is included only once.

Upvotes: 2

Related Questions