CODEWITHSUNDEEP
pythontimeoutpython-requests
Elhamer Yacine
Elhamer Yacine

Reputation: 15

If request timeout skip url python requests

I would like the following script to try every url in url_list, and if it exist print it exist(url) and if not print don't(url) and if request timeout skip to the next url using "requests" lib:

url_list = ['www.google.com','www.urlthatwilltimeout.com','www.urlthatdon\'t exist']

def exist:
    if request.status_code == 200:
    print"exist{0}".format(url)
else:
    print"don\'t{0}".format(url)
a = 0

while (a < 2):
url = urllist[a]
try:
    request = requests.get(url, timeout=10)
except request.timeout:#any option that is similar?
    print"timed out"
    continue
validate()
a+=1

Upvotes: 0

Views: 4207

Answers (1)

ErikR
ErikR

Reputation: 52029

Based on this SO answer below is code which will limit the total time taken by a GET request as well as discern other exceptions that may happen.

Note that in requests 2.4.0 and later you may specify a connection timeout and read timeout by using the syntax:

requests.get(..., timeout=(...conn timeout..., ...read timeout...))

The read timeout, however, only specifies the timeout between individual read calls, not a timeout for the entire request.

Code:

import requests
import eventlet
eventlet.monkey_patch()

url_list = ['http://localhost:3000/delay/0',
            'http://localhost:3000/delay/20', 
            'http://localhost:3333/',         # no server listening
            'http://www.google.com'
           ]

for url in url_list:
     try:
        with eventlet.timeout.Timeout(1):
          response = requests.get(url)
        print "OK -", url
     except requests.exceptions.ReadTimeout:
        print "READ TIMED OUT -", url
     except requests.exceptions.ConnectionError:
        print "CONNECT ERROR -", url
     except eventlet.timeout.Timeout, e:
        print "TOTAL TIMEOUT -", url
     except requests.exceptions.RequestException, e:
        print "OTHER REQUESTS EXCEPTION -", url, e

And here is an express server you can use to test it:

var express = require('express');
var sleep = require('sleep')
var app = express();

app.get('/delay/:secs', function(req, res) {
  var secs = parseInt( req.params.secs )
  sleep.sleep(secs)
  res.send('Done sleeping for ' + secs + ' seconds')  
});

app.listen(3000, function () {
  console.log('Example app listening on port 3000!');
});

Upvotes: 2

Related Questions