CODEWITHSUNDEEP
pythondnspython-requests
Kantium
Kantium

Reputation: 652

DNS timeout on 'requests' python library

For my project, I have to check the status of a website (on a shared hosting).

I use Python requests library.

def getStatusCode(url):
    try:
        return requests.head(url,timeout=0.3).status_code
    except:
        return -1

This code works great under MacOS 10.10 with Python3.4 with an url like http://www.google.com. If I unplug my ISP cable, I immediately got an exception.

Under Ubuntu Server 14.04 with Python3.4, if I unplug my ISP cable, I never get a timeout error. Same problem on Raspbian.

After some tests, if I replace the url with an IP http://216.58.212.100, Ubuntu Server raise me an exception, but as I'm on a shared web hosting so I can't use an IP.

After some research I found there is a difference between timeout in requests library and DNS lookup that not performed by it but by the OS.

So my question is what is the most beautiful way to solve this ? Do I need to add extra timeout exception in Python like : Timeout on a function call

Thank you

Upvotes: 6

Views: 4855

Answers (2)

Charlie
Charlie

Reputation: 2231

Now that I have a better understanding of your problem - I think there's a better approach which is to to use your OS ping application which shouldn't be hard to do in Python - for example. You should also average 1000s of requests and look at mean, standard deviation, outliers, etc. The reason for this is that if one request takes say 500ms and you want a resolution of 1ms you will need to spawn at least 500 requests to get anything close to the resolution you want.

The problem with using Pythons urllib(2) is that it won't perform nearly as well as a system level call so you'll have difficulty spawning enough threads to get the timing resolution you want.

Finally, I would check your result again a commercial product to make sure your results are similar. For example (no affiliation): http://www.thinkbroadband.com/ping.

Upvotes: 0

Kantium
Kantium

Reputation: 652

Based on Charlie's encouragement, I post here my two solutions

For the first one I added the host in the request header, so I can put the IP address as url and avoir DNS lookup.

def getStatusCode(url):
    headers = {'host': 'www.example.com'}
    try:
        return requests.head(url,timeout=0.3,headers=headers).status_code
    except:
        return -1

print(getStatusCode('http://1.2.3.4'))

The second solution is based on the use of signals but have a resolution of one second.

class timeout:
    def __init__(self, seconds=1, error_message='Timeout'):
        self.seconds = seconds
        self.error_message = error_message
    def handle_timeout(self, signum, frame):
        raise TimeoutError(self.error_message)
    def __enter__(self):
        signal.signal(signal.SIGALRM, self.handle_timeout)
        signal.alarm(self.seconds)
    def __exit__(self, type, value, traceback):
        signal.alarm(0)

def getStatusCode(url):
    try:
        return requests.head(url,timeout=0.3).status_code
    except:
        return -1

with timeout(seconds=1):
    print(getStatusCode('http://www.example.com'))

(This solution is from Thomas Ahle at https://stackoverflow.com/a/22348885/3896729)

Upvotes: 1

Related Questions