CODEWITHSUNDEEP
pythonhttpurllib2httplibpython-requests
user179169
user179169

Reputation:

Python-Requests close http connection

I was wondering, how do you close a connection with Requests (python-requests.org)?

With httplib it's HTTPConnection.close(), but how do I do the same with Requests?

Code:

r = requests.post("https://stream.twitter.com/1/statuses/filter.json", data={'track':toTrack}, auth=('username', 'passwd'))
for line in r.iter_lines():
    if line:
        self.mongo['db'].tweets.insert(json.loads(line))

Upvotes: 76

Views: 241516

Answers (8)

user1995868
user1995868

Reputation: 243

Based on the latest requests(2.25.1), the requests.<method> will close the connection by default

with sessions.Session() as session:
    return session.request(method=method, url=url, **kwargs)

https://github.com/psf/requests/blob/master/requests/api.py#L60

Thus, if you use the latest version of requests, it seems we don't need to close the connection by ourselves.

Also, if you need to send multiple times of requests with the same session, it's better to use requests.Session() instead of open/close the connection multiple times. EX:

with requests.Session() as s:
    r = s.get('https://example.org/1/')
    print(r.text)
    r = s.get('https://example.org/2/')
    print(r.text)
    r = s.get('https://example.org/3/')
    print(r.text)

Upvotes: 3

stwhite
stwhite

Reputation: 3275

I came to this question looking to solve the "too many open files" error, but I am using requests.session() in my code. A few searches later and I came up with an answer on the Python Requests Documentation which suggests to use the with block so that the session is closed even if there are unhandled exceptions:

with requests.Session() as s:
    s.get('http://google.com')

If you're not using Session you can actually do the same thing: https://2.python-requests.org/en/master/user/advanced/#session-objects

with requests.get('http://httpbin.org/get', stream=True) as r:
    # Do something

Upvotes: 75

allergique
allergique

Reputation: 71

To remove the "keep-alive" header in requests, I just created it from the Request object and then send it with Session

headers = {
'Host' : '1.2.3.4',
'User-Agent' : 'Test client (x86_64-pc-linux-gnu 7.16.3)',
'Accept' : '*/*',
'Accept-Encoding' : 'deflate, gzip',
'Accept-Language' : 'it_IT'
}

url = "https://stream.twitter.com/1/statuses/filter.json"
#r = requests.get(url, headers = headers) #this triggers keep-alive: True
s = requests.Session()
r = requests.Request('GET', url, headers)

Upvotes: -1

user1990614
user1990614

Reputation: 39

this works for me:

res = requests.get(<url>, timeout=10).content
requests.session().close()

Upvotes: 2

Oleg Gryb
Oleg Gryb

Reputation: 5259

I think a more reliable way of closing a connection is to tell the sever explicitly to close it in a way compliant with HTTP specification:

HTTP/1.1 defines the "close" connection option for the sender to signal that the connection will be closed after completion of the response. For example,

   Connection: close

in either the request or the response header fields indicates that the connection SHOULD NOT be considered `persistent' (section 8.1) after the current request/response is complete.

The Connection: close header is added to the actual request:

r = requests.post(url=url, data=body, headers={'Connection':'close'})

Upvotes: 81

hatcher
hatcher

Reputation: 401

please use response.close() to close to avoid "too many open files" error

for example:

r = requests.post("https://stream.twitter.com/1/statuses/filter.json", data={'track':toTrack}, auth=('username', 'passwd'))
....
r.close()

Upvotes: 30

Wilfred Hughes
Wilfred Hughes

Reputation: 31181

On Requests 1.X, the connection is available on the response object:

r = requests.post("https://stream.twitter.com/1/statuses/filter.json",
                  data={'track': toTrack}, auth=('username', 'passwd'))

r.connection.close()

Upvotes: 13

Felix Fung
Felix Fung

Reputation: 1757

As discussed here, there really isn't such a thing as an HTTP connection and what httplib refers to as the HTTPConnection is really the underlying TCP connection which doesn't really know much about your requests at all. Requests abstracts that away and you won't ever see it.

The newest version of Requests does in fact keep the TCP connection alive after your request.. If you do want your TCP connections to close, you can just configure the requests to not use keep-alive.

s = requests.session()
s.config['keep_alive'] = False

Upvotes: 34

Related Questions