CODEWITHSUNDEEP
python-3.xhttpsurllib3
bdemchak
bdemchak

Reputation: 273

In urllib3, HTTP request hangs ... but doesn't in curl

(Thanks for taking a look at this!)

I'm trying to use python3 and simple urllib3 http.request to read HTML from https://login.morganstanleyclientserv.com.

It seems like the server is resetting the connection, and eventually urllib3's retries give up.

Is there a TLS negotiation issue here? If so, how can urllib3 compensate?

Or is the problem elsewhere? How to troubleshoot this?

I have tried the identical(?) transaction using curl ... it returns the expected HTML without any delay.

I also tried reading from a different site (e.g., https://client.schwab.com/Login/SignOn/CustomerCenterLogin.aspx) ... no problem.

Chrome loads https://login.morganstanleyclientserv.com without problem.

uname -a ; python3 -V returns:

Linux ubuntu 4.18.0-17-generic #18~18.04.1-Ubuntu SMP Fri Mar 15 15:27:12 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Python 3.6.7

This is the curl that works:

curl -v --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" --header "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3" --header "Accept-Encoding: text/plain" --header "Accept-Language: en-US,en;q=0.9" --output foo  https://login.morganstanleyclientserv.com

This is the python3 + urllib3 code that hangs (after printing 1, then 2, but not anything else):

import urllib3
import certifi

print (1)
try:
    http = urllib3.PoolManager(cert_reqs = 'CERT_REQUIRED', 
                               ca_certs = certifi.where())

    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36',
               'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3',
               'Accept-Encoding': 'text/plain',
               'Accept-Language':'en-US,en;q=0.9'
               }
    print (2)
# *** This hangs ***
    r = http.request("GET", "https://login.morganstanleyclientserv.com", headers)
    print (3)
    print (r.data)
    print (4)
except Exception as e:
    print(e)
except:
    print("error")

Upvotes: 5

Views: 1579

Answers (1)

bdemchak
bdemchak

Reputation: 273

As a python newbie, I neglected to name the headers parameter in the http.request call. It should have read:

r = http.request("GET", "https://login.morganstanleyclientserv.com", headers=headers)

Thanks to Edeki!

Upvotes: 1

Related Questions