How to remove Http 401 Unauthorized error using python3

Question

I am trying to access web resource secured with HTTPBasicAuth using python3. I am providing the right credentials still it is showing 401 unauthorized error.

I am using urllib for doing it as I failed to do it using requests module because of ssl and proxy error. In urllib it just shows 401 unauthorized error hence I am able to hit that website.

Any suggestion how to solve this issue?
(I cannot post the credentials publicly for security reasons, that's why I just used *)

import urllib.request
import urllib.parse
import ssl
import requests
from requests.auth import HTTPBasicAuth
line =1
try:

    line += 1#2
    myssl = ssl.create_default_context()#Bypass SSL verification when trying to access a secured website
    myssl.check_hostname = False
    myssl.verify_mode = ssl.CERT_NONE#SSL verification disabled
    USERNAME = '******'
    PASSWORD = '******'
    login_data = dict(username=USERNAME, password=PASSWORD)


    headers = {}
    headers['User-Agent'] = "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.27 Safari/537.17"
    p = urllib.request.HTTPPasswordMgrWithPriorAuth()
    p.add_password(None,'https://bmh309668.rbeigcn.com:44330/sap/opu/odata/sap/IBAS_PLANT_MAINTENANCE_SRV/CodeGroups/',USERNAME,PASSWORD)
    handler = urllib.request.HTTPBasicAuthHandler(p)
    opener = urllib.request.build_opener(handler)
    urllib.request.install_opener(opener)


    line += 1#3
    data = urllib.parse.urlencode(login_data)
    data = data.encode('utf-8')#while using post we can send a byte format not a string so encode it to utf-8
    line += 1#4
    req = urllib.request.Request('https://bmh309668.rbeigcn.com:44330/sap/opu/odata/sap/IBAS_PLANT_MAINTENANCE_SRV/CodeGroups/',data=data, headers = headers)#request for the web page
    line += 1#5
    response = urllib.request.urlopen(req,context=myssl,data=data)#by using context=myssl we disable SSL verification
    line += 1#6
    #x = urllib.request.urlopen('https://bmh309668.rbeigcn.com:44330/sap/opu/odata/sap/IBAS_PLANT_MAINTENANCE_SRV/CodeGroups/',context=myssl,data=data)
    line += 1#7
    print(response.read())#print the data

except Exception as e:
    print("Exception raised at line number: ",line)
    print(str(e))

Getting exception at line number 5

Answer 1

Took help from Hultner's code and then disabled the urllib3 warnings. Now it is working.

import requests
import urllib3

line = 1 #line number denotes the exact place where the exception is raised
try:
    # Your configuration
    USERNAME = "******"
    PASSWORD = "******"
    URI = "https://bmh309668.rbeigcn.com:44330/sap/opu/odata/sap/IBAS_PLANT_MAINTENANCE_SRV/CodeGroups/"
    # specify the user agent
    USER_AGENT = "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.27 Safari/537.17"
    # verify the host. Skips verification if set to false
    CERT_FILE = False
    #diable the warnings
    urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
    urllib3.disable_warnings(urllib3.exceptions.HTTPError)

    # Create a Session to contain you basic AUTH, it will also persist your cookies
    line += 1#2
    authed_session = requests.Session()

    # Add your credentials
    line += 1#3
    authed_session.auth = (USERNAME, PASSWORD)

    # Cert verification, will not verify on false
    line += 1#4
    authed_session.verify = CERT_FILE

    line += 1#5
    authed_session.headers.update({'User-Agent': USER_AGENT})

    # Fetch the actual data
    line += 1#6
    fetched_data = authed_session.get(URI)
    line += 1#7
    print(fetched_data.text)
except Exception as e:
    print(line)
    print(str(e))

Answer 2

I can see that you import requests but doesn't use it. It's a excellent library for this use case.

Your core problem seems to be that you want to fetch data from a site with a self signed TLS/SSL-cert which is also protected by HTTPBasicAuth.

I've verified with a similar setup on my own server that below code should work for that case.

import requests

# Your configuration
USERNAME = "***"
PASSWORD = "***"
URI = "https://bmh309668.rbeigcn.com:44330/sap/opu/odata/sap/IBAS_PLANT_MAINTENANCE_SRV/CodeGroups/"
# Do you need to specify the user agent? Does the site run a white/blacklist?
USER_AGENT = "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.27 Safari/537.17"
# WARNING: You should point this to your cert file for the server to actually
# verify the host. Skips verification if set to false
CERT_FILE = False

# Create a Session to contain you basic AUTH, it will also persist your cookies
authed_session = requests.Session()

# Add your credentials
authed_session.auth = (USERNAME, PASSWORD)

# Cert verification, will not verify on false
authed_session.verify = CERT_FILE

authed_session.headers.update({'User-Agent': USER_AGENT})

# Fetch the actual data
fetched_data = authed_session.get(URI)
print(fetched_data.text)

It will work with CERT_FILE set to false but I highly recommend to specifying the path to the certfile for the host you use, so it can actually verify the host.