CODEWITHSUNDEEP
pythonpython-3.xweb-scrapingpython-requests
coder101
coder101

Reputation: 455

requests.post script for webscraping not working

I'm trying to scrape some data from https://ocfs.ny.gov/main/childcare/ccfs_template.asp without having a count limit on number of records per page. The developer tools show a Post method accessing "https://apps.netforge.ny.gov/dcfs/Search/Search HTP/1.1" upon clicking Search (need to insert a space in the Name field before Search will execute).

I want to download all the data into a file. My code uses the requests.post module, but I'm not sure if I'm using it correctly. Error I get is shown below my code. Appreciate some guidance on how I should modify it. Fairly new to python.

Code as follows:

import requests, csv

dataArg={'Criteria.ModalityCode':'', 'Criteria.CountyID':'', 'Criteria.SchoolDistrict':'', 'Criteria.ZipCode':'', 'Criteria.FacilityName':'+', 'Criteria.RegistrationID':'', 'Criteria.MedicationOnly':'false', 'Criteria.NonTraditionalHoursOnly':'false', 'Criteria.ShowOpenOnly':'true', 'Criteria.ShowOpenOnly':'false', 'Paging.PageSize':''}
dataCsv = requests.post('https://apps.netforge.ny.gov/dcfs/Search/Search HTP/1.1',data=dataArg)

openFile = open('nydata', 'wb')
for chunk in dataCsv.iter_content(1000000):
    openFile.write(chunk)

open_csv = open('nydata')
csv_reader = csv.reader(open_csv)
list_data = list(csv_reader)

Error:

File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3\connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3\connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3\connection.py", line 326, in connect
    ssl_context=context)
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3\util\ssl_.py", line 329, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\ssl.py", line 814, in __init__
    self.do_handshake()
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\ssl.py", line 1068, in do_handshake
    self._sslobj.do_handshake()
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests\adapters.py", line 440, in send
    timeout=timeout
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3\connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3\util\retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='apps.netforge.ny.gov', port=443): Max retries exceeded with url: /dcfs/Search/Search%20HTTP/1.1?Criteria.ModalityCode=&Criteria.CountyID=&Criteria.SchoolDistrict=&Criteria.ZipCode=&Criteria.FacilityName=+&Criteria.RegistrationID=&Criteria.MedicationOnly=false&Criteria.NonTraditionalHoursOnly=false&Criteria.ShowOpenOnly=true&Criteria.ShowOpenOnly=false&Paging.PageSize= (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\NY.py", line 3, in <module>
    dataCsv = requests.get('https://apps.netforge.ny.gov/dcfs/Search/Search HTTP/1.1?Criteria.ModalityCode=&Criteria.CountyID=&Criteria.SchoolDistrict=&Criteria.ZipCode=&Criteria.FacilityName=+&Criteria.RegistrationID=&Criteria.MedicationOnly=false&Criteria.NonTraditionalHoursOnly=false&Criteria.ShowOpenOnly=true&Criteria.ShowOpenOnly=false&Paging.PageSize=')
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests\api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests\api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests\sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests\sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests\adapters.py", line 506, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='apps.netforge.ny.gov', port=443): Max retries exceeded with url: /dcfs/Search/Search%20HTTP/1.1?Criteria.ModalityCode=&Criteria.CountyID=&Criteria.SchoolDistrict=&Criteria.ZipCode=&Criteria.FacilityName=+&Criteria.RegistrationID=&Criteria.MedicationOnly=false&Criteria.NonTraditionalHoursOnly=false&Criteria.ShowOpenOnly=true&Criteria.ShowOpenOnly=false&Paging.PageSize= (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)'),))
>>> 
== RESTART: C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\NY.py ==
Traceback (most recent call last):
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3\connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3\connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3\connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3\connection.py", line 326, in connect
    ssl_context=context)
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3\util\ssl_.py", line 329, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\ssl.py", line 814, in __init__
    self.do_handshake()
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\ssl.py", line 1068, in do_handshake
    self._sslobj.do_handshake()
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests\adapters.py", line 440, in send
    timeout=timeout
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3\connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3\util\retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='apps.netforge.ny.gov', port=443): Max retries exceeded with url: /dcfs/Search/Search%20HTP/1.1 (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\NY.py", line 19, in <module>
    dataCsv = requests.post('https://apps.netforge.ny.gov/dcfs/Search/Search HTP/1.1',data=dataArg)
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests\api.py", line 112, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests\api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests\sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests\sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\Karun\AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests\adapters.py", line 506, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='apps.netforge.ny.gov', port=443): Max retries exceeded with url: /dcfs/Search/Search%20HTP/1.1 (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)'),))

Upvotes: 1

Views: 203

Answers (1)

Keyur Potdar
Keyur Potdar

Reputation: 7248

First of all, the POST method is accessing https://apps.netforge.ny.gov/dcfs/Search/Search and not https://apps.netforge.ny.gov/dcfs/Search/Search HTP/1.1.

Regarding SSL Cert Verification, the docs say:

Requests verifies SSL certificates for HTTPS requests, just like a web browser. By default, SSL verification is enabled, and Requests will throw a SSLError if it's unable to verify the certificate.

So, you can set verify=False to overcome this. But, note that you should't use this in a production code.

Finally, using this code will give you the page:

data = {
    'Criteria.ModalityCode': '',
    'Criteria.CountyID': '',
    'Criteria.SchoolDistrict': '',
    'Criteria.ZipCode': '',
    'Criteria.FacilityName': '+',
    'Criteria.RegistrationID': '',
    'Criteria.MedicationOnly': 'false',
    'Criteria.NonTraditionalHoursOnly': 'false',
    'Criteria.ShowOpenOnly': 'false',
    'Paging.PageSize': ''
}

dataCsv = requests.post('https://apps.netforge.ny.gov/dcfs/Search/Search', data=data, verify=False)

Upvotes: 1

Related Questions