python requests post query fails:cookies?

Question

I am trying to execute this post but I get server error 500:

import requests
base_url = "https://www.assurland.com/ws/CarVehiculeSearch.asmx"
url = "%s/%s"% (base_url,"GetCarBodyTypeListByCarAlim")
pars ={"CarAlim":"DIES","CarType": "A7", "CodeMake": "AUDI", "FirstDrivingDate": "2015-09-22"}

with requests.Session() as s:
    r = s.post(url,data=pars)
    print r.status_code

 ## 500 

I think I need to set cookies or something else.

Thank you in advance for any help.

Answer 1

The problem is the date format that is badly handled in the site.

That error 500 carries this description:

Erreur lors de la génération du document XML. ---> Le type common.FormDataListItem n'était pas attendu. Utilisez l'attribut XmlInclude ou SoapInclude pour spécifier les types qui ne sont pas connus statiquement.

Try it from your browser on their page for a more user friendly debug: https://www.assurland.com/ws/CarVehiculeSearch.asmx?op=GetCarBodyTypeListByCarAlim

Note: "attribut XmlInclude ou SoapInclude pour spécifier les types qui ne sont pas connus statiquement"

So I tried a little search to enter date in another format.

As example:

Fri, 19 Jan 2001

or 19 Jan 2001

You get 200 OK request - in this case an empty result.

Unfortunately after a certain year, it looks that format is also not accepted: 22 Sep 2001 yields 200 OK but 22 Sep 2015 no more - it goes until 2008-12-31.

Try with another datetime format... But it looks there is anything there... take another site :)

Answer 2

Most API access points on that site appear to be broken. There is nothing wrong with your code per-se. I can't actually get the site to respond to any requests in their own webinterface, even when using a regular browser.

The main API documentation details (in French) that you can also use the SOAP protocol to create queries; Python has several SOAP client libraries you could choose from.

However, I've had no luck in getting that to work either. Using the excellent zeep library I tried to access the simpler GetMainCarMakeListByFirstDrivingDate endpoint (note that I had to use a datetime() object to model the timestamp):

>>> from zeep import Client
>>> client = Client('https://www.assurland.com/ws/CarVehiculeSearch.asmx?WSDL')
>>> client.service.GetMainCarMakeListByFirstDrivingDate(FirstDrivingDate=datetime(2015, 9, 22))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mjpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/site-packages/zeep/client.py", line 25, in __call__
    self._op_name, args, kwargs)
  File "/Users/mjpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/site-packages/zeep/wsdl/bindings/soap.py", line 109, in send
    return self.process_reply(client, operation_obj, response)
  File "/Users/mjpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/site-packages/zeep/wsdl/bindings/soap.py", line 145, in process_reply
    return self.process_error(doc, operation)
  File "/Users/mjpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/site-packages/zeep/wsdl/bindings/soap.py", line 223, in process_error
    detail=fault_node.find('detail'))
zeep.exceptions.Fault: <exception str() failed>

a Fault is the SOAP equivalent of a 500 error here.

When I enable debug logging, we can see that the server is having problems producing a response:

>>> client.service.GetMainCarMakeListByFirstDrivingDate(FirstDrivingDate=datetime(2015, 9, 22))
zeep.transports: HTTP Post to https://www.assurland.com/ws/CarVehiculeSearch.asmx:
<?xml version='1.0' encoding='utf-8'?>
<soap-env:Envelope xmlns:soap-env="http://schemas.xmlsoap.org/soap/envelope/">
  <soap-env:Body>
    <ns0:GetMainCarMakeListByFirstDrivingDate xmlns:ns0="http://tempuri.org/">
      <ns0:FirstDrivingDate>2015-09-22T00:00:00</ns0:FirstDrivingDate>
    </ns0:GetMainCarMakeListByFirstDrivingDate>
  </soap-env:Body>
</soap-env:Envelope>

zeep.transports: HTTP Response from https://www.assurland.com/ws/CarVehiculeSearch.asmx (status: 500):
<?xml version="1.0" encoding="utf-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><soap:Body><soap:Fault><faultcode>soap:Server</faultcode><faultstring>Le serveur n'a pas pu traiter la demande. ---&gt; Erreur lors de la génération du document XML. ---&gt; Le type common.FormDataListItem n'était pas attendu. Utilisez l'attribut XmlInclude ou SoapInclude pour spécifier les types qui ne sont pas connus statiquement.</faultstring><detail /></soap:Fault></soap:Body></soap:Envelope>
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mjpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/site-packages/zeep/client.py", line 25, in __call__
    self._op_name, args, kwargs)
  File "/Users/mjpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/site-packages/zeep/wsdl/bindings/soap.py", line 109, in send
    return self.process_reply(client, operation_obj, response)
  File "/Users/mjpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/site-packages/zeep/wsdl/bindings/soap.py", line 145, in process_reply
    return self.process_error(doc, operation)
  File "/Users/mjpieters/Development/venvs/stackoverflow-2.7/lib/python2.7/site-packages/zeep/wsdl/bindings/soap.py", line 223, in process_error
    detail=fault_node.find('detail'))
zeep.exceptions.Fault: <exception str() failed>

Hidden in there is the error message:

Le serveur n'a pas pu traiter la demande. ---> Erreur lors de la génération du document XML. ---> Le type common.FormDataListItem n'était pas attendu. Utilisez l'attribut XmlInclude ou SoapInclude pour spécifier les types qui ne sont pas connus statiquement.

or, with some Google Translate help, in English:

The server could not process the request. ---> Error generating the XML document. ---> The type common.FormDataListItem was not expected. Use XmlInclude or SoapInclude attribute to specify types that are not known statically.

Since we are not sending any common.FormDataListItem types, and the message complains that no XML document could be generated, this looks like a server-side programming error to me.

There are methods that do work, either via SOAP:

>>> client.service.GetCarTypeListByCodeMake(CodeMake='BMW', FirstDrivingDate=datetime(2016, 1, 1))
['I3', 'I8', 'M2', 'M3', 'M4', 'M5', 'M6', 'SERIE 1 II', 'SERIE 2', 'SERIE 3 VI', 'SERIE 4', 'SERIE 5', 'SERIE 6', 'SERIE 7', 'X1', 'X3', 'X4', 'X5', 'X6', 'Z4']
>>> client.service.GetAllCarTypeListByCodeMake(CodeMake='BMW')
['1502', '1600', '1602', '1800', '1802', '2000', '2002', '2500', '3,0', '3,3', '315', '316', '318', '320', '323', '324', '325', '328', '330', '518', '520', '523', '524', '525', '528', '530', '535', '540', '545', '550', '628', '630', '633', '635', '645', '650', '725', '728', '730', '732', '733', '735', '740', '745', '750', '760', '840', '850', 'I3', 'I8', 'L7', 'M2', 'M3', 'M4', 'M5', 'M535', 'M6', 'M635', 'SERIE 1', 'SERIE 1 II', 'SERIE 2', 'SERIE 3', 'SERIE 3 (SUITE)', 'SERIE 3 VI', 'SERIE 4', 'SERIE 5', 'SERIE 6', 'SERIE 7', 'X1', 'X3', 'X4', 'X5', 'X6', 'Z1', 'Z3', 'Z4', 'Z8']

or via posting application/x-www-form-urlencoded data with requests:

>>> response = requests.post('https://www.assurland.com/ws/CarVehiculeSearch.asmx/GetCarTypeListByCodeMake', data={'CodeMake': 'BMW', 'FirstDrivingDate': datetime(2016, 1, 1)})
>>> print(response.text)
<?xml version="1.0" encoding="utf-8"?>
<ArrayOfAnyType xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://tempuri.org/">
  <anyType xsi:type="xsd:string">I3</anyType>
  <anyType xsi:type="xsd:string">I8</anyType>
  <anyType xsi:type="xsd:string">M2</anyType>
  <anyType xsi:type="xsd:string">M3</anyType>
  <anyType xsi:type="xsd:string">M4</anyType>
  <anyType xsi:type="xsd:string">M5</anyType>
  <anyType xsi:type="xsd:string">M6</anyType>
  <anyType xsi:type="xsd:string">SERIE 1 II</anyType>
  <anyType xsi:type="xsd:string">SERIE 2</anyType>
  <anyType xsi:type="xsd:string">SERIE 3 VI</anyType>
  <anyType xsi:type="xsd:string">SERIE 4</anyType>
  <anyType xsi:type="xsd:string">SERIE 5</anyType>
  <anyType xsi:type="xsd:string">SERIE 6</anyType>
  <anyType xsi:type="xsd:string">SERIE 7</anyType>
  <anyType xsi:type="xsd:string">X1</anyType>
  <anyType xsi:type="xsd:string">X3</anyType>
  <anyType xsi:type="xsd:string">X4</anyType>
  <anyType xsi:type="xsd:string">X5</anyType>
  <anyType xsi:type="xsd:string">X6</anyType>
  <anyType xsi:type="xsd:string">Z4</anyType>
</ArrayOfAnyType>

You'll probably have to contact the maintainers of this API to get this sorted out, this is not something you can get to work on the Python side.