CODEWITHSUNDEEP
pythonpostcurlpython-requestsmultipartform-data
Michael Helwig
Michael Helwig

Reputation: 530

Python equivalent to curls --form: Create multipart form-data post request with data in "form" parameter

I'm looking for a python equivalent to this curl command:

 curl --referer "https://myreferer" --insecure --form "myparam=1234" https://myurl

which results in the following request (taken from httpbin.org/post):

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "myparam": "1234"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Connection": "close", 
    "Content-Length": "142", 
    "Content-Type": "multipart/form-data; boundary=----------------------------29a1ce32cc53", 
    "Host": "httpbin.org", 
    "Referer": "https://speedport.ip/hcti_start_passwort.stm", 
    "User-Agent": "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3", 
    "X-Request-Id": "c67c4461-89d2-4c9f-a9f4-ebfe312c026c"
  }, 
...

As you can see, the data "myparam" is delivered in a "form" parameter.

I tried to construct such a request via pythons requests module and came close with this code:

import requests
payload={'myparam':'1234'}
url="http://httpbin.org/post"
headers={'User-Agent': 'Mozilla 5.0','referer':'https://myreferer'}
r = requests.post(url, files=payload, headers=headers,verify=False)

But the requests library puts the data in the "files" parameter. So the resulting request looks like this:

{
  "args": {}, 
  "data": "", 
  "files": {
    "pws": "1234"
  }, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Connection": "close", 
    "Content-Length": "143", 
    "Content-Type": "multipart/form-data; boundary=a878ad29e28d47ffb00e0631319ed0e2", 
    "Host": "httpbin.org", 
    "Referer": "https://myreferer", 
    "User-Agent": "Mozilla 5.0", 
    "X-Request-Id": "60f5d65e-789a-47fe-bba3-dab88f9bbb65"
...

So the data is delivered in the wrong place, namely within the "files" parameter, which makes Apache choke with a "501 Not Implemented" response.

Can somebody suggest how to do such a request in Python? (I know I could just call curl as a subprocess but since I want to do many of these requests I'd like to have a python-only soulution (which is hopefully more performant)).

And, as you may have noted, I also need to accept a self-signed certificate and send a referer header.

I would be glad if anybody could suggest an easy way to solve this.

Thanks!

Edit: I already tried using the "data"-param of the requests.post command but this results in a different content type header (application/x-www-form-urlencoded). Please note the content type header of the curl request.

Edit: What I probably need is to simply sending the right Content-Type header, multipart/form-data, via the headers param of the requests.post command. But I would also have to calculate the "boundary"-part of the multipart/form-data header string. I suppose there must be an easer way than manually constructing the header and calculating boundaries.

Upvotes: 4

Views: 5198

Answers (2)

Jan Vlcinsky
Jan Vlcinsky

Reputation: 44112

Using file-like object for files results in multipart/form-data content type

Let us prepare all what we need for the call, starting with "usual" stuff:

>>> import requests
>>> data = {"myparam": "1234"}
>>> headers = {'User-Agent': 'Mozilla 5.0','referer':'https://myreferer'}

The trick to force requests to use "multipart/form-data" is to give it at leas one file-like object.

>>> from StringIO import StringIO
>>> buff = StringIO("")

buff is now the file-like object we can pass in as value of files argument.

>>> req = requests.post(url, data=data, headers=headers, stream=True, files=buff)
>>> print req.text
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "myparam": "1234"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Connection": "close", 
    "Content-Length": "130", 
    "Content-Type": "multipart/form-data; boundary=0b3bbec1f5c844a1b7377aacfe701f02", 
    "Host": "httpbin.org", 
    "Referer": "https://myreferer", 
    "User-Agent": "Mozilla 5.0", 
    "X-Request-Id": "988a0467-1c32-45aa-a75c-fba5aa8d632e"
  }, 
  "json": null, 
  "origin": "85.160.45.204", 
  "url": "http://httpbin.org/post"
}

If you would communicate with https using self-signed certificate, use verify=False: 

>>> req = requests.post(url, data=data, headers=headers, stream=True, files=buff, verify=False)

Help for requests.request also notes, that value of verify could be "A CA_BUNDLE path", so you could explicitly make sure, that the server is using the self-signed certificate you expect. But with this I have never experimented.

Upvotes: 3

Ian Stapleton Cordasco
Ian Stapleton Cordasco

Reputation: 28777

Unfortunately if you don't want to send the data as a file, you have to use a third party library -- requests_toolbelt. Once you pip install requests-toolbelt, you can then do

from requests_toolbelt import MultipartEncoder
import requests

payload = MultipartEncoder({'myparam': '1234'})
r = requests.post(url, data=payload, headers={'Content-Type': payload.content_type})

Naturally you can also have the other headers set as well, this is just a quick example of using the toolbelt to your needs.

If you want to validate the certificate, you can pass a string with the full path to the PEM file, e.g.,

r = requests.get('https://somesite.com', verify='/Users/mhelwig/certificate.pem')

Upvotes: 3

Related Questions