Posting Multi-part form encoded dict

Question

I am trying to use a third party API to upload a file. The API documentation is listed here: https://developers.procore.com/reference/project-folders-and-files#create-project-file

The API documents describe uploading a multipart/form-data body (RFC 2388) with the following information:

{
  "file": {
    "parent_id": 12,
    "name": "test_file.pdf",
    "is_tracked": true,
    "explicit_permissions": true,
    "description": "This file is good",
    "data": "foobar",
    "upload_uuid": "1QJ83Q56CVQR4X3C0JG7YV86F8",
    "custom_field_%{custom_field_definition_id}": custom field value
  }
}

I am using the python requests library and am NOT able to get this to work. I have tried using the data={} field in the requests.post() but this does not send it as multipart/form-data, I have tried using files field with the above passed in as a dict however I get the following error: TypeError: a bytes-like object is required, not 'dict'

I have no idea how to get this to work, thanks for any insight anyone can provide.

What has been tried:

sendData = {
    "file": {
        "parent_id":procoreDetails['destFolderId'],
        "name":f.name,
        "is_tracked": True,
        "explicit_permissions": True,
        "upload_uuid":instructions["uuid"]
    }
}

r = requests.post(
    headers = headers,
    url = settings.API_PROC_BASE+"/vapid/files",
    params={"project_id": procoreDetails['projectId']},
    files=sendData
)

Trace with error:

Traceback (most recent call last):
  File "./d2processor.py", line 68, in <module>
    d2processor()
  File "./d2processor.py", line 60, in d2processor
    procoreDetails=procoreDetails
  File "/mnt/gluster-vol1/Source/d2/autoAnnotate.py", line 991, in classify
    files=sendData
  File "/home/mnewman/envs/py36/lib/python3.6/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/home/mnewman/envs/py36/lib/python3.6/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/mnewman/envs/py36/lib/python3.6/site-packages/requests/sessions.py", line 516, in request
    prep = self.prepare_request(req)
  File "/home/mnewman/envs/py36/lib/python3.6/site-packages/requests/sessions.py", line 459, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/home/mnewman/envs/py36/lib/python3.6/site-packages/requests/models.py", line 317, in prepare
    self.prepare_body(data, files, json)
  File "/home/mnewman/envs/py36/lib/python3.6/site-packages/requests/models.py", line 505, in prepare_body
    (body, content_type) = self._encode_files(files, data)
  File "/home/mnewman/envs/py36/lib/python3.6/site-packages/requests/models.py", line 169, in _encode_files
    body, content_type = encode_multipart_formdata(new_fields)
  File "/home/mnewman/envs/py36/lib/python3.6/site-packages/urllib3/filepost.py", line 90, in encode_multipart_formdata
    body.write(data)
TypeError: a bytes-like object is required, not 'dict'

This leads to the following error: TypeError: a bytes-like object is required, not 'dict'

However this does force requests to use multipart/form-data body

I have also attempted to use:

sendData = {
    "file": {
        "parent_id":procoreDetails['destFolderId'],
        "name":f.name,
        "is_tracked": True,
        "explicit_permissions": True,
        "upload_uuid":instructions["uuid"]
    }
}

r = requests.post(
    headers = headers,
    url = settings.API_PROC_BASE+"/vapid/files",
    params={"project_id": procoreDetails['projectId']},
    data=sendData
)

Error message from trying to send it with the data field:

Post to Procore: 400 - {"errors":"param is missing or the value is empty: file"} 

Here is the exact header the above sends (which does not do what the API documents above require (i.e. its using application/x-www-form-urlencoded instead of the required multipart/form-data)):

{'User-Agent': 'python-requests/2.23.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Authorization': 'Bearer <key removed>', 'Procore-Company-Id': '<removed?', 'Content-Length': '83', 'Content-Type': 'application/x-www-form-urlencoded'}

Doing this requests DOES NOT use the multipart/form-data header. When I send this to the server it reports that the required "file" field is missing or empty.

The documentation (linked above) for the api endpoint make it explicit that the above body structure must be sent as multipart/form-data body (RFC 2388) however I can not seem to find a way to do this in requests.

Answer 1

So the solution was provided by the API developer and I am posting it here as it is a syntax nuance that would be applicable in other situations.

Recap, the issue is getting requests to send the following structure with multipart/form-data header:

{
  "file": {
    "parent_id": 12,
    "name": "test_file.pdf",
    "is_tracked": true,
    "explicit_permissions": true,
    "description": "This file is good",
    "data": "foobar",
    "upload_uuid": "1QJ83Q56CVQR4X3C0JG7YV86F8",
    "custom_field_%{custom_field_definition_id}": custom field value
  }
}

The solution is to use the following code to build the above structure:

sendData = {
    "file[parent_id]":procoreDetails['destFolderId'],
    "file[name]":fNameUpload,
}
files = [
    ('file[data]',open(f.filePath,'rb'))
]
r = requests.post(
    headers = headers,
    url = settings.API_PROC_BASE+"/vapid/files",
    params={"project_id": procoreDetails['projectId']},
    files=files,
    data=sendData
)

Note the use of file[parent_id] structure that allows requests to build the proper body nested dict. I can not actually use a nested dict passed to data= or files=.

Answer 2

HTTP data is sent and received as strings, and it’s the responsibility of the server and client to serialize and deserialize as required. The error message you’re getting (TypeError: a bytes-like object is required, not 'dict') basically means “you gave me a dictionary, but I want a string.” (Python’s bytes object is a basically string-like object for ASCII characters.)

You haven’t posted your code, but I suspect it should be something like this:

requests.post(..., data=str(my_dict))