CODEWITHSUNDEEP
pythonpostmultipartform-datahttpx
krizzo
krizzo

Reputation: 1883

Post multipart/form-data using pythons httpx library only form data

How do I send a POST with python HTTPX that will minic a CURL POST that works? This is to an opengear rest API. I believe it has to do something with the data field of the post.

This is the working curl hitting the rest API properly.

curl -s -k \
-X POST \
-H "Authorization: Token ${TOKEN}" \
-H "Content-Type: multipart/form-data" \
-F firmware_url="https://example.com/path/to/file/file.name" \
-F firmware_options="-R" \
https://${SERVER}/api/v2/system/firmware_upgrade

Based on httpx documentation this should be a simple call with a dictionary of the fields passed in the post method. It also says that posts with files defaults to streaming. It does not clearly state posts with out files use streaming. Based on the results from the logs of the nginx server it appears that it's not a streaming process.

I'm modifying the header content-type as the server is expecting a multipart/form-data and the httpx.post defaults to the application/x-www-form-urlencoded.

The python and different types of posts that I've attempted not all at once.

# Get a session token and creates the httpx.Client object api_client.
# Multiple GETs are called from the client before trying to post.
...
# POST section
headers = {"Content-Type": "multipart/form-data"}
data = {
    "firmware_url": f"https://example.com/path/to/file/file.name",
    "firmware_options": "-R"
}

# Simple post
res = api_client.post(f"https://{SERVER}/api/v2/system/firmware_upgrade", data=data, headers=headers)

# Data converted to json string
res = api_client.post(f"https://{SERVER}/api/v2/system/firmware_upgrade", data=json.dumps(data), headers=headers)

# Data converted to byte encoded json string
byte_data = json.dumps(data).encode(encoding="utf-8")
res = api_client.post(f"https://{SERVER}/api/v2/system/firmware_upgrade", data=data, headers=headers)

The nginx logs show different lines depending on which post is used.

CURL with -F form fields (working, appears to be a stream)
::ffff:10.0.0.1 - - [24/Jan/2025:22:39:47 +0000] "POST /api/v2/system/firmware_upgrade HTTP/1.1" 200 54 "-" "curl/8.5.0" rt=0.054 uct="0.001" uht="0.054" urt="0.054"

HTTPX with raw data dict
::ffff:10.0.0.1 - - [24/Jan/2025:22:38:51 +0000] "POST /api/v2/system/firmware_upgrade HTTP/1.1" 000 0 "-" "python-httpx/0.28.1" rt=0.000 uct="-" uht="-" urt="-"
::ffff:10.0.0.1 - - [24/Jan/2025:22:38:51 +0000] "firmware_url=https%3A%2F%2Fexample.com%2Fpath%2Fto%2Ffile%2Ffile.name&firmware_options=-R" 400 157 "-" "-" rt=0.027 uct="-" uht="-" urt="-"

HTTPX with JSON dumps
::ffff:10.0.0.1 - - [24/Jan/2025:22:38:08 +0000] "POST /api/v2/system/firmware_upgrade HTTP/1.1" 000 0 "-" "python-httpx/0.28.1" rt=0.000 uct="-" uht="-" urt="-"
::ffff:10.0.0.1 - - [24/Jan/2025:22:38:08 +0000] "{\x22firmware_url\x22: \x22https://example.com/path/to/file/file.name\x22, \x22firmware_options\x22: \x22-R\x22}" 400 157 "-" "-" rt=0.028 uct="-" uht="-" urt="-"

HTTPX with JSON dumps to byte encoded
::ffff:10.0.0.1 - - [27/Jan/2025:19:01:45 +0000] "POST /api/v2/system/firmware_upgrade HTTP/1.1" 000 0 "-" "python-httpx/0.28.1" rt=0.000 uct="-" uht="-" urt="-"
::ffff:10.0.0.1 - - [27/Jan/2025:19:01:45 +0000] "{\x22firmware_url\x22: \x22https://example.com/path/to/file/file.name\x22, \x22firmware_options\x22: \x22-R\x22}" 400 157 "-" "-" rt=0.006 uct="-" uht="-" urt="-"

Upvotes: 0

Views: 163

Answers (1)

Serhii Fomenko
Serhii Fomenko

Reputation: 1030

The official documentation states that form fields should be passed using the data keyword, using the dict object.

It does not clearly state posts with out files use streaming

Just passing form fields using POST will not use streaming. To make sure of this I did a bit of searching of the source code, this can be seen here, in the Request.encode_request method, if only data is passed, we will eventually get the headers and ByteStream object as the result of the function - encode_urlencoded_data, after which it will be read immediately into memory. For example, if we send something like this: httpx.post(url=url, data={'name': 'Foo'}), on the server I will see request.body=b'name=Foo'.

Since your endpoint accepts to send either file or firmware_url, my idea is as follows::

import httpx  
  
  
class GeneratorReader:  
    def __init__(self, gen_func):  
        self.gen_func = gen_func  
        self.active_gen = None  
  
    def read(self, chunk_size=1024):  
        if self.active_gen is None:  
            self.active_gen = self.gen_func(chunk_size=chunk_size)  
        return next(self.active_gen, None)  
  
  
file_url = 'https://example.com/path/to/file/file.name'  
client = httpx.Client()  
  
with client.stream(method='GET', url=file_url) as response:  
    client.post(  
        url='https://${SERVER}/api/v2/system/firmware_upgrade',  
        files={'file': GeneratorReader(gen_func=response.iter_bytes)}, 
        headers={  
            'Authorization': 'Token ${TOKEN}',  
            'Content-Type': 'multipart/form-data',  
        },  
    )

Essentially - all this code does is read your file using the stream method so as not to load everything into memory and redirects the data to the target endpoint. The GeneratorReader, we need because files={'file': ...}, must be either a string-like or file-like object with a read method.

Try doing it this way, should work. Alternatively, you could download your remote file (for example, through httpx) and save it as a local file and use something like this: files={'file': open('file.name', 'rb')}. It might also be interesting if you suddenly need to pass some metadata.

Upvotes: 0

Related Questions