CODEWITHSUNDEEP
pythonasync-awaitpython-requestsstreamfastapi
tamarlev
tamarlev

Reputation: 11

how to increase the chunk size when sending an http stream-upload request through request.post and receiving it through fastapi

This is fastapi http side:

from fastapi import Request
import aiofiles

@app.post('/upload')
async def upload(request: Request):
    try:
        filename = request.headers['filename']
        async with aiofiles.open(filename, 'wb') as f:
            async for chunk in request.stream():
                await f.write(chunk)
    except Exception:
        return {"message": "There was an error uploading the file"}
 
    return {"message": f"Successfuly uploaded {filename}"}

this is the request side:

import requests
import time

with open("images/1.png", "rb") as f:
    data = f.read()

url = 'http://127.0.0.1:8000/upload'
headers = {'filename': '1.png'}

start = time.time()
resp = requests.post(url=url, data=data, headers=headers)
end = time.time() - start

print(f'Elapsed time is {end} seconds.', '\n')
print(resp.json())

I spend a lot of time on this part:

async with aiofiles.open(filename, 'wb') as f:
    async for chunk in request.stream():
        await f.write(chunk)

I uploaded 256MB of file and I saw that the size of chunk of every call to request.stream() gives me 256k chunk of data (on my system) which means I will have to run f.write(chunk) around 10 times. I was wondering if there is a way to increase chunk size to 1MB so I can reduce the number of writes. My target is to improve the performance by using stream vs regular upload:

from fastapi import File, UploadFile
import aiofiles

@app.post("/upload")
async def upload(file: UploadFile = File(...)):
    try:
        async with aiofiles.open(file.filename, 'wb') as f:
            while contents := await file.read(1024): # async read chunk
                await f.write(contents)
    except Exception:
        return {"message": "There was an error uploading the file"}
    finally:
        await file.close()

    return {"message": f"Successfuly uploaded {file.filename}"}

which is very slow.

Upvotes: 1

Views: 1937

Answers (1)

Yuqi Wang
Yuqi Wang

Reputation: 243

In you request side, you can try

def gen_file_stream():
    with open(file_name, "rb") as f:
        while True:
            data = f.read(1024*1024)
            if not data:
                break
            yield data
resp=requests.post(url,data=gen_file_stream())

It may set your stream size in the chunk and help reduce your file operations

Upvotes: 1

Related Questions