How to Upload File using FastAPI?

Question

I am using FastAPI to upload a file according to the official documentation, as shown below:

@app.post("/create_file")
async def create_file(file: UploadFile = File(...)):
      file2store = await file.read()
      # some code to store the BytesIO(file2store) to the other database

When I send a request using Python requests library, as shown below:

f = open(".../file.txt", 'rb')
files = {"file": (f.name, f, "multipart/form-data")}
requests.post(url="SERVER_URL/create_file", files=files)

the file2store variable is always empty. Sometimes (rarely seen), it can get the file bytes, but almost all the time it is empty, so I can't restore the file to the other database.

I also tried the bytes rather than UploadFile, but I get the same results. Is there something wrong with my code, or is the way I use FastAPI to upload a file wrong?

Answer 1

First, as per FastAPI documentation, you need to install python-multipart—if you haven't already—as uploaded files are sent as "form data". For instance:

pip install python-multipart

The examples below use the .file attribute of the UploadFile object to get the actual Python file (i.e., SpooledTemporaryFile), which allows you to call the SpooledTemporaryFile's methods, such as .read() and .close(), without having to await them. It is important, however, to define your endpoint with def in this case—otherwise, if the endpoint was defined with async def, such operations would block the server until they are completed. In FastAPI, a normal def endpoint is run in an external threadpool that is then awaited, instead of being called directly (as it would block the server). I would highly suggest having a look at this answer, which explains the difference between def and async def endpoints, as well as provides a number of solutions when one needs to run blocking operations inside async def endpoints.

The SpooledTemporaryFile used by FastAPI/Starlette has the max_size attribute set to 1 MB, meaning that the data are spooled in memory until the file size exceeds 1 MB, at which point the data are written to a temporary file on disk, under the OS's temp directory. Hence, if you uploaded a file larger than 1 MB, it wouldn't be stored in memory, and calling file.file.read() would actually read the data from disk into memory. Thus, if the file is too large to fit into your server's RAM, you should rather read the file in chunks and process one chunk at a time, as described in "Read the File in chunks" section below.

As explained above and in this answer as well, FastAPI/Starlette uses AnyIO threads to run blocking functions, such as endpoints defined with normal def, in an external threadpool and then await them (so that FastAPI would still work asynchronously), in order to prevent them from blocking the event loop (of the main thread), and hence, the entire server. Therefore, every time an HTTP request arrives at an endpoint defined with normal def, a new thread will be spawned (or an idle thread will be used, if available), and thus, depending on the requirements of your project, the expected traffic (i.e., number of users simultaneously accessing your API), as well as any other blocking functions in your API that will eventually run in that threadpool (see the linked answers above for more details), you might need to adjust the maximum number of threads in that threadpool (see the linked answer above on how to do that).

However, you should always aim at using asynchronous code (i.e., using async/await), wherever is possible, as async code runs directly in the event loop which runs in a single thread (in this case, the main thread). One option would be to define the endpoint with async def and use the asynchronous read()/write()/close()/etc. file methods provided by FastAPI, as demonstrated in this answer. You should, however, note that, as explained in this answer, FastAPI, behind the scenes, actually calls the corresponding synchronous Python File methods in a separate thread from the external threadpool described earlier. Thus, it might or might not make a big difference (always perform and compare tests before choosing one approach over the other).

Note that in the bottom part of this answer, as well as in this answer, another approach is explained and demonstrated on how to upload large files in chunks, using Starlette's request.stream(), which results in considerably minimizing the time required to upload files, as well as avoiding the use of threads from that threadpool. Thus, I would highly recommend taking a look.

Upload Single File

app.py

from fastapi import File, UploadFile, HTTPException

@app.post("/upload")
def upload(file: UploadFile = File(...)):
    try:
        contents = file.file.read()
        with open(file.filename, 'wb') as f:
            f.write(contents)
    except Exception:
        raise HTTPException(status_code=500, detail='Something went wrong')
    finally:
        file.file.close()

    return {"message": f"Successfully uploaded {file.filename}"}

Read the File in chunks

As described earlier and in this answer, if the file is too big to fit into memory—for instance, if you have 8GB of RAM, you can't load a 50GB file (not to mention that the available RAM will always be less than the total amount installed on your machine, as other applications will be using some of the RAM)—you should rather load the file into memory in chunks and process the data one chunk at a time. This method, however, may take longer to complete, depending on the chunk size you choose—in the example below, the chunk size is 1024 * 1024 bytes (i.e., 1MB). You can adjust the chunk size as desired.

from fastapi import File, UploadFile, HTTPException
        
@app.post("/upload")
def upload(file: UploadFile = File(...)):
    try:
        with open(file.filename, 'wb') as f:
            while contents := file.file.read(1024 * 1024):
                f.write(contents)
    except Exception:
        raise HTTPException(status_code=500, detail='Something went wrong')
    finally:
        file.file.close()

    return {"message": f"Successfully uploaded {file.filename}"}

Another option would be to use shutil.copyfileobj(), which is used to copy the contents of a file-like object to another file-like object (have a look at this answer too). By default, the data is read in chunks with the default buffer (chunk) size being 1MB (i.e., 1024 * 1024 bytes) for Windows and 64KB for other platforms, as shown in the source code here. You can specify the buffer size by passing the optional length parameter. Note: If negative length value is passed, the entire contents of the file will be read instead—see f.read() as well, which .copyfileobj() uses under the hood (as can be seen in the source code here).

from fastapi import File, UploadFile, HTTPException
import shutil
        
@app.post("/upload")
def upload(file: UploadFile = File(...)):
    try:
        with open(file.filename, 'wb') as f:
            shutil.copyfileobj(file.file, f)
    except Exception:
        raise HTTPException(status_code=500, detail='Something went wrong')
    finally:
        file.file.close()
        
    return {"message": f"Successfully uploaded {file.filename}"}

test.py (using requests)

import requests

url = 'http://127.0.0.1:8000/upload'
file = {'file': open('images/1.png', 'rb')}
resp = requests.post(url=url, files=file) 
print(resp.json())

test.py (using httpx)

import httpx

url = 'http://127.0.0.1:8000/upload'
file = {'file': open('images/1.png', 'rb')}
resp = httpx.post(url=url, files=file) 
print(resp.json())

For an HTML <form> example, see here.

Upload Multiple (List of) Files

app.py

from fastapi import File, UploadFile, HTTPException
from typing import List

@app.post("/upload")
def upload(files: List[UploadFile] = File(...)):
    for file in files:
        try:
            contents = file.file.read()
            with open(file.filename, 'wb') as f:
                f.write(contents)
        except Exception:
            raise HTTPException(status_code=500, detail='Something went wrong')
        finally:
            file.file.close()

    return {"message": f"Successfuly uploaded {[file.filename for file in files]}"}    

Read the Files in chunks

As described earlier in this answer, if you expect some rather large file(s) and don't have enough RAM to accommodate all the data from the beginning to the end, you should rather load the file into memory in chunks, thus processing the data one chunk at a time (Note: adjust the chunk size as desired, below that is 1024 * 1024 bytes).

from fastapi import File, UploadFile, HTTPException
from typing import List

@app.post("/upload")
def upload(files: List[UploadFile] = File(...)):
    for file in files:
        try:
            with open(file.filename, 'wb') as f:
                while contents := file.file.read(1024 * 1024):
                    f.write(contents)
        except Exception:
            raise HTTPException(status_code=500, detail='Something went wrong')
        finally:
            file.file.close()
            
    return {"message": f"Successfuly uploaded {[file.filename for file in files]}"}   

or, using shutil.copyfileobj():

from fastapi import File, UploadFile, HTTPException
from typing import List
import shutil

@app.post("/upload")
def upload(files: List[UploadFile] = File(...)):
    for file in files:
        try:
            with open(file.filename, 'wb') as f:
                shutil.copyfileobj(file.file, f)
        except Exception:
            raise HTTPException(status_code=500, detail='Something went wrong')
        finally:
            file.file.close()

    return {"message": f"Successfuly uploaded {[file.filename for file in files]}"}  

test.py (using requests)

import requests

url = 'http://127.0.0.1:8000/upload'
files = [('files', open('images/1.png', 'rb')), ('files', open('images/2.png', 'rb'))]
resp = requests.post(url=url, files=files) 
print(resp.json())

test.py (using httpx)

import httpx

url = 'http://127.0.0.1:8000/upload'
files = [('files', open('images/1.png', 'rb')), ('files', open('images/2.png', 'rb'))]
resp = httpx.post(url=url, files=files) 
print(resp.json())

For an HTML <form> example, see here.

Answer 2

@app.post("/create_file/")
async def image(image: UploadFile = File(...)):
    print(image.file)
    # print('../'+os.path.isdir(os.getcwd()+"images"),"*************")
    try:
        os.mkdir("images")
        print(os.getcwd())
    except Exception as e:
        print(e) 
    file_name = os.getcwd()+"/images/"+image.filename.replace(" ", "-")
    with open(file_name,'wb+') as f:
        f.write(image.file.read())
        f.close()
   file = jsonable_encoder({"imagePath":file_name})
   new_image = await add_image(file)
   return {"filename": new_image}