CODEWITHSUNDEEP
pythonfile-uploadpython-requests
bonju0102
bonju0102

Reputation: 31

How to upload a bunch of files with Python requests module

When I using the requests module to upload at least 400 files with one request, I got the Too many open files error on macOS system.

I have tried ulimit -n 20000.

Also checked:

However, it didn't work.

Here is my code:

import os
import requests


url = 'http://127.0.0.1:8000/api/upload'
file_path = '/Users/BonJu/Projects/downloads'
file_list = os.listdir(file_path)
files = []
for file in file_list:
    try:
        source = open(os.path.join(file_path, file), 'rb')
        files.append(('file', source))

    except Exception as e:
        print('File: %s, Error: %s' % (file, e.__str__()))
        continue

response = requests.post(url=url, data={'uploader': 'admin'}, files=files)

Terminal result:

File: test_252.docx, Error: [Errno 24] Too many open files: '/Users/BonJu/Projects/downloads/test_252.docx'
File: test_253.docx, Error: [Errno 24] Too many open files: '/Users/BonJu/Projects/downloads/test_253.docx'
File: test_254.docx, Error: [Errno 24] Too many open files: '/Users/BonJu/Projects/downloads/test_254.docx'
...
File: test_418.docx, Error: [Errno 24] Too many open files: '/Users/BonJu/Projects/downloads/test_418.docx'

Because it is an API server which links the sent log files to the issue I need all the files to be sent in one request, otherwise the manager will get several mails and won't be able to address the issues.

Is there any solution for this situation?

Solution

I finally adjusted my API to save a temp file for storing the upload logs and pass a status parameter to control the final output.

my code:

payload = {
    'status': 'finish',
    'uploader': 'admin'
}
response = requests.post(url=url, data=payload, files=files)

API:

@api_view(['post'])
def upload(request, debug, api_version):
    status = request.POST.get('status')
    file_list = request.FILES.getlist('file')

    if status == 'finish':
        # open the temp file and insert the last logs then output
        
    else:
        # create/insert the logs and save to a temp file

    return Response({'status': status, 'files': file_list})

Upvotes: 3

Views: 144

Answers (1)

Mogi
Mogi

Reputation: 614

I would suggest make a single tar file from all of them.

or closing the files after reading their content (instead of leaving their file descriptor open until you post the request

Upvotes: 1

Related Questions