CODEWITHSUNDEEP
pythonrestherokupython-requests
Bailey Smith
Bailey Smith

Reputation: 2933

Post large files to heroku with Python

I'm creating a python script that will post audio files to an API. It's mostly working; however, the application is on Heroku and large files seem to be getting nabbed by heroku's memory limit. I tried using multipart/form-data, but it seems like that's not enough. I'm getting 503 errors for anything over about 5 minutes. 

import requests
import os
import glob
from requests_toolbelt import MultipartEncoder

token = 'XXXX'
headers = {'Authorization': 'Bearer ' + token, 'Content-Type': 'application/json'}
base_url = 'https://test.com/api/'

for file in glob.glob("*.mp3"):
    files = MultipartEncoder(fields={'file': (file, open(file, "rb"), 'audio/mpeg')})
    audio_headers = {'Authorization': 'Bearer ' + token, 'Content-Type': files.content_type}
    add_file = requests.post(base_url, headers=audio_headers, data=files)

Upvotes: 2

Views: 1240

Answers (1)

andrean
andrean

Reputation: 6796

Heroku has a 30 second limit for all requests. Larger files will certainly exceed that period and thus the request times out. The only solution is to post to a 3rd party storage directly from the browser (e.g. Amazon S3), and then send a request to heroku with the location of the uploaded data. Using a background process you'll be able to download the file and process it however you want.

Upvotes: 1

Related Questions