CODEWITHSUNDEEP
pythontimeoutpyramid
André Luiz
André Luiz

Reputation: 7282

Python Pyramid increase request timeout

I'm working on a Python Pyramid rest api and in one of the requests I need to process an excel and for each line of it I get the GPS coordinates and do a lot of validation, this means this only request may take some 10 minutes to process and I need to return a response to the caller with a json. I can't send it via WebSocket of anything else.

My question is: how do I increase the timeout for this request? This is my method:

@view_config(route_name='client_api_upload',
             request_method='POST',
             renderer='json',
             permission='upload client')
def client_api_upload(request):

    client_id = request.POST['client_id']
    filename = request.POST['file'].filename
    input_file = request.POST['file'].file
    extension = os.path.splitext(filename)[1]
    new_file_name = str(uuid.uuid4())+extension
    file_path = os.path.join(tempfile.gettempdir(), new_file_name)

    with open(file_path, 'wb') as output_file:
        shutil.copyfileobj(input_file, output_file)

    importer = ClientBuildingImporter(client_id=client_id, db_session=request.dbsession)
    importer.import_building_from_excel(excel_file_path=file_path)

    return importer.import_result

Thanks for any help

Upvotes: 1

Views: 903

Answers (2)

Mikko Ohtamaa
Mikko Ohtamaa

Reputation: 83758

The correct approach is not to execute long running tasks in the requests.

What you can do is

  • Make an initial request to kick off the task, returns immediately. This request writes the task to a database.
  • Have a background worker process (e.g. Celery) to process tasks
  • Have a front end polling / JavaScript / redirect loop to check if the background process has completed the task

Upvotes: 1

Sergey
Sergey

Reputation: 12407

I don't think it's Pyramid closing the connection, more likely it's the webserver (Apache/Nginx etc.) which serves your application. Pyramid itself is usually happy to just chug along as long as it is necessary.

I agree with @SamMason that there are better (but more complex) ways to properly execute long-running tasks, but in some controlled circumstances it may be fine to have a long-running request which does a lot of processing. I've had a few migration/maintenance scripts which took hours to finish, worked just fine.

The trick to keep all the participants (webserver, proxies, browser, etc.) happy is to make sure that there's a trickle of data returned by your application, don't let the HTTP connection just sit there idle.

With Pyramid, unfortunately, streaming requests is not completely straightforward but there are some resources which may point you in the right direction.

Upvotes: 2

Related Questions