CODEWITHSUNDEEP
pythonhttpclient-serverpython-asyncioaiohttp
George
George

Reputation: 151

Python AIOHTTP send a request right after returning a response

Please consider the following. There's a system that asks for data using HTTP POST methods. Right after sending such a request, the system waits for an HTTP response with a status code and data as separate messages. The existing system is built in a way that it won't accept a response with status code and data combined, which is, to be honest, doesn't make sense to me. On my side, I need to implement a system, which will receive such requests and provide data to clients. I decided to use the AIOHTTP library to solve this problem. As I'm very new to AIOHTTP, I can't find a way to send data back to the client right after returning a response. The existing system which sends requests also has an endpoint on its side. So, what I think of doing, is to return a response with a status code to the client and then as a client send a POST request to the provided endpoint. So my system will work both as a client and as a server. Now, what I do not understand, is how to implement this using AIOHTTP.

Let's say I have the following endpoint on my side with a handler. Please, consider this to only be pseudocode.

async def init():
    app = web.Application()
    app.add_routes([web.post('/endpoint/', handle)])
    app_runner = web.AppRunner(app)
    await app_runner.setup()
    site = web.TCPSite(runner=app_runner, host='127.0.0.1', port=8008)
    await site.start()

async def handle(request):
    data = await request.text()
    result = await process(data)  # Data processing routine. Might be time-consuming.
    await session.post(SERVER_ENDPOINT, data=result)  # Let's say I have session in this block 
                                                      # and I'm sending data back to the client.
    return web.Response(status=200)  # Returning a status without data.

Now, I need web.Response(status=200) to happen as soon as possible and only then process received data and send data back to the client. What I thought of doing is to wrap data processing and request sending in a task and adding it to a queue. Now, I always need the response to be sent first and I'm afraid that when using tasks, this might not be always true, or is it? Might the task be completed before returning a response? Is AIOHTTP good for this task? Should I consider something else?

Update #1

I've found a method called finish_response. Might it be used to implement something like this?

async def handler(self, request):
    self.finish_response(web.Response(status=200))  # Just an example.
    self.session.post(SERVER_ENDPOINT, data=my_data)
    return True  # or something

Upvotes: 0

Views: 2404

Answers (1)

RPalmer
RPalmer

Reputation: 556

aiohttp has a sibling project called aiojobs, which is used to handle background tasks. There is an example of how aiojobs integrates with aiohttp in their documentation.

So, modifying your example to work with aiojobs:

import aiojobs.aiohttp

async def init():
    app = web.Application()
    app.add_routes([web.post('/endpoint/', handle)])
    # We must setup AIOJobs from AIOHTTP app
    aiojobs.aiohttp.setup(app)
    app_runner = web.AppRunner(app)
    await app_runner.setup()
    site = web.TCPSite(runner=app_runner, host='127.0.0.1', port=8008)
    await site.start()

async def handle(request):
    data = await request.text()
    result = await process(data)  # Data processing routine. Might be time-consuming.
    # Here we create the background task
    await aiojobs.aiohttp.spawn(request, session.post(SERVER_ENDPOINT, data=result))
    # The response should return as soon as the task is created - it does not wait for the task to finish.
    return web.Response(status=200)  # Returning a status without data.

If you want the await process(data) to also be scheduled as a task, then you can move both calls into a seperate function, and schedule them together:

async def push_to_server(data):
  result = await process(data)
  await session.post(SERVER_ENDPOINT, data=result))

async def handle(request):
    data = await request.text()
    await aiojobs.aiohttp.spawn(request, push_to_server(data))
    return web.Response(status=200)

If you want to make sure the response is sent before the push_to_server coroutine is called, then you can make use of asyncio events:

import asyncio

async def push_to_server(data, start):
  await start.wait()
  result = await process(data)
  await session.post(SERVER_ENDPOINT, data=result))

async def handle(request):
    data = await request.text()
    start = asyncio.Event()
    await aiojobs.aiohttp.spawn(request, push_to_server(data, start))
    response = web.Response(status=200)
    await response.prepare(request)
    await response.write_eof()
    start.set()
    return response

Here, await response.prepare(request) and await response.write_eof() is just a long-winded way of sending the response, but allows us to call the start.set() afterwards, which will trigger the push_to_server functionality which is waiting on that event (await start.wait()).

Upvotes: 2

Related Questions