CODEWITHSUNDEEP
pythonasynchronouspython-asyncioaiohttp
Ritik Patel
Ritik Patel

Reputation: 69

How to resolve OSError: [Errno 24] Too many open files when performing POST calls with three different URLs using aiohttp?

I have csv file from which one column serves as the data that is passed in two of the POST method calls. I have three different URLs which need to be called subsequently in the code. I define the session using aiohttp and loop for the column length(one whose cell values are sent in two of the POST calls). The data obtained from the responses is stored in a list and that is used for a third POST call with different url.

The first two calls seem to execute fine and pretty fast(i printed the responses for them and verified), but when execution comes to the third one, this error shows up:

OSError: [Errno 24] Too many open files

I have tried some solutions which suggest to specify connector=aiohttp.TCPConnector(verify_ssl=False) in aiohttp.ClientSession(connector=aiohttp.TCPConnector(verify_ssl=False)) But this doesn't work. What should be the best way to tackle this? The second URL curently that i use is an HTTP one, localhost url, which will be changed to https cloud based url eventually.

Here is the example code depicting the situation:

import aiohttp
import pandas as pd
import asyncio
import requests
import json
import time

start_time = time.time()

df = pd.read_csv('Sample.csv', header=None, dtype=str)
RequestBodyvalues = df.iloc[:, [0]].values

async def get_FirstUrlresponse(session, url, requestBody, headers):
    async with session.post(url, data = json.dumps(requestBody), headers = headers) as resp:
        response = await resp.json()
        return response

async def get_SecondUrlresponse(session, url, requestBody, headers):
    async with session.post(url, data = json.dumps(requestBody), headers = headers) as resp:
        response = await resp.json()
        return response

async def get_ThirdUrlresponse(session, url, requestBody, headers):
    async with session.post(url, data = json.dumps(requestBody), headers = headers) as resp:
        response = await resp.json()
        return response

async def main():
    async with aiohttp.ClientSession() as session:
        FirstUrlTasks = []
        SecondUrlTasks = []
        for reqBody in RequestBodyvalues:
            firstUrl = 'https://firstUrl.com/searchByValue'
            secondUrl = 'http://secondUrl.com/someEndpoint'
            requestBody = {'value': reqBody}
            headers = {'Authorization' : 'Bearer Token', 
            'content-type': 'application/json'}

            FirstUrlTasks.append(asyncio.ensure_future(get_FirstUrlresponse(session, firstUrl, requestBody, headers)))
            SecondUrlTasks.append(asyncio.ensure_future(get_SecondUrlresponse(session, secondUrl, requestBody, headers)))
        
        firstUrlResponses = await asyncio.gather(*FirstUrlTasks)
        secondUrlresponses = await asyncio.gather(*SecondUrlTasks)

        valuesForThridUrl = []
        for secondUrlresponse in secondUrlresponses:
            #Logic to fetch values to pass to Third Url stored in list

        ThirdUrlTasks = []
        for value in valuesForThridUrl:
            ThirdUrl = 'https://thirdUrl.com/someEndpoint'
            requestBody = {'reqBody': value}
            headers = {'Authorization' : 'Bearer Token', 
            'content-type': 'application/json'}

            ThirdUrlTasks.append(asyncio.ensure_future(get_ThirdUrlresponse(session, ThirdUrl, requestBody, headers)))

        thirdUrlresponses = await asyncio.gather(*ThirdUrlTasks)

asyncio.run(main())

Upvotes: 5

Views: 4530

Answers (1)

Ali Irani
Ali Irani

Reputation: 607

See open file limits with this command

ulimit -n

and then increase limit

ulimit -n NUM

Upvotes: 9

Related Questions