CODEWITHSUNDEEP
python-3.xpython-asyncio
Jonathan
Jonathan

Reputation: 11375

Asyncio connecting to socket fails unexpectedly

I'm trying to lookup about 10,000 domains via whois with the following code.

async def lookup(server, port, query, sema):
    async with sema as sema:
        try:
            reader, writer = await asyncio.open_connection(server, port)
        except:
            return {}
        writer.write(query.encode("ISO-8859-1"))
        await writer.drain()
        data = b""
        while True:
            d = await reader.read(4096)
            if not d:
                break
            data += d
        writer.close()
        data = data.decode("ISO-8859-1")
        return data

However I repeatedly get the error 'Connect Failed'. If I try a single lookup it goes through which means the whois server is up. I've also increased the ulimit to 10,000 but I'm limiting lookups to only a 1000 at a time with a semaphore.

Upvotes: 0

Views: 295

Answers (1)

Martijn Pieters
Martijn Pieters

Reputation: 1124978

The whois server is almost certainly rate limiting you. Not all whois servers are built to scale to 1000s of concurrent connections from a single IP address.

Limit your rate further, lower the semaphore limit or switch to a leaky bucket rate limiter.

Alternatively, find a whois API provider that offers higher query rate options, or better yet, supports bulk queries.

Upvotes: 1

Related Questions