CODEWITHSUNDEEP
python-3.xaws-lambdapython-requests
LG555
LG555

Reputation: 165

Python requests timeout in AWS Lambda

Python requests is timing out from AWS Lambda for below url's.

https://www.nseindia.com/get-quotes/derivatives?symbol=NIFTY https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY

And its working for other urls.

Working url - https://support.oneskyapp.com/hc/en-us/article_attachments/202761627/example_1.json

Below is the code I am using.

import requests

def lambda_handler(event, lambda_context):
    headers = {"User-Agent":'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36'}
    
    with requests.session() as rq:
        print("Loading cookies")
        rq.get("https://www.nseindia.com/get-quotes/derivatives?symbol=NIFTY", headers=headers)

        print("Execute get request")
        result = rq.get("https://www.nseindia.com/api/option-chain-indices?symbol=NIFTY", headers=headers)

My Lambda is using Python3.7 runtime, not associated with any VPC and timeout is set to 1 minute. I tried with Lambda execution timeout of 5 minutes still no luck.

Output

Response: { "errorMessage": "2020-11-17T04:58:34.039Z 0f69a07d-6139-4aab-b1cf-310b19243735 Task timed out after 60.06 seconds" }

Request ID: "0f69a07d-6139-4aab-b1cf-310b19243735"

Function logs: START RequestId: 0f69a07d-6139-4aab-b1cf-310b19243735 Version: $LATEST Starting request session... Loading cookies END RequestId: 0f69a07d-6139-4aab-b1cf-310b19243735 REPORT RequestId: 0f69a07d-6139-4aab-b1cf-310b19243735 Duration: 60060.48 ms Billed Duration: 60000 ms Memory Size: 128 MB Max Memory Used: 59 MB Init Duration: 383.95 ms 2020-11-17T04:58:34.039Z 0f69a07d-6139-4aab-b1cf-310b19243735 Task timed out after 60.06 seconds

I am guessing this would be related to User-Agent headers. Any help would be highly appreciated.

Upvotes: 4

Views: 999

Answers (1)

ivica
ivica

Reputation: 1426

It could be that https://www.nseindia.com/ is blocking AWS IPs because they don't want to be scraped?

This is what I got when trying to open it from one of my EC2 machines

[ec2-user@ip-10-2-120-90 ~]$ curl https://www.nseindia.com/get-quotes/derivatives?symbol=NIFTY
<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>

You don't have permission to access "http&#58;&#47;&#47;www&#46;nseindia&#46;com&#47;get&#45;quotes&#47;derivatives&#63;" on this server.<P>
Reference&#32;&#35;18&#46;d6d3dead&#46;1605602802&#46;2598bb12
</BODY>
</HTML>

Upvotes: 1

Related Questions