CODEWITHSUNDEEP
pythonpython-3.xpython-requestsgrequests
Rafael
Rafael

Reputation: 3196

POSTing and GETing with grequests

I'm using grequests to scape websites faster. However, I also need to login to the website.

Before (just using requests) I could do:

where headers is my User-Agent. 

with requests.Session() as s: 
    s.headers.update(headers)
    s.post(loginURL, files = data)
    s.get(scrapeURL)

Using grequests I've only been able to pass headers by doing: 

rs = (grequests.get(u, headers=header) for u in urls)
response = grequests.map(rs)

Is there anyway to do a POST at the same time so I can login? The login URL is differnt than the URL(s) I'm scrapping.

Upvotes: 2

Views: 2018

Answers (2)

Padraic Cunningham
Padraic Cunningham

Reputation: 180391

You can pass in the Session object exactly the same as the headers:

with requests.Session() as s: 
    s.headers.update(headers)
    s.post(loginURL, files = data)
    s.get(scrapeURL)


    rs = (grequests.get(u, headers=header, session=s) for u in urls)
    response = grequests.map(rs)

Upvotes: 3

wim
wim

Reputation: 362557

First login the session, then pass it explicitly to your grequest like this:

requests = []
for url in urls:
    request = grequests.AsyncRequest(
        method='GET', 
        url=url, 
        session=session,
    )
    requests.append(request)

Upvotes: 3

Related Questions