CODEWITHSUNDEEP
pythonseleniumpython-requests
West
West

Reputation: 2570

How to use Requests library with Selenium, in Python

I would like to log into a webpage using Selenium and use the logged in session to do subsequent requests using the Requests library. My code so far is as shown:

from selenium import webdriver
import requests

driver = webdriver.Chrome()
driver.get("https://www.linkedin.com/uas/login?")

Once I get to the log in page I simply put in my login details then once logged in I want to be able to get info from specific pages using the requests library instead. How can I get this working?

Upvotes: 8

Views: 19261

Answers (1)

West
West

Reputation: 2570

Ok just figured this out for anyone with this challenge. Its simply passing the cookies from selenium to the requests session:

from selenium import webdriver
import requests

driver = webdriver.Chrome()
driver.get("https://www.linkedin.com/uas/login?")

s = requests.Session()
# Set correct user agent
selenium_user_agent = driver.execute_script("return navigator.userAgent;")
s.headers.update({"user-agent": selenium_user_agent})

for cookie in driver.get_cookies():
    s.cookies.set(cookie['name'], cookie['value'], domain=cookie['domain'])

response = s.get("https://linkedin/example_page.com")

Upvotes: 27

Related Questions