CODEWITHSUNDEEP
pythonselenium
David542
David542

Reputation: 110482

How to grab headers in python selenium-webdriver

I am trying to grab the headers in selenium webdriver. Something similar to the following:

>>> import requests
>>> res=requests.get('http://google.com')
>>> print res.headers

I need to use the Chrome webdriver because it supports flash and some other things that I need to test a web page. Here is what I have so far in Selenium:

from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://login.comcast.net/login?r=comcast.net&s=oauth&continue=https%3A%2F%2Flogin.comcast.net%2Foauth%2Fauthorize%3Fclient_id%3Dxtv-account-selector%26redirect_uri%3Dhttps%3A%2F%2Fxtv-pil.xfinity.com%2Fxtv-authn%2Fxfinity-cb%26response_type%3Dcode%26scope%3Dopenid%2520https%3A%2F%2Flogin.comcast.net%2Fapi%2Flogin%26state%3Dhttps%3A%2F%2Ftv.xfinity.com%2Fpartner-success.html%26prompt%3Dlogin%26response%3D1&reqId=18737431-624b-44cb-adf0-2a85d91bd662&forceAuthn=1&client_id=xtv-account-selector')
driver.find_element_by_css_selector('#user').send_keys('[email protected]')
driver.find_element_by_css_selector('#passwd').send_keys('XXY')
driver.find_element_by_css_selector('#passwd').submit()
print driver.headers ### How to do this?

I have seen some other answers that recommend running an entire selenium server to get this information (https://github.com/derekargueta/selenium-profiler). How would I get it using something similar to the above with Webdriver?

Upvotes: 14

Views: 41480

Answers (5)

BabyPoopSoup
BabyPoopSoup

Reputation: 1

You can use the JAVASCRIPT built-in method.

It only can be done once the driver has already been created though.

from selenium import webdriver
driver = webdriver.Chrome()
# Store it in a variable and print the value
agent = driver.execute_script("return navigator.userAgent")
print(agent)
# directly print the value
print(driver.execute_script("return navigator.userAgent"))

Upvotes: 0

munish
munish

Reputation: 4654

Now, it is very easy i suppose https://pypi.org/project/selenium-wire/ it is an extension of selenium. use from seleniumwire import webdriver and proceed as usual.

Upvotes: 7

Rafael Ribeiro
Rafael Ribeiro

Reputation: 321

You could try Mobilenium, a python package (still in development) that binds BrowserMob Proxy and Selenium.

An usage example:

>>> from mobilenium import mobidriver
>>>
>>> browsermob_path = 'path/to/browsermob-proxy'
>>> mob = mobidriver.Firefox(browsermob_binary=browsermob_path)
>>> mob.get('http://python-requests.org')
301
>>> mob.response['redirectURL']
'http://docs.python-requests.org'
>>> mob.headers['Content-Type']
'application/json; charset=utf8'
>>> mob.title
'Requests: HTTP for Humans \u2014 Requests 2.13.0 documentation'
>>> mob.find_elements_by_tag_name('strong')[1].text
'Behold, the power of Requests'

Upvotes: 5

J. Does
J. Does

Reputation: 863

You can get the header via the log (source from Mma's answer)

from selenium import webdriver
import json
driver = webdriver.PhantomJS(executable_path=r"your_path")
har = json.loads(driver.get_log('har')[0]['message']) # get the log
print('headers: ', har['log']['entries'][0]['request']['headers'])

Upvotes: 2

elethan
elethan

Reputation: 17003

Unfortunately, you cannot get this information from the Selenium webdriver, nor will you be able to any time in the near future it seems. An excerpt from a very long conversation on the subject:

This feature isn't going to happen.

The gist of the main reason being, from what I gather from the discussion, that the webdriver is meant for "driving the browser", and extending the API beyond that primary goal will, in the opinion of the developers, cause the overall quality and reliability of the API to suffer.

One potential workaround that I have seen suggested in a number of places, including the conversation linked above, is to use BrowserMob Proxy, which can be used to capture HTTP content, and can be used with selenium - though the linked example does not use the Python selenium API. It does seem that there is a Python wrapper for BrowserMob Proxy, but I cannot vouch for it's efficacy since I have never used it.

Upvotes: 14

Related Questions