how many requests does the code below make to the server?

Question

from lxml import etree
from selenium import webdriver
url = 'https://mountainproject.com'
driver.get(url) #I know this makes a request
html = driver.page_source #does this line also make a request?
dom = etree.HTML(html) #does this line also make a request?

I dont want to be making to many requests to the url so could any clarify if the 5th/6th line of code makes a request to the server?

Thank you.

Answer 1

There are actually quite a few requests made under the hood when executing your code. I found it makes the following requests:

https://mountainproject.com/ 301 text/html; charset=iso-8859-1
https://www.mountainproject.com/ 200 text/html; charset=UTF-8
https://www.mountainproject.com/css/ap-vendor-full.css?id=fa0c443b73aac8a18b4b 200 text/css
https://www.mountainproject.com/css/styles-shared-all.css?id=1e17ec741b720d85e959 200 text/css
https://www.mountainproject.com/css/styles-site-climb.css?id=712703d159d0ec101dd2 200 text/css
https://www.mountainproject.com/css/froala_editor.pkgd.min-3-1-0.css 200 text/css
https://www.mountainproject.com/js/ap-vendor-full.js?id=1945ee864736d4f641db 200 text/javascript
https://www.mountainproject.com/js/sites/all.js?id=a65a925fba88e4d1fc48 200 text/javascript
https://www.mountainproject.com/js/climb-main.js?id=d7aff6c924e13b9f5aa3 200 text/javascript
https://appleid.cdn-apple.com/appleauth/static/jsapi/appleid/1/en_US/appleid.auth.js 200 application/javascript;charset=UTF-8
https://www.mountainproject.com/img/icons/closeX-lightBg.svg 200 image/svg+xml
https://www.mountainproject.com/img/arrows/nextGray.svg 200 image/svg+xml
https://www.mountainproject.com/img/social/white-facebook.svg 200 image/svg+xml
https://www.mountainproject.com/img/climb/logoHexInvert.svg 200 image/svg+xml
https://www.mountainproject.com/img/hamburgerIcon.svg 200 image/svg+xml
https://www.mountainproject.com/img/apple.png 200 image/png
https://www.mountainproject.com/img/climb/logoHex2.svg 200 image/svg+xml
https://www.mountainproject.com/img/android.png 200 image/png
https://www.mountainproject.com/fonts/RobotoCondensed700.woff2 200 None
https://www.mountainproject.com/img/arrowLeft.svg 200 image/svg+xml
https://www.mountainproject.com/fonts/roboto-v19-latin-regular.woff2 200 None
https://www.mountainproject.com/img/arrowRight.svg 200 image/svg+xml
https://www.mountainproject.com/img/stars/starBlueHalf.svg 200 image/svg+xml
https://www.mountainproject.com/img/stars/starBlue.svg 200 image/svg+xml
https://www.mountainproject.com/img/partners/climbingMag.png 200 image/png
https://www.mountainproject.com/js/maps/maps.js?id=b47ec6c62549d52ee912 200 text/javascript
https://cdn2.apstatic.com/photos/climb/114286419_sqsmall_1524441422.jpg 200 image/jpeg
https://www.mountainproject.com/js/components/search/main.js?id=0448767c13131f4c9e6c 200 text/javascript
https://www.mountainproject.com/img/arrowRightBlack.svg 200 image/svg+xml
https://www.mountainproject.com/img/appleBlack.svg 200 image/svg+xml
https://www.mountainproject.com/img/androidBlack.svg 200 image/svg+xml
https://www.mountainproject.com/img/climb/appIcon.svg 200 image/svg+xml
https://www.mountainproject.com/img/partners/af-black.png 200 image/png
https://www.mountainproject.com/img/camera.svg 200 image/svg+xml
https://cdn2.apstatic.com/photos/climb/2200035_medium_1557851488.jpg 200 image/jpeg
https://www.mountainproject.com/img/partners/af-4.png 200 image/png
https://www.mountainproject.com/img/partners/alpinist.png 200 image/png
https://www.mountainproject.com/img/searchWhite.svg 200 image/svg+xml
https://www.googletagmanager.com/gtm.js?id=GTM-KX76S82 200 application/javascript; charset=UTF-8
https://www.mountainproject.com/img/waitWhite.gif 200 image/gif
https://www.mountainproject.com/img/icons/closeX-darkBg.svg 200 image/svg+xml
https://www.googletagmanager.com/gtag/js?id=G-TZ07N3455D&l=dataLayer&cx=c 200 application/javascript; charset=UTF-8
https://www.gstatic.com/charts/loader.js?_=1621032662221 200 text/javascript
https://www.google.com/recaptcha/api.js?onload=prepCaptchas&render=explicit 200 text/javascript; charset=UTF-8
https://www.google-analytics.com/analytics.js 200 text/javascript
https://connect.facebook.net/en_US/fbevents.js 200 application/x-javascript; charset=utf-8
https://www.google-analytics.com/g/collect?v=2&tid=G-TZ07N3455D&gtm=2oe5c1&_p=1314722123&sr=1792x1120&ul=en-us&cid=103056203.1621032664&_s=1&dl=https%3A%2F%2Fwww.mountainproject.com%2F&dt=Rock%20Climbing%20Guides%3A%20Routes%2C%20Photos%20%26%20Forum%20%7C%20Mountain%20Project&sid=1621032663&sct=1&seg=0&en=page_view&_fv=1&_nsi=1&_ss=1 204 text/plain
https://connect.facebook.net/signals/config/438433750939555?v=2.9.39&r=stable 200 application/x-javascript; charset=utf-8
https://www.mountainproject.com/photos/news/climbing-8e4bb4ceb980b8d477178f8608699457.jpg 200 image/jpeg
https://www.mountainproject.com/photos/news/accessfund-5e92e2476376733db69523c0d9f571c0.jpg 200 image/jpeg
https://www.gstatic.com/charts/50/loader.js 200 text/javascript
https://www.google-analytics.com/gtm/js?id=GTM-KHB2CJL&t=gtm2&cid=103056203.1621032664 200 application/javascript; charset=UTF-8
https://www.gstatic.com/recaptcha/releases/f-bnnOuahiYKuei7dmAd3kgv/recaptcha__en.js 200 text/javascript
https://www.facebook.com/tr/?id=438433750939555&ev=PageView&dl=https%3A%2F%2Fwww.mountainproject.com%2F&rl=&if=false&ts=1621032664161&sw=1792&sh=1120&v=2.9.39&r=stable&ec=0&o=30&fbp=fb.1.1621032664160.290931468&it=1621032663879&coo=false&exp=l1&rqm=GET 200 image/gif
https://www.google-analytics.com/j/collect?v=1&_v=j90&a=1314722123&t=pageview&_s=1&dl=https%3A%2F%2Fwww.mountainproject.com%2F&ul=en-us&de=UTF-8&dt=Rock%20Climbing%20Guides%3A%20Routes%2C%20Photos%20%26%20Forum%20%7C%20Mountain%20Project&sd=30-bit&sr=1792x1120&vp=1200x926&je=0&_u=aCDAAAADQAAAAC~&jid=1618144214&gjid=1075963567&cid=103056203.1621032664&tid=UA-407891-1&_gid=1600094628.1621032664&_r=1&gtm=2wg5c1KX76S82&z=1139774133 200 text/plain
https://www.gstatic.com/charts/50/css/core/tooltip.css 200 text/css
https://www.gstatic.com/charts/50/css/util/util.css 200 text/css
https://www.gstatic.com/charts/50/js/jsapi_compiled_default_module.js 200 text/javascript
https://www.gstatic.com/charts/50/js/jsapi_compiled_graphics_module.js 200 text/javascript
https://www.gstatic.com/charts/50/js/jsapi_compiled_ui_module.js 200 text/javascript
https://stats.g.doubleclick.net/j/collect?t=dc&aip=1&_r=3&v=1&_v=j90&tid=UA-407891-1&cid=103056203.1621032664&jid=1618144214&gjid=1075963567&_gid=1600094628.1621032664&_u=aCDAAAACQAAAAC~&z=492030330 200 text/plain
https://www.gstatic.com/charts/50/js/jsapi_compiled_corechart_module.js 200 text/javascript
https://www.facebook.com/tr/?id=438433750939555&ev=Microdata&dl=https%3A%2F%2Fwww.mountainproject.com%2F&rl=&if=false&ts=1621032664666&cd[DataLayer]=%5B%5D&cd[Meta]=%7B%22title%22%3A%22Rock%20Climbing%20Guides%3A%20Routes%2C%20Photos%20%26%20Forum%20%7C%20Mountain%20Project%22%2C%22meta%3Adescription%22%3A%22Find%20rock%20climbing%20routes%2C%20photos%2C%20and%20guides%20for%20every%20state%2C%20along%20with%20experiences%20and%20advice%20from%20fellow%20climbers.%22%7D&cd[OpenGraph]=%7B%22og%3Atitle%22%3A%22Rock%20Climbing%20Guides%3A%20Routes%2C%20Photos%20%26%20Forum%20%7C%20Mountain%20Project%22%2C%22og%3Asite_name%22%3A%22Mountain%20Project%22%2C%22og%3Atype%22%3A%22website%22%2C%22og%3Adescription%22%3A%22Find%20rock%20climbing%20routes%2C%20photos%2C%20and%20guides%20for%20every%20state%2C%20along%20with%20experiences%20and%20advice%20from%20fellow%20climbers.%22%2C%22og%3Aimage%22%3A%22https%3A%2F%2Fcdn2.apstatic.com%2Fclimb%2F111569437_medium_1494295483.jpg%22%2C%22og%3Aurl%22%3A%22https%3A%2F%2Fwww.mountainproject.com%22%7D&cd[Schema.org]=%5B%5D&cd[JSON-LD]=%5B%5D&sw=1792&sh=1120&v=2.9.39&r=stable&ec=1&o=30&fbp=fb.1.1621032664160.290931468&it=1621032663879&coo=false&es=automatic&tm=3&exp=l1&rqm=GET 200 image/gif
https://secure.quantserve.com/quant.js 200 application/javascript
https://rules.quantcount.com/rules-p-7h0NMm9UmA2V9.js 200 application/javascript
https://pixel.quantserve.com/pixel;r=1699603804;source=gtm;rf=0;a=p-7h0NMm9UmA2V9;url=https%3A%2F%2Fwww.mountainproject.com%2F;uh=65ce955d6ae6;uht=2;fpan=1;fpa=P0-724355007-1621032665310;pbcn=u;pbc=;ns=0;ce=1;qjs=1;qv=1558287b-20210421211215;cm=;gdpr=0;ref=;d=mountainproject.com;je=0;sr=1792x1120x30;dst=1;et=1621032665310;tzo=420;ogl=title.Rock%20Climbing%20Guides%3A%20Routes%252C%20Photos%20%26%20Forum%20%7C%20Mountain%20Project%2Csite_name.Mountain%20Project%2Ctype.website%2Cdescription.Find%20rock%20climbing%20routes%252C%20photos%252C%20and%20guides%20for%20every%20state%252C%20along%20with%20experi%2Cimage.https%3A%2F%2Fcdn2%252Eapstatic%252Ecom%2Fclimb%2F111569437_medium_1494295483%252Ejpg%2Curl.https%3A%2F%2Fwww%252Emountainproject%252Ecom 200 image/gif

How do I know this? This is a perfect use case for seleniumwire, which automatically captures all http requests made. I just used this code to return all the requests:

from lxml import etree
from seleniumwire import webdriver
import chromedriver_autoinstaller

chromedriver_autoinstaller.install()
options = webdriver.ChromeOptions()

driver = webdriver.Chrome(options=options)
url = 'https://mountainproject.com'
driver.get(url) #I know this makes a request
html = driver.page_source #does this line also make a request?
dom = etree.HTML(html) #does this line also make a request?
for request in driver.requests:

    print(
        request.url,
        request.response.status_code,
        request.response.headers['Content-Type']
    )

I'm not sure if there's a way to determine which requests are made by which line of code, maybe someone else can help if you need to figure that out.