CODEWITHSUNDEEP
python-3.xweb-scrapingbeautifulsoup
UnderConfident
UnderConfident

Reputation: 41

Web scraping: How to tell in general if a page has content rendered in javascript

How can you tell in general when a website is rendering content in javascript? I typically use bs4 to scrape and I when I can't find a tag, I'm not sure if its because its javascript rendered (which bs4 cant detect) or if I did something wrong.

Upvotes: 3

Views: 537

Answers (1)

Rusty Robot
Rusty Robot

Reputation: 1845

Compare the output of your request with the html returned from a browser request. In Chrome and Firefox, press F12 and the console will appear. Under the network tab you can see all the requests that have been made. If the Network tab is empty, refresh the page. The response from the first request in the Network tab should match the response you recieved from the Python request. If it doesn't match, either your request differs from the browser request, or javascript is doing some post processing.

Subsequent requests in the Network tab may be from javascript running, from iframes, images, or much more.

Upvotes: 3

Related Questions