CODEWITHSUNDEEP
javascriptpythonweb-scrapingdryscrape
denski
denski

Reputation: 2040

JS web-scraping doesn't find some elements

I'm wanting to use Dryscrape to scrape hotel room price results from Google. for example the rhs_block here https://www.google.co.uk/search?q=The+Taj+Mahal+Palace+hotel

However it doesn't seem to render and then collect javascript and I'm wondering where I might be going wrong.?

import dryscrape
from bs4 import BeautifulSoup

dryscrape.start_xvfb()
session = dryscrape.Session()

my_url = 'https://www.google.ie/search?q=The+Taj+Mahal+Palace+hotel'
session.visit(my_url)
response = session.body()
soup = BeautifulSoup(response, "lxml")

# prices = soup.find('div', {"class" : "rhs_block"})
prices = soup.find('div', {"class" : "lhpr-content-item"})

print prices

I've tested this on a simple js render page, so it does work. Any pointers would be greatly appreciated, as dryscrape is fairly new to me.

Upvotes: 0

Views: 269

Answers (1)

Dan-Dev
Dan-Dev

Reputation: 9430

Google doesn't like you User Agent setting. Under this line:

session = dryscrape.Session()

Add the line:

session.set_header("User-Agent", "Mozilla/5.0 (Windows NT 5.1; rv:41.0) Gecko/20100101 Firefox/41.0")

Then it will output:

<div class="lhpr-content-item" data-key="8"><div class="_qS"><a class="_dkf" data-dp="€176" data-pid="8" data-ved="0ahUKEwisoqOeisjVAhXCI1AKHWhQAm0QwDEIzAEoBDAW" href="https://www.google.com/travel/clk?pc=AA80Osxnd1Ycj04hDym-ZpFIn9a-iLsqE7UNxtLtnVS5khTT2PvlxyLaBSJZKt9V3zLJWmUBQJedYFG2CzsGB4Ru572oiGIF3i-UYsg1BBFNbDFPhXelW-FNo6lefLaSbCcPqO1W6rOEQT_ev6stedzfqyjT2Y7QnMNz5TGkr1zDWIfI6iQgV2l7mcMhzxHV7GKVjTjhX6KL-CT3c_9wBPpKVa1MICyikHUOf72incZ6e9TF1aMGcNKf6W91fdU__ZJOv3jByF7bkPQNOWM" onmousedown="return rwt(this,'','','','23','AFQjCNG0CN8A7n-gxtETpYwsGydozaH1Yg','','0ahUKEwisoqOeisjVAhXCI1AKHWhQAm0QwDEIzAEoBDAW','','',event)"><img alt="The Taj Mahal Palace" class="_Tjf" data-deferred="1" id="zemJWeyHEsLHwALooInoBguid_8" onload="google&amp;&amp;google.aft&amp;&amp;google.aft(this)" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAYAAACNiR0NAAAAAXNSR0IArs4c6QAAAS9JREFUOBFjTJ9v7sXwn2EmIyOjDAMF4P///08YGBnSmahhGMgdYAcBHcZEqcuQPQUyiwlZgBpsuIGTYw8xTE84zsDMxEKRuXADKTIFSTNW57AwsTIk2zcxaEiZMVx+fJRhweFGhn///4J9gKQXzPz84z3DxrMzGI7e3gTmY3WhubIng4G8AwMHKxeDqZIrg7GCM7o5cD4vhyBDsGkenI/VQJBrkAE6H1kOxAZZDANYvXzy7g4GDUkTiJefHGU493A/TD1BGquBIBfNB4YbOQDDwClxh8kxB64Hw8C///7AJYlhoKdbDAPzlzgykGIoKDMgA6yxjKyAVDbVDWTMmG/xn1RXIKtnYmQGloBA+P8fWBgjDJEVE8NGT/RM4JKWGJ1EqAGZxQQqtqlhKKwKAAB1/VzfnOVWVwAAAABJRU5ErkJggg=="/><div class="_uFf"><img alt="book action chevron" class="_hEj" onload="google&amp;&amp;google.aft&amp;&amp;google.aft(this)" src="https://www.gstatic.com/images/icons/material/system/2x/chevron_right_grey600_24dp.png"/><div class="_akf"><span><span class="_bkf"><span class="_FQr"><span class="_V0p">€176</span></span></span></span></div><div class="_zbu"><span class="_Zjf">The Taj Mahal Palace</span><span class="_aMr"><span class="_bMr"> · </span><span>Official website</span></span></div></div></a></div></div>

Upvotes: 1

Related Questions