CODEWITHSUNDEEP
scrapy
ic10503
ic10503

Reputation: 130

Scrapy shell XPATH not working

I tried the following XPATH in XPATHHelper in Chrome and XPather in Firefox and it always displays all the snippets(ie the description of the search results) in google search result page, but it does not work in the Scrapy shell:

//span[@class='st']

In case it matters, I invoke scrapy shell like this: 

scrapy  shell "http://www.google.com/search?q=myQuery"

and I say hxs.select("//span[@class='st']"). This always returns an empty list. Any clues as to why this could be happening?

Upvotes: 1

Views: 933

Answers (3)

hugsbrugs
hugsbrugs

Reputation: 3621

In Firefox url bar type : about:config find the line javascript.enable and change its value to false

Install FireFinder extension Open Firebug (F12)

and then enjoy scraping google like xpath expression :

//*[@id="search"]//li[@class="g"]/div[@class="s"]//cite

Upvotes: 0

Chris Hawkes
Chris Hawkes

Reputation: 12430

sometimes sites will not work with Javascript Disabled (Applebees.com for example) so you have to use an actual browser like Selenium.

Upvotes: 0

DrColossos
DrColossos

Reputation: 12998

Scrapy is not able to "parse" sites that need Javascript execution. What different developer consoles show you is the already interpreted and executed site with all Javascripts applied.

Since Google displays its resulst with the help of Javascript, the Scrapy on its own can't handle this.

Upvotes: 1

Related Questions