CODEWITHSUNDEEP
pythonxpathscrapy
Juned Ansari
Juned Ansari

Reputation: 5275

xpath doesnot extract contents using scrapy shell

I am able to fetch webpage content using XPath helper google chrome extension on website but same path if I apply using sxrapy shell it does not work.

using XPath chrome extension:

webpage = anywebsite.com

XPath = //section[@class='rslwrp']/div/ul/li//h2[@class='store-name']/span/a/span/text()

but same thing when I apply using scrapy shell it returns null why?

scrapy shell "www.mywebsite.com"
In [15]: response.xpath("//section[@class='rslwrp']/div/ul/li//h2[@class='store-
    ...: name']/span/a/span")                                                   
Out[15]: []

Upvotes: 0

Views: 51

Answers (1)

salparadise
salparadise

Reputation: 5815

It appears that setting the user-agent, it works:

scrapy shell "www.mywebsite" -s USER_AGENT="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/53.0"

Result:

In [1]: response
Out[1]: <200 https://www.mywebsite.com>

Also there was a space needed after 'rslwrp' so 'rslwrp '

In [9]: response.xpath("//section[@class='rslwrp ']/div/ul/li//h2[@class='store-name']/span/a/span")
Out[9]:
[<Selector xpath="//section[@class='rslwrp ']/div/ul/li//h2[@class='store-name']/span/a/span" data='<span class="lng_cont_name">Brand Cit...'>,
 <Selector xpath="//section[@class='rslwrp ']/div/ul/li//h2[@class='store-name']/span/a/span" data='<span class="lng_cont_name">Kuntal Ou...'>,
 <Selector xpath="//section[@class='rslwrp ']/div/ul/li//h2[@class='store-name']/span/a/span" data='<span class="lng_cont_name">Brands Cl...'>,
 <Selector xpath="//section[@class='rslwrp ']/div/ul/li//h2[@class='store-name']/span/a/span" data='<span class="lng_cont_name">4g Fashio...'>,
 <Selector xpath="//section[@class='rslwrp ']/div/ul/li//h2[@class='store-name']/span/a/span" data='<span class="lng_cont_name">Paridhan ...'>,
 <Selector xpath="//section[@class='rslwrp ']/div/ul/li//h2[@class='store-name']/span/a/span" data='<span class="lng_cont_name">Harish Ga...'>,
 <Selector xpath="//section[@class='rslwrp ']/div/ul/li//h2[@class='store-name']/span/a/span" data='<span class="lng_cont_name">Maa Vaish...'>,
 <Selector xpath="//section[@class='rslwrp ']/div/ul/li//h2[@class='store-name']/span/a/span" data='<span class="lng_cont_name">Shoppers ...'>,
 <Selector xpath="//section[@class='rslwrp ']/div/ul/li//h2[@class='store-name']/span/a/span" data='<span class="lng_cont_name">Bachoomal...'>,
 <Selector xpath="//section[@class='rslwrp ']/div/ul/li//h2[@class='store-name']/span/a/span" data='<span class="lng_cont_name">Vishal Me...'>]

Upvotes: 2

Related Questions