CODEWITHSUNDEEP
python-2.7xpathscrapyscrapy-shell
Mxracer888
Mxracer888

Reputation: 341

How do I extract text with the Xpath in Scrapy?

Alright, it seems like I've tried everything and just can't figure out what is going on here. The direct link to the site I am trying to scrape is http://www.ammofast.com/collections/pistol/products/aguila-25-auto-acp-full-metal-jacket-50-gr-1000-round-case

And I want to extract the title and the price. My best attempts just end up returning an empty [] 

User-Ps-MacBook-Pro:ammo user$ scrapy shell "http://www.ammofast.com/collections/pistolproducts/aguila-25-auto-acp-full-metal-jacket-50-gr-1000-round-case"
...
>>> response.xpath('//div[@id="product-header-title"]/h1[@id="product-title"]/text()').extract
<bound method SelectorList.extract of []>

I have tried a handfull a different variations but all to no avail. I guess I don't quite fully understand the XPath structure just yet, but that's why I'm practicing :) Thanks for the help! I am on Scrapy 1.1.0 as well.

Upvotes: 1

Views: 631

Answers (1)

Padraic Cunningham
Padraic Cunningham

Reputation: 180502

The price is loaded using Javascript, in the source you can see:

window.ShopifyAnalytics.lib.track(
          "Viewed Product",
          {"id":705964349,"name":"Aguila .25 Auto ACP Full Metal Jacket 50 Gr 1000 Round Case - 1000 \/ 50","price":"329.99","currency":"USD","sku":"","brand":"Aguila","category":".25 ACP","nonInteraction":true}
        );

            });

You can see if you look at the source returned that there is actually no text in p class="price":

 <div id="purchase">
                <p class="price"></p>                                 
                <input class="btn" type="submit" name="add" id="add-to-cart" value="Add to Cart" />
              </div>

            </div><!-- /.options -->

What you could do is parse that function code or get the price from the option tag:

In [15]: response.xpath("//*[@id='product-title']/text()")
Out[15]: [<Selector xpath="//*[@id='product-title']/text()" data=u'Aguila .25 Auto ACP Full Metal Jacket 50'>]

In [16]: response.xpath("//option/text()")
Out[16]:  [<Selector xpath="//*[@id='product-select']/option/text()" data=u'1000 / 50 - $329.99'>]

Upvotes: 1

Related Questions