CODEWITHSUNDEEP
xmlscrapy
Klark
Klark

Reputation: 31

How to parse html item embedded items in xml file with Scrapy

I am trying to parse an xml feed with XMLFeedSpider

In the XML feed I want to extract the 'price' item :

<span class="price" id="product-price-2037">19,77 €</span>

but this price item is in html code inside a tag as it follows :

<channel>
<item>
<title>
<![CDATA[ product title ]]>
</title>
<meta http-equiv="X-UA-Compatible" content="IE=8"/>
<link>http://example.com/apage.html</link>
<description>
<![CDATA[
<table><tr><td><a href="http://example.com/apage.html">
<img src="http://example.com/media/catalog/product/aimage173.jpg" border="0" align="left" height="75" width="75"></a></td>
<td style="text-decoration:none;"> <div class="price-bframe"> <p class="old-price"> <span class="price-label">Prix normal :</span>
<span class="price" id="old-price-2895037">40,00 €</span> </p>
<p class="special-price"> <span class="price-label">Prix spécial :</span>
<span class="price" id="product-price-2037">19,77 €</span> </p> </div> </td></tr></table>
]]>
</description>
</item>
</channel>

Here is my actual spider : 

from scrapy.contrib.spiders import XMLFeedSpider
from scrapy.selector import XmlXPathSelector
from tutorial.items import DmozItem

class DmozSpider(XMLFeedSpider):
name = 'myspidername'
allowed_domains = ["example.com"]
start_urls = ['http://example.com/rss/catalog/new/store_id/1/']
iterator = 'iternodes'
itertag = 'channel'

def parse_node(self, response, node):
    title = node.select('item/title/text()').extract()
    link = node.select('item/link/text()').extract()
    price = node.select('*[@class=price"]text()').extract()
    item = DmozItem()
    item['title'] = title
    item['link'] = link
    item['price'] = price
    return item

The result :

Invalid Xpath: *[@class=price"]text()

Upvotes: 2

Views: 1198

Answers (1)

Mirage
Mirage

Reputation: 31568

I think that is because your path is invalid try this

[@class=price"]/text()

I think you missed the slash before the text

Upvotes: 1

Related Questions