CODEWITHSUNDEEP
pythonxmlxpathscrapy
J.Zil
J.Zil

Reputation: 2449

Scrapy - Issue with xpath on an xml crawl

I'm trying to make a simple spider to grab some xml and spit it out in a new format for an experiment. However it seems there is extra code contained within the xml which is spat out. The format I want is like this (no extra code or value tag) along the lines of this: <body>Don't forget me this weekend!</body>

I think I am using xpath wrong but I'm not sure what I'm doing wrong.

Spider

from scrapy.contrib.spiders import XMLFeedSpider
from crawler.items import CrawlerItem

class SiteSpider(XMLFeedSpider):
    name = 'site'
    allowed_domains = ['www.w3schools.com']
    start_urls = ['http://www.w3schools.com/xml/note.xml']
    itertag = 'note'

    def parse_node(self, response):
        xxs = XmlXPathSelector(response)
        to = xxs.select('//to')
        who = xxs.select('//from')
        heading = xxs.select('//heading')   
        body = xxs.select('//body')           
        return item

Input 

<note>
<to>Tove</to>
<from>Jani</from>
<heading>Reminder</heading>
<body>Don't forget me this weekend!</body>
</note>

Curreont (wrong) Output

<?xml version="1.0" encoding="UTF-8"?>
<items>
   <item>
      <body>
         <value>&lt;body&gt;Don't forget me this weekend!&lt;/body&gt;</value>
      </body>
      <to>
         <value>&lt;to&gt;Tove&lt;/to&gt;</value>
      </to>
      <who>
         <value>&lt;from&gt;Jani&lt;/from&gt;</value>
      </who>
      <heading>
         <value>&lt;heading&gt;Reminder&lt;/heading&gt;</value>
      </heading>
   </item>
</items>

Upvotes: 1

Views: 708

Answers (1)

alecxe
alecxe

Reputation: 473753

The signature of parse_node() is incorrect. There should be selector argument given which you should call xpath() method on, example:

def parse_node(self, response, selector):
    to = selector.xpath('//to/text()').extract()
    who = selector.xpath('//from/text()').extract()
    print to, who

Prints:

[u'Tove'] [u'Jani']

Upvotes: 1

Related Questions