CODEWITHSUNDEEP
pythonxpathscrapy
leboMagma
leboMagma

Reputation: 88

Scrapy Spider scrapes content Partially and leaving others

I have a scrapy spider defined, it can scrape all names and some storiesand the xpath definded cannot capture the stories,from https://www.cancercarenorthwest.com/survivor-stories,

# -*- coding: utf-8 -*-

import scrapy
from scrapy.contrib.loader import ItemLoader
from scrapy.contrib.spiders import CrawlSpider,Rule
from scrapy.selector import XmlXPathSelector
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from cancerstories.items import CancerstoriesItem

class LungcancerSpider(CrawlSpider):
    name = "lungcancer"
    allowed_domains = ["coloncancercoalition.org"]
    start_urls = (
        'http://www.coloncancercoalition.org/community/stories/survivor-stories/',
    )
    rules = (
             Rule(SgmlLinkExtractor(allow=[r'http://www.coloncancercoalition.org/\d+/\d+/\d+/\w+']),callback='parse_page',follow=True),
             )

    def parse_page(self, response):
        Li = ItemLoader(item=CancerstoriesItem(),response=response)
        Li.add_xpath('name', '/html/body/div[4]/div[1]/div[1]/div/h1/text()')
        Li.add_xpath('story','//../div/div/p/text()')

        yield Li.load_item()

Upvotes: 1

Views: 46

Answers (1)

alecxe
alecxe

Reputation: 473873

I think you need to join the texts of all the paragraphs under the post content:

Li.add_xpath('story', '//div[@class="post-content"]/div/p/text()', Join(" "))

where Join() is the output processor imported as:

from scrapy.loader.processors import Join

Upvotes: 1

Related Questions