CODEWITHSUNDEEP
python-3.xweb-crawlerscrapy
Alex
Alex

Reputation: 27

How to use Scrapy to crawl data on the second level of a Page

I want to use scrapy spider to get data (question title + content & answer) from all posts of the following website:

https://forums.att.com/t5/custom/page/page-id/latest-activity/category-id/Customer_Care/page/1?page-type=latest-solutions-topics

The problem is I just dont know how to make it first to follow the link of the post and then to crawl the data of all 15 posts/site.

{import scrapy

class ArticleSpider(scrapy.Spider): name = "post" start_urls = ['https://forums.att.com/t5/Data-Messaging-Features-Internet/Throttling-for-unlimited-data/m-p/4805201#M73235']

def parse(self, response):
    SET_SELECTOR = 'body'
    for post in response.css(SET_SELECTOR):

        # Selector for title, content and answer
        TITLE_SELECTOR = '.lia-message-subject h5 ::text'
        CONTENT_SELECTOR = '.lia-message-body-content'
        ANSWER_SELECTOR = '.lia-message-body-content'

        yield {

            # [0].extract() = extract_first()
            'Qtitle': post.css(TITLE_SELECTOR)[0].extract(),
            'Qcontent': post.css(CONTENT_SELECTOR)[0].extract(),
            'Answer': post.css(ANSWER_SELECTOR)[1].extract(),
        }
    # Running through all 173 pages
    NEXT_PAGE_SELECTOR = '.lia-paging-page-next a ::attr(href)'
    next_page = response.css(NEXT_PAGE_SELECTOR).extract_first()
    if next_page:
        yield scrapy.Request(
            response.urljoin(next_page),
            callback=self.parse
        )}

I hope u can help me out. Thanks in advance!

Upvotes: 1

Views: 1458

Answers (1)

vold
vold

Reputation: 1549

You need to add a method for scraping post content. You can rewrite your spider code like this (I use xpath selector):

# -*- coding: utf-8 -*-
import scrapy  

class ArticleSpider(scrapy.Spider):
    name = "post"
    start_urls = ['https://forums.att.com/t5/custom/page/page-id/latest-activity/category-id/Customer_Care/page/1?page-type=latest-solutions-topics']

    def parse(self, response):
        for post_link in response.xpath('//h2//a/@href').extract():
            link = response.urljoin(post_link)
            yield scrapy.Request(link, callback=self.parse_post)

        # Checks if the main page has a link to next page if True keep parsing.
        next_page = response.xpath('(//a[@rel="next"])[1]/@href').extract_first()
        if next_page:
            yield scrapy.Request(next_page, callback=self.parse)

    def parse_post(self, response):
        # Scrape title, content from post.
        for post in response.xpath('//div[contains(@class, "lia-quilt-forum-message")]'):
            item = dict()
            item['title'] = post.xpath('.//h5/text()').extract_first()
            item['content'] = post.xpath('.//div[@class="lia-message-body-content"]//text()').extract()
            yield item

        # If the post page has a link to next page keep parsing.
        next_page = response.xpath('(//a[@rel="next"])[1]/@href').extract_first()
        if next_page:
            yield scrapy.Request(next_page, callback=self.parse_post)

This code parses all links from the main page and calls parse _post methods for scraping each post content. Both parse and parse_post methods check if there is next link and if True proceed scraping.

Upvotes: 2

Related Questions