CODEWITHSUNDEEP
pythonweb-scrapingscrapysplash-screen
user11645632
user11645632

Reputation:

Next button in scrapy

I need to use an if statement inside the if statement I already have to determine when my scraping program clicks the next button so I can do something once that happens. The current if statement just determines if there is a next button on the page. But I cannot figure out how to determine when the next button is actually clicked.

            # Finds next page button
            priority = response.meta['priority']
            next_page = response.xpath('//a[contains(., "- Next>>")]/@href').get()
            # If it exists and there is a next page enter if statement
            if next_page is not None:
                # Go to next page
                yield response.follow(next_page, self.parse, priority=priority, meta={'priority': priority})

Upvotes: 0

Views: 222

Answers (1)

Umair Ayub
Umair Ayub

Reputation: 21201

Have a flag in meta key to determine if that link came from being clicked the NEXT button

def parse(self, response):

    if response.meta.get('isNextClicked', False):
        #Next was clicked

    # Finds next page button
    priority = response.meta['priority']    

    next_page = response.xpath('//a[contains(., "- Next>>")]/@href').get()
    # If it exists and there is a next page enter if statement
    if next_page is not None:
        # Go to next page
        yield response.follow(next_page, self.parse, priority=priority, meta={'priority': priority, 'isNextClicked': True})

Upvotes: 2

Related Questions