CODEWITHSUNDEEP
pythonscrapy
Alessandro
Alessandro

Reputation: 175

Python Scrapy Function Call

I try to call the getNext() function from the main parse function that scrappy calls but it never gets called. 

class BlogSpider(scrapy.Spider):
      # User agent.
      name = 'Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.133 Mobile Safari/535.19'
      start_urls = ['http://www.tricksforums.org/best-free-movie-streaming-sites-to/']

      def getNext(self):
        print("Getting next ... ")
        # Check if next link in DB is valid and crawl.
        try:
          nextUrl = myDb.getNextUrl()
          urllib.urlopen(nextUrl).getcode()
          yield scrapy.Request(nextUrl['link'])
        except IOError as e:
          print("Server can't be reached", e.code)
          yield self.getNext()

      def parse(self, response):
        print("Parsing link: ", response.url)
        # Get all urls for futher crawling.
        all_links = hxs.xpath('*//a/@href').extract()
        for link in all_links:
          if validators.url(link) and not myDb.existUrl(link) and not myDb.visited(link):
            myDb.addUrl(link)
        print("Getting next?")
        yield self.getNext()

I tried with and without yield before it .. what's the issue ? And what's this yield supposed to be ? :)

Upvotes: 3

Views: 1388

Answers (1)

alecxe
alecxe

Reputation: 473783

You are trying to yield a generator, but meant to yield from a generator.

If you are on Python 3.3+, you can use yield from:

yield from self.getNext()

Or, simply do the return self.getNext().

Upvotes: 3

Related Questions