CODEWITHSUNDEEP
amazon-web-serviceslambdascrapytwistedaws-lambda
firefox
firefox

Reputation: 43

Scrapy throws error ReactorNotRestartable when runnning on AWS Lambda

I have deployed a scrapy project which crawls whenever an lambda api requests comes.

It runs perfectly for the first api call but later on it fails and throws ReactorNotRestartable error.

As far as I can understand the AWS Lambda ecosystem is not killing the process, hence reactor is still present in the memory.

The lambda log error is as follows:

Traceback (most recent call last):
File "/var/task/aws-lambda.py", line 42, in run_company_details_scrapy
process.start()
File "./lib/scrapy/crawler.py", line 280, in start
reactor.run(installSignalHandlers=False)  # blocking call
File "./lib/twisted/internet/base.py", line 1242, in run
self.startRunning(installSignalHandlers=installSignalHandlers)
File "./lib/twisted/internet/base.py", line 1222, in startRunning
ReactorBase.startRunning(self)
File "./lib/twisted/internet/base.py", line 730, in startRunning
raise error.ReactorNotRestartable()
ReactorNotRestartable

The lambda handler function is:

def run_company_details_scrapy(event, context):
   process = CrawlerProcess()
   process.crawl(CompanyDetailsSpidySpider)
   process.start()

I had a workaround by not stopping the reactor by inserting a flag in the start function

process.start(stop_after_crawl=False)

But the problem with this was that I had to wait until the lambda call timed out.

Tried other solutions, but none of them seems to work.Can anyone guide me how to solve this problem.

Upvotes: 4

Views: 2968

Answers (5)

Wise Lin
Wise Lin

Reputation: 11

Try this! It works for me!

Use Crochet to setup

import json
import logging
import os
import threading
import boto3
import scrapy
from fake_useragent import UserAgent
from scrapy.crawler import CrawlerRunner
from scrapy.utils.project import get_project_settings
from crochet import setup

# Initialize Crochet
setup()

# Configure logging
logging.getLogger('scrapy').propagate = False
ua = UserAgent()
region = os.getenv('REGION')
sqs = boto3.client('sqs', region_name=region)
queue_links = os.getenv('queue_links')


class MySpider(scrapy.Spider):
    name = "example"
    allowed_domains = ["www.example.com.tw"]
    start_urls = ["https://www.example.com.tw/main/Main.jsp"]
    user_agent = ua.random
    batch_size = 10  # batch size

    def __init__(self, *args, **kwargs):
        super(MySpider, self).__init__(*args, **kwargs)
        self.links = []

    def parse(self, response, **kwargs):
        try:
            sub_menus = response.css('.subMenu')
            for sub_menu in sub_menus:
                sub_menu_links = sub_menu.css("#topArea .dul .BTDME a::attr(href)")
                for sub_menu_link in sub_menu_links:
                    link = str(sub_menu_link.get())
                    if link.find("https") != -1 and link.find("category") != -1:
                        try:
                            self.links.append(link)

                        except Exception as e:
                            logging.error(f'SeleniumRequest: error > {e}, link: {sub_menu_link.get()}')
        except Exception as e:
            logging.error(str(e))

class LambdaRunner:
    def __init__(self):
        self.finished = threading.Event()
        self.results = []

    def run_spider(self):
        # Create a CrawlerRunner with project settings
        settings = get_project_settings()
        runner = CrawlerRunner(settings)

        # Create an instance of the spider class
        spider_cls = MySpider

        # Callback function to handle the spider results
        def handle_results(result):
            self.results.append(result)

            # Check if the spider has finished running
            if len(self.results) == 1:
                self.finished.set()

        # Start the first spider run
        deferred = runner.crawl(spider_cls)
        deferred.addCallback(handle_results)

        # Start the reactor
        runner.join()

    def wait_for_completion(self):
        self.finished.wait()

    def get_results(self):
        return self.results


def handler(event, context):
    try:
        runner = LambdaRunner()
        runner.run_spider()
        runner.wait_for_completion()

        return {
            'statusCode': 200,
            'body': json.dumps({'message': 'Completed!'})
        }
    except Exception as e:
        logging.exception(e)
        return {
            'statusCode': 500,
            'body': json.dumps({'error': str(e)})
        }

And you have to remove this setting in your settings.py of scrapy

# TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

Upvotes: 0

Kishan Patel
Kishan Patel

Reputation: 61

I faced error ReactorNotRestartable on AWS lambda and after I came to this solution

By default, the asynchronous nature of scrapy is not going to work well with Cloud Functions, as we'd need a way to block on the crawl to prevent the function from returning early and the instance being killed before the process terminates.

Instead, we can use scrapydo to run your existing spider in a blocking fashion:





import scrapy
import scrapy.crawler as crawler
rom scrapy.spiders import CrawlSpider
import scrapydo

scrapydo.setup()

# your spider
class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ['http://quotes.toscrape.com/tag/humor/']

    def parse(self, response):
        for quote in response.css('div.quote'):
            print(quote.css('span.text::text').extract_first())

scrapydo.run_spider(QuotesSpider)

Upvotes: 2

declension
declension

Reputation: 4185

Had the same problem recently, and Crochet didn't want to work for various reasons.

Eventually we went for the dirty solution: just call sys.exit(0) (or sys.exit(1) if an error was caught, not that anything looks at the return code AFAICT) at the end of the lambda handler function. This worked perfectly.

Obviously no good if you're intending to return a response from your Lambda, but if you're using Scrapy, data's probably being persisted already via your Pipelines, with a scheduler as the trigger for your Lambda, so no response needed.

Note: you will get a notice from AWS in CloudWatch:

RequestId: xxxx Process exited before completing request

Upvotes: 4

jschnurr
jschnurr

Reputation: 1191

This problem isn't unique to AWS Lambda - see running a spider in a Celery task.

You might try ScrapyScript (disclosure: I wrote it). It spawns a subprocess to support the Twisted reactor, blocks until all of the supplied spiders have finished, and then exits. It was written with Celery in mind, but the use case is similar.

In your case, this should work:

from scrapyscript import Job, Processor
def run_company_details_scrapy(event, context):
    job = Job(CompanyDetailsSpidySpider())
    Processor().run(job)`

Upvotes: 2

Jean-Paul Calderone
Jean-Paul Calderone

Reputation: 48335

You could try using https://pypi.python.org/pypi/crochet to coordinate use of a reactor running in a non-main thread from the Lambda-controlled main thread.

Crochet will do the threaded reactor initialization for you and provides tools to make it easy to call code in the reactor thread from the main (and get the results).

This might be more in line with the expectations Lambda has of your code.

Upvotes: 1

Related Questions