CODEWITHSUNDEEP
pythonpyspider
Kristin Chia
Kristin Chia

Reputation: 11

Pyspider HTTP 599 issue

I have been trying to fix this issue but upon running pyspider to crawl tesla.com. I keep getting "Exception: HTTP 599: HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)". However, if I were to scrape scarpy.org. The code would run successfully.

from pyspider.libs.base_handler import *


class Handler(BaseHandler):
    crawl_config = {
    }

    @every(minutes=24 * 60)
    def on_start(self):
        self.crawl('https://www.tesla.com', callback=self.index_page, validate_cert=False)

    @config(age=10 * 24 * 60 * 60)
    def index_page(self, response):
        for each in response.doc('a[href^="http"]').items():
            self.crawl(each.attr.href, callback=self.detail_page, validate_cert=False)

    @config(priority=2)
    def detail_page(self, response):
        return {
            "url": response.url,
            "title": response.doc('title').text(),
        }

How do I bypass or solve this error message?

Able to work now with the same code, however, would be great if anyone here knows what are some ways to prevent like websites from giving such errors.

Upvotes: 0

Views: 26

Answers (0)

Related Questions