CODEWITHSUNDEEP
scrapy
Eric G
Eric G

Reputation: 3517

Error downloading PDF files

I have the following (simplified) code:

import os
import scrapy

class TestSpider(scrapy.Spider):
    name = 'test_spider'
    start_urls = ['http://www.pdf995.com/samples/pdf.pdf', ]

    def parse(self, response):
        save_path = 'test'
        file_name = 'test.pdf'
        self.save_page(response, save_path, file_name)

    def save_page(self, response, save_dir, file_name):
        os.makedirs(save_dir, exist_ok=True)
        with open(os.path.join(save_dir, file_name), 'wb') as afile:
            afile.write(response.body)

When i run it, I get this error: 

[scrapy.core.scraper] ERROR: Error downloading <GET http://www.pdf995.com/samples/pdf.pdf>
Traceback (most recent call last):
File "C:\Python36\lib\site-packages\twisted\internet\defer.py", line 1301, in _inlineCallbacks
    result = g.send(result)
File "C:\Python36\lib\site-packages\scrapy\core\downloader\middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
File "C:\Python36\lib\site-packages\twisted\internet\defer.py", line 1278, in returnValue
    raise _DefGen_Return(val)
twisted.internet.defer._DefGen_Return: <200 http://www.pdf995.com/samples/pdf.pdf>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):   
File "C:\Python36\lib\site-packages\twisted\internet\defer.py", line 1301, in _inlineCallbacks
    result = g.send(result)   
File "C:\Python36\lib\site-packages\scrapy\core\downloader\middleware.py", line 53, in process_response
    spider=spider)   
File "C:\Python36\lib\site-packages\scrapy_beautifulsoup\middleware.py", line 16, in process_response
    return response.replace(body=str(BeautifulSoup(response.body, self.parser)))   
File "C:\Python36\lib\site-packages\scrapy\http\response\__init__.py", line 79, in replace
    return cls(*args, **kwargs)   
File "C:\Python36\lib\site-packages\scrapy\http\response\__init__.py", line 20, in __init__
    self._set_body(body)   
File "C:\Python36\lib\site-packages\scrapy\http\response\__init__.py", line 55, in _set_body
    "Response body must be bytes. " 
TypeError: Response body must be bytes. If you want to pass unicode body use TextResponse or HtmlResponse.

Do I need to introduce a middleware or something to handle this? This looks like it should be valid, at least by other examples.

Note: at the moment I'm not using a pipeline because there in my real spider I have a lot of checks on whether the related item has been scraped, validating if this pdf belongs to the item, and checking a custom name of a pdf to see if it was downloaded. And as mentioned, many samples did what I'm doing here so I thought it would be easier and work.

Upvotes: 0

Views: 301

Answers (1)

Tarun Lalwani
Tarun Lalwani

Reputation: 146520

The issue because of your own scrapy_beautifulsoup\middleware.py which is trying to replace the return response.replace(body=str(BeautifulSoup(response.body, self.parser))).

You need to correct that and that should fix the issue

Upvotes: 1

Related Questions