How to store scrapy shell output/response to variable instead of html file

I am trying store the html code into variable called response using cmdline.execute as shown in below code ,but it is unavailable to store and program code breaks at scrapy shell, can anyone tell me how to store raw html to variable

import scrapy

from scrapy import cmdline

linkedinnurl = "https://stackoverflow.com/users/5597065/adnan-stab=profile"

response = cmdline.execute("scrapy shell https://stackoverflow.com/users/5597065/adnan-s?tab=profile".split()))

print(response)

Upvotes: 2

Views: 963

Answers (1)

Amrit
Amrit

Reputation: 2165

You can do like this to store raw html to variable:

 class MySpider(scrapy.Spider):
        def parse(self, res):
            with open(dynamic_file_name_function(res.url), 'w') as f:
                f.write(res.body)

if you don't need dynamic file name then just do :

 class MySpider(scrapy.Spider):
        def parse(self, res):
            with open(your_file_path, 'w') as f:
                f.write(res.body)

Upvotes: 2

Related Questions