CODEWITHSUNDEEP
scrapysplash-screen
Arunkumar N
Arunkumar N

Reputation: 55

how to handle multiple return values in scrapy from splash

i'm using scrapy with splash, in my splash i can send multiple values but in my scrapy code i could not handle all.for example, this my splash script

splash_script = """
    function main(splash)
      local url = splash.args.url
      return {
        html = splash:html(),
        number = 1
      }
    end
    """

The method trigger splash from scrapy

yield scrapy.Request(
              url= response.urljoin(url),
              callback = self.product_details,
              errback=self.error,
              dont_filter=True,
              meta = {
                'splash':{
                  'endpoint': 'render.html',
                  'cache_args': ['lua_source'],
                  'args' :{
                     'index': index,
                     'http_method':'GET',
                     'lua_source': self.splash_script,
                   }
               }
          },
        )

The call back method

def product_details(self,response):
    print response.body

This method receives only html content, i cant see the number

Upvotes: 3

Views: 1162

Answers (1)

josh
josh

Reputation: 36

Your are printing response.body . This only includes the html.

You have to use response.data to see the 1.

You can also access the elements individually:

response.data['html']

or 

response.data['number']

And when you return stuff, make sure you are assigning it in the return statement:

NOT-

html = splash:html()
number = 1
return {number,html}

BUT

return {number = 1, html = splash:html()}

Basically, you have to assign the JSON keys in the return statement even if you might have done so outside. Extra info but that really screwed me up and you might run into the same problem.

Upvotes: 1

Related Questions