CODEWITHSUNDEEP
pythoncookiesscrapypython-requests
Saurav Pathak
Saurav Pathak

Reputation: 836

How do I send cookies data through scrapy Request?

I've been trying to send cookie in headers to this url and scrape json response. Cookie to be sent is cookie:"tid=fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg;"

On using python's requests library, I get needed json data. Here's my Python code. 

import requests
url = "https://amazon.zappos.com/mobileapi/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3"
cookie = "tid=fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg;"
headers = {'cookie':cookie}
requests.get(url, headers=headers).json()

Here's my spider code: 

def parse(self, response):
        cookie = "tid=fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg;"
        url = "https://amazon.zappos.com/mobileapif/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3"
        request = scrapy.Request(url=url, cookies ={'cookie': cookie},callback=self.parsejson)
        yield request

    def parsejson(self, response):
        jsonresponse = json.loads(response.text)
        print(jsonresponse)

But spider gives this error. 

2019-07-16 15:59:59 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://amazon.zappos.com/c/404> from <GET https://amazon.zappos.com/mobileapif/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3>
2019-07-16 15:59:59 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <GET https://amazon.zappos.com/c/404>
Cookie: cookie=tid=fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg

2019-07-16 16:00:00 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://amazon.zappos.com/c/404> (referer: https://amazon.zappos.com/mobileapi/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3)
2019-07-16 16:00:00 [scrapy.core.scraper] ERROR: Spider error processing <GET https://amazon.zappos.com/c/404> (referer: https://amazon.zappos.com/mobileapi/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3)
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/Users/mac/scrapy/projects/zappos/zappos/spiders/zappo.py", line 18, in parsejson
    jsonresponse = json.loads(response.text)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
2019-07-16 16:00:00 [scrapy.core.engine] INFO: Closing spider (finished)

Upvotes: 0

Views: 2065

Answers (1)

gangabass
gangabass

Reputation: 10666

You need to use key: value format for cookies:

def parse(self, response):
        # cookie = "tid=fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg;"
        url = "https://amazon.zappos.com/mobileapi/janus/recos/get?item=9280133&limit=5&limit_0=5&limit_1=10&limit_2=10&limit_3=15&teen=4826753&widgets=detail-1,detail-2,detail-vis-sims,detail-3"
        request = scrapy.Request(url=url, cookies ={'tid': "fpopRl3WSNQ1tdxH1bZTTR15LhuBRnJg"},callback=self.parsejson)
        yield request

UPDATE Looks like you have a typo in an url (mobileapif vs mobileapi). Check updated code.

Upvotes: 3

Related Questions