CODEWITHSUNDEEP
scrapy
ZengJuchen
ZengJuchen

Reputation: 4811

Why my scrapy response body is empty?

I'm learning scrapy recently. And I tried use its simplist way to fetch a response body, but I got an empty string.

Here is my code:

>>> from scrapy.http import Response
>>> r = Response('http://zenofpython.blog.163.com/blog/static/23531705420146124552782')
>>> r.body
''
>>> r.headers
{}
>>> r.status
200

And with no difficulty, I can visit the url I used above for scrapy Response through browser.It has rich content.

What mistake I've made here?

Upvotes: 1

Views: 5247

Answers (2)

Levon
Levon

Reputation: 11952

Another reason for your problem can be that the site requires User-Agent header. Try it like this 

scrapy shell http://www.to.somewhere -s USER_AGENT='Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9) Gecko/20080705 Firefox/3.0 Kapiko/3.0'

You can read more here

Upvotes: 3

marven
marven

Reputation: 1846

You're supposed to fetch a Request and get a Response object in return.

Try doing:

r = Request(url='http://zenofpython.blog.163.com/blog/static/23531705420146124552782')
fetch(r)

on scrapy shell and you'll be able to get the result as a Response object.

print response.body

Upvotes: 1

Related Questions