CODEWITHSUNDEEP
pythonweb-crawlerscrapy
Alok
Alok

Reputation: 10534

How to find out size of crawled webpage in Scrapy?

I am learning Scrapy.
I want to find out size of crawled webpage or size of response in KB or MB etc using Scrapy.
I can find out length of content of crawled webpage usingresponse.body
what is the simplest way to findout how much data is getting downloaded per request?

I tried to understand this solution which is similar to my my requirement. but I am not able to understand this code. 

parse(self, response):
    url=response.url
    content=response.body
    #download_size=

Upvotes: 1

Views: 3011

Answers (2)

Sam
Sam

Reputation: 360

You could simply use the built in response.body and apply len on it. Simply do

size=str(len(response.body)),

Upvotes: 1

codefreak
codefreak

Reputation: 7131

You can get size by using information provided by reading content-length from headers property of Response Object. 

parse(self, response):
    url=response.url
    content=response.body
    #response length in bytes
    download_size= int(response.headers['content-length'])

Upvotes: 1

Related Questions