CODEWITHSUNDEEP
pythonweb-scrapingscrapy
sensei_jake
sensei_jake

Reputation: 13

Can you set conditions when scraping with Scrapy?

I was wondering if I could set some conditions that have to be met for the information to be stored (doing web-scraping with Scrapy version 1.7.3).

For example, only storing the movies with a rating greater than 7 while scraping IMDB's website.

Or would I have to do it manually when looking through the output file? (I am currently outputting the data as a CSV file)

Upvotes: 1

Views: 126

Answers (1)

eLRuLL
eLRuLL

Reputation: 18799

This is an interesting question, and yes, scrapy can totally help you with this. There some approaches you can take, if it is only for manipulating the items before actually "returning" them (which means it is already an output) maybe I'd recommend to use Item Loaders which basically helps you setup rules per field on each item.

For actually dropping items with the respective rules I'd suggest you to use and Item Pipeline which serves as a final filter before again returning the items, in this case it would be interesting for you to combine it with something like Cerberus which helps you define whole item schemas and according to that, drop or return an item.

Upvotes: 1

Related Questions