CODEWITHSUNDEEP
pythonbeautifulsoupscrapy
Mirage
Mirage

Reputation: 31548

Can the use of Beautiful Soup with Scrapy increase the performance

I am doing crawling everything in scrapy. I have seen that many people are using beautiful Soup for parsing.

I just wanted to know that is there any advantage in terms of speed , efficiency or more slectrors etc which help me in creating spiders and crawlers or scrapy alone should be enough for me

Upvotes: 0

Views: 1954

Answers (2)

Petri
Petri

Reputation: 5006

The performance of using BeautifulSoup instead of the Scrapy builtin parser-selector mechanishm depends on a number of things: for example, it allows for different parsers to be used; lxml being the fastest of them. There are some other things that can be done to improve BeautifulSoup performance, as well. Overall, you may reach similar performance but in general, using BeautifulSoup will not deliver speed advantages.

However, BeautifulSoup does provide some advantageous alternative extraction APIs and selector mechanisms that Scrapy does not offer.

In particular, the CSS selectors are something you may find very convenient.

Upvotes: 2

akhter wahab
akhter wahab

Reputation: 4085

well the answer is you should try to parse couple of pages with HtmlSelector then Using beautiful Soup. and find some stats.

2ndly most of people use beautiful Soup even lxml for parsing because they already used to for using this.

Scrapy basic motive is Crawling if you are not comfortable with Xpath you can go with beautiful Soup , lxml (although lxml package also support xpath) even Only Regex for Parsing.

Upvotes: 0

Related Questions