CODEWITHSUNDEEP
javascriptpythonhtmlparsingscreen-scraping
melania
melania

Reputation: 1

How to scrape a dynamic page using scrapy or selenium?

I am trying to scrape a web dynamic page but with scrapy, I don't retrieve all the information, because the information I want is dynamic.

I tried to use Selenium but unlike Scrapy, you cannot specify headers and the website is blocking me. (I cannot name the website because scraping is illegal, but it is a well-known travel metasearch engine...). I have a page with a captcha solver (re-captcha so hard to resolve with a script) because the website detects me as a bot.

I want to have all the dynamic information and have access to the website, but impossible with selenium and scrapy is not giving me all the information I want and I think that it is because the page is not fully loaded when scrapy read it.

Upvotes: 0

Views: 294

Answers (1)

Yash Shukla
Yash Shukla

Reputation: 141

You can use selenium along with beautiful soup, you can scrape all data.' Please follow this like . https://code.tutsplus.com/tutorials/modern-web-scraping-with-beautifulsoup-and-selenium--cms-30486.

Upvotes: 1

Related Questions