CODEWITHSUNDEEP
pythonweb-crawlerscrapy
ombak
ombak

Reputation: 43

How to get values from conditional drop down fields with Scrapy/Python?

This tutorial helped me in order to gather the first drop down list entries on a website. But I am not able to get the data of the second conditional/dependent drop down box with Scrapy.

Assuming the following procedure is the way to go, how does step 2 work? The values never appear so far...

  1. Get list entries of first drop down box
  2. Select 1st value of first drop down box.
  3. get all dependent drop down values.
  4. iterate though the first drop down list...

I think the code below needs to be adjusted as the code is for a submit form:

    for i in range(len(brandCategoryList)):

        # Generate new request for each brand category's page
        yield FormRequest("http://www.xxxxxxxxxx.com",
                    method='POST',                         
                    formdata={'BrandName':'','CatBrand':brandCategoryList[i],'submit1':'Find+Product'},
                    callback=self.parseBrandPage,
                    meta={'brandCategoryId':i,'brandCategoryName':brandCategoryList[i]})

Thanks

Upvotes: 4

Views: 2406

Answers (1)

abeyer
abeyer

Reputation: 745

Scrapy won't run any javascript code in the page you are scraping, and dependent drop-downs often rely on javascript to populate their options based on the choice in their controlling drop-down.

If you need to automate/scrape content which requires javascript, you may want to consider another library. Start here for some pointers.

Upvotes: 2

Related Questions