CODEWITHSUNDEEP
pythonweb-scrapingscrapy
Maverick
Maverick

Reputation: 799

Scrapy, scrape from drop down menu

I am trying to scrape the names of some towns/cities from a drop down menu.

The html looks like:

option value="/salaries/THIS_IS_WHAT_I_WANT" data-tn-link data-tn-element="loc_city[]">        Acton

I have tried this: 

def parse_cities(self, response):
        for city in response.xpath('//select[@id="cmp-salary-loc-select"]/option/text()').extract():
            print(scrapy.Request(response.urljoin("/"+city)))

With no luck.

I know that this

response.xpath('//select[@id="cmp-salary-loc-select"]/option').extract()

Returns all the html I need, But just can't seem to get the values?

Any pointers would be appreciated. I haven't posted the link as I do want to try figure this one out rather than someone do it for me! But will add in if you guys think it would help?

Upvotes: 1

Views: 4144

Answers (1)

Enrique Dev
Enrique Dev

Reputation: 119

def parse_cities(self, response):
  for city in response.xpath('//select[@id="cmp-salary-loc-select"]/option/@value').extract():
     yield scrapy.Request(response.urljoin("/"+city), callback=some_method)

Upvotes: 1

Related Questions