CODEWITHSUNDEEP
pythonformsweb-scrapingscrapy
user1329187
user1329187

Reputation:

How to use Scrapy FormRequest.from_response() method to submit a form?

I am trying to parse some info from this page and then reload the page with other options from the drop-down menu called "Packungsgröße". There are 3 options, so I am trying to scrape the same page for all the options.

I found out that I can use the form of class="upprice_config" and POST it with the correct data. I do not know exactly which data I need to supply, so I supply all I have. Then I call the same function for parsing.

Any way, it does not work the way I want. It does not give me any error and seems it executes the form submission, but I do not get the required page.

Here is my code:

... def parse_product(self, response):
120     sc_menu = response.xpath('//form[@class="upprice_config"]/div/select')
121         if sc_menu:
122             sel_name = sc_menu[0].xpath('@name').extract()[0]
124             for opt in sc_menu[0].xpath('option'):
126                 if opt.xpath('@selected'):
127                     selected = opt.xpath('text()').extract()[0]
129                     product['options'] = selected.strip()
130 
131                     yield product
132 
133                 # extract value
134                 value = opt.xpath('@value').extract()[0]
136                 # submit form for the new option
137                 self.submit_form(response, value, sel_name)
138         else:
139             product['options'] = ''
140 
141             yield product
142 
143 
144 # submit form for the new option
145 def submit_form(self, response, value, sel_name):
146     formdata = {'method': 'POST',
147                 'action': response.url,
148                 'value': value}
149     scrapy.http.FormRequest.from_response(response, 
150                 formdata = formdata, 
151                 clickdata = {'name': sel_name},
152                 callback = self.parse_product)

Can somebody point me to what I am doing wrong here. If there is a better way to do it, please share with me.

Upvotes: 0

Views: 2135

Answers (1)

akhter wahab
akhter wahab

Reputation: 4085

if you are using firefox try to install firebug and check what data website is posting

i can see they are posting option value 

select class="form-control" onchange="this.form.submit();" name="group[1]">
  <option value="3"> 3kg </option>
  <option value="233" selected="selected"> 10kg </option>
  <option value="3603"> 25kg </option>

either you can post that data or i just checked you can Make get requests too

something like 

http://www.zooroyal.de/muehldorfer-pferdeleckerli-bio-bronchial?c=6259&group[1]=233
http://www.zooroyal.de/muehldorfer-pferdeleckerli-bio-bronchial?c=6259&group[1]=3603

Upvotes: 1

Related Questions