CODEWITHSUNDEEP
javascriptpythonpostscrapy
Zoheir
Zoheir

Reputation: 23

scrapy filling out POST form

I'm trying to use scrapy to fill a POST form, to try to book train tickets.

I thought that the FormRequest class can do the thing, but I can't deal with the javascript form. Scrapy crawler returns nothing.

The fileds I used are sufficient to send the form.

import scrapy

from scrapy.item import Item, Field
from scrapy.http import FormRequest
from scrapy.spider import BaseSpider

class SncfItem(Item):
 title = Field()
 link = Field()
 desc = Field()

class SncfSpider(scrapy.Spider):
name = "sncf"
allowed_domains = ["voyages-sncf.com"]
start_urls = (
    'http://www.voyages-sncf.com/billet-train',
)

def parse(self, response):

    yield FormRequest.from_response(response,
                                    formname='saisie',
                                    formdata={'ORIGIN_CITY': 'Gare de Lyon (Paris)',
                                              'DESTINATION_CITY': 'Lyon Part-Dieu',
                                              'OUTWARD_DATE': '03.06.2015'},
                                    callback=self.parse1)

def parse1(self, response):
    print response.status

Can anyone tell me please if I missed a step while using mySpider?

Any help would be greatly appreciated.
Thank's

scrapy crawl sncf -o items.xml  -t xml
2015-06-01 23:13:54+0200 [sncf] INFO: Spider opened
2015-06-01 23:13:54+0200 [sncf] INFO: Crawled 0 pages (at 0 pages/min),       scraped 0 items (at 0 items/min)
2015-06-01 23:13:54+0200 [scrapy] DEBUG: Telnet console listening on   127.0.0.1:6024
2015-06-01 23:13:54+0200 [scrapy] DEBUG: Web service listening on 127.0.0.1:6081
2015-06-01 23:13:55+0200 [sncf] DEBUG: Crawled (200) <GET     http://www.voyages-sncf.com/billet-train> (referer: None)
2015-06-01 23:13:56+0200 [sncf] DEBUG: Redirecting (302) to <GET   http://www.voyages-sncf.com/billet-train> from <POST http://www.voyages-   sncf.com/vsc/train-ticket/>
2015-06-01 23:13:56+0200 [sncf] DEBUG: Crawled (200) <GET   http://www.voyages-sncf.com/billet-train> (referer: http://www.voyages-  sncf.com/billet-train)
200
2015-06-01 23:13:56+0200 [sncf] INFO: Closing spider (finished)

Upvotes: 2

Views: 1149

Answers (1)

Elias Dorneles
Elias Dorneles

Reputation: 23796

You're getting a redirect because the date format is invalid.

I replayed that request in Scrapy shell doing the following:

$ scrapy shell http://www.voyages-sncf.com/billet-train
.... a few log messages later, I get a shell with the response...
>>> # first I recreate the FormRequest per your code:
>>> fr = FormRequest.from_response(response,
                                  formname='saisie',
                                  formdata={'ORIGIN_CITY': 'Gare de Lyon (Paris)',
                                           'DESTINATION_CITY': 'Lyon Part-Dieu',
                                           'OUTWARD_DATE': '03.06.2015'})
>>> # checked the url and method:
>>> fr.url
'http://www.voyages-sncf.com/vsc/train-ticket/'
>>> fr.method
'POST'
>>> fetch(fr)  # execute the request
>>> view(response)  # opened the result in browser

Looking at the result, I saw a validation error message for the date, saying: "Le format de la date d'aller que vous avez saisi est incorrect. Nous vous invitons à utiliser le calendrier."

Upvotes: 2

Related Questions