CODEWITHSUNDEEP
pythonpython-2.7scrapy
user2208349
user2208349

Reputation: 7709

403 error when scrapying this website

I am trying to crawl a website

this website returns Error, query failed in the body. then I click on Find USED tab. then I click search to get the result. the search button is actually making a post reqeust and get the data.

This is my spider:

    def start_requests(self):
        for url in self.start_urls:
            yield self.make_requests_from_url(url)

    def make_requests_from_url(self, url):
        return Request(url,cookies={'PHPSESSID':'0a94ce3bf2484d5102a047b86f5b6c17','__utm':'154876456.1461047540.1397668365.1397668365.1397668365.1',', callback=self.page_parse)

    def parse(self,response):
        sel = Selector(response)
        print sel

I got this response:

2014-04-16 21:04:27+0300 [XXX] DEBUG: Crawled (403) <POST http://website> (referer: None)

what am I doing wrong please?

I analysed the request when I clicked the search button and this is the request

http://www.autodealer.ae/plugins/ad/buy.php?q=used+cars+dubai



User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Cookie: PHPSESSID=0a94ce3bf2484d5102a047b86f5b6c17;
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 131

what am i doing wrong?

or how to crawl that website please?

am I doing wrong on my spider?

Upvotes: 0

Views: 1043

Answers (1)

yorodm
yorodm

Reputation: 4461

The site requires authentication of some sorts and you are providing a fake PHPSESSID cookie with your request object. Your Python code should authenticate first and then keep sending request to the site.

-------- EDITED ----------

Posting to that URL causes a 403 error.

$ curl -X POST "SITE EDITED"

 <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
 <html><head>
 <title>403 Forbidden</title>
 </head><body>
 <h1>Forbidden</h1>
 <p>You don't have permission to access /plugins/ad/buy.php
  on this server.</p>
 <p>Additionally, a 404 Not Found
    error was encountered while trying to use an ErrorDocument to handle the request.

 </p>
 </body></html>

Upvotes: 1

Related Questions