CODEWITHSUNDEEP
pythonscrapy
Siddharth
Siddharth

Reputation: 5209

Scrapy is redirecting direct links

I am using CrawlSpider to crawl and extract data from a webpage.

The start url has only one link (which is a direct link) and from that link data has to be scraped (data is being successfully scraped if I use BaseSpider and give the direct link). However, when I run the CrawlSpider, it is getting some weired 301 request and control is not coming to parse_item() which does the scraping.

I have a localhost page called main_page.html which has a link to http://www.flipkart.com/office-supplies/pen/faber-castell/itmd4fpkgkd7e3fj?pid=pend4fpkyk2w9fd4&_l=U0SclLhlhi7jGPVIA8xWyA--&_r=tcVDd6I7AkBG9cR2hX21MA--&ref=5b471a78-5264-4e09-ba61-03f8965e10d0 which I think it a direct link. On running the crawl spider, I see the following output at scrapy-

2011-12-05 15:54:34+0530 [flipkart_spider] DEBUG: Crawled (200) http://localhost/main_page.html> (referer: None)
2011-12-05 15:54:35+0530 [flipkart_spider] DEBUG: Redirecting (301) to http://www.flipkart.com/office-supplies/pen/faber-castell/itmd4fpkgkd7e3fj?pid=pend4fpkyk2w9fd4&_l=U0SclLhlhi7jGPVIA8xWyA--&_r=tcVDd6I7AkBG9cR2hX21MA--&ref=5b471a78-5264-4e09-ba61-03f8965e10d0> from http://www.flipkart.com/office-supplies/pen/faber-castell/itmd4fpkgkd7e3fj?_l=U0SclLhlhi7jGPVIA8xWyA--&_r=tcVDd6I7AkBG9cR2hX21MA--&pid=pend4fpkyk2w9fd4&ref=5b471a78-5264-4e09-ba61-03f8965e10d0>

I have a print statement in parse_item() which is not called. what is wrong here?

Upvotes: 1

Views: 1528

Answers (1)

Ski
Ski

Reputation: 14487

I think you should remove everything after ? in your url so it will become: http://www.flipkart.com/office-supplies/pen/faber-castell/itmd4fpkgkd7e3fj

Update:

In most cases attributes in query string are only used for things that do not affect content, for example: from which page you came here, which menu item is active, session id. If you copy urls from your browser into crawler it is possible that some of arguments in url has something to do with your user-agent and give unpredictable results when the same url is opened with different agent.

You will also save your self from lots of duplicate pages if you know which arguments are safe to be removed from url. Usually you can access same items from different categories, but every time the url of item will be different because argument ref= I think tells information from which page you came here.

Upvotes: 2

Related Questions