CODEWITHSUNDEEP
asp.nethttpiisurl-rewritingroutes
user1372494
user1372494

Reputation:

Unknown 302 redirect at root. Crawlers cannot follow through

The whole thing started when I noticed Facebook Debugger and other crawler tools are unable to parse my page. Facebook throws a critical error saying that it cannot follow the redirect. I believe Search Engine bots are hitting the same end. The website is functioning normally via all major web browsers.

It's probably worth to mention I am experimenting with ASP.NET Routing, using Web Forms under IIS8.

Given a website (http://example.com), here's what happens.

Case 1: Trying to access the root, this is what I get with a Web Sniffer simulator

enter image description here

Case 1 observations:

First thing I notice is '302' redirect instead of '200 OK'. It gives a 302 redirect with or without leading 'www'.

I noticed is that the Location Header is simply "/", confirmed by the page from IIS that I cannot see with a regular browser, which says the page is moved to "/". I believe something messes up at this point and the crawler is unable to follow through for some reason.

Case 2: Trying to access a given category page with a Web Sniffer simulator

Case 2 observations:

As you might figured out already, identical to case 1. And once again Facebook debugger cannot go through it, resulting to a redirect it cannot follow.

enter image description here

Questions:

1: How can I force an absolute path in the location headers instead of relative and will this be sufficient for the crawlers to follow through?

2: What could cause a 302 redirect happening in the first place in both www and non-www versions of the website?

Upvotes: 0

Views: 691

Answers (1)

Marco Miltenburg
Marco Miltenburg

Reputation: 6138

Your web application is most likely depending on a cookie. The application sends a Set-Cookie header and redirects to the same page in order to receive a new request with the cookie data available. Search engines / bots, the Facebook bot and your Web Sniffer simulator will not send that cookie data and hence the web application keeps sending the 302 redirect responses.

The solution is to change your application to not require cookies for just simply viewing your web pages.

Upvotes: 2

Related Questions