Display PDF in ASP.Net such that it cannot be crawled

Question

The behaviour I am trying to implement is as follows:

• Step1: click on the following link http://www.sigmaaldrich.com/catalog/Lookup.do?N5=All&N3=mode+matchpartialmax&N4=benzene&D7=0&D10=benzene&N1=S_ID&ST=RS&N25=0&F=PR
• Step2: Click any of the MSDS hyperlink to the right of the page.
• Step3: The target page displays the pdf after a brief pause.

The path of the pdf is not revealed anywhere and this pdf cannot be crawled.

Any pointers in how to implement this functionality using ASP.Net 3.5 will be much appreciated.

Answer 1

Create a robots.txt file and place it in thee root folder of you web application. Files are not crawled but you can link them with normal tag. No need for server side processing:

# robots.txt
Disallow: /FolderWithFilesThatCannotCrawled/

More Details

Answer 2

It looks like a javascript postback. A LinkButton should provide the same behavior.

However, do realize you're violating REST principles with this approach, and it's very hackish.