user3497702
Reputation: 861
How to test robots.txt that has disallows urls that contain somekeyword in ASP.NET MVC web app
I have an ASP.NET MVC web application. I added robots.txt to disallow a url that contains a key "somewords".
Please advise how can I test that a crawler is not allowed to access the page in local and test environment?
I cannot use any third-party tool in my environment since I need to get permission to use that.
Upvotes: 0
Views: 47
Answers (0)
Related Questions
- How do you handle multiple submit buttons in ASP.NET MVC Framework?
- How to get all Errors from ASP.Net MVC modelState?
- How can I properly handle 404 in ASP.NET MVC?
- How to add Web API to an existing ASP.NET MVC 4 Web Application project?
- How to render an ASP.NET MVC view as a string?
- ASP.NET MVC controller actions that return JSON or partial html
- How do you create a dropdownlist from an enum in ASP.NET MVC?
- How to get ELMAH to work with ASP.NET MVC [HandleError] attribute?
- ASP.NET MVC runtime installation requirement in "test" server