CODEWITHSUNDEEP
asp.net-mvcbotsrobots.txtelmah
Tyler Morrow
Tyler Morrow

Reputation: 961

How can I effectively prevent bot requests or at least prevent them from clogging my logs?

I assume the errors I'm getting are caused by bots that have:

Here are some of the errors:

There are other errors for specific URLs that used to exist, but have since been removed.

Is there a way to prevent bots from hitting these links or is it something I'll have to deal with by filtering out specific requests in ELMAH?

Upvotes: 1

Views: 366

Answers (1)

Peter
Peter

Reputation: 1715

Unfortunately, due to the amount of bots out there and the variety of ways that they are coded to attack or scrape your website, you will not be able to prevent all of these errors. However, you can easily choose to ignore specific types of errors in Elmah. Here is a sample of a filter in the <elmah> section of a web.config file:

<errorFilter>
  <test>
    <or>
      <and>
        <!-- filter all errors out that fall in the range 400-499 -->
        <greater binding="HttpStatusCode" value="399" type="Int32" />
        <lesser binding="HttpStatusCode" value="500" type="Int32" />
      </and>

      <regex binding="BaseException.Message" pattern="A potentially dangerous \b.+?\b value was detected from the client" caseSensitive="false" />

      <regex binding="BaseException.Message" pattern="he provided anti-forgery token was meant for user" caseSensitive="false" />        
    </or>
  </test>
</errorFilter>

That will filter out all the 404s etc. by only including errors less than 400 or greater than 499, and excludes a couple of specific .NET exceptions that are commonly triggered by malicious bots. From there, you can tweak to suit...

Upvotes: 1

Related Questions