CODEWITHSUNDEEP
javahttptomcatweb-crawler
James
James

Reputation: 18379

Apache Tomcat http dies, stops taking requests

I have been dealing with an aggressive bot crawler/attack for the last week. The bot is distributed with random ips, and agents string, so difficult to block, but I have another thread for that. This issue is that a flood of http requests can cause Tomcat to die.

The Tomcat process is still okay, and is not out of memory, it just stops taking http or https requests. It will timeout any http request, but will still take https requests (if http is being attacked, sometimes https dies as well).

I have seen error previously with "too many open files" so I changed the file limit from 10000 to 50000, that seemed to help, at least the https dying, but http still dies. I don't see the "too many open files" recently.

Seems like an extreme amount of open files, why would Tomcat open so many file, could it have a file leak under high load? The server sometimes is okay for 6 months (so can't be a leak under normal conditions), but has died under high load before.

The website is a large site with >1 million pages (dynamic content) and >1 million hits per day.

What happens when Tomcat gets a flood of http requests (like >100 per second for a prolonged duration), I assume requests will start backing up, if using a thread pool there will be no threads left, will it continue to pool requests until something breaks, or will it start rejecting requests?

Is there are way to start rejecting requests after a certain amount of backup? Seems like the only way to prevent a death or crash under extreme load.

My http and https config is different, so maybe that is related to why http dies. https is using maxThreads where as http is not, (how many threads is the default?)

<Connector port="80" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="443"
               URIEncoding="UTF-8" />

<Connector port="443" protocol="org.apache.coyote.http11.Http11NioProtocol"
               maxThreads="150" SSLEnabled="true" scheme="https" secure="true"
               clientAuth="false" sslProtocol="TLS"
               keystoreFile="..." keystorePass="..."
               URIEncoding="UTF-8"/>

Using Tomcat v8.5.47, CentOS 7.6, Oracle Java 1.8

Upvotes: 1

Views: 470

Answers (1)

Jimmy1976
Jimmy1976

Reputation: 11

Each socket connection is counted as an open file by the operating system. If you are using the ulimit command or similar depending on operating system then don't be afraid to set this an even higher number such as 250,000 for a large site such as this.

If not specified, maxThreads is set to 200. Try increasing this also, as along as you are not saturating the server's CPU when doing so the maxThreads can be increased to help mitigate DDOS attacks.

Upvotes: 1

Related Questions