CODEWITHSUNDEEP
apacheamazon-web-servicesmod-wsgidenial-of-service
Cristiano Coelho
Cristiano Coelho

Reputation: 1735

Apache mod_wsgi slowloris DoS protection

Assuming the following setup:

A simple slowloris attack like the one described here, using a "slow" request body: https://www.blackmoreops.com/2015/06/07/attack-website-using-slowhttptest-in-kali-linux/

The above attack, with just 15 requests (same as mod_wsgi threads) can easily lock the server until a timeout happens, either due to:

However, with just 15 concurrent "slow" requests, I was able to lock the server up to 60 seconds.

Repeating the same but with a more bizarre number, like 4096 requests, pretty much locks the server permanently since there will be always a new request that needs to be served by mod_wsgi once the previous times out.

I would expect that the load balancer should handle/detect this before even sending requests to apache, which it already does for similar attacks (partial headers, or tcp syn flood attacks never hit apache which is nice)

What options are available to help against this? I know there's no failproof option since these kind of attacks are difficult to detect and protect, but it's quite silly that the server can be locked that easily.

Also, if the wsgi application never reads request body, I would expect for the issue to not happen as well since the request should return immediately, but I'm not sure about this or the internals of mod_wsgi, for example, this is true when using a local dev wsgi server (the attack files since the request body is never read) but the attack succeeds when using mod_wsgi, which leads me to think it tries to read the body even before sending it to the wsgi code.

Upvotes: 0

Views: 1247

Answers (3)

Graham Dumpleton
Graham Dumpleton

Reputation: 58533

In mod_wsgi daemon mode there are a bunch of options to further help to combat such attacks by recovering from it and discarding queued requests as well which have been waiting too long. Try your tests using mod_wsgi-express as it defines defaults for a lot of these options whereas when using mod_wsgi yourself directly, there are no defaults. Use mod_wsgi-express start-server --help to see what defaults are. The actual options you want to look at for mod_wsgi daemon mode are request-timeout, connect-timeout, socket-timeout and queue-timeout. There are also other options related to buffer sizes and listener backlog you can play with. Do note that ultimately the listen backlog of the main Apache worker processes can still be an issue because it usually defaults to 500, which means a lot of requests can queue up stuck before you can even tag them with a time so as to help discard the backlog by tracking queue time.

You can find the documentation at:

On the point of whether mod_wsgi reads the request body before sending it, no it doesn't. Apache itself because it reads in block may partially read the request body when reading the headers, but it shouldn't block on it. Once the full request headers are passed off to mod_wsgi and sent through to the daemon process, then mod_wsgi will start transferring the request body.

Upvotes: 1

John Hanley
John Hanley

Reputation: 81386

Slowloris is a very simple Denial-of-Service attack. This is easy to detect and block.

Detecting and preventing DoS and DDos attacks are complex topics with many solutions. In your case you are making the situation worse by using outdated software and picking a low worker thread count so that the problem arises quickly.

A combination of services are available that would be used to manage Dos and DDos attacks.

The front-end of the total system would be protected by a firewall. Typically this firewall would include a Web Application Firewall to understand the nuances of HTTP protocols. In the AWS world, Amazon WAF and Shield are commonly used.

Another service that helps is a CDN. Amazon CloudFront uses Amazon Shield so it has good DDoS support.

The next step is to combine load balancers with auto scaling mechanisms. When the health checks start to fail (caused by Slowloris), the auto scaler will begin launching new instances and terminating failed instances. However, a sustained Slowloris attack will just hit the new servers. This is why the Web Application Firewall needs to detect the attack and start blocking it.

For your studies, take a look at mod_reqtimeout. This is an effective and tuneable solution for Apache for most Slowloris attacks.

[Update]

In the Amazon DDoS White Paper June 2015, Slowloris is specifically mentioned.

On AWS, you can use Amazon CloudFront and AWS WAF to defend your application against these attacks. Amazon CloudFront allows you to cache static content and serve it from AWS Edge Locations that can help reduce the load on your origin. Additionally, Amazon CloudFront can automatically close connections from slow-reading or slow-writing attackers (e.g., Slowloris).

Amazon DDoS White Paper June 2015

Upvotes: 2

Sean Mitchell
Sean Mitchell

Reputation: 457

Soloution:
If you are getting hit, I recommend you go to a provider that protects against DDoS attacks. However your best bet would be to programatically block the IP once it has been decided that it is being malicious. If you receive two large Content-Length POST requests than you should block the IP for a few minutes for suspicious activities. Many large companies are very cheap, and some of them are free for the basic package such as Cloud Flare. I use them for my company and I am beyond happy to have them!

Edit: Their job is literally just to protect you. That is it.

Upvotes: 0

Related Questions