Reputation: 6125
I'm trying to be bad.
I have a Python Flask application that's used by a lot of other services, and I want to simulate it being unreliable. I'm trying to implement something vaguely like a lightweight Chaos Monkey by causing this critical Flask application to randomly disconnect incoming requests, so that we can observe what happens in other applications when their requests are unpredictably dropped.
(Why? I have real-world servers running these Flask apps in industrial environments with extremely poor wireless network connectivity. I need to demonstrate that the rest of the system talking to these apps can reasonably recover even when the network drops most of the packets passing through it. Statistically killing requests on an otherwise-stable connection seems like a pretty good technique for doing so.)
The first part of the code is pretty straightforward: Add a before_request
hook that randomly chooses whether this request is going to fail:
PSEUDORANDOM = # ...custom repeatable RNG here...
THRESHOLD = # ...configurable reliability setting from 0 to 1...
@flask.before_request
def before_request_callback():
if PSEUDORANDOM.random() > THRESHOLD:
# ...what to do here?
Importantly, I don't want to return a 500 Internal Server Error
or a 4xx Sorry
response. I don't want any response to be sent back to the caller at all — I want to be able to write socket.close()
— or better yet, socket.shutdown()
. I want to simulate a crash or a major application failure or a network failure.
But I can't figure out how to get access to the underlying socket itself; it's wrapped in enough layers of objects that I can't find whichever one is actually holding it. I've read through the source code of the Request
classes in the werkzeug
package, and it has almost everything you could want — except the socket. The closest I can get is the Request.stream
property, but invoking its close()
method doesn't seem to do anything at all. Scouring StackOverflow questions for the last half hour hasn't yielded anybody else doing this either, apparently.
So how do I access the underlying socket so I can make this Flask application randomly appear to fail?
Upvotes: 0
Views: 891
Reputation: 46
This article (still reachable via the wayback machine) demonstrates a way of accessing the raw socket while processing a request. Shutting down the socket will then trigger a connection timed our error on the client side.
from flask import Flask, request
import socket
import werkzeug
class ErrorTestingWSGIRequestHandler(werkzeug.serving.WSGIRequestHandler):
def make_environ(self):
environ = super(ErrorTestingWSGIRequestHandler, self).make_environ()
environ['socket'] = self.connection
return environ
app = Flask(__name__)
@app.route('/')
def main():
return '<a href="/error">Fail now!</a>'
@app.route('/error')
def error():
request.environ["socket"].shutdown(socket.SHUT_RDWR)
return "not shown"
if __name__ == "__main__":
app.run(request_handler=ErrorTestingWSGIRequestHandler)
Upvotes: 3