CODEWITHSUNDEEP
statisticsmonitoring
Malcolm Box
Malcolm Box

Reputation: 4036

What are good ways to create real-time stats for high-load webservers?

Say I have a bunch of webservers each serving 100's of requests/s, and I want to see real time stats like:

Or in general for a bunch of timestamped events, I want to see real-time derived statistics - what's the best way to go about it?

I've considered having each GET request update a global counter somewhere, then sampling that at various intervals, but at the event rates I'm seeing it's hard to get a distributed counter that's fast enough.

Any ideas welcome!

Added: Servers are Linux running Apache/mod_wsgi, with a Python (Django) stack. Added: To give a sense of the event rates I want to track stats for, they're coming in at over 10K events/s. Even incrementing a distributed counter at that rate is a challenge.

Upvotes: 1

Views: 287

Answers (4)

Graham Dumpleton
Graham Dumpleton

Reputation: 58523

You might like to help us try out the beta of our agent for application performance monitoring in Python web applications.

http://newrelic.com

It delves more into the application performance rather than just the web server, but since any bottlenecks aren't generate going to be the web server, but your application then that is going to be more useful anyway.

Disclaimer. I work for New Relic and this is the project I am working on. It is a paid product, but the beta means it is free for now with all features. Later when that changes, if you didn't want to pay for it, their is still a Lite subscription level which is free and which gives you basic web metrics reporting which still covers some of what you are after. Anyway, right now would be a great opportunity to make use of it to debug your performance while you can.

Upvotes: 1

Jack
Jack

Reputation: 15872

Look at using WebSockets, their overhead is much smaller than a HTTP request, they are very well suited to real-time web applications. See: http://nodeknockout.com/ for Node based websocket examples.

http://en.wikipedia.org/wiki/WebSocket

You will need to run a daemon if you want to run it on your apache server.

Also take a look at: http://kaazing.com/ if you wan't less hassle, but are willing to fork out some cash.

http://websocket.org/img/poll-ws-compare.gif

Upvotes: 1

Steve B
Steve B

Reputation: 37660

On the Windows side, Perfmonance monitor is the tool you should investigate.

As Jared O'Connor said, you should precise what kind of web server you want to monitor.

Upvotes: 0

Jared O'Connor
Jared O'Connor

Reputation: 473

Virtually all good servers provide this kind of functionality out of the box. For example, Apache has the mod_status module and Glassfish supports JMX. Furthermore, there are many commercial packages for monitoring clusters, such as Hyperic and Zenoss.

What web or application server are you using? It is difficult to provide a solution without that information.

Upvotes: 1

Related Questions