Pluggable cluster management system

Question

Problem:
We have a cluster of servers, running on Java (Tomcat). Each server exposes a lot of information via JMX. It takes a lot of time to go through JMX to make sure the cluster is in valid state.

In some cases it's enough to check a certain status in every node. In other cases the data is spread across the nodes, so some logical analysis is required. Sometimes too much data needs to be analyzed and it cannot be done manually.

The question:
Is there a cluster management system that would provide a platform for automating such tests?

Requirements:

• It has to be able to be extended by plugins (preferably written in Java), so for some complex customized test all we need is to develop a plugin with Business Logic.
• It has to provide some JMX client platform
• GUI/JMX interface for running tests and seeing results
• Scheduling
• SNMP monitoring

Answer 1

This does not satisfy all of your requirements, but you should take a look at JGroups

(A Toolkit for Reliable Multicast Communication)

Essentially, it is a java library for creating clustered nodes that can be kept in synch with each other via various protocols using mutlicast or unicast. It includes a rich set of Building Blocks to help you build the functional stack you need. Your customization of the stack can be implemented using the Building Blocks and/or your own custom state management or cluster invokable "business methods". By that I mean, you could define a business method called int getOpenPortCount() which you could then invoke, not on a single node, but on the cluster. Each attached node would then invoke the method locally and return the result of the invocation, resulting in your cluster invocation returning [effectively] a value of int[] whose length is the number of nodes on your JGroups cluster.

It has to provide some JMX client platform

I am not completely sure what you mean by this, but there is no built in JMX-Connector as such. However, you may not need it since you can communicate with individual nodes directly, or all nodes through the cluster using the JGroups API.

GUI/JMX interface for running tests and seeing results

I don't think you'll find anything like this, but, since you would essentially be using pure Java, you could use a combination of the JGroups API, the JMX API, jUnit (or TestNG) and an eclipse based test-runner which provides a fairly decent testing harness and visualizing UI.

Scheduling

You can schedule events to be executed across the cluster using the JGroups TimeScheduler, Quartz or simply use ScheduledThreadPoolExecutor.

SNMP monitoring

JGroups supports JMX for monitoring, which in turn can be bridged through the JVM's SNMP agent. This jboss specific link will give you an idea of how this can be implemented.