Best approach to an extendible statistics system

Question

Ok so - I need to implement a statistics/data-points/data-sources system.

I basically want to pass data periodically to the 'root' and have it process and update the relevant properties for access throughout the application - as data sources for graphing, labels, status checks etc.

I was wondering if there were some real world examples of this from users that have handled something like this in the past. I googled the hell out of this and I keep getting a mixed bag of results as to what I should do and I hate just programming and 'letting the pieces fall in place'. I need a direction.

Edit for clarity: The data sources will be:

• Local files (xml most likely),
• Local sql,
• Remotely acquired json data,
• Remotely acquired sql.

Types of subsystems (limited list, just for illustrative purposes):

• Connection statuses - both bool and text,
• Graphing/Gridview data sources,
• Processing/Predictive methods (eg probability distribution etc),
• General statistical profiles based on client/dept,
• General statistical profiles based on date/time/spans,
• more...

As I said, a lot of these sources can be used in collaboration to update segments of data, should the need arise (which it likely will). One piece of information can be used across multiple systems, but there will be times when a fetch will be very specific for one point.

I hope that made it a little bit clearer... maybe. I would like to handle all the data processing in one area if possible. It'll be easier to work with as the flow increases over time.

I wrote down some thoughts on it as brainstormed the idea.

• The observer pattern

  This pattern seems good, however it does have some drawbacks in the sense that all subs will be notified instead of selective ones. Which will force me to either check the data then process, or create multiple observable objects for each type of data and have it cascade to the subs. I definitely like how extendable this is, also allows me to sub to multiple types of data sources should the need arise. On the other hand also seems like a lot of work to get any sort of results. Paying it forward as it were.

• Strategy

  This pattern also seems relevant but in a different way. Store the processing of the raw data separately and just have a parent class that holds all the statistical information (so to speak). I like this because all information is stored centrally and the 'nodes' process it and return it. Allows for easy access and storage, however the amount of properties (unless I split it up, likely) will be huge.

• Custom events.

  Now - I guess this COULD be seen as a reinvention of the first one. But I do like the control it offers.

• A combination of observer and strategy.

  This could be weird but hear me out. So You have your observable object have the data passed to it, which cascades down to appropriate subs that will process that information for different reasons, then using the strategy from each of those subs and process the information accordingly and pass it back to the sub for storage / access.

  An example of this would be periodically withdrawing data from some kind of source; this information can be used to update multiple areas of the system (observer), but each area needs it processed a different way (strategy).

Is this logic sound or should I be looking at it a different way. I do need this to be extendable and scalable as the system could potentially be handling 'large' volumes.

Thoughts? Tried to be specific but remain on topic.

Answer 1

I ended out going with a combination of observer and strategy with a few customs events thrown in. Funny how that works. It actually works very well - lightweight, extendable and scalable on testing with 'large' (5-7gigs) of input. Desired results every time. Although assistance didn't happen I thought I would share the fact that the observer/strategy combination actually worked well.