CODEWITHSUNDEEP
javaspring
Cromat
Cromat

Reputation: 180

Fast Multithreaded Online Processing Application Framework Suggestions

I am looking for a pattern and/or framework which can model the following problem in an easily configurable way.

Every say 3 minutes, I needs to have a set of jobs kick off in a web application context that will concurrently hit web services to obtain the latest version of data, and push it off to a database. The problem is the database will be being heavily used to read the data from to do tons of complex calculations on the data. We are currently using spring so I have been looking at Spring Batch to run this process does anyone have any suggestions/patterns/examples of using Spring or other technologies of a similar system?

Upvotes: 0

Views: 165

Answers (3)

Vladimir Tsvetkov
Vladimir Tsvetkov

Reputation: 3013

I think it's worth also looking into frameworks like camel-integration. Also take a look at the so called Enterprise Integration Patterns. Check the catalog - it might provide you with some useful vocabulary to think about the scaling/scheduling problem at hand.

The framework itself integrates really well with Spring.

Upvotes: 1

beny23
beny23

Reputation: 35048

Is refactoring the job out of the web application and into a standalone app a possibility?

That way you could stick the batch job onto a separate batch server (so that the extra load of the batch job wouldn't impact your web application), which then calls the web services and updates the database. The job can then be kicked off using something like cron or Autosys.

We're using Spring-Batch for exactly this purpose.

The database design would also depend on what the batched data is used for. If it is for reporting purposes, I would recommend separating the operational database from the reporting database, using a database link to obtain the required data from the operational database into the reporting database and then running the complex queries on the reporting database. That way the load is shifted off the operational database.

Upvotes: 1

Kevin Bowersox
Kevin Bowersox

Reputation: 94499

We have used ServletContextlisteners to kick off TimerTasks in our web applications when we needed processes to run repeatedly. The ServletContextListener kicks off when the app server starts the application or when the application is restarted. Then the timer tasks act like a separate thread that repeats your code for the specified period of time.

ServletContextListener http://www.javabeat.net/examples/2009/02/26/servletcontextlistener-example/

TimerTask http://enos.itcollege.ee/~jpoial/docs/tutorial/essential/threads/timer.html

Upvotes: 1

Related Questions