Huge data processing/ HPC in Java - suggest me how to begin

Question

I am thinking to work on a programming problem for which, I suppose, I will need to know a lot of advanced programming concepts. For some reasons I have decided to code it in Java - even though I am not proficient in it. So I want you to help me with suggestions, guidance, pointers to resources, books, tutorials or any generic advises that you think is pertinent.

Here is the basic nature of my problem:

1. I need to create a client-server architecture. Server supports multiple concurrent clients. Clients send it simple instructions (may be server exposes some kind of API/ runs listener on specific port), server executes the instructions and send result back to client.

2. The main job of the server is to do huge volume of data processing based on the instructions given to it. It takes data from backend database/ file systems. Data volume can easily surge up to ~ 200GB - 700GB. Data will be usually streamed to it, but it may require to hold huge volume of data in memory cache during processing (and if RAM is not enough, then page it to disk). Computations are generally numerically intensive in nature (let's say taking the inverse of a matrix)

3. The server should be able to do multithreading (I don't know what this term mean in Java, what I wish is, the server should be able to distribute the job in multiple parallel sub-processes.)

4. The server itself should be very lightweight. I Do NOT need any GUI Interface.

5. It will be great if I design it in a way so that I can integrate it later with HPC frameworks like Hadoop.

Now if I got to do this, what kind of programming do I need to learn? By the way, I have good understanding on OOP, I am somewhat familiar with Data Structures and algorithms, I know basic Java (never done any network or multithreaded programming in Java before, but have used typical oop concepts, generics, comparable interfaces etc.). I basically work in database programming, but have also done lot of C, C++, C#, Python in the past.

Given the requirement and my background, please suggest,

1. How should I begin to work on this project? What is the way to architect the project?
2. Should I create some basic API definitions first and then start working on the details?
3. Should I follow any particular design pattern? Where to learn them from?
4. What are the things I need to learn in Java and where to learn them from?
5. What is the best way to read huge data in memory? Is Java nio good solution?
6. If I instantiate a class with huge amount of data, would it work? (example, let's say I have a Vector class to represent a matrix with millions of elements and the constructor of the class reads huge data set in the memory). What's the best way to handle that?

Answer 1

First thing I would like to say that as per your explanation of the things you seem to be in a pretty good shape to use java as your server side language.

The kind of client server architecture you choose may depend on what kind of clients actually you are serving to. Would they be typical GUI or CUI based desktop clients or the web clients.

In the latter case you could use Spring Framework in a normal fashion and for the former one you could go further to explore Spring's support for Restful Web services. I would advise not to go with socket or TCP based networking solutions or use java networking.

Spring's RESTful API gives you a very cool abstraction over things like networking and multi threading even for a desktop based client. In case of a desktop client you can use JSON/XML as response and can use HttpClient library for making calls to server, which is a very cool abstraction of the underlying networking stuff.

Further up Spring's design patterns follow a very linear flow of data. A lot of your fundamental design considerations are catered by the Spring itself using Dependency Injection and Inversion of Control which are extremely simple to incorporate.

For a detailed analysis of design patterns related to specific requirements I would suggest you to read the book called Java Design Patterns: A Tutorial of Addison Wesley publications and the author is James W. Cooper.

One more thing about the API design. It would be preferable for you to first create a API specification and then go further to implement them.

Answer 2

You will want to define how the client and server will talk to eachother. The easiest way is to use established protocols such as HTTP by creating REST services that the client can call without much coding.

Most frameworks that support HTTP create several listeners that run in different threads. This gives you multi threading out of the box.

I'd suggest looking into I prefer Spring Controllers. Spring is fairly light weight.

If you want to use these frameworks, you will want to quickly find, and incorporate them into your application for compilation and packaging.

I would suggest looking into Maven for this. It's a big time saver. In particular using archetypes to create your project's folder structure, and auto download dependencies, and their dependencies.

Finally my words of wisdom. Ensure your services are singleton stateless services. This means you only create the objects once, and each thread uses the same objects. There is lots less garbage collection happening. This makes a huge difference when processing large amounts of requests.

Be careful not to use class level variables to hold state, in these services. If you do, different threads will over write each others data.