CODEWITHSUNDEEP
javatcpmessage-queue
m1h
m1h

Reputation: 21

read tcp stream with java while writing to database

I am not sure what the best solution for following situation is:

My Java-program is permanently reading from a tcp-stream. At the same time it is neccessary to persist these data to a database. The amount of data which should be written to the database could differ

I have read a lot of about message-queueing-systems and so on. In detail, my solution would consider using a LinkedBlockingQueue. Thereby, there are two threads: a) Starting a producer-threat which will perform reading from the tcp-stream b) Starting a consumer-threat which will write the (parsed) data from the stream to the database

The (example-)code looks like following:

Main.java

public static void main(String[] args) {
        LinkedBlockingQueue queue = new LinkedBlockingQueue(50);
        Producer producer = new Producer(queue);
        Consumer consumer = new Consumer(queue, producer);

Produer.java

public class Producer implements Runnable {
  private LinkedBlockingQueue queue;
  private boolean running;

  public Producer(LinkedBlockingQueue queue) {
        this.queue = queue;
        running = true;
    }

 @Override
  public void run() {
    //read TCP-Stream here and save parsed messages to queue
  }
  public boolean isRunning() {
        return running;
  }

Consumer.java

public class Consumer implements Runnable {
    private Producer producer;
    private LinkedBlockingQueue queue;

    public Consumer(LinkedBlockingQueue queue, Producer producer) {
        this.queue = queue;
        this.producer = producer;
    }
   @Override
   public void run() {
       //insert data into database here
       if(producer.isRunning()) {
          //while producer is running, data needs to be inserted to database
       }
   }

Is this a solution you would recommend to use? Or do you know better solutions?

Thank you!

Upvotes: 0

Views: 114

Answers (1)

peterh
peterh

Reputation: 19235

Your own suggestion is very good.

Ultimately what you are trying to solve is the problem of back pressure, i.e. if you are receiving data faster than you can write them to database. This may happen just because there's a lot of data arriving or simply because your destination is temporarily unavailable. Either way it is a situation you need to deal with.

In your proposed solution this is handled by an in-memory staging area (= your queue). As long as you have enough memory and you are not too concerned about loosing data on power outages then the in-memory strategy will work fine for you. A burst will be absorbed by growing memory in your Java application. Not a problem per se but remember that when your queue is finally drained the JVM GC will kick in and release the memory again from the JVM heap. But from the outside, i.e. from the point of view of the OS, the memory will possibly never be released. The JVM is very, very conservative about releasing memory back to the OS. Again, in most cases this is not a problem.

If you have more strict needs then you need to consider a more "robust" staging area instead of RAM, for example local disk. From my experience your proposed solution fits 95% of use cases.

Upvotes: 2

Related Questions