CODEWITHSUNDEEP
javamultithreadingexecutorservice
Raju
Raju

Reputation: 1

How to track Java multithread output as per file records sequentially

I need to read bulk data file and process the data (like data validation, tracking validation failures and insert into data base etc..) using java multithreading.

I wanted to use a main thread to read data from file and process the data using multitheading and ExecutorService.

I need to track every record output once validation and insertion are done against to database.

My questions are:

Upvotes: 0

Views: 236

Answers (2)

Gray
Gray

Reputation: 116908

  • How can I track output of every record?
  • Can I put one common variable and increase the count of failure record data?

As @JBNizet mentioned, you can use an AtomicInteger for both success and failure counts. Something like the following:

  private final AtomicInteger successCounter = new AtomicInteger(0);
  private final AtomicInteger failureCounter = new AtomicInteger(0);
  ...
  threadPool.submit(new MyRunnable(successCounter, failureCounter));
  ...
  public class MyRunnable {
     private AtomicInteger successCounter;
     private AtomicInteger failureCounter;
     public MyRunnable(AtomicInteger successCounter, AtomicInteger failureCounter) {
        this.successCounter = successCounter;
        this.failureCounter = failureCounter;
     }
     public run() {
        ...
        if (worked) {
           successCounter.incrementAndGet();
        } else {
           failureCounter.incrementAndGet();
        }
     }
  }

I need to track how many records are failed sequentially. Is it possible?

One way you could do this is have a transaction number on each line and save the last failed transaction-number. Then you can either count a failure as sequential or reset the sequential counter if the current failed transaction number is not (last+1).

  if (worked) {
     lastFailedTransactionNumber = 0;
     sequentialFailureCount = 0;
  } else {
     if (transactionNumber == lastFailedTransactionNumber + 1) {
        sequentialFailureCount++;
        ... maybe log it or something?
     }
     lastFailedTransactionNumber = transactionNumber;
  }

Upvotes: 3

JB Nizet
JB Nizet

Reputation: 692003

You could pass a unique instance of AtomicInteger to every validation task, and have the task increment this AtomicInteger each time a failure happens. An AtomicInteger is thread-safe, so you won't have any concurrency issue by doing this.

Regarding the output of every record, you just need to add the output to a thread-safe collection (like a Collections.synchronizedList() or a ConcurrentLinkedQueue, for example).

Upvotes: 2

Related Questions