CODEWITHSUNDEEP
javamultithreading
Raju
Raju

Reputation: 1

File reading using main thread and process data using multithreading and ExecutorService

I have a requirement to read bulk data file and process the data (like data validation, tracking validation failures and insert into data base etc..) using multithreading. I wanted to use main thread to reading data from file and process the data using an ExecutorService.

  1. Is this the correact approach? if yes, how can I do this?

  2. Also, I need to tract every record output once validation and insetion done against to database. How can I do track output of every record? Can put it one common variable and increasing the count of failure record data?

  3. Also I need to track how many records are failed sequentially?

Thanks.

Upvotes: 0

Views: 1507

Answers (3)

Drona
Drona

Reputation: 7234

I would suggest to split the file into virtual segment and allow your executors to pick and process these segments. Reading the file in the main thread would end up creating bottleneck. Allow each thread to read its segment separately and parallelly. Refer to the following post How to implement Concurrent read to a file mapped to memory in Java?

Upvotes: 2

Carmelo Foti
Carmelo Foti

Reputation: 93

Your idea is correct IMO, i would do something like:

while ((line = filereader.readLine()) != null) {
    Mycallable mycallable = new Mycallable(line);
    executor.submit(mycallable);
            }

Upvotes: 0

dbf
dbf

Reputation: 6499

Yes it is correct, but you will have an improvement if processing takes significant time (comparing to reading file), processing of pieces are independent and you have multiple cores. Read your data to any container (if it fits in RAM) and use any ExecService tutorials like this: http://tutorials.jenkov.com/java-util-concurrent/executorservice.html

Upvotes: 0

Related Questions