CODEWITHSUNDEEP
javamultithreading
user550738
user550738

Reputation:

how to maintain a list of threads?

I have hundreds of files to process. I do each file one at a time and it takes 30 minutes.

I'm thinking I can do this processing in 10 simultaneous threads, 10 files at a time, and I might be able to do it in 3 minutes instead of 30.

My question is, what is the "correct" way to manage my 10 threads? And when one is done, create a new one to a max number of 10.

This is what I have so far ... is this the "correct" way to do it? 

public class ThreadTest1 {

    public static int idCounter = 0;

    public class MyThread extends Thread {

        private int id;

        public MyThread() {
            this.id = idCounter++;
        }

        public void run() {
                    // this run method represents the long-running file processing
            System.out.println("I'm thread '"+this.id+"' and I'm going to sleep for 5 seconds!");
            try {
                Thread.sleep(5000);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
            System.out.println("I'm thread '"+this.id+"' and I'm done sleeping!");
        }

    }

    public void go() {

        int MAX_NUM_THREADS = 10;

        List<MyThread> threads = new ArrayList<MyThread>();

            // this for loop represents the 200 files that need to be processed
        for (int i=0; i<200; i++) {

            // if we've reached the max num of threads ...
            while (threads.size() == MAX_NUM_THREADS) {
                // loop through the threads until we find a dead one and remove it
                for (MyThread t : threads) {
                    if (!t.isAlive()) {
                        threads.remove(t);
                        break;
                    }
                }

            }

            // add new thread
            MyThread t = new MyThread();
            threads.add(t);
            t.start();

        }

    }

    public static void main(String[] args) {
        new ThreadTest1().go();
    }

}

Upvotes: 0

Views: 1185

Answers (2)

John B
John B

Reputation: 32969

I would suggest using Camel's File component if you are open to it. The component will handle all the issues with concurrency to ensure that multiple threads don't try to process the same file. The biggest challenge with making your code multi-threaded is making sure the threads don't interact. Let a framework take care of this for you.

Example:

 from("file://incoming?maxMessagesPerPoll=1&idempotent=true&moveFailed=failed&move=processed&readLock=none")
       .threads(10).process()

Upvotes: 0

pomkine
pomkine

Reputation: 1615

You can use ExecutorService to manage you threads.

And you can add while loop to thread run method to execute file processing task repeatedly. Also you can read about BlockingQueue usage. I think it will fit perfectly to allocate new files (tasks) between threads.

Upvotes: 2

Related Questions